Build an Image Segmentation Deep Learning Model with Python from Unreal Engine 5 Generated Images

Procedural generation can be used to create randomized versions of data. This is useful for when you need to generate large datasets quickly.

In this project, we use Unreal blueprints to generate 3D scenes in the Unreal Engine. The generated 3D scenes include grass, rocks and bushes in varying positions. We used blueprints to enabled a segmented view of the scene, where different objects like grass, a rock, a bush are painted in a unique colour.

The purpose is to have a realistic 3D scene and a segmented 3D scene, so that we can train a neural network to take a 3D scene and create a segmented 3D scene. Image segmentation is one type of computer vision, used for object detection, image recognition and more. We use another blueprint to take screenshots of the scene and its segmented version. This is the basis of the dataset we are creating. We can move around in the scene and capture many screenshots of a 3D world from multiple angles.

The power of scripting allows us to generate tons of image data that mimics the real world. The result is a dataset of images of a 3D world. We build a supervised neural network and train the model on the image data. The model learns how to take an image of a 3D world and generate a segmented version of that image.