A tool for drawing and editing images in mixed reality using Unity 3D and OpenAI API, making it easier for designers to create and edit images from doodle in unity
Game Engine: Unity 3D+ Oculus integration | Images: Photoshop | Models: Maya, Blender: Gen AI: Openai API
For this demo, I have created a novel interaction method for swapping between brushes using finger pose detection and to adjust size of the brush, raise and pinch the other hand and drag through space.
While drawing, using controllers might not be the best idea as grabbing a big controller in hands changes the feel of holding a small brush. Hand tracking based brushes seems quite easy to use in space for movement.
Below is the demo involving integrated gen AI to create art from doodle and editing followed by creating different variations of your final output.
There are three parts to this project:
1. Image generation from doodle and input prompt.
2. Edit the generated image by erasing the pixels and using it as a mask for the API with the original image.
3. Create and explore variations to your final output.