"Transformirror" by Daito Manabe and Kyle McDonald, produced by Rhizomatiks
This installation art piece is a fusion of real-time image-to-image transformation using SDXL-Turbo and the application of Stable Audio's Text-To-Music technology. When visitors are captured by a webcam, their images are instantly transformed and displayed on a large LED screen in a varied visual format. Additionally, this work incorporates 100 prompts generated by ChatGPT, used for transforming people's images and creating background music. It offers a glimpse into the creativity of AI and the mysteries of text interpretation. This installation allows visitors to experience the strange and enchanting visual and auditory textures produced by the cutting-edge real-time Image-To-Image and Text-To-Music technologies.
Transformirror processes video at 30fps at 1024x1024.
We use a square center crop from a 1080p webcam and and create batches of 4 sequential JPG frames that are sent to two workers using ZeroMQ over a local Gigabit network.
The workers decode and process the frames using SDXL-Turbo with stable-fast on a 4090 using Hugging Face diffusers, plus some custom code to keep seeds fixed within a batch.
The workers then encode and send back results to the manager in around 250-300ms. The manager handles de-jitter of the returned frames. The total latency of the system is around 500-800ms.
For music generation, we use Stable Audio, employing the same prompts used in the Image-To-Image process.
For our Christmas project, The 100 prompts utilized in the installation were also generated using ChatGPT.We provided ChatGPT with special instructions to create prompts that transform human figures into images reminiscent of Santa Claus or Christmas themes. These prompts were conceived and created in English, totaling 100 different variations. Examples include themes like "Dystopian Christmas" or "Party People Christmas," offering a diverse range of variations. This approach enriches the visual experience of the installation and enhances the festive atmosphere of Christmas.
While we utilized prompts specifically for Christmas in this instance, the technology allows for the adaptation of prompts to suit various events, parties, countries, or environments.
This adaptability enables the installation to automatically generate an infinite array of interactive experiences tailored to the ambiance and theme of the occasion.
For example, it can provide unique visual experiences for Halloween, international festivals, or themes inspired by natural environments. This flexibility makes the installation even more engaging and dynamic, offering visitors a continually fresh and surprising experience.