🅰️i Hank CuttleBarf - Speaking Intelligable Through Motion Capturing Video Driving His Appearance
0#motioncapture #drivingvideo #ai
🅰 ️ i Hank CuttleBarf using img2video and txt2video img2img it was possible to make a sketch I made come to life in a photorealistic manner. The animations the video creating LLMs are capable of are quite limited. Often speech is mimicked and absolutely nonsensical.
I wanted Hank to be able to talk reasonably intelligently and have is face / head have the appropriate expressions. A solution was to create a video of my own head saying what I wanted Hank to say. Then extract motion capture data out of it and map that on a model of Hank and synchronize it with the audio. The result is a video my motion capture data inhabiting a model of Hank.
This is the motion data attached to a still stationary model of Hank, it might also be possible to map the motion capture of my face (eyes, mouth, eyebrows) onto an existing video and have a moving about Hank say the things he needs to say. Uncanning and quite eerie to see.
This is done with python, a set of LLM tools on my own PC at home. It has a Ryzen9 3900x 12 core 24 threads CPU and an Nvidia RTX 4060Ti with 16Gb of RAM. A meager machine when it comes to creating a.i. but good enough for me. The input video needs to be 15fps on my system otherwise the LLM starts to lag animating the video with my motion data.
The tales of Hank CuttleBarf continue…
Also available on VHS 📼
Credits:
Concept, art design, video editing scripting by Mark Vergeer
The individuals depicted in this video have all been generated from pencil drawings to photorealistic avatars and have been used in Bing, ChatGPT and locally run A.I. models through Pinokio and Stable Diffusion, Easy Diffusion for Free.
Audio tracks used licensed from Epidemic Sound.
The “person” in this video does not exist in real life. Any likeness to a real person is absolutely coincidental. I sketched an older balding man in a suit wearing a tie ala car-salesman that is a little overweight. And i am quite surprised at how realistic looking the man ended up looking. His name will be “Hank Cluttlebarf”
ChatGPT created his likeness from the sketch I did using this prompt:
1. “Create a series photorealistic images from this art but keep the original shape proportions and line lengths. Make it look like this could actually exist in the real world with realistic textures. You can add realistic shadows. Don’t change any of the form or details of what was created. Show a distinct series of facial expressions. “
2. “Create photorealistic images from this art but keep the original shape proportions and line lengths. Make it look like this could actually exist in the real world with realistic textures. You can add realistic shadows. Don’t change any of the form or details of what was created. “
Thanks for watching - LactobacillusPrime
My Twitter: NOT - it's rather dead with it's fake ✔ ️ and rather toxic communihttps://twitter.com/LactobacillusPp
My Faceboohttps://www.facebook.com/LactobacillusPrimeRetroGamingg
My Instagrahttps://www.instagram.com/lactobacillus_primee
NOTICE:
"Copyright Disclaimer Under Section 107 of the Copyright Act 1976, allowance is made for "fair use" for purposes such as criticism, comment, news reporting, teaching, scholarship, and research. Fair use is a use permitted by copyright statute that might otherwise be infringing. Non-profit, educational or personal use tips the balance in favor of fair use."