Pexote Jack

A 100% AI Film Project

When deciding on a project for my CG Pro class, I wanted to make something completely with AI, without viewers knowing it was made completely with AI.

Workflow:

  • Backgrounds - mostly FLUX Dev in Comfy w/ lots of outpainting

  • Character animations - first frames w/ FLUX Kontext, composited onto chroma key backgrounds then animated using JSON prompts w/ Google VEO 3, w/ lots of still-capture to use as next I2V frame to continue animation

  • Vertical screen animations - All Midjourney, w/ same still-capture process for the ones over 20 seconds.

  • Voice & sound effects - All Eleven Labs lip synced to Veo's outputs.

  • Upscaling with Topaz Astra

  • Compositing - Davinci Resolve & Fusion

The Making of Pexote Jack

Screenshot of a computer screen showing a desktop workspace with a dark theme. The window is titled 'Pexote Jack AI Project' with various folders and files listed on the left side, including subfolders and individual files related to an AI project. The right side displays a list of document titles and snippets, such as 'Pexote Jack JSON prompts,' 'JSON prompt creation,' and instructions involving JSON and ComfyUI.
Screenshot of a digital project management interface showing project details for 'Pexote Jack,' a dark animated comedy series. The interface includes project names, files, and task status, with a sidebar listing various creative projects, and a search and help section.

Episode Synopsis:

A 1930s cartoon character shows up for a meeting thinking it's his big comeback, but the exec reveals they've already created an optimized AI version of him using his past films.

Project Setup

To get started, I setup a Project within various LLMs to help manage prompting, as well as technical help. This was incredibly important, as prompting became more complex as the project went on.

I started off with a Project in Claude, which was super helpful to have the instructions compiled and the project all in one place.

As time went on, I began to migrate over to ChatGPT for prompts, etc. I found the face that ChatGPT can access the internet in real time is SUPER helpful for AI, where things change constantly. For example, giving ChatGPT instructions to go find prompting instructions for different models, like FLUX Kontext, and to use those instructions when writing prompts for me was super helpful.

Various handwritten and typed notes and scripts arranged on a wooden table.
A screenplay page depicting scenes in the Viral.ly office and lobby, featuring a character called Jack, a cartoonish man in 1930s style, and a female robot at a mobile kiosk.
Screenshot of a digital media platform interface showing a list of video projects titled 'Pexote Jack,' with episodes labeled V9, V8, V7, and others, including dates and a share button.

Screenplay:

I've played around before w/ LLMs and screenwriting, but having Projects set up in various LLMs was the power I had been lacking in this department before. Also, I started as a writer, so I'm less reliant to lean on AI for screenwriting in general.

However, one of the really fun experiments with this episode was this - I gave each LLM a different persona:

Claude: Charlie Kaufman (existential)
ChatGPT: Joel & Ethan Coen (dark humor + intricate plot)
Super Grok: Dan Harmon (unhinged w/ pop culture references)

Then I gave them the instructions to rewrite my original script using UCB's sketch comedy principals, and give me back the rewritten draft. Then, I took each draft, and had the different LLMs give "notes" / critiques to each others' drafts. Then, I kinda did that process a few times, then made my own notes, and rewrote the whole thing from scratch.

Overall, I'd say it was pretty effective - there might be like 3 or 4 lines of dialogue that came from the process, which is alot in a 5 minute sketch comedy, (and a screenriter originally, dialogue is something writer's are generally super precious about.)

ComfyUI:

I have nothing but good things to say about Comfy. It's incredibly powerful, yet simple once you understand it. I recorded myself making this project each day (forgot occasionally,) so I have HOURS of footage of me making Pexote Jack, the majority of which is in Comfy. Once I'm finally finished with my the final cut, I want to create a "Making of" video where I can explain some things more in detail.

But put simply, there are simply too many things from Comfy to even begin to share here, so more on that soon. But above is a video of me sharing a specific FLUX workflow with the class, utilizing different LoRAs together with a Canny Edge control net for character design development.

Pexote Jack - a cartoon jackalope character with large antlers, a big smile, and wearing shorts and shoes.
Pexote Jack - A cartoon black-and-white character with antlers, large eyes, a smiling face, and a mischievous expression, wearing shorts and gloves.
Character design of Pexote Jack in various poses, showing a 180-degree turn, with antlers, large eyes, gloves, shorts with an X on the front, and large shoes.

Character Design:

I had already created the titular character in 3D for a short I made in Unreal Engine, (you can see it on my Director page,) but that was made with AI as well. I started with a sketch I made in Procreate, turned that into a 3D model with Kaedim, then rigged it, and finished off the black and white design in Unreal Engine.

But he was always meant to be 2D, cel style. And now with AI, I felt I could make that happen. So using the 3D model, and trying hundereds of different iterations, I finally landed on a design that became the hero design above.

And once I had the hero design, I was off to the races using FLUX Kontext for character sheet creation.

You can see the full character design exploration in this slides deck I put together.

A large window overlooking a city skyline with tall buildings, green hills, and mountains in the distance.
Empty modern room with large floor-to-ceiling windows overlooking downtown Los Angeles with city skyscrapers and palm trees under a clear blue sky.
A gray door with a black handle closed in a white minimalist room with a polished floor and ceiling lights.
A cartoon character with a deer's head and a human body sitting on a white chair against a green background.
A JSON code snippet describing a character with raised eyebrows, blinking, and a camera focus, likely a cartoon or animated character from the 1930s rubber hose era.
Colorful 3D cartoon jackalope character with big antlers, wearing shorts and boots, standing with arms outstretched and smiling.
A woman in a professional black suit and blue shirt, shown in multiple pose variations from front, side, back, and three-quarter angles, with a final colored illustration.
City skyline viewed through large glass windows from inside a building, with a night sky overhead.
Outline view of an indoor balcony with a glass railing, overlooking a city skyline with palm trees at dusk.
Outline drawing of a conference room with a closed double door at the center, high ceiling with recessed lighting, and blank walls.
A JSON code snippet for a performance scene with a character named Pexote Jack, seated in the same pose, in front of a camera and chroma background, dressed in classic 1930s rubber hose cartoon style.
A cartoon jackalope character with a big smile, wearing brown shorts, and standing against a plain white background.
A spacious room with large floor-to-ceiling windows showing a city skyline with buildings and a few tall palm trees outside. The room has marble walls, columns, and flooring with reflections of the windows and cityscape.

Backgrounds:

To develop a 360 degree view of the main office setting for shot / reverse shot use in the film, the workflow I eventually landed on was text-to-image prompting in Midjourney for photorealistic imagery that matched the location, which were then used for control nets to generate 2d cel style animation backgrounds.

That, and like 2K iterations, and ALOT of FLUX Outpainting.

A cartoon deer with a big smile, wearing shorts and gloves, standing on a green screen background.
Cartoon reindeer character with black body, white face and hands, large white shoes, antlers, and a black shirt with a gray X, standing against a green background.
A spacious, bright indoor area with large floor-to-ceiling windows showcasing a cityscape with skyscrapers and mountains, complemented by red-roofed houses and tall palm trees outside. The interior features white marble columns and a polished white marble floor reflecting natural light.
Empty lobby with marble walls and flooring, double wooden doors in center, and ceiling with recessed lighting.
Cartoon woman with black hair in a business suit sitting on a white bench against a green background.
Screenshot of JSON code for describing a cartoon animation of a jackal, including close-up shots of its face and movements.

Image 2 Video & Google Veo 3 JSON Prompting

Once I got into a good workflow for Image 2 Video, I'd start with the character on a chroma key background, which I've been separating out in Davinci Resolve / compositing.

After trying Midjourney, Runway, and Google Veo, I landed on Veo to be the best for this project. Mostly this was due to the power of JSON prompts, which seem to really help Google be specific on what it is you want each character to do.

I'm not sure why JSON prompting is kinda getting scoffed at? Maybe because it's reminiscent of the "prompt engineer" fad from a few years ago, but I found JSON prompting to be extremely powerful w/ Google Veo 3. Notably, for now, Midjourney can't digest prompts this long and specific, which really limits what that platform can do.

Animated cartoon reindeer character sitting on a chair with a bright green background.

Upscaling:

Just now getting into Upscaling w/ Topaz labs, but it does a pretty good job on Jack, except his "pie cut" eyes got tanslated into different shapes in the upscaling process, which is too specific of a character design for me to use. For Marrissa's character, I'm expecting it will be much better, and am planing on doing as run through of all her final clips w/ Topaz Astra.

Once I'm done with the final composite, I'm planning on experimenting with upscaling the entire scene when it's finished.

A person with a beard and dark hair sits at a table in a dimly lit space, wearing a dark shirt. They appear to be smiling or thinking, with hands near their mouth. The background has a warm glow from lamps or lights.

Dialogue & Audio:

Eleven Labs has come A LONG way. Utilizing LLMs prompting, I was able to design a custom voice for Jack - that I love - and with a few small tweaks in Davinci Resolve, I'm able to up the pitch to be more cartoony, etc.

Also notable is the Sound Effects Eleven Labs is able to generate. Normally, things like sound effects are one of those things that requires TONS of time to find the right ones, get them royalty free, etc. So being able to just generate them is a huge, huge, huge time saver.

Anyway, using the audio generated from Google Veo was super helpful for lip syncing the animations, which is another huge time saver in post.

The screenshot shows a video editing software interface with multiple panels and tools. In the upper right corner, a man with a beard, wearing a cap and white shirt, is visible in a small live video feed. The main workspace displays a timeline at the bottom with nodes and connectors, and the central area features three visual scenes: one with a woman in a white dress walking into a church, and two large screens showing the same scene on a blue wall, with editing and positioning controls on the right.
Screenshot of DaVinci Resolve Studio 20 video editing software interface with a timeline of a digital project titled 'PexoteJackComposites', showing video clips, a preview window with characters, and a node graph at the bottom for color grading or effects.
A scene from an animated film editing software showing a sketch of a reindeer standing on a bench, with a background of a door and walls in a studio setting.
Computer screen displaying a 3D modeling software with a crane model, green background, and various editing tools. Inset video of a man with headphones holding his head, appearing focused.

Compositing and Post:

Sine I decided to use a Chroma Key approach, there's obviously a lot of compositing that goes on. I'm really comfortable in Davinci Resolve, so did almost everything in Fusion.

Final Pixel

A cartoon-style hallway with bright blue walls and ceiling, decorated with framed animated posters. The shiny tiled floor reflects the pictures and a cartoon character standing in front of an elevator door.