C4D OpenPose Rig for Stable Diffusion [3D C4D → AI Stable Diffusion]

OpenPose Custom Character Rig in Cinema 4D scene (3D) using any kind of rigged character animation (Rokoko, Mixamo and more...), use it to create AI generated images/videos (with Controlnet in Stable Diffusion).

This is a tool I've been using in personal/professional productions. It allows me to get the most precise control over character animation and directing over AI generations. I feel that this workflow/setup allows a full creative control over AI. Can't wait to see what people will be able to do with this :)

WORKFLOW

• Stable Diffusion (AI / Automatic1111) → txt2img / Deforum, Frame Interpolation, Upscale
• C4D (3D / v.2024.4.1) → Custom Open Pose Rig + Viewport renderer
• Octane Material (3D) - Legacy friendly
• Rokoko Studio (Motion capture / release-2.4.4.7 ) → motion capture suit
• After effects / Media Encoder (Montage, Editing)

WHAT'S IN THE PACKAGE

• 1 C4D Scene file
• 1 Documentation file
• Stable diffusion image generation .txt file

SUMMARY

This project is based on the retranscription of a color code to render a character animation using 3D software (here C4D) that is ready to use in Stable Diffusion. The name of resource is "OpenPose COCO 18 Color Reference Chart" by U/GBJI (cf. spreadsheet).

/!\ DO NOT CHANGE THE COLOR OF THE LAYER AND MATERIALS / THEY ALL RESPOND TO A #HEX CODE. IF YOU DO THE OPENPOSE RIG WON'T WORK PROPERLY. /!\

OpenPose Bone color scheme → Cinema4D (2024.4.1) custom rig setup. Update: the C4D interface screenshot is an old C4D file with wrong colors that has been corrected :)

HOW TO MAKE IT WORK ?

SETUP C4D FILE:

• Be sure that the .C4D file has those "Color Management'' Parameters. (In "Edit" → "Setting Scene" or keyboard shortcuts to get this "Project" window is: Ctrl + D).
• Also, you NEED to have your viewport shading mode in Constant Shading "Display" → select "Constant Shading".

  

IMPORT CHARACTER ANIMATION SOURCE + SETUP THE OPEN POSE RIG:

• In the scene, import the character animation you would like to transfer to the OpenPose Rig (it can be from Mixamo, Rokoko...etc).
• The OpenPose Rig is composed of Spheres (Joints) and Sweep objects (Pairs Bones) that you can find in the "Objects tab", under the null named "Null:OpenPose_Rig".

• Select the Sphere Objects (named from "0" to "17") one by one and click on their Constraint tag. Now, in the "Attribute tab" → "Clamp" you can modify the "target" to the relevant bone of the animation rig. (ex: 2 →Right shoulder). You don't need to touch the Sweep objects (Pairs Bones), as they are already set up.
• Put the distance parameter of the Constraint Tag to "0 cm".
• From this point, the OpenPose Rig should stick and follow the Animation Rig you've just imported.

  

UPDATE - OCCLUDER OBJ:

• I've added a "Occlusion" Extrude Object (from a spline) in the "Null:Animation_Rig" as a test to get better results when the character rotates; it can mess up the AI generation.
  

RENDER PROCESS (IMAGES AND VIDEOS) + RENDER SETTINGS:

• Double check that your viewport is in "Constant Shading" mode and the Color Management parameters → cf. first part of the doc "SETUP C4D FILE".
• Be sure that you have a "Sky" Object with a black material (HEX #000000), it will create the black background that's needed for the AI Generation.
• "Render settings" → Select "Viewport Renderer".
• "Viewport Renderer Tab" → Copy the "Basic Properties" and "Filter" parameters as the screenshot.
• Be sure not to lock any Layers of "Null:OpenPose_Rig" otherwise it will appear gray.
  

  

QUICK STABLE DIFFUSION INTRODUCTION:

I've used this setup in Stable Diffusion Automatic 1111 (local), in SDXL and SD1.5 / txt2img and Deforum. Here I won't go into the details step-by-step process but I share screenshots of the way I configure Stable Diffusion to get to this result. Take note of the yellow squares, they are important parameters.

Stable Diffusion

• Check Yellow
• Uncheck Red

ControlNet (v1.1.452)

• OpenPose "prepocessor: none" and "Model: "thibaud_xl_openpose [c7b9cadd]"

In the documentation, I'll add the .txt file of this image, so you'll have the full list of parameters.

SHOWCASE "Shaping AI output with 3D & VFX!" (2024.03)

Here's a research I've worked on with this technique, I added more ControlNet from 3D export passes to improve the result and control over the AI generation:

• ControlNet: T2I-Adapteur - OpenPose - Depth
• 3D Pass for VFX: Z-Depth - OpenPose Rig - Material ID - Object ID

Please consider supporting this with a fair price as it takes a long time and effort to build that kind of tool. Don't hesitate to tag me on your own projects and experimentations so I can see how far this tool can be pushed!

Special thanks to Alexandre Rivaux (https://www.instagram.com/arivaux/) for his help in learning how to tame AI and Florian Coppier (https://www.instagram.com/pti_lofi/) for helping me find the right way to link AI and 3D.

If you have any questions, you can contact me by email: theo.rocq@gmail.com

📱 Follow me on instagram : https://www.instagram.com/theorocq/