Advanced RGB + Depth Reference Editing in One Pass

Summary

Pixel’s announcement explores an advanced video workflow that combines an RGB reference for character appearance with a depth reference for geometry and camera position. It covers character replacement, action transfer, scene preservation, audio-driven lip sync, three workflow variants, and how per-frame depth extraction can reduce drift.

DEV Community

Reference Edit Advanced: RGB + Depth Composite Reference in a Single Video Pass

Most reference-edit workflows use one reference. This one uses two simultaneously: an RGB reference for appearance and a depth reference for geometry. The combination unlocks doing four things in a single pass that normally need four passes. What the two references each carry RGB reference — who the character is. Face structure, hair, costume, colour. Depth reference — where everything is. Camera position, subject distance from lens, scene layout. Using both means the new character inherits appearance from one and spatial position from the other. Using only RGB, the character lands wherever the model feels like putting it. The four jobs in one pass Character replacement — the subject becomes the reference character. Action inheritance — the motion of the source performance transfers. Scene and camera preservation — because depth is conditioned, the camera does not wander and the set does not rebuild itself between frames. Audio-driven lip sync — when audio is supplied, mouth movement is driven by it. Doing these separately is how most pipelines end up with a character that has the right face but the wrong head angle, because each pass re-decides. Three variants File Use Standard Default settings Silent (无声版) No audio conditioning Face Refine Extra pass on the face after the main render The Face Refine variant is worth the extra time whenever the character is in frame larger than about a third of the image height. At smaller sizes the refine pass adds nothing. Where it breaks Depth conditioning is precise but rigid. If the source performance moves the subject closer to camera in a way the depth map does not describe (because the depth was extracted from a single frame), the composite drifts at that moment. The fix is a per-frame depth extraction, which is much more expensive — and the episode shows the difference. Files HF: https://huggingface.co/AI-KSK/minimax-h3-role-swap-reference-edit ModelScope: https://modelscope.cn/models/Sunjian520/minimax-h3-role-swap-reference-edit Video walkthrough (Chinese): https://youtu.be/UespHFT9cjc Bilibili: https://www.bilibili.com/video/BV1VjaD6jEU3/ Notes Three ComfyUI workflow exports; H3 weights and the depth extractor must exist locally. Tutorial material, not a product. Independent AI-KSK release; not affiliated with the model authors.
Advanced RGB + Depth Reference Editing in One Pass
Powered by Readybot.io

The latest from Pixel

Incident Response: Look Beyond the Red Alert

Pixel’s DEV Community announcement explores how teams can fixate on one failing service while overlooking the real cause of an outage, such as a full …

Sach: AI Fact-Checks WhatsApp Forwards in India

Pixel’s announcement introduces Sach, a tool that checks forwarded messages in English, Hindi, Telugu, and 14 other Indian languages against official sources. It gives cited …

Java 25 Removes SecurityManager: What to Fix

Pixel’s announcement explains how Java 24’s JEP 486 permanently disabled the SecurityManager, breaking test workarounds that intercept System.exit. It recommends returning exit codes or throwing …