ControlNet and img2img over your clay renders, the honest workflow

The problem with pure text to image AI for architecture is that it hallucinates the geometry. You describe your building, the model gives you a building that looks like yours but is not yours. For a concept board that is fine. For anything the client compares back to the plans it is a disaster. ControlNet, and specifically the depth and canny conditioning models, is the one AI workflow I use that keeps the geometry honest.

The idea is simple. You render a clay pass out of Blender or 3ds Max with no materials, just white surfaces and basic lighting. You feed that render into Stable Diffusion XL through ControlNet with a depth or canny conditioning image derived from the same render. The AI generates a photoreal image that follows the exact geometry of your clay pass but adds materials, lighting nuance, atmosphere, and detail. Because the geometry is locked, the client cannot spot drift between the drawings and the render.

The tooling stack I use is ComfyUI with SDXL 1.0 base, an architectural LoRA fine tune, and ControlNet Depth SDXL. Automatic1111 works too but ComfyUI gives you the node graph control that matters for repeatable production work. If you have never touched ComfyUI, budget two evenings to learn the interface. After that the payoff is significant.

The clay render itself matters. A flat white matte clay with soft ambient light and one directional key light gives ControlNet the depth information it needs. Do not render your clay with a proper skylight setup because the shading confuses the depth extraction. Just white matte, ambient, and key. Render at the same resolution you want your final output, typically 1920 by 1080 or 2048 square. Higher resolution ControlNet inputs give sharper conditioning.

The two ControlNet models I use are Depth and Canny. Depth uses a depth map of the scene and preserves the volumetric layout. Canny uses an edge detection pass and preserves the linework, which is better for facades with heavy detail like mullions, panel joints, and cornices. On most interior work Depth is enough. On exterior work with articulated facades I run both, blending them at 0.6 and 0.4 strength respectively.

The denoise strength setting is the single most important knob. Low denoise, around 0.2 to 0.3, keeps the render extremely close to your clay input, which means AI adds material and lighting nuance but does not change composition or lighting direction. Medium denoise, 0.4 to 0.6, lets the AI reinterpret the lighting more, which can give beautiful atmospheric results but occasionally shifts the mood away from your brief. High denoise above 0.7 is basically text to image with a hint from your clay and is rarely what you want.

Prompt writing for img2img is different from text to image. Because the geometry is already there, you should describe the material and lighting only, not the architecture. Photoreal interior, oak floor, plaster walls, morning east light with soft haze, cinematic 35mm, is a better prompt than describing the room shape or window placement, because that information is already coming from your clay pass. Redundancy in the prompt confuses the model.

The workflow I run in the studio for a concept variation set looks like this. Model the scene, render a clay pass at final resolution, extract depth and canny in ComfyUI, run SDXL through ControlNet at 0.3 denoise for a tight variant and 0.5 denoise for a mood variant. Four variants per prompt, three prompts per concept, gives the client twelve concept boards from one clay render in about 90 minutes. Compare that to the two days it would take to lookdev and render twelve full variants and the value is obvious.

This workflow does not replace final delivery renders. What it replaces is the concept and material exploration phase, where you would traditionally do rough post over a clay in Photoshop. ControlNet over clay is faster, produces more variants, and lets the client react to atmospheric decisions before you commit to full lookdev. Once the direction is agreed, you go back to your normal Corona or V-Ray pipeline for the final delivery.

There are two failure modes to watch for. First, ControlNet can lose thin geometry like fine mullions or slender balustrades because the depth model does not resolve them well. If your facade depends on fine linework, use Canny not Depth, or use both. Second, at low denoise the AI can leave patches that look like clay because it did not add enough material information. Bumping denoise up to 0.35 usually fixes this without shifting geometry visibly.

The hardware question is worth naming. SDXL through ControlNet on an RTX 4090 runs a single 2k image in about 25 seconds. On a 5090 it runs in about 12. On anything older than a 3080 the wait times become long enough that you will stop using the workflow. For studios investing in AI capable hardware this year, the 5090 is the strongest single card option and comfortable for two artists to share through a render node.

The last thing worth saying is that this workflow respects the client and the architecture in a way that pure text to image does not. When you show a client a ControlNet variant, the building is theirs. The window layout is theirs. The proportions are theirs. What the AI added is material feeling, lighting mood, and atmosphere. That is the honest place for AI in an archviz pipeline right now.