Architects' tools

first test of published today SD3. it's definitely not a revolution, but not bad at all for the base model. waiting for custom solutions, it should be cool. #comfyui

yesterday, among other things, we discussed with my students at an ai workshop how to control parts of an image with different prompts. this allows, for example, to apply different models or techniques to one or another part of the image. on these slides, the urban environment is made only with the RealVisXL model, and for the building in the center of the frame we additionally used LORA models designed by Ismail Seleit. #comfyui

I sometimes give workshops for architectural studios on AI+3D modeling (Grasshopper and Blender) and for that I have prepared various guides. This one is dedicated to the ComfyUI installation process and contains links to my favorite models, I think it's worth sharing: https://docs.google.com/document/d/1RH5z-_r0CCrKOy1V1rSUsxVv5BPErpvygzh4_VzCQQs/edit?usp=sharing

What is important to understand about AI rendering in general: 0. AI rendering does not replace normal rendering, but you can use them in combination. You can also use both techniques for different purposes. Normal rendering gives more control than AI, and this is often a key factor, but sometimes you want the opposite, i.e. less control and more randomness (creativity). 1. You can't say that AI rendering is faster, on my PC these two images are calculated in about the same time (first Blender, second Stable Diffusion), around 3-4 minutes. 2. The input image you use (for example, a screenshot from the 3D software) in any case will change in details on the final rendering, strongly or not, depending on the settings and resolution of the input. 3. Stable Diffusion does not work well with small details in the image, it does not recognize them well, and during rendering it can turn them into something else. 4. A lot depends on the details of the input image, if we have high LOD and we just need to play with light and atmosphere, it is one story, if there are not enough details and we tell to AI that it should add them, it will do it for the whole image, including changing what we might want to keep, and it will have to be solved somehow with masks or post-processing. 5. As with normal rendering, post-processing is necessary, something to add, remove, make color correction .... 6. To get a good result, you must first develop the whole process, maybe even train your own models. It takes quite a long time, and then it must be maintained and updated. 7. Using AI renderer is technically difficult and there is a lot to learn there

what is important to understand about this workflow: 0. You can load any images as inputs, but those that are a reference for rendering should be in good resolution (2048 pixels on the long side or better) if you want relatively accurate transfer of object contours to the final render. For an IPAdapter image the resolution is not so important. 1. The same workflow can also be used for idea or form finding, when you don't have a detailed 3D model, but only a mass model, simple physical model or a sketch (also on the internet people love all kinds of fucked up stuff like a render from crumpled paper or from a forest cone , I'm not a fan of that, but feel free))). For such purposes you need to add a preprocessor for the first ControlNet and use only one input image instead of two. Also, for more creative results, you will need to make the parameters in ControlNet not so strong, I've highlighted them on the screenshot.

and this is an example of three different cameras with the same image in IPAdapter.

examples of rendering one camera with three different images for IPAdapter, other settings the same for all results

demonstration of how it looks like in the software viewport. of course here you have full control, you can hide and add objects, that is, everything like for a normal render process.

if you use #rhino for 3D modeling, high-resolution screenshots are exported there with the _ViewCaptureToFile command black and white screenshot is not just for fun - it is a special graphic used for some ControlNet. you can achieve such a result automatically in ComfyUI with the help of preprocessor, but creating your own image allows you to take better control over the contours of objects in the frame. I customized the visual styles in Rhino, which is not hard to do, but to save your time, here are the files:

workflow file and images for the test, but of course you can use your own inputs. in this example: the top two images are screenshots from the Rhino project we developed in Cloud Cooperation, I wrote about it earlier. the third image is needed for IPAdapter, from which we take the atmosphere for rendering (lighting, color palette, materials...)

I promised to finalize the workflow for AI renderer and share it, now it's time to do that. I get the best results when using two ControlNet + IPAdapter together. Details and files in the next posts tonight. #comfyui

A competition project for the extension of a puppet theater in Stara Zagora, Bulgaria, which we did at Cloud Cooperation Modeling: Rhino+Revit Rendering: Blender Our team: Mohamed Abdelhadi Markus Prossnigg Benjamin Schmidt Irina Sorokina Albert Sumin #myworkdiary #blender #rhino #revit I need to post this project here as it is important to the next ComfyUI workflow that I will be featuring

But that's not all, I usually use Photoshop and Lightroom for image post-processing. in Photoshop I remove artifacts using the built-in ai. in Lightroom I do light and color correction.

I also tried to use images from midjourney 6.0 as inputs, these three

some examples of what I got, with different values of the seed parameter

the main difference of this version of workflow is prompt, it sets a new atmosphere relative to the original image from midjourney, but the image itself affects the composition and materials. the realism of materials and overall detail is achieved by checkpoint and lora models.