Architects' tools

The first three parameters are simple: - choose a name for your model (literally any name, but I prefer to use a prefix with a reference to the base model, in this case FLUX_) - specify the path to the folder in Google Drive where your dataset is located - specify the path to the folder in Google Drive where LORA models and images from test renders will be saved.

Google Colab, what we need: - to start somewhere around 100 compute units for a few tests (you can buy them here) - google drive for 100 GB (link). Next, let's open Colab, to which I already gave you a link. and go to setting up the parameters, located in the second section Step 2. Setup Config #comfyui #lora #flux

tomorrow we will analyze the parameters in Google Colab in detail. if someone has time to build his dataset, you can test it right away. an important point is that we will need a paid Colab, because we need a graphics card A100. in the free subscription at least it is not always possible to connect to it, at most it will not be available at all. for one session of training I spent about 30 compute units, which is about 3 eu.

Image descriptions can of course be done yourself manually, but I've trained GPT for this task, here's a link to it: https://chatgpt.com/g/g-wvc9iwYuc-image-captioner just upload images without asking anything, it knows what to do. the output will be a neat table. If you want to know what criteria are used to make descriptions, there is a button “Show the list of parameters...” on the start page. The only point is that ChatGPT does not work well with large amounts of uploaded materials, so it is better to upload images in portions, 5 at a time for example.

In order to start training a model, we need a dataset. proper preparation and processing of the dataset is the key to success. the tests for this material I made on the basis of photos of Coop Himmelb(l)au projects, which I downloaded from the company's website, where they are in good enough quality. The dataset contains 57 photos of about 20 different projects. in the folder it looks like on the screenshot. don't pay attention to the json file and the _latent_cache folder, they are created automatically when you start the training. we only need images and text files with descriptions of these images, the names of png and txt files must match.

a bit delayed, but it's time to talk about training LORA models for FLUX. i think i'm going to split this material into a few days. let's start with the training technique itself: we'll use this tool. https://github.com/ostris/ai-toolkit If you have 24Gb+ graphics card, you can run it locally. Ostris has links to tutorials on github page I mentioned above. but I used Google Colab. Artem Svetozarov adapted the original colab by adding separate windows with parameters, and I, in turn, slightly tweaked the code to allow different models to be used. but this link is specifically for Flux Dev only (others in the next posts): https://colab.research.google.com/drive/1xWiIQFpCx7aEkgEd_aBmrHm9hH5iBMIb?usp=sharing #comfyui #lora #flux

I promised to post information on how to train LORA on FLUX this week, but there is a new version of the model, which I also need to try in tests, if it is dramatically better and there is no point in posting old information, so I need some more time.

The way FLUX captures architectural style details when training LORA is quite impressive. This is a selection of images generated through models that I have trained on Coop Himmelb(l)au projects. Conclusions and details on how to train I'll publish next week. #comfyui

I've been doing a lot of model training for FLUX in the last couple of weeks, and there are some interesting findings there. i'll try to summarize the information and make posts soon. for now, here's a spreadsheet i'm creating to track progress during training. #comfyui

I've updated the link, yesterday it didn't work

LORA collection that I use in my practice https://civitai.com/collections/4905737 #comfyui

posted another one of my LORA on civitai. https://civitai.com/models/776612?modelVersionId=868589 #comfyui #ai

Today is the last day to register for our December webinar at a discounted price https://designmorphine.com/education/cellular-diffusions-v1-0

To save memory it makes sense to download and use GGUF Flux models. on the ip-adapter page they recommend this one specifically: flux1-dev-Q4_0.gguf, there are also other recommendations for running on weak machines (and for Flux almost anything is a weak machine).

about accuracy, this is an example of the original image (left) and the result (right). you can see that the style is completely different, here, rather, we are talking about the fact that the features of the object from the original image are transferred, not the style. In the ip-adapter node we have almost no settings, except for strength, unlike SDXL, where there are many parameters. nevertheless, the result is interesting and the approach itself can be used for form finding. and you can also add LORA, I did so in the example that I shared above. I used my own LORA.

Let's talk about ip-adapter for flux. what you need to know about it: - first, it is not as accurate as the similar approach for sdxl; - second, you need even more graphics memory than just for flux, but there is a nuance, which I will tell you next post today; - third, ip-adapter for flux can't be combined with control net (or I haven't found how to do it in a good way. formally mistoline cn works with xlabs sampler but results are incredibly bad in my tests), nevertheless control net doesn't work well with the architecture tasks yet, so it's not a big loss. #comfyui

And of course, those who managed to complete the task received certificates. The assignment was not simple and was time-consuming, just like the course itself.