ML for Curious

Preparation works at NeurIPS 2023

WIth SDXL inpainting one can achieve really impressive results in product design. Played a bit yesterday evening to put my favorite product on the spot ☺️

⚡️ The Chosen One: Consistent Characters in Text-to-Image Diffusion Models Brand new paper that achieves consistency across generations. Other methods - to achieve consistency need multiple images of the same object as an input. This method is self sufficient and follows a simple heuristic: depicted in the algorithm. In simple words: ✅ N images are generated given the same prompt with different seeds ✅ Generated images are going through a feature extractor ✅ Clusters are formed from the feature vectors ✅ Most "cohesive" cluster is chosen ✅ LoRA parameters are optimized given the images of the cluster 5 steps above are repeated until images generated by the model are "too similar". Drawbacks: ❗️Problems with consistency of multiple characters ❗️Spurious attributes ❗️Character inconsistency for very generic prompts ❗️Too many hyper-parameters to take care of

Dall-3 is part of chatGPT now: https://openai.com/dall-e-3 The image quality is astonishing. The paper details will be shared later. For the image above the following prompt was used: "medieval city of Yerevan, surrounded with walls. Castle is erected in the city center surrouned by a ditch. Knights are riding their horses, people in the market having their usual day "

Yet another paper from google that emphasises the importance of the alignment between the prompts and the image content. They also provide a very interesting and lightweight procedure to generate better image captions automatically. In their experiments they have prepared around 100 images with really good and detailed captions and fine-tuned a pre-trained model with them. The model was able to adapt really quickly, and the subsequent fine-tuning of SD model brought significant improvements. Paper Link: https://arxiv.org/pdf/2310.16656.pdf

✅ LoRA (low-rank adaptation) is awesome because it adapts only a small, low-rank subset of parameters of a base LLM. ✅ QLoRA is awesome because it lowered memory requirements even further by quantizing the base model weights. ✅ QA-LoRA is even more awesome as it takes QLoRA a step further and also quantizes the LoRA (adapter) weights, avoiding a costly conversion of the quantized base model weights back into 16-bit when adding the adapter weights. Table 2 shows that QA-LoRA is about 2x faster than QLoRA for fine-tuning. However, a much smaller number of parameters was used for the adapter weights. I believe it would have been fairer to use the same number of parameters for both when comparing their speeds. Paper Link: https://arxiv.org/abs/2309.14717

@Artificial_intelligence_ai 🎓 BayesDLL: Bayesian Deep Learning Library New Bayesian neural network library for PyTorch for large-scale deep network 🖥 Github: https://github.com/samsunglabs/bayesdll 📕 Paper: https://arxiv.org/abs/2309.12928v1 ⭐️ Dataset: https://paperswithcode.com/dataset/oxford-102-flower To learn AI from basics: YouTube: https://www.youtube.com/c/asifimmanad Telegram: https://t.me/Artificial_intelligence_AI

LoRA roulette from HuggingFace, a gamification on combining different LoRAs. https://huggingface.co/spaces/multimodalart/lora-roulette

With several lines of code, one can render images of higher quality with FreeU. https://www.youtube.com/watch?v=-CZ5uWxvX30

⚡️ IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Models The paper dives into the topic of prompting the pre-trained models with images. Main contributions: 1. The trained adaptor is lightweight and can be trained on the backbone of any diffusion model 2. Adaptor introduces a decoupled X-attention mechanism which allows to multi-modality to achieve desirable results. As a consequence the adaptor is also pluggable. Provided interface allows to put weights to define the strength of image prompt vs text prompt 3. IP-adapter is also compatible with other types of adaptors such as ControlNet. Methodology: 1. The image is mapped to sequence of features. 2. X-attention map is created between image-sequence-features and latent-noise-vector 3. X-attention maps from image branch and text-branch are summed together Note: Query matrix is shared between to X-attention maps PaperLink: https://arxiv.org/pdf/2308.06721.pdf Code: https://github.com/tencent-ailab/IP-Adapter/tree/main

⚡️New ImagePromt adatpters in action!!! The order of operations: 1. Take original photo 2. Extract canny edges 3. Generate an image using SDXL using only prompts and save it (text-prompt) 4. Use SDXL+IP-Adaptor+ControlNet: Conditions are on the generated image and canny edges only Paper description will post later :)

The math is sloppy but the idea is good: https://towardsdatascience.com/which-features-are-harmful-for-your-classification-model-6227859a44a6

⚡️One of the papers one should be definitely be excited about: InstructDiffusion: A Generalist Modeling Interface for Vision Tasks Authors found a way for a unifying image task representation with the help of diffusion models. Instead of formulating keypoint detection, semantic segmentation, image enhancement as separate tasks, they formulate it as image-to-image problem. And each task is expressed via natural language: ✅ Keypoint detection -> "Mark the left shoulder of the person with a blue circle" ✅ Semantic Segmentation -> " Mark the pixels of cat in the mirror to blue and leave the rest unchanged" ✅ Enhancement -> "Remove watermark from this picture", "Purify this photo by removing noise" The model shows generalization capabilities to tasks it was not explicitly trained on. One has to note - authors were really creative in collecting datasets :) Project page: https://gengzigang.github.io/instructdiffusion.github.io/