ML for Curious

ControlNet action, on how to stylize your images with SDXL. comfyUI workflow is uploaded

finally had a chance to play with SDXL 1.0 model. I hope you'd recognize the topic 😜 If not, ask in the comments :)

There is no recording on talks unfortunately, however, many talks are accompanied with slides and papers. https://causal-machine-learning.github.io/kdd2023-workshop/

"We've just launched fine-tuning for GPT-3.5 Turbo! Fine-tuning lets you train the model on your company's data and run it at scale. Early tests have shown that fine-tuned GPT-3.5 Turbo can match or exceed GPT-4 on narrow tasks:" https://openai.com/blog/gpt-3-5-turbo-fine-tuning-and-api-updates

The article offers a comprehensive guide on instruction-tuning Llama 2 from Meta AI, outlining the ability to create an instruction dataset. This dataset aids in fine-tuning Llama 2 to generate specific instructions based on input, facilitating tasks like personalized email writing. The tutorial covers defining use cases, creating prompt templates, and using TRL and the SFTTrainer for instruction-tuning, all executed on an AWS EC2 instance with an NVIDIA A10G GPU. https://www.philschmid.de/instruction-tune-llama-2

Runway Gen-2 model is just awesome: https://www.youtube.com/watch?v=Lym-1Q57QC4

Models like DistilBERT are popular because LLMs are usually too large for most contexts, but creating such pruned versions of a model usually require retraining. In this paper, researchers propose an approach that does not require retraining a pre-trained LLM for pruning. The approach is relatively simple and based on determining the weight importance for pruning based on computing element-wise product between the weight magnitude and norm of input activations.

Topological Data Analysis - the new kid in town! Summary 👇 ✅ Robust to noise and outliers: TDA is based on the topology of the data, which is more resistant to noise than geometric methods like t-SNE and UMAP. ✅ Multiscale Analysis: TDA can detect and represent the data structure at multiple scales, allowing for a more complete and nuanced understanding of the data. t-SNE and UMAP are mainly capturing local structures in the data and may miss global structures. ✅ Ability to learn complex patterns: Self-supervised deep learning allows the model to learn complex, hierarchical representations of the data. It is useful for high-dimensional data with non-linear relationships. ✅Parameter-free: This means that it does not require a lot of tuning or prior knowledge of the data. t-SNE and UMAP require careful parameter selection to achieve good results. ✅ Scalability: TDA is scalable to large datasets, and recent advances in computational methods makes TDA stand out. https://datarefiner.com/feed/why-tda

8 annoying A/B testing mistakes every engineer should know 💡 This article highlights common A/B testing pitfalls, which encompasses: 1) including unaffected users in experiments 2) only viewing aggregate results and neglecting subgroup insights 3) not setting a predetermined experiment duration 4) running full-scale experiments without preliminary testing 5) neglecting counter metrics that measure unintended negative effects 6) failing to account for seasonality in user behavior 7) testing unclear hypotheses 8) relying too heavily on A/B tests for decision-making https://youtu.be/C2ICJbICfsg

My darling has arrived:)

Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Models The paper tackles two problems: ✅ “catastrophic neglect”, where one or more of the subjects of the prompt are not generated ✅ incorrect “attribute binding”, where the model binds attributes to the wrong subjects or fails to bind them entirely Solution: ✅ Extracts subset tokens S from the prompt ✅ Define an auxiliary Loss function to be used during the backward pass of DM ✅ Conduct an optimisation step during backward pass to increase the values in cross-attention maps corresponding to S Paper: https://arxiv.org/pdf/2301.13826.pdf

Extracting Training Data from Diffusion Models As privacy plays an important role in generative tasks, this paper attempts to show that diffusion models are vulnerable to different kind of attacks. And they are able to find duplicated images. Where the paper falls short IMHO is that it finds around < 200 near-duplicate images for Stable Diffusion model from 16 million training samples. Which is less than 0.0000125% of the data. Where practitioners should take care is when the custom training data contains a lot of sensitive information e.g. fashion models, company logos.

Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translation Main observations: ✅ spatial features extracted from intermediate decoder layers encode localized semantic information ✅ self-attention - representing the affinities between the spatial features, allows to retain fine layout and shape details Summary: We want to obtain a target image - based on input image and target prompt. ✅ obtain an initial noise from the input image using DDIM. ✅ Two processes are happening in parallel: 1. to obtain the input image (M1) 2. to obtain the target image (M2) ✅ On each step, certain feature maps from the decoder and self-attention layers from M2 are replaced by corresponding layers from M1. Pros: ✅ Works both - on synthesised and real images. ✅ Findings in the paper are aligned with findings of prompt-to-prompt ✅ No retraining of DM is required ⚡️DDIM inversion of synthesised images might be in conflict with "Null-text Inversion" paper. Project page: https://pnp-diffusion.github.io/

Using Bayesian Optimisation to optimize Chocolate-Chip Cookie recipes Quite entertaining study by Googles team on how to use optimisation techniques in our daily problems. One has to point out though that for the evaluation one might still need to have a lot of resources: Paper: https://storage.googleapis.com/pub-tools-public-publication-data/pdf/f52ac33bc9d1adecd3a8037a7009b185fd934f0e.pdf Library used in the paper: https://github.com/google/vizier

ok this becomes a habit: I use all the limit for the post for explanation and forget about the paper reference: https://arxiv.org/pdf/2212.05973.pdf

Making Guidance practical I will use classifier guidance as an example. However, one can find many more formulations in the paper. Main Observations: ✅ Publicly off-shelf classifier models perform poorly on the noisy inputs (usually used to obtain the gradient to make a step in the backward pass) ✅ One would need to obtain labeled data to augment the inputs with noise to train a classifier - EXPENSIVE Solution: 1. Collect dataset without labels 2. Bucketize the diffusion steps T = 1000 to 5 buckets {1-200, 200-400, 400-600, 600-800, 800-1000} 3. Run a diffusion forward pass - randomly save a noisy image from every bucket. Outcome - 6 datasets (5 noisy, 1 noiseless) 4. Take off-shelf classifier - freeze it. For each noisy dataset fine-tune the copy : a. Noiseless image -> the frozen model b. Noisy image -> copy model c. Minimise KL-divergence between logits 5. By the end of step 4 one has 5 expert models. Inference - for a step t use the expert model from bucket where t falls for guidance.

Ran out of space in the previous post: Paper link: https://openaccess.thecvf.com/content/CVPR2023/papers/Lu_Specialist_Diffusion_Plug-and-Play_Sample-Efficient_Fine-Tuning_of_Text-to-Image_Diffusion_Models_To_CVPR_2023_paper.pdf

Specialist Diffusion Models - one more really cool addition to the box of generating personalized content Contributions: ✅ Image/Text augmentations during the training process (by paying attention to aug. leakage) ✅ Content Loss - disentangles the knowledge of content (from pre-trained DM) and style (from few-shot examples). ✅ Sparse updating to increase the training speed Pros: ✅ Achieves really impressive results and is able to capture the style from few examples ✅ Relies requires low computational power ✅ Works really well in combination with textual inversion Cons: ⚡️The results in the paper seem to be heavily cherry picked (too many cats and castles). More diverse examples would definitely help. ⚡️Ablation study - seemed to reveal the effect of the content loss. While effect of augmentations could be attributed to the random sampling. ⚡️Supplementary material and discussion on Limitations are missing Other than that BIG CONGRATULATIONS to PicsArt team! (Paper is accepted to CVPR2023)