Stability AI and its multimodal AI research lab, DeepFloyd, have announced the release of their latest research project, DeepFloyd IF. This powerful text-to-image model is available on a non-commercial, research-permissible license, allowing other research labs to examine and experiment with advanced text-to-image generation approaches. Stability AI aims to release a fully open-source version of the DeepFloyd IF model in the future.
The DeepFloyd IF model utilizes the T5-XXL-1.1 language model as a text encoder to better understand text prompts. With the addition of text-image cross-attention layers, the model generates coherent and clear text alongside objects of different properties appearing in various spatial relations.
One of the most impressive features of the DeepFloyd IF model is its ability to generate photorealistic images. It boasts an impressive zero-shot FID score of 6.66 on the COCO dataset, which is a main metric used to evaluate the performance of text-to-image models. The lower the FID score, the better the performance of the model.
The DeepFloyd IF model also has the ability to generate images with non-standard aspect ratios, including vertical or horizontal, as well as the standard square aspect. It can conduct zero-shot image-to-image translations, allowing for image modification without the need for fine-tuning. This is achieved through a three-step process that involves resizing the original image to 64 pixels, adding noise through forward diffusion, and using backward diffusion with a new text prompt to denoise the image. The output can be further modified using super-resolution modules through text description prompts.
The DeepFloyd IF model has significant implications for research labs that are interested in advanced text-to-image generation approaches. It provides an opportunity to examine and experiment with the latest techniques, including the use of language models and text-image cross-attention layers. The model’s ability to generate photorealistic images and modify image style, patterns, and details without fine-tuning is a significant breakthrough in the field of text-to-image generation.
Stability AI intends to release a fully open-source version of the DeepFloyd IF model in the future, further democratizing access to the latest text-to-image generation techniques. This will enable researchers to further develop and refine the technology, potentially leading to new and innovative applications in various fields, including art, design, and visual storytelling.
https://stability.ai/blog/deepfloyd-if-text-to-image-model
https://huggingface.co/spaces/DeepFloyd/IF