You thought you wouldn't have to hear about AI today? Well, think again! It's 2023 after all! (Don't worry, you don't need any prior AI expertise — we'll explain all you need to know.)

If you've ever been part of any fringe internet community, I'm sure you've encountered some amount of cat pictures. The way those communities work is simple, the more cat pictures you post, the more popular you are. (Though some communities prefer other animals, like shiba inus or frogs, but the general concept is the same.) But the amount of REAL cat pictures you can post is limited; wouldn't it be great if you could generate cat pictures on demand, just from a text prompt?

Generative AI models such as Stable Diffusion do just that. Today we'll look at a custom version of that, let's call it Cat Diffusion, specifically designed for the generation of cute cat memes.

In Cat Diffusion, each prompt is first converted by a so-called text encoder into a $$$k$$$-dimensional embedding vector of integers $$$(v_1, v_2, \ldots, v_k)$$$. Think of this as a compressed vector containing all the necessary information to reconstruct the sentence's meaning. Sentences that are more alike are closer together; those that are different are further apart. This embedding vector is then passed to a so-called diffusion model, transforming this embedding into a picture.

Example of a few different prompts in a two-dimensional embedding space (not Cat Diffusion); similar sentences stick together.

The diffusion process, which transforms the embedding vector into an image in many small steps. Conveniently for the AI, Yoda's fingers are cropped from the image.

Each of the elements in the embedding vector represents, in some sense, a property; it could represent the type of the cat, or its pose, or the picture's brightness. [In practice, the meaning of those properties is much more abstract and doesn't easily map to real-world concepts; understanding embedding spaces is a huge research topic in machine learning.]

A prompt consists of some number of words. Cat Diffusion uses a very simple text encoder: The $$$i$$$-th word is associated with a certain set of property-value pairs $$$\left \{(p_{i,1}, v_{i,1}), (p_{i,2}, v_{i,2}), \ldots, (p_{i,m_i}, v_{i,m_i}) \right\}$$$. The text encoder now starts with the embedding vector where all properties are set to $$$0$$$; then, it goes left-to-right through the entire prompt, and for the $$$i$$$-th word, for every $$$j \in \{1, \ldots, m_i\}$$$, it sets the $$$p_{i,j}$$$-th property in the embedding vector to $$$v_{i,j}$$$.

You already know what cat picture you want to generate, and what properties it should have. Can you find a prompt that encodes into the exact embedding vector $$$t$$$ that you want, if one exists?