2.1. Recap: Text#

A typical, text-only LLM is a stack of self-attention layers, with an embedding layer on the input side and the lm_head (token prediction) layer on the output side.

In such a LLM, text data is handle first by a tokenizer; then the embedding layer. Only then, text is represented as a (sequence of) vector(s); it is embedded on a vector space; which is then digested by the Transformer, a large language model. Before training, this structure – somewhat effective tokenization and vector representation – gives a room for the vector sequence to become semantically meaningful.

While training, the embedding layer is trained simultaneously, to output a semantically meaningful representation of the text, ${\mathbf{H}}_{\mathtt{q}}$.

text -> [tokenizer] -> integer seq -> [embedding layer] -> vector seq -> [Transformers]  

2.2. Image: Encoder and Adaptor#

Likewise, we need some way to process an image as an input of a LLM. Let’s break this process into an image encoder and an image adaptor. On a high level, this process is the same as we did to the text. As a result, we would get a vector representation ${\mathbf{H}}_{\mathtt{v}}$.

Encoder outputs a vector representation (or a sequence of vector) ${\mathbf{Z}}_{\mathtt{v}}=g({\mathbf{X}}_{\mathtt{v}})$. Here, $g()$ refers to the image encoder and ${\mathbf{X}}_{\mathtt{v}}$ is the input image. The encoder can be, for example, a ViT [DBK+20], or any other image encoder.

Adaptor is usually a linear layer (matrix multiplication) $\mathbf{W}$ that matches the vector dimension. In other words, ${\mathbf{H}}_{\mathtt{v}}=\mathbf{W}\cdot {\mathbf{Z}}_{\mathtt{v}}$. For example, if we’re using llama 7B model, $\mathbf{W}$ would convert the input visual features to a sequence of 4096-dim vectors.

This process is illustrated nicely in [LLWL24], where visual-input instruction task was performed (e.g., visual question-answer and image captioning).

At the end, the Transformer architecture ($f_{\varphi }$ in the image) performs the same task – to process input (${\mathbf{H}}_{\mathtt{v}}$ and ${\mathbf{H}}_{\mathtt{q}}$), through the self-attention layers, to generate the optimal outputs ${\mathbf{X}}_{\mathtt{a}}$. During training, the language model learns to understand not only the text representation but also the image representation.