Recently, the paper "Multi-modal Auto-regressive Modeling via Visual Tokens," co-authored by Prof. Ping Wang's team at the Laboratory and Shanghai Jiao Tong University, was accepted as a poster presentation at ACM Multimedia 2024 (the 32nd ACM International Conference on Multimedia). The conference will be held in Melbourne, Australia, from October 28 to November 1, 2024. ACM Multimedia is a Class A academic conference recommended by the China Computer Federation (CCF). This year, 4,385 submissions entered the review stage, of which 1,149 were accepted, an acceptance rate of 26.20%.

Large language models (LLMs) learn general semantic information and strong reasoning capabilities from natural language corpora through auto-regressive modeling on large-scale unlabeled text. However, extending this approach to multimodal scenarios poses challenges: image information is processed as continuous visual embeddings in LMMs, so discrete supervision labels for a classification loss cannot be obtained.

This paper achieves, for the first time, multimodal auto-regressive modeling with a unified objective, and explores the distribution of visual features within the internal semantic space of LLMs, as well as the possibility of representing different modalities with textual information. To address this problem, the researchers propose VT-LMM (Visual Token guided Large Multi-Modal Model), which introduces the concept of visual tokens to construct the distribution of visual features in the semantic space of the LMM and provide supervision information for visual modeling.

For each patch in an image, the researchers display in the figure the token with the highest probability among its corresponding visual tokens. The results show that visual tokens successfully transform visual features into the textual semantic space.

This study achieves, for the first time, multimodal auto-regressive modeling with a single objective. Visual tokens are used to construct the form in which information from different modalities exists in a shared semantic space, thereby providing supervision labels for both visual modeling and language modeling and explicitly supplementing the model's learning of the distribution of visual content. Furthermore, the researchers investigate the consistent form in which information from different modalities exists in the internal semantic space of large models, as well as the feasibility of using linguistic information to represent features of different modalities.