Learning Modal-Mixed Chain-of-Thought Reasoning with Latent Embeddings

(Shao et al., 2025)

This work proposes a way for multimodal models to reason with both words and tiny visual sketches that live in latent space. These sketches are generated on the fly, guided by the model’s language reasoning, and they help on vision intensive puzzles. The team trains this in two stages, first with supervised traces, then with reinforcement learning. Results across many tasks show steady gains over language only chain of thought.

Why This Is Interesting

Chain of thought made language models better at math and logic by getting them to show their work. That works less well when the hard part is visual, like tracking shapes, rotations, or tiny differences that are painful to describe in words. The paper’s core idea is simple and appealing: let the model switch between words and compact visual latents during reasoning, so it can offload the purely visual bits to lightweight embeddings while keeping language for high level logic.

How It Works

The method reuses the vision encoder and connector from the base VLM to define the latent space.
A diffusion head turns a hidden language state into a compact visual latent.
The model writes normal text until it emits a start control token, generates latent embeddings autoregressively, closes with an end control token, and goes back to text.
Training mixes supervised fine tuning on interleaved traces with reinforcement learning that teaches when to switch modalities and how long the sketch segments should be.

Results And Limits

On spatial and inductive tasks, the modal mixed model improves over the base VLM variants.
On visual search, spot the difference, and auxiliary lines, the approach lifts the base model’s scores.
Removing latents and training on text only hurts.
Replacing the diffusion head with a simple similarity loss helps a bit, but still trails the full method.
RL helped perception tasks a lot, yet some abstract logic dimensions saw smaller gains.

References

2025

arXiv

Learning Modal-Mixed Chain-of-Thought Reasoning with Latent Embeddings

Y. Shao, K. Zhou, Z. Xu, M. Atif Quamar, S. Hao, Z. Wang, Z. Hu, and B. Huang

Under Review

Abs arXiv

We study how to extend chain-of-thought (CoT) beyond language to better handle multimodal reasoning. While CoT helps LLMs and VLMs articulate intermediate steps, its text-only form often fails on vision-intensive problems where key intermediate states are inherently visual. We introduce modal-mixed CoT, which interleaves textual tokens with compact visual sketches represented as latent embeddings. To bridge the modality gap without eroding the original knowledge and capability of the VLM, we use the VLM itself as an encoder and train the language backbone to reconstruct its own intermediate vision embeddings, to guarantee the semantic alignment of the visual latent space. We further attach a diffusion-based latent decoder, invoked by a special control token and conditioned on hidden states from the VLM. In this way, the diffusion head carries fine-grained perceptual details while the VLM specifies high-level intent, which cleanly disentangles roles and reduces the optimization pressure of the VLM. Training proceeds in two stages: supervised fine-tuning on traces that interleave text and latents with a joint next-token and latent-reconstruction objective, followed by reinforcement learning that teaches when to switch modalities and how to compose long reasoning chains. Extensive experiments across 11 diverse multimodal reasoning tasks, demonstrate that our method yields better performance than language-only and other CoT methods.