Paper Banana: From Prompt to Publication-Ready Figure

Skip the design software. Paper Banana turns your research ideas into clean paper figures, workflow diagrams, and scientific illustrations in minutes.

Editable

Examples:

Paste paper content directly or describe the figure you need

3 Method
Large language models (LLMs) have recently demonstrated remarkable multimodal understanding capabilities when equipped with visual encoders and lightweight projection modules. Existing multimodal training paradigms, however, primarily optimize language modeling objectives, leaving the visual representations weakly constrained during training. Consequently, visual information may be partially discarded as it propagates through the language model layers.

To address this limitation, we propose Reconstructing Visual Tokens via Denoising (ROSS), a training framework that introduces an auxiliary visual reconstruction objective into standard multimodal LLM training. Rather than solely predicting textual outputs, the model is additionally required to preserve sufficient visual information for reconstructing image latent representations under a diffusion objective.

3.1 Visual Token Construction

Given an input image I, we first employ a pretrained SigLIP visual encoder to extract image features. Let f_I = E_siglip(I) denote the output feature sequence. A trainable projector H_phi maps these features into the language model embedding space: v = H_phi(f_I), where v in R^{N x D} represents a sequence of N visual tokens. For textual instructions, we utilize the tokenizer associated with the base language model to obtain text embeddings w in R^{(T-N) x D}. The visual and textual tokens are concatenated to form the final input sequence X = [v; w], which is subsequently processed by a shared LLM. Unlike prior approaches that only supervise textual outputs, ROSS explicitly supervises the visual prefix positions.

3.2 Teacher Latent Representation

To provide dense visual supervision, we introduce a frozen latent-space teacher derived from the KL-16 VAE used in diffusion models. For each image I, the teacher encoder produces clean latent tokens z_0 = E_VAE(I), where z_0 in R^{M x C}. Following the standard diffusion formulation, a timestep t ~ Uniform(1, T_d) is sampled, and Gaussian noise is injected: z_t = sqrt(alpha_bar_t) z_0 + sqrt(1 - alpha_bar_t) epsilon, with epsilon ~ N(0, I). The resulting noisy latent representation serves as the reconstruction target for the denoising branch. Importantly, both the KL-16 encoder and the noise schedule remain frozen during training.

3.3 Shared LLM Processing

The multimodal sequence X = [v; w] is fed into a transformer-based language model. Let h = LLM(X) denote the hidden states. The first N positions h_{1:N} correspond to visual outputs, while the remaining positions h_{N+1:T} correspond to textual outputs. Textual outputs participate in the standard autoregressive language modeling objective. Visual outputs are routed into a dedicated reconstruction branch.

3.4 Visual Reconstruction via Denoising

The key idea of ROSS is to force visual outputs generated by the language model to retain image information. To achieve this, we train a denoiser J_pi that predicts diffusion noise conditioned on noisy latent tokens z_t, diffusion timestep t, and visual hidden states h_{1:N}. Formally, epsilon_hat = J_pi(z_t, t, h_{1:N}). The denoiser is optimized using the standard diffusion objective: L_visual = E[||epsilon - epsilon_hat||^2]. By minimizing this objective, the model learns visual representations that are informative enough to reconstruct original image latents.

3.5 Language Modeling Objective

For text generation, we retain the conventional next-token prediction loss: L_text = -1/(T-N) sum_{i=N+1}^{T} log p(x_i | x_{<i}, v). This objective ensures that language generation performance remains comparable to existing multimodal LLM training pipelines.

3.6 Joint Optimization

The final training objective combines the language modeling loss and the visual reconstruction loss: L_ROSS = L_text + lambda L_visual, where lambda controls the balance between linguistic and visual supervision. In practice, we find that setting lambda = 1 provides stable optimization across multiple model scales.

3.7 Training Configuration

Unless otherwise specified, all experiments use a frozen SigLIP visual encoder, a frozen KL-16 VAE encoder, a decoder-free diffusion denoiser, and a Llama-family language model backbone. Trainable parameters include the projector H_phi, the shared LLM, and the denoiser J_pi, while the visual encoder, teacher tokenizer, and text tokenizer remain frozen.

Prompt (4378/15000)

Stylized prompt

Preset prompt (editable)

Model

Aspect ratio

Resolution

Format

Paper Banana Gallery

Explore prompt examples for model architectures, method workflows, evaluation panels, and dense paper-style research figures.