Three different mazes produced the exact same output. Measuring where in the pipeline the image's information actually gets lost.
Retrained with EOS-appended labels, then tested generation across four maze seeds: two from the training set (0 and 5), two held out (600 and 601). Stopping now works in all four cases. Generation terminates with <|im_end|> instead of rambling past the answer.
Then looked at the actual predicted points.
seed 0 (in-training): (249,83) (249,249) (416,249) (416,416) (582,416) (582,582) (749,582) (749,749) (915,749) (915,915)
seed 5 (in-training): (249,83) (249,249) (416,249) (416,416) (582,416) (582,582) (749,582) (749,749) (915,749) (915,915)
seed 600 (held-out): (249,83) (249,249) (416,249) (416,416) (582,416) (582,582) (749,582) (749,749) (915,749) (915,915)
seed 601 (held-out): (similar zigzag shape, slightly longer)Seeds 0, 5, and 600 produced the exact identical point sequence. Three different mazes, one memorised generic answer. Not "close but imperfect navigation." The model is not looking at the image at all. It is reproducing "the general shape maze solutions tend to have in this training set" as a pure language pattern, completely independent of what maze is actually shown.
The obvious next move would have been "train longer" or "increase LoRA capacity." But neither is well-motivated without knowing where the image's distinguishing information actually gets lost. So instead: measure it directly, at each stage of the pipeline, on the same set of maze images.
patch_embeds = vlm.vision_encoder(pixel_values) # raw, frozen ViT output
vision_embeds = vlm.projector(patch_embeds) # after the trained projectorPairwise cosine similarity between five different maze images, at each stage:
| Stage | Different-maze similarity |
|---|---|
Raw ViT patch_embeds | ~0.48 to 0.58 |
Projected vision_embeds | ~0.80 to 0.84 |
(Confirmed first that the input images themselves genuinely differ: pixel-level mean absolute difference 0.05 to 0.12 across seeds, so this is not an input bug.)
The frozen ViT differentiates mazes reasonably well: similarity well below 1.0. The projector then compresses much of that differentiation away, pushing already-different images into embeddings that look substantially more alike than what it was given.
A plausible explanation: the projector was trained in stage 1 entirely on natural Flickr8k photographs. People, animals, scenes. Maze line-drawings, flat white backgrounds, thin black lines, no texture or colour, are wildly out-of-distribution for whatever the projector learned to key off of. It may be folding many different synthetic diagrams into a fairly generic region of Qwen's embedding space, simply because it never learned what distinguishes one maze from another.
Combined with LoRA rank 8 (not much room for Qwen's attention to amplify whatever weak distinguishing signal does survive the projector), the two effects compound into the fully collapsed, image-independent output observed.
Talking through why increasing rank might help surfaced a cleaner way to describe what rank actually is.
LoRALinear is structurally an autoencoder bottleneck sitting next to the frozen layer. lora_A compresses the input down to rank numbers (the encoder). lora_B expands those numbers back out to the real output size (the decoder). Rank is the width of that narrow waist in the middle: not a target size to "expand to," but the number of independent directions of adjustment allowed through the pinch point before being blown back up to full size.
No matter how the expansion step is done, it can never encode more independent patterns of variation than the waist allowed through. That is the whole mechanism behind why rank caps correction complexity. Rank 8 means the LoRA adapter can only ever express adjustments that live in an 8-dimensional subspace, regardless of the size of the matrices on either side.
Bumped LORA_RANK from 8 to 32, a 4x increase, still tiny relative to full fine-tuning, and retrained.
This targets the second half of the compounding problem (giving attention more room to use whatever signal exists) without yet touching the projector side, which the diagnostic points at as the more likely root cause. Worth checking whether more attention capacity alone moves the needle before deciding whether the projector itself needs more or different training.
Once the rank-32 run finishes: re-check GPU memory (rank 32 quadruples LoRA's parameter count, though it remains small overall), then re-run the same four-seed generation test.
If the model now differentiates mazes: the rank was the bottleneck, and the projector's signal survived after all. If it still produces the same generic path: the projector's signal-narrowing is the harder bottleneck and needs addressing directly, with more projector training, maze-specific alignment data in stage 1, or both. That result will determine the next move cleanly.