This session tried three more angles at the model-ignoring-the-image problem. Each one failed cleanly. Worth writing down honestly before figuring out what is actually going on.
Before any of the projector work, a genuinely important catch: every maze so far had used the exact same two corners as start and end, always (0,0) and (width-1, height-1). That is not a detail. It is a free, memorisable fact the model could exploit without ever reading the image at all. "Paths always run top-left to bottom-right" is enough to produce a plausible-looking answer without perceiving anything.
Worth being honest about a real misconception that came up while fixing this: the assumption was that the model somehow knew where start and end were from the image. It did not and could not have, since render_image() never drew anything indicating either position. The model only appeared to know because the answer was a training-set-wide constant.
Fix: Maze.random_start_end(seed) picks two distinct corners deterministically per maze. render_image() now accepts start and end and draws green and red dot markers at their cell centres.
One important question before building this: whether varying the end without marking it would work as a simpler middle ground. It would not. If the end varies and is not indicated anywhere, the task becomes unsolvable in principle. No information anywhere, image or convention, would tell the model where the goal is. Varying something requires marking it; they have to go together.
Hypothesis: maybe the problem is that stage 2 trains too many things simultaneously, LoRA, projector, and point-token overlay, and the projector never gets focused gradient pressure to actually differentiate mazes.
What was built: froze ViT and Qwen completely (no LoRA at all), trained only vlm.projector.parameters() on a simplified label, just the point-coordinate list with no reasoning narration. Narration text is identical across mazes anyway and would dilute the signal. Removing LoRA was deliberate: isolate 100% of gradient pressure onto the projector alone.
Result: pairwise cosine similarity across different mazes' projected embeddings came out to 0.8671, barely different from, if anything slightly worse than, the joint-trained version's 0.80 to 0.84.
What it rules out: it is not "LoRA compensating for a weak projector." The projector alone, given focused training, still collapses maze embeddings.
Best explanation: removing LoRA did not just isolate a training signal, it may have removed the only mechanism that made embeddings usable at all. Frozen Qwen's attention was shaped entirely by natural-language relationships during pretraining. It was never taught to attend differently based on vision-token content. Stage 1's captioning task working at all may have been a property specific to natural photographs, not a guarantee that holds for line-drawing images fed into a completely unadapted Qwen.
Already covered in Part 10 but included here for completeness: quadrupling LoRA rank produced no change in the collapsed output. Seeds 0, 5, and 600 still generated the identical point sequence.
What it rules out: attention capacity was not the bottleneck.
Two separate fixes failing pointed at something structural. The plain 2-layer MLP projector applies to each of the 196 patches independently. It can never let one patch's features be influenced by another's. Telling two maze layouts apart plausibly requires comparing wall configurations across patches, something a per-patch-independent MLP cannot represent regardless of how it is trained.
This was the planned future experiment (Q-Former and Perceiver-Resampler style from Part 4), moved up from "after everything else works" to "the actual fix stage 2 needs."
What was built: AttentionProjector in vlm/projector.py. 32 learned query vectors cross-attend over all 196 patch embeddings, using a hand-written CrossAttention where queries are Q and patches are K and V, unlike self-attention where all three come from the same input. Residual and norm around both the cross-attention and a feedforward block, reusing the same decoder-layer skeleton used everywhere else in the project. The old Projector MLP class was reused as the feedforward block, a nice consolidation.
Since this replaces the projector's actual parameters (different module, different shapes entirely), every existing checkpoint became obsolete. Had to rebuild from scratch: fresh stage 1 on natural photos, then fresh stage 2 on top. train_stage1.py and train_stage2.py needed zero code changes. VLM.forward calls self.projector(patch_embeds) polymorphically, so swapping which class gets instantiated in VLM.from_pretrained was the only change needed anywhere.
One shape change worth noting: 32 query tokens plus 8 text tokens gives 40 total sequence length, down from 196 plus 8. The token compression benefit from Part 4 held up: peak GPU memory during stage 2 dropped from ~6.3GB to ~5.5GB.
Result: measured after stage 2 on the matched comparison, 0.9989. Essentially total collapse, and unambiguously worse than the MLP's number under the same conditions.
What it rules out: per-patch independence was not the (only) problem, and pooling and averaging can be actively harmful for this specific kind of image.
Best explanation: attention pooling has a strong built-in bias toward averaging. 32 queries doing softmax-weighted averaging over 196 mostly-white, sparse-line patches can wash out the small, sparse differences between mazes more thoroughly than an MLP that at least leaves every patch's output untouched. With only 500 examples and 1 epoch, the attention mechanism likely never learned to attend selectively. It probably settled into something close to uniform averaging, a worse failure mode for sparse, low-detail images than the MLP's inability to cross-reference patches ever was.
| Fix tried | Similarity result | What it rules out |
|---|---|---|
| LoRA rank 8 to 32 | Unchanged, identical output | Attention capacity was not the bottleneck |
| Isolate projector, no LoRA | 0.867, no better | Not "LoRA compensating for a weak projector" |
| MLP to attention-based projector | 0.9989, worse | Per-patch independence was not the only problem, and averaging can be actively harmful |
Three different fixes tried, three clean negative results, each ruling something out precisely. Every fix so far assumed the model or training setup was the limiting factor. They were not.
The next hypothesis worth testing before any more architecture or training changes: the maze images themselves may be the limiting factor.
These images are visually bland. Thin black lines on white. No colour, no texture, no gradients. Different mazes may simply not present enough visually distinctive features for a ViT pretrained on natural photographs to differentiate between them, regardless of what happens downstream. SigLIP was trained on photographs of the real world. A flat line-drawing maze is far outside that distribution.
If this is true, no amount of projector architecture tuning or LoRA capacity increase would fix it. The signal simply is not there for the ViT to extract in the first place. That is the hypothesis to test next, before touching anything else.
Test the "maze images are too visually bland" hypothesis directly. If confirmed, the options shift away from downstream architecture changes and toward either making the images richer (colour-coded walls, shading, texture) or fine-tuning the ViT itself rather than keeping it frozen. Worth knowing which of those is true before spending more compute on the wrong layer.