Hand-Writing a Vision-Language Model, Part 12: Coloured Walls, Split Learning Rates, and a Sharper Look at the Images Themselves

Testing Whether Colour Helps the Frozen ViT

The hypothesis from Part 11 was that maze images are nearly monochrome, black lines on a white background, while SigLIP was pretrained on rich, colourful natural photographs. Maybe that domain gap alone limits how much signal the frozen ViT can extract, independent of anything about the projector.

Tested cheaply, no training involved. Just re-measured raw ViT similarity across the same five test mazes with different wall-colouring schemes:

Real effect, right direction, but modest. Colour helps a little, not dramatically. Random per-segment colouring won out over the structural scheme, so render_image() got a random_colors option with a color_seed parameter (deterministic per maze), and MazeDataset now bakes it in.

Three Changes at Once, and a Confusing Result

Combined three things for the next full retrain: reverted to the MLP projector (the attention swap made things worse in Part 11), baked in random-coloured walls, and gave the projector its own higher learning rate (5e-4 versus LoRA and embedding's 1e-4) via separate AdamW parameter groups. The rationale: the projector needs to unlearn a Flickr8k-photograph bias while LoRA learns the task from scratch, so it may need to move faster.

Result: 0.9089 similarity. Worse than the original unmodified MLP baseline (0.80 to 0.84) from before any of this session's changes.

Worth being honest about the real methodological problem this surfaced: three things changed at once, colour, learning rate, and a completely fresh stage 1 retrain from a new random initialisation. There is no way to isolate which one hurt, or whether a fresh random initialisation alone introduces enough run-to-run variance to explain the swing. Flagged plainly rather than pushing forward with more untested changes stacked on top.

A Sharper Hypothesis: Not "No Colour," But "Mostly Blank Canvas"

Rather than test for run-to-run noise next, tested a much more specific version of the data hypothesis.

Most of a maze image is blank interior space. Only the patches that actually straddle a wall line carry any differentiating information at all. A patch sitting inside an open cell looks like blank white in every maze, regardless of that maze's actual layout. Comparing all 196 patches equally, as every similarity measurement so far had done, means the share of trivially-identical blank patches could be diluting whatever real differentiation exists in the minority of wall-containing patches.

Tested directly, no training involved. Classified each of the 196 ViT patches as "contains wall or marker content" or "always blank" using the raw pixel array before normalisation. A patch counts as wall-containing if any pixel in it deviates meaningfully from pure white. Then measured cosine similarity separately within each group:

Confirms part of the intuition directly. Wall-containing patches genuinely carry more differentiating signal than blank ones (0.41 versus 0.53). A real dilution effect, not imagined.

But it is a modest gap, not the dramatic "blank patches are trivially near-1.0 identical" that would fully explain the collapse. Even isolating to only the informative patches, 0.41 similarity is still a long way from the 0.80 to 0.9989 collapse measured after training. The projector and training process are independently compounding the problem on top of whatever mild dilution exists in the raw data. Both effects are real and probably additive, but neither one alone is the whole story.

One genuine surprise: approximately 46% of patches (90 to 92 out of 196) turned out to contain wall content. A much higher fraction than assumed going in. "Mostly blank canvas" was the working assumption before actually measuring. Worth remembering: check an assumption about the data directly before treating it as established, even when it sounds obviously true.

Where This Leaves Things

Real, measured, but partial explanations stacking up:

• Mild dilution from blank patches, confirmed this session
• The projector's inherent inability to relate patches to each other, Part 4's original concern, still unresolved
• Training-recipe and initialisation variance, still unquantified

None of these individually accounts for the full severity of the collapse. The next step before attributing any more swings to specific deliberate changes: measure run-to-run variance directly.

What Is Next

Measure run-to-run variance to establish a proper baseline for how much these similarity numbers move on their own before treating the next result as meaningful. Then decide which single variable to change next, not three at once.