The two halves of the project are now connected and forward-pass verified. Then the more valuable part: actually understanding what SigLIP's training does, and what the wrong turns looked like.
Three small pieces finished the wiring.
1. qwen3.model.Model.forward now accepts inputs_embeds directly:
def forward(self, input_ids=None, inputs_embeds=None):
if inputs_embeds is None:
inputs_embeds = self.embed_tokens(input_ids)
x = inputs_embeds
...This lets something else compute the embedding sequence and hand it in directly, bypassing Qwen's own token lookup. Necessary because a vision patch embedding was never a vocabulary ID to begin with. There is nothing for embed_tokens to look up for that part of the sequence.
2. The VLM glue class in vlm/model.py:
def forward(self, pixel_values, input_ids):
patch_embeds = self.vision_encoder(pixel_values)
vision_embeds = self.projector(patch_embeds)
text_embeds = self.qwen_model.embed_tokens(input_ids)
inputs_embeds = torch.cat([vision_embeds, text_embeds], dim=1)
return self.qwen_model(inputs_embeds=inputs_embeds)Simple prepend-and-concatenate. Vision patches come first, text tokens follow. No new special token needed anywhere. The VisionTransformer is frozen, the Projector is trainable, qwen3.model.Model is frozen.
3. scripts/verify_vlm.py confirmed:
Output shape (1, 204, 151936): 196 vision patches plus 8 text tokens, exactly as expected. The freeze split is correct: ViT and Qwen both fully frozen, Projector the only piece with requires_grad=True anywhere.
Before touching stage-1 training, the goal was to make sure "SigLIP produces embeddings such that matching pairs end up close together" was actually understood, not just repeated back. The wrong turns are worth documenting because each one was a reasonable guess that turned out to be subtly off.
Given the project's eventual goal involves <point> and <box> outputs, it seemed plausible SigLIP itself produced bounding boxes or coordinate-shaped outputs.
It does not. SigLIP is an image-text matching model (like CLIP, with a sigmoid loss instead of a softmax contrastive loss), and its vision tower outputs generic per-patch feature vectors. Nothing box or coordinate shaped. The eventual <point> and <box> text output is entirely Qwen's job, learned later from labelled training data. The ViT has no concept of "object" at all.
The assumption was that SigLIP links individual words to specific image regions: "landscape" pointing at the sky, "person" pointing at a face. It does not.
The whole caption collapses into one vector. The whole image collapses into one vector (that is what the attention-pooling head, which is skipped in this project, is for). The comparison is entirely global: does this whole-image vector match this whole-caption vector. No per-word, per-region grounding happens anywhere in SigLIP.
The guess was that image and text might get processed together through some shared layer, since they end up "in the same space."
Actually two completely separate networks: a vision tower and a text tower, confirmed directly from the vision_model.* and text_model.* split in the checkpoint back in Part 3. Zero interaction during encoding. "Same space" only means both towers' final outputs happen to be the same width (768), by design. Like converting Celsius and Fahrenheit both to Kelvin: totally different conversion processes, comparable only once they land in the same final unit.
Cosine similarity and dot product are just arithmetic. They work on any two same-sized vectors, trained or not. What training actually does is sculpt both towers' weights so that this fixed, untrained-in-itself operation becomes meaningful: pushing matching-pair vectors' dot product up, mismatched-pair vectors' dot product down, over millions of examples, until "high dot product" reliably means "these actually match."
Nothing about the operation itself needs training. What needs training is for the vectors it is applied to to encode compatible information in the first place. This is exactly the same problem the Projector solves. Two independently-trained models do not share a coordinate system for free.
A reasonable-sounding assumption: the projector decides which patch information is relevant and filters accordingly.
Structurally wrong. A plain Linear layer transforms each of the 196 patch vectors independently. Patch 47's output cannot be influenced by patch 12's input. There is no mechanism for one position to see another. The projector is not selecting or deciding anything. It is closer to a dialect translation: remapping each vector from SigLIP's 768-dimensional coordinate system into something that looks like a plausible token embedding in Qwen's 1024-dimensional coordinate system. The actual "what is relevant to what" job belongs entirely to Qwen's own attention, later, once vision and text tokens sit together in one sequence.
By analogy to SigLIP's own contrastive training (which needs both matching and mismatching pairs), the assumption was that stage-1 training would require labelled true/false data.
It does not. Stage 1 uses ordinary next-token prediction loss, the same mechanism used to pretrain Qwen on plain text. It only ever needs the correct target sequence. There is no negative-example step. The loss measures how far the model's predicted next-word distribution was from the one real answer, at every position in the caption. Stage 1 only needs (image, real caption) pairs. Nothing true/false about it.
| Stage 1: Projector Alignment | Stage 2: Visual Primitives Fine-tune | |
|---|---|---|
| Frozen | ViT + Qwen | ViT only |
| Trained | Projector only | Qwen via LoRA, likely Projector too |
| Data | (image, real caption) pairs, generic | (maze image, reasoning + coordinates) pairs, custom |
| Loss | Next-token prediction (cross-entropy) | Next-token prediction (cross-entropy) |
| Goal | Make vision embeddings interpretable to Qwen | Teach Qwen to reason about and point at things in images |
Same loss mechanism both times. What changes is which weights are unfrozen and what the target text looks like.
Plan stage-1 training properly: pick an image-caption dataset, design the training loop, decide hyperparameters within the 8GB RAM and VRAM ceiling, then move to stage-2 LoRA fine-tuning on the maze dataset.
Only after the full training pipeline is built and working does the MLP Projector get swapped for an attention-based one (Q-Former or Perceiver Resampler style) as a deliberate experiment to compare against the MLP baseline. Deliberately sequenced last, not next.