The vision half of the project. A hand-written ViT loaded with real SigLIP weights, verified layer by layer, and the first piece of the bridge to Qwen.
The original plan was to load the ViT via plain AutoModel.from_pretrained and treat it as a black box. Only the Qwen side was supposed to be reimplemented by hand.
Mid-build, that felt wrong. The whole point of this project is to understand what is happening at every layer, not just the language model side. So the ViT got the exact same treatment as Qwen: hand-written architecture, real pretrained weights loaded in, verified against AutoModel as ground truth, then frozen for use in the glue class.
ViT choice: google/siglip-base-patch16-224 — approximately 93M parameters, hidden size 768, 196 patch tokens. Same "develop small first" reasoning as Qwen3-0.6B versus 4B, now that both RAM and VRAM are confirmed 8GB constraints.
Same discipline as Qwen: read the weights before writing any code.
First genuine surprise: the downloaded siglip/model.safetensors is 812MB. A 93M-parameter vision-only model at fp32 should be approximately 372MB. The repo ships the full dual-encoder SiglipModel, both a vision tower and a text tower, since SigLIP is trained as a CLIP-style contrastive image-text model. 408 tensors total. Only 208 (vision_model.*) are needed. The other half (text_model.*, logit_scale, logit_bias) gets ignored entirely.
Second wrinkle: config.json's vision_config only lists patch_size: 16. Everything else is inherited from SiglipVisionConfig's library defaults and is not written to the file at all. Reading the raw JSON directly, as worked cleanly for Qwen, is not enough here. Resolving the effective config requires SiglipVisionConfig.from_pretrained(model_dir), which merges the file's overrides with the class defaults.
Resolved config:
| Parameter | Value |
|---|---|
| num_hidden_layers | 12 |
| hidden_size | 768 |
| num_attention_heads | 12 (head_dim 64, standard hidden_size / num_heads) |
| intermediate_size | 3072 |
| patch_size / image_size | 16 / 224 (196 patches, 14×14 grid) |
| hidden_act | gelu_pytorch_tanh |
| layer_norm_eps | 1e-6 |
Key differences from Qwen confirmed from real tensor shapes before writing any code:
bias=False)nn.Embedding(196, 768) added once at the input, no RoPEConv2d(3, 768, kernel_size=16, stride=16)q_norm or k_norm anywhere (no Qwen3-style per-head norm)vision_model.head.* exists in the checkpoint but is deliberately skipped — a VLM projector needs per-patch embeddings, not a single pooled vectorclass MultiHeadSelfAttention(nn.Module):
def __init__(self, hidden_size, num_heads, bias=True):
super().__init__()
self.num_heads = num_heads
self.head_dim = hidden_size // num_heads
self.scale = self.head_dim**-0.5
self.q_proj = nn.Linear(hidden_size, hidden_size, bias=bias)
self.k_proj = nn.Linear(hidden_size, hidden_size, bias=bias)
self.v_proj = nn.Linear(hidden_size, hidden_size, bias=bias)
self.out_proj = nn.Linear(hidden_size, hidden_size, bias=bias)
def forward(self, x):
batch, seq_len, hidden_size = x.shape
q = self.q_proj(x).view(batch, seq_len, self.num_heads, self.head_dim).transpose(1, 2)
k = self.k_proj(x).view(batch, seq_len, self.num_heads, self.head_dim).transpose(1, 2)
v = self.v_proj(x).view(batch, seq_len, self.num_heads, self.head_dim).transpose(1, 2)
attn_weights = (q @ k.transpose(-2, -1)) * self.scale
attn_weights = torch.softmax(attn_weights.float(), dim=-1).to(q.dtype)
attn_output = attn_weights @ v
attn_output = attn_output.transpose(1, 2).contiguous().view(batch, seq_len, hidden_size)
return self.out_proj(attn_output)Writing this after the Qwen attention block made the contrast immediate. This is much closer to the original textbook attention mechanism. Side by side with qwen3/modules.py, here is everything Qwen3 added that is simply absent here.
No GQA. Equal Q/K/V head counts (12 each), no repeat_kv, no KV-head sharing at all.
No RoPE. Position was injected once via the added position embedding at the input. Nothing rotates Q/K per layer.
No causal mask. Every patch attends to every other patch. There is no "future" to hide in an image the way there is in left-to-right generated text. The whole image is visible at once.
No q_norm or k_norm. That per-head RMSNorm was a Qwen3-specific stabilisation trick, not part of the original attention design.
A useful reminder: GQA, RoPE, and QK-norm are all relatively recent refinements. None of them are load-bearing for attention to function at all. They are optimisations for a specific regime (long autoregressive text generation) that a bidirectional image encoder simply does not need.
class MLP(nn.Module):
def __init__(self, hidden_size, intermediate_size, bias=True):
super().__init__()
self.fc1 = nn.Linear(hidden_size, intermediate_size, bias=bias)
self.fc2 = nn.Linear(intermediate_size, hidden_size, bias=bias)
def forward(self, x):
x = self.fc1(x)
x = torch.nn.functional.gelu(x, approximate="tanh")
return self.fc2(x)
class EncoderLayer(nn.Module):
def __init__(self, hidden_size, num_heads, intermediate_size, layer_norm_eps, bias=True):
super().__init__()
self.layer_norm1 = nn.LayerNorm(hidden_size, eps=layer_norm_eps)
self.self_attn = MultiHeadSelfAttention(hidden_size, num_heads, bias=bias)
self.layer_norm2 = nn.LayerNorm(hidden_size, eps=layer_norm_eps)
self.mlp = MLP(hidden_size, intermediate_size, bias=bias)
def forward(self, x):
residual = x
x = self.self_attn(self.layer_norm1(x))
x = residual + x
residual = x
x = self.mlp(self.layer_norm2(x))
x = residual + x
return xTwo matrices (fc1 and fc2), not SwiGLU's gated three. Plain Linear to GELU to Linear, no gating branch at all. The EncoderLayer reuses the exact same pre-norm and residual skeleton as Qwen's DecoderLayer, x = residual + sublayer(norm(x)), twice. That pattern turns out to be architecture-agnostic. It carries over unchanged from a causal decoder to a bidirectional encoder.
torch.nn.LayerNorm was used directly rather than hand-writing it. Its mechanics (mean-subtract, divide by std, scale and shift) were already the contrast point when RMSNorm was built in Part 1. Writing it again would not have added new understanding.
class VisionTransformer(nn.Module):
def __init__(self, hidden_size, num_layers, num_heads, intermediate_size,
patch_size, image_size, num_channels=3, layer_norm_eps=1e-6, bias=True):
super().__init__()
num_patches = (image_size // patch_size) ** 2
self.patch_embedding = nn.Conv2d(num_channels, hidden_size,
kernel_size=patch_size, stride=patch_size, bias=bias)
self.position_embedding = nn.Embedding(num_patches, hidden_size)
self.layers = nn.ModuleList([
EncoderLayer(hidden_size, num_heads, intermediate_size, layer_norm_eps, bias=bias)
for _ in range(num_layers)
])
self.post_layernorm = nn.LayerNorm(hidden_size, eps=layer_norm_eps)
def forward(self, pixel_values):
patch_embeds = self.patch_embedding(pixel_values).flatten(2).transpose(1, 2)
num_patches = patch_embeds.shape[1]
position_ids = torch.arange(num_patches, device=pixel_values.device)
x = patch_embeds + self.position_embedding(position_ids)
for layer in self.layers:
x = layer(x)
return self.post_layernorm(x)The patch embedding is the one genuinely new mechanical idea here versus anything in the Qwen side. A single strided Conv2d with kernel_size == stride == patch_size slices a 224x224 image into a 14x14 grid of non-overlapping 16x16 patches and linearly projects each one to a 768-dimensional vector in one operation. The convolution's stride ensures patches never overlap, so each output position corresponds to exactly one patch.
The from_config and from_pretrained classmethods follow the exact same factory pattern as qwen3.model.Model. Third time reusing that pattern now.
Loaded real weights, ran a forward pass on a (1, 3, 224, 224) dummy tensor. Output shape (1, 196, 768) as expected.
Same style as the Qwen verification: run both models on the same input, compare outputs.
HuggingFace's loader prints a wall of "UNEXPECTED key" warnings when loading SiglipVisionModel from this checkpoint directory. That is just it noting it ignored the text-tower weights, expected given the dual-encoder checkpoint. Not a problem with the hand-written code.
allclose(atol=1e-4) passes.class Projector(nn.Module):
def __init__(self, vision_hidden_size, qwen_hidden_size):
super().__init__()
self.fc1 = nn.Linear(vision_hidden_size, qwen_hidden_size)
self.fc2 = nn.Linear(qwen_hidden_size, qwen_hidden_size)
def forward(self, x):
x = self.fc1(x)
x = torch.nn.functional.gelu(x)
return self.fc2(x)This is the first piece of code in the entire project that touches both halves of the VLM. It maps SigLIP's 768-dimensional patch embeddings into Qwen3-0.6B's 1024-dimensional embedding space. Everything upstream (ViT) and downstream (Qwen) stays frozen. This small MLP is the only thing that gets trained in stage 1: image-caption alignment.
The architecture is deliberately minimal: two linear layers with a GELU activation between them. The ViT and Qwen stacks both have hundreds of millions of parameters encoding rich representations. The projector's job is not to understand vision or language. It is to learn a mapping between two representation spaces that were trained independently. A small trainable bridge between two large frozen systems.
(2, 196, 1024), parameter count matches.Extend qwen3.model.Model.forward to accept inputs_embeds directly, bypassing its own token-embedding lookup. Then the VLM glue class that concatenates projected vision embeddings with Qwen's text embeddings and runs the combined sequence through the frozen Qwen stack.
That is the actual moment the two halves of this project meet.