A prototype born from the NEAT thermal camera project. What if you could generate a depth-aware image of a building interior using its location as a guide?
The NEAT thermal camera project could identify heat signatures through smoke. It draws contours around objects based on thermal intensity. But contours alone have a limitation: they tell you where something is, not what it looks like in three-dimensional space. They are flat outlines, not depth-aware representations.
The question this project asks is: can we generate a plausible image of what a scene looks like, conditioned on where in the world that scene is?
The idea is inspired by what Google Street View does: marrying location metadata to a visual representation of a place. Except the goal here is not a clean outdoor photograph. It is a real-time generated overlay that could sit on top of thermal contours and give depth context to what the contours are outlining.
Think of it as breadcrumbs. You feed the network location coordinates for a known building. The network generates an image that reflects what that building's interior might look like. That generated image then marries onto the NEAT contours, giving a firefighter or emergency responder a richer picture of what they are navigating through, even when the scene is obscured by smoke.
This is a prototype. An incomplete one. But the idea is sound and worth documenting.
A standard GAN generates images from random noise. There is no control over what gets generated. A Conditional GAN (cGAN) introduces a label that the generator is conditioned on. The label steers what the generator produces.
In this case the label is location data: coordinates, building identifiers, spatial metadata. The kind of information that acts as a fingerprint for a specific place. The generator takes both a random latent vector and the location label, and is trained to produce an image that is coherent with that location.
The discriminator also sees the label. It learns to reject images that do not match the label, not just images that look unrealistic in general. Both networks are aware of the location context.
The dataset for this prototype is 704 images from a custom photo collection, preprocessed to 126x126 and normalised to the range [-0.5, 0.5].
Each image is converted to a numpy array, transposed to channel-first format, and saved to disk. The dataloader reads these numpy arrays back as tensors at training time, pairing each image with its label vector.
Both networks are adapted from the DCGAN face generation work, extended with conditioning layers.
The generator (generator_upsample) takes two inputs: a 10-dimensional latent vector drawn from a uniform distribution, and a 384-dimensional label vector representing location. Each input has its own conditioning branch.
Latent branch: a linear layer projects the 10-dimensional latent vector to a 4x4 spatial feature map.
Label branch: an embedding layer maps the 384-dimensional label to a 100-dimensional representation, which is then projected to a separate 4x4 spatial feature map.
Both branches are concatenated along the channel dimension, giving a 1024-channel 4x4 feature map as input to the upsampling body. The body then progressively upsamples through six stages:
1024 → 512 → 256 → 128 → 64 → 32 → 3 channels (RGB, 126x126)Each stage uses upsample-then-convolve (factor of 2) to avoid checkerboard artefacts. The label embedding means the generator is not just producing a random image: it is being nudged toward a particular kind of scene based on the conditioning signal.
The discriminator (discriminator_conv) also takes two inputs: a 126x126 RGB image and the 384-dimensional label vector.
The label conditioning branch uses an embedding layer to project the label to a 10-dimensional representation, then a linear layer expands this to match the spatial dimensions of the image (47,628 values, reshaped to 126x126). This conditioning map is concatenated with the image as an additional channel, giving a 6-channel input (3 RGB + 3 from the label conditioning).
The discriminator then runs seven convolutional stages with stride 2, progressively halving spatial dimensions while increasing channel depth from 6 to 1024, outputting a single real/fake score.
Gaussian noise injection with epoch-based decay (decay_gauss_std) is applied throughout, carrying over the same stability mechanism established in the DCGAN work.
Both gradient flow plots are monitored throughout training (plot_grad_flow), for generator and discriminator independently. The same lesson from the DCGAN project applies here: discriminator gradient health determines the quality of signal the generator receives. If the discriminator is not learning cleanly, the generator receives no useful signal regardless of how well it is architected.
This prototype is unfinished. The current labels are random noise, which means the network is learning to be conditioned on a signal that carries no real meaning. It cannot currently demonstrate the core thesis of the idea.
The real work remaining:
Building a meaningful location encoding. The 384-dimensional label vector needs to be derived from actual spatial data: GPS coordinates, building footprint metadata, or a learned embedding from architectural drawings or floor plans. The vector needs to encode something that meaningfully differentiates one location from another.
Post-generation cleaning. A building on fire does not look the same as a building in a clean Street View photograph. Whatever the generator produces needs to be adapted to degraded conditions: smoke, heat distortion, partial occlusion. This is not a trivial problem.
Integration with the NEAT contours. The generated image is ultimately meant to be an overlay, not a standalone output. Marrying the generated depth representation onto the thermal contours from the NEAT project requires careful alignment between the two modalities.
This is a project I intend to return to with a fresh perspective. The idea is genuinely novel and the architecture to pursue it is in place. What it needs now is real location data and a clearer pipeline from spatial metadata to conditioning vector.
Alongside the main LBCGAN work, a separate notebook was started to investigate how to properly evaluate the quality of the generated output, specifically in the context of fusing it with thermal contours.
The issue with standard distribution metrics like KL-Divergence is that they only measure the difference in pixel value distributions between two images. They do not care about where those pixels are spatially. A generated image could have exactly the right colour distribution but a completely wrong spatial layout, and KL-Divergence would not catch that.
For a system where the generated image has to align spatially with real-time thermal contours, that is a serious problem. A loss function that does not account for spatial structure cannot tell you whether the generated scene is actually coherent with the thermal overlay.
The investigation started with image brightness distribution as a baseline, then moved toward methods that account for spatial pixel distribution rather than just value distribution. This work is early stage but directly relevant to the post-fusion quality problem. It is the beginning of answering: once the two systems are talking to each other, how do you measure whether the result is actually good?
This project sits at the intersection of three things: generative modelling from the DCGAN work, evolutionary spatial adaptation from the NEAT thermal camera work, and location-aware conditioning.
The vision is a system where NEAT evolves to identify heat signatures and draw contours in real time, LBCGAN generates a location-conditioned scene representation, and the two are fused into a single overlay that gives depth and context to what the thermal camera is seeing.
None of that is finished. But the components exist. The NEAT implementation works. The conditional GAN architecture trains. The remaining work is the glue between them and the quality of the conditioning signal.
Full source: github.com/Kikumu/LBCGAN