A from-scratch NEAT implementation that tunes its own colour space parameters to identify objects through smoke — no labels required.
Thermal cameras see heat, not light. That makes them genuinely useful in smoke-filled environments where a standard camera sees nothing. Firefighters use them to identify people, obstacles, and hotspots when visibility is zero.
But raw thermal data is just a temperature grid. To make it actionable, you need to extract structure from it: contours, boundaries, shapes. The challenge is that the HSV colour space parameters for meaningful contour detection vary dramatically depending on the scene. A person in a cool room looks completely different from a piece of burning furniture. Static thresholds break immediately.
The question I wanted to answer: could a genetic algorithm learn to tune those parameters on the fly, adapting its colour space mask to whatever it was seeing?
The thermal sensor is an MLX90640, a 24x32 pixel thermal array running at 8Hz, interfaced to a Raspberry Pi via I2C. Raw output is a 768-value temperature grid. That gets interpolated up to 240x320 for display, then resized to 128x128 before being fed into the network, giving a 16,384-dimensional input.
The output is 6 values corresponding to the HSV trackbar parameters OpenCV uses for colour masking:
These six numbers define the colour range the mask accepts. Get them right and you get clean contours around heat signatures. Get them wrong and you get noise.
The standard approach would be to train a neural network with a fixed architecture. But the problem has an awkward property: you don't have ground truth labels. There's no dataset of "correct" HSV parameters for any given thermal frame. You can't supervise this.
What you can do is define a fitness function that measures output quality without needing labels. If the contours the mask produces are clean and low-noise, the parameters are good. If they're noisy and fragmented, they're bad.
NEAT (NeuroEvolution of Augmenting Topologies) is a good fit here because it:
The entire NEAT implementation is hand-rolled. No library, no framework. Every class, every mutation operator, every crossover function written from scratch. The full source is on GitHub: github.com/Kikumu/Raspberry-Pi-Thermal-cam-g-a.
Neuron: stores a value, a status (input=0, output=1, hidden=2), and a list of gene indices that connect to it.
connectionGene: a single weighted connection between two neurons. Carries an innovation number for ancestry tracking, an enable/disable flag, and a weight. This is NEAT's core data structure. The innovation number is what allows meaningful crossover between topologically different genomes.
createNewGenome: the genome class. Holds a gene dictionary (keyed by innovation number) and a neuron dictionary. Also carries all the genome's own mutation rates — link mutation chance, node mutation chance, perturbance, step size — which themselves evolve over time.
Four distinct mutation operators run each generation:
Point mutation (pointMutate): perturbs existing weights by a small step value (90% of the time) or replaces them entirely with a random value in [-2, 2] (10% of the time). The perturbance rate and step size both self-adapt each generation.
Node mutation (mutateNodeGeneInGenome): takes an existing enabled connection, disables it, and inserts a new neuron in the middle. The input-to-new connection gets weight 1.0 (preserving existing behaviour). The new-to-output connection inherits the original weight. This is how NEAT grows structure.
Link mutation (mutateConnectionGeneInGenome): samples two random neurons and, if they don't already share a connection, creates a new one with a random weight. Tries up to 5 times before giving up. This is how NEAT adds capacity.
Activation mutation (enable_disable_genes_candidacy): randomly enables or disables existing connections, letting the network effectively prune pathways that aren't contributing.
There's also a sign flip mutation (flip_sign_chance) that flips the sign of existing weights. Controlled by separate positive and negative flip probabilities that self-adapt independently.
When two genomes breed, matching genes (same innovation number) are inherited randomly from either parent. Disjoint and excess genes — those that only appear in one parent — are inherited from the more fit parent. This is the key NEAT insight. Innovation numbers act as a genetic timestamp that allows meaningful alignment between networks with completely different topologies.
The child genome's neuron dictionary is built by merging both parents' neuron dictionaries, then resetting all neuron values to 0.
Genomes are grouped into species based on compatibility distance, a weighted combination of disjoint genes, excess genes, and average weight difference between matching genes. The constants are tunable:
disjointmentConstant = 0.3
weightImportanceConstant = 0.84
excessGenesConstant = 0.5
speciesDistanceLimiter = 3Species protect innovation. A genome with a new structural change will initially perform poorly. Speciation gives it room to improve within its niche before competing against the full population. Without this, structural innovations get killed off before they have time to show their value.
This is the part that ties everything together. The genome's six output values get applied directly to OpenCV's HSV trackbars, generating a colour mask over the thermal image. The mask then goes through Canny edge detection to produce contours.
Fitness is measured as the inverse of image noise. Two metrics:
estimate_sigma from scikit-image: estimates noise standard deviation in the masked imagecv2.Laplacian variance: measures edge sharpness and focusA clean mask over a real heat signature produces sharp, well-defined contours with low noise. A bad mask produces fragmented garbage. The genetic algorithm never sees the thermal image directly. It just sees a score that says how clean its output was.
PopulationSize = 20
NumberOfGenerations = 800
Exposure = 10 # seconds per genome evaluation
Steps = 2 # feed-forward iterations per frameEach genome gets 10 seconds of live thermal input to accumulate a fitness score. The network runs feed-forward inference on each frame, outputs HSV values, generates a mask, scores it, and updates its score. After all genomes in a generation are evaluated, selection, crossover, and mutation produce the next generation.
The mutateInEnvironment flag allows genomes to mutate during their exposure window, not just between generations. This makes the search more exploratory at the cost of evaluation stability.
The system produces an OpenCV window showing the evolving contour mask in real time. Early generations produce noise. The HSV parameters are effectively random and the masks are meaningless. Over generations, fitter genomes produce cleaner separation of heat signatures from background. The network learns, without any labels, what "a meaningful thermal contour" looks like, because the fitness function rewards it for producing one.
Building this before I had deep RL experience is interesting to look back on. A few things stand out:
The fitness function is the hardest part. Getting a signal that genuinely reflects what you want, not a proxy that gets gamed, is the same challenge in RL reward shaping. I'd spend more time on this now before touching the evolutionary loop.
NEAT's sample efficiency is poor. 20 genomes x 10 seconds x 800 generations is a long time for a simple 6-output problem. Modern approaches like CMA-ES or even a simple RL formulation (treat HSV parameters as a continuous action space) would converge faster. That said, NEAT's topology search is genuinely valuable when you don't know what architecture you need, which was the case here.
The from-scratch implementation taught me things a library never would. Every bug I hit — crossover returning the wrong parent's genes, speciation thresholds that never split, mutations that silently did nothing — forced me to understand the algorithm at the level that actually mattered. That understanding is what informed the NEAT+SAC hybrid architecture I later designed for the drone project.
The drone project is the direct descendant of this work. The NEAT+SAC hybrid uses the same evolutionary intuition: NEAT searches topology, SAC trains weights, applied to continuous control rather than parameter tuning. If you want the full context on that, the companion post covers it.
The NEAT algorithm implemented in this project is based on the original published research:
Stanley, K. O., & Miikkulainen, R. (2002). Evolving Neural Networks through Augmenting Topologies. Evolutionary Computation, 10(2), 99-127. nn.cs.utexas.edu/downloads/papers/stanley.ec02.pdf
This is an independent from-scratch implementation of the concepts described in the paper, built in Python for a thermal camera computer vision application.