Building a Convolutional Neural Network From Scratch in C++

The Starting Point

In August 2019 I built a convolutional neural network in C++. No Python, no PyTorch, no TensorFlow. Just Eigen for matrix operations and OpenCV for image loading.

Every convolution, every activation function, every derivative, every weight update written by hand. No autograd. No automatic differentiation.

The task was binary image classification. Cat or dog.

The Architecture

The network is a multi-stage CNN with the following structure:

Input image (100x100 greyscale, via OpenCV)
    Conv Layer 1: 4 filters, 5x5 kernel, stride 4
    Max/Avg Pooling 1
    Conv Layer 2: 4 filters, 2x2 kernel
    Max/Avg Pooling 2
    Conv Layer 3: 2 filters, 2x2 kernel, stride 2
    Flatten
    Fully Connected Layer 1:  8 → 10 neurons
    Fully Connected Layer 2: 10 → 10 neurons
    Fully Connected Layer 3: 10 →  2 neurons
    Softmax output

The output is a two-class softmax: probability of dog vs probability of cat. The training loop runs for 1000 epochs on a single image.

The Components

Every component is a separate class with its own .cpp and .h file.

Training: Activation Functions and Derivatives

The Training class is the mathematical foundation. It implements every activation function and its derivative from scratch.

Sigmoid:

double Training::fncSigmoid(double n) {
    return 1.0 / (1.0 + exp(-n));
}
double Training::fncSigmoidDerivative(double n) {
    return n * (1 - n);
}

Swish (alternative to ReLU, included as an option):

double Training::funcSwish(double n) {
    return n / (1.0 + exp(-n));
}

Softmax and its derivative (for the output layer), categorical cross-entropy, and cross-entropy derivative are all implemented explicitly. The softmax derivative uses the full formula:

double x1 = (exp(softmaxVal_1) * exp(softmaxVal_2)) / pow((exp(softmaxVal_1) + exp(softmaxVal_2)), 2.0);

CostFunction: Loss and Gradients

The CostFunction class implements MSE per output neuron, overall network cost, and cost derivatives:

void CostFunction::costRes_1() {
    double res = (1.0 / (2.0 * 2.0));
    double out1 = actual_out[0] - network_out[0];
    double out2 = actual_out[1] - network_out[1];
    double res_3 = res * (pow(out1, 2.0) + pow(out2, 2.0));
    costdat = res_3;
}

The cost derivative per output neuron feeds directly into the backpropagation pass as the error signal.

Convolve: Convolution Layers

The Convolve class implements three successive convolution stages. Each stage:

1. Initialises filter weights from a uniform random distribution, scaled by the number of filter parameters (dividing by 25 for 5x5 filters, by 4 for 2x2 filters) to prevent activation saturation
2. Slides the filter across the input using nested loops
3. Extracts input chunks using Eigen's .block() method
4. Computes element-wise multiplication and sums into a pre-activation value
5. Applies sigmoid activation to every element of the result
6. Stores the output into the corresponding feature map

Filter weights persist between forward passes and are updated during backpropagation. The data1, data2, data3 vectors store the weights for each convolution stage respectively.

Output size calculation is noted explicitly in comments:

//outputsize calc = (inputwidth - filterwidth)/stride + 1

The first convolution takes the raw 100x100 greyscale pixel array (loaded via OpenCV) and produces 4 feature maps. Each subsequent stage narrows the spatial dimensions while the filter weights learn to detect increasingly abstract features.

LayerMaxPooling and LayerAvgPooling: Pooling

Both max and average pooling are implemented. The pooling classes use Eigen's .block() to extract windows and either .maxCoeff() or .mean() to reduce them.

Layer: Fully Connected Layers

The Layer class handles three fully connected layers. Each forwardPropagate function initialises weights on the first pass (scaled by number of inputs), builds weight and activation matrices using Eigen, computes dot products between activation vectors and weight slices, applies sigmoid activation, and stores neuron outputs for the next layer and for backpropagation.

The three FC layers go 8 → 10 → 10 → 2 neurons, with the final two values passed to softmax for class probabilities.

Backpropagation

Backpropagation is implemented manually through both the fully connected layers and the convolutional layers.

For the FC layers, the weight update rule is:

new_weight = associated_weight + (learning_rate * fncSigmoidDerivative(out_data));

For the convolutional layers, the derivative is propagated back through the flattened feature maps, with the weight update scaled by the number of filter parameters:

sum = (1.0 / (25.0 * sum));
weights_temp[weights_loop] = weights_temp[weights_loop] + learning_rate * sum;

The learning rate is set to 0.5 throughout.

The Training Loop

The full pipeline runs in main.cpp in a loop of 1000 epochs:

while (epochs < 1000) {
    // Load image with OpenCV
    Mat img_gray = imread("1.jpg");
    cvtColor(img_gray, img, cv::COLOR_BGR2GRAY);

    // Forward pass through conv stages and pooling
    conv.convole1(darray, epochs);
    maxpooling.poolConv(conv.featureMapData1[i], epochs);
    conv.convolve2(maxpooling.pooledConv[i], i, epochs);
    ...

    // Flatten and fully connected
    conv.flatten(epochs);
    LayerFunc.forwardPropagate(conv.Flattened_features);
    LayerFunc.forwardPropagate2(LayerFunc.firstLayerData);
    LayerFunc.forwardPropagate3(LayerFunc.secondLayerData);

    // Softmax output
    trn.funcSoftmax();

    // Cost
    cst.costRes_1();

    // Backpropagation
    LayerFunc.backpropagation(cst.derivative_cost, trn.softmax_derivative_values, conv.Flattened_features);
    conv.backpropagation(maxpooling.pooledConv1, maxpooling.pooledConv);
}

The console outputs predictions and overall network cost at each epoch:

Predictions:
42.731% sure its a dog
57.268% sure its a cat
Overall Network Cost: 0.0812...

What This Project Taught

Memory management matters. In Python, tensors clean themselves up. In C++, every pointer allocation needs thinking about. The use of double** for the average pooling output and new double*[60] for dynamic allocation are things you simply never touch in Python.

Eigen is not PyTorch. Eigen does linear algebra on fixed-size matrices at compile time. Every matrix dimension has to be known statically. When spatial dimensions change between convolution stages, you cannot just reshape. You have to explicitly declare new matrix types. This constraint forces you to think carefully about spatial dimensions at every step.

Backpropagation is not magic. When you cannot call .backward(), you have to know exactly what derivative you are computing and why. Writing the sigmoid derivative as n * (1 - n) by hand, then applying it at every layer, makes the chain rule concrete in a way that reading about it never does.

The architecture roadmap was ambitious. There is an RNN.h file in the repo, empty, a placeholder. The plan was to extend this into a recurrent architecture. That did not happen at the time. But the intent was there.

Looking Back

This is old code. It has rough edges. The data_loop reset logic in the FC backprop, the epochs == 0 branching to handle first-pass weight initialisation separately, the hard-coded spatial dimensions throughout. None of it would survive a code review today.

But it is also the project that made every subsequent neural network feel comprehensible. When PyTorch came into the picture later, it did not feel like magic. It felt like a very well-engineered wrapper around the things this project built by hand.

next read

Learning NLP From the Ground Up

next read

A Deep Dive Into Reinforcement Learning: Seven Algorithms From Scratch

Full source: github.com/Kikumu/Neural-networks-