A personal project building an autonomous quadcopter from scratch — simulation, deep RL, and a custom physical build.
This project began with a simple question: could I train a drone to hover using reinforcement learning, then transfer that learned policy onto a real drone I built myself?
No pre-built frameworks. No autopilot. Just a physics simulation, a neural network, and eventually a 250mm carbon fibre quadcopter with motors wired directly to a Raspberry Pi.
I'm a Senior AI Engineer by day, and this is the project I work on in the background — partly because autonomous systems genuinely interest me, partly because building something physical you can hold in your hands is a completely different kind of satisfaction to deploying a model behind an API.
Before any RL could happen, I needed a physics environment that actually modelled a real drone. I built a custom Gymnasium-compatible environment in PyBullet, parameterised specifically for the hardware I planned to build:
The observation space is 16-dimensional: position (x, y, z), velocity, roll/pitch/yaw, angular velocity, and distance to target. The action space is 4-dimensional — one throttle command per motor.
The first serious problem was a broken thrust formula. The original KT-based calculation produced a MAX_THRUST of approximately 0.00000004 N — meaning the drone would have needed a throttle value of 15,770,589 to hover, which is completely outside the action space. Every training run produced flat reward because motor commands had literally zero physical effect.
The fix was to back-derive from known physics:
MAX_THRUST = (DRONE_MASS * GRAVITY) / (4 * 0.5) # hover at throttle=0.5One line. That single change converted training from completely non-functional to producing positive reward at episode zero.
The first algorithm I trained was Soft Actor-Critic (SAC) with Prioritised Experience Replay (PER).
The actor (FCGP) is a dense skip-connection network with BatchNorm on the state input. The critic (FCQV) uses twin Q-networks with action injected at every hidden layer rather than only at the input. The replay buffer uses rank-based priority weighting, and a UTD (update-to-data) ratio of 4.
Best result: ~+4.07 average reward at episode 900.
Getting there wasn't smooth. A few things that broke along the way:
Q value overestimation was the persistent headache. Q values would collapse to -25 on longer runs. The root cause was the out_of_bounds threshold being set at 5m — the drone could drift laterally for a long time, accumulating exponential penalties that corrupted the Q targets. Tightening the boundary to 1.5m fixed the instability.
A checkpoint save/load bug froze the alpha parameter on resume — the alpha optimiser state wasn't being saved, so every time training continued from a checkpoint, alpha reset and froze. Small bug, significant impact on long training runs.
The airtime bonus was the single most impactful reward shaping change of the whole project — adding a small reward just for staying airborne turned consistently negative episodes into positive ones from the very start of training.
While SAC was sitting at +4 reward after 900 episodes, I tried Twin Delayed DDPG (TD3) with PER as a comparison.
It's not a tuning problem — it's an algorithm-task fit problem. Hover is fundamentally a deterministic stabilisation task: there's one right answer (keep the drone level at 1m) and the agent needs to learn to execute it consistently. SAC's entropy maximisation encourages exploration and stochasticity, which actively works against that. TD3 has no such pressure — it just learns the deterministic policy directly.
This doesn't mean SAC is wrong for the project. For more complex navigation tasks — following a moving target, avoiding obstacles, handling wind disturbance — SAC's exploration properties become an advantage. The plan is to use TD3 for hover/stabilisation and SAC for higher-level navigation, consistent with how published robotics RL papers approach this kind of hybrid task structure.
After the TD3 comparison, I went back to the SAC implementation and into the literature. The DroQ paper (Hiraoka et al., 2021) makes a counterintuitive point: naively increasing the UTD ratio on vanilla SAC makes things worse, not better.
The reason is that Q estimation bias compounds with every additional critic update. Without regularisation, cranking UTD causes the Q networks to diverge. The key insight from the paper is that the instability is entirely a critic problem — the actor doesn't bootstrap off its own outputs, so it's not affected.
Critic architecture changes:
LayerNorm on hidden layer outputsDropout(p=0.01) on hidden layers (per paper — not 0.05)BatchNorm1d on the raw state input staysTraining loop restructure:
The result was an immediate, noticeable improvement in training behaviour. Q metrics stabilised, and the reward curve started climbing more consistently than previous SAC runs.
The most experimental thread of the project is a hybrid architecture combining NEAT (NeuroEvolution of Augmenting Topologies) with SAC. Instead of fixing the network topology and training weights, use NEAT to evolve the topology itself, while SAC trains the weights within each candidate architecture. NEAT's fitness function is the RL reward.
As a preliminary step I wrote a Lua NEAT implementation from scratch as a learning exercise — just to deeply understand the algorithm before trying to hybridise it. That turned up several interesting bugs in my own understanding of crossover and speciation.
The next implementation milestone is the genome-to-PyTorch-actor bridge — converting a NEAT genome into a valid PyTorch module that SAC's training loop can then optimise.
The simulation is always the means, not the end. The goal is to deploy the trained policy on a drone I'm building myself.
| Component | Part |
|---|---|
| Frame | ZMR250 250mm carbon fibre |
| Motors | T-Motor VELOX 2207 1950KV ×4 |
| ESC | Skystars 45A BLHeli32 4-in-1 |
| Props | Tarot 5045 tri-blade |
| Battery | HRB 2200mAh 4S LiPo |
| Companion computer | Raspberry Pi Zero 2W |
| Altitude | VL53L1X laser lidar |
| Orientation / angular velocity | BNO085 IMU |
| Lateral velocity | PMW3901 optical flow |
The key design decision: no flight controller. Most drone projects use a Pixhawk or Betaflight FC as the stabilisation layer. I'm not doing that. The Raspberry Pi talks directly to the ESCs, and the trained RL policy is the only control loop.
When the physical build is complete, the sim-to-real transfer is essentially an observation source swap:
| Simulation | Real hardware |
|---|---|
p.getBasePosition() | VL53L1X altitude + PMW3901 optical flow |
p.getBaseVelocity() | PMW3901 + BNO085 integration |
| Roll / pitch / yaw | BNO085 |
| Angular velocity | BNO085 gyroscope |
The policy itself doesn't change. Just what feeds into the observation array.
The simulation right now is perfect. PyBullet gives the agent exact position, velocity, and orientation with zero noise. Real sensors don't work that way — the BNO085 drifts, the VL53L1X has measurement noise, the PMW3901 optical flow has latency. A policy trained on perfect information will struggle the moment it meets real hardware.
The next step before any physical deployment is domain randomisation — making the simulation deliberately imperfect so the policy learns robustness rather than memorising ideal conditions:
Once the policy trains robustly across a wide range of randomised conditions, the real hardware's imperfections fall within the distribution it's already learned to handle.