Implementing and comparing policy gradient, actor-critic, and off-policy methods in PyTorch on OpenAI Gym environments.
Using a library like Stable-Baselines3 to train an RL agent takes a few lines of code. Understanding what is actually happening takes considerably longer.
This repo is the result of going through the major families of deep reinforcement learning algorithms and implementing each one from scratch in PyTorch. No abstraction layers. No pre-built agents. Just the math, translated into code, run against OpenAI Gym environments including Pendulum-v1 and MountainCarContinuous-v0.
The goal was not just to get each algorithm working, but to understand where each one sits in the landscape, what problem it was designed to solve, and how it compares to the others. Seven algorithms implemented in total.
Before the implementations, it helps to have a map. RL algorithms split broadly into two families:
Policy-based methods directly learn a policy: a mapping from states to actions. They optimise the policy directly via gradient ascent on expected reward.
Value-based methods learn a value function: an estimate of how good a state or state-action pair is. The policy is derived from the value function rather than learned directly.
Actor-critic methods combine both. An actor learns the policy. A critic learns the value function. The critic's estimates reduce the variance of the policy gradient, making training more stable.
The seven algorithms here span all three families and represent a natural progression from simple to complex.
The starting point. VPG directly optimises the policy by computing the gradient of expected return and ascending it.
The architecture is two separate networks: a policy network (linearApproximator_FCDAP) that outputs an action distribution, and a value network (linearApproximator_FCV) that estimates the state value. Both are fully connected with configurable hidden dimensions.
The key implementation detail is the return calculation. For each trajectory, discounted returns are computed using:
discounts = np.logspace(0, Trajectory_length, num=Trajectory_length, base=gamma, endpoint=False)
returns = np.array([np.sum(discounts[:Trajectory_length-t] * reward_store[t:]) for t in range(Trajectory_length)])The value network's estimate is then subtracted from the returns to get the advantage: how much better the actual outcome was than expected. This reduces variance without introducing bias.
The policy loss is the negative log probability of the taken action, weighted by the advantage. The value loss is MSE between predicted state values and actual returns.
A2C addresses VPG's single-trajectory limitation by running multiple environments in parallel and synchronising updates.
The implementation uses Python's multiprocessing module with mp.Pipe for communication between worker processes and the main process. Each worker runs its own copy of the environment and sends experiences back via a pipe:
class Multiprocess_env():
def __init__(self, env, n_seeds):
self.env_pipes = [mp.Pipe() for seed in range(self.n_seeds)]
self.envs = [mp.Process(target=self.work, args=(seed, self.env_pipes[seed][1]))
for seed in range(self.n_seeds)]The key difference from VPG: all workers share the same global network, updates are synchronous (all workers finish their rollout before the update happens), and the advantage estimate benefits from multiple environment perspectives simultaneously.
The shared architecture (linearApproximator_A2C) outputs both a policy logit distribution and a state value from a single forward pass, sharing the lower layers between both heads.
A3C takes A2C's parallelism and removes the synchronisation requirement. Workers update the global network asynchronously, without waiting for each other.
The critical engineering challenge is that standard optimisers are not thread-safe. The solution is a custom SharedAdam optimiser where the internal state tensors are explicitly moved to shared memory:
class SharedAdam(torch.optim.Adam):
def __init__(self, params, ...):
super().__init__(params, ...)
for group in self.param_groups:
for p in group['params']:
state['shared_step'] = torch.zeros(1).share_memory_()
state['exp_avg'] = torch.zeros_like(p.data).share_memory_()
state['exp_avg_sq'] = torch.zeros_like(p.data).share_memory_()Each worker runs its own local copy of the policy and value networks, trains locally for max_steps, then pushes gradients to the global network and pulls the updated weights back. Ten workers run in parallel.
Workers are seeded differently (local_worker_seed = main_seed + worker_rank) to ensure they explore different parts of the state space. The asynchronous updates act as a form of implicit experience diversity.
GAE is an upgrade to the advantage calculation, not a new algorithm. It is applied on top of A3C.
Standard advantage: A(s,a) = R - V(s). This is high variance because R is a single sample of the return. GAE introduces a parameter gae_tau (lambda) that interpolates between the high-variance Monte Carlo return and the low-variance (but biased) one-step TD error:
The optimize_model function gains a gae_tau parameter and the advantage computation is modified accordingly. Everything else in A3C stays the same.
In practice, GAE significantly stabilises training on environments with longer episodes where the return estimate is particularly noisy. The trade-off between bias and variance becomes a tunable hyperparameter rather than a fixed property of the algorithm.
The first off-policy algorithm in the set. DDPG is designed for continuous action spaces, where policy gradient methods struggle with the high-dimensional action distribution.
The key insight: instead of learning a stochastic policy and sampling from it, learn a deterministic policy mu(s) that maps states directly to actions. The critic then evaluates state-action pairs.
The actor (linearApproximator_FCDCAP) uses a Tanh output to bound actions to the valid range. The critic (linearApproximator_FCQV) takes the state and action as separate inputs, injecting the action at the first hidden layer rather than at the input:
if i == 0:
hidden_input_layer += self.action_outputs_sizeExploration in a deterministic policy requires explicit noise. This implementation uses Ornstein-Uhlenbeck (OU) noise, which produces temporally correlated noise appropriate for physical control tasks:
x = self.x_prev + self.theta * (self.mu - self.x_prev) * self.dt + \
self.sigma * np.sqrt(self.dt) * np.random.normal(size=self.mu.shape)Target networks are updated via Polyak averaging rather than hard copies, providing smoother learning:
target_weight_update = (1.0 - tau) * target_weights.data + tau * online_weights.dataPrioritised Experience Replay (PER) is layered on top. Rather than sampling uniformly from the replay buffer, experiences are ranked by their absolute TD error and sampled proportionally. Higher-error experiences are replayed more frequently. Importance sampling weights correct for the resulting bias.
TD3 directly addresses DDPG's main failure mode: Q value overestimation.
When the critic overestimates Q values, the actor exploits this by selecting actions the critic incorrectly scores highly. The actor and critic then reinforce each other's errors. TD3 introduces three fixes.
Two independent Q networks, both trained separately. The Bellman target uses the minimum of the two Q estimates rather than either one:
q_target = min(Q1(next_state, next_action), Q2(next_state, next_action))Taking the minimum is pessimistic by design. It systematically underestimates rather than overestimates, which is a much safer failure mode.
The actor updates less frequently than the critics (every update_online_policy steps). The critics are given time to stabilise before the actor acts on their estimates.
Noise is added to the target action when computing the Bellman target, preventing the actor from exploiting narrow Q function peaks.
The implementation maintains six networks: online and target versions of both the actor and two critics. The update_online_model function handles the update_online_policy flag to implement the delayed updates.
SAC is the most sophisticated algorithm in the set, and arguably the most practically useful for continuous control.
The key difference from all previous algorithms: SAC adds an entropy term to the reward. The agent is not just rewarded for getting high returns — it is also rewarded for behaving as randomly as possible while still getting high returns. This maximum entropy objective encourages exploration throughout training, not just at the start.
The actor (FCGP) outputs a Gaussian distribution over actions rather than a single action. Mean and log standard deviation are learned as separate network heads. The log standard deviation is clamped to prevent numerical instability in the distribution sampling.
The entropy coefficient alpha is itself learned automatically via a Lagrangian constraint, targeting a desired entropy level. This eliminates the need to manually tune exploration.
SAC uses twin Q networks like TD3 to control overestimation, and PER for sample efficiency. The replay buffer uses a dictionary keyed by timestep, allowing O(1) deletion of the oldest experience when the buffer is full. The Agent class wires everything together:
class Agent():
def __init__(self, env, ReplayMemory, online_policy, online_q_a, online_q_b,
online_policy_optimizer, online_q_network_optimizer_a, online_q_network_optimizer_b,
target_q_a, target_q_b, gamma=0.99, alpha_pr=0.4, beta_pr=0.3,
tau=0.01, n_ep=120, max_steps=100000, memory_size=1000000, batch_size=50,
policy_update_step=2, target_update=50, ...)| Algorithm | Policy Type | On/Off Policy | Key Mechanism | Best For |
|---|---|---|---|---|
| VPG | Stochastic | On-policy | Policy gradient + baseline | Simple, low-dimensional |
| A2C | Stochastic | On-policy | Parallel synchronous workers | Faster on-policy learning |
| A3C | Stochastic | On-policy | Async workers, SharedAdam | CPU-parallelised training |
| A3C + GAE | Stochastic | On-policy | Bias-variance trade-off | Longer episode tasks |
| DDPG + PER | Deterministic | Off-policy | OU noise, Polyak averaging | Continuous control |
| TD3 + PER | Deterministic | Off-policy | Twin critics, delayed updates | Stable continuous control |
| SAC + PER | Stochastic | Off-policy | Maximum entropy, auto-alpha | Sample-efficient continuous |
Building all seven from scratch, back to back, in the same codebase gives you a feel for the field's progression that reading papers alone does not. Each algorithm is a response to a specific failure mode in the one before it. A2C fixes VPG's sample efficiency. A3C fixes A2C's synchronisation bottleneck. GAE fixes the noisy advantage estimate. TD3 fixes DDPG's overestimation. SAC fixes TD3's deterministic exploration.
The through-line across all of them is the same basic challenge: how do you get a reliable learning signal from a sparse, delayed, noisy reward? Every technique here is a different answer to that question.
This repo did not stay academic. The SAC and TD3 implementations here are the direct ancestors of the work on an autonomous quadcopter.
The drone project takes everything learned across these seven algorithms and applies it to a real physical control problem: teaching a 250mm quadcopter to hover and navigate using a policy trained entirely in simulation, then transferred to hardware with no flight controller. The Raspberry Pi runs the trained policy directly, with the same observation-to-action loop these notebooks implement.
SAC with PER became the primary algorithm, later upgraded with DroQ-style critic regularisation (LayerNorm and Dropout on the Q networks) to stabilise training at higher UTD ratios. TD3 with PER was used as a benchmark and significantly outperformed SAC on the hover task, which led to a deeper understanding of when deterministic versus stochastic policies are the right tool.
The pattern is the same as what this repo explores: each algorithm is a response to a failure mode in the previous one. The drone project just added one more constraint on top: the policy eventually has to run in the real world on constrained hardware.
Full source: github.com/Kikumu/Reinforcement-Learning-repo