Skip to article frontmatterSkip to article content
Site not loading correctly?

This may be due to an incorrect BASE_URL configuration. See the MyST Documentation for reference.

Execution Tactics

Introduction

Chapter Optimal Execution Theory addressed the scheduling problem: given a large order to execute over a horizon [0,T][0, T], what sequence of child order sizes {vti}\{v_{t_i}\} minimises expected cost for a given level of risk aversion? The Almgren–Chriss model and its extensions produce optimal schedules that specify, at each time step tit_i, a target volume viv_i to trade in the next slice [ti,ti+1][t_i, t_{i+1}].

A slice, however, is not an execution. Sending a market order of size viv_i to the exchange is always an option, but it is rarely optimal: it consumes the spread and accelerates local price impact within the slice. Execution tactics are the micro-level algorithms that implement each slice in the limit order book, deciding dynamically whether to post a limit order (and at what price level), upgrade to a market order, or split the slice across multiple venues. Where strategies resolve the tension between market impact and timing risk at the scale of minutes to hours, tactics resolve the analogous tension between execution probability and price improvement at the scale of seconds to minutes.

This chapter develops the theory and practice of execution tactics. Section Fill probability model recalls the fill probability model from Chapter Modelling the Limit Order Book — the core stochastic ingredient for all tactic design. Section Single-market execution tactic formulates the single-market placement problem as an optimal stochastic control problem, derives the Hamilton–Jacobi–Bellman equation, and characterizes the solution as an aggressiveness matrix indexed by remaining time and remaining quantity. Section Smart order routing extends the framework to smart order routing across multiple venues. Section Heuristic execution tactics describes the menu of heuristic tactics offered by brokers and exchanges, and section Reinforcement learning for execution tactics develops the reinforcement learning approach to tactic design, which bypasses the need for explicit model calibration.

Fill probability model

Section Probability of filling a limit order of Chapter Modelling the Limit Order Book develops the fill probability model in full, including derivation, calibration, extensions with additional LOB features, and monotonicity constraints. Here we recall the key result needed for tactic design.

Following Avellaneda and Stoikov Avellaneda & Stoikov, 2008 and Cont, Stoikov, and Talreja Cont et al., 2010, fill events for a limit order placed δ\delta ticks from the mid-price are modelled as a Poisson process with depth-dependent intensity:

λ(δ)=Aekδ,\lambda(\delta) = A \, e^{-k\delta},

where A>0A > 0 is the base fill rate and k>0k > 0 is the depth sensitivity. The probability of a fill within a time window τ\tau is:

P(fillδ,τ)=1eλ(δ)τ=1exp ⁣(Aekδτ).P(\text{fill} \mid \delta, \tau) = 1 - e^{-\lambda(\delta)\tau} = 1 - \exp\!\left(-A e^{-k\delta} \tau\right).

In practice, AA and kk are calibrated from historical LOB data by regressing observed fill rates against placement depth, controlling for spread, order imbalance, and time of day (see Section Probability of filling a limit order). Typical values for liquid equity markets are A0.5A \sim 0.52.0 fills per minute at zero depth and k1k \sim 13 per tick.

Single-market execution tactic

Problem setup

We now formulate the tactic’s control problem. The tactic is responsible for executing QQ shares (a sell order, without loss of generality) within a window [0,T][0, T]. The state at time tt is the remaining quantity qt{0,1,,Q}q_t \in \{0, 1, \ldots, Q\}; the control is the placement depth δt0\delta_t \geq 0 at which to post a limit sell order.

Dynamics. Fills arrive as a Poisson process with rate λ(δt)\lambda(\delta_t); each fill reduces the remaining quantity by one unit:

qt+=qt1with intensity λ(δt)q_{t^+} = q_t - 1 \quad \text{with intensity } \lambda(\delta_t)

We normalize the tick size to one, so a fill at depth δ\delta yields proceeds of δ\delta ticks above the mid-price.

Terminal condition. If any quantity remains at TT, it must be sold immediately via a market order. The forced liquidation incurs a cost of bb ticks per unit below mid-price:

ΠTterminal=bqT,b>0\Pi_T^{\text{terminal}} = -b \, q_T, \qquad b > 0

The parameter bb is a market property — the effective half-spread plus market impact of an unplanned emergency order — estimated from historical trade data during the calibration period alongside AA and kk. It is not a free parameter that the tactic designer controls.

Objective. The tactic maximises the expected CARA (Constant Absolute Risk Aversion) utility of total proceeds, with risk aversion coefficient γ>0\gamma > 0:

J(t,q)=sup{δs}stEt ⁣[eγΠT]J(t, q) = \sup_{\{\delta_s\}_{s \geq t}} \mathbb{E}_t \!\left[-e^{-\gamma \Pi_T}\right]

where ΠT=fillsδfillbqT\Pi_T = \sum_{\text{fills}} \delta_{\text{fill}} - b\,q_T is total proceeds including the terminal liquidation. The risk aversion parameter γ\gamma is the tactic’s aggressiveness control: a high-γ\gamma agent dislikes the possibility of reaching the deadline with unsold inventory and posts aggressively to ensure execution; a low-γ\gamma agent (near risk-neutral) prioritises price improvement and posts passively. This formulation is consistent with the exponential utility framework used in the market-making chapters.

Certainty equivalent. Working directly with the exponential utility is cumbersome. We transform to the certainty equivalent value function, which expresses values in the same units as raw proceeds (ticks):

H(t,q)1γln ⁣(J(t,q))H(t, q) \equiv -\frac{1}{\gamma}\ln\!\bigl(-J(t, q)\bigr)

The boundary conditions become H(T,q)=bqH(T, q) = -b\,q (terminal liquidation cost) and H(t,0)=0H(t, 0) = 0 (no inventory, no further proceeds).

The HJB equation and optimal placement

Let Φ(t,q)J(t,q)\Phi(t,q) \equiv -J(t,q) denote the minimised disutility (Φ>0\Phi > 0, related to HH by Φ=eγH\Phi = e^{-\gamma H}). Applying Bellman’s principle over [t,t+dt][t, t+dt] and taking dt0dt \to 0, the Hamilton–Jacobi–Bellman equation for Φ\Phi is:

0=tΦ+minδ0  Aekδ ⁣[eγδΦ(t,q1)Φ(t,q)]0 = \partial_t \Phi + \min_{\delta \geq 0}\; Ae^{-k\delta}\!\left[e^{-\gamma\delta}\Phi(t,q-1) - \Phi(t,q)\right]

Setting the derivative with respect to δ\delta to zero gives the first-order condition:

k ⁣[eγδΦq1Φq]γeγδΦq1=0(k+γ)eγδΦq1=kΦq-k\!\left[e^{-\gamma\delta}\Phi_{q-1} - \Phi_q\right] - \gamma e^{-\gamma\delta}\Phi_{q-1} = 0 \qquad \Rightarrow \qquad (k+\gamma)\,e^{-\gamma\delta^*}\Phi_{q-1} = k\,\Phi_q

Converting to the certainty-equivalent HH (using Φq=eγHq\Phi_q = e^{-\gamma H_q}) and defining:

ΔH(t,q)H(t,q)H(t,q1)0\Delta H(t, q) \equiv H(t, q) - H(t, q-1) \leq 0

the optimal placement depth is:

δ(t,q)=1γln ⁣(1+γk)+ΔH(t,q)\boxed{\delta^*(t, q) = \frac{1}{\gamma}\ln\!\left(1 + \frac{\gamma}{k}\right) + \Delta H(t, q)}

The first term, 1γln(1+γ/k)\frac{1}{\gamma}\ln(1+\gamma/k), is the risk-comfort depth: the base placement a risk-averse agent maintains even without inventory urgency. It replaces the 1/k1/k of the risk-neutral case, and has three key properties:

The interpretation of ΔH\Delta H is unchanged: the optimal placement depth equals the risk-comfort depth minus the absolute urgency ΔH(t,q)|\Delta H(t,q)|. A market order is optimal when δ(t,q)<0\delta^*(t,q) < 0, i.e., when urgency exceeds the risk-comfort depth.

Substituting the optimal δ\delta^* back and transforming to HH gives the reduced ODE in remaining time τ=Tt\tau = T - t:

τH(τ,q)=Ak+γ ⁣(kk+γ) ⁣k/γCγexp ⁣(kΔH(τ,q))\partial_\tau H(\tau, q) = \underbrace{\frac{A}{k+\gamma}\!\left(\frac{k}{k+\gamma}\right)^{\!k/\gamma}}_{C_\gamma}\exp\!\left(-k\,\Delta H(\tau,q)\right)

The constant CγC_\gamma encodes both the market fill-rate parameters (A,k)(A, k) and the agent’s risk aversion γ\gamma. As γ0\gamma \to 0, CγA/(ek)C_\gamma \to A/(ek), recovering the risk-neutral ODE. The structure of the equation — and hence the backward induction algorithm — is identical to the risk-neutral case; only the driving constant changes.

Analytical solution for q=1q = 1. When only one unit remains, H(τ,0)=0H(\tau, 0) = 0, so ΔH(τ,1)=H(τ,1)\Delta H(\tau, 1) = H(\tau, 1) and the ODE becomes:

τH(τ,1)=CγekH(τ,1)\partial_\tau H(\tau, 1) = C_\gamma\, e^{-k H(\tau, 1)}

This separable ODE has the solution (with initial condition H(0,1)=bH(0,1) = -b):

H(τ,1)=1kln ⁣(ekb+kCγτ)H(\tau, 1) = \frac{1}{k}\ln\!\left(e^{-kb} + k\,C_\gamma\,\tau\right)

The optimal placement depth for the last unit is:

δ(τ,1)=1γln ⁣(1+γk)+1kln ⁣(ekb+kCγτ)\delta^*(\tau, 1) = \frac{1}{\gamma}\ln\!\left(1+\frac{\gamma}{k}\right) + \frac{1}{k}\ln\!\left(e^{-kb} + k\,C_\gamma\,\tau\right)

As τ0\tau \to 0: δ(0,1)=1γln(1+γ/k)b\delta^*(0,1) = \frac{1}{\gamma}\ln(1+\gamma/k) - b, which triggers a market order when b>1γln(1+γ/k)b > \frac{1}{\gamma}\ln(1+\gamma/k) — a condition that is increasingly easy to satisfy as γ\gamma increases. As τ\tau \to \infty: δ(τ,1)\delta^*(\tau,1) grows without bound — with ample time, even a risk-averse agent posts very passively. For q>1q > 1, the system is solved numerically by backward induction on the grid (τ,q)(\tau, q).

The aggressiveness matrix

The solution δ(t,q)\delta^*(t, q) is most naturally displayed as a two-dimensional look-up table indexed by the normalised remaining time t/Tt/T and the normalised remaining quantity q/Qq/Q — the aggressiveness matrix of the tactic. Each cell of the matrix specifies the optimal placement depth in ticks at that (time, quantity) state.

Aggressiveness matrices for three levels of risk aversion \gamma (low: \gamma = 0.2, moderate: \gamma = 2, high: \gamma = 10; with A = 1 fill/min, k = 1.0 per tick, b = 1.0 tick calibrated, T = 10 min). Colour indicates optimal placement depth in ticks: darker (blue) regions correspond to deeper, more passive placement; lighter (yellow/red) regions trigger aggressive or market orders. In all cases, the tactic becomes more aggressive as time expires or as the remaining fraction q/Q approaches 1. A higher risk aversion \gamma shifts the entire matrix towards more aggressive placements, shrinking the risk-comfort depth and enlarging the market-order region.

Figure 1:Aggressiveness matrices for three levels of risk aversion γ\gamma (low: γ=0.2\gamma = 0.2, moderate: γ=2\gamma = 2, high: γ=10\gamma = 10; with A=1A = 1 fill/min, k=1.0k = 1.0 per tick, b=1.0b = 1.0 tick calibrated, T=10T = 10 min). Colour indicates optimal placement depth in ticks: darker (blue) regions correspond to deeper, more passive placement; lighter (yellow/red) regions trigger aggressive or market orders. In all cases, the tactic becomes more aggressive as time expires or as the remaining fraction q/Qq/Q approaches 1. A higher risk aversion γ\gamma shifts the entire matrix towards more aggressive placements, shrinking the risk-comfort depth and enlarging the market-order region.

Several properties of the aggressiveness matrix are economically intuitive:

In practice, the matrix is pre-computed off-line using backward induction and stored as a look-up table. At each step of the tactic, the current state (t,q)(t, q) is mapped to the nearest grid cell, and the corresponding depth δ\delta^* is read off. This makes real-time execution computationally trivial.

Multi-order splitting

The single-order formulation posts one limit order at a time. In practice, it is common to split the remaining quantity into mm simultaneous child orders, each of size q/mq/m, posted at (potentially different) price levels. The HJB equation for the multi-order case with mm simultaneous orders is:

tH(t,q)=maxδ1,,δm0j=1mλ(δj) ⁣[δj+H(t,q1)H(t,q)]-\partial_t H(t, q) = \max_{\delta_1, \ldots, \delta_m \geq 0} \sum_{j=1}^m \lambda(\delta_j)\!\left[\delta_j + H(t, q-1) - H(t, q)\right]

Because the objective is separable — each order’s contribution depends only on its own δj\delta_j and the common term ΔH(t,q)\Delta H(t,q) — the optimal δj\delta_j^* are identical for all jj:

δj(t,q)=1γln ⁣(1+γk)+ΔH(t,q)for all j=1,,m\delta_j^*(t, q) = \frac{1}{\gamma}\ln\!\left(1+\frac{\gamma}{k}\right) + \Delta H(t, q) \quad \text{for all } j = 1, \ldots, m

All orders are placed at the same optimal depth as in the single-order case Cartea & Jaimungal, 2015. The only effect of splitting is to multiply the effective fill rate: with mm orders at depth δ\delta^*, the aggregate fill rate is mλ(δ)m\lambda(\delta^*), reducing the expected time to the next fill by a factor of mm. The aggressiveness matrix is therefore unchanged; only the inventory dynamics change (fills arrive mm times faster on average). Splitting reduces the risk of the tactic finishing the window with large residual inventory, at no cost to price improvement.

Smart order routing

Multi-venue framework

Modern financial markets are fragmented: the same instrument trades simultaneously on multiple lit exchanges, dark pools, and bilateral platforms with different liquidity profiles. A Smart Order Router (SOR) is a component that distributes the tactic’s child orders across these venues, seeking to fill the required volume at the lowest total cost.

We model KK trading venues, each characterized by its own depth-dependent fill rate:

λk(δ)=Akekkδ,k=1,,K\lambda_k(\delta) = A_k \, e^{-k_k \delta}, \quad k = 1, \ldots, K

where AkA_k is the base fill rate at venue kk and kkk_k controls how steeply fill rates fall with depth. Venues with high AkA_k (high volume) and low kkk_k (book does not thin rapidly) are most liquid. The tactic simultaneously posts one limit order per venue, with depth δt(k)\delta^{(k)}_t at venue kk.

Optimal venue allocation

With multiple venues, the HJB equation becomes:

tH(t,q)=maxδ(1),,δ(K)0k=1Kλk(δ(k)) ⁣[δ(k)+H(t,q1)H(t,q)]-\partial_t H(t, q) = \max_{\delta^{(1)}, \ldots, \delta^{(K)} \geq 0} \sum_{k=1}^K \lambda_k(\delta^{(k)}) \!\left[\delta^{(k)} + H(t, q-1) - H(t,q)\right]

The multi-venue objective is again separable across venues for a given ΔH(t,q)\Delta H(t,q). The first-order condition for each venue independently yields:

δ(k)(t,q)=1γln ⁣(1+γkk)+ΔH(t,q),k=1,,K\delta^{(k)*}(t, q) = \frac{1}{\gamma}\ln\!\left(1 + \frac{\gamma}{k_k}\right) + \Delta H(t, q), \quad k = 1, \ldots, K

The optimal depth at each venue follows the same formula as the single-venue case, with the risk-comfort depth 1γln(1+γ/kk)\frac{1}{\gamma}\ln(1+\gamma/k_k) being venue-specific through kkk_k. This has a direct economic interpretation: venues where fill rates fall off slowly with depth (small kkk_k) attract deeper, more passive placements, while venues where fill rates fall off rapidly (large kkk_k) attract more aggressive placements. Additionally, for a given γ\gamma, depth is tighter everywhere compared with the risk-neutral benchmark.

The substituted HJB becomes:

tH(t,q)=k=1KCγ(k)exp ⁣(kkΔH(t,q)),Cγ(k)=Akkk+γ ⁣(kkkk+γ) ⁣kk/γ-\partial_t H(t,q) = \sum_{k=1}^K C_\gamma^{(k)} \exp\!\left(-k_k \, \Delta H(t,q)\right), \qquad C_\gamma^{(k)} = \frac{A_k}{k_k+\gamma}\!\left(\frac{k_k}{k_k+\gamma}\right)^{\!k_k/\gamma}

The SOR aggregates CARA-adjusted fill rate contributions from all venues simultaneously, without requiring an explicit allocation of volume to each venue.

Smart order routing across two venues. Left: fill rates \lambda_k(\delta) for a liquid venue (high A_1, low k_1; blue) and an illiquid venue (low A_2, high k_2; orange). The liquid venue attracts deeper passive placements; the illiquid venue is approached more aggressively. Right: simulated P&L distributions (10,000 runs, Q = 100, T = 10 min) comparing single-venue execution (liquid, illiquid) against the two-venue SOR. The SOR dominates both single-venue alternatives in both mean and variance.

Figure 2:Smart order routing across two venues. Left: fill rates λk(δ)\lambda_k(\delta) for a liquid venue (high A1A_1, low k1k_1; blue) and an illiquid venue (low A2A_2, high k2k_2; orange). The liquid venue attracts deeper passive placements; the illiquid venue is approached more aggressively. Right: simulated P&L distributions (10,000 runs, Q=100Q = 100, T=10T = 10 min) comparing single-venue execution (liquid, illiquid) against the two-venue SOR. The SOR dominates both single-venue alternatives in both mean and variance.

Practical considerations

Beyond the theoretical model, SOR design involves several practical dimensions:

Dark pools. Dark pools are off-exchange trading venues that do not display their order book; fills occur at the mid-price without market impact, but fill rates are uncertain and typically lower than on lit venues. The fill rate model for dark pools requires a different calibration — fills are generated by crossing contra-side orders, not by price movement. The seminal analysis of the dark pool problem is due to Ganchev et al. (2010), who frame it as a censored exploration problem.

Latency and queue position. Fill rates in limit order books depend not only on depth but on queue position at each level. An order that arrives early at a given price level will be filled before later arrivals at the same level (price-time priority). The model above implicitly assumes the order is at the back of the queue; incorporating queue position leads to a richer state space and has been studied in the literature on optimal placement with latency Cont et al., 2010.

Price divergence across venues. In fragmented markets, mid-prices can temporarily diverge across venues, especially at high frequency. A SOR that ignores this may place its order at the “wrong” venue. Best-execution regulations (MiFID II in Europe, Regulation NMS in the US) require systematic evaluation of execution quality across venues and oblige brokers to seek the best available price.

Heuristic execution tactics

Beyond the model-based approaches above, a range of heuristic tactics are widely used in practice, supported natively by exchanges and prime brokers. These tactics encode practical execution intuitions that are simpler to implement than full HJB solutions, and serve as useful benchmarks.

Pegged orders

A pegged order tracks a reference price dynamically, automatically re-pricing the order as market conditions change. The most common variants are:

Pegged orders reduce the operational complexity of managing limit orders in fast-moving markets: without pegging, a passive limit order placed at the mid at time t0t_0 may be far inside the book (or even outside the book entirely) by time t1t_1 if the price has moved. Pegging ensures the order remains continuously visible at the intended level without manual re-pricing.

Iceberg and reserve orders

A large passive limit order that is fully displayed in the order book signals the trader’s intention to other market participants, who may adjust their own behaviour — pulling displayed orders, widening spreads, or trading against the disclosed position — to extract rents from the informed side. Iceberg orders (also called reserve orders or hidden quantity orders) mitigate this information leakage by displaying only a small “tip” qtipq_{\text{tip}} in the public order book while reserving the remainder qhidden=Qqtipq_{\text{hidden}} = Q - q_{\text{tip}} invisibly. When the tip is fully filled, the next tranche is automatically posted.

The trade-off in choosing the tip size is as follows. A larger tip signals more information to the market but earns better queue priority (since the displayed tip competes for fills on a first-come, first-served basis with its full size). A smaller tip minimises information leakage but may execute more slowly if queue priority is lost each time the tip is exhausted. Empirical evidence suggests iceberg orders execute at slightly better prices than equivalent market orders but slower than equivalent fully-disclosed limit orders, reflecting the information leakage channel Cartea et al., 2015.

Liquidity seeking and sniping

Liquidity seeking tactics scan multiple venues simultaneously for executable liquidity at or better than a target price. When liquidity is detected — a visible limit order on the contra-side at a favourable price — the SOR fires a market order (or a marketable limit order) to capture it before it disappears. These tactics are opportunistic: they are passive in the absence of favourable liquidity and aggressive when it appears.

Sniping is a closely related strategy in which the tactic waits for a specific price level to be reached and then submits an aggressive order. In the context of execution tactics, a sniper may target a pre-specified price S<S0S^* < S_0 (for a buy) below which it activates, treating the price as a favourable entry point. The risk of sniping is adverse selection: the price may be moving rapidly to the target level precisely because informed traders are selling, making the achieved entry point worse in expectation than the target.

Participation rate tactics

Participation rate (PoV) tactics constrain the execution rate to a fixed percentage π\pi of market volume, so that the tactic does not trade significantly faster than the prevailing market pace. Within each measurement interval [ti,ti+1][t_i, t_{i+1}], the tactic computes the market volume ViV_i and submits enough orders to have traded πVi\pi V_i shares in aggregate. If ahead of pace, it switches to passive limit orders; if behind pace, it switches to more aggressive orders or market orders.

PoV tactics are natural for VWAP strategies (see chapter Optimal Execution Theory): a constant participation rate π=Q/Vday\pi = Q/V^{\text{day}} (where VdayV^{\text{day}} is the expected daily volume) approximates the VWAP schedule. In practice, the participation rate is adjusted intraday based on updated volume predictions from the volume models discussed in chapter Modelling the Limit Order Book.

Reinforcement learning for execution tactics

RL formulation

The optimal placement problem of section Single-market execution tactic rests on explicit assumptions: Poisson fill processes with exponential depth dependence, a known terminal liquidation cost, and a model that is stationary within the execution window. In practice, fill rates vary with LOB state, intraday patterns, and recent order flow in ways that are difficult to capture parsimoniously. Reinforcement learning (RL) provides an alternative that learns the optimal policy from simulated or historical experience, without requiring explicit model specification.

The tactic is formulated as a Markov Decision Process (MDP):

rt={atif a fill occurs at step tbqTif t=T and qT>00otherwiser_t = \begin{cases} a_t & \text{if a fill occurs at step } t \\ -b \, q_T & \text{if } t = T \text{ and } q_T > 0 \\ 0 & \text{otherwise} \end{cases}

This linear reward corresponds to the expected-proceeds (risk-neutral) objective — the γ0\gamma \to 0 limit of the CARA criterion. The RL agent therefore learns to approximate the risk-neutral limit of the HJB policy. The theoretically grounded approach to risk-averse execution is to use the CARA HJB directly with an explicit γ\gamma, but the RL framework is valuable precisely when the state space is richer than (τ,q)(\tau, q) and the fill rate model is non-stationary or complex.

The RL agent learns a policy πθ(s)\pi_\theta(\mathbf{s}) that maximises the expected discounted sum of rewards E[tρtrt]\mathbb{E}[\sum_t \rho^t r_t] (where ρ\rho is the discount factor) by estimating π\pi from experience rather than from explicit HJB solution.

The RL framework is particularly powerful when (i) the state space is richer than just (τ,q)(\tau, q) — including LOB features that are difficult to model analytically — and (ii) the fill rate model is non-stationary or has complex dependence structure.

Q-learning and deep Q-networks

The seminal application of RL to execution tactics is due to Nevmyvaka, Feng, and Kearns Nevmyvaka et al., 2006, who applied tabular Q-learning to the order placement problem on NASDAQ data. Their state representation included remaining quantity, remaining time, bid-ask spread, and short-term price momentum. The action space was a discretized set of placement levels. Evaluated on 1.5 years of NASDAQ data, the RL policy outperformed TWAP by approximately 50 basis points on average for large-cap stocks — a significant improvement for institutional-scale executions.

In the tabular Q-learning approach, the action-value function Q(s,a)Q(\mathbf{s}, a) is maintained as a look-up table and updated via the Bellman recursion:

Q(st,at)Q(st,at)+α ⁣[rt+γmaxaQ(st+1,a)Q(st,at)]Q(\mathbf{s}_t, a_t) \leftarrow Q(\mathbf{s}_t, a_t) + \alpha\!\left[r_t + \gamma \max_{a'} Q(\mathbf{s}_{t+1}, a') - Q(\mathbf{s}_t, a_t)\right]

where α\alpha is the learning rate and γ\gamma is the discount factor. The policy is ϵ\epsilon-greedy: with probability ϵ\epsilon a random action is taken (exploration), and with probability 1ϵ1-\epsilon the greedy action argmaxaQ(s,a)\arg\max_a Q(\mathbf{s},a) is selected.

When the state space is continuous or high-dimensional, the tabular representation becomes infeasible. The Deep Q-Network (DQN) replaces the table with a neural network Qθ(s,a)Q_\theta(\mathbf{s}, a) parameterized by weights θ\theta, trained by minimising the Bellman residual loss:

L(θ)=E ⁣[(r+γmaxaQθˉ(s,a)Qθ(s,a))2]\mathcal{L}(\theta) = \mathbb{E}\!\left[\left(r + \gamma \max_{a'} Q_{\bar{\theta}}(\mathbf{s}', a') - Q_\theta(\mathbf{s}, a)\right)^2\right]

where QθˉQ_{\bar{\theta}} is a periodically updated target network that stabilises training by decoupling the moving target from the network being updated. A replay buffer stores historical transitions (s,a,r,s)(\mathbf{s}, a, r, \mathbf{s}'), and mini-batches are drawn uniformly at random to break temporal correlations in the training data.

Ning et al. (2021) apply a Double DQN architecture — which reduces Q-value overestimation by separating action selection from value evaluation — to execution with limit and market orders. Their model is trained on a calibrated synthetic LOB and evaluated against IS and VWAP benchmarks, showing that the learned policy adapts placement depth to real-time LOB features in a way that the static aggressiveness matrix cannot.

Training a DQN execution tactic. Left: training curve — mean episodic reward per episode over 2,000 episodes (smoothed with a 50-episode rolling average). The agent progresses from random exploration to approximately optimal policy. Right: learned Q-values as a function of remaining time \tau and remaining fraction \phi for the action a = 0 (best ask placement), showing that the DQN has learned a representation qualitatively consistent with the theoretical aggressiveness matrix: higher Q-values (greener) at early times and low remaining fractions, lower Q-values (redder) at late times with large remaining fractions where more aggressive actions dominate.

Figure 3:Training a DQN execution tactic. Left: training curve — mean episodic reward per episode over 2,000 episodes (smoothed with a 50-episode rolling average). The agent progresses from random exploration to approximately optimal policy. Right: learned Q-values as a function of remaining time τ\tau and remaining fraction ϕ\phi for the action a=0a = 0 (best ask placement), showing that the DQN has learned a representation qualitatively consistent with the theoretical aggressiveness matrix: higher Q-values (greener) at early times and low remaining fractions, lower Q-values (redder) at late times with large remaining fractions where more aggressive actions dominate.

Policy gradient methods

Q-learning operates in a discretized action space. When the placement depth is treated as a continuous variable — or when the policy must be stochastic to handle exploration and variability in LOB conditions — policy gradient methods are more appropriate. The Proximal Policy Optimisation (PPO) algorithm Sutton & Barto, 2018 maintains a parameterized policy πθ(as)\pi_\theta(a|\mathbf{s}) (e.g., a Gaussian over continuous depths) and updates it by ascending a clipped surrogate objective that prevents destabilising large policy updates:

LCLIP(θ)=Et ⁣[min ⁣(ρt(θ)A^t,  clip(ρt(θ),1ϵ,1+ϵ)A^t)]\mathcal{L}^{\text{CLIP}}(\theta) = \mathbb{E}_t\!\left[\min\!\left(\rho_t(\theta)\hat{A}_t,\; \text{clip}(\rho_t(\theta), 1-\epsilon, 1+\epsilon)\hat{A}_t\right)\right]

where ρt(θ)=πθ(atst)/πθold(atst)\rho_t(\theta) = \pi_\theta(a_t|\mathbf{s}_t)/\pi_{\theta_{\text{old}}}(a_t|\mathbf{s}_t) is the probability ratio and A^t\hat{A}_t is the estimated advantage function.

Actor-critic methods (A2C, A3C) combine a policy network (actor) with a value network (critic) that estimates the expected return from each state. The critic reduces the variance of policy gradient estimates by providing a baseline for the advantage function. This structure mirrors the relationship between the optimal control value function H(t,q)H(t,q) and the optimal policy δ(t,q)\delta^*(t,q): the critic approximates HH and the actor approximates δ\delta^*.

Recent developments in RL for execution have explored several directions. Reward shaping augments the terminal execution cost with intermediate milestones (e.g., rewarding the agent for maintaining pace relative to a VWAP schedule). Multi-agent frameworks model the interaction between the execution tactic and other market participants — momentum traders, market makers — explicitly, training the tactic in a game-theoretic rather than single-agent setting. Offline RL trains on historical order book data without an explicit simulator, avoiding the model misspecification that can occur when training on a stylised generative model. All of these extensions move in the direction of reducing the gap between the theoretical ideal and robust practical deployment.

Exercises

References
  1. Avellaneda, M., & Stoikov, S. (2008). High-frequency trading in a limit order book. Quantitative Finance, 8(3), 217–224. 10.1080/14697680701381228
  2. Cont, R., Stoikov, S., & Talreja, R. (2010). A Stochastic Model for Order Book Dynamics. Operations Research, 58(3), 549–563. 10.1287/opre.1090.0780
  3. Cartea, Á., & Jaimungal, S. (2015). Optimal Execution with Limit and Market Orders. Quantitative Finance, 15(8), 1279–1291. 10.1080/14697688.2015.1032543
  4. Ganchev, K., Nevmyvaka, Y., Kearns, M., & Wortman Vaughan, J. (2010). Censored Exploration and the Dark Pool Problem. Communications of the ACM, 53(5), 99–107. 10.1145/1735223.1735248
  5. Cartea, Á., Jaimungal, S., & Penalva, J. (2015). Algorithmic and High-Frequency Trading. Cambridge University Press.
  6. Nevmyvaka, Y., Feng, Y., & Kearns, M. (2006). Reinforcement Learning for Optimized Trade Execution. Proceedings of the 23rd International Conference on Machine Learning (ICML), 673–680.
  7. Ning, B., Lin, F. H. T., & Jaimungal, S. (2021). Double Deep Q-Network for Optimal Execution Strategies. Applied Mathematical Finance, 28(2), 109–139. 10.1080/1350486X.2021.1980553
  8. Sutton, R. S., & Barto, A. G. (2018). Reinforcement Learning: An Introduction (2nd ed.). MIT Press.