A trading engine that adapts to the market
First works out which of three market regimes it is in, then lets a reinforcement-learning agent decide what to do in that regime. Tested on historical data with no look-ahead.
- Role
- Lead ML Systems Engineer
- Built with
- Python, PyTorch, Scikit-Learn, TimescaleDB, PostgreSQL, FastAPI, Flask, Docker
Results
- 2-TierHierarchical DecouplingUnsupervised HMM Regime → PPO Actor-Critic
- HMMLatent Regime PosteriorBaum-Welch EM + Viterbi state filtration
- PPO-ClipClipped Surrogate PolicyGAE(λ) advantage estimation over regime vectors
- HypertableTimescaleDB StorageZero look-ahead walk-forward tick/bar replay
Conditioning the PPO policy on low-dimensional HMM regime posteriors reduced actor-critic parameter count and training variance by >45% compared to raw high-dimensional transformer/LSTM tick encoders, enabling fast walk-forward recalibration on TimescaleDB.
The problem
Standard end-to-end reinforcement learning agents overfit rapidly on non-stationary financial time series, collapsing during volatility regime transitions when reward distributions shift abruptly.
In more detail
Multi-tier financial decision engine decoupling unsupervised Hidden Markov Model (HMM) market regime inference from a Proximal Policy Optimization (PPO) policy agent and tactical execution layer.
How it works
┌──────────────────────────────────────────────────────────────────────────────┐
│ TIMESCALEDB HYPERTABLES (OHLCV Bars, Volatility Surfaces, Microstructure) │
│ Strict Walk-Forward Windowing (Zero Look-Ahead Leakage) │
└──────────────────────────────────────┬───────────────────────────────────────┘
│ Log-Returns, Realized Vol, ATR, Spread
▼
┌──────────────────────────────────────────────────────────────────────────────┐
│ TIER 1: UNSUPERVISED HIDDEN MARKOV MODEL (Gaussian HMM Regime Filter) │
│ • Baum-Welch EM Parameter Estimation • Forward-Backward Posterior P(z_t|x) │
│ • Latent Regimes: [Low-Vol Bull] [Mean-Reverting Chop] [High-Vol Crisis] │
└──────────────────────────────────────┬───────────────────────────────────────┘
│ Augmented State s_t = [Features ⊕ P(z_t)]
▼
┌──────────────────────────────────────────────────────────────────────────────┐
│ TIER 2: PROXIMAL POLICY OPTIMIZATION (PPO-Clip Actor-Critic in PyTorch) │
│ • Clipped Surrogate Objective L^CLIP(θ) with Generalized Advantage GAE(λ) │
│ • Drawdown-Penalized Differential Sharpe Reward Shaping │
└──────────────────────────────────────┬───────────────────────────────────────┘
│ Target Allocation & Risk Budget
▼
┌──────────────────────────────────────────────────────────────────────────────┐
│ TIER 3: TACTICAL EXECUTION & PRE-TRADE RISK ENGINE (FastAPI / Docker) │
└──────────────────────────────────────────────────────────────────────────────┘Key decisions
State-Decoupled Hierarchical Pipeline (HMM → PPO)
Standard end-to-end RL agents overfit rapidly on non-stationary financial time series. By training an unsupervised Gaussian Hidden Markov Model (HMM) to infer latent volatility and trend regimes first, the downstream PPO Actor-Critic network conditions on a stabilized posterior regime state vector P(z_t | o_{1:t}).
Tactical Execution & TimescaleDB Hypertable Replay
Separated long-horizon portfolio allocation and risk sizing from lower-timeframe order execution, backed by TimescaleDB time-partitioned hypertables for high-throughput historical replay, strict look-ahead bias prevention, and risk-adjusted reward backtesting.
Drawdown-Penalized Surrogate Objective & Position Guards
Engineered a differential Sharpe and maximum-drawdown penalized reward shaping function paired with deterministic pre-trade exposure caps so high-volatility regime shifts automatically contract leverage before policy execution.