how to detect overfitting trading strategy tests

What overfitting looks like

An overfitted strategy shows performance metrics in backtesting that are too good to be plausible: a win rate consistently above 75-80% (institutional quant funds with genuine edges operate around 51-55%), a Sharpe ratio above 3.0 in backtesting (a sustained live Sharpe above 2.0 is world-class), a strategy that only works in one specific historical period, or one that only works for a single exact parameter combination.

Test 1: The jitter test (parameter sensitivity)

A strategy with a genuine edge should stay profitable across a range of parameter values near the optimized settings -- not just for one exact combination.

How to do it: take each parameter and test the strategy at values 5-10% above and below the optimized value. If it uses a 50-day moving average, test 45, 47, 50, 53, and 55 days.

What it means: a robust strategy performs similarly across the whole range; an overfitted one collapses when the parameter moves even slightly. If a 50-day average works but 48 days loses money, the edge is not real.

Test 2: Out-of-sample test

Never use 100% of your data for development. Split it: use 70-80% to build and optimize the strategy (in-sample), and hold back 20-30% as out-of-sample data the strategy has never seen. Test the finished strategy on that held-back data without changing anything.

Critical rule: once you have seen the out-of-sample result, you cannot tweak the strategy and retest on the same data. If you do, that data is now contaminated -- it has effectively become in-sample.

Test 3: Cross-asset test

If a strategy captures a real market principle (momentum, mean reversion, volatility breakout), it should generalize at least partly to similar markets. Apply the exact same rules to correlated assets -- for example, a momentum strategy built on Volvo B tested on Sandvik and Atlas Copco. If it produces catastrophic losses on every correlated peer and only works on one stock, it is memorizing that one price history, not capturing a pattern.

Test 4: Complexity check

Every extra rule or parameter multiplies overfitting risk. The most robust strategies typically have 2-4 core parameters with a clear rationale. Red flags: more than 5-6 conditions to trigger a signal; rules tied to specific weekdays or months with no economic reason; or a strategy you cannot explain in one sentence.

Red flag metrics summary

TRION and overfitting detection

TRION AI agents assess overfitting risk during strategy review -- checking parameter sensitivity, whether the metrics are plausible, and whether the logic has a clear market rationale rather than appearing curve-fitted to history. This happens before paper trading begins.

TRION was built around an honest validation sequence rather than a promise. It is a paper-only research and validation workstation: you describe a strategy idea in plain English, read the compiled logic line by line, and backtest it against real stored market data. When a metric cannot be computed honestly, TRION shows "N/A" instead of inventing a number.

TRION does not place real orders, does not connect to a broker, and does not promise profit. The current beta is simulation-only and paper-only. AI assists with drafting and explanation; it does not approve, activate, or execute anything. Humans make every decision.