Why Backtesting Can Mislead: Supplementing Backtesting with Forward Testing, Simulation, and Stress Testing

Backtesting is a powerful tool in the world of trading, helping to validate and refine our trading strategies. However, it can also be misleading due to a host of biases that may distort the effectiveness of a strategy. In this article, we explore these biases and discuss how to supplement backtesting with forward testing, simulation techniques, and stress testing to gain a more holistic understanding of our trading strategies.

Common Pitfalls of Backtesting: Overfitting and Historical Biases

One of the most common pitfalls of backtesting is overfitting. When a model is excessively tailored to historical data, it may perform well on backtests only to falter in live markets. This happens because the strategy might capture noise and trends that are specific to a particular historical period, which may not be replicable in the future.

A second common bias is the reliance on historical volatility or correlations. Traders often assume that past market dynamics will repeat, leading to a false sense of security. However, market conditions can shift suddenly, as seen during the 2020 market downturn. This sudden shift can make even the most well-tested strategy fail in real-world conditions.

Supplementing Backtesting with Forward Testing and Simulation

To supplement backtesting, it is crucial to incorporate robust forward testing and simulation techniques that factor in real-time market dynamics. Forward testing involves applying a strategy to real market data to assess its performance. This approach is crucial because it simulates actual trading scenarios, revealing how a strategy would perform under various market conditions.

Furthermore, incorporating simulation techniques can provide valuable insights. Monte Carlo simulations, for instance, can model a wide range of potential future scenarios, offering a probabilistic assessment of a strategy's effectiveness. These simulations can help identify potential risk factors and areas for improvement, enhancing the robustness of the strategy.

Stress Testing: Preparing for Adverse Conditions

Stress testing is another essential component of a comprehensive risk assessment. By testing your strategies under various adverse scenarios, you can gain insights into how they might perform in unexpected market conditions. For example, you can simulate market crashes, rapid liquidity withdrawal, and other extreme events. This approach helps you understand the worst-case scenarios and the strategies' ability to withstand these conditions.

For instance, in 2020, the global market experienced a sudden and severe downturn. Strategies that seemed bulletproof in previous bull markets fared poorly in this challenging environment. This highlights the importance of stress testing to ensure that your trading strategies can handle unexpected and extreme market events.

Qualitative Assessments and Market Insights

In addition to quantitative tests, incorporating qualitative assessments can enhance your approach to trading. Scrutinizing changes in market structure or investor sentiment can provide valuable insights into potential risks and opportunities. For example, shifts in the distribution of market liquidity, changes in trading volumes, or alterations in investor behavior can all impact the performance of your strategies.

Qualitative assessments can also help you stay informed about emerging trends and potential risks. For example, if you notice a sudden increase in short selling or a change in the number of institutions participating in a particular market, this could signal potential risks that need to be considered.

Anecdotes from a Seasoned Trader

Robert Kehres, a seasoned entrepreneur, fund manager, and quantitative trader, shares his insights from navigating both bull and bear markets. At 20, he worked at LIM Advisors, the longest continually operating hedge fund in Asia. Robert then became a quantitative trader at J.P. Morgan and later a hedge fund manager at 18 Salisbury Capital. His entrepreneurial journey includes founding Dynamify, a B2B enterprise FB SaaS platform, and Yoho, a productivity SaaS platform.

Robert's most recent ventures include Petronius Capital, an equity derivatives proprietary trading firm, and KOTH Gaming, a fantasy sports gambling digital casino. He holds a BA in Physics and Computer Science from Cambridge and an MSc in Mathematics from Oxford. Robert's experience highlights the importance of adaptability and constant learning in the ever-evolving world of trading.

From Robert's experience, past performance is no guarantee of future results. The ability to adapt and stay informed about changing market conditions is crucial in identifying potential risks and opportunities. By supplementing backtesting with forward testing, stress testing, and qualitative assessments, traders can develop more robust and resilient trading strategies.