Backtesting is a topic that always attracts one set of traders and despised by the others 🙂 The topic is pretty vast and this blogpost is just a basic primer on Backtesting. It is gonna be a long one but i believe, it will be worth your time 🙂

Let us etch something in our mind very clearly before getting into details – Discretionary trading methodologies/ideas cannot be backtested. You need a set of rigid entry/exit rules to backtest the idea. There are no 2 ways about it. One can act smart and twist words around it but there is no backtesting for discretionary systems – period.

Introduction

Backtesting an idea (in trading terms) is to gauge how a set of trading rules performed on historical data. But, many folks who come into trading just do data mining. Data mining simply scans historical dataset for rules that would have worked in the past. We have read this disclaimer various times that ‘past performance does not guarantee future results’ But still, the professed value of backtesting is based on the premise that historical tendencies/patterns repeat.

Backtesting and its efficacy

So, the popular question is if ‘past performance does not guarantee future results, why backtest an idea/system in the first place? One could very well save the time, effort and sleepless nights right?’ This is an argument that has some validity, up to a point. It is definitely true that past performance is not indicative of future results. But does it therefore mean that historical testing has no validity? I don’t think so.

Here is a case in point. Let’s say you want to build a model of the sun rising. Every day for a month, you get up before dawn, and wait for the sun to appear. Every day, it rises in the east. So, you build your model, run it for tomorrow, and it “predicts” that the sun will rise in the east. Will it? Who knows for certain? Some strange axis switching or earth rotation reversing could occur overnight or a cosmic phenomenon might occur (which could be understood only by Late Stephen Hawkings 🙂 ) and the sun could rise in the north, south, or west. Highly unlikely, yes, but so was the financial crisis of 2008, Nifty flash crash in Oct 2012 and the recent coronovirus debacle. Outlier and unexpected events can and do happen.

If such a calamity occurs, does it mean that the model is useless and never should have been built? No, but certainly you would have to now take into account that the world you modeled has changed significantly. So, having a model based on history is much better than completely guessing. With guessing, you are likely to be looking the wrong way when the sun rises tomorrow morning 🙂

Strategy and psyche

Before getting into the nitty-gritty of backtesting, I would like to clear few hitches of system development. It is imperative that the trader should come up with a system to backtest based on his/her traits and psyche.

1. Let us say that you are a profit-target based trader and dont believe in trailing the profits much. If someone gives you a profitable(system with positive expectancy) blackbox system to trade, sooner or later, you will abandon the system or will tweak it in few trades. Reason is very simple. The system skeleton and ideologies are not attuned to your psychology(the belief you have about how markets operate). A profit target based trader cannot trade ‘trailing stoploss’ based system (and vice-versa is also true) as there will be internal conflict. Please understand that psychology is like fingerprint – it is unique. Hence, it is very important that we choose a system that is attuned to our psychology.

2. If a trader cannot digest 10 losses in a row of a trend following system, there is no point devising one and backtesting it. If your psychology/understanding about the markets do not match what the system can offer, then you shouldn’t be trading that system. Manipulating the rules because you are intolerant with the results can turn a winning system into a big losing one. If you can’t live within the anticipated parameters, then you need to develop or find another system. If one is devising a system for intraday to backtest and he is known to hesitate on taking quick decisions, then it could prove costly in live trading. If a trader vacillates for even two minutes to enter a trade in an intraday system, then that could be the deciding factor between a winner and a loser. With a longer-term trading system, the entry is generally not so critical and hesitation errors are forgiven usually.

Ways of producing backtesting results

Before venturing into the topics in detail, i would like to state that am a big proponent of ‘bar-by-bar’ backtesting as it has number of advantages (but takes a lot of time) compared to automated backtesting (using code).

1. Historical backtesting

This is a kind of testing done by most of the charting software nowadays. You create a few lines of code and they spit out the results with myriad number of statistical parameters. This kind of testing also encapsulates the major problem in backtesting – over-optimisation and curve-fitting (did not get into the definition of these terms as they are commonly known). There is virtually no chance that the results in the future will be close to the ‘optimized’ results. The results are just too “tuned” to the data used in the test.

If a system has indicators (with parameter numbers) incorporated in it, there is higher chance of optimization and curve fitting. Once can play from 1 to 500 as value of the moving average and the s/w will spit out results in seconds. A trader will be tempted to choose the best MA (say 283) out of 500 MAs and guess what, real trading would suck big time as curve-fitting has been done nicely with the past data.

2. Out of sample testing (OOS)

If one is persistent on optimization in backtesting, this kind of testing can be a savior. So, here a trader can test 3 years of historical data testing (automated) and run numerous iterations of optimization but run the following year bar-by-bar. Of course, nothing can beat bar-by-bar testing of a well formed idea (with zero optimization) but this will help those folks who are a bit lazy in backtesting bar-by-bar.

3. Walk forward testing

Walk‐forward analysis is simply the aggregate of many out‐of‐sample periods, stitched together. So, 2 years of automated testing + 1 year of OOS testing followed by 2 years of automated testing followed by 1 year of OOS testing. Once we have a set of 4-5 years of OOS testing, we put them all together and create the statistical parameters. In this testing method, we can get the best of both worlds – a bit of bar-by-bar testing and optimization (if needed)

Importance of data in backtesting

For an accurate evaluation of any system, the data must be impeccable. Online free data will not cut it (a saying flashes through my mind right now – ‘penny wise, pound foolish’). Without correct data, system testing is useless.

Purity of data is another very crucial requirement. Any anomalies or mispriced quotes will have a direct impact on the system’s test results and can skew the results extensively. Cleaning of data is not an easy task and often must be relegated to professional data providers.

Parameters to look out in backtesting

As most of these terms are self-explanatory, I would just like to list out the most important parameters to watch out for.

Percentage profitable, Total number of trades, Max consecutive winning/losing trades, Net points (after cost), Maximum drawdown, max time take to come out of DD (Recovery factor), Ratio of average yearly profit to max DD (Calmar ratio). Max loss and max profit are not that important to me as they are singular events.

I am not a big fan of the popularized ratio – Sharpe ratio. There are 3 reasons for it.

1. It does not include the actual annual return but only the average monthly return. Thus, irregularities in the return are not recognized.

2. It does not distinguish between upside and downside fluctuations. As a result, it penalizes upside fluctuations in the same way as downside fluctuations.

3. It does not distinguish between intermittent and consecutive losses.

Optimization of backtested system

Once we have a trading methodology and an initial set of parameters, it is time to fine tune our approach. The principal folly of optimization is the tendency to curve fit. The problem is more pronounced with systems that involve indicators with numbers (like moving averages, RSI, etc). So, if one is dogged to optimize using a software, use the walk forward testing technique discussed earlier. By doing this testing, one can easily assuage the apprehension of curve fitting to a large extent. Word of caution here – fine-tuning a system just increases the level of false confidence that eventually will be dashed in real time when the system fails. I have backtested numerous systems in the past and have always followed this route – take a concept/idea, create rules based on the concept and test it bar-by-bar without optimization even a single time. Yes – it is an arduous task but in my opinion, it’s well worth the effort and time.

Final thoughts on backtesting

Traders are especially captivated of the percent profitability, or accuracy of a system. In practice, we should not target a specific win/loss ratio – the goal is to create a profitable system (= system with positive expectancy) in accordance with the trader’s psyche. It doesn’t matter whether you get there via relatively few but comparatively large winners, as is the case of trend-following systems, or smaller wins with a tendency to compensate losses and much more. It also could be a combination of the above. Eventually, the totality of the system is all that matters.

Please be cognizant of the fact that several trading luminaries have owed their successful careers to simple trend-following systems. These systems tend to produce winning trades close to 30%. And there is almost always some giveback of profit before a winning trade is closed out. A position’s equity high is never anticipated and profit targets are not used. It’s therefore easier for a trade to wind up as a loser. So essentially, the cardinal parameters of a system (with positive expectancy) could vary from trader to trader but search for the holy-grail system has to stop sometime.

Happy backtesting all !!