Backtesting An Idea/System

Backtesting is a topic that always attracts one set of traders and despised by the others 🙂 The topic is pretty vast and this blogpost is just a basic primer on Backtesting. It is gonna be a long one but i believe, it will be worth your time 🙂

Let us etch something in our mind very clearly before getting into details – Discretionary trading methodologies/ideas cannot be backtested. You need a set of rigid entry/exit rules to backtest the idea. There are no 2 ways about it. One can act smart and twist words around it but there is no backtesting for discretionary systems – period.

Introduction

Backtesting an idea (in trading terms) is to gauge how a set of trading rules performed on historical data. But, many folks who come into trading just do data mining. Data mining simply scans historical dataset for rules that would have worked in the past. We have read this disclaimer various times that ‘past performance does not guarantee future results’ But still, the professed value of backtesting is based on the premise that historical tendencies/patterns repeat.

Backtesting and its efficacy

So, the popular question is if ‘past performance does not guarantee future results, why backtest an idea/system in the first place? One could very well save the time, effort and sleepless nights right?’ This is an argument that has some validity, up to a point. It is definitely true that past performance is not indicative of future results. But does it therefore mean that historical testing has no validity? I don’t think so.

Here is a case in point. Let’s say you want to build a model of the sun rising. Every day for a month, you get up before dawn, and wait for the sun to appear. Every day, it rises in the east. So, you build your model, run it for tomorrow, and it “predicts” that the sun will rise in the east. Will it? Who knows for certain? Some strange axis switching or earth rotation reversing could occur overnight or a cosmic phenomenon might occur (which could be understood only by Late Stephen Hawkings 🙂 ) and the sun could rise in the north, south, or west. Highly unlikely, yes, but so was the financial crisis of 2008, Nifty flash crash in Oct 2012 and the recent coronovirus debacle. Outlier and unexpected events can and do happen.

If such a calamity occurs, does it mean that the model is useless and never should have been built? No, but certainly you would have to now take into account that the world you modeled has changed significantly. So, having a model based on history is much better than completely guessing. With guessing, you are likely to be looking the wrong way when the sun rises tomorrow morning 🙂

Strategy and psyche

Before getting into the nitty-gritty of backtesting, I would like to clear few hitches of system development. It is imperative that the trader should come up with a system to backtest based on his/her traits and psyche.

1. Let us say that you are a profit-target based trader and dont believe in trailing the profits much. If someone gives you a profitable(system with positive expectancy) blackbox system to trade, sooner or later, you will abandon the system or will tweak it in few trades. Reason is very simple. The system skeleton and ideologies are not attuned to your psychology(the belief you have about how markets operate). A profit target based trader cannot trade ‘trailing stoploss’ based system (and vice-versa is also true) as there will be internal conflict. Please understand that psychology is like fingerprint – it is unique. Hence, it is very important that we choose a system that is attuned to our psychology.

2. If a trader cannot digest 10 losses in a row of a trend following system, there is no point devising one and backtesting it. If your psychology/understanding about the markets do not match what the system can offer, then you shouldn’t be trading that system. Manipulating the rules because you are intolerant with the results can turn a winning system into a big losing one. If you can’t live within the anticipated parameters, then you need to develop or find another system. If one is devising a system for intraday to backtest and he is known to hesitate on taking quick decisions, then it could prove costly in live trading. If a trader vacillates for even two minutes to enter a trade in an intraday system, then that could be the deciding factor between a winner and a loser. With a longer-term trading system, the entry is generally not so critical and hesitation errors are forgiven usually.

Ways of producing backtesting results

Before venturing into the topics in detail, i would like to state that am a big proponent of ‘bar-by-bar’ backtesting as it has number of advantages (but takes a lot of time) compared to automated backtesting (using code).

1. Historical backtesting

This is a kind of testing done by most of the charting software nowadays. You create a few lines of code and they spit out the results with myriad number of statistical parameters. This kind of testing also encapsulates the major problem in backtesting – over-optimisation and curve-fitting (did not get into the definition of these terms as they are commonly known). There is virtually no chance that the results in the future will be close to the ‘optimized’ results. The results are just too “tuned” to the data used in the test.

If a system has indicators (with parameter numbers) incorporated in it, there is higher chance of optimization and curve fitting. Once can play from 1 to 500 as value of the moving average and the s/w will spit out results in seconds. A trader will be tempted to choose the best MA (say 283) out of 500 MAs and guess what, real trading would suck big time as curve-fitting has been done nicely with the past data.

2. Out of sample testing (OOS)

If one is persistent on optimization in backtesting, this kind of testing can be a savior. So, here a trader can test 3 years of historical data testing (automated) and run numerous iterations of optimization but run the following year bar-by-bar. Of course, nothing can beat bar-by-bar testing of a well formed idea (with zero optimization) but this will help those folks who are a bit lazy in backtesting bar-by-bar.

3. Walk forward testing

Walk‐forward analysis is simply the aggregate of many out‐of‐sample periods, stitched together. So, 2 years of automated testing + 1 year of OOS testing followed by 2 years of automated testing followed by 1 year of OOS testing. Once we have a set of 4-5 years of OOS testing, we put them all together and create the statistical parameters. In this testing method, we can get the best of both worlds – a bit of bar-by-bar testing and optimization (if needed)

Importance of data in backtesting

For an accurate evaluation of any system, the data must be impeccable. Online free data will not cut it (a saying flashes through my mind right now – ‘penny wise, pound foolish’). Without correct data, system testing is useless.

Purity of data is another very crucial requirement. Any anomalies or mispriced quotes will have a direct impact on the system’s test results and can skew the results extensively. Cleaning of data is not an easy task and often must be relegated to professional data providers.

Parameters to look out in backtesting

As most of these terms are self-explanatory, I would just like to list out the most important parameters to watch out for.

Percentage profitable, Total number of trades, Max consecutive winning/losing trades, Net points (after cost), Maximum drawdown, max time take to come out of DD (Recovery factor), Ratio of average yearly profit to max DD (Calmar ratio). Max loss and max profit are not that important to me as they are singular events.

I am not a big fan of the popularized ratio – Sharpe ratio. There are 3 reasons for it.

1. It does not include the actual annual return but only the average monthly return. Thus, irregularities in the return are not recognized.

2. It does not distinguish between upside and downside fluctuations. As a result, it penalizes upside fluctuations in the same way as downside fluctuations.

3. It does not distinguish between intermittent and consecutive losses.

Optimization of backtested system

Once we have a trading methodology and an initial set of parameters, it is time to fine tune our approach. The principal folly of optimization is the tendency to curve fit. The problem is more pronounced with systems that involve indicators with numbers (like moving averages, RSI, etc). So, if one is dogged to optimize using a software, use the walk forward testing technique discussed earlier. By doing this testing, one can easily assuage the apprehension of curve fitting to a large extent. Word of caution here – fine-tuning a system just increases the level of false confidence that eventually will be dashed in real time when the system fails. I have backtested numerous systems in the past and have always followed this route – take a concept/idea, create rules based on the concept and test it bar-by-bar without optimization even a single time. Yes – it is an arduous task but in my opinion, it’s well worth the effort and time.

Final thoughts on backtesting

Traders are especially captivated of the percent profitability, or accuracy of a system. In practice, we should not target a specific win/loss ratio – the goal is to create a profitable system (= system with positive expectancy) in accordance with the trader’s psyche. It doesn’t matter whether you get there via relatively few but comparatively large winners, as is the case of trend-following systems, or smaller wins with a tendency to compensate losses and much more. It also could be a combination of the above. Eventually, the totality of the system is all that matters.

Please be cognizant of the fact that several trading luminaries have owed their successful careers to simple trend-following systems. These systems tend to produce winning trades close to 30%. And there is almost always some giveback of profit before a winning trade is closed out. A position’s equity high is never anticipated and profit targets are not used. It’s therefore easier for a trade to wind up as a loser. So essentially, the cardinal parameters of a system (with positive expectancy) could vary from trader to trader but search for the holy-grail system has to stop sometime.

Happy backtesting all !!


See all posts »

Procrastination And Ways To Overcome It

I always wondered why traders never get to the point of creating a solid trading plan/system and the reasons behind this phenomenon.

I love writing articles about trading psychology and this topic made me come out of hibernation. This blogpost is a small attempt to see the probable reasons behind procrastinating in formulating a system – from a psychological perspective.

» More »


See all posts »

Achilles heel of a Discretionary trader

Achilles’ heel was an unguarded weakness that ultimately brought down a hero of Greek mythology and this post is an attempt to understand the things that has to be kept in mind while designing a trading methodology as a discretionary trader.

As many of you know that am a mechanical(rule based) trader, my trading day is pretty monotonous and boring, many a times. The other group of traders are often called as ‘discretionary traders’. Both groups have their advantages/disadvantages and one should follow what they are comfortable with. At the end of the day, both discretionary and systematic traders have the same goal – making money.

What is discretionary/mechanical trading?

Discretionary trading is decision based trading – when the trading idea shows up in the charts, the trader decides(at that moment) whether to take the trade or not based on current market conditions. Discretionary trading does not mean random trading. Every trader(mechanical or discretionary) has a methodology to enter/exit the market – there are no two ways about it. For example, even if all the conditions are met for a trade, a discretionary trader will not take the trade as volatility is too low (current market condition) – so, basically the trader decides to let it go seeing the current market condition.

On the other hand, Mechanical trading is driven by rules. The trader do not make any decision on taking the trade but the system does. If A happens, then the trader go long and if B happens, the trader goes short. There is no element of decision making involved by the trader as everything is planned out beforehand and the trader just has to execute those trades.

Designing a trading methodology as a discretionary trader

There are certain things discretionary traders should keep in mind when they design their system. More often than not, the psychological pressure of making the ‘quality’ decision of whether to take the trade or not can overwhelm a discretionary trader. To understand this behavior, we need to analyze how our brain works.

A. How brain works?

Evolution has effectively given us (human beings) a dumb brain and a smart brain. The smart brain runs the show most of the times unless a threat is present/perceived and consequently, the dumb brain takes over. Why? The reason is very simple – Dumb brain is faster at making simple/resolute decisions. This avoids people taking a long time on making a choice as time is critical in predatory situations. If we take too much time to take a decision on ‘fight or flight’ situations, there is a good possibility of ending up in the jaws of a predator 🙂

B. Markets are there to get me

With this understanding in mind, one can easily comprehend why this could create a problem in trading as our natural responses can be inappropriate and the way we view/assess info changes when the dumb brain takes over.

Our brains are built to handle belligerent situations and hence, even in a normal situation in trading (like a loss – by the way, many dont consider trading loss as normal 🙂 ), we tend to do things like making “revenge trades”, which is treating the market like an adversary.

Think about it for a second – this is tantamount to curve fitting data so that it fits our trading model but only here, we are fitting the market to how our behavior and natural responses are designed to interact with predators. This is exactly why we see people saying ‘Markets are there to get me’

C. Demo trading and tunnel vision

As a discretionary trader, when we do demo trading of our ideas, we almost always get superlative results. The reasons could be multi-fold but the important one is so evident. When we practice in a non-stressful situation, we evaluate our success based on how our smart brain handles the situation. Under stress, in real trading, we might find that we fail to notice things that are obvious when we look at the same information after the stress has passed. People tend to overlook/ignore information that is contradictory to their analysis of the situation — popularly known as tunnel vision“.

Let us sit on this ‘tunnel vision’ for a moment. Like the airport traffic controller who ignores contradictory information (as he is affected by stress), traders fail to exit a losing trade, because our discriminative attention ignores things that indicate it’s time to get out of the trade. Practice (or demo trading) does help, but many traders find that their behavior is different under stress.

So, when a discretionary trader designs/practices a system, he needs to consider human behavior, psychology, and the human factors that were discussed here. In another post, i will discuss the difficulties in being a mechanical(systematic) trader.

Final thoughts

As it generally happens in any endeavor specific to a competitive pursuit, people end up with hallowed beliefs without subjecting them to rational scrutiny. The subject of ‘which style of trading is better’ is one such associated with the business of trading and is a favorite ‘peg to hang’ blame for failure in becoming a successful trader by many.

This blogpost does not favor one over the other – Pick the one that best suits you and pursue trading in that direction.

Happy trading all !!


See all posts »

rss | E-Mail | Twitter