Development Process — Strategy Generation

This content has been archived. It may no longer be relevant In today’s efficient markets, finding a trading edge is never easy. StrategyQuant is a strategy development platform that uses genetic evolution to automatically generate novel strategies. These strategies can be subsequently funneled through a series of robustness tests, ranging from Monte Carlo simulations to walk forward optimizations, helping you select a strategy that is likely to maintain its edge in live trading. Originally created for the forex market, StrategyQuant has since expanded to the futures and equities markets as well. Due to the efficiency with which profitable strategies can be generated, it has now superseded Metatrader4 as my primary development platform. The strategies developed in StrategyQuant can be exported to Metatrader, TradeStation and MultiCharts for live trading. Some of StrategyQuant’s key functionalities are summarized below: Strategy generation can either be done manually, as is the case for the MT4 development workflow, or automatically using StrategyQuant’s genetic algorithm. For manual generation, StrategyQuant comes bundled with AlgoWizard, an intuitive strategy builder that allows you to code your algorithm using human-readable pseudocode. The final strategy developed in the MT4 workflow is shown below.

Figure 1: The CCI trend system previously developed can be conveniently coded using AlgoWizard

Automatic strategy generation can be achieved using either StrategyQuant’s genetic evolution or random generator. In both cases, strategy building blocks (e.g. indicators, entry/exit types) are specified beforehand, which StrategyQuant then combines to produce a viable strategy. This process is discussed in more detail below. Regardless of which route you take, programming knowledge is not required.

1.2. Robustness Testing

Regardless of how well executed it is, your backtest probably represents a single run over a particular piece of historical data. Because markets are always changing, you can expect your strategy performance to differ once it is exposed to different market conditions. Robustness tests give you an estimate of the risks and performance deterioration you can expect to encounter in live trading.

Figure 2: StrategyQuant comes with a variety of robustness tests

I frequently use Monte Carlo simulations and walk forward optimizations for my robustness testing. We will make use of these after our strategies have been generated.

1.3. Data Management

StrategyQuant comes with its inbuilt data management module, QuantDataManager, allowing you to download/import data from a multitude of sources. Since I mainly trade forex, I download both one-minute and tick forex data from Dukascopy, for use during strategy generation and robustness testing. This tick data can also be exported to MT4/5, allowing you to run 99% quality backtests in MT4/5.

Figure 3: StrategyQuant offers numerous data sources to suit your trading market

Similar to the MT4 development workflow, we will use StrategyQuant to develop a trend system for the hourly GBPJPY. Taking advantage of StrategyQuant’s efficient backtest engine, we will expand the backtest period from 5 years to 16.5 years (August 2003 – Jun 2020). We will start by automatically generating a large basket of potential strategies, which will then be subjected to increasingly stringent robustness tests. This article will cover the specifics of the strategy generation process, while the subsequent robustness testing will be discussed here.

2. Genetic Algorithms

Strategy generation using genetic evolution is one of StrategyQuant’s unique features. Before we dive into the details of the strategy generation process, let’s take a brief excursion into the field of genetic algorithms. Inspired by Darwin’s theory of evolution, genetic algorithms mimic the biological processes of natural selection, reproduction and mutation to create increasingly profitable strategies. Think about the various elements of your strategy, and how many possibilities exist for each of them. For example, what indicator should you use for your entry? Should you use a market or limit order to enter the market? Should you then add a time stop or a profit target? It quickly becomes evident that there are infinite ways to build a strategy. Genetic algorithms can help by: 1. Shrinking the search space by removing unprofitable system elements, significantly reducing computation time 2. Discovering optimal combinations of system elements/indicators which you might never have thought of The sections below will give a brief overview of the genetic evolution process. For a more in-depth explanation, I refer you to Denny Hermawanto’s excellent paper titled ‘Genetic Algorithm for Solving Simple Mathematical Equality Problem.’ The terms used in genetic algorithms are analogous to those used in biology.

  • Population: This refers to the pool of strategies you currently have. To kick-start the evolution process, an initial population will be randomly generated using the input information
  • Generation: The entire population during each iteration of the strategies’ evolution
  • Fitness: A measure of how well the strategy meets the desired goals. Fitness functions are metrics used to quantify the strategy’s fitness. Net profit is a simple example.
  • Crossover: Like human reproduction, crossovers create a new generation of strategies by combining two parents from the current generation
  • Selection: The process of giving fitter strategies a higher likelihood of being used for crossovers
  • Mutation: The random replacement of one of the strategy’s elements with another from the input information

The specifics of each genetic algorithm will vary, but the general outline of the process is as follows:

Figure 4: Basic representation of the inner workings of a genetic algorithm

The inputs specified (e.g. indicators used, entry/exit types) are randomly combined to form a viable strategy, creating the initial population. Fitness is computed for each strategy in the population, and fitter strategies are assigned higher probabilities of being used for crossovers. During each crossover, the two ‘parent’ strategies are split at a randomly determined point and the resulting building blocks are spliced together to create an ‘offspring’ strategy. Figure 5 illustrates this concept:

Figure 5: Typical crossover operation used in genetic algorithms

A split point was randomly chosen to be after the third building block. The child thus receives the first 3 blocks from mom, while the remaining 2 come from dad. This child subsequently replaces one of the parent strategies. The final operation in each evolution cycle is mutation, where one of the strategy’s building blocks is randomly replaced by another from the pool of inputs. The child created above is used to illustrate this:

Figure 6: Typical mutation operation used in genetic algorithms

After crossover and mutation are complete, the next generation of strategies is created. The processes of selection, crossover and mutation are repeated until the evolution is stopped, thus progressively creating fitter strategies. Note that genetic algorithms (and trading) operate based on probabilities. There is no guarantee that the next generation of strategies will be fitter than the current one, but over the course of evolution spanning multiple generations, strategy fitness is very likely to improve.

3. Building Strategies in StrategyQuant

Armed with an understanding of genetic algorithms, it’s time to dissect StrategyQuant’s generation process. As illustrated in Figure 7 below, we begin by configuring our strategy elements and genetic settings. These inputs are fed into the genetic algorithm, which progressively generates profitable strategies using the evolutionary processes described above. Finally, strategies which pass our performance filters are saved into the databank. Let’s briefly discuss the build configuration for our hourly GBPJPY trend system.

Figure 7: Strategy generation flow chart for StrategyQuant

3.1. Building Blocks

We’ll begin by selecting the entry and exit types for our strategy, followed by the indicators used.

Figure 8: Our trend strategy will contain stop entry orders, along with a mandatory stop loss and time stop

As we discovered in the Market Research section, GBPJPY has high volatility and trends well, although frequent whipsaws and false breakouts should be expected. Let’s try to eliminate some of the false entries by using stop entry orders, where the entry price is set at a certain distance above (below) the current market price for long (short) trades. Some market confirmation is thus required before we enter the market. There are numerous options for the exit types. I prefer using stop losses to control risk. For simplicity, let’s select only the stop loss and time stop. The CCI trend strategy developed in MT4 also happens to contain these two exits. You may wish to experiment with different exit types, depending on the type of strategy you wish to build, and the market you will trade it on.

Figure 9: Some common indicators have been selected to form the backbone of our strategy

StrategyQuant comes loaded with a whole bunch of indicators. You can even import your own custom indicators. I prefer using ‘traditional’ indicators such as Bollinger bands and moving averages. Don’t let their simplicity fool you; they have stood the test of time because they work.

3.2. Parameter Ranges

Let’s now specify the parameter ranges for our building blocks above.

Figure 10: Parameter ranges for stop losses and indicators

Both pip and ATR-based stop losses will be tested, and the indicator lookback periods will be within the 10-100 range. Since the genetic algorithm will be running countless iterations with different combinations of building blocks, your selected ranges should be large enough to accommodate a variety of market conditions and strategy types. The entry rule will use no more than 2 conditions; adding more may negatively affect the strategy’s robustness.

3.3. Money Management

Money management amplifies your profits when your strategy performs well, and preserves your capital during periods of drawdown. It is thus advisable to have some sort of money management or position sizing scheme in place during actual trading. During strategy development, however, I prefer to use a fixed lot size for the backtest. Consider the two equity curves below:

Figure 11: The tripling of profits shown in the bottom backtest is purely due to variable position sizing

Both are identical strategies with $3000 initial capital. The first strategy traded a fixed 0.1 lots throughout, producing a $6100 profit. The second strategy traded 0.1 lots for every $3000 of account balance, meaning position sizes increased as the account grew, eventually producing $18600 profit. This tripling of profits was purely due to position sizing. From the equity curves, it is easy to be tricked into thinking that the second strategy has a superior trading edge. By removing position sizing during development, we can get a clearer picture of the strategy’s edge over the course of the backtest.

Figure 12: Configuring our strategy to trade 0.01 lots throughout the backtest

For convenience, I keep my lot size fixed at 0.01. The very popular fixed fractional position sizing method is also available. Your position sizing method can be optimized to suit a particular backtest, although this introduces too much complexity for my liking.

3.4. Backtest Data

The data settings are fairly similar to what you get in MT4, except that you can factor in slippage, commissions and minimum order distance. The ‘Selected timeframe only’ test precision is similar to MT4’s ‘Open prices only’ model.

Figure 13: Detailed transaction costs and an 8-year out-of-sample test will be included during strategy generation

One of my favourite features is the ability to run out-of-sample tests during strategy generation itself. This means testing your strategy on fresh data that was not previously used for optimization, and is a popular method of determining how your strategy will perform in live trading. To take advantage of this inbuilt robustness test, you will need to break down the data portion into training (in-sample) and out-of-sample portions. Only the training data (white portion) will be used for strategy optimization. Your training data should be long enough to cover different market conditions, while leaving sufficient out-of-sample data for robustness testing. For simplicity, let’s just use a 50-50 data split; 8 years of data should be sufficient for each portion. In Section 3.6, we will set our performance filters to select only strategies with good out-of-sample performance.

3.5. Genetic Settings

If you decide to use random strategy generation instead of genetic evolution, this section will not be relevant. In random generation, building blocks are randomly combined to create strategies; the evolutionary processes (selection, crossover, mutation) are not used. There is some argument that random generation lowers the amount of curve fitting and thus produces more robust strategies. I am unsure about this, but I know it takes far longer to generate profitable strategies with random generation. All the strategies I am currently trading are a result of genetic evolution. This section is where we input the evolution settings for our genetic algorithm. These algorithms are the subject of ongoing scientific research, and choosing the optimal parameters is a non-trivial task. I find that the default settings in StrategyQuant work reasonably well.

Figure 14: Key settings for the genetic evolution. The optimal settings may vary depending on your strategy and market. 

The recommended population size is 10-100. A population that is too small provides little diversity and leaves little room for improvement, while an excessively large one causes long computation times. With islands, StrategyQuant allows several populations of strategies to be simultaneously evolved. The product of each island’s population size and the number of islands gives you the total number of strategies managed by the algorithm. Strategies can migrate from island to island, helping to ‘rejuvenate’ some strategies whose fitness may have reached a plateau. This should only happen occasionally though; otherwise we would lose the diversity of the independent islands. Crossover probability refers to the probability that each strategy will be used for crossover. A probability of 1 means the subsequent generation of strategies will be created entirely from crossovers of the current strategies. In the literature, values of 0.5-0.9 are commonly used. Mutation probability is the likelihood that a randomly selected element from each strategy will be swapped with another element from the pool of building blocks. A low mutation probability is beneficial because it adds diversity. If it is set too high, the evolution will become too random, and population fitness will not increase.

Figure 15: The return/maximum drawdown ratio is a popular measure of risk-adjusted performance, and will be used to drive the genetic evolution.

This is where we determine our fitness function. I discuss some essential characteristics of performance metrics here, and the return/drawdown ratio is one popular option that meets our needs. You also have the option of creating a custom weighted fitness function consisting of numerous metrics. The ‘main data backtest’ option means that only our strategies’ performance on GBPJPY will be used to compute fitness. An alternative is to compute the fitness over an entire portfolio of markets. This is a great way to build robust strategies, but computation time will rise significantly. Finally, let’s talk about evolution management. This corresponds to the ‘End Build?’ block in Figure 7.

Figure 16: The genetic algorithm will run continuously for 6 hours. If fitness stagnates, evolution will restart from generation #1.

I usually let the entire evolution process repeat itself after the maximum number of generations has been reached. You can decide to end the evolution after a set amount of time, or after a target number of strategies has been generated. Genetic algorithms operate based on probabilities, and there is a small chance that the strategies get weaker as they evolve. If this happens, you have the option of restarting the evolution if population fitness stagnates over a set number of generations.

3.6. Performance Filters

Since the genetic algorithm will be generating hundreds of strategies per minute, performance filters are needed to separate the wheat from the chaff. It is inevitable that certain combinations of building blocks will not produce viable strategies. For example, some entry conditions may result in zero trades, or the entry and exit price levels may be too close together. To counter this, StrategyQuant comes with a slew of automatic filters shown below.

Figure 17: Automatic and custom filters in StrategyQuant

Custom filters allow you to set your own performance benchmarks for the build. To screen for consistency over the backtest period, you can specify performance filters for every data segment. I try not to be too strict at this point; the subsequent robustness tests tend to be extremely demanding. Since the 16-year data period was equally split into the training and out-of-sample segments, let’s specify the filters for each segment. For simplicity, I will only use the following: 1. Number of trades At least 300 over the entire backtest period, which translates to about 1.5 trades per month. 2. Profit factor A 1.30 profit factor over each 8-year period is about the minimum performance I can accept. 3. Expectancy I use this to check whether transactions costs can be covered. With a 0.01 lot size, the $0.50 specified translates to about 5 pips. 4. Longest trade With the array of economic reports that are periodically released, I do not wish to hold my trades for longer than 30 days. That’s it for settings!

4. Strategy Generation Results

Using the above settings, a 6-hour build was done on the hourly GBPJPY. Note that genetic algorithms are computationally intensive programs. The build statistics are shown below:

Figure 18: Build statistics for our hourly GBPJPY strategy after 6 hours

The genetic algorithm iterated through half a million strategies, of which 98% failed our performance filters. The reasons for failure are shown above. Our databank reached its limit of 9999 strategies, and these strategies will be sent for robustness testing next.

5. Manual Strategy Generation & Custom Templates

If you have a promising trading idea to test, you can simply code it in AlgoWizard and run it through your performance filters and/or other robustness tests. A middle ground between manual and automatic generation is the use of custom strategy templates. Each automatically generated strategy is based on a template, where placeholder blocks are used for each trading condition and action. The genetic algorithm then proceeds to fill these placeholders with the building blocks specified. By creating a custom strategy template, you can create your own logic structure for the generated strategies. More information using templates can be found here. StrategyQuant’s core functionalities, namely automatic strategy generation and robustness testing, were briefly discussed. The concept of genetic algorithms was introduced and we set up StrategyQuant’s genetic algorithm to generate a trend system on the hourly GBPJPY. After running the algorithm for 6 hours and feeding the resulting strategies through our performance filters, a healthy databank of strategies was produced. These strategies will now need to pass our robustness tests before they can be considered for live trading. Next: Additional Market Testing

A

Additional Market Testing

A

Walk-Forward Optimization

Supercharge your strategy development with StrategyQuant

Access 14-day FREE trial here!

Get up to USD 300 discount!

Strategies need improvement?

Use QuantAnalyzer’s powerful analysis tools

Try the FREE version here!

Get 20% Discount here!

Hey there, Wayne here! I’m on a mission to develop robust algorithmic trading strategies for the forex markets. Trading Tact is where I share my trading methods and insights.