August 13, 2007
IO – Out of Sample and Walk Forward Testing
This page is obsolete. Current versions of AmiBroker feature built-in non-exhaustive, smart multithreaded optimizer and walk-forward engine.
As a more thorough verification that a system will perform as anticipated, we should always test the system with out of sample data or in other words with data that has not been seen by the In Sample optimization process graphically represented by:
This can be accomplished in AmiBroker by:
– Setting the from and to dates for our system to wherever we want
– Performing an optimization and choosing the parameter values to use going forward
– Changing the default values of the optimization statements
– Moving the from and to dates forward in time
– Running a back test to see how well the system performs.
Besides automating the whole process above, IO also offers more advanced alternatives such as a fully automated calendar or signal based, anchored or rolling Walk Forward optimization and Out of Sample testing which can be made to be very thorough. Graphically the anchored and rolling walk forward processes look like this:
Without automated tools such as this almost no one performs Walk Forward testing because of the amount of manual intervention that is required. For example think about the manual steps required to perform a Walk Forward test over a 3 year period 3 months at a time which are:
– Set the from and to dates for the original optimization to begin as of some date in time and end as of 3 years ago
– Perform the optimization and choose which parameter values to use going forward
– Change the default values of the optimization statements
– Move the from and to dates forward
– Run a back test for the first three months of out of sample data and record the results
– Then repeat the whole process eleven times, each time moving the end date ( anchored ) or beginning and ending dates ( rolling ) 3 months closer until you run out of data.
Assuming one had the means to manually stitch together the out of sample equity curve this then would provide a real life picture of how the system performed over a 3 year Out of Sample period with reoptimization occurring every 3 months.
The above can be accomplished in IO with no manual intervention and a single Walk Forward Directive which is written like this:
– WFAuto: Anchored: 3: Months
As a result even if it takes 15 minutes to optimize each of the 12 segments to accumulate the data necessary to build and show the tables and the combined equity curves, it can all be done unattended. As a result one only need to set up a run, get it started and then go find something else of interest to do. Besides the tabular results that are produced by IO it is also capable, with an included AFL, of showing an accurate composite of the In and Out of Sample equity curve in AmiBroker that looks like what is below:
The middle pane in the template above shows my replacement for the standard AmiBroker equity curve for the current In Sample optimization. The bottom pane shows the full Walk Forward results and is constructed on the fly as each new Walk Forward segment occurs. The section to the left of the thick vertical bar in the lower pane is the original In Sample optimization period. The sections to the right separated by thinner vertical bars are each of the Out of Sample periods in the Walk Forward analysis.
IO is also capable, with a slightly different form of the directive, of performing signal based Walk Forward processes which calls for reoptimization every time some user selected signal type ( Buy, Sell, Short, Cover, Entry, Exit, Any ) occurs. By definition this implies a variable length Out of Sample period that is dependent on when signals actually occur.
These are advanced features in IO.
A shareware version of IO with full documentation can be found in the AmiBroker Files Section …
http://groups.yahoo.com/group/amibroker/files/IO.zip
Filed by Fred at 5:22 am under Intelligent Optimization
Comments Off on IO – Out of Sample and Walk Forward Testing