High-Precision Delay and Interval Timing

Before continuing with this post you should carefully read the AmiBroker Help topic for the getPerformanceCounter().

Measuring time is an important aspect of all real-time intraday trading systems. Typical tasks requiring high-resolution timing include:

  1. Limiting the message rate to Interactive Brokers (IB) to 50/sec (API Error 100).
  2. Inserting small delays before polling IB Status, to allow for Internet delays.
  3. Staggering (interlacing) portfolio trades to spread out the action.
  4. Measuring and optimizing AFL execution time.
  5. Modifying orders after a small delay (to ensure fills).
  6. Periodic execution of tasks, for example, Watchlist scanning, Display and Status refresh, calculations based on slow changing variables, etc.
  7. Time-Stamping events, for example order placement.
  8. Collecting/preprocessing quotes.
  9. Overlay live tick-charts on faster and easier to manage 1-minute charts.

Most of these tasks can be accomplished using just three custom timing functions:

  1. GetElapsedTime(): A function that returns elapsed time since reset.
  2. SetDelay(): A function that returns time left to reach a future time.
  3. GetDelayTrigger(): A function that returns a trigger when a delay times out.

This post provides example functions in a demo application. To allow the use of many timers each function requires you to provide a TimerName, which will be used to retrieve timer information. Static variables are Global and can be read from anywhere; this means you have to be careful not to cross-reference the timers by using the same TimerName from different panes or windows. When running multiple copies of the same code you will need to key the TimerNames. For more on how to do this, see Keying Static Variables in the Real-Time AFL Programming category.

The timers below are implemented using the getPerformanceCounter(). This function returns the amount of time elapsed since the computer was last started. Tomasz recently explained this as follows: “The underlying high frequency counter runs all the time since computer start. What ‘reset’ flag really does is to store last value so next time you read it, it gets subtracted from last value giving you the difference. If reset is false, the last value is set to zero, and you get the original number of ‘clock ticks’ since computer start“.

The timers in this post do their own sampling of the underlying high frequency counter, and the getPerformanceCounter() Reset argument is always left set to False.

The getPerformanceCounter() returns values with microsecond resolution; however, the practical accuracy is severely limited by interruptions from the computer’s operating system. Do not expect much better than about 50-millisecond absolute accuracy. Aside from designing your own dedicated trading hardware (to replace the PC) there isn’t much that can be done about this. If you are brave, you can experiment with increasing program priority in your Task Manager window.

Chart refreshes are most often initiated by an arriving quote, but they can also be initiated by mouse clicks, tooltip, and various chart operations. This means that, when market activity is low and things are not happening as fast as you would like, you can force extra AFL executions by clicking on your chart. You can verify this by running the code below and, while clicking rapidly on the chart, observe that the timer counts displayed will update more rapidly.

You can ensure a one-second chart refresh by adding a RequestTimedRefresh(1) to your code. If the frequency of your arriving data is slow, your AFL code may execute only sporadically. Since your code must execute to read your timers, the resolution of your timers will be limited by the chart refresh rate. If your chart refreshes once a second your timing resolution will be one second!

Normally most of the AFL code in an Indicator window executes when your chart refreshes; however, to obtain speed advantages, you may execute non-critical sections (like account information and System Status) of your code less frequently using a timer. You can also execute small sections of code more frequently by placing them inside a well-controlled loop. If you do this, be sure to limit the maximum time your code can spend inside the loop to one second or less.

For fast trading systems, the frequency of AFL executions (chart refreshes) may be slow and this may make it difficult for you to get LMT fills. There is no way to have a program that requires 50 milliseconds per pass to execute 20 times per second.

Considering the interval between AFL executions, it is important to plan the layout of your code so that all events are handled in the most efficient order. If you don’t, transmittance of your order could well be delayed by up to a full second. There are situations where you want to invoke an immediate re-execution of your code. In some cases you might want to do this after placing an order to check order status before the next quote or refresh. Although it should be used sparingly this is possible by calling the RefreshAll():

1
2
3
4
5
function RefreshAll() 
{ 
oAB = CreateObject("Broker.Application"); 
oAB.RefreshAll(); 
}

This function can only be called once a second; calling it faster will not result in more frequent chart refreshes. This means you should only call it when really needed.

The code presented below is for demonstration only. The getElapsedTime() lets you measure elapsed time from the moment of Reset. The first argument passes the name you assign to the static timer; this allows you to use the same function to time different events. The second argument is a Reset flag. When this Reset is True, the function samples the underlying high frequency counter and uses it for later reference. When you call the function with the Reset argument set to False, it calculates the elapsed time by subtracting the earlier sampled value from the current value of the Performance Counter.

The setDelay() function lets you Start, Read, and Cancel a time delay. The TimerName argument functions as in the getElapsedTime(). Calling the setDelay() with the mSecDelay argument set to a non-zero value will start the Delay timer. Calling it with the mSecDelay argument set to zero will make it return the current count-down time in millseconds. Calling the function with the cancel argument set to True will terminate the delay.

The getDelayTrigger() function returns a trigger. This is a signal that is true for only one pass through the code. Triggers are frequently used in real-time trading systems. They are needed to prevent multiple actions when a signal becomes True.

To run the code, copy the formula to an Indicator and click Insert. You’ll see a chart window like Figure 1 below:

Figure 1. Result from running the example code.
timerdisplay.jpg

The example code maintains three timers, T1, T2 and T3. All timing values are expressed in milliseconds. In Figure-1 the Elapsed Time shown is measured from timer Reset. The Delay shown is the time remaining after Start, until the delay times out. The line for Timer T2 shows that its Delay just timed-out and produced a trigger. Timer T3 still has a Delay in progress. Right-click on the chart to open the Param window:

Figure 2. Param window.
timerparam.png

If you click one of the timer Resets in the Param window you’ll see the ElapsedTime in the corresponding row go to zero, and then start to increment sporadically when your chart refreshes. Without live data this would be at approximately 1-second intervals, as determined by the RequestTimedRefresh(1);

If you click Start for one of the timers this will start a delay. You can see how it counts down in the Delay column. Click the timer’s Cancel to terminate the Delay. Note that whenever a Delay times out, the word “Trigger” briefly appears in the third column.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
Filename = StrLeft(_DEFAULT_NAME(),StrLen(_DEFAULT_NAME())-2);
 
function RefreshAll()
	{
	oAB = CreateObject("Broker.Application");
	oAB.RefreshAll();
	}
 
function getElapsedTime( TimerName, Reset )
	{
	if( Reset ) 
		{
		TimeRef= GetPerformanceCounter(False);
		StaticVarSet(TimerName,TimeRef);
		}
	TimeRef = Nz(StaticVarGet(TimerName));
	ElapsedTime = GetPerformanceCounter(False) - TimeRef;
	return ElapsedTime;
	}
 
function setDelay( TimerName, MsecDelay, Cancel )
	{
	HRCounter= GetPerformanceCounter(False);
	if( Cancel ) 
		{
		StaticVarSet("TO"+TimerName,-1);
		StaticVarSet("DS"+TimerName, 0);
		}
	else if( MsecDelay ) 
		{
		StaticVarSet("TO"+TimerName,HRCounter+MsecDelay );
		}
	TimeOutTime = Nz(StaticVarGet("TO"+TimerName));
	DelayCount = Max(0, TimeOutTime - HRCounter );
	StaticVarSet("DC"+Timername, DelayCount);
	return DelayCount;
	}
 
function getDelayTrigger( TimerName )
	{
	DelayCount = Nz(StaticVarGet("DC"+TimerName));
	DelayState = DelayCount > 0;
	PrevDelayState = Nz(StaticVarGet("DS"+TimerName));
	StaticVarSet("DS"+TimerName, DelayState);
	return DelayState < PrevDelayState;
	}
 
RequestTimedRefresh( 1);
 
_SECTION_BEGIN("TIMER 1");
Reset1 = ParamTrigger("1 - Reset","RESET");
MSecDelay1 = Param("1 - Delay (mS)",1000,0,10000,10);
if( ParamTrigger("1 - Start", "START") ) setDelay( "Timer1", MSecDelay1, 0 );
if( ParamTrigger("1 - Cancel", "CANCEL") ) setDelay( "Timer1", MSecDelay1, 1 );
_SECTION_END();
 
_SECTION_BEGIN("TIMER 2");
Reset2 = ParamTrigger("2 - Reset","RESET");
MSecDelay2 = Param("2 - Delay (mS)",4000,0,10000,10);
if( ParamTrigger("2 - Start", "START") ) setDelay( "Timer2", MSecDelay2, 0 );
if( ParamTrigger("2 - Cancel", "CANCEL") ) setDelay( "Timer2", MSecDelay2, 1 );
_SECTION_END();
 
_SECTION_BEGIN("TIMER 3");
Reset3 = ParamTrigger("3 - Reset","RESET");
MSecDelay3 = Param("3 - Delay (mS)",8000,0,10000,10);
if( ParamTrigger("3 - Start", "START") ) setDelay( "Timer3", MSecDelay3, 0 );
if( ParamTrigger("3 - Cancel", "CANCEL") ) setDelay( "Timer3", MSecDelay3, 1 );
_SECTION_END();
 
ET1 	= getElapsedTime( "Timer1", Reset1 );
ETA1 	= setDelay( "Timer1", 0, 0 );
TT1 	= getDelayTrigger( "Timer1");
 
ET2 	= getElapsedTime( "Timer2", Reset2 );
ETA2 	= setDelay( "Timer2", 0, 0 );
TT2 	= getDelayTrigger( "Timer2");
 
ET3 	= getElapsedTime( "Timer3", Reset3 );
ETA3 	= setDelay( "Timer3", 0, 0 );
TT3 	= getDelayTrigger( "Timer3");
 
Title = "\nFilename: "+Filename+"\n\n"+
"MilliSeconds Since Computer Startup: "+NumToStr(GetPerformanceCounter(False),1.3)+"\n\n"+
"Timer     ElapsedTime     Delay     Trigger\n\n"+
"T1      "+NumToStr(ET1,10.0)+"     "+NumToStr(ETA1,7.0)+"      "+WriteIf(TT1,"TRIGGER1","")+"\n\n"+
"T2      "+NumToStr(ET2,10.0)+"     "+NumToStr(ETA2,7.0)+"      "+WriteIf(TT2,"TRIGGER2","")+"\n\n"+
"T3      "+NumToStr(ET3,10.0)+"     "+NumToStr(ETA3,7.0)+"      "+WriteIf(TT3,"TRIGGER3","");

Edited by Al Venosa.

1 Star2 Stars3 Stars4 Stars5 Stars (2 votes, average: 3.50 out of 5)
Loading ... Loading ...

Data Resources – Real Time Equity

A list of unqualified data resources with a bias towards Real Time equity data (mainly for the American market). Some of the sites do provide data for other financial instruments as well.

The list is dynamic and subject to change without notice.

The list is ordered according to when the author researched the site and it is not sorted by rank or order of merit.

CAUTION – MOST OF THE LINKS CONTAIN COMMERCIAL REFERENCES OR LINK TO COMMERCIAL SITES.

NOTE: THE AUTHOR HAS NO COMMERCIAL AFFILIATIONS OF ANY KIND AND RECEIVES NO GRATUITIES, OR BENEFITS, FROM ANY COMMERCIAL ORGANIZATION OR INDIVIDUAL INVOLVED IN COMMERCIAL ACTIVITIES. NOR IS HE PERSONALLY ENGAGED IN ANY COMMERCIAL ACTIVITIES, RELATED TO TRADING, IN ANY WAY. THE LINKS ARE PROVIDED AS A RESOURCE AND FOR EDUCATIONAL PURPOSES. THEY ARE NOT A RECOMMENDATION, ON THE PART OF THE AUTHOR, NOR DO THEY CONSTITUTE INVESTMENT ADVICE AND SHOULD NOT BE CONSTRUED AS SUCH.

4) Paritech

http://www.paritech.com.au/default-au.htm

A Melbourne (Australian) based provider of Real Time market data for the Australian Stock Exchange.

 Additional Products:

Unlimited real time data for the Australian markets using Paritech’s Pulse software or DDE functionality – Real Time export function – export 50 time series live fields and events out of PULSE into your own application e.g. Excel and other compatible software.

http://www.paritech.com.au/support/pulse/features/dde.asp

3) TechnicalTrends

 http://www.technicaltrends.com/

Indian stock-market data – quality end of day and intra day data for stocks, futures & commodities traded on Indian exchanges (NSE, BSE, NCDEX and MCX). Extensive historical data is available. Intra day data (30 minute) is available for ALL stocks in the NSE.  End of day data is provided in Computrac format (now popular as Metastock format)  while intra day data is in text files easily accessed by all technical analysis software. Commodities data is provided for all contracts listed on the NCDEX & MCX.  Data is available in individual, as well as continuous contracts.

2) ViraTech

http://www.viratechindia.com/index.html

Indian stock-market data – intraday data for active cash stocks, as well as ALL current month Futures Stocks & Indices. Historical data / Continuity / Offline option – with a large intraday history, you have completely continuous authentic data for over 15000 5-minute bars.

1)  eSignal

A reliable vendor for intraday data feed in Amibroker format including International baskets e.g. futures of National Stock Exchange(NSE, India).

1 Star2 Stars3 Stars4 Stars5 Stars (No Ratings Yet)
Loading ... Loading ...

Data Resources – EOD Equity

A list of unqualified data resources with a bias towards EOD equity data (mainly for the American market). Some of the sites do provide data for other financial instruments as well.

This is an unofficial resource guide. For a complete list of AmiBroker supported data providers refer to: http://www.amibroker.com/guide/h_quotes.html

The list is dynamic and subject to change without notice. It is ordered according to when the author researched the site and it is not sorted by rank or order of merit.

CAUTION – MOST OF THE LINKS CONTAIN COMMERCIAL REFERENCES OR LINK TO COMMERCIAL SITES.

NOTE: THE AUTHOR HAS NO COMMERCIAL AFFILIATIONS OF ANY KIND AND RECEIVES NO GRATUITIES, OR BENEFITS, FROM ANY COMMERCIAL ORGANIZATION OR INDIVIDUAL INVOLVED IN COMMERCIAL ACTIVITIES. NOR IS HE PERSONALLY ENGAGED IN ANY COMMERCIAL ACTIVITIES, RELATED TO TRADING, IN ANY WAY. THE LINKS ARE PROVIDED AS A RESOURCE AND FOR EDUCATIONAL PURPOSES. THEY ARE NOT A RECOMMENDATION, ON THE PART OF THE AUTHOR, NOR DO THEY CONSTITUTE INVESTMENT ADVICE AND SHOULD NOT BE CONSTRUED AS SUCH.

6) Paritech

http://www.paritech.com.au/default-au.htm

A Melbourne (Australian) based provider of market data utilizing Paritech’s DataDirector software.

Which Markets does Data Director Support?

DataDirector allows you to register for and download from one to multiple world markets:

Australian Stock Exchange (ASX)
Sydney Futures Exchange (SFE)
New York Stock Exchange (NYSE)
National Association of Securities Dealers (NASDAQ)
American Stock Exchange (AMEX)
London Stock Exchange – Domestic
Singapore Stock Exchange (SES)
Overseas Indices, Foreign Exchange Rates and Futures

Note: the bias is towards the Australian market and the Metastock format (all of the features listed may not apply to all exchanges – international customers should contact Paritech support for information on international data).

What format is the downloaded data in ?

DataDirector has a filter that enables you to format the output files (ASCII, Comma Separated Variable (CSV), Metastock) and to handle corporate actions (these can include splits, name changes and additions) plus Sector (GIC’s) and Favourites organization.

Where is the data sourced from?

Data is sourced, where possible, directly from the relevant Stock Exchange and also Reuters, one of the world’s leading suppliers of Real Time data feeds.

What happens if I miss a days download?

DataDirector allows you to download up to two months history data if you are unable to download for a period of time. Longer periods can be downloaded by calling our technical support staff.

How can I obtain historical data for periods longer than the two months available for downloading?

Historical data CD’s are available for all the supported markets (free with subscription packages).

The History CD for the Australian Stock Exchange (ASX) is fully back adjusted to ensure the integrity of your database is perfect. All ASX splits, ASX name changes and ASX consolidations are completed. The ASX History is typically 14-16 years, plus many of the blue chip stocks and indices go back over 25 years. History is also available for ETO’s (Exchange Traded Options) and Warrants.

History CD’s for the United States (NYSE, NASDAQ, Amex), Europe (LSE) and Singapore (SSX) markets going back over all securities approximately 10 years.

http://www.paritech.com.au/AU/products/data/renewal.asp

Additional Products:

a) COT (Commitment of Traders Reports) historical databases available.

http://www.paritech.com.au/AU/products/data/cotdata.asp

b) Unlimited real time data for the Australian markets using Paritech’s Pulse software or DDE functionality – Real Time export function – export 50 time series live fields and events out of PULSE into your own application e.g. Excel and other compatible software.

http://www.paritech.com.au/support/pulse/features/dde.asp

c) EOD and historical EOD data for:

Futures Prices – financial and commodity futures from 13 international exchanges. 
Cash Prices – major international stock indices and cash metal prices.

 http://www.paritech.com.au/AU/products/data/datatools.asp

d) EOD data for Foreign Exchange – 13 spot cross-rates against the U.S. Dollar.

 http://www.paritech.com.au/AU/products/data/datatools.asp

 

5) TechnicalTrends

 http://www.technicaltrends.com/

Indian stock-market data – quality end of day and intra day data for stocks, futures & commodities traded on Indian exchanges (NSE, BSE, NCDEX and MCX). Extensive historical data is available. Intra day data (30 minute) is available for ALL stocks in the NSE.  End of day data is provided in Computrac format (now popular as Metastock format)  while intra day data is in text files easily accessed by all technical analysis software. Commodities data is provided for all contracts listed on the NCDEX & MCX.  Data is available in individual, as well as continuous contracts.

 

4) ViraTech

http://www.viratechindia.com/index.html

Indian stock-market data – complete historical data and daily updates for all stocks at BSE, NSE and CSE including complete Futures & Options data plus exhaustive Commodity, Forex and World Market data . Metastock format automatically adjusted for Splits / Bonus / Rights / Name Changes etc. as and when they occur.

Complete Historical Data would be provided for :

  • BSE (From 1984, Sensex from 1979 with volumes)
  • NSE (From its inception in 1994)
  • US Stocks (including Nasdaq 100, S&P 500, ADR / GDR’s) comprising of almost 25 years of history!
  • 40+ World Market Indices starting from 1980
  • International (starting 1970) and National Commodity data (from Inception of Exchange’s)

3) BSE-NSE2005

http://in.groups.yahoo.com/group/bse-nse2005/

A community service Indian Yahoo Group created for the purpose of sharing data files. 

Join the forum to get Free End of Day Data for Bombay Stock Exchange {BSE}, National Stock Exchange {NSE}, Futures Data for F&O Segment of National Stock Exchange and Commodities Data for NCDEX MCX. The data includes not only the OHLC but also Delivery volumes which is a value addition and which cannot be found from data elsewhere.
We provide the Data in ASCII format and as such can be directly imported into most Charting softwares such as AmiBroker, MetaStock, etc.

2) JustData

 http://www.justdata.com.au/

EOD data for Forex, Stocks, Commodities and Futures from Australia, Asia, Europe, Africa, North and South America (stock market data from over 90 World Exchanges) http://www.justdata.com.au/Products/BodhiFreeway/bg_data.htm#stocks

Stock data is biased towards the Australian market (ASX equity data includes accurate dividend adjustment, using dividend values as recorded by the ASX, GIC’s sector classification, de-listed stock and current fundamental data).

Historical data – shipped as a CD ROM – available with data subscription packages or standalone – deep histories – Equities & Indices for international markets – Equities, Indices and Warrants for London and the major Asian and American markets – Equities, Indices, Income, Warrants and Options for the Australian market – plus World Indices – plus Futures (Commodities) & Indices for America, Europe and Asia – plus the dollar cross rates (Forex) for the major currencies.

http://www.justdata.com.au/Products/BodhiHistory/index.htm

Bodhi History includes expired symbols on the ASX since 1990 and on other Exchanges since JustData started disseminating them.

The future and commodity markets contain continuous contracts and current contracts for all traded symbols.

Choose to adjust your databases for Structural Events such as Share Splits; Distribution events such as Renounceable issues; Capital Change and Dividends (the author is not certain of this applies to all markets). You can also adjust Overseas Future & Commodity data at the time of a contract swap.

Compatible with AmiBroker (select ASCII, CSV or spreadsheet formats).

1) Commodity Systems Inc

http://www.csidata.com/

EOD and historical databases for:

  1. North American and World futures (Commodities).
  2. U.S. and non-U.S. stocks and stock indices (mainly Canada and U.K).
  3. Options on World indices
  4. Futures options.
  5. Mutual Funds.
  6. Briese’s Commitment Of Traders (COT) data.
  7. Single stock futures.
  8. Plus many fundamental and econometric series such as interest rates and housing starts.
  9. Data for de-listed stocks available by subscription.
  10. Long historical databases available.
  11. Historical inter-market correlation studies.
  12. and more.

Downloadable text files for the current constituents of the supported markets.

http://www.csidata.com/csi/exchangelist/index.html

1 Star2 Stars3 Stars4 Stars5 Stars (No Ratings Yet)
Loading ... Loading ...

Yahoo Data FAQ’s

An unsorted, unqualified list of FAQ’s relating to anything and everything to do with Yahoo! Finance data.

The list is dynamic and subject to change without notice.

Comments, where included, are the opinion of the individuals concerned and are not necessarily correct. In some cases they may be ‘out of date’ as circumstances change with time.

CAUTION - SOME OF THE LINKS CONTAIN COMMERCIAL REFERENCES OR LINK TO COMMERCIAL SITES.

NOTE: THE AUTHOR HAS NO COMMERCIAL AFFILIATIONS OF ANY KIND AND RECEIVES NO GRATUITIES, OR BENEFITS, FROM ANY COMMERCIAL ORGANIZATION OR INDIVIDUAL INVOLVED IN COMMERCIAL ACTIVITIES. NOR IS HE PERSONALLY ENGAGED IN ANY COMMERCIAL ACTIVITIES, RELATED TO TRADING, IN ANY WAY. THE LINKS ARE PROVIDED AS A RESOURCE AND FOR EDUCATIONAL PURPOSES. THEY ARE NOT A RECOMMENDATION, ON THE PART OF THE AUTHOR, NOR DO THEY CONSTITUTE INVESTMENT ADVICE AND SHOULD NOT BE CONSTRUED AS SUCH. 

 

Q7. I want to use Yahoo data for the ASX. Is it accurate?

A7. The use of Yahoo data for ASX stocks is not good. 

1.    No delisted history – any system testing you do based on Yahoo data is subject to population bias.  ie you can only test on the stocks that are currently listed.

2.    Significant inaccuracies for any stock exchange that trades in increments less than 1c.  This includes the Australian Stock Exchange (ASX), Singapore Exchange, some NASDAQ stocks and most OTCBB & Pink Sheet stocks.  This happens because Yahoo’s data is rounded to the nearest cent.  For example, there are 14000 listed instruments on the ASX.  Over 8000 of them are currently trading below 20c which means they move in either 0.5 or 0.1c increments.  Since Yahoo only prices in cents this introduce VERY SIGNIFICANT innacuracies.  – eg. a 2.6c shown as 3c represents a 15% error). 

3.    Normal dividends are adjusted by subtracting all historical data by the dividend amount.  Exchanges such as the ASX do not adjust price data for standard dividends.  By diluting it the way they do, any technical analysis or back-testing no longer shows a correct % return, nor does it show critical technical levels such as support and resistance.

4.    No sector information – critical if you want to perform sector-based analysis.

5.    Significant number of missing stock splits and other capital adjustments.

Re the rounding, if the data is adjusted (for splits/rights issues etc.) then you want plenty of decimal places so your testing shows the correct % gain.  In years past, the ASX had a different mechanism for determining the open and closing price of a stock, ending up with a weighted average of the bids/asks to come up with one open/close price (which would often be a fractional amount).  They changed this method a few years ago to more of a volume-matching method so that it’s a “normal” trading level.

Richard Dale.

Norgate Investor Services
- Premium quality Stock, Futures and Foreign Exchange Data for
  markets in Australia, Asia, Canada, Europe, UK & USA -
www.premiumdata.net

AmiBrokerYahooGroup message#68314 “Yahoo EOD data not always correct ?” http://finance.groups.yahoo.com/group/amibroker/message/68314

 

Q6. I am downloading EOD data from Yahoo to an ASX 200 group on a daily basis and I am also using another source. I have noticed discrepancies in quotes between these sources. It appears that Yahoo is incorrect. Sometimes it can be 5 cents or more difference.

Differences may be because one source is adjusting for dividends while the other is not. AmiQuote/Yahoo is adjusting, by default, for splits and
dividends but you can turn adjustment off by editing Formats\aqh.format file (using Notepad) and removing ADJCLOSE field.

Tomasz Janeczko – amibroker.com

 

 Q5. For two days in a row Yahoo data for US stocks retrieved with Amiquote has been very spotty – data comes through for some symbols but not
others?

A5. This is a Yahoo problem. For example if a stock is missing the LOW price of the day AmiQuote, by default, does not import data with missing LOW-HIGH prices.
You can however turn this off – however LOW price will be still missing – in fact it will use lowest of open/close as a low).

To accept it, despite missing LOW, you can go to aqd.format file (inside Formats subdirectory) and remove $STRICT 1 line.

Tomasz Janeczko – AmiBroker.com

AmiBrokerYahooGroup  message #116918     “Yahoo Data Very Spotty Last Two Days”  http://finance.groups.yahoo.com/group/amibroker/message/116918

 

Q4. AmiQuote does not automatically list tickers for the exchanges supported by Yahoo!Finance. Are there any other Yahoo down-loaders, compatible with AmiBroker, that do?

A4. Yes, there are at least two that the author is aware of. They are both commercially available.

a) Bodhi Freeway (formerly Bodhi Gold) is a down-loader, with Yahoo capability, that allow users to select the tickers they want to download. It is distributed by JustData and is free when bundled with their data subscription packages).

http://www.justdata.com.au/Products/BodhiFreeway/bg_yahoo.htm

They produce data files in the Metastock format that can be dynamically referenced in AmiBroker using the Metastock plugin.

http://www.justdata.com.au/Training/Tutorials/Bodhi3/amibroker.htm

Sector classification is not available with Yahoo international data.

At the time of writing, Bodhi Freeway, to purchase as a stand-alone product is approximately twice the price of AmiQuote.

b) MLDownloader

http://www.trading-tools.com/

Download End-of-Day stock quotes on 50+ exchanges worldwide. Forex, commodity futures, market indices, mutual funds, money market funds, ETF, corporate bonds, and metals are also supported.
Stores data in MetaStock or in the form of ASCII files.
Supports all common MetaStock formats (Version 6.52, 8.0, 9.0 and 10.0 (6000 securities per directory)).
Stock splits are performed automatically.
Notification when dividends were distributed.
Automatic update of indices and maintenance of your symbol lists.
Download attempts can easily be repeated – Detects symbol changes.
Downloading data can be scheduled.
Automatically connecting, downloading and disconnecting for analogue modem users.
MLDownloader supports multiple stock quotes servers like Yahoo, MSN.com or Quote.com
Easy way to create symbol list from existing MetaStock database and continue updating historical EOD data.
Auto-re-download of missing data.
Option of Daily, Weekly and Monthly or Intraday data.

At the time of writing, MLD, is approximately twice the price of AmiQuote.

 

Q3. I am having problems with the US-Stocks database. There are 8630 symbols in the database. Yahoo has no historical data for over 300 of them. Amiquote puts ‘junk’ (html 404 error) in the data file for missing historical data.

A3. Authors comments – Not all download errors are caused by a lack of historical data. Sometimes this occurs because there have been symbol (name) changes or companies have restructured in some way (mergers, de-listings) and this is not reflected in the Yahoo database. Typically the majority of them are .OB (Bulletin Board) stock. This is not an error caused by AmiQuote, AmiBroker or the US _Stocks database.

Within AmiBroker they can be reported as NoQuotes and sent to a Watch list by using the Database Purify Tool (they have been sorted into a NoQuotes watch list by the developer of the US_Stocks template). They can be deleted from the database by deleting them from the Watch list, or alternatively users can check the Yahoo site to find the reason for the ’failure to download’ and make repairs to the database where necessary. 

Examples of how to use the Database Purification Tool, and other a tips and tricks to help with quality control of databases, have been documented in this section of the Users’ Knowledge Base.

 

Q2. I recently purchased AmiBroker & downloaded historical quotes from Yahoo. The volume display will not display values greater than 2,147,483,648.00 for indexes. All the recent volume values stop at this level. Is there a way to increase the display limit for volume?

A2. 2,147,483,648 is the upper limit for integers in AmiBroker. The volume, as it is imported from Yahoo, can be adjusted down using decimal factors.  Refer to the AmiBroker Knowledge Base article http://www.amibroker.com/kb/index.php?s=Yahoo by Tomasz Janeczko.  Also search the Yahoo message board archives, or the Help manual, for $volfactor to find additional information.

AmiBrokerYahooGroup  message #116336     “Volume Display Stops at 2,147,483,648.00 on Indexes”    http://finance.groups.yahoo.com/group/amibroker/message/116336

 

Q1. Is there a template for downloading Yahoo data, plus sectors, for the international  markets like Germany or India?

A1. Authors comments – sorry but it appears not (unless someone comes forward and tells us otherwise). Generally the products and services available for the American market are superior to the other Yahoo supported markets e.g. for the American market a database is maintained, including sector classification, for approximately 8000 stocks (the sector  information can be found on the Yahoo!Finance Profile page for each stock).

By comparison, the Company information, including the Profile, is not available for the leading German or Indian stocks, let alone a range representative of those market as a whole.

Germany – the top 30 company Adidas has no active links in the Company section.

YFAQ003

India – the top 100 company Bank of India has no active links in the Company section.

YFAQ004

Further to that, many of the Y!Finance international sites do not provide constituent lists for ‘local’ indices.

The only way around this appears to be:

  • - for people to dig in and find some local information to use,
  • - use the global indices constituents for international markets,
  • - ‘practice’ on the American markets then ’purchase’ local data, for comparison and trading, later,
  • - or, pay for the information.

Unfortunately the trend is towards ‘pay for play’ e.g. index lists for German stocks are downloadable from the Xetra site http://deutsche-boerse.com/dbag/dispatch/en/kir/gdb_navigation/technology/20_Applications/10_Trading/10_Xetra_Trading_Platform but only on a subscription basis.

Standard and Poors maintain global indices that cover 70-80% of the equity market for the major international regions. Constituent lists, including GIC’s classifications, are downloadable from their site but only approximately 30-40 German companies are deemed liquid enough to make that list.

http://www2.standardandpoors.com/portal/site/sp/en/us/page.home/home/1,1,1,1,0,0,0,0,0,0,0,0,0,0,0,0.html

Alternatively, “If you can’t beat them, join them”.  Consider trading the American market and enjoy the benefits of participation in a liquid market (commissions might be cheaper than trading in your own country).

AmiBrokerYahooGroup message #116257  ”Flags to get industry data from yahoo”  http://finance.groups.yahoo.com/group/amibroker/message/116257

1 Star2 Stars3 Stars4 Stars5 Stars (2 votes, average: 3.00 out of 5)
Loading ... Loading ...

Database Management FAQ’s

An unsorted, unqualified list of FAQ’s relating to anything and everything to do with database management.

The list is dynamic and subject to change without notice.

 

Q6 (a). What happens when my original (MS) data source adds, or deletes, a stock code or merges two codes together?
Do I have to take care of this myself?

A6.  Use the METASTOCK PLUGIN  http://www.amibroker.com/guide/h_extsources.html  to read the MS database directly (so you don’t need to re-import all the time).

Re: understanding categories.

Categories are NOT folders (like in Windows Explorer). The difference is fundamental. In the Explorer a file appears (usually) only once in the given
tree leaf. In the Workspace symbol tree a given symbol appears multiple times because it appears in every category leaf that it belongs to, even if this is the same symbol and it exists only as a single entity. One symbol belongs to MANY categories at the same time. For example AAPL (Apple Inc.) will belong to"Stocks" group category, "Nasdaq" market category, "Information" sector category, "Comp-Computer Mfg" industry category, and may also belong to several watch lists and favorites category ALL at the same time.That’s why ONE symbol will appear in many leaves of the symbol tree. Now if you delete the SYMBOL it will of course disappear from ALL categories because you have deleted the SYMBOL.

There are two types of categories:

1. with mutually exclusive membership: groups,markets, sectors/industries – it means that symbol MUST belong to ONE group, ONE market, and ONEsector/industry at a time – you can move the symbol from one group/market/sector/industry to another but you can not remove this assignment – create "Unassigned" group/market/sector/industry instead.
2. with free membership: watch lists/favorites/indexes – it means that a symbol may belong to ANY number (including zero) of watchlists (and to favorite/indexcategory too)In this case you can remove this assignment by Watch List -> RemoveWatch lists are covered in detail in the User’s Guide:
http://www.amibroker.com/guide/h_watchlist.html     (the same is available when you press F1 from AmiBroker)

Another thing: if you use "DELETE" button in Symbol->Categoriesit displays the warning:"This will delete selected symbols FROM THE DATABASE".Again: it will delete FROM THE DATABASE, not from the category.

Tomasz Janeczko – amibroker.com

Authors comments: As a rule of thumb, where data is accessed directly from an external source, WITHOUT a local (AmiBroker) database enabled, changes made to the database by the provider will be reflected at the AB user interface.

Where data is accessed from an external source, WITH a local (AmiBroker) database enabled, database changes will be at the discretion of the user. Depending on the access method, new symbols will be automatically added to the database e.g. when using the MS plug-in, and redundant symbols, or symbols that change, will remain in the database (with their data). The user has the option to merge changed symbols or delete old symbols (the symbol lists are stored in the local AmiBroker database, independently of the plug-in).

Where a local AmiBroker database is used as a standalone (data is updated via AmiQuote) the user has complete control over which symbols to add and delete etc (once again redundant symbols, and their data, will remain in the database unless manually deleted).

Q6 (b). How can I merge two symbols together after a name change?

A6 (b). Pick the symbol, that you want to retain in the database (master symbol), from the All symbols list and select Symbols >> Merge from the AB file menu. Click on the symbol, that you want to assimilate into the master, from the drop-down list in the Merge quotes window that opens, select from the options available (overwrite duplicate quotes, delete the slave symbol after assimilation or assign an alias name) and then click on OK.

 

Q5. Does anybody know how to check the number of stock symbols in a database?

A5. Hover the mouse pointer over the Data information section of the AmiBroker Status Bar (in the bottom right of the AB screen). A tool-tip will report on the database file name and path, the number of symbols in the database and the number of quotes in the current symbol.

 

Q4. Currently, my EOD data is supplied in Metastock format from an external supplier (JustData)…how can I setup to use it in AmiBroker?

It is stored in the following folders/directories on my pc:
C: – MyDatabases – Metastock –
ASX – 0
ASX – 1
ASX – A
ASX – B
ASX – C, etc, etc
LSE – 0
LSE – 1
LSE – A
LSE – B, etc, etc
World Indices

At the moment I trade multiple exchanges (ASX, LSE) as shown above.
To use this data within Amibroker, I am aware that I can use the Metastock Plug-in to recreate my databases in Amibroker, as per this link http://www.amibroker.com/guide/h_extsources.html

However, when it comes to creating the equivalent databases in Amibroker, how would others recommend I create them?

For example, would I need a different Amibroker Database for:
1. each current folder (as listed above),
2. 1 for all the ASX data, 1 for all the LSE data, etc
3. 1 database which captures ALL my data (LSE, ASX & Indices)?

If option 3 is suitable, then how would I distinguish in Amibroker which exchange a security belongs to (LSE or ASX)?

A4. Authors comments – AmiBroker doesn’t provide for easy ways to reference symbols that are in one database from within another, so with that restriction in mind it depends on how you want to use the data. If you are going to cross reference World indices to local or international markets, then you would be better off merging them into one database. This takes some work, so if you are not going to do inter-market analysis it is a matter of personal choice whether the benefits are worth the effort.

From an organizational point of view, international exchanges can be ‘managed’ at the Market, Group or Watch list level (refer to Users’ Knowledge Base >> Setup A Custom Database for assistance with organizing Categories within AmiBroker).

Adapting some of the basic procedures outlined in that tutorial you can either:

1) Create a new local AmiBroker database and use the Metastock Wizard to import the symbols into Watch lists, folder by folder, and then assign them to Categories from there (with the help of categoryAddSymbol).

2) Create a new database and access the symbols using the Metastock plug-in (this requires nominating the folders that will be referenced by selecting them in the File >> Database settings >> Configure window). From there you will need to import Watch lists, that correspond to the Metastock folders, and then assign them to Categories (using categoryAddSymbol once again).

Method 1 will create a static database, for historical back-testing only, and method 2 will create a database that dynamically updates to the Metastock database.

Both database types rely on organizing the symbols into Categories, after the initial setup, if you want to replicate Metastock folders.

 

Q3. I have a big database with intraday 1-sec quotes. How can I change the quotes interval in the database to shrink the database size ?

A3. You can use Exploration to generate any desired interval (set Periodicity in AA) and export to ASCII then import back to a new database.
http://www.amibroker.com/kb/2006/03/04/how-to-export-quotations-from-amibroker-to-csv-file/

Tomasz Janeczko – amibroker.com

AmiBrokerYahooGroup message #116071 "How can I convert AmiBroker tick database to AmiBroker 1-minute dat" http://finance.groups.yahoo.com/group/amibroker/message/116071

 

Q2. Is it possible to enter negative values via the Quote Editor?

A2. There is special flag $ALLOWNEG 1 that you can use to allow negative data during ASCII import.

Tomasz Janeczko – amibroker.com

AmiBrokerYahooGroup  #116322 "How to enter negative values in quote editor?"  http://finance.groups.yahoo.com/group/amibroker/message/116322

 

Q1. Is it possible to import tick data with the Wizard in AB?

A1. Tick mode _is_ supported by the wizard.
You just need to add $TICKMODE 1 in the "Additional Commands" field in the wizard and use "Close" for Last tick price and "Volume" for last tick size.

Tomasz Janeczko – amibroker.com

AmiBrokerYahooGroup message #116178  "Import TickData"  http://finance.groups.yahoo.com/group/amibroker/message/116178

1 Star2 Stars3 Stars4 Stars5 Stars (No Ratings Yet)
Loading ... Loading ...

Yahoo’s Data Providers (v2)

Yahoo sources data from a number of providers who are acknowledged at the bottom of the Y!Finance page: http://finance.yahoo.com/ 

Yahoo001

The partner page link in that section opens a list of delayed intra-day data suppliers: http://finance.yahoo.com/exchanges

As well as that, other providers are acknowledged in other places e.g. the S&P500 index is provided by Comstock and industry information, “on 7500 publicly traded U.S. companies”, is from Hemscott Americas etc.

The obvious point is that there are, at the least, two historical quotes databases, a fundamental database and several delayed intra-day data streams used, in different ways and in different places, at the site and there is no relationship between them in anyway. Some pages are a compendium, sourced from different providers, and it is difficult to be certain, in some cases, what is coming from where.

The main points to be aware of:

  • 1) The intra-day quotes, for listings on the major US exchanges, come directly from the exchanges.
  • 2) Intra-day quotes, for non-U.S. listings, are supplied by third party providers e.g. Comstock and Telekurs.
  • 3) Fundamental data, for American exchange listings, is supplied by Capital IQ, a division of Standard and Poor’s.

Note: The Capital IQ  database is the logical choice as the underlying database for two stock screeners, although this is not confirmed at the site:

Java screener – http://screener.finance.yahoo.com/newscreener.html  

HTML screener -  http://screener.finance.yahoo.com/stocks.html 

At the time of writing the HTML database contains 8467 listings, but sadly only allows users to view screened lists 20 at a time, while the newer Java screener, which has superior exporting abilities, only references NYSE, Nasdaq and Amex stock i.e. 4370 listings.

Bulletin Board (.OB) listings are included in the HTML screener database but Pink Sheets symbols (.PK) are not (to obtain all listings for the databases run a filter using price >= 0 as the criteria).

  • 4) Historical data, for the U.S. markets, is supplied by Commodity Service Inc (CSI) and is updated after hours (in a trial conducted by the author the update was logged at 8 PM EST (USA) i.e. four hours after the close).
  • 5) Historical and delayed intra-day data for the non-U.S. markets is supplied by Hemscott Americas (historical data is updated after hours).

Note: International listings require a suffix to identify the exchange.

For a list of suffixes for each exchange refer to: http://finance.yahoo.com/exchanges

  • 6) The stock listings in the CSI (historical) database are not the same as for the Capital IQ (fundamental) database, or any other database for that matter. 
  • 7) Only one of the ‘databases’ has a downloadable index, albeit a limited download, while access to other resources at the site is on a symbol by symbol basis.

There is, however, the ‘mother of all indexes’ at: http://biz.yahoo.com/i/ (it seems likely that this index is provided by CSI, based on it’s size and content e.g. it is mainly comprised of US and Canadian listings and the number of constituents also approximates the CSI universe.

The patchwork nature of the site sometimes produces surprising outcomes. 

One example:

Japanese financial services company Acom Co Ltd is an ADR traded on the OTC market. It is not a constituent of the HTML screened list, and it does not have a ‘Yahoo’ chart. A search for the company by name, using Symbol Lookup, sometimes returns a result ……. 

Yahoo002

……….and at other times it doesn’t (pity about that).

Yahoo004

 In spite of all that it does have quotation and historical data at the Yahoo site.

Note:  To go to Yahoo!Finance site and enter ACMUY.PK in the Enter Symbol(s) input box and click on GET QUOTES at any Y!Finance page.

Yahoo002

Clicking on the Historical Prices link, in the Quotes section of the left hand side bar, will open approximately two years of data for this symbol. (Yes – Amiquote will download the data).

This is a result that is typical for OTC ADR’s.

All in all a mixed bag from Yahoo!Finance.

  • Version 1 – Sept21/2007
  • Version 2 – Sept23/2007 – Corrected minor errors and added links.

Written using Windows XPHome and Internet Explorer v7.

1 Star2 Stars3 Stars4 Stars5 Stars (2 votes, average: 5.00 out of 5)
Loading ... Loading ...

Gap-Trading demo, Session Timing Example

This post shows how to add intraday session timing to the Real-Time Gap-Trading (RTGT) system developed in the previous post. However, before tackling session timing, there are a few things you should be aware about:

Time-Synchronization

In real-time trading there are many functions that are timed with reference to your system’s clock. It is therefore imperative that you always synchronize your computer clock with an Internet timeserver before each trading session.

Tools -> Preferences -> Intraday settings

Data Timestamps can be aligned to either the start or the end of the base period. The code developed here uses time of FIRST tick inside bar, i.e., the data timestamp returns the time at the start of the bar. This is when the first quote for the new period arrives and defines the Open for the new bar. Click Tools -> Preferences -> Intraday to set this option:

Backtesting and Bar-Replay

During backtesting the timing resolution will be equal to the periodicity set in AA Settings. If you are comparing Backtester signals with the signals displayed on your chart, you must make sure that the AA and Chart use the same periodicity.

During Bar Replay the timing resolution will be equal to the greater of the base interval of your database or the Step Interval selected in the Bar-Replay window.

During Backtesting and Bar-Replay, AmiBroker will refer to the timestamp to know how prices change over time. Hence, you will have no choice but to time events, such as session timing, with reference to the data timestamp.

Live Trading

When trading from an Indicator, the data timestamp is rounded to the selected chart-periodicity, i.e., if you display a 1-minute chart, the timing resolution will be one minute. This means you cannot implement delays based in seconds using the data timestamps of a minute database. This is why most functions in a real-time trading system use the computer clock as reference.

You can, with caution, use either the data timestamp or the system’s clock to control your session. However, since the data timestamp is dependent on the arrival of new quotes (ignoring data padding), using data timestamp could be unreliable. If you want higher timing resolution, you could create a 5-second database. However, this means working with slow backfills and slow AFL executions, due to lengthy data histories. To keep things simple, we will use a one-minute database.

Session Timing

When you are developing real-time trading systems, it is often handy, even essential sometimes, to develop several code versions. Typically, these might include:

1) A version for Backtesting and Bar-Reply. This version would use the TimeNum function for timing.
2) A Real-Time trading version. This version would use the system clock (Now()) for timing and would be highly optimized for AFL execution speed.
3) A DebugView version. This is a useful intermediate development stage that lets you run your system in real-time without any TWS interfacing; it logs trades to DebugView instead of sending them to the TWS.

ParamTime() input statements are used to set the start- and end-time of the trading session. The code below is only intended for preliminary system evaluation using the Backtester: hence, is uses time-numbers for session timing.

The StartOfSession and EndOfSession variables are triggers (they last only one bar). They are used to initialize processes at the start of the session and clean up processes at the end of the session. The InSessionTime variable is True during trading and is used to control processes that must be turned On/Off depending on whether you are trading or not. These processes will be covered in future posts.

A TimeFrame Parameter has been added to help visualize how the system works in different timeframes. To see the equity for different timeframes, just drag the slider to see the system response to any timeframe from 1 minute to 1 hour. Having added Session Timing you can now explore whether this system is more profitable during certain hours of the trading day. I suggest you perform an individual backtest on you favorite watchlist; you may be surprised to find that with some systems there is no need to trade all day. More…

For debugging purposes, you can turn On/Off a colored ribbon to display the Start- (Green), End- (Red) and In-Session (Yellow) variables.

For convenience the code below includes all previously developed parts. Just copy to an Indicator formula window, and click Apply.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
_SECTION_BEGIN("SESSION TIMING"); 
TimeFrame= Param("Chart Timeframe (min)",1,1,60,1)*60; 
TimeFrameSet(TimeFrame); 
TN = TimeNum(); 
ParamStartTime = ParamTime("Session Start","09:30:00"); 
ParamEndTime = ParamTime("Session End","15:59:00"); 
InSessionTime = TN >= ParamStartTime AND TN <= ParamEndTime; 
StartOfSession = InSessionTime > Ref(InSessionTime,-1); 
EndOfSession = InSessionTime < Ref(InSessionTime,-1); 
InsessionTime = InSessionTime OR EndOfSession;
_SECTION_END();
 
Buy = O < Ref(L,-1) AND InsessionTime; 
BuyPrice = O; 
Sell = L < Ref(L,-1) OR EndOfSession; 
SellPrice = C; 
Short = O > Ref(H,-1) AND InsessionTime; 
ShortPrice = O; 
Cover = H > Ref(H,-1) OR EndOfSession; 
CoverPrice = C; 
SetTradeDelays(0,0,0,0); 
SetOption("CommissionMode",3); 
SetOption("CommissionAmount", 0.005); 
E=Equity(1); 
TradeEquity = ValueWhen(Sell OR Cover,E); 
 
_SECTION_BEGIN("PLOTTING");
Plot(C,"",1,128); 
if( ParamToggle("Session Timing Ribbons","HIDE|SHOW",0) ) 
{ 
Plot( InSessionTime,"",7,styleArea|styleOwnScale|styleNoLabel,0,60); 
Plot( StartOfSession,"",5,styleArea|styleOwnScale|styleNoLabel,0,30); 
Plot( EndOfSession,"",4,styleArea|styleOwnScale|styleNoLabel,0,30); 
} 
OutOfSessionColor = ParamColor("Out of Session",colorDarkRed); 
Plot(NOT InSessionTime,"",OutOfSessionColor,styleArea|styleOwnScale|styleNoLabel,0,1); 
PlotShapes(IIf(Buy, shapeSmallUpTriangle, shapeNone),5,0,BuyPrice,0); 
PlotShapes(IIf(Sell, shapeHollowDownTriangle, shapeNone),4,0,SellPrice,0); 
PlotShapes(IIf(Cover, shapeHollowUpTriangle, shapeNone),5,0,CoverPrice,0); 
PlotShapes(IIf(Short, shapeSmallDownTriangle, shapeNone),4,0,ShortPrice,0); 
if(ParamToggle("Equity curve","HIDE|SHOW",0) ) 
{ 
Plot(TradeEquity,"",2,styleStaircase|styleOwnScale); 
}
_SECTION_END();

Edited by Al Venosa.


1 Star2 Stars3 Stars4 Stars5 Stars (6 votes, average: 4.17 out of 5)
Loading ... Loading ...

System Ideas on the Internet

You are invited to submit links to system ideas in comments to this post.

Gap Trading Strategies – Stockcharts
Intraday Moving Average Crossover with Position Sizing – NeoTicker
Volatility-Breakout-Systems – Traders Log
Ten day High/Low system – StockWeblog
Reversion Systems – SeekingAlpha
Systems Traders Club: Trader Club Bulletins.

1 Star2 Stars3 Stars4 Stars5 Stars (4 votes, average: 4.25 out of 5)
Loading ... Loading ...

Designing a Tradable System – Spikes

The phenomenon that is the basis of many trading systems is the observation and trading of an exceptional price movement followed by a pullback.

An extreme example of the pullback phenomenon would be a Spike as shown in the chart below. Because the price change is so extreme, the pullback or correction appears instantaneous. There is no clear market response, i.e., traders at large are not inclined to take the price change seriously.

The problem is that inadvertently you can easily write code that trades these spikes. Only when you start trading such a system will you discover that your orders are not filled because the volume just isn’t there. This is a common reason why backtested and real results may sometimes differ substantially. You may have designed a system that is completely rational, backtests perfectly, and stands up to the most detailed technical scrutiny, only to find out that in real trading it fails completely.

You might think that by increasing the timeframe, for example to 15-minute or even daily, you can minimize this problem. However, while doing this may make the spikes less prominent, the tradability will not improve. Consider the spike in the 15-minute chart below:

Adding a few percent bands makes this look like a real trading opportunity. It looks so easy! However, the Low of the bar is still created by a single trade and the chance to get your order filled would still be minimal. Designing trading systems around minimal-volume price changes is one of the easiest traps for a real-time system developer to fall into. When designing an intraday trading system you should design your code to minimize the divergence of the backtester with respect to real-trading results. You can do this by working in the smallest time frame possible. Even when trading at hourly intervals you should write your code in the minute (or even Tick) timeframe.

There are a number of ways in which to do this. Take a look at the 10:30 AM spike in the 15-minute chart below and consider how you would determine its tradability:

The fact is that there is no way to tell whether the 10:30 AM High is tradable. However, expanding the chart to the 1-minute timeframe, as shown below, lets you clearly see a gradual reversal pattern. This means your order could probably have been filled somewhere near the top of the 15-minute spike shown earlier.

Running your Backtester in the 1-minute timeframe and looking for one-bar confirmations may drop your backtester performance, but your results would have been closer to that which can be obtained in real trading. In this case you would have separate Backtester and Trading code versions for your system; the Backtester code would include signal confirmation while your Trading code would not.

Edited by Al Venosa.

1 Star2 Stars3 Stars4 Stars5 Stars (7 votes, average: 4.29 out of 5)
Loading ... Loading ...

System-Design Pitfalls

When you are designing a real-time trading system, many things can go wrong. This post is intended to alert you to some of the potential pitfalls. However, that is all it can do. Only experience can teach you how to prevent them. Be aware that even the most experienced designers will make some of these mistakes repeatedly.

Since documenting all potential pitfalls with coding examples would consume too much time and space, they are, for now, only briefly commented on. Most of them will trigger a user response of “Oh yeah, that happened to me!”. If you need a more detailed explanation you can post questions in a comment to this post

No rules exist to prove that a trading system is free from coding or logical errors. However, two indicators are fairly reliable in suggesting you may have a problem:

1) Your profits are simply too good to be true. In this case you have no choice but to work through the code line by line, trying to find lines of code that look into the future. If that doesn’t reveal any errors, then you would have to inspect the plotted signals and trade list trade by trade.
2) Your system is very profitable trading Long but not Short, or Short buy not Long. When this happens, you may have an error in either the Long or Short parts of your code, and comparing the two sections will often reveal the problem (this only works for reversal systems). However, it could also be that your code is correct but that your trading principle is overly trend sensitive. This would almost certainly get you in trouble when the trend reverses. In this case no other cure exists than to re-think the basic system.

When designing high-frequency trading systems, i.e., those whose trade durations are in minutes, everything changes, and many traditional procedures fall apart. Internet delays, data delays, bad data (spikes), temporary system freezes (Windows sometimes has a mind of its own!), lagging status reports, TWS problems, etc., all become critical issues that will prevent you from obtaining a close match with the Backtester.

Many of these problems will only surface when you start trading real money. Hence, the final stages of developing a trading system should always involve trading real money. Here is where the Interactive Brokers account simulator (paper-trading account) may be an indispensable tool since you can test your system in real time without committing real dollars. But, since the market does not see your trades, even paper-trading results will differ from trading real money. In general, the faster you trade, the greater your real-trading results will deviate from your backtest results. You should also be aware that commissions play a much greater role on performance of high-frequency trading systems because trade profits are smaller.

No matter how you go about it, troubleshooting a complex trading system will almost always be a tedious and boring job that could keep you busy for several days or weeks. If you find that certain problems continue to resurface, they are likely related to your personal development style, and you may be able to write some code that checks for these specific problems. See the Debugging category for some ideas.

The list below, which is not exhaustive, is presented to caution you that many areas can lead to problems. Some are obvious, while others may be expanded on as needed and time allows.

- High/Low precedence (contrary to EOD where the Backtester is unable to determine which came first, the entry/exit or the high/low, in realtime there can be no ambiguity in price precedence).
- Data Delays (real-time data may be delayed for various reasons and time periods (Internet delays, lack of quotes, packets vs. ticks, etc.).
- Low Liquidity (there may be no-volume trading periods).
- Data Holes (bars with no trades).
- Data Spikes (high spikes without volume may trigger trades).
- Data Padding (a bar without data may be padded).
- Premature Padding (the last bar may be a padded bar).
- Data Accuracy (prices you receive aren’t always accurate).
- Random Slippage (you will rarely get the expected price).
- Breakout slippage (you will rarely get the Breakout price of your system).
- Survivorship Bias (companies that didn’t do well and stopped trading won’t be in your database, i.e., you are working above average stocks).
- Lucky Trades (a series of lucky trades may look like good performance).
- Parameter Over-Optimizing (optimized parameters are rarely stable over time).
- Design Over-Optimizing (frequent testing is like running an optimization and may be leading to false conclusions).
- Out–of-Bound Prices (with PriceBoundChecking turned ON, AmiBroker forces the trade price within the High-Low range, this may hide pricing errors).
- Price Rounding (prices may be rounded or truncated by the broker).
- Wrong Use of >= and <= (when using both <= and >= in the same statement, only the first equal condition will ever be seen).
- Comparing Floating Point Numbers (calculated values can have many decimal places, either round values or use the AlmostEqual()).
- Chart Justification (make sure you are looking at the Last bar!).
- System Mortality (no system will work forever).
- Sharing Trading Systems (sharing systems with other traders may result in over-trading a system).
- Being Duped by a Trend (a rallying ticker may make your system look like the HG (holy grail).
- Tricking AmiBroker (AmiBroker has its limits; it is possible to write esoteric code that will produce wrong results).
- Order Visibility (placing your order for every trader to see may influence the orders they place).
- Making the Market (extreme example: if you place a MKT order during a no-trading period you will change the chart).
- Window/Pane Execution Order (when passing variables between panes or windows do not assume that they execute in a fixed order, more).
- Trading at the Open (order execution at the start/end of day is different from midday because of volatility and data delays).
- IB Data Snap Shots (snapshots are only representative of prices traded).
- Trade Delays (make sure you understand your trade delays when backtesting).
- EOD and Intraday Gaps (There is no time interval in RT gaps).
- Time Zones (make sure your computer and database timezones are properly set).
- Very Short Time-Frames (prices jump and are less contiguous).
- Setting LMT Prices (consider rounding for faster order executions).
- 24-Hour vs. RTH (Regular Trading Hour) Backtesting (extended hours can rarely be traded like RTH due to huge bid/ask spreads and low volume).
- Static Variables Naming (use unique names for your static variables).
- Incorrect Computer Time (computer time offset from market time can cause real problems).
- Look-Ahead Problems (not all look-ahead coding problems are obvious).
- Buy/Sell Precedence in a Loop (be aware that AB and custom AFL loops enforce a Buy/Sell priority).
- RT Candle Discrepancies (RT Candles may be different from later backfills, especially in the opening print).
- Bars Loaded (consider bars-loaded with respect to execution speed and loops).
- Signal lifetime (signal strength quickly decays over bars in high frequency trading).
- SameBarExits (Sell signals may act as a qualifier for Buy signals).
- Designing systems based on High and Low triggers (these may fill in the Backtester but not in real trading). more…
- Using the wrong CommissionMode and/or CommissionAmount can make any system look good, or bad…
- Using zero TradeDelays is OK if you code the delays in your system’s code, else you may be looking into the future.

Edited by Al Venosa

1 Star2 Stars3 Stars4 Stars5 Stars (5 votes, average: 4.40 out of 5)
Loading ... Loading ...
« Previous PageNext Page »