Users' Knowledge Base

DRAFT ONLY - SUBJECT TO CHANGE

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

//P_InverseRTM  
//A SYSTEM BASED ON THE INVERSE OF A REVERSION TO MEAN  
//The one bar ROC of three moving averages is used to test for a bull trend 
 
ShortTermMA = ROC(MA(C,30),1);
MediumTermMA = ROC(MA(C,50),1);
LongTermMA = ROC(MA(C,100),1);  
 
Condition1 = ShortTermMA > 0 AND MediumTermMA > 0 AND LongTermMA > 0;  
 
Buy = Condition1;
Sell = 0;  
 
shape = Buy * shapeUpArrow + Sell * shapeDownArrow;
PlotShapes( shape, IIf( Buy, colorGreen, colorRed ), 0, IIf( Buy, Low, High ) );

This plots an up arrow for every bar where the 3 MA’s are moving up.

Another way to do the same thing (the bull condition is meet a little earlier).

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

//P_InverseRTM  
//A SYSTEM BASED ON THE INVERSE OF A REVERSION TO MEAN  
//The one bar ROC of three moving averages is used to test for a bull trend  
 
//ShortTermMA = ROC(MA(C,30),1);
//MediumTermMA = ROC(MA(C,50),1);
//LongTermMA = ROC(MA(C,100),1);  
 
ShortTermMA = MA(C,30);
MediumTermMA = MA(C,50);
LongTermMA = MA(C,100);  
 
Condition1 = ShortTermMA > MediumTermMA AND MediumTermMA > LongTermMA;  
 
Buy = Condition1;
Sell = 0;  
 
shape = Buy * shapeUpArrow + Sell * shapeDownArrow;
PlotShapes( shape, IIf( Buy, colorGreen, colorRed ), 0, IIf( Buy, Low, High ) );

Add a condition to screen for reversal bars (using the second MA method).

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

//P_InverseRTM  
//A SYSTEM BASED ON THE INVERSE OF A REVERSION TO MEAN  
//The one bar ROC of three moving averages is used to test for a bull trend  
 
//ShortTermMAChange = ROC(MA(C,30),1);
//MediumTermMAChange = ROC(MA(C,50),1);
//LongTermMAChange = ROC(MA(C,100),1);  
 
ShortTermMA = MA(C,30);
MediumTermMA = MA(C,50);
LongTermMA = MA(C,100);  
 
ReverseTrend = Ref(L,-1) > Ref(ShortTermMA,-1) AND L < ShortTermMA;  
 
Condition1 = ShortTermMA > MediumTermMA AND MediumTermMA > LongTermMA;
Condition2 = ReverseTrend == 1;  
 
Buy = Condition2;// AND Condition2;
Sell = 0;  
 
shape = Buy * shapeUpArrow + Sell * shapeDownArrow;
PlotShapes( shape, IIf( Buy, colorGreen, colorRed ), 0, IIf( Buy, Low, High ) );

The first buy signal (marked by the green up arrow) meets the system criteria but the second violates the 50 MA rule and the third breaks the MA bias rule.

Changing back to the first MA method:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

//P_InverseRTM  
//A SYSTEM BASED ON THE INVERSE OF A REVERSION TO MEAN  
//The one bar ROC of three moving averages is used to test for a bull trend  
 
ShortTermMAChange = ROC(MA(C,30),1);
MediumTermMAChange = ROC(MA(C,50),1);
LongTermMAChange = ROC(MA(C,100),1);  
 
ShortTermMA = MA(C,30);
MediumTermMA = MA(C,50);
LongTermMA = MA(C,100);  
 
ReverseTrend = Ref(L,-1) > Ref(ShortTermMA,-1) AND L < ShortTermMA;  
 
//Condition1 = ShortTermMA > MediumTermMA AND MediumTermMA > LongTermMA;
Condition1 = ShortTermMAChange > 0 AND MediumTermMAChange > 0 AND LongTermMAChange > 0;  
Condition2 = ReverseTrend;  
 
Buy = Condition1 AND Condition2;
Sell = 0;  
 
shape = Buy * shapeUpArrow + Sell * shapeDownArrow;
PlotShapes( shape, IIf( Buy, colorGreen, colorRed ), 0, IIf( Buy, Low, High ) );

This restores the MA bias as per the rules.

Adding a third condition excludes trend reversals that cross the 50 MA (on the close).

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

//P_InverseRTM  
//A SYSTEM BASED ON THE INVERSE OF A REVERSION TO MEAN  
//The one bar ROC of three moving averages is used to test for a bull trend  
 
ShortTermMAChange = ROC(MA(C,30),1);
MediumTermMAChange = ROC(MA(C,50),1);
LongTermMAChange = ROC(MA(C,100),1);  
 
ShortTermMA = MA(C,30);
MediumTermMA = MA(C,50);
LongTermMA = MA(C,100);  
 
ReverseTrend = Ref(L,-1) > Ref(ShortTermMA,-1) AND L < ShortTermMA;  
 
//Condition1 = ShortTermMA > MediumTermMA AND MediumTermMA > LongTermMA;
Condition1 = ShortTermMAChange > 0 AND MediumTermMAChange > 0 AND LongTermMAChange > 0;
Condition2 = ReverseTrend;
Condition3 = C > MediumTermMA;  
 
Buy = Condition1 AND Condition2 AND Condition3;
Sell = 0;  
 
shape = Buy * shapeUpArrow + Sell * shapeDownArrow;
PlotShapes( shape, IIf( Buy, colorGreen, colorRed ), 0, IIf( Buy, Low, High ) );

That ’screens’ the charts for bars that meet the trade conditions (raw signal) - to turn it into a system it needs specific entry and exit rules (stops) - followed by testing and the addition of money management rules.

Moving up the chart, and looking at a longer period, the ’system’ produced two wins and one loss in 9 months of ‘trading’ one stock from the S&P500 (that isn’t a typical outcome and the author picked a favorable part of the chart.

Filed by brian_z at 11:54 pm under Ideas (Experimental)
No Comments

(4 votes, average: 4.75 out of 5)

 Loading ...

Backfilled vs Real-Time Data issues

If you are not sure what HFAT (High Frequency Automated Trading) is all about, please Google the topic. This post highlights some of the problems you may encounter when venturing into HFAT of stocks. The views expressed here are based on personal experiences and/or may be anecdotal; not everything that happens in real-time trading is easy to explain. If you have technical insight and see inaccuracies, please comment for the benefit of future readers.

Designing and implementing high frequency trading systems is, from a trader’s viewpoint, probably the ultimate experience. To see and hear trades executed every few seconds and see the profits rolling in should give any trader an unprecedented high.

The catch is that to design an HFAT system that works with real money is very different from designing one using local data. The smaller the time frame the greater the impact of small data discrepancies. In sub-minute time frames, your HFAT system may perform very differently with local data than with real-time, raw market data.

A typical problem is that real-time live market data are delayed by up to several hundred milliseconds and that quotes may arrive out of sequence. What you see on your charts may be several quotes after the trade took place. This flawed data is what your trading system is trading and must be designed to work with. The charts you see in AmiBroker are mostly backfilled and/or updated after trading hours. At the end of a trading day, you will have data in your database that have a mixture of backfilled data (time and data errors have been corrected) and raw data (flawed) that were collected during the current day’s trading session. You may also have several lengthy data gaps that were introduced when you shut down the system and/or you lost your data feed.

While the procedure may vary for different data providers, quotes that are received in real time will lag in time. Since bar periods during live data collection are based on your computer clock, quotes may end up in the next bar due to their delayed arrival. The data used to backfill your database come from a different data server and will be time stamped. This allows AmiBroker to correct the position of quotes that were received out of sequence. This process removes the real time delays that were present when the data was received.

Since there are no delays with backfilled data, your backfilled data look ahead by several hundred milliseconds with respect to the data you will eventually be trading.

It is not unusual to develop a system with 5-second backfilled data (where all bad ticks and time-stamp errors have been corrected by the data provider) and obtain Holy Grail performance only to find out that when traded with real-time streaming data (where the data is delayed, contains bad ticks and time-stamp errors), the system is a total failure. The following charts illustrate this problem. The data to the left of the red line is backfilled and the data to the right of the Red line is data collected in real time. White is the equity.

You will not be able to visually judge whether data are backfilled or raw. The differences will only show up by running a trading system on the data; your trading system may be the only way to distinguish between backfilled and raw data. The chart below shows a close up of the data change.

Backfilling the above database and performing another Backtest over the same period produces a very different equity:

There is no guarantee that a system developed on one type of data will perform equally well with the other. When you first encounter a major equity drawdown, you may assume that this was just “a bad day”; after all, all trading systems have them. You may have developed and backtested your system over thousands of trades, covering a period of six months or more. You have been a good student and have used all the recommended methods to validate your trading system. You have tested in- and out-of-sample, applied intelligent optimizations, used Walk-Forward testing, performed Monte-Carlo analysis, and the list goes on. After being so thorough, how could you go wrong? You are ready to trade real money tomorrow and make your first 50% in one day!

The point is that all this effort is wasted time if the data used during development aren’t 100% identical to what you will be trading with.

The best way to develop an HFAT system is to use real live market data. The earlier you change from local or edemo data to real data, the more time you will save, and the more disappointments you will be spared. An HFAT system can never be completed off line, with a local database, or with simulated edemo Data. Its design must always include a significant paper trading and real-money phase.

Another problem when trading your IB paper-trading (simulated) account is that the user does not know the rules Interactive Brokers uses to decide whether an order should be executed or not. These execution criteria may change without warning. This imposes an artificial order to your executions that is unreal; the simulated market conditions will be different from those encountered in real trading. You may well develop a trading system that exploits IB’s way of processing to give you unreal performance, but such a system would fail in real trading.

Also, your paper-trades are not seen by, and cannot influence, the market. When trading real money your orders could be setting a new High or a Low, or if you are trading large sums, you could draw the price up or down. This means that even if your system performs extremely well in simulated trading, this is no guarantee that your system will perform well trading real money.

Using your simulated account to validate your system should never be your final validation before trading for profits; you should always include a real-money evaluation phase in your development plan. Your first real trades should never be to make money; they should be planned to validate your system under varied conditions.

Market behavior is very complex; be prepared for the unexpected and never skip a development step because something works extremely well. For example you might be testing your system using your IB simulated paper-trading account and see your profits skyrocket too fast to follow, perhaps having 90% winners and RARs that are out of this world. When this happens, it is extremely exciting and fun to watch; it is a rare experience that must be appreciated. It suggests that Holy-Grails are possible. But are they? Such favorable trading conditions may last for a few hundred trades, a few hours, or perhaps a few days. This can happen when technical conditions and market behavior are all just perfect for your system. Some unknown factor just made everything work perfectly. When you experience this, you’ll be analyzing your charts, trading log, execution report, etc. for weeks to follow. The fact is that it may never happen again, and you may never know what really happened.

Order and Position Status

IB Position size reporting may be erratic, is always delayed, and may include transient information. If you are trading fast and you use the IB Position Size to determine your next action, this will be a problem. This is especially the case with reversal systems where Covers may be processed before the Buys, and there may be many partial fills. For example, if you are reversing 100 shares, going alternatively Long and Short, you might read position sizes of 0, 100, 200, and even 300 shares. Do not base your system’s action solely on a single reading of the position size; your protective mechanisms will shut down your system many times a day. If a position is not what it is supposed to be on 5 consecutive queries (at quote interval), you may want to close all positions, suspend operation and continue later, or shut down the system and retry later.

Reporting of order status appears more reliable and stable. Usually it seems unnecessary to repeat Order Status queries.

IB Snapshots

Not addressed in this post is the matter of Snapshots however it is extremely important for real-time traders to understand how IB compresses and transmits its data. This topic has been discussed on several forums, for more information on IB data in general please read the following threads:

AmiBroker user group: Interactive Brokers Plug-in dropping volume data
IB’s Discussion Board: Globex Ticks snapshot or reality?
AmiBroker User Group: AB Tick Bar Analysis

The IB Maximum Message Rate

IB has a limit to the maximum rate of messages (order related) you can transmit per second. The rate of queries is not limited. The current limit is 50 messages per second. If you exceed this rate, IB will produce an error code, and if you continue to exceed the message rate, IB will suspend your connection. This, of course, should be prevented at all cost. The message rates are documented here. How to introduce real-time delays measuring in milliseconds is documented in the post on High Precision Interval and Delay Timing.

Internet Delays

Order and Position status is subject to a 50-400 milliseconds Internet delay. This delay will vary with your location and type of Internet connection to IB. You can test this delay by pinging the IB server. To do this type ping gw1.ibllc.com on command in the Start->Run windows (for Windows XP), and click the run button. A window, as shown below, will appear and show you the delays for three consecutive queries (pings) to IB:

If you encounter excessive delays or cannot connect at all, you can get more details about how your connection is routed by running tracert gw1.ibllc.com in the same manner. You may want to browse the Technical FAQ at IB for related items.

Edited by Al Venosa.

Filed by Herman at 8:55 am under Real-Time System Design
12 Comments

(5 votes, average: 5 out of 5)

 Loading ...

An unsorted, unqualified list of FAQ’s relating to anything and everything to do with AmiQuote.

The list is dynamic and subject to change without notice.

Q1. Hi, my Amiquote crashes on Windows Vista. Sometimes after Amiquote downloads some tickers data it crashes with following message: ”Quote MFC Application has stopped working”. Any suggestions?

A1. We actually tested it and AmiQuote only crashes on Vista when you are using cracked AmiBroker.
If you are using TRIAL, legal AmiBroker or REGISTERED, legal AmiBroker it works fine.

Tomasz Janeczko
amibroker.com

Filed by brian_z at 12:11 am under Data
No Comments

(3 votes, average: 3.67 out of 5)

 Loading ...

Q. Does anyone know how to buy tomorrow and sell the next day across your entire portfolio?

Basically, I want to:
1) Get the buy signal at the end of Day 1
2) Buy the stock on the opening of Day 2
3) Sell the stock on the opening of Day 3

A. There are multiple ways to do this in Amibroker. Below I give an example (using HHV to signal new 220 bar highs) of how this can be built into a portfolio type system.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32

//Code by Ed Pottasch 
// if you want to buy and sell, with delay, inside a portfolio type system 
 
SetBarsRequired(10000,10000); 
SetOption("MaxOpenPositions", 10 ); 
SetOption("UsePrevBarEquityForPosSizing",True); 
SetOption("PriceBoundChecking", False); 
PositionSize = -10; 
SetTradeDelays(0,0,0,0); 
 
// delay of exit with respect to entry in bars
sellDelay = Optimize("sellDelay", 1, 1, 50, 1); 
 
// buy signal when H exceeds H of past 220 bars
Buy = H > Ref(HHV(H,220),-1); 
 
// entry at the open of the bar following the signal bar
Buy = Ref(Buy,-1); BuyPrice = O; 
 
// remove excessive signals from initial signal
Buy = ExRemSpan(Buy, sellDelay); 
 
// sell sellDelay bars after entry at the open
Sell = BarsSince(Buy) == sellDelay; SellPrice = O; 
 
SetChartOptions(0, chartShowDates); 
GraphXSpace = 5; 
Plot(C,"C",1,64); 
PlotShapes(IIf(Buy,shapeUpArrow,0),colorWhite, layer = 0, yposition = BuyPrice, offset = 0 ); 
PlotShapes(IIf(Sell,shapeDownArrow,0),colorYellow, layer = 0, yposition = SellPrice, offset = 0 ); 
 
PositionScore = 1/Ref(RSI(sellDelay),-1);

Comments and code by Edward Pottasch.

AmiBrokerYahooGroup message# “How to Buy Tomorrow and Sell the Next Day Using Your Whole Portfolio” http://finance.groups.yahoo.com/group/amibroker/message/116945

ATTACHED FILE:

b_higherhighs.afl

Filed by brian_z at 10:38 pm under Backtester
No Comments

(No Ratings Yet)

 Loading ...

A bar range, for processing, can be stipulated at the formula level.

The following code plots the Close, over a four month period, using the DateNum function (BarIndex can also be used in a similar way).

1
2
3
4
5
6
7
8
9
10

//P_SelectedRange 
 
///////////Plot a selected date range in the middle of chart using BarIndex or DateNum 
//Plot(IIf(BarIndex() >= 978 AND BarIndex() <= 988,C,Null),"",1,1);
Plot(IIf(DateNum() >= 1050101 AND DateNum() <= 1050430,C,Null),"",1,1); 
 
///////////Plot a selected range at the end of a chart 
LastBarIndex = LastValue(BarIndex()); 
//Plot(IIf(BarIndex() >= LastBarIndex - 10 AND BarIndex() <= LastBarIndex(),C,1)," ",8,1);
//Plot(IIf(BarIndex() >= LastBarIndex - 10,C,1)," ",8,1);

If the last line of code is uncommented the Close,for the last 10 bars only, will be plotted.

The formula can be adapted for use in other modes (filter, scan etc).

ATTACHED FILE:

p_selectedrange.afl

Filed by brian_z at 9:43 pm under AFL - The Basics
No Comments

(2 votes, average: 4 out of 5)

 Loading ...

Scan can be used, in combination with AddToComposite (ATC), to create customized broad market indicators.

1) Run the following code as a Scan against the constituents of an index (for this example an EOD  database comprising only constituents of the the Dow Jones Index “^DJI”  has been used).

Note: Since there are no other symbols in the database All symbols were selected in Automatic Analysis  and the filter was not used i.e. instructions 1-3 (inclusive) in the formula were bypassed.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

// Breadth of Stocks
// Develops composites that can be plotted or manipulated as you wish
// How to Run
// 1) Select Group - Stocks or Market -NYSE,Nasdq, AMEX
// 2) Select an issue with a long history in the current ticker window
// 3) Set APPLY TO to use filter Market - Stocks
// 4) Set RANGE to one bar... 1 n last quotations
// 5) Press SCAN
// adapted from code by Dick Areehoi 
 
//===================52 Weeks New High - Lows  ================================== 
 
Buy = Sell = 0;//allows Scan to run 
 
//Find New Highs over the past 52 weeks. 
 
AddToComposite(IIf(C > HHV(Ref(C, -1), 250), 1,0), "~StocksNewHi","X"); 
//Same for lows next. 
AddToComposite(IIf(C < LLV(Ref(C, -1), 250), 1,0), "~StocksNewLo","X"); 
 
//Assign (define) variables for plots 
NewHigh = Foreign("~StocksNewHi","X"); 
NewLow = Foreign("~StocksNewLo","X"); 
DIFF_H_L = NewHigh-NewLow; 
 
Plot(NewHigh,"NewHigh",colorGreen,styleThick|styleHistogram); 
Plot(NewLow*-1,"NewLow",colorRed,styleThick|styleHistogram); 
Plot(0,"",1); 
//Plot(Diff_H_L,"Diff_H_L",colorBlack,styleLine)

Nothing will be reported in the AA Results pane by the scan.

However two pseudo tickers will have been added to the All symbols Tree.

2) Insert S_52WeekHiLo, as an indicator, then modify S_52WeekHiLo to plot only the difference (comment out the high and low plots and un-comment the difference plot) and save it as S_52weekHiLoDiff.

3) Insert S_52weekHiLoDiff as an indicator (it will automatically open in a new pane).

The Hi/Lo plots are breadth indicators for the broad market (^DJI in this example) and might produce signals that are suitable for trading the index. Different periods for the indicator, and/or offsetting the Hi/Lo periods, might produce some interesting variations on the theme. Another strategy might be to use the broad market indicator for timing entries to single issues,  e.g. a ’bull’ period in the breadth indicator coincides with a ’bull’ period for AIG (a constituent stock of the ^DJI).

CAUTION: THIS IS A VERY FAVORABLE SECTION OF THE CHART FOR THE BREADTH INDICATOR/STOCK PERFORMANCE COMBINATION. IT DOESN’T AUTOMATICALLY FOLLOW THAT IT WILL FORM THE BASIS OF A GOOD TRADING SYSTEM. “ONE SWALLOW A SUMMER DOES NOT MAKE”.

Note: In the AIG screenshot, although the upper chart shows prices for the current symbol the lower charts still plotting the global or breadth indicators, since they are referencing the foreign composite symbols (the current symbol shown in the indicator titles is not relevant). Also, the chart has been changed to Weekly view (in Weekly view the breadth indicators report the number of stocks making new highs or lows on the last day of the week).

The current chart can be ‘marked’ to provide a visual cue as to when that particular issue makes a new high.

To ‘mark’ charts when the price moves into new territory:

1) Save the following formula as P_NewHiLo.

2) Manually enter the number of bars required for the look-back period.

Note: The Hi and Lo plots can use different periods. For this example half the Hi look-back period was used for the Lo.

3) Left click on the formula, in the Chart Tree, and, holding the mouse button down, drag and drop it on the price chart, as an overlay.

1
2
3
4
5

//P_NewHiLo
//Overlay on a price chart. 
 
PlotShapes( IIf(H > HHV(Ref(H, -1), 250) AND BarIndex() > 250, shapeSmallCircle,0) , colorGreen,0,High,10); 
PlotShapes( IIf(L < LLV(Ref(L, -1), 125) AND BarIndex() > 125, shapeSmallCircle,0) , colorRed,0,Low,-10);

Explorer can be used to find which symbols are making new highs or lows, on any given day, or over any date range e.g. the solitary issue that made a new low on 11/10/2005 can be isolated.

To report on issues that a making a new High or Low:

1) Run an exploration using the following formula.

1
2
3
4
5
6
7
8
9

//X_NewHiLo 
 
//NewHi = H > HHV(Ref(H, -1),250);
NewLo = L < LLV(Ref(L, -1),125); 
 
Filter = NewLo; 
 
//AddColumn(NewHi, "NewHi");
AddColumn(NewLo, "NewLo");

2) Select All symbols, set the date range to the day, inclusive, and click on Explore.

GM is reported as the stock that is at a new 125 day low.

Clicking on the “GM” row, in the results pane, will bring up the chart for a visual confirmation.

Expanding the date range can return a list of the stocks that correspond to ‘bullish’ and ‘bearish’ periods for the NewHiLo Indicator e.g. the following Exploration reports the list of stocks that contributed to the ‘bullish’ NewHiLo indicator in the last quarter of 2005 (approximately). 

Clicking on the Date/Time column header will sort the NewHi’s sequentially, by date, to show which companies lead the ^ DJI ‘bull run’, to a new annual high on the 1/09/2006  the ^DJI.

The ‘bulls’ can also be compared to the ‘bears’ from the preceding ‘bearish’ period.

To report on stocks making new lows:

1) Set the date range to match the period marked in red. 

2) Change the formula to suit (by un-commenting the NewLo’s code) and then run a fresh exploration.

It can be interesting to track the performance of the ‘bulls’ or the ‘bears’, from that period, over the next few months.

To compare relative performance of the October 2005 ‘bears’:

1) Set ^DJI as the current chart.

2) Insert the Relative Performance formula, from Charts >> Basic Charts as a chart (a Properties of: Relative Performance window will open).

3) Position the cursor in the Tickers input box and enter the symbols, from the AA report, with a comma between each.

4) Click on OK and the performance of each, relative to the current ticker, will be plotted. 

5) Position the selector line on the reference date (in this case that is 11/10/2005) and scroll back until the reference day is the first bar in the chart (the relative performance of each symbol will be automatically recalculated to start from the first bar in the chart).

Interestingly, only 3/13 of the ‘bears’ under-perform, relative to the ^DJI, over the next (approx) six months.

CAUTION: RESULTS WILL VARY WITH THE TIME PERIOD SELECTED.

The exclusivity of a new high or low can also be tested.

A variation of the ‘bull side’ code used in P_NewHiNewLo adds a vertical line to the plot if a new 250 bar high is the first for a selected time period.

1
2
3
4
5
6
7
8
9
10

//P_HigherHi v2 
 
PlotShapes( IIf(H > HHV(Ref(H, -1), 250) AND BarIndex() > 250, shapeSmallCircle,0) , colorGreen,0,High,10); 
 
//P_ExclusiveHigherHi 
 
NewHi = H > HHV(Ref(H, -1), 250) AND BarIndex() > 250; 
ExclusivePeriods = Param("ExclusivePeriods",1,1,250,1);
ExclusiveNewHi = ValueWhen(NewHi ==1, BarIndex(),1) - ValueWhen(NewHi == 1, BarIndex(),2) >= ExclusivePeriods AND NewHi; 
Plot(ExclusiveNewHi,"ExclusiveNewHigh",2,2|styleOwnScale);

In this example the formula has ‘marked’ a new 250 bar high, that is the first one for 182 bars, with a red line.

Note: It would be rare to find a 250 bar new high that is the first occurence in exactly 182 bars so the formula reports, in this example, on any new high that is exclusive for at least 182 bars.

The exclusivity period for any new high can be found by changing the parameter setting.

To change the ExclusivePeriods parameter setting:

1) Right click inside the chart (the Properties window will open).

2) Left click in the space to the right of the default setting (the parameter slider control will appear).

3) Left click and hold the mouse button down while dragging the slider arrow left or right, to change the setting.

Note: The exact number of bars, since the previous 250 bar new high, can be found for any bar by dragging the slider until the plotted ‘marker’ line disappears from the chart and then moving the setting back one period.

The veracity of the code can be cross-checked by plotting barindex() in a separate pane, positioning the selector line on the relevant highs and calculating the difference between the reported barindex() numbers.

1

Plot(BarIndex(),_DEFAULT_NAME(),1,1);

For the complete code, that contains considerably more than the vignette provided here, download the attached S_52HiLoMaster file, or refer to the AmiBrokerYahooGroup message #116405     “Getting New High-New Low data into AB using AmiQuote? ” http://finance.groups.yahoo.com/group/amibroker/message/116405

For additional discussion on identifying exclusive new highs, within a defined period, refer to the AmiBrokerYahooGroup message #  116897  ”How to find out when a new 52 week Low was set in the past 90 days?”    http://finance.groups.yahoo.com/group/amibroker/message/116897    

ATTACHED FILES:

• s_52weekhilo.afl
• s_52weekhilodiff.afl
• s_52weekhilomaster.afl
• s_52weekhilo-v2.afl
  p_newhilo.afl
  x_newhilo.afl
• p_higherhi-v2.afl

_{version 2 - Oct29/2007 - adds examples of plotshapes for HiLo’s and explore NewHiLo with the corresponding formula files uploaded.}

Filed by brian_z at 11:13 am under Scans
2 Comments

(4 votes, average: 4.5 out of 5)

 Loading ...

Before continuing with this post you should carefully read the AmiBroker Help topic for the getPerformanceCounter().

Measuring time is an important aspect of all real-time intraday trading systems. Typical tasks requiring high-resolution timing include:

1. Limiting the message rate to Interactive Brokers (IB) to 50/sec (API Error 100).
2. Inserting small delays before polling IB Status, to allow for Internet delays.
3. Staggering (interlacing) portfolio trades to spread out the action.
4. Measuring and optimizing AFL execution time.
5. Modifying orders after a small delay (to ensure fills).
6. Periodic execution of tasks, for example, Watchlist scanning, Display and Status refresh, calculations based on slow changing variables, etc.
7. Time-Stamping events, for example order placement.
8. Collecting/preprocessing quotes.
9. Overlay live tick-charts on faster and easier to manage 1-minute charts.

Most of these tasks can be accomplished using just three custom timing functions:

1. GetElapsedTime(): A function that returns elapsed time since reset.
2. SetDelay(): A function that returns time left to reach a future time.
3. GetDelayTrigger(): A function that returns a trigger when a delay times out.

This post provides example functions in a demo application. To allow the use of many timers each function requires you to provide a TimerName, which will be used to retrieve timer information. Static variables are Global and can be read from anywhere; this means you have to be careful not to cross-reference the timers by using the same TimerName from different panes or windows. When running multiple copies of the same code you will need to key the TimerNames. For more on how to do this, see Keying Static Variables in the Real-Time AFL Programming category.

The timers below are implemented using the getPerformanceCounter(). This function returns the amount of time elapsed since the computer was last started. Tomasz recently explained this as follows: “The underlying high frequency counter runs all the time since computer start. What ‘reset’ flag really does is to store last value so next time you read it, it gets subtracted from last value giving you the difference. If reset is false, the last value is set to zero, and you get the original number of ‘clock ticks’ since computer start“.

The timers in this post do their own sampling of the underlying high frequency counter, and the getPerformanceCounter() Reset argument is always left set to False.

The getPerformanceCounter() returns values with microsecond resolution; however, the practical accuracy is severely limited by interruptions from the computer’s operating system. Do not expect much better than about 50-millisecond absolute accuracy. Aside from designing your own dedicated trading hardware (to replace the PC) there isn’t much that can be done about this. If you are brave, you can experiment with increasing program priority in your Task Manager window.

Chart refreshes are most often initiated by an arriving quote, but they can also be initiated by mouse clicks, tooltip, and various chart operations. This means that, when market activity is low and things are not happening as fast as you would like, you can force extra AFL executions by clicking on your chart. You can verify this by running the code below and, while clicking rapidly on the chart, observe that the timer counts displayed will update more rapidly.

You can ensure a one-second chart refresh by adding a RequestTimedRefresh(1) to your code. If the frequency of your arriving data is slow, your AFL code may execute only sporadically. Since your code must execute to read your timers, the resolution of your timers will be limited by the chart refresh rate. If your chart refreshes once a second your timing resolution will be one second!

Normally most of the AFL code in an Indicator window executes when your chart refreshes; however, to obtain speed advantages, you may execute non-critical sections (like account information and System Status) of your code less frequently using a timer. You can also execute small sections of code more frequently by placing them inside a well-controlled loop. If you do this, be sure to limit the maximum time your code can spend inside the loop to one second or less.

For fast trading systems, the frequency of AFL executions (chart refreshes) may be slow and this may make it difficult for you to get LMT fills. There is no way to have a program that requires 50 milliseconds per pass to execute 20 times per second.

Considering the interval between AFL executions, it is important to plan the layout of your code so that all events are handled in the most efficient order. If you don’t, transmittance of your order could well be delayed by up to a full second. There are situations where you want to invoke an immediate re-execution of your code. In some cases you might want to do this after placing an order to check order status before the next quote or refresh. Although it should be used sparingly this is possible by calling the RefreshAll():

1
2
3
4
5

function RefreshAll() 
{ 
oAB = CreateObject("Broker.Application"); 
oAB.RefreshAll(); 
}

This function can only be called once a second; calling it faster will not result in more frequent chart refreshes. This means you should only call it when really needed.

The code presented below is for demonstration only. The getElapsedTime() lets you measure elapsed time from the moment of Reset. The first argument passes the name you assign to the static timer; this allows you to use the same function to time different events. The second argument is a Reset flag. When this Reset is True, the function samples the underlying high frequency counter and uses it for later reference. When you call the function with the Reset argument set to False, it calculates the elapsed time by subtracting the earlier sampled value from the current value of the Performance Counter.

The setDelay() function lets you Start, Read, and Cancel a time delay. The TimerName argument functions as in the getElapsedTime(). Calling the setDelay() with the mSecDelay argument set to a non-zero value will start the Delay timer. Calling it with the mSecDelay argument set to zero will make it return the current count-down time in millseconds. Calling the function with the cancel argument set to True will terminate the delay.

The getDelayTrigger() function returns a trigger. This is a signal that is true for only one pass through the code. Triggers are frequently used in real-time trading systems. They are needed to prevent multiple actions when a signal becomes True.

To run the code, copy the formula to an Indicator and click Insert. You’ll see a chart window like Figure 1 below:

Figure 1. Result from running the example code.

The example code maintains three timers, T1, T2 and T3. All timing values are expressed in milliseconds. In Figure-1 the Elapsed Time shown is measured from timer Reset. The Delay shown is the time remaining after Start, until the delay times out. The line for Timer T2 shows that its Delay just timed-out and produced a trigger. Timer T3 still has a Delay in progress. Right-click on the chart to open the Param window:

Figure 2. Param window.

If you click one of the timer Resets in the Param window you’ll see the ElapsedTime in the corresponding row go to zero, and then start to increment sporadically when your chart refreshes. Without live data this would be at approximately 1-second intervals, as determined by the RequestTimedRefresh(1);

If you click Start for one of the timers this will start a delay. You can see how it counts down in the Delay column. Click the timer’s Cancel to terminate the Delay. Note that whenever a Delay times out, the word “Trigger” briefly appears in the third column.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88

Filename = StrLeft(_DEFAULT_NAME(),StrLen(_DEFAULT_NAME())-2);
 
function RefreshAll()
	{
	oAB = CreateObject("Broker.Application");
	oAB.RefreshAll();
	}
 
function getElapsedTime( TimerName, Reset )
	{
	if( Reset ) 
		{
		TimeRef= GetPerformanceCounter(False);
		StaticVarSet(TimerName,TimeRef);
		}
	TimeRef = Nz(StaticVarGet(TimerName));
	ElapsedTime = GetPerformanceCounter(False) - TimeRef;
	return ElapsedTime;
	}
 
function setDelay( TimerName, MsecDelay, Cancel )
	{
	HRCounter= GetPerformanceCounter(False);
	if( Cancel ) 
		{
		StaticVarSet("TO"+TimerName,-1);
		StaticVarSet("DS"+TimerName, 0);
		}
	else if( MsecDelay ) 
		{
		StaticVarSet("TO"+TimerName,HRCounter+MsecDelay );
		}
	TimeOutTime = Nz(StaticVarGet("TO"+TimerName));
	DelayCount = Max(0, TimeOutTime - HRCounter );
	StaticVarSet("DC"+Timername, DelayCount);
	return DelayCount;
	}
 
function getDelayTrigger( TimerName )
	{
	DelayCount = Nz(StaticVarGet("DC"+TimerName));
	DelayState = DelayCount > 0;
	PrevDelayState = Nz(StaticVarGet("DS"+TimerName));
	StaticVarSet("DS"+TimerName, DelayState);
	return DelayState < PrevDelayState;
	}
 
RequestTimedRefresh( 1);
 
_SECTION_BEGIN("TIMER 1");
Reset1 = ParamTrigger("1 - Reset","RESET");
MSecDelay1 = Param("1 - Delay (mS)",1000,0,10000,10);
if( ParamTrigger("1 - Start", "START") ) setDelay( "Timer1", MSecDelay1, 0 );
if( ParamTrigger("1 - Cancel", "CANCEL") ) setDelay( "Timer1", MSecDelay1, 1 );
_SECTION_END();
 
_SECTION_BEGIN("TIMER 2");
Reset2 = ParamTrigger("2 - Reset","RESET");
MSecDelay2 = Param("2 - Delay (mS)",4000,0,10000,10);
if( ParamTrigger("2 - Start", "START") ) setDelay( "Timer2", MSecDelay2, 0 );
if( ParamTrigger("2 - Cancel", "CANCEL") ) setDelay( "Timer2", MSecDelay2, 1 );
_SECTION_END();
 
_SECTION_BEGIN("TIMER 3");
Reset3 = ParamTrigger("3 - Reset","RESET");
MSecDelay3 = Param("3 - Delay (mS)",8000,0,10000,10);
if( ParamTrigger("3 - Start", "START") ) setDelay( "Timer3", MSecDelay3, 0 );
if( ParamTrigger("3 - Cancel", "CANCEL") ) setDelay( "Timer3", MSecDelay3, 1 );
_SECTION_END();
 
ET1 	= getElapsedTime( "Timer1", Reset1 );
ETA1 	= setDelay( "Timer1", 0, 0 );
TT1 	= getDelayTrigger( "Timer1");
 
ET2 	= getElapsedTime( "Timer2", Reset2 );
ETA2 	= setDelay( "Timer2", 0, 0 );
TT2 	= getDelayTrigger( "Timer2");
 
ET3 	= getElapsedTime( "Timer3", Reset3 );
ETA3 	= setDelay( "Timer3", 0, 0 );
TT3 	= getDelayTrigger( "Timer3");
 
Title = "\nFilename: "+Filename+"\n\n"+
"MilliSeconds Since Computer Startup: "+NumToStr(GetPerformanceCounter(False),1.3)+"\n\n"+
"Timer     ElapsedTime     Delay     Trigger\n\n"+
"T1      "+NumToStr(ET1,10.0)+"     "+NumToStr(ETA1,7.0)+"      "+WriteIf(TT1,"TRIGGER1","")+"\n\n"+
"T2      "+NumToStr(ET2,10.0)+"     "+NumToStr(ETA2,7.0)+"      "+WriteIf(TT2,"TRIGGER2","")+"\n\n"+
"T3      "+NumToStr(ET3,10.0)+"     "+NumToStr(ETA3,7.0)+"      "+WriteIf(TT3,"TRIGGER3","");

Edited by Al Venosa.

Filed by Herman at 9:09 pm under Real-Time AFL Programming
2 Comments

(2 votes, average: 3.5 out of 5)

 Loading ...

OBJECTIVE

The objective for this tutorial is to setup a Nasdaq Composite database including Group, Sector , Index, and Watch-list categorization.

SPECIFICATIONS

• AmiBroker: Standard v5.0 
• Amiquote: v1.94 registered
• Time Frame: Daily
• Data Source: Yahoo
• Data Type: US Stocks
• OS: Windows XP (Home)
• Software: Microsoft Office XP Professional 2002
• Browser: Internet Explorer v7.0

PRE-REQUISITES

Readers should be familiar with the Users’ Knowledge Base (UKB) posts, Exchanges >> Nasdaq’s Stock Screener and Indexes >> The Nasdaq Indexes, that explain how to download the ticker lists, used in this example, from the Nasdaq site.

INTRODUCTION

This article builds on the ‘template’ established in the UKB post Database Management >> Setup A Custom Database by applying the same ‘top-down’ method, of setting up a database, to the broad based Nasdaq Composite Index (^IXIC). The ‘template’ has been adapted to suit the new Watch list procedures introduced in AmiBroker v 5.0 and also to accommodate the available data set for the Nasdaq Exchange.

Note: An unequivocal method is given, to avoid confusing readers, but in reality there are many possible custom databases and several ways of constructing each of them within AmiBroker. Readers will need to creatively apply the skills and tools demonstrated in this post, and possibly a few more besides, when building custom databases of their own. 

PREPARATION

The component lists, used in this example, are recorded in Table 1 (down-loadable copies are attached near the end of this post).

Table 1 

As stated in the pre-requistes section, they were downloaded from the Nasdaq Exchange site.  Most of them can also be downloaded, in batches, from the Yahoo!Finance site http://finance.yahoo.com/indices?e=nasdaq by selecting the Components link on that page ( a one-off check showed that the lists used by the author and the lists available at Yahoo did not tally but they they were downloaded at different times, within the range of a week or two).

Note: The NasdaqHealthcare sector is not included in the Yahoo listings (the Healthcare Index symbol, ^IXHC, does, however, return prices and a component list when entered in a Get Quotes search). For this example, the Nasdaq lists were preferred, over the Yahoo lists, because of the superior downloading capabilities and (assumed) superior accuracy of the Nasdaq versions.

The list of sub-indexes, that are derived from the Nasdaq Composite, wasn’t available as a download at either site and was manually compiled by the author (refer to Table 2).

Table 2

MEMBERSHIP TESTS

Some planning is required when assigning symbols to Categories as the sub-Categories need to be unique and the order in which the assignments are made can affect the outcome (if the sub-groups are not unique symbols with joint membership will be moved to the most recent assignment leaving the sub-groups that are assigned first short). Mutual lists, that are not exclusive need, to be incorporated into the database at the Watch list level, where joint membership of sub-Categories is permitted. It is, therefore, wise to test membership at all levels before proceeding with the setup.

A test of all component lists , for mutual exclusivity, can be carried out, in Excel. Example files, suitable for this purpose, can be downloaded from the Attached Files section of this post (refer to MembershipTestGroups.xls, MembershipTestSectors.xls and MembershipTestSubIndexes.xls).

Using the membership test file the following results were obtained:

• 155 symbols in the Composite list do not have a match in the National Market and Capital Market (Nat/Cap) lists.
• 50 symbols in the Nat/Cap list do not have a match in the Composite list.
• The Nat/Cap lists are unique, relative to each other.
• There are two symbols in the Composite list that are not included in the Sector lists  i.e. APRO and FTSW.
• 170 Biotech symbols are also members of the Healthcare sector.
• All the other Sector lists are unique, relative to each other.
• The adjusted count, when BioTech symbols are deducted from the Healthcare count, totals 3092 which equals the Composite total of 3094 minus 2 (refer to Table 3). 
• The Nasdaq100 and the NasdaqFinancial100 are unique lists that are part of the NasdaqComposite.
• The Nasdaq100 Technology and the Nasdaq100 Ex-Tech are unique lists that together make up the Nasdaq100.

DATABASE DESIGN       

Based on the membership test results the rational for the database design is:

• The Composite members will makeup the global database, at the All Symbols level, with the remaining Categories subordinate to that list.
• Since the Composite Index is based on one market the Market list is a surrogate for the global database. 
• 14 Indexes, that track different subsets of the Composite, are not included in any lists and will be added to the database at the global level. This will increase the total number of symbols in the database to 3108 == 3094 (Composite) + 14 (Indexes).
• The Nat/Cap markets are major, mutually exclusive, lists that will comprise the next tier of categorization at the Group level.
• 155 symbols, that are not in the Nat/Cap lists, will remain unclassified at the Group level.
• The 50 symbols that are exclusive to the Nat/Cap list will not be added to the database so that the database remains faithful to the Composite at the global level.
• Sectors and Industries will be entered in a one to one relationship, since no information is available for Industry classifications.
• As symbols can not be members of more than one sub-Category, at the Sector/Industry level, the 170 symbols that have joint membership of the BioTech and Healthcare sectors will not be included in Healthcare (this will mean that the Healthcare sector, for this database, will not be a true surrogate of the Healthcare Sector Index).
• The complete Healthcare Sector list and the Nasdaq100 sub-Indexes will be included as Watch lists. 

The operations required to setup the database are:

• - save a copy of the original broker files in the C:/Program Files/AmiBroker (installation) folder,
• - save the database specific broker files in the installation folder,
• - backup the database specific broker files into a spare folder in the installation folder (optional), 
• - create a new database and manually name the Markets and Groups,
• - use the ASCII Wizard to import the Composite symbols (this adds symbols to the database, complete with Information),
• - import the Indexes into a Watch list folder using the ASCII Wizard (this adds symbols to the database, complete with Information),
• - use the ASCII Wizard to assign the Nat/Cap symbols to their Groups (this moves the symbols from the Unclassified sub-category to the Nat/Cap sub-categories within Groups),
• - backup the database specific files (constituent lists) into a spare folder in the installation folder (optional), 
• - import the Sector lists into ‘temporary’ Watch lists, via the Watchlists folder,
• - import current data to the database,
• - perform quality control checks (test for NoQuotes and global database membership),
• - repair or delete NoQuotes, 
• - and then assign symbols to sectors using the X_categoryAddSymbols  formula,
• - quality check the sector assignments,
• - empty the ‘temporary’ Watch lists,
• - assign index symbols to the Indexes folder (move them from the temporary Indexes WL using the X_categoryAddSymbols  formula),
• - manually assign the index symbols their respective Sector/Industry,
• - import the Composite sub-Indexes to ‘permanent’ Watch list folders,
• - and finally import historical data as required.

Refer to Table 3 for a ’summary’ of the process.

Table 3

Note: The Capital and National Market files do contain Symbol and Fullname information but since they are only being moved, from to one sub-Category to another, the Fullname can be ignored (the symbols already exist in the database and the Fullname information field was populated at the global level).

The Group assignment could have been done in other ways, however, for this example the Import Wizard was used, for this task, to demonstrate it’s assignment capabilities.

Once the user has a database structure ‘mapped out’ the creation process can start (additional effort at the ‘design’ stage will often save time later as unexpected occurrences, during the setup, can force a re-run or two.

CREATE A NEW DATABASE

The first step in the process is to create a new database.

To create the database:

1) Make a backup copy of the original broker.industries and broker.sector files in the C:/Program Files/AmiBroker (installation) folder (or the equivalent).

2) Download the broker.industries and broker.sector files attached to this post and Save them in the installation folder to replace the original versions.

Note: A handy tip is to create a new folder in the AmiBroker directory and save copies of the database specific broker files there, as a backup, before pasting them into the ‘working’ directory (save them in a Files folder or in a files folder with the same name as the database e.g. Data_NasdaqComposite_Files). The database backup files are then only one click away from the AmiBroker directory if they need to be checked, or reinstalled to the root folder following a mishap.

3) Create a new database called Data_NasdaqComposite, using the default EOD settings.

When the database is created the Sectors and Industries lists will automatically be set to the broker file specifications (the Sectors and Industries have a one to one relationship as no Industry information is available, for this example).

The Markets and Groups categories are not established when the database is created and they need to be set manually.

4) Check both categories and Edit the first name in the list as required  (for this example the Markets default needs to be Nasdaq and the Groups default needs to be Unclassified).

Once the Markets and Groups have be named symbols, and symbol information, can be imported into the database at the global level.

To import from a CSV file using the Wizard:

1) Open the Import Wizard using the File > Import Wizard command from the menu bar.

2) Click on Pick files.

3) Select the file to be imported from the file browser window that pops up and click on Open.

Note: NasdaqComposite.csv is a copy of index_components.csv that was downloaded in the post: UKB >> The Nasdaq Indexes

The selected file will be listed in the Import Wizard’s information pane.

4) Click on Next >.

The Define Fields window will open with a ‘preview’ of the file format in the Data file sample pane.

In this example the file has a two header rows, the first data column contains the Name (Fullname) and the second column contains the Symbol (Ticker). The data columns are separated by a comma and there is no quotation data included.

5) Define the fields according to the file format:

•           use the drop-down menus to set Column 1 to Fullname, Column 2 to Ticker and the remaining columns to Skip (ignore) anything else,
•           set the Separator to Comma(,),
•           pick Unclassified from the Group drop-down menu,
•           skip the first two lines (title rows) by entering 2 into the Skip first: input box,
•           check Log errors (optional),
•           check Automatically add new symbols (all of the symbols are new to the database and need to be added),
•           check No quotation data (the list does not contain any quotation data as only Tickers and Fullnames are to be imported).

When the settings are correct click on Next > and an Additional Settings window will open.

6) For this example there is no need to save the import format.

    Click on Finish.

7) Click on Next > to complete the Symbol/Information importation.

All of the symbols are arranged in one market (Nasdaq), Unclassified at the Group level and listed under the default Sector/Industry (NasdaqBank 0) as expected.

Since the Indexes are not included in the composite list they also need to be imported, in ASCII format, to add the symbols and symbol specific information to the global database.

To enter the Index symbols to a Watch list and the database:

1) Use the Import Wizard to import the data from the NasdaqIndexes.CSV file (set the Group sub-category to Unclassified and the Watch list (WL) to List 0 using the drop-down menus) .

2) The index symbols will be imported into Watch list 0 and enter the database under the Nasdaq|Unclassified|NasdaqBank 0 sub-categories.

3) Rename the Watch list to Indexes.

4) Save the database using the File >> Save Database command.

Note: It isn’t essential to save at this stage but it is a good precaution to take before continuing on.

This establishes Data_NasdaqComposite as a ‘global’ database that represents a broad cross-section of Nasdaq listings plus the associated indexes.

From there it can be categorized according to personal preference.

CATEGORIZE THE DATABASE CONSTITUENTS

Setup Group Assignments

From the membership test results the outcome of entering the Group assignments should be to move most of the symbols from the Unclassified group to the National Market & Capital Market groups, leaving 155 symbols and 14 indexes behind as Unclassified. The 50 symbols that are exclusive to the Nat/Cap list will not be added to the database.

To assign symbols to Groups using the ASCII Wizard:

1) Enter a name for the Groups, in the order required, and click on O.K.

Note: The order is permanent i.e. it will not be automatically sorted by alphabetical order in the Workspace Symbol Tree.

2) Open the NationalMarket component file in the ASCII Wizard and pick National Market 1 from the Group drop-down menu, to assign the symbols to the relevant group. Then deselect Automatically add new symbols to prevent symbols that are unique to the NM list from being added to the database.

Select the other settings as determined by the file format.

Note: The lists are from a different source to the other lists, used in this example, and the columns are in a different order such that the Symbol column precedes the Name column (always look at the preview pane carefully and don’t take the formats for granted).

Since the Import Wizard is being used for assignment, rather than information input, there is no need to use any data field other than the Ticker (the Fullname data is ’skipped’).

3) Click on Next >

An error report window will pop-up to notify the user that some symbols in the list were not imported.

4) Click on Yes to read the report.

Note: Readers who are working their way through the post might find minor differences, between their outcomes and the images used, as the author wrote the post over a period of weeks, using two computers and different file versions.

The report is in plain text format and it can be saved, or printed, for future reference.

5) Repeat the Group assignment procedure, using the Capital Market file, to finalize the Groups re-organization.

If a mistake is made in the number of lines skipped, when assigning symbols to sub-categories with the Wizard, the error can be corrected manually.

To manually reassign an Unclassified symbol e.g. EGHT to another sub-category:

1) Go to Symbol >> Organize assignments on the menu bar (the Assignments organizer window will open).

2) Select the Groups radio button then use the drop-down menu to pick Unclassified from the left hand list and Capital Market 2 from the right hand.

3) Scroll down the Unclassified list to find EGHT and select it before clicking on the Selected right arrow button to move the symbol to the nominated group.

When the Workspace Symbol Tree is refreshed it will reflect the new assignment for EGHT.

SETUP SECTOR AND INDUSTRY ASSIGNMENTS

Setting up the Sectors and Industries is a little more complicated and requires the use of an AFL formula.

The formula does the work of assigning symbols to the sectors but before it can be used some preparation is required:

• downloaded Industry lists, in CSV or Excel format, need to be converted into a Watch-list compliant (.tls) format and saved into the Database Watchlist Folder (in the installation directory),
• price data needs to be imported and redundant symbols deleted (AFL formulas will not work with symbols that do not contain quotes).  

Once those criteria have been meet the database Watchlist folder lists can be simultaneously imported into the Symbol Tree Watch lists (from AmiBroker v5.00 upwards).

Note: it wasn’t the intention of the developer that users would take this path so it does involve a minor ‘workaround’ (use at your own risk - the author has experienced a minor glitch or two using this procedure).

To import Symbols into the Symbol Tree Watch lists:

1) Open the relevant file that contains the Sector specific symbol lists e.g. NasdaqBank 0.xls

2) Copy the symbols from the spreadsheet column and Paste them into a text editor, like NotePad, (this will produce a plain text file with a single symbol per line).

2) Edit out the title rows, if any, and Save the file as a .tls file in a backup folder in the AmiBroker directory (they can be stored in the same folder as the backup broker files e.g. Data_NasdaqComposite_Files).

3) Close AmiBroker.

4)  Copy and Paste the Sector list files into the Database Watchlists Folder e.g. C:\Program Files\AmiBroker\Data_NasdaqComposite\WatchLists (it isn’t necessary to save them in any particular order).

3) Open AmiBroker again and the ticker lists will have been added to the Watchlist Master File (index.txt) which is also located in the Database WatchLists Folder.

Note: They will be added from List 64 on (inclusive). The first 64 lists (0-63 inclusive) have been reserved to maintain backward compatibility with older versions.

They will also take up their rightful place in the Workspace panel.

If the mouse pointer is hovered over a Watch list folder a tool-tip, with a report on the WL number and the symbol tally, will appear (the number of symbols per WL can be cross referenced to the records made at the design stage - refer to Table 3).

Downloading data and quality checking the database

Assigning symbols to Industries requires use of an AFL formula. Formulas will only work with symbols that contain at least one data point so it is necessary to import some data to the database. Current data will be used for this purpose since it downloads a lot quicker than historical data.

To download current data:

1) Open AmiQuote while AmiBroker is open with Data_NasdaqCombined as the selected database.

2) Get the tickers from AmiBroker using the Tools menu option and Automatically import data with Yahoo current as the select source (this will add one quote to all of the symbols that traded on the day).

Note: It is recommended to wait until after the daily close to download current data, for this purpose, so that stocks with low volume have a chance to register some trading activity during the day.

Once data has been imported it also provides an opportunity to remove, or repair, any redundant symbols in the database before downloading historical data (historical data will download faster if AmiQuote doesn’t have to report a lot of errors).

A list of the redundant symbols in the database can be made by using AmiBroker’s Data Purify tool.

To identify and quarantine redundant symbols:

1) Open the Database Purify tool (DPT) and enter the first symbol, from the All Symbols list, as the Reference symbol.

Note: Check that the symbol chosen as the reference does contain, at least, one quote.

2) Uncheck all of the report options, except for Report missing quotes.

3) Select All symbols and Last n = 1 as the settings and then click on Analyse.

A list of all stocks that DO NOT have a current quote will be returned.

4) Right click anywhere inside the list and pick Add all symbols to watch list from the context menu that opens.

Note: The DPT does not report a symbol count in the output list. Since AmiBroker doesn’t recognize No Quotes anywhere else the only way to obtain a count, for those who are using AB 4.91 and below, is to Copy the ‘list’ and Paste it into a spreadsheet program (for AB v4.92 and above the WL tool-tip will report the number of No Quotes symbols).

5) Select a vacant watch list to receive the output and click on OK.

Note: The importance of naming all Watch list folders as soon as they are ‘filled’ can readily be seen. In this example, if List 1 already contains data the outcome will be a bit of a mess that the user has to clean up (as noted by Dr Howard Bandy in his book Quantitative Trading Systems). So, in ‘real life’ the smart thing to do would be to name a Watch list folder "NoQuotes", prior to starting the database quality checks, and then send the DPT catch there.

All of the symbols identified as having no data will now be listed in Watch list 1.

Note: The redundant symbols will still remain in other Watch lists and Categories (assigning a symbol to a Watch list does not move it from anywhere else).

Sometimes a symbol list can contain a ‘rogue’ (unwanted) symbol that sneaks into the database by the backdoor (WL importing). Any symbols that were added to the database, via the Watch lists, will not have an associated Fullname (the Watch lists only contain Tickers without any additional information) so a test for symbols without a Fullname will find them).

A quick test, for global membership, can be carried out using the Explorer. For this example the test will be run against All symbols using the X_FullName formula.

To test All symbols in the global database for ‘true’ membership:

1) Run an exploration on All symbols using X_Fullname.

2) Sort the Results by clicking on the Fullname column. If there are any ‘rogue’ symbols present they will be sorted to the top of the list and have a blank in the Fullname cell. In this example there are no symbols without a Fullname which indicates that the database design and importation procedures were tight.

The number of symbols in the global database, that contain data, can be noted from the Status Bar at the bottom of the AA window.

Summing the symbol numbers, for stocks with and without data (NoQuotes), should equal the global database count (the Composite + Indexes tallies). For this database 2895 + 213 = = 3094 + 14 == 3108 (all present and correct!).

To construct a ‘working’ database it is advisable to ‘manually’ check each redundant symbol e.g. against Nasdaq’s lists or Yahoo’s lists. The reason for ‘failure’ to download current data should be found and corrections made, according to the preferences of the database owner (in some cases the symbol doesn’t exist and in others it will exist but have no volume for the day). The solutions, to problems encountered, can be wide ranging e.g. for traders who want liquidity, all ‘no volume for the day stocks’ can be deleted (they will be all that is left over in the ‘NoQuotes’ folder after ’symbol does not exist’ stocks have been deleted or repaired), while those who want to trade the small caps, or low volume stocks, can ‘run’ a short historical download for the leftover ‘no daily volume’ stocks and decide what action to take on the results they obtain.

While it might not be a realistic option, for a ‘working’ database, all ‘NoQuotes’ symbols will be deleted for this example.

To delete NoQuotes:

1) Open the Symbol >> Organize assignments window.

2) Select the Watch lists radio button.

3) Pick NoQuotes from the left hand drop-down.

4) Hold down the Shift key while selecting the first and last entry in the list (the list will turn blue to indicate that all of the symbols have been selected).

5) Click on the Delete button.

A warning message will ask for confirmation of the delete.

6) Click on OK.

The selected symbols will be removed from all levels of the database.

The tool-tip can be used to sum the new Watch list counts and check the total against the global database total of 2895.

Assigning Watch list constituents to Industries

Once all of the symbols in the database contain data they can be assigned to their Industries using X_categoryAddSymbols v4

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

/*X_CategoryAddSymbols v4*/   
//assigns Watch list members to the Industry with exactly the same name as the Watch list    
//Watch lists and Industries can be in any order    
//manually enter the first and (last Watch list number +1) for Watch lists that contain assignment symbols    
//tested OK with AB v5.0  
Filter = 1;  
for(WL = 64; WL < 73; WL++)//manually input the first and (last WL number + 1)  
&#160;&#160; {  
&#160;&#160;&#160; WLName = CategoryGetName(categoryWatchlist,WL);    
&#160;&#160;&#160; Ticker = WriteIf(InWatchList(WL), Name(),"");     
&#160;&#160;&#160; AddTextColumn(Ticker, WLName);     
&#160;&#160;&#160; IN = CategoryFind(WLName,categoryIndustry); //IndustryNumber to match selected WL Name  
&#160;&#160;&#160;&#160;&#160;&#160; if(InWatchList(WL))    
&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160; {     
&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160; CategoryAddSymbol("",categoryIndustry,IN);     
&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160; }  
&#160;&#160; }

To assign symbols to their industries:

1) Hover the mouse pointer over the first and last Watch list folders, that contain assignment lists, to check the WL number.

2) Enter the first and (last Watch list number + 1) to the formula, as the loop variable and the counter upper limit.

3) Run X_categoryAddSymbols v4 in Explorer with All symbols, n last days and N = 1 selected.

The symbols will be assigned to Industries, according to their Watch list membership (the Watch lists constituents will be reported in the AA window).

4) Click on the Result pane Titles to sort the column by Watch list and scroll down to manually cross check WL membership against Sector membership.

The Results can also be exported to a spreadsheet for printing or saving for reference.

Quality checks on the assignment process can also be run using X_Fullname.

To check Sector assignments against Watch lists:

1) Check use filter in the AA window and click on Define to set the filter criteria.

The Filter settings window will open.

2) Click on Clear to remove the previous settings (Includes and Excludes should both be free of redundant settings).

3) Select a Sector from the drop-down list and click on OK.

4) Run an Exploration using n last days and n = 1.

5) Leave the Automatic Analysis window open and right click on the corresponding Watch list folder, to open the context menu, then alphabetically Sort the list.

The Sector count, from the AA window status bar, can be checked against the corresponding Watch list count and the list members can be checked against each other by scrolling the lists.

Once the Sectors have been checked, and all of the symbols assigned correctly, the ‘temporary’ Watch lists can be emptied.

To empty Watch lists:

  1) Right click on the Watch list folder and select Erase (make empty) from the context menu that opens.

2) Click on Yes to continue when a warning message opens.

Alternatively, to empty a larger number of Watch lists:

1) Close AmiBroker.

2)  Delete all the temporary Watch list (.tls) files from the C:/Program Files/AmiBroker/WatchLists folder.

3)  Open AmiBroker (the ‘temporary’ Watch lists will be empty and no longer visible in the Symbol Tree).

Note: Users can elect to view empty WL’s by right clicking on a WL folder and deselecting Hide Empty watchlists from the context menu that opens.

The next step, in setting up the Nasdaq Composite Database, is to move the symbols in the Indexes WL to the Indexes Category.

To assign symbols to the Index folder:

1) Open AA and set the filter to the Indexes Watch list.

2) Run categoryAddSymbol v5.0 as an Exploration.

The index symbols listed in Watch list 0 will be assigned to the Indexes folder.

3) Watch list 0 can now be emptied.

That completes the reorganization of the index symbols into their own category. They will also be listed at the top of the default sub-category in all the other categories (automatic alphabetical sorting brings them to the top). As they are an exclusive group they can be moved to their own sub-category at any level. For this example they will be left in the Unclassified sub-category at the Group level and individually moved to their corresponding Sector/Industry (this could also have been done earlier by including them in the Industry lists).

To move the Index symbols from the default Industry sub-category to their own Industry:

1) Go to Symbols >> Organize assignments and select the Industries radio button on the Assignments Organizer window that opens.

2) Pick NasdaqBank 0 (the default) from the left hand drop-down list and then select the index symbol to move.

3) Pick the target industry from the right hand drop-down and then click on the left to right arrow button to move the symbol to it.

4) Repeat the procedure for all of the indexes that have a corresponding industry sub-category.

After all of the Sector indexes have been assigned five index symbols remain in NasdaqBank 0 (the composite index and four N100 indexes). They can be moved by naming an Industry as Unclassified and using the Assignments organizer to move them to it.

When all of the assignments are complete, and the watch list folders have been emptied, the symbols that comprise the permanent watch lists can be imported.

To import ‘permanent’ Watch lists:

1) Rename WL’s to match the lists that are to be imported.

2) Go to Menu bar >> Symbol >> Watch list >> Import and the Select watch list(s) window will open.

3) Select the File radio button and click on OK (a file browser window will open).

4)  Choose the required file from the list and click on OK (the symbols in the file will be imported into the selected WL).

Note: The file needs to be a text file with one ticker per line.

5) Repeat the process for the remaining WL’s including the original (complete) Healthcare Sector list.

Finally, the database can be saved and historical data downloaded by opening Amiquote, getting the ticker list from the open database and downloading the required range of quotes in the usual way.

CONCLUSION

That completes this tutorial that demonstrates one method of creating a database. It includes examples of the full range of graphical tools and basic skills that users require to allow them to customize their own databases, according to their needs and the data at their disposal. Some alternative methods of setting up a database will be dealt with, at a basic level, in future posts (including user defined ASCII importing and, possibly, low level ‘automation’).

ATTACHED FILES:

broker industries and broker sector files: brokersectors.txt and brokerindustries.txt

Nasdaq composite file: nasdaqcomposite.xls

Nasdaq markets file: nasdaqmarketssept2007.xls

Nasdaq National and Capital markets files: nasdaqcapitalsept2007.xls  and  nasdaqnationalsept2007.xls

Excel file used to compare Group membership: membershiptestgroups.xls

Excel file used to compare Sector membership: membershiptestsectors.xls

Excel file used to compare Index membership: membershiptestsubindexes.xls

Nasdaq Sector Indices files:

 nasdaqbank-0.txt

nasdaqbiotech-1.txt

nasdaqcomputer-2.txt

nasdaqhealthcareminusbio-3.txt

nasdaqindustrial-4.txt

nasdaqinsurance-5.txt

 nasdaqotherfinance-6.txt

nasdaqtelecommunications-7.txt

 nasdaqtransportation-8.txt

nasdaqhealthcare.txt

Nasdaq Sub-Indexes files:

nasdaq100.txt

nasdaq100extech.txt

nasdaq100tech.txt

nasdaqfinancial100.txt

Filed by brian_z at 11:05 am under Database Management
2 Comments

(No Ratings Yet)

 Loading ...

A list of unqualified data resources with a bias towards Real Time equity data (mainly for the American market). Some of the sites do provide data for other financial instruments as well.

The list is dynamic and subject to change without notice.

The list is ordered according to when the author researched the site and it is not sorted by rank or order of merit.

CAUTION - MOST OF THE LINKS CONTAIN COMMERCIAL REFERENCES OR LINK TO COMMERCIAL SITES.

NOTE: THE AUTHOR HAS NO COMMERCIAL AFFILIATIONS OF ANY KIND AND RECEIVES NO GRATUITIES, OR BENEFITS, FROM ANY COMMERCIAL ORGANIZATION OR INDIVIDUAL INVOLVED IN COMMERCIAL ACTIVITIES. NOR IS HE PERSONALLY ENGAGED IN ANY COMMERCIAL ACTIVITIES, RELATED TO TRADING, IN ANY WAY. THE LINKS ARE PROVIDED AS A RESOURCE AND FOR EDUCATIONAL PURPOSES. THEY ARE NOT A RECOMMENDATION, ON THE PART OF THE AUTHOR, NOR DO THEY CONSTITUTE INVESTMENT ADVICE AND SHOULD NOT BE CONSTRUED AS SUCH.

4) Paritech

http://www.paritech.com.au/default-au.htm

A Melbourne (Australian) based provider of Real Time market data for the Australian Stock Exchange.

 Additional Products:

Unlimited real time data for the Australian markets using Paritech’s Pulse software or DDE functionality - Real Time export function - export 50 time series live fields and events out of PULSE into your own application e.g. Excel and other compatible software.

http://www.paritech.com.au/support/pulse/features/dde.asp

3) TechnicalTrends

 http://www.technicaltrends.com/

Indian stock-market data - quality end of day and intra day data for stocks, futures & commodities traded on Indian exchanges (NSE, BSE, NCDEX and MCX). Extensive historical data is available. Intra day data (30 minute) is available for ALL stocks in the NSE.  End of day data is provided in Computrac format (now popular as Metastock format)  while intra day data is in text files easily accessed by all technical analysis software. Commodities data is provided for all contracts listed on the NCDEX & MCX.  Data is available in individual, as well as continuous contracts.

2) ViraTech

http://www.viratechindia.com/index.html

Indian stock-market data - intraday data for active cash stocks, as well as ALL current month Futures Stocks & Indices. Historical data / Continuity / Offline option - with a large intraday history, you have completely continuous authentic data for over 15000 5-minute bars.

1)  eSignal

A reliable vendor for intraday data feed in Amibroker format including International baskets e.g. futures of National Stock Exchange(NSE, India).

Filed by brian_z at 11:08 am under Data Resources
No Comments

(No Ratings Yet)

 Loading ...

An unsorted, unqualified list of FAQ’s on anything and everything to do with Real Time equity data.

The list is dynamic and subject to change without notice.

CAUTION - SOME OF THE LINKS CONTAIN COMMERCIAL REFERENCES, OR LINK TO COMMERCIAL SITES.

NOTE: THE AUTHOR HAS NO COMMERCIAL AFFILIATIONS OF ANY KIND WITH, AND RECEIVES NO GRATUITIES OR BENEFITS FROM, ANY COMMERCIAL ORGANIZATION OR INDIVIDUAL INVOLVED IN COMMERCIAL ACTIVITIES. NOR IS HE PERSONALLY ENGAGED IN ANY COMMERCIAL UNDERTAKINGS, RELATED TO TRADING, IN ANY WAY.  THE LINKS ARE PROVIDED AS A RESOURCE AND FOR EDUCATIONAL PURPOSES ONLY. THEY ARE NOT A RECOMMENDATION, ON THE PART OF THE AUTHOR, NOR DO THEY CONSTITUTE INVESTMENT ADVICE AND SHOULD NOT BE CONSTRUED AS SUCH. 

To be commenced at a future date.

Filed by brian_z at 11:06 am under Data
1 Comment

(No Ratings Yet)

 Loading ...