Importing Data Text Files
Tickblaze can import historical market data from text files in various formats, the connection process works as follows:
•Create a Text Files Data Connection
•Specify the folder that contains your text files
•Use Tickblaze as usual. Whenever you perform an action that requires historical data (e.g., viewing a Chart or Backtesting), Tickblaze will search the text files for a file that contains the data and then import it. Alternatively go to the Data Warehouse and use the Import Wizard to import the data from the connection.
Filenames
Whenever Tickblaze needs to import data, it searches for a file that contains that data inside the specified folder. It expects files that contain data for a certain symbol to have that symbol in their filename. The filename is case-insensitive, and the file extension is always ignored.
For example, if you're requesting data for AAPL then the following filenames are acceptable (i.e., Tickblaze will read these files):
•Just the symbol. E.g.: AAPL.csv, or aapl.txt.
•Start with the symbol; then a punctuation character; and then the rest of the filename is ignored. E.g.: AAPL-20161101.txt, or AAPL-data.blah.
However, the following filenames will not be accepted:
•data.txt - because it doesn't contain "AAPL"
•data-AAPL.txt - because it doesn't start with "AAPL"
•AAPLdata.txt - because there's no punctuation character after "AAPL"
Filenames by Instrument Type
Here's what Tickblaze expects symbols to look like:
•Equities - the symbol as-is. E.g.: AAPL, IBM.
•Forex - the symbol without slashes or any other punctuation characters. E.g.: EURUSD. (But not EUR-USD.)
•Futures - the symbol in the filename should be formatted as SYMBOL_MM_YYYY, where MM_YYYY are the contract month and year. E.g.: ES_09_2016.
Tickblaze supports several futures contract types. The symbol shown above (e.g, "ES_09_2016") is used to import a single contract. But if you want to import a continuous contract then do the following:
•Prepare a single text file that contains long-range historical data for the contract. When naming this file, add the string "#0" to the end of the symbol. E.g.: "ES_09_2016#0". The string "#0" indicates that the file contains data for a "Continuous By Data Provider" contract. In Tickblaze, when you select the futures contract, indicate that you want to use the contract type "Continuous By Data Provider".
•If you want to import contracts of type "Continuous Back-Adjusted" or "Continuous Non Back-Adjusted" then you need to provide multiple files: one for each of the individual contracts that together form the continuous contract. E.g., your files would be named: "ES_09_2016", "ES_06_2016", "ES_03_2016", "ES_12_2015", etc.
Examples
Tickblaze auto-detects the file format. It recognizes a wide variety of formats, which are described below. But first, here are some examples of files that Tickblaze is able to understand:
Date,Open,High,Low,Close,Volume,OpenInterest
2021-11-02,119.26,121.36,119.25,121.25,293091,0
2021-11-03,121,123.49,120.33,122.52,419470,0
2021-11-04,122.7,123.82,121.62,122.2,399557,0
2021-11-05,121.83,122.69,120.18,121.37,357944,0
2021-11-06,121.26,121.81,120.62,121,289695,0
2021-11-09,121.02,121.81,119.7,120.41,314079,0
2021-11-10,120.45,120.45,116.06,116.4,541001,0
When Tickblaze tries to import this file, it automatically recognizes the following attributes of the file format:
•Header Lines = 1 (Tickblaze ignores lines at the start of the file that don't match the format of the rest of the lines)
•Bar Type = Day (vs. Minute, Tick or Bid/Ask)
•Date Format = YYYY-MM-DD (year in 4 digits; month in 2 digits; day in 2 digits)
•Field Delimiter = comma (the character that separates different fields)
•Decimal Delimiter = period (the character that separates the whole and fractional parts of a price)
Here's another file, which uses a different format:
Date Time Open High Low Close Volume OpenInterest
14112016 09:01:00 42,86 42,96 42,85 42,92 1138 0
14112016 09:02:00 42,92 42,92 42,75 42,77 980 0
14112016 09:03:00 42,76 42,85 42,74 42,84 900 0
14112016 09:04:00 42,84 42,88 42,8 42,86 567 0
14112016 09:05:00 42,87 42,87 42,81 42,85 415 0
14112016 09:06:00 42,86 42,96 42,85 42,9 878 0
14112016 09:07:00 42,9 42,93 42,86 42,91 430 0
14112016 09:08:00 42,91 42,91 42,89 42,9 221 0
14112016 09:09:00 42,9 42,91 42,82 42,87 422 0
14112016 09:10:00 42,86 42,93 42,86 42,91 372 0
When Tickblaze tries to import this file, it automatically recognizes the following attributes of the file format:
•Header Lines = 2 (including the blank line on Line 2)
•Bar Type = Minute
•Date Format = DDMMYYYY
•Time Format = HH:MM:SS (hour in 2 digits; minute in 2 digits; second in 2 digits)
•Field Delimiter = space
•Decimal Delimiter = comma
Supported File Formats
General Format
Files should have one bar per line.
There can optionally be some header lines at the start of the file: e.g., a line that contains the name of the fields. Tickblaze automatically ignores these lines.
Short files might not be recognized correctly; make sure the files have plenty of data.
Supported Bar Types
Each file contains data for a single bar type. The following bar types are supported:
•Day
•Minute
•Tick - each row specifies which type of tick it contains (Bid/Ask/LastTrade)
•Bid/Ask - similar to Tick bars, but each row contains both Bid and Ask ticks. This is used in forex.
The way Tickblaze determines whether a file contains tick bars or bid/ask bars is by the symbol: for forex symbols, it assumes that the file contains bid/ask bars, unless the file format doesn't match, at which points it tests whether the files contains regular tick bars. For other symbols, it assumes that the file contains tick bars.
Field Order
Tickblaze expects fields to appear in a certain order. Note that Tickblaze does not read the field order from the header line, even if it exists: header lines are always ignored.
Day Bars
•Date, Open, High, Low, Close, Volume, Open Interest
The "Open Interest" field is optional.
Minute Bars
•Date+Time, Open, High, Low, Close, Volume, Open Interest
•Date, Time, Open, High, Low, Close, Volume, Open Interest
The difference between these two formats is in how the Date and Time are formatted. In the first format (Date+Time), the date and time are part of the same field. In the second format (Date, Time) they are two separate fields.
The "Open Interest" field is optional.
Tick Bars
•Date+Time, Price, Size, Type (Bid/Ask/LastTrade)
•Date, Time, Price, Size, Type
The "Type" field is optional; if it's missing then the tick is assumed to be a Last Trade tick. The difference between these two formats is in how the Date and Time are formatted. In the first format (Date+Time), the date and time are part of the same field. In the second format (Date, Time) they are two separate fields.
Bid/Ask Bars
•Date+Time, Bid Price, Ask Price, Bid Size, Ask Size
•Date, Time, Bid Price, Ask Price, Bid Size, Ask Size
The Bid Size and Ask Size are expected to be the full size of the trade, e.g. 300,000. They shouldn't specify the number of lots in the trade (e.g., 3). The difference between these two formats is in how the Date and Time are formatted. In the first format (Date+Time), the date and time are part of the same field. In the second format (Date, Time) they are two separate fields.
Field Delimiters
The following characters may be used to separate fields:
•Comma
•Space
•Tab
•Semicolon
Decimal Delimiters
The following characters may be used to separate the whole and fractional parts of a price:
•Period
•Comma
Date/Time Formats
The following date formats are supported:
•yyyy-MM-dd
•yyyy-dd-MM
•MM-dd-yyyy
•dd-MM-yyyy
•MMddyyyy
•ddMMyyyy
•yyyyMMdd
•yyyyddMM
•MM/dd/yyyy
•dd/MM/yyyy
•yyyy/MM/dd
•yyyy/dd/MM
•MM.dd.yyyy
•dd.MM.yyyy
•yyyy.MM.dd
•yyyy.dd.MM
•MMddyy
•ddMMyy
•yyMMdd
•yyddMM
•M/d/yyyy
•d/M/yyyy
•yyyy/M/d
•yyyy/d/M
•M-d-yyyy
•dd-M-yyyy
•yyyy-M-d
•yyyy-d-M
•d-MMM-yyyy
The following time formats are supported:
•HH:mm:ss
•HH.mm.ss
•HHmmss
•HH:mm
•HH.mm
•HHmm
•H:mm
•H.mm
•HH:mm:ss.fff
•HH:mm:ss:fff
•HH.mm.ss.fff
•HHmmss.fff
•HHmmssfff
("fff" = milliseconds)
Text Files Data Connection Settings
In addition to the auto-detected file format attributes (above), there are a couple of attributes that Tickblaze can't auto-detect, so you must specify them yourself:
•Import From - the folder that contains your data files.
•Time Zone - which timezone the timestamps in the file belong to. The default option, "Use exchange time", means that the timezone is assumed to be the one where the exchange is located. E.g., if the imported symbol is AAPL, which is traded on NASDAQ, then the timezone is assumed to be US Eastern Time. If a different timezone is used (not the exchange's timezone) then you must specify it here.
•Minute Bars Timestamp - when importing minutes data, this determines whether the timestamps belong to the beginning or end of the minute.