in Search

Data converter very slow ascii2Meatastock

Last post 12-31-2007, 7:12 by Pyradius. 3 replies.
Sort Posts: Previous Next
  •  12-30-2007, 5:26 26181

    Data converter very slow ascii2Meatastock

    My data provider giving me updates in ASCII format and i need to convert it to Metastock format using downloader. The issue is it take long time to convert all the securities in the text file.

    Now i have 1 year worth of data to be converted but it take 24hours to do 1 month of conversion,, if i need to do the whole year it might take few months !!

    is there any thing i can do to speed up the process..

     

  •  12-30-2007, 20:06 26185 in reply to 26181

    Re: Data converter very slow ascii2Meatastock

    It will go much quicker if you can consolidate/sort all of your daily files into a single file and then sort them by code.

    Let's say you have 5000 securities to convert for each day.

    If you have daily files, it will process them one day at a time, opening and closing each of the 5000 securities for each day.

    If you consolidate them into one file, sort them, then convert, you'll find that it will open each security only once, dramatically increasing the speed.

    e.g. I was able to convert over 18000 securities each with around 20 years of history securities in about 2 hours on my pretty quick PC.

    Another way of speeding things up is to exclude both your ASCII and MetaStock folders from your real-time virus checking.   This typically results in a 50% reduction in time on my PC to read/write files in this area.

    The alternative is to subscribe to a data vendor that can automatically provide your data in MetaStock format with no conversion required.

    The other thing to watch out for is that daily ASCII files have no way of providing any corporate actions (such as stock splits, code change and delistings) so your database will rapidly become untidy unless you really keep on top of the maintenance.

    Cheers,
    Richard.







    www.premiumdata.net
  •  12-31-2007, 3:18 26187 in reply to 26185

    Re: Data converter very slow ascii2Meatastock

    Hi Richard

    Thank you for the valuable infor.. I have turned off the virus checker and it improve the speed dramatically.

    I have 1 year of ascii data each day is a separate file that has around 2300 securities. IS that what you mean by grouping. I didnt follow your argument.

    With regards to maintaining the database, My provider is the one giving the ASCII files so i assume its already done by the data provider

    cheers

  •  12-31-2007, 7:12 26189 in reply to 26187

    Re: Data converter very slow ascii2Meatastock

    Security programs can definitely slow the process down as you noted.  What Richard was suggesting is that all the data and all the securities reside in a single file so that The DownLoader only has to open one file to convert everything.
View as RSS news feed in XML