Another update to the utility:
Added the ability to specify a symbol database location.
Added the ability to read types and groups from symbol databases 9.2 and later
Fills in Types and Groups combo boxes.

---

Equis International, Inc.
A Reuters Company
90 S. 400 W Suite 620
SLC, UT 84101

Just tried the Jan 12th file on my 9.1EOD database.  Because I'm paranoid, I set it to delete securities not traded for 2000 days.  As expected, it reported deleting 0 symbols.

But it didn't crash or report any errors and my database seems intact. So far, so good!  I think it'll be a handy utility.  As development progresses, of course, it would be nice to see some sort of progress indication to confirm it's working and help us estimate remaining processing time.  I have a WD Raptor RAID 0 drive setup, so it's pretty fast, but it would be useful to have that feedback.

I just thought of a useful (to me, anyway) addition to the program - instead of deleting files, how about the option to scan and prepare a report (printable, savable to file) of symbols meeting the input parameters?  Sometimes I have a "stuck" datafile that hasn't been updated in a long time due to a symbol change, typo etc. and this function would help spot those so I can attempt a fix.

I use ML Downloader with Yahoo data, and it catches most of these, but if the symbol isn't in MLD for some reason it won't be picked up.

---

When you see that in order to produce, you need to obtain permission from men who produce nothing - when you see that money is flowing to those who deal, not in goods, but in favors - when you see that men get richer by graft and by pull than by work, and your laws don't protect you against them, but protect them against you - when you see corruption being rewarded and honesty becoming a self-sacrifice - you may know that your society is doomed. - Ayn Rand

A progress bar isn't all that doable. I have no idea how much longer it will be since I don't yet know how many files and folders I have to search. Trying to find out before hand would make the process take twice as long (well, almost).
It should be easy enough to have it show somewhere what folder it is in and even what security it is currently looking at so you can watch the text fly by as it processes. That would probably be a feature I work on last since it is a nice feature and doesn't actually make it more usable.

What would I show in the list? The file location and symbol name?

---

Equis International, Inc.
A Reuters Company
90 S. 400 W Suite 620
SLC, UT 84101

I should have been a little clearer; generating a progress bar would require more overhead than it's worth.

If you could show file location, symbol name and the number of symbols processed so far (assuming updating this wouldn't cost too much processing time) it would be very useful.  After running it once or twice, we'll know approximately how many symbols it will process in total and can estimate overall progress.

If updating this display eats up too much time, maybe updating it once a second, or every 5 seconds would suffice.  Certainly doesn't have to be realtime or updated for every single symbol unless the overhead cost is trivial.

Thanks for plugging away at it.

---

When you see that in order to produce, you need to obtain permission from men who produce nothing - when you see that money is flowing to those who deal, not in goods, but in favors - when you see that men get richer by graft and by pull than by work, and your laws don't protect you against them, but protect them against you - when you see corruption being rewarded and honesty becoming a self-sacrifice - you may know that your society is doomed. - Ayn Rand

Haha! Cool. It was a lot less work than I thought to throw in that ability to display the currenty security being processed.
The program still doesn't respond to anything while it is processing, but now you can see at the bottom of the window what security it is working on in real time.

I threw in another feature I was working on in this build. It makes it so that the Browse dialogs will start at the folder that's in the db and security location text boxes so you don't have to dig down to that area.

I'm uploading the new version now.

---

Equis International, Inc.
A Reuters Company
90 S. 400 W Suite 620
SLC, UT 84101

Brandon: Thanks for writing this. I am going to download it now. Are there instructions on how to use it? Harry

Brandon: After reading the related thread on this subject, it is not clear what versions of Metastock can use this utility. I have version 8.01. Harry

Right now, it doesn't really matter what version of MetaStock you have. It will read your Types and Groups from your symbol database if you have version 9.2 or later, but doesn't yet do anything with that information. I only put a copy up that does that much with the database because I need people testing it and making sure it does what it should. Since you have version 8, your Type and Group shouldn't fill in.

When all the functions in the utility are done, it probably will only work with 9.2 and above. After I finish, I'll see how much more free time I have to do older versions.

The only function that currently really does anything is the delete securites X or more days old. To use that feature, all you have to do is specify the location of the securities you want to clean up (delete old ones from) and check the recursive box if you want it to search all the sub folders too. Then you enter in a number of days that you want to check for as not traded. If the security has that many days that it hasn't traded, it will be deleted when you click delete. It will show you a progress of what folder and security it is currently processing and at the end, it will tell you how many securities it deleted.

This utility seems to be working fine with the little testing it has received. Just remember that not all the functions are done (the ones that aren't done are disabled) and that it is compeletly unsupported.

---

Equis International, Inc.
A Reuters Company
90 S. 400 W Suite 620
SLC, UT 84101

I just wanted to say I still remember this project. :)

I'm sorry it is going so slowly.
I got a lot busier and it is a lot more difficult than I was hoping.

I may add a couple other options or make a new utility for various functions to get you guys something while I figure out how to make it read the symbol database the way I need it to.

---

Equis International, Inc.
A Reuters Company
90 S. 400 W Suite 620
SLC, UT 84101

Hi Brandan

Having read your post and checked out your utility out like to add a few more usability comments.  I download my end of day data using Reuters which notifies me of symbol changes, symbols that are not updated, and other errors.  My data files are spread across several folders with symbols appearing multiple times across those.  At present the method of deleting symbols that may be in several folders is cumbersome and very time-consuming, more especially since this needs to be done on a daily basis.  Likewise the method of changing symbols have been renamed it is also very time-consuming.  Occasionally I need to move a security from one folder to another and again this is very time-consuming.

In my humble opinion the downloader is woefully inadequate carrying out these daily housekeeping tasks since maintaining the data files is such an essential task.  Using a symbol list provided by metastock, is useful but only a starting step.  I would therefore like to suggest that a facility to simply find and delete sets of symbols, find a move set of symbols, and change of symbol tickers will be a great addition to the utility.  It seems to me that development of the downloader into a truly useful facility has been set aside for many years at the expense of improving the charting side of metastock, adding ever more indicators and so on.  There is little point in having all these one of all indicators if the data on which it is based is so hard to keep up-to-date.

I hope you'll given more time to expand and improve on your facility and I am definitely champing at the bit to try anything that makes maintaining data easier.  slipperx

That's basically what this program is intended to do.

You chose the Type (and optionally choose Group) and specify a folder. I creates any needed securities (anything new since the last time done, if ever).

You can then choose to delete any securities by either not existing in the symbol database or by having not traded for x days.

I haven't had time to get the symbol database functionality of the utility working, but everything that doesn't require the symbol database does work.

Between projects at work, home, and school, I haven't had time to finish this. I apologize and I do plan to get to it. The symbol database is just a lot harder to get in here than I had figured it would be.
I should maybe set aside a couple hours a week to work on this so at least it gets somewhere.

---

Equis International, Inc.
A Reuters Company
90 S. 400 W Suite 620
SLC, UT 84101

Hey Branden.....

Are you still working on this project?

A feature, which we may have already discussed, that would be extremely helpful, would be to review the collection report, process the "renames" and go out and rename the stocks.

Thanks

Dean

No, unfortunately, the symbol database reading code is so integrated into the DownLoader, I'd pretty much need to use all of the resources from the DownLoader in order to use it.
I've also been very busy again.
I'll look at it some more again in the future.
The current version that's still available will delete securities based on dates left not traded, but is unable to use the symbol database.

Good news for your question though, I am currently working on a utility to rename securities based on a CSV file.
You specify CurrentSymbol,NewSymbol in a CSV file and the utility will go through the specified data folder(s) and change the symbols.
It should be out about the first week of October.
I'll look into if I'm able to see the file that has the security change notifications from DownLoader and allow that to be a file that can be used.

---

Equis International, Inc.
A Reuters Company
90 S. 400 W Suite 620
SLC, UT 84101

Hi Branden, nice utility to remove symbols from your local DB that have not been updated in a while. I just downloaded it and really appreciate your hard work! I have a set of custom folders setup nothing fancy (Amex, Nasdaq1, Nasdaq2, Mutual Funds...under my Metastock Data folder..),but it does have a folder named Mutual Funds. I assume it will just ignore these symbols when looking for stocks to delete? Please clarify.

Also I use the Downloader to receive all the stocks/mutual fund data on the exchanges and average about 20 symbol changes daily. I do these manually and I assume others do the same. Are there any plans to create a utility to help with this? I pull about a ten year history of prices from stocks and mutual funds and it helps me to have this quasi custom folder list mentioned before instead of the list Equis offers, which if I understand would allow me to just update new stocks symbols but then I would have to download the 10 year history for all stocks/mutual funds as it purges the old list out. May be too much time involvled in a new download. I guess my question is have you heard of any one trying to solve this by creating an automatic system to keep your data up to date?

If not what's the most efficient way of keeping up to date with symbol changes? 

It does nothing with the symbol database. The idea of this utility was to use the symbol database to take care of add, removing, and renaming files, but turned out the code for the symbol database is just far too complex to put into the utility.

It doesn't look at the symbol when deleting based on x days not traded. So it wouldn't matter if it was stock or mutual fund. You provide a main folder and check if you want it to look at all the folders in that folder as well.

---

Equis International, Inc.
A Reuters Company
90 S. 400 W Suite 620
SLC, UT 84101