There are a number of ways to download free daily data, however I believe most of them involve software (e.g. MLDownloader, HQuotes, MetaQuotes) that makes one request at a time (for each symbol) to web sources like Yahoo or MSN. Am I correct?

This would be an extremely inefficient way of downlading data for entire exchanges and multiple exchanges each day. Making almost 10,000 HTTP requests (for which you need to wait for 10,000 responses back from the server) to download around a mere 500KB of data, and to do this every weekday, is absurd.

Is there any way anywhere to simply specify that you want to download all stock data of an exchange of a single day and have it done in one round trip to the server?

On further research of Yahoo, I've found that Yahoo can accept a maximum of 200 symbols at a time. This means for a list of 10,000 symbols, there would be 50 round trips to the server to download less than 1MB of data.

How do the quote download programs work with a list of 10,000 symbols? Do they submit 200 symbols at a time, or 1 at a time?

I still think it would be much better and faster if it was possible to download all data in one go.

For the amount of time you are expending trying to find sources of free data, and the amount of time you are going to have to spend cleaning your free data, quality checking, dealing with splits and consolidations, securities no longer trading etc, you are much better off just buying your data from a decent data vendor.

Have a read through the Forum on this issue and see all the problems traders have had when trying to save a couple of hundred dollars by downloading "free" data which ends up costing them extra hours per day, every day, in checking and re-formatting, or worse, people who lost much more than just a couple of hundred dollars because they made trades based on bad data.  If you are going to trade, you had better make sure the data on which you are basing decisions is, within reason, as accurate as possible.

Many data vendors also have neat tools that allow fast download and distribution in a single mouse-click. <edit> or even allows you to automate the process so that you don't have to do anything, you just wake up and all the overnight information has already been downloaded and distributed.  For an example of what services a good data vendor offers (including a free trial), see Richard Dale's services at http://www.premiumdata.net/. </edit>


Hope this helps.

wabbit

---

MS: 6.52 EOD, 7.x EOD, 8.0 PRO, 9.2 PRO w/QC, 10 PRO w/QC & MDK
For custom MetaStock programming : http://www.wabbit.com.au
My SkyPE status :
My SkyPE account : wabbit.com.au

Hi Wabbit, you must feel like a broken record each time you reply to a message about free data!

I'm aware of the differences between free data and data that costs money.

One major advantage of using data from Yahoo is that they provide data of over 50 international exchanges.

You may not be interested in trading on other exchanges but there are probably many others who are, and the data is free to download and play with before deciding to get serious and paying for good historical and daily data of a particular exchange. It's good to see how strategies that perform well in one market perform in other markets.

Now if there was a fast and easy way to download the historical data and daily updates of all stocks in all exchanges then that would be superb.

Even during the "learning" process, if you test your systems against dodgy data, it doesn't matter what exchange it comes from, you are going to get bogus results.  Even during system development and testing, it is imperative to have clean,  reliable data.


wabbit


Post Script:
There is a belief that good trading comes from a good education about the markets in which we are trading.  Many traders spend long hours every day trying to learn as much as they can about one market, let alone trawling through 50 or more international markets.  Within the market they are trading, many people have to limit their universe of tradable stocks because they know they cannot possibly watch the entire market place; it would be virtually impossible to gain an understanding of every stock on a single exchange, let alone every exchange.

A lot has been written about data mining and over-optimisation of systems to fit particular historical situations.  Needless to say, just about all of it is negative.  I would guess there are only a few successful traders who have successfully fitted their trading rules to multiple markets.  I would guess the number who have tried and failed to be the majority.

---

MS: 6.52 EOD, 7.x EOD, 8.0 PRO, 9.2 PRO w/QC, 10 PRO w/QC & MDK
For custom MetaStock programming : http://www.wabbit.com.au
My SkyPE status :
My SkyPE account : wabbit.com.au