Checking on Jakarta Composite Index (JCI) and other stock exchange indexes, and then writing them down, can be a tedious process. Luckily Yahoo Finance provides a readily downloadable information in CSV format. So, for example, for JCI you can download the latest information here.
I am interested in this kind of data for playing around, for example, using them for data visualisation. I thought Yahoo Finance will have RSS feeds, but it turned out the feeds are only for news. I also thought that IFTTT couldn’t download files (turned out it can, but not really suitable for my purpose). So I turned to cron and curl to automate the job. I write down the process here mainly as note to myself (I don’t usually write shell scripts).
The idea is to download the file at particular time (I choose 4.30pm) every working day. This means this kind of entry in crontab:
30 16 * * 1-5 sh /your/path/to/home/directory/bin/download-jkse.sh
This means the script download-jkse.sh will be executed at 16.30 every day of the month, every month, but only from Monday to Friday (working days). Next step is to write the script itself.
The main work is done by curl, which downloads the appropriate file at chosen time:
curl "http://download.finance.yahoo.com/d/quotes.csv?s=%5EJKSE&f=sl1d1t1c1ohgv&e=.csv" >> /home/gombang/JKSE/jkse.csv
The URL is quoted, because the special characters in the URL are playing havoc with curl. We only want one file at all times, so we concatenate the downloaded file to the end of a CSV file (jkse.csv), using operator
That’s should be the only line in the script, if only I had got an access to an always online server. I don’t, so I have to check whether my laptop is fully connected to Internet. I do this using
nmcli, the command line client of NetworkManager. If we are not connected, wait for 5 minutes, and check again. If connected, download the file.
The full short script turned out like this:
while [ $(nmcli networking connectivity) != "full" ]; do
curl "http://download.finance.yahoo.com/d/quotes.csv?s=%5EJKSE&f=sl1d1t1c1ohgv&e=.csv" >> /your/path/to/home/JKSE/jkse.csv
This added the latest data to file jkse.csv everyday at 4.30pm, or later, if at the time we are not connected, or (thanks to anacron), we are not up yet. This script is easily extended to download additional files like, for example, data of another stock exchange. We may need to check whether a particular day is a holiday (then don’t download any data), but then it maybe easier just to clean up the csv file later.
- Introduction to cron
- Bash programming howto