bieh.net

So, a while back we spent a little while working on a system to generate and do some fairly advanced filtering on RSS feeds of Trademe search listings (available in user-friendly form here). It’s been pretty neglected since then, although we’ve fixed the odd bug and added a few tweaks here and there. Anyway, recently I had a reason to glance at the log files, and was suprised to see it was getting a fair number of hits – over 12k in April alone. Granted, a lot of this is due to the fact that some feed readers make requests far too often, but still

Because everyone likes graphs, here’s a more detailed breakdown:

Trademe RSS requests/day

(click the image or here for the raw data)

Note that these are only successful requests. If TM denied my request, or Apache decided to lock up out of spite, or (most likely) I was messing around with the script and broke it, it doesn’t get logged. I suspect that may account for the dips. Not entirely sure what happened with that big spike though.

Anyway, should anyone want it, the source is here. It’s fairly rough in places, be warned:

http://bieh.net/svn/tmscraper

Running ‘./scraper.py http://trademe.co.nz’ may be a good place to start.