bieh.net

I shamelessly ripped off someone elses Greasemonkey script and now have RSS auto-discovery on all Trademe pages backed off the previous TM RSS+search stuff. The script can be found here:

Trademe.co.nz RSS

Because I feel guilty not posting an image in several consecutive posts, here’s a screenshot with a picture of what I mean:

Note that of course you will need the Greasemonkey extension for Firefox installed to use this.

EDIT: Updated it to not show (non-working) RSS feeds on auction pages, MyTradeMe, Sell, and Community sections.

EDIT #2: Bumped this post to show up on planet nztech, as I guess the readers of that are exactly the people who’d like this.

Apparently I’ve been added to Planet NZTech. Let’s see if it works.

EDIT: Apparently it does

The trademe search stuff has had a few updates applied:

• It now has all the same common search / filter options as the TM pages (ie: filter on buy-now, new vs used, etc)
• Various nasty bugs fixed, so it might actually work properly now
• It looks a little nicer, too (though still pretty ugly). It now also features a disclaimer, which we figured was probably a good idea

Enjoy.

EDIT: If you want to use this, use the Greasemonkey script. It’s just easier

So after my last post, we spent some time thinking about how the system could become more useful. We ended up deciding the best thing we could do would be to use the parser and RSS renderer to help build a better search system for Trademe listings.

The current TM search system kinda works, but I’ve always felt it was a bit limited – for example, if I search for ‘iPhone 3G’, the majority of the results are for cases and pouches. Sure, you can drill down to the category to get rid of a lot of the cruft, but people still insist on posting phone cases in the category that should be reserved just for actual phones.

This proved to be annoying when using my RSS feed generator, as I was constantly getting alerts that a cheap phone had turned up, and it turned out in most cases to be some sort of useless accessory.

So, we took the afternoon off and built this:

Trademe Filtered RSS Feed Generator Search Tool

(yeah, the name needs work).

Basically the way it works is:

• You paste in a URL or provide some search terms to build a URL
• We fetch the page and parse it to automatically pull out all the attributes (title, buynow, photo, etc) in the auctions
• A web form is then generated that lets you select filters on those attributes.
• Behind the scenes it’s building up a simple little query ( something like “buynow < 1000 | size = 42″ ) based on what you select in the form. This is then run over the page to extract the listings that match
• The listing objects are rendered to RSS same as before, and you are returned a URL that you can use in a feed reader as normal.

It’s actually kind of neat – it autodetects most of the attributes, and you can build up quite complex queries fairly easily:

• Cars for sale
• Houses for rent
• LCD TVs

As a good example of how this is useful, I’ve always found that in many cases, auctions without photos tended to be a bit dodgy when you’re looking for furniture and so on. Now you can filter on this: Bookshelves with pictures, Bookshelves without pictures.

The previously-mentioned ‘iPhone, but no cases’ search works nicely too with the excluded words feature. Another tactic there would be to filter to remove auctions with a buy-now / bid price less than some amount. However, that would mean if someone posted one for $1 by accident I’d miss out

Anyway, I’m sure there’s still a bunch of bugs and such in it. Feel free to post them here and I’ll see what I can do.

UPDATE: OK, so there were a couple of minor teething troubles, mostly to do with me breaking the buy-now option. That’s now fixed, I hope. I did some more work on the parser, and it should work properly for Property, Motors, and kinda sorta works for Jobs. TMJobs doesn’t look like it was made by the same people as the rest of it, so the names and everything are different, causing it not to play nice with the rest of my system.

Original Post: Today I got annoyed at my local auction site Trademe for not offering RSS feeds of anything. What I wanted was feeds notifying me of new search results and listings in categories – yes, TM offer email notifications for both these cases, but I spend a lot more time in Google Reader than I do in my email client. That, and their email only comes once a day, whereas RSS can be updated a lot more frequently.

I also wanted to (optionally) only be notified when a buy-now was available below some amount. Basically I want to see cheap stuff before anyone else has a chance to notice it, but I don’t want to do any of the work normally associated with that (ie: hitting F5 a lot)

Trademe do have a fairly limited public API. However, the closest I could find to what I wanted was this post, which wasn’t quite what I was after.

So I wrote a small script that downloads a trademe URL and extract the relevant information into a simple computer-readable format. Then I wrote another one that reads that format and renders it to RSS. After that I made it into a CGI. Then I needed a simple way of building up the URLs to the script with the correct escaping and parameters so I ran up a quick webpage for it. Then people complained that the webpage looked awful and kept asking the same questions, so I made the page a bit more complicated. It was a bit of a slippery slope, if I’m to be honest…

Anyway, I’ve put up a copy of it on this server to act as a stable and theoretically usable version. If anyone wants to use it, it’s here:

Trademe RSS Generator

I’m quite happy with the end result. By using feeds like this one, I automatically get notified of cheap PS3 games, for example

If anyone has any other suggestions as to what I could do with live machine-readable data of TM listings, feel free to leave a comment below.

My roads (mostly) now form automatic intersections when I place them. This makes it nice and easy to do city-block style grids.

I just need some traffic lights and I’m set

Just because I could, the Universe client now takes a screenshot when it exits, and pushes it to my server. It’s then displayed on the right, with a bigger copy below:

This picture should change pretty often when I’m doing active development, but it might look kinda strange if I’m squashing bugs

I implemented some initial support for procedural cities tonight. At the moment this consists of one static ‘building’ made out of some primitives, and a road system (which is what I spent most of my time playing with). While this doesn’t look particularly interesting at the moment, I now have most of the groundwork laid to do generate more complex buildings

I now have some initial support for multiple planets, and different kinds of planets. So, I made a ‘moon’. It looks kinda neat, even in it’s current semi-broken state.

It looks pretty nasty from height, though. Because there’s basically only one texture and no distance fog, you can see the squares of the quadtree quite clearly. But I’ll leave that for another night.

Tonight I spent some time implementing a bunch of overdue optimizations and fixes to the terrain engine. Basically all the boring stuff I’d been putting off because it was going to be painful without much payoff

It was totally worth it, however. FPS is way up, CPU usage is down, and the annoying popping artifacts are, if not gone, at least reduced. A bunch of things still need working on, but it’s a nice step forward.

None of these changes make good screenshots, but I’d hate to lose my streak of screenshot posts. So, enjoy this shot of the improved system looking pretty much the same as it did before.