January 2010
3 posts
Marco's feed is not nice to the crawler
The crawler is trying to conserve as much bandwidth as possible and Marco’s verbose description field is throwing this effort off…
It’s clear this is being dumped from the Tumblr ‘Description’ field, but It’s also got his google analytics tracking code buried within which seems superfluous. Just sayin’.
Debugging crawler, progress slow
I’ve managed to create an entirely new, lean crawler however testing it has proved tricky. I’m juggling this work with paying work so its unfortunately taking a back seat to that at the moment.
Road to Recovery
Happy New Decade all.
So sorry for the long downtime and associated, unacceptable radio silence on my part. We are turning over a new leaf here at Rososo and hope to be up and running soon.
A bit of background on the crash:
Rososo has a crawler that has to go check your feeds every X minutes to find out when they have last updated. To do this, the previous crawl method was relying on 3rd...