Skip to content Skip to navigation

January 2008

Go Crawl Yourself


First off, happy new year, blah, blah, blah, [insert navel-gazing year-in-review/big-plans-for-2008 post here]. There, that's done.

At the moment I'm working on some post-relaunch tweaks for Gothic BC and was looking at the logs. In between the people who think I get up at 6:00 a.m. after getting in from the club at 5:00 a.m. and am going to magically download and catalogue 400 photos off my camera, weed through them for the 150 or so worth posting, processes them, and have them posted on the site by 6:15 a.m. (reality: I got up about an hour ago and the camera is still in the bag) who are already scouring the site for last night's pictures, there are a gazillion hits from someone crawling the site for video content with a VEOH client.

This is uncool. First off, there is no video content on Gothic BC, nor will there be for some time to come, if ever. Secondly, as I understand it VEOH is effectively a P2P service which would mean I could potentially have dozens of these clients crawling the site effectively amounting to, given my limited bandwidth, a DDoS attack. No thanks.

Apache rewrite module to the rescue:

RewriteCond %{HTTP_USER_AGENT} veoh [NC]

RewriteRule .* http://www.veoh.com/ [F,L]


Pages