Say you’re writting a Javascript or Flash enabled web application that has the browser send off HTTP requests occasionally. What can you do when a bug in your application causes clients to send a constant stream of HTTP requests to your poor server?

I recently ran into this type of problem and had to figure out how keep our servers from being overloaded by a number of clients that were sending a constant stream of requests to our servers. Lucky for us, the URLs that were being requested by these “rogue” clients were easily distinguished from “valid” requests.

The first step is finding some way to intercept the requests and do something different with just those requests. For my case, these rogue requests were all being handled by a 404 page, which was a PHP script. My first idea was to delay the response sent to these rouge clients, by just adding a sleep of a few minutes to this PHP page. This worked as advertised since the rouge clients were waiting until the each request was completed before sending another request, but I was still looking for something a bit better.

My second idea was to try altering the response for these requests to be a ’301 Moved Permanently’ response instead of a ’404 Not Found’. My hope was that if I specified a location of say ‘http://localhost/’ that all subsequent requests destined for these rouge URLs would instead be sent to the localhost, thereby no longer bothering our servers with these silly requests. Unfortunately, this didn’t seem to work for me in my tests.

My final and satisfactory solution was to send some special cache headers in response to these rogue requests. I added a few lines found in an example in the ever useful PHP documentation. These lines told the Flash script to cache the URL they just requested for 30 days. Subsequent requests for these URLs were just looked up in the local cache by the Flash script, stopping the steady stream of requests sent to our servers for these worthless URLs. Success!

The library is great in that they have a ton of books. I think they have all or most of the O’Reilly books, which is key. However, the drawback is that the site itself is pretty horrible. It is horrible from a user interface perspective and performance wise. Every single action triggers a request to their super slow servers. Several actions require a few too many clicks which translate into waiting for their servers to respond. They should allow users to control page navigation simply by using the keyboard. They also fail to do the basics like remember what font setting I used last. Needless to say the site could use a major overhaul to bring it into the 21st century.

Lately in my spare time, I’ve started trying to add the features to Nutch that would allow us to use it. It’s fun. I recently submitted my first patch to the Nutch developers list. Hopefully I did everything well enough to get it commited to CVS. This patch allows users to specify Perl 5 regular expressions, which will get applied to all URLs that Nutch encounters. It’s useful for stuff like stripping out session IDs in URLs.

I’ve got a few more features that need to be added. I found another drawback to the way the crawler for Nutch was written. You can specify any number of threads to be running at the same time. However, currently it won’t allow two different threads to download from the same IP simultaneously. This is not good considering all of our websites look to the crawler as just 1 IP. I’ll probably have to make some changes there. Hopefully it’ll be relatively straightforward and easy.

Cool use of Nutch: Creative Commons Search (via: Doug)