lukebaker.org » Nutch http://lukebaker.org lukebaker.org Tue, 08 Mar 2011 22:39:20 +0000 en hourly 1 http://wordpress.org/?v=3.3.1 Nutch and Bittorrent http://lukebaker.org/archives/2005/05/25/nutch-and-bittorrent/ http://lukebaker.org/archives/2005/05/25/nutch-and-bittorrent/#comments Thu, 26 May 2005 00:31:11 +0000 Luke http://lukebaker.org/archives/2005/05/25/nutch-and-bittorrent/ Cool. The new bittorrent search engine uses Nutch.

]]> http://lukebaker.org/archives/2005/05/25/nutch-and-bittorrent/feed/ 0 Nutch Shorterm Goals http://lukebaker.org/archives/2004/09/07/nutch-shorterm-goals/ http://lukebaker.org/archives/2004/09/07/nutch-shorterm-goals/#comments Tue, 07 Sep 2004 23:18:44 +0000 Luke http://lukebaker.org/archives/2004/09/07/nutch-shorterm-goals/
  • Ability to use regular expressions for URL substitutions.
  • Allow users to to search using url:Store/View/Product/1001
  • Faster crawling of websites that look like one (1) IP address.
  • Some sort of templating engine for creating search results pages. Maybe use Velocity?
    • ]]>     http://lukebaker.org/archives/2004/09/07/nutch-shorterm-goals/feed/ 2     Nutch patch #1 http://lukebaker.org/archives/2004/09/06/nutch-patch-1/ http://lukebaker.org/archives/2004/09/06/nutch-patch-1/#comments Mon, 06 Sep 2004 21:09:10 +0000 Luke http://lukebaker.org/archives/2004/09/06/nutch-patch-1/ At work I was told to investigate other options for a search engine that would search just the sites that we host. While I was doing that I came across Nutch. It looked pretty sweet but not quite something that would fit our current needs. We needed a few more features. Currently at work we’re looking at a Google Search Appliance. It costs a pretty penny, but would be nice because hopefully that would be something we could just “set it and forget it.”

    Lately in my spare time, I’ve started trying to add the features to Nutch that would allow us to use it. It’s fun. I recently submitted my first patch to the Nutch developers list. Hopefully I did everything well enough to get it commited to CVS. This patch allows users to specify Perl 5 regular expressions, which will get applied to all URLs that Nutch encounters. It’s useful for stuff like stripping out session IDs in URLs.

    I’ve got a few more features that need to be added. I found another drawback to the way the crawler for Nutch was written. You can specify any number of threads to be running at the same time. However, currently it won’t allow two different threads to download from the same IP simultaneously. This is not good considering all of our websites look to the crawler as just 1 IP. I’ll probably have to make some changes there. Hopefully it’ll be relatively straightforward and easy.

    Cool use of Nutch: Creative Commons Search (via: Doug)

    ]]>     http://lukebaker.org/archives/2004/09/06/nutch-patch-1/feed/ 0