du willst einen eigenen webcrawler?
dann kannst du den alturo root server für 19,99 aber vergessen..
da brauchste schon ne hellmachine
am besten bei www.1st-housing.de oder www.ipx-server.de zusammenstellen
Hier verschiedene Crawler / Spider / Suchmaschinen...
Heritrix: Internet Archive Web Crawler
--> <a href='http://sourceforge.net/projects/archive-crawler/' target='_blank'>http://sourceforge.net/projects/archive-crawler/</a>
The archive-crawler project is building a flexible, extensible, robust, and scalable web crawler capable of fetching, archiving, and analyzing the full diversity and breadth of internet-accesible content.

PHPCrawl
--> <a href='http://sourceforge.net/projects/phpcrawl/' target='_blank'>http://sourceforge.net/projects/phpcrawl/</a>
PHPCrawl is a class written in PHP for crawling/spidering websites. It supports filters, limiters, cookie-handling and other features. By overriding a special method of the class users can decide what should happen to the data the webcrawler finds.

WebSPHINX
--> <a href='http://sourceforge.net/projects/websphinx/' target='_blank'>http://sourceforge.net/projects/websphinx/</a>
WebSPHINX is a web crawler (robot, spider) Java class library, originally developed by Robert Miller of Carnegie Mellon University. Multithreaded, tollerant HTML parsing, URL filtering and page classification, pattern matching, mirroring, and more.