PHP-Spider Features :

• supports two traversal algorithms: breadth-first and depth-first
• supports crawl depth limiting, queue size limiting and max downloads limiting
• supports adding custom URI discovery logic, based on XPath, CSS selectors, or plain old PHP
• comes with a useful set of URI filters, such as Domain limiting
• supports custom URI filters, both prefetch (URI) and postfetch (Resource content)
• supports custom request handling logic
• comes with a useful set of persistence handlers (memory, file. Redis soon to follow)
• supports custom persistence handlers
• collects statistics about the crawl for reporting
• dispatches useful events, allowing developers to add even more custom behavior
• supports a politeness policy
• will soon come with many default discoverers: RSS, Atom, RDF, etc.
• will soon support multiple queueing mechanisms (file, memcache, redis)
• will eventually support distributed spidering with a central queue