PHP Classes

Spider website: Crawl a site and retrieve the the URL of all links

Recommend this page to a friend!
     
  Info   View files Files   Install with Composer Install with Composer   Download Download   Reputation   Support forum   Blog    
Ratings Unique User Downloads Download Rankings
StarStarStar 45%Total: 3,048 All time: 1,198 This week: 488Up
Version License PHP version Categories
spider 0.1GNU General Publi...5.0HTML, PHP 5, Searching
Description 

Author

Karol Janyst

This class can be used to crawl a site and retrieve the the URL of all links.

It can retrieve a page of a site and follow all links recursively to retrieve all the site URLs.

The class can restrict the crawling to URLs with a given extension and avoids accessing pages listed in the site robots.txt file, or pages set with the no index or no follow meta tags.

Picture of Karol Janyst
Name: Karol Janyst <contact>
Classes: 2 packages by
Country: Poland Poland

 

  Files folder image Files (2)  
File Role Description
Plain text file spider.class.php Class Main class file
Accessible without login Plain text file example.php Example Example file

The PHP Classes site has supported package installation using the Composer tool since 2013, as you may verify by reading this instructions page.
Install with Composer Install with Composer
Downloadspider-2009-06-08.zip 3KB
Downloadspider-2009-06-08.tar.gz 3KB
Install with ComposerInstall with Composer
 Version Control Unique User Downloads Download Rankings  
 0%
Total:3,048
This week:0
All time:1,198
This week:488Up
User Ratings User Comments (2)
 All time
Utility:62%StarStarStarStar
Consistency:78%StarStarStarStar
Documentation:-
Examples:65%StarStarStarStar
Tests:-
Videos:-
Overall:45%StarStarStar
Rank:3319
 
I ran a simple test using this class.
14 years ago (Oliver Lillie)		22%StarStar
It's got great potential, but.
15 years ago (F Philip DeGeorge)		55%StarStarStar
 

  Applications that use this package  
No pages of applications that use this class were specified.

Add link image If you know an application of this package, send a message to the author to add a link here.