Have you ever wanted to be able to know for sure if a web spider from Google, Yahoo, Bing etc. has crawled your site? Sure you have! But web spiders these days are getting intelligent and are not leaving a sure footprint of who they are. Here at CrawlerSpy we take a Community approach taking non-identifying aggregate information of our users who decide to install our Wordpress Plug-in, and perform advanced heuristic analysis and when we detect a possible spider we send information on that spider to a central server for further analysis. Only when we are confident we have successfully detected a spider do we add that ip to our master database.
Access our Database how you require. Available formats are CSV, Flat Text which are updated every 2 hours, or use our realtime lightning fast RESTful API.
Do you run a Wordpress Blog? Save money on a CrawlerSpy subscription by installing our Wordpress plugin. Our Wordpress plugin performs a quick analysis of your visitors and decides is it is a web spider. Only if a spider is detected CrawlerSpy Servers are notified of the visitors ip address and submitted for more detailed analysis.
The CrawlerSpy Wordpress plug-in is designed to be unobtrusive, use zero resources and completely undetectable by your visitors and spiders, and will not impact the performance of your web site.
Download and Install