pfindBot

pfindBot is the crawler of pfind.pro, a search engine for parts of industrial and heavy equipment. It reads the product pages of parts sellers, so that a buyer who searches for a part number finds the offer and goes to the seller's own page to buy.

What it fetches

Product pages, product feeds and sitemaps of parts sellers, and robots.txt. Product images are fetched by pfindBot-Image, to show a thumbnail next to the link to the seller. pfindBot never places orders, fills in forms or logs in.

How it behaves

It obeys robots.txt, including Crawl-delay, and re-reads it at least once a day. It requests pages at a modest pace and slows down when a site answers 429 or 503.

How to recognise it

The user agent contains pfindBot/1.0 (+https://pfind.pro/bot) for pages and pfindBot-Image/1.0 (+https://pfind.pro/bot) for images.

How to limit or block it

Add a group for pfindBot to your robots.txt. For example, to keep it away from the whole site:

User-agent: pfindBot
Disallow: /

A group for pfindBot also applies to pfindBot-Image, unless you give the image crawler a group of its own. The change takes effect within a day.

Contact

Questions or a problem with pfindBot: [email protected].