aitcom-collector — the AIT Community data collector

AIT Community is a non-profit community for people who build with AI. Our members use the data collector to gather public information for their own research, such as the posts of a feed or the repositories of an open-source organization.

If you saw this address in your server logs, one of our members asked us to read a public page on your site.

• It reads your robots.txt first and follows it.
• It visits each site at most once a second, counting all our members together.
• When your site asks it to slow down (429 or 503), it waits, and it stops after the third time.
• It only reads public pages over https. It never logs in and never fills in forms.
• It always tells you who it is (see below).

aitcom-collector/1.0 (+https://aitcommunity.org/collectors/about)

Add this to your robots.txt to keep the collector away from your whole site:

User-agent: aitcom-collector
Disallow: /

You can also block only part of your site, for example Disallow: /private/.

Prefer that we never visit your domain? Email info@klevox.com with your domain name. We add it to our block list, which also covers its subdomains.