What it is

KommioBot is the crawler of Kommio's knowledge base. A support team using Kommio can add a website as a knowledge source, so that their agents and AI assistant can answer from its pages. The crawler then fetches those pages on that workspace's request, when the source is added and when the team refreshes it. It does not roam the web: every site it reads was added by a Kommio customer, usually their own site or documentation, and it stays on the host it was given.

How it behaves

It reads robots.txt first and follows the group addressed to KommioBot, else the * group, including Allow, Disallow (longest match wins, Allow on a tie) and Crawl-delay. A robots.txt that cannot be read because of a server error stops the crawl.
It pauses 300 ms between requests; a longer Crawl-delay wins, up to five seconds.
It stays on the start page's host, follows at most five redirects on that host, reads at most 50 pages by default (500 at most, as the workspace sets) up to three link hops (ten at most), and reads at most 2 MB of a page.
It skips pages marked noindex, and binary links. It keeps the main text of a page for search; it does not run scripts or submit forms.
It never fetches private network addresses unless the Kommio operator has allowed them.

How to block it

Add a group for its user agent to your robots.txt:

User-agent: KommioBot
Disallow: /

To keep it out of part of a site only, list those paths instead of /; to slow it down, add Crawl-delay: 5. A change takes effect on the next crawl. You can also refuse requests whose User-Agent header starts with KommioBot at your web server or CDN.

Contact

If a Kommio customer is crawling a site they should not, or you have any other question about this crawler, write to [email protected] with the site and the time of the requests, and we will follow it up with the workspace.