Robots.txt monitoring and alerts

Tiaki has built in robots.txt-monitoring and alerts. When we detect an issue with your robots.txt we double-check it and then notify you via email or SMS (or other channels of your choice).

Robots.txt checks

Disallow all

The most common mistake is to add the code Disallow: / to the robots.txt. This of course results in blocking search engines. We notify at once so you can fix it before Googlebot stops crawling your site.


Adding UTF-8 BOM (Byte order mark) to robots.txt is a common mistake and we will notify if we find it. To learn more read this help article on robots.txt.

Other errors

We are always looking to add more checks as we learn about new issues. More features to come.

Try robots.txt monitoring with Tiaki