The program that reads a law firm's own website for Burt.

What BurtBot is

What it's for

Burt is AI intake chat for personal-injury law firms. BurtBot reads the text on a firm's own website so Burt's chat assistant can answer visitors' questions about that firm — things like practice areas, office locations, and hours — using the firm's own words.

When it reads a site

BurtBot reads one website per law firm that uses Burt: the site the firm told Burt is its own. It reads it when the firm starts a paid plan, when the firm asks for a read, and about once a week after that (sometimes less often) to keep the answers up to date. It does not wander: it only reads pages on that one site. If a firm pointed BurtBot at a site that is not theirs, email us (see below) and we will stop it.

Its user agent

BurtBot identifies itself with this exact User-Agent string on every request, so it's easy to spot in your logs:

BurtBot/1.0 (+https://hiburt.com/burtbot)

How it behaves

It follows robots.txt

robots.txt is a small file a site can publish to tell automated readers which pages they may and may not visit. BurtBot checks it before reading anything else. If your file has a set of lines (a "group") that starts with User-agent: BurtBot, that group applies; otherwise BurtBot follows a * (everyone) group if there is one.

If robots.txt can't be reached

If your site has no robots.txt (a "not found" answer), BurtBot treats that as "no rules" and may read the site, like other readers do. If BurtBot cannot load the file at all (your server returns an error code in the 500s, times out, or cannot be reached), it treats that as "do not read this site" and stops without reading any page. It tries again on the next read.

It stays on the firm's own site

BurtBot only ever reads pages on the website address a firm gave Burt (or, if it gave none, the website where it installed Burt's chat) and its www. version — for example example-law.com and www.example-law.com. It never follows a link off that site, and does not read any other site.

It reads one page at a time

BurtBot fetches one page, waits at least one second, then fetches the next. If the site's own robots.txt asks for a longer wait (a Crawl-delay), BurtBot waits that long instead, up to 10 seconds. A shorter wait is ignored — BurtBot never goes faster than one page a second. One read fetches at most 100 pages, plus your robots.txt and sitemap files.

It reads text, nothing else

BurtBot keeps the plain text of a page and the links to other pages on the same site, and throws away the page's underlying code once it has that text. The text is used to draft answers for that one firm's chat; the firm checks them before they go live. It never logs in, never fills out or submits a form, and never runs a page's JavaScript — if a page's content only shows up after JavaScript runs, BurtBot can't read it and simply notes that it couldn't.

Block BurtBot

Add a group for BurtBot to your site's robots.txt. This example blocks it from your whole site:

You can also block part of your site: BurtBot reads Allow and Disallow lines, including * and a trailing $, and the longest matching rule wins. If your file has neither a BurtBot group nor a * group, BurtBot reads the site.

Even when blocked, BurtBot will still fetch your robots.txt now and then to check the rule. It reads nothing else.

Questions, or ask us to stop

If you have questions about BurtBot, or you want it to stop reading your site, email hello@hiburt.com with your site's address. A person will turn off reads of your site, usually within one business day. A robots.txt rule works on its own and takes effect at the next read.