About our crawler

briefingbitesBot/1.0 (+https://briefingbites.com/bot) is the content crawler for briefingbites, a briefing service for accounting firms. It reads public news pages and RSS feeds from a small, fixed list of tax and accounting publishers so that we can summarize and link to their reporting.

How it behaves

• It checks and obeys robots.txt on every host it scrapes; disallowed hosts are skipped entirely.
• It makes at most one request per second per host, with low overall concurrency.
• It honors ETag and Last-Modified headers to avoid refetching unchanged pages.
• It never attempts to bypass paywalls or logins.
• Our briefings quote at most one short phrase per article and always link the original.

Publisher requests

If you'd like us to stop crawling your site, adjust our rate, or discuss anything else, email support@briefingbites.com and we'll act on it promptly — or disallow briefingbitesBot in your robots.txt, which our crawler obeys automatically.