# Allow crawling of all content User-agent: SemrushBot Disallow: / User-agent: SemrushBot-SA Disallow: / User-agent: AhrefsBot Disallow: / User-agent: Claudebot Disallow: / User-agent: DotBot Disallow: / User-agent: Barkrowler Disallow: / User-Agent: ImagesiftBot Disallow: / User-agent: * Disallow: