Search engines: Googlebot, Bingbot, YandexBot, Baiduspider, DuckDuckBot, Applebot, Qwantbot, SeznamBot, Sogou.
AI & LLM crawlers: ChatGPT-User, GPTBot, Google-Extended, ClaudeBot, Claude-Web, PerplexityBot, Cohere, Anthropics, OAI-SearchBot, Quillbot, YouBot, MyCentralAIScraperBot.
SEO tools: AhrefsBot, SemrushBot, MJ12bot, DotBot, DataForSeoBot, Awario bots, SEOkicks, Botify, Jetslide, peer39.
Social & sharing: facebookexternalhit, FacebookBot, Twitterbot (X), Pinterestbot, Slackbot, Meta external fetchers.
Security & cloud: AliyunSecBot, Amazonbot, Google-CloudVertexBot and more.
Scrapers & research: BLEXBot, Bytespider, CCBot, Diffbot, DuckAssistBot, EchoboxBot, FriendlyCrawler, ImagesiftBot, magpie-crawler, NewsNow, news-please, omgili, Poseidon Research Crawler, Quora-Bot, Scrapy, SeekrBot, SeznamHomepageCrawler, TaraGroup, Timpibot, TurnitinBot, ViennaTinyBot, ZoomBot, ZoominfoBot.