Skip to content

Commit f91c9b0

Browse files
committed
Add a few more unwanted agents to robots.txt.
[skip ci]
1 parent 54ce097 commit f91c9b0

File tree

1 file changed

+14
-0
lines changed

1 file changed

+14
-0
lines changed

robots.txt

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,16 +1,30 @@
1+
User-agent: AI2Bot
2+
User-agent: AmazonBot
3+
User-agent: Applebot
14
User-agent: Applebot-Extended
25
User-agent: Bytespider
36
User-agent: CCBot
7+
User-agent: ChatGPT-User
48
User-agent: ClaudeBot
9+
User-agent: cohere-training-data-crawler
510
User-agent: Diffbot
11+
User-agent: DuckAssistBot
612
User-agent: FacebookBot
713
User-agent: Google-Extended
814
User-agent: GPTBot
915
User-agent: Kangaroo Bot
1016
User-agent: Meta-ExternalAgent
17+
User-agent: Meta-ExternalFetcher
18+
User-agent: NovaAct
1119
User-agent: omgili
20+
User-agent: Operator
21+
User-agent: OAI-SearchBot
22+
User-agent: PanguBot
23+
User-agent: PerplexityBot
24+
User-agent: Perplexity-User
1225
User-agent: Timpibot
1326
User-agent: Webzio-Extended
27+
User-agent: YouBot
1428
Disallow: /
1529

1630
User-agent: *

0 commit comments

Comments
 (0)