爬虫识别支持 IPv6 地址访问 了解详情
ClaudeBot 是由 Anthropic 运营的网络爬虫,用于下载其 LLM(大型语言模型)的训练数据,为 Claude 等 AI 产品提供支持。
根据行业标准,Anthropic 使用各种数据源进行模型开发,例如通过网络爬虫收集的来自互联网的公开数据。作为 Anthropic 构建安全可靠的前沿系统和推动负责任的人工智能开发领域的使命的一部分。
ClaudeBot 搜集收集数据的原则:
Anthropic 的数据收集应该是透明的。用户代理令牌 ClaudeBot 标识了 Anthropic 的通用网络爬虫。
Anthropic 的抓取不应具有 侵扰性 或 破坏性 。Anthropic 的目标是通过深思熟虑地考虑抓取相同域的速度并在适当的情况下尊重抓取延迟来将干扰降到最低。
根据行业标准,Anthropic 使用各种数据源进行模型开发,例如通过网络爬虫收集的来自互联网的公开数据。作为我们构建安全可靠的前沿系统和推动负责任的人工智能开发领域的使命的一部分。
ClaudeBot 遵守 robots.txt 协议,如果需要屏蔽整个网站禁止 ClaudeBot 抓取写法如下:
User-agent: ClaudeBot
Disallow: /
如果需要延缓 ClaudeBot 抓取速度,写法如下:
User-agent: ClaudeBot
Crawl-delay: 1
# | IP 地址 | Hostname | 国家代码 | 旗帜 |
---|---|---|---|---|
1 | 3.238.96.44 | ec2-3-238-96-44.compute-1.amazonaws.com | US | |
2 | 3.208.87.93 | ec2-3-208-87-93.compute-1.amazonaws.com | US | |
3 | 3.236.108.253 | ec2-3-236-108-253.compute-1.amazonaws.com | US | |
4 | 34.207.78.138 | ec2-34-207-78-138.compute-1.amazonaws.com | US | |
5 | 18.232.50.58 | ec2-18-232-50-58.compute-1.amazonaws.com | US | |
6 | 3.235.235.45 | ec2-3-235-235-45.compute-1.amazonaws.com | US | |
7 | 44.192.10.200 | ec2-44-192-10-200.compute-1.amazonaws.com | US | |
8 | 3.88.30.42 | ec2-3-88-30-42.compute-1.amazonaws.com | US | |
9 | 3.235.251.150 | ec2-3-235-251-150.compute-1.amazonaws.com | US | |
10 | 44.211.131.205 | ec2-44-211-131-205.compute-1.amazonaws.com | US |
未找到任何关于 claudebot 爬虫的信息。
# | IP 地址 | Hostname | 国家代码 | 旗帜 |
---|---|---|---|---|
1 | 3.137.176.67 | ec2-3-137-176-67.us-east-2.compute.amazonaws.com | US | |
2 | 3.145.53.16 | ec2-3-145-53-16.us-east-2.compute.amazonaws.com | US | |
3 | 3.144.45.101 | ec2-3-144-45-101.us-east-2.compute.amazonaws.com | US | |
4 | 3.147.195.193 | ec2-3-147-195-193.us-east-2.compute.amazonaws.com | US | |
5 | 3.129.92.231 | ec2-3-129-92-231.us-east-2.compute.amazonaws.com | US | |
6 | 3.145.100.135 | ec2-3-145-100-135.us-east-2.compute.amazonaws.com | US | |
7 | 3.22.51.241 | ec2-3-22-51-241.us-east-2.compute.amazonaws.com | US | |
8 | 18.119.172.8 | ec2-18-119-172-8.us-east-2.compute.amazonaws.com | US | |
9 | 18.119.114.15 | ec2-18-119-114-15.us-east-2.compute.amazonaws.com | US | |
10 | 18.227.48.4 | ec2-18-227-48-4.us-east-2.compute.amazonaws.com | US |
不清楚是否为 Claude 的爬虫程序,通过向 claudebot@anthropic.com
发送邮件失败(We're writing to let you know that the group you tried to contact (claudebot) may not exist),可能并不是真正的 Claude 爬虫,需要大家注意!
# | IP 地址 | Hostname | 国家代码 | 旗帜 |
---|---|---|---|---|
1 | 18.118.162.180 | ec2-18-118-162-180.us-east-2.compute.amazonaws.com | US | |
2 | 18.117.78.145 | ec2-18-117-78-145.us-east-2.compute.amazonaws.com | US | |
3 | 18.224.69.84 | ec2-18-224-69-84.us-east-2.compute.amazonaws.com | US | |
4 | 18.119.130.36 | ec2-18-119-130-36.us-east-2.compute.amazonaws.com | US | |
5 | 18.191.168.142 | ec2-18-191-168-142.us-east-2.compute.amazonaws.com | US | |
6 | 3.149.255.208 | ec2-3-149-255-208.us-east-2.compute.amazonaws.com | US | |
7 | 18.217.120.254 | ec2-18-217-120-254.us-east-2.compute.amazonaws.com | US | |
8 | 3.137.198.39 | ec2-3-137-198-39.us-east-2.compute.amazonaws.com | US | |
9 | 3.147.48.240 | ec2-3-147-48-240.us-east-2.compute.amazonaws.com | US | |
10 | 3.16.82.7 | ec2-3-16-82-7.us-east-2.compute.amazonaws.com | US |
根据行业标准,Anthropic 使用各种数据源进行模型开发,例如通过网络爬虫收集的来自互联网的公开数据。作为我们构建安全可靠的前沿系统和推动负责任的人工智能开发领域的使命的一部分。
ClaudeBot 遵守 robots.txt 协议,如果需要屏蔽整个网站禁止 ClaudeBot 抓取写法如下:
User-agent: ClaudeBot
Disallow: /
如果需要延缓 ClaudeBot 抓取速度,写法如下:
User-agent: ClaudeBot
Crawl-delay: 1
# | IP 地址 | Hostname | 国家代码 | 旗帜 |
---|---|---|---|---|
1 | 3.138.125.2 | ec2-3-138-125-2.us-east-2.compute.amazonaws.com | US | |
2 | 18.116.40.177 | ec2-18-116-40-177.us-east-2.compute.amazonaws.com | US | |
3 | 18.221.187.121 | ec2-18-221-187-121.us-east-2.compute.amazonaws.com | US | |
4 | 3.141.47.221 | ec2-3-141-47-221.us-east-2.compute.amazonaws.com | US | |
5 | 18.190.156.212 | ec2-18-190-156-212.us-east-2.compute.amazonaws.com | US | |
6 | 3.15.202.4 | ec2-3-15-202-4.us-east-2.compute.amazonaws.com | US | |
7 | 3.12.41.106 | ec2-3-12-41-106.us-east-2.compute.amazonaws.com | US | |
8 | 3.133.160.156 | ec2-3-133-160-156.us-east-2.compute.amazonaws.com | US | |
9 | 18.118.200.197 | ec2-18-118-200-197.us-east-2.compute.amazonaws.com | US | |
10 | 3.133.147.252 | ec2-3-133-147-252.us-east-2.compute.amazonaws.com | US |