Копия источника
Does Anthropic crawl data from the web, and how can site owners block the crawler? | Anthropic Help Center Skip to main content Claude Release notes Get started with Claude Get started with Claude What are some things I can use Claude for? Where can I access Claude? How up-to-date is Claude's training data?
What interfaces can I use to access Claude? Choose a Claude plan Verify your phone number How to gift a Claude subscription How to redeem a Claude gift subscription Account management Log in to your Claude account How to get support How do I change the email address associated with my account? I would like to input sensitive data into my chats with Claude. Who can view my conversations?
How do I log out of all active sessions? Export your Claude data Delete your Claude account Why is a coupon or promotion not available for my account? Managing your active sessions Understanding your billing address and tax calculation Notice regarding consumption tax (JCT) for Japanese customers Configuring session security settings How to get support for Claude for Government Public Sector FAQs Claude 4 Invite Contest Conversation management Delete or rename a conversation Share and unshare chats Use incognito chats Use Claude’s chat search and memory to build on previous context Why Claude switched models in your conversation with Fable 5 or Fable 5.1 Why Claude switched models in your conversation with Opus 5 Features and capabilities What are artifacts and how do I use them? Publish and share artifacts When should I use web search, extended thinking, and research?
Enable and use web search Use research on Claude Can Claude produce images? Upload files to Claude Create and edit files with Claude How does Claude handle mathematical equations and calculations? What are projects? Retrieval augmented generation (RAG) for projects What are skills?
Use skills in Claude How to create custom skills Browse skills, connectors, and plugins in one directory Use Claude for Excel Use Claude in Xcode Use Claude in Microsoft Foundry Use Claude for PowerPoint Visual and interactive content Import and export your memory from Claude Work across Microsoft 365 apps Use Claude for Microsoft 365 with third-party platforms Custom visuals in chat and Cowork Use dictation in Office agents Use Claude for Word Available beta and research preview features Use Claude Security Get started with Claude Design Set up your design system in Claude Design Use Claude for Outlook Use plugins in Claude See your monthly recap Get started with Claude Science Personalization and settings Understanding Claude's personalization features Customizing your appearance settings How to use Claude in your preferred language Change the model, effort, and thinking settings Set break reminders and quiet hours Troubleshooting Troubleshoot Claude error messages Claude is providing incorrect or misleading responses. What’s going on? Claude is producing links that don’t work and falsely claiming that it has sent emails or produced external documents. What’s going on?
Adapt to new model personas after deprecations Usage and limits Usage limit best practices How do usage and length limits work? Buy usage bundles Use the Claude Agent SDK with your Claude plan Claude Fable models on your plan Pro and Max plans General How large is the context window on paid Claude plans? Paid plan billing FAQs Why was my card declined? Add or update your paid Claude account’s tax or VAT ID Why am I being asked to verify my payment method?
Cancel your Pro or Max subscription How can I create and manage projects? Request a refund for a paid Claude plan Manage usage credits for paid Claude plans Holiday 2025 Usage Promotion Understanding your Pro or Max plan invoices Pro plan What is the Pro plan? How do I sign up for the Pro plan? How to change your Pro plan from monthly to annual billing Max plan What is the Max plan?
How do I sign up for the Max plan? Team and Enterprise plans Plan overviews What is the Enterprise plan? What is the Team plan? Claude Team plan for scientists Get started Get started with the Team plan Move your personal Claude account to a Team or Enterprise organization Find and join a Team or Enterprise organization Migrate your organization from Team to Enterprise Join an organization via invite link Respond to an Enterprise domain claim on your Claude account Claude Code on Console to Enterprise migration Set up Claude for Teachers for your school or district Billing How is my Team plan bill calculated?
Team plan billing FAQs Change your Team plan from monthly to annual billing Add or update your Team plan's tax or VAT ID Cancel your organization's Team plan subscription How am I billed for my Enterprise plan? Understanding your Team plan invoices Admin management Roles and permissions Purchase and manage seats on Team plans Purchase and manage seats on Enterprise plans Manage members on Team and Enterprise plans Claude Enterprise consumption guide Manage usage credits for Team and seat-based Enterprise plans Manage user feedback settings on Team and Enterprise plans Disable public projects for your organization Manage plugins for your organization Account management FAQs Manage groups and group spend limits on Enterprise plans Manage custom roles on Enterprise plans Set up role-based permissions on Enterprise plans Set organization instructions Claude Design admin guide for Team and Enterprise plans Claim and migrate accounts on your domain Designate support contacts for human support Claude Enterprise Admin API reference guide Claude Enterprise activation promo for Claude Code and Cowork Set a default model for your organization Restrict verified-domain connectors to your Enterprise Manage model access for your organization Create surveys for your organization Analytics and usage View usage analytics for Team and Enterprise plans Configure a custom OpenTelemetry collector for Office agents Use analytics chat to ask Claude about usage Capabilities Manage project visibility and sharing Use enterprise search Use visual and interactive content on Team and Enterprise plans Provision and manage skills for your organization What is Claude Tag? Get started with skill and plugin scanning Security and compliance Does Anthropic Act as a Data Processor or Controller? Can you delete data that I sent via Team and Enterprise plans?
How can I access the personal information that Anthropic has on my account? What happens to a user's data when they are removed from a Team or Enterprise organization? Export your organization's data Who owns and manages the data of my team? HIPAA-ready Enterprise plans Data retention practices for Covered Models Configure custom data retention controls for Enterprise plans Enable US-only inference for your organization Access audit logs Access the Compliance API Get started with Claude Compliance API integrations Restrict access to Claude with IP allowlisting Enforce network-level access control with Tenant Restrictions What are customer-managed encryption keys (CMEK)?
Inference hooks overview Identity management (SSO, JIT, SCIM) Important considerations before enabling single sign-on (SSO) and JIT/SCIM provisioning Set up single sign-on (SSO) Set up JIT or SCIM provisioning Switching to a different Identity Provider (IdP) How SCIM sync works for Enterprise organizations Setup by identity provider Google Workspace SSO setup Microsoft Entra ID SSO setup Okta SSO setup OneLogin SSO setup Ping Identity SSO setup Troubleshooting by identity provider Google Workspace SSO/SCIM email mismatch Microsoft Entra ID SSO/SCIM email mismatch Okta SSO/SCIM email mismatch OneLogin SSO/SCIM email mismatch Ping Identity SSO/SCIM email mismatch Claude Cowork Get started with Claude Cowork Use Claude Cowork safely Use Claude Cowork on web, desktop, and mobile Use Claude Cowork on Team and Enterprise plans Use the built-in browser in Claude Cowork Set up browser use in Claude Cowork for Team and Enterprise plans Use artifacts in Claude Cowork Schedule recurring tasks in Claude Cowork Assign tasks from anywhere in Claude Cowork Organize your tasks with projects in Claude Cowork Let Claude use your computer in Cowork Monitor Claude Cowork activity with OpenTelemetry Claude Cowork architecture overview Claude Code Claude Code model configuration Use Claude Code with your Pro or Max plan Automated Security Reviews in Claude Code Use Claude Code with your Team or Enterprise plan Claude Code usage analytics Manage API key environment variables in Claude Code Claude Code FAQ Claude Code on the web Set up Code Review for Claude Code Use Claude Code CLI with a screen reader Launch guides Your first day in Claude Code Models, usage, and limits in Claude Code Troubleshoot Claude Code installation and authentication Give Claude context: CLAUDE.md and better prompts Claude Code cheatsheet Claude Code user FAQ Claude Code power user tips Claude Code: Common developer use cases Claude Code communications kit Claude Code champion kit Claude Desktop General Install Claude Desktop Deploy Claude Desktop for Windows Deploy Claude Desktop for macOS Enterprise configuration for Claude Desktop Use quick entry with Claude Desktop on Mac Open Claude Desktop with a link Desktop extensions Enabling and using the desktop extension allowlist Deploying enterprise-grade MCP servers with desktop extensions Getting Started with Local MCP Servers on Claude Desktop Claude Mobile apps General Use dictation on Claude Mobile Use voice mode Minimum age requirement access restriction Open the Claude mobile app with a link Claude for iOS Install Claude for iOS How do I sign up for Claude Pro on the Claude app for iOS? Use Claude app intents, shortcuts, and widgets on iOS Access Claude for iOS on your Lock Screen, Control Center, and Action button How to update Claude for iOS Use Claude with iOS apps Claude for Android Install Claude for Android How do I sign up for Claude Pro on Claude for Android? Use the Claude widget on Android How to update Claude for Android Use Claude with Android apps Claude API and Console Use the Claude API and Console How can I access the Claude API? Where is the Claude API supported?
Log in to your Console account I created a Claude Console organization - how do I start using the Claude API? How do I use the playground? Cost and Usage Reporting in the Claude Console Creating and managing Workspaces in the Claude Console I have a paid Claude subscription (Pro, Max, Team, or Enterprise plans). Why do I have to pay separately to use the Claude API and Console?
Claude Console roles and permissions Manage user feedback settings on Claude Console About the Development Partner Program Anthropic's AI for Science Program Customizing your Console appearance settings Inviting members to the Claude Console Find and join a Console organization Assign a program to workspaces in Claude Console Claude API usage and best practices Our approach to rate limits for the Claude API I’m planning to launch a product using the Claude API. What steps should I take to ensure I’m not violating Anthropic’s Usage Policy? What should I do if I suspect my API key has been compromised? I’ve been using other large language models (LLMs) to power my product.
How should I alter my prompts when switching to Claude? API Key Best Practices: Keeping Your Keys Safe and Secure How can I get higher rate limits on the Claude API? Pricing and billing How do I pay for my Claude API usage? How can I learn more about Claude API pricing?
Add or update your Claude Console organization's tax or VAT ID Understanding your Claude API invoices API troubleshooting Where can I find your API documentation? I’m encountering 429 errors, and I’m worried my rate limit is too low. What should I do? Where can I go for Claude API support and assistance?
I'm getting an API connection error. How can I fix it? How can I check connectivity to the Claude API? API FAQ Can I have a Claude account and a Console account?
Can I use the Claude API for individual use? What is the External Researcher Access Program? How can I delete my Claude Console account? API prompt design Introduction to prompt design Claude’s response to my prompt is too brief.
My prompt isn’t giving me a helpful answer. API and distillation Preserved thinking: changing how the Messages API handles thinking blocks to protect against distillation Connectors General Use connectors to extend Claude's capabilities When to use desktop and web connectors Manage Claude’s tool access Anthropic Connectors Directory FAQ Anthropic Software Directory Policy Anthropic Software Directory Terms How Claude suggests connected apps Authorize MCP connectors for your entire organization Pre-Built Connectors Use Google Workspace connectors Use the GitHub integration Connect to Microsoft 365 Set up the Microsoft 365 connector Microsoft 365 connector security guide Use interactive connectors in Claude Get started with 1Password for Claude Custom Connectors Get started with custom connectors using remote MCP Claude in Chrome Get started with Claude in Chrome Claude in Chrome permissions guide Use Claude in Chrome safely Claude in Chrome admin controls Claude in Chrome troubleshooting Claude for Education Get started with Claude for Education at your university (for Owners/Admins) Use Claude for Education at your university Set up the Claude LTI in Canvas by Instructure Who owns and manages the data of my Claude for Education account? Privacy and legal Where can I learn more about Anthropic's Privacy practices? Business Associate Agreements (BAA) for Commercial Customers What Certifications has Anthropic obtained?
What is Anthropic’s policy for handling governmental requests for user information? Why am I receiving an 'Output blocked by content filtering policy' error? I think a user is infringing my copyright or other intellectual property. How do I report it?
How do I view and sign your Data Processing Addendum (DPA)? Why am I receiving an 'Output blocked by content filtering policy' error? Updates to our Acceptable Use Policy (now “Usage Policy”), Consumer Terms of Service, and Privacy Policy Consumer Terms of Service Updates Terms of Service Updates Official Anthropic marketing email addresses Report, block, and remove content from Claude Does Anthropic crawl data from the web, and how can site owners block the crawler? Online Safety Contacts Claude 4 Invite Sweepstakes Official Rules Designated point of contact for users in the EU Can I use my Outputs to train an AI model?
Report a concern: Australian DIS Standard compliance Unauthorized Anthropic stock sales and investment scams Covered Models Covered Models under a Business Associate Agreement (BAA) Using Claude for Legal Work: Privilege, Confidentiality, and How to Think About Configuration Claude for Teachers: your data and our terms How Claude marks AI-generated content Safeguards Our Approach to User Safety Exceptions to our Usage Policy Safeguards warnings and appeals API Safeguards Tools Law Enforcement Requests Responsible Use of Anthropic's Models: Guidelines for Organizations Serving Minors Report, block, and remove content from Claude CSAM Detection and Reporting Public Vulnerability Reporting Using Agents According to Our Usage Policy Creating a new account after deletion Model Safety Bug Bounty Program Crisis Helpline Support in Claude Identity verification on Claude Real-time cyber safeguards on Claude Opus and Sonnet Age assurance on Claude Child safety guidance for developers Amazon Bedrock What is Amazon Bedrock? How do I get access to Claude in Amazon Bedrock? I use Claude in Amazon Bedrock. Who do I contact for customer support inquiries?
How do I learn more about Amazon and Anthropic’s strategic collaboration? Where do I find Claude in Amazon Bedrock documentation? What AWS Regions are Claude models available in Amazon Bedrock? Claude for Government Get started with Claude for Government SSO login Set up SCIM in Claude for Government Organization instructions in Claude for Government MCP connectors MCP: Individual connectors MCP: Web Search Model availability in Claude for Government Classification banner in Claude for Government As per industry standard, Anthropic uses a variety of robots to gather data from the public web for model development, to search the web, and to retrieve web content at users’ direction.
Anthropic uses different robots to enable website owner transparency and choice. Below is information on the three robots that Anthropic uses and how to set your site preferences to enable those you want to access your content and limit those you don’t. Bot Use What happens when you disable it ClaudeBot ClaudeBot helps enhance the utility and safety of our generative AI models by collecting web content that could potentially contribute to their training. When a site restricts ClaudeBot access, it signals that the site's future materials should be excluded from our AI model training datasets.
Claude-User Claude-User supports Claude AI users. When individuals ask questions to Claude, it may access websites using a Claude-User agent. Claude-User allows site owners to control which sites can be accessed through these user-initiated requests. Disabling Claude-User on your site prevents our system from retrieving your content in response to a user query, which may reduce your site's visibility for user-directed web search.
Claude-SearchBot Claude-SearchBot navigates the web to improve search result quality for users. It analyzes online content specifically to enhance the relevance and accuracy of search responses. Disabling Claude-SearchBot on your site prevents our system from indexing your content for search optimization, which may reduce your site's visibility and accuracy in user search results. As part of our mission to build safe and reliable frontier systems and advance the field of responsible AI development, we’re sharing the principles by which we collect data as well as instructions on how to opt out of our crawling going forward: Our collection of data should be transparent.
Anthropic uses the Bots described above to access web content. Our crawling should not be intrusive or disruptive. We aim for minimal disruption by being thoughtful about how quickly we crawl the same domains and respecting Crawl-delay where appropriate. Anthropic’s Bots respect “do not crawl” signals by honoring industry standard directives in robots.txt.
Anthropic’s Bots respect anti-circumvention technologies (e.g., we will not attempt to bypass CAPTCHAs for the sites we crawl.) To limit crawling activity, we support the non-standard Crawl-delay extension to robots.txt. An example of this might be: User-agent: ClaudeBot Crawl-delay: 1 To block a Bot from your entire website, add this to the robots.txt file in your top-level directory. Please do this for every subdomain that you wish to opt out from. An example of this is: User-agent: ClaudeBot Disallow: / Opting out of being crawled by Anthropic Bots requires modifying the robots.txt file in the manner above.
Alternate methods like blocking IP address(es) from which Anthropic Bots operates may not work correctly or persistently guarantee an opt-out, as doing so impedes our ability to read your robots.txt file. If a crawler has a source IP address on this list, it indicates that the crawler is coming from Anthropic. You can learn more about our data handling practices and commitments at our Help Center. If you have further questions, or believe that our Bots may be malfunctioning, please reach out to [email protected].
Please reach out from an email that includes the domain you are contacting us about, as it is otherwise difficult to verify reports. You can be notified of substantial changes to this article by clicking here and completing the form: Subscribe to updates Did this answer your question? Related Articles Report, block, and remove content from Claude Business Associate Agreements (BAA) for Commercial Customers How to get support Does Anthropic Act as a Data Processor or Controller? Report, block, and remove content from Claude