# Robots.txt

Control search engine indexing and link following for your Documentation.AI site, then verify the published robots.txt configuration.

## Overview

Documentation.AI automatically generates and serves a `robots.txt` file at `/robots.txt` on your live docs site. The file includes directives for search engine indexing and link following, along with a reference to your sitemap.

Configure these directives from the **SEO** section in Site Config in the Editor top toolbar, or directly in [`documentation.json`](/content/docs/customize/site-configuration#seo-configuration/index.html).

`robots.txt` always reflects the last published version of your documentation. Publish your docs after any SEO configuration change to update the file on your live site.

## Enable or disable indexing and following

Control how search engines treat your docs by enabling or disabling indexing and link following. Documentation.AI stores these settings under `seo.robots:index` and `seo.robots:follow` in `documentation.json`. Both settings default to `true`.

### Using the Editor

Use Site Config when you want to change robots behavior without editing configuration files.

Open Site Config

In the Editor, click **Site Config** in the top toolbar, then select **SEO**.

Adjust robots directives

- Turn indexing **on or off** to control whether search engines index your docs pages. This maps to `seo.robots:index`.
- Turn following **on or off** to control whether search engines follow links on your docs pages. This maps to `seo.robots:follow`.

Publish changes

- Changes appear in the **Changes** panel alongside content edits.
- Publish your documentation so the updated robots configuration is included in the build.
- After publishing completes, reload `/robots.txt` on your live site to confirm that it reflects the new settings.

Site Config changes to robots directives only take effect after a new publish. If `/robots.txt` does not look updated, check that the latest publish finished successfully and that you are viewing the correct environment or domain.

### Using documentation.json

Define robots behavior directly in `documentation.json` when you manage site configuration in source control.

```json
{
  "seo": {
    "robots:index": true,
    "robots:follow": true
  }
}
```  
`seo.robots:index` controls whether search engines may index your documentation pages. Set it to `false` to return a non-indexing directive.

`seo.robots:follow` controls whether search engines may follow links from your documentation pages. Set it to `false` to return a non-following directive.

## Access and verify robots.txt

Verify the published file from your live site after changing either robots directive.

Open your live docs site

Open the public domain or custom domain where your documentation is published.

Open robots.txt

Append `/robots.txt` to your docs base URL. Confirm that the request returns HTTP `200` and that the file contains the expected indexing, following, and sitemap directives.

Check crawler access

Confirm that search engine tools such as Google Search Console can access the published file.

Custom domains and subpath deployments use the same relative path. Open `/robots.txt` from the base URL where your documentation is published.

## Robots and access control

Access control can override the indexing behavior you configure for a public documentation site. Private or protected pages always receive `noindex, nofollow` directives, and unauthorized visitors do not receive the pages' real metadata.

A fully private site returns a robots file that disallows crawling. Configure which pages require authentication from [Access control](/content/docs/customize/access-control/overview/index.html).

Sitemap generation is separate from robots indexing. Disabling indexing does not disable the sitemap. See [Sitemaps](/content/docs/seo-and-geo/sitemap/index.html) for sitemap configuration and behavior.