How to Create a robots.txt for Next.js

robots.txt tells search engine crawlers which pages they can and can't crawl. Every public Next.js site should have one. Without it, Googlebot crawls everything — including API routes, internal redirects, and pages you'd rather not index.

App Router (Next.js 13+): Use robots.ts

In the App Router, Next.js can generate robots.txt dynamically from a TypeScript file. Create:

app/robots.ts

import { MetadataRoute } from 'next'

export default function robots(): MetadataRoute.Robots {
  return {
    rules: {
      userAgent: '*',
      allow: '/',
      disallow: ['/api/', '/_next/', '/admin/'],
    },
    sitemap: 'https://www.yoursite.com/sitemap.xml',
  }
}

Next.js serves this at /robots.txt automatically. No static file needed. The output:

User-agent: *
Allow: /
Disallow: /api/
Disallow: /_next/
Disallow: /admin/

Sitemap: https://www.yoursite.com/sitemap.xml

Multiple rules for different bots

export default function robots(): MetadataRoute.Robots {
  return {
    rules: [
      {
        userAgent: 'Googlebot',
        allow: '/',
        disallow: '/private/',
      },
      {
        userAgent: ['Applebot', 'Bingbot'],
        allow: '/',
      },
      {
        userAgent: '*',
        allow: '/',
        disallow: ['/api/', '/admin/'],
      },
    ],
    sitemap: 'https://www.yoursite.com/sitemap.xml',
  }
}

Pages Router (Next.js 12 and earlier): Static File

In the Pages Router, place a static file at:

public/robots.txt

Next.js serves everything in public/ at the root URL. The file content:

User-agent: *
Allow: /
Disallow: /api/
Disallow: /admin/

Sitemap: https://www.yoursite.com/sitemap.xml

This approach also works in the App Router if you prefer a static file over the TypeScript approach.

robots.txt Directives Explained

| Directive | Meaning | |-----------|---------| | User-agent: * | Applies to all crawlers | | User-agent: Googlebot | Applies to Google only | | Allow: / | Allow crawling of this path | | Disallow: /admin/ | Block crawling of this path | | Sitemap: | Location of your XML sitemap | | Crawl-delay: 10 | Wait 10 seconds between requests (not supported by Google) |

Rules are processed top to bottom. The most specific rule wins. An explicit Allow overrides a broader Disallow.

What to Disallow in a Next.js App

Disallow: /api/

API routes don't need to be indexed. They return JSON, not HTML. Blocking them saves crawl budget.

Disallow: /_next/

Next.js internal build files. Google generally doesn't index these anyway, but it's clean to exclude them.

Disallow: /admin/

Any authenticated admin area. These pages shouldn't be in search results regardless of whether they're behind auth.

Disallow: /dashboard/
Disallow: /account/
Disallow: /settings/

User-specific pages that shouldn't appear in search results.

What NOT to Disallow

Don't block CSS and JavaScript files. Googlebot renders pages like a browser — it needs your CSS and JS to understand your page layout and content. Blocking /_next/static/ prevents Google from rendering your pages correctly.

Don't block images unless intentional. If you want image search traffic, keep images crawlable.

Don't block your sitemap location. Some robots.txt files accidentally disallow /sitemap.xml — that defeats the whole purpose.

robots.txt Does Not Equal noindex

robots.txt tells crawlers not to visit a URL. It does not remove a page from the index. A page can already be indexed from a previous crawl and backlinks, and Disallow won't de-index it.

To prevent indexing, use a meta robots tag:

<meta name="robots" content="noindex, nofollow" />

Or in Next.js App Router layout/page:

export const metadata = {
  robots: {
    index: false,
    follow: false,
  },
}

Testing Your robots.txt

Direct URL check — visit https://yoursite.com/robots.txt to confirm it's serving correctly
Google Search Console — use the robots.txt tester in GSC to test specific URLs against your rules
Fetch as Googlebot — in GSC, use "URL Inspection" → "Test Live URL" to see what Googlebot can access

Full Example for a Next.js Marketing Site

app/robots.ts:

import { MetadataRoute } from 'next'

export default function robots(): MetadataRoute.Robots {
  return {
    rules: {
      userAgent: '*',
      allow: '/',
      disallow: [
        '/api/',
        '/admin/',
        '/dashboard/',
        '/account/',
      ],
    },
    sitemap: 'https://www.yoursite.com/sitemap.xml',
  }
}

This allows everything public, blocks API and authenticated routes, and points crawlers to your sitemap.

Dynamic robots.txt Based on Environment

You might want to block all crawlers on staging and preview deployments — you don't want Vercel preview URLs indexed:

// app/robots.ts
import { MetadataRoute } from 'next'

export default function robots(): MetadataRoute.Robots {
  const isProduction = process.env.VERCEL_ENV === 'production';

  if (!isProduction) {
    return {
      rules: {
        userAgent: '*',
        disallow: '/',
      },
    };
  }

  return {
    rules: {
      userAgent: '*',
      allow: '/',
      disallow: ['/api/', '/admin/'],
    },
    sitemap: 'https://www.yoursite.com/sitemap.xml',
  };
}

On any non-production deployment (preview, staging), this returns Disallow: / — blocking all crawlers. On production, it serves the normal permissive rules.

Sitemap Integration

Every robots.txt should include a Sitemap: line pointing to your XML sitemap. In Next.js App Router, create a sitemap.ts alongside robots.ts:

// app/sitemap.ts
import { MetadataRoute } from 'next'

export default function sitemap(): MetadataRoute.Sitemap {
  return [
    {
      url: 'https://www.yoursite.com',
      lastModified: new Date(),
      changeFrequency: 'monthly',
      priority: 1,
    },
    {
      url: 'https://www.yoursite.com/tools',
      lastModified: new Date(),
      changeFrequency: 'weekly',
      priority: 0.8,
    },
  ]
}

Next.js serves this at /sitemap.xml. Reference it in your robots.ts with:

sitemap: 'https://www.yoursite.com/sitemap.xml',

Google and Bing both read the Sitemap: directive and use it to discover pages faster than crawling alone.

Common robots.txt Mistakes in Next.js Projects

Blocking /_next/static/ — this blocks your CSS and JavaScript. Googlebot won't be able to render your pages properly, and you'll see "page cannot be rendered" warnings in Google Search Console.

Using robots.txt to hide sensitive pages — robots.txt is public. Anyone can read it and see exactly which URLs you're trying to hide. Use authentication to protect private pages — not robots.txt.

Not deploying to production — if you're using the public/robots.txt approach and you test locally but forget to commit the file, it won't be served in production.

Disallowing the sitemap URL — always check that your Sitemap: URL is not blocked by a Disallow rule above it.

For your full developer toolkit — JSON formatters, image tools, and more — visit Mizakii's free developer tools.

How to Create a robots.txt for Next.js (App Router and Pages Router)