gohugoio-hugo/docs/content/extras/robots-txt.md

lastmod	date	menu	next	prev	title	weight
2015-12-08	2013-07-09	main parent extras	/community/mailing-list	/extras/urls	Table of Contents	120

Hugo can generated customized robots.txt in the [same way than any other template]({{< ref "templates/go-templates.md" >}}).

By default it generates a robots.txt which allows everything, it looks exactly

User-agent: *

To disable it just set disableRobotsTXT option to false in the [command line]({{< ref "commands/hugo.md" >}}) or [configuration file]({{< ref "overview/configuration.md" >}}).

Hugo will use the template robots.txt following the list starting with the one with more priority

• /layouts/robots.txt
• /themes/THEME/layout/robots.txt

An example of a robots.txt layout is:

User-agent: *

{{range .Data.Pages}}
Disallow: {{.RelPermalink}}{{end}}

This template disallows and all the pages of the site creating one Disallow entry for each one.