hugo-theme-flat/layouts/robots.txt

User-agent: *
Allow: /posts/
Allow: /tags/
Allow: /categories/
Allow: /about/

Disallow: *
Sitemap: {{ .Site.BaseURL }}sitemap.xml
enable robots.txt by default use CRLF in robots.txt 2022-05-10 13:34:51 +00:00			`User-agent: *`
			`Allow: /posts/`
			`Allow: /tags/`
			`Allow: /categories/`
			`Allow: /about/`

			`Disallow: *`
			`Sitemap: {{ .Site.BaseURL }}sitemap.xml`