Guide: Hide your Lemmy instance from search engines
Guide: Hide your Lemmy instance from search engines
Due to the nature of the default robots.txt and the meta tags in Lemmy, search engines will index even non-local communities. This leads to results that are undesirable, such as unrelated/undesirable content being associated with your instance.
As of today, lemmy-ui does not allow hiding non-local (or any) communities from Google and other search engines. If you, like me, do not want your instance to be associated with other content, you can add a custom robots.txt and response headers to avoid indexing.
In nginx, simply add this:
undefined
# Disallow all search engines location / { ... add_header X-Robots-Tag noindex; } location = /robots.txt { add_header Content-Type text/plain; return 200 "User-agent: *\nDisallow: /\n"; }
Here's a commit in my fork of the lemmy-ansible playbook. And here's a corresponding issue I opened in lemmy-ui.
I hope this helps someone :-)