Manage your sitemap, crawler rules, and structured data
Understand the different jobs of sitemap.xml, robots.txt, llms.txt, canonical URLs, and JSON-LD.
These settings help services discover URLs and understand your site. They solve different problems, so change the setting that matches your goal. Publishing a page, allowing a crawler, and appearing in a search result remain separate steps.
Check the automatic sitemap
Open Settings → Sitemap and select View XML. Your sitemap is available at /sitemap.xml on the public site.
Neopress builds it from published, public pages and eligible published CMS entries. A CMS item needs its public collection and detail-page template; a scheduled item with a future publication time is not included yet. Public language variants are included according to the site's available published languages.
If a URL is missing, open that exact address and check its publication status, visibility, language, and related CMS page before editing a sitemap manually. Creating a draft alone does not add a live page.
Control crawling with robots.txt
Open Settings → Robots.txt. The default file allows crawling and provides the sitemap address. If you replace it with custom text, you replace the default file, so preserve the rules and sitemap line you still need.
- Identify the crawler and paths your change should affect.
- Edit the rules and click Save changes.
- Use View TXT and check the complete public file, especially any
Disallowrules.
Saving an empty robots.txt editor restores the default behavior. Crawler rules are not password protection and should not be used to keep confidential content private.
Add llms.txt only when useful
Ask the Neopress AI to create a custom llms.txt using your site description and the public pages you want to highlight. For example: “Create llms.txt with a short site introduction and links to our services, pricing, and contact pages. Use the actual public URLs.” Review the content, then open /llms.txt on your site to verify it. This is a saved site setting, not an automatically generated inventory; removing its contents makes the public URL return 404.
This optional file does not guarantee AI discovery or citation. Google does not require it for its AI search features. See Google's guidance.
Set structured data and preferred URLs
Under Settings → General → Advanced Settings, use JSON-LD Structured Data for facts that match the visible content. Choose site scope for site-wide facts and a page or CMS entry for page-specific information. Review AI-generated values, correct validation errors, and save.
Neopress provides a default canonical URL based on the public domain and page path. A page's Canonical URL can be deliberately edited when another URL is the preferred version. Avoid directing unrelated pages to your homepage. Language alternates are generated for available public language versions; review those versions before assuming every translation is reachable.
Finish by opening the public file or page you changed. Valid metadata and structured data help describe content, but do not promise indexing, a particular search appearance, or a rich result.