Robots.txt and sitemap, ready to upload.
Tell search engines which pages to crawl and which to skip. Fill both files from your live site, then copy them to your site's root folder.
Your site
robots.txt
sitemap.xml
Upload, then test
Open yourdomain.com/robots.txt in a browser to check it loads. Then audit a few pages here to confirm they are not blocked by mistake.
Submit the sitemap
In Google Search Console, open Sitemaps, enter sitemap.xml and submit. Search Console then reports how many pages Google has indexed.
Never block CSS or JS
Google needs your stylesheets and scripts to see the page the way visitors do. Block only private areas such as admin, cart and checkout.
About robots.txt and sitemaps
Where do I upload robots.txt and sitemap.xml?
In the root folder of your website, so they open at yourdomain.com/robots.txt and yourdomain.com/sitemap.xml. On most hosts that is the public_html or www folder. WordPress SEO plugins create both files for you.
Does blocking a page in robots.txt remove it from Google?
No. Robots.txt stops Google from crawling a page, but the address can still appear in results if other pages link to it. To keep a page out of Google, let it be crawled and add a noindex meta tag instead.
Which pages belong in the sitemap?
Only pages you want to appear in Google: pages that return status 200, are not blocked by robots.txt, have no noindex tag and are the canonical version. Leave out thank-you pages, carts, logins and filtered duplicates.
Does Google use priority and changefreq?
Google has said it ignores both. It does use lastmod when the date is accurate, so update it only when the page content really changes.