Use a middleware function. This way the robots.txt will be handled before any session, cookieParser, etc: app.use(‘/robots.txt’, function (req, res, next) { res.type(‘text/plain’) res.send(“User-agent: *\nDisallow: /”); }); With express 4 app.get now gets handled in the order it appears so you can just use that: app.get(‘/robots.txt’, function (req, res) { res.type(‘text/plain’); res.send(“User-agent: *\nDisallow: /”); });
Here’s a solutions if you want to disallow query strings: Disallow: /*?* or if you want to be more precise on your query string: Disallow: /*?dir=*&order=*&p=* You can also add to the robots.txt which url to allow Allow: /new-printer$ The $ will make sure only the /new-printer will be allowed. More info: http://code.google.com/web/controlcrawlindex/docs/robots_txt.html Advanced Usage … Read more
The best way is to set static_url_path to root url from flask import Flask app = Flask(__name__, static_folder=”static”, static_url_path=””)
That file will allow all crawlers access User-agent: * Allow: / This basically allows all user agents (the *) to all parts of the site (the /).
According to the official documentation on sitemaps.org it needs to be a full URL: You can specify the location of the Sitemap using a robots.txt file. To do this, simply add the following line including the full URL to the sitemap: Sitemap: http://www.example.com/sitemap.xml