CDN

Sometimes you want to serve a different robots.txt file based on request URL, this could be because you run your site directly from CDN like I do. This means you’ll be able to get duplicate content, and to disallow this, we want to serve a different robots.txt file, or set specific headers. As a small … Continued