ExpressionEngine® Answers is a question and answer site for administrators, end users, developers and designers for ExpressionEngine® CMS. It only takes a minute to sign up.
Sign up to join this community
I have a static site that pulls in pop up content using ajax. I don't want search engines to index those ajax pages. I know there are ways to block pages with
<head>
<META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">
</head>
But I don't have that on each of the ajax pages. so wondering what the best approach would be.
I would think just putting those pages into a robots.txt file would be effective. Just a plain text file that sits in your website's root directory and includes basic site information (e.g. about location of your sitemap etc.). This can include statements instructing Google's robots to ignore specific files and folders - e.g.
Disallow: /ajax/ will tell google to ignore any files in the /ajax directory.
Disallow: /ajax/file.html will tell google to ignore the file /ajax/file.html
Using robots.txt means you can set these exclusions without having to put information into the files themselves, and also exclude directories.
HTH
