0

The SEO agency in charge of my client's website has unearthed a significant duplication issue due to which the business may get penalized.

We have setup the expressionengine cms at secretcmsurl.domain.com and somehow google has managed to index all CMS related urls which are obviously the same as the main websites. We now need to un-index these cms urls.

When I suggested that we add no-follow robots.txt entries, the agency said that it would reflect on the main website as well.

How do I now add a <meta name="robots" content="noindex, nofollow"> meta tag only to the backend cms templates? Do I write a plugin? If so, what kind of plugin?

  • Can you please clarify. Do you have a site using EE within which you have a public facing site AND you have also built a CMS on the front-end rather than use EE's own Control panel area? – foamcow Oct 6 '14 at 7:51
  • Yes I have a public site using EE and No - I have not built a CMS on the front-end. The CMS is simply hosted on a subdomain that is accessible via a public url – Amit Erandole Oct 6 '14 at 17:01
2

It's a simple matter of adding the robots meta tag to the templates which you don't wish to be indexed.

Then use Google's Webmaster Tools to remove those URLs from its index.

You could create a 'dynamic' robots file that applies only to your subdomain by using an htaccess rule like so: (from http://moz.com/community/q/block-an-entire-subdomain-with-robots-txt)

RewriteEngine on
RewriteCond %{HTTP_HOST} ^subdomain.website.com$
RewriteRule ^robotx\.txt$ robots-subdomain.txt

Then add:

User-agent: *
Disallow: / 

You should also read the rest of that thread and specifically this

The very next thing I would do is sort out how the site is set up. Having the subdomain and root domain pointing at the same place sounds like a recipe for disaster.

How are the "CMS" pages being crawled? Surely it requires a log-in to access them?

At the very least turn on http authentication for the templates you use for the CMS portion of the site to prevent them being crawled by anything that doesn't have the necessary credentials.

| improve this answer | |
  • ok @foamcow I will try this tomorrow and come back to you – Amit Erandole Oct 6 '14 at 17:02
  • It does require a login to access them but the agency is concerned that google will index them anyway – Amit Erandole Oct 6 '14 at 17:09
  • How can they be indexed if they are behind a log in? Just ensure that a redirect is issued if the user isn't logged in. – foamcow Oct 6 '14 at 18:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.