2

I've inherited an ee site, having never really worked with one before, and the robots.txt has multiple things that I would normally assume shouldn't be in there, such as:

  • /assets/
  • /images/
  • /themes/
  • and some specific files.

Given my understanding these items will now never be crawled by the search engines.

Also putting specific files in there just easily reveals them to the wider world, it's not really helping security is it? In fact the opposite.

Anyone with any advice would be appreciated, if there are folders that should be in there for ee then that's fine.

1

Well, There is no specific files or folders to put compulsory in robots.txt for EE. This is a file where people usually put things to allow or disallow search engine to crawl given folders or URLs. You can also pass sitemap in robots.txt file.

Specific answer to your question will be, It's okay to remove those folders from robots.txt if you want to crawl them. Nothing to do with security.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.