I've inherited an ee site, having never really worked with one before, and the robots.txt has multiple things that I would normally assume shouldn't be in there, such as:
- and some specific files.
Given my understanding these items will now never be crawled by the search engines.
Also putting specific files in there just easily reveals them to the wider world, it's not really helping security is it? In fact the opposite.
Anyone with any advice would be appreciated, if there are folders that should be in there for ee then that's fine.