Robots.txt is not found when placed in web root: where to place?

Question

What would be the correct place to put my robots.txt file? In wordpress i was used to putting this in the web root but in EE this doesnt seem to work, when opened in webbrowser i get 404 error.

Answer 1

Sounds like your htaccess file is rewriting the request and serving it to ExpressionEngine index.php. Ensure you have RewriteCond %{REQUEST_FILENAME} !-f in your htaccess file which will ignore all physical file requests and not ask EE to serve the request.

# Directs all EE web requests through the site index file
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule ^(.*)$ /index.php/$1 [L]

Alternatively, don't have a physical file on the server, create the robots.txt file as a template in the main template group folder. You might need to utilise template routes to remove the template group name.

Answer 2

Very much depends on how your server is set up, but usually it should go in the same folder that you have index.php - which is usually the folder public_html or some such (depending on what server you are using).

If that is not finding it - check to see if you have some .htaccess processing going on: that might mess it up.