How do you "hide" sites from search engines during development?

Question

I'm in the process of tweaking my base install and was wondering how you hide sites from search engines when deployed to a publicly accessible DEV or STAGING server?

I currently:

• Add a robot.txt file to disallow crawling (not sure how much that really helps though)
• Use global vars to hide GA code from all but PROD servers
• On occasion set the site to "offline" and force people to login

How do you do it? Do you simply keep the site "offline" and make the clients/stakeholders login? Or is there another fancy way of keeping the site "open" to those who know the URL but don't want to log in?

Answer 1

The best way to do this is to use a .htpasswd file to prevent anyone (including spiders) from seeing the site if they are not authorized. This is much better than a robots.txt file because unlike robots.txt - it cannot be ignored. If a spider doesn't have your password, they cannot view the content.

I typically like using a single, shared login for viewing the site. If all visitors to the site ask their browsers to remember the password, normally they will only need to enter it once.

Some hosts, such as DreamHost, will provide a method to automatically create a .htpasswd file for you. Otherwise, it's pretty easy to do manually.

First create or modify your .htaccess file in the root of your site to add authorization directives:

AuthType Basic
AuthUserFile /home/USERNAME/.htpasswd
AuthName "EE Staging Site"
require valid-user

We will create the .htpasswd file in a moment, which actually contains the credentials for the login. Whatever location you want to store this file in, you should enter above after the AuthUserFile directive. Lots of hosts have separate directories within your home directory for each site, so you may end up doing something like this:

AuthType Basic
AuthUserFile /home/USERNAME/example.com/.htpasswd
AuthName "EE Staging Site"
require valid-user

At any rate, once you've decided where to store the .htpasswd file and created the above .htaccess you need to then create the actual .htpasswd file. This is pretty easy:

/usr/bin/htpasswd -c /home/USERNAME/example.com/.htpasswd STAGINGUSER

Replace USERNAME with you actual username (and otherwise adjust the path as needed) STAGINGUSER use with a username you want visitors to have to enter, such as "staging":

/usr/bin/htpasswd -c /home/USERNAME/example.com/.htpasswd staging

In this example, we are creating a new username within the .htpasswd file named "staging". After running this line, it will prompt you for the actual password. Enter it, then enter the confirmation when prompted as well.

For Nginx, you can look into how to create the equivilant of a .htaccess file here and here.

The Nginx config file looks like this:

location  /  {
  auth_basic            "EE Staging Site";
  auth_basic_user_file  /home/USERNAME/example.com/.htpasswd;
}

The same instructions to create the .htpasswd file should apply, if you have Apache installed. If not, you can find alternative ways of creating the file at the second link.

Answer 2

I actually use the following along with Focus Lab's master config:

First I create php file called robots.php in root and I add the following to it:

<?php 
header("Content-Type: text/plain");
require_once('../config/config.env.php');
echo "#".ENV_FULL." version of robots";
?>

<?php if(ENV != 'prod'): ?>
User-agent: *
Disallow: /
<?php endif; ?>

###
# Add unsafe robots here to keep away
###

Making sure I'm linking to the config.env.php so I can check if we are on production or not.

Then in my .htaccess fill I add this:

# Use Dynamic robots.txt file
# ------------------------------
RewriteRule robots\.txt /robots.php [L]

Which does our redirect when robots hit robots.txt which in turn gives them the contents of the robots.php file.

Note: if we aren't on production then we are giving them the following:

User-agent: *
Disallow: /