Exclude ?URL redirect from site 404's

Question

I'm administering EE site with forum. On the forum, there are many redirects in the form of http://domain.com/?URL=http://third-party-site.com/page.html

Recently, Google started returning 404 for my site if the page on third-party site (which I can't control) does nit exist. What should I do to avoid that, except for blocking "?URL" urls in robots.txt (especially having in mind that the links existed always, but started returning errors just recently, EE version has not been updated)

Answer 1

You can do this in your htaccess

RewriteCond %{QUERY_STRING} URL [NC]
RewriteRule .* - [R=404,L]

And, EE will return a 404 error code. BUT, EE seems to always return the page that says To proceed to the URL you have requested, click the link below. It won't return your 404 template that you've set in EE (EE seems to give that ?URL parameter precedent over its 404 template processing.)

So, I'd worry that the above is not enough to right things with the Google index, and I'd throw the stronger 410 "Gone" error, instead--plus route it to a static error page outside of EE:

#optional, show a static file as an error document
ErrorDocument 410 /410.html
#
RewriteCond %{QUERY_STRING} URL [NC]
RewriteRule .* - [R=410,L]

EDITED: "Gone" is status code 410 -- corrected, above

I think Google (and other search engines) recognize that people who know how to throw 400 errors aren't kidding, and so those "gone" resources can be removed from the search index right away ;-)

---

Note that, if you don't use EE for your 404 page--e.g., you instead have ErrorDocument 404 /404.html set in your htaccess pointing to a static page, then the original example of throwing a 404 will totally circumvent EE, and show your static 404 error page.

Answer 2

One option is to add <meta name=”robots” content=”noindex, nofollow”> to the forum header so none of the pages on the forum are indexed. You can do that here:

themes > forum_themes > default > forum_global > meta_tags.html