Strange URLs appeared in Webmaster tools

Question

Got a very weird issue with an old EE (1.7) site, not one i've seen before.

After a server migration, we started getting a number of odd URLs being reported in Webmaster tools, all throwing errors (301 mainly)

The URLs were structured differently, but nearly all contained a web or email custom field in there somewhere - something like this

category/post-type/www.url.com/P0/

I've absolutely no idea how these URLs are being generated or how they are being formed. Anyone got any ideas where I could start to look? Completely at a loss.

Cheers Matt

Answer 1

I have seen this several times over the years. If you don't properly have your 404 pages some other website out there is generating these urls and they are coming back valid. Then when google indexes those sites the urls show up. Google doesn't know these are invalid URLs. The best way to prevent this is to make sure you tighten your 404s. Make sure you have any wrong group and/or page go to a 404 page.

Here is a good article on it:

http://joviawebstudio.com/index_ee.php/blog/guide_to_404_pages_with_expressionengine/

Big thing is using Strict URLs and having require_entry="yes" set in your channel entry tags.

Hope this helps!

Brad

Answer 2

Usually this happens when:

1. A client has a field that is supposed to contain a URL starting with http:// and they enter data without the http://. so it becomes a relative URL

2. There no error checking in the template to ensure that crazy long URLs are avoided.

It's actually not that hard to fix.

1. Do a quick look at all the values in the database for that field (or create a quick template that outputs that field for all entries ; add http:// to the entries. (Bonus: Add instructions to that URL collecting field that tells people to remember the http

2. Add this to the /category/post-type template, anywhere:

   {if segment_4 != ""}{redirect=404}{/if}

This says, if there's extra segments on the end of the URL (i.e. you'd never expect a valid segment_4 for that part of the site) then display the 404 page.

You can clean up some more of the bad pages with Strict URLs and having require_entry="yes" set, as another responder says, but the quickest fixes and likely causes are these.