3

I have a site with a few hundred pages that I've inherited. It is all built with static html files, one file for every page. It's clear that the previous developer didn't really have a plan for the site as each page as many different elements that seem to be all randomly combined.

What I'd like to do is convert the site to be ExpressionEngine based. Building the site is no problem, but importing the content does not look like it's going to be very fun. Does anyone know if there is a way to some how scrape the main content of each page and get it into a form at that EE can use and do it all automatically?

Improve this question
1
  • 2
    Unfortunately I don't think there is. I was in the same boat earlier this year. I first setup EE and then allowed my client to dump/enter the content in
    – Siebird
    Nov 20, 2012 at 20:23

4 Answers 4

Reset to default
6

Consider Yahoo Pipes for scraping content from webpages. You can export it to a number of formats and then use Solspace's Importer or AJW's DataGrab to move it into place on EE. You can find an walkthrough on how to grab content via Pipes here:

http://www.daybarr.com/blog/yahoo-pipes-tutorial-an-example-using-the-fetch-page-module-to-make-a-web-scraper

Improve this answer
2
  • 1
    This looks to be the best solution, for me, so far. Thanks.
    – CreateSean
    Nov 21, 2012 at 16:16
  • I did an import with codeigniter and querypath. first grabbed the listing pages via the pagination links. Then read in all pages with querypath. (the latter is xml/xhtml library that lets you access the page dom, much like jquery does in the browser)
    – GDmac
    Nov 28, 2012 at 10:00
4

It's a bit rough and ready but I wrote a EE 2.x plug-in to crib content from a static site we have to convert to EE. It had tons of news entries and there wasa common pattern in the markup we were able to leverage to scrape the content and insert it as a channel entry.

You may need to download/fork this and adjust to fit your needs.

https://github.com/ninefour/crib.pi.ee_addon

Hope that's of some help. :)

Improve this answer
3

One option would be to build a PHP parser that would take the site map and loop through each page, collecting the content in the main content div and saving that into the database (or CSV file with title and other fields that could be imported then via Data Grab).

As long as all the content is in the same main div, you should be good to go. You could even have it check for the first H1 on the page and have it save that as the Title field. If you want to get really fancy, you could also have the PHP file keep track of site hierarchy and import the necessary information for a structure.

You could start here for some information on some basic code samples for the php parser. https://webmasters.stackexchange.com/questions/2969/possible-to-only-load-specifiv-div-of-a-website

It may take a few hours to build the script, but I think it would be a lot quicker than copying all the pages over.

Improve this answer
0
0

Using http://fuckyeahmarkdown.com/ with the readability option turned on may get you much of the way to getting just the content without the structure imported in markdown format.

Improve this answer
3
  • Not sure why I got downvoted for that. If you are trying to pull in just the content of a page in a format that is easy to get into expressionengine then running through that site will provide you with a nicely markdown formated output with its best guess of providing just the content. Oh well.
    – UltraBob
    Nov 24, 2012 at 7:22
  • 1
    This works for a single page maybe. Not suited for importing a whole site.
    – GDmac
    Nov 28, 2012 at 9:58
  • Seems fairly easy to script to me, but I see your point.
    – UltraBob
    Nov 28, 2012 at 13:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.