How to import API data into EE

Question

If I wanted to consume API feeds (restful) and import that data into EE as entries, is that possible?

Ideally I would like to import a bulk amount of channels (once only), then a bunch of categories for each channel (once only/or once a blue moon), and then I want ti be able to fetch the JSON and use it with some mapping to entries in EE.

Is Solspace - importer the best tool to do this, or will I be better writing some manual PHP scripts to import them all into EE.

The final site will be thousands of pages, and then I am going to augment them with further data made available over an API.

I am really hoping EE will scale to this, as otherwise I need to write the system from scratch - how large does EE scale to?

Looking at say 500k unique visits a month viewing this imported data.

Or would this; be a better option

http://reinos.nl/expressionengine/webservice/docs

---

Excuse my long reply! * I am the thread author, just found my proper SE account!

Hello,

How was your experience using DataGrab, so I am correct in thinking if I had an API, I could make it import that into an entry, and also some custom fields?

Could you share with me as much detail as you wish, about how you found that process, and how you did it?

I would have 250 channels to create, with up to 30 categories in each, and each category would have 5-10 pages.

Lastly, is there a way I could make all the channels share the same template, all the categories the same, and all the pages the same? I.E three templates, without me duplicating them numerous times? Is there a way to smooth that flow too, I want to automate much of the entry creation as possible.

Say I had weather data, I want to be able to make a weather page for up to... 500 cities programatically, but I want each one to be a separate page. like

domain.com/channel/category/city

I am very familiar with EE, just never tried to automate content/entry creation and this would be at scale.

With datagrab, I could import the data through the API, but I can still edit it too through the usual edit content pages correct?

If I can't figure out how to do it with DataGrab, then I fear I would have to write an Angular App from scratch, but I love the EE interface and stuff.

Lastly, which plugin do you think is best? Datagrab sounds like it has more features, but I have bought SolSpace plugins before and they have always been good..

Answer 1

I've used DataGrab to achieve this in the past. Solspace Importer will also do the same.

Answer 2

You can setup both Solspace Importer and DataGrab to handle imports via CSV, RSS, JSON request either at a single time or on a schedule without issue.

Without getting into the debate of server setup you shouldn't have issues with serving that amount of visitors on a monthly basis assuming your server setup is sufficient. I have a client site that serves almost 500k unique visits a day without issues. You can see some large sites in a recently updated blog post from Hop Studios

Answer 3

If all "channels" have the exact same channel custom fields, you could just create 1 channel and assign a common custom field group. You could then differentiate each entry (what you're calling "pages") based on categories. Your categories would be your city names. With that setup, you could indeed have one template, and entries would display based on the category in the URL.

For example, if you have a "weather" template group with a template named "index". You could get all entries with the "New York" category by going to: http://example.com/weather/category/new-york

The weather/index template would likely contain a {exp:channel:entries} tag pair, at minimum, to display entry data.

Note: When the template's name is index, it's is not required in the URL. Any other name would need to have the template's name in the URL, eg. http://example.com/weather/entries/category/new-york

Also note that Solspace Importer is an add-on for importing channel entry data into a channel based on external data (eg. CSV file). It does not create the actual channel for you, nor does it create custom fields for a channel. It does, however, create entries in your pre-established channel.