Need to pull member information and custom fields from an external SQL (not mySQL) database into EE. The database is too big to export to csv (9gb) and import using Datagrab/Importer. Would JSON be option here as it will be a lot smaller size? So create a page on their server that is in the correct JSON format and then import the feed using Datagrab/Importer?
1 Answer
For an operation of this size, you should be using an enterprise solution, not a web application. You should be exporting the data from the other SQL database in chunks so you can move it gently. What is the other SQL format?
Directly moving the data files would be better than running a 9gb over-the-web import (too many chances for data to get wrecked, lose track of where you're at, etc..). I would recommend physically moving the data as a whole to the EE installation server, and then using a tool like MySQL Workbench or SequelPro, run your imports one table at a time. Then if you run into a failure during the process, you don't have to start over.
answered May 28, 2015 at 19:01
-
Thanks for the help, the other database is in an SQL format. What happens if data is changing on a daily basis? Would it not be better to query the database each time a request is made and print the results in EE? This way I don't have to copy the entire database. Basically I need a user to supply their username and password which is on the external database then print results of additional information for that user.– Thomas KCommented May 29, 2015 at 6:02
-
Whoa, ok, step back for a second. Can you verify an encrypted connection between the the EE server and the external database? If not, please do not even consider the over-the-web option if you are doing credentials authentication. What's the deal with this external database? Can it just live in production, or does it need to be shut down at some point, requiring actual migration? Commented May 29, 2015 at 12:09