60

This is a topic which is a major headache for us and I'd like to see how other folks tackle it.

We manage some pretty big sites which run on ExpressionEngine. Our clients sometimes ask for changes and enhancements which take a long time to develop, and so we naturally do that development work on a separate copy of their live website. However, when we come to bring those changes back to the production site it's a nightmare, for the following reasons

  1. Software version changes (i.e. modules added/replace, EE upgrades performed, module upgrades performed)
  2. EE 'Structural' changes (channels/fields/templates/snippets which have been added/removed/changed)
  3. Content changes

1 isn't too painful, 2 is a pain but we've developed some tooling to help us with that (and I believe Christopher Imrie's Site Manager can help with that too). It's number 3 which is a major problem. It's not just the fact that new content has been added, it's the inter-dependencies between all the various tables (and add-on tables) which make it almost impossible to handle.

How are you handling this?

Improve this question

8 Answers 8

Reset to default
59

A long time ago I came across a graphic which presented it like this:

Code should flow in this direction:

Development >> Staging >> Production

Content, on the other hand, should flow in the opposite direction:

Development << Staging << Production

That is, you should populate your development and staging environments with live content (to test your latest changes), then move your template code to staging and production. In this case code refers to your ExpressionEngine template code.

This is obviously easier said than done in ExpressionEngine since content and structure is stored together in the database, and there is no migration system to speak of. However, I believe it's still important to try and follow this as closely as possible.

Therefore, the way I do things is to document any changes you make to your content structure during development (channels, custom fields etc). Then, if possible, make those structural changes on a staging site, before bringing your template code onto staging. If that works, make the structural changes on production, before updating your template code.

Ideally, any structural changes you make won't break your existing template code (e.g. try to avoid renaming custom fields just for the sake of it). This is why it's important to test things on a staging site first, so you know your live site will not go down as you update your content structure.

Make sense?

Improve this answer
3
  • It makes sense ,but in my experience we can't develop on dev without also making changes to content. For example, a site was recently migrated over to Structure. That required adding new pages, merging channels, moving EE File fields to Channel Images etc etc. It would not be possible to make these changes on the production site prior to go-live because they are intrinsically linked to the structural and code changes which we made. The only solution I can see for this is to make a "content freeze" on the production site then make the changes on dev - a difficult thing to sell to the customer
    – James
    Nov 16, 2012 at 14:18
  • 11
    In my experience a "content freeze" is something that is pretty common and most of my clients have been able to accommodate it fairly well. It's something that you don't want to drop on them out of nowhere, but if you present it to them in an upfront and the nature of the beast type way, usually they are understanding. Not to mention it's fairly common across other platforms similar to EE.
    – Bryant
    Nov 16, 2012 at 14:47
  • 6
    I like the idea of a "content freeze" for static/brochure websites, but how would you handle it on an eCommerce website?
    – Mutual
    Nov 26, 2012 at 21:35
25

In short, we handle these issues on a case-by-case basis. Adrian described it well. Here are two slides from a presentation I did on the topic: schema flow & content flow

The thing that makes the described process easy for us is that we rarely work on multiple large features at a time. We build one, deploy. Build another, deploy. So we don't have to pick and choose which features get deployed at certain times (which would be a huge pain).

Improve this answer
1
  • Adrian's comment about code flow and content flow combined with Erik's about one feature at a time are a great start. They don't solve every problem but I'm not sure there is a silver bullet if you let things get too complex.
    – Tim Print
    Nov 16, 2012 at 16:39
9

I outlined some of the methodologies I have encountered for handling migrations in my EECI talk where I demoed Site Manager. You can find my slide deck here:

https://speakerdeck.com/chrisimrie/eeci-2012-us-ee-and-web-agency

Thanks for the kind words about Site Manager. Early days, but it is definitely our solution to migrating content between local/dev/production sites.

Improve this answer
1
  • 1
    How is Site Manager coming along? Still at about the same state as it was at EECIConf 2012?
    – Stuart McCoy
    Dec 5, 2012 at 16:43
3

I follow pretty much just what Adrian Macneil explained. I use git and Beanstalk for deployments. It makes it easy to deploy one branch (dev for example) to the staging server online. Once that code is all ready, I merge it with the production branch, and deploy that to the live/production server.

For future reference, there's an add-on being developed for syncing fields, channels, etc. between servers. It's still in alpha, but it looks very promising.

https://github.com/ckimrie/Site-Manager

Improve this answer
2

Just published an addon on github to trace db changes into files that can be deployed to your staging server. On the staging server you can re-execute the db changes made on your local development environment (manual or use a hook).

The basics work, but lots of room for improvement.

https://github.com/fccotech/ee-db-trace

Feel free to use/extend/branch etc.. (or create issues if you have trouble making it work)

I also created a forum thread in the EllisLab forums because this might be to META for SE:

http://ellislab.com/forums/viewthread/242709/

Improve this answer
1
2

I would recommend looking at the database migration features of HeidiSQL and Navicat. They both provide robust migration mechanisms that can synchronize the MySQL database underlying your EE instances.

Improve this answer
3
  • I've been experimenting with merging dev and production databases with Git. Basically exporting the databases as .sql files in separate branches, then merging them & managing conflicts with a diff tool. I've only done preliminary experiments though with mixed success - what is your opinion on that?
    – kmgdev
    Nov 19, 2012 at 22:00
  • I really like the idea in theory. I haven't tried it personally but it seems like it certainly could work. The main issue you'll run into with either method is that EE's primary keys for it's database records are not globally unique - if new entries or fields are added on both sides, different entries will be wanting to use the same IDs. That's where it gets messy, either way you do it.
    – Isaac Raway
    Nov 28, 2012 at 17:35
  • 1
    One quick word of caution about this sort of approach: You need to be carefully keeping multiple versions of SQL dumps in Git - because of the way that Git calculates deltas, it's very easy for minor DB changes to result in effectively adding the full size of the dump to the repo size for each change, which massively reduces one of Git's main advantages - it's lightweightness. Very large repos can get slow to work with. For "normal" apps dumping and versioning the scheme works fine, but EE's data structure doesn't make that straightforward.
    – Tom Davies
    Nov 30, 2012 at 19:45
1

I documented our workflow for this in response to "Move ExpressionEngine from development to live", including a link to a video of our git/MySQL steps.

Improve this answer
0

I am doing this all day now. I have 3 installs of EE running the same website. One is a personally hosted version of the website, the other is the client provided preview site and lastly we have the live site.

As Adrian answered I think he totally nailed the flow, but as others have said freezing content is sometimes not an option. You have to then figure out a way to make edits in your development environment and get those edits to fit as closely as possible to the live site.

To echo what Erik said by only deploying one feature at time, that is also helpful too. It keeps overlapping edits to minimum. Even still sometimes that is not an option either. Some clients request a batch of changes at once and then want to preview them before they go live.

What I've done is I track everything!

  1. First thing I do is make a full backup of all site files and database from the live site.
  2. I'll load these files into my development site. This gets me to the latest version and I'll current at least temporarily. If content is added to the live site I'm not going to get those changes. You'll have to be aware of this and make changes to your code if needed when you move to the live site.
  3. Track Everything: I track every template I edit. I track by line number and code added so its easy to push to the live site.
  4. Breaking up the updates you make into babysteps is useful. By babysteps I mean i create an outline of the steps I'm taking and I've very detailed about it too. (This does add time to the project, but I would prefer to add time in this way than adding time trying to figure out something I've already done).
  5. If i add modules, I track which modules I've installed and the steps to get it working including all of the settings.
  6. If i update js or css I also track by line number and code added.

So once my edits are done I end up with a lengthy document that contains every step I took from start to finish. Once approved I begin migrating my changes to the client hosted preview site and hopefully in this stage we nail any issues before going to the live site. Of course i adjust my notes as needed. That step is like a second chance to run through my steps in my document. I test my notes against the process.

I repeat all of these steps once again when i migrate to the live site if all goes well on the preview site.

I'm in the middle of this now. At the very end, once all of my changes are on the live site I'm going to export from the live site and update both of my staging websites. That way all of the sites will be on the most current version.

I have some specific issues in regards to Entry ID numbers not matching in the development, staging, and production sites. I want to get my databases in sync at the end of this procoess so that all of my templates will sync up as well. Right now I have to make a few changes to some templates so they load the correct entry by the ID.

Excellent documentation is the key to success here.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.