Managing data in development production cycle

One of the issues hotly dicussed in Drupal community, is the process of relating the dev server with the production. Imagine, you need to make some changes to the web site. You make a snapshot, and create a dev server. You make changes on a dev server, and then you want to move them to production. But, doh! On the production, you have some new users and content. What is the optimal way to implement the changes and merge the data back, keeping the data and config loses minimal? I asked the devs from the Google+ Drupal Community to share their best practices, and have come up with my own set of practices that I am using as well. So, here are some suggestions.

1. Keep the loop in mind.

Think of making changes as not of a one way stream, but of a loop.
Wrong way: Make a snapshot. Develop. Merge. Develop. Merge.
Right way: Make a snapshot. Develop. Merge. Make a snapshot.

2. Make incremental iterations.

The rule of thumb is that the more changes you make to the dev version, the harder it will be to merge them back. Do smaller changes and merge them back regularly, in smaller portions. I prefer to roll out one large feature, or a set of smaller features, per iteration, rather then storing them up and then hurdling them out.

3. Do database snapshots both before and after merging an iteration.

The iterations are tested before merging to production, but let's face it - there are cases, when the problem surfaces later. And sometimes, that problem can be unexpectedly large. This is rare, but a snapshot of the pre-iteration database, and the files (if they get overwritten), can save the day. You may ask, why take a snapshot after making an iteration? Because it's the next restore point now, and because you will use that snapshot in dev.

4. Use the snapshot of the production for your next iteration.

Have you every run in a situation, when you make an iteration, and merge it into production. Then, you do another iteration, and merge it into production. Then you make a third iteration. You merge the third iteration into production, and production has now pages that look broken. Why? Because content was added into production between those iterations, that you did not take to account while developing and testing. So, after a successful merge, take a snapshot of production, and develop on it, rather than continuing with your setup from the previous iteration.

5. Handle data types separate.

Which are the data types that we have? We have:

  1. Code. Themes and modules.
  2. Config. Settings, stored in database. (Settings, that are stored in Drupal object codes, yml, and JSON, can be handled like code proper).
  3. Content. Nodes and blocks.
  4. Files. Mostly images and multimedia.

Handle these separately. Code in git. Config - with Features or Settings modules. Content - with node export. In most cases, files the easiest things to syncronize.

How does this work in practice?

  1. I take a snapshot and put it on the dev server. This is easy, since you can do it raw dump/import and copy files.
  2. I work on the dev server.
  3. Then, I commit the code to git.
  4. Export the changed settings with the Settings module.
  5. On production, I make a backup snapshot.
  6. Put the site offile.
  7. Import the settings first, and check out the code and copy the files afterwards. (If you do the code and files first, you may get WSOD and be unable to import required settings).
  8. Clear cache, and verify changes have been applied succesfully.
  9. Put the site online and ask a few more prople to test and click around.
  10. I take a snapshot and put it in the dev server. It will be the basis for the next iteration.