Three Month Schema Change

I've just finished a schema change that took a lot longer than expected. I now understand how I could have done it a lot quicker.

I'm used to working in an Investment Bank where we have a few hours after New York close on Friday and Tokyo opens on a Monday to perform installs and upgrades.

Feedme is online 24/7 and has a few thousand users, so I decided to try and perform a schema migration online. This turned out to be a lot more difficult than I first thought. Probably because I allowed scope creep and ended up rewriting much of the server side of the application.

Drivers

1. I needed to change the entity groups to move all a user's data into a single entity group.  I could not make updates to a single user in a single transaction (GAE Transactions only work inside single entity groups).
2. I needed to change the URLs that a 3rd parties used to POST data to my server.
3. I needed to make some changes to my MVP framework.

Problems

1. Transactions on single entity groups, means that I couldn't make updates to a users data in a single transaction. So migrating all their data into single entity group could not be done in a single transaction.
2. Distributed subscriptions could miss updates if I wasn't subscribed on my new URLs before unsubscribing the old ones.
3. Always online means end users should not notice a break in service, duplicates or missing data.

Finite State Machines

The transaction and distributed subscription issues meant that I would have to perform asynchronous operations optimistically. This would have to happen over transaction boundaries. So I've have to journal  the fact that I'd performed the operation and deal with the results or errors in another transaction. I chose to do this by simply storing a state in each record that I was changing and having a  FSM manage what should happen on future events. I did this on several levels with a hierarchy of FSMs.