MongoDB Transaction Pattern

Transcript

1. MongoDB Transaction Pattern Who? Ian Yang From? Intridea Inc. When?

   2013-05-15 Wed

2. Outline 1 Overview 2 Write Operation Properties 3 Two Phrase

   Commit 4 Recovering 5 Rollback 6 Multiple Applications 7 References

3. Transaction (ACID) Atomicity all or nothing. Consistency does not violate

   any integrity constraints Isolation allow concurrent execution of transactions Durability Transaction is persistent after commit

4. MongoDB Features Atomicity Only on the level of a single

   document. Write Concern Return write operation result

5. MongoDB Transaction Atomicity done or rollback Consistency rollback may leave

   system in inconsistent state. Isolation other transactions can see intermediate result. Durability write concern, replica set, journal.

6. Requirements of Write Operations in Transaction Revertible for recover and

   rollback Idempotent for resume, recover and rollback

7. Not Revertible db.users.save(user); db.users.update ({ username: "ian"}, user); db.users.update ({

   username: "ian"}, {$set: {balance: 120}});

8. Revertible db.users.update ({ username: "ian"}, {$inc: {balance: 20}}); db.users.update ({

   username: "ian"}, {$inc: {balance: -20}}); db.users.update ({ username: "ian"}, {$push: {tx: 20}}); db.users.update ({ username: "ian"}, {$pull: {tx: 20}});

9. Not Idempotent db.users.update ({ username: "ian"}, {$inc: {balance: 20}}); db.users.update

   ({ username: "ian"}, {$push: {tx: 20}}); db.users.update( {username: "ian", balance: 100}, {$inc: {balance: 20}} );

10. Idempotent db.users.update( {username: "ian", tx: {$ne: tx._id}}, {$inc: {balance: 20},

    $push: {tx: tx._id}} ); db.users.update( {username: "ian", tx: tx._id}, {$inc: {balance: -20}, $pull: {tx: tx._id}} );

11. Sample Transfer 20 dollars from Ian to Daniel

12. Create Transaction Transaction provides a unique identifier, enough information to

    recover and rollback, a document for atomic write operation.

13. var tx = { _id: ObjectId (), source: ian._id ,

    dest: daniel._id , amount: 20 }; db.transactions.create(tx);

14. Transaction State Machine State represents transaction stages Different actions should

    be taken in different stages to recover or rollback

15. Step 1: Create Transaction var tx = { _id: ObjectId

    (), source: ian._id , dest: daniel._id , amount: 20, state: 'initial ' }; db.transactions.create(tx);

16. Step 2: initial → pending db.transactions.update( {_id: tx._id , state:

    'initial '}, {$set: {state: 'pending '}} );

17. Step 3: Apply Transaction db.users.update( {_id: tx.source , tx: {$ne:

    tx._id}}, {$push: {tx: tx._id}, $inc: { balance: - tx.amount }} ); db.users.update( {_id: tx.dest , tx: {$ne: tx._id}}, {$push: {tx: tx._id}, $inc: { balance: tx.amount }} );

18. Step 4: pending → committed db.transactions.update( {_id: tx._id , state:

    'pending '}, {$set: {state: 'committed '}} );

19. Step 5: Remove Pending Transaction db.users.update( {_id: tx.source , tx:

    tx._id}, {$pull: {tx: tx._id}} ); db.users.update( {_id: tx.dest , tx: tx._id}, {$pull: {tx: tx._id}} );

20. Step 6: committed → done db.transactions.update( {_id: tx._id , state:

    'committed '}, {$set: {state: 'done'}} );

21. Recover Initial Transactions Recover from Step 2: initial → pending

22. Recover Pending Transactions Recover from Step 3: Apply Transactions

23. Recover Committed Transactions Recover from Step 5: Remove Pending Transaction

24. General Rule Recover from the Step just following the step

    transaction just entered current state. State transition creates a check point

25. Rollback Operations Better to create a new transaction in reverse

    direction for Committed Transactions and Done Transactions Use following steps to rollback initial and pending transactions

26. Rollback 1: state → canceling db.transactions.update( {_id: t._id , state:

    {$in: ['initial ', ' pending ']}}, {$set: {state: 'canceling '}} );

27. Rollback 2: Undo the Transaction db.users.update( {_id: tx.dest , tx:

    {$ne: tx._id}}, {$pull: {tx: tx._id}, $inc: { balance: - tx.amount }} ); db.users.update( {_id: tx.source , tx: tx._id}, {$pull: {tx: tx._id}, $inc: { balance: tx.amount }} );

28. Rollback 3: canceling → canceled db.transactions.update( {_id: t._id , state:

    {$in: ['canceling ']}}, {$set: {state: 'canceled '}} );

29. Recover Rollback Restart from Rollback 2 for canceling transactions

30. Conclusion Create a document for any transaction Store transaction info

    in the document Apply only Revertible and Idempotent operations

31. Conclusion (Cont.) Transit state to save current work Transit state

    if several operation branches exists

32. Enough? Only 1 application can handle a give transaction at

    any point in time.

33. Background A picks a pending transaction to recover B picks

    also a pending transaction to recover

34. Scenario 1 A is so slow, when it starts apply

    transaction, B already has done the transaction. A must rollback. If A exists abnormally, the transaction is in inconsistent state.

35. Scenario 2 B just finished Step 5: Removing Pending Transaction

    A applied transactions and exited abnormally B change state to done without error

36. Solution findAndModify lock db.transactions.findAndModify ({ query: {state: "initial", application: {$exists:

    0}}, update: {$set: {state: "pending", application: "A1"}}, new: true });

37. Lock Release db.transactions.findAndModify ({ query: {application: "A1"}, update: {$unset: {application:

    1}}, new: true });

38. Lock Forever? Application exists abnormally before release the lock Solution:

    timeout

39. Timeout Lock now = new Date(); timeout = new Date(now.valueOf

    () + 60 * 10 * 1000); db.transactions.findAndModify ({ query: {state: "initial", lockUntil: {$not: {$gte: now}}}, update: {$set: {state: "pending", lockUntil: timeout}}, new: true });

40. Client Requirements Client timeout to ensure the operation is finished

    in expected time. Check time after return from block IO Defer transactions cleanup

41. References http://docs.mongodb.org/ manual/tutorial/ perform-two-phase-commits/ http://docs.mongodb.org/ manual/tutorial/ isolate-sequence-of-operations/

42. Q & A