|
| 1 | +--- |
| 2 | +layout: post |
| 3 | +title: "Measuring AWS RDS production downtime" |
| 4 | +excerpt: "Getting comfortable and experienced with the inevitable RDS downtime when modifying instances" |
| 5 | +tags: [rds, databases, rds, aws] |
| 6 | +comments: true |
| 7 | +--- |
| 8 | + |
| 9 | +RDS is arguably the least seamless service of the AWS platform. The act of |
| 10 | + |
| 11 | +While . This is |
| 12 | +also probably the reason why |
| 13 | + |
| 14 | +> TL;DR When resizing AWS RDS instances, it is better to practice beforehand |
| 15 | + what will happpen, to measure in order to plan production downtime |
| 16 | + communication etc. AWS promise of a few minutes quantifies better 2-3 minutes. |
| 17 | + When a replica is attached, things can go pearshaped and support has to be |
| 18 | +poked (it is a known issue) |
| 19 | + |
| 20 | + |
| 21 | +## What AWS says |
| 22 | + |
| 23 | + |
| 24 | +## How it actually happens |
| 25 | + |
| 26 | +{% highlight bash %} |
| 27 | +date |
| 28 | + while [ 1 ] |
| 29 | + do |
| 30 | + nc -vvvz strike-virginia.c3ryd7sklhx3.us-east-1.rds.amazonaws.com 3306 |
| 31 | + sleep 5 |
| 32 | + date |
| 33 | + done |
| 34 | +date |
| 35 | +{% endhighlight %} |
| 36 | + |
| 37 | +{% highlight bash %} |
| 38 | +[giovanni@giovanni-kerad ~/KeradCode/goldenmanager.com$for I in $(seq 1 360); do date; dig +short |
| 39 | +strike-virginia.c3ryd7sklhx3.us-east-1.rds.amazonaws.com; echo; sleep 10; done |
| 40 | +Wed Sep 9 13:03:33 CEST 2015 |
| 41 | +ec2-52-20-165-239.compute-1.amazonaws.com. |
| 42 | +52.20.165.239 |
| 43 | +{% endhighlight %} |
| 44 | + |
| 45 | +DNS switch |
| 46 | + |
| 47 | +{% highlight bash %} |
| 48 | +Wed Sep 9 13:14:49 CEST 2015 |
| 49 | +ec2-52-20-165-239.compute-1.amazonaws.com. |
| 50 | +52.20.165.239 |
| 51 | + |
| 52 | +Wed Sep 9 13:14:59 CEST 2015 |
| 53 | +ec2-52-22-6-238.compute-1.amazonaws.com. |
| 54 | +52.22.6.238 |
| 55 | + |
| 56 | +Wed Sep 9 13:15:09 CEST 2015 |
| 57 | +ec2-52-22-6-238.compute-1.amazonaws.com. |
| 58 | +52.22.6.238 |
| 59 | +{% endhighlight %} |
| 60 | +Actually salvia consectetur, hoodie duis lomo YOLO sunt sriracha. Aute pop-up brunch farm-to-table odio, salvia irure occaecat. Sriracha small batch literally skateboard. Echo Park nihil hoodie, aliquip forage artisan laboris. Trust fund reprehenderit nulla locavore. Stumptown raw denim kitsch, keffiyeh nulla twee dreamcatcher fanny pack ullamco 90's pop-up est culpa farm-to-table. Selfies 8-bit do pug odio. |
| 61 | + |
| 62 | +<figure class="half"> |
| 63 | + <a href="/images/RDS_downsize_tiempo_master.png"><img |
| 64 | +src="/images/RDS_downsize_tiempo_master.png"></a> |
| 65 | + <a href="/images/RDS_downsize_tiempo_slave.png"><img |
| 66 | +src="/images/RDS_downsize_tiempo_slave.png"></a> |
| 67 | + <figcaption>Caption describing these two images.</figcaption> |
| 68 | +</figure> |
| 69 | + |
| 70 | + |
| 71 | +{% highlight bash %} |
| 72 | +Thu Aug 27 13:04:12 CEST 2015 |
| 73 | +Connection to strike-virginia.c3ryd7sklhx3.us-east-1.rds.amazonaws.com 3306 port |
| 74 | +[tcp/mysql] succeeded! |
| 75 | +Thu Aug 27 13:04:17 CEST 2015 |
| 76 | +Connection to strike-virginia.c3ryd7sklhx3.us-east-1.rds.amazonaws.com 3306 port |
| 77 | +[tcp/mysql] succeeded! |
| 78 | +{% endhighlight %} |
| 79 | +RUN THE DOWNSCALE COMMAND...DB Instance state in AWS console changes in a little |
| 80 | +time to Modifying |
| 81 | +{% highlight bash %} |
| 82 | +Thu Aug 27 13:04:22 CEST 2015 |
| 83 | +Connection to strike-virginia.c3ryd7sklhx3.us-east-1.rds.amazonaws.com 3306 port |
| 84 | +[tcp/mysql] succeeded! |
| 85 | +Thu Aug 27 13:04:28 CEST 2015 |
| 86 | +Connection to strike-virginia.c3ryd7sklhx3.us-east-1.rds.amazonaws.com 3306 port |
| 87 | +[tcp/mysql] succeeded! |
| 88 | +{% endhighlight %} |
| 89 | +…. ~ 6 minutes afterwards….refused |
| 90 | +{% highlight bash %} |
| 91 | +Thu Aug 27 13:11:43 CEST 2015 |
| 92 | +nc: connect to strike-virginia.c3ryd7sklhx3.us-east-1.rds.amazonaws.com port |
| 93 | +3306 (tcp) failed: Connection refused |
| 94 | +Thu Aug 27 13:11:48 CEST 2015 |
| 95 | +nc: connect to strike-virginia.c3ryd7sklhx3.us-east-1.rds.amazonaws.com port |
| 96 | +3306 (tcp) failed: Connection timed out |
| 97 | +{% endhighlight %} |
| 98 | +…. 2:13 minutes afterwards….connected again. |
| 99 | +{% highlight bash %} |
| 100 | +Thu Aug 27 13:14:01 CEST 2015 |
| 101 | +Connection to strike-virginia.c3ryd7sklhx3.us-east-1.rds.amazonaws.com 3306 port |
| 102 | +[tcp/mysql] succeeded! |
| 103 | +Thu Aug 27 13:14:06 CEST 2015 |
| 104 | +Connection to strike-virginia.c3ryd7sklhx3.us-east-1.rds.amazonaws.com 3306 port |
| 105 | +[tcp/mysql] succeeded! |
| 106 | +{% endhighlight %} |
| 107 | + |
| 108 | +### |
| 109 | + |
| 110 | +Maybe mention that when downscaling replica, which is obviously not MultiAZ, the |
| 111 | +endpoint of course stays the same, connection is lost for some minutes, AND |
| 112 | +IP of the replica instance would change (i.e. stop and start change IP, as in |
| 113 | +EC2 with no Elastic IP). |
0 commit comments