Skip to content

Commit bc43de7

Browse files
committed
Got the material for post from here and there
1 parent 9ba1799 commit bc43de7

File tree

3 files changed

+113
-0
lines changed

3 files changed

+113
-0
lines changed
Lines changed: 113 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,113 @@
1+
---
2+
layout: post
3+
title: "Measuring AWS RDS production downtime"
4+
excerpt: "Getting comfortable and experienced with the inevitable RDS downtime when modifying instances"
5+
tags: [rds, databases, rds, aws]
6+
comments: true
7+
---
8+
9+
RDS is arguably the least seamless service of the AWS platform. The act of
10+
11+
While . This is
12+
also probably the reason why
13+
14+
> TL;DR When resizing AWS RDS instances, it is better to practice beforehand
15+
what will happpen, to measure in order to plan production downtime
16+
communication etc. AWS promise of a few minutes quantifies better 2-3 minutes.
17+
When a replica is attached, things can go pearshaped and support has to be
18+
poked (it is a known issue)
19+
20+
21+
## What AWS says
22+
23+
24+
## How it actually happens
25+
26+
{% highlight bash %}
27+
date
28+
while [ 1 ]
29+
do
30+
nc -vvvz strike-virginia.c3ryd7sklhx3.us-east-1.rds.amazonaws.com 3306
31+
sleep 5
32+
date
33+
done
34+
date
35+
{% endhighlight %}
36+
37+
{% highlight bash %}
38+
[giovanni@giovanni-kerad ~/KeradCode/goldenmanager.com$for I in $(seq 1 360); do date; dig +short
39+
strike-virginia.c3ryd7sklhx3.us-east-1.rds.amazonaws.com; echo; sleep 10; done
40+
Wed Sep 9 13:03:33 CEST 2015
41+
ec2-52-20-165-239.compute-1.amazonaws.com.
42+
52.20.165.239
43+
{% endhighlight %}
44+
45+
DNS switch
46+
47+
{% highlight bash %}
48+
Wed Sep 9 13:14:49 CEST 2015
49+
ec2-52-20-165-239.compute-1.amazonaws.com.
50+
52.20.165.239
51+
52+
Wed Sep 9 13:14:59 CEST 2015
53+
ec2-52-22-6-238.compute-1.amazonaws.com.
54+
52.22.6.238
55+
56+
Wed Sep 9 13:15:09 CEST 2015
57+
ec2-52-22-6-238.compute-1.amazonaws.com.
58+
52.22.6.238
59+
{% endhighlight %}
60+
Actually salvia consectetur, hoodie duis lomo YOLO sunt sriracha. Aute pop-up brunch farm-to-table odio, salvia irure occaecat. Sriracha small batch literally skateboard. Echo Park nihil hoodie, aliquip forage artisan laboris. Trust fund reprehenderit nulla locavore. Stumptown raw denim kitsch, keffiyeh nulla twee dreamcatcher fanny pack ullamco 90's pop-up est culpa farm-to-table. Selfies 8-bit do pug odio.
61+
62+
<figure class="half">
63+
<a href="/images/RDS_downsize_tiempo_master.png"><img
64+
src="/images/RDS_downsize_tiempo_master.png"></a>
65+
<a href="/images/RDS_downsize_tiempo_slave.png"><img
66+
src="/images/RDS_downsize_tiempo_slave.png"></a>
67+
<figcaption>Caption describing these two images.</figcaption>
68+
</figure>
69+
70+
71+
{% highlight bash %}
72+
Thu Aug 27 13:04:12 CEST 2015
73+
Connection to strike-virginia.c3ryd7sklhx3.us-east-1.rds.amazonaws.com 3306 port
74+
[tcp/mysql] succeeded!
75+
Thu Aug 27 13:04:17 CEST 2015
76+
Connection to strike-virginia.c3ryd7sklhx3.us-east-1.rds.amazonaws.com 3306 port
77+
[tcp/mysql] succeeded!
78+
{% endhighlight %}
79+
RUN THE DOWNSCALE COMMAND...DB Instance state in AWS console changes in a little
80+
time to Modifying
81+
{% highlight bash %}
82+
Thu Aug 27 13:04:22 CEST 2015
83+
Connection to strike-virginia.c3ryd7sklhx3.us-east-1.rds.amazonaws.com 3306 port
84+
[tcp/mysql] succeeded!
85+
Thu Aug 27 13:04:28 CEST 2015
86+
Connection to strike-virginia.c3ryd7sklhx3.us-east-1.rds.amazonaws.com 3306 port
87+
[tcp/mysql] succeeded!
88+
{% endhighlight %}
89+
…. ~ 6 minutes afterwards….refused
90+
{% highlight bash %}
91+
Thu Aug 27 13:11:43 CEST 2015
92+
nc: connect to strike-virginia.c3ryd7sklhx3.us-east-1.rds.amazonaws.com port
93+
3306 (tcp) failed: Connection refused
94+
Thu Aug 27 13:11:48 CEST 2015
95+
nc: connect to strike-virginia.c3ryd7sklhx3.us-east-1.rds.amazonaws.com port
96+
3306 (tcp) failed: Connection timed out
97+
{% endhighlight %}
98+
…. 2:13 minutes afterwards….connected again.
99+
{% highlight bash %}
100+
Thu Aug 27 13:14:01 CEST 2015
101+
Connection to strike-virginia.c3ryd7sklhx3.us-east-1.rds.amazonaws.com 3306 port
102+
[tcp/mysql] succeeded!
103+
Thu Aug 27 13:14:06 CEST 2015
104+
Connection to strike-virginia.c3ryd7sklhx3.us-east-1.rds.amazonaws.com 3306 port
105+
[tcp/mysql] succeeded!
106+
{% endhighlight %}
107+
108+
###
109+
110+
Maybe mention that when downscaling replica, which is obviously not MultiAZ, the
111+
endpoint of course stays the same, connection is lost for some minutes, AND
112+
IP of the replica instance would change (i.e. stop and start change IP, as in
113+
EC2 with no Elastic IP).

images/RDS_downsize_tiempo_master.png

48.6 KB
Loading

images/RDS_downsize_tiempo_slave.png

55.5 KB
Loading

0 commit comments

Comments
 (0)