Skip to content

AutonomousDatabase CR stuck at Provisioning or Terminating #112

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
laguvard opened this issue Jun 19, 2024 · 4 comments
Closed

AutonomousDatabase CR stuck at Provisioning or Terminating #112

laguvard opened this issue Jun 19, 2024 · 4 comments

Comments

@laguvard
Copy link

During a series of about a dozen AutonomousDatabase create/delete cycles, the CR remained stuck 3 times at either the Provisioning or the Terminating state even after the OCI ADB transitioned to Available or Terminated. This was due to a transient 500 error.

  1. It appears that there was no attempt to recover. The logs show:
DEBUG 2024/06/14 04:14:56.145788 asm_amd64.s:1650: Retry policy to use: {MaximumNumberAttempts=1, MinSleepBetween=0, MaxSleepBetween=0, ExponentialBackoffBase=0, NonEventuallyConsistentPolicy=<nil>}
  1. There was no indication on the CR (e.g., a condition) to show the error.

Here's the error:

INFO 2024/06/14 04:14:57.577023 client.go:463: Dump Response HTTP/1.1 500 Internal Server Error^M
Connection: close^M
Content-Length: 502^M
Cache-Control: must-revalidate,no-cache,no-store^M
Content-Type: text/html;charset=iso-8859-1^M
Date: Fri, 14 Jun 2024 04:14:56 GMT^M
Opc-Request-Id: 527875f6252c7e89ba40f484d3b47ef9/8077408D57D9A0CABBEC59A796F17A7F/C0B75A80CCCA38741AFB0B4820DD1D99^M
Strict-Transport-Security: max-age=31536000; includeSubDomains;^M
X-Content-Type-Options: nosniff^M
^M
<html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
<title>Error 500 Request failed.</title>
</head>
<body><h2>HTTP ERROR 500 Request failed.</h2>
<table>
<tr><th>URI:</th><td>/20160918/autonomousDatabases/ocid1.autonomousdatabase.oc1.iad.anuwcljs3lzhqhyadm6k5rdkjelgvsbp3k3c4ym4jqaivgemcsp6mgoqgqeq</td></tr>
<tr><th>STATUS:</th><td>500</td></tr>
<tr><th>MESSAGE:</th><td>Request failed.</td></tr>
<tr><th>SERVLET:</th><td>jersey</td></tr>
</table>

</body>
</html>
DEBUG 2024/06/14 04:14:57.577078 client.go:465: Error response could not be parsed due to: invalid character '<' looking for beginning of value
2024-06-14T04:14:57Z    ERROR   controllers.database.AutonomousDatabase.validateOperation.manageError   UpdateFailed    {"Namespace/Name": {"name":"oraclepdb-152abed7-367a-4631-b94e-156cf7b3ae59-adb","namespace":"test"}, "error": "[Error returned by Database Service. Http Status Code: 500. Error Code: BadErrorResponse. Opc request id: c35ed8b3f385c5d8191cfce81e1e6ee0/27F2B9A32C934160A70414F24047B67E/F7C056F133118AEEBC3207A1A6CD87FA. Message: Failed to parse json from response body due to: invalid character '<' looking for beginning of value. With response body <html>\n<head>\n<meta http-equiv=\"Content-Type\" content=\"text/html;charset=ISO-8859-1\"/>\n<title>Error 500 Request failed.</title>\n</head>\n<body><h2>HTTP ERROR 500 Request failed.</h2>\n<table>\n<tr><th>URI:</th><td>/20160918/autonomousDatabases/ocid1.autonomousdatabase.oc1.iad.anuwcljs3lzhqhyadm6k5rdkjelgvsbp3k3c4ym4jqaivgemcsp6mgoqgqeq</td></tr>\n<tr><th>STATUS:</th><td>500</td></tr>\n<tr><th>MESSAGE:</th><td>Request failed.</td></tr>\n<tr><th>SERVLET:</th><td>jersey</td></tr>\n</table>\n\n</body>\n</html>\n.\nOperation Name: GetAutonomousDatabase\nTimestamp: 2024-06-14 04:14:54 +0000 GMT\nClient Version: Oracle-GoSDK/65.49.3\nRequest Endpoint: GET https://database.us-ashburn-1.oraclecloud.com/20160918/autonomousDatabases/ocid1.autonomousdatabase.oc1.iad.anuwcljs3lzhqhyadm6k5rdkjelgvsbp3k3c4ym4jqaivgemcsp6mgoqgqeq
@laguvard
Copy link
Author

Version 1.1.0

@ting-lan-wang
Copy link
Member

ting-lan-wang commented Oct 24, 2024 •
edited
Loading

Hi @laguvard thanks for reporting the issues. For the first one, it can be fixed by increasing the retry number in each requests, and make the controller requeue the request when the OCI ADB is in intermediate states like Provisioning or Terminating. The both issues will be fixed in the next release.

@laguvard
Copy link
Author

Thanks @ting-lan-wang. Does Closed mean that you've merged the fixes?

@ting-lan-wang
Copy link
Member

Hi @laguvard The fixes were delivered internally. They will go public in the next version (v1.2.0).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@laguvard @ting-lan-wang