RUST-1894 Retry KMS requests on transient errors #1281
Conversation
isabelatkinson
force-pushed
the
retry-kms-requests
branch
from
a4a663e to
f0de22a
Compare
| @@ -1,6 +1,6 @@ | |||
| [profile.default] | |||
| test-threads = 1 | |||
| default-filter = 'not test(test::happy_eyeballs)' | |||
| default-filter = 'not test(test::happy_eyeballs) and not test(kms_retry)' | |||
There was a problem hiding this comment.
POC for skipping tests locally that need additional setup (in this case, a mock server running). We don't need to include this in the CI filter because the tests in csfle are only run by the CSFLE tasks, in which case the mock server needed here has been started by start-servers.sh.
Cargo.toml
Outdated
| @@ -91,7 +91,7 @@ hmac = "0.12.1" | |||
| once_cell = "1.19.0" | |||
| log = { version = "0.4.17", optional = true } | |||
| md-5 = "0.10.1" | |||
| mongocrypt = { git = "https://github.com/mongodb/libmongocrypt-rust.git", branch = "main", optional = true, version = "0.2.0" } | |||
| mongocrypt = { git = "https://github.com/isabelatkinson/libmongocrypt-rust", branch = "retry-kms", optional = true } | |||
There was a problem hiding this comment.
will revert once mongodb/libmongocrypt-rust#37 is merged, this is causing the cargo-deny failure
| @@ -3493,6 +3493,167 @@ async fn range_explicit_encryption_defaults() -> Result<()> { | |||
| Ok(()) | |||
| } | |||
|
|
|||
| // Prose Test 24. KMS Retry Tests | |||
| #[tokio::test] | |||
| // using openssl causes errors after configuring a network failpoint | |||
There was a problem hiding this comment.
I was getting opaque SSL errors from the Python server after configuring network errors with openssl-tls enabled - we're not gaining any useful coverage by running this test on rustls and openssl, so it didn't seem worth it to try to make it work
| let certificate_file = std::fs::read(&certificate_file_path).unwrap(); | ||
|
|
||
| let set_failpoint = |kind: &str, count: u8| { | ||
| // create a fresh client for each request to avoid hangs |
There was a problem hiding this comment.
Some more odd behavior: when I reused a single HTTP client for setting failpoints, this line would hang for 90 seconds and then connect successfully. I saw a few issues filed in reqwest's repo describing similar behavior that fixed it by re-creating the client for each request, so I just did the same here.
No description provided.