HTTP2 multiplexed requests get stuck in presence of "slow readers", Stream/Connection window is not being updated #2899
Description
Version
hyper 0.14.19
h2 0.3.13
Platform
Fedora 36
Linux fedora-wb-dev 5.17.11-300.fc36.x86_64 #1 SMP PREEMPT Wed May 25 15:04:05 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Description
We use hyper as a server and as a client in distributed application. Hyper client is configured as .http2_only(true).
Some GET requests are getting stuck. Those requests stream large response body (tens of megabytes). Some part of response body is transferred (several megabytes) and then both server and client are stuck with zero bytes being transferred. This only happens with HTTP2 in use.
The following code always reproduces this on my machine:
# Cargo.toml
[package]
name = "hyperreader"
version = "0.1.0"
edition = "2021"
# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html
[dependencies]
hyper = { version = "0.14.13", features = ["full"] }
tokio = { version = "1.12.0", features = ["full"] }
tokio-stream="*"// main.rs
use std::convert::Infallible;
use std::error::Error;
use std::mem::forget;
use std::net::SocketAddr;
use std::sync::Arc;
use std::time::Duration;
use hyper::{Body, Request, Response};
use hyper::client::HttpConnector;
use hyper::service::{make_service_fn, service_fn};
use tokio::spawn;
use tokio::time::sleep;
use tokio_stream::StreamExt;
const URL: &'static str = "http://127.0.0.1:1234";
const PORT: u16 = 1234;
const BODY_SIZE: usize = 10 * 1024 * 1024;
#[tokio::main]
async fn main() -> Result<(), Box<dyn Error>> {
spawn(
hyper::server::Server::bind(&SocketAddr::from(([127, 0, 0, 1], PORT)))
.serve(make_service_fn(|_| async { Ok::<_, Infallible>(service_fn(serve_file)) }))
);
let client = hyper::client::Builder::default()
.http2_only(true)
.build::<_, Body>(HttpConnector::new());
let client = Arc::new(client);
for i in 0..3 {
eprintln!("Sending {}", i);
let uri = URL.parse().unwrap();
let response = client.get(uri).await.unwrap();
eprintln!("Received response {}", i);
forget(response);
}
eprintln!("Sleeping 1 sec...");
sleep(Duration::from_secs(1)).await;
eprintln!("Sending request...");
let uri = URL.parse().unwrap();
let mut response = client.get(uri).await.unwrap();
eprintln!("Received response...");
let mut count = 0;
while let Some(chunk) = response.body_mut().next().await {
count += chunk?.len();
println!("{}/{} bytes", count, BODY_SIZE);
}
Ok(())
}
async fn serve_file(_req: Request<Body>) -> Result<Response<Body>, Infallible> {
let data = vec![0; BODY_SIZE];
Ok(Response::new(Body::from(data)))
}(note that the constants may be specific for a particular machine and may need tweaking for this to reproduce)
I expected to see this happen:
The program would run to completion and would print:
Sending 0
Received response 0
Sending 1
Received response 1
Sending 2
Received response 2
Sleeping 1 sec...
Sending request...
Received response...
// many lines skipped
10485760/10485760 bytes
Instead, this happened:
The program prints the following lines and then does nothing:
Sending 0
Received response 0
Sending 1
Received response 1
Sending 2
Received response 2
Sleeping 1 sec...
Sending request...
Received response...
In wireshark I see that neither client nor server don't send WINDOW_UPDATE frames during data transfer (except at the connection establishment). I suspect that this is a cause for this behavior.