response only available on next poll

[icing]

Version
master

Platform
Darwin xxx 22.5.0 Darwin Kernel Version 22.5.0: Mon Apr 24 20:51:50 PDT 2023; root:xnu-

Description

See curl/curl#11203 for the user report and find my analysis in the comments.

tl;dr

Calling hyper_executor_poll() triggers read of a response from the network IO layer, but does not return a HYPER_TASK_RESPONSE. Instead it returns HYPER_TASK_EMPTY.

The response sits undelivered inside the hyper stack. No socket activity is happening and curl's transfer loop checks much later again on the transfer, calling hyper_executor_poll() again. This does not do any network IO, but delivers the HYPER_TASK_RESPONSE right away.

This delays curls transfers significantly.

Please advice if you regard this as a bug in hyper or advice how curl shall reliably work around this behaviour. Thanks!

[seanmonstar]

I think I'm understanding what you're getting at. A related task has finished polling, but the one that delivers the response also needs to be polled. I'll take a closer look at what's happening in hyper's executor. What causes curl to call it again? A timer?

[icing]

Yes, curl visits the transfer again due to a timer, which can vary. This delays the overall transfer.

In the sequence I describe in the curl issue, a hyper_contex_waker was "woken" before. That, I assume triggered the IO during the first call to hyper_executor_poll() which did not return the response it reads. A subsequent call to the identical hyper_executor_poll() (not triggering IO) then gives the response.

We would expect the first call already to deliver the response.

[seanmonstar]

Ok, after investigating (remembering) how this code works, here's the update: hyper_executor_poll will do work its queue of tasks, and whenever any of them is ready, it will immediately stop and yield it in the return value. The best way for it to be used is to call hyper_executor_poll until it returns NULL, signaling that there's no more work that can be done.

There's a few internals tasks that get spawned, for connection maintenance, which when they yield, will have a value of HYPER_TASK_EMPTY. They probably aren't interesting to you, and can mostly be ignored. The best way to check if an interesting task has finished is to check any assigned userdata.

[icing]

Thanks for the advice. Will try that in curl.