Description
What steps did you take and what happened:
I deploy an app doing GRPC ServerSideStreaming, during a "long" period of time (few minutes). With Contour and the GatewayAPI, I can't configure a timeout above 15s and so I keep getting this error in my app:
16:38:16.523 [main] ERROR -- ❌ Error after PT15.013748S during ServerSide Streaming replies: CANCELLED
16:38:16.524 [main] ERROR -- ❌ Error:
io.grpc.StatusRuntimeException: CANCELLED
I deployed the app with the following GRPCRoute
:
apiVersion: gateway.networking.k8s.io/v1
kind: GRPCRoute
metadata:
name: grpc
spec:
parentRefs:
- name: gateway
namespace: projectcontour
hostnames:
- server.k3d-for-grpc.local
rules:
- matches:
- method:
service: com.gitlab.davinkevin.experimentations.armeria.grpc.hello.HelloService
method: Hello
- method:
service: com.gitlab.davinkevin.experimentations.armeria.grpc.hello.HelloService
method: Hellos
- method:
service: com.gitlab.davinkevin.experimentations.armeria.grpc.hello.HelloService
method: Dialog
- method:
service: grpc.reflection.v1alpha.ServerReflection
method: ServerReflectionInfo
backendRefs:
- name: server
port: 8081
sessionPersistence:
absoluteTimeout: 18s
idleTimeout: 28s
And the Contour gateway is deployed with ContourDeployment
:
kind: ContourDeployment
apiVersion: projectcontour.io/v1alpha1
metadata:
namespace: projectcontour
name: gateway
spec:
runtimeSettings:
envoy:
timeouts:
requestTimeout: 20s
connectionIdleTimeout: 25s
connectionShutdownGracePeriod: 30s
connectTimeout: 35s
delayedCloseTimeout: 40s
maxConnectionDuration: 45s
streamIdleTimeout: 50s
NOTE: I used different values to see which one has an impact on the behavior. It seems none were able to change something and I'm always blocked at 15s… even if I set all of them to 600s
.
What did you expect to happen:
I expect one of those values to have an impact on the GRPC communication.
Anything else you would like to add:
I searched both the GatewayAPI spec and the Contour reference documentation and found nothing. Sorry if I missed the key, but it means it's pretty well hidden 😅.
Environment:
- Contour version: 1.32.0
- Kubernetes version: (use
kubectl version
): 1.30+ - Kubernetes installer & version: k3s/k3d
- Cloud provider or hardware configuration: docker-for-mac
- OS (e.g. from
/etc/os-release
): Linux distribution built-in docker-for-mac