ghcr.io/docling-project/docling-serve-cu124:latest crashes as soon it uses GPU

[Gylfkxjyjdll]

I encounter the same issue, and this solution does not provide the desired outcome.
It is important to note that the command „count: 0 # nvidia-smi“ does not utilize the GPU -> it has to be at least count: 1. this is not the same as: id

Originally posted by @Gylfkxjyjdll in #132

[dolfim-ibm]

We just built a new release image yesterday. Can you please give it another try?

Do you have more details on the environment you are using? Which CUDA version, which GPU?

[Gylfkxjyjdll]

I attempted the latest release yesterday, and the behavior remains the same.

nvidia-smi --query-gpu=name --format=csv
name
NVIDIA GeForce RTX 5090
NVIDIA GeForce RTX 4070 Ti SUPER
NVIDIA GeForce RTX 3090

GPU Driver: NVIDIA-SMI 570.144 Driver Version: 570.144 CUDA Version: 12.8

docling-serve-1  | ERROR:docling_serve.engines.async_local.worker:Worker 0 failed to process job 67b77649-d8ef-4e84-939f-2ee9125a6796: 500: CUDA error: no kernel image is available for execution on the device
docling-serve-1  | CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
docling-serve-1  | For debugging consider passing CUDA_LAUNCH_BLOCKING=1
docling-serve-1  | Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

bash-5.1$ python -c "import torch; print('CUDA:', torch.cuda.is_available(), '\nArchs:', torch.cuda.get_arch_list())"
CUDA: True 
Archs: ['sm_50', 'sm_60', 'sm_70', 'sm_75', 'sm_80', 'sm_86', 'sm_90']

okay, only 3090 works, 4070 and 5090 not

[heapsoftware]

Im using 5090s as well, so need support for this too.

Per OpenwebUI docs about using docling with openwebui and wanting to do PDF image OCRing, when I run the following per the OWUI docs, it crashes with an error:

docker run --gpus all -p 5001:5001 -e DOCLING_SERVE_ENABLE_UI=true --platform linux/amd64 quay.io/docling-project/docling-serve

I get.....
│ /opt/app-root/lib64/python3.12/site-packages/transformers/utils/import_utils │
│ .py:2048 in getattr │
│ │
│ 2045 │ │ │ │ module = self._get_module(self._class_to_module[name] │
│ 2046 │ │ │ │ value = getattr(module, name) │
│ 2047 │ │ │ except (ModuleNotFoundError, RuntimeError) as e: │
│ ❱ 2048 │ │ │ │ raise ModuleNotFoundError( │
│ 2049 │ │ │ │ │ f"Could not import module '{name}'. Are this obje │
│ 2050 │ │ │ │ ) from e │
│ 2051 │
│ │
│ ╭───────────────────────────────── locals ─────────────────────────────────╮ │
│ │ name = 'RTDetrForObjectDetection' │ │
│ │ self = <module 'transformers' from │ │
│ │ '/opt/app-root/lib64/python3.12/site-packages/transformers/__ini… │ │
│ ╰──────────────────────────────────────────────────────────────────────────╯ │
╰──────────────────────────────────────────────────────────────────────────────╯
ModuleNotFoundError: Could not import module 'RTDetrForObjectDetection'. Are
this object's requirements defined correctly?

When I try the cuda version it says no GPUs found, since im running 5090s and 12.4 doesnt support those.