do_picture_description doesnt work with local HuggingFaceTB--SmolVLM-256M-Instruct

[khabibulloevm]

Hi @dolfim-ibm, I'm trying to use do_picture_description with a locally downloaded model (HuggingFaceTB--SmolVLM-256M-Instruct) via the /v1alpha/convert/file/async endpoint.

I'm passing the parameter as:

"do_picture_description": true,
"picture_description_local": {
  "repo_id": "/docling-models/HuggingFaceTB--SmolVLM-256M-Instruct"
}

However, I consistently receive a 422 error:

Input should be a valid dictionary or object to extract fields from

Before calling the endpoint, I downloaded all required models using docling-tools models download -o /docling-models, and I’m launching the API with:

docling-serve run --artifacts-path /docling-models

Here are the model folders I have under /docling-models:

I saw in another issue that do_picture_description might not yet be fully supported in /v1alpha/convert/file/async.

Could you please confirm:

Whether do_picture_description is supported for this endpoint?

Whether using a local model for picture_description_local should work in this context?

If not, is there an alternative endpoint or method recommended for image captioning with local models?

Thanks in advance for the clarification!

[dolfim-ibm]

I think this is similar to what is discussed in #196. It should get solved soon.

[khabibulloevm]

I think this is similar to what is discussed in #196. It should get solved soon.

When do_picture_description: true is set, is it required to explicitly pass the picture_description_local parameter?

Or is it sufficient to run docling-serve with --artifacts-path, assuming that all required models (like SmolVLM) are already present in the artifacts directory?

[vaclcer]

With v0.12.0, /v1alpha/convert/source exports into MD fine. Great!

But /v1alpha/convert/file is still broken. When using a standard:

"do_picture_description": "true",
"picture_description_local": {
    "repo_id": "HuggingFaceTB/SmolVLM-256M-Instruct",
    "prompt": "Describe the image in detail.",
    "generation_config": {
        "max_new_tokens": 200,
        "do_sample": False   
    }
}

I get response:

File "/opt/app-root/src/docling_serve/helper_functions.py", line 84, in as_form_func
docling  |     raise ValueError(f"Invalid JSON for field '{field_name}': {e}")
docling  | ValueError: Invalid JSON for field 'picture_description_local': 1 validation error for nullable[PictureDescriptionLocal]
docling  |   Invalid JSON: expected value at line 1 column 1 [type=json_invalid, input_value='generation_config', input_type=str]

Even though this corresponds with

docling-serve/docling_serve/datamodel/convert.py

Line 31 in b299af0

class PictureDescriptionLocal(BaseModel):

What am I not seeing here?

[vaclcer]

Ah, it needs to be added as a JSON string! Then it works too... Is this intentional? 😄

"picture_description_local": '''
{
    "repo_id": "ibm-granite/granite-vision-3.2-2b"
}
'''

[dolfim-ibm]

I think the specifics depend on the framework and language. I used the following with curl.

curl -X 'POST' \
  'http://localhost:5001/v1alpha/convert/file' \
  -H 'accept: application/json' \
  -H 'Content-Type: multipart/form-data' \
  -F 'do_picture_description=true' \
  -F 'image_export_mode=placeholder' \
  -F 'do_ocr=false' \
  -F 'picture_description_api={"url":"http://localhost:11434/v1/chat/completions","params":{"model":"granite3.2-vision:2b"},"timeout":60,"prompt":"Describe this image in a few sentences. "}' \
  -F 'files=@tests/2206.01062v1.pdf' \
  -F 'to_formats=json'