Skip to content

triton_python_backend_stub: Permission denied #8162

Closed
@Zheka001

Description

@Zheka001

Description
Problem with using triton_python_backend_stub in new image. Prepared triton_python_backend_stub by command inside nvcr.io/nvidia/tritonserver:25.03-py3 with python3.10.

Copy it to git repo. When trying to start triton-server i get:

I0421 10:08:17.917892 1 python_be.cc:1811] "Using Python execution env /opt/ruclip-venv"
I0421 10:08:17.918016 1 pb_env.cc:267] "Returning canonical path since EXECUTION_ENV_PATH does not contain compressed path. Path: /opt/ruclip-venv"
I0421 10:08:17.918905 1 stub_launcher.cc:385] "Starting Python backend stub: source /opt/ruclip-venv/bin/activate && exec env LD_LIBRARY_PATH=/opt/ruclip-venv/lib:$LD_LIBRARY_PATH /tmp/folderIbyKYQ/triton_python_backend_stub /tmp/folderIbyKYQ/1/model.py triton_python_backend_shm_region_6bd0fcc4-6ea1-49b4-a9f4-041f37b06c1b 1048576 1048576 1 /opt/tritonserver/backends/python 336 ruclip_visual_preprocessing DEFAULT"
/bin/dash: 0: cannot open /tmp/folderIbyKYQ/triton_python_backend_stub: Permission denied

And process stopped.

Triton Information
What version of Triton are you using?

nvcr.io/nvidia/tritonserver:25.03-py3

Are you using the Triton container or did you build it yourself?
Using a container made based on this nvcr.io/nvidia/tritonserver:25.03-py3 with pre-installed apt-get libs and python libs.

To Reproduce
Steps to reproduce the behavior.

  1. Prepare python_backend_stub in nvcr.io/nvidia/tritonserver:25.03-py3.
apt-get update
apt-get install cmake rapidjson-dev libarchive-dev zlib1g-dev -y

apt install software-properties-common -y
add-apt-repository ppa:deadsnakes/ppa
apt install python3.10 -y
apt install python3.10-dev -y

rm /usr/bin/python3
ln -s /usr/bin/python3.10 /usr/bin/python3

git clone https://github.com/triton-inference-server/python_backend -b r25.03
cd python_backend
mkdir build && cd build
cmake -DTRITON_ENABLE_GPU=ON -DTRITON_BACKEND_REPO_TAG=r25.03 -DTRITON_COMMON_REPO_TAG=r25.03 -DTRITON_CORE_REPO_TAG=r25.03 -DCMAKE_INSTALL_PREFIX:PATH=`pwd`/install ..
make triton-python-backend-stub
  1. Copy binary triton-python-backend-stub to working git repo with models.
  2. Prepare working image of triton with python3.10-venv
...
RUN apt-get update \
    && apt-get install --no-install-recommends \
    libsndfile1=1.2.2-1ubuntu5.24.04.1 \
    ffmpeg=7:6.1.1-3ubuntu5 \
    python3.10-venv=3.10.17-1+noble1 \
    python3.10-dev=3.10.17-1+noble1 \
    -y \
    && apt-get clean \
    && rm -rf /var/lib/apt/lists/*

# Создаем виртуальное окружение venv в указанной директории (например, /opt/venv1)
ENV VENV_DIR=/opt/special-venv
RUN python3.10 -m venv $VENV_DIR

RUN $VENV_DIR/bin/pip install --no-cache-dir --upgrade pip \
    && $VENV_DIR/bin/pip install --no-cache-dir wheel \
    && $VENV_DIR/bin/pip install --no-cache-dir cython==3.0.12 \
...

  1. Starting model with config:
    parameters: {
    key: "EXECUTION_ENV_PATH",
    value: {string_value: "/opt/special-venv"}
    }

  2. Start tritonserver with this model with "--log-verbose=3", get an error:

I0421 10:08:17.917892 1 python_be.cc:1811] "Using Python execution env /opt/ruclip-venv"
I0421 10:08:17.918016 1 pb_env.cc:267] "Returning canonical path since EXECUTION_ENV_PATH does not contain compressed path. Path: /opt/ruclip-venv"
I0421 10:08:17.918905 1 stub_launcher.cc:385] "Starting Python backend stub: source /opt/ruclip-venv/bin/activate && exec env LD_LIBRARY_PATH=/opt/ruclip-venv/lib:$LD_LIBRARY_PATH /tmp/folderIbyKYQ/triton_python_backend_stub /tmp/folderIbyKYQ/1/model.py triton_python_backend_shm_region_6bd0fcc4-6ea1-49b4-a9f4-041f37b06c1b 1048576 1048576 1 /opt/tritonserver/backends/python 336 ruclip_visual_preprocessing DEFAULT"
/bin/dash: 0: cannot open /tmp/folderIbyKYQ/triton_python_backend_stub: Permission denied

Describe the models (framework, inputs, outputs), ideally include the model configuration file (if using an ensemble include the model configuration file for that as well).

Expected behavior
A clear and concise description of what you expected to happen.
Starting model with custom python_backend_stub

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions