`LocalModel.build_cpr_model` fails when using base_image containing `/` about python-aiplatform HOT 5 OPEN

Thanks for reporting this. It may be a bug on our side. To unblock you, could you temporarily set the base_image without / locally? Thanks.

from python-aiplatform.

@abcdefgs0324 Thank you for giving me the idea of renaming the image name locally! It turned out / was not the root cause. After I gave a short name without / to nvidia/cuda:11.1.1-devel-ubuntu20.04, the error happened to my environment. I'll get back to you once I can narrow down the cases where the error happens. Thanks

from python-aiplatform.

Hi @abcdefgs0324,

Is there any suggestion how can a user set up base_image with nvidia/pytorch on board?

Two options below produce the error described in the original post:

• base_image="us-docker.pkg.dev/vertex-ai/prediction/pytorch-gpu.2-0:latest"
• base_image="pytorch/pytorch:2.1.0-cuda12.1-cudnn8-runtime"

from python-aiplatform.

What is the purpose of this image? Is it meant to be a base for CPR or just an example?

from python-aiplatform.

@abcdefgs0324 Thank you for giving me the idea of renaming the image name locally! It turned out / was not the root cause. After I gave a short name without / to nvidia/cuda:11.1.1-devel-ubuntu20.04, the error happened to my environment. I'll get back to you once I can narrow down the cases where the error happens. Thanks

Did you end up figuring out what caused this? I am having the same issue. I wish there was a way to view the internal output from Docker to be able to debug this.

EDIT--
I was able to grab the debug info locally by changing the following in site-packages/google/cloud/aiplatform/docker_utils/local_util.py, line 59 :

Change _logger.info(line) to print(line)

        for line in out:
            print(line)
            # _logger.info(line)

There is probably a way to instantiate a logger locally to catch these lines? I am not too familiar with logging, but I did notice that the logging does work when this runs in the cloud.

Anyway, my problem was just a problem with the requirements.txt file.

from python-aiplatform.