Vertex AI LLM SDK BatchPredictionJob blocks program execution about python-aiplatform HOT 6 CLOSED

You're right, the sync parameter is not used at all in this function.

The sync parameter is handled by the following decorator:

@base.optional_sync(return_input_arg="empty_batch_prediction_job")

from python-aiplatform.

In next release we're adding the BatchPredictionJob.submit() method that does not wait for job completion (there will be job.wait_for_completion() for that).

from python-aiplatform.

If you look at the method definition in https://github.com/googleapis/python-aiplatform/blob/main/google/cloud/aiplatform/models.py#L3762 you will see parameter sync, it's not documented there, but here it's just passed to the jobs.BatchPredictionJob.create so if you look at its definition you will see the sync parameter documented here
so just call

text_model.batch_predict(
  source_uri=["gs://BUCKET_NAME/test_table.jsonl"],
  destination_uri_prefix="gs://BUCKET_NAME/tmp/2023-05-25-vertex-LLM-Batch-Prediction/result3",
  # Optional:
  model_parameters={
      "maxOutputTokens": "200",
      "temperature": "0.2",
      "topP": "0.95",
      "topK": "40",
  },
  sync=False,

and it should not hang.

from python-aiplatform.

Hey,

Thank you very much for addressing this.
Unfortunately this does not solve the problem.

I actually tried that before opening the bug and tried again now.
After adding the sync=False parameter the program still hangs, and the parameter doesn't seem to have any effect.

Following the links you provided, I can see that eventually, the sync parameter is not evaluated when calling the _block_until_complete method here

Any ideas? Is this a bug?

from python-aiplatform.

You're right, the sync parameter is not used at all in this function. Yes, I think it is a bug, because it behaves differently than what is in the docs.

from python-aiplatform.

I'm not sure we can just switch TextGenerationModel.batch_predict to non-blocking version since it can be considered a breaking change. What do you think? Would such change break workflows?

from python-aiplatform.