Why blocking HPOJobs with "not having enough combinations of hyperparameter ranges"? about amazon-sagemaker-operator-for-k8s HOT 4 CLOSED

Hi,

The ValidationException is coming from SageMaker when using parameterRanges in the hyperParameterTuningJobConfig.

semantic of MaxNumberOfJob enforces a lower bound on the number of combinations

What type of parameter ranges and their scale is being used when you when you see this error ?

doesn't seem like a status where the job should be held in Reconciling state

Jobs stuck in reconciling state after Validation error occurs looks like an issue. I will try to reproduce it on our end. Please provide us with minimum reproducible sample input.

Thanks

from amazon-sagemaker-operator-for-k8s.

I was able to replicate the job stuck in ReconcilingTuningJob status with the below job definition

apiVersion: sagemaker.aws.amazon.com/v1
kind: HyperparameterTuningJob
metadata:
  name: kmeans-mnist-hpo-3
spec:
  region: us-east-1
  hyperParameterTuningJobConfig:
    strategy: Bayesian
    hyperParameterTuningJobObjective:
      type: Minimize
      metricName: test:msd
    resourceLimits:
      maxNumberOfTrainingJobs: 10
      maxParallelTrainingJobs: 5
    parameterRanges:
      categoricalParameterRanges:
      - name: init_method
        values:
        - 'random'
        - 'kmeans++'
  trainingJobDefinition:
    staticHyperParameters:
      - name: k
        value: '10'
      - name: feature_dim
        value: '784'
    algorithmSpecification:
      trainingImage: 382416733822.dkr.ecr.us-east-1.amazonaws.com/kmeans:1
      trainingInputMode: File
    roleArn: <REPLACE_ME>
    inputDataConfig:
    - channelName: train
      dataSource:
        s3DataSource:
          s3DataType: S3Prefix
          s3Uri: s3://<REPLACE_ME>/mnist_kmeans_example/train_data/
          s3DataDistributionType: FullyReplicated
      compressionType: None
      recordWrapperType: None
      inputMode: File
    outputDataConfig:
      s3OutputPath: s3://<REPLACE_ME>/mnist_kmeans_example/output
    resourceConfig:
      instanceType: ml.m4.xlarge
      instanceCount: 1
      volumeSizeInGB: 25
    stoppingCondition:
      maxRuntimeInSeconds: 3600

from amazon-sagemaker-operator-for-k8s.

Hi @surajkota! Sorry for the late reply.

What type of parameter ranges and their scale is being used when you when you see this error ?

So in my case, the validation failed when I set the maxNumberOfTrainingJob to 10 and only have, say, an integer parameter range (e.g., num_round) from 1 to 3. The ScalingType was Linear. Since you

I was able to replicate the job stuck in ReconcilingTuningJob status with the below job definition

In your example it seems that you are encountering ReconcilingTuningJob because there are only two possible training job configurations: init_method = ['random', 'kmeans++'] and you have maxNumberOfTrainingJobs=10. This aligns with my experience.

What I am trying to get at in this issue is that, even if the number of possible number of configurations is smaller than maxNumberOfTrainingJobs, SageMaker should still let the job proceed. maxNumberOfTrainingJobs should enforce only an upper limit on the number of training jobs that will be launched when the total number of possibilities is larger; its semantic should not enforce that the hpo job needs to have at least maxNumberOfTrainingJobs of training jobs.

I hope that makes sense :)

from amazon-sagemaker-operator-for-k8s.

Please use the latest version of SageMaker Operator - https://github.com/aws/amazon-sagemaker-operator-for-k8s#migrate-resources-to-the-new-sagemaker-operators-for-kubernetes

from amazon-sagemaker-operator-for-k8s.