aws / amazon-sagemaker-operator-for-k8s Goto Github PK

View Code? Open in Web Editor NEW

148.0 11.0 53.0 20.17 MB

Amazon SageMaker operator for Kubernetes

Home Page: https://docs.aws.amazon.com/sagemaker/latest/dg/kubernetes-sagemaker-operators.html

License: Apache License 2.0

Makefile 0.61% Go 89.46% Shell 9.55% Python 0.34% Mustache 0.04%

aws sagemaker kubernetes-operator kubebuilder golang kubernetes

amazon-sagemaker-operator-for-k8s's Introduction

Amazon SageMaker Operators for Kubernetes

Introduction

Amazon SageMaker Operators for Kubernetes are operators that can be used to train machine learning models, optimize hyperparameters for a given model, run batch transform jobs over existing models, and set up inference endpoints. With these operators, users can manage their jobs in Amazon SageMaker from their Kubernetes cluster in Amazon Elastic Kubernetes Service EKS.

Migrate resources to the new SageMaker Operators for Kubernetes

⚠️ This project will reach its end-of-life (EOL) on Feb 15, 2023 along with Amazon Elastic Kubernetes Service Kubernetes 1.21. If you are currently using version v1.2.2 or below of this SageMaker Operators for Kubernetes, we recommend migrating your resources to the latest SageMaker Operators for Kubernetes, the ACK service controller for Amazon SageMaker based on AWS Controllers for Kubernetes (ACK).

For information on the migration steps, see Migrate resources to the latest Operators.
For answers to frequently asked questions (FAQ) regarding the end of support of this version of SageMaker Operators for Kubernetes, see Announcing the End of Support of the Original Version of SageMaker Operator for Kubernetes.

Usage

:note: The following steps do not install the latest version of SageMaker Operators for Kubernetes. Find the link to the new ACK-based SageMaker Operators for Kubernetes project in the warning message above.

First, you must install the operators. After installation is complete, create a TrainingJob YAML specification by following one of the samples, like samples/xgboost-mnist-trainingjob.yaml. Then, use kubectl to create and monitor the progress of your job:

$ kubectl apply -f xgboost-mnist-trainingjob.yaml
trainingjob.sagemaker.aws.amazon.com/xgboost-mnist created

$ kubectl get trainingjob
NAME            STATUS       SECONDARY-STATUS   CREATION-TIME          SAGEMAKER-JOB-NAME
xgboost-mnist   InProgress   Starting           2019-11-26T23:38:11Z   xgboost-mnist-cf1e16fb10a511eaaa450a350733ba06

Once the job starts training, you can use a kubectl plugin to stream training logs:

$ kubectl get trainingjob
NAME            STATUS       SECONDARY-STATUS   CREATION-TIME          SAGEMAKER-JOB-NAME
xgboost-mnist   InProgress   Training           2019-11-26T23:38:11Z   xgboost-mnist-cf1e16fb10a511eaaa450a350733ba06

$ kubectl smlogs trainingjob xgboost-mnist | head -n 5
"xgboost-mnist" has SageMaker TrainingJobName "xgboost-mnist-cf1e16fb10a511eaaa450a350733ba06" in region "us-east-2", status "InProgress" and secondary status "Training"
xgboost-mnist-cf1e16fb10a511eaaa450a350733ba06/algo-1-1574811611 2019-11-26 15:41:13.449 -0800 PST Arguments: train
xgboost-mnist-cf1e16fb10a511eaaa450a350733ba06/algo-1-1574811611 2019-11-26 15:41:13.449 -0800 PST [2019-11-26:23:41:10:INFO] Running standalone xgboost training.
xgboost-mnist-cf1e16fb10a511eaaa450a350733ba06/algo-1-1574811611 2019-11-26 15:41:13.45 -0800 PST [2019-11-26:23:41:10:INFO] File size need to be processed in the node: 1122.95mb. Available memory size in the node: 8501.08mb
xgboost-mnist-cf1e16fb10a511eaaa450a350733ba06/algo-1-1574811611 2019-11-26 15:41:13.45 -0800 PST [2019-11-26:23:41:10:INFO] Determined delimiter of CSV input is ','
xgboost-mnist-cf1e16fb10a511eaaa450a350733ba06/algo-1-1574811611 2019-11-26 15:41:13.45 -0800 PST [23:41:10] S3DistributionType set as FullyReplicated

The Amazon SageMaker Operators for Kubernetes enable management of SageMaker TrainingJobs, HyperParameterTuningJobs, BatchTransformJobs and HostingDeployments (Endpoints). Create and monitor them using the same kubectl tool as above.

To install the operators onto your Kubernetes cluster, follow our User Guide.

YAML Examples

To make a YAML spec, follow one of the below examples as a guide. Replace values like RoleARN, S3 input buckets and S3 output buckets with values that correspond to your account.

Releases

Amazon SageMaker Operator for Kubernetes adheres to the SemVer specification. Each release updates the major version tag (eg. vX), a major/minor version tag (eg. vX.Y) and a major/minor/patch version tag (eg. vX.Y.Z), as well as new versions of the smlogs binary with URLs of the same versioning formats. To see a full list of all releases, refer to our Github releases page.

We also maintain a latest tag, which is updated to stay in line with the master branch. We do not recommend installing this on any production cluster, as any new major versions updated on the master branch will introduce breaking changes.

Contributing

amazon-sagemaker-operator-for-k8s is an open source project. See CONTRIBUTING for details.

License

This project is distributed under the Apache License, Version 2.0, see LICENSE and NOTICE for more information.

amazon-sagemaker-operator-for-k8s's People

Contributors

Stargazers

Watchers

amazon-sagemaker-operator-for-k8s's Issues

Why blocking HPOJobs with "not having enough combinations of hyperparameter ranges"?

We saw HPOJobs stuck in Reconciling state because of not enough combinations of hyperparmeter ranges:

Unable to create HyperParameter Tuning Job: ValidationException: You don't have enough combinations of hyperparameter ranges. The total number of hyperparameter combinations for the provided ranges [3.0] must be equal or greater than the value of MaxNumberOfJobs, [10]. Provide additional ranges."

It is a bit confusing to see the semantic of MaxNumberOfJob enforces a lower bound on the number of combinations, and I feel this doesn't seem like a status where the job should be held in Reconciling state. May I ask what's the reason to block such jobs?

Async Endpoint Request

Hello!

Will there be any support for Async Inference Endpoints?

Upgrade aws-sdk to v1

What would you like to be added:
Upgrade dependency to the stable aws-sdk v1.. it's a breaking change so a lot will need to be updated.

Why is this needed:
Unable to consume any of new SDK types because we have a dependency on the operator.

Short name for jobs

K8s often has short name for their resources. We should model the job name as short job name.

One suggestion

hyperparametertuningjobs      ===>  hptj or hpo
trainingjobs   ===> tj
batchtransformjobs  ===>  btj
endpointconfigs  ===>  epc or ec
hostingdeployments ===> hd 
models  ==> m

We should make sure we don't conflict with any existing k8s resources short name

Submitted TrainingJob won't invoke SageMaker

I added a comment at the bottom to this previous closed issue: [https://github.com//issues/99]

Could we please reopen the issue?

Error Building SageMaker Types due to missing types in common/manual_deepcopy

What happened:
Error Building SageMaker Types due to missing types in common/manual_deepcopy
(base) afccd2:example nj$ make all
go: creating new go.mod: module tmp
go: found sigs.k8s.io/controller-tools/cmd/controller-gen in sigs.k8s.io/controller-tools v0.2.5
/devel/projects/go_tutorial/bin/controller-gen object:headerFile="hack/boilerplate.go.txt" paths="./..."
go fmt ./...
controllers/guestbook_controller.go
go vet ./...
github.com/aws/amazon-sagemaker-operator-for-k8s/api/v1/common
../../../go_tutorial/pkg/mod/github.com/aws/amazon-sagemaker-operator-for-[email protected]/api/v1/common/manual_deepcopy.go:28:19: tag.DeepCopy undefined (type Tag has no field or method DeepCopy)
make: *** [vet] Error 2

What you expected to happen:
Packaged types refer to types in zz_generated_deepcopy which are missing

How to reproduce it (as minimally and precisely as possible):

Import of sagemaker types in Go Client fails build

import (
trainingjobv1 "github.com/aws/amazon-sagemaker-operator-for-k8s/api/v1/trainingjob"
)

Anything else we need to know?:

Environment:

Kubernetes version (use kubectl version):
Operator version (controller image tag): v1.1.0
OS (e.g: cat /etc/os-release):
Kernel (e.g. uname -a):
Installation method:
Others:

Error reconciling customizedMetricSpecification HostingAutoscalingPolicy

What happened: Operator fails to reconcile HAP while using PolicyName other than the "default" - IE; customizedMetricSpecification

What you expected to happen: Operator reconcile the HAP by creating it without error in Sagemaker Endpoint.

How to reproduce it (as minimally and precisely as possible):

Create an HostingEndpoint :

apiVersion: sagemaker.aws.amazon.com/v1
kind: HostingDeployment
metadata:
  name: my-model-endpoint
spec:
  region: ap-south-1
  endpointName: my-endpoint
  productionVariants:
    - variantName: AllTraffic
      modelName: my-model
      initialInstanceCount: 1
      instanceType: ml.c5.xlarge
      initialVariantWeight: 1
  models:
    - name: my-model
      executionRoleArn: arn:aws:iam::XXXXXXXX:role/sagemaker-role
      containers:
        - containerHostname: my-model
          modelDataUrl: s3://XXXX 
          image: XXXX # Removed

Create a HostingAutoscalingPolicy using customizedMetricSpecification

apiVersion: sagemaker.aws.amazon.com/v1
kind: HostingAutoscalingPolicy
metadata:
  name: my-model-endpoint-scaling-policy
spec:
    resourceId:
      - endpointName: my-model-endpoint
        variantName: AllTraffic
    region: ap-south-1
    policyName: CPU-ScalingPolicy
    policyType: TargetTrackingScaling
    minCapacity: 1
    maxCapacity: 3
    targetTrackingScalingPolicyConfiguration:
      targetValue: 80.0
      customizedMetricSpecification:
        metricName: CPUUtilization
        namespace: /aws/sagemaker/Endpoints
        dimensions:
           - name: EndpointName
             value: my-model-endpoint
           - name: VariantName
             value: AllTraffic
        statistic: Average
        unit: Percent

The operator produce the following failure log:

2021-11-05T18:49:49.241Z        INFO    controllers.HostingAutoscalingPolicy    Getting resource        {"hostingautoscalingpolicy": "my-namespace/my-model-endpoint-scaling-policyl"}
2021-11-05T18:49:49.241Z        INFO    controllers.HostingAutoscalingPolicy    Loaded AWS config       {"hostingautoscalingpolicy": "my-namespace/my-model-endpoint-scaling-policyl"}
2021-11-05T18:49:49.488Z        INFO    controllers.HostingAutoscalingPolicy    Determined action for AutoscalingJob    {"hostingautoscalingpolicy": "my-namespace/my-model-endpoint-scaling-policyl", "action": "NeedsCreate"}
2021-11-05T18:49:49.918Z        INFO    controllers.HostingAutoscalingPolicy    Got an error while reconciling HostingAutoscalingPolicy, will retry     {"hostingautoscalingpolicy": "my-namespace/my-model-endpoint-scaling-policyl", "err": "Unable to apply HostingAutoscalingPolicy: hosting autoscaling policy was not applied, description is empty", "errVerbose": "hosting autoscaling policy was not applied, description is empty\nUnable to apply HostingAutoscalingPolicy\ngithub.com/aws/amazon-sagemaker-operator-for-k8s/controllers/hostingautoscalingpolicy.(*Reconciler).reconcileHostingAutoscalingPolicy\n\t/workspace/controllers/hostingautoscalingpolicy/hostingautoscalingpolicy_controller.go:192\ngithub.com/aws/amazon-sagemaker-operator-for-k8s/controllers/hostingautoscalingpolicy.(*Reconciler).Reconcile\n\t/workspace/controllers/hostingautoscalingpolicy/hostingautoscalingpolicy_controller.go:112\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:235\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:209\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:188\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:90\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357"}

Environment:

Kubernetes version (use kubectl version): 1.20
Operator version (controller image tag): 1.2.1
Installation method: github.com/aws/amazon-sagemaker-operator-for-k8s/config/installers/rolebasedcreds?ref=v1.2.1
Others:

Establish a regular release cadence?

What would you like to be added:
I looks like there were no releases created since April. Ideally we would have these go out on some regular cadence.

Why is this needed:

Explore integration with Spinnaker

Spinnaker is a CD platform that some are using to orchestrate ML pipelines.

We should investigate ways that our operators can integrate with it so that Spinnaker pipelines can use SageMaker.

AccessDenied issue

Hi, I just followed the instructions and deployed a sample training job. Yet, somehow my IAM role seems to be getting in the way.

code used to deploy training job

kubectl apply -f xgboost-mnist-trainingjob.yaml

kubectl get trainingjob

NAME                        STATUS                   SECONDARY-STATUS   CREATION-TIME          SAGEMAKER-JOB-NAME
xgboost-mnist               ReconcilingTrainingJob                      2020-07-23T04:08:31Z   <sagemaker job name>
xgboost-mnist-2           ReconcilingTrainingJob                      2020-07-23T05:23:10Z   <sagemaker job name2>

Description of one of those reconciling training jobs

kubectl describe trainingjob/xgboost-mnist

Name:         xgboost-mnist
Namespace:    default
Labels:       <none>
Annotations:  API Version:  sagemaker.aws.amazon.com/v1
Kind:         TrainingJob
Metadata:
  Creation Timestamp:  2020-07-23T04:08:31Z
  Finalizers:
    sagemaker-operator-finalizer
  Generation:        2
  Resource Version:  28787
  Self Link:         /apis/sagemaker.aws.amazon.com/v1/namespaces/default/trainingjobs/xgboost-mnist
  UID:              <uid>
Spec:
  Algorithm Specification:
    Training Image:       433757028032.dkr.ecr.us-west-2.amazonaws.com/xgboost:1
    Training Input Mode:  File
  Hyper Parameters:
    Name:   max_depth
    Value:  5
    Name:   eta
    Value:  0.2
    Name:   gamma
    Value:  4
    Name:   min_child_weight
    Value:  6
    Name:   silent
    Value:  0
    Name:   objective
    Value:  multi:softmax
    Name:   num_class
    Value:  10
    Name:   num_round
    Value:  10
  Input Data Config:
    Channel Name:      train
    Compression Type:  None
    Content Type:      text/csv
    Data Source:
      s3DataSource:
        s3DataDistributionType:  FullyReplicated
        s3DataType:              S3Prefix
        s3Uri:                   <Replaced with my s3 uri>
    Channel Name:                validation
    Compression Type:            None
    Content Type:                text/csv
    Data Source:
      s3DataSource:
        s3DataDistributionType:  FullyReplicated
        s3DataType:              S3Prefix
        s3Uri:                   <Replaced with my s3 uri>
  Output Data Config:
    s3OutputPath:  <Replaced with my s3 uri>
  Region:          us-west-2
  Resource Config:
    Instance Count:     1
    Instance Type:      ml.m4.xlarge
    Volume Size In GB:  5
  Role Arn:             <Replace with my IAM arn>
  Stopping Condition:
    Max Runtime In Seconds:  86400
  Tags:
    Key:              tagKey
    Value:            tagValue
  Training Job Name:  <jobname>
Status:
  Additional:  Unable to describe SageMaker training job: WebIdentityErr: failed to retrieve credentials
caused by: AccessDenied: Not authorized to perform sts:AssumeRoleWithWebIdentity
                                 status code: 403, request id: <request id>
  Cloud Watch Log URL:           <log url, although log does not exist>
  Sage Maker Training Job Name:  <sagemaker training job name>
  Training Job Status:           ReconcilingTrainingJob
Events:                          <none>

checking OIDC

aws eks describe-cluster --name <cluster name> --region us-west-2 --query cluster.identity.oidc.issuer --output text

https://oidc.eks.us-west-2.amazonaws.com/id/<oidc ID>

IAM role used in the training job (SageMakerFullAccess Permission attached)

aws iam get-role --role-name <role name>

{
    "Role": {
        "Path": "/",
        "RoleName": <role name>,
        "RoleId": <role id>,
        "Arn": <role arn used in the training job>,
        "CreateDate": "2020-07-23T04:18:37+00:00",
        "AssumeRolePolicyDocument": {
            "Version": "2012-10-17",
            "Statement": [
                {
                    "Effect": "Allow",
                    "Principal": {
                        "Federated": "arn:aws:iam::<account number>:oidc-provider/oidc.eks.us-west-2.amazonaws.com/id/<oidc id>"
                    },
                    "Action": "sts:AssumeRoleWithWebIdentity",
                    "Condition": {
                        "StringEquals": {
                            "oidc.eks.us-west-2.amazonaws.com/id/<oidc id>:sub": "system:serviceaccount:sagemaker-k8s-operator-system:sagemaker-k8s-operator-default",
                            "oidc.eks.us-west-2.amazonaws.com/id/<oidc id>:aud": "sts.amazonaws.com"
                        }
                    }
                }
            ]
        },
        "MaxSessionDuration": 3600,
        "RoleLastUsed": {}
    }
}

Judging by the description status, IAM role which I made according to the sagemaker documentation, must be the issue. Could you guide me on how to fix this?

Also, since there's sagemaker operator for k8s, is it also possible to deploy sagemaker training jobs onto my EKS cluster?

On-premise K8S cluster integrated with Sagmaker operator

What would you like to be added:
The solution might be work fine with EKS, but when i want to copy this solution to on premise data center K8S cluster, and i want give authorization by using IAM user's AK/SK. and i am not sure it will works and how it can be worked.
Why is this needed:
Many of our cluster are build in on premise data center, and our way to manager credential is using AK/SK. Not sure if it will be acceptable.

Create an example where operator spec struct can be used in go-client

Customer request: #118

README.md broken link

The "install the operators" phrase in the README.me file points to the What is SageMaker page, it's been redirected to an unhelpful link.

Migrating to v1 as v1beta is deprecated

Starting with k8s 1.22 the v1beta CustomResourceDefinition will be deprecated see https://kubernetes.io/blog/2021/07/14/upcoming-changes-in-kubernetes-1-22/#api-changes

Which means that the code in this repo will no longer be applicable. Is there any plans to migrate the definitions?

error: no kind "TrainingJob" is registered for version "sagemaker.aws.amazon.com/v1" in scheme "k8s.io/kubectl/pkg/scheme/scheme.go:28"

KmsKeyId will constantly reconcile if not using ARN

What happened:
When creating any custom resource with a KmsKeyId field specified using an ID, rather than an ARN, the reconciler will get stuck trying to infinitely reconcile it.

What you expected to happen:
The custom resource is only reconciled once and then reaches a stable state.

How to reproduce it (as minimally and precisely as possible):
Include a KmsKeyId with any ID to a custom resource (such as EndpointConfig).

Anything else we need to know?:
The describe response for KmsKeyId is always an ARN. This is what causes the difference that the reconciler gets stuck trying to fix.

Environment:

Kubernetes version (use kubectl version):
Operator version (controller image tag):
OS (e.g: cat /etc/os-release):
Kernel (e.g. uname -a):
Installation method:
Others:

Install just a subset of CRDs for the TrainingJob Operator?

Hey Sagemaker Team,
My use case only requires the TrainingJobs CRD. However, if the other CRD's (batchtransformjobs, models) are not installed, then the deployment fails to start. Is there a flag I can pass into the deployment to fix this?

Also, is this repo properly packaged to be installed as a go module. I want to be able to access the
TrainingJobSpec struct here so that I can create CRD's with Client-go rather than yaml + kubectl commands.

How can i pass endpointName & custom container env variables on HostingDeployment ?

@surajkota

How can i pass endpointName & custom container env variables on HostingDeployment ?

error: no kind "TrainingJob" is registered for version "sagemaker.aws.amazon.com/v1" in scheme "k8s.io/kubectl/pkg/scheme/scheme.go:28"

Unify the model name between batch transform and hosting deployments.

Currently model name in hosting deployment uses kubernetes name of model while model in batch transform uses sagemaker model name.

Ideally we should have same mechanism for name, given that kubernetes user will look from kubernetes lens.

StepScaling policy support

What would you like to be added:
It would be great is this tool will allow to use StepScaling policy additionally to TargetTrackingScaling policy.

Why is this needed:
Some companies currently using StepScaling policy as a infrastructure standard and they could not use amazon-sagemaker-operator-for-k8s as it doesn't support this policy.

Improve HAP logging message

What would you like to be added:
As discussed in this issue there are two action items here -

Improve the existing HAP controller log message to be more specific.
Verify the server side errors are surfacing correctly for HAP (I did not see the right error message when specifying the incorrect endpoint region. Need to verify if this is due to incorrect operator implementation or GoSDK)

Why is this needed:
Improve debugging and thus better user experience.

Add typed clients/informers/listers for Sagemaker CRDs

What would you like to be added:
In addition to the generated DeepCopy functions, may this project also include generated code for clients, informers, and listers? The documentation on using the generators is a bit sparse, but I have found this blog post by Stefan Schimanski to be a good resource on using the scripts. There is also a master script (generator-group.sh) that can be used to generate all the code at once.

Why is this needed:

This will enable a native/full-featured experience for users to programmatically interact with Sagemaker's CRDs. Otherwise, users have to rely on dynamic clients/informers and type casting, which is a bit more error prone and inconvenient.

unable to kick off the sagemaker job

Deployed the sample mnist training job but seems its not getting invoked on the SageMaker

kubectl describe TrainingJob            
Name:         xgboost-mnist
Namespace:    default
Labels:       <none>
Annotations:  kubectl.kubernetes.io/last-applied-configuration:
                {"apiVersion":"sagemaker.aws.amazon.com/v1","kind":"TrainingJob","metadata":{"annotations":{},"name":"xgboost-mnist","namespace":"default"...
API Version:  sagemaker.aws.amazon.com/v1
Kind:         TrainingJob
Metadata:
  Creation Timestamp:  2020-03-09T06:58:17Z
  Generation:          1
  Resource Version:    117181
  Self Link:           /apis/sagemaker.aws.amazon.com/v1/namespaces/default/trainingjobs/xgboost-mnist
  UID:                 5a907178-61d3-11ea-b461-02efd6507006
Spec:
  Algorithm Specification:
    Training Image:       825641698319.dkr.ecr.us-east-2.amazonaws.com/xgboost:latest
    Training Input Mode:  File
  Hyper Parameters:
    Name:   max_depth
    Value:  5
    Name:   eta
    Value:  0.2
    Name:   gamma
    Value:  4
    Name:   min_child_weight
    Value:  6
    Name:   silent
    Value:  0
    Name:   objective
    Value:  multi:softmax
    Name:   num_class
    Value:  10
    Name:   num_round
    Value:  10
  Input Data Config:
    Channel Name:      train
    Compression Type:  None
    Content Type:      text/csv
    Data Source:
      S 3 Data Source:
        S 3 Data Distribution Type:  FullyReplicated
        S 3 Data Type:               S3Prefix
        S 3 Uri:                     s3://<MY-BUCKET>/xgboost-mnist/train/
    Channel Name:                    validation
    Compression Type:                None
    Content Type:                    text/csv
    Data Source:
      S 3 Data Source:
        S 3 Data Distribution Type:  FullyReplicated
        S 3 Data Type:               S3Prefix
        S 3 Uri:                     s3://<MY-BUCKET>/xgboost-mnist/validation/
  Output Data Config:
    S 3 Output Path:  s3://<MY-BUCKET>/xgboost-mnist/models/
  Region:             us-east-2
  Resource Config:
    Instance Count:     1
    Instance Type:      ml.m4.xlarge
    Volume Size In GB:  5
  Role Arn:             arn:aws:iam::<ACCOUNT>:role/sagemaker_execution_role
  Stopping Condition:
    Max Runtime In Seconds:  86400```

SageMaker Operator Types fails KubeBuilder Pattern validation check

What happened:
SageMaker Operator Types, when included as part of KubeBuilder V2 custom CRD definition fail due to validation errors of unescaped regex patterns.

/go/bin/controller-gen "crd:trivialVersions=true" rbac:roleName=manager-role webhook paths="./..." output:crd:artifacts:config=config/crd/bases
/go/pkg/mod/github.com/aws/amazon-sagemaker-operator-for-k8s@v1.0.1-0.20200410212604-780c48ecb21a/api/v1/common/sagemaker_api.go:488:2: extra arguments provided: "://([^/]+)/?(.*)$" (at <input>:1:12)
/go/pkg/mod/github.com/aws/amazon-sagemaker-operator-for-k8s@v1.0.1-0.20200410212604-780c48ecb21a/api/v1/common/sagemaker_api.go:110:2: extra arguments provided: "://([^/]+)/?(.*)$" (at <input>:1:12)
/go/pkg/mod/github.com/aws/amazon-sagemaker-operator-for-k8s@v1.0.1-0.20200410212604-780c48ecb21a/api/v1/common/sagemaker_api.go:82:2: extra arguments provided: "://([^/]+)/?(.*)$" (at <input>:1:12)
/go/pkg/mod/github.com/aws/amazon-sagemaker-operator-for-k8s@v1.0.1-0.20200410212604-780c48ecb21a/api/v1/common/sagemaker_api.go:103:2: extra arguments provided: "://([^/]+)/?(.*)$" (at <input>:1:12)
/go/pkg/mod/github.com/aws/amazon-sagemaker-operator-for-k8s@v1.0.1-0.20200410212604-780c48ecb21a/api/v1/common/sagemaker_api.go:466:2: extra arguments provided: "://([^/]+)/?(.*)$" (at <input>:1:12)
/go/pkg/mod/github.com/aws/amazon-sagemaker-operator-for-k8s@v1.0.1-0.20200410212604-780c48ecb21a/api/v1/common/sagemaker_api.go:450:2: extra arguments provided: "://([^/]+)/?(.*)$" (at <input>:1:12)
/go/pkg/mod/github.com/aws/amazon-sagemaker-operator-for-k8s@v1.0.1-0.20200410212604-780c48ecb21a/api/v1/common/sagemaker_api.go:500:2: extra arguments provided: "://([^/]+)/?(.*)$" (at <input>:1:12)
/go/pkg/mod/github.com/aws/amazon-sagemaker-operator-for-k8s@v1.0.1-0.20200410212604-780c48ecb21a/api/v1/common/sagemaker_api.go:515:2: extra arguments provided: "://([^/]+)/?(.*)$" (at <input>:1:12)
/go/pkg/mod/github.com/aws/amazon-sagemaker-operator-for-k8s@v1.0.1-0.20200410212604-780c48ecb21a/api/v1/common/sagemaker_api.go:500:2: extra arguments provided: "://([^/]+)/?(.*)$" (at <input>:1:12)
/go/pkg/mod/github.com/aws/amazon-sagemaker-operator-for-k8s@v1.0.1-0.20200410212604-780c48ecb21a/api/v1/common/sagemaker_api.go:450:2: extra arguments provided: "://([^/]+)/?(.*)$" (at <input>:1:12)
/go/pkg/mod/github.com/aws/amazon-sagemaker-operator-for-k8s@v1.0.1-0.20200410212604-780c48ecb21a/api/v1/common/sagemaker_api.go:82:2: extra arguments provided: "://([^/]+)/?(.*)$" (at <input>:1:12)
/go/pkg/mod/github.com/aws/amazon-sagemaker-operator-for-k8s@v1.0.1-0.20200410212604-780c48ecb21a/api/v1/common/sagemaker_api.go:466:2: extra arguments provided: "://([^/]+)/?(.*)$" (at <input>:1:12)
/go/pkg/mod/github.com/aws/amazon-sagemaker-operator-for-k8s@v1.0.1-0.20200410212604-780c48ecb21a/api/v1/common/sagemaker_api.go:103:2: extra arguments provided: "://([^/]+)/?(.*)$" (at <input>:1:12)
/go/pkg/mod/github.com/aws/amazon-sagemaker-operator-for-k8s@v1.0.1-0.20200410212604-780c48ecb21a/api/v1/common/sagemaker_api.go:488:2: extra arguments provided: "://([^/]+)/?(.*)$" (at <input>:1:12)
/go/pkg/mod/github.com/aws/amazon-sagemaker-operator-for-k8s@v1.0.1-0.20200410212604-780c48ecb21a/api/v1/common/sagemaker_api.go:515:2: extra arguments provided: "://([^/]+)/?(.*)$" (at <input>:1:12)
/go/pkg/mod/github.com/aws/amazon-sagemaker-operator-for-k8s@v1.0.1-0.20200410212604-780c48ecb21a/api/v1/common/sagemaker_api.go:110:2: extra arguments provided: "://([^/]+)/?(.*)$" (at <input>:1:12)

What you expected to happen:
KubeBuilder should generate CRD specification which includes AWS SageMaker Operator Types

How to reproduce it (as minimally and precisely as possible):

import (
	commonv1 "github.com/aws/amazon-sagemaker-operator-for-k8s/api/v1/common"
	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)

// GuestbookSpec defines the desired state of Guestbook
type GuestbookSpec struct {
	// INSERT ADDITIONAL SPEC FIELDS - desired state of cluster
	// Important: Run "make" to regenerate code after modifying this file

	AlgorithmSpecification *commonv1.AlgorithmSpecification `json:"algorithmSpecification"`

	EnableInterContainerTrafficEncryption *bool `json:"enableInterContainerTrafficEncryption,omitempty"`

	EnableNetworkIsolation *bool `json:"enableNetworkIsolation,omitempty"`
...
//Run make install with above  types in custom operator
make install

Anything else we need to know?:
Tried copying the above types and escaped the regex pattern with quotes (// +kubebuilder:validation:Pattern='^(https|s3)://([^/]+)/?(.*)$') and everything worked

Environment:

Kubernetes version (use kubectl version):Version: version.Version{KubeBuilderVersion:"2.3.1", KubernetesVendor:"1.16.4", GitCommit:"8b53abeb4280186e494b726edf8f54ca7aa64a49", BuildDate:"2020-03-26T16:42:00Z", GoOs:"unknown", GoArch:"unknown"}
Operator version (controller image tag): github.com/aws/amazon-sagemaker-operator-for-k8s v1.0.1-0.20200410212604-780c48ecb21a
OS (e.g: cat /etc/os-release):
Kernel (e.g. uname -a):
Installation method:
Others:

how to use amazon-sagemaker-operator-for-k8s in ec2 k8s like Minikube

I build a Minikube inside a ec2, and I attach the policy AmazonEC2ContainerRegistryFullAccess and AmazonSageMakerFullAccess as a role to my ec2 instance inside of eke's OIDC, I change the Role Arn in installer.yaml to the Role Arn for ec2, and run kubectl apply -f installer.yaml, however, the pods can't be installed.
kubectl -n sagemaker-k8s-operator-system get pods
the ready only 1/2, and status is imagepullbackoff
kubectl -n sagemaker-k8s-operator-system describe pods
It said that the 957583890962.dkr.ecr.us-east-1.amazonaws.com/amazon-sagemaker-operator-for-k8s:v1 docker image is no basic auth credentials, I try to modify the yaml 957583890962.dkr.ecr.us-east-1.amazonaws.com/amazon-sagemaker-operator-for-k8s:v1 to 640106867763.dkr.ecr.us-east-1.amazonaws.com/amazon-sagemaker-operator-for-k8s:v1 because my ec2 in us-west-2 but it happened the same , I also tried to
$(aws ecr get-login --registry-ids myaccountid --region us-west-2 --no-include-email)
and it said login succeed but it happened still.
the aws configure worked well. what could I to solve the problem or how to use amazon-sagemaker-operator-for-k8s in a normal k8s inside of eks?

Add checksum file to enable the installation of the chart from helm

What would you like to be added:
Add a file with checksums (index.yaml) to enable helm or terraform to install the chart automatically from this github repo.

Why is this needed:
For chart installation automation.

aws / amazon-sagemaker-operator-for-k8s Goto Github PK

amazon-sagemaker-operator-for-k8s's Introduction

Amazon SageMaker Operators for Kubernetes

Introduction

Migrate resources to the new SageMaker Operators for Kubernetes

Usage

YAML Examples

Releases

Contributing

License

amazon-sagemaker-operator-for-k8s's People

Contributors

Stargazers

Watchers

Forkers

amazon-sagemaker-operator-for-k8s's Issues

Recommend Projects

Recommend Topics

Recommend Org