Comments (9)
With #44 merged, @pm3310 can you try to see if the errors you were getting persist?
@andreas-grivas You should not need to use your workaround anymore, let me know if that's not the case and I'll get to it 😄
from sagify.
New version deployed https://pypi.org/project/sagify/ :-D Version 0.12.1 is available via pypi
from sagify.
I think this may be because an additional tag seems to be added to image_name from two locations after the changes in 0.11.0.
The first time docker_tag is added is in sagify/api/cloud.py:
image_name = config.image_name+':'+docker_tag
however image_name is passed as an argument to functions in sagify/sagemaker/sagemaker.py, which then call:
def _construct_image_location(self, image_name):
account = self.boto_session.client('sts').get_caller_identity()['Account']
region = self.boto_session.region_name
return '{account}.dkr.ecr.{region}.amazonaws.com/{image}:latest'.format(
account=account,
region=region,
image=image_name
)
Note that in the return statement ':latest' is always appended - so I think what potentially happens is you get an image_name with ':something:something' appended and sagemaker doesn't like that.
I found these changes have solved this error for me for now:
https://github.com/andreas-grivas/sagify/commit/d23cccd19106c50f1f7a50e17cbf9e0aac636755
Until this is investigated further, a quick fix is to specify the name parameter in the call to sage.Model() - eg. sage.Model(name='MyModel', restofparams) in the deploy function under sagify/sagemaker/sagemaker.py:
from sagify.
I think I found the issue. SageMaker doesn't like underscores (_
) in the image names :-(
from sagify.
Does sagify add underscores anywhere though?
from sagify.
@pm3310 It could be underscores too, but what @andreas-grivas suggests is definitely happening and needs fixing.
A colon is appended to the image name every time we call sagify cloud ...
, regardless of whether a tag was provided or not.
SageMakerClient
though also appends the latest tag every time as well.
In other words, if not tag is provided, we end up with an image name like this:
name-img::latest
(note the double colon)
and when a tag e.g. 'my-tag' is indeed provided the image name looks like this:
name-img:my-tag:latest
The fix is straightforward I think. We need to check for the existence of a tag in the above linked lines and append the tag or latest
respectively only if needed.
The docker tag feature was implemented rather hastily. In addition to the above, no additional method doc entries were added where the docker_tag
parameter was added; we can add them as part of fixing the main issue.
from sagify.
@andreas-grivas Would you like to become a Sagify active contributor? If you wish so, I can give you contributor access permissions :-) That means you'll be able to deploy new versions and review PRs :-)
from sagify.
@pm3310 Thank you Pavlos, that would be super awesome but it is still uncertain if sagemaker fits our needs - so I don't know if I will be working with sagify a lot in the future. @ilazakis @pm3310 Thank you both for fixing this :)
from sagify.
Hey @andreas-grivas no worries :-) We're planning to launch an enterprise solution early next year for training/deploying ML models on SageMaker much easier. Of course, it's going to be based on this open-source project :-)
The benefits of the solution will be:
- Hide engineering details of SageMaker and focus more on ML
- Keeping track of historical training jobs (who, what, where, when, which git commit)
- Deploy ML models as REST endpoints in 1 click from UI
Please, let me know if you'd like to be of the early adopters
from sagify.
Related Issues (20)
- Exploit SageMaker managed spot training HOT 2
- Adding the possibility to drop deployed models and endpoints HOT 3
- Optimize Docker image
- CreateTrainingJob operation: Invalid MaxWaitTimeInSeconds HOT 3
- Invalid type for parameter ExternalId, value: None HOT 8
- Amazon SageMaker Autopilot Support HOT 1
- Cloud deploy not working HOT 1
- push docker image fails with AWS CLI v2 HOT 4
- Error with credentials location when using a role and profile for sagify cloud train HOT 3
- No Local GPU HOT 2
- error when pushing docker image : no credentials found HOT 2
- Feature request: support SageMaker SDK v2 HOT 2
- motivation is missing HOT 1
- Replace Travis with GitHub Actions HOT 1
- AWS profile choice default throws IndexError HOT 2
- Version conflict ulrlib while running sagify init HOT 4
- Installing the latest version downloads all versions of `boto3` and `botocore` until it fails HOT 3
- Library stopped being maintained? HOT 4
- SageMaker Inference with GPU
- Are there any plans to support multi-model endpoints and/or multi-container? HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from sagify.