flmu / mlflow-tracking-server Goto Github PK

View Code? Open in Web Editor NEW

67.0 2.0 19.0 53 KB

MLFLow Tracking Server based on Docker and AWS S3

Home Page: https://www.mlflow.org/docs/latest/tracking.html

License: MIT License

Shell 43.86% Dockerfile 13.81% Python 42.33%

docker docker-image mlflow mlflow-tracking-server docker-file databricks machine-learning ui aws aws-s3

mlflow-tracking-server's Introduction

MLFLow Tracking Server Based on Docker and AWS S3

This repo provides a docker image of MLFLow Tracking Server which is based on sqlite, an internal file system for metadata (e.g. parameters, metrics) and an AWS S3 Bucket for files and artifacts.

Prerequisites

Before you start the MLFlow Tracking Server, you must create an AWS S3 Bucket and the corresponding credentials.

AWS Account
AWS CLI
Create a new AWS S3 Bucket
Add a new AWS User to to the system
Assign the following policy to the user:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "s3:ListAllMyBuckets",
                "s3:HeadBucket"
            ],
            "Resource": "*"
        },
        {
            "Sid": "VisualEditor1",
            "Effect": "Allow",
            "Action": "s3:*",
            "Resource": [
                "arn:aws:s3:::<_AWS_BUCKET_NAME_>",
                "arn:aws:s3:::<_AWS_BUCKET_NAME_>/*"
            ]
        }
    ]
}

Run the MLFlow Tracking Server

$ docker run \
    --rm \
    --name mlflow-tracking-server \
    -p 5000:5000 \
    -e PORT=5000 \
    -e FILE_DIR=/mlflow \
    -e AWS_BUCKET=<YOUR_AWS_BUCKET> \
    -e AWS_ACCESS_KEY_ID=<YOUR_AWS_ACCESS_KEY_ID> \
    -e AWS_SECRET_ACCESS_KEY=<YOUR_AWS_SECRET_ACCESS_KEY> \
    foxrider/mlflow-tracking-server:0.2.0

Access to http://127.0.0.1:5000

Environment Variables for the MLFlow Tracking Server

Required

Key	Description
`FILE_DIR`	Directory for artifacts and metadata (e.g. parameters, metrics)
`AWS_BUCKET`	Name of AWS S3 Bucket that will contain the artifacts
`AWS_ACCESS_KEY_ID`	AWS-Access-Key that you have created in the `Prerequisite` section
`AWS_SECRET_ACCESS_KEY`	AWS-Secret-Access-Key that you have created in the `Prerequisite` section

Optional

Key	Description	Default
`PORT`	Value for `listen` directive	`5000`

Test the MLFlow Tracking Server

Prerequisite

Install the AWS CLI
Add the AWS credentials to the client. You can either set the environment variables AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY on your client or use the AWS CLI command aws configure.

Simple Python Script

See this link

Contribution

Pull requests are welcome :)

Project Initiator

Florian Muchow

Contributors

Thanks to all the contributors

mlflow-tracking-server's People

Contributors

Stargazers

Watchers

Forkers

kielrodriguez asbe vic0777 goldminer09 zenaptix-lab minyk codegram lightwindy bsteenput aivanni guillaumelab oschistad jmeller 0xkoios tuhinsharma121 minglunwu szymonrucinski np-completed minhtuannn

mlflow-tracking-server's Issues

TODO: Migrate to Github Action Workflow / Remove Travis

Change the Storage Model for the MLFlow Tracking Server

Change the artifact storage model from an internal file system to AWS S3

No module named 'boto3'

When trying to access artifacts/data in S3 storage, I get the following error:

[2019-09-27 11:56:02 +0000] [46] [INFO] Starting gunicorn 19.9.0
[2019-09-27 11:56:02 +0000] [46] [INFO] Listening at: http://0.0.0.0:5000 (46)
[2019-09-27 11:56:02 +0000] [46] [INFO] Using worker: sync
[2019-09-27 11:56:02 +0000] [49] [INFO] Booting worker with pid: 49
[2019-09-27 11:56:02 +0000] [50] [INFO] Booting worker with pid: 50
[2019-09-27 11:56:02 +0000] [82] [INFO] Booting worker with pid: 82
[2019-09-27 11:56:03 +0000] [114] [INFO] Booting worker with pid: 114
/usr/local/lib/python3.7/site-packages/jinja2/utils.py:485: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
  from collections import MutableMapping
/usr/local/lib/python3.7/site-packages/jinja2/runtime.py:318: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
  from collections import Mapping
2019/09/27 11:56:42 ERROR mlflow.server: Exception on /ajax-api/2.0/preview/mlflow/artifacts/list [GET]
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 2446, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1951, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1820, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/local/lib/python3.7/site-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1949, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1935, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/usr/local/lib/python3.7/site-packages/mlflow/server/handlers.py", line 101, in wrapper
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/mlflow/server/handlers.py", line 352, in _list_artifacts
    artifact_entities = _get_artifact_repo(run).list_artifacts(path)
  File "/usr/local/lib/python3.7/site-packages/mlflow/store/s3_artifact_repo.py", line 65, in list_artifacts
    s3_client = self._get_s3_client()
  File "/usr/local/lib/python3.7/site-packages/mlflow/store/s3_artifact_repo.py", line 27, in _get_s3_client
    import boto3
ModuleNotFoundError: No module named 'boto3'

This can be fixed by running pip install boto3

Any way I can look into the code that builds the docker image?

Error when mlflow tries to write to /mlflow folder

I'm trying to use your image and use the tracking server when I run models from a notebook run through a server spinned by jupyterhub, which runs in a different docker container (I start both with docker-compose). However, although mlflow works correctly and I can see my model runs in the UI, I get an error message when mlflow tries to write the artifact to the /mlflow folder:

PermissionError: [Errno 13] Permission denied: '/mlflow'

Do you have any idea of how I can get the two containers to play nicely with each other? (sorry if the question is dumb, I'm a complete docker newbie).

--file-store deprecated

mlflow 1.0.0 deprecated --file-store:
https://github.com/mlflow/mlflow/releases

docker image no longer runs given that it pulls in the latest mlflow

Port issue

The given port can not be used.

flmu / mlflow-tracking-server Goto Github PK

mlflow-tracking-server's Introduction

MLFLow Tracking Server Based on Docker and AWS S3

Prerequisites

Run the MLFlow Tracking Server

Environment Variables for the MLFlow Tracking Server

Required

Optional

Test the MLFlow Tracking Server

Prerequisite

Simple Python Script

Contribution

Project Initiator

Contributors

mlflow-tracking-server's People

Contributors

Stargazers

Watchers

Forkers

mlflow-tracking-server's Issues

Recommend Projects

Recommend Topics

Recommend Org