henripal / labnotebook Goto Github PK

This is very cool!

Check out comet.ml for a similar and robust application.

We should cycle color per series instead of per graph. A given series should have the same color in the left-hand menu and both charts.

Good morning,

I was trying to install and try labnotebook but an error prevented me from continuing, could you please guide me in solving the problem, if possible?

when I do pip install. or pip3 install . this error follows:

Processing /home/belo/Libs/labnotebook
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 2237, in resolve
        return functools.reduce(getattr, self.attrs, module)
    AttributeError: module 'setuptools.dist' has no attribute 'check_specifier'
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-kspoor7r-build/setup.py", line 19, in <module>
        'start_backend = labnotebook.backend.api:start_backend'
      File "/usr/lib/python3.5/distutils/core.py", line 108, in setup
        _setup_distribution = dist = klass(attrs)
      File "/usr/lib/python3/dist-packages/setuptools/dist.py", line 272, in __init__
        _Distribution.__init__(self,attrs)
      File "/usr/lib/python3.5/distutils/dist.py", line 281, in __init__
        self.finalize_options()
      File "/usr/lib/python3/dist-packages/setuptools/dist.py", line 327, in finalize_options
        ep.load()(self, ep.name, value)
      File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 2229, in load
        return self.resolve()
      File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 2239, in resolve
        raise ImportError(str(exc))
    ImportError: module 'setuptools.dist' has no attribute 'check_specifier'
    
    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-kspoor7r-build/

Best Regards,

David Belo

I followed the instructions in basic_usage.ipynb but the frontend doesn't show any experiments. sqlalchemy method works fine.

And another point: does it makes sense to serve the index.html file instead of making a user to "double click" it? It is not always possible/easy to access a remote filesystem.

This could be done by smarter data-loading rather than the current brute force approach.
Open to other ideas.

This would optionally store the postgres password, username, host, and port, which would be used by start_backend and by the main package.
This could also contain the desired API host and port.

• live update doesn't update custom fields
• hangs when custom fields dict is not present

API log is full of:

127.0.0.1 - - [28/Mar/2018 01:15:49] "GET /steps/1 HTTP/1.1" 500 -
Traceback (most recent call last):
File "/home/koder/workspace/venvs/workload_placement/lib/python3.6/site-packages/flask/app.py", line 1997, in call
return self.wsgi_app(environ, start_response)
File "/home/koder/workspace/venvs/workload_placement/lib/python3.6/site-packages/flask/app.py", line 1985, in wsgi_app
response = self.handle_exception(e)
File "/home/koder/workspace/venvs/workload_placement/lib/python3.6/site-packages/flask_restful/init.py", line 273, in error_router
return original_handler(e)
File "/home/koder/workspace/venvs/workload_placement/lib/python3.6/site-packages/flask_cors/extension.py", line 161, in wrapped_function
return cors_after_request(app.make_response(f(*args, **kwargs)))
File "/home/koder/workspace/venvs/workload_placement/lib/python3.6/site-packages/flask/app.py", line 1540, in handle_exception
reraise(exc_type, exc_value, tb)
File "/home/koder/workspace/venvs/workload_placement/lib/python3.6/site-packages/flask/_compat.py", line 32, in reraise
raise value.with_traceback(tb)
File "/home/koder/workspace/venvs/workload_placement/lib/python3.6/site-packages/flask/app.py", line 1982, in wsgi_app
response = self.full_dispatch_request()
File "/home/koder/workspace/venvs/workload_placement/lib/python3.6/site-packages/flask/app.py", line 1614, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/home/koder/workspace/venvs/workload_placement/lib/python3.6/site-packages/flask_restful/init.py", line 273, in error_router
return original_handler(e)
File "/home/koder/workspace/venvs/workload_placement/lib/python3.6/site-packages/flask_cors/extension.py", line 161, in wrapped_function
return cors_after_request(app.make_response(f(*args, **kwargs)))
File "/home/koder/workspace/venvs/workload_placement/lib/python3.6/site-packages/flask/app.py", line 1517, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/home/koder/workspace/venvs/workload_placement/lib/python3.6/site-packages/flask/_compat.py", line 32, in reraise
raise value.with_traceback(tb)
File "/home/koder/workspace/venvs/workload_placement/lib/python3.6/site-packages/flask/app.py", line 1612, in full_dispatch_request
rv = self.dispatch_request()
File "/home/koder/workspace/venvs/workload_placement/lib/python3.6/site-packages/flask/app.py", line 1598, in dispatch_request
return self.view_functionsrule.endpoint
File "/home/koder/workspace/venvs/workload_placement/lib/python3.6/site-packages/flask_restful/init.py", line 480, in wrapper
resp = resource(*args, **kwargs)
File "/home/koder/workspace/venvs/workload_placement/lib/python3.6/site-packages/flask/views.py", line 84, in view
return self.dispatch_request(*args, **kwargs)
File "/home/koder/workspace/venvs/workload_placement/lib/python3.6/site-packages/flask_restful/init.py", line 595, in dispatch_request
resp = meth(*args, **kwargs)
File "/home/koder/workspace/venvs/workload_placement/lib/python3.6/site-packages/labnotebook/backend/api.py", line 45, in get
ts.trainloss).filter(ts.run_id == run_id).all()
File "/home/koder/workspace/venvs/workload_placement/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 2737, in all
return list(self)
File "/home/koder/workspace/venvs/workload_placement/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 2889, in iter
return self._execute_and_instances(context)
File "/home/koder/workspace/venvs/workload_placement/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 2912, in _execute_and_instances
result = conn.execute(querycontext.statement, self._params)
File "/home/koder/workspace/venvs/workload_placement/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 948, in execute
return meth(self, multiparams, params)
File "/home/koder/workspace/venvs/workload_placement/lib/python3.6/site-packages/sqlalchemy/sql/elements.py", line 269, in _execute_on_connection
return connection._execute_clauseelement(self, multiparams, params)
File "/home/koder/workspace/venvs/workload_placement/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1060, in _execute_clauseelement
compiled_sql, distilled_params
File "/home/koder/workspace/venvs/workload_placement/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1200, in _execute_context
context)
File "/home/koder/workspace/venvs/workload_placement/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1413, in _handle_dbapi_exception
exc_info
File "/home/koder/workspace/venvs/workload_placement/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause
reraise(type(exception), exception, tb=exc_tb, cause=cause)
File "/home/koder/workspace/venvs/workload_placement/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 186, in reraise
raise value.with_traceback(tb)
File "/home/koder/workspace/venvs/workload_placement/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1193, in _execute_context
context)
File "/home/koder/workspace/venvs/workload_placement/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 507, in do_execute
cursor.execute(statement, parameters)
sqlalchemy.exc.InternalError: (psycopg2.InternalError) current transaction is aborted, commands ignored until end of transaction block
[SQL: 'SELECT trainingsteps.step_id AS trainingsteps_step_id, trainingsteps.timestep AS trainingsteps_timestep, trainingsteps.run_id AS trainingsteps_run_id, trainingsteps.trainacc AS trainingsteps_trainacc, trainingsteps.valacc AS trainingsteps_valacc, trainingsteps.trainloss AS trainingsteps_trainloss \nFROM trainingsteps \nWHERE trainingsteps.run_id = %(run_id_1)s'] [parameters: {'run_id_1': '1'}] (Background on this error at: http://sqlalche.me/e/2j85)

python 3.6, postgresql-9.6(9.6.8-0ubuntu0.17.10)

pip freeze
absl-py==0.1.12
aniso8601==3.0.0
astor==0.6.2
astroid==1.6.0
attrs==17.4.0
better-exceptions==0.2.1
bleach==1.5.0
bokeh==0.12.13
cachetools==2.0.1
certifi==2018.1.18
chardet==3.0.4
click==6.7
colorama==0.3.9
cycler==0.10.0
Cython==0.27.3
decorator==4.2.1
entrypoints==0.2.3
fbprophet==0.2.1
flake8==3.5.0
flake8-polyfill==1.0.2
Flask==0.12.2
Flask-Cors==3.0.3
Flask-RESTful==0.3.6
gast==0.2.0
google-auth==1.3.0
gprof2dot==2017.9.19
grpcio==1.10.0
h5py==2.7.1
html5lib==0.9999999
idna==2.6
ipaddress==1.0.19
ipdb==0.11
ipykernel==4.8.2
ipython==6.2.1
ipython-genutils==0.2.0
ipywidgets==7.1.2
iso8601==0.1.12
isort==4.2.15
itsdangerous==0.24
jedi==0.11.1
Jinja2==2.10
json-rpc==1.10.8
jsonschema==2.6.0
jupyter==1.0.0
jupyter-client==5.2.3
jupyter-console==5.2.0
jupyter-core==4.4.0
jupyterlab==0.31.12
jupyterlab-launcher==0.10.5
Keras==2.1.5
kubernetes==4.0.0
labnotebook==0.1
lazy-object-proxy==1.3.1
line-profiler==2.1.2
logging-tree==1.7
mando==0.6.4
Markdown==2.6.11
MarkupSafe==1.0
marshmallow==2.15.0
matplotlib==2.1.1
mccabe==0.6.1
mistune==0.8.3
mypy==0.560
nbconvert==5.3.1
nbformat==4.4.0
notebook==5.4.1
numpy==1.14.0
oauthlib==2.0.6
pandas==0.22.0
pandocfilters==1.4.2
parso==0.1.1
petname==2.2
pew==1.1.2
pexpect==4.4.0
pickleshare==0.7.4
Pillow==5.0.0
pipenv==11.9.0
pkg-resources==0.0.0
pluggy==0.6.0
plumbum==1.6.5
prometheus-client==0.1.1
prompt-toolkit==1.0.15
protobuf==3.5.2.post1
psutil==5.4.3
psycopg2==2.7.4
ptyprocess==0.5.2
py==1.5.2
pyasn1==0.4.2
pyasn1-modules==0.2.1
pycodestyle==2.3.1
pyflakes==1.6.0
Pygments==2.2.0
pylint==1.8.1
pyparsing==2.2.0
pystan==2.17.1.0
pytest==3.4.0
pytest-profiling==1.2.11
python-dateutil==2.7.0
pytz==2017.3
PyYAML==3.12
pyzmq==17.0.0
qtconsole==4.3.1
radon==2.2.0
requests==2.18.4
requests-oauthlib==0.8.0
rpyc==3.4.4
rsa==3.4.2
scikit-learn==0.19.1
scipy==1.0.0
Send2Trash==1.5.0
serpent==1.23
simplegeneric==0.8.1
six==1.11.0
sklearn==0.0
SQLAlchemy==1.2.5
tensorboard==1.6.0
tensorflow==1.6.0
tensorflow-gpu==1.6.0
termcolor==1.1.0
terminado==0.8.1
testpath==0.3.1
texttable==1.2.1
torch==0.3.1
torchfile==0.1.0
tornado==5.0.1
traitlets==4.3.2
typed-ast==1.1.0
urllib3==1.22
virtualenv==15.2.0
virtualenv-clone==0.3.0
visdom==0.1.7
wcwidth==0.1.7
webencodings==0.5.1
websocket-client==0.40.0
Werkzeug==0.14.1
widgetsnbextension==3.1.4
wrapt==1.10.11

solution with no time constraints: Kafka + Socket.io but that would be overkill for this project I think.
Anyone have a 'light touch' idea?

After the work is completed, the backend server stops running.
The situation is not corrected by the completion of the accompanying python processes in the Task Manager.
The backend is restarted only after the system is rebooted.
System:
Windows 8.1 x64
Python 3.6.1: Anaconda custom (64 bit)
labnotebook==0.1

I run the basic_usage.ipynb with anaconda 3.5 , it threw some error to me.

InvalidRequestError Traceback (most recent call last)
in ()
1 # we start the experiment and output it to an 'experiment' variable
2 # we can then pass this experiment to step_experiment and end_experiment
----> 3 experiment = labnotebook.start_experiment(model_desc = model_desc)
4
5

D:\Anaconda35\lib\site-packages\labnotebook\main.py in start_experiment(dt, gpu_id, model_desc)
54 model_desc = model_desc)
55 labnotebook.session.add(xp)
---> 56 labnotebook.session.commit()
57
58 return xp

D:\Anaconda35\lib\site-packages\sqlalchemy\orm\session.py in commit(self)
941 raise sa_exc.InvalidRequestError("No transaction is begun.")
942
--> 943 self.transaction.commit()
944
945 def prepare(self):

D:\Anaconda35\lib\site-packages\sqlalchemy\orm\session.py in commit(self)
463
464 def commit(self):
--> 465 self._assert_active(prepared_ok=True)
466 if self._state is not PREPARED:
467 self._prepare_impl()

D:\Anaconda35\lib\site-packages\sqlalchemy\orm\session.py in _assert_active(self, prepared_ok, rollback_ok, deactive_ok, closed_msg)
274 "first issue Session.rollback()."
275 " Original exception was: %s"
--> 276 % self._rollback_exception
277 )
278 elif not deactive_ok:

InvalidRequestError: This Session's transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: (psycopg2.ProgrammingError) 错误: 关系 "experiments" 不存在
LINE 1: INSERT INTO experiments (run_id, dt, duration, gpu, model_de...
^
[SQL: "INSERT INTO experiments (run_id, dt, duration, gpu, model_desc, final_trainloss, final_trainacc, final_valacc, completed) VALUES (nextval('run_is_seq'), %(dt)s, %(duration)s, %(gpu)s, %(model_desc)s, %(final_trainloss)s, %(final_trainacc)s, %(final_valacc)s, %(completed)s) RETURNING experiments.run_id"] [parameters: {'gpu': 0, 'completed': False, 'duration': None, 'final_valacc': None, 'final_trainacc': None, 'model_desc': '{"sigma": 0.1}', 'dt': datetime.datetime(2018, 3, 20, 21, 17, 58, 230308), 'final_trainloss': None}] (Background on this error at: http://sqlalche.me/e/f405)

``