hepdata / hepdata_lib Goto Github PK

View Code? Open in Web Editor NEW

15.0 4.0 35.0 968 KB

Library for getting your data into HEPData

Home Page: https://hepdata-lib.readthedocs.io

License: MIT License

Python 98.89% Shell 0.11% Dockerfile 1.00%

hacktoberfest hepdata

hepdata_lib's Introduction

hepdata_lib

Library for getting your data into HEPData

Documentation: https://hepdata-lib.readthedocs.io

This code works with Python 3.6, 3.7, 3.8, 3.9, 3.10, 3.11 or 3.12.

Installation

It is highly recommended you install hepdata_lib into a virtual environment.

python -m pip install hepdata_lib

Alternatively, install from conda-forge using a conda ecosystem package manager:

conda install --channel conda-forge hepdata-lib

If you are not sure about your Python environment, please also see below how to use hepdata_lib in a Docker or Apptainer container.

Getting started

For using hepdata_lib, you don't even need to install it, but can use the binder or SWAN (CERN-only) services using one of the buttons below:

You can also use the Docker image:

docker run --rm -it -p 8888:8888 -v ${PWD}:/home/hepdata ghcr.io/hepdata/hepdata_lib:latest

And then point your browser to http://localhost:8888 and use the token that is printed out. The output will end up in your current working directory (${PWD}).

If you prefer a shell, instead run:

docker run --rm -it -p 8888:8888 -v ${PWD}:/home/hepdata ghcr.io/hepdata/hepdata_lib:latest bash

If on CERN LXPLUS or anywhere else where there is Apptainer available but not Docker, you can still use the docker image:

export APPTAINER_CACHEDIR="/tmp/$(whoami)/apptainer"
apptainer shell -B /afs -B /eos docker://ghcr.io/hepdata/hepdata_lib:latest bash

Unpacking the image can take a few minutes the first time you use it. Please be patient. Both EOS and AFS should be available and the output will be in your current working directory.

Further examples

There are a few more examples available that can directly be run using the binder links below or using SWAN (CERN-only, please use LCG release LCG_94 or later) and selecting the corresponding notebook manually:

External dependencies

Make sure that you have ROOT in your $PYTHONPATH and that the convert command is available by adding its location to your $PATH if needed.

A ROOT installation is not strictly required if your input data is not in a ROOT format, for example, if your input data is provided as text files or scikit-hep/hist histograms. Most of the hepdata_lib functionality can be used without a ROOT installation, other than the RootFileReader and CFileReader classes, and other functions of the hepdata_lib.root_utils module.

hepdata_lib's People

Contributors

Stargazers

Watchers

hepdata_lib's Issues

Integration with pyhf JSON workspaces

/cc @matthewfeickert @lukasheinrich -- we should probably file an issue here to investigate the possibility of getting the HEPData to handle pyhf JSON specifications (additionally teaching it to export the given specification to root+xml as well if needed).

I'm hoping to use this issue as a place to hold discussion on this. For reference, we do have a JSON schema that fully specifies the workspace and will be releasing a pyhf version on pypi shortly that contains the v1.0.0 of this schema.

Need tests for jupyter notebooks

Currently, we do not know if our notebooks are still working if we change the code. I think we could make use of papermill to run the notebooks in our tests.

combine_limit example showcases bad practice?

In the combine_limit example both 1-sigma and 2-sigma uncertainties are added to the expected value, but this seems to go against the CMS Higgs HepData recommendation.
From https://twiki.cern.ch/twiki/bin/viewauth/CMS/HiggsHepData:

For expected limits ("Brazil plots"), please note that it is not recommended to give the 1σ and 2σ bands as independent errors, since the "sum errors" button will sum them in quadrature and give meaningless results. It is better to give the expected limit twice (as two columns/graphs), one with the 1-sigma and the other with the 2-sigma uncertainty.

Docker container fails to build due to issue with installing jupyter

This also affects binder, possibly SWAN as well, legacy python issue. Solution is probably to fix ipython==5.7 and ipykernel=4.10 for python 2.7 as described at https://stackoverflow.com/questions/52733094/error-ipykernel-requires-python-version-3-4-or-above

Collecting ipykernel (from jupyter)
  Downloading https://files.pythonhosted.org/packages/11/0b/95330660f8cc5d63428b9886c800ea8d68842fd866389cf579acca4915be/ipykernel-5.1.0.tar.gz (101kB)
    Complete output from command python setup.py egg_info:
    ERROR: ipykernel requires Python version 3.4 or above.

    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-623fZJ/ipykernel/
You are using pip version 8.1.1, however version 18.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

Add flag to disable features requiring ROOT

There are use cases where the tool is used but no ROOT installation is available. We should add an option that disables all features that use ROOT to make the other features work independently (e.g. reading in a text file). For ROOT TTrees, we could fall back to uproot.

Create_files error with Python3

There is an error when creating the output file in a Python3 environment
submission.create_files(outdir)

The issue is cause by line 31 in init.py:
return dumper.represent_dict(data.iteritems())

It could be fixed by using:
return dumper.represent_dict(data.items())

I have also tested that this change does not harm the use with Python2.

Here the error message from the Getting_started notebook on SWAN

Click to expand the full error message

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-13-a8ab94870854> in <module>
      1 outdir = "example_output"
----> 2 submission.create_files(outdir)

~/.local/lib/python3.6/site-packages/hepdata_lib/__init__.py in create_files(self, outdir)
    515         # Write all the tables
    516         for table in self.tables:
--> 517             table.write_output(outdir)
    518 
    519         # Copy additional resource files

~/.local/lib/python3.6/site-packages/hepdata_lib/__init__.py in write_output(self, outdir)
    264         """
    265         self.write_images(outdir)
--> 266         self.write_yaml(outdir)
    267 
    268     def write_images(self, outdir):

~/.local/lib/python3.6/site-packages/hepdata_lib/__init__.py in write_yaml(self, outdir)
    339             outdir, '{NAME}.yaml'.format(NAME=shortname))
    340         with open(outfile_path, 'w') as outfile:
--> 341             yaml.dump(table, outfile, default_flow_style=False)
    342 
    343         # Add entry to central submission file

/cvmfs/sft.cern.ch/lcg/views/LCG_96python3/x86_64-centos7-gcc8-opt/lib/python3.6/site-packages/yaml/__init__.py in dump(data, stream, Dumper, **kwds)
    288     If stream is None, return the produced string instead.
    289     """
--> 290     return dump_all([data], stream, Dumper=Dumper, **kwds)
    291 
    292 def safe_dump_all(documents, stream=None, **kwds):

/cvmfs/sft.cern.ch/lcg/views/LCG_96python3/x86_64-centos7-gcc8-opt/lib/python3.6/site-packages/yaml/__init__.py in dump_all(documents, stream, Dumper, default_style, default_flow_style, canonical, indent, width, allow_unicode, line_break, encoding, explicit_start, explicit_end, version, tags, sort_keys)
    276         dumper.open()
    277         for data in documents:
--> 278             dumper.represent(data)
    279         dumper.close()
    280     finally:

/cvmfs/sft.cern.ch/lcg/views/LCG_96python3/x86_64-centos7-gcc8-opt/lib/python3.6/site-packages/yaml/representer.py in represent(self, data)
     25 
     26     def represent(self, data):
---> 27         node = self.represent_data(data)
     28         self.serialize(node)
     29         self.represented_objects = {}

/cvmfs/sft.cern.ch/lcg/views/LCG_96python3/x86_64-centos7-gcc8-opt/lib/python3.6/site-packages/yaml/representer.py in represent_data(self, data)
     46         data_types = type(data).__mro__
     47         if data_types[0] in self.yaml_representers:
---> 48             node = self.yaml_representers[data_types[0]](self, data)
     49         else:
     50             for data_type in data_types:

/cvmfs/sft.cern.ch/lcg/views/LCG_96python3/x86_64-centos7-gcc8-opt/lib/python3.6/site-packages/yaml/representer.py in represent_dict(self, data)
    205 
    206     def represent_dict(self, data):
--> 207         return self.represent_mapping('tag:yaml.org,2002:map', data)
    208 
    209     def represent_set(self, data):

/cvmfs/sft.cern.ch/lcg/views/LCG_96python3/x86_64-centos7-gcc8-opt/lib/python3.6/site-packages/yaml/representer.py in represent_mapping(self, tag, mapping, flow_style)
    116         for item_key, item_value in mapping:
    117             node_key = self.represent_data(item_key)
--> 118             node_value = self.represent_data(item_value)
    119             if not (isinstance(node_key, ScalarNode) and not node_key.style):
    120                 best_style = False

/cvmfs/sft.cern.ch/lcg/views/LCG_96python3/x86_64-centos7-gcc8-opt/lib/python3.6/site-packages/yaml/representer.py in represent_data(self, data)
     46         data_types = type(data).__mro__
     47         if data_types[0] in self.yaml_representers:
---> 48             node = self.yaml_representers[data_types[0]](self, data)
     49         else:
     50             for data_type in data_types:

/cvmfs/sft.cern.ch/lcg/views/LCG_96python3/x86_64-centos7-gcc8-opt/lib/python3.6/site-packages/yaml/representer.py in represent_list(self, data)
    197         #            break
    198         #if not pairs:
--> 199             return self.represent_sequence('tag:yaml.org,2002:seq', data)
    200         #value = []
    201         #for item_key, item_value in data:

/cvmfs/sft.cern.ch/lcg/views/LCG_96python3/x86_64-centos7-gcc8-opt/lib/python3.6/site-packages/yaml/representer.py in represent_sequence(self, tag, sequence, flow_style)
     90         best_style = True
     91         for item in sequence:
---> 92             node_item = self.represent_data(item)
     93             if not (isinstance(node_item, ScalarNode) and not node_item.style):
     94                 best_style = False

/cvmfs/sft.cern.ch/lcg/views/LCG_96python3/x86_64-centos7-gcc8-opt/lib/python3.6/site-packages/yaml/representer.py in represent_data(self, data)
     46         data_types = type(data).__mro__
     47         if data_types[0] in self.yaml_representers:
---> 48             node = self.yaml_representers[data_types[0]](self, data)
     49         else:
     50             for data_type in data_types:

/cvmfs/sft.cern.ch/lcg/views/LCG_96python3/x86_64-centos7-gcc8-opt/lib/python3.6/site-packages/yaml/representer.py in represent_dict(self, data)
    205 
    206     def represent_dict(self, data):
--> 207         return self.represent_mapping('tag:yaml.org,2002:map', data)
    208 
    209     def represent_set(self, data):

/cvmfs/sft.cern.ch/lcg/views/LCG_96python3/x86_64-centos7-gcc8-opt/lib/python3.6/site-packages/yaml/representer.py in represent_mapping(self, tag, mapping, flow_style)
    116         for item_key, item_value in mapping:
    117             node_key = self.represent_data(item_key)
--> 118             node_value = self.represent_data(item_value)
    119             if not (isinstance(node_key, ScalarNode) and not node_key.style):
    120                 best_style = False

/cvmfs/sft.cern.ch/lcg/views/LCG_96python3/x86_64-centos7-gcc8-opt/lib/python3.6/site-packages/yaml/representer.py in represent_data(self, data)
     46         data_types = type(data).__mro__
     47         if data_types[0] in self.yaml_representers:
---> 48             node = self.yaml_representers[data_types[0]](self, data)
     49         else:
     50             for data_type in data_types:

/cvmfs/sft.cern.ch/lcg/views/LCG_96python3/x86_64-centos7-gcc8-opt/lib/python3.6/site-packages/yaml/representer.py in represent_list(self, data)
    197         #            break
    198         #if not pairs:
--> 199             return self.represent_sequence('tag:yaml.org,2002:seq', data)
    200         #value = []
    201         #for item_key, item_value in data:

/cvmfs/sft.cern.ch/lcg/views/LCG_96python3/x86_64-centos7-gcc8-opt/lib/python3.6/site-packages/yaml/representer.py in represent_sequence(self, tag, sequence, flow_style)
     90         best_style = True
     91         for item in sequence:
---> 92             node_item = self.represent_data(item)
     93             if not (isinstance(node_item, ScalarNode) and not node_item.style):
     94                 best_style = False

/cvmfs/sft.cern.ch/lcg/views/LCG_96python3/x86_64-centos7-gcc8-opt/lib/python3.6/site-packages/yaml/representer.py in represent_data(self, data)
     46         data_types = type(data).__mro__
     47         if data_types[0] in self.yaml_representers:
---> 48             node = self.yaml_representers[data_types[0]](self, data)
     49         else:
     50             for data_type in data_types:

~/.local/lib/python3.6/site-packages/hepdata_lib/__init__.py in dict_representer(dumper, data)
     29 def dict_representer(dumper, data):
     30     """represent dict."""
---> 31     return dumper.represent_dict(data.iteritems())
     32 
     33 

AttributeError: 'collections.defaultdict' object has no attribute 'iteritems'

Parse LateX tables

There's been the request/idea to automatically parse LateX tables so that they can be included in HEPData. I guess several examples will have to serve as test cases for this.

Add script to test correct setup

Should check the following:

can ROOT be imported?
is imagemagick available?
is hepdata_lib installed?

Maybe also create some simple table.

Problem with Getting_started.ipynb

Seeing an issue when using ROOT 6.12.04 on Mac OS with python 2.7.14.

/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/yaml/representer.pyc in represent_data(self, data)
55 data_types = self.get_classobj_bases(data.class)+list(data_types)
56 if data_types[0] in self.yaml_representers:
---> 57 node = self.yaml_representers[data_types[0]](self, data)
58 else:
59 for data_type in data_types:

TypeError: unbound method represent_dict() must be called with Representer instance as first argument (got Dumper instance instead)

Add support for relative uncertainties

@GraemeWatt pointed out to me that you can specify relative uncertainties by giving the uncertainty value as a string like 1.0% or 15.3%. We should support and document this possibility.

ImageMagick failure on Ubuntu 18.04

There seems to be an issue with convert failing on Ubuntu 18.04. If you see errors like this one:

RuntimeError: convert-im6.q16: not authorized `input/PDF/Figure_004.pdf' @ error/constitute.c/ReadImage/412.

you can check out this fix on stackoverflow. It seems that ImageMagick now prohibits certain operations by default to avoid security vulnerabilities due to malicious code embedded in certain media files. Unfortunately, the fix requires the user to edit the policy file /etc/ImageMagick-6/policy.xml, which requires admin privileges. I am not sure what exactly triggers the issue, so it's hard to say how many users will be affected. If this becomes a widespread problem, we should look into replacing our use of ImageMagick with something else.

readthedocs build failure: missing ROOT dependency.

For autodoc to work, we somehow need to make ROOT work on the readthedocs build machine.
The trouble is that AFAIK there is no way of executing a custom shell script before the build, so we cannot copy&paste the solution we use for travis.

One hacky solution would be to somehow cram the ROOT installation (e.g. via binary download as done with travis) into setup.py. In that case, we would have to make sure that this really only happens when installing the library on the build server and not for users.

pylint currently not working

pylint was disabled since it was giving issues with the python2.7 tests on TravisCI. We can probably make it work again for the python3 tests only.

Replace keyword lists

The lists are now available in the central hepdata documentation:

https://hepdata-submission.readthedocs.io/en/latest/keywords/observables.html
https://hepdata-submission.readthedocs.io/en/latest/keywords/phrases.html
https://hepdata-submission.readthedocs.io/en/latest/keywords/partlist.html

We can just link to them.

"is" (object identity) operator not appropriate...

to compare returned values, among other things.
Specifically, the line

hepdata_lib/hepdata_lib/__init__.py

Line 117 in 0cbc5ac

if lenvar and (lenvar is not lenunc):

bit me: ValueError: Length of uncertainty list (616)is not the same as length of Variablevalues list (616)!.

Have a look at
http://blog.lerner.co.il/why-you-should-almost-never-use-is-in-python/

Or just in the interpreter on an x86_64 machine with Python 2.7.11 (default, Dec 19 2016, 17:44:19) [GCC 5.3.0] on linux2:

>>> 6 is 6
True
>>> 600 is 600
True
>>> x,y=len(range(6)),len(range(6))
>>> x==y,x is y
(True, True)
>>> x,y=len(range(600)),len(range(600))
>>> x==y,x is y
(True, False)

Document how to add a new test

We should have clear documentation so that contributors know how to to it.

"Open in SWAN" link does not open the notebook

When clicking on the link https://cern.ch/swanserver/cgi-bin/go?projurl=https://github.com/HEPData/hepdata_lib.git in the manual, a page with empty projects list is opened in SWAN and you have to insert the link manually to open the notebook.
Perhaps the SWAN API has changed?

Add bumpversion developer documentation

Clean up the following

Workflow

git checkout -b featurebranch
hack-hack-hack
git add -u
git commit
git push origin featurebranch

--> merge

git checkout master
git pull
bumpversion patch
git push origin master --tags

.bumpversion.cfg contains the list of files that contain the version numbers, e.g. setup.py

patch = + 0.0.1
minor = + 0.1.0
major = + 1

Check if ImageMagick is available

Currently, if one has added an image to a table, the convert command will fail when calling create_files(outdir). Instead, we should check whether the command is available and print out a warning.

Add linter to test

Would suggest to use pylint, which is pretty strict, but also gives useful hints how to fix the errors. Furthermore, provides good configurability via pylintrc, and can be integrated with pytest.

SWAN stack does not have ImageMagick installed

Therefore, convert is not available. Requested this to be added to next stack: https://its.cern.ch/jira/browse/UCA-159

add TH1 reading functionality

I have some example code available.

Include hepdata observables, phrases, and particles in documentation

The following files should be available as rendered html:

Also, it would make sense to extend this at some point, because not all use-cases are covered.

Can probably follow instructions as in https://stackoverflow.com/questions/36797384/embed-external-html-file-using-iframe-in-sphinx

Rework ROOT file reading

Relates to #99

The current purpose of ROOTFileReader is to provide a tool to extract arrays of numbers from histogram and graph axis, or possibly trees. Most of these features are implemented in uproot in a much nicer fashion, so there is no point for us to maintain effectively duplicate code. Instead of using ROOTFileReader as a ROOT-to-array converter, we should use it as a ROOT-to-hepdata_lib.Variable converter.

The following operations can definitely be handled by uproot:

TGraph -> array conversion.
TH1 -> array conversion.
TH2 -> array conversion. Caveat: need 1D array format for dependent variable consistent with what hepdata expects.
TTree -> array conversion.

The only operation we need ROOT for would be to retrieve histograms from TCanvas primitives.

Add documentation on the "reading data" part

Should explain functionality of the RootFileReader. Can also mention other nifty things such as np.loadtxt

Binder build broken

Trying to check whether #58 breaks the binder examples, I saw that the build of the docker container is broken:

Error: typed_ast only runs on Python 3.3 and above.

Full log:

Waiting for build to start...
Cloning into '/tmp/repo2docker3_1m49y8'...
HEAD is now at c30c2d9 Bump version: 0.2.2 → 0.2.3
Using DockerBuildPack builder
Step 1/23 : FROM rootproject/root-ubuntu16:6.12
 ---> 64b627b5345f
Step 2/23 : ENV MAGICK_HOME=/usr
 ---> Using cache
 ---> b8e2ba317335
Step 3/23 : ENV ROOT_VERSION=6.12/07
 ---> Using cache
 ---> 0aab77c6c27f
Step 4/23 : LABEL maintainer="Clemens Lange [email protected]"
 ---> Using cache
 ---> c8d429d773ca
Step 5/23 : ARG BUILD_DATE
 ---> Using cache
 ---> 0ad50da5ce90
Step 6/23 : ARG VCS_REF
 ---> Using cache
 ---> fb698c85520c
Step 7/23 : ARG VCS_URL
 ---> Using cache
 ---> 15339e1c019e
Step 8/23 : LABEL org.label-schema.build-date=$BUILD_DATE       org.label-schema.name="hepdata_lib Docker image"       org.label-schema.description="Provide environment in which to run hepdata_lib"       org.label-schema.url="https://github.com/clelange/hepdata_lib"       org.label-schema.vcs-ref=$VCS_REF       org.label-schema.vcs-url=$VCS_URL       org.label-schema.vendor="CERN"       org.label-schema.version=$ROOT_VERSION       org.label-schema.schema-version="1.0"
 ---> Using cache
 ---> d8abe21387ea
Step 9/23 : USER root
 ---> Using cache
 ---> e6e77d79a7fe
Step 10/23 : RUN apt-get update -qq     && ln -sf /usr/share/zoneinfo/UTC /etc/localtime     && apt-get -y install         python-dev         python-numpy         python-pip         imagemagick     && localedef -i en_US -f UTF-8 en_US.UTF-8     && rm -rf /packages /var/lib/apt/lists/*     && pip install --no-cache-dir         jupyter        metakernel         zmq         notebook==5.*         hepdata_lib
 ---> Running in ba99b71db44e
Reading package lists...
Building dependency tree...
Reading state information...
python-numpy is already the newest version (1:1.11.0-1ubuntu1).
python-dev is already the newest version (2.7.12-1~16.04).
The following additional packages will be installed:
  ghostscript gsfonts imagemagick-6.q16 imagemagick-common libcups2
  libcupsfilters1 libcupsimage2 libdjvulibre-text libdjvulibre21 libgs9
  libgs9-common libijs-0.35 libjbig2dec0 liblqr-1-0 libmagickcore-6.q16-2
  libmagickcore-6.q16-2-extra libmagickwand-6.q16-2 libnetpbm10
  libpython-all-dev librsvg2-2 librsvg2-common libwmf0.2-7 netpbm poppler-data
  python-all python-all-dev python-pip-whl python-setuptools python-wheel
Suggested packages:
  ghostscript-x hpijs imagemagick-doc autotrace cups-bsd | lpr | lprng
  enscript ffmpeg gimp gnuplot grads graphviz hp2xx html2ps libwmf-bin mplayer
  povray radiance sane-utils texlive-base-bin transfig ufraw-batch cups-common
  fonts-droid texlive-lang-cjk inkscape librsvg2-bin libwmf0.2-7-gtk
  poppler-utils fonts-japanese-mincho | fonts-ipafont-mincho
  fonts-japanese-gothic | fonts-ipafont-gothic fonts-arphic-ukai
  fonts-arphic-uming fonts-nanum python-setuptools-doc
The following NEW packages will be installed:
  ghostscript gsfonts imagemagick imagemagick-6.q16 imagemagick-common
  libcupsfilters1 libcupsimage2 libdjvulibre-text libdjvulibre21 libgs9
  libgs9-common libijs-0.35 libjbig2dec0 liblqr-1-0 libmagickcore-6.q16-2
  libmagickcore-6.q16-2-extra libmagickwand-6.q16-2 libnetpbm10
  libpython-all-dev librsvg2-2 librsvg2-common libwmf0.2-7 netpbm poppler-data
  python-all python-all-dev python-pip python-pip-whl python-setuptools
  python-wheel
The following packages will be upgraded:
  libcups2
1 upgraded, 30 newly installed, 0 to remove and 74 not upgraded.
Need to get 16.1 MB of archives.
After this operation, 59.4 MB of additional disk space will be used.
Get:1 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 imagemagick-common all 8:6.8.9.9-7ubuntu5.12 [41.5 kB]
Get:2 http://archive.ubuntu.com/ubuntu xenial/main amd64 liblqr-1-0 amd64 0.4.2-2 [23.6 kB]
Get:3 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 libmagickcore-6.q16-2 amd64 8:6.8.9.9-7ubuntu5.12 [1579 kB]
Get:4 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 libmagickwand-6.q16-2 amd64 8:6.8.9.9-7ubuntu5.12 [288 kB]
Get:5 http://archive.ubuntu.com/ubuntu xenial/main amd64 poppler-data all 0.4.7-7 [1495 kB]
Get:6 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 libcups2 amd64 2.1.3-4ubuntu0.5 [197 kB]
Get:7 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 libcupsfilters1 amd64 1.8.3-2ubuntu3.4 [80.5 kB]
Get:8 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 libcupsimage2 amd64 2.1.3-4ubuntu0.5 [16.1 kB]
Get:9 http://archive.ubuntu.com/ubuntu xenial/main amd64 libijs-0.35 amd64 0.35-12 [15.5 kB]
Get:10 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 libjbig2dec0 amd64 0.12+20150918-1ubuntu0.1 [55.3 kB]
Get:11 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 libgs9-common all 9.18~dfsg~0-0ubuntu2.8 [2979 kB]
Get:12 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 libgs9 amd64 9.18~dfsg~0-0ubuntu2.8 [2057 kB]
Get:13 http://archive.ubuntu.com/ubuntu xenial/main amd64 gsfonts all 1:8.11+urwcyr1.0.7~pre44-4.2ubuntu1 [3374kB]
Get:14 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 ghostscript amd64 9.18~dfsg~0-0ubuntu2.8 [40.9 kB]
Get:15 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 imagemagick-6.q16 amd64 8:6.8.9.9-7ubuntu5.12[387 kB]
Get:16 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 imagemagick amd64 8:6.8.9.9-7ubuntu5.12 [44.8kB]
Get:17 http://archive.ubuntu.com/ubuntu xenial/main amd64 libdjvulibre-text all 3.5.27.1-5 [49.3 kB]
Get:18 http://archive.ubuntu.com/ubuntu xenial/main amd64 libdjvulibre21 amd64 3.5.27.1-5 [583 kB]
Get:19 http://archive.ubuntu.com/ubuntu xenial/main amd64 librsvg2-2 amd64 2.40.13-3 [93.9 kB]
Get:20 http://archive.ubuntu.com/ubuntu xenial/main amd64 libwmf0.2-7 amd64 0.2.8.4-10.5ubuntu1 [147 kB]
Get:21 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 libmagickcore-6.q16-2-extra amd64 8:6.8.9.9-7ubuntu5.12 [59.7 kB]
Get:22 http://archive.ubuntu.com/ubuntu xenial/main amd64 libnetpbm10 amd64 2:10.0-15.3 [58.1 kB]
Get:23 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 libpython-all-dev amd64 2.7.12-1~16.04 [1006 B]
Get:24 http://archive.ubuntu.com/ubuntu xenial/main amd64 librsvg2-common amd64 2.40.13-3 [5034 B]
Get:25 http://archive.ubuntu.com/ubuntu xenial/main amd64 netpbm amd64 2:10.0-15.3 [997 kB]
Get:26 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 python-all amd64 2.7.12-1~16.04 [996 B]
Get:27 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 python-all-dev amd64 2.7.12-1~16.04 [1016 B]
Get:28 http://archive.ubuntu.com/ubuntu xenial-updates/universe amd64 python-pip-whl all 8.1.1-2ubuntu0.4 [1110kB]
Get:29 http://archive.ubuntu.com/ubuntu xenial-updates/universe amd64 python-pip all 8.1.1-2ubuntu0.4 [144 kB]
Get:30 http://archive.ubuntu.com/ubuntu xenial/main amd64 python-setuptools all 20.7.0-1 [169 kB]
Get:31 http://archive.ubuntu.com/ubuntu xenial/universe amd64 python-wheel all 0.29.0-1 [48.0 kB]
debconf: delaying package configuration, since apt-utils is not installed
Fetched 16.1 MB in 1s (9479 kB/s)
Selecting previously unselected package imagemagick-common.
(Reading database ... 53417 files and directories currently installed.)
Preparing to unpack .../imagemagick-common_8%3a6.8.9.9-7ubuntu5.12_all.deb ...
Unpacking imagemagick-common (8:6.8.9.9-7ubuntu5.12) ...
Selecting previously unselected package liblqr-1-0:amd64.
Preparing to unpack .../liblqr-1-0_0.4.2-2_amd64.deb ...
Unpacking liblqr-1-0:amd64 (0.4.2-2) ...
Selecting previously unselected package libmagickcore-6.q16-2:amd64.
Preparing to unpack .../libmagickcore-6.q16-2_8%3a6.8.9.9-7ubuntu5.12_amd64.deb ...
Unpacking libmagickcore-6.q16-2:amd64 (8:6.8.9.9-7ubuntu5.12) ...
Selecting previously unselected package libmagickwand-6.q16-2:amd64.
Preparing to unpack .../libmagickwand-6.q16-2_8%3a6.8.9.9-7ubuntu5.12_amd64.deb ...
Unpacking libmagickwand-6.q16-2:amd64 (8:6.8.9.9-7ubuntu5.12) ...
Selecting previously unselected package poppler-data.
Preparing to unpack .../poppler-data_0.4.7-7_all.deb ...
Unpacking poppler-data (0.4.7-7) ...
Preparing to unpack .../libcups2_2.1.3-4ubuntu0.5_amd64.deb ...
Unpacking libcups2:amd64 (2.1.3-4ubuntu0.5) over (2.1.3-4ubuntu0.4) ...
Selecting previously unselected package libcupsfilters1:amd64.
Preparing to unpack .../libcupsfilters1_1.8.3-2ubuntu3.4_amd64.deb ...
Unpacking libcupsfilters1:amd64 (1.8.3-2ubuntu3.4) ...
Selecting previously unselected package libcupsimage2:amd64.
Preparing to unpack .../libcupsimage2_2.1.3-4ubuntu0.5_amd64.deb ...
Unpacking libcupsimage2:amd64 (2.1.3-4ubuntu0.5) ...
Selecting previously unselected package libijs-0.35:amd64.
Preparing to unpack .../libijs-0.35_0.35-12_amd64.deb ...
Unpacking libijs-0.35:amd64 (0.35-12) ...
Selecting previously unselected package libjbig2dec0.
Preparing to unpack .../libjbig2dec0_0.12+20150918-1ubuntu0.1_amd64.deb ...
Unpacking libjbig2dec0 (0.12+20150918-1ubuntu0.1) ...
Selecting previously unselected package libgs9-common.
Preparing to unpack .../libgs9-common_9.18~dfsg~0-0ubuntu2.8_all.deb ...
Unpacking libgs9-common (9.18~dfsg~0-0ubuntu2.8) ...
Selecting previously unselected package libgs9:amd64.
Preparing to unpack .../libgs9_9.18~dfsg~0-0ubuntu2.8_amd64.deb ...
Unpacking libgs9:amd64 (9.18~dfsg~0-0ubuntu2.8) ...
Selecting previously unselected package gsfonts.
Preparing to unpack .../gsfonts_1%3a8.11+urwcyr1.0.7~pre44-4.2ubuntu1_all.deb ...
Unpacking gsfonts (1:8.11+urwcyr1.0.7~pre44-4.2ubuntu1) ...
Selecting previously unselected package ghostscript.
Preparing to unpack .../ghostscript_9.18~dfsg~0-0ubuntu2.8_amd64.deb ...
Unpacking ghostscript (9.18~dfsg~0-0ubuntu2.8) ...
Selecting previously unselected package imagemagick-6.q16.
Preparing to unpack .../imagemagick-6.q16_8%3a6.8.9.9-7ubuntu5.12_amd64.deb ...
Unpacking imagemagick-6.q16 (8:6.8.9.9-7ubuntu5.12) ...
Selecting previously unselected package imagemagick.
Preparing to unpack .../imagemagick_8%3a6.8.9.9-7ubuntu5.12_amd64.deb ...
Unpacking imagemagick (8:6.8.9.9-7ubuntu5.12) ...
Selecting previously unselected package libdjvulibre-text.
Preparing to unpack .../libdjvulibre-text_3.5.27.1-5_all.deb ...
Unpacking libdjvulibre-text (3.5.27.1-5) ...
Selecting previously unselected package libdjvulibre21:amd64.
Preparing to unpack .../libdjvulibre21_3.5.27.1-5_amd64.deb ...
Unpacking libdjvulibre21:amd64 (3.5.27.1-5) ...
Selecting previously unselected package librsvg2-2:amd64.
Preparing to unpack .../librsvg2-2_2.40.13-3_amd64.deb ...
Unpacking librsvg2-2:amd64 (2.40.13-3) ...
Selecting previously unselected package libwmf0.2-7:amd64.
Preparing to unpack .../libwmf0.2-7_0.2.8.4-10.5ubuntu1_amd64.deb ...
Unpacking libwmf0.2-7:amd64 (0.2.8.4-10.5ubuntu1) ...
Selecting previously unselected package libmagickcore-6.q16-2-extra:amd64.
Preparing to unpack .../libmagickcore-6.q16-2-extra_8%3a6.8.9.9-7ubuntu5.12_amd64.deb ...
Unpacking libmagickcore-6.q16-2-extra:amd64 (8:6.8.9.9-7ubuntu5.12) ...
Selecting previously unselected package libnetpbm10.
Preparing to unpack .../libnetpbm10_2%3a10.0-15.3_amd64.deb ...
Unpacking libnetpbm10 (2:10.0-15.3) ...
Selecting previously unselected package libpython-all-dev:amd64.
Preparing to unpack .../libpython-all-dev_2.7.12-1~16.04_amd64.deb ...
Unpacking libpython-all-dev:amd64 (2.7.12-1~16.04) ...
Selecting previously unselected package librsvg2-common:amd64.
Preparing to unpack .../librsvg2-common_2.40.13-3_amd64.deb ...
Unpacking librsvg2-common:amd64 (2.40.13-3) ...
Selecting previously unselected package netpbm.
Preparing to unpack .../netpbm_2%3a10.0-15.3_amd64.deb ...
Unpacking netpbm (2:10.0-15.3) ...
Selecting previously unselected package python-all.
Preparing to unpack .../python-all_2.7.12-1~16.04_amd64.deb ...
Unpacking python-all (2.7.12-1~16.04) ...
Selecting previously unselected package python-all-dev.
Preparing to unpack .../python-all-dev_2.7.12-1~16.04_amd64.deb ...
Unpacking python-all-dev (2.7.12-1~16.04) ...
Selecting previously unselected package python-pip-whl.
Preparing to unpack .../python-pip-whl_8.1.1-2ubuntu0.4_all.deb ...
Unpacking python-pip-whl (8.1.1-2ubuntu0.4) ...
Selecting previously unselected package python-pip.
Preparing to unpack .../python-pip_8.1.1-2ubuntu0.4_all.deb ...
Unpacking python-pip (8.1.1-2ubuntu0.4) ...
Selecting previously unselected package python-setuptools.
Preparing to unpack .../python-setuptools_20.7.0-1_all.deb ...
Unpacking python-setuptools (20.7.0-1) ...
Selecting previously unselected package python-wheel.
Preparing to unpack .../python-wheel_0.29.0-1_all.deb ...
Unpacking python-wheel (0.29.0-1) ...
Processing triggers for libc-bin (2.23-0ubuntu10) ...
Processing triggers for fontconfig (2.11.94-0ubuntu1.1) ...
Processing triggers for man-db (2.7.5-1) ...
Processing triggers for hicolor-icon-theme (0.15-0ubuntu1) ...
Processing triggers for mime-support (3.59ubuntu1) ...
Processing triggers for libgdk-pixbuf2.0-0:amd64 (2.32.2-1ubuntu1.4) ...
Setting up imagemagick-common (8:6.8.9.9-7ubuntu5.12) ...
Setting up liblqr-1-0:amd64 (0.4.2-2) ...
Setting up libmagickcore-6.q16-2:amd64 (8:6.8.9.9-7ubuntu5.12) ...
Setting up libmagickwand-6.q16-2:amd64 (8:6.8.9.9-7ubuntu5.12) ...
Setting up poppler-data (0.4.7-7) ...
Setting up libcups2:amd64 (2.1.3-4ubuntu0.5) ...
Setting up libcupsfilters1:amd64 (1.8.3-2ubuntu3.4) ...
Setting up libcupsimage2:amd64 (2.1.3-4ubuntu0.5) ...
Setting up libijs-0.35:amd64 (0.35-12) ...
Setting up libjbig2dec0 (0.12+20150918-1ubuntu0.1) ...
Setting up libgs9-common (9.18~dfsg~0-0ubuntu2.8) ...
update-alternatives: using /usr/share/ghostscript/9.18 to provide /usr/share/ghostscript/current (ghostscript-current) in auto mode
Setting up libgs9:amd64 (9.18~dfsg~0-0ubuntu2.8) ...
Setting up gsfonts (1:8.11+urwcyr1.0.7~pre44-4.2ubuntu1) ...
Setting up ghostscript (9.18~dfsg~0-0ubuntu2.8) ...
Setting up imagemagick-6.q16 (8:6.8.9.9-7ubuntu5.12) ...
Setting up imagemagick (8:6.8.9.9-7ubuntu5.12) ...
update-alternatives: using /usr/bin/compare-im6 to provide /usr/bin/compare (compare) in auto mode
update-alternatives: using /usr/bin/animate-im6 to provide /usr/bin/animate (animate) in auto mode
update-alternatives: using /usr/bin/convert-im6 to provide /usr/bin/convert (convert) in auto mode
update-alternatives: using /usr/bin/composite-im6 to provide /usr/bin/composite (composite) in auto mode
update-alternatives: using /usr/bin/conjure-im6 to provide /usr/bin/conjure (conjure) in auto mode
update-alternatives: using /usr/bin/import-im6 to provide /usr/bin/import (import) in auto mode
update-alternatives: using /usr/bin/identify-im6 to provide /usr/bin/identify (identify) in auto mode
update-alternatives: using /usr/bin/stream-im6 to provide /usr/bin/stream (stream) in auto mode
update-alternatives: using /usr/bin/display-im6 to provide /usr/bin/display (display) in auto mode
update-alternatives: using /usr/bin/montage-im6 to provide /usr/bin/montage (montage) in auto mode
update-alternatives: using /usr/bin/mogrify-im6 to provide /usr/bin/mogrify (mogrify) in auto mode
Setting up libdjvulibre-text (3.5.27.1-5) ...
Setting up libdjvulibre21:amd64 (3.5.27.1-5) ...
Setting up librsvg2-2:amd64 (2.40.13-3) ...
Setting up libwmf0.2-7:amd64 (0.2.8.4-10.5ubuntu1) ...
Setting up libmagickcore-6.q16-2-extra:amd64 (8:6.8.9.9-7ubuntu5.12) ...
Setting up libnetpbm10 (2:10.0-15.3) ...
Setting up libpython-all-dev:amd64 (2.7.12-1~16.04) ...
Setting up librsvg2-common:amd64 (2.40.13-3) ...
Setting up netpbm (2:10.0-15.3) ...
Setting up python-all (2.7.12-1~16.04) ...
Setting up python-all-dev (2.7.12-1~16.04) ...
Setting up python-pip-whl (8.1.1-2ubuntu0.4) ...
Setting up python-pip (8.1.1-2ubuntu0.4) ...
Setting up python-setuptools (20.7.0-1) ...
Setting up python-wheel (0.29.0-1) ...
Processing triggers for libc-bin (2.23-0ubuntu10) ...
Processing triggers for libgdk-pixbuf2.0-0:amd64 (2.32.2-1ubuntu1.4) ...
Collecting jupyter
  Downloading https://files.pythonhosted.org/packages/83/df/0f5dd132200728a86190397e1ea87cd76244e42d39ec5e88efd25b2abd7e/jupyter-1.0.0-py2.py3-none-any.whl
Collecting metakernel
  Downloading https://files.pythonhosted.org/packages/49/90/2ef457aa76ca3a7cd2b7f1f1baf97cbd12817e7430394ade52d143105e98/metakernel-0.20.14-py2.py3-none-any.whl (100kB)
Collecting zmq
  Downloading https://files.pythonhosted.org/packages/6e/78/833b2808793c1619835edb1a4e17a023d5d625f4f97ff25ffff986d1f472/zmq-0.0.0.tar.gz
Collecting notebook==5.*
  Downloading https://files.pythonhosted.org/packages/5e/7c/7fd8e9584779d65dfcad9fa2e09c76131a41f999f853a9c7026ed8585586/notebook-5.6.0-py2.py3-none-any.whl (8.9MB)
Collecting hepdata_lib
  Downloading https://files.pythonhosted.org/packages/60/cd/d6b4a383957e1d3ef7efa49d3a595b9c86a4e89428ab33eb62f7777fcf2a/hepdata_lib-0.2.3-py2.py3-none-any.whl
Collecting nbconvert (from jupyter)
  Downloading https://files.pythonhosted.org/packages/39/ea/280d6c0d92f8e3ca15fd798bbcc2ea141489f9539de7133d8fe10ea4b049/nbconvert-5.3.1-py2.py3-none-any.whl (387kB)
Collecting ipywidgets (from jupyter)
  Downloading https://files.pythonhosted.org/packages/ed/96/8ad75d90820d7e1a06fa0f4cc25521a5854e35e75f2b22e65299343a0c3d/ipywidgets-7.3.2-py2.py3-none-any.whl (110kB)
Collecting qtconsole (from jupyter)
  Downloading https://files.pythonhosted.org/packages/90/ff/047e0dca2627b162866920e7aa93f04523c0ae81e5c67060eec85701992d/qtconsole-4.3.1-py2.py3-none-any.whl (108kB)
Collecting jupyter-console (from jupyter)
  Downloading https://files.pythonhosted.org/packages/77/82/6469cd7fccf7958cbe5dce2e623f1e3c5e27f1bb1ad36d90519bc2d5d370/jupyter_console-5.2.0-py2.py3-none-any.whl
Collecting ipykernel (from jupyter)
  Downloading https://files.pythonhosted.org/packages/8e/65/c7ca3e3d05f9bd51b3010076b84f4e7304b12d0abf62a48f6cec2c90c019/ipykernel-4.8.2-py2-none-any.whl (108kB)
Collecting pexpect>=4.2 (from metakernel)
  Downloading https://files.pythonhosted.org/packages/89/e6/b5a1de8b0cc4e07ca1b305a4fcc3f9806025c1b651ea302646341222f88b/pexpect-4.6.0-py2.py3-none-any.whl (57kB)
Collecting pyzmq (from zmq)
  Downloading https://files.pythonhosted.org/packages/31/c2/60e77eef43deb145dfbd89d805fd652e0c1f1f460ad7ea9035763c1da763/pyzmq-17.1.0-cp27-cp27mu-manylinux1_x86_64.whl (998kB)
Collecting Send2Trash (from notebook==5.*)
  Downloading https://files.pythonhosted.org/packages/13/2e/ea40de0304bb1dc4eb309de90aeec39871b9b7c4bd30f1a3cdcb3496f5c0/Send2Trash-1.5.0.tar.gz
Collecting ipython-genutils (from notebook==5.*)
  Downloading https://files.pythonhosted.org/packages/fa/bc/9bd3b5c2b4774d5f33b2d544f1460be9df7df2fe42f352135381c347c69a/ipython_genutils-0.2.0-py2.py3-none-any.whl
Collecting jinja2 (from notebook==5.*)
  Downloading https://files.pythonhosted.org/packages/7f/ff/ae64bacdfc95f27a016a7bed8e8686763ba4d277a78ca76f32659220a731/Jinja2-2.10-py2.py3-none-any.whl (126kB)
Collecting jupyter-client>=5.2.0 (from notebook==5.*)
  Downloading https://files.pythonhosted.org/packages/94/dd/fe6c4d683b09eb05342bd2816b7779663f71762b4fa9c2d5203d35d17354/jupyter_client-5.2.3-py2.py3-none-any.whl (89kB)
Collecting nbformat (from notebook==5.*)
  Downloading https://files.pythonhosted.org/packages/da/27/9a654d2b6cc1eaa517d1c5a4405166c7f6d72f04f6e7eea41855fe808a46/nbformat-4.4.0-py2.py3-none-any.whl (155kB)
Collecting traitlets>=4.2.1 (from notebook==5.*)
  Downloading https://files.pythonhosted.org/packages/93/d6/abcb22de61d78e2fc3959c964628a5771e47e7cc60d53e9342e21ed6cc9a/traitlets-4.3.2-py2.py3-none-any.whl (74kB)
Collecting tornado>=4 (from notebook==5.*)
  Downloading https://files.pythonhosted.org/packages/45/ec/f2a03a0509bcfca336bef23a3dab0d07504893af34fd13064059ba4a0503/tornado-5.1.tar.gz (516kB)
Collecting terminado>=0.8.1 (from notebook==5.*)
  Downloading https://files.pythonhosted.org/packages/2e/20/a26211a24425923d46e1213b376a6ee60dc30bcdf1b0c345e2c3769deb1c/terminado-0.8.1-py2.py3-none-any.whl
Collecting jupyter-core>=4.4.0 (from notebook==5.*)
  Downloading https://files.pythonhosted.org/packages/1d/44/065d2d7bae7bebc06f1dd70d23c36da8c50c0f08b4236716743d706762a8/jupyter_core-4.4.0-py2.py3-none-any.whl (126kB)
Collecting prometheus-client (from notebook==5.*)
  Downloading https://files.pythonhosted.org/packages/a1/b1/08de091b392fec31da9bd3f5edd9214ec1c6931dd81641610ac20f3ff934/prometheus_client-0.3.1.tar.gz
Collecting ipaddress; python_version == "2.7" (from notebook==5.*)
  Downloading https://files.pythonhosted.org/packages/fc/d0/7fc3a811e011d4b388be48a0e381db8d990042df54aa4ef4599a31d39853/ipaddress-1.0.22-py2.py3-none-any.whl
Collecting pylint (from hepdata_lib)
  Downloading https://files.pythonhosted.org/packages/38/2a/c55cbca6d1f9b34d6bad25e68d7093493fd34fa2945a8a09322897fee530/pylint-2.1.0.tar.gz (555kB)
Collecting future (from hepdata_lib)
  Downloading https://files.pythonhosted.org/packages/00/2b/8d082ddfed935f3608cc61140df6dcbf0edea1bc3ab52fb6c29ae3e81e85/future-0.16.0.tar.gz (824kB)
Requirement already satisfied (use --upgrade to upgrade): numpy in /usr/lib/python2.7/dist-packages (from hepdata_lib)
Collecting PyYAML (from hepdata_lib)
  Downloading https://files.pythonhosted.org/packages/9e/a3/1d13970c3f36777c583f136c136f804d70f500168edc1edea6daa7200769/PyYAML-3.13.tar.gz (270kB)
Collecting pandocfilters>=1.4.1 (from nbconvert->jupyter)
  Downloading https://files.pythonhosted.org/packages/4c/ea/236e2584af67bb6df960832731a6e5325fd4441de001767da328c33368ce/pandocfilters-1.4.2.tar.gz
Collecting entrypoints>=0.2.2 (from nbconvert->jupyter)
  Downloading https://files.pythonhosted.org/packages/cc/8b/4eefa9b47f1910b3d2081da67726b066e379b04ca897acfe9f92bac56147/entrypoints-0.2.3-py2.py3-none-any.whl
Collecting testpath (from nbconvert->jupyter)
  Downloading https://files.pythonhosted.org/packages/15/19/d7bc380c479a184e4a5a9ce516e4e2a773165f89b445f7cdced83d94de25/testpath-0.3.1-py2.py3-none-any.whl (161kB)
Collecting mistune>=0.7.4 (from nbconvert->jupyter)
  Downloading https://files.pythonhosted.org/packages/c8/8c/87f4d359438ba0321a2ae91936030110bfcc62fef752656321a72b8c1af9/mistune-0.8.3-py2.py3-none-any.whl
Collecting pygments (from nbconvert->jupyter)
  Downloading https://files.pythonhosted.org/packages/02/ee/b6e02dc6529e82b75bb06823ff7d005b141037cb1416b10c6f00fc419dca/Pygments-2.2.0-py2.py3-none-any.whl (841kB)
Collecting bleach (from nbconvert->jupyter)
  Downloading https://files.pythonhosted.org/packages/30/b6/a8cffbb9ab4b62b557c22703163735210e9cd857d533740c64e1467d228e/bleach-2.1.3-py2.py3-none-any.whl
Collecting ipython<6.0.0,>=4.0.0; python_version < "3.3" (from ipywidgets->jupyter)
  Downloading https://files.pythonhosted.org/packages/b0/88/d996ab8be22cea1eaa18baee3678a11265e18cf09974728d683c51102148/ipython-5.8.0-py2-none-any.whl (760kB)
Collecting widgetsnbextension~=3.3.0 (from ipywidgets->jupyter)
  Downloading https://files.pythonhosted.org/packages/1e/dd/847f6e5cb98caf819e2dca50109611cc0e43b54e0f0d740ee8586a954b18/widgetsnbextension-3.3.1-py2.py3-none-any.whl (2.2MB)
Collecting prompt-toolkit<2.0.0,>=1.0.0 (from jupyter-console->jupyter)
  Downloading https://files.pythonhosted.org/packages/d1/b0/1a6c262da35c779dd79550137aa7c298a424987240a28792ec5ccf48f848/prompt_toolkit-1.0.15-py2-none-any.whl (247kB)
Collecting ptyprocess>=0.5 (from pexpect>=4.2->metakernel)
  Downloading https://files.pythonhosted.org/packages/d1/29/605c2cc68a9992d18dada28206eeada56ea4bd07a239669da41674648b6f/ptyprocess-0.6.0-py2.py3-none-any.whl
Collecting MarkupSafe>=0.23 (from jinja2->notebook==5.*)
  Downloading https://files.pythonhosted.org/packages/4d/de/32d741db316d8fdb7680822dd37001ef7a448255de9699ab4bfcbdf4172b/MarkupSafe-1.0.tar.gz
Collecting python-dateutil>=2.1 (from jupyter-client>=5.2.0->notebook==5.*)
  Downloading https://files.pythonhosted.org/packages/cf/f5/af2b09c957ace60dcfac112b669c45c8c97e32f94aa8b56da4c6d1682825/python_dateutil-2.7.3-py2.py3-none-any.whl (211kB)
Collecting jsonschema!=2.5.0,>=2.4 (from nbformat->notebook==5.*)
  Downloading https://files.pythonhosted.org/packages/77/de/47e35a97b2b05c2fadbec67d44cfcdcd09b8086951b331d82de90d2912da/jsonschema-2.6.0-py2.py3-none-any.whl
Collecting decorator (from traitlets>=4.2.1->notebook==5.*)
  Downloading https://files.pythonhosted.org/packages/bc/bb/a24838832ba35baf52f32ab1a49b906b5f82fb7c76b2f6a7e35e140bac30/decorator-4.3.0-py2.py3-none-any.whl
Requirement already satisfied (use --upgrade to upgrade): six in /usr/lib/python2.7/dist-packages (from traitlets>=4.2.1->notebook==5.*)
Collecting enum34; python_version == "2.7" (from traitlets>=4.2.1->notebook==5.*)
  Downloading https://files.pythonhosted.org/packages/c5/db/e56e6b4bbac7c4a06de1c50de6fe1ef3810018ae11732a50f15f62c7d050/enum34-1.1.6-py2-none-any.whl
Collecting futures (from tornado>=4->notebook==5.*)
  Downloading https://files.pythonhosted.org/packages/2d/99/b2c4e9d5a30f6471e410a146232b4118e697fa3ffc06d6a65efde84debd0/futures-3.2.0-py2-none-any.whl
Collecting singledispatch (from tornado>=4->notebook==5.*)
  Downloading https://files.pythonhosted.org/packages/c5/10/369f50bcd4621b263927b0a1519987a04383d4a98fb10438042ad410cf88/singledispatch-3.4.0.3-py2.py3-none-any.whl
Collecting backports_abc>=0.4 (from tornado>=4->notebook==5.*)
  Downloading https://files.pythonhosted.org/packages/7d/56/6f3ac1b816d0cd8994e83d0c4e55bc64567532f7dc543378bd87f81cebc7/backports_abc-0.5-py2.py3-none-any.whl
Collecting astroid>=2.0.0 (from pylint->hepdata_lib)
  Downloading https://files.pythonhosted.org/packages/70/5c/61bc9b9feda83bb081fd40903107475959e08f92270bd7465d88a934afa9/astroid-2.0.2.tar.gz (273kB)
Collecting isort>=4.2.5 (from pylint->hepdata_lib)
  Downloading https://files.pythonhosted.org/packages/41/d8/a945da414f2adc1d9e2f7d6e7445b27f2be42766879062a2e63616ad4199/isort-4.3.4-py2-none-any.whl (45kB)
Collecting mccabe (from pylint->hepdata_lib)
  Downloading https://files.pythonhosted.org/packages/87/89/479dc97e18549e21354893e4ee4ef36db1d237534982482c3681ee6e7b57/mccabe-0.6.1-py2.py3-none-any.whl
Collecting configparser>=3.5; python_version == "2.7" (from entrypoints>=0.2.2->nbconvert->jupyter)
  Downloading https://files.pythonhosted.org/packages/7c/69/c2ce7e91c89dc073eb1aa74c0621c3eefbffe8216b3f9af9d3885265c01c/configparser-3.5.0.tar.gz
Collecting html5lib!=1.0b1,!=1.0b2,!=1.0b3,!=1.0b4,!=1.0b5,!=1.0b6,!=1.0b7,!=1.0b8,>=0.99999999pre (from bleach->nbconvert->jupyter)
  Downloading https://files.pythonhosted.org/packages/a5/62/bbd2be0e7943ec8504b517e62bab011b4946e1258842bc159e5dfde15b96/html5lib-1.0.1-py2.py3-none-any.whl (117kB)
Collecting backports.shutil-get-terminal-size; python_version == "2.7" (from ipython<6.0.0,>=4.0.0; python_version < "3.3"->ipywidgets->jupyter)
  Downloading https://files.pythonhosted.org/packages/7d/cd/1750d6c35fe86d35f8562091737907f234b78fdffab42b29c72b1dd861f4/backports.shutil_get_terminal_size-1.0.0-py2.py3-none-any.whl
Requirement already satisfied (use --upgrade to upgrade): setuptools>=18.5 in /usr/lib/python2.7/dist-packages (from ipython<6.0.0,>=4.0.0; python_version < "3.3"->ipywidgets->jupyter)
Collecting pathlib2; python_version == "2.7" or python_version == "3.3" (from ipython<6.0.0,>=4.0.0; python_version < "3.3"->ipywidgets->jupyter)
  Downloading https://files.pythonhosted.org/packages/66/a7/9f8d84f31728d78beade9b1271ccbfb290c41c1e4dc13dbd4997ad594dcd/pathlib2-2.3.2-py2.py3-none-any.whl
Collecting simplegeneric>0.8 (from ipython<6.0.0,>=4.0.0; python_version < "3.3"->ipywidgets->jupyter)
  Downloading https://files.pythonhosted.org/packages/3d/57/4d9c9e3ae9a255cd4e1106bb57e24056d3d0709fc01b2e3e345898e49d5b/simplegeneric-0.8.1.zip
Collecting pickleshare (from ipython<6.0.0,>=4.0.0; python_version < "3.3"->ipywidgets->jupyter)
  Downloading https://files.pythonhosted.org/packages/9f/17/daa142fc9be6b76f26f24eeeb9a138940671490b91cb5587393f297c8317/pickleshare-0.7.4-py2.py3-none-any.whl
Collecting wcwidth (from prompt-toolkit<2.0.0,>=1.0.0->jupyter-console->jupyter)
  Downloading https://files.pythonhosted.org/packages/7e/9f/526a6947247599b084ee5232e4f9190a38f398d7300d866af3ab571a5bfe/wcwidth-0.1.7-py2.py3-none-any.whl
Collecting functools32; python_version == "2.7" (from jsonschema!=2.5.0,>=2.4->nbformat->notebook==5.*)
  Downloading https://files.pythonhosted.org/packages/c5/60/6ac26ad05857c601308d8fb9e87fa36d0ebf889423f47c3502ef034365db/functools32-3.2.3-2.tar.gz
Collecting lazy_object_proxy (from astroid>=2.0.0->pylint->hepdata_lib)
  Downloading https://files.pythonhosted.org/packages/52/7e/f0f570ba363e15251bb9fd452257ec2aff91be0187a08a893afbd8ae225f/lazy_object_proxy-1.3.1-cp27-cp27mu-manylinux1_x86_64.whl (56kB)
Collecting wrapt (from astroid>=2.0.0->pylint->hepdata_lib)
  Downloading https://files.pythonhosted.org/packages/a0/47/66897906448185fcb77fc3c2b1bc20ed0ecca81a0f2f88eda3fc5a34fc3d/wrapt-1.10.11.tar.gz
Collecting typing; python_version < "3.5" (from astroid>=2.0.0->pylint->hepdata_lib)
  Downloading https://files.pythonhosted.org/packages/0d/4d/4e5985d075d241d686a1663fa1f88b61d544658d08c1375c7c6aac32afc3/typing-3.6.4-py2-none-any.whl
Collecting typed_ast; python_version < "3.7" and implementation_name == "cpython" (from astroid>=2.0.0->pylint->hepdata_lib)
  Downloading https://files.pythonhosted.org/packages/52/cf/2ebc7d282f026e21eed4987e42e10964a077c13cfc168b42f3573a7f178c/typed-ast-1.1.0.tar.gz (200kB)
    Complete output from command python setup.py egg_info:
    Error: typed_ast only runs on Python 3.3 and above.

    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-2uRM6T/typed-ast/
You are using pip version 8.1.1, however version 18.0 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Removing intermediate container ba99b71db44e
The command '/bin/sh -c apt-get update -qq     && ln -sf /usr/share/zoneinfo/UTC /etc/localtime     && apt-get -y install         python-dev         python-numpy         python-pip         imagemagick     && localedef -i en_US -f UTF-8 en_US.UTF-8     && rm -rf /packages /var/lib/apt/lists/*     && pip install --no-cache-dir         jupyter         metakernel         zmq         notebook==5.*         hepdata_lib' returned a non-zero code: 1

Support for TMultiGraph and THStack

Support for TMultiGraph could be probably added to the wishlist. Support for TMultiGraph could make importing complex graphs much easier, and avoid painful conversions from TMultiGraph to TGraph in some cases.

Improve table add_image function

Currently, a call to Table.add_image immediately triggers creation of output files. As a consequence, you need to pass the submission output folder to the Table.add_image function.

Probably better:

Change Table.add_image to only store the path of the image in the Table
Add Table.create_image_files function that writes thumbnails, etc for all image paths stored in the Table.
Rename Table.write_yaml to Table.create_yaml_files
Create new Table.create_files function that calls Table.create_image_files and Table.create_yaml_files
Finally, replace call to Table.write_yaml in Submission.create_files with a call to Table.create_files. The output directory can be passed all the way down.

Limit table name length to 64 characters

This limit is enforced by the hepdata servers. Trying to upload a submission with a table with name length > 64 characters will fail. We should prevent users from doing that.

Dump YAML using CSafeDumper (or SafeDumper if not available)

In hepdata_lib/__init__.py, replace:

try:
    from yaml import CLoader as Loader, CDumper as Dumper
except ImportError:
    from yaml import Loader, Dumper

with:

try:
    from yaml import CSafeLoader as Loader, CSafeDumper as Dumper
except ImportError:
    from yaml import SafeLoader as Loader, SafeDumper as Dumper

This will ensure that only standard YAML tags are produced by the Dumper and not arbitrary Python objects. The latter will not be accepted by the hepdata-validator. It's not clear if hepdata_lib uses the Loader anywhere, so it could be removed if not.

Document use of pytest in documentation

With the new development documentation (#96) we only discuss unittest, but we actually run pytest, which has some additional features that would make writing tests easier. We should add this to the documentation.

add possibility to read TGraph from `.C` file

I have some example code that one can build on.

library needs refactoring

The library has reached > 1000 lines and pylint is complaining, so we should move some of the classes into separated files and then enable too-many-lines (d9ed87f).

PyYAML warnings on install

This is a PyYAML problem (not our fault). Fixed in yaml/pyyaml@298e079

So we just have to wait for PyYAML 3.13 to be released with this fix included

Need to migrate services to webhooks using GitHub or OAuth apps

See https://developer.github.com/changes/2018-04-25-github-services-deprecation/ - by end of January 2019.

We're currently making use of docker and travisCI services building of the docker container and tests, respectively.

I've applied for TravisCI, but we need to wait for them to make this possible. For the Docker container we might have to use GitHub actions.

Use docstrings for automated documentation

Then we can profit from http://www.sphinx-doc.org/en/master/ext/autodoc.html

RootFileReader tests fail on lxplus

Using #45, I tried running the tests on lxplus with CMSSW_8_0_20, several of them failed. I fixed most of them, but two remain:

_______________________________________________________________ TestRootFileReader.test_read_hist_2d_asymmetric_errors ________________________________________________________________

self = <test_rootfilereader.TestRootFileReader testMethod=test_read_hist_2d_asymmetric_errors>

    def test_read_hist_2d_asymmetric_errors(self):
        """Test the read_hist_2d function with asymmetric errors."""
        fpath = "testfile.root"

        # Create test histogram
        NX = 17
        NY = 17
        Nfill = 1000

        hist = ROOT.TH2D("test2_asym", "test2d_asym", NX, 0, 1, NY, 0, 1)
        hist.SetBinErrorOption(ROOT.TH1.kPoisson)
        for val in np.random.normal(loc=0.5, scale=0.15, size=(Nfill, 2)):
            hist.Fill(*val)

        backup_hist = hist.Clone("backup")

        # Write to file
        rfile = ROOT.TFile(fpath, "RECREATE")
        hist.SetDirectory(rfile)
        hist.Write("test2d_asym")
        rfile.Close()

        # Read back
        reader = RootFileReader(fpath)
        points = reader.read_hist_2d("test2d_asym")

        # Check keys
        self.assertTrue(set(["x", "y", "z", "x_edges", "y_edges", "dz"]) == set(points.keys()))

        # Check length
        for v in points.values():
            self.assertTrue(len(v) == NX*NY)

        # Look up original bins and compare
        for x, y, z, dz in zip(points["x"], points["y"], points["z"], points["dz"]):
            ibin = backup_hist.Fill(x, y, 0)
            ibinx = ctypes.c_int()
            ibiny = ctypes.c_int()
            ibinz = ctypes.c_int()
            backup_hist.GetBinXYZ(ibin, ibinx, ibiny, ibinz)
            self.assertTrue(float_compare(backup_hist.GetXaxis().GetBinCenter(ibinx.value), x))
            self.assertTrue(float_compare(backup_hist.GetYaxis().GetBinCenter(ibiny.value), y))
            self.assertTrue(float_compare(backup_hist.GetBinContent(ibin), z))
            self.assertTrue(tuple_compare(
                (-backup_hist.GetBinErrorLow(ibinx.value, ibiny.value),
>                backup_hist.GetBinErrorUp(ibinx.value, ibiny.value)), dz))
E           AssertionError: False is not true

tests/test_rootfilereader.py:302: AssertionError

and


self = <test_rootfilereader.TestRootFileReader testMethod=test_read_hist_2d_symmetric_errors>

    def test_read_hist_2d_symmetric_errors(self):
        """Test the read_hist_2d function with symmetric errors."""
        fpath = "testfile.root"

        # Create test histogram
        NX = 100
        NY = 100
        x_values = [0.5 + x for x in range(NX)]
        y_values = [0.5 + x for x in range(NY)]
        z_values = np.random.uniform(-1e3, 1e3, (NX, NY))
        dz_values = np.random.uniform(0, 1e3, (NX, NY))

        hist = ROOT.TH2D("test2d_sym", "test2d_sym", NX, 0, NX, NY, 0, NY)

        for ix in range(1, hist.GetNbinsX()+1):
            for iy in range(1, hist.GetNbinsY()+1):
                ibin = hist.GetBin(ix, iy)
                hist.SetBinContent(ibin, z_values[ix-1][iy-1])
                hist.SetBinError(ibin, dz_values[ix-1][iy-1])

        backup_hist = hist.Clone("backup")
        rfile = ROOT.TFile(fpath, "RECREATE")
        hist.SetDirectory(rfile)
        hist.Write("test2d_sym")
        rfile.Close()

        reader = RootFileReader(fpath)
        points = reader.read_hist_2d("test2d_sym")

        # Check keys
        self.assertTrue(set(["x", "y", "z", "x_edges", "y_edges", "dz"]) == set(points.keys()))

        # Check length
        for v in points.values():
            self.assertTrue(len(v) == NX*NY)

        # Check unordered contents
        self.assertTrue(set(points["x"]) == set(x_values))
        self.assertTrue(set(points["y"]) == set(y_values))

        # Look up original bins and compare
        for x, y, z, dz in zip(points["x"], points["y"], points["z"], points["dz"]):
>           ibin = backup_hist.Fill(x, y, 0)
E           AttributeError: 'PyROOT_NoneType' object has no attribute 'Fill'

tests/test_rootfilereader.py:245: AttributeError

@AndreasAlbert - could you please have a look and fix the tests? I'm not 100% sure what you were trying to achieve with them.

ROOT file content meaning

Currently, the content of the ROOT file looks like:

 KEY: TH1F	Hist1D_y1;1	doi:10.17182/hepdata.79044.v1/t1
 KEY: TH1F	Hist1D_y1_e1;1	doi:10.17182/hepdata.79044.v1/t1
 KEY: TH1F	Hist1D_y1_e2;1	doi:10.17182/hepdata.79044.v1/t1
 KEY: TGraphAsymmErrors	Graph1D_y1;1	doi:10.17182/hepdata.79044.v1/t1

What is the meaning of Hist1D_y1 histogram? I expected it to contain sum of stat. and syst. errors. However it's not the case. I see that (pseudocode):

errors("Hist1D_y1") != errors("Graph1D_y1")
# but at the same time
(errors("Hist1D_y1_e1") ** 2 + errors("Hist1D_y1_e2") ** 2) ** 0.5 == errors("Graph1D_y1")

Not sure if it's the right place, to ask, but I wasn't able to find an answer to my question.

Lint testing code

Document use of TGraph in a notebook

I would suggest to have some simple example that can be run in both myBinder and SWAN interactively. It's probably best to use a plot from an analysis that's already in HEPData. We can also provide examples for different analyses.
@AndreasAlbert - can you suggest one from EXO?

Increase resolution of the image conversion

The resolution of the current 'pdf -> png image convert' is too bad.
The density of the Root pdf image output is set to 300 by default. But the defalut density of the ImageMagick output is 72, which will lead to a big resolution lose in the convert output.
My opinoin is to consider adding a option '-density 300' when doing the convert in

hepdata_lib/hepdata_lib/__init__.py

Line 288 in 0e18ed2

command = "convert -flatten -fuzz 1% -trim +repage {} {}/{}".format(

to get rid of it. Though I don't know if there is a better solution(such as read the image resolution, density before the convert) due to my poor coding skill...

Python version problems

There is a versioning problem somewhere.

When I try to install on lxplus, I run into the error below.
This is the same error that kills our readthedocs build, so it is not lxplus specific.

# Enable python 27
$ scl enable python27 bash
$ python -V
Python 2.7.3

# Make PyROOT work
$ . /afs/cern.ch/sw/lcg/external/gcc/4.9/x86_64-slc6-gcc49-opt/setup.sh
$ . /afs/cern.ch/sw/lcg/app/releases/ROOT/6.06.06/x86_64-slc6-gcc49-opt/root/bin/thisroot.sh

# Install from local repo copy
$ python -m pip install -e hepdata_lib/ --user

Obtaining file:///afs/cern.ch/work/a/aalbert/public/2018-06-07_hepdata_monojet/hepdata_lib
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/afs/cern.ch/work/a/aalbert/public/2018-06-07_hepdata_monojet/hepdata_lib/setup.py", line 43, in <module>
        'Source': 'https://github.com/clelange/hepdata_lib',
      File "/afs/cern.ch/user/a/aalbert/.local/lib/python2.7/site-packages/setuptools/__init__.py", line 128, in setup
        _install_setup_requires(attrs)
      File "/afs/cern.ch/user/a/aalbert/.local/lib/python2.7/site-packages/setuptools/__init__.py", line 123, in _install_setup_requires
        dist.fetch_build_eggs(dist.setup_requires)
      File "/afs/cern.ch/user/a/aalbert/.local/lib/python2.7/site-packages/setuptools/dist.py", line 514, in fetch_build_eggs
        replace_conflicting=True,
      File "/afs/cern.ch/user/a/aalbert/.local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 770, in resolve
        replace_conflicting=replace_conflicting
      File "/afs/cern.ch/user/a/aalbert/.local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 1053, in best_match
        return self.obtain(req, installer)
      File "/afs/cern.ch/user/a/aalbert/.local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 1065, in obtain
        return installer(requirement)
      File "/afs/cern.ch/user/a/aalbert/.local/lib/python2.7/site-packages/setuptools/dist.py", line 581, in fetch_build_egg
        return cmd.easy_install(req)
      File "/afs/cern.ch/user/a/aalbert/.local/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 673, in easy_install
        return self.install_item(spec, dist.location, tmpdir, deps)
      File "/afs/cern.ch/user/a/aalbert/.local/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 699, in install_item
        dists = self.install_eggs(spec, download, tmpdir)
      File "/afs/cern.ch/user/a/aalbert/.local/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 884, in install_eggs
        return self.build_and_install(setup_script, setup_base)
      File "/afs/cern.ch/user/a/aalbert/.local/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 1152, in build_and_install
        self.run_setup(setup_script, setup_base, args)
      File "/afs/cern.ch/user/a/aalbert/.local/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 1140, in run_setup
        raise DistutilsError("Setup script exited with %s" % (v.args[0],))
    distutils.errors.DistutilsError: Setup script exited with Error: typed_ast only runs on Python 3.3 and above.
    
    ----------------------------------------

Add documentation for binned variables

It seems we haven't documented how to use binned variables, unless I missed it...

Uncertainty setter does not allow second argument

With the current setup, there is no way to pass the nominal optional argument to the values setter function. See here

Improve setup documentation

Use-cases to cover:

working on lxplus (e.g. with CMSSW), which needs pip install --user hepdata_lib
working on local computer (recommend virtualenv)

possibility to round numbers
possibility to limit the number of decimal
possibility to limit the number of significant digits
The last could be obtained by having the possibility to quote numbers in EE format and them limit the number of figures in the mantissa.

Cheers,
Claude

Move development status to Beta

We're still in Alpha development status on PyPI. There are still a few features that are useful to have, but I'd propose we move to Beta now with what will be version 0.3.

@AndreasAlbert What do you think?