neither package_data or data_files get installed about sampleproject HOT 25 CLOSED

Neither package_data or data_files should be part of any "regular" project. They are ridiculously hard to get right. data_files behaves inconsistently across tools. I consider those anti-patterns in packaging [1]. Having a correct MANIFEST.in and include_package_data=True is most always the right choice.

[1] If you have the patience watch this

from sampleproject.

OT: Setuptools is such a mess, and the defaults mean I have to look up how to do this every year or so.

I'm glad work is being done on it, but having delved inside, I think we'd be better off chucking it out and using something else at this point.

from sampleproject.

I'm gonna go ahead and close this -- packagers should use MANIFEST.in and include_package_data=True to correctly choose which files get included.

You can look at the pip's setup.py and MANIFEST.in for a more concrete example.

If anyone has similar/other issues related to which files get included, please check if there's a related issue filed on setuptools' issue tracker, otherwise please file a new issue there. :)

Thanks everyone for chiming in here!

from sampleproject.

@ionelmc MANIFEST.in + include_package_data=True as best practice, So true! I spent hours debugging what's wrong with package_data and finally solved it by adding a MANIFEST.

from sampleproject.

@fduxiao it's funny, as I only package something about once a year, I always, without fail - forget how to add non .py files to my python package, python packaging is almost always painful in some way.

from sampleproject.

With Python 3 I've found out that the MANIFEST.in is not honored anymore in the wheel, but it is in the tar.gz! Maybe this was always the case but I just noticed now.

Anyway, here is the fix: to force the wheel to honor the MANIFEST.in, you need to specify both include_package_data and packages.

If your Python module is inside a folder "mymodule", here is how the setup looks like:

setup( ...
    include_package_data=True,  # use MANIFEST.in to include config file
    packages=['mymodule'],  # to force the wheel to use the MANIFEST.in
)

If all your python scripts are at the root (and not inside a subfolder), use:

setup( ...
    include_package_data=True,  # use MANIFEST.in to include config file
    packages=['.'],  # to force the wheel to use the MANIFEST.in
)

from sampleproject.

When will this be fixed. You are justing wasting others' time, which is a kind of murder!

from sampleproject.

Using include_package_data=True fixed the issue for me. Thank's @ionelmc !

from sampleproject.

@ionelmc what is the way to include some data with a project, that will get installed into the virtualenv (and allow installation using pip) ?

from sampleproject.

@stuaxo MANIFEST.in + include_package_data=True is my default approach. I'm not really sure what you're asking me, describe your situation more.

from sampleproject.

@stuaxo In my situation, I use cython to accelerate my web application, which requires non python templates and will be used further by other projects, and I found the only way that works is putting an empty init.py into the templates dir and tell cython not to handle it (they are found automatically). It cost me more then 3 hours to find it.
I think it will be great if there's a simple way to add non python files.

from sampleproject.

Hello all,

Yesterday for the first time I faced a need to create a Python package. I didn't think it would be such a pain if it's supposed to have data files! I've skimmed through the documentation, but it's so diverse and inaccurate that I can't wrap my head around it.

I set up a simple directory structure:

.
├── config.yml
├── MANIFEST.in
├── sample
│   ├── hello.py
│   └── __init__.py
└── setup.py

The contents of MANIFEST.in is:

include config.yml

The setup.py is:

from setuptools import setup

setup(
    name='sample',
    version='1.0',
    packages=['sample'],
    include_package_data=True,
)

When I do pip install . or python setup.py install, a directory is created under .../lib/python3.6/site-packages, but there's no config.yml that I hoped would be included there, hence I cannot access it from code.

What am I doing wrong? Any help is appreciated. Thank you.

PS: Have watched @ionelmc 's presentation and tried the following MANIFEST.in: graft <package_name> <data_dir> but to no avail.

from sampleproject.

@andreypanin config.yml is not inside the package named sample thus it doesn't qualify as package data, thus include_package_data=True won't have any effect. Move it inside the sample directory and use graft sample in the manifest (just that).

from sampleproject.

can you provide details with a paste of your output

from sampleproject.

ok, I see. it works when installing (using pip or easy_install) from a built sdist, but not using "python setup.py install" from a checkout. I wasn't aware. will look into it.

from sampleproject.

I think this is related to @ericfrederich issue?

Test 1) If I do python setup.py install I get an egg containing both package_data.dat and data_file, similarly if I set zip_safe to False, I have those files both in the sample directory.

Test 2) I create a wheel using pip wheel --wheel-dir=. sampleproject and when I do pip install sample-1.2.0-py2.py3-none-any.whlthe my_data\data_file works, and sample_data.dat

Test 3) But when I create a sdist (python setup.py sdist), the zip always contains package_data.dat (even if I comment out the package_data section), and never contains data_file.

error: can't copy 'data\data_file': doesn't exist or not a regular file

Test 4) Now if I remove package_data.dat from MANIFEST.in, it only includes the package_data as I would have expected...I guess when creating an sdist, these files are ALWAYS added even after python 2.6 due to MANIFEST.in

I guess this is all related to the same problem that's been plaguing sdist vs bdist forever http://blog.codekills.net/2011/07/15/lies,-more-lies-and-python-packaging-documentation-on--package_data-/ Although I don't know how much of that is still accurate or translates to setuptools, but I DO know the same problems appear to be happening...It's just this sample doesn't appear to work for both... YET?

Python 2.7.9 Windows 64-bit and Linux 64-Bit
Setuptools 15.2 and 16.0 (And I even tried 1.0 for a few tests, nothing really changed)

from sampleproject.

yea @ionelmc , I think the sample project and the distribution tutorial needs some rehaul in this department. just recommend the easiest thing (like in your video), and maybe have an advanced section to cover all the caveats etc... that are so hard to remember.

from sampleproject.

from sampleproject.

@fduxiao is the workaround of MANIFEST.in (docs here) and setup(..., include_package_data=True) not working for you?

from sampleproject.

@nicktimko See my fork here. I'm not sure whether I write the MANIFEST.in correctly or not.

from sampleproject.

Saying graft data would be slightly simpler than recursive-include data *, but does it not work now?

StackOverflow might be a better forum for such a question than here.

from sampleproject.

@nicktimko It never works if my package has only compile modules (You see I just cythonized the modules and the data just fail to be installed). The conclusion I can get from stackoverflow is that it's only a dirty lie. I'm not asking a question. I'm saying one could not put package data for a package of compile modules. It may be caused by the fact that setup never thinks compiled modules as packages (they are under some folder) and I cannot specify the folder is actually a package, so can anyone find the reason and fix it?

from sampleproject.

@fduxiao I had to do the thing with a fake __init__.py but it feels like a hack, the kind that might just stop working some day.

from sampleproject.

Is this problem solved? I can use package_data to include my image files successfully into my package.

Python 3.7.3 64bit
setuptools - 40.8.0

from sampleproject.

To be clear, using a different build tool like flit can help avoid these setuptools specific issues.

from sampleproject.