jaraco / path Goto Github PK

When trying to install :

.
.
.
running install_egg_info
running egg_info
creating path.py.egg-info
writing path.py.egg-info/PKG-INFO
writing top-level names to path.py.egg-info/top_level.txt
writing dependency_links to path.py.egg-info/dependency_links.txt
writing manifest file 'path.py.egg-info/SOURCES.txt'
reading manifest file 'path.py.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'path.py.egg-info/SOURCES.txt'
Copying path.py.egg-info to /tmp/SBo/package-path.py/usr/lib64/python2.7/site-packages/path.py-5.2-py2.7.egg-info
running install_scripts
Usage: grep [OPTION]... PATTERN [FILE]...
Try 'grep --help' for more information.

and broke here.

path.py really lack of a quick introduction in a README.rst file. It would make adoption much easier since pypi and github can display these as HTML.

I can try to write one if you want.

This code can't work :

    def samefile(self): return self.module.samefile(self)

Since the signature of samefile is:

os.path.samefile(f1, f2)

I'd expect something like:

    def samefile(self, othe): return self.module.samefile(self, other)

And a matching test in the unit test file (there is currently None).

I can provide a PL but for such a minor fix I'm pretty sure it's 10 time faster to do it on you side.

def newer(targets, sources, flags):
"""
Compares sources to targets and produces a list of files where the sources are
newer than the targets. Sources & targets can be:
1. Single files
2. Lists of files
3. Directories -- all files that exist in both directories are compared. Directories
are recursed.
4. Lists of files and directories. These are compared positionally.
5. flag indicates whether file name/contents should be the same on source as
target, or to compare timestamp only.
6. Another flag indicates that lack of target file means "not up to date."
"""

(Was using paver which doesn't have any support for this kind of checking; thought I'd float the idea by you to see if it's a fit for path).

So we can chain these methods.

This allow to quickly create a file by doing something like:

foo = (tempdir()  / 'test.txt').write_text('bar')

And have a reference to it.

It would be useful if one could specify which path module to use instead of os.path when creating the path object.

The use case I have for this is when working with Git on Windows: os.path uses backslash as directory separator, but Git uses slash. Although Git understands backslashes as input, it uses forward slashes in its output; parsing and manipulating these with path.py results in a mix of backslashes and forward slashes.

Hey Jason, is there anything preventing a release of path.py? I have some code that relies on path extending unicode (from dd5d16a) and would also like to use Xentac's patch for cascading makedirs.

If there is somethnig in the way, let me know, maybe I can fix it.

I can read this comment in the source code, but can see any use of it. Is that a planned feature or something I don't understand ?

There are some problems people encounter file manipulating path that could deserve some love :

• finding a new name to create / copy a file to a place with a file with a similar name exists (with a suffix, or a counter)
• normalizing names so they become cross platform "safe" such as killing non ascii chars, replacing spaces with underscore and optionaly limiting the name size.

Do you think such code would have it's place in path.py ?

"Storing your password in plaintext is clearly not ideal. If anyone knows of a workaround I would love to hear it."

import keyring
keyring.set_password('pypi', 'john', 'hunter123')
Please enter password for encrypted keyring:
keyring.get_password('pypi', 'john')
u'hunter123'

Some docs:
http://lateral.netmanagers.com.ar/weblog/posts/BB858.html
http://askubuntu.com/questions/89082/storing-passwords-with-python-keyring

Seems there can be an issue creating the warning text when using the walk method in warn mode. I suspect this might only happen on python 2.

  File "/usr/lib64/python2.7/site-packages/flexget/plugins/filter/exists.py", line 44, in on_task_filter
    for p in folder.walk(errors='warn'):
  File "/usr/lib64/python2.7/site-packages/path.py", line 576, in walk
    TreeWalkWarning)
  File "/usr/lib64/python2.7/warnings.py", line 29, in _show_warning
    file.write(formatwarning(message, category, filename, lineno, line))
  File "/usr/lib64/python2.7/warnings.py", line 38, in formatwarning
    s =  "%s:%s: %s: %s\n" % (filename, lineno, category.__name__, message)
UnicodeEncodeError: 'ascii' codec can't encode characters in position 46-60: ordinal not in range(128)

p = path('/a/b/c/d.py')

p[1]
path('b')

p[-1]
path('d.py')

p[-3:]
path('b/c/d.py')

and maybe pop() and maybe iteration over the path parts (for p in path, despite splitall())...

It is very rare you need the entire list in memory, and when you do, an simple list() call turn the generator into the list anyway.

If you don't want to touch listdir and such because it is supposed to be the exact proxy to the os.path.listdir which does return a list, we can make a iterdir(), iterdirs() and iterfiles() method which use os.path.walk and yield to achieve the result.

I can make a patch and a pull request if you don't wish to code it yourself.

In both python2.7 and python 3.4 I get:

======================================================================
ERROR: test_walkdirs_with_unicode_name (__main__.TestUnicodePaths)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test_path.py", line 789, in test_walkdirs_with_unicode_name
    p = path(self.tempdir)
AttributeError: 'TestUnicodePaths' object has no attribute 'tempdir'

I used to love jorendorff's original path module, and just picked up this updated version to use on a project I'm working on. My first quick tests in the console left me a bit puzzled with a few corner cases. Maybe they are by design, but I think intuitively they should work differently.

Path + file

>>> path('/tmp') + 'README.txt'
path(u'/tmpREADME.txt')
>>> path('/tmp/') + 'README.txt'
path(u'/tmp/README.txt')

The trailing slash makes a difference, but both paths point to the same folder, so adding a file name should intuitively yield the same path. Especially considering this:

>>> path('/tmp/').abspath()
path(u'/tmp')

If you call abspath() on a path, the trailing slash gets stripped.

File in path

>>> p = path('./')
>>> p.files()[0]
path(u'./README.txt')
>>> (p + 'README.txt').isfile()
True
>>> 'README.txt' in p
False

It would maybe make sense if a path behaved in this respect like a collection of files and folders instead of a string.

The pathlib module is like a limited path.py. It does have some good ideas such as:

http://docs.python.org/dev/library/pathlib.html#pathlib.PurePath.parents

It would be nice to have it in path.py. I often do p.parent.parent.parent, this would make it shorter/

I have this use case:

new_dir = os.path.join( var_a, var_b, var_c )

where each var evaluates to string and so, the first var is not a path object.

to use path.py i need to do this:

new_dir = path('').joinpath( var_a, var_b, var_c )

or:

new_dir = path.joinpath( path(var_a), var_b, var_c )

or:

new_dir = path( var_a ).joinpath( var_b, var_c )

and i have lists too, which is a pain:

path('').joinpath(*List) ?
path.joinpath(path(p) for p in List) ?
path.joinpath(map(path, List)) ?
sum(map(path, List)) ?!
path(path.os.sep.join(List)) ?! :(

what do you think about pass lists to the constructor? or maybe joinpath to be a staticmethod?

I see we use property() as a fonction here:

namebase = property(
        _get_namebase, None, None,
        """ The same as path.name, but with one file extension stripped off.

For example, path('/home/guido/python.tar.gz').name == 'python.tar.gz',
but path('/home/guido/python.tar.gz').namebase == 'python.tar'
""")

and @Property here:

    @property
    def uncshare(self):
        """
The UNC mount point for this path.
This is empty for paths on local drives.
"""
        unc, r = self.module.splitunc(self)
        return self._next_class(unc)

I first though property() was preferred for backward compatibility. If it's not the cas, why not the use of @Property on _get_namebase ?

It's just curiosity really, it doesn't affect the lib usability.

This context manager allow a shortcut to modify a file in place :

http://www.zopatista.com/python/2013/11/26/inplace-file-rewriting/

We could use it with a syntaxe like :

with path('stuff').inplace(encoding='utf8'') as (orig, res):
    for line in orig:
        res.write(line.upper())

path.path.listdir() can fail with UnicodeDecodeError.

On Linux with Python 2:

$ echo -e 'r\xE9mi' | xargs touch
$ python
Python 2.7.7rc1 (default, May 21 2014, 12:59:42)
[GCC 4.8.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import path
>>> path.path('.').listdir()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/tmp/venv/local/lib/python2.7/site-packages/path.py", line 485, in listdir
    if self._next_class(child).fnmatch(pattern)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe9 in position 1: ordinal not in range(128)

I know you're not the only project out there that gets it wrong, but in my quest for something sensible I did test yours, so I thought I'd let you know. My search goes on!

As far as I can tell, the various calls to glob are not sorted by file name in path.py.

It's a common source of mistake to assume that glob sorts by file name. As path.py makes life easier for many things relative to file handling, I was wondering if you would be open to adding a 'sorted' after each call to glob? I am happy doing the pull request.

path.py could check the errno :

http://docs.python.org/2/library/errno.html

Then check the file permissions and user permissions/group, etc.

This way it could give a move complete error message.

E.G:

Instead of saying : "permission denied". It could say : "permission denied : the file has 755 and belong to user "user1" while this script is executer as user "user2" and don't belong to "user1" group.

This result would be of course different from OS to OS.

I'm trying to be a good boy here and use the .format() method rather than % to format my strings. The problem is that using .format() with a path returns a string and not a path.

Could you provide a .format() method which returns a path ?

>>> from path import path
>>> p = path("test {}")
>>> p
path(u'test {}')
>>> p.format("failed")
u'test failed'

Note tha tthe behavior is the same with %, i don't know if it can be changed:

>>> from path import path
>>> p = path("test %s")
>>> p
path(u'test %s')
>>> p % "failed"
u'test failed'

Returning None is not a feature used in any way.

All methods that don't need to return something could return self, making it possible to do calls like:

PROJECT_DIR = path(file).realpath.parent
TMP_DIR = (PROJECT_DIR.parent / 'tmp').makedirs_p()

Could be nice to be able to to:

with path('stuff') as p:

I have written a post about writing clean-up scripts in Python using path.py. It has got a lot of attention recently and I believe that it will also aid people in learning path.py. If you think this is viable then feel free to add this link to the README.md file. http://freepythontips.wordpress.com/2014/01/23/python-101-writing-a-cleanup-script/

I gave some thoughts to the discussion we had here : #26

The status proposal gave me an idea. We could actually generalize since the path class is becomming big, and maybe it would be beneficial to start separating responsabilities in the lib.

Some actions are clearly not related to a path, but to a file such as :

• calculating md5
• gettings stats
• getting text from file
• getting mimetype
  etc

Also, the fact the class path is supposed to deal with only a path make it difficult to add handy features that are file related without making it a strange melting pot.

This is why I'm suggesting we make a namespace dedicated to file operations. We could simply name it file, and it would be used this way :

>>> p = path('/etc/fstab')
>>> p.file.file_operation()

This would remove from the main namespace a lot of methods, making shell exploration easier. It would also be the occasion to remove duplicates, matching the Zen of Python "there should be only one way to do it". Currently we keep several names to match the os.path API and aliases for compat reasons. While this is supposed to help, with the current size of the lib, it's becoming less and less convenient to instinctly find the proper method without several trials or looking at the source code.

Enventually, it allows us to make the API more modern and adopt a fluid syntaxe :

statvfs() -> p.file.statvfs()
atime and getatime() -> p.file.atime
mtime and getmtime() -> p.file.mtime
ctime and getctime() -> p.file.ctime
utime -> p.file.utime
write_text() -> p.file.write(data)
write_bytes() -> p.file.write(data, 'b')
text() -> p.file.text
?() -> p.file.bytes
lstat() -> p.file.lstat()
read_hash() -> p.file.hash()
read_hexhash() -> p.file.hash().hexdigest()
read_md5() -> p.file.md5()
chunks() -> p.file.chunks()
getsize() -> p.file.size
stat() -> p.file.stat()
inplace() -> p.file.inplace()

file could also be a callable that opens the file and returns a file like object :

for line in p.file():
    # do stuff

The file() callable would accept all the parameters codec.open accept, a bit like what path.open does, but with encoding as well (set to 'utf8' as default like in Python 3).

However, it would not returns a stdlib file like object, but rather a proxy (or a monkeypatched deepcopy for perfs) with all the same attributes except an additional .path returning the original path object used to open the file :

>>> p.file().path is p
True

Last time you shared with me your concern about being backward incompatible, so i'm suggesting this to be added in an experimental branch in which we can play with what can be the future of path.py. Including all rejected features that would have made it backward incompatible, just to see where it leads us.

This branch can be used for future features such as asynchronous file system manipulations, name wrangling, advanced permission checks or anything that is not acceptable to put in the good old path.py.

path.py is now very stable and feature complete, and I don't think it's going to move much. We can safely support this version ad vitam and consider freezing it in the comming years while trying new things on the experimental branch.

Another benefit of the experimental branch is we can drop 2.7 support and focus on Python 3, making the codebase a bit smaller. path.py as is will always be supported for 2.7 and 3 and always be available. We don't really need to touch it.

Currently we got either methods or property to access some information.

I guess that the idea is:

• if it's named after os or shutils methods, it stays a method
• if it's a new method or an alias, make it a property

This make sense for people like me knowing the os API because I used it so much, but I've seen people struggling using path.py because they never know if they should use parenthesis or not and have to test their code countless times before getting it right.

Plus with all the alias, it clutters the namespace and make shell fiddling less easy.

What about making a new version that breaks compatibility and make everything accessible using properties when possible ?

If you use path.py, it's not to use the old os/shutils syntax anyway, so we could make it much easier to use.

Is there a method to know if a path is a file or a directory?
Thanks

http://pypy.python.org/pypi should be http://pypi.python.org/pypi or http://pypi.python.org/pypi/path.py

When iterating over a file list, there can be errors, and for now, path.py deals it this way :

    def walkfiles(self, pattern=None, errors='strict'):
        [...]
        for child in childList:
            try:
                isfile = child.isfile()
                isdir = not isfile and child.isdir()
            except:
                if errors == 'ignore':
                    continue
                elif errors == 'warn':
                    warnings.warn(
                        "Unable to access '%s': %s"
                        % (self, sys.exc_info()[1]),
                        TreeWalkWarning)
                    continue
                else:
                    raise

This does give some options, but it limits the choices between "shut up", "scream" or "crash". You may want to have other behaviors : logging, permission change, user change, deletion, accumulation of the errors in a list, and so on.

We could allow any behavior by just doing:

    def walkfiles(self, pattern=None, errors='strict'):
        [...]
        for child in childList:
            try:
                isfile = child.isfile()
                isdir = not isfile and child.isdir()
            except Exception as e:

                # If errors is a callable, call it by passing the file
                # file name and the exception so it can do something about it
                if callable(errors):
                    res = errors(child, e)
                    # if the callable returns something, yield it (it can
                    # this way do something to the file, 
                    # then yield it anyway if needed, considering the error
                    # solved or just ignore the file by not calling return)
                    if res is not None:
                        yield res

                elif errors == 'ignore':
                    continue
                elif errors == 'warn':
                    warnings.warn(
                        "Unable to access '%s': %s"
                        % (self, sys.exc_info()[1]),
                        TreeWalkWarning)
                    continue
                else:
                    raise

Then you would use it this way :

errors_to_deal_with_later = []
on_errors = lambda f, e: errors_to_deal_with_later.append((f, e))
for p in path('/etc').walkfiles(errors=on_errors):
    do_stuff(p)

for f, e in errors_to_deal_with_later:
    do_stuff_with_all_errors(on_errors)

Of course, the lambda could do anything : sending an email, adding the faulty path in a file, making a remote call, etc. This gives you the power to do something for each faulty files, not just the first one causing troubles and preventing the whole iteration to carry on.

We can apply this logic at several places in the path.py code and it doesn't affect the API, just add flexibility. The Overhead is not really important since we are already in an except clause (the whole stacktrace is been collected) so it's already a very slow logical path anyway.

BTW: nothing related but, what's the use of isdir = not isfile and child.isdir() ?

Provide a cd() method such as:

def cd(self):
    os.chdir(self)

It would be very handy in the shell, where using the context manager is not confortable.

It's not always easy to convert what you want to do ("add 'r' permission to groups) to permission code, espacially when you have to take in consideration the current permissions to do so.

It could be nice to have the hability to do:

path('/stuff').chmod('g+r')

It's more complicated than it seems though.

Uid and Gui works, but are are to manipulate. Since path.py is all about making your life easier, what about making this as easy as knowing the username ? And if nothing is passed, it could even be the current user by default.

The documentation says that path "may be installed using setuptools, distribute, or pip" but only gives an example of easy_install.

Which left me a little upset when I tried to install the package.

airhead:Projects sholden$ pip install path
Downloading/unpacking path
  Could not find any downloads that satisfy the requirement path
No distributions at all found for path
Storing complete log in /Users/sholden/.pip/pip.log

Pip installation should just work. Maybe the notes could mention that since path is a modue rther than a package it should be installed with

pip install path.py

The sphinx documentation is missing all methods without a docstring. The best thing to do would be to add a docstring to every method, but in the meantime I think that it is possible to include methods with no docstrings in the documentation. For most methods this would be enough.

If we implement #58, we can start working on suche features that have more their place in the file namespace.

This code returns a unique filename by incrementing a counter in the name :

import re
import os

def get_unique_path(path):

    while os.path.exists(path):
        base, ext = os.path.splitext(path)
        try:
            base, counter, _ = re.split(r" \((\d+)\)$", base)
        except ValueError:
            counter = 0

        path = "%s (%s)%s" % (base, int(counter) + 1, ext)

    return path

The time has come for me to step back and limit my involvement in this project. In order to avoid it being abandoned, I'd like to identify a successor to take over maintenance. I'm happy to mentor a new maintainer and contribute incidentally, but a volunteer is needed to maintain the project. I would like to transfer the repository and assign ownership to the path.py project in PyPI. Please express your interest here (preferred) or message me privately if appropriate.

Sometime you want to change the group. Sometime you want to change the owner. But not both.

This requires reading the current ownership, to assign back the same as the original one when we don't want to change it since os.chown() force us to pass 2 params.

Hi Jason,

If I was a good person, I'd submit a patch for this, but instead I'm just going to supply an issue. os.path.relpath() takes an optional "start" parameter, but path.path.relpath() does not accept such a parameter. I know you like to keep the APIs in sync, so I thought I'd mention it.

Creating a temporary file or dir is a very common task and manipulating the result with path.py is really nice.

Can we cut down even more on imports and make path proxy calls to tempfile like it does for os and shutil ?

I can make a patch and a pull request if you don't feel like coding it.

The path constructor allows construction from None. It probably should raise a TypeError in this case.

> path.path(None)
path(u'None')

We could extract this piece of code from _hash():

    def _hash(self, hash_name):
        with self.open('rb') as f:
            m = hashlib.new(hash_name)

            # from here 
            while True:
                d = f.read(8192)
                if not d:
                    break

            # to here
                m.update(d)
            return m

And make it a generator. Indeed, while loops are ugly in Python, and it would be nice to have a iterable for of doing this. E.G:

    def chunks(size, *args, **kwargs):
        with open(self, *args, **kwargs) as f:
            while True:
                yield f.read(8192)
                if not d:
                    break

And so any code like _hash would become as simple as :

    def _hash(self, hash_name):
        m = hashlib.new(hash_name)
        for chunk in self.chunk(8192)
            m.update(chunk)
        return m

Again, if you think I'm opening too many issues and not enough PL, let me know, I writting that on the fly while I'm batching the code review for #26

first compile regex for fnmatch then apply the match using the case insensitive re flag.

Honestly.. this may just be a crazy corner case.. I work with satellite imagery and it's nice to be able to match .TIF and .tif and .TiF

Some of my readers build a documentation from your source because they couldn't find any link to it:

http://docs.rebattu.fr/path.py/api.html

Is there an official one ?

If yes, please put a link in the readme. If not, a simple hook from github to read the doc should build it automatically, so people don't have to look at the source to find the API doc.

It's the small things...

path.copy() uses directly shutils.copy(). From python version 3.3 onwards, shutils.copy() returns the new path. It would be nice to always return the new path for all python version. This si also the case for other functions such as copy2, copyfile...

It can also be viewed as compatibility with python < 3.3.

The relevant documentation page is https://docs.python.org/3/library/shutil.html#shutil.copy

Since e708619, Path.open uses io.open which return a stream, which apparentely is not compatible with a File object, which is what is expected for example in numpy.ndarray.tofile.

Maybe the fault is in numpy, but this is still a compatibility problem with previous path.py versions.

I'm quite curious to know the wisdom and ideas of using _next_class, a ClassProperty, to create instances in lots of member functions.

I typically try to adhere to PEP8 standards. So, when I update this script to be used with a project I am working on, I have to go through and change all "path" instances to "Path". I am somewhat OCD, so I have to change it in the comments, as well. I was wondering if there would be any consideration to changing the class name to Path and simply adding the following for backwards compatibility:

path = Path

Also, slightly related to this issue is the fact that the _always_unicode classmethod uses "path" as a local variable name. That is not good practice, either.