Git Product home page Git Product logo

fs.sshfs's People

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar

fs.sshfs's Issues

COPYING fle is not in package

Hi
I added manually the COPYING file to the recipe, because it did not made it into the pypi package.

You may want to add it into a MANIFEST

Release 0.6.2 available?

Martin,

Is it possible to release v.0.6.2?
The current version 0.6.1 on PYPI doesn't include the commit 23a3f9f (Change remote platfrom management).

As a result, it is not possible to use it on Windows without because of missing modules pwd and grp.

Best regards,
Gennady

fs.sshfs 2.0.20 cannot walk a nonunicode filename

fs.sshfs cannot walk the following nonunicode file:

>>> os.listdir('.')
['Ma\udcf1ana']

The special character is a Spanish n~, in an unknown, 8 bit encoding.

The error looks like:

 File "/home/dstromberg/src/pyfilesystem-tests/lib/python3.6/site-packages/paramiko/sftp_client.py", line 222, in listdir_attr
   filename = msg.get_text()
 File "/home/dstromberg/src/pyfilesystem-tests/lib/python3.6/site-packages/paramiko/message.py", line 178, in get_text
   return u(self.get_string())
 File "/home/dstromberg/src/pyfilesystem-tests/lib/python3.6/site-packages/paramiko/py3compat.py", line 143, in u
   return s.decode(encoding)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xf1 in position 2: invalid continuation byte

Feature Request: Support atomic move and rename

Assertion as follows fails. Expect a pass, it is certain that SFTP is powerful enough to support renaming.

assert "supports_rename" in remote_fs.getmeta() and remote_fs.getmeta()["supports_rename"]

And it costs when moving big files around, which is merely copying the whole file without any optimization. I thought it was paramiko not supporting rename initially but after checking out the its documentation I found this functionality have already been implemented quite well. Two methods are provided: rename and posix_rename, both are atomic I suppose.

Thank you very much.

Cannot pass look_for_keys via FS URL

While it is possible to pass look_for_keys to the underlying paramiko Client as follows

from fs.sshfs import SSHFS

my_fs = SSHFS(..., look_for_keys=False)

I am unable to do so when using an FS URL:

import fs

my_fs = fs.open_fs("ssh://[user[:password]@]host[:port]/[directory]?look-for-keys=False")
my_fs = fs.open_fs("ssh://[user[:password]@]host[:port]/[directory]?look_for_keys=False")

Looking at SSHOpener, this could be remedied by passing look_for_keys in when constructing SSHFS, much like e.g. pkey is.

Feature Request: Expose missing host key policy as a parameter

Hello,

We are trying to implement validation of host keys against a whitelist of known hosts keys, and would be interested in the ability to change the missing host key policy to something other than AutoAddPolicy. Would it be possible to add such an option?

working with files..

This would be rather an incompetence on my part than an issue reporting, but I spent last 6 hours and still cannot figure out how to move(copy) files in and out of my SFTP directory.

I did manage to get into SFTP directory,

my_fs = SSHFS("OO.22.167.217", user="1234", passwd="111", timeout=10, port=25761, keepalive=10, compress=False)
print(my_fs)
a = my_fs.listdir('/qwe')
print(a)

['111.txt', 'demo3.py', 'handout.pdf']

This indeed prints out items in the root! ..but I'm stuck here for hours.
Would you be kind enough to post a snippet of scripts, moving c:\111.txt to my_fs?

(..found a negligible typo: paswd should be passwd.)

Error: UnsupportedProtocol: protocol 'ssh' is not supported, when trying from google-colab

Hi,

I tried to use fs.sshfs with a google-colab notebook, but I get the following error
UnsupportedProtocol: protocol 'ssh' is not supported

Installing the package was successful (console output pasted below).

To reproduce, type the following commands (except for replacing the address with a proper host)

!pip install fs.sshfs
import fs
my_fs = fs.open_fs("ssh://[user[:password]@]host[:port]/[directory]")

Thank you for your help

Here is a dump of the pip install console output


Collecting fs.sshfs
  Downloading fs.sshfs-0.8.0-py2.py3-none-any.whl
Requirement already satisfied: configparser~=3.2; python_version < "3" in /usr/local/lib/python2.7/dist-packages (from fs.sshfs)
Requirement already satisfied: six~=1.10 in /usr/local/lib/python2.7/dist-packages (from fs.sshfs)
Collecting fs~=2.0.7 (from fs.sshfs)
  Downloading fs-2.0.18-py2.py3-none-any.whl (106kB)
    100% |β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 112kB 4.5MB/s 
Collecting paramiko~=2.0 (from fs.sshfs)
  Downloading paramiko-2.4.0-py2.py3-none-any.whl (192kB)
    100% |β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 194kB 3.3MB/s 
Requirement already satisfied: pytz in /usr/local/lib/python2.7/dist-packages (from fs~=2.0.7->fs.sshfs)
Requirement already satisfied: enum34~=1.1.6; python_version < "3.4" in /usr/local/lib/python2.7/dist-packages (from fs~=2.0.7->fs.sshfs)
Collecting appdirs~=1.4.3 (from fs~=2.0.7->fs.sshfs)
  Downloading appdirs-1.4.3-py2.py3-none-any.whl
Requirement already satisfied: setuptools in /usr/lib/python2.7/dist-packages (from fs~=2.0.7->fs.sshfs)
Collecting cryptography>=1.5 (from paramiko~=2.0->fs.sshfs)
  Downloading cryptography-2.1.4-cp27-cp27mu-manylinux1_x86_64.whl (2.2MB)
    100% |β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 2.2MB 478kB/s 
Collecting pynacl>=1.0.1 (from paramiko~=2.0->fs.sshfs)
  Downloading PyNaCl-1.2.1-cp27-cp27mu-manylinux1_x86_64.whl (696kB)
    100% |β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 706kB 1.6MB/s 
Requirement already satisfied: pyasn1>=0.1.7 in /usr/local/lib/python2.7/dist-packages (from paramiko~=2.0->fs.sshfs)
Collecting bcrypt>=3.1.3 (from paramiko~=2.0->fs.sshfs)
  Downloading bcrypt-3.1.4-cp27-cp27mu-manylinux1_x86_64.whl (57kB)
    100% |β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 61kB 9.7MB/s 
Collecting cffi>=1.7; platform_python_implementation != "PyPy" (from cryptography>=1.5->paramiko~=2.0->fs.sshfs)
  Downloading cffi-1.11.5-cp27-cp27mu-manylinux1_x86_64.whl (407kB)
    100% |β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 409kB 2.9MB/s 
Requirement already satisfied: idna>=2.1 in /usr/local/lib/python2.7/dist-packages (from cryptography>=1.5->paramiko~=2.0->fs.sshfs)
Collecting asn1crypto>=0.21.0 (from cryptography>=1.5->paramiko~=2.0->fs.sshfs)
  Downloading asn1crypto-0.24.0-py2.py3-none-any.whl (101kB)
    100% |β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 102kB 7.4MB/s 
Collecting ipaddress; python_version < "3" (from cryptography>=1.5->paramiko~=2.0->fs.sshfs)
  Downloading ipaddress-1.0.19.tar.gz
Collecting pycparser (from cffi>=1.7; platform_python_implementation != "PyPy"->cryptography>=1.5->paramiko~=2.0->fs.sshfs)
  Downloading pycparser-2.18.tar.gz (245kB)
    100% |β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 256kB 3.9MB/s 
Building wheels for collected packages: ipaddress, pycparser
  Running setup.py bdist_wheel for ipaddress ... -οΏ½ οΏ½done
  Stored in directory: /content/.cache/pip/wheels/d7/6b/69/666188e8101897abb2e115d408d139a372bdf6bfa7abb5aef5
  Running setup.py bdist_wheel for pycparser ... -οΏ½ οΏ½\οΏ½ οΏ½done
  Stored in directory: /content/.cache/pip/wheels/95/14/9a/5e7b9024459d2a6600aaa64e0ba485325aff7a9ac7489db1b6
Successfully built ipaddress pycparser
Installing collected packages: appdirs, fs, pycparser, cffi, asn1crypto, ipaddress, cryptography, pynacl, bcrypt, paramiko, fs.sshfs
Successfully installed appdirs-1.4.3 asn1crypto-0.24.0 bcrypt-3.1.4 cffi-1.11.5 cryptography-2.1.4 fs-2.0.18 fs.sshfs-0.8.0 ipaddress-1.0.19 paramiko-2.4.0 pycparser-2.18 pynacl-1.2.1

listdir hangs indefinitely

Just trying to use fs.sshfs to access an sftp server but I'm finding that listdir is hanging indefinitely.

I've tracked it down to this line:

uname_sys = self._exec_command("uname -s")

...which calls the paramiko SSHClient.exec_command without a timeout argument:

fs.sshfs/fs/sshfs/sshfs.py

Lines 294 to 295 in 33f981b

_, out, err = self._client.exec_command(cmd)
return out.read().strip() if not err.read().strip() else None

Unfortunately, for whatever reason this seems to never return for me.

setuptools parameter extras_requires

if the setup_requires parameter could be moved to extras_requires, we don't need all these testing libraries
for a conda-forge package build.

cleanup not needed imports

test_create and test_sshfs have an import docker which is not needed.
some other libs also imported and not used.

SFTP file transfers extremely slow

Hi,

I am extremely grateful for this extension to PyFilesystem, thank you.

I have found that SFTP uploads are extremely slow, by orders of magnitude when compared to the same files transferred with FileZilla.

Some initial research seems to indicate that setting the set_pipelined() mode before a transfer will help, a lot. But I am having some trouble figuring out where I should attemp to set this.

I appreciate any help or thoughts you may have,

Regards,

Geoff

Improving the throughput of scandir

The current implementation of fs.sshfs only overrides listdir and not scandir, and so inherits the FS base class' default implementation of scandir. That implementation does a getinfo on every item in the directory, which ends up with an lstat via sshfs. That incurs a network round-trip for each file in the directory. That O(n) penalty can really add up, especially for operations that need the size or modification time details, such as copy or mirror.

However, the sftp protocol allows us to stat as part of the list operation all in one go, so sshfs could override scandir to take advantage of that optimization and minimize round-trips.

Locally, I have monkey-patched sshfs to use this optimization (see code below). I wonder if we could incorporate a version of this patch into fs.sshfs directly?

def _sshfs_scandir(self, path, namespaces=None, page=None):
    _namespaces = namespaces or ()
    _start, _end = (page or (0, None))
    _path = self.validatepath(path)
    _type = self.gettype(_path)
    if _type is not fs.enums.ResourceType.directory:
        raise fs.errors.DirectoryExpected(path)
    with fs.sshfs.error_tools.convert_sshfs_errors('scandir', path):
        for i, _stat_result in enumerate(self._sftp.listdir_attr(_path)):
            if i < _start:
                continue
            if _end is not None and i >= _end:
                break
            _name = _stat_result.filename
            yield self._make_info(_name, _stat_result, _namespaces)

Note that here I still have a gettype call in the critical path. I believe we could remove that as well, with some careful exception handling in the with block, to make sure we get DirectoryExepected in case of a bad path.

Don't read past end of file

I'm dealing with an SFTP server that errors out if you send a read() for more bytes than the file has. Not sure if this is a common thing or my server is just not following specs (wouldn't surprise me).

I solved this by manually tracking the filesize and bytes read, but I'm wondering if there'd be any cons to just having this in the package?

Two options would be:

  • Do what s3fs does and download into a temp file so that users won't encounter discrepancies between real filesystems and sftp
  • Emulate reading beyond the last byte without actually sending requests

Problems installing on Python2.7

Hi @althonos

I'm struggling to install fs.sshfs on Python2.7. I get the following error when importing...

>>> import fs.sshfs
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named sshfs

Although bizarrely I see ssh and sftp in the list of openers.

Everything appears to work fine on Python3

Suspect this may be related to #12

I've tried upgrading setuptools and distribute, but no luck.

Let me know if I can help with the debugging.

Mount remote filesystem to local

Hi,

I am trying to mount a remote filesystem to my local server. I don't see any example online.
I tried few things and not getting the correct result.

can anyone please share an example with me.

Highly appreciate any info!!!

ModuleNotFoundError: No module named 'fs.sshfs' - Setup.py install not working for some reason

This is certainly me misunderstanding something, but I haven't been able to figure it out.

Error: ModuleNotFoundError: No module named 'fs.sshfs'

Replication steps:

  1. Setup a new venv (Python 3.8.5)
  2. Run python setup.py install (With setup.py below)
  3. Open up your interpreter (or anything) and run from fs.sshfs import SSHFS

The part that is really throwing me for a loop is this

  1. After doing those steps run pip install fs.sshfs , pip will let you know you have all the requirements (But if you test with the python interpreter you will see this still doesn't' work)
  2. Install a new venv
  3. Run pip install fs.sshfs
  4. Now running from fs.sshfs import SSHFS works perfectly fine!

Very confusing for me. I'm trying to extend https://github.com/hotgluexyz/target-csv , but I"m not having very much luck. I'm going to dump fs.sshfs right now for using paramiko directly unfortunately due to this issue :/

Full error


Python 3.8.5 (default, May 27 2021, 13:30:53)
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from fs.sshfs import SSHFS
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'fs.sshfs'

setup.py

#!/usr/bin/env python

from setuptools import setup

setup(name='target-csv',
      version='0.3.3',
      description='Singer.io target for writing CSV files',
      author='Stitch',
      url='https://singer.io',
      classifiers=['Programming Language :: Python :: 3 :: Only'],
      py_modules=['target_csv'],
      install_requires=[
          'fs.sshfs==1.0.0',
          'jsonschema==2.6.0',
          'singer-python>=5.1.0,<=5.3.1',
      ],
      entry_points='''
          [console_scripts]
          target-csv=target_csv:main
      ''',
)

pypi Version 0.10.1 and pyFilesystem2 version

Hi

I need some understanding what the ~Operator in setup.cfg means and why the table of used modules is not showing the needed versions.

The pypi version lists 2.4.0 of PyFilesstem2 :
https://pypi.org/project/fs.sshfs/

it installs
fs 2.3.1
If 2.4.0 works, can you adopt the value in setup.cfg?

Also some of the other package have updates. pip install fs.sshfs installs on my system:

six                                1.12.0 
paramiko                      2.4.2
cached-property           1.5.1 

It looks like
https://pypi.org/project/fs.sshfs/
does not list the PyFileSystem2 version needed but always the newest available?

regards
Reimar

Paramiko errors, arbitrary configuration

I have a setup where I connect to a server via user/password. In my environment, connection here fails with an error like "no existing session". This is a paramiko error.

In paramiko, I can get around this by passing look_for_keys=False and allow_agent=False.
Currently, paramiko is initialized with

look_for_keys=True if (pkey and keyfile) is None else False,

which makes it tricky to disable look_for_keys. Also, it seems it's not possible to set additional arguments in the call.

I was thinking that maybe the final options that are passed to paramiko.connect could be made updateable by the SSHFS user. This could be done by creating an argument dictionary, and updating it from **kwargs right before passing it to paramiko.connect.

If that seems like an acceptable solution, I can create created a PR.

Setup AppVeyor CI to test on Windows

Since it was confirmed there is a more than zero person that are using this library on a Windows machine, it would be interesting to have some tests running in a Windows environment.

The trouble is, our tests run on Docker, and I've no idea how difficult it will be to setup Docker inside AppVeyor.

Feel free to add below any resource that may help

`create` option not honored

Hi,

If I pass the create option to an SFTP endpoint, it fails with fs.errors.ResourceNotFound. The host exists, and I can makedirs after connecting to the root.

For example:

import fs
sftp = fs.open_fs('sftp://example.com/path/to/save', create=True)

raises fs.errors.ResourceNotFound

import fs
root = fs.open_fs('sftp://example.com')
sftp = root.makedirs('/path/to/save')

Works just fine.

Thoughts?

Geoff

Unsupported SSH connection when using PyInstaller

I am connecting via SSH using fs.sshfs like this,

my_fs = fs.open_fs(u'ssh://myuser:[email protected]:22/share/directory/')

It is working fine when I run the script using PyCharm. However, when I create my EXE using PyInstaller and I run the executable I get this error:

fs.opener.errors.UnsupportedProtocol: protocol 'ssh' is not supported

I think that the PyInstaller doesn't include fs.sshfs and I have tried also to run PyInstaller with the following option:

--hidden-import=fs.sshfs

The error persists.

v2.1.0 of fs was released

Hi
there is a build rule fs ~=2.1.0 and v2.1.0 of fs was released

Are there known issues?

regards
Reimar

Find the latest file on a remote Server

Here is my scenario -

i have files on a remote server in a certain directory and I want to loop thru this remote server and directory and identify the latest files. I want to pass a file name as a wildcard and for that specified file name, it has to loop the remote server and directory and return the latest file.

The documentation -- https://pyfilesystem2.readthedocs.io/en/latest/
doesn't list this specific scenario. and i am looking for some example on how to do it.

any information is highly appreciated !!!!

Support for Paramiko 3.x

This package currently pins paramiko ~= 2.0. Where possible could the package test with paramiko 3.x and pin paramiko >= 2.0 instead?

"SSHException: Channel" closed on server with no shell access

Calling listdir on real sftp server (without shell access) produces attached traceback. It seems that this server forbids executing external commands (via self._exec_command).
It works with following patch:

index 777cfc0..0f29093 100644
--- a/fs/sshfs/sshfs.py
+++ b/fs/sshfs/sshfs.py
@@ -308,8 +308,11 @@ class SSHFS(FS):
         Returns:
             str: the platform of the remote server, as in `sys.platform`.
         """
-        uname_sys = self._exec_command("uname -s")
-        sysinfo = self._exec_command("sysinfo")
+        try:
+            uname_sys = self._exec_command("uname -s")
+            sysinfo = self._exec_command("sysinfo")
+        except paramiko.ssh_exception.SSHException:
+            return "unknown"
         if uname_sys is not None:
             if uname_sys == b"FreeBSD":
                 return "freebsd"

Attached traceback:

    149         _path = self.validatepath(path)
    150 
--> 151         _type = self.gettype(_path)
    152         if _type is not ResourceType.directory:
    153             raise errors.DirectoryExpected(path)

~/.virtualenvs/fs-tests/lib/python3.6/site-packages/fs/base.py in gettype(self, path)
    851 
    852         """
--> 853         resource_type = self.getdetails(path).type
    854         return resource_type
    855 

~/.virtualenvs/fs-tests/lib/python3.6/site-packages/fs/base.py in getdetails(self, path)
   1540 
   1541         """
-> 1542         return self.getinfo(path, namespaces=["details"])
   1543 
   1544     def check(self):

~/.virtualenvs/fs-tests/lib/python3.6/site-packages/fs/sshfs/sshfs.py in getinfo(self, path, namespaces)
    136         with convert_sshfs_errors('getinfo', path):
    137             _stat = self._sftp.lstat(_path)
--> 138             return self._make_info(basename(_path), _stat, namespaces)
    139 
    140     def geturl(self, path, purpose='download'):  # noqa: D102

~/.virtualenvs/fs-tests/lib/python3.6/site-packages/fs/sshfs/sshfs.py in _make_info(self, name, stat_result, namespaces)
    357         }
    358         if 'details' in namespaces:
--> 359             info['details'] = self._make_details_from_stat(stat_result)
    360         if 'stat' in namespaces:
    361             info['stat'] = {

~/.virtualenvs/fs-tests/lib/python3.6/site-packages/fs/sshfs/sshfs.py in _make_details_from_stat(self, stat_result)
    379 
    380         details['created'] = getattr(stat_result, 'st_birthtime', None)
--> 381         ctime_key = 'created' if self.platform == "win32" else 'metadata_changed'
    382         details[ctime_key] = getattr(stat_result, 'st_ctime', None)
    383         return details

~/.virtualenvs/fs-tests/lib/python3.6/site-packages/property_cached/__init__.py in __get__(self, obj, cls)
     94             return self
     95         with self.lock:
---> 96             return super(threaded_cached_property, self).__get__(obj, cls)
     97 
     98     def __set__(self, obj, value):

~/.virtualenvs/fs-tests/lib/python3.6/site-packages/property_cached/__init__.py in __get__(self, obj, cls)
     54         value = self.cache.get(obj, self._sentinel)
     55         if value is self._sentinel:
---> 56             value = self.cache[obj] = self.func(obj)
     57 
     58         return value

~/.virtualenvs/fs-tests/lib/python3.6/site-packages/fs/sshfs/sshfs.py in platform(self)
    309             str: the platform of the remote server, as in `sys.platform`.
    310         """
--> 311         uname_sys = self._exec_command("uname -s")
    312         sysinfo = self._exec_command("sysinfo")
    313         if uname_sys is not None:

~/.virtualenvs/fs-tests/lib/python3.6/site-packages/fs/sshfs/sshfs.py in _exec_command(self, cmd)
    344             None: if the error pipe of the command was not empty
    345         """
--> 346         _, out, err = self._client.exec_command(cmd, timeout=self._timeout)
    347         return out.read().strip() if not err.read().strip() else None
    348 

~/.virtualenvs/fs-tests/lib/python3.6/site-packages/paramiko/client.py in exec_command(self, command, bufsize, timeout, get_pty, environment)
    512         if environment:
    513             chan.update_environment(environment)
--> 514         chan.exec_command(command)
    515         stdin = chan.makefile_stdin("wb", bufsize)
    516         stdout = chan.makefile("r", bufsize)

~/.virtualenvs/fs-tests/lib/python3.6/site-packages/paramiko/channel.py in _check(self, *args, **kwds)
     70         ):
     71             raise SSHException("Channel is not open")
---> 72         return func(self, *args, **kwds)
     73 
     74     return _check

~/.virtualenvs/fs-tests/lib/python3.6/site-packages/paramiko/channel.py in exec_command(self, command)
    255         self._event_pending()
    256         self.transport._send_user_message(m)
--> 257         self._wait_for_event()
    258 
    259     @open_only

~/.virtualenvs/fs-tests/lib/python3.6/site-packages/paramiko/channel.py in _wait_for_event(self)
   1224         if e is None:
   1225             e = SSHException("Channel closed.")
-> 1226         raise e
   1227 
   1228     def _set_closed(self):

SSHException: Channel closed.

Unable to create filesystem if directory is link

If I try to open
"sftp:://@/home/"
it works fine;
but
"sftp:://@/bin/"
is not working because 'bin' is a link to '/usr/bin/'

In old implementation of fs ( pyrhonV3.4 ) iboth cases works fine
-Valeri

Inconsistent namespace packaging approach

Observation
This package uses the native namespace packaging approach (no init file in fs folder). Also, this package depends on pyfilesystem2 and shares this namespace folder 'fs' with pyfilesystem2.

However, pyfilesystem2 uses the pkg_resources-style namespace approach to declare its namespace folder 'fs' (see fs/init.py).

As a result, this package is inconsistent with the recommendation in pkg_resources-style namespace approach -

If you are creating a new distribution within an existing namespace package that uses this method then it’s recommended to continue using this as the different methods are not cross-compatible and it’s not advisable to try to migrate an existing package.

Problem
In use cases where these two packages are downloaded in seperate directories and both of them are added to the python systempath, the package import does not work due to the aforementioned inconsistency. This is exactly the problem if one uses bazel to manage these external dependencies in a project.

Suggestion
Please use the consistent namespace packaging convention as suggested in the link above.

fs.sshfs not threadsafe?

When I try to make multiple calls to fs.open in a ThreadPoolExecutor my program hangs :(

Is fs.sshfs expected to be threadsafe? Is this a known issue?

platform any but not windows currently?

I am not sure if it is possible to run this from windows. At least momently appveyor cries about pwd and grp. I was not able to verify this before I was able to use the conda-forge CIs.

If that principle is possiple then may be moving the imports
after

if self.platform in Platform.Unix

solves it?

or if that is only optional needed info['access'] may be excluded for windows

 if 'access' in namespaces:
            info['access'] = self._make_access_from_stat(stat_result)

but the imports have to be moved too, or enclosed by an exception.

https://anaconda.org/conda-forge/paramiko seems to be a valid windows package.

Error when closing python terminal

>>> import fs
>>> sfs = fs.open_fs('ssh://user@host/datafiles')
Exception ignored in: <bound method FS.__del__ of ClosingSubFS(<fs.sshfs.sshfs.SSHFS object at 0x7ff99b0e5fd0>, '/datafiles')>
Traceback (most recent call last):
  File "venv/lib64/python3.6/site-packages/fs/base.py", line 117, in __del__
  File "venv/lib64/python3.6/site-packages/fs/subfs.py", line 63, in close
  File "venv/lib64/python3.6/site-packages/fs/sshfs/sshfs.py", line 130, in close
TypeError: super() argument 1 must be type, not None
Exception ignored in: <bound method FS.__del__ of <fs.sshfs.sshfs.SSHFS object at 0x7ff99b0e5fd0>>
Traceback (most recent call last):
  File "venv/lib64/python3.6/site-packages/fs/base.py", line 117, in __del__
  File "venv/lib64/python3.6/site-packages/fs/sshfs/sshfs.py", line 130, in close
TypeError: super() argument 1 must be type, not None

Entrypoints

I just updated fs and I think it blew away fs.sshfs??

AFAICS fs.sshfs creates a sshfs folder in the fs namespace which is never going to work?

I'm confused though because it seems entrypoint support was added over in PyFilesystem/pyfilesystem2#56?

...and there are entrypoints defined in setup.cfg

fs.sshfs/setup.cfg

Lines 56 to 59 in d79362c

[options.entry_points]
fs.opener =
ssh = fs.opener.sshfs:SSHOpener
sftp = fs.opener.sshfs:SSHOpener

It seems to me that with entrypoints there's no need for sshfs to be embedded within fs?

Timeout handling on exotic SFTP servers

Hello,

Recently we've started working with Syncplify.me server. Apparently, it doesn't send an EOF which makes commands like listdir raise a timeout. This seems related to #15, but since it happens every time, we needed to work it around.
The problem has also been reported directly in Paramiko's tracker (see paramiko/paramiko#109) and apparently it also applies to Windows 10 OpenSSH implementation.
Currently I made a wrapper which forces stdout and stderr buffers to get closed by the client after reaching a certain timeout (based on a workaround posted here: paramiko/paramiko#109 (comment)). Here's my approach: https://github.com/pylogy/fs.sshfs/tree/fix-exec-timeout (it feels a bit hackish, so I haven't posted a PR yet). I'm also considering adding this directly to Paramiko after figuring out where exactly it should be handled.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    πŸ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. πŸ“ŠπŸ“ˆπŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❀️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.