Git Product home page Git Product logo

byteskeptical / sftpretty Goto Github PK

View Code? Open in Web Editor NEW
28.0 2.0 8.0 3.14 MB

Provides multi-threaded routines and high level protocol abstractions for a pretty quick & simple file transfer experience. Drop in replacement for pysftp.

Home Page: http://docs.sftpretty.com/

License: BSD 3-Clause "New" or "Revised" License

Python 100.00%
paramiko file-transfer-protocol file-transfer pysftp scp sftp sftp-client sftp-download sftp-server sftp-upload

sftpretty's Introduction

sftpretty

A pretty quick and simple interface to paramiko SFTP. Provides multi-threaded routines with progress notifications for reliable, asynchronous transfers. A Python3 optimized fork of pysftp with additional features & improvements.

  • Built-in retry decorator
  • Hash function for integrity checking
  • Improved local & remote directory mapping
  • Improved logging mechanism
  • More tests
  • Multi-threaded directory transfers
  • OpenSSH config file support
  • Progress notifications
  • Support for ciphers, compression, digests, kex & key type options
  • Support for disabled algorithms
  • Support for ED25519 & ECDSA keys
  • Support for private key passwords
  • Thread-safe connection manager
  • Transfer Resumption

Example

from sftpretty import CnOpts, Connection


# Basic

with Connection('hostname', username='me', password='secret') as sftp:
    # Temporarily chdir to public/.
    with sftp.cd('public'):
        # Upload file to public/ on remote.
        sftp.put('/my/local/filename')
        # Download a remote file from public/.
        sftp.get('remote_file')


with Connection('hostname', private_key='~/.ssh/id_ed25519',
                private_key_pass='secret') as sftp:
    # Upload local directory to remote_directory.
    sftp.put_d('/my/local', '/remote_directory')

    # Recursively download a remote_directory and save it to /tmp locally.
    sftp.get_r('remote_directory', '/tmp')


# Advanced

# Use password authentication
with Connection('hostname', username='me', password='secret') as sftp:
    # Upload local directory to remote_directory. On occurance of any
    # exception or child of, passed in the tuple, retry the operation.
    # Between each attempt increment a pause equal to backoff * delay.
    # Run a total of tries (six) times including the first attempt.
    sftp.put_d('/my/local', '/remote_directory', backoff=2, delay=1,
               exceptions=(NoValidConnectionsError, socket.timeout,
                           SSHException), tries=6)


# Use public key authentication
with Connection('hostname', private_key='~/.ssh/id_ed25519') as sftp:
    # Resume the download of a bigfile and save it to /mnt locally.
    sftp.get('bigfile', '/mnt', preserve_mtime=True, resume=True)


# Use public key authentication with optional private key password
with Connection('hostname', private_key='~/.ssh/id_ed25519',
                private_key_pass='secret') as sftp:
    # Recursively download a remote_directory and save it to /tmp locally.
    # Don't confirm files, useful in a scenario where the server removes
    # the remote file immediately after download. Preserve remote mtime on
    # local copy. Limit the thread pool connections to the server.
    sftp.get_r('remote_directory', '/tmp', confirm=False,
               preserve_mtime=True, workers=6)


# Use OpenSSH config for public key authentication. Configuration
# connection values are prioritized when available. Credentials still need
# to be provided. There may be a significant delta between your ssh program
# and support for newer security option algorithms due to lagging support
# in paramiko.
cnopts = CnOpts(config='~/.ssh/config', knownhosts='server.pub')
with Connection('alias', cnopts=cnopts, private_key_pass='secret') as sftp:
    # Rename existing file on remote server
    sftp.rename('/remote/old_name', '/remote/new_name')


# Pass custom host key file for verification 
cnopts = CnOpts(knownhosts='sftpserver.pub')
# Use connection options to set preferred encryption standards
cnopts.ciphers= ('aes256-ctr', 'aes128-ctr')
cnopts.digests = ('hmac-sha2-512', 'hmac-sha2-256')
cnopts.kex = ('ecdh-sha2-nistp521', 'ecdh-sha2-nistp384')
cnopts.key_types = ('ssh-ed25519', 'ecdsa-sha2-nistp521')
# Turn on verbose logging and set custom log file
cnopts.log = '/var/log/backups/daily.log'
cnopts.log_level = 'debug'
# Pass options object directly to connection object
with Connection('hostname', cnopts=cnopts, private_key='~/.ssh/id_backup',
                private_key_pass='secret') as sftp:
    # Aggressively retry important operation
    sftp.put_r('/local_backup', '/remote_backup', backoff=2, delay=1,
               exceptions=socket.timeout, preserve_mtime=True, tries=11)

Additional Information

Requirements

paramiko >= 1.17.0

Supports

Tested on Python 3.6, 3.7, 3.8, 3.9, 3.10, 3.11

sftpretty's People

Contributors

byteskeptical avatar coreyhartley avatar erans avatar ofiliojo avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar

sftpretty's Issues

ChannelException while using get_r

We had been using version 1.0.6 for a few weeks in our repo, as well as what is in root for the last few days, and started to notice this exception consistently happening when attempting to use get_r. Most files we tried to get came through, but we would not be able to grab the last few every time. Here is the traceback:

Traceback (most recent call last):
  File "C:\Program Files\Python38\lib\multiprocessing\pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "C:\Users\git\src\interfaces\sbc_interface.py", line 902, in _download_directory
    sftp.get_r(".", local_directory)
  File "C:\Users\git\.venv\lib\site-packages\sftpretty\__init__.py", line 497, in get_r
    self.get_d(
  File "C:\Users\git\.venv\lib\site-packages\sftpretty\__init__.py", line 448, in get_d
    raise err
  File "C:\Users\git\.venv\lib\site-packages\sftpretty\__init__.py", line 445, in get_d
    data = future.result()
  File "C:\Program Files\Python38\lib\concurrent\futures\_base.py", line 432, in result
    return self.__get_result()
  File "C:\Program Files\Python38\lib\concurrent\futures\_base.py", line 388, in __get_result
    raise self._exception
  File "C:\Program Files\Python38\lib\concurrent\futures\thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "C:\Users\git\.venv\lib\site-packages\sftpretty\__init__.py", line 366, in get
    _get(self, remotefile, localpath=localpath, callback=callback,
  File "C:\Users\git\.venv\lib\site-packages\sftpretty\__init__.py", line 355, in _get
    with self._sftp_channel() as channel:
  File "C:\Program Files\Python38\lib\contextlib.py", line 113, in __enter__
    return next(self.gen)
  File "C:\Users\git\.venv\lib\site-packages\sftpretty\__init__.py", line 244, in _sftp_channel
    raise err
  File "C:\Users\git\.venv\lib\site-packages\sftpretty\__init__.py", line 230, in _sftp_channel
    _channel = SFTPClient.from_transport(self._transport)
  File "C:\Users\git\.venv\lib\site-packages\paramiko\sftp_client.py", line 164, in from_transport
    chan = t.open_session(
  File "C:\Users\git\.venv\lib\site-packages\paramiko\transport.py", line 925, in open_session
    return self.open_channel(
  File "C:\Users\git\.venv\lib\site-packages\paramiko\transport.py", line 1067, in open_channel
    raise e
paramiko.ssh_exception.ChannelException: ChannelException(2, 'Connect failed')

Not sure if this was something being currently worked on or not, just wanted to shed some light on a potential issue. I was able to get pysftp working again on our repo and I can confirm that it did not happen using that dependency, with the same code.

We also noticed ChannelException(1, 'Administratively prohibited') in some instances as well.

when knowhosts directory doesn't exist. it will keep throwing FileNotFoundError regardless we set hostkeys=None

Currently using 1.1.2 version.
I have been using this way to set the hostkeys to None

cnopts.hostkeys = None

Apparently it doesnt matter because from the line 99 in __init__.py file, it will load from knownhosts.

Currently having an issue when using the client in a container that doesn't have knownhosts directory.
Even though I set hostkeys to None as above, it will try to resolve the default knownhosts which will raise FileNotFoundError.

So the workaround is to set the knownhosts to None, which will automatically set hotkeys to None as well. The only issue for this one is the type, it expects a string so it complains if it's None.

line 99

        if knownhosts is not None:
            try:
                self.hostkeys.load(Path(knownhosts).resolve().as_posix())
            except FileNotFoundError:
                # no known_hosts in the default unix location
                raise UserWarning(
                    f'No file or host key found in [{knownhosts}]. '
                    'You will need to explicitly load host keys '
                    '(cnopts.hostkeys.load(filename)) or disable host '
                    'key verification (cnopts.hostkeys = None).'
                )
            else:
                if len(self.hostkeys.items()) == 0:
                    raise HostKeysException('No host keys found!')
        else:
            self.hostkeys = None

So,

  1. Can we set setting the hostkeys to None taking priority. It will not try to load the default knownhosts directory.
  2. Or if not, I think making the knownhosts type to Optional[str] is okay for now.

Thank you!

Connection duplicated

Hallo,

If i'm useing the same connection function more as one time the connections duplicated on the next run.
Frist time works fine, on second time is my Connection duplicated.

The script is in a Flask backend and is scheduled every 2 min. (without scheduler is the same issue)

Code:

with Connection(URL, username=USER, private_key=KEYPATH, cnopts=cnopts) as sftp:
    print("oepn Connection", sftp.listdir(), file=sys.stderr)
    with sftp.cd('share'):
        for item in sftp.listdir():
            ...

Log:

gateway-server-1  | Friday, 16. June 2023 01:13:38 PM!
gateway-server-1  | [2023-06-16 13:13:38,935] INFO - [xyz.com] Host Key:
gateway-server-1  |        Name: ssh-rsa
gateway-server-1  |        Fingerprint: yxcs
gateway-server-1  |        Size: 4096
gateway-server-1  | open Connection ['share']
gateway-server-1  | [2023-06-16 13:13:40,273] INFO - Current Working Directory: [/data/share]
gateway-server-1  | [2023-06-16 13:13:40,288] INFO - [INFO] Start download file(s) for instance "00000_04_a_test"
gateway-server-1  | [2023-06-16 13:13:40,599] INFO - Current Working Directory: [/data/share]
gateway-server-1  | [2023-06-16 13:13:40,897] INFO - Current Working Directory: [/data/share]
gateway-server-1  | [2023-06-16 13:13:41,167] INFO - Current Working Directory: [/data/share/00000_04_a_test]
gateway-server-1  | [2023-06-16 13:13:41,180] INFO - [INFO] Downloaded 0 file(s) from instance "00000_04_a_test"
gateway-server-1  | [2023-06-16 13:13:41,438] INFO - Current Working Directory: [/data/share/00000_04_a_test]
gateway-server-1  | [2023-06-16 13:13:41,710] INFO - Current Working Directory: [/data/share]

gateway-server-1  | Friday, 16. June 2023 01:15:41 PM
gateway-server-1  | [2023-06-16 13:15:41,816] INFO - [xyz.com] Host Key:
gateway-server-1  |        Name: ssh-rsa
gateway-server-1  |        Fingerprint: yxcs
gateway-server-1  |        Size: 4096
gateway-server-1  | [2023-06-16 13:15:41,816] INFO - [xyz.com] Host Key:
gateway-server-1  |        Name: ssh-rsa
gateway-server-1  |        Fingerprint: yxcs
gateway-server-1  |        Size: 4096
gateway-server-1  | open Connection ['share']
gateway-server-1  | [2023-06-16 13:15:42,876] INFO - Current Working Directory: [/data/share]
gateway-server-1  | [2023-06-16 13:15:42,876] INFO - Current Working Directory: [/data/share]
gateway-server-1  | [2023-06-16 13:15:42,892] INFO - [INFO] Start download file(s) for instance "00000_04_a_test"
gateway-server-1  | [2023-06-16 13:15:42,892] INFO - [INFO] Start download file(s) for instance "00000_04_a_test"
gateway-server-1  | [2023-06-16 13:15:43,118] INFO - Current Working Directory: [/data/share]
gateway-server-1  | [2023-06-16 13:15:43,118] INFO - Current Working Directory: [/data/share]
gateway-server-1  | [2023-06-16 13:15:43,406] INFO - Current Working Directory: [/data/share]
gateway-server-1  | [2023-06-16 13:15:43,406] INFO - Current Working Directory: [/data/share]
gateway-server-1  | [2023-06-16 13:15:43,649] INFO - Current Working Directory: [/data/share/00000_04_a_test]
gateway-server-1  | [2023-06-16 13:15:43,649] INFO - Current Working Directory: [/data/share/00000_04_a_test]
gateway-server-1  | [2023-06-16 13:15:43,663] INFO - [INFO] Downloaded 0 file(s) from instance "00000_04_a_test"
gateway-server-1  | [2023-06-16 13:15:43,663] INFO - [INFO] Downloaded 0 file(s) from instance "00000_04_a_test"
gateway-server-1  | [2023-06-16 13:15:43,884] INFO - Current Working Directory: [/data/share/00000_04_a_test]
gateway-server-1  | [2023-06-16 13:15:43,884] INFO - Current Working Directory: [/data/share/00000_04_a_test]
gateway-server-1  | [2023-06-16 13:15:44,117] INFO - Current Working Directory: [/data/share]
gateway-server-1  | [2023-06-16 13:15:44,117] INFO - Current Working Directory: [/data/share]

gateway-server-1  | Friday, 16. June 2023 01:17:41 PM
gateway-server-1  | [2023-06-16 13:17:41,868] INFO - [xyz.com] Host Key:
gateway-server-1  |        Name: ssh-rsa
gateway-server-1  |        Fingerprint: yxcs
gateway-server-1  |        Size: 4096
gateway-server-1  | [2023-06-16 13:17:41,868] INFO - [xyz.com] Host Key:
gateway-server-1  |        Name: ssh-rsa
gateway-server-1  |        Fingerprint: yxcs
gateway-server-1  |        Size: 4096
gateway-server-1  | [2023-06-16 13:17:41,868] INFO - [xyz.com] Host Key:
gateway-server-1  |        Name: ssh-rsa
gateway-server-1  |        Fingerprint: yxcs
gateway-server-1  |        Size: 4096
gateway-server-1  | open Connection ['share']
gateway-server-1  | [2023-06-16 13:17:43,056] INFO - Current Working Directory: [/data/share]
gateway-server-1  | [2023-06-16 13:17:43,056] INFO - Current Working Directory: [/data/share]
gateway-server-1  | [2023-06-16 13:17:43,056] INFO - Current Working Directory: [/data/share]
gateway-server-1  | [2023-06-16 13:17:43,072] INFO - [INFO] Start download file(s) for instance "00000_04_a_test"
gateway-server-1  | [2023-06-16 13:17:43,072] INFO - [INFO] Start download file(s) for instance "00000_04_a_test"
gateway-server-1  | [2023-06-16 13:17:43,072] INFO - [INFO] Start download file(s) for instance "00000_04_a_test"
gateway-server-1  | [2023-06-16 13:17:43,321] INFO - Current Working Directory: [/data/share]
gateway-server-1  | [2023-06-16 13:17:43,321] INFO - Current Working Directory: [/data/share]
gateway-server-1  | [2023-06-16 13:17:43,321] INFO - Current Working Directory: [/data/share]
gateway-server-1  | [2023-06-16 13:17:43,596] INFO - Current Working Directory: [/data/share]
gateway-server-1  | [2023-06-16 13:17:43,596] INFO - Current Working Directory: [/data/share]
gateway-server-1  | [2023-06-16 13:17:43,596] INFO - Current Working Directory: [/data/share]
gateway-server-1  | [2023-06-16 13:17:43,958] INFO - Current Working Directory: [/data/share/00000_04_a_test]
gateway-server-1  | [2023-06-16 13:17:43,958] INFO - Current Working Directory: [/data/share/00000_04_a_test]
gateway-server-1  | [2023-06-16 13:17:43,958] INFO - Current Working Directory: [/data/share/00000_04_a_test]
gateway-server-1  | [2023-06-16 13:17:43,975] INFO - [INFO] Downloaded 0 file(s) from instance "00000_04_a_test"
gateway-server-1  | [2023-06-16 13:17:43,975] INFO - [INFO] Downloaded 0 file(s) from instance "00000_04_a_test"
gateway-server-1  | [2023-06-16 13:17:43,975] INFO - [INFO] Downloaded 0 file(s) from instance "00000_04_a_test"
gateway-server-1  | [2023-06-16 13:17:44,234] INFO - Current Working Directory: [/data/share/00000_04_a_test]
gateway-server-1  | [2023-06-16 13:17:44,234] INFO - Current Working Directory: [/data/share/00000_04_a_test]
gateway-server-1  | [2023-06-16 13:17:44,234] INFO - Current Working Directory: [/data/share/00000_04_a_test]
gateway-server-1  | [2023-06-16 13:17:44,492] INFO - Current Working Directory: [/data/share]
gateway-server-1  | [2023-06-16 13:17:44,492] INFO - Current Working Directory: [/data/share]
gateway-server-1  | [2023-06-16 13:17:44,492] INFO - Current Working Directory: [/data/share]

gateway-server-1  | Friday, 16. June 2023 01:19:41 PM
gateway-server-1  | [2023-06-16 13:19:41,846] INFO - [xyz.com] Host Key:
gateway-server-1  |        Name: ssh-rsa
gateway-server-1  |        Fingerprint: yxcs
gateway-server-1  |        Size: 4096
gateway-server-1  | [2023-06-16 13:19:41,846] INFO - [xyz.com] Host Key:
gateway-server-1  |        Name: ssh-rsa
gateway-server-1  |        Fingerprint: yxcs
gateway-server-1  |        Size: 4096
gateway-server-1  | [2023-06-16 13:19:41,846] INFO - [xyz.com] Host Key:
gateway-server-1  |        Name: ssh-rsa
gateway-server-1  |        Fingerprint: yxcs
gateway-server-1  |        Size: 4096
gateway-server-1  | [2023-06-16 13:19:41,846] INFO - [xyz.com] Host Key:
gateway-server-1  |        Name: ssh-rsa
gateway-server-1  |        Fingerprint: yxcs
gateway-server-1  |        Size: 4096
gateway-server-1  | oepn Connection ['share']
gateway-server-1  | [2023-06-16 13:19:43,043] INFO - Current Working Directory: [/data/share]
gateway-server-1  | [2023-06-16 13:19:43,043] INFO - Current Working Directory: [/data/share]
gateway-server-1  | [2023-06-16 13:19:43,043] INFO - Current Working Directory: [/data/share]
gateway-server-1  | [2023-06-16 13:19:43,043] INFO - Current Working Directory: [/data/share]
gateway-server-1  | [2023-06-16 13:19:43,059] INFO - [INFO] Start download file(s) for instance "00000_04_a_test"
gateway-server-1  | [2023-06-16 13:19:43,059] INFO - [INFO] Start download file(s) for instance "00000_04_a_test"
gateway-server-1  | [2023-06-16 13:19:43,059] INFO - [INFO] Start download file(s) for instance "00000_04_a_test"
gateway-server-1  | [2023-06-16 13:19:43,059] INFO - [INFO] Start download file(s) for instance "00000_04_a_test"
gateway-server-1  | [2023-06-16 13:19:43,347] INFO - Current Working Directory: [/data/share]
gateway-server-1  | [2023-06-16 13:19:43,347] INFO - Current Working Directory: [/data/share]
gateway-server-1  | [2023-06-16 13:19:43,347] INFO - Current Working Directory: [/data/share]
gateway-server-1  | [2023-06-16 13:19:43,347] INFO - Current Working Directory: [/data/share]
gateway-server-1  | [2023-06-16 13:19:43,660] INFO - Current Working Directory: [/data/share]
gateway-server-1  | [2023-06-16 13:19:43,660] INFO - Current Working Directory: [/data/share]
gateway-server-1  | [2023-06-16 13:19:43,660] INFO - Current Working Directory: [/data/share]
gateway-server-1  | [2023-06-16 13:19:43,660] INFO - Current Working Directory: [/data/share]
gateway-server-1  | [2023-06-16 13:19:43,904] INFO - Current Working Directory: [/data/share/00000_04_a_test]
gateway-server-1  | [2023-06-16 13:19:43,904] INFO - Current Working Directory: [/data/share/00000_04_a_test]
gateway-server-1  | [2023-06-16 13:19:43,904] INFO - Current Working Directory: [/data/share/00000_04_a_test]
gateway-server-1  | [2023-06-16 13:19:43,904] INFO - Current Working Directory: [/data/share/00000_04_a_test]
gateway-server-1  | [2023-06-16 13:19:43,922] INFO - [INFO] Downloaded 0 file(s) from instance "00000_04_a_test"
gateway-server-1  | [2023-06-16 13:19:43,922] INFO - [INFO] Downloaded 0 file(s) from instance "00000_04_a_test"
gateway-server-1  | [2023-06-16 13:19:43,922] INFO - [INFO] Downloaded 0 file(s) from instance "00000_04_a_test"
gateway-server-1  | [2023-06-16 13:19:43,922] INFO - [INFO] Downloaded 0 file(s) from instance "00000_04_a_test"
gateway-server-1  | [2023-06-16 13:19:44,156] INFO - Current Working Directory: [/data/share/00000_04_a_test]
gateway-server-1  | [2023-06-16 13:19:44,156] INFO - Current Working Directory: [/data/share/00000_04_a_test]
gateway-server-1  | [2023-06-16 13:19:44,156] INFO - Current Working Directory: [/data/share/00000_04_a_test]
gateway-server-1  | [2023-06-16 13:19:44,156] INFO - Current Working Directory: [/data/share/00000_04_a_test]
gateway-server-1  | [2023-06-16 13:19:44,406] INFO - Current Working Directory: [/data/share]
gateway-server-1  | [2023-06-16 13:19:44,406] INFO - Current Working Directory: [/data/share]
gateway-server-1  | [2023-06-16 13:19:44,406] INFO - Current Working Directory: [/data/share]
gateway-server-1  | [2023-06-16 13:19:44,406] INFO - Current Working Directory: [/data/share]

Need less logging

Question how can I turn down the amount of logging that sftpretty does on info?

put_d() unexpected remote path handling and issue with absolute Windows paths

I found two issues with the put_d() function.

  1. It looks like it doesn't accept absolute Windows paths.
    I get the error:

ValueError: 'C:\\Users\\johnm\\PycharmProjects\\project\\.other\\folder\\file' does not start with '\\'

  1. When I call put_d("local_folder", "folder"), I get the following error:

File "C:\Users\johnm\PycharmProjects\project\venv\lib\site-packages\sftpretty\__init__.py", line 563, in put
confirm=confirm, preserve_mtime=preserve_mtime)
File "C:\Users\johnm\PycharmProjects\project\venv\lib\site-packages\sftpretty\__init__.py", line 554, in _put
confirm=confirm)
File "C:\Users\johnm\PycharmProjects\project\venv\lib\site-packages\paramiko\sftp_client.py", line 759, in put
return self.putfo(fl, remotepath, file_size, callback, confirm)
File "C:\Users\johnm\PycharmProjects\project\venv\lib\site-packages\paramiko\sftp_client.py", line 714, in putfo
with self.file(remotepath, "wb") as fr:
File "C:\Users\johnm\PycharmProjects\project\venv\lib\site-packages\paramiko\sftp_client.py", line 372, in open
t, msg = self._request(CMD_OPEN, filename, imode, attrblock)
File "C:\Users\johnm\PycharmProjects\project\venv\lib\site-packages\paramiko\sftp_client.py", line 822, in _request
return self._read_response(num)
File "C:\Users\johnm\PycharmProjects\project\venv\lib\site-packages\paramiko\sftp_client.py", line 874, in _read_response
self._convert_status(msg)
File "C:\Users\johnm\PycharmProjects\project\venv\lib\site-packages\paramiko\sftp_client.py", line 903, in _convert_status
raise IOError(errno.ENOENT, text)
FileNotFoundError: [Errno 2] Path not found.

As I understand, the path it refers to is the remote path. When I debug it, in put_d(), the paths it creates are the following:
image
As you can see, the remotedir "folder/local_folder/file" is probably the issue. In my remote location, it creates the folder "folder". This of course makes sense as it successfully calls self.mkdir_p(remotedir) in line 599. Do you think this has to do with the location of the self.mkdir_p(remotedir) call? Should there be code that also creates the subfolders of the "local_folder" in the remote location(here the "folder/local_folder" subfolder)?

Thank you for your time.

Doesn't use SSH config file.

I think this is technically a feature request, not a bug, but:

I got the library working fine as a replacement for pysftp, which is good, but there's one thing about pysftp that bothered me and it's still here: it doesn't read the user's ~/.ssh/config file (or seem to have a function you can call to read it).

I shouldn't have to force my program (using the library) to get the host IP address (if using a custom name, such as "home"), username, and keyfile to use with the server, when the user may very well already have all that information stored inside their config file.

Even if it doesn't read it by default, the option should be there to read it via some function.

Windows: No hostkey for host [localhost] found - Part 2

Using 'localhost' throughout and the basic 'get' example from the Readme (see #20). The outcome is the same ie. the hostkey is not found.

One odd thing in the test_sftp.py output is it says Name: ssh-ed25519. In previous runs it said Name: ssh-rsa.

Details below:

> ssh-keygen -t rsa -b 4096
Generating public/private rsa key pair.
Enter file in which to save the key (C:\Users\dines/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in C:\Users\dines/.ssh/id_rsa
Your public key has been saved in C:\Users\dines/.ssh/id_rsa.pub
The key fingerprint is:
SHA256:BAraPwRjqXDCYe32nhMW1jG8H+OxRblaUipg8iRZ1zk dines@T2
The key's randomart image is:
+---[RSA 4096]----+
|.oB.o.o.. . .    |
|+=oO =.= E +     |
|o+o O o = = .    |
|.  = + = * +     |
|  . = . S X      |
|     =   =       |
|    o o          |
|     +           |
|      .          |
+----[SHA256]-----+
-----BEGIN OPENSSH PRIVATE KEY-----
b3BlbnNzaC1rZXktdjEAAAAABG5vbmUAAAAEbm9uZQAAAAAAAAABAAACFwAAAAdzc2gtcn
NhAAAAAwEAAQAAAgEAuD15gTVxdEaqWzxFRM6X7tPoh9ebNWvLe+/aQ8gPpJL5eD8nEOJX
1vwPFO1eR9BJC362W1CzvJTWKw+zzuHkj+gjnEqStp5/uNvE2gkU1kiMSbNlaRTAbEmcIj
qmggT+5fFzW71JK7ixX3926FS89t9ryRIiMC7l7bCRijeWAnM8x7IINf+J1OBA4vWRo7CN
6iHeBunSINr0Hqu4G/EZzq+pece6kWsOrej5/XPl3CF82KWsbP2x/uz6DbCONuAf5iPum5
AMzBiqhaVGQDKg6ypWhPcB+za1nZzlFzOxAmm7zXYfol8GIUgkxh/O3LKV0TTbrhEMwkz2
s4UfDHz0xa5w9tSAXKt2P2aOcfjw7NbLGpeqD8e8NJQ4xIE/X9bdxVJbO0x2Op5LmzJlf7
xqcgehl9myq8kQud02hHkLL6HgAYXF+2rNKL/eP4crqsZH7/pXEotIE2c4nRgEPvSI26gS
kK3hc7pzbrB7GnglBWyK4s3y8jsy0mHqasjtMm/3i4VxpaRBVRTel1SFmorN2FtF4Icv+N
IpToiKi1s0SHn8YE1ascZkwQnrGyeY30hu0dscyuzYQbgxeUwFzB1sZABDy0p5A973/bW8
/sfhx6UvDswpgA+5+U25AgRgciRbl2NrsjTYm22j7wvKe+/K1heho/7MII9kT0jHs1hhoq
cAAAdAsvP5HbLz+R0AAAAHc3NoLXJzYQAAAgEAuD15gTVxdEaqWzxFRM6X7tPoh9ebNWvL
e+/aQ8gPpJL5eD8nEOJX1vwPFO1eR9BJC362W1CzvJTWKw+zzuHkj+gjnEqStp5/uNvE2g
kU1kiMSbNlaRTAbEmcIjqmggT+5fFzW71JK7ixX3926FS89t9ryRIiMC7l7bCRijeWAnM8
x7IINf+J1OBA4vWRo7CN6iHeBunSINr0Hqu4G/EZzq+pece6kWsOrej5/XPl3CF82KWsbP
2x/uz6DbCONuAf5iPum5AMzBiqhaVGQDKg6ypWhPcB+za1nZzlFzOxAmm7zXYfol8GIUgk
xh/O3LKV0TTbrhEMwkz2s4UfDHz0xa5w9tSAXKt2P2aOcfjw7NbLGpeqD8e8NJQ4xIE/X9
bdxVJbO0x2Op5LmzJlf7xqcgehl9myq8kQud02hHkLL6HgAYXF+2rNKL/eP4crqsZH7/pX
EotIE2c4nRgEPvSI26gSkK3hc7pzbrB7GnglBWyK4s3y8jsy0mHqasjtMm/3i4VxpaRBVR
Tel1SFmorN2FtF4Icv+NIpToiKi1s0SHn8YE1ascZkwQnrGyeY30hu0dscyuzYQbgxeUwF
zB1sZABDy0p5A973/bW8/sfhx6UvDswpgA+5+U25AgRgciRbl2NrsjTYm22j7wvKe+/K1h
eho/7MII9kT0jHs1hhoqcAAAADAQABAAACABvaj2gNYF2n4VFxxzcj1KxytRD0c66b4WOX
W3Qr5tGGttiLE6jlz0ndSZpsWHia8LAgNAmSJdo41jgkY5a/aAKpodibHZsmU0DrWNNt1A
yhYzQxPv+n939nkYdLPqkhgDfbTQltkIZfodFCQAPX68mW9YCskSafsHtC2n3+8VrleSbn
r34aaAga7zRg0+ffZVUkUaDR8Mau04FfjLIMvU4WGJUksPwTJ2D7e+/pebLSDUHjqjHAGQ
zlXth6/MUnFSuB+yzhlk203PLfjcNe5B0phXMMtOv51XNCqQ/c7MHkolPpq9asbwzRD140
qN2M0kGSsDOcuEUXxaRWV99XdNm8XQEPmUcgQXne5JjnQwOVyt4UncoLIy7wgH0dSRnZWG
lp7HVXmA10349QPyF9f03nzydZa8o2S0AYDnKBfTTWzldQVM0tV+lxYaMxUbgrks2mx4+a
Bovc0aODhHTtvOcgzwmBWKxuisSwB3t190voZWi+wYubBVSsK0GAj3z66x510Rb9ui/H5N
dv0gpEQDHfNdlGjo5F2TauBLKJGGyI6b/70zLUtWvH4h+czP/Ryp479qPTvBQySmzZUh+E
mFCXGb3lCA4jXwoWxRdlgw0o8n5ouUz08yXr6NoxwJKIjMyJzdS0I2iLheoXRj5fytisrd
CzSztpfEkf746+BL6hAAABAQCzH4OYGDXKzdymyOnXbnJYYNZfndhLhDuJ4LA/HMwos+wA
uXC6y3/dE7Cp6/rUIccpmBaOCQ7oRmztd6aI55e18a7dvU2qIqedqqXAnh91PfdWqPA7QH
XkPBDdfmp4id02PCaXBtCJGUWZEUPnwvhAmf2DWGPz2Z7TE5YZlWakxdmxrRLZEPvGdlrA
ZdObKEAuLFPo9AG7h1G609yc2BSeSh+zW2dLLpwzVs3gHaPu2iK9iGdrel9KRuR24CtiEw
LVyRHCatYSpkrva/Y11ZMkWXA4X2Z/YMBg7CDupJs+zTPmhpxDE95XE0wyXf2ewYcOzuHj
eyQ/pysXfzzYyGPRAAABAQDcdXXo5k8rtBBAB9GCYIyBKQ5ZA/e+tAyMx12EugDWxC/Fzv
wKm10UpbPJoPP9JxXrexwlE8iWsxsPeLQiLUREvd5Y5XvJK4J+BLbf9KWJA0eD/pfM5UtI
vt11Z+5ldnuc801IA9DDEFlk4AxCReNhwlIAH2/t0VlGfYirBDbtU4l/im8zCqxjgY7Gr/
zN8vF+E+khEz6AOwwFRJ9QVi8B6jxxQwaxLtCfN7ZnukwuNzfh2IQvD/V5DtMxerd9pxFP
OgHPZfEpuJUaQX5T7vp9U9oMpQceinsFF57ylKrlllSeP4hrOUhI71PWUZfi0NMUFoYkFo
ETfq4cjlVy2CZVAAABAQDV8TwtK0so7aPS9GQBrSM7u18/r4iPdaeSdvEEa/gaZV9VNO9D
KZdxIK4N9mth0D4OymtcfBFXOWLGVoAJkWEu9OsXESNInkUtaPhMjXyTfUM9rZQ6+LHqPp
+RCq6NKgUnYmGrd8pCizQFh4kOoWJVk097ZXuR4M8o3N9Uw4eXg9SGRsqTFQfttEdfzh74
bMdBCg94GYCjzD5mCnsjFBwfIU859MU7ppQxtIse1UHL6uobdAeiz25KiiAQcQO1KzJBJw
J1cXQuwgGfiMcuVICSoGqCwTME0ciDknM6cdsPPxVU+cQ/Bqhvd8tW8wJHpfm5etrathHf
8sqB4mEpwwkLAAAACGRpbmVzQFQyAQI=
-----END OPENSSH PRIVATE KEY-----
> sftpserver -k C:\Users\dines\.ssh\id_rsa -l DEBUG
DEBUG:sftpserver.__main__:Serving C:\Users\dines over sftp at localhost:3373
DEBUG:sftpserver.__main__:Starting a new thread
INFO:paramiko.transport:Connected (version 2.0, client OpenSSH_for_Windows_8.6)
INFO:paramiko.transport:Auth rejected (none).
INFO:paramiko.transport:Auth granted (publickey).
DEBUG:sftpserver.__main__:1 active sessions
ERROR:paramiko.transport:Socket exception: An existing connection was forcibly closed by the remote host (10054)
> sftp -P 3373 localhost
The authenticity of host '[localhost]:3373 ([127.0.0.1]:3373)' can't be established.
RSA key fingerprint is SHA256:BAraPwRjqXDCYe32nhMW1jG8H+OxRblaUipg8iRZ1zk.
This key is not known by any other names
Are you sure you want to continue connecting (yes/no/[fingerprint])?
Warning: Permanently added '[localhost]:3373' (RSA) to the list of known hosts.
Connected to localhost.

[localhost]:3373 ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC4PXmBNXF0RqpbPEVEzpfu0+iH15s1a8t779pDyA+kkvl4PycQ4lfW/A8U7V5H0EkLfrZbULO8lNYrD7PO4eSP6COcSpK2nn+428TaCRTWSIxJs2VpFMBsSZwiOqaCBP7l8XNbvUkruLFff3boVLz232vJEiIwLuXtsJGKN5YCczzHsgg1/4nU4EDi9ZGjsI3qId4G6dIg2vQeq7gb8RnOr6l5x7qRaw6t6Pn9c+XcIXzYpaxs/bH+7PoNsI424B/mI+6bkAzMGKqFpUZAMqDrKlaE9wH7NrWdnOUXM7ECabvNdh+iXwYhSCTGH87cspXRNNuuEQzCTPazhR8MfPTFrnD21IBcq3Y/Zo5x+PDs1ssal6oPx7w0lDjEgT9f1t3FUls7THY6nkubMmV/vGpyB6GX2bKryRC53TaEeQsvoeABhcX7as0ov94/hyuqxkfv+lcSi0gTZzidGAQ+9IjbqBKQreFzunNusHsaeCUFbIrizfLyOzLSYepqyO0yb/eLhXGlpEFVFN6XVIWais3YW0Xghy/40ilOiIqLWzRIefxgTVqxxmTBCesbJ5jfSG7R2xzK7NhBuDF5TAXMHWxkAEPLSnkD3vf9tbz+x+HHpS8OzCmAD7n5TbkCBGByJFuXY2uyNNibbaPvC8p778rWF6Gj/swgj2RPSMezWGGipw==

# test_sftp.py
from sftpretty import Connection

with Connection('localhost', username='admin', password='admin') as sftp:
    sftp.get('../../src/source.txt', localpath='../../dst/destination.txt')
> python test_sftp.py
[2023-02-27 17:39:30,614] INFO - [localhost] Host Key:
        Name: ssh-ed25519
        Fingerprint: b'0c407c4e6f0d6eb800a376c649524f0f'
        Size: 256
Traceback (most recent call last):
  File "C:\Users\dines\test_sftp.py", line 3, in <module>
    with Connection('localhost', username='admin', password='admin') as sftp:
  File "C:\Users\dines\AppData\Local\Programs\Python\Python310\lib\site-packages\sftpretty\__init__.py", line 140, in __init__
    self._start_transport(host, port, compress=compress)
  File "C:\Users\dines\AppData\Local\Programs\Python\Python310\lib\site-packages\sftpretty\__init__.py", line 309, in _start_transport
    raise err
  File "C:\Users\dines\AppData\Local\Programs\Python\Python310\lib\site-packages\sftpretty\__init__.py", line 295, in _start_transport
    user_hostkey = self._cnopts.get_hostkey(host)
  File "C:\Users\dines\AppData\Local\Programs\Python\Python310\lib\site-packages\sftpretty\__init__.py", line 98, in get_hostkey
    raise SSHException(f'No hostkey for host [{host}] found.')
paramiko.ssh_exception.SSHException: No hostkey for host [localhost] found.

Windows: No hostkey for host [localhost] found.

cnopts = CnOpts(knownhosts='C:/Users/dines/.ssh/known_hosts')

Traceback (most recent call last):
  File "C:\Users\dines\..test_sftp.py", line 48, in <module>
    main(sys.argv)
  File "C:\Users\dines\..test_sftp.py", line 41, in main
    with Connection(host='localhost', port=3373, username='admin', password='admin', cnopts=cnopts) as sftp:
  File "C:\Users\dines\AppData\Local\Programs\Python\Python310\lib\site-packages\sftpretty\__init__.py", line 140, in __init__
    self._start_transport(host, port, compress=compress)
  File "C:\Users\dines\AppData\Local\Programs\Python\Python310\lib\site-packages\sftpretty\__init__.py", line 309, in _start_transport
    raise err
  File "C:\Users\dines\AppData\Local\Programs\Python\Python310\lib\site-packages\sftpretty\__init__.py", line 295, in _start_transport
    user_hostkey = self._cnopts.get_hostkey(host)
  File "C:\Users\dines\AppData\Local\Programs\Python\Python310\lib\site-packages\sftpretty\__init__.py", line 98, in get_hostkey
    raise SSHException(f'No hostkey for host [{host}] found.')
paramiko.ssh_exception.SSHException: No hostkey for host [localhost] found.

Looking at:
https://github.com/byteskeptical/sftpretty/blob/d8e2b07eaa050fa12cb32a37213e2f118248b173/sftpretty/__init__.py#L50
https://github.com/byteskeptical/sftpretty/blob/d8e2b07eaa050fa12cb32a37213e2f118248b173/sftpretty/__init__.py#L75

The code looks as if it should work or is something missing for Windows?

Can't catch IOError when retry is enabled during uploading a file

python: 3.10.6
sftpretty:1.1.1
paramiko: 3.3.1
Ubuntu 22.04.3 LTS

When trying to catch error with a retry enabled, when the IOError is thrown, it will not be caught and it will give this stack trace:

.venv/lib/python3.10/site-packages/sftpretty/helpers.py:149: in _retry
    return f(*args, **kwargs)
.venv/lib/python3.10/site-packages/sftpretty/__init__.py:603: in _put
    remote_attributes = channel.put(localfile,
.venv/lib/python3.10/site-packages/paramiko/sftp_client.py:759: in put
    return self.putfo(fl, remotepath, file_size, callback, confirm)
.venv/lib/python3.10/site-packages/paramiko/sftp_client.py:714: in putfo
    with self.file(remotepath, "wb") as fr:
.venv/lib/python3.10/site-packages/paramiko/sftp_client.py:372: in open
    t, msg = self._request(CMD_OPEN, filename, imode, attrblock)
.venv/lib/python3.10/site-packages/paramiko/sftp_client.py:822: in _request
    return self._read_response(num)
.venv/lib/python3.10/site-packages/paramiko/sftp_client.py:874: in _read_response
    self._convert_status(msg)
.venv/lib/python3.10/site-packages/paramiko/sftp_client.py:905: in _convert_status
    raise IOError(errno.EACCES, text)
E   PermissionError: [Errno 13] Permission denied

During handling of the above exception, another exception occurred:
tests/ephemeral_performance/conftest.py:492: in run_sod_schedule
    sftp_client.upload_file(
src/common/clients/sftp.py:41: in upload_file
    sftp.put(localfile, remotedir, logger=logger, **kwargs)
.venv/lib/python3.10/site-packages/sftpretty/__init__.py:614: in put
    return _put(self, localfile, remotepath=remotepath, callback=callback,
.venv/lib/python3.10/site-packages/sftpretty/helpers.py:150: in _retry
    except all_exception_types as e:
E   TypeError: catching classes that do not inherit from BaseException is not allowed

Steps to reproduce:

run a similar code like this, but the file exists already in the remote directory so it will give us permission denied error.

try:
    sftp.put(localfile, remotefile, tries=3)
except IOError:
    print("Error caught")

get_r copies remote folder structure into localdir

We have been using the following code (formerly pysftp) to navigate to the remote directory, to then copy its contents to a local directory:

sftp.chdir(remote_directory)
sftp.get_r(".", local_directory)

If our remote path was structured like \path\to\remote\data\* (where * is our data), and our local directory was \path\local, our directory structure now looks like \path\local\path\to\remote\data\* after using get_r. We would expect to see the directory structure look like \path\local\* after using it.

AttributeError: str object has no attribute root for put_d in localdir.root

in put_d method, when we pass the localdir arg a string, it will give this error:

  File "/home/smuslim/repos/qa-all-python-tests/.venv/lib/python3.10/site-packages/sftpretty/__init__.py", line 860, in <listcomp>
    .joinpath(localpath.relative_to(localdir.root).as_posix())
AttributeError: 'str' object has no attribute 'root'

For Linux, I don't see anywhere localdir as a Path instance but in this line of code it's calling the Path method.

Expected: put_d works when the localdir arguments is a valid localpath string.

sftpretty adding console log to root logger

It appears that here sftpretty is adding a console log handler to the root logger: https://github.com/byteskeptical/sftpretty/blob/root/sftpretty/__init__.py#L252

This is causing our own logging to have duplicate logs for all code running after an sftpretty upload. I can submit a PR that adds a new option to toggle this behavior on or off, if that seems acceptable? Or if the contributors would prefer passing a name to getLogger instead of nothing, then users of it can customize sftpretty's logging behavior completely.

Question: Is the put_d method expected to upload files to remote path with the same path as localpath?

Question:
Is the put_d method expected to upload files to remote path relative to localpath's path?

Example I have two files
file1.txt and file2.txt
in my local path /home/mawaliya/test/
that I want to upload to
a remote sftp folder inbox

When I call put_d("/home/mawaliya/test", "inbox")
Remote paths for each file will be relative to local directory.
For example:
inbox/home/mawaliya/test/file1.txt

I do not have these directories set up remotely, so it throws IOError - FileNotFoundError.

Is it expected ?
I am expecting my remote path for the files to be:
inbox/file1.txt
inbox/file2.txt

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.