Git Product home page Git Product logo

sftp_client's Introduction

SFTP Client

Build Coverage Status Module Version Hex Docs Total Download License Last Updated

An Elixir SFTP Client that wraps Erlang's ssh and ssh_sftp.

Prerequisites

  • Erlang 24 or greater
  • Elixir 1.11 or greater

Installation

The package can be installed by adding :sftp_client to your list of dependencies in mix.exs:

def deps do
  [
    {:sftp_client, "~> 2.1"}
  ]
end

Usage

There are bang (!) counterparts for almost all available functions.

Connect & Disconnect

To open a new connection to an SFTP server:

{:ok, conn} = SFTPClient.connect(host: "ftp.myhost.com")

Due to a change in OTP 24 that removed automatic public key lookup in the sftp module you may need to define the modify_algorithms option to avoid "Key exchange failed" errors:

{:ok, conn} = SFTPClient.connect(
  host: "ftp.myhost.com",
  modify_algorithms: [
    append: [
      public_key: [:"ssh-rsa"]
    ]
  ]
)

Refer to the docs for SFTPClient.Operations.Connect.connect/1 to find out all available options.

It is strongly recommended to close a connection after your operations have completed:

SFTPClient.disconnect(conn)

For short-lived connections you can also use a function as second argument. After the function body has run or raises, the connection is automatically closed.

SFTPClient.connect([host: "ftp.myhost.com"], fn conn ->
  # Do something with conn
end)

Download

To download a file from the server you can use the following function.

SFTPClient.download_file(conn, "my/remote/dir/file.jpg", "my/dir/local-file.jpg")
# => {:ok, "my/dir/local-file.jpg"}

When the third argument is an existing directory on your file system, the file is downloaded to a file with the same name as the one on the server.

SFTPClient.download_file(conn, "my/remote/dir/image.png", "my/local/dir")
# => {:ok, "my/local/dir/image.png"}

It is also possible to use Streams to download data into a file or memory.

source_stream = SFTPClient.stream_file!(conn, "my/remote/file.jpg")
target_stream = File.stream!("my/local/file.jpg")

source_stream
|> Stream.into(target_stream)
|> Stream.run()

Upload

To upload are file from the file system you can use the following function.

SFTPClient.upload_file(conn, "my/local/dir/file.jpg", "my/remote/dir/file.jpg")
# => {:ok, "my/remote/dir/file.jpg"}

You can also use Streams to upload data. Please make sure to set a proper chunk size or the upload may be very slow.

source_stream = File.stream!("my/local/file.jpg", [], 32_768)
target_stream = SFTPClient.stream_file!(conn, "my/remote/file.jpg")

source_stream
|> Stream.into(target_stream)
|> Stream.run()

List Directory

SFTPClient.list_dir(conn, "my/dir")
# => {:ok, ["my/dir/file_1.jpg", "my/dir/file_2.jpg", ...]}

Create Directory

SFTPClient.make_dir(conn, "my/new/dir")

Note that this operation fails unless the parent directory exists.

Delete

To delete a file:

SFTPClient.delete_file(conn, "my/remote/file.jpg")

To delete a directory:

SFTPClient.delete_dir(conn, "my/remote/dir")

Note that a directory cannot be deleted as long as it still contains files.

Rename

To move/rename a file or directory:

SFTPClient.rename(conn, "my/remote/file.jpg", "my/remote/new-file.jpg")

File Info

You can retrieve meta data about a file from the server such as file size, modification time, file permissions, owner and so on.

SFTPClient.file_info(conn, "my/remote/file.jpg")
# => {:ok, %File.Stat{...}}

Refer to the File.Stat docs for a list of available file information.

Symbolic Links

There are also a couple of functions that handle symlinks.

It is possible to get the target of a symlink.

SFTPClient.read_link(conn, "my/remote/link.jpg")
# => {:ok, "my/remote/file.jpg"}

You can retrieve meta data about symlinks, similar to file_info/2.

SFTPClient.link_info(conn, "my/remote/link.jpg")
# => {:ok, %File.Stat{...}}

And you are able to create symlinks.

SFTPClient.make_link(conn, "my/remote/link.jpg", "my/remote/file.jpg")

License

This library is released under the MIT License. See the LICENSE.md file for further details.

sftp_client's People

Contributors

alecostard avatar brendonpierson avatar kafaichoi avatar kianmeng avatar morinap avatar piacsek avatar tlux avatar tudborg avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

sftp_client's Issues

Protocol Collectable not implemented for %SFTPClient.Stream

Hi i'm trying to upload a file using SFTPClient.upload_file!(conn, local_path, remote_path).

but i'm getting

(Protocol.UndefinedError) protocol Collectable not implemented for %SFTPClient.Stream{chunk_size: 32768, conn: %SFTPClient.Conn{channel_pid: #PID<0.661.0>, config: %SFTPClient.Config{connect_timeout: 5000, dsa_pass_phrase: nil, ecdsa_pass_phrase: nil, host: "host_ip", inet: :inet, key_cb: {SFTPClient.KeyProvider, [private_key_path: nil, private_key_pass_phrase: nil]}, operation_timeout: :infinity, password: nil, port: port, private_key_pass_phrase: nil, private_key_path: nil, rsa_pass_phrase: 'secret, sftp_vsn: nil, system_dir: nil, user: 'user', user_dir: 'keys_dir'}, conn_ref: #PID<0.659.0>}, path: "remote_path"} of type SFTPClient.Stream (a struct). This protocol is implemented for the following type(s): Ecto.Adapters.SQL.Stream, Postgrex.Stream, IO.Stream, HashDict, File.Stream, Map, MapSet, HashSet, BitString, List (elixir 1.10.2) lib/collectable.ex:1: Collectable.impl_for!/1 (elixir 1.10.2) lib/collectable.ex:77: Collectable.into/1 (elixir 1.10.2) lib/stream.ex:531: Stream.do_into/5 (elixir 1.10.2) lib/stream.ex:649: Stream.run/1 (sftp_client 1.4.0) lib/sftp_client/operations/upload_file.ex:37: SFTPClient.Operations.UploadFile.upload_file!/3

what do i am doing wrong?

"module SFTPClient is not available" (related to the issue: add :sftp_client to "application" in the mix.exs file ?)

Found it !
That happens when I make and run a prod release (Elixir 1.9.0):
set MIX_ENV=prod mix release
then myprogram start
then when I try a command (myprogram rpc "mycommand(args)":
=> crash and
** (UndefinedFunctionError) function SFTPClient.connect/1 is undefined (module SFTPClient is not available)
When I had :sftp_client to applications or extra_applications options in mix.exs, everything is OK.
Did I forget something when I made the release ?
That doesn't happen with iex -S mix (and yes, the config files have the same content: dev.exs == releases.exs ;) )

[Edit] Seems to be a common issue : Here and here

ip bindings

what is right way to define ip interface for outgoing connections?

Cannot install inside of Livebook Desktop

When trying to install in Livebook desktop on Mac with the following command:

Mix.install(
  [
    {:sftp_client, "~> 1.4"}
  ],
  force: true
)

The following error is received:

== Compilation error in file lib/sftp_client/ssh_xfer_attr.ex ==
** (RuntimeError) error parsing file /Applications/Livebook.app/Contents/Resources/rel/vendor/otp/lib/ssh-5.0.1/src/ssh_xfer.hrl, got: {:error, :enoent}
    (elixir 1.15.7) lib/record/extractor.ex:84: Record.Extractor.read_file/2
    (elixir 1.15.7) lib/record/extractor.ex:50: Record.Extractor.extract_record/2
    lib/sftp_client/ssh_xfer_attr.ex:8: (module)
could not compile dependency :sftp_client, "mix compile" failed. Errors may have been logged above. You can recompile this dependency with "mix deps.compile sftp_client --force", update it with "mix deps.update sftp_client" or clean it with "mix deps.clean sftp_client"

In diving deeper on this, it is because the vendor OTP that is installed with Livebook desktop does not include the src folder for ssh-5.0.1. Is this something that can be changed in a future version?

Does not work with OTP 23

Hi.

I'm using OTP 23 and cannot connect to the server. The problem is here:

https://github.com/i22-digitalagentur/sftp_client/blob/ff9d2f069140a668ad8736e05bb44be941ab0313/lib/sftp_client/key_provider.ex#L11-L15

warning: :ssh_file.add_host_key/3 is undefined or private. Did you mean one of:

      * add_host_key/4

  lib/sftp_client/key_provider.ex:12: SFTPClient.KeyProvider.add_host_key/3

warning: :ssh_file.is_host_key/4 is undefined or private. Did you mean one of:

      * is_host_key/5

  lib/sftp_client/key_provider.ex:15: SFTPClient.KeyProvider.is_host_key/4

The dirty fix by adding an additional parameter to both functions works for me, but I'm not sure how to fix it properly to preserve the compatibility with older OTP versions.

How to make pub/priv key work with this client library

For the traveler who is struggling to get their private key to work:

  1. you need to generate the key-pair in PEM format. For instance, if you using ssh-keygen to create your keys: ssh-keygen -t rsa -m PEM -f id_rsa -C"[email protected]"

  2. ed25519 does not seem to be compatible, at this time, with this client and/or the underlying ssh_sftp client. rsa, however, works.

I found that when I specified a user_dir param with client config, ssh_file would generate a known_hosts file in the user_dir path with an algorithm I didn't even specify. Not sure what's going on with that! So, I am not using this parameter when connecting:

{:ok, conn} =
      SFTPClient.connect(
      host: "localhost",
      port: 2022,
      user: "username",
      private_key_path: "absolute/path/to/private_key",
      sftp_vsn: 2
   )

Good luck.

list_dir() hangs if folder is empty

Hi there,

I’m experiencing something weird. I can connect to SFTP servers just fine, all the operations I’ve tried work, but when I do a list_dir command, it hangs forever if the folder contains no files.

For example this:

    {:ok, conn} = SFTPClient.connect(get_in(state, [:config, :server]))
    {:ok, stat} = SFTPClient.file_info(conn, "in/images/products")
    IO.puts("Stat result #{inspect(stat)}")
    {:ok, list} = SFTPClient.list_dir(conn, "in/images/products")
    IO.inspect(list)
    SFTPClient.disconnect(conn)

If the "in/images/products" folder contains files, everything is hunky-dory, I get a list back.

However, if "in/images/products" has no files (either being completely empty, or having just subfolders), list_dir does not return. The following lines are not executed, I’ve tried waiting at least 10 minutes, much longer than my SFTP timeout, so something’s weird.

Let me know if there’s anything I can do to help reproduce this problem.

Deprecate stream_file/1

As the behavior of stream_file/1 may be confusing (see #8), the function should be deprecated. Instead stream_file!/1 should be used.

SFTPClient.upload_file is way slower than the others functions

First, thanks a lot for SFTPClient !
I'm currently developping a SFTP Client/Server and after using another SFTP library, I give a try to yours at the moment.
I made some tests on a local Windows PC (2 SFTP servers on differents ports exchanging files).
The transfer rates are all very nice (from 23 to 41 Mb/s), streamed or not...
Except for direct upload (SFTPClient.upload_file function) for which the rate falls down to 0,6 to 0.9 Mb/s only.
I isolated yours functions (SFTPClient.connect and SFTPClient.upload_file) in case this fall could come from my own code but with the same results.
Do you have an idea about what could cause that ?
Best regards.

SFTP with private rsa key

HI,
I'm trying to make a SFTP connection with private rsa key.
The service works and i can reach it with CLI and filezilla.

I tried a lot of config combination. The key is a classic id_rsa. the keys are generated with ssh-keygen command on cli.
I tried with something like [host: 'sftp', user: 'user', user_dir: 'keys', private_key_path: 'keys/id_rsa'] and also implementing manually a key_cb but i receive the same error everytime.
The error i get is this one:

10:16:06.539 [info]  Erlang SSH :client 4.9 (OpenSSL 1.1.1d  10 Sep 2019).
Server: 'SSH-2.0-OpenSSH_7.4p1 Debian-10+deb9u4'
Disconnects with code = 14 [RFC4253 11.1]: Unable to connect using the available authentication methods
State = {userauth,client}
Module = ssh_connection_handler, Line = 931.
Details:
  User auth failed for: "user"


{:error,
 %SFTPClient.ConnError{
   message: "Unable to connect using the available authentication methods"
 }}

Looking at the State in the error, it seems that is missing the public/private key authentication method, but i can't find a solution.

Do you have some ideas why this doesn't work? What am I doing wrong?

Thanks

Dialyzer errors on Elixir 1.16

Hi. Thanks for the library!

While updating an application to Elixir 1.16.1 I got a dialyzer error on an sftp_client funcion call. This made me try running dialyzer on this repo (with the same Elixir version) and I got the following

lib/sftp_client.ex:207:no_return
Function upload_file!/3 has no local return.
________________________________________________________________________________
lib/sftp_client/operations/upload_file.ex:31:no_return
Function upload_file!/3 has no local return.
________________________________________________________________________________
lib/sftp_client/operations/upload_file.ex:32:call
The function call will not succeed.

File.stream!(_local_path :: any(), [], 32768)

breaks the contract
(Path.t(), :line | pos_integer(), [stream_mode()]) :: File.Stream.t()

________________________________________________________________________________
done (warnings were emitted)
Halting VM with exit status 2

This happens because on Elixir 1.16.0 the File.stream!(file, options, line_or_bytes) was deprecated and the spec reflects only the latest version. The issue elixir-lang/elixir#13212 has more details and a possible solution.

Support more/all options of :ssh_sftp.start_channel()

Hi there.
Is there any way to pass an additional option to the underlying :ssh_sftp.start_channel/3 call? I have to connect to a pretty outdated server and for that reason I need to modify configuration algorithms via the modify_algorithms option. It seems to be not possible right now since the option has to be listed in the Config struct.

(FunctionClauseError) no function clause matching in :ssh.connect/4

Error to connect to SFTP

The following arguments were given to :ssh.connect/4:
    
        # 1
        'host'
    
        # 2
        "9003"
    
        # 3
        [
          user_interaction: false,
          user: 'admin',
          silently_accept_hosts: true,
          quiet_mode: true,
          password: 'admin!',
          key_cb: {SFTPClient.KeyProvider,
           [private_key_path: nil, private_key_pass_phrase: nil]},
          inet: :inet,
          connect_timeout: 5000
        ]
  
        # 4
        5000

call:

{:ok, conn} = SFTPClient.connect(host: "host", user: "admin", password: "admin", port: "9003")

[Improvement] Resuming file transfer ?

...Such as the "reget" sftp command which is unfortunally not supported (yet) by the Erlang ssh_sftp library(*) but can be REALLY usefull when an upload/download of a huge file suddenly stops.

(*) But the apread and apwrite could help... ^^ ;)

stream_file bangs on SFTPClient.OperationError

Hi! First of all, thanks for the great work!
I bumped into an exception in a non-bang function running a simple download pipeline, when using code like

case SftpClient.stream_file(conn, path) do
  {:ok, data} -> do_something_with(data)
  {:error, reason} -> {:error, reason}
end
** (SFTPClient.OperationError) Operation failed: permission_denied
    (sftp_client) lib/sftp_client/operation_util.ex:37: SFTPClient.OperationUtil.may_bang!/1
    (elixir) lib/stream.ex:1392: anonymous fn/5 in Stream.resource/3
    (elixir) lib/enum.ex:3023: Enum.reverse/1
    (elixir) lib/enum.ex:2668: Enum.to_list/1
    (myapp) lib/worker.ex:433: Worker.download/2

Can't access the sftp at the moment, but the error is pretty clear

I'm using 1.3.5 but didn't see any related diff in last commits
Didn't dig into the problem yet, pretty noob in elixir, but will try to take a look (and in the meanwhile add some defensive try rescue :P )

write_file @doc, @spec and implementation do not match

  @doc """
  Reads a file from the server, and returns the data as String.
  """
  @spec write_file(Conn.t(), Path.t(), String.t() | [String.t()]) ::
          :ok | {:error, SFTPClient.error()}
  def write_file(%Conn{} = conn, path, data) when is_binary(data) do
    write_file(conn, path, [data])
  end
  • returns a string according to @doc
  • returns :ok or {:error, _} according to @spec
  • data is_binary according to implementation
  • data can be String.t or [String.t] according to @spec

read_file crashes on OTP 27

After this removal of a deprecated function ( which was only there for debugging apparently ):
erlang/otp@b39f33e

the following crashes when reading a file:

14:48:47.726 [error] Process #PID<0.269.0> raised an exception
** (FunctionClauseError) no function clause matching in :file_io_server.file_request/2
    (kernel 10.0) file_io_server.erl:227: :file_io_server.file_request(:pid2name, {:state, {:file_descriptor, :prim_file, %{handle: #Reference<0.1131244897.1132068896.124240>, owner: #PID<0.269.0>, r_buffer: #Reference<0.1131244897.1132068865.127900>, r_ahead_size: 0}}, #PID<0.268.0>, #Reference<0.1131244897.1131937793.127901>, "", :binary, :latin1})
    (kernel 10.0) file_io_server.erl:185: :file_io_server.server_loop/1

:file_io_server.file_request(:pid2name, ...) does not exist anymore.

this will show up in a stacktrace as follows:

     stacktrace:
     (ssh 5.2) ssh_sftp.erl:1043: :ssh_sftp.read_file/3
     (sftp_client 2.0.1) lib/sftp_client/operations/read_file.ex:17: SFTPClient.Operations.ReadFile.read_file/2

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.