Git Product home page Git Product logo

s3-sftp-sync's Introduction

S3 SFTP Sync

Syncs files from a SFTP server to an S3 bucket.

Install:

pip install git+https://github.com/CityOfPhiladelphia/s3-sftp-sync.git

Usage:

s3_sftp_sync
s3_sftp_sync --config-file /path/to/config.conf
s3_sftp_sync --logging-file /path/to/logging_config.conf
s3_sftp_sync --config-file /path/to/config.conf --logging-file /path/to/logging_config.conf

Configuration File

sftp:
  hostname: secure-ftp.phila.gov ## SFTP server host or IP
  username: Opaproperty ## SFTP username
  password: password ## SFTP password
  verify_host_key: false ## Wether to verify the SFTP host, set to true if possible
s3:
  bucket: opa-sftp-backup ## bucket to place SFTP data in
  key_prefix: sftp ## key prefix for S3 destination files eg /sftp/original/file/path
  aws_access_key_id: SOMEKEYID ## an AWS access ke id. Omi if you would like to use instance role or ~/.aws
  aws_secret_access_key: SOMEKEY ## an AWS access key secret. Omit of you would like to use insstance role or ~/.aws
incremental_sync:
  last_modified_s3_key: sftp-sync-last-modified ## Key used for incremental sync

Logging Configuration File

Standard python logging file in ini format.

s3-sftp-sync's People

Contributors

awm33 avatar floptical avatar

Stargazers

avatar avatar Ming Fang avatar avatar Christopher Tull avatar Anton Lebedev avatar avatar avatar António Gusmão avatar Rob Cowie avatar Aaron Marburg avatar Pranav avatar

Watchers

Karissa Demi avatar Tim Wisniewski avatar James Cloos avatar Andy Rothwell avatar avatar Tom Swanson avatar Andrew Kennel avatar Charletta Kidd avatar Alexander M. Waldman avatar avatar

Forkers

fergusstrange faberpro christophertull vpol caceres-joseph

s3-sftp-sync's Issues

Issue with config file

Thanks for sharing this!! I am having an issue running it though. I am getting the following error (I also put a redacted version of my config below as well):

/home/ubuntu/.local/lib/python3.8/site-packages/s3_sftp_sync/cli.py:36: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
  config = yaml.load(file)
Starting sync
Traceback (most recent call last):
  File "/home/ubuntu/.local/bin/s3_sftp_sync", line 8, in <module>
    sys.exit(main())
  File "/home/ubuntu/.local/lib/python3.8/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/home/ubuntu/.local/lib/python3.8/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/home/ubuntu/.local/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/ubuntu/.local/lib/python3.8/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/home/ubuntu/.local/lib/python3.8/site-packages/s3_sftp_sync/cli.py", line 117, in main
    response = s3.get_object(Bucket=bucket, Key=key)
  File "/home/ubuntu/.local/lib/python3.8/site-packages/botocore/client.py", line 386, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/home/ubuntu/.local/lib/python3.8/site-packages/botocore/client.py", line 677, in _make_api_call
    request_dict = self._convert_to_request_dict(
  File "/home/ubuntu/.local/lib/python3.8/site-packages/botocore/client.py", line 725, in _convert_to_request_dict
    request_dict = self._serializer.serialize_to_request(
  File "/home/ubuntu/.local/lib/python3.8/site-packages/botocore/validate.py", line 319, in serialize_to_request
    raise ParamValidationError(report=report.generate_report())
botocore.exceptions.ParamValidationError: Parameter validation failed:
Invalid type for parameter Key, value: None, type: <class 'NoneType'>, valid types: <class 'str'>

Config File:

sftp:
  hostname: sftp.sitetouse.io
  username: username
  password: password
  verify_host_key: true

s3:
  bucket: syncbucket1
  key_prefix: /sitetouse
  aws_access_key_id: accesskey
  aws_secret_access_key: secretkey

# remove this section to use full sync
# incremental_sync:
#  last_modified_s3_key: sftp-sync-last-modified ## Key used for incremental sync

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.