kjohnston / pgbackups-archive Goto Github PK

View Code? Open in Web Editor NEW

118.0 118.0 41.0 91 KB

Automates archival of Heroku PGBackups to S3

License: MIT License

Ruby 96.87% Shell 3.13%

pgbackups-archive's People

Contributors

Stargazers

Watchers

pgbackups-archive's Issues

Could not find a valid gem 'heroku'

I ran gem install pgbackups-archive but I am getting ERROR: Could not find a valid gem 'heroku' (<= 3.32, >= 3.28.6) in any repository error.

deprecated heroku gem

Post-install message from heroku:
 !    The `heroku` gem has been deprecated and replaced with the Heroku Toolbelt.
 !    Download and install from: https://toolbelt.heroku.com
 !    For API access, see: https://github.com/heroku/heroku.rb

https://blog.heroku.com/archives/2012/10/15/upgrading-to-the-heroku-toolbelt

NameError: uninitialized constant Heroku::JSPlugin

When I run

heroku run rake pgbackups:archive --trace --account myaccount -a myappname

I get the following error.

Running rake pgbackups:archive --trace attached to terminal... up, run.9289
** Invoke pgbackups:archive (first_time)
** Execute pgbackups:archive
rake aborted!
NameError: uninitialized constant Heroku::JSPlugin

/app/vendor/bundle/ruby/1.9.1/gems/heroku-3.39.4/lib/heroku/command.rb:22:in load' /app/vendor/bundle/ruby/1.9.1/gems/pgbackups-archive-1.0.2/lib/pgbackups-archive/job.rb:16:ininitialize'
/app/vendor/bundle/ruby/1.9.1/gems/pgbackups-archive-1.0.2/lib/pgbackups-archive/job.rb:12:in new' /app/vendor/bundle/ruby/1.9.1/gems/pgbackups-archive-1.0.2/lib/pgbackups-archive/job.rb:12:incall'
/app/vendor/bundle/ruby/1.9.1/gems/pgbackups-archive-1.0.2/lib/tasks/pgbackups_archive.rake:5:in block (2 levels) in <top (required)>' /app/vendor/bundle/ruby/1.9.1/gems/rake-10.3.2/lib/rake/task.rb:240:incall'
/app/vendor/bundle/ruby/1.9.1/gems/rake-10.3.2/lib/rake/task.rb:240:in block in execute' /app/vendor/bundle/ruby/1.9.1/gems/rake-10.3.2/lib/rake/task.rb:235:ineach'
/app/vendor/bundle/ruby/1.9.1/gems/rake-10.3.2/lib/rake/task.rb:235:in execute' /app/vendor/bundle/ruby/1.9.1/gems/rake-10.3.2/lib/rake/task.rb:179:inblock in invoke_with_call_chain'
/app/vendor/ruby-1.9.3/lib/ruby/1.9.1/monitor.rb:211:in mon_synchronize' /app/vendor/bundle/ruby/1.9.1/gems/rake-10.3.2/lib/rake/task.rb:172:ininvoke_with_call_chain'
/app/vendor/bundle/ruby/1.9.1/gems/rake-10.3.2/lib/rake/task.rb:165:in invoke' /app/vendor/bundle/ruby/1.9.1/gems/rake-10.3.2/lib/rake/application.rb:150:ininvoke_task'
/app/vendor/bundle/ruby/1.9.1/gems/rake-10.3.2/lib/rake/application.rb:106:in block (2 levels) in top_level' /app/vendor/bundle/ruby/1.9.1/gems/rake-10.3.2/lib/rake/application.rb:106:ineach'
/app/vendor/bundle/ruby/1.9.1/gems/rake-10.3.2/lib/rake/application.rb:106:in block in top_level' /app/vendor/bundle/ruby/1.9.1/gems/rake-10.3.2/lib/rake/application.rb:115:inrun_with_threads'
/app/vendor/bundle/ruby/1.9.1/gems/rake-10.3.2/lib/rake/application.rb:100:in top_level' /app/vendor/bundle/ruby/1.9.1/gems/rake-10.3.2/lib/rake/application.rb:78:inblock in run'
/app/vendor/bundle/ruby/1.9.1/gems/rake-10.3.2/lib/rake/application.rb:176:in standard_exception_handling' /app/vendor/bundle/ruby/1.9.1/gems/rake-10.3.2/lib/rake/application.rb:75:inrun'
/app/vendor/bundle/ruby/1.9.1/gems/rake-10.3.2/bin/rake:33:in <top (required)>' /app/vendor/bundle/ruby/1.9.1/bin/rake:23:inload'
/app/vendor/bundle/ruby/1.9.1/bin/rake:23:in `

'
Tasks: TOP => pgbackups:archive

Any thoughts?

Issue running heroku run rake pgbackups:archive -a myapp

I'm not sure if this is a problem with my heroku configuration but when I run the rake task to test it using:

heroku run rake pgbackups:archive -a myapp

I get:

Running rake pgbackups:archive attached to terminal... up, run.3069
rake aborted!
URI::InvalidURIError: bad URI(is not URI?):
/app/vendor/bundle/ruby/2.1.0/gems/heroku-3.9.1/lib/heroku/client/pgbackups.rb:9:in `initialize'

Has anyone else had that issue?

@kjohnston is fantastic!

I was just thinking about doing this and like magic, it's done. Thank you for saving me time. A+

RestClient::InternalServerError: 500 Internal Server Error

Hi,
I really like this GEM and want to use it for my scheduled external backups to AWS.
Actually it runs in about 60% of the cases. Sometimes though it crashes and no backup is done.
I am not sure if this is a Heroku problem or something else.

I use:

heroku/toolbelt/3.35.1 (i386-mingw32) ruby/1.9.3
pgbackup:Archive -1.0.2
Windows 7

To reproduce the issue I execute the pgbackups:archive command from my local maschine: C:>heroku run rake pgbackups:archive --trace --app my-archiverHerokuApp

The serverside log looks like this:
2015-05-06T14:16:09.362259+00:00 heroku[run.7542]: State changed from starting to up
2015-05-06T14:16:09.575591+00:00 heroku[run.7542]: Awaiting client
2015-05-06T14:16:09.684460+00:00 heroku[run.7542]: Starting process with command bundle exec rake pgbackups:archive --trace
2015-05-06T14:18:28.456025+00:00 heroku[run.7542]: Process exited with status 1
2015-05-06T14:18:28.469804+00:00 heroku[run.7542]: State changed from up to complete

The Client Side log looks like this:
C:>heroku run rake pgbackups:archive --trace --app my-archiverHerokuApp
Running rake pgbackups:archive --trace attached to terminal... up, run.7542
** Invoke pgbackups:archive (first_time)
** Execute pgbackups:archive
----> long pause here!! <-----
rake aborted!
RestClient::InternalServerError: 500 Internal Server Error
/app/vendor/bundle/ruby/2.0.0/gems/rest-client-1.8.0/lib/restclient/abstract_response.rb:74:in return!' /app/vendor/bundle/ruby/2.0.0/gems/rest-client-1.8.0/lib/restclient/request.rb:495:inprocess_result'
/app/vendor/bundle/ruby/2.0.0/gems/rest-client-1.8.0/lib/restclient/request.rb:421:in block in transmit' /app/vendor/ruby-2.0.0/lib/ruby/2.0.0/net/http.rb:852:instart'
/app/vendor/bundle/ruby/2.0.0/gems/rest-client-1.8.0/lib/restclient/request.rb:413:in transmit' /app/vendor/bundle/ruby/2.0.0/gems/rest-client-1.8.0/lib/restclient/request.rb:176:inexecute'
/app/vendor/bundle/ruby/2.0.0/gems/rest-client-1.8.0/lib/restclient/request.rb:41:in execute' /app/vendor/bundle/ruby/2.0.0/gems/rest-client-1.8.0/lib/restclient/resource.rb:51:inget'
/app/vendor/bundle/ruby/2.0.0/gems/heroku-3.32.0/lib/heroku/client/heroku_postgresql_backups.rb:55:in block (2 levels) in http_get' /app/vendor/bundle/ruby/2.0.0/gems/heroku-3.32.0/lib/heroku/helpers.rb:104:inretry_on_exception'
/app/vendor/bundle/ruby/2.0.0/gems/heroku-3.32.0/lib/heroku/client/heroku_postgresql_backups.rb:54:in block in http_get' /app/vendor/bundle/ruby/2.0.0/gems/heroku-3.32.0/lib/heroku/client/heroku_postgresql_backups.rb:106:inchecking_client_version'
/app/vendor/bundle/ruby/2.0.0/gems/heroku-3.32.0/lib/heroku/client/heroku_postgresql_backups.rb:53:in http_get' /app/vendor/bundle/ruby/2.0.0/gems/heroku-3.32.0/lib/heroku/client/heroku_postgresql_backups.rb:20:intransfers_get'
/app/vendor/bundle/ruby/2.0.0/gems/heroku-3.32.0/lib/heroku/command/pg_backups.rb:383:in poll_transfer' /app/vendor/bundle/ruby/2.0.0/gems/pgbackups-archive-1.0.2/lib/pgbackups-archive/job.rb:37:incapture'
/app/vendor/bundle/ruby/2.0.0/gems/pgbackups-archive-1.0.2/lib/pgbackups-archive/job.rb:22:in call' /app/vendor/bundle/ruby/2.0.0/gems/pgbackups-archive-1.0.2/lib/pgbackups-archive/job.rb:12:incall'
/app/vendor/bundle/ruby/2.0.0/gems/pgbackups-archive-1.0.2/lib/tasks/pgbackups_archive.rake:5:in block (2 levels) in <top (required)>' /app/vendor/bundle/ruby/2.0.0/gems/rake-10.4.2/lib/rake/task.rb:240:incall'
/app/vendor/bundle/ruby/2.0.0/gems/rake-10.4.2/lib/rake/task.rb:240:in block in execute' /app/vendor/bundle/ruby/2.0.0/gems/rake-10.4.2/lib/rake/task.rb:235:ineach'
/app/vendor/bundle/ruby/2.0.0/gems/rake-10.4.2/lib/rake/task.rb:235:in execute' /app/vendor/bundle/ruby/2.0.0/gems/rake-10.4.2/lib/rake/task.rb:179:inblock in invoke_with_call_chain'
/app/vendor/ruby-2.0.0/lib/ruby/2.0.0/monitor.rb:211:in mon_synchronize' /app/vendor/bundle/ruby/2.0.0/gems/rake-10.4.2/lib/rake/task.rb:172:ininvoke_with_call_chain'
/app/vendor/bundle/ruby/2.0.0/gems/rake-10.4.2/lib/rake/task.rb:165:in invoke' /app/vendor/bundle/ruby/2.0.0/gems/rake-10.4.2/lib/rake/application.rb:150:ininvoke_task'
/app/vendor/bundle/ruby/2.0.0/gems/rake-10.4.2/lib/rake/application.rb:106:in block (2 levels) in top_level' /app/vendor/bundle/ruby/2.0.0/gems/rake-10.4.2/lib/rake/application.rb:106:ineach'
/app/vendor/bundle/ruby/2.0.0/gems/rake-10.4.2/lib/rake/application.rb:106:in block in top_level' /app/vendor/bundle/ruby/2.0.0/gems/rake-10.4.2/lib/rake/application.rb:115:inrun_with_threads'
/app/vendor/bundle/ruby/2.0.0/gems/rake-10.4.2/lib/rake/application.rb:100:in top_level' /app/vendor/bundle/ruby/2.0.0/gems/rake-10.4.2/lib/rake/application.rb:78:inblock in run'
/app/vendor/bundle/ruby/2.0.0/gems/rake-10.4.2/lib/rake/application.rb:176:in standard_exception_handling' /app/vendor/bundle/ruby/2.0.0/gems/rake-10.4.2/lib/rake/application.rb:75:inrun'
/app/vendor/bundle/ruby/2.0.0/gems/rake-10.4.2/bin/rake:33:in <top (required)>' /app/vendor/bundle/ruby/2.0.0/bin/rake:23:inload'
/app/vendor/bundle/ruby/2.0.0/bin/rake:23:in `

'
Tasks: TOP => pgbackups:archive

Here are the config:vars I set. (As DATABASE_URL is the default I didn't set "PGBACKUPS_DATABASE")
PGBACKUPS_APP: my-backupappname
PGBACKUPS_AWS_ACCESS_KEY_ID:
PGBACKUPS_AWS_SECRET_ACCESS_KEY:
PGBACKUPS_BUCKET: aws-bucket
PGBACKUPS_REGION: us-east-1

I have no Idea of ruby but could it be a dead lock with a timeout issue?
If anyone could help it would be really great!

Regards,
Matthias

Error requiring rake task in README for non-rails apps

btw I had to put this in my Rakefile:

require "pgbackups-archive"
load "tasks/pgbackups_archive.rake"

mainly writing this in case others come across this issue -- happy to submit a PR to either make the load statement unnecessary OR fix the README. Let me know if/how you'd like me to handle this. Thanks for the gem!

PS rationale for this is I made a separate standalone app to handle backups; I don't like putting this stuff into web-facing apps where possible

ReadMe update

Awesome gem, thanks for creating this. The link to the pro tip on how to create a policy seems to contain outdated information (i.e. the policy format is no longer valid), here's the policy that worked for us:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::bucket-name"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:PutObject"
            ],
            "Resource": [
                "arn:aws:s3:::bucket-name/*"
            ]
        }
    ]
}

Also, the heroku command to add the scheduler add-on has changed to:

heroku addons:create scheduler:standard

Bump or remove direct heroku gem dependency due to CVE warnings

Per the latest (double) indirect CVE warnings, your gem also depends on a vulnerable rest-client version. See heroku/legacy-cli#1470

You may want to wait and see what they do, or consider wrapping the CLI app commands instead maybe (ick, but... what are you gonna do?)

DEPRECATION WARNING: railtie_name is deprecated

Hey,

After add the gem to my app I am getting the following warning.

DEPRECATION WARNING: railtie_name is deprecated and has no effect. (called from require at /usr/ruby1.9.2/lib/ruby/gems/1.9.1/gems/bundler-1.0.7/lib/bundler/runtime.rb:64

I assume its being caused by "lib/pgbackups-archive/railtie.rb" line 7

Would removing the call to railtie_name cause any other problem?

Having trouble running tests

I can't remember how i used to run tests, but i ran into trouble when i tried to run the test suite. Can somebody who knows add instructions to the README?

thx!

Backups failing during Heroku API sunset brownout

Hi,

Heroku is sunsetting their old API version, and right now they are performing a 1-hour brownout to help uncover cases that will break when the API is really sunset. Turns out pgbackups-archive might be one such breakage.

I manually ran bundle exec rake pgbackups:archive on my app, intending to test it during this brownout, and it failed with the following errors:

rake aborted!
Heroku::API::Errors::ErrorWithResponse: Expected(200) <=> Actual(410 Gone)

body: "{\"id\":\"gone\",\"error\":\"This version of the API has been Sunset.\\nPlease see https://devcenter.heroku.com/changelog-items/1147 for more information.\\n\"}"
/app/vendor/bundle/ruby/2.4.0/gems/excon-0.55.0/lib/excon/middlewares/expects.rb:7:in `response_call'
/app/vendor/bundle/ruby/2.4.0/gems/excon-0.55.0/lib/excon/middlewares/response_parser.rb:9:in `response_call'
/app/vendor/bundle/ruby/2.4.0/gems/excon-0.55.0/lib/excon/connection.rb:388:in `response'
/app/vendor/bundle/ruby/2.4.0/gems/excon-0.55.0/lib/excon/connection.rb:252:in `request'
/app/vendor/bundle/ruby/2.4.0/gems/heroku-api-0.3.23/lib/heroku/api.rb:83:in `request'
/app/vendor/bundle/ruby/2.4.0/gems/heroku-3.32.0/lib/heroku/auth.rb:20:in `request'
/app/vendor/bundle/ruby/2.4.0/gems/heroku-api-0.3.23/lib/heroku/api/attachments.rb:6:in `get_attachments'
/app/vendor/bundle/ruby/2.4.0/gems/heroku-3.32.0/lib/heroku/helpers/heroku_postgresql.rb:99:in `app_attachments'
/app/vendor/bundle/ruby/2.4.0/gems/heroku-3.32.0/lib/heroku/helpers/heroku_postgresql.rb:104:in `hpg_databases'
/app/vendor/bundle/ruby/2.4.0/gems/heroku-3.32.0/lib/heroku/helpers/heroku_postgresql.rb:159:in `hpg_resolve'
/app/vendor/bundle/ruby/2.4.0/gems/heroku-3.32.0/lib/heroku/helpers/heroku_postgresql.rb:58:in `resolve'
/app/vendor/bundle/ruby/2.4.0/gems/pgbackups-archive-1.1.0/lib/pgbackups-archive/job.rb:35:in `capture'
/app/vendor/bundle/ruby/2.4.0/gems/pgbackups-archive-1.1.0/lib/pgbackups-archive/job.rb:22:in `call'
/app/vendor/bundle/ruby/2.4.0/gems/pgbackups-archive-1.1.0/lib/pgbackups-archive/job.rb:12:in `call'
/app/vendor/bundle/ruby/2.4.0/gems/pgbackups-archive-1.1.0/lib/tasks/pgbackups_archive.rake:5:in `block (2 levels) in <top (required)>'
/app/vendor/bundle/ruby/2.4.0/gems/bugsnag-5.2.0/lib/bugsnag/rake.rb:12:in `execute_with_bugsnag'
/app/vendor/bundle/ruby/2.4.0/gems/rake-12.0.0/exe/rake:27:in `<top (required)>'
/app/vendor/bundle/ruby/2.4.0/gems/bundler-1.13.7/lib/bundler/cli/exec.rb:74:in `load'
/app/vendor/bundle/ruby/2.4.0/gems/bundler-1.13.7/lib/bundler/cli/exec.rb:74:in `kernel_load'
/app/vendor/bundle/ruby/2.4.0/gems/bundler-1.13.7/lib/bundler/cli/exec.rb:27:in `run'
/app/vendor/bundle/ruby/2.4.0/gems/bundler-1.13.7/lib/bundler/cli.rb:332:in `exec'
/app/vendor/bundle/ruby/2.4.0/gems/bundler-1.13.7/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
/app/vendor/bundle/ruby/2.4.0/gems/bundler-1.13.7/lib/bundler/vendor/thor/lib/thor/invocation.rb:126:in `invoke_command'
/app/vendor/bundle/ruby/2.4.0/gems/bundler-1.13.7/lib/bundler/vendor/thor/lib/thor.rb:359:in `dispatch'
/app/vendor/bundle/ruby/2.4.0/gems/bundler-1.13.7/lib/bundler/cli.rb:20:in `dispatch'
/app/vendor/bundle/ruby/2.4.0/gems/bundler-1.13.7/lib/bundler/vendor/thor/lib/thor/base.rb:440:in `start'
/app/vendor/bundle/ruby/2.4.0/gems/bundler-1.13.7/lib/bundler/cli.rb:11:in `start'
/app/vendor/bundle/ruby/2.4.0/gems/bundler-1.13.7/exe/bundle:34:in `block in <top (required)>'
/app/vendor/bundle/ruby/2.4.0/gems/bundler-1.13.7/lib/bundler/friendly_errors.rb:100:in `with_friendly_errors'
/app/vendor/bundle/ruby/2.4.0/gems/bundler-1.13.7/exe/bundle:26:in `<top (required)>'
/app/bin/bundle:3:in `load'
/app/bin/bundle:3:in `<main>'
Excon::Error::Gone: Expected(200) <=> Actual(410 Gone)

body: "{\"id\":\"gone\",\"error\":\"This version of the API has been Sunset.\\nPlease see https://devcenter.heroku.com/changelog-items/1147 for more information.\\n\"}"
/app/vendor/bundle/ruby/2.4.0/gems/excon-0.55.0/lib/excon/middlewares/expects.rb:7:in `response_call'
/app/vendor/bundle/ruby/2.4.0/gems/excon-0.55.0/lib/excon/middlewares/response_parser.rb:9:in `response_call'
/app/vendor/bundle/ruby/2.4.0/gems/excon-0.55.0/lib/excon/connection.rb:388:in `response'
/app/vendor/bundle/ruby/2.4.0/gems/excon-0.55.0/lib/excon/connection.rb:252:in `request'
/app/vendor/bundle/ruby/2.4.0/gems/heroku-api-0.3.23/lib/heroku/api.rb:83:in `request'
/app/vendor/bundle/ruby/2.4.0/gems/heroku-3.32.0/lib/heroku/auth.rb:20:in `request'
/app/vendor/bundle/ruby/2.4.0/gems/heroku-api-0.3.23/lib/heroku/api/attachments.rb:6:in `get_attachments'
/app/vendor/bundle/ruby/2.4.0/gems/heroku-3.32.0/lib/heroku/helpers/heroku_postgresql.rb:99:in `app_attachments'
/app/vendor/bundle/ruby/2.4.0/gems/heroku-3.32.0/lib/heroku/helpers/heroku_postgresql.rb:104:in `hpg_databases'
/app/vendor/bundle/ruby/2.4.0/gems/heroku-3.32.0/lib/heroku/helpers/heroku_postgresql.rb:159:in `hpg_resolve'
/app/vendor/bundle/ruby/2.4.0/gems/heroku-3.32.0/lib/heroku/helpers/heroku_postgresql.rb:58:in `resolve'
/app/vendor/bundle/ruby/2.4.0/gems/pgbackups-archive-1.1.0/lib/pgbackups-archive/job.rb:35:in `capture'
/app/vendor/bundle/ruby/2.4.0/gems/pgbackups-archive-1.1.0/lib/pgbackups-archive/job.rb:22:in `call'
/app/vendor/bundle/ruby/2.4.0/gems/pgbackups-archive-1.1.0/lib/pgbackups-archive/job.rb:12:in `call'
/app/vendor/bundle/ruby/2.4.0/gems/pgbackups-archive-1.1.0/lib/tasks/pgbackups_archive.rake:5:in `block (2 levels) in <top (required)>'
/app/vendor/bundle/ruby/2.4.0/gems/bugsnag-5.2.0/lib/bugsnag/rake.rb:12:in `execute_with_bugsnag'
/app/vendor/bundle/ruby/2.4.0/gems/rake-12.0.0/exe/rake:27:in `<top (required)>'
/app/vendor/bundle/ruby/2.4.0/gems/bundler-1.13.7/lib/bundler/cli/exec.rb:74:in `load'
/app/vendor/bundle/ruby/2.4.0/gems/bundler-1.13.7/lib/bundler/cli/exec.rb:74:in `kernel_load'
/app/vendor/bundle/ruby/2.4.0/gems/bundler-1.13.7/lib/bundler/cli/exec.rb:27:in `run'
/app/vendor/bundle/ruby/2.4.0/gems/bundler-1.13.7/lib/bundler/cli.rb:332:in `exec'
/app/vendor/bundle/ruby/2.4.0/gems/bundler-1.13.7/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
/app/vendor/bundle/ruby/2.4.0/gems/bundler-1.13.7/lib/bundler/vendor/thor/lib/thor/invocation.rb:126:in `invoke_command'
/app/vendor/bundle/ruby/2.4.0/gems/bundler-1.13.7/lib/bundler/vendor/thor/lib/thor.rb:359:in `dispatch'
/app/vendor/bundle/ruby/2.4.0/gems/bundler-1.13.7/lib/bundler/cli.rb:20:in `dispatch'
/app/vendor/bundle/ruby/2.4.0/gems/bundler-1.13.7/lib/bundler/vendor/thor/lib/thor/base.rb:440:in `start'
/app/vendor/bundle/ruby/2.4.0/gems/bundler-1.13.7/lib/bundler/cli.rb:11:in `start'
/app/vendor/bundle/ruby/2.4.0/gems/bundler-1.13.7/exe/bundle:34:in `block in <top (required)>'
/app/vendor/bundle/ruby/2.4.0/gems/bundler-1.13.7/lib/bundler/friendly_errors.rb:100:in `with_friendly_errors'
/app/vendor/bundle/ruby/2.4.0/gems/bundler-1.13.7/exe/bundle:26:in `<top (required)>'
/app/bin/bundle:3:in `load'
/app/bin/bundle:3:in `<main>'
Tasks: TOP => pgbackups:archive
(See full trace by running task with --trace)

I'm running version 1.1.0 of pgbackups-archive. What needs to happen to fix this?

Thanks! :-)

Bump rake version in Gemspec

Looks like rake is locked at "~> 0.9.2" in the pgbackups-archive.gemspec. There's a newer version of rake out (v10.1.0). Could we bump the dependency, or would something go terribly wrong if we did so?

Backup error from Heroku to AWS S3

Heroku::API::Errors::Forbidden: Expected(200) <=> Actual(403 Forbidden)

body: "{"id":"forbidden","error":"You do not have access to the app 'myapp'."}"/app/vendor/bundle/ruby/2.1.0/gems/excon-0.45.4/lib/excon/middlewares/expects.rb:6:in `response_call'

Is there any plan to add automatic deletion of older backups?

I'm managing a 100mb backup. If I plan to do it every hour, it's over 2GB a day. If I don't manually delete the old ones once in a while it can easily become costly.

Is there any plan to add such feature i.e. configuring to keep only last n backups?

Error accessing S3 bucket when PGBACKUPS_REGION is set

When the PGBACKUPS_REGION is set the rake task fails because it tries to use bucketname.s3-us-west-2.amazonaws.com and AWS responds with a 301 and an error that you should use bucketname.s3.amazonaws.com instead.

rake aborted!
Expected(200) <=> Actual(301 Moved Permanently)
  request => {:chunk_size=>1048576, :connect_timeout=>60, :headers=>{"Date"=>"Tue, 23 Oct 2012 18:19:47 +0000", "Authorization"=>"[removed]", "Host"=>"[removed].s3-us-west-2.amazonaws.com:443"}, :instrumentor_name=>"excon", :mock=>false, :nonblock=>true, :read_timeout=>60, :retry_limit=>4, :ssl_ca_file=>"/app/vendor/bundle/ruby/1.9.1/gems/excon-0.16.7/data/cacert.pem", :ssl_verify_peer=>true, :write_timeout=>60, :host=>"[removed].s3-us-west-2.amazonaws.com", :host_port=>"[removed].s3-us-west-2.amazonaws.com:443", :path=>"/", :port=>"443", :query=>{}, :scheme=>"https", :expects=>200, :idempotent=>true, :method=>"GET", :response_block=>#<Proc:0x00000006d0e750@/app/vendor/bundle/ruby/1.9.1/gems/fog-1.6.0/lib/fog/core/connection.rb:16 (lambda)>}
  response => #<Excon::Response:0x00000006cf97b0 @body="<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>PermanentRedirect</Code><Message>The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.</Message><RequestId>[removed]</RequestId><Bucket>[removed]</Bucket><HostId>[removed]</HostId><Endpoint>s3.amazonaws.com</Endpoint></Error>", @headers={"x-amz-request-id"=>"[removed]", "x-amz-id-2"=>"[removed]", "Content-Type"=>"application/xml", "Transfer-Encoding"=>"chunked", "Date"=>"Tue, 23 Oct 2012 18:19:49 GMT", "Server"=>"AmazonS3"}, @status=301>

If fixed the issue for me by removing that setting: heroku config:remove PGBACKUPS_REGION

r14 memory issues

Each time i run the rake task, I get an R14 out of memory. It started happening when my database got to be around 300M. I'm not exactly sure where it's happening but after discussing with heroku support, it sounds like it's on the fog upload to S3. Has anyone else had similar issues? Are others uploading databases larger than 300M?

thx!

Can't make it work

Hey,

First issue here, waou.
I'm really interested into using this gem/app but unfortunately, I can't make it work.

Quick steps:
I've followed all the points mentionned in the readme.md as with the external app.

I still get this error:

$ backup run rake pgbackups:archive      
Running `rake pgbackups:archive` attached to terminal... up, run.8035
** [NewRelic][09/19/13 21:57:52 +0000 0db350d9-dfb0-4e35-8042-cd74310ccf8f (2)] INFO : Starting the New Relic agent in "development" environment.
** [NewRelic][09/19/13 21:57:52 +0000 0db350d9-dfb0-4e35-8042-cd74310ccf8f (2)] INFO : To prevent agent startup add a NEWRELIC_ENABLE=false environment variable or modify the "development" section of your newrelic.yml.
** [NewRelic][09/19/13 21:57:52 +0000 0db350d9-dfb0-4e35-8042-cd74310ccf8f (2)] INFO : Reading configuration from config/newrelic.yml
** [NewRelic][09/19/13 21:57:52 +0000 0db350d9-dfb0-4e35-8042-cd74310ccf8f (2)] INFO : Disabling the Request Sampler.
** [NewRelic][09/19/13 21:57:52 +0000 0db350d9-dfb0-4e35-8042-cd74310ccf8f (2)] WARN : Agent is configured not to use SSL when communicating with New Relic's servers
** [NewRelic][09/19/13 21:57:52 +0000 0db350d9-dfb0-4e35-8042-cd74310ccf8f (2)] INFO : Environment: development
** [NewRelic][09/19/13 21:57:52 +0000 0db350d9-dfb0-4e35-8042-cd74310ccf8f (2)] INFO : No known dispatcher detected.
** [NewRelic][09/19/13 21:57:52 +0000 0db350d9-dfb0-4e35-8042-cd74310ccf8f (2)] INFO : Application: copass-backup
** [NewRelic][09/19/13 21:57:52 +0000 0db350d9-dfb0-4e35-8042-cd74310ccf8f (2)] INFO : Installing Net instrumentation
** [NewRelic][09/19/13 21:57:52 +0000 0db350d9-dfb0-4e35-8042-cd74310ccf8f (2)] INFO : Finished instrumentation
** [NewRelic][09/19/13 21:57:52 +0000 0db350d9-dfb0-4e35-8042-cd74310ccf8f (2)] INFO : Reporting to: https://rpm.newrelic.com/accounts/430318/applications/2119748
** [NewRelic][09/19/13 21:57:52 +0000 0db350d9-dfb0-4e35-8042-cd74310ccf8f (2)] INFO : Starting the New Relic agent in "development" environment.
** [NewRelic][09/19/13 21:57:52 +0000 0db350d9-dfb0-4e35-8042-cd74310ccf8f (2)] INFO : To prevent agent startup add a NEWRELIC_ENABLE=false environment variable or modify the "development" section of your newrelic.yml.
** [NewRelic][09/19/13 21:57:52 +0000 0db350d9-dfb0-4e35-8042-cd74310ccf8f (2)] INFO : Reading configuration from config/newrelic.yml
.rake aborted!
404 Resource Not Found
/app/vendor/bundle/ruby/2.0.0/gems/rest-client-1.6.7/lib/restclient/abstract_response.rb:48:in `return!'
/app/vendor/bundle/ruby/2.0.0/gems/rest-client-1.6.7/lib/restclient/request.rb:230:in `process_result'
/app/vendor/bundle/ruby/2.0.0/gems/rest-client-1.6.7/lib/restclient/request.rb:178:in `block in transmit'
/app/vendor/bundle/ruby/2.0.0/gems/rest-client-1.6.7/lib/restclient/request.rb:172:in `transmit'
/app/vendor/bundle/ruby/2.0.0/gems/rest-client-1.6.7/lib/restclient/request.rb:64:in `execute'
/app/vendor/bundle/ruby/2.0.0/gems/rest-client-1.6.7/lib/restclient/request.rb:33:in `execute'
/app/vendor/bundle/ruby/2.0.0/gems/rest-client-1.6.7/lib/restclient/resource.rb:51:in `get'
/app/vendor/bundle/ruby/2.0.0/gems/heroku-2.34.0/lib/heroku/client/pgbackups.rb:70:in `block in get'
/app/vendor/bundle/ruby/2.0.0/gems/heroku-2.34.0/lib/heroku/client/pgbackups.rb:93:in `check_errors'
/app/vendor/bundle/ruby/2.0.0/gems/heroku-2.34.0/lib/heroku/client/pgbackups.rb:69:in `get'
/app/vendor/bundle/ruby/2.0.0/gems/heroku-2.34.0/lib/heroku/client/pgbackups.rb:36:in `get_transfer'
/app/vendor/bundle/ruby/2.0.0/gems/pgbackups-archive-0.2.0/lib/pgbackups-archive/heroku/client/pgbackups_archive.rb:33:in `capture'
/app/vendor/bundle/ruby/2.0.0/gems/pgbackups-archive-0.2.0/lib/pgbackups-archive/heroku/client/pgbackups_archive.rb:10:in `perform'
/app/Rakefile:9:in `call'
(eval):3:in `block in call_with_newrelic_transaction_trace'
/app/vendor/bundle/ruby/2.0.0/gems/newrelic_rpm-3.6.3.111/lib/new_relic/agent/instrumentation/controller_instrumentation.rb:318:in `perform_action_with_newrelic_trace'
(eval):2:in `call_with_newrelic_transaction_trace'
/app/Rakefile:19:in `block (2 levels) in <top (required)>'
/app/vendor/bundle/ruby/2.0.0/gems/rake-10.0.4/lib/rake/task.rb:246:in `call'
/app/vendor/bundle/ruby/2.0.0/gems/rake-10.0.4/lib/rake/task.rb:246:in `block in execute'
/app/vendor/bundle/ruby/2.0.0/gems/rake-10.0.4/lib/rake/task.rb:241:in `each'
/app/vendor/bundle/ruby/2.0.0/gems/rake-10.0.4/lib/rake/task.rb:241:in `execute'
/app/vendor/bundle/ruby/2.0.0/gems/rake-10.0.4/lib/rake/task.rb:184:in `block in invoke_with_call_chain'
/app/vendor/bundle/ruby/2.0.0/gems/rake-10.0.4/lib/rake/task.rb:177:in `invoke_with_call_chain'
/app/vendor/bundle/ruby/2.0.0/gems/rake-10.0.4/lib/rake/task.rb:170:in `invoke'
/app/vendor/bundle/ruby/2.0.0/gems/rake-10.0.4/lib/rake/application.rb:143:in `invoke_task'
/app/vendor/bundle/ruby/2.0.0/gems/rake-10.0.4/lib/rake/application.rb:101:in `block (2 levels) in top_level'
/app/vendor/bundle/ruby/2.0.0/gems/rake-10.0.4/lib/rake/application.rb:101:in `each'
/app/vendor/bundle/ruby/2.0.0/gems/rake-10.0.4/lib/rake/application.rb:101:in `block in top_level'
/app/vendor/bundle/ruby/2.0.0/gems/rake-10.0.4/lib/rake/application.rb:110:in `run_with_threads'
/app/vendor/bundle/ruby/2.0.0/gems/rake-10.0.4/lib/rake/application.rb:95:in `top_level'
/app/vendor/bundle/ruby/2.0.0/gems/rake-10.0.4/lib/rake/application.rb:73:in `block in run'
/app/vendor/bundle/ruby/2.0.0/gems/rake-10.0.4/lib/rake/application.rb:160:in `standard_exception_handling'
/app/vendor/bundle/ruby/2.0.0/gems/rake-10.0.4/lib/rake/application.rb:70:in `run'
Tasks: TOP => pgbackups:archive
(See full trace by running task with --trace)
** [NewRelic][09/19/13 21:57:55 +0000 0db350d9-dfb0-4e35-8042-cd74310ccf8f (2)] INFO : Starting Agent shutdown

I have tried with ENV variables configs, and with straight newrelic.yml config. Here's my config/newrelic.yml file

production:
  error_collector:
    capture_source: true
    enabled: true
    ignore_errors: ActionController::RoutingError
  apdex_t: 0.5
  ssl: false
  monitor_mode: true
  license_key: <licence_key>
  developer_mode: false
  app_name: <app_name>
  transaction_tracer:
    record_sql: obfuscated
    enabled: true
    stack_trace_threshold: 0.5
    transaction_threshold: apdex_f
  capture_params: false
  log_level: info
development:
  error_collector:
    capture_source: true
    enabled: true
    ignore_errors: ActionController::RoutingError
  apdex_t: 0.5
  ssl: false
  monitor_mode: true
  license_key: <licence_key>
  developer_mode: true
  app_name: <licence_key>
  transaction_tracer:
    record_sql: obfuscated
    enabled: true
    stack_trace_threshold: 0.5
    transaction_threshold: apdex_f
  capture_params: false
  log_level: info

And maybe I made a mistake in ENV['PGBACKUPS_DATABASE_URL'] I've set my ENV['PGBACKUPS_URL'] from the original app. Is that right?

And my last question is about bucket CORS, shouldn't I set something specific here.
I've allowed GET and POST to the bucket from http://app_name.herokuapp.com but it could not make any sense.

If you could help, it would be awesome, I'm really interested into having manual backups.

incorrect info in README

I believe this is incorrect:

Note: You can use paid-for versions of pgbackups if you'd like, however the dev and basic
database offerings only support the free (plus) version.

I'm running auto-month on both dev and basic plans, and all pgbackups plans are listed as being free.

PGBACKUPS_APP in file path

Could you use the app name (PGBACKUPS_APP in the file path? (here: https://github.com/kjohnston/pgbackups-archive/blob/master/lib/pgbackups-archive/job.rb#L98, preferably in second position).
It makes it possible to have a central backup bucket (I would use that because I have several deployments of the same app, all running in production).

Failure expiring backup

Hi there,

We recently upgraded to 1.0.2 and everything worked great, until we reached 30 backups when the PgbackupsArchive::Job#expire method kicks in. We're now getting NoMethodError: undefined method 'backup_num' for #<Heroku::Command::Pg:0x007eff79f54900> from lib/pgbackups-archive/job.rb:69. We're using heroku version 3.32.0. Setting PGBACKUPS_KEEP to a larger number got things going again.

I'll dig into this more over the weekend/next week, but I thought I'd open this issue to see if anyone's seen it before.

Thanks!

Support for Google Storage

I used to use the heroku_cloud_storage gem for backups to s3 and Google Storage, but that gem has since been abandoned and no longer works due to the change in Heroku's backup api. This gem looks promising but it does not appear to support backups to Google Storage. Would the authors of this gem be interested in having it also support backups to Google Storage?

You can find their implementation of Google Storage backups here: https://github.com/kamui/heroku_cloud_backup/blob/master/lib/heroku_cloud_backup.rb#L68

another approach using heroku toolbelt and aws-cli

@kjohnston - I wanted to run this by you. I had some issues with pgbackups-archive and ran it by the heroku folks. They said pgbackups-archive was on an unsupported version of the heroku api. They recommended we use the heroku toolbelt pgbackups:capture. So, i created a simple project that leverages the heroku toolbelt and aws cli to capture the backup and copy it to your s3 bucket.

https://github.com/kbaum/heroku-database-backups

All it really is a heroku project with a bash script that you can schedule to run as often as you like. There is very little code though there is room for much improvement i am sure.

Let me know what you think. Apologize in advanced for creating an issue for this but i couldn't figure out a better way to get in touch with you.

thx!

Where do you get the collaborator api key?

I can't figure out where this would be?

heroku config:add HEROKU_API_KEY="collaborator-api-key"

undefined method `closed?' for Hash

Running the rake task on Heroku gives me the following error:

undefined method `closed?' for #<Hash:0x0000000575e680>
/app/vendor/bundle/ruby/1.9.1/gems/rest-client-1.6.7/lib/restclient/payload.rb:109:in `close'
/app/vendor/bundle/ruby/1.9.1/gems/rest-client-1.6.7/lib/restclient/request.rb:66:in `ensure in execute'
/app/vendor/bundle/ruby/1.9.1/gems/rest-client-1.6.7/lib/restclient/request.rb:66:in `execute'
/app/vendor/bundle/ruby/1.9.1/gems/rest-client-1.6.7/lib/restclient/request.rb:33:in `execute'
/app/vendor/bundle/ruby/1.9.1/gems/rest-client-1.6.7/lib/restclient/resource.rb:67:in `post'
/app/vendor/bundle/ruby/1.9.1/gems/heroku-2.33.0/lib/heroku/client/pgbackups.rb:78:in `block in post'
/app/vendor/bundle/ruby/1.9.1/gems/heroku-2.33.0/lib/heroku/client/pgbackups.rb:93:in `check_errors'
/app/vendor/bundle/ruby/1.9.1/gems/heroku-2.33.0/lib/heroku/client/pgbackups.rb:77:in `post'
/app/vendor/bundle/ruby/1.9.1/gems/heroku-2.33.0/lib/heroku/client/pgbackups.rb:26:in `create_transfer'
/app/vendor/bundle/ruby/1.9.1/gems/pgbackups-archive-0.0.3/lib/pgbackups-archive/heroku/client/pgbackups_archive.rb:15:in `capture'
/app/vendor/bundle/ruby/1.9.1/gems/pgbackups-archive-0.0.3/lib/tasks/pgbackups_archive.rake:5:in `block (2 levels) in <top (required)>'

Not sure how to diagnose that - any ideas?

Recommend require: false in Gemfile

Hi there, thanks for the great gem. I just wanted to point out that adding the gem to the Gemfile as you recommend added 6 seconds to our boot time on Heroku. We eventually changed it to this: gem 'pgbackups-archive', require: false

We then just made our own rake task like so:

namespace :pgbackups do

  desc "Perform a pgbackups backup then archive to S3."
  task :archive do
    require 'pgbackups-archive'
    Heroku::Client::PgbackupsArchive.perform
  end

end

Thought this might inform a slight update to the codebase since adding 6 seconds to a boot on Heroku (mostly because of the fog gem) is a pretty big hit against the 60 seconds allowed.

Best,
Matt

Getting bad URI error in rake task

Hey - I tried this with and without the PGBACKUPS_DATABASE_URL, and also tried changing the database url manually to have an https:// instead of postgres:// in front - any idea what's causing this?

Tried:
PGBACKUPS_DATABASE_URL=DATABASE_URL
PGBACKUPS_DATABASE_URL=https://user:[email protected]:5512/username
(also tried having no pgbackups_database_url)

output

** Invoke pgbackups:archive (first_time)
** Execute pgbackups:archive
rake aborted!
bad URI(is not URI?):
/app/vendor/ruby-1.9.2/lib/ruby/1.9.1/uri/common.rb:156:in split' /app/vendor/ruby-1.9.2/lib/ruby/1.9.1/uri/common.rb:174:inparse'
/app/vendor/ruby-1.9.2/lib/ruby/1.9.1/uri/common.rb:628:in parse' /app/vendor/bundle/ruby/1.9.1/gems/heroku-2.32.14/lib/heroku/client/pgbackups.rb:9:ininitialize'

rubyzip Gemfile compatibility issue?

Hi - looks like an awesome tool you've got here. I was planning on automating this myself but why reinvent the wheel?

My Gemfile contains:

gem "pgbackups-archive", "1.0.0"

And with bundle install command, I get:

Bundler could not find compatible versions for gem "rubyzip":
  In Gemfile:
    pgbackups-archive (= 1.0.0) ruby depends on
      heroku (>= 3.28.6, ~> 3.28) ruby depends on
        rubyzip (= 0.9.9) ruby

    roo (>= 0) ruby depends on
      rubyzip (1.1.6)

This is a Rails 4.1.x application. Heroku Cedar 14.

Compatability with pg:backups

The old pgbackups addon has been deprecated. Is this gem compatible with the new pg:backups?

I'm having trouble because the PGBACKUPS_URL is not defined.

kjohnston / pgbackups-archive Goto Github PK

pgbackups-archive's People

Contributors

Stargazers

Watchers

Forkers

pgbackups-archive's Issues

Recommend Projects

Recommend Topics

Recommend Org