Comments (7)
simonw
commented on August 19, 2024
Using Litestream from Homebrew: https://litestream.io/install/mac/
% brew install benbjohnson/litestream/litestream
...
==> Upgrading benbjohnson/litestream/litestream
0.3.6 -> 0.3.7
% litestream version
v0.3.7
from s3-credentials.
simonw
commented on August 19, 2024
Created a read-write user against a prefix in a new bucket:
% s3-credentials create litestream-test-20220117 -c --prefix litestream-test/
Created bucket: litestream-test-20220117
Created user: 's3.read-write.litestream-test-20220117' with permissions boundary: 'arn:aws:iam::aws:policy/AmazonS3FullAccess'
Attached policy s3.read-write.litestream-test-20220117 to user s3.read-write.litestream-test-20220117
Created access key for user: s3.read-write.litestream-test-20220117
{
"UserName": "s3.read-write.litestream-test-20220117",
"AccessKeyId": "AKIAWXFXAIOZA3TTU5FX",
"Status": "Active",
"SecretAccessKey": "...",
"CreateDate": "2022-01-17 20:52:00+00:00"
}
% s3-credentials list-user-policies s3.read-write.litestream-test-20220117
User: s3.read-write.litestream-test-20220117
PolicyName: s3.read-write.litestream-test-20220117
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:GetBucketLocation"
],
"Resource": [
"arn:aws:s3:::litestream-test-20220117"
]
},
{
"Effect": "Allow",
"Action": [
"s3:ListBucket"
],
"Resource": [
"arn:aws:s3:::litestream-test-20220117"
],
"Condition": {
"StringLike": {
"s3:prefix": [
"litestream-test/*"
]
}
}
},
{
"Effect": "Allow",
"Action": [
"s3:GetObject",
"s3:GetObjectAcl",
"s3:GetObjectLegalHold",
"s3:GetObjectRetention",
"s3:GetObjectTagging"
],
"Resource": [
"arn:aws:s3:::litestream-test-20220117/litestream-test/*"
]
},
{
"Effect": "Allow",
"Action": [
"s3:PutObject",
"s3:DeleteObject"
],
"Resource": [
"arn:aws:s3:::litestream-test-20220117/litestream-test/*"
]
}
]
}
from s3-credentials.
simonw
commented on August 19, 2024
Running litestream against /tmp/litestream-demo.db:
% sqlite-utils create-database /tmp/litestream-demo.db
% AWS_ACCESS_KEY_ID="AKIAWXFXAIOZA3TTU5FX" \
AWS_SECRET_ACCESS_KEY="..." \
litestream replicate /tmp/litestream-demo.db s3://litestream-test-20220117/litestream-test/litestream-demo.db
litestream v0.3.7
initialized db: /tmp/litestream-demo.db
replicating to: name="s3" type="s3" bucket="litestream-test-20220117" path="litestream-test/litestream-demo.db" region="" endpoint="" sync-interval=1s
/tmp/litestream-demo.db: sync: new generation "0c076b9824ef9805", no generation exists
/tmp/litestream-demo.db(s3): snapshot written 0c076b9824ef9805/00000000
from s3-credentials.
simonw
commented on August 19, 2024
Looks like it's working:
~ % s3-credentials list-bucket litestream-test-20220117
{
"Key": "litestream-test/litestream-demo.db/generations/0c076b9824ef9805/snapshots/00000000.snapshot.lz4",
"LastModified": "2022-01-17 20:56:07+00:00",
"ETag": "\"92414853a857979dc8af748913ed431d\"",
"Size": 298,
"StorageClass": "STANDARD"
}
{
"Key": "litestream-test/litestream-demo.db/generations/0c076b9824ef9805/wal/00000000_00000000.wal.lz4",
"LastModified": "2022-01-17 20:56:07+00:00",
"ETag": "\"05674acf4609b1667fc8a4a39fe9ea99\"",
"Size": 428,
"StorageClass": "STANDARD"
}
{
"Key": "litestream-test/litestream-demo.db/generations/0c076b9824ef9805/wal/00000001_00000000.wal.lz4",
"LastModified": "2022-01-17 20:57:07+00:00",
"ETag": "\"761aab0af62fc9ec8a4e161276aea2b2\"",
"Size": 119,
"StorageClass": "STANDARD"
}
from s3-credentials.
simonw
commented on August 19, 2024
I ran this:
echo '{"name": "Cleo"}' | sqlite-utils insert /tmp/litestream-demo.db dogs -
And now the output of list-bucket is:
{
"Key": "litestream-test/litestream-demo.db/generations/0c076b9824ef9805/snapshots/00000000.snapshot.lz4",
"LastModified": "2022-01-17 20:56:07+00:00",
"ETag": "\"92414853a857979dc8af748913ed431d\"",
"Size": 298,
"StorageClass": "STANDARD"
}
{
"Key": "litestream-test/litestream-demo.db/generations/0c076b9824ef9805/wal/00000000_00000000.wal.lz4",
"LastModified": "2022-01-17 20:56:07+00:00",
"ETag": "\"05674acf4609b1667fc8a4a39fe9ea99\"",
"Size": 428,
"StorageClass": "STANDARD"
}
{
"Key": "litestream-test/litestream-demo.db/generations/0c076b9824ef9805/wal/00000001_00000000.wal.lz4",
"LastModified": "2022-01-17 20:57:07+00:00",
"ETag": "\"761aab0af62fc9ec8a4e161276aea2b2\"",
"Size": 119,
"StorageClass": "STANDARD"
}
{
"Key": "litestream-test/litestream-demo.db/generations/0c076b9824ef9805/wal/00000001_00001038.wal.lz4",
"LastModified": "2022-01-17 21:01:02+00:00",
"ETag": "\"98eff5e58034d570f516de7ee7f62b7b\"",
"Size": 404,
"StorageClass": "STANDARD"
}
{
"Key": "litestream-test/litestream-demo.db/generations/0c076b9824ef9805/wal/00000002_00000000.wal.lz4",
"LastModified": "2022-01-17 21:01:02+00:00",
"ETag": "\"83db0c7d0330a396d055bd4aa9714e52\"",
"Size": 119,
"StorageClass": "STANDARD"
}
from s3-credentials.
simonw
commented on August 19, 2024
Gonna try restoring the database using those same credentials.
https://litestream.io/reference/restore/#replica-url-restore
$ litestream restore -o /tmp/my.db s3://mybkt.litestream.io/my.db
So trying:
% AWS_ACCESS_KEY_ID="AKIAWXFXAIOZA3TTU5FX" \
AWS_SECRET_ACCESS_KEY="..." \
litestream restore -o /tmp/restored.db s3://litestream-test-20220117/litestream-test/litestream-demo.db
% ls -lah /tmp/restored.db
-rw------- 1 simon wheel 16K Jan 17 13:03 /tmp/restored.db
% sqlite-utils dump /tmp/restored.db
BEGIN TRANSACTION;
CREATE TABLE _litestream_lock (id INTEGER);
CREATE TABLE _litestream_seq (id INTEGER PRIMARY KEY, seq INTEGER);
INSERT INTO "_litestream_seq" VALUES(1,2);
CREATE TABLE [dogs] (
[name] TEXT
);
INSERT INTO "dogs" VALUES('Cleo');
COMMIT;
That worked!
from s3-credentials.
simonw
commented on August 19, 2024
I'm going to try replicating to an S3 folder outside of the --prefix, which should not be allowed.
% AWS_ACCESS_KEY_ID="AKIAWXFXAIOZA3TTU5FX" \
AWS_SECRET_ACCESS_KEY="..." \
litestream replicate /tmp/litestream-demo.db s3://litestream-test-20220117/not-allowed/litestream-demo.db
litestream v0.3.7
initialized db: /tmp/litestream-demo.db
replicating to: name="s3" type="s3" bucket="litestream-test-20220117" path="not-allowed/litestream-demo.db" region="" endpoint="" sync-interval=1s
/tmp/litestream-demo.db(s3): monitor error: AccessDenied: Access Denied
status code: 403, request id: G9PYW5FZ8H8YX4TD, host id: m3JAabyPL8/JrN9NLhj4jGzV+wSjF9H3/tYZb0lKI/2OJ9DDdTuUAzIdb45097oaoWTr4Q038eI=
So that worked too! Or rather it broke with the expected error message.
from s3-credentials.
Related Issues (20)
- Way to make an existing bucket public or private HOT 1
- Convert README into documentation website HOT 3
- Make it easier to add extra policy statements HOT 10
- Provide a `--profile` option to allow AWS profile selection HOT 3
- Using --policy should imply --user-permissions-boundary=none HOT 2
- s3-credentials.AmazonS3FullAccess has MaxSessionDuration 3600, should be 12 hours HOT 5
- KeyError if listing bucket with no items returned
- s3-credentials list-buckets --details should show region and website URL, if configured HOT 2
- `s3-credentials get-objects` command HOT 7
- `get-objects/put-objects` `--skip` and `--skip-hash` options HOT 1
- Add the options to add tags to the created resources HOT 3
- `set-public-policy` command HOT 5
- Add s3:PutObjectAcl to write policies HOT 3
- `s3-credentials delete-objects` command HOT 11
- Mysterious test failure in `test_put_objects` HOT 4
- debug-bucket command HOT 3
- Command to make a bucket public HOT 4
- `s3-credentials create name-of-bucket --create-bucket --public` fails with error HOT 4
- `s3-credentials list-bucket --urls` option HOT 1
- CI failures, including ImportError: cannot import name 'mock_s3' from 'moto' HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from s3-credentials.