Mozilla Sync Storage built with Rust.
- Rust stable
- MySQL 5.7 (or compatible)
-* libmysqlclient (
brew install mysql
on macOS,apt-get install libmysqlclient-dev
on Ubuntu) - Go
- Cmake
- Pkg-config
- Openssl
Depending on your OS, you may also need to install libgrpcdev
,
libcurl4-openssl-dev
, and protobuf-compiler-grpc
. Note: if the
code complies cleanly, but generates a Segmentation Fault within
Sentry init, you probably are missing libcurl4-openssl-dev
.
- Follow the instructions below to use either MySQL or Spanner as your DB.
- Now
cp config/local.example.toml config/local.toml
. Openconfig/local.toml
and make sure you have the desired settings configured. For a complete list of available configuration options, check out docs/config.md. make run_local
starts the server in debug mode, using your newlocal.toml
file for config options. Or, simplycargo run
with your own config options provided as env vars.- Visit
http://localhost:8000/__heartbeat__
to make sure the server is running.
Durable sync needs only a valid mysql DSN in order to set up connections to a MySQL database. The database can be local and is usually specified with a DSN like:
mysql://_user_:_password_@_host_/_database_
To setup a fresh MySQL DB and user: (mysql -u root
):
```
CREATE USER "sample_user"@"localhost" IDENTIFIED BY "sample_password";
CREATE DATABASE syncstorage_rs;
GRANT ALL PRIVILEGES on syncstorage_rs.* to sample_user@localhost;
```
Spanner requires a key in order to access the database. It's important that you know which keys have access to the spanner database. Contact your administrator to find out. One you know the key, log into the Google Cloud Console Service Accounts page. Be sure to select the correct project.
- Locate the email identifier of the access key and pick the vertical dot menu at the far right of the row.
- Select "Create Key" from the pop-up menu.
- Select "JSON" from the Dialog Box.
A proper key file will be downloaded to your local directory. It's important to safeguard that key file. For this example, we're going to name the file
service-account.json
.
The proper key file is in JSON format. An example file is provided below, with private information replaced by "...
"
{
"type": "service_account",
"project_id": "...",
"private_key_id": "...",
"private_key": "...",
"client_email": "...",
"client_id": "...",
"auth_uri": "https://accounts.google.com/o/oauth2/auth",
"token_uri": "https://oauth2.googleapis.com/token",
"auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
"client_x509_cert_url": "..."
}
Note, that unlike MySQL, there is no automatic migrations facility. Currently Spanner schema must be hand edited and modified.
To point to a GCP hosted Spanner instance from your local machine, follow these steps:
- Download the key file as shown above.
- Open
local.toml
and replacedatabase_url
with a link to your spanner instance. - Open the Makefile and ensure you've correctly set you
PATH_TO_GRPC_CERT
. make run_spanner
.- Visit
http://localhost:8000/__heartbeat__
to make sure the server is running.
This currently requires access to the mozilla-rust-sdk repo. If you don't have it, this will be made public soon; we'll update the README here when that happens.
- Make sure you have Docker installed locally.
- Copy the contents of mozilla-rust-sdk into top level root dir here.
- Change cargo.toml mozilla-rust-sdk entry to point to
"path = "mozilla-rust-sdk/googleapis-raw"
instead of the parent dir. - Comment out the
image
value undersyncstorage-rs
in docker-compose.yml, and add this instead:build: context: .
- Adjust the MySQL db creds in docker-compose.yml to match your local setup.
make docker_start
- You can verify it's working by visiting localhost:8000/heartbeat
This will walk you through the steps to connect this project to your local copy of Firefox.
-
Follow the steps outlined above for running this project using MySQL.
-
Setup a local copy of syncserver, with a few special changes to syncserver.ini; make sure that you're using the following values (in addition to all of the other defaults):
[server:main] port = 5000 [syncserver] public_url = http://localhost:5000/ # This value needs to match your "master_secret" for syncstorage-rs! secret = INSERT_SECRET_KEY_HERE [tokenserver] node_url = http://localhost:8000 sqluri = pymysql://sample_user:[email protected]/syncstorage_rs [endpoints] sync-1.5 = "http://localhost:8000/1.5/1"```
-
In Firefox, go to
about:config
. Changeidentity.sync.tokenserver.uri
tohttp://localhost:5000/token/1.0/sync/1.5
. -
Restart Firefox. Now, try syncing. You should see new BSOs in your local MySQL instance.
- If you want to connect to the existing Sentry project for local development, login to Sentry, and go to the page with api keys. Copy the
DSN
value. - Comment out the
human_logs
line in yourconfig/local.toml
file. - You can force an error to appear in Sentry by adding a
panic!
into main.rs, just before the finalOk(())
. - Now,
SENTRY_DSN={INSERT_DSN_FROM_STEP_1_HERE} make run_local
. - You may need to stop the local server after it hits the panic! before errors will appear in Sentry.
We use env_logger: set the RUST_LOG
env var.
make test
- open the Makefile to adjust your SYNC_DATABASE_URL
as needed.
Functional tests live in server-syncstorage and can be run against a local server, e.g.:
-
If you haven't already followed the instructions here to get all the dependencies for the server-syncstorage repo, you should start there.
-
Install (Python) server-syncstorage:
$ git clone https://github.com/mozilla-services/server-syncstorage/ $ cd server-syncstorage $ make build
-
Run an instance of syncstorage-rs (
cargo run
in this repo). -
To run all tests:
$ ./local/bin/python syncstorage/tests/functional/test_storage.py http://localhost:8000#<SOMESECRET>
-
Individual tests can be specified via the
SYNC_TEST_PREFIX
env var:$ SYNC_TEST_PREFIX=test_get_collection \ ./local/bin/python syncstorage/tests/functional/test_storage.py http://localhost:8000#<SOMESECRET>
Open a PR after doing the following:
- Bump the version number in Cargo.toml.
cargo build --release
- Build with the release profile release mode.clog -C CHANGELOG.md
- Generate release notes. We're using clog for release notes. Add a-p
,-m
or-M
flag to denote major/minor/patch version, ieclog -C CHANGELOG.md -p
.
Once your PR merges, then go ahead and create an official GitHub release.
rm Cargo.lock; cargo clean;
- Try this if you're having problems compiling.