mozilla-services / autograph Goto Github PK

segfault on test like TestMain / waiting for socket / socket cannot be used

saw this on a docs-only change (but this rebuild cleared the logs) and :ulfr has mentioned seeing something like this too.

Will add an example if I see it again.

It would make the audit log better if a checksum of the data signed was included.

Upon initialization, publish the chain of trust in a S3 bucket. The file will contain, in this order:

1. end entity certificate
2. intermediate signing cert
3. root cert

What should the chain of trust file be named? It should be something unique. We could take the public key hash of the EE cert, or we could use the Subject.CN + expiration date. The latter is more human readable.

@mozmark / @franziskuskiefer: thoughts?

Instead of storing private keys in the yaml config file, Autograph should support storing keys in a PKCS11-compliant HSM, and let the HSM perform crypto operations.

When doing this request:

requests.post('http://localhost:8000/signature', auth=self.auth, json=[{
    "input": payload
}])

I've got this response:

{u'hashalgorithm': u'sha384', u'signature': u'Noao6zwWwGhHnKhKcfF2QKEe_qxVpU_qt41BjGOo6FpLIkvHPUu5FmN6SgCQr5EyMpFCk3bitnZ65NenxOvrR9hIMPhuja1ZW8MoZt0TJFfZ7OXI3LvUSwlpCxcmSxDz', u'encoding': u'b64url'}

The hash algorithm is specified to "sha384" but it actually isn't (and autograph has no way to know it)

To go along with

Possibly limited to the max nginx body size.

Nice to have, but I was thinking it'd be handy to have a CLI option to parse the config file and print the matrix of authorizations for ids by signers.

Currently we get:

Content-Type: text/plain; charset=utf-8

It should be:

Content-Type: application/json; charset=utf-8

Autograph should support signing MAR files used to update Firefox. Ideally, MAR signing will start using the AMO PKI at the same time.

All signing operations should be logged (and possibly chained) to provide an audit trace of the activity of the signers.

On-the-fly key generation is slow. Autograph should use idle periods to pregenerate RSA keys and store them in a in-memory cache.

Spec at https://source.android.com/security/apksigning/v2

Example code that verifies at https://github.com/avast/apkverifier

This should be added to the FileSigner interface of the apk signer.

When a certificate nears expiration, the lambda returns

Errors found during monitoring: errorString
null

with no indication of that actual issue. The issue is mentioned earlier is the log output but not obvious from this message alone.

That is fail the request before hashing it here:

if the content type isn't application/json or any other accepted type.

refs: sirupsen/logrus#543

Ran into this running go test -cover

I think we just need to update the vendored version, since it's fixed upstream.

Instead of loading the private key in a yaml value, the autograph conf should take the path to a PEM encoded private key and certificate.

That would allow to verify chain of trust, publish the chain file to S3 and monitor expiration dates in datadog.

all inbound calls to the API should have valid Hawk Authorization tokens

on https://github.com/mozilla-services/autograph/blob/master/docs/endpoints.rst and its use for signing APKs and XPIs

this should be avoided here https://github.com/mozilla-services/autograph/blob/master/tools/sign-addon/sign.py#L18-L20 because you hve no guarantee no one will do something with the file before you start to actually create it.

use the secure version in the stdlib that will hand you an FD to make sure no one else is tampering with the same file

It looks like autograph cannot be install with Go 1.5 anymore:
https://travis-ci.org/Kinto/kinto-signer/jobs/139498637

Works fine with 1.6:
https://travis-ci.org/Kinto/kinto-signer/jobs/140562373

Autograph should support XPI signing.

cc @mozkeeler, @franziskuskiefer and @jcjones

Autograph needs a signer to issue COSE signatures for add-ons.

As there isn't a COSE library for golang yet, we'll need to write one, but we can reuse existing cbor encoders to help.

The format of signature requests is too obscure:

• input is used for any type of data input, whether it's a hash or raw data
• hashwith tells autograph if input needs to be hashed
• template tells autograph if the input needs to be modified before hashing

This leads to confusion. I'd like to simplify as follows:

• b64rawinput is a base64 encoded string that contains raw, non hashed, input data.
• b64hashinput is a base64 encoded string that contains hashed data

with the following rules:

1. modifications on b64hashinput are forbidden, meaning if either hashwith or template are set on a signature request that contains non-empty b64hashinput, an error will be returned to the caller.
2. b64hashinput and b64rawinput are mutually exclusive. If both are non-empty in a signature request, an error is returned to the caller.
3. if only b64rawinput is set, and no hashwith value is set, the hash algorithm in inferred from the private key (for example: P-384 will use SHA-384).
4. On b64rawinput, no template is applied unless explicitly requested. A consequence of this: on b64rawinput, the content-signature value is not returned if no template is requested (content-signature requires template: content-signature in the signature request). On b64hashinput, content-signature is always returned, as we assume the caller performed the templating prior to calling autograph.

Examples:

1. caller wants a signature on a hash

[ { "b64hashinput": "oidh19OQSHAD971eh==" }]

1. caller wants a content-signature on raw input:

[ { "b64rawinput": "Y2FyaWJvdW1hdXJpY2UK", "template": "content-signature" } ]

@ncloudioj @almet : thoughts?

The bugzilla signer lambda function is an autograph client that watches a given bugzilla component for attachments to sign. The idea is to rely on bugzilla as a tracking tool while automating the signing operations of addons and mobile applications. The workflow would work as follows:

1. developer creates bug in product X component Y to request signing of application A
2. bugzilla-signer wakes up and poll the rest api for new bugs (created in last X minutes)
3. bugzilla-signer finds the new bug and grabs the attachment. Optionally, we can filter to only take bugs that have receive relman+, or something similar.
4. bugzilla-signer parses the attachment to decide how to handle it. Android apps and add-ons have in ID in their manifest that can be mapped against a configuration file to decide how to sign. Otherwise, we could also use the product/component location for that. bugzilla-signer then calls autograph with the right set of credentials to sign the attachment.
5. bugzilla-signer gets the signed attachment back and push it to bugzilla by creating a new attachment in the bug.

Hey @jvehent ,

Having a quick question about the return value. The signature is currently of the JSON array type, for example,

"signatures": [
      {
        "encoding": "b64url",
        "signature": "PWUsOnvlhZV0I4k4hwGFMc3LQcUlS-l1UwD0cNevPv3ux7T9moHX_JZHc75cmnyo-hUkW6s-c6AaNr_dyxg2528OLY53voIqwTsiYll1iPElS9TV0xOo3awuwnYcctOp",
        "hashalgorithm": "sha256"
      }
    ] 

So is there any special reason for that? To me this is a bit unnecessary if autograph only attaches one signature to each piece of content.

The present heartbeat endpoint is functional but could use a bit more awesomeness. Particularly, as more features are implemented, it would be good if they were rolled into the health check here.

Here's a really good example from the Hello project:
https://github.com/mozilla-services/loop-server/blob/a4068bc919ed02ec870caaa49492f7401d55c2be/loop/routes/home.js#L35-L97

test Fx can successfully install signed for XPIs #76 and apply MAR #48 update files.

Both test should probably have xpcshell tests which could either:

1. download nightly and xpcshell.tests e.g. wget http://ftp.mozilla.org/pub/firefox/nightly/latest-mozilla-central/firefox-62.0a1.en-US.linux-x86_64.tar.bz2 http://ftp.mozilla.org/pub/firefox/nightly/latest-mozilla-central/firefox-62.0a1.en-US.linux-x86_64.xpcshell.tests.zip (currently about ~70MB)
2. extract them next to each other

or:

1. clone mozilla-central/unified and run bootstrap (or ideally a subset)

For the XPIs tests we would then:

1. sign a webextension
2. enable the dev-root in the Fx config
3. run something like https://searchfox.org/mozilla-central/source/security/manager/ssl/tests/unit/test_signed_apps.js that loads our new root then tries to verify the signed extension signatures

Many daemon programs provide a command option (usually -p <port>) so that users can change the default port the program listens on.

Can we add this feature to Autograph too?

The config file I'm using is checked into a secret repository and shared across the team. It is inconvenient and hard to maintain a local copy that uses a non-default port.

I'm seeing TestRsaCaching fail intermittently in CI and local envs. I've only checked the branch I'm working on.

--- FAIL: TestRsaCaching (12.87s)
        xpi_test.go:349: key retrieval from cache took more than 10ms

Is anyone else seeing or has seen this test fail? If so, can we switch it to a https://golang.org/pkg/testing/#Benchmark or test with large timeout and benchmark?

The API should be built into a docker container in travis-ci.

Hello I tried to call autograph using httpie.

Installation

pip install httpie requests-hawk

Try:

echo '[{"template": "content-signature", "input": "y0hdfsN8tHlCG82JLywb4d2U+VGWWry8dzwIC3Hk6j32mryUHxUel9SWM5TWkk0d", "keyid": "appkey1"}]' | \
    http POST http://localhost:8000/signature -v --auth-type hawk \
        --auth alice:fs5wgcer9qj819kfptdlp8gm227ewxnzvsuj9ztycsx08hfhzu

It works with the default alice token (in the github autograph.yml file)

As soon as I change the key for alice it yaml file, it doesn't work anymore.

$ echo '[{"template": "content-signature", "input": "y0hdfsN8tHlCG82JLywb4d2U+VGWWry8dzwIC3Hk6j32mryUHxUel9SWM5TWkk0d", "keyid": "appkey1"}]' | \
>         http POST http://localhost:8000/signature -v --auth-type hawk \
>             --auth alice:fs5wgcer9qj819kfptdlp8gm227ewxnzvsuj9ztycsx08hfhzu
{"Timestamp":1455633064467077546,"Type":"app.log","Logger":"Autograph","Hostname":"rhubscher-ThinkPad-W540","EnvVersion":"2.0","Pid":13448,"Fields":{"msg":"handlers.go:149: signing operation succeeded:[{\"ref\":\"2m45wwnlwsnan38f3wkanzuo39\",\"certificate\":{\"encryptionkey\":\"MHYwEAYHKoZIzj0CAQYFK4EEACIDYgAE4k3FmG7dFoOt3Tuzl76abTRtK8sb_r_ibCSeVKa96RbrOX2ciscz_TT8wfqBYS_8cN4zMe1-f7wRmkNrCUojZR1ZKmYM2BeiUOMlMoqk2O7-uwsn1DwNQSYP58TkvZt6\"},\"signatures\":[{\"encoding\":\"b64url\",\"signature\":\"Fx2aiRm3w0VSdl7LSbt_On2clVxXATAcxkefRgedEqB3clWN1kKhxs44BchztAEXM3p7FBEhclT6r2NhOQNkmk-K0M-b6ovw7VrhMCwLRfLakirpF_w_4Er0I5T0d64B\",\"hashalgorithm\":\"sha384\"}]}]"}}
POST /signature HTTP/1.1
Accept: application/json
Accept-Encoding: gzip, deflate
Authorization: Hawk mac="k+bNQzF0qK8Kl5Xt9Qxmk7k450HRwZUUd96AccsW1/I=", hash="oy0PUEUqRuMkFNekH2vtAy4jdA60zhii+XWlfMPqXHc=", id="alice", ts="1455633064", nonce="dm2sXU"
Connection: keep-alive
Content-Length: 133
Content-Type: application/json
Host: localhost:8000
User-Agent: HTTPie/0.9.3

[
    {
        "input": "y0hdfsN8tHlCG82JLywb4d2U+VGWWry8dzwIC3Hk6j32mryUHxUel9SWM5TWkk0d", 
        "keyid": "appkey1", 
        "template": "content-signature"
    }
]

HTTP/1.1 201 Created
Content-Length: 438
Content-Type: text/plain; charset=utf-8
Date: Tue, 16 Feb 2016 14:31:04 GMT

[{"ref":"2m45wwnlwsnan38f3wkanzuo39","certificate":{"encryptionkey":"MHYwEAYHKoZIzj0CAQYFK4EEACIDYgAE4k3FmG7dFoOt3Tuzl76abTRtK8sb_r_ibCSeVKa96RbrOX2ciscz_TT8wfqBYS_8cN4zMe1-f7wRmkNrCUojZR1ZKmYM2BeiUOMlMoqk2O7-uwsn1DwNQSYP58TkvZt6"},"signatures":[{"encoding":"b64url","signature":"Fx2aiRm3w0VSdl7LSbt_On2clVxXATAcxkefRgedEqB3clWN1kKhxs44BchztAEXM3p7FBEhclT6r2NhOQNkmk-K0M-b6ovw7VrhMCwLRfLakirpF_w_4Er0I5T0d64B","hashalgorithm":"sha384"}]}]

POST /signature HTTP/1.1
Accept: application/json
Accept-Encoding: gzip, deflate
Authorization: Hawk mac="T5L6w+VyPzX3Vm4cpJ7F7UHUeWEAC3Qfrj9dgF05Wjo=", hash="oy0PUEUqRuMkFNekH2vtAy4jdA60zhii+XWlfMPqXHc=", id="alice", ts="1455633130", nonce="3Iua8s"
Connection: keep-alive
Content-Length: 133
Content-Type: application/json
Host: localhost:8000
User-Agent: HTTPie/0.9.3

[
    {
        "input": "y0hdfsN8tHlCG82JLywb4d2U+VGWWry8dzwIC3Hk6j32mryUHxUel9SWM5TWkk0d", 
        "keyid": "appkey1", 
        "template": "content-signature"
    }
]

HTTP/1.1 401 Unauthorized
Content-Length: 53
Content-Type: text/plain; charset=utf-8
Date: Tue, 16 Feb 2016 14:32:10 GMT
X-Content-Type-Options: nosniff

authorization verification failed: hawk: invalid MAC

Monitoring should could test a signature on a validation key to make sure the service is running correctly.

@Micheletto: which format and where should the test output go?

Upon initialization of the autograph service, private keys and CSRs must be generated and submitted to the HSM for signature by the intermediates.

https://docs.google.com/document/d/1rGVyiLYvZyKE2oHcSVx-vBmQRKhs1kLLgn7xeCs6qKs/edit

GET /__version__
{
    “source”: “https://github.com/mozilla-services/autograph”,
    “version”: “1.0.0”,
    “commit”: “a5bc5f656571613ef4491f33504ecf36cb390677”
}

Implement a basic API endpoint to serve signature to callers

Non need for python anymore https://aws.amazon.com/blogs/compute/announcing-go-support-for-aws-lambda/

Callers should have access to specific signing keys based on their authorization.

This warning should trigger 39 days before expiration with a weekly email and 15 days before expiration with a daily page (via pagerduty).

Instead of storing authorizations in a config yaml, Autograph should rely on a database to allow for new authorization to be created without redeploying the service.

This would mean storing hawk tokens also in database, so we should consider the security implications of doing so.

It would allow us to remove the sops integration, and by doing so also remove the large dependency tree that goes with it.

failing test case here for cherry-picking: 1eccabc

See contentsig for a prototype.

Need test cases

• signature operations
• response time