Just update to go 1.10 and a project using this as a dependency fails to build

Dependency config:

[[projects]]
  branch = "master"
  name = "github.com/google/certificate-transparency-go"
  packages = [
    ".",
    "asn1",
    "client",
    "client/configpb",
    "jsonclient",
    "tls",
    "x509",
    "x509/pkix"
  ]
  revision = "686d1c84287487373dd38c9faaf05aa79acacf26"

Build error:

# vendor/github.com/google/certificate-transparency-go/x509
vendor/github.com/google/certificate-transparency-go/x509/root_cgo_darwin.go:224: cannot use nil as type _Ctype_CFDataRef in assignment
vendor/github.com/google/certificate-transparency-go/x509/root_cgo_darwin.go:225: cannot use nil as type _Ctype_CFDataRef in assignment
vendor/github.com/google/certificate-transparency-go/x509/root_cgo_darwin.go:235: cannot convert nil to type _Ctype_CFDataRef
make: *** [build] Error 2

Go version:

go version go1.10 darwin/amd64

Appears to be related to #131 but it's already closed

See

tbs.Extensions[keyAt].Value = issuerKeyID
if err != nil {
	return nil, fmt.Errorf("failed to marshal new auth key ID: %v", err)
}

It seems like the err != nil condition is dead code. Notice that err isn't set in any of the immediately preceding lines. I'm not sure what's supposed to be happening here.

crt.sh's new log monitor (which uses the certificate-transparency-go/x509 certificate parser) tripped over these certificates:
https://crt.sh/?cablint=1172&iCAID=62434&minNotBefore=2018-01-01

The problem is that one of the EKU "OIDs" is zero-length, and therefore...
https://github.com/google/certificate-transparency-go/blob/master/asn1/asn1.go#L287

Would it be possible to relax this check in asn1.go?

Hi,

There are large numbers of certs that come from CT logs that the current x509/ASN1 golang libraries are not able to parse. In the worst case, ~10% of the skydiver certs don't parse correctly, all due to the same error. You can replicate with an example cert here: https://play.golang.org/p/IL-Ey-fSXt

This behavior is both in the golang crypto/x509 package and the ct x509 package - does anyone have any ideas as to why this would happen?

Thanks!

Make it possible to have a configuration where a single set of frontends can send requests to multiple backends. For example to split backend traffic by region.

Seeing these errors on tip (52c0590)

../../../google/certificate-transparency-go/x509/root_cgo_darwin.go:210: cannot use nil as type _Ctype_CFDataRef in assignment
../../../google/certificate-transparency-go/x509/root_cgo_darwin.go:211: cannot use nil as type _Ctype_CFDataRef in assignment
../../../google/certificate-transparency-go/x509/root_cgo_darwin.go:221: cannot convert nil to type _Ctype_CFDataRef

👋 Hi folks,

I have a small command line tool that that is importing "github.com/google/certificate-transparency-go" to reference the SignedTreeHead type exported from the ct package.

As a side effect of that import my command line tool is receiving a new -allow_verification_with_non_compliant_keys command line flag. It looks like this comes from a global flag.Bool in signatures.go:

I wouldn't expect a library import to be defining command line flags and there doesn't appear to be a simple way to discard the flag from the importing code. Is this something you folks would consider changing?

Thanks!

Which requires URL filtering if the operator wants to hide potentially sensitive information. Probably a separate listener should be used for the various debug/prom endpoints instead.

I notice we're using ctxhttp, but net/http now has a way to set a Context on a request object. It would be a little nicer to use the standard library. Any objections to switching over? This would break compat for Go 1.6 and below.

Right now go.mod depends on:

github.com/letsencrypt/pkcs11key v2.0.1-0.20170608213348-396559074696+incompatible // indirect

However, I've fixed the go modules "incompatible" problem and released a couple of new versions. I recommend importing pkcs11key as github.com/letsencrypt/pkcs11key/v4, which is the modules-compatible import path.

I'm a beginner of certificate-transparency. And i need to deploy a full ct-log server (not in aws, google cloud etc.), it must be support to store 3 billon logs at least. But i have no idea about how many QPS of each ctfe api such as add-chain, get-sth, get-proof-by-hash and so on will be requeired in a general ct log server. Is there anyone who has experience could help me ？and another problem is if a general Mariadb(mysql) can support these logs store and requests. if not，could anyone give me some other suggestions about DB in ct-log？ Thanks ！

the ctfe apis:

	AddChainPath          = "/ct/v1/add-chain"
	AddPreChainPath       = "/ct/v1/add-pre-chain"
	GetSTHPath            = "/ct/v1/get-sth"
	GetEntriesPath        = "/ct/v1/get-entries"
	GetProofByHashPath    = "/ct/v1/get-proof-by-hash"
	GetSTHConsistencyPath = "/ct/v1/get-sth-consistency"
	GetRootsPath          = "/ct/v1/get-roots"
	GetEntryAndProofPath  = "/ct/v1/get-entry-and-proof"

Either capture (x := x) or pass the variable w in the func literals in scanner/scanner.go (currently lines 295 and 301)

It looks like Scanners naïvely pass through the context instead of checking it for deadline expiration or cancelation themselves, resulting in somewhat perverse behavior: instead of gracefully shutting down, when the context passed to a Scanner's Scan() function is canceled, fetcherJob() goroutine(s) go into an infinite loop, continually trying to fetch entries and failing.

I am seeing the following error trying to use this package on Windows using go-1.11beta1:

13:11:05 ....\vendor\github.com\google\certificate-transparency\go\x509\root_windows.go:112:3: cannot use uintptr(unsafe.Pointer(sslPara)) (type uintptr) as type syscall.Pointer in field value

This is apparently caused by the following change in golang: golang/go@4869ec0

and requires a one-line fix identical to the one in here: golang/go@4869ec0?diff=unified#diff-9a0dd81b66ea632b270ed4c40cb71e43

which of course will break compilation with golang < 1.11, so a more complicated fix is required.

Found while trying to compile Docker with go-1.11beta1 on Windows (moby/moby#37358 (comment))

Currently each time a STH is retrieved via get-sth a new signature is created and since CT almost always uses ECDSA these signatures are non-deterministic. This means that when the STH doesn't change between signatures it can be used to uniquely fingerprint the user that retrieved it. It seems like there are two solutions to this problem:

• Use RFC6979 deterministic ECDSA signatures
• Cache the STH

The first seems like a lot of work, although perfectly doable and would allow the CTFE to continue being essentially stateless while the second seems a lot simpler but does require some more significant architectural changes as well as giving the CTFE more information about how the Trillian log signer operates.

With error

got RPC bad response, possible extra info: proof:<> leaf:<merkle_leaf_hash:"F\005\200\343\366p:\347|\315&\331U\257\311\317\027\363q\2647t\215?\336/\302\304\335\203\323v" leaf_value:"\000\000\000\000\001c$b\026z\000\000\000\00520\202\005.0\202\003\026\240\003\002\001\002\002\0206\001\"\004p$\037\251KQb\251\343c\233\3240\r\006\t*\206H\206\367\r\001\001\r\005\0000\0261\0240\022\006\003U\004\003\023\013esignit.org0\036\027\r140620183754Z\027\r300620184746Z0\0261\0240\022\006\003U\004\003\023\013esignit.org0\202\002\"0\r\006\t*\206H\206\367\r\001\001\001\005\000\003\202\002\017\0000\202\002\n\002\202\002\001\000\267)Y\307\374\323/\024C\262\2115\341s\264gs\357X\302\030\340u\253\243F\013\200!|u\213\367\265F\215\222cw\202\031P\275\217\024p\034\207\330\263\016\004h\220\001\303\007\306\001\t\347w'Y\034\234\3308\350\032\031\003\222#\335Kbm\264\321\001\320\037\327M\241\257h|\t\207`\222\200\200[\245\232VM\252\206\313\006\336\354\010\tC\315K\362\036\355g\201:[\306L\322\373x\232|\352\220\360S\370\031\262\2155\306\374\307\006\205\204\320\002\311\235\261\206C\004\007A\331\216\027\327\364@\310\275\036\364\301\335I\272\021\241\370`\373\307y1\340\264TB \035u?G\207\370\303?\351K\r\005#\334qX\224wQB\277\250\345\2526\305Xe\371\202\273\257\244\2604-{<RL\013\262\232\202\364\367\200\360\237\350g]\277\021K\247Dl\204>oF\310\037\034i\312\2271\037\272\3010\212Pq\266|%*\215T6O#\324\032\204\243O@\304\"\270\021:\350\251jnr\316\217\271F\007\033\033\245M\203\307\304\013\361oz\242\236\235Lkg\210\\\227\252\033\2307e1\014\303H\371\366\365\217\321\273\020\332h\252\364\325~<Z\341\025\024\352\346}g(\016\212#\231\017\203\371\342\361\020\026\327\242\273\203\214\372e\216\233\225e\227\334\217^\241\317s\232\"\t\374\257\010\325s}\252j>\201\233h#:\241\013\\\035\320\034k\325i\023g\203N\215\372\315\367E\243}X`\023\204\215cl}|)\361\372\315v\344\277/\250&Y\212\335\367Q\336\022\203\225_fP*\240K\247\234\235[\275\261\r'\202\"\276d\272\n\235c\254![\315.\2022\345s\331\203w\317\313\020d\020*z\361\315\347\226v8\220\033\365\260p\353\036\364dd\264\354\307\332\013\346\220\237\257c\315\263\3268kd\376\021\022\200:O\346\230\031\037\002\003\001\000\001\243x0v0\013\006\003U\035\017\004\004\003\002\001\2060\017\006\003U\035\023\001\001\377\004\0050\003\001\001\3770\035\006\003U\035\016\004\026\004\024\363\206\260R\253h?\277\026\203\300Qt\237\357\272\230}\221\3210\022\006\t+\006\001\004\001\2027\025\001\004\005\002\003\001\000\0010#\006\t+\006\001\004\001\2027\025\002\004\026\004\024J\277\005?\3778-\3365\342x\300G\315\250\225\335e\023i0\r\006\t*\206H\206\367\r\001\001\r\005\000\003\202\002\001\000D\276\035\244\314\217\265\272\2051\375\351\214\353\236\020\276\223:Z\330\020H\223.l\254\nx\0269P\3300\300\332p\232\3330\210B\263u\234\001\210f\365\304G\241>\260\n\317G{j\"\010\020\352'`J!!^\254\316]>\027\236\2477^\274\327y\365\224\373\002\215\322\322\003\307\343\201\375\274\216\371S)\312\313F\247\262\n~\000=\247a\352\016\377\025\030\206rb1\020S\337\375SSP\026{\227\207\266\3176\364\350\374B\253\312\003h\353\033\001V\027C?\000\320\245\371\320\245\021f\267B\332I\324\203Q\327\320\237\350\255Z]i\017Te\205M\247e\200m\357oa0\202\027\221]\2673T\372rC\242:C\304^`\270\307\275n'\313\335\204\357w\316\330C\013\007\357\352>Q\273G_\211F\361[\022~k\357&\351M\005\216\221\2676D\245\233\362Eb/A\250L\366\244\353\314Zl\306^0\310\364\201\014\362\303g\246/Uk\027\275N\017\250\263\226M\263\335\253\200A\333\203\375\220\\^b1\036\376\260\320\361\267u\375_\316\025\224\3043J\361u\220\3037\333\326\237&\363\227\314K\345Y\215,1\245\326\356\211\254\345o\0135@X\377\276/\331yQ;\254@\217\301=#\036o>\246\261\337/\2059n\017\210\320!\266\265\377\254\316\301\360khX\002\"KBOq\204\",\255\347\334w\255\324\242\254\274\243\316g\220\003\323d\302\201\277\032\251E\225'V\231\000\212\376\300\213\373=9\322\272\243/I\004\300\211\177\017\251\334\tw\301\276\020\251\ng+\"VE\256\354\236:[\010I\2365\321=p\376\222\312\360\345\341l*\365\334f\373\003\230\341\277Y\347\363]\\A\244{]E\341r~\226\253\365U\376\204\031.\351\301b\372\030:\315\335s\314)\005\246\317\256\264a\221\323f7\244bo\244\3572\000\000" extra_data:"\000\000\000" leaf_identity_hash:"\306e\027[\326\016\327\n^\3523D\313\345\321\246\223oy\274'\2508NLsB\000A|'\237" queue_timestamp:<seconds:1525323798 nanos:142668298 > integrate_timestamp:<seconds:1525323807 nanos:516704183 > > signed_log_root:<timestamp_nanos:1527638451878542800 root_hash:"\337\217\321\220\317\235\357\325\003N\0221\372\314mJ\034\0065m\025\232+a\003\375XHfp\257\020" tree_size:24501948 tree_revision:896617 key_hint:"}\027B\307b\177\264q" log_root:"\000\001\000\000\000\000\001u\336\274 \337\217\321\220\317\235\357\325\003N\0221\372\314mJ\034\0065m\025\232+a\003\375XHfp\257\020\0253C\024\005r\275\320\000\000\000\000\000\r\256i\000\000" log_root_signature:"0F\002!\000\243\245\377;\217\221P\3140\336\005\367\212GL\023\013u9\351N\215\262\304<\364\362\250\200\3220\212\002!\000\377\263\230Ksl\272\342\233\212\200a\363\237\345~SJ\245\225\303\363K\234\235\016\221\370\327=\r\356" >

from https://github.com/google/certificate-transparency-go/blob/master/trillian/ctfe/handlers.go#L808-L810. Presumably because there isn't a proof because you can't do that when the tree only has one entry.

While this is reasonable, it would be better if it returned an empty array so that clients do not need a special case for this scenario.

At the moment, if the expected MMD is set too low (and --ignore-errors is set) you can end up with a situation where get-proof-by-hash never succeeds, but the tools continues despite only seeing errors.

My theory for what happens is:

• a cert is picked for get-proof-by-hash checking, but its not integrated yet (as --mmd is set too low)
• this gives a 404 error
• as we're in --ignore-errors mode we try the operation again
• a different, newer, cert is picked for get-proof-by-hash checking
• also not integrated, giving a 404 error
• repeat until there are no certs left that were submitted more than --mmd ago
• at this point we skip the get-proof-by-hash operation, and count it as a success.

Since the code has moved, I've reopnened this issue here:

Hi,

i've tried to use the scanner.go implementation to fetch a large number of certificates stored in a ct log and save them into a csv file (for code see here).
After some time i receive the following error message:

2017/05/02 16:11:36.017863 scanner.go:420: Processed: 66147 certs (to index 66147). Throughput: 1836.16 ETA: 14 hours 51 minutes 57 seconds
fatal error: runtime: out of memory

The go process takes up 15GB and then crashes due to no more memory being available on my system.

I've never written much in golang, but suspect the context.TODO() to be at fault here:
https://github.com/google/certificate-transparency/blob/master/go/scanner/scanner.go#L281

Best,
Matthias

Example:
$ curl "ct.googleapis.com/logs/argon2020/ct/v1/get-sth-consistency?first=1000&last=1000"
Bad Request
failed to parse consistency range: strconv.ParseInt: parsing "": invalid syntax

'last' is not a valid URL query parameter for this API (it should be 'second').

I suspect the issue is in func parseGetSTHConsistencyRange

We've seen some evidence that Trillian CTFE is generating SCTs for pre-certs that are not valid. This is being investigated.

I am working on adding IPv6 to Kubernetes. I found that Kubernetes has a dependency on the x509 pkg. In reviewing this pkg, the marshalSANs function does not support IPv6 addresses. Will you accept a pull request that updates this function for IPv6?

The most recent tagged release is from August 2018. Would you mind tagging a more recent release?

This is particularly relevant now that go modules are the norm, since go modules seems to strongly prefer tagged versions over arbitrary commits.

CTFE thinks that trillians GetLeavesByIndex RPC will return a non-error response when the requested range is higher than the current tree size and that it can check the response itself for this case, this is incorrect and causes 500s.

(Tracking reference in Trillian for a possible better case is: google/trillian#1162).

Running

GOOS=darwin GOARCH=amd64 go build -o /go/bin/foo -tags "nopkcs11" -ldflags "-X metadata.Version=1.1.0-beta-snapshot-6f7fe1cac" ./cmd/

on linux gives this error:

vendor/github.com/google/certificate-transparency-go/x509/root_stub.go:9:6: (*Certificate).systemVerify redeclared in this block
	previous declaration at vendor/github.com/google/certificate-transparency-go/x509/root_darwin.go:26:6
vendor/github.com/google/certificate-transparency-go/x509/root_stub.go:13:6: initSystemRoots redeclared in this block
	previous declaration at vendor/github.com/google/certificate-transparency-go/x509/root.go:20:6

Although I configured my database not to have a password for the user test@localhost, I'm getting the following error when running ./demo-script.sh in /trillian/integration:

W0723 15:28:42.047124 8336 tree_storage.go:81] Failed to set strict mode on mysql db: Error 1045: Access denied for user 'test'@'localhost' (using password: YES)

(To be more specific, I set the password to the empty string, which appears to be the closest MySQL can get to not having a password. At any rate, the command mysql --host=127.0.0.1 --port=3306 successfully connects.)

I then get this error, which I assume is due to the first one (or due to the same cause as the first one):

E0723 15:28:50.232953 8348 admin.go:57] Admin server unavailable: rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = "transport: Error while dialing dial tcp [::1]:6962: connect: connection refused"

crt.sh's new log monitor (which uses the certificate-transparency-go/x509 certificate parser) tripped over this certificate:
https://crt.sh/?id=628153824&opt=zlint

The problem is that the public exponent is DER-encoded as 0x020400010001 instead of 0x0203010001. Whilst I'd expect the Go standard library's encoding/asn1 decoder to complain about this, ISTM that certificate-transparency-go/asn1 should be more permissive here.

Go's new official tool for dependency management, dep, strongly encourages the use of semver-tagged releases. Would you consider starting to tag releases occasionally in this repo? Thanks!

Fix up scripts/k8s configs, improve docs etc.

crt.sh's CT monitor (which uses c-t-go) was getting stuck trying to process https://crt.sh/?id=1552595593, due to the public key's elliptic curve being unsupported.

I don't for a moment think that the patch below would form an acceptable PR; but just FYI, this is what I've done to crt.sh's local copy of c-t-go's x509.go in order to temporarily work around this problem...

diff --git a/x509/x509.go b/x509/x509.go
index f70a546..b9e3366 100644
--- a/x509/x509.go
+++ b/x509/x509.go
@@ -1403,7 +1403,7 @@ func parsePublicKey(algo PublicKeyAlgorithm, keyData *publicKeyInfo, nfe *NonFat
                }
                namedCurve := namedCurveFromOID(*namedCurveOID, nfe)
                if namedCurve == nil {
-                       return nil, fmt.Errorf("x509: unsupported elliptic curve %v", namedCurveOID)
+                       return nil, nil
                }
                x, y := elliptic.Unmarshal(namedCurve, asn1Data)
                if x == nil {

Scanner re-implements LogClient.GetEntries with better (more granular) error handling and asynchronous processing. We should implement a method on LogClient that supports this use-case better, rather than copy-pasting the code.

It would be helpful if there was a Dockerfile and Kubernetes config for Migrillian, so that it could quickly and easily be spun up in the Cloud.

Both https://ct.googleapis.com/logs/argon2020/ct/v1/add-pre-chain and https://ct.cloudflare.com/logs/nimbus2020/ct/v1/add-pre-chain reject the submission below. The error message is "x509: a root or intermediate certificate is not authorized for an extended key usage: EKU not permitted: asn1.ObjectIdentifier{2, 16, 840, 1, 113741, 1, 2, 3}".

The intermediate certificate contains 2 EKU OIDs: serverAuth and clientAuth.
The leaf certificate contains 3 EKU OIDs: serverAuth, clientAuth and 2.16.840.1.113741.1.2.3.

I believe submissions of this sort should be accepted. I'm not aware of any browser that would reject this chain, and AFAICT it doesn't violate any RFC5280 rules.

{"chain":["MIIFYzCCBEugAwIBAgIQQzf/4JkQvNGXkeOd4uPCtTANBgkqhkiG9w0BAQsFADCBizELMAkGA1UEBhMCR0IxGzAZBgNVBAgTEkdyZWF0ZXIgTWFuY2hlc3RlcjEQMA4GA1UEBxMHU2FsZm9yZDEaMBgGA1UEChMRQ09NT0RPIENBIExpbWl0ZWQxMTAvBgNVBAMTKENPTU9ETyBTSEEtMiBQcm8gU2VyaWVzIFNlY3VyZSBTZXJ2ZXIgQ0EwHhcNMTgwMzI4MDAwMDAwWhcNMjAwNjI1MjM1OTU5WjBEMSEwHwYDVQQLExhEb21haW4gQ29udHJvbCBWYWxpZGF0ZWQxHzAdBgNVBAMTFndlYm1haWwuYmxpdHp2ZXJsYWcuZGUwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCevQmeLZdssLoq1wxfXS2DyHBXrNquLoPolAcSDi4edMfYPG0in+l2zNh/LZbgpULZX2rRYV2cGPJBSx15ini56Kv4EX7RjPH6JYxCIgNCCcN1LPdsgHjpo5Gj7M+C8FRG7hHFpqf1EzEK+rfQJm3rZdj6BGO2vq41Hha+hiK5aBWj6HDn95+0Km7HwadPiRV7iXHUnjKjU6iP0bZ/i12lhVL5aTHt0iHAwXVj29teVmWoZ0G0dGNQKroaac/fjrW+N0DgHbaeb6YoVTRdmOCzucSl40FGjZO8tBIG6a/Kbgf9PdS0LL8PBC0qdNVa6hT0aarQCjwCEZloFRjlGQC7AgMBAAGjggIHMIICAzAfBgNVHSMEGDAWgBSAtisNQAoP3z+kv78D9LF2a9XnjzAdBgNVHQ4EFgQUGGZGLNPDTZDmbVQlRuBmB1igOHkwDgYDVR0PAQH/BAQDAgWgMAwGA1UdEwEB/wQCMAAwKQYDVR0lBCIwIAYIKwYBBQUHAwEGCCsGAQUFBwMCBgpghkgBhvhNAQIDMFAGA1UdIARJMEcwOwYMKwYBBAGyMQECAQMEMCswKQYIKwYBBQUHAgEWHWh0dHBzOi8vc2VjdXJlLmNvbW9kby5uZXQvQ1BTMAgGBmeBDAECATBPBgNVHR8ESDBGMESgQqBAhj5odHRwOi8vY3JsLmNvbW9kb2NhLmNvbS9DT01PRE9TSEEtMlByb1Nlcmllc1NlY3VyZVNlcnZlckNBLmNybDCBgAYIKwYBBQUHAQEEdDByMEoGCCsGAQUFBzAChj5odHRwOi8vY3J0LmNvbW9kb2NhLmNvbS9DT01PRE9TSEEtMlByb1Nlcmllc1NlY3VyZVNlcnZlckNBLmNydDAkBggrBgEFBQcwAYYYaHR0cDovL29jc3AuY29tb2RvY2EuY29tMD0GA1UdEQQ2MDSCFndlYm1haWwuYmxpdHp2ZXJsYWcuZGWCGnd3dy53ZWJtYWlsLmJsaXR6dmVybGFnLmRlMBMGCisGAQQB1nkCBAMBAf8EAgUAMA0GCSqGSIb3DQEBCwUAA4IBAQCky5AVMHbFLUxgDwdKEmOGHaXWYoroMvx/0qvGrPAdE186ytx8op7InuEWTX+GhIzjd2ynR60GY2DozBm2xZyf5WdL4Q7CXhXnqS/3DchM9t5B7NoaJYhTL+LHGq4j3xLrerpA+ffYGwqnCuK8rZdZPDCW8VSL50ATJHp8osyuAiJaqZsAhzgTvA5JOOxM8VZd7UdCOiMQTLx2sgoLlDx99sCcM6VS5wtqGla2MTOM68BlQkerK5x6XaEQaCWCW/TMBk62DmSQJxLwuqRgXDFVwMnjJBsGFATQ286BuNjjQtKeSKgNz5LgjVbZXncFcaBdLfcVAH9A/XHxG3+SeMwM","MIIEszCCA5ugAwIBAgIRALm8Dvlxug7myHs2fXKPFsMwDQYJKoZIhvcNAQEMBQAwezELMAkGA1UEBhMCR0IxGzAZBgNVBAgMEkdyZWF0ZXIgTWFuY2hlc3RlcjEQMA4GA1UEBwwHU2FsZm9yZDEaMBgGA1UECgwRQ29tb2RvIENBIExpbWl0ZWQxITAfBgNVBAMMGEFBQSBDZXJ0aWZpY2F0ZSBTZXJ2aWNlczAeFw0xNTAyMTIwMDAwMDBaFw0zMDAyMTEyMzU5NTlaMIGLMQswCQYDVQQGEwJHQjEbMBkGA1UECBMSR3JlYXRlciBNYW5jaGVzdGVyMRAwDgYDVQQHEwdTYWxmb3JkMRowGAYDVQQKExFDT01PRE8gQ0EgTGltaXRlZDExMC8GA1UEAxMoQ09NT0RPIFNIQS0yIFBybyBTZXJpZXMgU2VjdXJlIFNlcnZlciBDQTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAKfDF7ksP98PglnEw+2reeiYfnyLHnFh6nHteiwESpgltt6C6LZs3DO3Kjd0ZLutEhY1slAt2+26sHAsy2kA+OIDxwW3n06ScYQHP3lCZuhjMuUT+id2gcJhtzlvag+MiPgUyIWJS6YRsnlOtRR0gIJj4yyBiexPVHUmgBgF5UjJWaL/2V99mrfvIZdTkcuF0e9sW8FyChyz0LA5YLlWCAXM+0OOCBv6sbxSrcT8VO/tR3+LDssPoOZ2Id/YHF+T4CXUtZ7uCC9/DQhXEx5nKHffKoCokzHxVR35+VeSEu2bfiGxhWR4gYd7dm3reN91ORIMStG5npyxE0NqMg6kKRkCAwEAAaOCAR8wggEbMB8GA1UdIwQYMBaAFKARCiM+lvEH7OKvKe+CpX/QMKS0MB0GA1UdDgQWBBSAtisNQAoP3z+kv78D9LF2a9XnjzAOBgNVHQ8BAf8EBAMCAYYwEgYDVR0TAQH/BAgwBgEB/wIBADAdBgNVHSUEFjAUBggrBgEFBQcDAQYIKwYBBQUHAwIwGwYDVR0gBBQwEjAGBgRVHSAAMAgGBmeBDAECATBDBgNVHR8EPDA6MDigNqA0hjJodHRwOi8vY3JsLmNvbW9kb2NhLmNvbS9BQUFDZXJ0aWZpY2F0ZVNlcnZpY2VzLmNybDA0BggrBgEFBQcBAQQoMCYwJAYIKwYBBQUHMAGGGGh0dHA6Ly9vY3NwLmNvbW9kb2NhLmNvbTANBgkqhkiG9w0BAQwFAAOCAQEAcaRIjF1YZ7ivBA30KvRTacpH8N5bdoHYzvaF6j4vutHLvuOg8BQXB7d5+USfSp4Qr6E5SUfCvy4gnKWsKsXQH3vK/lE6nL/a8Ygh9/vT8wZxKkA/FfEIVojYQ+pSUANDcepe6BtmF0WL7Pr/03swQDBUI8VzXMuDlUVSFbPUesUOkR4h1Rmh9RVfG+mCS20queLLMcTCFJg7cz3HrViPJ/vp/YYsctfXRHoEngqg8FqvFpmaz6zEDh0Wd1odBGwAfcI/oGwSmn6zsPZ5mBfOcsxxtwyH6sLpQQidsQc1VtwUp/c6vOf8m30O32OHrKOwltKNq+CM8x+kc2suZlR4ow=="]}

The CT Hammer has a bunch of options to configure its behaviour, however there is also a hard-coded 50% chance of any submission being a duplicate.

This should be configurable too.

Trillian has a /healthz endpoint that returns HTTP 200 OK and the string "ok" when it is ready to serve. The CTFE does not. It would be nice, for consistency at least, if it did. It appears that both "/" and "/metrics" are used in varying places as a substitute.

Currently, the set of trusted roots for ctfe on GKE is stored in a configmap, but there's a size limit on this, which could mean that logs trusting a large number of roots might have issues when configuring.

It looks like there was an attempt to de-flake this last year, but it's not been 100% successful: 2174cda

running go test
ok  	github.com/google/certificate-transparency-go	8.096s	coverage: 44.8% of statements
ok  	github.com/google/certificate-transparency-go/asn1	1.105s	coverage: 85.2% of statements
ok  	github.com/google/certificate-transparency-go/client	23.788s	coverage: 77.1% of statements
?   	github.com/google/certificate-transparency-go/client/ctclient	[no test files]
ok  	github.com/google/certificate-transparency-go/fixchain	2.399s	coverage: 83.3% of statements
?   	github.com/google/certificate-transparency-go/fixchain/main	[no test files]
--- FAIL: TestRateLimiterGoroutines (13.88s)
	limiter_test.go:34: #4: Too many operations per second. Expected ~1000, got 1238
FAIL
coverage: 100.0% of statements
FAIL	github.com/google/certificate-transparency-go/fixchain/ratelimiter	28.332s

Over on mozilla/tls-observatory#352 we're looking at allowing invalid extensions through since the the project is an observatory. I brought up looking at other community crypto/x509 forks in hopes someone else has created a more permissive fork already. (This looks to exist in github.com/zmap/zcrypto/x509.)

The bug here is to report a couple certificates which don't pass the crypto/x509 fork in this repository.

Certificates: DPDHL TLS CA I3 and Suva Root CA

Parsing fails with the following:

$ go run /tmp/parse.go 
12729537.der: x509: failed to parse dnsName constraint "leserservice-media.de "
12728748.der: x509: failed to parse rfc822Name constraint "@suva.ch"

If anyone has a fork with a running Travis build then I think it will currently always fail, because there's no GCE key. Maybe only run if on the master repo?

Example: https://travis-ci.org/daviddrysdale/certificate-transparency-go/jobs/349719137

Google Cloud SDK 191.0.0
bq 2.0.29
core 2018.02.23
gsutil 4.28
ERROR: (gcloud.auth.activate-service-account) Could not read json file /home/travis/gcloud-service-key.json: No JSON object could be decoded
Updated property [core/project].
Updated property [container/cluster].
WARNING: Property [zone] is overridden by environment setting [CLOUDSDK_COMPUTE_ZONE=us-central1-a]
Updated property [compute/zone].
ERROR: (gcloud.container.clusters.get-credentials) You do not currently have an active account selected.
Please run:
 $ gcloud auth login
to obtain new credentials, or if you have already logged in with a
different account:
 $ gcloud config set account ACCOUNT
to select an already authenticated account to use.
The connection to the server localhost:8080 was refused - did you specify the right host or port?
Missing ctfe config map.

Ensure you have a PEM file containing all the roots your log should accept.
and a working CTFE configuration file, then create a CTFE configmap by
running the following command:
 kubectl create configmap \
    --from-file=roots=path/to/ctfe-configmap.yaml \
    --from-file=ctfe-config-file=path/to/ct_server.cfg

Add documentation on how to use Migrillian for:

• migrating CT logs to Trillian
• operating CT log mirrors

When building ./preload/preloader, the following packages are missing:

github.com/gogo/protobuf/proto
github.com/golang/protobuf/proto
github.com/golang/protobuf/ptypes
github.com/golang/protobuf/ptypes/timestamp
golang.org/x/net/context/ctxhttp

I suggest using dep for vendoring. It might be as easy as running dep init, which worked perfectly for me, and reduced the vendor folder size, too. It only updated github.com/coreos/etcd and github.com/grpc-ecosystem/grpc-gateway/runtime.

When RejectExpired is turned on in the config, Trillian not only rejects certificates with NotAfter in the past, it also rejects certificates with NotBefore in the future. This is a bit surprising because it contradicts the name, even though the behavior is documented in the config PB.

Jacob Hoffman-Andrews [12:30 PM]
It looks like it might just be a happenstance of using the x509 library's DisableTimeChecks:

verifyOpts := x509.VerifyOptions{                                                                                                                           
    Roots:             validationOpts.trustedRoots.CertPool(),                                                                                                
    CurrentTime:       now,                                                                                                                                   
    Intermediates:     intermediatePool.CertPool(),                                                                                                           
    DisableTimeChecks: !validationOpts.rejectExpired,

Would the Trillian team be open to change that to unconditionally set DisableTimeChecks: true, and then have a specific check that NotAfter is in the future, if rejectExpired is true? Right now the behavior of rejectExpired doesn't really match the name (though it does match the documentation). Also, forward-dating certificates is not technically illegal, and seems like a separate thing from expired certificates.

There should be a run of the integration tests using Trillian and the CTFE running from within Docker containers. This would check that the Docker configs are working correctly and that the CTFE is compatible with the latest published images for Trillian. This could be extended to also try the latest tagged release of Trillian, to detect changes in CTFE that are not backwards-compatible.

Trillian's integration tests already use its Docker images, see:
https://github.com/google/trillian/blob/72c17b947c009a93a424a734829f113094419986/.travis.yml#L112

Hi folks 👋,

While implementing a mock CT server for some tests using this project I noticed the ct.SignedCertificateTimestamp type from types.go is missing json struct tags:

The RFC 6962 sections referenced in that comment (3.2, 4.1 and 4.2) provide JSON encoded responses. In my own code I work around this by creating a throw-away type with the required JSON tags but it seems like this type could provide the tags itself and save users some work :-)

The main function in ct_server\main.go does not properly check the return value of invoking setupAndRegister. Specifically, https://github.com/google/certificate-transparency-go/blob/master/trillian/ctfe/ct_server/main.go#L151 does not assign the return value of the invocation of setupAndRegister to "err". This defeats the following error check, leading to subsequent undesired behavior (such as SIGSEGVs, etc.) instead of failing fast with an appropriate error message and exiting.

e.g. if the server returns garbled JSON, or some intermediate server replaces the JSON response with an error page, then it would be useful for the client to be able to access the raw HTTP response.

One way of doing this would be to define an error struct to be returned by jsonclient.GetAndParse that contains the http.Response rather than have GetAndParse return the body and a simple error as it does now.

Another way would be to leave jsonclient.GetAndParse as it is and define a richer error struct in the client package that client.LogClient can return from the GetFoo calls.

Note;

1. something like the above would be useful also in the jsonclient.PostAndParse API and the client.Add{Pre}Chain APIs that use it
2. it would be good to be able to inspect the raw response even in successful cases.

I was recently attempting to log a certificate issued by a name-constrained CA to Argon, and I got this error: "x509: a root or intermediate certificate is not authorized to sign in this domain". I believe this behavior is incorrect.

If you need an example certificate that fails, here's 300k of them. You could also try the certificate currently used by https://vt.edu/.

It looks like the Go x509 library refuses to verify certificates from name-constrained CAs unless the DNSName was supplied in the verification options. (golang/go#15196 (comment) alludes to this behavior) Since the CTFE doesn't do that, it fails to log these certificates. This was fixed in golang/go@9e76ce7, which should be released as part of Go 1.10.

Basically a simple config option to turn a log read-only. Probably the easiest way to do this would be to add a quick check in addChainInternal along the lines of if frozen { return http.StatusBadRequest, errors.New("log is read-only") }.

Proof verification code in merkletree/ looks the same as in Trillian's merkle/ folder. Could we just reuse Trillian code here? Some features in Trillian are also needed in this repo, e.g. range inclusion verification will be used in Migrillian to verify the fetched batches.