hoophq / sequence Goto Github PK

View Code? Open in Web Editor NEW

491.0 13.0 31.0 44 KB

Immutable, scalable, and easy to use ledger service.

License: Apache License 2.0

Dockerfile 1.11% Clojure 98.89%

ledger bookkeeping fintech double-entry-accounting merkle-tree hash-chain

sequence's Introduction

Sequence

Website: https://decimals.app/sequence
Community: Decimals Slack
Documentation: https://decimals.stoplight.io/docs/sequence/

Sequence is an API that store validate and reports asset movements. Also known as a Ledger. Sequence is immutable, scalable, and easy to use.

The key features of Sequence are:

Multi-Currency: Store, move and analyse any asset, from regular currencies like USD, to shelf items.
Multi-Tenant: Run multiple ledgers using the same infrastructure. Simply setup multiple tenants in the configurations and use the different API keys.
No-SQL powered: Sequence runs on top of a No-SQL database. It is horizontally scalable from the 12-factors container to the persistency layer.
Immutable: Most existing ledger use database updates. This is bad for a ledger. Sequence is immutable. The design of the database allows for consistency of balances without a single field using updateds.
API simplicity: Sequence creates things as you use them. When you send value to an account, it is gets created if it does not exist. It gets out of the way. Use it, and things will work.
Analytics: Send asset movement events to multiple destinations. Use it for analytics, fraud-detection, anything.

Getting Started

docker-compose

Clone and cd into the repository
Use docker-compose to bring up DynamoDB local and Sequence

git clone https://github.com/decimals/sequence.git

cd sequence

docker-compose up

Docker

Start the DynamoDB local container
Start the Sequence container

docker run -p 8000:8000 amazon/dynamodb-local

docker run -p 8910:8910 docker.pkg.github.com/decimals/sequence/sequence:0.0.1

Configurations

All configurations are loaded from environment variables. The available configurations are:

environment variable	configuration	dev profile
`DYNAMODB_ENDPOINT`	The host for the DynamoDB instance. Mostly used for local development.	`http://localhost:8000`
`SEGMENT_KEY`	Optional Segment.io key to generate analytics events.
`KEYS`	A string with a list of tenants and their `sha256` API keys digests. In json format.	`[{ "name": "test","email": "[email protected]", "public-key": "abc", "secret-key-hash": "a665a45920422f9d417e4867efdc4fb8a04a1f3fff1fa07e998e86f7f7a27ae3"}]`
`ALLOWED_ORIGINS`	A list of accepted CORS hosts. Example: `ALLOWED_ORIGINS= '["https://decimals.stoplight.io"]'`

Configurations can also be loaded from the profiles.clj file, where the dev configurations are setup.

Developing Sequence

Start a new REPL: lein repl
Start Sequence in dev-mode: (def dev-serv (start-dev))
Connect your editor to the running REPL session. Re-evaluated code will be seen immediately in the service.

sequence's People

Contributors

Stargazers

Watchers

sequence's Issues

Allow to pass CORS configuration

Allow configuration of allowed origins by environment variables.

(def service-map
  {::http/routes i/routes
   ::http/allowed-origins {:allowed-origins
                           ["https://decimals.stoplight.io", <-- This should come from config
                            "http://localhost:3000"] <-- I had to add for development :)
                           :methods "GET,POST"}

Support Idempotent requests

Idempotent requests are important to enable consistency over the network with other services using the ledger.

An optional x-request-id header should ensure only once processing per unique value of the header. When not present, this rule should not be applied.

Error when will execute lein repl

I'm following this guide, but I'm with a error on line 15

https://github.com/runopsio/sequence#developing-sequence

Cors error

I'm with a cors error, can you help me ?

sequence_1  | INFO  io.pedestal.http.cors  - {:msg "cors request processing", :origin "http://localhost:3000", :allowed nil, :line 84}

Sequence is compatible with any database allowing conditional puts. We should implement connectors to multiple databases to allow the usage in different environments. This will be the first adapter implementation. It will pave the way for new adapter to be created. The implementation must include:

A better decoupling of the DB adapter logic from the ledger
The implementation of the adapter for the Datomic database
Proper documentation about how to implement new adapters

Invalid credentials

Hi,

I cloned the repo and it's running on Docker. I'm trying to run the postman collection but so far I'm getting 403 "Invalid credentials." error message.

I'm following instructions from the GitHub repo and your documentation.

https://docs.decimals.app/docs/sequence/ZG9jOjEyOTQ0MDY-getting-started#try-out-the-api-collections

This is my docker compose:

"secret-key-hash": "a665a45920422f9d417e4867efdc4fb8a04a1f3fff1fa07e998e86f7f7a27ae3"}]' # -> 123

I doubled check the secret-key-hash using https://emn178.github.io/online-tools/sha256.html and it gives the same value for 123

I'm trying to hit the following endpoint: http://localhost:8910/v1/transactions?account=Alice

In postman, as far as I understand, public_key is not really that important? I see some requests only use it on the body to seed initial values, so its value shouldn't matter that much, does it? In case it does in the postman variables I'm using "public_key = abc"

About the secret_key variable in postman, I've tried the below values in the "current value" column of the environmental variables, I cleared "initial value" column just to be safe.

I'm encoding using: https://www.utilities-online.info/base64

Basic 123
Basic 123:
Basic MTIz
Basic MTIzOg==
Basic a665a45920422f9d417e4867efdc4fb8a04a1f3fff1fa07e998e86f7f7a27ae3

None of those secret_key values have worked so far, I always get the same error message: "Invalid credentials." with a 401 Unauthorized code. I even tried the solution out-of-the-box and with the secrets and all that as configured in the repo and still facing the same issue.

I'm not sure what I'm doing wrong or what values I'm incorrectly setting up.

P.D. This is a fantastic work based on what I've read in the documentation and is exactly what I'm looking for, hope you can help me with this authorization issue I'm having. Thank you!!!

Docs about asset movement events

As per the home page: "Send asset movement events to multiple destinations". (I would love to be able to subscribe to events on specific (or all) accounts through a pub/sub pattern.)

Will this eventually be documented?

Thanks for a really interesting product – I love the philosophy and the design choices.

Docs about pagination

As we discussed, just some simple docs revealing the pagination capabilities 👍

Error starting the server on docker build

Hello, I just cloned your project, I wasn't able to pull the docker from github registry, so I changed the docker compose to build instead, not sure if all the configuration is there in order to build, but I got this error:

sequence_1  | Creating your server...
sequence_1  | Exception in thread "main" java.lang.RuntimeException: could not start [#'decimals.analytics/analytics] due to
sequence_1  | 	at mount.core$up$fn__14040.invoke(core.cljc:80)

Click to expand full error

sequence_1  | Creating your server...
sequence_1  | Exception in thread "main" java.lang.RuntimeException: could not start [#'decimals.analytics/analytics] due to
sequence_1  | 	at mount.core$up$fn__14040.invoke(core.cljc:80)
sequence_1  | 	at mount.core$up.invokeStatic(core.cljc:80)
sequence_1  | 	at mount.core$up.invoke(core.cljc:78)
sequence_1  | 	at mount.core$bring.invokeStatic(core.cljc:247)
sequence_1  | 	at mount.core$bring.invoke(core.cljc:239)
sequence_1  | 	at mount.core$start.invokeStatic(core.cljc:289)
sequence_1  | 	at mount.core$start.doInvoke(core.cljc:281)
sequence_1  | 	at clojure.lang.RestFn.invoke(RestFn.java:397)
sequence_1  | 	at decimals.transport$start.invokeStatic(transport.clj:24)
sequence_1  | 	at decimals.transport$start.invoke(transport.clj:23)
sequence_1  | 	at decimals.transport$_main.invokeStatic(transport.clj:31)
sequence_1  | 	at decimals.transport$_main.doInvoke(transport.clj:27)
sequence_1  | 	at clojure.lang.RestFn.invoke(RestFn.java:397)
sequence_1  | 	at clojure.lang.AFn.applyToHelper(AFn.java:152)
sequence_1  | 	at clojure.lang.RestFn.applyTo(RestFn.java:132)
sequence_1  | 	at decimals.transport.main(Unknown Source)
sequence_1  | Caused by: java.lang.IllegalArgumentException: No matching field found: message for class java.lang.NullPointerException
sequence_1  | 	at clojure.lang.Reflector.getInstanceField(Reflector.java:397)
sequence_1  | 	at clojure.lang.Reflector.invokeNoArgInstanceMember(Reflector.java:440)
sequence_1  | 	at decimals.analytics$fn__21388.invokeStatic(analytics.clj:9)
sequence_1  | 	at decimals.analytics$fn__21388.invoke(analytics.clj:7)
sequence_1  | 	at mount.core$record_BANG_.invokeStatic(core.cljc:74)
sequence_1  | 	at mount.core$record_BANG_.invoke(core.cljc:73)
sequence_1  | 	at mount.core$up$fn__14040.invoke(core.cljc:81)
sequence_1  | 	... 15 more
sequence_sequence_1 exited with code 1

The only think I changed in the docker-compose file was:

services:
  sequence:
     build: .
     # image: docker.pkg.github.com/decimals/sequence/sequence:0.0.1
     # the rest is the same...

I haven't worked with clojure before so it's hard for me to debug. I assume it might be due I haven't provided the Segment.io key? Not sure but since is related to the analytics I think it could be an issue with that.

Happy to help debugging and looking forward to test this project :)

Docs about data model, add comments to code

I am having a really hard time following your code (I admittedly have no Clojure experience which doesn't help), it seems the comments are just related to things that were fixed. I also see your NoSQL Workshop file, but there is nothing that I can just jump into and read how you are storing the data and using the generically named indexes of LSI1 and GSI1.

I would highly recommend adding more comments to your code to describe what functions are doing, and adding documentation of the database schema and row examples of what the raw data looks like.

I did read your blog post regarding Sequence at https://decimals.substack.com/p/things-i-wish-i-knew-before-building

One of the things I was looking for was your merkle tree code to see how you were handling this - but a search of your sources for merkle, tree, or root turns up nothing, so I don't know if you just called it something different or if its not implemented in the current version.

`map->md5` is not predictive

The function sequence.crypto/map->md5 does not return the same output for inputs that are semantically the same. An example is given in the bellow code.

(let [m {:foo 300 :toto 600}
      t (assoc (assoc {} :toto 600) :foo 300)]
  (println m (map->md5 m)) ;; {:foo 300, :toto 600} d8a50edd5aeb65be51deb87eefc48f93
  (println t (map->md5 t)))  ;; {:toto 600, :foo 300} 52a197cb51d65d93a047df1b7fa994aa

The root cause of the issue is that part that convert a map to a byte array. JSON is a predictive mechanism to do such thing. It is very easy to find two different json serialization of the same underlying data structure; {"foo": 4} and {"foo": 4} comes immediately to my minds.

The solution involves walking the data data structure in a predictable way and compute the hash while doing this. I am sure there is a java library that already implements this logic for you.

Multiple concurrent requests results in incorrect 200 responses

If you send multiple concurrent requests like:

POST transactions andrios -> lucas (amount 5) 
POST transactions andrios -> lucas (amount 10) 
POST transactions andrios -> lucas (amount 15) 
POST transactions andrios -> lucas (amount 20) 
POST transactions andrios -> lucas (amount 25)

You'll get multiple 200 OK responses, but only one of this transactions actualy succeeded.

Duplicate entry prevention

In some cases backend could generate multiple requests to the ledger with the same transaction data resulting in a duplicate entry. One way to prevent this could be setting transaction id to HASH(order.id) - posting this more than once should result in failure.

Is this possible? Or there is a better approach to this?

Website and documentation is down

Currently down:

Metadata not returning on transactions

When creating or fetching transactions, the metadata is not returned.

To reproduce:

Given the transaction created with this payload:

{
    "from": "lucas",
    "to": "andrios",
    "amount": 100,
    "currency": "BRL",
    "metadata": [{
        "test": true
    }]
}

I should get back two transactions with the same metadata, but what I get back does not contain metadata:

[
    {
        "amount": 100,
        "date": "2020-09-12T09:21:23.764Z",
        "currency": "BRL",
        "balance": 500,
        "from": "lucas",
        "id": "16ea0f46389d2511cae81954309a339b",
        "to": "andrios"
    },
    {
        "amount": 100,
        "date": "2020-09-12T09:21:23.767Z",
        "currency": "BRL",
        "balance": 100,
        "from": "lucas",
        "id": "c44eacd2b08586b54b4ab179fdea09af",
        "to": "andrios"
    }
]

Internal server error: exception

Hi, I'm playing around with Sequence and bumped into trouble early on:

docker-compose up

Starting sequence_dynamodb_1 ... done
Creating sequence_sequence_1 ... done

curl http://localhost:8910/v1/transactions
-H 'Authorization: Basic MjyNj0Mwo='
-d '{
"from": "abc",
"to": "Alice",
"amount": 1000,
"currency": "usd"
}'

Internal server error: exception

The Sequence console reports the following:

sequence_1  | INFO  io.pedestal.http  - {:msg "POST /v1/transactions", :line 80}
sequence_1  | INFO  io.pedestal.http.cors  - {:msg "cors request processing", :origin nil, :allowed nil, :line 84}
sequence_1  | ERROR i.p.http.impl.servlet-interceptor  - {:msg "error-ring-response triggered", :context {:io.pedestal.interceptor.chain/stack (#Interceptor{:name :io.pedestal.http.impl.servlet-interceptor/stylobate} #Interceptor{:name :io.pedestal.http.impl.servlet-interceptor/terminator-injector}), :request {:protocol "HTTP/1.1", :async-supported? true, :remote-addr "172.19.0.1", :servlet-response #object[org.eclipse.jetty.server.Response 0x5b15bdac "HTTP/1.1 200 \nDate: Fri, 18 Sep 2020 14:58:38 GMT\r\n\r\n"], :servlet #object[io.pedestal.http.servlet.FnServlet 0x718000a1 "io.pedestal.http.servlet.FnServlet@718000a1"], :headers {"user-agent" "curl/7.64.1", "authorization" "Basic MjyNj0Mwo=", "host" "localhost:8910", "accept" "*/*", "content-length" "73", "content-type" "application/x-www-form-urlencoded"}, :server-port 8910, :servlet-request #object[org.eclipse.jetty.server.Request 0x71f2f53b "Request(POST //localhost:8910/v1/transactions)@71f2f53b"], :content-length 73, :content-type "application/x-www-form-urlencoded", :path-info "/v1/transactions", :url-for #object[clojure.lang.Delay 0x13737d55 {:status :pending, :val nil}], :uri "/v1/transactions", :server-name "localhost", :query-string nil, :path-params {}, :body #object[org.eclipse.jetty.server.HttpInputOverHTTP 0x2462d585 "HttpInputOverHTTP@2462d585[c=0,q=0,[0]=null,s=STREAM]"], :scheme :http, :request-method :post, :context-path ""}, :bindings {#'io.pedestal.http.route/*url-for* #object[clojure.lang.Delay 0x13737d55 {:status :pending, :val nil}]}, :enter-async [#object[io.pedestal.http.impl.servlet_interceptor$start_servlet_async 0x1b651f9c "io.pedestal.http.impl.servlet_interceptor$start_servlet_async@1b651f9c"]], :io.pedestal.interceptor.chain/terminators (#object[io.pedestal.http.impl.servlet_interceptor$terminator_inject$fn__13520 0x526ce478 "io.pedestal.http.impl.servlet_interceptor$terminator_inject$fn__13520@526ce478"]), :servlet-response #object[org.eclipse.jetty.server.Response 0x5b15bdac "HTTP/1.1 200 \nDate: Fri, 18 Sep 2020 14:58:38 GMT\r\n\r\n"], :route {:path "/v1/transactions", :method :post, :path-re #"/\Qv1\E/\Qtransactions\E", :path-parts ["v1" "transactions"], :interceptors [#Interceptor{:name :io.pedestal.http/json-body} #Interceptor{:name :auth} #Interceptor{:name :parse-tx} #Interceptor{:name :spec-tx} #Interceptor{:name :from-balance} #Interceptor{:name :genesis-balance} #Interceptor{:name :check-balance} #Interceptor{:name :hash-txs} #Interceptor{:name :chain-tx}], :route-name :transactions-post, :path-params {}, :io.pedestal.http.route.prefix-tree/satisfies-constraints? #object[clojure.core$constantly$fn__5672 0x6f1e7c72 "clojure.core$constantly$fn__5672@6f1e7c72"]}, :servlet #object[io.pedestal.http.servlet.FnServlet 0x718000a1 "io.pedestal.http.servlet.FnServlet@718000a1"], :servlet-request #object[org.eclipse.jetty.server.Request 0x71f2f53b "Request(POST //localhost:8910/v1/transactions)@71f2f53b"], :url-for #object[clojure.lang.Delay 0x13737d55 {:status :pending, :val nil}], :io.pedestal.interceptor.chain/execution-id 9, :servlet-config #object[org.eclipse.jetty.servlet.ServletHolder$Config 0x6672e704 "org.eclipse.jetty.servlet.ServletHolder$Config@6672e704"], :async? #object[io.pedestal.http.impl.servlet_interceptor$servlet_async_QMARK_ 0x13938483 "io.pedestal.http.impl.servlet_interceptor$servlet_async_QMARK_@13938483"]}, :line 253}
sequence_1  | clojure.lang.ExceptionInfo: java.lang.NullPointerException in Interceptor :auth - 
sequence_1  | 	at io.pedestal.interceptor.chain$throwable__GT_ex_info.invokeStatic(chain.clj:35)
sequence_1  | 	at io.pedestal.interceptor.chain$throwable__GT_ex_info.invoke(chain.clj:32)
sequence_1  | 	at io.pedestal.interceptor.chain$try_f.invokeStatic(chain.clj:57)
sequence_1  | 	at io.pedestal.interceptor.chain$try_f.invoke(chain.clj:44)
sequence_1  | 	at io.pedestal.interceptor.chain$process_all_with_binding.invokeStatic(chain.clj:171)
sequence_1  | 	at io.pedestal.interceptor.chain$process_all_with_binding.invoke(chain.clj:146)
sequence_1  | 	at io.pedestal.interceptor.chain$process_all$fn__9198.invoke(chain.clj:188)
sequence_1  | 	at clojure.lang.AFn.applyToHelper(AFn.java:152)
sequence_1  | 	at clojure.lang.AFn.applyTo(AFn.java:144)
sequence_1  | 	at clojure.core$apply.invokeStatic(core.clj:665)
sequence_1  | 	at clojure.core$with_bindings_STAR_.invokeStatic(core.clj:1973)
sequence_1  | 	at clojure.core$with_bindings_STAR_.doInvoke(core.clj:1973)
sequence_1  | 	at clojure.lang.RestFn.invoke(RestFn.java:425)
sequence_1  | 	at io.pedestal.interceptor.chain$process_all.invokeStatic(chain.clj:186)
sequence_1  | 	at io.pedestal.interceptor.chain$process_all.invoke(chain.clj:182)
sequence_1  | 	at io.pedestal.interceptor.chain$enter_all.invokeStatic(chain.clj:235)
sequence_1  | 	at io.pedestal.interceptor.chain$enter_all.invoke(chain.clj:229)
sequence_1  | 	at io.pedestal.interceptor.chain$execute.invokeStatic(chain.clj:379)
sequence_1  | 	at io.pedestal.interceptor.chain$execute.invoke(chain.clj:352)
sequence_1  | 	at io.pedestal.interceptor.chain$execute.invokeStatic(chain.clj:389)
sequence_1  | 	at io.pedestal.interceptor.chain$execute.invoke(chain.clj:352)
sequence_1  | 	at io.pedestal.http.impl.servlet_interceptor$interceptor_service_fn$fn__13545.invoke(servlet_interceptor.clj:351)
sequence_1  | 	at io.pedestal.http.servlet.FnServlet.service(servlet.clj:28)
sequence_1  | 	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:873)
sequence_1  | 	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:542)
sequence_1  | 	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
sequence_1  | 	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345)
sequence_1  | 	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
sequence_1  | 	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)
sequence_1  | 	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
sequence_1  | 	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247)
sequence_1  | 	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
sequence_1  | 	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
sequence_1  | 	at org.eclipse.jetty.server.Server.handle(Server.java:505)
sequence_1  | 	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:370)
sequence_1  | 	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:267)
sequence_1  | 	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
sequence_1  | 	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
sequence_1  | 	at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
sequence_1  | 	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
sequence_1  | 	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
sequence_1  | 	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
sequence_1  | 	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
sequence_1  | 	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
sequence_1  | 	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:698)
sequence_1  | 	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:804)
sequence_1  | 	at java.lang.Thread.run(Thread.java:748)
sequence_1  | Caused by: java.lang.NullPointerException: null
sequence_1  | 	at clojure.core$subs.invokeStatic(core.clj:4987)
sequence_1  | 	at clojure.core$subs.invoke(core.clj:4981)
sequence_1  | 	at decimals.security$remove_colon.invokeStatic(security.clj:34)
sequence_1  | 	at decimals.security$remove_colon.invoke(security.clj:33)
sequence_1  | 	at decimals.security$header_key.invokeStatic(security.clj:42)
sequence_1  | 	at decimals.security$header_key.invoke(security.clj:39)
sequence_1  | 	at decimals.security$apikey_auth.invokeStatic(security.clj:56)
sequence_1  | 	at decimals.security$apikey_auth.invoke(security.clj:54)
sequence_1  | 	at decimals.interceptors$fn__21455.invokeStatic(interceptors.clj:162)
sequence_1  | 	at decimals.interceptors$fn__21455.invoke(interceptors.clj:161)
sequence_1  | 	at io.pedestal.interceptor.chain$try_f.invokeStatic(chain.clj:54)
sequence_1  | 	... 44 common frames omitted
sequence_1  | INFO  i.p.http.impl.servlet-interceptor  - {:msg "sending error", :message "Internal server error: exception", :line 215}

Somehow related to the fact that I'm using a false authentication. If I remove the auth header, server gives me {"message":"Invalid credentials."}.

Any ideas?

How would one model an "auth-capture" scenario?

I'm exploring sequence and trying to apply it to a problem where an account may "promise" to pay up to X, and then end up paying Y (where typically Y < X). So if I start with 100 in my ledger, and promise 50, my balance is still 100 from one perspective, but I only have 50 left to promise. Then as that promise is resolved as 30 instead of 50, my balance is 70, and I have 70 left to promise.

Would this be best as two separate ledgers? One ledger with metadata distinguishing promises from movements (but balance computation becomes hard)?

Thanks!

Unable to find image on docker

can you help me with this ?

when docker will search a image he does this error