Hey Hi!
Abhishek here
This is submission to challenge proposed by the Matic team!
Basically, I had to index Kovan net for the latest 10k blocks and store that in the database and expose that DB by API.

My first thought was to request blocks one by one.But, in search of an optimized solution, I found Batch requests, which decrease the load on network traffic by doing requests in the batch. So here, 10000 blocks are requested in batches of 100. There is a problem with this approach though, as batch.execute() doesn't return promise or has a callback, you can't tell when the batch is done executing.
Performance ⬆️
Reliability ⬇️

Reliability issue solved using "retry(blockNumber, web3)" in controllers/ dbcontroller.js
Aim : For the requests failed in batch
Desc: Some requests may get ECONNRESET, depending on our call frequency and connection
For them, we are doing one by one (sync), by putting a good amount of time between each API call.

Given requests are independent of each other, the async model is the best choice.
Performance ⬆️
Reliability ⬇️

Reliability issue solved using delayedExecute in controllers/ dbcontroller.js

Aim: To distribute the instantaneous load on the database
Desc: delayedExecute is nothing but a wrapper around batch.execute(). Consider we use batch.execute() directly, once endpoint responds, the Async Update calls falls heavily on DB, resulting in load on memory and DB. Due to this load we may miss the update, resulting in inconsistent data.
To solve this what I did is, I added delay to each execution and increased it sequentially.
Ex. For the first batch : 0 sec, Second: 12 sec, Third: 24 sec.
At 0 all counters start, the first batch tx start indexing
After 12-sec Second batch tx start indexing
Please don't assume that all tx of one batch must get indexed in 12 sec period only, it may go to the next window, most time it does as load increases.
We are not ensuring that here, all we want is the distribution of load on period.

Analogy :
Suppose you want to eat bowl of rice.
You dont go on and directly pour bowl in mouth, as your system is not capable to handle it.
And you dont go and put one grain one by one.
What you do is you make batches from bowl of rice (batchRequest) and take one batch one by one (delayedExecute).

Please go through Demo for better view

My first thought was to keep the user as key and all the tx details as value.
But then I realized that I am repeating tx details for both the "From" and "To" account.
So I was storing a lot of reductant data.
Then I optimized it by dividing a single model into two models
One for User -> Tx Hash
One for TxHash -> Details
Efficiency ⬆️

At first, I was requesting from only one endpoint, then I created 3 for better performance
Performance ⬆️

1. Clone the Repo and npm install
2. Copy paste secret.json sent over mail into the config folder (If not received ping me on [email protected])
3. npm start

1. To Index 10k blocks from latest block : localhost:3000/api/genesis
2. To get tx of user : localhost:3000/api/getdetails/:address
3. To get all transactions : localhost:3000/api/getalltx
4. To get All users : localhost:3000/api/getallusers

Controllers: You will find most of the business logic here only
Config : MongoDB URI credentials
Models: Database Models
Routes: API routes