Comments (2)
I think I know what the root cause of this is and why it only showed up on production. Anytime data is retrieved from a record in the tstore backend, the full tlog tree is fetched. The tlog tree contains a leaf entry for every piece of record data. The entries are light weight, but when you have 30k votes on a proposals, that is going to still be over 30k leaves in the tlog tree. Prior performance testing revealed that when retrieving the tlog tree, every 10k leaves will add roughly 100ms onto the retrieval response.
The pi summary contains the proposal status, which is a status that is determined at runtime based on various proposal metadata. Proposals in the "Approved" politeiagui tab required two different calls be made to the tstore backend in order to determine their status. The first call retrieves record metadata and the second call retrieves billing status metadata. Each of these calls result in the tlog tree being retrieved. When a tree contains 30k leaves, as a result of having roughly that many DCR ticket votes cast on it, then each tlog tree retrieval is going to take ~300ms. Two separate tree retrievals means that it will take a minimum of 600ms to determine the proposal status of an approved proposal. The politieagui list view fetches the pi summaries in batches of 5. These requests are executed serially by the backend. This means that the tlog tree retrievals alone for this batch of calls is going to take ~3000ms (5 summaries per batch * 2 tree retrievals per summary * 300ms per tree retreival).
This was overlooked during development because the tlog trees in our test enviroments and on test-proposals don't have 30k leaves.
There are two things that we can do to help alleviate this performance bottleneck.
- Have the backend run batched read requests concurrently.
- Memoize certain parts of the proposal status call. For example, once a proposal vote has finished, its proposal status can never move backwards, i.e. move back to voting or under discussion. We can take advantage of this fact and write the proposal status function to memoize certain results in memory. This memory cache should be lazy loaded and should have an upper limit for the number of proposals results that it memoizes.
from politeia.
Performance testing was done using a batch of 5 proposals with the proposal status of "active" and where each proposal had ~1000 tlog leaves. The pi summaries call was taking ~650ms to return when tested locally and without any performance improvements. The combined improvements made in 1584 and 1586 brought this response time down to ~90ms. An 86% improvement.
This should bring the production 5 second response time down to around 1 second.
from politeia.
Related Issues (20)
- Forbid @ in usernames HOT 2
- [cms] Fresh domain invoices are not shown if using an old month
- politeiavoter: Bunches ignoring interrupt signal.
- comments: Allow retrieving all comment votes.
- Unstable order of data in Proposal Bundle download HOT 6
- comments: Add temporary edits setting. HOT 1
- comments: Add `allowextradata` to the policy reply.
- multi: Add routes page sizes as plugin settings.
- politeiad: MySQL max placeholder limit hit.
- politeiad: Invalid token bug. HOT 4
- tstore: Allow plugins to cache data in the kv store.
- pictl: Return timestamps of all comments when no IDs provided.
- Empty comments are being allowed HOT 1
- mysql/mariadb select statement bug on mac
- Invalidate all sessions of a user on password change
- Cannot Submit Proposal - getting error HOT 1
- Add rate limiting to authenticated password reset HOT 1
- Comments Bundle missing for Abandoned proposals
- [cms] Add grandfathered DCCs
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from politeia.