steemit / hivemind Goto Github PK
View Code? Open in Web Editor NEWDeveloper-friendly microservice powering social networks on the Steem blockchain.
License: MIT License
Developer-friendly microservice powering social networks on the Steem blockchain.
License: MIT License
store:
Example from hivemind/hive/db/schema.py
from sqlalchemy.dialects.mysql import (
CHAR, SMALLINT, TINYINT,
TINYTEXT, MEDIUMTEXT, DOUBLE,
)
Hivemind needs to know if jussi is configured so it can upgrade to batch requests for certain calls.
Use hive_api.py
as the starting point.
Currently hive indexer logging is basic console output, though tuned to produce useful output at the INFO level at a reasonable volume. Hive server logging could use some work. And generally there's a mix of prints and logger.getLogger, with hacks for other packages' noisy loggers (sa & jsonrpcserver). Hive's logging would benefit from a refactoring and polishing.
logger
config, or use other package?This issue is a container for various test results
I tried updating hive to the latest version, but the schema is out of date.
2017-09-28T10:24:00.298529363Z sqlalchemy.exc.OperationalError: (_mysql_exceptions.OperationalError) (1054, "Unknown column 'is_valid' in 'field list'")
Could you please add db migrations, or at least publish a simple textfile with SQL required to upgrade /w dates or commit hashes?
if the process is behind on blocks by a certain threshold it needs to go back to the sync routine
todo:
consider:
hive_posts_cache:
depth
, get_post_stats
valsHive could accept a comment along with a reblog. How would we handle multi-reblogs?
work began in https://github.com/steemit/hivemind/tree/postgres branch
21 specific records on 2018-01-22 from 19:33:36 to 22:03:51 are refusing to update.
SELECT * FROM hive_posts_cache WHERE is_paidout = '0' AND payout_at < '2018-01-24 16:33:42'
Within a period of 29 blocks, post 28009432
was created, deleted, and re-created. Sync detected that it was about to skip indexing this post and aborted.
The solution is likely to sort cache list by id before writing. Deleted posts may be breaking the assumption about sequential inserts when we process blocks in batch.
Provides more flexibility in how we interpret data, and leaves possibilities open for UI and reports.
It doesn't make sense to force-override the community field for comments, since they would not be accessible anyway. They should still be part of the tree, just flagged automatically.
For root posts, it's not yet clear which is the ideal approach.
Where to use env vars vs. args?
Implement a block streamer which lets us trail N blocks behind head, possibly adjusting to network conditions.
Leftover items from #242 to evaluate:
Tier 1
Tier 2
Add new sync strategy: sync from jussi, if it's configured. It allows us to request blocks in large batches.
One way to approximate the current rep score is to sum of all of a user's posts net_rshares. Where this differs from follow
plugin is that users with lower rep can bring down those with higher rep.
In the short term it may be ideal to keep it in steemd (see steemit/steem#1425). Long term, we may want to implement some other algorithm entirely.
This happens sometimes. Possibly due to db error log?
Oct 11 08:14:11 docker/cd60a907a168[7347]: [SYNC] Got block 16231286 (68.7/s, 1096rps 73wps) -- 0.0m remaining
Oct 11 08:14:11 docker/cd60a907a168[7347]: [INIT] *** Initial sync complete. Rebuilding cache. ***
Oct 11 08:14:11 docker/cd60a907a168[7347]: [INIT] Found 15034490 missing post cache entries
Oct 11 08:14:56 kernel: [1692646.139464] device-mapper: thin: 253:2: switching pool to out-of-data-space (error IO) mode
Oct 11 08:14:56 kernel: [1692646.144088] EXT4-fs warning (device dm-4): ext4_end_bio:314: I/O error -28 writing to inode 528049 (offset 1048576000 size 8388608 starting block 2519552)
Oct 11 08:14:56 kernel: [1692646.150472] Buffer I/O error on device dm-4, logical block 2519552
Oct 11 08:14:56 kernel: [1692646.153929] Buffer I/O error on device dm-4, logical block 2519553
Oct 11 08:14:56 kernel: [1692646.157063] Buffer I/O error on device dm-4, logical block 2519554
Oct 11 08:14:56 kernel: [1692646.160150] Buffer I/O error on device dm-4, logical block 2519555
Oct 11 08:14:56 kernel: [1692646.163072] Buffer I/O error on device dm-4, logical block 2519556
Oct 11 08:14:56 kernel: [1692646.165959] Buffer I/O error on device dm-4, logical block 2519557
Oct 11 08:14:56 kernel: [1692646.168900] Buffer I/O error on device dm-4, logical block 2519558
Oct 11 08:14:56 kernel: [1692646.171721] Buffer I/O error on device dm-4, logical block 2519559
Oct 11 08:14:56 kernel: [1692646.174681] Buffer I/O error on device dm-4, logical block 2519560
Oct 11 08:14:56 kernel: [1692646.177909] Buffer I/O error on device dm-4, logical block 2519561
Oct 11 08:14:56 kernel: [1692646.181019] EXT4-fs warning (device dm-4): ext4_end_bio:314: I/O error -28 writing to inode 528049 (offset 1048576000 size 8388608 starting block 2519808)
Oct 11 08:14:56 kernel: [1692646.187836] EXT4-fs warning (device dm-4): ext4_end_bio:314: I/O error -28 writing to inode 528049 (offset 1048576000 size 8388608 starting block 2520064)
Oct 11 08:14:56 kernel: [1692646.194356] EXT4-fs warning (device dm-4): ext4_end_bio:314: I/O error -28 writing to inode 528049 (offset 1048576000 size 8388608 starting block 2520320)
Oct 11 08:14:56 kernel: [1692646.200836] EXT4-fs warning (device dm-4): ext4_end_bio:314: I/O error -28 writing to inode 528049 (offset 1048576000 size 8388608 starting block 2520576)
Oct 11 08:14:56 kernel: [1692646.207192] EXT4-fs warning (device dm-4): ext4_end_bio:314: I/O error -28 writing to inode 528049 (offset 1048576000 size 8388608 starting block 2520832)
Oct 11 08:14:56 kernel: [1692646.214593] EXT4-fs warning (device dm-4): ext4_end_bio:314: I/O error -28 writing to inode 528049 (offset 1056964608 size 8388608 starting block 2521088)
Oct 11 08:14:56 kernel: [1692646.221018] EXT4-fs warning (device dm-4): ext4_end_bio:314: I/O error -28 writing to inode 528049 (offset 1056964608 size 8388608 starting block 2521344)
Oct 11 08:14:56 kernel: [1692646.227704] EXT4-fs warning (device dm-4): ext4_end_bio:314: I/O error -28 writing to inode 528049 (offset 1056964608 size 8388608 starting block 2521600)
Oct 11 08:14:56 kernel: [1692646.234261] EXT4-fs warning (device dm-4): ext4_end_bio:314: I/O error -28 writing to inode 528049 (offset 1056964608 size 8388608 starting block 2521856)
Oct 11 08:14:56 kernel: [1692646.271736] JBD2: Detected IO errors while flushing file data on dm-4-8
Oct 11 08:15:05 kernel: [1692655.338808] EXT4-fs warning: 287 callbacks suppressed
Oct 11 08:15:05 kernel: [1692655.342246] EXT4-fs warning (device dm-4): ext4_end_bio:314: I/O error -28 writing to inode 528049 (offset 0 size 0 starting block 2519552)
Oct 11 08:15:05 kernel: [1692655.351639] buffer_io_error: 75798 callbacks suppressed
Oct 11 08:15:05 kernel: [1692655.354860] Buffer I/O error on device dm-4, logical block 2519552
Oct 11 08:15:05 kernel: [1692655.358579] Buffer I/O error on device dm-4, logical block 2519553
Oct 11 08:15:05 kernel: [1692655.362446] Buffer I/O error on device dm-4, logical block 2519554
Oct 11 08:15:05 kernel: [1692655.366161] Buffer I/O error on device dm-4, logical block 2519555
Oct 11 08:15:05 kernel: [1692655.370238] Buffer I/O error on device dm-4, logical block 2519556
Oct 11 08:15:05 kernel: [1692655.373862] Buffer I/O error on device dm-4, logical block 2519557
Oct 11 08:15:05 kernel: [1692655.377500] Buffer I/O error on device dm-4, logical block 2519558
Oct 11 08:15:05 kernel: [1692655.381224] Buffer I/O error on device dm-4, logical block 2519559
Oct 11 08:15:05 kernel: [1692655.384238] Buffer I/O error on device dm-4, logical block 2519560
Oct 11 08:15:05 kernel: [1692655.387449] Buffer I/O error on device dm-4, logical block 2519561
Oct 11 08:15:05 kernel: [1692655.390647] EXT4-fs warning (device dm-4): ext4_end_bio:314: I/O error -28 writing to inode 528049 (offset 1361182720 size 6160384 starting block 2519808)
Oct 11 08:15:05 kernel: [1692655.402506] EXT4-fs warning (device dm-4): ext4_end_bio:314: I/O error -28 writing to inode 528049 (offset 1361182720 size 6160384 starting block 2520064)
Oct 11 08:15:05 kernel: [1692655.411143] EXT4-fs warning (device dm-4): ext4_end_bio:314: I/O error -28 writing to inode 528049 (offset 1361182720 size 6160384 starting block 2520320)
Oct 11 08:15:05 kernel: [1692655.420756] EXT4-fs warning (device dm-4): ext4_end_bio:314: I/O error -28 writing to inode 528049 (offset 1361182720 size 6160384 starting block 2520576)
Oct 11 08:15:05 kernel: [1692655.429972] EXT4-fs warning (device dm-4): ext4_end_bio:314: I/O error -28 writing to inode 528049 (offset 1361182720 size 6160384 starting block 2520832)
Oct 11 08:15:05 kernel: [1692655.439005] EXT4-fs warning (device dm-4): ext4_end_bio:314: I/O error -28 writing to inode 528049 (offset 1361182720 size 6160384 starting block 2521088)
Oct 11 08:15:05 kernel: [1692655.448575] EXT4-fs warning (device dm-4): ext4_end_bio:314: I/O error -28 writing to inode 528049 (offset 1361182720 size 6160384 starting block 2521344)
Oct 11 08:15:05 kernel: [1692655.457532] EXT4-fs warning (device dm-4): ext4_end_bio:314: I/O error -28 writing to inode 528049 (offset 1361182720 size 6160384 starting block 2521600)
Oct 11 08:15:05 kernel: [1692655.466855] EXT4-fs warning (device dm-4): ext4_end_bio:314: I/O error -28 writing to inode 528049 (offset 1361182720 size 6160384 starting block 2521856)
Oct 11 08:15:06 kernel: [1692655.516649] JBD2: Detected IO errors while flushing file data on dm-4-8
Oct 11 08:15:09 docker/cd60a907a168[7347]: #033[93m[SQL][58385ms] SELECT id, author, permlink FROM hive_posts WHERE is_deleted = 0 AND id > (SELECT IFNULL(MAX(post_id), 0) FROM hive_posts_cache) ORDER BY id LIMIT 1000000#033[0m
Oct 11 08:15:15 docker/cd60a907a168[7347]: Traceback (most recent call last):
Oct 11 08:15:15 docker/cd60a907a168[7347]: File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/engine/base.py", line 1182, in _execute_context
Oct 11 08:15:15 docker/cd60a907a168[7347]: context)
Oct 11 08:15:15 docker/cd60a907a168[7347]: File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/engine/default.py", line 470, in do_execute
Oct 11 08:15:15 docker/cd60a907a168[7347]: cursor.execute(statement, parameters)
Oct 11 08:15:15 docker/cd60a907a168[7347]: File "/usr/local/lib/python3.5/dist-packages/MySQLdb/cursors.py", line 250, in execute
Oct 11 08:15:15 docker/cd60a907a168[7347]: self.errorhandler(self, exc, value)
Oct 11 08:15:15 docker/cd60a907a168[7347]: File "/usr/local/lib/python3.5/dist-packages/MySQLdb/connections.py", line 50, in defaulterrorhandler
Oct 11 08:15:15 docker/cd60a907a168[7347]: raise errorvalue
Oct 11 08:15:15 docker/cd60a907a168[7347]: File "/usr/local/lib/python3.5/dist-packages/MySQLdb/cursors.py", line 247, in execute
Oct 11 08:15:15 docker/cd60a907a168[7347]: res = self._query(query)
Oct 11 08:15:15 docker/cd60a907a168[7347]: File "/usr/local/lib/python3.5/dist-packages/MySQLdb/cursors.py", line 411, in _query
Oct 11 08:15:15 docker/cd60a907a168[7347]: rowcount = self._do_query(q)
Oct 11 08:15:15 docker/cd60a907a168[7347]: File "/usr/local/lib/python3.5/dist-packages/MySQLdb/cursors.py", line 374, in _do_query
Oct 11 08:15:15 docker/cd60a907a168[7347]: db.query(q)
Oct 11 08:15:15 docker/cd60a907a168[7347]: File "/usr/local/lib/python3.5/dist-packages/MySQLdb/connections.py", line 277, in query
Oct 11 08:15:15 docker/cd60a907a168[7347]: _mysql.connection.query(self, query)
Oct 11 08:15:15 docker/cd60a907a168[7347]: _mysql_exceptions.ProgrammingError: (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '' at line 1")
Oct 11 08:15:15 docker/cd60a907a168[7347]:
Oct 11 08:15:15 docker/cd60a907a168[7347]: The above exception was the direct cause of the following exception:
Oct 11 08:15:15 docker/cd60a907a168[7347]:
need an API that works similarly to get_state
for fetching full discussions along with all relevant commenters' metadata
Responses should be as close as possible to steemd, making it easy for devs to switch to hive APIs.
Encountered a case where some rows were missing from the posts cache table. This should not be possible because rows are always written in sequence, within a transaction. It's possible this was a side-effect of dev testing.
The issue is triggered by a slight (and appropriate) tightening of the rpc request in
steem-js
. The following was added, which is triggering Chrome to decide it needs to sent an OPTIONS call before the actual post:headers: { Accept: 'application/json, text/plain, */*', 'Content-Type': 'application/json', },
https://github.com/aio-libs/aiohttp-cors looks promising.
OPTIONS call support is part of the HTTP standard. If our services properly support HTTP it will make development going forward more predictable, and stable.
It also currently eases development when using tools that assume full HTTP support, without needing to setup and run a local jussi instance.
evaluate:
root_comment
usefulAs a user of sufficient privilege within a community, I want to be able to pin 1 or more posts to the top of that community's feed; in an order chosen by me.
Spec out the classes of blacklists and how to read/write
Currently there is a loop which checks dgp until expected block is detected. This makes a lot of calls to jussi (and returned data is imprecise because calls cached by 3s). Each call can be up to 100-200ms which is a significant waste. Since blocks are at regular 3s intervals, hive should just request blocks at their ETA.
Also, once we have a known amount of idle time between blocks, it can be used for maintenance.
In condenser we need to replace:
currently with steemd all "posts" queries take a start (author,permlink) as a cursor to load successive pages from. hive just uses offset/limit. if we want to replicate existing APIs (nearly) 100% we would need to add the cursor option. this involves a bit more complexity because we'll need to perform a lookup on the column we're ordering by based on the author-permlink provided to know on which value to start. aside from the extra queries an upside is that seeking could be more efficient than simple offset; it's more infinite-scroll friendly; and, may be more consistent when result rows are 'moving' around.
for reference:
struct discussion_query {
void validate()const{
FC_ASSERT( filter_tags.find(tag) == filter_tags.end() );
FC_ASSERT( limit <= 100 );
}
string tag;
uint32_t limit = 0;
set<string> filter_tags;
set<string> select_authors; ///< list of authors to include, posts not by this author are filtered
set<string> select_tags; ///< list of tags to include, posts without these tags are filtered
uint32_t truncate_body = 0; ///< the number of bytes of the post body to return, 0 for all
optional<string> start_author;
optional<string> start_permlink;
optional<string> parent_author;
optional<string> parent_permlink;
};
maybe:
in the future we may need a summary table to assist certain queries.
potentially
tables
Hive's max memory usage should be no more than 1-2GB currently; let's set a limit (at least in dev) to verify.
https://docs.python.org/2/library/resource.html
Related to #49
Currently in condenser, resteem state is stored client-side. This makes it minimally usable but unreliable if it is to support un-resteeming.
The naive solution would be to return all resteeming accounts for each requested post, just as votes are currently. A better solution would be to specify an account context and return user-specific state from hive.
in case of fork:
Why is the python base being used in dockerfile instead of the same ubuntu-based baseimage we're using in sbds
? Reusing infrastructure wherever possible is preferred.
Particularly for get_content bottleneck.
Currently, the software relies extensively on 3 specific API calls: get_content
, get_block
, and get_dynamic_global_properties
. It calls these through steem-python
, with which there have been issues that may affect HA. It also doesn't support websockets. Evaluate whether if it's worth to run the whole stack vs purpose-specific API routines.
I guess the follow plugin doesn't necessarily enforce constraints on what is a valid username, and there is one with ศ
in it.
It breaks hive indexer like this:
INFO:sqlalchemy.engine.base.Engine:
INSERT IGNORE INTO hive_follows (follower, following, created_at, state)
VALUES (%s, %s, %s, %s) ON DUPLICATE KEY UPDATE state = %s
INFO:sqlalchemy.engine.base.Engine:('tuakanamorgan', 'najem\u202c', '2017-06-27T20:42:03', 1, 1)
INFO:sqlalchemy.engine.base.Engine:ROLLBACK
Traceback (most recent call last):
File "/usr/local/bin/hive", line 9, in <module>
load_entry_point('hivemind', 'console_scripts', 'hive')()
File "/usr/local/lib/python3.5/dist-packages/click/core.py", line 722, in __call__
return self.main(*args, **kwargs)
File "/usr/local/lib/python3.5/dist-packages/click/core.py", line 697, in main
rv = self.invoke(ctx)
File "/usr/local/lib/python3.5/dist-packages/click/core.py", line 1066, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/local/lib/python3.5/dist-packages/click/core.py", line 1066, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/local/lib/python3.5/dist-packages/click/core.py", line 895, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/local/lib/python3.5/dist-packages/click/core.py", line 535, in invoke
return callback(*args, **kwargs)
File "/app/hive/indexer/cli.py", line 22, in index_from_steemd
run()
File "/app/hive/indexer/core.py", line 421, in run
sync_from_steemd(is_initial_sync)
File "/app/hive/indexer/core.py", line 335, in sync_from_steemd
dirty |= process_blocks(blocks, is_initial_sync)
File "/app/hive/indexer/core.py", line 278, in process_blocks
dirty |= process_block(block, is_initial_sync)
File "/app/hive/indexer/core.py", line 264, in process_block
process_json_follow_op(account, op_json, date)
File "/app/hive/indexer/core.py", line 173, in process_json_follow_op
query(sql, fr=follower, fg=following, at=block_date, state=state)
File "/app/hive/db/methods.py", line 17, in query
res = conn.execute(query, **kwargs)
File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/engine/base.py", line 945, in execute
return meth(self, multiparams, params)
File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/sql/elements.py", line 263, in _execute_on_connection
return connection._execute_clauseelement(self, multiparams, params)
File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/engine/base.py", line 1053, in _execute_clauseelement
compiled_sql, distilled_params
File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/engine/base.py", line 1189, in _execute_context
context)
File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/engine/base.py", line 1405, in _handle_dbapi_exception
util.reraise(*exc_info)
File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/util/compat.py", line 187, in reraise
raise value
File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/engine/base.py", line 1182, in _execute_context
context)
File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/engine/default.py", line 470, in do_execute
cursor.execute(statement, parameters)
File "/usr/local/lib/python3.5/dist-packages/MySQLdb/cursors.py", line 234, in execute
args = tuple(map(db.literal, args))
File "/usr/local/lib/python3.5/dist-packages/MySQLdb/connections.py", line 318, in literal
s = self.escape(o, self.encoders)
File "/usr/local/lib/python3.5/dist-packages/MySQLdb/connections.py", line 225, in unicode_literal
return db.string_literal(str(u).encode(db.encoding))
UnicodeEncodeError: 'latin-1' codec can't encode character '\u202c' in position 5: ordinal not in range(256)
Saw several of these occurring in dev, not sure what the cause is:
Traceback (most recent call last):
File "/usr/local/lib/python3.5/dist-packages/bottle.py", line 862, in _handle
return route.call(**args)
File "/usr/local/lib/python3.5/dist-packages/bottle.py", line 1740, in wrapper
rv = callback(*a, **ka)
File "/app/hive/sbds/jsonrpc.py", line 64, in rpc
self.logger.error('Parse Error, Not JSON', extra=request.body.read())
File "/usr/lib/python3.5/logging/__init__.py", line 1308, in error
self._log(ERROR, msg, args, **kwargs)
File "/usr/lib/python3.5/logging/__init__.py", line 1414, in _log
exc_info, func, extra, sinfo)
File "/usr/lib/python3.5/logging/__init__.py", line 1388, in makeRecord
raise KeyError("Attempt to overwrite %r in LogRecord" % key)
KeyError: 'Attempt to overwrite 109 in LogRecord'
KeyError: 'Attempt to overwrite 51 in LogRecord'
etc
bonus
evaluate:
self
as null-comm? allows sub to trending blogs
For bulk requests, add the ability to retrieve batches in parallel, and determine optimal batch/worker sizes for jussi/steemd.
Condenser's current top-level dropdowns exist outside the theming context. They cannot respond to 'night mode' theme changes (or any others for that matter)
Though theme-dark is set, the dropdown exists outside the regular tree.
Two possible solutions (of many) add the theme class to either the <body>
tag, or the root of the tree that holds the dropdown.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.