An interactive art installation that excavates messages embedded in the Bitcoin blockchain
Home Page: https://messagesfromthemines.brangerbriz.com
License: GNU General Public License v3.0
Steps to reproduce:
Currently our free ipstack account doesn't allow for HTTPS queries (free tier is HTTP) only. This causes a mixed content error when loaded in browser as the rest of the contents are loaded over HTTPS. Solutions include:
Currently, our account free only allows 10,000 requests a month, which we may pass if the site blows up. The basic plan jumps that to 50,000.
Table columns:
Where should this live? Dedicated AWS server or elsewhere?
Have some kind of about page on the frontend so that users can learn about the project/understand what they are looking at. It would be nice to also have UI help/explanation here too.
The speed quirk in the current implementation is fine to show the project in the context of an art installation because the page is rarely refreshed. For an online version we should probably fix this though. My thought is that we could have a constant speed but have the text constantly wrap around without spacing. E.g.
# currently it works like this
[start] ip_1 | ip_2 | ip_3 [end]
# it could work like this
[start] ip_1 | ip_2 | ip_3 | ip_1 | ip_2 | ip_3 ip_1 | ip_2 [end]In my initial cloud server tests it seems like /block?id=XXXX API calls are taking between 15-20 seconds each. This may be because of network latency as I am port forwarding the JSONRPC port from bbpwn in the studio. It is not uncommon for this API endpoint to deliver 6MB of content. We will need to test running bitcoind on the AWS VPS to know what true latency will look like, but this doesn't look great just yet. It is possible that we could remove much of the JSONRPC functionality from the online version to reduce latency when exploring messages. Investigate...
Currently the frontend does not gracefully handle window resize events.
[mysql] (0.09 sec) SELECT DISTINCT SQL_CACHE block_height FROM op_return_address_messages_unique WHERE valid = 1 AND `data` LIKE CONCAT('%', HEX('hitler'), '%') ORDER BY block_height;
[http] GET /api/block?index=NaN
[error] error getting block
{ Error: ER_BAD_FIELD_ERROR: Unknown column 'NaN' in 'where clause'
at Query.Sequence._packetToError (/home/ubuntu/messages-from-the-mines/mftm-backend/node_modules/mysql/lib/protocol/sequences/Sequence.js:52:14)
at Query.ErrorPacket (/home/ubuntu/messages-from-the-mines/mftm-backend/node_modules/mysql/lib/protocol/sequences/Query.js:77:18)
at Protocol._parsePacket (/home/ubuntu/messages-from-the-mines/mftm-backend/node_modules/mysql/lib/protocol/Protocol.js:279:23)
at Parser.write (/home/ubuntu/messages-from-the-mines/mftm-backend/node_modules/mysql/lib/protocol/Parser.js:76:12)
at Protocol.write (/home/ubuntu/messages-from-the-mines/mftm-backend/node_modules/mysql/lib/protocol/Protocol.js:39:16)
at Socket.<anonymous> (/home/ubuntu/messages-from-the-mines/mftm-backend/node_modules/mysql/lib/Connection.js:103:28)
at emitOne (events.js:116:13)
at Socket.emit (events.js:211:7)
at addChunk (_stream_readable.js:263:12)
at readableAddChunk (_stream_readable.js:250:11)
--------------------
at Protocol._enqueue (/home/ubuntu/messages-from-the-mines/mftm-backend/node_modules/mysql/lib/protocol/Protocol.js:145:48)
at PoolConnection.query (/home/ubuntu/messages-from-the-mines/mftm-backend/node_modules/mysql/lib/Connection.js:208:25)
at Object.getBlock (/home/ubuntu/messages-from-the-mines/mftm-backend/src/utils.js:76:13)
at dbPool.getConnection (/home/ubuntu/messages-from-the-mines/mftm-backend/server.js:141:11)
at Ping.onOperationComplete [as _callback] (/home/ubuntu/messages-from-the-mines/mftm-backend/node_modules/mysql/lib/Pool.js:110:5)
at Ping.Sequence.end (/home/ubuntu/messages-from-the-mines/mftm-backend/node_modules/mysql/lib/protocol/sequences/Sequence.js:88:24)
at Ping.Sequence.OkPacket (/home/ubuntu/messages-from-the-mines/mftm-backend/node_modules/mysql/lib/protocol/sequences/Sequence.js:97:8)
at Protocol._parsePacket (/home/ubuntu/messages-from-the-mines/mftm-backend/node_modules/mysql/lib/protocol/Protocol.js:279:23)
at Parser.write (/home/ubuntu/messages-from-the-mines/mftm-backend/node_modules/mysql/lib/protocol/Parser.js:76:12)
at Protocol.write (/home/ubuntu/messages-from-the-mines/mftm-backend/node_modules/mysql/lib/protocol/Protocol.js:39:16)
code: 'ER_BAD_FIELD_ERROR',
errno: 1054,
sqlMessage: 'Unknown column \'NaN\' in \'where clause\'',
sqlState: '42S22',
index: 0,
sql: 'SELECT block_height, block_hash, block_timestamp FROM coinbase_messages WHERE block_height = NaN;' }
[error] error getting block
{ Error: ER_BAD_FIELD_ERROR: Unknown column 'NaN' in 'where clause'
at Query.Sequence._packetToError (/home/ubuntu/messages-from-the-mines/mftm-backend/node_modules/mysql/lib/protocol/sequences/Sequence.js:52:14)
at Query.ErrorPacket (/home/ubuntu/messages-from-the-mines/mftm-backend/node_modules/mysql/lib/protocol/sequences/Query.js:77:18)
at Protocol._parsePacket (/home/ubuntu/messages-from-the-mines/mftm-backend/node_modules/mysql/lib/protocol/Protocol.js:279:23)
at Parser.write (/home/ubuntu/messages-from-the-mines/mftm-backend/node_modules/mysql/lib/protocol/Parser.js:76:12)
at Protocol.write (/home/ubuntu/messages-from-the-mines/mftm-backend/node_modules/mysql/lib/protocol/Protocol.js:39:16)
at Socket.<anonymous> (/home/ubuntu/messages-from-the-mines/mftm-backend/node_modules/mysql/lib/Connection.js:103:28)
at emitOne (events.js:116:13)
at Socket.emit (events.js:211:7)
at addChunk (_stream_readable.js:263:12)
at readableAddChunk (_stream_readable.js:250:11)
--------------------
at Protocol._enqueue (/home/ubuntu/messages-from-the-mines/mftm-backend/node_modules/mysql/lib/protocol/Protocol.js:145:48)
at PoolConnection.query (/home/ubuntu/messages-from-the-mines/mftm-backend/node_modules/mysql/lib/Connection.js:208:25)
at Object.getBlock (/home/ubuntu/messages-from-the-mines/mftm-backend/src/utils.js:80:13)
at dbPool.getConnection (/home/ubuntu/messages-from-the-mines/mftm-backend/server.js:141:11)
at Ping.onOperationComplete [as _callback] (/home/ubuntu/messages-from-the-mines/mftm-backend/node_modules/mysql/lib/Pool.js:110:5)
at Ping.Sequence.end (/home/ubuntu/messages-from-the-mines/mftm-backend/node_modules/mysql/lib/protocol/sequences/Sequence.js:88:24)
at Ping.Sequence.OkPacket (/home/ubuntu/messages-from-the-mines/mftm-backend/node_modules/mysql/lib/protocol/sequences/Sequence.js:97:8)
at Protocol._parsePacket (/home/ubuntu/messages-from-the-mines/mftm-backend/node_modules/mysql/lib/protocol/Protocol.js:279:23)
at Parser.write (/home/ubuntu/messages-from-the-mines/mftm-backend/node_modules/mysql/lib/protocol/Parser.js:76:12)
at Protocol.write (/home/ubuntu/messages-from-the-mines/mftm-backend/node_modules/mysql/lib/protocol/Protocol.js:39:16)
code: 'ER_BAD_FIELD_ERROR',
errno: 1054,
sqlMessage: 'Unknown column \'NaN\' in \'where clause\'',
sqlState: '42S22',
index: 0,
sql: 'SELECT block_height, block_hash, block_timestamp FROM address_messages WHERE block_height = NaN;' }
/home/ubuntu/messages-from-the-mines/mftm-backend/node_modules/mysql/lib/protocol/Parser.js:80
throw err; // Rethrow non-MySQL errors
^
Error: Can't set headers after they are sent.
at validateHeader (_http_outgoing.js:491:11)
at ServerResponse.setHeader (_http_outgoing.js:498:3)
at ServerResponse.header (/home/ubuntu/messages-from-the-mines/mftm-backend/node_modules/express/lib/response.js:767:10)
at ServerResponse.contentType (/home/ubuntu/messages-from-the-mines/mftm-backend/node_modules/express/lib/response.js:595:15)
at ServerResponse.sendStatus (/home/ubuntu/messages-from-the-mines/mftm-backend/node_modules/express/lib/response.js:357:8)
at utils.getBlock (/home/ubuntu/messages-from-the-mines/mftm-backend/server.js:145:11)
at cb (/home/ubuntu/messages-from-the-mines/mftm-backend/src/utils.js:89:4)
at Query.connection.query [as _callback] (/home/ubuntu/messages-from-the-mines/mftm-backend/src/utils.js:80:39)
at Query.Sequence.end (/home/ubuntu/messages-from-the-mines/mftm-backend/node_modules/mysql/lib/protocol/sequences/Sequence.js:88:24)
at Query.ErrorPacket (/home/ubuntu/messages-from-the-mines/mftm-backend/node_modules/mysql/lib/protocol/sequences/Query.js:90:8)
mftm-backend/server.js was not written for fault tolerance. There are tons of throw err statements when certain services like JSONRPC communication fail. Remove these fails and log errors instead so that occasional errors won't crash the entire site once it is live.
Blocklist queries on initial pageload and whenever a filter is applied are very costly to the SQL server. There isn't a lot we can do about the filter queries, but we could cache the initial pageload query as the results will only change each time we parse re-parse the blockchain (and review messages). In particular, here are the queries I intend to cache:
SELECT DISTINCT block_height FROM coinbase_messages_unique ORDER BY block_height;
SELECT DISTINCT block_height FROM address_messages_unique ORDER BY block_height;
SELECT DISTINCT block_height FROM op_return_address_messages_unique ORDER BY block_height;
SELECT DISTINCT block_height FROM coinbase_messages_unique WHERE valid = 1 ORDER BY block_height;
SELECT DISTINCT block_height FROM address_messages_unique WHERE valid = 1 ORDER BY block_height;
SELECT DISTINCT block_height FROM op_return_address_messages_unique WHERE valid = 1 ORDER BY block_height;
SELECT DISTINCT block_height FROM coinbase_messages_unique WHERE nsfw = 0 ORDER BY block_height;
SELECT DISTINCT block_height FROM address_messages_unique WHERE nsfw = 0 ORDER BY block_height;
SELECT DISTINCT block_height FROM op_return_address_messages_unique WHERE nsfw = 0 ORDER BY block_height;
SELECT DISTINCT block_height FROM coinbase_messages_unique WHERE bookmarked = 1 ORDER BY block_height;
SELECT DISTINCT block_height FROM address_messages_unique WHERE bookmarked = 1 ORDER BY block_height;
SELECT DISTINCT block_height FROM op_return_address_messages_unique WHERE bookmarked = 1 ORDER BY block_height;
Use the same y-axis as all valid unfiltered messages.
Server logging is kind of arbitrary right now with lots of dangling console.log() statements left over from testing. This should be cleaned up.
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.