This is toolchain compsed by many small tools. The goal has been reduce 1 functionality per tool. below, or on the website you can find more contextual details. If you want get in touch, claudio at tracking dot exposed is the email to write to, the link lead to the PGP key.
Clone the repository, npm install
, have mongodb running on localhost.
There are two kind of scripts: listening services (they listen via HTTP and execute REST commands) and standalone scripts
Exists two kind of boxes in invi.sible.link system: the Vantage Point (VP) and the Coordinator. In my test infrastructure, the Coordinator is https://invi.sible.link and the three VP are ws-vp.sible.link, ams-vp.sible.link, hk-vp.sible.link
Using the code in this repository you have all the required software for replicate the analysis I'm doing. Both the boxes install the same package and needs mongodb, just they use different executables.
on the Coordinator server:
npm run storyteller
storyteller
is a binary located in bin/storyteller
using the configuration file config/storyteller.json
. It listen on the port 7000 in HTTP and serve the public website of the Coordinator box.
npm run vigile
vigile
is a binary located in bin/vigile
using the configuration file config/vigile.json
. It is consulted by the VPs to get the website to be tested.
When you are managing a campaign there is this command to be executed in the Coordinator box:
npm run social-pressure
Here a more extensive explanation on how Campaign works.
on the vantage point:
npm run exposer
These are executed when new tests and campaign are going to begin. The commands are intended to be repeated follwing the crontab scheduling chapter below.
coordinate the test for a list of websites. Requirement: a CSV list, a unique name to be associate to that test:
DEBUG=* bin/directionTool.js --csv ../amtrex/culture-list.csv --taskName culture
This command is intended to store a list of website in testing. With the following example you'll download a campaign package, a set of files respecting this specifics
[email protected]:tracking-exposed/intrex.git
cd intrex
npm i
The command above generate such output:
directionTool CSV source defined in ../intrex/italian-media-2016.csv, I hope is an absolute path +0ms
directionTool Unspecified 'needsfile' ENV, using default config/dailyNeeds.json +2ms
directionTool Using config/dailyNeeds.json as needs generator +1ms
directionTool Importing CSV ../intrex/italian-media-2016.csv +1ms
directionTool content {"needName":"basic","lastFor":{"number":28,"period":"h"},"startFrom":"midnight"} +3ms
directionTool Timeframe: startFrom midnight (midnight|now), lastFor {"number":28,"period":"h"} +1ms
directionTool Window start Mon Apr 17 2017 00:00:00 GMT+0200 end Tue Apr 18 2017 04:00:00 GMT+0200 +2ms
directionTool 39 lines → keys [site,description] 'rank' will be add +2ms
directionTool Read 39 sites, everything with rank < 100 will be stripped off +3ms
directionTool Generated 39 needs +6ms
directionTool The first is {
"subjectId": "1c40afc685a418a5098aee28a33f818f5b6c3e14",
"taskName": "intrex",
"href": "http://www.repubblica.it",
"description": "11717",
"rank": 1,
"needName": "basic",
"start": "2017-04-17T00:00:00.000Z",
"end": "2017-04-18T00:00:00.000Z",
"id": "58b5d753e732edb69d2cfdf3d683a8b1e76ce49b"
} +0ms
lib:mongo writeMany: DONE, in promises 39 objects +36ms
This command has set a list of task to be executed. The VPs contact vigile
asking if there are some task to be done. If vigile
has something to give back it log:
route:getTasks Ӫ getTasks max 30 from casa +2m
route:getTasks taskList returns 30 tasks updated for VP [casa] +76ms
route:getTasks Ϣ getTasks max 30 from casa +2m
route:getTasks taskList returns 9 tasks updated for VP [casa] +22ms
Logs the number of task provided to the VP, named casa in this log. With a maximum of 30 task returned every time, and 39 website to be tested,
If there are not new task to be delegated, vigile
log with some underscores, like this:
route:getTasks _________ casa +5ms
In the previous section we saw ↓
- how
directionTool
(inbin/directionTool
) - create
promises
(in mongodb configured inconfig/vigile.json
) - these promises are dispatch by
vigile
(bin/vigile
listening on 7200) - Vantage Point contact
vigile
when is executedbin/chopsticks
- Vantage Point get the site to be analyzed and save their results in mongodb (
config/chopsticks.json
) - Vantage Point notify to
vigile
if the analysis was successful or not,promises
get updated with this result
Then, from the box running vigile
and storyteller
we want retrieve the results obtained and do analysis, aggregation, visualizations.
DEBUG=* bin/campaignChecker.js --config config/experimentsCampaign.json --campaign intrex
- campaign: has to be a previously used
taskName
and has to be a field present inconfig/experimentsCampaign.json
This create in mongodb:
- as many entries in
evidences
as the request/response - as many entries in
surface
as the website tested - one entry in
sankeys
thug is a powerful instrument used to detect client side attacks. It extend a javascript sandbox and this suits very well to our scope. It is not mandatory for the system to work, but might provide unique results.
Following https://buffer.github.io/thug/doc/build.html
# aptitude install libboost-all-dev graphviz libffi-dev libfuzzy-dev autoconf libgraphviz-dev pkg-config python-pip
# aptitude install libtiff5-dev libjpeg8-dev zlib1g-dev libfreetype6-dev liblcms2-dev libwebp-dev tcl8.6-dev tk8.6-dev python-tk
# aptitude install python-libemu libemu-dev
# pip install pygraphviz --install-option="--include-path=/usr/include/graphviz" --install-option="--library-path=/usr/lib/graphviz/"
# pip install pyv8
# pip install thug
(otherwise I'll try with docker it don't work in the vantage point: the requirements here are a lot)
please refer to the file THUG.md
storyteller: web publisher of lists and results
There are different components because they might run in separated machines, and because some of them are HTTP servers, other are cron-based exectuion pipelines.
Additionally, to replicate the advocacy declined for CodingRights experiments, this might imply different boxes, management, etc.
Runs in the public web, 7000
- publish the results
- publish the project work-o-meter
- report to vigile on the access stats
- implement basic API
- fetch the data from machete
Some high level data and visualisation takes sense only if computed separately.
I was thinking to generalize this sequence of tasks through the toll named
machete
but it is proven an unproper planning, because the reduction process
changes too much to be generalized.
Therefore, the two scheduled execution currently implemented are:
It look at the API /api/v1/system/info
every hour, and keep track of the
load average, disk usage, memorized object in every machine contacted.
It look through all the Vantage Point to get information over the subject of a specific investigation, and update a daily report on the subject.
Runs in one admin controller machine, access restricted, 7200
execute activities in user time, no scheduled tasks
- provide direction to chopsticks, it is where test list are stored
- receive all the results of every promises resolved (choptstick, machete, socialpressure)
- receive all the anomalies reported by all the components
- provide stats for the admin on what is happening
- is managed by the adminstrators and only by them
Runs on the vantage point
perform web connection, is a pipeline, can be run in parallel, scheduled, it constantly execute itself and look for operation to do.
composition pipeline: retriveLists, phantom | thug, (phantomNavigator?) phantomCleaner | thugCleaner, reportAnomalies, fullifyPromises, mongo
Runs where chopstick run, 7300
expose via API in a raw version what the chopsticks exectutions, can't be performed by chopstick itself because this is a webserver and chopstick has different execution pattern.
Runs whenever
Tooks input from scheduled resuls, generate output for the social media and contains the logic to automatize activities
git clone [email protected]:vecna/invi.sible.link.git
mkdir bin
cd bin
ln -s ../invi.sible.link/bin/ISL-scheduled
crontab -e
*/2 * * * * bin/ISL-scheduled --task chopsticks
bin/ISL-scheduled --task campaign --campaign Brasil --config config/brlocal.json --taskName Brasil
* campaign: is looked in the config/brlocal as the campaign selector
* taskName: is saved in the db (evidences and surface) field `task` used as selector in `/api/v1/mostUniqueTrackers/$taskName`