This repository contains both the front-end (Coordinator - Python) and back-end (Peer - Golang) code to deliver the ftillite body of work.
Folder | Description |
---|---|
Coordinator | Contains the client and Compute Manager code written in Python. The coordinator node will run a single instance of both the Client and Compute Manager. |
Peer | Contains the Segment Manager code written in golang. Each peer node will run a Segment Manager instance. |
DataGenerator | Contains code to generate synthetic data and load this data into the postgreSQL db. |
Documentation | Contains all documentation relevant to a technical user / developer of the repo. |
Below is a diagram that conveys the core solution design.
<Removed>
<Removed>
The FTILLITE solution can be run using docker-compose. The docker-compose.yml file defines two profiles, all
and debug
. The all
profile runs each service specified in the compose file, and the debug
profile runs everything except the first Peer segment, to allow for this to be run in debug more within an IDE. Open a console at the root directory of the solution and enter
make start_all
This will start a similar set of containers that are configured to run in AWS. The Jupyter Notebooks and RabbitMQ admin interfaces can both be accessed in the same way as described for AWS but by substituting localhost as the IP Address. Note there may be a need to change connection strings in the notebook if you're running a hybrid between an IDE and docker containers.
To stop the containers use
make stop_all
Note that with any updates to the repository you need to run the following for those changes to apply:
make build_all
make start_all
To update the database schema ensure that the postgres container is running. From the Database folder run the update-postgres-container.sh
script. Note that you may need to change the name of the target container within the script from ftillite-postgres-1
to ftillite_postgres_1
depending on your environment.
To populate the database to use the auxdb_read command, perform the following with the FTILLITE system running via the above docker-compose command. Note that you will want to use a linux environment such as WSL due to Numpy limitations on windows systems for the data generation.
make generate_data
Note that you can tweak parameters on the data generation; see DataGenerator/README.MD for details.
Also note that if you have an older version of the database running or have changed any of the db details in the .env file, then you'll need to do the following in order to get the schemas and users setup correctly for each Peer node:
-
Run
make stop_all
-
Delete the postgres-data folder
-
Run
make start_all
The following tools are expected to be installed on your machine:
- Go
- Docker
- NVIDIA toolkit 11.4 (optional)
Before the Go code can be compiled you must have built libftcrypto and have the shared object file and header in Peer/lib/. Building libftcrypto can be done by:
- Initialising and updating the Git submodule (Peer/lib/libftcrypto):
- git submodule init
- git submodule update
- If you don't have the NVIDIA toolkit available locally, update the build-libftcrypto.sh script to use docker and comment out the use of nvcc.
- Run build-libftcrypto.sh to build the shared object file and copy the header file to Peer/lib
- Make sure to set the
CUDA_ARCH
andCUDA_VERSION
environment variables to match your system's specifications.
Note: If you're using Visual Studio Code, you may need to open the Peer/segment/types/ed25519array_gpu.go file and click the "regenerate cgo definitions" link at the top of the file to stop VSCode from showing errors in the codebase.
There are two changes required when updates are available from libftcrypto, both should be done to keep everything consistent.
Updating libftcrypto in the repository is required so that local builds of ftillite Peer nodes are linked against the same libftcrypto library.
To update the repository:
- Create a new branch/merge request for the update change.
- From within the submodule path,
Peer/lib/libftcrypto
, rungit checkout master && git pull
to grab the latest from upstream master. - Navigate back out of the submodule path and commit the change.
- Run
Peer/lib/build-libftcrypto.sh
to rebuild libftcrypto and update your local library (i.e.,Peer/lib/libftcrypto.so
).
To run the python tests, you will need to install all packages, including the ftillite library firstly with:
cd Coordinator
pip install -r requirements.txt
pip install -e ftillitelibrary
Then to run the tests, run the either of the following in the project root directory:
make test_py_non_gpu
- for non-GPU environmentsmake test_py_gpu
- for GPU environments
To profile the code, you can either run the following:
python -m cProfile <Python Module name>.py
Or you can add code programatically, which allows you to output data to a CSV.
See Coordinator/ftillitelibrary/tests/check_listmap.py
for an example of this.
Note that cumtime
and ncalls
are the most crucial metrics, as all others are obscured due to the heavy lifting being done on the Go-Side.
To run the tests, run the either of the following in the project root directory:
make test_go_non_gpu
- for non-GPU environmentsmake test_go_gpu
- for GPU environments
Note that it is recommended not to use the VsCode GUI as test cases that throw a panic will silently fail.
In order to run test cases for the Ed25519 type, you will either need to run you test cases with the flag --gpu=true
or in Vscode, add a settings.json to the root directory .vscode
of this repo with the below contents:
{
"go.testEnvVars": {
"gpu": "true"
}
}
You may also need to extend the VScode setting Test Timeout
to 30m
instead of the default 30s
.
Also when adding a python test, you will need to:
- Name the Python module / file
test_<rest of name>.py
and - functions within this module will also need to be defined as
def test_<test name>()
for pytest to register these new test cases.
The following steps can be used to implement a new FTILLite command. We'll implement the sha3_256
as an example, as it requires changes to both the Python library and the Go backend.
The Coordinator/ftillitelibrary/ftillite
directory contains the ftillite
Python module, in which is the client.py
file containing the class hierarchy described in the FTILLite documentation.
SHA3_256
is defined as a top-level function of the ftillite
module, so the following code can be placed anywhere outside
any of the class blocks in the client.py
file. To keep the file organised, these are generally placed at the bottom of the file.
def sha3_256(data):
return data.context()._exec_command(f'sha3_256 1 {data.handle()}')
Explanation of the code:
-
The
data
parameter is Bytearray Array whose elements will be hashed. The result will be a Bytearray Array of the same length, but strictly 32 bytes wide, containing the hashes. -
.context()
returns the FTILContext associated with the FTILLite variable (data
). -
_exec_command
is a function defined the FTILContext which sends the command to the message broker. -
"sha3_256 1 {data.handle()}"
is the command string sent to the Peers (i.e., the Go backend). Before the command string is sent, the following preprocessing steps are applied:sha3_256
will be prefixed withcommand_
1
will be replaced with a new variable handle.0
can be used to indicate that no handle should be created, which are typically used for in-place commands.{data.handle()}
is replaced with the handle of thedata
variable.
The exact format of the command string will vary between commands, however the important thing is that both the Python and Go code agree on what the format is.
The segment
package in Peer/segment
contains the Go implementations of the commands. To define a new command in Go:
- Define a new command constant in
command_constants.go
, e.g.,CommandSHA3256 = "command_sha3_256"
. The constants value must match the command name as sent by the Python library. - Create a new Go function which will contain the logic of the command. The type of the function is
func ([]string) (string, error)
, which has:- An array of parameters sent by Python, in the case of
sha3_256
, the first parameter will be the result handle, and the second thedata
variable handle. - A string result, or an error result. The format of the string result can be found in the
FTILContext :: _parse_result
Python function in theclient.py
file.
- An array of parameters sent by Python, in the case of
- Register the function in
command_registration.go
, e.g.,s.RegisterCommand(CommandSHA3256, s.sha3_256)
. - Write unit tests.
<Removed>