att / qujata Goto Github PK
View Code? Open in Web Editor NEWNext-Gen Crypto. Quantified.
License: MIT License
Next-Gen Crypto. Quantified.
License: MIT License
Add a new api to list the message size options in bytes: 0 bytes, 1 byte, 2 bytes, 100 bytes, 1KB, 100KB, 200KB, 1MB, 2MB, 10MB.
Implement bytes throughput and request count throughput metrics in a single test run of an algorithm and its selected amount of iterations.
After calculating the new metrics, you will add them as two new fields to our results JSON, requestThroughput
and bytesThroughput
, to the existing JSON data structure. These fields will represent the per-second throughput of algorithm executions and data processed respectively.
These new metrics shall also be visualized in the UI.
The current JSON data structure appears as follows:
{
"algorithm": "hqc256",
"id": 189,
"iterations": 10000,
"results": {
"averageCPU": 3.1,
"averageMemory": 476
}
}
The proposed JSON structure with the new fields is as follows:
{
"algorithm": "hqc256",
"id": 189,
"iterations": 10000,
"results": {
"averageCPU": 3.1,
"averageMemory": 476,
"requestThroughput": 100,
"bytesThroughput": 520
}
}
Data Collection: Per run, we need the number of iterations, the total message size (=iterations * message size), and the total time taken for the test run.
Message Size Calculation: Calculate the size of an HTTP request by adding the size of the request line, headers, and the payload (for POST requests). The size of the request line includes the HTTP method, URL, and HTTP version. Headers include fields like Host, User-Agent, Accept, and others. The payload is the data being sent to the server (for POST requests).
Python snippet:
# Calculate the size of the request line (method + URL + HTTP version)
# We add 4 to account for the 3 spaces and 1 end-of-line character that are part of the request line
request_line_size = sys.getsizeof(method) + sys.getsizeof(url) + sys.getsizeof('HTTP/1.1') + 4
# Calculate the size of the headers
headers_size = sum(sys.getsizeof(header) for header in headers.values())
# Calculate the size of the payload
data_size = sys.getsizeof(data)
# Add up the sizes to get the total size
total_size = request_line_size + headers_size + data_size
Throughput Calculation: Calculate the throughput per second for iterations and bytes:
For requests count: Throughput in count/second = (Total Count of Requests) / (Total Time in Seconds)
For bytes: Throughput in bytes/second = (Total Data in Bytes) / (Total Time in Seconds)
If the total time is less than a second, the throughput has to be scaled up proportionally to represent an estimate for a full second.
For example, if the total time is 30 seconds (0.5 seconds) and there are 500 requests, the formula becomes 500 / 0.5 resulting in 1000 requests per second.
Field Addition: After calculating the throughput values, add them to the respective JSON objects under the results
field as requestThroughput
and bytesThroughput
.
Metrics in DB: When saving the throughput values, make sure to name the metrics as follows: requestThroughputPerSecond
and bytesThroughputSecond
.
Visualization: After adding the new fields to the JSON, add the metrics to the UI visualization, both the table and the graphs.
The size of an HTTP request is correctly calculated by adding the size of the request line, headers, and the payload (for POST requests).
The values for requestThroughput
correctly represent the number of iterations that would be completed per second, given the total count of iterations and the total time for the test run. This should be calculated even if the total time is less than a second, in which case the throughput should be scaled up proportionally.
The values for bytesThroughput
correctly represent the volume of data that would be processed per second, given the total volume of data and the total time for the test run. This should be calculated even if the total time is less than a second, in which case the throughput should be scaled up proportionally.
The new fields requestThroughput
and bytesThroughput
are correctly added to the respective JSON objects under the results
field.
These new metrics show up in the UI results table, and the graphs.
Collect new metrics - TLS Handshake, Data Transfer Time.
Read this blog post for help:
https://blog.josephscott.org/2011/10/14/timing-details-with-curl/
For all supported Qujata algorithms, support the following
Be the first to suggest how to break this feature into individual tasks
Create a table and graphs out of the JSON in the researcher Qujata portal.
Implement it according to the metrics and params that are already available.
Summary table: Algorithm | Iterations | Message size | CPU | Memory | Error rate | Bytes throughput | Messages throughput | TLS handshake time
Bar chart graphs:
X = Algorithm + Iteration + Message size, Y = Metric.
Total graphs = number of metrics count(testRuns[x].results])
Each combination of iterations + message size + algorithm will be represented as another “Bar” in the graph.
Example:
Algorithm1.cpu = 25.5 (1000, 20)
Algorithm1.cpu = 40 (1000, 1024)
Algorithm2.cpu = 30.5 (1000, 20)
Algorithm2.cpu = 50 (1000, 1024)
Multi series graphs:
Line per algorithm, Y = Metric (e.g. Avg CPU), X = message size
Line per algorithm, Y = Metric (e.g. Avg CPU), X = number of iterations
Total graph combinations = numbers of metrics * 2 (1 message size + 1 for iterations)
Filters:
Algorithm(s) by name
Optional: Algorithm(s) by family (e.g. all BIKELs, all FRODOs.)
Optional: Algorithm(s) by NIST round (e.g. R1, R3, R5.)
Number of iterations
Message size
Allow exporting Test Suite results, with all its 'Test-Runs', to a file.
JSON example:
{
"id": 1,
"name": "test1",
"description": "test1",
"start_time": "Thu, 14 Dec 2023 15:37:31 GMT",
"end_time": "Thu, 14 Dec 2023 15:38:39 GMT",
"environment_info": {
"codeRelease": "1.1.0",
"cpu": "RELACE_WITH_CPU",
"cpuArchitecture": "RELACE_WITH_CPU_ARCHITECTURE",
"cpuClockSpeed": "RELACE_WITH_CLOCK_SPEED",
"cpuCores": 0,
"nodeSize": "RELACE_WITH_NODE_SIZE",
"operatingSystem": "RELACE_WITH_OPERATING_SYSTEM",
"resourceName": "RELACE_WITH_RESOURCE_NAME"
},
"testRuns": [
{
"algorithm": "Algorithm1",
"iterations": 1000,
"messageSizeBytes": 1024,
"results":
{
"averageCPU": 25.5,
"averageMemory": 512,
"errorRate": 0.05,
"bytesThroughput": 2048000,
"messagesThroughput": 500,
"averageTLSHandshakeTime": 10.2
}
}
...
]
Currently the cpu metric represents cpu cores usage
this metric should be changed to CPU_CORES_USAGE
and a new metric CPU_PRECENTAGE_USAGE should be created
Implement a scheduler that will set up test runs. The purpose of it is running multiple predefined experiments, and let them run on the background.
when application is loaded, run first test to find the maximum concurrency in the current environment
The idea is to analyze our reports and to provide coherent insights on the data that we collected.
This is one of the main goals of this project.
Figma
Insights section "View more"
Insights section "View less"
Analysis should be able to answer the following questions for our users and ourselves:
insights
.insights
property is populated.insights
property. Make sure to review the genAI generated insights and modify as needed.Out of scope: automate the genAI insights generation from an Azure instance of the latest gpt model after a run is executed.
calculate the total requests run time and weight and return in response.
Currently, the entire application is working without the message size parameter.
As a backend developer, we need to make sure the application is making her analyze process including the message size.
As a frontend developer, it needs to be presented on the following section:
Predefined list of options in the UI
0 bytes, 1 byte, 2 bytes, 100 bytes, 1KB, 100KB, 200KB, 1MB, 2MB, 10MB
These values will be provided in bytes from the backend.
The user should be able to add more options as needed. (In bytes)
The dropdown should take the bytes that are given as input and show them in the correct unit.
The number itself in the dropdown should not exceed 3 digits. (500 KB is ok, 1024KB is not OK. Should be 1 MB).
Below is a JS library + usage snippet that can help with that.
const filesize = require('filesize');
function convertBytesToHumanReadable(bytesValue) {
/**
* Convert bytes to human-readable units with a maximum of 3 digits.
*/
return filesize(bytesValue, {round: 3});
}
// Example usage:
const bytesValue1 = 500 * 1024; // 500 KB
const bytesValue2 = 1024 * 1024; // 1024 KB
console.log(convertBytesToHumanReadable(bytesValue1)); // Output: 500.000 KB
console.log(convertBytesToHumanReadable(bytesValue2)); // Output: 1.000 MB
We should consider the "message size" parameter throughout the entire application.
Support multiple platforms on which our benchmarking orchestration can run:
Analyze difference in results between the platforms
Add a new param to our experiments: message size. Send different sizes of payload in every cURL request to the NGINX server.
Display line charts to examine the effect of message size and PQ algorithms.
An article on Medium platform
Calculate the actual concurrency and save as a metric in DB
Implement and deploy a react-based portal with JSON visualizations of selected test suites to GitHub pages.
Static page showing static results exported from our findings.
Figma
Report Selector, No navigation
React app spin up switch: The "View experiments page" should have 2 modes. The first one is for local run / environment run (already implemented), where you have a backend to run your experiments.
The second mode is for the benchmarking reports, displaying our experiment runs from a folder in our main git branch.
You need to implement the switch of how you spin up the React application (There is a PR on it).
There is an environment variable that you should rely on, which will tell you in which mode you should spin up the app.
In the static mode, you should spin up the app with a different main page: View experiments page with its static mode.
View experiments page enhancement - Static mode: page report state must have include the following:
Our static portal will be deployed from our main code build, but with a build version dedicated to "Pages".
This dedicated build will have a few tweaks:
graph TD
Dev("Developer") -->|Push changes| GH("GitHub Repo")
GH -->|Trigger workflow| GA("GitHub Actions")
GA --> BE("Environment Configuration")
subgraph DockerBuild["Docker Build"]
BE -->|BUILD_ENV=gh-pages| GHP("GitHub Pages Build Settings")
BE -->|BUILD_ENV!=gh-pages| STD("Standard Build Settings")
end
DockerBuild --> IMG("Docker Image")
IMG -->|Extract static content| SC("Static Content")
SC -->|Deploy to GitHub Pages| Pages("GitHub Pages")
classDef default fill:#ddf,stroke:#33a,stroke-width:2px, border-radius:10px;
classDef build fill:#bbf,stroke:#33a,stroke-width:2px, border-radius:10px;
classDef settings fill:#ccf,stroke:#33a,stroke-width:2px, border-radius:10px;
class DockerBuild,IMG,SC,Pages build
class BE,GHP,STD settings
Environment Variables: Define necessary environment variables in .env
files for different environments:
.env
(local development):
REACT_APP_MAIN_PAGE=MainPageLocal.js
.env.gh-pages
(GitHub Pages):
REACT_APP_MAIN_PAGE=MainPageGitHubPages.js
Fetching from our GitHub repo reports folder
Fetch All File Names in a GitHub Repo Folder
const fetchFileNames = async () => {
const apiUrl = `https://api.github.com/repos/${owner}/${repo}/contents/${path}?ref=${branch}`;
try {
const response = await fetch(apiUrl);
const data = await response.json();
const fileNames = data.filter(item => item.type === 'file').map(file => file.name);
console.log('File names:', fileNames);
} catch (error) {
console.error('Error fetching file names:', error);
}
};
Fetch a Single File by Name from a GitHub Repo
const fetchFileContent = async () => {
const fileUrl = `https://raw.githubusercontent.com/${owner}/${repo}/${branch}/${path}/${fileName}`;
try {
const response = await fetch(fileUrl);
const content = await response.text(); // Use .json() if you're fetching a JSON file
setFileContent(content);
} catch (error) {
console.error('Error fetching file content:', error);
}
};
For both snippets, you need to replace owner
, repo
, path
, and fileName
with actual values. The branch
parameter defaults to 'main'
.
Dynamic Main Page Import: Use dynamic imports based on the environment variable to load the main page component:
import React, { lazy, Suspense } from 'react';
const MainPage = lazy(() => import(`./pages/${process.env.REACT_APP_MAIN_PAGE}`));
function App() {
return (
<Suspense fallback={<div>Loading...</div>}>
<MainPage />
</Suspense>
);
}
export default App;
Ensure the Dockerfile copies the correct environment file based on the build context:
# Example Dockerfile snippet
COPY . .
RUN if [ "$BUILD_ENV" = "gh-pages" ]; then \
cp .env.gh-pages .env; \
fi
# Continue with Docker build steps...
Update the workflow to build the Docker image with the BUILD_ENV
argument and include steps for deploying to GitHub Pages:
jobs:
build-and-deploy:
runs-on: ubuntu-latest
steps:
- name: Checkout code
uses: actions/checkout@v2
- name: Build Docker image for GitHub Pages
run: docker build ./portal --file ./portal/Dockerfile --tag my-website-image --build-arg BUILD_ENV=gh-pages
# Steps for copying static content from the Docker image and deploying to GitHub Pages
This solution integrates the dynamic main page switch with the data fetching setup. The main page component is dynamically imported based on an environment variable, which allows for different main pages in different environments (e.g., local development vs. GitHub Pages). The data fetching function adapts to the data source (API or GitHub JSON files) and processes the response accordingly.
The list of algorithms that we support should always be aligned with the algorithms specified in:
https://github.com/open-quantum-safe/oqs-provider#algorithms
Here is the list of algorithm family and the algorithms we currently support:
Algorithm | Type | Quantum Safe/Hybrid | Round 1 | Round 2 | Round 3 | Round 4 | Supported algorithms |
---|---|---|---|---|---|---|---|
CRYSTALS-Dilithium | SIG | Quantum Safe | Yes | Yes | Yes | TBD | - |
CRYSTALS-Kyber | KEM | Quantum Safe | Yes | Yes | Yes | TBD | kyber1024, kyber512, kyber768, p256_kyber512, p384_kyber768, x25519_kyber768 |
FALCON | SIG | Quantum Safe | Yes | Yes | Yes | TBD | - |
SPHINCS+ | SIG | Quantum Safe | Yes | Yes | Yes | TBD | - |
BIKE | KEM | Quantum Safe | No | No | TBD | TBD | bikel1, bikel3, bikel5 |
Classic McEliece | KEM | Quantum Safe | No | No | TBD | TBD | - |
HQC | KEM | Quantum Safe | No | No | TBD | TBD | hqc128, hqc192, hqc256 |
SIKE | KEM | Quantum Safe | No | No | TBD | TBD | - |
NTRU | KEM | Quantum Safe | No | No | - | - | - |
NTRU Prime | KEM | Quantum Safe | No | No | - | - | - |
Picnic | SIG | Quantum Safe | No | No | - | - | - |
Rainbow | SIG | Quantum Safe | No | No | - | - | - |
Saber | KEM | Quantum Safe | No | No | - | - | - |
FrodoKEM | KEM | Quantum Safe | No | No | - | - | frodo1344aes, frodo1344shake, frodo640aes, frodo640shake, frodo976aes, frodo976shake |
LAC | KEM | Quantum Safe | No | - | - | - | - |
LEDAcrypt | KEM | Quantum Safe | No | - | - | - | - |
LIMA | KEM | Quantum Safe | No | - | - | - | - |
LUOV | SIG | Quantum Safe | No | - | - | - | - |
MQDSS | SIG | Quantum Safe | No | - | - | - | - |
NewHope | KEM | Quantum Safe | No | - | - | - | - |
qTESLA | SIG | Quantum Safe | No | - | - | - | - |
ROLLO | KEM | Quantum Safe | No | - | - | - | - |
RQC | KEM | Quantum Safe | No | - | - | - | - |
ThreeBears | KEM | Quantum Safe | No | - | - | - | - |
XMSS | SIG | Quantum Safe | No | - | - | - | - |
xXE | KEM | Quantum Safe | No | - | - | - | - |
KEMTLS | KEM | Hybrid | - | No | - | - | - |
OQS | KEM | Hybrid | - | No | - | - | - |
pqNTRUSign | SIG | Hybrid | - | No | - | - | - |
P-256 | KEM | Classic | - | - | - | - | prime256v1 |
P-384 | KEM | Classic | - | - | - | - | secp384r1 |
Add requestThroughput and bytesThroughput to the test suite api response json.
currently qujata-curl runs as a single pod on the k8s cluster
in order to support better concurrency, qujata-curl should run as DaemonSet on all nodes in the K8S cluster
Implement "error rate" metric. Each experiment must log the number of runs that failed.
Be the first to suggest how to break this feature into individual tasks
Change the default nginx api to post request includes body
get throughput info from curl, calculate and save the requestThroughput and bytesThroughput metrics.
Poll the backend to check the current run's status and better reflect to the user the progress of it.
We want to observe different setups of scaling with different concurrency levels.
This will be done by simulating many clients, and observing server pods scaling up.
We would like to examine the impacts of PQC on such common use cases, measuring memory, CPU, throughput and more.
We need to test the following scenarios by minimum.
IoT Device Control
• Number of Requests: 300
• Request Size: Tiny (e.g., 100 bytes)
• Algorithms: Quantum-Safe, Hybrid, Classic
• Concurrency: 30
• Scenario: Evaluate the impact of algorithms on real-time control and monitoring of IoT devices.
Healthcare Records Access
• Number of Requests: 500
• Request Size: Medium (e.g., 1MB)
• Concurrency: 50
• Scenario: Simulate a healthcare application for accessing patient records.
Social Media Surge
• Number of Requests: 1000
• Request Size: Medium (e.g., 1MB)
• Concurrency: 100
• Scenario: Emulate a social media platform during a viral event or trending topic.
Online Banking Transactions
• Number of Requests: 2000
• Request Size: Medium (e.g., 1MB)
• Concurrency: 200
• Scenario: Assess the impact of algorithms on the security and speed of financial transactions.
Ride-Sharing Peak Hours
• Number of Requests: 3000
• Request Size: Tiny (e.g., 100 bytes)
• Algorithms: Quantum-Safe, Hybrid, Classic
• Concurrency: 300
• Scenario: Simulate a ride-sharing app during rush hours in a busy city.
Online Retail Peak Sale
• Number of Requests: 5000
• Request Size: Large (e.g., 10MB)
• Concurrency: 500
• Scenario: Simulate an online retail store during a peak shopping season or sale event.
Video Streaming Service
• Number of Requests: 10,000
• Request Size: Large (e.g., 10MB)
• Algorithms: Quantum-Safe, Hybrid, Classic
• Concurrency: 1000
• Scenario: Evaluate how well your application handles a surge in video streaming requests during a major live event.
Content Delivery Network (CDN)
• Number of Requests: 20,000
• Request Size: Very Large (e.g., 100MB+)
• Algorithms: Quantum-Safe, Hybrid, Classic
• Concurrency: 2000
• Scenario: Measure how algorithms affect the speed and efficiency of delivering very large media files during a global event.
As a frontend developer, it needs to be presented on the following section:
I had some time to think about the execution of the project, as a side observer. As a Frontend developer without permissions to the Figma file, I developed an interesting way to interact with the server.
Past Experiments Exposure:
Export Features:
New Test Initiation:
Running Test Tracing:
JSON View:
Unified Graph:
Data Normalization:
Dedicated Domain:
I know it is not one of the predicted steps that have been declared in the project, but it is worth sharing with you so we can pivot or jump-start ideas worth exploring.
Implement the steps specified in:
https://github.com/jgehrcke/github-repo-stats
This will persist the traffic data and display HTML reports of our traffic
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.