Git Product home page Git Product logo

openmined / pysyft Goto Github PK

View Code? Open in Web Editor NEW
9.3K 199.0 2.0K 663.22 MB

Perform data science on data that remains in someone else's server

Home Page: https://www.openmined.org/

License: Apache License 2.0

Python 72.52% Shell 1.36% Jupyter Notebook 19.38% Dockerfile 0.35% JavaScript 0.66% TypeScript 1.34% CSS 0.06% PowerShell 0.01% HCL 0.12% Cap'n Proto 0.01% HTML 0.01% Svelte 4.02% Smarty 0.17%
deep-learning secure-computation pytorch privacy cryptography python syft federated-learning hacktoberfest

pysyft's Introduction



Syft Logo

Perform data science on data that remains in someone else's server

Quickstart

LinuxmacOSWindowsDockerPodmanKubernetes

Install Client

$ pip install -U syft[data_science]

Launch Server

# from Jupyter / Python
import syft as sy
sy.requires(">=0.8.6,<0.8.7")
node = sy.orchestra.launch(name="my-domain", port=8080, dev_mode=True, reset=True)
# or from the command line
$ syft launch --name=my-domain --port=8080 --reset=True

Starting syft-node server on 0.0.0.0:8080

Launch Client

import syft as sy
sy.requires(">=0.8.6,<0.8.7")
domain_client = sy.login(port=8080, email="[email protected]", password="changethis")

PySyft in 10 minutes

📝 API Example Notebooks

Deploy Kubernetes Helm Chart

Note: Assuming we have a Kubernetes cluster already setup.

1. Add and update Helm repo for Syft

helm repo add openmined https://openmined.github.io/PySyft/helm
helm repo update openmined

2. Search for available Syft versions

helm search repo openmined/syft --versions --devel

3. Set your preferred Syft Chart version

SYFT_VERSION="<paste the chart version number>"

4. Provisioning Helm Charts

helm install my-domain openmined/syft --version $SYFT_VERSION --namespace syft --create-namespace --set ingress.className="traefik"

Ingress Controllers

For Azure AKS

helm install ... --set ingress.className="azure-application-gateway"

For AWS EKS

helm install ... --set ingress.className="alb"

For Google GKE we need the gce annotation annotation.

helm install ... --set ingress.class="gce"

Deploy to a Container Engine or Cloud

  1. Install our handy 🛵 cli tool which makes deploying a Domain or Gateway server to Docker or VM a one-liner:
    pip install -U hagrid

  2. Then run our interactive jupyter Install 🧙🏽‍♂️ WizardBETA:
    hagrid quickstart

  3. In the tutorial you will learn how to install and deploy:
    PySyft = our numpy-like 🐍 Python library for computing on private data in someone else's Domain

    PyGrid = our 🐳 docker / 🐧 vm Domain & Gateway Servers where private data lives

Docs and Support

Install Notes

  • HAGrid 0.3 Requires: 🐍 python 🐙 git - Run: pip install -U hagrid
  • Interactive Install 🧙🏽‍♂️ WizardBETA Requires 🛵 hagrid: - Run: hagrid quickstart
  • PySyft 0.8.1 Requires: 🐍 python 3.10 - 3.12 - Run: pip install -U syft
  • PyGrid Requires: 🐳 docker, 🦦 podman or ☸️ kubernetes - Run: hagrid launch ...

Versions

0.9.0 - Coming soon...
0.8.7 (Beta) - dev branch 👈🏽 API - Coming soon...
0.8.6 (Stable) - API

Deprecated:

PySyft and PyGrid use the same version and its best to match them up where possible. We release weekly betas which can be used in each context:

PySyft (Stable): pip install -U syft
PyGrid (Stable) hagrid launch ... tag=latest

PySyft (Beta): pip install -U syft --pre
PyGrid (Beta): hagrid launch ... tag=beta

HAGrid is a cli / deployment tool so the latest version of hagrid is usually the best.

What is Syft?

Syft

Syft is OpenMined's open source stack that provides secure and private Data Science in Python. Syft decouples private data from model training, using techniques like Federated Learning, Differential Privacy, and Encrypted Computation. This is done with a numpy-like interface and integration with Deep Learning frameworks, so that you as a Data Scientist can maintain your current workflow while using these new privacy-enhancing techniques.

Why should I use Syft?

Syft allows a Data Scientist to ask questions about a dataset and, within privacy limits set by the data owner, get answers to those questions, all without obtaining a copy of the data itself. We call this process Remote Data Science. It means in a wide variety of domains across society, the current risks of sharing information (copying data) with someone such as, privacy invasion, IP theft and blackmail will no longer prevent the vast benefits such as innovation, insights and scientific discovery which secure access will provide.

No more cold calls to get access to a dataset. No more weeks of wait times to get a result on your query. It also means 1000x more data in every domain. PySyft opens the doors to a streamlined Data Scientist workflow, all with the individual's privacy at its heart.

Terminology

👨🏻‍💼 Data Owners

👩🏽‍🔬 Data Scientists

Provide datasets which they would like to make available for study by an outside party they may or may not fully trust has good intentions.

Are end users who desire to perform computations or answer a specific question using one or more data owners' datasets.

🏰 Domain Server

🔗 Gateway Server

Manages the remote study of the data by a Data Scientist and allows the Data Owner to manage the data and control the privacy guarantees of the subjects under study. It also acts as a gatekeeper for the Data Scientist's access to the data to compute and experiment with the results.

Provides services to a group of Data Owners and Data Scientists, such as dataset search and bulk project approval (legal / technical) to participate in a project. A gateway server acts as a bridge between it's members (Domains) and their subscribers (Data Scientists) and can provide access to a collection of domains at once.

Community

Courses

Contributors

OpenMined and Syft appreciates all contributors, if you would like to fix a bug or suggest a new feature, please see our guidelines.

Contributors

Supporters

Open Collective

OpenMined is a fiscally sponsored 501(c)(3) in the USA. We are funded by our generous supporters on Open Collective.

Contributors

Disclaimer

Syft is under active development and is not yet ready for pilots on private data without our assistance. As early access participants, please contact us via Slack or email if you would like to ask a question or have a use case that you would like to discuss.

License

Apache License 2.0
Person icons created by Freepik - Flaticon

pysyft's People

Contributors

abyesilyurt avatar alfred-openmined-bot avatar bcebere avatar bharathgs avatar bitsofsteve avatar cereallarceny avatar dhreb avatar eelcovdw avatar gmuraru avatar hereismari avatar iamtrask avatar ionesiojunior avatar ishanmi avatar kamathhrishi avatar karlhigley avatar khoaguin avatar kiendang avatar koenvanderveen avatar lariffle avatar madhavajay avatar midokura-silvia avatar param-29 avatar peterchung241 avatar rasswanth-s avatar robert-wagner avatar shubham3121 avatar tcp avatar teo-milea avatar tudorcebere avatar yashgorana avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

pysyft's Issues

Implement Default adbmm Functionality in Base Tensor Type

User Story A: As a Data Scientist using Syft's Base Tensor type, we want to implement a default method for computing operations on a Tensor of arbitrary type. For this ticket to be complete, addbmm() should return a new tensor and addbmm_() should perform the operation inline. For a reference on the operation this performs check out PyTorch's documentation.

Acceptance Criteria:

  • If the Base Tensor type's attribute "encrypted" is set to True, it should return a NotImplemented error.
  • a unit test demonstrating the correct operation on the Base Tensor type implemented over int and float Tensors.
  • inline documentation in the python code. For inspiration on inline documentation, please check out PyTorch's documentation for this operator.

Implement Default clone Functionality for Base Tensor Type

User Story A: As a Data Scientist using Syft's Base Tensor type, we want to implement a default method for computing operations on a Tensor of arbitrary type. For this ticket to be complete, clone() should return a new tensor that's an exact copy. For a reference on the operation this performs check out PyTorch's documentation.

Acceptance Criteria:

  • If the Base Tensor type's attribute "encrypted" is set to True, it should return a NotImplemented error.
  • a unit test demonstrating the correct operation on the Base Tensor type implemented over int and float Tensors.
  • inline documentation in the python code. For inspiration on inline documentation, please check out PyTorch's documentation for this operator.

Implement Default le Functionality for Base Tensor Type

User Story A: As a Data Scientist using Syft's Base Tensor type, we want to implement a default method for computing operations on a Tensor of arbitrary type. For this ticket to be complete, le() should return a new tensor and le_() should perform the operation inline. For a reference on the operation this performs check out PyTorch's documentation.

Acceptance Criteria:

  • If the Base Tensor type's attribute "encrypted" is set to True, it should return a NotImplemented error.
  • a unit test demonstrating the correct operation on the Base Tensor type implemented over int and float Tensors.
  • inline documentation in the python code. For inspiration on inline documentation, please check out PyTorch's documentation for this operator.

Add Additive/Multiplicative Depth Tracking

NOTE: This ticket requires minimal work on ISSUE 10 to have been already started. Specifically, all work on this ticket should be over the generic Tensor type, the specs of which are described within that issue.

Description: Homomorphic Encryption has a fundamental limitation in that after a certain number of additions and multiplications, noise builds up in the cyphertext and endangers the decrypt-ability of the cyphertext. In order to make our tensors easy to use, we need to provide tracking around noise growth by tracking the number of additions and multiplications involved in creating each particular value in any given tensor. This tracking needs to either automatically bootstrap (remove noise) or flag a warning (error) if the noise is about to render the cyphertext undecryptable. Furthermore, as each encryption technique has slightly different needs in this area, our tensor object needs to implement this functionality in a way that is agnostic to the encryption technique.

Acceptance Criteria:

  1. For the Paillier, FV, (and YASHE if it's up in time) algorithms, be able to predict when a number is about to become undecryptable with > 99.99999% recall given arbitrary parameterization of the encryption algorithm. The "how" is not descriptive (be creative!) as long as its efficient. Unit Tests should demonstrate this level of accuracy.
  2. Implement the logic of (1) in such a way that the user is not aware of it (abstracted behind the operations)
  3. Implement the logic of (1) in such a way that it is agnostic of the HE algorithm (including changing the required functions in the encryption algorithm's abstract interface.
  4. Implement 1 of 3 settings as options for when a number is about to become undecryptable:
    a. the number is locally bootstrapped (make this the default if available)
    b. a call is made to an outside server (which you should also build in Syft) where the number is decrypted and then re-encrypted with the same key (make this the first backup after the default)
    c. an error is thrown (as a last resort)
  5. All functionality should work given arbitrary encryption parameters (Increasing key sizes can increase the max noise tolerated regardless of the algorithm being used)
  6. Complete benchmarking test suite for all abstract tensor operations so that their speed can be automatically reported by running the script with a given encryption algorithm. In this way, we should be able to see the tradeoff between speed and stability (bigger keys == slower but larger multiplicative depth). This should allow us to find the optimal tradeoff between the speed of any operation (which gets faster if you bootstrap more often) and the frequency of bootstrapping (which takes time).

Implement Default fmod Functionality for Base Tensor Type

User Story A: As a Data Scientist using Syft's Base Tensor type, we want to implement a default method for computing operations on a Tensor of arbitrary type. For this ticket to be complete, fmod() should return a new tensor and fmod_() should perform the operation inline. For a reference on the operation this performs check out PyTorch's documentation.

Acceptance Criteria:

  • If the Base Tensor type's attribute "encrypted" is set to True, it should return a NotImplemented error.
  • a unit test demonstrating the correct operation on the Base Tensor type implemented over int and float Tensors.
  • inline documentation in the python code. For inspiration on inline documentation, please check out PyTorch's documentation for this operator.

Implement Default chunk Functionality for Base Tensor Type

User Story A: As a Data Scientist using Syft's Base Tensor type, we want to implement a default method for computing operations on a Tensor of arbitrary type. For this ticket to be complete, chunk() should return a new tensor. For a reference on the operation this performs check out PyTorch's documentation.

Acceptance Criteria:

  • If the Base Tensor type's attribute "encrypted" is set to True, it should return a NotImplemented error.
  • a unit test demonstrating the correct operation on the Base Tensor type implemented over int and float Tensors.
  • inline documentation in the python code. For inspiration on inline documentation, please check out PyTorch's documentation for this operator.

Implement Default cauchy_ Functionality for Base Tensor Type

User Story A: As a Data Scientist using Syft's Base Tensor type, we want to implement a default method for computing operations on a Tensor of arbitrary type. For this ticket to be complete, cauchy_() should perform the operation inline. For a reference on the operation this performs check out PyTorch's documentation.

Acceptance Criteria:

  • If the Base Tensor type's attribute "encrypted" is set to True, it should return a NotImplemented error.
  • a unit test demonstrating the correct operation on the Base Tensor type implemented over int and float Tensors.
  • inline documentation in the python code. For inspiration on inline documentation, please check out PyTorch's documentation for this operator.

Implement Default half Functionality for Base Tensor Type

User Story A: As a Data Scientist using Syft's Base Tensor type, we want to implement a default method for computing operations on a Tensor of arbitrary type. For this ticket to be complete, half() should return a new tensor. For a reference on the operation this performs check out PyTorch's documentation.

Acceptance Criteria:

  • If the Base Tensor type's attribute "encrypted" is set to True, it should return a NotImplemented error.
  • a unit test demonstrating the correct operation on the Base Tensor type implemented over int and float Tensors.
  • inline documentation in the python code. For inspiration on inline documentation, please check out PyTorch's documentation for this operator.

Implement Default cumprod Functionality for Base Tensor Type

User Story A: As a Data Scientist using Syft's Base Tensor type, we want to implement a default method for computing operations on a Tensor of arbitrary type. For this ticket to be complete, cumprod() should return a new tensor. For a reference on the operation this performs check out PyTorch's documentation.

Acceptance Criteria:

  • If the Base Tensor type's attribute "encrypted" is set to True, it should return a NotImplemented error.
  • a unit test demonstrating the correct operation on the Base Tensor type implemented over int and float Tensors.
  • inline documentation in the python code. For inspiration on inline documentation, please check out PyTorch's documentation for this operator.

Implement Default eq Functionality for Base Tensor Type

User Story A: As a Data Scientist using Syft's Base Tensor type, we want to implement a default method for computing operations on a Tensor of arbitrary type. For this ticket to be complete, eq() should return a new tensor and eq_() should perform the operation inline. For a reference on the operation this performs check out PyTorch's documentation.

Acceptance Criteria:

  • If the Base Tensor type's attribute "encrypted" is set to True, it should return a NotImplemented error.
  • a unit test demonstrating the correct operation on the Base Tensor type implemented over int and float Tensors.
  • inline documentation in the python code. For inspiration on inline documentation, please check out PyTorch's documentation for this operator.

Implement Default gt Functionality for Base Tensor Type

User Story A: As a Data Scientist using Syft's Base Tensor type, we want to implement a default method for computing operations on a Tensor of arbitrary type. For this ticket to be complete, gt() should return a new tensor and gt_() should perform the operation inline. For a reference on the operation this performs check out PyTorch's documentation.

Acceptance Criteria:

  • If the Base Tensor type's attribute "encrypted" is set to True, it should return a NotImplemented error.
  • a unit test demonstrating the correct operation on the Base Tensor type implemented over int and float Tensors.
  • inline documentation in the python code. For inspiration on inline documentation, please check out PyTorch's documentation for this operator.

Implement Default dot Functionality for Base Tensor Type

User Story A: As a Data Scientist using Syft's Base Tensor type, we want to implement a default method for computing operations on a Tensor of arbitrary type. For this ticket to be complete, dot() should return a new tensor. For a reference on the operation this performs check out PyTorch's documentation.

Acceptance Criteria:

  • If the Base Tensor type's attribute "encrypted" is set to True, it should return a NotImplemented error.
  • a unit test demonstrating the correct operation on the Base Tensor type implemented over int and float Tensors.
  • inline documentation in the python code. For inspiration on inline documentation, please check out PyTorch's documentation for this operator.

Abstract Fixed Point Precision

Description: All of our Homomorphic Encryption algorithms operate on Integers under the hood. However, we want to be able to perform math on numbers of varying precision (most popularly, Floats). Thus, we need someone to put together a generic Fixed Point Precision wrapper that can provide this functionality around encryption algorithms that are Integers such that the following types are supported:

  • Byte (8-bit float)

  • Half (16-bit float)

  • Float (32-bit float)

  • Double (64-bit float)

  • Arbitrary (n-bit float)

  • Char (8-bit int)

  • Short (16-bit int)

  • Int (32-bit int)

  • Long (64-bit int)

  • Arbitrary (n-bit int)

Acceptance Criteria:

  • All types above are supported assuming 32-bit integer as input.
  • All basic operations are supported (+-*/ and inline versions).
  • Exponents can be decrypted
  • Unit Tests show that it works with all types above given Integers underneath
  • Unit Tests show that it works in the encrypted domain (using Paillier is fine)

Implement Default exp Functionality for Base Tensor Type

User Story A: As a Data Scientist using Syft's Base Tensor type, we want to implement a default method for computing operations on a Tensor of arbitrary type. For this ticket to be complete, exp() should return a new tensor and exp_() should perform the operation inline. For a reference on the operation this performs check out PyTorch's documentation.

Acceptance Criteria:

  • If the Base Tensor type's attribute "encrypted" is set to True, it should return a NotImplemented error.
  • a unit test demonstrating the correct operation on the Base Tensor type implemented over int and float Tensors.
  • inline documentation in the python code. For inspiration on inline documentation, please check out PyTorch's documentation for this operator.

Implement Default fill_() Functionality for Base Tensor Type

User Story A: As a Data Scientist using Syft's Base Tensor type, we want to implement a default method for computing operations on a Tensor of arbitrary type. For this ticket to be complete, fill_() should perform the operation inline. For a reference on the operation this performs check out PyTorch's documentation.

Acceptance Criteria:

  • If the Base Tensor type's attribute "encrypted" is set to True, it should return a NotImplemented error.
  • a unit test demonstrating the correct operation on the Base Tensor type implemented over int and float Tensors.
  • inline documentation in the python code. For inspiration on inline documentation, please check out PyTorch's documentation for this operator.

Implement Default floor Functionality for Base Tensor Type

User Story A: As a Data Scientist using Syft's Base Tensor type, we want to implement a default method for computing operations on a Tensor of arbitrary type. For this ticket to be complete, floor() should return a new tensor and floor_() should perform the operation inline. For a reference on the operation this performs check out PyTorch's documentation.

Acceptance Criteria:

  • If the Base Tensor type's attribute "encrypted" is set to True, it should return a NotImplemented error.
  • a unit test demonstrating the correct operation on the Base Tensor type implemented over int and float Tensors.
  • inline documentation in the python code. For inspiration on inline documentation, please check out PyTorch's documentation for this operator.

Implement Default mean Functionality for Base Tensor Type

User Story A: As a Data Scientist using Syft's Base Tensor type, we want to implement a default method for computing operations on a Tensor of arbitrary type. For this ticket to be complete, mean() should return a new tensor. For a reference on the operation this performs check out PyTorch's documentation.

Acceptance Criteria:

  • If the Base Tensor type's attribute "encrypted" is set to True, it should return a NotImplemented error.
  • a unit test demonstrating the correct operation on the Base Tensor type implemented over int and float Tensors.
  • inline documentation in the python code. For inspiration on inline documentation, please check out PyTorch's documentation for this operator.

Implement Default cumsum Functionality for Base Tensor Type

User Story A: As a Data Scientist using Syft's Base Tensor type, we want to implement a default method for computing operations on a Tensor of arbitrary type. For this ticket to be complete, cumsum() should return a new tensor. For a reference on the operation this performs check out PyTorch's documentation.

Acceptance Criteria:

  • If the Base Tensor type's attribute "encrypted" is set to True, it should return a NotImplemented error.
  • a unit test demonstrating the correct operation on the Base Tensor type implemented over int and float Tensors.
  • inline documentation in the python code. For inspiration on inline documentation, please check out PyTorch's documentation for this operator.

Internal, Fast "from scratch" Re-build of Paillier

Description: We are currently wrapping a Python implementation of the Paillier cryptosystem. While it is a great implementation, we want to be able to integrate one into our project a bit more closely. Furthermore, we would like to leverage lower level libraries for C, BLAS, and/or GPU integration when available. As such, this project endeavors to rebuild Paillier from scratch in a way that is more performant than the current (while also leaning on (ISSUE 12) for fixed point precision instead of having it deeply coupled to the encryption algorithm (as in python-paillier).

Acceptance Criteria:

  1. The new implementation supports the major homomorphic operations (with unit tests demonstrating)
    a. Public / Private Key Generation, Serialization, and Deserialization
    b. Encryption and Decryption
    c. Addition Between Encrypted Numbers
    d. Multiplication of an Encrypted Number by a Decrypted Scalar
    e. Leverages vector "packing" for increased performance (thus, the base data structure is a vector, not a scalar)
  2. The new implementation by default calls a highly performant runtime at least as fast as THIS
  3. The new implementation can support BLAS if available (and if BLAS is faster than (2))
  4. The new implementation can support CUDA and/or OpenCL GPU if available on a machine.
  5. This implementation is constructed in a new library called PyPaillier (OpenMined/PyPaillier) and integrated into Syft proper.
  6. Benchmarking suite documents the performance across a range of key sizes and backends.

Implement Default ge Functionality for Base Tensor Type

User Story A: As a Data Scientist using Syft's Base Tensor type, we want to implement a default method for computing operations on a Tensor of arbitrary type. For this ticket to be complete, ge() should return a new tensor and ge_() should perform the operation inline. For a reference on the operation this performs check out PyTorch's documentation.

Acceptance Criteria:

  • If the Base Tensor type's attribute "encrypted" is set to True, it should return a NotImplemented error.
  • a unit test demonstrating the correct operation on the Base Tensor type implemented over int and float Tensors.
  • inline documentation in the python code. For inspiration on inline documentation, please check out PyTorch's documentation for this operator.

Implement Default bernoulli Functionality for Base Tensor Type

User Story A: As a Data Scientist using Syft's Base Tensor type, we want to implement a default method for computing operations on a Tensor of arbitrary type. For this ticket to be complete, bernoulli() should return a new tensor and bernoulli_() should perform the operation inline. For a reference on the operation this performs check out PyTorch's documentation.

Acceptance Criteria:

  • If the Base Tensor type's attribute "encrypted" is set to True, it should return a NotImplemented error.
  • a unit test demonstrating the correct operation on the Base Tensor type implemented over int and float Tensors.
  • inline documentation in the python code. For inspiration on inline documentation, please check out PyTorch's documentation for this operator.

Implement Default ceil Functionality for Base Tensor Type

User Story A: As a Data Scientist using Syft's Base Tensor type, we want to implement a default method for computing operations on a Tensor of arbitrary type. For this ticket to be complete, ceil() should return a new tensor and ceil_() should perform the operation inline. For a reference on the operation this performs check out PyTorch's documentation.

Acceptance Criteria:

  • If the Base Tensor type's attribute "encrypted" is set to True, it should return a NotImplemented error.
  • a unit test demonstrating the correct operation on the Base Tensor type implemented over int and float Tensors.
  • inline documentation in the python code. For inspiration on inline documentation, please check out PyTorch's documentation for this operator.

Implement Default addcdiv Functionality in Base Tensor Type

User Story A: As a Data Scientist using Syft's Base Tensor type, we want to implement a default method for computing the batch matrix multiply + add of a Tensor of arbitrary type. addcdiv() should return a new tensor and addcdiv_() should perform the operation inline. For a reference on the operation this performs check out PyTorch's documentation.

Acceptance Criteria:

  • If the Base Tensor type's attribute "encrypted" is set to True, it should return a NotImplemented error.
  • a unit test demonstrating the correct operation on the Base Tensor type implemented over int and float Tensors.
  • inline documentation in the python code. For inspiration on inline documentation, please check out PyTorch's documentation for this operator.

Implement Default histc Functionality for Base Tensor Type

User Story A: As a Data Scientist using Syft's Base Tensor type, we want to implement a default method for computing operations on a Tensor of arbitrary type. For this ticket to be complete, histc() should return a new tensor. For a reference on the operation this performs check out PyTorch's documentation.

Acceptance Criteria:

  • If the Base Tensor type's attribute "encrypted" is set to True, it should return a NotImplemented error.
  • a unit test demonstrating the correct operation on the Base Tensor type implemented over int and float Tensors.
  • inline documentation in the python code. For inspiration on inline documentation, please check out PyTorch's documentation for this operator.

Implement Default masked Functionality for Base Tensor Type

User Story A: As a Data Scientist using Syft's Base Tensor type, we want to implement a default method for computing operations on a Tensor of arbitrary type. For this ticket to be complete, masked_select() should return a new tensor and masked_scatter_(), masked_fill_() should perform their operations inline. For a reference on the operation this performs check out PyTorch's documentation.

Acceptance Criteria:

  • If the Base Tensor type's attribute "encrypted" is set to True, it should return a NotImplemented error.
  • a unit test demonstrating the correct operation on the Base Tensor type implemented over int and float Tensors.
  • inline documentation in the python code. For inspiration on inline documentation, please check out PyTorch's documentation for this operator.

Implement Default log Functionality for Base Tensor Type

User Story A: As a Data Scientist using Syft's Base Tensor type, we want to implement a default method for computing operations on a Tensor of arbitrary type. For this ticket to be complete, log(), log1p(), and log_normal() should return a new tensor and log_(), log1p_(), and log_normal_() should perform the operation inline. For a reference on the operation this performs check out PyTorch's documentation.

Acceptance Criteria:

  • If the Base Tensor type's attribute "encrypted" is set to True, it should return a NotImplemented error.
  • a unit test demonstrating the correct operation on the Base Tensor type implemented over int and float Tensors.
  • inline documentation in the python code. For inspiration on inline documentation, please check out PyTorch's documentation for this operator.

Implement Default addmm Functionality in Base Tensor Type

User Story A: As a Data Scientist using Syft's Base Tensor type, we want to implement a default method for computing each operation on a Tensor of arbitrary type. addmm_() should return a new tensor and addmm_() should perform the operation inline. For a reference on the operation this performs check out PyTorch's documentation.

Acceptance Criteria:

  • If the Base Tensor type's attribute "encrypted" is set to True, it should return a NotImplemented error.
  • a unit test demonstrating the correct operation of addmm() and addmm_() on the Base Tensor type implemented over int and float Tensors.
  • inline documentation in the python code. For inspiration on inline documentation, please check out PyTorch's documentation for this operator.

Implement Default bmm Functionality for Base Tensor Type

User Story A: As a Data Scientist using Syft's Base Tensor type, we want to implement a default method for computing operations on a Tensor of arbitrary type. For this ticket to be complete, bmm() should return a new tensor. For a reference on the operation this performs check out PyTorch's documentation.

Acceptance Criteria:

  • If the Base Tensor type's attribute "encrypted" is set to True, it should return a NotImplemented error.
  • a unit test demonstrating the correct operation on the Base Tensor type implemented over int and float Tensors.
  • inline documentation in the python code. For inspiration on inline documentation, please check out PyTorch's documentation for this operator.

Implement Default gather Functionality for Base Tensor Type

User Story A: As a Data Scientist using Syft's Base Tensor type, we want to implement a default method for computing operations on a Tensor of arbitrary type. For this ticket to be complete, gather() should return a new tensor. For a reference on the operation this performs check out PyTorch's documentation.

Acceptance Criteria:

  • If the Base Tensor type's attribute "encrypted" is set to True, it should return a NotImplemented error.
  • a unit test demonstrating the correct operation on the Base Tensor type implemented over int and float Tensors.
  • inline documentation in the python code. For inspiration on inline documentation, please check out PyTorch's documentation for this operator.

Implement Base Tensor Object

In this ticket, we want to create a new basic type called a "Tensor". A Tensor is a nested list of numbers with an arbitrary number of dimensions. With one dimension, it's a list of numbers known as a "vector". With two dimensions, it's a list of lists of numbers known as a "matrix". In this ticket, we want to build our implementation of Tensor with inspiration from PyTorch's Tensor object (Tensor Docs).

So, in this ticket, you should build a basic tensor class. You should be able to pass in as an argument an arbitrary shape (number of rows, columns, etc.) when you create it. Furthermore, we want this basic class to support elementwise addition, subtraction, multiplication, and division.

Acceptance Criteria:

  • init supports initializing a tensor with an arbitrary number of dimensions and arbitrary scalar type (test with float and int)
  • elementwise operators for +-/* and their inline operators should be overridden
  • elementwise operations should check to make sure both tensors are of identical dimension
  • elementwise operators should also support (and automatically detect!) the input of a single number, which should be applied to the entire tensor (aka... multiply every number by "5").

Implement Default geometric Functionality for Base Tensor Type

User Story A: As a Data Scientist using Syft's Base Tensor type, we want to implement a default method for computing operations on a Tensor of arbitrary type. For this ticket to be complete, geometric_() should perform the operation inline. For a reference on the operation this performs check out PyTorch's documentation.

Acceptance Criteria:

  • If the Base Tensor type's attribute "encrypted" is set to True, it should return a NotImplemented error.
  • a unit test demonstrating the correct operation on the Base Tensor type implemented over int and float Tensors.
  • inline documentation in the python code. For inspiration on inline documentation, please check out PyTorch's documentation for this operator.

Implement Default addmv Functionality on Base Tensor Type

User Story A: As a Data Scientist using Syft's Base Tensor type, we want to implement a default method for computing operations on a Tensor of arbitrary type. For this ticket to be complete, addmv() should return a new tensor and addmv_() should perform the operation inline. For a reference on the operation this performs check out PyTorch's documentation.

Acceptance Criteria:

  • If the Base Tensor type's attribute "encrypted" is set to True, it should return a NotImplemented error.
  • a unit test demonstrating the correct operation on the Base Tensor type implemented over int and float Tensors.
  • inline documentation in the python code. For inspiration on inline documentation, please check out PyTorch's documentation for this operator.

Implement Default frac Functionality for Base Tensor Type

User Story A: As a Data Scientist using Syft's Base Tensor type, we want to implement a default method for computing operations on a Tensor of arbitrary type. For this ticket to be complete, frac() should return a new tensor and frac_() should perform the operation inline. For a reference on the operation this performs check out PyTorch's documentation.

Acceptance Criteria:

  • If the Base Tensor type's attribute "encrypted" is set to True, it should return a NotImplemented error.
  • a unit test demonstrating the correct operation on the Base Tensor type implemented over int and float Tensors.
  • inline documentation in the python code. For inspiration on inline documentation, please check out PyTorch's documentation for this operator.

Implement Default expand_as Functionality for Base Tensor Type

User Story A: As a Data Scientist using Syft's Base Tensor type, we want to implement a default method for computing operations on a Tensor of arbitrary type. For this ticket to be complete, expand_as() should return a new tensor. For a reference on the operation this performs check out PyTorch's documentation.

Acceptance Criteria:

  • If the Base Tensor type's attribute "encrypted" is set to True, it should return a NotImplemented error.
  • a unit test demonstrating the correct operation on the Base Tensor type implemented over int and float Tensors.
  • inline documentation in the python code. For inspiration on inline documentation, please check out PyTorch's documentation for this operator.

Implement Default index Functionality for Base Tensor Type

User Story A: As a Data Scientist using Syft's Base Tensor type, we want to implement a default method for computing operations on a Tensor of arbitrary type. For this ticket to be complete, index() and index_select() should return a new tensor and index_add_(), index_copy_(), and index_fill_() should perform the operation inline. For a reference on the operation this performs check out PyTorch's documentation.

Acceptance Criteria:

  • If the Base Tensor type's attribute "encrypted" is set to True, it should return a NotImplemented error.
  • a unit test demonstrating the correct operation on the Base Tensor type implemented over int and float Tensors.
  • inline documentation in the python code. For inspiration on inline documentation, please check out PyTorch's documentation for this operator.

Implement Default inverse Functionality for Base Tensor Type

User Story A: As a Data Scientist using Syft's Base Tensor type, we want to implement a default method for computing operations on a Tensor of arbitrary type. For this ticket to be complete, inverse should return a new tensor. For a reference on the operation this performs check out PyTorch's documentation.

Acceptance Criteria:

  • If the Base Tensor type's attribute "encrypted" is set to True, it should return a NotImplemented error.
  • a unit test demonstrating the correct operation on the Base Tensor type implemented over int and float Tensors.
  • inline documentation in the python code. For inspiration on inline documentation, please check out PyTorch's documentation for this operator.

Implement Default diag Functionality for Base Tensor Type

User Story A: As a Data Scientist using Syft's Base Tensor type, we want to implement a default method for computing operations on a Tensor of arbitrary type. For this ticket to be complete, diag() should return a new tensor. For a reference on the operation this performs check out PyTorch's documentation.

Acceptance Criteria:

  • If the Base Tensor type's attribute "encrypted" is set to True, it should return a NotImplemented error.
  • a unit test demonstrating the correct operation on the Base Tensor type implemented over int and float Tensors.
  • inline documentation in the python code. For inspiration on inline documentation, please check out PyTorch's documentation for this operator.

Implement Default baddbmm Functionality for Base Tensor Type

User Story A: As a Data Scientist using Syft's Base Tensor type, we want to implement a default method for computing operations on a Tensor of arbitrary type. For this ticket to be complete, baddbmm() should return a new tensor and baddbmm_() should perform the operation inline. For a reference on the operation this performs check out PyTorch's documentation.

Acceptance Criteria:

  • If the Base Tensor type's attribute "encrypted" is set to True, it should return a NotImplemented error.
  • a unit test demonstrating the correct operation on the Base Tensor type implemented over int and float Tensors.
  • inline documentation in the python code. For inspiration on inline documentation, please check out PyTorch's documentation for this operator.

Add Generic Tensor as Basic Type

Description: In the Proof-of-Concept Syft implementation, the basic type was a Float object contained within the Paillier.py class. This class contains overridden functions for various mathematical operators, so that the class can then be used inside of numpy arrays, as in our basic linear network linear.py. However, this approach, while simple, has several issues:

  • Homomorphic Encryption uses a technique called "packing" which can make efficient use when entire vectors of information are encrypted all at once. This was impossible when forcing all encrypted values to be simple scalars, resulting in a negative impact on performance.
  • Any function was callable upon the numpy arrays holding encrypted values, even those which were almost guaranteed to cause overflow within the encrypted cyphertexts, corrupting their values. While this was convenient, we need to implement the basic Tensor type ourselves so that we can have better warnings/automation for when bootstrapping should occur.
  • There was not a clear path to fast GPU support, which also requires vectorized operations.

In this work, we want to remedy these issues by building a generic tensor type, allowing us to handle encrypted values in the optimal way under the hood without requiring advanced knowledge by the user. The API for this tensor type will take direct inspiration from the unencrypted tensor classes done in the PyTorch framework, as it is a long term desire to eventually integrate with PyTorch if it becomes feasible to do so.

Acceptance Criteria:

  1. The generic tensor type must support the following operations inline with PyTorch:
    a. abs(), abs_(), add(value), add_(value), addbmm(), addbmm_(), addcdiv(), addcdiv_(), addcmul(), addcmul_(), addmm(), admm_(), addmv(), addmv_(), addr(), addr_(), baddbmm(), baddbmm_(), bernoulli(), bernoulli_(), bmm(), cauchy_(), ceil(), ceil_(), char(), chunk(), clamp(), clamp_(), clone(), contiguous(), copy_(), cpu(), cuda() - returns "not yet supported", cross(), cumprod(), cumsum(), diag(), dim(), dist(), div(), div_(), dot(), double(), eq(), eq_(), equal(), exp(), exp_(), expand(), expand_as(), exponential(), fill_(), float(), floor(), floor_(), fmod(), fmod_(), frac(), frac_(), gather(), ge(), ge_(), geometric_(), gt(), gt_(), half(), histc(), index(), index_add_(), index_copy_(), index_fill_(), index_select(), int(), inverse(), is_contiguous(), is_cuda(), is_signed(), le(), le_(), lerp(), lerp_(), log(), log1p(), log1p_(), log_(), log_normal_(), long(), lt(), lt_(), masked_scatter_(), masked_fill_(), masked_select(), matmul(), max(), mean(), median(), min(), mm(), mode(), mul(), mul_(), multinomial(), mv(), narrow(), ndimension(), ne(), ne_(), neg(), neg_(), nelement(), new(), nonzero(), norm(), normal_(), numel(), numpy(), permute(), pow(), pow_(), prod(), random_(), reciprocal(), reciprocal_(), remainder(), remainder_(), renorm(), renodm_(), repeat(), resize_(), resize_as_(), round(), round_(), rsqrt(), rsqrt_(), scatter_(), select(), set_(), short(), sigmoid(), sigmoid_(), sign(), sign_(), size(), split(), sqrt(), sqrt_(), squeeze(), squeeze_(), stride(), sub(), sub_(), sum(), t(), t_(), tolist(), topk(), trace(), transpose(), transpose_(), type(), unfold(), uniform_(), unsqueeze(), unsqueeze_(), view(), view_as(), zero_()
    b. For all functions above that pull from a distribution, optionally use a lookup table (abstract this?)
    c. For all functions above that use a function, optionally use an interpolation (abstract this?)

  2. The generic tensor type must be able to support an arbitrary number of dimensions for all operations.

  3. The generic tensor type must overload +-*/ and apply the appropriate operator (based on the dimensions of the tensors involved.

  4. The generic tensor type must prefer to use vector storage (under the hood) when encrypted to maximize the usefulness of packing in homomorphic encryption

  5. Any operations that are unsupported in the encrypted domain should include error messages

  6. str and repr should follow the conventions set in PyTorch.

  7. Additionally, there should be "encrypt(publickey)" and "decrypt(prikey)" functions which accept any key of the abstract type public or private key.

  8. All functionality within this base class should be of abstract type regardless of encryption type (specialized functionality for any encryption scheme should live within that scheme).

  9. Before this ticket is closed, this functionality should be shown to work with at least one homomorphic encryption strategy

Ticket Creator: @iamtrask

Implement Default Absolute Value Functions in Base Tensor Type

User Story A: As a Data Scientist using Syft's Base Tensor type, we want to implement a default method for computing the elementwise absolute value of a Tensor of arbitrary type. abs() should return a new tensor and abs_ should perform the operation inline. For a great reference on how

Acceptance Criteria:

  • If the Base Tensor type's attribute "encrypted" is set to True, it should return a NotImplemented error.
  • a unit test demonstrating the correct operation of abs() and abs_() on the Base Tensor type implemented over int and float Tensors.
  • inline documentation in the python code. For inspiration on inline documentation, please check out PyTorch's documentation for this operator.

Implement Default addcmul Functionality to Base Tensor Type

User Story A: As a Data Scientist using Syft's Base Tensor type, we want to implement a default method for computing operations on a Tensor of arbitrary type. For this ticket to be complete, addcmul() should return a new tensor and addcmul_() should perform the operation inline. For a reference on the operation this performs check out PyTorch's documentation.

Acceptance Criteria:

  • If the Base Tensor type's attribute "encrypted" is set to True, it should return a NotImplemented error.
  • a unit test demonstrating the correct operation on the Base Tensor type implemented over int and float Tensors.
  • inline documentation in the python code. For inspiration on inline documentation, please check out PyTorch's documentation for this operator.

Implement Default exponential() Functionality for Base Tensor Type

User Story A: As a Data Scientist using Syft's Base Tensor type, we want to implement a default method for computing operations on a Tensor of arbitrary type. For this ticket to be complete, exponential(). For a reference on the operation this performs check out PyTorch's documentation.

Acceptance Criteria:

  • If the Base Tensor type's attribute "encrypted" is set to True, it should return a NotImplemented error.
  • a unit test demonstrating the correct operation on the Base Tensor type implemented over int and float Tensors.
  • inline documentation in the python code. For inspiration on inline documentation, please check out PyTorch's documentation for this operator.

Implement Default lerp Functionality for Base Tensor Type

User Story A: As a Data Scientist using Syft's Base Tensor type, we want to implement a default method for computing operations on a Tensor of arbitrary type. For this ticket to be complete, lerp() should return a new tensor and lerp_() should perform the operation inline. For a reference on the operation this performs check out PyTorch's documentation.

Acceptance Criteria:

  • If the Base Tensor type's attribute "encrypted" is set to True, it should return a NotImplemented error.
  • a unit test demonstrating the correct operation on the Base Tensor type implemented over int and float Tensors.
  • inline documentation in the python code. For inspiration on inline documentation, please check out PyTorch's documentation for this operator.

Implement Default dim Functionality for Base Tensor Type

User Story A: As a Data Scientist using Syft's Base Tensor type, we want to implement a default method for computing operations on a Tensor of arbitrary type. For this ticket to be complete, diag() should return a list of the dimensions of the tensor. For a reference on the operation this performs check out PyTorch's documentation.

Acceptance Criteria:

  • If the Base Tensor type's attribute "encrypted" is set to True, it should return a NotImplemented error.
  • a unit test demonstrating the correct operation on the Base Tensor type implemented over int and float Tensors.
  • inline documentation in the python code. For inspiration on inline documentation, please check out PyTorch's documentation for this operator.

Implement Default lt Functionality for Base Tensor Type

User Story A: As a Data Scientist using Syft's Base Tensor type, we want to implement a default method for computing operations on a Tensor of arbitrary type. For this ticket to be complete, lt() should return a new tensor and lt_() should perform the operation inline. For a reference on the operation this performs check out PyTorch's documentation.

Acceptance Criteria:

  • If the Base Tensor type's attribute "encrypted" is set to True, it should return a NotImplemented error.
  • a unit test demonstrating the correct operation on the Base Tensor type implemented over int and float Tensors.
  • inline documentation in the python code. For inspiration on inline documentation, please check out PyTorch's documentation for this operator.

Implement Default equal Functionality for Base Tensor Type

User Story A: As a Data Scientist using Syft's Base Tensor type, we want to implement a default method for computing operations on a Tensor of arbitrary type. For this ticket to be complete, equal() should return a new tensor. For a reference on the operation this performs check out PyTorch's documentation.

Acceptance Criteria:

  • If the Base Tensor type's attribute "encrypted" is set to True, it should return a NotImplemented error.
  • a unit test demonstrating the correct operation on the Base Tensor type implemented over int and float Tensors.
  • inline documentation in the python code. For inspiration on inline documentation, please check out PyTorch's documentation for this operator.

Implement Default clamp Functionality for Base Tensor Type

User Story A: As a Data Scientist using Syft's Base Tensor type, we want to implement a default method for computing operations on a Tensor of arbitrary type. For this ticket to be complete, clamp() should return a new tensor and clamp_() should perform the operation inline. For a reference on the operation this performs check out PyTorch's documentation.

Acceptance Criteria:

  • If the Base Tensor type's attribute "encrypted" is set to True, it should return a NotImplemented error.
  • a unit test demonstrating the correct operation on the Base Tensor type implemented over int and float Tensors.
  • inline documentation in the python code. For inspiration on inline documentation, please check out PyTorch's documentation for this operator.

Implement Default matmul Functionality for Base Tensor Type

User Story A: As a Data Scientist using Syft's Base Tensor type, we want to implement a default method for computing operations on a Tensor of arbitrary type. For this ticket to be complete, matmul() should return a new tensor. For a reference on the operation this performs check out PyTorch's documentation.

Acceptance Criteria:

  • If the Base Tensor type's attribute "encrypted" is set to True, it should return a NotImplemented error.
  • a unit test demonstrating the correct operation on the Base Tensor type implemented over int and float Tensors.
  • inline documentation in the python code. For inspiration on inline documentation, please check out PyTorch's documentation for this operator.

Key-Rotation Support for Additive HE

Background: It has been alleged that vanilla Learning with Errors can be extremely performant for additive homomorphic encryption. In this work, we seek to verify that this is indeed true and achieve similar performance gains.

Acceptance Criteria:

  • the LWE approach in the above paper is constructed and shown to work for homomorphic addition and scalar multiplication (with unit tests)
  • key switching interfaces are added
  • the homomorphic encryption algorithm is in a new repository called PyFastLWE

Implement Default max Functionality for Base Tensor Type

User Story A: As a Data Scientist using Syft's Base Tensor type, we want to implement a default method for computing operations on a Tensor of arbitrary type. For this ticket to be complete, max() should return functionality based on what is passed into the method. For a reference on the operation this performs check out PyTorch's documentation.

Acceptance Criteria:

  • If the Base Tensor type's attribute "encrypted" is set to True, it should return a NotImplemented error.
  • a unit test demonstrating the correct operation on the Base Tensor type implemented over int and float Tensors.
  • inline documentation in the python code. For inspiration on inline documentation, please check out PyTorch's documentation for this operator.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.