Git Product home page Git Product logo

kiprotect / kodex Goto Github PK

View Code? Open in Web Editor NEW
105.0 9.0 16.0 3.81 MB

A privacy and security engineering toolkit: Discover, understand, pseudonymize, anonymize, encrypt and securely share sensitive and personal data: Privacy and security as code.

Home Page: https://heykodex.com

License: GNU Affero General Public License v3.0

Python 0.07% Makefile 0.11% Go 62.98% Dockerfile 0.01% Shell 0.01% SCSS 12.13% JavaScript 0.04% Sass 6.68% CSS 17.98%
privacy security privacy-protection privacy-enhancing-technologies privacy-engineering security-engineering anonymization pseudonymization anonymize pseudonymize

kodex's Introduction

Kodex (Community Edition - CE) is an open-source toolkit for privacy and security engineering. It helps you to automate data security and data protection measures in your data engineering workflows. It offers the following functionality:

  • Read data items from a variety of sources such as files, databases or message queues.
  • Protect these data items using various privacy- & security enhancing transformations, like de-identification, masking, pseudonymization, anonymization or encryption.
  • Send the protected items to a variety of destinations.

With Kodex, you can describe your data protection and data security workflows using a simple, declarative configuration language: Just like DevOps tools let you describe infrastructure as code, Kodex is a PrivacyOps & SecurityOps tool that let you describe privacy and security measures as code.

Kodex takes care of the boring and difficult aspects of privacy, such as

  • Key management: Kodex manages encryption and pseudonymization keys for you (if you want that).
  • Parameter management: Kodex keeps track of how every single data item was processed so you can prove the compliance of your data workflows and create an audit trail.
  • Data transformation: Kodex implements modern cryptographic and statistical techniques to protect your data.

Getting started

To download and install Kodex from source, simply run

git clone https://github.com/kiprotect/kodex
cd kiprotect

make
make install

Documentation

You can find the official documentation at https://heykodex.com/docs.

Transforming data

Kodex reads its configuration from so-called blueprints. To get an idea of how this works, check out our blueprints repository, which contains example blueprints together with instructions on how to run them. You can install these blueprints via Kodex (requires Internet access):

kodex blueprints download

Alternatively, you can copy them to your machine manually, please refer to the documentation for more details. To then run the pseudonymization example, simply type

# pseudonymize the example data and write it to a file named 'pseudonymized.json'
kodex run pseudonymization/examples/data-types/pseudonymize

# depseudonymize the data again and print the result on stdout
kodex run pseudonymization/examples/data-types/depseudonymize

That's it! Kodex takes care of generating and storing cryptographic parameters for the pseudonymization. If you want to manually enter a key instead to generate parameters, you can do that too:

# pseudonymize the data with a user-supplied key
kodex run pseudonymization/examples/data-types/pseudonymize-with-key

# depseudonymize with a key as well
kodex run pseudonymization/examples/data-types/depseudonymize-with-key

Running the tests

Kodex comes with a suite of automated unit tests, which you can run with Make:

make test

Testing Plugins

The plugin test may fail with an error message "plugin was built with a different version of package internal/cpu" which can occur if the compile flags used for compiling the main code and the plugin differ. This might happen e.g. if you perform race condition detection tests. To fix the problem, go to the plugin folder and run

make clean
make

Running the benchmarks

Kodex also comes with a number of benchmarks that you can run as follows:

make bench

Status & Roadmap

This is still an early version of Kodex and does not contain many features yet. We will progressively port more functionality from our Enterprise Edition (EE). The following features are next up on our list:

  • Anonymization: Anonymize streaming data using differentially private aggregations.
  • Discovery: Discover sensitive and personal information in your structured and unstructured data.
  • Encryption: Encrypt and decrypt structured data.
  • Data Mapping: Analyze and map your data infrastructure.
  • Consent Management: Manage and enforce processing purposes and user consent for all your data streams.

Enterprise Edition

Our open-source work is made possible by commercially offering a Kodex enterprise edition (EE), which extends the community edition (CE) with functionality that supports a deployment of Kodex in a professional enterprise environment. It includes e.g. the following functionality:

  • Advanced, SQL-based configuration & parameter management and storage.
  • REST-based API to control all Kodex functionality.
  • Web interface to manage and monitor data streams.
  • More advanced data transformations.
  • Role-based access control mechanism.

Are you interested to learn more about Kodex EE? Just visit our website or get in touch with us!

License

Kodex is currently licensed under the Affero GPL license, version 3 (AGPL-3.0). See the license file for details. In addition, we also offer a commercial license that allow you to directly integrate the Kodex code into closed-source software without disclosing your own code. If you're interested in buying a commercial license, please get in touch with us.

Why Affero GPL?

The Affero GPL license is a strong copyleft license that allows you to freely use Kodex for commercial and non-commercial purposes. If you use the software as a standalone tool without integrating it with your own software code (i.e. you do not import and compile it as a Go library in your own Go code) its use will not affect your own software code in any way. In that respect, Kodex can be used as freely as other Linux tools provided under a GPL license.

However, if you integrate the Kodex code with your own software code and distribute or offer that software as a web service, you will have to make the source code of your software available under a compatible license. Similarly, if you modify or extend Kodex and either distribute it or offer it as a service you will have to make the source code of your changes available as well. This ensures that improvements which you make to Kodex will benefit the entire user community.

I need a different license

If you have trouble using Kodex-CE due to the license terms, please get in touch with us. We offer a commercial license that enables you to integrate Kodex with your own software code without being affected by the terms of the AGPL license.

Contact us

Do you have trouble getting Kodex to run? Do you want to suggest a new feature or report a bug? Please open an issue in this issue tracker. If it's something that you'd like to discuss directly with us, please send us an e-mail, we love to hear from you!

Spread the word

Are you using Kodex in your organization and like it? Please let the world know! Spreading the word about it and giving us feedback helps us to improve the software.

kodex's People

Contributors

thmull avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

kodex's Issues

Error compiling Kodex from source code

When running make command I get the following error:

package ./api/definitions
        imports github.com/kiprotect/kodex/definitions
        imports github.com/kiprotect/kodex/cmd
        imports github.com/kiprotect/kodex/cmd/api
        imports github.com/kiprotect/kodex/api/helpers/controller
        imports github.com/kiprotect/kodex/helpers
        imports github.com/kiprotect/kodex/metering
        imports github.com/go-redis/redis
        ../../go/src/github.com/go-redis/redis/cluster.go:17:2: use of internal package github.com/redis/go-redis/internal not allowed
package ./api/definitions
        imports github.com/kiprotect/kodex/definitions
        imports github.com/kiprotect/kodex/cmd
        imports github.com/kiprotect/kodex/cmd/api
        imports github.com/kiprotect/kodex/api/helpers/controller
        imports github.com/kiprotect/kodex/helpers
        imports github.com/kiprotect/kodex/metering
        imports github.com/go-redis/redis
        ../../go/src/github.com/go-redis/redis/cluster.go:18:2: use of internal package github.com/redis/go-redis/internal/hashtag not allowed
package ./api/definitions
        imports github.com/kiprotect/kodex/definitions
        imports github.com/kiprotect/kodex/cmd
        imports github.com/kiprotect/kodex/cmd/api
        imports github.com/kiprotect/kodex/api/helpers/controller
        imports github.com/kiprotect/kodex/helpers
        imports github.com/kiprotect/kodex/metering
        imports github.com/go-redis/redis
        ../../go/src/github.com/go-redis/redis/command.go:12:2: use of internal package github.com/redis/go-redis/internal/hscan not allowed
package ./api/definitions
        imports github.com/kiprotect/kodex/definitions
        imports github.com/kiprotect/kodex/cmd
        imports github.com/kiprotect/kodex/cmd/api
        imports github.com/kiprotect/kodex/api/helpers/controller
        imports github.com/kiprotect/kodex/helpers
        imports github.com/kiprotect/kodex/metering
        imports github.com/go-redis/redis
        ../../go/src/github.com/go-redis/redis/cluster.go:19:2: use of internal package github.com/redis/go-redis/internal/pool not allowed
package ./api/definitions
        imports github.com/kiprotect/kodex/definitions
        imports github.com/kiprotect/kodex/cmd
        imports github.com/kiprotect/kodex/cmd/api
        imports github.com/kiprotect/kodex/api/helpers/controller
        imports github.com/kiprotect/kodex/helpers
        imports github.com/kiprotect/kodex/metering
        imports github.com/go-redis/redis
        ../../go/src/github.com/go-redis/redis/cluster.go:20:2: use of internal package github.com/redis/go-redis/internal/proto not allowed
package ./api/definitions
        imports github.com/kiprotect/kodex/definitions
        imports github.com/kiprotect/kodex/cmd
        imports github.com/kiprotect/kodex/cmd/api
        imports github.com/kiprotect/kodex/api/helpers/controller
        imports github.com/kiprotect/kodex/helpers
        imports github.com/kiprotect/kodex/metering
        imports github.com/go-redis/redis
        ../../go/src/github.com/go-redis/redis/cluster.go:21:2: use of internal package github.com/redis/go-redis/internal/rand not allowed
package ./api/definitions
        imports github.com/kiprotect/kodex/definitions
        imports github.com/kiprotect/kodex/cmd
        imports github.com/kiprotect/kodex/cmd/api
        imports github.com/kiprotect/kodex/api/helpers/controller
        imports github.com/kiprotect/kodex/helpers
        imports github.com/kiprotect/kodex/metering
        imports github.com/go-redis/redis
        ../../go/src/github.com/go-redis/redis/command.go:14:2: use of internal package github.com/redis/go-redis/internal/util not allowed
make: *** [Makefile:20: dep] Error 1

When running make install, I get the same error.

How can I solve this problem to be able to compile kodex?

Error running Kodex from release bin file

When trying to run Kodex from the release bin file I was unable to.

When running the first command

kodex blueprints download

I got the error:

ERRO[0000] no blueprint paths specified 

So I tried specifying the path, according to the documentation and run

kodex blueprints download https://my.blueprints/repo.zip

and got the same error.

Then, I tried to access the URL in a browser and got an error since the domain could not be found.

So I decided to download the blueprints manually from your Blueprints repository as listed on the documentation.

I placed the blueprints and the yaml file inside a folder, under the kodex bin file. When running the command

kodex run pseudonymization/examples/data-types/pseudonymize

I got the error

ERRO[0000] stat pseudonymization/examples/data-types/pseudonymize.yml: no such file or directory 
ERRO[0000] blueprint 'pseudonymization/examples/data-types/pseudonymize.yml' with version '' not found 

Then I placed the blueprints at the same directory level as the kodex bin executable file and while running the same command, I got the error

ERRO[0000] stat home/dinis/Desktop/kodex-tags/latest-bin/pseudonymization/examples/data-types/input.json: no such file or directory 

The path of the blueprints, relative to the kodex bin file is

pseudonymization/examples/data-types/input.json

It appears kodex has resolved the path to the absolute path, which, in my system is

home/dinis/Desktop/kodex-tags/latest-bin/pseudonymization/examples/data-types/input.json

However, kodex didn't include the / at the beginning, therefore trying to access that absolute path as a relative path, which will obviously not work. I checked that by placing the missing / at the beginning, the problem is fixed and the file is accessible from other programs. I cannot understand why kodex would try to access an absolute path as a relative path, when I have never specified that path on any configuration.

How can I solve this problem to be able to run kodex?

Staged Processing

Motivation

Many privacy-enhancing transformations require multiple stages. Generalizing attributes for example requires us to define a generalization hierarchy first. Then, in a second step we can apply this hierarchy to the data items. This requires us to process items in stages.

Examples:

  • Generalization hierarchy:

    • Stage 1:
      • Analyze value distribution in items.
    • Stage 2:
      • Generalize items with the given distribution.
  • k-Anonymity:

    • Stage 1:
      • Analyze attribute frequencies.

Implementation Proposal

To enable such staged processing, we plan to make the following additions to the Kodex stream processing mechanisms:

  • Add a numerical stage attribute to the Config model.
  • Add a Batch model that stores information about the processing of a given stage for a number of items.
  • Add an internal buffering mechanism (using internal channels) that enables us to buffer items for multi-stage processing.
  • Make the group store functionality currently implemented in the anonymization/aggregation action available to all actions as a means to perform distributed, parallel computation on data items.
  • Change the scheduler to enable staged processing of data items.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.