Git Product home page Git Product logo

kiprotect / kodex Goto Github PK

View Code? Open in Web Editor NEW
106.0 9.0 16.0 3.81 MB

A privacy and security engineering toolkit: Discover, understand, pseudonymize, anonymize, encrypt and securely share sensitive and personal data: Privacy and security as code.

Home Page: https://heykodex.com

License: GNU Affero General Public License v3.0

Python 0.07% Makefile 0.11% Go 62.98% Dockerfile 0.01% Shell 0.01% SCSS 12.13% JavaScript 0.04% Sass 6.68% CSS 17.98%
privacy security privacy-protection privacy-enhancing-technologies privacy-engineering security-engineering anonymization pseudonymization anonymize pseudonymize

kodex's Issues

Error running Kodex from release bin file

When trying to run Kodex from the release bin file I was unable to.

When running the first command

kodex blueprints download

I got the error:

ERRO[0000] no blueprint paths specified

So I tried specifying the path, according to the documentation and run

kodex blueprints download https://my.blueprints/repo.zip

and got the same error.

Then, I tried to access the URL in a browser and got an error since the domain could not be found.

So I decided to download the blueprints manually from your Blueprints repository as listed on the documentation.

I placed the blueprints and the yaml file inside a folder, under the kodex bin file. When running the command

kodex run pseudonymization/examples/data-types/pseudonymize

I got the error

ERRO[0000] stat pseudonymization/examples/data-types/pseudonymize.yml: no such file or directory 
ERRO[0000] blueprint 'pseudonymization/examples/data-types/pseudonymize.yml' with version '' not found

Then I placed the blueprints at the same directory level as the kodex bin executable file and while running the same command, I got the error

ERRO[0000] stat home/dinis/Desktop/kodex-tags/latest-bin/pseudonymization/examples/data-types/input.json: no such file or directory

The path of the blueprints, relative to the kodex bin file is

pseudonymization/examples/data-types/input.json

It appears kodex has resolved the path to the absolute path, which, in my system is

home/dinis/Desktop/kodex-tags/latest-bin/pseudonymization/examples/data-types/input.json

However, kodex didn't include the / at the beginning, therefore trying to access that absolute path as a relative path, which will obviously not work. I checked that by placing the missing / at the beginning, the problem is fixed and the file is accessible from other programs. I cannot understand why kodex would try to access an absolute path as a relative path, when I have never specified that path on any configuration.

How can I solve this problem to be able to run kodex?

Error compiling Kodex from source code

When running make command I get the following error:

package ./api/definitions
        imports github.com/kiprotect/kodex/definitions
        imports github.com/kiprotect/kodex/cmd
        imports github.com/kiprotect/kodex/cmd/api
        imports github.com/kiprotect/kodex/api/helpers/controller
        imports github.com/kiprotect/kodex/helpers
        imports github.com/kiprotect/kodex/metering
        imports github.com/go-redis/redis
        ../../go/src/github.com/go-redis/redis/cluster.go:17:2: use of internal package github.com/redis/go-redis/internal not allowed
package ./api/definitions
        imports github.com/kiprotect/kodex/definitions
        imports github.com/kiprotect/kodex/cmd
        imports github.com/kiprotect/kodex/cmd/api
        imports github.com/kiprotect/kodex/api/helpers/controller
        imports github.com/kiprotect/kodex/helpers
        imports github.com/kiprotect/kodex/metering
        imports github.com/go-redis/redis
        ../../go/src/github.com/go-redis/redis/cluster.go:18:2: use of internal package github.com/redis/go-redis/internal/hashtag not allowed
package ./api/definitions
        imports github.com/kiprotect/kodex/definitions
        imports github.com/kiprotect/kodex/cmd
        imports github.com/kiprotect/kodex/cmd/api
        imports github.com/kiprotect/kodex/api/helpers/controller
        imports github.com/kiprotect/kodex/helpers
        imports github.com/kiprotect/kodex/metering
        imports github.com/go-redis/redis
        ../../go/src/github.com/go-redis/redis/command.go:12:2: use of internal package github.com/redis/go-redis/internal/hscan not allowed
package ./api/definitions
        imports github.com/kiprotect/kodex/definitions
        imports github.com/kiprotect/kodex/cmd
        imports github.com/kiprotect/kodex/cmd/api
        imports github.com/kiprotect/kodex/api/helpers/controller
        imports github.com/kiprotect/kodex/helpers
        imports github.com/kiprotect/kodex/metering
        imports github.com/go-redis/redis
        ../../go/src/github.com/go-redis/redis/cluster.go:19:2: use of internal package github.com/redis/go-redis/internal/pool not allowed
package ./api/definitions
        imports github.com/kiprotect/kodex/definitions
        imports github.com/kiprotect/kodex/cmd
        imports github.com/kiprotect/kodex/cmd/api
        imports github.com/kiprotect/kodex/api/helpers/controller
        imports github.com/kiprotect/kodex/helpers
        imports github.com/kiprotect/kodex/metering
        imports github.com/go-redis/redis
        ../../go/src/github.com/go-redis/redis/cluster.go:20:2: use of internal package github.com/redis/go-redis/internal/proto not allowed
package ./api/definitions
        imports github.com/kiprotect/kodex/definitions
        imports github.com/kiprotect/kodex/cmd
        imports github.com/kiprotect/kodex/cmd/api
        imports github.com/kiprotect/kodex/api/helpers/controller
        imports github.com/kiprotect/kodex/helpers
        imports github.com/kiprotect/kodex/metering
        imports github.com/go-redis/redis
        ../../go/src/github.com/go-redis/redis/cluster.go:21:2: use of internal package github.com/redis/go-redis/internal/rand not allowed
package ./api/definitions
        imports github.com/kiprotect/kodex/definitions
        imports github.com/kiprotect/kodex/cmd
        imports github.com/kiprotect/kodex/cmd/api
        imports github.com/kiprotect/kodex/api/helpers/controller
        imports github.com/kiprotect/kodex/helpers
        imports github.com/kiprotect/kodex/metering
        imports github.com/go-redis/redis
        ../../go/src/github.com/go-redis/redis/command.go:14:2: use of internal package github.com/redis/go-redis/internal/util not allowed
make: *** [Makefile:20: dep] Error 1

When running make install, I get the same error.

How can I solve this problem to be able to compile kodex?

Staged Processing

Motivation

Many privacy-enhancing transformations require multiple stages. Generalizing attributes for example requires us to define a generalization hierarchy first. Then, in a second step we can apply this hierarchy to the data items. This requires us to process items in stages.

Examples:

  • Generalization hierarchy:

    • Stage 1:
      • Analyze value distribution in items.
    • Stage 2:
      • Generalize items with the given distribution.

  • k-Anonymity:

    • Stage 1:
      • Analyze attribute frequencies.

Implementation Proposal

To enable such staged processing, we plan to make the following additions to the Kodex stream processing mechanisms:

  • Add a numerical stage attribute to the Config model.
  • Add a Batch model that stores information about the processing of a given stage for a number of items.
  • Add an internal buffering mechanism (using internal channels) that enables us to buffer items for multi-stage processing.
  • Make the group store functionality currently implemented in the anonymization/aggregation action available to all actions as a means to perform distributed, parallel computation on data items.
  • Change the scheduler to enable staged processing of data items.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.