Hash transformer is too slow about greenmask HOT 8 CLOSED

We will review the implementation. But I suspect it is due to the hash function. I think we should provide hash function choice and hash function params. We will try to resolve it in the next release.

For now, I can suggest a temporal solution. You can use a simple shell script to implement any hashing function. For instance

#!/bin/bash

while read line
do
   printf "%s" "$line" | md5sum | awk '{print $1}'
done

And the config can be like:

    - schema: "humanresources"
      name: "employee"
      transformers:
        - name: "Cmd"
          params:
            driver:
              name: "text"
            expected_exit_code: -1
            skip_on_null_input: true
            executable: "/var/lib/playground/test.sh"
            columns:
              - name: "jobtitle"

The result

Read about Cmd transformer

from greenmask.

Another finding about the Hash transformer, it seems to generate duplicate values even for a small set of values.

I have a table users with 1000 records and the email column is unique (UNIQUE INDEX), when I use the Hash I get an error during pg_restore: ERROR: could not create unique index \"users_email_ux\""

from greenmask.

Never mind, it was my fault, the UNIQUE INDEX has a FILTER condition allowing duplicate emails for deleted users.
Sorry for the false alarm.

from greenmask.

I gave it a try, the new hash transformer is crazy fast now, thank you

from greenmask.

I see that the current hash implementation is using an encryption algorithm, maybe a hash algorithm would be faster.

I'm going to push back this requirement for now until there is a built-in solution for it, thanks

from greenmask.

Agreed. We will try to deliver it soon, but any contribution is appreciated. Thank you!

from greenmask.

Yeah, a collision was caused. I will rewrite the implementation with the possibility of choosing a hash function (md5, sha1, SHA224/256/384/512). The expected release date is 14 February. Thank you so much for reporting.

from greenmask.

FIxed in v0.1.5

from greenmask.