See: https://github.com/riboseinc/oss-guides/tree/master/for_new_projects/ruby

Roadmap #3

https://en.wikipedia.org/wiki/Signcryption

Currently, there's no way to specify the range of rows to run transcryption on.
Thus, in the case of failure in the middle of a transcryption, or some other unfortunate situations where there are rows under the old encryption config and some other rows under the new config, there's no tool provided to fix them.

I propose to extend Transcryptor::*::ReEncryptStatement#re_encrypt_column with a way to specify additional SELECT clauses, perhaps as AREL / String (maybe via Transcyptor::AbstractAdapter#select_rows(table_name, columns)?).

Roadmap #3

(foreseen to depend on #4)

http://blog.honeybadger.io/zero-downtime-migrations-of-large-databases-using-rails-postgres-and-redis/

Migrating large datasets takes time. Making key migrations not incur downtime would be crucial in time-critical production environments.

The above link outlines what's needed to facilitate this task.

From @dadooda

Explain strategy first, add concrete details as tool functionality gets implemented.

The “strategy” doesn’t change over time and is roughly like this:

1. In the project, create fields for new encrypted values using regular Rails migration(s).
2. Run stand-alone Transcryptor against the prepared database(s). This step should fail safe.
3. Once everything settles, create migration(s) to remove old encrypted fields/data.

Roadmap #3

Currently, transcryptor uses attr_encrypted & Encryptor, which in turn use OpenSSL for cryptographic operations.

We want support for non-OpenSSL operations, e.g.:

• PGP, or rnp (ruby-rnp)
• botan (ruby-botan)
• sm4
• NOTE: This could potentially be a separate project by itself.

Context

Inside a migration file, we specify the configs like so:

class RemoveSaltFromStuff < ActiveRecord::Migration

  def old_configs
    { # old configuration of attr_encrypted for column
      key:           ->(u) { ENV['key'] },
      mode:          :per_attribute_iv_and_salt,
      algorithm:     'aes-256-cbc',
      encode_salt:   false,
      insecure_mode: true,
    }
  end

  def new_configs
    { # new configuration of attr_encrypted for column
      key:       ->(u) { ENV['key'] },
      mode:      :per_attribute_iv,
      algorithm: 'aes-256-gcm',
    }
  end

  def up
    re_encrypt_column :keys, :stuff,
                      old_configs,
                      new_configs
  end

  def down
    re_encrypt_column :keys, :stuff,
                      new_configs,
                      old_configs
  end

end

What is expected

The u in the ->(u) { ENV['key'] } would be an object that has access to all the column values of the row that Transcryptor is iterating on.

What actually happens

u is instead instantiated with a PORO object but only contains the encrypted field and its metadata (@encrypted_stuff and @encrypted_attributes in this example).

Create an interface for a user to generate a migration(s), something like:

rails generate transcryptor migration name [here can be a list of fields]

Tables' names and fields might be provided as an argument.

Setup

We're running MySQL with the mysql2 adapter.
activerecord at version 4.1.16.

What was run

class RemoveSaltFromStuff < ActiveRecord::Migration

  def old_configs
    { # old configuration of attr_encrypted for column
      key: ->(u) { ENV['key'] },
      mode: :per_attribute_iv_and_salt,
      algorithm: 'aes-256-cbc',
      encode_salt: false,
      insecure_mode: true,
    }
  end

  def new_configs
    { # new configuration of attr_encrypted for column
      key: ->(u) { ENV['key'] },
      mode: :per_attribute_iv,
      algorithm: 'aes-256-gcm',
    }
  end

  def up
    re_encrypt_column :keys, :stuff,
      old_configs, new_configs
  end

  def down
    re_encrypt_column :keys, :stuff,
      new_configs,
      old_configs
  end

end

What is expected

When I run db:migrate, no errors would be returned and the migration is successfully completed.

What actually happens

The following trace would be produced and db:migrate is aborted.

wrong number of arguments (given 1, expected 3)/opt/ribose/bundle/gems/activerecord-4.1.16/lib/active_record/connection_adapters/mysql2_adapter.rb:253:in `exec_delete'
/opt/ribose/src/transcryptor/lib/transcryptor/active_record/adapter.rb:14:in `update_row'
/opt/ribose/src/transcryptor/lib/transcryptor/instance.rb:48:in `block in re_encrypt'
/opt/ribose/src/transcryptor/lib/transcryptor/instance.rb:45:in `each'
/opt/ribose/src/transcryptor/lib/transcryptor/instance.rb:45:in `re_encrypt'
/opt/ribose/src/transcryptor/lib/transcryptor/active_record/re_encrypt_statement.rb:5:in `re_encrypt_column'
...

It seems that the parameter parity of ConnectionAdapters::Mysql2Adapter#exec_delete and the adapter we have aren't matching?

Roadmap #3

• add first used datetime of the encryption key
• remind if used for over e.g. a year

As an example

class ReencryptUsersAndDocumentsWithNewKeys < ActiveRecord::Migration
  def change
    # for the default configuration of attr_encrypted
    re_encrypt_column(
      :users,
      :email,
      { key: -> { ENV['old_master_encryption_key'] } },
      { key: -> { ENV['new_master_encryption_key'] } }
    )

    # for the default configuration of attr_encrypted without salt to with salt
    add_column :users, :encrypted_email_salt, :string
    re_encrypt_column(
      :users,
      :email,
      { key: ENV['old_master_encryption_key'], mode: :per_attribute_iv  }, # :per_attribute_iv is optional because it is default
      { key: ENV['new_master_encryption_key'], mode: :per_attribute_iv_and_salt }
    )

    # Default options should be the same as in attr_encrypted gem
    # prefix:            'encrypted_',
    # suffix:            '',
    # if:                true,
    # unless:            false,
    # encode:            false,
    # encode_iv:         true,
    # encode_salt:       true,
    # default_encoding:  'm',
    # marshal:           false,
    # marshaler:         Marshal,
    # dump_method:       'dump',
    # load_method:       'load',
    # encryptor:         Encryptor,
    # encrypt_method:    'encrypt',
    # decrypt_method:    'decrypt',
    # mode:              :per_attribute_iv,
    # algorithm:         'aes-256-gcm',
    # allow_empty_value: false
  end
end

Goal

1. Seamless transition from one encryption scheme to another.
2. Preferrably not to have to change Model code.

Implies

1. No generated artifacts (e.g. .rb files) should break Rails.
2. DB migrations should not break running Rails server.
3. No need to modify Model code to add / rename / remove attr_encrypted
   columns.
4. No restart of Rails server should be necessary, except for the first time
   when set-up code is deployed.
5. Encryption configuration (encryption schema) changes are stored away from
   Model definitions.
   Perhaps in a config file, or even another DB table.

Potential tools

Instead of running rake tasks to run transcryptions, rails-data-migrations
seems like a good alternative. It stores its migration history in the table
data_migrations.

DB

Currently,

• DB schema: db/schema.rb
• DB migrations: db/migration/xxx_.rb
• model encryption configs: app/models/xxx.rb

Column

Some ideas on encryption schema:

• one config item per encrypted column

db/encryption_schema.yaml

---yaml
- class_name: User
  - column: ssn
    - config: { algo, salt, etc. }

migration:

• transcrypt old config to new config
• new config linked to schema?

db/transcryption/2017125_transcrypt_user_ssn.rb

class TranscryptUserSSN < Transcryption
  def old
    { algo, salt, etc. }
  end

  def new
    #
    # { algo, salt, etc. }
    #
    # or perhaps something like:
    #
    Transcryptor.schema(:user, :ssn)
  end
end

config:

• define columns to encrypt
• encryption configs are read from Transcryptor.schema(:user, :ssn)

app/models/user.rb

class User < AR
  attr_transcrypted :ssn
end

How to make this work? What else needs to be defined? What's the closest we can acheive?

cc: @DmitryDrobotov

See: https://github.com/riboseinc/oss-guides/tree/master/for_new_projects/ruby

Will be great to have special class/self-method which takes settings from somewhere (e.g. transcryptor.yml) and re-encrypts keys based on its data.

This basically means migrating a column from unencrypted to encrypted, and vice versa.

It would be invaluable for a project to be able to safely encrypt data from existing columns that are not yet encrypted, or to be able to e.g. declassify data. This feature would seem like a natural step.

In terms of zero-downtime, it means supporting things like:

class User < ActiveRecord::Base
  attr_transcryptor :ssn,
                    old: {
                      key: proc { |user|
                        ENV['OLD_USER_SSN_ENC_KEY'] || ENV['USER_SSN_ENC_KEY']
                      },
                      mode: :per_attribute_iv,
                      algorithm: 'aes-256-gcm'
                    },
                    new: {}

  # ...
end

^ Read from the old column, but write to both old and new columns. The new column (e.g. ssn) is written _un_encrypted.

class User < ActiveRecord::Base
  attr_transcryptor :ssn,
                    new: {
                      key: proc { |user|
                        ENV['OLD_USER_SSN_ENC_KEY'] || ENV['USER_SSN_ENC_KEY']
                      },
                      mode: :per_attribute_iv,
                      algorithm: 'aes-256-gcm'
                    },
                    old: {}

  # ...
end

^ As above, but the new column (e.g. encrypted_ssn) is written _en_crypted.

And from a Migration's perspective, it means something like:

        # Encrypt column
        re_encrypt_column(
          :my_table,
          :column_1,
          {},
          { key: '2asd2asd2asd2asd2asd2asd2asd2asd' }
        )

        # Decrypt column
        re_encrypt_column(
          :my_table,
          :column_1,
          { key: '2asd2asd2asd2asd2asd2asd2asd2asd' },
          {}
        )

From @dadooda

Create a robust yet customizable command-line tool for transcryptor

https://github.com/attr-encrypted/attr_encrypted works with ActiveRecord, DataMapper, or Sequel. So, It will be great to support all of them.

• #4 Support lazy re-encryption
  • add first used datetime of the encryption key
  • remind if used for over e.g. a year
• #5 Support zero-downtime migration
• #6 Support periodic re-encryption
  • may opt to ask user for new keys, or auto-generate
  • key expiry to be per-table or per-column
  • Options to not allow old keys
• #7 Key activemodel / interface
• #8 Support non-attr_encrypted keys (which uses OpenSSL)
  • e.g. PGP, or rnp(https://github.com/riboseinc/ruby-rnp)
  • e.g. botan2(https://github.com/riboseinc/ruby-botan)
  • e.g. sm4(https://en.wikipedia.org/wiki/SM4_Algorithm)
  • NOTE: This could potentially be a separate project by itself.
• #9 Support "signcryption"

From @dadooda

Data transcryption must always take place in a controlled transaction, since things half-done might ruin existing data. Combining it with Rails migration specifics might make things unnecessarily difficult. It’s comparatively easy to spook Rails with a carelessly written migration and then not to be able to roll the spoiled data back.

Roadmap #3

Roadmap #3

• key expiry policy to be stored in a config {object | file | DB table / record}
• may opt to ask user for new keys, or auto-generate
• option for key expiry to be per-table or per-column
• option to not allow old keys

There are still many pending tests. Complete them, and add more as appropriate to cover as much code base as possible.

There are many undocumented configurations, and would benefit from having documenation.
These include:

• encode64_* and decode64_* for both transcryptor.enc and .dec,
• allowed parameter types for iv and salt for transcryptor.enc,
  • true|false, String, ...?
• etc.

Steps

class RemoveSaltFromStuff < ActiveRecord::Migration

  def old_configs
    { # old configuration of attr_encrypted for column
      key: ->(u) { ENV['key'] },
      mode: :per_attribute_iv_and_salt,
      algorithm: 'aes-256-cbc',
      encode_salt: false,
      insecure_mode: true,
    }
  end

  def new_configs
    { # new configuration of attr_encrypted for column
      key: ->(u) { ENV['key'] },
      mode: :per_attribute_iv,
      algorithm: 'aes-256-gcm',
    }
  end

  def up
    re_encrypt_column :keys, :stuff,
      old_configs,
      new_configs
  end

  def down
    re_encrypt_column :keys, :stuff,
      new_configs,
      old_configs
  end

end

What is expected

When I run db:migrate, the keys.encrypted_stuff_salt fields should be set to NULL.

What actually happens

The SQL statement produced doesn't seem to contain things like SET encrypted_stuff_salt = NULL. ( #20 is blocking this SQL to be executed.)

The generated SQL would contain things like ... WHERE my_column = NULL ... where it should be ... WHERE my_column IS NULL ....