Changing Your Ecto Encryption Key

Author’s Note: I’ve released an open-source Hex package that implements the approach to encryption I describe in this post. Read the announcement post here.

READ THIS FIRST: Encrypting Data with Ecto

In an earlier post, I wrote about how to encrypt data with Ecto, Elixir’s database library. However, I didn’t cover how to change your encryption key, which you’ll definitely want to do periodically. I want to show how do that in this post.

Tag Your Ciphertext

In order to migrate to a new key, you’re going to need to be able to distinguish which key was used to encrypt any given text. This can be done very simply by prepending a single byte to every encrypted binary:

key_id = <<1>>
key_id <> <<5, 138, ...>

This byte is the ID of the key that was used, and the decryptor can easily use it to find the correct key.

If you’re not looking to migrate to a new key right now, this is really all you need to do. If all your ciphertext includes a key ID, you’ll be able to distinguish between ciphertexts, and you’ll be able to follow the rest of these steps later when you want to migrate to a new key.

Change Config

We need to change our config.exs to support multiple keys:

config :my_app, MyApp.AES,
  keys: %{
    <<1>> => :base64.decode("..."), # assuming you store keys in base64
    <<2>> => :base64.decode("...")  # otherwise, straight binary will do
  },
  default: <<1>> # The ID of the key we want to use

The first key, <<1>>, is the legacy key we’re migrating away from. <<2>> is the key we are migrating to. We’ll leave the :default as key <<1>> until we are ready to make the switch.

Change Encryptor

Now, we need to upgrade the encrypt/1 function to prepend the key ID of the key that was used for the encryption:

# Store config in module attributes for slightly faster lookup at runtime,
# and the ability to use them as function parameter defaults
@config  Application.get_env(:my_app, MyApp.AES)
@keys    @config[:keys]
@default @config[:default]

# Add another parameter, to allow encrypting with a specific key
def encrypt(ciphertext, key_id \\ @default) do
  iv    = :crypto.strong_rand_bytes(16)
  state = :crypto.stream_init(:aes_ctr, @keys[key_id], iv) # use specified key
  {_state, ciphertext} = :crypto.stream_encrypt(state, to_string(plaintext))
  key_id <> iv <> ciphertext # prepend key byte to iv and ciphertext
end

The key ID will now be included in every ciphertext. We now just have to update the decrypt/1 function to find this key ID properly:

def decrypt(ciphertext) do
  # The first byte is the key ID
  <<key_id::binary-1, iv::binary-16, ciphertext::binary>> = ciphertext
  state = :crypto.stream_init(:aes_ctr, @keys[key_id], iv) # use specified key
  {_state, plaintext} = :crypto.stream_decrypt(state, ciphertext)
  plaintext
end

With that, our encryptor can encrypt and decrypt values generated by any of our keys. By extension, this means that Ecto can read encrypted data from the database, no matter which key was used to generate it.

It’s also a good idea to add a simple function to the encryptor so that other modules can access the current default key ID without having to know where it is stored in config.exs

def key_id do
  @default
end

Change Model(s)

There isn’t actually a lot that we have to change on our model to get things basically working. If we switch to a new key at this point, nothing will break, and records will be gradually upgraded to the new key. This is how that works:

• Suppose that a given User record was encrypted with key <<1>>.
• We then switch the config so that the :default key is now <<2>>.
• When our user record is loaded, the decryptor will detect that it is encrypted with key <<1>> and successfully decrypt it using that key.
• When it is saved, it will be encrypted with key <<2>>.

This means that the software can continue to function without any downtime after we switch the keys. This is a big win, especially if you’re dealing with a lot of encrypted data that takes a long time to migrate.

Gradual Upgrading Not Enough

In the likely event that you want to re-encrypt your data with the new key more quickly, you’ll need to make some changes to your User model. First off, you want to track which rows have been migrated and which have not. Add an indexed binary field called :encryption_key_id to the model.

schema "users" do
  field :name, MyApp.EncryptedField
  field :email, MyApp.EncryptedField
  field :email_hash, MyApp.HashField
  field :encryption_key_id, :binary
end

Then, set this field with a callback. I’ve renamed the set_hashed_fields callback to set_defaults for this example:

before_insert :set_defaults
before_update :set_defaults

defp set_defaults(changeset) do
  changeset
  |> put_change(:encryption_key_id, MyApp.AES.key_id)
  |> put_change(:email_hash, get_field(changeset, :email))
end

The :encryption_key_id field will now contain the key ID which was used to encrypt the row. This will help us find rows which haven’t been upgraded yet.

Mix Task

We’ve come to the final step, creating a mix task to proactively migrate up all of our data to the new key. Here’s what it could look like:

# lib/mix/tasks/encryption.migrate.ex
defmodule Mix.Tasks.Encryption.Migrate do
  use Mix.Task

  import Ecto.Query
  import Logger, only: [info: 1]

  alias MyApp.Repo

  # Store the current key ID in a module attribute. This is the key that we are
  # going to migrate to, controlled by `:default` in `config.exs`.
  #
  # I'm doing it this way rather than calling the function directly so we can
  # use it in pattern matching below.
  @key_id MyApp.AES.key_id

  def run(args) do
    # Ensure that MyApp.Repo is started and available for use
    Mix.Task.run "app.start", args

    # Migrate our User model
    # You could migrate any other models that use encryption here.
    migrate MyApp.User
  end

  defp migrate(model) do
    info "=== Migrating #{model} Model ==="
    ids = ids_for(model)
    info "#{length(ids)} records found needing migration"

    # Migrate all the records found
    for id <- ids do
      # Records are fetched individually to ensure that they haven't changed
      # between being loaded and saved. We don't want to overwrite any changes
      # that occur while this task is running.
      Repo.get(model, id) |> migrate_record
    end

    info "=== Migration Complete ==="
  end

  # Returns all the IDs of records which, according to :encryption_key_id, have
  # not yet migrated to @key_id.
  defp ids_for(model) do
    query = from m in model, where:  m.encryption_key_id != ^@key_id,
                             select: m.id
    Repo.all(query)
  end

  # Do nothing if the record has been automagically migrated by app usage since
  # we queried for IDs at the start of this task.
  defp migrate_record(%{encryption_key_id: @key_id}), do: nil

  # If the record doesn't match the above definition, and therefore has not been
  # migrated, then simply save it back to the database. This will trigger
  # MyApp.AES.encrypt, which will save the data encrypted with the new key.
  defp migrate_record(record), do: Repo.update!(record)
end

The Great Migration

We’re ready to migrate! We can just change the :default in config.exs from <<1>> to <<2>>, and then run our mix task:

$ mix encryption.migrate
[info] === Migrating Elixir.MyApp.User Model ===
[info] 10 records found needing migration
[info] === Migration Complete ===

Conclusion

This approach has many things to recommend it:

• It’s simple, and therefore maintainable.
• There’s zero downtime while switching encryption keys.
• A minimum of extra fields are needed. (1 per table)
• You can verify that all the data has been migrated with a simple query on :encryption_key_id.
• You can switch to a new key in three steps:
  • Add a new key to the :keys map.
  • Change the :default to the new key ID.
  • Run mix encryption.migrate.

If you want to see all the code, I’ve posted it on Github. Hope this helps someone out there!