HCP Vault Performance Replication

10min
HCP
Vault

NOTE: Performance Replication is a HCP Vault Plus feature.

What is HCP Vault Performance Replication

Vault provides the critical services of identity management, secrets storage, and policy management. This functionality is expected to be highly available and to scale as the number of clients and their functional needs increase; at the same time, operators would like to ensure that a common set of policies are enforced globally, and a consistent set of secrets and keys are exposed to applications that need to interoperate. Vault Enterprise Performance Replication provides scalability across regions.

Replication operates on a leader/follower model, wherein a leader cluster (known as a primary) is linked to its follower secondary clusters. The primary cluster acts as the system of record and asynchronously replicates most Vault data. The secondary cluster mirrors the configuration of its primary cluster; however, it keeps track of its own tokens and leases.

If a user action modifies the underlying shared state, the secondary cluster forwards the request to the primary cluster to handle; this is transparent to the client. In practice, most high-volume workloads (reads in the kv backend, encryption/decryption operations in transit, etc.) can be satisfied by the local secondary, allowing Vault to scale relatively horizontally.

Scenario introduction

In this tutorial, you are going to create an HCP Vault primary cluster in the US East region, and the secondary cluster in the US West region.

Prerequisites

An HCP account with appropriate permissions to initiate this change.
If you are new to HCP Vault, go through the Getting Started with HCP Vault tutorials.
Install Vault on your local machine.

Create HashiCorp Virtual Networks

Create HashiCorp Virtual Networks (HVNs) in the US East region and the US West region before creating HCP Vault clusters.

Launch the HCP Portal and login.
Click HashiCorp Virtual Network, and click Create network.
At the Create a HashiCorp Virtual Network page, enter hvn-us-east-1 in the Network name field.
Select N. Virginia (us-east-1) from the Region selection drop-down list.
Accept or modify the default CIDR block.
NOTE: The HashiCorp Virtual Network CIDR block should not overlap with your existing private network address space.
Click Create network. This takes a few minutes.
From the HashiCorp Virtual Network page, click Create network again to create another HVN.
Enter hvn-us-west-2 in the Network name field, and select Oregon (us-west-2) from the Region selection drop-down list.
Enter 172.24.16.0/20 in the CIDR block field.
NOTE: The IP addresses should not overlap between hvn-us-east-1 and hvn-us-west-2. Since the CIDR for hvn-us-east-1 was 172.25.16.0/20, set the CIDR for hvn-us-west-2 to 172.24.16.0/20.
Click Create network. Wait until the HVN creation completes.
Click Back to Networks in the left navigation menu to view the HVNs.

Create a primary cluster

NOTE: This step creates a new HCP Vault custer. You can enable performance replication on an already existing cluster instead. However, the cluster must be a Plus tier cluster. If not, change the cluster to Plus first.

Click Vault in the left navigation menu, and click the Create cluster button.
Enter vault-cluster-primary in the Cluster ID field.
Under the Vault tier section, click the radio button for Plus.
Select the hvn-us-east-1 under the Network section.
Shift the toggle button for the Allow public connections from outside your selected network option.
NOTE: All new HCP Vault clusters are configured with public access disabled by default. Making your HCP Vault cluster publicly accessible is not recommended, however for ease of completing this tutorial we have you enable it. You can learn how to connect to a private HCP Vault cluster in the Connect an Amazon Transit Gateway to your HashiCorp Virtual Network or Peering an AWS VPC with HashiCorp Cloud Platform (HCP) tutorials.
Under the Cluster size section, click the Small radio button.
Click the Create cluster button. Wait for the cluster to initialize before proceeding.

Create sample data

To demonstrate the replication capabilities you will create two namespaces and enable the key/value v2 secrets engine.

Under Cluster URLs, click Public Cluster URL.
In a terminal, set the VAULT_ADDR environment variable to the copied address.
```
$ export VAULT_ADDR=<Public_Cluster_URL>
```
Return to the Overview page and click Generate token.
Within a few moments, a new token will be generated.
Copy the Admin Token.
Return to the terminal and set the VAULT_TOKEN environment variable.
```
$ export VAULT_TOKEN=<token>
```
Set the VAULT_NAMESPACE environment variable to admin.
```
$ export VAULT_NAMESPACE=admin
```

Create a namespace named replicate-namespace.

$ vault namespace create replicate-namespace

Key     Value
---     -----
id      LmNop
path    admin/replicate-namespace/

Enable the K/V v2 secrets engine in the replicate-namespace namespace with the path replicate-secrets.

$ vault secrets enable -namespace=admin/replicate-namespace -path=replicate-secrets kv-v2

Success! Enabled the kv-v2 secrets engine at: replicate-secrets/

Enable the K/V v2 secrets engine in the replicate-namespace namespace with the path do-not-replicate-secrets.

$ vault secrets enable -namespace=admin/replicate-namespace -path=do-not-replicate-secrets kv-v2

Success! Enabled the kv-v2 secrets engine at: do-not-replicate-secrets/

List the enabled secrets engines in the replicate-namespace namespace.

$ vault secrets list -namespace=admin/replicate-namespace

Path                 Type            Accessor                 Description
----                 ----            --------                 -----------
cubbyhole/           ns_cubbyhole    ns_cubbyhole_8bd101b3    per-token private secret storage
do-not-replicate/    kv              kv_3b7c4595              n/a
identity/            ns_identity     ns_identity_03413f20     identity store
replicate/           kv              kv_35088ce2              n/a
sys/                 ns_system       ns_system_d52af70f       system endpoints used for control, policy and debugging

Create a second namespace named do-not-replicate-namespace.

$ vault namespace create do-not-replicate-namespace

Key     Value
---     -----
id      HiJkl
path    admin/do-not-replicate-namespace/

Under Quick actions click Generate token.
Within a few moments, a new token will be generated.
Copy the Admin Token.
Under Access web UI, click the Public Cluster link.
A new browser window will open.
The login page is displayed. By default Vault enables the token authentication method.
Enter the copied token in the Token field.
Click Sign In.
Click Access in top navigation menu and then click Namespaces.
Click the Create Namespace + button.
In the Path field enter replicate-namespace and click the Save button.
Click the Create Namespace + button.
In the Path field enter do-not-replicate-namespace and click the Save button.
Click admin in the top navigation menu and click replicate-namespace.
Click Secrets in top navigation menu.
Click the Enable new engine + button.
Click the KV radio button and click the Next button.
In the Path field enter replicate-secrets and click the Enable Engine button.
Click Secrets in the top navigation menu.
Click the Enable new engine + button.
Click the KV radio button and click the Next button.
In the Path field enter do-not-replicate-secrets and click the Enable Engine button.

Create a performance secondary cluster

In the Vault overview page, click Replication in the left navigation menu.
Click Set up replication.
In the Create a Vault replication secondary page, enter vault-cluster-secondary in the Cluster ID field.
Select hvn-us-west-2 under HashiCorp Virtual Network.
Observe the Replication paths filter switch.
By default, all namespaces and mount paths will be replicated to the secondary cluster.
Click the Replicate all namespace and mount paths switch. The Deny access to the following namespaces and mount paths text box will appear.
In the text box enter do-not-replicate-namespace and click the Add button.
This will prevent the do-not-replicate-namespace namespace you created earlier from being replicated to the secondary cluster, including any mount paths in the namespace.
A second text box will appear. Enter replicate-namespace/do-not-replicate-secrets and click the Add button.
This will prevent the K/V v2 secret engine you enabled earlier from being replicated to the secondary cluster, even though its parent namespace is being replicated.
Click Create secondary. This takes a few minutes.

Validate cluster replication

Once the vault-cluster-secondary cluster deployment completes, verify the sample data you created was replicated as expected.

When the cluster status changes to Running, click the vault-cluster-secondary link.
In the Configuration pane, click the Public Cluster URL.
In a terminal, update the VAULT_ADDR environment variable to the copied address.
```
$ export VAULT_ADDR=<Public_Cluster_URL>
```
Return to the Overview page and click Generate token.
Within a few moments, a new token will be generated.
Copy the Admin Token.
In a terminal, update the VAULT_TOKEN environment variable to store the token value.
```
$ export VAULT_TOKEN=<Admin_Token>
```
View the list of namespaces available in the secondary cluster.
```
$ vault namespace list

Keys
----
replicate-namespace/
```
The do-not-replicate-namespace namespace was not replicated to the secondary cluster because you added the namespace path to the deny filter.

View the list of secrets engines available in the replicate-namespace.

$ vault secrets list -namespace=admin/replicate-namespace

Path                 Type            Accessor                 Description
----                 ----            --------                 -----------
cubbyhole/           ns_cubbyhole    ns_cubbyhole_8bd101b3    per-token private secret storage
identity/            ns_identity     ns_identity_03413f20     identity store
replicate-secrets    kv              kv_35088ce2              n/a
sys/                 ns_system       ns_system_d52af70f       system endpoints used for control, policy and debugging

The do-not-replicate-secrets K/V secret engine was not replicated, even though the namespace was replicated because you added the secret engine path to the deny filter.

When the cluster status changes to Running, click the vault-cluster-secondary link.
Under Quick actions and click Generate token.
Within a few moments, a new token will be generated.
Copy the Admin Token.
Under Access web UI, click the Public Cluster URL.
When the Vault UI launches in a new tab/window, enter the admin token in the Token field.
Click Sign In.
Click Access in top navigation menu and then click Namespaces.
The do-not-replicate-namespace namespace was not replicated to the secondary cluster because you added the namespace path to the deny filter.
Click admin in the top navigation menu and click replicate-namespace.
Click Secrets in top navigation menu.
The do-not-replicate-secrets K/V secret engine was not replicated, even though the namespace was replicated because you added the secret engine path to the deny filter.

Local secrets engines and auth methods

You can also enable secrets engines and/or auth methods locally if you want to disallow them from being replicated across the clusters.

Local secrets engine

Enable an auth method via CLI

To enable a local secrets engine or auth method via Vault CLI, use the -local flag.

Example: The following command enables the AppRole auth method locally at us_east_approle path. This auth method configuration will not be replicated to other clusters within the replication group.

$ vault auth enable -local -path=us_east_approle approle

Delete clusters

A primary cluster can not be deleted while it has an active secondary cluster. The secondary cluster needs to be deleted before deleting the primary cluster.

From the Vault page, click vault-cluster-secondary.
In the Overview page, click Manage > Delete cluster.
When prompted to confirm, enter DELETE in the text field and click Delete. Wait until the cluster deletion completes.
Repeat the steps to delete the primary cluster (vault-cluster-primary).

After clusters are deleted, you can delete the HVNs as well.

Dependencies: A primary cluster can not be deleted or scaled below Plus tier while it still has an active secondary cluster. A secondary cluster would need to be deleted before the primary could be deleted or scaled below Plus.