Dynamic, Synced Configuration

Waypoint can sync application configuration values with external systems such as Kubernetes ConfigMaps, HashiCorp Vault, Amazon Parameter Store, and more. These are known as dynamic configuration values. As the values change, Waypoint will automatically update your deployments and restart your application.

Dynamic values make it easy to utilize external configuration stores and the functionality they provide, such as automatic credential rotation in systems such as Vault or Amazon Parameter Store.

This functionality requires the Waypoint entrypoint.

Configuration Sources

The following is a list of currently available configuration sources and their associated documentation:

• Terraform Cloud - Read Terraform state outputs from Terraform Cloud.
• Kubernetes - Read values from ConfigMaps and Secrets.
• Vault - Read values from Vault secrets, including dynamic secrets such as database credentials.
• AWS SSM - Read configuration values from AWS SSM Parameter Store.
• Consul - Read values from Consul key value storage.

Setting Dynamic Values via waypoint.hcl

Dynamic configuration values are set in the waypoint.hcl file using the dynamic function within the same config stanza as static configurations. This specifies that a configuration value is synced with an external system dynamically.

config {
  env = {
    PORT = dynamic("kubernetes", {
      name = "my-config-map" # ConfigMap name
      key  = "port"
    })
  }
}

The example above syncs the PORT environment variable with the "port" key in the "my-config-map" Kubernetes ConfigMap. The "sync" actively watches the defined source, updates the environment variable whenever the value changes, and restarts your application.

The dynamic function has the following signature:

dynamic(SOURCE, PARAMS)

The SOURCE parameter is a string specifying the config source to sync with and PARAMS is a map of parameters for that config source. The set of parameters is dependent on the source being used. The list of sources supported is in the Configuration Sources section.

Unsetting Dynamic Values

To unset a dynamic configuration value, delete it from the waypoint.hcl file and then use the CLI to set it to an empty string as shown below. Deleting the key from the Waypoint file and syncing it will not be enough.

$ waypoint config set PORT=

Source Settings

Configuration sources sometimes support global settings for their behavior or require global settings to be set for them to function at all. For example, Vault has some global settings around how to authenticate. Without setting these, Vault configurations in applications will not work. These source settings are set at the server level and used by all dynamic calls.

To set configuration source settings, the waypoint config source-set CLI command is used:

$ waypoint config source-set -type=vault -config="auth=aws"

The documentation for the source settings available to a configuration source are alongside their parameters in the plugin documentation.

Scoped Source Settings

The waypoint config source-set command's -scope flag will configure a given config source to be used within a specific scope. The supported values are below. For a given app or project, the config source with the "most specific" scope will be used.

• global: Source setting applied to all projects and their applications
• project: Applies source setting only to this project
• app: Applies source setting only to this application

For example, you may have three config sources configured in your Waypoint server, one at the global scope, one at the project scope for project foo, and one at the app scope for app bar, in project foo. These have been configured with the commands below, for the Vault plugin. The Vault token set for each of these config sources is tailored to each scope, with only the necessary set of permissions for each.

$ waypoint config source-set -scope=global -type=vault -config=addr=https://vault.service.consul:8200 -config=token=waypoint-global-token

$ waypoint config source-set -scope=project -p=foo -type=vault -config=addr=https://vault.service.consul:8200 -config=token=foo-project-token

$ waypoint config source-set -scope=app -p=foo -a=bar -type=vault -config=addr=https://vault.service.consul:8200 -config=token=bar-app-token

In accordance with the scopes set, waypoint config source-get will return the config source corresponding with the specified scope.

$ waypoint config source-get -scope=global -type=vault
» Config Source Info:
   Type: vault
  Scope: global

  KEY  |           VALUE
--------+----------------------------
  addr  | https://vault.service.consul:8200
  token | waypoint-global-token

$ waypoint config source-get -scope=project -p=foo -type=vault
Config Source Info:
       Type: vault
      Scope: project
    Project: foo

  KEY  |           VALUE
--------+----------------------------
  addr  | https://vault.service.consul:8200
  token | foo-project-token

$ waypoint config source-get -scope=app -p=foo -a=bar -type=vault
Config Source Info:
       Type: vault
      Scope: app
    Project: foo
        App: bar

  KEY  |           VALUE
--------+----------------------------
  addr  | https://vault.service.consul:8200
  token | bar-app-token

In the third of the series of commands above, were the app-scoped config source absent, the project-scoped source would be returned. Were the project-scoped config source absent as well, the global-scoped config source would be returned.

To view all config sources, the all value can be set for the scope:

$ waypoint config source-get -scope=all
       TYPE       |  SCOPE  | PROJECT | APP | WORKSPACE
------------------+---------+---------+-----+------------
  vault           | global  |         |     |
  vault           | project | foo     |     |
  vault           | app     | foo     | bar |

The Waypoint Entrypoint will always attempt to retrieve configuration for the scope of the deployed app.

In addition to the global, project, and app scopes, when setting a config source, it may also be scoped to a specific workspace. This can be achieved by using the -workspace (or -w) flag. The config source set for a given workspace will be used only for operations and app instances deployed within that workspace.

$ waypoint config source-set -scope=app -p=foo -a=bar -w=foobar -type=vault -config=addr=https://vault.service.consul:8200 -config=token=bar-app-foobar-workspace-token

$ waypoint config source-get -scope=app -p=foo -a=bar -w=foobar -type=vault
Config Source Info:
       Type: vault
      Scope: app
    Project: foo
        App: bar
  Workspace: foobar

  KEY  |           VALUE
--------+----------------------------
  addr  | https://vault.service.consul:8200
  token | bar-app-foobar-workspace-token

Specifying workspace-scoped config sources is especially useful when your Waypoint server is used for releasing applications to multiple environments. A Vault config source for the dev workspace may have a different address or less restrictive credentials to be used than deployments in the prod workspace, for example.

Deleting Configuration Source Settings

To unset the configuration after it has been set with waypoint config source-set, specify the -delete flag.

$ waypoint config source-set -type=vault -delete

Debugging Dynamic Configuration

If a configuration value is not available (the environment variable is not set or the value is not what you expect), then the best debugging tool is waypoint logs. The logs command will show entrypoint logs where you will see any errors related to configuration syncing.

The waypoint logs output below is from an application deployed with Waypoint using Kubernetes ConfigMaps but deployed within a Nomad cluster. The lines have been purposely shortened slightly to improve readability on the website.

$ waypoint logs
7PMDER: [WARN]  entrypoint.config.watcher: error reading configuration values, all will be dropped: source=kubernetes
err="unable to load in-cluster configuration, KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT must be defined"
7PMDER: [INFO]  entrypoint.child: starting child process: args=[/usr/local/bin/docker-entrypoint.sh, node, app.js]
cmd=/usr/local/bin/docker-entrypoint.sh
7PMDER: Example app listening on port 3000!

You can see the first line has an error reading configuration value. If you keep scrolling right, you can see the source is kubernetes and the err is shown. In this case, it's telling us that it couldn't load any in-cluster configuration. Upon thinking about this, we can realize that we didn't deploy to a Kubernetes cluster so this error makes sense, and perhaps we need to do something else to enable out-of-cluster authentication.

To iterate on configuration, change the waypoint.hcl file and run waypoint config sync. This allows you to change configuration without doing a full redeploy.

Listing Environment Variables

If you don't see any errors in the log, it may be possible the environment variables are set correctly. To verify that environment variables are set, open a shell with waypoint exec and run env. This command will list the environment variables that are available to your application.

$ waypoint exec /bin/sh
# env
NODE_VERSION=13.14.0
HOSTNAME=55882281dde1
YARN_VERSION=1.22.4
PORT=3000
HOME=/root
TERM=xterm-256color
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
PWD=/usr/src/app