# Organize data in vault

Cycloid credentials manager is based on vault, this allows you to create key/value credentials or also key with several fields and values.

Credentials manager lets you specify or override the path of the vault key for advanced user who want to limit the scope of variables based on https://concourse-ci.org/creds.html

Before adding stuff in credentials manager, you should ask yourself some questions:

  • Do I need a unique credential? ex: a Slack hook url could be the same for all projects.
  • Where would my credentials be used? In the pipeline, Terraform or Ansible.
  • Is my credential linked to a specific project?
  • Will this project have several env? (staging/prod)
  • Will my credentials be different for each env?

Those could result in 4 ways to store your credentials:

  1. Regular key value
# Format
name: value

# Example
slack_hook_url: https://...
1
2
3
4
5
  1. Env named key value
# Format
env_name: value

# Example
prod_htaccess: foo
staging_htaccess: bar
1
2
3
4
5
6
  1. Key containing value for each envs
# Format
name:
  Env: value
  Env2: value

# Example
htaccess:
  prod: foo
  staging: bar
1
2
3
4
5
6
7
8
9
  1. Global secret containing several key values for each env
# Format
path_name:
  Env_name: value
  Env_name2: value
  Env2_name: value
  Env2_name2: value

# Example
lambda_secrets:
  prod_htaccess: foo
  prod_key: 123
  staging_htaccess: bar
  staging_key: 456
1
2
3
4
5
6
7
8
9
10
11
12
13

This use case could help when using the vault provider in Terraform. For each credential path, you have to define it in Terraform to use it. It might be easier for example to declare all secret env vars for a lambda in the same path instead of splitting them into several paths.

# Vault in the pipeline

Pipelines are based on Concourse. So the behavior is exactly the one described here: creds and here: credential-lookup-rules

A variable in Concourse is represented by ((myvar)). There is no difference with regular variables and vault variables in a pipeline. During the creation of the pipeline, Concourse will evaluate and render all the variables provided. For each variables not provided during the insert, Concourse will assume that they are vault variables.

So variables will stay in the ((myvar)) format in the pipeline. And will be rendered using vault only during the run time of the job.

Vault variables specification When resolving a parameter such as ((foo_param)), it will look in the following credential paths, in order:

  1. PIPELINE_NAME/foo_param
  2. foo_param

That means you can define or override credentials for a specific pipeline scope.

You can specify the field to fetch via a . syntax, e.g. ((foo.bar)). This will look into your credentials for the path foo containing field name bar.

Specific field name Vault credentials are actually key-value, so if you indicate only a path like ((foo)) Concourse will default to the field name “value” corresponding to the credential foo.value field.

# Vault in Terraform

Using vault in Terraform requires 2 main steps:

Configure the pipeline to provide a vault token to Terraform Configure a provider vault into Terraform: https://www.terraform.io/docs/providers/vault/index.html

Create a YAML anchors in the pipeline to be able to use it with all Terraform jobs (plan and apply).

shared:
  # Task: connect on vault API with approle and generate a token from it
  - &vault-approle-login
    task: vault-approle-login
    config:
      platform: linux
      image_resource:
        type: docker-image
        source:
          repository: cycloid/cycloid-toolkit
          tag: latest
      run:
        path: /usr/bin/vault-approle-login
      outputs:
      - name: vault-token
        path: vault-token
    params:
      VAULT_ROLEID: ((vault.role_id))
      VAULT_SECRETID: ((vault.secret_id))
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

Then call the YAML anchors in the Terraform jobs and specify the var_files generated containing the vault token.

- name: terraform-plan-((project))-((env))

...
      - *vault-approle-login

      - put: terraform-((project))-((env))
        params:
          plan_only: true
          terraform_source: merged-((project))/
          var_files:
              - vault-token/terraform.tfvars
1
2
3
4
5
6
7
8
9
10
11

The last step on Terraform side: declare a vault provider and a vault secret matching your credential path.

#
# Vault provider
#
variable "vault_token" {}

provider "vault" {
  address = "https://vault.cycloid.io"
  token   = "${var.vault_token}"
}

variable "build_team_name" {}
variable "build_pipeline_name" {}

#
# Vault secrets
#


# build_team_name is provided by the pipeline
data "vault_generic_secret" "lambda_secrets" {
  path = "cycloid/${var.build_team_name}/lambda_secrets"
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

Then simply use it in Terraform:

# Static key example
key = "${data.vault_generic_secret.lambda_secrets.data["prod_key"]}"

# Key from variable example (with key_name=prod_key)
Key = "${data.vault_generic_secret.lambda_secrets.data[var.key_name]}"

# Mixed static and variable as key
key = "${data.vault_generic_secret.lambda_read_secrets.data[format("%s_%s", var.env, "key")]}"
1
2
3
4
5
6
7
8

# Vault provider in StackCraft

Before using Terraform Vault provider into Stackcraft, please have a look on how do Cycloid credentials work into Terraform.

In this example we will use this Cycloid credential :

stackcraft_cycloid_cred

The important information is to know the path of the credential, in this case custom_cyclobot-account. And the other thing to know is the name of the fields you want to use (here login and password).

Go on StackCraft, configure the Vault provider with those values if you are using Cycloid SaaS version:

  • address: https://vault.cycloid.io
  • token : ${var.vault_token}

Define 2 new variables, vault_token and build_team_name in StackCraft using the top left variable menu :

stackcraft_variables

In schema view, look for vault_generic_secret component. It should be located under the Data Sources section. Drag and drop it and name it for example cyclobot-account.

Configure the path with the following value cycloid/${var.build_team_name}/custom_cyclobot-account.

stackcraft_vault_generic

From this point, you can use your credential into StackCraft.

As seen earlier, our example custom_cyclobot-account contained one field called login and one called password. You can directly use those fields as variables into StackCraft using the following syntax :

  • ${data.vault_generic_secret.cyclobot-account.data["login"]}
  • ${data.vault_generic_secret.cyclobot-account.data["password"]}

The last step When you create a project from this new generated stack, make sure the Vault token provided by the pipeline is forwarded to the Terraform code.

To do so, during the project creation wizard add those 2 lines in the Terraform section:

At the beginning of the file, add those 2 lines to define variables filled by the pipeline:

variable "vault_token" {}
variable "build_team_name" {}
1
2

Then specify the previously created variables into the call of the Terraform Module below:




 
 


module "..." {
  source = ...

	vault_token = "${var.vault_token}"
	build_team_name = "${var.build_team_name}"
}
1
2
3
4
5
6

The final file should looks like:

stackcraft_project_config