Config and catalog repositories

Introduction

Catalog repositories are a git repository (or just a branch) containing one or more stacks. They allow you to provide a way to share a catalog of stacks to your users's inside your organization.

Config repositories are a git repository (or branch) containing the configuration for your stacks for each environment they are used.

All cycloid projects uses stacks to perform IaC. The config contains any variables, templates and files that will be merged with the stack to configure the stack for a given project in a specific environment. This notion is explained in the stack section.

You can create as much catalog and configuration repositories as you need.

tip

You can also group everything on one repo and make two orphan branch, stacks and config.

This can be convenient an allow you to keep stacks and their relative config closer.

See the example below.

Prerequisites

• A git source code provider. We support most git provider out there.
• Valid git credentials for your repositories stored in Cycloid credentials.
  • Use a SSH Key credential for SSH urls
  • Use a basic auth credential for HTTPS urls
  • Credentials should have read and write access on repositories.

note

We recommend using SSH Keys, but we also support HTTP tokens (depending on your git provider)

• One repository or one branch for stacks
• One repository or one branch for config

warning

The specified Git repositories has to be accessible by Cycloid's API and potentially by Cycloid workers if you want to use it inside a pipeline.

important

If your Git server uses a different port than the standard 22, please note that the url format should follow the syntax from the git protocol:

ssh://[<user>@]<host>[:<port>]/<path-to-git-repo>

git://<host>[:<port>]/<path-to-git-repo>

http[s]://<host>[:<port>]/<path-to-git-repo>

For example, if your git server ssh port is 2222 the url should be as follow:

ssh://git@your.ssh.server:2222/repo.git

Quickstart

1. Create two repositories (or two orphaned branches), one for stacks and one for configs.

The most common and convenient way is to create two orphaned branches.

The most common and handy way is to create two orphaned branches

To do so, simply clone your Git repository and run the following commands:

# Config
git checkout --orphan  config
git rm -rf .
touch README.md
git add .
git commit -m "Init config"
git push origin config

# Stack
git checkout --orphan  stacks
git rm -rf .
touch README.md
git add .
git commit -m "Init stacks"
git push origin stacks

This will create 2 new orphan branches in the Git repository. Why orphan? Because those branches will contain totally different content. Making them orphan will create completely different branches with their own history in order to avoid merging them together.

2. Generate the related credentials and include in cycloid (lookup credentials management)

3. In the cycloid console, go to the Stacks > Catalog repositories menu.

4. Fill in the name, url, credential and branch of the stack repository.

You can choose the branch via a dropdown

You will be able to see your catalog repository on the list, and refresh it.

5. Go to Projects > Config repositories.

6. Click on Add config repository

7. Fill in the name, url, credential and branch of the config repository.

tip

You can set a config repo as default in your current org.

This is useful to centralize configuration on one repository.

Concepts

Default config repo

You can set a default config repository in an organization.

This repository will be filled as config repo for each new project.

Each project can override locally the config repo they use.

You can change it at project creation:

Or on the project view, Configuration tab:

Migrate a config or catalog repository

You may need at some point to change the reference to your config or catalog repo. This may occur for example when changing your git provider.

It is not possible (and shouldn't) to delete a catalog repo that contains stacks that are currently in use in at least 1 project.

So we implemented a way to just change the url of the config repo to update it in-place to all projects.

danger

Be careful to the following steps.

Doing this wrong could break projects an pipeline inside your organization !

Also, migrating a config / catalog repo could mean that you have to update all your legacy project/env that do not use cycloid special variables for config and catalog repo reference.

Finally, you will also loose git references history on concourse git resources.

This migration will take time an put some downtime on your pipeline.

tip

This process is far easier if your ensure that all your stacks use cycloid special vars to reference catalog and config repo.

Migrate a catalog repository

1. Copy all the stacks from the old catalog repository to the new.

warning

The structure of the repository and the relative path of all the stacks should stay the same !

2. Create or update the Git credential for the new repository in cycloid console.

3. Go to Stacks -> Catalog repositories and click on your catalog repository

4. You can then edit the catalog repo.

   To change the url, you must click on the padlock on the right of the URL field, this will allow you to change the git URL.

   Make sure it is valid.

   You can also ajust the Credential and branch for the new repository.

   Then save your changes.

5. You can then refresh the catalog repo to ensure that the operation was successful.

   If your are just migrating your stacks to another repo without adding new stacks, you should have this dialog after the refresh:

6. At this stage, you should check all your project and migrate all the reference from the old catalog repo to the new one.

note

If you need help with this proccess, feel free to reach us.

We can help you plan and execute this process and minimise downtime.

Migrate a config repo

Migrating a config repository is similar to the catalog repo.

1. Copy the configs from the old repo to the new.

2. Go to Projects -> Config repositories and update the config repo to migrate.

tip

Same as the config repo, you can update the url, credential and branch.

3. Check and update all your project.