WARNING

This section is made short on purpose, because it would not make sense to paraphrase Concourse's documentation; as Cycloid is based on Concourse, the concept of pipeline is defined by Concourse itself.

The core element of a stack is a Pipeline. Each stack contains a template of a pipeline and a sample of variables. (example Stack magento pipeline template).

A pipeline describes the workflow of a stack, how to create the application and also how to automate and orchestrate the deployment of new releases.

Pipeline

Pipeline are described in YAML format.

Each stack contains a YAML template of the pipeline and a sample of variables to configure it.

When you create a project from a stack, the template is read by Cycloid to create a running pipeline.

It's important to understand the difference between a pipeline template and a running pipeline:

The template of a pipeline is contained in the stack repository. This template is used only when you create an environment. Which means, if you edit the pipeline using the Dashboard, you actually edit the running pipeline created from the template. Not the template itself.

Which also means, updating the template will not update the running pipelines that you have already created. See Refresh pipeline section to refresh your running pipeline using the latest version of the template.

Important

Make sure to always keep in mind the difference between running pipeline and pipeline template described above.

To know little bit more about pipeline variables, feel free to Click here. It's also possible to use Cycloid credentials as value for those variables : Use Cycloid Credentials in a pipeline.

Vocabulary

A pipeline is a definition of jobs and resources within YAML files.

Jobs execute tasks first based on the versions of resources given as inputs.

For example a time resource could trigger a job every day at 12PM, whereas a git resource could trigger a job based on the commits/tags/branches of a git repository. These resources can also be used to do actions: report status on git repository, or a notify a slack channel.

During their executions jobs can alter resources to create artifact outputs. These outputs can be directly given to other jobs or be used later on.

You can visit other sections of our documentation for a better understanding such as the pipeline examples, or create your first private stack.

Feel also free to visit our community catalog for complete example of stacks using various common softwares.

Refresh a running pipeline from the template

Running pipelines can shift from a template, because you updated the template or you edited directly the running pipeline using the Cycloid dashboard.

If you feel the need to sync back your running pipeline from the template in the stack :

  1. Simply go on Pipelines view.
  2. Select the desired pipeline.
  3. Click on top right Reset pipeline button.
  4. A wizard will display the code of the pipeline and the variables from the sample in the stack.
  5. Make sure you enter the right values in the variables
  6. Click on Save

TIP

We currently recommennd to backup the variables you provided to configure your pipeline to help you if you reset it.

Note : We are currently working on a feature provide the value of the variables from your running pipeline during a refresh.

Understanding a pipeline display

Pipeline

A pipeline contains resources (black) and jobs (green). A resource is for example a git repository.

The resource is linked to a job in this example terraform plan. The full link Full Link indicates that the plan job will automatically trigger if a new git commit is detected by the resource. But in some cases like above for the terraform apply job, there is only dot link Dot Link which means that nothing will automatically trigger this job.

We usually use this to do some kind of “approval” to let the user validate the output of the plan, then trigger manually the apply of terraform. To do that just click on the apply job and press the "+" button to trigger the job.

Trigger Job

To know the details of a build, just click on a job. It will display each steps done, the status and also each resource version (git commit, ...) used during this build.

Job Details

Special Cycloid variables

As seen in the section Special .cycloid.yml file you can provide a pipeline and sample of files displayed to the user of the stack (Variables, Terraform and Ansible samples).

If you are creating your own stack, you might be looking for a way to add some bits of templating on those files.

Cycloid console provides a minimal templating to share information like the project name or the env name of a stack.

To use it, it’s really simple, you can define those keywords:

($ environment $)
($ project $)
($ organization_canonical $)
1
2
3

A real example here in the magento pipeline variables sample:

https://github.com/cycloid-community-catalog/stack-magento/blob/master/pipeline/variables.sample.yml

Which during the creation of the stack is rendered by the console:

Variables

Style guide

Reduce log outputs

Some standard commands used in pipelines generate a lot of output lines or progress bars. Which makes the whole build log output sometimes hard to read. To reduce this output, this is our tips for common commands :

Apk

apk add -q --no-progress ...
1

Curl/wget

curl -s ...
wget -q ...
1
2

Pip

pip -q install ...
1

Easy_install

easy_install --quiet ...
1

APT

apt-get update -qq > /dev/null
apt-get install -yqq ... > /dev/null
1
2

Npm

npm install --silent
npm build --silent ...
yarn install --silent
1
2
3

Php-composer

composer -q ...
php composer.phar install -q
1
2

Ansible / Packer-ansible

Make sure all ansible command are prefixed or have this env var ANSIBLE_STDOUT_CALLBACK=actionable to use callback/actionable - shows only items that need attention

Same case for Packer, make sure all shell commands also respect the points above (apt, ...)

Eg the packer.json for the ansible call

{
    ...
    "type": "ansible-local",
    "command": "ANSIBLE_STDOUT_CALLBACK=actionable ANSIBLE_FORCE_COLOR=1 PYTHONUNBUFFERED=1 ansible-playbook",
    ...
}
1
2
3
4
5
6

Use bash trap in scripts

One of the pattern is to kill a process faster, when a job succeeds or fails and control better the log output. So that you can display a simple "OK" when everything went fine, or the full logs when it failed.

This is a dummy example that you can put at the beginning of a script in a task, which illustrates a way to implement it:

path: /bin/bash
args:
- -ec
- |
  function finish {
      if [ $rc != 0 ]; then
        echo "Build finished with error, displaying the verbose log file ..."
        cat verbose_log_file
      fi
  }
  trap 'rc=$?; set +e; finish' EXIT

  ... doing some stuff ...

  make test > verbose_log_file

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

Common pipeline errors

Job/task : invalid tag format

Error message :

Error

resource script '/opt/resource/check []' failed: exit status 1

stderr:
failed to construct tagged reference: invalid tag format
1
2
3
4

This error usually occurs because of a typo in the docker image name which contain also the tag :

      - task: foo
        config:
          image_resource:
            source:
              repository: docker:dind
1
2
3
4
5

Solution

Use the expected tag field in pipeline configuration to provide a specific docker image.

            source:
              repository: docker
              tag: dind
1
2
3

Job/task : 401 Unauthorized

Error

resource script '/opt/resource/check []' failed: exit status 1

stderr:
failed to fetch digest: 401 Unauthorized
1
2
3
4

This error usually occurs because the docker image specified does not exist on dockerhub :

            source:
              repository: docker/foobar
1
2

Solution

Ensure the image specified is a valid one using docker pull docker/foobar

Resource/Job : Expected to find variables

Error

Checking failed

Expected to find variables: git_access
1
2
3

Which mean the pipeline is looking for a variable or a credential which is not configured.

It could come from a missing variable defined into your pipeline and not in the variables.sample.yaml. Or most of the time it come from a missing credential required by the pipeline. See more about credentials here. And how configure and use it into a pipeline here. This error usually occurs because the docker image specified does not exist on dockerhub :

  - name: git_master
    type: git
    source:
      private_key: ((git_access.ssh_key))
1
2
3
4

Solution

In this case the solution was to create a new Cycloid credential nammed access of type git in our organization.