WARNING

This section is made short on purpose, because it would not make sense to paraphrase Concourse's documentation; as Cycloid is based on Concourse, the concept of pipeline is defined by Concourse itself.

Vocabulary

A pipeline is a definition of jobs and resources within YAML files.

Jobs execute tasks first based on the versions of resources given as inputs.

For example a time resource could trigger a job every day at 12PM, whereas a git resource could trigger a job based on the commits/tags/branches of a git repository. These resources can also be used to do actions: report status on git repository, or a notify a slack channel.

During their executions jobs can alter resources to create artifact outputs. These outputs can be directly given to other jobs or be used later on.

You can visit other sections of our documentation for a better understanding such as the pipeline examples, or create your first private stack.

Feel also free to visit our community catalog for complete example of stacks using various common softwares.

Understanding a pipeline display

Pipeline

A pipeline contains resources (black) and jobs (green). A resource is for example a git repository.

The resource is linked to a job in this example terraform plan. The full link Full Link indicates that the plan job will automatically trigger if a new git commit is detected by the resource. But in some cases like above for the terraform apply job, there is only dot link Dot Link which means that nothing will automatically trigger this job.

We usually use this to do some kind of “approval” to let the user validate the output of the plan, then trigger manually the apply of terraform. To do that just click on the apply job and press the "+" button to trigger the job.

Trigger Job

To know the details of a build, just click on a job. It will display each steps done, the status and also each resource version (git commit, ...) used during this build.

Job Details

Special Cycloid variables

As seen in the section Special .cycloid.yml file you can provide a pipeline and sample of files displayed to the user of the stack (Variables, Terraform and Ansible samples).

If you are creating your own stack, you might be looking for a way to add some bits of templating on those files.

Cycloid console provides a minimal templating to share information like the project name or the env name of a stack.

To use it, it’s really simple, you can define those keywords:

($ environment $)
($ project $)
($ organization_canonical $)
1
2
3

A real example here in the magento pipeline variables sample:

https://github.com/cycloid-community-catalog/stack-magento/blob/master/pipeline/variables.sample.yml

Which during the creation of the stack is rendered by the console:

Variables

Style guide

Reduce log outputs

Some standard commands used in pipelines generate a lot of output lines or progress bars. Which makes the whole build log output sometimes hard to read. To reduce this output, this is our tips for common commands :

Apk

apk add -q --no-progress ...
1

Curl/wget

curl -s ...
wget -q ...
1
2

Pip

pip -q install ...
1

Easy_install

easy_install --quiet ...
1

APT

apt-get update -qq > /dev/null
apt-get install -yqq ... > /dev/null
1
2

Npm

npm install --silent
npm build --silent ...
yarn install --silent
1
2
3

Php-composer

composer -q ...
php composer.phar install -q
1
2

Ansible / Packer-ansible

Make sure all ansible command are prefixed or have this env var ANSIBLE_STDOUT_CALLBACK=actionable to use callback/actionable - shows only items that need attention

Same case for Packer, make sure all shell commands also respect the points above (apt, ...)

Eg the packer.json for the ansible call

{
    ...
    "type": "ansible-local",
    "command": "ANSIBLE_STDOUT_CALLBACK=actionable ANSIBLE_FORCE_COLOR=1 PYTHONUNBUFFERED=1 ansible-playbook",
    ...
}
1
2
3
4
5
6

Use bash trap in scripts

One of the pattern is to kill a process faster, when a job succeeds or fails and control better the log output. So that you can display a simple "OK" when everything went fine, or the full logs when it failed.

This is a dummy example that you can put at the beginning of a script in a task, which illustrates a way to implement it:

path: /bin/bash
args:
- -ec
- |
  function finish {
      if [ $rc != 0 ]; then
        echo "Build finished with error, displaying the verbose log file ..."
        cat verbose_log_file
      fi
  }
  trap 'rc=$?; set +e; finish' EXIT

  ... doing some stuff ...

  make test > verbose_log_file

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

Common pipeline errors

Job/task : invalid tag format

Error message :

Error

resource script '/opt/resource/check []' failed: exit status 1

stderr:
failed to construct tagged reference: invalid tag format
1
2
3
4

This error usually occurs because of a typo in the docker image name which contain also the tag :

      - task: foo
        config:
          image_resource:
            source:
              repository: docker:dind
1
2
3
4
5

Solution

Use the expected tag field in pipeline configuration to provide a specific docker image.

            source:
              repository: docker
              tag: dind
1
2
3

Job/task : 401 Unauthorized

Error

resource script '/opt/resource/check []' failed: exit status 1

stderr:
failed to fetch digest: 401 Unauthorized
1
2
3
4

This error usually occurs because the docker image specified does not exist on dockerhub :

            source:
              repository: docker/foobar
1
2

Solution

Ensure the image specified is a valid one using docker pull docker/foobar

Resource/Job : Expected to find variables

Error

Checking failed

Expected to find variables: git_access
1
2
3

Which mean the pipeline is looking for a variable or a credential which is not configured.

It could come from a missing variable defined into your pipeline and not in the variables.sample.yaml. Or most of the time it come from a missing credential required by the pipeline. See more about credentials here. And how configure and use it into a pipeline here. This error usually occurs because the docker image specified does not exist on dockerhub :

  - name: git_master
    type: git
    source:
      private_key: ((git_access.ssh_key))
1
2
3
4

Solution

In this case the solution was to create a new Cycloid credential nammed access of type git in our organization.