Why ?

The pipeline engine used in Cycloid Devops Platform is composed of 2 components:

  • Scheduler : Server side on which workers are connected. It schedules pipeline jobs and spreads tasks on your workers.
  • Workers : Controlled by the scheduler to run the jobs of your pipelines.

There are severals ways to deploy your own workers, it can differ a bit if you subscribe to Cycloid Devops Platform:

  • Dedicated: AWS
  • Dedicated: Onprem
  • SaaS

All methods rely on the source code of the public stack-external-worker.

If you subscribed to Cycloid Devops Platform in the SaaS version, the current easiest method to deploy your own workers on AWS is the CloudFormation way.

Deploy Cycloid worker on AWS with CloudFormation

AWS

As described above, the CloudFormation deployment is based on a template located under our stack-external-worker git repository.

The CloudFormation stack will create an AWS autoscaling group with EC2 Cycloid workers. The default values are already pre-filled to deploy and connect workers on our public SaaS, then you will be able to modify them according to your needs.

cloudformation-stack

Follow these steps to create a CloudFormation stack:

  1. Connect on http://console.cycloid.io.
  2. Click on the name of your Organization within the left menu Manage org
  3. Go into the CI/CD workers tab.
  4. Select the region in which you want to deploy the workers.
  5. Click on the button Launch CloudFormation Stack. It will redirect you to Amazon Console. (See comments at the end of the page)
  6. Configure the CloudFormation stack to match your AWS configuration:
    • KeyName : The EC2 Key Pair to allow SSH access to the instances.
    • MaxNumberOfWorkers: Maximum number of Cycloid workers.
    • SSHSecGroup: Instances from this security group will be allowed to connect using SSH on Cycloid workers.
    • MetricsSecGroup: Instances from this security group will be allowed to read metrics endpoint on Cycloid workers. Can be a random security group if you don't have monitoring.
    • VpcId: VpcId of your existing Virtual Private Cloud (VPC).
    • Subnets: The list of SubnetIds in your Virtual Private Cloud (VPC).
  7. Select the checkbox I acknowledge that AWS CloudFormation might create IAM resources. The stack will create an IAM role for the workers.
  8. Click on Create button to create the stack.

After the deployment of the CloudFormation stack, get back on the CI/CD workers tab in Cycloid dashboard. It will display all your connected workers and their status:

worker-status

Note: Sometimes, CloudFormation returns an error 400 (Bad Request), we are aware about of this and we are working with AWS to help them to fix it. Meanwhile, there are two options to solve it:

  • Once you clicked on the button Launch CloudFormation Stack and have realized that you have a blank page or similar, you can copy the url and paste it in a new tab, then you will be redirected to the correct page.
  • Go to CloudFormation home page, then click back on Launch CloudFormation Stack button and it will work as expected.

Change Cycloid worker size

  1. Connect on AWS Management Console
  2. Go on the CloudFormation service page.
  3. Select the cycloid-ci-workers stack and click on the top right Update button.
    • Select the Use current template checkbox.
    • Click on next.
  4. Select the desired InstanceType.
  5. Click next up to the Update stack button.

Update from the latest version of Cycloid CloudFormation stack

  1. Connect on AWS Management Console
  2. Go on the CloudFormation service page.
  3. Select the cycloid-ci-workers stack and click on the top right Update button.
    • Select the Replace current template checkbox.
    • Click on Amazon S3 URL.
    • Put the following url https://s3-eu-west-1.amazonaws.com/cycloid-cloudformation/external-worker-aws-cf-template.yaml.
    • Click next up to the Update stack button.

Deploy Cycloid worker on GCP with Deployment manager

GCP

The Deployment manager is based on a template located under our stack-external-worker git repository.

The Deployment manager stack will create an GCP autoscaler with Cycloid workers on Compute engine. The default values are already pre-filled to deploy and connect workers on our public SaaS, then you will be able to modify them according to your needs.

deployment-manager-stack

Before you start, make sure you installed gcloud and configured it. Starting with deployment manager

Follow these steps to create a Deployment manager stack:

  1. Connect on http://console.cycloid.io.
  2. Beside, open also a console terminal on your computer.
  3. On the left menu click on Credentials.
  4. In the list of credentials, you should have Cycloid-worker-keys. Click on it.
  5. In your terminal write : export CYCLOID_WORKER_KEY=$(base64 -w 0).
    • Then in the prompt paste the content of the ssh_prv field from Cycloid-worker-keys credential.
    • And press ctrl + d
  6. Click on the name of your Organization within the left menu Manage org
  7. Get the name of your organization and type export CYCLOID_ORG=<organization name> in your terminal.
  8. Get the ci_team_member value and type export CYCLOID_TEAM=<value of ci_team_member> in your terminal.
  9. Download the deployment manager template from our git : wget -O /tmp/external-worker-gcp-template.yaml.sample https://raw.githubusercontent.com/cycloid-community-catalog/stack-external-worker/master/extra/gcp/external-worker-gcp-template.yaml.sample
  10. Remplace the shell variables in this sample using envsubst command. : envsubst < /tmp/external-worker-gcp-template.yaml.sample > external-worker-gcp-template.yaml
  11. Deploy the stack in your terminal gcloud deployment-manager deployments create cycloid-worker --config external-worker-gcp-template.yaml (more details here)

Note

You can edit the external-worker-gcp-template.yaml file to select the number of deployed worker or the compute size.

instanceType: n1-standard-4
numberOfWorkers: 1
1
2

After the deployment of the stack, go on CI/CD workers tab in Cycloid dashboard. It will display all your connected workers and their status:

worker-status