Why ?

The pipeline engine used in Cycloid Devops Framework is composed of 2 components:

  • Scheduler : Server side on which workers are connected. It schedules pipeline jobs and spreads tasks on your workers.
  • Workers : Controlled by the scheduler to run the jobs of your pipelines.

There are severals ways to deploy your own workers, it can differ a bit if you subscribe to Cycloid Devops Framework:

  • Dedicated: AWS
  • Dedicated: Onprem
  • SaaS

All methods rely on the source code of the public stack-external-worker.

If you subscribed to Cycloid Devops Framework in the SaaS version, the current easiest method to deploy your own workers on AWS is the CloudFormation way.

Local: Deploy a local Cycloid worker using Docker

Docker

TIP

This solution is perfect to try Cycloid product but for production worker, we advice you to use the dedicated setup for each cloud providers.

To run a local Cycloid worker it requires to have a machine with Docker installed on it.

Follow these steps to create a local worker based on Docker:

  1. Connect on http://console.cycloid.io.
  2. On the left menu click on Credentials.
  3. In the list of credentials, find Cycloid-worker-keys. Click on it.
    • Copy the content of the ssh_prv field to Cycloid worker key input below.
  4. Click on the name of your Organization within the left menu Manage org
    • Copy the ci_team_member value to Ci team member input below.
  5. Click on button Generate below.
Worker configuration for Docker

GenerateBack

After the deployment of the stack, go on CI/CD workers tab in Cycloid dashboard. It will display all your connected workers and their statuses:

worker-status

AWS : Deploy Cycloid worker on AWS with CloudFormation

AWS

As described above, the CloudFormation deployment is based on a template located under our stack-external-worker git repository.

The CloudFormation stack will create an AWS autoscaling group with EC2 Cycloid workers. The default values are already pre-filled to deploy and connect workers on our public SaaS, then you will be able to modify them according to your needs.

cloudformation-stack

Follow these steps to create a CloudFormation stack:

  1. Connect on http://console.cycloid.io.
  2. Click on the name of your Organization within the left menu Manage org
  3. Go into the CI/CD workers tab.
  4. Select the region in which you want to deploy the workers.
  5. Click on the button Launch CloudFormation Stack. It will redirect you to Amazon Console. (See comments at the end of the page)
  6. Configure the CloudFormation stack to match your AWS configuration:
    • KeyName: The EC2 Key Pair to allow SSH access to the instances.
    • MaxNumberOfWorkers: Maximum number of Cycloid workers.
    • SSHSecGroup: Instances from this security group will be allowed to connect using SSH on Cycloid workers.
    • MetricsSecGroup: Instances from this security group will be allowed to read metrics endpoint on Cycloid workers. Can be a random security group if you don't have monitoring.
    • VpcId: VpcId of your existing Virtual Private Cloud (VPC).
    • Subnets: The list of SubnetIds in your Virtual Private Cloud (VPC).
  7. Select the checkbox I acknowledge that AWS CloudFormation might create IAM resources. The stack will create an IAM role for the workers.
  8. Click on Create button to create the stack.

After the deployment of the CloudFormation stack, get back on the CI/CD workers tab in Cycloid dashboard. It will display all your connected workers and their status:

worker-status

Note: Sometimes, CloudFormation returns an error 400 (Bad Request), we are aware about of this and we are working with AWS to help them to fix it. Meanwhile, there are two options to solve it:

  • Once you clicked on the button Launch CloudFormation Stack and have realized that you have a blank page or similar, you can copy the url and paste it in a new tab, then you will be redirected to the correct page.
  • Go to CloudFormation home page, then click back on Launch CloudFormation Stack button and it will work as expected.

Change Cycloid worker size

  1. Connect on AWS Management Console
  2. Go on the CloudFormation service page.
  3. Select the cycloid-ci-workers stack and click on the top right Update button.
    • Select the Use current template checkbox.
    • Click on next.
  4. Select the desired InstanceType.
  5. Click next up to the Update stack button.

Update from the latest version of Cycloid CloudFormation stack

  1. Connect on AWS Management Console
  2. Go on the CloudFormation service page.
  3. Select the cycloid-ci-workers stack and click on the top right Update button.
    • Select the Replace current template checkbox.
    • Click on Amazon S3 URL.
    • Put the following url https://s3-eu-west-1.amazonaws.com/cycloid-cloudformation/external-worker-aws-cf-template.yaml.
    • Click next up to the Update stack button.

GCP : Deploy Cycloid worker on GCP with Deployment manager

GCP

The Deployment manager is based on a template located under our stack-external-worker git repository.

The Deployment manager stack will create an GCP autoscaler with Cycloid workers on Compute engine. The default values are already pre-filled to deploy and connect workers on our public SaaS, then you will be able to modify them according to your needs.

deployment-manager-stack

Before you start, make sure you installed gcloud and configured it. Starting with deployment manager

Follow these steps to create a Deployment manager stack:

  1. Connect on http://console.cycloid.io.
  2. On the left menu click on Credentials.
  3. In the list of credentials, you should have Cycloid-worker-keys. Click on it.
    • Copy the content of the ssh_prv field to Cycloid worker key input below.
  4. Click on the name of your Organization within the left menu Manage org
    • Copy the name of your organization to Cycloid organization input below.
    • Copy the ci_team_member value to Ci team member input below.
  5. Click on button Generate below. more details here
Worker configuration for GCP

GenerateBack

Note

You can edit the external-worker-gcp-template.yaml file to select the number of deployed worker or the compute size.

instanceType: n1-standard-4
numberOfWorkers: 1
1
2

After the deployment of the stack, go on CI/CD workers tab in Cycloid dashboard. It will display all your connected workers and their status:

worker-status

Azure : Deploy Cycloid worker on Azure with Resource Manager Template

Azure

The Azure Resource Manager template is located under our stack-external-worker git repository.

The template will create an Azure Virtual Machine Scale Set of Cycloid workers. The default values are already pre-filled to deploy and connect workers on our public SaaS, then you will be able to modify them according to your needs.

azure-resource-manager-template

Follow these steps to deploy the Azure Resource Manager template:

  1. Click on the following button to load the external-worker azure template for deployment. It will redirect you to the Azure Console.
  2. Beside that, connect on http://console.cycloid.io in another browser tab.
  3. On the left menu click on Credentials.
  4. In the list of credentials, you should have Cycloid-worker-keys. Click on it.
    • Copy the content of the ssh_prv field to Cycloid worker key input below and click Generate.
    • Copy the generated output into the Worker Key field of the Azure Console openned with the step 1.
  5. Click on the name of your Organization within the left menu Manage org
  6. Copy the name of your organization into the Customer Tag field of the Azure Console openned with the step 1.
  7. Copy the ci_team_member value into the Team ID field of the Azure Console openned with the step 1.
  8. Configure the Custom deployment to match your needs:
    • Resource group: Select or create a Resource Group for this deployment.
    • Number Of Workers: Maximum number of Cycloid workers.
    • Virtual Network Name: Virtual Network name of your existing Virtual Network.
    • Subnet Name: Subnet name of your existing Subnet in your existing Virtual Network.
    • Admin Password of Key: The SSH public key or password to allow SSH access to the instances.
    • Sg Authorized Cidrs: IP CIDRs that will be allowed to connect using SSH on Cycloid workers.
  9. Click on Purchase button to create the stack.
Worker configuration for Azure

GenerateBack

After the Custom template deployment, go to the CI/CD workers tab in Cycloid dashboard. It will display all your connected workers and their status:

worker-status