CI Implementations with Tekton Resolvers and Pipelines As Code

For sometime now we have been, together with DevOps empowered application teams and DevOps engineers, using the OpenShift Pipelines, OpenShift's productised version of tekton.dev, as the Continuous Integration (CI) technology of choice when onboarding new applications or migrating existing ones into Red Hat's kubernetes.

Recently, the article on Migration from ClusterTasks to Tekton Resolvers in OpenShift Pipelines by colleagues Vincent Demeester and Koustav Saha and the GA of Pipelines as code prompted me to perform a refresher on the resources used as a baseline when starting a completely new implementation/migration or introducing the topic of tekton to a team.

In this article we will go through the manner in which we performed a classic CICD release cycle with tekton and gitops as well as how we can overhaul the existing material to give practical substance to all possible manners of utilsing tasks in pipelines beyond installing them as ClusterTask or Task as well as moving from pipeline webhooks for CI (and CD) to pipeline as code.

In the accompanying tekton-gitops-workshop github repository the setup populates:

• the OpenShift Gitops and Openshift Pipeline operators in an OpenShift cluster
• the Gitea Git Repository provider with 2 repositories
  • application-source repository hosts the application source
  • application-deploy hosts an ArgoCD Application that in turn bootstraps via an ApplicationSet the environment based applications for dev, test, prod, delivers and monitors the kubernetes configurations for the application in each environment.
• 2 OpenShift pipelines
  • ci-pipeline: runs the application testing, packaging and containerisation process
  • cd-pipeline: with a PR from the ci-pipeline to the application-deploy git repo the CD process initiates the process of delivering the new application deployments into OpenShift
• Various OpenShift Pipelines objects (Task, ClusterTask, Pipeline, EventListerner, TriggerBinding, TriggerTemplate) are installed in order to facilitate the pipeline implementation and triggering.

The resulting setup offers the ability through Git repository updates (as shown in the following diagram) to run through a lifecycle of Continuous Integration and Continuous Delivery with the tekton pipelines whilst the Continuous Deployment is handled by the ArgoCD component and again promoted via git Pull Requests.

Between the article mentioned earlier, the OpenShift Remote Pipelines Tasks Resolvers documentation and tekton's documentation there is very good information on how to configure the resolvers, what are the resolver types and purpose of each. We shall concentrate on how we made the changes to the original pipelines to take advantage of the resolver functionality.

It is important to note at this point that the 2 pipelines contain the following type of tasks:

• ClusterTask resources

  • git-clone (clones a repo from the provided url)
  • s2i-java (clones a Git repository and builds and pushes a container image using S2I and a Java builder image).

• Custom Task resources

  • next-env (works out the next environment to promote the updated image to)
  • sync-argo (executes an argocd sync to update the targeted environment resources)
  • create-pr-to-deploy (creates a PR towards the application-deploy repository with an updated deployment to the latest produced image)
  • generate-version(generates application version based on timestamp and pom version.)
  • mock(a task that mocks pipeline activities)
  • tag-image(Tags the image to the generated version)

A cluster resolver is a Tekton Pipelines feature that allows you to reference tasks and pipelines from other namespaces, therefore it can act as a cluster wide known location for DevOps engineers to maintain tasks on behalf of pipeline creators who can in-turn reference them from that namespace.

In the cluster-resolver branch the setup has been updated with the 2 aforementioned ClusterTask resources git-clone and s2i-java

• implemented as Task resources and
• installed in sk-workshop-ci-components namespace

The ci-pipeline has been updated to use now the resolver: cluster to reference the task from the CI components namespace.

 1  # ------------ CLONE APP SOURCE ------------ #
 2    - name: git-app-clone
 3      taskRef:
 4        resolver: cluster
 5        params:
 6          - name: kind
 7            value: task
 8          - name: name
 9            value: git-clone-custom
10          - name: namespace
11            value: $(params.CI_NS_PREFIX)-workshop-ci-components
12   ...
13  # ------------ BUILD IMAGE ------------ #
14    - name: build-image
15      runAfter:
16      - nexus-upload
17      taskRef:
18        resolver: cluster
19        params:
20          - name: kind
21            value: task
22          - name: name
23            value: s2i-java-custom
24          - name: namespace
25            value: $(params.CI_NS_PREFIX)-workshop-ci-components   

A git resolver is a Tekton Pipelines feature that allows you to reference tasks and pipelines from Git repositories., therefore it can act as a common team or organization source control location for DevOps engineers to maintain tasks on behalf of pipeline creators who can in-turn reference them from that repository.

In the git-resolver branch the setup has been updated with the 2 aforementioned ClusterTask resources git-clone and s2i-java

• implemented as Task resources,
• now available from a purposed setup tekton-catalog repository with a versioned entry for 0.1/git-clone.yaml and 0.1/s2i-java.yaml

The ci-pipeline has been updated to use now the resolver: git to reference the task from tekton-catalog repository.

 1    # ------------ CLONE APP SOURCE ------------ #
 2    - name: git-app-clone
 3      taskRef:
 4        resolver: git
 5        params:
 6          - name: url
 7            value: 'https://github.com/tektoncd/catalog.git'
 8          - name: revision
 9            value: main
10          - name: pathInRepo
11            value: task/git-clone/0.9/git-clone.yaml
12   ...
13    # ------------ BUILD IMAGE ------------ #
14    - name: build-image
15      runAfter:
16        - nexus-upload
17      taskRef:
18        resolver: git
19        params:
20          - name: url
21            value: https://github.com/skoussou/tekton-catalog.git
22          - name: revision
23            value: main
24          - name: pathInRepo
25            value: ci-tasks/resources/s2i-java-custom/0.1/s2i-java.yaml

A bundle resolver is a feature in Tekton Pipelines that allows you to reference Tekton resources from a Tekton bundle image, therefore it can act as a common set of prepared images by devops engineers to maintain tasks on behalf of pipeline creators who can in-turn reference them from the bundles.

In the bundle-resolver branch the setup has been updated with the aforementioned ClusterTask resource s2i-java

• implemented as a Task resource,
• now available from a purposed prepared image bundle stored in quay.io/skoussou/s2i-java repository.

Firstly, the bundle was built from the 0.1/s2i-java.yaml and versioned to 0.1

1tkn bundle push quay.io/skoussou/s2i-java:0.1 -f ci-tasks/resources/s2i-java-custom/0.1/s2i-java.yaml
2
3*Warning*: This is an experimental command, it's usage and behavior can change in the next release(s)
4Creating Tekton Bundle:
5	- Added Task: s2i-java-custom to image
6
7Pushed Tekton Bundle to quay.io/skoussou/s2i-java@sha256:88fb756feeab8e79cf43ba4845cb4b5f2bc27c38f528bba95708c23b34a75908

The ci-pipeline has been updated to use now the resolver: bundles to reference the image bundle from quay.io repository.

 1# ------------ BUILD IMAGE ------------ #
 2  - name: build-image
 3    runAfter:
 4      - nexus-upload
 5    taskRef:
 6      resolver: bundles
 7      params:
 8        - name: bundle
 9          value: quay.io/skoussou/s2i-java:0.1
10        - name: name
11          value: s2i-java-custom
12        - name: kind
13          value: task

With the usage of resolvers we have moved away from managing pipeline tasks in a distributed and uncontrolled manner. However, up until this point the remainder of the pipeline resources (including webhooks, eventlisteners, triggers, templates and pipelines), required in order to automatically trigger the ci-pipeline in OpenShift, are still managed separately from the application-source repository, not to mention that they are required to be implemented and maintained adding an extra overhead.

Pipelines-as-code (Generally Available on OpenShift Pipelines) which allows to ship the CI/CD pipelines within the same git repository as the application, making it easier to keep both of them in sync in terms of release updates, is the feature we are now going to utilise to migrate the ci-pipeline to.

An important note here is that pipeline-as-code functionality relies on integration with supported Git Repository Providers and as the current setup utilises gitea, which is not a supported Pipeline as Code GIT Repository hosting provider, the application source code is placed in a separate github application-source repository.

Creation of a PipelineRun Resource

In order to achieve the migration

• the pipeline-as-code branch no longer contains the resources trigger-quarkus-app-push.yaml and pipeline-ci.yaml, and
• instead they are replaced with a PipelineRun resource under the .tekton directory.

The PipelineRun contains the following important configurations in order to materialise the pipeline and all the necessary functionality including tasks, triggers etc.

1. The definitions when the pipeline will be initiated inside OpenShift include:

• on a push or pull request [1]
• on the main branch (of the source code repository) [2]
• when the pom.xml has also been updated [3]

 1  apiVersion: tekton.dev/v1beta1
 2  kind: PipelineRun
 3  metadata:
 4    name: quarkus-app
 5    annotations:
 6      # The event we are targeting as seen from the webhook payload
 7      # this can be an array too, i.e: [pull_request, push]
 8      pipelinesascode.tekton.dev/on-event: "[pull_request, push]"
 9
10      # The branch or tag we are targeting (ie: main, refs/tags/*)
11      pipelinesascode.tekton.dev/on-target-branch: "[main]"
12
13      # Executes only for specific paths
14      pipelinesascode.tekton.dev/on-cel-expression: |
15                event == "push" && "pom.xml".pathChanged()   

2. References [4] to remote tasks via the Pipeline as Code resolver in order to retrieve the tasks either from git repository or the hub.

 1  # git-clone custom task
 2  pipelinesascode.tekton.dev/task: "https://github.com/skoussou/tekton-catalog/blob/main/ci-tasks/resources/git-clone-custom/0.1/git-clone.yaml"
 3
 4  # Use maven task from the hub to test our Java project
 5  pipelinesascode.tekton.dev/task-1: "maven"
 6   
 7  # additional custom tasks
 8  pipelinesascode.tekton.dev/task-2: "https://github.com/skoussou/tekton-catalog/blob/main/ci-tasks/resources/task-generate-version/0.1/task-generate-version.yaml"
 9  pipelinesascode.tekton.dev/task-3: "https://github.com/skoussou/tekton-catalog/blob/main/ci-tasks/resources/mock/0.1/task-mock.yaml"    
10  pipelinesascode.tekton.dev/task-4: "https://github.com/skoussou/tekton-catalog/blob/main/ci-tasks/resources/s2i-java-custom/0.1/s2i-java.yaml"    
11  pipelinesascode.tekton.dev/task-5: "https://github.com/skoussou/tekton-catalog/blob/main/ci-tasks/resources/task-tag-image/0.1/task-tag-image.yaml"    
12  pipelinesascode.tekton.dev/task-6: "https://github.com/skoussou/tekton-catalog/blob/main/ci-tasks/resources/task-create-pr-to-deploy/0.1/task-create-pr-to-deploy.yaml"       

3. The parameters which determine:

• dynamic expandable parameters which can be pulled from the commit
• or hardcoded [5] ones.

4. The pipelineSpec which determines the pipeline either as:

• a reference via the annotation pipelinesascode.tekton.dev/pipeline: "<https://git.provider/raw/pipeline.yaml>"
• or (as delivered here) a list of tasks [6]

Pipelines as Code integration with a GitHub App

GitHub Apps act as a point of integration with Red Hat OpenShift Pipelines and in order to achieve this a single GitHub App for all cluster users is configured. This can be performed both via tkn app CLI or manually, here the former has been selected.

In order to configure Pipelines as Code to access the newly created GitHub App the following is executed whilst logged into OpenShift and within the openshift-pipelines namespace. As a result a Secret resource pipelines-as-code-secret is created in openshift-pipelines enabling the integration.

Note: the Github App name must be unique.

 1$ oc project openshift-pipelines
 2$ tkn pac bootstrap
 3  => Checking if Pipelines as Code is installed.
 4  ✓ Pipelines as Code is already installed.
 5  ? Enter the name of your GitHub application:  test-ppln-as-code
 6  👀 I have detected an OpenShift Route on: https://pipelines-as-code-controller-openshift-pipelines.apps.<CLUSTER_NAME>.<DOMAIN_NAME>
 7  ? Do you want me to use it? Yes
 8  🌍 Starting a web browser on http://localhost:8080, click on the button to create your GitHub APP
 9  🔑 Secret pipelines-as-code-secret has been created in the openshift-pipelines namespace
10  🚀 You can now add your newly created application on your repository by going to this URL:
11          
12  https://github.com/apps/test-ppln-as-code
13          
14  💡 Don't forget to run the "tkn pac create repo" to create a new Repository CRD on your cluster.

The new Github App is associated with the source repository.

Finally the following command adds a webhook and a Repository CR automatically for the pipeline as code to process events from application-source repository and run the pipeline in sk-workshop-components namespace based on the Repository CR.

1$ tkn pac create repository
2  ? Enter the Git repository url (default: https://github.com/skoussou/tekton-gitops-workshop):  https://github.com/skoussou/application-source
3  ? Please enter the namespace where the pipeline should run (default: openshift-pipelines): sk-workshop-components
4  ✓ Repository skoussou-application-source has been created in sk-workshop-components namespace
5  ℹ Directory .tekton has been created.
6  ✓ A basic template has been created in .tekton/pipelinerun.yaml, feel free to customize it.
7  ℹ You can test your pipeline by pushing generated template to your git repository

Now the pipeline as code reacts to any push event and initiates a new PipelineRun (for more pac information read here).

The results of the PipelineRun are not only reported in the OpenShift console but additionally shared back on the github repo.

Conclusion

The efforts have showcased how as a team we can better organize both pipeline components such as tasks and pipelines in order to promote re-usability, versioning and maintenance. There are still some questions open on the use of pipeline as code in multi component applications however the ease of setting up the flow in comparison to the complexity of maintaining all tekton CRs to achieve the same effect is undoubtedly impressive.