Why I Learned the Gitlab Runner Helm Chart in 45 Minutes

Many people may be thinking, "it took you 45 minutes to figure out how to deploy your own Gitlab runners?".  To which I reply, "yes, but hear me out".

Circumstances were such that I had some funky access on one of our program's Gitlab instances.  I was a member of a group in which much of my contributions were focused.  This group was bloated in its membership count and lacking in thoughtful permissions (like most all groups on this Gitlab).  However the ace up my sleeve was a group given to me months ago that must have been forgotten because I was the owner and membership was tight.

I hate the idea of sharing runners for pipelines I must verify security of all operations for my repo.  After all, it took me two days to figure out how to get my runner to commit and push the results of the pipeline back to the merge request it was running against.  Although that problem set is a different story entirely, the fundamental problem we are discussing now is trust, or rather zero trust.  I did not deploy the runners that were executing my pipeline, and the implications of a compromised runner could be catastrophic.  I felt something must be done.  So I decided to do something.

The first order of business was to get a Kubernetes cluster up in a secure environment solely under my control.  This was achieved by deploying this cluster in a privileged environment that most of the organization had no access to, thereby ensuring the persons of concern could not reach this cluster to protest my unilateral decision to take control over my pipelines.  

With my beastly cluster in hand, it was time to do my deep dive into "how do I deploy Gitlab Runners".  It was about 15 minutes of scanning my Google query results until I found the helm chart I was after.

Using my God mode permissions in my isolated Gitlab group I began deploying runner after runner until I had substantial compute to handle the load of my various pipelines and ensured that no repo other than the ones I intended could execute pipeline code using these newly deployed runners on my newly deployed cluster.  The procedure I followed to get this win was as follows:

helm repo add gitlab https://charts.gitlab.io
mkdir -m ~/charts/gitlab/runners && cd ~/charts/gitlab/runners
helm fetch gitlab/gitlab-runner
tar -xf gitlab* && cd gitlab-*

Once inside the gitlab-runner chart directory I was lazy.  I did not create a custom values.yaml file for each runner I deployed, but simply modified values.yaml on every helm install.  For my needs this was fine.  

You will need to install Helm v3 to execute the following but you should have had that before reading this far.  However, for completeness, if you do not already have helm installed, you can install it with the following:

curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3
chmod +x get_helm.sh
./get_helm.sh # you will be prompted for your system password...enter it

Yay, now you have helm and you should still be in that chart directory.  Unless you're a slob, you will need to remove that install script from your working directory by executing rm get_helm.sh.

Back to my story since that is what I want to tell you.  I had my new kubeconfig at ~/.kube/config to tell helm where to deploy these and I was in my gitlab-runners chart directory.  There were values still to be changed.

I like VSCode, so that's what I used.  In my editor I opened the values.yaml file and made some simple modifications.  I entered the URL of the Gitlab instance I was targeting, and my Gitlab group's registration token.  I also named each runner a unique name.  I uncommented the resource limits for the pods I was going to deploy and doubled their CPU and memory values.  After that was finished I switched back to my terminal, where I was still in the runner's chart directory.

Now the easy part.  To deploy a helm chart from a fetched chart one must do nothing special other than specify the root directory of the chart during the helm install command.  In this case my command looked like this:

helm install $RUNNER_NAME .

$RUNNER_NAME is the same unique name I set in the values.yaml file.  Within seconds I had my first runner registered to my group.  A runner of my own, not to be shared with any untrusted user on the Gitlab instance I was operating within.  

One runner is never enough so I repeated this procedure 9 more times, ensuring I had ample resources to do what I needed to do.

That's all folks.  Helm is easy, runners aren't rocket science, but you really should consider what runners do and who is using them.  Who controls them is perhaps the most important consideration, and when it comes to securing your source code, my advice is to trust no one but yourself.

Whether this has been helpful or not, is irrelevant, the upshot here is think about your security posture, and change what you can to improve it, always.