Helmfile vs Flux

First, I need to say that I'm not really good at Kustomize, and maybe everything can be done better.

Let's say we need to install Bitnami Postgres to 2 clusters using helm charts.

• Passwords should be stored in git encrypted with SOPS
• One database needs to be 10Gb in size, and another just 1Gb

With flux we're creating two manifests:

---
apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: HelmRepository
metadata:
  name: bitnami
  namespace: flux-system
spec:
  interval: 1m0s
  type: default
  url: https://charts.bitnami.com/bitnami
---
apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
  creationTimestamp: null
  name: postgresql
  namespace: flux-system
spec:
  chart:
    spec:
      chart: postgresql
      sourceRef:
        kind: HelmRepository
        name: bitnami
        namespace: flux-system
      version: 13.3.1
  install:
    crds: Create
    createNamespace: true
  interval: 1m0s
  releaseName: postgresql
  targetNamespace: database

With helmfile it's going to be this:

repositories:
  - name: bitnami
    url: https://charts.bitnami.com/bitnami

releases:
  - name: postgresql
    chart: bitnami/postgresql
    version: 13.3.1
    namespace: database

So it's 30 lines of code against 8.

Then we need to add values. With flux we would add something like that to the manifest.

  valuesFrom:
  - kind: ConfigMap
    name: database-postgresql-values.postgresql.yaml
    valuesKey: database-postgresql-values.postgresql.yaml
  - kind: ConfigMap
    name: database-postgresql-values.spec.postgresql.yaml
    valuesKey: database-postgresql-values.spec.postgresql.yaml

Spec means special in that case

And we need to create 2 configmaps (or it can be just one, cause I wanted to reuse common values, but I think kustomize won't let me use the folder that is not in that kustomize context, so I'm not sure how)

And it's either 4 or 7 lines of code in addition

apiVersion: v1
data:
  database-postgresql-values.spec.postgresql.yaml: |
    persistence:
      size: 1Gi    
kind: ConfigMap
metadata:
  creationTimestamp: null
  name: database-postgresql-values.spec.postgresql.yaml
  namespace: flux-system
---
apiVersion: v1
data:
  database-postgresql-values.postgresql.yaml: |
    architecture: standalone

    auth:
      database: postgres

    metrics:
      enabled: false    
kind: ConfigMap
metadata:
  creationTimestamp: null
  name: database-postgresql-values.postgresql.yaml
  namespace: flux-system

In case it's just one configmap, it would be something like:

apiVersion: v1
data:
  database-postgresql-values.postgresql.yaml: |
    persistence:
      size: 1Gi
    architecture: standalone
    auth:
      database: postgres
    metrics:
      enabled: false    
kind: ConfigMap
metadata:
  creationTimestamp: null
  name: database-postgresql-values.postgresql.yaml
  namespace: flux-system

So it's either about 25 or 15 lines of code. Also, it's not just you go to the chart's values and creates a values file, you need to create a whole configmap, so you either need to remember the whole CM spec,cmds to create it, or you need to find a reference somewhere to copy-paste.

Also, since it's just static manifests, names of configmaps and keys are not set once and reused. So it's error-prone configuration, especially when you can't just debug it, and you need to deploy it to spot a problem. Once you update a configmap name/key, you need not to forget to update the Release manifest.

With helmfile it's going to be something like that.

Add it to the helmfile:

    values:
      - ./values/common/values.postgresql.yaml
      - ./values/cluster-1/values.postgresql.yaml

And then values

---
architecture: standalone

auth:
  database: postgres

metrics:
  enabled: false
---
persistence:
  size: 1Gi

So, additional ~10 lines of code.

And the secret thingy

I'm pretty syre it can be done better, since I'm not good at Kustomize, so it might be a bad point.

How am I doing that?

I'm creating a Kubernetes secret where data is encrypted with Sops. To be honest I encrypt the whole secret, but it's just because I've screwed up the sops config part. But the idea is that there is a k8s secret with encrypted data. I won't count lines here, because probably it should be less that I have, approximately it's about 10 lines of code, because it's the whole manifest. And it has the same problem ConfigMap has, because it's a static manifest. but with my approach it's also required to have a secret-generator, that is decrypting secrets with the ksops plugin

---
apiVersion: viaduct.ai/v1
kind: ksops
metadata:
  name: shoebill-secret-gen
files:
  - src/secrets/database-postgresql-secrets.postgresql.yaml

And it's 6 additional lines + a system dependency (ksops). But I've been told that I don't have to do it this way, so it might be not required at all.

Helmfile:

    secrets:
      - ./values/{{ .Environment.Name}}/secrets.postgresql.yaml

So 2 lines are added to the helmfile, and a secret, that in unencrypted state is 4 lines of code

And already at this step it about 65 vs 24. for one chart in one cluster. With the helmfile you don't need to add k8s objects boilerplate around values, you just use them as they are defined in a chart, you don't need to use k8s object boilerplate code around release and repo. And you don't need to check names of secret/configmaps/keys, so they correspond in two static objects.

Now, let's add a second cluster:

Maybe if you use Kustomize with variables, you can avoid it, so I'm not sure

But probably, it would be just 1-to-1 copypaste of the first cluster with one line changed, so I'd expect to have 130 lines of code. And with helmfile I'd do it this way

---
environments:
  cluster-1:
  cluster-2:
---
repositories:
  - name: bitnami
    url: https://charts.bitnami.com/bitnami

releases:
  - name: postgresql
    chart: bitnami/postgresql
    version: 13.3.1
    namespace: database
    values:
      - ./values/common/values.postgresql.yaml
      - ./values/{{ .Environment.Name }}/values.postgresql.yaml
    secrets:
      - ./values/{{ .Environment.Name}}/secrets.postgresql.yaml

So it's just about 4 lines of code added to the helmfile, and 3 added as new values, so 31 in total.

Then if you need to change something that is common for there tho clusters (helmchart version, or common values), with helmfile you'll need to update it once, and with flux you'll have to do it as many times, as you have clusters, since it's going to be copy-paste, it's boring and hence error-prone task.