Skip to content

Latest commit

 

History

History
369 lines (262 loc) · 11.7 KB

readme.md

File metadata and controls

369 lines (262 loc) · 11.7 KB

Lemmings

Kubernetes cluster configuration that uses GitOps to keep state.

Includes Flux, Helm, cert-manager, Nginx Ingress Controller and Sealed Secrets.

Contributors

Versions

  • 2020-02-13 Flux 1.1, fluxcd.io annotations, and Helm 3
  • 2019-11-07 Flux 0.16, flux.weave.works annotations and Helm 2

Warning

This Lemmings guide has since 2021 been superseded by a Flux v2 based guide, the Double Dragon. Please follow it for a more up to date cluster management with gitops.

Pre requisite

Kubernetes tools and cluster

  • brew install kubectl

  • Create Kubernetes cluster (see Kubernetes as a Service providers below)

  • Set up Kubernetes context (see provider CLIs and kubectx below)

  • Test cluster connection:

    kubectl cluster-info

Lemmings install

Fork/Clone repository

  • brew install hub;
hub clone flurdy/lemmings my-lemmings;
cd my-lemmings;
hub create -p my-lemmings;
hub remote add upstream flurdy/lemmings;
git push -u origin master

  • Replace my-lemmings with whatever you want to call your cluster

  • Flux can also talk to Bitbucket, Gitlab and self-hosted repos

Install Helm

  • Helm released v3 in November 2019.
  • If you have other clusters using Helm 2 tread carefully to make sure you can support both v2 and v3.

Helm 3

  brew install helm@3

  • Vanilla Helm 3 includes no chart repositories. Lets add the core one

    helm repo add stable \
    https:// kubernetes-charts.storage.googleapis.com/

Helm 2

  • If you need to use Helm 2, refer to an earlier version of Lemmings more aimed at Helm 2 + Tiller
  • Note you may need to specify specific flux.weave.works annotations and HelmRelease versions to keep it working as things start to assume Helm 3

Configure cert-manager

  • Change email address in:
    • issuer/issuer-staging.yml
    • issuer/issuer-prod.yml
  • git add -p issuer;
git commit -m "Updated email in certificate issuers";
git push

Install Flux

helm repo add fluxcd https://charts.fluxcd.io;
helm repo update;
kubectl apply -f flux/crd-flux-helm.yml;
kubectl create namespace flux;
helm upgrade -i flux \
   --namespace=flux \
   --set helm.versions=v3 \
   --set git.url=git@github.com:YOURUSERNAME/my-lemmings \
   fluxcd/flux
  • Replace YOURUSERNAME/me-lemmings with your username and repository.

Install Flux Helm Operator

helm upgrade -i helm-operator \
   --namespace=flux \
   --set git.ssh.secretName=flux-git-deploy \
   --set helm.versions=v3 \
   fluxcd/helm-operator

Fluxctl & SSH

  • Install fluxctl and generate SSH key

    brew install fluxctl;
export FLUX_FORWARD_NAMESPACE=flux;
fluxctl identity --k8s-fwd-ns flux

  • Add Flux's SSH key to your github repo with write access:

  • Tail log:

    kubectl logs -n flux deploy/flux -f

Your GitOps based Kubernetes cluster is live!

If you waited a few minutes then your GitOps configured Kubernetes cluster is now live.

Your first application

  • (We baked one earlier for you: workflows/hello)

  • View/Edit workflows/hello/deployment.yml

    apiVersion: apps/v1
kind: Deployment
metadata:
  name: hello-deployment
  annotations:
    fluxcd.io/automated: "true"
spec:
  selector:
    matchLabels:
      app: hello
  replicas: 2
  template:
    metadata:
      labels:
        app: hello
    spec:
      containers:
      - name: hello-container
        image: nginxdemos/hello:0.2
        ports:
        - containerPort: 80
    • Note the fluxcd.io/automated annotation which tells Flux to upgrade it when possible.

  • Edit workflows/hello/service.yml

    apiVersion: v1
kind: Service
metadata:
  name: hello-service
spec:
  selector:
     app: hello
  ports:
  - protocol: TCP
     port: 80
     targetPort: 80

  • Edit workflows/hello/ingress.yml

    apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: hello-ingress
  annotations:
    kubernetes.io/ingress.class: nginx
#    cert-manager.io/cluster-issuer: letsencrypt-staging
spec:
#  tls:
#  - hosts:
#    - hello.example.com
#    secretName: letsencrypt-certificate-staging
  rules:
  - host: hello.example.com
    http:
      paths:
      - backend:
          serviceName: hello-service
          servicePort: 80

    The SSL certificate part is commented out for now as it wont work without a real domain that Letsencrypt can do a reverse verification call to.

Launch application

  • If you made any changes:

    git add workflows/hello
git commit -m "App: Hello"
git push

  • Wait (max 5 minutes) for Flux to detect your changes and apply them

  • Or if impatient:

    fluxctl sync

  • Check if the Hello cluster resources are running

    kubectl get deployment hello-deployment
kubectl get service hello-service
kubectl get ingress hello-ingress
kubectl get pods -l app=hello

Test application

  • Find the ingress controller's External IP

    kubectl get services nginx-ingress-controller

  • Use curl to resolve the URL. Replace 11.22.33.44 with the external IP.

  • And lynx to view it

    curl -H "Host: hello.example.com" \
  --resolve hello.example.com:80:11.22.33.44 \
  --resolve hello.example.com:443:11.22.33.44 \
  http://hello.example.com | lynx -stdin

  • This should show a basic hello world page, with an Nginx logo and some server address, name and date details.

Update application

  • Now if ever hub.docker.com/r/nginxdemos/hello bumps its version to higher than 0.2, Flux will detect it and bump your version in Kubernetes, and also commit the change back to your Git repository.
  • Unfortunately that image has not been updated for 2 years and unlikely to be updated, but when you start to use your own images they will be automagically updated if the deployment's annotation fluxcd.io/automated is set to true.

Delete application

git rm -r workflows/hello
git commit -m "Removed app Hello"
git push
fluxctl sync
kube delete ingress hello-ingress
kube delete service hello-service
kube delete deployment hello-deployment

Sealed Secrets

  • If you need secrets, (for private docker registry authentication, database passwords, API tokens), DO NOT commit them in plain text to the Git repository.

  • Instead we will use Sealed Secrets to encrypt them.

  • Check if Sealed Secrets was installed by Flux:

    kube get services sealed-secrets -n kube-system

  • Install CLI

    brew install kubeseal

  • Fetch the cluster's Sealed Secrets' public key

    kubeseal --fetch-cert \
   --controller-namespace=flux \
   --controller-name=sealed-secrets \
   > sealed-secrets/sealed-secrets-cert.pem
git add sealed-secrets/sealed-secrets-cert.pem
git commit -m "Sealed secret public key"

  • Create secrets but do not apply them to the cluster.

    • E.g with the --dry-run argument.

      mkdir secrets;
kubectl create secret generic basic-auth \
   --from-literal=user=admin \
   --from-literal=password=admin \
   --dry-run \
   -o json > secrets/basic-auth.json

  • Encrypt secret and transform to a sealed secret:

    kubeseal --format=yaml \
   --cert=sealed-secrets/sealed-secrets-cert.pem \
   < secrets/basic-auth.json \
   > secrets/secret-basic-auth.yml
git add secrets/secret-basic-auth.yml
rm secrets/basic-auth.json
git commit -m "Sealed basic auth secret"
git push

  • After a little time you should see the secret in your cluster

    kubectl describe secret basic-auth

  • Afterwards if you no longer need it, delete the file and secret:

    git rm secrets/secret-basic-auth.yml
git commit -m "Removed basic auth"
git push
fluxctl sync
kubectl delete secret basic-auth
kubectl delete SealedSecret basic-auth

  • Note: Step by step Docker Registry Cookbook

Go wild

  • Add/update your deployments, services, charts, docker registries, secrets, etc

Don't touch

  • Once Flux is running, by convention avoid using kubectl create|apply etc.
  • Nearly all changes should be via Git and Flux.
  • Any kubectl interaction should be read only.
  • Flux will not remove some resources, so you might need kubectl delete occasionally.

More information, alternatives, suggestions

Notes:

  • Certain operation takes a few minutes, e.g. pod creation.
  • cert-manager 0.11+ requires kubernetes 1.15+
  • Client tools are also available on Linux, Windows and more.