Illumina Innovates with Rancher and Kubernetes
Rancher uses cert-manager to automatically generate and renew TLS certificates for HA deployments of Rancher. As of Fall 2019, two important changes to cert-manager are set to occur that you need to take action on if you have an HA deployment of Rancher:
To address these changes, this guide will do two things:
Note: The namespace used in these instructions depends on the namespace cert-manager is currently installed in. If it is in kube-system use that in the instructions below. You can verify by running kubectl get pods --all-namespaces and checking which namespace the cert-manager-* pods are listed in. Do not change the namespace cert-manager is running in or this can cause issues.
kubectl get pods --all-namespaces
In order to upgrade cert-manager, follow these instructions:
Back up existing resources as a precaution
kubectl get -o yaml --all-namespaces issuer,clusterissuer,certificates > cert-manager-backup.yaml
Delete the existing deployment
helm delete --purge cert-manager
Install the CustomResourceDefinition resources separately
kubectl apply -f https://raw.githubusercontent.com/jetstack/cert-manager/release-0.9/deploy/manifests/00-crds.yaml
Label the kube-system namespace to disable resource validation
kubectl label namespace kube-system certmanager.k8s.io/disable-validation=true
Add the Jetstack Helm repository
helm repo add jetstack https://charts.jetstack.io
Update your local Helm chart repository cache
helm repo update
Install the new version of cert-manager
helm install --version 0.9.1 --name cert-manager --namespace kube-system jetstack/cert-manager
Before you can perform the upgrade, you must prepare your air gapped environment by adding the necessary container images to your private registry and downloading or rendering the required Kubernetes manifest files.
Follow the guide to Prepare your Private Registry with the images needed for the upgrade.
From a system connected to the internet, add the cert-manager repo to Helm
helm repo add jetstack https://charts.jetstack.io helm repo update
Fetch the latest cert-manager chart available from the Helm chart repository.
helm fetch jetstack/cert-manager --version v0.9.1
Render the cert manager template with the options you would like to use to install the chart. Remember to set the image.repository option to pull the image from your private registry. This will create a cert-manager directory with the Kubernetes manifest files.
image.repository
cert-manager
helm template ./cert-manager-v0.9.1.tgz --output-dir . \ --name cert-manager --namespace kube-system \ --set image.repository=<REGISTRY.YOURDOMAIN.COM:PORT>/quay.io/jetstack/cert-manager-controller --set webhook.image.repository=<REGISTRY.YOURDOMAIN.COM:PORT>/quay.io/jetstack/cert-manager-webhook --set cainjector.image.repository=<REGISTRY.YOURDOMAIN.COM:PORT>/quay.io/jetstack/cert-manager-cainjector
Download the required CRD file for cert-manager
curl -L -o cert-manager/cert-manager-crd.yaml https://raw.githubusercontent.com/jetstack/cert-manager/release-0.9/deploy/manifests/00-crds.yaml
Delete the existing cert-manager installation
kubectl -n kube-system delete deployment,sa,clusterrole,clusterrolebinding -l 'app=cert-manager' -l 'chart=cert-manager-v0.5.2'
kubectl apply -f cert-manager/cert-manager-crd.yaml
Install cert-manager
kubectl -n kube-system apply -R -f ./cert-manager
Once you’ve installed cert-manager, you can verify it is deployed correctly by checking the kube-system namespace for running pods:
kubectl get pods --namespace kube-system NAME READY STATUS RESTARTS AGE cert-manager-7cbdc48784-rpgnt 1/1 Running 0 3m cert-manager-webhook-5b5dd6999-kst4x 1/1 Running 0 3m cert-manager-cainjector-3ba5cd2bcd-de332x 1/1 Running 0 3m
If the ‘webhook’ pod (2nd line) is in a ContainerCreating state, it may still be waiting for the Secret to be mounted into the pod. Wait a couple of minutes for this to happen but if you experience problems, please check cert-manager’s troubleshooting guide.
Note: The above instructions ask you to add the disable-validation label to the kube-system namespace. Here are additional resources that explain why this is necessary: Information on the disable-validation label Information on webhook validation for certificates
Note: The above instructions ask you to add the disable-validation label to the kube-system namespace. Here are additional resources that explain why this is necessary:
Cert-manager has deprecated the use of the certificate.spec.acme.solvers field and will drop support for it completely in an upcoming release.
certificate.spec.acme.solvers
Per the cert-manager documentation, a new format for configuring ACME certificate resources was introduced in v0.8. Specifically, the challenge solver configuration field was moved. Both the old format and new are supported as of v0.9, but support for the old format will be dropped in an upcoming release of cert-manager. The cert-manager documentation strongly recommends that after upgrading you update your ACME Issuer and Certificate resources to the new format.
Details about the change and migration instructions can be found in the cert-manager v0.7 to v0.8 upgrade instructions.