Illumina Innovates with Rancher and Kubernetes
The following instructions will guide you through upgrading a high-availability Rancher server installation.
Review the Known Upgrade Issues and Caveats
Air Gap Installs Only: Collect and Populate Images for the new Rancher server version
Follow the guide to populate your private registry with the images for the Rancher version that you want to upgrade to.
During upgrade, you’ll enter a series of commands, filling placeholders with data from your environment. These placeholders are denoted with angled brackets and all capital letters (<EXAMPLE>).
<EXAMPLE>
Here’s an example of a command with a placeholder:
docker stop <RANCHER_CONTAINER_NAME>
In this command, <RANCHER_CONTAINER_NAME> is the name of your Rancher container.
<RANCHER_CONTAINER_NAME>
Cross reference the image and reference table below to learn how to obtain this placeholder data. Write down or copy this information before starting the upgrade.
Terminal docker ps Command, Displaying Where to Find <RANCHER_CONTAINER_TAG> and <RANCHER_CONTAINER_NAME>
docker ps
<RANCHER_CONTAINER_TAG>
v2.1.3
festive_mestorf
<RANCHER_VERSION>
<DATE>
2018-12-19
You can obtain <RANCHER_CONTAINER_TAG> and <RANCHER_CONTAINER_NAME> by logging into your Rancher Server by remote connection and entering the command to view the containers that are running: docker ps. You can also view containers that are stopped using a different command: docker ps -a. Use these commands for help anytime during while creating backups.
docker ps -a
During upgrade, you create a copy of the data from your current Rancher container and a backup in case something goes wrong. Then you deploy the new version of Rancher in a new container using your existing data. Follow the steps to upgrade Rancher server:
Using a remote Terminal connection, log into the node running your Rancher Server.
Stop the container currently running Rancher Server. Replace <RANCHER_CONTAINER_NAME> with the name of your Rancher container.
Use the command below, replacing each placeholder, to create a data container from the Rancher container that you just stopped.
docker create --volumes-from <RANCHER_CONTAINER_NAME> --name rancher-data rancher/rancher:<RANCHER_CONTAINER_TAG>
From the data container that you just created (rancher-data), create a backup tarball (rancher-data-backup-<RANCHER_VERSION>-<DATE>.tar.gz).
rancher-data
rancher-data-backup-<RANCHER_VERSION>-<DATE>.tar.gz
This tarball will serve as a rollback point if something goes wrong during upgrade. Use the following command, replacing each placeholder.
docker run --volumes-from rancher-data -v $PWD:/backup busybox tar zcvf /backup/rancher-data-backup-<RANCHER_VERSION>-<DATE>.tar.gz /var/lib/rancher
Step Result: When you enter this command, a series of commands should run.
Enter the ls command to confirm that the backup tarball was created. It will have a name similar to rancher-data-backup-<RANCHER_VERSION>-<DATE>.tar.gz.
ls
[rancher@ip-10-0-0-50 ~]$ ls rancher-data-backup-v2.1.3-20181219.tar.gz
Move your backup tarball to a safe location external from your Rancher Server.
Pull the image of the Rancher version that you want to upgrade to.
<RANCHER_VERSION_TAG>
docker pull rancher/rancher:<RANCHER_VERSION_TAG>
Start a new Rancher server container using the data from the rancher-data container. Remember to pass in all the environment variables that you had used when you started the original container.
Note: After upgrading Rancher Server, data from your upgraded server is now saved to the rancher-data container for use in future upgrades. Important: Do not stop the upgrade after initiating it, even if the upgrade process seems longer than expected. Stopping the upgrade may result in database migration errors during future upgrades. Did you… Use a proxy? See HTTP Proxy Configuration Configure custom CA root certificate to access your services? See Custom CA root certificate Record all transactions with the Rancher API? See API Auditing
Note: After upgrading Rancher Server, data from your upgraded server is now saved to the rancher-data container for use in future upgrades.
Important: Do not stop the upgrade after initiating it, even if the upgrade process seems longer than expected. Stopping the upgrade may result in database migration errors during future upgrades.
Did you…
Choose from the following options:
Select which option you had installed Rancher server
If you have selected to use the Rancher generated self-signed certificate, you add the –volumes-from rancher-data to the command that you had started your original Rancher server container.
–volumes-from rancher-data
docker run -d --volumes-from rancher-data \ --restart=unless-stopped \ -p 80:80 -p 443:443 \ rancher/rancher:<RANCHER_VERSION_TAG>
If you have selected to bring your own self-signed certificate, you add the –volumes-from rancher-data to the command that you had started your original Rancher server container and need to have access to the same certificate that you had originally installed with.
Reminder of the Cert Prerequisite: The certificate files must be in PEM format. In your certificate file, include all intermediate certificates in the chain. Order your certificates with your certificate first, followed by the intermediates. For an example, see SSL FAQ / Troubleshooting.
<CERT_DIRECTORY>
<FULL_CHAIN.pem>
<PRIVATE_KEY.pem>
<CA_CERTS>
docker run -d --volumes-from rancher-data \ --restart=unless-stopped \ -p 80:80 -p 443:443 \ -v /<CERT_DIRECTORY>/<FULL_CHAIN.pem>:/etc/rancher/ssl/cert.pem \ -v /<CERT_DIRECTORY>/<PRIVATE_KEY.pem>:/etc/rancher/ssl/key.pem \ -v /<CERT_DIRECTORY>/<CA_CERTS.pem>:/etc/rancher/ssl/cacerts.pem \ rancher/rancher:<RANCHER_VERSION_TAG>
If you have selected to use a certificate signed by a recognized CA, you add the –volumes-from rancher-data to the command that you had started your original Rancher server container and need to have access to the same certificates that you had originally installed with. Remember to include –no-cacerts as an argument to the container to disable the default CA certificate generated by Rancher.
–no-cacerts
Reminder of the Cert Prerequisite: The certificate files must be in PEM format. In your certificate file, include all intermediate certificates provided by the recognized CA. Order your certificates with your certificate first, followed by the intermediates. For an example, see SSL FAQ / Troubleshooting.
docker run -d --volumes-from rancher-data \ --restart=unless-stopped \ -p 80:80 -p 443:443 \ -v /<CERT_DIRECTORY>/<FULL_CHAIN.pem>:/etc/rancher/ssl/cert.pem \ -v /<CERT_DIRECTORY>/<PRIVATE_KEY.pem>:/etc/rancher/ssl/key.pem \ rancher/rancher:<RANCHER_VERSION_TAG> \ --no-cacerts
Remember: Let’s Encrypt provides rate limits for requesting new certificates. Therefore, limit how often you create or destroy the container. For more information, see Let’s Encrypt documentation on rate limits.
If you have selected to use Let’s Encrypt certificates, you add the --volumes-from rancher-data to the command that you had started your original Rancher server container and need to provide the domain that you had used when you originally installed Rancher.
--volumes-from rancher-data
Reminder of the Cert Prerequisites: Create a record in your DNS that binds your Linux host IP address to the hostname that you want to use for Rancher access (rancher.mydomain.com for example). Open port TCP/80 on your Linux host. The Let’s Encrypt http-01 challenge can come from any source IP address, so port TCP/80 must be open to all IP addresses.
Reminder of the Cert Prerequisites:
rancher.mydomain.com
TCP/80
<YOUR.DNS.NAME>
docker run -d --volumes-from rancher-data \ --restart=unless-stopped \ -p 80:80 -p 443:443 \ rancher/rancher:<RANCHER_VERSION_TAG> \ --acme-domain <YOUR.DNS.NAME>
For security purposes, SSL (Secure Sockets Layer) is required when using Rancher. SSL secures all Rancher network communication, like when you login or interact with a cluster.
Did you… Configure custom CA root certificate to access your services? See Custom CA root certificate. Record all transactions with the Rancher API? See API Auditing.
system-charts
<REGISTRY.YOURDOMAIN.COM:PORT>
docker run -d --volumes-from rancher-data \ --restart=unless-stopped \ -p 80:80 -p 443:443 \ -e CATTLE_SYSTEM_DEFAULT_REGISTRY=<REGISTRY.YOURDOMAIN.COM:PORT> \ # Set a default private registry to be used in Rancher -e CATTLE_SYSTEM_CATALOG=bundled \ #Available as of v2.3.0, use the packaged Rancher system charts <REGISTRY.YOURDOMAIN.COM:PORT>/rancher/rancher:<RANCHER_VERSION_TAG>
Reminder of the Prerequisite: The certificate files must be in PEM format. In your certificate file, include all intermediate certificates in the chain. Order your certificates with your certificate first, followed by the intermediates. For an example, see SSL FAQ / Troubleshooting.
docker run -d --restart=unless-stopped \ -p 80:80 -p 443:443 \ -v /<CERT_DIRECTORY>/<FULL_CHAIN.pem>:/etc/rancher/ssl/cert.pem \ -v /<CERT_DIRECTORY>/<PRIVATE_KEY.pem>:/etc/rancher/ssl/key.pem \ -v /<CERT_DIRECTORY>/<CA_CERTS.pem>:/etc/rancher/ssl/cacerts.pem \ -e CATTLE_SYSTEM_DEFAULT_REGISTRY=<REGISTRY.YOURDOMAIN.COM:PORT> \ # Set a default private registry to be used in Rancher -e CATTLE_SYSTEM_CATALOG=bundled \ #Available as of v2.3.0, use the packaged Rancher system charts <REGISTRY.YOURDOMAIN.COM:PORT>/rancher/rancher:<RANCHER_VERSION_TAG>
If you have selected to use a certificate signed by a recognized CA, you add the –volumes-from rancher-data to the command that you had started your original Rancher server container and need to have access to the same certificates that you had originally installed with.
Reminder of the Prerequisite: The certificate files must be in PEM format. In your certificate file, include all intermediate certificates provided by the recognized CA. Order your certificates with your certificate first, followed by the intermediates. For an example, see SSL FAQ / Troubleshooting.
Note: Use the --no-cacerts as argument to the container to disable the default CA certificate generated by Rancher.
--no-cacerts
docker run -d --volumes-from rancher-data \ --restart=unless-stopped \ -p 80:80 -p 443:443 \ --no-cacerts \ -v /<CERT_DIRECTORY>/<FULL_CHAIN.pem>:/etc/rancher/ssl/cert.pem \ -v /<CERT_DIRECTORY>/<PRIVATE_KEY.pem>:/etc/rancher/ssl/key.pem \ -e CATTLE_SYSTEM_DEFAULT_REGISTRY=<REGISTRY.YOURDOMAIN.COM:PORT> \ # Set a default private registry to be used in Rancher -e CATTLE_SYSTEM_CATALOG=bundled \ #Available as of v2.3.0, use the packaged Rancher system charts <REGISTRY.YOURDOMAIN.COM:PORT>/rancher/rancher:<RANCHER_VERSION_TAG>
Log into Rancher. Confirm that the upgrade succeeded by checking the version displayed in the bottom-left corner of the browser window.
Having network issues in your user clusters following upgrade? See Restoring Cluster Networking.
Having network issues in your user clusters following upgrade?
See Restoring Cluster Networking.
Remove the previous Rancher Server container. If you only stop the previous Rancher Server container (and don’t remove it), the container may restart after the next server reboot.
If your upgrade does not complete successfully, you can roll back Rancher server and its data back to its last healthy state. For more information, see Single Node Rollback.