Upgrade Supercluster

This section describes how to install a newer software version on PCEs in a Supercluster.

Before Upgrading

Before you upgrade the Supercluster, perform the following:

Back up the PCE.

Before the upgrade, back up the leader and all member databases and each PCE's runtime_env.yml file. See Back Up Superclusterfor information.
Ensure all PCEs are in a healthy state.

Before upgrading, make sure all PCEs in the entire Supercluster are in a healthy state. In the PCE web console, check the PCE Health page to make sure the PCE health status is Normal.

Types of Supercluster Upgrade

You can choose to perform a simple upgrade or a rolling upgrade.

Supercluster simple upgrade: The Supercluster simple upgrade procedure requires you to set all the PCEs in the Supercluster to runlevel 1 for the duration of the upgrade. During a simple upgrade, the Supercluster is not fully operational. See Supercluster Simple Upgrade.
Supercluster rolling upgrade: Rolling upgrade keeps the Supercluster operational while individual PCEs are upgraded one at a time. With a rolling upgrade, the Supercluster continues to operate. See Supercluster Rolling Upgrade.

NOTE:

Supercluster rolling upgrade is supported only for a hotfix or a maintenance release. The major and minor release numbers in the installed and upgrade versions must match. For example, you can do a rolling upgrade from 21.2.0 to 21.2.1.

Supercluster Simple Upgrade

A Supercluster simple upgrade follows these general steps:

On all PCEs, quiesce the data replication.
Upgrade the software on all nodes of all PCEs.
Migrate the database on all PCEs.
Bring all PCEs back to runlevel 5.

Steps for Upgrade

Quiesce data replication.
1. On any node in the PCE cluster, bring all PCEs to runlevel 2:
```
$ sudo -u ilo-pce illumio-pce-ctl set-runlevel 2
```
2. In the PCE clusters, repeat step (a) for all leaders and all members.
  
  The cluster status should be RUNNING.
3. On any node in all PCE clusters, verify that the set-runlevel command finished and the cluster status is RUNNING:
```
$ sudo -u ilo-pce illumio-pce-ctl cluster-status -w
```
  Do not proceed to the next step until the set-runlevel command finishes.
4. Quiesce database replication.
  
  On any node, run the following command. Repeat this command on every PCE.
```
$ sudo -u ilo-pce illumio-pce-db-management supercluster-quiesce timeout_in_seconds
```
  This command waits for data replication to finish, which can take some time. To set a time limit, use timeout_in_seconds (default: 600). If the command doesn't complete within this time, it will stop. You must then run the command again.
  
  Expected output when database replication is successfully quiesced:
```
Replication is complete.
```
Upgrade the software.

Because this is a simple upgrade, you upgrade the software on all nodes of all PCEs in parallel.
1. On any node, stop the PCE cluster:
```
$ sudo -u ilo-pce illumio-pce-ctl cluster-stop
```
2. The packages to install depend on the type of PCE node:
  - Core nodes: Two packages, the PCE RPM and UI RPM.
  - Data nodes: One package, the PCE RPM.
  On each core node in the cluster, log in as root and install the PCE RPM and UI RPM. Be sure to specify both of the RPM file names on the command line:
```
$ rpm -Uvh illumio_pce_rpm illumio_ui_rpm
```
  For illumio_pce_rpm and illumio_ui_rpm, substitute the paths and filenames of the two RPM files you downloaded from the Illumio Support portal.
3. On each data node in the cluster, log in as root and install the PCE RPM:
```
$ rpm -Uvh <illumio_pce_rpm>
```
  For illumio_pce_rpm, substitute the path and filename of the software you downloaded from the Illumio Support portal.
4. On any node, start each cluster at runlevel 1:
```
$ sudo -u ilo-pce illumio-pce-ctl start --runlevel 1
```
Update the runtime environment file (runtime_env.yml). See What's New and Changed in This Release to determine whether any changes to runtime_env.yml are required to upgrade. If changes are required:
1. On all nodes in the cluster, update the runtime_env.yml file.
2. On all nodes in the cluster, check the validity of the runtime_env.yml file:
```
$ sudo -u ilo-pce illumio-pce-ctl check-env
```
  If any issues are reported by this command, correct them before moving on to the next step.
Migrate the PCE database.
1. On any node of every upgraded PCE, run the following command:
```
$ sudo -u ilo-pce illumio-pce-db-management migrate --upgrade-type simple
```
2. The migration might take some time to complete. Check the progress with the following command:
```
$ sudo -u ilo-pce illumio-pce-db-management supercluster-upgrade-status
```
3. On any node in the first PCE cluster, bring all PCEs to runlevel 2. Repeat this step on all the other PCEs.
```
$ sudo -u ilo-pce illumio-pce-ctl set-runlevel 2
```
4. For all leader and member PCE clusters, repeat step b. Verify that all PCEs in the Supercluster are at runlevel 2.
5. Wait until agent_slony_service and login_slony_service are up and running. These service names appear in bright blue or may have a pound character (#) appended, depending on which color option was chosen when starting the PCE, --color or --no-color. Do not restart the PCE. This step could take some time, depending on how recently you upgraded the PCE software. Run the following command to monitor the progress:
```
$ sudo -u ilo-pce illumio-pce-ctl cluster-status -w
```
  Issue the command again, when needed, until the services are ready
Bring PCEs back to operational status.
1. On any node for each PCE, set the runlevel to 5 :
```
$ sudo -u ilo-pce illumio-pce-ctl set-runlevel 5
```
  Setting the runlevel can take time to complete.
2. On any node in all PCE clusters, verify that the set-runlevel command finished and the cluster status is RUNNING:
```
$ sudo -u ilo-pce illumio-pce-ctl cluster-status -w
```
  NOTE:
  Due to the time it takes to replicate new database tables across all the PCEs, the upgrade might take longer than usual. The delay occurs when you bring the PCE to runlevel 2 or 5 from runlevel 1 after upgrading the software. The wait time depends on the number of new tables that are part of the upgrade. The wait might be up to 20 minutes.
3. Verify that you can log into the PCE web console on each PCE in the Supercluster.

Upgrade is complete.

NOTE:

In rare cases, you might receive an error when attempting to log into the PCE web console. When this happens, run the following command on all nodes, and try logging in again:

$ sudo -u ilo-pce illumio-pce-ctl restart

Supercluster Rolling Upgrade

In a rolling upgrade, the PCEs are upgraded one by one. The PCE that is being upgraded is at runlevel 1, while all the other PCEs are fully operational (runlevel 5).

NOTE:

Supercluster rolling upgrade is supported only for a hotfix or a maintenance release. The major and minor release numbers in the installed and upgrade versions must match. Also, due to a software change in 21.2.2, you can only do a rolling upgrade when the installed and upgrade versions are both before 21.2.2 or when the installed and upgrade versions are both 21.2.2 or later. For example, you can do a rolling upgrade from or from 21.2.0 to 21.2.1 or from 21.2.2 to 21.2.7, but not from 21.2.1 to 21.2.7.

A Supercluster rolling upgrade follows these general steps:

Upgrade the software on all nodes of the leader PCE.
Migrate the database on the leader PCE.
Bring the leader PCE back to runlevel 5.
Repeat these steps for each member PCE.

Steps for Upgrade

Upgrade the software on the leader PCE.
1. On any node of the leader PCE, stop the PCE cluster:
```
$ sudo -u ilo-pce illumio-pce-ctl cluster-stop
```
2. The packages to install depend on the type of PCE node:
  - Core nodes: Two packages, the PCE RPM and UI RPM.
  - Data nodes: One package, the PCE RPM.
  On each core node in the cluster, log in as root and install the PCE RPM and UI RPM. Be sure to specify both of the RPM file names on the command line:
```
$ rpm -Uvh illumio_pce_rpm illumio_ui_rpm
```
  For illumio_pce_rpm and illumio_ui_rpm, substitute the paths and filenames of the two RPM files you downloaded from the Illumio Support portal.
3. On each data node in the cluster, log in as root and install the PCE RPM:
```
$ rpm -Uvh <illumio_pce_rpm>
```
  For illumio_pce_rpm, substitute the path and filename of the software you downloaded from the Illumio Support portal.
4. On any node, start the cluster at runlevel 1:
```
$ sudo -u ilo-pce illumio-pce-ctl start --runlevel 1
```
Update the runtime environment file (runtime_env.yml). See What's New and Changed in This Release to determine whether any changes to runtime_env.yml are required to upgrade. If changes are required:
1. On all nodes in the cluster, update the runtime_env.yml file.
2. On all nodes in the cluster, check the validity of the runtime_env.yml file:
```
$ sudo -u ilo-pce illumio-pce-ctl check-env
```
  If any issues are reported by this command, correct them before moving on to the next step.
Migrate the PCE database on the leader PCE.
1. On any node of the leader PCE, run the following command:
```
$ sudo -u ilo-pce illumio-pce-db-management migrate --upgrade-type rolling
```
2. The migration might take some time to complete. Check the progress with the following command:
```
$ sudo -u ilo-pce illumio-pce-db-management supercluster-upgrade-status
```
Bring the leader PCE back to operational status.
1. On any node of the leader PCE, set the runlevel to 5 :
```
$ sudo -u ilo-pce illumio-pce-ctl set-runlevel 5
```
  Setting the runlevel can take time to complete.
2. On any node of the leader PCE, verify that the set-runlevel command finished and the cluster status is RUNNING:
```
$ sudo -u ilo-pce illumio-pce-ctl cluster-status -w
```
Upgrade the software on a member PCE.
1. On any node of the member PCE, stop the PCE cluster:
```
$ sudo -u ilo-pce illumio-pce-ctl cluster-stop
```
2. On all nodes of the member PCE, install the new version of the PCE. For information, see the PCE Installation and Upgrade Guide.
3. On any node, start the cluster at runlevel 1:
```
$ sudo -u ilo-pce illumio-pce-ctl start --runlevel 1
```
Migrate the PCE database on the member PCE.
1. On any node of the member PCE, run the following command:
```
$ sudo -u ilo-pce illumio-pce-db-management migrate
```
2. The migration might take some time to complete. Check the progress with the following command:
```
$ sudo -u ilo-pce illumio-pce-db-management supercluster-upgrade-status
```
Bring the member PCE back to operational status.
1. On any node of the member PCE, set the runlevel to 5 :
```
$ sudo -u ilo-pce illumio-pce-ctl set-runlevel 5
```
  Setting the runlevel can take time to complete.
2. On any node of the member PCE, verify that the set-runlevel command finished and the cluster status is RUNNING:
```
$ sudo -u ilo-pce illumio-pce-ctl cluster-status -w
```
Repeat steps 4 through 6 for each additional member PCE.
Verify that you can log in to the PCE web console on each PCE in the Supercluster.

Upgrade is complete.

NOTE:

In rare cases, you might receive an error when attempting to log into the PCE web console. When this happens, run the following command on all nodes, and try logging in again:

$ sudo -u ilo-pce illumio-pce-ctl restart

During Supercluster Upgrade

During a rolling upgrade, if you log in to one of the PCEs, you will see a banner that states the Supercluster is in the process of a rolling upgrade.

The PCE Health page on the Leader displays the upgrade status for each PCE. The Upgrade Status column shows Pending if the PCE is in the process of being upgraded, and it shows Complete when the upgrade is complete. When the upgrade is finished, the Upgrade Status column no longer appears.

Supercluster Listen Only Mode

The PCE “Listen Only” mode allows you stop the PCE from sending policy changes to your VENs. Enabling Listen Only mode for the PCE is typically used in these situations:

During PCE maintenance windows, and when starting the PCE back up.
After restoring the PCE from a backup.
During maintenance windows for other parts of your network environment.

In Listen Only mode, VENs still report updated workload information to the PCE, but the PCE does not modify the firewall rules on any workloads or send any updates from the PCE to the VENs. Also, the PCE does not mark workloads as Offline and does not remove them from policy when Listen Only mode is enabled.

When this mode is enabled, you can still write policy, pair new workloads, provision policy changes, assign or change workload Labels, but changes will not be sent to the VENs until you disable Listen Only mode. You can disable Listen Only mode when you are ready to resume normal policy operations.

Enable PCE Listen Only Mode

On all nodes in the PCE cluster, stop the PCE software:
```
$ sudo -u ilo-pce illumio-pce-ctl stop
```
On all nodes in the PCE cluster, set the node at runlevel 1:
```
$ sudo -u ilo-pce illumio-pce-ctl start --runlevel 1
```

On any data node, enable Listen Only mode:

$ sudo -u ilo-pce illumio-pce-ctl listen-only-mode enable

Set the PCE runlevel to 5:

$ sudo -u ilo-pce illumio-pce-ctl set-runlevel 5

Disable PCE Listen Only Mode

NOTE:

The command to disable PCE Listen Only mode can be executed at either runlevel 1 or 5.

On all nodes in the PCE cluster, stop the PCE software:
```
$ sudo -u ilo-pce illumio-pce-ctl stop
```
On all nodes in the PCE cluster, set the node to runlevel 1:
```
$ sudo -u ilo-pce illumio-pce-ctl start --runlevel 1
```

On any data node, disable Listen Only mode:

$ sudo -u ilo-pce illumio-pce-ctl listen-only-mode disable

Set the PCE runlevel to 5:

$ sudo -u ilo-pce illumio-pce-ctl set-runlevel 5