Upgrade the PCE

This section describes how to upgrade the PCE and its UI together. To upgrade the UI alone, see UI-Only Upgrade for information.

When upgrading the PCE and UI packages together, perform the following high-level tasks:

Verify that all Upgrade Prerequisites are met.
Perform PCE installation planning and prerequisite steps if they have not already been done on this PCE, as described in Preparing for PCE Installation.
Back up the PCE.
Download the software.
Stop the PCE.
Install the new PCE and UI.
Update the runtime environment file.
Migrate the database.
Set runlevel to 5.
Verify successful upgrade.

Back Up the PCE

When you are upgrading from a previous PCE version, the first step is to back up your existing data.

Back Up PCE Data

(On an SNC, skip this step.) Before you back up the PCE, determine which data node is running the agent_traffic_redis_server service:

$ sudo -u ilo-pce illumio-pce-ctl cluster-status

You see the following output:

SERVICES (runlevel: 5) NODES (Reachable: 1 of 1)
====================== ===========================
agent_background_worker_service 192.168.33.90
agent_service NOT RUNNING
agent_slony_service 192.168.33.90
agent_traffic_redis_cache 192.168.33.90
agent_traffic_redis_server 192.168.33.90          <=== run the dump command from this node
agent_traffic_service NOT RUNNING
...

On the data node that is running the agent_traffic_redis_server service, run the following command:
```
$ sudo -u ilo-pce illumio-pce-db-management dump --file <location-of-db-dump-file>
```
In location-of-db-dump-file, enter a file name.

NOTE:
On an SNC, run this command on the single node.
After the dump command finishes, copy the backup files to a fault-tolerant storage location.

Back up the PCE Runtime Environment File

Store a copy of each node's runtime_env.yml file on a system that is not part of the cluster or Supercluster. The default location of the PCE Runtime Environment File is /etc/illumio-pce/ runtime_env.yml.

Download the Software

Download the software from the Illumio Support portal (login required).
On the core nodes only, copy the Illumio PCE UI RPM file to the /tmp folder. In the following steps, this file is referred to as illumio_ui_rpm.
On all nodes in the cluster, copy the Illumio PCE software RPM file to the /tmp folder. In the following steps, this file is referred to as illumio_pce_rpm.

Stop the PCE

On all nodes in the cluster, stop the PCE:

$ sudo -u ilo-pce illumio-pce-ctl stop --wait

On all nodes in the cluster, verify the PCE status is STOPPED:
```
$ sudo -u ilo-pce illumio-pce-ctl status -sv --wait
```

Install the New PCE and UI

The packages to install depend on the type of PCE node:

Core nodes: Two packages, the PCE RPM and UI RPM.
Data nodes: One package, the PCE RPM.

On each core node in the cluster, log in as root and install the PCE RPM and the UI RPM. Be sure to specify both of the RPM file names on the command line:
```
$ rpm -Uvh illumio_pce_rpm illumio_ui_rpm
```
For illumio_pce_rpm and illumio_ui_rpm, substitute the paths and filenames of the two RPM files you downloaded from the Illumio Support portal.
On each data node in the cluster, log in as root and install the PCE RPM:
```
$ rpm -Uvh illumio_pce_rpm
```
For illumio_pce_rpm, substitute the path and filename of the software you downloaded from the Illumio Support portal.

Update the Runtime Environment File

See What's New and Changed in This Release to determine if any changes to the PCE Runtime Environment File (runtime_env.yml) are required to upgrade.

WARNING:

The cluster_type runtime parameter must be set on all PCE nodes starting in PCE 21.5.0, except on a single node cluster (SNC). If you are upgrading from a version earlier than 21.5.0, be sure to set this parameter. For details about what value to use on each type of node, see Reference: PCE Runtime Parameters.

To make changes to the runtime environment configuration:

On all nodes in the cluster, update the runtime_env.yml file.
On all nodes in the cluster, check the validity of the runtime_env.yml file:
```
$ sudo -u ilo-pce illumio-pce-ctl check-env
```
If any issues are reported by this command, correct them before moving on to the next step.

Migrate the PCE Database

Ensure that you have upgraded all nodes in the cluster to the same version before you perform these steps. Otherwise, none of the nodes in your cluster will start.

On all nodes in the cluster, start the PCE:

$ sudo -u ilo-pce illumio-pce-ctl start --runlevel 1

For some upgrades, you might be prompted to upgrade the database PostgreSQL software on one of the data nodes.

At the prompt, enter yes to continue the upgrade.

If you do not see this prompt, go to the next step.

Upgrade needed. The agent database is currently running at postgres version 9.6 and needs to be upgraded to version 11.4
Upgrade needed. The traffic database is currently running at postgres version 9.6 and needs to be upgraded to version 11.4
The PCE software will now upgrade to a newer version of the postgres software from an older version.
You need to migrate the PCE database after this process has finished.
Prior to this upgrade, Illumio recommends you make a backup of the relevant data directories:
/var/illumio_pce_data/persistent
The upgrade must not be interrupted, otherwise the data might get corrupted and prevent the PCE from starting.
Do you wish to continue with the database upgrade. [yes/no]: yes
Proceeding with database upgrade.
Please wait until the system has reached runlevel 1.

On all nodes in the cluster, verify the PCE status:

$ sudo -u ilo-pce illumio-pce-ctl status -sv --wait

Verify that the runlevel is 1:

$ sudo -u ilo-pce illumio-pce-ctl get-runlevel

On any node in the PCE cluster, migrate the database to the latest schema version:
```
$ sudo -u ilo-pce illumio-pce-db-management migrate
```
NOTE:
This command can take some time to complete, depending on the amount of data. To check progress, view the Health page in the PCE web console. If the migration is still underway, you will see a message like "Traffic database migration in progress."

Set Runlevel 5

Bring the PCE to runlevel 5, full operation:

$ sudo -u ilo-pce illumio-pce-ctl set-runlevel 5

IMPORTANT:

If you did not run the illumio-pce-db-management migrate command on the primary database, you cannot bring the node up to runlevel 5 and you cannot start the other nodes in the cluster. If some of the nodes in the cluster are already running, they will shut down until you successfully migrate the database. If you attempt to start the upgraded PCE cluster without migrating the database, this error is displayed:

$ sudo -u ilo-pce illumio-pce-ctl start
Starting Illumio Runtime STARTING 20.96s
$ 
$ Stopping PCE software: DB migrations mismatch for DB: avenger_executor_dev: Missing migrations.

Verify Success

On all nodes in the cluster, verify the PCE status:
```
$ sudo -u ilo-pce illumio-pce-ctl status -s -v -w
```

On any node, verify the cluster status:

$ sudo -u ilo-pce illumio-pce-ctl cluster-status -w

When you have a front end load balancer (F5 or DNS), make sure that the load balancer is sending requests to the two core nodes in the cluster.
Log into the PCE web console, pair a VEN, and verify VEN sync status is showing as “Verified” for a few randomly selected workloads. Select a Workload's details page and verify that the Policy Sync section shows “Verified.”
Test that the firewall is correctly configured by connecting to restricted services using telnet. When the PCE is running, connections on TCP port 8300 should be accepted from other PCE hosts but rejected otherwise. Connections on TCP port 8443 should be accepted from all sources (unless modified using the -p option).