Back Up Supercluster

You need to perform regular backups on all PCEs in the Supercluster.

Different data is backed up depending on whether you run the backup from the Supercluster leader or a member: 

  • Leader backup: Contains all Supercluster replicated data, including workloads, labels, rulesets, rules, services, organization events, workload traffic data, and Supercluster configuration data.
  • Member backup: Contains the member's local data, including login information, workload traffic data, and Supercluster configuration data.

  • All PCE nodes' runtime environment file: The runtime_env.yml is not included in the backup and must be backed up separately for each node. The default location of the PCE Runtime Environment File is /etc/illumio-pce/runtime_env.yml. When the location is different on your system, you can find it by checking the value of the ILLUMIO_RUNTIME_ENV environment variable.

When to Back Up

Follow your own organization's policies and procedures for backup, including frequency (such as, hourly, daily, or weekly) and retention of backups offsite or on a system other than any of the Supercluster nodes.

Illumio recommends taking backups in the following situations:

  • Before and after a PCE version upgrade
  • After pairing a large number of VENs
  • After updating a large number of workloads (such as, changing workload policy state or applying labels)
  • After provisioning major policy changes
  • After making major changes in your environment that affect workload information (such as, an IP address changes)
  • Before and after adding new PCEs to your Supercluster
  • After you assign a new leader
  • On-demand backups before the procedures documented in this guide, such as migration and upgrade

Determine Data Node of All PCEs

The data node is the node that runs the agent_traffic_redis_server service. To determine the data node, run the following command:

$ sudo -u ilo-pce illumio-pce-ctl cluster-status

Expected output:

SERVICES (runlevel: 5) NODES (Reachable: 1 of 1)
====================== ===========================
agent_background_worker_service 192.168.33.90
agent_service NOT RUNNING
agent_slony_service 192.168.33.90
agent_traffic_redis_cache 192.168.33.90
agent_traffic_redis_server 192.168.33.90          <=== Run backup command on this node
agent_traffic_service NOT RUNNING
...
NOTE:

Check for agent_traffic_redis_server on a data node before every backup, because this service can be running on either data node.

Back Up Each PCE's Data

For the leader and every member PCE in your Supercluster, perform these steps:

  1. Log into the node running the agent_traffic_redis_server service.
  2. Create a directory for the backup file that is not one of the PCE software's installation directories.
  3. Grant both the ilo-pce user and the user who will run the backup command Read and Write permissions to this directory.

  4. Run the following command:

    $ sudo -u ilo-pce install_root/illumio-pce-db-management supercluster-data-dump --file desired_location_of_backup_file
  5. Repeat these steps for every PCE in the Supercluster.

Copy Leader Backup to Members

Copy the backup file that you just made on the leader PCE to the data0 node of each member PCE. The leader's data is readily available to every member so that you can more quickly restore the entire Supercluster. You can copy the file to any file system location of the member data0 node, except for the PCE software's installation directories.

Back Up Leader and Member Runtime Environment Files

Store a copy of each node's runtime_env.yml file on a system that is not part of the Supercluster. By default, the PCE Runtime Environment File is stored in /etc/illumio-pce/runtime_env.yml. When the location is different on your system, locate the file by checking the ILLUMIO_RUNTIME_ENV environment variable.