PCE Troubleshooting Scenarios

This section describes issues that can arise during PCE installation or upgrade and how to resolve them.

Session Limits Too Low

(RHEL7+ only)

Symptom:

The expected session limits, configured in /etc/security/limits.conf, might not be in effect for the PCE. This can cause a severe performance impact that may go unnoticed for some time.

Cause:

It typically affects systems which have a PAM authentication configuration utilizing the loginuid module. This type of configuration relies on systemd rather than the traditional security limits for the application's session limits. The issue can also arise if the documented installation preparation steps are not followed (see Process and File Limits) or if the values are later altered.

You can verify the session limits by inspecting a running PCE with the following command:

cat /proc/$(pgrep -f config_listener.rb)/limits | grep -e open -e processes

Solution:

If the output of this command shows values for nofile and nproc that are lower than required (see Process and File Limits), provide an override file to properly configure these limits.

Create the following file on the PCE:

/etc/systemd/system/illumio-pce.service.d/override.conf
Add the following lines to the file and save:
```
[Service]
LimitNOFILE=65535
LimitNPROC=65535
```
Reboot or restart the PCE.
Run the grep command above again and inspect the config_listener.rb session limits to ensure that they are now correct.

Database Migrations Mismatch

Symptom:

Error message “Stopping PCE software: DB migrations mismatch for DB: avenger_executor_dev: Missing migrations” when you try to bring the PCE to runlevel 5.

Cause:

Attempted to start the upgraded PCE without migrating the database. If you did not run the illumio-pce-db-management migrate command on the primary database node, you will not be able to bring any PCE node up to runlevel 5, and you will not be able to start the other nodes in the cluster. If some of the nodes in the cluster are already running, they will be shut down until you successfully migrate the database.

Solution:

Follow the steps in Migrate the PCE Database.

Database Already Exists

Symptom:

The illumio-pce-db-management setup command finishes abnormally with Exit Code = 1 and displays the following type of messages:

Database 'traffic_prod' already exists
...
psql:/opt/illumio-pce/illumio/webservices/traffic_query/db/structure.sql:52: ERROR:  relation "ar_internal_metadata" already exists
rake aborted!
failed to execute:
psql -v ON_ERROR_STOP=1 -q -f /opt/illumio-pce/illumio/webservices/traffic_query/db/structure.sql traffic_prod

Cause:

The database has already been set up.

Solution:

If you are trying to set up a new database, you must first remove the existing database.

WARNING:

All existing data will be lost.

To remove the database and all its data, run the command illumio-pce-db-management drop. Then, retry the illumio-pce-db-management setup command.

PCE UI Missing

Symptom:

Error message displayed in the browser, such as “PCE UI Missing.”

Cause:

The PCE package has been installed without the UI package.

Solution:

Install the UI package as described in UI-Only Upgrade.