Requirements for PCE Installation

Before installing the PCE, be sure your underlying systems are sufficient to successfully install and run the PCE. Check all the following system requirements.

Load Balancer Requirements

For MNC; does not apply in a SNC.

A server load balancer or DNS-level load balancer is required to distribute traffic to the PCE core nodes.

Configure the load balancer to use the Illumio REST API to monitor which cluster core nodes are available to receive requests. See the REST API Developer Guide for exact usage.

GET [api_version]/node_available

No authentication is required to call this API. An HTTP status code of 200 means the node is able to receive requests. Any other status code or no response means the node is unable accept requests. Unhealthy or unresponsive nodes should be removed from the load balancing pool.

  • The PCE Health Check API can experience up to a 30-second delay to return the actual status of the node.
  • In the 4x2 configuration, a maximum of two core nodes are available (return a status code of 200) at any time.
  • When using a DNS load balancer, it should only serve IP addresses for the cluster FQDN of those nodes that respond with a 200 to the /node_available API. For rapid failover in the event of a core node failure, Illumio recommends a DNS TTL of between 30 and 60 seconds.

PCE IP Address

Illumio recommends a statically-assigned IP address. By default, the PCE automatically uses the first available private IP address on the node. The PCE does not automatically bind to a public IP address.

When you use a public IP address or the node has multiple interfaces, you need to configure the PCE with the interface you want to use. To do so, set internal_service_ip in the configuration file runtime_env.yml. For example:

internal_service_ip: 10.2.8.89

To configure networking, see your OS vendor's documentation on the ifcfg-ethN script.

DNS Requirements

Your Domain Name System (DNS) must resolve the PCE's fully qualified domain name (FQDN). The FQDN must be resolvable on all managed workloads, on all nodes in the PCE cluster, and for all users of the PCE web console and REST API.

If you are using DNS-level load balancing, the PCE FQDN should resolve to the IP addresses of the core nodes. If you are using a server load balancer, the PCE FQDN should resolve to the VIPs of the server load balancer.

SMTP Requirements

An SMTP relay is required to send user invitations and “forgot password” email replies from the PCE.

The SMTP configuration parameter during PCE installation is smtp_relay_address. Allowable values are either an IP address with its SMTP port (default 587) or a resolvable FQDN with the SMTP port.

TLS Requirements

PCE communication is secured using the Transport Layer Security (TLS) protocol, the successor to the deprecated Secure Sockets Layer (SSL) protocol. TLS is used for securing the following communication sessions:

  • User access to the PCE web console and REST API over the HTTPS protocol.

  • Communication between the PCE and VENs.

    VEN-to-PCE communications for the EventService (default is port 8444) are secured by the ECDHE suite of cryptographic ciphers, which use an elliptic curve Diffie-Hellman key exchange. This exchange is signed with RSA signature algorithms.

  • Communication between PCE nodes in a multi-node cluster.

If you want to generate a temporary, self-signed certificate, see Understanding Illumio Trial Certificates in the Knowledge Base (log in required).

For an in-depth discussion of deploying the PCE with TLS, see Preparing Certificates for a PCE deployment in the Knowledge Base (log in required).

X.509 Certificate

An X.509 server certificate must be installed on each PCE node during installation. When any client (the VEN) opens a TLS session to the PCE (for example, pairing a workload, accessing the PCE web console, retrieving updated policy), the PCE presents the server certificate to secure the communication. The server certificate is uploaded as part of a certificate bundle that contains the server certificate and the chain of CA certificates (Intermediate or Root) to establish the chain of trust back to a Root CA.

CAUTION:

The client must be able to validate the chain of trust back to the Root CA for this certificate; otherwise, the TLS handshake fails. You might need to add all the certificates in the chain of trust to the keychain of the client.

The certificate package for the Illumio PCE must meet the following basic criteria:

  • The file must contain PEM-encoded certificates.
  • The subject value and issuer of the certificate must start with a leading slash character (/).
  • As a best practice, duplicate the subject in the Subject Alternative Name (subjectAltName).
  • The certificate's signature algorithm must be SHA256WithRSAEncryption.
  • The certificate's signature algorithm must not be RSASSA-PSS.
  • The file must contain the server certificate and the entire certificate chain necessary to establish the chain of trust back to a Root CA.
    1. The package must include all of the CA certificates (Intermediate and/or Root) needed to establish the chain of trust back to a Root CA.
      • If the certificate is generated by a Private CA, all certificates in the chain of trust back to the Root CA must be included. This includes the Root CA certificate and any applicable Intermediate CA certificates.
      • If the certificate is generated by a major Public CA (such as, VeriSign, GeoTrust, Entrust, or Thawte), any Intermediate CA certificates needed to establish the chain of trust back to the Public Root CA must be included.

    2. Pay careful attention to the order of the certificates in the bundle. The server certificate must be first. If you have an Apache-style bundle generated by a standard certificate request process, you need to open the file in a text editor and reverse the order of the certificates. Apache always expects the root certificate to come first, then any intermediates in order (from the root down), and the server certificate is last. The PCE uses nginx, which expects the opposite order. For additional details, see the Nginx documentation.

      The certificate bundle should look something like this:

      -----BEGIN CERTIFICATE-----
<server cert goes here>
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
<intermediate CA cert goes here>
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
<root CA cert goes here>
-----END CERTIFICATE-----
  • All certificates in the bundle must be valid for the current date, which depends on the system time being set correctly.
  • A trusted root store must be available for OpenSSL to validate certificates.
  • The certificate must match the PCE FQDN, which can be an exact match (for example, pce.mycompany.com) or a wildcard match (for example, *. mycompany.com)

The certificate must support both Server and Client authentication. Client authentication is used between nodes in an MNC. Run the following command and verify TLS Web Server Authentication, TLS Web Client Authentication appears within the X509v3 Extended Key Usage section.

$ openssl x509 -text -noout -in pce.mycompany.com.bundle.crt
...
X509v3 Extended Key Usage:
         TLS Web Server Authentication, TLS Web Client Authentication
...

RSASSA-PSS Signature Algorithm Not Supported

The certificate signature algorithm RSASSA-PSS, which is based on PKCS 1 version 2.1, is not supported, because it cannot be validated. This limitation is a widely known problem with this signature algorithm.

The PCE certificate requires the SHA256WithRSEencryption signature.

CAUTION:

If you use Microsoft Certificate Authority (CA) to sign PCE certificates, make sure to use the SHA256WithRSEencryption. PKCS#1 version 2.1 is enabled by default on Microsoft CAs and produces the unsupported RSASSA-PSS signature algorithm.

Private Keys

The private key that matches the X.509 certificate must be installed on each PCE node during installation, and the following guidelines must be met:

  • The private key must be PEM-encoded.
  • The file must not be encoded.
  • The file must not be password protected.

TLS Versions for Communications

The PCE uses Transport Layer Security (TLS) version 1.2 by default for VEN-to-PCE communications, the PCE's web server for the PCE web console, and the REST API.

  • The PCE default minimum version is TLS 1.2.
  • For VEN versions 18.1 and later, all VENs use TLS 1.2.
  • Older operating systems might not support TLS 1.2. For example, SUSE VEN version 17.1.x, a legacy VEN version that is no longer supported by Illumio, requires minimum version TLS 1.0.
  • Windows Server 2008 R2 SP1: The HTTP Client library, WinHttp, does not have the necessary API to limit SSL negotiation only to TLS 1.2. This must be configured through the Registry. See the Microsoft Support article “Update to enable TLS 1.1 and TLS 1.2.”

Changing Minimum TLS version

The default minimum TLS version is TLS 1.2. The minimum TLS version is configurable, but it is recommended that you leave the setting at its default of 1.2. Earlier TLS versions, such as 1.0 and 1.1, are considered less secure than 1.2, so it is recommended you do not use them. In some rare circumstances, you might need to change the minimum TLS version, such as when using older VEN operating systems that do not support TLS 1.2. In addition, you should verify that any browser you use is capable of negotiating the minimum version you set.

If you want to change the minimum TLS version, edit the following parameter in runtime_env.yml:

min_tls_version

The value of min_tls_version configures the PCE front end ports in runtime_env.yml:

  • front_end_https_port (default 8443)
  • front_end_https_management_port (defaults to front_end_https_port)
  • front_end_event_service_port (default 8444)

Allowable values:

  • tls1_0 allows TLS 1.0, 1.1, and 1.2.
  • tls1_1 allows TLS 1.1 and 1.2.
  • tls1_2 allows TLS 1.2.

(Optional) Configure SAML IdP for User Login

After installation, you can configure the PCE to rely on an external, third-party SAML identity provider system. See Single Sign-on Configuration in the PCE Administration Guide. The guide has step-by-step details for a wide variety of IdPs.

OS Setup and Package Dependencies

For information, see PCE OS Support and Package Dependencies on the Illumio Support portal.

NTP

Set up a Network Time Protocol (NTP) client for time synchronization. It is recommended that you use chrony, although ntpd can also be used. On RHEL8, chrony is the default.

To install and configure the NTP client, use the procedure in the documentation for the client on your operating system.

After you finish installing the PCE, you can use the following command to verify that the NTP client is installed, running, and synchronized to a time source:

# sudo -u ilo-pce illumio-pce-env check

IPTables

For the initial installation, you should disable iptables.

If iptables is enabled, you must configure it to allow inbound HTTPS connections to the PCE core nodes and service ports.

# service iptables stop    
# On CentOS 7.x, use the systemctl stop firewalld command.
# chkconfig iptables off

Language: UTF-8

Set the system language to a UTF-8 variant of English: either en_US.UTF-8 or en_GB.UTF-8.

Set the variable LANG="en_US.UTF-8" or LANG="en_GB.UTF-8" in the file /etc/locale.conf.

Trusted Public CA Store

A trusted root public Certificate Authority (CA) store must be available for OpenSSL to validate certificates.

If you rely on a certificate signed by a public CA, be sure to install the latest public root CA certificates ca-certificates package.

# yum install ca-certificates

When your certificate is signed by a private CA or the signing CAs are already included in each node's trusted root CA store, the ca-certificates package is not required.

PCE Internal Syslog

The PCE comes with an internal syslog configuration. The purpose of the PCE internal syslog is to help organizations use syslog without installing it themselves. See (Optional) Configure PCE Internal Syslog.

Process and File Limits and Kernel Parameters

This section describes how to set the process and file limits and OS kernel parameters that are required for PCE operation. The approach is different depending on whether you are configuring an SNC or MNC, and which operating system you are using, so look for the appropriate sections in the discussion that follows.

Three categories of settings must be configured:

  • Process and file limits
  • OS kernel parameters
  • Kernel module tuning
WARNING: The parameter modifications described in this section are strict requirements and must be followed to ensure proper functionality of the Illumio Core. If an Illumio support case is opened, and analysis finds that these parameters are not met, you will be directed to meet these requirements before any additional troubleshooting can be performed.

Keep the following in mind when managing these parameters:

  • Root access is needed for many of these procedures. Befor you start, be sure you have login credentials for a user account with root permissions.
  • When your settings are already greater than these, you do not need to reduce them to these values.
  • Make sure you do not have any automated processes that change these values.

SNC Process and File Limits and OS Kernel Parameters

The following table shows the required process and file limits for single-node clusters. To set these values, see Set and Verify Process and File Limits.

Parameter

Value

core (hard)

0

core (soft)

0

nofile (hard)1

65535

nofile (soft)1

65535

nproc (hard)

65535

nproc (soft)

65535

1 When you run additional processes on the PCE, such as monitoring or other operations processes, you might need to increase the value of nofile.

The following table shows the required OS kernel parameter values for single-node clusters. To set these values, see Set and Verify OS Kernel Parameters.

Parameter

Value

fs.file-max

2000000

net.core.somaxconn

16384

vm.overcommit_memory

1

The following table shows the required SNC kernel module tuning. To set this value, see Tune the Kernel Module.

Parameter

Value

nf_conntrack hashsize

262144

MNC Process and File Limits and OS Kernel Parameters

The following table shows the required process and file limits for multi-node clusters. To set these values, see Set and Verify Process and File Limits.

Parameter

Core Nodes

Data Nodes

core (hard)

0 0

core (soft)

0 0

nofile (hard)1

65535 65535

nofile (soft)1

65535 65535

nproc (hard)

65535 65535

nproc (soft)

65535 65535

The following table shows the required OS kernel parameter values for multi-node clusters. To set these values, see Set and Verify OS Kernel Parameters.

Parameter

Core Nodes

Data Nodes

fs.file-max

2000000 2000000

net.core.somaxconn

16384 Use system default setting

vm.overcommit_memory

Use system default setting 1

The following table shows the required kernel module tuning. To set this value, see Tune the Kernel Module.

Parameter

Core Nodes

nf_conntrack hashsize

262144

Set and Verify Process and File Limits

Process and file limits are set by editing configuration files and issuing commands. The techniques vary depending on the operating system version and which system management daemon you are using, systemd or init.d. If you are not sure which system management daemon is being used, run the following command:

$ ps -p1 | grep "init|upstart|systemd"

CentOS 7.x or RHEL 7.x/8.x with systemd

On every core and data node, do the following steps:

  1. As root, edit the following Illumio-specific configuration file:

    /etc/systemd/system/illumio-pce.service.d/override.conf

  2. Place the following lines in the file.:

    [Service]
LimitCORE=0
LimitNOFILE=65535
LimitNPROC=65535

  3. Reload the daemon configuration and restart the service to apply the change:

    # systemctl daemon-reload
# systemctl restart illumio-pce.service

  4. Verify that the correct settings are now in effect. As the PCE runtime user, run the following command. Verify that the output is as shown:

    $ sudo -u ilo-pce systemctl show illumio-pce.service | egrep "LimitCORE|LimitNPROC|LimitNOFILE"
LimitCORE=0
LimitNOFILE=65535
LimitNPROC=65535
See also the Linux systemd man page, especially "Table 1. Resource limit directives."

CentOS 7.x or RHEL 7.x/8.x with init.d

On every core and data node, do the following steps:

  1. As root, edit the following Illumio-specific configuration file:

    /etc/security/limits.d/99-illumio.conf

  2. Place the following lines in the file. The ilo-pce on each line indicates that the limits apply to only the PCE runtime user, which is ilo-pce unless this default user name was overridden during PCE installation. If you want the limits to apply to all users, use asterisks (*) instead of ilo-pce.

    ilo-pce    soft    core      0
ilo-pce    hard    core      0
ilo-pce    soft    nofile    65535
ilo-pce    hard    nofile    65535
ilo-pce    soft    nproc     65535
ilo-pce    hard    nproc     65535

  3. Apply the change:

    # sysctl -p /etc/security/limits.d/99-illumio.conf

  4. Verify that the correct settings are now in effect. As the PCE runtime user, run the following commands. Verify that the output is as shown:

    $ sudo -u ilo-pce ulimit -n
65535

$ sudo -u ilo-pce ulimit -u
65535

$ sudo -u ilo-pce ulimit -c
0

Set and Verify OS Kernel Parameters

Kernel parameters are set by editing configuration files and issuing commands. The commands are the same on all PCE-supported versions of CentOS and RHEL, but the techniques vary depending on whether you are configuring an SNC or MNC.

SNC: Set and verify OS kernel parameters

  1. As root, edit the following Illumio-specific configuration file:

    /etc/sysctl.d/99-illumio.conf

  2. Place the following lines in the file:

    fs.file-max          = 2000000
vm.overcommit_memory = 1
net.core.somaxconn   = 16384

  3. Apply the settings:

    # sysctl -p /etc/sysctl.d/99-illumio.conf

  4. Verify that the correct settings are now in effect. As the PCE runtime user, run the following command. Verify that the output is as shown:

    $ sudo -u ilo-pce sysctl -a 2>/dev/null | egrep "fs.file-max|vm.overcommit_memory|net.core.somaxconn"
fs.file-max = 2000000
net.core.somaxconn = 16384
vm.overcommit_memory = 1

For more information, see Configuring Kernel Parameters at Runtime in the Red Hat documentation.

MNC: Set and verify OS kernel parameters

  1. As root, on each core node, edit /etc/sysctl.d/99-illumio.conf and add the following lines:

    fs.file-max          = 2000000
net.core.somaxconn   = 16384

  2. As you go, on each core node, apply the settings:

    # sysctl -p /etc/sysctl.d/99-illumio.conf

  3. As root, on each data node, edit /etc/sysctl.d/99-illumio.conf and add the following lines:

    fs.file-max          = 2000000
vm.overcommit_memory = 1

  4. As you go, on each data node, apply the settings:

    # sysctl -p /etc/sysctl.d/99-illumio.conf

  5. Verify that the correct settings are now in effect. As the PCE runtime user, run the following command. Verify that the output is as shown:

    $ sudo -u ilo-pce sysctl -a 2>/dev/null | egrep "fs.file-max|vm.overcommit_memory|net.core.somaxconn"
fs.file-max = 2000000
net.core.somaxconn = 16384
vm.overcommit_memory = 1

For more information, see Configuring Kernel Parameters at Runtime in the Red Hat documentation.

Tune the Kernel Module

Adjust the hash size setting for the kernel conntrack module as follows. For this setting, the commands are the same on all PCE-supported versions of CentOS and RHEL.

On all core nodes:

  1. As root, run the following commands to tune the kernel conntrack module. The commands take effect immediately.

    # modprobe nf_conntrack
# echo 262144 > /sys/module/nf_conntrack/parameters/hashsize

  2. Run the following command to apply the same setting automatically on reboot:

    # echo "options nf_conntrack hashsize=262144" > /etc/modprobe.d/illumio.conf

  3. Verify that the correct setting is now in effect. Run the following command to inspect the hash size. Verify that the output is as shown:

    # cat /sys/module/nf_conntrack/parameters/hashsize
262144

About Your Organization Name and ID

An organization is a group of policies and users in the Illumio Core. An organization can contain any number of users, workloads, and policy objects (rulesets, IP lists, services, and security settings). When you sign up with Illumio, you will receive an email invitation to create your company's organization in Illumio Core.

Have ready your full organization name, which you specify at installation.

For on-premise PCE deployments, installation creates an organization identifier (org ID) and assigns the value of 1 to org ID. The value 1 distinguishes your on-premises PCE from the Illumio Core Cloud (SaaS) service, where each customer has a unique org ID.

The org ID is needed with the REST API, where you set org-ID to 1 for the on-premises PCE, and for other purposes.