Local Policy Convergence Controller

The local policy convergence controller provides a deterministic way of setting the readiness state of pods in your cluster after local policy has converged. By controlling the readiness state of pods, you can prevent them from receiving and sending traffic through Kubernetes until they are ready. Using a controller ensures that the network and security infrastructure is ready for a multi-microservice application.

In this release, the Kubernetes Custom Pod Conditions feature introduced in v1.14 is available for containerized VENs.

About the Controller Behavior

By default, the readiness gate is not specified on a pod spec and the C-VEN does not affect the readiness state of the pod regardless of annotations or Illumio managed state.

When the Illumio readiness gate is specified on a pod spec, the PCE completes the following actions when a new pod is created:

  1. Sends the C-VEN policy for the new pod P.
  2. When pod P is managed, the C-VEN applies local policy for the new pod P.
  3. The C-VEN waits for a timer to expire to allow peers to apply policy on their end (such as, updating the new pod P IP address).

    By default, the timer uses the following values:

    • If the pod is managed by Illumio, the timer is set to 15 seconds.
    • If the pod is not managed by Illumio, the timer is set to 0 seconds.

      TIP:

      To configure a custom value for the timer duration, see Timer Customization.

  4. The C-VEN sets the readiness gate pod condition to “True.”

    The pod is now considered “Ready” by Kubernetes.

Configure the Illumio Readiness Gate

To use a local policy convergence controller, specify the Illumio readiness gate under readinessGates.conditionType in the pod spec YAML.

See the following example pod spec YAML file:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-deploy
spec:
  selector:
    matchLabels:
      app: my-pod
  replicas: 2
  template:
    metadata:
      labels:
        app: my-pod
    spec:
      readinessGates:                                   # <----- declare readiness gates
      - conditionType: "com.illumio.policy-ready"       # <----- Illumio policy convergence readiness gate
      containers:
      - name: my-pod-web
        image: nginx
        ports:
        - containerPort: 80

Timer Customization

You can customize the timer cluster-wide or pre-pod.

NOTE:

When configuring a custom timer by using the DaemonSet environment variable or an annotation, you are limited to specifying 0-300 seconds.

Cluster Wide Timer Customization

To customize the timer duration on a cluster-wide basis, set the readiness gate timer variable in the C-VEN DaemonSet YAML.

See the following YAML file:

...
containers:
  - name: illumio-ven
    env:
      - name: ILO_SERVER
        valueFrom:
          secretKeyRef:
            name: illumio-ven-config
            key: ilo_server
      - name: ILO_CODE
        valueFrom:
          secretKeyRef:
            name: illumio-ven-config
            key: ilo_code
      - name: ILO_K8S_NODE_NAME
        valueFrom:
          fieldRef:
            fieldPath: spec.nodeName
      - name: ILO_K8S_READINESS_TIMER         # <--- custom readiness gate timer across the cluster
        value: "20"                           # <--- timer value
...

Pre-pod Timer Customization

To customize the timer duration for specific pods, set the Illumio readiness gate timer annotation on the pod spec.

See the following example deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-deploy
spec:
  selector:
    matchLabels:
      app: my-pod
  replicas: 2
  template:
    metadata:
      labels:
        app: my-pod
      annotations:
        com.illumio.readiness-gate-timer: "20"          # <----- custom readiness gate timer for all pods in this deployment
    spec:
      readinessGates:
      - conditionType: "com.illumio.policy-ready"
      containers:
      - name: my-pod-web
        image: nginx
        ports:
        - containerPort: 80

Track the State of the Readiness Gate

You can track the state of the readiness gate by running either of the following commands:

  • kubectl get pod -o wide

  • kubectl get ep -o wide

Example: State of the Readiness Gate

This example shows a cluster with Kubelink and the C-VEN deployed and running. When you initially deploy or scaled up the Illumio Readiness Gate, you see the following values:

NOTE:

The state of gate readiness appears in the "READINESS GATES" column.

$ kubectl get pod,ep -o wide
NAME                             READY   STATUS    RESTARTS   AGE     IP           NODE       NOMINATED NODE   READINESS GATES
pod/my-deploy-855dfbf94f-gwz7c   1/1     Running   1          4d20h   172.17.0.7   ubuntu20   <none>           0/1
pod/my-deploy-855dfbf94f-p7czp   1/1     Running   1          4d20h   172.17.0.6   ubuntu20   <none>           0/1
 
NAME                   ENDPOINTS                         AGE
endpoints/kubernetes   10.0.2.15:8443                    19d
endpoints/my-service                                     4d22h

In this example, the readiness gates are marked as 0/1 for both pods and my-service does not have any available endpoints. After the VEN has processed the policy for the new pods and the timer expires, it sets the readiness gate to “True” for each pod and you see the following output:

$ kubectl get pod,ep -o wide
NAME                             READY   STATUS    RESTARTS   AGE     IP           NODE       NOMINATED NODE   READINESS GATES
pod/my-deploy-855dfbf94f-gwz7c   1/1     Running   1          4d20h   172.17.0.7   ubuntu20   <none>           1/1
pod/my-deploy-855dfbf94f-p7czp   1/1     Running   1          4d20h   172.17.0.6   ubuntu20   <none>           1/1
 
NAME                   ENDPOINTS                         AGE
endpoints/kubernetes   10.0.2.15:8443                    19d
endpoints/my-service   172.17.0.6:9376,172.17.0.7:9376   4d22h

To view greater detail about the pod conditions, run the command kubectl get pod <pod name> -o yaml:

$ kubectl get pod my-deploy-855dfbf94f-gwz7c -o yaml
...
status:
  conditions:
  - lastProbeTime: null                         // <--
    lastTransitionTime: "2021-05-18T20:26:26Z"  // <--
    message: Pod Policy Ready                     // <-- this pod condition is set by VEN
    reason: PolicyReady                         // <--
    status: "True"                              // <--
    type: illumio.com/policy-ready              // <--
  - lastProbeTime: null
    lastTransitionTime: "2021-05-18T20:25:51Z"
    status: "True"
    type: Initialized
  - lastProbeTime: null
    lastTransitionTime: "2021-05-19T19:56:24Z"
    status: "True"
    type: Ready                                 // <-- this is only set to True after all readiness gates are set to True
  - lastProbeTime: null
    lastTransitionTime: "2021-05-19T19:56:24Z"
    status: "True"
    type: ContainersReady
  - lastProbeTime: null
    lastTransitionTime: "2021-05-18T20:25:51Z"
    status: "True"
    type: PodScheduled
...