Bulk Traffic Loader

This Public Experimental API allows you to upload a snapshot of static traffic connections from a non-Illumio environment to the PCE so you can visualize what your network traffic data looks like in the PCE Illumination map without having to install VENs on your workloads.

NOTE:

Bulk operations are rate limited to 1,000 items per operation.

Bulk Traffic Loader API Methods

Functionality HTTP URI
Upload a collection of traffic data to the PCE

POST

[api_version][org_href]/agents/bulk_traffic_flows

Workflow to Upload Bulk Traffic

To upload bulk traffic data to the PCE:

  1. Obtain network traffic information from your environment (using netstat or another network utility). For each traffic connection, obtain the source IP address, destination IP address, destination port, and protocol.
  2. Create unmanaged workloads in the PCE.
  3. In the PCE, map both the source IP address and destination IP address for each traffic flow to the unmanaged workloads. An unmanaged workload is required for each source and destination IP address that represents traffic in the Illumination map. This must be done before you load the traffic data.
  4. Use the Bulk Traffic Loader API to upload the CSV-formatted traffic data into the PCE.

Create Collection of Unmanaged Workloads

Using this method, you can create a collection of unmanaged workloads in your organization using a single API call, rather than having to individually create individual workloads one at a time.

When you create a collection of workloads using this method, these workloads are recorded in the PCE as “unmanaged,” which means that no VEN has been installed on the workloads and no policy can be applied to them.

Create an unmanaged workload for each source IP address and destination IP address before you use the Bulk Traffic Loader API.

URI for Creating a Collection of Unmanaged Workloads

PUT [api_version][org_href]/workloads/bulk_create

Request Body

When you create a collection of workloads, you pass a JSON request body that defines the workload details.

Although this example illustrates the request body for a single workload, you can add as many workloads as you want using the bulk_create method.

Request Properties

Property Description Type Required
name Short, friendly name of the workload. String Yes
description Long description of the workload. String No
external_data_set An external data set identifier. For example, if this workload information is stored in an external database. String No
external_data_reference An external data set reference path. For example, if this workload information is stored in an external database. String No
hostname The hostname reported by the workload. String Yes
service_principal_name The Kerberos Service Principal Name (SPN). This property is only relevant if you have configured Kerberos in your environment to authenticate VEN-to-PCE communication. String No
distinguished_name

The X.509 Subject distinguished name, used if you want this workload to use machine authentication between VENs and other hosts.

NOTE:

This property has an API exposure level of Public Experimental, which means it is not intended for production use and might change in future releases. For more information, see API Classification and Version.

 String No
public_ip The public IP address of the workload. String No
interfaces

List of all functioning network interfaces on the workload.

Properties for workload network interfaces: 

  • name: The interface name with a string data type
  • link state: The state of the interface connection with a string data type; one of three values:
    • up: Interface is communicating.
    • down: Interface is not communicating.
    • unknown: State of the interface is unknown.
  • address: The IP address assigned to the interface with a string data type
  • cidr_block: The number of bits in the subnet (for example, /24 is 255.255.255.0) with an integer data type
  • default_gateway_address: The default IP address of the default gateway, either IPv4 or IPv6 with an integer data type
  • friendly_name: The friendly name given to the interface with a string data type
 Array No
service_provider Name of the service provider that is hosting the workload. String No
data_center The name of the data center where the workload is being hosted.    
data_center_zone The zone inside the data center hosting the workload. String No
os_id Unique OS identifier given by Illumio to the workload. String No
os_detail Additional descriptive information about the workload OS String No
online Indicates whether the workload is online and can communicate with the PCE. Boolean. No
labels Labels that are attached to the workload. Array. No
"agent" (VEN) properties hash:
mode

This property defines the policy state of the workload's VEN. There are three possible states: 

idle: A state in which the VEN does not take control of the workload’s iptables (Linux) or Windows firewall (Windows), but rather provides the PCE relevant details about the workload, such as the workload IP address, operating system, and geographical location.

illuminated: The Illuminated policy state has two variations: Build and Test (configurable in the PCE web console):

  • Build is a state in which the VEN introspects all open ports on a workload and reports the flow of traffic between it and other workloads to the PCE. In this state, the PCE displays the flow of traffic to and from the workload. The PCE provides insight into the data center and the applications running in it. No traffic is blocked in this state.

  • Test is a state which allows you to apply all rules and visualize all the traffic that would be blocked if you placed the workloads into the enforced policy state. No traffic is blocked in this state.

enforced: A state in which all rules definitions and restrictions are enforced.

 String No
log_traffic(deprecated) If set to True, then traffic flows from the VEN are logged. Boolean No
visibility_level

The amount of data the VEN collects and reports to the PCE from a workload in the enforced mode (policy state), so you can control resource demands on workloads.

The higher levels of detail are useful for visualizing traffic flows in the Illumination map inside the PCE web console. If this parameter is not set, then VEN visibility level is set to flow_summary.

  • flow_summary: (“High Detail” in the PCE web console) The VEN collects traffic connection details (source IP, destination IP, protocol, and source and destination port) for both allowed and blocked connections. This option creates traffic links in the Illumination map and is typically used during the building and testing phase of your security policy.
  • flow_drops: (“Less Detail” in the PCE web console.) The VEN only collects traffic connection details (source IP, destination IP, protocol, and source and destination port) for blocked connections. This option provides less detail for Illumination but demands fewer system resources from a workload and is typically used for policy enforcement.
  • flow_off: (“No Detail” in the PCE web console.) The VEN does not collect any details about traffic connections. This option provides no Illumination detail and demands the least amount of resources from workloads. This mode is useful when you are satisfied with the rules that have been created and do not need additional overhead from observing workload communication.
    

JSON Payload

{
    "name": "workload 0",
    "description": "workload desc 0",
    "service_principal_name": "spn 0",
    "hostname": "workload-0.example.com",
    "public_ip": "24.1.1.1",
    "ephemeral": false,
    "external_data_set": "cmdb",
    "external_data_reference": "0",
    "interfaces": [
      {
        "name": "eth0",
        "link_state": "up",
        "address": "10.0.0.2",
        "cidr_block": 24,
        "ip_version": 4,
        "default_gateway_address": "10.0.0.1",
        "friendly_name": "wan"
      }
    ],
    "labels": [
      {"href": "/orgs/2/labels/1"},
      {"href": "/orgs/2/labels/9"}
    ],
    "service_provider": "service provider",
    "data_center": "central data center",
    "os_id": "os id 0",
    "os_detail": "os detail 0",
    "online": true,
    "agent": {
      "config": {
        "enforcement_mode": "full",
        "visibility_level": "flow_summary"
      }
    }
 }

Curl Command to Create Unmanaged Workload Collection

This curl example illustrates how to create two workloads using the Workload Bulk Create API. This example assumes you have already created API keys for PCE authentication.

curl -i -X PUT https://pce.mycompany.com:8443/api/v2/orgs/2/workloads/bulk_create -H "Content-Type:application/json" -u $KEY:$TOKEN -d '[{"name":"web_app1","description": "workload desc 0","service_principal_name":"spn 0","hostname":"workload-0.example.com", "public_ip":"24.1.0.1","ephemeral":false,"external_data_set":"cmdb","external_data_reference": "0","interfaces":[{"name":"eth0","link_state":"up","address":"10.0.0.2","cidr_block":24, "ip_version":4,"default_gateway_address":"10.0.0.1","friendly_name":"wan"}],"labels":[{"href":"/orgs/2/labels/1"},{"href":"/orgs/2/labels/9"}],"service_provider":"service provider","data_center":"central data center","os_id":"os id 0","os_detail": "os detail 0","online":true,"agent":{"config":{"enforcementmode":"full", "visibility_level":"flow_summary"}}}]'

Upload Bulk Traffic Flows

NOTE:

Illumio recommends limiting bulk traffic operations to a maximum of 1,000 traffic flows per operation.

This method uses a POST command and requires a payload of multiple network traffic connections (source IP address, destination IP address, destination port, and protocol for each flow) passed as a single string of data in the CSV format.

URI for Uploading Bulk Traffic Flows

POST [api_version][org_href]/agents/bulk_traffic_flows

Header

When making this call, specify in the header that the data type is CSV and provide the CSV file format version. For example:

X-Bulk-Traffic-Load-CSV-Version:1

CSV Input Format

The traffic flow data that this method sent to the PCE in the CSV format must contain the following information:

Column Type Comment
src String (IP address) Must be a correctly-formatted IPv4 or IPv6 address. This IP address must map to an unmanaged workload in the PCE before you load the traffic data to the PCE.
dst String (IP address) Must be a correctly-formatted IPv4 or IPv6 address. This IP address must map to an unmanaged workload in the PCE before you load the traffic data to the PCE.
destination port Integer Destination port (port exposed by the destination IP address).
protocol Integer Protocol integer that must be converted to its Assigned Internet Protocol Numbers (for example., “tcp,” must be converted it to its IANA number 6) before you make the API call.
NOTE:

The network protocol for each connection must be translated to its assigned internet protocol number before you use this API to send the data. These protocol numbers can be found on the Internet Assigned Numbers Authority (IANA) website.

CSV Payload

For each network traffic connection, provide the source IP address, destination IP address, destination port, and network protocol (translated to its assigned internet protocol number). Each new connection entry must be specified by an \n to indicate a line break in the CSV file.

Example of a flows.csv file: 

"192.3.106.142,192.3.66.238,10123,6\n192.3.106.142,192.5.179.150,5723,6\n192.3.199.114,192.4.114.14,49241,6 \

192.4.113.239,192.4.113.240,1573,6\n192.4.205.21,192.1.189.138,13724,6\n192.4.205.21,192.1.189.138,1556,6 \

192.1.189.137,192.1.189.136,1563,6\n192.1.189.137,192.1.189.136,1563,6\n192.1.189.137,192.1.189.135,6200,6 \

192.1.189.137,192.1.189.136,1575,6\n"

Curl Commands to Upload Bulk Traffic Flows from CSV

If you have a pre-formatted CSV file, you can use the following curl command to upload the traffic flow data (where @/path/flows.csv represents the network path of the .csv file):

curl -i -X POST https://pce.illum.io:8443/api/v2/orgs/2/agents/bulk_traffic_flows -H "X-Bulk-Traffic-Load-CSV-Version: 1" -u $KEY:$TOKEN --data-binary "@/{path}/flows.csv"

This curl example shows two traffic connections being uploaded to the PCE:

curl -i -X POST https://mypce.mycompany.com:8443/api/v2/orgs/2/agents/bulk_traffic_flows -u $KEY:$TOKEN -H "X-Bulk-Traffic-Load-CSV-Version: 1" --data "10.0.0.1,10.0.0.2,23,6\n10.0.0.4,10.0.0.5,45,6"

This curl example shows a CSV file with traffic connections uploaded to the PCE:

curl -i -X POST https://devtest167.ilabs.io:8443/api/v2/orgs/1/agents/bulk_traffic_flows -T "bulk_upload.csv" -H "X-Bulk-Traffic-Load-CSV-Version: 1" -H "Content-Type: text/csv" -u $KEY:$TOKEN

Response Body

After successfully uploading traffic data with this API, an HTTP Status code of “201 Created” is returned along with a JSON object in the response body that contains a summary of the operation:

  • num_flows_received: Total number of flows sent to the PCE
  • num_flows_failed: Number of flows that failed to be uploaded
  • failed_flows: Array of CSV lines that failed to parse or match a Workload

Example:

{ 
    "num_flows_received": 10,
    "num_flows_failed": 0 
    "failed_flows": []
}