Explorer

The Public Experimental Explorer APIs search and analyze PCE traffic flows for auditing, reporting, and troubleshooting. You can search for traffic flows between workloads or hosts, labeled workloads, or IP addresses, and you can restrict the search by specific port numbers and protocols.

There are three APIs for the traffic flows search:

Traffic Analysis Queries

This was the basic traffic analyzer for queries that is now deprecated:

Functionality HTTP URI

Search the PCE traffic data database to discover traffic patterns and write policy.

POST

[api_version][org_href]/traffic_flows/traffic_analysis_queries

NOTE: This API is now DERECATED and replaced with Asynchronous Queries for Traffic Flows, which has an additional parameter, query_name, and the max-results limit is raised from 100,000 to 200,000.

Create Traffic Analysis Queries 

POST [api_version][org_href]/traffic_flows/traffic_analysis_queries

Properties

Property Description Type Req
start_date Starting date for the query. If left empty,
the default interpretation is “today,” which is
“now” minus 24 hours.		 Date-time string
(YYYY-MM-DDTHH:MM:SS)		 Yes
end_date Ending date for the query.
If left empty, the default interpretation is
“today,” which is “now” plus 24 hours. 		Data-time string
(YYYY-MM-DDTHH:MM:SS)		 Yes
sources

Source labels, workloads, or IP addresses
to include or exclude in the search.

The response can contain up to five matching IP addresses.

NOTE:  The response returns sources as consumers.
Sources are treated as consumers for the purposes of the request; the response returns the source of an individual flow as src.

Sub-properties: 

  • include: Targets that can be included
    are workloads, labels, or IP addresses identified by their HREF and structured as an array of JSON objects.
    If this property is left empty, then include means consider “ALL” or “ANY” of the object type.
  • exclude: Targets that can be excluded
    are workloads, labels, or IP addresses identified by their HREF and structured as a JSON array.
    • When IP List is present in the consumer
      part of a traffic query, traffic from
      workloads that belong to that IP List will not be returned by default.
      If users want to see that traffic, they need to set exclude_workloads_
      from_ip_list_query: false
    • When IP List is present in the provider part of traffic query, traffic to workloads that belong to that IP List will not be returned by default.
      If the user wishes to see that traffic,
      they need to set
      exclude_workloads_from_ip_
      list_query: false
 Object

 

 

 

 

 

 






Yes




Yes
destinations

Target labels, workloads, or IP addresses to include or exclude in the search.

The response returns targets as providers.

Required sub-properties: 

  • include: Targets that can be included
    are workloads, labels, or IP addresses identified by their HREF and structured as an array of JSON objects.
    If this property is left empty, then include
    means consider “ALL” or “ANY” of the
    object type.
  • exclude: Targets that can be excluded
    are workloads, labels, or IP addresses identified by their HREF and structured as a JSON array.
    If this property is left empty, then exclude means exclude “NONE” of the object types.
Object

 

 

 

 

Yes





Yes
services

Services (5-tuple of port/to_port/proto/process/service) to include or exclude. Not all properties of the service subobjects are required.

Required properties:

  • include: Destination ports that must be matched in the traffic flows to be returned. An empty value is interpreted as “match any or all port values.”
  • exclude: Traffic flows with ports in this
    set are excluded from the final result.
    An empty value is interpreted as
    “exclude no ports” from the search.

Properties of the includeand exclude subobjects:

  • port: Port Number (integer 0-65535).
    Also the starting port when specifying a range.
  • to_port: High end of port range inclusive
    if specifying a range.
    If not specifying a range then don't send this:
  • proto: Protocol number. For the expected proto values
    see IANA Protocol Numbers.
  • process_name
  • windows_service_name
 

 

 

 

 

Yes


Yes
policy_decisions

Allows you to filter the query based on
policy decision:

  • allowed: Allowed traffic.
  • potentially_blocked: Allowed but
    potentially blocked traffic.
    This type of traffic occurs when a
    workload VEN is in the test policy state.
  • blocked: Blocked traffic.
  • unknown
Array of strings Yes
max_results Maximum number of flows to return Integer No
exclude_workloads_from_ip_list_query Exclude workload traffic when IP List is provided either in consumer or provider part of the traffic query

Boolean

 Default is: true

Response Properties

Property Description Type Required
src

Workloads (Workload), IP addresses (ip), or labels (label) that are consuming the service of the traffic flow returned in the response.

workload

ip

label



 

String
String
JSON list of strings

 Yes
dst

Workloads (workload), IP addresses (ip), or labels (label) that are providing the service of the traffic connection in the response.

workload

ip

label



 

String

String

JSON list of strings

 Yes
service

Port, protocol, process, service name and user_name of this flow

Additional required properties:

  • port: Destination port
  • proto :IANA protocol number
  • process_name: Process Name for this flow

  • windows_service_name: Windows Service Name for this flow

  • user_name: User Name for this flow

Object

 

 

Integer

Integer

String

String

String

 Yes
num_connections Number of traffic flows reported in the connection. Integer Yes
policy_decisions

If there is a rule in place for the returned traffic data, this property indicates the policy decision for the flow. Indicates if the traffic flow was allowed, potentially blocked (but allowed), or blocked.

  • allowed: Allowed traffic
  • potentially_blocked: Allowed but potentially blocked traffic
  • blocked: Blocked traffic
  • unknown

 String Yes
flow_direction Flow direction is inbound or outbound. String Yes
transmission Transmission type is broadcast or multicast. String No
timestamp_range

The time range during which the flows were reported in date-time format. Includes both of these sub-properties: 

  • first_detected: The first time this flow was detected within the time range specified by the query
  • last_detected: The last time this flow was detected within the time range specified by the query.
 String Yes
state

State of the flow, which can be one of the following values:

  • active
  • closed
  • timed-out
  • snapshot
  • new
  • incomplete
String No
dst_bo Bytes sent till now by the destination over the flow during the interval. Integer  
dst_bi Bytes received till now by the destination over the flow during the interval. Integer  
icmp_type ICMP type for the flow Integer  
icmp_code ICMP code for the flow Integer  
network

PCE network on which this flow was observed.
Additional properties:

  • name (network name)

  • href (network href)

Object

 

 

String

String

 Yes

Filter for Managed Services

This API allows you to filter allmanaged services, such as workloads, virtual services, and so on.

Functionality HTTP URI
Get a list of Virtual Servers GET [api_version][org_href]/sec_policy/:version/virtual_servers
Get a specified Virtual Server GET [api_version][org_href]/sec_policy/:version/virtual_servers/:uuid

 

Example Explorer Search

One preliminary method of creating policy is to make sure that different datacenter environments are segmented from each other, such as separating development and testing environments from production environments. Before writing policy rules to allow or block this traffic, determine if there are any traffic flows between them.

With Explorer, you can ask the PCE: 

Were there any traffic flows between my development and production environments, over any port besides port 80, excluding any Workloads that have a "Domain Controller" role label?

To construct this query, specify the following information: 

  • sources: In the JSON request, include the HREF of the “Development” environment label and exclude the HREF of the “Domain Controller” role label.
  • targets: In the JSON request, include the HREF of the “Production” environment label and exclude any workloads that do not have a role label assigned to them.
  • ports: Search for traffic over any port except port 80.

Curl Command to Search Traffic Data with Explorer

curl -i -X POST https://pce.my-company.com:8443/api/v2/orgs/7/traffic_flows/traffic_analysis_queries -H "Content-Type: application/json" -u $KEY:$TOKEN -T explorer-example.json

The above command is listing contents of explorer-example.json to specify the query.

The example should be similar to the following:

explorer-example.json

{
  "sources": {
    "include": [
      [
        {
          "label": {
            "href": "/orgs/1/labels/20"
          }
        }
      ]
    ],
    "exclude": [
      {
        "label": {
          "href": "/orgs/1/labels/21"
        }
      }
    ]
  },
  "destinations": {
    "include": [
      [
        {
          "label": {
            "href": "/orgs/1/labels/202"
          }
        }
      ]
    ],
    "exclude": [
      {
        "label": {
          "href": "/orgs/1/labels/205"
        }
      }
    ]
  },
  "services": {
    "include": [
      {
        "port": 80
      }
    ],
    "exclude": [
      {
        "port": 888
      }
    ]
  },
  "policy_decisions": ["allowed"], 
  "max_results": 2
}

Response

The response shows that there is a single traffic flow between your Development and Production Environment, the consumer of which is a workload identified by its IP address and labels, communicating with an unidentified IP address on TCP (number 6) over port 80. 

 {
	"src": {
	    "ip": "10.2.3.161",
            "workload": {
	        "href": "/orgs/1/workloads/3a5f2482-1188-4449-b035-0c7880486a4f",
		"hostname": "johns-test-client5",
		"name": null,
		"os_type": linux,
		 "labels": [{
		    "href": "/orgs/1/labels/144",
		     "key": "app",
		     "value": "test_app_1"  
        }, 
        {
		  "href": "/orgs/1/labels/143",
			"key": "role",
			"value": "test_access_1"
		}]
	    }
	},
	"dst": {
		"ip": "10.2.3.165",
		"workload": {
		    "href": "/orgs/1/workloads/87a10385-7466-488f-bce4-899de08974a0",
		    "hostname": "johns-test-server-4",
		    "name": null,
	            "os_type": linux,
		    "labels": [{
			"href": "/orgs/1/labels/145",
			"key": "app",
			"value": "test_app_2"
		}, {
			"href": "/orgs/1/labels/151",
			"key": "env",
			"value": "test_env_2"
		}, {
			"href": "/orgs/1/labels/149",
			"key": "loc",
			"value": "test_place_1"
		}, {
			"href": "/orgs/1/labels/143",
			"key": "role",
			"value": "test_access_1"
		}]
	}
    },
	"port": 14000,
	"proto": 17,
	"num_connections": 2,
	"policy_decision": "allowed",
	"timestamp_range": {
		"last_detected": "2020-03-03T00:38:12Z",
		"first_detected": "2020-03-03T00:38:12Z"
	}
}, {
	"src": {
	    "ip": "10.2.3.161",
	    "workload": {
		    "href": "/orgs/1/workloads/3a5f2482-1188-4449-b035-0c7880486a4f",
		    "hostname": "johns-test-client5",
		    "name": null,
		    "os_type": linux,
		    "labels": [{
			"href": "/orgs/1/labels/144",
			"key": "app",
			"value": "test_app_1"
			.........
		{
			"href": "/orgs/1/labels/143",
			"key": "role",
			"value": "test_access_1"
		}]
	}
    },
	"dst": {
		"ip": "10.2.3.165",
		"workload": {
		    "href": "/orgs/1/workloads/87a10385-7466-488f-bce4-899de08974a0",
		    "hostname": "johns-test-server-4",
		    "name": null,
		    "os_type": linux,
		    "labels": [{
			"href": "/orgs/1/labels/145",
			"key": "app",
			"value": "test_app_2"
			..........
			{
  			"href": "/orgs/1/labels/143",
			"key": "role",
			"value": "test_access_1"
			}]
		}
	},
	"port": 14000,
	"proto": 6,
	"num_connections": 6,
	"policy_decision": "allowed",
	"timestamp_range": {
		"last_detected": "2020-03-03T00:38:10Z",
		"first_detected": "2020-03-03T00:38:10Z"
	}
}

Asynchronous Queries for Traffic Flows

This new API is very similar to the existing one,Traffic Analysis Queries, only that the new one has a new parameter, query_name, while the max_results limit is raised to 200,000. Currently there is no change to POST[api_version][org_href]/traffic_flows/traffic_analysis_queries,which will eventually be deprecated.

The maximum of returned results in POST [api_version][org_href]/traffic_flows/traffic_analysis_queries is100,000, which is a reasonable number a user can view in the UI. However, when Explorer is used for capturing all traffic flows into a CSV file to build rules offline, the queries take longer to return, traffic data contains more than 100,000 rows, and so on.

Explorer queries are required to support both the single-node and multi-node Explorer in the SuperCluster environment. Therefore, limitation of 100,000 results was raised to 200,000 to better support SuperCluster environments in Explorer.

Async Queries API Methods

Functionality HTTP URI
Create a new async traffic query

POST

[api_version][org_href]traffic_flows/async_queries

Get a collection of async traffic queries

GET

[api_version][org_href]traffic_flows/async_queries

Download the completed async traffic query results GET api_version][org_href]traffic_flows_async/queries/:uuid/dowload
Update an async traffic query (request cancellation of the queued async query)

PUT

[api_version][org_href]traffic_flows/async_queries/:uuid

Delete the completed async traffic query DELETE [api_version][org_href]traffic_flows/async_queries/:uuid
NOTE:This new API is very similar to the existing one,Traffic Analysis Queries, only that the new one has a new parameter, query_name, and the max-results limit is raised to 200,000.

Parameters for GET [api_version][org_href] traffic_flows/async_queries

Property Description Type Required
org_id Organization Integer Yes

Parameters for POST [api_version][org_href] traffic_flows/async_queries

Property Description Type Required
org_id Organization Integer Yes
query_name Name of the query. String Yes
sources Source labels, workloads, IP addresses to include or exclude.
 Object Yes
services Services (5-tuple of port/to_port/proto/process/service) to include or exclude.
 Object Yes
max_results

Maximum number of flows to return.
"minimum": 0,

"maximum": 200000

 Integer No
status Query status String Yes
start_date

Starting date for query


 String/date No
end_date Ending date for query

String/date

 No
ip_list   Object Yes
ip_address IP address value String Yes
boundary_decisions List of boundary decisions String  

Example Async Explorer Queries

Curl command for traffic_flows_async_queries_post

curl -i -u api_1195cf055cf8a834c:148afd87ecc980900eaf10d6c54e6c0f607b22e0dbf768dd007e51e731096282 https://devtest0.ilabs.io:8443/api/v2/orgs/1/traffic_flows/async_queries -H "Content-Type: application/json" -X POST -d '{"sources":{"include":[[{"workload":{"href":"/orgs/1/workloads/a3ffb374-f6c6-4cce-ac57-642c66f1498f"}}]],"exclude":[]},"destinations":{"include":[[]],"exclude":[]},"services":{"include":[],"exclude":[]},"sources_destinations_query_op":"and","start_date":"2016-01-29T17:04:03.149Z","end_date":"2021-01-29T17:06:03.151Z","policy_decisions":[],"max_results":1000,"query_name":"worklaod test"}'

Response:

 HTTP/1.1 202 Accepted
	content-location: 7734501b-74a2-47a4-9ded-77bf4ceea938
	content-type: application/json
	content-length: 615
	x-request-id: 00c8fa00-dbd8-4a28-a5c7-354fb5ae3886
	cache-control: no-store
	x-frame-options: DENY
	x-xss-protection: 1; mode=block
	x-content-type-options: nosniff
{"status":"queued","href":"/orgs/1/traffic_flows/async_queries/7734501b-74a2-47a4-9ded-77bf4ceea938","created_by":{"href":"/users/1"},"query_parameters":{"sources":{"include":[[{"workload":{"href":"/orgs/1/workloads/a3ffb374-f6c6-4cce-ac57-642c66f1498f"}}]],"exclude":[]},"destinations":{"include":[[]],"exclude":[]},"services":{"include":[],"exclude":[]},"sources_destinations_query_op":"and","start_date":"2016-01-29T17:04:03.149Z","end_date":"2021-01-29T17:06:03.151Z","policy_decisions":[],"max_results":1000,"query_name":"worklaod test"},"created_at":"2021-04-09T20:50:30Z","updated_at":"2021-04-09T20:50:30Z"}

Curl command for traffic_flows/async_queries_get

This query gets the collection of all async jobs for the current user, including anything that was already submitted.

curl -i -u api_1195cf055cf8a834c:148afd87ecc980900eaf10d6c54e6c0f607b22e0dbf768dd007e51e731096282 https://devtest0.ilabs.io:8443/api/v2/orgs/1/traffic_flows/async_queries

Response

HTTP/1.1 200 OK
	content-type: application/json
	content-length: 1510
	x-request-id: fcf065e5-e465-4161-ba98-542182734c38
	cache-control: no-store
	x-frame-options: DENY
	x-xss-protection: 1; mode=block
	x-content-type-options: nosniff
[{"matches_count":1984,"flows_count":1000,"status":"completed","href":"/orgs/1/traffic_flows/async_queries/88675fbd-a88e-44bd-b358-2d6f2fc4f95a","result":"/orgs/1/traffic_flows/async_queries/88675fbd-a88e-44bd-b358-2d6f2fc4f95a/download","created_by":{"href":"/users/1"},"query_parameters":{"sources":{"include":[[{"workload":{"href":"/orgs/1/workloads/a3ffb374-f6c6-4cce-ac57-642c66f1498f"}}]],"exclude":[]},"destinations":{"include":[[]],"exclude":[]},"services":{"include":[],"exclude":[]},"sources_destinations_query_op":"and","start_date":"2016-01-29T17:04:03.149Z","end_date":"2021-01-29T17:06:03.151Z","policy_decisions":[],"max_results":1000,"query_name":"worklaod tesrrrrrt"},"created_at":"2021-04-09T20:50:19Z","updated_at":"2021-04-09T20:50:27Z"},{"matches_count":1984,"flows_count":1000,"status":"completed","href":"/orgs/1/traffic_flows/async_queries/7734501b-74a2-47a4-9ded-77bf4ceea938","result":"/orgs/1/traffic_flows/async_queries/7734501b-74a2-47a4-9ded-77bf4ceea938/download","created_by":{"href":"/users/1"},"query_parameters":{"sources":{"include":[[{"workload":{"href":"/orgs/1/workloads/a3ffb374-f6c6-4cce-ac57-642c66f1498f"}}]],"exclude":[]},"destinations":{"include":[[]],"exclude":[]},"services":{"include":[],"exclude":[]},"sources_destinations_query_op":"and","start_date":"2016-01-29T17:04:03.149Z","end_date":"2021-01-29T17:06:03.151Z","policy_decisions":[],"max_results":1000,"query_name":"worklaod test"},"created_at":"2021-04-09T20:50:30Z","updated_at":"2021-04-09T20:50:32Z"

Curl command for traffic_flows/async_queries/:uuid_get

This query gets a specific job included in the collection.

curl -i -u $KEY:$TOKEN https://devtest0.ilabs.io:8443/api/v2/orgs/1/traffic_flows/async_queries/88675fbd-a88e-44bd-b358-2d6f2fc4f95a

Response

HTTP/1.1 200 OK
	content-type: application/json
	content-length: 756
	x-request-id: f328b845-8542-4b96-a128-43aefdf7ba5a
	cache-control: no-store
	x-frame-options: DENY
	x-xss-protection: 1; mode=block
	x-content-type-options: nosniff
{"matches_count":1984,"flows_count":1000,"status":"completed",
"href":"/orgs/1/traffic_flows/async_queries/88675fbd-a88e-44bd-b358-2d6f2fc4f95a",
"result":"/orgs/1/traffic_flows/async_queries/88675fbd-a88e-44bd-b358-2d6f2fc4f95a/download",
"created_by":{"href":"/users/1"},"query_parameters":{"sources":{"include":[[{"workload":{"href":"/orgs/1/workloads/a3ffb374-f6c6-4cce-ac57-642c66f1498f"}}]],"exclude":[]},"destinations":{"include":[[]],"exclude":[]},"services":{"include":[],"exclude":[]},"sources_destinations_query_op":"and","start_date":"2016-01-29T17:04:03.149Z","end_date":"2021-01-29T17:06:03.151Z","policy_decisions":[],"max_results":1000,"query_name":"worklaod tesrrrrrt"},"created_at":"2021-04-09T20:50:19Z","updated_at":"2021-04-09T20:50:27Z"}

Asynchronous Queries

Explorer provides a view to query traffic data to help users build segmentation / intra / extra scope rules. Previously, it caped the maximum of returned results at 100,000, which is enough as a number of results a user can view in the UI. However, when you use Explorer for capturing all traffic flows into a CSV file to build rules offline, the queries take longer to return, traffic data contains more than 100,000 rows, and so on.

To better support SuperCluster environments in Explorer, limitation of 100,000 results requires a new approach in using Explorer queries. Explorer queries are required to support both the single-node and multi-node Explorer in the SuperCluster environment.

New APIs for asynchronous queries are introduced while there is no change to the existing APIs, which will be later deprecated.

Async Queries API Methods

Functionality HTTP URI
Create a new async traffic query

POST

[api_version][org_href]traffic_flows_async_queries

Get a collection of async traffic queries

GET

[api_version][org_href]traffic_flows_async_queries

Get an async traffic query. Asynchronous explorer query status. GET [api_version][org_href]traffic_flows_async_query/:uuid
Update an async traffic query (request cancellation of the queued async query)

PUT

[api_version][org_href]traffic_flows_async_queries/:uuid

Delete the completed async traffic query DELETE [api_version][org_href]traffic_flows_async_queries/:uuid

Properties

Property Description Type Required
href (GET) Query URI String Yes
status (GET) Current query status String Yes
creted_at Timestamp in UTC when this query was created date/time Yes
created_by User who created this query

String

Format URI

 Yes
query_parameters Explorer query parameters Object Yes
query_name (GET, POST) Name of the query. String Yes
sources

(GET, POST) Source labels, workloads, or IP addresses to include or exclude in the search.

Sub-properties: 

  • include: Targets that can be included
  • exclude: Targets that
Object

Yes

 

 

 

 
destinations

(GET, POST)Target labels, wo rkloads, IP addresses, domain names, transmission to include or exclude

   Yes
services (GET, POST) Services (5-tuple of port/to_port/proto/process/service) to include or exclude Object Yes
policy_decisions (GET, POST) List of policy decisions

Array

 Yes
result (GET) Result download URI, availble only if status is completed String No
updated_at Timestamp in UTC when this async query was last updated date/time No
start_date (GET, POST) Starting date for the query Data-time string (YYYY-MM-DDTHH:MM:SS) No
end_date (GET, POST) Ending date for the query. Data-time string (YYYY-MM-DDTHH:MM:SS) No
matches_count Query result count Integer No
flows_count Result count after query limits and RBAC filtering are applied Integer No
sources_destinations_query_op

(GET, POST) Query logical operator between sources and destinations.

Default is "and"

 String

No

 

 

 
regions Region-specific response metadata. Required properties:
pce_fqdn
responded		 Integer No
max_results

(GET, POST) Maximum number of flows to return.
"minimum": 0,

"maximum": 200000

 Integer No
exclude_workloads_from_ip_list_query (GET, POST) Exclude workload traffic when IP List is provided either in consumer or provider part of traffic query. Default is TRUE Boolean No
boundary_decisions List of boundary decisions String  

Database Metrics

This Public Experimental endpoint provides you the organization-specific insight into the current traffic database. It gives you ability to monitor how big the traffic database is and how much data you can store. It also gives information about how many days of data is available.

Database Metrics API Method

Functionality HTTP URI
Returns the organization database usage metrics. Provides to customers organization-specific insight into current traffic database size (#days, #GB).

GET

[api_version][org_href]traffic_flows/database_metrics

Parameters for Database Metrics

Property Description Type Required
flows_days Organization's total number of days of flow data Integer Yes
flows_days_limit Organization's limit on the total number of days of flow data Integer Yes
flows_oldest_day Organization's oldest day of flow data (yyyy-mm-dd) String, date No
flows_size_gb Organization's limit on the total number of gigabytes of flow data
 Number Yes
flows_size_gb_limit Organization's limit on the total number of gigabytes of flow data
 Number Yes
updated_at Timestamp in UTC when these flow metrics were generated String,
date/time		 Yes

Curl Command to get Database Metrics

curl -i -u  api_1f0c3ebf44dcafb19:6de325a25e7e4383de1dea9d8c7ce230773ad4d8a5cbb62980e7d176254f263d https://test.mylab.io:8443/api/v2/orgs/1/traffic_flows/database_metrics

Response

{"flows_days":2,"flows_size_gb":0.2695770263671875,"flows_days_limit":90,"flows_size_gb_limit":26,"updated_at":"2021-11-17T20:50:45Z"}%