Explorer

Explorer queries are used to search and analize PCE traffic flows.

The Public Experimental APIs search and analyze PCE traffic flows for auditing, reporting, and troubleshooting. You can search for traffic flows between workloads or hosts, labeled workloads, or IP addresses, and you can restrict the search by specific port numbers and protocols.

There are two APIs for the traffic flows search:

Traffic Analysis Queries

This was the basic traffic analyzer for queries that is deprecated

Functionality HTTP URI

Search the PCE traffic data database to discover traffic patterns and write policy.

POST

[api_version][org_href]/traffic_flows/traffic_analysis_queries

NOTE: This API is now DERECATED and replaced with Asynchronous Queries for Traffic Flows, which has an additional parameter, query_name, and the max-results limit is raised from 100,000 to 200,000.

Request Parameters for POST

The following table describes the request parameters for POST [api_version][org_href]/traffic_flows/traffic_analysis_queries.

Property Description Type Req
start_date Starting date for the query. If left empty,
the default interpretation is “today,” which is
“now” minus 24 hours.		 Date-time string
(YYYY-MM-DDTHH:MM:SS)		 Yes
end_date Ending date for the query.
If left empty, the default interpretation is
“today,” which is “now” plus 24 hours. 		Data-time string
(YYYY-MM-DDTHH:MM:SS)		 Yes
sources

Source labels, workloads, or IP addresses
to include or exclude in the search.

The response can contain up to 50 matching
IP addresses.

The response returns sources as consumers.

Sub-properties: 

  • include: Targets that can be included
    are workloads, labels, or IP addresses identifiedby their HREF and structured
    as an array of JSON objects.
    If this property is left empty, then include
    means consider “ALL” or “ANY” of the
    object type.
  • exclude: Targets that can be excluded
    are workloads, labels, or IP addresses identified by their HREF and structured as a JSON array.
    • When IP List is present in the consumer
      part of a traffic query, traffic from
      workloads that belong to that IP List will
      not be returned by default.
      If the user wishes to see that traffic,
      they need to set exclude_workloads_
      from_ip_list_query: false
    • When IP List is present in the provider
      part of traffic query, traffic to workloads
      that belong to that IP List
      will not be returned by default.
      If the user wishes to see that traffic,
      they need to set
      exclude_workloads_from_ip_
      list_query: false
 Array of JSON objects

Yes

 

 

 

No

 
destinations

Target labels, workloads, or IP addresses to
include or exclude in the search.

The response returns targets as providers.

Required sub-properties: 

  • include: Targets that can be included
    are workloads, labels, or IP addresses identified by their HREF and
    structured as an array of JSON objects.
    If this property is left empty, then include
    means consider “ALL” or “ANY” of the
    object type.
  • exclude: Targets that can be excluded
    are workloads, labels, or IP addresses identified by their HREF
    and structured as a JSON array.
    If this property is left empty, then exclude means exclude “NONE” of the object types.
Array of JSON objects

Yes

 

 

 

 

Yes

Yes
services

Services (5-tuple of port/to_port/proto/process/service) to include or exclude

Required properties:

  • include: Destination ports that must be matched
    in the traffic flows to be returned. An empty value is interpreted as “match any or all port values.”
  • exclude: Traffic flows with ports in this
    set are excluded from the final result.
    An empty value is interpreted as
    “exclude no ports” from the search.

Additional properties:

  • port: Port Number (integer 0-65535).
    Also the starting port when specifying a range.
  • to_port: High end of port range inclusive
    if specifying a range.
    If not specifying a range then don't
    send this
  • proto: Protocol number
  • process_name
  • windows_services_name
 

Yes

Yes

 

 

 

 

No
policy_decisions

Allows you to filter the query based on
policy decision:

  • allowed: Allowed traffic.
  • potentially_blocked: Allowed but
    potentially blocked traffic.
    This type of traffic occurs when a
    workload VEN is in the test policy state.
  • blocked: Blocked traffic.
  • unknown
Array of strings Yes
max_results Maximum number of flows to return Integer No
exclude_workloads_from_ip_list_query Exclude workload traffic when IP List is
provided either in consumer or
provider part of the traffic query

Boolean

 Default is: true

Response Properties

Property Description Required/Type
src

Workloads (Workload), IP addresses (ip), or labels (label) that are consuming the service of the traffic flow returned in the response.

workload

ip

label

Required


String
String
JSON list of strings
dst

Workloads (workload), IP addresses (ip), or labels (label) that are providing the service of the traffic connection in the response.

workload

ip

label

Required

String

String

JSON list of strings
service

Port, protocol, process, service name and user_name of this flow

Additional required properties:

  • port: Destination port
  • proto :IANA protocol number
  • process_name: Process Name for this flow

  • windows_service_name: Windows Service Name for this flow

  • user_name: User Name for this flow

Required, Object

 

Integer

Integer

String

String

String
num_connections Number of traffic flows reported in the connection. Required, Integer
policy_decisions

If there is a rule in place for the returned traffic data, this property indicates the policy decision for the flow. Indicates if the traffic flow was allowed, potentially blocked (but allowed), or blocked.

  • allowed: Allowed traffic
  • potentially_blocked: Allowed but potentially blocked traffic
  • blocked: Blocked traffic
  • unknown

 Required, String
flow_direction Flow direction is inbound or outbound. Required, String
transmission Transmission type is broadcast or multicast. Not required, String
timestamp_range

The time range during which the flows were reported in date-time format. Includes both of these sub-properties: 

  • first_detected: The first time this flow was detected within the time range specified by the query
  • last_detected: The last time this flow was detected within the time range specified by the query
 Required, String
state

State of the flow, which can be one of the following values:

  • active
  • closed
  • timed-out
  • snapshot
  • new
  • incomplete
Not required, String
dst_bo Bytes sent till now by the destination over the flow during the interval Integer
dst_bi Bytes received till now by the destination over the flow during the interval Integer
icmp_type ICMP type for the flow Integer
icmp_code ICMP code for the flow Integer

Example Explorer Search

One preliminary method of creating policy is to make sure that different datacenter environments are segmented from each other, such as separating development and testing environments from production environments. Before writing policy rules to allow or block this traffic, determine if there are any traffic flows between them.

With Explorer, you can ask the PCE: 

Were there any traffic flows between my development and production environments, over any port besides port 80, excluding any Workloads that have a "Domain Controller" role label?

To construct this query, specify the following information: 

  • sources: In the JSON request, include the HREF of the “Development” environment label and exclude the HREF of the “Domain Controller” role label.
  • targets: In the JSON request, include the HREF of the “Production” environment label and exclude any workloads that do not have a role label assigned to them.
  • ports: Search for traffic over any port except port 80.

Request Body

            {
    "sources": {
        "include": [
            [
                {
                    "label": {
                        "href": "/orgs/45/labels/7",
                        /* HREF of an environment label named Development. */
                    }
                }
            ]
        ],
        "exclude": [
            {
                "label":{
                    "href": "/orgs/45/labels/45",          
                     /* HREF of an role label named Domain Controller. */
                }
            }
        ]
    },
    "destinations": {
        "include":[
            [
                {
                    "label": {
                        "href": "/orgs/45/labels/8",
                         /* HREF of an environment label named Production. */
                    }
                }
            ]
        ],
        "exclude": [
            {
                "label": {
                    "href": "/orgs/45/labels?key=role&exists=false"
                     /* Exclude any workloads that have no role labels defined. */
                }
            }
        ]
    },
    "services": {
        "include": [
            [
                {
                    "port": "21"
                    /* Include port number 21 for TCP traffic. */
                    },
                {
                    "proto": "6"
                    /* Include TCP number. */
                    }
            ]
        ],
         "exclude": [
            [
                {
                    "port": "5"
                    /* Exclude port number 5 for UDP traffic. */
                },
                {
                    "proto": "17"
                   /* Exclude UDP protocol identified by number. */
                 }
            ]
        ]
    },
      "policy_decision": [
                {
                    "allowed", "blocked"
                },
        ]
    },
}

Curl Command to Search Traffic Data with Explorer

curl -i -X POST https://pce.my-company.com:8443/api/v2/orgs/7/traffic_flows/traffic_analysis_queries -H "Content-Type: application/json" -u $KEY:$TOKEN -T explorer-example.json

The above command is listing contents of explorer-example.json to specify the query.

The example should be similar to the following:

explorer-example.json

            {
  "sources": {
    "include": [
      [
        {
          "label": {
            "href": "/orgs/1/labels/20"
          }
        }
      ]
    ],
    "exclude": [
      {
        "label": {
          "href": "/orgs/1/labels/21"
        }
      }
    ]
  },
  "destinations": {
    "include": [
      [
        {
          "label": {
            "href": "/orgs/1/labels/202"
          }
        }
      ]
    ],
    "exclude": [
      {
        "label": {
          "href": "/orgs/1/labels/205"
        }
      }
    ]
  },
  "services": {
    "include": [
      {
        "port": 80
      }
    ],
    "exclude": [
      {
        "port": 888
      }
    ]
  },
  "policy_decisions": ["allowed"], 
  "max_results": 2
}

Response

The response shows that there is a single traffic flow between your Development and Production Environment, the consumer of which is a workload identified by its IP address and labels, communicating with an unidentified IP address on TCP (number 6) over port 80. 

            {
	"src": {
		"ip": "10.2.3.161",
		"workload": {
			"href": "/orgs/1/workloads/3a5f2482-1188-4449-b035-0c7880486a4f",
			"hostname": "jorge-test-client5",
			"name": null,
			"os_type": linux,
			"labels": [{
				"href": "/orgs/1/labels/144",
				"key": "app",
				"value": "test_app_1"
			}, 
{
				"href": "/orgs/1/labels/143",
				"key": "role",
				"value": "test_access_1"
			}]
		}
	},
	"dst": {
		"ip": "10.2.3.165",
		"workload": {
			"href": "/orgs/1/workloads/87a10385-7466-488f-bce4-899de08974a0",
			"hostname": "jorge-test-server-4",
			"name": null,
			"os_type": linux,
			"labels": [{
				"href": "/orgs/1/labels/145",
				"key": "app",
				"value": "test_app_2"
			}, {
				"href": "/orgs/1/labels/151",
				"key": "env",
				"value": "test_env_2"
			}, {
				"href": "/orgs/1/labels/149",
				"key": "loc",
				"value": "test_place_1"
			}, {
				"href": "/orgs/1/labels/143",
				"key": "role",
				"value": "test_access_1"
			}]
		}
	},
	"port": 14000,
	"proto": 17,
	"num_connections": 2,
	"policy_decision": "allowed",
	"timestamp_range": {
		"last_detected": "2020-03-03T00:38:12Z",
		"first_detected": "2020-03-03T00:38:12Z"
	}
}, {
	"src": {
		"ip": "10.2.3.161",
		"workload": {
			"href": "/orgs/1/workloads/3a5f2482-1188-4449-b035-0c7880486a4f",
			"hostname": "jorge-test-client5",
			"name": null,
			"os_type": linux,
			"labels": [{
				"href": "/orgs/1/labels/144",
				"key": "app",
				"value": "test_app_1"
			.........
			{
				"href": "/orgs/1/labels/143",
				"key": "role",
				"value": "test_access_1"
			}]
		}
	},
	"dst": {
		"ip": "10.2.3.165",
		"workload": {
			"href": "/orgs/1/workloads/87a10385-7466-488f-bce4-899de08974a0",
			"hostname": "jorge-test-server-4",
			"name": null,
			"os_type": linux,
			"labels": [{
				"href": "/orgs/1/labels/145",
				"key": "app",
				"value": "test_app_2"
			..........
			{
  				"href": "/orgs/1/labels/143",
				"key": "role",
				"value": "test_access_1"
			}]
		}
	},
	"port": 14000,
	"proto": 6,
	"num_connections": 6,
	"policy_decision": "allowed",
	"timestamp_range": {
		"last_detected": "2020-03-03T00:38:10Z",
		"first_detected": "2020-03-03T00:38:10Z"
	}
}

Asynchronous Queries for Traffic Flows

NOTE:This new API is very similar to the existing one,Traffic Analysis Queries, only that the new one has a new parameter, query_name, while the max-results limit is raised to 200,000.
Currently there is no change to POST[api_version][org_href]/traffic_flows/traffic_analysis_queries,which will eventually be deprecated.

The maximum of returned results in POST [api_version][org_href]/traffic_flows/traffic_analysis_queries is100,000, which is a reasonable number a user can view in the UI. However, when Explorer is used for capturing all traffic flows into a CSV file to build rules offline, the queries take longer to return, traffic data contains more than 100,000 rows, and so on.

Explorer queries are required to support both the single-node and multi-node Explorer in the SuperCluster environment. Therefore, limitation of 100,000 results was raised to 200,000 to better support SuperCluster environments in Explorer.

Async Queries API Methods

Functionality HTTP URI
Create a new async traffic query

POST

[api_version][org_href]traffic_flows_async_queries

Get a collection of async traffic queries

GET

[api_version][org_href]traffic_flows_async_queries

Download the completed async traffic query results GET api_version][org_href]traffic_flows_async_queries/:uuid/dowload
Update an async traffic query (request cancellation of the queued async query)

PUT

[api_version][org_href]traffic_flows_async_queries/:uuid

Delete the completed async traffic query DELETE [api_version][org_href]traffic_flows_async_queries/:uuid

Properties for Async Queries

NOTE:This new API is very similar to the existing one,Traffic Analysis Queries, only that the new one has a new parameter, query_name, and the max-results limit is raised to 200,000.
Property Description Type Required
query_name (POST) Name of the query. String Yes
sources (POST) Source labels, workloads, IP addresses to include or exclude:.
Required parameters:
include, exclude
 Object Yes
destinations (POST)Target labels, workloads, IP addresses, domain names, transmission to include or exclude.
Required parameters:
include, exclude		   Yes
services (POST)Services (5-tuple of port/to_port/proto/process/service) to include or exclude.
Required parameters:

include, exclude		 Object Yes
policy_decisions (POST)List of policy decisions. Available types are:
allowed

potentially_blocked

blocked

unknown

 Array Yes
max_results

(POST) Maximum number of flows to return.
"minimum": 0,

"maximum": 200000

 Integer No
status (PUT, GET) Query status String Yes
href (GET) Query URI String Yes
created_at

(GET) Timestamp in UTC when this query was created


 date/time Yes
created_by (GET) User who created this query

String

Format URI

 Yes
updated_at (GET) Timestamp in UTC when this async query was last updated date/time No
generated_at (GET) Async query generation timestamp in UTC date/time No
       
query_parameters (GET) Explorer query parameters Object Yes
matches_count (GET) Query result count Integer No
flows_count (GET) Result count after query limits and RBAC filtering are applied Integer No
reqions

(GET) Region-specific response metadata.

Required properties: 

  • pce_fqdn

  • responded

This section is present only in supercluster leader queries.

 Object No

Example Async Explorer Queries

Curl command for traffic_flows_async_queries_post

curl -i -u api_1195cf055cf8a834c:148afd87ecc980900eaf10d6c54e6c0f607b22e0dbf768dd007e51e731096282 https://devtest0.ilabs.io:8443/api/v2/orgs/1/traffic_flows/async_queries -H "Content-Type: application/json" -X POST -d

Response:

'{"sources":{"include":[[{"workload":{"href":"/orgs/1/workloads/a3ffb374-f6c6-4cce-ac57-642c66f1498f"}}]],"exclude":[]},"destinations":{"include":[[]],"exclude":[]},"services":{"include":[],"exclude":[]},"sources_destinations_query_op":"and","start_date":"2016-01-29T17:04:03.149Z","end_date":"2021-01-29T17:06:03.151Z","policy_decisions":[],"max_results":1000,"query_name":"worklaod test"}'  
	HTTP/1.1 202 Accepted
	content-location: 7734501b-74a2-47a4-9ded-77bf4ceea938
	content-type: application/json
	content-length: 615
	x-request-id: 00c8fa00-dbd8-4a28-a5c7-354fb5ae3886
	cache-control: no-store
	x-frame-options: DENY
	x-xss-protection: 1; mode=block
	x-content-type-options: nosniff
{"status":"queued","href":"/orgs/1/traffic_flows/async_queries/7734501b-74a2-47a4-9ded-77bf4ceea938","created_by":{"href":"/users/1"},"query_parameters":{"sources":{"include":[[{"workload":{"href":"/orgs/1/workloads/a3ffb374-f6c6-4cce-ac57-642c66f1498f"}}]],"exclude":[]},"destinations":{"include":[[]],"exclude":[]},"services":{"include":[],"exclude":[]},"sources_destinations_query_op":"and","start_date":"2016-01-29T17:04:03.149Z","end_date":"2021-01-29T17:06:03.151Z","policy_decisions":[],"max_results":1000,"query_name":"worklaod test"},"created_at":"2021-04-09T20:50:30Z","updated_at":"2021-04-09T20:50:30Z"}

Curl command for for traffic_flows/async_queries_get

This query gets the collection of all async jobs for the current user, including anything that was already submitted.

curl -i -u api_1195cf055cf8a834c:148afd87ecc980900eaf10d6c54e6c0f607b22e0dbf768dd007e51e731096282 https://devtest0.ilabs.io:8443/api/v2/orgs/1/traffic_flows/async_queries

Response

HTTP/1.1 200 OK
	content-type: application/json
	content-length: 1510
	x-request-id: fcf065e5-e465-4161-ba98-542182734c38
	cache-control: no-store
	x-frame-options: DENY
	x-xss-protection: 1; mode=block
	x-content-type-options: nosniff
[{"matches_count":1984,"flows_count":1000,"status":"completed","href":"/orgs/1/traffic_flows/async_queries/88675fbd-a88e-44bd-b358-2d6f2fc4f95a","result":"/orgs/1/traffic_flows/async_queries/88675fbd-a88e-44bd-b358-2d6f2fc4f95a/download","created_by":{"href":"/users/1"},"query_parameters":{"sources":{"include":[[{"workload":{"href":"/orgs/1/workloads/a3ffb374-f6c6-4cce-ac57-642c66f1498f"}}]],"exclude":[]},"destinations":{"include":[[]],"exclude":[]},"services":{"include":[],"exclude":[]},"sources_destinations_query_op":"and","start_date":"2016-01-29T17:04:03.149Z","end_date":"2021-01-29T17:06:03.151Z","policy_decisions":[],"max_results":1000,"query_name":"worklaod tesrrrrrt"},"created_at":"2021-04-09T20:50:19Z","updated_at":"2021-04-09T20:50:27Z"},{"matches_count":1984,"flows_count":1000,"status":"completed","href":"/orgs/1/traffic_flows/async_queries/7734501b-74a2-47a4-9ded-77bf4ceea938","result":"/orgs/1/traffic_flows/async_queries/7734501b-74a2-47a4-9ded-77bf4ceea938/download","created_by":{"href":"/users/1"},"query_parameters":{"sources":{"include":[[{"workload":{"href":"/orgs/1/workloads/a3ffb374-f6c6-4cce-ac57-642c66f1498f"}}]],"exclude":[]},"destinations":{"include":[[]],"exclude":[]},"services":{"include":[],"exclude":[]},"sources_destinations_query_op":"and","start_date":"2016-01-29T17:04:03.149Z","end_date":"2021-01-29T17:06:03.151Z","policy_decisions":[],"max_results":1000,"query_name":"worklaod test"},"created_at":"2021-04-09T20:50:30Z","updated_at":"2021-04-09T20:50:32Z"

Curl command for traffic_flows/async_queries/:uuid_get

This query gets a specific job included in the collection.

curl -i -u api_1195cf055cf8a834c:148afd87ecc980900eaf10d6c54e6c0f607b22e0dbf768dd007e51e731096282 https://devtest0.ilabs.io:8443/api/v2/orgs/1/traffic_flows/async_queries/88675fbd-a88e-44bd-b358-2d6f2fc4f95a

Response

HTTP/1.1 200 OK
	content-type: application/json
	content-length: 756
	x-request-id: f328b845-8542-4b96-a128-43aefdf7ba5a
	cache-control: no-store
	x-frame-options: DENY
	x-xss-protection: 1; mode=block
	x-content-type-options: nosniff
{"matches_count":1984,"flows_count":1000,"status":"completed",
"href":"/orgs/1/traffic_flows/async_queries/88675fbd-a88e-44bd-b358-2d6f2fc4f95a",
"result":"/orgs/1/traffic_flows/async_queries/88675fbd-a88e-44bd-b358-2d6f2fc4f95a/download",
"created_by":{"href":"/users/1"},"query_parameters":{"sources":{"include":[[{"workload":{"href":"/orgs/1/workloads/a3ffb374-f6c6-4cce-ac57-642c66f1498f"}}]],"exclude":[]},"destinations":{"include":[[]],"exclude":[]},"services":{"include":[],"exclude":[]},"sources_destinations_query_op":"and","start_date":"2016-01-29T17:04:03.149Z","end_date":"2021-01-29T17:06:03.151Z","policy_decisions":[],"max_results":1000,"query_name":"worklaod tesrrrrrt"},"created_at":"2021-04-09T20:50:19Z","updated_at":"2021-04-09T20:50:27Z"}