Collect Flow Records from F5

The example listed in the following steps uses a virtual edition of the F5 BIG-IP appliance in AWS and the Illumio FlowLink application to gather and parse flow data.

Requirements

• FlowLink (flow collector)
• F5 BIG-IP system with LTM
• A virtual server configured on F5 box

Create a Pool for Flow Collector

To create a pool of flow collectors to receive the flow record messages from the F5 system:

1. In the F5 UI, click Main > Local Traffic > Pools > Pool Lists > Create.
2. Enter a unique name in the Name field, which represents the flow collector.
3. A Health Monitor is not required. If you want to see if the F5 system can reach the flow collector, select gateway_icmp and move it to the Active box.
4. In the New Member section, configure the collector IP address.
5. Click Add.

If you are using IPFIX, use the following configuration:

Field	Value
Node Name	Enter the Collector IP address
Service Port	4739

If you are using NetFlow, use the following configuration:

Field	Value
Node Name	Enter the Collector IP address
Service Port	2055



6. Click Finished.

The below example shows two (2) different nodes configured in one pool. Both nodes have the IP address. However, one is for IPFIX and one is for NetFlow. Even though F5 allows two nodes in the pool, it is recommended to only have one node enabled (either 2055 or 4739).

Example with NetFlow enabled and IPFIX disabled:

Create a Log Destination

To create a log destination to stream the logs in either IPFIX or NetFlow V9 format to the Pool:

1. In the F5 UI, click Main > System > Logs > Configuration > Log Destinations > Create.
2. Enter a unique name in the Name field, which represents the flow collector.
3. In the Type field, select IPFIX.
4. Configure the IPFIX Settings.

If you are using IPFIX, use the following configuration:

Field	Value
Protocol	Select IPFIX
Pool Name	Select the pool created earlier
Transport Profile	UDP

If you are using NetFlow, use the following configuration:

Field	Value
Protocol	Select NetFlow V9
Pool Name	Select the pool created earlier
Transport Profile	UDP

5. Click Finished.

Example of a Log Destination configuration with NetFlow:

Create a Log Publisher

To create a log publisher to send logs to the specified log destination:

1. In the F5 UI, click Main > System > Logs > Configuration > Log Publishers > Create.
2. Enter a unique name in the Name field, which represents the flow collector.
3. In the Destination field, move your log destination from Available to Selected.
4. Click Finished.

Create an iRule

To create an iRule to which it parses network traffic and sends flow records to the specified log publisher:

1. Go to Main > iRules > iRule List > Create.
2. Enter a unique name in the Name field, which represents the flow collector.
3. In the Definition text field, enter the rules for parsing traffic. Ensure the iRule points to the log publisher created earlier.
4. Click Finished.

In the iRule example shown below, replace <insert_log_publisher_name_here> with the name of the log publisher.

when RULE_INIT {
  set static::http_rule1_dest ""
  set static::http_rule1_tmplt ""
}
 
 
# CLIENT_ACCEPTED event to initiate IPFIX destination and template
when CLIENT_ACCEPTED {
  set start [clock clicks -milliseconds]
  if { $static::http_rule1_dest == ""} {
    # open the logging destination if it has not been opened yet
    set static::http_rule1_dest [IPFIX::destination open -publisher /Common/<insert_log_publisher_name_here>]
  }
  if { $static::http_rule1_tmplt == ""} {
    # if the template has not been created yet, create the template
    set static::http_rule1_tmplt [IPFIX::template create "flowStartMilliseconds \
                                                          sourceIPv4Address \
                                                          sourceIPv6Address \
                                                          destinationIPv4Address \
                                                          destinationIPv6Address \
                                                          sourceTransportPort \
                                                          destinationTransportPort \
                                                          protocolIdentifier \
                                                          octetTotalCount \
                                                          packetTotalCount \
                                                          octetDeltaCount \
                                                          packetDeltaCount \
                                                          postNATSourceIPv4Address \
                                                          postNATSourceIPv6Address \
                                                          postNATDestinationIPv4Address \
                                                          postNATDestinationIPv6Address \
                                                          postNAPTSourceTransportPort \
                                                          postNAPTDestinationTransportPort \
                                                          postOctetTotalCount \
                                                          postPacketTotalCount \
                                                          postOctetDeltaCount \
                                                          postPacketDeltaCount \
                                                          flowEndMilliseconds \ "]
                                                           
  }
  set rule1_msg1 [IPFIX::msg create $static::http_rule1_tmplt]
}
 
# SERVER_CONNECTED event to initiate flow data to specified log publisher and populate 5 tuples
when SERVER_CONNECTED {
  set client_closed_flag 0
  set server_closed_flag 0
  IPFIX::msg set $rule1_msg1 flowStartMilliseconds $start
  IPFIX::msg set $rule1_msg1 protocolIdentifier [IP::protocol]
   
  # Clientside
  if { [clientside {IP::version}] equals "4" } {
    # Client IPv4 address
    IPFIX::msg set $rule1_msg1 sourceIPv4Address [IP::client_addr]
    # BIG-IP IPv4 VIP address
    IPFIX::msg set $rule1_msg1 destinationIPv4Address [clientside {IP::local_addr}]
  } else {
    # Client IPv6 address
    IPFIX::msg set $rule1_msg1 sourceIPv6Address [IP::client_addr]
    # BIG-IP IPv6 VIP address
    IPFIX::msg set $rule1_msg1 destinationIPv6Address [clientside {IP::local_addr}]
  }
  # Client port
  IPFIX::msg set $rule1_msg1 sourceTransportPort [TCP::client_port]
  # BIG-IP VIP port
  IPFIX::msg set $rule1_msg1 destinationTransportPort [clientside {TCP::local_port}]
 
  # Serverside
  if { [serverside {IP::version}] equals "4" } {
    # BIG-IP IPv4 self IP address
    IPFIX::msg set $rule1_msg1 postNATSourceIPv4Address [IP::local_addr]
    # Server IPv4 IP address
    IPFIX::msg set $rule1_msg1 postNATDestinationIPv4Address [IP::server_addr]
  } else {
    # BIG-IP IPv6 self IP address
    IPFIX::msg set $rule1_msg1 postNATSourceIPv6Address [IP::local_addr]
    # Server IPv6 IP address
    IPFIX::msg set $rule1_msg1 postNATDestinationIPv6Address [IP::server_addr]
  }
  # BIG-IP self IP port
  IPFIX::msg set $rule1_msg1 postNAPTSourceTransportPort [TCP::local_port]
  # Server port
  IPFIX::msg set $rule1_msg1 postNAPTDestinationTransportPort [TCP::server_port]
}
  
# SERVER_CLOSED event to collect IP pkts and bytes count on serverside
when SERVER_CLOSED {
  set server_closed_flag 1
  # when flow is completed, BIG-IP to server REQUEST pkts and bytes count
  IPFIX::msg set $rule1_msg1 octetTotalCount [IP::stats bytes out]
  IPFIX::msg set $rule1_msg1 packetTotalCount [IP::stats pkts out]
  # when flow is completed, server to BIG-IP RESPONSE pkts and bytes count
  IPFIX::msg set $rule1_msg1 octetDeltaCount [IP::stats bytes in]
  IPFIX::msg set $rule1_msg1 packetDeltaCount [IP::stats pkts in]
    IPFIX::destination send $static::http_rule1_dest $rule1_msg1
}
 
# CLIENT_CLOSED event to collect IP pkts and bytes count on clientside
when CLIENT_CLOSED {
  set client_closed_flag 1
  # when flow is completed, client to BIG-IP REQUEST pkts and bytes octetDeltaCount
  IPFIX::msg set $rule1_msg1 postOctetTotalCount [IP::stats bytes in]
  IPFIX::msg set $rule1_msg1 postPacketTotalCount [IP::stats pkts in]
  # when flow is completed, BIG-IP to client RESPONSE pkts and bytes count
  IPFIX::msg set $rule1_msg1 postOctetDeltaCount [IP::stats bytes out]
  IPFIX::msg set $rule1_msg1 postPacketDeltaCount [IP::stats pkts out]
  # record the client closed time in ms
  IPFIX::msg set $rule1_msg1 flowEndMilliseconds [clock click -milliseconds]
    # send the IPFIX log
    IPFIX::destination send $static::http_rule1_dest $rule1_msg1
}

Apply the iRule to a Virtual Server

To apply the iRule to a virtual server whose traffic you want to parse:

1. Go to Main > Virtual Server > Virtual Server List.
2. Select the virtual server you want to monitor.
3. Click the Resources tab. In the iRule section, click Manage.
4. Select the iRule that you previously created and move the iRule from Available to Enable.
5. Click Finished.

Example of a Virtual Server Resources page with the new iRule applied:

Create a Route Entry

By default, all traffic is sent out of the management interface. However, F5 does not support flow exports via the management NIC. You must add a route to force traffic, which is destined to the flow collector to leave a self-IP interface.

To create a route entry, if the F5 self-IP is unable to reach the flow collector:

1. In the F5 UI, click Main > Network > Routes > Add.
2. In the Properties section, create a route entry to send the flow records from F5 to the external flow collector IP address.

For Resource, select the Use Gateway option.