Statistics

While running the RTC Testbench it collects statistics about various aspects including round trip and one way times, errors, processing and workload execution. This sections explains what statistics are collected and how they can be obtained and visualized.

Statistics

The following table shows all gathered statistics. All statistics are collected per traffic class.

Statistics

Name	Description
FrameIdErrors	Is incremented if a traffic class receives a packet with wrong Profinet frame id. This could happen e.g., due to a wrong XDP eBPF program. Should be zero.
FramesReceived	How many frames have been received and processed.
FramesSent	How many frames have been transmitted.
Oneway[Min,Max,Av] [us]	One way time measures the time between sending a packet from reference to mirror (or the other way around) and receiving it on the other side. The time stamps for this are taken in software on the application level.
OnewayOutliers	Incremented if one way time is greater than cycle time. Should be zero.
OutofOrderErrors	Is incremented if a frame does not have the expected sequence number. This can indicate out of order transmission, but will also be increased if a packet is lost. Should be zero.
PayloadErrors	Is incremented if a frame does not have the expected payload. May happen due to driver bugs. Should be zero.
ProcBatch[Min,Max,Av] [us]	Latency from the first RX hardware timestamp to the last TX hardware timestamp (batch processing latency per cycle). See Processing Latency for detailed information.
ProcBatchOutliers	Is incremented if a ProcBatch latency is greater than cycle time. This usually indicates some real time issue (e.g., kernel, driver, hardware, …). Should be zero.
ProcFirst[Min,Max,Av] [us]	Latency from the first RX hardware timestamp to the first TX hardware timestamp (first-frame processing latency per cycle). See Processing Latency for detailed information.
ProcFirstOutliers	Is incremented if a ProcFirst latency is greater than cycle time. This usually indicates some real time issue (e.g., kernel, driver, hardware, …). Should be zero.
RoundTrip[Min,Max,Av] [us]	Round trip delay measures the time from reference to mirror and back. The delay is calculated based on software timestamps. Also the mirror sends back the frame not immediately, but rather in next cycle. Therefore, the round trip time should be less than 2 * cycle time. Note: In the JSON/MQTT output the minimum is named RoundTripTimeMin.
RoundTripOutliers	Is incremented if a round trip time is greater than 2 * cycle time. This usually indicates some real time issue (e.g., kernel, driver, hardware, …). Should be zero.
Rx[Min,Max,Av] [us]	Latency from NIC hardware to user space based on hardware timestamps. See Processing Latency for detailed information.
RxHw2Sw[Min,Max,Av] [us]	Latency from NIC hardware to the earliest available software timestamp (the XDP program for AF_XDP, or the kernel’s software RX timestamp for AF_PACKET) based on hardware timestamps. See Processing Latency for detailed information.
RxWorkload[Min,Max,Av] [us]	Duration of workload execution. Reported if one workload CPU is configured. See Real Time Compute Workloads.
RxWorkloadOutliers	Is incremented if a workload did not finish before the Tx thread of the traffic class runs. Should be zero.
RxWorkload<N>[Min,Max,Av,Outliers]	Same as above, but per workload thread N. Reported instead of the ones above if multiple workload CPUs are configured.
RxSw2App[Min,Max,Av] [us]	Latency from the earliest available software timestamp (the XDP program for AF_XDP, or the kernel’s software RX timestamp for AF_PACKET) to user space. See Processing Latency for detailed information.
Tx[Min,Max,Av] [us]	Latency from user space enqueue to hardware transmit based on hardware timestamps. See Processing Latency for detailed information.
TxHwTimestampMissing	Is incremented if hardware timestamp requested, but only software timestamp available. See Processing Latency for detailed information.

Note

Some statistics are only available under certain conditions: The hardware timestamp based ones require the RX_TIMESTAMP and/or TX_TIMESTAMP build options, are limited to the traffic classes TsnHigh, TsnLow, Rtc, Rta and GenericL2 and work with both AF_XDP and AF_PACKET. The workload statistics are only available, if a workload is configured. For detailed information about hardware timestamping, configuration requirements, and troubleshooting, see Processing Latency.

File log

All statistics are logged once per collection interval into a file log in text form.

MQTT / Grafana

Furthermore, the statistics can be submitted with MQTT and visualized with Grafana.

See MQTT and Grafana Integration for more information.

JSON/UDP

In addition, there is a JSON/UDP logger. This one submits the statistics in JSON representation per UDP once per collection interval. This is useful to get the statistics into other tools.

In order to use the JSON/UDP logging, the following configuration parameters can be used:

JSON/UDP configuration options

Option	Description
StatsCollectionIntervalNS	Interval in which a summary of the data is generated, typically 1s
LogJson	Enable and/or disable JSON logging
LogJsonThreadPriority	Thread priority for the logging thread, usually low < 7
LogJsonThreadCpu	Thread CPU affinity
LogJsonHost	IP address/host name where to send the JSON/UDP packets
LogJsonPort	Port used by the UDP communication
LogJsonMeasurementName	Used to distinguish measurements coming from different machines

The code base contains a script (scripts/stat.pl) to collect and print the statistics data. The user has to provide the UDP port (-p), the measurement name (-m), the traffic class (-t) and the statistics of interest (-s, wildcards are supported).

Example:

./scripts/stat.pl -p 8888 -m testbench1 -t TsnHigh -s FramesSent -s FramesReceived -s RxWorkload*
Measurement: testbench1 -- TC: TsnHigh
  FramesReceived: 0
  FramesSent: 24016
  RxWorkloadAv: 0
  RxWorkloadMax: 0
  RxWorkloadMin: 18446744073709551615