Coresight - HW Assisted Tracing on ARM
======================================
Author: Mathieu Poirier mathieu.poirier@linaro.org
Date: September 11th, 2014
Introduction
Coresight is an umbrella of technologies allowing for the debugging of ARM
based SoC. It includes solutions for JTAG and HW assisted tracing. This
document is concerned with the latter.
HW assisted tracing is becoming increasingly useful when dealing with systems
that have many SoCs and other components like GPU and DMA engines. ARM has
developed a HW assisted tracing solution by means of different components, each
being added to a design at synthesis time to cater to specific tracing needs.
Components are generally categorised as source, link and sinks and are
(usually) discovered using the AMBA bus.
“Sources” generate a compressed stream representing the processor instruction
path based on tracing scenarios as configured by users. From there the stream
flows through the coresight system (via ATB bus) using links that are connecting
the emanating source to a sink(s). Sinks serve as endpoints to the coresight
implementation, either storing the compressed stream in a memory buffer or
creating an interface to the outside world where data can be transferred to a
host without fear of filling up the onboard coresight memory buffer.
At typical coresight system would look like this:
**** AMBA AXI ****===||
* ||
^ ^ | ||
| | * **
0000000 ::::: 0000000 ::::: ::::: @@@@@@@ ||||||||||||
0 CPU 0<–>: C : 0 CPU 0<–>: C : : C : @ STM @ || System ||
|->0000000 : T : |->0000000 : T : : T :<—>@@@@@ || Memory ||
| #######<–>: I : | #######<–>: I : : I : @@@<-| ||||||||||||
| # ETM # ::::: | # PTM # ::::: ::::: @ |
| ##### ^ ^ | ##### ^ ! ^ ! . | |||||||||
| |->### | ! | |->### | ! | ! . | || DAP ||
| | # | ! | | # | ! | ! . | |||||||||
| | . | ! | | . | ! | ! . | | |
| | . | ! | | . | ! | ! . | | *
| | . | ! | | . | ! | ! . | | SWD/
| | . | ! | | . | ! | ! . | | JTAG
*<-|
*** AMBA Debug APB ****
| . ! . ! ! . |
| . * . * * . |
**** Cross Trigger Matrix (CTM) ***
| . ^ . . |
| * ! * * |
** AMBA Advanced Trace Bus (ATB) **
| ! =============== |
| * ===== F =====<———|
| ::::::::: ==== U ====
|–>:: CTI ::<!! === N ===
| ::::::::: ! == N ==
| ^ * == E ==
| ! &&&&&&&&& IIIIIII == L ==
|——>&& ETB &&<……II I =======
| ! &&&&&&&&& II I .
| ! I I .
| ! I REP I<……….
| ! I I
| !!>&&&&&&&&& II I *Source: ARM ltd.
|——>& TPIU &<……II I DAP = Debug Access Port
&&&&&&&&& IIIIIII ETM = Embedded Trace Macrocell
; PTM = Program Trace Macrocell
; CTI = Cross Trigger Interface
* ETB = Embedded Trace Buffer
To trace port TPIU= Trace Port Interface Unit
SWD = Serial Wire Debug
While on target configuration of the components is done via the APB bus,
all trace data are carried out-of-band on the ATB bus. The CTM provides
a way to aggregate and distribute signals between CoreSight components.
The coresight framework provides a central point to represent, configure and
manage coresight devices on a platform. This first implementation centers on
the basic tracing functionality, enabling components such ETM/PTM, funnel,
replicator, TMC, TPIU and ETB. Future work will enable more
intricate IP blocks such as STM and CTI.
Acronyms and Classification
Acronyms:
PTM: Program Trace Macrocell
ETM: Embedded Trace Macrocell
STM: System trace Macrocell
ETB: Embedded Trace Buffer
ITM: Instrumentation Trace Macrocell
TPIU: Trace Port Interface Unit
TMC-ETR: Trace Memory Controller, configured as Embedded Trace Router
TMC-ETF: Trace Memory Controller, configured as Embedded Trace FIFO
CTI: Cross Trigger Interface
Classification:
Source:
ETMv3.x ETMv4, PTMv1.0, PTMv1.1, STM, STM500, ITM
Link:
Funnel, replicator (intelligent or not), TMC-ETR
Sinks:
ETBv1.0, ETB1.1, TPIU, TMC-ETF
Misc:
CTI
Device Tree Bindings
See Documentation/devicetree/bindings/arm/coresight.txt for details.
As of this writing drivers for ITM, STMs and CTIs are not provided but are
expected to be added as the solution matures.
Framework and implementation
The coresight framework provides a central point to represent, configure and
manage coresight devices on a platform. Any coresight compliant device can
register with the framework for as long as they use the right APIs:
struct coresight_device *coresight_register(struct coresight_desc *desc);
void coresight_unregister(struct coresight_device *csdev);
The registering function is taking a “struct coresight_device *csdev” and
register the device with the core framework. The unregister function takes
a reference to a “struct coresight_device”, obtained at registration time.
If everything goes well during the registration process the new devices will
show up under /sys/bus/coresight/devices, as showns here for a TC2 platform:
root:# ls /sys/bus/coresight/devices/#
replicator 20030000.tpiu 2201c000.ptm 2203c000.etm 2203e000.etm
20010000.etb 20040000.funnel 2201d000.ptm 2203d000.etm
root:
The functions take a “struct coresight_device”, which looks like this:
struct coresight_desc {
enum coresight_dev_type type;
struct coresight_dev_subtype subtype;
const struct coresight_ops *ops;
struct coresight_platform_data pdata;
struct device *dev;
const struct attribute_group *groups;
};
The “coresight_dev_type” identifies what the device is, i.e, source link or
sink while the “coresight_dev_subtype” will characterise that type further.
The “struct coresight_ops” is mandatory and will tell the framework how to
perform base operations related to the components, each component having
a different set of requirement. For that “struct coresight_ops_sink”,
“struct coresight_ops_link” and “struct coresight_ops_source” have been
provided.
The next field, “struct coresight_platform_data *pdata” is acquired by calling
“of_get_coresight_platform_data()”, as part of the driver’s _probe routine and
“struct device *dev” gets the device reference embedded in the “amba_device”:
static int etm_probe(struct amba_device *adev, const struct amba_id *id)
{
…
…
drvdata->dev = &adev->dev;
…
}
Specific class of device (source, link, or sink) have generic operations
that can be performed on them (see “struct coresight_ops”). The
“**groups” is a list of sysfs entries pertaining to operations
specific to that component only. “Implementation defined” customisations are
expected to be accessed and controlled using those entries.
How to use the tracer modules
There are two ways to use the Coresight framework: 1) using the perf cmd line
tools and 2) interacting directly with the Coresight devices using the sysFS
interface. Preference is given to the former as using the sysFS interface
requires a deep understanding of the Coresight HW. The following sections
provide details on using both methods.
- Using the sysFS interface:
Before trace collection can start, a coresight sink needs to be identified.
There is no limit on the amount of sinks (nor sources) that can be enabled at
any given moment. As a generic operation, all device pertaining to the sink
class will have an “active” entry in sysfs:
root:/sys/bus/coresight/devices# ls
replicator 20030000.tpiu 2201c000.ptm 2203c000.etm 2203e000.etm
20010000.etb 20040000.funnel 2201d000.ptm 2203d000.etm
root:/sys/bus/coresight/devices# ls 20010000.etb
enable_sink status trigger_cntr
root:/sys/bus/coresight/devices# echo 1 > 20010000.etb/enable_sink
root:/sys/bus/coresight/devices# cat 20010000.etb/enable_sink
1
root:/sys/bus/coresight/devices#
At boot time the current etm3x driver will configure the first address
comparator with “_stext” and “_etext”, essentially tracing any instruction
that falls within that range. As such “enabling” a source will immediately
trigger a trace capture:
root:/sys/bus/coresight/devices# echo 1 > 2201c000.ptm/enable_source
root:/sys/bus/coresight/devices# cat 2201c000.ptm/enable_source
1
root:/sys/bus/coresight/devices# cat 20010000.etb/status
Depth: 0x2000
Status: 0x1
RAM read ptr: 0x0
RAM wrt ptr: 0x19d3 <—– The write pointer is moving
Trigger cnt: 0x0
Control: 0x1
Flush status: 0x0
Flush ctrl: 0x2001
root:/sys/bus/coresight/devices#
Trace collection is stopped the same way:
root:/sys/bus/coresight/devices# echo 0 > 2201c000.ptm/enable_source
root:/sys/bus/coresight/devices#
The content of the ETB buffer can be harvested directly from /dev:
root:/sys/bus/coresight/devices# dd if=/dev/20010000.etb
of=~/cstrace.bin
64+0 records in
64+0 records out
32768 bytes (33 kB) copied, 0.00125258 s, 26.2 MB/s
root:/sys/bus/coresight/devices#
The file cstrace.bin can be decompressed using “ptm2human”, DS-5 or Trace32.
Following is a DS-5 output of an experimental loop that increments a variable up
to a certain value. The example is simple and yet provides a glimpse of the
wealth of possibilities that coresight provides.
Info Tracing enabled
Instruction 106378866 0x8026B53C E52DE004 false PUSH {lr}
Instruction 0 0x8026B540 E24DD00C false SUB sp,sp,#0xc
Instruction 0 0x8026B544 E3A03000 false MOV r3,#0
Instruction 0 0x8026B548 E58D3004 false STR r3,[sp,#4]
Instruction 0 0x8026B54C E59D3004 false LDR r3,[sp,#4]
Instruction 0 0x8026B550 E3530004 false CMP r3,#4
Instruction 0 0x8026B554 E2833001 false ADD r3,r3,#1
Instruction 0 0x8026B558 E58D3004 false STR r3,[sp,#4]
Instruction 0 0x8026B55C DAFFFFFA true BLE {pc}-0x10 ; 0x8026b54c
Timestamp Timestamp: 17106715833
Instruction 319 0x8026B54C E59D3004 false LDR r3,[sp,#4]
Instruction 0 0x8026B550 E3530004 false CMP r3,#4
Instruction 0 0x8026B554 E2833001 false ADD r3,r3,#1
Instruction 0 0x8026B558 E58D3004 false STR r3,[sp,#4]
Instruction 0 0x8026B55C DAFFFFFA true BLE {pc}-0x10 ; 0x8026b54c
Instruction 9 0x8026B54C E59D3004 false LDR r3,[sp,#4]
Instruction 0 0x8026B550 E3530004 false CMP r3,#4
Instruction 0 0x8026B554 E2833001 false ADD r3,r3,#1
Instruction 0 0x8026B558 E58D3004 false STR r3,[sp,#4]
Instruction 0 0x8026B55C DAFFFFFA true BLE {pc}-0x10 ; 0x8026b54c
Instruction 7 0x8026B54C E59D3004 false LDR r3,[sp,#4]
Instruction 0 0x8026B550 E3530004 false CMP r3,#4
Instruction 0 0x8026B554 E2833001 false ADD r3,r3,#1
Instruction 0 0x8026B558 E58D3004 false STR r3,[sp,#4]
Instruction 0 0x8026B55C DAFFFFFA true BLE {pc}-0x10 ; 0x8026b54c
Instruction 7 0x8026B54C E59D3004 false LDR r3,[sp,#4]
Instruction 0 0x8026B550 E3530004 false CMP r3,#4
Instruction 0 0x8026B554 E2833001 false ADD r3,r3,#1
Instruction 0 0x8026B558 E58D3004 false STR r3,[sp,#4]
Instruction 0 0x8026B55C DAFFFFFA true BLE {pc}-0x10 ; 0x8026b54c
Instruction 10 0x8026B54C E59D3004 false LDR r3,[sp,#4]
Instruction 0 0x8026B550 E3530004 false CMP r3,#4
Instruction 0 0x8026B554 E2833001 false ADD r3,r3,#1
Instruction 0 0x8026B558 E58D3004 false STR r3,[sp,#4]
Instruction 0 0x8026B55C DAFFFFFA true BLE {pc}-0x10 ; 0x8026b54c
Instruction 6 0x8026B560 EE1D3F30 false MRC p15,#0x0,r3,c13,c0,#1
Instruction 0 0x8026B564 E1A0100D false MOV r1,sp
Instruction 0 0x8026B568 E3C12D7F false BIC r2,r1,#0x1fc0
Instruction 0 0x8026B56C E3C2203F false BIC r2,r2,#0x3f
Instruction 0 0x8026B570 E59D1004 false LDR r1,[sp,#4]
Instruction 0 0x8026B574 E59F0010 false LDR r0,[pc,#16] ; [0x8026B58C] = 0x80550368
Instruction 0 0x8026B578 E592200C false LDR r2,[r2,#0xc]
Instruction 0 0x8026B57C E59221D0 false LDR r2,[r2,#0x1d0]
Instruction 0 0x8026B580 EB07A4CF true BL {pc}+0x1e9344 ; 0x804548c4
Info Tracing enabled
Instruction 13570831 0x8026B584 E28DD00C false ADD sp,sp,#0xc
Instruction 0 0x8026B588 E8BD8000 true LDM sp!,{pc}
Timestamp Timestamp: 17107041535
- Using perf framework:
Coresight tracers are represented using the Perf framework’s Performance
Monitoring Unit (PMU) abstraction. As such the perf framework takes charge of
controlling when tracing gets enabled based on when the process of interest is
scheduled. When configured in a system, Coresight PMUs will be listed when
queried by the perf command line tool:
linaro@linaro-nano:~$ ./perf list pmu
List of pre-defined events (to be used in -e):
cs_etm// [Kernel PMU event]
linaro@linaro-nano:~$
Regardless of the number of tracers available in a system (usually equal to the
amount of processor cores), the “cs_etm” PMU will be listed only once.
A Coresight PMU works the same way as any other PMU, i.e the name of the PMU is
listed along with configuration options within forward slashes ‘/‘. Since a
Coresight system will typically have more than one sink, the name of the sink to
work with needs to be specified as an event option. Names for sink to choose
from are listed in sysFS under ($SYSFS)/bus/coresight/devices:
root@linaro-nano:~# ls /sys/bus/coresight/devices/
20010000.etf 20040000.funnel 20100000.stm 22040000.etm
22140000.etm 230c0000.funnel 23240000.etm 20030000.tpiu
20070000.etr 20120000.replicator 220c0000.funnel
23040000.etm 23140000.etm 23340000.etm
root@linaro-nano:~# perf record -e cs_etm/@20070000.etr/u --per-thread program
The syntax within the forward slashes ‘/‘ is important. The ‘@’ character
tells the parser that a sink is about to be specified and that this is the sink
to use for the trace session.
More information on the above and other example on how to use Coresight with
the perf tools can be found in the “HOWTO.md” file of the openCSD gitHub
repository [3].
2.1) AutoFDO analysis using the perf tools:
perf can be used to record and analyze trace of programs.
Execution can be recorded using ‘perf record’ with the cs_etm event,
specifying the name of the sink to record to, e.g:
perf record -e cs_etm/@20070000.etr/u --per-thread
The ‘perf report’ and ‘perf script’ commands can be used to analyze execution,
synthesizing instruction and branch events from the instruction trace.
‘perf inject’ can be used to replace the trace data with the synthesized events.
The –itrace option controls the type and frequency of synthesized events
(see perf documentation).
Note that only 64-bit programs are currently supported - further work is
required to support instruction decode of 32-bit Arm programs.
Generating coverage files for Feedback Directed Optimization: AutoFDO
‘perf inject’ accepts the –itrace option in which case tracing data is
removed and replaced with the synthesized events. e.g.
perf inject --itrace --strip -i perf.data -o perf.data.new
Below is an example of using ARM ETM for autoFDO. It requires autofdo
(https://github.com/google/autofdo) and gcc version 5. The bubble
sort example is from the AutoFDO tutorial (https://gcc.gnu.org/wiki/AutoFDO/Tutorial).
$ gcc-5 -O3 sort.c -o sort
$ taskset -c 2 ./sort
Bubble sorting array of 30000 elements
5910 ms
$ perf record -e cs_etm/@20070000.etr/u --per-thread taskset -c 2 ./sort
Bubble sorting array of 30000 elements
12543 ms
[ perf record: Woken up 35 times to write data ]
[ perf record: Captured and wrote 69.640 MB perf.data ]
$ perf inject -i perf.data -o inj.data --itrace=il64 --strip
$ create_gcov --binary=./sort --profile=inj.data --gcov=sort.gcov -gcov_version=1
$ gcc-5 -O3 -fauto-profile=sort.gcov sort.c -o sort_autofdo
$ taskset -c 2 ./sort_autofdo
Bubble sorting array of 30000 elements
5806 ms
How to use the STM module
Using the System Trace Macrocell module is the same as the tracers - the only
difference is that clients are driving the trace capture rather
than the program flow through the code.
As with any other CoreSight component, specifics about the STM tracer can be
found in sysfs with more information on each entry being found in [1]:
root@genericarmv8:# ls /sys/bus/coresight/devices/20100000.stm#
enable_source hwevent_select port_enable subsystem uevent
hwevent_enable mgmt port_select traceid
root@genericarmv8:
Like any other source a sink needs to be identified and the STM enabled before
being used:
root@genericarmv8:# echo 1 > /sys/bus/coresight/devices/20010000.etf/enable_sink# echo 1 > /sys/bus/coresight/devices/20100000.stm/enable_source
root@genericarmv8:
From there user space applications can request and use channels using the devfs
interface provided for that purpose by the generic STM API:
root@genericarmv8:# ls -l /dev/20100000.stm#
crw——- 1 root root 10, 61 Jan 3 18:11 /dev/20100000.stm
root@genericarmv8:
Details on how to use the generic STM API can be found here [2].
[1]. Documentation/ABI/testing/sysfs-bus-coresight-devices-stm
[2]. Documentation/trace/stm.rst
[3]. https://github.com/Linaro/perf-opencsd
