mirror of
git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
synced 2025-08-05 16:54:27 +00:00
52092c1d50
Corrects a spelling mistake in Documentation/trace/coresight/panic.rst where "incase" was used instead of "in case". Signed-off-by: Hendrik Hamerlinck <hendrik.hamerlinck@hammernet.be> Signed-off-by: Jonathan Corbet <corbet@lwn.net> Message-ID: <20250513110931.15072-1-hendrik.hamerlinck@hammernet.be>
362 lines
14 KiB
ReStructuredText
362 lines
14 KiB
ReStructuredText
===================================================
|
|
Using Coresight for Kernel panic and Watchdog reset
|
|
===================================================
|
|
|
|
Introduction
|
|
------------
|
|
This documentation is about using Linux coresight trace support to
|
|
debug kernel panic and watchdog reset scenarios.
|
|
|
|
Coresight trace during Kernel panic
|
|
-----------------------------------
|
|
From the coresight driver point of view, addressing the kernel panic
|
|
situation has four main requirements.
|
|
|
|
a. Support for allocation of trace buffer pages from reserved memory area.
|
|
Platform can advertise this using a new device tree property added to
|
|
relevant coresight nodes.
|
|
|
|
b. Support for stopping coresight blocks at the time of panic
|
|
|
|
c. Saving required metadata in the specified format
|
|
|
|
d. Support for reading trace data captured at the time of panic
|
|
|
|
Allocation of trace buffer pages from reserved RAM
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
A new optional device tree property "memory-region" is added to the
|
|
Coresight TMC device nodes, that would give the base address and size of trace
|
|
buffer.
|
|
|
|
Static allocation of trace buffers would ensure that both IOMMU enabled
|
|
and disabled cases are handled. Also, platforms that support persistent
|
|
RAM will allow users to read trace data in the subsequent boot without
|
|
booting the crashdump kernel.
|
|
|
|
Note:
|
|
For ETR sink devices, this reserved region will be used for both trace
|
|
capture and trace data retrieval.
|
|
For ETF sink devices, internal SRAM would be used for trace capture,
|
|
and they would be synced to reserved region for retrieval.
|
|
|
|
|
|
Disabling coresight blocks at the time of panic
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
In order to avoid the situation of losing relevant trace data after a
|
|
kernel panic, it would be desirable to stop the coresight blocks at the
|
|
time of panic.
|
|
|
|
This can be achieved by configuring the comparator, CTI and sink
|
|
devices as below::
|
|
|
|
Trigger on panic
|
|
Comparator --->External out --->CTI -->External In---->ETR/ETF stop
|
|
|
|
Saving metadata at the time of kernel panic
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
Coresight metadata involves all additional data that are required for a
|
|
successful trace decode in addition to the trace data. This involves
|
|
ETR/ETF/ETB register snapshot etc.
|
|
|
|
A new optional device property "memory-region" is added to
|
|
the ETR/ETF/ETB device nodes for this.
|
|
|
|
Reading trace data captured at the time of panic
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
Trace data captured at the time of panic, can be read from rebooted kernel
|
|
or from crashdump kernel using a special device file /dev/crash_tmc_xxx.
|
|
This device file is created only when there is a valid crashdata available.
|
|
|
|
General flow of trace capture and decode in case of kernel panic
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
1. Enable source and sink on all the cores using the sysfs interface.
|
|
ETR sinks should have trace buffers allocated from reserved memory,
|
|
by selecting "resrv" buffer mode from sysfs.
|
|
|
|
2. Run relevant tests.
|
|
|
|
3. On a kernel panic, all coresight blocks are disabled, necessary
|
|
metadata is synced by kernel panic handler.
|
|
|
|
System would eventually reboot or boot a crashdump kernel.
|
|
|
|
4. For platforms that supports crashdump kernel, raw trace data can be
|
|
dumped using the coresight sysfs interface from the crashdump kernel
|
|
itself. Persistent RAM is not a requirement in this case.
|
|
|
|
5. For platforms that supports persistent RAM, trace data can be dumped
|
|
using the coresight sysfs interface in the subsequent Linux boot.
|
|
Crashdump kernel is not a requirement in this case. Persistent RAM
|
|
ensures that trace data is intact across reboot.
|
|
|
|
Coresight trace during Watchdog reset
|
|
-------------------------------------
|
|
The main difference between addressing the watchdog reset and kernel panic
|
|
case are below,
|
|
|
|
a. Saving coresight metadata need to be taken care by the
|
|
SCP(system control processor) firmware in the specified format,
|
|
instead of kernel.
|
|
|
|
b. Reserved memory region given by firmware for trace buffer and metadata
|
|
has to be in persistent RAM.
|
|
Note: This is a requirement for watchdog reset case but optional
|
|
in kernel panic case.
|
|
|
|
Watchdog reset can be supported only on platforms that meet the above
|
|
two requirements.
|
|
|
|
Sample commands for testing a Kernel panic case with ETR sink
|
|
-------------------------------------------------------------
|
|
|
|
1. Boot Linux kernel with "crash_kexec_post_notifiers" added to the kernel
|
|
bootargs. This is mandatory if the user would like to read the tracedata
|
|
from the crashdump kernel.
|
|
|
|
2. Enable the preloaded ETM configuration::
|
|
|
|
#echo 1 > /sys/kernel/config/cs-syscfg/configurations/panicstop/enable
|
|
|
|
3. Configure CTI using sysfs interface::
|
|
|
|
#./cti_setup.sh
|
|
|
|
#cat cti_setup.sh
|
|
|
|
|
|
cd /sys/bus/coresight/devices/
|
|
|
|
ap_cti_config () {
|
|
#ETM trig out[0] trigger to Channel 0
|
|
echo 0 4 > channels/trigin_attach
|
|
}
|
|
|
|
etf_cti_config () {
|
|
#ETF Flush in trigger from Channel 0
|
|
echo 0 1 > channels/trigout_attach
|
|
echo 1 > channels/trig_filter_enable
|
|
}
|
|
|
|
etr_cti_config () {
|
|
#ETR Flush in from Channel 0
|
|
echo 0 1 > channels/trigout_attach
|
|
echo 1 > channels/trig_filter_enable
|
|
}
|
|
|
|
ctidevs=`find . -name "cti*"`
|
|
|
|
for i in $ctidevs
|
|
do
|
|
cd $i
|
|
|
|
connection=`find . -name "ete*"`
|
|
if [ ! -z "$connection" ]
|
|
then
|
|
echo "AP CTI config for $i"
|
|
ap_cti_config
|
|
fi
|
|
|
|
connection=`find . -name "tmc_etf*"`
|
|
if [ ! -z "$connection" ]
|
|
then
|
|
echo "ETF CTI config for $i"
|
|
etf_cti_config
|
|
fi
|
|
|
|
connection=`find . -name "tmc_etr*"`
|
|
if [ ! -z "$connection" ]
|
|
then
|
|
echo "ETR CTI config for $i"
|
|
etr_cti_config
|
|
fi
|
|
|
|
cd ..
|
|
done
|
|
|
|
Note: CTI connections are SOC specific and hence the above script is
|
|
added just for reference.
|
|
|
|
4. Choose reserved buffer mode for ETR buffer::
|
|
|
|
#echo "resrv" > /sys/bus/coresight/devices/tmc_etr0/buf_mode_preferred
|
|
|
|
5. Enable stop on flush trigger configuration::
|
|
|
|
#echo 1 > /sys/bus/coresight/devices/tmc_etr0/stop_on_flush
|
|
|
|
6. Start Coresight tracing on cores 1 and 2 using sysfs interface
|
|
|
|
7. Run some application on core 1::
|
|
|
|
#taskset -c 1 dd if=/dev/urandom of=/dev/null &
|
|
|
|
8. Invoke kernel panic on core 2::
|
|
|
|
#echo 1 > /proc/sys/kernel/panic
|
|
#taskset -c 2 echo c > /proc/sysrq-trigger
|
|
|
|
9. From rebooted kernel or crashdump kernel, read crashdata::
|
|
|
|
#dd if=/dev/crash_tmc_etr0 of=/trace/cstrace.bin
|
|
|
|
10. Run opencsd decoder tools/scripts to generate the instruction trace.
|
|
|
|
Sample instruction trace dump
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
Core1 dump::
|
|
|
|
A etm4_enable_hw: ffff800008ae1dd4
|
|
CONTEXT EL2 etm4_enable_hw: ffff800008ae1dd4
|
|
I etm4_enable_hw: ffff800008ae1dd4:
|
|
d503201f nop
|
|
I etm4_enable_hw: ffff800008ae1dd8:
|
|
d503201f nop
|
|
I etm4_enable_hw: ffff800008ae1ddc:
|
|
d503201f nop
|
|
I etm4_enable_hw: ffff800008ae1de0:
|
|
d503201f nop
|
|
I etm4_enable_hw: ffff800008ae1de4:
|
|
d503201f nop
|
|
I etm4_enable_hw: ffff800008ae1de8:
|
|
d503233f paciasp
|
|
I etm4_enable_hw: ffff800008ae1dec:
|
|
a9be7bfd stp x29, x30, [sp, #-32]!
|
|
I etm4_enable_hw: ffff800008ae1df0:
|
|
910003fd mov x29, sp
|
|
I etm4_enable_hw: ffff800008ae1df4:
|
|
a90153f3 stp x19, x20, [sp, #16]
|
|
I etm4_enable_hw: ffff800008ae1df8:
|
|
2a0003f4 mov w20, w0
|
|
I etm4_enable_hw: ffff800008ae1dfc:
|
|
900085b3 adrp x19, ffff800009b95000 <reserved_mem+0xc48>
|
|
I etm4_enable_hw: ffff800008ae1e00:
|
|
910f4273 add x19, x19, #0x3d0
|
|
I etm4_enable_hw: ffff800008ae1e04:
|
|
f8747a60 ldr x0, [x19, x20, lsl #3]
|
|
E etm4_enable_hw: ffff800008ae1e08:
|
|
b4000140 cbz x0, ffff800008ae1e30 <etm4_starting_cpu+0x50>
|
|
I 149.039572921 etm4_enable_hw: ffff800008ae1e30:
|
|
a94153f3 ldp x19, x20, [sp, #16]
|
|
I 149.039572921 etm4_enable_hw: ffff800008ae1e34:
|
|
52800000 mov w0, #0x0 // #0
|
|
I 149.039572921 etm4_enable_hw: ffff800008ae1e38:
|
|
a8c27bfd ldp x29, x30, [sp], #32
|
|
|
|
..snip
|
|
|
|
149.052324811 chacha_block_generic: ffff800008642d80:
|
|
9100a3e0 add x0,
|
|
I 149.052324811 chacha_block_generic: ffff800008642d84:
|
|
b86178a2 ldr w2, [x5, x1, lsl #2]
|
|
I 149.052324811 chacha_block_generic: ffff800008642d88:
|
|
8b010803 add x3, x0, x1, lsl #2
|
|
I 149.052324811 chacha_block_generic: ffff800008642d8c:
|
|
b85fc063 ldur w3, [x3, #-4]
|
|
I 149.052324811 chacha_block_generic: ffff800008642d90:
|
|
0b030042 add w2, w2, w3
|
|
I 149.052324811 chacha_block_generic: ffff800008642d94:
|
|
b8217882 str w2, [x4, x1, lsl #2]
|
|
I 149.052324811 chacha_block_generic: ffff800008642d98:
|
|
91000421 add x1, x1, #0x1
|
|
I 149.052324811 chacha_block_generic: ffff800008642d9c:
|
|
f100443f cmp x1, #0x11
|
|
|
|
|
|
Core 2 dump::
|
|
|
|
A etm4_enable_hw: ffff800008ae1dd4
|
|
CONTEXT EL2 etm4_enable_hw: ffff800008ae1dd4
|
|
I etm4_enable_hw: ffff800008ae1dd4:
|
|
d503201f nop
|
|
I etm4_enable_hw: ffff800008ae1dd8:
|
|
d503201f nop
|
|
I etm4_enable_hw: ffff800008ae1ddc:
|
|
d503201f nop
|
|
I etm4_enable_hw: ffff800008ae1de0:
|
|
d503201f nop
|
|
I etm4_enable_hw: ffff800008ae1de4:
|
|
d503201f nop
|
|
I etm4_enable_hw: ffff800008ae1de8:
|
|
d503233f paciasp
|
|
I etm4_enable_hw: ffff800008ae1dec:
|
|
a9be7bfd stp x29, x30, [sp, #-32]!
|
|
I etm4_enable_hw: ffff800008ae1df0:
|
|
910003fd mov x29, sp
|
|
I etm4_enable_hw: ffff800008ae1df4:
|
|
a90153f3 stp x19, x20, [sp, #16]
|
|
I etm4_enable_hw: ffff800008ae1df8:
|
|
2a0003f4 mov w20, w0
|
|
I etm4_enable_hw: ffff800008ae1dfc:
|
|
900085b3 adrp x19, ffff800009b95000 <reserved_mem+0xc48>
|
|
I etm4_enable_hw: ffff800008ae1e00:
|
|
910f4273 add x19, x19, #0x3d0
|
|
I etm4_enable_hw: ffff800008ae1e04:
|
|
f8747a60 ldr x0, [x19, x20, lsl #3]
|
|
E etm4_enable_hw: ffff800008ae1e08:
|
|
b4000140 cbz x0, ffff800008ae1e30 <etm4_starting_cpu+0x50>
|
|
I 149.046243445 etm4_enable_hw: ffff800008ae1e30:
|
|
a94153f3 ldp x19, x20, [sp, #16]
|
|
I 149.046243445 etm4_enable_hw: ffff800008ae1e34:
|
|
52800000 mov w0, #0x0 // #0
|
|
I 149.046243445 etm4_enable_hw: ffff800008ae1e38:
|
|
a8c27bfd ldp x29, x30, [sp], #32
|
|
I 149.046243445 etm4_enable_hw: ffff800008ae1e3c:
|
|
d50323bf autiasp
|
|
E 149.046243445 etm4_enable_hw: ffff800008ae1e40:
|
|
d65f03c0 ret
|
|
A ete_sysreg_write: ffff800008adfa18
|
|
|
|
..snip
|
|
|
|
I 149.05422547 panic: ffff800008096300:
|
|
a90363f7 stp x23, x24, [sp, #48]
|
|
I 149.05422547 panic: ffff800008096304:
|
|
6b00003f cmp w1, w0
|
|
I 149.05422547 panic: ffff800008096308:
|
|
3a411804 ccmn w0, #0x1, #0x4, ne // ne = any
|
|
N 149.05422547 panic: ffff80000809630c:
|
|
540001e0 b.eq ffff800008096348 <panic+0xe0> // b.none
|
|
I 149.05422547 panic: ffff800008096310:
|
|
f90023f9 str x25, [sp, #64]
|
|
E 149.05422547 panic: ffff800008096314:
|
|
97fe44ef bl ffff8000080276d0 <panic_smp_self_stop>
|
|
A panic: ffff80000809634c
|
|
I 149.05422547 panic: ffff80000809634c:
|
|
910102d5 add x21, x22, #0x40
|
|
I 149.05422547 panic: ffff800008096350:
|
|
52800020 mov w0, #0x1 // #1
|
|
E 149.05422547 panic: ffff800008096354:
|
|
94166b8b bl ffff800008631180 <bust_spinlocks>
|
|
N 149.054225518 bust_spinlocks: ffff800008631180:
|
|
340000c0 cbz w0, ffff800008631198 <bust_spinlocks+0x18>
|
|
I 149.054225518 bust_spinlocks: ffff800008631184:
|
|
f000a321 adrp x1, ffff800009a98000 <pbufs.0+0xbb8>
|
|
I 149.054225518 bust_spinlocks: ffff800008631188:
|
|
b9405c20 ldr w0, [x1, #92]
|
|
I 149.054225518 bust_spinlocks: ffff80000863118c:
|
|
11000400 add w0, w0, #0x1
|
|
I 149.054225518 bust_spinlocks: ffff800008631190:
|
|
b9005c20 str w0, [x1, #92]
|
|
E 149.054225518 bust_spinlocks: ffff800008631194:
|
|
d65f03c0 ret
|
|
A panic: ffff800008096358
|
|
|
|
Perf based testing
|
|
------------------
|
|
|
|
Starting perf session
|
|
~~~~~~~~~~~~~~~~~~~~~
|
|
ETF::
|
|
|
|
perf record -e cs_etm/panicstop,@tmc_etf1/ -C 1
|
|
perf record -e cs_etm/panicstop,@tmc_etf2/ -C 2
|
|
|
|
ETR::
|
|
|
|
perf record -e cs_etm/panicstop,@tmc_etr0/ -C 1,2
|
|
|
|
Reading trace data after panic
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
Same sysfs based method explained above can be used to retrieve and
|
|
decode the trace data after the reboot on kernel panic.
|