2019-06-04 10:11:33 +02:00
|
|
|
/* SPDX-License-Identifier: GPL-2.0-only */
|
2005-04-16 15:20:36 -07:00
|
|
|
/*
|
|
|
|
|
* linux/arch/arm/vfp/vfpinstr.h
|
|
|
|
|
*
|
|
|
|
|
* Copyright (C) 2004 ARM Limited.
|
|
|
|
|
* Written by Deep Blue Solutions Limited.
|
|
|
|
|
*
|
|
|
|
|
* VFP instruction masks.
|
|
|
|
|
*/
|
|
|
|
|
#define INST_CPRTDO(inst) (((inst) & 0x0f000000) == 0x0e000000)
|
|
|
|
|
#define INST_CPRT(inst) ((inst) & (1 << 4))
|
|
|
|
|
#define INST_CPRT_L(inst) ((inst) & (1 << 20))
|
|
|
|
|
#define INST_CPRT_Rd(inst) (((inst) & (15 << 12)) >> 12)
|
|
|
|
|
#define INST_CPRT_OP(inst) (((inst) >> 21) & 7)
|
|
|
|
|
#define INST_CPNUM(inst) ((inst) & 0xf00)
|
|
|
|
|
#define CPNUM(cp) ((cp) << 8)
|
|
|
|
|
|
|
|
|
|
#define FOP_MASK (0x00b00040)
|
|
|
|
|
#define FOP_FMAC (0x00000000)
|
|
|
|
|
#define FOP_FNMAC (0x00000040)
|
|
|
|
|
#define FOP_FMSC (0x00100000)
|
|
|
|
|
#define FOP_FNMSC (0x00100040)
|
|
|
|
|
#define FOP_FMUL (0x00200000)
|
|
|
|
|
#define FOP_FNMUL (0x00200040)
|
|
|
|
|
#define FOP_FADD (0x00300000)
|
|
|
|
|
#define FOP_FSUB (0x00300040)
|
|
|
|
|
#define FOP_FDIV (0x00800000)
|
|
|
|
|
#define FOP_EXT (0x00b00040)
|
|
|
|
|
|
|
|
|
|
#define FOP_TO_IDX(inst) ((inst & 0x00b00000) >> 20 | (inst & (1 << 6)) >> 4)
|
|
|
|
|
|
|
|
|
|
#define FEXT_MASK (0x000f0080)
|
|
|
|
|
#define FEXT_FCPY (0x00000000)
|
|
|
|
|
#define FEXT_FABS (0x00000080)
|
|
|
|
|
#define FEXT_FNEG (0x00010000)
|
|
|
|
|
#define FEXT_FSQRT (0x00010080)
|
|
|
|
|
#define FEXT_FCMP (0x00040000)
|
|
|
|
|
#define FEXT_FCMPE (0x00040080)
|
|
|
|
|
#define FEXT_FCMPZ (0x00050000)
|
|
|
|
|
#define FEXT_FCMPEZ (0x00050080)
|
|
|
|
|
#define FEXT_FCVT (0x00070080)
|
|
|
|
|
#define FEXT_FUITO (0x00080000)
|
|
|
|
|
#define FEXT_FSITO (0x00080080)
|
|
|
|
|
#define FEXT_FTOUI (0x000c0000)
|
|
|
|
|
#define FEXT_FTOUIZ (0x000c0080)
|
|
|
|
|
#define FEXT_FTOSI (0x000d0000)
|
|
|
|
|
#define FEXT_FTOSIZ (0x000d0080)
|
|
|
|
|
|
|
|
|
|
#define FEXT_TO_IDX(inst) ((inst & 0x000f0000) >> 15 | (inst & (1 << 7)) >> 7)
|
|
|
|
|
|
|
|
|
|
#define vfp_get_sd(inst) ((inst & 0x0000f000) >> 11 | (inst & (1 << 22)) >> 22)
|
2007-09-25 15:22:24 +01:00
|
|
|
#define vfp_get_dd(inst) ((inst & 0x0000f000) >> 12 | (inst & (1 << 22)) >> 18)
|
2005-04-16 15:20:36 -07:00
|
|
|
#define vfp_get_sm(inst) ((inst & 0x0000000f) << 1 | (inst & (1 << 5)) >> 5)
|
2007-09-25 15:22:24 +01:00
|
|
|
#define vfp_get_dm(inst) ((inst & 0x0000000f) | (inst & (1 << 5)) >> 1)
|
2005-04-16 15:20:36 -07:00
|
|
|
#define vfp_get_sn(inst) ((inst & 0x000f0000) >> 15 | (inst & (1 << 7)) >> 7)
|
2007-09-25 15:22:24 +01:00
|
|
|
#define vfp_get_dn(inst) ((inst & 0x000f0000) >> 16 | (inst & (1 << 7)) >> 3)
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
#define vfp_single(inst) (((inst) & 0x0000f00) == 0xa00)
|
|
|
|
|
|
|
|
|
|
#define FPSCR_N (1 << 31)
|
|
|
|
|
#define FPSCR_Z (1 << 30)
|
|
|
|
|
#define FPSCR_C (1 << 29)
|
|
|
|
|
#define FPSCR_V (1 << 28)
|
|
|
|
|
|
ARM: 9410/1: vfp: Use asm volatile in fmrx/fmxr macros
Floating point instructions in userspace can crash some arm kernels
built with clang/LLD 17.0.6:
BUG: unsupported FP instruction in kernel mode
FPEXC == 0xc0000780
Internal error: Oops - undefined instruction: 0 [#1] ARM
CPU: 0 PID: 196 Comm: vfp-reproducer Not tainted 6.10.0 #1
Hardware name: BCM2835
PC is at vfp_support_entry+0xc8/0x2cc
LR is at do_undefinstr+0xa8/0x250
pc : [<c0101d50>] lr : [<c010a80c>] psr: a0000013
sp : dc8d1f68 ip : 60000013 fp : bedea19c
r10: ec532b17 r9 : 00000010 r8 : 0044766c
r7 : c0000780 r6 : ec532b17 r5 : c1c13800 r4 : dc8d1fb0
r3 : c10072c4 r2 : c0101c88 r1 : ec532b17 r0 : 0044766c
Flags: NzCv IRQs on FIQs on Mode SVC_32 ISA ARM Segment none
Control: 00c5387d Table: 0251c008 DAC: 00000051
Register r0 information: non-paged memory
Register r1 information: vmalloc memory
Register r2 information: non-slab/vmalloc memory
Register r3 information: non-slab/vmalloc memory
Register r4 information: 2-page vmalloc region
Register r5 information: slab kmalloc-cg-2k
Register r6 information: vmalloc memory
Register r7 information: non-slab/vmalloc memory
Register r8 information: non-paged memory
Register r9 information: zero-size pointer
Register r10 information: vmalloc memory
Register r11 information: non-paged memory
Register r12 information: non-paged memory
Process vfp-reproducer (pid: 196, stack limit = 0x61aaaf8b)
Stack: (0xdc8d1f68 to 0xdc8d2000)
1f60: 0000081f b6f69300 0000000f c10073f4 c10072c4 dc8d1fb0
1f80: ec532b17 0c532b17 0044766c b6f9ccd8 00000000 c010a80c 00447670 60000010
1fa0: ffffffff c1c13800 00c5387d c0100f10 b6f68af8 00448fc0 00000000 bedea188
1fc0: bedea314 00000001 00448ebc b6f9d000 00447608 b6f9ccd8 00000000 bedea19c
1fe0: bede9198 bedea188 b6e1061c 0044766c 60000010 ffffffff 00000000 00000000
Call trace:
[<c0101d50>] (vfp_support_entry) from [<c010a80c>] (do_undefinstr+0xa8/0x250)
[<c010a80c>] (do_undefinstr) from [<c0100f10>] (__und_usr+0x70/0x80)
Exception stack(0xdc8d1fb0 to 0xdc8d1ff8)
1fa0: b6f68af8 00448fc0 00000000 bedea188
1fc0: bedea314 00000001 00448ebc b6f9d000 00447608 b6f9ccd8 00000000 bedea19c
1fe0: bede9198 bedea188 b6e1061c 0044766c 60000010 ffffffff
Code: 0a000061 e3877202 e594003c e3a09010 (eef16a10)
---[ end trace 0000000000000000 ]---
Kernel panic - not syncing: Fatal exception in interrupt
---[ end Kernel panic - not syncing: Fatal exception in interrupt ]---
This is a minimal userspace reproducer on a Raspberry Pi Zero W:
#include <stdio.h>
#include <math.h>
int main(void)
{
double v = 1.0;
printf("%fn", NAN + *(volatile double *)&v);
return 0;
}
Another way to consistently trigger the oops is:
calvin@raspberry-pi-zero-w ~$ python -c "import json"
The bug reproduces only when the kernel is built with DYNAMIC_DEBUG=n,
because the pr_debug() calls act as barriers even when not activated.
This is the output from the same kernel source built with the same
compiler and DYNAMIC_DEBUG=y, where the userspace reproducer works as
expected:
VFP: bounce: trigger ec532b17 fpexc c0000780
VFP: emulate: INST=0xee377b06 SCR=0x00000000
VFP: bounce: trigger eef1fa10 fpexc c0000780
VFP: emulate: INST=0xeeb40b40 SCR=0x00000000
VFP: raising exceptions 30000000
calvin@raspberry-pi-zero-w ~$ ./vfp-reproducer
nan
Crudely grepping for vmsr/vmrs instructions in the otherwise nearly
idential text for vfp_support_entry() makes the problem obvious:
vmlinux.llvm.good [0xc0101cb8] <+48>: vmrs r7, fpexc
vmlinux.llvm.good [0xc0101cd8] <+80>: vmsr fpexc, r0
vmlinux.llvm.good [0xc0101d20] <+152>: vmsr fpexc, r7
vmlinux.llvm.good [0xc0101d38] <+176>: vmrs r4, fpexc
vmlinux.llvm.good [0xc0101d6c] <+228>: vmrs r0, fpscr
vmlinux.llvm.good [0xc0101dc4] <+316>: vmsr fpexc, r0
vmlinux.llvm.good [0xc0101dc8] <+320>: vmrs r0, fpsid
vmlinux.llvm.good [0xc0101dcc] <+324>: vmrs r6, fpscr
vmlinux.llvm.good [0xc0101e10] <+392>: vmrs r10, fpinst
vmlinux.llvm.good [0xc0101eb8] <+560>: vmrs r10, fpinst2
vmlinux.llvm.bad [0xc0101cb8] <+48>: vmrs r7, fpexc
vmlinux.llvm.bad [0xc0101cd8] <+80>: vmsr fpexc, r0
vmlinux.llvm.bad [0xc0101d20] <+152>: vmsr fpexc, r7
vmlinux.llvm.bad [0xc0101d30] <+168>: vmrs r0, fpscr
vmlinux.llvm.bad [0xc0101d50] <+200>: vmrs r6, fpscr <== BOOM!
vmlinux.llvm.bad [0xc0101d6c] <+228>: vmsr fpexc, r0
vmlinux.llvm.bad [0xc0101d70] <+232>: vmrs r0, fpsid
vmlinux.llvm.bad [0xc0101da4] <+284>: vmrs r10, fpinst
vmlinux.llvm.bad [0xc0101df8] <+368>: vmrs r4, fpexc
vmlinux.llvm.bad [0xc0101e5c] <+468>: vmrs r10, fpinst2
I think LLVM's reordering is valid as the code is currently written: the
compiler doesn't know the instructions have side effects in hardware.
Fix by using "asm volatile" in fmxr() and fmrx(), so they cannot be
reordered with respect to each other. The original compiler now produces
working kernels on my hardware with DYNAMIC_DEBUG=n.
This is the relevant piece of the diff of the vfp_support_entry() text,
from the original oopsing kernel to a working kernel with this patch:
vmrs r0, fpscr
tst r0, #4096
bne 0xc0101d48
tst r0, #458752
beq 0xc0101ecc
orr r7, r7, #536870912
ldr r0, [r4, #0x3c]
mov r9, #16
-vmrs r6, fpscr
orr r9, r9, #251658240
add r0, r0, #4
str r0, [r4, #0x3c]
mvn r0, #159
sub r0, r0, #-1207959552
and r0, r7, r0
vmsr fpexc, r0
vmrs r0, fpsid
+vmrs r6, fpscr
and r0, r0, #983040
cmp r0, #65536
bne 0xc0101d88
Fixes: 4708fb041346 ("ARM: vfp: Reimplement VFP exception entry in C code")
Signed-off-by: Calvin Owens <calvin@wbinvd.org>
Signed-off-by: Russell King (Oracle) <rmk+kernel@armlinux.org.uk>
2024-07-29 17:05:51 +01:00
|
|
|
#define fmrx(_vfp_) ({ \
|
|
|
|
|
u32 __v; \
|
|
|
|
|
asm volatile (".fpu vfpv2\n" \
|
|
|
|
|
"vmrs %0, " #_vfp_ \
|
|
|
|
|
: "=r" (__v) : : "cc"); \
|
|
|
|
|
__v; \
|
|
|
|
|
})
|
|
|
|
|
|
|
|
|
|
#define fmxr(_vfp_, _var_) ({ \
|
|
|
|
|
asm volatile (".fpu vfpv2\n" \
|
|
|
|
|
"vmsr " #_vfp_ ", %0" \
|
|
|
|
|
: : "r" (_var_) : "cc"); \
|
|
|
|
|
})
|
ARM: 8991/1: use VFP assembler mnemonics if available
The integrated assembler of Clang 10 and earlier do not allow to access
the VFP registers through the coprocessor load/store instructions:
arch/arm/vfp/vfpmodule.c:342:2: error: invalid operand for instruction
fmxr(FPEXC, fpexc & ~(FPEXC_EX|FPEXC_DEX|FPEXC_FP2V|FPEXC_VV|FPEXC_TRAP_MASK));
^
arch/arm/vfp/vfpinstr.h:79:6: note: expanded from macro 'fmxr'
asm("mcr p10, 7, %0, " vfpreg(_vfp_) ", cr0, 0 @ fmxr " #_vfp_ ", %0"
^
<inline asm>:1:6: note: instantiated into assembly here
mcr p10, 7, r0, cr8, cr0, 0 @ fmxr FPEXC, r0
^
This has been addressed with Clang 11 [0]. However, to support earlier
versions of Clang and for better readability use of VFP assembler
mnemonics still is preferred.
Ideally we would replace this code with the unified assembler language
mnemonics vmrs/vmsr on call sites along with .fpu assembler directives.
The GNU assembler supports the .fpu directive at least since 2.17 (when
documentation has been added). Since Linux requires binutils 2.21 it is
safe to use .fpu directive. However, binutils does not allow to use
FPINST or FPINST2 as an argument to vmrs/vmsr instructions up to
binutils 2.24 (see binutils commit 16d02dc907c5):
arch/arm/vfp/vfphw.S: Assembler messages:
arch/arm/vfp/vfphw.S:162: Error: operand 0 must be FPSID or FPSCR pr FPEXC -- `vmsr FPINST,r6'
arch/arm/vfp/vfphw.S:165: Error: operand 0 must be FPSID or FPSCR pr FPEXC -- `vmsr FPINST2,r8'
arch/arm/vfp/vfphw.S:235: Error: operand 1 must be a VFP extension System Register -- `vmrs r3,FPINST'
arch/arm/vfp/vfphw.S:238: Error: operand 1 must be a VFP extension System Register -- `vmrs r12,FPINST2'
Use as-instr in Kconfig to check if FPINST/FPINST2 can be used. If they
can be used make use of .fpu directives and UAL VFP mnemonics for
register access.
This allows to build vfpmodule.c with Clang and its integrated assembler.
[0] https://reviews.llvm.org/D59733
Link: https://github.com/ClangBuiltLinux/linux/issues/905
Signed-off-by: Stefan Agner <stefan@agner.ch>
Signed-off-by: Russell King <rmk+kernel@armlinux.org.uk>
2020-07-09 11:21:27 +01:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
u32 vfp_single_cpdo(u32 inst, u32 fpscr);
|
|
|
|
|
u32 vfp_single_cprt(u32 inst, u32 fpscr, struct pt_regs *regs);
|
|
|
|
|
|
|
|
|
|
u32 vfp_double_cpdo(u32 inst, u32 fpscr);
|