public-mirrors/linux
1
0
Fork 0
mirror of git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git synced 2025-08-05 16:54:27 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

* Mitigate Indirect Target Selection (ITS) issue

Browse source 
-----BEGIN PGP SIGNATURE-----
 
 iQIzBAABCgAdFiEEV76QKkVc4xCGURexaDWVMHDJkrAFAmgebIwACgkQaDWVMHDJ
 krCGSA/+I+W/uqiz58Z2Zu4RrXMYFfKJxacF7My9wnOyRxaJduS3qrz1E5wHqBId
 f6M8wDx9nS24UxDkBbi84NdtlG1zj8nV8djtszGKVeqHG2DcQMMOXBKZSjOmTo2b
 GIZ3a3xEqXaFfnGQxXSZrvtHIwCmv10H2oyGHu0vBp/SJuWXNg72oivOGhbm0uWs
 0/bdIK8+1sW7OAmhhKdvMVpmzL8TQJnkUHSkQilPB2Tsf9wWDfeY7kDkK5YwQpk2
 ZK+hrmwCFXQZELY65F2+y/cFim/F38HiqVdvIkV1wFSVqVVE9hEKJ4BDZl1fXZKB
 p4qpDFgxO27E/eMo9IZfxRH4TdSoK6YLWo9FGWHKBPnciJfAeO9EP/AwAIhEQRdx
 YZlN9sGS6ja7O1Eh423BBw6cFj6ta0ck2T1PoYk32FXc6sgqCphsfvBD3+tJxz8/
 xoZ3BzoErdPqSXbH5cSI972kQW0JLESiMTZa827qnJtT672t6uBcsnnmR0ZbJH1f
 TJCC9qgwpBiEkiGW3gwv00SC7CkXo3o0FJw0pa3MkKHGd7csxBtGBHI1b6Jj+oB0
 yWf1HxSqwrq2Yek8R7lWd4jIxyWfKriEMTu7xCMUUFlprKmR2RufsADvqclNyedQ
 sGBCc4eu1cpZp2no/IFm+IvkuzUHnkS/WNL1LbZ9YI8h8unjZHE=
 =UVgZ
 -----END PGP SIGNATURE-----

Merge tag 'its-for-linus-20250509' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip

Pull x86 ITS mitigation from Dave Hansen:
 "Mitigate Indirect Target Selection (ITS) issue.

  I'd describe this one as a good old CPU bug where the behavior is
  _obviously_ wrong, but since it just results in bad predictions it
  wasn't wrong enough to notice. Well, the researchers noticed and also
  realized that thus bug undermined a bunch of existing indirect branch
  mitigations.

  Thus the unusually wide impact on this one. Details:

  ITS is a bug in some Intel CPUs that affects indirect branches
  including RETs in the first half of a cacheline. Due to ITS such
  branches may get wrongly predicted to a target of (direct or indirect)
  branch that is located in the second half of a cacheline. Researchers
  at VUSec found this behavior and reported to Intel.

  Affected processors:

   - Cascade Lake, Cooper Lake, Whiskey Lake V, Coffee Lake R, Comet
     Lake, Ice Lake, Tiger Lake and Rocket Lake.

  Scope of impact:

   - Guest/host isolation:

     When eIBRS is used for guest/host isolation, the indirect branches
     in the VMM may still be predicted with targets corresponding to
     direct branches in the guest.

   - Intra-mode using cBPF:

     cBPF can be used to poison the branch history to exploit ITS.
     Realigning the indirect branches and RETs mitigates this attack
     vector.

   - User/kernel:

     With eIBRS enabled user/kernel isolation is *not* impacted by ITS.

   - Indirect Branch Prediction Barrier (IBPB):

     Due to this bug indirect branches may be predicted with targets
     corresponding to direct branches which were executed prior to IBPB.
     This will be fixed in the microcode.

  Mitigation:

  As indirect branches in the first half of cacheline are affected, the
  mitigation is to replace those indirect branches with a call to thunk that
  is aligned to the second half of the cacheline.

  RETs that take prediction from RSB are not affected, but they may be
  affected by RSB-underflow condition. So, RETs in the first half of
  cacheline are also patched to a return thunk that executes the RET aligned
  to second half of cacheline"

* tag 'its-for-linus-20250509' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip:
  selftest/x86/bugs: Add selftests for ITS
  x86/its: FineIBT-paranoid vs ITS
  x86/its: Use dynamic thunks for indirect branches
  x86/ibt: Keep IBT disabled during alternative patching
  mm/execmem: Unify early execmem_cache behaviour
  x86/its: Align RETs in BHB clear sequence to avoid thunking
  x86/its: Add support for RSB stuffing mitigation
  x86/its: Add "vmexit" option to skip mitigation on some CPUs
  x86/its: Enable Indirect Target Selection mitigation
  x86/its: Add support for ITS-safe return thunk
  x86/its: Add support for ITS-safe indirect thunk
  x86/its: Enumerate Indirect Target Selection (ITS) bug
  Documentation: x86/bugs/its: Add ITS documentation
This commit is contained in:
Linus Torvalds 2025-05-11 17:23:03 -07:00
commit 6f5bf947ba
35 changed files with 1581 additions and 49 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

1
Documentation/ABI/testing/sysfs-devices-system-cpu
View file

@ -511,6 +511,7 @@ Description: information about CPUs heterogeneity.
What: /sys/devices/system/cpu/vulnerabilities What: /sys/devices/system/cpu/vulnerabilities
/sys/devices/system/cpu/vulnerabilities/gather_data_sampling /sys/devices/system/cpu/vulnerabilities/gather_data_sampling
/sys/devices/system/cpu/vulnerabilities/indirect_target_selection
/sys/devices/system/cpu/vulnerabilities/itlb_multihit /sys/devices/system/cpu/vulnerabilities/itlb_multihit
/sys/devices/system/cpu/vulnerabilities/l1tf /sys/devices/system/cpu/vulnerabilities/l1tf
/sys/devices/system/cpu/vulnerabilities/mds /sys/devices/system/cpu/vulnerabilities/mds

1
Documentation/admin-guide/hw-vuln/index.rst
View file

@ -23,3 +23,4 @@ are configurable at compile, boot or run time.
gather_data_sampling gather_data_sampling
reg-file-data-sampling reg-file-data-sampling
rsb rsb
indirect-target-selection

168
Documentation/admin-guide/hw-vuln/indirect-target-selection.rst Normal file
View file

@ -0,0 +1,168 @@
.. SPDX-License-Identifier: GPL-2.0
Indirect Target Selection (ITS)
===============================
ITS is a vulnerability in some Intel CPUs that support Enhanced IBRS and were
released before Alder Lake. ITS may allow an attacker to control the prediction
of indirect branches and RETs located in the lower half of a cacheline.
ITS is assigned CVE-2024-28956 with a CVSS score of 4.7 (Medium).
Scope of Impact
---------------
- **eIBRS Guest/Host Isolation**: Indirect branches in KVM/kernel may still be
predicted with unintended target corresponding to a branch in the guest.
- **Intra-Mode BTI**: In-kernel training such as through cBPF or other native
gadgets.
- **Indirect Branch Prediction Barrier (IBPB)**: After an IBPB, indirect
branches may still be predicted with targets corresponding to direct branches
executed prior to the IBPB. This is fixed by the IPU 2025.1 microcode, which
should be available via distro updates. Alternatively microcode can be
obtained from Intel's github repository [#f1]_.
Affected CPUs
-------------
Below is the list of ITS affected CPUs [#f2]_ [#f3]_:
======================== ============ ==================== ===============
Common name Family_Model eIBRS Intra-mode BTI
Guest/Host Isolation
======================== ============ ==================== ===============
SKYLAKE_X (step >= 6) 06_55H Affected Affected
ICELAKE_X 06_6AH Not affected Affected
ICELAKE_D 06_6CH Not affected Affected
ICELAKE_L 06_7EH Not affected Affected
TIGERLAKE_L 06_8CH Not affected Affected
TIGERLAKE 06_8DH Not affected Affected
KABYLAKE_L (step >= 12) 06_8EH Affected Affected
KABYLAKE (step >= 13) 06_9EH Affected Affected
COMETLAKE 06_A5H Affected Affected
COMETLAKE_L 06_A6H Affected Affected
ROCKETLAKE 06_A7H Not affected Affected
======================== ============ ==================== ===============
- All affected CPUs enumerate Enhanced IBRS feature.
- IBPB isolation is affected on all ITS affected CPUs, and need a microcode
update for mitigation.
- None of the affected CPUs enumerate BHI_CTRL which was introduced in Golden
Cove (Alder Lake and Sapphire Rapids). This can help guests to determine the
host's affected status.
- Intel Atom CPUs are not affected by ITS.
Mitigation
----------
As only the indirect branches and RETs that have their last byte of instruction
in the lower half of the cacheline are vulnerable to ITS, the basic idea behind
the mitigation is to not allow indirect branches in the lower half.
This is achieved by relying on existing retpoline support in the kernel, and in
compilers. ITS-vulnerable retpoline sites are runtime patched to point to newly
added ITS-safe thunks. These safe thunks consists of indirect branch in the
second half of the cacheline. Not all retpoline sites are patched to thunks, if
a retpoline site is evaluated to be ITS-safe, it is replaced with an inline
indirect branch.
Dynamic thunks
~~~~~~~~~~~~~~
From a dynamically allocated pool of safe-thunks, each vulnerable site is
replaced with a new thunk, such that they get a unique address. This could
improve the branch prediction accuracy. Also, it is a defense-in-depth measure
against aliasing.
Note, for simplicity, indirect branches in eBPF programs are always replaced
with a jump to a static thunk in __x86_indirect_its_thunk_array. If required,
in future this can be changed to use dynamic thunks.
All vulnerable RETs are replaced with a static thunk, they do not use dynamic
thunks. This is because RETs get their prediction from RSB mostly that does not
depend on source address. RETs that underflow RSB may benefit from dynamic
thunks. But, RETs significantly outnumber indirect branches, and any benefit
from a unique source address could be outweighed by the increased icache
footprint and iTLB pressure.
Retpoline
~~~~~~~~~
Retpoline sequence also mitigates ITS-unsafe indirect branches. For this
reason, when retpoline is enabled, ITS mitigation only relocates the RETs to
safe thunks. Unless user requested the RSB-stuffing mitigation.
RSB Stuffing
~~~~~~~~~~~~
RSB-stuffing via Call Depth Tracking is a mitigation for Retbleed RSB-underflow
attacks. And it also mitigates RETs that are vulnerable to ITS.
Mitigation in guests
^^^^^^^^^^^^^^^^^^^^
All guests deploy ITS mitigation by default, irrespective of eIBRS enumeration
and Family/Model of the guest. This is because eIBRS feature could be hidden
from a guest. One exception to this is when a guest enumerates BHI_DIS_S, which
indicates that the guest is running on an unaffected host.
To prevent guests from unnecessarily deploying the mitigation on unaffected
platforms, Intel has defined ITS_NO bit(62) in MSR IA32_ARCH_CAPABILITIES. When
a guest sees this bit set, it should not enumerate the ITS bug. Note, this bit
is not set by any hardware, but is **intended for VMMs to synthesize** it for
guests as per the host's affected status.
Mitigation options
^^^^^^^^^^^^^^^^^^
The ITS mitigation can be controlled using the "indirect_target_selection"
kernel parameter. The available options are:
======== ===================================================================
on (default) Deploy the "Aligned branch/return thunks" mitigation.
If spectre_v2 mitigation enables retpoline, aligned-thunks are only
deployed for the affected RET instructions. Retpoline mitigates
indirect branches.
off Disable ITS mitigation.
vmexit Equivalent to "=on" if the CPU is affected by guest/host isolation
part of ITS. Otherwise, mitigation is not deployed. This option is
useful when host userspace is not in the threat model, and only
attacks from guest to host are considered.
stuff Deploy RSB-fill mitigation when retpoline is also deployed.
Otherwise, deploy the default mitigation. When retpoline mitigation
is enabled, RSB-stuffing via Call-Depth-Tracking also mitigates
ITS.
force Force the ITS bug and deploy the default mitigation.
======== ===================================================================
Sysfs reporting
---------------
The sysfs file showing ITS mitigation status is:
/sys/devices/system/cpu/vulnerabilities/indirect_target_selection
Note, microcode mitigation status is not reported in this file.
The possible values in this file are:
.. list-table::
* - Not affected
- The processor is not vulnerable.
* - Vulnerable
- System is vulnerable and no mitigation has been applied.
* - Vulnerable, KVM: Not affected
- System is vulnerable to intra-mode BTI, but not affected by eIBRS
guest/host isolation.
* - Mitigation: Aligned branch/return thunks
- The mitigation is enabled, affected indirect branches and RETs are
relocated to safe thunks.
* - Mitigation: Retpolines, Stuffing RSB
- The mitigation is enabled using retpoline and RSB stuffing.
References
----------
.. [#f1] Microcode repository - https://github.com/intel/Intel-Linux-Processor-Microcode-Data-Files
.. [#f2] Affected Processors list - https://www.intel.com/content/www/us/en/developer/topic-technology/software-security-guidance/processors-affected-consolidated-product-cpu-model.html
.. [#f3] Affected Processors list (machine readable) - https://github.com/intel/Intel-affected-processor-list

18
Documentation/admin-guide/kernel-parameters.txt
View file

@ -2202,6 +2202,23 @@
different crypto accelerators. This option can be used different crypto accelerators. This option can be used
to achieve best performance for particular HW. to achieve best performance for particular HW.
indirect_target_selection= [X86,Intel] Mitigation control for Indirect
Target Selection(ITS) bug in Intel CPUs. Updated
microcode is also required for a fix in IBPB.
on: Enable mitigation (default).
off: Disable mitigation.
force: Force the ITS bug and deploy default
mitigation.
vmexit: Only deploy mitigation if CPU is affected by
guest/host isolation part of ITS.
stuff: Deploy RSB-fill mitigation when retpoline is
also deployed. Otherwise, deploy the default
mitigation.
For details see:
Documentation/admin-guide/hw-vuln/indirect-target-selection.rst
init= [KNL] init= [KNL]
Format: <full_path> Format: <full_path>
Run specified binary instead of /sbin/init as init Run specified binary instead of /sbin/init as init
@ -3693,6 +3710,7 @@
expose users to several CPU vulnerabilities. expose users to several CPU vulnerabilities.
Equivalent to: if nokaslr then kpti=0 [ARM64] Equivalent to: if nokaslr then kpti=0 [ARM64]
gather_data_sampling=off [X86] gather_data_sampling=off [X86]
indirect_target_selection=off [X86]
kvm.nx_huge_pages=off [X86] kvm.nx_huge_pages=off [X86]
l1tf=off [X86] l1tf=off [X86]
mds=off [X86] mds=off [X86]

12
arch/x86/Kconfig
View file

@ -2711,6 +2711,18 @@ config MITIGATION_SSB
of speculative execution in a similar way to the Meltdown and Spectre of speculative execution in a similar way to the Meltdown and Spectre
security vulnerabilities. security vulnerabilities.
config MITIGATION_ITS
bool "Enable Indirect Target Selection mitigation"
depends on CPU_SUP_INTEL && X86_64
depends on MITIGATION_RETPOLINE && MITIGATION_RETHUNK
select EXECMEM
default y
help
Enable Indirect Target Selection (ITS) mitigation. ITS is a bug in
BPU on some Intel CPUs that may allow Spectre V2 style attacks. If
disabled, mitigation cannot be enabled via cmdline.
See <file:Documentation/admin-guide/hw-vuln/indirect-target-selection.rst>
endif endif
config ARCH_HAS_ADD_PAGES config ARCH_HAS_ADD_PAGES

20
arch/x86/entry/entry_64.S
View file

@ -1525,7 +1525,9 @@ SYM_CODE_END(rewind_stack_and_make_dead)
* ORC to unwind properly. * ORC to unwind properly.
* *
* The alignment is for performance and not for safety, and may be safely * The alignment is for performance and not for safety, and may be safely
* refactored in the future if needed. * refactored in the future if needed. The .skips are for safety, to ensure
* that all RETs are in the second half of a cacheline to mitigate Indirect
* Target Selection, rather than taking the slowpath via its_return_thunk.
*/ */
SYM_FUNC_START(clear_bhb_loop) SYM_FUNC_START(clear_bhb_loop)
ANNOTATE_NOENDBR ANNOTATE_NOENDBR
@ -1536,10 +1538,22 @@ SYM_FUNC_START(clear_bhb_loop)
call 1f call 1f
jmp 5f jmp 5f
.align 64, 0xcc .align 64, 0xcc
/*
* Shift instructions so that the RET is in the upper half of the
* cacheline and don't take the slowpath to its_return_thunk.
*/
.skip 32 - (.Lret1 - 1f), 0xcc
ANNOTATE_INTRA_FUNCTION_CALL ANNOTATE_INTRA_FUNCTION_CALL
1: call 2f 1: call 2f
RET .Lret1: RET
.align 64, 0xcc .align 64, 0xcc
/*
* As above shift instructions for RET at .Lret2 as well.
*
* This should be ideally be: .skip 32 - (.Lret2 - 2f), 0xcc
* but some Clang versions (e.g. 18) don't like this.
*/
.skip 32 - 18, 0xcc
2: movl $5, %eax 2: movl $5, %eax
3: jmp 4f 3: jmp 4f
nop nop
@ -1547,7 +1561,7 @@ SYM_FUNC_START(clear_bhb_loop)
jnz 3b jnz 3b
sub $1, %ecx sub $1, %ecx
jnz 1b jnz 1b
RET .Lret2: RET
5: lfence 5: lfence
pop %rbp pop %rbp
RET RET

32
arch/x86/include/asm/alternative.h
View file

@ -6,6 +6,7 @@
#include <linux/stringify.h> #include <linux/stringify.h>
#include <linux/objtool.h> #include <linux/objtool.h>
#include <asm/asm.h> #include <asm/asm.h>
#include <asm/bug.h>
#define ALT_FLAGS_SHIFT 16 #define ALT_FLAGS_SHIFT 16
@ -124,6 +125,37 @@ static __always_inline int x86_call_depth_emit_accounting(u8 **pprog,
} }
#endif #endif
#ifdef CONFIG_MITIGATION_ITS
extern void its_init_mod(struct module *mod);
extern void its_fini_mod(struct module *mod);
extern void its_free_mod(struct module *mod);
extern u8 *its_static_thunk(int reg);
#else /* CONFIG_MITIGATION_ITS */
static inline void its_init_mod(struct module *mod) { }
static inline void its_fini_mod(struct module *mod) { }
static inline void its_free_mod(struct module *mod) { }
static inline u8 *its_static_thunk(int reg)
{
WARN_ONCE(1, "ITS not compiled in");
return NULL;
}
#endif
#if defined(CONFIG_MITIGATION_RETHUNK) && defined(CONFIG_OBJTOOL)
extern bool cpu_wants_rethunk(void);
extern bool cpu_wants_rethunk_at(void *addr);
#else
static __always_inline bool cpu_wants_rethunk(void)
{
return false;
}
static __always_inline bool cpu_wants_rethunk_at(void *addr)
{
return false;
}
#endif
#ifdef CONFIG_SMP #ifdef CONFIG_SMP
extern void alternatives_smp_module_add(struct module *mod, char *name, extern void alternatives_smp_module_add(struct module *mod, char *name,
void *locks, void *locks_end, void *locks, void *locks_end,

3
arch/x86/include/asm/cpufeatures.h
View file

@ -481,6 +481,7 @@
#define X86_FEATURE_AMD_HETEROGENEOUS_CORES (21*32 + 6) /* Heterogeneous Core Topology */ #define X86_FEATURE_AMD_HETEROGENEOUS_CORES (21*32 + 6) /* Heterogeneous Core Topology */
#define X86_FEATURE_AMD_WORKLOAD_CLASS (21*32 + 7) /* Workload Classification */ #define X86_FEATURE_AMD_WORKLOAD_CLASS (21*32 + 7) /* Workload Classification */
#define X86_FEATURE_PREFER_YMM (21*32 + 8) /* Avoid ZMM registers due to downclocking */ #define X86_FEATURE_PREFER_YMM (21*32 + 8) /* Avoid ZMM registers due to downclocking */
#define X86_FEATURE_INDIRECT_THUNK_ITS (21*32 + 9) /* Use thunk for indirect branches in lower half of cacheline */
/* /*
* BUG word(s) * BUG word(s)
@ -533,4 +534,6 @@
#define X86_BUG_BHI X86_BUG(1*32 + 3) /* "bhi" CPU is affected by Branch History Injection */ #define X86_BUG_BHI X86_BUG(1*32 + 3) /* "bhi" CPU is affected by Branch History Injection */
#define X86_BUG_IBPB_NO_RET X86_BUG(1*32 + 4) /* "ibpb_no_ret" IBPB omits return target predictions */ #define X86_BUG_IBPB_NO_RET X86_BUG(1*32 + 4) /* "ibpb_no_ret" IBPB omits return target predictions */
#define X86_BUG_SPECTRE_V2_USER X86_BUG(1*32 + 5) /* "spectre_v2_user" CPU is affected by Spectre variant 2 attack between user processes */ #define X86_BUG_SPECTRE_V2_USER X86_BUG(1*32 + 5) /* "spectre_v2_user" CPU is affected by Spectre variant 2 attack between user processes */
#define X86_BUG_ITS X86_BUG(1*32 + 6) /* "its" CPU is affected by Indirect Target Selection */
#define X86_BUG_ITS_NATIVE_ONLY X86_BUG(1*32 + 7) /* "its_native_only" CPU is affected by ITS, VMX is not affected */
#endif /* _ASM_X86_CPUFEATURES_H */ #endif /* _ASM_X86_CPUFEATURES_H */

8
arch/x86/include/asm/msr-index.h
View file

@ -211,6 +211,14 @@
* VERW clears CPU Register * VERW clears CPU Register
* File. * File.
*/ */
#define ARCH_CAP_ITS_NO BIT_ULL(62) /*
* Not susceptible to
* Indirect Target Selection.
* This bit is not set by
* HW, but is synthesized by
* VMMs for guests to know
* their affected status.
*/
#define MSR_IA32_FLUSH_CMD 0x0000010b #define MSR_IA32_FLUSH_CMD 0x0000010b
#define L1D_FLUSH BIT(0) /* #define L1D_FLUSH BIT(0) /*

10
arch/x86/include/asm/nospec-branch.h
View file

@ -336,10 +336,14 @@
#else /* __ASSEMBLER__ */ #else /* __ASSEMBLER__ */
#define ITS_THUNK_SIZE 64
typedef u8 retpoline_thunk_t[RETPOLINE_THUNK_SIZE]; typedef u8 retpoline_thunk_t[RETPOLINE_THUNK_SIZE];
typedef u8 its_thunk_t[ITS_THUNK_SIZE];
extern retpoline_thunk_t __x86_indirect_thunk_array[]; extern retpoline_thunk_t __x86_indirect_thunk_array[];
extern retpoline_thunk_t __x86_indirect_call_thunk_array[]; extern retpoline_thunk_t __x86_indirect_call_thunk_array[];
extern retpoline_thunk_t __x86_indirect_jump_thunk_array[]; extern retpoline_thunk_t __x86_indirect_jump_thunk_array[];
extern its_thunk_t __x86_indirect_its_thunk_array[];
#ifdef CONFIG_MITIGATION_RETHUNK #ifdef CONFIG_MITIGATION_RETHUNK
extern void __x86_return_thunk(void); extern void __x86_return_thunk(void);
@ -363,6 +367,12 @@ static inline void srso_return_thunk(void) {}
static inline void srso_alias_return_thunk(void) {} static inline void srso_alias_return_thunk(void) {}
#endif #endif
#ifdef CONFIG_MITIGATION_ITS
extern void its_return_thunk(void);
#else
static inline void its_return_thunk(void) {}
#endif
extern void retbleed_return_thunk(void); extern void retbleed_return_thunk(void);
extern void srso_return_thunk(void); extern void srso_return_thunk(void);
extern void srso_alias_return_thunk(void); extern void srso_alias_return_thunk(void);

336
arch/x86/kernel/alternative.c
View file

@ -18,6 +18,7 @@
#include <linux/mmu_context.h> #include <linux/mmu_context.h>
#include <linux/bsearch.h> #include <linux/bsearch.h>
#include <linux/sync_core.h> #include <linux/sync_core.h>
#include <linux/execmem.h>
#include <asm/text-patching.h> #include <asm/text-patching.h>
#include <asm/alternative.h> #include <asm/alternative.h>
#include <asm/sections.h> #include <asm/sections.h>
@ -31,6 +32,8 @@
#include <asm/paravirt.h> #include <asm/paravirt.h>
#include <asm/asm-prototypes.h> #include <asm/asm-prototypes.h>
#include <asm/cfi.h> #include <asm/cfi.h>
#include <asm/ibt.h>
#include <asm/set_memory.h>
int __read_mostly alternatives_patched; int __read_mostly alternatives_patched;
@ -124,6 +127,165 @@ const unsigned char * const x86_nops[ASM_NOP_MAX+1] =
#endif #endif
}; };
#ifdef CONFIG_FINEIBT
static bool cfi_paranoid __ro_after_init;
#endif
#ifdef CONFIG_MITIGATION_ITS
static struct module *its_mod;
static void *its_page;
static unsigned int its_offset;
/* Initialize a thunk with the "jmp *reg; int3" instructions. */
static void *its_init_thunk(void *thunk, int reg)
{
u8 *bytes = thunk;
int offset = 0;
int i = 0;
#ifdef CONFIG_FINEIBT
if (cfi_paranoid) {
/*
* When ITS uses indirect branch thunk the fineibt_paranoid
* caller sequence doesn't fit in the caller site. So put the
* remaining part of the sequence (<ea> + JNE) into the ITS
* thunk.
*/
bytes[i++] = 0xea; /* invalid instruction */
bytes[i++] = 0x75; /* JNE */
bytes[i++] = 0xfd;
offset = 1;
}
#endif
if (reg >= 8) {
bytes[i++] = 0x41; /* REX.B prefix */
reg -= 8;
}
bytes[i++] = 0xff;
bytes[i++] = 0xe0 + reg; /* jmp *reg */
bytes[i++] = 0xcc;
return thunk + offset;
}
void its_init_mod(struct module *mod)
{
if (!cpu_feature_enabled(X86_FEATURE_INDIRECT_THUNK_ITS))
return;
mutex_lock(&text_mutex);
its_mod = mod;
its_page = NULL;
}
void its_fini_mod(struct module *mod)
{
if (!cpu_feature_enabled(X86_FEATURE_INDIRECT_THUNK_ITS))
return;
WARN_ON_ONCE(its_mod != mod);
its_mod = NULL;
its_page = NULL;
mutex_unlock(&text_mutex);
for (int i = 0; i < mod->its_num_pages; i++) {
void *page = mod->its_page_array[i];
execmem_restore_rox(page, PAGE_SIZE);
}
}
void its_free_mod(struct module *mod)
{
if (!cpu_feature_enabled(X86_FEATURE_INDIRECT_THUNK_ITS))
return;
for (int i = 0; i < mod->its_num_pages; i++) {
void *page = mod->its_page_array[i];
execmem_free(page);
}
kfree(mod->its_page_array);
}
static void *its_alloc(void)
{
void *page __free(execmem) = execmem_alloc(EXECMEM_MODULE_TEXT, PAGE_SIZE);
if (!page)
return NULL;
if (its_mod) {
void *tmp = krealloc(its_mod->its_page_array,
(its_mod->its_num_pages+1) * sizeof(void *),
GFP_KERNEL);
if (!tmp)
return NULL;
its_mod->its_page_array = tmp;
its_mod->its_page_array[its_mod->its_num_pages++] = page;
execmem_make_temp_rw(page, PAGE_SIZE);
}
return no_free_ptr(page);
}
static void *its_allocate_thunk(int reg)
{
int size = 3 + (reg / 8);
void *thunk;
#ifdef CONFIG_FINEIBT
/*
* The ITS thunk contains an indirect jump and an int3 instruction so
* its size is 3 or 4 bytes depending on the register used. If CFI
* paranoid is used then 3 extra bytes are added in the ITS thunk to
* complete the fineibt_paranoid caller sequence.
*/
if (cfi_paranoid)
size += 3;
#endif
if (!its_page || (its_offset + size - 1) >= PAGE_SIZE) {
its_page = its_alloc();
if (!its_page) {
pr_err("ITS page allocation failed\n");
return NULL;
}
memset(its_page, INT3_INSN_OPCODE, PAGE_SIZE);
its_offset = 32;
}
/*
* If the indirect branch instruction will be in the lower half
* of a cacheline, then update the offset to reach the upper half.
*/
if ((its_offset + size - 1) % 64 < 32)
its_offset = ((its_offset - 1) | 0x3F) + 33;
thunk = its_page + its_offset;
its_offset += size;
return its_init_thunk(thunk, reg);
}
u8 *its_static_thunk(int reg)
{
u8 *thunk = __x86_indirect_its_thunk_array[reg];
#ifdef CONFIG_FINEIBT
/* Paranoid thunk starts 2 bytes before */
if (cfi_paranoid)
return thunk - 2;
#endif
return thunk;
}
#endif
/* /*
* Nomenclature for variable names to simplify and clarify this code and ease * Nomenclature for variable names to simplify and clarify this code and ease
* any potential staring at it: * any potential staring at it:
@ -581,7 +743,8 @@ static int emit_indirect(int op, int reg, u8 *bytes)
return i; return i;
} }
static int emit_call_track_retpoline(void *addr, struct insn *insn, int reg, u8 *bytes) static int __emit_trampoline(void *addr, struct insn *insn, u8 *bytes,
void *call_dest, void *jmp_dest)
{ {
u8 op = insn->opcode.bytes[0]; u8 op = insn->opcode.bytes[0];
int i = 0; int i = 0;
@ -602,7 +765,7 @@ static int emit_call_track_retpoline(void *addr, struct insn *insn, int reg, u8
switch (op) { switch (op) {
case CALL_INSN_OPCODE: case CALL_INSN_OPCODE:
__text_gen_insn(bytes+i, op, addr+i, __text_gen_insn(bytes+i, op, addr+i,
__x86_indirect_call_thunk_array[reg], call_dest,
CALL_INSN_SIZE); CALL_INSN_SIZE);
i += CALL_INSN_SIZE; i += CALL_INSN_SIZE;
break; break;
@ -610,7 +773,7 @@ static int emit_call_track_retpoline(void *addr, struct insn *insn, int reg, u8
case JMP32_INSN_OPCODE: case JMP32_INSN_OPCODE:
clang_jcc: clang_jcc:
__text_gen_insn(bytes+i, op, addr+i, __text_gen_insn(bytes+i, op, addr+i,
__x86_indirect_jump_thunk_array[reg], jmp_dest,
JMP32_INSN_SIZE); JMP32_INSN_SIZE);
i += JMP32_INSN_SIZE; i += JMP32_INSN_SIZE;
break; break;
@ -625,6 +788,48 @@ clang_jcc:
return i; return i;
} }
static int emit_call_track_retpoline(void *addr, struct insn *insn, int reg, u8 *bytes)
{
return __emit_trampoline(addr, insn, bytes,
__x86_indirect_call_thunk_array[reg],
__x86_indirect_jump_thunk_array[reg]);
}
#ifdef CONFIG_MITIGATION_ITS
static int emit_its_trampoline(void *addr, struct insn *insn, int reg, u8 *bytes)
{
u8 *thunk = __x86_indirect_its_thunk_array[reg];
u8 *tmp = its_allocate_thunk(reg);
if (tmp)
thunk = tmp;
return __emit_trampoline(addr, insn, bytes, thunk, thunk);
}
/* Check if an indirect branch is at ITS-unsafe address */
static bool cpu_wants_indirect_its_thunk_at(unsigned long addr, int reg)
{
if (!cpu_feature_enabled(X86_FEATURE_INDIRECT_THUNK_ITS))
return false;
/* Indirect branch opcode is 2 or 3 bytes depending on reg */
addr += 1 + reg / 8;
/* Lower-half of the cacheline? */
return !(addr & 0x20);
}
#else /* CONFIG_MITIGATION_ITS */
#ifdef CONFIG_FINEIBT
static bool cpu_wants_indirect_its_thunk_at(unsigned long addr, int reg)
{
return false;
}
#endif
#endif /* CONFIG_MITIGATION_ITS */
/* /*
* Rewrite the compiler generated retpoline thunk calls. * Rewrite the compiler generated retpoline thunk calls.
* *
@ -699,6 +904,15 @@ static int patch_retpoline(void *addr, struct insn *insn, u8 *bytes)
bytes[i++] = 0xe8; /* LFENCE */ bytes[i++] = 0xe8; /* LFENCE */
} }
#ifdef CONFIG_MITIGATION_ITS
/*
* Check if the address of last byte of emitted-indirect is in
* lower-half of the cacheline. Such branches need ITS mitigation.
*/
if (cpu_wants_indirect_its_thunk_at((unsigned long)addr + i, reg))
return emit_its_trampoline(addr, insn, reg, bytes);
#endif
ret = emit_indirect(op, reg, bytes + i); ret = emit_indirect(op, reg, bytes + i);
if (ret < 0) if (ret < 0)
return ret; return ret;
@ -732,6 +946,7 @@ void __init_or_module noinline apply_retpolines(s32 *start, s32 *end)
int len, ret; int len, ret;
u8 bytes[16]; u8 bytes[16];
u8 op1, op2; u8 op1, op2;
u8 *dest;
ret = insn_decode_kernel(&insn, addr); ret = insn_decode_kernel(&insn, addr);
if (WARN_ON_ONCE(ret < 0)) if (WARN_ON_ONCE(ret < 0))
@ -748,6 +963,12 @@ void __init_or_module noinline apply_retpolines(s32 *start, s32 *end)
case CALL_INSN_OPCODE: case CALL_INSN_OPCODE:
case JMP32_INSN_OPCODE: case JMP32_INSN_OPCODE:
/* Check for cfi_paranoid + ITS */
dest = addr + insn.length + insn.immediate.value;
if (dest[-1] == 0xea && (dest[0] & 0xf0) == 0x70) {
WARN_ON_ONCE(cfi_mode != CFI_FINEIBT);
continue;
}
break; break;
case 0x0f: /* escape */ case 0x0f: /* escape */
@ -775,6 +996,21 @@ void __init_or_module noinline apply_retpolines(s32 *start, s32 *end)
#ifdef CONFIG_MITIGATION_RETHUNK #ifdef CONFIG_MITIGATION_RETHUNK
bool cpu_wants_rethunk(void)
{
return cpu_feature_enabled(X86_FEATURE_RETHUNK);
}
bool cpu_wants_rethunk_at(void *addr)
{
if (!cpu_feature_enabled(X86_FEATURE_RETHUNK))
return false;
if (x86_return_thunk != its_return_thunk)
return true;
return !((unsigned long)addr & 0x20);
}
/* /*
* Rewrite the compiler generated return thunk tail-calls. * Rewrite the compiler generated return thunk tail-calls.
* *
@ -791,7 +1027,7 @@ static int patch_return(void *addr, struct insn *insn, u8 *bytes)
int i = 0; int i = 0;
/* Patch the custom return thunks... */ /* Patch the custom return thunks... */
if (cpu_feature_enabled(X86_FEATURE_RETHUNK)) { if (cpu_wants_rethunk_at(addr)) {
i = JMP32_INSN_SIZE; i = JMP32_INSN_SIZE;
__text_gen_insn(bytes, JMP32_INSN_OPCODE, addr, x86_return_thunk, i); __text_gen_insn(bytes, JMP32_INSN_OPCODE, addr, x86_return_thunk, i);
} else { } else {
@ -808,7 +1044,7 @@ void __init_or_module noinline apply_returns(s32 *start, s32 *end)
{ {
s32 *s; s32 *s;
if (cpu_feature_enabled(X86_FEATURE_RETHUNK)) if (cpu_wants_rethunk())
static_call_force_reinit(); static_call_force_reinit();
for (s = start; s < end; s++) { for (s = start; s < end; s++) {
@ -1022,8 +1258,6 @@ int cfi_get_func_arity(void *func)
static bool cfi_rand __ro_after_init = true; static bool cfi_rand __ro_after_init = true;
static u32 cfi_seed __ro_after_init; static u32 cfi_seed __ro_after_init;
static bool cfi_paranoid __ro_after_init = false;
/* /*
* Re-hash the CFI hash with a boot-time seed while making sure the result is * Re-hash the CFI hash with a boot-time seed while making sure the result is
* not a valid ENDBR instruction. * not a valid ENDBR instruction.
@ -1436,6 +1670,19 @@ static int cfi_rand_callers(s32 *start, s32 *end)
return 0; return 0;
} }
static int emit_paranoid_trampoline(void *addr, struct insn *insn, int reg, u8 *bytes)
{
u8 *thunk = (void *)__x86_indirect_its_thunk_array[reg] - 2;
#ifdef CONFIG_MITIGATION_ITS
u8 *tmp = its_allocate_thunk(reg);
if (tmp)
thunk = tmp;
#endif
return __emit_trampoline(addr, insn, bytes, thunk, thunk);
}
static int cfi_rewrite_callers(s32 *start, s32 *end) static int cfi_rewrite_callers(s32 *start, s32 *end)
{ {
s32 *s; s32 *s;
@ -1477,9 +1724,14 @@ static int cfi_rewrite_callers(s32 *start, s32 *end)
memcpy(bytes, fineibt_paranoid_start, fineibt_paranoid_size); memcpy(bytes, fineibt_paranoid_start, fineibt_paranoid_size);
memcpy(bytes + fineibt_caller_hash, &hash, 4); memcpy(bytes + fineibt_caller_hash, &hash, 4);
ret = emit_indirect(op, 11, bytes + fineibt_paranoid_ind); if (cpu_wants_indirect_its_thunk_at((unsigned long)addr + fineibt_paranoid_ind, 11)) {
if (WARN_ON_ONCE(ret != 3)) emit_paranoid_trampoline(addr + fineibt_caller_size,
continue; &insn, 11, bytes + fineibt_caller_size);
} else {
ret = emit_indirect(op, 11, bytes + fineibt_paranoid_ind);
if (WARN_ON_ONCE(ret != 3))
continue;
}
text_poke_early(addr, bytes, fineibt_paranoid_size); text_poke_early(addr, bytes, fineibt_paranoid_size);
} }
@ -1706,29 +1958,66 @@ Efault:
return false; return false;
} }
static bool is_paranoid_thunk(unsigned long addr)
{
u32 thunk;
__get_kernel_nofault(&thunk, (u32 *)addr, u32, Efault);
return (thunk & 0x00FFFFFF) == 0xfd75ea;
Efault:
return false;
}
/* /*
* regs->ip points to a LOCK Jcc.d8 instruction from the fineibt_paranoid_start[] * regs->ip points to a LOCK Jcc.d8 instruction from the fineibt_paranoid_start[]
* sequence. * sequence, or to an invalid instruction (0xea) + Jcc.d8 for cfi_paranoid + ITS
* thunk.
*/ */
static bool decode_fineibt_paranoid(struct pt_regs *regs, unsigned long *target, u32 *type) static bool decode_fineibt_paranoid(struct pt_regs *regs, unsigned long *target, u32 *type)
{ {
unsigned long addr = regs->ip - fineibt_paranoid_ud; unsigned long addr = regs->ip - fineibt_paranoid_ud;
u32 hash;
if (!cfi_paranoid || !is_cfi_trap(addr + fineibt_caller_size - LEN_UD2)) if (!cfi_paranoid)
return false; return false;
__get_kernel_nofault(&hash, addr + fineibt_caller_hash, u32, Efault); if (is_cfi_trap(addr + fineibt_caller_size - LEN_UD2)) {
*target = regs->r11 + fineibt_preamble_size; *target = regs->r11 + fineibt_preamble_size;
*type = regs->r10; *type = regs->r10;
/*
* Since the trapping instruction is the exact, but LOCK prefixed,
* Jcc.d8 that got us here, the normal fixup will work.
*/
return true;
}
/* /*
* Since the trapping instruction is the exact, but LOCK prefixed, * The cfi_paranoid + ITS thunk combination results in:
* Jcc.d8 that got us here, the normal fixup will work. *
* 0: 41 ba 78 56 34 12 mov $0x12345678, %r10d
* 6: 45 3b 53 f7 cmp -0x9(%r11), %r10d
* a: 4d 8d 5b f0 lea -0x10(%r11), %r11
* e: 2e e8 XX XX XX XX cs call __x86_indirect_paranoid_thunk_r11
*
* Where the paranoid_thunk looks like:
*
* 1d: <ea> (bad)
* __x86_indirect_paranoid_thunk_r11:
* 1e: 75 fd jne 1d
* __x86_indirect_its_thunk_r11:
* 20: 41 ff eb jmp *%r11
* 23: cc int3
*
*/ */
return true; if (is_paranoid_thunk(regs->ip)) {
*target = regs->r11 + fineibt_preamble_size;
*type = regs->r10;
regs->ip = *target;
return true;
}
Efault:
return false; return false;
} }
@ -2031,6 +2320,8 @@ static noinline void __init alt_reloc_selftest(void)
void __init alternative_instructions(void) void __init alternative_instructions(void)
{ {
u64 ibt;
int3_selftest(); int3_selftest();
/* /*
@ -2057,6 +2348,9 @@ void __init alternative_instructions(void)
*/ */
paravirt_set_cap(); paravirt_set_cap();
/* Keep CET-IBT disabled until caller/callee are patched */
ibt = ibt_save(/*disable*/ true);
__apply_fineibt(__retpoline_sites, __retpoline_sites_end, __apply_fineibt(__retpoline_sites, __retpoline_sites_end,
__cfi_sites, __cfi_sites_end, true); __cfi_sites, __cfi_sites_end, true);
@ -2080,6 +2374,8 @@ void __init alternative_instructions(void)
*/ */
apply_seal_endbr(__ibt_endbr_seal, __ibt_endbr_seal_end); apply_seal_endbr(__ibt_endbr_seal, __ibt_endbr_seal_end);
ibt_restore(ibt);
#ifdef CONFIG_SMP #ifdef CONFIG_SMP
/* Patch to UP if other cpus not imminent. */ /* Patch to UP if other cpus not imminent. */
if (!noreplace_smp && (num_present_cpus() == 1 || setup_max_cpus <= 1)) { if (!noreplace_smp && (num_present_cpus() == 1 || setup_max_cpus <= 1)) {

170
arch/x86/kernel/cpu/bugs.c
View file

@ -49,6 +49,7 @@ static void __init srbds_select_mitigation(void);
static void __init l1d_flush_select_mitigation(void); static void __init l1d_flush_select_mitigation(void);
static void __init srso_select_mitigation(void); static void __init srso_select_mitigation(void);
static void __init gds_select_mitigation(void); static void __init gds_select_mitigation(void);
static void __init its_select_mitigation(void);
/* The base value of the SPEC_CTRL MSR without task-specific bits set */ /* The base value of the SPEC_CTRL MSR without task-specific bits set */
u64 x86_spec_ctrl_base; u64 x86_spec_ctrl_base;
@ -66,6 +67,14 @@ static DEFINE_MUTEX(spec_ctrl_mutex);
void (*x86_return_thunk)(void) __ro_after_init = __x86_return_thunk; void (*x86_return_thunk)(void) __ro_after_init = __x86_return_thunk;
static void __init set_return_thunk(void *thunk)
{
if (x86_return_thunk != __x86_return_thunk)
pr_warn("x86/bugs: return thunk changed\n");
x86_return_thunk = thunk;
}
/* Update SPEC_CTRL MSR and its cached copy unconditionally */ /* Update SPEC_CTRL MSR and its cached copy unconditionally */
static void update_spec_ctrl(u64 val) static void update_spec_ctrl(u64 val)
{ {
@ -178,6 +187,7 @@ void __init cpu_select_mitigations(void)
*/ */
srso_select_mitigation(); srso_select_mitigation();
gds_select_mitigation(); gds_select_mitigation();
its_select_mitigation();
} }
/* /*
@ -1118,7 +1128,7 @@ do_cmd_auto:
setup_force_cpu_cap(X86_FEATURE_RETHUNK); setup_force_cpu_cap(X86_FEATURE_RETHUNK);
setup_force_cpu_cap(X86_FEATURE_UNRET); setup_force_cpu_cap(X86_FEATURE_UNRET);
x86_return_thunk = retbleed_return_thunk; set_return_thunk(retbleed_return_thunk);
if (boot_cpu_data.x86_vendor != X86_VENDOR_AMD && if (boot_cpu_data.x86_vendor != X86_VENDOR_AMD &&
boot_cpu_data.x86_vendor != X86_VENDOR_HYGON) boot_cpu_data.x86_vendor != X86_VENDOR_HYGON)
@ -1153,7 +1163,7 @@ do_cmd_auto:
setup_force_cpu_cap(X86_FEATURE_RETHUNK); setup_force_cpu_cap(X86_FEATURE_RETHUNK);
setup_force_cpu_cap(X86_FEATURE_CALL_DEPTH); setup_force_cpu_cap(X86_FEATURE_CALL_DEPTH);
x86_return_thunk = call_depth_return_thunk; set_return_thunk(call_depth_return_thunk);
break; break;
default: default:
@ -1187,6 +1197,145 @@ do_cmd_auto:
pr_info("%s\n", retbleed_strings[retbleed_mitigation]); pr_info("%s\n", retbleed_strings[retbleed_mitigation]);
} }
#undef pr_fmt
#define pr_fmt(fmt) "ITS: " fmt
enum its_mitigation_cmd {
ITS_CMD_OFF,
ITS_CMD_ON,
ITS_CMD_VMEXIT,
ITS_CMD_RSB_STUFF,
};
enum its_mitigation {
ITS_MITIGATION_OFF,
ITS_MITIGATION_VMEXIT_ONLY,
ITS_MITIGATION_ALIGNED_THUNKS,
ITS_MITIGATION_RETPOLINE_STUFF,
};
static const char * const its_strings[] = {
[ITS_MITIGATION_OFF] = "Vulnerable",
[ITS_MITIGATION_VMEXIT_ONLY] = "Mitigation: Vulnerable, KVM: Not affected",
[ITS_MITIGATION_ALIGNED_THUNKS] = "Mitigation: Aligned branch/return thunks",
[ITS_MITIGATION_RETPOLINE_STUFF] = "Mitigation: Retpolines, Stuffing RSB",
};
static enum its_mitigation its_mitigation __ro_after_init = ITS_MITIGATION_ALIGNED_THUNKS;
static enum its_mitigation_cmd its_cmd __ro_after_init =
IS_ENABLED(CONFIG_MITIGATION_ITS) ? ITS_CMD_ON : ITS_CMD_OFF;
static int __init its_parse_cmdline(char *str)
{
if (!str)
return -EINVAL;
if (!IS_ENABLED(CONFIG_MITIGATION_ITS)) {
pr_err("Mitigation disabled at compile time, ignoring option (%s)", str);
return 0;
}
if (!strcmp(str, "off")) {
its_cmd = ITS_CMD_OFF;
} else if (!strcmp(str, "on")) {
its_cmd = ITS_CMD_ON;
} else if (!strcmp(str, "force")) {
its_cmd = ITS_CMD_ON;
setup_force_cpu_bug(X86_BUG_ITS);
} else if (!strcmp(str, "vmexit")) {
its_cmd = ITS_CMD_VMEXIT;
} else if (!strcmp(str, "stuff")) {
its_cmd = ITS_CMD_RSB_STUFF;
} else {
pr_err("Ignoring unknown indirect_target_selection option (%s).", str);
}
return 0;
}
early_param("indirect_target_selection", its_parse_cmdline);
static void __init its_select_mitigation(void)
{
enum its_mitigation_cmd cmd = its_cmd;
if (!boot_cpu_has_bug(X86_BUG_ITS) || cpu_mitigations_off()) {
its_mitigation = ITS_MITIGATION_OFF;
return;
}
/* Retpoline+CDT mitigates ITS, bail out */
if (boot_cpu_has(X86_FEATURE_RETPOLINE) &&
boot_cpu_has(X86_FEATURE_CALL_DEPTH)) {
its_mitigation = ITS_MITIGATION_RETPOLINE_STUFF;
goto out;
}
/* Exit early to avoid irrelevant warnings */
if (cmd == ITS_CMD_OFF) {
its_mitigation = ITS_MITIGATION_OFF;
goto out;
}
if (spectre_v2_enabled == SPECTRE_V2_NONE) {
pr_err("WARNING: Spectre-v2 mitigation is off, disabling ITS\n");
its_mitigation = ITS_MITIGATION_OFF;
goto out;
}
if (!IS_ENABLED(CONFIG_MITIGATION_RETPOLINE) ||
!IS_ENABLED(CONFIG_MITIGATION_RETHUNK)) {
pr_err("WARNING: ITS mitigation depends on retpoline and rethunk support\n");
its_mitigation = ITS_MITIGATION_OFF;
goto out;
}
if (IS_ENABLED(CONFIG_DEBUG_FORCE_FUNCTION_ALIGN_64B)) {
pr_err("WARNING: ITS mitigation is not compatible with CONFIG_DEBUG_FORCE_FUNCTION_ALIGN_64B\n");
its_mitigation = ITS_MITIGATION_OFF;
goto out;
}
if (boot_cpu_has(X86_FEATURE_RETPOLINE_LFENCE)) {
pr_err("WARNING: ITS mitigation is not compatible with lfence mitigation\n");
its_mitigation = ITS_MITIGATION_OFF;
goto out;
}
if (cmd == ITS_CMD_RSB_STUFF &&
(!boot_cpu_has(X86_FEATURE_RETPOLINE) || !IS_ENABLED(CONFIG_MITIGATION_CALL_DEPTH_TRACKING))) {
pr_err("RSB stuff mitigation not supported, using default\n");
cmd = ITS_CMD_ON;
}
switch (cmd) {
case ITS_CMD_OFF:
its_mitigation = ITS_MITIGATION_OFF;
break;
case ITS_CMD_VMEXIT:
if (boot_cpu_has_bug(X86_BUG_ITS_NATIVE_ONLY)) {
its_mitigation = ITS_MITIGATION_VMEXIT_ONLY;
goto out;
}
fallthrough;
case ITS_CMD_ON:
its_mitigation = ITS_MITIGATION_ALIGNED_THUNKS;
if (!boot_cpu_has(X86_FEATURE_RETPOLINE))
setup_force_cpu_cap(X86_FEATURE_INDIRECT_THUNK_ITS);
setup_force_cpu_cap(X86_FEATURE_RETHUNK);
set_return_thunk(its_return_thunk);
break;
case ITS_CMD_RSB_STUFF:
its_mitigation = ITS_MITIGATION_RETPOLINE_STUFF;
setup_force_cpu_cap(X86_FEATURE_RETHUNK);
setup_force_cpu_cap(X86_FEATURE_CALL_DEPTH);
set_return_thunk(call_depth_return_thunk);
if (retbleed_mitigation == RETBLEED_MITIGATION_NONE) {
retbleed_mitigation = RETBLEED_MITIGATION_STUFF;
pr_info("Retbleed mitigation updated to stuffing\n");
}
break;
}
out:
pr_info("%s\n", its_strings[its_mitigation]);
}
#undef pr_fmt #undef pr_fmt
#define pr_fmt(fmt) "Spectre V2 : " fmt #define pr_fmt(fmt) "Spectre V2 : " fmt
@ -2607,10 +2756,10 @@ static void __init srso_select_mitigation(void)
if (boot_cpu_data.x86 == 0x19) { if (boot_cpu_data.x86 == 0x19) {
setup_force_cpu_cap(X86_FEATURE_SRSO_ALIAS); setup_force_cpu_cap(X86_FEATURE_SRSO_ALIAS);
x86_return_thunk = srso_alias_return_thunk; set_return_thunk(srso_alias_return_thunk);
} else { } else {
setup_force_cpu_cap(X86_FEATURE_SRSO); setup_force_cpu_cap(X86_FEATURE_SRSO);
x86_return_thunk = srso_return_thunk; set_return_thunk(srso_return_thunk);
} }
if (has_microcode) if (has_microcode)
srso_mitigation = SRSO_MITIGATION_SAFE_RET; srso_mitigation = SRSO_MITIGATION_SAFE_RET;
@ -2800,6 +2949,11 @@ static ssize_t rfds_show_state(char *buf)
return sysfs_emit(buf, "%s\n", rfds_strings[rfds_mitigation]); return sysfs_emit(buf, "%s\n", rfds_strings[rfds_mitigation]);
} }
static ssize_t its_show_state(char *buf)
{
return sysfs_emit(buf, "%s\n", its_strings[its_mitigation]);
}
static char *stibp_state(void) static char *stibp_state(void)
{ {
if (spectre_v2_in_eibrs_mode(spectre_v2_enabled) && if (spectre_v2_in_eibrs_mode(spectre_v2_enabled) &&
@ -2982,6 +3136,9 @@ static ssize_t cpu_show_common(struct device *dev, struct device_attribute *attr
case X86_BUG_RFDS: case X86_BUG_RFDS:
return rfds_show_state(buf); return rfds_show_state(buf);
case X86_BUG_ITS:
return its_show_state(buf);
default: default:
break; break;
} }
@ -3061,6 +3218,11 @@ ssize_t cpu_show_reg_file_data_sampling(struct device *dev, struct device_attrib
{ {
return cpu_show_common(dev, attr, buf, X86_BUG_RFDS); return cpu_show_common(dev, attr, buf, X86_BUG_RFDS);
} }
ssize_t cpu_show_indirect_target_selection(struct device *dev, struct device_attribute *attr, char *buf)
{
return cpu_show_common(dev, attr, buf, X86_BUG_ITS);
}
#endif #endif
void __warn_thunk(void) void __warn_thunk(void)

63
arch/x86/kernel/cpu/common.c
View file

@ -1227,6 +1227,10 @@ static const __initconst struct x86_cpu_id cpu_vuln_whitelist[] = {
#define GDS BIT(6) #define GDS BIT(6)
/* CPU is affected by Register File Data Sampling */ /* CPU is affected by Register File Data Sampling */
#define RFDS BIT(7) #define RFDS BIT(7)
/* CPU is affected by Indirect Target Selection */
#define ITS BIT(8)
/* CPU is affected by Indirect Target Selection, but guest-host isolation is not affected */
#define ITS_NATIVE_ONLY BIT(9)
static const struct x86_cpu_id cpu_vuln_blacklist[] __initconst = { static const struct x86_cpu_id cpu_vuln_blacklist[] __initconst = {
VULNBL_INTEL_STEPS(INTEL_IVYBRIDGE, X86_STEP_MAX, SRBDS), VULNBL_INTEL_STEPS(INTEL_IVYBRIDGE, X86_STEP_MAX, SRBDS),
@ -1238,22 +1242,25 @@ static const struct x86_cpu_id cpu_vuln_blacklist[] __initconst = {
VULNBL_INTEL_STEPS(INTEL_BROADWELL_G, X86_STEP_MAX, SRBDS), VULNBL_INTEL_STEPS(INTEL_BROADWELL_G, X86_STEP_MAX, SRBDS),
VULNBL_INTEL_STEPS(INTEL_BROADWELL_X, X86_STEP_MAX, MMIO), VULNBL_INTEL_STEPS(INTEL_BROADWELL_X, X86_STEP_MAX, MMIO),
VULNBL_INTEL_STEPS(INTEL_BROADWELL, X86_STEP_MAX, SRBDS), VULNBL_INTEL_STEPS(INTEL_BROADWELL, X86_STEP_MAX, SRBDS),
VULNBL_INTEL_STEPS(INTEL_SKYLAKE_X, X86_STEP_MAX, MMIO | RETBLEED | GDS), VULNBL_INTEL_STEPS(INTEL_SKYLAKE_X, 0x5, MMIO | RETBLEED | GDS),
VULNBL_INTEL_STEPS(INTEL_SKYLAKE_X, X86_STEP_MAX, MMIO | RETBLEED | GDS | ITS),
VULNBL_INTEL_STEPS(INTEL_SKYLAKE_L, X86_STEP_MAX, MMIO | RETBLEED | GDS | SRBDS), VULNBL_INTEL_STEPS(INTEL_SKYLAKE_L, X86_STEP_MAX, MMIO | RETBLEED | GDS | SRBDS),
VULNBL_INTEL_STEPS(INTEL_SKYLAKE, X86_STEP_MAX, MMIO | RETBLEED | GDS | SRBDS), VULNBL_INTEL_STEPS(INTEL_SKYLAKE, X86_STEP_MAX, MMIO | RETBLEED | GDS | SRBDS),
VULNBL_INTEL_STEPS(INTEL_KABYLAKE_L, X86_STEP_MAX, MMIO | RETBLEED | GDS | SRBDS), VULNBL_INTEL_STEPS(INTEL_KABYLAKE_L, 0xb, MMIO | RETBLEED | GDS | SRBDS),
VULNBL_INTEL_STEPS(INTEL_KABYLAKE, X86_STEP_MAX, MMIO | RETBLEED | GDS | SRBDS), VULNBL_INTEL_STEPS(INTEL_KABYLAKE_L, X86_STEP_MAX, MMIO | RETBLEED | GDS | SRBDS | ITS),
VULNBL_INTEL_STEPS(INTEL_KABYLAKE, 0xc, MMIO | RETBLEED | GDS | SRBDS),
VULNBL_INTEL_STEPS(INTEL_KABYLAKE, X86_STEP_MAX, MMIO | RETBLEED | GDS | SRBDS | ITS),
VULNBL_INTEL_STEPS(INTEL_CANNONLAKE_L, X86_STEP_MAX, RETBLEED), VULNBL_INTEL_STEPS(INTEL_CANNONLAKE_L, X86_STEP_MAX, RETBLEED),
VULNBL_INTEL_STEPS(INTEL_ICELAKE_L, X86_STEP_MAX, MMIO | MMIO_SBDS | RETBLEED | GDS), VULNBL_INTEL_STEPS(INTEL_ICELAKE_L, X86_STEP_MAX, MMIO | MMIO_SBDS | RETBLEED | GDS | ITS | ITS_NATIVE_ONLY),
VULNBL_INTEL_STEPS(INTEL_ICELAKE_D, X86_STEP_MAX, MMIO | GDS), VULNBL_INTEL_STEPS(INTEL_ICELAKE_D, X86_STEP_MAX, MMIO | GDS | ITS | ITS_NATIVE_ONLY),
VULNBL_INTEL_STEPS(INTEL_ICELAKE_X, X86_STEP_MAX, MMIO | GDS), VULNBL_INTEL_STEPS(INTEL_ICELAKE_X, X86_STEP_MAX, MMIO | GDS | ITS | ITS_NATIVE_ONLY),
VULNBL_INTEL_STEPS(INTEL_COMETLAKE, X86_STEP_MAX, MMIO | MMIO_SBDS | RETBLEED | GDS), VULNBL_INTEL_STEPS(INTEL_COMETLAKE, X86_STEP_MAX, MMIO | MMIO_SBDS | RETBLEED | GDS | ITS),
VULNBL_INTEL_STEPS(INTEL_COMETLAKE_L, 0x0, MMIO | RETBLEED), VULNBL_INTEL_STEPS(INTEL_COMETLAKE_L, 0x0, MMIO | RETBLEED | ITS),
VULNBL_INTEL_STEPS(INTEL_COMETLAKE_L, X86_STEP_MAX, MMIO | MMIO_SBDS | RETBLEED | GDS), VULNBL_INTEL_STEPS(INTEL_COMETLAKE_L, X86_STEP_MAX, MMIO | MMIO_SBDS | RETBLEED | GDS | ITS),
VULNBL_INTEL_STEPS(INTEL_TIGERLAKE_L, X86_STEP_MAX, GDS), VULNBL_INTEL_STEPS(INTEL_TIGERLAKE_L, X86_STEP_MAX, GDS | ITS | ITS_NATIVE_ONLY),
VULNBL_INTEL_STEPS(INTEL_TIGERLAKE, X86_STEP_MAX, GDS), VULNBL_INTEL_STEPS(INTEL_TIGERLAKE, X86_STEP_MAX, GDS | ITS | ITS_NATIVE_ONLY),
VULNBL_INTEL_STEPS(INTEL_LAKEFIELD, X86_STEP_MAX, MMIO | MMIO_SBDS | RETBLEED), VULNBL_INTEL_STEPS(INTEL_LAKEFIELD, X86_STEP_MAX, MMIO | MMIO_SBDS | RETBLEED),
VULNBL_INTEL_STEPS(INTEL_ROCKETLAKE, X86_STEP_MAX, MMIO | RETBLEED | GDS), VULNBL_INTEL_STEPS(INTEL_ROCKETLAKE, X86_STEP_MAX, MMIO | RETBLEED | GDS | ITS | ITS_NATIVE_ONLY),
VULNBL_INTEL_TYPE(INTEL_ALDERLAKE, ATOM, RFDS), VULNBL_INTEL_TYPE(INTEL_ALDERLAKE, ATOM, RFDS),
VULNBL_INTEL_STEPS(INTEL_ALDERLAKE_L, X86_STEP_MAX, RFDS), VULNBL_INTEL_STEPS(INTEL_ALDERLAKE_L, X86_STEP_MAX, RFDS),
VULNBL_INTEL_TYPE(INTEL_RAPTORLAKE, ATOM, RFDS), VULNBL_INTEL_TYPE(INTEL_RAPTORLAKE, ATOM, RFDS),
@ -1318,6 +1325,32 @@ static bool __init vulnerable_to_rfds(u64 x86_arch_cap_msr)
return cpu_matches(cpu_vuln_blacklist, RFDS); return cpu_matches(cpu_vuln_blacklist, RFDS);
} }
static bool __init vulnerable_to_its(u64 x86_arch_cap_msr)
{
/* The "immunity" bit trumps everything else: */
if (x86_arch_cap_msr & ARCH_CAP_ITS_NO)
return false;
if (boot_cpu_data.x86_vendor != X86_VENDOR_INTEL)
return false;
/* None of the affected CPUs have BHI_CTRL */
if (boot_cpu_has(X86_FEATURE_BHI_CTRL))
return false;
/*
* If a VMM did not expose ITS_NO, assume that a guest could
* be running on a vulnerable hardware or may migrate to such
* hardware.
*/
if (boot_cpu_has(X86_FEATURE_HYPERVISOR))
return true;
if (cpu_matches(cpu_vuln_blacklist, ITS))
return true;
return false;
}
static void __init cpu_set_bug_bits(struct cpuinfo_x86 *c) static void __init cpu_set_bug_bits(struct cpuinfo_x86 *c)
{ {
u64 x86_arch_cap_msr = x86_read_arch_cap_msr(); u64 x86_arch_cap_msr = x86_read_arch_cap_msr();
@ -1452,6 +1485,12 @@ static void __init cpu_set_bug_bits(struct cpuinfo_x86 *c)
if (cpu_has(c, X86_FEATURE_AMD_IBPB) && !cpu_has(c, X86_FEATURE_AMD_IBPB_RET)) if (cpu_has(c, X86_FEATURE_AMD_IBPB) && !cpu_has(c, X86_FEATURE_AMD_IBPB_RET))
setup_force_cpu_bug(X86_BUG_IBPB_NO_RET); setup_force_cpu_bug(X86_BUG_IBPB_NO_RET);
if (vulnerable_to_its(x86_arch_cap_msr)) {
setup_force_cpu_bug(X86_BUG_ITS);
if (cpu_matches(cpu_vuln_blacklist, ITS_NATIVE_ONLY))
setup_force_cpu_bug(X86_BUG_ITS_NATIVE_ONLY);
}
if (cpu_matches(cpu_vuln_whitelist, NO_MELTDOWN)) if (cpu_matches(cpu_vuln_whitelist, NO_MELTDOWN))
return; return;

2
arch/x86/kernel/ftrace.c
View file

@ -354,7 +354,7 @@ create_trampoline(struct ftrace_ops *ops, unsigned int *tramp_size)
goto fail; goto fail;
ip = trampoline + size; ip = trampoline + size;
if (cpu_feature_enabled(X86_FEATURE_RETHUNK)) if (cpu_wants_rethunk_at(ip))
__text_gen_insn(ip, JMP32_INSN_OPCODE, ip, x86_return_thunk, JMP32_INSN_SIZE); __text_gen_insn(ip, JMP32_INSN_OPCODE, ip, x86_return_thunk, JMP32_INSN_SIZE);
else else
memcpy(ip, retq, sizeof(retq)); memcpy(ip, retq, sizeof(retq));

6
arch/x86/kernel/module.c
View file

@ -266,6 +266,8 @@ int module_finalize(const Elf_Ehdr *hdr,
ibt_endbr = s; ibt_endbr = s;
} }
its_init_mod(me);
if (retpolines || cfi) { if (retpolines || cfi) {
void *rseg = NULL, *cseg = NULL; void *rseg = NULL, *cseg = NULL;
unsigned int rsize = 0, csize = 0; unsigned int rsize = 0, csize = 0;
@ -286,6 +288,9 @@ int module_finalize(const Elf_Ehdr *hdr,
void *rseg = (void *)retpolines->sh_addr; void *rseg = (void *)retpolines->sh_addr;
apply_retpolines(rseg, rseg + retpolines->sh_size); apply_retpolines(rseg, rseg + retpolines->sh_size);
} }
its_fini_mod(me);
if (returns) { if (returns) {
void *rseg = (void *)returns->sh_addr; void *rseg = (void *)returns->sh_addr;
apply_returns(rseg, rseg + returns->sh_size); apply_returns(rseg, rseg + returns->sh_size);
@ -326,4 +331,5 @@ int module_finalize(const Elf_Ehdr *hdr,
void module_arch_cleanup(struct module *mod) void module_arch_cleanup(struct module *mod)
{ {
alternatives_smp_module_del(mod); alternatives_smp_module_del(mod);
its_free_mod(mod);
} }

4
arch/x86/kernel/static_call.c
View file

@ -81,7 +81,7 @@ static void __ref __static_call_transform(void *insn, enum insn_type type,
break; break;
case RET: case RET:
if (cpu_feature_enabled(X86_FEATURE_RETHUNK)) if (cpu_wants_rethunk_at(insn))
code = text_gen_insn(JMP32_INSN_OPCODE, insn, x86_return_thunk); code = text_gen_insn(JMP32_INSN_OPCODE, insn, x86_return_thunk);
else else
code = &retinsn; code = &retinsn;
@ -90,7 +90,7 @@ static void __ref __static_call_transform(void *insn, enum insn_type type,
case JCC: case JCC:
if (!func) { if (!func) {
func = __static_call_return; func = __static_call_return;
if (cpu_feature_enabled(X86_FEATURE_RETHUNK)) if (cpu_wants_rethunk())
func = x86_return_thunk; func = x86_return_thunk;
} }

10
arch/x86/kernel/vmlinux.lds.S
View file

@ -505,6 +505,16 @@ PROVIDE(__ref_stack_chk_guard = __stack_chk_guard);
"SRSO function pair won't alias"); "SRSO function pair won't alias");
#endif #endif
#if defined(CONFIG_MITIGATION_ITS) && !defined(CONFIG_DEBUG_FORCE_FUNCTION_ALIGN_64B)
. = ASSERT(__x86_indirect_its_thunk_rax & 0x20, "__x86_indirect_thunk_rax not in second half of cacheline");
. = ASSERT(((__x86_indirect_its_thunk_rcx - __x86_indirect_its_thunk_rax) % 64) == 0, "Indirect thunks are not cacheline apart");
. = ASSERT(__x86_indirect_its_thunk_array == __x86_indirect_its_thunk_rax, "Gap in ITS thunk array");
#endif
#if defined(CONFIG_MITIGATION_ITS) && !defined(CONFIG_DEBUG_FORCE_FUNCTION_ALIGN_64B)
. = ASSERT(its_return_thunk & 0x20, "its_return_thunk not in second half of cacheline");
#endif
#endif /* CONFIG_X86_64 */ #endif /* CONFIG_X86_64 */
/* /*

4
arch/x86/kvm/x86.c
View file

@ -1584,7 +1584,7 @@ EXPORT_SYMBOL_GPL(kvm_emulate_rdpmc);
ARCH_CAP_PSCHANGE_MC_NO | ARCH_CAP_TSX_CTRL_MSR | ARCH_CAP_TAA_NO | \ ARCH_CAP_PSCHANGE_MC_NO | ARCH_CAP_TSX_CTRL_MSR | ARCH_CAP_TAA_NO | \
ARCH_CAP_SBDR_SSDP_NO | ARCH_CAP_FBSDP_NO | ARCH_CAP_PSDP_NO | \ ARCH_CAP_SBDR_SSDP_NO | ARCH_CAP_FBSDP_NO | ARCH_CAP_PSDP_NO | \
ARCH_CAP_FB_CLEAR | ARCH_CAP_RRSBA | ARCH_CAP_PBRSB_NO | ARCH_CAP_GDS_NO | \ ARCH_CAP_FB_CLEAR | ARCH_CAP_RRSBA | ARCH_CAP_PBRSB_NO | ARCH_CAP_GDS_NO | \
ARCH_CAP_RFDS_NO | ARCH_CAP_RFDS_CLEAR | ARCH_CAP_BHI_NO) ARCH_CAP_RFDS_NO | ARCH_CAP_RFDS_CLEAR | ARCH_CAP_BHI_NO | ARCH_CAP_ITS_NO)
static u64 kvm_get_arch_capabilities(void) static u64 kvm_get_arch_capabilities(void)
{ {
@ -1618,6 +1618,8 @@ static u64 kvm_get_arch_capabilities(void)
data |= ARCH_CAP_MDS_NO; data |= ARCH_CAP_MDS_NO;
if (!boot_cpu_has_bug(X86_BUG_RFDS)) if (!boot_cpu_has_bug(X86_BUG_RFDS))
data |= ARCH_CAP_RFDS_NO; data |= ARCH_CAP_RFDS_NO;
if (!boot_cpu_has_bug(X86_BUG_ITS))
data |= ARCH_CAP_ITS_NO;
if (!boot_cpu_has(X86_FEATURE_RTM)) { if (!boot_cpu_has(X86_FEATURE_RTM)) {
/* /*

48
arch/x86/lib/retpoline.S
View file

@ -367,6 +367,54 @@ SYM_FUNC_END(call_depth_return_thunk)
#endif /* CONFIG_MITIGATION_CALL_DEPTH_TRACKING */ #endif /* CONFIG_MITIGATION_CALL_DEPTH_TRACKING */
#ifdef CONFIG_MITIGATION_ITS
.macro ITS_THUNK reg
/*
* If CFI paranoid is used then the ITS thunk starts with opcodes (0xea; jne 1b)
* that complete the fineibt_paranoid caller sequence.
*/
1: .byte 0xea
SYM_INNER_LABEL(__x86_indirect_paranoid_thunk_\reg, SYM_L_GLOBAL)
UNWIND_HINT_UNDEFINED
ANNOTATE_NOENDBR
jne 1b
SYM_INNER_LABEL(__x86_indirect_its_thunk_\reg, SYM_L_GLOBAL)
UNWIND_HINT_UNDEFINED
ANNOTATE_NOENDBR
ANNOTATE_RETPOLINE_SAFE
jmp *%\reg
int3
.align 32, 0xcc /* fill to the end of the line */
.skip 32 - (__x86_indirect_its_thunk_\reg - 1b), 0xcc /* skip to the next upper half */
.endm
/* ITS mitigation requires thunks be aligned to upper half of cacheline */
.align 64, 0xcc
.skip 29, 0xcc
#define GEN(reg) ITS_THUNK reg
#include <asm/GEN-for-each-reg.h>
#undef GEN
.align 64, 0xcc
SYM_FUNC_ALIAS(__x86_indirect_its_thunk_array, __x86_indirect_its_thunk_rax)
SYM_CODE_END(__x86_indirect_its_thunk_array)
.align 64, 0xcc
.skip 32, 0xcc
SYM_CODE_START(its_return_thunk)
UNWIND_HINT_FUNC
ANNOTATE_NOENDBR
ANNOTATE_UNRET_SAFE
ret
int3
SYM_CODE_END(its_return_thunk)
EXPORT_SYMBOL(its_return_thunk)
#endif /* CONFIG_MITIGATION_ITS */
/* /*
* This function name is magical and is used by -mfunction-return=thunk-extern * This function name is magical and is used by -mfunction-return=thunk-extern
* for the compiler to generate JMPs to it. * for the compiler to generate JMPs to it.

3
arch/x86/mm/init_32.c
View file

@ -30,6 +30,7 @@
#include <linux/initrd.h> #include <linux/initrd.h>
#include <linux/cpumask.h> #include <linux/cpumask.h>
#include <linux/gfp.h> #include <linux/gfp.h>
#include <linux/execmem.h>
#include <asm/asm.h> #include <asm/asm.h>
#include <asm/bios_ebda.h> #include <asm/bios_ebda.h>
@ -755,6 +756,8 @@ void mark_rodata_ro(void)
pr_info("Write protecting kernel text and read-only data: %luk\n", pr_info("Write protecting kernel text and read-only data: %luk\n",
size >> 10); size >> 10);
execmem_cache_make_ro();
kernel_set_to_readonly = 1; kernel_set_to_readonly = 1;
#ifdef CONFIG_CPA_DEBUG #ifdef CONFIG_CPA_DEBUG

3
arch/x86/mm/init_64.c
View file

@ -34,6 +34,7 @@
#include <linux/gfp.h> #include <linux/gfp.h>
#include <linux/kcore.h> #include <linux/kcore.h>
#include <linux/bootmem_info.h> #include <linux/bootmem_info.h>
#include <linux/execmem.h>
#include <asm/processor.h> #include <asm/processor.h>
#include <asm/bios_ebda.h> #include <asm/bios_ebda.h>
@ -1391,6 +1392,8 @@ void mark_rodata_ro(void)
(end - start) >> 10); (end - start) >> 10);
set_memory_ro(start, (end - start) >> PAGE_SHIFT); set_memory_ro(start, (end - start) >> PAGE_SHIFT);
execmem_cache_make_ro();
kernel_set_to_readonly = 1; kernel_set_to_readonly = 1;
/* /*

7
arch/x86/net/bpf_jit_comp.c
View file

@ -663,7 +663,10 @@ static void emit_indirect_jump(u8 **pprog, int reg, u8 *ip)
{ {
u8 *prog = *pprog; u8 *prog = *pprog;
if (cpu_feature_enabled(X86_FEATURE_RETPOLINE_LFENCE)) { if (cpu_feature_enabled(X86_FEATURE_INDIRECT_THUNK_ITS)) {
OPTIMIZER_HIDE_VAR(reg);
emit_jump(&prog, its_static_thunk(reg), ip);
} else if (cpu_feature_enabled(X86_FEATURE_RETPOLINE_LFENCE)) {
EMIT_LFENCE(); EMIT_LFENCE();
EMIT2(0xFF, 0xE0 + reg); EMIT2(0xFF, 0xE0 + reg);
} else if (cpu_feature_enabled(X86_FEATURE_RETPOLINE)) { } else if (cpu_feature_enabled(X86_FEATURE_RETPOLINE)) {
@ -685,7 +688,7 @@ static void emit_return(u8 **pprog, u8 *ip)
{ {
u8 *prog = *pprog; u8 *prog = *pprog;
if (cpu_feature_enabled(X86_FEATURE_RETHUNK)) { if (cpu_wants_rethunk()) {
emit_jump(&prog, x86_return_thunk, ip); emit_jump(&prog, x86_return_thunk, ip);
} else { } else {
EMIT1(0xC3); /* ret */ EMIT1(0xC3); /* ret */

3
drivers/base/cpu.c
View file

@ -600,6 +600,7 @@ CPU_SHOW_VULN_FALLBACK(spec_rstack_overflow);
CPU_SHOW_VULN_FALLBACK(gds); CPU_SHOW_VULN_FALLBACK(gds);
CPU_SHOW_VULN_FALLBACK(reg_file_data_sampling); CPU_SHOW_VULN_FALLBACK(reg_file_data_sampling);
CPU_SHOW_VULN_FALLBACK(ghostwrite); CPU_SHOW_VULN_FALLBACK(ghostwrite);
CPU_SHOW_VULN_FALLBACK(indirect_target_selection);
static DEVICE_ATTR(meltdown, 0444, cpu_show_meltdown, NULL); static DEVICE_ATTR(meltdown, 0444, cpu_show_meltdown, NULL);
static DEVICE_ATTR(spectre_v1, 0444, cpu_show_spectre_v1, NULL); static DEVICE_ATTR(spectre_v1, 0444, cpu_show_spectre_v1, NULL);
@ -616,6 +617,7 @@ static DEVICE_ATTR(spec_rstack_overflow, 0444, cpu_show_spec_rstack_overflow, NU
static DEVICE_ATTR(gather_data_sampling, 0444, cpu_show_gds, NULL); static DEVICE_ATTR(gather_data_sampling, 0444, cpu_show_gds, NULL);
static DEVICE_ATTR(reg_file_data_sampling, 0444, cpu_show_reg_file_data_sampling, NULL); static DEVICE_ATTR(reg_file_data_sampling, 0444, cpu_show_reg_file_data_sampling, NULL);
static DEVICE_ATTR(ghostwrite, 0444, cpu_show_ghostwrite, NULL); static DEVICE_ATTR(ghostwrite, 0444, cpu_show_ghostwrite, NULL);
static DEVICE_ATTR(indirect_target_selection, 0444, cpu_show_indirect_target_selection, NULL);
static struct attribute *cpu_root_vulnerabilities_attrs[] = { static struct attribute *cpu_root_vulnerabilities_attrs[] = {
&dev_attr_meltdown.attr, &dev_attr_meltdown.attr,
@ -633,6 +635,7 @@ static struct attribute *cpu_root_vulnerabilities_attrs[] = {
&dev_attr_gather_data_sampling.attr, &dev_attr_gather_data_sampling.attr,
&dev_attr_reg_file_data_sampling.attr, &dev_attr_reg_file_data_sampling.attr,
&dev_attr_ghostwrite.attr, &dev_attr_ghostwrite.attr,
&dev_attr_indirect_target_selection.attr,
NULL NULL
}; };

2
include/linux/cpu.h
View file

@ -78,6 +78,8 @@ extern ssize_t cpu_show_gds(struct device *dev,
extern ssize_t cpu_show_reg_file_data_sampling(struct device *dev, extern ssize_t cpu_show_reg_file_data_sampling(struct device *dev,
struct device_attribute *attr, char *buf); struct device_attribute *attr, char *buf);
extern ssize_t cpu_show_ghostwrite(struct device *dev, struct device_attribute *attr, char *buf); extern ssize_t cpu_show_ghostwrite(struct device *dev, struct device_attribute *attr, char *buf);
extern ssize_t cpu_show_indirect_target_selection(struct device *dev,
struct device_attribute *attr, char *buf);
extern __printf(4, 5) extern __printf(4, 5)
struct device *cpu_device_create(struct device *parent, void *drvdata, struct device *cpu_device_create(struct device *parent, void *drvdata,

11
include/linux/execmem.h
View file

@ -4,6 +4,7 @@
#include <linux/types.h> #include <linux/types.h>
#include <linux/moduleloader.h> #include <linux/moduleloader.h>
#include <linux/cleanup.h>
#if (defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS)) && \ #if (defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS)) && \
!defined(CONFIG_KASAN_VMALLOC) !defined(CONFIG_KASAN_VMALLOC)
@ -53,7 +54,7 @@ enum execmem_range_flags {
EXECMEM_ROX_CACHE = (1 << 1), EXECMEM_ROX_CACHE = (1 << 1),
}; };
#ifdef CONFIG_ARCH_HAS_EXECMEM_ROX #if defined(CONFIG_ARCH_HAS_EXECMEM_ROX) && defined(CONFIG_EXECMEM)
/** /**
* execmem_fill_trapping_insns - set memory to contain instructions that * execmem_fill_trapping_insns - set memory to contain instructions that
* will trap * will trap
@ -93,9 +94,15 @@ int execmem_make_temp_rw(void *ptr, size_t size);
* Return: 0 on success or negative error code on failure. * Return: 0 on success or negative error code on failure.
*/ */
int execmem_restore_rox(void *ptr, size_t size); int execmem_restore_rox(void *ptr, size_t size);
/*
* Called from mark_readonly(), where the system transitions to ROX.
*/
void execmem_cache_make_ro(void);
#else #else
static inline int execmem_make_temp_rw(void *ptr, size_t size) { return 0; } static inline int execmem_make_temp_rw(void *ptr, size_t size) { return 0; }
static inline int execmem_restore_rox(void *ptr, size_t size) { return 0; } static inline int execmem_restore_rox(void *ptr, size_t size) { return 0; }
static inline void execmem_cache_make_ro(void) { }
#endif #endif
/** /**
@ -170,6 +177,8 @@ void *execmem_alloc(enum execmem_type type, size_t size);
*/ */
void execmem_free(void *ptr); void execmem_free(void *ptr);
DEFINE_FREE(execmem, void *, if (_T) execmem_free(_T));
#ifdef CONFIG_MMU #ifdef CONFIG_MMU
/** /**
* execmem_vmap - create virtual mapping for EXECMEM_MODULE_DATA memory * execmem_vmap - create virtual mapping for EXECMEM_MODULE_DATA memory

5
include/linux/module.h
View file

@ -586,6 +586,11 @@ struct module {
atomic_t refcnt; atomic_t refcnt;
#endif #endif
#ifdef CONFIG_MITIGATION_ITS
int its_num_pages;
void **its_page_array;
#endif
#ifdef CONFIG_CONSTRUCTORS #ifdef CONFIG_CONSTRUCTORS
/* Constructor functions. */ /* Constructor functions. */
ctor_fn_t *ctors; ctor_fn_t *ctors;

40
mm/execmem.c
View file

@ -254,6 +254,34 @@ out_unlock:
return ptr; return ptr;
} }
static bool execmem_cache_rox = false;
void execmem_cache_make_ro(void)
{
struct maple_tree *free_areas = &execmem_cache.free_areas;
struct maple_tree *busy_areas = &execmem_cache.busy_areas;
MA_STATE(mas_free, free_areas, 0, ULONG_MAX);
MA_STATE(mas_busy, busy_areas, 0, ULONG_MAX);
struct mutex *mutex = &execmem_cache.mutex;
void *area;
execmem_cache_rox = true;
mutex_lock(mutex);
mas_for_each(&mas_free, area, ULONG_MAX) {
unsigned long pages = mas_range_len(&mas_free) >> PAGE_SHIFT;
set_memory_ro(mas_free.index, pages);
}
mas_for_each(&mas_busy, area, ULONG_MAX) {
unsigned long pages = mas_range_len(&mas_busy) >> PAGE_SHIFT;
set_memory_ro(mas_busy.index, pages);
}
mutex_unlock(mutex);
}
static int execmem_cache_populate(struct execmem_range *range, size_t size) static int execmem_cache_populate(struct execmem_range *range, size_t size)
{ {
unsigned long vm_flags = VM_ALLOW_HUGE_VMAP; unsigned long vm_flags = VM_ALLOW_HUGE_VMAP;
@ -274,9 +302,15 @@ static int execmem_cache_populate(struct execmem_range *range, size_t size)
/* fill memory with instructions that will trap */ /* fill memory with instructions that will trap */
execmem_fill_trapping_insns(p, alloc_size, /* writable = */ true); execmem_fill_trapping_insns(p, alloc_size, /* writable = */ true);
err = set_memory_rox((unsigned long)p, vm->nr_pages); if (execmem_cache_rox) {
if (err) err = set_memory_rox((unsigned long)p, vm->nr_pages);
goto err_free_mem; if (err)
goto err_free_mem;
} else {
err = set_memory_x((unsigned long)p, vm->nr_pages);
if (err)
goto err_free_mem;
}
err = execmem_cache_add(p, alloc_size); err = execmem_cache_add(p, alloc_size);
if (err) if (err)

9
tools/objtool/arch/x86/decode.c
View file

@ -189,6 +189,15 @@ int arch_decode_instruction(struct objtool_file *file, const struct section *sec
op2 = ins.opcode.bytes[1]; op2 = ins.opcode.bytes[1];
op3 = ins.opcode.bytes[2]; op3 = ins.opcode.bytes[2];
/*
* XXX hack, decoder is buggered and thinks 0xea is 7 bytes long.
*/
if (op1 == 0xea) {
insn->len = 1;
insn->type = INSN_BUG;
return 0;
}
if (ins.rex_prefix.nbytes) { if (ins.rex_prefix.nbytes) {
rex = ins.rex_prefix.bytes[0]; rex = ins.rex_prefix.bytes[0];
rex_w = X86_REX_W(rex) >> 3; rex_w = X86_REX_W(rex) >> 3;

1
tools/testing/selftests/Makefile
View file

@ -121,6 +121,7 @@ TARGETS += user_events
TARGETS += vDSO TARGETS += vDSO
TARGETS += mm TARGETS += mm
TARGETS += x86 TARGETS += x86
TARGETS += x86/bugs
TARGETS += zram TARGETS += zram
#Please keep the TARGETS list alphabetically sorted #Please keep the TARGETS list alphabetically sorted
# Run "make quicktest=1 run_tests" or # Run "make quicktest=1 run_tests" or

3
tools/testing/selftests/x86/bugs/Makefile Normal file
View file

@ -0,0 +1,3 @@
TEST_PROGS := its_sysfs.py its_permutations.py its_indirect_alignment.py its_ret_alignment.py
TEST_FILES := common.py
include ../../lib.mk

164
tools/testing/selftests/x86/bugs/common.py Executable file
View file

@ -0,0 +1,164 @@
#!/usr/bin/env python3
# SPDX-License-Identifier: GPL-2.0
#
# Copyright (c) 2025 Intel Corporation
#
# This contains kselftest framework adapted common functions for testing
# mitigation for x86 bugs.
import os, sys, re, shutil
sys.path.insert(0, '../../kselftest')
import ksft
def read_file(path):
if not os.path.exists(path):
return None
with open(path, 'r') as file:
return file.read().strip()
def cpuinfo_has(arg):
cpuinfo = read_file('/proc/cpuinfo')
if arg in cpuinfo:
return True
return False
def cmdline_has(arg):
cmdline = read_file('/proc/cmdline')
if arg in cmdline:
return True
return False
def cmdline_has_either(args):
cmdline = read_file('/proc/cmdline')
for arg in args:
if arg in cmdline:
return True
return False
def cmdline_has_none(args):
return not cmdline_has_either(args)
def cmdline_has_all(args):
cmdline = read_file('/proc/cmdline')
for arg in args:
if arg not in cmdline:
return False
return True
def get_sysfs(bug):
return read_file("/sys/devices/system/cpu/vulnerabilities/" + bug)
def sysfs_has(bug, mitigation):
status = get_sysfs(bug)
if mitigation in status:
return True
return False
def sysfs_has_either(bugs, mitigations):
for bug in bugs:
for mitigation in mitigations:
if sysfs_has(bug, mitigation):
return True
return False
def sysfs_has_none(bugs, mitigations):
return not sysfs_has_either(bugs, mitigations)
def sysfs_has_all(bugs, mitigations):
for bug in bugs:
for mitigation in mitigations:
if not sysfs_has(bug, mitigation):
return False
return True
def bug_check_pass(bug, found):
ksft.print_msg(f"\nFound: {found}")
# ksft.print_msg(f"\ncmdline: {read_file('/proc/cmdline')}")
ksft.test_result_pass(f'{bug}: {found}')
def bug_check_fail(bug, found, expected):
ksft.print_msg(f'\nFound:\t {found}')
ksft.print_msg(f'Expected:\t {expected}')
ksft.print_msg(f"\ncmdline: {read_file('/proc/cmdline')}")
ksft.test_result_fail(f'{bug}: {found}')
def bug_status_unknown(bug, found):
ksft.print_msg(f'\nUnknown status: {found}')
ksft.print_msg(f"\ncmdline: {read_file('/proc/cmdline')}")
ksft.test_result_fail(f'{bug}: {found}')
def basic_checks_sufficient(bug, mitigation):
if not mitigation:
bug_status_unknown(bug, "None")
return True
elif mitigation == "Not affected":
ksft.test_result_pass(bug)
return True
elif mitigation == "Vulnerable":
if cmdline_has_either([f'{bug}=off', 'mitigations=off']):
bug_check_pass(bug, mitigation)
return True
return False
def get_section_info(vmlinux, section_name):
from elftools.elf.elffile import ELFFile
with open(vmlinux, 'rb') as f:
elffile = ELFFile(f)
section = elffile.get_section_by_name(section_name)
if section is None:
ksft.print_msg("Available sections in vmlinux:")
for sec in elffile.iter_sections():
ksft.print_msg(sec.name)
raise ValueError(f"Section {section_name} not found in {vmlinux}")
return section['sh_addr'], section['sh_offset'], section['sh_size']
def get_patch_sites(vmlinux, offset, size):
import struct
output = []
with open(vmlinux, 'rb') as f:
f.seek(offset)
i = 0
while i < size:
data = f.read(4) # s32
if not data:
break
sym_offset = struct.unpack('<i', data)[0] + i
i += 4
output.append(sym_offset)
return output
def get_instruction_from_vmlinux(elffile, section, virtual_address, target_address):
from capstone import Cs, CS_ARCH_X86, CS_MODE_64
section_start = section['sh_addr']
section_end = section_start + section['sh_size']
if not (section_start <= target_address < section_end):
return None
offset = target_address - section_start
code = section.data()[offset:offset + 16]
cap = init_capstone()
for instruction in cap.disasm(code, target_address):
if instruction.address == target_address:
return instruction
return None
def init_capstone():
from capstone import Cs, CS_ARCH_X86, CS_MODE_64, CS_OPT_SYNTAX_ATT
cap = Cs(CS_ARCH_X86, CS_MODE_64)
cap.syntax = CS_OPT_SYNTAX_ATT
return cap
def get_runtime_kernel():
import drgn
return drgn.program_from_kernel()
def check_dependencies_or_skip(modules, script_name="unknown test"):
for mod in modules:
try:
__import__(mod)
except ImportError:
ksft.test_result_skip(f"Skipping {script_name}: missing module '{mod}'")
ksft.finished()

150
tools/testing/selftests/x86/bugs/its_indirect_alignment.py Executable file
View file

@ -0,0 +1,150 @@
#!/usr/bin/env python3
# SPDX-License-Identifier: GPL-2.0
#
# Copyright (c) 2025 Intel Corporation
#
# Test for indirect target selection (ITS) mitigation.
#
# Test if indirect CALL/JMP are correctly patched by evaluating
# the vmlinux .retpoline_sites in /proc/kcore.
# Install dependencies
# add-apt-repository ppa:michel-slm/kernel-utils
# apt update
# apt install -y python3-drgn python3-pyelftools python3-capstone
#
# Best to copy the vmlinux at a standard location:
# mkdir -p /usr/lib/debug/lib/modules/$(uname -r)
# cp $VMLINUX /usr/lib/debug/lib/modules/$(uname -r)/vmlinux
#
# Usage: ./its_indirect_alignment.py [vmlinux]
import os, sys, argparse
from pathlib import Path
this_dir = os.path.dirname(os.path.realpath(__file__))
sys.path.insert(0, this_dir + '/../../kselftest')
import ksft
import common as c
bug = "indirect_target_selection"
mitigation = c.get_sysfs(bug)
if not mitigation or "Aligned branch/return thunks" not in mitigation:
ksft.test_result_skip("Skipping its_indirect_alignment.py: Aligned branch/return thunks not enabled")
ksft.finished()
if c.sysfs_has("spectre_v2", "Retpolines"):
ksft.test_result_skip("Skipping its_indirect_alignment.py: Retpolines deployed")
ksft.finished()
c.check_dependencies_or_skip(['drgn', 'elftools', 'capstone'], script_name="its_indirect_alignment.py")
from elftools.elf.elffile import ELFFile
from drgn.helpers.common.memory import identify_address
cap = c.init_capstone()
if len(os.sys.argv) > 1:
arg_vmlinux = os.sys.argv[1]
if not os.path.exists(arg_vmlinux):
ksft.test_result_fail(f"its_indirect_alignment.py: vmlinux not found at argument path: {arg_vmlinux}")
ksft.exit_fail()
os.makedirs(f"/usr/lib/debug/lib/modules/{os.uname().release}", exist_ok=True)
os.system(f'cp {arg_vmlinux} /usr/lib/debug/lib/modules/$(uname -r)/vmlinux')
vmlinux = f"/usr/lib/debug/lib/modules/{os.uname().release}/vmlinux"
if not os.path.exists(vmlinux):
ksft.test_result_fail(f"its_indirect_alignment.py: vmlinux not found at {vmlinux}")
ksft.exit_fail()
ksft.print_msg(f"Using vmlinux: {vmlinux}")
retpolines_start_vmlinux, retpolines_sec_offset, size = c.get_section_info(vmlinux, '.retpoline_sites')
ksft.print_msg(f"vmlinux: Section .retpoline_sites (0x{retpolines_start_vmlinux:x}) found at 0x{retpolines_sec_offset:x} with size 0x{size:x}")
sites_offset = c.get_patch_sites(vmlinux, retpolines_sec_offset, size)
total_retpoline_tests = len(sites_offset)
ksft.print_msg(f"Found {total_retpoline_tests} retpoline sites")
prog = c.get_runtime_kernel()
retpolines_start_kcore = prog.symbol('__retpoline_sites').address
ksft.print_msg(f'kcore: __retpoline_sites: 0x{retpolines_start_kcore:x}')
x86_indirect_its_thunk_r15 = prog.symbol('__x86_indirect_its_thunk_r15').address
ksft.print_msg(f'kcore: __x86_indirect_its_thunk_r15: 0x{x86_indirect_its_thunk_r15:x}')
tests_passed = 0
tests_failed = 0
tests_unknown = 0
with open(vmlinux, 'rb') as f:
elffile = ELFFile(f)
text_section = elffile.get_section_by_name('.text')
for i in range(0, len(sites_offset)):
site = retpolines_start_kcore + sites_offset[i]
vmlinux_site = retpolines_start_vmlinux + sites_offset[i]
passed = unknown = failed = False
try:
vmlinux_insn = c.get_instruction_from_vmlinux(elffile, text_section, text_section['sh_addr'], vmlinux_site)
kcore_insn = list(cap.disasm(prog.read(site, 16), site))[0]
operand = kcore_insn.op_str
insn_end = site + kcore_insn.size - 1 # TODO handle Jcc.32 __x86_indirect_thunk_\reg
safe_site = insn_end & 0x20
site_status = "" if safe_site else "(unsafe)"
ksft.print_msg(f"\nSite {i}: {identify_address(prog, site)} <0x{site:x}> {site_status}")
ksft.print_msg(f"\tvmlinux: 0x{vmlinux_insn.address:x}:\t{vmlinux_insn.mnemonic}\t{vmlinux_insn.op_str}")
ksft.print_msg(f"\tkcore: 0x{kcore_insn.address:x}:\t{kcore_insn.mnemonic}\t{kcore_insn.op_str}")
if (site & 0x20) ^ (insn_end & 0x20):
ksft.print_msg(f"\tSite at safe/unsafe boundary: {str(kcore_insn.bytes)} {kcore_insn.mnemonic} {operand}")
if safe_site:
tests_passed += 1
passed = True
ksft.print_msg(f"\tPASSED: At safe address")
continue
if operand.startswith('0xffffffff'):
thunk = int(operand, 16)
if thunk > x86_indirect_its_thunk_r15:
insn_at_thunk = list(cap.disasm(prog.read(thunk, 16), thunk))[0]
operand += ' -> ' + insn_at_thunk.mnemonic + ' ' + insn_at_thunk.op_str + ' <dynamic-thunk?>'
if 'jmp' in insn_at_thunk.mnemonic and thunk & 0x20:
ksft.print_msg(f"\tPASSED: Found {operand} at safe address")
passed = True
if not passed:
if kcore_insn.operands[0].type == capstone.CS_OP_IMM:
operand += ' <' + prog.symbol(int(operand, 16)) + '>'
if '__x86_indirect_its_thunk_' in operand:
ksft.print_msg(f"\tPASSED: Found {operand}")
else:
ksft.print_msg(f"\tPASSED: Found direct branch: {kcore_insn}, ITS thunk not required.")
passed = True
else:
unknown = True
if passed:
tests_passed += 1
elif unknown:
ksft.print_msg(f"UNKNOWN: unexpected operand: {kcore_insn}")
tests_unknown += 1
else:
ksft.print_msg(f'\t************* FAILED *************')
ksft.print_msg(f"\tFound {kcore_insn.bytes} {kcore_insn.mnemonic} {operand}")
ksft.print_msg(f'\t**********************************')
tests_failed += 1
except Exception as e:
ksft.print_msg(f"UNKNOWN: An unexpected error occurred: {e}")
tests_unknown += 1
ksft.print_msg(f"\n\nSummary:")
ksft.print_msg(f"PASS: \t{tests_passed} \t/ {total_retpoline_tests}")
ksft.print_msg(f"FAIL: \t{tests_failed} \t/ {total_retpoline_tests}")
ksft.print_msg(f"UNKNOWN: \t{tests_unknown} \t/ {total_retpoline_tests}")
if tests_failed == 0:
ksft.test_result_pass("All ITS return thunk sites passed")
else:
ksft.test_result_fail(f"{tests_failed} ITS return thunk sites failed")
ksft.finished()

109
tools/testing/selftests/x86/bugs/its_permutations.py Executable file
View file

@ -0,0 +1,109 @@
#!/usr/bin/env python3
# SPDX-License-Identifier: GPL-2.0
#
# Copyright (c) 2025 Intel Corporation
#
# Test for indirect target selection (ITS) cmdline permutations with other bugs
# like spectre_v2 and retbleed.
import os, sys, subprocess, itertools, re, shutil
test_dir = os.path.dirname(os.path.realpath(__file__))
sys.path.insert(0, test_dir + '/../../kselftest')
import ksft
import common as c
bug = "indirect_target_selection"
mitigation = c.get_sysfs(bug)
if not mitigation or "Not affected" in mitigation:
ksft.test_result_skip("Skipping its_permutations.py: not applicable")
ksft.finished()
if shutil.which('vng') is None:
ksft.test_result_skip("Skipping its_permutations.py: virtme-ng ('vng') not found in PATH.")
ksft.finished()
TEST = f"{test_dir}/its_sysfs.py"
default_kparam = ['clearcpuid=hypervisor', 'panic=5', 'panic_on_warn=1', 'oops=panic', 'nmi_watchdog=1', 'hung_task_panic=1']
DEBUG = " -v "
# Install dependencies
# https://github.com/arighi/virtme-ng
# apt install virtme-ng
BOOT_CMD = f"vng --run {test_dir}/../../../../../arch/x86/boot/bzImage "
#BOOT_CMD += DEBUG
bug = "indirect_target_selection"
input_options = {
'indirect_target_selection' : ['off', 'on', 'stuff', 'vmexit'],
'retbleed' : ['off', 'stuff', 'auto'],
'spectre_v2' : ['off', 'on', 'eibrs', 'retpoline', 'ibrs', 'eibrs,retpoline'],
}
def pretty_print(output):
OKBLUE = '\033[94m'
OKGREEN = '\033[92m'
WARNING = '\033[93m'
FAIL = '\033[91m'
ENDC = '\033[0m'
BOLD = '\033[1m'
# Define patterns and their corresponding colors
patterns = {
r"^ok \d+": OKGREEN,
r"^not ok \d+": FAIL,
r"^# Testing .*": OKBLUE,
r"^# Found: .*": WARNING,
r"^# Totals: .*": BOLD,
r"pass:([1-9]\d*)": OKGREEN,
r"fail:([1-9]\d*)": FAIL,
r"skip:([1-9]\d*)": WARNING,
}
# Apply colors based on patterns
for pattern, color in patterns.items():
output = re.sub(pattern, lambda match: f"{color}{match.group(0)}{ENDC}", output, flags=re.MULTILINE)
print(output)
combinations = list(itertools.product(*input_options.values()))
ksft.print_header()
ksft.set_plan(len(combinations))
logs = ""
for combination in combinations:
append = ""
log = ""
for p in default_kparam:
append += f' --append={p}'
command = BOOT_CMD + append
test_params = ""
for i, key in enumerate(input_options.keys()):
param = f'{key}={combination[i]}'
test_params += f' {param}'
command += f" --append={param}"
command += f" -- {TEST}"
test_name = f"{bug} {test_params}"
pretty_print(f'# Testing {test_name}')
t = subprocess.Popen(command, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
t.wait()
output, _ = t.communicate()
if t.returncode == 0:
ksft.test_result_pass(test_name)
else:
ksft.test_result_fail(test_name)
output = output.decode()
log += f" {output}"
pretty_print(log)
logs += output + "\n"
# Optionally use tappy to parse the output
# apt install python3-tappy
with open("logs.txt", "w") as f:
f.write(logs)
ksft.finished()

139
tools/testing/selftests/x86/bugs/its_ret_alignment.py Executable file
View file

@ -0,0 +1,139 @@
#!/usr/bin/env python3
# SPDX-License-Identifier: GPL-2.0
#
# Copyright (c) 2025 Intel Corporation
#
# Test for indirect target selection (ITS) mitigation.
#
# Tests if the RETs are correctly patched by evaluating the
# vmlinux .return_sites in /proc/kcore.
#
# Install dependencies
# add-apt-repository ppa:michel-slm/kernel-utils
# apt update
# apt install -y python3-drgn python3-pyelftools python3-capstone
#
# Run on target machine
# mkdir -p /usr/lib/debug/lib/modules/$(uname -r)
# cp $VMLINUX /usr/lib/debug/lib/modules/$(uname -r)/vmlinux
#
# Usage: ./its_ret_alignment.py
import os, sys, argparse
from pathlib import Path
this_dir = os.path.dirname(os.path.realpath(__file__))
sys.path.insert(0, this_dir + '/../../kselftest')
import ksft
import common as c
bug = "indirect_target_selection"
mitigation = c.get_sysfs(bug)
if not mitigation or "Aligned branch/return thunks" not in mitigation:
ksft.test_result_skip("Skipping its_ret_alignment.py: Aligned branch/return thunks not enabled")
ksft.finished()
c.check_dependencies_or_skip(['drgn', 'elftools', 'capstone'], script_name="its_ret_alignment.py")
from elftools.elf.elffile import ELFFile
from drgn.helpers.common.memory import identify_address
cap = c.init_capstone()
if len(os.sys.argv) > 1:
arg_vmlinux = os.sys.argv[1]
if not os.path.exists(arg_vmlinux):
ksft.test_result_fail(f"its_ret_alignment.py: vmlinux not found at user-supplied path: {arg_vmlinux}")
ksft.exit_fail()
os.makedirs(f"/usr/lib/debug/lib/modules/{os.uname().release}", exist_ok=True)
os.system(f'cp {arg_vmlinux} /usr/lib/debug/lib/modules/$(uname -r)/vmlinux')
vmlinux = f"/usr/lib/debug/lib/modules/{os.uname().release}/vmlinux"
if not os.path.exists(vmlinux):
ksft.test_result_fail(f"its_ret_alignment.py: vmlinux not found at {vmlinux}")
ksft.exit_fail()
ksft.print_msg(f"Using vmlinux: {vmlinux}")
rethunks_start_vmlinux, rethunks_sec_offset, size = c.get_section_info(vmlinux, '.return_sites')
ksft.print_msg(f"vmlinux: Section .return_sites (0x{rethunks_start_vmlinux:x}) found at 0x{rethunks_sec_offset:x} with size 0x{size:x}")
sites_offset = c.get_patch_sites(vmlinux, rethunks_sec_offset, size)
total_rethunk_tests = len(sites_offset)
ksft.print_msg(f"Found {total_rethunk_tests} rethunk sites")
prog = c.get_runtime_kernel()
rethunks_start_kcore = prog.symbol('__return_sites').address
ksft.print_msg(f'kcore: __rethunk_sites: 0x{rethunks_start_kcore:x}')
its_return_thunk = prog.symbol('its_return_thunk').address
ksft.print_msg(f'kcore: its_return_thunk: 0x{its_return_thunk:x}')
tests_passed = 0
tests_failed = 0
tests_unknown = 0
tests_skipped = 0
with open(vmlinux, 'rb') as f:
elffile = ELFFile(f)
text_section = elffile.get_section_by_name('.text')
for i in range(len(sites_offset)):
site = rethunks_start_kcore + sites_offset[i]
vmlinux_site = rethunks_start_vmlinux + sites_offset[i]
try:
passed = unknown = failed = skipped = False
symbol = identify_address(prog, site)
vmlinux_insn = c.get_instruction_from_vmlinux(elffile, text_section, text_section['sh_addr'], vmlinux_site)
kcore_insn = list(cap.disasm(prog.read(site, 16), site))[0]
insn_end = site + kcore_insn.size - 1
safe_site = insn_end & 0x20
site_status = "" if safe_site else "(unsafe)"
ksft.print_msg(f"\nSite {i}: {symbol} <0x{site:x}> {site_status}")
ksft.print_msg(f"\tvmlinux: 0x{vmlinux_insn.address:x}:\t{vmlinux_insn.mnemonic}\t{vmlinux_insn.op_str}")
ksft.print_msg(f"\tkcore: 0x{kcore_insn.address:x}:\t{kcore_insn.mnemonic}\t{kcore_insn.op_str}")
if safe_site:
tests_passed += 1
passed = True
ksft.print_msg(f"\tPASSED: At safe address")
continue
if "jmp" in kcore_insn.mnemonic:
passed = True
elif "ret" not in kcore_insn.mnemonic:
skipped = True
if passed:
ksft.print_msg(f"\tPASSED: Found {kcore_insn.mnemonic} {kcore_insn.op_str}")
tests_passed += 1
elif skipped:
ksft.print_msg(f"\tSKIPPED: Found '{kcore_insn.mnemonic}'")
tests_skipped += 1
elif unknown:
ksft.print_msg(f"UNKNOWN: An unknown instruction: {kcore_insn}")
tests_unknown += 1
else:
ksft.print_msg(f'\t************* FAILED *************')
ksft.print_msg(f"\tFound {kcore_insn.mnemonic} {kcore_insn.op_str}")
ksft.print_msg(f'\t**********************************')
tests_failed += 1
except Exception as e:
ksft.print_msg(f"UNKNOWN: An unexpected error occurred: {e}")
tests_unknown += 1
ksft.print_msg(f"\n\nSummary:")
ksft.print_msg(f"PASSED: \t{tests_passed} \t/ {total_rethunk_tests}")
ksft.print_msg(f"FAILED: \t{tests_failed} \t/ {total_rethunk_tests}")
ksft.print_msg(f"SKIPPED: \t{tests_skipped} \t/ {total_rethunk_tests}")
ksft.print_msg(f"UNKNOWN: \t{tests_unknown} \t/ {total_rethunk_tests}")
if tests_failed == 0:
ksft.test_result_pass("All ITS return thunk sites passed.")
else:
ksft.test_result_fail(f"{tests_failed} failed sites need ITS return thunks.")
ksft.finished()

65
tools/testing/selftests/x86/bugs/its_sysfs.py Executable file
View file

@ -0,0 +1,65 @@
#!/usr/bin/env python3
# SPDX-License-Identifier: GPL-2.0
#
# Copyright (c) 2025 Intel Corporation
#
# Test for Indirect Target Selection(ITS) mitigation sysfs status.
import sys, os, re
this_dir = os.path.dirname(os.path.realpath(__file__))
sys.path.insert(0, this_dir + '/../../kselftest')
import ksft
from common import *
bug = "indirect_target_selection"
mitigation = get_sysfs(bug)
ITS_MITIGATION_ALIGNED_THUNKS = "Mitigation: Aligned branch/return thunks"
ITS_MITIGATION_RETPOLINE_STUFF = "Mitigation: Retpolines, Stuffing RSB"
ITS_MITIGATION_VMEXIT_ONLY = "Mitigation: Vulnerable, KVM: Not affected"
ITS_MITIGATION_VULNERABLE = "Vulnerable"
def check_mitigation():
if mitigation == ITS_MITIGATION_ALIGNED_THUNKS:
if cmdline_has(f'{bug}=stuff') and sysfs_has("spectre_v2", "Retpolines"):
bug_check_fail(bug, ITS_MITIGATION_ALIGNED_THUNKS, ITS_MITIGATION_RETPOLINE_STUFF)
return
if cmdline_has(f'{bug}=vmexit') and cpuinfo_has('its_native_only'):
bug_check_fail(bug, ITS_MITIGATION_ALIGNED_THUNKS, ITS_MITIGATION_VMEXIT_ONLY)
return
bug_check_pass(bug, ITS_MITIGATION_ALIGNED_THUNKS)
return
if mitigation == ITS_MITIGATION_RETPOLINE_STUFF:
if cmdline_has(f'{bug}=stuff') and sysfs_has("spectre_v2", "Retpolines"):
bug_check_pass(bug, ITS_MITIGATION_RETPOLINE_STUFF)
return
if sysfs_has('retbleed', 'Stuffing'):
bug_check_pass(bug, ITS_MITIGATION_RETPOLINE_STUFF)
return
bug_check_fail(bug, ITS_MITIGATION_RETPOLINE_STUFF, ITS_MITIGATION_ALIGNED_THUNKS)
if mitigation == ITS_MITIGATION_VMEXIT_ONLY:
if cmdline_has(f'{bug}=vmexit') and cpuinfo_has('its_native_only'):
bug_check_pass(bug, ITS_MITIGATION_VMEXIT_ONLY)
return
bug_check_fail(bug, ITS_MITIGATION_VMEXIT_ONLY, ITS_MITIGATION_ALIGNED_THUNKS)
if mitigation == ITS_MITIGATION_VULNERABLE:
if sysfs_has("spectre_v2", "Vulnerable"):
bug_check_pass(bug, ITS_MITIGATION_VULNERABLE)
else:
bug_check_fail(bug, "Mitigation", ITS_MITIGATION_VULNERABLE)
bug_status_unknown(bug, mitigation)
return
ksft.print_header()
ksft.set_plan(1)
ksft.print_msg(f'{bug}: {mitigation} ...')
if not basic_checks_sufficient(bug, mitigation):
check_mitigation()
ksft.finished()