public-mirrors/linux
1
0
Fork 0
mirror of git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git synced 2025-08-05 16:54:27 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions
linux/arch/arm64/kvm/inject_fault.c

340 lines
8.7 KiB
C
Raw Permalink Normal View History

treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 234 Based on 1 normalized pattern(s): this program is free software you can redistribute it and or modify it under the terms of the gnu general public license version 2 as published by the free software foundation this program is distributed in the hope that it will be useful but without any warranty without even the implied warranty of merchantability or fitness for a particular purpose see the gnu general public license for more details you should have received a copy of the gnu general public license along with this program if not see http www gnu org licenses extracted by the scancode license scanner the SPDX license identifier GPL-2.0-only has been chosen to replace the boilerplate/reference in 503 file(s). Signed-off-by: Thomas Gleixner <tglx@linutronix.de> Reviewed-by: Alexios Zavras <alexios.zavras@intel.com> Reviewed-by: Allison Randal <allison@lohutok.net> Reviewed-by: Enrico Weigelt <info@metux.net> Cc: linux-spdx@vger.kernel.org Link: https://lkml.kernel.org/r/20190602204653.811534538@linutronix.de Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
2019-06-03 07:44:50 +02:00
// SPDX-License-Identifier: GPL-2.0-only
arm64: KVM: fault injection into a guest Implement the injection of a fault (undefined, data abort or prefetch abort) into a 64bit guest. Reviewed-by: Christopher Covington <cov@codeaurora.org> Reviewed-by: Catalin Marinas <catalin.marinas@arm.com> Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
2012-12-17 12:27:42 +00:00
/*
arm64: KVM: 32bit guest fault injection Add fault injection capability for 32bit guests. Reviewed-by: Christopher Covington <cov@codeaurora.org> Reviewed-by: Catalin Marinas <catalin.marinas@arm.com> Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
2013-02-06 11:29:35 +00:00
* Fault injection for both 32 and 64bit guests.
arm64: KVM: fault injection into a guest Implement the injection of a fault (undefined, data abort or prefetch abort) into a 64bit guest. Reviewed-by: Christopher Covington <cov@codeaurora.org> Reviewed-by: Catalin Marinas <catalin.marinas@arm.com> Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
2012-12-17 12:27:42 +00:00
*
* Copyright (C) 2012,2013 - ARM Ltd
* Author: Marc Zyngier <marc.zyngier@arm.com>
*
* Based on arch/arm/kvm/emulate.c
* Copyright (C) 2012 - Virtual Open Systems and Columbia University
* Author: Christoffer Dall <c.dall@virtualopensystems.com>
*/
#include <linux/kvm_host.h>
#include <asm/kvm_emulate.h>
KVM: arm64: nv: Support virtual EL2 exceptions Support injecting exceptions and performing exception returns to and from virtual EL2. This must be done entirely in software except when taking an exception from vEL0 to vEL2 when the virtual HCR_EL2.{E2H,TGE} == {1,1} (a VHE guest hypervisor). [maz: switch to common exception injection framework, illegal exeption return handling] Reviewed-by: Ganapatrao Kulkarni <gankulkarni@os.amperecomputing.com> Signed-off-by: Jintack Lim <jintack.lim@linaro.org> Signed-off-by: Christoffer Dall <christoffer.dall@arm.com> Signed-off-by: Marc Zyngier <maz@kernel.org> Link: https://lore.kernel.org/r/20230209175820.1939006-10-maz@kernel.org Signed-off-by: Oliver Upton <oliver.upton@linux.dev>
2023-02-09 17:58:11 +00:00
#include <asm/kvm_nested.h>
arm64: KVM: fault injection into a guest Implement the injection of a fault (undefined, data abort or prefetch abort) into a 64bit guest. Reviewed-by: Christopher Covington <cov@codeaurora.org> Reviewed-by: Catalin Marinas <catalin.marinas@arm.com> Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
2012-12-17 12:27:42 +00:00
#include <asm/esr.h>
KVM: arm64: Factor out helper for selecting exception target EL Pull out the exception target selection from pend_sync_exception() for general use. Use PSR_MODE_ELxh as a shorthand for the target EL, as SP_ELx selection is handled further along in the hyp's exception emulation. Reviewed-by: Marc Zyngier <maz@kernel.org> Link: https://lore.kernel.org/r/20250708172532.1699409-16-oliver.upton@linux.dev Signed-off-by: Oliver Upton <oliver.upton@linux.dev>
2025-07-08 10:25:20 -07:00
static unsigned int exception_target_el(struct kvm_vcpu *vcpu)
KVM: arm64: nv: Support virtual EL2 exceptions Support injecting exceptions and performing exception returns to and from virtual EL2. This must be done entirely in software except when taking an exception from vEL0 to vEL2 when the virtual HCR_EL2.{E2H,TGE} == {1,1} (a VHE guest hypervisor). [maz: switch to common exception injection framework, illegal exeption return handling] Reviewed-by: Ganapatrao Kulkarni <gankulkarni@os.amperecomputing.com> Signed-off-by: Jintack Lim <jintack.lim@linaro.org> Signed-off-by: Christoffer Dall <christoffer.dall@arm.com> Signed-off-by: Marc Zyngier <maz@kernel.org> Link: https://lore.kernel.org/r/20230209175820.1939006-10-maz@kernel.org Signed-off-by: Oliver Upton <oliver.upton@linux.dev>
2023-02-09 17:58:11 +00:00
{
/* If not nesting, EL1 is the only possible exception target */
KVM: arm64: Factor out helper for selecting exception target EL Pull out the exception target selection from pend_sync_exception() for general use. Use PSR_MODE_ELxh as a shorthand for the target EL, as SP_ELx selection is handled further along in the hyp's exception emulation. Reviewed-by: Marc Zyngier <maz@kernel.org> Link: https://lore.kernel.org/r/20250708172532.1699409-16-oliver.upton@linux.dev Signed-off-by: Oliver Upton <oliver.upton@linux.dev>
2025-07-08 10:25:20 -07:00
if (likely(!vcpu_has_nv(vcpu)))
return PSR_MODE_EL1h;
KVM: arm64: nv: Support virtual EL2 exceptions Support injecting exceptions and performing exception returns to and from virtual EL2. This must be done entirely in software except when taking an exception from vEL0 to vEL2 when the virtual HCR_EL2.{E2H,TGE} == {1,1} (a VHE guest hypervisor). [maz: switch to common exception injection framework, illegal exeption return handling] Reviewed-by: Ganapatrao Kulkarni <gankulkarni@os.amperecomputing.com> Signed-off-by: Jintack Lim <jintack.lim@linaro.org> Signed-off-by: Christoffer Dall <christoffer.dall@arm.com> Signed-off-by: Marc Zyngier <maz@kernel.org> Link: https://lore.kernel.org/r/20230209175820.1939006-10-maz@kernel.org Signed-off-by: Oliver Upton <oliver.upton@linux.dev>
2023-02-09 17:58:11 +00:00
/*
* With NV, we need to pick between EL1 and EL2. Note that we
* never deal with a nesting exception here, hence never
* changing context, and the exception itself can be delayed
* until the next entry.
*/
switch(*vcpu_cpsr(vcpu) & PSR_MODE_MASK) {
case PSR_MODE_EL2h:
case PSR_MODE_EL2t:
KVM: arm64: Factor out helper for selecting exception target EL Pull out the exception target selection from pend_sync_exception() for general use. Use PSR_MODE_ELxh as a shorthand for the target EL, as SP_ELx selection is handled further along in the hyp's exception emulation. Reviewed-by: Marc Zyngier <maz@kernel.org> Link: https://lore.kernel.org/r/20250708172532.1699409-16-oliver.upton@linux.dev Signed-off-by: Oliver Upton <oliver.upton@linux.dev>
2025-07-08 10:25:20 -07:00
return PSR_MODE_EL2h;
KVM: arm64: nv: Support virtual EL2 exceptions Support injecting exceptions and performing exception returns to and from virtual EL2. This must be done entirely in software except when taking an exception from vEL0 to vEL2 when the virtual HCR_EL2.{E2H,TGE} == {1,1} (a VHE guest hypervisor). [maz: switch to common exception injection framework, illegal exeption return handling] Reviewed-by: Ganapatrao Kulkarni <gankulkarni@os.amperecomputing.com> Signed-off-by: Jintack Lim <jintack.lim@linaro.org> Signed-off-by: Christoffer Dall <christoffer.dall@arm.com> Signed-off-by: Marc Zyngier <maz@kernel.org> Link: https://lore.kernel.org/r/20230209175820.1939006-10-maz@kernel.org Signed-off-by: Oliver Upton <oliver.upton@linux.dev>
2023-02-09 17:58:11 +00:00
case PSR_MODE_EL1h:
case PSR_MODE_EL1t:
KVM: arm64: Factor out helper for selecting exception target EL Pull out the exception target selection from pend_sync_exception() for general use. Use PSR_MODE_ELxh as a shorthand for the target EL, as SP_ELx selection is handled further along in the hyp's exception emulation. Reviewed-by: Marc Zyngier <maz@kernel.org> Link: https://lore.kernel.org/r/20250708172532.1699409-16-oliver.upton@linux.dev Signed-off-by: Oliver Upton <oliver.upton@linux.dev>
2025-07-08 10:25:20 -07:00
return PSR_MODE_EL1h;
KVM: arm64: nv: Support virtual EL2 exceptions Support injecting exceptions and performing exception returns to and from virtual EL2. This must be done entirely in software except when taking an exception from vEL0 to vEL2 when the virtual HCR_EL2.{E2H,TGE} == {1,1} (a VHE guest hypervisor). [maz: switch to common exception injection framework, illegal exeption return handling] Reviewed-by: Ganapatrao Kulkarni <gankulkarni@os.amperecomputing.com> Signed-off-by: Jintack Lim <jintack.lim@linaro.org> Signed-off-by: Christoffer Dall <christoffer.dall@arm.com> Signed-off-by: Marc Zyngier <maz@kernel.org> Link: https://lore.kernel.org/r/20230209175820.1939006-10-maz@kernel.org Signed-off-by: Oliver Upton <oliver.upton@linux.dev>
2023-02-09 17:58:11 +00:00
case PSR_MODE_EL0t:
KVM: arm64: Factor out helper for selecting exception target EL Pull out the exception target selection from pend_sync_exception() for general use. Use PSR_MODE_ELxh as a shorthand for the target EL, as SP_ELx selection is handled further along in the hyp's exception emulation. Reviewed-by: Marc Zyngier <maz@kernel.org> Link: https://lore.kernel.org/r/20250708172532.1699409-16-oliver.upton@linux.dev Signed-off-by: Oliver Upton <oliver.upton@linux.dev>
2025-07-08 10:25:20 -07:00
return vcpu_el2_tge_is_set(vcpu) ? PSR_MODE_EL2h : PSR_MODE_EL1h;
KVM: arm64: nv: Support virtual EL2 exceptions Support injecting exceptions and performing exception returns to and from virtual EL2. This must be done entirely in software except when taking an exception from vEL0 to vEL2 when the virtual HCR_EL2.{E2H,TGE} == {1,1} (a VHE guest hypervisor). [maz: switch to common exception injection framework, illegal exeption return handling] Reviewed-by: Ganapatrao Kulkarni <gankulkarni@os.amperecomputing.com> Signed-off-by: Jintack Lim <jintack.lim@linaro.org> Signed-off-by: Christoffer Dall <christoffer.dall@arm.com> Signed-off-by: Marc Zyngier <maz@kernel.org> Link: https://lore.kernel.org/r/20230209175820.1939006-10-maz@kernel.org Signed-off-by: Oliver Upton <oliver.upton@linux.dev>
2023-02-09 17:58:11 +00:00
default:
BUG();
}
}
KVM: arm64: nv: Ensure Address size faults affect correct ESR For historical reasons, Address size faults are first injected into the guest as an SEA and ESR_EL1 is subsequently modified to reflect the correct FSC. Of course, when dealing with a vEL2 this should poke ESR_EL2. Reviewed-by: Marc Zyngier <maz@kernel.org> Link: https://lore.kernel.org/r/20250708172532.1699409-17-oliver.upton@linux.dev Signed-off-by: Oliver Upton <oliver.upton@linux.dev>
2025-07-08 10:25:21 -07:00
static enum vcpu_sysreg exception_esr_elx(struct kvm_vcpu *vcpu)
{
if (exception_target_el(vcpu) == PSR_MODE_EL2h)
return ESR_EL2;
return ESR_EL1;
}
static enum vcpu_sysreg exception_far_elx(struct kvm_vcpu *vcpu)
{
if (exception_target_el(vcpu) == PSR_MODE_EL2h)
return FAR_EL2;
return FAR_EL1;
}
KVM: arm64: Factor out helper for selecting exception target EL Pull out the exception target selection from pend_sync_exception() for general use. Use PSR_MODE_ELxh as a shorthand for the target EL, as SP_ELx selection is handled further along in the hyp's exception emulation. Reviewed-by: Marc Zyngier <maz@kernel.org> Link: https://lore.kernel.org/r/20250708172532.1699409-16-oliver.upton@linux.dev Signed-off-by: Oliver Upton <oliver.upton@linux.dev>
2025-07-08 10:25:20 -07:00
static void pend_sync_exception(struct kvm_vcpu *vcpu)
{
if (exception_target_el(vcpu) == PSR_MODE_EL1h)
kvm_pend_exception(vcpu, EXCEPT_AA64_EL1_SYNC);
else
kvm_pend_exception(vcpu, EXCEPT_AA64_EL2_SYNC);
}
KVM: arm64: Route SEAs to the SError vector when EASE is set One of the finest additions of FEAT_DoubleFault2 is the ability for software to request *synchronous* external aborts be taken to the SError vector, which of coure are *asynchronous* in nature. Opinions be damned, implement the architecture and send SEAs to the SError vector if EASE is set for the target context. Reviewed-by: Marc Zyngier <maz@kernel.org> Link: https://lore.kernel.org/r/20250708172532.1699409-18-oliver.upton@linux.dev Signed-off-by: Oliver Upton <oliver.upton@linux.dev>
2025-07-08 10:25:22 -07:00
static void pend_serror_exception(struct kvm_vcpu *vcpu)
{
if (exception_target_el(vcpu) == PSR_MODE_EL1h)
kvm_pend_exception(vcpu, EXCEPT_AA64_EL1_SERR);
else
kvm_pend_exception(vcpu, EXCEPT_AA64_EL2_SERR);
}
static bool __effective_sctlr2_bit(struct kvm_vcpu *vcpu, unsigned int idx)
{
u64 sctlr2;
if (!kvm_has_sctlr2(vcpu->kvm))
return false;
if (is_nested_ctxt(vcpu) &&
!(__vcpu_sys_reg(vcpu, HCRX_EL2) & HCRX_EL2_SCTLR2En))
return false;
if (exception_target_el(vcpu) == PSR_MODE_EL1h)
sctlr2 = vcpu_read_sys_reg(vcpu, SCTLR2_EL1);
else
sctlr2 = vcpu_read_sys_reg(vcpu, SCTLR2_EL2);
return sctlr2 & BIT(idx);
}
static bool effective_sctlr2_ease(struct kvm_vcpu *vcpu)
{
return __effective_sctlr2_bit(vcpu, SCTLR2_EL1_EASE_SHIFT);
}
KVM: arm64: nv: Honor SError routing effects of SCTLR2_ELx.NMEA As the name might imply, when NMEA is set SErrors are non-maskable and can be taken regardless of PSTATE.A. As is the recurring theme with DoubleFault2, the effects on SError routing are entirely backwards to this. If at EL1, NMEA is *not* considered for SError routing when TMEA is set and the exception is taken to EL2 when PSTATE.A is set. Link: https://lore.kernel.org/r/20250708172532.1699409-20-oliver.upton@linux.dev Signed-off-by: Oliver Upton <oliver.upton@linux.dev>
2025-07-08 10:25:24 -07:00
static bool effective_sctlr2_nmea(struct kvm_vcpu *vcpu)
{
return __effective_sctlr2_bit(vcpu, SCTLR2_EL1_NMEA_SHIFT);
}
arm64: KVM: fault injection into a guest Implement the injection of a fault (undefined, data abort or prefetch abort) into a 64bit guest. Reviewed-by: Christopher Covington <cov@codeaurora.org> Reviewed-by: Catalin Marinas <catalin.marinas@arm.com> Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
2012-12-17 12:27:42 +00:00
static void inject_abt64(struct kvm_vcpu *vcpu, bool is_iabt, unsigned long addr)
{
unsigned long cpsr = *vcpu_cpsr(vcpu);
arm64: KVM: Set cpsr before spsr on fault injection We need to set cpsr before determining the spsr bank, as the bank depends on the target exception level of the injection, not the current mode of the vcpu. Normally this is one in the same (EL1), but not when we manage to trap an EL0 fault. It still doesn't really matter for the 64-bit EL0 case though, as vcpu_spsr() unconditionally uses the EL1 bank for that. However the 32-bit EL0 case gets fun, as that path will lead to the BUG() in vcpu_spsr32(). This patch fixes the assignment order and also modifies some white space in order to better group pairs of lines that have strict order. Cc: stable@vger.kernel.org # v4.5 Signed-off-by: Andrew Jones <drjones@redhat.com> Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
2016-07-22 10:38:46 -04:00
bool is_aarch32 = vcpu_mode_is_32bit(vcpu);
KVM: arm64: Treat ESR_EL2 as a 64-bit register ESR_EL2 was defined as a 32-bit register in the initial release of the ARM Architecture Manual for Armv8-A, and was later extended to 64 bits, with bits [63:32] RES0. ARMv8.7 introduced FEAT_LS64, which makes use of bits [36:32]. KVM treats ESR_EL1 as a 64-bit register when saving and restoring the guest context, but ESR_EL2 is handled as a 32-bit register. Start treating ESR_EL2 as a 64-bit register to allow KVM to make use of the most significant 32 bits in the future. The type chosen to represent ESR_EL2 is u64, as that is consistent with the notation KVM overwhelmingly uses today (u32), and how the rest of the registers are declared. Signed-off-by: Alexandru Elisei <alexandru.elisei@arm.com> Reviewed-by: Marc Zyngier <maz@kernel.org> Link: https://lore.kernel.org/r/20220425114444.368693-5-alexandru.elisei@arm.com Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
2022-04-25 12:44:43 +01:00
u64 esr = 0;
arm64: KVM: fault injection into a guest Implement the injection of a fault (undefined, data abort or prefetch abort) into a 64bit guest. Reviewed-by: Christopher Covington <cov@codeaurora.org> Reviewed-by: Catalin Marinas <catalin.marinas@arm.com> Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
2012-12-17 12:27:42 +00:00
KVM: arm64: Route SEAs to the SError vector when EASE is set One of the finest additions of FEAT_DoubleFault2 is the ability for software to request *synchronous* external aborts be taken to the SError vector, which of coure are *asynchronous* in nature. Opinions be damned, implement the architecture and send SEAs to the SError vector if EASE is set for the target context. Reviewed-by: Marc Zyngier <maz@kernel.org> Link: https://lore.kernel.org/r/20250708172532.1699409-18-oliver.upton@linux.dev Signed-off-by: Oliver Upton <oliver.upton@linux.dev>
2025-07-08 10:25:22 -07:00
/* This delight is brought to you by FEAT_DoubleFault2. */
if (effective_sctlr2_ease(vcpu))
pend_serror_exception(vcpu);
else
pend_sync_exception(vcpu);
arm64: KVM: fault injection into a guest Implement the injection of a fault (undefined, data abort or prefetch abort) into a 64bit guest. Reviewed-by: Christopher Covington <cov@codeaurora.org> Reviewed-by: Catalin Marinas <catalin.marinas@arm.com> Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
2012-12-17 12:27:42 +00:00
/*
* Build an {i,d}abort, depending on the level and the
* instruction set. Report an external synchronous abort.
*/
if (kvm_vcpu_trap_il_is32bit(vcpu))
arm64: kvm: move to ESR_ELx macros Now that we have common ESR_ELx macros, make use of them in the arm64 KVM code. The addition of <asm/esr.h> to the include path highlighted badly ordered (i.e. not alphabetical) include lists; these are changed to alphabetical order. There should be no functional change as a result of this patch. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Reviewed-by: Christoffer Dall <christoffer.dall@linaro.org> Cc: Catalin Marinas <catalin.marinas@arm.com> Cc: Marc Zyngier <marc.zyngier@arm.com> Cc: Peter Maydell <peter.maydell@linaro.org> Cc: Will Deacon <will.deacon@arm.com>
2014-11-24 13:59:30 +00:00
esr |= ESR_ELx_IL;
arm64: KVM: fault injection into a guest Implement the injection of a fault (undefined, data abort or prefetch abort) into a 64bit guest. Reviewed-by: Christopher Covington <cov@codeaurora.org> Reviewed-by: Catalin Marinas <catalin.marinas@arm.com> Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
2012-12-17 12:27:42 +00:00
/*
* Here, the guest runs in AArch64 mode when in EL1. If we get
* an AArch32 fault, it means we managed to trap an EL0 fault.
*/
if (is_aarch32 || (cpsr & PSR_MODE_MASK) == PSR_MODE_EL0t)
arm64: kvm: move to ESR_ELx macros Now that we have common ESR_ELx macros, make use of them in the arm64 KVM code. The addition of <asm/esr.h> to the include path highlighted badly ordered (i.e. not alphabetical) include lists; these are changed to alphabetical order. There should be no functional change as a result of this patch. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Reviewed-by: Christoffer Dall <christoffer.dall@linaro.org> Cc: Catalin Marinas <catalin.marinas@arm.com> Cc: Marc Zyngier <marc.zyngier@arm.com> Cc: Peter Maydell <peter.maydell@linaro.org> Cc: Will Deacon <will.deacon@arm.com>
2014-11-24 13:59:30 +00:00
esr |= (ESR_ELx_EC_IABT_LOW << ESR_ELx_EC_SHIFT);
arm64: KVM: fault injection into a guest Implement the injection of a fault (undefined, data abort or prefetch abort) into a 64bit guest. Reviewed-by: Christopher Covington <cov@codeaurora.org> Reviewed-by: Catalin Marinas <catalin.marinas@arm.com> Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
2012-12-17 12:27:42 +00:00
else
arm64: kvm: move to ESR_ELx macros Now that we have common ESR_ELx macros, make use of them in the arm64 KVM code. The addition of <asm/esr.h> to the include path highlighted badly ordered (i.e. not alphabetical) include lists; these are changed to alphabetical order. There should be no functional change as a result of this patch. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Reviewed-by: Christoffer Dall <christoffer.dall@linaro.org> Cc: Catalin Marinas <catalin.marinas@arm.com> Cc: Marc Zyngier <marc.zyngier@arm.com> Cc: Peter Maydell <peter.maydell@linaro.org> Cc: Will Deacon <will.deacon@arm.com>
2014-11-24 13:59:30 +00:00
esr |= (ESR_ELx_EC_IABT_CUR << ESR_ELx_EC_SHIFT);
arm64: KVM: fault injection into a guest Implement the injection of a fault (undefined, data abort or prefetch abort) into a 64bit guest. Reviewed-by: Christopher Covington <cov@codeaurora.org> Reviewed-by: Catalin Marinas <catalin.marinas@arm.com> Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
2012-12-17 12:27:42 +00:00
if (!is_iabt)
kvm: arm64: Fix EC field in inject_abt64 The EC field of the constructed ESR is conditionally modified by ORing in ESR_ELx_EC_DABT_LOW for a data abort. However, ESR_ELx_EC_SHIFT is missing from this condition. Signed-off-by: Matt Evans <matt.evans@arm.com> Acked-by: Marc Zyngier <marc.zyngier@arm.com> Cc: <stable@vger.kernel.org> Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
2016-05-16 13:54:56 +01:00
esr |= ESR_ELx_EC_DABT_LOW << ESR_ELx_EC_SHIFT;
arm64: KVM: fault injection into a guest Implement the injection of a fault (undefined, data abort or prefetch abort) into a 64bit guest. Reviewed-by: Christopher Covington <cov@codeaurora.org> Reviewed-by: Catalin Marinas <catalin.marinas@arm.com> Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
2012-12-17 12:27:42 +00:00
KVM: arm64: nv: Support virtual EL2 exceptions Support injecting exceptions and performing exception returns to and from virtual EL2. This must be done entirely in software except when taking an exception from vEL0 to vEL2 when the virtual HCR_EL2.{E2H,TGE} == {1,1} (a VHE guest hypervisor). [maz: switch to common exception injection framework, illegal exeption return handling] Reviewed-by: Ganapatrao Kulkarni <gankulkarni@os.amperecomputing.com> Signed-off-by: Jintack Lim <jintack.lim@linaro.org> Signed-off-by: Christoffer Dall <christoffer.dall@arm.com> Signed-off-by: Marc Zyngier <maz@kernel.org> Link: https://lore.kernel.org/r/20230209175820.1939006-10-maz@kernel.org Signed-off-by: Oliver Upton <oliver.upton@linux.dev>
2023-02-09 17:58:11 +00:00
esr |= ESR_ELx_FSC_EXTABT;
KVM: arm64: nv: Ensure Address size faults affect correct ESR For historical reasons, Address size faults are first injected into the guest as an SEA and ESR_EL1 is subsequently modified to reflect the correct FSC. Of course, when dealing with a vEL2 this should poke ESR_EL2. Reviewed-by: Marc Zyngier <maz@kernel.org> Link: https://lore.kernel.org/r/20250708172532.1699409-17-oliver.upton@linux.dev Signed-off-by: Oliver Upton <oliver.upton@linux.dev>
2025-07-08 10:25:21 -07:00
vcpu_write_sys_reg(vcpu, addr, exception_far_elx(vcpu));
vcpu_write_sys_reg(vcpu, esr, exception_esr_elx(vcpu));
arm64: KVM: fault injection into a guest Implement the injection of a fault (undefined, data abort or prefetch abort) into a 64bit guest. Reviewed-by: Christopher Covington <cov@codeaurora.org> Reviewed-by: Catalin Marinas <catalin.marinas@arm.com> Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
2012-12-17 12:27:42 +00:00
}
static void inject_undef64(struct kvm_vcpu *vcpu)
{
KVM: arm64: Treat ESR_EL2 as a 64-bit register ESR_EL2 was defined as a 32-bit register in the initial release of the ARM Architecture Manual for Armv8-A, and was later extended to 64 bits, with bits [63:32] RES0. ARMv8.7 introduced FEAT_LS64, which makes use of bits [36:32]. KVM treats ESR_EL1 as a 64-bit register when saving and restoring the guest context, but ESR_EL2 is handled as a 32-bit register. Start treating ESR_EL2 as a 64-bit register to allow KVM to make use of the most significant 32 bits in the future. The type chosen to represent ESR_EL2 is u64, as that is consistent with the notation KVM overwhelmingly uses today (u32), and how the rest of the registers are declared. Signed-off-by: Alexandru Elisei <alexandru.elisei@arm.com> Reviewed-by: Marc Zyngier <maz@kernel.org> Link: https://lore.kernel.org/r/20220425114444.368693-5-alexandru.elisei@arm.com Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
2022-04-25 12:44:43 +01:00
u64 esr = (ESR_ELx_EC_UNKNOWN << ESR_ELx_EC_SHIFT);
arm64: KVM: fault injection into a guest Implement the injection of a fault (undefined, data abort or prefetch abort) into a 64bit guest. Reviewed-by: Christopher Covington <cov@codeaurora.org> Reviewed-by: Catalin Marinas <catalin.marinas@arm.com> Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
2012-12-17 12:27:42 +00:00
KVM: arm64: nv: Support virtual EL2 exceptions Support injecting exceptions and performing exception returns to and from virtual EL2. This must be done entirely in software except when taking an exception from vEL0 to vEL2 when the virtual HCR_EL2.{E2H,TGE} == {1,1} (a VHE guest hypervisor). [maz: switch to common exception injection framework, illegal exeption return handling] Reviewed-by: Ganapatrao Kulkarni <gankulkarni@os.amperecomputing.com> Signed-off-by: Jintack Lim <jintack.lim@linaro.org> Signed-off-by: Christoffer Dall <christoffer.dall@arm.com> Signed-off-by: Marc Zyngier <maz@kernel.org> Link: https://lore.kernel.org/r/20230209175820.1939006-10-maz@kernel.org Signed-off-by: Oliver Upton <oliver.upton@linux.dev>
2023-02-09 17:58:11 +00:00
pend_sync_exception(vcpu);
arm64: KVM: fault injection into a guest Implement the injection of a fault (undefined, data abort or prefetch abort) into a 64bit guest. Reviewed-by: Christopher Covington <cov@codeaurora.org> Reviewed-by: Catalin Marinas <catalin.marinas@arm.com> Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
2012-12-17 12:27:42 +00:00
/*
* Build an unknown exception, depending on the instruction
* set.
*/
if (kvm_vcpu_trap_il_is32bit(vcpu))
arm64: kvm: move to ESR_ELx macros Now that we have common ESR_ELx macros, make use of them in the arm64 KVM code. The addition of <asm/esr.h> to the include path highlighted badly ordered (i.e. not alphabetical) include lists; these are changed to alphabetical order. There should be no functional change as a result of this patch. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Reviewed-by: Christoffer Dall <christoffer.dall@linaro.org> Cc: Catalin Marinas <catalin.marinas@arm.com> Cc: Marc Zyngier <marc.zyngier@arm.com> Cc: Peter Maydell <peter.maydell@linaro.org> Cc: Will Deacon <will.deacon@arm.com>
2014-11-24 13:59:30 +00:00
esr |= ESR_ELx_IL;
arm64: KVM: fault injection into a guest Implement the injection of a fault (undefined, data abort or prefetch abort) into a 64bit guest. Reviewed-by: Christopher Covington <cov@codeaurora.org> Reviewed-by: Catalin Marinas <catalin.marinas@arm.com> Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
2012-12-17 12:27:42 +00:00
KVM: arm64: nv: Ensure Address size faults affect correct ESR For historical reasons, Address size faults are first injected into the guest as an SEA and ESR_EL1 is subsequently modified to reflect the correct FSC. Of course, when dealing with a vEL2 this should poke ESR_EL2. Reviewed-by: Marc Zyngier <maz@kernel.org> Link: https://lore.kernel.org/r/20250708172532.1699409-17-oliver.upton@linux.dev Signed-off-by: Oliver Upton <oliver.upton@linux.dev>
2025-07-08 10:25:21 -07:00
vcpu_write_sys_reg(vcpu, esr, exception_esr_elx(vcpu));
arm64: KVM: fault injection into a guest Implement the injection of a fault (undefined, data abort or prefetch abort) into a 64bit guest. Reviewed-by: Christopher Covington <cov@codeaurora.org> Reviewed-by: Catalin Marinas <catalin.marinas@arm.com> Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
2012-12-17 12:27:42 +00:00
}
KVM: arm64: Consolidate exception injection Move the AArch32 exception injection code back into the inject_fault.c file, removing the need for a few non-static functions now that AArch32 host support is a thing of the past. Signed-off-by: Marc Zyngier <maz@kernel.org>
2020-10-15 09:45:24 +01:00
#define DFSR_FSC_EXTABT_LPAE 0x10
#define DFSR_FSC_EXTABT_nLPAE 0x08
#define DFSR_LPAE BIT(9)
KVM: arm64: Move AArch32 exceptions over to AArch64 sysregs The use of the AArch32-specific accessors have always been a bit annoying on 64bit, and it is time for a change. Let's move the AArch32 exception injection over to the AArch64 encoding, which requires us to split the two halves of FAR_EL1 into DFAR and IFAR. This enables us to drop the preempt_disable() games on VHE, and to kill the last user of the vcpu_cp15() macro. Signed-off-by: Marc Zyngier <maz@kernel.org>
2020-10-27 22:23:28 +00:00
#define TTBCR_EAE BIT(31)
KVM: arm64: Consolidate exception injection Move the AArch32 exception injection code back into the inject_fault.c file, removing the need for a few non-static functions now that AArch32 host support is a thing of the past. Signed-off-by: Marc Zyngier <maz@kernel.org>
2020-10-15 09:45:24 +01:00
static void inject_undef32(struct kvm_vcpu *vcpu)
{
KVM: arm64: Move vcpu PC/Exception flags to the input flag set The PC update flags (which also deal with exception injection) is one of the most complicated use of the flag we have. Make it more fool prof by: - moving it over to the new accessors and assign it to the input flag set - turn the combination of generic ELx flags with another flag indicating the target EL itself into an explicit set of flags for each EL and vector combination - add a new accessor to pend the exception This is otherwise a pretty straightformward conversion. Reviewed-by: Fuad Tabba <tabba@google.com> Reviewed-by: Reiji Watanabe <reijiw@google.com> Signed-off-by: Marc Zyngier <maz@kernel.org>
2022-05-28 12:38:18 +01:00
kvm_pend_exception(vcpu, EXCEPT_AA32_UND);
KVM: arm64: Consolidate exception injection Move the AArch32 exception injection code back into the inject_fault.c file, removing the need for a few non-static functions now that AArch32 host support is a thing of the past. Signed-off-by: Marc Zyngier <maz@kernel.org>
2020-10-15 09:45:24 +01:00
}
/*
* Modelled after TakeDataAbortException() and TakePrefetchAbortException
* pseudocode.
*/
KVM: arm64: Move AArch32 exceptions over to AArch64 sysregs The use of the AArch32-specific accessors have always been a bit annoying on 64bit, and it is time for a change. Let's move the AArch32 exception injection over to the AArch64 encoding, which requires us to split the two halves of FAR_EL1 into DFAR and IFAR. This enables us to drop the preempt_disable() games on VHE, and to kill the last user of the vcpu_cp15() macro. Signed-off-by: Marc Zyngier <maz@kernel.org>
2020-10-27 22:23:28 +00:00
static void inject_abt32(struct kvm_vcpu *vcpu, bool is_pabt, u32 addr)
KVM: arm64: Consolidate exception injection Move the AArch32 exception injection code back into the inject_fault.c file, removing the need for a few non-static functions now that AArch32 host support is a thing of the past. Signed-off-by: Marc Zyngier <maz@kernel.org>
2020-10-15 09:45:24 +01:00
{
KVM: arm64: Move AArch32 exceptions over to AArch64 sysregs The use of the AArch32-specific accessors have always been a bit annoying on 64bit, and it is time for a change. Let's move the AArch32 exception injection over to the AArch64 encoding, which requires us to split the two halves of FAR_EL1 into DFAR and IFAR. This enables us to drop the preempt_disable() games on VHE, and to kill the last user of the vcpu_cp15() macro. Signed-off-by: Marc Zyngier <maz@kernel.org>
2020-10-27 22:23:28 +00:00
u64 far;
u32 fsr;
/* Give the guest an IMPLEMENTATION DEFINED exception */
if (vcpu_read_sys_reg(vcpu, TCR_EL1) & TTBCR_EAE) {
fsr = DFSR_LPAE | DFSR_FSC_EXTABT_LPAE;
} else {
KVM: arm64: Fix typos Fix typos, most reported by "codespell arch/arm64". Only touches comments, no code changes. Signed-off-by: Bjorn Helgaas <bhelgaas@google.com> Cc: James Morse <james.morse@arm.com> Cc: Suzuki K Poulose <suzuki.poulose@arm.com> Cc: Zenghui Yu <yuzenghui@huawei.com> Cc: Catalin Marinas <catalin.marinas@arm.com> Cc: Will Deacon <will@kernel.org> Cc: linux-arm-kernel@lists.infradead.org Cc: kvmarm@lists.linux.dev Reviewed-by: Zenghui Yu <yuzenghui@huawei.com> Reviewed-by: Randy Dunlap <rdunlap@infradead.org> Link: https://lore.kernel.org/r/20240103231605.1801364-6-helgaas@kernel.org Signed-off-by: Oliver Upton <oliver.upton@linux.dev>
2024-01-03 17:16:02 -06:00
/* no need to shuffle FS[4] into DFSR[10] as it's 0 */
KVM: arm64: Move AArch32 exceptions over to AArch64 sysregs The use of the AArch32-specific accessors have always been a bit annoying on 64bit, and it is time for a change. Let's move the AArch32 exception injection over to the AArch64 encoding, which requires us to split the two halves of FAR_EL1 into DFAR and IFAR. This enables us to drop the preempt_disable() games on VHE, and to kill the last user of the vcpu_cp15() macro. Signed-off-by: Marc Zyngier <maz@kernel.org>
2020-10-27 22:23:28 +00:00
fsr = DFSR_FSC_EXTABT_nLPAE;
}
KVM: arm64: Consolidate exception injection Move the AArch32 exception injection code back into the inject_fault.c file, removing the need for a few non-static functions now that AArch32 host support is a thing of the past. Signed-off-by: Marc Zyngier <maz@kernel.org>
2020-10-15 09:45:24 +01:00
KVM: arm64: Move AArch32 exceptions over to AArch64 sysregs The use of the AArch32-specific accessors have always been a bit annoying on 64bit, and it is time for a change. Let's move the AArch32 exception injection over to the AArch64 encoding, which requires us to split the two halves of FAR_EL1 into DFAR and IFAR. This enables us to drop the preempt_disable() games on VHE, and to kill the last user of the vcpu_cp15() macro. Signed-off-by: Marc Zyngier <maz@kernel.org>
2020-10-27 22:23:28 +00:00
far = vcpu_read_sys_reg(vcpu, FAR_EL1);
KVM: arm64: Consolidate exception injection Move the AArch32 exception injection code back into the inject_fault.c file, removing the need for a few non-static functions now that AArch32 host support is a thing of the past. Signed-off-by: Marc Zyngier <maz@kernel.org>
2020-10-15 09:45:24 +01:00
if (is_pabt) {
KVM: arm64: Move vcpu PC/Exception flags to the input flag set The PC update flags (which also deal with exception injection) is one of the most complicated use of the flag we have. Make it more fool prof by: - moving it over to the new accessors and assign it to the input flag set - turn the combination of generic ELx flags with another flag indicating the target EL itself into an explicit set of flags for each EL and vector combination - add a new accessor to pend the exception This is otherwise a pretty straightformward conversion. Reviewed-by: Fuad Tabba <tabba@google.com> Reviewed-by: Reiji Watanabe <reijiw@google.com> Signed-off-by: Marc Zyngier <maz@kernel.org>
2022-05-28 12:38:18 +01:00
kvm_pend_exception(vcpu, EXCEPT_AA32_IABT);
KVM: arm64: Move AArch32 exceptions over to AArch64 sysregs The use of the AArch32-specific accessors have always been a bit annoying on 64bit, and it is time for a change. Let's move the AArch32 exception injection over to the AArch64 encoding, which requires us to split the two halves of FAR_EL1 into DFAR and IFAR. This enables us to drop the preempt_disable() games on VHE, and to kill the last user of the vcpu_cp15() macro. Signed-off-by: Marc Zyngier <maz@kernel.org>
2020-10-27 22:23:28 +00:00
far &= GENMASK(31, 0);
far |= (u64)addr << 32;
vcpu_write_sys_reg(vcpu, fsr, IFSR32_EL2);
KVM: arm64: Consolidate exception injection Move the AArch32 exception injection code back into the inject_fault.c file, removing the need for a few non-static functions now that AArch32 host support is a thing of the past. Signed-off-by: Marc Zyngier <maz@kernel.org>
2020-10-15 09:45:24 +01:00
} else { /* !iabt */
KVM: arm64: Move vcpu PC/Exception flags to the input flag set The PC update flags (which also deal with exception injection) is one of the most complicated use of the flag we have. Make it more fool prof by: - moving it over to the new accessors and assign it to the input flag set - turn the combination of generic ELx flags with another flag indicating the target EL itself into an explicit set of flags for each EL and vector combination - add a new accessor to pend the exception This is otherwise a pretty straightformward conversion. Reviewed-by: Fuad Tabba <tabba@google.com> Reviewed-by: Reiji Watanabe <reijiw@google.com> Signed-off-by: Marc Zyngier <maz@kernel.org>
2022-05-28 12:38:18 +01:00
kvm_pend_exception(vcpu, EXCEPT_AA32_DABT);
KVM: arm64: Move AArch32 exceptions over to AArch64 sysregs The use of the AArch32-specific accessors have always been a bit annoying on 64bit, and it is time for a change. Let's move the AArch32 exception injection over to the AArch64 encoding, which requires us to split the two halves of FAR_EL1 into DFAR and IFAR. This enables us to drop the preempt_disable() games on VHE, and to kill the last user of the vcpu_cp15() macro. Signed-off-by: Marc Zyngier <maz@kernel.org>
2020-10-27 22:23:28 +00:00
far &= GENMASK(63, 32);
far |= addr;
vcpu_write_sys_reg(vcpu, fsr, ESR_EL1);
KVM: arm64: Consolidate exception injection Move the AArch32 exception injection code back into the inject_fault.c file, removing the need for a few non-static functions now that AArch32 host support is a thing of the past. Signed-off-by: Marc Zyngier <maz@kernel.org>
2020-10-15 09:45:24 +01:00
}
KVM: arm64: Move AArch32 exceptions over to AArch64 sysregs The use of the AArch32-specific accessors have always been a bit annoying on 64bit, and it is time for a change. Let's move the AArch32 exception injection over to the AArch64 encoding, which requires us to split the two halves of FAR_EL1 into DFAR and IFAR. This enables us to drop the preempt_disable() games on VHE, and to kill the last user of the vcpu_cp15() macro. Signed-off-by: Marc Zyngier <maz@kernel.org>
2020-10-27 22:23:28 +00:00
vcpu_write_sys_reg(vcpu, far, FAR_EL1);
KVM: arm64: Consolidate exception injection Move the AArch32 exception injection code back into the inject_fault.c file, removing the need for a few non-static functions now that AArch32 host support is a thing of the past. Signed-off-by: Marc Zyngier <maz@kernel.org>
2020-10-15 09:45:24 +01:00
}
KVM: arm64: nv: Respect exception routing rules for SEAs Synchronous external aborts are taken to EL2 if ELIsInHost() or HCR_EL2.TEA=1. Rework the SEA injection plumbing to respect the imposed routing of the guest hypervisor and opportunistically rephrase things to make their function a bit more obvious. Reviewed-by: Marc Zyngier <maz@kernel.org> Link: https://lore.kernel.org/r/20250708172532.1699409-6-oliver.upton@linux.dev Signed-off-by: Oliver Upton <oliver.upton@linux.dev>
2025-07-08 10:25:10 -07:00
static void __kvm_inject_sea(struct kvm_vcpu *vcpu, bool iabt, u64 addr)
arm64: KVM: fault injection into a guest Implement the injection of a fault (undefined, data abort or prefetch abort) into a 64bit guest. Reviewed-by: Christopher Covington <cov@codeaurora.org> Reviewed-by: Catalin Marinas <catalin.marinas@arm.com> Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
2012-12-17 12:27:42 +00:00
{
KVM: arm/arm64: Introduce vcpu_el1_is_32bit We have numerous checks around that checks if the HCR_EL2 has the RW bit set to figure out if we're running an AArch64 or AArch32 VM. In some cases, directly checking the RW bit (given its unintuitive name), is a bit confusing, and that's not going to improve as we move logic around for the following patches that optimize KVM on AArch64 hosts with VHE. Therefore, introduce a helper, vcpu_el1_is_32bit, and replace existing direct checks of HCR_EL2.RW with the helper. Reviewed-by: Julien Grall <julien.grall@arm.com> Reviewed-by: Julien Thierry <julien.thierry@arm.com> Acked-by: Marc Zyngier <marc.zyngier@arm.com> Reviewed-by: Andrew Jones <drjones@redhat.com> Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org> Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
2017-12-13 22:56:48 +01:00
if (vcpu_el1_is_32bit(vcpu))
KVM: arm64: nv: Respect exception routing rules for SEAs Synchronous external aborts are taken to EL2 if ELIsInHost() or HCR_EL2.TEA=1. Rework the SEA injection plumbing to respect the imposed routing of the guest hypervisor and opportunistically rephrase things to make their function a bit more obvious. Reviewed-by: Marc Zyngier <maz@kernel.org> Link: https://lore.kernel.org/r/20250708172532.1699409-6-oliver.upton@linux.dev Signed-off-by: Oliver Upton <oliver.upton@linux.dev>
2025-07-08 10:25:10 -07:00
inject_abt32(vcpu, iabt, addr);
arm64: KVM: Fix host crash when injecting a fault into a 32bit guest When injecting a fault into a misbehaving 32bit guest, it seems rather idiotic to also inject a 64bit fault that is only going to corrupt the guest state. This leads to a situation where we perform an illegal exception return at EL2 causing the host to crash instead of killing the guest. Just fix the stupid bug that has been there from day 1. Cc: <stable@vger.kernel.org> Reported-by: Russell King <rmk+kernel@arm.linux.org.uk> Tested-by: Russell King <rmk+kernel@arm.linux.org.uk> Signed-off-by: Marc Zyngier <marc.zyngier@arm.com> Signed-off-by: Will Deacon <will.deacon@arm.com>
2015-08-27 16:10:01 +01:00
else
KVM: arm64: nv: Respect exception routing rules for SEAs Synchronous external aborts are taken to EL2 if ELIsInHost() or HCR_EL2.TEA=1. Rework the SEA injection plumbing to respect the imposed routing of the guest hypervisor and opportunistically rephrase things to make their function a bit more obvious. Reviewed-by: Marc Zyngier <maz@kernel.org> Link: https://lore.kernel.org/r/20250708172532.1699409-6-oliver.upton@linux.dev Signed-off-by: Oliver Upton <oliver.upton@linux.dev>
2025-07-08 10:25:10 -07:00
inject_abt64(vcpu, iabt, addr);
arm64: KVM: fault injection into a guest Implement the injection of a fault (undefined, data abort or prefetch abort) into a 64bit guest. Reviewed-by: Christopher Covington <cov@codeaurora.org> Reviewed-by: Catalin Marinas <catalin.marinas@arm.com> Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
2012-12-17 12:27:42 +00:00
}
KVM: arm64: nv: Respect exception routing rules for SEAs Synchronous external aborts are taken to EL2 if ELIsInHost() or HCR_EL2.TEA=1. Rework the SEA injection plumbing to respect the imposed routing of the guest hypervisor and opportunistically rephrase things to make their function a bit more obvious. Reviewed-by: Marc Zyngier <maz@kernel.org> Link: https://lore.kernel.org/r/20250708172532.1699409-6-oliver.upton@linux.dev Signed-off-by: Oliver Upton <oliver.upton@linux.dev>
2025-07-08 10:25:10 -07:00
static bool kvm_sea_target_is_el2(struct kvm_vcpu *vcpu)
arm64: KVM: fault injection into a guest Implement the injection of a fault (undefined, data abort or prefetch abort) into a 64bit guest. Reviewed-by: Christopher Covington <cov@codeaurora.org> Reviewed-by: Catalin Marinas <catalin.marinas@arm.com> Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
2012-12-17 12:27:42 +00:00
{
KVM: arm64: nv: Take "masked" aborts to EL2 when HCRX_EL2.TMEA is set HCRX_EL2.TMEA further modifies the external abort behavior where unmasked aborts are taken to EL1 and masked aborts are taken to EL2. It's rather weird when you consider that SEAs are, well, *synchronous* and therefore not actually maskable. However, for the purposes of exception routing, they're considered "masked" if the A flag is set. This gets a bit hairier when considering the fact that TMEA also enables vSErrors, i.e. KVM has delegated the HW vSError context to the guest hypervisor. We can keep the vSError context delegation as-is by taking advantage of a couple properties: - If SErrors are unmasked, the 'physical' SError can be taken in-context immediately. In other words, KVM can emulate the EL1 SError while preserving vEL2's ownership of the vSError context. - If SErrors are masked, the 'physical' SError is taken to EL2 immediately and needs the usual nested exception entry. Note that the new in-context handling has the benign effect where unmasked SError injections are emulated even for non-nested VMs. Reviewed-by: Marc Zyngier <maz@kernel.org> Link: https://lore.kernel.org/r/20250708172532.1699409-19-oliver.upton@linux.dev Signed-off-by: Oliver Upton <oliver.upton@linux.dev>
2025-07-08 10:25:23 -07:00
if (__vcpu_sys_reg(vcpu, HCR_EL2) & (HCR_TGE | HCR_TEA))
return true;
if (!vcpu_mode_priv(vcpu))
return false;
return (*vcpu_cpsr(vcpu) & PSR_A_BIT) &&
(__vcpu_sys_reg(vcpu, HCRX_EL2) & HCRX_EL2_TMEA);
KVM: arm64: nv: Respect exception routing rules for SEAs Synchronous external aborts are taken to EL2 if ELIsInHost() or HCR_EL2.TEA=1. Rework the SEA injection plumbing to respect the imposed routing of the guest hypervisor and opportunistically rephrase things to make their function a bit more obvious. Reviewed-by: Marc Zyngier <maz@kernel.org> Link: https://lore.kernel.org/r/20250708172532.1699409-6-oliver.upton@linux.dev Signed-off-by: Oliver Upton <oliver.upton@linux.dev>
2025-07-08 10:25:10 -07:00
}
int kvm_inject_sea(struct kvm_vcpu *vcpu, bool iabt, u64 addr)
{
lockdep_assert_held(&vcpu->mutex);
if (is_nested_ctxt(vcpu) && kvm_sea_target_is_el2(vcpu))
return kvm_inject_nested_sea(vcpu, iabt, addr);
__kvm_inject_sea(vcpu, iabt, addr);
return 1;
arm64: KVM: fault injection into a guest Implement the injection of a fault (undefined, data abort or prefetch abort) into a 64bit guest. Reviewed-by: Christopher Covington <cov@codeaurora.org> Reviewed-by: Catalin Marinas <catalin.marinas@arm.com> Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
2012-12-17 12:27:42 +00:00
}
KVM: arm64: Inject exception on out-of-IPA-range translation fault When taking a translation fault for an IPA that is outside of the range defined by the hypervisor (between the HW PARange and the IPA range), we stupidly treat it as an IO and forward the access to userspace. Of course, userspace can't do much with it, and things end badly. Arguably, the guest is braindead, but we should at least catch the case and inject an exception. Check the faulting IPA against: - the sanitised PARange: inject an address size fault - the IPA size: inject an abort Reported-by: Christoffer Dall <christoffer.dall@arm.com> Signed-off-by: Marc Zyngier <maz@kernel.org>
2022-04-21 15:38:10 +01:00
void kvm_inject_size_fault(struct kvm_vcpu *vcpu)
{
unsigned long addr, esr;
addr = kvm_vcpu_get_fault_ipa(vcpu);
addr |= kvm_vcpu_get_hfar(vcpu) & GENMASK(11, 0);
KVM: arm64: nv: Respect exception routing rules for SEAs Synchronous external aborts are taken to EL2 if ELIsInHost() or HCR_EL2.TEA=1. Rework the SEA injection plumbing to respect the imposed routing of the guest hypervisor and opportunistically rephrase things to make their function a bit more obvious. Reviewed-by: Marc Zyngier <maz@kernel.org> Link: https://lore.kernel.org/r/20250708172532.1699409-6-oliver.upton@linux.dev Signed-off-by: Oliver Upton <oliver.upton@linux.dev>
2025-07-08 10:25:10 -07:00
__kvm_inject_sea(vcpu, kvm_vcpu_trap_is_iabt(vcpu), addr);
KVM: arm64: Inject exception on out-of-IPA-range translation fault When taking a translation fault for an IPA that is outside of the range defined by the hypervisor (between the HW PARange and the IPA range), we stupidly treat it as an IO and forward the access to userspace. Of course, userspace can't do much with it, and things end badly. Arguably, the guest is braindead, but we should at least catch the case and inject an exception. Check the faulting IPA against: - the sanitised PARange: inject an address size fault - the IPA size: inject an abort Reported-by: Christoffer Dall <christoffer.dall@arm.com> Signed-off-by: Marc Zyngier <maz@kernel.org>
2022-04-21 15:38:10 +01:00
/*
* If AArch64 or LPAE, set FSC to 0 to indicate an Address
* Size Fault at level 0, as if exceeding PARange.
*
* Non-LPAE guests will only get the external abort, as there
KVM: arm64: Fix repeated words in comments Delete the redundant word 'to'. Signed-off-by: Jingyu Wang <jingyuwang_vip@163.com> Reviewed-by: Mukesh Ojha <quic_mojha@quicinc.com> Signed-off-by: Marc Zyngier <maz@kernel.org> Link: https://lore.kernel.org/r/20230309075919.169518-1-jingyuwang_vip@163.com
2023-03-09 15:59:19 +08:00
* is no way to describe the ASF.
KVM: arm64: Inject exception on out-of-IPA-range translation fault When taking a translation fault for an IPA that is outside of the range defined by the hypervisor (between the HW PARange and the IPA range), we stupidly treat it as an IO and forward the access to userspace. Of course, userspace can't do much with it, and things end badly. Arguably, the guest is braindead, but we should at least catch the case and inject an exception. Check the faulting IPA against: - the sanitised PARange: inject an address size fault - the IPA size: inject an abort Reported-by: Christoffer Dall <christoffer.dall@arm.com> Signed-off-by: Marc Zyngier <maz@kernel.org>
2022-04-21 15:38:10 +01:00
*/
if (vcpu_el1_is_32bit(vcpu) &&
!(vcpu_read_sys_reg(vcpu, TCR_EL1) & TTBCR_EAE))
return;
KVM: arm64: nv: Ensure Address size faults affect correct ESR For historical reasons, Address size faults are first injected into the guest as an SEA and ESR_EL1 is subsequently modified to reflect the correct FSC. Of course, when dealing with a vEL2 this should poke ESR_EL2. Reviewed-by: Marc Zyngier <maz@kernel.org> Link: https://lore.kernel.org/r/20250708172532.1699409-17-oliver.upton@linux.dev Signed-off-by: Oliver Upton <oliver.upton@linux.dev>
2025-07-08 10:25:21 -07:00
esr = vcpu_read_sys_reg(vcpu, exception_esr_elx(vcpu));
KVM: arm64: Inject exception on out-of-IPA-range translation fault When taking a translation fault for an IPA that is outside of the range defined by the hypervisor (between the HW PARange and the IPA range), we stupidly treat it as an IO and forward the access to userspace. Of course, userspace can't do much with it, and things end badly. Arguably, the guest is braindead, but we should at least catch the case and inject an exception. Check the faulting IPA against: - the sanitised PARange: inject an address size fault - the IPA size: inject an abort Reported-by: Christoffer Dall <christoffer.dall@arm.com> Signed-off-by: Marc Zyngier <maz@kernel.org>
2022-04-21 15:38:10 +01:00
esr &= ~GENMASK_ULL(5, 0);
KVM: arm64: nv: Ensure Address size faults affect correct ESR For historical reasons, Address size faults are first injected into the guest as an SEA and ESR_EL1 is subsequently modified to reflect the correct FSC. Of course, when dealing with a vEL2 this should poke ESR_EL2. Reviewed-by: Marc Zyngier <maz@kernel.org> Link: https://lore.kernel.org/r/20250708172532.1699409-17-oliver.upton@linux.dev Signed-off-by: Oliver Upton <oliver.upton@linux.dev>
2025-07-08 10:25:21 -07:00
vcpu_write_sys_reg(vcpu, esr, exception_esr_elx(vcpu));
KVM: arm64: Inject exception on out-of-IPA-range translation fault When taking a translation fault for an IPA that is outside of the range defined by the hypervisor (between the HW PARange and the IPA range), we stupidly treat it as an IO and forward the access to userspace. Of course, userspace can't do much with it, and things end badly. Arguably, the guest is braindead, but we should at least catch the case and inject an exception. Check the faulting IPA against: - the sanitised PARange: inject an address size fault - the IPA size: inject an abort Reported-by: Christoffer Dall <christoffer.dall@arm.com> Signed-off-by: Marc Zyngier <maz@kernel.org>
2022-04-21 15:38:10 +01:00
}
arm64: KVM: fault injection into a guest Implement the injection of a fault (undefined, data abort or prefetch abort) into a 64bit guest. Reviewed-by: Christopher Covington <cov@codeaurora.org> Reviewed-by: Catalin Marinas <catalin.marinas@arm.com> Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
2012-12-17 12:27:42 +00:00
/**
* kvm_inject_undefined - inject an undefined instruction into the guest
KVM: arm64: Fix inject_fault.c kernel-doc warnings Fix kernel-doc warnings. arch/arm64/kvm/inject_fault.c:210: warning: Function parameter or member 'vcpu' not described in 'kvm_inject_undefined' Signed-off-by: Tian Tao <tiantao6@hisilicon.com> Signed-off-by: Marc Zyngier <maz@kernel.org> Acked-by: Will Deacon <will@kernel.org> Link: https://lore.kernel.org/r/1600154512-44624-1-git-send-email-tiantao6@hisilicon.com
2020-09-15 15:21:52 +08:00
* @vcpu: The vCPU in which to inject the exception
arm64: KVM: fault injection into a guest Implement the injection of a fault (undefined, data abort or prefetch abort) into a 64bit guest. Reviewed-by: Christopher Covington <cov@codeaurora.org> Reviewed-by: Catalin Marinas <catalin.marinas@arm.com> Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
2012-12-17 12:27:42 +00:00
*
* It is assumed that this code is called from the VCPU thread and that the
* VCPU therefore is not currently executing guest code.
*/
void kvm_inject_undefined(struct kvm_vcpu *vcpu)
{
KVM: arm/arm64: Introduce vcpu_el1_is_32bit We have numerous checks around that checks if the HCR_EL2 has the RW bit set to figure out if we're running an AArch64 or AArch32 VM. In some cases, directly checking the RW bit (given its unintuitive name), is a bit confusing, and that's not going to improve as we move logic around for the following patches that optimize KVM on AArch64 hosts with VHE. Therefore, introduce a helper, vcpu_el1_is_32bit, and replace existing direct checks of HCR_EL2.RW with the helper. Reviewed-by: Julien Grall <julien.grall@arm.com> Reviewed-by: Julien Thierry <julien.thierry@arm.com> Acked-by: Marc Zyngier <marc.zyngier@arm.com> Reviewed-by: Andrew Jones <drjones@redhat.com> Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org> Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
2017-12-13 22:56:48 +01:00
if (vcpu_el1_is_32bit(vcpu))
KVM: arm64: Consolidate exception injection Move the AArch32 exception injection code back into the inject_fault.c file, removing the need for a few non-static functions now that AArch32 host support is a thing of the past. Signed-off-by: Marc Zyngier <maz@kernel.org>
2020-10-15 09:45:24 +01:00
inject_undef32(vcpu);
arm64: KVM: Fix host crash when injecting a fault into a 32bit guest When injecting a fault into a misbehaving 32bit guest, it seems rather idiotic to also inject a 64bit fault that is only going to corrupt the guest state. This leads to a situation where we perform an illegal exception return at EL2 causing the host to crash instead of killing the guest. Just fix the stupid bug that has been there from day 1. Cc: <stable@vger.kernel.org> Reported-by: Russell King <rmk+kernel@arm.linux.org.uk> Tested-by: Russell King <rmk+kernel@arm.linux.org.uk> Signed-off-by: Marc Zyngier <marc.zyngier@arm.com> Signed-off-by: Will Deacon <will.deacon@arm.com>
2015-08-27 16:10:01 +01:00
else
inject_undef64(vcpu);
arm64: KVM: fault injection into a guest Implement the injection of a fault (undefined, data abort or prefetch abort) into a 64bit guest. Reviewed-by: Christopher Covington <cov@codeaurora.org> Reviewed-by: Catalin Marinas <catalin.marinas@arm.com> Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
2012-12-17 12:27:42 +00:00
}
arm64: KVM: Add Virtual Abort injection helper Now that we're able to context switch the HCR_EL2.VA bit, let's introduce a helper that injects an Abort into a vcpu. Signed-off-by: Marc Zyngier <marc.zyngier@arm.com> Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
2016-09-06 14:02:01 +01:00
KVM: arm64: nv: Take "masked" aborts to EL2 when HCRX_EL2.TMEA is set HCRX_EL2.TMEA further modifies the external abort behavior where unmasked aborts are taken to EL1 and masked aborts are taken to EL2. It's rather weird when you consider that SEAs are, well, *synchronous* and therefore not actually maskable. However, for the purposes of exception routing, they're considered "masked" if the A flag is set. This gets a bit hairier when considering the fact that TMEA also enables vSErrors, i.e. KVM has delegated the HW vSError context to the guest hypervisor. We can keep the vSError context delegation as-is by taking advantage of a couple properties: - If SErrors are unmasked, the 'physical' SError can be taken in-context immediately. In other words, KVM can emulate the EL1 SError while preserving vEL2's ownership of the vSError context. - If SErrors are masked, the 'physical' SError is taken to EL2 immediately and needs the usual nested exception entry. Note that the new in-context handling has the benign effect where unmasked SError injections are emulated even for non-nested VMs. Reviewed-by: Marc Zyngier <maz@kernel.org> Link: https://lore.kernel.org/r/20250708172532.1699409-19-oliver.upton@linux.dev Signed-off-by: Oliver Upton <oliver.upton@linux.dev>
2025-07-08 10:25:23 -07:00
static bool serror_is_masked(struct kvm_vcpu *vcpu)
{
KVM: arm64: nv: Honor SError routing effects of SCTLR2_ELx.NMEA As the name might imply, when NMEA is set SErrors are non-maskable and can be taken regardless of PSTATE.A. As is the recurring theme with DoubleFault2, the effects on SError routing are entirely backwards to this. If at EL1, NMEA is *not* considered for SError routing when TMEA is set and the exception is taken to EL2 when PSTATE.A is set. Link: https://lore.kernel.org/r/20250708172532.1699409-20-oliver.upton@linux.dev Signed-off-by: Oliver Upton <oliver.upton@linux.dev>
2025-07-08 10:25:24 -07:00
return (*vcpu_cpsr(vcpu) & PSR_A_BIT) && !effective_sctlr2_nmea(vcpu);
KVM: arm64: nv: Take "masked" aborts to EL2 when HCRX_EL2.TMEA is set HCRX_EL2.TMEA further modifies the external abort behavior where unmasked aborts are taken to EL1 and masked aborts are taken to EL2. It's rather weird when you consider that SEAs are, well, *synchronous* and therefore not actually maskable. However, for the purposes of exception routing, they're considered "masked" if the A flag is set. This gets a bit hairier when considering the fact that TMEA also enables vSErrors, i.e. KVM has delegated the HW vSError context to the guest hypervisor. We can keep the vSError context delegation as-is by taking advantage of a couple properties: - If SErrors are unmasked, the 'physical' SError can be taken in-context immediately. In other words, KVM can emulate the EL1 SError while preserving vEL2's ownership of the vSError context. - If SErrors are masked, the 'physical' SError is taken to EL2 immediately and needs the usual nested exception entry. Note that the new in-context handling has the benign effect where unmasked SError injections are emulated even for non-nested VMs. Reviewed-by: Marc Zyngier <maz@kernel.org> Link: https://lore.kernel.org/r/20250708172532.1699409-19-oliver.upton@linux.dev Signed-off-by: Oliver Upton <oliver.upton@linux.dev>
2025-07-08 10:25:23 -07:00
}
KVM: arm64: nv: Honor SError exception routing / masking To date KVM has used HCR_EL2.VSE to track the state of a pending SError for the guest. With this bit set, hardware respects the EL1 exception routing / masking rules and injects the vSError when appropriate. This isn't correct for NV guests as hardware is oblivious to vEL2's intentions for SErrors. Better yet, with FEAT_NV2 the guest can change the routing behind our back as HCR_EL2 is redirected to memory. Cope with this mess by: - Using a flag (instead of HCR_EL2.VSE) to track the pending SError state when SErrors are unconditionally masked for the current context - Resampling the routing / masking of a pending SError on every guest entry/exit - Emulating exception entry when SError routing implies a translation regime change Reviewed-by: Marc Zyngier <maz@kernel.org> Link: https://lore.kernel.org/r/20250708172532.1699409-7-oliver.upton@linux.dev Signed-off-by: Oliver Upton <oliver.upton@linux.dev>
2025-07-08 10:25:11 -07:00
static bool kvm_serror_target_is_el2(struct kvm_vcpu *vcpu)
KVM: arm64: Set an impdef ESR for Virtual-SError using VSESR_EL2. Prior to v8.2's RAS Extensions, the HCR_EL2.VSE 'virtual SError' feature generated an SError with an implementation defined ESR_EL1.ISS, because we had no mechanism to specify the ESR value. On Juno this generates an all-zero ESR, the most significant bit 'ISV' is clear indicating the remainder of the ISS field is invalid. With the RAS Extensions we have a mechanism to specify this value, and the most significant bit has a new meaning: 'IDS - Implementation Defined Syndrome'. An all-zero SError ESR now means: 'RAS error: Uncategorized' instead of 'no valid ISS'. Add KVM support for the VSESR_EL2 register to specify an ESR value when HCR_EL2.VSE generates a virtual SError. Change kvm_inject_vabt() to specify an implementation-defined value. We only need to restore the VSESR_EL2 value when HCR_EL2.VSE is set, KVM save/restores this bit during __{,de}activate_traps() and hardware clears the bit once the guest has consumed the virtual-SError. Future patches may add an API (or KVM CAP) to pend a virtual SError with a specified ESR. Cc: Dongjiu Geng <gengdongjiu@huawei.com> Reviewed-by: Marc Zyngier <marc.zyngier@arm.com> Signed-off-by: James Morse <james.morse@arm.com> Reviewed-by: Christoffer Dall <christoffer.dall@linaro.org> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
2018-01-15 19:39:01 +00:00
{
KVM: arm64: nv: Take "masked" aborts to EL2 when HCRX_EL2.TMEA is set HCRX_EL2.TMEA further modifies the external abort behavior where unmasked aborts are taken to EL1 and masked aborts are taken to EL2. It's rather weird when you consider that SEAs are, well, *synchronous* and therefore not actually maskable. However, for the purposes of exception routing, they're considered "masked" if the A flag is set. This gets a bit hairier when considering the fact that TMEA also enables vSErrors, i.e. KVM has delegated the HW vSError context to the guest hypervisor. We can keep the vSError context delegation as-is by taking advantage of a couple properties: - If SErrors are unmasked, the 'physical' SError can be taken in-context immediately. In other words, KVM can emulate the EL1 SError while preserving vEL2's ownership of the vSError context. - If SErrors are masked, the 'physical' SError is taken to EL2 immediately and needs the usual nested exception entry. Note that the new in-context handling has the benign effect where unmasked SError injections are emulated even for non-nested VMs. Reviewed-by: Marc Zyngier <maz@kernel.org> Link: https://lore.kernel.org/r/20250708172532.1699409-19-oliver.upton@linux.dev Signed-off-by: Oliver Upton <oliver.upton@linux.dev>
2025-07-08 10:25:23 -07:00
if (is_hyp_ctxt(vcpu) || vcpu_el2_amo_is_set(vcpu))
return true;
if (!(__vcpu_sys_reg(vcpu, HCRX_EL2) & HCRX_EL2_TMEA))
return false;
KVM: arm64: nv: Honor SError routing effects of SCTLR2_ELx.NMEA As the name might imply, when NMEA is set SErrors are non-maskable and can be taken regardless of PSTATE.A. As is the recurring theme with DoubleFault2, the effects on SError routing are entirely backwards to this. If at EL1, NMEA is *not* considered for SError routing when TMEA is set and the exception is taken to EL2 when PSTATE.A is set. Link: https://lore.kernel.org/r/20250708172532.1699409-20-oliver.upton@linux.dev Signed-off-by: Oliver Upton <oliver.upton@linux.dev>
2025-07-08 10:25:24 -07:00
/*
* In another example where FEAT_DoubleFault2 is entirely backwards,
* "masked" as it relates to the routing effects of HCRX_EL2.TMEA
* doesn't consider SCTLR2_EL1.NMEA. That is to say, even if EL1 asked
* for non-maskable SErrors, the EL2 bit takes priority if A is set.
*/
if (vcpu_mode_priv(vcpu))
return *vcpu_cpsr(vcpu) & PSR_A_BIT;
/*
* Otherwise SErrors are considered unmasked when taken from EL0 and
* NMEA is set.
*/
KVM: arm64: nv: Take "masked" aborts to EL2 when HCRX_EL2.TMEA is set HCRX_EL2.TMEA further modifies the external abort behavior where unmasked aborts are taken to EL1 and masked aborts are taken to EL2. It's rather weird when you consider that SEAs are, well, *synchronous* and therefore not actually maskable. However, for the purposes of exception routing, they're considered "masked" if the A flag is set. This gets a bit hairier when considering the fact that TMEA also enables vSErrors, i.e. KVM has delegated the HW vSError context to the guest hypervisor. We can keep the vSError context delegation as-is by taking advantage of a couple properties: - If SErrors are unmasked, the 'physical' SError can be taken in-context immediately. In other words, KVM can emulate the EL1 SError while preserving vEL2's ownership of the vSError context. - If SErrors are masked, the 'physical' SError is taken to EL2 immediately and needs the usual nested exception entry. Note that the new in-context handling has the benign effect where unmasked SError injections are emulated even for non-nested VMs. Reviewed-by: Marc Zyngier <maz@kernel.org> Link: https://lore.kernel.org/r/20250708172532.1699409-19-oliver.upton@linux.dev Signed-off-by: Oliver Upton <oliver.upton@linux.dev>
2025-07-08 10:25:23 -07:00
return serror_is_masked(vcpu);
KVM: arm64: Set an impdef ESR for Virtual-SError using VSESR_EL2. Prior to v8.2's RAS Extensions, the HCR_EL2.VSE 'virtual SError' feature generated an SError with an implementation defined ESR_EL1.ISS, because we had no mechanism to specify the ESR value. On Juno this generates an all-zero ESR, the most significant bit 'ISV' is clear indicating the remainder of the ISS field is invalid. With the RAS Extensions we have a mechanism to specify this value, and the most significant bit has a new meaning: 'IDS - Implementation Defined Syndrome'. An all-zero SError ESR now means: 'RAS error: Uncategorized' instead of 'no valid ISS'. Add KVM support for the VSESR_EL2 register to specify an ESR value when HCR_EL2.VSE generates a virtual SError. Change kvm_inject_vabt() to specify an implementation-defined value. We only need to restore the VSESR_EL2 value when HCR_EL2.VSE is set, KVM save/restores this bit during __{,de}activate_traps() and hardware clears the bit once the guest has consumed the virtual-SError. Future patches may add an API (or KVM CAP) to pend a virtual SError with a specified ESR. Cc: Dongjiu Geng <gengdongjiu@huawei.com> Reviewed-by: Marc Zyngier <marc.zyngier@arm.com> Signed-off-by: James Morse <james.morse@arm.com> Reviewed-by: Christoffer Dall <christoffer.dall@linaro.org> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
2018-01-15 19:39:01 +00:00
}
KVM: arm64: nv: Honor SError exception routing / masking To date KVM has used HCR_EL2.VSE to track the state of a pending SError for the guest. With this bit set, hardware respects the EL1 exception routing / masking rules and injects the vSError when appropriate. This isn't correct for NV guests as hardware is oblivious to vEL2's intentions for SErrors. Better yet, with FEAT_NV2 the guest can change the routing behind our back as HCR_EL2 is redirected to memory. Cope with this mess by: - Using a flag (instead of HCR_EL2.VSE) to track the pending SError state when SErrors are unconditionally masked for the current context - Resampling the routing / masking of a pending SError on every guest entry/exit - Emulating exception entry when SError routing implies a translation regime change Reviewed-by: Marc Zyngier <maz@kernel.org> Link: https://lore.kernel.org/r/20250708172532.1699409-7-oliver.upton@linux.dev Signed-off-by: Oliver Upton <oliver.upton@linux.dev>
2025-07-08 10:25:11 -07:00
static bool kvm_serror_undeliverable_at_el2(struct kvm_vcpu *vcpu)
arm64: KVM: Add Virtual Abort injection helper Now that we're able to context switch the HCR_EL2.VA bit, let's introduce a helper that injects an Abort into a vcpu. Signed-off-by: Marc Zyngier <marc.zyngier@arm.com> Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
2016-09-06 14:02:01 +01:00
{
KVM: arm64: nv: Honor SError exception routing / masking To date KVM has used HCR_EL2.VSE to track the state of a pending SError for the guest. With this bit set, hardware respects the EL1 exception routing / masking rules and injects the vSError when appropriate. This isn't correct for NV guests as hardware is oblivious to vEL2's intentions for SErrors. Better yet, with FEAT_NV2 the guest can change the routing behind our back as HCR_EL2 is redirected to memory. Cope with this mess by: - Using a flag (instead of HCR_EL2.VSE) to track the pending SError state when SErrors are unconditionally masked for the current context - Resampling the routing / masking of a pending SError on every guest entry/exit - Emulating exception entry when SError routing implies a translation regime change Reviewed-by: Marc Zyngier <maz@kernel.org> Link: https://lore.kernel.org/r/20250708172532.1699409-7-oliver.upton@linux.dev Signed-off-by: Oliver Upton <oliver.upton@linux.dev>
2025-07-08 10:25:11 -07:00
return !(vcpu_el2_tge_is_set(vcpu) || vcpu_el2_amo_is_set(vcpu));
}
int kvm_inject_serror_esr(struct kvm_vcpu *vcpu, u64 esr)
{
lockdep_assert_held(&vcpu->mutex);
if (is_nested_ctxt(vcpu) && kvm_serror_target_is_el2(vcpu))
return kvm_inject_nested_serror(vcpu, esr);
if (vcpu_is_el2(vcpu) && kvm_serror_undeliverable_at_el2(vcpu)) {
vcpu_set_vsesr(vcpu, esr);
vcpu_set_flag(vcpu, NESTED_SERROR_PENDING);
return 1;
}
KVM: arm64: nv: Take "masked" aborts to EL2 when HCRX_EL2.TMEA is set HCRX_EL2.TMEA further modifies the external abort behavior where unmasked aborts are taken to EL1 and masked aborts are taken to EL2. It's rather weird when you consider that SEAs are, well, *synchronous* and therefore not actually maskable. However, for the purposes of exception routing, they're considered "masked" if the A flag is set. This gets a bit hairier when considering the fact that TMEA also enables vSErrors, i.e. KVM has delegated the HW vSError context to the guest hypervisor. We can keep the vSError context delegation as-is by taking advantage of a couple properties: - If SErrors are unmasked, the 'physical' SError can be taken in-context immediately. In other words, KVM can emulate the EL1 SError while preserving vEL2's ownership of the vSError context. - If SErrors are masked, the 'physical' SError is taken to EL2 immediately and needs the usual nested exception entry. Note that the new in-context handling has the benign effect where unmasked SError injections are emulated even for non-nested VMs. Reviewed-by: Marc Zyngier <maz@kernel.org> Link: https://lore.kernel.org/r/20250708172532.1699409-19-oliver.upton@linux.dev Signed-off-by: Oliver Upton <oliver.upton@linux.dev>
2025-07-08 10:25:23 -07:00
/*
* Emulate the exception entry if SErrors are unmasked. This is useful if
* the vCPU is in a nested context w/ vSErrors enabled then we've already
* delegated he hardware vSError context (i.e. HCR_EL2.VSE, VSESR_EL2,
* VDISR_EL2) to the guest hypervisor.
KVM: arm64: Populate ESR_ELx.EC for emulated SError injection The hardware vSError injection mechanism populates ESR_ELx.EC as part of ESR propagation and the contents of VSESR_EL2 populate the ISS field. Of course, this means our emulated injection needs to set up the EC correctly for an SError too. Fixes: ce66109cec86 ("KVM: arm64: nv: Take "masked" aborts to EL2 when HCRX_EL2.TMEA is set") Link: https://lore.kernel.org/r/20250708230632.1954240-2-oliver.upton@linux.dev Signed-off-by: Oliver Upton <oliver.upton@linux.dev>
2025-07-08 16:06:31 -07:00
*
* As we're emulating the SError injection we need to explicitly populate
* ESR_ELx.EC because hardware will not do it on our behalf.
KVM: arm64: nv: Take "masked" aborts to EL2 when HCRX_EL2.TMEA is set HCRX_EL2.TMEA further modifies the external abort behavior where unmasked aborts are taken to EL1 and masked aborts are taken to EL2. It's rather weird when you consider that SEAs are, well, *synchronous* and therefore not actually maskable. However, for the purposes of exception routing, they're considered "masked" if the A flag is set. This gets a bit hairier when considering the fact that TMEA also enables vSErrors, i.e. KVM has delegated the HW vSError context to the guest hypervisor. We can keep the vSError context delegation as-is by taking advantage of a couple properties: - If SErrors are unmasked, the 'physical' SError can be taken in-context immediately. In other words, KVM can emulate the EL1 SError while preserving vEL2's ownership of the vSError context. - If SErrors are masked, the 'physical' SError is taken to EL2 immediately and needs the usual nested exception entry. Note that the new in-context handling has the benign effect where unmasked SError injections are emulated even for non-nested VMs. Reviewed-by: Marc Zyngier <maz@kernel.org> Link: https://lore.kernel.org/r/20250708172532.1699409-19-oliver.upton@linux.dev Signed-off-by: Oliver Upton <oliver.upton@linux.dev>
2025-07-08 10:25:23 -07:00
*/
if (!serror_is_masked(vcpu)) {
pend_serror_exception(vcpu);
KVM: arm64: Populate ESR_ELx.EC for emulated SError injection The hardware vSError injection mechanism populates ESR_ELx.EC as part of ESR propagation and the contents of VSESR_EL2 populate the ISS field. Of course, this means our emulated injection needs to set up the EC correctly for an SError too. Fixes: ce66109cec86 ("KVM: arm64: nv: Take "masked" aborts to EL2 when HCRX_EL2.TMEA is set") Link: https://lore.kernel.org/r/20250708230632.1954240-2-oliver.upton@linux.dev Signed-off-by: Oliver Upton <oliver.upton@linux.dev>
2025-07-08 16:06:31 -07:00
esr |= FIELD_PREP(ESR_ELx_EC_MASK, ESR_ELx_EC_SERROR);
KVM: arm64: nv: Take "masked" aborts to EL2 when HCRX_EL2.TMEA is set HCRX_EL2.TMEA further modifies the external abort behavior where unmasked aborts are taken to EL1 and masked aborts are taken to EL2. It's rather weird when you consider that SEAs are, well, *synchronous* and therefore not actually maskable. However, for the purposes of exception routing, they're considered "masked" if the A flag is set. This gets a bit hairier when considering the fact that TMEA also enables vSErrors, i.e. KVM has delegated the HW vSError context to the guest hypervisor. We can keep the vSError context delegation as-is by taking advantage of a couple properties: - If SErrors are unmasked, the 'physical' SError can be taken in-context immediately. In other words, KVM can emulate the EL1 SError while preserving vEL2's ownership of the vSError context. - If SErrors are masked, the 'physical' SError is taken to EL2 immediately and needs the usual nested exception entry. Note that the new in-context handling has the benign effect where unmasked SError injections are emulated even for non-nested VMs. Reviewed-by: Marc Zyngier <maz@kernel.org> Link: https://lore.kernel.org/r/20250708172532.1699409-19-oliver.upton@linux.dev Signed-off-by: Oliver Upton <oliver.upton@linux.dev>
2025-07-08 10:25:23 -07:00
vcpu_write_sys_reg(vcpu, esr, exception_esr_elx(vcpu));
return 1;
}
KVM: arm64: nv: Honor SError exception routing / masking To date KVM has used HCR_EL2.VSE to track the state of a pending SError for the guest. With this bit set, hardware respects the EL1 exception routing / masking rules and injects the vSError when appropriate. This isn't correct for NV guests as hardware is oblivious to vEL2's intentions for SErrors. Better yet, with FEAT_NV2 the guest can change the routing behind our back as HCR_EL2 is redirected to memory. Cope with this mess by: - Using a flag (instead of HCR_EL2.VSE) to track the pending SError state when SErrors are unconditionally masked for the current context - Resampling the routing / masking of a pending SError on every guest entry/exit - Emulating exception entry when SError routing implies a translation regime change Reviewed-by: Marc Zyngier <maz@kernel.org> Link: https://lore.kernel.org/r/20250708172532.1699409-7-oliver.upton@linux.dev Signed-off-by: Oliver Upton <oliver.upton@linux.dev>
2025-07-08 10:25:11 -07:00
vcpu_set_vsesr(vcpu, esr & ESR_ELx_ISS_MASK);
*vcpu_hcr(vcpu) |= HCR_VSE;
return 1;
arm64: KVM: Add Virtual Abort injection helper Now that we're able to context switch the HCR_EL2.VA bit, let's introduce a helper that injects an Abort into a vcpu. Signed-off-by: Marc Zyngier <marc.zyngier@arm.com> Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
2016-09-06 14:02:01 +01:00
}
Reference in a new issue Copy permalink