2020-09-23 11:31:11 -07:00
|
|
|
// SPDX-License-Identifier: GPL-2.0-only
|
KVM: x86: Unify pr_fmt to use module name for all KVM modules
Define pr_fmt using KBUILD_MODNAME for all KVM x86 code so that printks
use consistent formatting across common x86, Intel, and AMD code. In
addition to providing consistent print formatting, using KBUILD_MODNAME,
e.g. kvm_amd and kvm_intel, allows referencing SVM and VMX (and SEV and
SGX and ...) as technologies without generating weird messages, and
without causing naming conflicts with other kernel code, e.g. "SEV: ",
"tdx: ", "sgx: " etc.. are all used by the kernel for non-KVM subsystems.
Opportunistically move away from printk() for prints that need to be
modified anyways, e.g. to drop a manual "kvm: " prefix.
Opportunistically convert a few SGX WARNs that are similarly modified to
WARN_ONCE; in the very unlikely event that the WARNs fire, odds are good
that they would fire repeatedly and spam the kernel log without providing
unique information in each print.
Note, defining pr_fmt yields undesirable results for code that uses KVM's
printk wrappers, e.g. vcpu_unimpl(). But, that's a pre-existing problem
as SVM/kvm_amd already defines a pr_fmt, and thankfully use of KVM's
wrappers is relatively limited in KVM x86 code.
Signed-off-by: Sean Christopherson <seanjc@google.com>
Reviewed-by: Paul Durrant <paul@xen.org>
Message-Id: <20221130230934.1014142-35-seanjc@google.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2022-11-30 23:09:18 +00:00
|
|
|
#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
|
|
|
|
|
|
2020-09-23 11:31:11 -07:00
|
|
|
#include <linux/kvm_host.h>
|
2025-06-11 15:45:06 -07:00
|
|
|
#include <linux/kvm_irqfd.h>
|
2020-09-23 11:31:11 -07:00
|
|
|
|
|
|
|
|
#include <asm/irq_remapping.h>
|
|
|
|
|
#include <asm/cpu.h>
|
|
|
|
|
|
|
|
|
|
#include "lapic.h"
|
2021-11-22 19:43:11 -05:00
|
|
|
#include "irq.h"
|
2020-09-23 11:31:11 -07:00
|
|
|
#include "posted_intr.h"
|
|
|
|
|
#include "trace.h"
|
|
|
|
|
#include "vmx.h"
|
2025-02-22 09:47:43 +08:00
|
|
|
#include "tdx.h"
|
2020-09-23 11:31:11 -07:00
|
|
|
|
|
|
|
|
/*
|
2021-10-08 19:12:21 -07:00
|
|
|
* Maintain a per-CPU list of vCPUs that need to be awakened by wakeup_handler()
|
|
|
|
|
* when a WAKEUP_VECTOR interrupted is posted. vCPUs are added to the list when
|
|
|
|
|
* the vCPU is scheduled out and is blocking (e.g. in HLT) with IRQs enabled.
|
|
|
|
|
* The vCPUs posted interrupt descriptor is updated at the same time to set its
|
|
|
|
|
* notification vector to WAKEUP_VECTOR, so that posted interrupt from devices
|
|
|
|
|
* wake the target vCPUs. vCPUs are removed from the list and the notification
|
|
|
|
|
* vector is reset when the vCPU is scheduled in.
|
2020-09-23 11:31:11 -07:00
|
|
|
*/
|
2021-12-08 01:52:16 +00:00
|
|
|
static DEFINE_PER_CPU(struct list_head, wakeup_vcpus_on_cpu);
|
2021-10-08 19:12:21 -07:00
|
|
|
/*
|
|
|
|
|
* Protect the per-CPU list with a per-CPU spinlock to handle task migration.
|
|
|
|
|
* When a blocking vCPU is awakened _and_ migrated to a different pCPU, the
|
|
|
|
|
* ->sched_in() path will need to take the vCPU off the list of the _previous_
|
|
|
|
|
* CPU. IRQs must be disabled when taking this lock, otherwise deadlock will
|
|
|
|
|
* occur if a wakeup IRQ arrives and attempts to acquire the lock.
|
|
|
|
|
*/
|
2021-12-08 01:52:16 +00:00
|
|
|
static DEFINE_PER_CPU(raw_spinlock_t, wakeup_vcpus_on_cpu_lock);
|
2020-09-23 11:31:11 -07:00
|
|
|
|
KVM: VMX: Use separate subclasses for PI wakeup lock to squash false positive
Use a separate subclass when acquiring KVM's per-CPU posted interrupts
wakeup lock in the scheduled out path, i.e. when adding a vCPU on the list
of vCPUs to wake, to workaround a false positive deadlock.
Chain exists of:
&p->pi_lock --> &rq->__lock --> &per_cpu(wakeup_vcpus_on_cpu_lock, cpu)
Possible unsafe locking scenario:
CPU0 CPU1
---- ----
lock(&per_cpu(wakeup_vcpus_on_cpu_lock, cpu));
lock(&rq->__lock);
lock(&per_cpu(wakeup_vcpus_on_cpu_lock, cpu));
lock(&p->pi_lock);
*** DEADLOCK ***
In the wakeup handler, the callchain is *always*:
sysvec_kvm_posted_intr_wakeup_ipi()
|
--> pi_wakeup_handler()
|
--> kvm_vcpu_wake_up()
|
--> try_to_wake_up(),
and the lock order is:
&per_cpu(wakeup_vcpus_on_cpu_lock, cpu) --> &p->pi_lock.
For the schedule out path, the callchain is always (for all intents and
purposes; if the kernel is preemptible, kvm_sched_out() can be called from
something other than schedule(), but the beginning of the callchain will
be the same point in vcpu_block()):
vcpu_block()
|
--> schedule()
|
--> kvm_sched_out()
|
--> vmx_vcpu_put()
|
--> vmx_vcpu_pi_put()
|
--> pi_enable_wakeup_handler()
and the lock order is:
&rq->__lock --> &per_cpu(wakeup_vcpus_on_cpu_lock, cpu)
I.e. lockdep sees AB+BC ordering for schedule out, and CA ordering for
wakeup, and complains about the A=>C versus C=>A inversion. In practice,
deadlock can't occur between schedule out and the wakeup handler as they
are mutually exclusive. The entirely of the schedule out code that runs
with the problematic scheduler locks held, does so with IRQs disabled,
i.e. can't run concurrently with the wakeup handler.
Use a subclass instead disabling lockdep entirely, and tell lockdep that
both subclasses are being acquired when loading a vCPU, as the sched_out
and sched_in paths are NOT mutually exclusive, e.g.
CPU 0 CPU 1
--------------- ---------------
vCPU0 sched_out
vCPU1 sched_in
vCPU1 sched_out vCPU 0 sched_in
where vCPU0's sched_in may race with vCPU1's sched_out, on CPU 0's wakeup
list+lock.
Signed-off-by: Yan Zhao <yan.y.zhao@intel.com>
Signed-off-by: Sean Christopherson <seanjc@google.com>
Reviewed-by: Maxim Levitsky <mlevitsk@redhat.com>
Message-ID: <20250401154727.835231-3-seanjc@google.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2025-04-01 08:47:27 -07:00
|
|
|
#define PI_LOCK_SCHED_OUT SINGLE_DEPTH_NESTING
|
|
|
|
|
|
2025-03-18 00:35:07 -06:00
|
|
|
static struct pi_desc *vcpu_to_pi_desc(struct kvm_vcpu *vcpu)
|
2020-09-23 11:31:11 -07:00
|
|
|
{
|
2025-03-14 14:06:48 -04:00
|
|
|
return &(to_vt(vcpu)->pi_desc);
|
2020-09-23 11:31:11 -07:00
|
|
|
}
|
|
|
|
|
|
2022-06-24 11:45:45 -04:00
|
|
|
static int pi_try_set_control(struct pi_desc *pi_desc, u64 *pold, u64 new)
|
2021-12-08 01:52:13 +00:00
|
|
|
{
|
|
|
|
|
/*
|
|
|
|
|
* PID.ON can be set at any time by a different vCPU or by hardware,
|
|
|
|
|
* e.g. a device. PID.control must be written atomically, and the
|
|
|
|
|
* update must be retried with a fresh snapshot an ON change causes
|
|
|
|
|
* the cmpxchg to fail.
|
|
|
|
|
*/
|
2022-06-24 11:45:45 -04:00
|
|
|
if (!try_cmpxchg64(&pi_desc->control, pold, new))
|
2021-12-08 01:52:13 +00:00
|
|
|
return -EBUSY;
|
|
|
|
|
|
|
|
|
|
return 0;
|
|
|
|
|
}
|
|
|
|
|
|
2020-09-23 11:31:11 -07:00
|
|
|
void vmx_vcpu_pi_load(struct kvm_vcpu *vcpu, int cpu)
|
|
|
|
|
{
|
|
|
|
|
struct pi_desc *pi_desc = vcpu_to_pi_desc(vcpu);
|
2025-03-14 14:06:48 -04:00
|
|
|
struct vcpu_vt *vt = to_vt(vcpu);
|
2020-09-23 11:31:11 -07:00
|
|
|
struct pi_desc old, new;
|
KVM: VMX: Handle PI descriptor updates during vcpu_put/load
Move the posted interrupt pre/post_block logic into vcpu_put/load
respectively, using the kvm_vcpu_is_blocking() to determining whether or
not the wakeup handler needs to be set (and unset). This avoids updating
the PI descriptor if halt-polling is successful, reduces the number of
touchpoints for updating the descriptor, and eliminates the confusing
behavior of intentionally leaving a "stale" PI.NDST when a blocking vCPU
is scheduled back in after preemption.
The downside is that KVM will do the PID update twice if the vCPU is
preempted after prepare_to_rcuwait() but before schedule(), but that's a
rare case (and non-existent on !PREEMPT kernels).
The notable wart is the need to send a self-IPI on the wakeup vector if
an outstanding notification is pending after configuring the wakeup
vector. Ideally, KVM would just do a kvm_vcpu_wake_up() in this case,
but the scheduler doesn't support waking a task from its preemption
notifier callback, i.e. while the task is right in the middle of
being scheduled out.
Note, setting the wakeup vector before halt-polling is not necessary:
once the pending IRQ will be recorded in the PIR, kvm_vcpu_has_events()
will detect this (via kvm_cpu_get_interrupt(), kvm_apic_get_interrupt(),
apic_has_interrupt_for_ppr() and finally vmx_sync_pir_to_irr()) and
terminate the polling.
Signed-off-by: Sean Christopherson <seanjc@google.com>
Reviewed-by: Maxim Levitsky <mlevitsk@redhat.com>
Message-Id: <20211208015236.1616697-5-seanjc@google.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2021-12-08 01:52:14 +00:00
|
|
|
unsigned long flags;
|
2020-09-23 11:31:11 -07:00
|
|
|
unsigned int dest;
|
|
|
|
|
|
|
|
|
|
/*
|
2021-10-08 19:12:13 -07:00
|
|
|
* To simplify hot-plug and dynamic toggling of APICv, keep PI.NDST and
|
|
|
|
|
* PI.SN up-to-date even if there is no assigned device or if APICv is
|
|
|
|
|
* deactivated due to a dynamic inhibit bit, e.g. for Hyper-V's SyncIC.
|
2020-09-23 11:31:11 -07:00
|
|
|
*/
|
2021-10-08 19:12:13 -07:00
|
|
|
if (!enable_apicv || !lapic_in_kernel(vcpu))
|
|
|
|
|
return;
|
|
|
|
|
|
KVM: VMX: Handle PI descriptor updates during vcpu_put/load
Move the posted interrupt pre/post_block logic into vcpu_put/load
respectively, using the kvm_vcpu_is_blocking() to determining whether or
not the wakeup handler needs to be set (and unset). This avoids updating
the PI descriptor if halt-polling is successful, reduces the number of
touchpoints for updating the descriptor, and eliminates the confusing
behavior of intentionally leaving a "stale" PI.NDST when a blocking vCPU
is scheduled back in after preemption.
The downside is that KVM will do the PID update twice if the vCPU is
preempted after prepare_to_rcuwait() but before schedule(), but that's a
rare case (and non-existent on !PREEMPT kernels).
The notable wart is the need to send a self-IPI on the wakeup vector if
an outstanding notification is pending after configuring the wakeup
vector. Ideally, KVM would just do a kvm_vcpu_wake_up() in this case,
but the scheduler doesn't support waking a task from its preemption
notifier callback, i.e. while the task is right in the middle of
being scheduled out.
Note, setting the wakeup vector before halt-polling is not necessary:
once the pending IRQ will be recorded in the PIR, kvm_vcpu_has_events()
will detect this (via kvm_cpu_get_interrupt(), kvm_apic_get_interrupt(),
apic_has_interrupt_for_ppr() and finally vmx_sync_pir_to_irr()) and
terminate the polling.
Signed-off-by: Sean Christopherson <seanjc@google.com>
Reviewed-by: Maxim Levitsky <mlevitsk@redhat.com>
Message-Id: <20211208015236.1616697-5-seanjc@google.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2021-12-08 01:52:14 +00:00
|
|
|
/*
|
|
|
|
|
* If the vCPU wasn't on the wakeup list and wasn't migrated, then the
|
|
|
|
|
* full update can be skipped as neither the vector nor the destination
|
KVM: VMX: Suppress PI notifications whenever the vCPU is put
Suppress posted interrupt notifications (set PID.SN=1) whenever the vCPU
is put, i.e. unloaded, not just when the vCPU is preempted, as KVM doesn't
do anything in response to a notification IRQ that arrives in the host,
nor does KVM rely on the Outstanding Notification (PID.ON) flag when the
vCPU is unloaded. And, the cost of scanning the PIR to manually set PID.ON
when loading the vCPU is quite small, especially relative to the cost of
loading (and unloading) a vCPU.
On the flip side, leaving SN clear means a notification for the vCPU will
result in a spurious IRQ for the pCPU, even if vCPU task is scheduled out,
running in userspace, etc. Even worse, if the pCPU is running a different
vCPU, the spurious IRQ could trigger posted interrupt processing for the
wrong vCPU, which is technically a violation of the architecture, as
setting bits in PIR aren't supposed to be propagated to the vIRR until a
notification IRQ is received.
The saving grace of the current behavior is that hardware sends
notification interrupts if and only if PID.ON=0, i.e. only the first
posted interrupt for a vCPU will trigger a spurious IRQ (for each window
where the vCPU is unloaded).
Ideally, KVM would suppress notifications before enabling IRQs in the
VM-Exit, but KVM relies on PID.ON as an indicator that there is a posted
interrupt pending in PIR, e.g. in vmx_sync_pir_to_irr(), and sadly there
is no way to ask hardware to set PID.ON, but not generate an interrupt.
That could be solved by using pi_has_pending_interrupt() instead of
checking only PID.ON, but it's not at all clear that would be a performance
win, as KVM would end up scanning the entire PIR whenever an interrupt
isn't pending.
And long term, the spurious IRQ window, i.e. where a vCPU is loaded with
IRQs enabled, can effectively be made smaller for hot paths by moving
performance critical VM-Exit handlers into the fastpath, i.e. by never
enabling IRQs for hot path VM-Exits.
Link: https://lore.kernel.org/r/20250611224604.313496-21-seanjc@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
2025-06-11 15:45:22 -07:00
|
|
|
* needs to be changed. Clear SN even if there is no assigned device,
|
|
|
|
|
* again for simplicity.
|
KVM: VMX: Handle PI descriptor updates during vcpu_put/load
Move the posted interrupt pre/post_block logic into vcpu_put/load
respectively, using the kvm_vcpu_is_blocking() to determining whether or
not the wakeup handler needs to be set (and unset). This avoids updating
the PI descriptor if halt-polling is successful, reduces the number of
touchpoints for updating the descriptor, and eliminates the confusing
behavior of intentionally leaving a "stale" PI.NDST when a blocking vCPU
is scheduled back in after preemption.
The downside is that KVM will do the PID update twice if the vCPU is
preempted after prepare_to_rcuwait() but before schedule(), but that's a
rare case (and non-existent on !PREEMPT kernels).
The notable wart is the need to send a self-IPI on the wakeup vector if
an outstanding notification is pending after configuring the wakeup
vector. Ideally, KVM would just do a kvm_vcpu_wake_up() in this case,
but the scheduler doesn't support waking a task from its preemption
notifier callback, i.e. while the task is right in the middle of
being scheduled out.
Note, setting the wakeup vector before halt-polling is not necessary:
once the pending IRQ will be recorded in the PIR, kvm_vcpu_has_events()
will detect this (via kvm_cpu_get_interrupt(), kvm_apic_get_interrupt(),
apic_has_interrupt_for_ppr() and finally vmx_sync_pir_to_irr()) and
terminate the polling.
Signed-off-by: Sean Christopherson <seanjc@google.com>
Reviewed-by: Maxim Levitsky <mlevitsk@redhat.com>
Message-Id: <20211208015236.1616697-5-seanjc@google.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2021-12-08 01:52:14 +00:00
|
|
|
*/
|
|
|
|
|
if (pi_desc->nv != POSTED_INTR_WAKEUP_VECTOR && vcpu->cpu == cpu) {
|
|
|
|
|
if (pi_test_and_clear_sn(pi_desc))
|
|
|
|
|
goto after_clear_sn;
|
2020-09-23 11:31:11 -07:00
|
|
|
return;
|
KVM: VMX: Handle PI descriptor updates during vcpu_put/load
Move the posted interrupt pre/post_block logic into vcpu_put/load
respectively, using the kvm_vcpu_is_blocking() to determining whether or
not the wakeup handler needs to be set (and unset). This avoids updating
the PI descriptor if halt-polling is successful, reduces the number of
touchpoints for updating the descriptor, and eliminates the confusing
behavior of intentionally leaving a "stale" PI.NDST when a blocking vCPU
is scheduled back in after preemption.
The downside is that KVM will do the PID update twice if the vCPU is
preempted after prepare_to_rcuwait() but before schedule(), but that's a
rare case (and non-existent on !PREEMPT kernels).
The notable wart is the need to send a self-IPI on the wakeup vector if
an outstanding notification is pending after configuring the wakeup
vector. Ideally, KVM would just do a kvm_vcpu_wake_up() in this case,
but the scheduler doesn't support waking a task from its preemption
notifier callback, i.e. while the task is right in the middle of
being scheduled out.
Note, setting the wakeup vector before halt-polling is not necessary:
once the pending IRQ will be recorded in the PIR, kvm_vcpu_has_events()
will detect this (via kvm_cpu_get_interrupt(), kvm_apic_get_interrupt(),
apic_has_interrupt_for_ppr() and finally vmx_sync_pir_to_irr()) and
terminate the polling.
Signed-off-by: Sean Christopherson <seanjc@google.com>
Reviewed-by: Maxim Levitsky <mlevitsk@redhat.com>
Message-Id: <20211208015236.1616697-5-seanjc@google.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2021-12-08 01:52:14 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
local_irq_save(flags);
|
2020-09-23 11:31:11 -07:00
|
|
|
|
|
|
|
|
/*
|
KVM: VMX: Handle PI descriptor updates during vcpu_put/load
Move the posted interrupt pre/post_block logic into vcpu_put/load
respectively, using the kvm_vcpu_is_blocking() to determining whether or
not the wakeup handler needs to be set (and unset). This avoids updating
the PI descriptor if halt-polling is successful, reduces the number of
touchpoints for updating the descriptor, and eliminates the confusing
behavior of intentionally leaving a "stale" PI.NDST when a blocking vCPU
is scheduled back in after preemption.
The downside is that KVM will do the PID update twice if the vCPU is
preempted after prepare_to_rcuwait() but before schedule(), but that's a
rare case (and non-existent on !PREEMPT kernels).
The notable wart is the need to send a self-IPI on the wakeup vector if
an outstanding notification is pending after configuring the wakeup
vector. Ideally, KVM would just do a kvm_vcpu_wake_up() in this case,
but the scheduler doesn't support waking a task from its preemption
notifier callback, i.e. while the task is right in the middle of
being scheduled out.
Note, setting the wakeup vector before halt-polling is not necessary:
once the pending IRQ will be recorded in the PIR, kvm_vcpu_has_events()
will detect this (via kvm_cpu_get_interrupt(), kvm_apic_get_interrupt(),
apic_has_interrupt_for_ppr() and finally vmx_sync_pir_to_irr()) and
terminate the polling.
Signed-off-by: Sean Christopherson <seanjc@google.com>
Reviewed-by: Maxim Levitsky <mlevitsk@redhat.com>
Message-Id: <20211208015236.1616697-5-seanjc@google.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2021-12-08 01:52:14 +00:00
|
|
|
* If the vCPU was waiting for wakeup, remove the vCPU from the wakeup
|
|
|
|
|
* list of the _previous_ pCPU, which will not be the same as the
|
|
|
|
|
* current pCPU if the task was migrated.
|
2020-09-23 11:31:11 -07:00
|
|
|
*/
|
KVM: VMX: Handle PI descriptor updates during vcpu_put/load
Move the posted interrupt pre/post_block logic into vcpu_put/load
respectively, using the kvm_vcpu_is_blocking() to determining whether or
not the wakeup handler needs to be set (and unset). This avoids updating
the PI descriptor if halt-polling is successful, reduces the number of
touchpoints for updating the descriptor, and eliminates the confusing
behavior of intentionally leaving a "stale" PI.NDST when a blocking vCPU
is scheduled back in after preemption.
The downside is that KVM will do the PID update twice if the vCPU is
preempted after prepare_to_rcuwait() but before schedule(), but that's a
rare case (and non-existent on !PREEMPT kernels).
The notable wart is the need to send a self-IPI on the wakeup vector if
an outstanding notification is pending after configuring the wakeup
vector. Ideally, KVM would just do a kvm_vcpu_wake_up() in this case,
but the scheduler doesn't support waking a task from its preemption
notifier callback, i.e. while the task is right in the middle of
being scheduled out.
Note, setting the wakeup vector before halt-polling is not necessary:
once the pending IRQ will be recorded in the PIR, kvm_vcpu_has_events()
will detect this (via kvm_cpu_get_interrupt(), kvm_apic_get_interrupt(),
apic_has_interrupt_for_ppr() and finally vmx_sync_pir_to_irr()) and
terminate the polling.
Signed-off-by: Sean Christopherson <seanjc@google.com>
Reviewed-by: Maxim Levitsky <mlevitsk@redhat.com>
Message-Id: <20211208015236.1616697-5-seanjc@google.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2021-12-08 01:52:14 +00:00
|
|
|
if (pi_desc->nv == POSTED_INTR_WAKEUP_VECTOR) {
|
KVM: VMX: Use separate subclasses for PI wakeup lock to squash false positive
Use a separate subclass when acquiring KVM's per-CPU posted interrupts
wakeup lock in the scheduled out path, i.e. when adding a vCPU on the list
of vCPUs to wake, to workaround a false positive deadlock.
Chain exists of:
&p->pi_lock --> &rq->__lock --> &per_cpu(wakeup_vcpus_on_cpu_lock, cpu)
Possible unsafe locking scenario:
CPU0 CPU1
---- ----
lock(&per_cpu(wakeup_vcpus_on_cpu_lock, cpu));
lock(&rq->__lock);
lock(&per_cpu(wakeup_vcpus_on_cpu_lock, cpu));
lock(&p->pi_lock);
*** DEADLOCK ***
In the wakeup handler, the callchain is *always*:
sysvec_kvm_posted_intr_wakeup_ipi()
|
--> pi_wakeup_handler()
|
--> kvm_vcpu_wake_up()
|
--> try_to_wake_up(),
and the lock order is:
&per_cpu(wakeup_vcpus_on_cpu_lock, cpu) --> &p->pi_lock.
For the schedule out path, the callchain is always (for all intents and
purposes; if the kernel is preemptible, kvm_sched_out() can be called from
something other than schedule(), but the beginning of the callchain will
be the same point in vcpu_block()):
vcpu_block()
|
--> schedule()
|
--> kvm_sched_out()
|
--> vmx_vcpu_put()
|
--> vmx_vcpu_pi_put()
|
--> pi_enable_wakeup_handler()
and the lock order is:
&rq->__lock --> &per_cpu(wakeup_vcpus_on_cpu_lock, cpu)
I.e. lockdep sees AB+BC ordering for schedule out, and CA ordering for
wakeup, and complains about the A=>C versus C=>A inversion. In practice,
deadlock can't occur between schedule out and the wakeup handler as they
are mutually exclusive. The entirely of the schedule out code that runs
with the problematic scheduler locks held, does so with IRQs disabled,
i.e. can't run concurrently with the wakeup handler.
Use a subclass instead disabling lockdep entirely, and tell lockdep that
both subclasses are being acquired when loading a vCPU, as the sched_out
and sched_in paths are NOT mutually exclusive, e.g.
CPU 0 CPU 1
--------------- ---------------
vCPU0 sched_out
vCPU1 sched_in
vCPU1 sched_out vCPU 0 sched_in
where vCPU0's sched_in may race with vCPU1's sched_out, on CPU 0's wakeup
list+lock.
Signed-off-by: Yan Zhao <yan.y.zhao@intel.com>
Signed-off-by: Sean Christopherson <seanjc@google.com>
Reviewed-by: Maxim Levitsky <mlevitsk@redhat.com>
Message-ID: <20250401154727.835231-3-seanjc@google.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2025-04-01 08:47:27 -07:00
|
|
|
raw_spinlock_t *spinlock = &per_cpu(wakeup_vcpus_on_cpu_lock, vcpu->cpu);
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* In addition to taking the wakeup lock for the regular/IRQ
|
|
|
|
|
* context, tell lockdep it is being taken for the "sched out"
|
|
|
|
|
* context as well. vCPU loads happens in task context, and
|
|
|
|
|
* this is taking the lock of the *previous* CPU, i.e. can race
|
|
|
|
|
* with both the scheduler and the wakeup handler.
|
|
|
|
|
*/
|
|
|
|
|
raw_spin_lock(spinlock);
|
|
|
|
|
spin_acquire(&spinlock->dep_map, PI_LOCK_SCHED_OUT, 0, _RET_IP_);
|
2025-03-14 14:06:48 -04:00
|
|
|
list_del(&vt->pi_wakeup_list);
|
KVM: VMX: Use separate subclasses for PI wakeup lock to squash false positive
Use a separate subclass when acquiring KVM's per-CPU posted interrupts
wakeup lock in the scheduled out path, i.e. when adding a vCPU on the list
of vCPUs to wake, to workaround a false positive deadlock.
Chain exists of:
&p->pi_lock --> &rq->__lock --> &per_cpu(wakeup_vcpus_on_cpu_lock, cpu)
Possible unsafe locking scenario:
CPU0 CPU1
---- ----
lock(&per_cpu(wakeup_vcpus_on_cpu_lock, cpu));
lock(&rq->__lock);
lock(&per_cpu(wakeup_vcpus_on_cpu_lock, cpu));
lock(&p->pi_lock);
*** DEADLOCK ***
In the wakeup handler, the callchain is *always*:
sysvec_kvm_posted_intr_wakeup_ipi()
|
--> pi_wakeup_handler()
|
--> kvm_vcpu_wake_up()
|
--> try_to_wake_up(),
and the lock order is:
&per_cpu(wakeup_vcpus_on_cpu_lock, cpu) --> &p->pi_lock.
For the schedule out path, the callchain is always (for all intents and
purposes; if the kernel is preemptible, kvm_sched_out() can be called from
something other than schedule(), but the beginning of the callchain will
be the same point in vcpu_block()):
vcpu_block()
|
--> schedule()
|
--> kvm_sched_out()
|
--> vmx_vcpu_put()
|
--> vmx_vcpu_pi_put()
|
--> pi_enable_wakeup_handler()
and the lock order is:
&rq->__lock --> &per_cpu(wakeup_vcpus_on_cpu_lock, cpu)
I.e. lockdep sees AB+BC ordering for schedule out, and CA ordering for
wakeup, and complains about the A=>C versus C=>A inversion. In practice,
deadlock can't occur between schedule out and the wakeup handler as they
are mutually exclusive. The entirely of the schedule out code that runs
with the problematic scheduler locks held, does so with IRQs disabled,
i.e. can't run concurrently with the wakeup handler.
Use a subclass instead disabling lockdep entirely, and tell lockdep that
both subclasses are being acquired when loading a vCPU, as the sched_out
and sched_in paths are NOT mutually exclusive, e.g.
CPU 0 CPU 1
--------------- ---------------
vCPU0 sched_out
vCPU1 sched_in
vCPU1 sched_out vCPU 0 sched_in
where vCPU0's sched_in may race with vCPU1's sched_out, on CPU 0's wakeup
list+lock.
Signed-off-by: Yan Zhao <yan.y.zhao@intel.com>
Signed-off-by: Sean Christopherson <seanjc@google.com>
Reviewed-by: Maxim Levitsky <mlevitsk@redhat.com>
Message-ID: <20250401154727.835231-3-seanjc@google.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2025-04-01 08:47:27 -07:00
|
|
|
spin_release(&spinlock->dep_map, _RET_IP_);
|
|
|
|
|
raw_spin_unlock(spinlock);
|
2020-09-23 11:31:11 -07:00
|
|
|
}
|
|
|
|
|
|
2021-10-08 19:12:20 -07:00
|
|
|
dest = cpu_physical_id(cpu);
|
|
|
|
|
if (!x2apic_mode)
|
|
|
|
|
dest = (dest << 8) & 0xFF00;
|
|
|
|
|
|
2022-06-24 11:45:45 -04:00
|
|
|
old.control = READ_ONCE(pi_desc->control);
|
2020-09-23 11:31:11 -07:00
|
|
|
do {
|
2022-06-24 11:45:45 -04:00
|
|
|
new.control = old.control;
|
2020-09-23 11:31:11 -07:00
|
|
|
|
KVM: VMX: Handle PI descriptor updates during vcpu_put/load
Move the posted interrupt pre/post_block logic into vcpu_put/load
respectively, using the kvm_vcpu_is_blocking() to determining whether or
not the wakeup handler needs to be set (and unset). This avoids updating
the PI descriptor if halt-polling is successful, reduces the number of
touchpoints for updating the descriptor, and eliminates the confusing
behavior of intentionally leaving a "stale" PI.NDST when a blocking vCPU
is scheduled back in after preemption.
The downside is that KVM will do the PID update twice if the vCPU is
preempted after prepare_to_rcuwait() but before schedule(), but that's a
rare case (and non-existent on !PREEMPT kernels).
The notable wart is the need to send a self-IPI on the wakeup vector if
an outstanding notification is pending after configuring the wakeup
vector. Ideally, KVM would just do a kvm_vcpu_wake_up() in this case,
but the scheduler doesn't support waking a task from its preemption
notifier callback, i.e. while the task is right in the middle of
being scheduled out.
Note, setting the wakeup vector before halt-polling is not necessary:
once the pending IRQ will be recorded in the PIR, kvm_vcpu_has_events()
will detect this (via kvm_cpu_get_interrupt(), kvm_apic_get_interrupt(),
apic_has_interrupt_for_ppr() and finally vmx_sync_pir_to_irr()) and
terminate the polling.
Signed-off-by: Sean Christopherson <seanjc@google.com>
Reviewed-by: Maxim Levitsky <mlevitsk@redhat.com>
Message-Id: <20211208015236.1616697-5-seanjc@google.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2021-12-08 01:52:14 +00:00
|
|
|
/*
|
|
|
|
|
* Clear SN (as above) and refresh the destination APIC ID to
|
|
|
|
|
* handle task migration (@cpu != vcpu->cpu).
|
|
|
|
|
*/
|
2021-10-08 19:12:20 -07:00
|
|
|
new.ndst = dest;
|
2024-04-23 10:41:05 -07:00
|
|
|
__pi_clear_sn(&new);
|
KVM: VMX: Handle PI descriptor updates during vcpu_put/load
Move the posted interrupt pre/post_block logic into vcpu_put/load
respectively, using the kvm_vcpu_is_blocking() to determining whether or
not the wakeup handler needs to be set (and unset). This avoids updating
the PI descriptor if halt-polling is successful, reduces the number of
touchpoints for updating the descriptor, and eliminates the confusing
behavior of intentionally leaving a "stale" PI.NDST when a blocking vCPU
is scheduled back in after preemption.
The downside is that KVM will do the PID update twice if the vCPU is
preempted after prepare_to_rcuwait() but before schedule(), but that's a
rare case (and non-existent on !PREEMPT kernels).
The notable wart is the need to send a self-IPI on the wakeup vector if
an outstanding notification is pending after configuring the wakeup
vector. Ideally, KVM would just do a kvm_vcpu_wake_up() in this case,
but the scheduler doesn't support waking a task from its preemption
notifier callback, i.e. while the task is right in the middle of
being scheduled out.
Note, setting the wakeup vector before halt-polling is not necessary:
once the pending IRQ will be recorded in the PIR, kvm_vcpu_has_events()
will detect this (via kvm_cpu_get_interrupt(), kvm_apic_get_interrupt(),
apic_has_interrupt_for_ppr() and finally vmx_sync_pir_to_irr()) and
terminate the polling.
Signed-off-by: Sean Christopherson <seanjc@google.com>
Reviewed-by: Maxim Levitsky <mlevitsk@redhat.com>
Message-Id: <20211208015236.1616697-5-seanjc@google.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2021-12-08 01:52:14 +00:00
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Restore the notification vector; in the blocking case, the
|
|
|
|
|
* descriptor was modified on "put" to use the wakeup vector.
|
|
|
|
|
*/
|
|
|
|
|
new.nv = POSTED_INTR_VECTOR;
|
2022-06-24 11:45:45 -04:00
|
|
|
} while (pi_try_set_control(pi_desc, &old.control, new.control));
|
2020-09-23 11:31:11 -07:00
|
|
|
|
KVM: VMX: Handle PI descriptor updates during vcpu_put/load
Move the posted interrupt pre/post_block logic into vcpu_put/load
respectively, using the kvm_vcpu_is_blocking() to determining whether or
not the wakeup handler needs to be set (and unset). This avoids updating
the PI descriptor if halt-polling is successful, reduces the number of
touchpoints for updating the descriptor, and eliminates the confusing
behavior of intentionally leaving a "stale" PI.NDST when a blocking vCPU
is scheduled back in after preemption.
The downside is that KVM will do the PID update twice if the vCPU is
preempted after prepare_to_rcuwait() but before schedule(), but that's a
rare case (and non-existent on !PREEMPT kernels).
The notable wart is the need to send a self-IPI on the wakeup vector if
an outstanding notification is pending after configuring the wakeup
vector. Ideally, KVM would just do a kvm_vcpu_wake_up() in this case,
but the scheduler doesn't support waking a task from its preemption
notifier callback, i.e. while the task is right in the middle of
being scheduled out.
Note, setting the wakeup vector before halt-polling is not necessary:
once the pending IRQ will be recorded in the PIR, kvm_vcpu_has_events()
will detect this (via kvm_cpu_get_interrupt(), kvm_apic_get_interrupt(),
apic_has_interrupt_for_ppr() and finally vmx_sync_pir_to_irr()) and
terminate the polling.
Signed-off-by: Sean Christopherson <seanjc@google.com>
Reviewed-by: Maxim Levitsky <mlevitsk@redhat.com>
Message-Id: <20211208015236.1616697-5-seanjc@google.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2021-12-08 01:52:14 +00:00
|
|
|
local_irq_restore(flags);
|
|
|
|
|
|
2020-09-23 11:31:11 -07:00
|
|
|
after_clear_sn:
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Clear SN before reading the bitmap. The VT-d firmware
|
|
|
|
|
* writes the bitmap and reads SN atomically (5.2.3 in the
|
|
|
|
|
* spec), so it doesn't really have a memory barrier that
|
|
|
|
|
* pairs with this, but we cannot do that and we need one.
|
|
|
|
|
*/
|
|
|
|
|
smp_mb__after_atomic();
|
|
|
|
|
|
|
|
|
|
if (!pi_is_pir_empty(pi_desc))
|
|
|
|
|
pi_set_on(pi_desc);
|
|
|
|
|
}
|
|
|
|
|
|
2021-11-22 19:43:11 -05:00
|
|
|
static bool vmx_can_use_vtd_pi(struct kvm *kvm)
|
|
|
|
|
{
|
2025-06-11 15:45:56 -07:00
|
|
|
/*
|
|
|
|
|
* Note, reading the number of possible bypass IRQs can race with a
|
|
|
|
|
* bypass IRQ being attached to the VM. vmx_pi_start_bypass() ensures
|
|
|
|
|
* blockng vCPUs will see an elevated count or get KVM_REQ_UNBLOCK.
|
|
|
|
|
*/
|
2025-04-01 09:18:02 -07:00
|
|
|
return irqchip_in_kernel(kvm) && kvm_arch_has_irq_bypass() &&
|
2025-06-11 15:45:56 -07:00
|
|
|
READ_ONCE(kvm->arch.nr_possible_bypass_irqs);
|
2021-11-22 19:43:11 -05:00
|
|
|
}
|
|
|
|
|
|
2020-09-23 11:31:11 -07:00
|
|
|
/*
|
KVM: VMX: Handle PI descriptor updates during vcpu_put/load
Move the posted interrupt pre/post_block logic into vcpu_put/load
respectively, using the kvm_vcpu_is_blocking() to determining whether or
not the wakeup handler needs to be set (and unset). This avoids updating
the PI descriptor if halt-polling is successful, reduces the number of
touchpoints for updating the descriptor, and eliminates the confusing
behavior of intentionally leaving a "stale" PI.NDST when a blocking vCPU
is scheduled back in after preemption.
The downside is that KVM will do the PID update twice if the vCPU is
preempted after prepare_to_rcuwait() but before schedule(), but that's a
rare case (and non-existent on !PREEMPT kernels).
The notable wart is the need to send a self-IPI on the wakeup vector if
an outstanding notification is pending after configuring the wakeup
vector. Ideally, KVM would just do a kvm_vcpu_wake_up() in this case,
but the scheduler doesn't support waking a task from its preemption
notifier callback, i.e. while the task is right in the middle of
being scheduled out.
Note, setting the wakeup vector before halt-polling is not necessary:
once the pending IRQ will be recorded in the PIR, kvm_vcpu_has_events()
will detect this (via kvm_cpu_get_interrupt(), kvm_apic_get_interrupt(),
apic_has_interrupt_for_ppr() and finally vmx_sync_pir_to_irr()) and
terminate the polling.
Signed-off-by: Sean Christopherson <seanjc@google.com>
Reviewed-by: Maxim Levitsky <mlevitsk@redhat.com>
Message-Id: <20211208015236.1616697-5-seanjc@google.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2021-12-08 01:52:14 +00:00
|
|
|
* Put the vCPU on this pCPU's list of vCPUs that needs to be awakened and set
|
|
|
|
|
* WAKEUP as the notification vector in the PI descriptor.
|
2020-09-23 11:31:11 -07:00
|
|
|
*/
|
KVM: VMX: Handle PI descriptor updates during vcpu_put/load
Move the posted interrupt pre/post_block logic into vcpu_put/load
respectively, using the kvm_vcpu_is_blocking() to determining whether or
not the wakeup handler needs to be set (and unset). This avoids updating
the PI descriptor if halt-polling is successful, reduces the number of
touchpoints for updating the descriptor, and eliminates the confusing
behavior of intentionally leaving a "stale" PI.NDST when a blocking vCPU
is scheduled back in after preemption.
The downside is that KVM will do the PID update twice if the vCPU is
preempted after prepare_to_rcuwait() but before schedule(), but that's a
rare case (and non-existent on !PREEMPT kernels).
The notable wart is the need to send a self-IPI on the wakeup vector if
an outstanding notification is pending after configuring the wakeup
vector. Ideally, KVM would just do a kvm_vcpu_wake_up() in this case,
but the scheduler doesn't support waking a task from its preemption
notifier callback, i.e. while the task is right in the middle of
being scheduled out.
Note, setting the wakeup vector before halt-polling is not necessary:
once the pending IRQ will be recorded in the PIR, kvm_vcpu_has_events()
will detect this (via kvm_cpu_get_interrupt(), kvm_apic_get_interrupt(),
apic_has_interrupt_for_ppr() and finally vmx_sync_pir_to_irr()) and
terminate the polling.
Signed-off-by: Sean Christopherson <seanjc@google.com>
Reviewed-by: Maxim Levitsky <mlevitsk@redhat.com>
Message-Id: <20211208015236.1616697-5-seanjc@google.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2021-12-08 01:52:14 +00:00
|
|
|
static void pi_enable_wakeup_handler(struct kvm_vcpu *vcpu)
|
2020-09-23 11:31:11 -07:00
|
|
|
{
|
|
|
|
|
struct pi_desc *pi_desc = vcpu_to_pi_desc(vcpu);
|
2025-03-14 14:06:48 -04:00
|
|
|
struct vcpu_vt *vt = to_vt(vcpu);
|
KVM: VMX: Handle PI descriptor updates during vcpu_put/load
Move the posted interrupt pre/post_block logic into vcpu_put/load
respectively, using the kvm_vcpu_is_blocking() to determining whether or
not the wakeup handler needs to be set (and unset). This avoids updating
the PI descriptor if halt-polling is successful, reduces the number of
touchpoints for updating the descriptor, and eliminates the confusing
behavior of intentionally leaving a "stale" PI.NDST when a blocking vCPU
is scheduled back in after preemption.
The downside is that KVM will do the PID update twice if the vCPU is
preempted after prepare_to_rcuwait() but before schedule(), but that's a
rare case (and non-existent on !PREEMPT kernels).
The notable wart is the need to send a self-IPI on the wakeup vector if
an outstanding notification is pending after configuring the wakeup
vector. Ideally, KVM would just do a kvm_vcpu_wake_up() in this case,
but the scheduler doesn't support waking a task from its preemption
notifier callback, i.e. while the task is right in the middle of
being scheduled out.
Note, setting the wakeup vector before halt-polling is not necessary:
once the pending IRQ will be recorded in the PIR, kvm_vcpu_has_events()
will detect this (via kvm_cpu_get_interrupt(), kvm_apic_get_interrupt(),
apic_has_interrupt_for_ppr() and finally vmx_sync_pir_to_irr()) and
terminate the polling.
Signed-off-by: Sean Christopherson <seanjc@google.com>
Reviewed-by: Maxim Levitsky <mlevitsk@redhat.com>
Message-Id: <20211208015236.1616697-5-seanjc@google.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2021-12-08 01:52:14 +00:00
|
|
|
struct pi_desc old, new;
|
2020-09-23 11:31:11 -07:00
|
|
|
|
2025-04-01 08:47:26 -07:00
|
|
|
lockdep_assert_irqs_disabled();
|
2021-10-08 19:12:15 -07:00
|
|
|
|
KVM: VMX: Use separate subclasses for PI wakeup lock to squash false positive
Use a separate subclass when acquiring KVM's per-CPU posted interrupts
wakeup lock in the scheduled out path, i.e. when adding a vCPU on the list
of vCPUs to wake, to workaround a false positive deadlock.
Chain exists of:
&p->pi_lock --> &rq->__lock --> &per_cpu(wakeup_vcpus_on_cpu_lock, cpu)
Possible unsafe locking scenario:
CPU0 CPU1
---- ----
lock(&per_cpu(wakeup_vcpus_on_cpu_lock, cpu));
lock(&rq->__lock);
lock(&per_cpu(wakeup_vcpus_on_cpu_lock, cpu));
lock(&p->pi_lock);
*** DEADLOCK ***
In the wakeup handler, the callchain is *always*:
sysvec_kvm_posted_intr_wakeup_ipi()
|
--> pi_wakeup_handler()
|
--> kvm_vcpu_wake_up()
|
--> try_to_wake_up(),
and the lock order is:
&per_cpu(wakeup_vcpus_on_cpu_lock, cpu) --> &p->pi_lock.
For the schedule out path, the callchain is always (for all intents and
purposes; if the kernel is preemptible, kvm_sched_out() can be called from
something other than schedule(), but the beginning of the callchain will
be the same point in vcpu_block()):
vcpu_block()
|
--> schedule()
|
--> kvm_sched_out()
|
--> vmx_vcpu_put()
|
--> vmx_vcpu_pi_put()
|
--> pi_enable_wakeup_handler()
and the lock order is:
&rq->__lock --> &per_cpu(wakeup_vcpus_on_cpu_lock, cpu)
I.e. lockdep sees AB+BC ordering for schedule out, and CA ordering for
wakeup, and complains about the A=>C versus C=>A inversion. In practice,
deadlock can't occur between schedule out and the wakeup handler as they
are mutually exclusive. The entirely of the schedule out code that runs
with the problematic scheduler locks held, does so with IRQs disabled,
i.e. can't run concurrently with the wakeup handler.
Use a subclass instead disabling lockdep entirely, and tell lockdep that
both subclasses are being acquired when loading a vCPU, as the sched_out
and sched_in paths are NOT mutually exclusive, e.g.
CPU 0 CPU 1
--------------- ---------------
vCPU0 sched_out
vCPU1 sched_in
vCPU1 sched_out vCPU 0 sched_in
where vCPU0's sched_in may race with vCPU1's sched_out, on CPU 0's wakeup
list+lock.
Signed-off-by: Yan Zhao <yan.y.zhao@intel.com>
Signed-off-by: Sean Christopherson <seanjc@google.com>
Reviewed-by: Maxim Levitsky <mlevitsk@redhat.com>
Message-ID: <20250401154727.835231-3-seanjc@google.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2025-04-01 08:47:27 -07:00
|
|
|
/*
|
|
|
|
|
* Acquire the wakeup lock using the "sched out" context to workaround
|
|
|
|
|
* a lockdep false positive. When this is called, schedule() holds
|
|
|
|
|
* various per-CPU scheduler locks. When the wakeup handler runs, it
|
|
|
|
|
* holds this CPU's wakeup lock while calling try_to_wake_up(), which
|
|
|
|
|
* can eventually take the aforementioned scheduler locks, which causes
|
|
|
|
|
* lockdep to assume there is deadlock.
|
|
|
|
|
*
|
|
|
|
|
* Deadlock can't actually occur because IRQs are disabled for the
|
|
|
|
|
* entirety of the sched_out critical section, i.e. the wakeup handler
|
|
|
|
|
* can't run while the scheduler locks are held.
|
|
|
|
|
*/
|
|
|
|
|
raw_spin_lock_nested(&per_cpu(wakeup_vcpus_on_cpu_lock, vcpu->cpu),
|
|
|
|
|
PI_LOCK_SCHED_OUT);
|
2025-03-14 14:06:48 -04:00
|
|
|
list_add_tail(&vt->pi_wakeup_list,
|
2021-12-08 01:52:16 +00:00
|
|
|
&per_cpu(wakeup_vcpus_on_cpu, vcpu->cpu));
|
|
|
|
|
raw_spin_unlock(&per_cpu(wakeup_vcpus_on_cpu_lock, vcpu->cpu));
|
2020-09-23 11:31:11 -07:00
|
|
|
|
2024-04-23 10:41:05 -07:00
|
|
|
WARN(pi_test_sn(pi_desc), "PI descriptor SN field set before blocking");
|
2021-12-08 01:52:13 +00:00
|
|
|
|
2022-06-24 11:45:45 -04:00
|
|
|
old.control = READ_ONCE(pi_desc->control);
|
2020-09-23 11:31:11 -07:00
|
|
|
do {
|
|
|
|
|
/* set 'NV' to 'wakeup vector' */
|
2022-06-24 11:45:45 -04:00
|
|
|
new.control = old.control;
|
2020-09-23 11:31:11 -07:00
|
|
|
new.nv = POSTED_INTR_WAKEUP_VECTOR;
|
2022-06-24 11:45:45 -04:00
|
|
|
} while (pi_try_set_control(pi_desc, &old.control, new.control));
|
2020-09-23 11:31:11 -07:00
|
|
|
|
KVM: VMX: Handle PI descriptor updates during vcpu_put/load
Move the posted interrupt pre/post_block logic into vcpu_put/load
respectively, using the kvm_vcpu_is_blocking() to determining whether or
not the wakeup handler needs to be set (and unset). This avoids updating
the PI descriptor if halt-polling is successful, reduces the number of
touchpoints for updating the descriptor, and eliminates the confusing
behavior of intentionally leaving a "stale" PI.NDST when a blocking vCPU
is scheduled back in after preemption.
The downside is that KVM will do the PID update twice if the vCPU is
preempted after prepare_to_rcuwait() but before schedule(), but that's a
rare case (and non-existent on !PREEMPT kernels).
The notable wart is the need to send a self-IPI on the wakeup vector if
an outstanding notification is pending after configuring the wakeup
vector. Ideally, KVM would just do a kvm_vcpu_wake_up() in this case,
but the scheduler doesn't support waking a task from its preemption
notifier callback, i.e. while the task is right in the middle of
being scheduled out.
Note, setting the wakeup vector before halt-polling is not necessary:
once the pending IRQ will be recorded in the PIR, kvm_vcpu_has_events()
will detect this (via kvm_cpu_get_interrupt(), kvm_apic_get_interrupt(),
apic_has_interrupt_for_ppr() and finally vmx_sync_pir_to_irr()) and
terminate the polling.
Signed-off-by: Sean Christopherson <seanjc@google.com>
Reviewed-by: Maxim Levitsky <mlevitsk@redhat.com>
Message-Id: <20211208015236.1616697-5-seanjc@google.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2021-12-08 01:52:14 +00:00
|
|
|
/*
|
|
|
|
|
* Send a wakeup IPI to this CPU if an interrupt may have been posted
|
|
|
|
|
* before the notification vector was updated, in which case the IRQ
|
|
|
|
|
* will arrive on the non-wakeup vector. An IPI is needed as calling
|
|
|
|
|
* try_to_wake_up() from ->sched_out() isn't allowed (IRQs are not
|
|
|
|
|
* enabled until it is safe to call try_to_wake_up() on the task being
|
|
|
|
|
* scheduled out).
|
|
|
|
|
*/
|
|
|
|
|
if (pi_test_on(&new))
|
2023-08-09 08:21:45 -07:00
|
|
|
__apic_send_IPI_self(POSTED_INTR_WAKEUP_VECTOR);
|
2020-09-23 11:31:11 -07:00
|
|
|
}
|
|
|
|
|
|
2022-04-19 23:45:10 +08:00
|
|
|
static bool vmx_needs_pi_wakeup(struct kvm_vcpu *vcpu)
|
|
|
|
|
{
|
|
|
|
|
/*
|
|
|
|
|
* The default posted interrupt vector does nothing when
|
|
|
|
|
* invoked outside guest mode. Return whether a blocked vCPU
|
|
|
|
|
* can be the target of posted interrupts, as is the case when
|
|
|
|
|
* using either IPI virtualization or VT-d PI, so that the
|
|
|
|
|
* notification vector is switched to the one that calls
|
|
|
|
|
* back to the pi_wakeup_handler() function.
|
|
|
|
|
*/
|
2025-02-22 09:47:43 +08:00
|
|
|
return (vmx_can_use_ipiv(vcpu) && !is_td_vcpu(vcpu)) ||
|
|
|
|
|
vmx_can_use_vtd_pi(vcpu->kvm);
|
2022-04-19 23:45:10 +08:00
|
|
|
}
|
|
|
|
|
|
KVM: VMX: Handle PI descriptor updates during vcpu_put/load
Move the posted interrupt pre/post_block logic into vcpu_put/load
respectively, using the kvm_vcpu_is_blocking() to determining whether or
not the wakeup handler needs to be set (and unset). This avoids updating
the PI descriptor if halt-polling is successful, reduces the number of
touchpoints for updating the descriptor, and eliminates the confusing
behavior of intentionally leaving a "stale" PI.NDST when a blocking vCPU
is scheduled back in after preemption.
The downside is that KVM will do the PID update twice if the vCPU is
preempted after prepare_to_rcuwait() but before schedule(), but that's a
rare case (and non-existent on !PREEMPT kernels).
The notable wart is the need to send a self-IPI on the wakeup vector if
an outstanding notification is pending after configuring the wakeup
vector. Ideally, KVM would just do a kvm_vcpu_wake_up() in this case,
but the scheduler doesn't support waking a task from its preemption
notifier callback, i.e. while the task is right in the middle of
being scheduled out.
Note, setting the wakeup vector before halt-polling is not necessary:
once the pending IRQ will be recorded in the PIR, kvm_vcpu_has_events()
will detect this (via kvm_cpu_get_interrupt(), kvm_apic_get_interrupt(),
apic_has_interrupt_for_ppr() and finally vmx_sync_pir_to_irr()) and
terminate the polling.
Signed-off-by: Sean Christopherson <seanjc@google.com>
Reviewed-by: Maxim Levitsky <mlevitsk@redhat.com>
Message-Id: <20211208015236.1616697-5-seanjc@google.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2021-12-08 01:52:14 +00:00
|
|
|
void vmx_vcpu_pi_put(struct kvm_vcpu *vcpu)
|
2020-09-23 11:31:11 -07:00
|
|
|
{
|
KVM: VMX: Handle PI descriptor updates during vcpu_put/load
Move the posted interrupt pre/post_block logic into vcpu_put/load
respectively, using the kvm_vcpu_is_blocking() to determining whether or
not the wakeup handler needs to be set (and unset). This avoids updating
the PI descriptor if halt-polling is successful, reduces the number of
touchpoints for updating the descriptor, and eliminates the confusing
behavior of intentionally leaving a "stale" PI.NDST when a blocking vCPU
is scheduled back in after preemption.
The downside is that KVM will do the PID update twice if the vCPU is
preempted after prepare_to_rcuwait() but before schedule(), but that's a
rare case (and non-existent on !PREEMPT kernels).
The notable wart is the need to send a self-IPI on the wakeup vector if
an outstanding notification is pending after configuring the wakeup
vector. Ideally, KVM would just do a kvm_vcpu_wake_up() in this case,
but the scheduler doesn't support waking a task from its preemption
notifier callback, i.e. while the task is right in the middle of
being scheduled out.
Note, setting the wakeup vector before halt-polling is not necessary:
once the pending IRQ will be recorded in the PIR, kvm_vcpu_has_events()
will detect this (via kvm_cpu_get_interrupt(), kvm_apic_get_interrupt(),
apic_has_interrupt_for_ppr() and finally vmx_sync_pir_to_irr()) and
terminate the polling.
Signed-off-by: Sean Christopherson <seanjc@google.com>
Reviewed-by: Maxim Levitsky <mlevitsk@redhat.com>
Message-Id: <20211208015236.1616697-5-seanjc@google.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2021-12-08 01:52:14 +00:00
|
|
|
struct pi_desc *pi_desc = vcpu_to_pi_desc(vcpu);
|
2021-10-08 19:12:18 -07:00
|
|
|
|
2022-04-19 23:45:10 +08:00
|
|
|
if (!vmx_needs_pi_wakeup(vcpu))
|
2020-09-23 11:31:11 -07:00
|
|
|
return;
|
|
|
|
|
|
KVM: VMX: Suppress PI notifications whenever the vCPU is put
Suppress posted interrupt notifications (set PID.SN=1) whenever the vCPU
is put, i.e. unloaded, not just when the vCPU is preempted, as KVM doesn't
do anything in response to a notification IRQ that arrives in the host,
nor does KVM rely on the Outstanding Notification (PID.ON) flag when the
vCPU is unloaded. And, the cost of scanning the PIR to manually set PID.ON
when loading the vCPU is quite small, especially relative to the cost of
loading (and unloading) a vCPU.
On the flip side, leaving SN clear means a notification for the vCPU will
result in a spurious IRQ for the pCPU, even if vCPU task is scheduled out,
running in userspace, etc. Even worse, if the pCPU is running a different
vCPU, the spurious IRQ could trigger posted interrupt processing for the
wrong vCPU, which is technically a violation of the architecture, as
setting bits in PIR aren't supposed to be propagated to the vIRR until a
notification IRQ is received.
The saving grace of the current behavior is that hardware sends
notification interrupts if and only if PID.ON=0, i.e. only the first
posted interrupt for a vCPU will trigger a spurious IRQ (for each window
where the vCPU is unloaded).
Ideally, KVM would suppress notifications before enabling IRQs in the
VM-Exit, but KVM relies on PID.ON as an indicator that there is a posted
interrupt pending in PIR, e.g. in vmx_sync_pir_to_irr(), and sadly there
is no way to ask hardware to set PID.ON, but not generate an interrupt.
That could be solved by using pi_has_pending_interrupt() instead of
checking only PID.ON, but it's not at all clear that would be a performance
win, as KVM would end up scanning the entire PIR whenever an interrupt
isn't pending.
And long term, the spurious IRQ window, i.e. where a vCPU is loaded with
IRQs enabled, can effectively be made smaller for hot paths by moving
performance critical VM-Exit handlers into the fastpath, i.e. by never
enabling IRQs for hot path VM-Exits.
Link: https://lore.kernel.org/r/20250611224604.313496-21-seanjc@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
2025-06-11 15:45:22 -07:00
|
|
|
/*
|
|
|
|
|
* If the vCPU is blocking with IRQs enabled and ISN'T being preempted,
|
|
|
|
|
* enable the wakeup handler so that notification IRQ wakes the vCPU as
|
|
|
|
|
* expected. There is no need to enable the wakeup handler if the vCPU
|
|
|
|
|
* is preempted between setting its wait state and manually scheduling
|
|
|
|
|
* out, as the task is still runnable, i.e. doesn't need a wake event
|
|
|
|
|
* from KVM to be scheduled in.
|
|
|
|
|
*
|
|
|
|
|
* If the wakeup handler isn't being enabled, Suppress Notifications as
|
|
|
|
|
* the cost of propagating PIR.IRR to PID.ON is negligible compared to
|
|
|
|
|
* the cost of a spurious IRQ, and vCPU put/load is a slow path.
|
|
|
|
|
*/
|
|
|
|
|
if (!vcpu->preempted && kvm_vcpu_is_blocking(vcpu) &&
|
2025-02-27 09:20:07 +08:00
|
|
|
((is_td_vcpu(vcpu) && tdx_interrupt_allowed(vcpu)) ||
|
|
|
|
|
(!is_td_vcpu(vcpu) && !vmx_interrupt_blocked(vcpu))))
|
KVM: VMX: Handle PI descriptor updates during vcpu_put/load
Move the posted interrupt pre/post_block logic into vcpu_put/load
respectively, using the kvm_vcpu_is_blocking() to determining whether or
not the wakeup handler needs to be set (and unset). This avoids updating
the PI descriptor if halt-polling is successful, reduces the number of
touchpoints for updating the descriptor, and eliminates the confusing
behavior of intentionally leaving a "stale" PI.NDST when a blocking vCPU
is scheduled back in after preemption.
The downside is that KVM will do the PID update twice if the vCPU is
preempted after prepare_to_rcuwait() but before schedule(), but that's a
rare case (and non-existent on !PREEMPT kernels).
The notable wart is the need to send a self-IPI on the wakeup vector if
an outstanding notification is pending after configuring the wakeup
vector. Ideally, KVM would just do a kvm_vcpu_wake_up() in this case,
but the scheduler doesn't support waking a task from its preemption
notifier callback, i.e. while the task is right in the middle of
being scheduled out.
Note, setting the wakeup vector before halt-polling is not necessary:
once the pending IRQ will be recorded in the PIR, kvm_vcpu_has_events()
will detect this (via kvm_cpu_get_interrupt(), kvm_apic_get_interrupt(),
apic_has_interrupt_for_ppr() and finally vmx_sync_pir_to_irr()) and
terminate the polling.
Signed-off-by: Sean Christopherson <seanjc@google.com>
Reviewed-by: Maxim Levitsky <mlevitsk@redhat.com>
Message-Id: <20211208015236.1616697-5-seanjc@google.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2021-12-08 01:52:14 +00:00
|
|
|
pi_enable_wakeup_handler(vcpu);
|
KVM: VMX: Suppress PI notifications whenever the vCPU is put
Suppress posted interrupt notifications (set PID.SN=1) whenever the vCPU
is put, i.e. unloaded, not just when the vCPU is preempted, as KVM doesn't
do anything in response to a notification IRQ that arrives in the host,
nor does KVM rely on the Outstanding Notification (PID.ON) flag when the
vCPU is unloaded. And, the cost of scanning the PIR to manually set PID.ON
when loading the vCPU is quite small, especially relative to the cost of
loading (and unloading) a vCPU.
On the flip side, leaving SN clear means a notification for the vCPU will
result in a spurious IRQ for the pCPU, even if vCPU task is scheduled out,
running in userspace, etc. Even worse, if the pCPU is running a different
vCPU, the spurious IRQ could trigger posted interrupt processing for the
wrong vCPU, which is technically a violation of the architecture, as
setting bits in PIR aren't supposed to be propagated to the vIRR until a
notification IRQ is received.
The saving grace of the current behavior is that hardware sends
notification interrupts if and only if PID.ON=0, i.e. only the first
posted interrupt for a vCPU will trigger a spurious IRQ (for each window
where the vCPU is unloaded).
Ideally, KVM would suppress notifications before enabling IRQs in the
VM-Exit, but KVM relies on PID.ON as an indicator that there is a posted
interrupt pending in PIR, e.g. in vmx_sync_pir_to_irr(), and sadly there
is no way to ask hardware to set PID.ON, but not generate an interrupt.
That could be solved by using pi_has_pending_interrupt() instead of
checking only PID.ON, but it's not at all clear that would be a performance
win, as KVM would end up scanning the entire PIR whenever an interrupt
isn't pending.
And long term, the spurious IRQ window, i.e. where a vCPU is loaded with
IRQs enabled, can effectively be made smaller for hot paths by moving
performance critical VM-Exit handlers into the fastpath, i.e. by never
enabling IRQs for hot path VM-Exits.
Link: https://lore.kernel.org/r/20250611224604.313496-21-seanjc@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
2025-06-11 15:45:22 -07:00
|
|
|
else
|
KVM: VMX: Handle PI descriptor updates during vcpu_put/load
Move the posted interrupt pre/post_block logic into vcpu_put/load
respectively, using the kvm_vcpu_is_blocking() to determining whether or
not the wakeup handler needs to be set (and unset). This avoids updating
the PI descriptor if halt-polling is successful, reduces the number of
touchpoints for updating the descriptor, and eliminates the confusing
behavior of intentionally leaving a "stale" PI.NDST when a blocking vCPU
is scheduled back in after preemption.
The downside is that KVM will do the PID update twice if the vCPU is
preempted after prepare_to_rcuwait() but before schedule(), but that's a
rare case (and non-existent on !PREEMPT kernels).
The notable wart is the need to send a self-IPI on the wakeup vector if
an outstanding notification is pending after configuring the wakeup
vector. Ideally, KVM would just do a kvm_vcpu_wake_up() in this case,
but the scheduler doesn't support waking a task from its preemption
notifier callback, i.e. while the task is right in the middle of
being scheduled out.
Note, setting the wakeup vector before halt-polling is not necessary:
once the pending IRQ will be recorded in the PIR, kvm_vcpu_has_events()
will detect this (via kvm_cpu_get_interrupt(), kvm_apic_get_interrupt(),
apic_has_interrupt_for_ppr() and finally vmx_sync_pir_to_irr()) and
terminate the polling.
Signed-off-by: Sean Christopherson <seanjc@google.com>
Reviewed-by: Maxim Levitsky <mlevitsk@redhat.com>
Message-Id: <20211208015236.1616697-5-seanjc@google.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2021-12-08 01:52:14 +00:00
|
|
|
pi_set_sn(pi_desc);
|
2020-09-23 11:31:11 -07:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Handler for POSTED_INTERRUPT_WAKEUP_VECTOR.
|
|
|
|
|
*/
|
|
|
|
|
void pi_wakeup_handler(void)
|
|
|
|
|
{
|
|
|
|
|
int cpu = smp_processor_id();
|
2022-04-06 19:25:02 +08:00
|
|
|
struct list_head *wakeup_list = &per_cpu(wakeup_vcpus_on_cpu, cpu);
|
|
|
|
|
raw_spinlock_t *spinlock = &per_cpu(wakeup_vcpus_on_cpu_lock, cpu);
|
2025-03-14 14:06:48 -04:00
|
|
|
struct vcpu_vt *vt;
|
2020-09-23 11:31:11 -07:00
|
|
|
|
2022-04-06 19:25:02 +08:00
|
|
|
raw_spin_lock(spinlock);
|
2025-03-14 14:06:48 -04:00
|
|
|
list_for_each_entry(vt, wakeup_list, pi_wakeup_list) {
|
2020-09-23 11:31:11 -07:00
|
|
|
|
2025-03-14 14:06:48 -04:00
|
|
|
if (pi_test_on(&vt->pi_desc))
|
|
|
|
|
kvm_vcpu_wake_up(vt_to_vcpu(vt));
|
2020-09-23 11:31:11 -07:00
|
|
|
}
|
2022-04-06 19:25:02 +08:00
|
|
|
raw_spin_unlock(spinlock);
|
2020-09-23 11:31:11 -07:00
|
|
|
}
|
|
|
|
|
|
2020-10-24 04:08:37 -04:00
|
|
|
void __init pi_init_cpu(int cpu)
|
2020-09-23 11:31:11 -07:00
|
|
|
{
|
2021-12-08 01:52:16 +00:00
|
|
|
INIT_LIST_HEAD(&per_cpu(wakeup_vcpus_on_cpu, cpu));
|
|
|
|
|
raw_spin_lock_init(&per_cpu(wakeup_vcpus_on_cpu_lock, cpu));
|
2020-09-23 11:31:11 -07:00
|
|
|
}
|
|
|
|
|
|
2025-03-18 00:35:07 -06:00
|
|
|
void pi_apicv_pre_state_restore(struct kvm_vcpu *vcpu)
|
|
|
|
|
{
|
|
|
|
|
struct pi_desc *pi = vcpu_to_pi_desc(vcpu);
|
|
|
|
|
|
|
|
|
|
pi_clear_on(pi);
|
|
|
|
|
memset(pi->pir, 0, sizeof(pi->pir));
|
|
|
|
|
}
|
|
|
|
|
|
2020-09-23 11:31:11 -07:00
|
|
|
bool pi_has_pending_interrupt(struct kvm_vcpu *vcpu)
|
|
|
|
|
{
|
|
|
|
|
struct pi_desc *pi_desc = vcpu_to_pi_desc(vcpu);
|
|
|
|
|
|
|
|
|
|
return pi_test_on(pi_desc) ||
|
|
|
|
|
(pi_test_sn(pi_desc) && !pi_is_pir_empty(pi_desc));
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
|
KVM: VMX: update vcpu posted-interrupt descriptor when assigning device
For VMX, when a vcpu enters HLT emulation, pi_post_block will:
1) Add vcpu to per-cpu list of blocked vcpus.
2) Program the posted-interrupt descriptor "notification vector"
to POSTED_INTR_WAKEUP_VECTOR
With interrupt remapping, an interrupt will set the PIR bit for the
vector programmed for the device on the CPU, test-and-set the
ON bit on the posted interrupt descriptor, and if the ON bit is clear
generate an interrupt for the notification vector.
This way, the target CPU wakes upon a device interrupt and wakes up
the target vcpu.
Problem is that pi_post_block only programs the notification vector
if kvm_arch_has_assigned_device() is true. Its possible for the
following to happen:
1) vcpu V HLTs on pcpu P, kvm_arch_has_assigned_device is false,
notification vector is not programmed
2) device is assigned to VM
3) device interrupts vcpu V, sets ON bit
(notification vector not programmed, so pcpu P remains in idle)
4) vcpu 0 IPIs vcpu V (in guest), but since pi descriptor ON bit is set,
kvm_vcpu_kick is skipped
5) vcpu 0 busy spins on vcpu V's response for several seconds, until
RCU watchdog NMIs all vCPUs.
To fix this, use the start_assignment kvm_x86_ops callback to kick
vcpus out of the halt loop, so the notification vector is
properly reprogrammed to the wakeup vector.
Reported-by: Pei Zhang <pezhang@redhat.com>
Signed-off-by: Marcelo Tosatti <mtosatti@redhat.com>
Message-Id: <20210526172014.GA29007@fuller.cnet>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2021-05-26 14:20:14 -03:00
|
|
|
/*
|
2025-06-11 15:45:56 -07:00
|
|
|
* Kick all vCPUs when the first possible bypass IRQ is attached to a VM, as
|
|
|
|
|
* blocking vCPUs may scheduled out without reconfiguring PID.NV to the wakeup
|
|
|
|
|
* vector, i.e. if the bypass IRQ came along after vmx_vcpu_pi_put().
|
KVM: VMX: update vcpu posted-interrupt descriptor when assigning device
For VMX, when a vcpu enters HLT emulation, pi_post_block will:
1) Add vcpu to per-cpu list of blocked vcpus.
2) Program the posted-interrupt descriptor "notification vector"
to POSTED_INTR_WAKEUP_VECTOR
With interrupt remapping, an interrupt will set the PIR bit for the
vector programmed for the device on the CPU, test-and-set the
ON bit on the posted interrupt descriptor, and if the ON bit is clear
generate an interrupt for the notification vector.
This way, the target CPU wakes upon a device interrupt and wakes up
the target vcpu.
Problem is that pi_post_block only programs the notification vector
if kvm_arch_has_assigned_device() is true. Its possible for the
following to happen:
1) vcpu V HLTs on pcpu P, kvm_arch_has_assigned_device is false,
notification vector is not programmed
2) device is assigned to VM
3) device interrupts vcpu V, sets ON bit
(notification vector not programmed, so pcpu P remains in idle)
4) vcpu 0 IPIs vcpu V (in guest), but since pi descriptor ON bit is set,
kvm_vcpu_kick is skipped
5) vcpu 0 busy spins on vcpu V's response for several seconds, until
RCU watchdog NMIs all vCPUs.
To fix this, use the start_assignment kvm_x86_ops callback to kick
vcpus out of the halt loop, so the notification vector is
properly reprogrammed to the wakeup vector.
Reported-by: Pei Zhang <pezhang@redhat.com>
Signed-off-by: Marcelo Tosatti <mtosatti@redhat.com>
Message-Id: <20210526172014.GA29007@fuller.cnet>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2021-05-26 14:20:14 -03:00
|
|
|
*/
|
2025-06-11 15:45:56 -07:00
|
|
|
void vmx_pi_start_bypass(struct kvm *kvm)
|
KVM: VMX: update vcpu posted-interrupt descriptor when assigning device
For VMX, when a vcpu enters HLT emulation, pi_post_block will:
1) Add vcpu to per-cpu list of blocked vcpus.
2) Program the posted-interrupt descriptor "notification vector"
to POSTED_INTR_WAKEUP_VECTOR
With interrupt remapping, an interrupt will set the PIR bit for the
vector programmed for the device on the CPU, test-and-set the
ON bit on the posted interrupt descriptor, and if the ON bit is clear
generate an interrupt for the notification vector.
This way, the target CPU wakes upon a device interrupt and wakes up
the target vcpu.
Problem is that pi_post_block only programs the notification vector
if kvm_arch_has_assigned_device() is true. Its possible for the
following to happen:
1) vcpu V HLTs on pcpu P, kvm_arch_has_assigned_device is false,
notification vector is not programmed
2) device is assigned to VM
3) device interrupts vcpu V, sets ON bit
(notification vector not programmed, so pcpu P remains in idle)
4) vcpu 0 IPIs vcpu V (in guest), but since pi descriptor ON bit is set,
kvm_vcpu_kick is skipped
5) vcpu 0 busy spins on vcpu V's response for several seconds, until
RCU watchdog NMIs all vCPUs.
To fix this, use the start_assignment kvm_x86_ops callback to kick
vcpus out of the halt loop, so the notification vector is
properly reprogrammed to the wakeup vector.
Reported-by: Pei Zhang <pezhang@redhat.com>
Signed-off-by: Marcelo Tosatti <mtosatti@redhat.com>
Message-Id: <20210526172014.GA29007@fuller.cnet>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2021-05-26 14:20:14 -03:00
|
|
|
{
|
2025-06-11 15:45:57 -07:00
|
|
|
if (WARN_ON_ONCE(!vmx_can_use_vtd_pi(kvm)))
|
KVM: VMX: update vcpu posted-interrupt descriptor when assigning device
For VMX, when a vcpu enters HLT emulation, pi_post_block will:
1) Add vcpu to per-cpu list of blocked vcpus.
2) Program the posted-interrupt descriptor "notification vector"
to POSTED_INTR_WAKEUP_VECTOR
With interrupt remapping, an interrupt will set the PIR bit for the
vector programmed for the device on the CPU, test-and-set the
ON bit on the posted interrupt descriptor, and if the ON bit is clear
generate an interrupt for the notification vector.
This way, the target CPU wakes upon a device interrupt and wakes up
the target vcpu.
Problem is that pi_post_block only programs the notification vector
if kvm_arch_has_assigned_device() is true. Its possible for the
following to happen:
1) vcpu V HLTs on pcpu P, kvm_arch_has_assigned_device is false,
notification vector is not programmed
2) device is assigned to VM
3) device interrupts vcpu V, sets ON bit
(notification vector not programmed, so pcpu P remains in idle)
4) vcpu 0 IPIs vcpu V (in guest), but since pi descriptor ON bit is set,
kvm_vcpu_kick is skipped
5) vcpu 0 busy spins on vcpu V's response for several seconds, until
RCU watchdog NMIs all vCPUs.
To fix this, use the start_assignment kvm_x86_ops callback to kick
vcpus out of the halt loop, so the notification vector is
properly reprogrammed to the wakeup vector.
Reported-by: Pei Zhang <pezhang@redhat.com>
Signed-off-by: Marcelo Tosatti <mtosatti@redhat.com>
Message-Id: <20210526172014.GA29007@fuller.cnet>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2021-05-26 14:20:14 -03:00
|
|
|
return;
|
|
|
|
|
|
|
|
|
|
kvm_make_all_cpus_request(kvm, KVM_REQ_UNBLOCK);
|
|
|
|
|
}
|
|
|
|
|
|
2025-06-11 15:45:06 -07:00
|
|
|
int vmx_pi_update_irte(struct kvm_kernel_irqfd *irqfd, struct kvm *kvm,
|
|
|
|
|
unsigned int host_irq, uint32_t guest_irq,
|
2025-06-11 15:45:31 -07:00
|
|
|
struct kvm_vcpu *vcpu, u32 vector)
|
2020-09-23 11:31:11 -07:00
|
|
|
{
|
2025-06-11 15:45:31 -07:00
|
|
|
if (vcpu) {
|
2025-06-11 15:45:34 -07:00
|
|
|
struct intel_iommu_pi_data pi_data = {
|
2025-06-11 15:45:31 -07:00
|
|
|
.pi_desc_addr = __pa(vcpu_to_pi_desc(vcpu)),
|
|
|
|
|
.vector = vector,
|
|
|
|
|
};
|
|
|
|
|
|
2025-06-11 15:45:34 -07:00
|
|
|
return irq_set_vcpu_affinity(host_irq, &pi_data);
|
2025-06-11 15:45:31 -07:00
|
|
|
} else {
|
|
|
|
|
return irq_set_vcpu_affinity(host_irq, NULL);
|
|
|
|
|
}
|
2020-09-23 11:31:11 -07:00
|
|
|
}
|