public-mirrors/linux
1
0
Fork 0
mirror of git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git synced 2025-08-05 16:54:27 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions
linux/arch/riscv/include/asm/mmu_context.h

54 lines
1.2 KiB
C
Raw Permalink Normal View History

treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 286 Based on 1 normalized pattern(s): this program is free software you can redistribute it and or modify it under the terms of the gnu general public license as published by the free software foundation version 2 this program is distributed in the hope that it will be useful but without any warranty without even the implied warranty of merchantability or fitness for a particular purpose see the gnu general public license for more details extracted by the scancode license scanner the SPDX license identifier GPL-2.0-only has been chosen to replace the boilerplate/reference in 97 file(s). Signed-off-by: Thomas Gleixner <tglx@linutronix.de> Reviewed-by: Allison Randal <allison@lohutok.net> Reviewed-by: Alexios Zavras <alexios.zavras@intel.com> Cc: linux-spdx@vger.kernel.org Link: https://lkml.kernel.org/r/20190529141901.025053186@linutronix.de Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
2019-05-29 07:18:00 -07:00
/* SPDX-License-Identifier: GPL-2.0-only */
RISC-V: Paging and MMU This patch contains code to manage the RISC-V MMU, including definitions of the page tables and the page walking code. Signed-off-by: Palmer Dabbelt <palmer@dabbelt.com>
2017-07-10 18:06:09 -07:00
/*
* Copyright (C) 2012 Regents of the University of California
RISC-V: Flush I$ when making a dirty page executable The RISC-V ISA allows for instruction caches that are not coherent WRT stores, even on a single hart. As a result, we need to explicitly flush the instruction cache whenever marking a dirty page as executable in order to preserve the correct system behavior. Local instruction caches aren't that scary (our implementations actually flush the cache, but RISC-V is defined to allow higher-performance implementations to exist), but RISC-V defines no way to perform an instruction cache shootdown. When explicitly asked to do so we can shoot down remote instruction caches via an IPI, but this is a bit on the slow side. Instead of requiring an IPI to all harts whenever marking a page as executable, we simply flush the currently running harts. In order to maintain correct behavior, we additionally mark every other hart as needing a deferred instruction cache which will be taken before anything runs on it. Signed-off-by: Andrew Waterman <andrew@sifive.com> Signed-off-by: Palmer Dabbelt <palmer@sifive.com>
2017-10-25 14:30:32 -07:00
* Copyright (C) 2017 SiFive
RISC-V: Paging and MMU This patch contains code to manage the RISC-V MMU, including definitions of the page tables and the page walking code. Signed-off-by: Palmer Dabbelt <palmer@dabbelt.com>
2017-07-10 18:06:09 -07:00
*/
#ifndef _ASM_RISCV_MMU_CONTEXT_H
#define _ASM_RISCV_MMU_CONTEXT_H
RISC-V: Add missing include Fixes: include/asm-generic/mm_hooks.h:20:11: warning: 'struct vm_area_struct' declared inside parameter list will not be visible outside of this definition or declaration include/asm-generic/mm_hooks.h:19:38: warning: 'struct mm_struct' declared inside parameter list will not be visible outside of this definition or declaration Signed-off-by: Olof Johansson <olof@lixom.net> Signed-off-by: Palmer Dabbelt <palmer@sifive.com>
2017-11-29 17:55:20 -08:00
#include <linux/mm_types.h>
RISC-V: Paging and MMU This patch contains code to manage the RISC-V MMU, including definitions of the page tables and the page walking code. Signed-off-by: Palmer Dabbelt <palmer@dabbelt.com>
2017-07-10 18:06:09 -07:00
#include <asm-generic/mm_hooks.h>
#include <linux/mm.h>
#include <linux/sched.h>
riscv: move switch_mm to its own file switch_mm is an expensive operations that has two users. flush_icache_deferred is only called within switch_mm and can be moved together. The function is expected to be more complicated when ASID support is added, so clean up eagerly. By moving them to a separate file we also removes some excessive dependency of tlbflush.h and cacheflush.h. Signed-off-by: Gary Guo <gary@garyguo.net> Reviewed-by: Anup Patel <anup@brainfault.org> Signed-off-by: Christoph Hellwig <hch@lst.de> Signed-off-by: Palmer Dabbelt <palmer@sifive.com>
2019-03-27 00:41:29 +00:00
void switch_mm(struct mm_struct *prev, struct mm_struct *next,
struct task_struct *task);
RISC-V: Paging and MMU This patch contains code to manage the RISC-V MMU, including definitions of the page tables and the page walking code. Signed-off-by: Palmer Dabbelt <palmer@dabbelt.com>
2017-07-10 18:06:09 -07:00
riscv: use asm-generic/mmu_context.h for no-op implementations Signed-off-by: Nicholas Piggin <npiggin@gmail.com> Reviewed-by: Pekka Enberg <penberg@kernel.org> Acked-by: Palmer Dabbelt <palmerdabbelt@google.com> Cc: Paul Walmsley <paul.walmsley@sifive.com> Cc: Palmer Dabbelt <palmer@dabbelt.com> Cc: Albert Ou <aou@eecs.berkeley.edu> Cc: linux-riscv@lists.infradead.org Signed-off-by: Arnd Bergmann <arnd@arndb.de>
2020-09-02 00:15:33 +10:00
#define activate_mm activate_mm
RISC-V: Paging and MMU This patch contains code to manage the RISC-V MMU, including definitions of the page tables and the page walking code. Signed-off-by: Palmer Dabbelt <palmer@dabbelt.com>
2017-07-10 18:06:09 -07:00
static inline void activate_mm(struct mm_struct *prev,
struct mm_struct *next)
{
riscv: Add support for the tagged address ABI When pointer masking is enabled for userspace, the kernel can accept tagged pointers as arguments to some system calls. Allow this by untagging the pointers in access_ok() and the uaccess routines. The uaccess routines must peform untagging in software because U-mode and S-mode have entirely separate pointer masking configurations. In fact, hardware may not even implement pointer masking for S-mode. Since the number of tag bits is variable, untagged_addr_remote() needs to know what PMLEN to use for the remote mm. Therefore, the pointer masking mode must be the same for all threads sharing an mm. Enforce this with a lock flag in the mm context, as x86 does for LAM. The flag gets reset in init_new_context() during fork(), as the new mm is no longer multithreaded. Reviewed-by: Charlie Jenkins <charlie@rivosinc.com> Tested-by: Charlie Jenkins <charlie@rivosinc.com> Signed-off-by: Samuel Holland <samuel.holland@sifive.com> Link: https://lore.kernel.org/r/20241016202814.4061541-6-samuel.holland@sifive.com Signed-off-by: Palmer Dabbelt <palmer@rivosinc.com>
2024-10-16 13:27:46 -07:00
#ifdef CONFIG_RISCV_ISA_SUPM
next->context.pmlen = 0;
#endif
RISC-V: Paging and MMU This patch contains code to manage the RISC-V MMU, including definitions of the page tables and the page walking code. Signed-off-by: Palmer Dabbelt <palmer@dabbelt.com>
2017-07-10 18:06:09 -07:00
switch_mm(prev, next, NULL);
}
RISC-V: Implement ASID allocator Currently, we do local TLB flush on every MM switch. This is very harsh on performance because we are forcing page table walks after every MM switch. This patch implements ASID allocator for assigning an ASID to a MM context. The number of ASIDs are limited in HW so we create a logical entity named CONTEXTID for assigning to MM context. The lower bits of CONTEXTID are ASID and upper bits are VERSION number. The number of usable ASID bits supported by HW are detected at boot-time by writing 1s to ASID bits in SATP CSR. We allocate new CONTEXTID on first MM switch for a MM context where the ASID is allocated from an ASID bitmap and VERSION is provide by an atomic counter. At time of allocating new CONTEXTID, if we run out of available ASIDs then: 1. We flush the ASID bitmap 2. Increment current VERSION atomic counter 3. Re-allocate ASID from ASID bitmap 4. Flush TLB on all CPUs 5. Try CONTEXTID re-assignment on all CPUs Please note that we don't use ASID #0 because it is used at boot-time by all CPUs for initial MM context. Also, newly created context is always assigned CONTEXTID #0 (i.e. VERSION #0 and ASID #0) which is an invalid context in our implementation. Using above approach, we have virtually infinite CONTEXTIDs on-top-of limited number of HW ASIDs. This approach is inspired from ASID allocator used for Linux ARM/ARM64 but we have adapted it for RISC-V. Overall, this ASID allocator helps us reduce rate of local TLB flushes on every CPU thereby increasing performance. This patch is tested on QEMU virt machine, Spike and SiFive Unleashed board. On QEMU virt machine, we see some (3-5% approx) performance improvement with SW emulated TLBs provided by QEMU. Unfortunately, the ASID bits of the SATP CSR are not implemented on Spike and SiFive Unleashed board so we don't see any change in performance. On real HW having all ASID bits implemented, the performance gains will be much more due improved sharing of TLB among different processes. Signed-off-by: Anup Patel <anup.patel@wdc.com> Reviewed-by: Palmer Dabbelt <palmerdabbelt@google.com> Signed-off-by: Palmer Dabbelt <palmerdabbelt@google.com>
2021-02-03 15:19:07 +05:30
#define init_new_context init_new_context
static inline int init_new_context(struct task_struct *tsk,
struct mm_struct *mm)
{
#ifdef CONFIG_MMU
atomic_long_set(&mm->context.id, 0);
#endif
riscv: Add support for the tagged address ABI When pointer masking is enabled for userspace, the kernel can accept tagged pointers as arguments to some system calls. Allow this by untagging the pointers in access_ok() and the uaccess routines. The uaccess routines must peform untagging in software because U-mode and S-mode have entirely separate pointer masking configurations. In fact, hardware may not even implement pointer masking for S-mode. Since the number of tag bits is variable, untagged_addr_remote() needs to know what PMLEN to use for the remote mm. Therefore, the pointer masking mode must be the same for all threads sharing an mm. Enforce this with a lock flag in the mm context, as x86 does for LAM. The flag gets reset in init_new_context() during fork(), as the new mm is no longer multithreaded. Reviewed-by: Charlie Jenkins <charlie@rivosinc.com> Tested-by: Charlie Jenkins <charlie@rivosinc.com> Signed-off-by: Samuel Holland <samuel.holland@sifive.com> Link: https://lore.kernel.org/r/20241016202814.4061541-6-samuel.holland@sifive.com Signed-off-by: Palmer Dabbelt <palmer@rivosinc.com>
2024-10-16 13:27:46 -07:00
if (IS_ENABLED(CONFIG_RISCV_ISA_SUPM))
clear_bit(MM_CONTEXT_LOCK_PMLEN, &mm->context.flags);
RISC-V: Implement ASID allocator Currently, we do local TLB flush on every MM switch. This is very harsh on performance because we are forcing page table walks after every MM switch. This patch implements ASID allocator for assigning an ASID to a MM context. The number of ASIDs are limited in HW so we create a logical entity named CONTEXTID for assigning to MM context. The lower bits of CONTEXTID are ASID and upper bits are VERSION number. The number of usable ASID bits supported by HW are detected at boot-time by writing 1s to ASID bits in SATP CSR. We allocate new CONTEXTID on first MM switch for a MM context where the ASID is allocated from an ASID bitmap and VERSION is provide by an atomic counter. At time of allocating new CONTEXTID, if we run out of available ASIDs then: 1. We flush the ASID bitmap 2. Increment current VERSION atomic counter 3. Re-allocate ASID from ASID bitmap 4. Flush TLB on all CPUs 5. Try CONTEXTID re-assignment on all CPUs Please note that we don't use ASID #0 because it is used at boot-time by all CPUs for initial MM context. Also, newly created context is always assigned CONTEXTID #0 (i.e. VERSION #0 and ASID #0) which is an invalid context in our implementation. Using above approach, we have virtually infinite CONTEXTIDs on-top-of limited number of HW ASIDs. This approach is inspired from ASID allocator used for Linux ARM/ARM64 but we have adapted it for RISC-V. Overall, this ASID allocator helps us reduce rate of local TLB flushes on every CPU thereby increasing performance. This patch is tested on QEMU virt machine, Spike and SiFive Unleashed board. On QEMU virt machine, we see some (3-5% approx) performance improvement with SW emulated TLBs provided by QEMU. Unfortunately, the ASID bits of the SATP CSR are not implemented on Spike and SiFive Unleashed board so we don't see any change in performance. On real HW having all ASID bits implemented, the performance gains will be much more due improved sharing of TLB among different processes. Signed-off-by: Anup Patel <anup.patel@wdc.com> Reviewed-by: Palmer Dabbelt <palmerdabbelt@google.com> Signed-off-by: Palmer Dabbelt <palmerdabbelt@google.com>
2021-02-03 15:19:07 +05:30
return 0;
}
riscv: add ASID-based tlbflushing methods Implement optimized version of the tlb flushing routines for systems using ASIDs. These are behind the use_asid_allocator static branch to not affect existing systems not using ASIDs. Signed-off-by: Guo Ren <guoren@linux.alibaba.com> [hch: rebased on top of previous cleanups, use the same algorithm as the non-ASID based code for local vs global flushes, keep functions as local as possible] Signed-off-by: Christoph Hellwig <hch@lst.de> Tested-by: Guo Ren <guoren@kernel.org> Signed-off-by: Palmer Dabbelt <palmerdabbelt@google.com>
2021-06-06 17:20:50 +02:00
DECLARE_STATIC_KEY_FALSE(use_asid_allocator);
riscv: Add support for the tagged address ABI When pointer masking is enabled for userspace, the kernel can accept tagged pointers as arguments to some system calls. Allow this by untagging the pointers in access_ok() and the uaccess routines. The uaccess routines must peform untagging in software because U-mode and S-mode have entirely separate pointer masking configurations. In fact, hardware may not even implement pointer masking for S-mode. Since the number of tag bits is variable, untagged_addr_remote() needs to know what PMLEN to use for the remote mm. Therefore, the pointer masking mode must be the same for all threads sharing an mm. Enforce this with a lock flag in the mm context, as x86 does for LAM. The flag gets reset in init_new_context() during fork(), as the new mm is no longer multithreaded. Reviewed-by: Charlie Jenkins <charlie@rivosinc.com> Tested-by: Charlie Jenkins <charlie@rivosinc.com> Signed-off-by: Samuel Holland <samuel.holland@sifive.com> Link: https://lore.kernel.org/r/20241016202814.4061541-6-samuel.holland@sifive.com Signed-off-by: Palmer Dabbelt <palmer@rivosinc.com>
2024-10-16 13:27:46 -07:00
#ifdef CONFIG_RISCV_ISA_SUPM
#define mm_untag_mask mm_untag_mask
static inline unsigned long mm_untag_mask(struct mm_struct *mm)
{
return -1UL >> mm->context.pmlen;
}
#endif
riscv: use asm-generic/mmu_context.h for no-op implementations Signed-off-by: Nicholas Piggin <npiggin@gmail.com> Reviewed-by: Pekka Enberg <penberg@kernel.org> Acked-by: Palmer Dabbelt <palmerdabbelt@google.com> Cc: Paul Walmsley <paul.walmsley@sifive.com> Cc: Palmer Dabbelt <palmer@dabbelt.com> Cc: Albert Ou <aou@eecs.berkeley.edu> Cc: linux-riscv@lists.infradead.org Signed-off-by: Arnd Bergmann <arnd@arndb.de>
2020-09-02 00:15:33 +10:00
#include <asm-generic/mmu_context.h>
RISC-V: Paging and MMU This patch contains code to manage the RISC-V MMU, including definitions of the page tables and the page walking code. Signed-off-by: Palmer Dabbelt <palmer@dabbelt.com>
2017-07-10 18:06:09 -07:00
#endif /* _ASM_RISCV_MMU_CONTEXT_H */
Reference in a new issue Copy permalink