2018-05-08 16:20:54 +02:00
|
|
|
// SPDX-License-Identifier: GPL-2.0
|
2015-12-03 18:21:29 +01:00
|
|
|
/*
|
2018-05-08 16:20:54 +02:00
|
|
|
* Copyright (C) 2015-2018 Etnaviv Project
|
2015-12-03 18:21:29 +01:00
|
|
|
*/
|
|
|
|
|
|
drm/etnaviv: rework MMU handling
This reworks the MMU handling to make it possible to have multiple MMU contexts.
A context is basically one instance of GPU page tables. Currently we have one
set of page tables per GPU, which isn't all that clever, as it has the
following two consequences:
1. All GPU clients (aka processes) are sharing the same pagetables, which means
there is no isolation between clients, but only between GPU assigned memory
spaces and the rest of the system. Better than nothing, but also not great.
2. Clients operating on the same set of buffers with different etnaviv GPU
cores, e.g. a workload using both the 2D and 3D GPU, need to map the used
buffers into the pagetable sets of each used GPU.
This patch reworks all the MMU handling to introduce the abstraction of the
MMU context. A context can be shared across different GPU cores, as long as
they have compatible MMU implementations, which is the case for all systems
with Vivante GPUs seen in the wild.
As MMUv1 is not able to change pagetables on the fly, without a
"stop the world" operation, which stops GPU, changes pagetables via CPU
interaction, restarts GPU, the implementation introduces a shared context on
MMUv1, which is returned whenever there is a request for a new context.
This patch assigns a MMU context to each GPU, so on MMUv2 systems there is
still one set of pagetables per GPU, but due to the shared context MMUv1
systems see a change in behavior as now a single pagetable set is used
across all GPU cores.
Signed-off-by: Lucas Stach <l.stach@pengutronix.de>
Reviewed-by: Philipp Zabel <p.zabel@pengutronix.de>
Reviewed-by: Guido Günther <agx@sigxcpu.org>
2019-07-05 19:17:24 +02:00
|
|
|
#include <linux/dma-mapping.h>
|
2019-06-30 07:21:03 +02:00
|
|
|
#include <linux/scatterlist.h>
|
|
|
|
|
|
2016-08-16 12:09:08 +02:00
|
|
|
#include "common.xml.h"
|
2017-01-16 16:09:51 +01:00
|
|
|
#include "etnaviv_cmdbuf.h"
|
2015-12-03 18:21:29 +01:00
|
|
|
#include "etnaviv_drv.h"
|
|
|
|
|
#include "etnaviv_gem.h"
|
|
|
|
|
#include "etnaviv_gpu.h"
|
|
|
|
|
#include "etnaviv_mmu.h"
|
|
|
|
|
|
drm/etnaviv: rework MMU handling
This reworks the MMU handling to make it possible to have multiple MMU contexts.
A context is basically one instance of GPU page tables. Currently we have one
set of page tables per GPU, which isn't all that clever, as it has the
following two consequences:
1. All GPU clients (aka processes) are sharing the same pagetables, which means
there is no isolation between clients, but only between GPU assigned memory
spaces and the rest of the system. Better than nothing, but also not great.
2. Clients operating on the same set of buffers with different etnaviv GPU
cores, e.g. a workload using both the 2D and 3D GPU, need to map the used
buffers into the pagetable sets of each used GPU.
This patch reworks all the MMU handling to introduce the abstraction of the
MMU context. A context can be shared across different GPU cores, as long as
they have compatible MMU implementations, which is the case for all systems
with Vivante GPUs seen in the wild.
As MMUv1 is not able to change pagetables on the fly, without a
"stop the world" operation, which stops GPU, changes pagetables via CPU
interaction, restarts GPU, the implementation introduces a shared context on
MMUv1, which is returned whenever there is a request for a new context.
This patch assigns a MMU context to each GPU, so on MMUv2 systems there is
still one set of pagetables per GPU, but due to the shared context MMUv1
systems see a change in behavior as now a single pagetable set is used
across all GPU cores.
Signed-off-by: Lucas Stach <l.stach@pengutronix.de>
Reviewed-by: Philipp Zabel <p.zabel@pengutronix.de>
Reviewed-by: Guido Günther <agx@sigxcpu.org>
2019-07-05 19:17:24 +02:00
|
|
|
static void etnaviv_context_unmap(struct etnaviv_iommu_context *context,
|
2017-09-07 16:52:13 +02:00
|
|
|
unsigned long iova, size_t size)
|
|
|
|
|
{
|
|
|
|
|
size_t unmapped_page, unmapped = 0;
|
|
|
|
|
size_t pgsize = SZ_4K;
|
|
|
|
|
|
|
|
|
|
while (unmapped < size) {
|
drm/etnaviv: rework MMU handling
This reworks the MMU handling to make it possible to have multiple MMU contexts.
A context is basically one instance of GPU page tables. Currently we have one
set of page tables per GPU, which isn't all that clever, as it has the
following two consequences:
1. All GPU clients (aka processes) are sharing the same pagetables, which means
there is no isolation between clients, but only between GPU assigned memory
spaces and the rest of the system. Better than nothing, but also not great.
2. Clients operating on the same set of buffers with different etnaviv GPU
cores, e.g. a workload using both the 2D and 3D GPU, need to map the used
buffers into the pagetable sets of each used GPU.
This patch reworks all the MMU handling to introduce the abstraction of the
MMU context. A context can be shared across different GPU cores, as long as
they have compatible MMU implementations, which is the case for all systems
with Vivante GPUs seen in the wild.
As MMUv1 is not able to change pagetables on the fly, without a
"stop the world" operation, which stops GPU, changes pagetables via CPU
interaction, restarts GPU, the implementation introduces a shared context on
MMUv1, which is returned whenever there is a request for a new context.
This patch assigns a MMU context to each GPU, so on MMUv2 systems there is
still one set of pagetables per GPU, but due to the shared context MMUv1
systems see a change in behavior as now a single pagetable set is used
across all GPU cores.
Signed-off-by: Lucas Stach <l.stach@pengutronix.de>
Reviewed-by: Philipp Zabel <p.zabel@pengutronix.de>
Reviewed-by: Guido Günther <agx@sigxcpu.org>
2019-07-05 19:17:24 +02:00
|
|
|
unmapped_page = context->global->ops->unmap(context, iova,
|
|
|
|
|
pgsize);
|
2017-09-07 16:52:13 +02:00
|
|
|
if (!unmapped_page)
|
|
|
|
|
break;
|
|
|
|
|
|
|
|
|
|
iova += unmapped_page;
|
|
|
|
|
unmapped += unmapped_page;
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
drm/etnaviv: rework MMU handling
This reworks the MMU handling to make it possible to have multiple MMU contexts.
A context is basically one instance of GPU page tables. Currently we have one
set of page tables per GPU, which isn't all that clever, as it has the
following two consequences:
1. All GPU clients (aka processes) are sharing the same pagetables, which means
there is no isolation between clients, but only between GPU assigned memory
spaces and the rest of the system. Better than nothing, but also not great.
2. Clients operating on the same set of buffers with different etnaviv GPU
cores, e.g. a workload using both the 2D and 3D GPU, need to map the used
buffers into the pagetable sets of each used GPU.
This patch reworks all the MMU handling to introduce the abstraction of the
MMU context. A context can be shared across different GPU cores, as long as
they have compatible MMU implementations, which is the case for all systems
with Vivante GPUs seen in the wild.
As MMUv1 is not able to change pagetables on the fly, without a
"stop the world" operation, which stops GPU, changes pagetables via CPU
interaction, restarts GPU, the implementation introduces a shared context on
MMUv1, which is returned whenever there is a request for a new context.
This patch assigns a MMU context to each GPU, so on MMUv2 systems there is
still one set of pagetables per GPU, but due to the shared context MMUv1
systems see a change in behavior as now a single pagetable set is used
across all GPU cores.
Signed-off-by: Lucas Stach <l.stach@pengutronix.de>
Reviewed-by: Philipp Zabel <p.zabel@pengutronix.de>
Reviewed-by: Guido Günther <agx@sigxcpu.org>
2019-07-05 19:17:24 +02:00
|
|
|
static int etnaviv_context_map(struct etnaviv_iommu_context *context,
|
2017-09-07 17:06:28 +02:00
|
|
|
unsigned long iova, phys_addr_t paddr,
|
|
|
|
|
size_t size, int prot)
|
2017-09-07 16:52:13 +02:00
|
|
|
{
|
|
|
|
|
unsigned long orig_iova = iova;
|
|
|
|
|
size_t pgsize = SZ_4K;
|
|
|
|
|
size_t orig_size = size;
|
|
|
|
|
int ret = 0;
|
|
|
|
|
|
|
|
|
|
while (size) {
|
drm/etnaviv: rework MMU handling
This reworks the MMU handling to make it possible to have multiple MMU contexts.
A context is basically one instance of GPU page tables. Currently we have one
set of page tables per GPU, which isn't all that clever, as it has the
following two consequences:
1. All GPU clients (aka processes) are sharing the same pagetables, which means
there is no isolation between clients, but only between GPU assigned memory
spaces and the rest of the system. Better than nothing, but also not great.
2. Clients operating on the same set of buffers with different etnaviv GPU
cores, e.g. a workload using both the 2D and 3D GPU, need to map the used
buffers into the pagetable sets of each used GPU.
This patch reworks all the MMU handling to introduce the abstraction of the
MMU context. A context can be shared across different GPU cores, as long as
they have compatible MMU implementations, which is the case for all systems
with Vivante GPUs seen in the wild.
As MMUv1 is not able to change pagetables on the fly, without a
"stop the world" operation, which stops GPU, changes pagetables via CPU
interaction, restarts GPU, the implementation introduces a shared context on
MMUv1, which is returned whenever there is a request for a new context.
This patch assigns a MMU context to each GPU, so on MMUv2 systems there is
still one set of pagetables per GPU, but due to the shared context MMUv1
systems see a change in behavior as now a single pagetable set is used
across all GPU cores.
Signed-off-by: Lucas Stach <l.stach@pengutronix.de>
Reviewed-by: Philipp Zabel <p.zabel@pengutronix.de>
Reviewed-by: Guido Günther <agx@sigxcpu.org>
2019-07-05 19:17:24 +02:00
|
|
|
ret = context->global->ops->map(context, iova, paddr, pgsize,
|
|
|
|
|
prot);
|
2017-09-07 16:52:13 +02:00
|
|
|
if (ret)
|
|
|
|
|
break;
|
|
|
|
|
|
|
|
|
|
iova += pgsize;
|
|
|
|
|
paddr += pgsize;
|
|
|
|
|
size -= pgsize;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/* unroll mapping in case something went wrong */
|
|
|
|
|
if (ret)
|
drm/etnaviv: rework MMU handling
This reworks the MMU handling to make it possible to have multiple MMU contexts.
A context is basically one instance of GPU page tables. Currently we have one
set of page tables per GPU, which isn't all that clever, as it has the
following two consequences:
1. All GPU clients (aka processes) are sharing the same pagetables, which means
there is no isolation between clients, but only between GPU assigned memory
spaces and the rest of the system. Better than nothing, but also not great.
2. Clients operating on the same set of buffers with different etnaviv GPU
cores, e.g. a workload using both the 2D and 3D GPU, need to map the used
buffers into the pagetable sets of each used GPU.
This patch reworks all the MMU handling to introduce the abstraction of the
MMU context. A context can be shared across different GPU cores, as long as
they have compatible MMU implementations, which is the case for all systems
with Vivante GPUs seen in the wild.
As MMUv1 is not able to change pagetables on the fly, without a
"stop the world" operation, which stops GPU, changes pagetables via CPU
interaction, restarts GPU, the implementation introduces a shared context on
MMUv1, which is returned whenever there is a request for a new context.
This patch assigns a MMU context to each GPU, so on MMUv2 systems there is
still one set of pagetables per GPU, but due to the shared context MMUv1
systems see a change in behavior as now a single pagetable set is used
across all GPU cores.
Signed-off-by: Lucas Stach <l.stach@pengutronix.de>
Reviewed-by: Philipp Zabel <p.zabel@pengutronix.de>
Reviewed-by: Guido Günther <agx@sigxcpu.org>
2019-07-05 19:17:24 +02:00
|
|
|
etnaviv_context_unmap(context, orig_iova, orig_size - size);
|
2017-09-07 16:52:13 +02:00
|
|
|
|
|
|
|
|
return ret;
|
|
|
|
|
}
|
|
|
|
|
|
2024-10-26 04:43:55 +08:00
|
|
|
static int etnaviv_iommu_map(struct etnaviv_iommu_context *context,
|
|
|
|
|
u32 iova, unsigned int va_len,
|
2023-10-07 15:03:12 +08:00
|
|
|
struct sg_table *sgt, int prot)
|
2024-10-26 04:43:55 +08:00
|
|
|
{
|
|
|
|
|
struct scatterlist *sg;
|
2015-12-03 18:21:29 +01:00
|
|
|
unsigned int da = iova;
|
2020-04-28 13:08:23 +02:00
|
|
|
unsigned int i;
|
2015-12-03 18:21:29 +01:00
|
|
|
int ret;
|
|
|
|
|
|
drm/etnaviv: rework MMU handling
This reworks the MMU handling to make it possible to have multiple MMU contexts.
A context is basically one instance of GPU page tables. Currently we have one
set of page tables per GPU, which isn't all that clever, as it has the
following two consequences:
1. All GPU clients (aka processes) are sharing the same pagetables, which means
there is no isolation between clients, but only between GPU assigned memory
spaces and the rest of the system. Better than nothing, but also not great.
2. Clients operating on the same set of buffers with different etnaviv GPU
cores, e.g. a workload using both the 2D and 3D GPU, need to map the used
buffers into the pagetable sets of each used GPU.
This patch reworks all the MMU handling to introduce the abstraction of the
MMU context. A context can be shared across different GPU cores, as long as
they have compatible MMU implementations, which is the case for all systems
with Vivante GPUs seen in the wild.
As MMUv1 is not able to change pagetables on the fly, without a
"stop the world" operation, which stops GPU, changes pagetables via CPU
interaction, restarts GPU, the implementation introduces a shared context on
MMUv1, which is returned whenever there is a request for a new context.
This patch assigns a MMU context to each GPU, so on MMUv2 systems there is
still one set of pagetables per GPU, but due to the shared context MMUv1
systems see a change in behavior as now a single pagetable set is used
across all GPU cores.
Signed-off-by: Lucas Stach <l.stach@pengutronix.de>
Reviewed-by: Philipp Zabel <p.zabel@pengutronix.de>
Reviewed-by: Guido Günther <agx@sigxcpu.org>
2019-07-05 19:17:24 +02:00
|
|
|
if (!context || !sgt)
|
2015-12-03 18:21:29 +01:00
|
|
|
return -EINVAL;
|
|
|
|
|
|
2020-04-28 13:08:23 +02:00
|
|
|
for_each_sgtable_dma_sg(sgt, sg, i) {
|
drm/etnaviv: Drop the offset in page manipulation
The etnaviv driver, both kernel space and user space, assumes that GPU page
size is 4KiB. Its IOMMU map/unmap 4KiB physical address range once a time.
If 'sg->offset != 0' is true, then the current implementation will map the
IOVA to a wrong area, which may lead to coherency problem. Picture 0 and 1
give the illustration, see below.
PA start drifted
|
|<--- 'sg_dma_address(sg) - sg->offset'
| .------ sg_dma_address(sg)
| | .---- sg_dma_len(sg)
|<-sg->offset->| |
V |<-->| Another one cpu page
+----+----+----+----+ +----+----+----+----+
|xxxx| |||||| |||||||||||||||||||||
+----+----+----+----+ +----+----+----+----+
^ ^ ^ ^
|<--- da_len --->| | |
| | | |
| .--------------' | |
| | .----------------' |
| | | .----------------'
| | | |
| | +----+----+----+----+
| | |||||||||||||||||||||
| | +----+----+----+----+
| |
| '--------------. da_len = sg_dma_len(sg) + sg->offset, using
| | 'sg_dma_len(sg) + sg->offset' will lead to GPUVA
+----+ ~~~~~~~~~~~~~+ collision, but min_t(unsigned int, da_len, va_len)
|xxxx| | will clamp it to correct size. But the IOVA will
+----+ ~~~~~~~~~~~~~+ be redirect to wrong area.
^
| Picture 0: Possibly wrong implementation.
GPUVA (IOVA)
--------------------------------------------------------------------------
.------- sg_dma_address(sg)
| .---- sg_dma_len(sg)
|<-sg->offset->| |
| |<-->| another one cpu page
+----+----+----+----+ +----+----+----+----+
| |||||| |||||||||||||||||||||
+----+----+----+----+ +----+----+----+----+
^ ^ ^ ^
| | | |
.--------------' | | |
| | | |
| .--------------' | |
| | .----------------' |
| | | .----------------'
| | | |
+----+ +----+----+----+----+
|||||| ||||||||||||||||||||| The first one is SZ_4K, the second is SZ_16K
+----+ +----+----+----+----+
^
| Picture 1: Perfectly correct implementation.
GPUVA (IOVA)
If sg->offset != 0 is true, IOVA will be mapped to wrong physical address.
Either because there doesn't contain the data or there contains wrong data.
Strictly speaking, the memory area that before sg_dma_address(sg) doesn't
belong to us, and it's likely that the area is being used by other process.
Because we don't want to introduce confusions about which part is visible
to the GPU, we assumes that the size of GPUVA is always 4KiB aligned. This
is very relaxed requirement, since we already made the decision that GPU
page size is 4KiB (as a canonical decision). And softpin feature is landed,
Mesa's util_vma_heap_alloc() will certainly report correct length of GPUVA
to kernel with desired alignment ensured.
With above statements agreed, drop the "offset in page" manipulation will
return us a correct implementation at any case.
Fixes: a8c21a5451d8 ("drm/etnaviv: add initial etnaviv DRM driver")
Signed-off-by: Sui Jingfeng <sui.jingfeng@linux.dev>
Signed-off-by: Lucas Stach <l.stach@pengutronix.de>
2024-11-15 20:32:44 +08:00
|
|
|
phys_addr_t pa = sg_dma_address(sg);
|
|
|
|
|
unsigned int da_len = sg_dma_len(sg);
|
2024-10-26 04:43:55 +08:00
|
|
|
unsigned int bytes = min_t(unsigned int, da_len, va_len);
|
2015-12-03 18:21:29 +01:00
|
|
|
|
2024-11-15 20:32:45 +08:00
|
|
|
VERB("map[%d]: %08x %pap(%x)", i, da, &pa, bytes);
|
2015-12-03 18:21:29 +01:00
|
|
|
|
2024-11-15 20:32:46 +08:00
|
|
|
if (!IS_ALIGNED(iova | pa | bytes, SZ_4K)) {
|
|
|
|
|
dev_err(context->global->dev,
|
|
|
|
|
"unaligned: iova 0x%x pa %pa size 0x%x\n",
|
|
|
|
|
iova, &pa, bytes);
|
|
|
|
|
ret = -EINVAL;
|
|
|
|
|
goto fail;
|
|
|
|
|
}
|
|
|
|
|
|
drm/etnaviv: rework MMU handling
This reworks the MMU handling to make it possible to have multiple MMU contexts.
A context is basically one instance of GPU page tables. Currently we have one
set of page tables per GPU, which isn't all that clever, as it has the
following two consequences:
1. All GPU clients (aka processes) are sharing the same pagetables, which means
there is no isolation between clients, but only between GPU assigned memory
spaces and the rest of the system. Better than nothing, but also not great.
2. Clients operating on the same set of buffers with different etnaviv GPU
cores, e.g. a workload using both the 2D and 3D GPU, need to map the used
buffers into the pagetable sets of each used GPU.
This patch reworks all the MMU handling to introduce the abstraction of the
MMU context. A context can be shared across different GPU cores, as long as
they have compatible MMU implementations, which is the case for all systems
with Vivante GPUs seen in the wild.
As MMUv1 is not able to change pagetables on the fly, without a
"stop the world" operation, which stops GPU, changes pagetables via CPU
interaction, restarts GPU, the implementation introduces a shared context on
MMUv1, which is returned whenever there is a request for a new context.
This patch assigns a MMU context to each GPU, so on MMUv2 systems there is
still one set of pagetables per GPU, but due to the shared context MMUv1
systems see a change in behavior as now a single pagetable set is used
across all GPU cores.
Signed-off-by: Lucas Stach <l.stach@pengutronix.de>
Reviewed-by: Philipp Zabel <p.zabel@pengutronix.de>
Reviewed-by: Guido Günther <agx@sigxcpu.org>
2019-07-05 19:17:24 +02:00
|
|
|
ret = etnaviv_context_map(context, da, pa, bytes, prot);
|
2015-12-03 18:21:29 +01:00
|
|
|
if (ret)
|
|
|
|
|
goto fail;
|
|
|
|
|
|
2024-10-26 04:43:55 +08:00
|
|
|
va_len -= bytes;
|
2015-12-03 18:21:29 +01:00
|
|
|
da += bytes;
|
|
|
|
|
}
|
|
|
|
|
|
2022-03-23 17:08:24 +01:00
|
|
|
context->flush_seq++;
|
|
|
|
|
|
2015-12-03 18:21:29 +01:00
|
|
|
return 0;
|
|
|
|
|
|
|
|
|
|
fail:
|
2020-04-28 13:08:23 +02:00
|
|
|
etnaviv_context_unmap(context, iova, da - iova);
|
2015-12-03 18:21:29 +01:00
|
|
|
return ret;
|
|
|
|
|
}
|
|
|
|
|
|
drm/etnaviv: rework MMU handling
This reworks the MMU handling to make it possible to have multiple MMU contexts.
A context is basically one instance of GPU page tables. Currently we have one
set of page tables per GPU, which isn't all that clever, as it has the
following two consequences:
1. All GPU clients (aka processes) are sharing the same pagetables, which means
there is no isolation between clients, but only between GPU assigned memory
spaces and the rest of the system. Better than nothing, but also not great.
2. Clients operating on the same set of buffers with different etnaviv GPU
cores, e.g. a workload using both the 2D and 3D GPU, need to map the used
buffers into the pagetable sets of each used GPU.
This patch reworks all the MMU handling to introduce the abstraction of the
MMU context. A context can be shared across different GPU cores, as long as
they have compatible MMU implementations, which is the case for all systems
with Vivante GPUs seen in the wild.
As MMUv1 is not able to change pagetables on the fly, without a
"stop the world" operation, which stops GPU, changes pagetables via CPU
interaction, restarts GPU, the implementation introduces a shared context on
MMUv1, which is returned whenever there is a request for a new context.
This patch assigns a MMU context to each GPU, so on MMUv2 systems there is
still one set of pagetables per GPU, but due to the shared context MMUv1
systems see a change in behavior as now a single pagetable set is used
across all GPU cores.
Signed-off-by: Lucas Stach <l.stach@pengutronix.de>
Reviewed-by: Philipp Zabel <p.zabel@pengutronix.de>
Reviewed-by: Guido Günther <agx@sigxcpu.org>
2019-07-05 19:17:24 +02:00
|
|
|
static void etnaviv_iommu_unmap(struct etnaviv_iommu_context *context, u32 iova,
|
2017-09-07 17:06:28 +02:00
|
|
|
struct sg_table *sgt, unsigned len)
|
2015-12-03 18:21:29 +01:00
|
|
|
{
|
2024-10-26 04:43:55 +08:00
|
|
|
etnaviv_context_unmap(context, iova, len);
|
2022-03-23 17:08:24 +01:00
|
|
|
|
|
|
|
|
context->flush_seq++;
|
2015-12-03 18:21:29 +01:00
|
|
|
}
|
|
|
|
|
|
drm/etnaviv: rework MMU handling
This reworks the MMU handling to make it possible to have multiple MMU contexts.
A context is basically one instance of GPU page tables. Currently we have one
set of page tables per GPU, which isn't all that clever, as it has the
following two consequences:
1. All GPU clients (aka processes) are sharing the same pagetables, which means
there is no isolation between clients, but only between GPU assigned memory
spaces and the rest of the system. Better than nothing, but also not great.
2. Clients operating on the same set of buffers with different etnaviv GPU
cores, e.g. a workload using both the 2D and 3D GPU, need to map the used
buffers into the pagetable sets of each used GPU.
This patch reworks all the MMU handling to introduce the abstraction of the
MMU context. A context can be shared across different GPU cores, as long as
they have compatible MMU implementations, which is the case for all systems
with Vivante GPUs seen in the wild.
As MMUv1 is not able to change pagetables on the fly, without a
"stop the world" operation, which stops GPU, changes pagetables via CPU
interaction, restarts GPU, the implementation introduces a shared context on
MMUv1, which is returned whenever there is a request for a new context.
This patch assigns a MMU context to each GPU, so on MMUv2 systems there is
still one set of pagetables per GPU, but due to the shared context MMUv1
systems see a change in behavior as now a single pagetable set is used
across all GPU cores.
Signed-off-by: Lucas Stach <l.stach@pengutronix.de>
Reviewed-by: Philipp Zabel <p.zabel@pengutronix.de>
Reviewed-by: Guido Günther <agx@sigxcpu.org>
2019-07-05 19:17:24 +02:00
|
|
|
static void etnaviv_iommu_remove_mapping(struct etnaviv_iommu_context *context,
|
2015-12-03 18:21:29 +01:00
|
|
|
struct etnaviv_vram_mapping *mapping)
|
|
|
|
|
{
|
|
|
|
|
struct etnaviv_gem_object *etnaviv_obj = mapping->object;
|
|
|
|
|
|
2020-10-29 15:20:56 +01:00
|
|
|
lockdep_assert_held(&context->lock);
|
|
|
|
|
|
drm/etnaviv: rework MMU handling
This reworks the MMU handling to make it possible to have multiple MMU contexts.
A context is basically one instance of GPU page tables. Currently we have one
set of page tables per GPU, which isn't all that clever, as it has the
following two consequences:
1. All GPU clients (aka processes) are sharing the same pagetables, which means
there is no isolation between clients, but only between GPU assigned memory
spaces and the rest of the system. Better than nothing, but also not great.
2. Clients operating on the same set of buffers with different etnaviv GPU
cores, e.g. a workload using both the 2D and 3D GPU, need to map the used
buffers into the pagetable sets of each used GPU.
This patch reworks all the MMU handling to introduce the abstraction of the
MMU context. A context can be shared across different GPU cores, as long as
they have compatible MMU implementations, which is the case for all systems
with Vivante GPUs seen in the wild.
As MMUv1 is not able to change pagetables on the fly, without a
"stop the world" operation, which stops GPU, changes pagetables via CPU
interaction, restarts GPU, the implementation introduces a shared context on
MMUv1, which is returned whenever there is a request for a new context.
This patch assigns a MMU context to each GPU, so on MMUv2 systems there is
still one set of pagetables per GPU, but due to the shared context MMUv1
systems see a change in behavior as now a single pagetable set is used
across all GPU cores.
Signed-off-by: Lucas Stach <l.stach@pengutronix.de>
Reviewed-by: Philipp Zabel <p.zabel@pengutronix.de>
Reviewed-by: Guido Günther <agx@sigxcpu.org>
2019-07-05 19:17:24 +02:00
|
|
|
etnaviv_iommu_unmap(context, mapping->vram_node.start,
|
2024-10-26 04:43:55 +08:00
|
|
|
etnaviv_obj->sgt, etnaviv_obj->size);
|
2015-12-03 18:21:29 +01:00
|
|
|
drm_mm_remove_node(&mapping->vram_node);
|
|
|
|
|
}
|
|
|
|
|
|
2022-07-14 12:31:42 +02:00
|
|
|
void etnaviv_iommu_reap_mapping(struct etnaviv_vram_mapping *mapping)
|
|
|
|
|
{
|
|
|
|
|
struct etnaviv_iommu_context *context = mapping->context;
|
|
|
|
|
|
|
|
|
|
lockdep_assert_held(&context->lock);
|
|
|
|
|
WARN_ON(mapping->use);
|
|
|
|
|
|
|
|
|
|
etnaviv_iommu_remove_mapping(context, mapping);
|
|
|
|
|
etnaviv_iommu_context_put(mapping->context);
|
|
|
|
|
mapping->context = NULL;
|
|
|
|
|
list_del_init(&mapping->mmu_node);
|
|
|
|
|
}
|
|
|
|
|
|
drm/etnaviv: rework MMU handling
This reworks the MMU handling to make it possible to have multiple MMU contexts.
A context is basically one instance of GPU page tables. Currently we have one
set of page tables per GPU, which isn't all that clever, as it has the
following two consequences:
1. All GPU clients (aka processes) are sharing the same pagetables, which means
there is no isolation between clients, but only between GPU assigned memory
spaces and the rest of the system. Better than nothing, but also not great.
2. Clients operating on the same set of buffers with different etnaviv GPU
cores, e.g. a workload using both the 2D and 3D GPU, need to map the used
buffers into the pagetable sets of each used GPU.
This patch reworks all the MMU handling to introduce the abstraction of the
MMU context. A context can be shared across different GPU cores, as long as
they have compatible MMU implementations, which is the case for all systems
with Vivante GPUs seen in the wild.
As MMUv1 is not able to change pagetables on the fly, without a
"stop the world" operation, which stops GPU, changes pagetables via CPU
interaction, restarts GPU, the implementation introduces a shared context on
MMUv1, which is returned whenever there is a request for a new context.
This patch assigns a MMU context to each GPU, so on MMUv2 systems there is
still one set of pagetables per GPU, but due to the shared context MMUv1
systems see a change in behavior as now a single pagetable set is used
across all GPU cores.
Signed-off-by: Lucas Stach <l.stach@pengutronix.de>
Reviewed-by: Philipp Zabel <p.zabel@pengutronix.de>
Reviewed-by: Guido Günther <agx@sigxcpu.org>
2019-07-05 19:17:24 +02:00
|
|
|
static int etnaviv_iommu_find_iova(struct etnaviv_iommu_context *context,
|
2016-08-19 23:43:40 +02:00
|
|
|
struct drm_mm_node *node, size_t size)
|
2015-12-03 18:21:29 +01:00
|
|
|
{
|
|
|
|
|
struct etnaviv_vram_mapping *free = NULL;
|
2017-02-02 21:04:38 +00:00
|
|
|
enum drm_mm_insert_mode mode = DRM_MM_INSERT_LOW;
|
2015-12-03 18:21:29 +01:00
|
|
|
int ret;
|
|
|
|
|
|
drm/etnaviv: rework MMU handling
This reworks the MMU handling to make it possible to have multiple MMU contexts.
A context is basically one instance of GPU page tables. Currently we have one
set of page tables per GPU, which isn't all that clever, as it has the
following two consequences:
1. All GPU clients (aka processes) are sharing the same pagetables, which means
there is no isolation between clients, but only between GPU assigned memory
spaces and the rest of the system. Better than nothing, but also not great.
2. Clients operating on the same set of buffers with different etnaviv GPU
cores, e.g. a workload using both the 2D and 3D GPU, need to map the used
buffers into the pagetable sets of each used GPU.
This patch reworks all the MMU handling to introduce the abstraction of the
MMU context. A context can be shared across different GPU cores, as long as
they have compatible MMU implementations, which is the case for all systems
with Vivante GPUs seen in the wild.
As MMUv1 is not able to change pagetables on the fly, without a
"stop the world" operation, which stops GPU, changes pagetables via CPU
interaction, restarts GPU, the implementation introduces a shared context on
MMUv1, which is returned whenever there is a request for a new context.
This patch assigns a MMU context to each GPU, so on MMUv2 systems there is
still one set of pagetables per GPU, but due to the shared context MMUv1
systems see a change in behavior as now a single pagetable set is used
across all GPU cores.
Signed-off-by: Lucas Stach <l.stach@pengutronix.de>
Reviewed-by: Philipp Zabel <p.zabel@pengutronix.de>
Reviewed-by: Guido Günther <agx@sigxcpu.org>
2019-07-05 19:17:24 +02:00
|
|
|
lockdep_assert_held(&context->lock);
|
2015-12-03 18:21:29 +01:00
|
|
|
|
|
|
|
|
while (1) {
|
|
|
|
|
struct etnaviv_vram_mapping *m, *n;
|
2016-12-22 08:36:29 +00:00
|
|
|
struct drm_mm_scan scan;
|
2015-12-03 18:21:29 +01:00
|
|
|
struct list_head list;
|
|
|
|
|
bool found;
|
|
|
|
|
|
drm/etnaviv: rework MMU handling
This reworks the MMU handling to make it possible to have multiple MMU contexts.
A context is basically one instance of GPU page tables. Currently we have one
set of page tables per GPU, which isn't all that clever, as it has the
following two consequences:
1. All GPU clients (aka processes) are sharing the same pagetables, which means
there is no isolation between clients, but only between GPU assigned memory
spaces and the rest of the system. Better than nothing, but also not great.
2. Clients operating on the same set of buffers with different etnaviv GPU
cores, e.g. a workload using both the 2D and 3D GPU, need to map the used
buffers into the pagetable sets of each used GPU.
This patch reworks all the MMU handling to introduce the abstraction of the
MMU context. A context can be shared across different GPU cores, as long as
they have compatible MMU implementations, which is the case for all systems
with Vivante GPUs seen in the wild.
As MMUv1 is not able to change pagetables on the fly, without a
"stop the world" operation, which stops GPU, changes pagetables via CPU
interaction, restarts GPU, the implementation introduces a shared context on
MMUv1, which is returned whenever there is a request for a new context.
This patch assigns a MMU context to each GPU, so on MMUv2 systems there is
still one set of pagetables per GPU, but due to the shared context MMUv1
systems see a change in behavior as now a single pagetable set is used
across all GPU cores.
Signed-off-by: Lucas Stach <l.stach@pengutronix.de>
Reviewed-by: Philipp Zabel <p.zabel@pengutronix.de>
Reviewed-by: Guido Günther <agx@sigxcpu.org>
2019-07-05 19:17:24 +02:00
|
|
|
ret = drm_mm_insert_node_in_range(&context->mm, node,
|
2018-03-09 12:53:34 +01:00
|
|
|
size, 0, 0, 0, U64_MAX, mode);
|
2015-12-03 18:21:29 +01:00
|
|
|
if (ret != -ENOSPC)
|
|
|
|
|
break;
|
|
|
|
|
|
|
|
|
|
/* Try to retire some entries */
|
drm/etnaviv: rework MMU handling
This reworks the MMU handling to make it possible to have multiple MMU contexts.
A context is basically one instance of GPU page tables. Currently we have one
set of page tables per GPU, which isn't all that clever, as it has the
following two consequences:
1. All GPU clients (aka processes) are sharing the same pagetables, which means
there is no isolation between clients, but only between GPU assigned memory
spaces and the rest of the system. Better than nothing, but also not great.
2. Clients operating on the same set of buffers with different etnaviv GPU
cores, e.g. a workload using both the 2D and 3D GPU, need to map the used
buffers into the pagetable sets of each used GPU.
This patch reworks all the MMU handling to introduce the abstraction of the
MMU context. A context can be shared across different GPU cores, as long as
they have compatible MMU implementations, which is the case for all systems
with Vivante GPUs seen in the wild.
As MMUv1 is not able to change pagetables on the fly, without a
"stop the world" operation, which stops GPU, changes pagetables via CPU
interaction, restarts GPU, the implementation introduces a shared context on
MMUv1, which is returned whenever there is a request for a new context.
This patch assigns a MMU context to each GPU, so on MMUv2 systems there is
still one set of pagetables per GPU, but due to the shared context MMUv1
systems see a change in behavior as now a single pagetable set is used
across all GPU cores.
Signed-off-by: Lucas Stach <l.stach@pengutronix.de>
Reviewed-by: Philipp Zabel <p.zabel@pengutronix.de>
Reviewed-by: Guido Günther <agx@sigxcpu.org>
2019-07-05 19:17:24 +02:00
|
|
|
drm_mm_scan_init(&scan, &context->mm, size, 0, 0, mode);
|
2015-12-03 18:21:29 +01:00
|
|
|
|
|
|
|
|
found = 0;
|
|
|
|
|
INIT_LIST_HEAD(&list);
|
drm/etnaviv: rework MMU handling
This reworks the MMU handling to make it possible to have multiple MMU contexts.
A context is basically one instance of GPU page tables. Currently we have one
set of page tables per GPU, which isn't all that clever, as it has the
following two consequences:
1. All GPU clients (aka processes) are sharing the same pagetables, which means
there is no isolation between clients, but only between GPU assigned memory
spaces and the rest of the system. Better than nothing, but also not great.
2. Clients operating on the same set of buffers with different etnaviv GPU
cores, e.g. a workload using both the 2D and 3D GPU, need to map the used
buffers into the pagetable sets of each used GPU.
This patch reworks all the MMU handling to introduce the abstraction of the
MMU context. A context can be shared across different GPU cores, as long as
they have compatible MMU implementations, which is the case for all systems
with Vivante GPUs seen in the wild.
As MMUv1 is not able to change pagetables on the fly, without a
"stop the world" operation, which stops GPU, changes pagetables via CPU
interaction, restarts GPU, the implementation introduces a shared context on
MMUv1, which is returned whenever there is a request for a new context.
This patch assigns a MMU context to each GPU, so on MMUv2 systems there is
still one set of pagetables per GPU, but due to the shared context MMUv1
systems see a change in behavior as now a single pagetable set is used
across all GPU cores.
Signed-off-by: Lucas Stach <l.stach@pengutronix.de>
Reviewed-by: Philipp Zabel <p.zabel@pengutronix.de>
Reviewed-by: Guido Günther <agx@sigxcpu.org>
2019-07-05 19:17:24 +02:00
|
|
|
list_for_each_entry(free, &context->mappings, mmu_node) {
|
2015-12-03 18:21:29 +01:00
|
|
|
/* If this vram node has not been used, skip this. */
|
|
|
|
|
if (!free->vram_node.mm)
|
|
|
|
|
continue;
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* If the iova is pinned, then it's in-use,
|
|
|
|
|
* so we must keep its mapping.
|
|
|
|
|
*/
|
|
|
|
|
if (free->use)
|
|
|
|
|
continue;
|
|
|
|
|
|
|
|
|
|
list_add(&free->scan_node, &list);
|
2016-12-22 08:36:29 +00:00
|
|
|
if (drm_mm_scan_add_block(&scan, &free->vram_node)) {
|
2015-12-03 18:21:29 +01:00
|
|
|
found = true;
|
|
|
|
|
break;
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
if (!found) {
|
|
|
|
|
/* Nothing found, clean up and fail */
|
|
|
|
|
list_for_each_entry_safe(m, n, &list, scan_node)
|
2016-12-22 08:36:29 +00:00
|
|
|
BUG_ON(drm_mm_scan_remove_block(&scan, &m->vram_node));
|
2015-12-03 18:21:29 +01:00
|
|
|
break;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* drm_mm does not allow any other operations while
|
|
|
|
|
* scanning, so we have to remove all blocks first.
|
|
|
|
|
* If drm_mm_scan_remove_block() returns false, we
|
|
|
|
|
* can leave the block pinned.
|
|
|
|
|
*/
|
|
|
|
|
list_for_each_entry_safe(m, n, &list, scan_node)
|
2016-12-22 08:36:29 +00:00
|
|
|
if (!drm_mm_scan_remove_block(&scan, &m->vram_node))
|
2015-12-03 18:21:29 +01:00
|
|
|
list_del_init(&m->scan_node);
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Unmap the blocks which need to be reaped from the MMU.
|
2016-01-21 15:20:50 +00:00
|
|
|
* Clear the mmu pointer to prevent the mapping_get finding
|
2015-12-03 18:21:29 +01:00
|
|
|
* this mapping.
|
|
|
|
|
*/
|
|
|
|
|
list_for_each_entry_safe(m, n, &list, scan_node) {
|
2022-07-14 12:31:42 +02:00
|
|
|
etnaviv_iommu_reap_mapping(m);
|
2015-12-03 18:21:29 +01:00
|
|
|
list_del_init(&m->scan_node);
|
|
|
|
|
}
|
|
|
|
|
|
2017-02-02 21:04:38 +00:00
|
|
|
mode = DRM_MM_INSERT_EVICT;
|
|
|
|
|
|
2015-12-03 18:21:29 +01:00
|
|
|
/*
|
|
|
|
|
* We removed enough mappings so that the new allocation will
|
2017-01-17 10:59:37 +01:00
|
|
|
* succeed, retry the allocation one more time.
|
2015-12-03 18:21:29 +01:00
|
|
|
*/
|
|
|
|
|
}
|
|
|
|
|
|
2016-08-19 23:43:40 +02:00
|
|
|
return ret;
|
|
|
|
|
}
|
|
|
|
|
|
2019-08-02 14:19:38 +02:00
|
|
|
static int etnaviv_iommu_insert_exact(struct etnaviv_iommu_context *context,
|
|
|
|
|
struct drm_mm_node *node, size_t size, u64 va)
|
|
|
|
|
{
|
2022-03-23 17:08:25 +01:00
|
|
|
struct etnaviv_vram_mapping *m, *n;
|
|
|
|
|
struct drm_mm_node *scan_node;
|
|
|
|
|
LIST_HEAD(scan_list);
|
|
|
|
|
int ret;
|
|
|
|
|
|
2020-10-29 15:20:56 +01:00
|
|
|
lockdep_assert_held(&context->lock);
|
|
|
|
|
|
2022-03-23 17:08:25 +01:00
|
|
|
ret = drm_mm_insert_node_in_range(&context->mm, node, size, 0, 0, va,
|
|
|
|
|
va + size, DRM_MM_INSERT_LOWEST);
|
|
|
|
|
if (ret != -ENOSPC)
|
|
|
|
|
return ret;
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* When we can't insert the node, due to a existing mapping blocking
|
|
|
|
|
* the address space, there are two possible reasons:
|
|
|
|
|
* 1. Userspace genuinely messed up and tried to reuse address space
|
|
|
|
|
* before the last job using this VMA has finished executing.
|
|
|
|
|
* 2. The existing buffer mappings are idle, but the buffers are not
|
|
|
|
|
* destroyed yet (likely due to being referenced by another context) in
|
|
|
|
|
* which case the mappings will not be cleaned up and we must reap them
|
|
|
|
|
* here to make space for the new mapping.
|
|
|
|
|
*/
|
|
|
|
|
|
|
|
|
|
drm_mm_for_each_node_in_range(scan_node, &context->mm, va, va + size) {
|
|
|
|
|
m = container_of(scan_node, struct etnaviv_vram_mapping,
|
|
|
|
|
vram_node);
|
|
|
|
|
|
|
|
|
|
if (m->use)
|
|
|
|
|
return -ENOSPC;
|
|
|
|
|
|
|
|
|
|
list_add(&m->scan_node, &scan_list);
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
list_for_each_entry_safe(m, n, &scan_list, scan_node) {
|
2022-07-14 12:31:42 +02:00
|
|
|
etnaviv_iommu_reap_mapping(m);
|
2022-03-23 17:08:25 +01:00
|
|
|
list_del_init(&m->scan_node);
|
|
|
|
|
}
|
|
|
|
|
|
2019-08-02 14:19:38 +02:00
|
|
|
return drm_mm_insert_node_in_range(&context->mm, node, size, 0, 0, va,
|
|
|
|
|
va + size, DRM_MM_INSERT_LOWEST);
|
|
|
|
|
}
|
|
|
|
|
|
drm/etnaviv: rework MMU handling
This reworks the MMU handling to make it possible to have multiple MMU contexts.
A context is basically one instance of GPU page tables. Currently we have one
set of page tables per GPU, which isn't all that clever, as it has the
following two consequences:
1. All GPU clients (aka processes) are sharing the same pagetables, which means
there is no isolation between clients, but only between GPU assigned memory
spaces and the rest of the system. Better than nothing, but also not great.
2. Clients operating on the same set of buffers with different etnaviv GPU
cores, e.g. a workload using both the 2D and 3D GPU, need to map the used
buffers into the pagetable sets of each used GPU.
This patch reworks all the MMU handling to introduce the abstraction of the
MMU context. A context can be shared across different GPU cores, as long as
they have compatible MMU implementations, which is the case for all systems
with Vivante GPUs seen in the wild.
As MMUv1 is not able to change pagetables on the fly, without a
"stop the world" operation, which stops GPU, changes pagetables via CPU
interaction, restarts GPU, the implementation introduces a shared context on
MMUv1, which is returned whenever there is a request for a new context.
This patch assigns a MMU context to each GPU, so on MMUv2 systems there is
still one set of pagetables per GPU, but due to the shared context MMUv1
systems see a change in behavior as now a single pagetable set is used
across all GPU cores.
Signed-off-by: Lucas Stach <l.stach@pengutronix.de>
Reviewed-by: Philipp Zabel <p.zabel@pengutronix.de>
Reviewed-by: Guido Günther <agx@sigxcpu.org>
2019-07-05 19:17:24 +02:00
|
|
|
int etnaviv_iommu_map_gem(struct etnaviv_iommu_context *context,
|
2016-08-19 23:43:40 +02:00
|
|
|
struct etnaviv_gem_object *etnaviv_obj, u32 memory_base,
|
2019-08-02 14:19:38 +02:00
|
|
|
struct etnaviv_vram_mapping *mapping, u64 va)
|
2016-08-19 23:43:40 +02:00
|
|
|
{
|
|
|
|
|
struct sg_table *sgt = etnaviv_obj->sgt;
|
|
|
|
|
struct drm_mm_node *node;
|
|
|
|
|
int ret;
|
|
|
|
|
|
|
|
|
|
lockdep_assert_held(&etnaviv_obj->lock);
|
|
|
|
|
|
drm/etnaviv: rework MMU handling
This reworks the MMU handling to make it possible to have multiple MMU contexts.
A context is basically one instance of GPU page tables. Currently we have one
set of page tables per GPU, which isn't all that clever, as it has the
following two consequences:
1. All GPU clients (aka processes) are sharing the same pagetables, which means
there is no isolation between clients, but only between GPU assigned memory
spaces and the rest of the system. Better than nothing, but also not great.
2. Clients operating on the same set of buffers with different etnaviv GPU
cores, e.g. a workload using both the 2D and 3D GPU, need to map the used
buffers into the pagetable sets of each used GPU.
This patch reworks all the MMU handling to introduce the abstraction of the
MMU context. A context can be shared across different GPU cores, as long as
they have compatible MMU implementations, which is the case for all systems
with Vivante GPUs seen in the wild.
As MMUv1 is not able to change pagetables on the fly, without a
"stop the world" operation, which stops GPU, changes pagetables via CPU
interaction, restarts GPU, the implementation introduces a shared context on
MMUv1, which is returned whenever there is a request for a new context.
This patch assigns a MMU context to each GPU, so on MMUv2 systems there is
still one set of pagetables per GPU, but due to the shared context MMUv1
systems see a change in behavior as now a single pagetable set is used
across all GPU cores.
Signed-off-by: Lucas Stach <l.stach@pengutronix.de>
Reviewed-by: Philipp Zabel <p.zabel@pengutronix.de>
Reviewed-by: Guido Günther <agx@sigxcpu.org>
2019-07-05 19:17:24 +02:00
|
|
|
mutex_lock(&context->lock);
|
2016-08-19 23:43:40 +02:00
|
|
|
|
|
|
|
|
/* v1 MMU can optimize single entry (contiguous) scatterlists */
|
drm/etnaviv: rework MMU handling
This reworks the MMU handling to make it possible to have multiple MMU contexts.
A context is basically one instance of GPU page tables. Currently we have one
set of page tables per GPU, which isn't all that clever, as it has the
following two consequences:
1. All GPU clients (aka processes) are sharing the same pagetables, which means
there is no isolation between clients, but only between GPU assigned memory
spaces and the rest of the system. Better than nothing, but also not great.
2. Clients operating on the same set of buffers with different etnaviv GPU
cores, e.g. a workload using both the 2D and 3D GPU, need to map the used
buffers into the pagetable sets of each used GPU.
This patch reworks all the MMU handling to introduce the abstraction of the
MMU context. A context can be shared across different GPU cores, as long as
they have compatible MMU implementations, which is the case for all systems
with Vivante GPUs seen in the wild.
As MMUv1 is not able to change pagetables on the fly, without a
"stop the world" operation, which stops GPU, changes pagetables via CPU
interaction, restarts GPU, the implementation introduces a shared context on
MMUv1, which is returned whenever there is a request for a new context.
This patch assigns a MMU context to each GPU, so on MMUv2 systems there is
still one set of pagetables per GPU, but due to the shared context MMUv1
systems see a change in behavior as now a single pagetable set is used
across all GPU cores.
Signed-off-by: Lucas Stach <l.stach@pengutronix.de>
Reviewed-by: Philipp Zabel <p.zabel@pengutronix.de>
Reviewed-by: Guido Günther <agx@sigxcpu.org>
2019-07-05 19:17:24 +02:00
|
|
|
if (context->global->version == ETNAVIV_IOMMU_V1 &&
|
2016-08-19 23:43:40 +02:00
|
|
|
sgt->nents == 1 && !(etnaviv_obj->flags & ETNA_BO_FORCE_MMU)) {
|
|
|
|
|
u32 iova;
|
|
|
|
|
|
|
|
|
|
iova = sg_dma_address(sgt->sgl) - memory_base;
|
|
|
|
|
if (iova < 0x80000000 - sg_dma_len(sgt->sgl)) {
|
|
|
|
|
mapping->iova = iova;
|
2022-03-23 17:08:23 +01:00
|
|
|
mapping->context = etnaviv_iommu_context_get(context);
|
drm/etnaviv: rework MMU handling
This reworks the MMU handling to make it possible to have multiple MMU contexts.
A context is basically one instance of GPU page tables. Currently we have one
set of page tables per GPU, which isn't all that clever, as it has the
following two consequences:
1. All GPU clients (aka processes) are sharing the same pagetables, which means
there is no isolation between clients, but only between GPU assigned memory
spaces and the rest of the system. Better than nothing, but also not great.
2. Clients operating on the same set of buffers with different etnaviv GPU
cores, e.g. a workload using both the 2D and 3D GPU, need to map the used
buffers into the pagetable sets of each used GPU.
This patch reworks all the MMU handling to introduce the abstraction of the
MMU context. A context can be shared across different GPU cores, as long as
they have compatible MMU implementations, which is the case for all systems
with Vivante GPUs seen in the wild.
As MMUv1 is not able to change pagetables on the fly, without a
"stop the world" operation, which stops GPU, changes pagetables via CPU
interaction, restarts GPU, the implementation introduces a shared context on
MMUv1, which is returned whenever there is a request for a new context.
This patch assigns a MMU context to each GPU, so on MMUv2 systems there is
still one set of pagetables per GPU, but due to the shared context MMUv1
systems see a change in behavior as now a single pagetable set is used
across all GPU cores.
Signed-off-by: Lucas Stach <l.stach@pengutronix.de>
Reviewed-by: Philipp Zabel <p.zabel@pengutronix.de>
Reviewed-by: Guido Günther <agx@sigxcpu.org>
2019-07-05 19:17:24 +02:00
|
|
|
list_add_tail(&mapping->mmu_node, &context->mappings);
|
2017-10-23 21:27:30 +02:00
|
|
|
ret = 0;
|
|
|
|
|
goto unlock;
|
2016-08-19 23:43:40 +02:00
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
node = &mapping->vram_node;
|
|
|
|
|
|
2019-08-02 14:19:38 +02:00
|
|
|
if (va)
|
2024-10-26 04:43:55 +08:00
|
|
|
ret = etnaviv_iommu_insert_exact(context, node, etnaviv_obj->size, va);
|
2019-08-02 14:19:38 +02:00
|
|
|
else
|
2024-10-26 04:43:55 +08:00
|
|
|
ret = etnaviv_iommu_find_iova(context, node, etnaviv_obj->size);
|
2017-10-23 21:27:30 +02:00
|
|
|
if (ret < 0)
|
|
|
|
|
goto unlock;
|
2015-12-03 18:21:29 +01:00
|
|
|
|
|
|
|
|
mapping->iova = node->start;
|
2024-10-26 04:43:55 +08:00
|
|
|
ret = etnaviv_iommu_map(context, node->start, etnaviv_obj->size, sgt,
|
2017-09-07 17:06:28 +02:00
|
|
|
ETNAVIV_PROT_READ | ETNAVIV_PROT_WRITE);
|
2015-12-03 18:21:29 +01:00
|
|
|
|
|
|
|
|
if (ret < 0) {
|
|
|
|
|
drm_mm_remove_node(node);
|
2017-10-23 21:27:30 +02:00
|
|
|
goto unlock;
|
2015-12-03 18:21:29 +01:00
|
|
|
}
|
|
|
|
|
|
2022-03-23 17:08:23 +01:00
|
|
|
mapping->context = etnaviv_iommu_context_get(context);
|
drm/etnaviv: rework MMU handling
This reworks the MMU handling to make it possible to have multiple MMU contexts.
A context is basically one instance of GPU page tables. Currently we have one
set of page tables per GPU, which isn't all that clever, as it has the
following two consequences:
1. All GPU clients (aka processes) are sharing the same pagetables, which means
there is no isolation between clients, but only between GPU assigned memory
spaces and the rest of the system. Better than nothing, but also not great.
2. Clients operating on the same set of buffers with different etnaviv GPU
cores, e.g. a workload using both the 2D and 3D GPU, need to map the used
buffers into the pagetable sets of each used GPU.
This patch reworks all the MMU handling to introduce the abstraction of the
MMU context. A context can be shared across different GPU cores, as long as
they have compatible MMU implementations, which is the case for all systems
with Vivante GPUs seen in the wild.
As MMUv1 is not able to change pagetables on the fly, without a
"stop the world" operation, which stops GPU, changes pagetables via CPU
interaction, restarts GPU, the implementation introduces a shared context on
MMUv1, which is returned whenever there is a request for a new context.
This patch assigns a MMU context to each GPU, so on MMUv2 systems there is
still one set of pagetables per GPU, but due to the shared context MMUv1
systems see a change in behavior as now a single pagetable set is used
across all GPU cores.
Signed-off-by: Lucas Stach <l.stach@pengutronix.de>
Reviewed-by: Philipp Zabel <p.zabel@pengutronix.de>
Reviewed-by: Guido Günther <agx@sigxcpu.org>
2019-07-05 19:17:24 +02:00
|
|
|
list_add_tail(&mapping->mmu_node, &context->mappings);
|
2017-10-23 21:27:30 +02:00
|
|
|
unlock:
|
drm/etnaviv: rework MMU handling
This reworks the MMU handling to make it possible to have multiple MMU contexts.
A context is basically one instance of GPU page tables. Currently we have one
set of page tables per GPU, which isn't all that clever, as it has the
following two consequences:
1. All GPU clients (aka processes) are sharing the same pagetables, which means
there is no isolation between clients, but only between GPU assigned memory
spaces and the rest of the system. Better than nothing, but also not great.
2. Clients operating on the same set of buffers with different etnaviv GPU
cores, e.g. a workload using both the 2D and 3D GPU, need to map the used
buffers into the pagetable sets of each used GPU.
This patch reworks all the MMU handling to introduce the abstraction of the
MMU context. A context can be shared across different GPU cores, as long as
they have compatible MMU implementations, which is the case for all systems
with Vivante GPUs seen in the wild.
As MMUv1 is not able to change pagetables on the fly, without a
"stop the world" operation, which stops GPU, changes pagetables via CPU
interaction, restarts GPU, the implementation introduces a shared context on
MMUv1, which is returned whenever there is a request for a new context.
This patch assigns a MMU context to each GPU, so on MMUv2 systems there is
still one set of pagetables per GPU, but due to the shared context MMUv1
systems see a change in behavior as now a single pagetable set is used
across all GPU cores.
Signed-off-by: Lucas Stach <l.stach@pengutronix.de>
Reviewed-by: Philipp Zabel <p.zabel@pengutronix.de>
Reviewed-by: Guido Günther <agx@sigxcpu.org>
2019-07-05 19:17:24 +02:00
|
|
|
mutex_unlock(&context->lock);
|
2015-12-03 18:21:29 +01:00
|
|
|
|
|
|
|
|
return ret;
|
|
|
|
|
}
|
|
|
|
|
|
drm/etnaviv: rework MMU handling
This reworks the MMU handling to make it possible to have multiple MMU contexts.
A context is basically one instance of GPU page tables. Currently we have one
set of page tables per GPU, which isn't all that clever, as it has the
following two consequences:
1. All GPU clients (aka processes) are sharing the same pagetables, which means
there is no isolation between clients, but only between GPU assigned memory
spaces and the rest of the system. Better than nothing, but also not great.
2. Clients operating on the same set of buffers with different etnaviv GPU
cores, e.g. a workload using both the 2D and 3D GPU, need to map the used
buffers into the pagetable sets of each used GPU.
This patch reworks all the MMU handling to introduce the abstraction of the
MMU context. A context can be shared across different GPU cores, as long as
they have compatible MMU implementations, which is the case for all systems
with Vivante GPUs seen in the wild.
As MMUv1 is not able to change pagetables on the fly, without a
"stop the world" operation, which stops GPU, changes pagetables via CPU
interaction, restarts GPU, the implementation introduces a shared context on
MMUv1, which is returned whenever there is a request for a new context.
This patch assigns a MMU context to each GPU, so on MMUv2 systems there is
still one set of pagetables per GPU, but due to the shared context MMUv1
systems see a change in behavior as now a single pagetable set is used
across all GPU cores.
Signed-off-by: Lucas Stach <l.stach@pengutronix.de>
Reviewed-by: Philipp Zabel <p.zabel@pengutronix.de>
Reviewed-by: Guido Günther <agx@sigxcpu.org>
2019-07-05 19:17:24 +02:00
|
|
|
void etnaviv_iommu_unmap_gem(struct etnaviv_iommu_context *context,
|
2015-12-03 18:21:29 +01:00
|
|
|
struct etnaviv_vram_mapping *mapping)
|
|
|
|
|
{
|
|
|
|
|
WARN_ON(mapping->use);
|
|
|
|
|
|
drm/etnaviv: rework MMU handling
This reworks the MMU handling to make it possible to have multiple MMU contexts.
A context is basically one instance of GPU page tables. Currently we have one
set of page tables per GPU, which isn't all that clever, as it has the
following two consequences:
1. All GPU clients (aka processes) are sharing the same pagetables, which means
there is no isolation between clients, but only between GPU assigned memory
spaces and the rest of the system. Better than nothing, but also not great.
2. Clients operating on the same set of buffers with different etnaviv GPU
cores, e.g. a workload using both the 2D and 3D GPU, need to map the used
buffers into the pagetable sets of each used GPU.
This patch reworks all the MMU handling to introduce the abstraction of the
MMU context. A context can be shared across different GPU cores, as long as
they have compatible MMU implementations, which is the case for all systems
with Vivante GPUs seen in the wild.
As MMUv1 is not able to change pagetables on the fly, without a
"stop the world" operation, which stops GPU, changes pagetables via CPU
interaction, restarts GPU, the implementation introduces a shared context on
MMUv1, which is returned whenever there is a request for a new context.
This patch assigns a MMU context to each GPU, so on MMUv2 systems there is
still one set of pagetables per GPU, but due to the shared context MMUv1
systems see a change in behavior as now a single pagetable set is used
across all GPU cores.
Signed-off-by: Lucas Stach <l.stach@pengutronix.de>
Reviewed-by: Philipp Zabel <p.zabel@pengutronix.de>
Reviewed-by: Guido Günther <agx@sigxcpu.org>
2019-07-05 19:17:24 +02:00
|
|
|
mutex_lock(&context->lock);
|
2015-12-03 18:21:29 +01:00
|
|
|
|
2022-03-23 17:08:22 +01:00
|
|
|
/* Bail if the mapping has been reaped by another thread */
|
|
|
|
|
if (!mapping->context) {
|
|
|
|
|
mutex_unlock(&context->lock);
|
|
|
|
|
return;
|
|
|
|
|
}
|
|
|
|
|
|
2015-12-03 18:21:29 +01:00
|
|
|
/* If the vram node is on the mm, unmap and remove the node */
|
drm/etnaviv: rework MMU handling
This reworks the MMU handling to make it possible to have multiple MMU contexts.
A context is basically one instance of GPU page tables. Currently we have one
set of page tables per GPU, which isn't all that clever, as it has the
following two consequences:
1. All GPU clients (aka processes) are sharing the same pagetables, which means
there is no isolation between clients, but only between GPU assigned memory
spaces and the rest of the system. Better than nothing, but also not great.
2. Clients operating on the same set of buffers with different etnaviv GPU
cores, e.g. a workload using both the 2D and 3D GPU, need to map the used
buffers into the pagetable sets of each used GPU.
This patch reworks all the MMU handling to introduce the abstraction of the
MMU context. A context can be shared across different GPU cores, as long as
they have compatible MMU implementations, which is the case for all systems
with Vivante GPUs seen in the wild.
As MMUv1 is not able to change pagetables on the fly, without a
"stop the world" operation, which stops GPU, changes pagetables via CPU
interaction, restarts GPU, the implementation introduces a shared context on
MMUv1, which is returned whenever there is a request for a new context.
This patch assigns a MMU context to each GPU, so on MMUv2 systems there is
still one set of pagetables per GPU, but due to the shared context MMUv1
systems see a change in behavior as now a single pagetable set is used
across all GPU cores.
Signed-off-by: Lucas Stach <l.stach@pengutronix.de>
Reviewed-by: Philipp Zabel <p.zabel@pengutronix.de>
Reviewed-by: Guido Günther <agx@sigxcpu.org>
2019-07-05 19:17:24 +02:00
|
|
|
if (mapping->vram_node.mm == &context->mm)
|
|
|
|
|
etnaviv_iommu_remove_mapping(context, mapping);
|
2015-12-03 18:21:29 +01:00
|
|
|
|
|
|
|
|
list_del(&mapping->mmu_node);
|
drm/etnaviv: rework MMU handling
This reworks the MMU handling to make it possible to have multiple MMU contexts.
A context is basically one instance of GPU page tables. Currently we have one
set of page tables per GPU, which isn't all that clever, as it has the
following two consequences:
1. All GPU clients (aka processes) are sharing the same pagetables, which means
there is no isolation between clients, but only between GPU assigned memory
spaces and the rest of the system. Better than nothing, but also not great.
2. Clients operating on the same set of buffers with different etnaviv GPU
cores, e.g. a workload using both the 2D and 3D GPU, need to map the used
buffers into the pagetable sets of each used GPU.
This patch reworks all the MMU handling to introduce the abstraction of the
MMU context. A context can be shared across different GPU cores, as long as
they have compatible MMU implementations, which is the case for all systems
with Vivante GPUs seen in the wild.
As MMUv1 is not able to change pagetables on the fly, without a
"stop the world" operation, which stops GPU, changes pagetables via CPU
interaction, restarts GPU, the implementation introduces a shared context on
MMUv1, which is returned whenever there is a request for a new context.
This patch assigns a MMU context to each GPU, so on MMUv2 systems there is
still one set of pagetables per GPU, but due to the shared context MMUv1
systems see a change in behavior as now a single pagetable set is used
across all GPU cores.
Signed-off-by: Lucas Stach <l.stach@pengutronix.de>
Reviewed-by: Philipp Zabel <p.zabel@pengutronix.de>
Reviewed-by: Guido Günther <agx@sigxcpu.org>
2019-07-05 19:17:24 +02:00
|
|
|
mutex_unlock(&context->lock);
|
2022-03-23 17:08:23 +01:00
|
|
|
etnaviv_iommu_context_put(context);
|
2015-12-03 18:21:29 +01:00
|
|
|
}
|
|
|
|
|
|
drm/etnaviv: rework MMU handling
This reworks the MMU handling to make it possible to have multiple MMU contexts.
A context is basically one instance of GPU page tables. Currently we have one
set of page tables per GPU, which isn't all that clever, as it has the
following two consequences:
1. All GPU clients (aka processes) are sharing the same pagetables, which means
there is no isolation between clients, but only between GPU assigned memory
spaces and the rest of the system. Better than nothing, but also not great.
2. Clients operating on the same set of buffers with different etnaviv GPU
cores, e.g. a workload using both the 2D and 3D GPU, need to map the used
buffers into the pagetable sets of each used GPU.
This patch reworks all the MMU handling to introduce the abstraction of the
MMU context. A context can be shared across different GPU cores, as long as
they have compatible MMU implementations, which is the case for all systems
with Vivante GPUs seen in the wild.
As MMUv1 is not able to change pagetables on the fly, without a
"stop the world" operation, which stops GPU, changes pagetables via CPU
interaction, restarts GPU, the implementation introduces a shared context on
MMUv1, which is returned whenever there is a request for a new context.
This patch assigns a MMU context to each GPU, so on MMUv2 systems there is
still one set of pagetables per GPU, but due to the shared context MMUv1
systems see a change in behavior as now a single pagetable set is used
across all GPU cores.
Signed-off-by: Lucas Stach <l.stach@pengutronix.de>
Reviewed-by: Philipp Zabel <p.zabel@pengutronix.de>
Reviewed-by: Guido Günther <agx@sigxcpu.org>
2019-07-05 19:17:24 +02:00
|
|
|
static void etnaviv_iommu_context_free(struct kref *kref)
|
2015-12-03 18:21:29 +01:00
|
|
|
{
|
drm/etnaviv: rework MMU handling
This reworks the MMU handling to make it possible to have multiple MMU contexts.
A context is basically one instance of GPU page tables. Currently we have one
set of page tables per GPU, which isn't all that clever, as it has the
following two consequences:
1. All GPU clients (aka processes) are sharing the same pagetables, which means
there is no isolation between clients, but only between GPU assigned memory
spaces and the rest of the system. Better than nothing, but also not great.
2. Clients operating on the same set of buffers with different etnaviv GPU
cores, e.g. a workload using both the 2D and 3D GPU, need to map the used
buffers into the pagetable sets of each used GPU.
This patch reworks all the MMU handling to introduce the abstraction of the
MMU context. A context can be shared across different GPU cores, as long as
they have compatible MMU implementations, which is the case for all systems
with Vivante GPUs seen in the wild.
As MMUv1 is not able to change pagetables on the fly, without a
"stop the world" operation, which stops GPU, changes pagetables via CPU
interaction, restarts GPU, the implementation introduces a shared context on
MMUv1, which is returned whenever there is a request for a new context.
This patch assigns a MMU context to each GPU, so on MMUv2 systems there is
still one set of pagetables per GPU, but due to the shared context MMUv1
systems see a change in behavior as now a single pagetable set is used
across all GPU cores.
Signed-off-by: Lucas Stach <l.stach@pengutronix.de>
Reviewed-by: Philipp Zabel <p.zabel@pengutronix.de>
Reviewed-by: Guido Günther <agx@sigxcpu.org>
2019-07-05 19:17:24 +02:00
|
|
|
struct etnaviv_iommu_context *context =
|
|
|
|
|
container_of(kref, struct etnaviv_iommu_context, refcount);
|
2015-12-03 18:21:29 +01:00
|
|
|
|
2019-07-05 19:17:27 +02:00
|
|
|
etnaviv_cmdbuf_suballoc_unmap(context, &context->cmdbuf_mapping);
|
2024-09-08 20:11:06 +08:00
|
|
|
mutex_destroy(&context->lock);
|
drm/etnaviv: rework MMU handling
This reworks the MMU handling to make it possible to have multiple MMU contexts.
A context is basically one instance of GPU page tables. Currently we have one
set of page tables per GPU, which isn't all that clever, as it has the
following two consequences:
1. All GPU clients (aka processes) are sharing the same pagetables, which means
there is no isolation between clients, but only between GPU assigned memory
spaces and the rest of the system. Better than nothing, but also not great.
2. Clients operating on the same set of buffers with different etnaviv GPU
cores, e.g. a workload using both the 2D and 3D GPU, need to map the used
buffers into the pagetable sets of each used GPU.
This patch reworks all the MMU handling to introduce the abstraction of the
MMU context. A context can be shared across different GPU cores, as long as
they have compatible MMU implementations, which is the case for all systems
with Vivante GPUs seen in the wild.
As MMUv1 is not able to change pagetables on the fly, without a
"stop the world" operation, which stops GPU, changes pagetables via CPU
interaction, restarts GPU, the implementation introduces a shared context on
MMUv1, which is returned whenever there is a request for a new context.
This patch assigns a MMU context to each GPU, so on MMUv2 systems there is
still one set of pagetables per GPU, but due to the shared context MMUv1
systems see a change in behavior as now a single pagetable set is used
across all GPU cores.
Signed-off-by: Lucas Stach <l.stach@pengutronix.de>
Reviewed-by: Philipp Zabel <p.zabel@pengutronix.de>
Reviewed-by: Guido Günther <agx@sigxcpu.org>
2019-07-05 19:17:24 +02:00
|
|
|
context->global->ops->free(context);
|
|
|
|
|
}
|
|
|
|
|
void etnaviv_iommu_context_put(struct etnaviv_iommu_context *context)
|
2015-12-03 18:21:29 +01:00
|
|
|
{
|
drm/etnaviv: rework MMU handling
This reworks the MMU handling to make it possible to have multiple MMU contexts.
A context is basically one instance of GPU page tables. Currently we have one
set of page tables per GPU, which isn't all that clever, as it has the
following two consequences:
1. All GPU clients (aka processes) are sharing the same pagetables, which means
there is no isolation between clients, but only between GPU assigned memory
spaces and the rest of the system. Better than nothing, but also not great.
2. Clients operating on the same set of buffers with different etnaviv GPU
cores, e.g. a workload using both the 2D and 3D GPU, need to map the used
buffers into the pagetable sets of each used GPU.
This patch reworks all the MMU handling to introduce the abstraction of the
MMU context. A context can be shared across different GPU cores, as long as
they have compatible MMU implementations, which is the case for all systems
with Vivante GPUs seen in the wild.
As MMUv1 is not able to change pagetables on the fly, without a
"stop the world" operation, which stops GPU, changes pagetables via CPU
interaction, restarts GPU, the implementation introduces a shared context on
MMUv1, which is returned whenever there is a request for a new context.
This patch assigns a MMU context to each GPU, so on MMUv2 systems there is
still one set of pagetables per GPU, but due to the shared context MMUv1
systems see a change in behavior as now a single pagetable set is used
across all GPU cores.
Signed-off-by: Lucas Stach <l.stach@pengutronix.de>
Reviewed-by: Philipp Zabel <p.zabel@pengutronix.de>
Reviewed-by: Guido Günther <agx@sigxcpu.org>
2019-07-05 19:17:24 +02:00
|
|
|
kref_put(&context->refcount, etnaviv_iommu_context_free);
|
2015-12-03 18:21:29 +01:00
|
|
|
}
|
|
|
|
|
|
drm/etnaviv: rework MMU handling
This reworks the MMU handling to make it possible to have multiple MMU contexts.
A context is basically one instance of GPU page tables. Currently we have one
set of page tables per GPU, which isn't all that clever, as it has the
following two consequences:
1. All GPU clients (aka processes) are sharing the same pagetables, which means
there is no isolation between clients, but only between GPU assigned memory
spaces and the rest of the system. Better than nothing, but also not great.
2. Clients operating on the same set of buffers with different etnaviv GPU
cores, e.g. a workload using both the 2D and 3D GPU, need to map the used
buffers into the pagetable sets of each used GPU.
This patch reworks all the MMU handling to introduce the abstraction of the
MMU context. A context can be shared across different GPU cores, as long as
they have compatible MMU implementations, which is the case for all systems
with Vivante GPUs seen in the wild.
As MMUv1 is not able to change pagetables on the fly, without a
"stop the world" operation, which stops GPU, changes pagetables via CPU
interaction, restarts GPU, the implementation introduces a shared context on
MMUv1, which is returned whenever there is a request for a new context.
This patch assigns a MMU context to each GPU, so on MMUv2 systems there is
still one set of pagetables per GPU, but due to the shared context MMUv1
systems see a change in behavior as now a single pagetable set is used
across all GPU cores.
Signed-off-by: Lucas Stach <l.stach@pengutronix.de>
Reviewed-by: Philipp Zabel <p.zabel@pengutronix.de>
Reviewed-by: Guido Günther <agx@sigxcpu.org>
2019-07-05 19:17:24 +02:00
|
|
|
struct etnaviv_iommu_context *
|
2019-07-05 19:17:27 +02:00
|
|
|
etnaviv_iommu_context_init(struct etnaviv_iommu_global *global,
|
|
|
|
|
struct etnaviv_cmdbuf_suballoc *suballoc)
|
2016-08-16 11:54:51 +02:00
|
|
|
{
|
2019-07-05 19:17:27 +02:00
|
|
|
struct etnaviv_iommu_context *ctx;
|
|
|
|
|
int ret;
|
|
|
|
|
|
drm/etnaviv: rework MMU handling
This reworks the MMU handling to make it possible to have multiple MMU contexts.
A context is basically one instance of GPU page tables. Currently we have one
set of page tables per GPU, which isn't all that clever, as it has the
following two consequences:
1. All GPU clients (aka processes) are sharing the same pagetables, which means
there is no isolation between clients, but only between GPU assigned memory
spaces and the rest of the system. Better than nothing, but also not great.
2. Clients operating on the same set of buffers with different etnaviv GPU
cores, e.g. a workload using both the 2D and 3D GPU, need to map the used
buffers into the pagetable sets of each used GPU.
This patch reworks all the MMU handling to introduce the abstraction of the
MMU context. A context can be shared across different GPU cores, as long as
they have compatible MMU implementations, which is the case for all systems
with Vivante GPUs seen in the wild.
As MMUv1 is not able to change pagetables on the fly, without a
"stop the world" operation, which stops GPU, changes pagetables via CPU
interaction, restarts GPU, the implementation introduces a shared context on
MMUv1, which is returned whenever there is a request for a new context.
This patch assigns a MMU context to each GPU, so on MMUv2 systems there is
still one set of pagetables per GPU, but due to the shared context MMUv1
systems see a change in behavior as now a single pagetable set is used
across all GPU cores.
Signed-off-by: Lucas Stach <l.stach@pengutronix.de>
Reviewed-by: Philipp Zabel <p.zabel@pengutronix.de>
Reviewed-by: Guido Günther <agx@sigxcpu.org>
2019-07-05 19:17:24 +02:00
|
|
|
if (global->version == ETNAVIV_IOMMU_V1)
|
2019-07-05 19:17:27 +02:00
|
|
|
ctx = etnaviv_iommuv1_context_alloc(global);
|
2016-08-16 11:54:51 +02:00
|
|
|
else
|
2019-07-05 19:17:27 +02:00
|
|
|
ctx = etnaviv_iommuv2_context_alloc(global);
|
|
|
|
|
|
|
|
|
|
if (!ctx)
|
|
|
|
|
return NULL;
|
|
|
|
|
|
|
|
|
|
ret = etnaviv_cmdbuf_suballoc_map(suballoc, ctx, &ctx->cmdbuf_mapping,
|
|
|
|
|
global->memory_base);
|
2019-10-16 16:10:21 +02:00
|
|
|
if (ret)
|
|
|
|
|
goto out_free;
|
|
|
|
|
|
|
|
|
|
if (global->version == ETNAVIV_IOMMU_V1 &&
|
|
|
|
|
ctx->cmdbuf_mapping.iova > 0x80000000) {
|
|
|
|
|
dev_err(global->dev,
|
|
|
|
|
"command buffer outside valid memory window\n");
|
|
|
|
|
goto out_unmap;
|
2019-07-05 19:17:27 +02:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
return ctx;
|
2019-10-16 16:10:21 +02:00
|
|
|
|
|
|
|
|
out_unmap:
|
|
|
|
|
etnaviv_cmdbuf_suballoc_unmap(ctx, &ctx->cmdbuf_mapping);
|
|
|
|
|
out_free:
|
|
|
|
|
global->ops->free(ctx);
|
|
|
|
|
return NULL;
|
drm/etnaviv: rework MMU handling
This reworks the MMU handling to make it possible to have multiple MMU contexts.
A context is basically one instance of GPU page tables. Currently we have one
set of page tables per GPU, which isn't all that clever, as it has the
following two consequences:
1. All GPU clients (aka processes) are sharing the same pagetables, which means
there is no isolation between clients, but only between GPU assigned memory
spaces and the rest of the system. Better than nothing, but also not great.
2. Clients operating on the same set of buffers with different etnaviv GPU
cores, e.g. a workload using both the 2D and 3D GPU, need to map the used
buffers into the pagetable sets of each used GPU.
This patch reworks all the MMU handling to introduce the abstraction of the
MMU context. A context can be shared across different GPU cores, as long as
they have compatible MMU implementations, which is the case for all systems
with Vivante GPUs seen in the wild.
As MMUv1 is not able to change pagetables on the fly, without a
"stop the world" operation, which stops GPU, changes pagetables via CPU
interaction, restarts GPU, the implementation introduces a shared context on
MMUv1, which is returned whenever there is a request for a new context.
This patch assigns a MMU context to each GPU, so on MMUv2 systems there is
still one set of pagetables per GPU, but due to the shared context MMUv1
systems see a change in behavior as now a single pagetable set is used
across all GPU cores.
Signed-off-by: Lucas Stach <l.stach@pengutronix.de>
Reviewed-by: Philipp Zabel <p.zabel@pengutronix.de>
Reviewed-by: Guido Günther <agx@sigxcpu.org>
2019-07-05 19:17:24 +02:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
void etnaviv_iommu_restore(struct etnaviv_gpu *gpu,
|
|
|
|
|
struct etnaviv_iommu_context *context)
|
|
|
|
|
{
|
|
|
|
|
context->global->ops->restore(gpu, context);
|
2016-08-16 11:54:51 +02:00
|
|
|
}
|
|
|
|
|
|
drm/etnaviv: rework MMU handling
This reworks the MMU handling to make it possible to have multiple MMU contexts.
A context is basically one instance of GPU page tables. Currently we have one
set of page tables per GPU, which isn't all that clever, as it has the
following two consequences:
1. All GPU clients (aka processes) are sharing the same pagetables, which means
there is no isolation between clients, but only between GPU assigned memory
spaces and the rest of the system. Better than nothing, but also not great.
2. Clients operating on the same set of buffers with different etnaviv GPU
cores, e.g. a workload using both the 2D and 3D GPU, need to map the used
buffers into the pagetable sets of each used GPU.
This patch reworks all the MMU handling to introduce the abstraction of the
MMU context. A context can be shared across different GPU cores, as long as
they have compatible MMU implementations, which is the case for all systems
with Vivante GPUs seen in the wild.
As MMUv1 is not able to change pagetables on the fly, without a
"stop the world" operation, which stops GPU, changes pagetables via CPU
interaction, restarts GPU, the implementation introduces a shared context on
MMUv1, which is returned whenever there is a request for a new context.
This patch assigns a MMU context to each GPU, so on MMUv2 systems there is
still one set of pagetables per GPU, but due to the shared context MMUv1
systems see a change in behavior as now a single pagetable set is used
across all GPU cores.
Signed-off-by: Lucas Stach <l.stach@pengutronix.de>
Reviewed-by: Philipp Zabel <p.zabel@pengutronix.de>
Reviewed-by: Guido Günther <agx@sigxcpu.org>
2019-07-05 19:17:24 +02:00
|
|
|
int etnaviv_iommu_get_suballoc_va(struct etnaviv_iommu_context *context,
|
2019-07-05 19:17:21 +02:00
|
|
|
struct etnaviv_vram_mapping *mapping,
|
|
|
|
|
u32 memory_base, dma_addr_t paddr,
|
|
|
|
|
size_t size)
|
2016-08-17 14:57:51 +02:00
|
|
|
{
|
drm/etnaviv: rework MMU handling
This reworks the MMU handling to make it possible to have multiple MMU contexts.
A context is basically one instance of GPU page tables. Currently we have one
set of page tables per GPU, which isn't all that clever, as it has the
following two consequences:
1. All GPU clients (aka processes) are sharing the same pagetables, which means
there is no isolation between clients, but only between GPU assigned memory
spaces and the rest of the system. Better than nothing, but also not great.
2. Clients operating on the same set of buffers with different etnaviv GPU
cores, e.g. a workload using both the 2D and 3D GPU, need to map the used
buffers into the pagetable sets of each used GPU.
This patch reworks all the MMU handling to introduce the abstraction of the
MMU context. A context can be shared across different GPU cores, as long as
they have compatible MMU implementations, which is the case for all systems
with Vivante GPUs seen in the wild.
As MMUv1 is not able to change pagetables on the fly, without a
"stop the world" operation, which stops GPU, changes pagetables via CPU
interaction, restarts GPU, the implementation introduces a shared context on
MMUv1, which is returned whenever there is a request for a new context.
This patch assigns a MMU context to each GPU, so on MMUv2 systems there is
still one set of pagetables per GPU, but due to the shared context MMUv1
systems see a change in behavior as now a single pagetable set is used
across all GPU cores.
Signed-off-by: Lucas Stach <l.stach@pengutronix.de>
Reviewed-by: Philipp Zabel <p.zabel@pengutronix.de>
Reviewed-by: Guido Günther <agx@sigxcpu.org>
2019-07-05 19:17:24 +02:00
|
|
|
mutex_lock(&context->lock);
|
2016-08-19 23:49:10 +02:00
|
|
|
|
2019-07-05 19:17:27 +02:00
|
|
|
if (mapping->use > 0) {
|
|
|
|
|
mapping->use++;
|
|
|
|
|
mutex_unlock(&context->lock);
|
|
|
|
|
return 0;
|
|
|
|
|
}
|
|
|
|
|
|
2019-07-05 19:17:21 +02:00
|
|
|
/*
|
|
|
|
|
* For MMUv1 we don't add the suballoc region to the pagetables, as
|
|
|
|
|
* those GPUs can only work with cmdbufs accessed through the linear
|
|
|
|
|
* window. Instead we manufacture a mapping to make it look uniform
|
|
|
|
|
* to the upper layers.
|
|
|
|
|
*/
|
drm/etnaviv: rework MMU handling
This reworks the MMU handling to make it possible to have multiple MMU contexts.
A context is basically one instance of GPU page tables. Currently we have one
set of page tables per GPU, which isn't all that clever, as it has the
following two consequences:
1. All GPU clients (aka processes) are sharing the same pagetables, which means
there is no isolation between clients, but only between GPU assigned memory
spaces and the rest of the system. Better than nothing, but also not great.
2. Clients operating on the same set of buffers with different etnaviv GPU
cores, e.g. a workload using both the 2D and 3D GPU, need to map the used
buffers into the pagetable sets of each used GPU.
This patch reworks all the MMU handling to introduce the abstraction of the
MMU context. A context can be shared across different GPU cores, as long as
they have compatible MMU implementations, which is the case for all systems
with Vivante GPUs seen in the wild.
As MMUv1 is not able to change pagetables on the fly, without a
"stop the world" operation, which stops GPU, changes pagetables via CPU
interaction, restarts GPU, the implementation introduces a shared context on
MMUv1, which is returned whenever there is a request for a new context.
This patch assigns a MMU context to each GPU, so on MMUv2 systems there is
still one set of pagetables per GPU, but due to the shared context MMUv1
systems see a change in behavior as now a single pagetable set is used
across all GPU cores.
Signed-off-by: Lucas Stach <l.stach@pengutronix.de>
Reviewed-by: Philipp Zabel <p.zabel@pengutronix.de>
Reviewed-by: Guido Günther <agx@sigxcpu.org>
2019-07-05 19:17:24 +02:00
|
|
|
if (context->global->version == ETNAVIV_IOMMU_V1) {
|
2019-07-05 19:17:21 +02:00
|
|
|
mapping->iova = paddr - memory_base;
|
2016-08-19 23:49:10 +02:00
|
|
|
} else {
|
2019-07-05 19:17:21 +02:00
|
|
|
struct drm_mm_node *node = &mapping->vram_node;
|
2016-08-19 23:49:10 +02:00
|
|
|
int ret;
|
|
|
|
|
|
drm/etnaviv: rework MMU handling
This reworks the MMU handling to make it possible to have multiple MMU contexts.
A context is basically one instance of GPU page tables. Currently we have one
set of page tables per GPU, which isn't all that clever, as it has the
following two consequences:
1. All GPU clients (aka processes) are sharing the same pagetables, which means
there is no isolation between clients, but only between GPU assigned memory
spaces and the rest of the system. Better than nothing, but also not great.
2. Clients operating on the same set of buffers with different etnaviv GPU
cores, e.g. a workload using both the 2D and 3D GPU, need to map the used
buffers into the pagetable sets of each used GPU.
This patch reworks all the MMU handling to introduce the abstraction of the
MMU context. A context can be shared across different GPU cores, as long as
they have compatible MMU implementations, which is the case for all systems
with Vivante GPUs seen in the wild.
As MMUv1 is not able to change pagetables on the fly, without a
"stop the world" operation, which stops GPU, changes pagetables via CPU
interaction, restarts GPU, the implementation introduces a shared context on
MMUv1, which is returned whenever there is a request for a new context.
This patch assigns a MMU context to each GPU, so on MMUv2 systems there is
still one set of pagetables per GPU, but due to the shared context MMUv1
systems see a change in behavior as now a single pagetable set is used
across all GPU cores.
Signed-off-by: Lucas Stach <l.stach@pengutronix.de>
Reviewed-by: Philipp Zabel <p.zabel@pengutronix.de>
Reviewed-by: Guido Günther <agx@sigxcpu.org>
2019-07-05 19:17:24 +02:00
|
|
|
ret = etnaviv_iommu_find_iova(context, node, size);
|
2016-08-19 23:49:10 +02:00
|
|
|
if (ret < 0) {
|
drm/etnaviv: rework MMU handling
This reworks the MMU handling to make it possible to have multiple MMU contexts.
A context is basically one instance of GPU page tables. Currently we have one
set of page tables per GPU, which isn't all that clever, as it has the
following two consequences:
1. All GPU clients (aka processes) are sharing the same pagetables, which means
there is no isolation between clients, but only between GPU assigned memory
spaces and the rest of the system. Better than nothing, but also not great.
2. Clients operating on the same set of buffers with different etnaviv GPU
cores, e.g. a workload using both the 2D and 3D GPU, need to map the used
buffers into the pagetable sets of each used GPU.
This patch reworks all the MMU handling to introduce the abstraction of the
MMU context. A context can be shared across different GPU cores, as long as
they have compatible MMU implementations, which is the case for all systems
with Vivante GPUs seen in the wild.
As MMUv1 is not able to change pagetables on the fly, without a
"stop the world" operation, which stops GPU, changes pagetables via CPU
interaction, restarts GPU, the implementation introduces a shared context on
MMUv1, which is returned whenever there is a request for a new context.
This patch assigns a MMU context to each GPU, so on MMUv2 systems there is
still one set of pagetables per GPU, but due to the shared context MMUv1
systems see a change in behavior as now a single pagetable set is used
across all GPU cores.
Signed-off-by: Lucas Stach <l.stach@pengutronix.de>
Reviewed-by: Philipp Zabel <p.zabel@pengutronix.de>
Reviewed-by: Guido Günther <agx@sigxcpu.org>
2019-07-05 19:17:24 +02:00
|
|
|
mutex_unlock(&context->lock);
|
2017-01-16 17:29:57 +01:00
|
|
|
return ret;
|
2016-08-19 23:49:10 +02:00
|
|
|
}
|
2019-07-05 19:17:21 +02:00
|
|
|
|
|
|
|
|
mapping->iova = node->start;
|
drm/etnaviv: rework MMU handling
This reworks the MMU handling to make it possible to have multiple MMU contexts.
A context is basically one instance of GPU page tables. Currently we have one
set of page tables per GPU, which isn't all that clever, as it has the
following two consequences:
1. All GPU clients (aka processes) are sharing the same pagetables, which means
there is no isolation between clients, but only between GPU assigned memory
spaces and the rest of the system. Better than nothing, but also not great.
2. Clients operating on the same set of buffers with different etnaviv GPU
cores, e.g. a workload using both the 2D and 3D GPU, need to map the used
buffers into the pagetable sets of each used GPU.
This patch reworks all the MMU handling to introduce the abstraction of the
MMU context. A context can be shared across different GPU cores, as long as
they have compatible MMU implementations, which is the case for all systems
with Vivante GPUs seen in the wild.
As MMUv1 is not able to change pagetables on the fly, without a
"stop the world" operation, which stops GPU, changes pagetables via CPU
interaction, restarts GPU, the implementation introduces a shared context on
MMUv1, which is returned whenever there is a request for a new context.
This patch assigns a MMU context to each GPU, so on MMUv2 systems there is
still one set of pagetables per GPU, but due to the shared context MMUv1
systems see a change in behavior as now a single pagetable set is used
across all GPU cores.
Signed-off-by: Lucas Stach <l.stach@pengutronix.de>
Reviewed-by: Philipp Zabel <p.zabel@pengutronix.de>
Reviewed-by: Guido Günther <agx@sigxcpu.org>
2019-07-05 19:17:24 +02:00
|
|
|
ret = etnaviv_context_map(context, node->start, paddr, size,
|
|
|
|
|
ETNAVIV_PROT_READ);
|
2016-08-19 23:49:10 +02:00
|
|
|
if (ret < 0) {
|
2019-07-05 19:17:21 +02:00
|
|
|
drm_mm_remove_node(node);
|
drm/etnaviv: rework MMU handling
This reworks the MMU handling to make it possible to have multiple MMU contexts.
A context is basically one instance of GPU page tables. Currently we have one
set of page tables per GPU, which isn't all that clever, as it has the
following two consequences:
1. All GPU clients (aka processes) are sharing the same pagetables, which means
there is no isolation between clients, but only between GPU assigned memory
spaces and the rest of the system. Better than nothing, but also not great.
2. Clients operating on the same set of buffers with different etnaviv GPU
cores, e.g. a workload using both the 2D and 3D GPU, need to map the used
buffers into the pagetable sets of each used GPU.
This patch reworks all the MMU handling to introduce the abstraction of the
MMU context. A context can be shared across different GPU cores, as long as
they have compatible MMU implementations, which is the case for all systems
with Vivante GPUs seen in the wild.
As MMUv1 is not able to change pagetables on the fly, without a
"stop the world" operation, which stops GPU, changes pagetables via CPU
interaction, restarts GPU, the implementation introduces a shared context on
MMUv1, which is returned whenever there is a request for a new context.
This patch assigns a MMU context to each GPU, so on MMUv2 systems there is
still one set of pagetables per GPU, but due to the shared context MMUv1
systems see a change in behavior as now a single pagetable set is used
across all GPU cores.
Signed-off-by: Lucas Stach <l.stach@pengutronix.de>
Reviewed-by: Philipp Zabel <p.zabel@pengutronix.de>
Reviewed-by: Guido Günther <agx@sigxcpu.org>
2019-07-05 19:17:24 +02:00
|
|
|
mutex_unlock(&context->lock);
|
2017-01-16 17:29:57 +01:00
|
|
|
return ret;
|
2016-08-19 23:49:10 +02:00
|
|
|
}
|
|
|
|
|
|
drm/etnaviv: rework MMU handling
This reworks the MMU handling to make it possible to have multiple MMU contexts.
A context is basically one instance of GPU page tables. Currently we have one
set of page tables per GPU, which isn't all that clever, as it has the
following two consequences:
1. All GPU clients (aka processes) are sharing the same pagetables, which means
there is no isolation between clients, but only between GPU assigned memory
spaces and the rest of the system. Better than nothing, but also not great.
2. Clients operating on the same set of buffers with different etnaviv GPU
cores, e.g. a workload using both the 2D and 3D GPU, need to map the used
buffers into the pagetable sets of each used GPU.
This patch reworks all the MMU handling to introduce the abstraction of the
MMU context. A context can be shared across different GPU cores, as long as
they have compatible MMU implementations, which is the case for all systems
with Vivante GPUs seen in the wild.
As MMUv1 is not able to change pagetables on the fly, without a
"stop the world" operation, which stops GPU, changes pagetables via CPU
interaction, restarts GPU, the implementation introduces a shared context on
MMUv1, which is returned whenever there is a request for a new context.
This patch assigns a MMU context to each GPU, so on MMUv2 systems there is
still one set of pagetables per GPU, but due to the shared context MMUv1
systems see a change in behavior as now a single pagetable set is used
across all GPU cores.
Signed-off-by: Lucas Stach <l.stach@pengutronix.de>
Reviewed-by: Philipp Zabel <p.zabel@pengutronix.de>
Reviewed-by: Guido Günther <agx@sigxcpu.org>
2019-07-05 19:17:24 +02:00
|
|
|
context->flush_seq++;
|
2016-08-19 23:49:10 +02:00
|
|
|
}
|
2019-07-05 19:17:21 +02:00
|
|
|
|
drm/etnaviv: rework MMU handling
This reworks the MMU handling to make it possible to have multiple MMU contexts.
A context is basically one instance of GPU page tables. Currently we have one
set of page tables per GPU, which isn't all that clever, as it has the
following two consequences:
1. All GPU clients (aka processes) are sharing the same pagetables, which means
there is no isolation between clients, but only between GPU assigned memory
spaces and the rest of the system. Better than nothing, but also not great.
2. Clients operating on the same set of buffers with different etnaviv GPU
cores, e.g. a workload using both the 2D and 3D GPU, need to map the used
buffers into the pagetable sets of each used GPU.
This patch reworks all the MMU handling to introduce the abstraction of the
MMU context. A context can be shared across different GPU cores, as long as
they have compatible MMU implementations, which is the case for all systems
with Vivante GPUs seen in the wild.
As MMUv1 is not able to change pagetables on the fly, without a
"stop the world" operation, which stops GPU, changes pagetables via CPU
interaction, restarts GPU, the implementation introduces a shared context on
MMUv1, which is returned whenever there is a request for a new context.
This patch assigns a MMU context to each GPU, so on MMUv2 systems there is
still one set of pagetables per GPU, but due to the shared context MMUv1
systems see a change in behavior as now a single pagetable set is used
across all GPU cores.
Signed-off-by: Lucas Stach <l.stach@pengutronix.de>
Reviewed-by: Philipp Zabel <p.zabel@pengutronix.de>
Reviewed-by: Guido Günther <agx@sigxcpu.org>
2019-07-05 19:17:24 +02:00
|
|
|
list_add_tail(&mapping->mmu_node, &context->mappings);
|
2019-07-05 19:17:21 +02:00
|
|
|
mapping->use = 1;
|
|
|
|
|
|
drm/etnaviv: rework MMU handling
This reworks the MMU handling to make it possible to have multiple MMU contexts.
A context is basically one instance of GPU page tables. Currently we have one
set of page tables per GPU, which isn't all that clever, as it has the
following two consequences:
1. All GPU clients (aka processes) are sharing the same pagetables, which means
there is no isolation between clients, but only between GPU assigned memory
spaces and the rest of the system. Better than nothing, but also not great.
2. Clients operating on the same set of buffers with different etnaviv GPU
cores, e.g. a workload using both the 2D and 3D GPU, need to map the used
buffers into the pagetable sets of each used GPU.
This patch reworks all the MMU handling to introduce the abstraction of the
MMU context. A context can be shared across different GPU cores, as long as
they have compatible MMU implementations, which is the case for all systems
with Vivante GPUs seen in the wild.
As MMUv1 is not able to change pagetables on the fly, without a
"stop the world" operation, which stops GPU, changes pagetables via CPU
interaction, restarts GPU, the implementation introduces a shared context on
MMUv1, which is returned whenever there is a request for a new context.
This patch assigns a MMU context to each GPU, so on MMUv2 systems there is
still one set of pagetables per GPU, but due to the shared context MMUv1
systems see a change in behavior as now a single pagetable set is used
across all GPU cores.
Signed-off-by: Lucas Stach <l.stach@pengutronix.de>
Reviewed-by: Philipp Zabel <p.zabel@pengutronix.de>
Reviewed-by: Guido Günther <agx@sigxcpu.org>
2019-07-05 19:17:24 +02:00
|
|
|
mutex_unlock(&context->lock);
|
2019-07-05 19:17:21 +02:00
|
|
|
|
|
|
|
|
return 0;
|
2016-08-17 14:57:51 +02:00
|
|
|
}
|
|
|
|
|
|
drm/etnaviv: rework MMU handling
This reworks the MMU handling to make it possible to have multiple MMU contexts.
A context is basically one instance of GPU page tables. Currently we have one
set of page tables per GPU, which isn't all that clever, as it has the
following two consequences:
1. All GPU clients (aka processes) are sharing the same pagetables, which means
there is no isolation between clients, but only between GPU assigned memory
spaces and the rest of the system. Better than nothing, but also not great.
2. Clients operating on the same set of buffers with different etnaviv GPU
cores, e.g. a workload using both the 2D and 3D GPU, need to map the used
buffers into the pagetable sets of each used GPU.
This patch reworks all the MMU handling to introduce the abstraction of the
MMU context. A context can be shared across different GPU cores, as long as
they have compatible MMU implementations, which is the case for all systems
with Vivante GPUs seen in the wild.
As MMUv1 is not able to change pagetables on the fly, without a
"stop the world" operation, which stops GPU, changes pagetables via CPU
interaction, restarts GPU, the implementation introduces a shared context on
MMUv1, which is returned whenever there is a request for a new context.
This patch assigns a MMU context to each GPU, so on MMUv2 systems there is
still one set of pagetables per GPU, but due to the shared context MMUv1
systems see a change in behavior as now a single pagetable set is used
across all GPU cores.
Signed-off-by: Lucas Stach <l.stach@pengutronix.de>
Reviewed-by: Philipp Zabel <p.zabel@pengutronix.de>
Reviewed-by: Guido Günther <agx@sigxcpu.org>
2019-07-05 19:17:24 +02:00
|
|
|
void etnaviv_iommu_put_suballoc_va(struct etnaviv_iommu_context *context,
|
2019-07-05 19:17:21 +02:00
|
|
|
struct etnaviv_vram_mapping *mapping)
|
2016-08-19 23:49:10 +02:00
|
|
|
{
|
2019-07-05 19:17:21 +02:00
|
|
|
struct drm_mm_node *node = &mapping->vram_node;
|
2016-08-19 23:49:10 +02:00
|
|
|
|
2019-07-05 19:17:27 +02:00
|
|
|
mutex_lock(&context->lock);
|
|
|
|
|
mapping->use--;
|
2019-07-05 19:17:21 +02:00
|
|
|
|
2019-07-05 19:17:27 +02:00
|
|
|
if (mapping->use > 0 || context->global->version == ETNAVIV_IOMMU_V1) {
|
|
|
|
|
mutex_unlock(&context->lock);
|
2019-07-05 19:17:21 +02:00
|
|
|
return;
|
2019-07-05 19:17:27 +02:00
|
|
|
}
|
2019-07-05 19:17:21 +02:00
|
|
|
|
drm/etnaviv: rework MMU handling
This reworks the MMU handling to make it possible to have multiple MMU contexts.
A context is basically one instance of GPU page tables. Currently we have one
set of page tables per GPU, which isn't all that clever, as it has the
following two consequences:
1. All GPU clients (aka processes) are sharing the same pagetables, which means
there is no isolation between clients, but only between GPU assigned memory
spaces and the rest of the system. Better than nothing, but also not great.
2. Clients operating on the same set of buffers with different etnaviv GPU
cores, e.g. a workload using both the 2D and 3D GPU, need to map the used
buffers into the pagetable sets of each used GPU.
This patch reworks all the MMU handling to introduce the abstraction of the
MMU context. A context can be shared across different GPU cores, as long as
they have compatible MMU implementations, which is the case for all systems
with Vivante GPUs seen in the wild.
As MMUv1 is not able to change pagetables on the fly, without a
"stop the world" operation, which stops GPU, changes pagetables via CPU
interaction, restarts GPU, the implementation introduces a shared context on
MMUv1, which is returned whenever there is a request for a new context.
This patch assigns a MMU context to each GPU, so on MMUv2 systems there is
still one set of pagetables per GPU, but due to the shared context MMUv1
systems see a change in behavior as now a single pagetable set is used
across all GPU cores.
Signed-off-by: Lucas Stach <l.stach@pengutronix.de>
Reviewed-by: Philipp Zabel <p.zabel@pengutronix.de>
Reviewed-by: Guido Günther <agx@sigxcpu.org>
2019-07-05 19:17:24 +02:00
|
|
|
etnaviv_context_unmap(context, node->start, node->size);
|
2019-07-05 19:17:21 +02:00
|
|
|
drm_mm_remove_node(node);
|
drm/etnaviv: rework MMU handling
This reworks the MMU handling to make it possible to have multiple MMU contexts.
A context is basically one instance of GPU page tables. Currently we have one
set of page tables per GPU, which isn't all that clever, as it has the
following two consequences:
1. All GPU clients (aka processes) are sharing the same pagetables, which means
there is no isolation between clients, but only between GPU assigned memory
spaces and the rest of the system. Better than nothing, but also not great.
2. Clients operating on the same set of buffers with different etnaviv GPU
cores, e.g. a workload using both the 2D and 3D GPU, need to map the used
buffers into the pagetable sets of each used GPU.
This patch reworks all the MMU handling to introduce the abstraction of the
MMU context. A context can be shared across different GPU cores, as long as
they have compatible MMU implementations, which is the case for all systems
with Vivante GPUs seen in the wild.
As MMUv1 is not able to change pagetables on the fly, without a
"stop the world" operation, which stops GPU, changes pagetables via CPU
interaction, restarts GPU, the implementation introduces a shared context on
MMUv1, which is returned whenever there is a request for a new context.
This patch assigns a MMU context to each GPU, so on MMUv2 systems there is
still one set of pagetables per GPU, but due to the shared context MMUv1
systems see a change in behavior as now a single pagetable set is used
across all GPU cores.
Signed-off-by: Lucas Stach <l.stach@pengutronix.de>
Reviewed-by: Philipp Zabel <p.zabel@pengutronix.de>
Reviewed-by: Guido Günther <agx@sigxcpu.org>
2019-07-05 19:17:24 +02:00
|
|
|
mutex_unlock(&context->lock);
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
size_t etnaviv_iommu_dump_size(struct etnaviv_iommu_context *context)
|
|
|
|
|
{
|
|
|
|
|
return context->global->ops->dump_size(context);
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
void etnaviv_iommu_dump(struct etnaviv_iommu_context *context, void *buf)
|
|
|
|
|
{
|
|
|
|
|
context->global->ops->dump(context, buf);
|
2016-08-19 23:49:10 +02:00
|
|
|
}
|
2019-07-05 19:17:21 +02:00
|
|
|
|
drm/etnaviv: rework MMU handling
This reworks the MMU handling to make it possible to have multiple MMU contexts.
A context is basically one instance of GPU page tables. Currently we have one
set of page tables per GPU, which isn't all that clever, as it has the
following two consequences:
1. All GPU clients (aka processes) are sharing the same pagetables, which means
there is no isolation between clients, but only between GPU assigned memory
spaces and the rest of the system. Better than nothing, but also not great.
2. Clients operating on the same set of buffers with different etnaviv GPU
cores, e.g. a workload using both the 2D and 3D GPU, need to map the used
buffers into the pagetable sets of each used GPU.
This patch reworks all the MMU handling to introduce the abstraction of the
MMU context. A context can be shared across different GPU cores, as long as
they have compatible MMU implementations, which is the case for all systems
with Vivante GPUs seen in the wild.
As MMUv1 is not able to change pagetables on the fly, without a
"stop the world" operation, which stops GPU, changes pagetables via CPU
interaction, restarts GPU, the implementation introduces a shared context on
MMUv1, which is returned whenever there is a request for a new context.
This patch assigns a MMU context to each GPU, so on MMUv2 systems there is
still one set of pagetables per GPU, but due to the shared context MMUv1
systems see a change in behavior as now a single pagetable set is used
across all GPU cores.
Signed-off-by: Lucas Stach <l.stach@pengutronix.de>
Reviewed-by: Philipp Zabel <p.zabel@pengutronix.de>
Reviewed-by: Guido Günther <agx@sigxcpu.org>
2019-07-05 19:17:24 +02:00
|
|
|
int etnaviv_iommu_global_init(struct etnaviv_gpu *gpu)
|
2015-12-03 18:21:29 +01:00
|
|
|
{
|
drm/etnaviv: rework MMU handling
This reworks the MMU handling to make it possible to have multiple MMU contexts.
A context is basically one instance of GPU page tables. Currently we have one
set of page tables per GPU, which isn't all that clever, as it has the
following two consequences:
1. All GPU clients (aka processes) are sharing the same pagetables, which means
there is no isolation between clients, but only between GPU assigned memory
spaces and the rest of the system. Better than nothing, but also not great.
2. Clients operating on the same set of buffers with different etnaviv GPU
cores, e.g. a workload using both the 2D and 3D GPU, need to map the used
buffers into the pagetable sets of each used GPU.
This patch reworks all the MMU handling to introduce the abstraction of the
MMU context. A context can be shared across different GPU cores, as long as
they have compatible MMU implementations, which is the case for all systems
with Vivante GPUs seen in the wild.
As MMUv1 is not able to change pagetables on the fly, without a
"stop the world" operation, which stops GPU, changes pagetables via CPU
interaction, restarts GPU, the implementation introduces a shared context on
MMUv1, which is returned whenever there is a request for a new context.
This patch assigns a MMU context to each GPU, so on MMUv2 systems there is
still one set of pagetables per GPU, but due to the shared context MMUv1
systems see a change in behavior as now a single pagetable set is used
across all GPU cores.
Signed-off-by: Lucas Stach <l.stach@pengutronix.de>
Reviewed-by: Philipp Zabel <p.zabel@pengutronix.de>
Reviewed-by: Guido Günther <agx@sigxcpu.org>
2019-07-05 19:17:24 +02:00
|
|
|
enum etnaviv_iommu_version version = ETNAVIV_IOMMU_V1;
|
|
|
|
|
struct etnaviv_drm_private *priv = gpu->drm->dev_private;
|
|
|
|
|
struct etnaviv_iommu_global *global;
|
|
|
|
|
struct device *dev = gpu->drm->dev;
|
|
|
|
|
|
|
|
|
|
if (gpu->identity.minor_features1 & chipMinorFeatures1_MMU_VERSION)
|
|
|
|
|
version = ETNAVIV_IOMMU_V2;
|
|
|
|
|
|
|
|
|
|
if (priv->mmu_global) {
|
|
|
|
|
if (priv->mmu_global->version != version) {
|
|
|
|
|
dev_err(gpu->dev,
|
|
|
|
|
"MMU version doesn't match global version\n");
|
|
|
|
|
return -ENXIO;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
priv->mmu_global->use++;
|
|
|
|
|
return 0;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
global = kzalloc(sizeof(*global), GFP_KERNEL);
|
|
|
|
|
if (!global)
|
|
|
|
|
return -ENOMEM;
|
|
|
|
|
|
|
|
|
|
global->bad_page_cpu = dma_alloc_wc(dev, SZ_4K, &global->bad_page_dma,
|
|
|
|
|
GFP_KERNEL);
|
|
|
|
|
if (!global->bad_page_cpu)
|
|
|
|
|
goto free_global;
|
|
|
|
|
|
|
|
|
|
memset32(global->bad_page_cpu, 0xdead55aa, SZ_4K / sizeof(u32));
|
|
|
|
|
|
|
|
|
|
if (version == ETNAVIV_IOMMU_V2) {
|
|
|
|
|
global->v2.pta_cpu = dma_alloc_wc(dev, ETNAVIV_PTA_SIZE,
|
|
|
|
|
&global->v2.pta_dma, GFP_KERNEL);
|
|
|
|
|
if (!global->v2.pta_cpu)
|
|
|
|
|
goto free_bad_page;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
global->dev = dev;
|
|
|
|
|
global->version = version;
|
|
|
|
|
global->use = 1;
|
|
|
|
|
mutex_init(&global->lock);
|
|
|
|
|
|
|
|
|
|
if (version == ETNAVIV_IOMMU_V1)
|
|
|
|
|
global->ops = &etnaviv_iommuv1_ops;
|
|
|
|
|
else
|
|
|
|
|
global->ops = &etnaviv_iommuv2_ops;
|
|
|
|
|
|
|
|
|
|
priv->mmu_global = global;
|
|
|
|
|
|
|
|
|
|
return 0;
|
|
|
|
|
|
|
|
|
|
free_bad_page:
|
|
|
|
|
dma_free_wc(dev, SZ_4K, global->bad_page_cpu, global->bad_page_dma);
|
|
|
|
|
free_global:
|
|
|
|
|
kfree(global);
|
|
|
|
|
|
|
|
|
|
return -ENOMEM;
|
2015-12-03 18:21:29 +01:00
|
|
|
}
|
|
|
|
|
|
drm/etnaviv: rework MMU handling
This reworks the MMU handling to make it possible to have multiple MMU contexts.
A context is basically one instance of GPU page tables. Currently we have one
set of page tables per GPU, which isn't all that clever, as it has the
following two consequences:
1. All GPU clients (aka processes) are sharing the same pagetables, which means
there is no isolation between clients, but only between GPU assigned memory
spaces and the rest of the system. Better than nothing, but also not great.
2. Clients operating on the same set of buffers with different etnaviv GPU
cores, e.g. a workload using both the 2D and 3D GPU, need to map the used
buffers into the pagetable sets of each used GPU.
This patch reworks all the MMU handling to introduce the abstraction of the
MMU context. A context can be shared across different GPU cores, as long as
they have compatible MMU implementations, which is the case for all systems
with Vivante GPUs seen in the wild.
As MMUv1 is not able to change pagetables on the fly, without a
"stop the world" operation, which stops GPU, changes pagetables via CPU
interaction, restarts GPU, the implementation introduces a shared context on
MMUv1, which is returned whenever there is a request for a new context.
This patch assigns a MMU context to each GPU, so on MMUv2 systems there is
still one set of pagetables per GPU, but due to the shared context MMUv1
systems see a change in behavior as now a single pagetable set is used
across all GPU cores.
Signed-off-by: Lucas Stach <l.stach@pengutronix.de>
Reviewed-by: Philipp Zabel <p.zabel@pengutronix.de>
Reviewed-by: Guido Günther <agx@sigxcpu.org>
2019-07-05 19:17:24 +02:00
|
|
|
void etnaviv_iommu_global_fini(struct etnaviv_gpu *gpu)
|
2015-12-03 18:21:29 +01:00
|
|
|
{
|
drm/etnaviv: rework MMU handling
This reworks the MMU handling to make it possible to have multiple MMU contexts.
A context is basically one instance of GPU page tables. Currently we have one
set of page tables per GPU, which isn't all that clever, as it has the
following two consequences:
1. All GPU clients (aka processes) are sharing the same pagetables, which means
there is no isolation between clients, but only between GPU assigned memory
spaces and the rest of the system. Better than nothing, but also not great.
2. Clients operating on the same set of buffers with different etnaviv GPU
cores, e.g. a workload using both the 2D and 3D GPU, need to map the used
buffers into the pagetable sets of each used GPU.
This patch reworks all the MMU handling to introduce the abstraction of the
MMU context. A context can be shared across different GPU cores, as long as
they have compatible MMU implementations, which is the case for all systems
with Vivante GPUs seen in the wild.
As MMUv1 is not able to change pagetables on the fly, without a
"stop the world" operation, which stops GPU, changes pagetables via CPU
interaction, restarts GPU, the implementation introduces a shared context on
MMUv1, which is returned whenever there is a request for a new context.
This patch assigns a MMU context to each GPU, so on MMUv2 systems there is
still one set of pagetables per GPU, but due to the shared context MMUv1
systems see a change in behavior as now a single pagetable set is used
across all GPU cores.
Signed-off-by: Lucas Stach <l.stach@pengutronix.de>
Reviewed-by: Philipp Zabel <p.zabel@pengutronix.de>
Reviewed-by: Guido Günther <agx@sigxcpu.org>
2019-07-05 19:17:24 +02:00
|
|
|
struct etnaviv_drm_private *priv = gpu->drm->dev_private;
|
|
|
|
|
struct etnaviv_iommu_global *global = priv->mmu_global;
|
|
|
|
|
|
2023-06-07 15:02:22 +02:00
|
|
|
if (!global)
|
|
|
|
|
return;
|
|
|
|
|
|
drm/etnaviv: rework MMU handling
This reworks the MMU handling to make it possible to have multiple MMU contexts.
A context is basically one instance of GPU page tables. Currently we have one
set of page tables per GPU, which isn't all that clever, as it has the
following two consequences:
1. All GPU clients (aka processes) are sharing the same pagetables, which means
there is no isolation between clients, but only between GPU assigned memory
spaces and the rest of the system. Better than nothing, but also not great.
2. Clients operating on the same set of buffers with different etnaviv GPU
cores, e.g. a workload using both the 2D and 3D GPU, need to map the used
buffers into the pagetable sets of each used GPU.
This patch reworks all the MMU handling to introduce the abstraction of the
MMU context. A context can be shared across different GPU cores, as long as
they have compatible MMU implementations, which is the case for all systems
with Vivante GPUs seen in the wild.
As MMUv1 is not able to change pagetables on the fly, without a
"stop the world" operation, which stops GPU, changes pagetables via CPU
interaction, restarts GPU, the implementation introduces a shared context on
MMUv1, which is returned whenever there is a request for a new context.
This patch assigns a MMU context to each GPU, so on MMUv2 systems there is
still one set of pagetables per GPU, but due to the shared context MMUv1
systems see a change in behavior as now a single pagetable set is used
across all GPU cores.
Signed-off-by: Lucas Stach <l.stach@pengutronix.de>
Reviewed-by: Philipp Zabel <p.zabel@pengutronix.de>
Reviewed-by: Guido Günther <agx@sigxcpu.org>
2019-07-05 19:17:24 +02:00
|
|
|
if (--global->use > 0)
|
|
|
|
|
return;
|
|
|
|
|
|
|
|
|
|
if (global->v2.pta_cpu)
|
|
|
|
|
dma_free_wc(global->dev, ETNAVIV_PTA_SIZE,
|
|
|
|
|
global->v2.pta_cpu, global->v2.pta_dma);
|
|
|
|
|
|
|
|
|
|
if (global->bad_page_cpu)
|
|
|
|
|
dma_free_wc(global->dev, SZ_4K,
|
|
|
|
|
global->bad_page_cpu, global->bad_page_dma);
|
|
|
|
|
|
|
|
|
|
mutex_destroy(&global->lock);
|
|
|
|
|
kfree(global);
|
|
|
|
|
|
|
|
|
|
priv->mmu_global = NULL;
|
2015-12-03 18:21:29 +01:00
|
|
|
}
|