drm/nouveau: nvkm/vmm: implement raw ops to manage uvmm

The new VM_BIND UAPI uses the DRM GPU VA manager to manage the VA space.
Hence, we a need a way to manipulate the MMUs page tables without goi

drm/nouveau: nvkm/vmm: implement raw ops to manage uvmm

The new VM_BIND UAPI uses the DRM GPU VA manager to manage the VA space.
Hence, we a need a way to manipulate the MMUs page tables without going
through the internal range allocator implemented by nvkm/vmm.

This patch adds a raw interface for nvkm/vmm to pass the resposibility
for managing the address space and the corresponding map/unmap/sparse
operations to the upper layers.

Reviewed-by: Dave Airlie <airlied@redhat.com>
Signed-off-by: Danilo Krummrich <dakr@redhat.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20230804182406.5222-11-dakr@redhat.com

show more ...

drm/nouveau/mmu/gp100: remove unused variable

Fixes a compilation issue introduced because I forgot to test with WERROR
enabled.

Cc: Stephen Rothwell <sfr@canb.auug.org.au>
Cc: DRI <dri-devel@lists

drm/nouveau/mmu/gp100: remove unused variable

Fixes a compilation issue introduced because I forgot to test with WERROR
enabled.

Cc: Stephen Rothwell <sfr@canb.auug.org.au>
Cc: DRI <dri-devel@lists.freedesktop.org>
Cc: nouveau@lists.freedesktop.org
Fixes: 404046cf4805 ("drm/nouveau/mmu/gp100-: drop unneeded assignment in the if condition.")
Acked-by: Adam Jackson <ajax@redhat.com>
Reviewed-by: Lyude Paul <lyude@redhat.com>
Signed-off-by: Karol Herbst <kherbst@redhat.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20211012133334.1737918-1-kherbst@redhat.com

show more ...

drm/nouveau/mmu/gp100-: drop unneeded assignment in the if condition.

In order to keep the code style consistency of the whole file,
the 'inst' assignments should be deleted.

The clang_analyzer com

drm/nouveau/mmu/gp100-: drop unneeded assignment in the if condition.

In order to keep the code style consistency of the whole file,
the 'inst' assignments should be deleted.

The clang_analyzer complains as follows:

drivers/gpu/drm/nouveau/nvkm/subdev/mmu/vmmgp100.c:499:8: warning:
Although the value storedto 'inst' is used in the enclosing expression,
the value is never actually read from 'inst'.

Karol: Removed unneeded brackets.

Reported-by: Zeal Robot <zealci@zte.com.cn>
Signed-off-by: Luo penghao <luo.penghao@zte.com.cn>
Reviewed-by: Karol Herbst <kherbst@redhat.com>
Signed-off-by: Karol Herbst <kherbst@redhat.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20210821021447.27097-1-luo.penghao@zte.com.cn

show more ...

drm/nouveau: rip out nvkm_client.super

No longer required now that userspace can't touch anything that might
need it, and should fix DRM MM operations racing with each other, and
the random hangs/cr

drm/nouveau: rip out nvkm_client.super

No longer required now that userspace can't touch anything that might
need it, and should fix DRM MM operations racing with each other, and
the random hangs/crashes that come with that.

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>
Reviewed-by: Lyude Paul <lyude@redhat.com>

show more ...

nouveau/svm: implement atomic SVM access

Some NVIDIA GPUs do not support direct atomic access to system memory via
PCIe. Instead this must be emulated by granting the GPU exclusive access
to the me

nouveau/svm: implement atomic SVM access

Some NVIDIA GPUs do not support direct atomic access to system memory via
PCIe. Instead this must be emulated by granting the GPU exclusive access
to the memory. This is achieved by replacing CPU page table entries with
special swap entries that fault on userspace access.

The driver then grants the GPU permission to update the page undergoing
atomic access via the GPU page tables. When CPU access to the page is
required a CPU fault is raised which calls into the device driver via MMU
notifiers to revoke the atomic access. The original page table entries
are then restored allowing CPU access to proceed.

Link: https://lkml.kernel.org/r/20210616105937.23201-11-apopple@nvidia.com
Signed-off-by: Alistair Popple <apopple@nvidia.com>
Reviewed-by: Ben Skeggs <bskeggs@redhat.com>
Cc: Christoph Hellwig <hch@lst.de>
Cc: Hugh Dickins <hughd@google.com>
Cc: Jason Gunthorpe <jgg@nvidia.com>
Cc: John Hubbard <jhubbard@nvidia.com>
Cc: "Matthew Wilcox (Oracle)" <willy@infradead.org>
Cc: Peter Xu <peterx@redhat.com>
Cc: Ralph Campbell <rcampbell@nvidia.com>
Cc: Shakeel Butt <shakeelb@google.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>

show more ...

nouveau: fix storing invalid ptes

When migrating a range of system memory to device private memory, some of
the pages in the address range may not be migrating. In this case, the non
migrating pages

nouveau: fix storing invalid ptes

When migrating a range of system memory to device private memory, some of
the pages in the address range may not be migrating. In this case, the non
migrating pages won't have a new GPU MMU entry to store but the
nvif_object_ioctl() NVIF_VMM_V0_PFNMAP method doesn't check the input and
stores a bad valid GPU page table entry.

Fix this by skipping the invalid input PTEs when updating the GPU page
tables.

Link: https://lore.kernel.org/r/20200723223004.9586-2-rcampbell@nvidia.com
Signed-off-by: Ralph Campbell <rcampbell@nvidia.com>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>

show more ...

drm/nouveau/vmm/gp100-: fix mapping 2MB sysmem pages

The nvif_object_ioctl() method NVIF_VMM_V0_PFNMAP wasn't correctly
setting the hardware specific GPU page table entries for 2MB sized
pages. Fix

drm/nouveau/vmm/gp100-: fix mapping 2MB sysmem pages

The nvif_object_ioctl() method NVIF_VMM_V0_PFNMAP wasn't correctly
setting the hardware specific GPU page table entries for 2MB sized
pages. Fix this by adding functions to set and clear PD0 GPU page
table entries.

Signed-off-by: Ralph Campbell <rcampbell@nvidia.com>
Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

show more ...

drm/nouveau/mmu/gp100-: enable mmu invalidate depth optimisation

This causes us to invalidate MMU only at the level we made modifications -
ie: if we've only modified PTEs, there's no need to have M

drm/nouveau/mmu/gp100-: enable mmu invalidate depth optimisation

This causes us to invalidate MMU only at the level we made modifications -
ie: if we've only modified PTEs, there's no need to have MMU dump the PDs
it's fetched into L2.

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

show more ...

nouveau: fix mapping 2MB sysmem pages

The nvif_object_ioctl() method NVIF_VMM_V0_PFNMAP wasn't correctly setting
the hardware specific GPU page table entries for 2MB sized pages. Fix this
by adding

nouveau: fix mapping 2MB sysmem pages

The nvif_object_ioctl() method NVIF_VMM_V0_PFNMAP wasn't correctly setting
the hardware specific GPU page table entries for 2MB sized pages. Fix this
by adding functions to set and clear PD0 GPU page table entries.

Link: https://lore.kernel.org/r/20200701225352.9649-4-rcampbell@nvidia.com
Signed-off-by: Ralph Campbell <rcampbell@nvidia.com>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>

show more ...

drm/nouveau/mmu: Add correct turing page kinds

Turing introduced a new simplified page kind
scheme, reducing the number of possible page
kinds from 256 to 16. It also is the first
NVIDIA GPU in whi

drm/nouveau/mmu: Add correct turing page kinds

Turing introduced a new simplified page kind
scheme, reducing the number of possible page
kinds from 256 to 16. It also is the first
NVIDIA GPU in which the highest possible page
kind value is not reserved as an "invalid" page
kind.

To address this, the invalid page kind is made
an explicit property of the MMU HAL, and a new
table of page kinds is added to the tu102 MMU
HAL.

One hardware change not addressed here is that
0x00 is technically no longer a supported page
kind, and pitch surfaces are instead intended to
share the block-linear generic page kind 0x06.
However, because that will be a rather invasive
change to nouveau and 0x00 still works fine in
practice on Turing hardware, addressing this new
behavior is deferred.

Signed-off-by: James Jones <jajones@nvidia.com>
Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

show more ...

drm/nouveau/mmu/gp100-: support vmms with gcc/tex replayable faults enabled

Some GPU units are capable of supporting "replayable" page faults, where
the execution unit will wait for SW to fixup GPU

drm/nouveau/mmu/gp100-: support vmms with gcc/tex replayable faults enabled

Some GPU units are capable of supporting "replayable" page faults, where
the execution unit will wait for SW to fixup GPU page tables rather than
triggering a channel-fatal fault.

This feature isn't useful (it's harmful, even) unless something like HMM
is being used to manage events appearing in the replayable fault buffer,
so, it's disabled by default.

This commit allows a client to request it be enabled.

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

show more ...

drm/nouveau/mmu/gp100-: add privileged methods for fault replay/cancel

Host methods exist to do at least some of what we need, but we are not
currently pushing replay/cancels through a channel like

drm/nouveau/mmu/gp100-: add privileged methods for fault replay/cancel

Host methods exist to do at least some of what we need, but we are not
currently pushing replay/cancels through a channel like UVM does as it's
not clear whether it's necessary in our case (UVM also updates PTEs with
the GPU).

UVM also pushes a software method for fault cancels on Pascal, seemingly
because the host methods don't appear to be sufficient. If/when we want
to push the replay/cancel on the GPU, we can re-purpose the cancellation
code here to implement that swmthd.

Keep it simple for now, until we figure out exactly what we need here.

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

show more ...

drm/nouveau/mmu: add a privileged method to directly manage PTEs

This provides a somewhat more direct method of manipulating the GPU page
tables, which will be required to support SVM.

Signed-off-b

drm/nouveau/mmu: add a privileged method to directly manage PTEs

This provides a somewhat more direct method of manipulating the GPU page
tables, which will be required to support SVM.

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

show more ...

drm/nouveau/mmu: support initialisation of client-managed address-spaces

NVKM is currently responsible for managing the allocation of a client's
GPU address-space, but there's various use-cases (ie.

drm/nouveau/mmu: support initialisation of client-managed address-spaces

NVKM is currently responsible for managing the allocation of a client's
GPU address-space, but there's various use-cases (ie. HMM address-space
mirroring) where giving a client more direct control is desirable.

This commit allows for a VMM to be created where the area allocated for
NVKM is limited to a client-specified window, the remainder of address-
space is controlled directly by the client.

Leaving a window is necessary to support various internal requirements,
but also to support existing allocation interfaces as not all of the HW
is capable of working with a HMM allocation.

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

show more ...

drm/nouveau/mmu/gf100-: virtualise setting pdb base address for invalidation

It appears that Pascal and newer need something different.

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

drm/nouveau/mmu/gf100-: make mmu invalidate function more general

Will want to reuse this for fault replay/cancellation swmthds.

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

drm/nouveau/mmu/gp100,gp10b: implement new vmm backend

Adds support for:
- 64KiB/2MiB big page sizes (128KiB not supported by HW with new PT layout).
- System-memory PTs.
- LPTE "invalid" state.
- (

drm/nouveau/mmu/gp100,gp10b: implement new vmm backend

Adds support for:
- 64KiB/2MiB big page sizes (128KiB not supported by HW with new PT layout).
- System-memory PTs.
- LPTE "invalid" state.
- (Tegra) Use of video memory aperture.
- Sparse PDEs/PTEs.
- Additional blocklinear kinds.
- 49-bit address-space.

GP100 supports an entirely new 5-level page table layout that provides
an expanded 49-bit address-space. It also supports the layout present
on previous generations, which we've been making do with until now.

This commit implements support for the new layout, and enables it by
default.

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

show more ...

drm/nouveau/mmu/gp100,gp10b: implement vmm on top of new base

Adds support for:
- Selection of old/new-style page table layout (GP100MmuLayout=0/1).
- System-memory PDs.

New layout disabled by defa

drm/nouveau/mmu/gp100,gp10b: implement vmm on top of new base

Adds support for:
- Selection of old/new-style page table layout (GP100MmuLayout=0/1).
- System-memory PDs.

New layout disabled by default for the moment, as we don't have a
backend that can handle it yet.

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

show more ...

Merge tag 'drm-next-2020-08-06' of git://anongit.freedesktop.org/drm/drm

Pull drm updates from Dave Airlie:
"New xilinx displayport driver, AMD support for two new GPUs (more
head

Merge tag 'drm-next-2020-08-06' of git://anongit.freedesktop.org/drm/drm

Pull drm updates from Dave Airlie:
"New xilinx displayport driver, AMD support for two new GPUs (more
header files), i915 initial support for RocketLake and some work on
their DG1 (discrete chip).

The core also grew some lockdep annotations to try and constrain what
drivers do with dma-fences, and added some documentation on why the
idea of indefinite fences doesn't work.

The long list is below.

I do have some fixes trees outstanding, but I'll follow up with those
later.

core:
- add user def flag to cmd line modes
- dma_fence_wait added might_sleep
- dma-fence lockdep annotations
- indefinite fences are bad documentation
- gem CMA functions used in more drivers
- struct mutex removal
- more drm_ debug macro usage
- set/drop master api fixes
- fix for drm/mm hole size comparison
- drm/mm remove invalid entry optimization
- optimise drm/mm hole handling
- VRR debugfs added
- uncompressed AFBC modifier support
- multiple display id blocks in EDID
- multiple driver sg handling fixes
- __drm_atomic_helper_crtc_reset in all drivers
- managed vram helpers

ttm:
- ttm_mem_reg handling cleanup
- remove bo offset field
- drop CMA memtype flag
- drop mappable flag

xilinx:
- New Xilinx ZynqMP DisplayPort Subsystem driver

nouveau:
- add CRC support
- start using NVIDIA published class header files
- convert all push buffer emission to new macros
- Proper push buffer space management for EVO/NVD channels.
- firmware loading fixes
- 2MiB system memory pages support on Pascal and newer

vkms:
- larger cursor support

i915:
- Rocketlake platform enablement
- Early DG1 enablement
- Numerous GEM refactorings
- DP MST fixes
- FBC, PSR, Cursor, Color, Gamma fixes
- TGL, RKL, EHL workaround updates
- TGL 8K display support fixes
- SDVO/HDMI/DVI fixes

amdgpu:
- Initial support for Sienna Cichlid GPU
- Initial support for Navy Flounder GPU
- SI UVD/VCE support
- expose rotation property
- Add support for unique id on Arcturus
- Enable runtime PM on vega10 boards that support BACO
- Skip BAR resizing if the bios already did id
- Major swSMU code cleanup
- Fixes for DCN bandwidth calculations

amdkfd:
- Track SDMA usage per process
- SMI events interface

radeon:
- Default to on chip GART for AGP boards on all arches
- Runtime PM reference count fixes

msm:
- headers regenerated causing churn
- a650/a640 display and GPU enablement
- dpu dither support for 6bpc panels
- dpu cursor fix
- dsi/mdp5 enablement for sdm630/sdm636/sdm66

tegra:
- video capture prep support
- reflection support

mediatek:
- convert mtk_dsi to bridge API

meson:
- FBC support

sun4i:
- iommu support

rockchip:
- register locking fix
- per-pixel alpha support PX30 VOP

mgag200:
- ported to simple and shmem helpers
- device init cleanups
- use managed pci functions
- dropped hw cursor support

ast:
- use managed pci functions
- use managed VRAM helpers
- rework cursor support

malidp:
- dev_groups support

hibmc:
- refactor hibmc_drv_vdac:

vc4:
- create TXP CRTC

imx:
- error path fixes and cleanups

etnaviv:
- clock handling and error handling cleanups
- use pin_user_pages"

* tag 'drm-next-2020-08-06' of git://anongit.freedesktop.org/drm/drm: (1747 commits)
drm/msm: use kthread_create_worker instead of kthread_run
drm/msm/mdp5: Add MDP5 configuration for SDM636/660
drm/msm/dsi: Add DSI configuration for SDM660
drm/msm/mdp5: Add MDP5 configuration for SDM630
drm/msm/dsi: Add phy configuration for SDM630/636/660
drm/msm/a6xx: add A640/A650 hwcg
drm/msm/a6xx: hwcg tables in gpulist
drm/msm/dpu: add SM8250 to hw catalog
drm/msm/dpu: add SM8150 to hw catalog
drm/msm/dpu: intf timing path for displayport
drm/msm/dpu: set missing flush bits for INTF_2 and INTF_3
drm/msm/dpu: don't use INTF_INPUT_CTRL feature on sdm845
drm/msm/dpu: move some sspp caps to dpu_caps
drm/msm/dpu: update UBWC config for sm8150 and sm8250
drm/msm/dpu: use right setup_blend_config for sm8150 and sm8250
drm/msm/a6xx: set ubwc config for A640 and A650
drm/msm/adreno: un-open-code some packets
drm/msm: sync generated headers
drm/msm/a6xx: add build_bw_table for A640/A650
drm/msm/a6xx: fix crashstate capture for A650
...

show more ...

nouveau: fix storing invalid ptes

When migrating a range of system memory to device private memory, some of
the pages in the address range may not be migrating. In this case, the non

nouveau: fix storing invalid ptes

When migrating a range of system memory to device private memory, some of
the pages in the address range may not be migrating. In this case, the non
migrating pages won't have a new GPU MMU entry to store but the
nvif_object_ioctl() NVIF_VMM_V0_PFNMAP method doesn't check the input and
stores a bad valid GPU page table entry.

Fix this by skipping the invalid input PTEs when updating the GPU page
tables.

Link: https://lore.kernel.org/r/20200723223004.9586-2-rcampbell@nvidia.com
Signed-off-by: Ralph Campbell <rcampbell@nvidia.com>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>

show more ...

drm/nouveau/vmm/gp100-: fix mapping 2MB sysmem pages

The nvif_object_ioctl() method NVIF_VMM_V0_PFNMAP wasn't correctly
setting the hardware specific GPU page table entries for 2MB sized

drm/nouveau/vmm/gp100-: fix mapping 2MB sysmem pages

The nvif_object_ioctl() method NVIF_VMM_V0_PFNMAP wasn't correctly
setting the hardware specific GPU page table entries for 2MB sized
pages. Fix this by adding functions to set and clear PD0 GPU page
table entries.

Signed-off-by: Ralph Campbell <rcampbell@nvidia.com>
Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

show more ...

drm/nouveau/mmu/gp100-: enable mmu invalidate depth optimisation

This causes us to invalidate MMU only at the level we made modifications -
ie: if we've only modified PTEs, there's no ne

drm/nouveau/mmu/gp100-: enable mmu invalidate depth optimisation

This causes us to invalidate MMU only at the level we made modifications -
ie: if we've only modified PTEs, there's no need to have MMU dump the PDs
it's fetched into L2.

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

show more ...

nouveau: fix mapping 2MB sysmem pages

The nvif_object_ioctl() method NVIF_VMM_V0_PFNMAP wasn't correctly setting
the hardware specific GPU page table entries for 2MB sized pages. Fix thi

nouveau: fix mapping 2MB sysmem pages

The nvif_object_ioctl() method NVIF_VMM_V0_PFNMAP wasn't correctly setting
the hardware specific GPU page table entries for 2MB sized pages. Fix this
by adding functions to set and clear PD0 GPU page table entries.

Link: https://lore.kernel.org/r/20200701225352.9649-4-rcampbell@nvidia.com
Signed-off-by: Ralph Campbell <rcampbell@nvidia.com>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>

show more ...

drm/nouveau/mmu: Add correct turing page kinds

Turing introduced a new simplified page kind
scheme, reducing the number of possible page
kinds from 256 to 16. It also is the first

drm/nouveau/mmu: Add correct turing page kinds

Turing introduced a new simplified page kind
scheme, reducing the number of possible page
kinds from 256 to 16. It also is the first
NVIDIA GPU in which the highest possible page
kind value is not reserved as an "invalid" page
kind.

To address this, the invalid page kind is made
an explicit property of the MMU HAL, and a new
table of page kinds is added to the tu102 MMU
HAL.

One hardware change not addressed here is that
0x00 is technically no longer a supported page
kind, and pitch surfaces are instead intended to
share the block-linear generic page kind 0x06.
However, because that will be a rather invasive
change to nouveau and 0x00 still works fine in
practice on Turing hardware, addressing this new
behavior is deferred.

Signed-off-by: James Jones <jajones@nvidia.com>
Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

show more ...

drm/nouveau/mmu/gp100-: support vmms with gcc/tex replayable faults enabled

Some GPU units are capable of supporting "replayable" page faults, where
the execution unit will wait for SW t

drm/nouveau/mmu/gp100-: support vmms with gcc/tex replayable faults enabled

Some GPU units are capable of supporting "replayable" page faults, where
the execution unit will wait for SW to fixup GPU page tables rather than
triggering a channel-fatal fault.

This feature isn't useful (it's harmful, even) unless something like HMM
is being used to manage events appearing in the replayable fault buffer,
so, it's disabled by default.

This commit allows a client to request it be enabled.

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

show more ...