system/devices/cxl.rst

3 From the view of a single host, CXL is an interconnect standard that
4 targets accelerators and memory devices attached to a CXL host.
6 software running on a QEMU emulated host or to the internals of
11 by considering only a single host and a static configuration.
14 with CXL Host Bridges, which have CXL Root Ports which may be directly
26   - Configuration space access
27   - BAR mapped memory accesses used for registers and mailboxes.
28   - MSI/MSI-X
29   - AER
30   - DOE mailboxes
31   - IDE
32   - Many other PCI express defined interfaces..
36   - Equivalent of accessing DRAM / NVDIMMs. Any access / feature
37     supported by the host for normal memory should also work for
46 --------------------
49 **Type 1:** These support coherent caching of host memory.  Example might
53 **Type 2:** These support coherent caching of host memory and host
55 by the host. This is a complex topic, so for more information on CXL
59 additional memory (HDM) to a CXL host including both volatile and
61 number of Type 3 memory devices using HDM Decoders in the host, host
65 ------------------------------
69 emulation challenging with host specific firmware being responsible
76 ----------------------
77 A CXL system is made up a Host with a number of 'standard components'
83 A CFMW consists of a particular range of Host Physical Address space
84 which is routed to particular CXL Host Bridges.  At time of generic
94 CXL Host Bridge (CXL HB)
96 A CXL host bridge is similar to the PCIe equivalent, but with a
97 specification defined register interface called CXL Host Bridge
99 space is described to system software via a CXL Host Bridge
107   a particularly Host Physical Address range to the target port
123 visibility to a particular host is generally the same as for
139 by creating an upstream port (cxl-upstream) and a number of
140 downstream ports on the internal switch bus (cxl-downstream).
142 CXL Memory Devices - Type 3
147 routing but for translation of the incoming host physical address (HPA)
151 ---------------------
162   |<------------------SYSTEM PHYSICAL ADDRESS MAP (1)----------------->|
201       | from host PA  | | PCI 0e:00.0 | | PCI df:00.0| | PCI e0:00.0  |
210     particular interleave setup across the CXL Host Bridges (HB)
215 (2) **Two CXL Host Bridges**. Each of these has 2 CXL Root Ports and
242     they will take the Host Physical Addresses of accesses and map
247   |<------------------SYSTEM PHYSICAL ADDRESS MAP (1)----------------->|
282             ---------------------------------------------------
296       | from host PA  | | PCI 10:00.0 | | PCI 11:00.0| | PCI 12:00.0  |
302 ---------------------
305   qemu-system-x86_64 -M q35,cxl=on -m 4G,maxmem=8G,slots=8 -smp 4 \
307   -object memory-backend-file,id=cxl-mem1,share=on,mem-path=/tmp/cxltest.raw,size=256M \
308   -object memory-backend-file,id=cxl-lsa1,share=on,mem-path=/tmp/lsa.raw,size=256M \
309   -device pxb-cxl,bus_nr=12,bus=pcie.0,id=cxl.1 \
310   -device cxl-rp,port=0,bus=cxl.1,id=root_port13,chassis=0,slot=2 \
311   -device cxl-type3,bus=root_port13,persistent-memdev=cxl-mem1,lsa=cxl-lsa1,id=cxl-pmem0 \
312   -M cxl-fmw.0.targets.0=cxl.1,cxl-fmw.0.size=4G
316   qemu-system-x86_64 -M q35,cxl=on -m 4G,maxmem=8G,slots=8 -smp 4 \
318   -object memory-backend-ram,id=vmem0,share=on,size=256M \
319   -device pxb-cxl,bus_nr=12,bus=pcie.0,id=cxl.1 \
320   -device cxl-rp,port=0,bus=cxl.1,id=root_port13,chassis=0,slot=2 \
321   -device cxl-type3,bus=root_port13,volatile-memdev=vmem0,id=cxl-vmem0 \
322   -M cxl-fmw.0.targets.0=cxl.1,cxl-fmw.0.size=4G
326   qemu-system-x86_64 -M q35,cxl=on -m 4G,maxmem=8G,slots=8 -smp 4 \
328   -object memory-backend-ram,id=vmem0,share=on,size=256M \
329   -object memory-backend-file,id=cxl-lsa0,share=on,mem-path=/tmp/lsa.raw,size=256M \
330   -device pxb-cxl,bus_nr=12,bus=pcie.0,id=cxl.1 \
331   -device cxl-rp,port=0,bus=cxl.1,id=root_port13,chassis=0,slot=2 \
332   -device cxl-type3,bus=root_port13,volatile-memdev=vmem0,lsa=cxl-lsa0,id=cxl-vmem0 \
333   -M cxl-fmw.0.targets.0=cxl.1,cxl-fmw.0.size=4G
336 interleave across 2 CXL host bridges.  Each host bridge has 2 CXL Root Ports, with
339   qemu-system-x86_64 -M q35,cxl=on -m 4G,maxmem=8G,slots=8 -smp 4 \
341   -object memory-backend-file,id=cxl-mem1,share=on,mem-path=/tmp/cxltest.raw,size=256M \
342   -object memory-backend-file,id=cxl-mem2,share=on,mem-path=/tmp/cxltest2.raw,size=256M \
343   -object memory-backend-file,id=cxl-mem3,share=on,mem-path=/tmp/cxltest3.raw,size=256M \
344   -object memory-backend-file,id=cxl-mem4,share=on,mem-path=/tmp/cxltest4.raw,size=256M \
345   -object memory-backend-file,id=cxl-lsa1,share=on,mem-path=/tmp/lsa.raw,size=256M \
346   -object memory-backend-file,id=cxl-lsa2,share=on,mem-path=/tmp/lsa2.raw,size=256M \
347   -object memory-backend-file,id=cxl-lsa3,share=on,mem-path=/tmp/lsa3.raw,size=256M \
348   -object memory-backend-file,id=cxl-lsa4,share=on,mem-path=/tmp/lsa4.raw,size=256M \
349   -device pxb-cxl,bus_nr=12,bus=pcie.0,id=cxl.1 \
350   -device pxb-cxl,bus_nr=222,bus=pcie.0,id=cxl.2 \
351   -device cxl-rp,port=0,bus=cxl.1,id=root_port13,chassis=0,slot=2 \
352   -device cxl-type3,bus=root_port13,persistent-memdev=cxl-mem1,lsa=cxl-lsa1,id=cxl-pmem0 \
353   -device cxl-rp,port=1,bus=cxl.1,id=root_port14,chassis=0,slot=3 \
354   -device cxl-type3,bus=root_port14,persistent-memdev=cxl-mem2,lsa=cxl-lsa2,id=cxl-pmem1 \
355   -device cxl-rp,port=0,bus=cxl.2,id=root_port15,chassis=0,slot=5 \
356   -device cxl-type3,bus=root_port15,persistent-memdev=cxl-mem3,lsa=cxl-lsa3,id=cxl-pmem2 \
357   -device cxl-rp,port=1,bus=cxl.2,id=root_port16,chassis=0,slot=6 \
358   -device cxl-type3,bus=root_port16,persistent-memdev=cxl-mem4,lsa=cxl-lsa4,id=cxl-pmem3 \
359 …-M cxl-fmw.0.targets.0=cxl.1,cxl-fmw.0.targets.1=cxl.2,cxl-fmw.0.size=4G,cxl-fmw.0.interleave-gran…
363   qemu-system-x86_64 -M q35,cxl=on -m 4G,maxmem=8G,slots=8 -smp 4 \
365   -object memory-backend-file,id=cxl-mem0,share=on,mem-path=/tmp/cxltest.raw,size=256M \
366   -object memory-backend-file,id=cxl-mem1,share=on,mem-path=/tmp/cxltest1.raw,size=256M \
367   -object memory-backend-file,id=cxl-mem2,share=on,mem-path=/tmp/cxltest2.raw,size=256M \
368   -object memory-backend-file,id=cxl-mem3,share=on,mem-path=/tmp/cxltest3.raw,size=256M \
369   -object memory-backend-file,id=cxl-lsa0,share=on,mem-path=/tmp/lsa0.raw,size=256M \
370   -object memory-backend-file,id=cxl-lsa1,share=on,mem-path=/tmp/lsa1.raw,size=256M \
371   -object memory-backend-file,id=cxl-lsa2,share=on,mem-path=/tmp/lsa2.raw,size=256M \
372   -object memory-backend-file,id=cxl-lsa3,share=on,mem-path=/tmp/lsa3.raw,size=256M \
373   -device pxb-cxl,bus_nr=12,bus=pcie.0,id=cxl.1 \
374   -device cxl-rp,port=0,bus=cxl.1,id=root_port0,chassis=0,slot=0 \
375   -device cxl-rp,port=1,bus=cxl.1,id=root_port1,chassis=0,slot=1 \
376   -device cxl-upstream,bus=root_port0,id=us0 \
377   -device cxl-downstream,port=0,bus=us0,id=swport0,chassis=0,slot=4 \
378   -device cxl-type3,bus=swport0,persistent-memdev=cxl-mem0,lsa=cxl-lsa0,id=cxl-pmem0 \
379   -device cxl-downstream,port=1,bus=us0,id=swport1,chassis=0,slot=5 \
380   -device cxl-type3,bus=swport1,persistent-memdev=cxl-mem1,lsa=cxl-lsa1,id=cxl-pmem1 \
381   -device cxl-downstream,port=2,bus=us0,id=swport2,chassis=0,slot=6 \
382   -device cxl-type3,bus=swport2,persistent-memdev=cxl-mem2,lsa=cxl-lsa2,id=cxl-pmem2 \
383   -device cxl-downstream,port=3,bus=us0,id=swport3,chassis=0,slot=7 \
384   -device cxl-type3,bus=swport3,persistent-memdev=cxl-mem3,lsa=cxl-lsa3,id=cxl-pmem3 \
385   -M cxl-fmw.0.targets.0=cxl.1,cxl-fmw.0.size=4G,cxl-fmw.0.interleave-granularity=4k
388 ------------
391 [persistent-memdev] attributes. [memdev] will default to a persistent memory
393 with [persistent-memdev].
396 ----------------------------
410 ----------
412  - Consortium website for specifications etc:
414  - Compute Express Link (CXL) Specification, Revision 3.1, August 2023