1.. SPDX-License-Identifier: GPL-2.0
2
3=================
4PCI NTB Function
5=================
6
7:Author: Kishon Vijay Abraham I <kishon@ti.com>
8
9PCI Non-Transparent Bridges (NTB) allow two host systems to communicate
10with each other by exposing each host as a device to the other host.
11NTBs typically support the ability to generate interrupts on the remote
12machine, expose memory ranges as BARs, and perform DMA.  They also support
13scratchpads, which are areas of memory within the NTB that are accessible
14from both machines.
15
16PCI NTB Function allows two different systems (or hosts) to communicate
17with each other by configuring the endpoint instances in such a way that
18transactions from one system are routed to the other system.
19
20In the below diagram, PCI NTB function configures the SoC with multiple
21PCI Endpoint (EP) instances in such a way that transactions from one EP
22controller are routed to the other EP controller. Once PCI NTB function
23configures the SoC with multiple EP instances, HOST1 and HOST2 can
24communicate with each other using SoC as a bridge.
25
26.. code-block:: text
27
28    +-------------+                                   +-------------+
29    |             |                                   |             |
30    |    HOST1    |                                   |    HOST2    |
31    |             |                                   |             |
32    +------^------+                                   +------^------+
33           |                                                 |
34           |                                                 |
35 +---------|-------------------------------------------------|---------+
36 |  +------v------+                                   +------v------+  |
37 |  |             |                                   |             |  |
38 |  |     EP      |                                   |     EP      |  |
39 |  | CONTROLLER1 |                                   | CONTROLLER2 |  |
40 |  |             <----------------------------------->             |  |
41 |  |             |                                   |             |  |
42 |  |             |                                   |             |  |
43 |  |             |  SoC With Multiple EP Instances   |             |  |
44 |  |             |  (Configured using NTB Function)  |             |  |
45 |  +-------------+                                   +-------------+  |
46 +---------------------------------------------------------------------+
47
48Constructs used for Implementing NTB
49====================================
50
51	1) Config Region
52	2) Self Scratchpad Registers
53	3) Peer Scratchpad Registers
54	4) Doorbell (DB) Registers
55	5) Memory Window (MW)
56
57
58Config Region:
59--------------
60
61Config Region is a construct that is specific to NTB implemented using NTB
62Endpoint Function Driver. The host and endpoint side NTB function driver will
63exchange information with each other using this region. Config Region has
64Control/Status Registers for configuring the Endpoint Controller. Host can
65write into this region for configuring the outbound Address Translation Unit
66(ATU) and to indicate the link status. Endpoint can indicate the status of
67commands issued by host in this region. Endpoint can also indicate the
68scratchpad offset and number of memory windows to the host using this region.
69
70The format of Config Region is given below. All the fields here are 32 bits.
71
72.. code-block:: text
73
74	+------------------------+
75	|         COMMAND        |
76	+------------------------+
77	|         ARGUMENT       |
78	+------------------------+
79	|         STATUS         |
80	+------------------------+
81	|         TOPOLOGY       |
82	+------------------------+
83	|    ADDRESS (LOWER 32)  |
84	+------------------------+
85	|    ADDRESS (UPPER 32)  |
86	+------------------------+
87	|           SIZE         |
88	+------------------------+
89	|   NO OF MEMORY WINDOW  |
90	+------------------------+
91	|  MEMORY WINDOW1 OFFSET |
92	+------------------------+
93	|       SPAD OFFSET      |
94	+------------------------+
95	|        SPAD COUNT      |
96	+------------------------+
97	|      DB ENTRY SIZE     |
98	+------------------------+
99	|         DB DATA        |
100	+------------------------+
101	|            :           |
102	+------------------------+
103	|            :           |
104	+------------------------+
105	|         DB DATA        |
106	+------------------------+
107
108
109  COMMAND:
110
111	NTB function supports three commands:
112
113	  CMD_CONFIGURE_DOORBELL (0x1): Command to configure doorbell. Before
114	invoking this command, the host should allocate and initialize
115	MSI/MSI-X vectors (i.e., initialize the MSI/MSI-X Capability in the
116	Endpoint). The endpoint on receiving this command will configure
117	the outbound ATU such that transactions to Doorbell BAR will be routed
118	to the MSI/MSI-X address programmed by the host. The ARGUMENT
119	register should be populated with number of DBs to configure (in the
120	lower 16 bits) and if MSI or MSI-X should be configured (BIT 16).
121
122	  CMD_CONFIGURE_MW (0x2): Command to configure memory window (MW). The
123	host invokes this command after allocating a buffer that can be
124	accessed by remote host. The allocated address should be programmed
125	in the ADDRESS register (64 bit), the size should be programmed in
126	the SIZE register and the memory window index should be programmed
127	in the ARGUMENT register. The endpoint on receiving this command
128	will configure the outbound ATU such that transactions to MW BAR
129	are routed to the address provided by the host.
130
131	  CMD_LINK_UP (0x3): Command to indicate an NTB application is
132	bound to the EP device on the host side. Once the endpoint
133	receives this command from both the hosts, the endpoint will
134	raise a LINK_UP event to both the hosts to indicate the host
135	NTB applications can start communicating with each other.
136
137  ARGUMENT:
138
139	The value of this register is based on the commands issued in
140	command register. See COMMAND section for more information.
141
142  TOPOLOGY:
143
144	Set to NTB_TOPO_B2B_USD for Primary interface
145	Set to NTB_TOPO_B2B_DSD for Secondary interface
146
147  ADDRESS/SIZE:
148
149	Address and Size to be used while configuring the memory window.
150	See "CMD_CONFIGURE_MW" for more info.
151
152  MEMORY WINDOW1 OFFSET:
153
154	Memory Window 1 and Doorbell registers are packed together in the
155	same BAR. The initial portion of the region will have doorbell
156	registers and the latter portion of the region is for memory window 1.
157	This register will specify the offset of the memory window 1.
158
159  NO OF MEMORY WINDOW:
160
161	Specifies the number of memory windows supported by the NTB device.
162
163  SPAD OFFSET:
164
165	Self scratchpad region and config region are packed together in the
166	same BAR. The initial portion of the region will have config region
167	and the latter portion of the region is for self scratchpad. This
168	register will specify the offset of the self scratchpad registers.
169
170  SPAD COUNT:
171
172	Specifies the number of scratchpad registers supported by the NTB
173	device.
174
175  DB ENTRY SIZE:
176
177	Used to determine the offset within the DB BAR that should be written
178	in order to raise doorbell. EPF NTB can use either MSI or MSI-X to
179	ring doorbell (MSI-X support will be added later). MSI uses same
180	address for all the interrupts and MSI-X can provide different
181	addresses for different interrupts. The MSI/MSI-X address is provided
182	by the host and the address it gives is based on the MSI/MSI-X
183	implementation supported by the host. For instance, ARM platform
184	using GIC ITS will have the same MSI-X address for all the interrupts.
185	In order to support all the combinations and use the same mechanism
186	for both MSI and MSI-X, EPF NTB allocates a separate region in the
187	Outbound Address Space for each of the interrupts. This region will
188	be mapped to the MSI/MSI-X address provided by the host. If a host
189	provides the same address for all the interrupts, all the regions
190	will be translated to the same address. If a host provides different
191	addresses, the regions will be translated to different addresses. This
192	will ensure there is no difference while raising the doorbell.
193
194  DB DATA:
195
196	EPF NTB supports 32 interrupts, so there are 32 DB DATA registers.
197	This holds the MSI/MSI-X data that has to be written to MSI address
198	for raising doorbell interrupt. This will be populated by EPF NTB
199	while invoking CMD_CONFIGURE_DOORBELL.
200
201Scratchpad Registers:
202---------------------
203
204  Each host has its own register space allocated in the memory of NTB endpoint
205  controller. They are both readable and writable from both sides of the bridge.
206  They are used by applications built over NTB and can be used to pass control
207  and status information between both sides of a device.
208
209  Scratchpad registers has 2 parts
210	1) Self Scratchpad: Host's own register space
211	2) Peer Scratchpad: Remote host's register space.
212
213Doorbell Registers:
214-------------------
215
216  Doorbell Registers are used by the hosts to interrupt each other.
217
218Memory Window:
219--------------
220
221  Actual transfer of data between the two hosts will happen using the
222  memory window.
223
224Modeling Constructs:
225====================
226
227There are 5 or more distinct regions (config, self scratchpad, peer
228scratchpad, doorbell, one or more memory windows) to be modeled to achieve
229NTB functionality. At least one memory window is required while more than
230one is permitted. All these regions should be mapped to BARs for hosts to
231access these regions.
232
233If one 32-bit BAR is allocated for each of these regions, the scheme would
234look like this:
235
236======  ===============
237BAR NO  CONSTRUCTS USED
238======  ===============
239BAR0    Config Region
240BAR1    Self Scratchpad
241BAR2    Peer Scratchpad
242BAR3    Doorbell
243BAR4    Memory Window 1
244BAR5    Memory Window 2
245======  ===============
246
247However if we allocate a separate BAR for each of the regions, there would not
248be enough BARs for all the regions in a platform that supports only 64-bit
249BARs.
250
251In order to be supported by most of the platforms, the regions should be
252packed and mapped to BARs in a way that provides NTB functionality and
253also makes sure the host doesn't access any region that it is not supposed
254to.
255
256The following scheme is used in EPF NTB Function:
257
258======  ===============================
259BAR NO  CONSTRUCTS USED
260======  ===============================
261BAR0    Config Region + Self Scratchpad
262BAR1    Peer Scratchpad
263BAR2    Doorbell + Memory Window 1
264BAR3    Memory Window 2
265BAR4    Memory Window 3
266BAR5    Memory Window 4
267======  ===============================
268
269With this scheme, for the basic NTB functionality 3 BARs should be sufficient.
270
271Modeling Config/Scratchpad Region:
272----------------------------------
273
274.. code-block:: text
275
276 +-----------------+------->+------------------+        +-----------------+
277 |       BAR0      |        |  CONFIG REGION   |        |       BAR0      |
278 +-----------------+----+   +------------------+<-------+-----------------+
279 |       BAR1      |    |   |SCRATCHPAD REGION |        |       BAR1      |
280 +-----------------+    +-->+------------------+<-------+-----------------+
281 |       BAR2      |            Local Memory            |       BAR2      |
282 +-----------------+                                    +-----------------+
283 |       BAR3      |                                    |       BAR3      |
284 +-----------------+                                    +-----------------+
285 |       BAR4      |                                    |       BAR4      |
286 +-----------------+                                    +-----------------+
287 |       BAR5      |                                    |       BAR5      |
288 +-----------------+                                    +-----------------+
289   EP CONTROLLER 1                                        EP CONTROLLER 2
290
291Above diagram shows Config region + Scratchpad region for HOST1 (connected to
292EP controller 1) allocated in local memory. The HOST1 can access the config
293region and scratchpad region (self scratchpad) using BAR0 of EP controller 1.
294The peer host (HOST2 connected to EP controller 2) can also access this
295scratchpad region (peer scratchpad) using BAR1 of EP controller 2. This
296diagram shows the case where Config region and Scratchpad regions are allocated
297for HOST1, however the same is applicable for HOST2.
298
299Modeling Doorbell/Memory Window 1:
300----------------------------------
301
302.. code-block:: text
303
304 +-----------------+    +----->+----------------+-----------+-----------------+
305 |       BAR0      |    |      |   Doorbell 1   +-----------> MSI-X ADDRESS 1 |
306 +-----------------+    |      +----------------+           +-----------------+
307 |       BAR1      |    |      |   Doorbell 2   +---------+ |                 |
308 +-----------------+----+      +----------------+         | |                 |
309 |       BAR2      |           |   Doorbell 3   +-------+ | +-----------------+
310 +-----------------+----+      +----------------+       | +-> MSI-X ADDRESS 2 |
311 |       BAR3      |    |      |   Doorbell 4   +-----+ |   +-----------------+
312 +-----------------+    |      |----------------+     | |   |                 |
313 |       BAR4      |    |      |                |     | |   +-----------------+
314 +-----------------+    |      |      MW1       +---+ | +-->+ MSI-X ADDRESS 3||
315 |       BAR5      |    |      |                |   | |     +-----------------+
316 +-----------------+    +----->-----------------+   | |     |                 |
317   EP CONTROLLER 1             |                |   | |     +-----------------+
318                               |                |   | +---->+ MSI-X ADDRESS 4 |
319                               +----------------+   |       +-----------------+
320                                EP CONTROLLER 2     |       |                 |
321                                  (OB SPACE)        |       |                 |
322                                                    +------->      MW1        |
323                                                            |                 |
324                                                            |                 |
325                                                            +-----------------+
326                                                            |                 |
327                                                            |                 |
328                                                            |                 |
329                                                            |                 |
330                                                            |                 |
331                                                            +-----------------+
332                                                             PCI Address Space
333                                                             (Managed by HOST2)
334
335Above diagram shows how the doorbell and memory window 1 is mapped so that
336HOST1 can raise doorbell interrupt on HOST2 and also how HOST1 can access
337buffers exposed by HOST2 using memory window1 (MW1). Here doorbell and
338memory window 1 regions are allocated in EP controller 2 outbound (OB) address
339space. Allocating and configuring BARs for doorbell and memory window1
340is done during the initialization phase of NTB endpoint function driver.
341Mapping from EP controller 2 OB space to PCI address space is done when HOST2
342sends CMD_CONFIGURE_MW/CMD_CONFIGURE_DOORBELL.
343
344Modeling Optional Memory Windows:
345---------------------------------
346
347This is modeled the same was as MW1 but each of the additional memory windows
348is mapped to separate BARs.
349