1==================================
2VDUSE - "vDPA Device in Userspace"
3==================================
4
5vDPA (virtio data path acceleration) device is a device that uses a
6datapath which complies with the virtio specifications with vendor
7specific control path. vDPA devices can be both physically located on
8the hardware or emulated by software. VDUSE is a framework that makes it
9possible to implement software-emulated vDPA devices in userspace. And
10to make the device emulation more secure, the emulated vDPA device's
11control path is handled in the kernel and only the data path is
12implemented in the userspace.
13
14Note that only virtio block device is supported by VDUSE framework now,
15which can reduce security risks when the userspace process that implements
16the data path is run by an unprivileged user. The support for other device
17types can be added after the security issue of corresponding device driver
18is clarified or fixed in the future.
19
20Create/Destroy VDUSE devices
21----------------------------
22
23VDUSE devices are created as follows:
24
251. Create a new VDUSE instance with ioctl(VDUSE_CREATE_DEV) on
26   /dev/vduse/control.
27
282. Setup each virtqueue with ioctl(VDUSE_VQ_SETUP) on /dev/vduse/$NAME.
29
303. Begin processing VDUSE messages from /dev/vduse/$NAME. The first
31   messages will arrive while attaching the VDUSE instance to vDPA bus.
32
334. Send the VDPA_CMD_DEV_NEW netlink message to attach the VDUSE
34   instance to vDPA bus.
35
36VDUSE devices are destroyed as follows:
37
381. Send the VDPA_CMD_DEV_DEL netlink message to detach the VDUSE
39   instance from vDPA bus.
40
412. Close the file descriptor referring to /dev/vduse/$NAME.
42
433. Destroy the VDUSE instance with ioctl(VDUSE_DESTROY_DEV) on
44   /dev/vduse/control.
45
46The netlink messages can be sent via vdpa tool in iproute2 or use the
47below sample codes:
48
49.. code-block:: c
50
51	static int netlink_add_vduse(const char *name, enum vdpa_command cmd)
52	{
53		struct nl_sock *nlsock;
54		struct nl_msg *msg;
55		int famid;
56
57		nlsock = nl_socket_alloc();
58		if (!nlsock)
59			return -ENOMEM;
60
61		if (genl_connect(nlsock))
62			goto free_sock;
63
64		famid = genl_ctrl_resolve(nlsock, VDPA_GENL_NAME);
65		if (famid < 0)
66			goto close_sock;
67
68		msg = nlmsg_alloc();
69		if (!msg)
70			goto close_sock;
71
72		if (!genlmsg_put(msg, NL_AUTO_PORT, NL_AUTO_SEQ, famid, 0, 0, cmd, 0))
73			goto nla_put_failure;
74
75		NLA_PUT_STRING(msg, VDPA_ATTR_DEV_NAME, name);
76		if (cmd == VDPA_CMD_DEV_NEW)
77			NLA_PUT_STRING(msg, VDPA_ATTR_MGMTDEV_DEV_NAME, "vduse");
78
79		if (nl_send_sync(nlsock, msg))
80			goto close_sock;
81
82		nl_close(nlsock);
83		nl_socket_free(nlsock);
84
85		return 0;
86	nla_put_failure:
87		nlmsg_free(msg);
88	close_sock:
89		nl_close(nlsock);
90	free_sock:
91		nl_socket_free(nlsock);
92		return -1;
93	}
94
95How VDUSE works
96---------------
97
98As mentioned above, a VDUSE device is created by ioctl(VDUSE_CREATE_DEV) on
99/dev/vduse/control. With this ioctl, userspace can specify some basic configuration
100such as device name (uniquely identify a VDUSE device), virtio features, virtio
101configuration space, the number of virtqueues and so on for this emulated device.
102Then a char device interface (/dev/vduse/$NAME) is exported to userspace for device
103emulation. Userspace can use the VDUSE_VQ_SETUP ioctl on /dev/vduse/$NAME to
104add per-virtqueue configuration such as the max size of virtqueue to the device.
105
106After the initialization, the VDUSE device can be attached to vDPA bus via
107the VDPA_CMD_DEV_NEW netlink message. Userspace needs to read()/write() on
108/dev/vduse/$NAME to receive/reply some control messages from/to VDUSE kernel
109module as follows:
110
111.. code-block:: c
112
113	static int vduse_message_handler(int dev_fd)
114	{
115		int len;
116		struct vduse_dev_request req;
117		struct vduse_dev_response resp;
118
119		len = read(dev_fd, &req, sizeof(req));
120		if (len != sizeof(req))
121			return -1;
122
123		resp.request_id = req.request_id;
124
125		switch (req.type) {
126
127		/* handle different types of messages */
128
129		}
130
131		len = write(dev_fd, &resp, sizeof(resp));
132		if (len != sizeof(resp))
133			return -1;
134
135		return 0;
136	}
137
138There are now three types of messages introduced by VDUSE framework:
139
140- VDUSE_GET_VQ_STATE: Get the state for virtqueue, userspace should return
141  avail index for split virtqueue or the device/driver ring wrap counters and
142  the avail and used index for packed virtqueue.
143
144- VDUSE_SET_STATUS: Set the device status, userspace should follow
145  the virtio spec: https://docs.oasis-open.org/virtio/virtio/v1.1/virtio-v1.1.html
146  to process this message. For example, fail to set the FEATURES_OK device
147  status bit if the device can not accept the negotiated virtio features
148  get from the VDUSE_DEV_GET_FEATURES ioctl.
149
150- VDUSE_UPDATE_IOTLB: Notify userspace to update the memory mapping for specified
151  IOVA range, userspace should firstly remove the old mapping, then setup the new
152  mapping via the VDUSE_IOTLB_GET_FD ioctl.
153
154After DRIVER_OK status bit is set via the VDUSE_SET_STATUS message, userspace is
155able to start the dataplane processing as follows:
156
1571. Get the specified virtqueue's information with the VDUSE_VQ_GET_INFO ioctl,
158   including the size, the IOVAs of descriptor table, available ring and used ring,
159   the state and the ready status.
160
1612. Pass the above IOVAs to the VDUSE_IOTLB_GET_FD ioctl so that those IOVA regions
162   can be mapped into userspace. Some sample codes is shown below:
163
164.. code-block:: c
165
166	static int perm_to_prot(uint8_t perm)
167	{
168		int prot = 0;
169
170		switch (perm) {
171		case VDUSE_ACCESS_WO:
172			prot |= PROT_WRITE;
173			break;
174		case VDUSE_ACCESS_RO:
175			prot |= PROT_READ;
176			break;
177		case VDUSE_ACCESS_RW:
178			prot |= PROT_READ | PROT_WRITE;
179			break;
180		}
181
182		return prot;
183	}
184
185	static void *iova_to_va(int dev_fd, uint64_t iova, uint64_t *len)
186	{
187		int fd;
188		void *addr;
189		size_t size;
190		struct vduse_iotlb_entry entry;
191
192		entry.start = iova;
193		entry.last = iova;
194
195		/*
196		 * Find the first IOVA region that overlaps with the specified
197		 * range [start, last] and return the corresponding file descriptor.
198		 */
199		fd = ioctl(dev_fd, VDUSE_IOTLB_GET_FD, &entry);
200		if (fd < 0)
201			return NULL;
202
203		size = entry.last - entry.start + 1;
204		*len = entry.last - iova + 1;
205		addr = mmap(0, size, perm_to_prot(entry.perm), MAP_SHARED,
206			    fd, entry.offset);
207		close(fd);
208		if (addr == MAP_FAILED)
209			return NULL;
210
211		/*
212		 * Using some data structures such as linked list to store
213		 * the iotlb mapping. The munmap(2) should be called for the
214		 * cached mapping when the corresponding VDUSE_UPDATE_IOTLB
215		 * message is received or the device is reset.
216		 */
217
218		return addr + iova - entry.start;
219	}
220
2213. Setup the kick eventfd for the specified virtqueues with the VDUSE_VQ_SETUP_KICKFD
222   ioctl. The kick eventfd is used by VDUSE kernel module to notify userspace to
223   consume the available ring. This is optional since userspace can choose to poll the
224   available ring instead.
225
2264. Listen to the kick eventfd (optional) and consume the available ring. The buffer
227   described by the descriptors in the descriptor table should be also mapped into
228   userspace via the VDUSE_IOTLB_GET_FD ioctl before accessing.
229
2305. Inject an interrupt for specific virtqueue with the VDUSE_INJECT_VQ_IRQ ioctl
231   after the used ring is filled.
232
233For more details on the uAPI, please see include/uapi/linux/vduse.h.
234