106164d2bSGeorge Zhang /* 206164d2bSGeorge Zhang * VMware VMCI Driver 306164d2bSGeorge Zhang * 406164d2bSGeorge Zhang * Copyright (C) 2012 VMware, Inc. All rights reserved. 506164d2bSGeorge Zhang * 606164d2bSGeorge Zhang * This program is free software; you can redistribute it and/or modify it 706164d2bSGeorge Zhang * under the terms of the GNU General Public License as published by the 806164d2bSGeorge Zhang * Free Software Foundation version 2 and no later version. 906164d2bSGeorge Zhang * 1006164d2bSGeorge Zhang * This program is distributed in the hope that it will be useful, but 1106164d2bSGeorge Zhang * WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY 1206164d2bSGeorge Zhang * or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License 1306164d2bSGeorge Zhang * for more details. 1406164d2bSGeorge Zhang */ 1506164d2bSGeorge Zhang 1606164d2bSGeorge Zhang #include <linux/vmw_vmci_defs.h> 1706164d2bSGeorge Zhang #include <linux/vmw_vmci_api.h> 1842281d20SAndy King #include <linux/highmem.h> 1906164d2bSGeorge Zhang #include <linux/kernel.h> 2042281d20SAndy King #include <linux/mm.h> 2106164d2bSGeorge Zhang #include <linux/module.h> 2206164d2bSGeorge Zhang #include <linux/mutex.h> 2342281d20SAndy King #include <linux/pagemap.h> 2442281d20SAndy King #include <linux/sched.h> 2542281d20SAndy King #include <linux/slab.h> 2606164d2bSGeorge Zhang #include <linux/socket.h> 2706164d2bSGeorge Zhang #include <linux/wait.h> 2806164d2bSGeorge Zhang 2906164d2bSGeorge Zhang #include "vmci_handle_array.h" 3006164d2bSGeorge Zhang #include "vmci_queue_pair.h" 3106164d2bSGeorge Zhang #include "vmci_datagram.h" 3206164d2bSGeorge Zhang #include "vmci_resource.h" 3306164d2bSGeorge Zhang #include "vmci_context.h" 3406164d2bSGeorge Zhang #include "vmci_driver.h" 3506164d2bSGeorge Zhang #include "vmci_event.h" 3606164d2bSGeorge Zhang #include "vmci_route.h" 3706164d2bSGeorge Zhang 3806164d2bSGeorge Zhang /* 3906164d2bSGeorge Zhang * In the following, we will distinguish between two kinds of VMX processes - 4006164d2bSGeorge Zhang * the ones with versions lower than VMCI_VERSION_NOVMVM that use specialized 4106164d2bSGeorge Zhang * VMCI page files in the VMX and supporting VM to VM communication and the 4206164d2bSGeorge Zhang * newer ones that use the guest memory directly. We will in the following 4306164d2bSGeorge Zhang * refer to the older VMX versions as old-style VMX'en, and the newer ones as 4406164d2bSGeorge Zhang * new-style VMX'en. 4506164d2bSGeorge Zhang * 4606164d2bSGeorge Zhang * The state transition datagram is as follows (the VMCIQPB_ prefix has been 4706164d2bSGeorge Zhang * removed for readability) - see below for more details on the transtions: 4806164d2bSGeorge Zhang * 4906164d2bSGeorge Zhang * -------------- NEW ------------- 5006164d2bSGeorge Zhang * | | 5106164d2bSGeorge Zhang * \_/ \_/ 5206164d2bSGeorge Zhang * CREATED_NO_MEM <-----------------> CREATED_MEM 5306164d2bSGeorge Zhang * | | | 5406164d2bSGeorge Zhang * | o-----------------------o | 5506164d2bSGeorge Zhang * | | | 5606164d2bSGeorge Zhang * \_/ \_/ \_/ 5706164d2bSGeorge Zhang * ATTACHED_NO_MEM <----------------> ATTACHED_MEM 5806164d2bSGeorge Zhang * | | | 5906164d2bSGeorge Zhang * | o----------------------o | 6006164d2bSGeorge Zhang * | | | 6106164d2bSGeorge Zhang * \_/ \_/ \_/ 6206164d2bSGeorge Zhang * SHUTDOWN_NO_MEM <----------------> SHUTDOWN_MEM 6306164d2bSGeorge Zhang * | | 6406164d2bSGeorge Zhang * | | 6506164d2bSGeorge Zhang * -------------> gone <------------- 6606164d2bSGeorge Zhang * 6706164d2bSGeorge Zhang * In more detail. When a VMCI queue pair is first created, it will be in the 6806164d2bSGeorge Zhang * VMCIQPB_NEW state. It will then move into one of the following states: 6906164d2bSGeorge Zhang * 7006164d2bSGeorge Zhang * - VMCIQPB_CREATED_NO_MEM: this state indicates that either: 7106164d2bSGeorge Zhang * 7206164d2bSGeorge Zhang * - the created was performed by a host endpoint, in which case there is 7306164d2bSGeorge Zhang * no backing memory yet. 7406164d2bSGeorge Zhang * 7506164d2bSGeorge Zhang * - the create was initiated by an old-style VMX, that uses 7606164d2bSGeorge Zhang * vmci_qp_broker_set_page_store to specify the UVAs of the queue pair at 7706164d2bSGeorge Zhang * a later point in time. This state can be distinguished from the one 7806164d2bSGeorge Zhang * above by the context ID of the creator. A host side is not allowed to 7906164d2bSGeorge Zhang * attach until the page store has been set. 8006164d2bSGeorge Zhang * 8106164d2bSGeorge Zhang * - VMCIQPB_CREATED_MEM: this state is the result when the queue pair 8206164d2bSGeorge Zhang * is created by a VMX using the queue pair device backend that 8306164d2bSGeorge Zhang * sets the UVAs of the queue pair immediately and stores the 8406164d2bSGeorge Zhang * information for later attachers. At this point, it is ready for 8506164d2bSGeorge Zhang * the host side to attach to it. 8606164d2bSGeorge Zhang * 8706164d2bSGeorge Zhang * Once the queue pair is in one of the created states (with the exception of 8806164d2bSGeorge Zhang * the case mentioned for older VMX'en above), it is possible to attach to the 8906164d2bSGeorge Zhang * queue pair. Again we have two new states possible: 9006164d2bSGeorge Zhang * 9106164d2bSGeorge Zhang * - VMCIQPB_ATTACHED_MEM: this state can be reached through the following 9206164d2bSGeorge Zhang * paths: 9306164d2bSGeorge Zhang * 9406164d2bSGeorge Zhang * - from VMCIQPB_CREATED_NO_MEM when a new-style VMX allocates a queue 9506164d2bSGeorge Zhang * pair, and attaches to a queue pair previously created by the host side. 9606164d2bSGeorge Zhang * 9706164d2bSGeorge Zhang * - from VMCIQPB_CREATED_MEM when the host side attaches to a queue pair 9806164d2bSGeorge Zhang * already created by a guest. 9906164d2bSGeorge Zhang * 10006164d2bSGeorge Zhang * - from VMCIQPB_ATTACHED_NO_MEM, when an old-style VMX calls 10106164d2bSGeorge Zhang * vmci_qp_broker_set_page_store (see below). 10206164d2bSGeorge Zhang * 10306164d2bSGeorge Zhang * - VMCIQPB_ATTACHED_NO_MEM: If the queue pair already was in the 10406164d2bSGeorge Zhang * VMCIQPB_CREATED_NO_MEM due to a host side create, an old-style VMX will 10506164d2bSGeorge Zhang * bring the queue pair into this state. Once vmci_qp_broker_set_page_store 10606164d2bSGeorge Zhang * is called to register the user memory, the VMCIQPB_ATTACH_MEM state 10706164d2bSGeorge Zhang * will be entered. 10806164d2bSGeorge Zhang * 10906164d2bSGeorge Zhang * From the attached queue pair, the queue pair can enter the shutdown states 11006164d2bSGeorge Zhang * when either side of the queue pair detaches. If the guest side detaches 11106164d2bSGeorge Zhang * first, the queue pair will enter the VMCIQPB_SHUTDOWN_NO_MEM state, where 11206164d2bSGeorge Zhang * the content of the queue pair will no longer be available. If the host 11306164d2bSGeorge Zhang * side detaches first, the queue pair will either enter the 11406164d2bSGeorge Zhang * VMCIQPB_SHUTDOWN_MEM, if the guest memory is currently mapped, or 11506164d2bSGeorge Zhang * VMCIQPB_SHUTDOWN_NO_MEM, if the guest memory is not mapped 11606164d2bSGeorge Zhang * (e.g., the host detaches while a guest is stunned). 11706164d2bSGeorge Zhang * 11806164d2bSGeorge Zhang * New-style VMX'en will also unmap guest memory, if the guest is 11906164d2bSGeorge Zhang * quiesced, e.g., during a snapshot operation. In that case, the guest 12006164d2bSGeorge Zhang * memory will no longer be available, and the queue pair will transition from 12106164d2bSGeorge Zhang * *_MEM state to a *_NO_MEM state. The VMX may later map the memory once more, 12206164d2bSGeorge Zhang * in which case the queue pair will transition from the *_NO_MEM state at that 12306164d2bSGeorge Zhang * point back to the *_MEM state. Note that the *_NO_MEM state may have changed, 12406164d2bSGeorge Zhang * since the peer may have either attached or detached in the meantime. The 12506164d2bSGeorge Zhang * values are laid out such that ++ on a state will move from a *_NO_MEM to a 12606164d2bSGeorge Zhang * *_MEM state, and vice versa. 12706164d2bSGeorge Zhang */ 12806164d2bSGeorge Zhang 12906164d2bSGeorge Zhang /* 13006164d2bSGeorge Zhang * VMCIMemcpy{To,From}QueueFunc() prototypes. Functions of these 13106164d2bSGeorge Zhang * types are passed around to enqueue and dequeue routines. Note that 13206164d2bSGeorge Zhang * often the functions passed are simply wrappers around memcpy 13306164d2bSGeorge Zhang * itself. 13406164d2bSGeorge Zhang * 13506164d2bSGeorge Zhang * Note: In order for the memcpy typedefs to be compatible with the VMKernel, 13606164d2bSGeorge Zhang * there's an unused last parameter for the hosted side. In 13706164d2bSGeorge Zhang * ESX, that parameter holds a buffer type. 13806164d2bSGeorge Zhang */ 13906164d2bSGeorge Zhang typedef int vmci_memcpy_to_queue_func(struct vmci_queue *queue, 14006164d2bSGeorge Zhang u64 queue_offset, const void *src, 14106164d2bSGeorge Zhang size_t src_offset, size_t size); 14206164d2bSGeorge Zhang typedef int vmci_memcpy_from_queue_func(void *dest, size_t dest_offset, 14306164d2bSGeorge Zhang const struct vmci_queue *queue, 14406164d2bSGeorge Zhang u64 queue_offset, size_t size); 14506164d2bSGeorge Zhang 14606164d2bSGeorge Zhang /* The Kernel specific component of the struct vmci_queue structure. */ 14706164d2bSGeorge Zhang struct vmci_queue_kern_if { 14806164d2bSGeorge Zhang struct page **page; 14906164d2bSGeorge Zhang struct page **header_page; 15006164d2bSGeorge Zhang void *va; 15106164d2bSGeorge Zhang struct mutex __mutex; /* Protects the queue. */ 15206164d2bSGeorge Zhang struct mutex *mutex; /* Shared by producer and consumer queues. */ 15306164d2bSGeorge Zhang bool host; 15406164d2bSGeorge Zhang size_t num_pages; 15506164d2bSGeorge Zhang bool mapped; 15606164d2bSGeorge Zhang }; 15706164d2bSGeorge Zhang 15806164d2bSGeorge Zhang /* 15906164d2bSGeorge Zhang * This structure is opaque to the clients. 16006164d2bSGeorge Zhang */ 16106164d2bSGeorge Zhang struct vmci_qp { 16206164d2bSGeorge Zhang struct vmci_handle handle; 16306164d2bSGeorge Zhang struct vmci_queue *produce_q; 16406164d2bSGeorge Zhang struct vmci_queue *consume_q; 16506164d2bSGeorge Zhang u64 produce_q_size; 16606164d2bSGeorge Zhang u64 consume_q_size; 16706164d2bSGeorge Zhang u32 peer; 16806164d2bSGeorge Zhang u32 flags; 16906164d2bSGeorge Zhang u32 priv_flags; 17006164d2bSGeorge Zhang bool guest_endpoint; 17106164d2bSGeorge Zhang unsigned int blocked; 17206164d2bSGeorge Zhang unsigned int generation; 17306164d2bSGeorge Zhang wait_queue_head_t event; 17406164d2bSGeorge Zhang }; 17506164d2bSGeorge Zhang 17606164d2bSGeorge Zhang enum qp_broker_state { 17706164d2bSGeorge Zhang VMCIQPB_NEW, 17806164d2bSGeorge Zhang VMCIQPB_CREATED_NO_MEM, 17906164d2bSGeorge Zhang VMCIQPB_CREATED_MEM, 18006164d2bSGeorge Zhang VMCIQPB_ATTACHED_NO_MEM, 18106164d2bSGeorge Zhang VMCIQPB_ATTACHED_MEM, 18206164d2bSGeorge Zhang VMCIQPB_SHUTDOWN_NO_MEM, 18306164d2bSGeorge Zhang VMCIQPB_SHUTDOWN_MEM, 18406164d2bSGeorge Zhang VMCIQPB_GONE 18506164d2bSGeorge Zhang }; 18606164d2bSGeorge Zhang 18706164d2bSGeorge Zhang #define QPBROKERSTATE_HAS_MEM(_qpb) (_qpb->state == VMCIQPB_CREATED_MEM || \ 18806164d2bSGeorge Zhang _qpb->state == VMCIQPB_ATTACHED_MEM || \ 18906164d2bSGeorge Zhang _qpb->state == VMCIQPB_SHUTDOWN_MEM) 19006164d2bSGeorge Zhang 19106164d2bSGeorge Zhang /* 19206164d2bSGeorge Zhang * In the queue pair broker, we always use the guest point of view for 19306164d2bSGeorge Zhang * the produce and consume queue values and references, e.g., the 19406164d2bSGeorge Zhang * produce queue size stored is the guests produce queue size. The 19506164d2bSGeorge Zhang * host endpoint will need to swap these around. The only exception is 19606164d2bSGeorge Zhang * the local queue pairs on the host, in which case the host endpoint 19706164d2bSGeorge Zhang * that creates the queue pair will have the right orientation, and 19806164d2bSGeorge Zhang * the attaching host endpoint will need to swap. 19906164d2bSGeorge Zhang */ 20006164d2bSGeorge Zhang struct qp_entry { 20106164d2bSGeorge Zhang struct list_head list_item; 20206164d2bSGeorge Zhang struct vmci_handle handle; 20306164d2bSGeorge Zhang u32 peer; 20406164d2bSGeorge Zhang u32 flags; 20506164d2bSGeorge Zhang u64 produce_size; 20606164d2bSGeorge Zhang u64 consume_size; 20706164d2bSGeorge Zhang u32 ref_count; 20806164d2bSGeorge Zhang }; 20906164d2bSGeorge Zhang 21006164d2bSGeorge Zhang struct qp_broker_entry { 21106164d2bSGeorge Zhang struct vmci_resource resource; 21206164d2bSGeorge Zhang struct qp_entry qp; 21306164d2bSGeorge Zhang u32 create_id; 21406164d2bSGeorge Zhang u32 attach_id; 21506164d2bSGeorge Zhang enum qp_broker_state state; 21606164d2bSGeorge Zhang bool require_trusted_attach; 21706164d2bSGeorge Zhang bool created_by_trusted; 21806164d2bSGeorge Zhang bool vmci_page_files; /* Created by VMX using VMCI page files */ 21906164d2bSGeorge Zhang struct vmci_queue *produce_q; 22006164d2bSGeorge Zhang struct vmci_queue *consume_q; 22106164d2bSGeorge Zhang struct vmci_queue_header saved_produce_q; 22206164d2bSGeorge Zhang struct vmci_queue_header saved_consume_q; 22306164d2bSGeorge Zhang vmci_event_release_cb wakeup_cb; 22406164d2bSGeorge Zhang void *client_data; 22506164d2bSGeorge Zhang void *local_mem; /* Kernel memory for local queue pair */ 22606164d2bSGeorge Zhang }; 22706164d2bSGeorge Zhang 22806164d2bSGeorge Zhang struct qp_guest_endpoint { 22906164d2bSGeorge Zhang struct vmci_resource resource; 23006164d2bSGeorge Zhang struct qp_entry qp; 23106164d2bSGeorge Zhang u64 num_ppns; 23206164d2bSGeorge Zhang void *produce_q; 23306164d2bSGeorge Zhang void *consume_q; 234e6389a13SDmitry Torokhov struct ppn_set ppn_set; 23506164d2bSGeorge Zhang }; 23606164d2bSGeorge Zhang 23706164d2bSGeorge Zhang struct qp_list { 23806164d2bSGeorge Zhang struct list_head head; 23906164d2bSGeorge Zhang struct mutex mutex; /* Protect queue list. */ 24006164d2bSGeorge Zhang }; 24106164d2bSGeorge Zhang 24206164d2bSGeorge Zhang static struct qp_list qp_broker_list = { 24306164d2bSGeorge Zhang .head = LIST_HEAD_INIT(qp_broker_list.head), 24406164d2bSGeorge Zhang .mutex = __MUTEX_INITIALIZER(qp_broker_list.mutex), 24506164d2bSGeorge Zhang }; 24606164d2bSGeorge Zhang 24706164d2bSGeorge Zhang static struct qp_list qp_guest_endpoints = { 24806164d2bSGeorge Zhang .head = LIST_HEAD_INIT(qp_guest_endpoints.head), 24906164d2bSGeorge Zhang .mutex = __MUTEX_INITIALIZER(qp_guest_endpoints.mutex), 25006164d2bSGeorge Zhang }; 25106164d2bSGeorge Zhang 25206164d2bSGeorge Zhang #define INVALID_VMCI_GUEST_MEM_ID 0 25306164d2bSGeorge Zhang #define QPE_NUM_PAGES(_QPE) ((u32) \ 25442281d20SAndy King (DIV_ROUND_UP(_QPE.produce_size, PAGE_SIZE) + \ 25542281d20SAndy King DIV_ROUND_UP(_QPE.consume_size, PAGE_SIZE) + 2)) 25606164d2bSGeorge Zhang 25706164d2bSGeorge Zhang 25806164d2bSGeorge Zhang /* 25906164d2bSGeorge Zhang * Frees kernel VA space for a given queue and its queue header, and 26006164d2bSGeorge Zhang * frees physical data pages. 26106164d2bSGeorge Zhang */ 26206164d2bSGeorge Zhang static void qp_free_queue(void *q, u64 size) 26306164d2bSGeorge Zhang { 26406164d2bSGeorge Zhang struct vmci_queue *queue = q; 26506164d2bSGeorge Zhang 26606164d2bSGeorge Zhang if (queue) { 26742281d20SAndy King u64 i = DIV_ROUND_UP(size, PAGE_SIZE); 26806164d2bSGeorge Zhang 26906164d2bSGeorge Zhang if (queue->kernel_if->mapped) { 27006164d2bSGeorge Zhang vunmap(queue->kernel_if->va); 27106164d2bSGeorge Zhang queue->kernel_if->va = NULL; 27206164d2bSGeorge Zhang } 27306164d2bSGeorge Zhang 27406164d2bSGeorge Zhang while (i) 27506164d2bSGeorge Zhang __free_page(queue->kernel_if->page[--i]); 27606164d2bSGeorge Zhang 27706164d2bSGeorge Zhang vfree(queue->q_header); 27806164d2bSGeorge Zhang } 27906164d2bSGeorge Zhang } 28006164d2bSGeorge Zhang 28106164d2bSGeorge Zhang /* 28206164d2bSGeorge Zhang * Allocates kernel VA space of specified size, plus space for the 28306164d2bSGeorge Zhang * queue structure/kernel interface and the queue header. Allocates 28406164d2bSGeorge Zhang * physical pages for the queue data pages. 28506164d2bSGeorge Zhang * 28606164d2bSGeorge Zhang * PAGE m: struct vmci_queue_header (struct vmci_queue->q_header) 28706164d2bSGeorge Zhang * PAGE m+1: struct vmci_queue 28806164d2bSGeorge Zhang * PAGE m+1+q: struct vmci_queue_kern_if (struct vmci_queue->kernel_if) 28906164d2bSGeorge Zhang * PAGE n-size: Data pages (struct vmci_queue->kernel_if->page[]) 29006164d2bSGeorge Zhang */ 29106164d2bSGeorge Zhang static void *qp_alloc_queue(u64 size, u32 flags) 29206164d2bSGeorge Zhang { 29306164d2bSGeorge Zhang u64 i; 29406164d2bSGeorge Zhang struct vmci_queue *queue; 29506164d2bSGeorge Zhang struct vmci_queue_header *q_header; 29642281d20SAndy King const u64 num_data_pages = DIV_ROUND_UP(size, PAGE_SIZE); 29706164d2bSGeorge Zhang const uint queue_size = 29806164d2bSGeorge Zhang PAGE_SIZE + 29906164d2bSGeorge Zhang sizeof(*queue) + sizeof(*(queue->kernel_if)) + 30006164d2bSGeorge Zhang num_data_pages * sizeof(*(queue->kernel_if->page)); 30106164d2bSGeorge Zhang 30206164d2bSGeorge Zhang q_header = vmalloc(queue_size); 30306164d2bSGeorge Zhang if (!q_header) 30406164d2bSGeorge Zhang return NULL; 30506164d2bSGeorge Zhang 30606164d2bSGeorge Zhang queue = (void *)q_header + PAGE_SIZE; 30706164d2bSGeorge Zhang queue->q_header = q_header; 30806164d2bSGeorge Zhang queue->saved_header = NULL; 30906164d2bSGeorge Zhang queue->kernel_if = (struct vmci_queue_kern_if *)(queue + 1); 31006164d2bSGeorge Zhang queue->kernel_if->header_page = NULL; /* Unused in guest. */ 31106164d2bSGeorge Zhang queue->kernel_if->page = (struct page **)(queue->kernel_if + 1); 31206164d2bSGeorge Zhang queue->kernel_if->host = false; 31306164d2bSGeorge Zhang queue->kernel_if->va = NULL; 31406164d2bSGeorge Zhang queue->kernel_if->mapped = false; 31506164d2bSGeorge Zhang 31606164d2bSGeorge Zhang for (i = 0; i < num_data_pages; i++) { 31706164d2bSGeorge Zhang queue->kernel_if->page[i] = alloc_pages(GFP_KERNEL, 0); 31806164d2bSGeorge Zhang if (!queue->kernel_if->page[i]) 31906164d2bSGeorge Zhang goto fail; 32006164d2bSGeorge Zhang } 32106164d2bSGeorge Zhang 32206164d2bSGeorge Zhang if (vmci_qp_pinned(flags)) { 32306164d2bSGeorge Zhang queue->kernel_if->va = 32406164d2bSGeorge Zhang vmap(queue->kernel_if->page, num_data_pages, VM_MAP, 32506164d2bSGeorge Zhang PAGE_KERNEL); 32606164d2bSGeorge Zhang if (!queue->kernel_if->va) 32706164d2bSGeorge Zhang goto fail; 32806164d2bSGeorge Zhang 32906164d2bSGeorge Zhang queue->kernel_if->mapped = true; 33006164d2bSGeorge Zhang } 33106164d2bSGeorge Zhang 33206164d2bSGeorge Zhang return (void *)queue; 33306164d2bSGeorge Zhang 33406164d2bSGeorge Zhang fail: 33506164d2bSGeorge Zhang qp_free_queue(queue, i * PAGE_SIZE); 33606164d2bSGeorge Zhang return NULL; 33706164d2bSGeorge Zhang } 33806164d2bSGeorge Zhang 33906164d2bSGeorge Zhang /* 34006164d2bSGeorge Zhang * Copies from a given buffer or iovector to a VMCI Queue. Uses 34106164d2bSGeorge Zhang * kmap()/kunmap() to dynamically map/unmap required portions of the queue 34206164d2bSGeorge Zhang * by traversing the offset -> page translation structure for the queue. 34306164d2bSGeorge Zhang * Assumes that offset + size does not wrap around in the queue. 34406164d2bSGeorge Zhang */ 34506164d2bSGeorge Zhang static int __qp_memcpy_to_queue(struct vmci_queue *queue, 34606164d2bSGeorge Zhang u64 queue_offset, 34706164d2bSGeorge Zhang const void *src, 34806164d2bSGeorge Zhang size_t size, 34906164d2bSGeorge Zhang bool is_iovec) 35006164d2bSGeorge Zhang { 35106164d2bSGeorge Zhang struct vmci_queue_kern_if *kernel_if = queue->kernel_if; 35206164d2bSGeorge Zhang size_t bytes_copied = 0; 35306164d2bSGeorge Zhang 35406164d2bSGeorge Zhang while (bytes_copied < size) { 35506164d2bSGeorge Zhang u64 page_index = (queue_offset + bytes_copied) / PAGE_SIZE; 35606164d2bSGeorge Zhang size_t page_offset = 35706164d2bSGeorge Zhang (queue_offset + bytes_copied) & (PAGE_SIZE - 1); 35806164d2bSGeorge Zhang void *va; 35906164d2bSGeorge Zhang size_t to_copy; 36006164d2bSGeorge Zhang 36106164d2bSGeorge Zhang if (!kernel_if->mapped) 36206164d2bSGeorge Zhang va = kmap(kernel_if->page[page_index]); 36306164d2bSGeorge Zhang else 36406164d2bSGeorge Zhang va = (void *)((u8 *)kernel_if->va + 36506164d2bSGeorge Zhang (page_index * PAGE_SIZE)); 36606164d2bSGeorge Zhang 36706164d2bSGeorge Zhang if (size - bytes_copied > PAGE_SIZE - page_offset) 36806164d2bSGeorge Zhang /* Enough payload to fill up from this page. */ 36906164d2bSGeorge Zhang to_copy = PAGE_SIZE - page_offset; 37006164d2bSGeorge Zhang else 37106164d2bSGeorge Zhang to_copy = size - bytes_copied; 37206164d2bSGeorge Zhang 37306164d2bSGeorge Zhang if (is_iovec) { 37406164d2bSGeorge Zhang struct iovec *iov = (struct iovec *)src; 37506164d2bSGeorge Zhang int err; 37606164d2bSGeorge Zhang 37706164d2bSGeorge Zhang /* The iovec will track bytes_copied internally. */ 37806164d2bSGeorge Zhang err = memcpy_fromiovec((u8 *)va + page_offset, 37906164d2bSGeorge Zhang iov, to_copy); 38006164d2bSGeorge Zhang if (err != 0) { 38106164d2bSGeorge Zhang kunmap(kernel_if->page[page_index]); 38206164d2bSGeorge Zhang return VMCI_ERROR_INVALID_ARGS; 38306164d2bSGeorge Zhang } 38406164d2bSGeorge Zhang } else { 38506164d2bSGeorge Zhang memcpy((u8 *)va + page_offset, 38606164d2bSGeorge Zhang (u8 *)src + bytes_copied, to_copy); 38706164d2bSGeorge Zhang } 38806164d2bSGeorge Zhang 38906164d2bSGeorge Zhang bytes_copied += to_copy; 39006164d2bSGeorge Zhang if (!kernel_if->mapped) 39106164d2bSGeorge Zhang kunmap(kernel_if->page[page_index]); 39206164d2bSGeorge Zhang } 39306164d2bSGeorge Zhang 39406164d2bSGeorge Zhang return VMCI_SUCCESS; 39506164d2bSGeorge Zhang } 39606164d2bSGeorge Zhang 39706164d2bSGeorge Zhang /* 39806164d2bSGeorge Zhang * Copies to a given buffer or iovector from a VMCI Queue. Uses 39906164d2bSGeorge Zhang * kmap()/kunmap() to dynamically map/unmap required portions of the queue 40006164d2bSGeorge Zhang * by traversing the offset -> page translation structure for the queue. 40106164d2bSGeorge Zhang * Assumes that offset + size does not wrap around in the queue. 40206164d2bSGeorge Zhang */ 40306164d2bSGeorge Zhang static int __qp_memcpy_from_queue(void *dest, 40406164d2bSGeorge Zhang const struct vmci_queue *queue, 40506164d2bSGeorge Zhang u64 queue_offset, 40606164d2bSGeorge Zhang size_t size, 40706164d2bSGeorge Zhang bool is_iovec) 40806164d2bSGeorge Zhang { 40906164d2bSGeorge Zhang struct vmci_queue_kern_if *kernel_if = queue->kernel_if; 41006164d2bSGeorge Zhang size_t bytes_copied = 0; 41106164d2bSGeorge Zhang 41206164d2bSGeorge Zhang while (bytes_copied < size) { 41306164d2bSGeorge Zhang u64 page_index = (queue_offset + bytes_copied) / PAGE_SIZE; 41406164d2bSGeorge Zhang size_t page_offset = 41506164d2bSGeorge Zhang (queue_offset + bytes_copied) & (PAGE_SIZE - 1); 41606164d2bSGeorge Zhang void *va; 41706164d2bSGeorge Zhang size_t to_copy; 41806164d2bSGeorge Zhang 41906164d2bSGeorge Zhang if (!kernel_if->mapped) 42006164d2bSGeorge Zhang va = kmap(kernel_if->page[page_index]); 42106164d2bSGeorge Zhang else 42206164d2bSGeorge Zhang va = (void *)((u8 *)kernel_if->va + 42306164d2bSGeorge Zhang (page_index * PAGE_SIZE)); 42406164d2bSGeorge Zhang 42506164d2bSGeorge Zhang if (size - bytes_copied > PAGE_SIZE - page_offset) 42606164d2bSGeorge Zhang /* Enough payload to fill up this page. */ 42706164d2bSGeorge Zhang to_copy = PAGE_SIZE - page_offset; 42806164d2bSGeorge Zhang else 42906164d2bSGeorge Zhang to_copy = size - bytes_copied; 43006164d2bSGeorge Zhang 43106164d2bSGeorge Zhang if (is_iovec) { 43206164d2bSGeorge Zhang struct iovec *iov = (struct iovec *)dest; 43306164d2bSGeorge Zhang int err; 43406164d2bSGeorge Zhang 43506164d2bSGeorge Zhang /* The iovec will track bytes_copied internally. */ 43606164d2bSGeorge Zhang err = memcpy_toiovec(iov, (u8 *)va + page_offset, 43706164d2bSGeorge Zhang to_copy); 43806164d2bSGeorge Zhang if (err != 0) { 43906164d2bSGeorge Zhang kunmap(kernel_if->page[page_index]); 44006164d2bSGeorge Zhang return VMCI_ERROR_INVALID_ARGS; 44106164d2bSGeorge Zhang } 44206164d2bSGeorge Zhang } else { 44306164d2bSGeorge Zhang memcpy((u8 *)dest + bytes_copied, 44406164d2bSGeorge Zhang (u8 *)va + page_offset, to_copy); 44506164d2bSGeorge Zhang } 44606164d2bSGeorge Zhang 44706164d2bSGeorge Zhang bytes_copied += to_copy; 44806164d2bSGeorge Zhang if (!kernel_if->mapped) 44906164d2bSGeorge Zhang kunmap(kernel_if->page[page_index]); 45006164d2bSGeorge Zhang } 45106164d2bSGeorge Zhang 45206164d2bSGeorge Zhang return VMCI_SUCCESS; 45306164d2bSGeorge Zhang } 45406164d2bSGeorge Zhang 45506164d2bSGeorge Zhang /* 45606164d2bSGeorge Zhang * Allocates two list of PPNs --- one for the pages in the produce queue, 45706164d2bSGeorge Zhang * and the other for the pages in the consume queue. Intializes the list 45806164d2bSGeorge Zhang * of PPNs with the page frame numbers of the KVA for the two queues (and 45906164d2bSGeorge Zhang * the queue headers). 46006164d2bSGeorge Zhang */ 46106164d2bSGeorge Zhang static int qp_alloc_ppn_set(void *prod_q, 46206164d2bSGeorge Zhang u64 num_produce_pages, 46306164d2bSGeorge Zhang void *cons_q, 464e6389a13SDmitry Torokhov u64 num_consume_pages, struct ppn_set *ppn_set) 46506164d2bSGeorge Zhang { 46606164d2bSGeorge Zhang u32 *produce_ppns; 46706164d2bSGeorge Zhang u32 *consume_ppns; 46806164d2bSGeorge Zhang struct vmci_queue *produce_q = prod_q; 46906164d2bSGeorge Zhang struct vmci_queue *consume_q = cons_q; 47006164d2bSGeorge Zhang u64 i; 47106164d2bSGeorge Zhang 47206164d2bSGeorge Zhang if (!produce_q || !num_produce_pages || !consume_q || 47306164d2bSGeorge Zhang !num_consume_pages || !ppn_set) 47406164d2bSGeorge Zhang return VMCI_ERROR_INVALID_ARGS; 47506164d2bSGeorge Zhang 47606164d2bSGeorge Zhang if (ppn_set->initialized) 47706164d2bSGeorge Zhang return VMCI_ERROR_ALREADY_EXISTS; 47806164d2bSGeorge Zhang 47906164d2bSGeorge Zhang produce_ppns = 48006164d2bSGeorge Zhang kmalloc(num_produce_pages * sizeof(*produce_ppns), GFP_KERNEL); 48106164d2bSGeorge Zhang if (!produce_ppns) 48206164d2bSGeorge Zhang return VMCI_ERROR_NO_MEM; 48306164d2bSGeorge Zhang 48406164d2bSGeorge Zhang consume_ppns = 48506164d2bSGeorge Zhang kmalloc(num_consume_pages * sizeof(*consume_ppns), GFP_KERNEL); 48606164d2bSGeorge Zhang if (!consume_ppns) { 48706164d2bSGeorge Zhang kfree(produce_ppns); 48806164d2bSGeorge Zhang return VMCI_ERROR_NO_MEM; 48906164d2bSGeorge Zhang } 49006164d2bSGeorge Zhang 49106164d2bSGeorge Zhang produce_ppns[0] = page_to_pfn(vmalloc_to_page(produce_q->q_header)); 49206164d2bSGeorge Zhang for (i = 1; i < num_produce_pages; i++) { 49306164d2bSGeorge Zhang unsigned long pfn; 49406164d2bSGeorge Zhang 49506164d2bSGeorge Zhang produce_ppns[i] = 49606164d2bSGeorge Zhang page_to_pfn(produce_q->kernel_if->page[i - 1]); 49706164d2bSGeorge Zhang pfn = produce_ppns[i]; 49806164d2bSGeorge Zhang 49906164d2bSGeorge Zhang /* Fail allocation if PFN isn't supported by hypervisor. */ 50006164d2bSGeorge Zhang if (sizeof(pfn) > sizeof(*produce_ppns) 50106164d2bSGeorge Zhang && pfn != produce_ppns[i]) 50206164d2bSGeorge Zhang goto ppn_error; 50306164d2bSGeorge Zhang } 50406164d2bSGeorge Zhang 50506164d2bSGeorge Zhang consume_ppns[0] = page_to_pfn(vmalloc_to_page(consume_q->q_header)); 50606164d2bSGeorge Zhang for (i = 1; i < num_consume_pages; i++) { 50706164d2bSGeorge Zhang unsigned long pfn; 50806164d2bSGeorge Zhang 50906164d2bSGeorge Zhang consume_ppns[i] = 51006164d2bSGeorge Zhang page_to_pfn(consume_q->kernel_if->page[i - 1]); 51106164d2bSGeorge Zhang pfn = consume_ppns[i]; 51206164d2bSGeorge Zhang 51306164d2bSGeorge Zhang /* Fail allocation if PFN isn't supported by hypervisor. */ 51406164d2bSGeorge Zhang if (sizeof(pfn) > sizeof(*consume_ppns) 51506164d2bSGeorge Zhang && pfn != consume_ppns[i]) 51606164d2bSGeorge Zhang goto ppn_error; 51706164d2bSGeorge Zhang } 51806164d2bSGeorge Zhang 51906164d2bSGeorge Zhang ppn_set->num_produce_pages = num_produce_pages; 52006164d2bSGeorge Zhang ppn_set->num_consume_pages = num_consume_pages; 52106164d2bSGeorge Zhang ppn_set->produce_ppns = produce_ppns; 52206164d2bSGeorge Zhang ppn_set->consume_ppns = consume_ppns; 52306164d2bSGeorge Zhang ppn_set->initialized = true; 52406164d2bSGeorge Zhang return VMCI_SUCCESS; 52506164d2bSGeorge Zhang 52606164d2bSGeorge Zhang ppn_error: 52706164d2bSGeorge Zhang kfree(produce_ppns); 52806164d2bSGeorge Zhang kfree(consume_ppns); 52906164d2bSGeorge Zhang return VMCI_ERROR_INVALID_ARGS; 53006164d2bSGeorge Zhang } 53106164d2bSGeorge Zhang 53206164d2bSGeorge Zhang /* 53306164d2bSGeorge Zhang * Frees the two list of PPNs for a queue pair. 53406164d2bSGeorge Zhang */ 535e6389a13SDmitry Torokhov static void qp_free_ppn_set(struct ppn_set *ppn_set) 53606164d2bSGeorge Zhang { 53706164d2bSGeorge Zhang if (ppn_set->initialized) { 53806164d2bSGeorge Zhang /* Do not call these functions on NULL inputs. */ 53906164d2bSGeorge Zhang kfree(ppn_set->produce_ppns); 54006164d2bSGeorge Zhang kfree(ppn_set->consume_ppns); 54106164d2bSGeorge Zhang } 54206164d2bSGeorge Zhang memset(ppn_set, 0, sizeof(*ppn_set)); 54306164d2bSGeorge Zhang } 54406164d2bSGeorge Zhang 54506164d2bSGeorge Zhang /* 54606164d2bSGeorge Zhang * Populates the list of PPNs in the hypercall structure with the PPNS 54706164d2bSGeorge Zhang * of the produce queue and the consume queue. 54806164d2bSGeorge Zhang */ 549e6389a13SDmitry Torokhov static int qp_populate_ppn_set(u8 *call_buf, const struct ppn_set *ppn_set) 55006164d2bSGeorge Zhang { 55106164d2bSGeorge Zhang memcpy(call_buf, ppn_set->produce_ppns, 55206164d2bSGeorge Zhang ppn_set->num_produce_pages * sizeof(*ppn_set->produce_ppns)); 55306164d2bSGeorge Zhang memcpy(call_buf + 55406164d2bSGeorge Zhang ppn_set->num_produce_pages * sizeof(*ppn_set->produce_ppns), 55506164d2bSGeorge Zhang ppn_set->consume_ppns, 55606164d2bSGeorge Zhang ppn_set->num_consume_pages * sizeof(*ppn_set->consume_ppns)); 55706164d2bSGeorge Zhang 55806164d2bSGeorge Zhang return VMCI_SUCCESS; 55906164d2bSGeorge Zhang } 56006164d2bSGeorge Zhang 56106164d2bSGeorge Zhang static int qp_memcpy_to_queue(struct vmci_queue *queue, 56206164d2bSGeorge Zhang u64 queue_offset, 56306164d2bSGeorge Zhang const void *src, size_t src_offset, size_t size) 56406164d2bSGeorge Zhang { 56506164d2bSGeorge Zhang return __qp_memcpy_to_queue(queue, queue_offset, 56606164d2bSGeorge Zhang (u8 *)src + src_offset, size, false); 56706164d2bSGeorge Zhang } 56806164d2bSGeorge Zhang 56906164d2bSGeorge Zhang static int qp_memcpy_from_queue(void *dest, 57006164d2bSGeorge Zhang size_t dest_offset, 57106164d2bSGeorge Zhang const struct vmci_queue *queue, 57206164d2bSGeorge Zhang u64 queue_offset, size_t size) 57306164d2bSGeorge Zhang { 57406164d2bSGeorge Zhang return __qp_memcpy_from_queue((u8 *)dest + dest_offset, 57506164d2bSGeorge Zhang queue, queue_offset, size, false); 57606164d2bSGeorge Zhang } 57706164d2bSGeorge Zhang 57806164d2bSGeorge Zhang /* 57906164d2bSGeorge Zhang * Copies from a given iovec from a VMCI Queue. 58006164d2bSGeorge Zhang */ 58106164d2bSGeorge Zhang static int qp_memcpy_to_queue_iov(struct vmci_queue *queue, 58206164d2bSGeorge Zhang u64 queue_offset, 58306164d2bSGeorge Zhang const void *src, 58406164d2bSGeorge Zhang size_t src_offset, size_t size) 58506164d2bSGeorge Zhang { 58606164d2bSGeorge Zhang 58706164d2bSGeorge Zhang /* 58806164d2bSGeorge Zhang * We ignore src_offset because src is really a struct iovec * and will 58906164d2bSGeorge Zhang * maintain offset internally. 59006164d2bSGeorge Zhang */ 59106164d2bSGeorge Zhang return __qp_memcpy_to_queue(queue, queue_offset, src, size, true); 59206164d2bSGeorge Zhang } 59306164d2bSGeorge Zhang 59406164d2bSGeorge Zhang /* 59506164d2bSGeorge Zhang * Copies to a given iovec from a VMCI Queue. 59606164d2bSGeorge Zhang */ 59706164d2bSGeorge Zhang static int qp_memcpy_from_queue_iov(void *dest, 59806164d2bSGeorge Zhang size_t dest_offset, 59906164d2bSGeorge Zhang const struct vmci_queue *queue, 60006164d2bSGeorge Zhang u64 queue_offset, size_t size) 60106164d2bSGeorge Zhang { 60206164d2bSGeorge Zhang /* 60306164d2bSGeorge Zhang * We ignore dest_offset because dest is really a struct iovec * and 60406164d2bSGeorge Zhang * will maintain offset internally. 60506164d2bSGeorge Zhang */ 60606164d2bSGeorge Zhang return __qp_memcpy_from_queue(dest, queue, queue_offset, size, true); 60706164d2bSGeorge Zhang } 60806164d2bSGeorge Zhang 60906164d2bSGeorge Zhang /* 61006164d2bSGeorge Zhang * Allocates kernel VA space of specified size plus space for the queue 61106164d2bSGeorge Zhang * and kernel interface. This is different from the guest queue allocator, 61206164d2bSGeorge Zhang * because we do not allocate our own queue header/data pages here but 61306164d2bSGeorge Zhang * share those of the guest. 61406164d2bSGeorge Zhang */ 61506164d2bSGeorge Zhang static struct vmci_queue *qp_host_alloc_queue(u64 size) 61606164d2bSGeorge Zhang { 61706164d2bSGeorge Zhang struct vmci_queue *queue; 61842281d20SAndy King const size_t num_pages = DIV_ROUND_UP(size, PAGE_SIZE) + 1; 61906164d2bSGeorge Zhang const size_t queue_size = sizeof(*queue) + sizeof(*(queue->kernel_if)); 62006164d2bSGeorge Zhang const size_t queue_page_size = 62106164d2bSGeorge Zhang num_pages * sizeof(*queue->kernel_if->page); 62206164d2bSGeorge Zhang 62306164d2bSGeorge Zhang queue = kzalloc(queue_size + queue_page_size, GFP_KERNEL); 62406164d2bSGeorge Zhang if (queue) { 62506164d2bSGeorge Zhang queue->q_header = NULL; 62606164d2bSGeorge Zhang queue->saved_header = NULL; 62706164d2bSGeorge Zhang queue->kernel_if = 62806164d2bSGeorge Zhang (struct vmci_queue_kern_if *)((u8 *)queue + 62906164d2bSGeorge Zhang sizeof(*queue)); 63006164d2bSGeorge Zhang queue->kernel_if->host = true; 63106164d2bSGeorge Zhang queue->kernel_if->mutex = NULL; 63206164d2bSGeorge Zhang queue->kernel_if->num_pages = num_pages; 63306164d2bSGeorge Zhang queue->kernel_if->header_page = 63406164d2bSGeorge Zhang (struct page **)((u8 *)queue + queue_size); 63506164d2bSGeorge Zhang queue->kernel_if->page = &queue->kernel_if->header_page[1]; 63606164d2bSGeorge Zhang queue->kernel_if->va = NULL; 63706164d2bSGeorge Zhang queue->kernel_if->mapped = false; 63806164d2bSGeorge Zhang } 63906164d2bSGeorge Zhang 64006164d2bSGeorge Zhang return queue; 64106164d2bSGeorge Zhang } 64206164d2bSGeorge Zhang 64306164d2bSGeorge Zhang /* 64406164d2bSGeorge Zhang * Frees kernel memory for a given queue (header plus translation 64506164d2bSGeorge Zhang * structure). 64606164d2bSGeorge Zhang */ 64706164d2bSGeorge Zhang static void qp_host_free_queue(struct vmci_queue *queue, u64 queue_size) 64806164d2bSGeorge Zhang { 64906164d2bSGeorge Zhang kfree(queue); 65006164d2bSGeorge Zhang } 65106164d2bSGeorge Zhang 65206164d2bSGeorge Zhang /* 65306164d2bSGeorge Zhang * Initialize the mutex for the pair of queues. This mutex is used to 65406164d2bSGeorge Zhang * protect the q_header and the buffer from changing out from under any 65506164d2bSGeorge Zhang * users of either queue. Of course, it's only any good if the mutexes 65606164d2bSGeorge Zhang * are actually acquired. Queue structure must lie on non-paged memory 65706164d2bSGeorge Zhang * or we cannot guarantee access to the mutex. 65806164d2bSGeorge Zhang */ 65906164d2bSGeorge Zhang static void qp_init_queue_mutex(struct vmci_queue *produce_q, 66006164d2bSGeorge Zhang struct vmci_queue *consume_q) 66106164d2bSGeorge Zhang { 66206164d2bSGeorge Zhang /* 66306164d2bSGeorge Zhang * Only the host queue has shared state - the guest queues do not 66406164d2bSGeorge Zhang * need to synchronize access using a queue mutex. 66506164d2bSGeorge Zhang */ 66606164d2bSGeorge Zhang 66706164d2bSGeorge Zhang if (produce_q->kernel_if->host) { 66806164d2bSGeorge Zhang produce_q->kernel_if->mutex = &produce_q->kernel_if->__mutex; 66906164d2bSGeorge Zhang consume_q->kernel_if->mutex = &produce_q->kernel_if->__mutex; 67006164d2bSGeorge Zhang mutex_init(produce_q->kernel_if->mutex); 67106164d2bSGeorge Zhang } 67206164d2bSGeorge Zhang } 67306164d2bSGeorge Zhang 67406164d2bSGeorge Zhang /* 67506164d2bSGeorge Zhang * Cleans up the mutex for the pair of queues. 67606164d2bSGeorge Zhang */ 67706164d2bSGeorge Zhang static void qp_cleanup_queue_mutex(struct vmci_queue *produce_q, 67806164d2bSGeorge Zhang struct vmci_queue *consume_q) 67906164d2bSGeorge Zhang { 68006164d2bSGeorge Zhang if (produce_q->kernel_if->host) { 68106164d2bSGeorge Zhang produce_q->kernel_if->mutex = NULL; 68206164d2bSGeorge Zhang consume_q->kernel_if->mutex = NULL; 68306164d2bSGeorge Zhang } 68406164d2bSGeorge Zhang } 68506164d2bSGeorge Zhang 68606164d2bSGeorge Zhang /* 68706164d2bSGeorge Zhang * Acquire the mutex for the queue. Note that the produce_q and 68806164d2bSGeorge Zhang * the consume_q share a mutex. So, only one of the two need to 68906164d2bSGeorge Zhang * be passed in to this routine. Either will work just fine. 69006164d2bSGeorge Zhang */ 69106164d2bSGeorge Zhang static void qp_acquire_queue_mutex(struct vmci_queue *queue) 69206164d2bSGeorge Zhang { 69306164d2bSGeorge Zhang if (queue->kernel_if->host) 69406164d2bSGeorge Zhang mutex_lock(queue->kernel_if->mutex); 69506164d2bSGeorge Zhang } 69606164d2bSGeorge Zhang 69706164d2bSGeorge Zhang /* 69806164d2bSGeorge Zhang * Release the mutex for the queue. Note that the produce_q and 69906164d2bSGeorge Zhang * the consume_q share a mutex. So, only one of the two need to 70006164d2bSGeorge Zhang * be passed in to this routine. Either will work just fine. 70106164d2bSGeorge Zhang */ 70206164d2bSGeorge Zhang static void qp_release_queue_mutex(struct vmci_queue *queue) 70306164d2bSGeorge Zhang { 70406164d2bSGeorge Zhang if (queue->kernel_if->host) 70506164d2bSGeorge Zhang mutex_unlock(queue->kernel_if->mutex); 70606164d2bSGeorge Zhang } 70706164d2bSGeorge Zhang 70806164d2bSGeorge Zhang /* 70906164d2bSGeorge Zhang * Helper function to release pages in the PageStoreAttachInfo 71006164d2bSGeorge Zhang * previously obtained using get_user_pages. 71106164d2bSGeorge Zhang */ 71206164d2bSGeorge Zhang static void qp_release_pages(struct page **pages, 71306164d2bSGeorge Zhang u64 num_pages, bool dirty) 71406164d2bSGeorge Zhang { 71506164d2bSGeorge Zhang int i; 71606164d2bSGeorge Zhang 71706164d2bSGeorge Zhang for (i = 0; i < num_pages; i++) { 71806164d2bSGeorge Zhang if (dirty) 71906164d2bSGeorge Zhang set_page_dirty(pages[i]); 72006164d2bSGeorge Zhang 72106164d2bSGeorge Zhang page_cache_release(pages[i]); 72206164d2bSGeorge Zhang pages[i] = NULL; 72306164d2bSGeorge Zhang } 72406164d2bSGeorge Zhang } 72506164d2bSGeorge Zhang 72606164d2bSGeorge Zhang /* 72706164d2bSGeorge Zhang * Lock the user pages referenced by the {produce,consume}Buffer 72806164d2bSGeorge Zhang * struct into memory and populate the {produce,consume}Pages 72906164d2bSGeorge Zhang * arrays in the attach structure with them. 73006164d2bSGeorge Zhang */ 73106164d2bSGeorge Zhang static int qp_host_get_user_memory(u64 produce_uva, 73206164d2bSGeorge Zhang u64 consume_uva, 73306164d2bSGeorge Zhang struct vmci_queue *produce_q, 73406164d2bSGeorge Zhang struct vmci_queue *consume_q) 73506164d2bSGeorge Zhang { 73606164d2bSGeorge Zhang int retval; 73706164d2bSGeorge Zhang int err = VMCI_SUCCESS; 73806164d2bSGeorge Zhang 73906164d2bSGeorge Zhang down_write(¤t->mm->mmap_sem); 74006164d2bSGeorge Zhang retval = get_user_pages(current, 74106164d2bSGeorge Zhang current->mm, 74206164d2bSGeorge Zhang (uintptr_t) produce_uva, 74306164d2bSGeorge Zhang produce_q->kernel_if->num_pages, 74406164d2bSGeorge Zhang 1, 0, produce_q->kernel_if->header_page, NULL); 74506164d2bSGeorge Zhang if (retval < produce_q->kernel_if->num_pages) { 74606164d2bSGeorge Zhang pr_warn("get_user_pages(produce) failed (retval=%d)", retval); 74706164d2bSGeorge Zhang qp_release_pages(produce_q->kernel_if->header_page, retval, 74806164d2bSGeorge Zhang false); 74906164d2bSGeorge Zhang err = VMCI_ERROR_NO_MEM; 75006164d2bSGeorge Zhang goto out; 75106164d2bSGeorge Zhang } 75206164d2bSGeorge Zhang 75306164d2bSGeorge Zhang retval = get_user_pages(current, 75406164d2bSGeorge Zhang current->mm, 75506164d2bSGeorge Zhang (uintptr_t) consume_uva, 75606164d2bSGeorge Zhang consume_q->kernel_if->num_pages, 75706164d2bSGeorge Zhang 1, 0, consume_q->kernel_if->header_page, NULL); 75806164d2bSGeorge Zhang if (retval < consume_q->kernel_if->num_pages) { 75906164d2bSGeorge Zhang pr_warn("get_user_pages(consume) failed (retval=%d)", retval); 76006164d2bSGeorge Zhang qp_release_pages(consume_q->kernel_if->header_page, retval, 76106164d2bSGeorge Zhang false); 76206164d2bSGeorge Zhang qp_release_pages(produce_q->kernel_if->header_page, 76306164d2bSGeorge Zhang produce_q->kernel_if->num_pages, false); 76406164d2bSGeorge Zhang err = VMCI_ERROR_NO_MEM; 76506164d2bSGeorge Zhang } 76606164d2bSGeorge Zhang 76706164d2bSGeorge Zhang out: 76806164d2bSGeorge Zhang up_write(¤t->mm->mmap_sem); 76906164d2bSGeorge Zhang 77006164d2bSGeorge Zhang return err; 77106164d2bSGeorge Zhang } 77206164d2bSGeorge Zhang 77306164d2bSGeorge Zhang /* 77406164d2bSGeorge Zhang * Registers the specification of the user pages used for backing a queue 77506164d2bSGeorge Zhang * pair. Enough information to map in pages is stored in the OS specific 77606164d2bSGeorge Zhang * part of the struct vmci_queue structure. 77706164d2bSGeorge Zhang */ 77806164d2bSGeorge Zhang static int qp_host_register_user_memory(struct vmci_qp_page_store *page_store, 77906164d2bSGeorge Zhang struct vmci_queue *produce_q, 78006164d2bSGeorge Zhang struct vmci_queue *consume_q) 78106164d2bSGeorge Zhang { 78206164d2bSGeorge Zhang u64 produce_uva; 78306164d2bSGeorge Zhang u64 consume_uva; 78406164d2bSGeorge Zhang 78506164d2bSGeorge Zhang /* 78606164d2bSGeorge Zhang * The new style and the old style mapping only differs in 78706164d2bSGeorge Zhang * that we either get a single or two UVAs, so we split the 78806164d2bSGeorge Zhang * single UVA range at the appropriate spot. 78906164d2bSGeorge Zhang */ 79006164d2bSGeorge Zhang produce_uva = page_store->pages; 79106164d2bSGeorge Zhang consume_uva = page_store->pages + 79206164d2bSGeorge Zhang produce_q->kernel_if->num_pages * PAGE_SIZE; 79306164d2bSGeorge Zhang return qp_host_get_user_memory(produce_uva, consume_uva, produce_q, 79406164d2bSGeorge Zhang consume_q); 79506164d2bSGeorge Zhang } 79606164d2bSGeorge Zhang 79706164d2bSGeorge Zhang /* 79806164d2bSGeorge Zhang * Releases and removes the references to user pages stored in the attach 79906164d2bSGeorge Zhang * struct. Pages are released from the page cache and may become 80006164d2bSGeorge Zhang * swappable again. 80106164d2bSGeorge Zhang */ 80206164d2bSGeorge Zhang static void qp_host_unregister_user_memory(struct vmci_queue *produce_q, 80306164d2bSGeorge Zhang struct vmci_queue *consume_q) 80406164d2bSGeorge Zhang { 80506164d2bSGeorge Zhang qp_release_pages(produce_q->kernel_if->header_page, 80606164d2bSGeorge Zhang produce_q->kernel_if->num_pages, true); 80706164d2bSGeorge Zhang memset(produce_q->kernel_if->header_page, 0, 80806164d2bSGeorge Zhang sizeof(*produce_q->kernel_if->header_page) * 80906164d2bSGeorge Zhang produce_q->kernel_if->num_pages); 81006164d2bSGeorge Zhang qp_release_pages(consume_q->kernel_if->header_page, 81106164d2bSGeorge Zhang consume_q->kernel_if->num_pages, true); 81206164d2bSGeorge Zhang memset(consume_q->kernel_if->header_page, 0, 81306164d2bSGeorge Zhang sizeof(*consume_q->kernel_if->header_page) * 81406164d2bSGeorge Zhang consume_q->kernel_if->num_pages); 81506164d2bSGeorge Zhang } 81606164d2bSGeorge Zhang 81706164d2bSGeorge Zhang /* 81806164d2bSGeorge Zhang * Once qp_host_register_user_memory has been performed on a 81906164d2bSGeorge Zhang * queue, the queue pair headers can be mapped into the 82006164d2bSGeorge Zhang * kernel. Once mapped, they must be unmapped with 82106164d2bSGeorge Zhang * qp_host_unmap_queues prior to calling 82206164d2bSGeorge Zhang * qp_host_unregister_user_memory. 82306164d2bSGeorge Zhang * Pages are pinned. 82406164d2bSGeorge Zhang */ 82506164d2bSGeorge Zhang static int qp_host_map_queues(struct vmci_queue *produce_q, 82606164d2bSGeorge Zhang struct vmci_queue *consume_q) 82706164d2bSGeorge Zhang { 82806164d2bSGeorge Zhang int result; 82906164d2bSGeorge Zhang 83006164d2bSGeorge Zhang if (!produce_q->q_header || !consume_q->q_header) { 83106164d2bSGeorge Zhang struct page *headers[2]; 83206164d2bSGeorge Zhang 83306164d2bSGeorge Zhang if (produce_q->q_header != consume_q->q_header) 83406164d2bSGeorge Zhang return VMCI_ERROR_QUEUEPAIR_MISMATCH; 83506164d2bSGeorge Zhang 83606164d2bSGeorge Zhang if (produce_q->kernel_if->header_page == NULL || 83706164d2bSGeorge Zhang *produce_q->kernel_if->header_page == NULL) 83806164d2bSGeorge Zhang return VMCI_ERROR_UNAVAILABLE; 83906164d2bSGeorge Zhang 84006164d2bSGeorge Zhang headers[0] = *produce_q->kernel_if->header_page; 84106164d2bSGeorge Zhang headers[1] = *consume_q->kernel_if->header_page; 84206164d2bSGeorge Zhang 84306164d2bSGeorge Zhang produce_q->q_header = vmap(headers, 2, VM_MAP, PAGE_KERNEL); 84406164d2bSGeorge Zhang if (produce_q->q_header != NULL) { 84506164d2bSGeorge Zhang consume_q->q_header = 84606164d2bSGeorge Zhang (struct vmci_queue_header *)((u8 *) 84706164d2bSGeorge Zhang produce_q->q_header + 84806164d2bSGeorge Zhang PAGE_SIZE); 84906164d2bSGeorge Zhang result = VMCI_SUCCESS; 85006164d2bSGeorge Zhang } else { 85106164d2bSGeorge Zhang pr_warn("vmap failed\n"); 85206164d2bSGeorge Zhang result = VMCI_ERROR_NO_MEM; 85306164d2bSGeorge Zhang } 85406164d2bSGeorge Zhang } else { 85506164d2bSGeorge Zhang result = VMCI_SUCCESS; 85606164d2bSGeorge Zhang } 85706164d2bSGeorge Zhang 85806164d2bSGeorge Zhang return result; 85906164d2bSGeorge Zhang } 86006164d2bSGeorge Zhang 86106164d2bSGeorge Zhang /* 86206164d2bSGeorge Zhang * Unmaps previously mapped queue pair headers from the kernel. 86306164d2bSGeorge Zhang * Pages are unpinned. 86406164d2bSGeorge Zhang */ 86506164d2bSGeorge Zhang static int qp_host_unmap_queues(u32 gid, 86606164d2bSGeorge Zhang struct vmci_queue *produce_q, 86706164d2bSGeorge Zhang struct vmci_queue *consume_q) 86806164d2bSGeorge Zhang { 86906164d2bSGeorge Zhang if (produce_q->q_header) { 87006164d2bSGeorge Zhang if (produce_q->q_header < consume_q->q_header) 87106164d2bSGeorge Zhang vunmap(produce_q->q_header); 87206164d2bSGeorge Zhang else 87306164d2bSGeorge Zhang vunmap(consume_q->q_header); 87406164d2bSGeorge Zhang 87506164d2bSGeorge Zhang produce_q->q_header = NULL; 87606164d2bSGeorge Zhang consume_q->q_header = NULL; 87706164d2bSGeorge Zhang } 87806164d2bSGeorge Zhang 87906164d2bSGeorge Zhang return VMCI_SUCCESS; 88006164d2bSGeorge Zhang } 88106164d2bSGeorge Zhang 88206164d2bSGeorge Zhang /* 88306164d2bSGeorge Zhang * Finds the entry in the list corresponding to a given handle. Assumes 88406164d2bSGeorge Zhang * that the list is locked. 88506164d2bSGeorge Zhang */ 88606164d2bSGeorge Zhang static struct qp_entry *qp_list_find(struct qp_list *qp_list, 88706164d2bSGeorge Zhang struct vmci_handle handle) 88806164d2bSGeorge Zhang { 88906164d2bSGeorge Zhang struct qp_entry *entry; 89006164d2bSGeorge Zhang 89106164d2bSGeorge Zhang if (vmci_handle_is_invalid(handle)) 89206164d2bSGeorge Zhang return NULL; 89306164d2bSGeorge Zhang 89406164d2bSGeorge Zhang list_for_each_entry(entry, &qp_list->head, list_item) { 89506164d2bSGeorge Zhang if (vmci_handle_is_equal(entry->handle, handle)) 89606164d2bSGeorge Zhang return entry; 89706164d2bSGeorge Zhang } 89806164d2bSGeorge Zhang 89906164d2bSGeorge Zhang return NULL; 90006164d2bSGeorge Zhang } 90106164d2bSGeorge Zhang 90206164d2bSGeorge Zhang /* 90306164d2bSGeorge Zhang * Finds the entry in the list corresponding to a given handle. 90406164d2bSGeorge Zhang */ 90506164d2bSGeorge Zhang static struct qp_guest_endpoint * 90606164d2bSGeorge Zhang qp_guest_handle_to_entry(struct vmci_handle handle) 90706164d2bSGeorge Zhang { 90806164d2bSGeorge Zhang struct qp_guest_endpoint *entry; 90906164d2bSGeorge Zhang struct qp_entry *qp = qp_list_find(&qp_guest_endpoints, handle); 91006164d2bSGeorge Zhang 91106164d2bSGeorge Zhang entry = qp ? container_of( 91206164d2bSGeorge Zhang qp, struct qp_guest_endpoint, qp) : NULL; 91306164d2bSGeorge Zhang return entry; 91406164d2bSGeorge Zhang } 91506164d2bSGeorge Zhang 91606164d2bSGeorge Zhang /* 91706164d2bSGeorge Zhang * Finds the entry in the list corresponding to a given handle. 91806164d2bSGeorge Zhang */ 91906164d2bSGeorge Zhang static struct qp_broker_entry * 92006164d2bSGeorge Zhang qp_broker_handle_to_entry(struct vmci_handle handle) 92106164d2bSGeorge Zhang { 92206164d2bSGeorge Zhang struct qp_broker_entry *entry; 92306164d2bSGeorge Zhang struct qp_entry *qp = qp_list_find(&qp_broker_list, handle); 92406164d2bSGeorge Zhang 92506164d2bSGeorge Zhang entry = qp ? container_of( 92606164d2bSGeorge Zhang qp, struct qp_broker_entry, qp) : NULL; 92706164d2bSGeorge Zhang return entry; 92806164d2bSGeorge Zhang } 92906164d2bSGeorge Zhang 93006164d2bSGeorge Zhang /* 93106164d2bSGeorge Zhang * Dispatches a queue pair event message directly into the local event 93206164d2bSGeorge Zhang * queue. 93306164d2bSGeorge Zhang */ 93406164d2bSGeorge Zhang static int qp_notify_peer_local(bool attach, struct vmci_handle handle) 93506164d2bSGeorge Zhang { 93606164d2bSGeorge Zhang u32 context_id = vmci_get_context_id(); 93706164d2bSGeorge Zhang struct vmci_event_qp ev; 93806164d2bSGeorge Zhang 93906164d2bSGeorge Zhang ev.msg.hdr.dst = vmci_make_handle(context_id, VMCI_EVENT_HANDLER); 94006164d2bSGeorge Zhang ev.msg.hdr.src = vmci_make_handle(VMCI_HYPERVISOR_CONTEXT_ID, 94106164d2bSGeorge Zhang VMCI_CONTEXT_RESOURCE_ID); 94206164d2bSGeorge Zhang ev.msg.hdr.payload_size = sizeof(ev) - sizeof(ev.msg.hdr); 94306164d2bSGeorge Zhang ev.msg.event_data.event = 94406164d2bSGeorge Zhang attach ? VMCI_EVENT_QP_PEER_ATTACH : VMCI_EVENT_QP_PEER_DETACH; 94506164d2bSGeorge Zhang ev.payload.peer_id = context_id; 94606164d2bSGeorge Zhang ev.payload.handle = handle; 94706164d2bSGeorge Zhang 94806164d2bSGeorge Zhang return vmci_event_dispatch(&ev.msg.hdr); 94906164d2bSGeorge Zhang } 95006164d2bSGeorge Zhang 95106164d2bSGeorge Zhang /* 95206164d2bSGeorge Zhang * Allocates and initializes a qp_guest_endpoint structure. 95306164d2bSGeorge Zhang * Allocates a queue_pair rid (and handle) iff the given entry has 95406164d2bSGeorge Zhang * an invalid handle. 0 through VMCI_RESERVED_RESOURCE_ID_MAX 95506164d2bSGeorge Zhang * are reserved handles. Assumes that the QP list mutex is held 95606164d2bSGeorge Zhang * by the caller. 95706164d2bSGeorge Zhang */ 95806164d2bSGeorge Zhang static struct qp_guest_endpoint * 95906164d2bSGeorge Zhang qp_guest_endpoint_create(struct vmci_handle handle, 96006164d2bSGeorge Zhang u32 peer, 96106164d2bSGeorge Zhang u32 flags, 96206164d2bSGeorge Zhang u64 produce_size, 96306164d2bSGeorge Zhang u64 consume_size, 96406164d2bSGeorge Zhang void *produce_q, 96506164d2bSGeorge Zhang void *consume_q) 96606164d2bSGeorge Zhang { 96706164d2bSGeorge Zhang int result; 96806164d2bSGeorge Zhang struct qp_guest_endpoint *entry; 96906164d2bSGeorge Zhang /* One page each for the queue headers. */ 97042281d20SAndy King const u64 num_ppns = DIV_ROUND_UP(produce_size, PAGE_SIZE) + 97142281d20SAndy King DIV_ROUND_UP(consume_size, PAGE_SIZE) + 2; 97206164d2bSGeorge Zhang 97306164d2bSGeorge Zhang if (vmci_handle_is_invalid(handle)) { 97406164d2bSGeorge Zhang u32 context_id = vmci_get_context_id(); 97506164d2bSGeorge Zhang 97606164d2bSGeorge Zhang handle = vmci_make_handle(context_id, VMCI_INVALID_ID); 97706164d2bSGeorge Zhang } 97806164d2bSGeorge Zhang 97906164d2bSGeorge Zhang entry = kzalloc(sizeof(*entry), GFP_KERNEL); 98006164d2bSGeorge Zhang if (entry) { 98106164d2bSGeorge Zhang entry->qp.peer = peer; 98206164d2bSGeorge Zhang entry->qp.flags = flags; 98306164d2bSGeorge Zhang entry->qp.produce_size = produce_size; 98406164d2bSGeorge Zhang entry->qp.consume_size = consume_size; 98506164d2bSGeorge Zhang entry->qp.ref_count = 0; 98606164d2bSGeorge Zhang entry->num_ppns = num_ppns; 98706164d2bSGeorge Zhang entry->produce_q = produce_q; 98806164d2bSGeorge Zhang entry->consume_q = consume_q; 98906164d2bSGeorge Zhang INIT_LIST_HEAD(&entry->qp.list_item); 99006164d2bSGeorge Zhang 99106164d2bSGeorge Zhang /* Add resource obj */ 99206164d2bSGeorge Zhang result = vmci_resource_add(&entry->resource, 99306164d2bSGeorge Zhang VMCI_RESOURCE_TYPE_QPAIR_GUEST, 99406164d2bSGeorge Zhang handle); 99506164d2bSGeorge Zhang entry->qp.handle = vmci_resource_handle(&entry->resource); 99606164d2bSGeorge Zhang if ((result != VMCI_SUCCESS) || 99706164d2bSGeorge Zhang qp_list_find(&qp_guest_endpoints, entry->qp.handle)) { 99806164d2bSGeorge Zhang pr_warn("Failed to add new resource (handle=0x%x:0x%x), error: %d", 99906164d2bSGeorge Zhang handle.context, handle.resource, result); 100006164d2bSGeorge Zhang kfree(entry); 100106164d2bSGeorge Zhang entry = NULL; 100206164d2bSGeorge Zhang } 100306164d2bSGeorge Zhang } 100406164d2bSGeorge Zhang return entry; 100506164d2bSGeorge Zhang } 100606164d2bSGeorge Zhang 100706164d2bSGeorge Zhang /* 100806164d2bSGeorge Zhang * Frees a qp_guest_endpoint structure. 100906164d2bSGeorge Zhang */ 101006164d2bSGeorge Zhang static void qp_guest_endpoint_destroy(struct qp_guest_endpoint *entry) 101106164d2bSGeorge Zhang { 101206164d2bSGeorge Zhang qp_free_ppn_set(&entry->ppn_set); 101306164d2bSGeorge Zhang qp_cleanup_queue_mutex(entry->produce_q, entry->consume_q); 101406164d2bSGeorge Zhang qp_free_queue(entry->produce_q, entry->qp.produce_size); 101506164d2bSGeorge Zhang qp_free_queue(entry->consume_q, entry->qp.consume_size); 101606164d2bSGeorge Zhang /* Unlink from resource hash table and free callback */ 101706164d2bSGeorge Zhang vmci_resource_remove(&entry->resource); 101806164d2bSGeorge Zhang 101906164d2bSGeorge Zhang kfree(entry); 102006164d2bSGeorge Zhang } 102106164d2bSGeorge Zhang 102206164d2bSGeorge Zhang /* 102306164d2bSGeorge Zhang * Helper to make a queue_pairAlloc hypercall when the driver is 102406164d2bSGeorge Zhang * supporting a guest device. 102506164d2bSGeorge Zhang */ 102606164d2bSGeorge Zhang static int qp_alloc_hypercall(const struct qp_guest_endpoint *entry) 102706164d2bSGeorge Zhang { 102806164d2bSGeorge Zhang struct vmci_qp_alloc_msg *alloc_msg; 102906164d2bSGeorge Zhang size_t msg_size; 103006164d2bSGeorge Zhang int result; 103106164d2bSGeorge Zhang 103206164d2bSGeorge Zhang if (!entry || entry->num_ppns <= 2) 103306164d2bSGeorge Zhang return VMCI_ERROR_INVALID_ARGS; 103406164d2bSGeorge Zhang 103506164d2bSGeorge Zhang msg_size = sizeof(*alloc_msg) + 103606164d2bSGeorge Zhang (size_t) entry->num_ppns * sizeof(u32); 103706164d2bSGeorge Zhang alloc_msg = kmalloc(msg_size, GFP_KERNEL); 103806164d2bSGeorge Zhang if (!alloc_msg) 103906164d2bSGeorge Zhang return VMCI_ERROR_NO_MEM; 104006164d2bSGeorge Zhang 104106164d2bSGeorge Zhang alloc_msg->hdr.dst = vmci_make_handle(VMCI_HYPERVISOR_CONTEXT_ID, 104206164d2bSGeorge Zhang VMCI_QUEUEPAIR_ALLOC); 104306164d2bSGeorge Zhang alloc_msg->hdr.src = VMCI_ANON_SRC_HANDLE; 104406164d2bSGeorge Zhang alloc_msg->hdr.payload_size = msg_size - VMCI_DG_HEADERSIZE; 104506164d2bSGeorge Zhang alloc_msg->handle = entry->qp.handle; 104606164d2bSGeorge Zhang alloc_msg->peer = entry->qp.peer; 104706164d2bSGeorge Zhang alloc_msg->flags = entry->qp.flags; 104806164d2bSGeorge Zhang alloc_msg->produce_size = entry->qp.produce_size; 104906164d2bSGeorge Zhang alloc_msg->consume_size = entry->qp.consume_size; 105006164d2bSGeorge Zhang alloc_msg->num_ppns = entry->num_ppns; 105106164d2bSGeorge Zhang 105206164d2bSGeorge Zhang result = qp_populate_ppn_set((u8 *)alloc_msg + sizeof(*alloc_msg), 105306164d2bSGeorge Zhang &entry->ppn_set); 105406164d2bSGeorge Zhang if (result == VMCI_SUCCESS) 105506164d2bSGeorge Zhang result = vmci_send_datagram(&alloc_msg->hdr); 105606164d2bSGeorge Zhang 105706164d2bSGeorge Zhang kfree(alloc_msg); 105806164d2bSGeorge Zhang 105906164d2bSGeorge Zhang return result; 106006164d2bSGeorge Zhang } 106106164d2bSGeorge Zhang 106206164d2bSGeorge Zhang /* 106306164d2bSGeorge Zhang * Helper to make a queue_pairDetach hypercall when the driver is 106406164d2bSGeorge Zhang * supporting a guest device. 106506164d2bSGeorge Zhang */ 106606164d2bSGeorge Zhang static int qp_detatch_hypercall(struct vmci_handle handle) 106706164d2bSGeorge Zhang { 106806164d2bSGeorge Zhang struct vmci_qp_detach_msg detach_msg; 106906164d2bSGeorge Zhang 107006164d2bSGeorge Zhang detach_msg.hdr.dst = vmci_make_handle(VMCI_HYPERVISOR_CONTEXT_ID, 107106164d2bSGeorge Zhang VMCI_QUEUEPAIR_DETACH); 107206164d2bSGeorge Zhang detach_msg.hdr.src = VMCI_ANON_SRC_HANDLE; 107306164d2bSGeorge Zhang detach_msg.hdr.payload_size = sizeof(handle); 107406164d2bSGeorge Zhang detach_msg.handle = handle; 107506164d2bSGeorge Zhang 107606164d2bSGeorge Zhang return vmci_send_datagram(&detach_msg.hdr); 107706164d2bSGeorge Zhang } 107806164d2bSGeorge Zhang 107906164d2bSGeorge Zhang /* 108006164d2bSGeorge Zhang * Adds the given entry to the list. Assumes that the list is locked. 108106164d2bSGeorge Zhang */ 108206164d2bSGeorge Zhang static void qp_list_add_entry(struct qp_list *qp_list, struct qp_entry *entry) 108306164d2bSGeorge Zhang { 108406164d2bSGeorge Zhang if (entry) 108506164d2bSGeorge Zhang list_add(&entry->list_item, &qp_list->head); 108606164d2bSGeorge Zhang } 108706164d2bSGeorge Zhang 108806164d2bSGeorge Zhang /* 108906164d2bSGeorge Zhang * Removes the given entry from the list. Assumes that the list is locked. 109006164d2bSGeorge Zhang */ 109106164d2bSGeorge Zhang static void qp_list_remove_entry(struct qp_list *qp_list, 109206164d2bSGeorge Zhang struct qp_entry *entry) 109306164d2bSGeorge Zhang { 109406164d2bSGeorge Zhang if (entry) 109506164d2bSGeorge Zhang list_del(&entry->list_item); 109606164d2bSGeorge Zhang } 109706164d2bSGeorge Zhang 109806164d2bSGeorge Zhang /* 109906164d2bSGeorge Zhang * Helper for VMCI queue_pair detach interface. Frees the physical 110006164d2bSGeorge Zhang * pages for the queue pair. 110106164d2bSGeorge Zhang */ 110206164d2bSGeorge Zhang static int qp_detatch_guest_work(struct vmci_handle handle) 110306164d2bSGeorge Zhang { 110406164d2bSGeorge Zhang int result; 110506164d2bSGeorge Zhang struct qp_guest_endpoint *entry; 110606164d2bSGeorge Zhang u32 ref_count = ~0; /* To avoid compiler warning below */ 110706164d2bSGeorge Zhang 110806164d2bSGeorge Zhang mutex_lock(&qp_guest_endpoints.mutex); 110906164d2bSGeorge Zhang 111006164d2bSGeorge Zhang entry = qp_guest_handle_to_entry(handle); 111106164d2bSGeorge Zhang if (!entry) { 111206164d2bSGeorge Zhang mutex_unlock(&qp_guest_endpoints.mutex); 111306164d2bSGeorge Zhang return VMCI_ERROR_NOT_FOUND; 111406164d2bSGeorge Zhang } 111506164d2bSGeorge Zhang 111606164d2bSGeorge Zhang if (entry->qp.flags & VMCI_QPFLAG_LOCAL) { 111706164d2bSGeorge Zhang result = VMCI_SUCCESS; 111806164d2bSGeorge Zhang 111906164d2bSGeorge Zhang if (entry->qp.ref_count > 1) { 112006164d2bSGeorge Zhang result = qp_notify_peer_local(false, handle); 112106164d2bSGeorge Zhang /* 112206164d2bSGeorge Zhang * We can fail to notify a local queuepair 112306164d2bSGeorge Zhang * because we can't allocate. We still want 112406164d2bSGeorge Zhang * to release the entry if that happens, so 112506164d2bSGeorge Zhang * don't bail out yet. 112606164d2bSGeorge Zhang */ 112706164d2bSGeorge Zhang } 112806164d2bSGeorge Zhang } else { 112906164d2bSGeorge Zhang result = qp_detatch_hypercall(handle); 113006164d2bSGeorge Zhang if (result < VMCI_SUCCESS) { 113106164d2bSGeorge Zhang /* 113206164d2bSGeorge Zhang * We failed to notify a non-local queuepair. 113306164d2bSGeorge Zhang * That other queuepair might still be 113406164d2bSGeorge Zhang * accessing the shared memory, so don't 113506164d2bSGeorge Zhang * release the entry yet. It will get cleaned 113606164d2bSGeorge Zhang * up by VMCIqueue_pair_Exit() if necessary 113706164d2bSGeorge Zhang * (assuming we are going away, otherwise why 113806164d2bSGeorge Zhang * did this fail?). 113906164d2bSGeorge Zhang */ 114006164d2bSGeorge Zhang 114106164d2bSGeorge Zhang mutex_unlock(&qp_guest_endpoints.mutex); 114206164d2bSGeorge Zhang return result; 114306164d2bSGeorge Zhang } 114406164d2bSGeorge Zhang } 114506164d2bSGeorge Zhang 114606164d2bSGeorge Zhang /* 114706164d2bSGeorge Zhang * If we get here then we either failed to notify a local queuepair, or 114806164d2bSGeorge Zhang * we succeeded in all cases. Release the entry if required. 114906164d2bSGeorge Zhang */ 115006164d2bSGeorge Zhang 115106164d2bSGeorge Zhang entry->qp.ref_count--; 115206164d2bSGeorge Zhang if (entry->qp.ref_count == 0) 115306164d2bSGeorge Zhang qp_list_remove_entry(&qp_guest_endpoints, &entry->qp); 115406164d2bSGeorge Zhang 115506164d2bSGeorge Zhang /* If we didn't remove the entry, this could change once we unlock. */ 115606164d2bSGeorge Zhang if (entry) 115706164d2bSGeorge Zhang ref_count = entry->qp.ref_count; 115806164d2bSGeorge Zhang 115906164d2bSGeorge Zhang mutex_unlock(&qp_guest_endpoints.mutex); 116006164d2bSGeorge Zhang 116106164d2bSGeorge Zhang if (ref_count == 0) 116206164d2bSGeorge Zhang qp_guest_endpoint_destroy(entry); 116306164d2bSGeorge Zhang 116406164d2bSGeorge Zhang return result; 116506164d2bSGeorge Zhang } 116606164d2bSGeorge Zhang 116706164d2bSGeorge Zhang /* 116806164d2bSGeorge Zhang * This functions handles the actual allocation of a VMCI queue 116906164d2bSGeorge Zhang * pair guest endpoint. Allocates physical pages for the queue 117006164d2bSGeorge Zhang * pair. It makes OS dependent calls through generic wrappers. 117106164d2bSGeorge Zhang */ 117206164d2bSGeorge Zhang static int qp_alloc_guest_work(struct vmci_handle *handle, 117306164d2bSGeorge Zhang struct vmci_queue **produce_q, 117406164d2bSGeorge Zhang u64 produce_size, 117506164d2bSGeorge Zhang struct vmci_queue **consume_q, 117606164d2bSGeorge Zhang u64 consume_size, 117706164d2bSGeorge Zhang u32 peer, 117806164d2bSGeorge Zhang u32 flags, 117906164d2bSGeorge Zhang u32 priv_flags) 118006164d2bSGeorge Zhang { 118106164d2bSGeorge Zhang const u64 num_produce_pages = 118242281d20SAndy King DIV_ROUND_UP(produce_size, PAGE_SIZE) + 1; 118306164d2bSGeorge Zhang const u64 num_consume_pages = 118442281d20SAndy King DIV_ROUND_UP(consume_size, PAGE_SIZE) + 1; 118506164d2bSGeorge Zhang void *my_produce_q = NULL; 118606164d2bSGeorge Zhang void *my_consume_q = NULL; 118706164d2bSGeorge Zhang int result; 118806164d2bSGeorge Zhang struct qp_guest_endpoint *queue_pair_entry = NULL; 118906164d2bSGeorge Zhang 119006164d2bSGeorge Zhang if (priv_flags != VMCI_NO_PRIVILEGE_FLAGS) 119106164d2bSGeorge Zhang return VMCI_ERROR_NO_ACCESS; 119206164d2bSGeorge Zhang 119306164d2bSGeorge Zhang mutex_lock(&qp_guest_endpoints.mutex); 119406164d2bSGeorge Zhang 119506164d2bSGeorge Zhang queue_pair_entry = qp_guest_handle_to_entry(*handle); 119606164d2bSGeorge Zhang if (queue_pair_entry) { 119706164d2bSGeorge Zhang if (queue_pair_entry->qp.flags & VMCI_QPFLAG_LOCAL) { 119806164d2bSGeorge Zhang /* Local attach case. */ 119906164d2bSGeorge Zhang if (queue_pair_entry->qp.ref_count > 1) { 120006164d2bSGeorge Zhang pr_devel("Error attempting to attach more than once\n"); 120106164d2bSGeorge Zhang result = VMCI_ERROR_UNAVAILABLE; 120206164d2bSGeorge Zhang goto error_keep_entry; 120306164d2bSGeorge Zhang } 120406164d2bSGeorge Zhang 120506164d2bSGeorge Zhang if (queue_pair_entry->qp.produce_size != consume_size || 120606164d2bSGeorge Zhang queue_pair_entry->qp.consume_size != 120706164d2bSGeorge Zhang produce_size || 120806164d2bSGeorge Zhang queue_pair_entry->qp.flags != 120906164d2bSGeorge Zhang (flags & ~VMCI_QPFLAG_ATTACH_ONLY)) { 121006164d2bSGeorge Zhang pr_devel("Error mismatched queue pair in local attach\n"); 121106164d2bSGeorge Zhang result = VMCI_ERROR_QUEUEPAIR_MISMATCH; 121206164d2bSGeorge Zhang goto error_keep_entry; 121306164d2bSGeorge Zhang } 121406164d2bSGeorge Zhang 121506164d2bSGeorge Zhang /* 121606164d2bSGeorge Zhang * Do a local attach. We swap the consume and 121706164d2bSGeorge Zhang * produce queues for the attacher and deliver 121806164d2bSGeorge Zhang * an attach event. 121906164d2bSGeorge Zhang */ 122006164d2bSGeorge Zhang result = qp_notify_peer_local(true, *handle); 122106164d2bSGeorge Zhang if (result < VMCI_SUCCESS) 122206164d2bSGeorge Zhang goto error_keep_entry; 122306164d2bSGeorge Zhang 122406164d2bSGeorge Zhang my_produce_q = queue_pair_entry->consume_q; 122506164d2bSGeorge Zhang my_consume_q = queue_pair_entry->produce_q; 122606164d2bSGeorge Zhang goto out; 122706164d2bSGeorge Zhang } 122806164d2bSGeorge Zhang 122906164d2bSGeorge Zhang result = VMCI_ERROR_ALREADY_EXISTS; 123006164d2bSGeorge Zhang goto error_keep_entry; 123106164d2bSGeorge Zhang } 123206164d2bSGeorge Zhang 123306164d2bSGeorge Zhang my_produce_q = qp_alloc_queue(produce_size, flags); 123406164d2bSGeorge Zhang if (!my_produce_q) { 123506164d2bSGeorge Zhang pr_warn("Error allocating pages for produce queue\n"); 123606164d2bSGeorge Zhang result = VMCI_ERROR_NO_MEM; 123706164d2bSGeorge Zhang goto error; 123806164d2bSGeorge Zhang } 123906164d2bSGeorge Zhang 124006164d2bSGeorge Zhang my_consume_q = qp_alloc_queue(consume_size, flags); 124106164d2bSGeorge Zhang if (!my_consume_q) { 124206164d2bSGeorge Zhang pr_warn("Error allocating pages for consume queue\n"); 124306164d2bSGeorge Zhang result = VMCI_ERROR_NO_MEM; 124406164d2bSGeorge Zhang goto error; 124506164d2bSGeorge Zhang } 124606164d2bSGeorge Zhang 124706164d2bSGeorge Zhang queue_pair_entry = qp_guest_endpoint_create(*handle, peer, flags, 124806164d2bSGeorge Zhang produce_size, consume_size, 124906164d2bSGeorge Zhang my_produce_q, my_consume_q); 125006164d2bSGeorge Zhang if (!queue_pair_entry) { 125106164d2bSGeorge Zhang pr_warn("Error allocating memory in %s\n", __func__); 125206164d2bSGeorge Zhang result = VMCI_ERROR_NO_MEM; 125306164d2bSGeorge Zhang goto error; 125406164d2bSGeorge Zhang } 125506164d2bSGeorge Zhang 125606164d2bSGeorge Zhang result = qp_alloc_ppn_set(my_produce_q, num_produce_pages, my_consume_q, 125706164d2bSGeorge Zhang num_consume_pages, 125806164d2bSGeorge Zhang &queue_pair_entry->ppn_set); 125906164d2bSGeorge Zhang if (result < VMCI_SUCCESS) { 126006164d2bSGeorge Zhang pr_warn("qp_alloc_ppn_set failed\n"); 126106164d2bSGeorge Zhang goto error; 126206164d2bSGeorge Zhang } 126306164d2bSGeorge Zhang 126406164d2bSGeorge Zhang /* 126506164d2bSGeorge Zhang * It's only necessary to notify the host if this queue pair will be 126606164d2bSGeorge Zhang * attached to from another context. 126706164d2bSGeorge Zhang */ 126806164d2bSGeorge Zhang if (queue_pair_entry->qp.flags & VMCI_QPFLAG_LOCAL) { 126906164d2bSGeorge Zhang /* Local create case. */ 127006164d2bSGeorge Zhang u32 context_id = vmci_get_context_id(); 127106164d2bSGeorge Zhang 127206164d2bSGeorge Zhang /* 127306164d2bSGeorge Zhang * Enforce similar checks on local queue pairs as we 127406164d2bSGeorge Zhang * do for regular ones. The handle's context must 127506164d2bSGeorge Zhang * match the creator or attacher context id (here they 127606164d2bSGeorge Zhang * are both the current context id) and the 127706164d2bSGeorge Zhang * attach-only flag cannot exist during create. We 127806164d2bSGeorge Zhang * also ensure specified peer is this context or an 127906164d2bSGeorge Zhang * invalid one. 128006164d2bSGeorge Zhang */ 128106164d2bSGeorge Zhang if (queue_pair_entry->qp.handle.context != context_id || 128206164d2bSGeorge Zhang (queue_pair_entry->qp.peer != VMCI_INVALID_ID && 128306164d2bSGeorge Zhang queue_pair_entry->qp.peer != context_id)) { 128406164d2bSGeorge Zhang result = VMCI_ERROR_NO_ACCESS; 128506164d2bSGeorge Zhang goto error; 128606164d2bSGeorge Zhang } 128706164d2bSGeorge Zhang 128806164d2bSGeorge Zhang if (queue_pair_entry->qp.flags & VMCI_QPFLAG_ATTACH_ONLY) { 128906164d2bSGeorge Zhang result = VMCI_ERROR_NOT_FOUND; 129006164d2bSGeorge Zhang goto error; 129106164d2bSGeorge Zhang } 129206164d2bSGeorge Zhang } else { 129306164d2bSGeorge Zhang result = qp_alloc_hypercall(queue_pair_entry); 129406164d2bSGeorge Zhang if (result < VMCI_SUCCESS) { 129506164d2bSGeorge Zhang pr_warn("qp_alloc_hypercall result = %d\n", result); 129606164d2bSGeorge Zhang goto error; 129706164d2bSGeorge Zhang } 129806164d2bSGeorge Zhang } 129906164d2bSGeorge Zhang 130006164d2bSGeorge Zhang qp_init_queue_mutex((struct vmci_queue *)my_produce_q, 130106164d2bSGeorge Zhang (struct vmci_queue *)my_consume_q); 130206164d2bSGeorge Zhang 130306164d2bSGeorge Zhang qp_list_add_entry(&qp_guest_endpoints, &queue_pair_entry->qp); 130406164d2bSGeorge Zhang 130506164d2bSGeorge Zhang out: 130606164d2bSGeorge Zhang queue_pair_entry->qp.ref_count++; 130706164d2bSGeorge Zhang *handle = queue_pair_entry->qp.handle; 130806164d2bSGeorge Zhang *produce_q = (struct vmci_queue *)my_produce_q; 130906164d2bSGeorge Zhang *consume_q = (struct vmci_queue *)my_consume_q; 131006164d2bSGeorge Zhang 131106164d2bSGeorge Zhang /* 131206164d2bSGeorge Zhang * We should initialize the queue pair header pages on a local 131306164d2bSGeorge Zhang * queue pair create. For non-local queue pairs, the 131406164d2bSGeorge Zhang * hypervisor initializes the header pages in the create step. 131506164d2bSGeorge Zhang */ 131606164d2bSGeorge Zhang if ((queue_pair_entry->qp.flags & VMCI_QPFLAG_LOCAL) && 131706164d2bSGeorge Zhang queue_pair_entry->qp.ref_count == 1) { 131806164d2bSGeorge Zhang vmci_q_header_init((*produce_q)->q_header, *handle); 131906164d2bSGeorge Zhang vmci_q_header_init((*consume_q)->q_header, *handle); 132006164d2bSGeorge Zhang } 132106164d2bSGeorge Zhang 132206164d2bSGeorge Zhang mutex_unlock(&qp_guest_endpoints.mutex); 132306164d2bSGeorge Zhang 132406164d2bSGeorge Zhang return VMCI_SUCCESS; 132506164d2bSGeorge Zhang 132606164d2bSGeorge Zhang error: 132706164d2bSGeorge Zhang mutex_unlock(&qp_guest_endpoints.mutex); 132806164d2bSGeorge Zhang if (queue_pair_entry) { 132906164d2bSGeorge Zhang /* The queues will be freed inside the destroy routine. */ 133006164d2bSGeorge Zhang qp_guest_endpoint_destroy(queue_pair_entry); 133106164d2bSGeorge Zhang } else { 133206164d2bSGeorge Zhang qp_free_queue(my_produce_q, produce_size); 133306164d2bSGeorge Zhang qp_free_queue(my_consume_q, consume_size); 133406164d2bSGeorge Zhang } 133506164d2bSGeorge Zhang return result; 133606164d2bSGeorge Zhang 133706164d2bSGeorge Zhang error_keep_entry: 133806164d2bSGeorge Zhang /* This path should only be used when an existing entry was found. */ 133906164d2bSGeorge Zhang mutex_unlock(&qp_guest_endpoints.mutex); 134006164d2bSGeorge Zhang return result; 134106164d2bSGeorge Zhang } 134206164d2bSGeorge Zhang 134306164d2bSGeorge Zhang /* 134406164d2bSGeorge Zhang * The first endpoint issuing a queue pair allocation will create the state 134506164d2bSGeorge Zhang * of the queue pair in the queue pair broker. 134606164d2bSGeorge Zhang * 134706164d2bSGeorge Zhang * If the creator is a guest, it will associate a VMX virtual address range 134806164d2bSGeorge Zhang * with the queue pair as specified by the page_store. For compatibility with 134906164d2bSGeorge Zhang * older VMX'en, that would use a separate step to set the VMX virtual 135006164d2bSGeorge Zhang * address range, the virtual address range can be registered later using 135106164d2bSGeorge Zhang * vmci_qp_broker_set_page_store. In that case, a page_store of NULL should be 135206164d2bSGeorge Zhang * used. 135306164d2bSGeorge Zhang * 135406164d2bSGeorge Zhang * If the creator is the host, a page_store of NULL should be used as well, 135506164d2bSGeorge Zhang * since the host is not able to supply a page store for the queue pair. 135606164d2bSGeorge Zhang * 135706164d2bSGeorge Zhang * For older VMX and host callers, the queue pair will be created in the 135806164d2bSGeorge Zhang * VMCIQPB_CREATED_NO_MEM state, and for current VMX callers, it will be 135906164d2bSGeorge Zhang * created in VMCOQPB_CREATED_MEM state. 136006164d2bSGeorge Zhang */ 136106164d2bSGeorge Zhang static int qp_broker_create(struct vmci_handle handle, 136206164d2bSGeorge Zhang u32 peer, 136306164d2bSGeorge Zhang u32 flags, 136406164d2bSGeorge Zhang u32 priv_flags, 136506164d2bSGeorge Zhang u64 produce_size, 136606164d2bSGeorge Zhang u64 consume_size, 136706164d2bSGeorge Zhang struct vmci_qp_page_store *page_store, 136806164d2bSGeorge Zhang struct vmci_ctx *context, 136906164d2bSGeorge Zhang vmci_event_release_cb wakeup_cb, 137006164d2bSGeorge Zhang void *client_data, struct qp_broker_entry **ent) 137106164d2bSGeorge Zhang { 137206164d2bSGeorge Zhang struct qp_broker_entry *entry = NULL; 137306164d2bSGeorge Zhang const u32 context_id = vmci_ctx_get_id(context); 137406164d2bSGeorge Zhang bool is_local = flags & VMCI_QPFLAG_LOCAL; 137506164d2bSGeorge Zhang int result; 137606164d2bSGeorge Zhang u64 guest_produce_size; 137706164d2bSGeorge Zhang u64 guest_consume_size; 137806164d2bSGeorge Zhang 137906164d2bSGeorge Zhang /* Do not create if the caller asked not to. */ 138006164d2bSGeorge Zhang if (flags & VMCI_QPFLAG_ATTACH_ONLY) 138106164d2bSGeorge Zhang return VMCI_ERROR_NOT_FOUND; 138206164d2bSGeorge Zhang 138306164d2bSGeorge Zhang /* 138406164d2bSGeorge Zhang * Creator's context ID should match handle's context ID or the creator 138506164d2bSGeorge Zhang * must allow the context in handle's context ID as the "peer". 138606164d2bSGeorge Zhang */ 138706164d2bSGeorge Zhang if (handle.context != context_id && handle.context != peer) 138806164d2bSGeorge Zhang return VMCI_ERROR_NO_ACCESS; 138906164d2bSGeorge Zhang 139006164d2bSGeorge Zhang if (VMCI_CONTEXT_IS_VM(context_id) && VMCI_CONTEXT_IS_VM(peer)) 139106164d2bSGeorge Zhang return VMCI_ERROR_DST_UNREACHABLE; 139206164d2bSGeorge Zhang 139306164d2bSGeorge Zhang /* 139406164d2bSGeorge Zhang * Creator's context ID for local queue pairs should match the 139506164d2bSGeorge Zhang * peer, if a peer is specified. 139606164d2bSGeorge Zhang */ 139706164d2bSGeorge Zhang if (is_local && peer != VMCI_INVALID_ID && context_id != peer) 139806164d2bSGeorge Zhang return VMCI_ERROR_NO_ACCESS; 139906164d2bSGeorge Zhang 140006164d2bSGeorge Zhang entry = kzalloc(sizeof(*entry), GFP_ATOMIC); 140106164d2bSGeorge Zhang if (!entry) 140206164d2bSGeorge Zhang return VMCI_ERROR_NO_MEM; 140306164d2bSGeorge Zhang 140406164d2bSGeorge Zhang if (vmci_ctx_get_id(context) == VMCI_HOST_CONTEXT_ID && !is_local) { 140506164d2bSGeorge Zhang /* 140606164d2bSGeorge Zhang * The queue pair broker entry stores values from the guest 140706164d2bSGeorge Zhang * point of view, so a creating host side endpoint should swap 140806164d2bSGeorge Zhang * produce and consume values -- unless it is a local queue 140906164d2bSGeorge Zhang * pair, in which case no swapping is necessary, since the local 141006164d2bSGeorge Zhang * attacher will swap queues. 141106164d2bSGeorge Zhang */ 141206164d2bSGeorge Zhang 141306164d2bSGeorge Zhang guest_produce_size = consume_size; 141406164d2bSGeorge Zhang guest_consume_size = produce_size; 141506164d2bSGeorge Zhang } else { 141606164d2bSGeorge Zhang guest_produce_size = produce_size; 141706164d2bSGeorge Zhang guest_consume_size = consume_size; 141806164d2bSGeorge Zhang } 141906164d2bSGeorge Zhang 142006164d2bSGeorge Zhang entry->qp.handle = handle; 142106164d2bSGeorge Zhang entry->qp.peer = peer; 142206164d2bSGeorge Zhang entry->qp.flags = flags; 142306164d2bSGeorge Zhang entry->qp.produce_size = guest_produce_size; 142406164d2bSGeorge Zhang entry->qp.consume_size = guest_consume_size; 142506164d2bSGeorge Zhang entry->qp.ref_count = 1; 142606164d2bSGeorge Zhang entry->create_id = context_id; 142706164d2bSGeorge Zhang entry->attach_id = VMCI_INVALID_ID; 142806164d2bSGeorge Zhang entry->state = VMCIQPB_NEW; 142906164d2bSGeorge Zhang entry->require_trusted_attach = 143006164d2bSGeorge Zhang !!(context->priv_flags & VMCI_PRIVILEGE_FLAG_RESTRICTED); 143106164d2bSGeorge Zhang entry->created_by_trusted = 143206164d2bSGeorge Zhang !!(priv_flags & VMCI_PRIVILEGE_FLAG_TRUSTED); 143306164d2bSGeorge Zhang entry->vmci_page_files = false; 143406164d2bSGeorge Zhang entry->wakeup_cb = wakeup_cb; 143506164d2bSGeorge Zhang entry->client_data = client_data; 143606164d2bSGeorge Zhang entry->produce_q = qp_host_alloc_queue(guest_produce_size); 143706164d2bSGeorge Zhang if (entry->produce_q == NULL) { 143806164d2bSGeorge Zhang result = VMCI_ERROR_NO_MEM; 143906164d2bSGeorge Zhang goto error; 144006164d2bSGeorge Zhang } 144106164d2bSGeorge Zhang entry->consume_q = qp_host_alloc_queue(guest_consume_size); 144206164d2bSGeorge Zhang if (entry->consume_q == NULL) { 144306164d2bSGeorge Zhang result = VMCI_ERROR_NO_MEM; 144406164d2bSGeorge Zhang goto error; 144506164d2bSGeorge Zhang } 144606164d2bSGeorge Zhang 144706164d2bSGeorge Zhang qp_init_queue_mutex(entry->produce_q, entry->consume_q); 144806164d2bSGeorge Zhang 144906164d2bSGeorge Zhang INIT_LIST_HEAD(&entry->qp.list_item); 145006164d2bSGeorge Zhang 145106164d2bSGeorge Zhang if (is_local) { 145206164d2bSGeorge Zhang u8 *tmp; 145306164d2bSGeorge Zhang 145406164d2bSGeorge Zhang entry->local_mem = kcalloc(QPE_NUM_PAGES(entry->qp), 145506164d2bSGeorge Zhang PAGE_SIZE, GFP_KERNEL); 145606164d2bSGeorge Zhang if (entry->local_mem == NULL) { 145706164d2bSGeorge Zhang result = VMCI_ERROR_NO_MEM; 145806164d2bSGeorge Zhang goto error; 145906164d2bSGeorge Zhang } 146006164d2bSGeorge Zhang entry->state = VMCIQPB_CREATED_MEM; 146106164d2bSGeorge Zhang entry->produce_q->q_header = entry->local_mem; 146206164d2bSGeorge Zhang tmp = (u8 *)entry->local_mem + PAGE_SIZE * 146342281d20SAndy King (DIV_ROUND_UP(entry->qp.produce_size, PAGE_SIZE) + 1); 146406164d2bSGeorge Zhang entry->consume_q->q_header = (struct vmci_queue_header *)tmp; 146506164d2bSGeorge Zhang } else if (page_store) { 146606164d2bSGeorge Zhang /* 146706164d2bSGeorge Zhang * The VMX already initialized the queue pair headers, so no 146806164d2bSGeorge Zhang * need for the kernel side to do that. 146906164d2bSGeorge Zhang */ 147006164d2bSGeorge Zhang result = qp_host_register_user_memory(page_store, 147106164d2bSGeorge Zhang entry->produce_q, 147206164d2bSGeorge Zhang entry->consume_q); 147306164d2bSGeorge Zhang if (result < VMCI_SUCCESS) 147406164d2bSGeorge Zhang goto error; 147506164d2bSGeorge Zhang 147606164d2bSGeorge Zhang entry->state = VMCIQPB_CREATED_MEM; 147706164d2bSGeorge Zhang } else { 147806164d2bSGeorge Zhang /* 147906164d2bSGeorge Zhang * A create without a page_store may be either a host 148006164d2bSGeorge Zhang * side create (in which case we are waiting for the 148106164d2bSGeorge Zhang * guest side to supply the memory) or an old style 148206164d2bSGeorge Zhang * queue pair create (in which case we will expect a 148306164d2bSGeorge Zhang * set page store call as the next step). 148406164d2bSGeorge Zhang */ 148506164d2bSGeorge Zhang entry->state = VMCIQPB_CREATED_NO_MEM; 148606164d2bSGeorge Zhang } 148706164d2bSGeorge Zhang 148806164d2bSGeorge Zhang qp_list_add_entry(&qp_broker_list, &entry->qp); 148906164d2bSGeorge Zhang if (ent != NULL) 149006164d2bSGeorge Zhang *ent = entry; 149106164d2bSGeorge Zhang 149206164d2bSGeorge Zhang /* Add to resource obj */ 149306164d2bSGeorge Zhang result = vmci_resource_add(&entry->resource, 149406164d2bSGeorge Zhang VMCI_RESOURCE_TYPE_QPAIR_HOST, 149506164d2bSGeorge Zhang handle); 149606164d2bSGeorge Zhang if (result != VMCI_SUCCESS) { 149706164d2bSGeorge Zhang pr_warn("Failed to add new resource (handle=0x%x:0x%x), error: %d", 149806164d2bSGeorge Zhang handle.context, handle.resource, result); 149906164d2bSGeorge Zhang goto error; 150006164d2bSGeorge Zhang } 150106164d2bSGeorge Zhang 150206164d2bSGeorge Zhang entry->qp.handle = vmci_resource_handle(&entry->resource); 150306164d2bSGeorge Zhang if (is_local) { 150406164d2bSGeorge Zhang vmci_q_header_init(entry->produce_q->q_header, 150506164d2bSGeorge Zhang entry->qp.handle); 150606164d2bSGeorge Zhang vmci_q_header_init(entry->consume_q->q_header, 150706164d2bSGeorge Zhang entry->qp.handle); 150806164d2bSGeorge Zhang } 150906164d2bSGeorge Zhang 151006164d2bSGeorge Zhang vmci_ctx_qp_create(context, entry->qp.handle); 151106164d2bSGeorge Zhang 151206164d2bSGeorge Zhang return VMCI_SUCCESS; 151306164d2bSGeorge Zhang 151406164d2bSGeorge Zhang error: 151506164d2bSGeorge Zhang if (entry != NULL) { 151606164d2bSGeorge Zhang qp_host_free_queue(entry->produce_q, guest_produce_size); 151706164d2bSGeorge Zhang qp_host_free_queue(entry->consume_q, guest_consume_size); 151806164d2bSGeorge Zhang kfree(entry); 151906164d2bSGeorge Zhang } 152006164d2bSGeorge Zhang 152106164d2bSGeorge Zhang return result; 152206164d2bSGeorge Zhang } 152306164d2bSGeorge Zhang 152406164d2bSGeorge Zhang /* 152506164d2bSGeorge Zhang * Enqueues an event datagram to notify the peer VM attached to 152606164d2bSGeorge Zhang * the given queue pair handle about attach/detach event by the 152706164d2bSGeorge Zhang * given VM. Returns Payload size of datagram enqueued on 152806164d2bSGeorge Zhang * success, error code otherwise. 152906164d2bSGeorge Zhang */ 153006164d2bSGeorge Zhang static int qp_notify_peer(bool attach, 153106164d2bSGeorge Zhang struct vmci_handle handle, 153206164d2bSGeorge Zhang u32 my_id, 153306164d2bSGeorge Zhang u32 peer_id) 153406164d2bSGeorge Zhang { 153506164d2bSGeorge Zhang int rv; 153606164d2bSGeorge Zhang struct vmci_event_qp ev; 153706164d2bSGeorge Zhang 153806164d2bSGeorge Zhang if (vmci_handle_is_invalid(handle) || my_id == VMCI_INVALID_ID || 153906164d2bSGeorge Zhang peer_id == VMCI_INVALID_ID) 154006164d2bSGeorge Zhang return VMCI_ERROR_INVALID_ARGS; 154106164d2bSGeorge Zhang 154206164d2bSGeorge Zhang /* 154306164d2bSGeorge Zhang * In vmci_ctx_enqueue_datagram() we enforce the upper limit on 154406164d2bSGeorge Zhang * number of pending events from the hypervisor to a given VM 154506164d2bSGeorge Zhang * otherwise a rogue VM could do an arbitrary number of attach 154606164d2bSGeorge Zhang * and detach operations causing memory pressure in the host 154706164d2bSGeorge Zhang * kernel. 154806164d2bSGeorge Zhang */ 154906164d2bSGeorge Zhang 155006164d2bSGeorge Zhang ev.msg.hdr.dst = vmci_make_handle(peer_id, VMCI_EVENT_HANDLER); 155106164d2bSGeorge Zhang ev.msg.hdr.src = vmci_make_handle(VMCI_HYPERVISOR_CONTEXT_ID, 155206164d2bSGeorge Zhang VMCI_CONTEXT_RESOURCE_ID); 155306164d2bSGeorge Zhang ev.msg.hdr.payload_size = sizeof(ev) - sizeof(ev.msg.hdr); 155406164d2bSGeorge Zhang ev.msg.event_data.event = attach ? 155506164d2bSGeorge Zhang VMCI_EVENT_QP_PEER_ATTACH : VMCI_EVENT_QP_PEER_DETACH; 155606164d2bSGeorge Zhang ev.payload.handle = handle; 155706164d2bSGeorge Zhang ev.payload.peer_id = my_id; 155806164d2bSGeorge Zhang 155906164d2bSGeorge Zhang rv = vmci_datagram_dispatch(VMCI_HYPERVISOR_CONTEXT_ID, 156006164d2bSGeorge Zhang &ev.msg.hdr, false); 156106164d2bSGeorge Zhang if (rv < VMCI_SUCCESS) 156206164d2bSGeorge Zhang pr_warn("Failed to enqueue queue_pair %s event datagram for context (ID=0x%x)\n", 156306164d2bSGeorge Zhang attach ? "ATTACH" : "DETACH", peer_id); 156406164d2bSGeorge Zhang 156506164d2bSGeorge Zhang return rv; 156606164d2bSGeorge Zhang } 156706164d2bSGeorge Zhang 156806164d2bSGeorge Zhang /* 156906164d2bSGeorge Zhang * The second endpoint issuing a queue pair allocation will attach to 157006164d2bSGeorge Zhang * the queue pair registered with the queue pair broker. 157106164d2bSGeorge Zhang * 157206164d2bSGeorge Zhang * If the attacher is a guest, it will associate a VMX virtual address 157306164d2bSGeorge Zhang * range with the queue pair as specified by the page_store. At this 157406164d2bSGeorge Zhang * point, the already attach host endpoint may start using the queue 157506164d2bSGeorge Zhang * pair, and an attach event is sent to it. For compatibility with 157606164d2bSGeorge Zhang * older VMX'en, that used a separate step to set the VMX virtual 157706164d2bSGeorge Zhang * address range, the virtual address range can be registered later 157806164d2bSGeorge Zhang * using vmci_qp_broker_set_page_store. In that case, a page_store of 157906164d2bSGeorge Zhang * NULL should be used, and the attach event will be generated once 158006164d2bSGeorge Zhang * the actual page store has been set. 158106164d2bSGeorge Zhang * 158206164d2bSGeorge Zhang * If the attacher is the host, a page_store of NULL should be used as 158306164d2bSGeorge Zhang * well, since the page store information is already set by the guest. 158406164d2bSGeorge Zhang * 158506164d2bSGeorge Zhang * For new VMX and host callers, the queue pair will be moved to the 158606164d2bSGeorge Zhang * VMCIQPB_ATTACHED_MEM state, and for older VMX callers, it will be 158706164d2bSGeorge Zhang * moved to the VMCOQPB_ATTACHED_NO_MEM state. 158806164d2bSGeorge Zhang */ 158906164d2bSGeorge Zhang static int qp_broker_attach(struct qp_broker_entry *entry, 159006164d2bSGeorge Zhang u32 peer, 159106164d2bSGeorge Zhang u32 flags, 159206164d2bSGeorge Zhang u32 priv_flags, 159306164d2bSGeorge Zhang u64 produce_size, 159406164d2bSGeorge Zhang u64 consume_size, 159506164d2bSGeorge Zhang struct vmci_qp_page_store *page_store, 159606164d2bSGeorge Zhang struct vmci_ctx *context, 159706164d2bSGeorge Zhang vmci_event_release_cb wakeup_cb, 159806164d2bSGeorge Zhang void *client_data, 159906164d2bSGeorge Zhang struct qp_broker_entry **ent) 160006164d2bSGeorge Zhang { 160106164d2bSGeorge Zhang const u32 context_id = vmci_ctx_get_id(context); 160206164d2bSGeorge Zhang bool is_local = flags & VMCI_QPFLAG_LOCAL; 160306164d2bSGeorge Zhang int result; 160406164d2bSGeorge Zhang 160506164d2bSGeorge Zhang if (entry->state != VMCIQPB_CREATED_NO_MEM && 160606164d2bSGeorge Zhang entry->state != VMCIQPB_CREATED_MEM) 160706164d2bSGeorge Zhang return VMCI_ERROR_UNAVAILABLE; 160806164d2bSGeorge Zhang 160906164d2bSGeorge Zhang if (is_local) { 161006164d2bSGeorge Zhang if (!(entry->qp.flags & VMCI_QPFLAG_LOCAL) || 161106164d2bSGeorge Zhang context_id != entry->create_id) { 161206164d2bSGeorge Zhang return VMCI_ERROR_INVALID_ARGS; 161306164d2bSGeorge Zhang } 161406164d2bSGeorge Zhang } else if (context_id == entry->create_id || 161506164d2bSGeorge Zhang context_id == entry->attach_id) { 161606164d2bSGeorge Zhang return VMCI_ERROR_ALREADY_EXISTS; 161706164d2bSGeorge Zhang } 161806164d2bSGeorge Zhang 161906164d2bSGeorge Zhang if (VMCI_CONTEXT_IS_VM(context_id) && 162006164d2bSGeorge Zhang VMCI_CONTEXT_IS_VM(entry->create_id)) 162106164d2bSGeorge Zhang return VMCI_ERROR_DST_UNREACHABLE; 162206164d2bSGeorge Zhang 162306164d2bSGeorge Zhang /* 162406164d2bSGeorge Zhang * If we are attaching from a restricted context then the queuepair 162506164d2bSGeorge Zhang * must have been created by a trusted endpoint. 162606164d2bSGeorge Zhang */ 162706164d2bSGeorge Zhang if ((context->priv_flags & VMCI_PRIVILEGE_FLAG_RESTRICTED) && 162806164d2bSGeorge Zhang !entry->created_by_trusted) 162906164d2bSGeorge Zhang return VMCI_ERROR_NO_ACCESS; 163006164d2bSGeorge Zhang 163106164d2bSGeorge Zhang /* 163206164d2bSGeorge Zhang * If we are attaching to a queuepair that was created by a restricted 163306164d2bSGeorge Zhang * context then we must be trusted. 163406164d2bSGeorge Zhang */ 163506164d2bSGeorge Zhang if (entry->require_trusted_attach && 163606164d2bSGeorge Zhang (!(priv_flags & VMCI_PRIVILEGE_FLAG_TRUSTED))) 163706164d2bSGeorge Zhang return VMCI_ERROR_NO_ACCESS; 163806164d2bSGeorge Zhang 163906164d2bSGeorge Zhang /* 164006164d2bSGeorge Zhang * If the creator specifies VMCI_INVALID_ID in "peer" field, access 164106164d2bSGeorge Zhang * control check is not performed. 164206164d2bSGeorge Zhang */ 164306164d2bSGeorge Zhang if (entry->qp.peer != VMCI_INVALID_ID && entry->qp.peer != context_id) 164406164d2bSGeorge Zhang return VMCI_ERROR_NO_ACCESS; 164506164d2bSGeorge Zhang 164606164d2bSGeorge Zhang if (entry->create_id == VMCI_HOST_CONTEXT_ID) { 164706164d2bSGeorge Zhang /* 164806164d2bSGeorge Zhang * Do not attach if the caller doesn't support Host Queue Pairs 164906164d2bSGeorge Zhang * and a host created this queue pair. 165006164d2bSGeorge Zhang */ 165106164d2bSGeorge Zhang 165206164d2bSGeorge Zhang if (!vmci_ctx_supports_host_qp(context)) 165306164d2bSGeorge Zhang return VMCI_ERROR_INVALID_RESOURCE; 165406164d2bSGeorge Zhang 165506164d2bSGeorge Zhang } else if (context_id == VMCI_HOST_CONTEXT_ID) { 165606164d2bSGeorge Zhang struct vmci_ctx *create_context; 165706164d2bSGeorge Zhang bool supports_host_qp; 165806164d2bSGeorge Zhang 165906164d2bSGeorge Zhang /* 166006164d2bSGeorge Zhang * Do not attach a host to a user created queue pair if that 166106164d2bSGeorge Zhang * user doesn't support host queue pair end points. 166206164d2bSGeorge Zhang */ 166306164d2bSGeorge Zhang 166406164d2bSGeorge Zhang create_context = vmci_ctx_get(entry->create_id); 166506164d2bSGeorge Zhang supports_host_qp = vmci_ctx_supports_host_qp(create_context); 166606164d2bSGeorge Zhang vmci_ctx_put(create_context); 166706164d2bSGeorge Zhang 166806164d2bSGeorge Zhang if (!supports_host_qp) 166906164d2bSGeorge Zhang return VMCI_ERROR_INVALID_RESOURCE; 167006164d2bSGeorge Zhang } 167106164d2bSGeorge Zhang 167206164d2bSGeorge Zhang if ((entry->qp.flags & ~VMCI_QP_ASYMM) != (flags & ~VMCI_QP_ASYMM_PEER)) 167306164d2bSGeorge Zhang return VMCI_ERROR_QUEUEPAIR_MISMATCH; 167406164d2bSGeorge Zhang 167506164d2bSGeorge Zhang if (context_id != VMCI_HOST_CONTEXT_ID) { 167606164d2bSGeorge Zhang /* 167706164d2bSGeorge Zhang * The queue pair broker entry stores values from the guest 167806164d2bSGeorge Zhang * point of view, so an attaching guest should match the values 167906164d2bSGeorge Zhang * stored in the entry. 168006164d2bSGeorge Zhang */ 168106164d2bSGeorge Zhang 168206164d2bSGeorge Zhang if (entry->qp.produce_size != produce_size || 168306164d2bSGeorge Zhang entry->qp.consume_size != consume_size) { 168406164d2bSGeorge Zhang return VMCI_ERROR_QUEUEPAIR_MISMATCH; 168506164d2bSGeorge Zhang } 168606164d2bSGeorge Zhang } else if (entry->qp.produce_size != consume_size || 168706164d2bSGeorge Zhang entry->qp.consume_size != produce_size) { 168806164d2bSGeorge Zhang return VMCI_ERROR_QUEUEPAIR_MISMATCH; 168906164d2bSGeorge Zhang } 169006164d2bSGeorge Zhang 169106164d2bSGeorge Zhang if (context_id != VMCI_HOST_CONTEXT_ID) { 169206164d2bSGeorge Zhang /* 169306164d2bSGeorge Zhang * If a guest attached to a queue pair, it will supply 169406164d2bSGeorge Zhang * the backing memory. If this is a pre NOVMVM vmx, 169506164d2bSGeorge Zhang * the backing memory will be supplied by calling 169606164d2bSGeorge Zhang * vmci_qp_broker_set_page_store() following the 169706164d2bSGeorge Zhang * return of the vmci_qp_broker_alloc() call. If it is 169806164d2bSGeorge Zhang * a vmx of version NOVMVM or later, the page store 169906164d2bSGeorge Zhang * must be supplied as part of the 170006164d2bSGeorge Zhang * vmci_qp_broker_alloc call. Under all circumstances 170106164d2bSGeorge Zhang * must the initially created queue pair not have any 170206164d2bSGeorge Zhang * memory associated with it already. 170306164d2bSGeorge Zhang */ 170406164d2bSGeorge Zhang 170506164d2bSGeorge Zhang if (entry->state != VMCIQPB_CREATED_NO_MEM) 170606164d2bSGeorge Zhang return VMCI_ERROR_INVALID_ARGS; 170706164d2bSGeorge Zhang 170806164d2bSGeorge Zhang if (page_store != NULL) { 170906164d2bSGeorge Zhang /* 171006164d2bSGeorge Zhang * Patch up host state to point to guest 171106164d2bSGeorge Zhang * supplied memory. The VMX already 171206164d2bSGeorge Zhang * initialized the queue pair headers, so no 171306164d2bSGeorge Zhang * need for the kernel side to do that. 171406164d2bSGeorge Zhang */ 171506164d2bSGeorge Zhang 171606164d2bSGeorge Zhang result = qp_host_register_user_memory(page_store, 171706164d2bSGeorge Zhang entry->produce_q, 171806164d2bSGeorge Zhang entry->consume_q); 171906164d2bSGeorge Zhang if (result < VMCI_SUCCESS) 172006164d2bSGeorge Zhang return result; 172106164d2bSGeorge Zhang 172206164d2bSGeorge Zhang /* 172306164d2bSGeorge Zhang * Preemptively load in the headers if non-blocking to 172406164d2bSGeorge Zhang * prevent blocking later. 172506164d2bSGeorge Zhang */ 172606164d2bSGeorge Zhang if (entry->qp.flags & VMCI_QPFLAG_NONBLOCK) { 172706164d2bSGeorge Zhang result = qp_host_map_queues(entry->produce_q, 172806164d2bSGeorge Zhang entry->consume_q); 172906164d2bSGeorge Zhang if (result < VMCI_SUCCESS) { 173006164d2bSGeorge Zhang qp_host_unregister_user_memory( 173106164d2bSGeorge Zhang entry->produce_q, 173206164d2bSGeorge Zhang entry->consume_q); 173306164d2bSGeorge Zhang return result; 173406164d2bSGeorge Zhang } 173506164d2bSGeorge Zhang } 173606164d2bSGeorge Zhang 173706164d2bSGeorge Zhang entry->state = VMCIQPB_ATTACHED_MEM; 173806164d2bSGeorge Zhang } else { 173906164d2bSGeorge Zhang entry->state = VMCIQPB_ATTACHED_NO_MEM; 174006164d2bSGeorge Zhang } 174106164d2bSGeorge Zhang } else if (entry->state == VMCIQPB_CREATED_NO_MEM) { 174206164d2bSGeorge Zhang /* 174306164d2bSGeorge Zhang * The host side is attempting to attach to a queue 174406164d2bSGeorge Zhang * pair that doesn't have any memory associated with 174506164d2bSGeorge Zhang * it. This must be a pre NOVMVM vmx that hasn't set 174606164d2bSGeorge Zhang * the page store information yet, or a quiesced VM. 174706164d2bSGeorge Zhang */ 174806164d2bSGeorge Zhang 174906164d2bSGeorge Zhang return VMCI_ERROR_UNAVAILABLE; 175006164d2bSGeorge Zhang } else { 175106164d2bSGeorge Zhang /* 175206164d2bSGeorge Zhang * For non-blocking queue pairs, we cannot rely on 175306164d2bSGeorge Zhang * enqueue/dequeue to map in the pages on the 175406164d2bSGeorge Zhang * host-side, since it may block, so we make an 175506164d2bSGeorge Zhang * attempt here. 175606164d2bSGeorge Zhang */ 175706164d2bSGeorge Zhang 175806164d2bSGeorge Zhang if (flags & VMCI_QPFLAG_NONBLOCK) { 175906164d2bSGeorge Zhang result = 176006164d2bSGeorge Zhang qp_host_map_queues(entry->produce_q, 176106164d2bSGeorge Zhang entry->consume_q); 176206164d2bSGeorge Zhang if (result < VMCI_SUCCESS) 176306164d2bSGeorge Zhang return result; 176406164d2bSGeorge Zhang 176506164d2bSGeorge Zhang entry->qp.flags |= flags & 176606164d2bSGeorge Zhang (VMCI_QPFLAG_NONBLOCK | VMCI_QPFLAG_PINNED); 176706164d2bSGeorge Zhang } 176806164d2bSGeorge Zhang 176906164d2bSGeorge Zhang /* The host side has successfully attached to a queue pair. */ 177006164d2bSGeorge Zhang entry->state = VMCIQPB_ATTACHED_MEM; 177106164d2bSGeorge Zhang } 177206164d2bSGeorge Zhang 177306164d2bSGeorge Zhang if (entry->state == VMCIQPB_ATTACHED_MEM) { 177406164d2bSGeorge Zhang result = 177506164d2bSGeorge Zhang qp_notify_peer(true, entry->qp.handle, context_id, 177606164d2bSGeorge Zhang entry->create_id); 177706164d2bSGeorge Zhang if (result < VMCI_SUCCESS) 177806164d2bSGeorge Zhang pr_warn("Failed to notify peer (ID=0x%x) of attach to queue pair (handle=0x%x:0x%x)\n", 177906164d2bSGeorge Zhang entry->create_id, entry->qp.handle.context, 178006164d2bSGeorge Zhang entry->qp.handle.resource); 178106164d2bSGeorge Zhang } 178206164d2bSGeorge Zhang 178306164d2bSGeorge Zhang entry->attach_id = context_id; 178406164d2bSGeorge Zhang entry->qp.ref_count++; 178506164d2bSGeorge Zhang if (wakeup_cb) { 178606164d2bSGeorge Zhang entry->wakeup_cb = wakeup_cb; 178706164d2bSGeorge Zhang entry->client_data = client_data; 178806164d2bSGeorge Zhang } 178906164d2bSGeorge Zhang 179006164d2bSGeorge Zhang /* 179106164d2bSGeorge Zhang * When attaching to local queue pairs, the context already has 179206164d2bSGeorge Zhang * an entry tracking the queue pair, so don't add another one. 179306164d2bSGeorge Zhang */ 179406164d2bSGeorge Zhang if (!is_local) 179506164d2bSGeorge Zhang vmci_ctx_qp_create(context, entry->qp.handle); 179606164d2bSGeorge Zhang 179706164d2bSGeorge Zhang if (ent != NULL) 179806164d2bSGeorge Zhang *ent = entry; 179906164d2bSGeorge Zhang 180006164d2bSGeorge Zhang return VMCI_SUCCESS; 180106164d2bSGeorge Zhang } 180206164d2bSGeorge Zhang 180306164d2bSGeorge Zhang /* 180406164d2bSGeorge Zhang * queue_pair_Alloc for use when setting up queue pair endpoints 180506164d2bSGeorge Zhang * on the host. 180606164d2bSGeorge Zhang */ 180706164d2bSGeorge Zhang static int qp_broker_alloc(struct vmci_handle handle, 180806164d2bSGeorge Zhang u32 peer, 180906164d2bSGeorge Zhang u32 flags, 181006164d2bSGeorge Zhang u32 priv_flags, 181106164d2bSGeorge Zhang u64 produce_size, 181206164d2bSGeorge Zhang u64 consume_size, 181306164d2bSGeorge Zhang struct vmci_qp_page_store *page_store, 181406164d2bSGeorge Zhang struct vmci_ctx *context, 181506164d2bSGeorge Zhang vmci_event_release_cb wakeup_cb, 181606164d2bSGeorge Zhang void *client_data, 181706164d2bSGeorge Zhang struct qp_broker_entry **ent, 181806164d2bSGeorge Zhang bool *swap) 181906164d2bSGeorge Zhang { 182006164d2bSGeorge Zhang const u32 context_id = vmci_ctx_get_id(context); 182106164d2bSGeorge Zhang bool create; 182206164d2bSGeorge Zhang struct qp_broker_entry *entry = NULL; 182306164d2bSGeorge Zhang bool is_local = flags & VMCI_QPFLAG_LOCAL; 182406164d2bSGeorge Zhang int result; 182506164d2bSGeorge Zhang 182606164d2bSGeorge Zhang if (vmci_handle_is_invalid(handle) || 182706164d2bSGeorge Zhang (flags & ~VMCI_QP_ALL_FLAGS) || is_local || 182806164d2bSGeorge Zhang !(produce_size || consume_size) || 182906164d2bSGeorge Zhang !context || context_id == VMCI_INVALID_ID || 183006164d2bSGeorge Zhang handle.context == VMCI_INVALID_ID) { 183106164d2bSGeorge Zhang return VMCI_ERROR_INVALID_ARGS; 183206164d2bSGeorge Zhang } 183306164d2bSGeorge Zhang 183406164d2bSGeorge Zhang if (page_store && !VMCI_QP_PAGESTORE_IS_WELLFORMED(page_store)) 183506164d2bSGeorge Zhang return VMCI_ERROR_INVALID_ARGS; 183606164d2bSGeorge Zhang 183706164d2bSGeorge Zhang /* 183806164d2bSGeorge Zhang * In the initial argument check, we ensure that non-vmkernel hosts 183906164d2bSGeorge Zhang * are not allowed to create local queue pairs. 184006164d2bSGeorge Zhang */ 184106164d2bSGeorge Zhang 184206164d2bSGeorge Zhang mutex_lock(&qp_broker_list.mutex); 184306164d2bSGeorge Zhang 184406164d2bSGeorge Zhang if (!is_local && vmci_ctx_qp_exists(context, handle)) { 184506164d2bSGeorge Zhang pr_devel("Context (ID=0x%x) already attached to queue pair (handle=0x%x:0x%x)\n", 184606164d2bSGeorge Zhang context_id, handle.context, handle.resource); 184706164d2bSGeorge Zhang mutex_unlock(&qp_broker_list.mutex); 184806164d2bSGeorge Zhang return VMCI_ERROR_ALREADY_EXISTS; 184906164d2bSGeorge Zhang } 185006164d2bSGeorge Zhang 185106164d2bSGeorge Zhang if (handle.resource != VMCI_INVALID_ID) 185206164d2bSGeorge Zhang entry = qp_broker_handle_to_entry(handle); 185306164d2bSGeorge Zhang 185406164d2bSGeorge Zhang if (!entry) { 185506164d2bSGeorge Zhang create = true; 185606164d2bSGeorge Zhang result = 185706164d2bSGeorge Zhang qp_broker_create(handle, peer, flags, priv_flags, 185806164d2bSGeorge Zhang produce_size, consume_size, page_store, 185906164d2bSGeorge Zhang context, wakeup_cb, client_data, ent); 186006164d2bSGeorge Zhang } else { 186106164d2bSGeorge Zhang create = false; 186206164d2bSGeorge Zhang result = 186306164d2bSGeorge Zhang qp_broker_attach(entry, peer, flags, priv_flags, 186406164d2bSGeorge Zhang produce_size, consume_size, page_store, 186506164d2bSGeorge Zhang context, wakeup_cb, client_data, ent); 186606164d2bSGeorge Zhang } 186706164d2bSGeorge Zhang 186806164d2bSGeorge Zhang mutex_unlock(&qp_broker_list.mutex); 186906164d2bSGeorge Zhang 187006164d2bSGeorge Zhang if (swap) 187106164d2bSGeorge Zhang *swap = (context_id == VMCI_HOST_CONTEXT_ID) && 187206164d2bSGeorge Zhang !(create && is_local); 187306164d2bSGeorge Zhang 187406164d2bSGeorge Zhang return result; 187506164d2bSGeorge Zhang } 187606164d2bSGeorge Zhang 187706164d2bSGeorge Zhang /* 187806164d2bSGeorge Zhang * This function implements the kernel API for allocating a queue 187906164d2bSGeorge Zhang * pair. 188006164d2bSGeorge Zhang */ 188106164d2bSGeorge Zhang static int qp_alloc_host_work(struct vmci_handle *handle, 188206164d2bSGeorge Zhang struct vmci_queue **produce_q, 188306164d2bSGeorge Zhang u64 produce_size, 188406164d2bSGeorge Zhang struct vmci_queue **consume_q, 188506164d2bSGeorge Zhang u64 consume_size, 188606164d2bSGeorge Zhang u32 peer, 188706164d2bSGeorge Zhang u32 flags, 188806164d2bSGeorge Zhang u32 priv_flags, 188906164d2bSGeorge Zhang vmci_event_release_cb wakeup_cb, 189006164d2bSGeorge Zhang void *client_data) 189106164d2bSGeorge Zhang { 189206164d2bSGeorge Zhang struct vmci_handle new_handle; 189306164d2bSGeorge Zhang struct vmci_ctx *context; 189406164d2bSGeorge Zhang struct qp_broker_entry *entry; 189506164d2bSGeorge Zhang int result; 189606164d2bSGeorge Zhang bool swap; 189706164d2bSGeorge Zhang 189806164d2bSGeorge Zhang if (vmci_handle_is_invalid(*handle)) { 189906164d2bSGeorge Zhang new_handle = vmci_make_handle( 190006164d2bSGeorge Zhang VMCI_HOST_CONTEXT_ID, VMCI_INVALID_ID); 190106164d2bSGeorge Zhang } else 190206164d2bSGeorge Zhang new_handle = *handle; 190306164d2bSGeorge Zhang 190406164d2bSGeorge Zhang context = vmci_ctx_get(VMCI_HOST_CONTEXT_ID); 190506164d2bSGeorge Zhang entry = NULL; 190606164d2bSGeorge Zhang result = 190706164d2bSGeorge Zhang qp_broker_alloc(new_handle, peer, flags, priv_flags, 190806164d2bSGeorge Zhang produce_size, consume_size, NULL, context, 190906164d2bSGeorge Zhang wakeup_cb, client_data, &entry, &swap); 191006164d2bSGeorge Zhang if (result == VMCI_SUCCESS) { 191106164d2bSGeorge Zhang if (swap) { 191206164d2bSGeorge Zhang /* 191306164d2bSGeorge Zhang * If this is a local queue pair, the attacher 191406164d2bSGeorge Zhang * will swap around produce and consume 191506164d2bSGeorge Zhang * queues. 191606164d2bSGeorge Zhang */ 191706164d2bSGeorge Zhang 191806164d2bSGeorge Zhang *produce_q = entry->consume_q; 191906164d2bSGeorge Zhang *consume_q = entry->produce_q; 192006164d2bSGeorge Zhang } else { 192106164d2bSGeorge Zhang *produce_q = entry->produce_q; 192206164d2bSGeorge Zhang *consume_q = entry->consume_q; 192306164d2bSGeorge Zhang } 192406164d2bSGeorge Zhang 192506164d2bSGeorge Zhang *handle = vmci_resource_handle(&entry->resource); 192606164d2bSGeorge Zhang } else { 192706164d2bSGeorge Zhang *handle = VMCI_INVALID_HANDLE; 192806164d2bSGeorge Zhang pr_devel("queue pair broker failed to alloc (result=%d)\n", 192906164d2bSGeorge Zhang result); 193006164d2bSGeorge Zhang } 193106164d2bSGeorge Zhang vmci_ctx_put(context); 193206164d2bSGeorge Zhang return result; 193306164d2bSGeorge Zhang } 193406164d2bSGeorge Zhang 193506164d2bSGeorge Zhang /* 193606164d2bSGeorge Zhang * Allocates a VMCI queue_pair. Only checks validity of input 193706164d2bSGeorge Zhang * arguments. The real work is done in the host or guest 193806164d2bSGeorge Zhang * specific function. 193906164d2bSGeorge Zhang */ 194006164d2bSGeorge Zhang int vmci_qp_alloc(struct vmci_handle *handle, 194106164d2bSGeorge Zhang struct vmci_queue **produce_q, 194206164d2bSGeorge Zhang u64 produce_size, 194306164d2bSGeorge Zhang struct vmci_queue **consume_q, 194406164d2bSGeorge Zhang u64 consume_size, 194506164d2bSGeorge Zhang u32 peer, 194606164d2bSGeorge Zhang u32 flags, 194706164d2bSGeorge Zhang u32 priv_flags, 194806164d2bSGeorge Zhang bool guest_endpoint, 194906164d2bSGeorge Zhang vmci_event_release_cb wakeup_cb, 195006164d2bSGeorge Zhang void *client_data) 195106164d2bSGeorge Zhang { 195206164d2bSGeorge Zhang if (!handle || !produce_q || !consume_q || 195306164d2bSGeorge Zhang (!produce_size && !consume_size) || (flags & ~VMCI_QP_ALL_FLAGS)) 195406164d2bSGeorge Zhang return VMCI_ERROR_INVALID_ARGS; 195506164d2bSGeorge Zhang 195606164d2bSGeorge Zhang if (guest_endpoint) { 195706164d2bSGeorge Zhang return qp_alloc_guest_work(handle, produce_q, 195806164d2bSGeorge Zhang produce_size, consume_q, 195906164d2bSGeorge Zhang consume_size, peer, 196006164d2bSGeorge Zhang flags, priv_flags); 196106164d2bSGeorge Zhang } else { 196206164d2bSGeorge Zhang return qp_alloc_host_work(handle, produce_q, 196306164d2bSGeorge Zhang produce_size, consume_q, 196406164d2bSGeorge Zhang consume_size, peer, flags, 196506164d2bSGeorge Zhang priv_flags, wakeup_cb, client_data); 196606164d2bSGeorge Zhang } 196706164d2bSGeorge Zhang } 196806164d2bSGeorge Zhang 196906164d2bSGeorge Zhang /* 197006164d2bSGeorge Zhang * This function implements the host kernel API for detaching from 197106164d2bSGeorge Zhang * a queue pair. 197206164d2bSGeorge Zhang */ 197306164d2bSGeorge Zhang static int qp_detatch_host_work(struct vmci_handle handle) 197406164d2bSGeorge Zhang { 197506164d2bSGeorge Zhang int result; 197606164d2bSGeorge Zhang struct vmci_ctx *context; 197706164d2bSGeorge Zhang 197806164d2bSGeorge Zhang context = vmci_ctx_get(VMCI_HOST_CONTEXT_ID); 197906164d2bSGeorge Zhang 198006164d2bSGeorge Zhang result = vmci_qp_broker_detach(handle, context); 198106164d2bSGeorge Zhang 198206164d2bSGeorge Zhang vmci_ctx_put(context); 198306164d2bSGeorge Zhang return result; 198406164d2bSGeorge Zhang } 198506164d2bSGeorge Zhang 198606164d2bSGeorge Zhang /* 198706164d2bSGeorge Zhang * Detaches from a VMCI queue_pair. Only checks validity of input argument. 198806164d2bSGeorge Zhang * Real work is done in the host or guest specific function. 198906164d2bSGeorge Zhang */ 199006164d2bSGeorge Zhang static int qp_detatch(struct vmci_handle handle, bool guest_endpoint) 199106164d2bSGeorge Zhang { 199206164d2bSGeorge Zhang if (vmci_handle_is_invalid(handle)) 199306164d2bSGeorge Zhang return VMCI_ERROR_INVALID_ARGS; 199406164d2bSGeorge Zhang 199506164d2bSGeorge Zhang if (guest_endpoint) 199606164d2bSGeorge Zhang return qp_detatch_guest_work(handle); 199706164d2bSGeorge Zhang else 199806164d2bSGeorge Zhang return qp_detatch_host_work(handle); 199906164d2bSGeorge Zhang } 200006164d2bSGeorge Zhang 200106164d2bSGeorge Zhang /* 200206164d2bSGeorge Zhang * Returns the entry from the head of the list. Assumes that the list is 200306164d2bSGeorge Zhang * locked. 200406164d2bSGeorge Zhang */ 200506164d2bSGeorge Zhang static struct qp_entry *qp_list_get_head(struct qp_list *qp_list) 200606164d2bSGeorge Zhang { 200706164d2bSGeorge Zhang if (!list_empty(&qp_list->head)) { 200806164d2bSGeorge Zhang struct qp_entry *entry = 200906164d2bSGeorge Zhang list_first_entry(&qp_list->head, struct qp_entry, 201006164d2bSGeorge Zhang list_item); 201106164d2bSGeorge Zhang return entry; 201206164d2bSGeorge Zhang } 201306164d2bSGeorge Zhang 201406164d2bSGeorge Zhang return NULL; 201506164d2bSGeorge Zhang } 201606164d2bSGeorge Zhang 201706164d2bSGeorge Zhang void vmci_qp_broker_exit(void) 201806164d2bSGeorge Zhang { 201906164d2bSGeorge Zhang struct qp_entry *entry; 202006164d2bSGeorge Zhang struct qp_broker_entry *be; 202106164d2bSGeorge Zhang 202206164d2bSGeorge Zhang mutex_lock(&qp_broker_list.mutex); 202306164d2bSGeorge Zhang 202406164d2bSGeorge Zhang while ((entry = qp_list_get_head(&qp_broker_list))) { 202506164d2bSGeorge Zhang be = (struct qp_broker_entry *)entry; 202606164d2bSGeorge Zhang 202706164d2bSGeorge Zhang qp_list_remove_entry(&qp_broker_list, entry); 202806164d2bSGeorge Zhang kfree(be); 202906164d2bSGeorge Zhang } 203006164d2bSGeorge Zhang 203106164d2bSGeorge Zhang mutex_unlock(&qp_broker_list.mutex); 203206164d2bSGeorge Zhang } 203306164d2bSGeorge Zhang 203406164d2bSGeorge Zhang /* 203506164d2bSGeorge Zhang * Requests that a queue pair be allocated with the VMCI queue 203606164d2bSGeorge Zhang * pair broker. Allocates a queue pair entry if one does not 203706164d2bSGeorge Zhang * exist. Attaches to one if it exists, and retrieves the page 203806164d2bSGeorge Zhang * files backing that queue_pair. Assumes that the queue pair 203906164d2bSGeorge Zhang * broker lock is held. 204006164d2bSGeorge Zhang */ 204106164d2bSGeorge Zhang int vmci_qp_broker_alloc(struct vmci_handle handle, 204206164d2bSGeorge Zhang u32 peer, 204306164d2bSGeorge Zhang u32 flags, 204406164d2bSGeorge Zhang u32 priv_flags, 204506164d2bSGeorge Zhang u64 produce_size, 204606164d2bSGeorge Zhang u64 consume_size, 204706164d2bSGeorge Zhang struct vmci_qp_page_store *page_store, 204806164d2bSGeorge Zhang struct vmci_ctx *context) 204906164d2bSGeorge Zhang { 205006164d2bSGeorge Zhang return qp_broker_alloc(handle, peer, flags, priv_flags, 205106164d2bSGeorge Zhang produce_size, consume_size, 205206164d2bSGeorge Zhang page_store, context, NULL, NULL, NULL, NULL); 205306164d2bSGeorge Zhang } 205406164d2bSGeorge Zhang 205506164d2bSGeorge Zhang /* 205606164d2bSGeorge Zhang * VMX'en with versions lower than VMCI_VERSION_NOVMVM use a separate 205706164d2bSGeorge Zhang * step to add the UVAs of the VMX mapping of the queue pair. This function 205806164d2bSGeorge Zhang * provides backwards compatibility with such VMX'en, and takes care of 205906164d2bSGeorge Zhang * registering the page store for a queue pair previously allocated by the 206006164d2bSGeorge Zhang * VMX during create or attach. This function will move the queue pair state 206106164d2bSGeorge Zhang * to either from VMCIQBP_CREATED_NO_MEM to VMCIQBP_CREATED_MEM or 206206164d2bSGeorge Zhang * VMCIQBP_ATTACHED_NO_MEM to VMCIQBP_ATTACHED_MEM. If moving to the 206306164d2bSGeorge Zhang * attached state with memory, the queue pair is ready to be used by the 206406164d2bSGeorge Zhang * host peer, and an attached event will be generated. 206506164d2bSGeorge Zhang * 206606164d2bSGeorge Zhang * Assumes that the queue pair broker lock is held. 206706164d2bSGeorge Zhang * 206806164d2bSGeorge Zhang * This function is only used by the hosted platform, since there is no 206906164d2bSGeorge Zhang * issue with backwards compatibility for vmkernel. 207006164d2bSGeorge Zhang */ 207106164d2bSGeorge Zhang int vmci_qp_broker_set_page_store(struct vmci_handle handle, 207206164d2bSGeorge Zhang u64 produce_uva, 207306164d2bSGeorge Zhang u64 consume_uva, 207406164d2bSGeorge Zhang struct vmci_ctx *context) 207506164d2bSGeorge Zhang { 207606164d2bSGeorge Zhang struct qp_broker_entry *entry; 207706164d2bSGeorge Zhang int result; 207806164d2bSGeorge Zhang const u32 context_id = vmci_ctx_get_id(context); 207906164d2bSGeorge Zhang 208006164d2bSGeorge Zhang if (vmci_handle_is_invalid(handle) || !context || 208106164d2bSGeorge Zhang context_id == VMCI_INVALID_ID) 208206164d2bSGeorge Zhang return VMCI_ERROR_INVALID_ARGS; 208306164d2bSGeorge Zhang 208406164d2bSGeorge Zhang /* 208506164d2bSGeorge Zhang * We only support guest to host queue pairs, so the VMX must 208606164d2bSGeorge Zhang * supply UVAs for the mapped page files. 208706164d2bSGeorge Zhang */ 208806164d2bSGeorge Zhang 208906164d2bSGeorge Zhang if (produce_uva == 0 || consume_uva == 0) 209006164d2bSGeorge Zhang return VMCI_ERROR_INVALID_ARGS; 209106164d2bSGeorge Zhang 209206164d2bSGeorge Zhang mutex_lock(&qp_broker_list.mutex); 209306164d2bSGeorge Zhang 209406164d2bSGeorge Zhang if (!vmci_ctx_qp_exists(context, handle)) { 209506164d2bSGeorge Zhang pr_warn("Context (ID=0x%x) not attached to queue pair (handle=0x%x:0x%x)\n", 209606164d2bSGeorge Zhang context_id, handle.context, handle.resource); 209706164d2bSGeorge Zhang result = VMCI_ERROR_NOT_FOUND; 209806164d2bSGeorge Zhang goto out; 209906164d2bSGeorge Zhang } 210006164d2bSGeorge Zhang 210106164d2bSGeorge Zhang entry = qp_broker_handle_to_entry(handle); 210206164d2bSGeorge Zhang if (!entry) { 210306164d2bSGeorge Zhang result = VMCI_ERROR_NOT_FOUND; 210406164d2bSGeorge Zhang goto out; 210506164d2bSGeorge Zhang } 210606164d2bSGeorge Zhang 210706164d2bSGeorge Zhang /* 210806164d2bSGeorge Zhang * If I'm the owner then I can set the page store. 210906164d2bSGeorge Zhang * 211006164d2bSGeorge Zhang * Or, if a host created the queue_pair and I'm the attached peer 211106164d2bSGeorge Zhang * then I can set the page store. 211206164d2bSGeorge Zhang */ 211306164d2bSGeorge Zhang if (entry->create_id != context_id && 211406164d2bSGeorge Zhang (entry->create_id != VMCI_HOST_CONTEXT_ID || 211506164d2bSGeorge Zhang entry->attach_id != context_id)) { 211606164d2bSGeorge Zhang result = VMCI_ERROR_QUEUEPAIR_NOTOWNER; 211706164d2bSGeorge Zhang goto out; 211806164d2bSGeorge Zhang } 211906164d2bSGeorge Zhang 212006164d2bSGeorge Zhang if (entry->state != VMCIQPB_CREATED_NO_MEM && 212106164d2bSGeorge Zhang entry->state != VMCIQPB_ATTACHED_NO_MEM) { 212206164d2bSGeorge Zhang result = VMCI_ERROR_UNAVAILABLE; 212306164d2bSGeorge Zhang goto out; 212406164d2bSGeorge Zhang } 212506164d2bSGeorge Zhang 212606164d2bSGeorge Zhang result = qp_host_get_user_memory(produce_uva, consume_uva, 212706164d2bSGeorge Zhang entry->produce_q, entry->consume_q); 212806164d2bSGeorge Zhang if (result < VMCI_SUCCESS) 212906164d2bSGeorge Zhang goto out; 213006164d2bSGeorge Zhang 213106164d2bSGeorge Zhang result = qp_host_map_queues(entry->produce_q, entry->consume_q); 213206164d2bSGeorge Zhang if (result < VMCI_SUCCESS) { 213306164d2bSGeorge Zhang qp_host_unregister_user_memory(entry->produce_q, 213406164d2bSGeorge Zhang entry->consume_q); 213506164d2bSGeorge Zhang goto out; 213606164d2bSGeorge Zhang } 213706164d2bSGeorge Zhang 213806164d2bSGeorge Zhang if (entry->state == VMCIQPB_CREATED_NO_MEM) 213906164d2bSGeorge Zhang entry->state = VMCIQPB_CREATED_MEM; 214006164d2bSGeorge Zhang else 214106164d2bSGeorge Zhang entry->state = VMCIQPB_ATTACHED_MEM; 214206164d2bSGeorge Zhang 214306164d2bSGeorge Zhang entry->vmci_page_files = true; 214406164d2bSGeorge Zhang 214506164d2bSGeorge Zhang if (entry->state == VMCIQPB_ATTACHED_MEM) { 214606164d2bSGeorge Zhang result = 214706164d2bSGeorge Zhang qp_notify_peer(true, handle, context_id, entry->create_id); 214806164d2bSGeorge Zhang if (result < VMCI_SUCCESS) { 214906164d2bSGeorge Zhang pr_warn("Failed to notify peer (ID=0x%x) of attach to queue pair (handle=0x%x:0x%x)\n", 215006164d2bSGeorge Zhang entry->create_id, entry->qp.handle.context, 215106164d2bSGeorge Zhang entry->qp.handle.resource); 215206164d2bSGeorge Zhang } 215306164d2bSGeorge Zhang } 215406164d2bSGeorge Zhang 215506164d2bSGeorge Zhang result = VMCI_SUCCESS; 215606164d2bSGeorge Zhang out: 215706164d2bSGeorge Zhang mutex_unlock(&qp_broker_list.mutex); 215806164d2bSGeorge Zhang return result; 215906164d2bSGeorge Zhang } 216006164d2bSGeorge Zhang 216106164d2bSGeorge Zhang /* 216206164d2bSGeorge Zhang * Resets saved queue headers for the given QP broker 216306164d2bSGeorge Zhang * entry. Should be used when guest memory becomes available 216406164d2bSGeorge Zhang * again, or the guest detaches. 216506164d2bSGeorge Zhang */ 216606164d2bSGeorge Zhang static void qp_reset_saved_headers(struct qp_broker_entry *entry) 216706164d2bSGeorge Zhang { 216806164d2bSGeorge Zhang entry->produce_q->saved_header = NULL; 216906164d2bSGeorge Zhang entry->consume_q->saved_header = NULL; 217006164d2bSGeorge Zhang } 217106164d2bSGeorge Zhang 217206164d2bSGeorge Zhang /* 217306164d2bSGeorge Zhang * The main entry point for detaching from a queue pair registered with the 217406164d2bSGeorge Zhang * queue pair broker. If more than one endpoint is attached to the queue 217506164d2bSGeorge Zhang * pair, the first endpoint will mainly decrement a reference count and 217606164d2bSGeorge Zhang * generate a notification to its peer. The last endpoint will clean up 217706164d2bSGeorge Zhang * the queue pair state registered with the broker. 217806164d2bSGeorge Zhang * 217906164d2bSGeorge Zhang * When a guest endpoint detaches, it will unmap and unregister the guest 218006164d2bSGeorge Zhang * memory backing the queue pair. If the host is still attached, it will 218106164d2bSGeorge Zhang * no longer be able to access the queue pair content. 218206164d2bSGeorge Zhang * 218306164d2bSGeorge Zhang * If the queue pair is already in a state where there is no memory 218406164d2bSGeorge Zhang * registered for the queue pair (any *_NO_MEM state), it will transition to 218506164d2bSGeorge Zhang * the VMCIQPB_SHUTDOWN_NO_MEM state. This will also happen, if a guest 218606164d2bSGeorge Zhang * endpoint is the first of two endpoints to detach. If the host endpoint is 218706164d2bSGeorge Zhang * the first out of two to detach, the queue pair will move to the 218806164d2bSGeorge Zhang * VMCIQPB_SHUTDOWN_MEM state. 218906164d2bSGeorge Zhang */ 219006164d2bSGeorge Zhang int vmci_qp_broker_detach(struct vmci_handle handle, struct vmci_ctx *context) 219106164d2bSGeorge Zhang { 219206164d2bSGeorge Zhang struct qp_broker_entry *entry; 219306164d2bSGeorge Zhang const u32 context_id = vmci_ctx_get_id(context); 219406164d2bSGeorge Zhang u32 peer_id; 219506164d2bSGeorge Zhang bool is_local = false; 219606164d2bSGeorge Zhang int result; 219706164d2bSGeorge Zhang 219806164d2bSGeorge Zhang if (vmci_handle_is_invalid(handle) || !context || 219906164d2bSGeorge Zhang context_id == VMCI_INVALID_ID) { 220006164d2bSGeorge Zhang return VMCI_ERROR_INVALID_ARGS; 220106164d2bSGeorge Zhang } 220206164d2bSGeorge Zhang 220306164d2bSGeorge Zhang mutex_lock(&qp_broker_list.mutex); 220406164d2bSGeorge Zhang 220506164d2bSGeorge Zhang if (!vmci_ctx_qp_exists(context, handle)) { 220606164d2bSGeorge Zhang pr_devel("Context (ID=0x%x) not attached to queue pair (handle=0x%x:0x%x)\n", 220706164d2bSGeorge Zhang context_id, handle.context, handle.resource); 220806164d2bSGeorge Zhang result = VMCI_ERROR_NOT_FOUND; 220906164d2bSGeorge Zhang goto out; 221006164d2bSGeorge Zhang } 221106164d2bSGeorge Zhang 221206164d2bSGeorge Zhang entry = qp_broker_handle_to_entry(handle); 221306164d2bSGeorge Zhang if (!entry) { 221406164d2bSGeorge Zhang pr_devel("Context (ID=0x%x) reports being attached to queue pair(handle=0x%x:0x%x) that isn't present in broker\n", 221506164d2bSGeorge Zhang context_id, handle.context, handle.resource); 221606164d2bSGeorge Zhang result = VMCI_ERROR_NOT_FOUND; 221706164d2bSGeorge Zhang goto out; 221806164d2bSGeorge Zhang } 221906164d2bSGeorge Zhang 222006164d2bSGeorge Zhang if (context_id != entry->create_id && context_id != entry->attach_id) { 222106164d2bSGeorge Zhang result = VMCI_ERROR_QUEUEPAIR_NOTATTACHED; 222206164d2bSGeorge Zhang goto out; 222306164d2bSGeorge Zhang } 222406164d2bSGeorge Zhang 222506164d2bSGeorge Zhang if (context_id == entry->create_id) { 222606164d2bSGeorge Zhang peer_id = entry->attach_id; 222706164d2bSGeorge Zhang entry->create_id = VMCI_INVALID_ID; 222806164d2bSGeorge Zhang } else { 222906164d2bSGeorge Zhang peer_id = entry->create_id; 223006164d2bSGeorge Zhang entry->attach_id = VMCI_INVALID_ID; 223106164d2bSGeorge Zhang } 223206164d2bSGeorge Zhang entry->qp.ref_count--; 223306164d2bSGeorge Zhang 223406164d2bSGeorge Zhang is_local = entry->qp.flags & VMCI_QPFLAG_LOCAL; 223506164d2bSGeorge Zhang 223606164d2bSGeorge Zhang if (context_id != VMCI_HOST_CONTEXT_ID) { 223706164d2bSGeorge Zhang bool headers_mapped; 223806164d2bSGeorge Zhang 223906164d2bSGeorge Zhang /* 224006164d2bSGeorge Zhang * Pre NOVMVM vmx'en may detach from a queue pair 224106164d2bSGeorge Zhang * before setting the page store, and in that case 224206164d2bSGeorge Zhang * there is no user memory to detach from. Also, more 224306164d2bSGeorge Zhang * recent VMX'en may detach from a queue pair in the 224406164d2bSGeorge Zhang * quiesced state. 224506164d2bSGeorge Zhang */ 224606164d2bSGeorge Zhang 224706164d2bSGeorge Zhang qp_acquire_queue_mutex(entry->produce_q); 224806164d2bSGeorge Zhang headers_mapped = entry->produce_q->q_header || 224906164d2bSGeorge Zhang entry->consume_q->q_header; 225006164d2bSGeorge Zhang if (QPBROKERSTATE_HAS_MEM(entry)) { 225106164d2bSGeorge Zhang result = 225206164d2bSGeorge Zhang qp_host_unmap_queues(INVALID_VMCI_GUEST_MEM_ID, 225306164d2bSGeorge Zhang entry->produce_q, 225406164d2bSGeorge Zhang entry->consume_q); 225506164d2bSGeorge Zhang if (result < VMCI_SUCCESS) 225606164d2bSGeorge Zhang pr_warn("Failed to unmap queue headers for queue pair (handle=0x%x:0x%x,result=%d)\n", 225706164d2bSGeorge Zhang handle.context, handle.resource, 225806164d2bSGeorge Zhang result); 225906164d2bSGeorge Zhang 226006164d2bSGeorge Zhang if (entry->vmci_page_files) 226106164d2bSGeorge Zhang qp_host_unregister_user_memory(entry->produce_q, 226206164d2bSGeorge Zhang entry-> 226306164d2bSGeorge Zhang consume_q); 226406164d2bSGeorge Zhang else 226506164d2bSGeorge Zhang qp_host_unregister_user_memory(entry->produce_q, 226606164d2bSGeorge Zhang entry-> 226706164d2bSGeorge Zhang consume_q); 226806164d2bSGeorge Zhang 226906164d2bSGeorge Zhang } 227006164d2bSGeorge Zhang 227106164d2bSGeorge Zhang if (!headers_mapped) 227206164d2bSGeorge Zhang qp_reset_saved_headers(entry); 227306164d2bSGeorge Zhang 227406164d2bSGeorge Zhang qp_release_queue_mutex(entry->produce_q); 227506164d2bSGeorge Zhang 227606164d2bSGeorge Zhang if (!headers_mapped && entry->wakeup_cb) 227706164d2bSGeorge Zhang entry->wakeup_cb(entry->client_data); 227806164d2bSGeorge Zhang 227906164d2bSGeorge Zhang } else { 228006164d2bSGeorge Zhang if (entry->wakeup_cb) { 228106164d2bSGeorge Zhang entry->wakeup_cb = NULL; 228206164d2bSGeorge Zhang entry->client_data = NULL; 228306164d2bSGeorge Zhang } 228406164d2bSGeorge Zhang } 228506164d2bSGeorge Zhang 228606164d2bSGeorge Zhang if (entry->qp.ref_count == 0) { 228706164d2bSGeorge Zhang qp_list_remove_entry(&qp_broker_list, &entry->qp); 228806164d2bSGeorge Zhang 228906164d2bSGeorge Zhang if (is_local) 229006164d2bSGeorge Zhang kfree(entry->local_mem); 229106164d2bSGeorge Zhang 229206164d2bSGeorge Zhang qp_cleanup_queue_mutex(entry->produce_q, entry->consume_q); 229306164d2bSGeorge Zhang qp_host_free_queue(entry->produce_q, entry->qp.produce_size); 229406164d2bSGeorge Zhang qp_host_free_queue(entry->consume_q, entry->qp.consume_size); 229506164d2bSGeorge Zhang /* Unlink from resource hash table and free callback */ 229606164d2bSGeorge Zhang vmci_resource_remove(&entry->resource); 229706164d2bSGeorge Zhang 229806164d2bSGeorge Zhang kfree(entry); 229906164d2bSGeorge Zhang 230006164d2bSGeorge Zhang vmci_ctx_qp_destroy(context, handle); 230106164d2bSGeorge Zhang } else { 230206164d2bSGeorge Zhang qp_notify_peer(false, handle, context_id, peer_id); 230306164d2bSGeorge Zhang if (context_id == VMCI_HOST_CONTEXT_ID && 230406164d2bSGeorge Zhang QPBROKERSTATE_HAS_MEM(entry)) { 230506164d2bSGeorge Zhang entry->state = VMCIQPB_SHUTDOWN_MEM; 230606164d2bSGeorge Zhang } else { 230706164d2bSGeorge Zhang entry->state = VMCIQPB_SHUTDOWN_NO_MEM; 230806164d2bSGeorge Zhang } 230906164d2bSGeorge Zhang 231006164d2bSGeorge Zhang if (!is_local) 231106164d2bSGeorge Zhang vmci_ctx_qp_destroy(context, handle); 231206164d2bSGeorge Zhang 231306164d2bSGeorge Zhang } 231406164d2bSGeorge Zhang result = VMCI_SUCCESS; 231506164d2bSGeorge Zhang out: 231606164d2bSGeorge Zhang mutex_unlock(&qp_broker_list.mutex); 231706164d2bSGeorge Zhang return result; 231806164d2bSGeorge Zhang } 231906164d2bSGeorge Zhang 232006164d2bSGeorge Zhang /* 232106164d2bSGeorge Zhang * Establishes the necessary mappings for a queue pair given a 232206164d2bSGeorge Zhang * reference to the queue pair guest memory. This is usually 232306164d2bSGeorge Zhang * called when a guest is unquiesced and the VMX is allowed to 232406164d2bSGeorge Zhang * map guest memory once again. 232506164d2bSGeorge Zhang */ 232606164d2bSGeorge Zhang int vmci_qp_broker_map(struct vmci_handle handle, 232706164d2bSGeorge Zhang struct vmci_ctx *context, 232806164d2bSGeorge Zhang u64 guest_mem) 232906164d2bSGeorge Zhang { 233006164d2bSGeorge Zhang struct qp_broker_entry *entry; 233106164d2bSGeorge Zhang const u32 context_id = vmci_ctx_get_id(context); 233206164d2bSGeorge Zhang bool is_local = false; 233306164d2bSGeorge Zhang int result; 233406164d2bSGeorge Zhang 233506164d2bSGeorge Zhang if (vmci_handle_is_invalid(handle) || !context || 233606164d2bSGeorge Zhang context_id == VMCI_INVALID_ID) 233706164d2bSGeorge Zhang return VMCI_ERROR_INVALID_ARGS; 233806164d2bSGeorge Zhang 233906164d2bSGeorge Zhang mutex_lock(&qp_broker_list.mutex); 234006164d2bSGeorge Zhang 234106164d2bSGeorge Zhang if (!vmci_ctx_qp_exists(context, handle)) { 234206164d2bSGeorge Zhang pr_devel("Context (ID=0x%x) not attached to queue pair (handle=0x%x:0x%x)\n", 234306164d2bSGeorge Zhang context_id, handle.context, handle.resource); 234406164d2bSGeorge Zhang result = VMCI_ERROR_NOT_FOUND; 234506164d2bSGeorge Zhang goto out; 234606164d2bSGeorge Zhang } 234706164d2bSGeorge Zhang 234806164d2bSGeorge Zhang entry = qp_broker_handle_to_entry(handle); 234906164d2bSGeorge Zhang if (!entry) { 235006164d2bSGeorge Zhang pr_devel("Context (ID=0x%x) reports being attached to queue pair (handle=0x%x:0x%x) that isn't present in broker\n", 235106164d2bSGeorge Zhang context_id, handle.context, handle.resource); 235206164d2bSGeorge Zhang result = VMCI_ERROR_NOT_FOUND; 235306164d2bSGeorge Zhang goto out; 235406164d2bSGeorge Zhang } 235506164d2bSGeorge Zhang 235606164d2bSGeorge Zhang if (context_id != entry->create_id && context_id != entry->attach_id) { 235706164d2bSGeorge Zhang result = VMCI_ERROR_QUEUEPAIR_NOTATTACHED; 235806164d2bSGeorge Zhang goto out; 235906164d2bSGeorge Zhang } 236006164d2bSGeorge Zhang 236106164d2bSGeorge Zhang is_local = entry->qp.flags & VMCI_QPFLAG_LOCAL; 236206164d2bSGeorge Zhang result = VMCI_SUCCESS; 236306164d2bSGeorge Zhang 236406164d2bSGeorge Zhang if (context_id != VMCI_HOST_CONTEXT_ID) { 236506164d2bSGeorge Zhang struct vmci_qp_page_store page_store; 236606164d2bSGeorge Zhang 236706164d2bSGeorge Zhang page_store.pages = guest_mem; 236806164d2bSGeorge Zhang page_store.len = QPE_NUM_PAGES(entry->qp); 236906164d2bSGeorge Zhang 237006164d2bSGeorge Zhang qp_acquire_queue_mutex(entry->produce_q); 237106164d2bSGeorge Zhang qp_reset_saved_headers(entry); 237206164d2bSGeorge Zhang result = 237306164d2bSGeorge Zhang qp_host_register_user_memory(&page_store, 237406164d2bSGeorge Zhang entry->produce_q, 237506164d2bSGeorge Zhang entry->consume_q); 237606164d2bSGeorge Zhang qp_release_queue_mutex(entry->produce_q); 237706164d2bSGeorge Zhang if (result == VMCI_SUCCESS) { 237806164d2bSGeorge Zhang /* Move state from *_NO_MEM to *_MEM */ 237906164d2bSGeorge Zhang 238006164d2bSGeorge Zhang entry->state++; 238106164d2bSGeorge Zhang 238206164d2bSGeorge Zhang if (entry->wakeup_cb) 238306164d2bSGeorge Zhang entry->wakeup_cb(entry->client_data); 238406164d2bSGeorge Zhang } 238506164d2bSGeorge Zhang } 238606164d2bSGeorge Zhang 238706164d2bSGeorge Zhang out: 238806164d2bSGeorge Zhang mutex_unlock(&qp_broker_list.mutex); 238906164d2bSGeorge Zhang return result; 239006164d2bSGeorge Zhang } 239106164d2bSGeorge Zhang 239206164d2bSGeorge Zhang /* 239306164d2bSGeorge Zhang * Saves a snapshot of the queue headers for the given QP broker 239406164d2bSGeorge Zhang * entry. Should be used when guest memory is unmapped. 239506164d2bSGeorge Zhang * Results: 239606164d2bSGeorge Zhang * VMCI_SUCCESS on success, appropriate error code if guest memory 239706164d2bSGeorge Zhang * can't be accessed.. 239806164d2bSGeorge Zhang */ 239906164d2bSGeorge Zhang static int qp_save_headers(struct qp_broker_entry *entry) 240006164d2bSGeorge Zhang { 240106164d2bSGeorge Zhang int result; 240206164d2bSGeorge Zhang 240306164d2bSGeorge Zhang if (entry->produce_q->saved_header != NULL && 240406164d2bSGeorge Zhang entry->consume_q->saved_header != NULL) { 240506164d2bSGeorge Zhang /* 240606164d2bSGeorge Zhang * If the headers have already been saved, we don't need to do 240706164d2bSGeorge Zhang * it again, and we don't want to map in the headers 240806164d2bSGeorge Zhang * unnecessarily. 240906164d2bSGeorge Zhang */ 241006164d2bSGeorge Zhang 241106164d2bSGeorge Zhang return VMCI_SUCCESS; 241206164d2bSGeorge Zhang } 241306164d2bSGeorge Zhang 241406164d2bSGeorge Zhang if (NULL == entry->produce_q->q_header || 241506164d2bSGeorge Zhang NULL == entry->consume_q->q_header) { 241606164d2bSGeorge Zhang result = qp_host_map_queues(entry->produce_q, entry->consume_q); 241706164d2bSGeorge Zhang if (result < VMCI_SUCCESS) 241806164d2bSGeorge Zhang return result; 241906164d2bSGeorge Zhang } 242006164d2bSGeorge Zhang 242106164d2bSGeorge Zhang memcpy(&entry->saved_produce_q, entry->produce_q->q_header, 242206164d2bSGeorge Zhang sizeof(entry->saved_produce_q)); 242306164d2bSGeorge Zhang entry->produce_q->saved_header = &entry->saved_produce_q; 242406164d2bSGeorge Zhang memcpy(&entry->saved_consume_q, entry->consume_q->q_header, 242506164d2bSGeorge Zhang sizeof(entry->saved_consume_q)); 242606164d2bSGeorge Zhang entry->consume_q->saved_header = &entry->saved_consume_q; 242706164d2bSGeorge Zhang 242806164d2bSGeorge Zhang return VMCI_SUCCESS; 242906164d2bSGeorge Zhang } 243006164d2bSGeorge Zhang 243106164d2bSGeorge Zhang /* 243206164d2bSGeorge Zhang * Removes all references to the guest memory of a given queue pair, and 243306164d2bSGeorge Zhang * will move the queue pair from state *_MEM to *_NO_MEM. It is usually 243406164d2bSGeorge Zhang * called when a VM is being quiesced where access to guest memory should 243506164d2bSGeorge Zhang * avoided. 243606164d2bSGeorge Zhang */ 243706164d2bSGeorge Zhang int vmci_qp_broker_unmap(struct vmci_handle handle, 243806164d2bSGeorge Zhang struct vmci_ctx *context, 243906164d2bSGeorge Zhang u32 gid) 244006164d2bSGeorge Zhang { 244106164d2bSGeorge Zhang struct qp_broker_entry *entry; 244206164d2bSGeorge Zhang const u32 context_id = vmci_ctx_get_id(context); 244306164d2bSGeorge Zhang bool is_local = false; 244406164d2bSGeorge Zhang int result; 244506164d2bSGeorge Zhang 244606164d2bSGeorge Zhang if (vmci_handle_is_invalid(handle) || !context || 244706164d2bSGeorge Zhang context_id == VMCI_INVALID_ID) 244806164d2bSGeorge Zhang return VMCI_ERROR_INVALID_ARGS; 244906164d2bSGeorge Zhang 245006164d2bSGeorge Zhang mutex_lock(&qp_broker_list.mutex); 245106164d2bSGeorge Zhang 245206164d2bSGeorge Zhang if (!vmci_ctx_qp_exists(context, handle)) { 245306164d2bSGeorge Zhang pr_devel("Context (ID=0x%x) not attached to queue pair (handle=0x%x:0x%x)\n", 245406164d2bSGeorge Zhang context_id, handle.context, handle.resource); 245506164d2bSGeorge Zhang result = VMCI_ERROR_NOT_FOUND; 245606164d2bSGeorge Zhang goto out; 245706164d2bSGeorge Zhang } 245806164d2bSGeorge Zhang 245906164d2bSGeorge Zhang entry = qp_broker_handle_to_entry(handle); 246006164d2bSGeorge Zhang if (!entry) { 246106164d2bSGeorge Zhang pr_devel("Context (ID=0x%x) reports being attached to queue pair (handle=0x%x:0x%x) that isn't present in broker\n", 246206164d2bSGeorge Zhang context_id, handle.context, handle.resource); 246306164d2bSGeorge Zhang result = VMCI_ERROR_NOT_FOUND; 246406164d2bSGeorge Zhang goto out; 246506164d2bSGeorge Zhang } 246606164d2bSGeorge Zhang 246706164d2bSGeorge Zhang if (context_id != entry->create_id && context_id != entry->attach_id) { 246806164d2bSGeorge Zhang result = VMCI_ERROR_QUEUEPAIR_NOTATTACHED; 246906164d2bSGeorge Zhang goto out; 247006164d2bSGeorge Zhang } 247106164d2bSGeorge Zhang 247206164d2bSGeorge Zhang is_local = entry->qp.flags & VMCI_QPFLAG_LOCAL; 247306164d2bSGeorge Zhang 247406164d2bSGeorge Zhang if (context_id != VMCI_HOST_CONTEXT_ID) { 247506164d2bSGeorge Zhang qp_acquire_queue_mutex(entry->produce_q); 247606164d2bSGeorge Zhang result = qp_save_headers(entry); 247706164d2bSGeorge Zhang if (result < VMCI_SUCCESS) 247806164d2bSGeorge Zhang pr_warn("Failed to save queue headers for queue pair (handle=0x%x:0x%x,result=%d)\n", 247906164d2bSGeorge Zhang handle.context, handle.resource, result); 248006164d2bSGeorge Zhang 248106164d2bSGeorge Zhang qp_host_unmap_queues(gid, entry->produce_q, entry->consume_q); 248206164d2bSGeorge Zhang 248306164d2bSGeorge Zhang /* 248406164d2bSGeorge Zhang * On hosted, when we unmap queue pairs, the VMX will also 248506164d2bSGeorge Zhang * unmap the guest memory, so we invalidate the previously 248606164d2bSGeorge Zhang * registered memory. If the queue pair is mapped again at a 248706164d2bSGeorge Zhang * later point in time, we will need to reregister the user 248806164d2bSGeorge Zhang * memory with a possibly new user VA. 248906164d2bSGeorge Zhang */ 249006164d2bSGeorge Zhang qp_host_unregister_user_memory(entry->produce_q, 249106164d2bSGeorge Zhang entry->consume_q); 249206164d2bSGeorge Zhang 249306164d2bSGeorge Zhang /* 249406164d2bSGeorge Zhang * Move state from *_MEM to *_NO_MEM. 249506164d2bSGeorge Zhang */ 249606164d2bSGeorge Zhang entry->state--; 249706164d2bSGeorge Zhang 249806164d2bSGeorge Zhang qp_release_queue_mutex(entry->produce_q); 249906164d2bSGeorge Zhang } 250006164d2bSGeorge Zhang 250106164d2bSGeorge Zhang result = VMCI_SUCCESS; 250206164d2bSGeorge Zhang 250306164d2bSGeorge Zhang out: 250406164d2bSGeorge Zhang mutex_unlock(&qp_broker_list.mutex); 250506164d2bSGeorge Zhang return result; 250606164d2bSGeorge Zhang } 250706164d2bSGeorge Zhang 250806164d2bSGeorge Zhang /* 250906164d2bSGeorge Zhang * Destroys all guest queue pair endpoints. If active guest queue 251006164d2bSGeorge Zhang * pairs still exist, hypercalls to attempt detach from these 251106164d2bSGeorge Zhang * queue pairs will be made. Any failure to detach is silently 251206164d2bSGeorge Zhang * ignored. 251306164d2bSGeorge Zhang */ 251406164d2bSGeorge Zhang void vmci_qp_guest_endpoints_exit(void) 251506164d2bSGeorge Zhang { 251606164d2bSGeorge Zhang struct qp_entry *entry; 251706164d2bSGeorge Zhang struct qp_guest_endpoint *ep; 251806164d2bSGeorge Zhang 251906164d2bSGeorge Zhang mutex_lock(&qp_guest_endpoints.mutex); 252006164d2bSGeorge Zhang 252106164d2bSGeorge Zhang while ((entry = qp_list_get_head(&qp_guest_endpoints))) { 252206164d2bSGeorge Zhang ep = (struct qp_guest_endpoint *)entry; 252306164d2bSGeorge Zhang 252406164d2bSGeorge Zhang /* Don't make a hypercall for local queue_pairs. */ 252506164d2bSGeorge Zhang if (!(entry->flags & VMCI_QPFLAG_LOCAL)) 252606164d2bSGeorge Zhang qp_detatch_hypercall(entry->handle); 252706164d2bSGeorge Zhang 252806164d2bSGeorge Zhang /* We cannot fail the exit, so let's reset ref_count. */ 252906164d2bSGeorge Zhang entry->ref_count = 0; 253006164d2bSGeorge Zhang qp_list_remove_entry(&qp_guest_endpoints, entry); 253106164d2bSGeorge Zhang 253206164d2bSGeorge Zhang qp_guest_endpoint_destroy(ep); 253306164d2bSGeorge Zhang } 253406164d2bSGeorge Zhang 253506164d2bSGeorge Zhang mutex_unlock(&qp_guest_endpoints.mutex); 253606164d2bSGeorge Zhang } 253706164d2bSGeorge Zhang 253806164d2bSGeorge Zhang /* 253906164d2bSGeorge Zhang * Helper routine that will lock the queue pair before subsequent 254006164d2bSGeorge Zhang * operations. 254106164d2bSGeorge Zhang * Note: Non-blocking on the host side is currently only implemented in ESX. 254206164d2bSGeorge Zhang * Since non-blocking isn't yet implemented on the host personality we 254306164d2bSGeorge Zhang * have no reason to acquire a spin lock. So to avoid the use of an 254406164d2bSGeorge Zhang * unnecessary lock only acquire the mutex if we can block. 254506164d2bSGeorge Zhang * Note: It is assumed that QPFLAG_PINNED implies QPFLAG_NONBLOCK. Therefore 254606164d2bSGeorge Zhang * we can use the same locking function for access to both the queue 254706164d2bSGeorge Zhang * and the queue headers as it is the same logic. Assert this behvior. 254806164d2bSGeorge Zhang */ 254906164d2bSGeorge Zhang static void qp_lock(const struct vmci_qp *qpair) 255006164d2bSGeorge Zhang { 255106164d2bSGeorge Zhang if (vmci_can_block(qpair->flags)) 255206164d2bSGeorge Zhang qp_acquire_queue_mutex(qpair->produce_q); 255306164d2bSGeorge Zhang } 255406164d2bSGeorge Zhang 255506164d2bSGeorge Zhang /* 255606164d2bSGeorge Zhang * Helper routine that unlocks the queue pair after calling 255706164d2bSGeorge Zhang * qp_lock. Respects non-blocking and pinning flags. 255806164d2bSGeorge Zhang */ 255906164d2bSGeorge Zhang static void qp_unlock(const struct vmci_qp *qpair) 256006164d2bSGeorge Zhang { 256106164d2bSGeorge Zhang if (vmci_can_block(qpair->flags)) 256206164d2bSGeorge Zhang qp_release_queue_mutex(qpair->produce_q); 256306164d2bSGeorge Zhang } 256406164d2bSGeorge Zhang 256506164d2bSGeorge Zhang /* 256606164d2bSGeorge Zhang * The queue headers may not be mapped at all times. If a queue is 256706164d2bSGeorge Zhang * currently not mapped, it will be attempted to do so. 256806164d2bSGeorge Zhang */ 256906164d2bSGeorge Zhang static int qp_map_queue_headers(struct vmci_queue *produce_q, 257006164d2bSGeorge Zhang struct vmci_queue *consume_q, 257106164d2bSGeorge Zhang bool can_block) 257206164d2bSGeorge Zhang { 257306164d2bSGeorge Zhang int result; 257406164d2bSGeorge Zhang 257506164d2bSGeorge Zhang if (NULL == produce_q->q_header || NULL == consume_q->q_header) { 257606164d2bSGeorge Zhang if (can_block) 257706164d2bSGeorge Zhang result = qp_host_map_queues(produce_q, consume_q); 257806164d2bSGeorge Zhang else 257906164d2bSGeorge Zhang result = VMCI_ERROR_QUEUEPAIR_NOT_READY; 258006164d2bSGeorge Zhang 258106164d2bSGeorge Zhang if (result < VMCI_SUCCESS) 258206164d2bSGeorge Zhang return (produce_q->saved_header && 258306164d2bSGeorge Zhang consume_q->saved_header) ? 258406164d2bSGeorge Zhang VMCI_ERROR_QUEUEPAIR_NOT_READY : 258506164d2bSGeorge Zhang VMCI_ERROR_QUEUEPAIR_NOTATTACHED; 258606164d2bSGeorge Zhang } 258706164d2bSGeorge Zhang 258806164d2bSGeorge Zhang return VMCI_SUCCESS; 258906164d2bSGeorge Zhang } 259006164d2bSGeorge Zhang 259106164d2bSGeorge Zhang /* 259206164d2bSGeorge Zhang * Helper routine that will retrieve the produce and consume 259306164d2bSGeorge Zhang * headers of a given queue pair. If the guest memory of the 259406164d2bSGeorge Zhang * queue pair is currently not available, the saved queue headers 259506164d2bSGeorge Zhang * will be returned, if these are available. 259606164d2bSGeorge Zhang */ 259706164d2bSGeorge Zhang static int qp_get_queue_headers(const struct vmci_qp *qpair, 259806164d2bSGeorge Zhang struct vmci_queue_header **produce_q_header, 259906164d2bSGeorge Zhang struct vmci_queue_header **consume_q_header) 260006164d2bSGeorge Zhang { 260106164d2bSGeorge Zhang int result; 260206164d2bSGeorge Zhang 260306164d2bSGeorge Zhang result = qp_map_queue_headers(qpair->produce_q, qpair->consume_q, 260406164d2bSGeorge Zhang vmci_can_block(qpair->flags)); 260506164d2bSGeorge Zhang if (result == VMCI_SUCCESS) { 260606164d2bSGeorge Zhang *produce_q_header = qpair->produce_q->q_header; 260706164d2bSGeorge Zhang *consume_q_header = qpair->consume_q->q_header; 260806164d2bSGeorge Zhang } else if (qpair->produce_q->saved_header && 260906164d2bSGeorge Zhang qpair->consume_q->saved_header) { 261006164d2bSGeorge Zhang *produce_q_header = qpair->produce_q->saved_header; 261106164d2bSGeorge Zhang *consume_q_header = qpair->consume_q->saved_header; 261206164d2bSGeorge Zhang result = VMCI_SUCCESS; 261306164d2bSGeorge Zhang } 261406164d2bSGeorge Zhang 261506164d2bSGeorge Zhang return result; 261606164d2bSGeorge Zhang } 261706164d2bSGeorge Zhang 261806164d2bSGeorge Zhang /* 261906164d2bSGeorge Zhang * Callback from VMCI queue pair broker indicating that a queue 262006164d2bSGeorge Zhang * pair that was previously not ready, now either is ready or 262106164d2bSGeorge Zhang * gone forever. 262206164d2bSGeorge Zhang */ 262306164d2bSGeorge Zhang static int qp_wakeup_cb(void *client_data) 262406164d2bSGeorge Zhang { 262506164d2bSGeorge Zhang struct vmci_qp *qpair = (struct vmci_qp *)client_data; 262606164d2bSGeorge Zhang 262706164d2bSGeorge Zhang qp_lock(qpair); 262806164d2bSGeorge Zhang while (qpair->blocked > 0) { 262906164d2bSGeorge Zhang qpair->blocked--; 263006164d2bSGeorge Zhang qpair->generation++; 263106164d2bSGeorge Zhang wake_up(&qpair->event); 263206164d2bSGeorge Zhang } 263306164d2bSGeorge Zhang qp_unlock(qpair); 263406164d2bSGeorge Zhang 263506164d2bSGeorge Zhang return VMCI_SUCCESS; 263606164d2bSGeorge Zhang } 263706164d2bSGeorge Zhang 263806164d2bSGeorge Zhang /* 263906164d2bSGeorge Zhang * Makes the calling thread wait for the queue pair to become 264006164d2bSGeorge Zhang * ready for host side access. Returns true when thread is 264106164d2bSGeorge Zhang * woken up after queue pair state change, false otherwise. 264206164d2bSGeorge Zhang */ 264306164d2bSGeorge Zhang static bool qp_wait_for_ready_queue(struct vmci_qp *qpair) 264406164d2bSGeorge Zhang { 264506164d2bSGeorge Zhang unsigned int generation; 264606164d2bSGeorge Zhang 264706164d2bSGeorge Zhang if (qpair->flags & VMCI_QPFLAG_NONBLOCK) 264806164d2bSGeorge Zhang return false; 264906164d2bSGeorge Zhang 265006164d2bSGeorge Zhang qpair->blocked++; 265106164d2bSGeorge Zhang generation = qpair->generation; 265206164d2bSGeorge Zhang qp_unlock(qpair); 265306164d2bSGeorge Zhang wait_event(qpair->event, generation != qpair->generation); 265406164d2bSGeorge Zhang qp_lock(qpair); 265506164d2bSGeorge Zhang 265606164d2bSGeorge Zhang return true; 265706164d2bSGeorge Zhang } 265806164d2bSGeorge Zhang 265906164d2bSGeorge Zhang /* 266006164d2bSGeorge Zhang * Enqueues a given buffer to the produce queue using the provided 266106164d2bSGeorge Zhang * function. As many bytes as possible (space available in the queue) 266206164d2bSGeorge Zhang * are enqueued. Assumes the queue->mutex has been acquired. Returns 266306164d2bSGeorge Zhang * VMCI_ERROR_QUEUEPAIR_NOSPACE if no space was available to enqueue 266406164d2bSGeorge Zhang * data, VMCI_ERROR_INVALID_SIZE, if any queue pointer is outside the 266506164d2bSGeorge Zhang * queue (as defined by the queue size), VMCI_ERROR_INVALID_ARGS, if 266606164d2bSGeorge Zhang * an error occured when accessing the buffer, 266706164d2bSGeorge Zhang * VMCI_ERROR_QUEUEPAIR_NOTATTACHED, if the queue pair pages aren't 266806164d2bSGeorge Zhang * available. Otherwise, the number of bytes written to the queue is 266906164d2bSGeorge Zhang * returned. Updates the tail pointer of the produce queue. 267006164d2bSGeorge Zhang */ 267106164d2bSGeorge Zhang static ssize_t qp_enqueue_locked(struct vmci_queue *produce_q, 267206164d2bSGeorge Zhang struct vmci_queue *consume_q, 267306164d2bSGeorge Zhang const u64 produce_q_size, 267406164d2bSGeorge Zhang const void *buf, 267506164d2bSGeorge Zhang size_t buf_size, 267606164d2bSGeorge Zhang vmci_memcpy_to_queue_func memcpy_to_queue, 267706164d2bSGeorge Zhang bool can_block) 267806164d2bSGeorge Zhang { 267906164d2bSGeorge Zhang s64 free_space; 268006164d2bSGeorge Zhang u64 tail; 268106164d2bSGeorge Zhang size_t written; 268206164d2bSGeorge Zhang ssize_t result; 268306164d2bSGeorge Zhang 268406164d2bSGeorge Zhang result = qp_map_queue_headers(produce_q, consume_q, can_block); 268506164d2bSGeorge Zhang if (unlikely(result != VMCI_SUCCESS)) 268606164d2bSGeorge Zhang return result; 268706164d2bSGeorge Zhang 268806164d2bSGeorge Zhang free_space = vmci_q_header_free_space(produce_q->q_header, 268906164d2bSGeorge Zhang consume_q->q_header, 269006164d2bSGeorge Zhang produce_q_size); 269106164d2bSGeorge Zhang if (free_space == 0) 269206164d2bSGeorge Zhang return VMCI_ERROR_QUEUEPAIR_NOSPACE; 269306164d2bSGeorge Zhang 269406164d2bSGeorge Zhang if (free_space < VMCI_SUCCESS) 269506164d2bSGeorge Zhang return (ssize_t) free_space; 269606164d2bSGeorge Zhang 269706164d2bSGeorge Zhang written = (size_t) (free_space > buf_size ? buf_size : free_space); 269806164d2bSGeorge Zhang tail = vmci_q_header_producer_tail(produce_q->q_header); 269906164d2bSGeorge Zhang if (likely(tail + written < produce_q_size)) { 270006164d2bSGeorge Zhang result = memcpy_to_queue(produce_q, tail, buf, 0, written); 270106164d2bSGeorge Zhang } else { 270206164d2bSGeorge Zhang /* Tail pointer wraps around. */ 270306164d2bSGeorge Zhang 270406164d2bSGeorge Zhang const size_t tmp = (size_t) (produce_q_size - tail); 270506164d2bSGeorge Zhang 270606164d2bSGeorge Zhang result = memcpy_to_queue(produce_q, tail, buf, 0, tmp); 270706164d2bSGeorge Zhang if (result >= VMCI_SUCCESS) 270806164d2bSGeorge Zhang result = memcpy_to_queue(produce_q, 0, buf, tmp, 270906164d2bSGeorge Zhang written - tmp); 271006164d2bSGeorge Zhang } 271106164d2bSGeorge Zhang 271206164d2bSGeorge Zhang if (result < VMCI_SUCCESS) 271306164d2bSGeorge Zhang return result; 271406164d2bSGeorge Zhang 271506164d2bSGeorge Zhang vmci_q_header_add_producer_tail(produce_q->q_header, written, 271606164d2bSGeorge Zhang produce_q_size); 271706164d2bSGeorge Zhang return written; 271806164d2bSGeorge Zhang } 271906164d2bSGeorge Zhang 272006164d2bSGeorge Zhang /* 272106164d2bSGeorge Zhang * Dequeues data (if available) from the given consume queue. Writes data 272206164d2bSGeorge Zhang * to the user provided buffer using the provided function. 272306164d2bSGeorge Zhang * Assumes the queue->mutex has been acquired. 272406164d2bSGeorge Zhang * Results: 272506164d2bSGeorge Zhang * VMCI_ERROR_QUEUEPAIR_NODATA if no data was available to dequeue. 272606164d2bSGeorge Zhang * VMCI_ERROR_INVALID_SIZE, if any queue pointer is outside the queue 272706164d2bSGeorge Zhang * (as defined by the queue size). 272806164d2bSGeorge Zhang * VMCI_ERROR_INVALID_ARGS, if an error occured when accessing the buffer. 272906164d2bSGeorge Zhang * Otherwise the number of bytes dequeued is returned. 273006164d2bSGeorge Zhang * Side effects: 273106164d2bSGeorge Zhang * Updates the head pointer of the consume queue. 273206164d2bSGeorge Zhang */ 273306164d2bSGeorge Zhang static ssize_t qp_dequeue_locked(struct vmci_queue *produce_q, 273406164d2bSGeorge Zhang struct vmci_queue *consume_q, 273506164d2bSGeorge Zhang const u64 consume_q_size, 273606164d2bSGeorge Zhang void *buf, 273706164d2bSGeorge Zhang size_t buf_size, 273806164d2bSGeorge Zhang vmci_memcpy_from_queue_func memcpy_from_queue, 273906164d2bSGeorge Zhang bool update_consumer, 274006164d2bSGeorge Zhang bool can_block) 274106164d2bSGeorge Zhang { 274206164d2bSGeorge Zhang s64 buf_ready; 274306164d2bSGeorge Zhang u64 head; 274406164d2bSGeorge Zhang size_t read; 274506164d2bSGeorge Zhang ssize_t result; 274606164d2bSGeorge Zhang 274706164d2bSGeorge Zhang result = qp_map_queue_headers(produce_q, consume_q, can_block); 274806164d2bSGeorge Zhang if (unlikely(result != VMCI_SUCCESS)) 274906164d2bSGeorge Zhang return result; 275006164d2bSGeorge Zhang 275106164d2bSGeorge Zhang buf_ready = vmci_q_header_buf_ready(consume_q->q_header, 275206164d2bSGeorge Zhang produce_q->q_header, 275306164d2bSGeorge Zhang consume_q_size); 275406164d2bSGeorge Zhang if (buf_ready == 0) 275506164d2bSGeorge Zhang return VMCI_ERROR_QUEUEPAIR_NODATA; 275606164d2bSGeorge Zhang 275706164d2bSGeorge Zhang if (buf_ready < VMCI_SUCCESS) 275806164d2bSGeorge Zhang return (ssize_t) buf_ready; 275906164d2bSGeorge Zhang 276006164d2bSGeorge Zhang read = (size_t) (buf_ready > buf_size ? buf_size : buf_ready); 276106164d2bSGeorge Zhang head = vmci_q_header_consumer_head(produce_q->q_header); 276206164d2bSGeorge Zhang if (likely(head + read < consume_q_size)) { 276306164d2bSGeorge Zhang result = memcpy_from_queue(buf, 0, consume_q, head, read); 276406164d2bSGeorge Zhang } else { 276506164d2bSGeorge Zhang /* Head pointer wraps around. */ 276606164d2bSGeorge Zhang 276706164d2bSGeorge Zhang const size_t tmp = (size_t) (consume_q_size - head); 276806164d2bSGeorge Zhang 276906164d2bSGeorge Zhang result = memcpy_from_queue(buf, 0, consume_q, head, tmp); 277006164d2bSGeorge Zhang if (result >= VMCI_SUCCESS) 277106164d2bSGeorge Zhang result = memcpy_from_queue(buf, tmp, consume_q, 0, 277206164d2bSGeorge Zhang read - tmp); 277306164d2bSGeorge Zhang 277406164d2bSGeorge Zhang } 277506164d2bSGeorge Zhang 277606164d2bSGeorge Zhang if (result < VMCI_SUCCESS) 277706164d2bSGeorge Zhang return result; 277806164d2bSGeorge Zhang 277906164d2bSGeorge Zhang if (update_consumer) 278006164d2bSGeorge Zhang vmci_q_header_add_consumer_head(produce_q->q_header, 278106164d2bSGeorge Zhang read, consume_q_size); 278206164d2bSGeorge Zhang 278306164d2bSGeorge Zhang return read; 278406164d2bSGeorge Zhang } 278506164d2bSGeorge Zhang 278606164d2bSGeorge Zhang /* 278706164d2bSGeorge Zhang * vmci_qpair_alloc() - Allocates a queue pair. 278806164d2bSGeorge Zhang * @qpair: Pointer for the new vmci_qp struct. 278906164d2bSGeorge Zhang * @handle: Handle to track the resource. 279006164d2bSGeorge Zhang * @produce_qsize: Desired size of the producer queue. 279106164d2bSGeorge Zhang * @consume_qsize: Desired size of the consumer queue. 279206164d2bSGeorge Zhang * @peer: ContextID of the peer. 279306164d2bSGeorge Zhang * @flags: VMCI flags. 279406164d2bSGeorge Zhang * @priv_flags: VMCI priviledge flags. 279506164d2bSGeorge Zhang * 279606164d2bSGeorge Zhang * This is the client interface for allocating the memory for a 279706164d2bSGeorge Zhang * vmci_qp structure and then attaching to the underlying 279806164d2bSGeorge Zhang * queue. If an error occurs allocating the memory for the 279906164d2bSGeorge Zhang * vmci_qp structure no attempt is made to attach. If an 280006164d2bSGeorge Zhang * error occurs attaching, then the structure is freed. 280106164d2bSGeorge Zhang */ 280206164d2bSGeorge Zhang int vmci_qpair_alloc(struct vmci_qp **qpair, 280306164d2bSGeorge Zhang struct vmci_handle *handle, 280406164d2bSGeorge Zhang u64 produce_qsize, 280506164d2bSGeorge Zhang u64 consume_qsize, 280606164d2bSGeorge Zhang u32 peer, 280706164d2bSGeorge Zhang u32 flags, 280806164d2bSGeorge Zhang u32 priv_flags) 280906164d2bSGeorge Zhang { 281006164d2bSGeorge Zhang struct vmci_qp *my_qpair; 281106164d2bSGeorge Zhang int retval; 281206164d2bSGeorge Zhang struct vmci_handle src = VMCI_INVALID_HANDLE; 281306164d2bSGeorge Zhang struct vmci_handle dst = vmci_make_handle(peer, VMCI_INVALID_ID); 281406164d2bSGeorge Zhang enum vmci_route route; 281506164d2bSGeorge Zhang vmci_event_release_cb wakeup_cb; 281606164d2bSGeorge Zhang void *client_data; 281706164d2bSGeorge Zhang 281806164d2bSGeorge Zhang /* 281906164d2bSGeorge Zhang * Restrict the size of a queuepair. The device already 282006164d2bSGeorge Zhang * enforces a limit on the total amount of memory that can be 282106164d2bSGeorge Zhang * allocated to queuepairs for a guest. However, we try to 282206164d2bSGeorge Zhang * allocate this memory before we make the queuepair 282306164d2bSGeorge Zhang * allocation hypercall. On Linux, we allocate each page 282406164d2bSGeorge Zhang * separately, which means rather than fail, the guest will 282506164d2bSGeorge Zhang * thrash while it tries to allocate, and will become 282606164d2bSGeorge Zhang * increasingly unresponsive to the point where it appears to 282706164d2bSGeorge Zhang * be hung. So we place a limit on the size of an individual 282806164d2bSGeorge Zhang * queuepair here, and leave the device to enforce the 282906164d2bSGeorge Zhang * restriction on total queuepair memory. (Note that this 283006164d2bSGeorge Zhang * doesn't prevent all cases; a user with only this much 283106164d2bSGeorge Zhang * physical memory could still get into trouble.) The error 283206164d2bSGeorge Zhang * used by the device is NO_RESOURCES, so use that here too. 283306164d2bSGeorge Zhang */ 283406164d2bSGeorge Zhang 283506164d2bSGeorge Zhang if (produce_qsize + consume_qsize < max(produce_qsize, consume_qsize) || 283606164d2bSGeorge Zhang produce_qsize + consume_qsize > VMCI_MAX_GUEST_QP_MEMORY) 283706164d2bSGeorge Zhang return VMCI_ERROR_NO_RESOURCES; 283806164d2bSGeorge Zhang 283906164d2bSGeorge Zhang retval = vmci_route(&src, &dst, false, &route); 284006164d2bSGeorge Zhang if (retval < VMCI_SUCCESS) 284106164d2bSGeorge Zhang route = vmci_guest_code_active() ? 284206164d2bSGeorge Zhang VMCI_ROUTE_AS_GUEST : VMCI_ROUTE_AS_HOST; 284306164d2bSGeorge Zhang 284406164d2bSGeorge Zhang /* If NONBLOCK or PINNED is set, we better be the guest personality. */ 284506164d2bSGeorge Zhang if ((!vmci_can_block(flags) || vmci_qp_pinned(flags)) && 284606164d2bSGeorge Zhang VMCI_ROUTE_AS_GUEST != route) { 284706164d2bSGeorge Zhang pr_devel("Not guest personality w/ NONBLOCK OR PINNED set"); 284806164d2bSGeorge Zhang return VMCI_ERROR_INVALID_ARGS; 284906164d2bSGeorge Zhang } 285006164d2bSGeorge Zhang 285106164d2bSGeorge Zhang /* 285206164d2bSGeorge Zhang * Limit the size of pinned QPs and check sanity. 285306164d2bSGeorge Zhang * 285406164d2bSGeorge Zhang * Pinned pages implies non-blocking mode. Mutexes aren't acquired 285506164d2bSGeorge Zhang * when the NONBLOCK flag is set in qpair code; and also should not be 285606164d2bSGeorge Zhang * acquired when the PINNED flagged is set. Since pinning pages 285706164d2bSGeorge Zhang * implies we want speed, it makes no sense not to have NONBLOCK 285806164d2bSGeorge Zhang * set if PINNED is set. Hence enforce this implication. 285906164d2bSGeorge Zhang */ 286006164d2bSGeorge Zhang if (vmci_qp_pinned(flags)) { 286106164d2bSGeorge Zhang if (vmci_can_block(flags)) { 286206164d2bSGeorge Zhang pr_err("Attempted to enable pinning w/o non-blocking"); 286306164d2bSGeorge Zhang return VMCI_ERROR_INVALID_ARGS; 286406164d2bSGeorge Zhang } 286506164d2bSGeorge Zhang 286606164d2bSGeorge Zhang if (produce_qsize + consume_qsize > VMCI_MAX_PINNED_QP_MEMORY) 286706164d2bSGeorge Zhang return VMCI_ERROR_NO_RESOURCES; 286806164d2bSGeorge Zhang } 286906164d2bSGeorge Zhang 287006164d2bSGeorge Zhang my_qpair = kzalloc(sizeof(*my_qpair), GFP_KERNEL); 287106164d2bSGeorge Zhang if (!my_qpair) 287206164d2bSGeorge Zhang return VMCI_ERROR_NO_MEM; 287306164d2bSGeorge Zhang 287406164d2bSGeorge Zhang my_qpair->produce_q_size = produce_qsize; 287506164d2bSGeorge Zhang my_qpair->consume_q_size = consume_qsize; 287606164d2bSGeorge Zhang my_qpair->peer = peer; 287706164d2bSGeorge Zhang my_qpair->flags = flags; 287806164d2bSGeorge Zhang my_qpair->priv_flags = priv_flags; 287906164d2bSGeorge Zhang 288006164d2bSGeorge Zhang wakeup_cb = NULL; 288106164d2bSGeorge Zhang client_data = NULL; 288206164d2bSGeorge Zhang 288306164d2bSGeorge Zhang if (VMCI_ROUTE_AS_HOST == route) { 288406164d2bSGeorge Zhang my_qpair->guest_endpoint = false; 288506164d2bSGeorge Zhang if (!(flags & VMCI_QPFLAG_LOCAL)) { 288606164d2bSGeorge Zhang my_qpair->blocked = 0; 288706164d2bSGeorge Zhang my_qpair->generation = 0; 288806164d2bSGeorge Zhang init_waitqueue_head(&my_qpair->event); 288906164d2bSGeorge Zhang wakeup_cb = qp_wakeup_cb; 289006164d2bSGeorge Zhang client_data = (void *)my_qpair; 289106164d2bSGeorge Zhang } 289206164d2bSGeorge Zhang } else { 289306164d2bSGeorge Zhang my_qpair->guest_endpoint = true; 289406164d2bSGeorge Zhang } 289506164d2bSGeorge Zhang 289606164d2bSGeorge Zhang retval = vmci_qp_alloc(handle, 289706164d2bSGeorge Zhang &my_qpair->produce_q, 289806164d2bSGeorge Zhang my_qpair->produce_q_size, 289906164d2bSGeorge Zhang &my_qpair->consume_q, 290006164d2bSGeorge Zhang my_qpair->consume_q_size, 290106164d2bSGeorge Zhang my_qpair->peer, 290206164d2bSGeorge Zhang my_qpair->flags, 290306164d2bSGeorge Zhang my_qpair->priv_flags, 290406164d2bSGeorge Zhang my_qpair->guest_endpoint, 290506164d2bSGeorge Zhang wakeup_cb, client_data); 290606164d2bSGeorge Zhang 290706164d2bSGeorge Zhang if (retval < VMCI_SUCCESS) { 290806164d2bSGeorge Zhang kfree(my_qpair); 290906164d2bSGeorge Zhang return retval; 291006164d2bSGeorge Zhang } 291106164d2bSGeorge Zhang 291206164d2bSGeorge Zhang *qpair = my_qpair; 291306164d2bSGeorge Zhang my_qpair->handle = *handle; 291406164d2bSGeorge Zhang 291506164d2bSGeorge Zhang return retval; 291606164d2bSGeorge Zhang } 291706164d2bSGeorge Zhang EXPORT_SYMBOL_GPL(vmci_qpair_alloc); 291806164d2bSGeorge Zhang 291906164d2bSGeorge Zhang /* 292006164d2bSGeorge Zhang * vmci_qpair_detach() - Detatches the client from a queue pair. 292106164d2bSGeorge Zhang * @qpair: Reference of a pointer to the qpair struct. 292206164d2bSGeorge Zhang * 292306164d2bSGeorge Zhang * This is the client interface for detaching from a VMCIQPair. 292406164d2bSGeorge Zhang * Note that this routine will free the memory allocated for the 292506164d2bSGeorge Zhang * vmci_qp structure too. 292606164d2bSGeorge Zhang */ 292706164d2bSGeorge Zhang int vmci_qpair_detach(struct vmci_qp **qpair) 292806164d2bSGeorge Zhang { 292906164d2bSGeorge Zhang int result; 293006164d2bSGeorge Zhang struct vmci_qp *old_qpair; 293106164d2bSGeorge Zhang 293206164d2bSGeorge Zhang if (!qpair || !(*qpair)) 293306164d2bSGeorge Zhang return VMCI_ERROR_INVALID_ARGS; 293406164d2bSGeorge Zhang 293506164d2bSGeorge Zhang old_qpair = *qpair; 293606164d2bSGeorge Zhang result = qp_detatch(old_qpair->handle, old_qpair->guest_endpoint); 293706164d2bSGeorge Zhang 293806164d2bSGeorge Zhang /* 293906164d2bSGeorge Zhang * The guest can fail to detach for a number of reasons, and 294006164d2bSGeorge Zhang * if it does so, it will cleanup the entry (if there is one). 294106164d2bSGeorge Zhang * The host can fail too, but it won't cleanup the entry 294206164d2bSGeorge Zhang * immediately, it will do that later when the context is 294306164d2bSGeorge Zhang * freed. Either way, we need to release the qpair struct 294406164d2bSGeorge Zhang * here; there isn't much the caller can do, and we don't want 294506164d2bSGeorge Zhang * to leak. 294606164d2bSGeorge Zhang */ 294706164d2bSGeorge Zhang 294806164d2bSGeorge Zhang memset(old_qpair, 0, sizeof(*old_qpair)); 294906164d2bSGeorge Zhang old_qpair->handle = VMCI_INVALID_HANDLE; 295006164d2bSGeorge Zhang old_qpair->peer = VMCI_INVALID_ID; 295106164d2bSGeorge Zhang kfree(old_qpair); 295206164d2bSGeorge Zhang *qpair = NULL; 295306164d2bSGeorge Zhang 295406164d2bSGeorge Zhang return result; 295506164d2bSGeorge Zhang } 295606164d2bSGeorge Zhang EXPORT_SYMBOL_GPL(vmci_qpair_detach); 295706164d2bSGeorge Zhang 295806164d2bSGeorge Zhang /* 295906164d2bSGeorge Zhang * vmci_qpair_get_produce_indexes() - Retrieves the indexes of the producer. 296006164d2bSGeorge Zhang * @qpair: Pointer to the queue pair struct. 296106164d2bSGeorge Zhang * @producer_tail: Reference used for storing producer tail index. 296206164d2bSGeorge Zhang * @consumer_head: Reference used for storing the consumer head index. 296306164d2bSGeorge Zhang * 296406164d2bSGeorge Zhang * This is the client interface for getting the current indexes of the 296506164d2bSGeorge Zhang * QPair from the point of the view of the caller as the producer. 296606164d2bSGeorge Zhang */ 296706164d2bSGeorge Zhang int vmci_qpair_get_produce_indexes(const struct vmci_qp *qpair, 296806164d2bSGeorge Zhang u64 *producer_tail, 296906164d2bSGeorge Zhang u64 *consumer_head) 297006164d2bSGeorge Zhang { 297106164d2bSGeorge Zhang struct vmci_queue_header *produce_q_header; 297206164d2bSGeorge Zhang struct vmci_queue_header *consume_q_header; 297306164d2bSGeorge Zhang int result; 297406164d2bSGeorge Zhang 297506164d2bSGeorge Zhang if (!qpair) 297606164d2bSGeorge Zhang return VMCI_ERROR_INVALID_ARGS; 297706164d2bSGeorge Zhang 297806164d2bSGeorge Zhang qp_lock(qpair); 297906164d2bSGeorge Zhang result = 298006164d2bSGeorge Zhang qp_get_queue_headers(qpair, &produce_q_header, &consume_q_header); 298106164d2bSGeorge Zhang if (result == VMCI_SUCCESS) 298206164d2bSGeorge Zhang vmci_q_header_get_pointers(produce_q_header, consume_q_header, 298306164d2bSGeorge Zhang producer_tail, consumer_head); 298406164d2bSGeorge Zhang qp_unlock(qpair); 298506164d2bSGeorge Zhang 298606164d2bSGeorge Zhang if (result == VMCI_SUCCESS && 298706164d2bSGeorge Zhang ((producer_tail && *producer_tail >= qpair->produce_q_size) || 298806164d2bSGeorge Zhang (consumer_head && *consumer_head >= qpair->produce_q_size))) 298906164d2bSGeorge Zhang return VMCI_ERROR_INVALID_SIZE; 299006164d2bSGeorge Zhang 299106164d2bSGeorge Zhang return result; 299206164d2bSGeorge Zhang } 299306164d2bSGeorge Zhang EXPORT_SYMBOL_GPL(vmci_qpair_get_produce_indexes); 299406164d2bSGeorge Zhang 299506164d2bSGeorge Zhang /* 299606164d2bSGeorge Zhang * vmci_qpair_get_consume_indexes() - Retrieves the indexes of the comsumer. 299706164d2bSGeorge Zhang * @qpair: Pointer to the queue pair struct. 299806164d2bSGeorge Zhang * @consumer_tail: Reference used for storing consumer tail index. 299906164d2bSGeorge Zhang * @producer_head: Reference used for storing the producer head index. 300006164d2bSGeorge Zhang * 300106164d2bSGeorge Zhang * This is the client interface for getting the current indexes of the 300206164d2bSGeorge Zhang * QPair from the point of the view of the caller as the consumer. 300306164d2bSGeorge Zhang */ 300406164d2bSGeorge Zhang int vmci_qpair_get_consume_indexes(const struct vmci_qp *qpair, 300506164d2bSGeorge Zhang u64 *consumer_tail, 300606164d2bSGeorge Zhang u64 *producer_head) 300706164d2bSGeorge Zhang { 300806164d2bSGeorge Zhang struct vmci_queue_header *produce_q_header; 300906164d2bSGeorge Zhang struct vmci_queue_header *consume_q_header; 301006164d2bSGeorge Zhang int result; 301106164d2bSGeorge Zhang 301206164d2bSGeorge Zhang if (!qpair) 301306164d2bSGeorge Zhang return VMCI_ERROR_INVALID_ARGS; 301406164d2bSGeorge Zhang 301506164d2bSGeorge Zhang qp_lock(qpair); 301606164d2bSGeorge Zhang result = 301706164d2bSGeorge Zhang qp_get_queue_headers(qpair, &produce_q_header, &consume_q_header); 301806164d2bSGeorge Zhang if (result == VMCI_SUCCESS) 301906164d2bSGeorge Zhang vmci_q_header_get_pointers(consume_q_header, produce_q_header, 302006164d2bSGeorge Zhang consumer_tail, producer_head); 302106164d2bSGeorge Zhang qp_unlock(qpair); 302206164d2bSGeorge Zhang 302306164d2bSGeorge Zhang if (result == VMCI_SUCCESS && 302406164d2bSGeorge Zhang ((consumer_tail && *consumer_tail >= qpair->consume_q_size) || 302506164d2bSGeorge Zhang (producer_head && *producer_head >= qpair->consume_q_size))) 302606164d2bSGeorge Zhang return VMCI_ERROR_INVALID_SIZE; 302706164d2bSGeorge Zhang 302806164d2bSGeorge Zhang return result; 302906164d2bSGeorge Zhang } 303006164d2bSGeorge Zhang EXPORT_SYMBOL_GPL(vmci_qpair_get_consume_indexes); 303106164d2bSGeorge Zhang 303206164d2bSGeorge Zhang /* 303306164d2bSGeorge Zhang * vmci_qpair_produce_free_space() - Retrieves free space in producer queue. 303406164d2bSGeorge Zhang * @qpair: Pointer to the queue pair struct. 303506164d2bSGeorge Zhang * 303606164d2bSGeorge Zhang * This is the client interface for getting the amount of free 303706164d2bSGeorge Zhang * space in the QPair from the point of the view of the caller as 303806164d2bSGeorge Zhang * the producer which is the common case. Returns < 0 if err, else 303906164d2bSGeorge Zhang * available bytes into which data can be enqueued if > 0. 304006164d2bSGeorge Zhang */ 304106164d2bSGeorge Zhang s64 vmci_qpair_produce_free_space(const struct vmci_qp *qpair) 304206164d2bSGeorge Zhang { 304306164d2bSGeorge Zhang struct vmci_queue_header *produce_q_header; 304406164d2bSGeorge Zhang struct vmci_queue_header *consume_q_header; 304506164d2bSGeorge Zhang s64 result; 304606164d2bSGeorge Zhang 304706164d2bSGeorge Zhang if (!qpair) 304806164d2bSGeorge Zhang return VMCI_ERROR_INVALID_ARGS; 304906164d2bSGeorge Zhang 305006164d2bSGeorge Zhang qp_lock(qpair); 305106164d2bSGeorge Zhang result = 305206164d2bSGeorge Zhang qp_get_queue_headers(qpair, &produce_q_header, &consume_q_header); 305306164d2bSGeorge Zhang if (result == VMCI_SUCCESS) 305406164d2bSGeorge Zhang result = vmci_q_header_free_space(produce_q_header, 305506164d2bSGeorge Zhang consume_q_header, 305606164d2bSGeorge Zhang qpair->produce_q_size); 305706164d2bSGeorge Zhang else 305806164d2bSGeorge Zhang result = 0; 305906164d2bSGeorge Zhang 306006164d2bSGeorge Zhang qp_unlock(qpair); 306106164d2bSGeorge Zhang 306206164d2bSGeorge Zhang return result; 306306164d2bSGeorge Zhang } 306406164d2bSGeorge Zhang EXPORT_SYMBOL_GPL(vmci_qpair_produce_free_space); 306506164d2bSGeorge Zhang 306606164d2bSGeorge Zhang /* 306706164d2bSGeorge Zhang * vmci_qpair_consume_free_space() - Retrieves free space in consumer queue. 306806164d2bSGeorge Zhang * @qpair: Pointer to the queue pair struct. 306906164d2bSGeorge Zhang * 307006164d2bSGeorge Zhang * This is the client interface for getting the amount of free 307106164d2bSGeorge Zhang * space in the QPair from the point of the view of the caller as 307206164d2bSGeorge Zhang * the consumer which is not the common case. Returns < 0 if err, else 307306164d2bSGeorge Zhang * available bytes into which data can be enqueued if > 0. 307406164d2bSGeorge Zhang */ 307506164d2bSGeorge Zhang s64 vmci_qpair_consume_free_space(const struct vmci_qp *qpair) 307606164d2bSGeorge Zhang { 307706164d2bSGeorge Zhang struct vmci_queue_header *produce_q_header; 307806164d2bSGeorge Zhang struct vmci_queue_header *consume_q_header; 307906164d2bSGeorge Zhang s64 result; 308006164d2bSGeorge Zhang 308106164d2bSGeorge Zhang if (!qpair) 308206164d2bSGeorge Zhang return VMCI_ERROR_INVALID_ARGS; 308306164d2bSGeorge Zhang 308406164d2bSGeorge Zhang qp_lock(qpair); 308506164d2bSGeorge Zhang result = 308606164d2bSGeorge Zhang qp_get_queue_headers(qpair, &produce_q_header, &consume_q_header); 308706164d2bSGeorge Zhang if (result == VMCI_SUCCESS) 308806164d2bSGeorge Zhang result = vmci_q_header_free_space(consume_q_header, 308906164d2bSGeorge Zhang produce_q_header, 309006164d2bSGeorge Zhang qpair->consume_q_size); 309106164d2bSGeorge Zhang else 309206164d2bSGeorge Zhang result = 0; 309306164d2bSGeorge Zhang 309406164d2bSGeorge Zhang qp_unlock(qpair); 309506164d2bSGeorge Zhang 309606164d2bSGeorge Zhang return result; 309706164d2bSGeorge Zhang } 309806164d2bSGeorge Zhang EXPORT_SYMBOL_GPL(vmci_qpair_consume_free_space); 309906164d2bSGeorge Zhang 310006164d2bSGeorge Zhang /* 310106164d2bSGeorge Zhang * vmci_qpair_produce_buf_ready() - Gets bytes ready to read from 310206164d2bSGeorge Zhang * producer queue. 310306164d2bSGeorge Zhang * @qpair: Pointer to the queue pair struct. 310406164d2bSGeorge Zhang * 310506164d2bSGeorge Zhang * This is the client interface for getting the amount of 310606164d2bSGeorge Zhang * enqueued data in the QPair from the point of the view of the 310706164d2bSGeorge Zhang * caller as the producer which is not the common case. Returns < 0 if err, 310806164d2bSGeorge Zhang * else available bytes that may be read. 310906164d2bSGeorge Zhang */ 311006164d2bSGeorge Zhang s64 vmci_qpair_produce_buf_ready(const struct vmci_qp *qpair) 311106164d2bSGeorge Zhang { 311206164d2bSGeorge Zhang struct vmci_queue_header *produce_q_header; 311306164d2bSGeorge Zhang struct vmci_queue_header *consume_q_header; 311406164d2bSGeorge Zhang s64 result; 311506164d2bSGeorge Zhang 311606164d2bSGeorge Zhang if (!qpair) 311706164d2bSGeorge Zhang return VMCI_ERROR_INVALID_ARGS; 311806164d2bSGeorge Zhang 311906164d2bSGeorge Zhang qp_lock(qpair); 312006164d2bSGeorge Zhang result = 312106164d2bSGeorge Zhang qp_get_queue_headers(qpair, &produce_q_header, &consume_q_header); 312206164d2bSGeorge Zhang if (result == VMCI_SUCCESS) 312306164d2bSGeorge Zhang result = vmci_q_header_buf_ready(produce_q_header, 312406164d2bSGeorge Zhang consume_q_header, 312506164d2bSGeorge Zhang qpair->produce_q_size); 312606164d2bSGeorge Zhang else 312706164d2bSGeorge Zhang result = 0; 312806164d2bSGeorge Zhang 312906164d2bSGeorge Zhang qp_unlock(qpair); 313006164d2bSGeorge Zhang 313106164d2bSGeorge Zhang return result; 313206164d2bSGeorge Zhang } 313306164d2bSGeorge Zhang EXPORT_SYMBOL_GPL(vmci_qpair_produce_buf_ready); 313406164d2bSGeorge Zhang 313506164d2bSGeorge Zhang /* 313606164d2bSGeorge Zhang * vmci_qpair_consume_buf_ready() - Gets bytes ready to read from 313706164d2bSGeorge Zhang * consumer queue. 313806164d2bSGeorge Zhang * @qpair: Pointer to the queue pair struct. 313906164d2bSGeorge Zhang * 314006164d2bSGeorge Zhang * This is the client interface for getting the amount of 314106164d2bSGeorge Zhang * enqueued data in the QPair from the point of the view of the 314206164d2bSGeorge Zhang * caller as the consumer which is the normal case. Returns < 0 if err, 314306164d2bSGeorge Zhang * else available bytes that may be read. 314406164d2bSGeorge Zhang */ 314506164d2bSGeorge Zhang s64 vmci_qpair_consume_buf_ready(const struct vmci_qp *qpair) 314606164d2bSGeorge Zhang { 314706164d2bSGeorge Zhang struct vmci_queue_header *produce_q_header; 314806164d2bSGeorge Zhang struct vmci_queue_header *consume_q_header; 314906164d2bSGeorge Zhang s64 result; 315006164d2bSGeorge Zhang 315106164d2bSGeorge Zhang if (!qpair) 315206164d2bSGeorge Zhang return VMCI_ERROR_INVALID_ARGS; 315306164d2bSGeorge Zhang 315406164d2bSGeorge Zhang qp_lock(qpair); 315506164d2bSGeorge Zhang result = 315606164d2bSGeorge Zhang qp_get_queue_headers(qpair, &produce_q_header, &consume_q_header); 315706164d2bSGeorge Zhang if (result == VMCI_SUCCESS) 315806164d2bSGeorge Zhang result = vmci_q_header_buf_ready(consume_q_header, 315906164d2bSGeorge Zhang produce_q_header, 316006164d2bSGeorge Zhang qpair->consume_q_size); 316106164d2bSGeorge Zhang else 316206164d2bSGeorge Zhang result = 0; 316306164d2bSGeorge Zhang 316406164d2bSGeorge Zhang qp_unlock(qpair); 316506164d2bSGeorge Zhang 316606164d2bSGeorge Zhang return result; 316706164d2bSGeorge Zhang } 316806164d2bSGeorge Zhang EXPORT_SYMBOL_GPL(vmci_qpair_consume_buf_ready); 316906164d2bSGeorge Zhang 317006164d2bSGeorge Zhang /* 317106164d2bSGeorge Zhang * vmci_qpair_enqueue() - Throw data on the queue. 317206164d2bSGeorge Zhang * @qpair: Pointer to the queue pair struct. 317306164d2bSGeorge Zhang * @buf: Pointer to buffer containing data 317406164d2bSGeorge Zhang * @buf_size: Length of buffer. 317506164d2bSGeorge Zhang * @buf_type: Buffer type (Unused). 317606164d2bSGeorge Zhang * 317706164d2bSGeorge Zhang * This is the client interface for enqueueing data into the queue. 317806164d2bSGeorge Zhang * Returns number of bytes enqueued or < 0 on error. 317906164d2bSGeorge Zhang */ 318006164d2bSGeorge Zhang ssize_t vmci_qpair_enqueue(struct vmci_qp *qpair, 318106164d2bSGeorge Zhang const void *buf, 318206164d2bSGeorge Zhang size_t buf_size, 318306164d2bSGeorge Zhang int buf_type) 318406164d2bSGeorge Zhang { 318506164d2bSGeorge Zhang ssize_t result; 318606164d2bSGeorge Zhang 318706164d2bSGeorge Zhang if (!qpair || !buf) 318806164d2bSGeorge Zhang return VMCI_ERROR_INVALID_ARGS; 318906164d2bSGeorge Zhang 319006164d2bSGeorge Zhang qp_lock(qpair); 319106164d2bSGeorge Zhang 319206164d2bSGeorge Zhang do { 319306164d2bSGeorge Zhang result = qp_enqueue_locked(qpair->produce_q, 319406164d2bSGeorge Zhang qpair->consume_q, 319506164d2bSGeorge Zhang qpair->produce_q_size, 319606164d2bSGeorge Zhang buf, buf_size, 319706164d2bSGeorge Zhang qp_memcpy_to_queue, 319806164d2bSGeorge Zhang vmci_can_block(qpair->flags)); 319906164d2bSGeorge Zhang 320006164d2bSGeorge Zhang if (result == VMCI_ERROR_QUEUEPAIR_NOT_READY && 320106164d2bSGeorge Zhang !qp_wait_for_ready_queue(qpair)) 320206164d2bSGeorge Zhang result = VMCI_ERROR_WOULD_BLOCK; 320306164d2bSGeorge Zhang 320406164d2bSGeorge Zhang } while (result == VMCI_ERROR_QUEUEPAIR_NOT_READY); 320506164d2bSGeorge Zhang 320606164d2bSGeorge Zhang qp_unlock(qpair); 320706164d2bSGeorge Zhang 320806164d2bSGeorge Zhang return result; 320906164d2bSGeorge Zhang } 321006164d2bSGeorge Zhang EXPORT_SYMBOL_GPL(vmci_qpair_enqueue); 321106164d2bSGeorge Zhang 321206164d2bSGeorge Zhang /* 321306164d2bSGeorge Zhang * vmci_qpair_dequeue() - Get data from the queue. 321406164d2bSGeorge Zhang * @qpair: Pointer to the queue pair struct. 321506164d2bSGeorge Zhang * @buf: Pointer to buffer for the data 321606164d2bSGeorge Zhang * @buf_size: Length of buffer. 321706164d2bSGeorge Zhang * @buf_type: Buffer type (Unused). 321806164d2bSGeorge Zhang * 321906164d2bSGeorge Zhang * This is the client interface for dequeueing data from the queue. 322006164d2bSGeorge Zhang * Returns number of bytes dequeued or < 0 on error. 322106164d2bSGeorge Zhang */ 322206164d2bSGeorge Zhang ssize_t vmci_qpair_dequeue(struct vmci_qp *qpair, 322306164d2bSGeorge Zhang void *buf, 322406164d2bSGeorge Zhang size_t buf_size, 322506164d2bSGeorge Zhang int buf_type) 322606164d2bSGeorge Zhang { 322706164d2bSGeorge Zhang ssize_t result; 322806164d2bSGeorge Zhang 322906164d2bSGeorge Zhang if (!qpair || !buf) 323006164d2bSGeorge Zhang return VMCI_ERROR_INVALID_ARGS; 323106164d2bSGeorge Zhang 323206164d2bSGeorge Zhang qp_lock(qpair); 323306164d2bSGeorge Zhang 323406164d2bSGeorge Zhang do { 323506164d2bSGeorge Zhang result = qp_dequeue_locked(qpair->produce_q, 323606164d2bSGeorge Zhang qpair->consume_q, 323706164d2bSGeorge Zhang qpair->consume_q_size, 323806164d2bSGeorge Zhang buf, buf_size, 323906164d2bSGeorge Zhang qp_memcpy_from_queue, true, 324006164d2bSGeorge Zhang vmci_can_block(qpair->flags)); 324106164d2bSGeorge Zhang 324206164d2bSGeorge Zhang if (result == VMCI_ERROR_QUEUEPAIR_NOT_READY && 324306164d2bSGeorge Zhang !qp_wait_for_ready_queue(qpair)) 324406164d2bSGeorge Zhang result = VMCI_ERROR_WOULD_BLOCK; 324506164d2bSGeorge Zhang 324606164d2bSGeorge Zhang } while (result == VMCI_ERROR_QUEUEPAIR_NOT_READY); 324706164d2bSGeorge Zhang 324806164d2bSGeorge Zhang qp_unlock(qpair); 324906164d2bSGeorge Zhang 325006164d2bSGeorge Zhang return result; 325106164d2bSGeorge Zhang } 325206164d2bSGeorge Zhang EXPORT_SYMBOL_GPL(vmci_qpair_dequeue); 325306164d2bSGeorge Zhang 325406164d2bSGeorge Zhang /* 325506164d2bSGeorge Zhang * vmci_qpair_peek() - Peek at the data in the queue. 325606164d2bSGeorge Zhang * @qpair: Pointer to the queue pair struct. 325706164d2bSGeorge Zhang * @buf: Pointer to buffer for the data 325806164d2bSGeorge Zhang * @buf_size: Length of buffer. 325906164d2bSGeorge Zhang * @buf_type: Buffer type (Unused on Linux). 326006164d2bSGeorge Zhang * 326106164d2bSGeorge Zhang * This is the client interface for peeking into a queue. (I.e., 326206164d2bSGeorge Zhang * copy data from the queue without updating the head pointer.) 326306164d2bSGeorge Zhang * Returns number of bytes dequeued or < 0 on error. 326406164d2bSGeorge Zhang */ 326506164d2bSGeorge Zhang ssize_t vmci_qpair_peek(struct vmci_qp *qpair, 326606164d2bSGeorge Zhang void *buf, 326706164d2bSGeorge Zhang size_t buf_size, 326806164d2bSGeorge Zhang int buf_type) 326906164d2bSGeorge Zhang { 327006164d2bSGeorge Zhang ssize_t result; 327106164d2bSGeorge Zhang 327206164d2bSGeorge Zhang if (!qpair || !buf) 327306164d2bSGeorge Zhang return VMCI_ERROR_INVALID_ARGS; 327406164d2bSGeorge Zhang 327506164d2bSGeorge Zhang qp_lock(qpair); 327606164d2bSGeorge Zhang 327706164d2bSGeorge Zhang do { 327806164d2bSGeorge Zhang result = qp_dequeue_locked(qpair->produce_q, 327906164d2bSGeorge Zhang qpair->consume_q, 328006164d2bSGeorge Zhang qpair->consume_q_size, 328106164d2bSGeorge Zhang buf, buf_size, 328206164d2bSGeorge Zhang qp_memcpy_from_queue, false, 328306164d2bSGeorge Zhang vmci_can_block(qpair->flags)); 328406164d2bSGeorge Zhang 328506164d2bSGeorge Zhang if (result == VMCI_ERROR_QUEUEPAIR_NOT_READY && 328606164d2bSGeorge Zhang !qp_wait_for_ready_queue(qpair)) 328706164d2bSGeorge Zhang result = VMCI_ERROR_WOULD_BLOCK; 328806164d2bSGeorge Zhang 328906164d2bSGeorge Zhang } while (result == VMCI_ERROR_QUEUEPAIR_NOT_READY); 329006164d2bSGeorge Zhang 329106164d2bSGeorge Zhang qp_unlock(qpair); 329206164d2bSGeorge Zhang 329306164d2bSGeorge Zhang return result; 329406164d2bSGeorge Zhang } 329506164d2bSGeorge Zhang EXPORT_SYMBOL_GPL(vmci_qpair_peek); 329606164d2bSGeorge Zhang 329706164d2bSGeorge Zhang /* 329806164d2bSGeorge Zhang * vmci_qpair_enquev() - Throw data on the queue using iov. 329906164d2bSGeorge Zhang * @qpair: Pointer to the queue pair struct. 330006164d2bSGeorge Zhang * @iov: Pointer to buffer containing data 330106164d2bSGeorge Zhang * @iov_size: Length of buffer. 330206164d2bSGeorge Zhang * @buf_type: Buffer type (Unused). 330306164d2bSGeorge Zhang * 330406164d2bSGeorge Zhang * This is the client interface for enqueueing data into the queue. 330506164d2bSGeorge Zhang * This function uses IO vectors to handle the work. Returns number 330606164d2bSGeorge Zhang * of bytes enqueued or < 0 on error. 330706164d2bSGeorge Zhang */ 330806164d2bSGeorge Zhang ssize_t vmci_qpair_enquev(struct vmci_qp *qpair, 330906164d2bSGeorge Zhang void *iov, 331006164d2bSGeorge Zhang size_t iov_size, 331106164d2bSGeorge Zhang int buf_type) 331206164d2bSGeorge Zhang { 331306164d2bSGeorge Zhang ssize_t result; 331406164d2bSGeorge Zhang 331506164d2bSGeorge Zhang if (!qpair || !iov) 331606164d2bSGeorge Zhang return VMCI_ERROR_INVALID_ARGS; 331706164d2bSGeorge Zhang 331806164d2bSGeorge Zhang qp_lock(qpair); 331906164d2bSGeorge Zhang 332006164d2bSGeorge Zhang do { 332106164d2bSGeorge Zhang result = qp_enqueue_locked(qpair->produce_q, 332206164d2bSGeorge Zhang qpair->consume_q, 332306164d2bSGeorge Zhang qpair->produce_q_size, 332406164d2bSGeorge Zhang iov, iov_size, 332506164d2bSGeorge Zhang qp_memcpy_to_queue_iov, 332606164d2bSGeorge Zhang vmci_can_block(qpair->flags)); 332706164d2bSGeorge Zhang 332806164d2bSGeorge Zhang if (result == VMCI_ERROR_QUEUEPAIR_NOT_READY && 332906164d2bSGeorge Zhang !qp_wait_for_ready_queue(qpair)) 333006164d2bSGeorge Zhang result = VMCI_ERROR_WOULD_BLOCK; 333106164d2bSGeorge Zhang 333206164d2bSGeorge Zhang } while (result == VMCI_ERROR_QUEUEPAIR_NOT_READY); 333306164d2bSGeorge Zhang 333406164d2bSGeorge Zhang qp_unlock(qpair); 333506164d2bSGeorge Zhang 333606164d2bSGeorge Zhang return result; 333706164d2bSGeorge Zhang } 333806164d2bSGeorge Zhang EXPORT_SYMBOL_GPL(vmci_qpair_enquev); 333906164d2bSGeorge Zhang 334006164d2bSGeorge Zhang /* 334106164d2bSGeorge Zhang * vmci_qpair_dequev() - Get data from the queue using iov. 334206164d2bSGeorge Zhang * @qpair: Pointer to the queue pair struct. 334306164d2bSGeorge Zhang * @iov: Pointer to buffer for the data 334406164d2bSGeorge Zhang * @iov_size: Length of buffer. 334506164d2bSGeorge Zhang * @buf_type: Buffer type (Unused). 334606164d2bSGeorge Zhang * 334706164d2bSGeorge Zhang * This is the client interface for dequeueing data from the queue. 334806164d2bSGeorge Zhang * This function uses IO vectors to handle the work. Returns number 334906164d2bSGeorge Zhang * of bytes dequeued or < 0 on error. 335006164d2bSGeorge Zhang */ 335106164d2bSGeorge Zhang ssize_t vmci_qpair_dequev(struct vmci_qp *qpair, 335206164d2bSGeorge Zhang void *iov, 335306164d2bSGeorge Zhang size_t iov_size, 335406164d2bSGeorge Zhang int buf_type) 335506164d2bSGeorge Zhang { 335606164d2bSGeorge Zhang ssize_t result; 335706164d2bSGeorge Zhang 335806164d2bSGeorge Zhang if (!qpair || !iov) 335906164d2bSGeorge Zhang return VMCI_ERROR_INVALID_ARGS; 336006164d2bSGeorge Zhang 336132b083a3SAndy King qp_lock(qpair); 336232b083a3SAndy King 336306164d2bSGeorge Zhang do { 336406164d2bSGeorge Zhang result = qp_dequeue_locked(qpair->produce_q, 336506164d2bSGeorge Zhang qpair->consume_q, 336606164d2bSGeorge Zhang qpair->consume_q_size, 336706164d2bSGeorge Zhang iov, iov_size, 336806164d2bSGeorge Zhang qp_memcpy_from_queue_iov, 336906164d2bSGeorge Zhang true, vmci_can_block(qpair->flags)); 337006164d2bSGeorge Zhang 337106164d2bSGeorge Zhang if (result == VMCI_ERROR_QUEUEPAIR_NOT_READY && 337206164d2bSGeorge Zhang !qp_wait_for_ready_queue(qpair)) 337306164d2bSGeorge Zhang result = VMCI_ERROR_WOULD_BLOCK; 337406164d2bSGeorge Zhang 337506164d2bSGeorge Zhang } while (result == VMCI_ERROR_QUEUEPAIR_NOT_READY); 337606164d2bSGeorge Zhang 337706164d2bSGeorge Zhang qp_unlock(qpair); 337806164d2bSGeorge Zhang 337906164d2bSGeorge Zhang return result; 338006164d2bSGeorge Zhang } 338106164d2bSGeorge Zhang EXPORT_SYMBOL_GPL(vmci_qpair_dequev); 338206164d2bSGeorge Zhang 338306164d2bSGeorge Zhang /* 338406164d2bSGeorge Zhang * vmci_qpair_peekv() - Peek at the data in the queue using iov. 338506164d2bSGeorge Zhang * @qpair: Pointer to the queue pair struct. 338606164d2bSGeorge Zhang * @iov: Pointer to buffer for the data 338706164d2bSGeorge Zhang * @iov_size: Length of buffer. 338806164d2bSGeorge Zhang * @buf_type: Buffer type (Unused on Linux). 338906164d2bSGeorge Zhang * 339006164d2bSGeorge Zhang * This is the client interface for peeking into a queue. (I.e., 339106164d2bSGeorge Zhang * copy data from the queue without updating the head pointer.) 339206164d2bSGeorge Zhang * This function uses IO vectors to handle the work. Returns number 339306164d2bSGeorge Zhang * of bytes peeked or < 0 on error. 339406164d2bSGeorge Zhang */ 339506164d2bSGeorge Zhang ssize_t vmci_qpair_peekv(struct vmci_qp *qpair, 339606164d2bSGeorge Zhang void *iov, 339706164d2bSGeorge Zhang size_t iov_size, 339806164d2bSGeorge Zhang int buf_type) 339906164d2bSGeorge Zhang { 340006164d2bSGeorge Zhang ssize_t result; 340106164d2bSGeorge Zhang 340206164d2bSGeorge Zhang if (!qpair || !iov) 340306164d2bSGeorge Zhang return VMCI_ERROR_INVALID_ARGS; 340406164d2bSGeorge Zhang 340506164d2bSGeorge Zhang qp_lock(qpair); 340606164d2bSGeorge Zhang 340706164d2bSGeorge Zhang do { 340806164d2bSGeorge Zhang result = qp_dequeue_locked(qpair->produce_q, 340906164d2bSGeorge Zhang qpair->consume_q, 341006164d2bSGeorge Zhang qpair->consume_q_size, 341106164d2bSGeorge Zhang iov, iov_size, 341206164d2bSGeorge Zhang qp_memcpy_from_queue_iov, 341306164d2bSGeorge Zhang false, vmci_can_block(qpair->flags)); 341406164d2bSGeorge Zhang 341506164d2bSGeorge Zhang if (result == VMCI_ERROR_QUEUEPAIR_NOT_READY && 341606164d2bSGeorge Zhang !qp_wait_for_ready_queue(qpair)) 341706164d2bSGeorge Zhang result = VMCI_ERROR_WOULD_BLOCK; 341806164d2bSGeorge Zhang 341906164d2bSGeorge Zhang } while (result == VMCI_ERROR_QUEUEPAIR_NOT_READY); 342006164d2bSGeorge Zhang 342106164d2bSGeorge Zhang qp_unlock(qpair); 342206164d2bSGeorge Zhang return result; 342306164d2bSGeorge Zhang } 342406164d2bSGeorge Zhang EXPORT_SYMBOL_GPL(vmci_qpair_peekv); 3425