65938e81 | 02-Apr-2024 |
Jens Axboe <axboe@kernel.dk> |
io_uring/kbuf: hold io_buffer_list reference over mmap
commit 561e4f9451d65fc2f7eef564e0064373e3019793 upstream.
If we look up the kbuf, ensure that it doesn't get unregistered until after we're do
io_uring/kbuf: hold io_buffer_list reference over mmap
commit 561e4f9451d65fc2f7eef564e0064373e3019793 upstream.
If we look up the kbuf, ensure that it doesn't get unregistered until after we're done with it. Since we're inside mmap, we cannot safely use the io_uring lock. Rely on the fact that we can lookup the buffer list under RCU now and grab a reference to it, preventing it from being unregistered until we're done with it. The lookup returns the io_buffer_list directly with it referenced.
Cc: stable@vger.kernel.org # v6.4+ Fixes: 5cf4f52e6d8a ("io_uring: free io_buffer_list entries via RCU") Signed-off-by: Jens Axboe <axboe@kernel.dk> Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
show more ...
|
6b9d49bc | 01-Apr-2024 |
Jens Axboe <axboe@kernel.dk> |
io_uring: use private workqueue for exit work
commit 73eaa2b583493b680c6f426531d6736c39643bfb upstream.
Rather than use the system unbound event workqueue, use an io_uring specific one. This avoids
io_uring: use private workqueue for exit work
commit 73eaa2b583493b680c6f426531d6736c39643bfb upstream.
Rather than use the system unbound event workqueue, use an io_uring specific one. This avoids dependencies with the tty, which also uses the system_unbound_wq, and issues flushes of said workqueue from inside its poll handling.
Cc: stable@vger.kernel.org Reported-by: Rasmus Karlsson <rasmus.karlsson@pajlada.com> Tested-by: Rasmus Karlsson <rasmus.karlsson@pajlada.com> Tested-by: Iskren Chernev <me@iskren.info> Link: https://github.com/axboe/liburing/issues/1113 Signed-off-by: Jens Axboe <axboe@kernel.dk> Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
show more ...
|
b392402d | 15-Mar-2024 |
Jens Axboe <axboe@kernel.dk> |
io_uring/kbuf: protect io_buffer_list teardown with a reference
commit 6b69c4ab4f685327d9e10caf0d84217ba23a8c4b upstream.
No functional changes in this patch, just in preparation for being able to
io_uring/kbuf: protect io_buffer_list teardown with a reference
commit 6b69c4ab4f685327d9e10caf0d84217ba23a8c4b upstream.
No functional changes in this patch, just in preparation for being able to keep the buffer list alive outside of the ctx->uring_lock.
Cc: stable@vger.kernel.org # v6.4+ Signed-off-by: Jens Axboe <axboe@kernel.dk> Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
show more ...
|
4c0a5da0 | 14-Mar-2024 |
Jens Axboe <axboe@kernel.dk> |
io_uring/kbuf: get rid of bl->is_ready
commit 3b80cff5a4d117c53d38ce805823084eaeffbde6 upstream.
Now that xarray is being exclusively used for the buffer_list lookup, this check is no longer needed
io_uring/kbuf: get rid of bl->is_ready
commit 3b80cff5a4d117c53d38ce805823084eaeffbde6 upstream.
Now that xarray is being exclusively used for the buffer_list lookup, this check is no longer needed. Get rid of it and the is_ready member.
Cc: stable@vger.kernel.org # v6.4+ Signed-off-by: Jens Axboe <axboe@kernel.dk> Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
show more ...
|
0f314c3b | 08-Mar-2024 |
Pavel Begunkov <asml.silence@gmail.com> |
io_uring: fix mshot io-wq checks
[ Upstream commit 3a96378e22cc46c7c49b5911f6c8631527a133a9 ]
When checking for concurrent CQE posting, we're not only interested in requests running from the poll h
io_uring: fix mshot io-wq checks
[ Upstream commit 3a96378e22cc46c7c49b5911f6c8631527a133a9 ]
When checking for concurrent CQE posting, we're not only interested in requests running from the poll handler but also strayed requests ended up in normal io-wq execution. We're disallowing multishots in general from io-wq, not only when they came in a certain way.
Cc: stable@vger.kernel.org Fixes: 17add5cea2bba ("io_uring: force multishot CQEs into task context") Signed-off-by: Pavel Begunkov <asml.silence@gmail.com> Link: https://lore.kernel.org/r/d8c5b36a39258036f93301cd60d3cd295e40653d.1709905727.git.asml.silence@gmail.com Signed-off-by: Jens Axboe <axboe@kernel.dk> Signed-off-by: Sasha Levin <sashal@kernel.org>
show more ...
|
4cbc5e93 | 15-Mar-2024 |
Pavel Begunkov <asml.silence@gmail.com> |
io_uring: fix poll_remove stalled req completion
[ Upstream commit 5e3afe580a9f5ca173a6bd55ffe10948796ef7e5 ]
Taking the ctx lock is not enough to use the deferred request completion infrastructure
io_uring: fix poll_remove stalled req completion
[ Upstream commit 5e3afe580a9f5ca173a6bd55ffe10948796ef7e5 ]
Taking the ctx lock is not enough to use the deferred request completion infrastructure, it'll get queued into the list but no one would expect it there, so it will sit there until next io_submit_flush_completions(). It's hard to care about the cancellation path, so complete it via tw.
Fixes: ef7dfac51d8ed ("io_uring/poll: serialize poll linked timer start with poll removal") Signed-off-by: Pavel Begunkov <asml.silence@gmail.com> Link: https://lore.kernel.org/r/c446740bc16858f8a2a8dcdce899812f21d15f23.1710514702.git.asml.silence@gmail.com Signed-off-by: Jens Axboe <axboe@kernel.dk> Signed-off-by: Sasha Levin <sashal@kernel.org>
show more ...
|
0b6f39c1 | 13-Mar-2024 |
Gabriel Krisman Bertazi <krisman@suse.de> |
io_uring: Fix release of pinned pages when __io_uaddr_map fails
[ Upstream commit 67d1189d1095d471ed7fa426c7e384a7140a5dd7 ]
Looking at the error path of __io_uaddr_map, if we fail after pinning th
io_uring: Fix release of pinned pages when __io_uaddr_map fails
[ Upstream commit 67d1189d1095d471ed7fa426c7e384a7140a5dd7 ]
Looking at the error path of __io_uaddr_map, if we fail after pinning the pages for any reasons, ret will be set to -EINVAL and the error handler won't properly release the pinned pages.
I didn't manage to trigger it without forcing a failure, but it can happen in real life when memory is heavily fragmented.
Signed-off-by: Gabriel Krisman Bertazi <krisman@suse.de> Fixes: 223ef4743164 ("io_uring: don't allow IORING_SETUP_NO_MMAP rings on highmem pages") Link: https://lore.kernel.org/r/20240313213912.1920-1-krisman@suse.de Signed-off-by: Jens Axboe <axboe@kernel.dk> Signed-off-by: Sasha Levin <sashal@kernel.org>
show more ...
|
59a53469 | 01-Mar-2024 |
Dan Carpenter <dan.carpenter@linaro.org> |
io_uring/net: fix overflow check in io_recvmsg_mshot_prep()
[ Upstream commit 8ede3db5061bb1fe28e2c9683329aafa89d2b1b4 ]
The "controllen" variable is type size_t (unsigned long). Casting it to int
io_uring/net: fix overflow check in io_recvmsg_mshot_prep()
[ Upstream commit 8ede3db5061bb1fe28e2c9683329aafa89d2b1b4 ]
The "controllen" variable is type size_t (unsigned long). Casting it to int could lead to an integer underflow.
The check_add_overflow() function considers the type of the destination which is type int. If we add two positive values and the result cannot fit in an integer then that's counted as an overflow.
However, if we cast "controllen" to an int and it turns negative, then negative values *can* fit into an int type so there is no overflow.
Good: 100 + (unsigned long)-4 = 96 <-- overflow Bad: 100 + (int)-4 = 96 <-- no overflow
I deleted the cast of the sizeof() as well. That's not a bug but the cast is unnecessary.
Fixes: 9b0fc3c054ff ("io_uring: fix types in io_recvmsg_multishot_overflow") Signed-off-by: Dan Carpenter <dan.carpenter@linaro.org> Link: https://lore.kernel.org/r/138bd2e2-ede8-4bcc-aa7b-f3d9de167a37@moroto.mountain Signed-off-by: Jens Axboe <axboe@kernel.dk> Signed-off-by: Sasha Levin <sashal@kernel.org>
show more ...
|
51d28472 | 27-Feb-2024 |
Jens Axboe <axboe@kernel.dk> |
io_uring/net: move receive multishot out of the generic msghdr path
[ Upstream commit c55978024d123d43808ab393a0a4ce3ce8568150 ]
Move the actual user_msghdr / compat_msghdr into the send and receiv
io_uring/net: move receive multishot out of the generic msghdr path
[ Upstream commit c55978024d123d43808ab393a0a4ce3ce8568150 ]
Move the actual user_msghdr / compat_msghdr into the send and receive sides, respectively, so we can move the uaddr receive handling into its own handler, and ditto the multishot with buffer selection logic.
Signed-off-by: Jens Axboe <axboe@kernel.dk> Stable-dep-of: 8ede3db5061b ("io_uring/net: fix overflow check in io_recvmsg_mshot_prep()") Signed-off-by: Sasha Levin <sashal@kernel.org>
show more ...
|
7b8fa7a0 | 31-Jan-2024 |
Jens Axboe <axboe@kernel.dk> |
io_uring: remove unconditional looping in local task_work handling
[ Upstream commit 9fe3eaea4a3530ca34a8d8ff00b1848c528789ca ]
If we have a ton of notifications coming in, we can be looping in her
io_uring: remove unconditional looping in local task_work handling
[ Upstream commit 9fe3eaea4a3530ca34a8d8ff00b1848c528789ca ]
If we have a ton of notifications coming in, we can be looping in here for a long time. This can be problematic for various reasons, mostly because we can starve userspace. If the application is waiting on N events, then only re-run if we need more events.
Fixes: c0e0d6ba25f1 ("io_uring: add IORING_SETUP_DEFER_TASKRUN") Signed-off-by: Jens Axboe <axboe@kernel.dk> Signed-off-by: Sasha Levin <sashal@kernel.org>
show more ...
|
6fc19b3d | 19-Dec-2023 |
Jens Axboe <axboe@kernel.dk> |
io_uring: drop any code related to SCM_RIGHTS
This is dead code after we dropped support for passing io_uring fds over SCM_RIGHTS, get rid of it.
Signed-off-by: Jens Axboe <axboe@kernel.dk> Signed-
io_uring: drop any code related to SCM_RIGHTS
This is dead code after we dropped support for passing io_uring fds over SCM_RIGHTS, get rid of it.
Signed-off-by: Jens Axboe <axboe@kernel.dk> Signed-off-by: Sasha Levin <sashal@kernel.org>
show more ...
|
22ccf61c | 29-Jan-2024 |
Jens Axboe <axboe@kernel.dk> |
io_uring/net: limit inline multishot retries
Commit 76b367a2d83163cf19173d5cb0b562acbabc8eac upstream.
If we have multiple clients and some/all are flooding the receives to such an extent that we c
io_uring/net: limit inline multishot retries
Commit 76b367a2d83163cf19173d5cb0b562acbabc8eac upstream.
If we have multiple clients and some/all are flooding the receives to such an extent that we can retry a LOT handling multishot receives, then we can be starving some clients and hence serving traffic in an imbalanced fashion.
Limit multishot retry attempts to some arbitrary value, whose only purpose serves to ensure that we don't keep serving a single connection for way too long. We default to 32 retries, which should be more than enough to provide fairness, yet not so small that we'll spend too much time requeuing rather than handling traffic.
Cc: stable@vger.kernel.org Depends-on: 704ea888d646 ("io_uring/poll: add requeue return code from poll multishot handling") Depends-on: 1e5d765a82f ("io_uring/net: un-indent mshot retry path in io_recv_finish()") Depends-on: e84b01a880f6 ("io_uring/poll: move poll execution helpers higher up") Fixes: b3fdea6ecb55 ("io_uring: multishot recv") Fixes: 9bb66906f23e ("io_uring: support multishot in recvmsg") Link: https://github.com/axboe/liburing/issues/1043 Signed-off-by: Jens Axboe <axboe@kernel.dk> Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
show more ...
|
7cbd3aa5 | 29-Jan-2024 |
Jens Axboe <axboe@kernel.dk> |
io_uring/poll: add requeue return code from poll multishot handling
Commit 704ea888d646cb9d715662944cf389c823252ee0 upstream.
Since our poll handling is edge triggered, multishot handlers retry int
io_uring/poll: add requeue return code from poll multishot handling
Commit 704ea888d646cb9d715662944cf389c823252ee0 upstream.
Since our poll handling is edge triggered, multishot handlers retry internally until they know that no more data is available. In preparation for limiting these retries, add an internal return code, IOU_REQUEUE, which can be used to inform the poll backend about the handler wanting to retry, but that this should happen through a normal task_work requeue rather than keep hammering on the issue side for this one request.
No functional changes in this patch, nobody is using this return code just yet.
Signed-off-by: Jens Axboe <axboe@kernel.dk> Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
show more ...
|