#
c03f05f1 |
| 29-Jul-2022 |
Al Viro <viro@zeniv.linux.org.uk> |
fix copy_page_from_iter() for compound destinations
had been broken for ITER_BVEC et.al. since ever (OK, v3.17 when ITER_BVEC had first appeared)...
Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
|
Revision tags: v5.15.58, v5.15.57, v5.15.56, v5.15.55, v5.15.54, v5.15.53, v5.15.52, v5.15.51, v5.15.50 |
|
#
f0f6b614 |
| 23-Jun-2022 |
Al Viro <viro@zeniv.linux.org.uk> |
copy_page_to_iter(): don't split high-order page in case of ITER_PIPE
... just shove it into one pipe_buffer.
Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
|
Revision tags: v5.15.49, v5.15.48, v5.15.47 |
|
#
310d9d5a |
| 11-Jun-2022 |
Al Viro <viro@zeniv.linux.org.uk> |
expand those iov_iter_advance()...
Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
|
#
746de1f8 |
| 14-Jun-2022 |
Al Viro <viro@zeniv.linux.org.uk> |
pipe_get_pages(): switch to append_pipe()
now that we are advancing the iterator, there's no need to treat the first page separately - just call append_pipe() in a loop.
Signed-off-by: Al Viro <vir
pipe_get_pages(): switch to append_pipe()
now that we are advancing the iterator, there's no need to treat the first page separately - just call append_pipe() in a loop.
Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
show more ...
|
#
eba2d3d7 |
| 10-Jun-2022 |
Al Viro <viro@zeniv.linux.org.uk> |
get rid of non-advancing variants
mechanical change; will be further massaged in subsequent commits
Reviewed-by: Jeff Layton <jlayton@kernel.org> Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
|
#
3cf42da3 |
| 17-Jun-2022 |
Al Viro <viro@zeniv.linux.org.uk> |
iov_iter: saner helper for page array allocation
All call sites of get_pages_array() are essenitally identical now. Replace with common helper...
Returns number of slots available in resulting arra
iov_iter: saner helper for page array allocation
All call sites of get_pages_array() are essenitally identical now. Replace with common helper...
Returns number of slots available in resulting array or 0 on OOM; it's up to the caller to make sure it doesn't ask to zero-entry array (i.e. neither maxpages nor size are allowed to be zero).
Reviewed-by: Jeff Layton <jlayton@kernel.org> Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
show more ...
|
#
85200084 |
| 17-Jun-2022 |
Al Viro <viro@zeniv.linux.org.uk> |
fold __pipe_get_pages() into pipe_get_pages()
... and don't mangle maxsize there - turn the loop into counting one instead. Easier to see that we won't run out of array that way. Note that special
fold __pipe_get_pages() into pipe_get_pages()
... and don't mangle maxsize there - turn the loop into counting one instead. Easier to see that we won't run out of array that way. Note that special treatment of the partial buffer in that thing is an artifact of the non-advancing semantics of iov_iter_get_pages() - if not for that, it would be append_pipe(), same as the body of the loop that follows it. IOW, once we make iov_iter_get_pages() advancing, the whole thing will turn into calculate how many pages do we want allocate an array (if needed) call append_pipe() that many times.
Reviewed-by: Jeff Layton <jlayton@kernel.org> Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
show more ...
|
#
0aa4fc32 |
| 10-Jun-2022 |
Al Viro <viro@zeniv.linux.org.uk> |
ITER_XARRAY: don't open-code DIV_ROUND_UP()
Reviewed-by: Jeff Layton <jlayton@kernel.org> Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
|
#
451c0ba9 |
| 17-Jun-2022 |
Al Viro <viro@zeniv.linux.org.uk> |
unify the rest of iov_iter_get_pages()/iov_iter_get_pages_alloc() guts
same as for pipes and xarrays; after that iov_iter_get_pages() becomes a wrapper for __iov_iter_get_pages_alloc().
Signed-off-
unify the rest of iov_iter_get_pages()/iov_iter_get_pages_alloc() guts
same as for pipes and xarrays; after that iov_iter_get_pages() becomes a wrapper for __iov_iter_get_pages_alloc().
Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
show more ...
|
#
68fe506f |
| 17-Jun-2022 |
Al Viro <viro@zeniv.linux.org.uk> |
unify xarray_get_pages() and xarray_get_pages_alloc()
same as for pipes
Reviewed-by: Jeff Layton <jlayton@kernel.org> Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
|
#
acbdeb83 |
| 17-Jun-2022 |
Al Viro <viro@zeniv.linux.org.uk> |
unify pipe_get_pages() and pipe_get_pages_alloc()
The differences between those two are * pipe_get_pages() gets a non-NULL struct page ** value pointing to preallocated array + array size. * pipe_g
unify pipe_get_pages() and pipe_get_pages_alloc()
The differences between those two are * pipe_get_pages() gets a non-NULL struct page ** value pointing to preallocated array + array size. * pipe_get_pages_alloc() gets an address of struct page ** variable that contains NULL, allocates the array and (on success) stores its address in that variable.
Not hard to combine - always pass struct page ***, have the previous pipe_get_pages_alloc() caller pass ~0U as cap for array size.
Reviewed-by: Jeff Layton <jlayton@kernel.org> Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
show more ...
|
#
c81ce28d |
| 17-Jun-2022 |
Al Viro <viro@zeniv.linux.org.uk> |
iov_iter_get_pages(): sanity-check arguments
zero maxpages is bogus, but best treated as "just return 0"; NULL pages, OTOH, should be treated as a hard bug.
get rid of now completely useless checks
iov_iter_get_pages(): sanity-check arguments
zero maxpages is bogus, but best treated as "just return 0"; NULL pages, OTOH, should be treated as a hard bug.
get rid of now completely useless checks in xarray_get_pages{,_alloc}().
Reviewed-by: Jeff Layton <jlayton@kernel.org> Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
show more ...
|
#
91329559 |
| 10-Jun-2022 |
Al Viro <viro@zeniv.linux.org.uk> |
iov_iter_get_pages_alloc(): lift freeing pages array on failure exits into wrapper
Incidentally, ITER_XARRAY did *not* free the sucker in case when iter_xarray_populate_pages() returned 0...
Review
iov_iter_get_pages_alloc(): lift freeing pages array on failure exits into wrapper
Incidentally, ITER_XARRAY did *not* free the sucker in case when iter_xarray_populate_pages() returned 0...
Reviewed-by: Jeff Layton <jlayton@kernel.org> Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
show more ...
|
#
12d426ab |
| 15-Jun-2022 |
Al Viro <viro@zeniv.linux.org.uk> |
ITER_PIPE: fold data_start() and pipe_space_for_user() together
All their callers are next to each other; all of them want the total amount of pages and, possibly, the offset in the partial final bu
ITER_PIPE: fold data_start() and pipe_space_for_user() together
All their callers are next to each other; all of them want the total amount of pages and, possibly, the offset in the partial final buffer.
Combine into a new helper (pipe_npages()), fix the bogosity in pipe_space_for_user(), while we are at it.
Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
show more ...
|
#
10f525a8 |
| 15-Jun-2022 |
Al Viro <viro@zeniv.linux.org.uk> |
ITER_PIPE: cache the type of last buffer
We often need to find whether the last buffer is anon or not, and currently it's rather clumsy: check if ->iov_offset is non-zero (i.e. that pipe is not emp
ITER_PIPE: cache the type of last buffer
We often need to find whether the last buffer is anon or not, and currently it's rather clumsy: check if ->iov_offset is non-zero (i.e. that pipe is not empty) if so, get the corresponding pipe_buffer and check its ->ops if it's &default_pipe_buf_ops, we have an anon buffer.
Let's replace the use of ->iov_offset (which is nowhere near similar to its role for other flavours) with signed field (->last_offset), with the following rules: empty, no buffers occupied: 0 anon, with bytes up to N-1 filled: N zero-copy, with bytes up to N-1 filled: -N
That way abs(i->last_offset) is equal to what used to be in i->iov_offset and empty vs. anon vs. zero-copy can be distinguished by the sign of i->last_offset.
Checks for "should we extend the last buffer or should we start a new one?" become easier to follow that way.
Note that most of the operations can only be done in a sane state - i.e. when the pipe has nothing past the current position of iterator. About the only thing that could be done outside of that state is iov_iter_advance(), which transitions to the sane state by truncating the pipe. There are only two cases where we leave the sane state: 1) iov_iter_get_pages()/iov_iter_get_pages_alloc(). Will be dealt with later, when we make get_pages advancing - the callers are actually happier that way. 2) iov_iter copied, then something is put into the copy. Since they share the underlying pipe, the original gets behind. When we decide that we are done with the copy (original is not usable until then) we advance the original. direct_io used to be done that way; nowadays it operates on the original and we do iov_iter_revert() to discard the excessive data. At the moment there's nothing in the kernel that could do that to ITER_PIPE iterators, so this reason for insane state is theoretical right now.
Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
show more ...
|
#
92acdc4f |
| 12-Jun-2022 |
Al Viro <viro@zeniv.linux.org.uk> |
ITER_PIPE: clean iov_iter_revert()
Fold pipe_truncate() into it, clean up. We can release buffers in the same loop where we walk backwards to the iterator beginning looking for the place where the
ITER_PIPE: clean iov_iter_revert()
Fold pipe_truncate() into it, clean up. We can release buffers in the same loop where we walk backwards to the iterator beginning looking for the place where the new position will be.
Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
show more ...
|
#
2c855de9 |
| 15-Jun-2022 |
Al Viro <viro@zeniv.linux.org.uk> |
ITER_PIPE: clean pipe_advance() up
instead of setting ->iov_offset for new position and calling pipe_truncate() to adjust ->len of the last buffer and discard everything after it, adjust ->len at th
ITER_PIPE: clean pipe_advance() up
instead of setting ->iov_offset for new position and calling pipe_truncate() to adjust ->len of the last buffer and discard everything after it, adjust ->len at the same time we set ->iov_offset and use pipe_discard_from() to deal with buffers past that.
Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
show more ...
|
#
ca591967 |
| 16-Jun-2022 |
Al Viro <viro@zeniv.linux.org.uk> |
ITER_PIPE: lose iter_head argument of __pipe_get_pages()
it's only used to get to the partial buffer we can add to, and that's always the last one, i.e. pipe->head - 1.
Signed-off-by: Al Viro <viro
ITER_PIPE: lose iter_head argument of __pipe_get_pages()
it's only used to get to the partial buffer we can add to, and that's always the last one, i.e. pipe->head - 1.
Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
show more ...
|
#
e3b42964 |
| 11-Jun-2022 |
Al Viro <viro@zeniv.linux.org.uk> |
ITER_PIPE: fold push_pipe() into __pipe_get_pages()
Expand the only remaining call of push_pipe() (in __pipe_get_pages()), combine it with the page-collecting loop there.
Note that the only reason
ITER_PIPE: fold push_pipe() into __pipe_get_pages()
Expand the only remaining call of push_pipe() (in __pipe_get_pages()), combine it with the page-collecting loop there.
Note that the only reason it's not a loop doing append_pipe() is that append_pipe() is advancing, while iov_iter_get_pages() is not. As soon as it switches to saner semantics, this thing will switch to using append_pipe().
Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
show more ...
|
#
8fad7767 |
| 14-Jun-2022 |
Al Viro <viro@zeniv.linux.org.uk> |
ITER_PIPE: allocate buffers as we go in copy-to-pipe primitives
New helper: append_pipe(). Extends the last buffer if possible, allocates a new one otherwise. Returns page and offset in it on succ
ITER_PIPE: allocate buffers as we go in copy-to-pipe primitives
New helper: append_pipe(). Extends the last buffer if possible, allocates a new one otherwise. Returns page and offset in it on success, NULL on failure. iov_iter is advanced past the data we've got.
Use that instead of push_pipe() in copy-to-pipe primitives; they get simpler that way. Handling of short copy (in "mc" one) is done simply by iov_iter_revert() - iov_iter is in consistent state after that one, so we can use that.
[Fix for braino caught by Liu Xinpeng <liuxp11@chinatelecom.cn> folded in] [another braino fix, this time in copy_pipe_to_iter() and pipe_zero(); caught by testcase from Hugh Dickins]
Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
show more ...
|
#
47b7fcae |
| 13-Jun-2022 |
Al Viro <viro@zeniv.linux.org.uk> |
ITER_PIPE: helpers for adding pipe buffers
There are only two kinds of pipe_buffer in the area used by ITER_PIPE.
1) anonymous - copy_to_iter() et.al. end up creating those and copying data there.
ITER_PIPE: helpers for adding pipe buffers
There are only two kinds of pipe_buffer in the area used by ITER_PIPE.
1) anonymous - copy_to_iter() et.al. end up creating those and copying data there. They have zero ->offset, and their ->ops points to default_pipe_page_ops.
2) zero-copy ones - those come from copy_page_to_iter(), and page comes from caller. ->offset is also caller-supplied - it might be non-zero. ->ops points to page_cache_pipe_buf_ops.
Move creation and insertion of those into helpers - push_anon(pipe, size) and push_page(pipe, page, offset, size) resp., separating them from the "could we avoid creating a new buffer by merging with the current head?" logics.
Acked-by: Jeff Layton <jlayton@kernel.org> Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
show more ...
|
#
2dcedb2a |
| 14-Jun-2022 |
Al Viro <viro@zeniv.linux.org.uk> |
ITER_PIPE: helper for getting pipe buffer by index
pipe_buffer instances of a pipe are organized as a ring buffer, with power-of-2 size. Indices are kept *not* reduced modulo ring size, so the buff
ITER_PIPE: helper for getting pipe buffer by index
pipe_buffer instances of a pipe are organized as a ring buffer, with power-of-2 size. Indices are kept *not* reduced modulo ring size, so the buffer refered to by index N is pipe->bufs[N & (pipe->ring_size - 1)].
Ring size can change over the lifetime of a pipe, but not while the pipe is locked. So for any iov_iter primitives it's a constant. Original conversion of pipes to this layout went overboard trying to microoptimize that - calculating pipe->ring_size - 1, storing it in a local variable and using through the function. In some cases it might be warranted, but most of the times it only obfuscates what's going on in there.
Introduce a helper (pipe_buf(pipe, N)) that would encapsulate that and use it in the obvious cases. More will follow...
Reviewed-by: Jeff Layton <jlayton@kernel.org> Reviewed-by: Christian Brauner (Microsoft) <brauner@kernel.org> Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
show more ...
|
Revision tags: v5.15.46, v5.15.45, v5.15.44, v5.15.43, v5.15.42, v5.18 |
|
#
fcb14cb1 |
| 22-May-2022 |
Al Viro <viro@zeniv.linux.org.uk> |
new iov_iter flavour - ITER_UBUF
Equivalent of single-segment iovec. Initialized by iov_iter_ubuf(), checked for by iter_is_ubuf(), otherwise behaves like ITER_IOVEC ones.
We are going to expose t
new iov_iter flavour - ITER_UBUF
Equivalent of single-segment iovec. Initialized by iov_iter_ubuf(), checked for by iter_is_ubuf(), otherwise behaves like ITER_IOVEC ones.
We are going to expose the things like ->write_iter() et.al. to those in subsequent commits.
New predicate (user_backed_iter()) that is true for ITER_IOVEC and ITER_UBUF; places like direct-IO handling should use that for checking that pages we modify after getting them from iov_iter_get_pages() would need to be dirtied.
DO NOT assume that replacing iter_is_iovec() with user_backed_iter() will solve all problems - there's code that uses iter_is_iovec() to decide how to poke around in iov_iter guts and for that the predicate replacement obviously won't suffice.
Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
show more ...
|
#
dd45ab9d |
| 17-Jun-2022 |
Al Viro <viro@zeniv.linux.org.uk> |
first_iovec_segment(): just return address
... and calculate the offset in the caller
Reviewed-by: Jeff Layton <jlayton@kernel.org> Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
|
#
59dbd7d0 |
| 21-Jun-2022 |
Al Viro <viro@zeniv.linux.org.uk> |
iov_iter: massage calling conventions for first_{iovec,bvec}_segment()
Pass maxsize by reference, return length via the same.
Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
|