Kernel/linux
1
0
Fork 0
mirror of https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git synced 2025-01-06 05:13:18 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity
ec26c225f0
linux/io_uring
History
Pavel Begunkov ec26c225f0 io_uring: merge iopoll and normal completion paths 
io_do_iopoll() and io_submit_flush_completions() are pretty similar,
both filling CQEs and then free a list of requests. Don't duplicate it
and make iopoll use __io_submit_flush_completions(), which also helps
with inlining and other optimisations.

For that, we need to first find all completed iopoll requests and splice
them from the iopoll list and then pass it down. This adds one extra
list traversal, which should be fine as requests will stay hot in cache.

CQ locking is already conditional, introduce ->lockless_cq and skip
locking for IOPOLL as it's protected by ->uring_lock.

We also add a wakeup optimisation for IOPOLL to __io_cq_unlock_post(),
so it works just like io_cqring_ev_posted_iopoll().

Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://lore.kernel.org/r/3840473f5e8a960de35b77292026691880f6bdbc.1692916914.git.asml.silence@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2023-08-24 17:16:19 -06:00
..
advise.c io_uring: always go async for unsupported fadvise flags 2023-01-29 15:18:26 -07:00
advise.h io_uring: split out fadvise/madvise operations 2022-07-24 18:39:11 -06:00
alloc_cache.h io_uring/rsrc: consolidate node caching 2023-04-12 12:09:41 -06:00
cancel.c io_uring/cancel: wire up IORING_ASYNC_CANCEL_OP for sync cancel 2023-07-17 10:05:48 -06:00
cancel.h io_uring/cancel: support opcode based lookup and cancelation 2023-07-17 10:05:48 -06:00
epoll.c io_uring: undeprecate epoll_ctl support 2023-05-26 20:22:41 -06:00
epoll.h io_uring: move epoll handler to its own file 2022-07-24 18:39:11 -06:00
fdinfo.c io_uring/fdinfo: get rid of ref tryget 2023-08-10 10:24:19 -06:00
fdinfo.h io_uring: move fdinfo helpers to its own file 2022-07-24 18:39:12 -06:00
filetable.c io_uring: add helpers to decode the fixed file file_ptr 2023-06-20 09:36:22 -06:00
filetable.h io_uring: add helpers to decode the fixed file file_ptr 2023-06-20 09:36:22 -06:00
fs.c io_uring: for requests that require async, force it 2023-01-29 15:18:26 -07:00
fs.h io_uring: split out filesystem related operations 2022-07-24 18:39:11 -06:00
io_uring.c io_uring: merge iopoll and normal completion paths 2023-08-24 17:16:19 -06:00
io_uring.h io_uring: merge iopoll and normal completion paths 2023-08-24 17:16:19 -06:00
io-wq.c io_uring/sqpoll: fix io-wq affinity when IORING_SETUP_SQPOLL is used 2023-08-16 13:40:28 -06:00
io-wq.h io_uring/sqpoll: fix io-wq affinity when IORING_SETUP_SQPOLL is used 2023-08-16 13:40:28 -06:00
kbuf.c for-6.4/io_uring-2023-04-21 2023-04-26 12:40:31 -07:00
kbuf.h io_uring: add support for user mapped provided buffer ring 2023-04-03 07:14:21 -06:00
Makefile io_uring: add zc notification infrastructure 2022-07-24 18:41:06 -06:00
msg_ring.c io_uring: use io_file_from_index in io_msg_grab_file 2023-06-20 09:36:22 -06:00
msg_ring.h io_uring: get rid of double locking 2022-12-07 06:47:13 -07:00
net.c io_uring: never overflow io_aux_cqe 2023-08-11 10:42:57 -06:00
net.h io_uring: Add KASAN support for alloc_caches 2023-04-03 07:16:14 -06:00
nop.c io_uring: kill extra io_uring_types.h includes 2022-07-24 18:39:14 -06:00
nop.h io_uring: move nop into its own file 2022-07-24 18:39:11 -06:00
notif.c io_uring/notif: add constant for ubuf_info flags 2023-04-15 14:21:04 -06:00
notif.h io_uring/notif: add constant for ubuf_info flags 2023-04-15 14:21:04 -06:00
opdef.c io_uring: Pass whole sqe to commands 2023-05-04 08:19:05 -06:00
opdef.h io_uring: Split io_issue_def struct 2023-01-29 15:17:41 -07:00
openclose.c fsnotify: move fsnotify_open() hook into do_dentry_open() 2023-06-12 10:43:45 +02:00
openclose.h io_uring: split out fixed file installation and removal 2022-07-24 18:39:16 -06:00
poll.c io_uring: never overflow io_aux_cqe 2023-08-11 10:42:57 -06:00
poll.h io_uring: avoid indirect function calls for the hottest task_work 2023-06-02 08:55:37 -06:00
refs.h io_uring: make io_uring_types.h public 2022-07-24 18:39:14 -06:00
rsrc.c io_uring/rsrc: keep one global dummy_ubuf 2023-08-11 10:42:57 -06:00
rsrc.h io_uring/rsrc: Annotate struct io_mapped_ubuf with __counted_by 2023-08-17 19:14:47 -06:00
rw.c io_uring: merge iopoll and normal completion paths 2023-08-24 17:16:19 -06:00
rw.h io_uring: avoid indirect function calls for the hottest task_work 2023-06-02 08:55:37 -06:00
slist.h io_uring: silence variable ‘prev’ set but not used warning 2023-03-09 10:10:58 -07:00
splice.c io_uring/splice: use fput() directly 2023-08-10 10:24:25 -06:00
splice.h io_uring: split out splice related operations 2022-07-24 18:39:11 -06:00
sqpoll.c io_uring/sqpoll: fix io-wq affinity when IORING_SETUP_SQPOLL is used 2023-08-16 13:40:28 -06:00
sqpoll.h io_uring/sqpoll: fix io-wq affinity when IORING_SETUP_SQPOLL is used 2023-08-16 13:40:28 -06:00
statx.c io_uring: for requests that require async, force it 2023-01-29 15:18:26 -07:00
statx.h io_uring: move statx handling to its own file 2022-07-24 18:39:11 -06:00
sync.c io_uring: for requests that require async, force it 2023-01-29 15:18:26 -07:00
sync.h io_uring: split out fs related sync/fallocate functions 2022-07-24 18:39:11 -06:00
tctx.c io_uring: Add io_uring_setup flag to pre-register ring fd and never install it 2023-05-16 08:06:00 -06:00
tctx.h io_uring: simplify __io_uring_add_tctx_node 2022-10-07 12:25:30 -06:00
timeout.c io_uring: never overflow io_aux_cqe 2023-08-11 10:42:57 -06:00
timeout.h io_uring: remove unused return from io_disarm_next 2022-09-21 13:15:01 -06:00
uring_cmd.c io_uring: simplify big_cqe handling 2023-08-24 17:16:19 -06:00
uring_cmd.h io_uring: Remove unnecessary BUILD_BUG_ON 2023-05-04 08:19:05 -06:00
xattr.c io_uring: for requests that require async, force it 2023-01-29 15:18:26 -07:00
xattr.h io_uring: move xattr related opcodes to its own file 2022-07-24 18:39:11 -06:00