Pavel Begunkov 8751d15426 io_uring: reduce scheduling due to tw ... Every task_work will try to wake the task to be executed, which causes excessive scheduling and additional overhead. For some tw it's justified, but others won't do much but post a single CQE. When a task waits for multiple cqes, every such task_work will wake it up. Instead, the task may give a hint about how many cqes it waits for, io_req_local_work_add() will compare against it and skip wake ups if #cqes + #tw is not enough to satisfy the waiting condition. Task_work that uses the optimisation should be simple enough and never post more than one CQE. It's also ignored for non DEFER_TASKRUN rings. Signed-off-by: Pavel Begunkov <asml.silence@gmail.com> Link: https://lore.kernel.org/r/d2b77e99d1e86624d8a69f7037d764b739dcd225.1680782017.git.asml.silence@gmail.com Signed-off-by: Jens Axboe <axboe@kernel.dk>		2023-04-06 16:23:28 -06:00
..
advise.c	io_uring: always go async for unsupported fadvise flags	2023-01-29 15:18:26 -07:00
advise.h	io_uring: split out fadvise/madvise operations	2022-07-24 18:39:11 -06:00
alloc_cache.h	io_uring/rsrc: add custom limit for node caching	2023-04-04 09:30:39 -06:00
cancel.c	io_uring/cancel: re-grab ctx mutex after finishing wait	2022-12-21 13:31:40 -07:00
cancel.h	io_uring: add sync cancelation API through io_uring_register()	2022-07-24 18:39:15 -06:00
epoll.c	io_uring: make io_kiocb_to_cmd() typesafe	2022-08-12 17:01:00 -06:00
epoll.h	io_uring: move epoll handler to its own file	2022-07-24 18:39:11 -06:00
fdinfo.c	capability: just use a 'u64' instead of a 'u32[2]' array	2023-03-01 10:01:22 -08:00
fdinfo.h	io_uring: move fdinfo helpers to its own file	2022-07-24 18:39:12 -06:00
filetable.c	io_uring/rsrc: fix null-ptr-deref in io_file_bitmap_get()	2023-03-22 11:04:55 -06:00
filetable.h	io_uring: kill hot path fixed file bitmap debug checks	2022-10-16 17:07:53 -06:00
fs.c	io_uring: for requests that require async, force it	2023-01-29 15:18:26 -07:00
fs.h	io_uring: split out filesystem related operations	2022-07-24 18:39:11 -06:00
io_uring.c	io_uring: reduce scheduling due to tw	2023-04-06 16:23:28 -06:00
io_uring.h	io_uring: reduce scheduling due to tw	2023-04-06 16:23:28 -06:00
io-wq.c	io_uring/io-wq: drop outdated comment	2023-04-03 07:16:15 -06:00
io-wq.h	io_uring: move list helpers to a separate file	2022-07-24 18:39:15 -06:00
kbuf.c	io_uring/kbuf: disallow mapping a badly aligned provided ring buffer	2023-04-03 07:16:14 -06:00
kbuf.h	io_uring: add support for user mapped provided buffer ring	2023-04-03 07:14:21 -06:00
Makefile	io_uring: add zc notification infrastructure	2022-07-24 18:41:06 -06:00
msg_ring.c	io_uring/msg_ring: let target know allocated index	2023-03-16 07:16:56 -06:00
msg_ring.h	io_uring: get rid of double locking	2022-12-07 06:47:13 -07:00
net.c	io_uring/net: avoid sending -ECONNABORTED on repeated connection requests	2023-03-20 20:44:45 -06:00
net.h	io_uring: Add KASAN support for alloc_caches	2023-04-03 07:16:14 -06:00
nop.c	io_uring: kill extra io_uring_types.h includes	2022-07-24 18:39:14 -06:00
nop.h	io_uring: move nop into its own file	2022-07-24 18:39:11 -06:00
notif.c	io_uring: reduce scheduling due to tw	2023-04-06 16:23:28 -06:00
notif.h	io_uring: reduce scheduling due to tw	2023-04-06 16:23:28 -06:00
opdef.c	io_uring,audit: don't log IORING_OP_MADVISE	2023-02-10 16:00:30 -07:00
opdef.h	io_uring: Split io_issue_def struct	2023-01-29 15:17:41 -07:00
openclose.c	io_uring: always go async for unsupported open flags	2023-01-29 15:18:26 -07:00
openclose.h	io_uring: split out fixed file installation and removal	2022-07-24 18:39:16 -06:00
poll.c	io_uring: encapsulate task_work state	2023-04-03 07:16:15 -06:00
poll.h	io_uring/poll: allow some retries for poll triggering spuriously	2023-02-25 20:10:13 -07:00
refs.h	io_uring: make io_uring_types.h public	2022-07-24 18:39:14 -06:00
rsrc.c	io_uring/rsrc: optimise io_rsrc_data refcounting	2023-04-04 09:30:39 -06:00
rsrc.h	io_uring/rsrc: add custom limit for node caching	2023-04-04 09:30:39 -06:00
rw.c	io_uring: reduce scheduling due to tw	2023-04-06 16:23:28 -06:00
rw.h	io_uring/rw: don't lose partial IO result on fail	2022-09-21 13:15:02 -06:00
slist.h	io_uring: silence variable ‘prev’ set but not used warning	2023-03-09 10:10:58 -07:00
splice.c	io_uring: for requests that require async, force it	2023-01-29 15:18:26 -07:00
splice.h	io_uring: split out splice related operations	2022-07-24 18:39:11 -06:00
sqpoll.c	io_uring/sqpoll: Do not set PF_NO_SETAFFINITY on sqpoll threads	2023-03-15 06:50:59 -06:00
sqpoll.h	io_uring: make io_sqpoll_wait_sq return void	2023-01-29 15:17:40 -07:00
statx.c	io_uring: for requests that require async, force it	2023-01-29 15:18:26 -07:00
statx.h	io_uring: move statx handling to its own file	2022-07-24 18:39:11 -06:00
sync.c	io_uring: for requests that require async, force it	2023-01-29 15:18:26 -07:00
sync.h	io_uring: split out fs related sync/fallocate functions	2022-07-24 18:39:11 -06:00
tctx.c	io_uring: rename 'in_idle' to 'in_cancel'	2023-02-22 09:57:23 -07:00
tctx.h	io_uring: simplify __io_uring_add_tctx_node	2022-10-07 12:25:30 -06:00
timeout.c	io_uring: encapsulate task_work state	2023-04-03 07:16:15 -06:00
timeout.h	io_uring: remove unused return from io_disarm_next	2022-09-21 13:15:01 -06:00
uring_cmd.c	io_uring/uring_cmd: assign ioucmd->cmd at async prep time	2023-04-05 09:30:18 -06:00
uring_cmd.h	io_uring: move uring_cmd handling to its own file	2022-07-24 18:39:11 -06:00
xattr.c	io_uring: for requests that require async, force it	2023-01-29 15:18:26 -07:00
xattr.h	io_uring: move xattr related opcodes to its own file	2022-07-24 18:39:11 -06:00