This patchset allows for io_uring zerocopy to support REQ_F_CQE_SKIP, skipping the normal completion notification, but not the zerocopy buffer release notification.
This patchset also includes a test to test these changes, and a patch to mini_liburing to enable io_uring_peek_cqe, which is needed for the test.
Oliver Crumrine (3): io_uring: Add REQ_F_CQE_SKIP support for io_uring zerocopy io_uring: Add io_uring_peek_cqe to mini_liburing io_uring: Support IOSQE_CQE_SKIP_SUCCESS in io_uring zerocopy test
io_uring/net.c | 6 +-- tools/include/io_uring/mini_liburing.h | 18 +++++++++ .../selftests/net/io_uring_zerocopy_tx.c | 37 +++++++++++++++++-- .../selftests/net/io_uring_zerocopy_tx.sh | 7 +++- 4 files changed, 59 insertions(+), 10 deletions(-)
In his patch to enable zerocopy networking for io_uring, Pavel Begunkov specifically disabled REQ_F_CQE_SKIP, as (at least from my understanding) the userspace program wouldn't receive the IORING_CQE_F_MORE flag in the result value.
To fix this, instead of keeping track of how many CQEs have been received, and subtracting notifs from that, programs can keep track of how many SQEs they have issued, and if a CQE is returned with an error, they can simply subtract from how many notifs they expect to receive.
Signed-off-by: Oliver Crumrine ozlinuxc@gmail.com --- io_uring/net.c | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-)
diff --git a/io_uring/net.c b/io_uring/net.c index 1e7665ff6ef7..822f49809b68 100644 --- a/io_uring/net.c +++ b/io_uring/net.c @@ -1044,9 +1044,6 @@ int io_send_zc_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe)
if (unlikely(READ_ONCE(sqe->__pad2[0]) || READ_ONCE(sqe->addr3))) return -EINVAL; - /* we don't support IOSQE_CQE_SKIP_SUCCESS just yet */ - if (req->flags & REQ_F_CQE_SKIP) - return -EINVAL;
notif = zc->notif = io_alloc_notif(ctx); if (!notif) @@ -1342,7 +1339,8 @@ void io_sendrecv_fail(struct io_kiocb *req) req->cqe.res = sr->done_io;
if ((req->flags & REQ_F_NEED_CLEANUP) && - (req->opcode == IORING_OP_SEND_ZC || req->opcode == IORING_OP_SENDMSG_ZC)) + (req->opcode == IORING_OP_SEND_ZC || req->opcode == IORING_OP_SENDMSG_ZC) && + !(req->flags & REQ_F_CQE_SKIP)) req->cqe.flags |= IORING_CQE_F_MORE; }
On 4/4/24 23:17, Oliver Crumrine wrote:
In his patch to enable zerocopy networking for io_uring, Pavel Begunkov specifically disabled REQ_F_CQE_SKIP, as (at least from my understanding) the userspace program wouldn't receive the IORING_CQE_F_MORE flag in the result value.
No. IORING_CQE_F_MORE means there will be another CQE from this request, so a single CQE without IORING_CQE_F_MORE is trivially fine.
The problem is the semantics, because by suppressing the first CQE you're loosing the result value. You might rely on WAITALL as other sends and "fail" (in terms of io_uring) the request in case of a partial send posting 2 CQEs, but that's not a great way and it's getting userspace complicated pretty easily.
In short, it was left out for later because there is a better way to implement it, but it should be done carefully
To fix this, instead of keeping track of how many CQEs have been received, and subtracting notifs from that, programs can keep track of
That's a benchmark way of doing it, more realistically it'd be more like
event_loop() { cqe = wait_cqe(); struct req *r = (struct req *)cqe->user_data; r->callback(r, cqe); }
send_zc_callback(req, cqe) { if (cqe->flags & F_MORE) { // don't free the req // we should wait for another CQE ... } }
how many SQEs they have issued, and if a CQE is returned with an error, they can simply subtract from how many notifs they expect to receive.
The design specifically untangles those two notions, i.e. there can be a notification even when the main CQE fails (ret<0). It's safer this way, even though AFAIK relying on errors would be fine with current users (TCP/UDP).
Signed-off-by: Oliver Crumrine ozlinuxc@gmail.com
io_uring/net.c | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-)
diff --git a/io_uring/net.c b/io_uring/net.c index 1e7665ff6ef7..822f49809b68 100644 --- a/io_uring/net.c +++ b/io_uring/net.c @@ -1044,9 +1044,6 @@ int io_send_zc_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe) if (unlikely(READ_ONCE(sqe->__pad2[0]) || READ_ONCE(sqe->addr3))) return -EINVAL;
- /* we don't support IOSQE_CQE_SKIP_SUCCESS just yet */
- if (req->flags & REQ_F_CQE_SKIP)
return -EINVAL;notif = zc->notif = io_alloc_notif(ctx); if (!notif) @@ -1342,7 +1339,8 @@ void io_sendrecv_fail(struct io_kiocb *req) req->cqe.res = sr->done_io; if ((req->flags & REQ_F_NEED_CLEANUP) &&
(req->opcode == IORING_OP_SEND_ZC || req->opcode == IORING_OP_SENDMSG_ZC))
(req->opcode == IORING_OP_SEND_ZC || req->opcode == IORING_OP_SENDMSG_ZC) &&!(req->flags & REQ_F_CQE_SKIP))
Pavel Begunkov wrote:
On 4/4/24 23:17, Oliver Crumrine wrote:
In his patch to enable zerocopy networking for io_uring, Pavel Begunkov specifically disabled REQ_F_CQE_SKIP, as (at least from my understanding) the userspace program wouldn't receive the IORING_CQE_F_MORE flag in the result value.
No. IORING_CQE_F_MORE means there will be another CQE from this request, so a single CQE without IORING_CQE_F_MORE is trivially fine.
The problem is the semantics, because by suppressing the first CQE you're loosing the result value. You might rely on WAITALL
That's already happening with io_send.
as other sends and "fail" (in terms of io_uring) the request in case of a partial send posting 2 CQEs, but that's not a great way and it's getting userspace complicated pretty easily.
In short, it was left out for later because there is a better way to implement it, but it should be done carefully
Maybe we could put the return values in the notifs? That would be a discrepancy between io_send and io_send_zc, though.
To fix this, instead of keeping track of how many CQEs have been received, and subtracting notifs from that, programs can keep track of
That's a benchmark way of doing it, more realistically it'd be more like
event_loop() { cqe = wait_cqe(); struct req *r = (struct req *)cqe->user_data; r->callback(r, cqe);
send_zc_callback(req, cqe) { if (cqe->flags & F_MORE) { // don't free the req // we should wait for another CQE ... } }
how many SQEs they have issued, and if a CQE is returned with an error, they can simply subtract from how many notifs they expect to receive.
The design specifically untangles those two notions, i.e. there can be a notification even when the main CQE fails (ret<0). It's safer this way, even though AFAIK relying on errors would be fine with current users (TCP/UDP).
Signed-off-by: Oliver Crumrine ozlinuxc@gmail.com
io_uring/net.c | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-)
diff --git a/io_uring/net.c b/io_uring/net.c index 1e7665ff6ef7..822f49809b68 100644 --- a/io_uring/net.c +++ b/io_uring/net.c @@ -1044,9 +1044,6 @@ int io_send_zc_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe)
if (unlikely(READ_ONCE(sqe->__pad2[0]) || READ_ONCE(sqe->addr3))) return -EINVAL;
/* we don't support IOSQE_CQE_SKIP_SUCCESS just yet */
if (req->flags & REQ_F_CQE_SKIP)
return -EINVAL;notif = zc->notif = io_alloc_notif(ctx); if (!notif)
@@ -1342,7 +1339,8 @@ void io_sendrecv_fail(struct io_kiocb *req) req->cqe.res = sr->done_io;
if ((req->flags & REQ_F_NEED_CLEANUP) &&
(req->opcode == IORING_OP_SEND_ZC || req->opcode == IORING_OP_SENDMSG_ZC))
(req->opcode == IORING_OP_SEND_ZC || req->opcode == IORING_OP_SENDMSG_ZC) &&!(req->flags & REQ_F_CQE_SKIP))-- Pavel Begunkov
On 4/5/24 21:04, Oliver Crumrine wrote:
Pavel Begunkov wrote:
On 4/4/24 23:17, Oliver Crumrine wrote:
In his patch to enable zerocopy networking for io_uring, Pavel Begunkov specifically disabled REQ_F_CQE_SKIP, as (at least from my understanding) the userspace program wouldn't receive the IORING_CQE_F_MORE flag in the result value.
No. IORING_CQE_F_MORE means there will be another CQE from this request, so a single CQE without IORING_CQE_F_MORE is trivially fine.
The problem is the semantics, because by suppressing the first CQE you're loosing the result value. You might rely on WAITALL
That's already happening with io_send.
Right, and it's still annoying and hard to use
as other sends and "fail" (in terms of io_uring) the request in case of a partial send posting 2 CQEs, but that's not a great way and it's getting userspace complicated pretty easily.
In short, it was left out for later because there is a better way to implement it, but it should be done carefully
Maybe we could put the return values in the notifs? That would be a discrepancy between io_send and io_send_zc, though.
Yes. And yes, having a custom flavour is not good. It'd only be well usable if apart from returning the actual result it also guarantees there will be one and only one CQE, then the userspace doesn't have to do the dancing with counting and checking F_MORE. In fact, I outlined before how a generic solution may looks like:
https://github.com/axboe/liburing/issues/824
The only interesting part, IMHO, is to be able to merge the main completion with its notification. Below is an old stash rebased onto for-6.10. The only thing missing is relinking, but maybe we don't even care about it. I need to cover it well with tests.
commit ca5e4fb6d105b5dfdf3768d46ce01529b7bb88c5 Author: Pavel Begunkov asml.silence@gmail.com Date: Sat Apr 6 15:46:38 2024 +0100
io_uring/net: introduce single CQE send zc mode
IORING_OP_SEND[MSG]_ZC requests are posting two completions, one to notify that the data was queued, and later a second, usually referred as "notification", to let the user know that the buffer used can be reused/freed. In some cases the user might not care about the main completion and would be content getting only the notification, which would allow to simplify the userspace.
One example is when after a send the user would be waiting for the other end to get the message and reply back not pushing any more data in the meantime. Another case is unreliable protocols like UDP, which do not require a confirmation from the other end before dropping buffers, and so the notifications are usually posted shortly after the send request is queued.
Add a flag merging completions into a single CQE. cqe->res will store the send's result as usual, and it will have IORING_CQE_F_NOTIF set if the buffer was potentially used. Timewise, it would be posted at the moment when the notification would have been originally completed.
Signed-off-by: Pavel Begunkov asml.silence@gmail.com
diff --git a/include/uapi/linux/io_uring.h b/include/uapi/linux/io_uring.h index 7bd10201a02b..e2b528c341c9 100644 --- a/include/uapi/linux/io_uring.h +++ b/include/uapi/linux/io_uring.h @@ -356,6 +356,7 @@ enum io_uring_op { #define IORING_RECV_MULTISHOT (1U << 1) #define IORING_RECVSEND_FIXED_BUF (1U << 2) #define IORING_SEND_ZC_REPORT_USAGE (1U << 3) +#define IORING_SEND_ZC_COMBINE_CQE (1U << 4)
/* * cqe.res for IORING_CQE_F_NOTIF if diff --git a/io_uring/net.c b/io_uring/net.c index a74287692071..052f030ab8f8 100644 --- a/io_uring/net.c +++ b/io_uring/net.c @@ -992,7 +992,19 @@ void io_send_zc_cleanup(struct io_kiocb *req) } }
-#define IO_ZC_FLAGS_COMMON (IORING_RECVSEND_POLL_FIRST | IORING_RECVSEND_FIXED_BUF) +static inline void io_sendzc_adjust_res(struct io_kiocb *req) +{ + struct io_sr_msg *sr = io_kiocb_to_cmd(req, struct io_sr_msg); + + if (sr->flags & IORING_SEND_ZC_COMBINE_CQE) { + sr->notif->cqe.res = req->cqe.res; + req->flags |= REQ_F_CQE_SKIP; + } +} + +#define IO_ZC_FLAGS_COMMON (IORING_RECVSEND_POLL_FIRST | \ + IORING_RECVSEND_FIXED_BUF | \ + IORING_SEND_ZC_COMBINE_CQE) #define IO_ZC_FLAGS_VALID (IO_ZC_FLAGS_COMMON | IORING_SEND_ZC_REPORT_USAGE)
int io_send_zc_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe) @@ -1022,6 +1034,8 @@ int io_send_zc_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe) if (zc->flags & ~IO_ZC_FLAGS_VALID) return -EINVAL; if (zc->flags & IORING_SEND_ZC_REPORT_USAGE) { + if (zc->flags & IORING_SEND_ZC_COMBINE_CQE) + return -EINVAL; io_notif_set_extended(notif); io_notif_to_data(notif)->zc_report = true; } @@ -1197,6 +1211,9 @@ int io_send_zc(struct io_kiocb *req, unsigned int issue_flags) else if (zc->done_io) ret = zc->done_io;
+ io_req_set_res(req, ret, IORING_CQE_F_MORE); + io_sendzc_adjust_res(req); + /* * If we're in io-wq we can't rely on tw ordering guarantees, defer * flushing notif to io_send_zc_cleanup() @@ -1205,7 +1222,6 @@ int io_send_zc(struct io_kiocb *req, unsigned int issue_flags) io_notif_flush(zc->notif); io_req_msg_cleanup(req, 0); } - io_req_set_res(req, ret, IORING_CQE_F_MORE); return IOU_OK; }
@@ -1258,6 +1274,9 @@ int io_sendmsg_zc(struct io_kiocb *req, unsigned int issue_flags) else if (sr->done_io) ret = sr->done_io;
+ io_req_set_res(req, ret, IORING_CQE_F_MORE); + io_sendzc_adjust_res(req); + /* * If we're in io-wq we can't rely on tw ordering guarantees, defer * flushing notif to io_send_zc_cleanup() @@ -1266,7 +1285,6 @@ int io_sendmsg_zc(struct io_kiocb *req, unsigned int issue_flags) io_notif_flush(sr->notif); io_req_msg_cleanup(req, 0); } - io_req_set_res(req, ret, IORING_CQE_F_MORE); return IOU_OK; }
@@ -1278,8 +1296,10 @@ void io_sendrecv_fail(struct io_kiocb *req) req->cqe.res = sr->done_io;
if ((req->flags & REQ_F_NEED_CLEANUP) && - (req->opcode == IORING_OP_SEND_ZC || req->opcode == IORING_OP_SENDMSG_ZC)) + (req->opcode == IORING_OP_SEND_ZC || req->opcode == IORING_OP_SENDMSG_ZC)) { req->cqe.flags |= IORING_CQE_F_MORE; + io_sendzc_adjust_res(req); + } }
int io_accept_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe)
mini_liburing.h is used by the io_uring tests as a stand-in for liburing. The next patch in this series needs the io_uring_peek_cqe function, and I speculate that future tests will too, so add that here.
Please scrutinize this code, I am somewhat new to io_uring, and whilst I have tested this and know it to work, I very well could've made a mistake somewhere without noticing.
Signed-off-by: Oliver Crumrine ozlinuxc@gmail.com --- tools/include/io_uring/mini_liburing.h | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+)
diff --git a/tools/include/io_uring/mini_liburing.h b/tools/include/io_uring/mini_liburing.h index 9ccb16074eb5..94dcc7f9aa33 100644 --- a/tools/include/io_uring/mini_liburing.h +++ b/tools/include/io_uring/mini_liburing.h @@ -182,6 +182,24 @@ static inline int io_uring_wait_cqe(struct io_uring *ring, return 0; }
+static inline int io_uring_peek_cqe(struct io_uring *ring, + struct io_uring_cqe **cqe_ptr) +{ + struct io_uring_cq *cq = &ring->cq; + const unsigned int mask = *cq->kring_mask; + unsigned int head = *cq->khead; + unsigned int tail = *cq->ktail; + + *cqe_ptr = NULL; + + read_barrier(); + if (head-tail) { + *cqe_ptr = &cq->cqes[head & mask]; + return 0; + } + return -EAGAIN; +} + static inline int io_uring_submit(struct io_uring *ring) { struct io_uring_sq *sq = &ring->sq;
Add support for the IOSQE_CQE_SKIP_SUCCESS flag in the io_uring_zerocopy_tx test, using the "-a" option. Instead of incrementing when IORING_CQE_F_MORE is set, remember how many SQEs are sent and simply wait on notifs instead of regular completions. For non-zc stuff, there won't be notifs or completions, so don't wait on either of those, but check the completion queue for errors at the end to make sure none have popped up.
The changes to the shell script run the tests both with and without the "-a" option.
Signed-off-by: Oliver Crumrine ozlinuxc@gmail.com --- .../selftests/net/io_uring_zerocopy_tx.c | 38 +++++++++++++++++-- .../selftests/net/io_uring_zerocopy_tx.sh | 7 +++- 2 files changed, 39 insertions(+), 6 deletions(-)
diff --git a/tools/testing/selftests/net/io_uring_zerocopy_tx.c b/tools/testing/selftests/net/io_uring_zerocopy_tx.c index 76e604e4810e..11a43594935f 100644 --- a/tools/testing/selftests/net/io_uring_zerocopy_tx.c +++ b/tools/testing/selftests/net/io_uring_zerocopy_tx.c @@ -50,8 +50,10 @@ enum { };
static bool cfg_cork = false; +static bool cfg_nocqe = false; static int cfg_mode = MODE_ZC_FIXED; static int cfg_nr_reqs = 8; +static int cfg_nr_completions = 8; static int cfg_family = PF_UNSPEC; static int cfg_payload_len; static int cfg_port = 8000; @@ -134,11 +136,21 @@ static void do_tx(int domain, int type, int protocol) if (mode == MODE_NONZC) { io_uring_prep_send(sqe, fd, payload, cfg_payload_len, msg_flags); + if (cfg_nocqe) { + sqe->flags |= IOSQE_CQE_SKIP_SUCCESS; + cfg_nr_completions--; + } sqe->user_data = NONZC_TAG; } else { io_uring_prep_sendzc(sqe, fd, payload, cfg_payload_len, msg_flags, zc_flags); + if (cfg_nocqe) { + sqe->flags |= IOSQE_CQE_SKIP_SUCCESS; + packets++; + compl_cqes++; + bytes += cfg_payload_len; + } if (mode == MODE_ZC_FIXED) { sqe->ioprio |= IORING_RECVSEND_FIXED_BUF; sqe->buf_index = buf_idx; @@ -153,7 +165,7 @@ static void do_tx(int domain, int type, int protocol)
if (cfg_cork) do_setsockopt(fd, IPPROTO_UDP, UDP_CORK, 0); - for (i = 0; i < cfg_nr_reqs; i++) { + for (i = 0; i < cfg_nr_completions; i++) { ret = io_uring_wait_cqe(&ring, &cqe); if (ret) error(1, ret, "wait cqe"); @@ -168,7 +180,9 @@ static void do_tx(int domain, int type, int protocol) if (compl_cqes <= 0) error(1, -EINVAL, "notification mismatch"); compl_cqes--; - i--; + if (!cfg_nocqe) + i--; io_uring_cqe_seen(&ring); continue; } @@ -200,6 +214,17 @@ static void do_tx(int domain, int type, int protocol) compl_cqes--; }
+ /* The above code does not account for a send error when + * IOSQE_CQE_SKIP_SUCCESS is set. This is operating under the + * assumption that an error CQE will get put on the ring before + * the above code completes: + */ + while (!io_uring_peek_cqe(&ring, &cqe)) { + if (cqe->res == -EAGAIN) + continue; + error(1, -EINVAL, "send failed"); + } + fprintf(stderr, "tx=%lu (MB=%lu), tx/s=%lu (MB/s=%lu)\n", packets, bytes >> 20, packets / (cfg_runtime_ms / 1000), @@ -221,7 +246,7 @@ static void do_test(int domain, int type, int protocol) static void usage(const char *filepath) { error(1, 0, "Usage: %s (-4|-6) (udp|tcp) -D<dst_ip> [-s<payload size>] " - "[-t<time s>] [-n<batch>] [-p<port>] [-m<mode>]", filepath); + "[-t<time s>] [-n<batch>] [-p<port>] [-m<mode>] [-a]", filepath); }
static void parse_opts(int argc, char **argv) @@ -239,7 +264,7 @@ static void parse_opts(int argc, char **argv) usage(argv[0]); cfg_payload_len = max_payload_len;
- while ((c = getopt(argc, argv, "46D:p:s:t:n:c:m:")) != -1) { + while ((c = getopt(argc, argv, "46aD:p:s:t:n:c:m:")) != -1) { switch (c) { case '4': if (cfg_family != PF_UNSPEC) @@ -274,6 +299,9 @@ static void parse_opts(int argc, char **argv) case 'm': cfg_mode = strtol(optarg, NULL, 0); break; + case 'a': + cfg_nocqe = true; + break; } }
@@ -302,6 +330,8 @@ static void parse_opts(int argc, char **argv) error(1, 0, "-s: payload exceeds max (%d)", max_payload_len); if (optind != argc - 1) usage(argv[0]); + + cfg_nr_completions = cfg_nr_reqs; }
int main(int argc, char **argv) diff --git a/tools/testing/selftests/net/io_uring_zerocopy_tx.sh b/tools/testing/selftests/net/io_uring_zerocopy_tx.sh index 123439545013..aeb4645b7891 100755 --- a/tools/testing/selftests/net/io_uring_zerocopy_tx.sh +++ b/tools/testing/selftests/net/io_uring_zerocopy_tx.sh @@ -25,11 +25,14 @@ readonly path_sysctl_mem="net.core.optmem_max" # No arguments: automated test if [[ "$#" -eq "0" ]]; then IPs=( "4" "6" ) + SKIPCQEs=("" "-a")
for IP in "${IPs[@]}"; do for mode in $(seq 1 3); do - $0 "$IP" udp -m "$mode" -t 1 -n 32 - $0 "$IP" tcp -m "$mode" -t 1 -n 1 + for cqe in "${SKIPCQEs[@]}"; do + $0 "$IP" udp -m "$mode" -t 1 -n 32 "$cqe" + $0 "$IP" tcp -m "$mode" -t 1 -n 1 "$cqe" + done done done
On 4/5/24 3:19 AM, Oliver Crumrine wrote:
Add support for the IOSQE_CQE_SKIP_SUCCESS flag in the io_uring_zerocopy_tx test, using the "-a" option. Instead of incrementing when IORING_CQE_F_MORE is set, remember how many SQEs are sent and simply wait on notifs instead of regular completions. For non-zc stuff, there won't be notifs or completions, so don't wait on either of those, but check the completion queue for errors at the end to make sure none have popped up.
The changes to the shell script run the tests both with and without the "-a" option.
Signed-off-by: Oliver Crumrine ozlinuxc@gmail.com
Acked-by: Muhammad Usama Anjum usama.anjum@collabora.com
.../selftests/net/io_uring_zerocopy_tx.c | 38 +++++++++++++++++-- .../selftests/net/io_uring_zerocopy_tx.sh | 7 +++- 2 files changed, 39 insertions(+), 6 deletions(-)
diff --git a/tools/testing/selftests/net/io_uring_zerocopy_tx.c b/tools/testing/selftests/net/io_uring_zerocopy_tx.c index 76e604e4810e..11a43594935f 100644 --- a/tools/testing/selftests/net/io_uring_zerocopy_tx.c +++ b/tools/testing/selftests/net/io_uring_zerocopy_tx.c @@ -50,8 +50,10 @@ enum { }; static bool cfg_cork = false; +static bool cfg_nocqe = false; static int cfg_mode = MODE_ZC_FIXED; static int cfg_nr_reqs = 8; +static int cfg_nr_completions = 8; static int cfg_family = PF_UNSPEC; static int cfg_payload_len; static int cfg_port = 8000; @@ -134,11 +136,21 @@ static void do_tx(int domain, int type, int protocol) if (mode == MODE_NONZC) { io_uring_prep_send(sqe, fd, payload, cfg_payload_len, msg_flags);
if (cfg_nocqe) {sqe->flags |= IOSQE_CQE_SKIP_SUCCESS;cfg_nr_completions--;} sqe->user_data = NONZC_TAG; } else { io_uring_prep_sendzc(sqe, fd, payload, cfg_payload_len, msg_flags, zc_flags);if (cfg_nocqe) {sqe->flags |= IOSQE_CQE_SKIP_SUCCESS;packets++;compl_cqes++;bytes += cfg_payload_len;} if (mode == MODE_ZC_FIXED) { sqe->ioprio |= IORING_RECVSEND_FIXED_BUF; sqe->buf_index = buf_idx;@@ -153,7 +165,7 @@ static void do_tx(int domain, int type, int protocol) if (cfg_cork) do_setsockopt(fd, IPPROTO_UDP, UDP_CORK, 0);
for (i = 0; i < cfg_nr_reqs; i++) {
for (i = 0; i < cfg_nr_completions; i++) { ret = io_uring_wait_cqe(&ring, &cqe); if (ret) error(1, ret, "wait cqe");@@ -168,7 +180,9 @@ static void do_tx(int domain, int type, int protocol) if (compl_cqes <= 0) error(1, -EINVAL, "notification mismatch"); compl_cqes--;
i--;
if (!cfg_nocqe)i--; io_uring_cqe_seen(&ring); continue; }@@ -200,6 +214,17 @@ static void do_tx(int domain, int type, int protocol) compl_cqes--; }
- /* The above code does not account for a send error when
* IOSQE_CQE_SKIP_SUCCESS is set. This is operating under the* assumption that an error CQE will get put on the ring before* the above code completes:*/- while (!io_uring_peek_cqe(&ring, &cqe)) {
if (cqe->res == -EAGAIN)continue;error(1, -EINVAL, "send failed");- }
- fprintf(stderr, "tx=%lu (MB=%lu), tx/s=%lu (MB/s=%lu)\n", packets, bytes >> 20, packets / (cfg_runtime_ms / 1000),
@@ -221,7 +246,7 @@ static void do_test(int domain, int type, int protocol) static void usage(const char *filepath) { error(1, 0, "Usage: %s (-4|-6) (udp|tcp) -D<dst_ip> [-s<payload size>] "
"[-t<time s>] [-n<batch>] [-p<port>] [-m<mode>]", filepath);
"[-t<time s>] [-n<batch>] [-p<port>] [-m<mode>] [-a]", filepath);} static void parse_opts(int argc, char **argv) @@ -239,7 +264,7 @@ static void parse_opts(int argc, char **argv) usage(argv[0]); cfg_payload_len = max_payload_len;
- while ((c = getopt(argc, argv, "46D:p:s:t:n:c:m:")) != -1) {
- while ((c = getopt(argc, argv, "46aD:p:s:t:n:c:m:")) != -1) { switch (c) { case '4': if (cfg_family != PF_UNSPEC)
@@ -274,6 +299,9 @@ static void parse_opts(int argc, char **argv) case 'm': cfg_mode = strtol(optarg, NULL, 0); break;
case 'a':cfg_nocqe = true;break;@@ -302,6 +330,8 @@ static void parse_opts(int argc, char **argv) error(1, 0, "-s: payload exceeds max (%d)", max_payload_len); if (optind != argc - 1) usage(argv[0]);
- cfg_nr_completions = cfg_nr_reqs;
} int main(int argc, char **argv) diff --git a/tools/testing/selftests/net/io_uring_zerocopy_tx.sh b/tools/testing/selftests/net/io_uring_zerocopy_tx.sh index 123439545013..aeb4645b7891 100755 --- a/tools/testing/selftests/net/io_uring_zerocopy_tx.sh +++ b/tools/testing/selftests/net/io_uring_zerocopy_tx.sh @@ -25,11 +25,14 @@ readonly path_sysctl_mem="net.core.optmem_max" # No arguments: automated test if [[ "$#" -eq "0" ]]; then IPs=( "4" "6" )
- SKIPCQEs=("" "-a")
for IP in "${IPs[@]}"; do for mode in $(seq 1 3); do
$0 "$IP" udp -m "$mode" -t 1 -n 32$0 "$IP" tcp -m "$mode" -t 1 -n 1
for cqe in "${SKIPCQEs[@]}"; do$0 "$IP" udp -m "$mode" -t 1 -n 32 "$cqe"$0 "$IP" tcp -m "$mode" -t 1 -n 1 "$cqe"done
On 4/4/24 23:16, Oliver Crumrine wrote:
This patchset allows for io_uring zerocopy to support REQ_F_CQE_SKIP, skipping the normal completion notification, but not the zerocopy buffer release notification.
It's an io_uring internal change not altering how it operates with the net layer, you don't need to CC the net list.
This patchset also includes a test to test these changes, and a patch to mini_liburing to enable io_uring_peek_cqe, which is needed for the test.
For the same reason tests should be in liburing, where all io_uring tests are, and the selftest can be dropped. See liburing/test/send-zerocopy.c
Oliver Crumrine (3): io_uring: Add REQ_F_CQE_SKIP support for io_uring zerocopy io_uring: Add io_uring_peek_cqe to mini_liburing io_uring: Support IOSQE_CQE_SKIP_SUCCESS in io_uring zerocopy test
io_uring/net.c | 6 +-- tools/include/io_uring/mini_liburing.h | 18 +++++++++ .../selftests/net/io_uring_zerocopy_tx.c | 37 +++++++++++++++++-- .../selftests/net/io_uring_zerocopy_tx.sh | 7 +++- 4 files changed, 59 insertions(+), 10 deletions(-)
linux-kselftest-mirror@lists.linaro.org
-
Muhammad Usama Anjum -
Oliver Crumrine -
Pavel Begunkov