Add RSEQ, restartable sequence, support and related selftest to RISCV. The Kconfig option HAVE_REGS_AND_STACK_ACCESS_API is also required by RSEQ because RSEQ will modify the content of pt_regs.sepc through instruction_pointer_set() during the fixup procedure. In order to select the config HAVE_REGS_AND_STACK_ACCESS_API, the missing APIs for accessing pt_regs are also added in this patch set.
The relevant RSEQ tests in kselftest require the Binutils patch "RISC-V: Fix linker problems with TLS copy relocs" to avoid placing PREINIT_ARRAY and TLS variable of librseq.so at the same address. https://sourceware.org/git/gitweb.cgi?p=binutils-gdb.git%3Ba=commit%3Bh=3e7b... A segmental fault will happen if binutils misses this patch.
Patrick Stählin (1): riscv: add required functions to enable HAVE_REGS_AND_STACK_ACCESS_API
Vincent Chen (2): riscv: Add support for restartable sequence rseq/selftests: Add support for riscv
Changes since v1: 1. Use the correct register name to access pt_regs
arch/riscv/Kconfig | 2 + arch/riscv/include/asm/ptrace.h | 29 +- arch/riscv/kernel/entry.S | 4 + arch/riscv/kernel/ptrace.c | 99 +++++ arch/riscv/kernel/signal.c | 2 + tools/testing/selftests/rseq/param_test.c | 23 ++ tools/testing/selftests/rseq/rseq-riscv.h | 622 ++++++++++++++++++++++++++++++ tools/testing/selftests/rseq/rseq.h | 2 + 8 files changed, 782 insertions(+), 1 deletion(-) create mode 100644 tools/testing/selftests/rseq/rseq-riscv.h
From: Patrick Stählin me@packi.ch
In order to select HAVE_REGS_AND_STACK_ACCESS_API, adding the APIs required by kprobes to access pt_regs and stack entries to the RISC-V ports.
Signed-off-by: Patrick Stählin me@packi.ch Signed-off-by: Vincent Chen vincent.chen@sifive.com --- arch/riscv/Kconfig | 1 + arch/riscv/include/asm/ptrace.h | 29 +++++++++++- arch/riscv/kernel/ptrace.c | 99 +++++++++++++++++++++++++++++++++++++++++ 3 files changed, 128 insertions(+), 1 deletion(-)
diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index 73f029eae0cc..fa98b279257e 100644 --- a/arch/riscv/Kconfig +++ b/arch/riscv/Kconfig @@ -66,6 +66,7 @@ config RISCV select ARCH_HAS_GCOV_PROFILE_ALL select HAVE_COPY_THREAD_TLS select HAVE_ARCH_KASAN if MMU && 64BIT + select HAVE_REGS_AND_STACK_ACCESS_API
config ARCH_MMAP_RND_BITS_MIN default 18 if 64BIT diff --git a/arch/riscv/include/asm/ptrace.h b/arch/riscv/include/asm/ptrace.h index ee49f80c9533..9fc685659eb1 100644 --- a/arch/riscv/include/asm/ptrace.h +++ b/arch/riscv/include/asm/ptrace.h @@ -8,6 +8,7 @@
#include <uapi/asm/ptrace.h> #include <asm/csr.h> +#include <linux/compiler.h>
#ifndef __ASSEMBLY__
@@ -59,7 +60,7 @@ struct pt_regs { #endif
#define user_mode(regs) (((regs)->status & SR_PP) == 0) - +#define MAX_REG_OFFSET offsetof(struct pt_regs, orig_a0)
/* Helpers for working with the instruction pointer */ static inline unsigned long instruction_pointer(struct pt_regs *regs) @@ -79,6 +80,12 @@ static inline unsigned long user_stack_pointer(struct pt_regs *regs) { return regs->sp; } + +static inline unsigned long kernel_stack_pointer(struct pt_regs *regs) +{ + return regs->sp; +} + static inline void user_stack_pointer_set(struct pt_regs *regs, unsigned long val) { @@ -101,6 +108,26 @@ static inline unsigned long regs_return_value(struct pt_regs *regs) return regs->a0; }
+int regs_query_register_offset(const char *name); + +unsigned long regs_get_kernel_stack_nth(struct pt_regs *regs, unsigned int n); +/** + * regs_get_register() - get register value from its offset + * @regs: pt_regs from which register value is gotten + * @offset: offset of the register. + * + * regs_get_register returns the value of a register whose offset from @regs. + * The @offset is the offset of the register in struct pt_regs. + * If @offset is bigger than MAX_REG_OFFSET, this returns 0. + */ +static inline unsigned long regs_get_register(struct pt_regs *regs, + unsigned int offset) +{ + if (unlikely(offset > MAX_REG_OFFSET)) + return 0; + + return *(unsigned long *)((unsigned long)regs + offset); +} #endif /* __ASSEMBLY__ */
#endif /* _ASM_RISCV_PTRACE_H */ diff --git a/arch/riscv/kernel/ptrace.c b/arch/riscv/kernel/ptrace.c index 407464201b91..bdfee5eff777 100644 --- a/arch/riscv/kernel/ptrace.c +++ b/arch/riscv/kernel/ptrace.c @@ -125,6 +125,105 @@ const struct user_regset_view *task_user_regset_view(struct task_struct *task) return &riscv_user_native_view; }
+struct pt_regs_offset { + const char *name; + int offset; +}; + +#define REG_OFFSET_NAME(r) {.name = #r, .offset = offsetof(struct pt_regs, r)} +#define REG_OFFSET_END {.name = NULL, .offset = 0} + +static const struct pt_regs_offset regoffset_table[] = { + REG_OFFSET_NAME(epc), + REG_OFFSET_NAME(ra), + REG_OFFSET_NAME(sp), + REG_OFFSET_NAME(gp), + REG_OFFSET_NAME(tp), + REG_OFFSET_NAME(t0), + REG_OFFSET_NAME(t1), + REG_OFFSET_NAME(t2), + REG_OFFSET_NAME(s0), + REG_OFFSET_NAME(s1), + REG_OFFSET_NAME(a0), + REG_OFFSET_NAME(a1), + REG_OFFSET_NAME(a2), + REG_OFFSET_NAME(a3), + REG_OFFSET_NAME(a4), + REG_OFFSET_NAME(a5), + REG_OFFSET_NAME(a6), + REG_OFFSET_NAME(a7), + REG_OFFSET_NAME(s2), + REG_OFFSET_NAME(s3), + REG_OFFSET_NAME(s4), + REG_OFFSET_NAME(s5), + REG_OFFSET_NAME(s6), + REG_OFFSET_NAME(s7), + REG_OFFSET_NAME(s8), + REG_OFFSET_NAME(s9), + REG_OFFSET_NAME(s10), + REG_OFFSET_NAME(s11), + REG_OFFSET_NAME(t3), + REG_OFFSET_NAME(t4), + REG_OFFSET_NAME(t5), + REG_OFFSET_NAME(t6), + REG_OFFSET_NAME(status), + REG_OFFSET_NAME(badaddr), + REG_OFFSET_NAME(cause), + REG_OFFSET_NAME(orig_a0), + REG_OFFSET_END, +}; + +/** + * regs_query_register_offset() - query register offset from its name + * @name: the name of a register + * + * regs_query_register_offset() returns the offset of a register in struct + * pt_regs from its name. If the name is invalid, this returns -EINVAL; + */ +int regs_query_register_offset(const char *name) +{ + const struct pt_regs_offset *roff; + + for (roff = regoffset_table; roff->name; roff++) + if (!strcmp(roff->name, name)) + return roff->offset; + return -EINVAL; +} + +/** + * regs_within_kernel_stack() - check the address in the stack + * @regs: pt_regs which contains kernel stack pointer. + * @addr: address which is checked. + * + * regs_within_kernel_stack() checks @addr is within the kernel stack page(s). + * If @addr is within the kernel stack, it returns true. If not, returns false. + */ +static bool regs_within_kernel_stack(struct pt_regs *regs, unsigned long addr) +{ + return (addr & ~(THREAD_SIZE - 1)) == + (kernel_stack_pointer(regs) & ~(THREAD_SIZE - 1)); +} + +/** + * regs_get_kernel_stack_nth() - get Nth entry of the stack + * @regs: pt_regs which contains kernel stack pointer. + * @n: stack entry number. + * + * regs_get_kernel_stack_nth() returns @n th entry of the kernel stack which + * is specified by @regs. If the @n th entry is NOT in the kernel stack, + * this returns 0. + */ +unsigned long regs_get_kernel_stack_nth(struct pt_regs *regs, unsigned int n) +{ + unsigned long *addr = (unsigned long *)kernel_stack_pointer(regs); + + addr += n; + if (regs_within_kernel_stack(regs, (unsigned long)addr)) + return *addr; + else + return 0; +} + void ptrace_disable(struct task_struct *child) { clear_tsk_thread_flag(child, TIF_SYSCALL_TRACE);
Add calls to rseq_signal_deliver(), rseq_handle_notify_resume() and rseq_syscall() to introduce RSEQ support.
1. Call the rseq_handle_notify_resume() function on return to userspace if TIF_NOTIFY_RESUME thread flag is set.
2. Call the rseq_signal_deliver() function to fixup on the pre-signal frame when a signal is delivered on top of a restartable sequence critical section.
3. Check that system calls are not invoked from within rseq critical sections by invoking rseq_signal() from ret_from_syscall(). With CONFIG_DEBUG_RSEQ, such behavior results in termination of the process with SIGSEGV.
Signed-off-by: Vincent Chen vincent.chen@sifive.com --- arch/riscv/Kconfig | 1 + arch/riscv/kernel/entry.S | 4 ++++ arch/riscv/kernel/signal.c | 2 ++ 3 files changed, 7 insertions(+)
diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index fa98b279257e..ded32979d33d 100644 --- a/arch/riscv/Kconfig +++ b/arch/riscv/Kconfig @@ -67,6 +67,7 @@ config RISCV select HAVE_COPY_THREAD_TLS select HAVE_ARCH_KASAN if MMU && 64BIT select HAVE_REGS_AND_STACK_ACCESS_API + select HAVE_RSEQ
config ARCH_MMAP_RND_BITS_MIN default 18 if 64BIT diff --git a/arch/riscv/kernel/entry.S b/arch/riscv/kernel/entry.S index bad4d85b5e91..89d3713b0aef 100644 --- a/arch/riscv/kernel/entry.S +++ b/arch/riscv/kernel/entry.S @@ -214,6 +214,10 @@ ENTRY(handle_exception) handle_syscall: /* save the initial A0 value (needed in signal handlers) */ REG_S a0, PT_ORIG_A0(sp) +#ifdef CONFIG_RSEQ_DEBUG + move a0, sp + call rseq_syscall +#endif /* * Advance SEPC to avoid executing the original * scall instruction on sret diff --git a/arch/riscv/kernel/signal.c b/arch/riscv/kernel/signal.c index 17ba190e84a5..d939c5de41c4 100644 --- a/arch/riscv/kernel/signal.c +++ b/arch/riscv/kernel/signal.c @@ -234,6 +234,7 @@ static void handle_signal(struct ksignal *ksig, struct pt_regs *regs) sigset_t *oldset = sigmask_to_save(); int ret;
+ rseq_signal_deliver(ksig, regs); /* Are we from a system call? */ if (regs->cause == EXC_SYSCALL) { /* Avoid additional syscall restarting via ret_from_exception */ @@ -316,5 +317,6 @@ asmlinkage __visible void do_notify_resume(struct pt_regs *regs, if (thread_info_flags & _TIF_NOTIFY_RESUME) { clear_thread_flag(TIF_NOTIFY_RESUME); tracehook_notify_resume(regs); + rseq_handle_notify_resume(NULL, regs); } }
----- On Mar 9, 2020, at 1:59 AM, Vincent Chen vincent.chen@sifive.com wrote: [...]
--- a/arch/riscv/kernel/signal.c +++ b/arch/riscv/kernel/signal.c @@ -234,6 +234,7 @@ static void handle_signal(struct ksignal *ksig, struct pt_regs *regs) sigset_t *oldset = sigmask_to_save(); int ret;
- rseq_signal_deliver(ksig, regs); /* Are we from a system call? */ if (regs->cause == EXC_SYSCALL) {
[...]
As Al Viro pointed out on IRC, the rseq_signal_deliver() should go after syscall restart handling, similarly to what is done on every other supported architecture.
Note that there is already an upstream commit derived on this non-upstream patch:
commit 9866d141a097 ("csky: Add support for restartable sequence")
which is broken in the same way.
I'm not sure why I was never CC'd on the csky patch. Considering that nobody bothered to implement the rseq selftests for csky, I don't see how any of this is tested. I would favor a revert of that commit until the testing glue is contributed. Unfortunately, the csky commit has been upstream since v5.7.
Thanks,
Mathieu
On Mon, Jul 19, 2021 at 10:43 PM Mathieu Desnoyers mathieu.desnoyers@efficios.com wrote:
----- On Mar 9, 2020, at 1:59 AM, Vincent Chen vincent.chen@sifive.com wrote: [...]
--- a/arch/riscv/kernel/signal.c +++ b/arch/riscv/kernel/signal.c @@ -234,6 +234,7 @@ static void handle_signal(struct ksignal *ksig, struct pt_regs *regs) sigset_t *oldset = sigmask_to_save(); int ret;
rseq_signal_deliver(ksig, regs); /* Are we from a system call? */ if (regs->cause == EXC_SYSCALL) {[...]
As Al Viro pointed out on IRC, the rseq_signal_deliver() should go after syscall restart handling, similarly to what is done on every other supported architecture.
Thanks for the notification. I will adjust the porting and try to send the patch again for review.
Note that there is already an upstream commit derived on this non-upstream patch:
commit 9866d141a097 ("csky: Add support for restartable sequence")
which is broken in the same way.
I'm not sure why I was never CC'd on the csky patch. Considering that nobody bothered to implement the rseq selftests for csky, I don't see how any of this is tested. I would favor a revert of that commit until the testing glue is contributed. Unfortunately, the csky commit has been upstream since v5.7.
Thanks,
Mathieu
-- Mathieu Desnoyers EfficiOS Inc. http://www.efficios.com
Add support for risc-v in the rseq selftests, which covers both 64-bit and 32-bit ISA with little endian mode.
Signed-off-by: Vincent Chen vincent.chen@sifive.com --- tools/testing/selftests/rseq/param_test.c | 23 ++ tools/testing/selftests/rseq/rseq-riscv.h | 622 ++++++++++++++++++++++++++++++ tools/testing/selftests/rseq/rseq.h | 2 + 3 files changed, 647 insertions(+) create mode 100644 tools/testing/selftests/rseq/rseq-riscv.h
diff --git a/tools/testing/selftests/rseq/param_test.c b/tools/testing/selftests/rseq/param_test.c index e8a657a5f48a..4032dfaa3af2 100644 --- a/tools/testing/selftests/rseq/param_test.c +++ b/tools/testing/selftests/rseq/param_test.c @@ -205,6 +205,29 @@ unsigned int yield_mod_cnt, nr_abort; "addiu " INJECT_ASM_REG ", -1\n\t" \ "bnez " INJECT_ASM_REG ", 222b\n\t" \ "333:\n\t" +#elif defined(__riscv) + +#define RSEQ_INJECT_INPUT \ + , [loop_cnt_1]"m"(loop_cnt[1]) \ + , [loop_cnt_2]"m"(loop_cnt[2]) \ + , [loop_cnt_3]"m"(loop_cnt[3]) \ + , [loop_cnt_4]"m"(loop_cnt[4]) \ + , [loop_cnt_5]"m"(loop_cnt[5]) \ + , [loop_cnt_6]"m"(loop_cnt[6]) + +#define INJECT_ASM_REG "t1" + +#define RSEQ_INJECT_CLOBBER \ + , INJECT_ASM_REG + +#define RSEQ_INJECT_ASM(n) \ + "lw " INJECT_ASM_REG ", %[loop_cnt_" #n "]\n\t" \ + "beqz " INJECT_ASM_REG ", 333f\n\t" \ + "222:\n\t" \ + "addi " INJECT_ASM_REG "," INJECT_ASM_REG ", -1\n\t" \ + "bnez " INJECT_ASM_REG ", 222b\n\t" \ + "333:\n\t" +
#else #error unsupported target diff --git a/tools/testing/selftests/rseq/rseq-riscv.h b/tools/testing/selftests/rseq/rseq-riscv.h new file mode 100644 index 000000000000..56b47db4a9a4 --- /dev/null +++ b/tools/testing/selftests/rseq/rseq-riscv.h @@ -0,0 +1,622 @@ +/* SPDX-License-Identifier: LGPL-2.1 OR MIT */ +/* + * Select the instruction "csrw mhartid, x0" as the RSEQ_SIG. Unlike + * other architecture, the ebreak instruction has no immediate field for + * distinguishing purposes. Hence, ebreak is not suitable as RSEQ_SIG. + * "csrw mhartid, x0" can also satisfy the RSEQ requirement because it + * is an uncommon instruction and will raise an illegal instruction + * exception when executed in all modes. + */ + +#if __ORDER_LITTLE_ENDIAN__ == 1234 +#define RSEQ_SIG 0xf1401073 /* csrr mhartid, x0 */ +#else +#error "Currently, RSEQ only supports Little-Endian version" +#endif + +#if __riscv_xlen == 64 +#define __REG_SEL(a, b) a +#elif __riscv_xlen == 32 +#define __REG_SEL(a, b) b +#endif + +#define REG_L __REG_SEL("ld ", "lw ") +#define REG_S __REG_SEL("sd ", "sw ") + +#define RISCV_FENCE(p, s) \ + __asm__ __volatile__ ("fence " #p "," #s : : : "memory") +#define rseq_smp_mb() RISCV_FENCE(rw, rw) +#define rseq_smp_rmb() RISCV_FENCE(r, r) +#define rseq_smp_wmb() RISCV_FENCE(w, w) +#define RSEQ_ASM_TMP_REG_1 "t6" +#define RSEQ_ASM_TMP_REG_2 "t5" +#define RSEQ_ASM_TMP_REG_3 "t4" +#define RSEQ_ASM_TMP_REG_4 "t3" + +#define rseq_smp_load_acquire(p) \ +__extension__ ({ \ + __typeof(*p) ____p1 = RSEQ_READ_ONCE(*p); \ + RISCV_FENCE(r, rw) \ + ____p1; \ +}) + +#define rseq_smp_acquire__after_ctrl_dep() rseq_smp_rmb() + +#define rseq_smp_store_release(p, v) \ +do { \ + RISCV_FENCE(rw, w); \ + RSEQ_WRITE_ONCE(*p, v); \ +} while (0) + + +#ifdef RSEQ_SKIP_FASTPATH +#include "rseq-skip.h" +#else /* !RSEQ_SKIP_FASTPATH */ + +#define __RSEQ_ASM_DEFINE_TABLE(label, version, flags, start_ip, \ + post_commit_offset, abort_ip) \ + ".pushsection __rseq_cs, "aw"\n" \ + ".balign 32\n" \ + __rseq_str(label) ":\n" \ + ".long " __rseq_str(version) ", " __rseq_str(flags) "\n" \ + ".quad " __rseq_str(start_ip) ", " \ + __rseq_str(post_commit_offset) ", " \ + __rseq_str(abort_ip) "\n" \ + ".popsection\n\t" \ + ".pushsection __rseq_cs_ptr_array, "aw"\n" \ + ".quad " __rseq_str(label) "b\n" \ + ".popsection\n" + +#define RSEQ_ASM_DEFINE_TABLE(label, start_ip, post_commit_ip, abort_ip) \ + __RSEQ_ASM_DEFINE_TABLE(label, 0x0, 0x0, start_ip, \ + (post_commit_ip - start_ip), abort_ip) + + +/* + * Exit points of a rseq critical section consist of all instructions outside + * of the critical section where a critical section can either branch to or + * reach through the normal course of its execution. The abort IP and the + * post-commit IP are already part of the __rseq_cs section and should not be + * explicitly defined as additional exit points. Knowing all exit points is + * useful to assist debuggers stepping over the critical section. + */ +#define RSEQ_ASM_DEFINE_EXIT_POINT(start_ip, exit_ip) \ + ".pushsection __rseq_exit_point_array, "aw"\n" \ + ".quad " __rseq_str(start_ip) ", " __rseq_str(exit_ip) "\n" \ + ".popsection\n" + +#define RSEQ_ASM_STORE_RSEQ_CS(label, cs_label, rseq_cs) \ + RSEQ_INJECT_ASM(1) \ + "la "RSEQ_ASM_TMP_REG_1 ", " __rseq_str(cs_label) "\n" \ + REG_S RSEQ_ASM_TMP_REG_1 ", %[" __rseq_str(rseq_cs) "]\n" \ + __rseq_str(label) ":\n" + +#define RSEQ_ASM_DEFINE_ABORT(label, abort_label) \ + "j 222f\n" \ + ".balign 4\n" \ + ".long " __rseq_str(RSEQ_SIG) "\n" \ + __rseq_str(label) ":\n" \ + "j %l[" __rseq_str(abort_label) "]\n" \ + "222:\n" + +#define RSEQ_ASM_OP_STORE(value, var) \ + REG_S "%[" __rseq_str(value) "], %[" __rseq_str(var) "]\n" + +#define RSEQ_ASM_OP_CMPEQ(var, expect, label) \ + REG_L RSEQ_ASM_TMP_REG_1 ", %[" __rseq_str(var) "]\n" \ + "bne "RSEQ_ASM_TMP_REG_1 ", %[" __rseq_str(expect) "] ," \ + __rseq_str(label) "\n" + +#define RSEQ_ASM_OP_CMPEQ32(var, expect, label) \ + "lw "RSEQ_ASM_TMP_REG_1 ", %[" __rseq_str(var) "]\n" \ + "bne "RSEQ_ASM_TMP_REG_1 ", %[" __rseq_str(expect) "] ," \ + __rseq_str(label) "\n" + +#define RSEQ_ASM_OP_CMPNE(var, expect, label) \ + REG_L RSEQ_ASM_TMP_REG_1 ", %[" __rseq_str(var) "]\n" \ + "beq "RSEQ_ASM_TMP_REG_1 ", %[" __rseq_str(expect) "] ," \ + __rseq_str(label) "\n" + +#define RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, label) \ + RSEQ_INJECT_ASM(2) \ + RSEQ_ASM_OP_CMPEQ32(current_cpu_id, cpu_id, label) + +#define RSEQ_ASM_OP_R_LOAD(var) \ + REG_L RSEQ_ASM_TMP_REG_1 ", %[" __rseq_str(var) "]\n" + +#define RSEQ_ASM_OP_R_STORE(var) \ + REG_S RSEQ_ASM_TMP_REG_1 ", %[" __rseq_str(var) "]\n" + +#define RSEQ_ASM_OP_R_LOAD_OFF(offset) \ + "add "RSEQ_ASM_TMP_REG_1 ", %[" __rseq_str(offset) "], " \ + RSEQ_ASM_TMP_REG_1 "\n" \ + REG_L RSEQ_ASM_TMP_REG_1 ", (" RSEQ_ASM_TMP_REG_1 ")\n" + +#define RSEQ_ASM_OP_R_ADD(count) \ + "add "RSEQ_ASM_TMP_REG_1 ", " RSEQ_ASM_TMP_REG_1 \ + ", %[" __rseq_str(count) "]\n" + +#define RSEQ_ASM_OP_FINAL_STORE(value, var, post_commit_label) \ + RSEQ_ASM_OP_STORE(value, var) \ + __rseq_str(post_commit_label) ":\n" + +#define RSEQ_ASM_OP_FINAL_STORE_RELEASE(value, var, post_commit_label) \ + "fence rw, w\n" \ + RSEQ_ASM_OP_STORE(value, var) \ + __rseq_str(post_commit_label) ":\n" + +#define RSEQ_ASM_OP_R_FINAL_STORE(var, post_commit_label) \ + REG_S RSEQ_ASM_TMP_REG_1 ", %[" __rseq_str(var) "]\n" \ + __rseq_str(post_commit_label) ":\n" + +#define RSEQ_ASM_OP_R_BAD_MEMCPY(dst, src, len) \ + "beqz %[" __rseq_str(len) "], 333f\n" \ + "mv " RSEQ_ASM_TMP_REG_1 ", %[" __rseq_str(len) "]\n" \ + "mv " RSEQ_ASM_TMP_REG_2 ", %[" __rseq_str(src) "]\n" \ + "mv " RSEQ_ASM_TMP_REG_3 ", %[" __rseq_str(dst) "]\n" \ + "222:\n" \ + "lb " RSEQ_ASM_TMP_REG_4 ", 0(" RSEQ_ASM_TMP_REG_2 ")\n" \ + "sb " RSEQ_ASM_TMP_REG_4 ", 0(" RSEQ_ASM_TMP_REG_3 ")\n" \ + "addi " RSEQ_ASM_TMP_REG_1 ", " RSEQ_ASM_TMP_REG_1 ", -1\n" \ + "addi " RSEQ_ASM_TMP_REG_2 ", " RSEQ_ASM_TMP_REG_2 ", 1\n" \ + "addi " RSEQ_ASM_TMP_REG_3 ", " RSEQ_ASM_TMP_REG_3 ", 1\n" \ + "bnez " RSEQ_ASM_TMP_REG_1 ", 222b\n" \ + "333:\n" + +static inline __attribute__((always_inline)) +int rseq_cmpeqv_storev(intptr_t *v, intptr_t expect, intptr_t newv, int cpu) +{ + RSEQ_INJECT_C(9) + + __asm__ __volatile__ goto ( + RSEQ_ASM_DEFINE_TABLE(1, 2f, 3f, 4f) + RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[cmpfail]) +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[error1]) + RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[error2]) +#endif + RSEQ_ASM_STORE_RSEQ_CS(2, 1b, rseq_cs) + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) + RSEQ_INJECT_ASM(3) + RSEQ_ASM_OP_CMPEQ(v, expect, %l[cmpfail]) + RSEQ_INJECT_ASM(4) +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1]) + RSEQ_ASM_OP_CMPEQ(v, expect, %l[error2]) +#endif + RSEQ_ASM_OP_FINAL_STORE(newv, v, 3) + RSEQ_INJECT_ASM(5) + RSEQ_ASM_DEFINE_ABORT(4, abort) + : /* gcc asm goto does not allow outputs */ + : [cpu_id] "r" (cpu), + [current_cpu_id] "m" (__rseq_abi.cpu_id), + [rseq_cs] "m" (__rseq_abi.rseq_cs), + [v] "m" (*v), + [expect] "r" (expect), + [newv] "r" (newv) + RSEQ_INJECT_INPUT + : "memory", RSEQ_ASM_TMP_REG_1 + : abort, cmpfail +#ifdef RSEQ_COMPARE_TWICE + , error1, error2 +#endif + ); + + return 0; +abort: + RSEQ_INJECT_FAILED + return -1; +cmpfail: + return 1; +#ifdef RSEQ_COMPARE_TWICE +error1: + rseq_bug("cpu_id comparison failed"); +error2: + rseq_bug("expected value comparison failed"); +#endif +} + +static inline __attribute__((always_inline)) +int rseq_cmpnev_storeoffp_load(intptr_t *v, intptr_t expectnot, + off_t voffp, intptr_t *load, int cpu) +{ + RSEQ_INJECT_C(9) + + __asm__ __volatile__ goto ( + RSEQ_ASM_DEFINE_TABLE(1, 2f, 3f, 4f) + RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[cmpfail]) +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[error1]) + RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[error2]) +#endif + RSEQ_ASM_STORE_RSEQ_CS(2, 1b, rseq_cs) + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) + RSEQ_INJECT_ASM(3) + RSEQ_ASM_OP_CMPNE(v, expectnot, %l[cmpfail]) + RSEQ_INJECT_ASM(4) +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1]) + RSEQ_ASM_OP_CMPNE(v, expectnot, %l[error2]) +#endif + RSEQ_ASM_OP_R_LOAD(v) + RSEQ_ASM_OP_R_STORE(load) + RSEQ_ASM_OP_R_LOAD_OFF(voffp) + RSEQ_ASM_OP_R_FINAL_STORE(v, 3) + RSEQ_INJECT_ASM(5) + RSEQ_ASM_DEFINE_ABORT(4, abort) + : /* gcc asm goto does not allow outputs */ + : [cpu_id] "r" (cpu), + [current_cpu_id] "m" (__rseq_abi.cpu_id), + [rseq_cs] "m" (__rseq_abi.rseq_cs), + [v] "m" (*v), + [expectnot] "r" (expectnot), + [load] "m" (*load), + [voffp] "r" (voffp) + RSEQ_INJECT_INPUT + : "memory", RSEQ_ASM_TMP_REG_1 + : abort, cmpfail +#ifdef RSEQ_COMPARE_TWICE + , error1, error2 +#endif + ); + return 0; +abort: + RSEQ_INJECT_FAILED + return -1; +cmpfail: + return 1; +#ifdef RSEQ_COMPARE_TWICE +error1: + rseq_bug("cpu_id comparison failed"); +error2: + rseq_bug("expected value comparison failed"); +#endif +} + +static inline __attribute__((always_inline)) +int rseq_addv(intptr_t *v, intptr_t count, int cpu) +{ + RSEQ_INJECT_C(9) + + __asm__ __volatile__ goto ( + RSEQ_ASM_DEFINE_TABLE(1, 2f, 3f, 4f) +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[error1]) +#endif + RSEQ_ASM_STORE_RSEQ_CS(2, 1b, rseq_cs) + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) + RSEQ_INJECT_ASM(3) +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1]) +#endif + RSEQ_ASM_OP_R_LOAD(v) + RSEQ_ASM_OP_R_ADD(count) + RSEQ_ASM_OP_R_FINAL_STORE(v, 3) + RSEQ_INJECT_ASM(4) + RSEQ_ASM_DEFINE_ABORT(4, abort) + : /* gcc asm goto does not allow outputs */ + : [cpu_id] "r" (cpu), + [current_cpu_id] "m" (__rseq_abi.cpu_id), + [rseq_cs] "m" (__rseq_abi.rseq_cs), + [v] "m" (*v), + [count] "r" (count) + RSEQ_INJECT_INPUT + : "memory", RSEQ_ASM_TMP_REG_1 + : abort +#ifdef RSEQ_COMPARE_TWICE + , error1 +#endif + ); + return 0; +abort: + RSEQ_INJECT_FAILED + return -1; +#ifdef RSEQ_COMPARE_TWICE +error1: + rseq_bug("cpu_id comparison failed"); +#endif +} + +static inline __attribute__((always_inline)) +int rseq_cmpeqv_trystorev_storev(intptr_t *v, intptr_t expect, + intptr_t *v2, intptr_t newv2, + intptr_t newv, int cpu) +{ + RSEQ_INJECT_C(9) + + __asm__ __volatile__ goto ( + RSEQ_ASM_DEFINE_TABLE(1, 2f, 3f, 4f) + RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[cmpfail]) +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[error1]) + RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[error2]) +#endif + RSEQ_ASM_STORE_RSEQ_CS(2, 1b, rseq_cs) + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) + RSEQ_INJECT_ASM(3) + RSEQ_ASM_OP_CMPEQ(v, expect, %l[cmpfail]) + RSEQ_INJECT_ASM(4) +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1]) + RSEQ_ASM_OP_CMPEQ(v, expect, %l[error2]) +#endif + RSEQ_ASM_OP_STORE(newv2, v2) + RSEQ_INJECT_ASM(5) + RSEQ_ASM_OP_FINAL_STORE(newv, v, 3) + RSEQ_INJECT_ASM(6) + RSEQ_ASM_DEFINE_ABORT(4, abort) + : /* gcc asm goto does not allow outputs */ + : [cpu_id] "r" (cpu), + [current_cpu_id] "m" (__rseq_abi.cpu_id), + [rseq_cs] "m" (__rseq_abi.rseq_cs), + [expect] "r" (expect), + [v] "m" (*v), + [newv] "r" (newv), + [v2] "m" (*v2), + [newv2] "r" (newv2) + RSEQ_INJECT_INPUT + : "memory", RSEQ_ASM_TMP_REG_1 + : abort, cmpfail +#ifdef RSEQ_COMPARE_TWICE + , error1, error2 +#endif + ); + + return 0; +abort: + RSEQ_INJECT_FAILED + return -1; +cmpfail: + return 1; +#ifdef RSEQ_COMPARE_TWICE +error1: + rseq_bug("cpu_id comparison failed"); +error2: + rseq_bug("expected value comparison failed"); +#endif +} + +static inline __attribute__((always_inline)) +int rseq_cmpeqv_trystorev_storev_release(intptr_t *v, intptr_t expect, + intptr_t *v2, intptr_t newv2, + intptr_t newv, int cpu) +{ + RSEQ_INJECT_C(9) + + __asm__ __volatile__ goto ( + RSEQ_ASM_DEFINE_TABLE(1, 2f, 3f, 4f) + RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[cmpfail]) +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[error1]) + RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[error2]) +#endif + RSEQ_ASM_STORE_RSEQ_CS(2, 1b, rseq_cs) + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) + RSEQ_INJECT_ASM(3) + RSEQ_ASM_OP_CMPEQ(v, expect, %l[cmpfail]) + RSEQ_INJECT_ASM(4) +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1]) + RSEQ_ASM_OP_CMPEQ(v, expect, %l[error2]) +#endif + RSEQ_ASM_OP_STORE(newv2, v2) + RSEQ_INJECT_ASM(5) + RSEQ_ASM_OP_FINAL_STORE_RELEASE(newv, v, 3) + RSEQ_INJECT_ASM(6) + RSEQ_ASM_DEFINE_ABORT(4, abort) + : /* gcc asm goto does not allow outputs */ + : [cpu_id] "r" (cpu), + [current_cpu_id] "m" (__rseq_abi.cpu_id), + [rseq_cs] "m" (__rseq_abi.rseq_cs), + [expect] "r" (expect), + [v] "m" (*v), + [newv] "r" (newv), + [v2] "m" (*v2), + [newv2] "r" (newv2) + RSEQ_INJECT_INPUT + : "memory", RSEQ_ASM_TMP_REG_1 + : abort, cmpfail +#ifdef RSEQ_COMPARE_TWICE + , error1, error2 +#endif + ); + + return 0; +abort: + RSEQ_INJECT_FAILED + return -1; +cmpfail: + return 1; +#ifdef RSEQ_COMPARE_TWICE +error1: + rseq_bug("cpu_id comparison failed"); +error2: + rseq_bug("expected value comparison failed"); +#endif +} + +static inline __attribute__((always_inline)) +int rseq_cmpeqv_cmpeqv_storev(intptr_t *v, intptr_t expect, + intptr_t *v2, intptr_t expect2, + intptr_t newv, int cpu) +{ + RSEQ_INJECT_C(9) + + __asm__ __volatile__ goto ( + RSEQ_ASM_DEFINE_TABLE(1, 2f, 3f, 4f) + RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[cmpfail]) +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[error1]) + RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[error2]) + RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[error3]) +#endif + RSEQ_ASM_STORE_RSEQ_CS(2, 1b, rseq_cs) + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) + RSEQ_INJECT_ASM(3) + RSEQ_ASM_OP_CMPEQ(v, expect, %l[cmpfail]) + RSEQ_INJECT_ASM(4) + RSEQ_ASM_OP_CMPEQ(v2, expect2, %l[cmpfail]) + RSEQ_INJECT_ASM(5) +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1]) + RSEQ_ASM_OP_CMPEQ(v, expect, %l[error2]) + RSEQ_ASM_OP_CMPEQ(v2, expect2, %l[error3]) +#endif + RSEQ_ASM_OP_FINAL_STORE(newv, v, 3) + RSEQ_INJECT_ASM(6) + RSEQ_ASM_DEFINE_ABORT(4, abort) + : /* gcc asm goto does not allow outputs */ + : [cpu_id] "r" (cpu), + [current_cpu_id] "m" (__rseq_abi.cpu_id), + [rseq_cs] "m" (__rseq_abi.rseq_cs), + [v] "m" (*v), + [expect] "r" (expect), + [v2] "m" (*v2), + [expect2] "r" (expect2), + [newv] "r" (newv) + RSEQ_INJECT_INPUT + : "memory", RSEQ_ASM_TMP_REG_1 + : abort, cmpfail +#ifdef RSEQ_COMPARE_TWICE + , error1, error2, error3 +#endif + ); + + return 0; +abort: + RSEQ_INJECT_FAILED + return -1; +cmpfail: + return 1; +#ifdef RSEQ_COMPARE_TWICE +error1: + rseq_bug("cpu_id comparison failed"); +error2: + rseq_bug("expected value comparison failed"); +error3: + rseq_bug("2nd expected value comparison failed"); +#endif +} + +static inline __attribute__((always_inline)) +int rseq_cmpeqv_trymemcpy_storev(intptr_t *v, intptr_t expect, + void *dst, void *src, size_t len, + intptr_t newv, int cpu) +{ + RSEQ_INJECT_C(9) + __asm__ __volatile__ goto ( + RSEQ_ASM_DEFINE_TABLE(1, 2f, 3f, 4f) + RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[cmpfail]) +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[error1]) + RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[error2]) +#endif + RSEQ_ASM_STORE_RSEQ_CS(2, 1b, rseq_cs) + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) + RSEQ_INJECT_ASM(3) + RSEQ_ASM_OP_CMPEQ(v, expect, %l[cmpfail]) + RSEQ_INJECT_ASM(4) +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1]) + RSEQ_ASM_OP_CMPEQ(v, expect, %l[error2]) +#endif + RSEQ_ASM_OP_R_BAD_MEMCPY(dst, src, len) + RSEQ_INJECT_ASM(5) + RSEQ_ASM_OP_FINAL_STORE(newv, v, 3) + RSEQ_INJECT_ASM(6) + RSEQ_ASM_DEFINE_ABORT(4, abort) + : /* gcc asm goto does not allow outputs */ + : [cpu_id] "r" (cpu), + [current_cpu_id] "m" (__rseq_abi.cpu_id), + [rseq_cs] "m" (__rseq_abi.rseq_cs), + [expect] "r" (expect), + [v] "m" (*v), + [newv] "r" (newv), + [dst] "r" (dst), + [src] "r" (src), + [len] "r" (len) + RSEQ_INJECT_INPUT + : "memory", RSEQ_ASM_TMP_REG_1, RSEQ_ASM_TMP_REG_2, + RSEQ_ASM_TMP_REG_3, RSEQ_ASM_TMP_REG_4 + : abort, cmpfail +#ifdef RSEQ_COMPARE_TWICE + , error1, error2 +#endif + ); + + return 0; +abort: + RSEQ_INJECT_FAILED + return -1; +cmpfail: + return 1; +#ifdef RSEQ_COMPARE_TWICE +error1: + rseq_bug("cpu_id comparison failed"); +error2: + rseq_bug("expected value comparison failed"); +#endif +} + +static inline __attribute__((always_inline)) +int rseq_cmpeqv_trymemcpy_storev_release(intptr_t *v, intptr_t expect, + void *dst, void *src, size_t len, + intptr_t newv, int cpu) +{ + RSEQ_INJECT_C(9) + + __asm__ __volatile__ goto ( + RSEQ_ASM_DEFINE_TABLE(1, 2f, 3f, 4f) + RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[cmpfail]) +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[error1]) + RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[error2]) +#endif + RSEQ_ASM_STORE_RSEQ_CS(2, 1b, rseq_cs) + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) + RSEQ_INJECT_ASM(3) + RSEQ_ASM_OP_CMPEQ(v, expect, %l[cmpfail]) + RSEQ_INJECT_ASM(4) +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1]) + RSEQ_ASM_OP_CMPEQ(v, expect, %l[error2]) +#endif + RSEQ_ASM_OP_R_BAD_MEMCPY(dst, src, len) + RSEQ_INJECT_ASM(5) + RSEQ_ASM_OP_FINAL_STORE_RELEASE(newv, v, 3) + RSEQ_INJECT_ASM(6) + RSEQ_ASM_DEFINE_ABORT(4, abort) + : /* gcc asm goto does not allow outputs */ + : [cpu_id] "r" (cpu), + [current_cpu_id] "m" (__rseq_abi.cpu_id), + [rseq_cs] "m" (__rseq_abi.rseq_cs), + [expect] "r" (expect), + [v] "m" (*v), + [newv] "r" (newv), + [dst] "r" (dst), + [src] "r" (src), + [len] "r" (len) + RSEQ_INJECT_INPUT + : "memory", RSEQ_ASM_TMP_REG_1, RSEQ_ASM_TMP_REG_2, + RSEQ_ASM_TMP_REG_3, RSEQ_ASM_TMP_REG_4 + : abort, cmpfail +#ifdef RSEQ_COMPARE_TWICE + , error1, error2 +#endif + ); + + return 0; +abort: + RSEQ_INJECT_FAILED + return -1; +cmpfail: + return 1; +#ifdef RSEQ_COMPARE_TWICE +error1: + rseq_bug("cpu_id comparison failed"); +error2: + rseq_bug("expected value comparison failed"); +#endif +} + +#endif /* !RSEQ_SKIP_FASTPATH */ diff --git a/tools/testing/selftests/rseq/rseq.h b/tools/testing/selftests/rseq/rseq.h index 3f63eb362b92..72efb6d3d84e 100644 --- a/tools/testing/selftests/rseq/rseq.h +++ b/tools/testing/selftests/rseq/rseq.h @@ -79,6 +79,8 @@ extern int __rseq_handled; #include <rseq-mips.h> #elif defined(__s390__) #include <rseq-s390.h> +#elif defined(__riscv) +#include <rseq-riscv.h> #else #error unsupported target #endif
On Sun, 08 Mar 2020 22:59:52 PDT (-0700), vincent.chen@sifive.com wrote:
Add support for risc-v in the rseq selftests, which covers both 64-bit and 32-bit ISA with little endian mode.
Signed-off-by: Vincent Chen vincent.chen@sifive.com
tools/testing/selftests/rseq/param_test.c | 23 ++ tools/testing/selftests/rseq/rseq-riscv.h | 622 ++++++++++++++++++++++++++++++ tools/testing/selftests/rseq/rseq.h | 2 + 3 files changed, 647 insertions(+) create mode 100644 tools/testing/selftests/rseq/rseq-riscv.h
There are a ton of checkpatch errors in here.
diff --git a/tools/testing/selftests/rseq/param_test.c b/tools/testing/selftests/rseq/param_test.c index e8a657a5f48a..4032dfaa3af2 100644 --- a/tools/testing/selftests/rseq/param_test.c +++ b/tools/testing/selftests/rseq/param_test.c @@ -205,6 +205,29 @@ unsigned int yield_mod_cnt, nr_abort; "addiu " INJECT_ASM_REG ", -1\n\t" \ "bnez " INJECT_ASM_REG ", 222b\n\t" \ "333:\n\t" +#elif defined(__riscv)
+#define RSEQ_INJECT_INPUT \
- , [loop_cnt_1]"m"(loop_cnt[1]) \
- , [loop_cnt_2]"m"(loop_cnt[2]) \
- , [loop_cnt_3]"m"(loop_cnt[3]) \
- , [loop_cnt_4]"m"(loop_cnt[4]) \
- , [loop_cnt_5]"m"(loop_cnt[5]) \
- , [loop_cnt_6]"m"(loop_cnt[6])
+#define INJECT_ASM_REG "t1"
+#define RSEQ_INJECT_CLOBBER \
- , INJECT_ASM_REG
+#define RSEQ_INJECT_ASM(n) \
- "lw " INJECT_ASM_REG ", %[loop_cnt_" #n "]\n\t" \
- "beqz " INJECT_ASM_REG ", 333f\n\t" \
- "222:\n\t" \
- "addi " INJECT_ASM_REG "," INJECT_ASM_REG ", -1\n\t" \
- "bnez " INJECT_ASM_REG ", 222b\n\t" \
- "333:\n\t"
#else #error unsupported target diff --git a/tools/testing/selftests/rseq/rseq-riscv.h b/tools/testing/selftests/rseq/rseq-riscv.h new file mode 100644 index 000000000000..56b47db4a9a4 --- /dev/null +++ b/tools/testing/selftests/rseq/rseq-riscv.h @@ -0,0 +1,622 @@ +/* SPDX-License-Identifier: LGPL-2.1 OR MIT */ +/*
- Select the instruction "csrw mhartid, x0" as the RSEQ_SIG. Unlike
- other architecture, the ebreak instruction has no immediate field for
- distinguishing purposes. Hence, ebreak is not suitable as RSEQ_SIG.
- "csrw mhartid, x0" can also satisfy the RSEQ requirement because it
- is an uncommon instruction and will raise an illegal instruction
- exception when executed in all modes.
- */
+#if __ORDER_LITTLE_ENDIAN__ == 1234 +#define RSEQ_SIG 0xf1401073 /* csrr mhartid, x0 */ +#else +#error "Currently, RSEQ only supports Little-Endian version" +#endif
+#if __riscv_xlen == 64 +#define __REG_SEL(a, b) a +#elif __riscv_xlen == 32 +#define __REG_SEL(a, b) b +#endif
+#define REG_L __REG_SEL("ld ", "lw ") +#define REG_S __REG_SEL("sd ", "sw ")
+#define RISCV_FENCE(p, s) \
- __asm__ __volatile__ ("fence " #p "," #s : : : "memory")
+#define rseq_smp_mb() RISCV_FENCE(rw, rw) +#define rseq_smp_rmb() RISCV_FENCE(r, r) +#define rseq_smp_wmb() RISCV_FENCE(w, w) +#define RSEQ_ASM_TMP_REG_1 "t6" +#define RSEQ_ASM_TMP_REG_2 "t5" +#define RSEQ_ASM_TMP_REG_3 "t4" +#define RSEQ_ASM_TMP_REG_4 "t3"
+#define rseq_smp_load_acquire(p) \ +__extension__ ({ \
- __typeof(*p) ____p1 = RSEQ_READ_ONCE(*p); \
- RISCV_FENCE(r, rw) \
- ____p1; \
+})
+#define rseq_smp_acquire__after_ctrl_dep() rseq_smp_rmb()
+#define rseq_smp_store_release(p, v) \ +do { \
- RISCV_FENCE(rw, w); \
- RSEQ_WRITE_ONCE(*p, v); \
+} while (0)
+#ifdef RSEQ_SKIP_FASTPATH +#include "rseq-skip.h" +#else /* !RSEQ_SKIP_FASTPATH */
+#define __RSEQ_ASM_DEFINE_TABLE(label, version, flags, start_ip, \
post_commit_offset, abort_ip) \- ".pushsection __rseq_cs, "aw"\n" \
- ".balign 32\n" \
- __rseq_str(label) ":\n" \
- ".long " __rseq_str(version) ", " __rseq_str(flags) "\n" \
- ".quad " __rseq_str(start_ip) ", " \
__rseq_str(post_commit_offset) ", " \__rseq_str(abort_ip) "\n" \- ".popsection\n\t" \
- ".pushsection __rseq_cs_ptr_array, "aw"\n" \
- ".quad " __rseq_str(label) "b\n" \
- ".popsection\n"
+#define RSEQ_ASM_DEFINE_TABLE(label, start_ip, post_commit_ip, abort_ip) \
- __RSEQ_ASM_DEFINE_TABLE(label, 0x0, 0x0, start_ip, \
(post_commit_ip - start_ip), abort_ip)+/*
- Exit points of a rseq critical section consist of all instructions outside
- of the critical section where a critical section can either branch to or
- reach through the normal course of its execution. The abort IP and the
- post-commit IP are already part of the __rseq_cs section and should not be
- explicitly defined as additional exit points. Knowing all exit points is
- useful to assist debuggers stepping over the critical section.
- */
+#define RSEQ_ASM_DEFINE_EXIT_POINT(start_ip, exit_ip) \
- ".pushsection __rseq_exit_point_array, "aw"\n" \
- ".quad " __rseq_str(start_ip) ", " __rseq_str(exit_ip) "\n" \
- ".popsection\n"
+#define RSEQ_ASM_STORE_RSEQ_CS(label, cs_label, rseq_cs) \
- RSEQ_INJECT_ASM(1) \
- "la "RSEQ_ASM_TMP_REG_1 ", " __rseq_str(cs_label) "\n" \
- REG_S RSEQ_ASM_TMP_REG_1 ", %[" __rseq_str(rseq_cs) "]\n" \
- __rseq_str(label) ":\n"
+#define RSEQ_ASM_DEFINE_ABORT(label, abort_label) \
- "j 222f\n" \
- ".balign 4\n" \
- ".long " __rseq_str(RSEQ_SIG) "\n" \
- __rseq_str(label) ":\n" \
- "j %l[" __rseq_str(abort_label) "]\n" \
- "222:\n"
+#define RSEQ_ASM_OP_STORE(value, var) \
- REG_S "%[" __rseq_str(value) "], %[" __rseq_str(var) "]\n"
+#define RSEQ_ASM_OP_CMPEQ(var, expect, label) \
- REG_L RSEQ_ASM_TMP_REG_1 ", %[" __rseq_str(var) "]\n" \
- "bne "RSEQ_ASM_TMP_REG_1 ", %[" __rseq_str(expect) "] ," \
__rseq_str(label) "\n"+#define RSEQ_ASM_OP_CMPEQ32(var, expect, label) \
- "lw "RSEQ_ASM_TMP_REG_1 ", %[" __rseq_str(var) "]\n" \
- "bne "RSEQ_ASM_TMP_REG_1 ", %[" __rseq_str(expect) "] ," \
__rseq_str(label) "\n"+#define RSEQ_ASM_OP_CMPNE(var, expect, label) \
- REG_L RSEQ_ASM_TMP_REG_1 ", %[" __rseq_str(var) "]\n" \
- "beq "RSEQ_ASM_TMP_REG_1 ", %[" __rseq_str(expect) "] ," \
__rseq_str(label) "\n"+#define RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, label) \
- RSEQ_INJECT_ASM(2) \
- RSEQ_ASM_OP_CMPEQ32(current_cpu_id, cpu_id, label)
+#define RSEQ_ASM_OP_R_LOAD(var) \
- REG_L RSEQ_ASM_TMP_REG_1 ", %[" __rseq_str(var) "]\n"
+#define RSEQ_ASM_OP_R_STORE(var) \
- REG_S RSEQ_ASM_TMP_REG_1 ", %[" __rseq_str(var) "]\n"
+#define RSEQ_ASM_OP_R_LOAD_OFF(offset) \
- "add "RSEQ_ASM_TMP_REG_1 ", %[" __rseq_str(offset) "], " \
RSEQ_ASM_TMP_REG_1 "\n" \- REG_L RSEQ_ASM_TMP_REG_1 ", (" RSEQ_ASM_TMP_REG_1 ")\n"
+#define RSEQ_ASM_OP_R_ADD(count) \
- "add "RSEQ_ASM_TMP_REG_1 ", " RSEQ_ASM_TMP_REG_1 \
", %[" __rseq_str(count) "]\n"+#define RSEQ_ASM_OP_FINAL_STORE(value, var, post_commit_label) \
- RSEQ_ASM_OP_STORE(value, var) \
- __rseq_str(post_commit_label) ":\n"
+#define RSEQ_ASM_OP_FINAL_STORE_RELEASE(value, var, post_commit_label) \
- "fence rw, w\n" \
- RSEQ_ASM_OP_STORE(value, var) \
- __rseq_str(post_commit_label) ":\n"
+#define RSEQ_ASM_OP_R_FINAL_STORE(var, post_commit_label) \
- REG_S RSEQ_ASM_TMP_REG_1 ", %[" __rseq_str(var) "]\n" \
- __rseq_str(post_commit_label) ":\n"
+#define RSEQ_ASM_OP_R_BAD_MEMCPY(dst, src, len) \
- "beqz %[" __rseq_str(len) "], 333f\n" \
- "mv " RSEQ_ASM_TMP_REG_1 ", %[" __rseq_str(len) "]\n" \
- "mv " RSEQ_ASM_TMP_REG_2 ", %[" __rseq_str(src) "]\n" \
- "mv " RSEQ_ASM_TMP_REG_3 ", %[" __rseq_str(dst) "]\n" \
- "222:\n" \
- "lb " RSEQ_ASM_TMP_REG_4 ", 0(" RSEQ_ASM_TMP_REG_2 ")\n" \
- "sb " RSEQ_ASM_TMP_REG_4 ", 0(" RSEQ_ASM_TMP_REG_3 ")\n" \
- "addi " RSEQ_ASM_TMP_REG_1 ", " RSEQ_ASM_TMP_REG_1 ", -1\n" \
- "addi " RSEQ_ASM_TMP_REG_2 ", " RSEQ_ASM_TMP_REG_2 ", 1\n" \
- "addi " RSEQ_ASM_TMP_REG_3 ", " RSEQ_ASM_TMP_REG_3 ", 1\n" \
- "bnez " RSEQ_ASM_TMP_REG_1 ", 222b\n" \
- "333:\n"
+static inline __attribute__((always_inline)) +int rseq_cmpeqv_storev(intptr_t *v, intptr_t expect, intptr_t newv, int cpu) +{
- RSEQ_INJECT_C(9)
- __asm__ __volatile__ goto (
RSEQ_ASM_DEFINE_TABLE(1, 2f, 3f, 4f)RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[cmpfail])+#ifdef RSEQ_COMPARE_TWICE
RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[error1])RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[error2])+#endif
RSEQ_ASM_STORE_RSEQ_CS(2, 1b, rseq_cs)RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f)RSEQ_INJECT_ASM(3)RSEQ_ASM_OP_CMPEQ(v, expect, %l[cmpfail])RSEQ_INJECT_ASM(4)+#ifdef RSEQ_COMPARE_TWICE
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1])RSEQ_ASM_OP_CMPEQ(v, expect, %l[error2])+#endif
RSEQ_ASM_OP_FINAL_STORE(newv, v, 3)RSEQ_INJECT_ASM(5)RSEQ_ASM_DEFINE_ABORT(4, abort): /* gcc asm goto does not allow outputs */: [cpu_id] "r" (cpu),[current_cpu_id] "m" (__rseq_abi.cpu_id),[rseq_cs] "m" (__rseq_abi.rseq_cs),[v] "m" (*v),[expect] "r" (expect),[newv] "r" (newv)RSEQ_INJECT_INPUT: "memory", RSEQ_ASM_TMP_REG_1: abort, cmpfail+#ifdef RSEQ_COMPARE_TWICE
, error1, error2+#endif
- );
- return 0;
+abort:
- RSEQ_INJECT_FAILED
- return -1;
+cmpfail:
- return 1;
+#ifdef RSEQ_COMPARE_TWICE +error1:
- rseq_bug("cpu_id comparison failed");
+error2:
- rseq_bug("expected value comparison failed");
+#endif +}
+static inline __attribute__((always_inline)) +int rseq_cmpnev_storeoffp_load(intptr_t *v, intptr_t expectnot,
off_t voffp, intptr_t *load, int cpu)+{
- RSEQ_INJECT_C(9)
- __asm__ __volatile__ goto (
RSEQ_ASM_DEFINE_TABLE(1, 2f, 3f, 4f)RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[cmpfail])+#ifdef RSEQ_COMPARE_TWICE
RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[error1])RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[error2])+#endif
RSEQ_ASM_STORE_RSEQ_CS(2, 1b, rseq_cs)RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f)RSEQ_INJECT_ASM(3)RSEQ_ASM_OP_CMPNE(v, expectnot, %l[cmpfail])RSEQ_INJECT_ASM(4)+#ifdef RSEQ_COMPARE_TWICE
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1])RSEQ_ASM_OP_CMPNE(v, expectnot, %l[error2])+#endif
RSEQ_ASM_OP_R_LOAD(v)RSEQ_ASM_OP_R_STORE(load)RSEQ_ASM_OP_R_LOAD_OFF(voffp)RSEQ_ASM_OP_R_FINAL_STORE(v, 3)RSEQ_INJECT_ASM(5)RSEQ_ASM_DEFINE_ABORT(4, abort): /* gcc asm goto does not allow outputs */: [cpu_id] "r" (cpu),[current_cpu_id] "m" (__rseq_abi.cpu_id),[rseq_cs] "m" (__rseq_abi.rseq_cs),[v] "m" (*v),[expectnot] "r" (expectnot),[load] "m" (*load),[voffp] "r" (voffp)RSEQ_INJECT_INPUT: "memory", RSEQ_ASM_TMP_REG_1: abort, cmpfail+#ifdef RSEQ_COMPARE_TWICE
, error1, error2+#endif
- );
- return 0;
+abort:
- RSEQ_INJECT_FAILED
- return -1;
+cmpfail:
- return 1;
+#ifdef RSEQ_COMPARE_TWICE +error1:
- rseq_bug("cpu_id comparison failed");
+error2:
- rseq_bug("expected value comparison failed");
+#endif +}
+static inline __attribute__((always_inline)) +int rseq_addv(intptr_t *v, intptr_t count, int cpu) +{
- RSEQ_INJECT_C(9)
- __asm__ __volatile__ goto (
RSEQ_ASM_DEFINE_TABLE(1, 2f, 3f, 4f)+#ifdef RSEQ_COMPARE_TWICE
RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[error1])+#endif
RSEQ_ASM_STORE_RSEQ_CS(2, 1b, rseq_cs)RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f)RSEQ_INJECT_ASM(3)+#ifdef RSEQ_COMPARE_TWICE
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1])+#endif
RSEQ_ASM_OP_R_LOAD(v)RSEQ_ASM_OP_R_ADD(count)RSEQ_ASM_OP_R_FINAL_STORE(v, 3)RSEQ_INJECT_ASM(4)RSEQ_ASM_DEFINE_ABORT(4, abort): /* gcc asm goto does not allow outputs */: [cpu_id] "r" (cpu),[current_cpu_id] "m" (__rseq_abi.cpu_id),[rseq_cs] "m" (__rseq_abi.rseq_cs),[v] "m" (*v),[count] "r" (count)RSEQ_INJECT_INPUT: "memory", RSEQ_ASM_TMP_REG_1: abort+#ifdef RSEQ_COMPARE_TWICE
, error1+#endif
- );
- return 0;
+abort:
- RSEQ_INJECT_FAILED
- return -1;
+#ifdef RSEQ_COMPARE_TWICE +error1:
- rseq_bug("cpu_id comparison failed");
+#endif +}
+static inline __attribute__((always_inline)) +int rseq_cmpeqv_trystorev_storev(intptr_t *v, intptr_t expect,
intptr_t *v2, intptr_t newv2,intptr_t newv, int cpu)+{
- RSEQ_INJECT_C(9)
- __asm__ __volatile__ goto (
RSEQ_ASM_DEFINE_TABLE(1, 2f, 3f, 4f)RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[cmpfail])+#ifdef RSEQ_COMPARE_TWICE
RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[error1])RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[error2])+#endif
RSEQ_ASM_STORE_RSEQ_CS(2, 1b, rseq_cs)RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f)RSEQ_INJECT_ASM(3)RSEQ_ASM_OP_CMPEQ(v, expect, %l[cmpfail])RSEQ_INJECT_ASM(4)+#ifdef RSEQ_COMPARE_TWICE
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1])RSEQ_ASM_OP_CMPEQ(v, expect, %l[error2])+#endif
RSEQ_ASM_OP_STORE(newv2, v2)RSEQ_INJECT_ASM(5)RSEQ_ASM_OP_FINAL_STORE(newv, v, 3)RSEQ_INJECT_ASM(6)RSEQ_ASM_DEFINE_ABORT(4, abort): /* gcc asm goto does not allow outputs */: [cpu_id] "r" (cpu),[current_cpu_id] "m" (__rseq_abi.cpu_id),[rseq_cs] "m" (__rseq_abi.rseq_cs),[expect] "r" (expect),[v] "m" (*v),[newv] "r" (newv),[v2] "m" (*v2),[newv2] "r" (newv2)RSEQ_INJECT_INPUT: "memory", RSEQ_ASM_TMP_REG_1: abort, cmpfail+#ifdef RSEQ_COMPARE_TWICE
, error1, error2+#endif
- );
- return 0;
+abort:
- RSEQ_INJECT_FAILED
- return -1;
+cmpfail:
- return 1;
+#ifdef RSEQ_COMPARE_TWICE +error1:
- rseq_bug("cpu_id comparison failed");
+error2:
- rseq_bug("expected value comparison failed");
+#endif +}
+static inline __attribute__((always_inline)) +int rseq_cmpeqv_trystorev_storev_release(intptr_t *v, intptr_t expect,
intptr_t *v2, intptr_t newv2,intptr_t newv, int cpu)+{
- RSEQ_INJECT_C(9)
- __asm__ __volatile__ goto (
RSEQ_ASM_DEFINE_TABLE(1, 2f, 3f, 4f)RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[cmpfail])+#ifdef RSEQ_COMPARE_TWICE
RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[error1])RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[error2])+#endif
RSEQ_ASM_STORE_RSEQ_CS(2, 1b, rseq_cs)RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f)RSEQ_INJECT_ASM(3)RSEQ_ASM_OP_CMPEQ(v, expect, %l[cmpfail])RSEQ_INJECT_ASM(4)+#ifdef RSEQ_COMPARE_TWICE
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1])RSEQ_ASM_OP_CMPEQ(v, expect, %l[error2])+#endif
RSEQ_ASM_OP_STORE(newv2, v2)RSEQ_INJECT_ASM(5)RSEQ_ASM_OP_FINAL_STORE_RELEASE(newv, v, 3)RSEQ_INJECT_ASM(6)RSEQ_ASM_DEFINE_ABORT(4, abort): /* gcc asm goto does not allow outputs */: [cpu_id] "r" (cpu),[current_cpu_id] "m" (__rseq_abi.cpu_id),[rseq_cs] "m" (__rseq_abi.rseq_cs),[expect] "r" (expect),[v] "m" (*v),[newv] "r" (newv),[v2] "m" (*v2),[newv2] "r" (newv2)RSEQ_INJECT_INPUT: "memory", RSEQ_ASM_TMP_REG_1: abort, cmpfail+#ifdef RSEQ_COMPARE_TWICE
, error1, error2+#endif
- );
- return 0;
+abort:
- RSEQ_INJECT_FAILED
- return -1;
+cmpfail:
- return 1;
+#ifdef RSEQ_COMPARE_TWICE +error1:
- rseq_bug("cpu_id comparison failed");
+error2:
- rseq_bug("expected value comparison failed");
+#endif +}
+static inline __attribute__((always_inline)) +int rseq_cmpeqv_cmpeqv_storev(intptr_t *v, intptr_t expect,
intptr_t *v2, intptr_t expect2,intptr_t newv, int cpu)+{
- RSEQ_INJECT_C(9)
- __asm__ __volatile__ goto (
RSEQ_ASM_DEFINE_TABLE(1, 2f, 3f, 4f)RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[cmpfail])+#ifdef RSEQ_COMPARE_TWICE
RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[error1])RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[error2])RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[error3])+#endif
RSEQ_ASM_STORE_RSEQ_CS(2, 1b, rseq_cs)RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f)RSEQ_INJECT_ASM(3)RSEQ_ASM_OP_CMPEQ(v, expect, %l[cmpfail])RSEQ_INJECT_ASM(4)RSEQ_ASM_OP_CMPEQ(v2, expect2, %l[cmpfail])RSEQ_INJECT_ASM(5)+#ifdef RSEQ_COMPARE_TWICE
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1])RSEQ_ASM_OP_CMPEQ(v, expect, %l[error2])RSEQ_ASM_OP_CMPEQ(v2, expect2, %l[error3])+#endif
RSEQ_ASM_OP_FINAL_STORE(newv, v, 3)RSEQ_INJECT_ASM(6)RSEQ_ASM_DEFINE_ABORT(4, abort): /* gcc asm goto does not allow outputs */: [cpu_id] "r" (cpu),[current_cpu_id] "m" (__rseq_abi.cpu_id),[rseq_cs] "m" (__rseq_abi.rseq_cs),[v] "m" (*v),[expect] "r" (expect),[v2] "m" (*v2),[expect2] "r" (expect2),[newv] "r" (newv)RSEQ_INJECT_INPUT: "memory", RSEQ_ASM_TMP_REG_1: abort, cmpfail+#ifdef RSEQ_COMPARE_TWICE
, error1, error2, error3+#endif
- );
- return 0;
+abort:
- RSEQ_INJECT_FAILED
- return -1;
+cmpfail:
- return 1;
+#ifdef RSEQ_COMPARE_TWICE +error1:
- rseq_bug("cpu_id comparison failed");
+error2:
- rseq_bug("expected value comparison failed");
+error3:
- rseq_bug("2nd expected value comparison failed");
+#endif +}
+static inline __attribute__((always_inline)) +int rseq_cmpeqv_trymemcpy_storev(intptr_t *v, intptr_t expect,
void *dst, void *src, size_t len,intptr_t newv, int cpu)+{
- RSEQ_INJECT_C(9)
- __asm__ __volatile__ goto (
RSEQ_ASM_DEFINE_TABLE(1, 2f, 3f, 4f)RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[cmpfail])+#ifdef RSEQ_COMPARE_TWICE
RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[error1])RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[error2])+#endif
RSEQ_ASM_STORE_RSEQ_CS(2, 1b, rseq_cs)RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f)RSEQ_INJECT_ASM(3)RSEQ_ASM_OP_CMPEQ(v, expect, %l[cmpfail])RSEQ_INJECT_ASM(4)+#ifdef RSEQ_COMPARE_TWICE
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1])RSEQ_ASM_OP_CMPEQ(v, expect, %l[error2])+#endif
RSEQ_ASM_OP_R_BAD_MEMCPY(dst, src, len)RSEQ_INJECT_ASM(5)RSEQ_ASM_OP_FINAL_STORE(newv, v, 3)RSEQ_INJECT_ASM(6)RSEQ_ASM_DEFINE_ABORT(4, abort): /* gcc asm goto does not allow outputs */: [cpu_id] "r" (cpu),[current_cpu_id] "m" (__rseq_abi.cpu_id),[rseq_cs] "m" (__rseq_abi.rseq_cs),[expect] "r" (expect),[v] "m" (*v),[newv] "r" (newv),[dst] "r" (dst),[src] "r" (src),[len] "r" (len)RSEQ_INJECT_INPUT: "memory", RSEQ_ASM_TMP_REG_1, RSEQ_ASM_TMP_REG_2,RSEQ_ASM_TMP_REG_3, RSEQ_ASM_TMP_REG_4: abort, cmpfail+#ifdef RSEQ_COMPARE_TWICE
, error1, error2+#endif
- );
- return 0;
+abort:
- RSEQ_INJECT_FAILED
- return -1;
+cmpfail:
- return 1;
+#ifdef RSEQ_COMPARE_TWICE +error1:
- rseq_bug("cpu_id comparison failed");
+error2:
- rseq_bug("expected value comparison failed");
+#endif +}
+static inline __attribute__((always_inline)) +int rseq_cmpeqv_trymemcpy_storev_release(intptr_t *v, intptr_t expect,
void *dst, void *src, size_t len,intptr_t newv, int cpu)+{
- RSEQ_INJECT_C(9)
- __asm__ __volatile__ goto (
RSEQ_ASM_DEFINE_TABLE(1, 2f, 3f, 4f)RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[cmpfail])+#ifdef RSEQ_COMPARE_TWICE
RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[error1])RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[error2])+#endif
RSEQ_ASM_STORE_RSEQ_CS(2, 1b, rseq_cs)RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f)RSEQ_INJECT_ASM(3)RSEQ_ASM_OP_CMPEQ(v, expect, %l[cmpfail])RSEQ_INJECT_ASM(4)+#ifdef RSEQ_COMPARE_TWICE
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1])RSEQ_ASM_OP_CMPEQ(v, expect, %l[error2])+#endif
RSEQ_ASM_OP_R_BAD_MEMCPY(dst, src, len)RSEQ_INJECT_ASM(5)RSEQ_ASM_OP_FINAL_STORE_RELEASE(newv, v, 3)RSEQ_INJECT_ASM(6)RSEQ_ASM_DEFINE_ABORT(4, abort): /* gcc asm goto does not allow outputs */: [cpu_id] "r" (cpu),[current_cpu_id] "m" (__rseq_abi.cpu_id),[rseq_cs] "m" (__rseq_abi.rseq_cs),[expect] "r" (expect),[v] "m" (*v),[newv] "r" (newv),[dst] "r" (dst),[src] "r" (src),[len] "r" (len)RSEQ_INJECT_INPUT: "memory", RSEQ_ASM_TMP_REG_1, RSEQ_ASM_TMP_REG_2,RSEQ_ASM_TMP_REG_3, RSEQ_ASM_TMP_REG_4: abort, cmpfail+#ifdef RSEQ_COMPARE_TWICE
, error1, error2+#endif
- );
- return 0;
+abort:
- RSEQ_INJECT_FAILED
- return -1;
+cmpfail:
- return 1;
+#ifdef RSEQ_COMPARE_TWICE +error1:
- rseq_bug("cpu_id comparison failed");
+error2:
- rseq_bug("expected value comparison failed");
+#endif +}
+#endif /* !RSEQ_SKIP_FASTPATH */ diff --git a/tools/testing/selftests/rseq/rseq.h b/tools/testing/selftests/rseq/rseq.h index 3f63eb362b92..72efb6d3d84e 100644 --- a/tools/testing/selftests/rseq/rseq.h +++ b/tools/testing/selftests/rseq/rseq.h @@ -79,6 +79,8 @@ extern int __rseq_handled; #include <rseq-mips.h> #elif defined(__s390__) #include <rseq-s390.h> +#elif defined(__riscv) +#include <rseq-riscv.h> #else #error unsupported target #endif
----- On Mar 26, 2020, at 11:49 AM, Palmer Dabbelt palmer@dabbelt.com wrote:
On Sun, 08 Mar 2020 22:59:52 PDT (-0700), vincent.chen@sifive.com wrote:
Add support for risc-v in the rseq selftests, which covers both 64-bit and 32-bit ISA with little endian mode.
Signed-off-by: Vincent Chen vincent.chen@sifive.com
tools/testing/selftests/rseq/param_test.c | 23 ++ tools/testing/selftests/rseq/rseq-riscv.h | 622 ++++++++++++++++++++++++++++++ tools/testing/selftests/rseq/rseq.h | 2 + 3 files changed, 647 insertions(+) create mode 100644 tools/testing/selftests/rseq/rseq-riscv.h
There are a ton of checkpatch errors in here.
Is it just my mail client or the main issue is:
ERROR: DOS line endings
?
As far as other issues are concerned, I know there are a few checkpatch false-positives that trigger for my rseq-{$ARCH}.h header, from which rseq-riscv.h is derived, because it has issues with extensive use of inline assembly.
Thanks,
Mathieu
On Fri, Mar 27, 2020 at 12:17 AM Mathieu Desnoyers mathieu.desnoyers@efficios.com wrote:
----- On Mar 26, 2020, at 11:49 AM, Palmer Dabbelt palmer@dabbelt.com wrote:
On Sun, 08 Mar 2020 22:59:52 PDT (-0700), vincent.chen@sifive.com wrote:
Add support for risc-v in the rseq selftests, which covers both 64-bit and 32-bit ISA with little endian mode.
Signed-off-by: Vincent Chen vincent.chen@sifive.com
tools/testing/selftests/rseq/param_test.c | 23 ++ tools/testing/selftests/rseq/rseq-riscv.h | 622 ++++++++++++++++++++++++++++++ tools/testing/selftests/rseq/rseq.h | 2 + 3 files changed, 647 insertions(+) create mode 100644 tools/testing/selftests/rseq/rseq-riscv.h
There are a ton of checkpatch errors in here.
Is it just my mail client or the main issue is:
ERROR: DOS line endings
?
I am not sure, but I did not run into this error in my environment.
As far as other issues are concerned, I know there are a few checkpatch false-positives that trigger for my rseq-{$ARCH}.h header, from which rseq-riscv.h is derived, because it has issues with extensive use of inline assembly.
Thanks,
Mathieu
Thank Mathieu for your explanation.
The errors reported by checkpatch.pl can be categorized into two cases. The first one is "need consistent spacing around %". such as ERROR: need consistent spacing around '%' (ctx:WxV) #628: FILE: tools/testing/selftests/rseq/rseq-riscv.h:572: + RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[error2]) where RSEQ_ASM_DEFINE_EXIT_POINT is defined as below #define RSEQ_ASM_DEFINE_EXIT_POINT(start_ip, exit_ip) \ ".pushsection __rseq_exit_point_array, "aw"\n" \ ".quad " __rseq_str(start_ip) ", " __rseq_str(exit_ip) "\n" \ ".popsection\n"
These errors were mainly found in the rseq-riscv.h. As Mathieu mentioned, the RSEQ_ASM_DEFINE_EXIT_POINT macro is used in the inline assembly, which the second argument %l[error2] indicates the error2 is a label and it locates outside of the inline assembly. To obey the syntax, I cannot add a space after % to fix this bug.
The second kind of error is "Macros with complex values should be enclosed in parentheses" such as ERROR: Macros with complex values should be enclosed in parentheses #27: FILE: tools/testing/selftests/rseq/param_test.c:210: +#define RSEQ_INJECT_INPUT \ + , [loop_cnt_1]"m"(loop_cnt[1]) \ + , [loop_cnt_2]"m"(loop_cnt[2]) \ + , [loop_cnt_3]"m"(loop_cnt[3]) \ + , [loop_cnt_4]"m"(loop_cnt[4]) \ + , [loop_cnt_5]"m"(loop_cnt[5]) \ + , [loop_cnt_6]"m"(loop_cnt[6]) In this case, it was a input operand list of inline assembly, so I could not add parentheses to enclose them. Except for these two kinds of error, there are two erros could be solved by adding parentheses. I also checked it and I think it would be safe. So, I mimic the implementations of other architecture without using parenthese. If needed, I think can add the parenthese to solve these two errors. Thanks
linux-kselftest-mirror@lists.linaro.org
-
Mathieu Desnoyers -
Palmer Dabbelt -
Vincent Chen