Hi Dave, Oleg, and All,
Short story ===========
It seems to me that ARMv7 uprobes need proper icache flush after xol write. Please look at [1] discussion for similar issue on ppc.
It seems that flush_dcache_page was sufficient for latter architectures of PPC but it does not look that it is good enough for ARMv7.
AFAIK know ARM V7 does not have "snooping Harvard caches" and needs something like __cpuc_coherent_user_range function call to sync up icache and dcache after instruction write through dcache.
Patch that I propose follows this cover letter. There I introduced weak arch_uprobe_xol_sync_dcache_icache function that does traditional flush_dcache_page call and I redefined this function to __cpuc_coherent_user_range call in ARM v7 > case.
[1] http://linux-kernel.2935.n7.nabble.com/Re-PATCH-6-9-uprobes-flush-cache-afte...
Longer story ============
I was trying Dave's armv7 uprobes with SystemTap on Arndale board. I used Linaro linux branch 3.14 based that contained Dave's armv7 uprobes topic code. I believe it should be pretty much the same as armv7 uprobes code that went to Russell's tree.
I was able to do one function simple test - it worked fine for me. But when I've tried to run many function like "probe process("foobar").function("*")" probe SystemTap my target process always crashed.
After quite a bit of chasing the issue, I was able to come up with test case that shows several probes installed against 'ls' process. First probe placed at 'push {r4, r5, r6, r7, r8, r9, r10, r11, lr}' instruction, which is first in _getopt_initialize function, then script adds few more probes at _getopt_initialize addresses that are executed latter. And in those probes I dump registers set and top of stack. By looking at execution of script one may easily conclude that it looks like that for each probe 'push {r4, r5, r6, r7, r8, r9, r10, r11, lr}' instruction is always executed - one may see 36 bytes increase of stack size and see copy of corresponding registers on the stack.
The code path is the following:
handle_swbp -> pre_ssout
pre_ssout -> xol_get_insn_slot
xol_get_insn_slot -> copy_to_page xol_get_insn_slot -> flush_dcache_page
pre_ssout -> arch_uprobe_pre_xol
pre_ssout function calls xol_get_insn_slot which finds available slot in XOL area, that is mapped into user process and copies required instruction into xol slot. After that it calls flush_dcache_page, but icache is not flushed in ARM case by this function. So I think the following thing happens: first time first xol slot got 'push {r4, r5, r6, r7, r8, r9, r10, r11, lr}' instruction and it retrieved into icache. Latter when other probes are executed the same first slot of xol area it will get different instruction but because icache is not flushed CPU keep executing 'push' instruction.
When I add the following testing patch that flush icache in arch_uprobe_pre_xol
[kamensky@kamensky-w530 git]$ git diff diff --git a/arch/arm/kernel/uprobes.c b/arch/arm/kernel/uprobes.c index f9bacee..ef34623 100644 --- a/arch/arm/kernel/uprobes.c +++ b/arch/arm/kernel/uprobes.c @@ -117,6 +117,8 @@ int arch_uprobe_pre_xol(struct arch_uprobe *auprobe, struct pt_regs *regs) { struct uprobe_task *utask = current->utask;
+ __flush_icache_all(); + if (auprobe->prehandler) auprobe->prehandler(auprobe, &utask->autask, regs);
it fixes my test problem, 'ls' executes fine, I am able to trace it with "process("foobar").function("*")" traces.
Having complete icache flush is far from optimal. I was looking for better and more optimal and targeted icache flush.
In discussion [1] and corresponding commit it was alluded that flush_icache_user_range should help but I don't think it will work for ARMv7. It seems that for ARMv7 flush_icache_user_range is not quite correct - it calls again 'flush_dcache_page(page)' which I don't think touches icache (other strange thing that it ignores len completely).
I looked at armv7 kprobes code, in similar area arch_prepare_kprobe function calls arch_prepare_kprobe which calls flush_insn macro which is defined as call flush_icache_range, which in turn defines as __cpuc_coherent_kern_range(s,e).
I looked at ptrace breakpoint write code as well that deals with similar issue.
As far as I see it seems that the best function to sync up icache and dcache of user land process is to use __cpuc_coherent_user_range function which sync up dcache and icache memory region of given size and at given address on current core. In case of Arndale it goes through v7_coherent_user_range function. Given uprobes single step like use case it seem syncing up cache on current core is sufficient. If someone can confirm that __cpuc_coherent_user_range is right choice for this situation it would be nice. Test shows it work, we would need to know for sure.
Next issue is how to integrate this call with cpu independent uprobes code. I introduced week arch_uprobe_xol_sync_dcache_icache function that calls flush_dcache_page as before, and for ARMv7 defined one that calls __cpuc_coherent_user_range. As far I understand flush_dcache_page was introduced for some relatively recent ppc CPU, it seems to be noop on x86. I was thinking that default weak version of arch_uprobe_xol_sync_dcache_icache should be empty and ppc define one that calls flush_dcache_page. But I have no way to test ppc case so I decided do conservative implementation and keep default version calling flush_dcache_page as it was before.
Thanks, Victor
Appendix: Test case that shows push instruction executed several times ======================================================================
SystemTap test script ---------------------
root@genericarmv7a:~/systemtap/test# cat ls_t4_not_r4.stp function print_memory(addr:long, size:long) { i = 0; addr2 = addr; while (i < size) { printf("0x%8.8x: ", addr2); while (i < size) { printf ("0x%8.8x ", user_uint32(addr2)); addr2 = addr2 + 4; i = i + 4; if (i%16 == 0) { break; } } printf("\n"); } }
probe process("/bin/ls.coreutils").function("_getopt_initialize") { sp = register("sp"); print_memory(sp, 64); print_regs(); printf("-> _getopt_initialize\n"); }
probe process("/bin/ls.coreutils").statement(0x0001b2e8) { sp = register("sp"); print_memory(sp, 64);
print_regs(); printf("-> 0x0001b2e8\n"); }
probe process("/bin/ls.coreutils").statement(0x0001b2d8) { sp = register("sp"); print_memory(sp, 64);
print_regs(); printf("-> 0x0001b2d8\n"); }
probe process("/bin/ls.coreutils").statement(0x0001b408) { sp = register("sp"); print_memory(sp, 64);
print_regs(); printf("-> 0x0001b408\n"); }
probe process("/bin/ls.coreutils").function("_getopt_internal_r") { // sp = register("sp"); // print_memory(sp, 64);
print_regs(); printf("-> _getopt_internal_r\n"); }
execution of script -------------------
Look at log of script execution. Check how $sp changes at each probe (+36). And see that r4, r5, r6, r7, r8, r9, r10, r11, lr registers are always at top of stack.
root@genericarmv7a:~/systemtap/test# stap -U -v ls_t4_not_r4.stp Pass 1: parsed user script and 100 library script(s) using 20520virt/16336res/1728shr/15260data kb, in 410usr/30sys/437real ms. Pass 2: analyzed script: 5 probe(s), 8 function(s), 3 embed(s), 2 global(s) using 21984virt/18568res/2620shr/16724data kb, in 1280usr/870sys/2507real ms. Pass 3: translated to C into "/tmp/stapPmXkE6/stap_06d9647b8c7b3327fecfc3259c39e0ed_6448_src.c" using 21984virt/18796res/2848shr/16724data kb, in 70usr/260sys/330real ms. Pass 4: compiled C into "stap_06d9647b8c7b3327fecfc3259c39e0ed_6448.ko" in 16050usr/1290sys/18328real ms. Pass 5: starting run. CPU: 1pc : [<0001b2cc>] lr : [<0001c1ac>] sp : 7eeb2940 ip : 00000001 fp : 0002a2d8 r10: 76ffe000 r9 : 0002a2d8 r8 : 7eeb29c4 r7 : 00000000 r6 : 00000000 r5 : 0002a2b0 r4 : 0002af40 r3 : 0001e914 r2 : 00020dc4 r1 : 7eeb2de4 r0 : 00000001 Flags: nZCv IRQs on FIQs on Mode USER_32 Segment user Control: 30C5387D Table: AC5A14C0 DAC: 55555555 -> _getopt_internal_r 0x7eeb28d8: 0xffffffff 0x00000000 0x76e97e34 0x76ffb8f8 0x7eeb28e8: 0x00000000 0x00000038 0x00000000 0x76f560c8 0x7eeb28f8: 0x00000077 0x00001500 0x00000005 0x000018b2 0x7eeb2908: 0x00000a3b 0x7f1c0300 0x01000415 0x76ffa4c0 CPU: 1pc : [<0001b2d8>] lr : [<0001c1ac>] sp : 7eeb28d8 ip : 00000001 fp : 0002a2d8 r10: 00000001 r9 : 0002a2d8 r8 : 7eeb29c4 r7 : 00000000 r6 : 00000000 r5 : 0002a2b0 r4 : 0002af40 r3 : 0001e914 r2 : 00020dc4 r1 : 7eeb2de4 r0 : 00000001 Flags: nzCv IRQs on FIQs on Mode USER_32 Segment user Control: 30C5387D Table: AC5A14C0 DAC: 55555555 -> 0x0001b2d8 0x7eeb28b4: 0x0002af40 0x0002a2b0 0x00000000 0x00000000 0x7eeb28c4: 0x7eeb29c4 0x0002a2d8 0x7eeb2de4 0x0001e914 0x7eeb28d4: 0x0001c1ac 0xffffffff 0x00000000 0x76e97e34 0x7eeb28e4: 0x76ffb8f8 0x00000000 0x00000038 0x00000000 CPU: 1pc : [<0001b2e8>] lr : [<0001c1ac>] sp : 7eeb28b4 ip : 00000001 fp : 0002a2d8 r10: 00000001 r9 : 0002a2d8 r8 : 7eeb29c4 r7 : 00000000 r6 : 00000000 r5 : 0002a2b0 r4 : 0002af40 r3 : 00000001 r2 : 00020dc4 r1 : 7eeb2de4 r0 : 00000001 Flags: nzCv IRQs on FIQs on Mode USER_32 Segment user Control: 30C5387D Table: AC5A14C0 DAC: 55555555 -> 0x0001b2e8 0x7eeb2890: 0x0002af40 0x0002a2b0 0x00000000 0x00000000 0x7eeb28a0: 0x7eeb29c4 0x00000001 0x00000001 0x0002a2d8 0x7eeb28b0: 0x0001c1ac 0x0002af40 0x0002a2b0 0x00000000 0x7eeb28c0: 0x00000000 0x7eeb29c4 0x0002a2d8 0x7eeb2de4 CPU: 1pc : [<0001b3f8>] lr : [<0001c1ac>] sp : 7eeb2890 ip : 00000001 fp : 0002a2d8 r10: 00000001 r9 : 0002a2d8 r8 : 7eeb29c4 r7 : 00000000 r6 : 00000000 r5 : 0002a2b0 r4 : 0002af40 r3 : 00000001 r2 : 00000000 r1 : 7eeb2de4 r0 : 00000001 Flags: nZCv IRQs on FIQs on Mode USER_32 Segment user Control: 30C5387D Table: AC5A14C0 DAC: 55555555 -> _getopt_initialize 0x7eeb286c: 0x0002af40 0x0002a2b0 0x00000000 0x00000000 0x7eeb287c: 0x7eeb29c4 0x0002a2d8 0x00000001 0x0002a2d8 0x7eeb288c: 0x0001c1ac 0x0002af40 0x0002a2b0 0x00000000 0x7eeb289c: 0x00000000 0x7eeb29c4 0x00000001 0x00000001 CPU: 1pc : [<0001b408>] lr : [<0001c1ac>] sp : 7eeb286c ip : 00000001 fp : 0002a2d8 r10: 00000001 r9 : 0002a2d8 r8 : 7eeb29c4 r7 : 00000000 r6 : 00000000 r5 : 0002a2b0 r4 : 0002af40 r3 : 00000001 r2 : 00000000 r1 : 7eeb2de4 r0 : 00000001 Flags: nzCv IRQs on FIQs on Mode USER_32 Segment user Control: 30C5387D Table: AC5A14C0 DAC: 55555555 -> 0x0001b408
gdb session of crashed ls command =================================
Looking at core of crashed ls process. Look at disassemble of function you can see uprobe breakpoints as <UNDEFINED> instruction. For instructions that should be there and which are executed through xol look at next section.
root@genericarmv7a:~# gdb /bin/ls.coreutils -c core GNU gdb (Linaro GDB) 7.6.1-2013.10 Copyright (C) 2013 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "arm-oe-linux-gnueabi". For bug reporting instructions, please see: http://bugs.launchpad.net/gdb-linaro/... Reading symbols from /bin/ls.coreutils...Reading symbols from /bin/.debug/ls.coreutils...done. done.
warning: core file may not match specified executable file. [New LWP 12424] Core was generated by `ls'. Program terminated with signal 11, Segmentation fault. #0 _getopt_initialize (argc=1, argv=0x7eeb29c4, posixly_correct=175936, d=0x2af40 <getopt_data>, optstring=0x2a2b0 <rpl_optind> "\001") at lib/getopt.c:241 241 if (optstring[0] == '-') (gdb) bt #0 _getopt_initialize (argc=1, argv=0x7eeb29c4, posixly_correct=175936, d=0x2af40 <getopt_data>, optstring=0x2a2b0 <rpl_optind> "\001") at lib/getopt.c:241 #1 _getopt_internal_r (argc=1, argv=0x7eeb29c4, optstring=0x2a2b0 <rpl_optind> "\001", longopts=0x2a2d8 <interrupt_signal>, longind=0x1c1ac <rpl_getopt_internal+72>, long_only=175936, d=0x2a2b0 <rpl_optind>, posixly_correct=0) at lib/getopt.c:361 #2 0x0002a2d8 in ?? () Cannot access memory at address 0x1 #3 0x0002a2d8 in ?? () Cannot access memory at address 0x1 Backtrace stopped: previous frame identical to this frame (corrupt stack?) (gdb) info reg r0 0x1 1 r1 0x7eeb2de4 2129341924 r2 0x0 0 r3 0x1 1 r4 0x2af40 175936 r5 0x2a2b0 172720 r6 0x0 0 r7 0x0 0 r8 0x7eeb29c4 2129340868 r9 0x2a2d8 172760 r10 0x1 1 r11 0x2a2d8 172760 r12 0x0 0 sp 0x7eeb2848 0x7eeb2848 lr 0x1c1ac 115116 pc 0x1b420 0x1b420 <_getopt_internal_r+340> cpsr 0x200f0010 537853968 (gdb) set height 0 (gdb) disassemble _getopt_internal_r Dump of assembler code for function _getopt_internal_r: 0x0001b2cc <+0>: ; <UNDEFINED> instruction: 0xe7f001f9 0x0001b2d0 <+4>: sub sp, sp, #68 ; 0x44 0x0001b2d4 <+8>: subs r10, r0, #0 0x0001b2d8 <+12>: ; <UNDEFINED> instruction: 0xe7f001f9 0x0001b2dc <+16>: str r3, [sp, #28] 0x0001b2e0 <+20>: str r1, [sp, #24] 0x0001b2e4 <+24>: ldr r3, [r4, #4] 0x0001b2e8 <+28>: ; <UNDEFINED> instruction: 0xe7f001f9 0x0001b2ec <+32>: str r3, [sp, #20] 0x0001b2f0 <+36>: ble 0x1b4f8 <_getopt_internal_r+556> 0x0001b2f4 <+40>: ldr r3, [r4] 0x0001b2f8 <+44>: mov r2, #0 0x0001b2fc <+48>: str r2, [r4, #12] 0x0001b300 <+52>: cmp r3, r2 0x0001b304 <+56>: beq 0x1b3f0 <_getopt_internal_r+292> 0x0001b308 <+60>: ldr r2, [r4, #16] 0x0001b30c <+64>: cmp r2, #0 0x0001b310 <+68>: beq 0x1b3f8 <_getopt_internal_r+300> 0x0001b314 <+72>: ldr r5, [sp, #12] 0x0001b318 <+76>: ldrb r3, [r5] 0x0001b31c <+80>: cmp r3, #45 ; 0x2d 0x0001b320 <+84>: cmpne r3, #43 ; 0x2b 0x0001b324 <+88>: ldrbeq r3, [r5, #1] 0x0001b328 <+92>: addeq r5, r5, #1 0x0001b32c <+96>: streq r5, [sp, #12] 0x0001b330 <+100>: cmp r3, #58 ; 0x3a 0x0001b334 <+104>: ldr r9, [r4, #20] 0x0001b338 <+108>: ldr r12, [sp, #20] 0x0001b33c <+112>: moveq r12, #0 0x0001b340 <+116>: cmp r9, #0 0x0001b344 <+120>: str r12, [sp, #20] 0x0001b348 <+124>: beq 0x1b458 <_getopt_internal_r+396> 0x0001b34c <+128>: ldrb r3, [r9] 0x0001b350 <+132>: cmp r3, #0 0x0001b354 <+136>: beq 0x1b458 <_getopt_internal_r+396> 0x0001b358 <+140>: str r9, [sp, #16] 0x0001b35c <+144>: ldr r12, [sp, #28] 0x0001b360 <+148>: cmp r12, #0 0x0001b364 <+152>: beq 0x1b880 <_getopt_internal_r+1460> 0x0001b368 <+156>: ldr r3, [r4] 0x0001b36c <+160>: ldr r5, [sp, #24] 0x0001b370 <+164>: str r3, [sp, #32] 0x0001b374 <+168>: ldr r3, [r5, r3, lsl #2] 0x0001b378 <+172>: ldrb r1, [r3, #1] 0x0001b37c <+176>: cmp r1, #45 ; 0x2d 0x0001b380 <+180>: beq 0x1b564 <_getopt_internal_r+664> 0x0001b384 <+184>: ldr r12, [sp, #108] ; 0x6c 0x0001b388 <+188>: cmp r12, #0 0x0001b38c <+192>: bne 0x1b548 <_getopt_internal_r+636> 0x0001b390 <+196>: ldr r9, [sp, #16] 0x0001b394 <+200>: add r6, r9, #1 0x0001b398 <+204>: str r6, [r4, #20] 0x0001b39c <+208>: ldrb r5, [r9] 0x0001b3a0 <+212>: ldr r0, [sp, #12] 0x0001b3a4 <+216>: mov r1, r5 0x0001b3a8 <+220>: bl 0x9db4 <strchr> 0x0001b3ac <+224>: ldrb r3, [r6] 0x0001b3b0 <+228>: cmp r3, #0 0x0001b3b4 <+232>: ldreq r3, [r4] 0x0001b3b8 <+236>: addeq r3, r3, #1 0x0001b3bc <+240>: streq r3, [r4] 0x0001b3c0 <+244>: sub r3, r5, #58 ; 0x3a 0x0001b3c4 <+248>: cmp r0, #0 0x0001b3c8 <+252>: cmpne r3, #1 0x0001b3cc <+256>: bhi 0x1b6f0 <_getopt_internal_r+1060> 0x0001b3d0 <+260>: ldr r12, [sp, #20] 0x0001b3d4 <+264>: cmp r12, #0 0x0001b3d8 <+268>: bne 0x1b6b0 <_getopt_internal_r+996> 0x0001b3dc <+272>: mov r1, #63 ; 0x3f 0x0001b3e0 <+276>: str r5, [r4, #8] 0x0001b3e4 <+280>: mov r0, r1 0x0001b3e8 <+284>: add sp, sp, #68 ; 0x44 0x0001b3ec <+288>: pop {r4, r5, r6, r7, r8, r9, r10, r11, pc} 0x0001b3f0 <+292>: mov r3, #1 0x0001b3f4 <+296>: str r3, [r4] 0x0001b3f8 <+300>: ; <UNDEFINED> instruction: 0xe7f001f9 0x0001b3fc <+304>: str r3, [r4, #36] ; 0x24 0x0001b400 <+308>: cmp r5, #0 0x0001b404 <+312>: str r3, [r4, #32] 0x0001b408 <+316>: ; <UNDEFINED> instruction: 0xe7f001f9 0x0001b40c <+320>: str r3, [r4, #20] 0x0001b410 <+324>: movne r0, #1 0x0001b414 <+328>: beq 0x1b530 <_getopt_internal_r+612> 0x0001b418 <+332>: ldr r12, [sp, #12] 0x0001b41c <+336>: str r0, [r4, #28] => 0x0001b420 <+340>: ldrb r3, [r12] 0x0001b424 <+344>: cmp r3, #45 ; 0x2d 0x0001b428 <+348>: beq 0x1b848 <_getopt_internal_r+1404> 0x0001b42c <+352>: cmp r3, #43 ; 0x2b 0x0001b430 <+356>: beq 0x1b868 <_getopt_internal_r+1436> 0x0001b434 <+360>: cmp r0, #0
disassemble of _getopt_internal_r function ------------------------------------------
Just to see what instructions got breakpoint
(gdb) disassemble _getopt_internal_r Dump of assembler code for function _getopt_internal_r: 0x0001b2cc <+0>: push {r4, r5, r6, r7, r8, r9, r10, r11, lr} 0x0001b2d0 <+4>: sub sp, sp, #68 ; 0x44 0x0001b2d4 <+8>: subs r10, r0, #0 0x0001b2d8 <+12>: ldr r4, [sp, #112] ; 0x70 0x0001b2dc <+16>: str r3, [sp, #28] 0x0001b2e0 <+20>: str r1, [sp, #24] 0x0001b2e4 <+24>: ldr r3, [r4, #4] 0x0001b2e8 <+28>: str r2, [sp, #12] 0x0001b2ec <+32>: str r3, [sp, #20] 0x0001b2f0 <+36>: ble 0x1b4f8 <_getopt_internal_r+556> 0x0001b2f4 <+40>: ldr r3, [r4] 0x0001b2f8 <+44>: mov r2, #0 0x0001b2fc <+48>: str r2, [r4, #12] 0x0001b300 <+52>: cmp r3, r2 0x0001b304 <+56>: beq 0x1b3f0 <_getopt_internal_r+292> 0x0001b308 <+60>: ldr r2, [r4, #16] 0x0001b30c <+64>: cmp r2, #0 0x0001b310 <+68>: beq 0x1b3f8 <_getopt_internal_r+300> 0x0001b314 <+72>: ldr r5, [sp, #12] 0x0001b318 <+76>: ldrb r3, [r5] 0x0001b31c <+80>: cmp r3, #45 ; 0x2d 0x0001b320 <+84>: cmpne r3, #43 ; 0x2b 0x0001b324 <+88>: ldrbeq r3, [r5, #1] 0x0001b328 <+92>: addeq r5, r5, #1 0x0001b32c <+96>: streq r5, [sp, #12] 0x0001b330 <+100>: cmp r3, #58 ; 0x3a 0x0001b334 <+104>: ldr r9, [r4, #20] 0x0001b338 <+108>: ldr r12, [sp, #20] 0x0001b33c <+112>: moveq r12, #0 0x0001b340 <+116>: cmp r9, #0 0x0001b344 <+120>: str r12, [sp, #20] 0x0001b348 <+124>: beq 0x1b458 <_getopt_internal_r+396> 0x0001b34c <+128>: ldrb r3, [r9] 0x0001b350 <+132>: cmp r3, #0 0x0001b354 <+136>: beq 0x1b458 <_getopt_internal_r+396> 0x0001b358 <+140>: str r9, [sp, #16] 0x0001b35c <+144>: ldr r12, [sp, #28] 0x0001b360 <+148>: cmp r12, #0 0x0001b364 <+152>: beq 0x1b880 <_getopt_internal_r+1460> 0x0001b368 <+156>: ldr r3, [r4] 0x0001b36c <+160>: ldr r5, [sp, #24] 0x0001b370 <+164>: str r3, [sp, #32] 0x0001b374 <+168>: ldr r3, [r5, r3, lsl #2] 0x0001b378 <+172>: ldrb r1, [r3, #1] 0x0001b37c <+176>: cmp r1, #45 ; 0x2d 0x0001b380 <+180>: beq 0x1b564 <_getopt_internal_r+664> 0x0001b384 <+184>: ldr r12, [sp, #108] ; 0x6c 0x0001b388 <+188>: cmp r12, #0 0x0001b38c <+192>: bne 0x1b548 <_getopt_internal_r+636> 0x0001b390 <+196>: ldr r9, [sp, #16] 0x0001b394 <+200>: add r6, r9, #1 0x0001b398 <+204>: str r6, [r4, #20] 0x0001b39c <+208>: ldrb r5, [r9] 0x0001b3a0 <+212>: ldr r0, [sp, #12] 0x0001b3a4 <+216>: mov r1, r5 0x0001b3a8 <+220>: bl 0x9db4 <strchr> 0x0001b3ac <+224>: ldrb r3, [r6] 0x0001b3b0 <+228>: cmp r3, #0 0x0001b3b4 <+232>: ldreq r3, [r4] 0x0001b3b8 <+236>: addeq r3, r3, #1 0x0001b3bc <+240>: streq r3, [r4] 0x0001b3c0 <+244>: sub r3, r5, #58 ; 0x3a 0x0001b3c4 <+248>: cmp r0, #0 0x0001b3c8 <+252>: cmpne r3, #1 0x0001b3cc <+256>: bhi 0x1b6f0 <_getopt_internal_r+1060> 0x0001b3d0 <+260>: ldr r12, [sp, #20] 0x0001b3d4 <+264>: cmp r12, #0 0x0001b3d8 <+268>: bne 0x1b6b0 <_getopt_internal_r+996> 0x0001b3dc <+272>: mov r1, #63 ; 0x3f 0x0001b3e0 <+276>: str r5, [r4, #8] 0x0001b3e4 <+280>: mov r0, r1 0x0001b3e8 <+284>: add sp, sp, #68 ; 0x44 0x0001b3ec <+288>: pop {r4, r5, r6, r7, r8, r9, r10, r11, pc} 0x0001b3f0 <+292>: mov r3, #1 0x0001b3f4 <+296>: str r3, [r4] 0x0001b3f8 <+300>: ldr r5, [sp, #116] ; 0x74 0x0001b3fc <+304>: str r3, [r4, #36] ; 0x24 0x0001b400 <+308>: cmp r5, #0 0x0001b404 <+312>: str r3, [r4, #32] 0x0001b408 <+316>: mov r3, #0 0x0001b40c <+320>: str r3, [r4, #20] 0x0001b410 <+324>: movne r0, #1 0x0001b414 <+328>: beq 0x1b530 <_getopt_internal_r+612> 0x0001b418 <+332>: ldr r12, [sp, #12] 0x0001b41c <+336>: str r0, [r4, #28] 0x0001b420 <+340>: ldrb r3, [r12] 0x0001b424 <+344>: cmp r3, #45 ; 0x2d 0x0001b428 <+348>: beq 0x1b848 <_getopt_internal_r+1404> 0x0001b42c <+352>: cmp r3, #43 ; 0x2b 0x0001b430 <+356>: beq 0x1b868 <_getopt_internal_r+1436> 0x0001b434 <+360>: cmp r0, #0
Victor Kamensky (1): ARM: uprobes need icache flush after xol write
arch/arm/kernel/uprobes.c | 6 ++++++ include/linux/uprobes.h | 3 +++ kernel/events/uprobes.c | 20 +++++++++++++++----- 3 files changed, 24 insertions(+), 5 deletions(-)
After instruction write into xol area, on ARM V7 architecture code need to flush icache to sync up dcache and icache. Having just 'flush_dcache_page(page)' call is not enough - it is possible to have stale instruction sitting in icache for given xol area slot address.
Introduce arch_uprobe_xol_sync_dcache_icache weak function that by default calls 'flush_dcache_page(page)' and on ARM define one that calls __cpuc_coherent_user_range.
Signed-off-by: Victor Kamensky victor.kamensky@linaro.org --- arch/arm/kernel/uprobes.c | 6 ++++++ include/linux/uprobes.h | 3 +++ kernel/events/uprobes.c | 20 +++++++++++++++----- 3 files changed, 24 insertions(+), 5 deletions(-)
diff --git a/arch/arm/kernel/uprobes.c b/arch/arm/kernel/uprobes.c index f9bacee..841750b 100644 --- a/arch/arm/kernel/uprobes.c +++ b/arch/arm/kernel/uprobes.c @@ -113,6 +113,12 @@ int arch_uprobe_analyze_insn(struct arch_uprobe *auprobe, struct mm_struct *mm, return 0; }
+void arch_uprobe_xol_sync_dcache_icache(struct page *page, + unsigned long addr, unsigned long size) +{ + __cpuc_coherent_user_range(addr, size); +} + int arch_uprobe_pre_xol(struct arch_uprobe *auprobe, struct pt_regs *regs) { struct uprobe_task *utask = current->utask; diff --git a/include/linux/uprobes.h b/include/linux/uprobes.h index edff2b9..2fe3dfd 100644 --- a/include/linux/uprobes.h +++ b/include/linux/uprobes.h @@ -32,6 +32,7 @@ struct vm_area_struct; struct mm_struct; struct inode; struct notifier_block; +struct page;
#define UPROBE_HANDLER_REMOVE 1 #define UPROBE_HANDLER_MASK 1 @@ -127,6 +128,8 @@ extern int arch_uprobe_exception_notify(struct notifier_block *self, unsigned l extern void arch_uprobe_abort_xol(struct arch_uprobe *aup, struct pt_regs *regs); extern unsigned long arch_uretprobe_hijack_return_addr(unsigned long trampoline_vaddr, struct pt_regs *regs); extern bool __weak arch_uprobe_ignore(struct arch_uprobe *aup, struct pt_regs *regs); +extern void __weak arch_uprobe_xol_sync_dcache_icache(struct page *page, + unsigned long addr, unsigned long size); #else /* !CONFIG_UPROBES */ struct uprobes_state { }; diff --git a/kernel/events/uprobes.c b/kernel/events/uprobes.c index 04709b6..0027883 100644 --- a/kernel/events/uprobes.c +++ b/kernel/events/uprobes.c @@ -1299,11 +1299,9 @@ static unsigned long xol_get_insn_slot(struct uprobe *uprobe) /* Initialize the slot */ copy_to_page(area->page, xol_vaddr, &uprobe->arch.ixol, sizeof(uprobe->arch.ixol)); - /* - * We probably need flush_icache_user_range() but it needs vma. - * This should work on supported architectures too. - */ - flush_dcache_page(area->page); + + arch_uprobe_xol_sync_dcache_icache(area->page, + xol_vaddr, sizeof(uprobe->arch.ixol));
return xol_vaddr; } @@ -1346,6 +1344,18 @@ static void xol_free_insn_slot(struct task_struct *tsk) } }
+void __weak arch_uprobe_xol_sync_dcache_icache(struct page *page, + unsigned long addr, unsigned long len) +{ + /* + * We probably need flush_icache_user_range() but it needs vma. + * This should work on most of architectures by default. If + * architecture needs to do something different it can define + * its own version of the function. + */ + flush_dcache_page(page); +} + /** * uprobe_get_swbp_addr - compute address of swbp given post-swbp regs * @regs: Reflects the saved state of the task after it has hit a breakpoint
On Mon, Apr 07, 2014 at 08:04:20PM -0700, Victor Kamensky wrote:
After instruction write into xol area, on ARM V7 architecture code need to flush icache to sync up dcache and icache. Having just 'flush_dcache_page(page)' call is not enough - it is possible to have stale instruction sitting in icache for given xol area slot address.
Introduce arch_uprobe_xol_sync_dcache_icache weak function that by default calls 'flush_dcache_page(page)' and on ARM define one that calls __cpuc_coherent_user_range.
You're right that something is needed.
Would flush_icache_range() not be the correct thing to call here?
Sync'ing the I and D sides for a whole page seems excessive to install a probe.
I believe __cpuc_coherent_kern_range() is not intended as an API, and should only be used in special situations in arch-specific code. It's the correct operation, but arm provides flush_icache_range as a wrapper to call.
flush_icache_range() is documented, with the correct effect, in Documentation/cachetlb.txt, so I believe that all arches should provide it, even if it's a no-op.
Cheers ---Dave
Signed-off-by: Victor Kamensky victor.kamensky@linaro.org
arch/arm/kernel/uprobes.c | 6 ++++++ include/linux/uprobes.h | 3 +++ kernel/events/uprobes.c | 20 +++++++++++++++----- 3 files changed, 24 insertions(+), 5 deletions(-)
diff --git a/arch/arm/kernel/uprobes.c b/arch/arm/kernel/uprobes.c index f9bacee..841750b 100644 --- a/arch/arm/kernel/uprobes.c +++ b/arch/arm/kernel/uprobes.c @@ -113,6 +113,12 @@ int arch_uprobe_analyze_insn(struct arch_uprobe *auprobe, struct mm_struct *mm, return 0; } +void arch_uprobe_xol_sync_dcache_icache(struct page *page,
unsigned long addr, unsigned long size)
+{
- __cpuc_coherent_user_range(addr, size);
+}
int arch_uprobe_pre_xol(struct arch_uprobe *auprobe, struct pt_regs *regs) { struct uprobe_task *utask = current->utask; diff --git a/include/linux/uprobes.h b/include/linux/uprobes.h index edff2b9..2fe3dfd 100644 --- a/include/linux/uprobes.h +++ b/include/linux/uprobes.h @@ -32,6 +32,7 @@ struct vm_area_struct; struct mm_struct; struct inode; struct notifier_block; +struct page; #define UPROBE_HANDLER_REMOVE 1 #define UPROBE_HANDLER_MASK 1 @@ -127,6 +128,8 @@ extern int arch_uprobe_exception_notify(struct notifier_block *self, unsigned l extern void arch_uprobe_abort_xol(struct arch_uprobe *aup, struct pt_regs *regs); extern unsigned long arch_uretprobe_hijack_return_addr(unsigned long trampoline_vaddr, struct pt_regs *regs); extern bool __weak arch_uprobe_ignore(struct arch_uprobe *aup, struct pt_regs *regs); +extern void __weak arch_uprobe_xol_sync_dcache_icache(struct page *page,
unsigned long addr, unsigned long size);
#else /* !CONFIG_UPROBES */ struct uprobes_state { }; diff --git a/kernel/events/uprobes.c b/kernel/events/uprobes.c index 04709b6..0027883 100644 --- a/kernel/events/uprobes.c +++ b/kernel/events/uprobes.c @@ -1299,11 +1299,9 @@ static unsigned long xol_get_insn_slot(struct uprobe *uprobe) /* Initialize the slot */ copy_to_page(area->page, xol_vaddr, &uprobe->arch.ixol, sizeof(uprobe->arch.ixol));
- /*
* We probably need flush_icache_user_range() but it needs vma.
* This should work on supported architectures too.
*/
- flush_dcache_page(area->page);
- arch_uprobe_xol_sync_dcache_icache(area->page,
xol_vaddr, sizeof(uprobe->arch.ixol));
return xol_vaddr; } @@ -1346,6 +1344,18 @@ static void xol_free_insn_slot(struct task_struct *tsk) } } +void __weak arch_uprobe_xol_sync_dcache_icache(struct page *page,
unsigned long addr, unsigned long len)
+{
- /*
* We probably need flush_icache_user_range() but it needs vma.
* This should work on most of architectures by default. If
* architecture needs to do something different it can define
* its own version of the function.
*/
- flush_dcache_page(page);
+}
/**
- uprobe_get_swbp_addr - compute address of swbp given post-swbp regs
- @regs: Reflects the saved state of the task after it has hit a breakpoint
-- 1.8.1.4
linaro-kernel mailing list linaro-kernel@lists.linaro.org http://lists.linaro.org/mailman/listinfo/linaro-kernel
On Tue, Apr 08, 2014 at 09:24:50AM +0100, Dave Martin wrote:
You're right that something is needed.
Would flush_icache_range() not be the correct thing to call here?
Sync'ing the I and D sides for a whole page seems excessive to install a probe.
I believe __cpuc_coherent_kern_range() is not intended as an API, and should only be used in special situations in arch-specific code. It's the correct operation, but arm provides flush_icache_range as a wrapper to call.
flush_icache_range() is documented, with the correct effect, in Documentation/cachetlb.txt, so I believe that all arches should provide it, even if it's a no-op.
The real question is, what is this code in xol_get_insn_slot() trying to do?
1. area->page is an anonymous page - it was allocated by alloc_page(GFP_HIGHUSER). This makes use of flush_dcache_page() questionable at best.
2. xol_get_insn_slot() appears to want to copy instructions into this page via the kernel mapping such that they're visible to userspace.
So, the first thing that we need to do is to push the written data out to a point where the instruction stream can see it. Then we need to ensure that the instruction stream can see the new instructions.
Practically for ARM, that means pushing it out of the D-cache and then making sure that there are no I-cache lines associated with the new instructions.
With VIVT instruction caches (yes, we still have them on ARMv7), we have to flush the I-cache lines associated with the userspace addresses - flushing the I-cache lines for the kernel mapping doesn't do anything for the stale lines in the userspace mapping.
copy_to_page() kmap_atomic()'s the page - this doesn't take account of VIPT aliasing caches, and so the writes may be in an alias of the user page. Plus, when the page is kunmap_atomic()'d, there's no guarantee that the written lines will be flushed out.
copy_from_page() suffers from a similar problem, though its saving grace is that the user mapping is execute-only (iow, read+execute only) so should never end up with any dirty cache lines associated with it.
flush_dcache_page() will ensure that the D-cache lines are flushed out on ARM (even though it's only defined to have an effect for page cache pages) but it does nothing for the I-cache. So, although it seems to partially work, it's definitely the wrong interface here.
However... the normal way we do this (for example, in the case of ptrace) is via copy_to_user_page() / copy_from_user_page(), and we have some pretty complex handling via those functions to deal with writing to the page and having the affected thread running on a different CPU to the one we wrote the new instructions. I'd recommend that uprobes looks at trying to re-use some of this infrastructure rather than re-inventing this wheel.
On 04/08/14 07:46, Russell King - ARM Linux wrote:
On Tue, Apr 08, 2014 at 09:24:50AM +0100, Dave Martin wrote:
You're right that something is needed.
Would flush_icache_range() not be the correct thing to call here?
Sync'ing the I and D sides for a whole page seems excessive to install a probe.
I believe __cpuc_coherent_kern_range() is not intended as an API, and should only be used in special situations in arch-specific code. It's the correct operation, but arm provides flush_icache_range as a wrapper to call.
flush_icache_range() is documented, with the correct effect, in Documentation/cachetlb.txt, so I believe that all arches should provide it, even if it's a no-op.
The real question is, what is this code in xol_get_insn_slot() trying to do?
area->page is an anonymous page - it was allocated by alloc_page(GFP_HIGHUSER). This makes use of flush_dcache_page() questionable at best.
xol_get_insn_slot() appears to want to copy instructions into this page via the kernel mapping such that they're visible to userspace.
So, the first thing that we need to do is to push the written data out to a point where the instruction stream can see it. Then we need to ensure that the instruction stream can see the new instructions.
Practically for ARM, that means pushing it out of the D-cache and then making sure that there are no I-cache lines associated with the new instructions.
With VIVT instruction caches (yes, we still have them on ARMv7), we have to flush the I-cache lines associated with the userspace addresses - flushing the I-cache lines for the kernel mapping doesn't do anything for the stale lines in the userspace mapping.
copy_to_page() kmap_atomic()'s the page - this doesn't take account of VIPT aliasing caches, and so the writes may be in an alias of the user page. Plus, when the page is kunmap_atomic()'d, there's no guarantee that the written lines will be flushed out.
copy_from_page() suffers from a similar problem, though its saving grace is that the user mapping is execute-only (iow, read+execute only) so should never end up with any dirty cache lines associated with it.
flush_dcache_page() will ensure that the D-cache lines are flushed out on ARM (even though it's only defined to have an effect for page cache pages) but it does nothing for the I-cache. So, although it seems to partially work, it's definitely the wrong interface here.
However... the normal way we do this (for example, in the case of ptrace) is via copy_to_user_page() / copy_from_user_page(), and we have some pretty complex handling via those functions to deal with writing to the page and having the affected thread running on a different CPU to the one we wrote the new instructions. I'd recommend that uprobes looks at trying to re-use some of this infrastructure rather than re-inventing this wheel.
Conincentally I ran into this same problem last weekend while testing my future thumb extensions to the uprobes code. I too tried a couple cache-flushing changes, and found most any change fixed the problem and the past intermittent systemtap failures I had seen (although I was unsure how much of that was due to happy side-effects from any particular flush operation). I found flush_icache_user_range() but I was scratching my head on how to use it instead of a heavy-weight flush as the uprobes code deliberately does not hang onto a copy of the user vma pointer. I tried a patch storing the mm struct pointer in the uprobes xol struct and looking up the vma info using it and the user virtual address, but I'm not sure that's any better than just hanging onto the vma pointer.
I also would not necessarily have assumed the kernel dcache flush was still needed, but I now see how this makes sense. Thanks for the explanation Russell, it answered several questions I had.
Unfortunately copy_to_user_page() also needs a pointer to a vma struct so, while it presumably provides the model to follow, it can't simply be dropped in.
Victor, do you want to own the action of coming up with a patch? If not I'm OK with taking this forward, although I suspect both of us will need help testing for anything other than ARM.
-dl
On Tue, Apr 08, 2014 at 09:05:49AM -0400, David Long wrote:
Unfortunately copy_to_user_page() also needs a pointer to a vma struct so, while it presumably provides the model to follow, it can't simply be dropped in.
Well, isn't this code doing the same thing as ptrace? It seems to want to modify a page in userspace of another process to change instructions that are going to be executed. That's what ptrace does, and ptrace already copes with all the issues there.
Given that we've already solved that problem, wouldn't it be a good idea if the tracing code would stop trying to reinvent broken solutions to problems we have already solved?
On 8 April 2014 06:30, Russell King - ARM Linux linux@arm.linux.org.uk wrote:
On Tue, Apr 08, 2014 at 09:05:49AM -0400, David Long wrote:
Unfortunately copy_to_user_page() also needs a pointer to a vma struct so, while it presumably provides the model to follow, it can't simply be dropped in.
Well, isn't this code doing the same thing as ptrace? It seems to want to modify a page in userspace of another process to change instructions that are going to be executed. That's what ptrace does, and ptrace already copes with all the issues there.
As I see it, the difference between ptrace use and xol single stepping it that in first all cores should be involved and potentially cache operation would be broadcasted. In case of uprobes only local core is involved. The way I read uprobes code xol slot will be used once by current core to execute one instruction, then it will hit next breakpoint and xol slot will be freed and potentially reused by next uprobe.
I assume that while doing xol single stepping task cannot migrate to another core, wondering whether it is absolutely true. It seems that similar ppc code that calls flush_dcache_page follow this assumption. If uprobes xol single stepping need to handle all cores it will be very expensive.
In ptrace case caches of all cores must be handled.
Having said above I do agree that large portion of flush_ptrace_access could be reused between these two use cases.
Thanks, Victor
Given that we've already solved that problem, wouldn't it be a good idea if the tracing code would stop trying to reinvent broken solutions to problems we have already solved?
-- FTTC broadband for 0.8mile line: now at 9.7Mbps down 460kbps up... slowly improving, and getting towards what was expected from it.
On 8 April 2014 07:09, Victor Kamensky victor.kamensky@linaro.org wrote:
On 8 April 2014 06:30, Russell King - ARM Linux linux@arm.linux.org.uk wrote:
On Tue, Apr 08, 2014 at 09:05:49AM -0400, David Long wrote:
Unfortunately copy_to_user_page() also needs a pointer to a vma struct so, while it presumably provides the model to follow, it can't simply be dropped in.
Well, isn't this code doing the same thing as ptrace? It seems to want to modify a page in userspace of another process to change instructions that are going to be executed. That's what ptrace does, and ptrace already copes with all the issues there.
As I see it, the difference between ptrace use and xol single stepping it that in first all cores should be involved and potentially cache operation would be broadcasted. In case of uprobes only local core is involved. The way I read uprobes code xol slot will be used once by current core to execute one instruction, then it will hit next breakpoint and xol slot will be freed and potentially reused by next uprobe.
I assume that while doing xol single stepping task cannot migrate to another core, wondering whether it is absolutely true. It seems that similar ppc code that calls flush_dcache_page follow this assumption. If uprobes xol single stepping need to handle all cores it will be very expensive.
In ptrace case caches of all cores must be handled.
Having said above I do agree that large portion of flush_ptrace_access could be reused between these two use cases.
Looking at flush_ptrace_access more closely. Now I am not sure that ptrace write code could easily reused.
1) flush_ptrace_access seems to handle both data and text segments write. In case of xol write we always know that it is code write
2) as I pointed before flush_ptrace_access handles smp case whereas xol write does not need to do that
3) flush_ptrace_access needs vma, which uprobe code does not have on layer where xol is installed. That is probably most critical issue that could stop suggested code sharing. And note that vma in flush_ptrace_access is needed only for cases cases 1) and 2) above, about which xol write does not really care. If we take only required pieces from flush_ptrace_access would not xol cache flush look like this:
void arch_uprobe_xol_sync_dcache_icache(struct page *page, unsigned long addr, unsigned long size) { if (icache_is_vipt_aliasing()) flush_icache_alias(page_to_pfn(page), addr, size); else __cpuc_coherent_user_range(addr, size); }
of course, above function should be properly places or flush_icache_alias function should be made globally visible.
Note on the side: flush_ptrace_access works with user-land memory but it still uses __cpuc_coherent_kern_range. I think it should use __cpuc_coherent_user_range. It looks like implementation wise it is the same but user variant seems more appropriate in given situation. Or am I missing something here?
Thanks, Victor
Thanks, Victor
Given that we've already solved that problem, wouldn't it be a good idea if the tracing code would stop trying to reinvent broken solutions to problems we have already solved?
-- FTTC broadband for 0.8mile line: now at 9.7Mbps down 460kbps up... slowly improving, and getting towards what was expected from it.
On Tue, Apr 08, 2014 at 08:35:01AM -0700, Victor Kamensky wrote:
Looking at flush_ptrace_access more closely. Now I am not sure that ptrace write code could easily reused.
- flush_ptrace_access seems to handle both data and text segments
write. In case of xol write we always know that it is code write
Of course it has to, but writing code is the harder of the two problems. With writes to data segments, the only thing that has to be dealt with is the data cache. With code, not only do you need to deal with the data cache, but you also need to deal with the instruction cache too.
- as I pointed before flush_ptrace_access handles smp case whereas
xol write does not need to do that
Are you sure about that?
If I'm reading the code correctly, uprobes inserts a trapping instruction into the userspace program. When that instruction is hit, it checks whether the thread is the desired one, and may request a slot in this magic page, which is when the write happens.
The uprobes special page is shared across all threads which share the mm_struct, so in the case of a multi-threaded program running on a SMP machine, this page is visible to multiple CPUs.
Is it possible for uprobes to be active on more than one thread at a time? If so, because that page is shared, you could end up writing to a partial cache line from two threads. From what I can see, ixol[] is two words, and there's normally 8 works per cache line on ARM, or occasionally 16.
So, the question now is: is it possible to have uprobes active on more than one thread, and for two threads to hit the uprobes processing, both needing a slot in the page, hitting the same cache line?
Now, what happens if thread 1 on CPU1 gets there first with its write. Then thread 2 on CPU2 gets there, causing the cache line to migrate to CPU2. Then CPU1 does it's (non-broadcasted) flush, meanwhile CPU2 then gets preempted and goes off and does something else.
Please tell me that can't happen. :)
- flush_ptrace_access needs vma, which uprobe code does not have on
layer where xol is installed. That is probably most critical issue that could stop suggested code sharing. And note that vma in flush_ptrace_access is needed only for cases cases 1) and 2) above, about which xol write does not really care. If we take only required pieces from flush_ptrace_access would not xol cache flush look like this:
void arch_uprobe_xol_sync_dcache_icache(struct page *page, unsigned long addr, unsigned long size) { if (icache_is_vipt_aliasing()) flush_icache_alias(page_to_pfn(page), addr, size); else __cpuc_coherent_user_range(addr, size); }
And this is only half the story.
What you're saying above is:
- if the *instruction* cache is an aliasing VIPT cache, then we can map the user page at the same colour and flush the data and instruction caches according to that colour. Problem: if you have a VIPT aliasing data cache, and you wrote to the page via a mapping which had a different colour (which is highly likely given you're using kmap_atomic()) then you are not flushing the new instructions out of the data cache. - if the *instruction* cache is not an aliasing VIPT cache, then we flush using the userspace mapping directly. Problem: what if the data cache is aliasing... same problem as the above case.
This is why we have the check for cache_is_vipt_aliasing() in there - that's not _just_ to deal with ptrace data modification, it's there for text modification as well.
Note on the side: flush_ptrace_access works with user-land memory but it still uses __cpuc_coherent_kern_range. I think it should use __cpuc_coherent_user_range.
No, because __cpuc_coherent_kern_range() there is used on the *kernel* mapping of the page, not on the *user* mapping (which may fault). The kernel mapping will never fault.
It looks like implementation wise it is the same but user variant seems more appropriate in given situation. Or am I missing something here?
They do seem to be the same for both cases, so we could probably remove the distinction. It's unlikely we'll ever see an implementation needing a difference between the two now.
On 04/08/14 12:19, Russell King - ARM Linux wrote:
On Tue, Apr 08, 2014 at 08:35:01AM -0700, Victor Kamensky wrote:
Looking at flush_ptrace_access more closely. Now I am not sure that ptrace write code could easily reused.
- flush_ptrace_access seems to handle both data and text segments
write. In case of xol write we always know that it is code write
Of course it has to, but writing code is the harder of the two problems. With writes to data segments, the only thing that has to be dealt with is the data cache. With code, not only do you need to deal with the data cache, but you also need to deal with the instruction cache too.
- as I pointed before flush_ptrace_access handles smp case whereas
xol write does not need to do that
Are you sure about that?
If I'm reading the code correctly, uprobes inserts a trapping instruction into the userspace program. When that instruction is hit, it checks whether the thread is the desired one, and may request a slot in this magic page, which is when the write happens.
The uprobes special page is shared across all threads which share the mm_struct, so in the case of a multi-threaded program running on a SMP machine, this page is visible to multiple CPUs.
Is it possible for uprobes to be active on more than one thread at a time? If so, because that page is shared, you could end up writing to a partial cache line from two threads. From what I can see, ixol[] is two words, and there's normally 8 works per cache line on ARM, or occasionally 16.
So, the question now is: is it possible to have uprobes active on more than one thread, and for two threads to hit the uprobes processing, both needing a slot in the page, hitting the same cache line?
Now, what happens if thread 1 on CPU1 gets there first with its write. Then thread 2 on CPU2 gets there, causing the cache line to migrate to CPU2. Then CPU1 does it's (non-broadcasted) flush, meanwhile CPU2 then gets preempted and goes off and does something else.
Please tell me that can't happen. :)
From arch/arm/include/asm/uprobes.h:
3) flush_ptra#define UPROBE_XOL_SLOT_BYTES 64
-dl
On 8 April 2014 09:19, Russell King - ARM Linux linux@arm.linux.org.uk wrote:
On Tue, Apr 08, 2014 at 08:35:01AM -0700, Victor Kamensky wrote:
Looking at flush_ptrace_access more closely. Now I am not sure that ptrace write code could easily reused.
- flush_ptrace_access seems to handle both data and text segments
write. In case of xol write we always know that it is code write
Of course it has to, but writing code is the harder of the two problems. With writes to data segments, the only thing that has to be dealt with is the data cache. With code, not only do you need to deal with the data cache, but you also need to deal with the instruction cache too.
- as I pointed before flush_ptrace_access handles smp case whereas
xol write does not need to do that
Are you sure about that?
If I'm reading the code correctly, uprobes inserts a trapping instruction into the userspace program. When that instruction is hit, it checks whether the thread is the desired one, and may request a slot in this magic page, which is when the write happens.
The uprobes special page is shared across all threads which share the mm_struct, so in the case of a multi-threaded program running on a SMP machine, this page is visible to multiple CPUs.
Is it possible for uprobes to be active on more than one thread at a time? If so, because that page is shared, you could end up writing to a partial cache line from two threads. From what I can see, ixol[] is two words, and there's normally 8 works per cache line on ARM, or occasionally 16.
So, the question now is: is it possible to have uprobes active on more than one thread, and for two threads to hit the uprobes processing, both needing a slot in the page, hitting the same cache line?
Now, what happens if thread 1 on CPU1 gets there first with its write. Then thread 2 on CPU2 gets there, causing the cache line to migrate to CPU2. Then CPU1 does it's (non-broadcasted) flush, meanwhile CPU2 then gets preempted and goes off and does something else.
Please tell me that can't happen. :)
- flush_ptrace_access needs vma, which uprobe code does not have on
layer where xol is installed. That is probably most critical issue that could stop suggested code sharing. And note that vma in flush_ptrace_access is needed only for cases cases 1) and 2) above, about which xol write does not really care. If we take only required pieces from flush_ptrace_access would not xol cache flush look like this:
void arch_uprobe_xol_sync_dcache_icache(struct page *page, unsigned long addr, unsigned long size) { if (icache_is_vipt_aliasing()) flush_icache_alias(page_to_pfn(page), addr, size); else __cpuc_coherent_user_range(addr, size); }
And this is only half the story.
What you're saying above is:
- if the *instruction* cache is an aliasing VIPT cache, then we can map the user page at the same colour and flush the data and instruction caches according to that colour. Problem: if you have a VIPT aliasing data cache, and you wrote to the page via a mapping which had a different colour (which is highly likely given you're using kmap_atomic()) then you are not flushing the new instructions out of the data cache.
- if the *instruction* cache is not an aliasing VIPT cache, then we flush using the userspace mapping directly. Problem: what if the data cache is aliasing... same problem as the above case.
This is why we have the check for cache_is_vipt_aliasing() in there - that's not _just_ to deal with ptrace data modification, it's there for text modification as well.
Thanks! I got it now. Did not fully understand flush_ptrace_access code. And did not catch (misread) difference between cache_is_vipt_aliasing and icache_is_vipt_aliasing().
Probably some technique that retrieves required information from vma like 1) is_exec = vma->vm_flags & VM_EXEC 2) core_in_mm = cpumask_test_cpu(smp_processor_id(), mm_cpumask(vma->vm_mm)) 3) broadcast = cache_ops_need_broadcast() and passes it to common function would allow to share common code and relieve need to search for vma after xol write call. After xol write will call common function will be called with is_exec = 1, core_in_mm = 1, broadcast = 0.
Although that common function will have seven parameters and will look a bit ugly and unnatural. It could be compensated by making it inline so flush_ptrace_access and flush_uprobe_xol_access function would effectively have its own copy and correctly optimized.
Other option is to have flush_ptrace_access that takes vma and flush_uprobe_xol_access function side by side in the same file with comment that begs to keep common logic between two functions. But not sure how that would work in long term.
Comments? Preferences? Any other suggestions how to deal with lack of vma in uprobe xol write case?
Thanks, Victor
Note on the side: flush_ptrace_access works with user-land memory but it still uses __cpuc_coherent_kern_range. I think it should use __cpuc_coherent_user_range.
No, because __cpuc_coherent_kern_range() there is used on the *kernel* mapping of the page, not on the *user* mapping (which may fault). The kernel mapping will never fault.
It looks like implementation wise it is the same but user variant seems more appropriate in given situation. Or am I missing something here?
They do seem to be the same for both cases, so we could probably remove the distinction. It's unlikely we'll ever see an implementation needing a difference between the two now.
-- FTTC broadband for 0.8mile line: now at 9.7Mbps down 460kbps up... slowly improving, and getting towards what was expected from it.
Sorry, I din't read the whole thread yet...
On 04/08, Russell King - ARM Linux wrote:
On Tue, Apr 08, 2014 at 09:05:49AM -0400, David Long wrote:
Unfortunately copy_to_user_page() also needs a pointer to a vma struct so, while it presumably provides the model to follow, it can't simply be dropped in.
Well, isn't this code doing the same thing as ptrace? It seems to want to modify a page in userspace of another process to change instructions that are going to be executed. That's what ptrace does, and ptrace already copes with all the issues there.
Yes, but it does get_user_pages(&vma) and thus it knows vma.
Given that we've already solved that problem, wouldn't it be a good idea if the tracing code would stop trying to reinvent broken solutions to problems we have already solved?
But uprobes can't do this. Of course, I am not saying this is impossible, but it would be nice to avoid mmap_sem/find_vma/etc.
Almost nobody (iirc only sparc?) actually uses this "vma" arguments. And at least the supported architectures do not (at least this is what I think after the quick grep).
Perhaps we can rolerate the hack below?
Oleg.
--- x/kernel/events/uprobes.c +++ x/kernel/events/uprobes.c @@ -1295,10 +1295,10 @@ static unsigned long xol_get_insn_slot(s copy_to_page(area->page, xol_vaddr, &uprobe->arch.ixol, sizeof(uprobe->arch.ixol)); /* - * We probably need flush_icache_user_range() but it needs vma. - * This should work on supported architectures too. + * The architectures we currently support do not really use vma. */ - flush_dcache_page(area->page); + flush_icache_user_range(NULL /* vma */, area->page, + xol_vaddr, sizeof(uprobe->arch.ixol));
return xol_vaddr; }
On 04/08, Oleg Nesterov wrote:
Almost nobody (iirc only sparc?)
arch/alpha, I meant.
actually uses this "vma" arguments. And at least the supported architectures do not (at least this is what I think after the quick grep).
Perhaps we can rolerate the hack below?
Oleg.
--- x/kernel/events/uprobes.c +++ x/kernel/events/uprobes.c @@ -1295,10 +1295,10 @@ static unsigned long xol_get_insn_slot(s copy_to_page(area->page, xol_vaddr, &uprobe->arch.ixol, sizeof(uprobe->arch.ixol)); /*
* We probably need flush_icache_user_range() but it needs vma.
* This should work on supported architectures too.
*/* The architectures we currently support do not really use vma.
- flush_dcache_page(area->page);
flush_icache_user_range(NULL /* vma */, area->page,
xol_vaddr, sizeof(uprobe->arch.ixol));
return xol_vaddr;
}
And perhaps we can change arch/alpha to accept vma == NULL?
Oleg.
--- x/arch/alpha/kernel/smp.c +++ x/arch/alpha/kernel/smp.c @@ -757,8 +757,13 @@ flush_icache_user_range(struct vm_area_s { struct mm_struct *mm = vma->vm_mm;
- if ((vma->vm_flags & VM_EXEC) == 0) - return; + if (vma) { + if ((vma->vm_flags & VM_EXEC) == 0) + return; + mm = vma->vm_mm; + } else { + mm = current->mm; + }
preempt_disable();
On Tue, Apr 08, 2014 at 05:27:35PM +0200, Oleg Nesterov wrote:
Sorry, I din't read the whole thread yet...
On 04/08, Russell King - ARM Linux wrote:
On Tue, Apr 08, 2014 at 09:05:49AM -0400, David Long wrote:
Unfortunately copy_to_user_page() also needs a pointer to a vma struct so, while it presumably provides the model to follow, it can't simply be dropped in.
Well, isn't this code doing the same thing as ptrace? It seems to want to modify a page in userspace of another process to change instructions that are going to be executed. That's what ptrace does, and ptrace already copes with all the issues there.
Yes, but it does get_user_pages(&vma) and thus it knows vma.
Given that we've already solved that problem, wouldn't it be a good idea if the tracing code would stop trying to reinvent broken solutions to problems we have already solved?
But uprobes can't do this. Of course, I am not saying this is impossible, but it would be nice to avoid mmap_sem/find_vma/etc.
Almost nobody (iirc only sparc?) actually uses this "vma" arguments. And at least the supported architectures do not (at least this is what I think after the quick grep).
Perhaps we can rolerate the hack below?
This has no effect at fixing the reported problem though:
#define flush_icache_user_range(vma,page,addr,len) \ flush_dcache_page(page)
so it results in no change.
The bigger question is... what is this function supposed to do? It's not been documented in Documentation/cachetlb.txt, and nothing in the kernel refers to this function - it is completely unused.
I think let's start out by killing this function - the semantics of this function have been lost, so it's not clear what it was supposed to do in its original form. Even going back to the start of git history, it looks like it was never used outside arch code.
On Tue, Apr 08, 2014 at 04:41:17PM +0100, Russell King - ARM Linux wrote:
I think let's start out by killing this function - the semantics of this function have been lost, so it's not clear what it was supposed to do in its original form. Even going back to the start of git history, it looks like it was never used outside arch code.
I floated a patch to remove flush_icache_user_range() to the architecture maintainers, giving the background to how this came about. I received the following reply from David Miller:
ptrace() accesses (via __access_remote_vm()) already use an existing helper function for these sorts of situations, in the form of copy_{to,from}_user_page(). I would suggest that uprobes uses that as well.
I think this is a very valid point, and echo's my point.
On Wed, Apr 09, 2014 at 05:18:26PM +0100, Russell King - ARM Linux wrote:
On Tue, Apr 08, 2014 at 04:41:17PM +0100, Russell King - ARM Linux wrote:
I think let's start out by killing this function - the semantics of this function have been lost, so it's not clear what it was supposed to do in its original form. Even going back to the start of git history, it looks like it was never used outside arch code.
I floated a patch to remove flush_icache_user_range() to the architecture maintainers, giving the background to how this came about. I received the following reply from David Miller:
ptrace() accesses (via __access_remote_vm()) already use an existing helper function for these sorts of situations, in the form of copy_{to,from}_user_page(). I would suggest that uprobes uses that as well.
I think this is a very valid point, and echo's my point.
More comments from David about using copy_{to,from}_user_page():
They should use it at first, then if there is a problem we can identify exactly what can be optimized or needs to be, and expand the interface as needed.
Let's not optimize first.
Please look into using this solution, thanks.
On 04/09, Russell King - ARM Linux wrote:
I floated a patch to remove flush_icache_user_range() to the architecture maintainers, giving the background to how this came about. I received the following reply from David Miller:
ptrace() accesses (via __access_remote_vm()) already use an existing helper function for these sorts of situations, in the form of copy_{to,from}_user_page(). I would suggest that uprobes uses that as well.
I think this is a very valid point, and echo's my point.
Well. So far I disagree, but let me reply tomorrow.
Sorry, right now I can't even read this thread.
Oleg.
Hi David,
On 8 April 2014 06:05, David Long dave.long@linaro.org wrote:
Victor, do you want to own the action of coming up with a patch? If not I'm OK with taking this forward, although I suspect both of us will need help testing for anything other than ARM.
Please take it forward. I can do some testing of patches you will come up with.
Thanks, Victor
linaro-kernel@lists.linaro.org