Arnd Bergmann arnd@arndb.de wrote on 08/26/2011 04:44:26 PM:
On Thursday 25 August 2011, Russell King - ARM Linux wrote:
Arnd, can you test this to make sure your gdb test case still works,
and
Mark, can you test this to make sure it fixes your problem please?
Hi Russell,
The patch in question was not actually from me but from Ulrich Weigand, so he's probably the right person to test your patch. I'm forwarding it in full to Uli for reference.
Hi Arnd, hi Russell,
sorry for the late reply, I've just returned from vacation today ...
I've not yet run the test, but just from reading through the patch it seems that this will at least partially re-introduce the problem my original patch was trying to fix.
The situation here is about GDB performing an "inferior function call", e.g. via the GDB "call" command. To do so, GDB will:
0. [ Have gotten control of the target process via some ptrace intercept previously, and then ... ] 1. Save the register state 2. Create a dummy frame on the stack and set up registers (PC, SP, argument registers, ...) as appropriate for a function call 3. Restart via PTRACE_CONTINUE [ ... at this point, the target process runs the function until it returns to a breakpoint instruction and GDB gets control again via another ptrace intercept ... ] 4. Restore the register state saved in [1.] 5. At some later point, continue the target process [at its original location] with PTRACE_CONTINUE
The problem now occurs if at point [0.] the target process just happened to be blocked in a restartable system call. For this sequence to then work as expected, two things have to happen:
- at point [3.], the kernel must *not* attempt to restart a system call, even though it thinks we're stopped in a restartable system call
- at point [5.], the kernel now *must* restart the originally interrupted system call, even though it thinks we're stopped at some breakpoint, and not within a system call
My patch achieved both these goals, while it would seem your patch only solves the first issue, not the second one. In fact, since any interaction with ptrace will always cause the TIF_SYS_RESTART flag to be *reset*, and there is no way at all to *set* it, there doesn't appear to be any way for GDB to achive that second goal.
[ With my patch, that second goal was implicitly achieved by the fact that at [1.] GDB would save a register state that already corresponds to the way things should be for restarting the system call. When that register set is then restored in [4.], restart just happens automatically without any further kernel intervention. ]
One way to fix this might be to make the TIF_SYS_RESTART flag itself visible to ptrace, so the GDB could save/restore it along with the rest of the register set; this would be similar to how that problem is handled on other platforms. However, there doesn't appear to be an obvious place for the flag in the ptrace register set ...
Bye, Ulrich
On Thu, Sep 01, 2011 at 03:41:22PM +0200, Ulrich Weigand wrote:
The problem now occurs if at point [0.] the target process just happened to be blocked in a restartable system call. For this sequence to then work as expected, two things have to happen:
at point [3.], the kernel must *not* attempt to restart a system call, even though it thinks we're stopped in a restartable system call
at point [5.], the kernel now *must* restart the originally interrupted system call, even though it thinks we're stopped at some breakpoint, and not within a system call
My patch achieved both these goals, while it would seem your patch only solves the first issue, not the second one. In fact, since any interaction with ptrace will always cause the TIF_SYS_RESTART flag to be *reset*, and there is no way at all to *set* it, there doesn't appear to be any way for GDB to achive that second goal.
...
One way to fix this might be to make the TIF_SYS_RESTART flag itself visible to ptrace, so the GDB could save/restore it along with the rest of the register set; this would be similar to how that problem is handled on other platforms. However, there doesn't appear to be an obvious place for the flag in the ptrace register set ...
Thanks for looking at this.
I don't think we can augment the ptrace register set - that would be a major API change which would immediately break lots of userspace, causing user stack overflows and such like.
I can't see a way out of this - and given the seriousness of the kernel side issue (causing kernel warnings), and that your change altered the strace behaviour (an unintended user-visible change) I think we're going to have to live with the gdb testcase failing until we can come up with a better fix for it.
I also wonder what the validity of this behaviour is - there are cases where you can't do what gdb's trying to do - eg, with a syscall using a restart block (-ERESTART_RESTARTBLOCK) because the restart information could be wiped out by a new syscall performed by the function gdb wants to run. Or when the program receives a signal for it to handle while running that function.
Russell King - ARM Linux linux@arm.linux.org.uk wrote on 09/01/2011 04:00:00 PM:
On Thu, Sep 01, 2011 at 03:41:22PM +0200, Ulrich Weigand wrote:
One way to fix this might be to make the TIF_SYS_RESTART flag itself visible to ptrace, so the GDB could save/restore it along with the rest of the register set; this would be similar to how that problem is handled on other platforms. However, there doesn't appear to be an obvious place for the flag in the ptrace register set ...
Thanks for looking at this.
I don't think we can augment the ptrace register set - that would be a major API change which would immediately break lots of userspace, causing user stack overflows and such like.
Well, it wouldn't need to go into the default register set (REGSET_GPR), but could be in a new register set. This shouldn't change anything for existing applications, but GDB could probe for support for the new regset and use it if present.
I can't see a way out of this - and given the seriousness of the kernel side issue (causing kernel warnings), and that your change altered the strace behaviour (an unintended user-visible change) I think we're going to have to live with the gdb testcase failing until we can come up with a better fix for it.
The change you suggest, with the addition of making the flag available to the debugger in a new register set, should be fine for GDB.
I agree that the race you pointed out in the initial mail to this thread needs to be fixed; an in fact, it seems that this race is also present on s390 ... In discussions with Martin, we were wondering however whether the fix you suggest does actually fully plug the race:
Assume the scenario you initally describe, where a first signal is ignored and leads to system call restart. With your latest patch, you call into syscall_restart which sets everything up to restart the call (with interrupts disabled). Then, you go back to user space, with PC pointing to the SVC that is to be restarted. Now assume that some interrupt is pending at this point. At the switch to user space, interrupts will be enabled. Can it now happen that this pending interrupt is recognized *before* the SVC is executed? (Maybe the hardware doesn't allow that on ARM ... it definitely *is* possible on s390, though.)
If so, can this not then lead to the exact same race you initially described (the interrupt causing some other task to be scheduled, and then an second signal to be delivered to the task about to restart the SVC)?
I guess one way to fix this would be to not actually go back to user space for system call restart at all, but instead just short- cut this in entry.S and simply go right back to the system call entry path. As a side benefit, this could eliminate the need to create a stack frame for the OABI RESTARTBLOCK case ... (This is what we do on s390 b.t.w., where there is a similar issue with the system call number being an immediate argument to the SVC instruction.)
I also wonder what the validity of this behaviour is - there are cases where you can't do what gdb's trying to do - eg, with a syscall using a restart block (-ERESTART_RESTARTBLOCK) because the restart information could be wiped out by a new syscall performed by the function gdb wants to run. Or when the program receives a signal for it to handle while running that function.
Yes, performing an inferior call during a system call that restarts using RESTARTBLOCK is broken. This is a problem on all architectures, and has been ever since the RESTARTBLOCK mechanism was introduced ...
To really fix this case would probably require some way for the debugger to save and restore the restore_block saved state. This is not quite trivial, since it would expose that state to user space, effectively creating a new ABI (and probably requiring sanity checks to ensure a valid state is restored). This probably cannot be fixed by one architecture for itself, but would need support from common kernel code.
Bye, Ulrich
On Fri, Sep 02, 2011 at 04:47:35PM +0200, Ulrich Weigand wrote:
Assume the scenario you initally describe, where a first signal is ignored and leads to system call restart. With your latest patch, you call into syscall_restart which sets everything up to restart the call (with interrupts disabled).
I don't think SIG_IGN signals even set the TIF work flag, so they never even cause a call into do_signal(). Therefore, as far as syscalls go, attempting to send a process (eg) a SIGINT which its handler is set to SIG_IGN results in the process not even being notified about the attempt - we won't even wake up while the syscall is sleeping.
To really fix this case would probably require some way for the debugger to save and restore the restore_block saved state. This is not quite trivial, since it would expose that state to user space, effectively creating a new ABI (and probably requiring sanity checks to ensure a valid state is restored). This probably cannot be fixed by one architecture for itself, but would need support from common kernel code.
Such state would have to be crytographically signed or kept entirely within the kernel, as it would otherwise mean that you could redirect the kernel PC to anywhere...
Russell King - ARM Linux linux@arm.linux.org.uk wrote on 09/02/2011 07:22:59 PM:
On Fri, Sep 02, 2011 at 04:47:35PM +0200, Ulrich Weigand wrote:
Assume the scenario you initally describe, where a first signal is ignored and leads to system call restart. With your latest patch, you call into syscall_restart which sets everything up to restart the call (with interrupts disabled).
I don't think SIG_IGN signals even set the TIF work flag, so they never even cause a call into do_signal(). Therefore, as far as syscalls go, attempting to send a process (eg) a SIGINT which its handler is set to SIG_IGN results in the process not even being notified about the attempt - we won't even wake up while the syscall is sleeping.
I don't see why SIG_IGN signals shouldn't set the TIF work flag; the decision whether to ignore a signal is only made once we've got to get_signal_to_deliver. In any case, whether or not the signal is SIG_IGN doesn't matter for the example at all; I'm simply talking about the case whether the first signal we get leads to system call restart, exactly the same as in the original example you initially described here: http://lists.arm.linux.org.uk/lurker/message/20110823.154329.a3e65f95.en.htm...
To really fix this case would probably require some way for the debugger to save and restore the restore_block saved state. This is not quite trivial, since it would expose that state to user space, effectively creating a new ABI (and probably requiring sanity checks to ensure a valid state is restored). This probably cannot be fixed by one architecture for itself, but would need support from common kernel code.
Such state would have to be crytographically signed or kept entirely within the kernel, as it would otherwise mean that you could redirect the kernel PC to anywhere...
Agreed, that's why the state would need to be verified (in the case of the function pointer, we probably would not want to export the kernel code address to user space in any case, but identify which of the possible target functions is to be called in some other manner).
Bye, Ulrich
On Fri, Sep 02, 2011 at 07:40:34PM +0200, Ulrich Weigand wrote:
Russell King - ARM Linux linux@arm.linux.org.uk wrote on 09/02/2011 07:22:59 PM:
On Fri, Sep 02, 2011 at 04:47:35PM +0200, Ulrich Weigand wrote:
Assume the scenario you initally describe, where a first signal is ignored and leads to system call restart. With your latest patch, you call into syscall_restart which sets everything up to restart the call (with interrupts disabled).
I don't think SIG_IGN signals even set the TIF work flag, so they never even cause a call into do_signal(). Therefore, as far as syscalls go, attempting to send a process (eg) a SIGINT which its handler is set to SIG_IGN results in the process not even being notified about the attempt - we won't even wake up while the syscall is sleeping.
I don't see why SIG_IGN signals shouldn't set the TIF work flag; the decision whether to ignore a signal is only made once we've got to get_signal_to_deliver.
Yes, having looked deeper, you seem to be right - but only if the thread is being ptraced. If it's not being ptraced, ignored signals don't make it that far.
And yes, we can end up processing the interrupt before the SVC is executed, which is still a hole. So we need to avoid doing the restart in userspace - which might actually make things easier. I'll take a look into that over the weekend.
Hi Russell,
Russell King - ARM Linux linux@arm.linux.org.uk wrote on 09/02/2011 07:48:12 PM:
On Fri, Sep 02, 2011 at 07:40:34PM +0200, Ulrich Weigand wrote:
Russell King - ARM Linux linux@arm.linux.org.uk wrote on 09/02/2011 07:22:59 PM:
On Fri, Sep 02, 2011 at 04:47:35PM +0200, Ulrich Weigand wrote:
Assume the scenario you initally describe, where a first signal is ignored and leads to system call restart. With your latest patch, you call into syscall_restart which sets everything up to restart the call (with interrupts disabled).
I don't think SIG_IGN signals even set the TIF work flag, so they never even cause a call into do_signal(). Therefore, as far as syscalls go, attempting to send a process (eg) a SIGINT which its handler is set to SIG_IGN results in the process not even being notified about the attempt - we won't even wake up while the syscall is sleeping.
I don't see why SIG_IGN signals shouldn't set the TIF work flag; the decision whether to ignore a signal is only made once we've got to get_signal_to_deliver.
Yes, having looked deeper, you seem to be right - but only if the thread is being ptraced. If it's not being ptraced, ignored signals don't make it that far.
And yes, we can end up processing the interrupt before the SVC is executed, which is still a hole. So we need to avoid doing the restart in userspace - which might actually make things easier. I'll take a look into that over the weekend.
After some more discussions with Uli I've now created a patch that hopefully will address the issue on s390. The new code always uses the TIF_RESTART_SVC work flag to restart the system call if there is no signal to deliver. Only if there is a signal to deliver the traditional rewind of the instruction pointer is used to restart the system call (for -ERESTARTNOINTR and -ERESTARTSYS with a SA_RESTART signal handler).
In case you are interested in the code is available at git://git390.marist.edu/pub/scm/linux-2.6.git features in particular git commit 4a6a001c39ac196530d8378408b61d6d2fee70d9.
blue skies, Martin
Russell King - ARM Linux linux@arm.linux.org.uk wrote:
And yes, we can end up processing the interrupt before the SVC is executed, which is still a hole. So we need to avoid doing the restart in userspace - which might actually make things easier. I'll take a look into that over the weekend.
Hi Russell, I was wondering whether is there any news on this? If you do rewrite the signal handler processing, please let me know so I can update GDB accordingly ... Thanks!
Mit freundlichen Gruessen / Best Regards
Ulrich Weigand
-- Dr. Ulrich Weigand | Phone: +49-7031/16-3727 STSM, GNU compiler and toolchain for Linux on System z and Cell/B.E. IBM Deutschland Research & Development GmbH Vorsitzender des Aufsichtsrats: Martin Jetter | Geschäftsführung: Dirk Wittkopp Sitz der Gesellschaft: Böblingen | Registergericht: Amtsgericht Stuttgart, HRB 243294
linaro-toolchain@lists.linaro.org
-
Martin Schwidefsky -
Russell King - ARM Linux -
Ulrich Weigand