Sometime after 5.10.105 (5.10.132 and 6.0) there is a change that makes setns(open("/proc/1/ns/net")) in the main process change the behaviour of other process threads.
I don't know how much is broken, but the following fails.
Create a network namespace (eg "test"). Create a 'bond' interface (eg "test0") in the namespace.
Then /proc/net/bonding/test0 only exists inside the namespace.
However if you run a program in the "test" namespace that does: - create a thread. - change the main thread to in "init" namespace. - try to open /proc/net/bonding/test0 in the thread. then the open fails.
I don't know how much else is affected and haven't tried to bisect (I can't create bonds on my normal test kernel).
The test program below shows the problem. Compile and run as: # ip netns exec test strace -f test_prog /proc/net/bonding/test0
The second open by the child should succeed, but fails.
I can't see any changes to the bonding code, so I suspect it is something much more fundamental. It might only affect /proc/net, but it might also affect which namespace sockets get created in. IIRC ls -l /proc/n/task/*/ns gives the correct namespaces.
David
#define _GNU_SOURCE
#include <fcntl.h> #include <unistd.h> #include <poll.h> #include <pthread.h> #include <sched.h>
#define delay(secs) poll(0,0, (secs) * 1000)
static void *thread_fn(void *file) { delay(2); open(file, O_RDONLY);
delay(5); open(file, O_RDONLY);
return NULL; }
int main(int argc, char **argv) { pthread_t id;
pthread_create(&id, NULL, thread_fn, argv[1]);
delay(1); open(argv[1], O_RDONLY);
delay(2); setns(open("/proc/1/ns/net", O_RDONLY), 0);
delay(1); open(argv[1], O_RDONLY);
delay(4);
return 0; }
- Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK Registration No: 1397386 (Wales)
On Sun, Sep 04, 2022 at 02:05:20PM +0000, David Laight wrote:
Sometime after 5.10.105 (5.10.132 and 6.0) there is a change that makes setns(open("/proc/1/ns/net")) in the main process change the behaviour of other process threads.
Can you use 'git bisect' to find the offending commit?
thanks,
greg k-h
7055197705709c59b8ab77e6a5c7d46d61edd96e Author: Alexey Dobriyan adobriyan@gmail.com Cc: Al Viro viro@zeniv.linux.org.uk Signed-off-by: Andrew Morton akpm@linux-foundation.org c6c75deda813 1fde6f21d90f
-----Original Message----- From: David Laight David.Laight@ACULAB.COM Sent: 04 September 2022 15:05
Sometime after 5.10.105 (5.10.132 and 6.0) there is a change that makes setns(open("/proc/1/ns/net")) in the main process change the behaviour of other process threads.
I don't know how much is broken, but the following fails.
Create a network namespace (eg "test"). Create a 'bond' interface (eg "test0") in the namespace.
Then /proc/net/bonding/test0 only exists inside the namespace.
However if you run a program in the "test" namespace that does:
- create a thread.
- change the main thread to in "init" namespace.
- try to open /proc/net/bonding/test0 in the thread.
then the open fails.
I don't know how much else is affected and haven't tried to bisect (I can't create bonds on my normal test kernel).
I've now bisected it. Prior to change 7055197705709c59b8ab77e6a5c7d46d61edd96e proc: fix dentry/inode overinstantiating under /proc/${pid}/net the setns() had no effect of either thread. Afterwards both threads see the entries in the init namespace.
However I think that in 5.10.105 the setns() did affect the thread it was run in. That might be the behaviour before c6c75deda813. proc: fix lookup in /proc/net subdirectories after setns(2)
There is also the earlier 1fde6f21d90f proc: fix /proc/net/* after setns(2)
From the commit messages it does look as though setns() should change what is seen, but just for the current thread. So it is currently broken - and has been since 5.18.0-rc4 and whichever stable branches the change was backported to.
David
The test program below shows the problem. Compile and run as: # ip netns exec test strace -f test_prog /proc/net/bonding/test0
The second open by the child should succeed, but fails.
I can't see any changes to the bonding code, so I suspect it is something much more fundamental. It might only affect /proc/net, but it might also affect which namespace sockets get created in. IIRC ls -l /proc/n/task/*/ns gives the correct namespaces.
David
#define _GNU_SOURCE
#include <fcntl.h> #include <unistd.h> #include <poll.h> #include <pthread.h> #include <sched.h>
#define delay(secs) poll(0,0, (secs) * 1000)
static void *thread_fn(void *file) { delay(2); open(file, O_RDONLY);
delay(5); open(file, O_RDONLY); return NULL;}
int main(int argc, char **argv) { pthread_t id;
pthread_create(&id, NULL, thread_fn, argv[1]); delay(1); open(argv[1], O_RDONLY); delay(2); setns(open("/proc/1/ns/net", O_RDONLY), 0); delay(1); open(argv[1], O_RDONLY); delay(4); return 0;}
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK Registration No: 1397386 (Wales)
- Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK Registration No: 1397386 (Wales)
On Mon, Sep 05, 2022 at 09:54:34AM +0000, David Laight wrote:
7055197705709c59b8ab77e6a5c7d46d61edd96e Author: Alexey Dobriyan adobriyan@gmail.com Cc: Al Viro viro@zeniv.linux.org.uk Signed-off-by: Andrew Morton akpm@linux-foundation.org c6c75deda813 1fde6f21d90f
-----Original Message----- From: David Laight David.Laight@ACULAB.COM Sent: 04 September 2022 15:05
Sometime after 5.10.105 (5.10.132 and 6.0) there is a change that makes setns(open("/proc/1/ns/net")) in the main process change the behaviour of other process threads.
Not again...
I don't know how much is broken, but the following fails.
Create a network namespace (eg "test"). Create a 'bond' interface (eg "test0") in the namespace.
Then /proc/net/bonding/test0 only exists inside the namespace.
However if you run a program in the "test" namespace that does:
- create a thread.
- change the main thread to in "init" namespace.
- try to open /proc/net/bonding/test0 in the thread.
then the open fails.
I don't know how much else is affected and haven't tried to bisect (I can't create bonds on my normal test kernel).
I've now bisected it. Prior to change 7055197705709c59b8ab77e6a5c7d46d61edd96e proc: fix dentry/inode overinstantiating under /proc/${pid}/net the setns() had no effect of either thread. Afterwards both threads see the entries in the init namespace.
However I think that in 5.10.105 the setns() did affect the thread it was run in. That might be the behaviour before c6c75deda813. proc: fix lookup in /proc/net subdirectories after setns(2)
There is also the earlier 1fde6f21d90f proc: fix /proc/net/* after setns(2)
From the commit messages it does look as though setns() should change what is seen, but just for the current thread. So it is currently broken - and has been since 5.18.0-rc4 and whichever stable branches the change was backported to.
David
The test program below shows the problem. Compile and run as: # ip netns exec test strace -f test_prog /proc/net/bonding/test0
The second open by the child should succeed, but fails.
I can't see any changes to the bonding code, so I suspect it is something much more fundamental. It might only affect /proc/net, but it might also affect which namespace sockets get created in.
How? setns(2) acts on "current", and sockets are created from current->nsproxy->net_ns?
IIRC ls -l /proc/n/task/*/ns gives the correct namespaces.
David
#define _GNU_SOURCE
#include <fcntl.h> #include <unistd.h> #include <poll.h> #include <pthread.h> #include <sched.h>
#define delay(secs) poll(0,0, (secs) * 1000)
static void *thread_fn(void *file) { delay(2); open(file, O_RDONLY);
delay(5); open(file, O_RDONLY); return NULL;}
int main(int argc, char **argv) { pthread_t id;
pthread_create(&id, NULL, thread_fn, argv[1]); delay(1); open(argv[1], O_RDONLY); delay(2); setns(open("/proc/1/ns/net", O_RDONLY), 0); delay(1); open(argv[1], O_RDONLY); delay(4); return 0;}
Can you test before this one? This is where it all started.
commit 1da4d377f943fe4194ffb9fb9c26cc58fad4dd24 Author: Alexey Dobriyan adobriyan@gmail.com Date: Fri Apr 13 15:35:42 2018 -0700
proc: revalidate misc dentries
From: Alexey Dobriyan
Sent: 05 September 2022 18:33
On Mon, Sep 05, 2022 at 09:54:34AM +0000, David Laight wrote:
7055197705709c59b8ab77e6a5c7d46d61edd96e Author: Alexey Dobriyan adobriyan@gmail.com Cc: Al Viro viro@zeniv.linux.org.uk Signed-off-by: Andrew Morton akpm@linux-foundation.org c6c75deda813 1fde6f21d90f
-----Original Message----- From: David Laight David.Laight@ACULAB.COM Sent: 04 September 2022 15:05
Sometime after 5.10.105 (5.10.132 and 6.0) there is a change that makes setns(open("/proc/1/ns/net")) in the main process change the behaviour of other process threads.
Not again...
The bisection gave a whack-a-mole patch :-)
I don't know how much is broken, but the following fails.
Create a network namespace (eg "test"). Create a 'bond' interface (eg "test0") in the namespace.
Then /proc/net/bonding/test0 only exists inside the namespace.
However if you run a program in the "test" namespace that does:
- create a thread.
- change the main thread to in "init" namespace.
- try to open /proc/net/bonding/test0 in the thread.
then the open fails.
I don't know how much else is affected and haven't tried to bisect (I can't create bonds on my normal test kernel).
I've now bisected it. Prior to change 7055197705709c59b8ab77e6a5c7d46d61edd96e proc: fix dentry/inode overinstantiating under /proc/${pid}/net the setns() had no effect of either thread. Afterwards both threads see the entries in the init namespace.
However I think that in 5.10.105 the setns() did affect the thread it was run in. That might be the behaviour before c6c75deda813. proc: fix lookup in /proc/net subdirectories after setns(2)
There is also the earlier 1fde6f21d90f proc: fix /proc/net/* after setns(2)
From the commit messages it does look as though setns() should change what is seen, but just for the current thread. So it is currently broken - and has been since 5.18.0-rc4 and whichever stable branches the change was backported to.
David
The test program below shows the problem. Compile and run as: # ip netns exec test strace -f test_prog /proc/net/bonding/test0
The second open by the child should succeed, but fails.
I can't see any changes to the bonding code, so I suspect it is something much more fundamental. It might only affect /proc/net, but it might also affect which namespace sockets get created in.
How? setns(2) acts on "current", and sockets are created from current->nsproxy->net_ns?
I was worried that things might be really badly broken. The bisection implied /proc/net rather than namespace setup itself.
IIRC ls -l /proc/n/task/*/ns gives the correct namespaces.
David
#define _GNU_SOURCE
#include <fcntl.h> #include <unistd.h> #include <poll.h> #include <pthread.h> #include <sched.h>
#define delay(secs) poll(0,0, (secs) * 1000)
static void *thread_fn(void *file) { delay(2); open(file, O_RDONLY);
delay(5); open(file, O_RDONLY); return NULL;}
int main(int argc, char **argv) { pthread_t id;
pthread_create(&id, NULL, thread_fn, argv[1]); delay(1); open(argv[1], O_RDONLY); delay(2); setns(open("/proc/1/ns/net", O_RDONLY), 0); delay(1); open(argv[1], O_RDONLY); delay(4); return 0;}
Can you test before this one? This is where it all started.
commit 1da4d377f943fe4194ffb9fb9c26cc58fad4dd24
That one really doesn't want to build with the toolchain I'm using. gcc generates a lot of warnings (mostly about function pointer types) and then objtool fails the build.
The changes seem to be about flushing the dnlc. In my case everything is pretty static. The namespace is created once just after boot. The real failing program just does: ip netns exec namespace program 3</sys/class/net process: clone() process: setns() thread: open("/proc/net/bonding/not_in_namespace") and expects the open() to fail. So it actually looks like the wrong namespace is being used.
It is interesting that /proc/net is affected by setns() whereas you have to go through hoops to access /sys/class/net. We pass an open fd to /sys/class/net in the init namespace through to the program so it can use openat(3, ...) to see entries in both namespaces.
The namespace exists to separate network traffic, not isolate applications.
David
- Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK Registration No: 1397386 (Wales)
From: Alexey Dobriyan
Sent: 05 September 2022 18:33
-----Original Message----- From: David Laight David.Laight@ACULAB.COM Sent: 04 September 2022 15:05
Sometime after 5.10.105 (5.10.132 and 6.0) there is a change that makes setns(open("/proc/1/ns/net")) in the main process changes the behaviour of other process threads.
Not again...
I've realised what is going on. It really isn't obvious at all. Quite possibly the last change did fix it - even though it broke our code.
/proc/net is a symlink to /proc/self/net. But that isn't what the code wants to open. What it needs is /proc/self/task/self/net. But there isn't a 'self' in /proc/self/task. Which makes it all a bit tedious (especially without gettid() in glibc). (This is a busybox/buildroot system, maybe I could add it!)
I'd probably have noticed earlier if the /proc/net symlink didn't exist. I guess that is for compatibility with pre-netns kernels.
David
- Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK Registration No: 1397386 (Wales)
TWIMC: this mail is primarily send for documentation purposes and for regzbot, my Linux kernel regression tracking bot. These mails usually contain '#forregzbot' in the subject, to make them easy to spot and filter.
On 06.09.22 12:48, David Laight wrote:
From: Alexey Dobriyan
Sent: 05 September 2022 18:33
-----Original Message----- From: David Laight David.Laight@ACULAB.COM Sent: 04 September 2022 15:05
Sometime after 5.10.105 (5.10.132 and 6.0) there is a change that makes setns(open("/proc/1/ns/net")) in the main process changes the behaviour of other process threads.
Not again...
I've realised what is going on. It really isn't obvious at all. Quite possibly the last change did fix it - even though it broke our code.
In that case this seems to be appropriate, unless I misunderstood things:
#regzbot invalid: apparently not a regression
/proc/net is a symlink to /proc/self/net. But that isn't what the code wants to open. What it needs is /proc/self/task/self/net. But there isn't a 'self' in /proc/self/task. Which makes it all a bit tedious (especially without gettid() in glibc). (This is a busybox/buildroot system, maybe I could add it!)
I'd probably have noticed earlier if the /proc/net symlink didn't exist. I guess that is for compatibility with pre-netns kernels.
David
TWIMC: this mail is primarily send for documentation purposes and for regzbot, my Linux kernel regression tracking bot. These mails usually contain '#forregzbot' in the subject, to make them easy to spot and filter.
[TLDR: I'm adding this regression report to the list of tracked regressions; all text from me you find below is based on a few templates paragraphs you might have encountered already already in similar form.]
Hi, this is your Linux kernel regression tracker. CCing the regression mailing list, as it should be in the loop for all regressions, as explained here: https://www.kernel.org/doc/html/latest/admin-guide/reporting-issues.html
On 04.09.22 16:05, David Laight wrote:
Sometime after 5.10.105 (5.10.132 and 6.0) there is a change that makes setns(open("/proc/1/ns/net")) in the main process change the behaviour of other process threads.
I don't know how much is broken, but the following fails.
Create a network namespace (eg "test"). Create a 'bond' interface (eg "test0") in the namespace.
Then /proc/net/bonding/test0 only exists inside the namespace.
However if you run a program in the "test" namespace that does:
- create a thread.
- change the main thread to in "init" namespace.
- try to open /proc/net/bonding/test0 in the thread.
then the open fails.
I don't know how much else is affected and haven't tried to bisect (I can't create bonds on my normal test kernel).
The test program below shows the problem. Compile and run as: # ip netns exec test strace -f test_prog /proc/net/bonding/test0
The second open by the child should succeed, but fails.
I can't see any changes to the bonding code, so I suspect it is something much more fundamental. It might only affect /proc/net, but it might also affect which namespace sockets get created in. IIRC ls -l /proc/n/task/*/ns gives the correct namespaces.
David
#define _GNU_SOURCE
#include <fcntl.h> #include <unistd.h> #include <poll.h> #include <pthread.h> #include <sched.h>
#define delay(secs) poll(0,0, (secs) * 1000)
static void *thread_fn(void *file) { delay(2); open(file, O_RDONLY);
delay(5); open(file, O_RDONLY); return NULL;}
int main(int argc, char **argv) { pthread_t id;
pthread_create(&id, NULL, thread_fn, argv[1]); delay(1); open(argv[1], O_RDONLY); delay(2); setns(open("/proc/1/ns/net", O_RDONLY), 0); delay(1); open(argv[1], O_RDONLY); delay(4); return 0;}
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK Registration No: 1397386 (Wales)
Thanks for the report. To be sure below issue doesn't fall through the cracks unnoticed, I'm adding it to regzbot, my Linux kernel regression tracking bot:
#regzbot ^introduced v5.10.105..v5.10.132 #regzbot title setns() affecting other threads (also in 6.0-rc) #regzbot ignore-activity
This isn't a regression? This issue or a fix for it are already discussed somewhere else? It was fixed already? You want to clarify when the regression started to happen? Or point out I got the title or something else totally wrong? Then just reply -- ideally with also telling regzbot about it, as explained here: https://linux-regtracking.leemhuis.info/tracked-regression/
Reminder for developers: When fixing the issue, add 'Link:' tags pointing to the report (the mail this one replies to), as explained for in the Linux kernel's documentation; above webpage explains why this is important for tracked regressions.
Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat)
P.S.: As the Linux kernel's regression tracker I deal with a lot of reports and sometimes miss something important when writing mails like this. If that's the case here, don't hesitate to tell me in a public reply, it's in everyone's interest to set the public record straight.
linux-stable-mirror@lists.linaro.org
-
Alexey Dobriyan -
David Laight -
Greg KH -
Thorsten Leemhuis