Dear Linux Community Members,
Over the years, various informal groups have formed within our community, serving purposes such as maintaining connections with companies and external bodies, handling sensitive information, making challenging decisions, and, at times, representing the community as a whole. These groups contribute significantly to our community's development and deserve our recognition and appreciation.
I'll name a few below that I identified from `Documentation/`: - Code of Conduct Committee conduct@kernel.org - Linux kernel security team security@kernel.org - Linux kernel hardware security team hardware-security@kernel.org - Kernel CVE assignment team cve@kernel.org - Stable Team for unpublished vulnerabilities stable@kernel.org (I suspect it's just an alias to regular stable team, but I found no evidence).
Over recent events, I've taken a closer look at how our community's governance operates, only to find that there's remarkably little public information available about those informal groups. With the exception of the Linux kernel hardware security team, it seems none of these groups maintain a public list of members that I can easily find.
Upon digging into the details, I’d like to raise a few concerns and offer some thoughts for further discussion:
- Absence of a Membership Register Our community is built on mutual trust. Without knowing who comprises these groups, it's understandably difficult for people to have full confidence in their work. A publicly available membership list would not only foster trust but also allow us to address our recognition and appreciation.
- Lack of Guidelines for Actions Many of these groups appear to operate without documented guidelines. While I trust each respectful individual's integrity, documented guidelines would enable the wider community to better understand and appreciate the roles and responsibilities involved.
- Insufficient Transparency in Decision-Making I fully respect the need for confidentiality in handling security matters, yet some degree of openness around decision-making processes is essential in my opinion. Releasing communications post-embargo, for instance, could promote understanding and prevent potential abuse of confidential procedures.
- No Conflict of Interest Policy Particularly in the case of the Code of Conduct Committee, there may arise situations where individuals face challenging decisions involving personal connections. A conflict of interest policy would provide valuable guidance in such circumstances.
Thank you for reading. I know none of us enjoy being pulled away by these non-technical concerns, we love coding after all. However, I feel these concerns are vital for the community's continued health. It might be a candidate of Linux TAB discussion.
I'm looking forward to everyone's input.
Thanks - Jiaxun
在2024年10月25日十月 下午4:15，Jiaxun Yang写道：
Dear Linux Community Members,
Over recent events, I've taken a closer look at how our community's governance operates, only to find that there's remarkably little public information available about those informal groups. With the exception of the Linux kernel hardware security team, it seems none of these groups maintain a public list of members that I can easily find.
Just to correct, people notified me that a membership list for Code of Conduct Committee is available at kernel.org [1] instead of in source tree.
[...]
Thanks
[1]: https://kernel.org/code-of-conduct.html
On 25/10/2024 17:15, Jiaxun Yang wrote:
Dear Linux Community Members,
Over the years, various informal groups have formed within our community, serving purposes such as maintaining connections with companies and external bodies, handling sensitive information, making challenging decisions, and, at times, representing the community as a whole. These groups contribute significantly to our community's development and deserve our recognition and appreciation.
I'll name a few below that I identified from `Documentation/`:
- Code of Conduct Committee conduct@kernel.org
- Linux kernel security team security@kernel.org
- Linux kernel hardware security team hardware-security@kernel.org
- Kernel CVE assignment team cve@kernel.org
- Stable Team for unpublished vulnerabilities stable@kernel.org (I suspect it's just an alias to regular stable team, but I found no evidence).
Over recent events, I've taken a closer look at how our community's governance operates, only to find that there's remarkably little public information available
Oh, spread more FUD under the cloak of helping the community. Reminds me something, wait, how was it? zx?
about those informal groups. With the exception of the Linux kernel hardware security team, it seems none of these groups maintain a public list of members that I can easily find.
Upon digging into the details, I’d like to raise a few concerns and offer some thoughts for further discussion:
- Absence of a Membership Register
Our community is built on mutual trust. Without knowing who comprises these groups, it's understandably difficult for people to have full confidence in their work.
No, you might have difficulty, not "all people" which you imply. Please stop creating sentences like you are speaking for others. You do not speak for others.
A publicly available membership list would not only foster trust but also allow us to address our recognition and appreciation.
Nope. For some of the groups it is very intentional to hide the membership. It was explained already why and should be pretty obvious.
- Lack of Guidelines for Actions
Many of these groups appear to operate without documented guidelines. While I trust each respectful individual's integrity, documented guidelines would enable the wider community to better understand and appreciate the roles and responsibilities involved.
Guidelines are well documented, although I understand something might be missing. Feel free to extend the existing documentation, as usual, patches are welcomed.
- Insufficient Transparency in Decision-Making
I fully respect the need for confidentiality in handling security matters, yet some degree of openness around decision-making processes is essential in my opinion. Releasing communications post-embargo, for instance, could promote understanding and prevent potential abuse of confidential procedures.
Again, unspecified FUD.
- No Conflict of Interest Policy
Particularly in the case of the Code of Conduct Committee, there may arise situations where individuals face challenging decisions involving personal connections. A conflict of interest policy would provide valuable guidance in such circumstances.
Feel free to propose patches instead of claiming there is problem for others. If you identify issue, propose a patch.
Several other your replies earlier were in similar tone. I am not going to engage in such discussions and probably neither other people, but some think that silence is approval or agreement. Thus this reply. for me this is just FUD.
Best regards, Krzysztof
在2024年10月26日十月 下午12:05，Krzysztof Kozlowski写道：
Oh, spread more FUD under the cloak of helping the community. Reminds me something, wait, how was it? zx?
I drafted this email with good will.
While I appreciate any constructive comments, this kind of unfair accusation is unacceptable.
I'm not demanding anyone to take action, I'm just trying to be helpful.
about those informal groups. With the exception of the Linux kernel hardware security team, it seems none of these groups maintain a public list of members that I can easily find.
Upon digging into the details, I’d like to raise a few concerns and offer some thoughts for further discussion:
- Absence of a Membership Register
Our community is built on mutual trust. Without knowing who comprises these groups, it's understandably difficult for people to have full confidence in their work.
No, you might have difficulty, not "all people" which you imply. Please stop creating sentences like you are speaking for others. You do not speak for others.
I never said "all" here, and just to quote:
"I am expressing the views of a number of people I talked to but it's not fair of me to name them."
The same applies to this email as well. I actually did a private RFC before sending it. Many people are unable to speak up here due to company affiliation and other concerns.
A publicly available membership list would not only foster trust but also allow us to address our recognition and appreciation.
Nope. For some of the groups it is very intentional to hide the membership. It was explained already why and should be pretty obvious.
I might be dumb in this case, do you mind giving me a pointer to the explanation? I can draft patch to make it clear in documents.
[...]
- No Conflict of Interest Policy
Particularly in the case of the Code of Conduct Committee, there may arise situations where individuals face challenging decisions involving personal connections. A conflict of interest policy would provide valuable guidance in such circumstances.
Feel free to propose patches instead of claiming there is problem for others. If you identify issue, propose a patch.
Thanks, I will. I'm just aiming to gather some feedback before proposing patches. I also welcome patches from those more qualified than myself.
Several other your replies earlier were in similar tone. I am not going to engage in such discussions and probably neither other people, but some think that silence is approval or agreement. Thus this reply. for me this is just FUD.
Again, I must decline to accept this sort of unfair accusation.
It's indeed not the tone I'm usually speaking on the mailing list. It ought to be more straightforward for technical communications. However, in these particularly challenging times, I'm striving to maintain a humble and respectful tone whilst ensuring my views are clearly spoken. I'd be grateful for comments expressing any dissatisfaction with my approach, but I feel that personal attack ultimately do nothing constructive.
Thanks.
Best regards, Krzysztof
On Fri, Oct 25, 2024 at 04:15:42PM +0100, Jiaxun Yang wrote:
Over recent events, I've taken a closer look at how our community's governance operates, only to find that there's remarkably little public information available about those informal groups.
There's quite a bit of information available in the Linux Kernel documentation. For example:
* https://www.kernel.org/doc/html/latest/process/security-bugs.html * https://www.kernel.org/doc/html/latest/process/code-of-conduct.html * https://www.kernel.org/code-of-conduct.html
Ultimately, though, governance model that we've used since the founding of the Benevolent Dictator model. For a description of this, see:
* https://wiki.p2pfoundation.net/Benevolent_Dictator
The reason why this model works for Open Source projects is that ultimately, the license allows the code to be forked, and someone could decide to take the Linux Kernel sources, and declare some new version, say: "Tedix". However, if I was delusional enough to do this, it's very likely no one would pay attention to me, and consider me a random madman (of which there are many on the Internet).
Ultmately, though, the reason why Linus continues to serve as the leader of the Linux is that there is a very large number of people that respect his judgement and technical acumen. And unlike in physical space where a dictator could (hypothetically) order tanks to fire on college-aged students, ultimately no one can force developers or companies to continue use or develop Linux.
Everything else follows from this. So for example, a maintainer or maintainer team can refuse to accept patches from a particular source. If someone disagrees with a decision, whether it is not accepting a patch, or request a patch that it be reverted, they can appeal to Linus. Linus ask the Maintainer for their reasons, or can decide to override the decision by accepting the patch into his tree, or reverting a patch. Ultimately, Linus can decide to relieve a maintainer of their duties by simply refusing to accept pull request from that maintainer, or by remoing the subsytem entirely from his sources.
As another example, the Code of Conduct committee has no inherent power to sanction developers, other than to make recommendations to people who actually do the work --- namely, Linus Torvalds, other maintainers, the people who run the mailing lists, etc. Like with Maintainers, their "power" comes from the respect that individuals on that body have with Linus and the other maintainers.
Yet another body which you didn't mention is the Linux Foundation Technical Advisory board. That body is elected, but the TAB has always made it clear that the primary power comes from the reputation and moral authority of the people who are elected to the TAB. Sure, The TAB chair has an at-large seat on the Linux Foundation board, but any influence that the TAB through the TAB chair might have is more because of their work and the force of their arguments.
More broadly, the model that I like to use is "servant leadership", and that's why I tell people who want to pursue taking up leadership roles in Linux. Most of the senior leadership spend a huge amount of their personal time, and have often made career choices that have prioritized working on Linux and other Open Source projects over monetary renumeration. Speaking for myself, I could have progressed farther in terms of position and salary. I made choices that traded the freedom and ability to work on Linux because that was more important to me, and there is an awful lot of what I do as a leader is to serve those people in the ext4 development community.
This is not true just in Linux; previously, I've served on the Security Area Advisory Group for the IETF, the standards body for the internet, and as working group chair for the ipsec working group when the IPSec protocols were first being standardied. Sure, I was part of the "governance" of the IETF, but one of the things you learn very quickly is that as a volunteer leader, your primary power is to stop things from happening. Hopefully, you're only stopping bad things from happening, and you can try to encourage and cajole volunteers spend time on what's necessary to make forward progress. And of course, you can spend your own personal time smoothing the way to enable the members of the community to make forward progress. And that takes us back to "servant leadership".
Cheers,
- Ted
P.S. Note that when I say "volunteer', I'm using this in a fairly broad/informal fashion. Yes, some members of the community have companies that pay our salaries to work on Linux. But as the ext4 maintainer, I don't have magement authority over the ext4 developer. I can refuse to take a patch; I can spend time creating testing infrastruture to make it easier for ext4 contributors to test their work; I can point out ways that some particular design might be good for ext4, and good for their company's business objectives, to the extent that I know their companies goals or they are willing to share those goals with me. But I can't *force* someone at SuSE or Oracle or IBM or Huawei to work on some particular ext4 feature or bug. Ultimately, either individuals (who might be hobbists) or companies, voluntarily choose to contribute to ext4, or the IPSec standard. And so that's why I call Linux and the IETF have much in common with a pure 100% volunteer organization, such as Doctors without Borders.
在2024年10月26日十月 下午3:56，Theodore Ts'o写道：
On Fri, Oct 25, 2024 at 04:15:42PM +0100, Jiaxun Yang wrote:
Over recent events, I've taken a closer look at how our community's governance operates, only to find that there's remarkably little public information available about those informal groups.
Hi Theodore,
Thanks for detailed comments! This kind of constructive discussions is what I'm always looking for.
There's quite a bit of information available in the Linux Kernel documentation. For example:
Thank you for the pointers. My concerns actually rooted from the first two documents, and I was directed to the third link off-list following my initial post.
In process/security-bugs, the term "security officers" is consistently mentioned, yet it's unclear who they are, what specific privileges they hold compared to regular developers, and how security fixes are expected to reach Linus's tree during an embargo period.
After reviewing the third link, I now have a clearer understanding of the CoC Committee, though it's unfortunate that this webpage is not directly referenced in the kernel documentation.
That being said, I'll try to improve the documentation on these things based on my observations. My background perhaps makes me particularly sensitive to some ambiguous language, especially where "constructive ambiguity" might be involved. Recent events make me started to look into those aspects in border community.
Ultimately, though, governance model that we've used since the founding of the Benevolent Dictator model. For a description of this, see:
The reason why this model works for Open Source projects is that ultimately, the license allows the code to be forked, and someone could decide to take the Linux Kernel sources, and declare some new version, say: "Tedix". However, if I was delusional enough to do this, it's very likely no one would pay attention to me, and consider me a random madman (of which there are many on the Internet).
Ultmately, though, the reason why Linus continues to serve as the leader of the Linux is that there is a very large number of people that respect his judgement and technical acumen. And unlike in physical space where a dictator could (hypothetically) order tanks to fire on college-aged students, ultimately no one can force developers or companies to continue use or develop Linux.
Everything else follows from this. So for example, a maintainer or maintainer team can refuse to accept patches from a particular source. If someone disagrees with a decision, whether it is not accepting a patch, or request a patch that it be reverted, they can appeal to Linus. Linus ask the Maintainer for their reasons, or can decide to override the decision by accepting the patch into his tree, or reverting a patch. Ultimately, Linus can decide to relieve a maintainer of their duties by simply refusing to accept pull request from that maintainer, or by remoing the subsytem entirely from his sources.
That aligns well with my understanding. I've witnessed many "escalations to Linus" and have been involved a few times myself. I'd say we (maybe not all) benefit from having a respected individual to make those difficult final decisions.
I have no intention of criticizing the system here.
As another example, the Code of Conduct committee has no inherent power to sanction developers, other than to make recommendations to people who actually do the work --- namely, Linus Torvalds, other maintainers, the people who run the mailing lists, etc. Like with Maintainers, their "power" comes from the respect that individuals on that body have with Linus and the other maintainers.
That's new to me. The `Enforcement` section in the CoC document initially gave me the impression that all participants were obligated to follow their decisions, but it turns out that's not the case.
I appreciate the clarification.
Yet another body which you didn't mention is the Linux Foundation Technical Advisory board. That body is elected, but the TAB has always made it clear that the primary power comes from the reputation and moral authority of the people who are elected to the TAB. Sure, The TAB chair has an at-large seat on the Linux Foundation board, but any influence that the TAB through the TAB chair might have is more because of their work and the force of their arguments.
I reviewed the TAB information pages before making that post and saw a clear membership list and guidelines for TAB, so I didn't view it as an "informal group". I even suggested that this could be a topic for TAB discussion at the end of my original post.
More broadly, the model that I like to use is "servant leadership", and that's why I tell people who want to pursue taking up leadership roles in Linux. Most of the senior leadership spend a huge amount of their personal time, and have often made career choices that have prioritized working on Linux and other Open Source projects over monetary renumeration. Speaking for myself, I could have progressed farther in terms of position and salary. I made choices that traded the freedom and ability to work on Linux because that was more important to me, and there is an awful lot of what I do as a leader is to serve those people in the ext4 development community.
Thank you for sharing this insight on servant leadership, it genuinely resonates. I deeply appreciate the personal sacrifices and commitment that you, other leaders and everyone involved, have made to prioritise the advancement of Linux and open-source projects over conventional career progression. It's truly inspiring to witness such dedication.
This is not true just in Linux; previously, I've served on the Security Area Advisory Group for the IETF, the standards body for the internet, and as working group chair for the ipsec working group when the IPSec protocols were first being standardied. Sure, I was part of the "governance" of the IETF, but one of the things you learn very quickly is that as a volunteer leader, your primary power is to stop things from happening. Hopefully, you're only stopping bad things from happening, and you can try to encourage and cajole volunteers spend time on what's necessary to make forward progress. And of course, you can spend your own personal time smoothing the way to enable the members of the community to make forward progress. And that takes us back to "servant leadership".
It's fortunate that people like you tirelessly contribute to driving the world forward. I think we can all agree that, despite the disputes all over the world, we're each striving to make the world a better place in our own ways.
I'm sorry this matter has taken up your attention, I'll see what I can do on my end.
Thanks
Cheers,
- TedP.S. Note that when I say "volunteer', I'm using this in a fairly broad/informal fashion. Yes, some members of the community have companies that pay our salaries to work on Linux. But as the ext4 maintainer, I don't have magement authority over the ext4 developer. I can refuse to take a patch; I can spend time creating testing infrastruture to make it easier for ext4 contributors to test their work; I can point out ways that some particular design might be good for ext4, and good for their company's business objectives, to the extent that I know their companies goals or they are willing to share those goals with me. But I can't *force* someone at SuSE or Oracle or IBM or Huawei to work on some particular ext4 feature or bug. Ultimately, either individuals (who might be hobbists) or companies, voluntarily choose to contribute to ext4, or the IPSec standard. And so that's why I call Linux and the IETF have much in common with a pure 100% volunteer organization, such as Doctors without Borders.
On Sat, Oct 26, 2024 at 05:33:16PM +0100, Jiaxun Yang wrote:
There's quite a bit of information available in the Linux Kernel documentation. For example:
Thank you for the pointers. My concerns actually rooted from the first two documents, and I was directed to the third link off-list following my initial post.
In process/security-bugs, the term "security officers" is consistently mentioned, yet it's unclear who they are, what specific privileges they hold compared to regular developers,
The "security officers" is just a small group of trusted maintainers (~20) who devote some of their spare time (and possibly some work time as well) reviewing, triaging, and forwarding some security reports that arrive there. I think the term "security officers" is used essentially to remind that reporters are interacting with real people without the heavy weight of processes or whatever some could possibly imagine.
I'm not aware of a public list of participants, though the list of participants is shared between the members from time to time when one arrives or leaves. I think discretion is key here because I'm not even sure that every participant's employer knows that they're on that list, and it's better this way, as some might try to exert pressure to try to get some early notifications, which is absolutely forbidden.
Participants are quite responsible people. Some have already left the list by lacking time to participate, some temporarily or definitely. New participants are sometimes asked to join because they're involved in many of the reports, and that way they can directly interact with the reporter without anyone having to review and forward them the messages first.
So if you wonder who's there, just ask yourself who can speed up the process by participating when there are frequent reports in their area of expertise, and you'll guess by yourself a few of them :-)
and how security fixes are expected to reach Linus's tree during an embargo period.
There's no hard-rule there. Some fixes are written by some of the team members because the bug is directly in the subsystem they maintain so for them the easiest path is to take the patch and add it to their pending queue. Most of the time the fixes are forwarded to maintainers not part of the list and they deal with them the way they're used to for other bugs reports. Most of the time bug reporters are told that their report is not critical and should be handled the regular way (as it's always better to have public discussions on fixes). It's super rare that fixes are merged directly by Linus himself. It could happen because there's a huge emergency, but history told us that bugs handled in emergency do not always result in the best fixes. Also if one is seeking discretion, the last thing to do is to merge the fix without sharing it on a public list, as that's what attracts suspecting eyes :-)
Also, for the vast majority of bug reports there's no embargo period requested by the reporter, as most of them just want bugs to be fixed. I think it might be less than 1-2% for which an embargo is requested, and that's fine because fixes don't wait. Most of the time once the fix is agreed upon by the different parties and passes the reporter's tests, it gets merged in the maintainer's tree.
I noticed many times that there are some fantasies around the security list because it's not public, so people in quest of amazing stories may imagine lots of stuff happening there. The reality is that it's exactly like any other topic list where bugs are discussed between maintainers and bug reporters, but the discussions are just not public since they would directly put many users around the world in trouble without even having a chance to protect themselves. Another benefit of not being public is that it's easier for reporters to share traces, captures etc. They don't need to waste their time anonymizing them (though most of the time there's absolutely nothing confidential shared anyway, but an IP or MAC address can remain without having to hide them as is often done on public reports).
Really there's nothing special about that list, it simply helps to put bug reporter in relation with the appropriate maintainers and save them from trivial mistakes, because it's always frightening to report a security issue to a project, you always fear you're sending to the wrong people and will cause unexpected trouble. That list is there to address this specific point, and to make sure the report is not forgotten.
I hope this clarifies its role a bit! Willy
On Sat, Oct 26, 2024 at 05:33:16PM +0100, Jiaxun Yang wrote:
That being said, I'll try to improve the documentation on these things based on my observations. My background perhaps makes me particularly sensitive to some ambiguous language, especially where "constructive ambiguity" might be involved. Recent events make me started to look into those aspects in border community.
Well, make no mistake, there *is* a lot of ambiguity, because we don't really have a centralized governance structure other than Linus has the benvelent dictator. The general philosophy is to have just as much structure as necessary, but no more. We do need to have a legal organization to sign contracts with hotels, caterers, etc., for the purposes of organizing conferences. That is one of the roles of the Linux Foundation. But just because the Linux Foundation organizes conferences, and accepts corporate donations, and pays Linus's salary, *doesn't* mean that they get to dictate to Linus what he does, or anything about what code does or doesn't get accepted into the Linux kernel. As Jim Zemlin, the Executive Director of the Linux Foundation has been known to have said, he works for Linus, and not the other way around.
This is not the only way to organize an open source project, of course. For example, the Rust community has a lot more structure process. I will note that this has not reduced the amount of organizational drama and contrversy. In fact, some might argue that their governance structures may have caused some of the more recent drama that lead to some people stepping down from official leadership roles....
Or you can take a look at some of the BSD projects, which early on had a lot of drama and some of the BSD forks based on who was officiallty part of the core team, and who wasn't (or who was thrown off of the core team). It's perhaps because of that drama in the early 90's that some of us who were around during that era rather consciously rejected the formation of anything like the BSD formal core team model, because we saw the dysfunction that could result from it.
There are limits to the informal model, of course. One of the ways that we have tried to make it scale is that there is great value in making sure that the kernel developers have face time with each other. It's one of the reasons why I organized the Linux Kernel Summit, which later morphed into the Maintainer's Summit. It's why there are many people who spend a huge amount of time organizing the Linux Plumbers Conference and other workshops, whether it's the Linux Security Symposium, or the Linux Storage, File Systems, and MM workshop, or Netconf. The ability for us to see each other face to face, and break bread together, makes the human relationships real in a way that avoids e-mail conversations alone can turning into flame wars.
More recently, some subsystem teams have started regular video chats. They aren't a substitute for in-person meetings, but they still are valuable in terms of having that higher bandwidth conversation where the non-verbal cues can humanize the personal connection.
Of course, like all things, there are tradeoff and limitations. Attendance at in-person meetings can be hampered by real-world considerations such as the cost of travel, or the need to get travel visas, or for people for whom English is not their primary language, they might be able to use Google Translate for e-mail, but that doesn't work that well for in-person meetings or video conferences. Some of these can be mitigated; the Linux Foundation has a travel scholarship fund for people who can't get corporate sponsorship for their travel, and many conferences now have a hybrid option for people who can't attend in person for whatever reason. But the language barrier can still be an issue for some. Maybe someday we will have something like Star Trek's universal translator...
The bottom line, though, that any organization is made up of *people*, and so there is no substitute for personal relationships and trust. If you don't have that, I doubt any amount of organizational structure can save you, and in fact, to the extent that some people might try to game the formal rules/structure, it might actually make things worse.
Cheers,
- Ted
Hello from random bystander watching all this,
You probably should have used the term 'in camera' [1] instead of informal groups.
[1]: https://en.wiktionary.org/wiki/in_camera : "1. In secret or in private (in an enclosed room, behind closed doors)."
Bye,
---- revi | 레비 (IPA: lɛbi) - https://revi.xyz - he/him https://revi.xyz/pronoun-is/ - What time is it in my timezone? https://issuetracker.revi.xyz/u/time - In this Korean name https://en.wikipedia.org/wiki/Korean_name, the family name is Hong https://en.wikipedia.org/wiki/Hong_(Korean_surname), which makes my name HONG Yongmin. - I reply when my time permits. Don't feel pressured to reply ASAP; take your time and respond at your schedule.
linux-stable-mirror@lists.linaro.org
-
Jiaxun Yang -
Krzysztof Kozlowski -
Theodore Ts'o -
Willy Tarreau -
Yongmin