Hi, Alexei!
On Wed, 27 May 2020 09:48:04 -0700, Alexei Starovoitov wrote:
On Wed, May 27, 2020 at 12:19 AM Yauheni Kaliuta yauheni.kaliuta@redhat.com wrote:
Hi, Alexei!
> On Tue, 26 May 2020 22:37:39 -0700, Alexei Starovoitov wrote:
On Tue, May 26, 2020 at 10:31 PM Yauheni Kaliuta yauheni.kaliuta@redhat.com wrote:
Hi, Andrii!
>>> On Tue, 26 May 2020 17:19:18 -0700, Andrii Nakryiko wrote:
On Fri, May 22, 2020 at 1:19 AM Yauheni Kaliuta yauheni.kaliuta@redhat.com wrote:
There is difference in depoying static and generated extra resource files between in/out of tree build and flavors:
- in case of unflavored out-of-tree build static files are not
available and must be copied as well as both static and generated files for flavored build.
So split the rules and variables. The name TRUNNER_EXTRA_GEN_FILES is chosen in analogy to TEST_GEN_* variants.
Can we keep them together but be smarter about what needs to be copied based on source/target directories? I would really like to not blow up all these rules.
I can try, ok, I just find it a bit more clear. But it's good to get some input from kselftest about OOT build in general.
I see no value in 'make install' of selftests/bpf and since it's broken just remove that makefile target.
Some CI systems perform testing next stage after building were build tree is not available anymore. So it's in use at the moment.
such CI systems can do 'cp -r' then
It's a discussion for linux-kselftest@ (added).
At the moment `make install` is generic kselftest functionality and since bpf is part of that infra it looks a bit strange to break it intentionally.
On Wed, May 27, 2020 at 10:02 AM Yauheni Kaliuta yauheni.kaliuta@redhat.com wrote:
Hi, Alexei!
On Wed, 27 May 2020 09:48:04 -0700, Alexei Starovoitov wrote:
On Wed, May 27, 2020 at 12:19 AM Yauheni Kaliuta yauheni.kaliuta@redhat.com wrote:
Hi, Alexei!
>> On Tue, 26 May 2020 22:37:39 -0700, Alexei Starovoitov wrote:
On Tue, May 26, 2020 at 10:31 PM Yauheni Kaliuta yauheni.kaliuta@redhat.com wrote:
Hi, Andrii!
>>>> On Tue, 26 May 2020 17:19:18 -0700, Andrii Nakryiko wrote:
On Fri, May 22, 2020 at 1:19 AM Yauheni Kaliuta yauheni.kaliuta@redhat.com wrote: > > There is difference in depoying static and generated extra resource > files between in/out of tree build and flavors: > > - in case of unflavored out-of-tree build static files are not > available and must be copied as well as both static and generated > files for flavored build. > > So split the rules and variables. The name TRUNNER_EXTRA_GEN_FILES > is chosen in analogy to TEST_GEN_* variants. >
Can we keep them together but be smarter about what needs to be copied based on source/target directories? I would really like to not blow up all these rules.
I can try, ok, I just find it a bit more clear. But it's good to get some input from kselftest about OOT build in general.
I see no value in 'make install' of selftests/bpf and since it's broken just remove that makefile target.
Some CI systems perform testing next stage after building were build tree is not available anymore. So it's in use at the moment.
such CI systems can do 'cp -r' then
It's a discussion for linux-kselftest@ (added).
At the moment `make install` is generic kselftest functionality and since bpf is part of that infra it looks a bit strange to break it intentionally.
selftests/bpf is only historically part of selftests. It probably should stop using kselftest build infra all together. We had breakages in selftests/bpf in the past only because of changes in kselftests bits.
On 5/27/20 11:05 AM, Alexei Starovoitov wrote:
On Wed, May 27, 2020 at 10:02 AM Yauheni Kaliuta yauheni.kaliuta@redhat.com wrote:
Hi, Alexei!
> On Wed, 27 May 2020 09:48:04 -0700, Alexei Starovoitov wrote:
On Wed, May 27, 2020 at 12:19 AM Yauheni Kaliuta yauheni.kaliuta@redhat.com wrote:
Hi, Alexei!
>>> On Tue, 26 May 2020 22:37:39 -0700, Alexei Starovoitov wrote:
On Tue, May 26, 2020 at 10:31 PM Yauheni Kaliuta yauheni.kaliuta@redhat.com wrote:
Hi, Andrii!
>>>>> On Tue, 26 May 2020 17:19:18 -0700, Andrii Nakryiko wrote:
> On Fri, May 22, 2020 at 1:19 AM Yauheni Kaliuta > yauheni.kaliuta@redhat.com wrote: >> >> There is difference in depoying static and generated extra resource >> files between in/out of tree build and flavors: >> >> - in case of unflavored out-of-tree build static files are not >> available and must be copied as well as both static and generated >> files for flavored build. >> >> So split the rules and variables. The name TRUNNER_EXTRA_GEN_FILES >> is chosen in analogy to TEST_GEN_* variants. >>
> Can we keep them together but be smarter about what needs to > be copied based on source/target directories? I would really > like to not blow up all these rules.
I can try, ok, I just find it a bit more clear. But it's good to get some input from kselftest about OOT build in general.
I see no value in 'make install' of selftests/bpf and since it's broken just remove that makefile target.
Some CI systems perform testing next stage after building were build tree is not available anymore. So it's in use at the moment.
such CI systems can do 'cp -r' then
It's a discussion for linux-kselftest@ (added).
At the moment `make install` is generic kselftest functionality and since bpf is part of that infra it looks a bit strange to break it intentionally.
selftests/bpf is only historically part of selftests. It probably should stop using kselftest build infra all together. We had breakages in selftests/bpf in the past only because of changes in kselftests bits.
The question is whether or not the breakages addresses quickly. Also, bpf keels breaking ksleftest builds and installs because it has dependencies on bleeding edge tools and causes problems for kselftest users.
You are pulling me into the discussion midway and I am missing the context for the discussion. There is another thread on this topic where Yauheni and I have been talking about bpf install.
I would say bpf install has never really worked from kselftest install mechanism.
Ideally all tests use kselftest common install rule to leverage the install and not have users do individual test installs. It isn't productive and cooperative to say let's have bpf test do its thing. It is part of selftests and we have to figure out how to have it consistently build and run.
It isn't building for me on Linux 5.7-rc7 at the moment, leave alone install.
The test Makefile has to handle OUTPUT directory. Please add me to the entire patch series especially if it changes selftests Makefile and lib.mk. I will review and try to see if we can make bpf install work under kselftest common infrastructure.
I recently fixed bugs in kvm test Makefile to address the install problems for cross-builds.
thanks, -- Shuah
On Wed, May 27, 2020 at 3:01 PM shuah shuah@kernel.org wrote:
On 5/27/20 11:05 AM, Alexei Starovoitov wrote:
On Wed, May 27, 2020 at 10:02 AM Yauheni Kaliuta yauheni.kaliuta@redhat.com wrote:
Hi, Alexei!
>> On Wed, 27 May 2020 09:48:04 -0700, Alexei Starovoitov wrote:
On Wed, May 27, 2020 at 12:19 AM Yauheni Kaliuta yauheni.kaliuta@redhat.com wrote:
Hi, Alexei!
>>>> On Tue, 26 May 2020 22:37:39 -0700, Alexei Starovoitov wrote:
On Tue, May 26, 2020 at 10:31 PM Yauheni Kaliuta yauheni.kaliuta@redhat.com wrote: > > Hi, Andrii! > > >>>>> On Tue, 26 May 2020 17:19:18 -0700, Andrii Nakryiko wrote: > > > On Fri, May 22, 2020 at 1:19 AM Yauheni Kaliuta > > yauheni.kaliuta@redhat.com wrote: > >> > >> There is difference in depoying static and generated extra resource > >> files between in/out of tree build and flavors: > >> > >> - in case of unflavored out-of-tree build static files are not > >> available and must be copied as well as both static and generated > >> files for flavored build. > >> > >> So split the rules and variables. The name TRUNNER_EXTRA_GEN_FILES > >> is chosen in analogy to TEST_GEN_* variants. > >> > > > Can we keep them together but be smarter about what needs to > > be copied based on source/target directories? I would really > > like to not blow up all these rules. > > I can try, ok, I just find it a bit more clear. But it's good to > get some input from kselftest about OOT build in general.
I see no value in 'make install' of selftests/bpf and since it's broken just remove that makefile target.
Some CI systems perform testing next stage after building were build tree is not available anymore. So it's in use at the moment.
such CI systems can do 'cp -r' then
It's a discussion for linux-kselftest@ (added).
At the moment `make install` is generic kselftest functionality and since bpf is part of that infra it looks a bit strange to break it intentionally.
selftests/bpf is only historically part of selftests. It probably should stop using kselftest build infra all together. We had breakages in selftests/bpf in the past only because of changes in kselftests bits.
The question is whether or not the breakages addresses quickly. Also, bpf keels breaking ksleftest builds and installs because it has dependencies on bleeding edge tools and causes problems for kselftest users.
You are pulling me into the discussion midway and I am missing the context for the discussion. There is another thread on this topic where Yauheni and I have been talking about bpf install.
I would say bpf install has never really worked from kselftest install mechanism.
Ideally all tests use kselftest common install rule to leverage the install and not have users do individual test installs. It isn't productive and cooperative to say let's have bpf test do its thing. It is part of selftests and we have to figure out how to have it consistently build and run.
It isn't building for me on Linux 5.7-rc7 at the moment, leave alone install.
The test Makefile has to handle OUTPUT directory. Please add me to the entire patch series especially if it changes selftests Makefile and lib.mk. I will review and try to see if we can make bpf install work under kselftest common infrastructure.
I prefer to keep selftests/bpf install broken. This forced marriage between kselftests and selftests/bpf never worked well. I think it's a time to free them up from each other.
On Wed, 27 May 2020 15:23:13 -0700, Alexei Starovoitov wrote:
I prefer to keep selftests/bpf install broken. This forced marriage between kselftests and selftests/bpf never worked well. I think it's a time to free them up from each other.
Alexei, it would be great if you could cooperate with other people instead of pushing your own way. The selftests infrastructure was put to the kernel to have one place for testing. Inventing yet another way to add tests does not help anyone. You don't own the kernel. We're community, we should cooperate.
Jiri
On Thu, May 28, 2020 at 10:05:57AM +0200, Jiri Benc wrote:
On Wed, 27 May 2020 15:23:13 -0700, Alexei Starovoitov wrote:
I prefer to keep selftests/bpf install broken. This forced marriage between kselftests and selftests/bpf never worked well. I think it's a time to free them up from each other.
Alexei, it would be great if you could cooperate with other people instead of pushing your own way. The selftests infrastructure was put to the kernel to have one place for testing. Inventing yet another way to add tests does not help anyone. You don't own the kernel. We're community, we should cooperate.
I agree, we rely on the infrastructure of the kselftests framework so that testing systems do not have to create "custom" frameworks to handle all of the individual variants that could easily crop up here.
Let's keep it easy for people to run and use these tests, to not do so is to ensure that they are not used, which is the exact opposite goal of creating tests.
thanks,
greg k-h
On Thu, May 28, 2020 at 12:56:31PM +0200, Greg KH wrote:
On Thu, May 28, 2020 at 10:05:57AM +0200, Jiri Benc wrote:
On Wed, 27 May 2020 15:23:13 -0700, Alexei Starovoitov wrote:
I prefer to keep selftests/bpf install broken. This forced marriage between kselftests and selftests/bpf never worked well. I think it's a time to free them up from each other.
Alexei, it would be great if you could cooperate with other people instead of pushing your own way. The selftests infrastructure was put to the kernel to have one place for testing. Inventing yet another way to add tests does not help anyone. You don't own the kernel. We're community, we should cooperate.
I agree, we rely on the infrastructure of the kselftests framework so that testing systems do not have to create "custom" frameworks to handle all of the individual variants that could easily crop up here.
Let's keep it easy for people to run and use these tests, to not do so is to ensure that they are not used, which is the exact opposite goal of creating tests.
Greg,
It is easy for people (bpf developers) to run and use the tests. Every developer runs them before submitting patches. New tests is a hard requirement for any new features. Maintainers run them for every push.
What I was and will push back hard is when other people (not bpf developers) come back with an excuse that some CI system has a hard time running these tests. It's the problem of weak CI. That CI needs to be fixed. Not the tests. The example of this is that we already have github/libbpf CI that runs selftests/bpf just fine. Anyone who wants to do another CI are welcome to copy paste what already works instead of burdening people (bpf developers) who run and use existing tests. I frankly have no sympathy to folks who put their own interest of their CI development in front of bpf community of developers. The main job of CI is to help developers and maintainers. Where helping means to not impose new dumb rules on developers because CI framework is dumb. Fix CI instead.
On 5/28/20 10:14 AM, Alexei Starovoitov wrote:
On Thu, May 28, 2020 at 12:56:31PM +0200, Greg KH wrote:
On Thu, May 28, 2020 at 10:05:57AM +0200, Jiri Benc wrote:
On Wed, 27 May 2020 15:23:13 -0700, Alexei Starovoitov wrote:
I prefer to keep selftests/bpf install broken. This forced marriage between kselftests and selftests/bpf never worked well. I think it's a time to free them up from each other.
Alexei, it would be great if you could cooperate with other people instead of pushing your own way. The selftests infrastructure was put to the kernel to have one place for testing. Inventing yet another way to add tests does not help anyone. You don't own the kernel. We're community, we should cooperate.
I agree, we rely on the infrastructure of the kselftests framework so that testing systems do not have to create "custom" frameworks to handle all of the individual variants that could easily crop up here.
Let's keep it easy for people to run and use these tests, to not do so is to ensure that they are not used, which is the exact opposite goal of creating tests.
Greg,
It is easy for people (bpf developers) to run and use the tests. Every developer runs them before submitting patches. New tests is a hard requirement for any new features. Maintainers run them for every push.
What I was and will push back hard is when other people (not bpf developers) come back with an excuse that some CI system has a hard time running these tests. It's the problem of weak CI. That CI needs to be fixed. Not the tests. The example of this is that we already have github/libbpf CI that runs selftests/bpf just fine. Anyone who wants to do another CI are welcome to copy paste what already works instead of burdening people (bpf developers) who run and use existing tests. I frankly have no sympathy to folks who put their own interest of their CI development in front of bpf community of developers. The main job of CI is to help developers and maintainers. Where helping means to not impose new dumb rules on developers because CI framework is dumb. Fix CI instead.
Here is what CI users are requesting:
- ability to install bpf test with other selftests using kselftest install. The common framework is in place and with minor changes to bpf test Makefile, we can make this happen. Others and myself are willing to work on this, so we can get bpf test coverage in test rings.
- be able to build and run existing tests without breaking the test build when new tests are added that have hard dependency on new versions of tools (llvm etc.). This isn't such a novel idea. We don't break kernel builds every single release and even when we require newer compiler releases. Plan the new tests with the intent to not break existing users and add new tests at the same time. We use min rev and not bleeding edge as the requirement for kernel build.
Requiring test rings upgrade to new versions of llvm is unreasonable. It places undue burden on the admins to do this every single release (may be even every rc cycle)
What is dumb about these requests and why is it not acceptable to just bpf when all other sub-systems keep adding tests continuously using the selftests framework so we can test the kernel better and our releases are of better quality.
If you check the volume of tests that get added every release, you can easily see it isn't hard.
Calling the needs of CI dumb is detrimental to kernel quality as these rings provide a very important function. Addressing their use-case helps get better test coverage for bpf and kernel areas that use bpf.
thanks, -- Shuah
Hi, Alexei,
On Thu, May 28, 2020 at 7:14 PM Alexei Starovoitov alexei.starovoitov@gmail.com wrote:
On Thu, May 28, 2020 at 12:56:31PM +0200, Greg KH wrote:
On Thu, May 28, 2020 at 10:05:57AM +0200, Jiri Benc wrote:
On Wed, 27 May 2020 15:23:13 -0700, Alexei Starovoitov wrote:
I prefer to keep selftests/bpf install broken. This forced marriage between kselftests and selftests/bpf never worked well. I think it's a time to free them up from each other.
Alexei, it would be great if you could cooperate with other people instead of pushing your own way. The selftests infrastructure was put to the kernel to have one place for testing. Inventing yet another way to add tests does not help anyone. You don't own the kernel. We're community, we should cooperate.
I agree, we rely on the infrastructure of the kselftests framework so that testing systems do not have to create "custom" frameworks to handle all of the individual variants that could easily crop up here.
Let's keep it easy for people to run and use these tests, to not do so is to ensure that they are not used, which is the exact opposite goal of creating tests.
Greg,
It is easy for people (bpf developers) to run and use the tests. Every developer runs them before submitting patches. New tests is a hard requirement for any new features. Maintainers run them for every push.
What I was and will push back hard is when other people (not bpf developers) come back with an excuse that some CI system has a hard time running these tests. It's the problem of weak CI. That CI needs to be fixed. Not the tests. The example of this is that we already have github/libbpf CI that runs selftests/bpf just fine. Anyone who wants to do another CI are welcome to copy paste what already works instead of burdening people (bpf developers) who run and use existing tests. I frankly have no sympathy to folks who put their own interest of their CI development in front of bpf community of developers. The main job of CI is to help developers and maintainers. Where helping means to not impose new dumb rules on developers because CI framework is dumb. Fix CI instead.
Any good reason why bpf selftests, residing under selftests/, should be an exception? "Breakages" is not, breakages are fixable.
On Thu, May 28, 2020 at 11:07:09AM -0600, Shuah Khan wrote:
On 5/28/20 10:14 AM, Alexei Starovoitov wrote:
On Thu, May 28, 2020 at 12:56:31PM +0200, Greg KH wrote:
On Thu, May 28, 2020 at 10:05:57AM +0200, Jiri Benc wrote:
On Wed, 27 May 2020 15:23:13 -0700, Alexei Starovoitov wrote:
I prefer to keep selftests/bpf install broken. This forced marriage between kselftests and selftests/bpf never worked well. I think it's a time to free them up from each other.
Alexei, it would be great if you could cooperate with other people instead of pushing your own way. The selftests infrastructure was put to the kernel to have one place for testing. Inventing yet another way to add tests does not help anyone. You don't own the kernel. We're community, we should cooperate.
I agree, we rely on the infrastructure of the kselftests framework so that testing systems do not have to create "custom" frameworks to handle all of the individual variants that could easily crop up here.
Let's keep it easy for people to run and use these tests, to not do so is to ensure that they are not used, which is the exact opposite goal of creating tests.
Greg,
It is easy for people (bpf developers) to run and use the tests. Every developer runs them before submitting patches. New tests is a hard requirement for any new features. Maintainers run them for every push.
What I was and will push back hard is when other people (not bpf developers) come back with an excuse that some CI system has a hard time running these tests. It's the problem of weak CI. That CI needs to be fixed. Not the tests. The example of this is that we already have github/libbpf CI that runs selftests/bpf just fine. Anyone who wants to do another CI are welcome to copy paste what already works instead of burdening people (bpf developers) who run and use existing tests. I frankly have no sympathy to folks who put their own interest of their CI development in front of bpf community of developers. The main job of CI is to help developers and maintainers. Where helping means to not impose new dumb rules on developers because CI framework is dumb. Fix CI instead.
Here is what CI users are requesting:
- ability to install bpf test with other selftests using kselftest install. The common framework is in place and with minor changes to bpf test Makefile, we can make this happen. Others and myself are willing to work on this, so we can get bpf test coverage in test rings.
so you're saying that bpf maintainers and all bpf developers now would need to incorporate new 'make install' step to their workflow because some unknown CI system that is not even functional decided to do 'make install' ? That's exactly my point about selfish CI developers who put their needs in front of bpf community of developers.
- be able to build and run existing tests without breaking the test build when new tests are added that have hard dependency on new versions of tools (llvm etc.). This isn't such a novel idea. We don't break kernel builds every single release and even when we require newer compiler releases. Plan the new tests with the intent to not break existing users and add new tests at the same time. We use min rev and not bleeding edge as the requirement for kernel build.
'existing users'? CI is not a user. CI is machine that should _help_ developers. Above two things (forcing install on humans and breaking day-to-day tests for bpf maintainers and developers) is the opposite of helping developers. Please do NOT use such useless CI as an excuse. Such CI should not be built in the first place when it slows down the development instead of helping it.
On Thu, May 28, 2020 at 08:10:57PM +0300, Yauheni Kaliuta wrote:
Hi, Alexei,
On Thu, May 28, 2020 at 7:14 PM Alexei Starovoitov alexei.starovoitov@gmail.com wrote:
On Thu, May 28, 2020 at 12:56:31PM +0200, Greg KH wrote:
On Thu, May 28, 2020 at 10:05:57AM +0200, Jiri Benc wrote:
On Wed, 27 May 2020 15:23:13 -0700, Alexei Starovoitov wrote:
I prefer to keep selftests/bpf install broken. This forced marriage between kselftests and selftests/bpf never worked well. I think it's a time to free them up from each other.
Alexei, it would be great if you could cooperate with other people instead of pushing your own way. The selftests infrastructure was put to the kernel to have one place for testing. Inventing yet another way to add tests does not help anyone. You don't own the kernel. We're community, we should cooperate.
I agree, we rely on the infrastructure of the kselftests framework so that testing systems do not have to create "custom" frameworks to handle all of the individual variants that could easily crop up here.
Let's keep it easy for people to run and use these tests, to not do so is to ensure that they are not used, which is the exact opposite goal of creating tests.
Greg,
It is easy for people (bpf developers) to run and use the tests. Every developer runs them before submitting patches. New tests is a hard requirement for any new features. Maintainers run them for every push.
What I was and will push back hard is when other people (not bpf developers) come back with an excuse that some CI system has a hard time running these tests. It's the problem of weak CI. That CI needs to be fixed. Not the tests. The example of this is that we already have github/libbpf CI that runs selftests/bpf just fine. Anyone who wants to do another CI are welcome to copy paste what already works instead of burdening people (bpf developers) who run and use existing tests. I frankly have no sympathy to folks who put their own interest of their CI development in front of bpf community of developers. The main job of CI is to help developers and maintainers. Where helping means to not impose new dumb rules on developers because CI framework is dumb. Fix CI instead.
Any good reason why bpf selftests, residing under selftests/, should be an exception? "Breakages" is not, breakages are fixable.
As I said early the location of bpf selftests in tools/testing/selftests/ was a historical mistake that needs to be corrected. There is no value in residing in that directory. kselftest are aiming to test the kernel whereas selftests/bpf are testing kernel, libbpf, bpftool, llvm, pahole. These are the tests for bpf ecosystem.
Hi!
On Thu, May 28, 2020 at 9:16 PM Alexei Starovoitov alexei.starovoitov@gmail.com wrote:
On Thu, May 28, 2020 at 11:07:09AM -0600, Shuah Khan wrote:
[...]
Here is what CI users are requesting:
- ability to install bpf test with other selftests using kselftest install. The common framework is in place and with minor changes to bpf test Makefile, we can make this happen. Others and myself are willing to work on this, so we can get bpf test coverage in test rings.
so you're saying that bpf maintainers and all bpf developers now would need to incorporate new 'make install' step to their workflow because some unknown CI system that is not even functional decided to do 'make install' ? That's exactly my point about selfish CI developers who put their needs in front of bpf community of developers.
May be, it can work both ways to make everybody happy :) (I haven't seen yet fundamental problems why not).
On Thu, May 28, 2020 at 11:29 AM Yauheni Kaliuta yauheni.kaliuta@redhat.com wrote:
Hi!
On Thu, May 28, 2020 at 9:16 PM Alexei Starovoitov alexei.starovoitov@gmail.com wrote:
On Thu, May 28, 2020 at 11:07:09AM -0600, Shuah Khan wrote:
[...]
Here is what CI users are requesting:
- ability to install bpf test with other selftests using kselftest install. The common framework is in place and with minor changes to bpf test Makefile, we can make this happen. Others and myself are willing to work on this, so we can get bpf test coverage in test rings.
so you're saying that bpf maintainers and all bpf developers now would need to incorporate new 'make install' step to their workflow because some unknown CI system that is not even functional decided to do 'make install' ? That's exactly my point about selfish CI developers who put their needs in front of bpf community of developers.
May be, it can work both ways to make everybody happy :) (I haven't seen yet fundamental problems why not).
then stop pretending and do 'cp -r' for your CI as you were suggested several times already. It works just fine for libbpf CI. Feel free to copy those scripts.
On 5/28/20 12:15 PM, Alexei Starovoitov wrote:
On Thu, May 28, 2020 at 11:07:09AM -0600, Shuah Khan wrote:
On 5/28/20 10:14 AM, Alexei Starovoitov wrote:
On Thu, May 28, 2020 at 12:56:31PM +0200, Greg KH wrote:
On Thu, May 28, 2020 at 10:05:57AM +0200, Jiri Benc wrote:
On Wed, 27 May 2020 15:23:13 -0700, Alexei Starovoitov wrote:
I prefer to keep selftests/bpf install broken. This forced marriage between kselftests and selftests/bpf never worked well. I think it's a time to free them up from each other.
Alexei, it would be great if you could cooperate with other people instead of pushing your own way. The selftests infrastructure was put to the kernel to have one place for testing. Inventing yet another way to add tests does not help anyone. You don't own the kernel. We're community, we should cooperate.
I agree, we rely on the infrastructure of the kselftests framework so that testing systems do not have to create "custom" frameworks to handle all of the individual variants that could easily crop up here.
Let's keep it easy for people to run and use these tests, to not do so is to ensure that they are not used, which is the exact opposite goal of creating tests.
Greg,
It is easy for people (bpf developers) to run and use the tests. Every developer runs them before submitting patches. New tests is a hard requirement for any new features. Maintainers run them for every push.
What I was and will push back hard is when other people (not bpf developers) come back with an excuse that some CI system has a hard time running these tests. It's the problem of weak CI. That CI needs to be fixed. Not the tests. The example of this is that we already have github/libbpf CI that runs selftests/bpf just fine. Anyone who wants to do another CI are welcome to copy paste what already works instead of burdening people (bpf developers) who run and use existing tests. I frankly have no sympathy to folks who put their own interest of their CI development in front of bpf community of developers. The main job of CI is to help developers and maintainers. Where helping means to not impose new dumb rules on developers because CI framework is dumb. Fix CI instead.
Here is what CI users are requesting:
- ability to install bpf test with other selftests using kselftest install. The common framework is in place and with minor changes to bpf test Makefile, we can make this happen. Others and myself are willing to work on this, so we can get bpf test coverage in test rings.
so you're saying that bpf maintainers and all bpf developers now would need to incorporate new 'make install' step to their workflow because some unknown CI system that is not even functional decided to do 'make install' ? That's exactly my point about selfish CI developers who put their needs in front of bpf community of developers.
There is no need change bpf maintainer and developer workflow. You don't have to use install option. Kselftest framework doesn't require a specific workflow and you can:
1. Build and run your tests from bpf directory if you choose to 2. Install to run on different target.
Adding install install option requires a change to bpf Makefile only to copy test that are built to install directory.
make kselftest-install from the main kernel Makefile in conjunction with selftests Makefile and lib.mk will handle all of that.
Sounds like there is a misunderstanding that bpf maintainer/developer workflow will have to change to support install. That is not the case. The reason kselftest exists on the first place is to have common framework to take care of build/run/install as a common layer so individual test writers don't have to worry about these details and write tests instead.
- be able to build and run existing tests without breaking the test build when new tests are added that have hard dependency on new versions of tools (llvm etc.). This isn't such a novel idea. We don't break kernel builds every single release and even when we require newer compiler releases. Plan the new tests with the intent to not break existing users and add new tests at the same time. We use min rev and not bleeding edge as the requirement for kernel build.
'existing users'?
I said existing tests not users. When you add new bpf tests, existing tests should continue to build and run without dependency on new revs of llvm.
CI is not a user. CI is machine that should _help_
developers. Above two things (forcing install on humans and breaking day-to-day tests for bpf maintainers and developers) is the opposite of helping developers. Please do NOT use such useless CI as an excuse. Such CI should not be built in the first place when it slows down the development instead of helping it.
LKFT test ring runs selftests now and Kernel CI will be soon. Various users run selftests in their own environments. LKFT ring admins have to build and install tests whether it is automated or not and look at the results. If bpf test doesn't build and/or installed, it won't run on test rings that qualify stable/next/main releases.
I don't understand why CI use-case is useless.
thanks, -- Shuah
On 5/28/20 12:34 PM, Alexei Starovoitov wrote:
On Thu, May 28, 2020 at 11:29 AM Yauheni Kaliuta yauheni.kaliuta@redhat.com wrote:
Hi!
On Thu, May 28, 2020 at 9:16 PM Alexei Starovoitov alexei.starovoitov@gmail.com wrote:
On Thu, May 28, 2020 at 11:07:09AM -0600, Shuah Khan wrote:
[...]
Here is what CI users are requesting:
- ability to install bpf test with other selftests using kselftest install. The common framework is in place and with minor changes to bpf test Makefile, we can make this happen. Others and myself are willing to work on this, so we can get bpf test coverage in test rings.
so you're saying that bpf maintainers and all bpf developers now would need to incorporate new 'make install' step to their workflow because some unknown CI system that is not even functional decided to do 'make install' ? That's exactly my point about selfish CI developers who put their needs in front of bpf community of developers.
May be, it can work both ways to make everybody happy :) (I haven't seen yet fundamental problems why not).
then stop pretending and do 'cp -r' for your CI as you were suggested several times already. It works just fine for libbpf CI. Feel free to copy those scripts.
Yauheni,
Let's drop discussing this patch set. I don't have all the patches and changes in this to weigh in. There is no need to add cp -r at all.
I want to take the discussion one level up to the use-cases.
thanks, -- Shuah
On 5/28/20 11:10 AM, Yauheni Kaliuta wrote:
Hi, Alexei,
On Thu, May 28, 2020 at 7:14 PM Alexei Starovoitov alexei.starovoitov@gmail.com wrote:
On Thu, May 28, 2020 at 12:56:31PM +0200, Greg KH wrote:
On Thu, May 28, 2020 at 10:05:57AM +0200, Jiri Benc wrote:
On Wed, 27 May 2020 15:23:13 -0700, Alexei Starovoitov wrote:
I prefer to keep selftests/bpf install broken. This forced marriage between kselftests and selftests/bpf never worked well. I think it's a time to free them up from each other.
Alexei, it would be great if you could cooperate with other people instead of pushing your own way. The selftests infrastructure was put to the kernel to have one place for testing. Inventing yet another way to add tests does not help anyone. You don't own the kernel. We're community, we should cooperate.
I agree, we rely on the infrastructure of the kselftests framework so that testing systems do not have to create "custom" frameworks to handle all of the individual variants that could easily crop up here.
Let's keep it easy for people to run and use these tests, to not do so is to ensure that they are not used, which is the exact opposite goal of creating tests.
Greg,
It is easy for people (bpf developers) to run and use the tests. Every developer runs them before submitting patches. New tests is a hard requirement for any new features. Maintainers run them for every push.
What I was and will push back hard is when other people (not bpf developers) come back with an excuse that some CI system has a hard time running these tests. It's the problem of weak CI. That CI needs to be fixed. Not the tests. The example of this is that we already have github/libbpf CI that runs selftests/bpf just fine. Anyone who wants to do another CI are welcome to copy paste what already works instead of burdening people (bpf developers) who run and use existing tests. I frankly have no sympathy to folks who put their own interest of their CI development in front of bpf community of developers. The main job of CI is to help developers and maintainers. Where helping means to not impose new dumb rules on developers because CI framework is dumb. Fix CI instead.
Any good reason why bpf selftests, residing under selftests/, should be an exception? "Breakages" is not, breakages are fixable.
Let's not talk about moving tests. I don't want to discuss that until we are all on the same page on what is the problem in adding install support to bpf Makefile.
It is possible that there is a misunderstanding that bpf maintainer and developer workflow will change. Which is definitely not needed. If this patch series requires it, it isn't correct and needs to be reworked.
thanks, -- Shuah
On Thu, May 28, 2020 at 12:59:06PM -0600, Shuah Khan wrote:
On 5/28/20 12:15 PM, Alexei Starovoitov wrote:
On Thu, May 28, 2020 at 11:07:09AM -0600, Shuah Khan wrote:
On 5/28/20 10:14 AM, Alexei Starovoitov wrote:
On Thu, May 28, 2020 at 12:56:31PM +0200, Greg KH wrote:
On Thu, May 28, 2020 at 10:05:57AM +0200, Jiri Benc wrote:
On Wed, 27 May 2020 15:23:13 -0700, Alexei Starovoitov wrote: > I prefer to keep selftests/bpf install broken. > This forced marriage between kselftests and selftests/bpf > never worked well. I think it's a time to free them up from each other.
Alexei, it would be great if you could cooperate with other people instead of pushing your own way. The selftests infrastructure was put to the kernel to have one place for testing. Inventing yet another way to add tests does not help anyone. You don't own the kernel. We're community, we should cooperate.
I agree, we rely on the infrastructure of the kselftests framework so that testing systems do not have to create "custom" frameworks to handle all of the individual variants that could easily crop up here.
Let's keep it easy for people to run and use these tests, to not do so is to ensure that they are not used, which is the exact opposite goal of creating tests.
Greg,
It is easy for people (bpf developers) to run and use the tests. Every developer runs them before submitting patches. New tests is a hard requirement for any new features. Maintainers run them for every push.
What I was and will push back hard is when other people (not bpf developers) come back with an excuse that some CI system has a hard time running these tests. It's the problem of weak CI. That CI needs to be fixed. Not the tests. The example of this is that we already have github/libbpf CI that runs selftests/bpf just fine. Anyone who wants to do another CI are welcome to copy paste what already works instead of burdening people (bpf developers) who run and use existing tests. I frankly have no sympathy to folks who put their own interest of their CI development in front of bpf community of developers. The main job of CI is to help developers and maintainers. Where helping means to not impose new dumb rules on developers because CI framework is dumb. Fix CI instead.
Here is what CI users are requesting:
- ability to install bpf test with other selftests using kselftest install. The common framework is in place and with minor changes to bpf test Makefile, we can make this happen. Others and myself are willing to work on this, so we can get bpf test coverage in test rings.
so you're saying that bpf maintainers and all bpf developers now would need to incorporate new 'make install' step to their workflow because some unknown CI system that is not even functional decided to do 'make install' ? That's exactly my point about selfish CI developers who put their needs in front of bpf community of developers.
There is no need change bpf maintainer and developer workflow. You don't have to use install option. Kselftest framework doesn't require a specific workflow and you can:
- Build and run your tests from bpf directory if you choose to
- Install to run on different target.
Adding install install option requires a change to bpf Makefile only to copy test that are built to install directory.
make kselftest-install from the main kernel Makefile in conjunction with selftests Makefile and lib.mk will handle all of that.
Sounds like there is a misunderstanding that bpf maintainer/developer workflow will have to change to support install. That is not the case. The reason kselftest exists on the first place is to have common framework to take care of build/run/install as a common layer so individual test writers don't have to worry about these details and write tests instead.
I don't think you understand the 'make install' implications. Not doing 'make install' means that all the Makefile changes that Yauheni is proposing will immediately bit rot. People are constantly adding new tests and changing makefile. 'make install' _will_ be broken instantly unless _humans_ incorporate it in their patch development process and maintainer incorporate that step into their workflow as well.
Ohh, but don't worry about this broken 'make install' is not an answer. It's broken now and I don't want to see patches that move it from one broken state into another broken state and at the same time add complexity to it.
That's very different from 'make install' doing 'cp -r' of the whole dir. In such case the chances of it going stale and broken are much lower.
- be able to build and run existing tests without breaking the test build when new tests are added that have hard dependency on new versions of tools (llvm etc.). This isn't such a novel idea. We don't break kernel builds every single release and even when we require newer compiler releases. Plan the new tests with the intent to not break existing users and add new tests at the same time. We use min rev and not bleeding edge as the requirement for kernel build.
'existing users'?
I said existing tests not users. When you add new bpf tests, existing tests should continue to build and run without dependency on new revs of llvm.
Who said that existing test stop building with new llvm ? Please check your facts.
If bpf test doesn't build and/or installed, it won't run on test rings that qualify stable/next/main releases.
That's the case today and I prefer to keep it this way. stable folks ignored our recommendations on how selftests/bpf should be run, so please don't come up with your ways of doing it and complain that something doesn't work.
On Thu, May 28, 2020 at 10:09 PM Shuah Khan skhan@linuxfoundation.org wrote:
On 5/28/20 11:10 AM, Yauheni Kaliuta wrote:
Hi, Alexei,
On Thu, May 28, 2020 at 7:14 PM Alexei Starovoitov alexei.starovoitov@gmail.com wrote:
On Thu, May 28, 2020 at 12:56:31PM +0200, Greg KH wrote:
On Thu, May 28, 2020 at 10:05:57AM +0200, Jiri Benc wrote:
On Wed, 27 May 2020 15:23:13 -0700, Alexei Starovoitov wrote:
I prefer to keep selftests/bpf install broken. This forced marriage between kselftests and selftests/bpf never worked well. I think it's a time to free them up from each other.
Alexei, it would be great if you could cooperate with other people instead of pushing your own way. The selftests infrastructure was put to the kernel to have one place for testing. Inventing yet another way to add tests does not help anyone. You don't own the kernel. We're community, we should cooperate.
I agree, we rely on the infrastructure of the kselftests framework so that testing systems do not have to create "custom" frameworks to handle all of the individual variants that could easily crop up here.
Let's keep it easy for people to run and use these tests, to not do so is to ensure that they are not used, which is the exact opposite goal of creating tests.
Greg,
It is easy for people (bpf developers) to run and use the tests. Every developer runs them before submitting patches. New tests is a hard requirement for any new features. Maintainers run them for every push.
What I was and will push back hard is when other people (not bpf developers) come back with an excuse that some CI system has a hard time running these tests. It's the problem of weak CI. That CI needs to be fixed. Not the tests. The example of this is that we already have github/libbpf CI that runs selftests/bpf just fine. Anyone who wants to do another CI are welcome to copy paste what already works instead of burdening people (bpf developers) who run and use existing tests. I frankly have no sympathy to folks who put their own interest of their CI development in front of bpf community of developers. The main job of CI is to help developers and maintainers. Where helping means to not impose new dumb rules on developers because CI framework is dumb. Fix CI instead.
Any good reason why bpf selftests, residing under selftests/, should be an exception? "Breakages" is not, breakages are fixable.
Let's not talk about moving tests. I don't want to discuss that until we are all on the same page on what is the problem in adding install support to bpf Makefile.
It is possible that there is a misunderstanding that bpf maintainer and developer workflow will change. Which is definitely not needed. If this patch series requires it, it isn't correct and needs to be reworked.
patch series does not change it. Running bpf tests in-place is one of my own usecases, I'm not going to break it :) The series tried to fix what does not work (install and build in $(OUTPUT) directory) but exists in the code. I'll include you in Cc for the respin(s) if you are interested.
The discussion is pretty much not connected to the patchset, but about bpf selftests and the infra in general.
On 5/28/20 1:20 PM, Yauheni Kaliuta wrote:
On Thu, May 28, 2020 at 10:09 PM Shuah Khan skhan@linuxfoundation.org wrote:
On 5/28/20 11:10 AM, Yauheni Kaliuta wrote:
Hi, Alexei,
On Thu, May 28, 2020 at 7:14 PM Alexei Starovoitov alexei.starovoitov@gmail.com wrote:
On Thu, May 28, 2020 at 12:56:31PM +0200, Greg KH wrote:
On Thu, May 28, 2020 at 10:05:57AM +0200, Jiri Benc wrote:
On Wed, 27 May 2020 15:23:13 -0700, Alexei Starovoitov wrote: > I prefer to keep selftests/bpf install broken. > This forced marriage between kselftests and selftests/bpf > never worked well. I think it's a time to free them up from each other.
Alexei, it would be great if you could cooperate with other people instead of pushing your own way. The selftests infrastructure was put to the kernel to have one place for testing. Inventing yet another way to add tests does not help anyone. You don't own the kernel. We're community, we should cooperate.
I agree, we rely on the infrastructure of the kselftests framework so that testing systems do not have to create "custom" frameworks to handle all of the individual variants that could easily crop up here.
Let's keep it easy for people to run and use these tests, to not do so is to ensure that they are not used, which is the exact opposite goal of creating tests.
Greg,
It is easy for people (bpf developers) to run and use the tests. Every developer runs them before submitting patches. New tests is a hard requirement for any new features. Maintainers run them for every push.
What I was and will push back hard is when other people (not bpf developers) come back with an excuse that some CI system has a hard time running these tests. It's the problem of weak CI. That CI needs to be fixed. Not the tests. The example of this is that we already have github/libbpf CI that runs selftests/bpf just fine. Anyone who wants to do another CI are welcome to copy paste what already works instead of burdening people (bpf developers) who run and use existing tests. I frankly have no sympathy to folks who put their own interest of their CI development in front of bpf community of developers. The main job of CI is to help developers and maintainers. Where helping means to not impose new dumb rules on developers because CI framework is dumb. Fix CI instead.
Any good reason why bpf selftests, residing under selftests/, should be an exception? "Breakages" is not, breakages are fixable.
Let's not talk about moving tests. I don't want to discuss that until we are all on the same page on what is the problem in adding install support to bpf Makefile.
It is possible that there is a misunderstanding that bpf maintainer and developer workflow will change. Which is definitely not needed. If this patch series requires it, it isn't correct and needs to be reworked.
patch series does not change it. Running bpf tests in-place is one of my own usecases, I'm not going to break it :)
Great. I wasn't suggesting you would. So there is no need for the concern that bpf developer/maintainer workflow will have to change to add install support.
The series tried to fix what does not work (install and build in $(OUTPUT) directory) but exists in the code. I'll include you in Cc for the respin(s) if you are interested.
Please do.
The discussion is pretty much not connected to the patchset, but about bpf selftests and the infra in general.
Yeah. We have to get a better understanding and be on the same page on this anyway.
thanks, -- Shuah
On 5/28/20 1:18 PM, Alexei Starovoitov wrote:
On Thu, May 28, 2020 at 12:59:06PM -0600, Shuah Khan wrote:
On 5/28/20 12:15 PM, Alexei Starovoitov wrote:
On Thu, May 28, 2020 at 11:07:09AM -0600, Shuah Khan wrote:
On 5/28/20 10:14 AM, Alexei Starovoitov wrote:
On Thu, May 28, 2020 at 12:56:31PM +0200, Greg KH wrote:
On Thu, May 28, 2020 at 10:05:57AM +0200, Jiri Benc wrote: > On Wed, 27 May 2020 15:23:13 -0700, Alexei Starovoitov wrote: >> I prefer to keep selftests/bpf install broken. >> This forced marriage between kselftests and selftests/bpf >> never worked well. I think it's a time to free them up from each other. > > Alexei, it would be great if you could cooperate with other people > instead of pushing your own way. The selftests infrastructure was put > to the kernel to have one place for testing. Inventing yet another way > to add tests does not help anyone. You don't own the kernel. We're > community, we should cooperate.
I agree, we rely on the infrastructure of the kselftests framework so that testing systems do not have to create "custom" frameworks to handle all of the individual variants that could easily crop up here.
Let's keep it easy for people to run and use these tests, to not do so is to ensure that they are not used, which is the exact opposite goal of creating tests.
Greg,
It is easy for people (bpf developers) to run and use the tests. Every developer runs them before submitting patches. New tests is a hard requirement for any new features. Maintainers run them for every push.
What I was and will push back hard is when other people (not bpf developers) come back with an excuse that some CI system has a hard time running these tests. It's the problem of weak CI. That CI needs to be fixed. Not the tests. The example of this is that we already have github/libbpf CI that runs selftests/bpf just fine. Anyone who wants to do another CI are welcome to copy paste what already works instead of burdening people (bpf developers) who run and use existing tests. I frankly have no sympathy to folks who put their own interest of their CI development in front of bpf community of developers. The main job of CI is to help developers and maintainers. Where helping means to not impose new dumb rules on developers because CI framework is dumb. Fix CI instead.
Here is what CI users are requesting:
- ability to install bpf test with other selftests using kselftest install. The common framework is in place and with minor changes to bpf test Makefile, we can make this happen. Others and myself are willing to work on this, so we can get bpf test coverage in test rings.
so you're saying that bpf maintainers and all bpf developers now would need to incorporate new 'make install' step to their workflow because some unknown CI system that is not even functional decided to do 'make install' ? That's exactly my point about selfish CI developers who put their needs in front of bpf community of developers.
There is no need change bpf maintainer and developer workflow. You don't have to use install option. Kselftest framework doesn't require a specific workflow and you can:
- Build and run your tests from bpf directory if you choose to
- Install to run on different target.
Adding install install option requires a change to bpf Makefile only to copy test that are built to install directory.
make kselftest-install from the main kernel Makefile in conjunction with selftests Makefile and lib.mk will handle all of that.
Sounds like there is a misunderstanding that bpf maintainer/developer workflow will have to change to support install. That is not the case. The reason kselftest exists on the first place is to have common framework to take care of build/run/install as a common layer so individual test writers don't have to worry about these details and write tests instead.
I don't think you understand the 'make install' implications. Not doing 'make install' means that all the Makefile changes that Yauheni is proposing will immediately bit rot. People are constantly adding new tests and changing makefile. 'make install' _will_ be broken instantly unless _humans_ incorporate it in their patch development process and maintainer incorporate that step into their workflow as well.
I don't think so. I think if you want to work with us on this, we can find a way. bpf isn't such a unique test that adding install will break it.
Ohh, but don't worry about this broken 'make install' is not an answer. It's broken now and I don't want to see patches that move it from one broken state into another broken state and at the same time add complexity to it.
Well! What you are saying "I don;t want to collaborate to find a solution to the problem".
That's very different from 'make install' doing 'cp -r' of the whole dir. In such case the chances of it going stale and broken are much lower.
- be able to build and run existing tests without breaking the test build when new tests are added that have hard dependency on new versions of tools (llvm etc.). This isn't such a novel idea. We don't break kernel builds every single release and even when we require newer compiler releases. Plan the new tests with the intent to not break existing users and add new tests at the same time. We use min rev and not bleeding edge as the requirement for kernel build.
'existing users'?
I said existing tests not users. When you add new bpf tests, existing tests should continue to build and run without dependency on new revs of llvm.
Who said that existing test stop building with new llvm ? Please check your facts.
I am basing this on a previous discussion on this topic. bpf test(s) that build stop building and the solution always has been upgrade to new tool chain. If this changed now and there is no hard tie to bpf test building and llvm release, great.
If bpf test doesn't build and/or installed, it won't run on test rings that qualify stable/next/main releases.
That's the case today and I prefer to keep it this way.
Why is that? Are you making a decision for the rest of the kernel with this approach? If bpf doesn't get tested in stable test rings, this isn't bpf problem alone. This is a kernel release issue.
stable folks ignored our recommendations on how selftests/bpf should be run, so please don't come up with your ways of doing it and complain that something doesn't work.
If you want to talk about history, bpf test was added in Oct 2016 and by then kselftest was in use with its run_tests and install support. So it is misleading to suggest that the framework came after the bpf test. bpf test was added to existing kselftest framework used by all other tests. Expecting the existing framework to change and adapt to a new test isn't reasonable.
When a new test is added, it has to work within the framework. Framework can be improved and changed, however not the way you are attempting to do in this thread. You can do this like others in the community to do by making changes and improvements.
Sadly, it doesn't appear we are getting anywhere productive with this thread. :(
thanks, -- Shuah
On 5/28/20 2:09 PM, Shuah Khan wrote:
On 5/28/20 1:18 PM, Alexei Starovoitov wrote:
On Thu, May 28, 2020 at 12:59:06PM -0600, Shuah Khan wrote:
On 5/28/20 12:15 PM, Alexei Starovoitov wrote:
On Thu, May 28, 2020 at 11:07:09AM -0600, Shuah Khan wrote:
On 5/28/20 10:14 AM, Alexei Starovoitov wrote:
On Thu, May 28, 2020 at 12:56:31PM +0200, Greg KH wrote: > On Thu, May 28, 2020 at 10:05:57AM +0200, Jiri Benc wrote: >> On Wed, 27 May 2020 15:23:13 -0700, Alexei Starovoitov wrote: >>> I prefer to keep selftests/bpf install broken. >>> This forced marriage between kselftests and selftests/bpf >>> never worked well. I think it's a time to free them up from >>> each other. >> >> Alexei, it would be great if you could cooperate with other people >> instead of pushing your own way. The selftests infrastructure >> was put >> to the kernel to have one place for testing. Inventing yet >> another way >> to add tests does not help anyone. You don't own the kernel. We're >> community, we should cooperate. > > I agree, we rely on the infrastructure of the kselftests > framework so > that testing systems do not have to create "custom" frameworks to > handle > all of the individual variants that could easily crop up here. > > Let's keep it easy for people to run and use these tests, to not > do so > is to ensure that they are not used, which is the exact opposite > goal of > creating tests.
Greg,
It is easy for people (bpf developers) to run and use the tests. Every developer runs them before submitting patches. New tests is a hard requirement for any new features. Maintainers run them for every push.
What I was and will push back hard is when other people (not bpf developers) come back with an excuse that some CI system has a hard time running these tests. It's the problem of weak CI. That CI needs to be fixed. Not the tests. The example of this is that we already have github/libbpf CI that runs selftests/bpf just fine. Anyone who wants to do another CI are welcome to copy paste what already works instead of burdening people (bpf developers) who run and use existing tests. I frankly have no sympathy to folks who put their own interest of their CI development in front of bpf community of developers. The main job of CI is to help developers and maintainers. Where helping means to not impose new dumb rules on developers because CI framework is dumb. Fix CI instead.
Here is what CI users are requesting:
- ability to install bpf test with other selftests using kselftest
install. The common framework is in place and with minor changes to bpf test Makefile, we can make this happen. Others and myself are willing to work on this, so we can get bpf test coverage in test rings.
so you're saying that bpf maintainers and all bpf developers now would need to incorporate new 'make install' step to their workflow because some unknown CI system that is not even functional decided to do 'make install' ? That's exactly my point about selfish CI developers who put their needs in front of bpf community of developers.
There is no need change bpf maintainer and developer workflow. You don't have to use install option. Kselftest framework doesn't require a specific workflow and you can:
- Build and run your tests from bpf directory if you choose to
- Install to run on different target.
Adding install install option requires a change to bpf Makefile only to copy test that are built to install directory.
make kselftest-install from the main kernel Makefile in conjunction with selftests Makefile and lib.mk will handle all of that.
Sounds like there is a misunderstanding that bpf maintainer/developer workflow will have to change to support install. That is not the case. The reason kselftest exists on the first place is to have common framework to take care of build/run/install as a common layer so individual test writers don't have to worry about these details and write tests instead.
I don't think you understand the 'make install' implications. Not doing 'make install' means that all the Makefile changes that Yauheni is proposing will immediately bit rot. People are constantly adding new tests and changing makefile. 'make install' _will_ be broken instantly unless _humans_ incorporate it in their patch development process and maintainer incorporate that step into their workflow as well.
I don't think so. I think if you want to work with us on this, we can find a way. bpf isn't such a unique test that adding install will break it.
Ohh, but don't worry about this broken 'make install' is not an answer. It's broken now and I don't want to see patches that move it from one broken state into another broken state and at the same time add complexity to it.
Well! What you are saying "I don;t want to collaborate to find a solution to the problem".
That's very different from 'make install' doing 'cp -r' of the whole dir. In such case the chances of it going stale and broken are much lower.
- be able to build and run existing tests without breaking the test
build when new tests are added that have hard dependency on new versions of tools (llvm etc.). This isn't such a novel idea. We don't break kernel builds every single release and even when we require newer compiler releases. Plan the new tests with the intent to not break existing users and add new tests at the same time. We use min rev and not bleeding edge as the requirement for kernel build.
'existing users'?
I said existing tests not users. When you add new bpf tests, existing tests should continue to build and run without dependency on new revs of llvm.
Who said that existing test stop building with new llvm ? Please check your facts.
I am basing this on a previous discussion on this topic. bpf test(s) that build stop building and the solution always has been upgrade to new tool chain. If this changed now and there is no hard tie to bpf test building and llvm release, great.
If bpf test doesn't build and/or installed, it won't run on test rings that qualify stable/next/main releases.
That's the case today and I prefer to keep it this way.
Why is that? Are you making a decision for the rest of the kernel with this approach? If bpf doesn't get tested in stable test rings, this isn't bpf problem alone. This is a kernel release issue.
stable folks ignored our recommendations on how selftests/bpf should be run, so please don't come up with your ways of doing it and complain that something doesn't work.
If you want to talk about history, bpf test was added in Oct 2016 and by then kselftest was in use with its run_tests and install support. So it is misleading to suggest that the framework came after the bpf test. bpf test was added to existing kselftest framework used by all other tests. Expecting the existing framework to change and adapt to a new test isn't reasonable.
When a new test is added, it has to work within the framework. Framework can be improved and changed, however not the way you are attempting to do in this thread. You can do this like others in the community to do by making changes and improvements.
In any case, kselftest framework lets you override INSTALL_RULE and define your own. Same is true for other rules: RUN_TESTS, EMIT_TESTS, CLEAN, INSTALL
bpf does this already for CLEAN.
# override lib.mk's default rules OVERRIDE_TARGETS := 1 override define CLEAN
If generic install rule doesn't work for you do override so bpf so users can install it.
thanks, -- Shuah
linux-kselftest-mirror@lists.linaro.org
-
Alexei Starovoitov -
Greg KH -
Jiri Benc -
shuah -
Shuah Khan -
Yauheni Kaliuta