I did a bisect which implicated f2bd1c5ae2cb0cf9525c9bffc0038c12dd7e1338 as the first bad commit.
Reverting this commit on 6.1.8 gives me working sound again.
I'm not clear why this is breaking 6.1.x since it appears to be in 6.0.18 (7494e2e6c55e), which was the last working package in Fedora for the 6.0 line. Maybe something else didn't make it into 6.1?
On Sat, Jan 28, 2023 at 12:09:54PM -0500, Jason Montleon wrote:
I did a bisect which implicated f2bd1c5ae2cb0cf9525c9bffc0038c12dd7e1338 as the first bad commit.
Reverting this commit on 6.1.8 gives me working sound again.
I'm not clear why this is breaking 6.1.x since it appears to be in 6.0.18 (7494e2e6c55e), which was the last working package in Fedora for the 6.0 line. Maybe something else didn't make it into 6.1?
So this is also broken in Linus's tree (6.2-rc5?)
thanks,
greg k-h
I have confirmed 6.2-rc5 is also broken and removing the same commit causes it to work again.
Thank you, Jason Montleon
On Sat, Jan 28, 2023 at 12:52 PM Greg KH gregkh@linuxfoundation.org wrote:
On Sat, Jan 28, 2023 at 12:09:54PM -0500, Jason Montleon wrote:
I did a bisect which implicated f2bd1c5ae2cb0cf9525c9bffc0038c12dd7e1338 as the first bad commit.
Reverting this commit on 6.1.8 gives me working sound again.
I'm not clear why this is breaking 6.1.x since it appears to be in 6.0.18 (7494e2e6c55e), which was the last working package in Fedora for the 6.0 line. Maybe something else didn't make it into 6.1?
So this is also broken in Linus's tree (6.2-rc5?)
thanks,
greg k-h
I ran a bisect back further while patching in f2bd1c5ae2cb0cf9525c9bffc0038c12dd7e1338.
When doing this, 3fd63658caed9494cca1d4789a66d3d2def2a0ab appears to be the commit in 6.1+ that it's interacting badly with. I couldn't revert this cleanly without reverting a handful of other commits, but doing so also results in working audio with 6.1.8 while leaving in f2bd1c5ae2cb. (e9441675edc1, 1cda83e42bf6, 37882100cd06, 0e213813df02, fb5987844808 were the others removed)
Hopefully that helps narrow it down for someone more familiar, otherwise I'll keep digging.
On Sat, Jan 28, 2023 at 7:23 PM Jason Montleon jmontleo@redhat.com wrote:
I have confirmed 6.2-rc5 is also broken and removing the same commit causes it to work again.
Thank you, Jason Montleon
On Sat, Jan 28, 2023 at 12:52 PM Greg KH gregkh@linuxfoundation.org wrote:
On Sat, Jan 28, 2023 at 12:09:54PM -0500, Jason Montleon wrote:
I did a bisect which implicated f2bd1c5ae2cb0cf9525c9bffc0038c12dd7e1338 as the first bad commit.
Reverting this commit on 6.1.8 gives me working sound again.
I'm not clear why this is breaking 6.1.x since it appears to be in 6.0.18 (7494e2e6c55e), which was the last working package in Fedora for the 6.0 line. Maybe something else didn't make it into 6.1?
So this is also broken in Linus's tree (6.2-rc5?)
thanks,
greg k-h
-- Jason Montleon | email: jmontleo@redhat.com Red Hat, Inc. | gpg key: 0x069E3022 Cell: 508-496-0663 | irc: jmontleo / jmontleon
-- Jason Montleon | email: jmontleo@redhat.com Red Hat, Inc. | gpg key: 0x069E3022 Cell: 508-496-0663 | irc: jmontleo / jmontleon
On 30.01.23 07:33, Jason Montleon wrote:
I ran a bisect back further while patching in f2bd1c5ae2cb0cf9525c9bffc0038c12dd7e1338.
Cezary Rojewski, you authored this change which appears to cause a regression. Do you maybe have an idea what's wrong here?
Also CCing Takashi, who merged that changes.
FWIW, this thread starts here:
https://lore.kernel.org/regressions/CALFERdzKUodLsm6=Ub3g2+PxpNpPtPq3bGBLbff...
Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat) -- Everything you wanna know about Linux kernel regression tracking: https://linux-regtracking.leemhuis.info/about/#tldr If I did something stupid, please tell me, as explained on that page.
When doing this, 3fd63658caed9494cca1d4789a66d3d2def2a0ab appears to be the commit in 6.1+ that it's interacting badly with. I couldn't revert this cleanly without reverting a handful of other commits, but doing so also results in working audio with 6.1.8 while leaving in f2bd1c5ae2cb. (e9441675edc1, 1cda83e42bf6, 37882100cd06, 0e213813df02, fb5987844808 were the others removed)
Hopefully that helps narrow it down for someone more familiar, otherwise I'll keep digging.
On Sat, Jan 28, 2023 at 7:23 PM Jason Montleon jmontleo@redhat.com wrote:
I have confirmed 6.2-rc5 is also broken and removing the same commit causes it to work again.
Thank you, Jason Montleon
On Sat, Jan 28, 2023 at 12:52 PM Greg KH gregkh@linuxfoundation.org wrote:
On Sat, Jan 28, 2023 at 12:09:54PM -0500, Jason Montleon wrote:
I did a bisect which implicated f2bd1c5ae2cb0cf9525c9bffc0038c12dd7e1338 as the first bad commit.
Reverting this commit on 6.1.8 gives me working sound again.
I'm not clear why this is breaking 6.1.x since it appears to be in 6.0.18 (7494e2e6c55e), which was the last working package in Fedora for the 6.0 line. Maybe something else didn't make it into 6.1?
So this is also broken in Linus's tree (6.2-rc5?)
thanks,
greg k-h
-- Jason Montleon | email: jmontleo@redhat.com Red Hat, Inc. | gpg key: 0x069E3022 Cell: 508-496-0663 | irc: jmontleo / jmontleon
-- Jason Montleon | email: jmontleo@redhat.com Red Hat, Inc. | gpg key: 0x069E3022 Cell: 508-496-0663 | irc: jmontleo / jmontleon
On 2023-01-30 12:27 PM, Linux kernel regression tracking (Thorsten Leemhuis) wrote:
On 30.01.23 07:33, Jason Montleon wrote:
I ran a bisect back further while patching in f2bd1c5ae2cb0cf9525c9bffc0038c12dd7e1338.
Cezary Rojewski, you authored this change which appears to cause a regression. Do you maybe have an idea what's wrong here?
Hello,
Sitting with Lukasz and debugging this.
Please note that messages seen in the logs: [ 8.538645] snd_soc_skl 0000:00:1f.3: ASoC: no sink widget found for AEC Capture [ 8.538654] snd_soc_skl 0000:00:1f.3: ASoC: Failed to add route echo_ref_out cpr 7 -> direct -> AEC Capture
and: [ 8.614993] rt5663 i2c-10EC5663:00: sysclk < 384 x fs, disable i2s asrc [ 8.617460] HDMI HDA Codec ehdaudio0D2: hdac_hdmi_present_sense: disconnect for pin:port 5:0 [ 8.617581] HDMI HDA Codec ehdaudio0D2: hdac_hdmi_present_sense: disconnect for pin:port 6:0 [ 8.617712] HDMI HDA Codec ehdaudio0D2: hdac_hdmi_present_sense: disconnect for pin:port 7:0
seem to be just noise, these are present even in working configuration. Looks like topology and AudioDSP firmware loads fine. However, one or more of sound cards component is deferred when probing what results in no error in the logs but still lack of audio.
We managed to reproduce the problem on Lukasz' Eve machine. If hardware cooperates, we should be able to narrow down the problem quickly from here. Our suspect - something (a patch or two) is missing in sound/soc in Fedora tree. Our CI which also covers upstream skylake-driver with HDAudio (e.g.: rt274/rt286/rt298) and I2S (same examples) configurations show no signs of regression. Thus, this is probably Chromebook-specific.
My opinion stands at - none of the patches mentioned during the discussion is the real problem. All of these were in fact addressing real problems with audio stack stability/reloading.
Will reply once more is known.
Kind regards, Czarek
On Mon, Jan 30, 2023 at 1:15 PM Cezary Rojewski cezary.rojewski@intel.com wrote:
On 2023-01-30 12:27 PM, Linux kernel regression tracking (Thorsten Leemhuis) wrote:
On 30.01.23 07:33, Jason Montleon wrote:
I ran a bisect back further while patching in f2bd1c5ae2cb0cf9525c9bffc0038c12dd7e1338.
Cezary Rojewski, you authored this change which appears to cause a regression. Do you maybe have an idea what's wrong here?
Hello,
Sitting with Lukasz and debugging this.
Please note that messages seen in the logs: [ 8.538645] snd_soc_skl 0000:00:1f.3: ASoC: no sink widget found for AEC Capture [ 8.538654] snd_soc_skl 0000:00:1f.3: ASoC: Failed to add route echo_ref_out cpr 7 -> direct -> AEC Capture
and: [ 8.614993] rt5663 i2c-10EC5663:00: sysclk < 384 x fs, disable i2s asrc [ 8.617460] HDMI HDA Codec ehdaudio0D2: hdac_hdmi_present_sense: disconnect for pin:port 5:0 [ 8.617581] HDMI HDA Codec ehdaudio0D2: hdac_hdmi_present_sense: disconnect for pin:port 6:0 [ 8.617712] HDMI HDA Codec ehdaudio0D2: hdac_hdmi_present_sense: disconnect for pin:port 7:0
seem to be just noise, these are present even in working configuration. Looks like topology and AudioDSP firmware loads fine. However, one or more of sound cards component is deferred when probing what results in no error in the logs but still lack of audio.
We managed to reproduce the problem on Lukasz' Eve machine. If hardware cooperates, we should be able to narrow down the problem quickly from here. Our suspect - something (a patch or two) is missing in sound/soc in Fedora tree. Our CI which also covers upstream skylake-driver with HDAudio (e.g.: rt274/rt286/rt298) and I2S (same examples) configurations show no signs of regression. Thus, this is probably Chromebook-specific.
My opinion stands at - none of the patches mentioned during the discussion is the real problem. All of these were in fact addressing real problems with audio stack stability/reloading.
Will reply once more is known.
Kind regards, Czarek
Dear Czarek, many thanks for the answer and taking care of it. If needed something from my side please jest let me know and I will try to do it.
Rgds Sasa
On 2023-01-30 1:22 PM, Sasa Ostrouska wrote:
Dear Czarek, many thanks for the answer and taking care of it. If needed something from my side please jest let me know and I will try to do it.
Hello Sasa,
Could you provide us with the topology and firmware binary present on your machine?
Audio topology is located at /lib/firmware and named:
9d71-GOOGLE-EVEMAX-0-tplg.bin -or- dfw_sst.bin
Firmware on the other hand is found in /lib/firmware/intel/. 'dsp_fw_kbl.bin' will lie there, it shall be a symlink pointing to an actual AudioDSP firmware binary.
The reasoning for these asks is fact that problem stopped reproducing on our end once we started playing with kernel versions (moved away from status quo with Fedora). Neither on Lukasz EVE nor on my SKL RVP. However, we might be using newer configuration files when compared to equivalent of yours.
Recent v6.2-rc5 broonie/sound/for-next - no repro Our internal tree based on Mark's for-next - no repro 6.1.7 stable [1] - no repro
Of course we will continue with our attempts. Will notify about the progress.
[1]: https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?h=v...
Kind regards, Czarek
On Tue, Jan 31, 2023 at 1:37 PM Cezary Rojewski cezary.rojewski@intel.com wrote:
On 2023-01-30 1:22 PM, Sasa Ostrouska wrote:
Dear Czarek, many thanks for the answer and taking care of it. If needed something from my side please jest let me know and I will try to do it.
Hello Sasa,
Hi Czarek,
Could you provide us with the topology and firmware binary present on your machine?
Sure, see below.
Audio topology is located at /lib/firmware and named:
9d71-GOOGLE-EVEMAX-0-tplg.bin
-rw-r--r--. 1 root root 37864 21 nov 20.18 9d71-GOOGLE-EVEMAX-0-tplg.bin
-or- dfw_sst.bin
Firmware on the other hand is found in /lib/firmware/intel/. 'dsp_fw_kbl.bin' will lie there, it shall be a symlink pointing to an actual AudioDSP firmware binary.
lrwxrwxrwx. 1 root root 23 23 gen 10.28 dsp_fw_kbl.bin.xz -> dsp_fw_kbl_v3402.bin.xz
The reasoning for these asks is fact that problem stopped reproducing on our end once we started playing with kernel versions (moved away from status quo with Fedora). Neither on Lukasz EVE nor on my SKL RVP. However, we might be using newer configuration files when compared to equivalent of yours.
Recent v6.2-rc5 broonie/sound/for-next - no repro Our internal tree based on Mark's for-next - no repro 6.1.7 stable [1] - no repro
Of course we will continue with our attempts. Will notify about the progress.
Ok, waiting for your good news, on the other hand if you need anything else feel free to ask.
Kind regards, Czarek
Rgds Sasa
On 2023-01-31 3:58 PM, Sasa Ostrouska wrote:
On Tue, Jan 31, 2023 at 1:37 PM Cezary Rojewski cezary.rojewski@intel.com wrote:
...
Hi Czarek,
Could you provide us with the topology and firmware binary present on your machine?
Sure, see below.
Audio topology is located at /lib/firmware and named:
9d71-GOOGLE-EVEMAX-0-tplg.bin
-rw-r--r--. 1 root root 37864 21 nov 20.18 9d71-GOOGLE-EVEMAX-0-tplg.bin
-or- dfw_sst.bin
Firmware on the other hand is found in /lib/firmware/intel/. 'dsp_fw_kbl.bin' will lie there, it shall be a symlink pointing to an actual AudioDSP firmware binary.
lrwxrwxrwx. 1 root root 23 23 gen 10.28 dsp_fw_kbl.bin.xz -> dsp_fw_kbl_v3402.bin.xz
I was hoping to receive the actual files. Names alone won't suffice. A bit of reverse engineering and I'll know exactly what topology I'm looking at.
On Tue, Jan 31, 2023 at 4:49 PM Cezary Rojewski cezary.rojewski@intel.com wrote:
On 2023-01-31 3:58 PM, Sasa Ostrouska wrote:
On Tue, Jan 31, 2023 at 1:37 PM Cezary Rojewski cezary.rojewski@intel.com wrote:
...
Hi Czarek,
Could you provide us with the topology and firmware binary present on your machine?
Sure, see below.
Audio topology is located at /lib/firmware and named:
9d71-GOOGLE-EVEMAX-0-tplg.bin
-rw-r--r--. 1 root root 37864 21 nov 20.18 9d71-GOOGLE-EVEMAX-0-tplg.bin
-or- dfw_sst.bin
Firmware on the other hand is found in /lib/firmware/intel/. 'dsp_fw_kbl.bin' will lie there, it shall be a symlink pointing to an actual AudioDSP firmware binary.
lrwxrwxrwx. 1 root root 23 23 gen 10.28 dsp_fw_kbl.bin.xz -> dsp_fw_kbl_v3402.bin.xz
I was hoping to receive the actual files. Names alone won't suffice. A bit of reverse engineering and I'll know exactly what topology I'm looking at.
Oh sorry, I didnt understood it. Here they are attached, But thosse are the ones I extracted from chromos[0] images as pointed out on Jasons github page [1]
I see I have plenty of them in /lib/firmware/ so I am not sure which ones you want. Here is the list of the ones I have:
lrwxrwxrwx. 1 root root 23 23 gen 10.28 /lib/firmware/intel/dsp_fw_kbl.bin.xz -> dsp_fw_kbl_v3402.bin.xz -rw-r--r--. 1 root root 131932 23 gen 10.28 /lib/firmware/intel/dsp_fw_kbl_v1037.bin.xz -rw-r--r--. 1 root root 132308 23 gen 10.28 /lib/firmware/intel/dsp_fw_kbl_v2042.bin.xz -rw-r--r--. 1 root root 133888 23 gen 10.28 /lib/firmware/intel/dsp_fw_kbl_v2630.bin.xz -rw-r--r--. 1 root root 133832 23 gen 10.28 /lib/firmware/intel/dsp_fw_kbl_v3266.bin.xz -rw-r--r--. 1 root root 134512 23 gen 10.28 /lib/firmware/intel/dsp_fw_kbl_v3402.bin.xz -rw-r--r--. 1 root root 134060 23 gen 10.28 /lib/firmware/intel/dsp_fw_kbl_v3420.bin.xz -rw-r--r--. 1 root root 131020 23 gen 10.28 /lib/firmware/intel/dsp_fw_kbl_v701.bin.xz
If you need all of them let me know.
[0] https://dl.google.com/dl/edgedl/chromeos/recovery/chromeos_15117.112.0_eve_r... [1] https://github.com/jmontleon/pixelbook-fedora#audio
On Tue, Jan 31, 2023 at 7:37 AM Cezary Rojewski cezary.rojewski@intel.com wrote:
On 2023-01-30 1:22 PM, Sasa Ostrouska wrote:
Dear Czarek, many thanks for the answer and taking care of it. If needed something from my side please jest let me know and I will try to do it.
Hello Sasa,
Could you provide us with the topology and firmware binary present on your machine?
Audio topology is located at /lib/firmware and named:
9d71-GOOGLE-EVEMAX-0-tplg.bin -or- dfw_sst.bin
Firmware on the other hand is found in /lib/firmware/intel/. 'dsp_fw_kbl.bin' will lie there, it shall be a symlink pointing to an actual AudioDSP firmware binary.
Maybe this is the problem.
I think most of us are pulling the topology and firmware from the chromeos recovery images for lack of any other known source, and it looks a little different than this. Those can be downloaded like so: https://gist.github.com/jmontleon/8899cb83138f2653f520fbbcc5b830a0
After placing the topology file you'll see these errors and audio will not work until they're also copied in place. snd_soc_skl 0000:00:1f.3: Direct firmware load for dsp_lib_dsm_core_spt_release.bin failed with error -2 snd_soc_skl 0000:00:1f.3: Direct firmware load for intel/dsp_fw_C75061F3-F2B2-4DCC-8F9F-82ABB4131E66.bin failed with error -2
Once those were in place, up to 6.0.18 audio worked.
Is there a better source for the topology file?
The reasoning for these asks is fact that problem stopped reproducing on our end once we started playing with kernel versions (moved away from status quo with Fedora). Neither on Lukasz EVE nor on my SKL RVP. However, we might be using newer configuration files when compared to equivalent of yours.
Recent v6.2-rc5 broonie/sound/for-next - no repro Our internal tree based on Mark's for-next - no repro 6.1.7 stable [1] - no repro
Of course we will continue with our attempts. Will notify about the progress.
Kind regards, Czarek
On 2023-01-31 4:16 PM, Jason Montleon wrote:
Maybe this is the problem.
I think most of us are pulling the topology and firmware from the chromeos recovery images for lack of any other known source, and it looks a little different than this. Those can be downloaded like so: https://gist.github.com/jmontleon/8899cb83138f2653f520fbbcc5b830a0
After placing the topology file you'll see these errors and audio will not work until they're also copied in place. snd_soc_skl 0000:00:1f.3: Direct firmware load for dsp_lib_dsm_core_spt_release.bin failed with error -2 snd_soc_skl 0000:00:1f.3: Direct firmware load for intel/dsp_fw_C75061F3-F2B2-4DCC-8F9F-82ABB4131E66.bin failed with error -2
Once those were in place, up to 6.0.18 audio worked.
Is there a better source for the topology file?
There was no other source for the topology files for a very long time. This has been quite recently changed with the introduction of avsdk [1] and avs-topology-xml [2] repos. These hold 2 major branches each:
1) main - for the avs-driver which has already replaced the skylake-driver in client department and shall soon do the same in the remaining areas 2) for-skylake-driver - for the skylake-driver, help downstream in dealing with bugs and ease the transition to the avs-driver
Updated AudioDSP firmware binaries should be shared soon, completing last few remaining steps. Not every library can be shared though. And can't comment on usage of 3rd party loadable modules outside of supported set of configurations.
[1]: https://github.com/thesofproject/avsdk [2]: https://github.com/thesofproject/avs-topology-xml
Kind regards, Czarek
On 1/31/2023 4:16 PM, Jason Montleon wrote:
On Tue, Jan 31, 2023 at 7:37 AM Cezary Rojewski cezary.rojewski@intel.com wrote:
On 2023-01-30 1:22 PM, Sasa Ostrouska wrote:
Dear Czarek, many thanks for the answer and taking care of it. If needed something from my side please jest let me know and I will try to do it.
Hello Sasa,
Could you provide us with the topology and firmware binary present on your machine?
Audio topology is located at /lib/firmware and named:
9d71-GOOGLE-EVEMAX-0-tplg.bin -or- dfw_sst.bin
Firmware on the other hand is found in /lib/firmware/intel/. 'dsp_fw_kbl.bin' will lie there, it shall be a symlink pointing to an actual AudioDSP firmware binary.
Maybe this is the problem.
I think most of us are pulling the topology and firmware from the chromeos recovery images for lack of any other known source, and it looks a little different than this. Those can be downloaded like so: https://gist.github.com/jmontleon/8899cb83138f2653f520fbbcc5b830a0
After placing the topology file you'll see these errors and audio will not work until they're also copied in place. snd_soc_skl 0000:00:1f.3: Direct firmware load for dsp_lib_dsm_core_spt_release.bin failed with error -2 snd_soc_skl 0000:00:1f.3: Direct firmware load for intel/dsp_fw_C75061F3-F2B2-4DCC-8F9F-82ABB4131E66.bin failed with error -2
Once those were in place, up to 6.0.18 audio worked.
Is there a better source for the topology file?
The reasoning for these asks is fact that problem stopped reproducing on our end once we started playing with kernel versions (moved away from status quo with Fedora). Neither on Lukasz EVE nor on my SKL RVP. However, we might be using newer configuration files when compared to equivalent of yours.
Recent v6.2-rc5 broonie/sound/for-next - no repro Our internal tree based on Mark's for-next - no repro 6.1.7 stable [1] - no repro
Of course we will continue with our attempts. Will notify about the progress.
Kind regards, Czarek
Hi Jason,
as I understand you've tried to do bisect, can you instead try building kernels checking out following tags: v6.1 v6.1.1 v6.1.2 v6.1.3 v6.1.4 v6.1.5 v6.1.6 v6.1.7 v6.1.8 and report when it stops working, so it narrows scope of what we look at? I assume that kernel builds are done using upstream stable kernel (from https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/).
Thanks, Amadeusz
On Wed, Feb 1, 2023 at 6:05 AM Amadeusz Sławiński amadeuszx.slawinski@linux.intel.com wrote:
On 1/31/2023 4:16 PM, Jason Montleon wrote:
On Tue, Jan 31, 2023 at 7:37 AM Cezary Rojewski cezary.rojewski@intel.com wrote:
On 2023-01-30 1:22 PM, Sasa Ostrouska wrote:
Dear Czarek, many thanks for the answer and taking care of it. If needed something from my side please jest let me know and I will try to do it.
Hello Sasa,
Could you provide us with the topology and firmware binary present on your machine?
Audio topology is located at /lib/firmware and named:
9d71-GOOGLE-EVEMAX-0-tplg.bin -or- dfw_sst.bin
Firmware on the other hand is found in /lib/firmware/intel/. 'dsp_fw_kbl.bin' will lie there, it shall be a symlink pointing to an actual AudioDSP firmware binary.
Maybe this is the problem.
I think most of us are pulling the topology and firmware from the chromeos recovery images for lack of any other known source, and it looks a little different than this. Those can be downloaded like so: https://gist.github.com/jmontleon/8899cb83138f2653f520fbbcc5b830a0
After placing the topology file you'll see these errors and audio will not work until they're also copied in place. snd_soc_skl 0000:00:1f.3: Direct firmware load for dsp_lib_dsm_core_spt_release.bin failed with error -2 snd_soc_skl 0000:00:1f.3: Direct firmware load for intel/dsp_fw_C75061F3-F2B2-4DCC-8F9F-82ABB4131E66.bin failed with error -2
Once those were in place, up to 6.0.18 audio worked.
Is there a better source for the topology file?
The reasoning for these asks is fact that problem stopped reproducing on our end once we started playing with kernel versions (moved away from status quo with Fedora). Neither on Lukasz EVE nor on my SKL RVP. However, we might be using newer configuration files when compared to equivalent of yours.
Recent v6.2-rc5 broonie/sound/for-next - no repro Our internal tree based on Mark's for-next - no repro 6.1.7 stable [1] - no repro
Of course we will continue with our attempts. Will notify about the progress.
Kind regards, Czarek
Hi Jason,
as I understand you've tried to do bisect, can you instead try building kernels checking out following tags: v6.1 v6.1.1 v6.1.2 v6.1.3 v6.1.4 v6.1.5 v6.1.6 v6.1.7 v6.1.8 and report when it stops working, so it narrows scope of what we look at? I assume that kernel builds are done using upstream stable kernel (from https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/).
Thanks, Amadeusz
Hi Amadeusz, Yes, I did the bisects using https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/
The only thing I did to these was add 392cc13c5ec72ccd6bbfb1bc2339502cc59dd285, otherwise audio breaks with the dai not registered error message in dmesg from the rt5514 bug from 6.0 and up. It wasn't added to 6.1 until rc6, I believe. If there's a better way to work around the multiple bugs I can try again, otherwise I will start working on builds from tags and see if I learn anything.
FWIW, I've seen two people complain that Arch isn't working either since it moved to 6.1. For the one who was trying, patching out the commit I came to with the first bisect did not regain them sound like it did for me. And yet Sasa reports Slackware is mostly working for him with 6.1.8 on Slackware. I don't know what to make of it, but thought I'd share in case it helps point someone else to something. https://github.com/jmontleon/pixelbook-fedora/issues/51#issuecomment-1410222... https://github.com/jmontleon/pixelbook-fedora/issues/51#issuecomment-1410673... https://github.com/jmontleon/pixelbook-fedora/issues/53#issuecomment-1408699...
Probably less relevant since they aren't from upstream and I know they don't mean as much, but I have tried 6.1.5-6.1.8 Fedora packages for certain, and went back trying several others from koji back into rc builds, although using prebuilt kernels, anything before 6.1-rc6 won't work, as mentioned above. Nothing worked. But as I said I'll build from tags and see if I can learn anything.
Thank you, Jason Montleon
I have built kernels for 6.0.19 (I don't think anyone confirmed whether or not it worked), plus every 6.1 tag from 6.1-rc1 up to 6.1.7. 6.0.19 worked. No 6.1 kernels worked. For rc1 to rc5 I built with and without the legacy dai renaming patch added in rc6 that I believe would be necessary, but it made no difference either way.
On Wed, Feb 1, 2023 at 9:33 AM Jason Montleon jmontleo@redhat.com wrote:
On Wed, Feb 1, 2023 at 6:05 AM Amadeusz Sławiński amadeuszx.slawinski@linux.intel.com wrote:
On 1/31/2023 4:16 PM, Jason Montleon wrote:
On Tue, Jan 31, 2023 at 7:37 AM Cezary Rojewski cezary.rojewski@intel.com wrote:
On 2023-01-30 1:22 PM, Sasa Ostrouska wrote:
Dear Czarek, many thanks for the answer and taking care of it. If needed something from my side please jest let me know and I will try to do it.
Hello Sasa,
Could you provide us with the topology and firmware binary present on your machine?
Audio topology is located at /lib/firmware and named:
9d71-GOOGLE-EVEMAX-0-tplg.bin -or- dfw_sst.bin
Firmware on the other hand is found in /lib/firmware/intel/. 'dsp_fw_kbl.bin' will lie there, it shall be a symlink pointing to an actual AudioDSP firmware binary.
Maybe this is the problem.
I think most of us are pulling the topology and firmware from the chromeos recovery images for lack of any other known source, and it looks a little different than this. Those can be downloaded like so: https://gist.github.com/jmontleon/8899cb83138f2653f520fbbcc5b830a0
After placing the topology file you'll see these errors and audio will not work until they're also copied in place. snd_soc_skl 0000:00:1f.3: Direct firmware load for dsp_lib_dsm_core_spt_release.bin failed with error -2 snd_soc_skl 0000:00:1f.3: Direct firmware load for intel/dsp_fw_C75061F3-F2B2-4DCC-8F9F-82ABB4131E66.bin failed with error -2
Once those were in place, up to 6.0.18 audio worked.
Is there a better source for the topology file?
The reasoning for these asks is fact that problem stopped reproducing on our end once we started playing with kernel versions (moved away from status quo with Fedora). Neither on Lukasz EVE nor on my SKL RVP. However, we might be using newer configuration files when compared to equivalent of yours.
Recent v6.2-rc5 broonie/sound/for-next - no repro Our internal tree based on Mark's for-next - no repro 6.1.7 stable [1] - no repro
Of course we will continue with our attempts. Will notify about the progress.
Kind regards, Czarek
Hi Jason,
as I understand you've tried to do bisect, can you instead try building kernels checking out following tags: v6.1 v6.1.1 v6.1.2 v6.1.3 v6.1.4 v6.1.5 v6.1.6 v6.1.7 v6.1.8 and report when it stops working, so it narrows scope of what we look at? I assume that kernel builds are done using upstream stable kernel (from https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/).
Thanks, Amadeusz
Hi Amadeusz, Yes, I did the bisects using https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/
The only thing I did to these was add 392cc13c5ec72ccd6bbfb1bc2339502cc59dd285, otherwise audio breaks with the dai not registered error message in dmesg from the rt5514 bug from 6.0 and up. It wasn't added to 6.1 until rc6, I believe. If there's a better way to work around the multiple bugs I can try again, otherwise I will start working on builds from tags and see if I learn anything.
FWIW, I've seen two people complain that Arch isn't working either since it moved to 6.1. For the one who was trying, patching out the commit I came to with the first bisect did not regain them sound like it did for me. And yet Sasa reports Slackware is mostly working for him with 6.1.8 on Slackware. I don't know what to make of it, but thought I'd share in case it helps point someone else to something. https://github.com/jmontleon/pixelbook-fedora/issues/51#issuecomment-1410222... https://github.com/jmontleon/pixelbook-fedora/issues/51#issuecomment-1410673... https://github.com/jmontleon/pixelbook-fedora/issues/53#issuecomment-1408699...
Probably less relevant since they aren't from upstream and I know they don't mean as much, but I have tried 6.1.5-6.1.8 Fedora packages for certain, and went back trying several others from koji back into rc builds, although using prebuilt kernels, anything before 6.1-rc6 won't work, as mentioned above. Nothing worked. But as I said I'll build from tags and see if I can learn anything.
Thank you, Jason Montleon
-- Jason Montleon | email: jmontleo@redhat.com Red Hat, Inc. | gpg key: 0x069E3022 Cell: 508-496-0663 | irc: jmontleo / jmontleon
On 2/4/2023 4:16 PM, Jason Montleon wrote:
I have built kernels for 6.0.19 (I don't think anyone confirmed whether or not it worked), plus every 6.1 tag from 6.1-rc1 up to 6.1.7. 6.0.19 worked. No 6.1 kernels worked. For rc1 to rc5 I built with and without the legacy dai renaming patch added in rc6 that I believe would be necessary, but it made no difference either way.
Hi,
thank you for trying to narrow it down, if I understand correctly -rc1 doesn't work, which means that problem was introduced somewhere between 6.0 and 6.1-rc1 (just for the sake of being sure, can you test 6.0 instead of 6.0.19?) There is one commit which I'm bit suspicious about: ef6f5494faf6a37c74990689a3bb3cee76d2544c it changes how HDMI are assigned and as a machine board present on EVE makes use of HDMI, it may potentially cause some problems. Can you try reverting it? (If reverting on top of v6.1.8 you need to revert both f9aafff5448b1d8d457052271cd9a11b24e4d0bd and ef6f5494faf6a37c74990689a3bb3cee76d2544c which has minor conflict, easily resolved with just adding both lines.
I also still wonder, why problem reproduces only on some distributions... any chance you can try and boot with pipewire/pulseaudio disabled and see if it still happens, iirc those tools try to check all FEs and this may be breaking something during enumeration.
Thanks, Amadeusz
On Wed, Feb 1, 2023 at 9:33 AM Jason Montleon jmontleo@redhat.com wrote:
On Wed, Feb 1, 2023 at 6:05 AM Amadeusz Sławiński amadeuszx.slawinski@linux.intel.com wrote:
On 1/31/2023 4:16 PM, Jason Montleon wrote:
On Tue, Jan 31, 2023 at 7:37 AM Cezary Rojewski cezary.rojewski@intel.com wrote:
On 2023-01-30 1:22 PM, Sasa Ostrouska wrote:
Dear Czarek, many thanks for the answer and taking care of it. If needed something from my side please jest let me know and I will try to do it.
Hello Sasa,
Could you provide us with the topology and firmware binary present on your machine?
Audio topology is located at /lib/firmware and named:
9d71-GOOGLE-EVEMAX-0-tplg.bin -or- dfw_sst.bin
Firmware on the other hand is found in /lib/firmware/intel/. 'dsp_fw_kbl.bin' will lie there, it shall be a symlink pointing to an actual AudioDSP firmware binary.
Maybe this is the problem.
I think most of us are pulling the topology and firmware from the chromeos recovery images for lack of any other known source, and it looks a little different than this. Those can be downloaded like so: https://gist.github.com/jmontleon/8899cb83138f2653f520fbbcc5b830a0
After placing the topology file you'll see these errors and audio will not work until they're also copied in place. snd_soc_skl 0000:00:1f.3: Direct firmware load for dsp_lib_dsm_core_spt_release.bin failed with error -2 snd_soc_skl 0000:00:1f.3: Direct firmware load for intel/dsp_fw_C75061F3-F2B2-4DCC-8F9F-82ABB4131E66.bin failed with error -2
Once those were in place, up to 6.0.18 audio worked.
Is there a better source for the topology file?
The reasoning for these asks is fact that problem stopped reproducing on our end once we started playing with kernel versions (moved away from status quo with Fedora). Neither on Lukasz EVE nor on my SKL RVP. However, we might be using newer configuration files when compared to equivalent of yours.
Recent v6.2-rc5 broonie/sound/for-next - no repro Our internal tree based on Mark's for-next - no repro 6.1.7 stable [1] - no repro
Of course we will continue with our attempts. Will notify about the progress.
Kind regards, Czarek
Hi Jason,
as I understand you've tried to do bisect, can you instead try building kernels checking out following tags: v6.1 v6.1.1 v6.1.2 v6.1.3 v6.1.4 v6.1.5 v6.1.6 v6.1.7 v6.1.8 and report when it stops working, so it narrows scope of what we look at? I assume that kernel builds are done using upstream stable kernel (from https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/).
Thanks, Amadeusz
Hi Amadeusz, Yes, I did the bisects using https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/
The only thing I did to these was add 392cc13c5ec72ccd6bbfb1bc2339502cc59dd285, otherwise audio breaks with the dai not registered error message in dmesg from the rt5514 bug from 6.0 and up. It wasn't added to 6.1 until rc6, I believe. If there's a better way to work around the multiple bugs I can try again, otherwise I will start working on builds from tags and see if I learn anything.
FWIW, I've seen two people complain that Arch isn't working either since it moved to 6.1. For the one who was trying, patching out the commit I came to with the first bisect did not regain them sound like it did for me. And yet Sasa reports Slackware is mostly working for him with 6.1.8 on Slackware. I don't know what to make of it, but thought I'd share in case it helps point someone else to something. https://github.com/jmontleon/pixelbook-fedora/issues/51#issuecomment-1410222... https://github.com/jmontleon/pixelbook-fedora/issues/51#issuecomment-1410673... https://github.com/jmontleon/pixelbook-fedora/issues/53#issuecomment-1408699...
Probably less relevant since they aren't from upstream and I know they don't mean as much, but I have tried 6.1.5-6.1.8 Fedora packages for certain, and went back trying several others from koji back into rc builds, although using prebuilt kernels, anything before 6.1-rc6 won't work, as mentioned above. Nothing worked. But as I said I'll build from tags and see if I can learn anything.
Thank you, Jason Montleon
-- Jason Montleon | email: jmontleo@redhat.com Red Hat, Inc. | gpg key: 0x069E3022 Cell: 508-496-0663 | irc: jmontleo / jmontleon
On Mon, Feb 6, 2023 at 4:04 AM Amadeusz Sławiński amadeuszx.slawinski@linux.intel.com wrote:
On 2/4/2023 4:16 PM, Jason Montleon wrote:
I have built kernels for 6.0.19 (I don't think anyone confirmed whether or not it worked), plus every 6.1 tag from 6.1-rc1 up to 6.1.7. 6.0.19 worked. No 6.1 kernels worked. For rc1 to rc5 I built with and without the legacy dai renaming patch added in rc6 that I believe would be necessary, but it made no difference either way.
Hi,
thank you for trying to narrow it down, if I understand correctly -rc1 doesn't work, which means that problem was introduced somewhere between 6.0 and 6.1-rc1 (just for the sake of being sure, can you test 6.0 instead of 6.0.19?) There is one commit which I'm bit suspicious about: ef6f5494faf6a37c74990689a3bb3cee76d2544c it changes how HDMI are assigned and as a machine board present on EVE makes use of HDMI, it may potentially cause some problems. Can you try reverting it? (If reverting on top of v6.1.8 you need to revert both f9aafff5448b1d8d457052271cd9a11b24e4d0bd and ef6f5494faf6a37c74990689a3bb3cee76d2544c which has minor conflict, easily resolved with just adding both lines.
Yes, happy to give that a shot and will report back.
I also still wonder, why problem reproduces only on some distributions... any chance you can try and boot with pipewire/pulseaudio disabled and see if it still happens, iirc those tools try to check all FEs and this may be breaking something during enumeration.
I can definitely try disabling pulseaudio and switching to pipewire and seeing if anything changes as well.
FWIW, I installed Arch on a thumb drive this weekend and was able to reproduce the issue and work around it by reverting the commit from my first bisect. So, for me it behaves just like Fedora. The instructions for Arch for building a custom kernel are great except they generalize the bootloader instructions, so you need to know what to do at the end to add the grub boot entries, if using grub for example, and I suspect that may be where the confusion came from, though I don't know. I'm trying to get one of the two to reproduce my results to confirm and at least get them a workaround.
I have slackware on another thumb drive already, but I have yet to even get it updated to 6.1.8.
If any of them behave differently I was hoping to tease out whether it's firmware, kernel config, or something else, but so far the first has been more of the same.
Thanks, Amadeusz
On Wed, Feb 1, 2023 at 9:33 AM Jason Montleon jmontleo@redhat.com wrote:
On Wed, Feb 1, 2023 at 6:05 AM Amadeusz Sławiński amadeuszx.slawinski@linux.intel.com wrote:
On 1/31/2023 4:16 PM, Jason Montleon wrote:
On Tue, Jan 31, 2023 at 7:37 AM Cezary Rojewski cezary.rojewski@intel.com wrote:
On 2023-01-30 1:22 PM, Sasa Ostrouska wrote:
> Dear Czarek, many thanks for the answer and taking care of it. If > needed something from my side please jest let me know > and I will try to do it.
Hello Sasa,
Could you provide us with the topology and firmware binary present on your machine?
Audio topology is located at /lib/firmware and named:
9d71-GOOGLE-EVEMAX-0-tplg.bin -or- dfw_sst.bin
Firmware on the other hand is found in /lib/firmware/intel/. 'dsp_fw_kbl.bin' will lie there, it shall be a symlink pointing to an actual AudioDSP firmware binary.
Maybe this is the problem.
I think most of us are pulling the topology and firmware from the chromeos recovery images for lack of any other known source, and it looks a little different than this. Those can be downloaded like so: https://gist.github.com/jmontleon/8899cb83138f2653f520fbbcc5b830a0
After placing the topology file you'll see these errors and audio will not work until they're also copied in place. snd_soc_skl 0000:00:1f.3: Direct firmware load for dsp_lib_dsm_core_spt_release.bin failed with error -2 snd_soc_skl 0000:00:1f.3: Direct firmware load for intel/dsp_fw_C75061F3-F2B2-4DCC-8F9F-82ABB4131E66.bin failed with error -2
Once those were in place, up to 6.0.18 audio worked.
Is there a better source for the topology file?
The reasoning for these asks is fact that problem stopped reproducing on our end once we started playing with kernel versions (moved away from status quo with Fedora). Neither on Lukasz EVE nor on my SKL RVP. However, we might be using newer configuration files when compared to equivalent of yours.
Recent v6.2-rc5 broonie/sound/for-next - no repro Our internal tree based on Mark's for-next - no repro 6.1.7 stable [1] - no repro
Of course we will continue with our attempts. Will notify about the progress.
Kind regards, Czarek
Hi Jason,
as I understand you've tried to do bisect, can you instead try building kernels checking out following tags: v6.1 v6.1.1 v6.1.2 v6.1.3 v6.1.4 v6.1.5 v6.1.6 v6.1.7 v6.1.8 and report when it stops working, so it narrows scope of what we look at? I assume that kernel builds are done using upstream stable kernel (from https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/).
Thanks, Amadeusz
Hi Amadeusz, Yes, I did the bisects using https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/
The only thing I did to these was add 392cc13c5ec72ccd6bbfb1bc2339502cc59dd285, otherwise audio breaks with the dai not registered error message in dmesg from the rt5514 bug from 6.0 and up. It wasn't added to 6.1 until rc6, I believe. If there's a better way to work around the multiple bugs I can try again, otherwise I will start working on builds from tags and see if I learn anything.
FWIW, I've seen two people complain that Arch isn't working either since it moved to 6.1. For the one who was trying, patching out the commit I came to with the first bisect did not regain them sound like it did for me. And yet Sasa reports Slackware is mostly working for him with 6.1.8 on Slackware. I don't know what to make of it, but thought I'd share in case it helps point someone else to something. https://github.com/jmontleon/pixelbook-fedora/issues/51#issuecomment-1410222... https://github.com/jmontleon/pixelbook-fedora/issues/51#issuecomment-1410673... https://github.com/jmontleon/pixelbook-fedora/issues/53#issuecomment-1408699...
Probably less relevant since they aren't from upstream and I know they don't mean as much, but I have tried 6.1.5-6.1.8 Fedora packages for certain, and went back trying several others from koji back into rc builds, although using prebuilt kernels, anything before 6.1-rc6 won't work, as mentioned above. Nothing worked. But as I said I'll build from tags and see if I can learn anything.
Thank you, Jason Montleon
-- Jason Montleon | email: jmontleo@redhat.com Red Hat, Inc. | gpg key: 0x069E3022 Cell: 508-496-0663 | irc: jmontleo / jmontleon
-- Jason Montleon | email: jmontleo@redhat.com Red Hat, Inc. | gpg key: 0x069E3022 Cell: 508-496-0663 | irc: jmontleo / jmontleon
On Mon, Feb 6, 2023 at 8:51 AM Jason Montleon jmontleo@redhat.com wrote:
On Mon, Feb 6, 2023 at 4:04 AM Amadeusz Sławiński amadeuszx.slawinski@linux.intel.com wrote:
On 2/4/2023 4:16 PM, Jason Montleon wrote:
I have built kernels for 6.0.19 (I don't think anyone confirmed whether or not it worked), plus every 6.1 tag from 6.1-rc1 up to 6.1.7. 6.0.19 worked. No 6.1 kernels worked. For rc1 to rc5 I built with and without the legacy dai renaming patch added in rc6 that I believe would be necessary, but it made no difference either way.
Hi,
thank you for trying to narrow it down, if I understand correctly -rc1 doesn't work, which means that problem was introduced somewhere between 6.0 and 6.1-rc1 (just for the sake of being sure, can you test 6.0 instead of 6.0.19?) There is one commit which I'm bit suspicious about: ef6f5494faf6a37c74990689a3bb3cee76d2544c it changes how HDMI are assigned and as a machine board present on EVE makes use of HDMI, it may potentially cause some problems. Can you try reverting it? (If reverting on top of v6.1.8 you need to revert both f9aafff5448b1d8d457052271cd9a11b24e4d0bd and ef6f5494faf6a37c74990689a3bb3cee76d2544c which has minor conflict, easily resolved with just adding both lines.
Yes, happy to give that a shot and will report back.
Removing f9aafff5448b1d8d457052271cd9a11b24e4d0bd and ef6f5494faf6a37c74990689a3bb3cee76d2544c did not make things work.
You may be onto something with pulseaudio and/or HDMI, however. When setting up Slackware I saw an interesting aplay hang. Normally aplay -l will list like this with working audio: $ aplay -l **** List of PLAYBACK Hardware Devices **** card 0: kblr55145663max [kbl-r5514-5663-max], device 0: Audio (*) [] Subdevices: 1/1 Subdevice #0: subdevice #0 card 0: kblr55145663max [kbl-r5514-5663-max], device 2: Headset Audio (*) [] Subdevices: 1/1 Subdevice #0: subdevice #0 card 0: kblr55145663max [kbl-r5514-5663-max], device 6: Hdmi1 (*) [] Subdevices: 1/1 Subdevice #0: subdevice #0 card 0: kblr55145663max [kbl-r5514-5663-max], device 7: Hdmi2 (*) [] Subdevices: 1/1 Subdevice #0: subdevice #0
Both on Slackware and Fedora with broken audio it hangs like so (haven't tried on Arch): $ aplay -l **** List of PLAYBACK Hardware Devices **** card 0: kblr55145663max [kbl-r5514-5663-max], device 0: Audio (*) [] Subdevices: 1/1 Subdevice #0: subdevice #0 card 0: kblr55145663max [kbl-r5514-5663-max], device 2: Headset Audio (*) [] Subdevices: 1/1 Subdevice #0: subdevice #0 card 0: kblr55145663max [kbl-r5514-5663-max], device 6: Hdmi1 (*) [] Subdevices: 1/1 Subdevice #0: subdevice #0
If I remove or disable pulseaudio it lists without hanging, but it's difficult for me to tell whether it's working since aplay, etc. seem to want pulseaudio to play anything. Shutdown hangs persist regardless.
Also, Slackware with 6.1.9 behaves as badly for me as everything else. If Sasa has working audio I do not know how he has managed to configure it. On each distro, as soon as I add topology and firmware files everything goes bad, regardless of whether I add ucm configuration or not, etc.
I also still wonder, why problem reproduces only on some distributions... any chance you can try and boot with pipewire/pulseaudio disabled and see if it still happens, iirc those tools try to check all FEs and this may be breaking something during enumeration.
I can definitely try disabling pulseaudio and switching to pipewire and seeing if anything changes as well.
FWIW, I installed Arch on a thumb drive this weekend and was able to reproduce the issue and work around it by reverting the commit from my first bisect. So, for me it behaves just like Fedora. The instructions for Arch for building a custom kernel are great except they generalize the bootloader instructions, so you need to know what to do at the end to add the grub boot entries, if using grub for example, and I suspect that may be where the confusion came from, though I don't know. I'm trying to get one of the two to reproduce my results to confirm and at least get them a workaround.
I have slackware on another thumb drive already, but I have yet to even get it updated to 6.1.8.
If any of them behave differently I was hoping to tease out whether it's firmware, kernel config, or something else, but so far the first has been more of the same.
Thanks, Amadeusz
On Wed, Feb 1, 2023 at 9:33 AM Jason Montleon jmontleo@redhat.com wrote:
On Wed, Feb 1, 2023 at 6:05 AM Amadeusz Sławiński amadeuszx.slawinski@linux.intel.com wrote:
On 1/31/2023 4:16 PM, Jason Montleon wrote:
On Tue, Jan 31, 2023 at 7:37 AM Cezary Rojewski cezary.rojewski@intel.com wrote: > > On 2023-01-30 1:22 PM, Sasa Ostrouska wrote: > >> Dear Czarek, many thanks for the answer and taking care of it. If >> needed something from my side please jest let me know >> and I will try to do it. > > > Hello Sasa, > > Could you provide us with the topology and firmware binary present on > your machine? > > Audio topology is located at /lib/firmware and named: > > 9d71-GOOGLE-EVEMAX-0-tplg.bin > -or- > dfw_sst.bin > > Firmware on the other hand is found in /lib/firmware/intel/. > 'dsp_fw_kbl.bin' will lie there, it shall be a symlink pointing to an > actual AudioDSP firmware binary. > Maybe this is the problem.
I think most of us are pulling the topology and firmware from the chromeos recovery images for lack of any other known source, and it looks a little different than this. Those can be downloaded like so: https://gist.github.com/jmontleon/8899cb83138f2653f520fbbcc5b830a0
After placing the topology file you'll see these errors and audio will not work until they're also copied in place. snd_soc_skl 0000:00:1f.3: Direct firmware load for dsp_lib_dsm_core_spt_release.bin failed with error -2 snd_soc_skl 0000:00:1f.3: Direct firmware load for intel/dsp_fw_C75061F3-F2B2-4DCC-8F9F-82ABB4131E66.bin failed with error -2
Once those were in place, up to 6.0.18 audio worked.
Is there a better source for the topology file?
> The reasoning for these asks is fact that problem stopped reproducing on > our end once we started playing with kernel versions (moved away from > status quo with Fedora). Neither on Lukasz EVE nor on my SKL RVP. > However, we might be using newer configuration files when compared to > equivalent of yours. > > Recent v6.2-rc5 broonie/sound/for-next - no repro > Our internal tree based on Mark's for-next - no repro > 6.1.7 stable [1] - no repro > > Of course we will continue with our attempts. Will notify about the > progress. > > > [1]: > https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?h=v... > > > Kind regards, > Czarek >
Hi Jason,
as I understand you've tried to do bisect, can you instead try building kernels checking out following tags: v6.1 v6.1.1 v6.1.2 v6.1.3 v6.1.4 v6.1.5 v6.1.6 v6.1.7 v6.1.8 and report when it stops working, so it narrows scope of what we look at? I assume that kernel builds are done using upstream stable kernel (from https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/).
Thanks, Amadeusz
Hi Amadeusz, Yes, I did the bisects using https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/
The only thing I did to these was add 392cc13c5ec72ccd6bbfb1bc2339502cc59dd285, otherwise audio breaks with the dai not registered error message in dmesg from the rt5514 bug from 6.0 and up. It wasn't added to 6.1 until rc6, I believe. If there's a better way to work around the multiple bugs I can try again, otherwise I will start working on builds from tags and see if I learn anything.
FWIW, I've seen two people complain that Arch isn't working either since it moved to 6.1. For the one who was trying, patching out the commit I came to with the first bisect did not regain them sound like it did for me. And yet Sasa reports Slackware is mostly working for him with 6.1.8 on Slackware. I don't know what to make of it, but thought I'd share in case it helps point someone else to something. https://github.com/jmontleon/pixelbook-fedora/issues/51#issuecomment-1410222... https://github.com/jmontleon/pixelbook-fedora/issues/51#issuecomment-1410673... https://github.com/jmontleon/pixelbook-fedora/issues/53#issuecomment-1408699...
Probably less relevant since they aren't from upstream and I know they don't mean as much, but I have tried 6.1.5-6.1.8 Fedora packages for certain, and went back trying several others from koji back into rc builds, although using prebuilt kernels, anything before 6.1-rc6 won't work, as mentioned above. Nothing worked. But as I said I'll build from tags and see if I can learn anything.
Thank you, Jason Montleon
-- Jason Montleon | email: jmontleo@redhat.com Red Hat, Inc. | gpg key: 0x069E3022 Cell: 508-496-0663 | irc: jmontleo / jmontleon
-- Jason Montleon | email: jmontleo@redhat.com Red Hat, Inc. | gpg key: 0x069E3022 Cell: 508-496-0663 | irc: jmontleo / jmontleon
On Mon, Feb 6, 2023 at 8:57 PM Jason Montleon jmontleo@redhat.com wrote:
On Mon, Feb 6, 2023 at 8:51 AM Jason Montleon jmontleo@redhat.com wrote:
On Mon, Feb 6, 2023 at 4:04 AM Amadeusz Sławiński amadeuszx.slawinski@linux.intel.com wrote:
On 2/4/2023 4:16 PM, Jason Montleon wrote:
I have built kernels for 6.0.19 (I don't think anyone confirmed whether or not it worked), plus every 6.1 tag from 6.1-rc1 up to 6.1.7. 6.0.19 worked. No 6.1 kernels worked. For rc1 to rc5 I built with and without the legacy dai renaming patch added in rc6 that I believe would be necessary, but it made no difference either way.
Hi,
thank you for trying to narrow it down, if I understand correctly -rc1 doesn't work, which means that problem was introduced somewhere between 6.0 and 6.1-rc1 (just for the sake of being sure, can you test 6.0 instead of 6.0.19?) There is one commit which I'm bit suspicious about: ef6f5494faf6a37c74990689a3bb3cee76d2544c it changes how HDMI are assigned and as a machine board present on EVE makes use of HDMI, it may potentially cause some problems. Can you try reverting it? (If reverting on top of v6.1.8 you need to revert both f9aafff5448b1d8d457052271cd9a11b24e4d0bd and ef6f5494faf6a37c74990689a3bb3cee76d2544c which has minor conflict, easily resolved with just adding both lines.
Yes, happy to give that a shot and will report back.
Removing f9aafff5448b1d8d457052271cd9a11b24e4d0bd and ef6f5494faf6a37c74990689a3bb3cee76d2544c did not make things work.
You may be onto something with pulseaudio and/or HDMI, however. When setting up Slackware I saw an interesting aplay hang. Normally aplay -l will list like this with working audio: $ aplay -l **** List of PLAYBACK Hardware Devices **** card 0: kblr55145663max [kbl-r5514-5663-max], device 0: Audio (*) [] Subdevices: 1/1 Subdevice #0: subdevice #0 card 0: kblr55145663max [kbl-r5514-5663-max], device 2: Headset Audio (*) [] Subdevices: 1/1 Subdevice #0: subdevice #0 card 0: kblr55145663max [kbl-r5514-5663-max], device 6: Hdmi1 (*) [] Subdevices: 1/1 Subdevice #0: subdevice #0 card 0: kblr55145663max [kbl-r5514-5663-max], device 7: Hdmi2 (*) [] Subdevices: 1/1 Subdevice #0: subdevice #0
Both on Slackware and Fedora with broken audio it hangs like so (haven't tried on Arch): $ aplay -l **** List of PLAYBACK Hardware Devices **** card 0: kblr55145663max [kbl-r5514-5663-max], device 0: Audio (*) [] Subdevices: 1/1 Subdevice #0: subdevice #0 card 0: kblr55145663max [kbl-r5514-5663-max], device 2: Headset Audio (*) [] Subdevices: 1/1 Subdevice #0: subdevice #0 card 0: kblr55145663max [kbl-r5514-5663-max], device 6: Hdmi1 (*) [] Subdevices: 1/1 Subdevice #0: subdevice #0
If I remove or disable pulseaudio it lists without hanging, but it's difficult for me to tell whether it's working since aplay, etc. seem to want pulseaudio to play anything. Shutdown hangs persist regardless.
Also, Slackware with 6.1.9 behaves as badly for me as everything else. If Sasa has working audio I do not know how he has managed to configure it. On each distro, as soon as I add topology and firmware files everything goes bad, regardless of whether I add ucm configuration or not, etc.
As already said, Jason, my slackware install still have a working audio after all updates. I have installed slackware the common way, makeing it use BTRFS, installed by booting it , run setup on slackware64-15.0 installed all packages except KDE as I use XCFE or GNOME. Since gnome is not part of Slackware I am on XFCE right now. After I installed slackware64-15.0 I boot into it, and then followed the steps in your AUDIO section from step 1 to step 5. So basically copied firmware into /lib/firmware, /lib/firmware/intel and opt/google/dsm After that I have reboot and sound was working already on kernel 5.15.80 . Then I run slackpkg update and upgrade-all to update everything to slackware64-current, which at the time had a 6.1.8 kernel. Slackware usually ships an unpatched kernel supplied as it is from Linus tree.
Thats it, and initially I know sound was hanging sometimes. But today I do not experience any hangs anymore.
Only thing I had to do was to enable the kbl-r5514-5663-max profile in pavucontrol. As by default it was at off position.
But if needed any file or log from Slackware instsall let me know.
Rgds Sasa
I've done some more digging. The only line that needs to be reverted from f2bd1c5ae2cb0cf9525c9bffc0038c12dd7e1338, moving from snd_hda_codec_device_init back to snd_hda_codec_device_new is: codec->core.exec_verb = codec_exec_verb; (https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/tree/sound/...)
I added a bunch of debug statements and all the code in codec_exec_verb runs at boot with this in snd_hda_codec_device_init, whereas it does not when in snd_hda_codec_device_new.
From what I can tell we end up in snd_hda_power_up_pm and then get hung up at snd_hdac_power_up.
There are a bunch of pin port messages that show up from hdac_hdmi_query_port_connlist when things are working, that never appear when broken: [ 14.618805] HDMI HDA Codec ehdaudio0D2: No connections found for pin:port 5:0 [ 14.619242] HDMI HDA Codec ehdaudio0D2: No connections found for pin:port 5:1 [ 14.619703] HDMI HDA Codec ehdaudio0D2: No connections found for pin:port 5:2 ...
I do see hdac_hdmi_runtime_suspend run a moment before things go bad, but I have no idea if it is related.
Without patching anything and CONFIG_PM unset everything works.
I don't know if that helps anyone see where the problem is. If not I'll keep plugging away.
Incidentally, commit 3fd63658caed9494cca1d4789a66d3d2def2a0ab, pointed to by my second bisect, starts making using of skl_codec_device_init where I believe snd_hda_codec_device_init is called and starts the problem. I believe this is why reverting either of the two works around the problem.
On Mon, Feb 6, 2023 at 2:57 PM Jason Montleon jmontleo@redhat.com wrote:
On Mon, Feb 6, 2023 at 8:51 AM Jason Montleon jmontleo@redhat.com wrote:
On Mon, Feb 6, 2023 at 4:04 AM Amadeusz Sławiński amadeuszx.slawinski@linux.intel.com wrote:
On 2/4/2023 4:16 PM, Jason Montleon wrote:
I have built kernels for 6.0.19 (I don't think anyone confirmed whether or not it worked), plus every 6.1 tag from 6.1-rc1 up to 6.1.7. 6.0.19 worked. No 6.1 kernels worked. For rc1 to rc5 I built with and without the legacy dai renaming patch added in rc6 that I believe would be necessary, but it made no difference either way.
Hi,
thank you for trying to narrow it down, if I understand correctly -rc1 doesn't work, which means that problem was introduced somewhere between 6.0 and 6.1-rc1 (just for the sake of being sure, can you test 6.0 instead of 6.0.19?) There is one commit which I'm bit suspicious about: ef6f5494faf6a37c74990689a3bb3cee76d2544c it changes how HDMI are assigned and as a machine board present on EVE makes use of HDMI, it may potentially cause some problems. Can you try reverting it? (If reverting on top of v6.1.8 you need to revert both f9aafff5448b1d8d457052271cd9a11b24e4d0bd and ef6f5494faf6a37c74990689a3bb3cee76d2544c which has minor conflict, easily resolved with just adding both lines.
Yes, happy to give that a shot and will report back.
Removing f9aafff5448b1d8d457052271cd9a11b24e4d0bd and ef6f5494faf6a37c74990689a3bb3cee76d2544c did not make things work.
You may be onto something with pulseaudio and/or HDMI, however. When setting up Slackware I saw an interesting aplay hang. Normally aplay -l will list like this with working audio: $ aplay -l **** List of PLAYBACK Hardware Devices **** card 0: kblr55145663max [kbl-r5514-5663-max], device 0: Audio (*) [] Subdevices: 1/1 Subdevice #0: subdevice #0 card 0: kblr55145663max [kbl-r5514-5663-max], device 2: Headset Audio (*) [] Subdevices: 1/1 Subdevice #0: subdevice #0 card 0: kblr55145663max [kbl-r5514-5663-max], device 6: Hdmi1 (*) [] Subdevices: 1/1 Subdevice #0: subdevice #0 card 0: kblr55145663max [kbl-r5514-5663-max], device 7: Hdmi2 (*) [] Subdevices: 1/1 Subdevice #0: subdevice #0
Both on Slackware and Fedora with broken audio it hangs like so (haven't tried on Arch): $ aplay -l **** List of PLAYBACK Hardware Devices **** card 0: kblr55145663max [kbl-r5514-5663-max], device 0: Audio (*) [] Subdevices: 1/1 Subdevice #0: subdevice #0 card 0: kblr55145663max [kbl-r5514-5663-max], device 2: Headset Audio (*) [] Subdevices: 1/1 Subdevice #0: subdevice #0 card 0: kblr55145663max [kbl-r5514-5663-max], device 6: Hdmi1 (*) [] Subdevices: 1/1 Subdevice #0: subdevice #0
If I remove or disable pulseaudio it lists without hanging, but it's difficult for me to tell whether it's working since aplay, etc. seem to want pulseaudio to play anything. Shutdown hangs persist regardless.
Also, Slackware with 6.1.9 behaves as badly for me as everything else. If Sasa has working audio I do not know how he has managed to configure it. On each distro, as soon as I add topology and firmware files everything goes bad, regardless of whether I add ucm configuration or not, etc.
I also still wonder, why problem reproduces only on some distributions... any chance you can try and boot with pipewire/pulseaudio disabled and see if it still happens, iirc those tools try to check all FEs and this may be breaking something during enumeration.
I can definitely try disabling pulseaudio and switching to pipewire and seeing if anything changes as well.
FWIW, I installed Arch on a thumb drive this weekend and was able to reproduce the issue and work around it by reverting the commit from my first bisect. So, for me it behaves just like Fedora. The instructions for Arch for building a custom kernel are great except they generalize the bootloader instructions, so you need to know what to do at the end to add the grub boot entries, if using grub for example, and I suspect that may be where the confusion came from, though I don't know. I'm trying to get one of the two to reproduce my results to confirm and at least get them a workaround.
I have slackware on another thumb drive already, but I have yet to even get it updated to 6.1.8.
If any of them behave differently I was hoping to tease out whether it's firmware, kernel config, or something else, but so far the first has been more of the same.
Thanks, Amadeusz
On Wed, Feb 1, 2023 at 9:33 AM Jason Montleon jmontleo@redhat.com wrote:
On Wed, Feb 1, 2023 at 6:05 AM Amadeusz Sławiński amadeuszx.slawinski@linux.intel.com wrote:
On 1/31/2023 4:16 PM, Jason Montleon wrote: > On Tue, Jan 31, 2023 at 7:37 AM Cezary Rojewski > cezary.rojewski@intel.com wrote: >> >> On 2023-01-30 1:22 PM, Sasa Ostrouska wrote: >> >>> Dear Czarek, many thanks for the answer and taking care of it. If >>> needed something from my side please jest let me know >>> and I will try to do it. >> >> >> Hello Sasa, >> >> Could you provide us with the topology and firmware binary present on >> your machine? >> >> Audio topology is located at /lib/firmware and named: >> >> 9d71-GOOGLE-EVEMAX-0-tplg.bin >> -or- >> dfw_sst.bin >> >> Firmware on the other hand is found in /lib/firmware/intel/. >> 'dsp_fw_kbl.bin' will lie there, it shall be a symlink pointing to an >> actual AudioDSP firmware binary. >> > Maybe this is the problem. > > I think most of us are pulling the topology and firmware from the > chromeos recovery images for lack of any other known source, and it > looks a little different than this. Those can be downloaded like so: > https://gist.github.com/jmontleon/8899cb83138f2653f520fbbcc5b830a0 > > After placing the topology file you'll see these errors and audio will > not work until they're also copied in place. > snd_soc_skl 0000:00:1f.3: Direct firmware load for > dsp_lib_dsm_core_spt_release.bin failed with error -2 > snd_soc_skl 0000:00:1f.3: Direct firmware load for > intel/dsp_fw_C75061F3-F2B2-4DCC-8F9F-82ABB4131E66.bin failed with > error -2 > > Once those were in place, up to 6.0.18 audio worked. > > Is there a better source for the topology file? > >> The reasoning for these asks is fact that problem stopped reproducing on >> our end once we started playing with kernel versions (moved away from >> status quo with Fedora). Neither on Lukasz EVE nor on my SKL RVP. >> However, we might be using newer configuration files when compared to >> equivalent of yours. >> >> Recent v6.2-rc5 broonie/sound/for-next - no repro >> Our internal tree based on Mark's for-next - no repro >> 6.1.7 stable [1] - no repro >> >> Of course we will continue with our attempts. Will notify about the >> progress. >> >> >> [1]: >> https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?h=v... >> >> >> Kind regards, >> Czarek >> > >
Hi Jason,
as I understand you've tried to do bisect, can you instead try building kernels checking out following tags: v6.1 v6.1.1 v6.1.2 v6.1.3 v6.1.4 v6.1.5 v6.1.6 v6.1.7 v6.1.8 and report when it stops working, so it narrows scope of what we look at? I assume that kernel builds are done using upstream stable kernel (from https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/).
Thanks, Amadeusz
Hi Amadeusz, Yes, I did the bisects using https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/
The only thing I did to these was add 392cc13c5ec72ccd6bbfb1bc2339502cc59dd285, otherwise audio breaks with the dai not registered error message in dmesg from the rt5514 bug from 6.0 and up. It wasn't added to 6.1 until rc6, I believe. If there's a better way to work around the multiple bugs I can try again, otherwise I will start working on builds from tags and see if I learn anything.
FWIW, I've seen two people complain that Arch isn't working either since it moved to 6.1. For the one who was trying, patching out the commit I came to with the first bisect did not regain them sound like it did for me. And yet Sasa reports Slackware is mostly working for him with 6.1.8 on Slackware. I don't know what to make of it, but thought I'd share in case it helps point someone else to something. https://github.com/jmontleon/pixelbook-fedora/issues/51#issuecomment-1410222... https://github.com/jmontleon/pixelbook-fedora/issues/51#issuecomment-1410673... https://github.com/jmontleon/pixelbook-fedora/issues/53#issuecomment-1408699...
Probably less relevant since they aren't from upstream and I know they don't mean as much, but I have tried 6.1.5-6.1.8 Fedora packages for certain, and went back trying several others from koji back into rc builds, although using prebuilt kernels, anything before 6.1-rc6 won't work, as mentioned above. Nothing worked. But as I said I'll build from tags and see if I can learn anything.
Thank you, Jason Montleon
-- Jason Montleon | email: jmontleo@redhat.com Red Hat, Inc. | gpg key: 0x069E3022 Cell: 508-496-0663 | irc: jmontleo / jmontleon
-- Jason Montleon | email: jmontleo@redhat.com Red Hat, Inc. | gpg key: 0x069E3022 Cell: 508-496-0663 | irc: jmontleo / jmontleon
-- Jason Montleon | email: jmontleo@redhat.com Red Hat, Inc. | gpg key: 0x069E3022 Cell: 508-496-0663 | irc: jmontleo / jmontleon
-- Jason Montleon | email: jmontleo@redhat.com Red Hat, Inc. | gpg key: 0x069E3022 Cell: 508-496-0663 | irc: jmontleo / jmontleon
I have a hung task call trace from a debug kernel in case it's helpful: https://gist.github.com/jmontleon/a6dff2ad949cc50bb8f162d7b306b320
On Thu, Feb 9, 2023 at 11:13 AM Jason Montleon jmontleo@redhat.com wrote:
I've done some more digging. The only line that needs to be reverted from f2bd1c5ae2cb0cf9525c9bffc0038c12dd7e1338, moving from snd_hda_codec_device_init back to snd_hda_codec_device_new is: codec->core.exec_verb = codec_exec_verb; (https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/tree/sound/...)
I added a bunch of debug statements and all the code in codec_exec_verb runs at boot with this in snd_hda_codec_device_init, whereas it does not when in snd_hda_codec_device_new.
From what I can tell we end up in snd_hda_power_up_pm and then get hung up at snd_hdac_power_up.
There are a bunch of pin port messages that show up from hdac_hdmi_query_port_connlist when things are working, that never appear when broken: [ 14.618805] HDMI HDA Codec ehdaudio0D2: No connections found for pin:port 5:0 [ 14.619242] HDMI HDA Codec ehdaudio0D2: No connections found for pin:port 5:1 [ 14.619703] HDMI HDA Codec ehdaudio0D2: No connections found for pin:port 5:2 ...
I do see hdac_hdmi_runtime_suspend run a moment before things go bad, but I have no idea if it is related.
Without patching anything and CONFIG_PM unset everything works.
I don't know if that helps anyone see where the problem is. If not I'll keep plugging away.
Incidentally, commit 3fd63658caed9494cca1d4789a66d3d2def2a0ab, pointed to by my second bisect, starts making using of skl_codec_device_init where I believe snd_hda_codec_device_init is called and starts the problem. I believe this is why reverting either of the two works around the problem.
On Mon, Feb 6, 2023 at 2:57 PM Jason Montleon jmontleo@redhat.com wrote:
On Mon, Feb 6, 2023 at 8:51 AM Jason Montleon jmontleo@redhat.com wrote:
On Mon, Feb 6, 2023 at 4:04 AM Amadeusz Sławiński amadeuszx.slawinski@linux.intel.com wrote:
On 2/4/2023 4:16 PM, Jason Montleon wrote:
I have built kernels for 6.0.19 (I don't think anyone confirmed whether or not it worked), plus every 6.1 tag from 6.1-rc1 up to 6.1.7. 6.0.19 worked. No 6.1 kernels worked. For rc1 to rc5 I built with and without the legacy dai renaming patch added in rc6 that I believe would be necessary, but it made no difference either way.
Hi,
thank you for trying to narrow it down, if I understand correctly -rc1 doesn't work, which means that problem was introduced somewhere between 6.0 and 6.1-rc1 (just for the sake of being sure, can you test 6.0 instead of 6.0.19?) There is one commit which I'm bit suspicious about: ef6f5494faf6a37c74990689a3bb3cee76d2544c it changes how HDMI are assigned and as a machine board present on EVE makes use of HDMI, it may potentially cause some problems. Can you try reverting it? (If reverting on top of v6.1.8 you need to revert both f9aafff5448b1d8d457052271cd9a11b24e4d0bd and ef6f5494faf6a37c74990689a3bb3cee76d2544c which has minor conflict, easily resolved with just adding both lines.
Yes, happy to give that a shot and will report back.
Removing f9aafff5448b1d8d457052271cd9a11b24e4d0bd and ef6f5494faf6a37c74990689a3bb3cee76d2544c did not make things work.
You may be onto something with pulseaudio and/or HDMI, however. When setting up Slackware I saw an interesting aplay hang. Normally aplay -l will list like this with working audio: $ aplay -l **** List of PLAYBACK Hardware Devices **** card 0: kblr55145663max [kbl-r5514-5663-max], device 0: Audio (*) [] Subdevices: 1/1 Subdevice #0: subdevice #0 card 0: kblr55145663max [kbl-r5514-5663-max], device 2: Headset Audio (*) [] Subdevices: 1/1 Subdevice #0: subdevice #0 card 0: kblr55145663max [kbl-r5514-5663-max], device 6: Hdmi1 (*) [] Subdevices: 1/1 Subdevice #0: subdevice #0 card 0: kblr55145663max [kbl-r5514-5663-max], device 7: Hdmi2 (*) [] Subdevices: 1/1 Subdevice #0: subdevice #0
Both on Slackware and Fedora with broken audio it hangs like so (haven't tried on Arch): $ aplay -l **** List of PLAYBACK Hardware Devices **** card 0: kblr55145663max [kbl-r5514-5663-max], device 0: Audio (*) [] Subdevices: 1/1 Subdevice #0: subdevice #0 card 0: kblr55145663max [kbl-r5514-5663-max], device 2: Headset Audio (*) [] Subdevices: 1/1 Subdevice #0: subdevice #0 card 0: kblr55145663max [kbl-r5514-5663-max], device 6: Hdmi1 (*) [] Subdevices: 1/1 Subdevice #0: subdevice #0
If I remove or disable pulseaudio it lists without hanging, but it's difficult for me to tell whether it's working since aplay, etc. seem to want pulseaudio to play anything. Shutdown hangs persist regardless.
Also, Slackware with 6.1.9 behaves as badly for me as everything else. If Sasa has working audio I do not know how he has managed to configure it. On each distro, as soon as I add topology and firmware files everything goes bad, regardless of whether I add ucm configuration or not, etc.
I also still wonder, why problem reproduces only on some distributions... any chance you can try and boot with pipewire/pulseaudio disabled and see if it still happens, iirc those tools try to check all FEs and this may be breaking something during enumeration.
I can definitely try disabling pulseaudio and switching to pipewire and seeing if anything changes as well.
FWIW, I installed Arch on a thumb drive this weekend and was able to reproduce the issue and work around it by reverting the commit from my first bisect. So, for me it behaves just like Fedora. The instructions for Arch for building a custom kernel are great except they generalize the bootloader instructions, so you need to know what to do at the end to add the grub boot entries, if using grub for example, and I suspect that may be where the confusion came from, though I don't know. I'm trying to get one of the two to reproduce my results to confirm and at least get them a workaround.
I have slackware on another thumb drive already, but I have yet to even get it updated to 6.1.8.
If any of them behave differently I was hoping to tease out whether it's firmware, kernel config, or something else, but so far the first has been more of the same.
Thanks, Amadeusz
On Wed, Feb 1, 2023 at 9:33 AM Jason Montleon jmontleo@redhat.com wrote:
On Wed, Feb 1, 2023 at 6:05 AM Amadeusz Sławiński amadeuszx.slawinski@linux.intel.com wrote: > > On 1/31/2023 4:16 PM, Jason Montleon wrote: >> On Tue, Jan 31, 2023 at 7:37 AM Cezary Rojewski >> cezary.rojewski@intel.com wrote: >>> >>> On 2023-01-30 1:22 PM, Sasa Ostrouska wrote: >>> >>>> Dear Czarek, many thanks for the answer and taking care of it. If >>>> needed something from my side please jest let me know >>>> and I will try to do it. >>> >>> >>> Hello Sasa, >>> >>> Could you provide us with the topology and firmware binary present on >>> your machine? >>> >>> Audio topology is located at /lib/firmware and named: >>> >>> 9d71-GOOGLE-EVEMAX-0-tplg.bin >>> -or- >>> dfw_sst.bin >>> >>> Firmware on the other hand is found in /lib/firmware/intel/. >>> 'dsp_fw_kbl.bin' will lie there, it shall be a symlink pointing to an >>> actual AudioDSP firmware binary. >>> >> Maybe this is the problem. >> >> I think most of us are pulling the topology and firmware from the >> chromeos recovery images for lack of any other known source, and it >> looks a little different than this. Those can be downloaded like so: >> https://gist.github.com/jmontleon/8899cb83138f2653f520fbbcc5b830a0 >> >> After placing the topology file you'll see these errors and audio will >> not work until they're also copied in place. >> snd_soc_skl 0000:00:1f.3: Direct firmware load for >> dsp_lib_dsm_core_spt_release.bin failed with error -2 >> snd_soc_skl 0000:00:1f.3: Direct firmware load for >> intel/dsp_fw_C75061F3-F2B2-4DCC-8F9F-82ABB4131E66.bin failed with >> error -2 >> >> Once those were in place, up to 6.0.18 audio worked. >> >> Is there a better source for the topology file? >> >>> The reasoning for these asks is fact that problem stopped reproducing on >>> our end once we started playing with kernel versions (moved away from >>> status quo with Fedora). Neither on Lukasz EVE nor on my SKL RVP. >>> However, we might be using newer configuration files when compared to >>> equivalent of yours. >>> >>> Recent v6.2-rc5 broonie/sound/for-next - no repro >>> Our internal tree based on Mark's for-next - no repro >>> 6.1.7 stable [1] - no repro >>> >>> Of course we will continue with our attempts. Will notify about the >>> progress. >>> >>> >>> [1]: >>> https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?h=v... >>> >>> >>> Kind regards, >>> Czarek >>> >> >> > > Hi Jason, > > as I understand you've tried to do bisect, can you instead try building > kernels checking out following tags: > v6.1 v6.1.1 v6.1.2 v6.1.3 v6.1.4 v6.1.5 v6.1.6 > v6.1.7 v6.1.8 > and report when it stops working, so it narrows scope of what we look > at? I assume that kernel builds are done using upstream stable kernel > (from https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/). > > Thanks, > Amadeusz > Hi Amadeusz, Yes, I did the bisects using https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/
The only thing I did to these was add 392cc13c5ec72ccd6bbfb1bc2339502cc59dd285, otherwise audio breaks with the dai not registered error message in dmesg from the rt5514 bug from 6.0 and up. It wasn't added to 6.1 until rc6, I believe. If there's a better way to work around the multiple bugs I can try again, otherwise I will start working on builds from tags and see if I learn anything.
FWIW, I've seen two people complain that Arch isn't working either since it moved to 6.1. For the one who was trying, patching out the commit I came to with the first bisect did not regain them sound like it did for me. And yet Sasa reports Slackware is mostly working for him with 6.1.8 on Slackware. I don't know what to make of it, but thought I'd share in case it helps point someone else to something. https://github.com/jmontleon/pixelbook-fedora/issues/51#issuecomment-1410222... https://github.com/jmontleon/pixelbook-fedora/issues/51#issuecomment-1410673... https://github.com/jmontleon/pixelbook-fedora/issues/53#issuecomment-1408699...
Probably less relevant since they aren't from upstream and I know they don't mean as much, but I have tried 6.1.5-6.1.8 Fedora packages for certain, and went back trying several others from koji back into rc builds, although using prebuilt kernels, anything before 6.1-rc6 won't work, as mentioned above. Nothing worked. But as I said I'll build from tags and see if I can learn anything.
Thank you, Jason Montleon
-- Jason Montleon | email: jmontleo@redhat.com Red Hat, Inc. | gpg key: 0x069E3022 Cell: 508-496-0663 | irc: jmontleo / jmontleon
-- Jason Montleon | email: jmontleo@redhat.com Red Hat, Inc. | gpg key: 0x069E3022 Cell: 508-496-0663 | irc: jmontleo / jmontleon
-- Jason Montleon | email: jmontleo@redhat.com Red Hat, Inc. | gpg key: 0x069E3022 Cell: 508-496-0663 | irc: jmontleo / jmontleon
-- Jason Montleon | email: jmontleo@redhat.com Red Hat, Inc. | gpg key: 0x069E3022 Cell: 508-496-0663 | irc: jmontleo / jmontleon
On 2023-02-09 5:13 PM, Jason Montleon wrote:
I've done some more digging. The only line that needs to be reverted from f2bd1c5ae2cb0cf9525c9bffc0038c12dd7e1338, moving from snd_hda_codec_device_init back to snd_hda_codec_device_new is: codec->core.exec_verb = codec_exec_verb; (https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/tree/sound/...)
I added a bunch of debug statements and all the code in codec_exec_verb runs at boot with this in snd_hda_codec_device_init, whereas it does not when in snd_hda_codec_device_new.
From what I can tell we end up in snd_hda_power_up_pm and then get hung up at snd_hdac_power_up.
There are a bunch of pin port messages that show up from hdac_hdmi_query_port_connlist when things are working, that never appear when broken: [ 14.618805] HDMI HDA Codec ehdaudio0D2: No connections found for pin:port 5:0 [ 14.619242] HDMI HDA Codec ehdaudio0D2: No connections found for pin:port 5:1 [ 14.619703] HDMI HDA Codec ehdaudio0D2: No connections found for pin:port 5:2 ...
I do see hdac_hdmi_runtime_suspend run a moment before things go bad, but I have no idea if it is related.
Without patching anything and CONFIG_PM unset everything works.
I don't know if that helps anyone see where the problem is. If not I'll keep plugging away.
Incidentally, commit 3fd63658caed9494cca1d4789a66d3d2def2a0ab, pointed to by my second bisect, starts making using of skl_codec_device_init where I believe snd_hda_codec_device_init is called and starts the problem. I believe this is why reverting either of the two works around the problem.
This is some exceptional debugging, Jason.
I believe this finding reveals a long standing problem that was covered by a very specific codec-fields initialization order:
During initial part of codec-device initialization, VERBs execution follows different flow than one happening once the device is fully initialized. This comes down to the if-statement preset in snd_hdac_exec_verb() and the fact codec_exec_verb() differs from snd_hdac_bus_exec_verb() in PM-handling - the latter is devoid of it.
That is until ->exec_verb gets initialized and codec_exec_verb() becomes the sole handler of VERB execution process. As PM is not yet configured at the time - snd_hda_codec_device_init() happens early, whereas PM configuration is done later with snd_hda_set_power_save() during skl_hda_audio_probe() in sound/soc/intel/boards/skl_hda_dsp_generic.c - it should not be touched yet.
I'm up for reverting this single line to where it was before the offending patch. We still want to avoid the page fault the very patch is addressing.
Does the proposed change address both problems? i.e. no sound and hang during shutdown?
Kind regards, Czarek
On Fri, Feb 10, 2023 at 8:10 AM Cezary Rojewski cezary.rojewski@intel.com wrote:
On 2023-02-09 5:13 PM, Jason Montleon wrote:
I've done some more digging. The only line that needs to be reverted from f2bd1c5ae2cb0cf9525c9bffc0038c12dd7e1338, moving from snd_hda_codec_device_init back to snd_hda_codec_device_new is: codec->core.exec_verb = codec_exec_verb; (https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/tree/sound/...)
I added a bunch of debug statements and all the code in codec_exec_verb runs at boot with this in snd_hda_codec_device_init, whereas it does not when in snd_hda_codec_device_new.
From what I can tell we end up in snd_hda_power_up_pm and then get hung up at snd_hdac_power_up.
There are a bunch of pin port messages that show up from hdac_hdmi_query_port_connlist when things are working, that never appear when broken: [ 14.618805] HDMI HDA Codec ehdaudio0D2: No connections found for pin:port 5:0 [ 14.619242] HDMI HDA Codec ehdaudio0D2: No connections found for pin:port 5:1 [ 14.619703] HDMI HDA Codec ehdaudio0D2: No connections found for pin:port 5:2 ...
I do see hdac_hdmi_runtime_suspend run a moment before things go bad, but I have no idea if it is related.
Without patching anything and CONFIG_PM unset everything works.
I don't know if that helps anyone see where the problem is. If not I'll keep plugging away.
Incidentally, commit 3fd63658caed9494cca1d4789a66d3d2def2a0ab, pointed to by my second bisect, starts making using of skl_codec_device_init where I believe snd_hda_codec_device_init is called and starts the problem. I believe this is why reverting either of the two works around the problem.
This is some exceptional debugging, Jason.
I believe this finding reveals a long standing problem that was covered by a very specific codec-fields initialization order:
During initial part of codec-device initialization, VERBs execution follows different flow than one happening once the device is fully initialized. This comes down to the if-statement preset in snd_hdac_exec_verb() and the fact codec_exec_verb() differs from snd_hdac_bus_exec_verb() in PM-handling - the latter is devoid of it.
Thank you for the explanation! I was not following this well, but it makes sense to me now.
That is until ->exec_verb gets initialized and codec_exec_verb() becomes the sole handler of VERB execution process. As PM is not yet configured at the time - snd_hda_codec_device_init() happens early, whereas PM configuration is done later with snd_hda_set_power_save() during skl_hda_audio_probe() in sound/soc/intel/boards/skl_hda_dsp_generic.c - it should not be touched yet.
I'm up for reverting this single line to where it was before the offending patch. We still want to avoid the page fault the very patch is addressing.
Does the proposed change address both problems? i.e. no sound and hang during shutdown?
Yes, moving the one line back fixes both the no sound and shutdown hang.
Thank you! Jason Montleon
Kind regards, Czarek
-- Jason Montleon | email: jmontleo@redhat.com Red Hat, Inc. | gpg key: 0x069E3022 Cell: 508-496-0663 | irc: jmontleo / jmontleon
On Fri, Feb 10, 2023 at 3:57 PM Jason Montleon jmontleo@redhat.com wrote:
On Fri, Feb 10, 2023 at 8:10 AM Cezary Rojewski cezary.rojewski@intel.com wrote:
On 2023-02-09 5:13 PM, Jason Montleon wrote:
I've done some more digging. The only line that needs to be reverted from f2bd1c5ae2cb0cf9525c9bffc0038c12dd7e1338, moving from snd_hda_codec_device_init back to snd_hda_codec_device_new is: codec->core.exec_verb = codec_exec_verb; (https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/tree/sound/...)
I added a bunch of debug statements and all the code in codec_exec_verb runs at boot with this in snd_hda_codec_device_init, whereas it does not when in snd_hda_codec_device_new.
From what I can tell we end up in snd_hda_power_up_pm and then get hung up at snd_hdac_power_up.
There are a bunch of pin port messages that show up from hdac_hdmi_query_port_connlist when things are working, that never appear when broken: [ 14.618805] HDMI HDA Codec ehdaudio0D2: No connections found for pin:port 5:0 [ 14.619242] HDMI HDA Codec ehdaudio0D2: No connections found for pin:port 5:1 [ 14.619703] HDMI HDA Codec ehdaudio0D2: No connections found for pin:port 5:2 ...
I do see hdac_hdmi_runtime_suspend run a moment before things go bad, but I have no idea if it is related.
Without patching anything and CONFIG_PM unset everything works.
I don't know if that helps anyone see where the problem is. If not I'll keep plugging away.
Incidentally, commit 3fd63658caed9494cca1d4789a66d3d2def2a0ab, pointed to by my second bisect, starts making using of skl_codec_device_init where I believe snd_hda_codec_device_init is called and starts the problem. I believe this is why reverting either of the two works around the problem.
This is some exceptional debugging, Jason.
I believe this finding reveals a long standing problem that was covered by a very specific codec-fields initialization order:
During initial part of codec-device initialization, VERBs execution follows different flow than one happening once the device is fully initialized. This comes down to the if-statement preset in snd_hdac_exec_verb() and the fact codec_exec_verb() differs from snd_hdac_bus_exec_verb() in PM-handling - the latter is devoid of it.
Thank you for the explanation! I was not following this well, but it makes sense to me now.
That is until ->exec_verb gets initialized and codec_exec_verb() becomes the sole handler of VERB execution process. As PM is not yet configured at the time - snd_hda_codec_device_init() happens early, whereas PM configuration is done later with snd_hda_set_power_save() during skl_hda_audio_probe() in sound/soc/intel/boards/skl_hda_dsp_generic.c - it should not be touched yet.
I'm up for reverting this single line to where it was before the offending patch. We still want to avoid the page fault the very patch is addressing.
Does the proposed change address both problems? i.e. no sound and hang during shutdown?
Yes, moving the one line back fixes both the no sound and shutdown hang.
Thank you! Jason Montleon
Kind regards, Czarek
-- Jason Montleon | email: jmontleo@redhat.com Red Hat, Inc. | gpg key: 0x069E3022 Cell: 508-496-0663 | irc: jmontleo / jmontleon
Congrats to all of you people !! Thanks for the fix. Sasa
[TLDR: This mail in primarily relevant for Linux kernel regression tracking. See link in footer if these mails annoy you.]
On 28.01.23 18:09, Jason Montleon wrote:
I did a bisect which implicated f2bd1c5ae2cb0cf9525c9bffc0038c12dd7e1338 as the first bad commit.
Reverting this commit on 6.1.8 gives me working sound again.
I'm not clear why this is breaking 6.1.x since it appears to be in 6.0.18 (7494e2e6c55e), which was the last working package in Fedora for the 6.0 line. Maybe something else didn't make it into 6.1?
In that case let me update the tracking data:
#regzbot introduced: f2bd1c5ae2cb0cf95 #regzbot ignore-activity
Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat) -- Everything you wanna know about Linux kernel regression tracking: https://linux-regtracking.leemhuis.info/about/#tldr That page also explains what to do if mails like this annoy you.
linux-stable-mirror@lists.linaro.org
-
Amadeusz Sławiński -
Cezary Rojewski -
Greg KH -
Jason Montleon -
Linux kernel regression tracking (#update) -
Linux kernel regression tracking (Thorsten Leemhuis)
-
Sasa Ostrouska