Hi all!
I am working with some Raspberry Pi boards and, while defining some particularities for booting via NFS, I came across an issue regarding the way the rootfs file is unpacked. To be more specific, after looking through the LAVA code, it seems that the untar_file() method runs on the conditional branch that states the tar members are specified. I do not understand how this implementation is designed in regards to tar archive manipulation, but the following scenario takes place:
* The rootfs archive we specify in the job definition is copied and renamed, the extension being changed from .tar.bz2 to a plain .tar (which seems a bit strange tom e, on its own) * Then, after unpacking to the LAVA temp job dir, only two directories are extracted from the whole rootfs archive. I did manage to „inverveene” while the job was running and looking in the temp dir to see exactly what gets extracted. Since the rest of the folders from the rootfs are not available, the job fails once the system starts booting
While investigating, I noticed that the untarf_file() method is invoked from download.py (/usr/lib/python2.7/dist-packages/lava_dispatcher/pipeline/actions/deploy/) - @ line 344 – with members being specified. I know that, because when I manually changed how this call is made and clearly specified “None” for the “member” positional argument, I get the following error at job runtime: https://paste.debian.net/1001252/ (the full job log can be found here: https://paste.debian.net/1001253/ )
The code changes I am talking about (and the code where I suspect something is ambiguous) are pointed out here: https://paste.debian.net/1001256/ . A job definition we use for this board integration can be analyzed here: https://paste.debian.net/1001257/
The full log of the initial job, that ran without my code changes, can be found here: https://paste.debian.net/1001254/ . The „kernel panic” message ocurs because, as stated in one of the errors, the “init” folder is missing, which is accurate, because only bin & dev are extracted from our rootfs tar.bz2 archive.
What could we do? Is this something that needs to be adjusted/fixed in LAVA?
Kind regards, Dragoș
I forgot mentioning that we use LAVA 2017.10-1.
From: Lava-users [mailto:lava-users-bounces@lists.linaro.org] On Behalf Of Dragos Iorgulescu Sent: Monday, December 18, 2017 12:47 PM To: lava-users@lists.linaro.org Subject: [Lava-users] Unpacking tar.bz2 root filesystems seems to malfunction
Hi all!
I am working with some Raspberry Pi boards and, while defining some particularities for booting via NFS, I came across an issue regarding the way the rootfs file is unpacked. To be more specific, after looking through the LAVA code, it seems that the untar_file() method runs on the conditional branch that states the tar members are specified. I do not understand how this implementation is designed in regards to tar archive manipulation, but the following scenario takes place:
* The rootfs archive we specify in the job definition is copied and renamed, the extension being changed from .tar.bz2 to a plain .tar (which seems a bit strange tom e, on its own) * Then, after unpacking to the LAVA temp job dir, only two directories are extracted from the whole rootfs archive. I did manage to „inverveene” while the job was running and looking in the temp dir to see exactly what gets extracted. Since the rest of the folders from the rootfs are not available, the job fails once the system starts booting
While investigating, I noticed that the untarf_file() method is invoked from download.py (/usr/lib/python2.7/dist-packages/lava_dispatcher/pipeline/actions/deploy/) - @ line 344 – with members being specified. I know that, because when I manually changed how this call is made and clearly specified “None” for the “member” positional argument, I get the following error at job runtime: https://paste.debian.net/1001252/https://url10.mailanyone.net/v1/?m=1eQsxY-0006tI-52&i=57e1b682&c=I4FXDEPBzsyMKM7czyB5dUescfmbezxbX419rfkW1M853SjHRcLmcQC3irth51XXmc6Shiqtu6rTzH4HTpka2oe61_j8APT6Hmv2KbTD1AcNwuEpDwFRDXbX_Ghz004MJl0LAmA9T1urJDfTp7pdT9xAiApewgHZo9-jMCUyjnPnhMNuufWFJ11G4MtHRnZUDqYV8Qcrjlkpl5oi-DTZiM73orhcSJxRc6q0R7-H1M8 (the full job log can be found here: https://paste.debian.net/1001253/https://url10.mailanyone.net/v1/?m=1eQsxY-0006tI-52&i=57e1b682&c=A8GVVtE9YH-daa12E-142CL-zdSTEVD30HeXcHD5NZ9ptLTiJ2OgKafAA5MJPbg6m5np7vxihC5zqOnzBm5O75aAKoCGNr3QdaBikNYXdWfDEuYUsxAVUHEdiyJF-iZNnPX-2nTYUdAHJWcwiw1ccrojI_a6d6OphIFWC0p4MAmgsZJYykcGeOndvvq5dJ5f96NP1ZCSrAHJIVTYEFwd1ke2SmtpgqUB1fYSGRQ_8LA )
The code changes I am talking about (and the code where I suspect something is ambiguous) are pointed out here: https://paste.debian.net/1001256/https://url10.mailanyone.net/v1/?m=1eQsxY-0006tI-52&i=57e1b682&c=t4dnV7iaOZUYlQU3cBixGPf743P2bUdH4pdPVHc6AElGD5QPsWMYPvHZEfBjUv4hJMJAVE699K76VuHeZAGZn4fMHB7DQQpcOW9iwhir94D-0AOR2bWGy6MXO0bfojiSOH2ISmqp9XM-XSiuXLI_CuKP7du_qe7neaWrkGmHa6FY2VfAPH0bL8ENrTxJ32fqPfXxGVF-dPXAMDVzoDu_bb31v5VmdbK5AkgQOyhV8yI . A job definition we use for this board integration can be analyzed here: https://paste.debian.net/1001257/https://url10.mailanyone.net/v1/?m=1eQsxY-0006tI-52&i=57e1b682&c=brW-sVo8LJ5PQ5NR-DDWJqqk9GRFySMnINzObFEScBaK2DHIIaGOdduotjZ_yrTbqsJC5SnrYBEvtNUii9T2zeBfQ1Uh0Yesgj1Z7uhvjYQCdgy1E9PmmXqxvD1RBIdRr29waadhSef0iRrOSqqayVwoTALZXgs4eUXMN2mdPiP6AuMw-vp4hBvgclj7m0WkWWdyKbkgjCivSCRvxzPT9pOF4xHcVB16X8hbWbcfIIs
The full log of the initial job, that ran without my code changes, can be found here: https://paste.debian.net/1001254/https://url10.mailanyone.net/v1/?m=1eQsxY-0006tI-52&i=57e1b682&c=JQaVFlVuTDCDSPdmGA3ruiE8BUQhlMVA7R56mjn87e6KNL5UWe5Y72O-_0kSziYYiURgvSk69FxcM1pGGw4XrPQs_FPXvsk04aa5CVM0uNsbi3oOqsiNRHYxTg3xrEUYSTOxGY9R3zdkuT4IlaSqBydQ1TZtnkU8H74vb4fX_jEG9mUtu0WGsdoLIbwe2yzeZPQXDX2_DS9Fe0HCPIKKfgew_vcJ6jc5bOiUk0eIWLw . The „kernel panic” message ocurs because, as stated in one of the errors, the “init” folder is missing, which is accurate, because only bin & dev are extracted from our rootfs tar.bz2 archive.
What could we do? Is this something that needs to be adjusted/fixed in LAVA?
Kind regards, Dragoș
Hi,
On Monday 18 December 2017 04:17 PM, Dragos Iorgulescu wrote:
The code changes I am talking about (and the code where I suspect something is ambiguous) are pointed out here: https://paste.debian.net/1001256/ .
A job definition we use for this board integration can be analyzed here: https://paste.debian.net/1001257/
LAVA expects both the compression format and the archive format specified. In your case as per your job definition the following must be specified:
https://paste.debian.net/1001400/
Pay attention to line 38 ie., 'archive: tar'.
Thank You.
Hi!
Thanks for the answer. Just tried that, and here is the output: - https://paste.debian.net/1001409/ (plain log) - https://paste.debian.net/1001411/ (job def)
What I don't understand is the way the archive is "saved":
Using bz2 decompression downloading file:///tftpboot/eltf/standard/Enea_Linux_7.0_raspberrypi3-64_enea-image-standard_2017-12-14_16-47-18/images/raspberrypi3-64/enea-image-standard-raspberrypi3-64.tar.bz2 saving as /var/lib/lava/dispatcher/tmp/525/tftp-deploy-jBJXcy/nfsrootfs/enea-image-standard-raspberrypi3-64.tar
Kind regards, Dragoș
-----Original Message----- From: Senthil Kumaran S [mailto:senthil.kumaran@linaro.org] Sent: Tuesday, December 19, 2017 8:07 AM To: Dragos Iorgulescu Dragos.Iorgulescu@enea.com; lava-users@lists.linaro.org Subject: Re: [Lava-users] Unpacking tar.bz2 root filesystems seems to malfunction
Hi,
On Monday 18 December 2017 04:17 PM, Dragos Iorgulescu wrote:
The code changes I am talking about (and the code where I suspect something is ambiguous) are pointed out here: https://paste.debian.net/1001256/ .
A job definition we use for this board integration can be analyzed here: https://paste.debian.net/1001257/
LAVA expects both the compression format and the archive format specified. In your case as per your job definition the following must be specified:
https://paste.debian.net/1001400/
Pay attention to line 38 ie., 'archive: tar'.
Thank You. -- Senthil Kumaran S http://www.stylesen.org/ http://www.sasenthilkumaran.com/
What I don't understand is the way the archive is "saved":
Using bz2 decompression downloading file:///tftpboot/eltf/standard/Enea_Linux_7.0_ raspberrypi3-64_enea-image-standard_2017-12-14_16-47-18/ images/raspberrypi3-64/enea-image-standard-raspberrypi3-64.tar.bz2 saving as /var/lib/lava/dispatcher/tmp/525/tftp-deploy-jBJXcy/ nfsrootfs/enea-image-standard-raspberrypi3-64.tar
Just to answser this question: the archive is decompressed on the fly (while downloading) so the resulting file is a ".tar".
Well, considering that, why is it still treated as a bz2 archive in the next step?
Because the next operation ends up with the following error: Unable to unpack /var/lib/lava/dispatcher/tmp/525/tftp-deploy-jBJXcy/nfsrootfs/enea-image-standard-raspberrypi3-64.tar: not a bzip2 file
Regards, Dragoș
From: Remi Duraffort [mailto:remi.duraffort@linaro.org] Sent: Tuesday, December 19, 2017 10:34 AM To: Dragos Iorgulescu Dragos.Iorgulescu@enea.com Cc: senthil.kumaran@linaro.org; lava-users@lists.linaro.org Subject: Re: [Lava-users] Unpacking tar.bz2 root filesystems seems to malfunction
What I don't understand is the way the archive is "saved":
Using bz2 decompression downloading file:///tftpboot/eltf/standard/Enea_Linux_7.0_raspberrypi3-64_enea-image-standard_2017-12-14_16-47-18/images/raspberrypi3-64/enea-image-standard-raspberrypi3-64.tar.bz2 saving as /var/lib/lava/dispatcher/tmp/525/tftp-deploy-jBJXcy/nfsrootfs/enea-image-standard-raspberrypi3-64.tar
Just to answser this question: the archive is decompressed on the fly (while downloading) so the resulting file is a ".tar".
-- Rémi Duraffort LAVA Team
On Tuesday 19 December 2017 02:14 PM, Dragos Iorgulescu wrote:
Well, considering that, why is it still treated as a bz2 archive in the next step?
Can you cross check if your tar.bz2 file is intact, by extracting it manually?
I tried an example with a Nexus4 device with the job definition - http://paste.debian.net/1001456/ and got job completed without any issues as seen in http://paste.debian.net/1001457/
Thank You.
Hi Kumaran,
Yes, I did extract manually in the first place (I did mention that in the original e-mail - I hope I am not mistaking). The archive is intact and all it's contents can be successfully extracted. As a matter a fact, I did extract the contents to a TFTP server and manually started the board using the same directives we specify in LAVA, and it worked flawlessly.
In case it matters, the archive is successfully extracted using "tar xjf archive_name.tar.bz2".
So, where could the problem be? All ideas & options are exhausted, as far as we are concerned (for now)...
Regards, Dragoș
-----Original Message----- From: Senthil Kumaran S [mailto:senthil.kumaran@linaro.org] Sent: Tuesday, December 19, 2017 3:47 PM To: Dragos Iorgulescu Dragos.Iorgulescu@enea.com; Remi Duraffort remi.duraffort@linaro.org Cc: lava-users@lists.linaro.org Subject: Re: [Lava-users] Unpacking tar.bz2 root filesystems seems to malfunction
On Tuesday 19 December 2017 02:14 PM, Dragos Iorgulescu wrote:
Well, considering that, why is it still treated as a bz2 archive in the next step?
Can you cross check if your tar.bz2 file is intact, by extracting it manually?
I tried an example with a Nexus4 device with the job definition - http://paste.debian.net/1001456/ and got job completed without any issues as seen in http://paste.debian.net/1001457/
Thank You. -- Senthil Kumaran S http://www.stylesen.org/ http://www.sasenthilkumaran.com/
On Tue, Dec 19, 2017 at 01:58:49PM +0000, Dragos Iorgulescu wrote:
Hi Kumaran,
Yes, I did extract manually in the first place (I did mention that in the original e-mail - I hope I am not mistaking). The archive is intact and all it's contents can be successfully extracted. As a matter a fact, I did extract the contents to a TFTP server and manually started the board using the same directives we specify in LAVA, and it worked flawlessly.
In case it matters, the archive is successfully extracted using "tar xjf archive_name.tar.bz2".
So, where could the problem be? All ideas & options are exhausted, as far as we are concerned (for now)...
Hi,
I don't know if this is relevant here, but I know there's still an ongoing problem with bzip2-compressed files compressed in parallel using pbzip2 - they're not supported properly when decompressing using the python bz2 module. Could that be the feault here?
Cheers,
Hi Steve!
I was kinda thinking the same thing in regards to the bz2 py module, but I am not aware of any ongoing problems. This is also one of the reasons I opened this thread 😊
Regards, Dragos
-----Original Message----- From: Steve McIntyre [mailto:steve.mcintyre@linaro.org] Sent: Tuesday, December 19, 2017 4:34 PM To: Dragos Iorgulescu Dragos.Iorgulescu@enea.com Cc: senthil.kumaran@linaro.org; Remi Duraffort remi.duraffort@linaro.org; lava-users@lists.linaro.org Subject: Re: [Lava-users] Unpacking tar.bz2 root filesystems seems to malfunction
On Tue, Dec 19, 2017 at 01:58:49PM +0000, Dragos Iorgulescu wrote:
Hi Kumaran,
Yes, I did extract manually in the first place (I did mention that in the original e-mail - I hope I am not mistaking). The archive is intact and all it's contents can be successfully extracted. As a matter a fact, I did extract the contents to a TFTP server and manually started the board using the same directives we specify in LAVA, and it worked flawlessly.
In case it matters, the archive is successfully extracted using "tar xjf archive_name.tar.bz2".
So, where could the problem be? All ideas & options are exhausted, as far as we are concerned (for now)...
Hi,
I don't know if this is relevant here, but I know there's still an ongoing problem with bzip2-compressed files compressed in parallel using pbzip2 - they're not supported properly when decompressing using the python bz2 module. Could that be the feault here?
Cheers,
Hi,
On Tuesday 19 December 2017 08:09 PM, Dragos Iorgulescu wrote:
I was kinda thinking the same thing in regards to the bz2 py module, but I am not aware of any ongoing problems. This is also one of the reasons I opened this thread 😊
I just gave a try with files compressed using pbzip2, See http://paste.debian.net/1001471/ - the checksum doesn't match.
The job did failed, but with a different error than what was reported initially as seen in http://paste.debian.net/1001472/
Thank You.
Is it possible to use another compression format, in order to bypass this? I just tried doing that and used the rootfs tar repacked as a gzip file. Even if if specify "compression: gz" in the job definition, it appears that LAVA expects this rootfs to be a bzip2 archive...
https://paste.debian.net/1001483/ -> plain log https://paste.debian.net/1001485/ -> job definition
Mind you that specifying the "archive: tar" attribute doesn't make a difference (tried that already)
-----Original Message----- From: Senthil Kumaran S [mailto:senthil.kumaran@linaro.org] Sent: Tuesday, December 19, 2017 5:00 PM To: Dragos Iorgulescu Dragos.Iorgulescu@enea.com; Steve McIntyre steve.mcintyre@linaro.org Cc: Remi Duraffort remi.duraffort@linaro.org; lava-users@lists.linaro.org Subject: Re: [Lava-users] Unpacking tar.bz2 root filesystems seems to malfunction
Hi,
On Tuesday 19 December 2017 08:09 PM, Dragos Iorgulescu wrote:
I was kinda thinking the same thing in regards to the bz2 py module, but I am not aware of any ongoing problems. This is also one of the reasons I opened this thread 😊
I just gave a try with files compressed using pbzip2, See http://paste.debian.net/1001471/ - the checksum doesn't match.
The job did failed, but with a different error than what was reported initially as seen in http://paste.debian.net/1001472/
Thank You. -- Senthil Kumaran S http://www.stylesen.org/ http://www.sasenthilkumaran.com/
On Tue, Dec 19, 2017 at 03:29:46PM +0000, Dragos Iorgulescu wrote:
Is it possible to use another compression format, in order to bypass this? I just tried doing that and used the rootfs tar repacked as a gzip file. Even if if specify "compression: gz" in the job definition, it appears that LAVA expects this rootfs to be a bzip2 archive...
https://paste.debian.net/1001483/ -> plain log https://paste.debian.net/1001485/ -> job definition
Mind you that specifying the "archive: tar" attribute doesn't make a difference (tried that already)
Right. We're using the python tarfile module, and that's where the error is coming from, e.g. in line 84 of your log. That's from lava_dispatcher/utils/compression.py:untar_file():
except tarfile.TarError as exc: raise JobError("Unable to unpack %s: %s" % (infile, str(exc)))
the tarfile module includes built-in support for gz and bz2 compression and decompression, and by default will attempt to auto-decompress things transparently. Maybe it's being confused by something in your rootfs. It's not double-compressed (or similar?), I hope? You're referencing a file: url so I can't grab it to take a look here.
Maybe we should be tarfile.open() with mode='r:' to turn off the auto-decompress here, as we've already decompressed it during the download...
Cheers,
This change could be attempted in compression.py, directly? I am currently trying anything I can think of in order to get past this.
-----Original Message----- From: Steve McIntyre [mailto:steve.mcintyre@linaro.org] Sent: Tuesday, December 19, 2017 5:53 PM To: Dragos Iorgulescu Dragos.Iorgulescu@enea.com Cc: senthil.kumaran@linaro.org; Remi Duraffort remi.duraffort@linaro.org; lava-users@lists.linaro.org Subject: Re: [Lava-users] Unpacking tar.bz2 root filesystems seems to malfunction
On Tue, Dec 19, 2017 at 03:29:46PM +0000, Dragos Iorgulescu wrote:
Is it possible to use another compression format, in order to bypass this? I just tried doing that and used the rootfs tar repacked as a gzip file. Even if if specify "compression: gz" in the job definition, it appears that LAVA expects this rootfs to be a bzip2 archive...
https://paste.debian.net/1001483/ -> plain log https://paste.debian.net/1001485/ -> job definition
Mind you that specifying the "archive: tar" attribute doesn't make a difference (tried that already)
Right. We're using the python tarfile module, and that's where the error is coming from, e.g. in line 84 of your log. That's from lava_dispatcher/utils/compression.py:untar_file():
except tarfile.TarError as exc: raise JobError("Unable to unpack %s: %s" % (infile, str(exc)))
the tarfile module includes built-in support for gz and bz2 compression and decompression, and by default will attempt to auto-decompress things transparently. Maybe it's being confused by something in your rootfs. It's not double-compressed (or similar?), I hope? You're referencing a file: url so I can't grab it to take a look here.
Maybe we should be tarfile.open() with mode='r:' to turn off the auto-decompress here, as we've already decompressed it during the download...
Cheers,
-
Dragos Iorgulescu -
Remi Duraffort -
Senthil Kumaran S -
Steve McIntyre