Re: [Lava-users] A deployment strategy using USB Gadget

On 28 March 2018 at 12:09, Ros Dos Santos, Alfonso (CT RDA DS EVO OPS DIA SE 1) alfonso.ros-dos-santos@evosoft.com wrote:

Hi everyone,

in the last few months my team and I have being using LAVA in our Continuous Integration process for the development of a Yocto Linux image for some development boards. These boards were not compatible with the available strategies in LAVA so we had to improvise a little.

That's fine and you have started on the right path by adding a new Strategy class.

These boards are however capable of booting from a USB device. Our idea was then to create a new deployment strategy that will download the image file into some Linux device with a OTG USB port and "expose" it using the g_mass_storage kernel module.

We have new support for devices like that - the WaRP7 support.

https://staging.validation.linaro.org/scheduler/job/211325

The OTG USB port will be connected to the test development board USB. For the booting strategy we use the already existing minimal boot where we simply power up the device and let it boot from the USB.

We would like to know your thoughts about this idea and if you see any value in these changes for a possible contribution.

There is certainly value. The best thing to do at this point is follow the documentation guidelines on contributing this upstream. Comments can be addressed within the code review system.

email-based changes are bad because there is no way to run the automation support to check each patchset.

One essential step is that all the existing unit tests succeed and that this new deployment has unit tests of it's own. This will need to include lava-server changes for the Jinja2 templates and lava-dispatcher changes to use the output of those templates.

In the board's device configuration we add the host to where download and mount the image

actions: deploy: methods: usbgadget: usb_gadget_host: {{ usb_gadget_host }}

We developed these changes on top of the lava-dispatcher verision 2017.7-1~bpo9+1 from the strerch-backports repository.

First things first, follow the docs to enable to LAVA repositories and upgrade your instance to the current LAVA production release: 2018.2 then use the developer documentation and the developer build scripts to prepare and install the latest master branch. All contributions need to be merged into the master branch.

Here is our patch with the changes:

We also added some options to apply a patch to the image boot partition to make it usb bootable, but if the image is already usb bootable it is not needed.

---

.../pipeline/actions/deploy/strategies.py | 1 + .../pipeline/actions/deploy/usbgadget.py | 254

With the removal of the V1 codebase, all these files have moved up a level - the pipeline directory no longer exists. So the patch needs to be rebased on the current git master branch.

+++++++++++++++++++++ 2 files changed, 255 insertions(+) create mode 100644 lava_dispatcher/pipeline/actions/deploy/usbgadget.py

diff --git a/lava_dispatcher/pipeline/actions/deploy/strategies.py b/lava_dispatcher/pipeline/actions/deploy/strategies.py index da1e155..cfd6438 100644 --- a/lava_dispatcher/pipeline/actions/deploy/strategies.py +++ b/lava_dispatcher/pipeline/actions/deploy/strategies.py @@ -32,3 +32,4 @@ from lava_dispatcher.pipeline.actions.deploy.lxc import Lxc from lava_dispatcher.pipeline.actions.deploy.iso import DeployIso from lava_dispatcher.pipeline.actions.deploy.nfs import Nfs from lava_dispatcher.pipeline.actions.deploy.vemsd import VExpressMsd +from lava_dispatcher.pipeline.actions.deploy.usbgadget import USBGadgetDeployment

UsbMassStorage or UsbGadget - you don't want to repeat Deployment there.

diff --git a/lava_dispatcher/pipeline/actions/deploy/usbgadget.py b/lava_dispatcher/pipeline/actions/deploy/usbgadget.py new file mode 100644 index 0000000..65347f4 --- /dev/null +++ b/lava_dispatcher/pipeline/actions/deploy/usbgadget.py @@ -0,0 +1,254 @@ +import os +import patch +import guestfs +import errno +import gzip

+from paramiko import SSHClient, AutoAddPolicy

That's adding a new dependency. We have strict requirements on which dependencies can be added according to availability on all the supported platforms. Also, we have support for making SSH connections with full support for configurable SSH options to cope with a wide variety of labs, via the existing Jinja2 templates. AutoAdd is a particularly bad idea - persistence of all kinds is to be avoided.

+from scp import SCPClient +from tempfile import mkdtemp +from shutil import rmtree

+from lava_dispatcher.pipeline.action import Pipeline, InfrastructureError, Action +from lava_dispatcher.pipeline.logical import Deployment +from lava_dispatcher.pipeline.actions.deploy import DeployAction

+from lava_dispatcher.pipeline.actions.deploy.download import (

• DownloaderAction,

+)

+from lava_dispatcher.pipeline.actions.deploy.overlay import (

• OverlayAction,

+)

+from lava_dispatcher.pipeline.actions.deploy.apply_overlay import (

• ApplyOverlayImage,

+)

+class PatchFileAction(Action):

There have been changes upstream to relocate the name, summary and description to classmethods.

• def __init__(self):

•    super(PatchFileAction, self).__init__()
  

•    self.name = "patch-image-file"
  

•    self.summary = "patch-image-file"
  

•    self.description = "patch-image-file"
  

• def run(self, connection, max_end_time, args=None):

•    connection = super(PatchFileAction, self).run(
  

•        connection, max_end_time, args)
  

•    decompressed_image = self.get_namespace_data(
  

•        action='download-action', label='image', key='file')
  

•    self.logger.debug("Image: %s", decompressed_image)
  

•    partition = self.parameters['patch'].get('partition')
  

•    if partition is None:
  

•        raise JobError(
  

•            "Unable to apply the patch, image without 'partition'")
  

•    patchfile = self.get_namespace_data(
  

•        action='download-action', label='file', key='patch')
  

•    destination = self.parameters['patch'].get('dst')
  

•    self.patch_file(decompressed_image, partition, destination,
  

patchfile)

•    return connection
  

• @staticmethod

static methods are not typically used. classmethod maybe or move the function into one of the utils/ classes. This is particularly useful as there are already utils/ classes using GuestFS

• def patch_file(image, partition, destination, patchfile):

•    """
  

•    Reads the destination file from the image, patchs it and writes
  

it back

•    to the image.
  

•    """
  

•    guest = guestfs.GuestFS(python_return_dict=True)
  

•    guest.add_drive(image)
  

•    guest.launch()
  

•    partitions = guest.list_partitions()
  

•    if not partitions:
  

•        raise InfrastructureError("Unable to prepare guestfs")
  

•    guest_partition = partitions[partition]
  

•    guest.mount(guest_partition, '/')
  

•    # create mount point
  

•    tmpd = mkdtemp()
  

•    # read the file to be patched
  

•    f_to_patch = guest.read_file(destination)
  

•    if destination.startswith('/'):
  

•        # copy the file locally
  

•        copy_dst = os.path.join(tmpd, destination[1:])
  

•    else:
  

•        copy_dst = os.path.join(tmpd, destination)
  

•    try:
  

•        os.makedirs(os.path.dirname(copy_dst))
  

•    except OSError as exc:
  

•        if exc.errno == errno.EEXIST:
  

•            pass
  

•        else:
  

•            raise
  

•    with open(copy_dst, 'w') as dst:
  

•        dst.write(f_to_patch)
  

•    # read the patch
  

•    ptch = patch.fromfile(patchfile)
  

•    # apply the patch
  

•    ptch.apply(root=tmpd)
  

•    # write the patched file back to the image
  

•    with open(copy_dst, 'r') as copy:
  

•        guest.write(destination, copy.read())
  

•    guest.umount(guest_partition)
  

•    # remove the mount point
  

•    rmtree(tmpd)
  

+class USBGadgetScriptAction(Action):

• def __init__(self, host):

•    super(USBGadgetScriptAction, self).__init__()
  

•    self.name = "deploy-usb-gadget"
  

•    self.summary = "deploy-usb-gadget"
  

•    self.description = "deploy-usb-gadget"
  

•    self.host = host
  

• def validate(self):

•    if 'deployment_data' in self.parameters:
  

•        lava_test_results_dir = self.parameters[
  

•            'deployment_data']['lava_test_results_dir']
  

•        lava_test_results_dir = lava_test_results_dir %
  

self.job.job_id

•        self.set_namespace_data(action='test', label='results',
  

•                                key='lava_test_results_dir',
  

value=lava_test_results_dir)

• def print_transfer_progress(self, filename, size, sent):

•    current_progress = (100 * sent) / size
  

•    if current_progress >= self.transfer_progress + 5:
  

•        self.transfer_progress = current_progress
  

•        self.logger.debug(
  

•            "Transferring file %s. Progress %d%%", filename,
  

current_progress)

• def run(self, connection, max_end_time, args=None):

•    connection = super(USBGadgetScriptAction, self).run(
  

•        connection, max_end_time, args)
  

•    # # Compressing the image file
  

•    uncompressed_image = self.get_namespace_data(
  

•        action='download-action', label='file', key='image')
  

•    self.logger.debug("Compressing the image %s", uncompressed_image)
  

•    compressed_image = uncompressed_image + '.gz'
  

•    with open(uncompressed_image) as f_in,
  

gzip.open(compressed_image, 'wb') as f_out:

•        f_out.writelines(f_in)
  

•    # # Try to connect to the usb gadget host
  

•    ssh = SSHClient()
  

•    ssh.set_missing_host_key_policy(AutoAddPolicy())
  

•    ssh.connect(hostname=self.host, username='root', password='')
  
  

All those must come from device configuration.

•    dest_file = os.path.join('/mnt/',
  

os.path.basename(compressed_image))

•    # # Clear /mnt folder
  

•    self.logger.debug("Clearing /mnt directory")
  

•    stdin, stdout, stderr = ssh.exec_command('rm -rf /mnt/*')
  
  

This needs to come from device configuration.

•    exit_code = stdout.channel.recv_exit_status()
  

•    if exit_code == 0:
  

•        self.logger.debug("/mnt clear")
  

•    else:
  

•        self.logger.error("Could not clear /mnt on secondary device")
  

•    # # Transfer the compressed image file
  

•    self.logger.debug(
  

•        "Transferring file %s to the usb gadget host",
  

compressed_image)

•    self.transfer_progress = 0
  

•    scp = SCPClient(ssh.get_transport(),
  

•                    progress=self.print_transfer_progress,
  

•                    socket_timeout=600.0)
  

•    scp.put(compressed_image, dest_file)
  

•    scp.close()
  

•    # # Decompress the sent image
  

•    self.logger.debug("Decompressing the file %s", dest_file)
  

•    stdin, stdout, stderr = ssh.exec_command('gzip -d %s' %
  

(dest_file))

•    exit_code = stdout.channel.recv_exit_status()
  

•    if exit_code == 0:
  

•        self.logger.debug("Decompressed file")
  

•    else:
  

•        self.logger.error("Could not decompress file: %s",
  

•                          stderr.readlines())
  

•    # # Run the g_mass_storage module
  

•    dest_file_uncompressed = dest_file[:-3]
  

•    self.logger.debug(
  

•        "Exposing the image %s as a usb storage",
  

dest_file_uncompressed)

•    stdin, stdout, stderr = ssh.exec_command('rmmod g_mass_storage')
  

•    exit_code = stdout.channel.recv_exit_status()
  

•    stdin, stdout, stderr = ssh.exec_command(
  

•        'modprobe g_mass_storage file=%s' % (dest_file_uncompressed))
  
  

Where is this being executed? DUT or dispatcher?

We already have run_command support for operations on the dispatcher, but modprobe is NOT suitable for use on the dispatcher within test jobs.

•    exit_code = stdout.channel.recv_exit_status()
  

•    if exit_code == 0:
  

•        self.logger.debug("Mounted mass storage file")
  

•    else:
  

•        self.logger.error("Could not mount file: %s",
  

•                          stderr.readlines())
  

•    ssh.close()
  

•    return connection
  

+class USBGadgetDeploymentAction(DeployAction):

• def __init__(self):

•    super(USBGadgetDeploymentAction, self).__init__()
  

•    self.name = 'usb-gadget-deploy'
  

•    self.description = "deploy images using the fake usb device"
  

•    self.summary = "deploy images"
  

• def populate(self, parameters):

•    self.internal_pipeline = Pipeline(
  

•        parent=self, job=self.job, parameters=parameters)
  

•    path = self.mkdtemp()
  

•    # Download the image
  

•    self.internal_pipeline.add_action(DownloaderAction('image',
  

path))

•    if self.test_needs_overlay(parameters):
  

•        self.internal_pipeline.add_action(OverlayAction())
  

•        self.internal_pipeline.add_action(ApplyOverlayImage())
  

•    # Patch it if needed
  

•    if 'patch' in parameters:
  

• self.internal_pipeline.add_action(DownloaderAction('patch', path))

•        self.internal_pipeline.add_action(PatchFileAction())
  

•    host = self.job.device['actions']['deploy'][
  

•        'methods']['usbgadget']['usb_gadget_host']
  

• self.internal_pipeline.add_action(USBGadgetScriptAction(host))

+class USBGadgetDeployment(Deployment):

• """
• Only for iot2000-usb
• """
• compatibility = 4
• def __init__(self, parent, parameters):

•    super(USBGadgetDeployment, self).__init__(parent)
  

•    self.priority = 1
  

•    self.action = USBGadgetDeploymentAction()
  

•    self.action.section = self.action_type
  

•    self.action.job = self.job
  

•    parent.add_action(self.action, parameters)
  

• @classmethod
• def accepts(cls, device, parameters):

•    """
  

•    Accept only iot2000-usb jobs
  

•    """
  

•    return device['device_type'] == 'iot2000-usb'
  

-- 2.7.4 _______________________________________________ Lava-users mailing list Lava-users@lists.linaro.org https://lists.linaro.org/mailman/listinfo/lava-users