Re: Master images are a mess

W dniu 07.12.2011 18:44, Paul Larson pisze:

On Wed, Dec 7, 2011 at 10:01 AM, Zygmunt Krynicki <zygmunt.krynicki@linaro.org mailto:zygmunt.krynicki@linaro.org> wrote:

Hi, sorry for the topic, I wanted to catch your attention.

This is a quick brain dump based on my own observations/battle with
master images last week.

1) Unless we use external USB/ETH adapters then cloning a master image
clones the mac address as well. This has serious consequences and I'm

This doesn't ring true. We do have different mac addresses, even on boards without flash and on-board ethernet.

How does it work? As far as I know mac address is burned in boot.scr, if you copy that (and tell me we don't) then we get duplicates.

Update: after a quick discussion on #linaro it seems that the mac address is actually burned into the hardware pack and lmc does not make one (at least not for panda). I have not verified this yet but if true then _all_ pandas with a given hwpack build get the same mac.

100% sure that's why lava-test had to be switched to the random UUID

Incorrect - I hunted down that problem. We can switch back if you really like, but I don't see any advantage to it.

What was the problem? I don't remember we actually traced that to the root cause.

mode. This problem applies to the master image mode. In the test image
the software can do anything so we may run with a random MAC or with
the mac that master images' boot loader set (we should check that).
Since making master images is a mess, unless is becomes automated I
will not be convinced that people just know how to make them properly
and are not simply copying from someone. There is no reproducible
master image creation process that ensure two people with the same
board can run a single test in a reproducible way! (different starting
rootfs/bootloader/package selection/random mistakes)

That's a pretty big exaggeration to say that it can't be done by others, or that it affects reproducibility of tests.

Try assisting others in getting master images. It takes a long while before you get from "lava installed" to "lava ran stream for the first time". The reason for that is provisioning a board is HARD and should not be.

The process isn't *that* hard. It's essentially just a nano image, a couple of extra packages installed, and add a few partitions. However, I do agree with the sentiment that this should be automated as much as possible.

2) Running code via serial on the master image is a mess. It is very
fragile. We need an agent on the board instead of a random master
image+serial shell. The agent will expose board identity, capabilities
and standard APIs to LAVA (notably the dispatcher).
The same API, if done sensibly, will work for software emulators and
hardware boards. Agent API for a software emulator can do different
things. Dispatcher should be based on agent API instead of ramming the
serial line.

This sounds like a good connect topic. It has some advantages, but also a lot of things to address.

3) The master image, as we know it today, should be booting remotely.
The boot loader can stay on the board until we can push it over USB.

em The problem is getting it to a state that we can push it over usb for every board. Not all boards support this, and the ones that do sometimes have issues with the tools to make it possible.

I don't want to push 100% over usb but pushing 99.9 (all except to boot loader) works for all boards as far as I know. This would give us controllable master image (hell we could install tests before turning the power on).

We've talked

about other solutions like a SD interface we can write from an external host over USB, then boot the board. One potential pitfall here is that this would mean we can no longer offload the lmc process with celery. It would HAVE to be done from the attached host.

I'm not proposing anything like that. I just want to keep the master rootfs + kernel away from the sd card. It is not in any way related to how we run testrootfs.

That means we are back

to serializing LMC processes, or we have a host for every single dev board!

I'm not proposing anything like that. LCM can still run anywhere we want.

The only thing that absolutely has to stay in the card is the lava
board identity file which would be generated from the web UI. There is

If that's needed, then why couldn't it be written when we deploy to the board?

It should be written once (when we create the master image) for a particular board. It should be written then and never touched before.

no reason to keep rootfs/kernel/initrd there. This means that a single
small card can fit all tests as well. It also means we can reset the
master image (as currently it is writeable by the board and can be
corrupted) before booting to ensure consistent behaviour. I did some
work on that and I managed to boot panda over NFS. Ideally I want to
boot over nbd (netblock device) which is much faster and with proper
"master image" init script we can expose a single read only net block
device to _all_ the boards.

4) With agent on each board, identity file on the SD card LAVA will
know if cloning happened. We could do dynamic board detection (unplug
the board -> it goes away, plug it back -> it shows up). We could move
a board from system to system and have 0config transitions.

Ok, you lost me here. In the last point you made, you seemed to be advocating for erasing the entire SD with the image we want to deploy.

I want to do it in stages as we have time / solve roadblocks:

1) Start with what we have but make it 100% automatic. Fix MAC issues. 2) Allow making the master rootfs empty (and very small as a partition), move to nfs or nbd for root filesystem. 3) Allow lava to manage master rootfs (create, revert to pristine) 4) Do tftp booting (kernel, initrd, device tree) so that we only keep uboot in the SD card. *) Fork a new R&D topic that will allow to put the bootloader remotely where possible. This would give us 100% remote booting. 5) Keep the cards with two files: master-boot.scr and master-uboot.bin, and an empty, small rootfs partition (for compatibility)

So once we reach this stage we have:

1) Roughly a few MB used on the SD card for any LAVA files (the boot loader and the script that tells it to do something smart). There is nothing unique to this and we can always DD the first few blocks to copy the partition layout and our boot partition over to any board. Only the identity file will need to be updated (but we don't have identity now so that's not a regression). 2) Each boot of the master image is identical apart from the current time. Lava can revert the snapshot or boot the board in read-only mode (where it will behave as a Ubuntu LTSP client). 3) We can install tests remotely if we want to. We don't even need to turn the board on. We can install the test. Boot the master image and ask it to deploy an already modified rootfs.

And there is joy and peace of mind. Everyone gets identical (perhaps even binary identical if we wipe logs and stuff like ssh host keys) master images.

5) Dispatcher should drop all configuration files. Sure it made sense
12 months ago when the idea was to run it standalone. Now all of that
configuration should be in the database and should be provided by the
scheduler to the dispatcher as a big serialized argument (or a file
descriptor or a temporary file on disk). Setting up the dispatcher for
a new instance is a pain and unless you can copy stuff from the
validation server and ask everyone around for help it's very hard to
get right. If master images could be constructed programmatically and
with a agent on each "master image" lava would just get that
configuration for free.

We should talk to our users about this though. We already *know* we have users that are using the dispatcher standalone today. I think it's possible to still move this config into the database, but we don't want to pull the rug out from under anyone without a good plan of how to get them standing again.

Sure. We can retain the old way but it's not how lava the stack should invoke it. This way we can manage this configuration in the web parts and evolve it as needed.

6) We should drop conmux. As in the lab we already have TCP/IP sockets
for the serial lines we could just provide my example serial->tcp
script as lava-serial service that people with directly attached
boards would use. We could get a similar lava-power service if that
would make sense. The lava-serial service could be started as an
instance for all USB/SERIAL adapters plugged in if we really wanted
(hello upstart!). The lava-power service would be custom and would
require some config but it is very rare. Only lab and me have
something like that. Again it should be instance based IMHO so I can
say: 'start lava-power CONF=/etc/lava-power/magic-hack.conf' and see
LAVA know about a power service. One could then say that a particular
board uses a particular serial and power services.

Another good topic for the connect I think. It needs a lot of fleshing out, but it sounds like you are basically suggesting we should recreate the functionality of conmux in lava.

Yes, but sans the perl parts and with lava integration (service subscription and more). Plus I don't expect us to hit all those conmux bugs that we cannot reasonably fix.

Conmux does two primary things for us:

1. It gives us a single interface for dealing with a variety of boards

We want a TCP socket for the serial. There are better ways of doing that.

2. It provides a safer, more convenient way of dealing with console and

power on boards.

I don't see how that is true. The console - sure, but you cannot even attempt to type there (all hell would break loose if conmux allowed this) so the usefulness is zero. The power point is true but that also integrates poorly with the rest of LAVA.

To console in to a board, and hardreset the power in

conmux: $ conmux-console panda01 $ ~$hardreset

Without it? [lookup wiki or database to find the console server/port] $ telnet console01 7001 [notice it's hung] ^] quit [lookup wiki or db to find the pdu server/port] $ telnet pdu01 [go through long menu driven interface to tell it to reset the port] ^] quit $ telnet console01 7001 [still hung... notice that you accidently reset some other board that someone was running a test on]

You don't think that was my suggestion do you?

Sure, we could provide a command line tool for looking up those things in the lava database, and give admins an easy interface to just say "take me to the console of this machine", or "hardreset this machine". If we did that, and also added attached serial multiplexing, we will have... rewritten conmux. :)

lava-server manage boardctl panda01 --reboot lava-server manage boardctl panda01 --serial-trace

The rest is done in the services, behind the scenes. You get true stuff in the web UI, you get APIs. Try building that on conmux without integration issues.

Without the bugs and with right lava integration, sounds good to me.

That beings said, I think it could still be useful because it can give us an API to call from within lava, rather than having to call out to a command line tool. However, I don't see a huge urgency for it at the moment.

I agree.

My personal view on priority is as follows:

1) Fixing master image story to a point where we can reliably build images such as the ones we have today

2) Putting identity file on the SD card, adding lava extension to manage devices (this will be the place that ultimately holds stuff like dispatcher configs, has actions to do stuff with a board).

3) Putting an agent in the master image so that we don't talk over serial ever again. Simplifying the dispatcher, coming up with a board API. Exposing the stuff in dashboard UI as sensible.

4) Doing more work on remote booting so that we can minimize space usage on the SD card and have immutable master image (that starts exactly the same on each boot because we control the read-only rootfs, initrd, kernel, device tree from the lava-server parts).

Best regards Zygmunt