On Wed, Dec 7, 2011 at 10:01 AM, Zygmunt Krynicki <zygmunt.krynicki@linaro.org> wrote:
Hi, sorry for the topic, I wanted to catch your attention.
This is a quick brain dump based on my own observations/battle with
master images last week.
1) Unless we use external USB/ETH adapters then cloning a master image
clones the mac address as well. This has serious consequences and I'm
This doesn't ring true. We do have different mac addresses, even on boards without flash and on-board ethernet.
100% sure that's why lava-test had to be switched to the random UUID
Incorrect - I hunted down that problem. We can switch back if you really like, but I don't see any advantage to it.
mode. This problem applies to the master image mode. In the test image
the software can do anything so we may run with a random MAC or with
the mac that master images' boot loader set (we should check that).
Since making master images is a mess, unless is becomes automated I
will not be convinced that people just know how to make them properly
and are not simply copying from someone. There is no reproducible
master image creation process that ensure two people with the same
board can run a single test in a reproducible way! (different starting
rootfs/bootloader/package selection/random mistakes)
That's a pretty big exaggeration to say that it can't be done by others, or that it affects reproducibility of tests. The process isn't *that* hard. It's essentially just a nano image, a couple of extra packages installed, and add a few partitions. However, I do agree with the sentiment that this should be automated as much as possible.
2) Running code via serial on the master image is a mess. It is very
fragile. We need an agent on the board instead of a random master
image+serial shell. The agent will expose board identity, capabilities
and standard APIs to LAVA (notably the dispatcher).
The same API, if done sensibly, will work for software emulators and
hardware boards. Agent API for a software emulator can do different
things. Dispatcher should be based on agent API instead of ramming the
serial line.
This sounds like a good connect topic. It has some advantages, but also a lot of things to address.
3) The master image, as we know it today, should be booting remotely.
The boot loader can stay on the board until we can push it over USB.
The problem is getting it to a state that we can push it over usb for every board. Not all boards support this, and the ones that do sometimes have issues with the tools to make it possible. We've talked about other solutions like a SD interface we can write from an external host over USB, then boot the board. One potential pitfall here is that this would mean we can no longer offload the lmc process with celery. It would HAVE to be done from the attached host. That means we are back to serializing LMC processes, or we have a host for every single dev board!
The only thing that absolutely has to stay in the card is the lava
board identity file which would be generated from the web UI. There is
If that's needed, then why couldn't it be written when we deploy to the board?
no reason to keep rootfs/kernel/initrd there. This means that a single
small card can fit all tests as well. It also means we can reset the
master image (as currently it is writeable by the board and can be
corrupted) before booting to ensure consistent behaviour. I did some
work on that and I managed to boot panda over NFS. Ideally I want to
boot over nbd (netblock device) which is much faster and with proper
"master image" init script we can expose a single read only net block
device to _all_ the boards.
4) With agent on each board, identity file on the SD card LAVA will
know if cloning happened. We could do dynamic board detection (unplug
the board -> it goes away, plug it back -> it shows up). We could move
a board from system to system and have 0config transitions.
Ok, you lost me here. In the last point you made, you seemed to be advocating for erasing the entire SD with the image we want to deploy.
5) Dispatcher should drop all configuration files. Sure it made sense
12 months ago when the idea was to run it standalone. Now all of that
configuration should be in the database and should be provided by the
scheduler to the dispatcher as a big serialized argument (or a file
descriptor or a temporary file on disk). Setting up the dispatcher for
a new instance is a pain and unless you can copy stuff from the
validation server and ask everyone around for help it's very hard to
get right. If master images could be constructed programmatically and
with a agent on each "master image" lava would just get that
configuration for free.
We should talk to our users about this though. We already *know* we have users that are using the dispatcher standalone today. I think it's possible to still move this config into the database, but we don't want to pull the rug out from under anyone without a good plan of how to get them standing again.
6) We should drop conmux. As in the lab we already have TCP/IP sockets
for the serial lines we could just provide my example serial->tcp
script as lava-serial service that people with directly attached
boards would use. We could get a similar lava-power service if that
would make sense. The lava-serial service could be started as an
instance for all USB/SERIAL adapters plugged in if we really wanted
(hello upstart!). The lava-power service would be custom and would
require some config but it is very rare. Only lab and me have
something like that. Again it should be instance based IMHO so I can
say: 'start lava-power CONF=/etc/lava-power/magic-hack.conf' and see
LAVA know about a power service. One could then say that a particular
board uses a particular serial and power services.
Another good topic for the connect I think. It needs a lot of fleshing out, but it sounds like you are basically suggesting we should recreate the functionality of conmux in lava. Conmux does two primary things for us:
1. It gives us a single interface for dealing with a variety of boards
2. It provides a safer, more convenient way of dealing with console and power on boards. To console in to a board, and hardreset the power in conmux:
$ conmux-console panda01
$ ~$hardreset
Without it?
[lookup wiki or database to find the console server/port]
$ telnet console01 7001
[notice it's hung]
^] quit
[lookup wiki or db to find the pdu server/port]
$ telnet pdu01
[go through long menu driven interface to tell it to reset the port]
^] quit
$ telnet console01 7001
[still hung... notice that you accidently reset some other board that someone was running a test on]
Sure, we could provide a command line tool for looking up those things in the lava database, and give admins an easy interface to just say "take me to the console of this machine", or "hardreset this machine". If we did that, and also added attached serial multiplexing, we will have... rewritten conmux. :)
That beings said, I think it could still be useful because it can give us an API to call from within lava, rather than having to call out to a command line tool. However, I don't see a huge urgency for it at the moment.
Thanks,
Paul Larson