Dear Lava team,
We're deploying Lava V2. So far we've been working on old servers to prototype our installation. We're now almost ready to order our definitive PCs. We have assessed some key features for our Lava server. Still, we're not 100% sure on how powerful it should be.
Is it possible for you to share the main characteristics of your current Lava servers (Number of cores, RAM, size of disk)? That would be helpful.
Thanks and best regards,
Denis Humeau
On 4 July 2016 at 10:32, Denis HUMEAU denis.humeau@st.com wrote:
Dear Lava team,
We’re deploying Lava V2. So far we’ve been working on old servers to prototype our installation. We’re now almost ready to order our definitive PCs.
You'll need to assess where those servers are struggling, whether it is simple CPU performance, RAM / swapping, storage limits or device connectivity issues.
We have assessed some key features for our Lava server. Still, we’re not 100% sure on how powerful it should be.
Is it possible for you to share the main characteristics of your current Lava servers (Number of cores, RAM, size of disk)? That would be helpful.
There are some hints in the documentation but it can largely only be hints as it depends on how that instance is going to be used.
The details will depend on:
0: how many workers are connected to each master 1: how many devices are connected to each worker 2: how many users / expected number of test jobs per day/hour etc. 3: what kinds of devices need to be supported - devices which can only be tested over USB will impact on your choice as many servers only have 1 or 2 USB host ports and USB hubs are problematic, even when powered. 4: whether you're doing any development work which would require a staging or test instance.
validation.linaro.org is a HP Proliant 8 cores with hyperthreading, 32GB RAM 4TB disk storage with separate backup storage. Workers need less RAM and less storage (e.g. 12 or 16GB and 500GB or 1TB respectively).
A developer lab may have a cubietruck (dual core Cortex-A7 2GB RAM and SATA support) which can manage ~4 devices as a worker but struggles to provide a server for more than a couple of users.
Hello Neil,
There are some hints in the documentation but it can largely only be hints as it depends on how that instance is going to be used.
The details will depend on:
0: how many workers are connected to each master 1: how many devices are connected to each worker 2: how many users / expected number of test jobs per day/hour etc. 3: what kinds of devices need to be supported - devices which can only be tested over USB will impact on your choice as many servers only have 1 or 2 USB host ports and USB hubs are problematic, even when powered.
For the usb hubs, ST boards does have an external power supply so we don't need the power-over-usb. Except this power supply issue (not applicable for ST boards), is it reliable to use a usb hub?
On 4 July 2016 at 15:41, Remi Duraffort remi.duraffort@linaro.org wrote:
Hello Neil,
There are some hints in the documentation but it can largely only be hints as it depends on how that instance is going to be used.
The details will depend on:
0: how many workers are connected to each master 1: how many devices are connected to each worker 2: how many users / expected number of test jobs per day/hour etc. 3: what kinds of devices need to be supported - devices which can only be tested over USB will impact on your choice as many servers only have 1 or 2 USB host ports and USB hubs are problematic, even when powered.
For the usb hubs, ST boards does have an external power supply so we don't need the power-over-usb. Except this power supply issue (not applicable for ST boards), is it reliable to use a usb hub?
The problems with USB hubs and the reason for requiring power to the hub (as well as to the board) is that during a board reset, the enumeration on the dispatcher / serial console server will fail / return a different UID if the hub loses power in between. The second problem is that no matter what the hub says on the box or on the hub, the actual chipset used inside will vary unpredictably and some chipsets just randomly reset. The third problem is that LAVA is testing prototype boards which may or may not have bugs in their own USB stacks (hardware and software). Put all these together and devices which only provide serial over USB can be very troublesome. It's not to power the device over USB, it's to power the hub during device reboots / disconnects. The more of these powered USB hubs get involved between the device and the dispatcher, the worse the problem becomes. Hence, if your dispatcher only has 2 USB host ports and is meant to talk serial to a dozen devices which can only provide serial over USB, things go wrong at random intervals. The Cambridge lab are experimenting with custom powered USB hubs from Cambrionix for these reasons.
Neil,
Can cubietruck be used as a dispatcher/worker in production ? So far we have used only x86 machines. Do ARM machines work as well ?
Thanks Sandeep
On Mon, Jul 4, 2016, 8:04 AM Neil Williams neil.williams@linaro.org wrote:
On 4 July 2016 at 15:41, Remi Duraffort remi.duraffort@linaro.org wrote:
Hello Neil,
There are some hints in the documentation but it can largely only be hints as it depends on how that instance is going to be used.
The details will depend on:
0: how many workers are connected to each master 1: how many devices are connected to each worker 2: how many users / expected number of test jobs per day/hour etc. 3: what kinds of devices need to be supported - devices which can only be tested over USB will impact on your choice as many servers only have 1 or 2 USB host ports and USB hubs are problematic, even when powered.
For the usb hubs, ST boards does have an external power supply so we
don't
need the power-over-usb. Except this power supply issue (not applicable
for
ST boards), is it reliable to use a usb hub?
The problems with USB hubs and the reason for requiring power to the hub (as well as to the board) is that during a board reset, the enumeration on the dispatcher / serial console server will fail / return a different UID if the hub loses power in between. The second problem is that no matter what the hub says on the box or on the hub, the actual chipset used inside will vary unpredictably and some chipsets just randomly reset. The third problem is that LAVA is testing prototype boards which may or may not have bugs in their own USB stacks (hardware and software). Put all these together and devices which only provide serial over USB can be very troublesome. It's not to power the device over USB, it's to power the hub during device reboots / disconnects. The more of these powered USB hubs get involved between the device and the dispatcher, the worse the problem becomes. Hence, if your dispatcher only has 2 USB host ports and is meant to talk serial to a dozen devices which can only provide serial over USB, things go wrong at random intervals. The Cambridge lab are experimenting with custom powered USB hubs from Cambrionix for these reasons.
--
Neil Williams
neil.williams@linaro.org http://www.linux.codehelp.co.uk/ _______________________________________________ Lava-users mailing list Lava-users@lists.linaro.org https://lists.linaro.org/mailman/listinfo/lava-users
On 7 July 2016 at 03:37, Sandeep Chawla sandeep@cyngn.com wrote:
Neil,
Can cubietruck be used as a dispatcher/worker in production ?
https://lava.debian.net/scheduler/worker/tweetypie.codehelp
This is a cubietruck (with SATA) dispatcher running in my home lab. It doesn't need to be a large SATA drive (it should be a small 2.5inch, not full size due to power requirements).
So far we have used only x86 machines. Do ARM machines work as well ?
ARMv7 devices typically end up with I/O limitations due to limited RAM and limited CPU speed. All ARMv7 boards used for dispatchers should be dual-core or better, with as much RAM as possible and will need (fast) SATA - so not SATA over USB. The number of devices supported by such a dispatcher will be less than the number used per dispatcher on validation.linaro.org. I have two devices on that cubietruck, I've previously had I/O latency issues when using 4 or more.
ARMv8 boards are going to be much better candidates than ARMv7 for dispatchers running lots of devices - the extra RAM and extra cores have a massive impact on the number of devices which can be supported without the simultaneous I/O requirements overwhelming the dispatcher when all devices are active at the same time. In validation.linaro.org, the average dispatcher is expected to cope with 20 devices, subject to limitations of USB connectivity. Devices are equally assigned across dispatchers to assist with performance and connectivity.
Thanks Sandeep
On Mon, Jul 4, 2016, 8:04 AM Neil Williams neil.williams@linaro.org wrote:
On 4 July 2016 at 15:41, Remi Duraffort remi.duraffort@linaro.org wrote:
Hello Neil,
There are some hints in the documentation but it can largely only be hints as it depends on how that instance is going to be used.
The details will depend on:
0: how many workers are connected to each master 1: how many devices are connected to each worker 2: how many users / expected number of test jobs per day/hour etc. 3: what kinds of devices need to be supported - devices which can only be tested over USB will impact on your choice as many servers only have 1 or 2 USB host ports and USB hubs are problematic, even when powered.
For the usb hubs, ST boards does have an external power supply so we don't need the power-over-usb. Except this power supply issue (not applicable for ST boards), is it reliable to use a usb hub?
The problems with USB hubs and the reason for requiring power to the hub (as well as to the board) is that during a board reset, the enumeration on the dispatcher / serial console server will fail / return a different UID if the hub loses power in between. The second problem is that no matter what the hub says on the box or on the hub, the actual chipset used inside will vary unpredictably and some chipsets just randomly reset. The third problem is that LAVA is testing prototype boards which may or may not have bugs in their own USB stacks (hardware and software). Put all these together and devices which only provide serial over USB can be very troublesome. It's not to power the device over USB, it's to power the hub during device reboots / disconnects. The more of these powered USB hubs get involved between the device and the dispatcher, the worse the problem becomes. Hence, if your dispatcher only has 2 USB host ports and is meant to talk serial to a dozen devices which can only provide serial over USB, things go wrong at random intervals. The Cambridge lab are experimenting with custom powered USB hubs from Cambrionix for these reasons.
--
Neil Williams
neil.williams@linaro.org http://www.linux.codehelp.co.uk/ _______________________________________________ Lava-users mailing list Lava-users@lists.linaro.org https://lists.linaro.org/mailman/listinfo/lava-users
-
Denis HUMEAU -
Neil Williams -
Remi Duraffort -
Sandeep Chawla