To be frank, *absolutely do not touch U-Boot*. Linaro does not offer support for either the Smartbook in U-Boot (it's not even been pushed to mainline yet) and I am fairly certain that the Smartbook in the kernel is not supported enough to get a reasonably working system. There's a very good chance you will brick something and not be able to recover it.
You say you have USB-serial, does that mean you have the little flat serial cable from under the keyboard and dongle attached? I don't understand what you mean by having a small ramfs with USB-serial support (debug support is not via USB, it's a special debug board. I know this is not terribly friendly, but it is how it is).
Serial settings are 115200,8n1 if not set explicitly I would start there (sometimes when rebooting systems will put some odd characters from the FIFO and it messes up speed detection). You probably need to make sure serial support is compiled into the kernel - check the config. It may not be! The Linaro kernel MAY have enough from the linux-imx branch to support the Smartbook but it may also not be configured correctly.
Please tell us your kernel boot arguments, give me a pointer to a git repository or commit of a Linaro branch I can look at to confirm what you're running.