On Thu, Oct 30, 2014 at 1:18 PM, Jens Wiklander <jens.wiklander@linaro.org> wrote:


On Thu, Oct 30, 2014 at 12:22 PM, Ard Biesheuvel <ard.biesheuvel@linaro.org> wrote:
On 30 October 2014 11:31, Ard Biesheuvel <ard.biesheuvel@linaro.org> wrote:
> On 30 October 2014 11:20, Jens Wiklander <jens.wiklander@linaro.org> wrote:
>> Hi toolchain champions,
>>
>> [please keep me in cc as I'm not subscribed to
>> linaro-toolchain@lists.linaro.org]
>>
>> In OP-TEE we are going to activate a pager which is an integrated part of
>> the statically linked secure OS binary (compiled for ARMv7/Aarch32 now, but
>> at some point also Aarch64).
>>
>> The pager in OP-TEE allows use of more memory than the amount of available
>> physical memory. This makes it possible to for instance have an OP-TEE
>> binary that requires more memory than the amount of available memory. What
>> the pager does is to map a physical page at the virtual address where the
>> memory is needed and populate it which what is expected on demand. The pager
>> also unmaps physical pages that hasn't been used in a while to be able to
>> recycle it.
>>
>> The code used by the pager to map and populate a page must always be mapped
>> since we would otherwise get a deadlock. The problem is that the pager is
>> also part of OP-TEE so we need to partition the binary in a way that all
>> code needed to handle a page fault is in one area in the binary and always
>> mapped.
>>
>> Annotating functions and such as it's done in the Linux kernel with __init
>> will not scale here since the pager will need routines from "third-party"
>> libraries. We can make small changes to the libraries but identifying and
>> annotating everything needed by the pager is too much. We would also run
>> into troubles with strings.
>>
>> I have a couple ideas below that I need help exploring.
>>
>> What if we do an incremental linking of the entire TEE Core with garbage
>> collect only keeping the entry functions of the pager? Then we would get an
>> object file with everything the pager depends on included but not much more.
>> It would be easy to put this single object file in a separate part of the
>> OP-TEE binary. The procedure would be something like:
>>
>> Compile everything with -ffunction-sections -fdata-sections
>> ld -i --gc-sections -u pager_entry -o pager_code.o $(objs) $(link-ldadd)
>> $(libgcc)
>> ld $(link-ldflags) pager_code.o $(objs) $(link-ldadd) $(libgcc)
>>
>> But the problem comes with linking in the rest of the code in the last step,
>> we would get lots of multiple defined symbols. We could create a
>> libtee_core.a file where we put all the $(objs) files and the linker would
>> only use the needed object files. But we would still have some multiple
>> defined symbols left since each .o file contains more than just one section.
>>
>> Any ideas how to solve this?
>>


After lots of experimentation I've finally found something that seems to work. What I do is something like:
Compile everything with -ffunction-sections -fdata-sections
ld -i --gc-sections -u pager_entry -o pager_code.o $(objs) $(link-ldadd) $(libgcc)
to get all required sections. Then I have a script generating a list of all the sections in a format suitable to include in our linker script. After that I link the OP-TEE binary as usual and voila, all pager related sections are in one area.

There's one small problem though, it seems that even if I compile everything with "-ffunction-sections -fdata-sections" there's still some stuff emitted in .rodata.
It seems that for instance:
printf("Hello world");
would result in putting the string "Hello world" in .rodata. Any ideas how to get these constant strings in one .rodata.something each instead?

Regards,
Jens