On 13 July 2016 at 10:35, Al Grant Al.Grant@arm.com wrote:
Hi,
When you see the libraries being mapped multiple times, are you just seeing the code and data segments? I see that too, I just ignore the data segments.
(Taking the liberty of CC'ing the list as this is probably a topic of interest)
Each time a library is mapped perf gets notified by the mm subsystem. Part of the notification is a new vm_area_struct that contains the new start address of the library (vm_area_struct::vm_start). Upon receiving the notification the new address is communicated to the ETM drivers which do the required filter configuration. That is all good and working well.
On ARM64 (because I _assume_ X86 folks didn't see this) we get 3 notifications. For example notification A will have address 0x7f93a60000 while, subsequently, notification B and C address 0x7f93a70000. Note that the latter two are 64K higher than the first one.
Once the last notification has been received the code in the main program is executed. That code (in the main program) jumps to library code mapped at the address it got from the first notification and not the last one, making the filter configuration all wrong.
As such I have to understand what notification B and C are for. Based on the vm_area_struct::vm_flags I'm guessing some sort of accounting feature but not sure yet. If I ignore notification B and C, things work amazingly well and one can really see the power offered by coresight.
That's where I'm at now.
Get back to me if you (or anyone else) want more information.
Mathieu
Al IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
On ARM64 (because I _assume_ X86 folks didn't see this) we get 3 notifications. For example notification A will have address 0x7f93a60000 while, subsequently, notification B and C address 0x7f93a70000. Note that the latter two are 64K higher than the first one.
Once the last notification has been received the code in the main program is executed. That code (in the main program) jumps to library code mapped at the address it got from the first notification and not the last one, making the filter configuration all wrong.
As such I have to understand what notification B and C are for. Based on the vm_area_struct::vm_flags I'm guessing some sort of accounting feature but not sure yet.
Code and data segment(s)? They will be mapped in separately with separate permissions. If you look at /proc/self/maps or /proc/<pid>/maps you will see 2 or 3 entries per library, depending on system.
vm_pgoff will indicate the file offset within the ELF file. This will tend to be zero for the main code segment and non-zero for data segment(s).
vm_flags will indicate the segment permissions. You would normally get at least one without VM_WRITE and at least one with VM_WRITE. You might get R+X, R+W+X, R+W, just R etc.
Something worth knowing here is that you're more likely to see VM_EXEC on data sections on ARM64 than on X86_64, because the READ_MEANS_EXEC personality bit is inherited in the process tree on ARM64 and not X86_64. At worst, if your init process has an executable stack then practically every data segment mapping in the system gets VM_EXEC! You can't assume that you will only get one VM_EXEC mapping and that it will be "the" main code section; or act as if this was true by treating your last VM_EXEC notification as the code section. If you want to pick a single code section then use the one with vm_pgoff set to zero.
Again this is shows up in /proc/xxx/maps - if you see 'x's on writeable data segments and even on non executable files like /usr/lib/locale/locale-archive, then you've got sticky VM_EXEC. It's a bug really, I keep meaning to write a patch for this.
Could you dump out what you're seeing, complete with vm_flags and vm_pgoff, and also /proc/pid/maps? Hopefully things will then be clear.
Al
If I ignore notification B and C, things work amazingly well and one can really see the power offered by coresight.
That's where I'm at now.
Get back to me if you (or anyone else) want more information.
Mathieu
Al IMPORTANT NOTICE: The contents of this email and any attachments are
confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
On 14 July 2016 at 05:16, Al Grant Al.Grant@arm.com wrote:
On ARM64 (because I _assume_ X86 folks didn't see this) we get 3 notifications. For example notification A will have address 0x7f93a60000 while, subsequently, notification B and C address 0x7f93a70000. Note that the latter two are 64K higher than the first one.
Once the last notification has been received the code in the main program is executed. That code (in the main program) jumps to library code mapped at the address it got from the first notification and not the last one, making the filter configuration all wrong.
As such I have to understand what notification B and C are for. Based on the vm_area_struct::vm_flags I'm guessing some sort of accounting feature but not sure yet.
Code and data segment(s)? They will be mapped in separately with separate permissions. If you look at /proc/self/maps or /proc/<pid>/maps you will see 2 or 3 entries per library, depending on system.
vm_pgoff will indicate the file offset within the ELF file. This will tend to be zero for the main code segment and non-zero for data segment(s).
vm_flags will indicate the segment permissions. You would normally get at least one without VM_WRITE and at least one with VM_WRITE. You might get R+X, R+W+X, R+W, just R etc.
Something worth knowing here is that you're more likely to see VM_EXEC on data sections on ARM64 than on X86_64, because the READ_MEANS_EXEC personality bit is inherited in the process tree on ARM64 and not X86_64. At worst, if your init process has an executable stack then practically every data segment mapping in the system gets VM_EXEC! You can't assume that you will only get one VM_EXEC mapping and that it will be "the" main code section; or act as if this was true by treating your last VM_EXEC notification as the code section. If you want to pick a single code section then use the one with vm_pgoff set to zero.
Again this is shows up in /proc/xxx/maps - if you see 'x's on writeable data segments and even on non executable files like /usr/lib/locale/locale-archive, then you've got sticky VM_EXEC. It's a bug really, I keep meaning to write a patch for this.
Could you dump out what you're seeing, complete with vm_flags and vm_pgoff, and also /proc/pid/maps? Hopefully things will then be clear.
That is very helpful, thanks for taking the time to write all this. Ok, I'll mofidy the example code (that I will also send) to execute for a little while - that way I can probe /proc/xyz/maps and correlate what's found there with the mapped addresses I get from the mm subsystem.
But it might not be today (I'll try really hard). The perf user space code no longer builds on the offical perf tree and I need to fix it right away.
Mathieu
Al
If I ignore notification B and C, things work amazingly well and one can really see the power offered by coresight.
That's where I'm at now.
Get back to me if you (or anyone else) want more information.
Mathieu
Al IMPORTANT NOTICE: The contents of this email and any attachments are
confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
On 14 July 2016 at 05:16, Al Grant Al.Grant@arm.com wrote:
On ARM64 (because I _assume_ X86 folks didn't see this) we get 3 notifications. For example notification A will have address 0x7f93a60000 while, subsequently, notification B and C address 0x7f93a70000. Note that the latter two are 64K higher than the first one.
Once the last notification has been received the code in the main program is executed. That code (in the main program) jumps to library code mapped at the address it got from the first notification and not the last one, making the filter configuration all wrong.
As such I have to understand what notification B and C are for. Based on the vm_area_struct::vm_flags I'm guessing some sort of accounting feature but not sure yet.
Code and data segment(s)? They will be mapped in separately with separate permissions. If you look at /proc/self/maps or /proc/<pid>/maps you will see 2 or 3 entries per library, depending on system.
vm_pgoff will indicate the file offset within the ELF file. This will tend to be zero for the main code segment and non-zero for data segment(s).
vm_flags will indicate the segment permissions. You would normally get at least one without VM_WRITE and at least one with VM_WRITE. You might get R+X, R+W+X, R+W, just R etc.
Something worth knowing here is that you're more likely to see VM_EXEC on data sections on ARM64 than on X86_64, because the READ_MEANS_EXEC personality bit is inherited in the process tree on ARM64 and not X86_64. At worst, if your init process has an executable stack then practically every data segment mapping in the system gets VM_EXEC! You can't assume that you will only get one VM_EXEC mapping and that it will be "the" main code section; or act as if this was true by treating your last VM_EXEC notification as the code section. If you want to pick a single code section then use the one with vm_pgoff set to zero.
Again this is shows up in /proc/xxx/maps - if you see 'x's on writeable data segments and even on non executable files like /usr/lib/locale/locale-archive, then you've got sticky VM_EXEC. It's a bug really, I keep meaning to write a patch for this.
Could you dump out what you're seeing, complete with vm_flags and vm_pgoff, and also /proc/pid/maps? Hopefully things will then be clear.
Al
Alright, here's what I have so far [1]. It is the /proc/xyz/maps of the example program (enclosed herein) I have been running with some enhancement, i.e the last 3 columns. If we take line #2, 0x000875 is the value of the vm_area_struct::vm_flags, 0x0 vm_area_struct::vm_pgoff and finally, if "MMP2" is present, it means that perf generated an MMPAP2 event for it. Flag definition for the vm_flags can be found here[2].
I can see a few trends:
1. libcstest.so.1.0 and libc-2.21.so have been mmap'ed 4 times, each time with the same vm_flags.
2. For _all_ the binaries that have been loaded, i.e main, libc-2.21.so, libcstest.so.1.0 and ld-2.21.so, Perf only generated an mmap2 event the first time the binary was loaded.
3. Only the binaries with the vm_flags ending with 0x75 generated an mmap2 event.
4. I have experimented a lot with this and, at least for my test library (libcstest.so.1.o) - I haven't tried with glibc, the code that gets executed falls within the range of the first map. It sounds reasonable since the same mapped area is marked as executable.
Logically, the first map is likely to be the executable code (hence the permission) and the other ones different sections in the binary file. I think that is in line with that you explained in your previous email.
I'm pretty sure that if I start objdump'ing libc-2.21.so and libcstest.so.1.0 I'll find interesting things at the offset pointed to by vm_pgoff. I really need to see what perf does to identify the maps that deserve sending an MMAP2 event to user space - that is likely the key.
But for today I'm out of time. I've enclosed everything you need to investigate (should you feel like it).
Mathieu
[1]. http://pastebin.com/4tYDLRzF [2]. http://lxr.free-electrons.com/source/include/linux/mm.h
If I ignore notification B and C, things work amazingly well and one can really see the power offered by coresight.
That's where I'm at now.
Get back to me if you (or anyone else) want more information.
Mathieu
Al IMPORTANT NOTICE: The contents of this email and any attachments are
confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
-
Al Grant -
Mathieu Poirier