Sorry it took a while to get back to you guys. I was visiting customers last week. Most of my comments are just highlighting the differences between TI's STM 1.0 driver and ST-E's STM 1.0 driver, but there are a few questions, observations and suggestions. At the end I included some discussion on TI's meta data and OST header requirements.
I have not had a chance to look at your actual implementation yet. Did you do anything to abstract the actual HW transport ports and control registers from the higher level driver functions?
I realize there is a lot here to work through so if you would rather schedule a conference call to talk through the differences I can do that. I would like to start work on a Linaro (Unified) STM Spec next week if I can get feedback from everybody over the next few days. I will be out of the office on 5/27 and 5/31.
I am especially interested in details of what you guys have in mind for a "common trace framework to receive STM drivers". If by framework you mean well defined APIs that are implemented for specific devices, then I think we are in agreement. What Michael and I have talked about is a common STM user mode experience across all Linaro supported devices, making Linux user mode code 100% portable between our devices.
ST-E STM Driver stm-trace.txt review:
1. Software Overview
In your "Software Overview" it states:
"The end of data packet is marked by a time stamp on latest byte(s) only."
I assume that user messages can be made up of any number of bytes, half-words, words or longs (what ever is most efficient) and you simply terminate the last element of the message with a time-stamp - right?
In the TI STM implementation a message can be any number and combination of bytes, half-words, or word transfers terminated with a time-stamp on the last element. In addition to that we also add an OST header to a message. (See below for discussion on OST header).
2. Lossless/Lossy modes.
TI only supports lossless mode for sw generated messages and is enforced in our hw implementation. Lossy mode is reserved for true hw messages.
I did not notice that you documented a way to modify this through the debugfs API or IOCTLS.
I am kind of thinking that may be ok since this is really a hw configuration choice in your case, but in the TI case the user does not get to make that choice.
3. Channel Assignment
TI makes the assignment with mknod using the minor number to assign a fixed channel. This allows the user mode application to overload the channel usage for categorizing data (not my idea). I think we see the error of our ways here and will be ok with a dynamic channel allocation.
I am thinking that for each unique pid a channel should be assigned when the device is opened. I would guess you are keeping a channel table around and write() just checks the table for a pid assignment (no time to look at your implementation yet), if none is found the first free channel is used. If you moved this function back to open then you could do the IOCTL STM_GET_CHANNEL_NO anytime, not just after the first write.
In write how do you flag an error if you exhaust the number of available channels?
4. Kernel API
TI does not support a Kernel API (yet). I can see that the Alloc/Free and File IO type functions are useful and should be standard.
Not sure what you mean by "lockless" trace functions?
It looks like your "low level atomic trace functions for 1, 2, 4 or 8 bytes" is similar to TI's binary library functions (not supported by the TI STM Driver). This is what we use the OST header for, allowing our tool chain to differentiate between different message formats, rather than just assuming the data is a simple stream of bytes.
5. Debugfs APIs.
TI used a different approach. The tool-chain on the host provides all the transport setup through JTAG, so our driver does not support setting up the actual STM data export (number of pins and clock rate). In our case device transport parameters must match the host receiver's collection setup.
With your approach the user can change the clock rate and export pin width effectively at any time. Our tools actually go through a calibration process during initialization so any changes to the device's transport setup (clock rate, number of pins data exported on) would cause the TI tool chain a lot of grief.
There are some parameters we know we need to add (like master enables). This are currently also handled by the host tools. TI's STM module allows up to 4 SW masters to be enabled (with id masks that can be used to enable multiple masters from the same group) and 4 HW masters that can be enabled at the same time as the SW masters. If the user tries to enable more than the HW allows do you have a mechanism to flag an error?
I don't have a lot of experience with debugfs but I am assuming it's primarily used for allowing scripts to configure a driver (like in your example) or extract information.
We may want to define a standard set of debugfs options whose implementation is vendor specific. But that raises some questions:
- How do we deal with options that don't make sense for a specific vendor? Maybe just doing nothing is acceptable or do we want to provide a discovery mechanism? - Would user scripts then also be vendor specific? We should probably make an effort to avoid this. A discovery mechanism may allow user mode scripts to be generic.
6. Mapped Channels
I believe the TI hw transport channel mapping is compatible. In the TI case a channel is mapped into two spaces, the first half is for non-timestamp transfers and the second half is for time-stamped transfers. When we write a message (from a user mode write call for example) we simply write all the data except the last element through the non-timestamp port, and then the last element is written to the time-stamped port. So I think we could be compatible here.
With that said I am not sure about exposing all channels to a user mode library. You are relying on the library to use the convention of getting a free channel from the driver to make sure there are no conflicts. If the channel assignment is made when you open the device, you could conceivably map just the address space needed for the single channel, thus eliminating the need to get a free channel from the driver. In the TI case a single channel's transport mapping is 4K bytes, which matches the typical PAGE_SIZE. I realize not all hw implementations will match up with the PAGE_SIZE, which may be why you simply map all the channels back to user space.
Since free channels can become busy rapidly, maybe a better convention would be to simply use another device node if the user wants the library STM data to be transmitted on a different STM channel than the current process. This may be a case where providing a mechanism (see meta data discussion below) to allow channels to be named for the toolchain may be a good idea (provide task name and process id).
7. 8-byte Writes
TI does not support 64-bit writes with our STM 1.0 module. We may need an IOCTL to get the largest transfer supported for the mmap case. For all other cases this should just be hidden in the device dependent code.
8. Kernel Internal Usage
I like the idea of having dedicated support in the driver for common kernel logging. Any ideas on how you would support kernel STM channel assignments without hard-coding?
We may need a mechanism to communicate the definition of each hard-coded channel to our tools.
The following are TI specific:
9. Data protection
In SMP systems if the processor is switched a new master is generated (in some TI devices). So we protect the data with a mutex to guarantee a complete message is generated by the same master.
10. Meta Data
Our user mode HW libraries use meta data to transport data needed to process the HW profiling STM messages. Items like processor speed, sampling rate, processing options, ... (just a predefined byte buffer our tool-chain understands). The meta data is currently broadcasted on a dedicated channel (255), which conflicts with your hard-coded channel for logging printk output. So we will need to resolve hard-coded conflicts.
We need the driver to support registration and transport of the meta data on demand from the library (when the HW master is disabled, in case the collection buffer is small and circular).
I am thinking an IOCTL could be used to register meta data and then the data simply broadcast on a STM channel (will need to figure out which one) when the HW master is enabled and disabled.
Meta data transmission is problematic for circular buffers (like ETB's) thus the reason for also sending meta data when a hw master is disabled. SW masters are not typically disabled, and our HW does not provide a transmission byte count (remember there are HW messages also being generated in the TI case). So there is no way from a driver we can tell when the recoding buffer will wrap even if the user told us the buffer size. I am thinking the best solution would be to force the user to gracefully disable the channel to get any sw channel meta data provided by the driver.
TI supports three cases of data capture: - DTC/Host collection (stop on buffer full) - DTC/Host collection (circular buffer) - ETB/on-chip collection (circular buffer)
Of if the user is at a point in their code where they know thery will stop recoding on the hOst or ETB, we provide an IOCTL that simply disables all channels.
In the ETB case we may want to simply disable any open STM channels when the user decides to stop recording as a fail safe mechanism.
Note: Periodic transmission of meta data into a small circular buffer will not work well. In cases where the data is sparse the buffer will simply be filled with meta data rather than useful data.
11. OST Headers
Adding an OST header to each message is a requirement for compatibility with TI's toolchain. There are a couple of ways to approach:
Completely hidden from the user - The device specific code will know if the header is necessary. On a write, prior to the copy from user space, the device independent code would have to make a call to get a properly sized memory buffer from the device dependent code that would include the header.
User enabled - Provide an IOCTL that allows the user to put the driver in a tool-chain specific mode (like add OST headers).
Regards, Doug Deao
________________________________________ From: Philippe Langlais [mailto:philippe.langlais@linaro.org] Sent: Wednesday, May 04, 2011 3:08 AM To: Deao, Douglas Cc: Linus Walleij Subject: Re: STM at UDS-Budapest
Hi Doug,
On STE ux500 platforms we have the same STM module (follow MIDP STP 1.0), I have already posted our current implementation to the LKML and Linaro ML, it's very similar to your proposal. I can't be present to the Linaro summit but Linus Walleij can replace me for this topic, he proposes to write a common trace framework to receive STM drivers. Attached all our current proposal and work around STM.
Regards Philippe Langlais ST-Ericsson On 3 May 2011 00:42, Deao, Douglas d-deao@ti.com wrote: I am hosting an introductory session on System Trace at the summit. TI's System Trace Module (STM) provides a common protocol for instrumentation messages across multiple cores and system level hardware profiling in complex SoCs. Attached is a whitepaper for background reading. Looking forward to meeting you at the summit. Regards, Doug Deao Texas Instruments
_______________________________________________ linaro-dev mailing list linaro-dev@lists.linaro.org http://lists.linaro.org/mailman/listinfo/linaro-dev
Hi Doug,
I initiate the work to build a hardware trace framework in the kernel, I'm not started the study to have a common userspace API for STM, thanks to this email we can start such a work, but it may be long (next week I'm in vacation till June 7th). See my detailed response for all your interrogations and my thoughts about STE STM implementation below:
On 25 May 2011 23:54, Deao, Douglas d-deao@ti.com wrote:
Sorry it took a while to get back to you guys. I was visiting customers last week. Most of my comments are just highlighting the differences between TI's STM 1.0 driver and ST-E's STM 1.0 driver, but there are a few questions, observations and suggestions. At the end I included some discussion on TI's meta data and OST header requirements.
I have not had a chance to look at your actual implementation yet. Did you do anything to abstract the actual HW transport ports and control registers from the higher level driver functions?
Yes, partially I think through IOCTLs & debugfs (see our stm.h userspace API)
I realize there is a lot here to work through so if you would rather schedule a conference call to talk through the differences I can do that. I would like to start work on a Linaro (Unified) STM Spec next week if I can get feedback from everybody over the next few days. I will be out of the office on 5/27 and 5/31.
I hope this email is enough.
I am especially interested in details of what you guys have in mind for a "common trace framework to receive STM drivers". If by framework you mean well defined APIs that are implemented for specific devices, then I think we are in agreement. What Michael and I have talked about is a common STM user mode experience across all Linaro supported devices, making Linux user mode code 100% portable between our devices.
For my point of view, the trace device framework must ease the integration of new hardware trace drivers in the kernel (not only STM MIPI) to present standard hooks in current trace infrastructure, but it can cover a common STM userspace API too.
ST-E STM Driver stm-trace.txt review:
- Software Overview
In your "Software Overview" it states:
"The end of data packet is marked by a time stamp on latest byte(s) only."
I assume that user messages can be made up of any number of bytes, half-words, words or longs (what ever is most efficient) and you simply terminate the last element of the message with a time-stamp - right?
Yes, the message buffer can be mis aligned
In the TI STM implementation a message can be any number and combination of bytes, half-words, or word transfers terminated with a time-stamp on the last element. In addition to that we also add an OST header to a message. (See below for discussion on OST header).
In our case the OST header is added by the external capture probe.
- Lossless/Lossy modes.
TI only supports lossless mode for sw generated messages and is enforced in our hw implementation. Lossy mode is reserved for true hw messages.
For STE, hw messages are always lossy, but sw generated messages could be configured in lossless (default) or lossy mode.
I did not notice that you documented a way to modify this through the debugfs API or IOCTLS.
I have 2 IOCTLs (STM_SET_MODE & STM_GET_MODE) and debugfs (masters_modes) interfaces for that.
I am kind of thinking that may be ok since this is really a hw configuration choice in your case, but in the TI case the user does not get to make that choice.
OK
- Channel Assignment
TI makes the assignment with mknod using the minor number to assign a fixed channel. This allows the user mode application to overload the channel usage for categorizing data (not my idea). I think we see the error of our ways here and will be ok with a dynamic channel allocation.
If we are agree with dynamic channel allocation then a common STM userspace API is possible I think and perhaps common STM driver.
I am thinking that for each unique pid a channel should be assigned when the device is opened. I would guess you are keeping a channel table around and write() just checks the table for a pid assignment (no time to look at your implementation yet), if none is found the first free channel is used. If you moved this function back to open then you could do the IOCTL STM_GET_CHANNEL_NO anytime, not just after the first write.
The reason behind this behavior is for our current STM user lib which open "/dev/stm" and alloc/free channels with IOCTL and never use write operation (only mmap + direct write) and in this case we don't want to loose a channel. I think we can change this behavior. We can support multiple channels allocated for one Process to avoid contention in multi-threaded process.
In write how do you flag an error if you exhaust the number of available channels?
I return -ENOMEM.
- Kernel API
TI does not support a Kernel API (yet). I can see that the Alloc/Free and File IO type functions are useful and should be standard.
Not sure what you mean by "lockless" trace functions?
Not protected against concurrent access, it's the responsibility to the caller.
It looks like your "low level atomic trace functions for 1, 2, 4 or 8 bytes" is similar to TI's binary library functions (not supported by the TI STM Driver). This is what we use the OST header for, allowing our tool chain to differentiate between different message formats, rather than just assuming the data is a simple stream of bytes.
I'm not ready to add an OST header to each trace messages to particularly for kernel function traces (ftrace has short messages 8 bytes size with high throughput), I rather to dedicate a specific channel for this, OST header has to be optional.
- Debugfs APIs.
TI used a different approach. The tool-chain on the host provides all the transport setup through JTAG, so our driver does not support setting up the actual STM data export (number of pins and clock rate). In our case device transport parameters must match the host receiver's collection setup.
With your approach the user can change the clock rate and export pin width effectively at any time. Our tools actually go through a calibration process during initialization so any changes to the device's transport setup (clock rate, number of pins data exported on) would cause the TI tool chain a lot of grief.
There are some parameters we know we need to add (like master enables). This are currently also handled by the host tools. TI's STM module allows up to 4 SW masters to be enabled (with id masks that can be used to enable multiple masters from the same group) and 4 HW masters that can be enabled at the same time as the SW masters. If the user tries to enable more than the HW allows do you have a mechanism to flag an error?
We can't face the same problem with STE STM, our hardware support 6 SW masters & 2 HW masters without restriction, all could be enabled at same time (managed by a bit field register)
I don't have a lot of experience with debugfs but I am assuming it's primarily used for allowing scripts to configure a driver (like in your example) or extract information.
It's the current way to configure kernel tracing infrastructure (from console, through scripts or user apps), kernel users are familiar with this interface.
We may want to define a standard set of debugfs options whose implementation is vendor specific. But that raises some questions:
- How do we deal with options that don't make sense for a specific vendor?
Maybe just doing nothing is acceptable or do we want to provide a discovery mechanism?
Do nothing may be sufficient, we can add STM IP specific string descriptor too, something like "vendor id+STM IP id+STP version"
- Would user scripts then also be vendor specific?
We should probably make an effort to avoid this. A discovery mechanism may allow user mode scripts to be generic.
Yes, but STM configuration scripts are device specific, to see.
- Mapped Channels
I believe the TI hw transport channel mapping is compatible. In the TI case a channel is mapped into two spaces, the first half is for non-timestamp transfers and the second half is for time-stamped transfers. When we write a message (from a user mode write call for example) we simply write all the data except the last element through the non-timestamp port, and then the last element is written to the time-stamped port. So I think we could be compatible here.
I think so, we have similar channel memory mapping but, one STM channel is accessed through this C union of 2 consecutive 64bits word (first for non-timestamp, 2nd with timestamp) /* One single channel mapping */ struct stm_channel { union { __u8 no_stamp8; __u16 no_stamp16; __u32 no_stamp32; __u64 no_stamp64; }; union { __u8 stamp8; __u16 stamp16; __u32 stamp32; __u64 stamp64; }; };
With that said I am not sure about exposing all channels to a user mode library. You are relying on the library to use the convention of getting a free channel from the driver to make sure there are no conflicts. If the channel assignment is made when you open the device, you could conceivably map just the address space needed for the single channel, thus eliminating the need to get a free channel from the driver. In the TI case a single channel's transport mapping is 4K bytes, which matches the typical PAGE_SIZE. I realize not all hw implementations will match up with the PAGE_SIZE, which may be why you simply map all the channels back to user space.
It's our case, the 256 channels is mapped in the same 4K memory page.
Since free channels can become busy rapidly, maybe a better convention would be to simply use another device node if the user wants the library STM data to be transmitted on a different STM channel than the current process. This may be a case where providing a mechanism (see meta data discussion below) to allow channels to be named for the toolchain may be a good idea (provide task name and process id).
In our current implementation, a new open on the same "/dev/stm" device allocate a new channel at first write.
- 8-byte Writes
TI does not support 64-bit writes with our STM 1.0 module. We may need an IOCTL to get the largest transfer supported for the mmap case. For all other cases this should just be hidden in the device dependent code.
I think so
- Kernel Internal Usage
I like the idea of having dedicated support in the driver for common kernel logging. Any ideas on how you would support kernel STM channel assignments without hard-coding?
But configurable through kernel configuration, the other way is to dedicate a specific channel for this signalization like your channel 255, not so easy to define.
We may need a mechanism to communicate the definition of each hard-coded channel to our tools.
to see
The following are TI specific:
- Data protection
In SMP systems if the processor is switched a new master is generated (in some TI devices). So we protect the data with a mutex to guarantee a complete message is generated by the same master.
OK, it's not our case.
- Meta Data
Our user mode HW libraries use meta data to transport data needed to process the HW profiling STM messages. Items like processor speed, sampling rate, processing options, ... (just a predefined byte buffer our tool-chain understands). The meta data is currently broadcasted on a dedicated channel (255), which conflicts with your hard-coded channel for logging printk output. So we will need to resolve hard-coded conflicts.
Easy to fix, we can reserve dedicated channels at driver init and change our hard-coded affectation.
We need the driver to support registration and transport of the meta data on demand from the library (when the HW master is disabled, in case the collection buffer is small and circular).
I am thinking an IOCTL could be used to register meta data and then the data simply broadcast on a STM channel (will need to figure out which one) when the HW master is enabled and disabled.
We don't face the same problematic
Meta data transmission is problematic for circular buffers (like ETB's) thus the reason for also sending meta data when a hw master is disabled. SW masters are not typically disabled, and our HW does not provide a transmission byte count (remember there are HW messages also being generated in the TI case). So there is no way from a driver we can tell when the recoding buffer will wrap even if the user told us the buffer size. I am thinking the best solution would be to force the user to gracefully disable the channel to get any sw channel meta data provided by the driver.
I think so, if I have well understood
TI supports three cases of data capture:
- DTC/Host collection (stop on buffer full)
- DTC/Host collection (circular buffer)
- ETB/on-chip collection (circular buffer)
We don't support ETB/on-chip collection, We use external probe or Lauterbach combiprobe for trace data collection (I think it's circular buffer)
Of if the user is at a point in their code where they know thery will stop recoding on the hOst or ETB, we provide an IOCTL that simply disables all channels.
In the ETB case we may want to simply disable any open STM channels when the user decides to stop recording as a fail safe mechanism.
Note: Periodic transmission of meta data into a small circular buffer will not work well. In cases where the data is sparse the buffer will simply be filled with meta data rather than useful data.
- OST Headers
Adding an OST header to each message is a requirement for compatibility with TI's toolchain. There are a couple of ways to approach:
Completely hidden from the user - The device specific code will know if the header is necessary. On a write, prior to the copy from user space, the device independent code would have to make a call to get a properly sized memory buffer from the device dependent code that would include the header.
User enabled - Provide an IOCTL that allows the user to put the driver in a tool-chain specific mode (like add OST headers).
I agree, see my previous remark on OST headers
Regards, Doug Deao
I attached my last STM patches which introduce trace framework.
Best Regards, Philippe Langlais
From: Philippe Langlais [mailto:philippe.langlais@linaro.org] Sent: Wednesday, May 04, 2011 3:08 AM To: Deao, Douglas Cc: Linus Walleij Subject: Re: STM at UDS-Budapest
Hi Doug,
On STE ux500 platforms we have the same STM module (follow MIDP STP 1.0), I have already posted our current implementation to the LKML and Linaro ML, it's very similar to your proposal. I can't be present to the Linaro summit but Linus Walleij can replace me for this topic, he proposes to write a common trace framework to receive STM drivers. Attached all our current proposal and work around STM.
Regards Philippe Langlais ST-Ericsson On 3 May 2011 00:42, Deao, Douglas d-deao@ti.com wrote: I am hosting an introductory session on System Trace at the summit. TI's System Trace Module (STM) provides a common protocol for instrumentation messages across multiple cores and system level hardware profiling in complex SoCs. Attached is a whitepaper for background reading.
Looking forward to meeting you at the summit.
Regards, Doug Deao Texas Instruments
linaro-dev mailing list linaro-dev@lists.linaro.org http://lists.linaro.org/mailman/listinfo/linaro-dev
Philippe,
So from your comments it seems to me there is nothing controversial. Do you have spec for the trace framework you have coded and sent in the patches?
Regards,
Doug
________________________________ From: Philippe Langlais [mailto:philippe.langlais@linaro.org] Sent: Thursday, May 26, 2011 10:20 AM To: Deao, Douglas; Linus Walleij; Lee Jones Cc: Arvind Chauhan; Michael Hope; linaro-dev@lists.linaro.org; pierre.peiffer@stericsson.com; loic.pallardy@stericsson.com Subject: Re: ST-E STM Driver Review
Hi Doug,
I initiate the work to build a hardware trace framework in the kernel, I'm not started the study to have a common userspace API for STM, thanks to this email we can start such a work, but it may be long (next week I'm in vacation till June 7th). See my detailed response for all your interrogations and my thoughts about STE STM implementation below: On 25 May 2011 23:54, Deao, Douglas <d-deao@ti.commailto:d-deao@ti.com> wrote: Sorry it took a while to get back to you guys. I was visiting customers last week. Most of my comments are just highlighting the differences between TI's STM 1.0 driver and ST-E's STM 1.0 driver, but there are a few questions, observations and suggestions. At the end I included some discussion on TI's meta data and OST header requirements.
I have not had a chance to look at your actual implementation yet. Did you do anything to abstract the actual HW transport ports and control registers from the higher level driver functions? Yes, partially I think through IOCTLs & debugfs (see our stm.h userspace API)
I realize there is a lot here to work through so if you would rather schedule a conference call to talk through the differences I can do that. I would like to start work on a Linaro (Unified) STM Spec next week if I can get feedback from everybody over the next few days. I will be out of the office on 5/27 and 5/31. I hope this email is enough.
I am especially interested in details of what you guys have in mind for a "common trace framework to receive STM drivers". If by framework you mean well defined APIs that are implemented for specific devices, then I think we are in agreement. What Michael and I have talked about is a common STM user mode experience across all Linaro supported devices, making Linux user mode code 100% portable between our devices. For my point of view, the trace device framework must ease the integration of new hardware trace drivers in the kernel (not only STM MIPI) to present standard hooks in current trace infrastructure, but it can cover a common STM userspace API too.
ST-E STM Driver stm-trace.txt review:
1. Software Overview
In your "Software Overview" it states:
"The end of data packet is marked by a time stamp on latest byte(s) only."
I assume that user messages can be made up of any number of bytes, half-words, words or longs (what ever is most efficient) and you simply terminate the last element of the message with a time-stamp - right? Yes, the message buffer can be mis aligned
In the TI STM implementation a message can be any number and combination of bytes, half-words, or word transfers terminated with a time-stamp on the last element. In addition to that we also add an OST header to a message. (See below for discussion on OST header). In our case the OST header is added by the external capture probe.
2. Lossless/Lossy modes.
TI only supports lossless mode for sw generated messages and is enforced in our hw implementation. Lossy mode is reserved for true hw messages. For STE, hw messages are always lossy, but sw generated messages could be configured in lossless (default) or lossy mode.
I did not notice that you documented a way to modify this through the debugfs API or IOCTLS. I have 2 IOCTLs (STM_SET_MODE & STM_GET_MODE) and debugfs (masters_modes) interfaces for that.
I am kind of thinking that may be ok since this is really a hw configuration choice in your case, but in the TI case the user does not get to make that choice. OK
3. Channel Assignment
TI makes the assignment with mknod using the minor number to assign a fixed channel. This allows the user mode application to overload the channel usage for categorizing data (not my idea). I think we see the error of our ways here and will be ok with a dynamic channel allocation. If we are agree with dynamic channel allocation then a common STM userspace API is possible I think and perhaps common STM driver.
I am thinking that for each unique pid a channel should be assigned when the device is opened. I would guess you are keeping a channel table around and write() just checks the table for a pid assignment (no time to look at your implementation yet), if none is found the first free channel is used. If you moved this function back to open then you could do the IOCTL STM_GET_CHANNEL_NO anytime, not just after the first write. The reason behind this behavior is for our current STM user lib which open "/dev/stm" and alloc/free channels with IOCTL and never use write operation (only mmap + direct write) and in this case we don't want to loose a channel. I think we can change this behavior. We can support multiple channels allocated for one Process to avoid contention in multi-threaded process.
In write how do you flag an error if you exhaust the number of available channels? I return -ENOMEM.
4. Kernel API
TI does not support a Kernel API (yet). I can see that the Alloc/Free and File IO type functions are useful and should be standard.
Not sure what you mean by "lockless" trace functions? Not protected against concurrent access, it's the responsibility to the caller.
It looks like your "low level atomic trace functions for 1, 2, 4 or 8 bytes" is similar to TI's binary library functions (not supported by the TI STM Driver). This is what we use the OST header for, allowing our tool chain to differentiate between different message formats, rather than just assuming the data is a simple stream of bytes. I'm not ready to add an OST header to each trace messages to particularly for kernel function traces (ftrace has short messages 8 bytes size with high throughput), I rather to dedicate a specific channel for this, OST header has to be optional.
5. Debugfs APIs.
TI used a different approach. The tool-chain on the host provides all the transport setup through JTAG, so our driver does not support setting up the actual STM data export (number of pins and clock rate). In our case device transport parameters must match the host receiver's collection setup.
With your approach the user can change the clock rate and export pin width effectively at any time. Our tools actually go through a calibration process during initialization so any changes to the device's transport setup (clock rate, number of pins data exported on) would cause the TI tool chain a lot of grief.
There are some parameters we know we need to add (like master enables). This are currently also handled by the host tools. TI's STM module allows up to 4 SW masters to be enabled (with id masks that can be used to enable multiple masters from the same group) and 4 HW masters that can be enabled at the same time as the SW masters. If the user tries to enable more than the HW allows do you have a mechanism to flag an error? We can't face the same problem with STE STM, our hardware support 6 SW masters & 2 HW masters without restriction, all could be enabled at same time (managed by a bit field register)
I don't have a lot of experience with debugfs but I am assuming it's primarily used for allowing scripts to configure a driver (like in your example) or extract information. It's the current way to configure kernel tracing infrastructure (from console, through scripts or user apps), kernel users are familiar with this interface.
We may want to define a standard set of debugfs options whose implementation is vendor specific. But that raises some questions:
- How do we deal with options that don't make sense for a specific vendor? Maybe just doing nothing is acceptable or do we want to provide a discovery mechanism? Do nothing may be sufficient, we can add STM IP specific string descriptor too, something like "vendor id+STM IP id+STP version" - Would user scripts then also be vendor specific? We should probably make an effort to avoid this. A discovery mechanism may allow user mode scripts to be generic. Yes, but STM configuration scripts are device specific, to see.
6. Mapped Channels
I believe the TI hw transport channel mapping is compatible. In the TI case a channel is mapped into two spaces, the first half is for non-timestamp transfers and the second half is for time-stamped transfers. When we write a message (from a user mode write call for example) we simply write all the data except the last element through the non-timestamp port, and then the last element is written to the time-stamped port. So I think we could be compatible here. I think so, we have similar channel memory mapping but, one STM channel is accessed through this C union of 2 consecutive 64bits word (first for non-timestamp, 2nd with timestamp) /* One single channel mapping */ struct stm_channel { union { __u8 no_stamp8; __u16 no_stamp16; __u32 no_stamp32; __u64 no_stamp64; }; union { __u8 stamp8; __u16 stamp16; __u32 stamp32; __u64 stamp64; }; };
With that said I am not sure about exposing all channels to a user mode library. You are relying on the library to use the convention of getting a free channel from the driver to make sure there are no conflicts. If the channel assignment is made when you open the device, you could conceivably map just the address space needed for the single channel, thus eliminating the need to get a free channel from the driver. In the TI case a single channel's transport mapping is 4K bytes, which matches the typical PAGE_SIZE. I realize not all hw implementations will match up with the PAGE_SIZE, which may be why you simply map all the channels back to user space. It's our case, the 256 channels is mapped in the same 4K memory page.
Since free channels can become busy rapidly, maybe a better convention would be to simply use another device node if the user wants the library STM data to be transmitted on a different STM channel than the current process. This may be a case where providing a mechanism (see meta data discussion below) to allow channels to be named for the toolchain may be a good idea (provide task name and process id). In our current implementation, a new open on the same "/dev/stm" device allocate a new channel at first write.
7. 8-byte Writes
TI does not support 64-bit writes with our STM 1.0 module. We may need an IOCTL to get the largest transfer supported for the mmap case. For all other cases this should just be hidden in the device dependent code. I think so
8. Kernel Internal Usage
I like the idea of having dedicated support in the driver for common kernel logging. Any ideas on how you would support kernel STM channel assignments without hard-coding? But configurable through kernel configuration, the other way is to dedicate a specific channel for this signalization like your channel 255, not so easy to define.
We may need a mechanism to communicate the definition of each hard-coded channel to our tools. to see
The following are TI specific:
9. Data protection
In SMP systems if the processor is switched a new master is generated (in some TI devices). So we protect the data with a mutex to guarantee a complete message is generated by the same master. OK, it's not our case.
10. Meta Data
Our user mode HW libraries use meta data to transport data needed to process the HW profiling STM messages. Items like processor speed, sampling rate, processing options, ... (just a predefined byte buffer our tool-chain understands). The meta data is currently broadcasted on a dedicated channel (255), which conflicts with your hard-coded channel for logging printk output. So we will need to resolve hard-coded conflicts. Easy to fix, we can reserve dedicated channels at driver init and change our hard-coded affectation.
We need the driver to support registration and transport of the meta data on demand from the library (when the HW master is disabled, in case the collection buffer is small and circular).
I am thinking an IOCTL could be used to register meta data and then the data simply broadcast on a STM channel (will need to figure out which one) when the HW master is enabled and disabled. We don't face the same problematic
Meta data transmission is problematic for circular buffers (like ETB's) thus the reason for also sending meta data when a hw master is disabled. SW masters are not typically disabled, and our HW does not provide a transmission byte count (remember there are HW messages also being generated in the TI case). So there is no way from a driver we can tell when the recoding buffer will wrap even if the user told us the buffer size. I am thinking the best solution would be to force the user to gracefully disable the channel to get any sw channel meta data provided by the driver. I think so, if I have well understood
TI supports three cases of data capture: - DTC/Host collection (stop on buffer full) - DTC/Host collection (circular buffer) - ETB/on-chip collection (circular buffer) We don't support ETB/on-chip collection, We use external probe or Lauterbach combiprobe for trace data collection (I think it's circular buffer)
Of if the user is at a point in their code where they know thery will stop recoding on the hOst or ETB, we provide an IOCTL that simply disables all channels.
In the ETB case we may want to simply disable any open STM channels when the user decides to stop recording as a fail safe mechanism.
Note: Periodic transmission of meta data into a small circular buffer will not work well. In cases where the data is sparse the buffer will simply be filled with meta data rather than useful data.
11. OST Headers
Adding an OST header to each message is a requirement for compatibility with TI's toolchain. There are a couple of ways to approach:
Completely hidden from the user - The device specific code will know if the header is necessary. On a write, prior to the copy from user space, the device independent code would have to make a call to get a properly sized memory buffer from the device dependent code that would include the header.
User enabled - Provide an IOCTL that allows the user to put the driver in a tool-chain specific mode (like add OST headers). I agree, see my previous remark on OST headers
Regards, Doug Deao
I attached my last STM patches which introduce trace framework.
Best Regards, Philippe Langlais
________________________________________ From: Philippe Langlais [mailto:philippe.langlais@linaro.orgmailto:philippe.langlais@linaro.org] Sent: Wednesday, May 04, 2011 3:08 AM To: Deao, Douglas Cc: Linus Walleij Subject: Re: STM at UDS-Budapest
Hi Doug,
On STE ux500 platforms we have the same STM module (follow MIDP STP 1.0), I have already posted our current implementation to the LKML and Linaro ML, it's very similar to your proposal. I can't be present to the Linaro summit but Linus Walleij can replace me for this topic, he proposes to write a common trace framework to receive STM drivers. Attached all our current proposal and work around STM.
Regards Philippe Langlais ST-Ericsson On 3 May 2011 00:42, Deao, Douglas <d-deao@ti.commailto:d-deao@ti.com> wrote: I am hosting an introductory session on System Trace at the summit. TI's System Trace Module (STM) provides a common protocol for instrumentation messages across multiple cores and system level hardware profiling in complex SoCs. Attached is a whitepaper for background reading.
Looking forward to meeting you at the summit.
Regards, Doug Deao Texas Instruments
_______________________________________________ linaro-dev mailing list linaro-dev@lists.linaro.orgmailto:linaro-dev@lists.linaro.org http://lists.linaro.org/mailman/listinfo/linaro-dev
On Thursday 26 May 2011, Philippe Langlais wrote:
I initiate the work to build a hardware trace framework in the kernel, I'm not started the study to have a common userspace API for STM, thanks to this email we can start such a work, but it may be long (next week I'm in vacation till June 7th).
I missed the initial parts of the conversation, but I think that regarding the user interface, the most important issue by far is integration into the perf events infrastructure. There is no room for having yet another API to get at tracing and performance data.
See my detailed response for all your interrogations and my thoughts about STE STM implementation below:
On 25 May 2011 23:54, Deao, Douglas d-deao@ti.com wrote:
Sorry it took a while to get back to you guys. I was visiting customers last week. Most of my comments are just highlighting the differences between TI's STM 1.0 driver and ST-E's STM 1.0 driver, but there are a few questions, observations and suggestions. At the end I included some discussion on TI's meta data and OST header requirements.
I have not had a chance to look at your actual implementation yet. Did you do anything to abstract the actual HW transport ports and control registers from the higher level driver functions?
Yes, partially I think through IOCTLs & debugfs (see our stm.h userspace API)
Any ioctls in addition to the ones defined in include/linux/perf_event.h will need to be coordinated with the perf developers.
Regarding a debugfs interface, you need to consider that all you cannot build stable interfaces in debugfs by definition. If you want to have long-term support for your subsystem, you would need a new file system.
I am especially interested in details of what you guys have in mind for a "common trace framework to receive STM drivers". If by framework you mean well defined APIs that are implemented for specific devices, then I think we are in agreement. What Michael and I have talked about is a common STM user mode experience across all Linaro supported devices, making Linux user mode code 100% portable between our devices.
For my point of view, the trace device framework must ease the integration of new hardware trace drivers in the kernel (not only STM MIPI) to present standard hooks in current trace infrastructure, but it can cover a common STM userspace API too.
Can you explain the difference between hardware trace drivers and PMU drivers? What does the common infrastructure for trace drivers do that is not already handled by ftrace or perf?
I am thinking that for each unique pid a channel should be assigned when the device is opened. I would guess you are keeping a channel table around and write() just checks the table for a pid assignment (no time to look at your implementation yet), if none is found the first free channel is used. If you moved this function back to open then you could do the IOCTL STM_GET_CHANNEL_NO anytime, not just after the first write.
The reason behind this behavior is for our current STM user lib which open "/dev/stm" and alloc/free channels with IOCTL and never use write operation (only mmap + direct write) and in this case we don't want to loose a channel. I think we can change this behavior. We can support multiple channels allocated for one Process to avoid contention in multi-threaded process.
/dev/stm does not sound particularly hardware independent, i.e. it would not be portable to other architectures. The kernel interface should realy abstract such differences. Why do you need an extra character device besides the perf file descriptor?
Arnd
Philippe,
Forgot to mention I to will be on vacation the week after you and will return on the 14th. I am only in three days next week but will try to put together a proposal for a common userspace STM API.
Regards, Doug
________________________________ From: Deao, Douglas Sent: Thursday, May 26, 2011 4:18 PM To: 'Philippe Langlais'; Linus Walleij; Lee Jones Cc: Arvind Chauhan; Michael Hope; linaro-dev@lists.linaro.org; pierre.peiffer@stericsson.com; loic.pallardy@stericsson.com Subject: RE: ST-E STM Driver Review
Philippe,
So from your comments it seems to me there is nothing controversial. Do you have spec for the trace framework you have coded and sent in the patches?
Regards,
Doug
________________________________ From: Philippe Langlais [mailto:philippe.langlais@linaro.org] Sent: Thursday, May 26, 2011 10:20 AM To: Deao, Douglas; Linus Walleij; Lee Jones Cc: Arvind Chauhan; Michael Hope; linaro-dev@lists.linaro.org; pierre.peiffer@stericsson.com; loic.pallardy@stericsson.com Subject: Re: ST-E STM Driver Review
Hi Doug,
I initiate the work to build a hardware trace framework in the kernel, I'm not started the study to have a common userspace API for STM, thanks to this email we can start such a work, but it may be long (next week I'm in vacation till June 7th). See my detailed response for all your interrogations and my thoughts about STE STM implementation below: On 25 May 2011 23:54, Deao, Douglas <d-deao@ti.commailto:d-deao@ti.com> wrote: Sorry it took a while to get back to you guys. I was visiting customers last week. Most of my comments are just highlighting the differences between TI's STM 1.0 driver and ST-E's STM 1.0 driver, but there are a few questions, observations and suggestions. At the end I included some discussion on TI's meta data and OST header requirements.
I have not had a chance to look at your actual implementation yet. Did you do anything to abstract the actual HW transport ports and control registers from the higher level driver functions? Yes, partially I think through IOCTLs & debugfs (see our stm.h userspace API)
I realize there is a lot here to work through so if you would rather schedule a conference call to talk through the differences I can do that. I would like to start work on a Linaro (Unified) STM Spec next week if I can get feedback from everybody over the next few days. I will be out of the office on 5/27 and 5/31. I hope this email is enough.
I am especially interested in details of what you guys have in mind for a "common trace framework to receive STM drivers". If by framework you mean well defined APIs that are implemented for specific devices, then I think we are in agreement. What Michael and I have talked about is a common STM user mode experience across all Linaro supported devices, making Linux user mode code 100% portable between our devices. For my point of view, the trace device framework must ease the integration of new hardware trace drivers in the kernel (not only STM MIPI) to present standard hooks in current trace infrastructure, but it can cover a common STM userspace API too.
ST-E STM Driver stm-trace.txt review:
1. Software Overview
In your "Software Overview" it states:
"The end of data packet is marked by a time stamp on latest byte(s) only."
I assume that user messages can be made up of any number of bytes, half-words, words or longs (what ever is most efficient) and you simply terminate the last element of the message with a time-stamp - right? Yes, the message buffer can be mis aligned
In the TI STM implementation a message can be any number and combination of bytes, half-words, or word transfers terminated with a time-stamp on the last element. In addition to that we also add an OST header to a message. (See below for discussion on OST header). In our case the OST header is added by the external capture probe.
2. Lossless/Lossy modes.
TI only supports lossless mode for sw generated messages and is enforced in our hw implementation. Lossy mode is reserved for true hw messages. For STE, hw messages are always lossy, but sw generated messages could be configured in lossless (default) or lossy mode.
I did not notice that you documented a way to modify this through the debugfs API or IOCTLS. I have 2 IOCTLs (STM_SET_MODE & STM_GET_MODE) and debugfs (masters_modes) interfaces for that.
I am kind of thinking that may be ok since this is really a hw configuration choice in your case, but in the TI case the user does not get to make that choice. OK
3. Channel Assignment
TI makes the assignment with mknod using the minor number to assign a fixed channel. This allows the user mode application to overload the channel usage for categorizing data (not my idea). I think we see the error of our ways here and will be ok with a dynamic channel allocation. If we are agree with dynamic channel allocation then a common STM userspace API is possible I think and perhaps common STM driver.
I am thinking that for each unique pid a channel should be assigned when the device is opened. I would guess you are keeping a channel table around and write() just checks the table for a pid assignment (no time to look at your implementation yet), if none is found the first free channel is used. If you moved this function back to open then you could do the IOCTL STM_GET_CHANNEL_NO anytime, not just after the first write. The reason behind this behavior is for our current STM user lib which open "/dev/stm" and alloc/free channels with IOCTL and never use write operation (only mmap + direct write) and in this case we don't want to loose a channel. I think we can change this behavior. We can support multiple channels allocated for one Process to avoid contention in multi-threaded process.
In write how do you flag an error if you exhaust the number of available channels? I return -ENOMEM.
4. Kernel API
TI does not support a Kernel API (yet). I can see that the Alloc/Free and File IO type functions are useful and should be standard.
Not sure what you mean by "lockless" trace functions? Not protected against concurrent access, it's the responsibility to the caller.
It looks like your "low level atomic trace functions for 1, 2, 4 or 8 bytes" is similar to TI's binary library functions (not supported by the TI STM Driver). This is what we use the OST header for, allowing our tool chain to differentiate between different message formats, rather than just assuming the data is a simple stream of bytes. I'm not ready to add an OST header to each trace messages to particularly for kernel function traces (ftrace has short messages 8 bytes size with high throughput), I rather to dedicate a specific channel for this, OST header has to be optional.
5. Debugfs APIs.
TI used a different approach. The tool-chain on the host provides all the transport setup through JTAG, so our driver does not support setting up the actual STM data export (number of pins and clock rate). In our case device transport parameters must match the host receiver's collection setup.
With your approach the user can change the clock rate and export pin width effectively at any time. Our tools actually go through a calibration process during initialization so any changes to the device's transport setup (clock rate, number of pins data exported on) would cause the TI tool chain a lot of grief.
There are some parameters we know we need to add (like master enables). This are currently also handled by the host tools. TI's STM module allows up to 4 SW masters to be enabled (with id masks that can be used to enable multiple masters from the same group) and 4 HW masters that can be enabled at the same time as the SW masters. If the user tries to enable more than the HW allows do you have a mechanism to flag an error? We can't face the same problem with STE STM, our hardware support 6 SW masters & 2 HW masters without restriction, all could be enabled at same time (managed by a bit field register)
I don't have a lot of experience with debugfs but I am assuming it's primarily used for allowing scripts to configure a driver (like in your example) or extract information. It's the current way to configure kernel tracing infrastructure (from console, through scripts or user apps), kernel users are familiar with this interface.
We may want to define a standard set of debugfs options whose implementation is vendor specific. But that raises some questions:
- How do we deal with options that don't make sense for a specific vendor? Maybe just doing nothing is acceptable or do we want to provide a discovery mechanism? Do nothing may be sufficient, we can add STM IP specific string descriptor too, something like "vendor id+STM IP id+STP version" - Would user scripts then also be vendor specific? We should probably make an effort to avoid this. A discovery mechanism may allow user mode scripts to be generic. Yes, but STM configuration scripts are device specific, to see.
6. Mapped Channels
I believe the TI hw transport channel mapping is compatible. In the TI case a channel is mapped into two spaces, the first half is for non-timestamp transfers and the second half is for time-stamped transfers. When we write a message (from a user mode write call for example) we simply write all the data except the last element through the non-timestamp port, and then the last element is written to the time-stamped port. So I think we could be compatible here. I think so, we have similar channel memory mapping but, one STM channel is accessed through this C union of 2 consecutive 64bits word (first for non-timestamp, 2nd with timestamp) /* One single channel mapping */ struct stm_channel { union { __u8 no_stamp8; __u16 no_stamp16; __u32 no_stamp32; __u64 no_stamp64; }; union { __u8 stamp8; __u16 stamp16; __u32 stamp32; __u64 stamp64; }; };
With that said I am not sure about exposing all channels to a user mode library. You are relying on the library to use the convention of getting a free channel from the driver to make sure there are no conflicts. If the channel assignment is made when you open the device, you could conceivably map just the address space needed for the single channel, thus eliminating the need to get a free channel from the driver. In the TI case a single channel's transport mapping is 4K bytes, which matches the typical PAGE_SIZE. I realize not all hw implementations will match up with the PAGE_SIZE, which may be why you simply map all the channels back to user space. It's our case, the 256 channels is mapped in the same 4K memory page.
Since free channels can become busy rapidly, maybe a better convention would be to simply use another device node if the user wants the library STM data to be transmitted on a different STM channel than the current process. This may be a case where providing a mechanism (see meta data discussion below) to allow channels to be named for the toolchain may be a good idea (provide task name and process id). In our current implementation, a new open on the same "/dev/stm" device allocate a new channel at first write.
7. 8-byte Writes
TI does not support 64-bit writes with our STM 1.0 module. We may need an IOCTL to get the largest transfer supported for the mmap case. For all other cases this should just be hidden in the device dependent code. I think so
8. Kernel Internal Usage
I like the idea of having dedicated support in the driver for common kernel logging. Any ideas on how you would support kernel STM channel assignments without hard-coding? But configurable through kernel configuration, the other way is to dedicate a specific channel for this signalization like your channel 255, not so easy to define.
We may need a mechanism to communicate the definition of each hard-coded channel to our tools. to see
The following are TI specific:
9. Data protection
In SMP systems if the processor is switched a new master is generated (in some TI devices). So we protect the data with a mutex to guarantee a complete message is generated by the same master. OK, it's not our case.
10. Meta Data
Our user mode HW libraries use meta data to transport data needed to process the HW profiling STM messages. Items like processor speed, sampling rate, processing options, ... (just a predefined byte buffer our tool-chain understands). The meta data is currently broadcasted on a dedicated channel (255), which conflicts with your hard-coded channel for logging printk output. So we will need to resolve hard-coded conflicts. Easy to fix, we can reserve dedicated channels at driver init and change our hard-coded affectation.
We need the driver to support registration and transport of the meta data on demand from the library (when the HW master is disabled, in case the collection buffer is small and circular).
I am thinking an IOCTL could be used to register meta data and then the data simply broadcast on a STM channel (will need to figure out which one) when the HW master is enabled and disabled. We don't face the same problematic
Meta data transmission is problematic for circular buffers (like ETB's) thus the reason for also sending meta data when a hw master is disabled. SW masters are not typically disabled, and our HW does not provide a transmission byte count (remember there are HW messages also being generated in the TI case). So there is no way from a driver we can tell when the recoding buffer will wrap even if the user told us the buffer size. I am thinking the best solution would be to force the user to gracefully disable the channel to get any sw channel meta data provided by the driver. I think so, if I have well understood
TI supports three cases of data capture: - DTC/Host collection (stop on buffer full) - DTC/Host collection (circular buffer) - ETB/on-chip collection (circular buffer) We don't support ETB/on-chip collection, We use external probe or Lauterbach combiprobe for trace data collection (I think it's circular buffer)
Of if the user is at a point in their code where they know thery will stop recoding on the hOst or ETB, we provide an IOCTL that simply disables all channels.
In the ETB case we may want to simply disable any open STM channels when the user decides to stop recording as a fail safe mechanism.
Note: Periodic transmission of meta data into a small circular buffer will not work well. In cases where the data is sparse the buffer will simply be filled with meta data rather than useful data.
11. OST Headers
Adding an OST header to each message is a requirement for compatibility with TI's toolchain. There are a couple of ways to approach:
Completely hidden from the user - The device specific code will know if the header is necessary. On a write, prior to the copy from user space, the device independent code would have to make a call to get a properly sized memory buffer from the device dependent code that would include the header.
User enabled - Provide an IOCTL that allows the user to put the driver in a tool-chain specific mode (like add OST headers). I agree, see my previous remark on OST headers
Regards, Doug Deao
I attached my last STM patches which introduce trace framework.
Best Regards, Philippe Langlais
________________________________________ From: Philippe Langlais [mailto:philippe.langlais@linaro.orgmailto:philippe.langlais@linaro.org] Sent: Wednesday, May 04, 2011 3:08 AM To: Deao, Douglas Cc: Linus Walleij Subject: Re: STM at UDS-Budapest
Hi Doug,
On STE ux500 platforms we have the same STM module (follow MIDP STP 1.0), I have already posted our current implementation to the LKML and Linaro ML, it's very similar to your proposal. I can't be present to the Linaro summit but Linus Walleij can replace me for this topic, he proposes to write a common trace framework to receive STM drivers. Attached all our current proposal and work around STM.
Regards Philippe Langlais ST-Ericsson On 3 May 2011 00:42, Deao, Douglas <d-deao@ti.commailto:d-deao@ti.com> wrote: I am hosting an introductory session on System Trace at the summit. TI's System Trace Module (STM) provides a common protocol for instrumentation messages across multiple cores and system level hardware profiling in complex SoCs. Attached is a whitepaper for background reading.
Looking forward to meeting you at the summit.
Regards, Doug Deao Texas Instruments
_______________________________________________ linaro-dev mailing list linaro-dev@lists.linaro.orgmailto:linaro-dev@lists.linaro.org http://lists.linaro.org/mailman/listinfo/linaro-dev
On 27 May 2011 10:45, Arnd Bergmann arnd@arndb.de wrote:
On Thursday 26 May 2011, Philippe Langlais wrote:
I initiate the work to build a hardware trace framework in the kernel,
I'm
not started the study to have a common userspace API for STM, thanks to this email we can start
such
a work, but it may be long (next week I'm in vacation till June 7th).
I missed the initial parts of the conversation, but I think that regarding the user interface, the most important issue by far is integration into the perf events infrastructure. There is no room for having yet another API to get at tracing and performance data.
I think that we misunderstood each other on the purpose of STM to particularly on the term "hardware trace": For more information see http://www.arm.com/products/system-ip/debug-trace/trace-macrocells-etm/cores... . This new user API is to manage trace resources (channels) and configuration of STM for the rest it's standard write only char driver to provide printf style debug interface. There is no interference with perf event infrastructure.
See my detailed response for all your interrogations and my thoughts
about
STE STM implementation below:
On 25 May 2011 23:54, Deao, Douglas d-deao@ti.com wrote:
Sorry it took a while to get back to you guys. I was visiting customers last week. Most of my comments are just highlighting the differences
between
TI's STM 1.0 driver and ST-E's STM 1.0 driver, but there are a few questions, observations and suggestions. At the end I included some discussion on TI's meta data and OST header requirements.
I have not had a chance to look at your actual implementation yet. Did
you
do anything to abstract the actual HW transport ports and control
registers
from the higher level driver functions?
Yes, partially I think through IOCTLs & debugfs (see our stm.h userspace API)
Any ioctls in addition to the ones defined in include/linux/perf_event.h will need to be coordinated with the perf developers.
Regarding a debugfs interface, you need to consider that all you cannot build stable interfaces in debugfs by definition. If you want to have long-term support for your subsystem, you would need a new file system.
OK
I am especially interested in details of what you guys have in mind for
a
"common trace framework to receive STM drivers". If by framework you
mean
well defined APIs that are implemented for specific devices, then I
think we
are in agreement. What Michael and I have talked about is a common STM
user
mode experience across all Linaro supported devices, making Linux user
mode
code 100% portable between our devices.
For my point of view, the trace device framework must ease the
integration
of new hardware trace drivers in the kernel (not only STM MIPI) to present standard hooks in current trace infrastructure, but it can cover a common STM userspace API too.
Can you explain the difference between hardware trace drivers and PMU drivers? What does the common infrastructure for trace drivers do that is not already handled by ftrace or perf?
STM is more an hardware to extract printf like traces and far less intrusive, it can be use instead of internal trace ring buffer to extract ftrace or perf results through an external probe.
I am thinking that for each unique pid a channel should be assigned
when
the device is opened. I would guess you are keeping a channel table
around
and write() just checks the table for a pid assignment (no time to look
at
your implementation yet), if none is found the first free channel is
used.
If you moved this function back to open then you could do the IOCTL STM_GET_CHANNEL_NO anytime, not just after the first write.
The reason behind this behavior is for our current STM user lib which
open
"/dev/stm" and alloc/free channels with IOCTL and never use write operation (only mmap + direct write) and in
this
case we don't want to loose a channel. I think we can change this behavior. We can support multiple channels allocated for one Process to avoid contention in multi-threaded process.
/dev/stm does not sound particularly hardware independent, i.e. it would not be portable to other architectures. The kernel interface should realy abstract such differences. Why do you need an extra character device besides the perf file descriptor?
Arnd
Philippe
Hi Philippe,
Some comments on stm-trace.txt, for points 1, 2, 3.
Maybe I can schedule a conference call this week to go over some of the notes, let me know.
Best regards, Arvind
-----Original Message----- From: Philippe Langlais [mailto:philippe.langlais@linaro.org] Sent: Thursday, May 26, 2011 8:50 PM To: Deao, Douglas; Linus Walleij; Lee Jones Cc: Arvind Chauhan; Michael Hope; linaro-dev@lists.linaro.org; pierre.peiffer@stericsson.com; loic.pallardy@stericsson.com Subject: Re: ST-E STM Driver Review
Hi Doug,
I initiate the work to build a hardware trace framework in the kernel, I'm not started the study to have a common userspace API for STM, thanks to this email we can start such a work, but it may be long (next week I'm in vacation till June 7th). See my detailed response for all your interrogations and my thoughts about STE STM implementation below: On 25 May 2011 23:54, Deao, Douglas d-deao@ti.com wrote: Sorry it took a while to get back to you guys. I was visiting customers last week. Most of my comments are just highlighting the differences between TI's STM 1.0 driver and ST-E's STM 1.0 driver, but there are a few questions, observations and suggestions. At the end I included some discussion on TI's meta data and OST header requirements.
I have not had a chance to look at your actual implementation yet. Did you do anything to abstract the actual HW transport ports and control registers from the higher level driver functions? Yes, partially I think through IOCTLs & debugfs (see our stm.h userspace API)
I realize there is a lot here to work through so if you would rather schedule a conference call to talk through the differences I can do that. I would like to start work on a Linaro (Unified) STM Spec next week if I can get feedback from everybody over the next few days. I will be out of the office on 5/27 and 5/31. I hope this email is enough.
I am especially interested in details of what you guys have in mind for a "common trace framework to receive STM drivers". If by framework you mean well defined APIs that are implemented for specific devices, then I think we are in agreement. What Michael and I have talked about is a common STM user mode experience across all Linaro supported devices, making Linux user mode code 100% portable between our devices. For my point of view, the trace device framework must ease the integration of new hardware trace drivers in the kernel (not only STM MIPI) to present standard hooks in current trace infrastructure, but it can cover a common STM userspace API too.
ST-E STM Driver stm-trace.txt review:
- Software Overview
In your "Software Overview" it states:
"The end of data packet is marked by a time stamp on latest byte(s) only."
I assume that user messages can be made up of any number of bytes, half- words, words or longs (what ever is most efficient) and you simply terminate the last element of the message with a time-stamp - right? Yes, the message buffer can be mis aligned
So the question is that should the driver do packetizing or should this be left to the tracing framework on top of STM driver (time-stamping the last element of the message can be left to the intermediate logic). In my view, STM driver should be a low level entity and provide simpler interface in terms of raw writes of 32bits or multiples - this may allow tracing framework to directly dump information over STM instead of buffering in between, making it less intrusive.
In the TI STM implementation a message can be any number and combination of bytes, half-words, or word transfers terminated with a time-stamp on the last element. In addition to that we also add an OST header to a message. (See below for discussion on OST header). In our case the OST header is added by the external capture probe.
Same as above, I think OST should be sitting on top of STM driver instead of being part of it - Allowing non-OST implementations to coexist.
- Lossless/Lossy modes.
TI only supports lossless mode for sw generated messages and is enforced in our hw implementation. Lossy mode is reserved for true hw messages. For STE, hw messages are always lossy, but sw generated messages could be configured in lossless (default) or lossy mode.
In my view, STM driver should expose all functionalities of the underlying hardware - leaving guaranteed/invariant choice to the tracing framework on top.
I did not notice that you documented a way to modify this through the debugfs API or IOCTLS. I have 2 IOCTLs (STM_SET_MODE & STM_GET_MODE) and debugfs (masters_modes) interfaces for that.
We expose channels through debugfs interface on demand basis. Channel allocation creates 2 debugfs nodes on STM mount point, one used for dumping trace data and other for configuring modes like guaranteed/invariant/timestamp etc. for the channel.
I am kind of thinking that may be ok since this is really a hw configuration choice in your case, but in the TI case the user does not get to make that choice. OK
- Channel Assignment
TI makes the assignment with mknod using the minor number to assign a fixed channel. This allows the user mode application to overload the channel usage for categorizing data (not my idea). I think we see the error of our ways here and will be ok with a dynamic channel allocation. If we are agree with dynamic channel allocation then a common STM userspace API is possible I think and perhaps common STM driver.
mknod would allow only 256 channels. We use debugfs nodes to allocate any number of channels, which appears automatically in debugfs mount point. When programatically accessing STM i/f we allow apps to allocate channels in terms of blocks using mmap node. When an app calls mmap API, we attach a page fault handler to the mapped region, thus when for the first time app touches this area it generates a page fault. In the handler, we allocate a block if available or preempt it from some dormant process and attach to the faulting VMA.
I am thinking that for each unique pid a channel should be assigned when the device is opened. I would guess you are keeping a channel table around and write() just checks the table for a pid assignment (no time to look at your implementation yet), if none is found the first free channel is used. If you moved this function back to open then you could do the IOCTL STM_GET_CHANNEL_NO anytime, not just after the first write. The reason behind this behavior is for our current STM user lib which open "/dev/stm" and alloc/free channels with IOCTL and never use write operation (only mmap + direct write) and in this case we don't want to loose a channel. I think we can change this behavior. We can support multiple channels allocated for one Process to avoid contention in multi-threaded process.
In write how do you flag an error if you exhaust the number of available channels? I return -ENOMEM.
We don't return error, we preempt blocks from other process
- Kernel API
TI does not support a Kernel API (yet). I can see that the Alloc/Free and File IO type functions are useful and should be standard.
Not sure what you mean by "lockless" trace functions? Not protected against concurrent access, it's the responsibility to the caller.
It looks like your "low level atomic trace functions for 1, 2, 4 or 8 bytes" is similar to TI's binary library functions (not supported by the TI STM Driver). This is what we use the OST header for, allowing our tool chain to differentiate between different message formats, rather than just assuming the data is a simple stream of bytes. I'm not ready to add an OST header to each trace messages to particularly for kernel function traces (ftrace has short messages 8 bytes size with high throughput), I rather to dedicate a specific channel for this, OST header has to be optional.
- Debugfs APIs.
TI used a different approach. The tool-chain on the host provides all the transport setup through JTAG, so our driver does not support setting up the actual STM data export (number of pins and clock rate). In our case device transport parameters must match the host receiver's collection setup.
With your approach the user can change the clock rate and export pin width effectively at any time. Our tools actually go through a calibration process during initialization so any changes to the device's transport setup (clock rate, number of pins data exported on) would cause the TI tool chain a lot of grief.
There are some parameters we know we need to add (like master enables). This are currently also handled by the host tools. TI's STM module allows up to 4 SW masters to be enabled (with id masks that can be used to enable multiple masters from the same group) and 4 HW masters that can be enabled at the same time as the SW masters. If the user tries to enable more than the HW allows do you have a mechanism to flag an error? We can't face the same problem with STE STM, our hardware support 6 SW masters & 2 HW masters without restriction, all could be enabled at same time (managed by a bit field register)
I don't have a lot of experience with debugfs but I am assuming it's primarily used for allowing scripts to configure a driver (like in your example) or extract information. It's the current way to configure kernel tracing infrastructure (from console, through scripts or user apps), kernel users are familiar with this interface.
We may want to define a standard set of debugfs options whose implementation is vendor specific. But that raises some questions:
- How do we deal with options that don't make sense for a specific
vendor? Maybe just doing nothing is acceptable or do we want to provide a discovery mechanism? Do nothing may be sufficient, we can add STM IP specific string descriptor too, something like "vendor id+STM IP id+STP version"
- Would user scripts then also be vendor specific?
We should probably make an effort to avoid this. A discovery mechanism may allow user mode scripts to be generic. Yes, but STM configuration scripts are device specific, to see.
- Mapped Channels
I believe the TI hw transport channel mapping is compatible. In the TI case a channel is mapped into two spaces, the first half is for non- timestamp transfers and the second half is for time-stamped transfers. When we write a message (from a user mode write call for example) we simply write all the data except the last element through the non- timestamp port, and then the last element is written to the time-stamped port. So I think we could be compatible here. I think so, we have similar channel memory mapping but, one STM channel is accessed through this C union of 2 consecutive 64bits word (first for non-timestamp, 2nd with timestamp) /* One single channel mapping */ struct stm_channel { union { __u8 no_stamp8; __u16 no_stamp16; __u32 no_stamp32; __u64 no_stamp64; }; union { __u8 stamp8; __u16 stamp16; __u32 stamp32; __u64 stamp64; }; };
With that said I am not sure about exposing all channels to a user mode library. You are relying on the library to use the convention of getting a free channel from the driver to make sure there are no conflicts. If the channel assignment is made when you open the device, you could conceivably map just the address space needed for the single channel, thus eliminating the need to get a free channel from the driver. In the TI case a single channel's transport mapping is 4K bytes, which matches the typical PAGE_SIZE. I realize not all hw implementations will match up with the PAGE_SIZE, which may be why you simply map all the channels back to user space. It's our case, the 256 channels is mapped in the same 4K memory page.
Since free channels can become busy rapidly, maybe a better convention would be to simply use another device node if the user wants the library STM data to be transmitted on a different STM channel than the current process. This may be a case where providing a mechanism (see meta data discussion below) to allow channels to be named for the toolchain may be a good idea (provide task name and process id). In our current implementation, a new open on the same "/dev/stm" device allocate a new channel at first write.
- 8-byte Writes
TI does not support 64-bit writes with our STM 1.0 module. We may need an IOCTL to get the largest transfer supported for the mmap case. For all other cases this should just be hidden in the device dependent code. I think so
- Kernel Internal Usage
I like the idea of having dedicated support in the driver for common kernel logging. Any ideas on how you would support kernel STM channel assignments without hard-coding? But configurable through kernel configuration, the other way is to dedicate a specific channel for this signalization like your channel 255, not so easy to define.
We may need a mechanism to communicate the definition of each hard-coded channel to our tools. to see
The following are TI specific:
- Data protection
In SMP systems if the processor is switched a new master is generated (in some TI devices). So we protect the data with a mutex to guarantee a complete message is generated by the same master. OK, it's not our case.
- Meta Data
Our user mode HW libraries use meta data to transport data needed to process the HW profiling STM messages. Items like processor speed, sampling rate, processing options, ... (just a predefined byte buffer our tool-chain understands). The meta data is currently broadcasted on a dedicated channel (255), which conflicts with your hard-coded channel for logging printk output. So we will need to resolve hard-coded conflicts. Easy to fix, we can reserve dedicated channels at driver init and change our hard-coded affectation.
We need the driver to support registration and transport of the meta data on demand from the library (when the HW master is disabled, in case the collection buffer is small and circular).
I am thinking an IOCTL could be used to register meta data and then the data simply broadcast on a STM channel (will need to figure out which one) when the HW master is enabled and disabled. We don't face the same problematic
Meta data transmission is problematic for circular buffers (like ETB's) thus the reason for also sending meta data when a hw master is disabled. SW masters are not typically disabled, and our HW does not provide a transmission byte count (remember there are HW messages also being generated in the TI case). So there is no way from a driver we can tell when the recoding buffer will wrap even if the user told us the buffer size. I am thinking the best solution would be to force the user to gracefully disable the channel to get any sw channel meta data provided by the driver. I think so, if I have well understood
TI supports three cases of data capture:
- DTC/Host collection (stop on buffer full)
- DTC/Host collection (circular buffer)
- ETB/on-chip collection (circular buffer)
We don't support ETB/on-chip collection, We use external probe or Lauterbach combiprobe for trace data collection (I think it's circular buffer)
Of if the user is at a point in their code where they know thery will stop recoding on the hOst or ETB, we provide an IOCTL that simply disables all channels.
In the ETB case we may want to simply disable any open STM channels when the user decides to stop recording as a fail safe mechanism.
Note: Periodic transmission of meta data into a small circular buffer will not work well. In cases where the data is sparse the buffer will simply be filled with meta data rather than useful data.
- OST Headers
Adding an OST header to each message is a requirement for compatibility with TI's toolchain. There are a couple of ways to approach:
Completely hidden from the user - The device specific code will know if the header is necessary. On a write, prior to the copy from user space, the device independent code would have to make a call to get a properly sized memory buffer from the device dependent code that would include the header.
User enabled - Provide an IOCTL that allows the user to put the driver in a tool-chain specific mode (like add OST headers). I agree, see my previous remark on OST headers
Regards, Doug Deao
I attached my last STM patches which introduce trace framework.
Best Regards, Philippe Langlais
From: Philippe Langlais [mailto:philippe.langlais@linaro.org] Sent: Wednesday, May 04, 2011 3:08 AM To: Deao, Douglas Cc: Linus Walleij Subject: Re: STM at UDS-Budapest
Hi Doug,
On STE ux500 platforms we have the same STM module (follow MIDP STP 1.0), I have already posted our current implementation to the LKML and Linaro ML, it's very similar to your proposal. I can't be present to the Linaro summit but Linus Walleij can replace me for this topic, he proposes to write a common trace framework to receive STM drivers. Attached all our current proposal and work around STM.
Regards Philippe Langlais ST-Ericsson On 3 May 2011 00:42, Deao, Douglas d-deao@ti.com wrote: I am hosting an introductory session on System Trace at the summit. TI's System Trace Module (STM) provides a common protocol for instrumentation messages across multiple cores and system level hardware profiling in complex SoCs. Attached is a whitepaper for background reading.
Looking forward to meeting you at the summit.
Regards, Doug Deao Texas Instruments
linaro-dev mailing list linaro-dev@lists.linaro.org http://lists.linaro.org/mailman/listinfo/linaro-dev
-- IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
Arvind,
1. It's a fundamental requirement that the driver supports redirected IO (stdout) and write() calls like any standard character driver. This makes STM really easy to use for most user applications that simply want to use existing printf() or write() calls to export logging messages without doing a lot of integration work. Plus this is what most of our customers are expecting and already using.
2. Device drivers should abstract the underlining hardware from user space applications, thus the reason we would like to see file operations standardized. With that said we are also ok with exposing the STM transport (through mmap) to user space supporting vendor specific libraries. I think we cover all our individual requirements this way.
3. Yes, we realize that STM channels need to be allocated dynamically from the driver.
I have started writing a generic STM specification document. Before I get much farther I think it's important we agree on requirements and architecture. I am trying to accommodate TI, STE and ARM capabilities and preserve our existing host tool chain methods. Will try to send out what I have so far tomorrow, and then schedule a call for early next week.
Have you guys made any progress on releasing your current STM driver?
Regards, Doug
-----Original Message----- From: Arvind Chauhan [mailto:Arvind.Chauhan@arm.com] Sent: Monday, June 20, 2011 7:58 AM To: Philippe Langlais; Deao, Douglas; Linus Walleij; Lee Jones Cc: Michael Hope; linaro-dev@lists.linaro.org; pierre.peiffer@stericsson.com; loic.pallardy@stericsson.com Subject: RE: ST-E STM Driver Review
Hi Philippe,
Some comments on stm-trace.txt, for points 1, 2, 3.
Maybe I can schedule a conference call this week to go over some of the notes, let me know.
Best regards, Arvind
-----Original Message----- From: Philippe Langlais [mailto:philippe.langlais@linaro.org] Sent: Thursday, May 26, 2011 8:50 PM To: Deao, Douglas; Linus Walleij; Lee Jones Cc: Arvind Chauhan; Michael Hope; linaro-dev@lists.linaro.org; pierre.peiffer@stericsson.com; loic.pallardy@stericsson.com Subject: Re: ST-E STM Driver Review
Hi Doug,
I initiate the work to build a hardware trace framework in the kernel, I'm not started the study to have a common userspace API for STM, thanks to this email we can start such a work, but it may be long (next week I'm in vacation till June 7th). See my detailed response for all your interrogations and my thoughts about STE STM implementation below: On 25 May 2011 23:54, Deao, Douglas d-deao@ti.com wrote: Sorry it took a while to get back to you guys. I was visiting customers last week. Most of my comments are just highlighting the differences between TI's STM 1.0 driver and ST-E's STM 1.0 driver, but there are a few questions, observations and suggestions. At the end I included some discussion on TI's meta data and OST header requirements.
I have not had a chance to look at your actual implementation yet. Did you do anything to abstract the actual HW transport ports and control registers from the higher level driver functions? Yes, partially I think through IOCTLs & debugfs (see our stm.h userspace API)
I realize there is a lot here to work through so if you would rather schedule a conference call to talk through the differences I can do that. I would like to start work on a Linaro (Unified) STM Spec next week if I can get feedback from everybody over the next few days. I will be out of the office on 5/27 and 5/31. I hope this email is enough.
I am especially interested in details of what you guys have in mind for a "common trace framework to receive STM drivers". If by framework you mean well defined APIs that are implemented for specific devices, then I think we are in agreement. What Michael and I have talked about is a common STM user mode experience across all Linaro supported devices, making Linux user mode code 100% portable between our devices. For my point of view, the trace device framework must ease the integration of new hardware trace drivers in the kernel (not only STM MIPI) to present standard hooks in current trace infrastructure, but it can cover a common STM userspace API too.
ST-E STM Driver stm-trace.txt review:
- Software Overview
In your "Software Overview" it states:
"The end of data packet is marked by a time stamp on latest byte(s) only."
I assume that user messages can be made up of any number of bytes, half- words, words or longs (what ever is most efficient) and you simply terminate the last element of the message with a time-stamp - right? Yes, the message buffer can be mis aligned
So the question is that should the driver do packetizing or should this be left to the tracing framework on top of STM driver (time-stamping the last element of the message can be left to the intermediate logic). In my view, STM driver should be a low level entity and provide simpler interface in terms of raw writes of 32bits or multiples - this may allow tracing framework to directly dump information over STM instead of buffering in between, making it less intrusive.
In the TI STM implementation a message can be any number and combination of bytes, half-words, or word transfers terminated with a time-stamp on the last element. In addition to that we also add an OST header to a message. (See below for discussion on OST header). In our case the OST header is added by the external capture probe.
Same as above, I think OST should be sitting on top of STM driver instead of being part of it - Allowing non-OST implementations to coexist.
- Lossless/Lossy modes.
TI only supports lossless mode for sw generated messages and is enforced in our hw implementation. Lossy mode is reserved for true hw messages. For STE, hw messages are always lossy, but sw generated messages could be configured in lossless (default) or lossy mode.
In my view, STM driver should expose all functionalities of the underlying hardware - leaving guaranteed/invariant choice to the tracing framework on top.
I did not notice that you documented a way to modify this through the debugfs API or IOCTLS. I have 2 IOCTLs (STM_SET_MODE & STM_GET_MODE) and debugfs (masters_modes) interfaces for that.
We expose channels through debugfs interface on demand basis. Channel allocation creates 2 debugfs nodes on STM mount point, one used for dumping trace data and other for configuring modes like guaranteed/invariant/timestamp etc. for the channel.
I am kind of thinking that may be ok since this is really a hw configuration choice in your case, but in the TI case the user does not get to make that choice. OK
- Channel Assignment
TI makes the assignment with mknod using the minor number to assign a fixed channel. This allows the user mode application to overload the channel usage for categorizing data (not my idea). I think we see the error of our ways here and will be ok with a dynamic channel allocation. If we are agree with dynamic channel allocation then a common STM userspace API is possible I think and perhaps common STM driver.
mknod would allow only 256 channels. We use debugfs nodes to allocate any number of channels, which appears automatically in debugfs mount point. When programatically accessing STM i/f we allow apps to allocate channels in terms of blocks using mmap node. When an app calls mmap API, we attach a page fault handler to the mapped region, thus when for the first time app touches this area it generates a page fault. In the handler, we allocate a block if available or preempt it from some dormant process and attach to the faulting VMA.
I am thinking that for each unique pid a channel should be assigned when the device is opened. I would guess you are keeping a channel table around and write() just checks the table for a pid assignment (no time to look at your implementation yet), if none is found the first free channel is used. If you moved this function back to open then you could do the IOCTL STM_GET_CHANNEL_NO anytime, not just after the first write. The reason behind this behavior is for our current STM user lib which open "/dev/stm" and alloc/free channels with IOCTL and never use write operation (only mmap + direct write) and in this case we don't want to loose a channel. I think we can change this behavior. We can support multiple channels allocated for one Process to avoid contention in multi-threaded process.
In write how do you flag an error if you exhaust the number of available channels? I return -ENOMEM.
We don't return error, we preempt blocks from other process
- Kernel API
TI does not support a Kernel API (yet). I can see that the Alloc/Free and File IO type functions are useful and should be standard.
Not sure what you mean by "lockless" trace functions? Not protected against concurrent access, it's the responsibility to the caller.
It looks like your "low level atomic trace functions for 1, 2, 4 or 8 bytes" is similar to TI's binary library functions (not supported by the TI STM Driver). This is what we use the OST header for, allowing our tool chain to differentiate between different message formats, rather than just assuming the data is a simple stream of bytes. I'm not ready to add an OST header to each trace messages to particularly for kernel function traces (ftrace has short messages 8 bytes size with high throughput), I rather to dedicate a specific channel for this, OST header has to be optional.
- Debugfs APIs.
TI used a different approach. The tool-chain on the host provides all the transport setup through JTAG, so our driver does not support setting up the actual STM data export (number of pins and clock rate). In our case device transport parameters must match the host receiver's collection setup.
With your approach the user can change the clock rate and export pin width effectively at any time. Our tools actually go through a calibration process during initialization so any changes to the device's transport setup (clock rate, number of pins data exported on) would cause the TI tool chain a lot of grief.
There are some parameters we know we need to add (like master enables). This are currently also handled by the host tools. TI's STM module allows up to 4 SW masters to be enabled (with id masks that can be used to enable multiple masters from the same group) and 4 HW masters that can be enabled at the same time as the SW masters. If the user tries to enable more than the HW allows do you have a mechanism to flag an error? We can't face the same problem with STE STM, our hardware support 6 SW masters & 2 HW masters without restriction, all could be enabled at same time (managed by a bit field register)
I don't have a lot of experience with debugfs but I am assuming it's primarily used for allowing scripts to configure a driver (like in your example) or extract information. It's the current way to configure kernel tracing infrastructure (from console, through scripts or user apps), kernel users are familiar with this interface.
We may want to define a standard set of debugfs options whose implementation is vendor specific. But that raises some questions:
- How do we deal with options that don't make sense for a specific
vendor? Maybe just doing nothing is acceptable or do we want to provide a discovery mechanism? Do nothing may be sufficient, we can add STM IP specific string descriptor too, something like "vendor id+STM IP id+STP version"
- Would user scripts then also be vendor specific?
We should probably make an effort to avoid this. A discovery mechanism may allow user mode scripts to be generic. Yes, but STM configuration scripts are device specific, to see.
- Mapped Channels
I believe the TI hw transport channel mapping is compatible. In the TI case a channel is mapped into two spaces, the first half is for non- timestamp transfers and the second half is for time-stamped transfers. When we write a message (from a user mode write call for example) we simply write all the data except the last element through the non- timestamp port, and then the last element is written to the time-stamped port. So I think we could be compatible here. I think so, we have similar channel memory mapping but, one STM channel is accessed through this C union of 2 consecutive 64bits word (first for non-timestamp, 2nd with timestamp) /* One single channel mapping */ struct stm_channel { union { __u8 no_stamp8; __u16 no_stamp16; __u32 no_stamp32; __u64 no_stamp64; }; union { __u8 stamp8; __u16 stamp16; __u32 stamp32; __u64 stamp64; }; };
With that said I am not sure about exposing all channels to a user mode library. You are relying on the library to use the convention of getting a free channel from the driver to make sure there are no conflicts. If the channel assignment is made when you open the device, you could conceivably map just the address space needed for the single channel, thus eliminating the need to get a free channel from the driver. In the TI case a single channel's transport mapping is 4K bytes, which matches the typical PAGE_SIZE. I realize not all hw implementations will match up with the PAGE_SIZE, which may be why you simply map all the channels back to user space. It's our case, the 256 channels is mapped in the same 4K memory page.
Since free channels can become busy rapidly, maybe a better convention would be to simply use another device node if the user wants the library STM data to be transmitted on a different STM channel than the current process. This may be a case where providing a mechanism (see meta data discussion below) to allow channels to be named for the toolchain may be a good idea (provide task name and process id). In our current implementation, a new open on the same "/dev/stm" device allocate a new channel at first write.
- 8-byte Writes
TI does not support 64-bit writes with our STM 1.0 module. We may need an IOCTL to get the largest transfer supported for the mmap case. For all other cases this should just be hidden in the device dependent code. I think so
- Kernel Internal Usage
I like the idea of having dedicated support in the driver for common kernel logging. Any ideas on how you would support kernel STM channel assignments without hard-coding? But configurable through kernel configuration, the other way is to dedicate a specific channel for this signalization like your channel 255, not so easy to define.
We may need a mechanism to communicate the definition of each hard-coded channel to our tools. to see
The following are TI specific:
- Data protection
In SMP systems if the processor is switched a new master is generated (in some TI devices). So we protect the data with a mutex to guarantee a complete message is generated by the same master. OK, it's not our case.
- Meta Data
Our user mode HW libraries use meta data to transport data needed to process the HW profiling STM messages. Items like processor speed, sampling rate, processing options, ... (just a predefined byte buffer our tool-chain understands). The meta data is currently broadcasted on a dedicated channel (255), which conflicts with your hard-coded channel for logging printk output. So we will need to resolve hard-coded conflicts. Easy to fix, we can reserve dedicated channels at driver init and change our hard-coded affectation.
We need the driver to support registration and transport of the meta data on demand from the library (when the HW master is disabled, in case the collection buffer is small and circular).
I am thinking an IOCTL could be used to register meta data and then the data simply broadcast on a STM channel (will need to figure out which one) when the HW master is enabled and disabled. We don't face the same problematic
Meta data transmission is problematic for circular buffers (like ETB's) thus the reason for also sending meta data when a hw master is disabled. SW masters are not typically disabled, and our HW does not provide a transmission byte count (remember there are HW messages also being generated in the TI case). So there is no way from a driver we can tell when the recoding buffer will wrap even if the user told us the buffer size. I am thinking the best solution would be to force the user to gracefully disable the channel to get any sw channel meta data provided by the driver. I think so, if I have well understood
TI supports three cases of data capture:
- DTC/Host collection (stop on buffer full)
- DTC/Host collection (circular buffer)
- ETB/on-chip collection (circular buffer)
We don't support ETB/on-chip collection, We use external probe or Lauterbach combiprobe for trace data collection (I think it's circular buffer)
Of if the user is at a point in their code where they know thery will stop recoding on the hOst or ETB, we provide an IOCTL that simply disables all channels.
In the ETB case we may want to simply disable any open STM channels when the user decides to stop recording as a fail safe mechanism.
Note: Periodic transmission of meta data into a small circular buffer will not work well. In cases where the data is sparse the buffer will simply be filled with meta data rather than useful data.
- OST Headers
Adding an OST header to each message is a requirement for compatibility with TI's toolchain. There are a couple of ways to approach:
Completely hidden from the user - The device specific code will know if the header is necessary. On a write, prior to the copy from user space, the device independent code would have to make a call to get a properly sized memory buffer from the device dependent code that would include the header.
User enabled - Provide an IOCTL that allows the user to put the driver in a tool-chain specific mode (like add OST headers). I agree, see my previous remark on OST headers
Regards, Doug Deao
I attached my last STM patches which introduce trace framework.
Best Regards, Philippe Langlais
From: Philippe Langlais [mailto:philippe.langlais@linaro.org] Sent: Wednesday, May 04, 2011 3:08 AM To: Deao, Douglas Cc: Linus Walleij Subject: Re: STM at UDS-Budapest
Hi Doug,
On STE ux500 platforms we have the same STM module (follow MIDP STP 1.0), I have already posted our current implementation to the LKML and Linaro ML, it's very similar to your proposal. I can't be present to the Linaro summit but Linus Walleij can replace me for this topic, he proposes to write a common trace framework to receive STM drivers. Attached all our current proposal and work around STM.
Regards Philippe Langlais ST-Ericsson On 3 May 2011 00:42, Deao, Douglas d-deao@ti.com wrote: I am hosting an introductory session on System Trace at the summit. TI's System Trace Module (STM) provides a common protocol for instrumentation messages across multiple cores and system level hardware profiling in complex SoCs. Attached is a whitepaper for background reading.
Looking forward to meeting you at the summit.
Regards, Doug Deao Texas Instruments
linaro-dev mailing list linaro-dev@lists.linaro.org http://lists.linaro.org/mailman/listinfo/linaro-dev
-- IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
Hi,
As stated by Douglas, a character driver is more natural for a printf like usage, therefore we need a mean to do packetizing (with optional OST header: can be supported by generic functions in kernel hw tracing framework). My current implementation use only one device node (/dev/stm) for all STM channels (no minor<256 restriction even for STM v2.0), each "open+write" on this device node allocate dynamically a new channel and IOCTLs are available to retrieve channel number. This driver covers low level interface too (direct write with no overhead from 1 to 8 bytes with mmap + IOCTLs to allocate/free/configure channels) for a usage with an optimized user space library.
The hardware tracing framework, I started to develop, only covers trace device register/unregister and offers a set of common hooks to the standard kernel tracing infrastructure, all things traced into trace ring buffer (ftrace) could be redirected in real time towards dedicated STM channels, it's only a draft, we can add OST header management here and others features.
I attach my current user+kernel STM interface. Now, I'm waiting of Doug's generic STM specification document for review and for pursue STM driver development for future upstreaming.
Best regards, Philippe
On 20 June 2011 14:57, Arvind Chauhan Arvind.Chauhan@arm.com wrote:
Hi Philippe,
Some comments on stm-trace.txt, for points 1, 2, 3.
Maybe I can schedule a conference call this week to go over some of the notes, let me know.
Best regards, Arvind
-----Original Message----- From: Philippe Langlais [mailto:philippe.langlais@linaro.org] Sent: Thursday, May 26, 2011 8:50 PM To: Deao, Douglas; Linus Walleij; Lee Jones Cc: Arvind Chauhan; Michael Hope; linaro-dev@lists.linaro.org; pierre.peiffer@stericsson.com; loic.pallardy@stericsson.com Subject: Re: ST-E STM Driver Review
Hi Doug,
I initiate the work to build a hardware trace framework in the kernel, I'm not started the study to have a common userspace API for STM, thanks to this email we can start such a work, but it may be long (next week I'm in vacation till June 7th). See my detailed response for all your interrogations and my thoughts about STE STM implementation below: On 25 May 2011 23:54, Deao, Douglas d-deao@ti.com wrote: Sorry it took a while to get back to you guys. I was visiting customers last week. Most of my comments are just highlighting the differences between TI's STM 1.0 driver and ST-E's STM 1.0 driver, but there are a few questions, observations and suggestions. At the end I included some discussion on TI's meta data and OST header requirements.
I have not had a chance to look at your actual implementation yet. Did you do anything to abstract the actual HW transport ports and control registers from the higher level driver functions? Yes, partially I think through IOCTLs & debugfs (see our stm.h userspace API)
I realize there is a lot here to work through so if you would rather schedule a conference call to talk through the differences I can do that. I would like to start work on a Linaro (Unified) STM Spec next week if I can get feedback from everybody over the next few days. I will be out of the office on 5/27 and 5/31. I hope this email is enough.
I am especially interested in details of what you guys have in mind for a "common trace framework to receive STM drivers". If by framework you mean well defined APIs that are implemented for specific devices, then I think we are in agreement. What Michael and I have talked about is a common STM user mode experience across all Linaro supported devices, making Linux user mode code 100% portable between our devices. For my point of view, the trace device framework must ease the integration of new hardware trace drivers in the kernel (not only STM MIPI) to present standard hooks in current trace infrastructure, but it can cover a common STM userspace API too.
ST-E STM Driver stm-trace.txt review:
- Software Overview
In your "Software Overview" it states:
"The end of data packet is marked by a time stamp on latest byte(s) only."
I assume that user messages can be made up of any number of bytes, half- words, words or longs (what ever is most efficient) and you simply terminate the last element of the message with a time-stamp - right? Yes, the message buffer can be mis aligned
So the question is that should the driver do packetizing or should this be left to the tracing framework on top of STM driver (time-stamping the last element of the message can be left to the intermediate logic). In my view, STM driver should be a low level entity and provide simpler interface in terms of raw writes of 32bits or multiples - this may allow tracing framework to directly dump information over STM instead of buffering in between, making it less intrusive.
In the TI STM implementation a message can be any number and combination of bytes, half-words, or word transfers terminated with a time-stamp on the last element. In addition to that we also add an OST header to a message. (See below for discussion on OST header). In our case the OST header is added by the external capture probe.
Same as above, I think OST should be sitting on top of STM driver instead of being part of it - Allowing non-OST implementations to coexist.
- Lossless/Lossy modes.
TI only supports lossless mode for sw generated messages and is enforced in our hw implementation. Lossy mode is reserved for true hw messages. For STE, hw messages are always lossy, but sw generated messages could be configured in lossless (default) or lossy mode.
In my view, STM driver should expose all functionalities of the underlying hardware - leaving guaranteed/invariant choice to the tracing framework on top.
I did not notice that you documented a way to modify this through the debugfs API or IOCTLS. I have 2 IOCTLs (STM_SET_MODE & STM_GET_MODE) and debugfs (masters_modes) interfaces for that.
We expose channels through debugfs interface on demand basis. Channel allocation creates 2 debugfs nodes on STM mount point, one used for dumping trace data and other for configuring modes like guaranteed/invariant/timestamp etc. for the channel.
I am kind of thinking that may be ok since this is really a hw configuration choice in your case, but in the TI case the user does not get to make that choice. OK
- Channel Assignment
TI makes the assignment with mknod using the minor number to assign a fixed channel. This allows the user mode application to overload the channel usage for categorizing data (not my idea). I think we see the error of our ways here and will be ok with a dynamic channel allocation. If we are agree with dynamic channel allocation then a common STM userspace API is possible I think and perhaps common STM driver.
mknod would allow only 256 channels. We use debugfs nodes to allocate any number of channels, which appears automatically in debugfs mount point. When programatically accessing STM i/f we allow apps to allocate channels in terms of blocks using mmap node. When an app calls mmap API, we attach a page fault handler to the mapped region, thus when for the first time app touches this area it generates a page fault. In the handler, we allocate a block if available or preempt it from some dormant process and attach to the faulting VMA.
I am thinking that for each unique pid a channel should be assigned when the device is opened. I would guess you are keeping a channel table around and write() just checks the table for a pid assignment (no time to look at your implementation yet), if none is found the first free channel is used. If you moved this function back to open then you could do the IOCTL STM_GET_CHANNEL_NO anytime, not just after the first write. The reason behind this behavior is for our current STM user lib which open "/dev/stm" and alloc/free channels with IOCTL and never use write operation (only mmap + direct write) and in this case we don't want to loose a channel. I think we can change this behavior. We can support multiple channels allocated for one Process to avoid contention in multi-threaded process.
In write how do you flag an error if you exhaust the number of available channels? I return -ENOMEM.
We don't return error, we preempt blocks from other process
- Kernel API
TI does not support a Kernel API (yet). I can see that the Alloc/Free and File IO type functions are useful and should be standard.
Not sure what you mean by "lockless" trace functions? Not protected against concurrent access, it's the responsibility to the caller.
It looks like your "low level atomic trace functions for 1, 2, 4 or 8 bytes" is similar to TI's binary library functions (not supported by the TI STM Driver). This is what we use the OST header for, allowing our tool chain to differentiate between different message formats, rather than just assuming the data is a simple stream of bytes. I'm not ready to add an OST header to each trace messages to particularly for kernel function traces (ftrace has short messages 8 bytes size with high throughput), I rather to dedicate a specific channel for this, OST header has to be optional.
- Debugfs APIs.
TI used a different approach. The tool-chain on the host provides all the transport setup through JTAG, so our driver does not support setting up the actual STM data export (number of pins and clock rate). In our case device transport parameters must match the host receiver's collection setup.
With your approach the user can change the clock rate and export pin width effectively at any time. Our tools actually go through a calibration process during initialization so any changes to the device's transport setup (clock rate, number of pins data exported on) would cause the TI tool chain a lot of grief.
There are some parameters we know we need to add (like master enables). This are currently also handled by the host tools. TI's STM module allows up to 4 SW masters to be enabled (with id masks that can be used to enable multiple masters from the same group) and 4 HW masters that can be enabled at the same time as the SW masters. If the user tries to enable more than the HW allows do you have a mechanism to flag an error? We can't face the same problem with STE STM, our hardware support 6 SW masters & 2 HW masters without restriction, all could be enabled at same time (managed by a bit field register)
I don't have a lot of experience with debugfs but I am assuming it's primarily used for allowing scripts to configure a driver (like in your example) or extract information. It's the current way to configure kernel tracing infrastructure (from console, through scripts or user apps), kernel users are familiar with this interface.
We may want to define a standard set of debugfs options whose implementation is vendor specific. But that raises some questions:
- How do we deal with options that don't make sense for a specific
vendor? Maybe just doing nothing is acceptable or do we want to provide a discovery mechanism? Do nothing may be sufficient, we can add STM IP specific string descriptor too, something like "vendor id+STM IP id+STP version"
- Would user scripts then also be vendor specific?
We should probably make an effort to avoid this. A discovery mechanism may allow user mode scripts to be generic. Yes, but STM configuration scripts are device specific, to see.
- Mapped Channels
I believe the TI hw transport channel mapping is compatible. In the TI case a channel is mapped into two spaces, the first half is for non- timestamp transfers and the second half is for time-stamped transfers. When we write a message (from a user mode write call for example) we simply write all the data except the last element through the non- timestamp port, and then the last element is written to the time-stamped port. So I think we could be compatible here. I think so, we have similar channel memory mapping but, one STM channel is accessed through this C union of 2 consecutive 64bits word (first for non-timestamp, 2nd with timestamp) /* One single channel mapping */ struct stm_channel { union { __u8 no_stamp8; __u16 no_stamp16; __u32 no_stamp32; __u64 no_stamp64; }; union { __u8 stamp8; __u16 stamp16; __u32 stamp32; __u64 stamp64; }; };
With that said I am not sure about exposing all channels to a user mode library. You are relying on the library to use the convention of getting a free channel from the driver to make sure there are no conflicts. If the channel assignment is made when you open the device, you could conceivably map just the address space needed for the single channel, thus eliminating the need to get a free channel from the driver. In the TI case a single channel's transport mapping is 4K bytes, which matches the typical PAGE_SIZE. I realize not all hw implementations will match up with the PAGE_SIZE, which may be why you simply map all the channels back to user space. It's our case, the 256 channels is mapped in the same 4K memory page.
Since free channels can become busy rapidly, maybe a better convention would be to simply use another device node if the user wants the library STM data to be transmitted on a different STM channel than the current process. This may be a case where providing a mechanism (see meta data discussion below) to allow channels to be named for the toolchain may be a good idea (provide task name and process id). In our current implementation, a new open on the same "/dev/stm" device allocate a new channel at first write.
- 8-byte Writes
TI does not support 64-bit writes with our STM 1.0 module. We may need an IOCTL to get the largest transfer supported for the mmap case. For all other cases this should just be hidden in the device dependent code. I think so
- Kernel Internal Usage
I like the idea of having dedicated support in the driver for common kernel logging. Any ideas on how you would support kernel STM channel assignments without hard-coding? But configurable through kernel configuration, the other way is to dedicate a specific channel for this signalization like your channel 255, not so easy to define.
We may need a mechanism to communicate the definition of each hard-coded channel to our tools. to see
The following are TI specific:
- Data protection
In SMP systems if the processor is switched a new master is generated (in some TI devices). So we protect the data with a mutex to guarantee a complete message is generated by the same master. OK, it's not our case.
- Meta Data
Our user mode HW libraries use meta data to transport data needed to process the HW profiling STM messages. Items like processor speed, sampling rate, processing options, ... (just a predefined byte buffer our tool-chain understands). The meta data is currently broadcasted on a dedicated channel (255), which conflicts with your hard-coded channel for logging printk output. So we will need to resolve hard-coded conflicts. Easy to fix, we can reserve dedicated channels at driver init and change our hard-coded affectation.
We need the driver to support registration and transport of the meta data on demand from the library (when the HW master is disabled, in case the collection buffer is small and circular).
I am thinking an IOCTL could be used to register meta data and then the data simply broadcast on a STM channel (will need to figure out which one) when the HW master is enabled and disabled. We don't face the same problematic
Meta data transmission is problematic for circular buffers (like ETB's) thus the reason for also sending meta data when a hw master is disabled. SW masters are not typically disabled, and our HW does not provide a transmission byte count (remember there are HW messages also being generated in the TI case). So there is no way from a driver we can tell when the recoding buffer will wrap even if the user told us the buffer size. I am thinking the best solution would be to force the user to gracefully disable the channel to get any sw channel meta data provided by the driver. I think so, if I have well understood
TI supports three cases of data capture:
- DTC/Host collection (stop on buffer full)
- DTC/Host collection (circular buffer)
- ETB/on-chip collection (circular buffer)
We don't support ETB/on-chip collection, We use external probe or Lauterbach combiprobe for trace data collection (I think it's circular buffer)
Of if the user is at a point in their code where they know thery will stop recoding on the hOst or ETB, we provide an IOCTL that simply disables all channels.
In the ETB case we may want to simply disable any open STM channels when the user decides to stop recording as a fail safe mechanism.
Note: Periodic transmission of meta data into a small circular buffer will not work well. In cases where the data is sparse the buffer will simply be filled with meta data rather than useful data.
- OST Headers
Adding an OST header to each message is a requirement for compatibility with TI's toolchain. There are a couple of ways to approach:
Completely hidden from the user - The device specific code will know if the header is necessary. On a write, prior to the copy from user space, the device independent code would have to make a call to get a properly sized memory buffer from the device dependent code that would include the header.
User enabled - Provide an IOCTL that allows the user to put the driver in a tool-chain specific mode (like add OST headers). I agree, see my previous remark on OST headers
Regards, Doug Deao
I attached my last STM patches which introduce trace framework.
Best Regards, Philippe Langlais
From: Philippe Langlais [mailto:philippe.langlais@linaro.org] Sent: Wednesday, May 04, 2011 3:08 AM To: Deao, Douglas Cc: Linus Walleij Subject: Re: STM at UDS-Budapest
Hi Doug,
On STE ux500 platforms we have the same STM module (follow MIDP STP 1.0), I have already posted our current implementation to the LKML and Linaro ML, it's very similar to your proposal. I can't be present to the Linaro summit but Linus Walleij can replace me for this topic, he proposes to write a common trace framework to receive STM drivers. Attached all our current proposal and work around STM.
Regards Philippe Langlais ST-Ericsson On 3 May 2011 00:42, Deao, Douglas d-deao@ti.com wrote: I am hosting an introductory session on System Trace at the summit. TI's System Trace Module (STM) provides a common protocol for instrumentation messages across multiple cores and system level hardware profiling in complex SoCs. Attached is a whitepaper for background reading.
Looking forward to meeting you at the summit.
Regards, Doug Deao Texas Instruments
linaro-dev mailing list linaro-dev@lists.linaro.org http://lists.linaro.org/mailman/listinfo/linaro-dev
-- IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
Hi Doug,
Some comments below.
Best regards, Arvind
-----Original Message----- From: Deao, Douglas [mailto:d-deao@ti.com] Sent: Tuesday, June 21, 2011 5:44 AM To: Arvind Chauhan; Philippe Langlais; Linus Walleij; Lee Jones Cc: Michael Hope; linaro-dev@lists.linaro.org; pierre.peiffer@stericsson.com; loic.pallardy@stericsson.com Subject: RE: ST-E STM Driver Review
Arvind,
- It's a fundamental requirement that the driver supports redirected IO
(stdout) and write() calls like any standard character driver. This makes STM really easy to use for most user applications that simply want to use existing printf() or write() calls to export logging messages without doing a lot of integration work. Plus this is what most of our customers are expecting and already using.
We made debugfs nodes allow userspace applications to redirect their trace details to STM channels, plus they can be opened as normal file using file system calls for writing. To enable various use cases: 1) Apps are statically compiled, and cannot be re-compiled These type of apps must use io-redirection, as not much can be done here.
2) Apps are shared compiled and cannot be re-compiled In these type of apps, I suggest using LD_PRE_LOADED option with a shared library (overriding standard IO and file-system calls). These overridden calls use mmap version of channels to dump trace. Also these IO calls can be modeled to be un-buffered. More involved though.
3) Apps can be re-compiled and source code base is small I suggest creating a highly tuned tracing framework using mmap interface, to provide non-intrusive trace mechanism.
4) Apps can be re-compiled, but source code is huge. 'Suggest using a library (created with weak symbol) to override IO and file-system calls.
This can cover various use case scenarios - varying degree of intrusive-ness, but will require support libraries and manifests. 'Surely requires greater discussion.
- Device drivers should abstract the underlining hardware from user
space applications, thus the reason we would like to see file operations standardized. With that said we are also ok with exposing the STM transport (through mmap) to user space supporting vendor specific libraries. I think we cover all our individual requirements this way.
- Yes, we realize that STM channels need to be allocated dynamically
from the driver.
I have started writing a generic STM specification document. Before I get much farther I think it's important we agree on requirements and architecture. I am trying to accommodate TI, STE and ARM capabilities and preserve our existing host tool chain methods. Will try to send out what I have so far tomorrow, and then schedule a call for early next week.
'May be good to consider virtualization scenarios too.
Have you guys made any progress on releasing your current STM driver?
We are making progress on this front - I shall be able to provide an update early next week.
Regards, Doug
-----Original Message----- From: Arvind Chauhan [mailto:Arvind.Chauhan@arm.com] Sent: Monday, June 20, 2011 7:58 AM To: Philippe Langlais; Deao, Douglas; Linus Walleij; Lee Jones Cc: Michael Hope; linaro-dev@lists.linaro.org; pierre.peiffer@stericsson.com; loic.pallardy@stericsson.com Subject: RE: ST-E STM Driver Review
Hi Philippe,
Some comments on stm-trace.txt, for points 1, 2, 3.
Maybe I can schedule a conference call this week to go over some of the notes, let me know.
Best regards, Arvind
-----Original Message----- From: Philippe Langlais [mailto:philippe.langlais@linaro.org] Sent: Thursday, May 26, 2011 8:50 PM To: Deao, Douglas; Linus Walleij; Lee Jones Cc: Arvind Chauhan; Michael Hope; linaro-dev@lists.linaro.org; pierre.peiffer@stericsson.com; loic.pallardy@stericsson.com Subject: Re: ST-E STM Driver Review
Hi Doug,
I initiate the work to build a hardware trace framework in the kernel, I'm not started the study to have a common userspace API for STM, thanks to this email we can start such a work, but it may be long (next week I'm in vacation till June 7th). See my detailed response for all your interrogations and my thoughts about STE STM implementation below: On 25 May 2011 23:54, Deao, Douglas d-deao@ti.com wrote: Sorry it took a while to get back to you guys. I was visiting customers last week. Most of my comments are just highlighting the differences between TI's STM 1.0 driver and ST-E's STM 1.0 driver, but there are a few questions, observations and suggestions. At the end I included some discussion on TI's meta data and OST header requirements.
I have not had a chance to look at your actual implementation yet. Did you do anything to abstract the actual HW transport ports and control registers from the higher level driver functions? Yes, partially I think through IOCTLs & debugfs (see our stm.h
userspace
API)
I realize there is a lot here to work through so if you would rather schedule a conference call to talk through the differences I can do that. I would like to start work on a Linaro (Unified) STM Spec next week if I can get feedback from everybody over the next few days. I
will
be out of the office on 5/27 and 5/31. I hope this email is enough.
I am especially interested in details of what you guys have in mind for a "common trace framework to receive STM drivers". If by framework you mean well defined APIs that are implemented for specific devices, then
I
think we are in agreement. What Michael and I have talked about is a common STM user mode experience across all Linaro supported devices, making Linux user mode code 100% portable between our devices. For my point of view, the trace device framework must ease the integration of new hardware trace drivers in the kernel (not only STM MIPI) to present standard hooks in current trace infrastructure, but it can cover a common STM userspace API too.
ST-E STM Driver stm-trace.txt review:
- Software Overview
In your "Software Overview" it states:
"The end of data packet is marked by a time stamp on latest byte(s) only."
I assume that user messages can be made up of any number of bytes,
half-
words, words or longs (what ever is most efficient) and you simply terminate the last element of the message with a time-stamp - right? Yes, the message buffer can be mis aligned
So the question is that should the driver do packetizing or should this be left to the tracing framework on top of STM driver (time-stamping the last element of the message can be left to the intermediate logic). In my view, STM driver should be a low level entity and provide simpler interface in terms of raw writes of 32bits or multiples - this may allow tracing framework to directly dump information over STM instead of buffering in between, making it less intrusive.
In the TI STM implementation a message can be any number and
combination
of bytes, half-words, or word transfers terminated with a time-stamp on the last element. In addition to that we also add an OST header to a message. (See below for discussion on OST header). In our case the OST header is added by the external capture probe.
Same as above, I think OST should be sitting on top of STM driver instead of being part of it - Allowing non-OST implementations to coexist.
- Lossless/Lossy modes.
TI only supports lossless mode for sw generated messages and is
enforced
in our hw implementation. Lossy mode is reserved for true hw messages. For STE, hw messages are always lossy, but sw generated messages could be configured in lossless (default) or lossy mode.
In my view, STM driver should expose all functionalities of the underlying hardware - leaving guaranteed/invariant choice to the tracing framework on top.
I did not notice that you documented a way to modify this through the debugfs API or IOCTLS. I have 2 IOCTLs (STM_SET_MODE & STM_GET_MODE) and debugfs (masters_modes) interfaces for that.
We expose channels through debugfs interface on demand basis. Channel allocation creates 2 debugfs nodes on STM mount point, one used for dumping trace data and other for configuring modes like guaranteed/invariant/timestamp etc. for the channel.
I am kind of thinking that may be ok since this is really a hw configuration choice in your case, but in the TI case the user does not get to make that choice. OK
- Channel Assignment
TI makes the assignment with mknod using the minor number to assign a fixed channel. This allows the user mode application to overload the channel usage for categorizing data (not my idea). I think we see the error of our ways here and will be ok with a dynamic channel
allocation.
If we are agree with dynamic channel allocation then a common STM userspace API is possible I think and perhaps common STM driver.
mknod would allow only 256 channels. We use debugfs nodes to allocate any number of channels, which appears automatically in debugfs mount point. When programatically accessing STM i/f we allow apps to allocate channels in terms of blocks using mmap node. When an app calls mmap API, we attach a page fault handler to the mapped region, thus when for the first time app touches this area it generates a page fault. In the handler, we allocate a block if available or preempt it from some dormant process and attach to the faulting VMA.
I am thinking that for each unique pid a channel should be assigned
when
the device is opened. I would guess you are keeping a channel table around and write() just checks the table for a pid assignment (no time to look at your implementation yet), if none is found the first free channel is used. If you moved this function back to open then you could do the IOCTL STM_GET_CHANNEL_NO anytime, not just after the first
write.
The reason behind this behavior is for our current STM user lib which open "/dev/stm" and alloc/free channels with IOCTL and never use write operation (only mmap + direct write) and in this case we don't want to loose a channel. I think we can change this behavior. We can support multiple channels allocated for one Process to avoid contention in multi-threaded process.
In write how do you flag an error if you exhaust the number of
available
channels? I return -ENOMEM.
We don't return error, we preempt blocks from other process
- Kernel API
TI does not support a Kernel API (yet). I can see that the Alloc/Free and File IO type functions are useful and should be standard.
Not sure what you mean by "lockless" trace functions? Not protected against concurrent access, it's the responsibility to
the
caller.
It looks like your "low level atomic trace functions for 1, 2, 4 or 8 bytes" is similar to TI's binary library functions (not supported by
the
TI STM Driver). This is what we use the OST header for, allowing our tool chain to differentiate between different message formats, rather than just assuming the data is a simple stream of bytes. I'm not ready to add an OST header to each trace messages to particularly for kernel function traces (ftrace has short messages 8 bytes size with high throughput), I rather to dedicate a specific channel for this, OST header has to be optional.
- Debugfs APIs.
TI used a different approach. The tool-chain on the host provides all the transport setup through JTAG, so our driver does not support
setting
up the actual STM data export (number of pins and clock rate). In our case device transport parameters must match the host receiver's collection setup.
With your approach the user can change the clock rate and export pin width effectively at any time. Our tools actually go through a calibration process during initialization so any changes to the
device's
transport setup (clock rate, number of pins data exported on) would cause the TI tool chain a lot of grief.
There are some parameters we know we need to add (like master enables). This are currently also handled by the host tools. TI's STM module allows up to 4 SW masters to be enabled (with id masks that can be used to enable multiple masters from the same group) and 4 HW masters that can be enabled at the same time as the SW masters. If the user tries to enable more than the HW allows do you have a mechanism to flag an
error?
We can't face the same problem with STE STM, our hardware support 6 SW masters & 2 HW masters without restriction, all could be enabled at
same
time (managed by a bit field register)
I don't have a lot of experience with debugfs but I am assuming it's primarily used for allowing scripts to configure a driver (like in your example) or extract information. It's the current way to configure kernel tracing infrastructure (from console, through scripts or user apps), kernel users are familiar with this interface.
We may want to define a standard set of debugfs options whose implementation is vendor specific. But that raises some questions:
- How do we deal with options that don't make sense for a specific
vendor? Maybe just doing nothing is acceptable or do we want to provide a discovery mechanism? Do nothing may be sufficient, we can add STM IP specific string descriptor too, something like "vendor id+STM IP id+STP version"
- Would user scripts then also be vendor specific?
We should probably make an effort to avoid this. A discovery mechanism may allow user mode scripts to be generic. Yes, but STM configuration scripts are device specific, to see.
- Mapped Channels
I believe the TI hw transport channel mapping is compatible. In the TI case a channel is mapped into two spaces, the first half is for non- timestamp transfers and the second half is for time-stamped transfers. When we write a message (from a user mode write call for example) we simply write all the data except the last element through the non- timestamp port, and then the last element is written to the time-
stamped
port. So I think we could be compatible here. I think so, we have similar channel memory mapping but, one STM channel is accessed through this C union of 2 consecutive 64bits word (first
for
non-timestamp, 2nd with timestamp) /* One single channel mapping */ struct stm_channel { union { __u8 no_stamp8; __u16 no_stamp16; __u32 no_stamp32; __u64 no_stamp64; }; union { __u8 stamp8; __u16 stamp16; __u32 stamp32; __u64 stamp64; }; };
With that said I am not sure about exposing all channels to a user mode library. You are relying on the library to use the convention of
getting
a free channel from the driver to make sure there are no conflicts. If the channel assignment is made when you open the device, you could conceivably map just the address space needed for the single channel, thus eliminating the need to get a free channel from the driver. In the TI case a single channel's transport mapping is 4K bytes, which matches the typical PAGE_SIZE. I realize not all hw implementations will match up with the PAGE_SIZE, which may be why you simply map all the channels back to user space. It's our case, the 256 channels is mapped in the same 4K memory page.
Since free channels can become busy rapidly, maybe a better convention would be to simply use another device node if the user wants the
library
STM data to be transmitted on a different STM channel than the current process. This may be a case where providing a mechanism (see meta data discussion below) to allow channels to be named for the toolchain may
be
a good idea (provide task name and process id). In our current implementation, a new open on the same "/dev/stm" device allocate a new channel at first write.
- 8-byte Writes
TI does not support 64-bit writes with our STM 1.0 module. We may need an IOCTL to get the largest transfer supported for the mmap case. For all other cases this should just be hidden in the device dependent
code.
I think so
- Kernel Internal Usage
I like the idea of having dedicated support in the driver for common kernel logging. Any ideas on how you would support kernel STM channel assignments without hard-coding? But configurable through kernel configuration, the other way is to dedicate a specific channel for this signalization like your channel 255, not so easy to define.
We may need a mechanism to communicate the definition of each hard-
coded
channel to our tools. to see
The following are TI specific:
- Data protection
In SMP systems if the processor is switched a new master is generated (in some TI devices). So we protect the data with a mutex to guarantee
a
complete message is generated by the same master. OK, it's not our case.
- Meta Data
Our user mode HW libraries use meta data to transport data needed to process the HW profiling STM messages. Items like processor speed, sampling rate, processing options, ... (just a predefined byte buffer our tool-chain understands). The meta data is currently broadcasted on
a
dedicated channel (255), which conflicts with your hard-coded channel for logging printk output. So we will need to resolve hard-coded conflicts. Easy to fix, we can reserve dedicated channels at driver init and
change
our hard-coded affectation.
We need the driver to support registration and transport of the meta data on demand from the library (when the HW master is disabled, in
case
the collection buffer is small and circular).
I am thinking an IOCTL could be used to register meta data and then the data simply broadcast on a STM channel (will need to figure out which one) when the HW master is enabled and disabled. We don't face the same problematic
Meta data transmission is problematic for circular buffers (like ETB's) thus the reason for also sending meta data when a hw master is
disabled.
SW masters are not typically disabled, and our HW does not provide a transmission byte count (remember there are HW messages also being generated in the TI case). So there is no way from a driver we can tell when the recoding buffer will wrap even if the user told us the buffer size. I am thinking the best solution would be to force the user to gracefully disable the channel to get any sw channel meta data provided by the driver. I think so, if I have well understood
TI supports three cases of data capture:
- DTC/Host collection (stop on buffer full)
- DTC/Host collection (circular buffer)
- ETB/on-chip collection (circular buffer)
We don't support ETB/on-chip collection, We use external probe or Lauterbach combiprobe for trace data collection (I think it's circular buffer)
Of if the user is at a point in their code where they know thery will stop recoding on the hOst or ETB, we provide an IOCTL that simply disables all channels.
In the ETB case we may want to simply disable any open STM channels
when
the user decides to stop recording as a fail safe mechanism.
Note: Periodic transmission of meta data into a small circular buffer will not work well. In cases where the data is sparse the buffer will simply be filled with meta data rather than useful data.
- OST Headers
Adding an OST header to each message is a requirement for compatibility with TI's toolchain. There are a couple of ways to approach:
Completely hidden from the user - The device specific code will know if the header is necessary. On a write, prior to the copy from user space, the device independent code would have to make a call to get a properly sized memory buffer from the device dependent code that would include the header.
User enabled - Provide an IOCTL that allows the user to put the driver in a tool-chain specific mode (like add OST headers). I agree, see my previous remark on OST headers
Regards, Doug Deao
I attached my last STM patches which introduce trace framework.
Best Regards, Philippe Langlais
From: Philippe Langlais [mailto:philippe.langlais@linaro.org] Sent: Wednesday, May 04, 2011 3:08 AM To: Deao, Douglas Cc: Linus Walleij Subject: Re: STM at UDS-Budapest
Hi Doug,
On STE ux500 platforms we have the same STM module (follow MIDP STP 1.0), I have already posted our current implementation to the LKML and Linaro ML, it's very similar to your proposal. I can't be present to the Linaro summit but Linus Walleij can replace
me
for this topic, he proposes to write a common trace framework to receive STM drivers. Attached all our current proposal and work around STM.
Regards Philippe Langlais ST-Ericsson On 3 May 2011 00:42, Deao, Douglas d-deao@ti.com wrote: I am hosting an introductory session on System Trace at the summit.
TI's
System Trace Module (STM) provides a common protocol for
instrumentation
messages across multiple cores and system level hardware profiling in complex SoCs. Attached is a whitepaper for background reading.
Looking forward to meeting you at the summit.
Regards, Doug Deao Texas Instruments
linaro-dev mailing list linaro-dev@lists.linaro.org http://lists.linaro.org/mailman/listinfo/linaro-dev
-- IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
-- IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
Please find attached the first few sections. I have not spent any time on formatting, but this should be enough to get us talking about requirements.
Let me know what you think.
Thanks, Doug
________________________________ From: Philippe Langlais [mailto:philippe.langlais@linaro.org] Sent: Tuesday, June 21, 2011 3:06 AM To: Arvind Chauhan Cc: Deao, Douglas; Linus Walleij; Lee Jones; Michael Hope; linaro-dev@lists.linaro.org; pierre.peiffer@stericsson.com; loic.pallardy@stericsson.com Subject: Re: ST-E STM Driver Review
Hi,
As stated by Douglas, a character driver is more natural for a printf like usage, therefore we need a mean to do packetizing (with optional OST header: can be supported by generic functions in kernel hw tracing framework). My current implementation use only one device node (/dev/stm) for all STM channels (no minor<256 restriction even for STM v2.0), each "open+write" on this device node allocate dynamically a new channel and IOCTLs are available to retrieve channel number. This driver covers low level interface too (direct write with no overhead from 1 to 8 bytes with mmap + IOCTLs to allocate/free/configure channels) for a usage with an optimized user space library.
The hardware tracing framework, I started to develop, only covers trace device register/unregister and offers a set of common hooks to the standard kernel tracing infrastructure, all things traced into trace ring buffer (ftrace) could be redirected in real time towards dedicated STM channels, it's only a draft, we can add OST header management here and others features.
I attach my current user+kernel STM interface. Now, I'm waiting of Doug's generic STM specification document for review and for pursue STM driver development for future upstreaming.
Best regards, Philippe
On 20 June 2011 14:57, Arvind Chauhan <Arvind.Chauhan@arm.commailto:Arvind.Chauhan@arm.com> wrote: Hi Philippe,
Some comments on stm-trace.txt, for points 1, 2, 3.
Maybe I can schedule a conference call this week to go over some of the notes, let me know.
Best regards, Arvind
-----Original Message----- From: Philippe Langlais [mailto:philippe.langlais@linaro.orgmailto:philippe.langlais@linaro.org] Sent: Thursday, May 26, 2011 8:50 PM To: Deao, Douglas; Linus Walleij; Lee Jones Cc: Arvind Chauhan; Michael Hope; linaro-dev@lists.linaro.orgmailto:linaro-dev@lists.linaro.org; pierre.peiffer@stericsson.commailto:pierre.peiffer@stericsson.com; loic.pallardy@stericsson.commailto:loic.pallardy@stericsson.com Subject: Re: ST-E STM Driver Review
Hi Doug,
I initiate the work to build a hardware trace framework in the kernel, I'm not started the study to have a common userspace API for STM, thanks to this email we can start such a work, but it may be long (next week I'm in vacation till June 7th). See my detailed response for all your interrogations and my thoughts about STE STM implementation below: On 25 May 2011 23:54, Deao, Douglas <d-deao@ti.commailto:d-deao@ti.com> wrote: Sorry it took a while to get back to you guys. I was visiting customers last week. Most of my comments are just highlighting the differences between TI's STM 1.0 driver and ST-E's STM 1.0 driver, but there are a few questions, observations and suggestions. At the end I included some discussion on TI's meta data and OST header requirements.
I have not had a chance to look at your actual implementation yet. Did you do anything to abstract the actual HW transport ports and control registers from the higher level driver functions? Yes, partially I think through IOCTLs & debugfs (see our stm.h userspace API)
I realize there is a lot here to work through so if you would rather schedule a conference call to talk through the differences I can do that. I would like to start work on a Linaro (Unified) STM Spec next week if I can get feedback from everybody over the next few days. I will be out of the office on 5/27 and 5/31. I hope this email is enough.
I am especially interested in details of what you guys have in mind for a "common trace framework to receive STM drivers". If by framework you mean well defined APIs that are implemented for specific devices, then I think we are in agreement. What Michael and I have talked about is a common STM user mode experience across all Linaro supported devices, making Linux user mode code 100% portable between our devices. For my point of view, the trace device framework must ease the integration of new hardware trace drivers in the kernel (not only STM MIPI) to present standard hooks in current trace infrastructure, but it can cover a common STM userspace API too.
ST-E STM Driver stm-trace.txt review:
- Software Overview
In your "Software Overview" it states:
"The end of data packet is marked by a time stamp on latest byte(s) only."
I assume that user messages can be made up of any number of bytes, half- words, words or longs (what ever is most efficient) and you simply terminate the last element of the message with a time-stamp - right? Yes, the message buffer can be mis aligned
So the question is that should the driver do packetizing or should this be left to the tracing framework on top of STM driver (time-stamping the last element of the message can be left to the intermediate logic). In my view, STM driver should be a low level entity and provide simpler interface in terms of raw writes of 32bits or multiples - this may allow tracing framework to directly dump information over STM instead of buffering in between, making it less intrusive.
In the TI STM implementation a message can be any number and combination of bytes, half-words, or word transfers terminated with a time-stamp on the last element. In addition to that we also add an OST header to a message. (See below for discussion on OST header). In our case the OST header is added by the external capture probe.
Same as above, I think OST should be sitting on top of STM driver instead of being part of it - Allowing non-OST implementations to coexist.
- Lossless/Lossy modes.
TI only supports lossless mode for sw generated messages and is enforced in our hw implementation. Lossy mode is reserved for true hw messages. For STE, hw messages are always lossy, but sw generated messages could be configured in lossless (default) or lossy mode.
In my view, STM driver should expose all functionalities of the underlying hardware - leaving guaranteed/invariant choice to the tracing framework on top.
I did not notice that you documented a way to modify this through the debugfs API or IOCTLS. I have 2 IOCTLs (STM_SET_MODE & STM_GET_MODE) and debugfs (masters_modes) interfaces for that.
We expose channels through debugfs interface on demand basis. Channel allocation creates 2 debugfs nodes on STM mount point, one used for dumping trace data and other for configuring modes like guaranteed/invariant/timestamp etc. for the channel.
I am kind of thinking that may be ok since this is really a hw configuration choice in your case, but in the TI case the user does not get to make that choice. OK
- Channel Assignment
TI makes the assignment with mknod using the minor number to assign a fixed channel. This allows the user mode application to overload the channel usage for categorizing data (not my idea). I think we see the error of our ways here and will be ok with a dynamic channel allocation. If we are agree with dynamic channel allocation then a common STM userspace API is possible I think and perhaps common STM driver.
mknod would allow only 256 channels. We use debugfs nodes to allocate any number of channels, which appears automatically in debugfs mount point. When programatically accessing STM i/f we allow apps to allocate channels in terms of blocks using mmap node. When an app calls mmap API, we attach a page fault handler to the mapped region, thus when for the first time app touches this area it generates a page fault. In the handler, we allocate a block if available or preempt it from some dormant process and attach to the faulting VMA.
I am thinking that for each unique pid a channel should be assigned when the device is opened. I would guess you are keeping a channel table around and write() just checks the table for a pid assignment (no time to look at your implementation yet), if none is found the first free channel is used. If you moved this function back to open then you could do the IOCTL STM_GET_CHANNEL_NO anytime, not just after the first write. The reason behind this behavior is for our current STM user lib which open "/dev/stm" and alloc/free channels with IOCTL and never use write operation (only mmap + direct write) and in this case we don't want to loose a channel. I think we can change this behavior. We can support multiple channels allocated for one Process to avoid contention in multi-threaded process.
In write how do you flag an error if you exhaust the number of available channels? I return -ENOMEM.
We don't return error, we preempt blocks from other process
- Kernel API
TI does not support a Kernel API (yet). I can see that the Alloc/Free and File IO type functions are useful and should be standard.
Not sure what you mean by "lockless" trace functions? Not protected against concurrent access, it's the responsibility to the caller.
It looks like your "low level atomic trace functions for 1, 2, 4 or 8 bytes" is similar to TI's binary library functions (not supported by the TI STM Driver). This is what we use the OST header for, allowing our tool chain to differentiate between different message formats, rather than just assuming the data is a simple stream of bytes. I'm not ready to add an OST header to each trace messages to particularly for kernel function traces (ftrace has short messages 8 bytes size with high throughput), I rather to dedicate a specific channel for this, OST header has to be optional.
- Debugfs APIs.
TI used a different approach. The tool-chain on the host provides all the transport setup through JTAG, so our driver does not support setting up the actual STM data export (number of pins and clock rate). In our case device transport parameters must match the host receiver's collection setup.
With your approach the user can change the clock rate and export pin width effectively at any time. Our tools actually go through a calibration process during initialization so any changes to the device's transport setup (clock rate, number of pins data exported on) would cause the TI tool chain a lot of grief.
There are some parameters we know we need to add (like master enables). This are currently also handled by the host tools. TI's STM module allows up to 4 SW masters to be enabled (with id masks that can be used to enable multiple masters from the same group) and 4 HW masters that can be enabled at the same time as the SW masters. If the user tries to enable more than the HW allows do you have a mechanism to flag an error? We can't face the same problem with STE STM, our hardware support 6 SW masters & 2 HW masters without restriction, all could be enabled at same time (managed by a bit field register)
I don't have a lot of experience with debugfs but I am assuming it's primarily used for allowing scripts to configure a driver (like in your example) or extract information. It's the current way to configure kernel tracing infrastructure (from console, through scripts or user apps), kernel users are familiar with this interface.
We may want to define a standard set of debugfs options whose implementation is vendor specific. But that raises some questions:
- How do we deal with options that don't make sense for a specific
vendor? Maybe just doing nothing is acceptable or do we want to provide a discovery mechanism? Do nothing may be sufficient, we can add STM IP specific string descriptor too, something like "vendor id+STM IP id+STP version"
- Would user scripts then also be vendor specific?
We should probably make an effort to avoid this. A discovery mechanism may allow user mode scripts to be generic. Yes, but STM configuration scripts are device specific, to see.
- Mapped Channels
I believe the TI hw transport channel mapping is compatible. In the TI case a channel is mapped into two spaces, the first half is for non- timestamp transfers and the second half is for time-stamped transfers. When we write a message (from a user mode write call for example) we simply write all the data except the last element through the non- timestamp port, and then the last element is written to the time-stamped port. So I think we could be compatible here. I think so, we have similar channel memory mapping but, one STM channel is accessed through this C union of 2 consecutive 64bits word (first for non-timestamp, 2nd with timestamp) /* One single channel mapping */ struct stm_channel { union { __u8 no_stamp8; __u16 no_stamp16; __u32 no_stamp32; __u64 no_stamp64; }; union { __u8 stamp8; __u16 stamp16; __u32 stamp32; __u64 stamp64; }; };
With that said I am not sure about exposing all channels to a user mode library. You are relying on the library to use the convention of getting a free channel from the driver to make sure there are no conflicts. If the channel assignment is made when you open the device, you could conceivably map just the address space needed for the single channel, thus eliminating the need to get a free channel from the driver. In the TI case a single channel's transport mapping is 4K bytes, which matches the typical PAGE_SIZE. I realize not all hw implementations will match up with the PAGE_SIZE, which may be why you simply map all the channels back to user space. It's our case, the 256 channels is mapped in the same 4K memory page.
Since free channels can become busy rapidly, maybe a better convention would be to simply use another device node if the user wants the library STM data to be transmitted on a different STM channel than the current process. This may be a case where providing a mechanism (see meta data discussion below) to allow channels to be named for the toolchain may be a good idea (provide task name and process id). In our current implementation, a new open on the same "/dev/stm" device allocate a new channel at first write.
- 8-byte Writes
TI does not support 64-bit writes with our STM 1.0 module. We may need an IOCTL to get the largest transfer supported for the mmap case. For all other cases this should just be hidden in the device dependent code. I think so
- Kernel Internal Usage
I like the idea of having dedicated support in the driver for common kernel logging. Any ideas on how you would support kernel STM channel assignments without hard-coding? But configurable through kernel configuration, the other way is to dedicate a specific channel for this signalization like your channel 255, not so easy to define.
We may need a mechanism to communicate the definition of each hard-coded channel to our tools. to see
The following are TI specific:
- Data protection
In SMP systems if the processor is switched a new master is generated (in some TI devices). So we protect the data with a mutex to guarantee a complete message is generated by the same master. OK, it's not our case.
- Meta Data
Our user mode HW libraries use meta data to transport data needed to process the HW profiling STM messages. Items like processor speed, sampling rate, processing options, ... (just a predefined byte buffer our tool-chain understands). The meta data is currently broadcasted on a dedicated channel (255), which conflicts with your hard-coded channel for logging printk output. So we will need to resolve hard-coded conflicts. Easy to fix, we can reserve dedicated channels at driver init and change our hard-coded affectation.
We need the driver to support registration and transport of the meta data on demand from the library (when the HW master is disabled, in case the collection buffer is small and circular).
I am thinking an IOCTL could be used to register meta data and then the data simply broadcast on a STM channel (will need to figure out which one) when the HW master is enabled and disabled. We don't face the same problematic
Meta data transmission is problematic for circular buffers (like ETB's) thus the reason for also sending meta data when a hw master is disabled. SW masters are not typically disabled, and our HW does not provide a transmission byte count (remember there are HW messages also being generated in the TI case). So there is no way from a driver we can tell when the recoding buffer will wrap even if the user told us the buffer size. I am thinking the best solution would be to force the user to gracefully disable the channel to get any sw channel meta data provided by the driver. I think so, if I have well understood
TI supports three cases of data capture:
- DTC/Host collection (stop on buffer full)
- DTC/Host collection (circular buffer)
- ETB/on-chip collection (circular buffer)
We don't support ETB/on-chip collection, We use external probe or Lauterbach combiprobe for trace data collection (I think it's circular buffer)
Of if the user is at a point in their code where they know thery will stop recoding on the hOst or ETB, we provide an IOCTL that simply disables all channels.
In the ETB case we may want to simply disable any open STM channels when the user decides to stop recording as a fail safe mechanism.
Note: Periodic transmission of meta data into a small circular buffer will not work well. In cases where the data is sparse the buffer will simply be filled with meta data rather than useful data.
- OST Headers
Adding an OST header to each message is a requirement for compatibility with TI's toolchain. There are a couple of ways to approach:
Completely hidden from the user - The device specific code will know if the header is necessary. On a write, prior to the copy from user space, the device independent code would have to make a call to get a properly sized memory buffer from the device dependent code that would include the header.
User enabled - Provide an IOCTL that allows the user to put the driver in a tool-chain specific mode (like add OST headers). I agree, see my previous remark on OST headers
Regards, Doug Deao
I attached my last STM patches which introduce trace framework.
Best Regards, Philippe Langlais
From: Philippe Langlais [mailto:philippe.langlais@linaro.orgmailto:philippe.langlais@linaro.org] Sent: Wednesday, May 04, 2011 3:08 AM To: Deao, Douglas Cc: Linus Walleij Subject: Re: STM at UDS-Budapest
Hi Doug,
On STE ux500 platforms we have the same STM module (follow MIDP STP 1.0), I have already posted our current implementation to the LKML and Linaro ML, it's very similar to your proposal. I can't be present to the Linaro summit but Linus Walleij can replace me for this topic, he proposes to write a common trace framework to receive STM drivers. Attached all our current proposal and work around STM.
Regards Philippe Langlais ST-Ericsson On 3 May 2011 00:42, Deao, Douglas <d-deao@ti.commailto:d-deao@ti.com> wrote: I am hosting an introductory session on System Trace at the summit. TI's System Trace Module (STM) provides a common protocol for instrumentation messages across multiple cores and system level hardware profiling in complex SoCs. Attached is a whitepaper for background reading.
Looking forward to meeting you at the summit.
Regards, Doug Deao Texas Instruments
linaro-dev mailing list linaro-dev@lists.linaro.orgmailto:linaro-dev@lists.linaro.org http://lists.linaro.org/mailman/listinfo/linaro-dev
-- IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
Hi Doug,
I can set up a voice bridge to get discussion rolling - let me know if following schedule suits all ('have tried to keep various TZ in mind)
24/June/2011, 15:30 GMT
Best regards, Arvind
From: Deao, Douglas [mailto:d-deao@ti.com] Sent: Wednesday, June 22, 2011 4:58 AM To: Philippe Langlais; Arvind Chauhan Cc: Linus Walleij; Lee Jones; Michael Hope; linaro-dev@lists.linaro.org; pierre.peiffer@stericsson.com; loic.pallardy@stericsson.com Subject: RE: ST-E STM Driver Review
Please find attached the first few sections. I have not spent any time on formatting, but this should be enough to get us talking about requirements.
Let me know what you think.
Thanks, Doug
________________________________ From: Philippe Langlais [mailto:philippe.langlais@linaro.org] Sent: Tuesday, June 21, 2011 3:06 AM To: Arvind Chauhan Cc: Deao, Douglas; Linus Walleij; Lee Jones; Michael Hope; linaro-dev@lists.linaro.org; pierre.peiffer@stericsson.com; loic.pallardy@stericsson.com Subject: Re: ST-E STM Driver Review
Hi,
As stated by Douglas, a character driver is more natural for a printf like usage, therefore we need a mean to do packetizing (with optional OST header: can be supported by generic functions in kernel hw tracing framework). My current implementation use only one device node (/dev/stm) for all STM channels (no minor<256 restriction even for STM v2.0), each "open+write" on this device node allocate dynamically a new channel and IOCTLs are available to retrieve channel number. This driver covers low level interface too (direct write with no overhead from 1 to 8 bytes with mmap + IOCTLs to allocate/free/configure channels) for a usage with an optimized user space library.
The hardware tracing framework, I started to develop, only covers trace device register/unregister and offers a set of common hooks to the standard kernel tracing infrastructure, all things traced into trace ring buffer (ftrace) could be redirected in real time towards dedicated STM channels, it's only a draft, we can add OST header management here and others features.
I attach my current user+kernel STM interface. Now, I'm waiting of Doug's generic STM specification document for review and for pursue STM driver development for future upstreaming.
Best regards, Philippe On 20 June 2011 14:57, Arvind Chauhan <Arvind.Chauhan@arm.commailto:Arvind.Chauhan@arm.com> wrote: Hi Philippe,
Some comments on stm-trace.txt, for points 1, 2, 3.
Maybe I can schedule a conference call this week to go over some of the notes, let me know.
Best regards, Arvind
-----Original Message----- From: Philippe Langlais [mailto:philippe.langlais@linaro.orgmailto:philippe.langlais@linaro.org] Sent: Thursday, May 26, 2011 8:50 PM To: Deao, Douglas; Linus Walleij; Lee Jones Cc: Arvind Chauhan; Michael Hope; linaro-dev@lists.linaro.orgmailto:linaro-dev@lists.linaro.org; pierre.peiffer@stericsson.commailto:pierre.peiffer@stericsson.com; loic.pallardy@stericsson.commailto:loic.pallardy@stericsson.com Subject: Re: ST-E STM Driver Review
Hi Doug,
I initiate the work to build a hardware trace framework in the kernel, I'm not started the study to have a common userspace API for STM, thanks to this email we can start such a work, but it may be long (next week I'm in vacation till June 7th). See my detailed response for all your interrogations and my thoughts about STE STM implementation below: On 25 May 2011 23:54, Deao, Douglas <d-deao@ti.commailto:d-deao@ti.com> wrote: Sorry it took a while to get back to you guys. I was visiting customers last week. Most of my comments are just highlighting the differences between TI's STM 1.0 driver and ST-E's STM 1.0 driver, but there are a few questions, observations and suggestions. At the end I included some discussion on TI's meta data and OST header requirements.
I have not had a chance to look at your actual implementation yet. Did you do anything to abstract the actual HW transport ports and control registers from the higher level driver functions? Yes, partially I think through IOCTLs & debugfs (see our stm.h userspace API)
I realize there is a lot here to work through so if you would rather schedule a conference call to talk through the differences I can do that. I would like to start work on a Linaro (Unified) STM Spec next week if I can get feedback from everybody over the next few days. I will be out of the office on 5/27 and 5/31. I hope this email is enough.
I am especially interested in details of what you guys have in mind for a "common trace framework to receive STM drivers". If by framework you mean well defined APIs that are implemented for specific devices, then I think we are in agreement. What Michael and I have talked about is a common STM user mode experience across all Linaro supported devices, making Linux user mode code 100% portable between our devices. For my point of view, the trace device framework must ease the integration of new hardware trace drivers in the kernel (not only STM MIPI) to present standard hooks in current trace infrastructure, but it can cover a common STM userspace API too.
ST-E STM Driver stm-trace.txt review:
- Software Overview
In your "Software Overview" it states:
"The end of data packet is marked by a time stamp on latest byte(s) only."
I assume that user messages can be made up of any number of bytes, half- words, words or longs (what ever is most efficient) and you simply terminate the last element of the message with a time-stamp - right? Yes, the message buffer can be mis aligned
So the question is that should the driver do packetizing or should this be left to the tracing framework on top of STM driver (time-stamping the last element of the message can be left to the intermediate logic). In my view, STM driver should be a low level entity and provide simpler interface in terms of raw writes of 32bits or multiples - this may allow tracing framework to directly dump information over STM instead of buffering in between, making it less intrusive.
In the TI STM implementation a message can be any number and combination of bytes, half-words, or word transfers terminated with a time-stamp on the last element. In addition to that we also add an OST header to a message. (See below for discussion on OST header). In our case the OST header is added by the external capture probe.
Same as above, I think OST should be sitting on top of STM driver instead of being part of it - Allowing non-OST implementations to coexist.
- Lossless/Lossy modes.
TI only supports lossless mode for sw generated messages and is enforced in our hw implementation. Lossy mode is reserved for true hw messages. For STE, hw messages are always lossy, but sw generated messages could be configured in lossless (default) or lossy mode.
In my view, STM driver should expose all functionalities of the underlying hardware - leaving guaranteed/invariant choice to the tracing framework on top.
I did not notice that you documented a way to modify this through the debugfs API or IOCTLS. I have 2 IOCTLs (STM_SET_MODE & STM_GET_MODE) and debugfs (masters_modes) interfaces for that.
We expose channels through debugfs interface on demand basis. Channel allocation creates 2 debugfs nodes on STM mount point, one used for dumping trace data and other for configuring modes like guaranteed/invariant/timestamp etc. for the channel.
I am kind of thinking that may be ok since this is really a hw configuration choice in your case, but in the TI case the user does not get to make that choice. OK
- Channel Assignment
TI makes the assignment with mknod using the minor number to assign a fixed channel. This allows the user mode application to overload the channel usage for categorizing data (not my idea). I think we see the error of our ways here and will be ok with a dynamic channel allocation. If we are agree with dynamic channel allocation then a common STM userspace API is possible I think and perhaps common STM driver.
mknod would allow only 256 channels. We use debugfs nodes to allocate any number of channels, which appears automatically in debugfs mount point. When programatically accessing STM i/f we allow apps to allocate channels in terms of blocks using mmap node. When an app calls mmap API, we attach a page fault handler to the mapped region, thus when for the first time app touches this area it generates a page fault. In the handler, we allocate a block if available or preempt it from some dormant process and attach to the faulting VMA.
I am thinking that for each unique pid a channel should be assigned when the device is opened. I would guess you are keeping a channel table around and write() just checks the table for a pid assignment (no time to look at your implementation yet), if none is found the first free channel is used. If you moved this function back to open then you could do the IOCTL STM_GET_CHANNEL_NO anytime, not just after the first write. The reason behind this behavior is for our current STM user lib which open "/dev/stm" and alloc/free channels with IOCTL and never use write operation (only mmap + direct write) and in this case we don't want to loose a channel. I think we can change this behavior. We can support multiple channels allocated for one Process to avoid contention in multi-threaded process.
In write how do you flag an error if you exhaust the number of available channels? I return -ENOMEM.
We don't return error, we preempt blocks from other process
- Kernel API
TI does not support a Kernel API (yet). I can see that the Alloc/Free and File IO type functions are useful and should be standard.
Not sure what you mean by "lockless" trace functions? Not protected against concurrent access, it's the responsibility to the caller.
It looks like your "low level atomic trace functions for 1, 2, 4 or 8 bytes" is similar to TI's binary library functions (not supported by the TI STM Driver). This is what we use the OST header for, allowing our tool chain to differentiate between different message formats, rather than just assuming the data is a simple stream of bytes. I'm not ready to add an OST header to each trace messages to particularly for kernel function traces (ftrace has short messages 8 bytes size with high throughput), I rather to dedicate a specific channel for this, OST header has to be optional.
- Debugfs APIs.
TI used a different approach. The tool-chain on the host provides all the transport setup through JTAG, so our driver does not support setting up the actual STM data export (number of pins and clock rate). In our case device transport parameters must match the host receiver's collection setup.
With your approach the user can change the clock rate and export pin width effectively at any time. Our tools actually go through a calibration process during initialization so any changes to the device's transport setup (clock rate, number of pins data exported on) would cause the TI tool chain a lot of grief.
There are some parameters we know we need to add (like master enables). This are currently also handled by the host tools. TI's STM module allows up to 4 SW masters to be enabled (with id masks that can be used to enable multiple masters from the same group) and 4 HW masters that can be enabled at the same time as the SW masters. If the user tries to enable more than the HW allows do you have a mechanism to flag an error? We can't face the same problem with STE STM, our hardware support 6 SW masters & 2 HW masters without restriction, all could be enabled at same time (managed by a bit field register)
I don't have a lot of experience with debugfs but I am assuming it's primarily used for allowing scripts to configure a driver (like in your example) or extract information. It's the current way to configure kernel tracing infrastructure (from console, through scripts or user apps), kernel users are familiar with this interface.
We may want to define a standard set of debugfs options whose implementation is vendor specific. But that raises some questions:
- How do we deal with options that don't make sense for a specific
vendor? Maybe just doing nothing is acceptable or do we want to provide a discovery mechanism? Do nothing may be sufficient, we can add STM IP specific string descriptor too, something like "vendor id+STM IP id+STP version"
- Would user scripts then also be vendor specific?
We should probably make an effort to avoid this. A discovery mechanism may allow user mode scripts to be generic. Yes, but STM configuration scripts are device specific, to see.
- Mapped Channels
I believe the TI hw transport channel mapping is compatible. In the TI case a channel is mapped into two spaces, the first half is for non- timestamp transfers and the second half is for time-stamped transfers. When we write a message (from a user mode write call for example) we simply write all the data except the last element through the non- timestamp port, and then the last element is written to the time-stamped port. So I think we could be compatible here. I think so, we have similar channel memory mapping but, one STM channel is accessed through this C union of 2 consecutive 64bits word (first for non-timestamp, 2nd with timestamp) /* One single channel mapping */ struct stm_channel { union { __u8 no_stamp8; __u16 no_stamp16; __u32 no_stamp32; __u64 no_stamp64; }; union { __u8 stamp8; __u16 stamp16; __u32 stamp32; __u64 stamp64; }; };
With that said I am not sure about exposing all channels to a user mode library. You are relying on the library to use the convention of getting a free channel from the driver to make sure there are no conflicts. If the channel assignment is made when you open the device, you could conceivably map just the address space needed for the single channel, thus eliminating the need to get a free channel from the driver. In the TI case a single channel's transport mapping is 4K bytes, which matches the typical PAGE_SIZE. I realize not all hw implementations will match up with the PAGE_SIZE, which may be why you simply map all the channels back to user space. It's our case, the 256 channels is mapped in the same 4K memory page.
Since free channels can become busy rapidly, maybe a better convention would be to simply use another device node if the user wants the library STM data to be transmitted on a different STM channel than the current process. This may be a case where providing a mechanism (see meta data discussion below) to allow channels to be named for the toolchain may be a good idea (provide task name and process id). In our current implementation, a new open on the same "/dev/stm" device allocate a new channel at first write.
- 8-byte Writes
TI does not support 64-bit writes with our STM 1.0 module. We may need an IOCTL to get the largest transfer supported for the mmap case. For all other cases this should just be hidden in the device dependent code. I think so
- Kernel Internal Usage
I like the idea of having dedicated support in the driver for common kernel logging. Any ideas on how you would support kernel STM channel assignments without hard-coding? But configurable through kernel configuration, the other way is to dedicate a specific channel for this signalization like your channel 255, not so easy to define.
We may need a mechanism to communicate the definition of each hard-coded channel to our tools. to see
The following are TI specific:
- Data protection
In SMP systems if the processor is switched a new master is generated (in some TI devices). So we protect the data with a mutex to guarantee a complete message is generated by the same master. OK, it's not our case.
- Meta Data
Our user mode HW libraries use meta data to transport data needed to process the HW profiling STM messages. Items like processor speed, sampling rate, processing options, ... (just a predefined byte buffer our tool-chain understands). The meta data is currently broadcasted on a dedicated channel (255), which conflicts with your hard-coded channel for logging printk output. So we will need to resolve hard-coded conflicts. Easy to fix, we can reserve dedicated channels at driver init and change our hard-coded affectation.
We need the driver to support registration and transport of the meta data on demand from the library (when the HW master is disabled, in case the collection buffer is small and circular).
I am thinking an IOCTL could be used to register meta data and then the data simply broadcast on a STM channel (will need to figure out which one) when the HW master is enabled and disabled. We don't face the same problematic
Meta data transmission is problematic for circular buffers (like ETB's) thus the reason for also sending meta data when a hw master is disabled. SW masters are not typically disabled, and our HW does not provide a transmission byte count (remember there are HW messages also being generated in the TI case). So there is no way from a driver we can tell when the recoding buffer will wrap even if the user told us the buffer size. I am thinking the best solution would be to force the user to gracefully disable the channel to get any sw channel meta data provided by the driver. I think so, if I have well understood
TI supports three cases of data capture:
- DTC/Host collection (stop on buffer full)
- DTC/Host collection (circular buffer)
- ETB/on-chip collection (circular buffer)
We don't support ETB/on-chip collection, We use external probe or Lauterbach combiprobe for trace data collection (I think it's circular buffer)
Of if the user is at a point in their code where they know thery will stop recoding on the hOst or ETB, we provide an IOCTL that simply disables all channels.
In the ETB case we may want to simply disable any open STM channels when the user decides to stop recording as a fail safe mechanism.
Note: Periodic transmission of meta data into a small circular buffer will not work well. In cases where the data is sparse the buffer will simply be filled with meta data rather than useful data.
- OST Headers
Adding an OST header to each message is a requirement for compatibility with TI's toolchain. There are a couple of ways to approach:
Completely hidden from the user - The device specific code will know if the header is necessary. On a write, prior to the copy from user space, the device independent code would have to make a call to get a properly sized memory buffer from the device dependent code that would include the header.
User enabled - Provide an IOCTL that allows the user to put the driver in a tool-chain specific mode (like add OST headers). I agree, see my previous remark on OST headers
Regards, Doug Deao
I attached my last STM patches which introduce trace framework.
Best Regards, Philippe Langlais
From: Philippe Langlais [mailto:philippe.langlais@linaro.orgmailto:philippe.langlais@linaro.org] Sent: Wednesday, May 04, 2011 3:08 AM To: Deao, Douglas Cc: Linus Walleij Subject: Re: STM at UDS-Budapest
Hi Doug,
On STE ux500 platforms we have the same STM module (follow MIDP STP 1.0), I have already posted our current implementation to the LKML and Linaro ML, it's very similar to your proposal. I can't be present to the Linaro summit but Linus Walleij can replace me for this topic, he proposes to write a common trace framework to receive STM drivers. Attached all our current proposal and work around STM.
Regards Philippe Langlais ST-Ericsson On 3 May 2011 00:42, Deao, Douglas <d-deao@ti.commailto:d-deao@ti.com> wrote: I am hosting an introductory session on System Trace at the summit. TI's System Trace Module (STM) provides a common protocol for instrumentation messages across multiple cores and system level hardware profiling in complex SoCs. Attached is a whitepaper for background reading.
Looking forward to meeting you at the summit.
Regards, Doug Deao Texas Instruments
linaro-dev mailing list linaro-dev@lists.linaro.orgmailto:linaro-dev@lists.linaro.org http://lists.linaro.org/mailman/listinfo/linaro-dev
-- IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
-- IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
Arvind,
15:30 GMT (10:30 AM CDT) works for me.
Thanks, Doug
________________________________ From: Arvind Chauhan [mailto:Arvind.Chauhan@arm.com] Sent: Thursday, June 23, 2011 10:34 AM To: Deao, Douglas; Philippe Langlais Cc: Linus Walleij; Lee Jones; Michael Hope; linaro-dev@lists.linaro.org; pierre.peiffer@stericsson.com; loic.pallardy@stericsson.com Subject: RE: ST-E STM Driver Review
Hi Doug,
I can set up a voice bridge to get discussion rolling - let me know if following schedule suits all ('have tried to keep various TZ in mind)
24/June/2011, 15:30 GMT
Best regards, Arvind
From: Deao, Douglas [mailto:d-deao@ti.com] Sent: Wednesday, June 22, 2011 4:58 AM To: Philippe Langlais; Arvind Chauhan Cc: Linus Walleij; Lee Jones; Michael Hope; linaro-dev@lists.linaro.org; pierre.peiffer@stericsson.com; loic.pallardy@stericsson.com Subject: RE: ST-E STM Driver Review
Please find attached the first few sections. I have not spent any time on formatting, but this should be enough to get us talking about requirements.
Let me know what you think.
Thanks, Doug
________________________________ From: Philippe Langlais [mailto:philippe.langlais@linaro.org] Sent: Tuesday, June 21, 2011 3:06 AM To: Arvind Chauhan Cc: Deao, Douglas; Linus Walleij; Lee Jones; Michael Hope; linaro-dev@lists.linaro.org; pierre.peiffer@stericsson.com; loic.pallardy@stericsson.com Subject: Re: ST-E STM Driver Review
Hi,
As stated by Douglas, a character driver is more natural for a printf like usage, therefore we need a mean to do packetizing (with optional OST header: can be supported by generic functions in kernel hw tracing framework). My current implementation use only one device node (/dev/stm) for all STM channels (no minor<256 restriction even for STM v2.0), each "open+write" on this device node allocate dynamically a new channel and IOCTLs are available to retrieve channel number. This driver covers low level interface too (direct write with no overhead from 1 to 8 bytes with mmap + IOCTLs to allocate/free/configure channels) for a usage with an optimized user space library.
The hardware tracing framework, I started to develop, only covers trace device register/unregister and offers a set of common hooks to the standard kernel tracing infrastructure, all things traced into trace ring buffer (ftrace) could be redirected in real time towards dedicated STM channels, it's only a draft, we can add OST header management here and others features.
I attach my current user+kernel STM interface. Now, I'm waiting of Doug's generic STM specification document for review and for pursue STM driver development for future upstreaming.
Best regards, Philippe On 20 June 2011 14:57, Arvind Chauhan <Arvind.Chauhan@arm.commailto:Arvind.Chauhan@arm.com> wrote: Hi Philippe,
Some comments on stm-trace.txt, for points 1, 2, 3.
Maybe I can schedule a conference call this week to go over some of the notes, let me know.
Best regards, Arvind
-----Original Message----- From: Philippe Langlais [mailto:philippe.langlais@linaro.orgmailto:philippe.langlais@linaro.org] Sent: Thursday, May 26, 2011 8:50 PM To: Deao, Douglas; Linus Walleij; Lee Jones Cc: Arvind Chauhan; Michael Hope; linaro-dev@lists.linaro.orgmailto:linaro-dev@lists.linaro.org; pierre.peiffer@stericsson.commailto:pierre.peiffer@stericsson.com; loic.pallardy@stericsson.commailto:loic.pallardy@stericsson.com Subject: Re: ST-E STM Driver Review
Hi Doug,
I initiate the work to build a hardware trace framework in the kernel, I'm not started the study to have a common userspace API for STM, thanks to this email we can start such a work, but it may be long (next week I'm in vacation till June 7th). See my detailed response for all your interrogations and my thoughts about STE STM implementation below: On 25 May 2011 23:54, Deao, Douglas <d-deao@ti.commailto:d-deao@ti.com> wrote: Sorry it took a while to get back to you guys. I was visiting customers last week. Most of my comments are just highlighting the differences between TI's STM 1.0 driver and ST-E's STM 1.0 driver, but there are a few questions, observations and suggestions. At the end I included some discussion on TI's meta data and OST header requirements.
I have not had a chance to look at your actual implementation yet. Did you do anything to abstract the actual HW transport ports and control registers from the higher level driver functions? Yes, partially I think through IOCTLs & debugfs (see our stm.h userspace API)
I realize there is a lot here to work through so if you would rather schedule a conference call to talk through the differences I can do that. I would like to start work on a Linaro (Unified) STM Spec next week if I can get feedback from everybody over the next few days. I will be out of the office on 5/27 and 5/31. I hope this email is enough.
I am especially interested in details of what you guys have in mind for a "common trace framework to receive STM drivers". If by framework you mean well defined APIs that are implemented for specific devices, then I think we are in agreement. What Michael and I have talked about is a common STM user mode experience across all Linaro supported devices, making Linux user mode code 100% portable between our devices. For my point of view, the trace device framework must ease the integration of new hardware trace drivers in the kernel (not only STM MIPI) to present standard hooks in current trace infrastructure, but it can cover a common STM userspace API too.
ST-E STM Driver stm-trace.txt review:
- Software Overview
In your "Software Overview" it states:
"The end of data packet is marked by a time stamp on latest byte(s) only."
I assume that user messages can be made up of any number of bytes, half- words, words or longs (what ever is most efficient) and you simply terminate the last element of the message with a time-stamp - right? Yes, the message buffer can be mis aligned
So the question is that should the driver do packetizing or should this be left to the tracing framework on top of STM driver (time-stamping the last element of the message can be left to the intermediate logic). In my view, STM driver should be a low level entity and provide simpler interface in terms of raw writes of 32bits or multiples - this may allow tracing framework to directly dump information over STM instead of buffering in between, making it less intrusive.
In the TI STM implementation a message can be any number and combination of bytes, half-words, or word transfers terminated with a time-stamp on the last element. In addition to that we also add an OST header to a message. (See below for discussion on OST header). In our case the OST header is added by the external capture probe.
Same as above, I think OST should be sitting on top of STM driver instead of being part of it - Allowing non-OST implementations to coexist.
- Lossless/Lossy modes.
TI only supports lossless mode for sw generated messages and is enforced in our hw implementation. Lossy mode is reserved for true hw messages. For STE, hw messages are always lossy, but sw generated messages could be configured in lossless (default) or lossy mode.
In my view, STM driver should expose all functionalities of the underlying hardware - leaving guaranteed/invariant choice to the tracing framework on top.
I did not notice that you documented a way to modify this through the debugfs API or IOCTLS. I have 2 IOCTLs (STM_SET_MODE & STM_GET_MODE) and debugfs (masters_modes) interfaces for that.
We expose channels through debugfs interface on demand basis. Channel allocation creates 2 debugfs nodes on STM mount point, one used for dumping trace data and other for configuring modes like guaranteed/invariant/timestamp etc. for the channel.
I am kind of thinking that may be ok since this is really a hw configuration choice in your case, but in the TI case the user does not get to make that choice. OK
- Channel Assignment
TI makes the assignment with mknod using the minor number to assign a fixed channel. This allows the user mode application to overload the channel usage for categorizing data (not my idea). I think we see the error of our ways here and will be ok with a dynamic channel allocation. If we are agree with dynamic channel allocation then a common STM userspace API is possible I think and perhaps common STM driver.
mknod would allow only 256 channels. We use debugfs nodes to allocate any number of channels, which appears automatically in debugfs mount point. When programatically accessing STM i/f we allow apps to allocate channels in terms of blocks using mmap node. When an app calls mmap API, we attach a page fault handler to the mapped region, thus when for the first time app touches this area it generates a page fault. In the handler, we allocate a block if available or preempt it from some dormant process and attach to the faulting VMA.
I am thinking that for each unique pid a channel should be assigned when the device is opened. I would guess you are keeping a channel table around and write() just checks the table for a pid assignment (no time to look at your implementation yet), if none is found the first free channel is used. If you moved this function back to open then you could do the IOCTL STM_GET_CHANNEL_NO anytime, not just after the first write. The reason behind this behavior is for our current STM user lib which open "/dev/stm" and alloc/free channels with IOCTL and never use write operation (only mmap + direct write) and in this case we don't want to loose a channel. I think we can change this behavior. We can support multiple channels allocated for one Process to avoid contention in multi-threaded process.
In write how do you flag an error if you exhaust the number of available channels? I return -ENOMEM.
We don't return error, we preempt blocks from other process
- Kernel API
TI does not support a Kernel API (yet). I can see that the Alloc/Free and File IO type functions are useful and should be standard.
Not sure what you mean by "lockless" trace functions? Not protected against concurrent access, it's the responsibility to the caller.
It looks like your "low level atomic trace functions for 1, 2, 4 or 8 bytes" is similar to TI's binary library functions (not supported by the TI STM Driver). This is what we use the OST header for, allowing our tool chain to differentiate between different message formats, rather than just assuming the data is a simple stream of bytes. I'm not ready to add an OST header to each trace messages to particularly for kernel function traces (ftrace has short messages 8 bytes size with high throughput), I rather to dedicate a specific channel for this, OST header has to be optional.
- Debugfs APIs.
TI used a different approach. The tool-chain on the host provides all the transport setup through JTAG, so our driver does not support setting up the actual STM data export (number of pins and clock rate). In our case device transport parameters must match the host receiver's collection setup.
With your approach the user can change the clock rate and export pin width effectively at any time. Our tools actually go through a calibration process during initialization so any changes to the device's transport setup (clock rate, number of pins data exported on) would cause the TI tool chain a lot of grief.
There are some parameters we know we need to add (like master enables). This are currently also handled by the host tools. TI's STM module allows up to 4 SW masters to be enabled (with id masks that can be used to enable multiple masters from the same group) and 4 HW masters that can be enabled at the same time as the SW masters. If the user tries to enable more than the HW allows do you have a mechanism to flag an error? We can't face the same problem with STE STM, our hardware support 6 SW masters & 2 HW masters without restriction, all could be enabled at same time (managed by a bit field register)
I don't have a lot of experience with debugfs but I am assuming it's primarily used for allowing scripts to configure a driver (like in your example) or extract information. It's the current way to configure kernel tracing infrastructure (from console, through scripts or user apps), kernel users are familiar with this interface.
We may want to define a standard set of debugfs options whose implementation is vendor specific. But that raises some questions:
- How do we deal with options that don't make sense for a specific
vendor? Maybe just doing nothing is acceptable or do we want to provide a discovery mechanism? Do nothing may be sufficient, we can add STM IP specific string descriptor too, something like "vendor id+STM IP id+STP version"
- Would user scripts then also be vendor specific?
We should probably make an effort to avoid this. A discovery mechanism may allow user mode scripts to be generic. Yes, but STM configuration scripts are device specific, to see.
- Mapped Channels
I believe the TI hw transport channel mapping is compatible. In the TI case a channel is mapped into two spaces, the first half is for non- timestamp transfers and the second half is for time-stamped transfers. When we write a message (from a user mode write call for example) we simply write all the data except the last element through the non- timestamp port, and then the last element is written to the time-stamped port. So I think we could be compatible here. I think so, we have similar channel memory mapping but, one STM channel is accessed through this C union of 2 consecutive 64bits word (first for non-timestamp, 2nd with timestamp) /* One single channel mapping */ struct stm_channel { union { __u8 no_stamp8; __u16 no_stamp16; __u32 no_stamp32; __u64 no_stamp64; }; union { __u8 stamp8; __u16 stamp16; __u32 stamp32; __u64 stamp64; }; };
With that said I am not sure about exposing all channels to a user mode library. You are relying on the library to use the convention of getting a free channel from the driver to make sure there are no conflicts. If the channel assignment is made when you open the device, you could conceivably map just the address space needed for the single channel, thus eliminating the need to get a free channel from the driver. In the TI case a single channel's transport mapping is 4K bytes, which matches the typical PAGE_SIZE. I realize not all hw implementations will match up with the PAGE_SIZE, which may be why you simply map all the channels back to user space. It's our case, the 256 channels is mapped in the same 4K memory page.
Since free channels can become busy rapidly, maybe a better convention would be to simply use another device node if the user wants the library STM data to be transmitted on a different STM channel than the current process. This may be a case where providing a mechanism (see meta data discussion below) to allow channels to be named for the toolchain may be a good idea (provide task name and process id). In our current implementation, a new open on the same "/dev/stm" device allocate a new channel at first write.
- 8-byte Writes
TI does not support 64-bit writes with our STM 1.0 module. We may need an IOCTL to get the largest transfer supported for the mmap case. For all other cases this should just be hidden in the device dependent code. I think so
- Kernel Internal Usage
I like the idea of having dedicated support in the driver for common kernel logging. Any ideas on how you would support kernel STM channel assignments without hard-coding? But configurable through kernel configuration, the other way is to dedicate a specific channel for this signalization like your channel 255, not so easy to define.
We may need a mechanism to communicate the definition of each hard-coded channel to our tools. to see
The following are TI specific:
- Data protection
In SMP systems if the processor is switched a new master is generated (in some TI devices). So we protect the data with a mutex to guarantee a complete message is generated by the same master. OK, it's not our case.
- Meta Data
Our user mode HW libraries use meta data to transport data needed to process the HW profiling STM messages. Items like processor speed, sampling rate, processing options, ... (just a predefined byte buffer our tool-chain understands). The meta data is currently broadcasted on a dedicated channel (255), which conflicts with your hard-coded channel for logging printk output. So we will need to resolve hard-coded conflicts. Easy to fix, we can reserve dedicated channels at driver init and change our hard-coded affectation.
We need the driver to support registration and transport of the meta data on demand from the library (when the HW master is disabled, in case the collection buffer is small and circular).
I am thinking an IOCTL could be used to register meta data and then the data simply broadcast on a STM channel (will need to figure out which one) when the HW master is enabled and disabled. We don't face the same problematic
Meta data transmission is problematic for circular buffers (like ETB's) thus the reason for also sending meta data when a hw master is disabled. SW masters are not typically disabled, and our HW does not provide a transmission byte count (remember there are HW messages also being generated in the TI case). So there is no way from a driver we can tell when the recoding buffer will wrap even if the user told us the buffer size. I am thinking the best solution would be to force the user to gracefully disable the channel to get any sw channel meta data provided by the driver. I think so, if I have well understood
TI supports three cases of data capture:
- DTC/Host collection (stop on buffer full)
- DTC/Host collection (circular buffer)
- ETB/on-chip collection (circular buffer)
We don't support ETB/on-chip collection, We use external probe or Lauterbach combiprobe for trace data collection (I think it's circular buffer)
Of if the user is at a point in their code where they know thery will stop recoding on the hOst or ETB, we provide an IOCTL that simply disables all channels.
In the ETB case we may want to simply disable any open STM channels when the user decides to stop recording as a fail safe mechanism.
Note: Periodic transmission of meta data into a small circular buffer will not work well. In cases where the data is sparse the buffer will simply be filled with meta data rather than useful data.
- OST Headers
Adding an OST header to each message is a requirement for compatibility with TI's toolchain. There are a couple of ways to approach:
Completely hidden from the user - The device specific code will know if the header is necessary. On a write, prior to the copy from user space, the device independent code would have to make a call to get a properly sized memory buffer from the device dependent code that would include the header.
User enabled - Provide an IOCTL that allows the user to put the driver in a tool-chain specific mode (like add OST headers). I agree, see my previous remark on OST headers
Regards, Doug Deao
I attached my last STM patches which introduce trace framework.
Best Regards, Philippe Langlais
From: Philippe Langlais [mailto:philippe.langlais@linaro.orgmailto:philippe.langlais@linaro.org] Sent: Wednesday, May 04, 2011 3:08 AM To: Deao, Douglas Cc: Linus Walleij Subject: Re: STM at UDS-Budapest
Hi Doug,
On STE ux500 platforms we have the same STM module (follow MIDP STP 1.0), I have already posted our current implementation to the LKML and Linaro ML, it's very similar to your proposal. I can't be present to the Linaro summit but Linus Walleij can replace me for this topic, he proposes to write a common trace framework to receive STM drivers. Attached all our current proposal and work around STM.
Regards Philippe Langlais ST-Ericsson On 3 May 2011 00:42, Deao, Douglas <d-deao@ti.commailto:d-deao@ti.com> wrote: I am hosting an introductory session on System Trace at the summit. TI's System Trace Module (STM) provides a common protocol for instrumentation messages across multiple cores and system level hardware profiling in complex SoCs. Attached is a whitepaper for background reading.
Looking forward to meeting you at the summit.
Regards, Doug Deao Texas Instruments
linaro-dev mailing list linaro-dev@lists.linaro.orgmailto:linaro-dev@lists.linaro.org http://lists.linaro.org/mailman/listinfo/linaro-dev
-- IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
-- IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
-
Arnd Bergmann -
Arvind Chauhan -
Deao, Douglas -
Philippe Langlais