Hi Al, Mike,

On Thu, Apr 23, 2020 at 03:26:06PM +0100, Mike Leach wrote:

[...]

...

The patch below does the kernel and userspace side but is not complete. The problem is that userspace perf creates the metadata copy of TRCCONFIGR based on its request (and fills in the other id registers by reading sysfs), but the detection of EL2/E2H happens in the kernel which adjusts TRCCONFIGR, and it's this config which is needed for decode. I see three ways round this:

• have userspace test to see if the kernel is EL2 (somehow) and adjust the

metadata to mirror what the kernel is doing

• have the kernel pass the adjusted TRCCONFIGR back so perf can put it in the

metadata

Not necessarily the entire configr but at least a flag to indicate VMID contexts are in play.

...

• have the perf decoder get the thread id from whichever of VMID and

CONTEXTID is available in a PE_CONTEXT element

Obviously, the last is simplest, but it's a bodge, and means that OpenCSD will see VMIDs when its TRCCONFIGR says it won't. It's kind of cleanest to get

OpenCSD won't care. VMID/CID are pass through for the decoder - it's up to the client to use them or not. We use the trace CONFIGR value to spot things like return stack. The ETMv4 protocol tells us if VMID / CID is present and just believe that. The decode doesn't validate what it gets in trace from what is set up in the config / id registers in general when the content self describes. (we do use the architecture version and D / I settings to do a broad brush rejection of certain packet types, but that is about it).

...

the real TRCCONFIGR somehow, but how do we do that?

As I mentioned on the other thread, I think this problem ties in with the sink information issue - both want to get kernel selections into the AUX DATA headers or something similar. A common approach will be of benefit here.

I did a quick check and found OpenCSD's header has defined a field to indicate the exception level; so just curious if we can simply check the exception level in OpenCSD's element, if the exception level is EL2, then perf tool uses VMID rather than CID for the process ID.

If this is feasible, then here don't depend on TRCCONFIGR anymore. Does this make sense? Sorry might introduce noise.

Thanks, Leo

Hi Al, Suzuki,

On Wed, May 06, 2020 at 10:17:15AM +0100, Suzuki Kuruppassery Poulose wrote:

On 05/05/2020 05:00 PM, Al Grant wrote:

...

...

...

The patch below does the kernel and userspace side but is not complete. The problem is that userspace perf creates the metadata copy of TRCCONFIGR based on its request (and fills in the other id registers by reading sysfs), but the detection of EL2/E2H happens in the kernel which adjusts TRCCONFIGR, and it's this config which is needed for decode. I

see three ways round this:

...

• have userspace test to see if the kernel is EL2 (somehow) and adjust

the metadata to mirror what the kernel is doing

• have the kernel pass the adjusted TRCCONFIGR back so perf can put it

in the metadata

• have the perf decoder get the thread id from whichever of VMID and

CONTEXTID is available in a PE_CONTEXT element

Obviously, the last is simplest, but it's a bodge, and means that OpenCSD will see VMIDs when its TRCCONFIGR says it won't. It's kind of cleanest to get the real TRCCONFIGR somehow, but how do we do that?

We do get TRCCONFIGR in the perf records. We should simply make sure we get the uptodate value (wherever we are getting it from).

The copy in PERF_RECORD_AUXINFO (which is a synthetic record created by userspace perf) is, I believe, as I said:

"userspace perf creates the metadata copy of TRCCONFIGR based on its request".

So if the kernel modifies it based on information only the kernel knows, there's no current way to get the actual value. That was what I was trying to address with my suggestions.

Have I missed some place the actual TRCCONFIGR is already being returned in other perf records?

The sysfs is one place where we could expose this and this could be then consumed by the perf tool while creating the TRCCONFIGR. Since the EL can't change, this could be a onetime probe activity.

I spent some time to look into this issue, I think we need to note one thing is: since the metadata is captured at the beginning when execute Perf tool, it should be prior to setting TRCCONFIGR in the CoreSight driver when enable perf event. Especially, if we consider for the per-thread case, if the traced task is not migrated on a specific CPU (let's say CPU_a), then CPU_a's TRCCONFIGR will not be set properly until the trace task is migrated to CPU_a.

So we can see in the perf tool it has the function cs_etmv4_get_config(), it doesn't read out any value from sysfs for the register TRCCONFIGR, alternatively it just generates a value for TRCCONFIGR based on perf's 'attr.config'. On the other hand, in the kernel side, it needs to maintain the same logic in the function etm4_parse_event_config(), which also parses 'attr.config' and set into TRCCONFIGR.

For fixing this issue, I think one potential direction is to change the function etm_event_init() in the kernel, we can use it to invoke a function like etm4_parse_event_config(), so this can allow the register TRCCONFIGR to be ready in the initialisation phase. Then, as Suzuki suggested, in the perf tool we can use sysfs node to read register TRCCONFIGR. To be honest, I don't verify this is feasible or not, but from reading the code, looks like this is feasible with below flow:

record__open() `-> evsel__open() `-> evsel__open_cpu() `-> perf_event_open() `-> perf_init_event() `-> perf_try_init_event() `-> etm_event_init() `-> Set TRCCONFIGR

record__synthesize() `-> perf_event__synthesize_auxtrace_info() `-> auxtrace_record__info_fill() `-> cs_etm_info_fill() `-> Use sysfs to read out TRCCONFIGR

Thanks, Leo

From: CoreSight coresight-bounces@lists.linaro.org On Behalf Of Mathieu Poirier Sent: 08 May 2020 16:44 To: leo.yan@linaro.org Cc: Coresight ML coresight@lists.linaro.org Subject: Re: Context id tracing with kernel at EL2

On Fri, 8 May 2020 at 00:11, Leo Yan <leo.yan@linaro.orgmailto:leo.yan@linaro.org> wrote: Hi Al, Suzuki,

On Wed, May 06, 2020 at 10:17:15AM +0100, Suzuki Kuruppassery Poulose wrote:

On 05/05/2020 05:00 PM, Al Grant wrote:

...

...

...

The patch below does the kernel and userspace side but is not complete. The problem is that userspace perf creates the metadata copy of TRCCONFIGR based on its request (and fills in the other id registers by reading sysfs), but the detection of EL2/E2H happens in the kernel which adjusts TRCCONFIGR, and it's this config which is needed for decode. I

see three ways round this:

...

• have userspace test to see if the kernel is EL2 (somehow) and adjust

the metadata to mirror what the kernel is doing

• have the kernel pass the adjusted TRCCONFIGR back so perf can put it

in the metadata

• have the perf decoder get the thread id from whichever of VMID and

CONTEXTID is available in a PE_CONTEXT element

Obviously, the last is simplest, but it's a bodge, and means that OpenCSD will see VMIDs when its TRCCONFIGR says it won't. It's kind of cleanest to get the real TRCCONFIGR somehow, but how do we do that?

We do get TRCCONFIGR in the perf records. We should simply make sure we get the uptodate value (wherever we are getting it from).

The copy in PERF_RECORD_AUXINFO (which is a synthetic record created by userspace perf) is, I believe, as I said:

"userspace perf creates the metadata copy of TRCCONFIGR based on its request".

So if the kernel modifies it based on information only the kernel knows, there's no current way to get the actual value. That was what I was trying to address with my suggestions.

Have I missed some place the actual TRCCONFIGR is already being returned in other perf records?

The sysfs is one place where we could expose this and this could be then consumed by the perf tool while creating the TRCCONFIGR. Since the EL can't change, this could be a onetime probe activity.

I'm joining this thread on the late and as such don't have all the context. But sysfs is never a good place to export run-time configuration registers as there could be multiple perf sessions active at the same time. This isn't a big issue for N:1 topologies but very real for 1:1.

I agree. Event-specific data (e.g. actual TRCCONFIGR) should be returned through some mechanism specific to the event. For example, we could have a new ioctl() specific to CoreSight events (“return trace config”). Or we could use one of the flag bits in the PERF_RECORD_AUX record. There are 64 bits, of which 4 are currently used as generic flags (PERF_AUX_FLAG_TRUNCATED etc.). If we could reserve some of those as event-specific flags (say PERF_AUX_FLAG_EVENT0..7), then for ETM AUX buffers we could use a flag to indicate if the thread id is in CID or VMID in the trace. That would be my preference.

The suggestion might have been to use sysfs to indicate if the kernel is at EL1 or EL2. That’s not runtime varying, so sysfs should be fine for that. However it then forces perf userspace to make the inference that at EL2 the thread id is always in VMID. That’s currently the case, but it’s the sort of heuristic rule we really don’t want to be pushing into perf.

And if we do that it will need to be captured at “perf record” time and stored in a new field in PERF_RECORD_AUXTRACE_INFO. The advantage of using an AUX flag is that we don’t change any formats – it’s just using a flag in an existing flags word to help us decode traces that are currently undecodeable.

Al

I spent some time to look into this issue, I think we need to note one thing is: since the metadata is captured at the beginning when execute Perf tool, it should be prior to setting TRCCONFIGR in the CoreSight driver when enable perf event. Especially, if we consider for the per-thread case, if the traced task is not migrated on a specific CPU (let's say CPU_a), then CPU_a's TRCCONFIGR will not be set properly until the trace task is migrated to CPU_a.

So we can see in the perf tool it has the function cs_etmv4_get_config(), it doesn't read out any value from sysfs for the register TRCCONFIGR, alternatively it just generates a value for TRCCONFIGR based on perf's 'attr.config'. On the other hand, in the kernel side, it needs to maintain the same logic in the function etm4_parse_event_config(), which also parses 'attr.config' and set into TRCCONFIGR.

For fixing this issue, I think one potential direction is to change the function etm_event_init() in the kernel, we can use it to invoke a function like etm4_parse_event_config(), so this can allow the register TRCCONFIGR to be ready in the initialisation phase. Then, as Suzuki suggested, in the perf tool we can use sysfs node to read register TRCCONFIGR. To be honest, I don't verify this is feasible or not, but from reading the code, looks like this is feasible with below flow:

record__open() `-> evsel__open() `-> evsel__open_cpu() `-> perf_event_open() `-> perf_init_event() `-> perf_try_init_event() `-> etm_event_init() `-> Set TRCCONFIGR

record__synthesize() `-> perf_event__synthesize_auxtrace_info() `-> auxtrace_record__info_fill() `-> cs_etm_info_fill() `-> Use sysfs to read out TRCCONFIGR

Thanks, Leo _______________________________________________ CoreSight mailing list CoreSight@lists.linaro.orgmailto:CoreSight@lists.linaro.org https://lists.linaro.org/mailman/listinfo/coresight IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

On Fri, 22 May 2020 at 09:30, Suzuki K Poulose suzuki.poulose@arm.com wrote:

On 05/08/2020 04:43 PM, Mathieu Poirier wrote:

...

On Fri, 8 May 2020 at 00:11, Leo Yan <leo.yan@linaro.org mailto:leo.yan@linaro.org> wrote:

Hi Al, Suzuki,

On Wed, May 06, 2020 at 10:17:15AM +0100, Suzuki Kuruppassery
Poulose wrote:
 > On 05/05/2020 05:00 PM, Al Grant wrote:
 > > > > The patch below does the kernel and userspace side but is
not complete.
 > > > > The problem is that userspace perf creates the metadata copy of
 > > > > TRCCONFIGR based on its request (and fills in the other id
registers
 > > > > by reading sysfs), but the detection of EL2/E2H happens in
the kernel
 > > > > which adjusts TRCCONFIGR, and it's this config which is
needed for decode. I
 > > > see three ways round this:
 > > > >
 > > > > - have userspace test to see if the kernel is EL2 (somehow)
and adjust
 > > > > the metadata to mirror what the kernel is doing
 > > > >
 > > > > - have the kernel pass the adjusted TRCCONFIGR back so perf
can put it
 > > > > in the metadata
 > > > >
 > > > > - have the perf decoder get the thread id from whichever of
VMID and
 > > > > CONTEXTID is available in a PE_CONTEXT element
 > > > >
 > > > > Obviously, the last is simplest, but it's a bodge, and
means that
 > > > > OpenCSD will see VMIDs when its TRCCONFIGR says it won't.
It's kind of
 > > > > cleanest to get the real TRCCONFIGR somehow, but how do we
do that?
 > > >
 > > > We do get TRCCONFIGR in the perf records. We should simply
make sure we get
 > > > the uptodate value (wherever we are getting it from).
 > >
 > > The copy in PERF_RECORD_AUXINFO (which is a synthetic record
created
 > > by userspace perf) is, I believe, as I said:
 > >
 > > "userspace perf creates the metadata copy of TRCCONFIGR based on
 > > its request".
 > >
 > > So if the kernel modifies it based on information only the
kernel knows,
 > > there's no current way to get the actual value. That was what I
was trying
 > > to address with my suggestions.
 > >
 > > Have I missed some place the actual TRCCONFIGR is already being
returned
 > > in other perf records?
 >
 > The sysfs is one place where we could expose this and this could
be then
 > consumed by the perf tool while creating the TRCCONFIGR. Since the EL
 > can't change, this could be a onetime probe activity.

I'm joining this thread on the late and as such don't have all the context. But sysfs is never a good place to export run-time configuration registers as there could be multiple perf sessions active at the same time. This isn't a big issue for N:1 topologies but very real for 1:1.

I don't see how this affected by the sink topology, as this is an ETM configuration.

By sysfs, I mean, not necessarily under /sys/bus/coreight/.../etmN/

but, under :

/sys/bus/event_source/.../cs_etm/

We already expose various different information there. e.g, nr_addr_filters, sinks/

So we could expose something like :

trcconfigr/

and could describe how each of the individual event configs affect the TRCCONFIGR register. So that the perf tool can construct the TRCCONFIGR register from scratch just by looking at the trcconfigr directory.

e.g,

# cd /sys/bus/event_source/devices/cs_etm/ # for f in $(find format -type f); do echo $(basename $f) - $(cat $f); done cycacc - config:12 sinkid - config2:0-31 contextid - config:14 retstack - config:29 timestamp - config:28

$ for f in $(find trcconfigr -type f); do echo $(basename $f) - $(cat $f); done cycacc - 0x10 // Bit 4 timestamp - 0x800 // Bit 11 contextid - 0x8080 // Bit15 | Bit 7 On EL2 kernels - 0x40 // Bit 6 on EL1 kernels retstack - 0x1000 // Bit12

So those are configurations that apply to specific trace sessions. When dealing with concurrent trace sessions there is no way to guarantee the information exported to sysfs applies to the trace session that would read the information.

That way, the perf tool can scale automatically to cover for the future config changes that affect the TRCCONFIGR. It simply needs to map the "config-name" to trace config by the kernel provided bits. This will also solve the etm3 vs etm4 differences in bit positions and hopefully for the future IPs.

Suzuki

...

I spent some time to look into this issue, I think we need to note one
thing is: since the metadata is captured at the beginning when execute
Perf tool, it should be prior to setting TRCCONFIGR in the CoreSight
driver when enable perf event.  Especially, if we consider for the
per-thread case, if the traced task is not migrated on a specific CPU
(let's say CPU_a), then CPU_a's TRCCONFIGR will not be set properly
until the trace task is migrated to CPU_a.

So we can see in the perf tool it has the function
cs_etmv4_get_config(), it doesn't read out any value from sysfs
for the register TRCCONFIGR, alternatively it just generates a value
for TRCCONFIGR based on perf's 'attr.config'.  On the other hand, in
the kernel side, it needs to maintain the same logic in the function
etm4_parse_event_config(), which also parses 'attr.config' and set
into TRCCONFIGR.

For fixing this issue, I think one potential direction is to change
the function etm_event_init() in the kernel, we can use it to invoke
a function like etm4_parse_event_config(), so this can allow the
register TRCCONFIGR to be ready in the initialisation phase.  Then,
as Suzuki suggested, in the perf tool we can use sysfs node to read
register TRCCONFIGR.  To be honest, I don't verify this is feasible
or not, but from reading the code, looks like this is feasible with
below flow:

   record__open()
     `-> evsel__open()
           `-> evsel__open_cpu()
                 `-> perf_event_open()
                       `-> perf_init_event()
                             `-> perf_try_init_event()
                                   `-> etm_event_init()
                                         `-> Set TRCCONFIGR

   record__synthesize()
     `-> perf_event__synthesize_auxtrace_info()
           `-> auxtrace_record__info_fill()
                 `-> cs_etm_info_fill()
                       `-> Use sysfs to read out TRCCONFIGR

Thanks,
Leo
_______________________________________________
CoreSight mailing list
CoreSight@lists.linaro.org <mailto:CoreSight@lists.linaro.org>
https://lists.linaro.org/mailman/listinfo/coresight

Hi,

On Tue, 26 May 2020 at 17:21, Suzuki K Poulose suzuki.poulose@arm.com wrote:

On 05/25/2020 04:56 PM, Mathieu Poirier wrote:

...

On Fri, 22 May 2020 at 09:30, Suzuki K Poulose suzuki.poulose@arm.com wrote:

...

On 05/08/2020 04:43 PM, Mathieu Poirier wrote:

...

On Fri, 8 May 2020 at 00:11, Leo Yan <leo.yan@linaro.org mailto:leo.yan@linaro.org> wrote:

 Hi Al, Suzuki,

 On Wed, May 06, 2020 at 10:17:15AM +0100, Suzuki Kuruppassery
 Poulose wrote:
  > On 05/05/2020 05:00 PM, Al Grant wrote:
  > > > > The patch below does the kernel and userspace side but is
 not complete.
  > > > > The problem is that userspace perf creates the metadata copy of
  > > > > TRCCONFIGR based on its request (and fills in the other id
 registers
  > > > > by reading sysfs), but the detection of EL2/E2H happens in
 the kernel
  > > > > which adjusts TRCCONFIGR, and it's this config which is
 needed for decode. I
  > > > see three ways round this:
  > > > >
  > > > > - have userspace test to see if the kernel is EL2 (somehow)
 and adjust
  > > > > the metadata to mirror what the kernel is doing
  > > > >
  > > > > - have the kernel pass the adjusted TRCCONFIGR back so perf
 can put it
  > > > > in the metadata
  > > > >
  > > > > - have the perf decoder get the thread id from whichever of
 VMID and
  > > > > CONTEXTID is available in a PE_CONTEXT element
  > > > >
  > > > > Obviously, the last is simplest, but it's a bodge, and
 means that
  > > > > OpenCSD will see VMIDs when its TRCCONFIGR says it won't.
 It's kind of
  > > > > cleanest to get the real TRCCONFIGR somehow, but how do we
 do that?
  > > >
  > > > We do get TRCCONFIGR in the perf records. We should simply
 make sure we get
  > > > the uptodate value (wherever we are getting it from).
  > >
  > > The copy in PERF_RECORD_AUXINFO (which is a synthetic record
 created
  > > by userspace perf) is, I believe, as I said:
  > >
  > > "userspace perf creates the metadata copy of TRCCONFIGR based on
  > > its request".
  > >
  > > So if the kernel modifies it based on information only the
 kernel knows,
  > > there's no current way to get the actual value. That was what I
 was trying
  > > to address with my suggestions.
  > >
  > > Have I missed some place the actual TRCCONFIGR is already being
 returned
  > > in other perf records?
  >
  > The sysfs is one place where we could expose this and this could
 be then
  > consumed by the perf tool while creating the TRCCONFIGR. Since the EL
  > can't change, this could be a onetime probe activity.

I'm joining this thread on the late and as such don't have all the context. But sysfs is never a good place to export run-time configuration registers as there could be multiple perf sessions active at the same time. This isn't a big issue for N:1 topologies but very real for 1:1.

I don't see how this affected by the sink topology, as this is an ETM configuration.

By sysfs, I mean, not necessarily under /sys/bus/coreight/.../etmN/

but, under :

/sys/bus/event_source/.../cs_etm/

We already expose various different information there. e.g, nr_addr_filters, sinks/

So we could expose something like :

trcconfigr/

and could describe how each of the individual event configs affect the TRCCONFIGR register. So that the perf tool can construct the TRCCONFIGR register from scratch just by looking at the trcconfigr directory.

e.g,

# cd /sys/bus/event_source/devices/cs_etm/ # for f in $(find format -type f); do echo $(basename $f) - $(cat $f); done cycacc - config:12 sinkid - config2:0-31 contextid - config:14 retstack - config:29 timestamp - config:28

$ for f in $(find trcconfigr -type f); do echo $(basename $f) - $(cat $f); done cycacc - 0x10 // Bit 4 timestamp - 0x800 // Bit 11 contextid - 0x8080 // Bit15 | Bit 7 On EL2 kernels - 0x40 // Bit 6 on EL1 kernels retstack - 0x1000 // Bit12

So those are configurations that apply to specific trace sessions. When dealing with concurrent trace sessions there is no way to guarantee the information exported to sysfs applies to the trace session that would read the information.

I think there is a bit of confusion here; The idea is the perf tool will construct the value of the traceconfigr based on the "options" selected by the user, just like it does now for "config1"/"config2"

i.e,

perf record -e cs_etm/cycacc,timestamp/ blah-blah

perf tool will :

map cycacc -> config1 |= BIT(12) and trcconfigr |= BIT($cat /sys/.../trcconfigr/cycacc)

and similarly for all the options. The values under the "trcconfigr" are static and only maps a given attribute to the traceconfig register bit.

Thus, any session can create the TRCCONFIGR accurately based on the events now, by dynamically looking up under the sysfs, rather than "compile time" static assumption of the the ETM model.

So how does this work when I copy the captured data off my target and try to decode offline?

Regards

Mike

Cheers Suzuki _______________________________________________ CoreSight mailing list CoreSight@lists.linaro.org https://lists.linaro.org/mailman/listinfo/coresight

HI Al,

On Tue, 26 May 2020 at 19:53, Al Grant Al.Grant@arm.com wrote:

...

-----Original Message----- From: CoreSight coresight-bounces@lists.linaro.org On Behalf Of Mike Leach Sent: 26 May 2020 17:33 To: Suzuki Poulose Suzuki.Poulose@arm.com Cc: Coresight ML coresight@lists.linaro.org Subject: Re: Context id tracing with kernel at EL2

Hi,

On Tue, 26 May 2020 at 17:21, Suzuki K Poulose suzuki.poulose@arm.com wrote:

...

On 05/25/2020 04:56 PM, Mathieu Poirier wrote:

...

On Fri, 22 May 2020 at 09:30, Suzuki K Poulose suzuki.poulose@arm.com

wrote:

...

...

...

On 05/08/2020 04:43 PM, Mathieu Poirier wrote:

...

On Fri, 8 May 2020 at 00:11, Leo Yan <leo.yan@linaro.org mailto:leo.yan@linaro.org> wrote:

 Hi Al, Suzuki,

 On Wed, May 06, 2020 at 10:17:15AM +0100, Suzuki Kuruppassery
 Poulose wrote:
  > On 05/05/2020 05:00 PM, Al Grant wrote:
  > > > > The patch below does the kernel and userspace side but is
 not complete.
  > > > > The problem is that userspace perf creates the metadata copy

of

...

...

...

...

  > > > > TRCCONFIGR based on its request (and fills in the other id
 registers
  > > > > by reading sysfs), but the detection of EL2/E2H happens in
 the kernel
  > > > > which adjusts TRCCONFIGR, and it's this config which is
 needed for decode. I
  > > > see three ways round this:
  > > > >
  > > > > - have userspace test to see if the kernel is EL2 (somehow)
 and adjust
  > > > > the metadata to mirror what the kernel is doing
  > > > >
  > > > > - have the kernel pass the adjusted TRCCONFIGR back so perf
 can put it
  > > > > in the metadata
  > > > >
  > > > > - have the perf decoder get the thread id from whichever of
 VMID and
  > > > > CONTEXTID is available in a PE_CONTEXT element
  > > > >
  > > > > Obviously, the last is simplest, but it's a bodge, and
 means that
  > > > > OpenCSD will see VMIDs when its TRCCONFIGR says it won't.
 It's kind of
  > > > > cleanest to get the real TRCCONFIGR somehow, but how do we
 do that?
  > > >
  > > > We do get TRCCONFIGR in the perf records. We should simply
 make sure we get
  > > > the uptodate value (wherever we are getting it from).
  > >
  > > The copy in PERF_RECORD_AUXINFO (which is a synthetic record
 created
  > > by userspace perf) is, I believe, as I said:
  > >
  > > "userspace perf creates the metadata copy of TRCCONFIGR based

on

...

...

...

...

  > > its request".
  > >
  > > So if the kernel modifies it based on information only the
 kernel knows,
  > > there's no current way to get the actual value. That was what I
 was trying
  > > to address with my suggestions.
  > >
  > > Have I missed some place the actual TRCCONFIGR is already being
 returned
  > > in other perf records?
  >
  > The sysfs is one place where we could expose this and this could
 be then
  > consumed by the perf tool while creating the TRCCONFIGR. Since the

EL

...

...

...

...

  > can't change, this could be a onetime probe activity.

I'm joining this thread on the late and as such don't have all the context. But sysfs is never a good place to export run-time configuration registers as there could be multiple perf sessions active at the same time. This isn't a big issue for N:1 topologies but very real for 1:1.

I don't see how this affected by the sink topology, as this is an ETM configuration.

By sysfs, I mean, not necessarily under /sys/bus/coreight/.../etmN/

but, under :

/sys/bus/event_source/.../cs_etm/

We already expose various different information there. e.g, nr_addr_filters, sinks/

So we could expose something like :

trcconfigr/

and could describe how each of the individual event configs affect the TRCCONFIGR register. So that the perf tool can construct the TRCCONFIGR register from scratch just by looking at the trcconfigr directory.

e.g,

# cd /sys/bus/event_source/devices/cs_etm/ # for f in $(find format -type f); do echo $(basename $f) - $(cat $f); done cycacc - config:12 sinkid - config2:0-31 contextid - config:14 retstack - config:29 timestamp - config:28

$ for f in $(find trcconfigr -type f); do echo $(basename $f) - $(cat $f); done cycacc - 0x10 // Bit 4 timestamp - 0x800 // Bit 11 contextid - 0x8080 // Bit15 | Bit 7 On EL2 kernels - 0x40 // Bit 6 on EL1 kernels retstack - 0x1000 // Bit12

So those are configurations that apply to specific trace sessions. When dealing with concurrent trace sessions there is no way to guarantee the information exported to sysfs applies to the trace session that would read the information.

I think there is a bit of confusion here; The idea is the perf tool will construct the value of the traceconfigr based on the "options" selected by the user, just like it does now for "config1"/"config2"

i.e,

perf record -e cs_etm/cycacc,timestamp/ blah-blah

perf tool will :

map cycacc -> config1 |= BIT(12) and trcconfigr |= BIT($cat /sys/.../trcconfigr/cycacc)

and similarly for all the options. The values under the "trcconfigr" are static and only maps a given attribute to the traceconfig register bit.

Thus, any session can create the TRCCONFIGR accurately based on the events now, by dynamically looking up under the sysfs, rather than "compile time" static assumption of the the ETM model.

So how does this work when I copy the captured data off my target and try to decode offline?

Same as it did before, but this time TRCCONFIGR in the perf.data's PERF_RECORD_AUXTRACE_INFO will correctly indicate that the thread id is in vmid.

Yeah - that makes sense - I skimmed Suzukis code and somehow assumed it was grabbing the data live on decode rather than the thing that makes most sense - putting it in the INFO structures after the record has finished.

The problem we have now is that when "perf record" is constructing PERF_RECORD_AUXTRACE_INFO, which is a synthetic record that the kernel has nothing to do with, the session-specific value of TRCCONFIGR perf uses is one that it synthesizes based on the options it is passing into the kernel, and this doesn't take account of any adjustments made by the kernel, e.g. to track CONTEXTIDR_EL2 in the vmid field. The idea is to have sysfs provide enough (non-session-specific) info about the kernel, so that "perf record" can synthesize a TRCCONFIGR that is both session-specific and adjusted to take account of what the kernel is doing. In effect "perf record" is mirroring the kernel's own logic.

The alternative is to have the kernel pass something back in a record (e.g. a flag in PERF_RECORD_AUX), again to help "perf record" construct a correct TRCCONFIGR.

It would be much cleaner if the kernel generated all metadata needed to decode the AUX buffer, but that’s not how perf is designed.

I did think that perf was designed for exactly the task of extracting data from the kernel to allow user side analysis!

However I would agree there doesn't appear to be an obvious route out for the specifics we need other than embed it in the data we are outputting - i.e. the raw trace buffer. sysfs is simpler as long as it is read at the end of the record session.

Regards

Mike

Al

...

Regards

Mike

...

Cheers Suzuki _______________________________________________ CoreSight mailing list CoreSight@lists.linaro.org https://lists.linaro.org/mailman/listinfo/coresight

-- Mike Leach Principal Engineer, ARM Ltd. Manchester Design Centre. UK _______________________________________________ CoreSight mailing list CoreSight@lists.linaro.org https://lists.linaro.org/mailman/listinfo/coresight

IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

On 05/26/2020 05:33 PM, Mike Leach wrote:

Hi,

On Tue, 26 May 2020 at 17:21, Suzuki K Poulose suzuki.poulose@arm.com wrote:

...

On 05/25/2020 04:56 PM, Mathieu Poirier wrote:

...

On Fri, 22 May 2020 at 09:30, Suzuki K Poulose suzuki.poulose@arm.com wrote:

...

On 05/08/2020 04:43 PM, Mathieu Poirier wrote:

...

On Fri, 8 May 2020 at 00:11, Leo Yan <leo.yan@linaro.org mailto:leo.yan@linaro.org> wrote:

  Hi Al, Suzuki,

  On Wed, May 06, 2020 at 10:17:15AM +0100, Suzuki Kuruppassery
  Poulose wrote:
   > On 05/05/2020 05:00 PM, Al Grant wrote:
   > > > > The patch below does the kernel and userspace side but is
  not complete.
   > > > > The problem is that userspace perf creates the metadata copy of
   > > > > TRCCONFIGR based on its request (and fills in the other id
  registers
   > > > > by reading sysfs), but the detection of EL2/E2H happens in
  the kernel
   > > > > which adjusts TRCCONFIGR, and it's this config which is
  needed for decode. I
   > > > see three ways round this:
   > > > >
   > > > > - have userspace test to see if the kernel is EL2 (somehow)
  and adjust
   > > > > the metadata to mirror what the kernel is doing
   > > > >
   > > > > - have the kernel pass the adjusted TRCCONFIGR back so perf
  can put it
   > > > > in the metadata
   > > > >
   > > > > - have the perf decoder get the thread id from whichever of
  VMID and
   > > > > CONTEXTID is available in a PE_CONTEXT element
   > > > >
   > > > > Obviously, the last is simplest, but it's a bodge, and
  means that
   > > > > OpenCSD will see VMIDs when its TRCCONFIGR says it won't.
  It's kind of
   > > > > cleanest to get the real TRCCONFIGR somehow, but how do we
  do that?
   > > >
   > > > We do get TRCCONFIGR in the perf records. We should simply
  make sure we get
   > > > the uptodate value (wherever we are getting it from).
   > >
   > > The copy in PERF_RECORD_AUXINFO (which is a synthetic record
  created
   > > by userspace perf) is, I believe, as I said:
   > >
   > > "userspace perf creates the metadata copy of TRCCONFIGR based on
   > > its request".
   > >
   > > So if the kernel modifies it based on information only the
  kernel knows,
   > > there's no current way to get the actual value. That was what I
  was trying
   > > to address with my suggestions.
   > >
   > > Have I missed some place the actual TRCCONFIGR is already being
  returned
   > > in other perf records?
   >
   > The sysfs is one place where we could expose this and this could
  be then
   > consumed by the perf tool while creating the TRCCONFIGR. Since the EL
   > can't change, this could be a onetime probe activity.

I'm joining this thread on the late and as such don't have all the context. But sysfs is never a good place to export run-time configuration registers as there could be multiple perf sessions active at the same time. This isn't a big issue for N:1 topologies but very real for 1:1.

I don't see how this affected by the sink topology, as this is an ETM configuration.

By sysfs, I mean, not necessarily under /sys/bus/coreight/.../etmN/

but, under :

/sys/bus/event_source/.../cs_etm/

We already expose various different information there. e.g, nr_addr_filters, sinks/

So we could expose something like :

trcconfigr/

and could describe how each of the individual event configs affect the TRCCONFIGR register. So that the perf tool can construct the TRCCONFIGR register from scratch just by looking at the trcconfigr directory.

e.g,

# cd /sys/bus/event_source/devices/cs_etm/ # for f in $(find format -type f); do echo $(basename $f) - $(cat $f); done cycacc - config:12 sinkid - config2:0-31 contextid - config:14 retstack - config:29 timestamp - config:28

$ for f in $(find trcconfigr -type f); do echo $(basename $f) - $(cat $f); done cycacc - 0x10 // Bit 4 timestamp - 0x800 // Bit 11 contextid - 0x8080 // Bit15 | Bit 7 On EL2 kernels - 0x40 // Bit 6 on EL1 kernels retstack - 0x1000 // Bit12

So those are configurations that apply to specific trace sessions. When dealing with concurrent trace sessions there is no way to guarantee the information exported to sysfs applies to the trace session that would read the information.

I think there is a bit of confusion here; The idea is the perf tool will construct the value of the traceconfigr based on the "options" selected by the user, just like it does now for "config1"/"config2"

i.e,

perf record -e cs_etm/cycacc,timestamp/ blah-blah

perf tool will :

map cycacc -> config1 |= BIT(12) and trcconfigr |= BIT($cat /sys/.../trcconfigr/cycacc)

and similarly for all the options. The values under the "trcconfigr" are static and only maps a given attribute to the traceconfig register bit.

Thus, any session can create the TRCCONFIGR accurately based on the events now, by dynamically looking up under the sysfs, rather than "compile time" static assumption of the the ETM model.

So how does this work when I copy the captured data off my target and try to decode offline?

Just like it works now. There is no change in flow for the perf tool. The only difference, how the perf tool would emit the TRCCONFIGR values in the perf.data samples. Instead of assuming that perf_event.attr.config1.CID => TRCCONFIGR[bit6], use the sysfs exported values to figure out the TRCCONFIGR setting.

Suzuki

On 09/16/2020 10:58 AM, Al Grant wrote:

...

Option 1: Trace Config exposed matching the format name:

$ pwd /sys/bus/event_source/devices/cs_etm

$ ls trace_config/ contextid cycacc retstack timestamp cs_etm# for f in $(find trace_config -type f) do echo $(basename $f) : $(cat $f) done

timestamp : 0x800 contextid : 0x40 cycacc : 0x10 retstack : 0x1000

Disadvantages : It is really tricky for the perf tool figure out the name of the "config" option from the event->attr.config bits. As the parsing of the format is not exported elsewhere (or I didn't look deep enough)

I don't like this. TRCCONFIG is a dynamic register whose value at any given time is specific to a perf session. Having it exposed in sysfs means that sysfs

Right, we are not exposing the TRCCONFIG for the *current session* (irrespective of what is running). This is a way for the perf tool to compute the TRCCONFIGR for a given event. The tool will consult the exposed files to construct the TRCCONFIGR (see the code below for e.g,). None of the values exposed are complete TRCCONFIGR values anyways.

now exposes data about any number of perf sessions, just so that each instance of "perf record" can read data about its own session - and hopefully read the right data at the right time. It might work but it's not very clean.

...

Option 2: Trace Config exposed as the "bit" positions.

$ ls trace_config/ 12 14 28 29

$ for f in $(find . -type f) do echo $(basename $f) : $(cat $f) done

28 : 0x800 14 : 0x40 12 : 0x10

29 : 0x1000

Advantage: The perf tool can iterate over the event->attr.config bits and do something like:

trconfigr = 0

for_each_set_bit(i, event->attr.config) trconfigr |= sysfs_read_trace_config(i)

where:

sysfs_read_trace_config(int i) { char path[MAX_LEN]; FILE *fp; unsigned long trcfg = 0;

sprintf(path, "%s/%d", $TRACE_CONFIG_PATH, i); fp = fopen(path, "r"); if (fp) fscanf(fp, "%lx", &trcfg); return trcfg; }

That is how this is going to be consumed. Again, the values exposed are not the ETM current configuration.

Suzuki

On Wed, Sep 16, 2020 at 09:58:52AM +0000, Al Grant wrote:

[...]

...

Thoughts ?

Another solution would be a specific immutable sysfs entry for the ETM, for the sole purpose of saying "I'm running at EL2 and will put the process id in VMID". Then "perf record" can read that. The question is then how it should put that in perf.data. It could either use TRCCONFIGR (which it's already faking) or TRCIDR2 as you suggest, but a more direct solution would be to use one of the spare header flags in PERF_RECORD_AUXDATA.

How about add a new node (e.g. "kern_el") under the sysfs?

/sys/bus/coresight/devices/etmX/mgmt/kern_el

perf tool can add a new entry in its metadata for "kern_el" when record, and when decode the CoreSight trace data, we can firstly get to know the kernel exception level from the metadata, and then based on this info to decide PID value from "context.context_id" or "context.vmid".

This approach seems to me is more directive than other options.

Would it be possible to reserve a few flags in the capabilities area of the main mmap buffer, to indicate "flags that tell you about the AUX data"? That would be another way for the kernel to tell userspace what it's doing.

I think Al suggests to change the data structure perf_event_mmap_page like below:

struct perf_event_mmap_page {

union { __u64 capabilities; struct { __u64 cap_bit0 : 1, /* Always 0, deprecated, see commit 860f085b74e9 */ cap_bit0_is_deprecated : 1, /* Always 1, signals that bit 0 is zero */

cap_user_rdpmc : 1, /* The RDPMC instruction can be used to read counts */ cap_user_time : 1, /* The time_{shift,mult,offset} fields are used */ cap_user_time_zero : 1, /* The time_zero field is used */ cap_user_time_short : 1, /* the time_{cycle,mask} fields are used */

cap_vhe : 1, /* System supports VHE or not */ cap_____res : 57; }; };

IMHO, I have concern for this since the capability 'cap_vhe' is specific for Arm64 arch and it cannot be commonly applied on other archs.

Thanks, Leo

On Fri, Sep 18, 2020 at 10:29:01PM +0000, Al Grant wrote:

[...]

...

...

Would it be possible to reserve a few flags in the capabilities area of the main mmap buffer, to indicate "flags that tell you about the AUX data"? That would be another way for the kernel to tell userspace what

it's doing.

I think Al suggests to change the data structure perf_event_mmap_page like below:

struct perf_event_mmap_page {

union {
    __u64   capabilities;
    struct {
            __u64   cap_bit0                : 1, /* Always 0, deprecated, see commit

860f085b74e9 */ cap_bit0_is_deprecated : 1, /* Always 1, signals that bit 0 is zero */

                    cap_user_rdpmc          : 1, /* The RDPMC instruction can be used to

read counts */ cap_user_time : 1, /* The time_{shift,mult,offset} fields are used */ cap_user_time_zero : 1, /* The time_zero field is used */ cap_user_time_short : 1, /* the time_{cycle,mask} fields are used */

                    cap_vhe                 : 1, /* System supports VHE or not */
                    cap_____res             : 57;
    };
};

IMHO, I have concern for this since the capability 'cap_vhe' is specific for Arm64 arch and it cannot be commonly applied on other archs.

I was thinking more that the mmap header would have something like

cap_eventflag1:1 cap_eventflag2:1

allocated for event-specific flags. Then those flags can be used for meanings specific to event. The same goes for flags in the perf record headers - if a handful of flags are reserved for event-specific use, their meanings can be different for different event types.

This approach might be difficult to extend, especially if one perf session enabls multiple events (and it's very flexiable for event numbers), so it leads to unlimited event flags.

Another potential issue is how to sent these event flags in the kernel. One way is to use a central place to set these event flags, like in the function arch_perf_update_userpage(), but these means this function needs to know every events ID and their associated event flags; another way is to spread event flag setting in individual PMU drivers, this brings a requirement for program flow: the event flag setting in PMU driver must be prior to reading mmap page in the perf tool.

Thanks Leo