Message-ID: <f9fddb60-6976-6d99-fcc2-8011d2fadeb7@arm.com>
Date:   Wed, 30 Aug 2023 17:04:57 +0100
From:   Suzuki K Poulose <suzuki.poulose@....com>
To:     scclevenger@...amperecomputing.com,
        Anshuman Khandual <anshuman.khandual@....com>,
        linux-arm-kernel@...ts.infradead.org
Cc:     Mike Leach <mike.leach@...aro.org>,
        James Clark <james.clark@....com>,
        Leo Yan <leo.yan@...aro.org>,
        Alexander Shishkin <alexander.shishkin@...ux.intel.com>,
        coresight@...ts.linaro.org, linux-kernel@...r.kernel.org,
        Sudeep Holla <sudeep.holla@....com>
Subject: Re: [PATCH V2] coresight: tmc: Make etr buffer mode user configurable
 from sysfs

Cc: Sudeep

Hi Steve

On 28/08/2023 17:35, Steve Clevenger wrote:
> 
> Hi Suzuki,
> 
> On 8/27/2023 2:35 PM, Suzuki K Poulose wrote:
>> Hi Steve
>>
>> On 26/08/2023 01:14, Steve Clevenger wrote:
>>>
>>> Unfortunately, I tested with the original patch not [PATCH V2]. I've
>>> remedied this. My results below:
>>>
>>> [root@...01sys-b212 linux]# cat
>>> /sys/devices/system/cpu/cpu123/ARMHC501\:23/tmc_etr35/buf_modes_available
>>> auto flat catu
>>> [root@...01sys-b212 linux]# cat
>>> /sys/devices/system/cpu/cpu123/ARMHC501\:23/tmc_etr35/buf_mode_preferred
>>> auto
>>> [root@...01sys-b212 linux]# echo "catu" >
>>> /sys/devices/system/cpu/cpu123/ARMHC501\:23/tmc_etr35/buf_mode_preferred
>>> [root@...01sys-b212 linux]# cat
>>> /sys/devices/system/cpu/cpu123/ARMHC501\:23/tmc_etr35/buf_mode_preferred
>>> catu
>>>
>>> As with the V1 patch, auto defaults to catu.
>>>
>>> I expected to see tmc-sg (former default) as an available mode, but do
>>> not. As I recall, the buffer mode defaulted to ETR scatter-gather prior
>>> to this patch. Must this capability now be explicitly advertised? I've
>>> seen this done as "arm,scatter-gather" in device trees, but not used by
>>> Ampere. Perhaps someone can enlighten me.
>>
>> Yes, you must add that property to the TMC-ETR node (for both DT and
>> ACPI). In the past, almost all of the TMC-ETRs (except Juno board)
>> locked up the system while using the SG mode (due to the interconnect
>> issues, something to do with the transaction). Thus, we decided to
>> add a property explicitly enabling this for a given platform.
>>
>> When you mentioned, it was using TMC-ETR SG mode, how did you verify
>> this ? Please be aware that the table allocation code etc are shared
>> by both TMC-SG and CATU.
>>
> 
> You might recall how this started. I had no way to test the CATU due to
> the order the ETR modes defaulted (Flat, ETR-SG, CATU). For test
> purposes, I programmatically swapped the ETR-SG/CATU order and could
> then verify CATU operation by the driver calling into CATU code. This

So, were you using the DT based boot for the above runs ?

> suggests Flat mode was bypassed, and the driver defaulted to ETR-SG
> prior to this hack. This didn't offer the user any control, hence my
> feature request. Note that most of the early Ampere self-hosted trace
> collection used ETR-SG. Now I can't select it.


> 
> How is this property described in the ACPI? The "ACPI for CoreSight™ 1.1
> Platform Design Document" (DEN0067) doesn't describe this.

This is not specified in the ACPI platform design document. I can get
it fixed. Ideally we need a property describing that the scatter-gather
mode is safe to use.

DT uses "arm,scatter-gather" property [0] and this is what we now expect
in the ACPI based systems too.

https://elixir.bootlin.com/linux/latest/source/Documentation/devicetree/bindings/arm/arm,coresight-tmc.yaml#:~:text=71-,72,-73

Does it sound fine ?

Suzuki

> 
> Thanks,
> Steve
> 
> 
>> Kind regards
>> Suzuki
>>
>>>
>>> Steve C.
>>>
>>> On 8/23/2023 4:10 PM, Steve Clevenger wrote:
>>>>
>>>> Here's some quick feedback. My system shows two modes available; auto
>>>> catu
>>>>
>>>> etr_buf_mode_current is writable. I expected to see tmc-sg (former
>>>> default) listed in etr_buf_modes_available but it doesn't show up.
>>>>
>>>> Note that both the auto and catu etr_buf_mode_current settings default
>>>> to catu. My understanding is auto should revert to the default behavior.
>>>> On my system the default was tmc-sg.
>>>>
>>>> More later.
>>>>
>>>> [root@...01sys-b212 kernel]# cat
>>>> /sys/devices/system/cpu/cpu20/ARMHC501\:60/tmc_etr96/etr_buf_modes_available
>>>>
>>>> auto catu
>>>> [root@...01sys-b212 kernel]# cat
>>>> /sys/devices/system/cpu/cpu20/ARMHC501\:60/tmc_etr96/etr_buf_mode_current
>>>> catu
>>>> [root@...01sys-b212 kernel]# echo "catu" >
>>>> /sys/devices/system/cpu/cpu20/ARMHC501\:60/tmc_etr96/etr_buf_mode_current
>>>> [root@...01sys-b212 kernel]# cat
>>>> /sys/devices/system/cpu/cpu20/ARMHC501\:60/tmc_etr96/etr_buf_mode_current
>>>> catu
>>>>
>>>> Steve C.
>>>>
>>>>
>>>> On 8/21/2023 12:40 PM, Steve Clevenger wrote:
>>>>>
>>>>> Hi Suzuki,
>>>>>
>>>>> I may be able to test it this week. You've already pointed me at the
>>>>> patch thread(s). The main holdup is I need to merge the 6.6 pending
>>>>> platform work in order to use the Ampere ACPI. I couldn't get these
>>>>> patches to apply directly to 6.4 last I tried.
>>>>>
>>>>> Steve C.
>>>>>
>>>>> On 8/18/2023 2:39 AM, Suzuki K Poulose wrote:
>>>>>> Cc: Steve
>>>>>>
>>>>>> Steve,
>>>>>>
>>>>>> Are you able to test this with CATU ?
>>>>>>
>>>>>>
>>>>>> On 18/08/2023 09:21, Anshuman Khandual wrote:
>>>>>>> Currently TMC-ETR automatically selects the buffer mode from all
>>>>>>> available
>>>>>>> methods in the following sequentially fallback manner - also in that
>>>>>>> order.
>>>>>>>
>>>>>>> 1. FLAT mode with or without IOMMU
>>>>>>> 2. TMC-ETR-SG (scatter gather) mode when available
>>>>>>> 3. CATU mode when available
>>>>>>>
>>>>>>> But this order might not be ideal for all situations. For example if
>>>>>>> there
>>>>>>> is a CATU connected to ETR, it may be better to use TMC-ETR scatter
>>>>>>> gather
>>>>>>> method, rather than CATU. But hard coding such order changes will
>>>>>>> prevent
>>>>>>> us from testing or using a particular mode. This change provides
>>>>>>> following
>>>>>>> new sysfs tunables for the user to control TMC-ETR buffer mode
>>>>>>> explicitly,
>>>>>>> if required. This adds following new sysfs files for buffer mode
>>>>>>> selection
>>>>>>> purpose explicitly in the user space.
>>>>>>>
>>>>>>> /sys/bus/coresight/devices/tmc_etr<N>/buf_modes_available
>>>>>>> /sys/bus/coresight/devices/tmc_etr<N>/buf_mode_preferred
>>>>>>>
>>>>>>> $ cat buf_modes_available
>>>>>>> auto flat tmc-sg catu    ------------------> Supported TMC-ETR buffer
>>>>>>> modes
>>>>>>>
>>>>>>> $ echo catu > buf_mode_preferred   -------> Explicit buffer mode
>>>>>>> request
>>>>>>>
>>>>>>> But explicit user request has to be within supported ETR buffer modes
>>>>>>> only.
>>>>>>> These sysfs interface files are exclussive to ETR, and hence these
>>>>>>> are
>>>>>>> not
>>>>>>> available for other TMC devices such as ETB or ETF etc.
>>>>>>>
>>>>>>> A new auto' mode (i.e ETR_MODE_AUTO) has been added to help fallback
>>>>>>> to the
>>>>>>> existing default behaviour, when user provided preferred buffer mode
>>>>>>> fails.
>>>>>>> ETR_MODE_FLAT and ETR_MODE_AUTO are always available as preferred
>>>>>>> modes.
>>>>>>>
>>>>>>> Cc: Suzuki K Poulose <suzuki.poulose@....com>
>>>>>>> Cc: Mike Leach <mike.leach@...aro.org>
>>>>>>> Cc: James Clark <james.clark@....com>
>>>>>>> Cc: Leo Yan <leo.yan@...aro.org>
>>>>>>> Cc: Alexander Shishkin <alexander.shishkin@...ux.intel.com>
>>>>>>> Cc: coresight@...ts.linaro.org
>>>>>>> Cc: linux-arm-kernel@...ts.infradead.org
>>>>>>> Cc: linux-kernel@...r.kernel.org
>>>>>>> Signed-off-by: Anshuman Khandual <anshuman.khandual@....com>
>>>>>>> ---
>>>>>>> This applies on v6.5-rc6
>>>>>>>
>>>>>>> Changes in V2:
>>>>>>>
>>>>>>> - Renamed sysfs file etr_buf_modes_available as buf_modes_available
>>>>>>> - Renamed sysfs file buf_mode_current as buf_mode_preferred
>>>>>>> - Renamed etr_supports_flat_mode() as etr_can_use_flat_mode()
>>>>>>> - Renamed coresight_tmc_groups[] as coresight_etf_groups[]
>>>>>>> - Reused coresight_tmc_group[] for trigger_cntr and buffer_size
>>>>>>> - Fallback trying ETR_MODE_AUTO when user preferred mode fails
>>>>>>> - Moved ETR sysfs details into coresight-tmc-etr.c
>>>>>>> - Dropped etr_can_use_flat_mode() check while offering ETR_MODE_FLAT
>>>>>>> in sysfs
>>>>>>> - Moved struct etr_buf_hw inside coresight-tmc-etr.c
>>>>>>> - Moved get_etr_buf_hw() and etr_can_use_flat_mode() inside
>>>>>>> coresight-tmc-etr.c
>>>>>>> - Updated month in
>>>>>>> Documentation/ABI/testing/sysfs-bus-coresight-devices-tmc
>>>>>>>
>>>>>>> Changes in V1:
>>>>>>>
>>>>>>> https://lore.kernel.org/all/20230728084837.276551-1-anshuman.khandual@arm.com/
>>>>>>>
>>>>>>>     .../testing/sysfs-bus-coresight-devices-tmc   |  16 +++
>>>>>>>     .../hwtracing/coresight/coresight-tmc-core.c  |  15 ++-
>>>>>>>     .../hwtracing/coresight/coresight-tmc-etr.c   | 111
>>>>>>> ++++++++++++++++--
>>>>>>>     drivers/hwtracing/coresight/coresight-tmc.h   |   3 +
>>>>>>>     4 files changed, 131 insertions(+), 14 deletions(-)
>>>>>>
>>>>>>
>>>>>> Looks good to me.
>>>>>>
>>>>>> Suzuki
>>>>>>
>>>>>>
>>

Powered by blists - more mailing lists