| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening PHC | |
|
Open Source and information security mailing list archives
| ||
|
Date: Tue, 03 Aug 2021 11:09:17 +0530
From: Sai Prakash Ranjan <saiprakash.ranjan@...eaurora.org>
To: Will Deacon <will@...nel.org>
Cc: Robin Murphy <robin.murphy@....com>,
Joerg Roedel <joro@...tes.org>,
iommu@...ts.linux-foundation.org,
linux-arm-kernel@...ts.infradead.org, linux-kernel@...r.kernel.org,
linux-arm-msm@...r.kernel.org,
Doug Anderson <dianders@...omium.org>,
Krishna Reddy <vdumpa@...dia.com>,
Thierry Reding <treding@...dia.com>,
Tomasz Figa <tfiga@...omium.org>,
Bjorn Andersson <bjorn.andersson@...aro.org>
Subject: Re: [PATCHv3] iommu/arm-smmu: Optimize ->tlb_flush_walk() for qcom
implementation
On 2021-08-02 21:13, Will Deacon wrote:
> On Wed, Jun 23, 2021 at 07:12:01PM +0530, Sai Prakash Ranjan wrote:
>> Currently for iommu_unmap() of large scatter-gather list with page
>> size
>> elements, the majority of time is spent in flushing of partial walks
>> in
>> __arm_lpae_unmap() which is a VA based TLB invalidation invalidating
>> page-by-page on iommus like arm-smmu-v2 (TLBIVA).
>>
>> For example: to unmap a 32MB scatter-gather list with page size
>> elements
>> (8192 entries), there are 16->2MB buffer unmaps based on the pgsize
>> (2MB
>> for 4K granule) and each of 2MB will further result in 512 TLBIVAs
>> (2MB/4K)
>> resulting in a total of 8192 TLBIVAs (512*16) for 16->2MB causing a
>> huge
>> overhead.
>>
>> On qcom implementation, there are several performance improvements for
>> TLB cache invalidations in HW like wait-for-safe (for realtime clients
>> such as camera and display) and few others to allow for cache
>> lookups/updates when TLBI is in progress for the same context bank.
>> So the cost of over-invalidation is less compared to the unmap latency
>> on several usecases like camera which deals with large buffers. So,
>> ASID based TLB invalidations (TLBIASID) can be used to invalidate the
>> entire context for partial walk flush thereby improving the unmap
>> latency.
>>
>> Non-strict mode can use this by default for all platforms given its
>> all about over-invalidation saving time on individual unmaps and
>> non-deterministic generally.
>>
>> For this example of 32MB scatter-gather list unmap, this change
>> results
>> in just 16 ASID based TLB invalidations (TLBIASIDs) as opposed to 8192
>> TLBIVAs thereby increasing the performance of unmaps drastically.
>>
>> Test on QTI SM8150 SoC for 10 iterations of iommu_{map_sg}/unmap:
>> (average over 10 iterations)
>>
>> Before this optimization:
>>
>> size iommu_map_sg iommu_unmap
>> 4K 2.067 us 1.854 us
>> 64K 9.598 us 8.802 us
>> 1M 148.890 us 130.718 us
>> 2M 305.864 us 67.291 us
>> 12M 1793.604 us 390.838 us
>> 16M 2386.848 us 518.187 us
>> 24M 3563.296 us 775.989 us
>> 32M 4747.171 us 1033.364 us
>>
>> After this optimization:
>>
>> size iommu_map_sg iommu_unmap
>> 4K 1.723 us 1.765 us
>> 64K 9.880 us 8.869 us
>> 1M 155.364 us 135.223 us
>> 2M 303.906 us 5.385 us
>> 12M 1786.557 us 21.250 us
>> 16M 2391.890 us 27.437 us
>> 24M 3570.895 us 39.937 us
>> 32M 4755.234 us 51.797 us
>>
>> This is further reduced once the map/unmap_pages() support gets in
>> which
>> will result in just 1 TLBIASID as compared to 16 TLBIASIDs.
>>
>> Real world data also shows big difference in unmap performance as
>> below:
>>
>> There were reports of camera frame drops because of high overhead in
>> iommu unmap without this optimization because of frequent unmaps
>> issued
>> by camera of about 100MB/s taking more than 100ms thereby causing
>> frame
>> drops.
>>
>> Signed-off-by: Sai Prakash Ranjan <saiprakash.ranjan@...eaurora.org>
>> ---
>>
>> Changes in v3:
>> * Move the logic to arm-smmu driver from io-pgtable (Robin)
>> * Use a new set of iommu_flush_ops->arm_smmu_s1_tlb_impl_ops and use
>> it for qcom impl
>>
>> Changes in v2:
>> * Add a quirk to choose tlb_flush_all in partial walk flush
>> * Set the quirk for QTI SoC implementation
>>
>> ---
>> drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c | 13 +++++++++++++
>> drivers/iommu/arm/arm-smmu/arm-smmu.c | 17 ++++++++++++++++-
>> 2 files changed, 29 insertions(+), 1 deletion(-)
>>
>> diff --git a/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c
>> b/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c
>> index 7771d40176de..218c71465819 100644
>> --- a/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c
>> +++ b/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c
>> @@ -10,6 +10,8 @@
>>
>> #include "arm-smmu.h"
>>
>> +extern const struct iommu_flush_ops arm_smmu_s1_tlb_impl_ops;
>> +
>> struct qcom_smmu {
>> struct arm_smmu_device smmu;
>> bool bypass_quirk;
>> @@ -146,6 +148,8 @@ static int qcom_adreno_smmu_init_context(struct
>> arm_smmu_domain *smmu_domain,
>> {
>> struct adreno_smmu_priv *priv;
>>
>> + pgtbl_cfg->tlb = &arm_smmu_s1_tlb_impl_ops;
>> +
>> /* Only enable split pagetables for the GPU device (SID 0) */
>> if (!qcom_adreno_smmu_is_gpu_device(dev))
>> return 0;
>> @@ -185,6 +189,14 @@ static const struct of_device_id
>> qcom_smmu_client_of_match[] __maybe_unused = {
>> { }
>> };
>>
>> +static int qcom_smmu_init_context(struct arm_smmu_domain
>> *smmu_domain,
>> + struct io_pgtable_cfg *pgtbl_cfg, struct device *dev)
>> +{
>> + pgtbl_cfg->tlb = &arm_smmu_s1_tlb_impl_ops;
>> +
>> + return 0;
>> +}
>> +
>> static int qcom_smmu_cfg_probe(struct arm_smmu_device *smmu)
>> {
>> unsigned int last_s2cr = ARM_SMMU_GR0_S2CR(smmu->num_mapping_groups
>> - 1);
>> @@ -308,6 +320,7 @@ static int qcom_smmu500_reset(struct
>> arm_smmu_device *smmu)
>> }
>>
>> static const struct arm_smmu_impl qcom_smmu_impl = {
>> + .init_context = qcom_smmu_init_context,
>> .cfg_probe = qcom_smmu_cfg_probe,
>> .def_domain_type = qcom_smmu_def_domain_type,
>> .reset = qcom_smmu500_reset,
>> diff --git a/drivers/iommu/arm/arm-smmu/arm-smmu.c
>> b/drivers/iommu/arm/arm-smmu/arm-smmu.c
>> index d3c6f54110a5..f3845e822565 100644
>> --- a/drivers/iommu/arm/arm-smmu/arm-smmu.c
>> +++ b/drivers/iommu/arm/arm-smmu/arm-smmu.c
>> @@ -341,6 +341,12 @@ static void arm_smmu_tlb_add_page_s1(struct
>> iommu_iotlb_gather *gather,
>> ARM_SMMU_CB_S1_TLBIVAL);
>> }
>>
>> +static void arm_smmu_tlb_inv_walk_impl_s1(unsigned long iova, size_t
>> size,
>> + size_t granule, void *cookie)
>> +{
>> + arm_smmu_tlb_inv_context_s1(cookie);
>> +}
>> +
>> static void arm_smmu_tlb_inv_walk_s2(unsigned long iova, size_t size,
>> size_t granule, void *cookie)
>> {
>> @@ -388,6 +394,12 @@ static const struct iommu_flush_ops
>> arm_smmu_s1_tlb_ops = {
>> .tlb_add_page = arm_smmu_tlb_add_page_s1,
>> };
>>
>> +const struct iommu_flush_ops arm_smmu_s1_tlb_impl_ops = {
>> + .tlb_flush_all = arm_smmu_tlb_inv_context_s1,
>> + .tlb_flush_walk = arm_smmu_tlb_inv_walk_impl_s1,
>> + .tlb_add_page = arm_smmu_tlb_add_page_s1,
>> +};
>
> Hmm, dunno about this. Wouldn't it be a lot cleaner if the
> tlb_flush_walk
> callbacks just did the right thing based on the smmu_domain (maybe in
> the
> arm_smmu_cfg?) rather than having an entirely new set of ops just
> because
> they're const and you can't overide the bit you want?
>
> I don't think there's really an awful lot qcom-specific about the
> principle
> here -- there's a trade-off between over-invalidation and invalidation
> latency. That happens on the CPU as well.
>
Sorry didn't understand, based on smmu_domain what? How do we make
this implementation specific? Do you mean something like a quirk?
The reason we didn't make this common was because nvidia folks weren't
so happy with that, you can find the discussion in this thread [1].
[1]
https://lore.kernel.org/lkml/20210609145315.25750-1-saiprakash.ranjan@codeaurora.org/
Thanks,
Sai
--
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a
member
of Code Aurora Forum, hosted by The Linux Foundation
Powered by blists - more mailing lists