[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-ID: <5a1e0243-263d-455a-83cd-c274d7d8befb@axentia.se>
Date: Thu, 2 Sep 2021 23:52:21 +0200
From: Peter Rosin <peda@...ntia.se>
To: Liam Beguin <liambeguin@...il.com>
Cc: Jonathan Cameron <jic23@...nel.org>, lars@...afoo.de,
linux-kernel@...r.kernel.org, linux-iio@...r.kernel.org,
devicetree@...r.kernel.org, robh+dt@...nel.org
Subject: Re: [PATCH v8 09/14] iio: afe: rescale: fix accuracy for small
fractional scales
On 2021-09-02 04:27, Liam Beguin wrote:
> On Mon, Aug 30, 2021 at 04:30:54PM +0200, Peter Rosin wrote:
>>
>> Having said that, if 'scale' and 'offset' indeed are static, then the gcd
>> cost can be mitigated by caching the result. Exact rescaling is always
>> nice...
>>
>> If IIO_VAL_INT overflows while rescaling, we are SOL whichever way we turn,
>> so ignore doing anything about that.
>
> I was thinking of using check_mul_overflow() to do something about the
> overflow, but I'm happy to leave it out for now.
My mistake, you are right. A sufficiently large denominator can of course
be used to dodge overflow in the numerator by sacrificing some accuracy
even if our maximum scale is still limited to 32 bits.
I apparently didn't have my brains about when I wrote the above...
Cheers,
Peter
Powered by blists - more mailing lists