linux-kernel - Re: [PATCH 2/3] media: vicodec: avoic clang frame size warning

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Message-ID: <eaf45207-2741-2e0d-4ae4-aabb36e4417d@xs4all.nl>
Date:   Tue, 19 Feb 2019 20:56:25 +0100
From:   Hans Verkuil <hverkuil@...all.nl>
To:     Arnd Bergmann <arnd@...db.de>,
        Hans Verkuil <hans.verkuil@...co.com>,
        Mauro Carvalho Chehab <mchehab@...nel.org>
Cc:     Nick Desaulniers <ndesaulniers@...gle.com>,
        Mark Brown <broonie@...nel.org>,
        Nathan Chancellor <natechancellor@...il.com>,
        Dafna Hirschfeld <dafna3@...il.com>,
        Tom aan de Wiel <tom.aandewiel@...il.com>,
        linux-media@...r.kernel.org, linux-kernel@...r.kernel.org
Subject: Re: [PATCH 2/3] media: vicodec: avoic clang frame size warning

On 2/19/19 6:01 PM, Arnd Bergmann wrote:
> Clang-9 makes some different inlining decisions compared to gcc, which
> leads to a warning about a possible stack overflow problem when building
> with CONFIG_KASAN, including when setting asan-stack=0, which avoids
> most other frame overflow warnings:
> 
> drivers/media/platform/vicodec/codec-fwht.c:673:12: error: stack frame size of 2224 bytes in function 'encode_plane'
> 
> Manually adding noinline_for_stack annotations in those functions
> called by encode_plane() or decode_plane() that require a significant
> amount of kernel stack makes this impossible to happen with any
> compiler.
> 
> Signed-off-by: Arnd Bergmann <arnd@...db.de>
> ---
>  drivers/media/platform/vicodec/codec-fwht.c | 18 ++++++++++--------
>  1 file changed, 10 insertions(+), 8 deletions(-)
> 
> diff --git a/drivers/media/platform/vicodec/codec-fwht.c b/drivers/media/platform/vicodec/codec-fwht.c
> index d1d6085da9f1..135d56bcc2c5 100644
> --- a/drivers/media/platform/vicodec/codec-fwht.c
> +++ b/drivers/media/platform/vicodec/codec-fwht.c
> @@ -47,7 +47,7 @@ static const uint8_t zigzag[64] = {
>  };
>  
>  
> -static int rlc(const s16 *in, __be16 *output, int blocktype)
> +static int noinline_for_stack rlc(const s16 *in, __be16 *output, int blocktype)
>  {
>  	s16 block[8 * 8];
>  	s16 *wp = block;
> @@ -106,8 +106,8 @@ static int rlc(const s16 *in, __be16 *output, int blocktype)
>   * This function will worst-case increase rlc_in by 65*2 bytes:
>   * one s16 value for the header and 8 * 8 coefficients of type s16.
>   */
> -static u16 derlc(const __be16 **rlc_in, s16 *dwht_out,
> -		 const __be16 *end_of_input)
> +static noinline_for_stack u16
> +derlc(const __be16 **rlc_in, s16 *dwht_out, const __be16 *end_of_input)
>  {
>  	/* header */
>  	const __be16 *input = *rlc_in;
> @@ -373,7 +373,8 @@ static void fwht(const u8 *block, s16 *output_block, unsigned int stride,
>   * Furthermore values can be negative... This is just a version that
>   * works with 16 signed data
>   */
> -static void fwht16(const s16 *block, s16 *output_block, int stride, int intra)
> +static void noinline_for_stack
> +fwht16(const s16 *block, s16 *output_block, int stride, int intra)
>  {
>  	/* we'll need more than 8 bits for the transformed coefficients */
>  	s32 workspace1[8], workspace2[8];
> @@ -456,7 +457,8 @@ static void fwht16(const s16 *block, s16 *output_block, int stride, int intra)
>  	}
>  }
>  
> -static void ifwht(const s16 *block, s16 *output_block, int intra)
> +static noinline_for_stack void
> +ifwht(const s16 *block, s16 *output_block, int intra)
>  {

Please add it for fwht as well. It makes no sense to have it for fwht16, ifwht
but not the fwht function.

Got to say this is all very magic...

I think it would be good to perhaps have a comment at the start of the source
that explains why noinline_for_stack is added to selected functions.

Patches 1 & 3 are fine, BTW.

Regards,

	Hans

>  	/*
>  	 * we'll need more than 8 bits for the transformed coefficients
> @@ -604,9 +606,9 @@ static int var_inter(const s16 *old, const s16 *new)
>  	return ret;
>  }
>  
> -static int decide_blocktype(const u8 *cur, const u8 *reference,
> -			    s16 *deltablock, unsigned int stride,
> -			    unsigned int input_step)
> +static noinline_for_stack int
> +decide_blocktype(const u8 *cur, const u8 *reference, s16 *deltablock,
> +		 unsigned int stride, unsigned int input_step)
>  {
>  	s16 tmp[64];
>  	s16 old[64];
>