| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Message-ID: <20250804180029.11b8c310@sal.lan>
Date: Mon, 4 Aug 2025 18:00:29 +0200
From: Mauro Carvalho Chehab <mchehab+huawei@...nel.org>
To: Jonathan Corbet <corbet@....net>
Cc: linux-doc@...r.kernel.org, linux-kernel@...r.kernel.org, Akira Yokosawa
<akiyks@...il.com>
Subject: Re: [PATCH 03/12] docs: kdoc: backslashectomy in kdoc_parser
Em Mon, 4 Aug 2025 14:58:18 +0200
Mauro Carvalho Chehab <mchehab+huawei@...nel.org> escreveu:
> Em Fri, 01 Aug 2025 08:21:49 -0600
> Jonathan Corbet <corbet@....net> escreveu:
>
> > Mauro Carvalho Chehab <mchehab+huawei@...nel.org> writes:
> >
> > > Em Thu, 31 Jul 2025 18:13:17 -0600
> > > Jonathan Corbet <corbet@....net> escreveu:
> > >
> > >> A lot of the regular expressions in this file have extraneous backslashes
> > >
> > > This one is a bit scary... It could actually cause issues somewhere.
> >
> > What kind of issues?
>
> I caught several issues in the past due to the lack of it. Don't
> recall the specific cases, but using reserved symbols without
> backslashes have giving me enough headaches.
>
> Yet, see POSIX rules for some cases:
>
> https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap09.html#tag_09_03
>
> like this one:
>
> "The character sequences "[.", "[=", and "[:" shall be special
> inside a bracket expression"
>
> Basically, if you don't know exactly what you're doing, and just
> place special characters there without extra case, you may be
> in serious troubles. And see, this is just for BRE (basic regular
> expressions). There are also other weirdness with ERE (extended
> regular expressions):
>
> "The <period>, <left-square-bracket>, <backslash>, and
> <left-parenthesis> shall be special except when used
> in a bracket expression"
>
> > > Also, IMHO, some expressions look worse on my eyes ;-)
> >
> > Here I think we're going to disagree. The extra backslashes are really
> > just visual noise as far as I'm concerned.
> >
> > >> that may have been needed in Perl, but aren't helpful here. Take them out
> > >> to reduce slightly the visual noise.
> > >
> > > No idea if Perl actually requires, but, at least for me, I do prefer to
> > > see all special characters properly escaped with a backslash. This way,
> > > it is a lot clearer that what it is expecting is a string, instead of
> > > using something that may affect regex processing.
> >
> > I guess my point is that, in the given cases, the characters in question
> > *aren't* special.
>
> They are special in the sense that we're using characters that
> have meanings in regular expressions and even placing them on
> a random order may cause POSIX violations (and eventually cause
> troubles if, for instance, we need to use "regex" instead of "re",
> or if someone fixes python native "re" to be more POSIX compliant.
>
> > >> - param = KernRe(r'[\[\)].*').sub('', param, count=1)
> > >> + param = KernRe(r'[)[].*').sub('', param, count=1)
> > >
> > > This one, for instance, IMHO looks a lot worse for my eyes to understand
> > > that there is a "[" that it is not an operator, but instead a string.
> > > The open close parenthesis also looks weird. My regex-trained eyes think
> > > that this would be part of a capture group.
> >
> > ...and mine say "that's in [brackets] why are you escaping it?" :)
>
> Heh, all those years writing and reviewing kernel code, for me
> seeing unmatched parenthesis/brackets really bugs me... perhaps
> it starts some sort of TOC syndrome ;-)
>
> Perhaps one alternative would be to have a separate var, like:
>
> # Before touching this, see:
> # https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap09.html#tag_09_04
> # As some char sequences inside brackets have special meanings
> escape_chars = ")["
>
> param = KernRe(rf'[{escape_chars}].*').sub('', param, count=1)
>
> or to use re_escape().
>
> > >> if dtype == "" and param.endswith("..."):
> > >> if KernRe(r'\w\.\.\.$').search(param):
> > >> @@ -405,7 +405,7 @@ class KernelDoc:
> > >>
> > >> for arg in args.split(splitter):
> > >> # Strip comments
> > >> - arg = KernRe(r'\/\*.*\*\/').sub('', arg)
> > >> + arg = KernRe(r'/\*.*\*/').sub('', arg)
> > >
> > > A pattern like /..../ is a standard way to pass search group with Regex
> > > on many languages and utils that accept regular expressions like the
> > > sed command. Dropping the backslash here IMHO makes it confusing ;-)
> >
> > ...but it is definitely not any such in Python and never has been, so
> > escaping slashes looks weird and makes the reader wonder what they are
> > missing.
>
> After re-reading, this specific change is actually ok, but yeah, I
> still need to read it twice or three times, as on sed, perl and other
> languages that are more POSIX compliant, /re/ means a regex delimiter:
>
> https://en.wikipedia.org/wiki/Regular_expression
>
> > > Seriously, IMHO this patch makes a lot worse to understand what brackets,
> > > parenthesis and dots are strings, and which ones are part of the regex
> > > syntax.
> >
> > So I guess I won't fight this one to the death, but I really do
> > disagree. Writing regexes in a non-canonical style just makes it harder
> > for anybody else who comes along to figure out what is going on; it
> > certainly made it harder for me.
>
> Heh, for me, my main concerns are:
> - unmatched brackets/parenthesis
> - POSIX violations - it may work today, but future Python versions
> that fix "re" module will cause regressions. It is also annoying
> to write/understand regex that only works on Python.
In time: I mean *possible* POSIX violations.
I very much prefer more backslashs than needed or use re.escape()
than to read thoughtfully POSIX and Python-specific specific rules.
Python, in particular, is not very reliable between versions - each
new version comes with a set of incompatible changes.
Regex in Python for instance had incompatible changes in 3.6, 3.7, 3.11
and 3.13 (according with a LLM query I did). None relevant for our cases,
but there were incompatible changes on 3.6, 3.7 and 3.13 on common
patterns like \b (affecting re.split) and \w (affecting utf). The number
of escaped chars on re.escape() also increased on 3.7.
In summary, if you agree with always escape brackets, curly brackets and
parenthesis inside brackets on kernel-doc, we should be free of not
opened/not closed "symbols" with is an annoyance at least for me, and
we should be freed of possible POSIX issues and undefined behavior(*).
On such case, feel free to add my Reviewed-by.
Regards,
Mauro
(*) Python spec:
https://docs.python.org/3/library/re.html#regular-expression-syntax
Says:
"Backslash either escapes characters which have special meaning in
a set such as '-', ']', '^' and '\\' itself..."
Here, "such as" is vague: it doesn't say anything about "[[]" or "[\[]".
As not escaping "[" is a POSIX violation, I'd say that this is undefined
behavior that could change with time specially if they want to stick
with POSIX and/or implement POSIX ERE in the future.
---
Btw, I asked a LLM (*) to generate a list of special chars inside brackets,
listing how such characters are handled. Neither Deepseek nor Chatgpt
considered brackets without escape as valid or good practice.
That's the Deepseek version:
+------------------+-----------+-----------+-----------+-----------+-----------+-----------+
| | Python re | Perl | POSIX ERE | POSIX BRE | grep -E | sed (BRE) |
+==================+===========+===========+===========+===========+===========+===========+
| ] (not first) | \] | \] | \] | \] | \] | \] |
+------------------+-----------+-----------+-----------+-----------+-----------+-----------+
| ] (first) | ] | ] | ] | ] | ] | ] |
+------------------+-----------+-----------+-----------+-----------+-----------+-----------+
| [ | \[ | \[ | \[ | \[ | \[ | \[ |
+------------------+-----------+-----------+-----------+-----------+-----------+-----------+
| - (middle) | \-, a-z | \-, a-z | \-, a-z | \-, a-z | \-, a-z | \- |
+------------------+-----------+-----------+-----------+-----------+-----------+-----------+
| - (first/last) | - | - | - | - | - | - |
+------------------+-----------+-----------+-----------+-----------+-----------+-----------+
| ^ (first) | ^ | ^ | ^ | ^ | ^ | \^ |
+------------------+-----------+-----------+-----------+-----------+-----------+-----------+
| ^ (not first) | ^ | ^ | \^ | \^ | \^ | \^ |
+------------------+-----------+-----------+-----------+-----------+-----------+-----------+
| \ | \\ | \\ | \\ | \\ | \\ | \\ |
+------------------+-----------+-----------+-----------+-----------+-----------+-----------+
| POSIX classes | [:alpha:] | [:alpha:] | [:alpha:] | [:alpha:] | [:alpha:] | [:alpha:] |
| (e.g. [:alpha:]) | | | | | | |
+------------------+-----------+-----------+-----------+-----------+-----------+-----------+
| Special POSIX | - | - | [.ch.] | [.ch.] | - | [.ch.] |
| collating [.ch.] | | | [=a=] | [=a=] | | [=a=] |
+------------------+-----------+-----------+-----------+-----------+-----------+-----------+
| * | * | * | * | \* | * | \* |
+------------------+-----------+-----------+-----------+-----------+-----------+-----------+
| $ | $ | $ | $ | \$ | $ | \$ |
+------------------+-----------+-----------+-----------+-----------+-----------+-----------+
| . | . | . | . | \. | . | \. |
+------------------+-----------+-----------+-----------+-----------+-----------+-----------+
And using the LLM query produced by Deepseek, (see below), that's Chatgpt results:
+------------------------------------+------------+--------+-------------+-------------+-----------+-------------+
| Feature | Python re | Perl | POSIX ERE | POSIX BRE | grep -E | sed (BRE) |
+====================================+============+========+=============+=============+===========+=============+
| `]` (first position) | literal | literal| literal | literal | literal | literal |
+------------------------------------+------------+--------+-------------+-------------+-----------+-------------+
| `]` (non-first position) | \] | \] | \] | \] | \] | \] |
+------------------------------------+------------+--------+-------------+-------------+-----------+-------------+
| `[` | literal | literal| literal | literal | literal | literal |
+------------------------------------+------------+--------+-------------+-------------+-----------+-------------+
| `-` (between literals) | range `a-z`| range | range | range | range | range |
+------------------------------------+------------+--------+-------------+-------------+-----------+-------------+
| `-` (start or end) | literal | literal| literal | literal | literal | literal |
+------------------------------------+------------+--------+-------------+-------------+-----------+-------------+
| `^` (first position) | negation | negation| negation | negation | negation | negation |
+------------------------------------+------------+--------+-------------+-------------+-----------+-------------+
| `^` (non-first position) | literal | literal| literal | literal | literal | literal |
+------------------------------------+------------+--------+-------------+-------------+-----------+-------------+
| `\` | \\ | \\ | \\ | \\ | \\ | \\ |
+------------------------------------+------------+--------+-------------+-------------+-----------+-------------+
| POSIX class `[:alpha:]` | literal | literal| supported | supported | supported | supported |
+------------------------------------+------------+--------+-------------+-------------+-----------+-------------+
| POSIX special `[.ch.]` | - | - | supported | supported | supported | supported |
+------------------------------------+------------+--------+-------------+-------------+-----------+-------------+
| POSIX equivalence `[=a=]` | - | - | supported | supported | supported | supported |
+------------------------------------+------------+--------+-------------+-------------+-----------+-------------+
| `*` | literal | literal| literal | literal | literal | literal |
+------------------------------------+------------+--------+-------------+-------------+-----------+-------------+
| `$` | literal | literal| literal | literal | literal | literal |
+------------------------------------+------------+--------+-------------+-------------+-----------+-------------+
| `.` | literal | literal| literal | literal | literal | literal |
+------------------------------------+------------+--------+-------------+-------------+-----------+-------------+
Yet, Chatgpt original result was wrong. I had to do an extra query
there:
- Please check again for [ inside brackets. POSIX spec mentions some sequences starting with it that aren't literal. Re-check it and update the table.
Chatgpt now gave a better answer:
You're correct to call this out — in **POSIX**, an unescaped `[` inside a character class can indeed initiate **special bracket expressions** (`[:class:]`, `[=equiv=]`, `[.collate.]`). This makes it **not always treated as literal**, depending on **context and position**.
Here is the **updated full table** in **strict ReST ASCII format**, reflecting that correction:
```
+------------------------------------+------------+--------+-------------+-------------+-----------+-------------+
| Feature | Python re | Perl | POSIX ERE | POSIX BRE | grep -E | sed (BRE) |
+====================================+============+========+=============+=============+===========+=============+
| `]` (first position) | literal | literal| literal | literal | literal | literal |
+------------------------------------+------------+--------+-------------+-------------+-----------+-------------+
| `]` (non-first position) | \] | \] | \] | \] | \] | \] |
+------------------------------------+------------+--------+-------------+-------------+-----------+-------------+
| `[` | literal | literal| special* | special* | special* | special* |
+------------------------------------+------------+--------+-------------+-------------+-----------+-------------+
| `-` (between literals) | range `a-z`| range | range | range | range | range |
+------------------------------------+------------+--------+-------------+-------------+-----------+-------------+
| `-` (start or end) | literal | literal| literal | literal | literal | literal |
+------------------------------------+------------+--------+-------------+-------------+-----------+-------------+
| `^` (first position) | negation | negation| negation | negation | negation | negation |
+------------------------------------+------------+--------+-------------+-------------+-----------+-------------+
| `^` (non-first position) | literal | literal| literal | literal | literal | literal |
+------------------------------------+------------+--------+-------------+-------------+-----------+-------------+
| `\` | \\ | \\ | \\ | \\ | \\ | \\ |
+------------------------------------+------------+--------+-------------+-------------+-----------+-------------+
| POSIX class `[:alpha:]` | literal | literal| supported | supported | supported | supported |
+------------------------------------+------------+--------+-------------+-------------+-----------+-------------+
| POSIX special `[.ch.]` | - | - | supported | supported | supported | supported |
+------------------------------------+------------+--------+-------------+-------------+-----------+-------------+
| POSIX equivalence `[=a=]` | - | - | supported | supported | supported | supported |
+------------------------------------+------------+--------+-------------+-------------+-----------+-------------+
| `*` | literal | literal| literal | literal | literal | literal |
+------------------------------------+------------+--------+-------------+-------------+-----------+-------------+
| `$` | literal | literal| literal | literal | literal | literal |
+------------------------------------+------------+--------+-------------+-------------+-----------+-------------+
| `.` | literal | literal| literal | literal | literal | literal |
+------------------------------------+------------+--------+-------------+-------------+-----------+-------------+
```
### Footnote:
* \* In **POSIX ERE**, **POSIX BRE**, **`grep -E`**, and **`sed`**, an unescaped `[` **immediately followed by `:`, `=`, or `.`** at the start of a character class (e.g., `[[:`, `[.=`, `[==`) initiates a **special POSIX expression** and is **not a literal**. To match a literal `[`, avoid placing it in this position or use placement later in the class. Escaping `[` (`\[`), while common in practice, is not guaranteed portable in POSIX.
Let me know if you'd like to add more edge cases (e.g., `[[:[:alpha:]:]]` or nested misuse).
-
LLM queries to produce the above result with Deepseek:
- list on an ascii table python changes to re and regex modules that caused backward-compatibility issues.
- for internal "re" module, compare changes for "[]" brakets that contain characters on it. what chars/sequences of special characters can cause troubles on Python? What's the difference in terms of needing backslashes when compared with POSIX, Perl, sed and grep (with -E)? Ignore unicode. We're interested just in ascii special chars.
- get the table (3) and transpose it. check if different versions do different cases. Include both POSIX ERE and BRE. Add special sequences. for instance, POSIX have special meanings for having "[" followed by some other special characters. Show only the table
To allow you to reproduce, I also asked:
- I want to pass to someone else the LLM requests for him to produce exactly the same table.
It produced the following LLM query:
<snip>
Create a comparison table in ReST format showing escaping rules for special ASCII characters inside regex character classes `[]`. Follow these specifications:
1. **Tools**: Compare Python `re`, Perl, POSIX ERE, POSIX BRE, `grep -E`, and `sed` (BRE)
2. **Structure**: Transposed layout (features as rows, tools as columns)
3. **Content**: Include all these cases:
- `]` (first vs. non-first position)
- `[`, `-`, `^`, `\`
- POSIX classes (`[:alpha:]`)
- POSIX specials (`[.ch.]`, `[=a=]`)
- `*`, `$`, `.`
4. **Rules**:
- Mark required escapes with `\`
- Show range syntax (`a-z`) where applicable
- Indicate "literal" when no escape needed
5. **Format**: Strict ReST ASCII table with grid lines
6. **Notes**:
- Exclude Unicode
- Highlight version differences in footnotes
- Keep all tool columns even if identical
- Use `-` for unsupported features
Provide only the final table with no commentary.
</snip>
Powered by blists - more mailing lists