[<prev] [next>] [<thread-prev] [day] [month] [year] [list]
Message-ID: <aYIB-SKmYsx9j-M2@pathway.suse.cz>
Date: Tue, 3 Feb 2026 15:11:05 +0100
From: Petr Mladek <pmladek@...e.com>
To: ysard <ysard_git@....fr>
Cc: John Ogness <john.ogness@...utronix.de>, linux-kernel@...r.kernel.org,
senozhatsky@...omium.org
Subject: Re: Regression: system freeze on resume from suspend introduced by
printk per-console suspended state
On Tue 2026-02-03 02:32:46, ysard wrote:
> On 2026-02-02 12:02:07, Petr Mladek wrote:
> > Could you please provide the full log, including the backtraces?
> > I am curious about the callers of the console_lock()...
>
> Sorry, I had been on the same grep since the beginning.
> Here are the complete logs.
>
>
> > 1. console_lock() API called by
> > echo suspend >/proc/driver/nvidia/suspend
>
> https://pastebin.com/6gv0bF1c
>
> > 2. console_lock() API called by the suspend test
> >
> > No freeze (expected):
> > $ sudo sh -c "
> > mkdir -p /var/run/nvidia-sleep \
> > && echo 2 > /var/run/nvidia-sleep/Xorg.vt_number \
> > && chvt 63 \
> > && systemctl suspend"
>
> https://pastebin.com/10JBnnRx
>
> > 3. No freeze with the 1st patch and uncommented synchronize_srcu() calls.:
> > $ sudo sh -c "
> > mkdir -p /var/run/nvidia-sleep \
> > && echo 2 > /var/run/nvidia-sleep/Xorg.vt_number \
> > && chvt 63 \
> > && echo suspend >/proc/driver/nvidia/suspend \
> > && systemctl suspend"
>
> https://pastebin.com/nqks8h7Z
Thanks a lot for the complete logs. Honestly, I do not see any obvious
ansver on our guestions.
1. Backtraces with nvidia functions
===================================
I have hoped that the logs would include some backtraces where a
function from the nvidia driver would try to get console_lock().
But I can't find any.
Especially the 1st log from
"echo suspend >/proc/driver/nvidia/suspend"
does not include any nvidia code.
Note that there are functions from the nvidia driver in the 3rd log.
But they are all with '?', for example:
2026-01-31T23:01:51.032779+01:00 kernel: [ T1426] printk: console lock API call [21]: console_lock() by pid=1426, comm=Xorg
2026-01-31T23:01:51.032793+01:00 kernel: [ T1426] Call Trace:
2026-01-31T23:01:51.032794+01:00 kernel: [ T1426] <TASK>
2026-01-31T23:01:51.032796+01:00 kernel: [ T1426] print_console_lock_caller+0x154/0x220
2026-01-31T23:01:51.032797+01:00 kernel: [ T1426] console_lock+0x1a/0x60
2026-01-31T23:01:51.032798+01:00 kernel: [ T1426] vt_ioctl+0xe6d/0x1430
2026-01-31T23:01:51.032799+01:00 kernel: [ T1426] tty_ioctl+0x261/0x8d0
2026-01-31T23:01:51.032799+01:00 kernel: [ T1426] ? _nv023604rm+0x9/0x50 [nvidia]
2026-01-31T23:01:51.032800+01:00 kernel: [ T1426] ? _nv023631rm+0x28/0xf0 [nvidia]
2026-01-31T23:01:51.032801+01:00 kernel: [ T1426] __x64_sys_ioctl+0x96/0xe0
2026-01-31T23:01:51.032812+01:00 kernel: [ T1426] do_syscall_64+0x85/0x640
2026-01-31T23:01:51.032814+01:00 kernel: [ T1426] ? _nv011691rm+0xbe/0x100 [nvidia]
2026-01-31T23:01:51.032814+01:00 kernel: [ T1426] ? _nv037812rm+0x11a/0x160 [nvidia]
It means that the addresses were found on stack but they were not part
of the current call chain.
The adresses might be on stack because they were passed as parameters
or so. But more likely they are there as a garbage (return adresses)
from previous stack users.
2. console_(try)lock() callers with suspended consoles
======================================================
The regression has gone after we restored the original behavior of
console_(try/un)lock() API after console_suspend_all().
I would expect that a console_(try/un)lock() API caller would get
blocked or do an out-of-memory access or so. But there seems to
be only one caller in the 3rd log.
Let me explain.
The are 4 console_lock()/console_unlock() calls from the pr_flush()
in console_suspend_all(). For example, the 4th one is:
2026-01-31T23:01:57.758969+01:00 kernel: [ T2628] printk: console lock API call [50]: console_unlock() by pid=2628, comm=systemd-sleep
2026-01-31T23:01:57.758971+01:00 kernel: [ T2628] Call Trace:
2026-01-31T23:01:57.758972+01:00 kernel: [ T2628] <TASK>
2026-01-31T23:01:57.758973+01:00 kernel: [ T2628] print_console_lock_caller+0x154/0x220
2026-01-31T23:01:57.758975+01:00 kernel: [ T2628] console_unlock+0x22/0xf0
2026-01-31T23:01:57.758976+01:00 kernel: [ T2628] ? down+0x1e/0x70
2026-01-31T23:01:57.758978+01:00 kernel: [ T2628] __pr_flush+0x11e/0x280
2026-01-31T23:01:57.758979+01:00 kernel: [ T2628] ? _printk+0x54/0x60
2026-01-31T23:01:57.758980+01:00 kernel: [ T2628] console_suspend_all+0x27/0xc0
2026-01-31T23:01:57.758981+01:00 kernel: [ T2628] suspend_devices_and_enter+0x11b/0xa20
2026-01-31T23:01:57.758983+01:00 kernel: [ T2628] pm_suspend+0x19b/0x620
2026-01-31T23:01:57.758984+01:00 kernel: [ T2628] state_store+0x6c/0xd0
Then there is one more console_lock() call, right before printing the
message into the log:
2026-01-31T23:01:57.759223+01:00 kernel: [ T2628] printk: console lock API call [51]: console_lock() by pid=2628, comm=systemd-sleep
2026-01-31T23:01:57.759225+01:00 kernel: [ T2628] Call Trace:
2026-01-31T23:01:57.759227+01:00 kernel: [ T2628] <TASK>
2026-01-31T23:01:57.759229+01:00 kernel: [ T2628] print_console_lock_caller+0x154/0x220
2026-01-31T23:01:57.759230+01:00 kernel: [ T2628] console_lock+0x1a/0x60
2026-01-31T23:01:57.759232+01:00 kernel: [ T2628] console_suspend_all+0x41/0xc0
2026-01-31T23:01:57.759233+01:00 kernel: [ T2628] suspend_devices_and_enter+0x11b/0xa20
2026-01-31T23:01:57.759248+01:00 kernel: [ T2628] pm_suspend+0x19b/0x620
2026-01-31T23:01:57.759251+01:00 kernel: [ T2628] state_store+0x6c/0xd0
2026-01-31T23:01:57.759253+01:00 kernel: [ T2628] kernfs_fop_write_iter+0x130/0x210
2026-01-31T23:01:57.759255+01:00 kernel: [ T2628] vfs_write+0x274/0x430
2026-01-31T23:01:57.759256+01:00 kernel: [ T2628] ksys_write+0x5e/0xe0
2026-01-31T23:01:57.759258+01:00 kernel: [ T2628] do_syscall_64+0x85/0x640
2026-01-31T23:01:57.759259+01:00 kernel: [ T2628] ? ___pte_offset_map+0x1b/0x110
2026-01-31T23:01:57.759260+01:00 kernel: [ T2628] ? __handle_mm_fault+0xbdb/0x1020
2026-01-31T23:01:57.759261+01:00 kernel: [ T2628] ? count_memcg_events+0xdd/0x1a0
2026-01-31T23:01:57.759263+01:00 kernel: [ T2628] ? handle_mm_fault+0xbf/0x300
2026-01-31T23:01:57.759264+01:00 kernel: [ T2628] ? do_user_addr_fault+0x2c1/0x880
2026-01-31T23:01:57.759266+01:00 kernel: [ T2628] ? irqentry_exit+0x7b/0x6b0
2026-01-31T23:01:57.759267+01:00 kernel: [ T2628] entry_SYSCALL_64_after_hwframe+0x76/0x7e
2026-01-31T23:01:57.759269+01:00 kernel: [ T2628] RIP: 0033:0x7f961aa5e687
2026-01-31T23:01:57.759270+01:00 kernel: [ T2628] Code: 48 89 fa 4c 89 df e8 58 b3 00 00 8b 93 08 03 00 00 59 5e 48 83 f8 fc 74 1a 5b c3 0f 1f 84 00 00 00 00 00 48 8b 44 24 10 0f 05 <5b> c3 0f 1f 80 00 00 00 00 83 e2 39 83 fa 08 75 de e8 23 ff ff ff
2026-01-31T23:01:57.759272+01:00 kernel: [ T2628] RSP: 002b:00007ffde2cb0040 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
2026-01-31T23:01:57.759273+01:00 kernel: [ T2628] RAX: ffffffffffffffda RBX: 00007f961b0f8c80 RCX: 00007f961aa5e687
2026-01-31T23:01:57.759275+01:00 kernel: [ T2628] RDX: 0000000000000004 RSI: 000055993c9fee10 RDI: 0000000000000007
2026-01-31T23:01:57.759276+01:00 kernel: [ T2628] RBP: 000055993c9fee10 R08: 0000000000000000 R09: 0000000000000000
2026-01-31T23:01:57.759278+01:00 kernel: [ T2628] R10: 0000000000000000 R11: 0000000000000202 R12: 0000000000000004
2026-01-31T23:01:57.759279+01:00 kernel: [ T2628] R13: 000055993c9f62a0 R14: 00007f961abb4e80 R15: 00007ffde2cb01b0
2026-01-31T23:01:57.759280+01:00 kernel: [ T2628] </TASK>
2026-01-31T23:01:57.759282+01:00 kernel: [ T2628] printk: console_suspend_all
Now, here is the critical section where the console_lock() API behaves
differently. But there there is no caller until the system gets
suspended:
2026-01-31T23:01:57.759283+01:00 kernel: [ T2641] sd 0:0:0:0: [sdc] Synchronizing SCSI cache
2026-01-31T23:01:57.759285+01:00 kernel: [ T269] ata1.00: Entering standby power mode
2026-01-31T23:01:57.759286+01:00 kernel: [ T2638] sd 1:0:0:0: [sda] Synchronizing SCSI cache
2026-01-31T23:01:57.759288+01:00 kernel: [ T2639] sd 2:0:0:0: [sdb] Synchronizing SCSI cache
2026-01-31T23:01:57.759289+01:00 kernel: [ T271] ata2.00: Entering standby power mode
2026-01-31T23:01:57.759290+01:00 kernel: [ T273] ata3.00: Entering standby power mode
2026-01-31T23:01:57.759291+01:00 kernel: [ T2628] ACPI: EC: interrupt blocked
2026-01-31T23:01:57.759293+01:00 kernel: [ T2628] ACPI: PM: Preparing to enter system sleep state S3
2026-01-31T23:01:57.759294+01:00 kernel: [ T2628] ACPI: EC: event blocked
2026-01-31T23:01:57.759296+01:00 kernel: [ T2628] ACPI: EC: EC stopped
2026-01-31T23:01:57.759297+01:00 kernel: [ T2628] ACPI: PM: Saving platform NVS memory
2026-01-31T23:01:57.759299+01:00 kernel: [ T2628] Disabling non-boot CPUs ...
2026-01-31T23:01:57.759300+01:00 kernel: [ T2628] smpboot: CPU 7 is now offline
2026-01-31T23:01:57.759301+01:00 kernel: [ T2628] smpboot: CPU 6 is now offline
2026-01-31T23:01:57.759303+01:00 kernel: [ T2628] smpboot: CPU 5 is now offline
2026-01-31T23:01:57.759304+01:00 kernel: [ T2628] smpboot: CPU 4 is now offline
2026-01-31T23:01:57.759306+01:00 kernel: [ T2628] smpboot: CPU 3 is now offline
2026-01-31T23:01:57.759307+01:00 kernel: [ T2628] smpboot: CPU 2 is now offline
2026-01-31T23:01:57.759309+01:00 kernel: [ T2628] smpboot: CPU 1 is now offline
Then the resume starts:
2026-01-31T23:01:57.759310+01:00 kernel: [ T2628] ACPI: PM: Low-level resume complete
2026-01-31T23:01:57.759312+01:00 kernel: [ T2628] ACPI: EC: EC started
2026-01-31T23:01:57.759313+01:00 kernel: [ T2628] ACPI: PM: Restoring platform NVS memory
2026-01-31T23:01:57.759314+01:00 kernel: [ T2628] Enabling non-boot CPUs ...
2026-01-31T23:01:57.759316+01:00 kernel: [ T2628] smpboot: Booting Node 0 Processor 1 APIC 0x2
2026-01-31T23:01:57.759317+01:00 kernel: [ T2628] CPU1 is up
2026-01-31T23:01:57.759319+01:00 kernel: [ T2628] smpboot: Booting Node 0 Processor 2 APIC 0x4
2026-01-31T23:01:57.759320+01:00 kernel: [ T2628] CPU2 is up
2026-01-31T23:01:57.759321+01:00 kernel: [ T2628] smpboot: Booting Node 0 Processor 3 APIC 0x6
2026-01-31T23:01:57.759323+01:00 kernel: [ T2628] CPU3 is up
2026-01-31T23:01:57.759324+01:00 kernel: [ T2628] smpboot: Booting Node 0 Processor 4 APIC 0x1
2026-01-31T23:01:57.759325+01:00 kernel: [ T2628] CPU4 is up
2026-01-31T23:01:57.759326+01:00 kernel: [ T2628] smpboot: Booting Node 0 Processor 5 APIC 0x3
2026-01-31T23:01:57.759328+01:00 kernel: [ T2628] CPU5 is up
2026-01-31T23:01:57.759329+01:00 kernel: [ T2628] smpboot: Booting Node 0 Processor 6 APIC 0x5
2026-01-31T23:01:57.759330+01:00 kernel: [ T2628] CPU6 is up
2026-01-31T23:01:57.759332+01:00 kernel: [ T2628] smpboot: Booting Node 0 Processor 7 APIC 0x7
2026-01-31T23:01:57.759333+01:00 kernel: [ T2628] CPU7 is up
2026-01-31T23:01:57.759335+01:00 kernel: [ T2628] ACPI: PM: Waking up from system sleep state S3
2026-01-31T23:01:57.759336+01:00 kernel: [ T2628] ACPI: EC: interrupt unblocked
2026-01-31T23:01:57.759337+01:00 kernel: [ T70] usb usb1: root hub lost power or was reset
2026-01-31T23:01:57.759339+01:00 kernel: [ T2650] xhci_hcd 0000:00:14.0: xHC error in resume, USBSTS 0x411, Reinit
2026-01-31T23:01:57.759340+01:00 kernel: [ T2650] usb usb3: root hub lost power or was reset
2026-01-31T23:01:57.759342+01:00 kernel: [ T2650] usb usb4: root hub lost power or was reset
2026-01-31T23:01:57.759343+01:00 kernel: [ T2628] ACPI: EC: event unblocked
2026-01-31T23:01:57.759344+01:00 kernel: [ T2655] usb usb2: root hub lost power or was reset
and finally, there is a console_lock API caller:
2026-01-31T23:01:57.759347+01:00 kernel: [ T97] printk: console lock API call [52]: console_lock() by pid=97, comm=kworker/2:1+events
2026-01-31T23:01:57.759349+01:00 kernel: [ T97] Call Trace:
2026-01-31T23:01:57.759350+01:00 kernel: [ T97] <TASK>
2026-01-31T23:01:57.759351+01:00 kernel: [ T97] print_console_lock_caller+0x154/0x220
2026-01-31T23:01:57.759353+01:00 kernel: [ T97] console_lock+0x1a/0x60
2026-01-31T23:01:57.759354+01:00 kernel: [ T97] console_callback+0x14/0x1e0
2026-01-31T23:01:57.759355+01:00 kernel: [ T97] process_one_work+0x19b/0x3c0
2026-01-31T23:01:57.759357+01:00 kernel: [ T97] worker_thread+0x256/0x370
2026-01-31T23:01:57.759358+01:00 kernel: [ T97] ? __pfx_worker_thread+0x10/0x10
2026-01-31T23:01:57.759360+01:00 kernel: [ T97] kthread+0xef/0x230
2026-01-31T23:01:57.759361+01:00 kernel: [ T97] ? __pfx_kthread+0x10/0x10
2026-01-31T23:01:57.759362+01:00 kernel: [ T97] ? __pfx_kthread+0x10/0x10
2026-01-31T23:01:57.759363+01:00 kernel: [ T97] ret_from_fork+0x20b/0x260
2026-01-31T23:01:57.759365+01:00 kernel: [ T97] ? __pfx_kthread+0x10/0x10
2026-01-31T23:01:57.759366+01:00 kernel: [ T97] ret_from_fork_asm+0x1a/0x30
2026-01-31T23:01:57.759367+01:00 kernel: [ T97] </TASK>
2026-01-31T23:01:57.759369+01:00 kernel: [ T97] printk: console lock API call [53]: console_unlock() by pid=97, comm=kworker/2:1+events
2026-01-31T23:01:57.759370+01:00 kernel: [ T97] Call Trace:
2026-01-31T23:01:57.759371+01:00 kernel: [ T97] <TASK>
2026-01-31T23:01:57.759373+01:00 kernel: [ T97] print_console_lock_caller+0x154/0x220
2026-01-31T23:01:57.759374+01:00 kernel: [ T97] console_unlock+0x22/0xf0
2026-01-31T23:01:57.759376+01:00 kernel: [ T97] process_one_work+0x19b/0x3c0
2026-01-31T23:01:57.759377+01:00 kernel: [ T97] worker_thread+0x256/0x370
2026-01-31T23:01:57.759378+01:00 kernel: [ T97] ? __pfx_worker_thread+0x10/0x10
2026-01-31T23:01:57.759380+01:00 kernel: [ T97] kthread+0xef/0x230
2026-01-31T23:01:57.759381+01:00 kernel: [ T97] ? __pfx_kthread+0x10/0x10
2026-01-31T23:01:57.759382+01:00 kernel: [ T97] ? __pfx_kthread+0x10/0x10
2026-01-31T23:01:57.759384+01:00 kernel: [ T97] ret_from_fork+0x20b/0x260
2026-01-31T23:01:57.759385+01:00 kernel: [ T97] ? __pfx_kthread+0x10/0x10
2026-01-31T23:01:57.759386+01:00 kernel: [ T97] ret_from_fork_asm+0x1a/0x30
2026-01-31T23:01:57.759388+01:00 kernel: [ T97] </TASK>
It seems to be from the console_callback() from the generic
drivers/tty/vt/vt.c code.
Sigh, I am not sure if it does anything which might be affected by
the state of the nvidia driver. Mabye, we should add more people
into Cc who are more familiar with the VT code and graphical
drivers.
Anyway, this was the last and only call before console_resume_all()
was called:
2026-01-31T23:01:57.759389+01:00 kernel: [ T2635] usb 1-1: reset high-speed USB device number 2 using ehci-pci
2026-01-31T23:01:57.759390+01:00 kernel: [ T2652] usb 2-1: reset high-speed USB device number 2 using ehci-pci
2026-01-31T23:01:57.759392+01:00 kernel: [ T97] psmouse serio4: synaptics: queried max coordinates: x [..5660], y [..4746]
2026-01-31T23:01:57.759393+01:00 kernel: [ T273] ata3: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
2026-01-31T23:01:57.759394+01:00 kernel: [ T271] ata2: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
2026-01-31T23:01:57.759396+01:00 kernel: [ T273] ata3.00: ACPI cmd f5/00:00:00:00:00:a0(SECURITY FREEZE LOCK) filtered out
2026-01-31T23:01:57.759397+01:00 kernel: [ T271] ata2.00: ACPI cmd f5/00:00:00:00:00:a0(SECURITY FREEZE LOCK) filtered out
2026-01-31T23:01:57.759398+01:00 kernel: [ T271] ata2.00: ACPI cmd ef/10:03:00:00:00:a0(SET FEATURES) filtered out
2026-01-31T23:01:57.759581+01:00 kernel: [ T273] ata3.00: ACPI cmd ef/10:03:00:00:00:a0(SET FEATURES) filtered out
2026-01-31T23:01:57.759584+01:00 kernel: [ T273] ata3.00: supports DRM functions and may not be fully accessible
2026-01-31T23:01:57.759586+01:00 kernel: [ T271] ata2.00: supports DRM functions and may not be fully accessible
2026-01-31T23:01:57.759588+01:00 kernel: [ T88] sd 2:0:0:0: [sdb] Starting disk
2026-01-31T23:01:57.759589+01:00 kernel: [ T273] ata3.00: ACPI cmd f5/00:00:00:00:00:a0(SECURITY FREEZE LOCK) filtered out
2026-01-31T23:01:57.759591+01:00 kernel: [ T273] ata3.00: ACPI cmd ef/10:03:00:00:00:a0(SET FEATURES) filtered out
2026-01-31T23:01:57.759593+01:00 kernel: [ T273] ata3.00: supports DRM functions and may not be fully accessible
2026-01-31T23:01:57.759594+01:00 kernel: [ T179] sd 1:0:0:0: [sda] Starting disk
2026-01-31T23:01:57.759596+01:00 kernel: [ T273] ata3.00: configured for UDMA/133
2026-01-31T23:01:57.759597+01:00 kernel: [ T271] ata2.00: ACPI cmd f5/00:00:00:00:00:a0(SECURITY FREEZE LOCK) filtered out
2026-01-31T23:01:57.759599+01:00 kernel: [ T271] ata2.00: ACPI cmd ef/10:03:00:00:00:a0(SET FEATURES) filtered out
2026-01-31T23:01:57.759600+01:00 kernel: [ T97] psmouse serio4: synaptics: queried min coordinates: x [1282..], y [1106..]
2026-01-31T23:01:57.759602+01:00 kernel: [ T271] ata2.00: supports DRM functions and may not be fully accessible
2026-01-31T23:01:57.759603+01:00 kernel: [ T271] ata2.00: configured for UDMA/133
2026-01-31T23:01:57.759605+01:00 kernel: [ T2667] usb 3-4: reset low-speed USB device number 3 using xhci_hcd
2026-01-31T23:01:57.759606+01:00 kernel: [ T2635] usb 1-1.3: reset high-speed USB device number 3 using ehci-pci
2026-01-31T23:01:57.759608+01:00 kernel: [ T2665] usb 3-1: reset high-speed USB device number 2 using xhci_hcd
2026-01-31T23:01:57.759609+01:00 kernel: [ T2628] printk: console_resume_all
2026-01-31T23:01:57.759610+01:00 kernel: [ T2628] printk: console lock API call [54]: console_unlock() by pid=2628, comm=systemd-sleep
2026-01-31T23:01:57.759612+01:00 kernel: [ T2628] Call Trace:
2026-01-31T23:01:57.759613+01:00 kernel: [ T2628] <TASK>
2026-01-31T23:01:57.759614+01:00 kernel: [ T2628] print_console_lock_caller+0x154/0x220
2026-01-31T23:01:57.759616+01:00 kernel: [ T2628] console_unlock+0x22/0xf0
2026-01-31T23:01:57.759617+01:00 kernel: [ T2628] console_resume_all+0xe4/0x100
2026-01-31T23:01:57.759619+01:00 kernel: [ T2628] suspend_devices_and_enter+0x340/0xa20
2026-01-31T23:01:57.759620+01:00 kernel: [ T2628] pm_suspend+0x19b/0x620
2026-01-31T23:01:57.759621+01:00 kernel: [ T2628] state_store+0x6c/0xd0
So, the problem might be in the console_callback() callbacks. But it
seems that most code paths are not called. Otherwise,
WARN_CONSOLE_UNLOCKED() would warned. Note that the reverted
console_lock() does not set "console_locked" when suspended.
Or we might miss some calls. For example, console_trylock() calls
in vprintk_emit() and fb_flashcursor() are not logged. But again
a lot of vt code is guarded by WARN_CONSOLE_UNLOCKED(). And these
parts were not called almost for sure.
Hmm, I am not sure how to move forward.
One more idea is to try shuffling the code in console_trylock()/
console_lock()/console_unlock() and see when the regression
will be visible again. I mean to restore the "broken" behavior
step by step:
1. set console_locked = 1 even when console_suspended
2. set console_may_schedule = 1 even when console_suspended
3. call __console_flush_and_unlock() even when console_suspended
I wonder which particular change would break the suspend.
It might be another clue.
Any other ideas are welcome.
Best Regards,
Petr
Powered by blists - more mailing lists