lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  linux-cve-announce  PHC 
Open Source and information security mailing list archives
 
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-ID: <1882425827.361582.1476816475343.JavaMail.zimbra@redhat.com>
Date:   Tue, 18 Oct 2016 14:47:55 -0400 (EDT)
From:   CAI Qian <caiqian@...hat.com>
To:     Rob Herring <robh@...nel.org>,
        Alessandro Zummo <a.zummo@...ertech.it>,
        Alexandre Belloni <alexandre.belloni@...e-electrons.com>
Cc:     Greg Kroah-Hartman <gregkh@...uxfoundation.org>,
        rtc-linux@...glegroups.com,
        linux-kernel <linux-kernel@...r.kernel.org>
Subject: Re: [4.9-rc1] kernel panic from `cat /proc/driver/rtc`

It turns out this panic can only be reproduced with CONFIG_DEBUG_TEST_DRIVER_REMOVE
enabled. There are some errors in dmesg when the config is enabled.

[   71.215937] rtc_cmos 00:00: RTC can wake from S4
[   71.218096] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio1/input/input2
[   71.232591] rtc_cmos 00:00: rtc core: registered rtc_cmos as rtc0
[   71.239518] rtc_cmos 00:00: alarms up to one month, y3k, 114 bytes nvram, hpet irqs
[   71.248160] rtc_cmos 00:00: RTC can wake from S4
[   71.267680] rtc_cmos: probe of 00:00 failed with error -16

It works fine without it.

$ cat /proc/driver/rtc 
rtc_time        : 14:30:56
rtc_date        : 2016-10-18
alrm_time        : 19:48:53
alrm_date        : 2016-10-18
alarm_IRQ        : no
alrm_pending        : no
update IRQ enabled        : no
periodic IRQ enabled        : no
periodic IRQ frequency        : 1024
max user IRQ frequency        : 64
24hr                : yes
periodic_IRQ        : no
update_IRQ        : no
HPET_emulated        : yes
BCD                : yes
DST_enable        : no
periodic_freq        : 1024
batt_status        : okay

   CAI Qian

----- Original Message -----
> From: "CAI Qian" <caiqian@...hat.com>
> To: rtc-linux@...glegroups.com, "linux-kernel" <linux-kernel@...r.kernel.org>
> Cc: "Alessandro Zummo" <a.zummo@...ertech.it>, "Alexandre Belloni" <alexandre.belloni@...e-electrons.com>
> Sent: Tuesday, October 18, 2016 9:28:12 AM
> Subject: [4.9-rc1] kernel panic from `cat /proc/driver/rtc`
> 
> This looks like new introduced in the 4.9 merge window. I have never saw any
> of
> those while testing v4.8.
>    CAI Qian
> 
> $ cat /proc/driver/rtc
> 
> [ 7890.728704] UBSAN: Undefined behaviour in drivers/rtc/rtc-cmos.c:433:10
> [ 7890.736088] member access within null pointer of type 'struct cmos_rtc'
> [ 7890.743472] CPU: 81 PID: 32522 Comm: proc01 Tainted: G        W
> 4.9.0-rc1 #32
> [ 7890.752017] Hardware name: Intel Corporation S2600WTT/S2600WTT, BIOS
> GRRFSDP1.86B.0271.R00.1510301446 10/30/2015
> [ 7890.763373]  ffff88080a54f868 ffffffff81d23184 0000000041b58ab3
> ffffffff8334713f
> [ 7890.771670]  ffffffff81d230c4 ffff88080a54f890 ffff88080a54f840
> ffff88081f324900
> [ 7890.779966]  ffffffff82ff7b40 ffffffff831279a0 00000000000001b1
> 00000000d3274681
> [ 7890.788262] Call Trace:
> [ 7890.790993]  [<ffffffff81d23184>] dump_stack+0xc0/0x12c
> [ 7890.796825]  [<ffffffff81d230c4>] ? _atomic_dec_and_lock+0xc4/0xc4
> [ 7890.803723]  [<ffffffff81e46d55>] ubsan_epilogue+0xd/0x8a
> [ 7890.809748]  [<ffffffff81e46f38>] __ubsan_handle_type_mismatch+0x166/0x434
> [ 7890.817421]  [<ffffffff81e46dd2>] ? ubsan_epilogue+0x8a/0x8a
> [ 7890.823738]  [<ffffffff81dce253>] ? __this_cpu_preempt_check+0x13/0x20
> [ 7890.831025]  [<ffffffff8131c580>] ? trace_hardirqs_on_caller+0x520/0x720
> [ 7890.838509]  [<ffffffff8253b351>] cmos_procfs+0x1b1/0x1e0
> [ 7890.844535]  [<ffffffff8253b1a0>] ? rtc_handler+0x140/0x140
> [ 7890.850754]  [<ffffffff82538580>] rtc_proc_show+0x180/0x640
> [ 7890.856973]  [<ffffffff82538400>] ? rtc_proc_open+0xd0/0xd0
> [ 7890.863196]  [<ffffffff817eb26d>] ? kasan_kmalloc+0xad/0xe0
> [ 7890.869419]  [<ffffffff81901d44>] seq_read+0x334/0x1400
> [ 7890.875252]  [<ffffffff81901a10>] ? seq_hlist_start_percpu+0x4a0/0x4a0
> [ 7890.882538]  [<ffffffff810952fb>] ? save_stack_trace+0x1b/0x20
> [ 7890.889050]  [<ffffffff817eaff6>] ? save_stack+0x46/0xd0
> [ 7890.894979]  [<ffffffff817eb821>] ? kasan_slab_free+0x71/0xb0
> [ 7890.901393]  [<ffffffff817e3fd9>] ? kmem_cache_free+0xe9/0x660
> [ 7890.907905]  [<ffffffff818affe0>] ? putname+0xe0/0x120
> [ 7890.913639]  [<ffffffff81319fb0>] ? print_usage_bug+0x700/0x700
> [ 7890.920250]  [<ffffffff81a27160>] proc_reg_read+0x110/0x270
> [ 7890.926470]  [<ffffffff81870ff6>] __vfs_read+0x106/0x990
> [ 7890.932398]  [<ffffffff81870ef0>] ? do_iter_readv_writev+0x840/0x840
> [ 7890.939490]  [<ffffffff81b35ab5>] ? selinux_file_permission+0x3c5/0x550
> [ 7890.946874]  [<ffffffff81b12ef6>] ? security_file_permission+0x176/0x220
> [ 7890.954354]  [<ffffffff81871ec8>] ? rw_verify_area+0xd8/0x380
> [ 7890.960767]  [<ffffffff81872288>] vfs_read+0x118/0x400
> [ 7890.966500]  [<ffffffff8187710f>] SyS_read+0xdf/0x1d0
> [ 7890.972137]  [<ffffffff81877030>] ? vfs_copy_file_range+0x8f0/0x8f0
> [ 7890.979132]  [<ffffffff81dce253>] ? __this_cpu_preempt_check+0x13/0x20
> [ 7890.986416]  [<ffffffff81877030>] ? vfs_copy_file_range+0x8f0/0x8f0
> [ 7890.993412]  [<ffffffff8100924d>] do_syscall_64+0x19d/0x540
> [ 7890.999631]  [<ffffffff82c76f24>] entry_SYSCALL64_slow_path+0x25/0x25
> [ 7891.006820]
> ================================================================================
> [ 7891.016322] kasan: CONFIG_KASAN_INLINE enabled
> [ 7891.021292] kasan: GPF could be caused by NULL-ptr deref or user memory
> access
> [ 7891.029371] general protection fault: 0000 [#1] PREEMPT SMP
> DEBUG_PAGEALLOC KASAN
> [ 7891.037722] Modules linked in: tun ext4 jbd2 mbcache loop veth
> ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4
> nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype iptable_filter xt_conntrack nf_nat
> nf_conntrack br_netfilter bridge stp llc overlay intel_rapl sb_edac
> edac_core x86_pkg_temp_thermal intel_powerclamp coretemp crct10dif_pclmul
> crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper
> ablk_helper cryptd intel_uncore iTCO_wdt iTCO_vendor_support pcspkr i2c_i801
> i2c_smbus sg mei_me mei lpc_ich shpchp ipmi_ssif mxm_wmi ipmi_si
> ipmi_msghandler wmi acpi_power_meter acpi_pad nfsd auth_rpcgss nfs_acl lockd
> grace sunrpc ip_tables xfs libcrc32c sr_mod sd_mod cdrom mgag200
> i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops
> ttm crc32c_intel drm ixgbe serio_raw ahci libahci libata mdio ptp i2c_core
> pps_core dca fjes dm_mirror dm_region_hash dm_log dm_mod
> [ 7891.127218] CPU: 81 PID: 32522 Comm: proc01 Tainted: G        W
> 4.9.0-rc1 #32
> [ 7891.135764] Hardware name: Intel Corporation S2600WTT/S2600WTT, BIOS
> GRRFSDP1.86B.0271.R00.1510301446 10/30/2015
> [ 7891.147124] task: ffff88081f324900 task.stack: ffff88080a548000
> [ 7891.153731] RIP: 0010:[<ffffffff8253b250>]  [<ffffffff8253b250>]
> cmos_procfs+0xb0/0x1e0
> [ 7891.162677] RSP: 0018:ffff88080a54f938  EFLAGS: 00010246
> [ 7891.168605] RAX: dffffc0000000000 RBX: 0000000000000000 RCX:
> 0000000000000000
> [ 7891.176569] RDX: 0000000000000000 RSI: ffffffff82e9a500 RDI:
> ffffed01014a9f20
> [ 7891.184534] RBP: ffff88080a54f990 R08: ffff88081f324900 R09:
> 0000000000000007
> [ 7891.192499] R10: ffff88080a54f780 R11: 0000000000000006 R12:
> 0000000000000002
> [ 7891.200463] R13: ffffffff831272e0 R14: 1ffff101014a9f39 R15:
> ffffffff83127d60
> [ 7891.208430] FS:  00007fe516b93800(0000) GS:ffff880e56800000(0000)
> knlGS:0000000000000000
> [ 7891.217461] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [ 7891.223873] CR2: 00007f153a7200a0 CR3: 0000000e48f80000 CR4:
> 00000000003406e0
> [ 7891.231838] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
> 0000000000000000
> [ 7891.239802] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7:
> 0000000000000400
> [ 7891.247765] Stack:
> [ 7891.250006]  0000012300000001 ffff880800000000 1ffff101014a9f39
> 00000000d3274681
> [ 7891.258299]  00000000d3274681 ffff880e108a9a40 ffff880e108a9a40
> ffff88084745b300
> [ 7891.266597]  ffffffff8253b1a0 1ffff101014a9f39 ffffffff83127d60
> ffff88080a54fad0
> [ 7891.274891] Call Trace:
> [ 7891.277619]  [<ffffffff8253b1a0>] ? rtc_handler+0x140/0x140
> [ 7891.283839]  [<ffffffff82538580>] rtc_proc_show+0x180/0x640
> [ 7891.290058]  [<ffffffff82538400>] ? rtc_proc_open+0xd0/0xd0
> [ 7891.296278]  [<ffffffff817eb26d>] ? kasan_kmalloc+0xad/0xe0
> [ 7891.302497]  [<ffffffff81901d44>] seq_read+0x334/0x1400
> [ 7891.308328]  [<ffffffff81901a10>] ? seq_hlist_start_percpu+0x4a0/0x4a0
> [ 7891.315613]  [<ffffffff810952fb>] ? save_stack_trace+0x1b/0x20
> [ 7891.322122]  [<ffffffff817eaff6>] ? save_stack+0x46/0xd0
> [ 7891.328050]  [<ffffffff817eb821>] ? kasan_slab_free+0x71/0xb0
> [ 7891.334463]  [<ffffffff817e3fd9>] ? kmem_cache_free+0xe9/0x660
> [ 7891.340972]  [<ffffffff818affe0>] ? putname+0xe0/0x120
> [ 7891.346707]  [<ffffffff81319fb0>] ? print_usage_bug+0x700/0x700
> [ 7891.353313]  [<ffffffff81a27160>] proc_reg_read+0x110/0x270
> [ 7891.359532]  [<ffffffff81870ff6>] __vfs_read+0x106/0x990
> [ 7891.365459]  [<ffffffff81870ef0>] ? do_iter_readv_writev+0x840/0x840
> [ 7891.372553]  [<ffffffff81b35ab5>] ? selinux_file_permission+0x3c5/0x550
> [ 7891.379936]  [<ffffffff81b12ef6>] ? security_file_permission+0x176/0x220
> [ 7891.387415]  [<ffffffff81871ec8>] ? rw_verify_area+0xd8/0x380
> [ 7891.393830]  [<ffffffff81872288>] vfs_read+0x118/0x400
> [ 7891.399564]  [<ffffffff8187710f>] SyS_read+0xdf/0x1d0
> [ 7891.405202]  [<ffffffff81877030>] ? vfs_copy_file_range+0x8f0/0x8f0
> [ 7891.412197]  [<ffffffff81dce253>] ? __this_cpu_preempt_check+0x13/0x20
> [ 7891.419483]  [<ffffffff81877030>] ? vfs_copy_file_range+0x8f0/0x8f0
> [ 7891.426478]  [<ffffffff8100924d>] do_syscall_64+0x19d/0x540
> [ 7891.432699]  [<ffffffff82c76f24>] entry_SYSCALL64_slow_path+0x25/0x25
> [ 7891.439881] Code: c7 c0 20 73 12 83 49 c7 c5 e0 72 12 83 4c 0f 49 e8 48 85
> db 0f 84 04 01 00 00 48 89 da 48 b8 00 00 00 00 00 fc ff df 48 c1 ea 03 <80>
> 3c 02 00 0f 85 dc 00 00 00 48 8b 1b 48 85 db 0f 84 bd 00 00
> [ 7891.461643] RIP  [<ffffffff8253b250>] cmos_procfs+0xb0/0x1e0
> [ 7891.467970]  RSP <ffff88080a54f938>
> [ 7891.478688] ---[ end trace 724fbd49872953c3 ]---
> [ 7891.483921] Kernel panic - not syncing: Fatal exception
> [ 7891.489875] Kernel Offset: disabled
> [ 7891.493787] ---[ end Kernel panic - not syncing: Fatal exception
> [ 7891.500491] ------------[ cut here ]------------
> [ 7891.505648] WARNING: CPU: 81 PID: 32522 at arch/x86/kernel/smp.c:127
> native_smp_send_reschedule+0x107/0x150
> [ 7891.516521] Modules linked in: tun ext4 jbd2 mbcache loop veth
> ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4
> nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype iptable_filter xt_conntrack nf_nat
> nf_conntrack br_netfilter bridge stp llc overlay intel_rapl sb_edac
> edac_core x86_pkg_temp_thermal intel_powerclamp coretemp crct10dif_pclmul
> crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper
> ablk_helper cryptd intel_uncore iTCO_wdt iTCO_vendor_support pcspkr i2c_i801
> i2c_smbus sg mei_me mei lpc_ich shpchp ipmi_ssif mxm_wmi ipmi_si
> ipmi_msghandler wmi acpi_power_meter acpi_pad nfsd auth_rpcgss nfs_acl lockd
> grace sunrpc ip_tables xfs libcrc32c sr_mod sd_mod cdrom mgag200
> i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops
> ttm crc32c_intel drm ixgbe serio_raw ahci libahci libata mdio ptp i2c_core
> pps_core dca fjes dm_mirror dm_region_hash dm_log dm_mod
> [ 7891.605894] CPU: 81 PID: 32522 Comm: proc01 Tainted: G      D W
> 4.9.0-rc1 #32
> [ 7891.614440] Hardware name: Intel Corporation S2600WTT/S2600WTT, BIOS
> GRRFSDP1.86B.0271.R00.1510301446 10/30/2015
> [ 7891.625798]  ffff880e56807a00 ffffffff81d23184 0000000041b58ab3
> ffffffff8334713f
> [ 7891.634094]  ffffffff81d230c4 000000000333a700 ffff880e568079c0
> ffffffff81293d7c
> [ 7891.642390]  00000000d3274681 ffff880e56807a78 0000000000000051
> 0000000000007f0a
> [ 7891.650686] Call Trace:
> [ 7891.653413]  <IRQ> [ 7891.655561]  [<ffffffff81d23184>]
> dump_stack+0xc0/0x12c
> [ 7891.661401]  [<ffffffff81d230c4>] ? _atomic_dec_and_lock+0xc4/0xc4
> [ 7891.668302]  [<ffffffff81293d7c>] ? rcu_read_unlock+0x3c/0x80
> [ 7891.674719]  [<ffffffff811b06ee>] __warn+0x19e/0x1d0
> [ 7891.680259]  [<ffffffff811b090d>] warn_slowpath_null+0x1d/0x20
> [ 7891.686769]  [<ffffffff810dbe57>] native_smp_send_reschedule+0x107/0x150
> [ 7891.694253]  [<ffffffff8126cddb>] try_to_wake_up+0x9fb/0x10d0
> [ 7891.700665]  [<ffffffff8126c3e0>] ? migrate_swap_stop+0xa70/0xa70
> [ 7891.707468]  [<ffffffff8126d662>] default_wake_function+0x42/0x70
> [ 7891.714271]  [<ffffffff812ed575>] __wake_up_common+0xe5/0x210
> [ 7891.720688]  [<ffffffff8198d33f>] ? ep_poll_callback+0x5ef/0x9d0
> [ 7891.727391]  [<ffffffff812ed703>] __wake_up_locked+0x13/0x20
> [ 7891.733707]  [<ffffffff8198cfc1>] ep_poll_callback+0x271/0x9d0
> [ 7891.740218]  [<ffffffff8198d230>] ? ep_poll_callback+0x4e0/0x9d0
> [ 7891.746921]  [<ffffffff8198cd50>] ? ep_poll_safewake+0x60/0x60
> [ 7891.753433]  [<ffffffff812ed6c3>] ? __wake_up+0x23/0x50
> [ 7891.759264]  [<ffffffff812ed575>] __wake_up_common+0xe5/0x210
> [ 7891.765679]  [<ffffffff812ed6d9>] __wake_up+0x39/0x50
> [ 7891.771320]  [<ffffffff8135cb3c>] wake_up_klogd_work_func+0x4c/0x60
> [ 7891.778319]  [<ffffffff815b249d>] irq_work_run_list+0xdd/0x1a0
> [ 7891.784829]  [<ffffffff815b31a6>] irq_work_tick+0x166/0x240
> [ 7891.791051]  [<ffffffff813d0682>] update_process_times+0x42/0x60
> [ 7891.797756]  [<ffffffff81409412>] tick_sched_handle+0x92/0x170
> [ 7891.804267]  [<ffffffff8140bf29>] tick_sched_timer+0x39/0xb0
> [ 7891.810584]  [<ffffffff813d3ad7>] __hrtimer_run_queues+0x3d7/0x12e0
> [ 7891.817580]  [<ffffffff813d704c>] ? hrtimer_interrupt+0x16c/0x5b0
> [ 7891.824381]  [<ffffffff8140bef0>] ? tick_sched_do_timer+0x200/0x200
> [ 7891.831378]  [<ffffffff813d3700>] ? enqueue_hrtimer+0x5a0/0x5a0
> [ 7891.837983]  [<ffffffff813d704c>] ? hrtimer_interrupt+0x16c/0x5b0
> [ 7891.844784]  [<ffffffff813d711b>] hrtimer_interrupt+0x23b/0x5b0
> [ 7891.851393]  [<ffffffff810e9f1d>] local_apic_timer_interrupt+0x9d/0x170
> [ 7891.858778]  [<ffffffff82c79880>] smp_apic_timer_interrupt+0x80/0xb0
> [ 7891.865869]  [<ffffffff82c788b3>] apic_timer_interrupt+0x93/0xa0
> [ 7891.872572]  <EOI> [ 7891.874720]  [<ffffffff81646e6d>] ?
> panic+0x354/0x3e1
> [ 7891.880364]  [<ffffffff81646e6a>] ? panic+0x351/0x3e1
> [ 7891.886002]  [<ffffffff81646b19>] ?
> percpu_up_read_preempt_enable.constprop.25+0xb9/0xb9
> [ 7891.895035]  [<ffffffff81068822>] oops_end+0xe2/0xf0
> [ 7891.900574]  [<ffffffff81068e50>] die+0x50/0x80
> [ 7891.905631]  [<ffffffff8105e732>] do_general_protection+0x242/0x430
> [ 7891.912627]  [<ffffffff82c77fb2>] general_protection+0x22/0x30
> [ 7891.919139]  [<ffffffff8253b250>] ? cmos_procfs+0xb0/0x1e0
> [ 7891.925261]  [<ffffffff8253b351>] ? cmos_procfs+0x1b1/0x1e0
> [ 7891.931480]  [<ffffffff8253b1a0>] ? rtc_handler+0x140/0x140
> [ 7891.937700]  [<ffffffff82538580>] rtc_proc_show+0x180/0x640
> [ 7891.943919]  [<ffffffff82538400>] ? rtc_proc_open+0xd0/0xd0
> [ 7891.950139]  [<ffffffff817eb26d>] ? kasan_kmalloc+0xad/0xe0
> [ 7891.956359]  [<ffffffff81901d44>] seq_read+0x334/0x1400
> [ 7891.962192]  [<ffffffff81901a10>] ? seq_hlist_start_percpu+0x4a0/0x4a0
> [ 7891.969477]  [<ffffffff810952fb>] ? save_stack_trace+0x1b/0x20
> [ 7891.975989]  [<ffffffff817eaff6>] ? save_stack+0x46/0xd0
> [ 7891.981918]  [<ffffffff817eb821>] ? kasan_slab_free+0x71/0xb0
> [ 7891.988332]  [<ffffffff817e3fd9>] ? kmem_cache_free+0xe9/0x660
> [ 7891.994842]  [<ffffffff818affe0>] ? putname+0xe0/0x120
> [ 7892.000577]  [<ffffffff81319fb0>] ? print_usage_bug+0x700/0x700
> [ 7892.007185]  [<ffffffff81a27160>] proc_reg_read+0x110/0x270
> [ 7892.013404]  [<ffffffff81870ff6>] __vfs_read+0x106/0x990
> [ 7892.019332]  [<ffffffff81870ef0>] ? do_iter_readv_writev+0x840/0x840
> [ 7892.026426]  [<ffffffff81b35ab5>] ? selinux_file_permission+0x3c5/0x550
> [ 7892.033802]  [<ffffffff81b12ef6>] ? security_file_permission+0x176/0x220
> [ 7892.041282]  [<ffffffff81871ec8>] ? rw_verify_area+0xd8/0x380
> [ 7892.047697]  [<ffffffff81872288>] vfs_read+0x118/0x400
> [ 7892.053432]  [<ffffffff8187710f>] SyS_read+0xdf/0x1d0
> [ 7892.059069]  [<ffffffff81877030>] ? vfs_copy_file_range+0x8f0/0x8f0
> [ 7892.066066]  [<ffffffff81dce253>] ? __this_cpu_preempt_check+0x13/0x20
> [ 7892.073352]  [<ffffffff81877030>] ? vfs_copy_file_range+0x8f0/0x8f0
> [ 7892.080348]  [<ffffffff8100924d>] do_syscall_64+0x19d/0x540
> [ 7892.086567]  [<ffffffff82c76f24>] entry_SYSCALL64_slow_path+0x25/0x25
> [ 7892.093756] ---[ end trace 724fbd49872953c4 ]---

Powered by blists - more mailing lists

Powered by Openwall GNU/*/Linux Powered by OpenVZ