[<prev] [next>] [thread-next>] [day] [month] [year] [list]
Message-ID: <Y3YhyMtswrhLy3nt@xpf.sh.intel.com>
Date: Thu, 17 Nov 2022 19:58:00 +0800
From: Pengfei Xu <pengfei.xu@...el.com>
To: <elver@...gle.com>
CC: <linux-kernel@...r.kernel.org>, <pengfei.xu@...el.com>,
<heng.su@...el.com>
Subject: [Syzkaller and bisect] There is "task hung in
perf_event_release_kernel" WARNING in v6.1-rc5 kernel
Hi Marco Elver and kernel developers,
Greeting!
There is "task hung in perf_event_release_kernel" WARNING in v6.1-rc5 kernel
+ below patch:
https://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git/commit/?h=perf/urgent&id=bb88f9695460bec25aa30ba9072595025cf6c8af
[ 300.223812] INFO: task repro:3009 blocked for more than 147 seconds.
[ 300.224414] Not tainted 6.1.0-rc5-kvmperfoverflow+ #10
[ 300.224931] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 300.225626] task:repro state:D stack:0 pid:3009 ppid:492 flags:0x00004002
[ 300.226375] Call Trace:
[ 300.226609] <TASK>
[ 300.226817] __schedule+0x2de/0x8f0
[ 300.227170] ? kmem_cache_free_bulk.part.80+0x651/0x680
[ 300.227682] schedule+0x5b/0xe0
[ 300.227998] schedule_preempt_disabled+0x1c/0x30
[ 300.228443] __mutex_lock.isra.17+0x480/0x710
[ 300.228856] ? __sanitizer_cov_trace_pc+0x25/0x60
[ 300.229302] __mutex_lock_slowpath+0x13/0x20
[ 300.229702] mutex_lock+0x30/0x40
[ 300.230021] perf_event_ctx_lock_nested.isra.103+0xd4/0x190
[ 300.230537] perf_event_release_kernel+0x77/0x370
[ 300.230988] ? __sanitizer_cov_trace_pc+0x25/0x60
[ 300.231437] ? perf_event_release_kernel+0x370/0x370
[ 300.231923] perf_release+0x22/0x30
[ 300.232267] __fput+0x10e/0x440
[ 300.232582] ____fput+0x1e/0x30
[ 300.232894] task_work_run+0xa6/0x100
[ 300.233250] do_exit+0x4d4/0x11f0
[ 300.233580] ? write_comp_data+0x2f/0x90
[ 300.233948] do_group_exit+0x58/0xe0
[ 300.234302] get_signal+0xf07/0xf10
[ 300.234642] arch_do_signal_or_restart+0x3b/0x7d0
[ 300.235080] ? __sanitizer_cov_trace_pc+0x25/0x60
[ 300.235521] ? do_futex+0x1a2/0x2e0
[ 300.235866] ? write_comp_data+0x2f/0x90
[ 300.236239] ? __sanitizer_cov_trace_pc+0x25/0x60
[ 300.236683] exit_to_user_mode_prepare+0xdc/0x180
[ 300.237132] syscall_exit_to_user_mode+0x1f/0x30
[ 300.237582] do_syscall_64+0x4a/0x90
[ 300.237931] entry_SYSCALL_64_after_hwframe+0x63/0xcd
[ 300.238404] RIP: 0033:0x7fa38e38a59d
[ 300.238752] RSP: 002b:00007fa38e242ea8 EFLAGS: 00000246 ORIG_RAX: 00000000000000ca
[ 300.239437] RAX: fffffffffffffe00 RBX: 0000000000000000 RCX: 00007fa38e38a59d
[ 300.240084] RDX: 0000000000000000 RSI: 0000000000000080 RDI: 0000000000405158
[ 300.240720] RBP: 00007fa38e242ec0 R08: 0000000000000000 R09: 0000000000000000
[ 300.241355] R10: 0000000000000000 R11: 0000000000000246 R12: 00007fff34812dce
[ 300.241983] R13: 00007fff34812dcf R14: 00007fff34812e60 R15: 00007fa38e243700
[ 300.242620] </TASK>
[ 300.242832] INFO: task repro:3001 blocked for more than 147 seconds.
[ 300.243402] Not tainted 6.1.0-rc5-kvmperfoverflow+ #10
[ 300.243941] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 300.244641] task:repro state:D stack:0 pid:3001 ppid:496 flags:0x00004004
[ 300.245401] Call Trace:
[ 300.245635] <TASK>
[ 300.245845] __schedule+0x2de/0x8f0
[ 300.246200] schedule+0x5b/0xe0
[ 300.246522] schedule_preempt_disabled+0x1c/0x30
[ 300.246972] __mutex_lock.isra.17+0x480/0x710
[ 300.247395] ? write_comp_data+0x2f/0x90
[ 300.247789] __mutex_lock_slowpath+0x13/0x20
[ 300.248200] mutex_lock+0x30/0x40
[ 300.248525] __do_sys_perf_event_open+0xb46/0x19b0
[ 300.248993] __x64_sys_perf_event_open+0x2f/0x40
[ 300.249441] do_syscall_64+0x3b/0x90
[ 300.249797] entry_SYSCALL_64_after_hwframe+0x63/0xcd
[ 300.250275] RIP: 0033:0x7fa38e38a59d
[ 300.250613] RSP: 002b:00007fa38e263e98 EFLAGS: 00000246 ORIG_RAX: 000000000000012a
[ 300.251298] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007fa38e38a59d
[ 300.251955] RDX: 0000000000000000 RSI: 00000000ffffffff RDI: 0000000020000080
[ 300.252602] RBP: 00007fa38e263ec0 R08: 0000000000000000 R09: 0000000000000000
[ 300.253250] R10: 0000000000000003 R11: 0000000000000246 R12: 00007fff34812dce
[ 300.253901] R13: 00007fff34812dcf R14: 00007fff34812e60 R15: 00007fa38e264700
[ 300.254563] </TASK>
[ 300.254783] INFO: task repro:3016 blocked for more than 147 seconds.
[ 300.255376] Not tainted 6.1.0-rc5-kvmperfoverflow+ #10
[ 300.255912] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 300.256623] task:repro state:D stack:0 pid:3016 ppid:488 flags:0x00004004
[ 300.257394] Call Trace:
[ 300.257632] <TASK>
[ 300.257844] __schedule+0x2de/0x8f0
[ 300.258196] schedule+0x5b/0xe0
[ 300.258522] schedule_preempt_disabled+0x1c/0x30
[ 300.258974] __mutex_lock.isra.17+0x480/0x710
[ 300.259401] __mutex_lock_slowpath+0x13/0x20
[ 300.259825] mutex_lock+0x30/0x40
[ 300.260160] perf_event_ctx_lock_nested.isra.103+0xd4/0x190
[ 300.260696] perf_try_init_event+0x191/0x280
[ 300.261118] perf_event_alloc+0xe30/0x1850
[ 300.261517] ? __fget_light+0xb0/0x220
[ 300.261890] __do_sys_perf_event_open+0x3cb/0x19b0
[ 300.262360] ? __sanitizer_cov_trace_pc+0x25/0x60
[ 300.262822] __x64_sys_perf_event_open+0x2f/0x40
[ 300.263282] do_syscall_64+0x3b/0x90
[ 300.263637] entry_SYSCALL_64_after_hwframe+0x63/0xcd
[ 300.264145] RIP: 0033:0x7fa38e38a59d
[ 300.264496] RSP: 002b:00007fa38e242e98 EFLAGS: 00000246 ORIG_RAX: 000000000000012a
[ 300.265208] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007fa38e38a59d
[ 300.265871] RDX: 0000000000000000 RSI: 00000000ffffffff RDI: 0000000020000000
[ 300.266523] RBP: 00007fa38e242ec0 R08: 0000000000000000 R09: 0000000000000000
[ 300.267177] R10: 0000000000000003 R11: 0000000000000246 R12: 00007fff34812dce
[ 300.267839] R13: 00007fff34812dcf R14: 00007fff34812e60 R15: 00007fa38e243700
[ 300.268501] </TASK>
[ 300.268718] INFO: task repro:3005 blocked for more than 147 seconds.
[ 300.269297] Not tainted 6.1.0-rc5-kvmperfoverflow+ #10
[ 300.269821] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 300.270527] task:repro state:D stack:0 pid:3005 ppid:487 flags:0x00004004
[ 300.271292] Call Trace:
[ 300.271523] <TASK>
[ 300.271741] __schedule+0x2de/0x8f0
[ 300.272096] schedule+0x5b/0xe0
[ 300.272421] schedule_preempt_disabled+0x1c/0x30
[ 300.272873] __mutex_lock.isra.17+0x480/0x710
[ 300.273293] ? __sanitizer_cov_trace_pc+0x25/0x60
[ 300.273741] __mutex_lock_slowpath+0x13/0x20
[ 300.274145] mutex_lock+0x30/0x40
[ 300.274472] __do_sys_perf_event_open+0xb46/0x19b0
[ 300.274943] __x64_sys_perf_event_open+0x2f/0x40
[ 300.275399] do_syscall_64+0x3b/0x90
[ 300.275767] entry_SYSCALL_64_after_hwframe+0x63/0xcd
[ 300.276253] RIP: 0033:0x7fa38e38a59d
[ 300.276603] RSP: 002b:00007fa38e263e98 EFLAGS: 00000246 ORIG_RAX: 000000000000012a
[ 300.277316] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007fa38e38a59d
[ 300.277984] RDX: 0000000000000000 RSI: 00000000ffffffff RDI: 0000000020000000
[ 300.278652] RBP: 00007fa38e263ec0 R08: 0000000000000000 R09: 0000000000000000
[ 300.279327] R10: 0000000000000003 R11: 0000000000000246 R12: 00007fff34812dce
[ 300.279999] R13: 00007fff34812dcf R14: 00007fff34812e60 R15: 00007fa38e264700
[ 300.280676] </TASK>
[ 300.280898] INFO: task repro:3007 blocked for more than 147 seconds.
[ 300.281492] Not tainted 6.1.0-rc5-kvmperfoverflow+ #10
[ 300.282019] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 300.282736] task:repro state:D stack:0 pid:3007 ppid:495 flags:0x00004002
[ 300.283518] Call Trace:
[ 300.283774] <TASK>
[ 300.283988] __schedule+0x2de/0x8f0
[ 300.284351] ? kmem_cache_free_bulk.part.80+0x651/0x680
[ 300.284863] schedule+0x5b/0xe0
[ 300.285193] schedule_preempt_disabled+0x1c/0x30
[ 300.285658] __mutex_lock.isra.17+0x480/0x710
[ 300.286081] ? __sanitizer_cov_trace_pc+0x25/0x60
[ 300.286544] __mutex_lock_slowpath+0x13/0x20
[ 300.286965] mutex_lock+0x30/0x40
[ 300.287310] perf_event_ctx_lock_nested.isra.103+0xd4/0x190
[ 300.287852] perf_event_release_kernel+0x77/0x370
[ 300.288319] ? __sanitizer_cov_trace_pc+0x25/0x60
[ 300.288784] ? perf_event_release_kernel+0x370/0x370
[ 300.289271] perf_release+0x22/0x30
[ 300.289625] __fput+0x10e/0x440
[ 300.289952] ____fput+0x1e/0x30
[ 300.290278] task_work_run+0xa6/0x100
[ 300.290651] do_exit+0x4d4/0x11f0
[ 300.290996] ? write_comp_data+0x2f/0x90
[ 300.291387] do_group_exit+0x58/0xe0
[ 300.291769] get_signal+0xf07/0xf10
[ 300.292122] ? wake_up_q+0x52/0xa0
[ 300.292468] arch_do_signal_or_restart+0x3b/0x7d0
[ 300.292927] ? __sanitizer_cov_trace_pc+0x25/0x60
[ 300.293387] ? do_futex+0x1a2/0x2e0
[ 300.293735] ? write_comp_data+0x2f/0x90
[ 300.294121] ? __sanitizer_cov_trace_pc+0x25/0x60
[ 300.294581] exit_to_user_mode_prepare+0xdc/0x180
[ 300.295049] syscall_exit_to_user_mode+0x1f/0x30
[ 300.295504] do_syscall_64+0x4a/0x90
[ 300.295879] entry_SYSCALL_64_after_hwframe+0x63/0xcd
[ 300.296373] RIP: 0033:0x7fa38e38a59d
[ 300.296723] RSP: 002b:00007fa38e263ea8 EFLAGS: 00000246 ORIG_RAX: 00000000000000ca
[ 300.297431] RAX: fffffffffffffe00 RBX: 0000000000000000 RCX: 00007fa38e38a59d
[ 300.298098] RDX: 0000000000000000 RSI: 0000000000000080 RDI: 0000000000405148
[ 300.298767] RBP: 00007fa38e263ec0 R08: 0000000000000000 R09: 0000000000000000
[ 300.299441] R10: 0000000000000000 R11: 0000000000000246 R12: 00007fff34812dce
[ 300.300128] R13: 00007fff34812dcf R14: 00007fff34812e60 R15: 00007fa38e264700
[ 300.300805] </TASK>
[ 300.301029] INFO: task repro:3018 blocked for more than 147 seconds.
[ 300.301627] Not tainted 6.1.0-rc5-kvmperfoverflow+ #10
[ 300.302163] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 300.302888] task:repro state:D stack:0 pid:3018 ppid:491 flags:0x00004004
[ 300.303693] Call Trace:
[ 300.303936] <TASK>
[ 300.304153] __schedule+0x2de/0x8f0
[ 300.304525] schedule+0x5b/0xe0
[ 300.304860] schedule_preempt_disabled+0x1c/0x30
[ 300.305327] __mutex_lock.isra.17+0x480/0x710
[ 300.305760] ? __sanitizer_cov_trace_pc+0x25/0x60
[ 300.306222] __mutex_lock_slowpath+0x13/0x20
[ 300.306649] mutex_lock+0x30/0x40
[ 300.306989] __do_sys_perf_event_open+0x108f/0x19b0
[ 300.307485] ? write_comp_data+0x2f/0x90
[ 300.307897] __x64_sys_perf_event_open+0x2f/0x40
[ 300.308363] do_syscall_64+0x3b/0x90
[ 300.308733] entry_SYSCALL_64_after_hwframe+0x63/0xcd
[ 300.309233] RIP: 0033:0x7fa38e38a59d
[ 300.309586] RSP: 002b:00007fa38e242e98 EFLAGS: 00000246 ORIG_RAX: 000000000000012a
[ 300.310297] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007fa38e38a59d
[ 300.310967] RDX: 0000000000000000 RSI: 00000000ffffffff RDI: 0000000020000000
[ 300.311511] RBP: 00007fa38e242ec0 R08: 0000000000000000 R09: 0000000000000000
[ 300.311955] R10: ffffffffffffffff R11: 0000000000000246 R12: 00007fff34812dce
[ 300.312415] R13: 00007fff34812dcf R14: 00007fff34812e60 R15: 00007fa38e243700
[ 300.313049] </TASK>
[ 300.313249] INFO: task repro:3015 blocked for more than 147 seconds.
[ 300.313798] Not tainted 6.1.0-rc5-kvmperfoverflow+ #10
[ 300.314277] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 300.314961] task:repro state:D stack:0 pid:3015 ppid:490 flags:0x00000004
[ 300.315679] Call Trace:
[ 300.315903] <TASK>
[ 300.316109] __schedule+0x2de/0x8f0
[ 300.316441] ? raw_spin_rq_lock_nested+0x2d/0x40
[ 300.316873] schedule+0x5b/0xe0
[ 300.317181] schedule_preempt_disabled+0x1c/0x30
[ 300.317616] __mutex_lock.isra.17+0x480/0x710
[ 300.317992] __mutex_lock_slowpath+0x13/0x20
[ 300.318379] mutex_lock+0x30/0x40
[ 300.318672] perf_event_ctx_lock_nested.isra.103+0xd4/0x190
[ 300.319179] perf_try_init_event+0x191/0x280
[ 300.319542] perf_event_alloc+0xe30/0x1850
[ 300.319934] ? __fget_light+0xb0/0x220
[ 300.320294] __do_sys_perf_event_open+0x3cb/0x19b0
[ 300.320740] ? write_comp_data+0x2f/0x90
[ 300.321099] __x64_sys_perf_event_open+0x2f/0x40
[ 300.321500] do_syscall_64+0x3b/0x90
[ 300.321844] entry_SYSCALL_64_after_hwframe+0x63/0xcd
[ 300.322273] RIP: 0033:0x7fa38e38a59d
[ 300.322589] RSP: 002b:00007fa38e263e98 EFLAGS: 00000246 ORIG_RAX: 000000000000012a
[ 300.323240] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007fa38e38a59d
[ 300.323891] RDX: 0000000000000000 RSI: 00000000ffffffff RDI: 0000000020000000
[ 300.324455] RBP: 00007fa38e263ec0 R08: 0000000000000000 R09: 0000000000000000
[ 300.325051] R10: 0000000000000003 R11: 0000000000000246 R12: 00007fff34812dce
[ 300.325629] R13: 00007fff34812dcf R14: 00007fff34812e60 R15: 00007fa38e264700
[ 300.326185] </TASK>
[ 300.326362] INFO: task repro:3021 blocked for more than 147 seconds.
[ 300.326868] Not tainted 6.1.0-rc5-kvmperfoverflow+ #10
[ 300.327315] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 300.327873] task:repro state:D stack:0 pid:3021 ppid:490 flags:0x00000004
[ 300.328537] Call Trace:
[ 300.328744] <TASK>
[ 300.328924] __schedule+0x2de/0x8f0
[ 300.329220] schedule+0x5b/0xe0
[ 300.329505] schedule_preempt_disabled+0x1c/0x30
[ 300.329884] __mutex_lock.isra.17+0x480/0x710
[ 300.330263] ? __sanitizer_cov_trace_pc+0x25/0x60
[ 300.330643] __mutex_lock_slowpath+0x13/0x20
[ 300.331010] mutex_lock+0x30/0x40
[ 300.331291] __do_sys_perf_event_open+0x108f/0x19b0
[ 300.331694] ? __sanitizer_cov_trace_pc+0x25/0x60
[ 300.332069] __x64_sys_perf_event_open+0x2f/0x40
[ 300.332446] do_syscall_64+0x3b/0x90
[ 300.332769] entry_SYSCALL_64_after_hwframe+0x63/0xcd
[ 300.333199] RIP: 0033:0x7fa38e38a59d
[ 300.333496] RSP: 002b:00007fa38e242e98 EFLAGS: 00000246 ORIG_RAX: 000000000000012a
[ 300.334114] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007fa38e38a59d
[ 300.334700] RDX: 0000000000000000 RSI: 00000000ffffffff RDI: 0000000020000080
[ 300.335263] RBP: 00007fa38e242ec0 R08: 0000000000000000 R09: 0000000000000000
[ 300.335847] R10: ffffffffffffffff R11: 0000000000000246 R12: 00007fff34812dce
[ 300.336453] R13: 00007fff34812dcf R14: 00007fff34812e60 R15: 00007fa38e243700
[ 300.337056] </TASK>
[ 300.337244] INFO: task repro:3020 blocked for more than 147 seconds.
[ 300.337749] Not tainted 6.1.0-rc5-kvmperfoverflow+ #10
[ 300.338200] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 300.338816] task:repro state:D stack:0 pid:3020 ppid:493 flags:0x00000004
[ 300.339505] Call Trace:
[ 300.339719] <TASK>
[ 300.339897] __schedule+0x2de/0x8f0
[ 300.340207] schedule+0x5b/0xe0
[ 300.340462] schedule_preempt_disabled+0x1c/0x30
[ 300.340867] __mutex_lock.isra.17+0x480/0x710
[ 300.341238] ? __sanitizer_cov_trace_pc+0x25/0x60
[ 300.341633] __mutex_lock_slowpath+0x13/0x20
[ 300.341979] mutex_lock+0x30/0x40
[ 300.342267] __do_sys_perf_event_open+0x108f/0x19b0
[ 300.342670] ? write_comp_data+0x2f/0x90
[ 300.343014] __x64_sys_perf_event_open+0x2f/0x40
[ 300.343409] do_syscall_64+0x3b/0x90
[ 300.343728] entry_SYSCALL_64_after_hwframe+0x63/0xcd
[ 300.344149] RIP: 0033:0x7fa38e38a59d
[ 300.344448] RSP: 002b:00007fa38e263e98 EFLAGS: 00000246 ORIG_RAX: 000000000000012a
[ 300.345073] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007fa38e38a59d
[ 300.345646] RDX: 0000000000000000 RSI: 00000000ffffffff RDI: 0000000020000000
[ 300.346176] RBP: 00007fa38e263ec0 R08: 0000000000000000 R09: 0000000000000000
[ 300.346745] R10: 00000000ffffffff R11: 0000000000000246 R12: 00007fff34812dce
[ 300.347345] R13: 00007fff34812dcf R14: 00007fff34812e60 R15: 00007fa38e264700
[ 300.347882] </TASK>
[ 300.348077] INFO: task repro:3022 blocked for more than 147 seconds.
[ 300.348606] Not tainted 6.1.0-rc5-kvmperfoverflow+ #10
[ 300.349082] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 300.349681] task:repro state:D stack:0 pid:3022 ppid:493 flags:0x00000004
[ 300.350310] Call Trace:
[ 300.350510] <TASK>
[ 300.350676] __schedule+0x2de/0x8f0
[ 300.350963] schedule+0x5b/0xe0
[ 300.351207] schedule_preempt_disabled+0x1c/0x30
[ 300.351555] __mutex_lock.isra.17+0x480/0x710
[ 300.351941] ? __sanitizer_cov_trace_pc+0x25/0x60
[ 300.352292] __mutex_lock_slowpath+0x13/0x20
[ 300.352618] mutex_lock+0x30/0x40
[ 300.352872] __do_sys_perf_event_open+0x108f/0x19b0
[ 300.353248] ? __sanitizer_cov_trace_pc+0x25/0x60
[ 300.353648] __x64_sys_perf_event_open+0x2f/0x40
[ 300.354019] do_syscall_64+0x3b/0x90
[ 300.354282] entry_SYSCALL_64_after_hwframe+0x63/0xcd
[ 300.354669] RIP: 0033:0x7fa38e38a59d
[ 300.354946] RSP: 002b:00007fa38e242e98 EFLAGS: 00000246 ORIG_RAX: 000000000000012a
[ 300.355508] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007fa38e38a59d
[ 300.356076] RDX: 0000000000000000 RSI: 00000000ffffffff RDI: 0000000020000080
[ 300.356643] RBP: 00007fa38e242ec0 R08: 0000000000000000 R09: 0000000000000000
[ 300.357220] R10: ffffffffffffffff R11: 0000000000000246 R12: 00007fff34812dce
[ 300.357724] R13: 00007fff34812dcf R14: 00007fff34812e60 R15: 00007fa38e243700
[ 300.358270] </TASK>
Bisected and found the first bad commit:
97ba62b278674293762c3d91f724f1bb922f04e0
"perf: Add support for SIGTRAP on perf events"
Reproduced code, kconfig and bisect info is in attached.
All detailed info is in link:
https://github.com/xupengfe/syzkaller_logs/tree/main/221116_021029_perf_event_release_kernel
If it's helpful, please add the "Reported-by" tag from me.
Thanks!
BR.
View attachment "bisect_info.log" of type "text/plain" (4091 bytes)
View attachment "repro.c" of type "text/plain" (15628 bytes)
View attachment "config-guest" of type "text/plain" (278992 bytes)
Powered by blists - more mailing lists