netdev - Re: Urgent Bug report: PPPoE ioctl(PPPIOCCONNECT): Transport endpoint is not connected

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives
Hash Suite for Android: free password hash cracker in your pocket
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-Id: <FFD368DF-4C89-494B-8E7B-35C2A139E277@gmail.com>
Date:   Wed, 11 Aug 2021 14:10:32 +0300
From:   Martin Zaharinov <micron10@...il.com>
To:     Pali Rohár <pali@...nel.org>,
        Guillaume Nault <gnault@...hat.com>
Cc:     Greg KH <gregkh@...uxfoundation.org>,
        netdev <netdev@...r.kernel.org>,
        Eric Dumazet <eric.dumazet@...il.com>
Subject: Re: Urgent  Bug report: PPPoE ioctl(PPPIOCCONNECT): Transport
 endpoint is not connected

And one more that see.

Problem is come when accel start finishing sessions,
Now in server have 2k users and restart on one of vlans 3 Olt with 400 users and affect other vlans ,
And problem is start when start destroying dead sessions from vlan with 3 Olt and this affect all other vlans.
May be kernel destroy old session slow and entrained other users by locking other sessions.
is there a way to speed up the closing of stopped/dead sessions.

Martin

> On 9 Aug 2021, at 18:15, Pali Rohár <pali@...nel.org> wrote:
> 
> On Sunday 08 August 2021 18:29:30 Martin Zaharinov wrote:
>> Hi Pali
>> 
>> Kernel 5.13.8
>> 
>> 
>> The problem is from kernel 5.8 > I try all major update 5.9, 5.10, 5.11 ,5.12
>> 
>> I use accel-pppd daemon (not pppd) .
> 
> I'm not using accel-pppd, so cannot help here.
> 
> I would suggest to try "git bisect" kernel version which started to be
> problematic for accel-pppd.
> 
> Providing state of ppp channels and ppp units could help to debug this
> issue, but I'm not sure if accel-pppd has this debug feature. IIRC only
> process which has ppp file descriptors can retrieve and dump this
> information.
> 
>> And yes after users started to connecting .
>> 
>> When system boot and connect first time all user connect without any problem .
>> In time of work user disconnect and connect (power cut , fiber cut or other problem in network) , but in time of spike (may be make lock or other problem ) disconnect ~ 400-500 users  and affect other users. Process go to load over 100% and In statistic I see many finishing connection and many start connection. 
>> And in this time in log get many lines with   ioctl(PPPIOCCONNECT): Transport endpoint is not connected. After finish (unlock or other) stop to see this error and system is back to normal. And connect all disconnected users.
>> 
>> Martin
>> 
>>> On 8 Aug 2021, at 18:23, Pali Rohár <pali@...nel.org> wrote:
>>> 
>>> Hello!
>>> 
>>> On Sunday 08 August 2021 18:14:09 Martin Zaharinov wrote:
>>>> Add Pali Rohár,
>>>> 
>>>> If have any idea .
>>>> 
>>>> Martin
>>>> 
>>>>> On 6 Aug 2021, at 7:40, Greg KH <gregkh@...uxfoundation.org> wrote:
>>>>> 
>>>>> On Thu, Aug 05, 2021 at 11:53:50PM +0300, Martin Zaharinov wrote:
>>>>>> Hi Net dev team
>>>>>> 
>>>>>> 
>>>>>> Please check this error :
>>>>>> Last time I write for this problem : https://www.spinics.net/lists/netdev/msg707513.html
>>>>>> 
>>>>>> But not find any solution.
>>>>>> 
>>>>>> Config of server is : Bonding port channel (LACP)  > Accel PPP server > Huawei switch.
>>>>>> 
>>>>>> Server is work fine users is down/up 500+ users .
>>>>>> But in one moment server make spike and affect other vlans in same server .
>>> 
>>> When this error started to happen? After kernel upgrade? After pppd
>>> upgrade? Or after system upgrade? Or when more users started to
>>> connecting?
>>> 
>>>>>> And in accel I see many row with this error.
>>>>>> 
>>>>>> Is there options to find and fix this bug.
>>>>>> 
>>>>>> With accel team I discus this problem  and they claim it is kernel bug and need to find solution with Kernel dev team.
>>>>>> 
>>>>>> 
>>>>>> [2021-08-05 13:52:05.294] vlan912: 24b205903d09718e: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
>>>>>> [2021-08-05 13:52:05.298] vlan912: 24b205903d097162: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
>>>>>> [2021-08-05 13:52:05.626] vlan641: 24b205903d09711b: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
>>>>>> [2021-08-05 13:52:11.000] vlan912: 24b205903d097105: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
>>>>>> [2021-08-05 13:52:17.852] vlan912: 24b205903d0971ae: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
>>>>>> [2021-08-05 13:52:21.113] vlan641: 24b205903d09715b: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
>>>>>> [2021-08-05 13:52:27.963] vlan912: 24b205903d09718d: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
>>>>>> [2021-08-05 13:52:30.249] vlan496: 24b205903d097184: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
>>>>>> [2021-08-05 13:52:30.992] vlan420: 24b205903d09718a: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
>>>>>> [2021-08-05 13:52:33.937] vlan640: 24b205903d0971cd: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
>>>>>> [2021-08-05 13:52:40.032] vlan912: 24b205903d097182: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
>>>>>> [2021-08-05 13:52:40.420] vlan912: 24b205903d0971d5: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
>>>>>> [2021-08-05 13:52:42.799] vlan912: 24b205903d09713a: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
>>>>>> [2021-08-05 13:52:42.799] vlan614: 24b205903d0971e5: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
>>>>>> [2021-08-05 13:52:43.102] vlan912: 24b205903d097190: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
>>>>>> [2021-08-05 13:52:43.850] vlan479: 24b205903d097153: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
>>>>>> [2021-08-05 13:52:43.850] vlan479: 24b205903d097141: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
>>>>>> [2021-08-05 13:52:43.852] vlan912: 24b205903d097198: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
>>>>>> [2021-08-05 13:52:43.977] vlan637: 24b205903d097148: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
>>>>>> [2021-08-05 13:52:44.528] vlan637: 24b205903d0971c3: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
>>>>> 
>>>>> These are userspace error messages, not kernel messages.
>>>>> 
>>>>> What kernel version are you using?
>>> 
>>> Yes, we need to know, what kernel version are you using.
>>> 
>>>>> thanks,
>>>>> 
>>>>> greg k-h
>>>> 
>>> 
>>> And also another question, what version of pppd daemon are you using?
>>> 
>>> Also, are you able to dump state of ppp channels and ppp units? It is
>>> needed to know to which tty device, file descriptor (or socket
>>> extension) is (or should be) particular ppp channel bounded.
>>