Date:   Fri, 1 Feb 2019 12:29:07 +0800
From:   David Chang <dchang@...e.com>
To:     Heiner Kallweit <hkallweit1@...il.com>
Cc:     Peter Ceiley <peter@...ley.net>,
        Realtek linux nic maintainers <nic_swsd@...ltek.com>,
        netdev@...r.kernel.org, Martti Laaksonen <martti.laaksonen@....fi>
Subject: Re: r8169 Driver - Poor Network Performance Since Kernel 4.19

On Jan 31, 2019 at 07:35:30 +0100, Heiner Kallweit wrote:
> Hi David, two more things:
> 
> 1. Could you please test a recent linux-next kernel?

Not tested yet. Will do if possible.

> 2. Please get a register dump (ethtool -d <if>) from 4.18 and 4.19
>    and compare them.

For your informaiton.

[with pcie_aspm=off]
--- v4.18.15	2019-02-01 12:11:56.019051828 +0800
+++ v4.9.11	2019-02-01 12:12:26.827439645 +0800
@@ -3,18 +3,19 @@
 Offset          Values
 ------          ------
 0x0000:         ec 8e b5 5a 2c f5 00 00 48 00 40 00 80 00 80 00
-0x0010:         00 10 38 0e 04 00 00 00 78 00 06 00 00 00 00 00
-0x0020:         00 f0 9b f6 03 00 00 00 00 00 00 00 00 00 00 00
+0x0010:         00 f0 ba 0d 04 00 00 00 78 00 06 00 00 00 00 00
+0x0020:         00 d0 35 f7 03 00 00 00 00 00 00 00 00 00 00 00
 0x0030:         00 00 00 00 00 00 00 0c 00 00 00 00 3f 80 00 00
-0x0040:         80 0f 10 57 0e cf 02 00 00 cf ba 34 00 00 00 00
-0x0050:         10 00 cf 18 60 11 00 01 11 11 11 00 00 00 00 00
+0x0040:         80 0f 10 57 0e cf 02 00 00 d8 c7 50 00 00 00 00
+0x0050:         10 00 cf 98 60 11 01 01 11 11 11 00 00 00 00 00
 0x0060:         00 00 00 00 3c 10 00 81 2c f0 00 80 93 00 80 f0
-0x0070:         00 6f 00 c4 b0 31 00 00 07 00 00 00 00 00 00 00
+0x0070:         00 6f 00 c4 b0 31 00 00 07 00 00 00 00 00 76 d0
 0x0080:         8b 06 01 00 00 00 00 00 00 00 00 00 00 00 00 00
 0x0090:         00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
 0x00a0:         00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
-0x00b0:         7f 04 00 00 00 00 00 00 e1 c1 05 d2 00 00 00 00
+0x00b0:         7f 04 00 00 00 00 00 00 ad 79 01 d2 00 00 00 00
 0x00c0:         00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
 0x00d0:         21 00 00 32 0e 00 00 00 00 00 00 40 06 11 fd 00
-0x00e0:         e1 20 51 51 00 30 94 f6 03 00 00 00 27 00 00 00
+0x00e0:         e1 20 51 51 00 e0 35 f7 03 00 00 00 27 00 00 00
 0x00f0:         3f 00 00 00 00 00 00 00 03 00 00 00 00 00 00 00

[pcie_aspm.policy=performance]
--- v4.18.15-p	2019-02-01 12:18:46.919221060 +0800
+++ v4.9.11-p	2019-02-01 12:19:09.207474824 +0800
@@ -3,18 +3,19 @@
 Offset          Values
 ------          ------
 0x0000:         ec 8e b5 5a 2c f5 00 00 48 00 40 00 80 00 80 00
-0x0010:         00 f0 bc 0d 04 00 00 00 78 00 06 00 00 00 00 00
-0x0020:         00 60 2e f7 03 00 00 00 00 00 00 00 00 00 00 00
+0x0010:         00 c0 22 09 04 00 00 00 78 00 06 00 00 00 00 00
+0x0020:         00 f0 e5 f4 03 00 00 00 00 00 00 00 00 00 00 00
 0x0030:         00 00 00 00 00 00 00 0c 00 00 00 00 3f 80 00 00
-0x0040:         80 0f 10 57 0e cf 02 00 00 53 50 1a 00 00 00 00
-0x0050:         10 00 cf 18 60 11 00 01 11 11 11 00 00 00 00 00
+0x0040:         80 0f 10 57 0e cf 02 00 00 d2 35 7b 00 00 00 00
+0x0050:         10 00 cf 98 60 11 01 01 11 11 11 00 00 00 00 00
 0x0060:         00 00 00 00 3c 10 00 81 2c f0 00 80 93 00 80 f0
-0x0070:         00 6f 00 c4 b0 31 00 00 07 00 00 00 00 00 00 00
+0x0070:         00 6f 00 c4 b0 31 00 00 07 00 00 00 00 00 a4 a0
 0x0080:         8b 06 01 00 00 00 00 00 00 00 00 00 00 00 00 00
 0x0090:         00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
 0x00a0:         00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
-0x00b0:         7f 04 00 00 00 00 00 00 e1 c1 05 d2 00 00 00 00
+0x00b0:         7f 04 00 00 00 00 00 00 ad 79 01 d2 00 00 00 00
 0x00c0:         00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
 0x00d0:         21 00 00 32 0e 00 00 00 00 00 00 40 06 11 fd 00
-0x00e0:         e1 20 51 51 00 70 2e f7 03 00 00 00 27 00 00 00
+0x00e0:         e1 20 51 51 00 00 e6 f4 03 00 00 00 27 00 00 00
 0x00f0:         3f 00 00 00 00 00 00 00 03 00 00 00 00 00 00 00

Thanks,
David

> Heiner
> 
> 
> On 31.01.2019 07:21, Heiner Kallweit wrote:
> > David, thanks for the link to the bug ticket.
> > I think only a proper bisect can help to find the offending commit.
> > 
> > Heiner
> > 
> > 
> > On 31.01.2019 03:32, David Chang wrote:
> >> Hi,
> >>
> >> We had a similr case here.
> >> - Realtek r8169 receive performance regression in kernel 4.19
> >>   https://bugzilla.suse.com/show_bug.cgi?id=1119649
> >>
> >> kernel: r8169 0000:01:00.0 eth0: RTL8168h/8111h, XID 54100880
> >> The major symptom is there are many rx_missed count.
> >>
> >>
> >> On Jan 30, 2019 at 20:15:45 +0100, Heiner Kallweit wrote:
> >>> Hi Peter,
> >>>
> >>> recently I had somebody where pcie_aspm=off for whatever reason didn't
> >>> do the trick, can you also check with pcie_aspm.policy=performance.
> >>
> >> We will give it a try later.
> >>
> >>> And please check with "ethtool -S <if>" whether the chip statistics
> >>> show a significant number of errors.
> >>>
> >>> If this doesn't help you may have to bisect to find the offending commit.
> >>
> >> We had tried fallback driver to a few previous commits as following,
> >> but with no luck.
> >>
> >> 9675931e6b65 r8169: re-enable MSI-X on RTL8168g (v4.19)
> >> 098b01ad9837 r8169: don't include asm headers directly (v4.19-rc1)
> >> a2965f12fde6 r8169: remove rtl8169_set_speed_xmii (v4.19-rc1)
> >> 6fcf9b1d4d6c r8169: fix runtime suspend (v4.19-rc1)
> >> e397286b8e89 r8169: remove TBI 1000BaseX support (v4.19-rc1)
> >>
> >> Thanks,
> >> David Chang
> >>
> >>>
> >>> Heiner
> >>>
> >>>
> >>> On 30.01.2019 10:59, Peter Ceiley wrote:
> >>>> Hi Heiner,
> >>>>
> >>>> I tried disabling the ASPM using the pcie_aspm=off kernel parameter
> >>>> and this made no difference.
> >>>>
> >>>> I tried compiling the 4.18.16 r8169.c with the 4.19.18 source and
> >>>> subsequently loaded the module in the running 4.19.18 kernel. I can
> >>>> confirm that this immediately resolved the issue and access to the NFS
> >>>> shares operated as expected.
> >>>>
> >>>> I presume this means it is an issue with the r8169 driver included in
> >>>> 4.19 onwards?
> >>>>
> >>>> To answer your last questions:
> >>>>
> >>>> Base Board Information
> >>>>     Manufacturer: Alienware
> >>>>     Product Name: 0PGRP5
> >>>>     Version: A02
> >>>>
> >>>> ... and yes, the RTL8168 is the onboard network chip.
> >>>>
> >>>> Regards,
> >>>>
> >>>> Peter.
> >>>>
> >>>> On Tue, 29 Jan 2019 at 17:44, Heiner Kallweit <hkallweit1@...il.com> wrote:
> >>>>>
> >>>>> Hi Peter,
> >>>>>
> >>>>> I think the vendor driver doesn't enable ASPM per default.
> >>>>> So it's worth a try to disable ASPM in the BIOS or via sysfs.
> >>>>> Few older systems seem to have issues with ASPM, what kind of
> >>>>> system / mainboard are you using? The RTL8168 is the onboard
> >>>>> network chip?
> >>>>>
> >>>>> Rgds, Heiner
> >>>>>
> >>>>>
> >>>>> On 29.01.2019 07:20, Peter Ceiley wrote:
> >>>>>> Hi Heiner,
> >>>>>>
> >>>>>> Thanks, I'll do some more testing. It might not be the driver - I
> >>>>>> assumed it was due to the fact that using the r8168 driver 'resolves'
> >>>>>> the issue. I'll see if I can test the r8169.c on top of 4.19 - this is
> >>>>>> a good idea.
> >>>>>>
> >>>>>> Cheers,
> >>>>>>
> >>>>>> Peter.
> >>>>>>
> >>>>>> On Tue, 29 Jan 2019 at 17:16, Heiner Kallweit <hkallweit1@...il.com> wrote:
> >>>>>>>
> >>>>>>> Hi Peter,
> >>>>>>>
> >>>>>>> at a first glance it doesn't look like a typical driver issue.
> >>>>>>> What you could do:
> >>>>>>>
> >>>>>>> - Test the r8169.c from 4.18 on top of 4.19.
> >>>>>>>
> >>>>>>> - Check whether disabling ASPM (/sys/module/pcie_aspm) has an effect.
> >>>>>>>
> >>>>>>> - Bisect between 4.18 and 4.19 to find the offending commit.
> >>>>>>>
> >>>>>>> Any specific reason why you think root cause is in the driver and not
> >>>>>>> elsewhere in the network subsystem?
> >>>>>>>
> >>>>>>> Heiner
> >>>>>>>
> >>>>>>>
> >>>>>>> On 28.01.2019 23:10, Peter Ceiley wrote:
> >>>>>>>> Hi Heiner,
> >>>>>>>>
> >>>>>>>> Thanks for getting back to me.
> >>>>>>>>
> >>>>>>>> No, I don't use jumbo packets.
> >>>>>>>>
> >>>>>>>> Bandwidth is *generally* good, and iperf results to my NAS provide
> >>>>>>>> over 900 Mbits/s in both circumstances. The issue seems to appear when
> >>>>>>>> establishing a connection and is most notable, for example, on my
> >>>>>>>> mounted NFS shares where it takes seconds (up to 10's of seconds on
> >>>>>>>> larger directories) to list the contents of each directory. Once a
> >>>>>>>> transfer begins on a file, I appear to get good bandwidth.
> >>>>>>>>
> >>>>>>>> I'm unsure of the best scientific data to provide you in order to
> >>>>>>>> troubleshoot this issue. Running the following
> >>>>>>>>
> >>>>>>>>     netstat -s |grep retransmitted
> >>>>>>>>
> >>>>>>>> shows a steady increase in retransmitted segments each time I list the
> >>>>>>>> contents of a remote directory, for example, running 'ls' on a
> >>>>>>>> directory containing 345 media files did the following using kernel
> >>>>>>>> 4.19.18:
> >>>>>>>>
> >>>>>>>> increased retransmitted segments by 21 and the 'time' command showed
> >>>>>>>> the following:
> >>>>>>>>     real    0m19.867s
> >>>>>>>>     user    0m0.012s
> >>>>>>>>     sys    0m0.036s
> >>>>>>>>
> >>>>>>>> The same command shows no retransmitted segments running kernel
> >>>>>>>> 4.18.16 and 'time' showed:
> >>>>>>>>     real    0m0.300s
> >>>>>>>>     user    0m0.004s
> >>>>>>>>     sys    0m0.007s
> >>>>>>>>
> >>>>>>>> ifconfig does not show any RX/TX errors nor dropped packets in either case.
> >>>>>>>>
> >>>>>>>> dmesg XID:
> >>>>>>>> [    2.979984] r8169 0000:03:00.0 eth0: RTL8168g/8111g,
> >>>>>>>> f8:b1:56:fe:67:e0, XID 4c000800, IRQ 32
> >>>>>>>>
> >>>>>>>> # lspci -vv
> >>>>>>>> 03:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd.
> >>>>>>>> RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 0c)
> >>>>>>>>     Subsystem: Dell RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller
> >>>>>>>>     Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
> >>>>>>>> ParErr- Stepping- SERR- FastB2B- DisINTx+
> >>>>>>>>     Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
> >>>>>>>> <TAbort- <MAbort- >SERR- <PERR- INTx-
> >>>>>>>>     Latency: 0, Cache Line Size: 64 bytes
> >>>>>>>>     Interrupt: pin A routed to IRQ 19
> >>>>>>>>     Region 0: I/O ports at d000 [size=256]
> >>>>>>>>     Region 2: Memory at f7b00000 (64-bit, non-prefetchable) [size=4K]
> >>>>>>>>     Region 4: Memory at f2100000 (64-bit, prefetchable) [size=16K]
> >>>>>>>>     Capabilities: [40] Power Management version 3
> >>>>>>>>         Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=375mA
> >>>>>>>> PME(D0+,D1+,D2+,D3hot+,D3cold+)
> >>>>>>>>         Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
> >>>>>>>>     Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit+
> >>>>>>>>         Address: 0000000000000000  Data: 0000
> >>>>>>>>     Capabilities: [70] Express (v2) Endpoint, MSI 01
> >>>>>>>>         DevCap:    MaxPayload 128 bytes, PhantFunc 0, Latency L0s
> >>>>>>>> <512ns, L1 <64us
> >>>>>>>>             ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
> >>>>>>>> SlotPowerLimit 10.000W
> >>>>>>>>         DevCtl:    CorrErr- NonFatalErr- FatalErr- UnsupReq-
> >>>>>>>>             RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
> >>>>>>>>             MaxPayload 128 bytes, MaxReadReq 4096 bytes
> >>>>>>>>         DevSta:    CorrErr+ NonFatalErr- FatalErr- UnsupReq- AuxPwr+ TransPend-
> >>>>>>>>         LnkCap:    Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit
> >>>>>>>> Latency L0s unlimited, L1 <64us
> >>>>>>>>             ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
> >>>>>>>>         LnkCtl:    ASPM L1 Enabled; RCB 64 bytes Disabled- CommClk+
> >>>>>>>>             ExtSynch- ClockPM+ AutWidDis- BWInt- AutBWInt-
> >>>>>>>>         LnkSta:    Speed 2.5GT/s (ok), Width x1 (ok)
> >>>>>>>>             TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
> >>>>>>>>         DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR+,
> >>>>>>>> OBFF Via message/WAKE#
> >>>>>>>>              AtomicOpsCap: 32bit- 64bit- 128bitCAS-
> >>>>>>>>         DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR+,
> >>>>>>>> OBFF Disabled
> >>>>>>>>              AtomicOpsCtl: ReqEn-
> >>>>>>>>         LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-
> >>>>>>>>              Transmit Margin: Normal Operating Range,
> >>>>>>>> EnterModifiedCompliance- ComplianceSOS-
> >>>>>>>>              Compliance De-emphasis: -6dB
> >>>>>>>>         LnkSta2: Current De-emphasis Level: -6dB,
> >>>>>>>> EqualizationComplete-, EqualizationPhase1-
> >>>>>>>>              EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
> >>>>>>>>     Capabilities: [b0] MSI-X: Enable+ Count=4 Masked-
> >>>>>>>>         Vector table: BAR=4 offset=00000000
> >>>>>>>>         PBA: BAR=4 offset=00000800
> >>>>>>>>     Capabilities: [d0] Vital Product Data
> >>>>>>>> pcilib: sysfs_read_vpd: read failed: Input/output error
> >>>>>>>>         Not readable
> >>>>>>>>     Capabilities: [100 v1] Advanced Error Reporting
> >>>>>>>>         UESta:    DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
> >>>>>>>> RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
> >>>>>>>>         UEMsk:    DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
> >>>>>>>> RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
> >>>>>>>>         UESvrt:    DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt-
> >>>>>>>> RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
> >>>>>>>>         CESta:    RxErr+ BadTLP+ BadDLLP+ Rollover- Timeout+ AdvNonFatalErr-
> >>>>>>>>         CEMsk:    RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
> >>>>>>>>         AERCap:    First Error Pointer: 00, ECRCGenCap+ ECRCGenEn-
> >>>>>>>> ECRCChkCap+ ECRCChkEn-
> >>>>>>>>             MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
> >>>>>>>>         HeaderLog: 00000000 00000000 00000000 00000000
> >>>>>>>>     Capabilities: [140 v1] Virtual Channel
> >>>>>>>>         Caps:    LPEVC=0 RefClk=100ns PATEntryBits=1
> >>>>>>>>         Arb:    Fixed- WRR32- WRR64- WRR128-
> >>>>>>>>         Ctrl:    ArbSelect=Fixed
> >>>>>>>>         Status:    InProgress-
> >>>>>>>>         VC0:    Caps:    PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
> >>>>>>>>             Arb:    Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
> >>>>>>>>             Ctrl:    Enable+ ID=0 ArbSelect=Fixed TC/VC=01
> >>>>>>>>             Status:    NegoPending- InProgress-
> >>>>>>>>     Capabilities: [160 v1] Device Serial Number 01-00-00-00-68-4c-e0-00
> >>>>>>>>     Capabilities: [170 v1] Latency Tolerance Reporting
> >>>>>>>>         Max snoop latency: 71680ns
> >>>>>>>>         Max no snoop latency: 71680ns
> >>>>>>>>     Kernel driver in use: r8169
> >>>>>>>>     Kernel modules: r8169
> >>>>>>>>
> >>>>>>>> Please let me know if you have any other ideas in terms of testing.
> >>>>>>>>
> >>>>>>>> Thanks!
> >>>>>>>>
> >>>>>>>> Peter.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> On Tue, 29 Jan 2019 at 05:28, Heiner Kallweit <hkallweit1@...il.com> wrote:
> >>>>>>>>>
> >>>>>>>>> On 28.01.2019 12:13, Peter Ceiley wrote:
> >>>>>>>>>> Hi,
> >>>>>>>>>>
> >>>>>>>>>> I have been experiencing very poor network performance since Kernel
> >>>>>>>>>> 4.19 and I'm confident it's related to the r8169 driver.
> >>>>>>>>>>
> >>>>>>>>>> I have no issue with kernel versions 4.18 and prior. I am experiencing
> >>>>>>>>>> this issue in kernels 4.19 and 4.20 (currently running/testing with
> >>>>>>>>>> 4.20.4 & 4.19.18).
> >>>>>>>>>>
> >>>>>>>>>> If someone could guide me in the right direction, I'm happy to help
> >>>>>>>>>> troubleshoot this issue. Note that I have been keeping an eye on one
> >>>>>>>>>> issue related to loading of the PHY driver, however, my symptoms
> >>>>>>>>>> differ in that I still have a network connection. I have attempted to
> >>>>>>>>>> reload the driver on a running system, but this does not improve the
> >>>>>>>>>> situation.
> >>>>>>>>>>
> >>>>>>>>>> Using the proprietary r8168 driver returns my device to proper working order.
> >>>>>>>>>>
> >>>>>>>>>> lshw shows:
> >>>>>>>>>>        description: Ethernet interface
> >>>>>>>>>>        product: RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller
> >>>>>>>>>>        vendor: Realtek Semiconductor Co., Ltd.
> >>>>>>>>>>        physical id: 0
> >>>>>>>>>>        bus info: pci@...0:03:00.0
> >>>>>>>>>>        logical name: enp3s0
> >>>>>>>>>>        version: 0c
> >>>>>>>>>>        serial:
> >>>>>>>>>>        size: 1Gbit/s
> >>>>>>>>>>        capacity: 1Gbit/s
> >>>>>>>>>>        width: 64 bits
> >>>>>>>>>>        clock: 33MHz
> >>>>>>>>>>        capabilities: pm msi pciexpress msix vpd bus_master cap_list
> >>>>>>>>>> ethernet physical tp aui bnc mii fibre 10bt 10bt-fd 100bt 100bt-fd
> >>>>>>>>>> 1000bt-fd autonegotiation
> >>>>>>>>>>        configuration: autonegotiation=on broadcast=yes driver=r8169
> >>>>>>>>>> duplex=full firmware=rtl8168g-2_0.0.1 02/06/13 ip=192.168.1.25
> >>>>>>>>>> latency=0 link=yes multicast=yes port=MII speed=1Gbit/s
> >>>>>>>>>>        resources: irq:19 ioport:d000(size=256)
> >>>>>>>>>> memory:f7b00000-f7b00fff memory:f2100000-f2103fff
> >>>>>>>>>>
> >>>>>>>>>> Kind Regards,
> >>>>>>>>>>
> >>>>>>>>>> Peter.
> >>>>>>>>>>
> >>>>>>>>> Hi Peter,
> >>>>>>>>>
> >>>>>>>>> the description "poor network performance" is quite vague, therefore:
> >>>>>>>>>
> >>>>>>>>> - Can you provide any measurements?
> >>>>>>>>> - iperf results before and after
> >>>>>>>>> - statistics about dropped packets (rx and/or tx)
> >>>>>>>>> - Do you use jumbo packets?
> >>>>>>>>>
> >>>>>>>>> Also help would be a "lspci -vv" output for the network card and
> >>>>>>>>> the dmesg output line with the chip XID.
> >>>>>>>>>
> >>>>>>>>> Heiner
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>
> > 
> 
> 

Powered by blists - more mailing lists