linux-kernel - [PATCH] USB: serial: cp210x: Fix error 32 when hardware flow control is enabled.

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [thread-next>] [day] [month] [year] [list]

Date:   Tue, 19 Jan 2021 10:42:33 +0000
From:   Pho Tran <Pho.Tran@...abs.com>
To:     "gregkh@...uxfoundation.org" <gregkh@...uxfoundation.org>,
        "johan@...nel.org" <johan@...nel.org>
CC:     "linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>,
        "linux-usb@...r.kernel.org" <linux-usb@...r.kernel.org>,
        Hung Nguyen <Hung.Nguyen@...abs.com>,
        Tung Pham <Tung.Pham@...abs.com>
Subject: [PATCH] USB: serial: cp210x: Fix error 32 when hardware flow control
  is enabled.

Fix error 32 returned by CP210X_SET_MHS when hardware flow control is enabled.

The root cause of error 32 is that user application (CoolTerm, linux-serial-test)
opened cp210x device with hardware flow control then attempt to control RTS/DTR pins.
In hardware flow control, RTS/DTR pins will be controlled by hardware only,
any attempt to control those pins will cause error 32 from the device.
This fix will block MHS command(command to control RTS/DTR pins) to the device
if hardware flow control is being used.

Signed-off-by: Pho Tran <pho.tran@...abs.com>
---
 drivers/usb/serial/cp210x.c | 27 +++++++++++++++++++++++++++
 1 file changed, 27 insertions(+)

diff --git a/drivers/usb/serial/cp210x.c b/drivers/usb/serial/cp210x.c
index fbb10dfc56e3..3694b7c62290 100644
--- a/drivers/usb/serial/cp210x.c
+++ b/drivers/usb/serial/cp210x.c
@@ -1211,6 +1211,12 @@ static int cp210x_tiocmset_port(struct usb_serial_port *port,
 		unsigned int set, unsigned int clear)
 {
 	u16 control = 0;
+	struct cp210x_flow_ctl flow_ctl;
+	u32 ctl_hs = 0;
+	u32 flow_repl = 0;
+	bool auto_dtr = false;
+	bool auto_rts = false;
+	int ret;
 
 	if (set & TIOCM_RTS) {
 		control |= CONTROL_RTS;
@@ -1231,6 +1237,27 @@ static int cp210x_tiocmset_port(struct usb_serial_port *port,
 
 	dev_dbg(&port->dev, "%s - control = 0x%.4x\n", __func__, control);
 
+	// Don't send MHS command if device in hardware flowcontrol mode
+	ret = cp210x_read_reg_block(port, CP210X_GET_FLOW, &flow_ctl, sizeof(flow_ctl));
+	if (ret)
+		return ret;
+
+	ctl_hs = le32_to_cpu(flow_ctl.ulControlHandshake);
+	flow_repl = le32_to_cpu(flow_ctl.ulFlowReplace);
+
+	if (CP210X_SERIAL_DTR_SHIFT(CP210X_SERIAL_DTR_FLOW_CTL) == (ctl_hs & CP210X_SERIAL_DTR_MASK))
+		auto_dtr = true;
+	else
+		auto_dtr = false;
+
+	if (CP210X_SERIAL_RTS_SHIFT(CP210X_SERIAL_RTS_FLOW_CTL) == (flow_repl & CP210X_SERIAL_RTS_MASK))
+		auto_rts = true;
+	else
+		auto_rts = false;
+
+	if (auto_dtr || auto_rts)
+		return 0;
+
 	return cp210x_write_u16_reg(port, CP210X_SET_MHS, control);
 }
 
-- 
2.17.1