linux-kernel - VT switch broken with docking station DP

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [thread-next>] [day] [month] [year] [list]

Message-ID: <s5h8tqp34co.wl-tiwai@suse.de>
Date:   Thu, 05 Jan 2017 16:37:27 +0100
From:   Takashi Iwai <tiwai@...e.de>
To:     dri-devel@...ts.freedesktop.org
Cc:     linux-kernel@...r.kernel.org
Subject: VT switch broken with docking station DP

Hi,

recently I noticed that VT console doesn't work any longer when I dock
a Dell E7270 laptop with a DP monitor.  The bug detail is like this:

At first, I boot the laptop without dock.  I can switch between X and
VT via ctrl-alt-F1, so far.  Then I dock it to a docking station
connected with a DP monitor.  Now, when I switch to VT, it behaves as
if frozen, the X graphics screen remains.  But actually it's only
graphics and the keyboard input is processed in VT.  I can go back to
X via alt-F7 again.  The situation remains until I undock and I kill X
once.

After looking more deeply at drm debug log, I found out that it's
caused by the drm atomic check.  Essentially, it's because eDP has the
lower resolution (1366x768) than DP (1920x1080).  Since booting with
eDP, the frame buffer size is 1366x768.  Then it hits the following
check in drm_atomic_plane_check():

	fb_width = state->fb->width << 16;
	fb_height = state->fb->height << 16;

	/* Make sure source coordinates are inside the fb. */
	if (state->src_w > fb_width ||
	    state->src_x > fb_width - state->src_w ||
	    state->src_h > fb_height ||
	    state->src_y > fb_height - state->src_h) {
		DRM_DEBUG_ATOMIC("Invalid source coordinates "
				 "%u.%06ux%u.%06u+%u.%06u+%u.%06u\n",
				 state->src_w >> 16, ((state->src_w & 0xffff) * 15625) >> 10,
				 state->src_h >> 16, ((state->src_h & 0xffff) * 15625) >> 10,
				 state->src_x >> 16, ((state->src_x & 0xffff) * 15625) >> 10,
				 state->src_y >> 16, ((state->src_y & 0xffff) * 15625) >> 10);
		return -ENOSPC;
	}

Actually after commenting out "return -ENOSPC", VT switch works fine.

But the code above made me wonder what's the requirement here.  IIRC,
the VT always worked on a display with a higher resolution even if the
frame buffer is smaller.  Only a part of display was used, but it was
OK, far better than the frozen graphics :)

Can we simply drop this check, or may we add a flag to skip it for VT
switching?  Or any better idea?

thanks,

Takashi