Date:   Thu, 6 Sep 2018 11:22:04 +0200
From:   Philipp Eppelt <philipp.eppelt@...nkonzept.com>
To:     linux-kernel@...r.kernel.org
Cc:     Thomas Gleixner <tglx@...utronix.de>
Subject: x86/apic: MSI address malformed for "flat" driver

Hi,

I believe the x86/APIC implementation does not behave according to the
Intel SDM specification, when it comes to composing MSI messages for the
"flat" APIC driver as of a31e58e129f73ab5b04016330b13ed51fde7a961 .


APIC "flat" driver and MSI address composing from the current master
(2018-09-02, 60c1f89241d49bacf71035470684a8d7b4bb46ea):

  static struct apic apic_flat __ro_after_init = {
  ...
	.irq_delivery_mode		= dest_Fixed,
	.irq_dest_mode			= 1, /* logical */
  ...
	.calc_dest_apicid		= apic_flat_calc_apicid,
  };


  static void irq_msi_compose_msg(struct irq_data *data, struct msi_msg
*msg) {
  ...
	msg->address_lo =
		MSI_ADDR_BASE_LO |
		((apic->irq_dest_mode == 0) ?
			MSI_ADDR_DEST_MODE_PHYSICAL :
			MSI_ADDR_DEST_MODE_LOGICAL) |
		MSI_ADDR_REDIRECTION_CPU |
		MSI_ADDR_DEST_ID(cfg->dest_apicid);
  ...
  }

The "flat" driver defines the MSI addressing scheme to be used as
logical addressing in flat mode. The MSI msg address is composed
accordingly, but sets MSI_ADDR_REDIRECTION_CPU which is a zero at bit[3].

The intel SDM states for the MSI address format (SDM vol.3 10.11):
31-20  0xfee
19-12  Destination ID (DID)
11-4   Reserved
3      Redirection Hint (RH)
2      Destination Mode (DM)
1-0    XX

The relation of RH and DM is, if RH is 0, DM is ignored and the DID
field is interpreted the same as are bits [63:56] in the IO-APIC,
meaning as local APIC ID.

If RH is 1 and DM is 0, physical addressing is used (see the
apic_physflat driver).
If RH is 1 and DM is 1, logical addressing is used which splits up into
flat and cluster mode determined by the APICs DFR and LDR using the
logical APIC address.


Currently, irq_msi_compose_msg composes for the "flat" driver an address
like 0xfee0'1004 for a 64-bit single-core system without IO-APIC and MSI
remapping and no ACPI (a virtual system).

That's incorrect because RH == 0 means the DID should show a local APIC
ID, but it shows a logical APIC ID for logical flat addressing (DM == 1,
DFR[31:28] == 0).
The LDR register is correctly set up as well, so the behavior is
consistent, but completely ignores the RH value.

The DID calculation producing the local APIC ID should be done by
"apic_default_calc_apicid", when the RH bit is not set.


That's my analysis I want to put up for discussion.

I hope to have included all necessary information on my setup, please
let me know if I missed something.


I don't have an overview over all affected parts in and around the APIC,
so I am currently not able to produce a patch (besides just changing
.calc_dest_apicid which makes the "flat" driver inconsistent).


Cheers,
Philipp


p.s. I am on vacation the next three weeks starting Saturday, so forgive
me for not answering in the meantime.


Kernel config: x86_64_defconfig
+
CONFIG_KERNEL_XZ=y
CONFIG_BLK_DEV_INITRD=y
CONFIG_CC_STACKPROTECTOR_REGULAR=y
CONFIG_PCI_MSI=y
CONFIG_OF=y
CONFIG_BLK_DEV_RAM=y
CONFIG_BLK_DEV_RAM_SIZE=16384
CONFIG_VIRTIO_BLK=y
CONFIG_SERIO_RAW=y
CONFIG_VT_HW_CONSOLE_BINDING=y
CONFIG_VIRTIO_CONSOLE=y
CONFIG_HW_RANDOM_VIRTIO=y
CONFIG_VIRTIO_PCI=y
# CONFIG_VIRTIO_PCI_LEGACY is not set
CONFIG_VIRTIO_INPUT=y
CONFIG_EXT3_FS=y
CONFIG_TMPFS=y
CONFIG_MESSAGE_LOGLEVEL_DEFAULT=7
CONFIG_DEBUG_FS=y
CONFIG_STACKTRACE=y
CONFIG_MEMTEST=y


virtual System:
x86-64 64-bit UP, 128MB RAM, no ACPI, no MSI remapping, no IO-APIC

-- 
philipp.eppelt@...nkonzept.com - Tel. 0351-41 883 221
http://www.kernkonzept.com

Powered by blists - more mailing lists