Message-ID:
 <MRWPR09MB80220E49BECD4D18E4AEC5C38F4CA@MRWPR09MB8022.eurprd09.prod.outlook.com>
Date: Sun, 6 Jul 2025 13:57:51 +0000
From: Pnina Feder <PNINA.FEDER@...ileye.com>
To: Alexandre Ghiti <alex@...ti.fr>, Björn Töpel
	<bjorn@...nel.org>
CC: Gregory Greenman <Gregory.Greenman@...ileye.com>, "bjorn@...osinc.com"
	<bjorn@...osinc.com>, "linux-kernel@...r.kernel.org"
	<linux-kernel@...r.kernel.org>, "linux-riscv@...ts.infradead.org"
	<linux-riscv@...ts.infradead.org>, "mick@....forth.gr" <mick@....forth.gr>,
	"palmer@...belt.com" <palmer@...belt.com>, "paul.walmsley@...ive.com"
	<paul.walmsley@...ive.com>, Vladimir Kondratiev
	<Vladimir.Kondratiev@...ileye.com>
Subject: RE: [PATCH 0/1] Fix for riscv vmcore issue

>Hi Pnina,
>
>>> Pnina!
>>>
>>> Pnina Feder <pnina.feder@...ileye.com> writes:
>>>
>>>> We are creating a vmcore using kexec on a Linux 6.15 RISC-V system 
>>>> and analyzing it with the crash tool on the host. This workflow used 
>>>> to work on Linux 6.14 but is now broken in 6.15.
>>> Thanks for reporting this!
>>>
>>>> The issue is caused by a change in the kernel:
>>>> In Linux 6.15, certain memblock sections are now marked as Reserved 
>>>> in /proc/iomem. The kexec tool excludes all Reserved regions when 
>>>> generating the vmcore, so these sections are missing from the dump.
>>>> How are you collecting the /proc/vmcore file? A full set of commands would be helpful.
>>>
>> We’ve defined in our system that when a process crashes, we call panic().
>> To handle crash recovery, we're using kexec with the following command:
>> kexec -p /Image --initrd=/rootfs.cpio --append "console=${con} earlycon=${earlycon} no4lvl"
>>
>> To simulate crash, we trigger it using:
>> sleep 100 & kill -6 $!
>>
>> This boots into the crash kernel (kdump), where we then copy the /proc/vmcore file back to the host for analysis.
>>
>>>> However, the kernel still uses addresses in these regions—for 
>>>> example, for IRQ pointers. Since the crash tool needs access to 
>>>> these memory areas to function correctly, their exclusion breaks the analysis.
>>> Wdym with "IRQ pointers"? Also, what version (sha1) of crash are you using?
>>>
>>> We are currently using crash-utility version 9.0.0 (master).
>>>  From the crash analysis logs, we observed errors like:
>>>
>> "......
>> IRQ stack pointer[0] is  ffffffd6fbdcc068
>>> crash: read error: kernel virtual address: ffffffd6fbdcc068  type: "IRQ stack pointer"
>> .....
>>
>>> <read_kdump: addr: ffffffff80edf1cc paddr: 8010df1cc cnt: 4>
>> <readmem: ffffffd6fbdd6880, KVADDR, "runqueues entry (per_cpu)", 3456, 
>> (FOE), 55acf03963e0>
>>> read_kdump: addr: ffffffd6fbdd6880 paddr: 8fbdd6880 cnt: 1920<
>> crash: read error: kernel virtual address: ffffffd6fbdd6880  type: "runqueues entry (per_cpu)"
>
>
>I can't reproduce this issue on qemu, booting with sv39. I'm using the latest kexec-tools (which recently merged riscv support), crash 9.0.0 and kernel 6.16.0-rc4. Note that I'm using crash in qemu.
>
>Are you able to reproduce this on qemu too?

Yes, I am using qemu too on main and crash kernel, with latest kexec-tools, crash 9.0.0 and kernel 6.15


>Maybe that's related to the config, can you share your config?

this is my dev_config 

CONFIG_SYSVIPC=y
CONFIG_POSIX_MQUEUE=y
CONFIG_AUDIT=y
CONFIG_NO_HZ_IDLE=y
CONFIG_HIGH_RES_TIMERS=y
CONFIG_BPF_SYSCALL=y
CONFIG_PREEMPT_RT=y
CONFIG_TASKSTATS=y
CONFIG_TASK_DELAY_ACCT=y
CONFIG_PSI=y
CONFIG_IKCONFIG=y
CONFIG_IKCONFIG_PROC=y
CONFIG_CGROUPS=y
CONFIG_MEMCG=y
CONFIG_CGROUP_SCHED=y
CONFIG_CFS_BANDWIDTH=y
CONFIG_RT_GROUP_SCHED=y
CONFIG_CGROUP_PIDS=y
CONFIG_CGROUP_FREEZER=y
CONFIG_CGROUP_HUGETLB=y
CONFIG_CPUSETS=y
CONFIG_CGROUP_DEVICE=y
CONFIG_CGROUP_CPUACCT=y
CONFIG_CGROUP_PERF=y
CONFIG_CGROUP_BPF=y
CONFIG_NAMESPACES=y
CONFIG_USER_NS=y
CONFIG_CHECKPOINT_RESTORE=y
CONFIG_BLK_DEV_INITRD=y
CONFIG_EXPERT=y
CONFIG_PROFILING=y
CONFIG_KEXEC=y
CONFIG_ARCH_VIRT=y
CONFIG_NONPORTABLE=y
CONFIG_SMP=y
CONFIG_NR_CPUS=32
CONFIG_HZ_1000=y
CONFIG_CPU_IDLE=y
CONFIG_MODULES=y
CONFIG_MODULE_UNLOAD=y
CONFIG_IOSCHED_BFQ=y
CONFIG_PAGE_REPORTING=y
CONFIG_PERCPU_STATS=y
CONFIG_NET=y
CONFIG_PACKET=y
CONFIG_UNIX=y
CONFIG_XFRM_USER=m
CONFIG_INET=y
CONFIG_IP_MULTICAST=y
CONFIG_IP_ADVANCED_ROUTER=y
CONFIG_INET_ESP=m
CONFIG_NETWORK_SECMARK=y
CONFIG_NETFILTER=y
CONFIG_IP_NF_IPTABLES=y
CONFIG_IP_NF_FILTER=y
CONFIG_BRIDGE=m
CONFIG_BRIDGE_VLAN_FILTERING=y
CONFIG_VLAN_8021Q=m
CONFIG_NET_SCHED=y
CONFIG_NET_CLS_CGROUP=m
CONFIG_NETLINK_DIAG=y
CONFIG_NET_L3_MASTER_DEV=y
CONFIG_CGROUP_NET_PRIO=y
CONFIG_FAILOVER=y
CONFIG_DEVTMPFS=y
CONFIG_DEVTMPFS_MOUNT=y
CONFIG_MTD=y
CONFIG_MTD_BLOCK=y
CONFIG_MTD_CFI=y
CONFIG_MTD_CFI_INTELEXT=y
CONFIG_MTD_PHYSMAP=y
CONFIG_MTD_PHYSMAP_OF=y
CONFIG_BLK_DEV_LOOP=y
CONFIG_BLK_DEV_LOOP_MIN_COUNT=0
CONFIG_VIRTIO_BLK=y
CONFIG_SCSI=y
CONFIG_BLK_DEV_SD=y
CONFIG_SCSI_VIRTIO=y
CONFIG_MD=y
CONFIG_BLK_DEV_DM=y
CONFIG_NETDEVICES=y
CONFIG_MACB=y
CONFIG_PCS_XPCS=m
CONFIG_SERIO_LIBPS2=y
CONFIG_VT_HW_CONSOLE_BINDING=y
CONFIG_LEGACY_PTY_COUNT=16
CONFIG_SERIAL_8250=y
CONFIG_SERIAL_8250_CONSOLE=y
CONFIG_SERIAL_OF_PLATFORM=y
CONFIG_SERIAL_EARLYCON_RISCV_SBI=y
CONFIG_VIRTIO_CONSOLE=y
CONFIG_HW_RANDOM=y
CONFIG_HW_RANDOM_VIRTIO=y
CONFIG_I2C=y
CONFIG_I2C_DESIGNWARE_CORE=y
CONFIG_SPI=y
CONFIG_PINCTRL=y
CONFIG_PINCTRL_SINGLE=y
CONFIG_GPIOLIB=y
CONFIG_GPIO_SYSFS=y
CONFIG_GPIO_DWAPB=y
CONFIG_GPIO_SIFIVE=y
CONFIG_POWER_SUPPLY=y
CONFIG_WATCHDOG=y
CONFIG_WATCHDOG_CORE=y
CONFIG_REGULATOR=y
CONFIG_REGULATOR_FIXED_VOLTAGE=y
CONFIG_BACKLIGHT_CLASS_DEVICE=m
CONFIG_SCSI_UFSHCD=y
CONFIG_SCSI_UFSHCD_PLATFORM=y
CONFIG_SCSI_UFS_DWC_TC_PLATFORM=y
CONFIG_RTC_CLASS=y
CONFIG_RTC_DRV_M41T80=y
CONFIG_DMADEVICES=y
CONFIG_SYNC_FILE=y
CONFIG_COMMON_CLK_EYEQ=y
CONFIG_RPMSG_CHAR=y
CONFIG_RPMSG_CTRL=y
CONFIG_RPMSG_VIRTIO=y
CONFIG_RESET_CONTROLLER=y
CONFIG_RESET_SIMPLE=y
CONFIG_GENERIC_PHY=y
CONFIG_EXT4_FS=y
CONFIG_EXT4_FS_POSIX_ACL=y
CONFIG_EXT4_FS_SECURITY=y
CONFIG_MSDOS_FS=y
CONFIG_VFAT_FS=y
CONFIG_TMPFS=y
CONFIG_TMPFS_POSIX_ACL=y
CONFIG_HUGETLBFS=y
CONFIG_KEYS=y
CONFIG_SECURITY=y
CONFIG_SECURITYFS=y
CONFIG_SECURITY_NETWORK=y
CONFIG_SECURITY_PATH=y
CONFIG_CRYPTO_RSA=y
CONFIG_CRYPTO_ECB=y
CONFIG_CRYPTO_BLAKE2B=m
CONFIG_CRYPTO_XXHASH=m
CONFIG_CRYPTO_USER_API_HASH=y
CONFIG_CRC_CCITT=m
CONFIG_CRC_ITU_T=y
CONFIG_CRC7=y
CONFIG_LIBCRC32C=m
CONFIG_PRINTK_TIME=y
CONFIG_DYNAMIC_DEBUG=y
CONFIG_DEBUG_INFO_DWARF5=y
CONFIG_DEBUG_FS=y
CONFIG_DEBUG_PAGEALLOC=y
CONFIG_PTDUMP_DEBUGFS=y
CONFIG_SCHED_STACK_END_CHECK=y
CONFIG_DEBUG_VM=y
CONFIG_DEBUG_VM_PGFLAGS=y
CONFIG_DEBUG_MEMORY_INIT=y
CONFIG_DEBUG_PER_CPU_MAPS=y
CONFIG_SOFTLOCKUP_DETECTOR=y
CONFIG_WQ_WATCHDOG=y
CONFIG_DEBUG_RT_MUTEXES=y
CONFIG_DEBUG_SPINLOCK=y
CONFIG_DEBUG_ATOMIC_SLEEP=y
CONFIG_DEBUG_LIST=y
CONFIG_DEBUG_PLIST=y
CONFIG_DEBUG_SG=y
CONFIG_RCU_EQS_DEBUG=y
CONFIG_MEMTEST=y

>> These failures occur consistently for addresses in the 0xffffffd000000000 region.
>
>
>FYI, this region is the direct mapping (see Documentation/arch/riscv/vm-layout.rst).
>
>Thanks,
>
>Alex
>
>
>> Upon inspection, we confirmed that the physical addresses corresponding to those virtual addresses are not present in the vmcore, as they fall under Reserved memory sections.
>> We tested a patch to kexec-tools that prevents exclusion of the Reserved-memblock section from the vmcore. With this patch, the issue no longer occurs, and crash analysis succeeds.
>> Note: I suspect the same issue exists on ARM64, as both the signal.c and kexec-tools implementations are similar.
>>
>>> Thanks!
>>> Björn

Powered by blists - more mailing lists