linux-kernel - vm: shmat() address for 32 vs 64 bit kernel

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [thread-next>] [day] [month] [year] [list]

Message-ID: <4981FEBF.5050603@msgid.tls.msk.ru>
Date:	Thu, 29 Jan 2009 22:08:47 +0300
From:	Michael Tokarev <mjt@....msk.ru>
To:	Linux-kernel <linux-kernel@...r.kernel.org>
Subject: vm: shmat() address for 32 vs 64 bit kernel

I were needed to run a legacy application which uses
Oracle 8 database.  Tried to run it on a 64bit kernel,
which is running on my machine for a long time, together
with 32bits userland.  Everything worked but not oracle,
which segfaults at startup on 64bits kernel, but works
just fine when running the same but 32bits kernel
(userspace is the same).

After much debugging I found this difference which may
be relevant.  It's the address returned by shmat() routine.

Here are the relevant stuff for 32bits:

shmget(0x4d21f3bc, 579067904, IPC_CREAT|IPC_EXCL|0640) = 174358532
shmat(174358532, 0, 0)            = 0x9569f000

Here, the segment ends at (0x9569f000+579067904)/1024/1024 = 2942Mb.

And here's what happens with 64bits kernel:

shmget(0xc9a7c840, 579067904, IPC_CREAT|IPC_EXCL|0640) = 294912
shmat(294912, 0, 0)               = 0xd5548000

Here, the segment ends at 3965Mb.

And right after that shmat, oracle process terminates with SIGSEGV
(on 64bits kernel, but continues just fine on 32bits).

I understand full well many bad points here: proprietary software,
old legacy code, not supported, buggy.

But the question really is: why the difference in the first place,
for 32bits userland, between 32 and 64bits kernels?  Can this diff
be "fixed" somehow?  For example, I remember kernel/user memory
split kconfig option - is there similar thing for 32bits userspace
on 64bits kernel?

(And for the record, in case others will encounter the same issue:
the process started it all sees this:

SQL> startup
ORA-24323: value not allowed
ORA-03113: end-of-file on communication channel

and in the alert file we see this:

Errors in file /usr/oracle/8.1/rdbms/log/ora_3365.trc:
ORA-00445: background process "PMON" did not start after 120 seconds
)

Thanks!

/mjt
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/