LD_PRELOAD and mixed architectures

[Dave Vitek]

Hi all,

I'm on an amd64 machine that has a mixture of i386 and amd64 binaries
present. I want to build a shared object file and use LD_PRELOAD to
inject it into every process in some process tree. The executables in
the tree could be a mixture of amd64 and i386 executables. Many other
operating systems have a way of using LD_PRELOAD in a mixed architecture
environment, but I'm not seeing a path to victory on netbsd.

I had a look at the source code for ld.so (the elf one anyway). It
doesn't search search LD_LIBRARY_PATH -- it just tries to open() the
filename in LD_PRELOAD relative to CWD and blows up if the architecture
doesn't match the process. Other operating systems facilitate this sort
of thing in various ways:

Solaris: LD_PRELOAD_32 and LD_PRELOAD_64
Linux: Search LD_LIBRARY_PATH, only fail if every attempt fails. Also,
"$LIB" is magical.
Mac OS X: Universal binaries

One (ugly) thought I had is to have the shared object file hook every
function in the "exec" family and fix LD_PRELOAD so it points at the
architecture of the binary about to be exec'd before delegating to the
real exec. Anyone have a less ugly alternative that doesn't require
modifying system files (e.g., adjusting ld.so)?

- Dave

[Gary Duzan]

In Message <***@grammatech.com>,
Dave Vitek <***@grammatech.com>wrote:

=>Hi all,
=>
=>I'm on an amd64 machine that has a mixture of i386 and amd64 binaries
=>present. I want to build a shared object file and use LD_PRELOAD to
=>inject it into every process in some process tree. The executables in
=>the tree could be a mixture of amd64 and i386 executables. Many other
=>operating systems have a way of using LD_PRELOAD in a mixed architecture
=>environment, but I'm not seeing a path to victory on netbsd.
=>
=>I had a look at the source code for ld.so (the elf one anyway). It
=>doesn't search search LD_LIBRARY_PATH -- it just tries to open() the
=>filename in LD_PRELOAD relative to CWD and blows up if the architecture
=>doesn't match the process. Other operating systems facilitate this sort
=>of thing in various ways:
=>
=>Solaris: LD_PRELOAD_32 and LD_PRELOAD_64
=>Linux: Search LD_LIBRARY_PATH, only fail if every attempt fails. Also,
=>"$LIB" is magical.
=>Mac OS X: Universal binaries
=>
=>One (ugly) thought I had is to have the shared object file hook every
=>function in the "exec" family and fix LD_PRELOAD so it points at the
=>architecture of the binary about to be exec'd before delegating to the
=>real exec. Anyone have a less ugly alternative that doesn't require
=>modifying system files (e.g., adjusting ld.so)?

Just a thought, but have you tried putting a 32-bit version of
the library under /emul/netbsd32 ?

Gary Duzan

[Greg Troxel]

Post by Gary Duzan
Just a thought, but have you tried putting a 32-bit version of
the library under /emul/netbsd32 ?

Great idea. Binaries running under emulation will look in
/emul/netbsd32 first (prepended to the file path), so that should work.

[Dave Vitek]

Post by Gary Duzan
=>Hi all,
=>
=>I'm on an amd64 machine that has a mixture of i386 and amd64 binaries
=>present. I want to build a shared object file and use LD_PRELOAD to
=>inject it into every process in some process tree. The executables in
=>the tree could be a mixture of amd64 and i386 executables. Many other
=>operating systems have a way of using LD_PRELOAD in a mixed architecture
=>environment, but I'm not seeing a path to victory on netbsd.
=>
=>I had a look at the source code for ld.so (the elf one anyway). It
=>doesn't search search LD_LIBRARY_PATH -- it just tries to open() the
=>filename in LD_PRELOAD relative to CWD and blows up if the architecture
=>doesn't match the process. Other operating systems facilitate this sort
=>
=>Solaris: LD_PRELOAD_32 and LD_PRELOAD_64
=>Linux: Search LD_LIBRARY_PATH, only fail if every attempt fails. Also,
=>"$LIB" is magical.
=>Mac OS X: Universal binaries
=>
=>One (ugly) thought I had is to have the shared object file hook every
=>function in the "exec" family and fix LD_PRELOAD so it points at the
=>architecture of the binary about to be exec'd before delegating to the
=>real exec. Anyone have a less ugly alternative that doesn't require
=>modifying system files (e.g., adjusting ld.so)?
Just a thought, but have you tried putting a 32-bit version of
the library under /emul/netbsd32 ?
Gary Duzan

Looks like things loaded due to LD_PRELOAD don't search /emul. Here is
a short experiment:

$ echo "int main(){ return 0; }" > nop.c
$ gcc -m32 nop.c -o nop
$ LD_PRELOAD=app/lib64/libhook.so ./nop
app/lib64/libhook.so: unrecognized file format2 [2 != 1]
$ LD_PRELOAD=app/lib/libhook.so ./nop
$ ls /emul/netbsd32/
libhook.so
$ LD_PRELOAD=libhook.so ./nop
Cannot open "libhook.so"
$ gcc -m64 nop.c -o nop
$ LD_PRELOAD=libhook.so ./nop
Cannot open "libhook.so"

It occurs to me that linux emulation processes may have similar challenges.

There is an existing problem report here:
http://gnats.netbsd.org/47509

Relevant code from ld.so_elf confirms that it just does a straight up
open() of the values in the environment variable and doesn't search
anywhere:

// ld_preload is raw value of env variable


if (ld_preload) {
/*
* Pre-load user-specified objects after the main program
* but before any shared object dependencies.
*/
dbg(("preloading objects"));
if (_rtld_preload(ld_preload) == -1)
_rtld_die();
}
...


int
_rtld_preload(const char *preload_path)
{
const char *path;
char *cp, *buf;
int status = 0;

if (preload_path != NULL && *preload_path != '\0') {
cp = buf = xstrdup(preload_path);
while ((path = strsep(&cp, " :")) != NULL && status == 0) {
if (!_rtld_load_object(path, _RTLD_MAIN))
status = -1;
else
dbg((" preloaded \"%s\"", path));
}
xfree(buf);
}

return status;
}


...

Obj_Entry *
_rtld_load_object(const char *filepath, int flags)
{
Obj_Entry *obj;
int fd = -1;
struct stat sb;
size_t pathlen = strlen(filepath);

for (obj = _rtld_objlist->next; obj != NULL; obj = obj->next)
if (pathlen == obj->pathlen && !strcmp(obj->path, filepath))
break;

/*
* If we didn't find a match by pathname, open the file and check
* again by device and inode. This avoids false mismatches caused
* by multiple links or ".." in pathnames.
*
* To avoid a race, we open the file and use fstat() rather than
* using stat().
*/
if (obj == NULL) {
if ((fd = open(filepath, O_RDONLY)) == -1) {
_rtld_error("Cannot open \"%s\"", filepath);
return NULL;
}

[Greg Troxel]

Post by Dave Vitek
Looks like things loaded due to LD_PRELOAD don't search /emul. Here
$ echo "int main(){ return 0; }" > nop.c
$ gcc -m32 nop.c -o nop
$ LD_PRELOAD=app/lib64/libhook.so ./nop
app/lib64/libhook.so: unrecognized file format2 [2 != 1]
$ LD_PRELOAD=app/lib/libhook.so ./nop
$ ls /emul/netbsd32/
libhook.so
$ LD_PRELOAD=libhook.so ./nop
Cannot open "libhook.so"
$ gcc -m64 nop.c -o nop
$ LD_PRELOAD=libhook.so ./nop
Cannot open "libhook.so"
It occurs to me that linux emulation processes may have similar challenges.
http://gnats.netbsd.org/47509
Relevant code from ld.so_elf confirms that it just does a straight up
open() of the values in the environment variable and doesn't search

So:

You are showing something looking for app/lib/libhook.so, and then
failing to find it as libhook.so. That's not a surprise.

I suggest first using absolute paths for libraries. That's the normal
case and more likely to work.

ld.so doesn't do any searching. The process of looking in
/emul/netbsd32 is built into namei and thus works at the system call
level.

Use ktrace to see what's happening with system calls.

[Dave Vitek]

Post by Greg Troxel

Post by Dave Vitek
Looks like things loaded due to LD_PRELOAD don't search /emul. Here
$ echo "int main(){ return 0; }" > nop.c
$ gcc -m32 nop.c -o nop
$ LD_PRELOAD=app/lib64/libhook.so ./nop
app/lib64/libhook.so: unrecognized file format2 [2 != 1]
$ LD_PRELOAD=app/lib/libhook.so ./nop
$ ls /emul/netbsd32/
libhook.so
$ LD_PRELOAD=libhook.so ./nop
Cannot open "libhook.so"
$ gcc -m64 nop.c -o nop
$ LD_PRELOAD=libhook.so ./nop
Cannot open "libhook.so"
It occurs to me that linux emulation processes may have similar challenges.
http://gnats.netbsd.org/47509
Relevant code from ld.so_elf confirms that it just does a straight up
open() of the values in the environment variable and doesn't search

You are showing something looking for app/lib/libhook.so, and then
failing to find it as libhook.so. That's not a surprise.
I suggest first using absolute paths for libraries. That's the normal
case and more likely to work.
ld.so doesn't do any searching. The process of looking in
/emul/netbsd32 is built into namei and thus works at the system call
level.
Use ktrace to see what's happening with system calls.

Thanks Greg; it works. I hadn't appreciated what directory structure
was needed under /emul. I'd still love to find a way that doesn't
require superuser, but this may be good enough for now.

- Dave

[g***@duzan.org]

=> Thanks Greg; it works. I hadn't appreciated what directory structure
=> was needed under /emul. I'd still love to find a way that doesn't
=> require superuser, but this may be good enough for now.

Good to hear. Sorry I didn't explain the requirements in more detail up
front. Good luck...

Gary Duzan