Issue
I am attempting to cross compile a package via buildroot. That package can call custom code via a dlopen call to the path of a .so shared library. However, the custom code requires a symbol from the parent executable which is calling dlopen. So the shared library is built with an include -I directive to the directory of the source code of the parent executable, along with -shared -fPIC.
On Debian the parent executable opens the shared library, and according to LD_DEBUG=all finds the symbol and loads the library correctly by searching the path to the parent executable.
On buildroot the parent executable opens the shared library, but fails upon trying to dynamically link the symbol from the executable itself. According to LD_DEBUG=all, there is a symbol lookup error: undefined symbol.
In my testing, I found the debian binary can load the shared library from both the debian AND buildroot systems, running on both the debian AND buildroot system. Therefore the problem most likely lies in the differences between the two parent executables.
I have diffed the readelf -a output (head below) and do not see any true differences between the debian and buildroot binary. Most notably the symbol the buildroot library cannot find is present in both binaries (also below)
ELF Header:
Magic: 7f 45 4c 46 02 01 01 00 00 00 00 00 00 00 00 00
Class: ELF64
Data: 2's complement, little endian
Version: 1 (current)
OS/ABI: UNIX - System V
ABI Version: 0
Type: DYN (Position-Independent Executable file)
Machine: AArch64
Version: 0x1
Entry point address: 0x34a00
Start of program headers: 64 (bytes into file)
Start of section headers: 1933304 (bytes into file)
Flags: 0x0
Size of this header: 64 (bytes)
Size of program headers: 56 (bytes)
Number of program headers: 9
Size of section headers: 64 (bytes)
Number of section headers: 30
Section header string table index: 29
Symbol it fails on:
5964: 0000000000049714 288 FUNC GLOBAL DEFAULT 13 plugin_hook_register
Furthermore, I have tried/checked:
- not stripping the buildroot binary
- building the debian binary with the same flags that buildroot uses
- Changing rpath on the executable
- using the executable from buildroot before the rpath hooks have been called
- Changing
LD_LIBRARY_PATH - Comparing the differences of ldd (of which there are none)
Any ideas on how to further inspect the binaries to determine why this symbol is not able to be found is much appreciated.
Edit:
nm -D output of the symbol for each binary:
Bad binary (GNU nm 2.40, full output) -->
U abort@GLIBC_2.17
U accept@GLIBC_2.17
U access@GLIBC_2.17
U arc4random@GLIBC_2.36
U atof@GLIBC_2.17
U atoi@GLIBC_2.17
U atol@GLIBC_2.17
U atoll@GLIBC_2.17
U backtrace@GLIBC_2.17
U backtrace_symbols@GLIBC_2.17
U bind@GLIBC_2.17
U calloc@GLIBC_2.17
U chdir@GLIBC_2.17
U chmod@GLIBC_2.17
U clearerr@GLIBC_2.17
U clock_gettime@GLIBC_2.17
U close@GLIBC_2.17
U closedir@GLIBC_2.17
U closelog@GLIBC_2.17
U connect@GLIBC_2.17
U __ctype_b_loc@GLIBC_2.17
U __ctype_tolower_loc@GLIBC_2.17
U __ctype_toupper_loc@GLIBC_2.17
w __cxa_finalize@GLIBC_2.17
U dlclose@GLIBC_2.34
U dlerror@GLIBC_2.34
U dlopen@GLIBC_2.34
U dlsym@GLIBC_2.34
U dup2@GLIBC_2.17
U epoll_create1@GLIBC_2.17
U epoll_ctl@GLIBC_2.17
U epoll_wait@GLIBC_2.17
U __errno_location@GLIBC_2.17
U eventfd@GLIBC_2.17
U exit@GLIBC_2.17
U fchmod@GLIBC_2.17
U fchown@GLIBC_2.17
U fclose@GLIBC_2.17
U fcntl64@GLIBC_2.28
U feof@GLIBC_2.17
U ferror@GLIBC_2.17
U fflush@GLIBC_2.17
U fgets@GLIBC_2.17
U fileno@GLIBC_2.17
U fopen64@GLIBC_2.17
U fork@GLIBC_2.17
U fprintf@GLIBC_2.17
U fputc@GLIBC_2.17
U fputs@GLIBC_2.17
U fread@GLIBC_2.17
U free@GLIBC_2.17
U freeaddrinfo@GLIBC_2.17
U fstat64@GLIBC_2.33
U fsync@GLIBC_2.17
U ftruncate64@GLIBC_2.17
U fwrite@GLIBC_2.17
U getaddrinfo@GLIBC_2.17
U __getauxval@GLIBC_2.17
U getc@GLIBC_2.17
U getcwd@GLIBC_2.17
U getenv@GLIBC_2.17
U geteuid@GLIBC_2.17
U getline@GLIBC_2.17
U getpagesize@GLIBC_2.17
U getpeername@GLIBC_2.17
U getpgid@GLIBC_2.17
U getpid@GLIBC_2.17
U getsockname@GLIBC_2.17
U getsockopt@GLIBC_2.17
U gettimeofday@GLIBC_2.17
w __gmon_start__
U isatty@GLIBC_2.17
U __isoc23_sscanf@GLIBC_2.38
U __isoc23_strtol@GLIBC_2.38
U __isoc23_strtoull@GLIBC_2.38
U __isoc99_fscanf@GLIBC_2.17
U __isoc99_sscanf@GLIBC_2.17
w _ITM_deregisterTMCloneTable
w _ITM_registerTMCloneTable
U kill@GLIBC_2.17
U __libc_start_main@GLIBC_2.34
U listen@GLIBC_2.17
U localtime@GLIBC_2.17
U localtime_r@GLIBC_2.17
U lockf64@GLIBC_2.17
U lseek64@GLIBC_2.17
U lstat64@GLIBC_2.33
U malloc@GLIBC_2.17
U memcmp@GLIBC_2.17
U memcpy@GLIBC_2.17
U __memcpy_chk@GLIBC_2.17
U memmove@GLIBC_2.17
U memset@GLIBC_2.17
U __memset_chk@GLIBC_2.17
U mkdir@GLIBC_2.17
U mmap64@GLIBC_2.17
U mremap@GLIBC_2.17
U MsQuicClose@msquic
U MsQuicOpenVersion@msquic
U munmap@GLIBC_2.17
U nanosleep@GLIBC_2.17
U open64@GLIBC_2.17
U opendir@GLIBC_2.17
U openlog@GLIBC_2.17
U perror@GLIBC_2.17
U pipe@GLIBC_2.17
U printf@GLIBC_2.17
U pthread_attr_destroy@GLIBC_2.17
U pthread_attr_init@GLIBC_2.17
U pthread_condattr_destroy@GLIBC_2.17
U pthread_condattr_init@GLIBC_2.17
U pthread_condattr_setclock@GLIBC_2.34
U pthread_cond_broadcast@GLIBC_2.17
U pthread_cond_destroy@GLIBC_2.17
U pthread_cond_init@GLIBC_2.17
U pthread_cond_signal@GLIBC_2.17
U pthread_cond_timedwait@GLIBC_2.17
U pthread_cond_wait@GLIBC_2.17
U pthread_create@GLIBC_2.34
U pthread_join@GLIBC_2.34
U pthread_mutexattr_destroy@GLIBC_2.34
U pthread_mutexattr_init@GLIBC_2.34
U pthread_mutexattr_settype@GLIBC_2.34
U pthread_mutex_destroy@GLIBC_2.17
U pthread_mutex_init@GLIBC_2.17
U pthread_mutex_lock@GLIBC_2.17
U pthread_mutex_trylock@GLIBC_2.34
U pthread_mutex_unlock@GLIBC_2.17
U pthread_rwlock_destroy@GLIBC_2.34
U pthread_rwlock_init@GLIBC_2.34
U pthread_rwlock_rdlock@GLIBC_2.34
U pthread_rwlock_unlock@GLIBC_2.34
U pthread_rwlock_wrlock@GLIBC_2.34
U pthread_self@GLIBC_2.17
U pthread_setname_np@GLIBC_2.34
U pthread_sigmask@GLIBC_2.32
U putchar@GLIBC_2.17
U puts@GLIBC_2.17
U read@GLIBC_2.17
U readdir64@GLIBC_2.17
U readlink@GLIBC_2.17
U readv@GLIBC_2.17
U realloc@GLIBC_2.17
U __register_atfork@GLIBC_2.17
U remove@GLIBC_2.17
U rename@GLIBC_2.17
U rewind@GLIBC_2.17
U rmdir@GLIBC_2.17
U sendmsg@GLIBC_2.17
U setsid@GLIBC_2.17
U setsockopt@GLIBC_2.17
U shutdown@GLIBC_2.17
U sigaction@GLIBC_2.17
U sigaddset@GLIBC_2.17
U sigemptyset@GLIBC_2.17
U sleep@GLIBC_2.17
U snprintf@GLIBC_2.17
U socket@GLIBC_2.17
U socketpair@GLIBC_2.17
U sprintf@GLIBC_2.17
U __sprintf_chk@GLIBC_2.17
U __stack_chk_fail@GLIBC_2.17
U __stack_chk_guard@GLIBC_2.17
U stat64@GLIBC_2.33
U stderr@GLIBC_2.17
U stdin@GLIBC_2.17
U stdout@GLIBC_2.17
U strcasecmp@GLIBC_2.17
U strcat@GLIBC_2.17
U __strcat_chk@GLIBC_2.17
U strchr@GLIBC_2.17
U strcmp@GLIBC_2.17
U strcpy@GLIBC_2.17
U __strcpy_chk@GLIBC_2.17
U strcspn@GLIBC_2.17
U strdup@GLIBC_2.17
U strerror@GLIBC_2.17
U strftime@GLIBC_2.17
U strlcpy@GLIBC_2.38
U strlen@GLIBC_2.17
U strncasecmp@GLIBC_2.17
U strncmp@GLIBC_2.17
U strncpy@GLIBC_2.17
U __strncpy_chk@GLIBC_2.17
U strnlen@GLIBC_2.17
U strrchr@GLIBC_2.17
U strstr@GLIBC_2.17
U strtod@GLIBC_2.17
U strtok@GLIBC_2.17
U strtok_r@GLIBC_2.17
U syscall@GLIBC_2.17
U sysconf@GLIBC_2.17
U system@GLIBC_2.17
U time@GLIBC_2.17
U unlink@GLIBC_2.17
U utimes@GLIBC_2.17
U vfprintf@GLIBC_2.17
U vprintf@GLIBC_2.17
U vsnprintf@GLIBC_2.17
U __vsyslog_chk@GLIBC_2.17
U write@GLIBC_2.17
U writev@GLIBC_2.17
Good binary (GNU nm 2.40, filtered output)
0000000000049714 T plugin_hook_register
They symbol is missing for the buildroot binary, but only from nm and not readelf.
Solution
how to further inspect the binaries to determine why this symbol is not able to be found
The symbol is present in the "regular" symbol table, but not in the dynamic symbol table. Only symbols present in the dynamic symbol table participate in dynamic linking.
A defined symbol that is missing from the dynamic symbol table must have been hidden intentionally (by default, all symbols are exported).
This can happen in a number of ways:
- the symbol may be marked with
__attribute__((visibility("hidden"))or"protected" - the symbol may be hidden in the linker version script
- compile command may have
-fvisibility=hiddenflag
So look for above differences in the two build environments. In other words, compare link and compile command lines, compare linker scripts (if any), compare pre-processed sources. At least one of these must be different.
Answered By - Employed Russian Answer Checked By - Marie Seifert (WPSolving Admin)