[SOLVED] Error loading shared libraries: cannot open shared object file :: on external hardware

Issue

I'm currently developing a C++ application which will reference multiple *.so libraries, each containing code for different machines - written in C. I also have one shared object containing code from a custom namespace Utilities, which (as the name implies) contains basic utilities useful for the application (written in C++, like the rest of the application).

Currently, utilities.so is the only (custom) library referenced by the application. The application compiles and links just fine, however executing it on the target hardware display the following error: bin/updater_v4test: error while loading shared libraries: ../../../bin/device_modules/utilities.so.1.0.0: cannot open shared object file: No such file or directory

When calling LDD on said application, the following output is shown: ../../../bin/device_modules/utilities.so.1.0.0 => not found

I've written a script which (upon make exiting without errors) collects all SOs and pushes them in to the /lib/ directory on the target hardware; meaning that upon each successful build of the application (whether in part or entirely), the updated files are pushed to the correct directory on the target hardware.

In an attempt to mimic other libraries I've used (e.g. zlib), I've attempted to create a symlink to the library file, with no luck (utilities.so.1.0.0 is the symlink, utilities.so is the actual SO). utilities.so.1.0.0 --> utilities.so

At the moment I'm at a loss, and I can't waste much more time in trying to figure this out myself.

---

Below is an excerpt from the utilities Makefile:

include ../../common/user.mk

LIB_DIR=../../../libs/lib/powerpc-linux-gnu
CFLAGS=-Wall -ggdb -I../../libs/include -I../../common -lpthread -lddc -std=c++0x -I../../libs/include/zlib
LDFLAGS=../../libs/lib/powerpc-linux-gnu/libcurl.so.4 ../../libs/include/minizip/.libs/libminizip.so.1.0.0 \
-L ../../libs/lib \
-L ../../libs/lib/powerpc-linux-gnu \
-Wl,-rpath-link,../../libs/lib/powerpc-linux-gnu \
-lrt -lddc -lpthread -shared
BIN_DIR=../bin
CANONICAL_BIN_DIR := $(shell readlink -f $(BIN_DIR))
CANONICAL_CUR_DIR := $(shell readlink -f "./")
COMP_OBJECTS := $(wildcard *.cpp)
OBJECTS := $(patsubst %.cpp,%.o,$(wildcard *.cpp))

all: objects utilities.so
    -cp --parents utilities.so $(GLOBAL_BIN_LIB)
    -ln -s utilities.so utilities.so.1.0.0
    -mv utilities.so.1.0.0 $(GLOBAL_BIN_LIB)
#   -cp --parents *.h $(GLOBAL_HEADER_DIR)
    -rm -f utilities.so
    -rm -f *.o
    for header in $(wildcard *.h); do \
        echo $$header; \
        ln -s $(CANONICAL_CUR_DIR)/$$header $(GLOBAL_HEADER_DIR)/$$header; \
    done;
    @printf "########## BUILT $^ ##########\n\n"

utilities.so: $(OBJECTS)
    ${CXX} $^ -o $@ ${LDFLAGS}

objects: $(COMP_OBJECTS)
    ${CXX} -c $^ ${CFLAGS}

---

And now an excerpt from the application Makefile. Here I've added a reference to the library in the linker flags, the header files are all included in from one directory.

include ../../../common/user.mk

LIB_DIR=../../../libs/lib/powerpc-linux-gnu
CFLAGS=-Wall -ggdb -I../../../libs/include -I${COMMON_DIR} -lpthread -lddc -std=c++0x -I../../../libs/include/zlib -I${HEADER_DIR}
LDFLAGS=../../../libs/lib/powerpc-linux-gnu/libcurl.so.4 ../../../libs/include/minizip/.libs/libminizip.so.1.0.0 ../../../bin/device_modules/utilities.so.1.0.0 \
    -L ../../../libs/lib \
    -L ../../../libs/lib/powerpc-linux-gnu \
    -Wl,-rpath-link,../../../libs/lib/powerpc-linux-gnu \
    -lrt -lddc -lpthread -L ${SO_DIR}
BIN_DIR=../bin
CANONICAL_BIN_DIR := $(shell readlink -f $(BIN_DIR))
CANONICAL_CUR_DIR := $(shell readlink -f "./")

all: test_update.bin
    -cp --parents test_update.bin $(BIN_DIR)
    -ln -s $(CANONICAL_BIN_DIR)/test_update.bin $(GLOBAL_BIN_APP)/test_update.bin
    -rm -f *.bin
    @printf "########## BUILT $^ ##########\n\n"

test_update.bin: main.o updaterdelegate.o commonfunctions.o tinyxml.o
    ${CXX} $^ -o $@ ${LDFLAGS}

####################
#  Required Files  #
####################

main.o: Main.cpp
    ${CXX} -c $^ -o $@ ${CFLAGS}

updaterdelegate.o: UpdaterDelegate.cpp
    ${CXX} -c $^ -o $@ ${CFLAGS}

commonfunctions.o: $(shell python -c "import os.path; print os.path.relpath('${IMPL_CMN_FUNC}'.replace('\"', ''), '${CANONICAL_CUR_DIR}'.replace('\"', ''))")
    ${CXX} -c $^ -o $@ ${CFLAGS}

tinyxml.o: $(shell python -c "import os.path; print os.path.relpath('${IMPL_TXML}'.replace('\"', ''), '${CANONICAL_CUR_DIR}'.replace('\"', ''))")
    ${CXX} -c $^ -o $@ ${CFLAGS}

####################
# /Required Files  #
####################

---

Just for gits and shiggles, I'll add the build output of the utilities.so and application.

utilities.so:

(Clean ommitted)

=============== CLEAN COMPLETE... BUILDING... ===============


make: Entering directory `~/_workspace/upv4/common/utils'
powerpc-linux-gnu-g++  -c ArgumentHandling.cpp Extensions.cpp Logging.cpp -Wall -ggdb -I../../libs/include -I../../common -lpthread -lddc -std=c++0x -I../../libs/include/zlib
ArgumentHandling.cpp: In member function ‘void Utilities::ArgumentHandler::freeMemory()’:
ArgumentHandling.cpp:136: warning: deleting ‘void*’ is undefined
powerpc-linux-gnu-g++  ArgumentHandling.o Extensions.o Logging.o -o utilities.so ../../libs/lib/powerpc-linux-gnu/libcurl.so.4 ../../libs/include/minizip/.libs/libminizip.so.1.0.0 -L ../../libs/lib -L ../../libs/lib/powerpc-linux-gnu -Wl,-rpath-link,../../libs/lib/powerpc-linux-gnu -lrt -lddc -lpthread -shared
cp --parents utilities.so """~/_workspace/upv4""/bin/device_modules"
ln -s utilities.so utilities.so.1.0.0
mv utilities.so.1.0.0 """~/_workspace/upv4""/bin/device_modules"
rm -f utilities.so
rm -f *.o
for header in ArgumentHandling.h Enumerations.h Extensions.h Logging.h; do \
        echo $header; \
        ln -s ~/_workspace/upv4/common/utils/$header """"~/_workspace/upv4""/bin/device_modules"/headers"/$header; \
    done;
ArgumentHandling.h
Enumerations.h
Extensions.h
Logging.h
########## BUILT objects utilities.so ##########

make: Leaving directory `~/_workspace/upv4/common/utils'



=============== BUILD COMPLETE... PARSING... ===============


========== Warnings ==========

Total: 1

ArgumentHandling.cpp:136: warning: deleting ‘void*’ is undefined


========== Errors ==========

Total: 0

test_update.bin:

(Clean ommitted)

=============== CLEAN COMPLETE... BUILDING... ===============


make: Entering directory `~/_workspace/_workspace/upv4/test/app/src'
powerpc-linux-gnu-g++  -c Main.cpp -o main.o -Wall -ggdb -I../../../libs/include -I"""~/_workspace/_workspace/upv4""/common" -lpthread -lddc -std=c++0x -I../../../libs/include/zlib -I"""""~/_workspace/_workspace/upv4""/bin/device_modules"/headers""
powerpc-linux-gnu-g++  -c UpdaterDelegate.cpp -o updaterdelegate.o -Wall -ggdb -I../../../libs/include -I"""~/_workspace/_workspace/upv4""/common" -lpthread -lddc -std=c++0x -I../../../libs/include/zlib -I"""""~/_workspace/_workspace/upv4""/bin/device_modules"/headers""
powerpc-linux-gnu-g++  -c ../../../common/commonFunctions.cpp -o commonfunctions.o -Wall -ggdb -I../../../libs/include -I"""~/_workspace/_workspace/upv4""/common" -lpthread -lddc -std=c++0x -I../../../libs/include/zlib -I"""""~/_workspace/_workspace/upv4""/bin/device_modules"/headers""
powerpc-linux-gnu-g++  -c ../../../common/xmlreader/tinyxml2.cpp -o tinyxml.o -Wall -ggdb -I../../../libs/include -I"""~/_workspace/_workspace/upv4""/common" -lpthread -lddc -std=c++0x -I../../../libs/include/zlib -I"""""~/_workspace/_workspace/upv4""/bin/device_modules"/headers""
powerpc-linux-gnu-g++  main.o updaterdelegate.o commonfunctions.o tinyxml.o -o test_update.bin ../../../libs/lib/powerpc-linux-gnu/libcurl.so.4 ../../../libs/include/minizip/.libs/libminizip.so.1.0.0 ../../../bin/device_modules/utilities.so.1.0.0 -L ../../../libs/lib -L ../../../libs/lib/powerpc-linux-gnu -Wl,-rpath-link,../../../libs/lib/powerpc-linux-gnu -lrt -lddc -lpthread -L """"~/_workspace/_workspace/upv4""/bin/device_modules""
cp --parents test_update.bin ../bin
ln -s ~/_workspace/_workspace/upv4/test/app/bin/test_update.bin """~/_workspace/_workspace/upv4""/bin"/test_update.bin
ln: failed to create symbolic link `~/_workspace/_workspace/upv4/bin/test_update.bin': File exists
make: [all] Error 1 (ignored)
rm -f *.bin
########## BUILT test_update.bin ##########

make: Leaving directory `~/_workspace/_workspace/upv4/test/app/src'



=============== BUILD COMPLETE... PARSING... ===============


========== Warnings ==========

Total: 0




========== Errors ==========

Total: 0

The build script is custom, I built it myself to rid myself of the entire make output when I don't need it, and it parses out all the errors and warnings. Due to the issue, I have turned make output back on.

Did I miss something during compilation/linking of the library, or is something going wrong during the linking of the application? I'm inclined to say it's during compilation/linking of the library - although I don't know what exactly is going wrong, as everything is compiling and linking just fine. Why is the application looking for the .so file in a path it can't possibly have?

I also made sure that the .so files are found with the $PATH variable, so the application should be able to find them.

Solution

The system normally searches for shared objects in a fixed set of directories. You can handle around this by defining the environment variable LD_LIBRARY_PATH and adding the directory where you install the shared libraries into.

Or you can add those libraries to some of the standard libraries directories and execute ldconfig -a to update the cache of shared libraries.

See ldconfig(8), for more info.

edit

There are two mechanisms to load a shared object.

• The first is the normal library loading mechanism, that specifies what has to be loaded at start executable time, and implies linking your executable with the shared objects. This is what you do in your Makefile. You specify the shared executables, and the ld.so.xxx shared object (which is linked to your application when you are dynamically linking it) loads and follows all the unresolved identifiers to find a place for them in the virtual address space. The ld.so.xxx object uses the /etc/ld.so.cache file, that is just a hash table with the directories and shared executables available to be loaded this way. That file is indexed by what is called the soname (which is something useful to allow different versions of the same library to coexist in the same system) and normally maps to the last versioned shared found in the list of directories listed in the file /etc/ld.so.conf (this is static information to accelerate the process of loading libraries and is generated at each boot of the system). If the shared was found at link time (this mechanism is not used by ld, but only to load the libraries at program start time) to have a soname, then that soname is searched in the /etc/ld.so.cache to find the final file that has to be loaded. This cache is built on each boot of the system, so you don't have to cope with that, but only if you don't want to reboot and you install a new library for system use. It is important to note that shared objects must be given sonames for this to work, and that the list of directories in the file /etc/ld.so.conf are the only directories used to search for files. This means that for a standard library to be usable, you need to put it in one of this directories (or add the directory to the list in /etc/ld.so.conf) and then execute ldconfig -a to rebuild the cache and let it include a reference for the file under that soname.

  The other way it to add to the list of search places a list in the form of a PATH variable. The variable is LD_LIBRARY_PATH so, in case you have your shared objects in ${HOME}/libs, you can add this line to your .profile:

  export LD_LIBRARY_PATH=${HOME}/libs
  

  or

  setenv LD_LIBRARY_PATH ${HOME}/libs
  

  this allows your library to live outside of the standard directories, but think twice, as this is a far less efficient way to load a file (as it involves processing a list of directories to look for the final shared objects, while the previous approach is direct, you ask for the file matching the soname that ld.so.xxx asks for, only one search, only one step)

• The second is to use a library function dlopen(3) that allow you to load a shared object and do some housekeeping before calling anything inside. The dl library allows you to search for a symbol in the shared executable and then decide if you interpret it as data or a jump target. dlopen() just opens and loads into the virtual address space a shared object. It resolves the dependencies (if requested to) and is the more flexible way (but it is also nontransparent) to load unknown code for execution. This is the way plugins normally work. You decide in a config file, or dynamically what to load, and then you load it. The program doesn't have to know previously the symbol table of what you are dealing with and you are free to implement whatever you want in the module you are loading.

All of these methods work with ELF binaries, so you have a lot of freedom, but also a lot of complexity on it.

more...

As i've seen from your compilation:

• you are including -l in the compile only commands, the libraries are only needed if you are linking an executable, don't put libraries in the compile phase.

• for a library to be searched and selected by the linker, it has to be named as lib<name>.so (without the version info at the end) so this means that you normally find three names for a standard library (let me use the math library -lm as an example):

  /usr/lib/libm.so.3.2.8 # this is the ELF file with the library contents. /usr/lib/libm.so.3 -> libm.so.3.2.8 # this is the soname used to create the library. /usr/lib/libm.so -> libm.so.3 # this is the actual file the ld(1) program searches for when using -lm.

those links have to be created, as the system normally doesn't. This is part of the installation procedure for a shared library. The soname link allows you to have different versions of a library and detect which one of them will be used at runtime (all must be compatible so you can interchange, when you make an incompatible modification, then you have to change the soname, so the system doesn't get confused on loading)

It is very important to know that the ld(1) program only selects a library if it is called lib<name>.so, with no version info. Indeed, the compiler first searchs for lib<name>.so, then for lib<name>.a, then it complains.

It is very important to put -L places to search for libraries before any -l option that will use those directories in the linking parameters.

You only need to run ldconfig -a and install the library in a system directory if you are not going to use the LD_LIBRARY_PATH mechanism. (this mechanism doesn't work for root account, for obvious reasons :))

Expecting this added comments give some light to the process.

Answered By - Luis Colorado

Answer Checked By - Robin (WPSolving Admin)