Fixing --as-needed problems

Put this line in spec file:

%define filterout_ld -Wl,--as-needed

But don't use it unless you really need to quickly build a package. Never ever try to commit such a change. It might be tempting to do so but if you think about it –as-needed did it's job well and disabled unneeded libraries. If building fails, it's the package that needs to get fixed, not the linker flags.

With –as-needed enabled, only libraries containing symbols required by object files are linked.

In other words, if you list 10 libraries and only 3 of them contain referenced symbols, others are silently discarded. This is a good thing but unfortunately some projects fail to list all their dependencies properly.

This normally works because if project requires libraries A and B and A itself lists B as a dependency, both A and B get pulled in even if the project itself only lists A.

Now assume that A is a superfluous (unneeded) dependency which gets dropped (or A requiring B is superfluous and B gets dropped from A). The symbols are no longer available.

Linker options are positional, and position is very important. It looks for missing symbols only in libraries coming after object file.

Correct positions are:

$ gcc $(LDFLAGS) -o $@ $(OBJECTS) $(LIBS)

and this generally works well.

If there is something like that:

$ gcc -Wl,--as-needed -l<library> <objects.o>

the library will never be linked, because it's not required at the time it's checked.

And with something like that:

$ gcc -l<library> -Wl,--as-needed <objects.o>

library will allways be linked, even if it isn't required.

Most of problems are with readline/ncurses and tinfo, and it's nice example: tinfo is required by both readline and ncurses, and both are linked with this library (now, when fixed readline is fixed).

But some packages link with readline or ncurses while they use only symbols from tinfo. Without –as-needed those executables work because they are linked with tinfo from readline/ncurses library. With –as-needed linking will not work, because readline/ncurses contain no symbols required by executable so they are not linked, it's dependencies naturally are neither linked. Thats why there is a need to pass -ltinfo. If it requires only symbols from tinfo it's ok to s/ncurses/tinfo/. But if it really requires readline/ncurses but there is some executable (or ./configure) which requires only tinfo both -lreadline/-lncurses and -ltinfo should be passed.

For a longer example take a look at real example nr. 1.

Other common problem is when package produces some shared libraries without linking them with all required libraries, and everything is linked at the end to one binary. So we have:

$ gcc -Wl,--as-needed -o executable <objects.o> -l2 -l1

It's usual situation when object files require only library 2, and -l1 is required by -l2. As I said linker only checks for symbols from objects, so -l1 is not linked. Normally it is easy to fix it, simply make sure while linking -l2 it is linked to -l1 (take a look at the second real example).

Very common situation, xmoto.spec. Actually, it's nothing new. Every linking behaves this way if some library is missing.

With -Wl,–as-needed enabled it stops on something like this:

g++ -Wl,--as-needed -o xmoto-edit BuiltInFont.o [...object files...] Packager.o -lGL -lcurl -lode -llualib50 -llua50 -lSDL_mixer -lbz2 -lz -lpng -ljpeg

VApp.o: In function `vapp::App::getRealTime()':
src/VApp.cpp:287: undefined reference to `SDL_GetTicks'
VApp.o: In function `vapp::App::getTime()':
src/VApp.cpp:284: undefined reference to `SDL_GetTicks'

[... bunch of missing SDL functions ...]

Editor.o: In function `vapp::EditorApp::viewDrawGrid()':
src/Editor.cpp:777: undefined reference to `SDL_GetMouseState'
Editor.o:src/Editor.cpp:46: more undefined references to `SDL_GetMouseState' follow
EditorMain.o: In function `main':
src/EditorMain.cpp:59: undefined reference to `SDL_Quit'

collect2: ld returned 1 exit status
make[1]: *** [xmoto-edit] Error 1

So let's try to find some of missing symbols:

$ grep SDL_GetMouseState /usr/lib64/libSDL*.so
Binary file /usr/lib64/libSDL.so matches

They are in -lSDL, but binary does not link with -lSDL. Edit Makefile by hand and add -lSDL at the same place -lSDL_mixer is: LIBS = -lcurl -lode -llualib50 -llua50 -lSDL_mixer -lSDL -lbz2 -lz -lpng -ljpeg

What we get after running make in build tree:

g++ -Wl,--as-needed -o xmoto-edit BuiltInFont.o [...object files...] Packager.o -lcurl -lode -llualib50 -llua50 -lSDL_mixer -lSDL -lbz2 -lz -lpng -ljpeg

VApp.o: In function `vapp::App::grabScreen()':
src/VApp.cpp:667: undefined reference to `glReadBuffer'
src/VApp.cpp:671: undefined reference to `glReadPixels'

[...more missing gl functions...]

src/Editor.cpp:1280: undefined reference to `glEnable'
src/Editor.cpp:1288: undefined reference to `glDisable'

collect2: ld returned 1 exit status

Same story:

$ grep glEnableClientState /usr/lib64/lib*.so
Binary file /usr/lib64/libGL.so matches

LIBS = -lcurl -lode -llualib50 -llua50 -lSDL_mixer -lSDL -lGL -lbz2 -lz -lpng -ljpeg

But take a look at spec file, -lGL thing was fixed there already:

%{__make} \
        GL_LIBS="-lGL"

Anyway, lets run make, and what we get ?

g++ -Wl,--as-needed -o xmoto-edit BuiltInFont.o [...object files...] Packager.o -lcurl -lode -llualib50 -llua50 -lSDL_mixer -lSDL -lGL -lbz2 -lz -lpng -ljpeg

make[1]: Leaving directory `/home/users/sparky/rpm/BUILD/xmoto-0.2.0'

It worked !

Answer is really easy: libSDL is required by SDL_mixer:

$ ldd /usr/lib64/libSDL_mixer-1.2.so.0.2.4 | grep SDL
        libSDL-1.2.so.0 => /usr/lib64/libSDL-1.2.so.0 (0x00002ab425307000)

but xmoto-edit contains no SDL_mixer symbols, that's why it wasn't linked with SDL_mixer, and there was nothing providing SDL library.

Can you see now why was it stupid to disable –as-needed ? It worked just perfectly disabling unneeded library !

Finally, fix for this may be:

%configure \
        LIBS="-lSDL -lGL"

or patching configure.in, and the result is:

Wrote: /home/users/sparky/rpm/RPMS/xmoto-0.2.0-2.x86_64.rpm

That's the most common and a little more difficult case, evolution-data-server:

Compilation stops at this place:

/bin/sh ../libtool --tag=CC --mode=link gcc -Wl,--as-needed -o test-source-selector  test-source-selector.o libedataserverui-1.2.la ../libedataserver/libedataserver-1.2.la -pthread -lglade-2.0 [...many -l libraries...]
-lgnome-keyring -lpthread

gcc -Wl,--as-needed -o .libs/test-source-selector test-source-selector.o -pthread ./.libs/libedataserverui-1.2.so [...many, many .so and -l libraries...] -lpthread

./.libs/libedataserverui-1.2.so: undefined reference to `glade_xml_new'
./.libs/libedataserverui-1.2.so: undefined reference to `gnome_keyring_find_items_sync'
./.libs/libedataserverui-1.2.so: undefined reference to `gnome_keyring_get_default_keyring_sync'
./.libs/libedataserverui-1.2.so: undefined reference to `gnome_keyring_attribute_list_free'
./.libs/libedataserverui-1.2.so: undefined reference to `glade_xml_get_widget'
./.libs/libedataserverui-1.2.so: undefined reference to `gnome_keyring_create_sync'
./.libs/libedataserverui-1.2.so: undefined reference to `gnome_keyring_item_delete_sync'
./.libs/libedataserverui-1.2.so: undefined reference to `gnome_keyring_item_create_sync'

collect2: ld returned 1 exit status
make[2]: *** [test-source-selector] Error 1

It's different case, now it isn't problem with binary file, but with shared library. And the difficulty is it shows up at executable-linking time, because undefined symbols in libraries are permited.

OK, lets try to fix that library:

$ grep gnome_keyring_item_delete_sync /usr/lib64/lib*.so
Binary file /usr/lib64/libgnome-keyring.so matches
$ grep glade_xml_new /usr/lib64/lib*.so
Binary file /usr/lib64/libglade-2.0.so matches

Editing Makefile we find:

GNOME_KEYRING_LIBS = -lgnome-keyring -lglib-2.0

So add this one and -lglade-2.0 to that library deps:

libedataserverui_1_2_la_LIBADD = \
        $(top_builddir)/addressbook/libebook/libebook-1.2.la    \
        $(GNOME_KEYRING_LIBS) -lglade-2.0 \
        $(E_DATA_SERVER_LIBS)

As it uses libtool, remove .la to force regeneration: $ rm libedataserverui-1.2.la and try to build $ make passes without problems.

Now we only need to fix it correcly, after looking at configure.in and Makefile.am one can see it was only a typo:

libedataserverui_1_2_la_LIBADD = \
        $(top_builddir)/addressbook/libebook/libebook-1.2.la    \
        $(E_DATA_SERVER_UI_LIBS)
                        ^^^

test-source-selector binary was linked with all libraries needed by libedataserverui-1.2.so, and the binary was the one who provided missing symbols to libedataserverui-1.2.so

You can check manually does library has all required symbols, simply running ' gcc library ', like this:

This is an example of correctly linked library:

$ gcc /usr/lib64/liblftp-tasks.so.0.0.0
/usr/lib64/gcc/x86_64-pld-linux/4.1.2/../../../../lib64/crt1.o: In function `_start':
(.text+0x20): undefined reference to `main'
collect2: ld returned 1 exit status

And incorrectly linked one:

$ gcc /usr/lib64/liblftp-jobs.so.0.0.0
/usr/lib64/gcc/x86_64-pld-linux/4.1.2/../../../../lib64/crt1.o: In function `_start':
(.text+0x20): undefined reference to `main'
/usr/lib64/liblftp-jobs.so.0.0.0: undefined reference to `StringSet::Remove(int)'
/usr/lib64/liblftp-jobs.so.0.0.0: undefined reference to `Buffer::Format(char const*, ...)'
[...]
/usr/lib64/liblftp-jobs.so.0.0.0: undefined reference to `SMTask::SuspendSlave()'
collect2: ld returned 1 exit status

Warning: some libraries may require to have unresolved symbols.

Two of most difficult of common problems at once, evolution.spec:

The difficulties are:

1. it appears in configure, what may be difficult to fix and difficult to find what the real problem is
2. the problem is not a missing library, but arguments order

configure stops with such message, which says nothing:

checking if pilot-link handles UTF-8 conversions... no
configure: error: evolution requires pilot-link to have working UTF-8 conversion routines

let's look at config.log:

gcc -o conftest -ggdb -O2 -DORBIT2=1 -pthread -I/usr/include/libgnome-2.0 [...many -I...] -I/usr/include/libxml2 -Wl,--as-needed -pthread -lgpilotd [...many -l...] -lglib-2.0 conftest.c >&5

/home/users/sparky/tmp/ccgrL9ll.o: In function `main':
/home/users/sparky/rpm/BUILD/evolution-2.7.90/conftest.c:64: undefined reference to `convert_ToPilotChar'

collect2: ld returned 1 exit status

function 'convert_ToPilotChar', may be found in passed library, but take a look at section about positions, arguments order is incorrect: libraries go before objects (conftest.c). It's very common when someone puts -l<> in LDFLAGS instead of LIBS, that was the case too.

Just take a look at my fix:

-       LDFLAGS_save="$LDFLAGS"
-       LDFLAGS="$LDFLAGS $GNOME_PILOT_LIBS"
+       LIBS_save="$LIBS"
+       LIBS="$LIBS $GNOME_PILOT_LIBS"
[...]
-       LDFLAGS="$LDFLAGS_save"
+       LIBS="$LIBS_save"

With this simple change everything works perfectly.