(LinuxWorld) — This is the third article in a series calling for a new installation paradigm for Linux. We have a few more technical issues to address, but it should be useful to start by summarizing the goals so we can keep them in mind as we study the problems. Software installation should meet these minimum requirements:

1. Installation should be easy enough for grandmothers and kids, but flexible enough for professionals to customize as needed.
2. Installation should consume as little time as possible.
3. Installations and updates should not break anything on your system, be it security or the proper functioning of another application.
4. Updates and maintenance should be as automatic and hands-off as possible.
5. The software-installation process should be distribution-agnostic so that the same exact package is as likely to install and run properly on Red Hat, SuSE, Mandrake, Debian or any other Linux distribution.
6. The software-installation process should be distribution version-agnostic so that the same exact package is as likely to install and run properly on Red Hat 9 as it is Red Hat 8.

No doubt some of these goals will be needlessly controversial if they are misunderstood, so let's clarify some points. When I say that installation should consume as little time as possible, I mean exactly that. Some Gentoo fans will no doubt argue that if they can launch an installation and let the process run unattended, then that installation takes very little time. This is perfectly acceptable in some situations, but it doesn't support one of our goals: Granny isn't likely to launch an installation and come back two days later to see if it's finished.

It is also potentially dangerous to make updates and maintenance automatic and hands-off, as any user of Microsoft software knows. However, the fact that one company implements this goal poorly does not invalidate the goal itself. Some Debian users run automatic updates nightly or weekly, and some people who like to live on the edge set up nightly or weekly scripts to download, build and install CVS versions of software. Granny isn't likely to do any of these things, but it would be nice if her software automatically plugged security holes without her intervention.

The Distro Bistro

Numbers 5 and 6 on our requirements list address the most visible problem with existing package-management solutions. There are several reasons why it can be difficult to install a Mandrake package on a Red Hat system or a SuSE package on a Debian system. Even if you get the package installed, the software may not run properly or run at all.

The first level of incompatibility comes at the package level. Mandrake, Red Hat, SuSE and others all use the RPM package format, but this by no means enforces any level of compatibility between the distributions. If you attempt to install a package intended for one distribution on another, the package manager will likely look for another package upon which this particular one depends, and it will complain if you don't have it installed.

When it complains, it most likely needs a shared library provided by that other package. The problem is that whether or not you actually have that shared library on your system is often irrelevant to the package manager. Although it needs the library — not the package that contains it — it looks for the existence of the package, not the library.

Conan the Librarian

We could design a packaging system to search for a library instead of the package that provides it. This raises other issues, however. What if the package manager looks in /lib and /usr/lib and concludes that libsomething.so.6 is missing? Let's assume the package manager solves the problem by installing the needed library in /usr/lib.

What if the library exists in /usr/lib/something or /usr/local/lib, and the package manager simply failed to search these locations? If this is the case, you will have two potentially conflicting libraries on your system. What if the library does not exist anywhere on the system now, but you later install an official package that puts another version of the library in /lib. Now you have two versions of the same library in the two most critical library paths, /lib and /usr/lib, and that's asking for trouble.

In our last installment, we discovered that this is complicated by the fact that applications can search for shared libraries in several ways, depending on many variables: how the application is built, how your system is configured, how the user environment is configured, and others. If you have conflicting libraries on your system, these variables could lead to a system that has no problems at all, a system where some of your applications have mysterious problems or fail, or a system that works fine but simultaneously loads multiple versions of a library when it doesn't need to.

LDD revisited

I was hoping that someone might wonder why I chose to use the ldd command to illustrate points about the way Linux handles shared libraries. I confess I accidentally left out the one clue I had intended to include — one likely to give away the method to my madness.

Here's the missing clue. The optional -r command-line parameter will tell ldd to perform the relocations for data objects and functions. This means it will essentially do a "dry-run" test to see what will happen when the application actually tries to load the shared library. If ldd cannot resolve a symbol, it reports that the symbol is unresolved.

What if we could incorporate ldd into the package manager so that it not only searches out where the application might find the needed libraries, but also pre-tests the link phase to see if the library it finds will supply all the necessary link symbols?

This sounds good, but it wouldn't work in practice, at least not the way ldd is designed today. For one thing, ldd doesn't even attempt to detect runtime plug-ins or library sub-dependencies and test the validity of those symbols, so it only gives you a partial picture of how the linker will resolve symbols when you run the application. For another, ldd is inappropriate for such use, because it introduces serious security risks. Finally, you have to run ldd with the same user privileges and environment variables that will be used at runtime in order for it to simulate the real-world link process accurately. This isn't feasible for the purpose of installing an application all users can access, as different users may have different environments.

Nevertheless, this approach is the one I had hoped readers would guess from the introduction of the topic on ldd, and perhaps if I had remembered to bait the hook, someone might have bit.

Be Gentoo with me

I also had an ulterior motive for mentioning Gentoo Linux earlier. Gentoo uses a system called portage, a derivative of the BSD Ports approach to installing packages. The Gentoo package installer, emerge, automatically resolves dependencies and downloads needed files much like the Debian apt-get system. Gentoo differs in that it doesn't just download binaries, it usually downloads the source code and compiles it on your system, applying any of the custom optimization settings you may choose for your particular machine. It is an effective way to get a finely tuned Linux box running, but it is certainly not the answer to our quest, as noted above.

But one aspect of Gentoo can serve to teach us a very interesting lesson. Gentoo has something called USE, which tells the system that you want it to compile in certain options on a regular basis. To cite the example in the Gentoo documentation, you can set Gentoo to include "gnome" in the USE string, after which it will always assume you want Gnome-aware applications to be compiled with Gnome support.

This dove-tails beautifully with the existing GNU automatic build system, which includes the utilities autoconf, automake and libtool. Combined, these utilities will search your system to see if it includes certain features and libraries. Assuming it finds what it needs, it automatically generates various files to make it possible for you to build the application to use these features. Gentoo's USE feature simply makes sure that you have the correct resources on your system so that autoconf, automake and libtool will find what they need to build in things like Gnome support.

Libtool and libltdl

The autoconf and automake tools are interesting. The functions they provide deserve investigation for use in a new installation paradigm. Libtool is perhaps the most important one to examine, since it specifically deals with shared libraries. Libtool specifically addresses the fact that different flavors of Unix support shared libraries differently, and some do not support shared libraries at all. Libtool attempts to hide these differences so that you can build the same application on different platforms without having to care how that platform handles libraries.

These utilities fall short of our goals, because they exist for an entirely different purpose. They don't exist to make it easy to install software. They make it easy to develop and build applications on differing flavors of Unix, including Linux, without having to address the significant differences in how these flavors provide features, load libraries, link libraries and so on.

One of the most interesting tools in this set is GNU libldtl. You might recall that we discussed a function called dlopen() in the last article. This is the function that dynamically loads a shared library. It has a number of limitations, and it is not universally supported. GNU libldtl is based on the same API (for example, lt_dlopen() functions much the same way as dlopen()), but it smooths out the differences across platforms and provides a number of enhancements to the existing API.

The idea behind libltdl is to insulate the programmer from the details of what library features a Unix system supports and how it supports them. The goal is for programmers who use libltdl to go about their business of using the features of shared libraries with the comfort of knowing their applications should run properly even on those platforms that do not support one or more of those features.

Summary

Hopefully, by now you can begin to see where I've been going with this line of thinking.

Existing GNU utilities can search out and resolve the differences between Unix platforms in order to make it easy to build the same application on these different platforms. One "gotcha" is that the GNU utilities only discover what exists and build the program based on the available features. Gentoo goes one more step and installs the features you want so that these GNU utilities will find and use them at build-time.

The question is not whether one can use specific utilities, such as ldd, autoconf, automake, libtool and libltdl, to reach our installation goals. I don't think that's likely, and it may not even be possible.

The question is much simpler. It goes something like this: It is obviously easier from a technical perspective to resolve compatibility problems across distributions of Linux than it is between entirely different Unix systems — systems that may not even load shared libraries the same way. Given that the GNU utilities accomplish the more difficult goal of cross-platform compatibility, then why should it be at all difficult to design a packaging system that solves the compatibility problems between distributions of the same Unix system, Linux?

I hope to demonstrate in the next installment that it shouldn't be hard at all. We'll apply some of what we have learned from the above utilities to tinker together a pseudo-package system that deals systematically with the potential incompatibilities between distributions and distribution versions, and solves these problems during the installation process. We'll also deal with some of the trade-offs of this system.