Managing and Deploying Software in a Linux Environment

Systems programmers familiar with IBM's mainframe operating systems expect to have certain software installation and maintenance tools, primarily SMP/E and SES. These tools are not available in the Open Source/Linux world, and some of the maintenance practices and concepts they represent are not available either. This paper examines the differences as well as similarities in tools, philosophy, and practices between the Mainframe and Open Source/Linux worlds, and some of the challenges they present.

Package Management Tools

The Open Source community uses a number of software/package management tools. The most commonly used package tool is the Red Hat Package Manager, or RPM, used by most major Linux distributions. Perhaps the second most common one is dpkg (usually in combination with A Package Tool, better known as apt-get), which is unique to the Debian GNU/Linux distribution.

RPM

Robert Young and Marc Ewing founded Red Hat Linux Software in 1995. Red Hat developed RPM for managing the creation and installation of both their source code (SRPMs) and executable code (RPMs). While Red Hat owns the RPM copyright and is the major contributor to RPM development, they encourage others in the Linux community to use RPM. As further incentive, they licensed RPM under the GNU Public License (GPL). Linux developers and vendors began using RPM until it eventually became known more as a community tool than a Red Hat product.

You can find RPM source code and additional information at http://www.rpm.org/. For an in-depth look at RPM and its capabilities, there is an online version of Maximum RPM by Edward C. Bailey, at http://www.rpm.org/max-rpm/.

dpkg and apt-get

Ian Murdock started the Debian GNU/Linux distribution in 1993 and named it for his wife, Deb(ra), and himself, Ian. The dpkg tool was written originally as a shell script. It was re-written a number of times, first in Perl and later in C. The set of programs named apt, described as the next-generation Debian package manager interface, were added in 1999 with Debian version 2.1. Unlike RPM and SRPM packages, which end in .rpm and .src.rpm extensions, Debian package names end with the extension .deb.

How RPM and dpkg differ from SMP/E and SES

RPM and dpkg both provide the essential functionality (and much more) you expect in such tools:

Install packages

Remove packages

Upgrade/downgrade packages

Package installation queries

RPM and dpkg have at least one feature in common that only recently has become available in the mainframe environment: installing software over a network connection. The Linux world has had this capability for quite a long while.

Perhaps the biggest difference between IBM-provided software management tools and RPM and dpkg is tracking granularity. RPM and dpkg track software only at the package level, while SMP/E and SES track software and its maintenance at a much finer level of granularity.

This level of tracking is sometimes referred to as element level. Element-level tracking allows a systems programmer to determine exactly what maintenance last touched a particular object module or what updates are contained in a specific fix. Element-level tracking is not available in either RPM or dpkg and most likely never will be.

How Unix/Linux maintenance philosophy differs from z/OS and z/VM

Because RPM and dpkg don't track software or maintenance at a specific fix or element level, most Linux distributions recommend complete package replacements. For example, to install a particular security patch, you might be advised to upgrade from Samba 2.2.7 to Samba 2.2.8.

One of the challenges of the replacement method is determining what else is going to change with the upgrade. A complete package replacement can cause changes such as new versions of configuration files, new user and group names added to /etc/passwd and /etc/groups, new versions of startup scripts, and so on. This might be more change than you want to introduce at a given time.

Since many Open Source projects provide specific source code patches to current levels of the package, there is an alternative to doing a complete package replacement. You can download the source code patch for your version, apply it, re-compile the affected module, and install it. If you choose this approach, we strongly recommend that you use, or interface with, the package management tool for your distribution to perform the process. For example, use RPM to build a new .rpm file and dpkg to build a new .deb file. Do not simply recompile the source code and copy the updated binary module into place. There are some reasons, discussed later, for not taking this approach.

While most, if not all, Open Source projects use problem-tracking tools, there is nothing that corresponds to an APAR or PTF as used in the z/OS environment. In a Linux environment, unique numbers are not used to identify fixes. The patch file creators/packagers can use whatever names they decide to use. This can make it very difficult to figure out if a particular patch is installed on the same package from different Linux providers. If it is really important to know, you must manually inspect the source package - a time consuming effort.

A key value of Open Source development is that many people have access to and can contribute to the source development. To enable access, many Open Source project owners provide Internet access to their work-in-progress via Concurrent Versioning System (CVS) or similar code management tools. Having access to the most current source provides you with more control over your systems and, hence, your business.

For example, you can retrieve and install a patch yourself rather than waiting for the patch to be delivered via standard Linux vendor channels. Typically, this approach is not possible with proprietary packages. Because commercial software vendors usually do not supply source code, you have less control over when and how patches are applied.

How Unix/Linux maintenance philosophy is similar to z/OS or z/VM

Because source code is readily available for open source packages and many IS installations need to maintain current software levels, there is a tendency to download the current source, compile the package, and install it. Taking this approach without using the appropriate package management tool for that platform may cause severe problems.

In many ways, this is just as bad as zapping modules or updating source code files outside the control of SMP/E or SES. You have no record of what was changed. Others working with the system may have no idea that the current system is not reflected accurately in the RPM or dpkg database. They might install a package that conflicts with or partially replaces the improperly installed package. If the improperly installed package partially replaced one that was in the RPM or dpkg database and that package is deleted, your Linux system may be disabled.

The list of drawbacks can get quite long. The best practice is to use the package tool that came with your Linux distribution or a recognized tool from a software vendor. If your Linux distribution has a package management tool, or the software vendor provides a product-specific tool, always use it. This may be inconvenient forcing you to learn how to create .rpm or .debs packages, but may save you hundreds of debugging hours.

We don't recommend that you create .rpm or .deb packages for commercial software products not provided in one of those formats. Having the appropriate product registration and files is very important. But performing the packaging yourself is a never-ending, tedious, and error-prone task that is best left to the software vendor.

The Unix/Linux community has long recognized the importance of keeping systems up, running, and applications available. As a result, you can perform many maintenance activities while keeping systems and applications online. You can bring down most system services, put updated versions in place, and bring the services back up without affecting the rest of the system. If the change involves only parameter files and not software, most services can reload their new parameters without interrupting execution. When dealing with components such as shells, editors, compilers, development tools, there is no service to bring down. Having to re-IPL a system because of maintenance is rare and usually the result of replacing the Linux kernel.

It is also critical to keep systems current, particularly in the area of security fixes. Because of the nature of Open Source, developers and vendors usually develop fixes for security exposures quickly and distribute them immediately. Trying to keep a security problem secret is futile when the entire world can see the source code.

Because the Open Source community does not, generally, hide security bugs, it may appear as though Linux and other Open Source packages have more security problems than their proprietary equivalents. Supporters of Open Source development say that it only appears that way. All software has problems, but you can't hide or deny problems in an Open Source environment. Regardless of which position you take, the Open Source community generally makes bugs known and distributes a fix as quickly as possible.

Experience has taught most mainframe shops not to install locally developed software in places like SYS1.LINKLIB (a common MVS location for system files) or on the 190 disks (a common VM/CMS location for system files). It is simpler to manage local software if you install it someplace else.

Similarly, when you install non-distribution Linux software, you should not put it in the same place(s) as the distribution software (most frequently /usr). According to the Filesystem Hierarchy Standard1 (FHS), the appropriate directory for add-on, or local, software is /opt or /usr/local. Reviewing the FHS gives you tremendous insight into the Unix/Linux file system and explains the rationale for putting discrete software components in specific, well-defined places.

Tool Limitations

Neither RPM nor dpkg can identify if a particular patch has been installed on a package or not. To determine if a specific patch is installed, first identify the revision level of the software package that includes the patch. Then, query the RPM or dpkg database to see if the package is at that level or higher. Tracking this for a small number of packages is time consuming. Doing it for tens or hundreds of packages is nearly impossible.

RPM checks available file system space before installing a package. Upgrading a package may not increase net disk space because many files are being replaced and only a few new files added.

However, RPM doesn't know this prior to starting the upgrade and requires enough free disk space to replicate the package. If you do not have enough file system free space available for RPM to duplicate the package, it may refuse to install a package upgrade. This can be frustrating, but reminds us of the importance of managing disk space efficiently.

Keep in mind that the checks RPM performs prior to installing are only queries to the RPM database. RPM is not searching your disks to determine what software is actually installed. This RPM design may cause RPM to indicate a package is not installed when, in reality, it is. The RPM database may not have a package installation record because someone may have installed it without using the RPM facility. Or the package might have been installed with RPM, but it may have been installed from a distribution that did not specify the file in question was being provided. Yet additional reasons to make sure you use RPM to install everything you possibly can and don't mix distributions.

Similar to pre-requisite and co-requisite chains that can make it difficult to install a particular z/OS fix, figuring out long package dependencies (and package conflicts) may be very frustrating. The apt-get function in Debian provides relief in this area. Apt get iteratively queries the local package database and the remote package servers to determine which additional packages and their dependencies you need to download and install to satisfy the current installation request. RPM does not have an equivalent - function, but Conectiva Linux has produced an APT-RPM package with similar capabilities. Most Linux distributions do not include APT-RPM, so if you want to use it, you must download and build APT-RPM for your specific Linux distribution.

Sharing Linux/390 File Systems Across z/VM Guests

Why you want to share file systems

If you are going to create a significant number of Linux systems (more than ten or so) in a z/VM environment, you'll waste a lot of disk space due to duplication of files. If some Linux systems are running the same version of a particular package, why maintain multiple copies of the same files? Maintaining separate copies of files also means you must apply patches or new versions on each system.

Sharing file systems in read-only mode using z/VM's minidisk sharing capabilities reduces software maintenance effort and saves disk space. You can also use VDISK (virtual disk in storage) for shared file systems. While VDISK puts more pressure on the storage and paging subsystems, it provides extremely high performance file access.

Why it's difficult

Linux/390 systems can share z/VM minidisks exclusively in read-only mode. If a system has a mini-disk mounted as read-write, the other Linux systems sharing the mini-disk view the file system as “dirty” when mounting it. The Linux system attempting to mount the read-write file system displays a warning message.

In some cases, a sharing Linux system tries to “fsck” the shared file system and generates I/O errors because it cannot write to the disk. Also, due to aggressive file caching by Linux, the sharing systems may not see changes to the disk by the “owning” system immediately. Therefore, it is critical for shared file systems to be mounted as read-only for all Linux systems.

This also means that the system that creates/updates the file system cannot ever be mounted read-write while other systems are reading the file system. This requires at least two copies of the file system - one for read-only use and a one that can be updated and eventually swapped with the in-use read-only file system. To mount a file system as read-only, you must specify read-only in three places:

The LINK entry in the CP directory for the Linux/390 guest

The kernel parameters for the DASD driver

The /etc/fstab file for the guest, including the parameter that tells the system not to fsck it at boot time.

Having to make the changes in so many places increases the chance that someone will fail to make one of them as changes are made in the future. While the only change that really matters is the CP directory update, missing one or both of the other two changes generates error messages that can be disconcerting.

Overcoming potential problems

RPM and dpkg database files are on the /var file system which is not typically shareable. Since it is unlikely that the software on most of your Linux/390 systems is identical, it is not feasible to share the RPM or dpkg database in the /var file system.

However, if you have an environment where you can share the RPM or dpkg database, there are a few steps you must take to make this happen. Move the package database to a part of the file system that is shared. Then, create symbolic links that point to the actual database file location. This equals a local usermod that must be maintained whenever you install a new version of RPM or Linux.

Maintaining packages when sharing

If you do share package databases, how can you add packages to just one of the systems that is sharing /usr? Many RPM packages put files into the /usr hierarchy, which is a problem if you want to install a package on just one system and /usr is read-only. Other components will probably go into /etc, which isn't shared. All of this creates additional challenges you need to address.

Possible work-arounds

Using NFS over a z/VM Guest LAN is one option. By defining a number of system templates within NFS, you can mount the templates needed for a specific Linux system. Basically, you have established a shared file system, but via NFS, and not by defining and mounting multiple read-only file systems.

This approach may increase the number of times you need to install and maintain products. If the number of templates is small, this approach can result in significant savings in people time compared to using real servers and read-only files.

Guest LANs provide a zero latency virtual network, so performance is typically very good. While this doesn't address the need to install local packages on particular systems, it does eliminate the need to declare read-only file system attributes in so many different places. To eliminate a single point of failure, consider two or more different Linux/390 guests acting as NFS servers in a virtual High Availability configuration within one z/VM system.

What can be shared

The /usr hierarchy is frequently cited as the best candidate for sharing. It is typically the largest part of the file system, so sharing it provides the most savings in disk space. /opt normally is much smaller, but is also a reasonable candidate for sharing.

Note

For file systems that are going to be shared, and therefore mounted read-only, it makes little sense to use a journaling file system such as reiserfs, ext3, or IBM's JFS. If a file system is mounted read-only, the system is not updated, so there's no need for the journal if the system crashes. If you avoid using journaling file systems, you will also avoid the obtuse error messages that can show up when a dirty file system is mounted and the system tries to replay the journal.

What cannot be shared

Since these two particular directories must be in the root (/) file system, you can not share anything else in the root file system either:

You cannot share any part of a file system that needs to be written to by any particular system. Examples include:

/home (although read-write sharing via NFS can be, and is frequently, done)

/tmp

/var

Mixing Software from Different Linux Distributions

Historically, mixing software from different Linux distributions has had poor results. Through increased standardization, the FHS and the Linux Standard Base (LSB) have reduced the number of problems this can cause, but it is still not recommended for a number of reasons:

Software packages probably were not compiled/linked against the same level of libraries/include files as the rest of your system.

Software packages may contain pre- or post-installation scripts that assume certain commands or facilities are available, but are not installed.

Software packagers don't use consistent names across distributions for the “provides,” “requires,” “supercedes,” etc. values that the package tools use to determine dependencies and conflicts.

Additional cooperation between the distribution providers could remedy this last item easily, but that hasn't been done. This causes a similar situation to the lock-in that many proprietary software suppliers try to achieve. If it were easy to mix and match packages from different distributions, there would be less reason to stick with a particular distribution.

Open Source Versus Proprietary Packages

Software quality

Like everything else in the world, Sturgeon's Law - “90% of everything is crud”, applies to Open Source Software. You should be picky about the Open Source packages you install and depend on.

Anyone can develop and distribute an Open Source software package. Some Open Source software authors do an exceptional job of testing their software. Some depend on the Open Source user community for testing. While most mainstream Open Source packages are thoroughly tested, be prepared to thoroughly test non-mainstream packages. Later we suggest some questions to consider when choosing an Open Source package.

Software longevity/viability

When you choose to use an Open Source package, consider how long it will be maintained by the author or Open Source community. A thorough look at the vast array of Open Source packages reveals that many of them haven't been updated in quite a while or have simply been abandoned. It may not be important to your IT shop if they have the resources necessary to take on that task. Most Open Source software users don't want to be in that role.

Again, this issue applies to proprietary software as well. If a commercial product is removed from the market, you may be out of luck when your license expires. With Open Source software, you at least have some choices and more control over your business:

Stay with the software and find someone to maintain it

Maintain it yourself

Cooperate with others to maintain it

When the time is right, select a different product.

The package's level of maturity. Has it been through the development cycle more than once or twice?

The reputation of the package. A number of Open Source packages are considered industry standards, - Apache, Bind, Per, Samba, Sendmail, Squid, for example. You can consider those packages safe choices.

The current install base. The larger it is, the better. If the current developers decide to take a hiatus (or a hike), the chances are good that someone else will take over the role. This has happened a number of times in the past.

Who are the developers and how many of them are there?

Have the developers been around for a long time and do they have a proven track record?

Do the developers work for a company that is committed to Linux and/or Open Source in general?

Do other software suppliers use this package you're considering as a component of their product? For example, IBM uses Apache as the engine for their IBM HTTP Server product. Clearly, IBM has a vested interest in the viability of the Apache project.

If a substantial part of your business relies on Open Source software, you must invest the time and resources to answer these questions.

Software license management

How much time and effort does your company spend managing software licenses? How sure are you that all the software you use is properly licensed? What is your legal liability/exposure if you're violating license requirements? Open Source software can help relieve some of this burden, due to the free redistribution clause in all Open Source Initiative (OSI)-compliant licenses. You can install Open Source software where you need it, when you need it, with little or no permission from anyone outside your company.

Other considerations

Does the product comply with open standards? If not, you are likely to experience integration problems, and some degree of lock-in. If the Open Source software packages you are using support open standards, replacing or upgrading them may be possible with a minimum of disruption. Open standards conformance can ease migration tasks.

Does the package run on all the platforms on which you run? Many Open Source packages run on a large number of Linux distribution and hardware platforms. Some packages have built-in limitations like embedded Intel assembler code, imposing hardware limits on the software. Or perhaps the software can only run on 32-bit machines or little-endian hardware. Investigate these issues before making a serious commitment to a package.

If the new package requires migration, how difficult will it be? Are there tools available to facilitate the migration?

Make sure you know where a package is installing files. Some Open Source developers ignore best practices and install code and data files into inappropriate places. This is true of both commercial and Open Source developers.

Conclusions

With more and more companies using Linux, the option of using Open Source software has also increased. As with any software package, Open Source is an attractive option in the appropriate situation. The use of Open Source software creates new requirements and challenges for managing software in an IT environment. Most of the considerations outlined here apply equally to commercial/proprietary software. Follow good software acquisition and management rules, regardless of the product's origin.

In the final analysis, what many people find so appealing about Open Source software is increased control over their IT destiny. Flexibility, portability, and the ability to deploy software more closely tied to business requirements are good things to have working for you.

Helping you maintain advantage

BMC Software Professional Services helps your company maintain its competitive advantage through a comprehensive suite of services that includes service level management consulting, installation, implementation, configuration, and customization. Our professional services and education offerings are designed to ensure the ongoing availability of critical business applications, maximize product potential, reduce project risk, deliver IT value to your business, and improve your operations. For more information about BMC Software Professional Services, visit http://www.bmc.com/profserv.

About BMC Software

BMC Software, Inc. [NYSE:BMC], is a leader in enterprise management. The company focuses on Assuring Business Availability® for its customers by helping them proactively improve service, reduce costs, and increase value to their business. BMC Software solutions span enterprise systems, applications, and databases. Founded in 1980, BMC Software has offices worldwide and is a member of the S&P 500, with fiscal year 2002 revenues of approximately $1.3 billion. Visit www.bmc.com to learn more.

Contents

Introduction