Closing gaps in the embedded Linux ecosystem: The ISV challenge – Part I
By Bill Weinberg
Open Source Architecture Specialist, OSDL
In this column, Bill examines the state of the embedded Linux ecosystem − how Independent Software Vendors (ISVs) regard Linux as a platform, how they port to Linux, and how they offer support for the ever-expanding range of Linux distributions and processing platforms.
Part I of the column examines factors that can slow or limit ISV adoption of Linux as a host/target. In Part II, the column details efforts by the OSDL and other organizations to facilitate and accelerate ISV adoption and support of Linux.
Inherited benefits
Embedded computing with Linux benefits enormously from the mainstream adop-
tion of Linux and open source for the enterprise. Embedded developers today leverage Linux to build intelligent devices ranging from telecommunications to home networking, from industrial control to medical instrumentation, and from home entertainment to smart phones. However, embedded development and deployment with Linux also face many of the same challenges as enterprise applications. Standing out among those hurdles is the validation and productization of Commercial Off-The-Shelf (COTS) software from ISVs on Linux platforms.
The ISV challenge
In the enterprise marketplace, ISVs view Linux as an emerging and increasingly dominant host platform for data center
and desktop applications.
Most visibly, Linux has been taking server and workstation market share away from proprietary UNIX, as well as winning white boxes, blades, and seats away from Microsoft Windows. The first ISVs to migrate their wares to Linux have been those with off-the-shelf UNIX product lines.
Enterprise and embedded ISVs choose
if and when to support Linux based on a mix of ROI factors. To understand these factors, the OSDL conducted a study in 2004 of several dozen top-tier (enterprise) ISVs. The highlights of the study are shown in Table 1.
Table 1
The study data bears some additional analysis. The number of distributions supported by an ISV seems innocuous out of context, but bear in mind that these same companies typically support only one or two versions of Windows or Solaris. The extreme of nine distributions is for ISVs in the storage area market, whose middleware and appliances must support regional distributions of Linux (such as Red Hat, SuSE, Mandriva, and Red Flag) and legacy distributions of Linux of the same for the lifetime of their appliance.
Surprisingly, many enterprise software vendors do not track actual deployment or sales by host OS, and consequently, they do not respond directly to movement in their existing customer base. A more common migration motivator is requests from new customers, or pending orders from large accounts.
The OSDL study also revealed the fol-lowing pervasive set of concerns shared by most ISVs:
Validation and testing of ISV-ware with each release of leading Linux distributions
Library version differences among distributions
Identifying distributions and versions to support
Packaging for and installation on mul-tiple distributions
Certainly, these concerns are shared by software houses targeting embedded as well. Following is a discussion on how these issues have the most impact on
ISV ROI.
Port once and run everywhere?
While the Linux kernel is impressively stable and community practices prevent the OS core from forking or fragmenting, it is very difficult to guarantee that an application binary built on one Linux distribution will run correctly on another – even within the family of IA/x86 distributions. While the divergences are small, and many programs simply do just run on one distribution or another, for complex applications these differences can be enough to derail successful installation and execution. Problem areas are listed in Table 2.
Table 2
In enterprise, companies like Red Hat, SuSE, Mandriva, and others address these challenges with a combination of platform certification programs and language in their Service Level Agreements (SLAs) that determines support policies for third-party commercial and open source programs not included in their own distributions. More encompassing efforts to address these issues have also included United Linux (now defunct) and the Linux Core Consortium.
Additional challenges
Independent software vendors who offer embedded applications face the above challenges in varying degrees, as well as additional challenges endemic to embed-ded development and deployment. For nominally embedded applications running on enterprise-like blades and embedded PCs, the challenges are mostly the same. With other more deeply embedded designs, other unique obstacles present themselves:
While there are probably fewer em-bedded ISVs overall, those ISVs may need to address a greater number of divergent platforms and application areas.
While all architectures share the same Linux kernel source base and most other operating system code, there are differences in maturity and feature support among CPU-type implementations.
There exist a larger number of relevant commercial embedded distributions and platforms than exist for enterprise Linux (each with smaller market shares and volumes), and there are more differences among those products (FSMLabs, MetroWerks, MontaVista, TimeSys, SysGo, Wind River).
While there exists the possibility of bi-nary compatibility across x86/IA-32
enterprise distributions, there is no comparable potential across the major embedded architectures (such as ARM, M68K, MIPS, PowerPC, SH, x86). Indeed, there are even source and binary validation issues among members of the same CPU family (such as Intel XScale vs. TI OMAP, IBM vs. Freescale PowerPCs).
Many embedded ISVs do not actually target Linux per se, but port to Linux using compatibility layers. On one hand, these abstractions minimize differences among embedded operating systems, but on the other hand, they mask and do not address importance distinctions between them (for example, threading library semantics).
In harmony with open source practices, many embedded ISVs have long-standing traditions of shipping source code. However, this supply method shifts the burden of embedded platform validation and support to their customers (and creates services opportunities for themselves or third parties).
Much ISV-ware was conceived to bridge the gap between enterprise and infrastructure technology and bare metal embedded applications. Much of this value added software addressed management middleware, file systems, and networking protocols – technology that is fully native to all embedded Linux platforms. To play in the embedded Linux market, embedded ISVs must decompose their offerings to face this reality, or force-fit their legacy wares onto the embedded Linux OS and stack, with questionable value-add for customers.
It is also interesting to examine how em-bedded software companies dip their toes into supporting Linux. Typically, embedded ISVs start by targeting their wares at the regionally-dominant enterprise Linux platform. In North America, this first port is usually to a version of Red Hat. Many ISVs go no further than this first platform-wise step, and then proceed to advertise they have accomplished Linux support.
More systematic Linux support by em-bedded ISVs usually involves validation on one or more architecture versions of a particular embedded Linux distribution, such as MontaVista Linux Carrier Grade Edition, or Wind River Platform for Networking Equipment, Linux Edition.
Release introductions
In both enterprise and embedded settings, ISVs face a challenge that arises from the misalignment of their own release cycles and those of the open source platforms to which they port their wares. Most ISVs productize their applications on one or two strategic host platforms, and then port to the other OSes in their stable. In-creasingly, at least one Linux distribution is usually among these key platforms.
Most ISVs of my acquaintance release new product versions every 12 to 24 months. Enterprise distribution suppliers like Red Hat and SuSE, and non-commercial distributions like Debian offer new releases as often as twice every year. Embedded platform new releases are usually introduced less often. The individual projects, whose code comprises the hundreds of packages that round out
a distribution, has its own release cycle that ranges from monthly drops to more stately (or stagnant) rhythms.
ISVs usually port their software to the current release of a given host platform, such as SuSE Desktop 9.2 or MontaVista Linux 3.1. The porting effort can occur early or later on in the platform release lifetime.
A distribution itself can draw upon hundreds, sometimes even thousands of individual project/packages. Distributions, as they approach their own code freeze dates, settle on the more recent, stable versions of their constituent package sets. Some of these packages versions might be only one month old, while others could date back 6 to 12 months.
Misalignment
The big challenge occurs when an ISV, either at porting time or during the lifetime of their own release, finds a bug in a pack-age within a distribution. The first recourse of most ISVs is to go to the distribution for a patch. The distribution supplier response, however, may be that the bug is fixed in the next release, even if the ISV needs the patch in the present release.
Should the requirement escalate in urgency, the ISV may elect to go to the project behind the package (for example, GNU binutils, qt). What they often find there is that the bug has been fixed in a subsequent release, or that the project maintainers are willing to accept or even create a patch, but on the current release, not on the release that was aggregated into the distribution. This mismatch is illustrated in Figure 1.
Figure 1
In the worst case, an ISV can be faced with package code that is 12 months old. The situation for embedded ISVs can be even more precarious, since embedded Linux platform providers are often more conservative (or less agile) than their enterprise counterparts.
A prime example comes from some em-bedded distribution suppliers, who as of this writing still ship platforms derived from the prior 2.4 Linux kernel, a full
18 months after other distributions (embedded and enterprise) have moved to the 2.6 kernel, and most projects only build and validate their code against the 2.6 base.
Meeting the ISV challenge
In the November issue of Embedded Computing Design, Part II of the OSDL column will examine community and commercial efforts to make it easier for ISVs, developers, and end users in the enterprise and embedded communities. These activities and initiatives include:
The Linux Standards Base from the Free Standards Group
OSDL ISV and IHV Forums
The Binary Regression Test project sponsored by the OSDL
The Linux Core Consortium
Partnering and validation programs at commercial distribution suppliers
. . . . .
Bill Weinberg brings more than 18 years embedded and open systems experience to his role as Open Source Architecture Specialist at the Open Source Development Labs. Bill can be contacted at bweinberg@osdl.org.
OSDL – home to Linus Torvalds, the creator of Linux – is dedicated to accelerating the growth and adoption of Linux in the enterprise. Contact the OSDL directly for membership and lab usage information.
