As Linux and Open Source software becomes ever more popular in embedded applications, developers and management at device OEMs strive to understand the Open Source community, its practices, and their own companies’ roles in Open Source. This column is intended to provide an overview of the processes and practices that drive Open Source development, both as a primer for the justification and use of Open Source Software (OSS) in embedded projects, and as an invitation to developers of all types to participate in those processes.
A disciplined process
Your company probably already uses an application or tool that was developed using Open Source methodologies. For example, your corporate website may use Apache with PHP or Perl, some portion of your corporate data may be warehoused with PostgreSQL, and you may now use or have probably used the Netscape browser. It is also extremely likely that your embedded projects are built with a member of the GNU Compiler Collection (GCC), and somewhere in your company you use Linux for enterprise or even embedded applications.
Open Source is not a free-for-all. Source code, while flowing freely from developers to distributions to end-users (and back) does not make its way helter-skelter into projects and from there into deployed devices. The Open Source development process is probably more disciplined than many proprietary software development processes. It has to be as the Open Source development process must be able to integrate, test, and quality-assure contributions from thousands of developers from around the globe.
The Linux kernel development process (Figure 1) in particular has been described as a benevolent dictatorship. Patches and enhancement are submitted for all parts of the Linux kernel by developers worldwide, thereby creating a borderless community. Vetting and integrating this diverse set of contributions falls to the 80 or so subsystem maintainers who focus on and contribute to particular aspects of the kernel such as file systems, memory management, and the system call interface.
Figure 1 [Kernal Dev Process]
Advancing versions of these subsystems are rolled up into patch sets that ultimately form experimental kernel versions (in the past, odd numbered kernel releases like 2.3.x and 2.5.x). When Linus Torvalds and his team at kernl.org are convinced that Linux kernel technology has advanced and matured sufficiently for commercial deployment, a new production kernel is born and handed off for testing to production kernel maintainers (for example, Marcelo Tosati for 2.4 and Andrew Morton for 2.6). It is normally from these stable production kernels that distribution suppliers build their GNU/Linux OS platforms. For the enterprise: Red Hat, SuSE, and others. For the embedded domain: MontaVista, TimeSys, Wind River, and others.
OSS Code Origins
Let us examine two origins of Open Source code:
Open Source code from new projects
Open Source code integration into mature projects
Open Source code from new projects
Open Source software projects are initiated in the same manner as proprietary projects – from a developer’s inspiration, or more formally, in response to requirements expressed by and collected from end-users.
OSS departs radically from traditional commercial software in that its inception need not take place within the confines of a traditional engineering organization. Sometimes new code is written from scratch, and other times, a piece of existing code is re-licensed under an Open Source license and released as a community project.
Some OSS projects, such as applications or middleware, are entirely free-standing. Others exist as patches to the Linux kernel or other OS components – usually device drivers, I/O subsystems, alternate schedulers, and other systems components. A project can exist indefinitely as an independent effort, and is useful as long as its maintainers update it to support current libraries and kernel versions.
If a project shows exceptional merit and broad utility, it can also be picked up by the Linux kernel maintainers and be integrated into the mainstream OS. Such examples include the adoption of the National Security Agency (NSA) Secure SE Linux as a standard build option in the 2.6 Linux kernel, and the incorporation of the Pre-emptible Linux Kernel Patch into the 2.5 and later production kernels. More typical examples are the hundreds of drivers developed by hardware vendors for their devices and later integrated into mainstream kernel trees.
Open Source code integration into mature projects
Larger, mature projects continue to evolve over time and benefit from the contributions of both core team developers as outside submitters. Understanding the path that new submissions follow is instructive for organizations hoping to participate in Open Source development, and for companies concerned about the integrity of the process (Figure 2).
Figure 2 [Code Lifecycle]
Newcomers to Open Source often speak of pushing code out to Open Source, but push is an inappropriate verb to describe the submission process. Individual contributors or organizations actually find it quite challenging to have their patches accepted into actual projects and distributions.
To begin with, a patch must be well-formed, which means coded and packaged per the established Open Source conventions, and per the requirements of the given project. Moreover, a patch must be of sufficient merit or novelty to stand out from the dozens (or hundreds) of other proposed patches in front of a project maintainer.
Once a submission actually makes it into a project, it will be exercised, tested, and scrutinized by that project’s user community. If it survives and becomes part of a project’s mainstream code, and the project is then picked up by a distribution, it will be subject to the integration, testing, and QA discipline that comprises that distribution’s added value.
It should be noted that end-users have complete project control, because the inclusion of each specific project package in the distribution into their project is entirely discretionary. Because you have the project source code, you can always return to the project that accepted and built the code if the need arises.
Open Source code quality assurance
Quality assurance for Open Source code occurs at several points in the code lifecycle, and at multiple places in the Open Source ecosystem. As described above, there exist three layers of purview over code submission and quality:
Project
Distribution
End-user
At the project level, there is project-specific testing (including build-testing) that is carried out as part of the project lifecycle. At the same time, the project community joins with the project audience to engage in usability and performance testing.
Project code integration into standard distributions is subject to both standards-based and supplier-specific testing, and the QA that each distribution team performs. In my own experience at an embedded distribution vendor, we performed standards-compliance testing (for example, LSB and POSIX), stability and robustness testing, real-time response and throughput benchmark testing, installation testing, and several other tests and QA checks. We also cross-pollinated by comparing test results among our eight supported architecture families.
The result of open source cooperation
If you extend the test and QA process cooperation out to several dozen distribution suppliers, add the bug reports from hundreds of thousands of embedded and enterprise users and developers, and then add the rapid repair and integration that is the hallmark of Open Source, you have a virtualized and borderless QA team that outpaces and outscales even the largest proprietary software organization.
OSDL Role in Open Source
The OSDL participates in Open Source development in four distinct capacities:
OSDL Initiatives
First, and most visibly, OSDL initiatives, promoted by and for member companies, collect and refine end-user requirements for each of the target areas: Carrier Grade, Data Center, and Desktop Linux. These requirements are mapped against the current capability set of Linux and its stack at any given moment in time with two outcomes: the provision of specifications and capabilities lists to distribution providers, and the fostering of new community development to fill gaps in Linux capabilities.
Performance and Regression Testing
Less visibly, the OSDL hosts a massive testing effort for both experimental and production versions of the Linux kernel. This test bed can also be applied to combinations of kernel builds and patch sets using the OSDL Scalable Test Platform (STP) combined with the Linux Patch Line Manager (PLM). OSDL members and associates can upload patches and leverage STP on-line at the OSDL website.
OSDL Member Contributions to Open Source
In response to OSDL initiatives and other organizational imperatives, the over 60 members of the OSDL make substantial contributions to Linux and to other Open Source projects in both enterprise and embedded areas. Founding members include IBM, Intel, HP, CA, Fujitsu, Hitachi, and NEC.
Direct Participation in OSS Development
OSDL staff members contribute directly to dozens of Open Source projects such as the Kernel, Asynchronous IO, Persistent Device Naming, Clustering, and Testing. The OSDL also compensates Linus Torvalds, and kernel maintainer Andrew Morton
. . . . .
Bill Weinberg brings over 17 years embedded and open systems experience to his role as Open Source Architecture Specialist at the Open Source Development Labs. Bill can be contacted at bweinberg@osdl.org.
OSDL – home to Linus Torvalds, the creator of Linux – is dedicated to accelerating the growth and adoption of Linux in the enterprise. Contact the OSDL directly for membership and lab usage information.
Open Source Development Labs, Inc.
12725 SW Millikan Way, Suite 400
Beaverton, OR 97005
Tel: 503-626-2455 • Fax: 503-626-2436
Website: www.osdl.org
