Considering its current prominence in the embedded world, it is surprising how few books on the subject of embedded Linux are available. I reviewed Craig Hollabaugh's Embedded Linux: Hardware, Software, and Interfacing and LinuxDevices.com had earlier reviewed John Lombardo's Embedded Linux, but few other books cover the subject. Karim Yaghmour's Building Embedded Linux Systems addresses O'Reilly's lack of coverage in the area. Yaghmour originated the Linux Trace Toolkit and has been a frequent contributor to other open source projects, so one expects -- and finds -- accurate and detailed technical information in this book.

Uncompromising approach

Yaghmour uses an uncompromising do-it-yourself approach to the subject. The reader is essentially told to sit down with nothing more than an internet-connected computer, download all the needed packages from their original sources, and build the target system software from scratch. No reliance on third-party free or proprietary tools will ease the effort or stand in the way of the user's completely understanding the process. As we shall see, this pedagogical approach is excellent for highly motivated developers wanting to understand every detail and having the time to invest. Others who want to pick and choose the areas they wish to spend time on will be more appreciative of Hollabaugh's approach, which relied on scripts to perform much of the work. But even if you don't have time for a DIY approach to building your system, you should read a copy of Building Embedded Linux Systems just to gain an understanding of the procedures happening underneath the surface.

Sadly, the book is poorly organized from the point of view of someone following the process step by step, or for simply reading the book cover to cover. The first three chapters provide background information on a number of technologies important to embedded Linux, always maddeningly stopping just before getting to the point of providing usable detail needed for implementation. The subsequent chapters then revisit each subject in detail without the background information. So a reader wishing to implement a system using, say, JFFS2, must flip between chapters 3 and 8 to get the full picture. However, Yaghmour makes up for the first three chapters with some of the best, most detailed, descriptions of the process of building an embedded Linux system in the subsequent chapters.

Here is a detailed chapter-by-chapter review . . .

Chapter 1 --Introduction

Chapter 1 provides an introduction to Linux, embedded systems, and Linux within the embedded market. Most LinuxDevices.com readers can safely skip this material unless you are completely new to the subject.

One interesting subject covered in some detail is licensing issues. This subject is scary for many developers considering using Linux in embedded systems, with relentless FUD from proprietary software vendors magnifying the genuinely complex issues resulting from the GPL and the RTLinux patent. The author does his best to simplify the issues, but sometimes raises more questions than answers. For example, he makes a distinction regarding the GPL's requirements regarding distributing modified source code, writing, "Packaging the unmodified software for the purpose of running it . . . is not subject to this provision [requiring distributing source code]." The author's interpretation seems to be that one can redistribute unmodified GPL software without including source code; he repeats this conclusion in different ways several times. This is not how the GPL is usually interpreted -- it seems to require you to make the source code available any time you distribute the software. Whether you have first modified it is not relevant to your responsibility, although on a practical level the fact that you have not modified it means that the source code must be available from where you obtained it originally. Yaghmour makes the same questionable distinction between modified and unmodified code when discussing the LGPL. His summary of the licensing issues is otherwise quite good, but the modified/unmodified distinction may leave readers confused.

Chapter 2 -- Basic Concepts

While Chapter 1 discussed basic Linux and embedded system concepts, Chapter 2 introduces the development model Yaghmour will use in the remainder of the book. This chapter contains considerable filler, as the author suggests various models of little practicality in building actual systems, along with those actually in common use among embedded Linux developers. He correctly recommends using Linux as the cross-development host operating system, but then goes on to cover the use of Windows with Cygwin, and even Unix, as host operating systems. It is fair to say that Linux is by far the most appropriate cross-development platform for developing Linux-based devices. To use Windows or, worse yet, some proprietary Unix workstation, for embedded Linux development is only to invite misery. It's not that it cannot be done, it's only that most of us have schedules to meet and cannot spend our time tracking down problems that could be avoided simply by using a $500 Linux box.

This is typical of the book's first three chapters -- one finds considerable filler here having little to hold the interest of a typically harried embedded system developer looking for practical help. Stay with me -- it gets better.

Chapter 3 -- Hardware Support

But not yet. Chapter 3 is perhaps the most frustrating of all. It introduces the hardware Yaghmour considers most typical in embedded Linux systems. The chapter starts with page-long introductions to each of the architectures appropriate for such systems -- x86, ARM, PPC, MIPS, SH, and 68k. It's hard to identify the usefulness of this material; a device's processor architecture is usually determined by other considerations, and when it isn't, a one-page summary is unlikely to be of sufficient help to a designer.

The rest of the chapter contains short introductions to some buses and peripheral interfaces supported by Linux -- PCI, USB, GPIB, MTD, IDE, and so on. In each case, he discusses the technology in general terms, usually providing both more and less information than needed -- he provides us with short histories of the technology, then stops short of providing information on how to use it in Linux. The detailed information is provided in later chapters, but a better organization would have been to provide all of it together in one place, including only the introductory material necessary for understanding the technology.

Of greatest usefulness in this chapter is the introduction to Linux flash-based storage technologies. Yaghmour's is one of the better descriptions I've read, providing a good introduction to the MTD subsystem and to the filesystems intended for use with flash chips.

This chapter contains the closest thing I found to a technical error in the book. The author's coverage of watchdog timers mentions a daemon that can be run for the sole purpose of periodically resetting the watchdog hardware, even providing a link to the daemon software. To be sure, he recommends against using this approach to watchdog handling. But separating watchdog control into an independent process is such a horribly wrong policy that even mentioning it as a possibility rises almost to the level of technical error.

Chapter 4 -- Development Tools

Finally in Chapter 4 we get to content useful on a practical level. This chapter covers development tools, with an excellent step-by-step description of building a cross-development toolchain. As always, the author obviously understands his material, and a reader will emerge with a pretty good understanding of the process after reading this chapter.

Some readers will be annoyed with the author's pedagogical approach. First, he refuses to provide any help in automating the building of your toolchain, writing, "I would rather you fully understand all the steps involved." So he walks us through every step in building the toolchain -- no scripts will be provided. I'm sure readers will appreciate his concern for their educational growth, but most of us are faced with hard choices in deciding how to spend our development time.

One can only imagine a typical programmer who has installed Linux on a desktop machine at home and developed some expertise in it. Having heard that Linux can be used for embedded systems, this designer has managed to persuade the boss to use Linux rather than VxWorks for the company's next device. Imagine his/her mounting sense of panic while slogging through Chapter 4, which would take a newbie days or even weeks to get through fully; compared to VxWorks and other proprietary systems, which of course come with fully functioning development environments.

But Yaghmour provides no sugar-coating for the issues. His introduction to the process of building a toolchain is unintentionally amusing, at least to someone who already understands the process. He correctly warns against using the most recent versions of the various toolchain components (gcc, binutils, glibc, and so on) and expecting they will work together correctly for your host and target. He writes, ". . . start with the most recent stable versions of each package and replace them one by one with older ones if they fail to build." He then goes on to cite a hypothetical example in which gcc 3.2.1 fails to build with binutils 2.13.2.1, so you revert to 3.2, then 3.1.1 and so on; then the same thing can happen with glibc 2.3.1, so you revert to 2.3, then 2.2.5. He also points out that even a combination that builds correctly is not guaranteed to work correctly. This sort of talk continues for a full page.

One need hardly point out that while such discussions are of interest to toolchain developers, the average embedded system developer will not have time or inclination to try every permutation of the toolchain components to find an optimal one. To be sure, Yaghmour does provide a table showing known good toolchain component combinations for each architecture, but only after the long scary discussion about trying every permutation.

The material in Chapter 4 is of top quality -- anyone using Linux for embedded systems should understand the toolchain they use. It's only the pedagogical approach that I found alternately frustrating and amusing. A useful alternative might have been to describe the process step-by-step, mention briefly that not all combinations of the toolchain components work, provide the table of known good combinations, keep this table up to date on a web site, and provide some scripts for those who don't wish to build their toolchain by hand.

Still, once you've gotten past the annoying pedagogical approach, this chapter is an excellent introduction to the GNU toolchain. I recommend reading it even if you intend to use one of the available scripts for building your toolchain -- you will emerge with a good basic understanding that will equip you to tackle most problems you encounter.

Also covered in Chapter 4 are alternatives to glibc (of which uClibc is the most important) and some languages less commonly used in embedded systems: Java, Perl, Python, and Ada.

Chapter 5 -- Kernel Considerations

After the toolchain, the next logical component to address is the kernel, and Yaghmour's coverage of it in Chapter 5 is nearly ideal. He rightly refers readers to the appropriate general Linux books for most of the material needed for configuring the kernel and writing device writers, concentrating instead on only those issues specific to embedded systems. This includes processor configuration, cross-compiling, and dealing with multiple configurations. This chapter strikes the right balance -- an experienced Linux user and programmer could read this short chapter and be ready to build an embedded kernel; those less experienced should first consult one of the many introductory books.

You can download and read Chapter 5 here (PDF file).

Chapter 6 -- Root Filesystem Content

Chapter 6 addresses the next logical subject -- root filesystem configuration. This chapter again strikes the right balance, addressing only those issues likely to be unfamiliar to desktop Linux users. Yaghmour emphasizes those components recommended for use in embedded systems, including excellent introductions to uClibc, busybox, tinylogin, and busybox init.

Chapter 7 -- Storage Device Manipulation

Chapter 7 covers the important issue of mass storage devices, an area that causes much confusion among embedded Linux newbies. The emphasis is naturally on flash devices, and Yaghmour covers the MTD layer and filesystem level issues adeptly and cleanly. This is one of the best introductions I have seen, with first rate coverage of the arcane art of handling the DiskOnChip devices. My only complaint concerns the chapter's vague coverage of swapping issues. The author correctly recommends that most embedded systems have swapping turned off, but inexplicably fails to provide the simple instructions to do so.

Chapter 8 -- Root Filesystem Setup

Chapter 8 examines the filesystem types commonly used in embedded systems, providing some help in understanding the tradeoffs involved in choosing one over the other. Good coverage of cramfs, jffs, and jffs2 is provided, as well as those filesystems commonly used for desktop and server systems, such as ext2 and ext3.

Chapter 9 -- Setting Up the Bootloader

The author's treatment of bootloaders in Chapter 9 is one of the best short introductions I've seen. This subject is one of the more confusing to those familiar with desktop Linux, in which typically you follow clear and simple rules to set up LILO or GRUB, and your system then just boots. Embedded Linux has a much wider range of bootloader options, many requiring considerable effort for configuration and setup. Yaghmour's coverage emphasizes U-Boot, apparently reflecting his experience with that ARM- and PPC-oriented bootloader.

Chapter 10 -- Setting Up Networking Services

Chapter 10 addresses setting up network services. Much of this material applies to Linux in general, and can thus be found in numerous references, but Yaghmour's treatment emphasizes those components having an emphasis on small-footprint implementation, such as the Boa web server. He also addresses cross-compilation issues, so his treatment adds an emphasis not found in desktop Linux references.

Chapter 11 -- Debugging Tools

Finally, Chapter 11 addresses debugging tools. The author covers gdb in eight pages, including cross-debugging using gdbserver. Granted, all of this information is available elsewhere, but I think many readers would have appreciated having more detail here. And although most embedded system developers are now most comfortable debugging using a graphical IDE, Yaghmour virtually ignores the subject, leaving the reader to grapple with configuring KDevelop or DDD for their cross-development use. The rest of the chapter concerns tracing and profiling, with a special emphasis on the author's Linux Trace Toolkit.

Summing it up

Clearly, Building Embedded Linux Systems is an important book that should be in the library of anyone using or considering Linux for embedded systems. It covers the core issues of the subject (cross-development host setup, kernel and root filesystem configuration, init system, bootloaders) and covers them clearly and at a good level of detail. The book is well worth its price for these chapters alone.

My objections concern what could have been covered but wasn't. Most significant would be uClinux, probably the hottest subject in the field today, and certain to be important in the future because it allows Linux to be used in a significantly expanded space of devices. One could argue that its recent integration into the mainstream kernel makes a separate treatment of uClinux unnecessary, but the use of Linux in MMU-less processors is still sufficiently different that a separate treatment is necessary.

Also notable for its absence is a full treatment of real-time issues. To be sure, the importance of real-time capabilities in embedded systems is often overstated, but designers should still have a full grasp of their options when choosing their platform. I think a chapter covering RTAI, RTLinux, and the preemptible kernel patch could easily be justified.

Finally, many readers would find helpful some coverage of higher-level functionality, particularly the construction of graphical user interfaces. Few embedded systems have available the resources to implement a typical desktop GUI environment utilizing X and a desktop environment. So a book on embedded Linux should have some coverage of how to implement framebuffer support on diverse hardware, and how to take advantage of this support using the various windowing toolkits.

Room for this additional material could easily have been made by eliminating or scaling back some of the areas I've mentioned. Most of the first three chapters are strictly introductory, often presented in far too wordy a style. Much of this could have been eliminated, with the remaining material integrated with the later chapters in a seemless way. Similarly, Chapter 10, concerning networking, presents some material that differs little from networking on desktops.

Despite these gripes, you should have Building Embedded Linux Systems for the unquestionably top quality material it does have. The book gives you a solid foundation in embedded Linux that you ought to have, whether building systems using one of the toolkits or strictly from scratch.

Karim Yaghmour has recently launched embeddedTUX.org, which will serve as a companion site to "Building Embedded Linux Systems", and will provide additional information about developing embedded systems based on the Linux kernel using only freely available open source and free software packages. An associated mailing list is intended to provide a forum for discussing embedded Linux topics.

• O'Reilly announces new book: "Building Embedded Linux Systems"
  
• Embedded Linux (a book by John Lombardo)
  
• Embedded Linux: Hardware, Software, and Interfacing (a book by Dr. Craig Hollabaugh)
  
• Book review: John Lombardo's Embedded Linux
  
• Introducing Embedded Linux: Hardware, Software and Interfacing
  
• Book review: Craig Hollabaugh's Embedded Linux
  
• Two more reviews of Craig Hollabaugh's 'Embedded Linux' book