Beowulf Software

Index

Overview

The Beowulf software environment is implemented as an add-on to commercially available, royalty-free base Linux distributions. These distributions include all of the software needed for a networked workstation: the kernel, Linux utilities, the GNU software suite, and many add-on packages. Initially we used the very popular Slackware distribution. We have now migrated to the RedHat distribution with its better package management and upgrade system.

The Beowulf distribution includes several programming environments (all developed elsewhere) and development libraries as individually installable packages. PVM, MPI, and BSP are all available. SYS-V--style IPC and p-threads are also supported. A considerable amount of work has gone into improving the network subsystem of the kernel and implementing fast network device support. Most of these changes have been incorporated into the kernel source code tree, although many device driver updates are still in the active development or tuning phase.

In the Beowulf scheme, as is common in NOW clusters, every node is responsible for running its own copy of the kernel and nodes are generally sovereign and autonomous at the kernel level. However, in the interests of presenting a more uniform system image to both users and applications, we have extended the Linux kernel to allow a loose ensemble of nodes to participate in a number of global namespaces. A guiding principle of these extensions is to have little increase in the kernel size or complexity and, most importantly, negligible impact on the individual processor performance.


Beowulf System Software

Network Device Drivers
We have long contributed to the development of the Linux networking code. Many of the Linux ethernet device drivers, and most of the device drivers for cost-effective high-performance network adapters were written by Donald Becker at CESDIS. A large portion of his time goes into maintaining and enhancing the performance of these device drivers. Fast Ethernet has long been a cluster staple, while Gigabit Ethernet has recently been tested.
Beowulf Ethernet Channel Bonding
One of the goals of the goals of the Beowulf project is to demonstrate scalable I/O using commodity subsystems. For scaling network I/O we devised a method to join multiple low-cost networks into a single logical network with higher bandwidth.
Beowulf PID space
Normal UNIX processes `belong' to the kernel running them and have a unique identifier within that context. In a parallel distributed scheme it is often convenient for UNIX processes to have a process ID that is unique across an entire cluster, spanning several kernels. To this end we have developed two different schemes. Both are loadable kernel modules which provide a cluster-wide process id space with remote delivery of signals.
PVM-TCL
A set of extensions to TCL that allow you to manipulate the PVM virtual machine directly. NOTE: This currently does not extend the message-passing calls to TCL, just the PVM configuration ones. One could, however, implement them from the base provided here.
Virtual Memory Pre-Pager
A loadable kernel module for Linux 2.0 that implements a non-blocking page read-in system call. With the proper run-time support, this new system call can dramatically reduce the run-time of "out-of-core" programs (programs that have data sets larger than the available physical memory) by allowing multiple pages to be read from disk prior to their actual use.
PPro Performance Counter Patches
Kernel patches and a small library to add kernel support for the Pentium Pro performance counters.
LM78 Hardware Monitor Driver (ver 0.2)
A loadable kernel module for Linux 2.0 that provides a /proc interface for the LM78 Hardware Monitor.
Intel PR440FX Netbooting Tools
Tools to assist with nebooting from built in network interface on the Intel PR440FX and possibly other AMI Bios motherboards.

Beowulf Software as RPMs

Linux kernel with channel bonding and other Beowulf tweaks.
Ohio State's LAM MPI 6.1 as RedHat rpms.
Oak Ridge National Lab's PVM 3.3.11 as RedHat rpms.
Gero Kulhmann's 'netboot' package for booting diskless x86 machines.
Beowulf base software distribution for RedHat Linux.

Programming Models

Distributed Computing Environments

There are several distributed application programming environments available on Beowulf. The most commonly used are the PVM and MPI environments (we use the LAM implementation), with BSP also available and used.

Network Virtual Memory / Distributed Shared Memory

The Linux kernel provides a VFS-like interface into the virtual memory system. This makes it simpler to add transparent distributed back-ends to implicitly managed namespaces. Page-based systems can be created that allow the entire memory of a cluster to be accessed either almost or completely transparently.

An additional environment being added to the Beowulf packages is page-based Network Virtual Memory (NVM), also known as Distributed Shared Memory (DSM). Page-based distributed shared memory uses the virtual memory hardware of the processor and a software-enforced ownership and consistency policy to give the illusion of a memory region shared among processes running an application.

The initial Beowulf DSM implementation was based on the ZOUNDS (Zero Overhead Unified Network DSM System) system from Sarnoff. Our own distributed shared memory package, implemented by Jason Crawford at GSFC, will soon be available (June 1997).

We have also evaluated a commercial DSM package, TreadMarks from Rice University.

Beowulf Technology

Beowulf systems are a good idea because technology advances have changed the economics of computing. The tradeoff now favors clusters of commodity technology over higher-performance but lower volume systems. A significant part of putting together a near-optimal system is "technology-tracking": Here are a few topics we actively follow:

Beowulf Software from other sites

Beowulf cluster software is being developed in as many places as Beowulf clusters are being built. Below are pointers I am aware of:

Other CESDIS Linux software

In addition to the above software, additional Linux-related software has been developed at CESDIS.

X10 ActiveHome device control for Linux
A complete X10 package for device control using the X10 ActiveHome computer interface. We have, at various times, joked about using X10 for a couple of things in our Beowulf context. A popular idea is to use an X10 RF remote to cycle the power on hung machines inside a locked machine room. :) Over the course of our project, man-days would be saved if we could reboot hung clusters without leaving our desks.
Mike Romberg's (romberg@fsl.noaa.gov) xosview patched for /proc/apm support
  • Binary RPM for i386
  • Source RPM

  • Author:Dan Ridge, newt@cesdis.gsfc.nasa.gov.
    Contact:newt@cesdis.gsfc.nasa.gov
    CESDIS.