The Beowulf Parallel Workstation is based on the Linux operating system. Linux [6] is a full-featured clone of the UNIX operating system originally designed for x86 processors and recently extended to support other architectures. Its feature set includes POSIX compatibility, a TCP/IP protocol stack with a BSD-compatible sockets interface, very broad device support, dynamically linked shared libraries, interprocess communication and an efficient virtual memory subsystem with unified buffer cache.
A feature especially important to Beowulf is that Linux is a free implementation of UNIX. It is distributed under the terms of the Free Software Foundation's GNU Public License which insures that source code to the system is available, that we can easily share improvements, and that there are there no per-node royalties. Beowulf uses many low-cost processing nodes, and the latter is important not only financially but also because the administrative overhead of adding and moving nodes could be substantial.
Linux supports most of the available portable programming tools for distributed environments. The most popular system has been the public domain version of PVM [12], the Parallel Virtual Machine library. Other tools such as MPI [8] and an RPC library are also available.
Scalable communications is achieved by a technique called ``channel bonding'' [9]. With our implementation, the hardware address of a primary network adaptor is duplicated on the secondary interfaces, and all packets received on the bonded networks are marked as if they came from the primary interface. This scheme requires each Ethernet segment to span all nodes; switches provide a transparent connect between segments. With this constraint the Ethernet packet contents are independent of the actual interface used. The software routing overhead of handling more general interconnect topologies is avoided; the only additional computation over using single network interface is the computationally simple task of distributing the packets over the available device transmit queues. The current method used is initially alternating packets among the available network interfaces to the point that their private queues are full, and then distributing additional packets to the not-yet-full queues.
The Routed Mesh topologies take advantage of IP packet forwarding, a standard feature of the Linux kernel. This allows the kernel to function as an IP packet router, accepting a packet on one interface, determining that it must be forwarded, potentially fragmenting it, and transmitting it out a second interface. While the Ethernet adapters used on Beowulf are ``bus masters'' that reduce CPU load by copying packet data to and from main memory autonomously, the machine doing the routing still has the software load of handling a receive interrupt, allocating temporary space for the forwarded packet and copying it there, consulting its routing table, and placing the packet on the target interfaces' transmit queue.