Beowulf Ethernet Channel Bonding

Overview

One of the goals of the goals of the Beouwlf project is to demonstrate scalable I/O using commodity subsystems. For scaling network I/O we devised a method to join multiple low-cost networks into a single logical network with higher bandwidth. The only additional work over a using single network interface is the computationally simple task of distributing the packets over the available device transmit queues.

We initially implement this method using two 10mbps ethernets on the first Beowulf system, later expanding it to three channels. Recent Beowulf systems have parallel 100Mbs Fast Ethernet channels.

Use

Except for the performance increase, channel bonding is transparent to the applications running on the cluster.

User-level control program 'ifenslave'

The system-visible interface to "channel bonding" is the 'ifenslave' command. This command is analogous to the 'ifconfig' command used to set up the primary network interface. The 'ifenslave' command copies the configuration of a "master" channel, commonly the lowest number network, to slave channels. It can optionally configure the slave channels to run in a receive-only mode, which is useful when initially configuring or shutting the down the additional network interfaces.

Implementation

To minimize protocol overhead and allow the latest possible load-balancing decision, Beowulf channel bonding is implemented at the device queue layer in the Linux kernel, below the IP protocol level. This has several advantages:

• Load-balancing may be done just before the hardware transmit queue, after logical address (e.g. IP address) and physical address (e.g. Ethernet station address) are added to the frame.
• Fragmented packets, usually large UDP/IP packets, take advantage of the of the multiple paths.
• Networks that fail completely first fill their hardware queues, and and are subsequently ignored.

• All connected machines must be on the same set of bonded networks.
• It's difficult to detect and compensate if one network segments (splits in the middle).

The implementation is divided into two parts, the kernel changes needed to support channel bonding, and the user-level control program 'ifenslave'.

The user-level control program 'ifenslave' is compiled from the source ifenslave.c.