With Fedora Core 4, Red Hat, for the first time, released its Global File System (GFS) as an open sourced, freely available distributed network file system. A distributed network file system builds on the basic concept of NFS as well as RAID techniques to create a file system implemented on multiple hosts across a large network, in effect, distributing the same file system among different hosts at a very low level.

Think of it as a kind of RAID array implemented across network hosts instead of just a single system. In effect, instead of each host relying on its own file systems on its own hard drive, they all share the same distributed file system that uses hard drives collected on different distributed servers. This provides far greater and efficient use of the storage available to the hosts and provides for a more centralized management of file system use. GFS can be run either directly connected to a SAN (Storage Area Network) or using GNBD (Global Network Block Device) storage connected over a LAN. The best performance is obtained from a SAN connection, whereas a GNBD format can be implemented easily using the storage on LAN (Ethernet) connected systems. Like RAID devices, mirroring, failover, and redundancy can help protect and recover data.

GFS separates the physical implementation from the logical format. A GFS file system appears as a set of logical volumes on one seamless logical device that can be mounted easily to any directory on your Linux file system. The logical volumes are created and managed by the Cluster Logical Volume Manager (CLVM), which is a cluster-enabled LVM. Physically, the file system is constructed from different storage resources known as cluster nodes distributed across your network. The administrator manages these nodes, providing needed mirroring or storage expansion. Should a node fail, GFS can fence a system off until it has recovered the node. Setting up a GFS file system requires planning. You have to determine ahead-of-time different settings such as the number and names of your GFS file systems, the nodes that will be able to mount the file systems, fencing methods, and the partitions and disks to use.

GFS, as implemented by Red Hat, uses a variety of install packages that do not appear related just by their names. This article provides a map of what packages you will need and how they are related, as well as basic implementation and operations for a GFS file system.

GFS Packages
To use GFS you have to install a number of different packages that include the GFS tools, the locking method you want to use, and configuration tools. The GFS tools and locking methods also have several corresponding kernel module and header packages. Be sure to choose the kernel module for the kind of kernel you're running. There are kernel module packages for the different types of kernels: i586, i686, SMP, and Xen. There are also packages for both i586 and i686 kernel headers, needed for compiling other modules. The Fedora Core 4 GFS tools and the GFS SMP kernel module (used for hyper-threading Pentiums and dual processors) packages are shown here. Updates for more recent kernel versions are also available.

GFS-6.1-0.pre22.3.i386.rpm
GFS-kernel-smp-2.6.11.5-20050505.133825.FC4.0.i686.rpm

Cluster configuration tools and servers are kept in the Cluster Configuration System package, ccs. Fencing tools used to isolate failed resources are kept in the fence package. The tools and servers needed to implement a GNBD form of GFS are located in the gnbd package. In addition, you need the kernel modules with the drivers to support GNBD devices, located in a gnbd-kernel package. LVM cluster support is located in the lvm2-cluster package. For gndb be sure to choose the one for your kernel. The Fedora 4 package names are listed here:

ccs
fence
gndb
gndb-kernel
lvm2-cluster

To run a cluster, you need both a cluster manager and a locking mechanism. There are two choices: the Cluster Manager (cman ) with the Distributed Lock Manager (dlm) or the Grand Unified Locking Manager (gulm). cman with dlm implements cluster management and locking directly using modules: cman and dlm. cman manages connections between cluster devices and services, and uses dlm to provide locking. The dlm locking mechanism operates as a kernel module with supporting libraries and is invoked with the cman script. gulm is an older method that operates, instead, as a server, providing both cluster management and locking. gulm can be invoked by the lock_gulmd script. It is considered an alternative to cman and dlm. You only need to install either gulm or the cman and dlm packages.

cman
cman-kernel
dlm
dlm-kernel
gulm
gulm-kernel

You can't run both cman and gulm at the same time. The cman script checks the /etc/cluster.conf file for the gulm configuration, and if found will not load the cman kernel module. Use the cman_tool to have a node join a cluster or remove a node from the cluster.

GFS Service Scripts
To start the GFS file system, run a series of scripts that start up the appropriate servers and load the needed modules. These are the ccsd script with start-up configuration detection, fenced for fencing support, cman for cluster management dlm locking, clvmd to detect activate up the CLVM GFS devices, and, finally, gfs to mount your GFS file systems. The scripts will check for any GFS configuration settings in the /etc/sysconfig/cluster file. To shut down the GFS file system service, use a reverse sequence with the stop option.

service ccsd start
service cman start
service fenced start
service clvmd start
service gfs start

As noted previously, you have a choice between two different locking mechanisms, gulm and dlm. The dlm method is used in the previous example, which is invoked by the cman script.

The clvmd script both starts the clvmd server and uses LVM commands vgscan and vgchange to locate and activate your cluster devices.

The gfs service script will mount GFS file systems to the locations specified in the /etd/fstab file. Entries are needed for all the GFS file systems you want to mount in /etc/fstab. The stop option will unmount the file systems.

Implementing a GFS File System
To set up a GFS file system, first create cluster devices using the physical volumes and organize them into logical volumes. Use the CLVM (Clustering Logical Volume Manager) to set up logical volumes from physical partitions (in the past you used a volume manager called pool to do this). You can then install GFS file systems on these logical volumes directly. CLVM operates like LVM, using the same commands. It works over a distributed network and requires that the clvmd server be running.

You can then configure your system with the Cluster Configuration System. Create a /etc/cluster.conf file and set up your configuration. The configuration will include information like the nodes used, fencing methods, and the locking method used (gulm or dlm). Consult the cluster.conf Man page for configuration details. For configuration you can also use the system-config-cluster tool, which provides an easy-to-use GUI interface, or set up your GFS file system (still under development for Fedora Core 4). You can select locking and fence methods as well as mount points and devices, and then start the ccsd server and test the configuration with the ccs_test tool.

service ccsd start
ccs_test mygfs

You then use the ccs_tool to create cluster.ccs, fence.ccs, and node.ccs configuration files. These files are organized into a CCS archive that is placed on each node and cluster device.

On each node start the ccsd configuration, fenced fencing server, and the locking method you want to use like dlm. Check the respective Man pages for details on the locking servers. You can start the servers with their service scripts as noted previously.

To create new file systems on the cluster devices, use the gfs_mkfs command and then mount them with the -t gfs option. The following command creates a GFS file system on the /dev/gv0/mgfs and then mounts it to the /gfs1 directory. For gfs_mkfs, the -t option indicates the lock table used and the -p option specifies the lock protocol. The -j option specifies the number of journals.

gfs_mkfs -t mycluster:mygfs -p lock_dlm -j 2 /dev/vg0/mgfs
mount -t gfs /dev/vg0/mgfs /gfs1

To have the gfs service script mount the GFS file system for you, you need to place an entry for it in the /etc/fstab file. If you don't want the file system automatically mounted, add the noauto option.

/dev/vg0/mgfs   /gfs1   gfs   noauto,defaults   0   0

• Resolving FAT File System Error “STOP 0x00000023”