| By Steve Jones |
Article Rating: |
|
| March 17, 2006 08:15 AM EST |
Reads: |
21,740 |
Here's what I did:
Create the PBS scripts to run the tests-(following the example script above - truncated version)
#!/bin/bash
#PBS -l nodes=1:ppn=2
cmd="/home/tools/bonnie++/sbin/bonnie++ -s 8000 -n 0 -f -d
/home/alice-nfs-directory/bonnie"
$cmd >& $PBS_O_WORKDIR/Log.d/run.nfs/log.bonnie.nfs.$PBS_JOBID
#!/bin/bash
#PBS -l nodes=1:ppn=2
cmd="/home/tools/bonnie++/sbin/bonnie++ -s 8000 -n 0 -f -d
/home/alice-panfs-directory/bonnie"
$cmd >& $PBS_O_WORKDIR/Log.d/run.panfs/log.bonnie.panfs.$PBS_JOBID
Run the tests-
Results from eight instances of the NFS script-
17.80MB/sec for concurrent write using NFS with eight dual-processor jobs
36.97MB/sec during read process
Results from eight instances of the PanFS script-
154MB/sec for concurrent write using PanFS with eight dual-processor jobs
190MB/sec during read process
The numbers were excellent, so I decided to scale it up:
Results from 16 instances of the NFS script-
20.59MB/sec for concurrent write using NFS with 16 dual-processor jobs
27.41MB/sec for during read process
Results from 16 instances of the PanFS script-
187MB/sec for concurrent write using PanFS with 16 dual-processor jobs
405MB/sec during read process
It's pretty much a no-brainer here. We moved to the new system. Having the Rocks Clusters platform let us integrate the Panasas solution in just under two hours.
Final Thoughts
First off, this may sound as if it's very simple to set up and manage a cluster. While the description above looks to be straightforward, the real work starts once your first user logs onto the front-end. I have a group of power users that I've unleashed on systems in the past and I'm comfortable with their ability to compile and run known good code and provide me with clear information related to the issues they encounter on the system. I wouldn't advise setting up a system and turning everyone loose on it since you might spend the next few weeks trying to resolve issues that may or may not be cluster-related.
This is where the Rocks discussion list comes in handy. Although you may have certain configuration and code-related differences with other Rocks installations, you share the same common thread with everyone, the same tools to troubleshoot and apply changes.
You'll also find your cluster may not be as different as you think it is from other clusters. I've been able to manage a large number of clusters without having the overhead of many system administrators primarily because of interactions with people on the Rocks list. By far the easiest system I've installed was a Penguin AMD Opteron cluster running Infinicon (now Silverstorm) Infiniband equipment. The team at the AMD Developer Center runs Rocks on the multiple clusters they use to provide service for developers running on their test systems and then rebuild their clusters each time a new client requests system time. They have been an excellent resource through the Rocks list with information about cluster building and day-to-day management.
About the Science
Researchers in the mechanical engineering and aeronautics & astronautics departments at Stanford University are involved in several research programs that require large-scale massively parallel computing resources to carry out first-of-a-kind simulations. The cluster is being used to compute the details of the flow and acoustic fields created by helicopters in forward flight. For this purpose two major simulation codes, SUmb and CDP, are run simultaneously to compute the flow in separate areas of the domain: SUmb computes the flow in the region near the surface of the blades where compressibility and viscous effects are dominant, and CDP resolves the wake portion where the identification of the strength and location of the trailing vortices is of fundamental importance.
SUmb (Stanford University multi-block) and CDP (named for the late Charles David Pierce) are both mas-|sively parallel flow solvers developed at Stanford under the sponsorship of the Department of Energy's ASC program. SUmb can be used to solve for the compressible flow in many applications including, but are not limited to, jet engines, subsonic and supersonic aircraft, helicopters, launch vehicles, space and re-entry vehicles, and a host of other research applications.
SUmb uses a multi-block structured meshing approach. The mesh is decomposed into a number of pieces that are distributed to each of the processors in a calculation and the Message Passing Interface (MPI) standard is used to communicate between processors using a high-bandwidth, low-latency network (InfiniBand in the case of our Rocks Cluster).
CDP uses a fully unstructured grid approach to allow more flexibility in concentrating the grid points in regions of interest. The code was developed to simulate the multiphase reacting flow in jet engine combustors, but can be applied more generally to simulate a variety of flows where important flow structure persists for a relatively long time, such as the trailing vortices generated by the helicopter's blade tips.
For coupled simulations where SUmb needs to interact with CDP, the codes use a Python-based interface to simplify access to the data structures, while allowing the core portions of the solution to be carried out using highly optimized compiled languages. Both codes have been routinely run on thousands of processors and have been readied for computations on large parallel computers such as BlueGene/L, which is expected to reach a total of about 130,000 processors.
Resources
Subscribe to Linux Email News Alerts
Subscribe via RSS
