Welcome!

Linux Authors: Carmen Gonzalez, Sematext Blog , Lori MacVittie, Pat Romanski, Trevor Parsons

Related Topics: Linux

Linux: Article

NFS v4 Testing

Preparing for enterprise deployment

The Network File System (NFS) is an important mechanism for sharing files among end users on a broad range of platforms. End users have relied on NFS to support mission-critical applications for several decades. However, in recent years, other shared file systems have been developed to provide features that earlier versions of NFS lacked. To compete and address real end-user needs, the new rev 4 of NFS was developed. As NFS version 4 becomes available for deployment, interest in it is growing. Does it deliver on its promises? Will it introduce performance gains or stability issues compared with NFS v3? How well does it fit in existing enterprise ecosystems? The new version offers performance and security features, but may also pose risks or other challenges, and leaves various open questions for IT managers.

In speaking with a number of member companies, OSDL became aware of a widespread industry interest in seeing these questions answered through organized testing of the Linux implementation of NFS 4. Moreover, there was a strong desire among OSDL member companies to see this testing work done in open community-driven processes to encourage involvement by a broad spectrum of participants. The OSDL lab was also asked by the NFS 4 development community to help organize such testing, which began a few months ago, but has already helped make NFS 4 ready for enterprise deployment.

Initiating the Effort at OSDL
The Linux NFS v4 testing effort at the Open Source Development Labs came about through a confluence of interests. During various OSDL meetings, presentations, and activities in 2004, the OSDL Test and Performance Department was frequent questioned about NFS. Did OSDL tools and tests support network testing? Was OSDL doing file system performance tests with NFS? Did OSDL plan to help with the NFS v4 development?

Independently, when the OSDL Test and Performance Department informally asked several companies for lists of testing avenues for OSDL to pursue, NFS found its way onto everyone's list; not necessarily as item number one, but always in the top five.

The OSDL Storage Special Interest Group (SIG) and Data Center Linux (DCL) Initiative had separately identified NFS testing as a priority item and were interested in supporting the effort.

The primary reason it got such a high priority is that DCL surveyed end-user member companies running mission-critical applications and found NFS to be a cornerstone technology. In many cases, it was a crucial part of their Linux deployments. Once NFS testing was identified as a priority, the Storage SIG, with the support of Bryce Harrington of the OSDL Test and Performance group, began work to stimulate community-based testing efforts.

With such widespread interest, gaining approval for the project was straightforward.

Benefits of NFS v4
Version 4 of NFS offers features that are geared to improving both security and performance. A full list of the new benefits is available at http://nfs.sourceforge.net/. In brief it:

  • Tracks file state: Unlike prior versions of NFS, in NFS v4 file state (locking, reading, writing) is tracked between the client and server
  • Permits lease-based locking: It lets the client take ownership of a file for a period of time; it must contact the server to extend the lease
  • Allows file delegation: NFS v4 servers can let NFS v4 clients modify cached files without contacting the server until the server notes that another client needs it and issues a 'callback'
  • Implements compound RPCs (Remote Procedure Calls): Multiple NFS operations (LOOKUP, OPEN, READ, etc.) can be combined into a single RPC request, thereby minimizing network round trips.
  • Supports security flavors: A number of sophisticated security mechanisms including Kerberos 5 and SPKM3 are implemented, and APIs are available to add new security mechanisms down the road
  • Supports ACLs: On POSIX systems and Windows, NFS v4 standardizes how ACLs are used. Named attributes are also added, allowing user and group names to be accessed as strings, not just numeric IDs.
  • Combines several distinct NFS protocols: Combines stat, NLM, mount, ACL, and NFS into a single protocol specification for better compatibility with network firewalls
  • Supports file migration and replication

The Challenge for NFS v4 Testing
End users who benefit the most from these new features tend to have the highest risk aversion to changing their infrastructure in terms of downtime and troubleshooting costs. The implication is that testing NFS v4 is more important than other Open Source projects. It also presents a great opportunity: if the testing is done well, NFS 4 could be applied to new problem areas that have been outside the scope of previous NFS versions, and empower the community.

From this perspective, the objectives for testing NFSv4 were:
a) to identify problems and improvement opportunities in the implementation so developers can achieve a better product
b) to establish NFS 4 as a technology ready for end-user deployment with rapid realization of benefits from migration
c) to enable end users and community members to participate in community-oriented testing efforts so they can ensure their needs will get full visibility. This insures that testing resources are applied efficiently on high-priority areas without duplication.

Finding a Community
An important goal highlighted by the IT staff surveyed was that testing activities be community-oriented. This orientation was called out for several reasons like avoiding lock-in, allowing for open peer review of results, and enabling the testing work to be shared broadly.

Experience showed that it's extremely difficult to create an Open Source project from scratch. Besides the development or testing work that the project was formed to do, projects must recruit participants, build visibility, package the results, answer questions from outside parties, build the infrastructure, and much more. On the other hand, joining an existing project brings many of these gains automatically. So when starting the NFS v4 project, our first step was to evaluate various mailing lists and find one that would best suit OSDL testing discussions. OSDL selected and joined the [email protected] list managed by the CITI NFS Version 4 developers.

Finding the "Problem"
As newcomers to NFS v4 the OSDL team needed to figure out what to do. While the team had been given a huge number of potential action items, we still didn't know exactly where our efforts should be directed. An easy initial assumption would be that simply running various tests would be an adequate return on the testing investment. However, instead of jumping right in to running tests, OSDL engaged with the existing testers and asked them what was needed near-term.

The OSDL team noticed early on that the community didn't use a bug tracker. In discussion with the community, it was apparent that it had a high level of confidence in its ability to handle bugs through its mailing list, and we decided to postpone the bug tracker investigation.

It became clear during these discussions that beyond running tests arbitrarily, an organized approach was needed that would help identify what testing really needed to be done. We needed to know what the priorities were, whether someone was already doing those tests, and what precisely should be achieved with each test. This list of priorities needed a wide buy-in from the testers, developers, and enterprise users, and needed to be openly available to anyone for review.

The general feeling was that the real problem to solve was to gain a broad top-down prioritized list of all aspects of testing so OSDL could track who was working on which item and the status of each.

The NFS v4 Testing Matrix - Evolution of the Plan
During our initial analysis, OSDL got a huge number of e-mails, presentations, and discussions about the testing needs of NFS v4.

The community needed a way to collect and organize these disparate ideas and plans to communicate testing needs coherently. Early on Mary Edie Meredith of OSDL, the DCL roadmap coordinator, suggested a test matrix to correlate test items with test programs and reference testing resources and staff. The final and best form for this document was a list of test items in a spreadsheet that resembled a "Work Breakdown Structure"(WBS). Like a typical WBS, the NFS v4 Test Matrix organized testing tasks into a numbered hierarchy.

At the highest level, the Testing Matrix now has five broad categories:

  1. Functional testing
  2. Interoperability testing
  3. Robustness testing
  4. Performance testing
  5. Security testing
Each category is divided into sections. Section I.A covers "Standards Compliance/Conformance Verification" and I.B identifies "Regression testing," for example. Each section is broken down into various general tasks with enough granularity to be done by a single organization (see Figure 1).

The NFS Testing Matrix was then circulated among members of both the NFS v4 community and the industry-at-large for feedback, additions, and prioritization suggestions. Done initially through e-mail, we found it easier to get participation by holding weekly conference calls to go through the matrix section-by-section.

In a number of cases the team found that community members were already working on tests. Tracking this existing activity in the Test Matrix helped other testers avoid duplication. This correlation also helped identify gaps where specific kinds of tests were needed, but where the existing tests lacked the necessary coverage. This information proved especially interesting for the test authors, giving clear direction about what to add to the tests, and why.

Latest Status
Over the past few months this testing effort has generated a number of improvements to the code. Several participants working on some of the functional, robustness, and performance test items in the matrix uncovered a number of bugs; our principle of working closely with the developers has allowed these issues to be addressed quickly and closed.

As an example of this work, OSDL attended the Connectathon event in late February and chose to focus on testing the installation of NFSv4 on SuSE as a learning exercise. The version of SuSE tested used the Heimdal version of Kerberos, which hadn't been widely tested and some of the libraries had compilation and configuration issues; working directly with the developers, we were able to generate and test patches to correct the behavior. These patches were incorporated into the mainline code base shortly thereafter.

Now that we know clearly what the needs are, we're ready to engage additional volunteers in testing. The team is using a combination of approaches to solicit help. First, we're trying to drill deeper into each task to help answer questions about how to do tests. Second, through publications such as this one we'll spread awareness of the NFS v4 testing effort to potential participants. Third, we're approaching companies involved in the community to encourage them to sign up to help. Fourth, we're reaching out to end users to validate priorities and determine where their test efforts converge with our efforts. Fifth, we're reaching out to the OSDL Security SIG to help define the security section of the test matrix. Finally, for items we can't find volunteers for, we'll report these areas as issues to the OSDL DCL committee for resolution, and that group will hopefully assign the resources needed right away.

The planning mechanism we'll implement next is "'Testing Checklists." Such checklists will provide pointers to items to be tested, identify non-obvious configuration directions, and outline other things to look for. OSDL is also developing testing tools to assist testers in doing tests, collecting information, and reporting the results. OSDL hopes to act as a central collection point for the results of NFS v4 testing efforts.

Building a Community-Owned Testing Methodology
There are a number of challenges to the community approach. Unlike traditional testing, where a single company owns the process and employs the staff to do it, in wide community-driven testing it can be hard to get every area filled.

Also, with Open Source the distinction between a developer and tester is much more blurred. This can sometimes result in more emphasis being put on development than testing. For NFS v4, a balance must be struck that includes strong emphasis on both testing and development.

A third challenge is the sheer complexity of the NFS code stack. Besides the NFS client and server code in the Linux kernel, there's a surrounding layer of utilities, administrative tools, file systems, add-ons like automounter and cachefs, and authentication services. Interaction between these pieces and NFS needs thorough testing with different versions and configuration settings and huge numbers of permutations. The OSDL team hopes that careful planning will contain the scope, and that open participation to a wider community will disperse the effort.

Fortunately, these weaknesses are all areas that dovetail well with corporate testing efforts since they are areas where those organizations have strengths. Companies bring employees that can be dedicated to specific tasks, people with the specific talents needed for the task. They can scale their contributions to match their business needs, providing an effective way to address complexities - if a given company needs a particular set of interactions thoroughly tested, then the business case will exist to justify funding a testing effort to do it.

By establishing a clear, well organized, and structured testing effort in the open NFS v4 community, we'll enable these organizations to participate better in testing; they can focus on their own priorities. By encouraging them to share their results openly, NFS v4 as a whole will be improved and the testing work will eventually be done.

Call to Action
Many features need testing in NFS, but of course, no company wants to do it all alone. Our hope is that by involving a variety of companies, even if each company's contribution is small, NFS v4 will get enough testing and validation to benefit from the new NFS v4 features reasonably soon, while avoiding the frustrations of incompletely tested software.

Twice a year, the larger NFS community (including developers for non-Linux platforms) get together at Connectathon and Bakeathon events, where implementations are tested against one another in a controlled network environment. OSDL participated in the 2005 Connectathon to interact with developers face-to-face and learn about setting up, using, and testing the code. Such face-to-face opportunities are invaluable in solving problems. We have also arranged a BOF meeting at this year's Ottawa Linux Symposium (OLS) to interact with the wider Open Source community.

Please review the NFS v4Test Matrix and look for areas of interest for your own organization. If you'd like to participate or track its progress, join the NFS v4 and Storage SIG at www.developer.osdl.org/dev/nfsv4/.

Appendix: High-Priority Items in the Test Matrix
High-priority testing areas identified by the community include the following:

Functional Testing

  • Standards compliance and conformance (POSIX, NFS specs)
  • State transitions
  • "Ecosystem compatibility" - glibc, krb5, Ipsec, ACLs (POSIX & NFS), automounter
  • Compatibility with the TCP protocol
  • Automounter direct map support
  • Use Case: Database functionality on NFS
  • Use Case: Clusters/migration/replication functionality (multiple clients)
  • Use Case: Web server
Interoperability Testing
  • Interoperability with mit-krb5 and IpSec v4 protocols
  • Interoperability between 32-bit and 64-bit clients and servers
  • Interoperability between big endian and little endian
  • Interoperability of the Linux NFS client with target server architectures/platforms
  • Ext3 interoperability
  • ACL interoperability of Linux and non-Linux clients and servers
Robustness Testing
  • Running workloads for two weeks under various conditions and interuptions
  • Resource limit testing (out of memory, disk space, inode, swap space)
  • Stress load testing
  • Scalability (max number of connections and file systems)
  • Recovery from problems while under light/normal/heavy loads
  • Automounter race conditions and remounting in corner cases
Performance Testing
  • Comparison of NFS v3 and NVS v4 for common use cases
  • Evaluate performance in load scenarios
  • Scalability performance - does performance degrade gracefully?
Security Testing
  • Review security feature design and assumptions
  • Code audit
  • Attack and penetration security review

More Stories By Lynn de la Torre

Lynn de la Torre is a member of OSDL and coordinates the activities of the DCL Working Group. Lynn has thirty years of experience in the data center, and has worked in operations, system administration, database administration, and software development. Prior to joining OSDL, Lynn was a project manager for a large data warehouse implementation.

More Stories By Bryce Harrington

Bryce is a Senior Performance Engineer at the Open Source Development Labs in Beaverton, Oregon leading OSDL's NFSv4 testing efforts. He graduated with a BSc in Aerospace Engineering from the University of Southern California and an MSc in Aeronautical Engineering from the California Institute of Technology in 1995.

Comments (0)

Share your thoughts on this story.

Add your comment
You must be signed in to add a comment. Sign-in | Register

In accordance with our Comment Policy, we encourage comments that are on topic, relevant and to-the-point. We will remove comments that include profanity, personal attacks, racial slurs, threats of violence, or other inappropriate material that violates our Terms and Conditions, and will block users who make repeated violations. We ask all readers to expect diversity of opinion and to treat one another with dignity and respect.


@ThingsExpo Stories
SYS-CON Events announced today that Windstream, a leading provider of advanced network and cloud communications, has been named “Silver Sponsor” of SYS-CON's 16th International Cloud Expo®, which will take place on June 9–11, 2015, at the Javits Center in New York, NY. Windstream (Nasdaq: WIN), a FORTUNE 500 and S&P 500 company, is a leading provider of advanced network communications, including cloud computing and managed services, to businesses nationwide. The company also offers broadband, phone and digital TV services to consumers primarily in rural areas.

ARMONK, N.Y., Nov. 20, 2014 /PRNewswire/ --  IBM (NYSE: IBM) today announced that it is bringing a greater level of control, security and flexibility to cloud-based application development and delivery with a single-tenant version of Bluemix, IBM's platform-as-a-service. The new platform enables developers to build ap...

The BPM world is going through some evolution or changes where traditional business process management solutions really have nowhere to go in terms of development of the road map. In this demo at 15th Cloud Expo, Kyle Hansen, Director of Professional Services at AgilePoint, shows AgilePoint’s unique approach to dealing with this market circumstance by developing a rapid application composition or development framework.
"BSQUARE is in the business of selling software solutions for smart connected devices. It's obvious that IoT has moved from being a technology to being a fundamental part of business, and in the last 18 months people have said let's figure out how to do it and let's put some focus on it, " explained Dave Wagstaff, VP & Chief Architect, at BSQUARE Corporation, in this SYS-CON.tv interview at @ThingsExpo, held Nov 4-6, 2014, at the Santa Clara Convention Center in Santa Clara, CA.
The major cloud platforms defy a simple, side-by-side analysis. Each of the major IaaS public-cloud platforms offers their own unique strengths and functionality. Options for on-site private cloud are diverse as well, and must be designed and deployed while taking existing legacy architecture and infrastructure into account. Then the reality is that most enterprises are embarking on a hybrid cloud strategy and programs. In this Power Panel at 15th Cloud Expo (http://www.CloudComputingExpo.com), moderated by Ashar Baig, Research Director, Cloud, at Gigaom Research, Nate Gordon, Director of T...
The Internet of Things is not new. Historically, smart businesses have used its basic concept of leveraging data to drive better decision making and have capitalized on those insights to realize additional revenue opportunities. So, what has changed to make the Internet of Things one of the hottest topics in tech? In his session at @ThingsExpo, Chris Gray, Director, Embedded and Internet of Things, discussed the underlying factors that are driving the economics of intelligent systems. Discover how hardware commoditization, the ubiquitous nature of connectivity, and the emergence of Big Data a...
“In the past year we've seen a lot of stabilization of WebRTC. You can now use it in production with a far greater degree of certainty. A lot of the real developments in the past year have been in things like the data channel, which will enable a whole new type of application," explained Peter Dunkley, Technical Director at Acision, in this SYS-CON.tv interview at @ThingsExpo, held Nov 4–6, 2014, at the Santa Clara Convention Center in Santa Clara, CA.
SYS-CON Events announced today that IDenticard will exhibit at SYS-CON's 16th International Cloud Expo®, which will take place on June 9-11, 2015, at the Javits Center in New York City, NY. IDenticard™ is the security division of Brady Corp (NYSE: BRC), a $1.5 billion manufacturer of identification products. We have small-company values with the strength and stability of a major corporation. IDenticard offers local sales, support and service to our customers across the United States and Canada. Our partner network encompasses some 300 of the world's leading systems integrators and security s...
DevOps Summit 2015 New York, co-located with the 16th International Cloud Expo - to be held June 9-11, 2015, at the Javits Center in New York City, NY - announces that it is now accepting Keynote Proposals. The widespread success of cloud computing is driving the DevOps revolution in enterprise IT. Now as never before, development teams must communicate and collaborate in a dynamic, 24/7/365 environment. There is no time to wait for long development cycles that produce software that is obsolete at launch. DevOps may be disruptive, but it is essential.
"People are a lot more knowledgeable about APIs now. There are two types of people who work with APIs - IT people who want to use APIs for something internal and the product managers who want to do something outside APIs for people to connect to them," explained Roberto Medrano, Executive Vice President at SOA Software, in this SYS-CON.tv interview at Cloud Expo, held Nov 4–6, 2014, at the Santa Clara Convention Center in Santa Clara, CA.
Nigeria has the largest economy in Africa, at more than US$500 billion, and ranks 23rd in the world. A recent re-evaluation of Nigeria's true economic size doubled the previous estimate, and brought it well ahead of South Africa, which is a member (unlike Nigeria) of the G20 club for political as well as economic reasons. Nigeria's economy can be said to be quite diverse from one point of view, but heavily dependent on oil and gas at the same time. Oil and natural gas account for about 15% of Nigera's overall economy, but traditionally represent more than 90% of the country's exports and as...
The Internet of Things is a misnomer. That implies that everything is on the Internet, and that simply should not be - especially for things that are blurring the line between medical devices that stimulate like a pacemaker and quantified self-sensors like a pedometer or pulse tracker. The mesh of things that we manage must be segmented into zones of trust for sensing data, transmitting data, receiving command and control administrative changes, and peer-to-peer mesh messaging. In his session at @ThingsExpo, Ryan Bagnulo, Solution Architect / Software Engineer at SOA Software, focused on desi...
"At our booth we are showing how to provide trust in the Internet of Things. Trust is where everything starts to become secure and trustworthy. Now with the scaling of the Internet of Things it becomes an interesting question – I've heard numbers from 200 billion devices next year up to a trillion in the next 10 to 15 years," explained Johannes Lintzen, Vice President of Sales at Utimaco, in this SYS-CON.tv interview at @ThingsExpo, held Nov 4–6, 2014, at the Santa Clara Convention Center in Santa Clara, CA.
"For over 25 years we have been working with a lot of enterprise customers and we have seen how companies create applications. And now that we have moved to cloud computing, mobile, social and the Internet of Things, we see that the market needs a new way of creating applications," stated Jesse Shiah, CEO, President and Co-Founder of AgilePoint Inc., in this SYS-CON.tv interview at 15th Cloud Expo, held Nov 4–6, 2014, at the Santa Clara Convention Center in Santa Clara, CA.
SYS-CON Events announced today that Gridstore™, the leader in hyper-converged infrastructure purpose-built to optimize Microsoft workloads, will exhibit at SYS-CON's 16th International Cloud Expo®, which will take place on June 9-11, 2015, at the Javits Center in New York City, NY. Gridstore™ is the leader in hyper-converged infrastructure purpose-built for Microsoft workloads and designed to accelerate applications in virtualized environments. Gridstore’s hyper-converged infrastructure is the industry’s first all flash version of HyperConverged Appliances that include both compute and storag...
Today’s enterprise is being driven by disruptive competitive and human capital requirements to provide enterprise application access through not only desktops, but also mobile devices. To retrofit existing programs across all these devices using traditional programming methods is very costly and time consuming – often prohibitively so. In his session at @ThingsExpo, Jesse Shiah, CEO, President, and Co-Founder of AgilePoint Inc., discussed how you can create applications that run on all mobile devices as well as laptops and desktops using a visual drag-and-drop application – and eForms-buildi...
We certainly live in interesting technological times. And no more interesting than the current competing IoT standards for connectivity. Various standards bodies, approaches, and ecosystems are vying for mindshare and positioning for a competitive edge. It is clear that when the dust settles, we will have new protocols, evolved protocols, that will change the way we interact with devices and infrastructure. We will also have evolved web protocols, like HTTP/2, that will be changing the very core of our infrastructures. At the same time, we have old approaches made new again like micro-services...
Code Halos - aka "digital fingerprints" - are the key organizing principle to understand a) how dumb things become smart and b) how to monetize this dynamic. In his session at @ThingsExpo, Robert Brown, AVP, Center for the Future of Work at Cognizant Technology Solutions, outlined research, analysis and recommendations from his recently published book on this phenomena on the way leading edge organizations like GE and Disney are unlocking the Internet of Things opportunity and what steps your organization should be taking to position itself for the next platform of digital competition.
The 3rd International Internet of @ThingsExpo, co-located with the 16th International Cloud Expo - to be held June 9-11, 2015, at the Javits Center in New York City, NY - announces that its Call for Papers is now open. The Internet of Things (IoT) is the biggest idea since the creation of the Worldwide Web more than 20 years ago.
As the Internet of Things unfolds, mobile and wearable devices are blurring the line between physical and digital, integrating ever more closely with our interests, our routines, our daily lives. Contextual computing and smart, sensor-equipped spaces bring the potential to walk through a world that recognizes us and responds accordingly. We become continuous transmitters and receivers of data. In his session at @ThingsExpo, Andrew Bolwell, Director of Innovation for HP's Printing and Personal Systems Group, discussed how key attributes of mobile technology – touch input, sensors, social, and ...