Welcome!

Linux Containers Authors: Liz McMillan, Elizabeth White, Automic Blog, Yeshim Deniz, XebiaLabs Blog

Related Topics: Linux Containers

Linux Containers: Article

NFS v4 Testing

Preparing for enterprise deployment

The Network File System (NFS) is an important mechanism for sharing files among end users on a broad range of platforms. End users have relied on NFS to support mission-critical applications for several decades. However, in recent years, other shared file systems have been developed to provide features that earlier versions of NFS lacked. To compete and address real end-user needs, the new rev 4 of NFS was developed. As NFS version 4 becomes available for deployment, interest in it is growing. Does it deliver on its promises? Will it introduce performance gains or stability issues compared with NFS v3? How well does it fit in existing enterprise ecosystems? The new version offers performance and security features, but may also pose risks or other challenges, and leaves various open questions for IT managers.

In speaking with a number of member companies, OSDL became aware of a widespread industry interest in seeing these questions answered through organized testing of the Linux implementation of NFS 4. Moreover, there was a strong desire among OSDL member companies to see this testing work done in open community-driven processes to encourage involvement by a broad spectrum of participants. The OSDL lab was also asked by the NFS 4 development community to help organize such testing, which began a few months ago, but has already helped make NFS 4 ready for enterprise deployment.

Initiating the Effort at OSDL
The Linux NFS v4 testing effort at the Open Source Development Labs came about through a confluence of interests. During various OSDL meetings, presentations, and activities in 2004, the OSDL Test and Performance Department was frequent questioned about NFS. Did OSDL tools and tests support network testing? Was OSDL doing file system performance tests with NFS? Did OSDL plan to help with the NFS v4 development?

Independently, when the OSDL Test and Performance Department informally asked several companies for lists of testing avenues for OSDL to pursue, NFS found its way onto everyone's list; not necessarily as item number one, but always in the top five.

The OSDL Storage Special Interest Group (SIG) and Data Center Linux (DCL) Initiative had separately identified NFS testing as a priority item and were interested in supporting the effort.

The primary reason it got such a high priority is that DCL surveyed end-user member companies running mission-critical applications and found NFS to be a cornerstone technology. In many cases, it was a crucial part of their Linux deployments. Once NFS testing was identified as a priority, the Storage SIG, with the support of Bryce Harrington of the OSDL Test and Performance group, began work to stimulate community-based testing efforts.

With such widespread interest, gaining approval for the project was straightforward.

Benefits of NFS v4
Version 4 of NFS offers features that are geared to improving both security and performance. A full list of the new benefits is available at http://nfs.sourceforge.net/. In brief it:

  • Tracks file state: Unlike prior versions of NFS, in NFS v4 file state (locking, reading, writing) is tracked between the client and server
  • Permits lease-based locking: It lets the client take ownership of a file for a period of time; it must contact the server to extend the lease
  • Allows file delegation: NFS v4 servers can let NFS v4 clients modify cached files without contacting the server until the server notes that another client needs it and issues a 'callback'
  • Implements compound RPCs (Remote Procedure Calls): Multiple NFS operations (LOOKUP, OPEN, READ, etc.) can be combined into a single RPC request, thereby minimizing network round trips.
  • Supports security flavors: A number of sophisticated security mechanisms including Kerberos 5 and SPKM3 are implemented, and APIs are available to add new security mechanisms down the road
  • Supports ACLs: On POSIX systems and Windows, NFS v4 standardizes how ACLs are used. Named attributes are also added, allowing user and group names to be accessed as strings, not just numeric IDs.
  • Combines several distinct NFS protocols: Combines stat, NLM, mount, ACL, and NFS into a single protocol specification for better compatibility with network firewalls
  • Supports file migration and replication

The Challenge for NFS v4 Testing
End users who benefit the most from these new features tend to have the highest risk aversion to changing their infrastructure in terms of downtime and troubleshooting costs. The implication is that testing NFS v4 is more important than other Open Source projects. It also presents a great opportunity: if the testing is done well, NFS 4 could be applied to new problem areas that have been outside the scope of previous NFS versions, and empower the community.

From this perspective, the objectives for testing NFSv4 were:
a) to identify problems and improvement opportunities in the implementation so developers can achieve a better product
b) to establish NFS 4 as a technology ready for end-user deployment with rapid realization of benefits from migration
c) to enable end users and community members to participate in community-oriented testing efforts so they can ensure their needs will get full visibility. This insures that testing resources are applied efficiently on high-priority areas without duplication.

Finding a Community
An important goal highlighted by the IT staff surveyed was that testing activities be community-oriented. This orientation was called out for several reasons like avoiding lock-in, allowing for open peer review of results, and enabling the testing work to be shared broadly.

Experience showed that it's extremely difficult to create an Open Source project from scratch. Besides the development or testing work that the project was formed to do, projects must recruit participants, build visibility, package the results, answer questions from outside parties, build the infrastructure, and much more. On the other hand, joining an existing project brings many of these gains automatically. So when starting the NFS v4 project, our first step was to evaluate various mailing lists and find one that would best suit OSDL testing discussions. OSDL selected and joined the [email protected] list managed by the CITI NFS Version 4 developers.

Finding the "Problem"
As newcomers to NFS v4 the OSDL team needed to figure out what to do. While the team had been given a huge number of potential action items, we still didn't know exactly where our efforts should be directed. An easy initial assumption would be that simply running various tests would be an adequate return on the testing investment. However, instead of jumping right in to running tests, OSDL engaged with the existing testers and asked them what was needed near-term.

The OSDL team noticed early on that the community didn't use a bug tracker. In discussion with the community, it was apparent that it had a high level of confidence in its ability to handle bugs through its mailing list, and we decided to postpone the bug tracker investigation.

It became clear during these discussions that beyond running tests arbitrarily, an organized approach was needed that would help identify what testing really needed to be done. We needed to know what the priorities were, whether someone was already doing those tests, and what precisely should be achieved with each test. This list of priorities needed a wide buy-in from the testers, developers, and enterprise users, and needed to be openly available to anyone for review.

The general feeling was that the real problem to solve was to gain a broad top-down prioritized list of all aspects of testing so OSDL could track who was working on which item and the status of each.

The NFS v4 Testing Matrix - Evolution of the Plan
During our initial analysis, OSDL got a huge number of e-mails, presentations, and discussions about the testing needs of NFS v4.

The community needed a way to collect and organize these disparate ideas and plans to communicate testing needs coherently. Early on Mary Edie Meredith of OSDL, the DCL roadmap coordinator, suggested a test matrix to correlate test items with test programs and reference testing resources and staff. The final and best form for this document was a list of test items in a spreadsheet that resembled a "Work Breakdown Structure"(WBS). Like a typical WBS, the NFS v4 Test Matrix organized testing tasks into a numbered hierarchy.

At the highest level, the Testing Matrix now has five broad categories:

  1. Functional testing
  2. Interoperability testing
  3. Robustness testing
  4. Performance testing
  5. Security testing
Each category is divided into sections. Section I.A covers "Standards Compliance/Conformance Verification" and I.B identifies "Regression testing," for example. Each section is broken down into various general tasks with enough granularity to be done by a single organization (see Figure 1).

The NFS Testing Matrix was then circulated among members of both the NFS v4 community and the industry-at-large for feedback, additions, and prioritization suggestions. Done initially through e-mail, we found it easier to get participation by holding weekly conference calls to go through the matrix section-by-section.

In a number of cases the team found that community members were already working on tests. Tracking this existing activity in the Test Matrix helped other testers avoid duplication. This correlation also helped identify gaps where specific kinds of tests were needed, but where the existing tests lacked the necessary coverage. This information proved especially interesting for the test authors, giving clear direction about what to add to the tests, and why.

Latest Status
Over the past few months this testing effort has generated a number of improvements to the code. Several participants working on some of the functional, robustness, and performance test items in the matrix uncovered a number of bugs; our principle of working closely with the developers has allowed these issues to be addressed quickly and closed.

As an example of this work, OSDL attended the Connectathon event in late February and chose to focus on testing the installation of NFSv4 on SuSE as a learning exercise. The version of SuSE tested used the Heimdal version of Kerberos, which hadn't been widely tested and some of the libraries had compilation and configuration issues; working directly with the developers, we were able to generate and test patches to correct the behavior. These patches were incorporated into the mainline code base shortly thereafter.

Now that we know clearly what the needs are, we're ready to engage additional volunteers in testing. The team is using a combination of approaches to solicit help. First, we're trying to drill deeper into each task to help answer questions about how to do tests. Second, through publications such as this one we'll spread awareness of the NFS v4 testing effort to potential participants. Third, we're approaching companies involved in the community to encourage them to sign up to help. Fourth, we're reaching out to end users to validate priorities and determine where their test efforts converge with our efforts. Fifth, we're reaching out to the OSDL Security SIG to help define the security section of the test matrix. Finally, for items we can't find volunteers for, we'll report these areas as issues to the OSDL DCL committee for resolution, and that group will hopefully assign the resources needed right away.

The planning mechanism we'll implement next is "'Testing Checklists." Such checklists will provide pointers to items to be tested, identify non-obvious configuration directions, and outline other things to look for. OSDL is also developing testing tools to assist testers in doing tests, collecting information, and reporting the results. OSDL hopes to act as a central collection point for the results of NFS v4 testing efforts.

Building a Community-Owned Testing Methodology
There are a number of challenges to the community approach. Unlike traditional testing, where a single company owns the process and employs the staff to do it, in wide community-driven testing it can be hard to get every area filled.

Also, with Open Source the distinction between a developer and tester is much more blurred. This can sometimes result in more emphasis being put on development than testing. For NFS v4, a balance must be struck that includes strong emphasis on both testing and development.

A third challenge is the sheer complexity of the NFS code stack. Besides the NFS client and server code in the Linux kernel, there's a surrounding layer of utilities, administrative tools, file systems, add-ons like automounter and cachefs, and authentication services. Interaction between these pieces and NFS needs thorough testing with different versions and configuration settings and huge numbers of permutations. The OSDL team hopes that careful planning will contain the scope, and that open participation to a wider community will disperse the effort.

Fortunately, these weaknesses are all areas that dovetail well with corporate testing efforts since they are areas where those organizations have strengths. Companies bring employees that can be dedicated to specific tasks, people with the specific talents needed for the task. They can scale their contributions to match their business needs, providing an effective way to address complexities - if a given company needs a particular set of interactions thoroughly tested, then the business case will exist to justify funding a testing effort to do it.

By establishing a clear, well organized, and structured testing effort in the open NFS v4 community, we'll enable these organizations to participate better in testing; they can focus on their own priorities. By encouraging them to share their results openly, NFS v4 as a whole will be improved and the testing work will eventually be done.

Call to Action
Many features need testing in NFS, but of course, no company wants to do it all alone. Our hope is that by involving a variety of companies, even if each company's contribution is small, NFS v4 will get enough testing and validation to benefit from the new NFS v4 features reasonably soon, while avoiding the frustrations of incompletely tested software.

Twice a year, the larger NFS community (including developers for non-Linux platforms) get together at Connectathon and Bakeathon events, where implementations are tested against one another in a controlled network environment. OSDL participated in the 2005 Connectathon to interact with developers face-to-face and learn about setting up, using, and testing the code. Such face-to-face opportunities are invaluable in solving problems. We have also arranged a BOF meeting at this year's Ottawa Linux Symposium (OLS) to interact with the wider Open Source community.

Please review the NFS v4Test Matrix and look for areas of interest for your own organization. If you'd like to participate or track its progress, join the NFS v4 and Storage SIG at www.developer.osdl.org/dev/nfsv4/.

Appendix: High-Priority Items in the Test Matrix
High-priority testing areas identified by the community include the following:

Functional Testing

  • Standards compliance and conformance (POSIX, NFS specs)
  • State transitions
  • "Ecosystem compatibility" - glibc, krb5, Ipsec, ACLs (POSIX & NFS), automounter
  • Compatibility with the TCP protocol
  • Automounter direct map support
  • Use Case: Database functionality on NFS
  • Use Case: Clusters/migration/replication functionality (multiple clients)
  • Use Case: Web server
Interoperability Testing
  • Interoperability with mit-krb5 and IpSec v4 protocols
  • Interoperability between 32-bit and 64-bit clients and servers
  • Interoperability between big endian and little endian
  • Interoperability of the Linux NFS client with target server architectures/platforms
  • Ext3 interoperability
  • ACL interoperability of Linux and non-Linux clients and servers
Robustness Testing
  • Running workloads for two weeks under various conditions and interuptions
  • Resource limit testing (out of memory, disk space, inode, swap space)
  • Stress load testing
  • Scalability (max number of connections and file systems)
  • Recovery from problems while under light/normal/heavy loads
  • Automounter race conditions and remounting in corner cases
Performance Testing
  • Comparison of NFS v3 and NVS v4 for common use cases
  • Evaluate performance in load scenarios
  • Scalability performance - does performance degrade gracefully?
Security Testing
  • Review security feature design and assumptions
  • Code audit
  • Attack and penetration security review

More Stories By Lynn de la Torre

Lynn de la Torre is a member of OSDL and coordinates the activities of the DCL Working Group. Lynn has thirty years of experience in the data center, and has worked in operations, system administration, database administration, and software development. Prior to joining OSDL, Lynn was a project manager for a large data warehouse implementation.

More Stories By Bryce Harrington

Bryce is a Senior Performance Engineer at the Open Source Development Labs in Beaverton, Oregon leading OSDL's NFSv4 testing efforts. He graduated with a BSc in Aerospace Engineering from the University of Southern California and an MSc in Aeronautical Engineering from the California Institute of Technology in 1995.

Comments (0)

Share your thoughts on this story.

Add your comment
You must be signed in to add a comment. Sign-in | Register

In accordance with our Comment Policy, we encourage comments that are on topic, relevant and to-the-point. We will remove comments that include profanity, personal attacks, racial slurs, threats of violence, or other inappropriate material that violates our Terms and Conditions, and will block users who make repeated violations. We ask all readers to expect diversity of opinion and to treat one another with dignity and respect.


@ThingsExpo Stories
The question before companies today is not whether to become intelligent, it’s a question of how and how fast. The key is to adopt and deploy an intelligent application strategy while simultaneously preparing to scale that intelligence. In her session at 21st Cloud Expo, Sangeeta Chakraborty, Chief Customer Officer at Ayasdi, will provide a tactical framework to become a truly intelligent enterprise, including how to identify the right applications for AI, how to build a Center of Excellence to ...
SYS-CON Events announced today that Massive Networks will exhibit at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Massive Networks mission is simple. To help your business operate seamlessly with fast, reliable, and secure internet and network solutions. Improve your customer's experience with outstanding connections to your cloud.
SYS-CON Events announced today that Datera, that offers a radically new data management architecture, has been named "Exhibitor" of SYS-CON's 21st International Cloud Expo ®, which will take place on Oct 31 - Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Datera is transforming the traditional datacenter model through modern cloud simplicity. The technology industry is at another major inflection point. The rise of mobile, the Internet of Things, data storage and Big...
Everything run by electricity will eventually be connected to the Internet. Get ahead of the Internet of Things revolution and join Akvelon expert and IoT industry leader, Sergey Grebnov, in his session at @ThingsExpo, for an educational dive into the world of managing your home, workplace and all the devices they contain with the power of machine-based AI and intelligent Bot services for a completely streamlined experience.
Because IoT devices are deployed in mission-critical environments more than ever before, it’s increasingly imperative they be truly smart. IoT sensors simply stockpiling data isn’t useful. IoT must be artificially and naturally intelligent in order to provide more value In his session at @ThingsExpo, John Crupi, Vice President and Engineering System Architect at Greenwave Systems, will discuss how IoT artificial intelligence (AI) can be carried out via edge analytics and machine learning techn...
In his session at @ThingsExpo, Arvind Radhakrishnen discussed how IoT offers new business models in banking and financial services organizations with the capability to revolutionize products, payments, channels, business processes and asset management built on strong architectural foundation. The following topics were covered: How IoT stands to impact various business parameters including customer experience, cost and risk management within BFS organizations.
SYS-CON Events announced today that CA Technologies has been named "Platinum Sponsor" of SYS-CON's 21st International Cloud Expo®, which will take place October 31-November 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. CA Technologies helps customers succeed in a future where every business - from apparel to energy - is being rewritten by software. From planning to development to management to security, CA creates software that fuels transformation for companies in the applic...
From 2013, NTT Communications has been providing cPaaS service, SkyWay. Its customer’s expectations for leveraging WebRTC technology are not only typical real-time communication use cases such as Web conference, remote education, but also IoT use cases such as remote camera monitoring, smart-glass, and robotic. Because of this, NTT Communications has numerous IoT business use-cases that its customers are developing on top of PaaS. WebRTC will lead IoT businesses to be more innovative and address...
SYS-CON Events announced today that CA Technologies has been named “Platinum Sponsor” of SYS-CON's 21st International Cloud Expo®, which will take place October 31-November 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. CA Technologies helps customers succeed in a future where every business – from apparel to energy – is being rewritten by software. From planning to development to management to security, CA creates software that fuels transformation for companies in the applic...
Internet of @ThingsExpo, taking place October 31 - November 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA, is co-located with 21st Cloud Expo and will feature technical sessions from a rock star conference faculty and the leading industry players in the world. The Internet of Things (IoT) is the most profound change in personal and enterprise IT since the creation of the Worldwide Web more than 20 years ago. All major researchers estimate there will be tens of billions devic...
WebRTC is great technology to build your own communication tools. It will be even more exciting experience it with advanced devices, such as a 360 Camera, 360 microphone, and a depth sensor camera. In his session at @ThingsExpo, Masashi Ganeko, a manager at INFOCOM Corporation, will introduce two experimental projects from his team and what they learned from them. "Shotoku Tamago" uses the robot audition software HARK to track speakers in 360 video of a remote party. "Virtual Teleport" uses a...
SYS-CON Events announced today that Calligo has been named “Bronze Sponsor” of SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Calligo is an innovative cloud service provider offering mid-sized companies the highest levels of data privacy. Calligo offers unparalleled application performance guarantees, commercial flexibility and a personalized support service from its globally located cloud platform...
SYS-CON Events announced today that Elastifile will exhibit at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 - Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Elastifile Cloud File System (ECFS) is software-defined data infrastructure designed for seamless and efficient management of dynamic workloads across heterogeneous environments. Elastifile provides the architecture needed to optimize your hybrid cloud environment, by facilitating efficient...
SYS-CON Events announced today that Cloudistics, an on-premises cloud computing company, has been named “Bronze Sponsor” of SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Launched in 2016, Cloudistics helps anyone bring the power of the cloud to the data center in an easy-to-use, on- premises cloud platform that automatically provides high performance resources for all types of applications: Docke...
SYS-CON Events announced today that Golden Gate University will exhibit at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Since 1901, non-profit Golden Gate University (GGU) has been helping adults achieve their professional goals by providing high quality, practice-based undergraduate and graduate educational programs in law, taxation, business and related professions. Many of its courses are taug...
SYS-CON Events announced today that Golden Gate University will exhibit at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Since 1901, non-profit Golden Gate University (GGU) has been helping adults achieve their professional goals by providing high quality, practice-based undergraduate and graduate educational programs in law, taxation, business and related professions. Many of its courses are taug...
SYS-CON Events announced today that Secure Channels, a cybersecurity firm, will exhibit at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Secure Channels, Inc. offers several products and solutions to its many clients, helping them protect critical data from being compromised and access to computer networks from the unauthorized. The company develops comprehensive data encryption security strategie...
Recently, WebRTC has a lot of eyes from market. The use cases of WebRTC are expanding - video chat, online education, online health care etc. Not only for human-to-human communication, but also IoT use cases such as machine to human use cases can be seen recently. One of the typical use-case is remote camera monitoring. With WebRTC, people can have interoperability and flexibility for deploying monitoring service. However, the benefit of WebRTC for IoT is not only its convenience and interopera...
SYS-CON Events announced today that SkyScale will exhibit at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. SkyScale is a world-class provider of cloud-based, ultra-fast multi-GPU hardware platforms for lease to customers desiring the fastest performance available as a service anywhere in the world. SkyScale builds, configures, and manages dedicated systems strategically located in maximum-security...
SYS-CON Events announced today that Datera will exhibit at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Datera offers a radically new approach to data management, where innovative software makes data infrastructure invisible, elastic and able to perform at the highest level. It eliminates hardware lock-in and gives IT organizations the choice to source x86 server nodes, with business model option...