Click here to close now.

Welcome!

Linux Authors: Liz McMillan, AppDynamics Blog, Pat Romanski, Elizabeth White, Carmen Gonzalez

Related Topics: Linux

Linux: Article

NFS v4 Testing

Preparing for enterprise deployment

The Network File System (NFS) is an important mechanism for sharing files among end users on a broad range of platforms. End users have relied on NFS to support mission-critical applications for several decades. However, in recent years, other shared file systems have been developed to provide features that earlier versions of NFS lacked. To compete and address real end-user needs, the new rev 4 of NFS was developed. As NFS version 4 becomes available for deployment, interest in it is growing. Does it deliver on its promises? Will it introduce performance gains or stability issues compared with NFS v3? How well does it fit in existing enterprise ecosystems? The new version offers performance and security features, but may also pose risks or other challenges, and leaves various open questions for IT managers.

In speaking with a number of member companies, OSDL became aware of a widespread industry interest in seeing these questions answered through organized testing of the Linux implementation of NFS 4. Moreover, there was a strong desire among OSDL member companies to see this testing work done in open community-driven processes to encourage involvement by a broad spectrum of participants. The OSDL lab was also asked by the NFS 4 development community to help organize such testing, which began a few months ago, but has already helped make NFS 4 ready for enterprise deployment.

Initiating the Effort at OSDL
The Linux NFS v4 testing effort at the Open Source Development Labs came about through a confluence of interests. During various OSDL meetings, presentations, and activities in 2004, the OSDL Test and Performance Department was frequent questioned about NFS. Did OSDL tools and tests support network testing? Was OSDL doing file system performance tests with NFS? Did OSDL plan to help with the NFS v4 development?

Independently, when the OSDL Test and Performance Department informally asked several companies for lists of testing avenues for OSDL to pursue, NFS found its way onto everyone's list; not necessarily as item number one, but always in the top five.

The OSDL Storage Special Interest Group (SIG) and Data Center Linux (DCL) Initiative had separately identified NFS testing as a priority item and were interested in supporting the effort.

The primary reason it got such a high priority is that DCL surveyed end-user member companies running mission-critical applications and found NFS to be a cornerstone technology. In many cases, it was a crucial part of their Linux deployments. Once NFS testing was identified as a priority, the Storage SIG, with the support of Bryce Harrington of the OSDL Test and Performance group, began work to stimulate community-based testing efforts.

With such widespread interest, gaining approval for the project was straightforward.

Benefits of NFS v4
Version 4 of NFS offers features that are geared to improving both security and performance. A full list of the new benefits is available at http://nfs.sourceforge.net/. In brief it:

  • Tracks file state: Unlike prior versions of NFS, in NFS v4 file state (locking, reading, writing) is tracked between the client and server
  • Permits lease-based locking: It lets the client take ownership of a file for a period of time; it must contact the server to extend the lease
  • Allows file delegation: NFS v4 servers can let NFS v4 clients modify cached files without contacting the server until the server notes that another client needs it and issues a 'callback'
  • Implements compound RPCs (Remote Procedure Calls): Multiple NFS operations (LOOKUP, OPEN, READ, etc.) can be combined into a single RPC request, thereby minimizing network round trips.
  • Supports security flavors: A number of sophisticated security mechanisms including Kerberos 5 and SPKM3 are implemented, and APIs are available to add new security mechanisms down the road
  • Supports ACLs: On POSIX systems and Windows, NFS v4 standardizes how ACLs are used. Named attributes are also added, allowing user and group names to be accessed as strings, not just numeric IDs.
  • Combines several distinct NFS protocols: Combines stat, NLM, mount, ACL, and NFS into a single protocol specification for better compatibility with network firewalls
  • Supports file migration and replication

The Challenge for NFS v4 Testing
End users who benefit the most from these new features tend to have the highest risk aversion to changing their infrastructure in terms of downtime and troubleshooting costs. The implication is that testing NFS v4 is more important than other Open Source projects. It also presents a great opportunity: if the testing is done well, NFS 4 could be applied to new problem areas that have been outside the scope of previous NFS versions, and empower the community.

From this perspective, the objectives for testing NFSv4 were:
a) to identify problems and improvement opportunities in the implementation so developers can achieve a better product
b) to establish NFS 4 as a technology ready for end-user deployment with rapid realization of benefits from migration
c) to enable end users and community members to participate in community-oriented testing efforts so they can ensure their needs will get full visibility. This insures that testing resources are applied efficiently on high-priority areas without duplication.

Finding a Community
An important goal highlighted by the IT staff surveyed was that testing activities be community-oriented. This orientation was called out for several reasons like avoiding lock-in, allowing for open peer review of results, and enabling the testing work to be shared broadly.

Experience showed that it's extremely difficult to create an Open Source project from scratch. Besides the development or testing work that the project was formed to do, projects must recruit participants, build visibility, package the results, answer questions from outside parties, build the infrastructure, and much more. On the other hand, joining an existing project brings many of these gains automatically. So when starting the NFS v4 project, our first step was to evaluate various mailing lists and find one that would best suit OSDL testing discussions. OSDL selected and joined the [email protected] list managed by the CITI NFS Version 4 developers.

Finding the "Problem"
As newcomers to NFS v4 the OSDL team needed to figure out what to do. While the team had been given a huge number of potential action items, we still didn't know exactly where our efforts should be directed. An easy initial assumption would be that simply running various tests would be an adequate return on the testing investment. However, instead of jumping right in to running tests, OSDL engaged with the existing testers and asked them what was needed near-term.

The OSDL team noticed early on that the community didn't use a bug tracker. In discussion with the community, it was apparent that it had a high level of confidence in its ability to handle bugs through its mailing list, and we decided to postpone the bug tracker investigation.

It became clear during these discussions that beyond running tests arbitrarily, an organized approach was needed that would help identify what testing really needed to be done. We needed to know what the priorities were, whether someone was already doing those tests, and what precisely should be achieved with each test. This list of priorities needed a wide buy-in from the testers, developers, and enterprise users, and needed to be openly available to anyone for review.

The general feeling was that the real problem to solve was to gain a broad top-down prioritized list of all aspects of testing so OSDL could track who was working on which item and the status of each.

The NFS v4 Testing Matrix - Evolution of the Plan
During our initial analysis, OSDL got a huge number of e-mails, presentations, and discussions about the testing needs of NFS v4.

The community needed a way to collect and organize these disparate ideas and plans to communicate testing needs coherently. Early on Mary Edie Meredith of OSDL, the DCL roadmap coordinator, suggested a test matrix to correlate test items with test programs and reference testing resources and staff. The final and best form for this document was a list of test items in a spreadsheet that resembled a "Work Breakdown Structure"(WBS). Like a typical WBS, the NFS v4 Test Matrix organized testing tasks into a numbered hierarchy.

At the highest level, the Testing Matrix now has five broad categories:

  1. Functional testing
  2. Interoperability testing
  3. Robustness testing
  4. Performance testing
  5. Security testing
Each category is divided into sections. Section I.A covers "Standards Compliance/Conformance Verification" and I.B identifies "Regression testing," for example. Each section is broken down into various general tasks with enough granularity to be done by a single organization (see Figure 1).

The NFS Testing Matrix was then circulated among members of both the NFS v4 community and the industry-at-large for feedback, additions, and prioritization suggestions. Done initially through e-mail, we found it easier to get participation by holding weekly conference calls to go through the matrix section-by-section.

In a number of cases the team found that community members were already working on tests. Tracking this existing activity in the Test Matrix helped other testers avoid duplication. This correlation also helped identify gaps where specific kinds of tests were needed, but where the existing tests lacked the necessary coverage. This information proved especially interesting for the test authors, giving clear direction about what to add to the tests, and why.

Latest Status
Over the past few months this testing effort has generated a number of improvements to the code. Several participants working on some of the functional, robustness, and performance test items in the matrix uncovered a number of bugs; our principle of working closely with the developers has allowed these issues to be addressed quickly and closed.

As an example of this work, OSDL attended the Connectathon event in late February and chose to focus on testing the installation of NFSv4 on SuSE as a learning exercise. The version of SuSE tested used the Heimdal version of Kerberos, which hadn't been widely tested and some of the libraries had compilation and configuration issues; working directly with the developers, we were able to generate and test patches to correct the behavior. These patches were incorporated into the mainline code base shortly thereafter.

Now that we know clearly what the needs are, we're ready to engage additional volunteers in testing. The team is using a combination of approaches to solicit help. First, we're trying to drill deeper into each task to help answer questions about how to do tests. Second, through publications such as this one we'll spread awareness of the NFS v4 testing effort to potential participants. Third, we're approaching companies involved in the community to encourage them to sign up to help. Fourth, we're reaching out to end users to validate priorities and determine where their test efforts converge with our efforts. Fifth, we're reaching out to the OSDL Security SIG to help define the security section of the test matrix. Finally, for items we can't find volunteers for, we'll report these areas as issues to the OSDL DCL committee for resolution, and that group will hopefully assign the resources needed right away.

The planning mechanism we'll implement next is "'Testing Checklists." Such checklists will provide pointers to items to be tested, identify non-obvious configuration directions, and outline other things to look for. OSDL is also developing testing tools to assist testers in doing tests, collecting information, and reporting the results. OSDL hopes to act as a central collection point for the results of NFS v4 testing efforts.

Building a Community-Owned Testing Methodology
There are a number of challenges to the community approach. Unlike traditional testing, where a single company owns the process and employs the staff to do it, in wide community-driven testing it can be hard to get every area filled.

Also, with Open Source the distinction between a developer and tester is much more blurred. This can sometimes result in more emphasis being put on development than testing. For NFS v4, a balance must be struck that includes strong emphasis on both testing and development.

A third challenge is the sheer complexity of the NFS code stack. Besides the NFS client and server code in the Linux kernel, there's a surrounding layer of utilities, administrative tools, file systems, add-ons like automounter and cachefs, and authentication services. Interaction between these pieces and NFS needs thorough testing with different versions and configuration settings and huge numbers of permutations. The OSDL team hopes that careful planning will contain the scope, and that open participation to a wider community will disperse the effort.

Fortunately, these weaknesses are all areas that dovetail well with corporate testing efforts since they are areas where those organizations have strengths. Companies bring employees that can be dedicated to specific tasks, people with the specific talents needed for the task. They can scale their contributions to match their business needs, providing an effective way to address complexities - if a given company needs a particular set of interactions thoroughly tested, then the business case will exist to justify funding a testing effort to do it.

By establishing a clear, well organized, and structured testing effort in the open NFS v4 community, we'll enable these organizations to participate better in testing; they can focus on their own priorities. By encouraging them to share their results openly, NFS v4 as a whole will be improved and the testing work will eventually be done.

Call to Action
Many features need testing in NFS, but of course, no company wants to do it all alone. Our hope is that by involving a variety of companies, even if each company's contribution is small, NFS v4 will get enough testing and validation to benefit from the new NFS v4 features reasonably soon, while avoiding the frustrations of incompletely tested software.

Twice a year, the larger NFS community (including developers for non-Linux platforms) get together at Connectathon and Bakeathon events, where implementations are tested against one another in a controlled network environment. OSDL participated in the 2005 Connectathon to interact with developers face-to-face and learn about setting up, using, and testing the code. Such face-to-face opportunities are invaluable in solving problems. We have also arranged a BOF meeting at this year's Ottawa Linux Symposium (OLS) to interact with the wider Open Source community.

Please review the NFS v4Test Matrix and look for areas of interest for your own organization. If you'd like to participate or track its progress, join the NFS v4 and Storage SIG at www.developer.osdl.org/dev/nfsv4/.

Appendix: High-Priority Items in the Test Matrix
High-priority testing areas identified by the community include the following:

Functional Testing

  • Standards compliance and conformance (POSIX, NFS specs)
  • State transitions
  • "Ecosystem compatibility" - glibc, krb5, Ipsec, ACLs (POSIX & NFS), automounter
  • Compatibility with the TCP protocol
  • Automounter direct map support
  • Use Case: Database functionality on NFS
  • Use Case: Clusters/migration/replication functionality (multiple clients)
  • Use Case: Web server
Interoperability Testing
  • Interoperability with mit-krb5 and IpSec v4 protocols
  • Interoperability between 32-bit and 64-bit clients and servers
  • Interoperability between big endian and little endian
  • Interoperability of the Linux NFS client with target server architectures/platforms
  • Ext3 interoperability
  • ACL interoperability of Linux and non-Linux clients and servers
Robustness Testing
  • Running workloads for two weeks under various conditions and interuptions
  • Resource limit testing (out of memory, disk space, inode, swap space)
  • Stress load testing
  • Scalability (max number of connections and file systems)
  • Recovery from problems while under light/normal/heavy loads
  • Automounter race conditions and remounting in corner cases
Performance Testing
  • Comparison of NFS v3 and NVS v4 for common use cases
  • Evaluate performance in load scenarios
  • Scalability performance - does performance degrade gracefully?
Security Testing
  • Review security feature design and assumptions
  • Code audit
  • Attack and penetration security review

More Stories By Lynn de la Torre

Lynn de la Torre is a member of OSDL and coordinates the activities of the DCL Working Group. Lynn has thirty years of experience in the data center, and has worked in operations, system administration, database administration, and software development. Prior to joining OSDL, Lynn was a project manager for a large data warehouse implementation.

More Stories By Bryce Harrington

Bryce is a Senior Performance Engineer at the Open Source Development Labs in Beaverton, Oregon leading OSDL's NFSv4 testing efforts. He graduated with a BSc in Aerospace Engineering from the University of Southern California and an MSc in Aeronautical Engineering from the California Institute of Technology in 1995.

Comments (0)

Share your thoughts on this story.

Add your comment
You must be signed in to add a comment. Sign-in | Register

In accordance with our Comment Policy, we encourage comments that are on topic, relevant and to-the-point. We will remove comments that include profanity, personal attacks, racial slurs, threats of violence, or other inappropriate material that violates our Terms and Conditions, and will block users who make repeated violations. We ask all readers to expect diversity of opinion and to treat one another with dignity and respect.


@ThingsExpo Stories
The 17th International Cloud Expo has announced that its Call for Papers is open. 17th International Cloud Expo, to be held November 3-5, 2015, at the Santa Clara Convention Center in Santa Clara, CA, brings together Cloud Computing, APM, APIs, Microservices, Security, Big Data, Internet of Things, DevOps and WebRTC to one location. With cloud computing driving a higher percentage of enterprise IT budgets every year, it becomes increasingly important to plant your flag in this fast-expanding business opportunity. Submit your speaking proposal today!
Hadoop as a Service (as offered by handful of niche vendors now) is a cloud computing solution that makes medium and large-scale data processing accessible, easy, fast and inexpensive. In his session at Big Data Expo, Kumar Ramamurthy, Vice President and Chief Technologist, EIM & Big Data, at Virtusa, will discuss how this is achieved by eliminating the operational challenges of running Hadoop, so one can focus on business growth. The fragmented Hadoop distribution world and various PaaS solutions that provide a Hadoop flavor either make choices for customers very flexible in the name of opti...
Cultural, regulatory, environmental, political and economic (CREPE) conditions over the past decade are creating cross-industry solution spaces that require processes and technologies from both the Internet of Things (IoT), and Data Management and Analytics (DMA). These solution spaces are evolving into Sensor Analytics Ecosystems (SAE) that represent significant new opportunities for organizations of all types. Public Utilities throughout the world, providing electricity, natural gas and water, are pursuing SmartGrid initiatives that represent one of the more mature examples of SAE. We have s...
The Internet of Things will put IT to its ultimate test by creating infinite new opportunities to digitize products and services, generate and analyze new data to improve customer satisfaction, and discover new ways to gain a competitive advantage across nearly every industry. In order to help corporate business units to capitalize on the rapidly evolving IoT opportunities, IT must stand up to a new set of challenges. In his session at @ThingsExpo, Jeff Kaplan, Managing Director of THINKstrategies, will examine why IT must finally fulfill its role in support of its SBUs or face a new round of...
One of the biggest challenges when developing connected devices is identifying user value and delivering it through successful user experiences. In his session at Internet of @ThingsExpo, Mike Kuniavsky, Principal Scientist, Innovation Services at PARC, described an IoT-specific approach to user experience design that combines approaches from interaction design, industrial design and service design to create experiences that go beyond simple connected gadgets to create lasting, multi-device experiences grounded in people's real needs and desires.
The 4th International Internet of @ThingsExpo, co-located with the 17th International Cloud Expo - to be held November 3-5, 2015, at the Santa Clara Convention Center in Santa Clara, CA - announces that its Call for Papers is open. The Internet of Things (IoT) is the biggest idea since the creation of the Worldwide Web more than 20 years ago.
Wearable devices have come of age. The primary applications of wearables so far have been "the Quantified Self" or the tracking of one's fitness and health status. We propose the evolution of wearables into social and emotional communication devices. Our BE(tm) sensor uses light to visualize the skin conductance response. Our sensors are very inexpensive and can be massively distributed to audiences or groups of any size, in order to gauge reactions to performances, video, or any kind of presentation. In her session at @ThingsExpo, Jocelyn Scheirer, CEO & Founder of Bionolux, will discuss ho...
The true value of the Internet of Things (IoT) lies not just in the data, but through the services that protect the data, perform the analysis and present findings in a usable way. With many IoT elements rooted in traditional IT components, Big Data and IoT isn’t just a play for enterprise. In fact, the IoT presents SMBs with the prospect of launching entirely new activities and exploring innovative areas. CompTIA research identifies several areas where IoT is expected to have the greatest impact.
Can call centers hang up the phones for good? Intuitive Solutions did. WebRTC enabled this contact center provider to eliminate antiquated telephony and desktop phone infrastructure with a pure web-based solution, allowing them to expand beyond brick-and-mortar confines to a home-based agent model. It also ensured scalability and better service for customers, including MUY! Companies, one of the country's largest franchise restaurant companies with 232 Pizza Hut locations. This is one example of WebRTC adoption today, but the potential is limitless when powered by IoT.
Every day we read jaw-dropping stats on the explosion of data. We allocate significant resources to harness and better understand it. We build businesses around it. But we’ve only just begun. For big payoffs in Big Data, CIOs are turning to cognitive computing. Cognitive computing’s ability to securely extract insights, understand natural language, and get smarter each time it’s used is the next, logical step for Big Data.
The Internet of Things will greatly expand the opportunities for data collection and new business models driven off of that data. In her session at @ThingsExpo, Esmeralda Swartz, CMO of MetraTech, discussed how for this to be effective you not only need to have infrastructure and operational models capable of utilizing this new phenomenon, but increasingly service providers will need to convince a skeptical public to participate. Get ready to show them the money!
SYS-CON Events announced today that MetraTech, now part of Ericsson, has been named “Silver Sponsor” of SYS-CON's 16th International Cloud Expo®, which will take place on June 9–11, 2015, at the Javits Center in New York, NY. Ericsson is the driving force behind the Networked Society- a world leader in communications infrastructure, software and services. Some 40% of the world’s mobile traffic runs through networks Ericsson has supplied, serving more than 2.5 billion subscribers.
17th Cloud Expo, taking place Nov 3-5, 2015, at the Santa Clara Convention Center in Santa Clara, CA, will feature technical sessions from a rock star conference faculty and the leading industry players in the world. Cloud computing is now being embraced by a majority of enterprises of all sizes. Yesterday's debate about public vs. private has transformed into the reality of hybrid cloud: a recent survey shows that 74% of enterprises have a hybrid cloud strategy. Meanwhile, 94% of enterprises are using some form of XaaS – software, platform, and infrastructure as a service.
The Internet of Things is not new. Historically, smart businesses have used its basic concept of leveraging data to drive better decision making and have capitalized on those insights to realize additional revenue opportunities. So, what has changed to make the Internet of Things one of the hottest topics in tech? In his session at @ThingsExpo, Chris Gray, Director, Embedded and Internet of Things, discussed the underlying factors that are driving the economics of intelligent systems. Discover how hardware commoditization, the ubiquitous nature of connectivity, and the emergence of Big Data a...
With major technology companies and startups seriously embracing IoT strategies, now is the perfect time to attend @ThingsExpo in Silicon Valley. Learn what is going on, contribute to the discussions, and ensure that your enterprise is as "IoT-Ready" as it can be! Internet of @ThingsExpo, taking place Nov 3-5, 2015, at the Santa Clara Convention Center in Santa Clara, CA, is co-located with 17th Cloud Expo and will feature technical sessions from a rock star conference faculty and the leading industry players in the world. The Internet of Things (IoT) is the most profound change in personal an...
All major researchers estimate there will be tens of billions devices - computers, smartphones, tablets, and sensors - connected to the Internet by 2020. This number will continue to grow at a rapid pace for the next several decades. With major technology companies and startups seriously embracing IoT strategies, now is the perfect time to attend @ThingsExpo, June 9-11, 2015, at the Javits Center in New York City. Learn what is going on, contribute to the discussions, and ensure that your enterprise is as "IoT-Ready" as it can be
Container frameworks, such as Docker, provide a variety of benefits, including density of deployment across infrastructure, convenience for application developers to push updates with low operational hand-holding, and a fairly well-defined deployment workflow that can be orchestrated. Container frameworks also enable a DevOps approach to application development by cleanly separating concerns between operations and development teams. But running multi-container, multi-server apps with containers is very hard. You have to learn five new and different technologies and best practices (libswarm, sy...
SYS-CON Events announced today that DragonGlass, an enterprise search platform, will exhibit at SYS-CON's 16th International Cloud Expo®, which will take place on June 9-11, 2015, at the Javits Center in New York City, NY. After eleven years of designing and building custom applications, OpenCrowd has launched DragonGlass, a cloud-based platform that enables the development of search-based applications. These are a new breed of applications that utilize a search index as their backbone for data retrieval. They can easily adapt to new data sets and provide access to both structured and unstruc...
As the Internet of Things unfolds, mobile and wearable devices are blurring the line between physical and digital, integrating ever more closely with our interests, our routines, our daily lives. Contextual computing and smart, sensor-equipped spaces bring the potential to walk through a world that recognizes us and responds accordingly. We become continuous transmitters and receivers of data. In his session at @ThingsExpo, Andrew Bolwell, Director of Innovation for HP's Printing and Personal Systems Group, discussed how key attributes of mobile technology – touch input, sensors, social, and ...
WebRTC defines no default signaling protocol, causing fragmentation between WebRTC silos. SIP and XMPP provide possibilities, but come with considerable complexity and are not designed for use in a web environment. In his session at @ThingsExpo, Matthew Hodgson, technical co-founder of the Matrix.org, discussed how Matrix is a new non-profit Open Source Project that defines both a new HTTP-based standard for VoIP & IM signaling and provides reference implementations.