|By Lynn de la Torre, Bryce Harrington||
|May 30, 2005 12:00 PM EDT||
The Network File System (NFS) is an important mechanism for sharing files among end users on a broad range of platforms. End users have relied on NFS to support mission-critical applications for several decades. However, in recent years, other shared file systems have been developed to provide features that earlier versions of NFS lacked. To compete and address real end-user needs, the new rev 4 of NFS was developed. As NFS version 4 becomes available for deployment, interest in it is growing. Does it deliver on its promises? Will it introduce performance gains or stability issues compared with NFS v3? How well does it fit in existing enterprise ecosystems? The new version offers performance and security features, but may also pose risks or other challenges, and leaves various open questions for IT managers.
In speaking with a number of member companies, OSDL became aware of a widespread industry interest in seeing these questions answered through organized testing of the Linux implementation of NFS 4. Moreover, there was a strong desire among OSDL member companies to see this testing work done in open community-driven processes to encourage involvement by a broad spectrum of participants. The OSDL lab was also asked by the NFS 4 development community to help organize such testing, which began a few months ago, but has already helped make NFS 4 ready for enterprise deployment.
Initiating the Effort at OSDL
The Linux NFS v4 testing effort at the Open Source Development Labs came about through a confluence of interests. During various OSDL meetings, presentations, and activities in 2004, the OSDL Test and Performance Department was frequent questioned about NFS. Did OSDL tools and tests support network testing? Was OSDL doing file system performance tests with NFS? Did OSDL plan to help with the NFS v4 development?
Independently, when the OSDL Test and Performance Department informally asked several companies for lists of testing avenues for OSDL to pursue, NFS found its way onto everyone's list; not necessarily as item number one, but always in the top five.
The OSDL Storage Special Interest Group (SIG) and Data Center Linux (DCL) Initiative had separately identified NFS testing as a priority item and were interested in supporting the effort.
The primary reason it got such a high priority is that DCL surveyed end-user member companies running mission-critical applications and found NFS to be a cornerstone technology. In many cases, it was a crucial part of their Linux deployments. Once NFS testing was identified as a priority, the Storage SIG, with the support of Bryce Harrington of the OSDL Test and Performance group, began work to stimulate community-based testing efforts.
With such widespread interest, gaining approval for the project was straightforward.
Benefits of NFS v4
Version 4 of NFS offers features that are geared to improving both security and performance. A full list of the new benefits is available at http://nfs.sourceforge.net/. In brief it:
- Tracks file state: Unlike prior versions of NFS, in NFS v4 file state (locking, reading, writing) is tracked between the client and server
- Permits lease-based locking: It lets the client take ownership of a file for a period of time; it must contact the server to extend the lease
- Allows file delegation: NFS v4 servers can let NFS v4 clients modify cached files without contacting the server until the server notes that another client needs it and issues a 'callback'
- Implements compound RPCs (Remote Procedure Calls): Multiple NFS operations (LOOKUP, OPEN, READ, etc.) can be combined into a single RPC request, thereby minimizing network round trips.
- Supports security flavors: A number of sophisticated security mechanisms including Kerberos 5 and SPKM3 are implemented, and APIs are available to add new security mechanisms down the road
- Supports ACLs: On POSIX systems and Windows, NFS v4 standardizes how ACLs are used. Named attributes are also added, allowing user and group names to be accessed as strings, not just numeric IDs.
- Combines several distinct NFS protocols: Combines stat, NLM, mount, ACL, and NFS into a single protocol specification for better compatibility with network firewalls
- Supports file migration and replication
The Challenge for NFS v4 Testing
End users who benefit the most from these new features tend to have the highest risk aversion to changing their infrastructure in terms of downtime and troubleshooting costs. The implication is that testing NFS v4 is more important than other Open Source projects. It also presents a great opportunity: if the testing is done well, NFS 4 could be applied to new problem areas that have been outside the scope of previous NFS versions, and empower the community.
From this perspective, the objectives for testing NFSv4 were:
a) to identify problems and improvement opportunities in the implementation so developers can achieve a better product
b) to establish NFS 4 as a technology ready for end-user deployment with rapid realization of benefits from migration
c) to enable end users and community members to participate in community-oriented testing efforts so they can ensure their needs will get full visibility. This insures that testing resources are applied efficiently on high-priority areas without duplication.
Finding a Community
An important goal highlighted by the IT staff surveyed was that testing activities be community-oriented. This orientation was called out for several reasons like avoiding lock-in, allowing for open peer review of results, and enabling the testing work to be shared broadly.
Experience showed that it's extremely difficult to create an Open Source project from scratch. Besides the development or testing work that the project was formed to do, projects must recruit participants, build visibility, package the results, answer questions from outside parties, build the infrastructure, and much more. On the other hand, joining an existing project brings many of these gains automatically. So when starting the NFS v4 project, our first step was to evaluate various mailing lists and find one that would best suit OSDL testing discussions. OSDL selected and joined the [email protected] list managed by the CITI NFS Version 4 developers.
Finding the "Problem"
As newcomers to NFS v4 the OSDL team needed to figure out what to do. While the team had been given a huge number of potential action items, we still didn't know exactly where our efforts should be directed. An easy initial assumption would be that simply running various tests would be an adequate return on the testing investment. However, instead of jumping right in to running tests, OSDL engaged with the existing testers and asked them what was needed near-term.
The OSDL team noticed early on that the community didn't use a bug tracker. In discussion with the community, it was apparent that it had a high level of confidence in its ability to handle bugs through its mailing list, and we decided to postpone the bug tracker investigation.
It became clear during these discussions that beyond running tests arbitrarily, an organized approach was needed that would help identify what testing really needed to be done. We needed to know what the priorities were, whether someone was already doing those tests, and what precisely should be achieved with each test. This list of priorities needed a wide buy-in from the testers, developers, and enterprise users, and needed to be openly available to anyone for review.
The general feeling was that the real problem to solve was to gain a broad top-down prioritized list of all aspects of testing so OSDL could track who was working on which item and the status of each.
The NFS v4 Testing Matrix - Evolution of the Plan
During our initial analysis, OSDL got a huge number of e-mails, presentations, and discussions about the testing needs of NFS v4.
The community needed a way to collect and organize these disparate ideas and plans to communicate testing needs coherently. Early on Mary Edie Meredith of OSDL, the DCL roadmap coordinator, suggested a test matrix to correlate test items with test programs and reference testing resources and staff. The final and best form for this document was a list of test items in a spreadsheet that resembled a "Work Breakdown Structure"(WBS). Like a typical WBS, the NFS v4 Test Matrix organized testing tasks into a numbered hierarchy.
At the highest level, the Testing Matrix now has five broad categories:
- Functional testing
- Interoperability testing
- Robustness testing
- Performance testing
- Security testing
The NFS Testing Matrix was then circulated among members of both the NFS v4 community and the industry-at-large for feedback, additions, and prioritization suggestions. Done initially through e-mail, we found it easier to get participation by holding weekly conference calls to go through the matrix section-by-section.
In a number of cases the team found that community members were already working on tests. Tracking this existing activity in the Test Matrix helped other testers avoid duplication. This correlation also helped identify gaps where specific kinds of tests were needed, but where the existing tests lacked the necessary coverage. This information proved especially interesting for the test authors, giving clear direction about what to add to the tests, and why.
Over the past few months this testing effort has generated a number of improvements to the code. Several participants working on some of the functional, robustness, and performance test items in the matrix uncovered a number of bugs; our principle of working closely with the developers has allowed these issues to be addressed quickly and closed.
As an example of this work, OSDL attended the Connectathon event in late February and chose to focus on testing the installation of NFSv4 on SuSE as a learning exercise. The version of SuSE tested used the Heimdal version of Kerberos, which hadn't been widely tested and some of the libraries had compilation and configuration issues; working directly with the developers, we were able to generate and test patches to correct the behavior. These patches were incorporated into the mainline code base shortly thereafter.
Now that we know clearly what the needs are, we're ready to engage additional volunteers in testing. The team is using a combination of approaches to solicit help. First, we're trying to drill deeper into each task to help answer questions about how to do tests. Second, through publications such as this one we'll spread awareness of the NFS v4 testing effort to potential participants. Third, we're approaching companies involved in the community to encourage them to sign up to help. Fourth, we're reaching out to end users to validate priorities and determine where their test efforts converge with our efforts. Fifth, we're reaching out to the OSDL Security SIG to help define the security section of the test matrix. Finally, for items we can't find volunteers for, we'll report these areas as issues to the OSDL DCL committee for resolution, and that group will hopefully assign the resources needed right away.
The planning mechanism we'll implement next is "'Testing Checklists." Such checklists will provide pointers to items to be tested, identify non-obvious configuration directions, and outline other things to look for. OSDL is also developing testing tools to assist testers in doing tests, collecting information, and reporting the results. OSDL hopes to act as a central collection point for the results of NFS v4 testing efforts.
Building a Community-Owned Testing Methodology
There are a number of challenges to the community approach. Unlike traditional testing, where a single company owns the process and employs the staff to do it, in wide community-driven testing it can be hard to get every area filled.
Also, with Open Source the distinction between a developer and tester is much more blurred. This can sometimes result in more emphasis being put on development than testing. For NFS v4, a balance must be struck that includes strong emphasis on both testing and development.
A third challenge is the sheer complexity of the NFS code stack. Besides the NFS client and server code in the Linux kernel, there's a surrounding layer of utilities, administrative tools, file systems, add-ons like automounter and cachefs, and authentication services. Interaction between these pieces and NFS needs thorough testing with different versions and configuration settings and huge numbers of permutations. The OSDL team hopes that careful planning will contain the scope, and that open participation to a wider community will disperse the effort.
Fortunately, these weaknesses are all areas that dovetail well with corporate testing efforts since they are areas where those organizations have strengths. Companies bring employees that can be dedicated to specific tasks, people with the specific talents needed for the task. They can scale their contributions to match their business needs, providing an effective way to address complexities - if a given company needs a particular set of interactions thoroughly tested, then the business case will exist to justify funding a testing effort to do it.
By establishing a clear, well organized, and structured testing effort in the open NFS v4 community, we'll enable these organizations to participate better in testing; they can focus on their own priorities. By encouraging them to share their results openly, NFS v4 as a whole will be improved and the testing work will eventually be done.
Call to Action
Many features need testing in NFS, but of course, no company wants to do it all alone. Our hope is that by involving a variety of companies, even if each company's contribution is small, NFS v4 will get enough testing and validation to benefit from the new NFS v4 features reasonably soon, while avoiding the frustrations of incompletely tested software.
Twice a year, the larger NFS community (including developers for non-Linux platforms) get together at Connectathon and Bakeathon events, where implementations are tested against one another in a controlled network environment. OSDL participated in the 2005 Connectathon to interact with developers face-to-face and learn about setting up, using, and testing the code. Such face-to-face opportunities are invaluable in solving problems. We have also arranged a BOF meeting at this year's Ottawa Linux Symposium (OLS) to interact with the wider Open Source community.
Please review the NFS v4Test Matrix and look for areas of interest for your own organization. If you'd like to participate or track its progress, join the NFS v4 and Storage SIG at www.developer.osdl.org/dev/nfsv4/.
Appendix: High-Priority Items in the Test Matrix
High-priority testing areas identified by the community include the following:
- Standards compliance and conformance (POSIX, NFS specs)
- State transitions
- "Ecosystem compatibility" - glibc, krb5, Ipsec, ACLs (POSIX & NFS), automounter
- Compatibility with the TCP protocol
- Automounter direct map support
- Use Case: Database functionality on NFS
- Use Case: Clusters/migration/replication functionality (multiple clients)
- Use Case: Web server
- Interoperability with mit-krb5 and IpSec v4 protocols
- Interoperability between 32-bit and 64-bit clients and servers
- Interoperability between big endian and little endian
- Interoperability of the Linux NFS client with target server architectures/platforms
- Ext3 interoperability
- ACL interoperability of Linux and non-Linux clients and servers
- Running workloads for two weeks under various conditions and interuptions
- Resource limit testing (out of memory, disk space, inode, swap space)
- Stress load testing
- Scalability (max number of connections and file systems)
- Recovery from problems while under light/normal/heavy loads
- Automounter race conditions and remounting in corner cases
- Comparison of NFS v3 and NVS v4 for common use cases
- Evaluate performance in load scenarios
- Scalability performance - does performance degrade gracefully?
- Review security feature design and assumptions
- Code audit
- Attack and penetration security review
With major technology companies and startups seriously embracing IoT strategies, now is the perfect time to attend @ThingsExpo 2016 in New York. Learn what is going on, contribute to the discussions, and ensure that your enterprise is as "IoT-Ready" as it can be! Internet of @ThingsExpo, taking place June 6-8, 2017, at the Javits Center in New York City, New York, is co-located with 20th Cloud Expo and will feature technical sessions from a rock star conference faculty and the leading industry p...
Jan. 22, 2017 07:45 PM EST Reads: 3,861
"LinearHub provides smart video conferencing, which is the Roundee service, and we archive all the video conferences and we also provide the transcript," stated Sunghyuk Kim, CEO of LinearHub, in this SYS-CON.tv interview at @ThingsExpo, held November 1-3, 2016, at the Santa Clara Convention Center in Santa Clara, CA.
Jan. 22, 2017 03:15 PM EST Reads: 1,686
Internet of @ThingsExpo, taking place June 6-8, 2017 at the Javits Center in New York City, New York, is co-located with the 20th International Cloud Expo and will feature technical sessions from a rock star conference faculty and the leading industry players in the world. @ThingsExpo New York Call for Papers is now open.
Jan. 22, 2017 02:30 PM EST Reads: 3,758
"There's a growing demand from users for things to be faster. When you think about all the transactions or interactions users will have with your product and everything that is between those transactions and interactions - what drives us at Catchpoint Systems is the idea to measure that and to analyze it," explained Leo Vasiliou, Director of Web Performance Engineering at Catchpoint Systems, in this SYS-CON.tv interview at 18th Cloud Expo, held June 7-9, 2016, at the Javits Center in New York Ci...
Jan. 22, 2017 02:00 PM EST Reads: 5,821
The 20th International Cloud Expo has announced that its Call for Papers is open. Cloud Expo, to be held June 6-8, 2017, at the Javits Center in New York City, brings together Cloud Computing, Big Data, Internet of Things, DevOps, Containers, Microservices and WebRTC to one location. With cloud computing driving a higher percentage of enterprise IT budgets every year, it becomes increasingly important to plant your flag in this fast-expanding business opportunity. Submit your speaking proposal ...
Jan. 22, 2017 02:00 PM EST Reads: 5,290
WebRTC is the future of browser-to-browser communications, and continues to make inroads into the traditional, difficult, plug-in web communications world. The 6th WebRTC Summit continues our tradition of delivering the latest and greatest presentations within the world of WebRTC. Topics include voice calling, video chat, P2P file sharing, and use cases that have already leveraged the power and convenience of WebRTC.
Jan. 22, 2017 01:45 PM EST Reads: 3,228
20th Cloud Expo, taking place June 6-8, 2017, at the Javits Center in New York City, NY, will feature technical sessions from a rock star conference faculty and the leading industry players in the world. Cloud computing is now being embraced by a majority of enterprises of all sizes. Yesterday's debate about public vs. private has transformed into the reality of hybrid cloud: a recent survey shows that 74% of enterprises have a hybrid cloud strategy.
Jan. 22, 2017 01:00 PM EST Reads: 4,392
Discover top technologies and tools all under one roof at April 24–28, 2017, at the Westin San Diego in San Diego, CA. Explore the Mobile Dev + Test and IoT Dev + Test Expo and enjoy all of these unique opportunities: The latest solutions, technologies, and tools in mobile or IoT software development and testing. Meet one-on-one with representatives from some of today's most innovative organizations
Jan. 22, 2017 12:45 PM EST Reads: 1,690
DevOps is being widely accepted (if not fully adopted) as essential in enterprise IT. But as Enterprise DevOps gains maturity, expands scope, and increases velocity, the need for data-driven decisions across teams becomes more acute. DevOps teams in any modern business must wrangle the ‘digital exhaust’ from the delivery toolchain, "pervasive" and "cognitive" computing, APIs and services, mobile devices and applications, the Internet of Things, and now even blockchain. In this power panel at @...
Jan. 22, 2017 11:45 AM EST Reads: 2,973
Data is the fuel that drives the machine learning algorithmic engines and ultimately provides the business value. In his session at Cloud Expo, Ed Featherston, a director and senior enterprise architect at Collaborative Consulting, discussed the key considerations around quality, volume, timeliness, and pedigree that must be dealt with in order to properly fuel that engine.
Jan. 22, 2017 11:30 AM EST Reads: 3,740
"A lot of times people will come to us and have a very diverse set of requirements or very customized need and we'll help them to implement it in a fashion that you can't just buy off of the shelf," explained Nick Rose, CTO of Enzu, in this SYS-CON.tv interview at 18th Cloud Expo, held June 7-9, 2016, at the Javits Center in New York City, NY.
Jan. 22, 2017 09:30 AM EST Reads: 4,771
The WebRTC Summit New York, to be held June 6-8, 2017, at the Javits Center in New York City, NY, announces that its Call for Papers is now open. Topics include all aspects of improving IT delivery by eliminating waste through automated business models leveraging cloud technologies. WebRTC Summit is co-located with 20th International Cloud Expo and @ThingsExpo. WebRTC is the future of browser-to-browser communications, and continues to make inroads into the traditional, difficult, plug-in web co...
Jan. 22, 2017 09:30 AM EST Reads: 3,025
Buzzword alert: Microservices and IoT at a DevOps conference? What could possibly go wrong? In this Power Panel at DevOps Summit, moderated by Jason Bloomberg, the leading expert on architecting agility for the enterprise and president of Intellyx, panelists peeled away the buzz and discuss the important architectural principles behind implementing IoT solutions for the enterprise. As remote IoT devices and sensors become increasingly intelligent, they become part of our distributed cloud enviro...
Jan. 22, 2017 08:30 AM EST Reads: 5,012
In 2014, Amazon announced a new form of compute called Lambda. We didn't know it at the time, but this represented a fundamental shift in what we expect from cloud computing. Now, all of the major cloud computing vendors want to take part in this disruptive technology. In his session at 20th Cloud Expo, John Jelinek IV, a web developer at Linux Academy, will discuss why major players like AWS, Microsoft Azure, IBM Bluemix, and Google Cloud Platform are all trying to sidestep VMs and containers...
Jan. 22, 2017 08:30 AM EST Reads: 987
SYS-CON Events announced today that MobiDev, a client-oriented software development company, will exhibit at SYS-CON's 20th International Cloud Expo®, which will take place June 6-8, 2017, at the Javits Center in New York City, NY, and the 21st International Cloud Expo®, which will take place October 31-November 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. MobiDev is a software company that develops and delivers turn-key mobile apps, websites, web services, and complex softw...
Jan. 22, 2017 07:15 AM EST Reads: 1,982
WebRTC is about the data channel as much as about video and audio conferencing. However, basically all commercial WebRTC applications have been built with a focus on audio and video. The handling of “data” has been limited to text chat and file download – all other data sharing seems to end with screensharing. What is holding back a more intensive use of peer-to-peer data? In her session at @ThingsExpo, Dr Silvia Pfeiffer, WebRTC Applications Team Lead at National ICT Australia, looked at differ...
Jan. 22, 2017 03:45 AM EST Reads: 5,077
Fact is, enterprises have significant legacy voice infrastructure that’s costly to replace with pure IP solutions. How can we bring this analog infrastructure into our shiny new cloud applications? There are proven methods to bind both legacy voice applications and traditional PSTN audio into cloud-based applications and services at a carrier scale. Some of the most successful implementations leverage WebRTC, WebSockets, SIP and other open source technologies. In his session at @ThingsExpo, Da...
Jan. 22, 2017 02:30 AM EST Reads: 2,920
Growth hacking is common for startups to make unheard-of progress in building their business. Career Hacks can help Geek Girls and those who support them (yes, that's you too, Dad!) to excel in this typically male-dominated world. Get ready to learn the facts: Is there a bias against women in the tech / developer communities? Why are women 50% of the workforce, but hold only 24% of the STEM or IT positions? Some beginnings of what to do about it! In her Day 2 Keynote at 17th Cloud Expo, Sandy Ca...
Jan. 22, 2017 02:30 AM EST Reads: 7,930
SYS-CON Media announced today that @WebRTCSummit Blog, the largest WebRTC resource in the world, has been launched. @WebRTCSummit Blog offers top articles, news stories, and blog posts from the world's well-known experts and guarantees better exposure for its authors than any other publication. @WebRTCSummit Blog can be bookmarked ▸ Here @WebRTCSummit conference site can be bookmarked ▸ Here
Jan. 22, 2017 02:00 AM EST Reads: 11,773
Manufacturers are embracing the Industrial Internet the same way consumers are leveraging Fitbits – to improve overall health and wellness. Both can provide consistent measurement, visibility, and suggest performance improvements customized to help reach goals. Fitbit users can view real-time data and make adjustments to increase their activity. In his session at @ThingsExpo, Mark Bernardo Professional Services Leader, Americas, at GE Digital, discussed how leveraging the Industrial Internet and...
Jan. 22, 2017 01:45 AM EST Reads: 6,598