|By Lynn de la Torre, Bryce Harrington||
|May 30, 2005 12:00 PM EDT||
The Network File System (NFS) is an important mechanism for sharing files among end users on a broad range of platforms. End users have relied on NFS to support mission-critical applications for several decades. However, in recent years, other shared file systems have been developed to provide features that earlier versions of NFS lacked. To compete and address real end-user needs, the new rev 4 of NFS was developed. As NFS version 4 becomes available for deployment, interest in it is growing. Does it deliver on its promises? Will it introduce performance gains or stability issues compared with NFS v3? How well does it fit in existing enterprise ecosystems? The new version offers performance and security features, but may also pose risks or other challenges, and leaves various open questions for IT managers.
In speaking with a number of member companies, OSDL became aware of a widespread industry interest in seeing these questions answered through organized testing of the Linux implementation of NFS 4. Moreover, there was a strong desire among OSDL member companies to see this testing work done in open community-driven processes to encourage involvement by a broad spectrum of participants. The OSDL lab was also asked by the NFS 4 development community to help organize such testing, which began a few months ago, but has already helped make NFS 4 ready for enterprise deployment.
Initiating the Effort at OSDL
The Linux NFS v4 testing effort at the Open Source Development Labs came about through a confluence of interests. During various OSDL meetings, presentations, and activities in 2004, the OSDL Test and Performance Department was frequent questioned about NFS. Did OSDL tools and tests support network testing? Was OSDL doing file system performance tests with NFS? Did OSDL plan to help with the NFS v4 development?
Independently, when the OSDL Test and Performance Department informally asked several companies for lists of testing avenues for OSDL to pursue, NFS found its way onto everyone's list; not necessarily as item number one, but always in the top five.
The OSDL Storage Special Interest Group (SIG) and Data Center Linux (DCL) Initiative had separately identified NFS testing as a priority item and were interested in supporting the effort.
The primary reason it got such a high priority is that DCL surveyed end-user member companies running mission-critical applications and found NFS to be a cornerstone technology. In many cases, it was a crucial part of their Linux deployments. Once NFS testing was identified as a priority, the Storage SIG, with the support of Bryce Harrington of the OSDL Test and Performance group, began work to stimulate community-based testing efforts.
With such widespread interest, gaining approval for the project was straightforward.
Benefits of NFS v4
Version 4 of NFS offers features that are geared to improving both security and performance. A full list of the new benefits is available at http://nfs.sourceforge.net/. In brief it:
- Tracks file state: Unlike prior versions of NFS, in NFS v4 file state (locking, reading, writing) is tracked between the client and server
- Permits lease-based locking: It lets the client take ownership of a file for a period of time; it must contact the server to extend the lease
- Allows file delegation: NFS v4 servers can let NFS v4 clients modify cached files without contacting the server until the server notes that another client needs it and issues a 'callback'
- Implements compound RPCs (Remote Procedure Calls): Multiple NFS operations (LOOKUP, OPEN, READ, etc.) can be combined into a single RPC request, thereby minimizing network round trips.
- Supports security flavors: A number of sophisticated security mechanisms including Kerberos 5 and SPKM3 are implemented, and APIs are available to add new security mechanisms down the road
- Supports ACLs: On POSIX systems and Windows, NFS v4 standardizes how ACLs are used. Named attributes are also added, allowing user and group names to be accessed as strings, not just numeric IDs.
- Combines several distinct NFS protocols: Combines stat, NLM, mount, ACL, and NFS into a single protocol specification for better compatibility with network firewalls
- Supports file migration and replication
The Challenge for NFS v4 Testing
End users who benefit the most from these new features tend to have the highest risk aversion to changing their infrastructure in terms of downtime and troubleshooting costs. The implication is that testing NFS v4 is more important than other Open Source projects. It also presents a great opportunity: if the testing is done well, NFS 4 could be applied to new problem areas that have been outside the scope of previous NFS versions, and empower the community.
From this perspective, the objectives for testing NFSv4 were:
a) to identify problems and improvement opportunities in the implementation so developers can achieve a better product
b) to establish NFS 4 as a technology ready for end-user deployment with rapid realization of benefits from migration
c) to enable end users and community members to participate in community-oriented testing efforts so they can ensure their needs will get full visibility. This insures that testing resources are applied efficiently on high-priority areas without duplication.
Finding a Community
An important goal highlighted by the IT staff surveyed was that testing activities be community-oriented. This orientation was called out for several reasons like avoiding lock-in, allowing for open peer review of results, and enabling the testing work to be shared broadly.
Experience showed that it's extremely difficult to create an Open Source project from scratch. Besides the development or testing work that the project was formed to do, projects must recruit participants, build visibility, package the results, answer questions from outside parties, build the infrastructure, and much more. On the other hand, joining an existing project brings many of these gains automatically. So when starting the NFS v4 project, our first step was to evaluate various mailing lists and find one that would best suit OSDL testing discussions. OSDL selected and joined the [email protected] list managed by the CITI NFS Version 4 developers.
Finding the "Problem"
As newcomers to NFS v4 the OSDL team needed to figure out what to do. While the team had been given a huge number of potential action items, we still didn't know exactly where our efforts should be directed. An easy initial assumption would be that simply running various tests would be an adequate return on the testing investment. However, instead of jumping right in to running tests, OSDL engaged with the existing testers and asked them what was needed near-term.
The OSDL team noticed early on that the community didn't use a bug tracker. In discussion with the community, it was apparent that it had a high level of confidence in its ability to handle bugs through its mailing list, and we decided to postpone the bug tracker investigation.
It became clear during these discussions that beyond running tests arbitrarily, an organized approach was needed that would help identify what testing really needed to be done. We needed to know what the priorities were, whether someone was already doing those tests, and what precisely should be achieved with each test. This list of priorities needed a wide buy-in from the testers, developers, and enterprise users, and needed to be openly available to anyone for review.
The general feeling was that the real problem to solve was to gain a broad top-down prioritized list of all aspects of testing so OSDL could track who was working on which item and the status of each.
The NFS v4 Testing Matrix - Evolution of the Plan
During our initial analysis, OSDL got a huge number of e-mails, presentations, and discussions about the testing needs of NFS v4.
The community needed a way to collect and organize these disparate ideas and plans to communicate testing needs coherently. Early on Mary Edie Meredith of OSDL, the DCL roadmap coordinator, suggested a test matrix to correlate test items with test programs and reference testing resources and staff. The final and best form for this document was a list of test items in a spreadsheet that resembled a "Work Breakdown Structure"(WBS). Like a typical WBS, the NFS v4 Test Matrix organized testing tasks into a numbered hierarchy.
At the highest level, the Testing Matrix now has five broad categories:
- Functional testing
- Interoperability testing
- Robustness testing
- Performance testing
- Security testing
The NFS Testing Matrix was then circulated among members of both the NFS v4 community and the industry-at-large for feedback, additions, and prioritization suggestions. Done initially through e-mail, we found it easier to get participation by holding weekly conference calls to go through the matrix section-by-section.
In a number of cases the team found that community members were already working on tests. Tracking this existing activity in the Test Matrix helped other testers avoid duplication. This correlation also helped identify gaps where specific kinds of tests were needed, but where the existing tests lacked the necessary coverage. This information proved especially interesting for the test authors, giving clear direction about what to add to the tests, and why.
Over the past few months this testing effort has generated a number of improvements to the code. Several participants working on some of the functional, robustness, and performance test items in the matrix uncovered a number of bugs; our principle of working closely with the developers has allowed these issues to be addressed quickly and closed.
As an example of this work, OSDL attended the Connectathon event in late February and chose to focus on testing the installation of NFSv4 on SuSE as a learning exercise. The version of SuSE tested used the Heimdal version of Kerberos, which hadn't been widely tested and some of the libraries had compilation and configuration issues; working directly with the developers, we were able to generate and test patches to correct the behavior. These patches were incorporated into the mainline code base shortly thereafter.
Now that we know clearly what the needs are, we're ready to engage additional volunteers in testing. The team is using a combination of approaches to solicit help. First, we're trying to drill deeper into each task to help answer questions about how to do tests. Second, through publications such as this one we'll spread awareness of the NFS v4 testing effort to potential participants. Third, we're approaching companies involved in the community to encourage them to sign up to help. Fourth, we're reaching out to end users to validate priorities and determine where their test efforts converge with our efforts. Fifth, we're reaching out to the OSDL Security SIG to help define the security section of the test matrix. Finally, for items we can't find volunteers for, we'll report these areas as issues to the OSDL DCL committee for resolution, and that group will hopefully assign the resources needed right away.
The planning mechanism we'll implement next is "'Testing Checklists." Such checklists will provide pointers to items to be tested, identify non-obvious configuration directions, and outline other things to look for. OSDL is also developing testing tools to assist testers in doing tests, collecting information, and reporting the results. OSDL hopes to act as a central collection point for the results of NFS v4 testing efforts.
Building a Community-Owned Testing Methodology
There are a number of challenges to the community approach. Unlike traditional testing, where a single company owns the process and employs the staff to do it, in wide community-driven testing it can be hard to get every area filled.
Also, with Open Source the distinction between a developer and tester is much more blurred. This can sometimes result in more emphasis being put on development than testing. For NFS v4, a balance must be struck that includes strong emphasis on both testing and development.
A third challenge is the sheer complexity of the NFS code stack. Besides the NFS client and server code in the Linux kernel, there's a surrounding layer of utilities, administrative tools, file systems, add-ons like automounter and cachefs, and authentication services. Interaction between these pieces and NFS needs thorough testing with different versions and configuration settings and huge numbers of permutations. The OSDL team hopes that careful planning will contain the scope, and that open participation to a wider community will disperse the effort.
Fortunately, these weaknesses are all areas that dovetail well with corporate testing efforts since they are areas where those organizations have strengths. Companies bring employees that can be dedicated to specific tasks, people with the specific talents needed for the task. They can scale their contributions to match their business needs, providing an effective way to address complexities - if a given company needs a particular set of interactions thoroughly tested, then the business case will exist to justify funding a testing effort to do it.
By establishing a clear, well organized, and structured testing effort in the open NFS v4 community, we'll enable these organizations to participate better in testing; they can focus on their own priorities. By encouraging them to share their results openly, NFS v4 as a whole will be improved and the testing work will eventually be done.
Call to Action
Many features need testing in NFS, but of course, no company wants to do it all alone. Our hope is that by involving a variety of companies, even if each company's contribution is small, NFS v4 will get enough testing and validation to benefit from the new NFS v4 features reasonably soon, while avoiding the frustrations of incompletely tested software.
Twice a year, the larger NFS community (including developers for non-Linux platforms) get together at Connectathon and Bakeathon events, where implementations are tested against one another in a controlled network environment. OSDL participated in the 2005 Connectathon to interact with developers face-to-face and learn about setting up, using, and testing the code. Such face-to-face opportunities are invaluable in solving problems. We have also arranged a BOF meeting at this year's Ottawa Linux Symposium (OLS) to interact with the wider Open Source community.
Please review the NFS v4Test Matrix and look for areas of interest for your own organization. If you'd like to participate or track its progress, join the NFS v4 and Storage SIG at www.developer.osdl.org/dev/nfsv4/.
Appendix: High-Priority Items in the Test Matrix
High-priority testing areas identified by the community include the following:
- Standards compliance and conformance (POSIX, NFS specs)
- State transitions
- "Ecosystem compatibility" - glibc, krb5, Ipsec, ACLs (POSIX & NFS), automounter
- Compatibility with the TCP protocol
- Automounter direct map support
- Use Case: Database functionality on NFS
- Use Case: Clusters/migration/replication functionality (multiple clients)
- Use Case: Web server
- Interoperability with mit-krb5 and IpSec v4 protocols
- Interoperability between 32-bit and 64-bit clients and servers
- Interoperability between big endian and little endian
- Interoperability of the Linux NFS client with target server architectures/platforms
- Ext3 interoperability
- ACL interoperability of Linux and non-Linux clients and servers
- Running workloads for two weeks under various conditions and interuptions
- Resource limit testing (out of memory, disk space, inode, swap space)
- Stress load testing
- Scalability (max number of connections and file systems)
- Recovery from problems while under light/normal/heavy loads
- Automounter race conditions and remounting in corner cases
- Comparison of NFS v3 and NVS v4 for common use cases
- Evaluate performance in load scenarios
- Scalability performance - does performance degrade gracefully?
- Review security feature design and assumptions
- Code audit
- Attack and penetration security review
There are several IoTs: the Industrial Internet, Consumer Wearables, Wearables and Healthcare, Supply Chains, and the movement toward Smart Grids, Cities, Regions, and Nations. There are competing communications standards every step of the way, a bewildering array of sensors and devices, and an entire world of competing data analytics platforms. To some this appears to be chaos. In this power panel at @ThingsExpo, moderated by Conference Chair Roger Strukhoff, Bradley Holt, Developer Advocate a...
Jun. 26, 2016 08:45 AM EDT Reads: 684
In his keynote at 18th Cloud Expo, Andrew Keys, Co-Founder of ConsenSys Enterprise, provided an overview of the evolution of the Internet and the Database and the future of their combination – the Blockchain. Andrew Keys is Co-Founder of ConsenSys Enterprise. He comes to ConsenSys Enterprise with capital markets, technology and entrepreneurial experience. Previously, he worked for UBS investment bank in equities analysis. Later, he was responsible for the creation and distribution of life sett...
Jun. 25, 2016 06:15 PM EDT Reads: 986
Cognitive Computing is becoming the foundation for a new generation of solutions that have the potential to transform business. Unlike traditional approaches to building solutions, a cognitive computing approach allows the data to help determine the way applications are designed. This contrasts with conventional software development that begins with defining logic based on the current way a business operates. In her session at 18th Cloud Expo, Judith S. Hurwitz, President and CEO of Hurwitz & ...
Jun. 25, 2016 03:00 PM EDT Reads: 1,545
In his general session at 18th Cloud Expo, Lee Atchison, Principal Cloud Architect and Advocate at New Relic, discussed cloud as a ‘better data center’ and how it adds new capacity (faster) and improves application availability (redundancy). The cloud is a ‘Dynamic Tool for Dynamic Apps’ and resource allocation is an integral part of your application architecture, so use only the resources you need and allocate /de-allocate resources on the fly.
Jun. 25, 2016 02:15 PM EDT Reads: 1,022
A strange thing is happening along the way to the Internet of Things, namely far too many devices to work with and manage. It has become clear that we'll need much higher efficiency user experiences that can allow us to more easily and scalably work with the thousands of devices that will soon be in each of our lives. Enter the conversational interface revolution, combining bots we can literally talk with, gesture to, and even direct with our thoughts, with embedded artificial intelligence, wh...
Jun. 25, 2016 01:45 PM EDT Reads: 883
19th Cloud Expo, taking place November 1-3, 2016, at the Santa Clara Convention Center in Santa Clara, CA, will feature technical sessions from a rock star conference faculty and the leading industry players in the world. Cloud computing is now being embraced by a majority of enterprises of all sizes. Yesterday's debate about public vs. private has transformed into the reality of hybrid cloud: a recent survey shows that 74% of enterprises have a hybrid cloud strategy. Meanwhile, 94% of enterpri...
Jun. 25, 2016 01:15 PM EDT Reads: 1,227
SYS-CON Events announced today that Bsquare has been named “Silver Sponsor” of SYS-CON's @ThingsExpo, which will take place on November 1–3, 2016, at the Santa Clara Convention Center in Santa Clara, CA. For more than two decades, Bsquare has helped its customers extract business value from a broad array of physical assets by making them intelligent, connecting them, and using the data they generate to optimize business processes.
Jun. 25, 2016 11:45 AM EDT Reads: 1,175
Internet of @ThingsExpo, taking place November 1-3, 2016, at the Santa Clara Convention Center in Santa Clara, CA, is co-located with 19th Cloud Expo and will feature technical sessions from a rock star conference faculty and the leading industry players in the world. The Internet of Things (IoT) is the most profound change in personal and enterprise IT since the creation of the Worldwide Web more than 20 years ago. All major researchers estimate there will be tens of billions devices - comp...
Jun. 25, 2016 11:15 AM EDT Reads: 1,193
There is little doubt that Big Data solutions will have an increasing role in the Enterprise IT mainstream over time. Big Data at Cloud Expo - to be held November 1-3, 2016, at the Santa Clara Convention Center in Santa Clara, CA - has announced its Call for Papers is open. Cloud computing is being adopted in one form or another by 94% of enterprises today. Tens of billions of new devices are being connected to The Internet of Things. And Big Data is driving this bus. An exponential increase is...
Jun. 25, 2016 11:00 AM EDT Reads: 1,289
Machine Learning helps make complex systems more efficient. By applying advanced Machine Learning techniques such as Cognitive Fingerprinting, wind project operators can utilize these tools to learn from collected data, detect regular patterns, and optimize their own operations. In his session at 18th Cloud Expo, Stuart Gillen, Director of Business Development at SparkCognition, discussed how research has demonstrated the value of Machine Learning in delivering next generation analytics to imp...
Jun. 25, 2016 11:00 AM EDT Reads: 498
The 19th International Cloud Expo has announced that its Call for Papers is open. Cloud Expo, to be held November 1-3, 2016, at the Santa Clara Convention Center in Santa Clara, CA, brings together Cloud Computing, Big Data, Internet of Things, DevOps, Digital Transformation, Microservices and WebRTC to one location. With cloud computing driving a higher percentage of enterprise IT budgets every year, it becomes increasingly important to plant your flag in this fast-expanding business opportuni...
Jun. 25, 2016 10:30 AM EDT Reads: 1,243
Cloud computing is being adopted in one form or another by 94% of enterprises today. Tens of billions of new devices are being connected to The Internet of Things. And Big Data is driving this bus. An exponential increase is expected in the amount of information being processed, managed, analyzed, and acted upon by enterprise IT. This amazing is not part of some distant future - it is happening today. One report shows a 650% increase in enterprise data by 2020. Other estimates are even higher....
Jun. 25, 2016 10:15 AM EDT Reads: 1,251
The cloud market growth today is largely in public clouds. While there is a lot of spend in IT departments in virtualization, these aren’t yet translating into a true “cloud” experience within the enterprise. What is stopping the growth of the “private cloud” market? In his general session at 18th Cloud Expo, Nara Rajagopalan, CEO of Accelerite, explored the challenges in deploying, managing, and getting adoption for a private cloud within an enterprise. What are the key differences between wh...
Jun. 25, 2016 10:00 AM EDT Reads: 677
Internet of @ThingsExpo, taking place November 1-3, 2016, at the Santa Clara Convention Center in Santa Clara, CA, is co-located with the 19th International Cloud Expo and will feature technical sessions from a rock star conference faculty and the leading industry players in the world and ThingsExpo Silicon Valley Call for Papers is now open.
Jun. 25, 2016 09:30 AM EDT Reads: 1,086
It is one thing to build single industrial IoT applications, but what will it take to build the Smart Cities and truly society changing applications of the future? The technology won’t be the problem, it will be the number of parties that need to work together and be aligned in their motivation to succeed. In his Day 2 Keynote at @ThingsExpo, Henrik Kenani Dahlgren, Portfolio Marketing Manager at Ericsson, discussed how to plan to cooperate, partner, and form lasting all-star teams to change t...
Jun. 25, 2016 07:45 AM EDT Reads: 1,062
Connected devices and the industrial internet are growing exponentially every year with Cisco expecting 50 billion devices to be in operation by 2020. In this period of growth, location-based insights are becoming invaluable to many businesses as they adopt new connected technologies. Knowing when and where these devices connect from is critical for a number of scenarios in supply chain management, disaster management, emergency response, M2M, location marketing and more. In his session at @Th...
Jun. 25, 2016 07:30 AM EDT Reads: 887
SYS-CON Events announced today that ReadyTalk, a leading provider of online conferencing and webinar services, has been named Vendor Presentation Sponsor at the 19th International Cloud Expo, which will take place on November 1–3, 2016, at the Santa Clara Convention Center in Santa Clara, CA. ReadyTalk delivers audio and web conferencing services that inspire collaboration and enable the Future of Work for today’s increasingly digital and mobile workforce. By combining intuitive, innovative tec...
Jun. 24, 2016 01:00 PM EDT Reads: 1,334
Amazon has gradually rolled out parts of its IoT offerings, but these are just the tip of the iceberg. In addition to optimizing their backend AWS offerings, Amazon is laying the ground work to be a major force in IoT - especially in the connected home and office. In his session at @ThingsExpo, Chris Kocher, founder and managing director of Grey Heron, explained how Amazon is extending its reach to become a major force in IoT by building on its dominant cloud IoT platform, its Dash Button strat...
Jun. 24, 2016 12:00 PM EDT Reads: 1,596
industrial company for a multi-year contract initially valued at over $4.0 million. In addition to DataV software, Bsquare will also provide comprehensive systems integration, support and maintenance services. DataV leverages advanced data analytics, predictive reasoning, data-driven diagnostics, and automated orchestration of remediation actions in order to improve asset uptime while reducing service and warranty costs.
Jun. 22, 2016 11:00 AM EDT Reads: 1,353
Vidyo, Inc., has joined the Alliance for Open Media. The Alliance for Open Media is a non-profit organization working to define and develop media technologies that address the need for an open standard for video compression and delivery over the web. As a member of the Alliance, Vidyo will collaborate with industry leaders in pursuit of an open and royalty-free AOMedia Video codec, AV1. Vidyo’s contributions to the organization will bring to bear its long history of expertise in codec technolo...
Jun. 19, 2016 12:45 PM EDT Reads: 1,247