Welcome!

Linux Containers Authors: Liz McMillan, Vaibhaw Pandey, Elizabeth White, Pat Romanski, Yeshim Deniz

Related Topics: @DevOpsSummit, Linux Containers, Containers Expo Blog

@DevOpsSummit: Blog Feed Post

Scaled Agile Factory - Going Beyond the Usual Suspects | @DevOpsSummit #DevOps

The question of how to scale Agile development has been around a while

If you address the question of how to scale Agile projects by considering what framework to use, you are only looking at one aspect of the problem. Scaling is all about coordination – managing enterprise considerations and cross program dependencies, and the defacto frameworks (SAFe, LeSS and DAD) focus on the people and process dimensions. However, in combination with a factory approach you may be able to automate many of the compliance and dependency management issues.

The question of how to scale Agile development has been around a while. In January 2015 I commented [1] on a clear trend in which my customers were voicing concerns about loss of consistency; inability to govern; lack of coordination and increasing time to market. Since then I have observed many large organizations adopting SAFe (Scaled Agile Framework) perhaps because it appears to be the only game in town, or perhaps there’s good marketing or there’s strength in numbers. But the criticisms of SAFe [2] haven’t gone away. The central and continuing concern is that SAFe compromises core Agile principles of self-organizing, cross functional teams that have full responsibility for the delivery of potentially shippable increments. And that SAFe looks suspiciously like WaterScrumFall because there’s a high overhead of portfolio, value stream and project management and in all probability intentional architecture. Now I have mixed opinions on SAFe; I see positives in value streams and product management which are important. And for organizations that have large programs, highly organized, structured approaches will be seen as lower risk, and indeed something that takes them some way down the Agile path, without straying too far from conventional management comfort zones. But the outcomes are unlikely to be inherently “agile”.

What SAFe does, is provide a rather conventional approach for a complex problem that most enterprises have – that is to deliver high integrity solutions that can work in an enterprise context that demands cross project dependency management, consistent data and reference architecture, collaboration with the existing portfolio complexities, compliance with enterprise standards and governance etc.

There are alternatives. Craig Larman and Bas Vodde in their books [3] and more recently with their LeSS initiative [4] have pursued a very different path that starts with Scrum and scales by understanding the needs for coordination while adhering to core Scrum principles. In their forthcoming book [5] they say, “LeSS is (1) lightweight, (2) simple to understand, and (3) difficult to master—due to essential complexity”. And this allows us to contrast the different approaches; while SAFe clearly works at some level, it has its roots in conventional large scale project management, whereas LeSS is lightweight, but requires much deeper understanding of the systems dynamics because of the inherent complexity of all the coordination requirements.
So the real question underlying Scaling Agile should be, “can we address some of the coordination requirements in a manner that reduces complexity and eliminates some of the need for additional layers of management or events?”

In my post Service Factory 2.0 [6] I describe the conceptual background of the Software Factory, ideas pioneered by my old friend Keith Short and Jack Greenfield while they were at Microsoft. Today these have evolved and become specialized around a framework of tools, repeatable processes and patterns for creation and assembly of services – manifest as first order components with formal interfaces. If you consider that all core business functionality is increasingly composed of services and their operations, this provides us with a reference architecture that by design implements separation of concerns.  Figure 1 below is a conceptual view of the scope of the service factory in terms of managed objects.

Figure 1

Naturally there will be other patterns involved in any solution including UX and workflow; but for most enterprise class solutions, a high proportion of the functionality should comprise services and business rules. This allows a consistent, highly automated approach to architecture, requirements specification, design and test. The factory effectively implements a proven service reference architecture as a platform which can be customized for individual programs, projects and technologies. While the factory platform is similar to an architecture runway in scope and intent, because it is model based the factory framework enables continuous evolution of the platform capabilities that are automatically inherited into work in progress factory code allowing program wide changes to be incorporated, often with minimal or no change to the service specific code. And the factory is a repository of the life cycle artifacts that can be leveraged for various management tasks including governance of traceability, impact analysis, etc.

Let’s consider the key coordination and management activities typically required in a scaled agile program and how the factory may facilitate that. In Figure 2 below I have used the SAFe layers of portfolio, program and team, albeit without the value stream level. But apart from that, the view is intended to be framework neutral. The diagram illustrates how the Factory provides a full life cycle backplane that establishes the underlying (meta) model that ensures all the participants are in compliance with core concepts from portfolio planning through to delivery and production. The populated model is therefore able to provide essential information particularly about dependencies, but equally complete traceability and governance between business requirements and delivered services and rules.
Figure 2

In the LeSS framework, Larman and Vodde disregard constructs such as ART (Agile Release Train) and 8 to 12 week PIs (Program Increments) and also the Hardening Sprint, preferring to use the standard Scrum duration of 2 to 4 weeks, with continuous integration.  We might assume these management constructs (ARTs and PIs) were invented because of the difficulty of effective inter team communications, and the overhead that creates. Larman and Vodde have a lot of good ideas about how to manage effective inter Teams communications, but their approach is essentially based on feature or story dependencies. In the factory environment inter-team dependencies can be more accurately based on service operation dependencies. But this is just the tip of the iceberg, in terms of how the factory can be leveraged to manage activity. The service specification becomes a pivotal work product; as it becomes available other teams and team members have access to detailed behaviour information that can inform and facilitate working ahead for developing depending services, test case development including automation assistance in case development, plus devops.

Another huge issue in Agile projects is managing technical debt. In the factory environment the standard patterns generate all the infrastructure and shared code, and when changes take place, as discussed above, teams can perform an architecture update, inheriting all the changes to bring their project in line with the latest reference architecture. And because the factory is driven by the specification level, independent of technology it is inherently late binding, allowing change if technology with minimum impact.

By utilizing work products that are part of a defined process based on the full life cycle reference architecture, the factory approach reduces the essential complexity of the scaling task. All moving parts are under management. This doesn’t detract from the inherent purity of the Agile approach. Rather, the defined process provides clarity over the minimum necessary “planning work” (intentional architecture, rules definition and business requirements). And the reference architecture and factory process provide a firm foundation on which emergent architecture can be more effectively carried out by the cross functional delivery team. This is likely to encourage programs to adopt a hybrid approach to their organizing model, embracing elements of LeSS like guidance to optimize multiple concurrent Scrum based sprints with continuous delivery, and to complement them with minimum necessary elements of SAFe, particularly those that are required to communicate to more conventional stakeholders. But, at least for mid-sized projects, maybe up to 10 teams, it seems overkill to adopt the formality of PI Planning and everything that comes with it.

In this post I have focused on the scaled Agile approach for projects, but it’s also useful to understand the factory has other important outcomes. First the highly modular nature of the reference architecture (see Figure 1) leads to “solutions” that are inherently agile. Because solutions are comprised primarily of services and rules with strong formal contracts, implemented using component based implementation patterns, the horizon of change is likely to be very constrained. Further the specification approach in which services, rules, data and processes are defined independent of implementation and technology provides high levels of visibility to the business, and high transparency and traceability of the implemented solution. Finally, we have a way to realize architecture in inherently agile solutions without Big Upfront Design, or indeed Big Upfront Organization, that will of course never become legacy because they can be continuously evolved.


[2] Reasoned criticisms of SAFe

[3] Scaling Lean & Agile, Craig Larman and Bas Vodde, Addison Wesley, 2009



Read the original blog entry...

More Stories By David Sprott

David Sprott is a consultant, researcher and educator specializing in service oriented architecture, application modernization and cloud computing. Since 1997 David founded and led the well known think tank CBDI Forum providing unique research and guidance around loose coupled architecture, technologies and practices to F5000 companies and governments worldwide. As CEO of Everware-CBDI International a UK based corporation, he directs the global research and international consulting operations of the leading independent advisors on Service Oriented Application Modernization.

@ThingsExpo Stories
Gemini is Yahoo’s native and search advertising platform. To ensure the quality of a complex distributed system that spans multiple products and components and across various desktop websites and mobile app and web experiences – both Yahoo owned and operated and third-party syndication (supply), with complex interaction with more than a billion users and numerous advertisers globally (demand) – it becomes imperative to automate a set of end-to-end tests 24x7 to detect bugs and regression. In th...
Widespread fragmentation is stalling the growth of the IIoT and making it difficult for partners to work together. The number of software platforms, apps, hardware and connectivity standards is creating paralysis among businesses that are afraid of being locked into a solution. EdgeX Foundry is unifying the community around a common IoT edge framework and an ecosystem of interoperable components.
"MobiDev is a software development company and we do complex, custom software development for everybody from entrepreneurs to large enterprises," explained Alan Winters, U.S. Head of Business Development at MobiDev, in this SYS-CON.tv interview at 21st Cloud Expo, held Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA.
Large industrial manufacturing organizations are adopting the agile principles of cloud software companies. The industrial manufacturing development process has not scaled over time. Now that design CAD teams are geographically distributed, centralizing their work is key. With large multi-gigabyte projects, outdated tools have stifled industrial team agility, time-to-market milestones, and impacted P&L stakeholders.
"Space Monkey by Vivent Smart Home is a product that is a distributed cloud-based edge storage network. Vivent Smart Home, our parent company, is a smart home provider that places a lot of hard drives across homes in North America," explained JT Olds, Director of Engineering, and Brandon Crowfeather, Product Manager, at Vivint Smart Home, in this SYS-CON.tv interview at @ThingsExpo, held Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA.
"Akvelon is a software development company and we also provide consultancy services to folks who are looking to scale or accelerate their engineering roadmaps," explained Jeremiah Mothersell, Marketing Manager at Akvelon, in this SYS-CON.tv interview at 21st Cloud Expo, held Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA.
In his session at 21st Cloud Expo, Carl J. Levine, Senior Technical Evangelist for NS1, will objectively discuss how DNS is used to solve Digital Transformation challenges in large SaaS applications, CDNs, AdTech platforms, and other demanding use cases. Carl J. Levine is the Senior Technical Evangelist for NS1. A veteran of the Internet Infrastructure space, he has over a decade of experience with startups, networking protocols and Internet infrastructure, combined with the unique ability to it...
"IBM is really all in on blockchain. We take a look at sort of the history of blockchain ledger technologies. It started out with bitcoin, Ethereum, and IBM evaluated these particular blockchain technologies and found they were anonymous and permissionless and that many companies were looking for permissioned blockchain," stated René Bostic, Technical VP of the IBM Cloud Unit in North America, in this SYS-CON.tv interview at 21st Cloud Expo, held Oct 31 – Nov 2, 2017, at the Santa Clara Conventi...
"Cloud Academy is an enterprise training platform for the cloud, specifically public clouds. We offer guided learning experiences on AWS, Azure, Google Cloud and all the surrounding methodologies and technologies that you need to know and your teams need to know in order to leverage the full benefits of the cloud," explained Alex Brower, VP of Marketing at Cloud Academy, in this SYS-CON.tv interview at 21st Cloud Expo, held Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clar...
Coca-Cola’s Google powered digital signage system lays the groundwork for a more valuable connection between Coke and its customers. Digital signs pair software with high-resolution displays so that a message can be changed instantly based on what the operator wants to communicate or sell. In their Day 3 Keynote at 21st Cloud Expo, Greg Chambers, Global Group Director, Digital Innovation, Coca-Cola, and Vidya Nagarajan, a Senior Product Manager at Google, discussed how from store operations and ...
"There's plenty of bandwidth out there but it's never in the right place. So what Cedexis does is uses data to work out the best pathways to get data from the origin to the person who wants to get it," explained Simon Jones, Evangelist and Head of Marketing at Cedexis, in this SYS-CON.tv interview at 21st Cloud Expo, held Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA.
SYS-CON Events announced today that CrowdReviews.com has been named “Media Sponsor” of SYS-CON's 22nd International Cloud Expo, which will take place on June 5–7, 2018, at the Javits Center in New York City, NY. CrowdReviews.com is a transparent online platform for determining which products and services are the best based on the opinion of the crowd. The crowd consists of Internet users that have experienced products and services first-hand and have an interest in letting other potential buye...
SYS-CON Events announced today that Telecom Reseller has been named “Media Sponsor” of SYS-CON's 22nd International Cloud Expo, which will take place on June 5-7, 2018, at the Javits Center in New York, NY. Telecom Reseller reports on Unified Communications, UCaaS, BPaaS for enterprise and SMBs. They report extensively on both customer premises based solutions such as IP-PBX as well as cloud based and hosted platforms.
It is of utmost importance for the future success of WebRTC to ensure that interoperability is operational between web browsers and any WebRTC-compliant client. To be guaranteed as operational and effective, interoperability must be tested extensively by establishing WebRTC data and media connections between different web browsers running on different devices and operating systems. In his session at WebRTC Summit at @ThingsExpo, Dr. Alex Gouaillard, CEO and Founder of CoSMo Software, presented ...
WebRTC is great technology to build your own communication tools. It will be even more exciting experience it with advanced devices, such as a 360 Camera, 360 microphone, and a depth sensor camera. In his session at @ThingsExpo, Masashi Ganeko, a manager at INFOCOM Corporation, introduced two experimental projects from his team and what they learned from them. "Shotoku Tamago" uses the robot audition software HARK to track speakers in 360 video of a remote party. "Virtual Teleport" uses a multip...
A strange thing is happening along the way to the Internet of Things, namely far too many devices to work with and manage. It has become clear that we'll need much higher efficiency user experiences that can allow us to more easily and scalably work with the thousands of devices that will soon be in each of our lives. Enter the conversational interface revolution, combining bots we can literally talk with, gesture to, and even direct with our thoughts, with embedded artificial intelligence, whic...
SYS-CON Events announced today that Evatronix will exhibit at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Evatronix SA offers comprehensive solutions in the design and implementation of electronic systems, in CAD / CAM deployment, and also is a designer and manufacturer of advanced 3D scanners for professional applications.
Leading companies, from the Global Fortune 500 to the smallest companies, are adopting hybrid cloud as the path to business advantage. Hybrid cloud depends on cloud services and on-premises infrastructure working in unison. Successful implementations require new levels of data mobility, enabled by an automated and seamless flow across on-premises and cloud resources. In his general session at 21st Cloud Expo, Greg Tevis, an IBM Storage Software Technical Strategist and Customer Solution Architec...
To get the most out of their data, successful companies are not focusing on queries and data lakes, they are actively integrating analytics into their operations with a data-first application development approach. Real-time adjustments to improve revenues, reduce costs, or mitigate risk rely on applications that minimize latency on a variety of data sources. In his session at @BigDataExpo, Jack Norris, Senior Vice President, Data and Applications at MapR Technologies, reviewed best practices to ...
An increasing number of companies are creating products that combine data with analytical capabilities. Running interactive queries on Big Data requires complex architectures to store and query data effectively, typically involving data streams, an choosing efficient file format/database and multiple independent systems that are tied together through custom-engineered pipelines. In his session at @BigDataExpo at @ThingsExpo, Tomer Levi, a senior software engineer at Intel’s Advanced Analytics gr...