Welcome!

@DevOpsSummit Authors: Jason Bloomberg, Stackify Blog, Aruna Ravichandran, Otto Berkes, Ayman Sayed

Related Topics: @DevOpsSummit, Linux Containers, Containers Expo Blog

@DevOpsSummit: Blog Feed Post

Scaled Agile Factory - Going Beyond the Usual Suspects | @DevOpsSummit #DevOps

The question of how to scale Agile development has been around a while

If you address the question of how to scale Agile projects by considering what framework to use, you are only looking at one aspect of the problem. Scaling is all about coordination – managing enterprise considerations and cross program dependencies, and the defacto frameworks (SAFe, LeSS and DAD) focus on the people and process dimensions. However, in combination with a factory approach you may be able to automate many of the compliance and dependency management issues.

The question of how to scale Agile development has been around a while. In January 2015 I commented [1] on a clear trend in which my customers were voicing concerns about loss of consistency; inability to govern; lack of coordination and increasing time to market. Since then I have observed many large organizations adopting SAFe (Scaled Agile Framework) perhaps because it appears to be the only game in town, or perhaps there’s good marketing or there’s strength in numbers. But the criticisms of SAFe [2] haven’t gone away. The central and continuing concern is that SAFe compromises core Agile principles of self-organizing, cross functional teams that have full responsibility for the delivery of potentially shippable increments. And that SAFe looks suspiciously like WaterScrumFall because there’s a high overhead of portfolio, value stream and project management and in all probability intentional architecture. Now I have mixed opinions on SAFe; I see positives in value streams and product management which are important. And for organizations that have large programs, highly organized, structured approaches will be seen as lower risk, and indeed something that takes them some way down the Agile path, without straying too far from conventional management comfort zones. But the outcomes are unlikely to be inherently “agile”.

What SAFe does, is provide a rather conventional approach for a complex problem that most enterprises have – that is to deliver high integrity solutions that can work in an enterprise context that demands cross project dependency management, consistent data and reference architecture, collaboration with the existing portfolio complexities, compliance with enterprise standards and governance etc.

There are alternatives. Craig Larman and Bas Vodde in their books [3] and more recently with their LeSS initiative [4] have pursued a very different path that starts with Scrum and scales by understanding the needs for coordination while adhering to core Scrum principles. In their forthcoming book [5] they say, “LeSS is (1) lightweight, (2) simple to understand, and (3) difficult to master—due to essential complexity”. And this allows us to contrast the different approaches; while SAFe clearly works at some level, it has its roots in conventional large scale project management, whereas LeSS is lightweight, but requires much deeper understanding of the systems dynamics because of the inherent complexity of all the coordination requirements.
So the real question underlying Scaling Agile should be, “can we address some of the coordination requirements in a manner that reduces complexity and eliminates some of the need for additional layers of management or events?”

In my post Service Factory 2.0 [6] I describe the conceptual background of the Software Factory, ideas pioneered by my old friend Keith Short and Jack Greenfield while they were at Microsoft. Today these have evolved and become specialized around a framework of tools, repeatable processes and patterns for creation and assembly of services – manifest as first order components with formal interfaces. If you consider that all core business functionality is increasingly composed of services and their operations, this provides us with a reference architecture that by design implements separation of concerns.  Figure 1 below is a conceptual view of the scope of the service factory in terms of managed objects.

Figure 1

Naturally there will be other patterns involved in any solution including UX and workflow; but for most enterprise class solutions, a high proportion of the functionality should comprise services and business rules. This allows a consistent, highly automated approach to architecture, requirements specification, design and test. The factory effectively implements a proven service reference architecture as a platform which can be customized for individual programs, projects and technologies. While the factory platform is similar to an architecture runway in scope and intent, because it is model based the factory framework enables continuous evolution of the platform capabilities that are automatically inherited into work in progress factory code allowing program wide changes to be incorporated, often with minimal or no change to the service specific code. And the factory is a repository of the life cycle artifacts that can be leveraged for various management tasks including governance of traceability, impact analysis, etc.

Let’s consider the key coordination and management activities typically required in a scaled agile program and how the factory may facilitate that. In Figure 2 below I have used the SAFe layers of portfolio, program and team, albeit without the value stream level. But apart from that, the view is intended to be framework neutral. The diagram illustrates how the Factory provides a full life cycle backplane that establishes the underlying (meta) model that ensures all the participants are in compliance with core concepts from portfolio planning through to delivery and production. The populated model is therefore able to provide essential information particularly about dependencies, but equally complete traceability and governance between business requirements and delivered services and rules.
Figure 2

In the LeSS framework, Larman and Vodde disregard constructs such as ART (Agile Release Train) and 8 to 12 week PIs (Program Increments) and also the Hardening Sprint, preferring to use the standard Scrum duration of 2 to 4 weeks, with continuous integration.  We might assume these management constructs (ARTs and PIs) were invented because of the difficulty of effective inter team communications, and the overhead that creates. Larman and Vodde have a lot of good ideas about how to manage effective inter Teams communications, but their approach is essentially based on feature or story dependencies. In the factory environment inter-team dependencies can be more accurately based on service operation dependencies. But this is just the tip of the iceberg, in terms of how the factory can be leveraged to manage activity. The service specification becomes a pivotal work product; as it becomes available other teams and team members have access to detailed behaviour information that can inform and facilitate working ahead for developing depending services, test case development including automation assistance in case development, plus devops.

Another huge issue in Agile projects is managing technical debt. In the factory environment the standard patterns generate all the infrastructure and shared code, and when changes take place, as discussed above, teams can perform an architecture update, inheriting all the changes to bring their project in line with the latest reference architecture. And because the factory is driven by the specification level, independent of technology it is inherently late binding, allowing change if technology with minimum impact.

By utilizing work products that are part of a defined process based on the full life cycle reference architecture, the factory approach reduces the essential complexity of the scaling task. All moving parts are under management. This doesn’t detract from the inherent purity of the Agile approach. Rather, the defined process provides clarity over the minimum necessary “planning work” (intentional architecture, rules definition and business requirements). And the reference architecture and factory process provide a firm foundation on which emergent architecture can be more effectively carried out by the cross functional delivery team. This is likely to encourage programs to adopt a hybrid approach to their organizing model, embracing elements of LeSS like guidance to optimize multiple concurrent Scrum based sprints with continuous delivery, and to complement them with minimum necessary elements of SAFe, particularly those that are required to communicate to more conventional stakeholders. But, at least for mid-sized projects, maybe up to 10 teams, it seems overkill to adopt the formality of PI Planning and everything that comes with it.

In this post I have focused on the scaled Agile approach for projects, but it’s also useful to understand the factory has other important outcomes. First the highly modular nature of the reference architecture (see Figure 1) leads to “solutions” that are inherently agile. Because solutions are comprised primarily of services and rules with strong formal contracts, implemented using component based implementation patterns, the horizon of change is likely to be very constrained. Further the specification approach in which services, rules, data and processes are defined independent of implementation and technology provides high levels of visibility to the business, and high transparency and traceability of the implemented solution. Finally, we have a way to realize architecture in inherently agile solutions without Big Upfront Design, or indeed Big Upfront Organization, that will of course never become legacy because they can be continuously evolved.


[2] Reasoned criticisms of SAFe

[3] Scaling Lean & Agile, Craig Larman and Bas Vodde, Addison Wesley, 2009



Read the original blog entry...

More Stories By David Sprott

David Sprott is a consultant, researcher and educator specializing in service oriented architecture, application modernization and cloud computing. Since 1997 David founded and led the well known think tank CBDI Forum providing unique research and guidance around loose coupled architecture, technologies and practices to F5000 companies and governments worldwide. As CEO of Everware-CBDI International a UK based corporation, he directs the global research and international consulting operations of the leading independent advisors on Service Oriented Application Modernization.

@DevOpsSummit Stories
"ZeroStack is a startup in Silicon Valley. We're solving a very interesting problem around bringing public cloud convenience with private cloud control for enterprises and mid-size companies," explained Kamesh Pemmaraju, VP of Product Management at ZeroStack, in this SYS-CON.tv interview at 21st Cloud Expo, held Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA.
"CA has been doing a lot of things in the area of DevOps. Now we have a complete set of tool sets in order to enable customers to go all the way from planning to development to testing down to release into the operations," explained Aruna Ravichandran, Vice President of Global Marketing and Strategy at CA Technologies, in this SYS-CON.tv interview at DevOps Summit at 21st Cloud Expo, held Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA.
"Infoblox does DNS, DHCP and IP address management for not only enterprise networks but cloud networks as well. Customers are looking for a single platform that can extend not only in their private enterprise environment but private cloud, public cloud, tracking all the IP space and everything that is going on in that environment," explained Steve Salo, Principal Systems Engineer at Infoblox, in this SYS-CON.tv interview at 21st Cloud Expo, held Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA.
SYS-CON Events announced today that CrowdReviews.com has been named “Media Sponsor” of SYS-CON's 22nd International Cloud Expo, which will take place on June 5–7, 2018, at the Javits Center in New York City, NY. CrowdReviews.com is a transparent online platform for determining which products and services are the best based on the opinion of the crowd. The crowd consists of Internet users that have experienced products and services first-hand and have an interest in letting other potential buyers learn their thoughts on their experience.
SYS-CON Events announced today that Telecom Reseller has been named “Media Sponsor” of SYS-CON's 22nd International Cloud Expo, which will take place on June 5-7, 2018, at the Javits Center in New York, NY. Telecom Reseller reports on Unified Communications, UCaaS, BPaaS for enterprise and SMBs. They report extensively on both customer premises based solutions such as IP-PBX as well as cloud based and hosted platforms.
While some developers care passionately about how data centers and clouds are architected, for most, it is only the end result that matters. To the majority of companies, technology exists to solve a business problem, and only delivers value when it is solving that problem. 2017 brings the mainstream adoption of containers for production workloads. In his session at 21st Cloud Expo, Ben McCormack, VP of Operations at Evernote, discussed how data centers of the future will be managed, how the public cloud best suits your organization, and what the future holds for operations and infrastructure engineers in a post-container world. Is a serverless world inevitable?
Enterprises are adopting Kubernetes to accelerate the development and the delivery of cloud-native applications. However, sharing a Kubernetes cluster between members of the same team can be challenging. And, sharing clusters across multiple teams is even harder. Kubernetes offers several constructs to help implement segmentation and isolation. However, these primitives can be complex to understand and apply. As a result, it’s becoming common for enterprises to end up with several clusters. This leads to a waste of cloud resources and increased operational overhead.
A strange thing is happening along the way to the Internet of Things, namely far too many devices to work with and manage. It has become clear that we'll need much higher efficiency user experiences that can allow us to more easily and scalably work with the thousands of devices that will soon be in each of our lives. Enter the conversational interface revolution, combining bots we can literally talk with, gesture to, and even direct with our thoughts, with embedded artificial intelligence, which can process our conversational commands and orchestrate the outcomes we request across our personal and professional realm of connected devices.
DevOps promotes continuous improvement through a culture of collaboration. But in real terms, how do you: Integrate activities across diverse teams and services? Make objective decisions with system-wide visibility? Use feedback loops to enable learning and improvement? With technology insights and real-world examples, in his general session at @DevOpsSummit, at 21st Cloud Expo, Andi Mann, Chief Technology Advocate at Splunk, explored how leading organizations use data-driven DevOps to close their feedback loops to drive continuous improvement.
SYS-CON Events announced today that Evatronix will exhibit at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Evatronix SA offers comprehensive solutions in the design and implementation of electronic systems, in CAD / CAM deployment, and also is a designer and manufacturer of advanced 3D scanners for professional applications.
Sanjeev Sharma Joins June 5-7, 2018 @DevOpsSummit at @Cloud Expo New York Faculty. Sanjeev Sharma is an internationally known DevOps and Cloud Transformation thought leader, technology executive, and author. Sanjeev's industry experience includes tenures as CTO, Technical Sales leader, and Cloud Architect leader. As an IBM Distinguished Engineer, Sanjeev is recognized at the highest levels of IBM's core of technical leaders.
We all know that end users experience the Internet primarily with mobile devices. From an app development perspective, we know that successfully responding to the needs of mobile customers depends on rapid DevOps – failing fast, in short, until the right solution evolves in your customers' relationship to your business. Whether you’re decomposing an SOA monolith, or developing a new application cloud natively, it’s not a question of using microservices – not doing so will be a path to eventual business failure.
"Cloud4U builds software services that help people build DevOps platforms for cloud-based software and using our platform people can draw a picture of the system, network, software," explained Kihyeon Kim, CEO and Head of R&D at Cloud4U, in this SYS-CON.tv interview at 21st Cloud Expo, held Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA.
Is advanced scheduling in Kubernetes achievable?Yes, however, how do you properly accommodate every real-life scenario that a Kubernetes user might encounter? How do you leverage advanced scheduling techniques to shape and describe each scenario in easy-to-use rules and configurations? In his session at @DevOpsSummit at 21st Cloud Expo, Oleg Chunikhin, CTO at Kublr, answered these questions and demonstrated techniques for implementing advanced scheduling. For example, using spot instances and cost-effective resources on AWS, coupled with the ability to deliver a minimum set of functionalities that cover the majority of needs – without configuration complexity.
As DevOps methodologies expand their reach across the enterprise, organizations face the daunting challenge of adapting related cloud strategies to ensure optimal alignment, from managing complexity to ensuring proper governance. How can culture, automation, legacy apps and even budget be reexamined to enable this ongoing shift within the modern software factory? In her Day 2 Keynote at @DevOpsSummit at 21st Cloud Expo, Aruna Ravichandran, VP, DevOps Solutions Marketing, CA Technologies, was joined by a panel of industry experts and real-world practitioners who shared their insight into an emerging set of best practices that lie at the heart of today's digital transformation.
SYS-CON Events announced today that Synametrics Technologies will exhibit at SYS-CON's 22nd International Cloud Expo®, which will take place on June 5-7, 2018, at the Javits Center in New York, NY. Synametrics Technologies is a privately held company based in Plainsboro, New Jersey that has been providing solutions for the developer community since 1997. Based on the success of its initial product offerings such as WinSQL, Xeams, SynaMan and Syncrify, Synametrics continues to create and hone innovative products that help customers get more from their computer applications, databases and infrastructure. To date, over one million users around the world have chosen Synametrics solutions to help power their accelerated business and personal computing needs.
As many know, the first generation of Cloud Management Platform (CMP) solutions were designed for managing virtual infrastructure (IaaS) and traditional applications. But that's no longer enough to satisfy evolving and complex business requirements. In his session at 21st Cloud Expo, Scott Davis, Embotics CTO, explored how next-generation CMPs ensure organizations can manage cloud-native and microservice-based application architectures, while also facilitating agile DevOps methodology. He explained how automation, orchestration and governance are fundamental to managing today's hybrid cloud environments and are critical for digital businesses to deliver services faster, with better user experience and higher quality, all while saving money.
SYS-CON Events announced today that Google Cloud has been named “Keynote Sponsor” of SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Companies come to Google Cloud to transform their businesses. Google Cloud’s comprehensive portfolio – from infrastructure to apps to devices – helps enterprises innovate faster, scale smarter, stay secure, and do more with data than ever before.
Cloud Expo | DXWorld Expo have announced the conference tracks for Cloud Expo 2018. Cloud Expo will be held June 5-7, 2018, at the Javits Center in New York City, and November 6-8, 2018, at the Santa Clara Convention Center, Santa Clara, CA. Digital Transformation (DX) is a major focus with the introduction of DX Expo within the program. Successful transformation requires a laser focus on being data-driven and on using all the tools available that enable transformation if they plan to survive over the long term. A total of 88% of Fortune 500 companies from a generation ago are now out of business. Only 12% still survive. Similar percentages are found throughout enterprises of all sizes.
Continuous Delivery makes it possible to exploit findings of cognitive psychology and neuroscience to increase the productivity and happiness of our teams. In his session at 22nd Cloud Expo | DXWorld Expo, Daniel Jones, CTO of EngineerBetter, will answer: How can we improve willpower and decrease technical debt? Is the present bias real? How can we turn it to our advantage? Can you increase a team’s effective IQ? How do DevOps & Product Teams increase empathy, and what impact does empathy have on productivity?
There is a huge demand for responsive, real-time mobile and web experiences, but current architectural patterns do not easily accommodate applications that respond to events in real time. Common solutions using message queues or HTTP long-polling quickly lead to resiliency, scalability and development velocity challenges. In his session at 21st Cloud Expo, Ryland Degnan, a Senior Software Engineer on the Netflix Edge Platform team, will discuss how by leveraging a reactive stream-based protocol, we have been able to solve many of these problems at the communication layer. This makes it possible to create rich application experiences and support use-cases such as mobile-to-mobile communication and large file transfers that would be difficult or cost-prohibitive with traditional networking.
You know you need the cloud, but you're hesitant to simply dump everything at Amazon since you know that not all workloads are suitable for cloud. You know that you want the kind of ease of use and scalability that you get with public cloud, but your applications are architected in a way that makes the public cloud a non-starter. You're looking at private cloud solutions based on hyperconverged infrastructure, but you're concerned with the limits inherent in those technologies. What do you do?
The 22nd International Cloud Expo | 1st DXWorld Expo has announced that its Call for Papers is open. Cloud Expo | DXWorld Expo, to be held June 5-7, 2018, at the Javits Center in New York, NY, brings together Cloud Computing, Digital Transformation, Big Data, Internet of Things, DevOps, Machine Learning and WebRTC to one location. With cloud computing driving a higher percentage of enterprise IT budgets every year, it becomes increasingly important to plant your flag in this fast-expanding business opportunity. Submit your speaking proposal today!
The past few years have brought a sea change in the way applications are architected, developed, and consumed—increasing both the complexity of testing and the business impact of software failures. How can software testing professionals keep pace with modern application delivery, given the trends that impact both architectures (cloud, microservices, and APIs) and processes (DevOps, agile, and continuous delivery)? This is where continuous testing comes in. D
In his Opening Keynote at 21st Cloud Expo, John Considine, General Manager of IBM Cloud Infrastructure, led attendees through the exciting evolution of the cloud. He looked at this major disruption from the perspective of technology, business models, and what this means for enterprises of all sizes. John Considine is General Manager of Cloud Infrastructure Services at IBM. In that role he is responsible for leading IBM’s public cloud infrastructure including strategy, development, and offering management. To date, IBM has launched more than 50 cloud data centers that span the globe. He has been building advanced technology, delivering “as a service” solutions, and managing infrastructure services for the past 20 years.