|By Michael Bushong||
|July 28, 2014 06:00 AM EDT||
The benefits of automation are well understood: more agile service provisioning, faster time to insight when there are issues, and a reduction in human error as manual interaction is reduced. Much of the premise behind long-term SDN architectural advantages is steeped in the hope that SDN will help enable and ultimately promote automation. But while centralizing control has significant operational advantages, by itself, it doesn’t actually address the most important requirement for automation.
If automation is going to be more than just reducing keystrokes, there will have to be a rise of workflow state.
Successfully managing a network is an exercise in constant iteration through network state. Whenever something needs to be done, the architect or operator examines her current frame of reference to figure out the starting point. That frame of reference usually starts with some implicit understanding of how the network is designed. From there, she takes some action. Maybe she pings an endpoint, checks the state of a BGP neighbor, or examines some interface statistics. Whatever the first step, the point is that she knows when she starts that there is work after the first step.
The information gleaned from the first step yields additional understanding. Her frame of reference changes as she now knows more than before. With her new position in referential space, she takes the next step. And the next, and the next after that. Each step yields a different piece of information, and the process of iterating through a constantly changing referential space ultimately yields some outcome or resolution.
Byproducts of iterative workflows
There are two major byproducts of this iterative approach to workflow. The first is that the starting point is rarely based on an absolute understanding of fact. Rather it is an interpretation that the individual operator or architect creates based on a number of somewhat soft conditions – knowledge, experience, intuition, whatever. This means that for each task, the workflow is somewhat unique, depending on the operator and the environment.
The impact here is important. If workflows are unique based on the operator and the conditions (i.e., the referential space or frame of reference), then the outcomes driven by those workflows are difficult to repeat. Part of why networking is so hard is that so much of it borders on arcane dark art. Science demands repeatability, but the very nature of workflow management in networking makes that challenging.
The second byproduct of networking’s iterative nature is that workflows frequently depend on a set of chained tasks, each of which has a dependency on the preceding task. To make things worse, that dependency is actually rarely known at the start of a a workflow. It’s not that tasks cannot be predictably chained – first, you look at the physical layer, and then you move up stack perhaps. But each subsequent task is executed based on not just the previous task but also the output of the previous task. This creates a complex set of if/then statements in most workflows.
Part of the challenge in automation is providing the logic to navigate the conditional nature of networking workflows.
“Network engineers need to think like programmers”
With the rise of movements like DevOps, “network engineers thinking like programmers” has become a popular phrase. This is a very important change in how we handle network architecture and operations. But there are subtleties here that get lost in the cliche.
First, when people toss the phrase around, they often mean that network engineers need to pick up a scripting language (Python, Ruby, even Perl). Thinking like a software developer has very little to do with programming languages. Languages are a way of expressing intent, but it’s entirely possible to know Python and think nothing like a developer.
Second, when people refer to programming in the context of DevOps, they generally mean that network operators need to think about configuration less as a collection of commands and more like code. Once you make that shift, then you can think about things like source code management, automated testing, and rapid deployment.
But networking needs to do more than just treat configuration as code. DevOps has more to do with deploying and validating changes. It doesn’t fundamentally change how workflows are executed, and it barely touches more operational tasks like troubleshooting network conditions.
Before anyone picks a religious battle over DevOps here, my point is not that DevOps is bad. It’s just that DevOps by itself is not sufficient. And there are things that ought to be done that are separate from DevOps.
Tiny feedback loops
So if thinking like a programmer isn’t about learning a programming language and it’s more than treating configuration as code, what is it?
Software development is really about creating something out of lots of tiny feedback loops. When you write functions, you don’t just execute some task. You generally execute that task and then return a value. The value provides some immediate feedback about the outcome. In some cases, the function returns the value of a computation; in other cases, it simply returns an indication that the function succeeded or failed.
These values are obviously then used by other functions, which allows us to string together small building blocks into complex chains. The important part? These chains can then be repeatably executed in a deterministic way.
Networking workflows shouldn’t be that different. Each individual activity yields some value (sometimes a specific value as when looking at some counter, other times a success or failure as with a ping). The problem is that while networking commands frequently return information, it is up to the operator themselves to parse this information, analyze what it means, and then take the next action.
What we need if we really want to make automation happen in ways that extend beyond just scripting keystrokes is a means of creating deterministic networking workflows. For this to happen, we need people who construct workflows to think more like developers. Each activity within a workflow needs to be a tiny feedback loop with explicit workflow state that is programmatically passed between workflow elements.
We actually instinctively do this at times. XML, NETCONF, and the like have been used to encapsulate networking inputs and outputs for awhile with the intent of making things parseable and thus more automatable.
But we stopped short. We made the outputs more automation-friendly without ever really creating workflows. So while we can programmatically act on values, it only works if someone has automated a particular workflow. As an industry, we haven’t gotten to actually addressing the workflow problem.
Maybe it’s the highly conditional nature of networking combined with the uniqueness of individual networks. Or maybe it’s that outside of a few automation savants, our industry doesn’t generally think about workflows the way a software developer would.
The bottom line
Networking workflows rely way too heavily on an iterative pass through referential space. The reason change is so scary and troubleshooting so hard is that very little in networking is actually deterministic. But if we really want to improve the overall user experience en route to making workflows both repeatable and reliable, we do need to start thinking a bit more like developers. It all starts with a more explicit understanding of the workflows we rely on, and the expression of feedback via some form of workflow state.
And for everyone betting on abstractions, just know that abstracting a poorly-defined workflow results in an equally poor abstraction. We need to be starting elsewhere.
[Today’s fun fact: Only male fireflies can fly. Take that, females!]
Cloud computing is unquestionably one of the driving forces of DevOps, as the automation of operations transforms enterprise software development. DevOps, however, is more than a technology trend, as it represents a move toward silo-busting, self-organizing horizontal teams that drive business velocity. At the same time, enterprise Digital Transformation represents an upheaval across the enterprise, as customer preferences and behavior drive enterprise technology decisions. This transformation extends well beyond technology, however, as companies must rethink their organizational structure, m...
Dec. 1, 2015 03:45 PM EST
Most of the IoT Gateway scenarios involve collecting data from machines/processing and pushing data upstream to cloud for further analytics. The gateway hardware varies from Raspberry Pi to Industrial PCs. The document states the process of allowing deploying polyglot data pipelining software with the clear notion of supporting immutability. In his session at @ThingsExpo, Shashank Jain, a development architect for SAP Labs, discussed the objective, which is to automate the IoT deployment process from development to production scenarios using Docker containers.
Dec. 1, 2015 03:00 PM EST Reads: 145
DevOps is about increasing efficiency, but nothing is more inefficient than building the same application twice. However, this is a routine occurrence with enterprise applications that need both a rich desktop web interface and strong mobile support. With recent technological advances from Isomorphic Software and others, rich desktop and tuned mobile experiences can now be created with a single codebase – without compromising functionality, performance or usability. In his session at DevOps Summit, Charles Kendrick, CTO and Chief Architect at Isomorphic Software, demonstrated examples of com...
Dec. 1, 2015 02:45 PM EST Reads: 448
OpsHub, Inc. has announced enhanced support for DevOps and Migration for both Team Foundation Server and Visual Studio On-line in a heterogeneous environment. With added support for build and release entities in OpsHub Integration Manager (OIM) Microsoft customers can now leverage Visual Studio build and release services to manage DevOps processes in a heterogeneous environment. With the enhanced support customers can manage the DevOps process in Team Foundation Server while undertaking activities in heterogeneous tools as a part of the automated process.
Dec. 1, 2015 01:42 PM EST
SYS-CON Events announced today that Catchpoint, a global leader in monitoring, and testing the performance of online applications, has been named "Silver Sponsor" of DevOps Summit New York, which will take place on June 7-9, 2016 at the Javits Center in New York City. Catchpoint radically transforms the way businesses manage, monitor, and test the performance of online applications. Truly understand and improve user experience with clear visibility into complex, distributed online systems.Founded in 2008 by four DoubleClick / Google executives with a passion for speed, reliability and overall ...
Dec. 1, 2015 12:15 PM EST
Actifio is powering new application development and testing services from Net3 Technologies (N3T), a managed cloud services provider. N3T's new Symmetry DevOps™ service builds on its existing Palmetto Virtual Data Center (PvDC) Cloud services for data backup and disaster recovery (DR) based on the Actifio Copy Data Virtualization platform. Previously, N3T's data protection and DR services were challenged by overlapping and inefficient legacy hardware and software platforms from multiple vendors, which meant higher costs in licensing and management overhead, and difficulty in responding rapi...
Dec. 1, 2015 11:30 AM EST
Discussions of cloud computing have evolved in recent years from a focus on specific types of cloud, to a world of hybrid cloud, and to a world dominated by the APIs that make today's multi-cloud environments and hybrid clouds possible. In this Power Panel at 17th Cloud Expo, moderated by Conference Chair Roger Strukhoff, panelists addressed the importance of customers being able to use the specific technologies they need, through environments and ecosystems that expose their APIs to make true change and transformation possible.
Dec. 1, 2015 10:00 AM EST Reads: 581
Microservices are a very exciting architectural approach that many organizations are looking to as a way to accelerate innovation. Microservices promise to allow teams to move away from monolithic "ball of mud" systems, but the reality is that, in the vast majority of organizations, different projects and technologies will continue to be developed at different speeds. How to handle the dependencies between these disparate systems with different iteration cycles? Consider the "canoncial problem" in this scenario: microservice A (releases daily) depends on a couple of additions to backend B (re...
Dec. 1, 2015 09:00 AM EST Reads: 485
Container technology is shaping the future of DevOps and it’s also changing the way organizations think about application development. With the rise of mobile applications in the enterprise, businesses are abandoning year-long development cycles and embracing technologies that enable rapid development and continuous deployment of apps. In his session at DevOps Summit, Kurt Collins, Developer Evangelist at Built.io, examined how Docker has evolved into a highly effective tool for application delivery by allowing increasingly popular Mobile Backend-as-a-Service (mBaaS) platforms to quickly crea...
Dec. 1, 2015 08:00 AM EST Reads: 398
Growth hacking is common for startups to make unheard-of progress in building their business. Career Hacks can help Geek Girls and those who support them (yes, that's you too, Dad!) to excel in this typically male-dominated world. Get ready to learn the facts: Is there a bias against women in the tech / developer communities? Why are women 50% of the workforce, but hold only 24% of the STEM or IT positions? Some beginnings of what to do about it! In her Day 2 Keynote at 17th Cloud Expo, Sandy Carter, IBM General Manager Cloud Ecosystem and Developers, and a Social Business Evangelist, wil...
Dec. 1, 2015 05:00 AM EST Reads: 624
Culture is the most important ingredient of DevOps. The challenge for most organizations is defining and communicating a vision of beneficial DevOps culture for their organizations, and then facilitating the changes needed to achieve that. Often this comes down to an ability to provide true leadership. As a CIO, are your direct reports IT managers or are they IT leaders? The hard truth is that many IT managers have risen through the ranks based on their technical skills, not their leadership ability. Many are unable to effectively engage and inspire, creating forward momentum in the direction...
Dec. 1, 2015 01:00 AM EST Reads: 438
In his General Session at DevOps Summit, Asaf Yigal, Co-Founder & VP of Product at Logz.io, explored the value of Kibana 4 for log analysis and provided a hands-on tutorial on how to set up Kibana 4 and get the most out of Apache log files. He examined three use cases: IT operations, business intelligence, and security and compliance. Asaf Yigal is co-founder and VP of Product at log analytics software company Logz.io. In the past, he was co-founder of social-trading platform Currensee, which was later acquired by OANDA. He was also an early employee of server performance-monitoring company...
Nov. 30, 2015 10:00 PM EST Reads: 292
Continuous processes around the development and deployment of applications are both impacted by -- and a benefit to -- the Internet of Things trend. To help better understand the relationship between DevOps and a plethora of new end-devices and data please welcome Gary Gruver, consultant, author and a former IT executive who has led many large-scale IT transformation projects, and John Jeremiah, Technology Evangelist at Hewlett Packard Enterprise (HPE), on Twitter at @j_jeremiah. The discussion is moderated by me, Dana Gardner, Principal Analyst at Interarbor Solutions.
Nov. 29, 2015 06:45 AM EST Reads: 757
With major technology companies and startups seriously embracing IoT strategies, now is the perfect time to attend @ThingsExpo 2016 in New York and Silicon Valley. Learn what is going on, contribute to the discussions, and ensure that your enterprise is as "IoT-Ready" as it can be! Internet of @ThingsExpo, taking place Nov 3-5, 2015, at the Santa Clara Convention Center in Santa Clara, CA, is co-located with 17th Cloud Expo and will feature technical sessions from a rock star conference faculty and the leading industry players in the world. The Internet of Things (IoT) is the most profound cha...
Nov. 28, 2015 12:00 PM EST Reads: 577
Internet of @ThingsExpo, taking place June 7-9, 2016 at Javits Center, New York City and Nov 1-3, 2016, at the Santa Clara Convention Center in Santa Clara, CA, is co-located with the 18th International @CloudExpo and will feature technical sessions from a rock star conference faculty and the leading industry players in the world and ThingsExpo New York Call for Papers is now open.
Nov. 27, 2015 12:00 PM EST Reads: 574
There are over 120 breakout sessions in all, with Keynotes, General Sessions, and Power Panels adding to three days of incredibly rich presentations and content. Join @ThingsExpo conference chair Roger Strukhoff (@IoT2040), June 7-9, 2016 in New York City, for three days of intense 'Internet of Things' discussion and focus, including Big Data's indespensable role in IoT, Smart Grids and Industrial Internet of Things, Wearables and Consumer IoT, as well as (new) IoT's use in Vertical Markets.
Nov. 25, 2015 12:00 PM EST Reads: 565
IBM has announced a new cloud-based service that enables developers to automatically translate cloud and mobile apps into the world's most-spoken languages. IBM Globalization Pipeline, now available in beta on Bluemix, IBM's Cloud platform, rapidly opens up new global markets to companies without requiring them to rebuild or redeploy their apps. The beta version will support English as the base language and nine additional languages including: French, German, Spanish, Brazilian Portuguese, Italian, Japanese, Simplified Chinese, Traditional Chinese, and Korean.
Nov. 25, 2015 08:00 AM EST
After more than five years of DevOps, definitions are evolving, boundaries are expanding, ‘unicorns’ are no longer rare, enterprises are on board, and pundits are moving on. Can we now look at an evolution of DevOps? Should we? Is the foundation of DevOps ‘done’, or is there still too much left to do? What is mature, and what is still missing? What does the next 5 years of DevOps look like? In this Power Panel at DevOps Summit, moderated by DevOps Summit Conference Chair Andi Mann, panelists looked back at what DevOps has become, and forward at what it might create next.
Nov. 20, 2015 05:00 PM EST Reads: 403
SYS-CON Events announced today that TechTarget has been named “Media Sponsor” of SYS-CON's 17th International Cloud Expo®, which will take place on November 3–5, 2015, at the Santa Clara Convention Center in Santa Clara, CA. TechTarget storage websites are the best online information resource for news, tips and expert advice for the storage, backup and disaster recovery markets.
Nov. 5, 2015 10:00 AM EST Reads: 587
SYS-CON Events announced today that O'Reilly Media has been named “Media Sponsor” of SYS-CON's 17th International Cloud Expo®, which will take place on November 3–5, 2015, at the Santa Clara Convention Center in Santa Clara, CA. O'Reilly spreads the knowledge of innovators through its technology books, online services, research, and tech conferences. An active participant in the technology community, O'Reilly has a long history of advocacy, meme-making, and evangelism.
Nov. 5, 2015 09:45 AM EST Reads: 555
Docker is hot. However, as Docker container use spreads into more mature production pipelines, there can be issues about control of Docker images to ensure they are production-ready. Is a promotion-based model appropriate to control and track the flow of Docker images from development to production? In his session at DevOps Summit, Fred Simon, Co-founder and Chief Architect of JFrog, will demonstrate how to implement a promotion model for Docker images using a binary repository, and then show how to distribute them to any kind of consumer, being it a customer or a data center.
Nov. 3, 2015 10:00 AM EST Reads: 737
In today's enterprise, digital transformation represents organizational change even more so than technology change, as customer preferences and behavior drive end-to-end transformation across lines of business as well as IT. To capitalize on the ubiquitous disruption driving this transformation, companies must be able to innovate at an increasingly rapid pace.
Nov. 3, 2015 09:00 AM EST Reads: 524
Webair, a leading provider of Cloud Hosting, Colocation and Managed solutions, today announces that its Chief Technology Officer, Sagi Brody, will speak at Cloud Expo 2015 Silicon Valley, to be held November 3-5, 2015, at the Santa Clara Convention Center in Santa Clara, California. Cloud Expo 2015 Silicon Valley is a world-class conference that brings together thought-leaders and cutting edge practitioners in the cloud / utility computing, Big Data, Internet of Things (IoT), Software-Defined Data Center (SDDC), DevOps and Web Real-Time Communication (WebRTC) space, which, in addition to prest...
Nov. 3, 2015 06:00 AM EST Reads: 516
DevOps is a software development method that places emphasis on communications between Software Engineering, Quality Assurance and IT Operations (SEQAITO ) with the goal to produce software and services to improve, increase the operational performance for the Enterprise. Communications is key not only between the SEQAITO team members but also the communication between the applications and the SEQAITO team. How can an organization provide the human communication and the application communication to the SEQAITO team to ensure the successful development, deployment of the application?
Nov. 3, 2015 05:00 AM EST Reads: 473
Logz.io, the Predictive ELK (Elasticsearch, Logstash and Kibana) log analytics cloud service company, announced today that it was chosen to speak at DevOps Summit, which will take place on November 3-5 in Santa Clara, California. Logz.io will explore the entire process that we have undergone – through research, benchmarking, implementation, optimization, and customer success – in developing a processing engine that can handle petabytes of data. We will also discuss the requirements of such an engine in terms of scalability, resilience, security, and availability along with how the architectur...
Nov. 3, 2015 12:45 AM EST Reads: 458