Welcome!

@DevOpsSummit Authors: Elizabeth White, XebiaLabs Blog, Liz McMillan, Pat Romanski, Mehdi Daoudi

Related Topics: @DevOpsSummit, Linux Containers, Containers Expo Blog, Apache, FinTech Journal

@DevOpsSummit: Blog Post

DevOps and SQL Review By @Datical | @DevOpsSummit [#DevOps]

DevOps patterns are in the constant crusade to bring high-quality products to market faster

Automating SQL Review to Save Time and Money

I’ve spent the majority of my tech career in startups.  I love the fast pace, the opportunity to learn new things, and the sense of accomplishment that comes from bringing a successful new product to market.  I began my career in Quality Assurance.  In startups, you rarely enjoy the low ratio of Developers to QA Engineers that you might in a large enterprise.  As a QA engineer in a startup, your inbox is always much more full than your outbox. You are the last gate before the next release so you’re always under the microscope. In an early stage startup you are most likely also the “Customer Support” team, so when an issue is hit in production you become VERY popular.

As someone in that position, I always kept an eye out for the right tools to lighten my load without sacrificing any of my own personal quality standards for the work I was doing.  This is how I came across FindBugs about 10 years ago.  The first time I ran it and shared the output with the development engineers on my team they felt that the tool emitted more false positives or “nitpicky” patterns than true bugs.  But over time, as we tweaked and extended the checks performed to cover our specific needs and correlated the data from FindBugs with actual counts of bugs found in test and production, FindBugs became an integral part of our nightly and on-demand builds.  The reports were an excellent early indicator of potential issues and allowed developers to rectify misdeeds before we used up testing cycles or troubleshooting time in operations.  The developers on my team also started committing fewer and fewer infractions as the daily reminders they got from our build system helped them to change their bad habits into safer, better performing, more stable code. Release cycles shortened, product quality improved, and customer satisfaction rose proving that an ounce of prevention really is worth a pound of cure.

As Enterprise IT embraces agile development practices and adopts DevOps patterns in the constant crusade to bring high-quality products to market faster, DBAs are really starting to feel the pinch.  The description above of a QA Engineer in a software startup is apt.  With more frequent releases the DBA’s inbox of SQL scripts to write, review, modify or optimize is always more full than her outbox. The DBA is the last bastion of defense for data quality, data security, and data platform performance and is therefore under constant scrutiny. When there is a production outage, the DBA is among the first called to respond.

One of the most time consuming tasks for the Fortune 50 DBAs we work with is SQL review.  Some DBAs are allocating 70% of their time manually reviewing SQL scripts.  They are checking for the same things in SQL that tools like FindBugs are looking for in Java code: code patterns that indicate logical problems, security flaws, performance issues, and non-compliance to internally defined best practices or externally mandated regulations.

It’s clear that DBA’s need a tool that does for them what FindBugs did for my team a decade ago.  Static analysis for SQL is nothing new, but current offerings only go so far.  Typically, they evaluate the SQL statements with no contextual sensitivity. This omission severely limits the productivity and quality gains that can be achieved because so much of Database Lifecycle Management is being aware of Who is doing What, Where and When.  For example, an organization may allow privilege grants and INSERT statements in a TEST environment, but never allows such activity in an automated session in PROD. Any static analysis tool for SQL must take environmental parameters into consideration.

Also complicating matters is the nature of database ‘versioning.’  While your application is packaged, versioned and replaced wholesale from release to release, the database schema that supports your application is persistent and evolves over time.  What’s more, external compliance standards (SOX or PCI DSS for example) and internal audit requirements often dictate that incremental changes to the database be rigidly controlled and tracked in a well-defined process. This means the DBA must also confirm (through manual process and reviewing SQL for the appropriate comments) that the change can be traced to its cause and the application of the change can be traced through each environment.

The Datical DB Rules Engine was designed and implemented to satisfy the unique set of challenges posed by SQL review & static analysis.  Here are just a few of the reasons that Datical DB enables acceleration through static analysis safely and sanely.

  • Models Make for Powerful Evaluation – Datical DB abstracts the application schema into a strictly defined and validated object model. Authoring powerful rules is fast, straightforward and simple. Once they are written they are enforced every time a Forecast or Deploy is performed on any database in the lifecycle.
  • Environmentally Aware Change Validation - The model includes information about the client environment and various database instances in your applications lifecycle. Your rules can be written to allow maximum flexibility in early stage environments and maximum security in sensitive environments simultaneously.
  • Easily Confirm Internal & External Audit Requirements – In Datical DB, everything you need to remain in compliance with external and internal audit requirements is tied tightly to individual changes in the Data Model.  Manual review to confirm auditability of change is replaced with automated checks that are executed every time you (or your automation frameworks) Forecast or Deploy.
  • Automatically Validate What’s Important to YOU - Provides the capability to customize analysis to cover internal best practices like naming conventions, SQL DOs and DON’Ts, and object dependency management
  • Automate The Boring Stuff. Get Back To The Fun Stuff - Like many static analysis tools for code, Datical DB integrates into your build and deployment systems in a few mouse clicks. Now every time you build or promote an application, Rules validations are performed and a report is generated for dissemination throughout the organization. Your DBAs, having considerably reduced the time they spent with eyes on the screen reading SQL, are concentrating on more strategic projects and problems.
  • Better Coding Means Fewer Bugs - DBAs author rules and share them with development.  Development then has a codified repository of what is and is not acceptable in their organization to work against. Fewer bugs escaping DEV saves time and money.
  • Increasing Operations Involvement In Database Development – The Rules Engine is tightly integrated with Datical DB Forecast.  This feature allows you to simulate database change without actually altering the target database.  When DBAs share their Rules with Operations, Operations can run nightly Forecasts against STAGE or PROD to ensure that what’s currently in DEV or TEST will comply with the stricter validations performed downstream, once again finding problems earlier in the lifecycle when they are cheaper and easier to fix.

More Stories By Pete Pickerill

Pete Pickerill is Vice President of Products and Co-founder of Datical. Pete is a software industry veteran who has built his career in Austin’s technology sector. Prior to co-founding Datical, he was employee number one at Phurnace Software and helped lead the company to a high profile acquisition by BMC Software, Inc. Pete has spent the majority of his career in successful startups and the companies that acquired them including Loop One (acquired by NeoPost Solutions), WholeSecurity (acquired by Symantec, Inc.) and Phurnace Software.

@DevOpsSummit Stories
In IT, we sometimes coin terms for things before we know exactly what they are and how they’ll be used. The resulting terms may capture a common set of aspirations and goals – as “cloud” did broadly for on-demand, self-service, and flexible computing. But such a term can also lump together diverse and even competing practices, technologies, and priorities to the point where important distinctions are glossed over and lost.
Most companies are adopting or evaluating container technology - Docker in particular - to speed up application deployment, drive down cost, ease management and make application delivery more flexible overall. As with most new architectures, this dream takes a lot of work to become a reality. Even when you do get your application componentized enough and packaged properly, there are still challenges for DevOps teams to making the shift to continuous delivery and achieving that reduction in cost and increase in speed.
Enterprise architects are increasingly adopting multi-cloud strategies as they seek to utilize existing data center assets, leverage the advantages of cloud computing and avoid cloud vendor lock-in. This requires a globally aware traffic management strategy that can monitor infrastructure health across data centers and end-user experience globally, while responding to control changes and system specification at the speed of today’s DevOps teams. In his session at 20th Cloud Expo, Josh Gray, Chief Architect at Cedexis, covered strategies for orchestrating global traffic achieving the highest-quality end-user experience while spanning multiple clouds and data centers and reacting at the velocity of modern development teams.
"At the keynote this morning we spoke about the value proposition of Nutanix, of having a DevOps culture and a mindset, and the business outcomes of achieving agility and scale, which everybody here is trying to accomplish," noted Mark Lavi, DevOps Solution Architect at Nutanix, in this SYS-CON.tv interview at @DevOpsSummit at 20th Cloud Expo, held June 6-8, 2017, at the Javits Center in New York City, NY.
In his session at 20th Cloud Expo, Mike Johnston, an infrastructure engineer at Supergiant.io, discussed how to use Kubernetes to set up a SaaS infrastructure for your business. Mike Johnston is an infrastructure engineer at Supergiant.io with over 12 years of experience designing, deploying, and maintaining server and workstation infrastructure at all scales. He has experience with brick and mortar data centers as well as cloud providers like Digital Ocean, Amazon Web Services, and Rackspace. His expertise is in automating deployment, management, and problem resolution in these environments, allowing his teams to run large transactional applications with high availability and the speed the consumer demands.
DevOps is under attack because developers don’t want to mess with infrastructure. They will happily own their code into production, but want to use platforms instead of raw automation. That’s changing the landscape that we understand as DevOps with both architecture concepts (CloudNative) and process redefinition (SRE). Rob Hirschfeld’s recent work in Kubernetes operations has led to the conclusion that containers and related platforms have changed the way we should be thinking about DevOps and controlling infrastructure. The rise of Site Reliability Engineering (SRE) is part of that redefinition of operations vs development roles in organizations.
SYS-CON Events announced today that Massive Networks will exhibit at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Massive Networks mission is simple. To help your business operate seamlessly with fast, reliable, and secure internet and network solutions. Improve your customer's experience with outstanding connections to your cloud.
All organizations that did not originate this moment have a pre-existing culture as well as legacy technology and processes that can be more or less amenable to DevOps implementation. That organizational culture is influenced by the personalities and management styles of Executive Management, the wider culture in which the organization is situated, and the personalities of key team members at all levels of the organization. This culture and entrenched interests usually throw a wrench in the works because of misaligned incentives.
SYS-CON Events announced today that Datera, that offers a radically new data management architecture, has been named "Exhibitor" of SYS-CON's 21st International Cloud Expo ®, which will take place on Oct 31 - Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Datera is transforming the traditional datacenter model through modern cloud simplicity. The technology industry is at another major inflection point. The rise of mobile, the Internet of Things, data storage and Big Data are challenging the existing design patterns of the Data Center. The increase in complexity of managing legacy systems alongside new systems is beyond the ability of most IT departments. This leads to multiple tiers of storage and high economic costs, during a time in which IT is expected to do more with less.
Given the popularity of the containers, further investment in the telco/cable industry is needed to transition existing VM-based solutions to containerized cloud native deployments. The networking architecture of the solution isolates the network traffic into different network planes (e.g., management, control, and media). This naturally makes support for multiple interfaces in container orchestration engines an indispensable requirement.
As many know, the first generation of Cloud Management Platform (CMP) solutions were designed for managing virtual infrastructure (IaaS) and traditional applications. But that’s no longer enough to satisfy evolving and complex business requirements. In his session at 21st Cloud Expo, Scott Davis, Embotics CTO, will explore how next-generation CMPs ensure organizations can manage cloud-native and microservice-based application architectures, while also facilitating agile DevOps methodology. He will explain how automation, orchestration and governance are fundamental to managing today’s hybrid cloud environments and are critical for digital businesses to deliver services faster, with better user experience and higher quality, all while saving money.
SYS-CON Events announced today that CA Technologies has been named "Platinum Sponsor" of SYS-CON's 21st International Cloud Expo®, which will take place October 31-November 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. CA Technologies helps customers succeed in a future where every business - from apparel to energy - is being rewritten by software. From planning to development to management to security, CA creates software that fuels transformation for companies in the application economy. With CA software at the center of their IT strategy, organizations can leverage the technology that changes the way we live - from the data center to the mobile device. CA's software and solutions help customers thrive in the new application economy by delivering the means to deploy, monitor and secure their applications and infrastructure.
While some vendors scramble to create and sell you a fancy solution for monitoring your spanking new Amazon Lambdas, hear how you can do it on the cheap using just built-in Java APIs yourself. By exploiting a little-known fact that Lambdas aren’t exactly single-threaded, you can effectively identify hot spots in your serverless code. In his session at @DevOpsSummit at 21st Cloud Expo, Dave Martin, Product owner at CA Technologies, will give a live demonstration and code walkthrough, showing how to overcome the challenges of monitoring S3 and RDS. This presentation will provide an overview of necessary Amazon Lambda concepts and discus how to integrate the monitoring data with other tools.
SYS-CON Events announced today that CA Technologies has been named “Platinum Sponsor” of SYS-CON's 21st International Cloud Expo®, which will take place October 31-November 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. CA Technologies helps customers succeed in a future where every business – from apparel to energy – is being rewritten by software. From planning to development to management to security, CA creates software that fuels transformation for companies in the application economy.
Internet of @ThingsExpo, taking place October 31 - November 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA, is co-located with 21st Cloud Expo and will feature technical sessions from a rock star conference faculty and the leading industry players in the world. The Internet of Things (IoT) is the most profound change in personal and enterprise IT since the creation of the Worldwide Web more than 20 years ago. All major researchers estimate there will be tens of billions devices - computers, smartphones, tablets, and sensors - connected to the Internet by 2020. This number will continue to grow at a rapid pace for the next several decades. With major technology companies and startups seriously embracing IoT strategies, now is the perfect time to attend @ThingsExpo in Silicon Valley. Learn what is going on, contribute to the discussions, and ensure that your enterprise...
As more and more companies are making the shift from on-premises to public cloud, the standard approach to DevOps is evolving. From encryption, compliance and regulations like GDPR, security in the cloud has become a hot topic. Many DevOps-focused companies have hired dedicated staff to fulfill these requirements, often creating further siloes, complexity and cost. This session aims to highlight existing DevOps cultural approaches, tooling and how security can be wrapped in every facet of the build and release cycle and how to get sales and customer facing resources wrapped in.
SYS-CON Events announced today that Calligo has been named “Bronze Sponsor” of SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Calligo is an innovative cloud service provider offering mid-sized companies the highest levels of data privacy. Calligo offers unparalleled application performance guarantees, commercial flexibility and a personalized support service from its globally located cloud platforms. Through its four pillars of focus, Calligo delivers a platform that businesses can trust to deliver the high level of service and protection they expect and is lacking in many cloud offerings.
SYS-CON Events announced today that Elastifile will exhibit at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 - Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Elastifile Cloud File System (ECFS) is software-defined data infrastructure designed for seamless and efficient management of dynamic workloads across heterogeneous environments. Elastifile provides the architecture needed to optimize your hybrid cloud environment, by facilitating efficient data access across cloud and on-premises boundaries - with all the advantages of public IaaS everywhere.
As DevOps methodologies expand their reach across the enterprise, organizations face the daunting challenge of adapting related cloud strategies to ensure optimal alignment, from managing complexity to ensuring proper governance. How can culture, automation, legacy apps and even budget be reexamined to enable this ongoing shift within the modern software factory?
SYS-CON Events announced today that Cloudistics, an on-premises cloud computing company, has been named “Bronze Sponsor” of SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Launched in 2016, Cloudistics helps anyone bring the power of the cloud to the data center in an easy-to-use, on- premises cloud platform that automatically provides high performance resources for all types of applications: Docker, Splunk, Hadoop, Citrix® VDI, and many other high performance workloads. With no onsite controllers to install or maintain, it’s easy to scale across a large site or multiple locations – all from a single, centralized dashboard.
With Cloud Foundry you can easily deploy and use apps utilizing websocket technology, but not everybody realizes that scaling them out is not that trivial. In his session at 21st Cloud Expo, Roman Swoszowski, CTO and VP, Cloud Foundry Services, at Grape Up, will show you an example of how to deal with this issue. He will demonstrate a cloud-native Spring Boot app running in Cloud Foundry and communicating with clients over websocket protocol that can be easily scaled horizontally and coordinate communication between multiple instances by using an additional message broker.
@DevOpsSummit at Cloud Expo taking place Oct 31 - Nov 2, 2017, at the Santa Clara Convention Center, Santa Clara, CA, is co-located with the 21st International Cloud Expo and will feature technical sessions from a rock star conference faculty and the leading industry players in the world. The widespread success of cloud computing is driving the DevOps revolution in enterprise IT. Now as never before, development teams must communicate and collaborate in a dynamic, 24/7/365 environment. There is no time to wait for long development cycles that produce software that is obsolete at launch. DevOps may be disruptive, but it is essential.
SYS-CON Events announced today that Golden Gate University will exhibit at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Since 1901, non-profit Golden Gate University (GGU) has been helping adults achieve their professional goals by providing high quality, practice-based undergraduate and graduate educational programs in law, taxation, business and related professions. Many of its courses are taught by faculty actively working in their field of expertise, providing students with skills that can be applied immediately. The new MS in Business Analytics, like most of its programs, is available fully online or in-person in downtown SF.
SYS-CON Events announced today that Golden Gate University will exhibit at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Since 1901, non-profit Golden Gate University (GGU) has been helping adults achieve their professional goals by providing high quality, practice-based undergraduate and graduate educational programs in law, taxation, business and related professions. Many of its courses are taught by faculty actively working in their field of expertise, providing students with skills that can be applied immediately. The new MS in Business Analytics, like most of its programs, is available fully online or in-person in downtown SF.
DevOps at Cloud Expo, taking place October 31 - November 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA, is co-located with 21st Cloud Expo and will feature technical sessions from a rock star conference faculty and the leading industry players in the world. The widespread success of cloud computing is driving the DevOps revolution in enterprise IT. Now as never before, development teams must communicate and collaborate in a dynamic, 24/7/365 environment. There is no time to wait for long development cycles that produce software that is obsolete at launch. DevOps may be disruptive, but it is essential.