In a previous job, I joined a team that was supposed to be introducing DevOps to the organisation.
It started out well -- we spent a few months hacking with Terraform, Docker, Vagrant, Kubernetes, and related technologies to implement an infrastructure-as-code approach -- automating the process of provisioning and decommissioning servers, and building a setup where development teams could deploy updates with a simple git push.
Unfortunately it all went downhill fairly rapidly. We ended up spending the majority of our time manually applying security patches to a bunch of snowflake servers that we'd lifted-and-shifted from another hosting provider to AWS, and fielding support requests from the development teams. Within a year, we were being told in no uncertain terms by our project manager that we were an operations team, not a development team.
It felt like a complete bait-and-switch. Within two years, I had left the organisation in question and moved on to a new job elsewhere doing actual development again. Last I heard, the entire team had been disbanded.
It sounds like the author of this article must have had a very similar experience. I wonder just how common it is. It seems that in many places, "DevOps" is all Ops and no Dev.
I was one of the biggest proponents of DevOps (even started and the DevOps orange county meetup) but ... after working at AWS for 5+ years, my entire perspective shifted.
While DevOps does work in some organizations, sometimes what you really need to do is operationalize your software development team. Don't get me wrong, system developers/engineers/operations are needed but for sometimes for a small four person start up, the best investment is improving the development team's operational posture (e.g. run books, alarming, monitoring).
I was a network engineer at my previous job, I was fed up with doing things manually so I learned to code better. Now I'm a full stack developer writing code for the network team of a different company.
One thing that has changed drastically over the last 15 years is the sharp divide between IT and development. It used to be that developers were clueless how to deploy their own apps, and IT was terrified of writing code preferring to do things manually, or purchase something prepackaged. Now I see developers that understand firewalls and load balancers, and IT folk that can script stuff out for themselves. I think we have the push behind DevOps to thank for that progress, and while there is still a long ways to go, I think it's a good thing overall.
I worked at a large internet company in the mid-late 00s, and the typical team composition was a handful of developers, a product manager, a designer (if you worked on customer facing UI), a project manager (for large enough projects), QAs, and a SRE.
Every team had dedicated engineers whose sole purpose was to place purchase orders for servers, set up and manage databases, deploy changes, monitor uptime and more. And this was standard practice across the industry, as soon as your company outgrew a shared host.
Even the thought of something like this would be absurd today. So I'd flip the "no one admits they don't to it" around – your company does DevOps already, whether you realize/admit it or not.
According to "The DevOps Handbook" DevOps is...
> the outcome of applying the most trusted principles from the domain of physical manufacturing and leadership to the IT value stream. DevOps relies on bodies of knowledge from Lean, Theory of Constraints, the Toyota Production System, resilience engineering, learning organizations, safety culture, human factors, and many others.
Anybody that's worked in an organization that "uses DevOps" or has "DevOps Engineers" knows that this simply isn't true in the vast majority of circumstances. Too often DevOps is just a new name for an ops team. "True" DevOps is all about aligning incentives, sharing responsibilities, and fostering collaboration between people who build and maintain systems whereas "DevOps" (with air quotes) is just a new spin on being a sysadmin with a greater emphasis on AWS and Terraform.
I agree the terms have become pointless. I have been thinking about opening an actual school for DevOps, or at least IT Ops. There is virtually no education for it, but it's a complex subject matter that requires years of experience. The SRE books are a peek under the covers but leave so much unsaid that it's really only useful at "doing operations", not digital transformation.
To properly build and run a high-performing organization, you need to understand at least 1) the entire history (up to today) of software development practice and production operations, 2) all of the competing requirements of product design, development, and operation, 3) architectural best practices, 4) the industry standards that apply to the work, and 5) tech product-aligned business management methods.
Basically you need a generalist who has 15+ years experience and has tried and failed at everything under the sun. I think of it like carpentry before there was a building code, and before the introduction of apprentices and licenses. Trying to build high-performance orgs without a master builder to show you is like trying to build skyscrapers after having built your own home (and having never seen a skyscraper). Or worse, taking an old barn (non-tech enterprise) and trying to build a skyscraper out of that. Without a trade school it will continue to be just a craft.
I feel like DevOps is a trap.
Most companies that have sufficient talent and organization/operational setup don't see a need for DevOps.
There seem to be a lot of other companies that are sucked in by its promises and think that it will solve their issues. Surprise... this didn't fix your shitty policies, disorganized/unempowered org structure, or cheap talent (cause you're unwilling to pay or train).
I assume there are a few places that can benefit from this. Maybe some medium to large tech companies with great procedures, documentation, and an empowered org structure. Especially if they're paying enough for people who can manage it. The other is probably for very small places wirh fairly simple tech where you can't afford entire teams of people so you empower someone to really own the system top to bottom (without ever calling it DevOps).
Recently I've been reading "When Agile Gets Physical" [1] and "Extreme Programming Explained":
1. "Extreme Programming Explained" explains the importance of separating values from principles and practices.
2. "When Agile Gets Physical" unpacks why using agile software best practices are horrible for hardware. This is "duh" if you're a hardware person, but the book goes back to agile/lean first principles, and then builds up an approach for agile hardware.
It got me thinking that we often conflate values and practices. Blinding adopting Scrum or DevOps without understanding some of the underlying value and principles leads to problems. If you understand the foundational principles, you're in a much better position to adopt/reject certain practices for your given situation.
In reality, most organizations just adopt a few agile practices, while rejecting the basic values.
[1] https://rapidlearningcycles.com/when-agile-gets-physical/
[2] https://www.amazon.com/Extreme-Programming-Explained-Embrace...
Not directly related to the question, but I personally hate the term "Sprint". Physiologically, a sprint involve by definition is a short-term high effort activity that can't be sustained without a break.
Does anyone use a different term that doesn't imply working oneself to death?
When I was a wee lad, “programmers” wrote software and sorta tested it and then kicked it over the fence to “systems administrators”, who chain-smoked and were very, very un-woke.
This was a bad system.
To the extent that “DevOps” means anything, it’s that the spheres of capability and responsibility need to have tons of overlap: SWE’s need to be in the oncall loop and know how the filesystem works, and SRE’s need to be serious coders.
That’s it: everyone codes, everyone operates. We all automate where feasible.
Sidebar: What’s with the crazy surge in BuzzFeed-style titles on the front page? It’s not a federal matter but it’s not a slight change YoY either.
My team at Nasdaq (I work on other products, not the stock exchange) does DevOps surprisingly well for not being a Real Tech Company TM. I don't know that the business sees benefits, as it meant that all of the devs on my team have had to learn Terraform, Helm, AWS, and Kubernetes in addition to the actual SpringBoot Java application code, so our feature velocity dropped quite a bit, but it was fun to learn, and it's probably better to have people who understand the full deployment and not just pass blame between teams during incidents.
I'm confused about DevOps - I thought it was about dev teams owning the infrastructure, testing and deployment pipelines, so that there is no clear delineation between they person who writes tests, the person who sets up CI/CD pipelines and the person who codes the app.
It has nothing to do with frequency of deployments, and it does not mean that no ops team is necessary for running the show, it's about team autonomy and better understanding of different teams' concerns because everyone is doing a bit of everything.
> If I were to generalize, I’d say most of the companies I’m picking on fall into one of two categories:
> They believe in DevOps ideas, but they’re unwilling to invest in changing their ways. They aren’t convinced that some of the scarier ideas (like deploying multiple times per day) will help after all.
Option number 3 is the biggest one for me: thinking that having a separate "DevOps" team from the Dev team that does ops via modern tools is DevOps. That may be a good approach for your company, but it ain't DevOps.
I think there is a third category - companies that have heard of DevOps, think it means that the development team and operations team talk to one another, which they already do (!) and shut down conversations after that.
I've come across this archetype in several contracting gigs and I find it to be especially pernicious as it gives the incumbents (who usually have more weight in the org) ammo to effectively do nothing.
The author is right that DevOps seems to often mean different things to different people. I would have loved it if they had taken more of an attempt at one definition than just that it's when developers have an understanding of how and where their code is deployed. That does sound sensible, but what other best practices does DevOps usually entail?
DevOps (and Agile) both usually run aground on the same question of trust. If you aren't willing to trust your delivery teams then you aren't going to see the leverage that both practices bring to an organization.
And if you aren't willing to trust your delivery teams then your organization has a much bigger problem than either DevOps or Agile can solve.
The problem with these fad methodologies is that it either works, or you are not doing it properly. The methodology can't be wrong.
It is like saying that C programs never crash, because if they crash, it means you are not using C properly. It is true, but no one will pretend that, everyone knows that under real life circumstances, C programs can crash, and there are plenty of other programming languages that address this issue, at the expense of other things, like performance or simplicity.
But for some reason those who push methodologies don't seem to consider that if that methodology is difficult to implement, or if it is not robust against things like conflicts of interest and office politics, or whatever, then maybe it is a problem in their methodology. It doesn't mean the methodology is bad, just that it doesn't work in every case, and that some tweaking may be necessary.
It kind of feels like devs have been duped for the past twenty or so years.
Want to save in project management? Just make devs responsible for planning and estimating, and call it "agile".
Want to remove sysadmins and ops? Make devs write their own configuration for the infrastructure, and say you do "devops".
Want to eliminate QA? Say that you do TDD.
> We need better definitions for these terms
When I point to the moon with my finger and say "moon", the fool looks at my finger and calls it the moon.
Is it more important to mindlessly tag yourself "Agile", with all the bells and whistles adorning the capital A, or would you rather just be agile according to the sought out goals and philosophy, and adapted to your specific situation?
Is it more important to have a clearer definition of "DevOps", so that you can more accurately implement all the prescribed workflows, or would you rather understand the pursued ideals, so that you can pick and choose what you need, according to your specific context, unencumbered by a slew of best, but ultimately useless, practices?
The entire article avoided talking about definitions of DevOps, while rambling a lot about how we have different ideas of what it means and how we need better definitions.
I appreciate the alert that there's a problem here, but there's no substance to the article.
DevOps has lost all meaning as the first paragraph in the article says. Anything it says after that is just semantics. Everyone is doing it on some level, but what that means to each company differs widely.
when John Willis explained it in 2016 devops had enormous potential for the initiated. id guess maybe 10% of anyone who says 'devops' does anything remotely close though.
devops in 2022 means if youre a dev, youre expected to know the bowels of the OS inside and out as well and be willing to spend hours debugging it alongside your normal programming work. if youre in ops? it means debugging someones undocumented middleware from nine years ago while the front of site burns so "rockstar" coders dont jump ship having to do it instead. if youre in dev or ops, it now means standups where you sit down for an hour and talk about where to put your story in jira or how to enter it at all, and where you discuss broad strategy too. instead of leaving after your status update, youre expected to stay. its just another meeting now.
and once boardmembers and leadership realized they could spare themselves the crucifixion of any real accountability after a breech, they turned devops into devSECops, meaning your infosec team sits in awkward silence as you explain why double defined nginx headers arent a feature and the same management team that made them part of the standups also unilaterally declared ssl3 can never be deprecated because customers.
I find the article somewhat ridiculous in that it takes the tone that everyone thinks the idea in question are good.
> They aren’t convinced that some of the scarier ideas (like deploying multiple times per day) will help after all
For some cases, rapid deployment might make sense (purely front end stuff). For a lot of cases, I'm firmly of the opinion that it does _not_ make sense. I've never worked on a project that deployed that often, and I don't feel the need to. I have depended on services that deployed that often and "oh, sorry we broke that, we'll fix it later; you'll just have to live with permanently broken data" gets my knickers in a twist every time.
> I don’t buy into all the ideas behind agile, particularly some of the Scrum-specific ceremonies.
That's because a lot of the Scrum people present it as "the" Agile. And a lot of the non-Scrum people are fairly convince that Scrum is awful. To which the Scrum people present "no true Scotsman" arguments.
I'm all for the underlying "listen to your customer, listen often, and change plans as often as necessary to make them happy" ideas of Agile. But every "methodology and process" for it I've seen completely loses that idea. It's infuriating.
> I think companies feel that if they aren’t going to “do DevOps”, that they need to rebrand the term “DevOps” to mean “what we happen to already do”.
> To me, the most egregious example of this is when a company simply changes the job titles of all its ops people. John use to be an “IT Operations” guy, abut now he’s a “DevOps Engineer”.
This is so true it hurts..
Agile is a philosophy, it is not sprints, story points or whatever crazy methodology an expensive Agile Coaching company has come up with. The same applies to DevOps. DevOps could just be a bash script, written by the developer, to feit their codebase. It doesn't have to be fancy or require an extensive amount of investment.
SRE seems the better approach. Make an error budget, measure reliability, stop releasing in favor of reliability engineering if the budget is missed, use the free time reduce toil. DevOps seems to degrade to disorganization because the incentives are not aligned between product development and software engineering.
Most SaaS are overcooked spaghetti floating on AWS alphabet soup. A devops mentality won't fix them.
I also go to the gym 4 times a week, go to sleep on a regular schedule and eat a balanced diet!
This is also very true in the presales engagements by top consulting firms under "Digital Transformation". These presales teams would come and show, on paper, how great they are with the agile practices, devops, microservices, you name it.
However, it's a totally opposite story during the actual implementations. I've witnessed that it took a top consulting firm 4 months to provision a sandbox environment while being 1 month away from the first release. When asked about their proud DevOps practices, you'd hear excuses that are in the blatant lie territory. They knew they could get by by managing the stakeholders instead of really doing what they had promised.
You go up at 5 AM in the morning. You ride your bike to a customer. You pass some security checks like, known guy, yes, no. You are shown to the servers in the cellar. Cause they are safe there. You get access to the admin account after 2 hours. You do some backup to a local disk. Somebody screams something but you can’t hear what because of the noise. You say yes and hopes it was about coffee even though the room is like a sauna. Now you pick up that CD and insert it. You are not allowed to use a USB-drive because it is mutable. Some clicks and the installation begin. And now you wait.
I am pro-devops.
While I generally agree about the annoyance with UI changes, and how that grinds on people, I hope we don't just handwave over the "CD for bug fixes is fine" bits.
It varies product-to-product obviously, but I'd guess there is also a significant chunk of folks that are pleased that their functional annoyances/blockers (bugs) can be fixed quickly, without waiting for a monthly or quarterly release.
These folks might not mind the fluctuating UIs, if it means they get the functional value they need asap.
I hate devops. Mainly because I don’t want to do it. I liked it better when there was a dedicated infrastructure team instead of pushing it down to developers and giving it a new name.
If you have a separate DevOps role in your org, arguably you're not doing DevOps. So probably most orgs aren't doing DevOps, at least those that are talking about it.
Devops means let a non-admin developer have access to aws and just buy everything he can find in order to try and fix his shortcomings.
It'd be better and cheaper to follow a sysop course
Corporate kayfabe. https://en.wikipedia.org/wiki/Kayfabe
I wonder how to use a devops flow for machine learning workloads. Something we are working on at my current job is how to balance the speed of releasing together with keeping data, pipeline and model versions aligned (where it for instance takes multiple days to release a new model).
Releasing new features which need a model retraining is currently a very jarring experience.
> We need better definitions for these terms
How about this? Let's describe levels of adoption. Think of the "normal form" concept for databases. Each level has a strict definition, and you can objectively observe which level you are using. And not every level is appropriate at all times.
So maybe a team could say, "we're at DevOps level 2," or, "we use level 3 Scrum."
> The ideas behind the DevOps movements undeniably changed the software development world for the better
I absolutely deny this.
I think one of the problems with the terms "Agile" and "DevOps" is that they are pretty general and generic terms that they are trying to cast very specific meaning too...
“So you’re saying you’re not an agile team?” Sure we are appropriately agile for our needs, no need to be rigid about it...
This is a content-free extremely meager blog post that doesn’t really belong on the front page of HN but the discussion in the comments is quite illuminating.
The discussion shows how a lot of people have wildly different ideas about what devops is and that in and of itself is interesting.
For me devops was never a great idea. Ideally you want one team to decide on the architecture. You do need front line ops, ideally they'd take some ownership and write some scripts, but the buck stops with one team for inhouse systems that is usually developers.
No one admits they don't do it because everyone does it. I have worked at 6 different companies since I've heard that term and all of them were able to deploy to an environment by clicking a button on a CI/CD tool. Maybe I'm the outlier?
The mistake was turning DevOps into a job position which resulted in silos being created more when the whole purpose was to remove/prevent silos and to allow developers to operate at their own pace.
All these processes are a template, a starting point. Begin there and optimise and refine to suit your team and your environment. Dogmatically following a process that doesn’t work for you is silly.
Just being devil’s advocate, whatever the profession is, if someone keeps changing things, how would it reflect on his or her maturity in getting things done?
I feel like devops became a marketing juggernaut then somehow evolved into creating a third team, apart from dev and ops, that does handoffs with both of them.
I only do as much 'ops' work as required by the idiots in charge. Its much more productive for me to be working on business solutions than screw around with getting traces and logs into datadog and figuring out all the permission issues in AWS. Someone can specialize in that knowledge instead of everybody needing to know everything.
This seems like an example of something similar to the no true scotsman fallacy. (it's inverse) These ideas are supposed to be aspirational, akin to asking "is it a true democracy" or "is it really capitalism". I don't entirely object to the notion of trying to be more precise, but gatekeeping "devops" or "agile" also seems to sort of miss the point.
"DevOps" is a corrupted term. It just means sysadmin at this point.
I see the point, but it’s funny that the author lays out the reason why this goalpost-moving is happening.
As pointed out, you sound like an idiot if you’re against Agile. So it’s a lot easier to get what you want if you just call it Agile.
I get that it’s not intellectually rigorous or whatever, but good grief, when has that ever mattered in business. Is there any class of concepts that survives unblemished after contact with reality?
Or, alternatively, it might not be a great idea in all cases.
Maybe your development team shouldn't develop terraform the four times a year they need it. It might even be faster to have two or three individuals with a singular vision for the system in a smaller team. (There may not be enough work for some teams to have someone with specific DevOps skills. However, that ends up in a partial-shared-service model where team members get called off for organizational concerns.)
Maybe you need a subject matter expert that will develop standards for teams and select a set of tools, setting those up so that teams can build off those for their needs. (This is why Ops are often shared services teams.)
Maybe it's not a great idea if the first team to encounter a problem picks the tool that works best for them, and the organization ends up with multiple tools because there was nobody with both the time and expertise dedicated to the analysis. (This is why Ops are often shared services teams.)
We need to break down barriers. We need to adopt valuable practices. However, the parts of DevOps that don't make it to the real world may just be because they don't solve the problems of the organization.
As a dev, the whole "rapid release" thing made a lot of sense to me. As a user, though, I've turned 180 degrees on it. I hate "rapid release" with a passion now. Which means I dislike Agile by consequence.
Rapid release means I can no longer rely on the software that engages in it. It's constantly shifting, with frequent changes in UI and workflow, and new and exciting bugs constantly appearing.
I miss the days when I could plan ahead for new versions, so I could time them to have the least possible disruption to my life.