What makes the challenge unique is that many companies slowly establish an internal process for automation and leverage some form of platform that they iterate over. At Keepit, this is virtually nonexistent. Instead, complex machine configuration needs to be standardized and automated, and the hurdles to get there are often bigger than they first seem. You typically need to consider:
- What previously had been changing on the machine?
- Who owns it?
- Where are the secrets kept?
- Are all the dependencies really being captured?
- Do I understand the security measures in place?
Keepit is an enterprise data-backup company. This means that security has always been a top priority. We own all our hardware. The entire infrastructure is isolated into datacenters, gateways, VLANs, and with firewall rules in between. All of this is being done inside of an air-gapped environment.
At another company, to begin using something like Terraform or CI/CD Docker jobs you might stand up a new service and get moving. A couple of days later, a prototype is ready to go. At Keepit, this is more challenging. You can’t just “move fast and break things”. You need to build each step that you’re used to simply having there already. In many cases you will need to be aware of:
- DHCP update process
- Firewall rules
- DNS configuration
- Kerberos
- LDAP
- …etc
Then, you must create the elementary pieces. The S3-compatible storage backend. The container registry that uses the S3-compatible backend. The automation definitions. You need to move content from the servers into Git so the changes are visible and can later be managed with more automation.
The real challenge
But the challenge doesn’t stop there. The most critical part of modernizing an organization’s operational maturity doesn’t come from the technical aspect at all. It comes from the culture of the people working there. They may feel challenged by these changes. They might not understand the end-goal. They may think this is going to break the process that took so long to get working in the first place.
Our engineering teams are used to deploying 1-2x per month to production. Everything, from building, testing, deploying, and releasing needs to change in order to improve that. But how do you go up against a process that has been crystallized by 10 years of stepwise movement to be able to do that? Our QA/Release teams have done the unthinkable: to make sense out of chaos. Because deploying manually to production on this scale even once a month is insanely hard. Do you really want to risk breaking that, and pissing off customers and coworkers?
The key to Keepit’s success so far has been to improve things which are already part of this brittle process. To educate companywide on what Elite DevOps looks like, and what each step for Keepit will look like to get there. What is the #1 priority?
Prioritization
And this is the final lesson learned at Keepit’s operational modernization journey. When there are 100000000 different CRITICAL things which all need to be improved at the same time, you need to have laser-focused prioritization on the most important tasks. Is it secret management software? Automation software? Is to split the monorepo and monolithic.
Deployment practices? Is it refactoring? Rearchitecting? Introducing HashiCorp Vault or Terraform? Updating scripts? Improving build times? The list goes on and on. And the only way to address the tech debt is to be aligned, as an organization, on exactly what the next most important steps are. Then, to put the blinders up, and solve that one problem correctly before moving on to the next.
Reporting
The DORA metrics are king when it comes to DevOps maturity. And reporting these numbers, as well as the initiatives aimed at improving those numbers. Small wins need to be celebrated, and the actual performance needs to be monitored. Without a scoreboard, how can anyone know when their team is winning?
Artykuł sponsorowany przez: https://www.keepit.com/