OpsLevel Logo
Product

Visibility

Catalog

Keep an automated record of truth

Integrations

Unify your entire tech stack

AI Engine

Restoring knowledge & generating insight

Standards

Scorecards

Measure and improve software health

Campaigns

Action on cross-cutting initiatives with ease

Checks

Get actionable insights

Developer Autonomy

Service Templates

Spin up new services within guardrails

Self-service Actions

Empower devs to do more on their own

Knowledge Center

Tap into API & Tech Docs in one single place

Featured Resource

March Product Updates
March Product Updates
Read more
Use Cases

Use cases

Improve Standards

Set and rollout best practices for your software

Drive Ownership

Build accountability and clarity into your catalog

Developer Experience

Free up your team to focus on high-impact work

Featured Resource

The Ultimate Guide to Microservices Versioning Best Practices
The Ultimate Guide to Microservices Versioning Best Practices
Read more
Customers
Our customers

We support leading engineering teams to deliver high-quality software, faster.

More customers
Hudl
Hudl goes from Rookie to MVP with OpsLevel
Read more
Hudl
Keller Williams
Keller Williams’ software catalog becomes a vital source of truth
Read more
Keller Williams
Duolingo
How Duolingo automates service creation and maintenance to tackle more impactful infra work
Read more
Duolingo
Resources
Our resources

Explore our library of helpful resources and learn what your team can do with OpsLevel.

All resources

Resource types

Blog

Resources, tips, and the latest in engineering insights

Guide

Practical resources to roll out new programs and features

Demo

Videos of our product and features

Events

Live and on-demand conversations

Interactive Demo

See OpsLevel in action

Pricing

Flexible and designed for your unique needs

Docs
Log In
Book a demo
Log In
Book a demo
No items found.
Share this
Table of contents
 link
 
Resources
Blog

DogOps: The Inventory

Product
Integration
Visibility
Automation
Platform engineer
Catalog
Service Detection
Life at OpsLevel
DogOps: The Inventory
Kyle Rockman
|
September 13, 2022

As we mentioned in the first post of this blog series for #DogOps, when registering services to Circular, your initial task is to build your service catalog.  After you have inventoried your services and their metadata, you can begin to locate any existing holes and areas for improvement.

Today we'll cover the different ways in which we register services to Circular, and the reasons why we choose these methods.

By Hand

We registered our first service in Circular by hand using the Rails App. We filled out all the applicable metadata by hand.  That metadata includes details like: the language and framework used, the responsible team (in this case, engineering), the aliases we use in our systems, and some quick links to tools like our Datadog dashboard and PagerDuty rotation.

When starting out with OpsLevel, we recommend registering your first few services by hand, in order to familiarize yourself with the possible metadata.

I can already hear you screaming, "WHY WOULD YOU DO THAT??!!!"

In our opinion, by-hand registration encourages you to consider the available locations for collecting data from within your existing infrastructure and tools. As your understanding of OpsLevel grows, you will more easily understand how our automated collection methods can map data to a service.

Often, customers who use Kubernetes will jump right into the Kubernetes sync. They end up having a hard time because their resources don't have the necessary metadata to derive an owner or lifecycle for a service. Instead, they might have been better served using our CLI and a bash for loop.

opslevel.yml

As our company grew, so did our number of engineering teams. We had a larger amount of things we wanted to register in Circular and track.

In our last post, we listed the various things we register as a service to Circular. The main reason driving the increase was a need for tracking ownership and maturity of each thing individually.

We choose to register things that do not have a long-lived process component via opslevel.yml. For example, our CLI and our Terraform provider will never run in our infrastructure, but we still want to track and improve them. Therefore, they count as services. We will delve into more detail about how we do this in a future post focused on the maturity rubric and campaigns.

We also treat the “main” branch as the current source of truth and cut our releases from that branch when we are ready. This means that we can easily change any of the metadata with a simple pull request, which gives us git history on the changes to that metadata.

Kubernetes Sync

At OpsLevel we run almost everything in Kubernetes clusters. Those clusters include applications like Nginx ingress controller, cert-manager, the rails application and sidekiq workers.

We use Flux - the gitops toolkit - to apply our Kubernetes manifests to all our different clusters. This process leaves us with the question: should we just drop an opslevel.yml file in there too? Or is there a better way?

In this case, we opted to use our Kubernetes sync for registering and reconciling deployment resources that appear in our cluster. This is beneficial in that when someone goes around the gitops flow and manually creates a deployment in one of our clusters, it shows up in Circular. A regular audit of "unowned" services in OpsLevel now helps us head off the question of "what is this thing" 6 months later.

This process also helped us discover an observability problem within our product. How is one notified of new services being created? We took this feature request back to the product team as a user experience improvement.

There is a downside to using the Kubernetes sync. We have to populate the metadata that would normally be in the opslevel.yml directly onto the deployment resources as annotations. Inspecting the data works well when all you have is kubectl. But this process also moves the data out of the main services code repository into our flux repository, where all our Kubernetes manifests live. We deemed this trade-off to be acceptable for the time being and we might re-evaluate this in the future.

We are currently experimenting with using a hybrid approach where some fields like description, language and framework come from opslevel.yml and tags & tools come from Kubernetes. The side where this shines is when you deploy something you don't own the code repository for such as cert-manager. Now we can just adjust the annotations on the manifests we pulled from the cert-manager project.

In the end, the Kubernetes sync is a great way to bootstrap a large number of services, even if it often lacks the necessary metadata for a good service catalog. You can double down on Kubernetes annotations and even bake them into a template, so that all future services get automatically registered and all the necessary metadata is collected. It’s is a larger effort, but it pays off in the long run as you democratize the creation of new services.

Terraform

Lastly, we use Terraform to provision our Kubernetes clusters. We have recently taken to using our Terraform provider to register each cluster as a service, so we can track relevant metadata and PagerDuty alerts about our clusters health.

In the long term, we envision having many more Kubernetes clusters across different regions. The prospect of these additional clusters triggered us to begin leveraging the Terraform provider in a preemptive step to set us up for success as we scale and build a global customer base that requires many more Kubernetes cluster.

In our case, Terraform was the best place to get and set the metadata for each cluster, and our Terraform provider easily facilitated this process. Because we use a Terraform module for Kubernetes cluster creation, any time we spin out a new cluster it is automatically registered correctly with all the metadata.

Bonus - Tags

As you can see, we use a variety of methods to register services. As such, allow us to make one key recommendation for long-term success as your catalog grows. Use tags! Ensure that you use a standard tag key to hold a value which denotes the importance of the service. That way, in the future, you'll know exactly where to look to change any of the metadata.

If you saw this tag on a service you would instantly know that to change the lifecycle field I need to go dig around in the Terraform configuration. This learning did lead us to give product feedback that this kind of functionality be built into the productive natively so keep your eyes peeled for this small feature popping up in the near future.

Until Next Time

We look forward to continuing this series and we hope you enjoy and find the content useful as you continue your service maturity journey at your company.

More resources

March Product Updates
Blog
March Product Updates

Some of the big releases from the month of March.

Read more
How Generative AI Is Changing Software Development: Key Insights from the DORA Report
Blog
How Generative AI Is Changing Software Development: Key Insights from the DORA Report

Discover the key findings from the 2024 DORA Report on Generative AI in Software Development. Learn how OpsLevel’s AI-powered tools enhance productivity, improve code quality, and simplify documentation, while helping developers avoid common pitfalls of AI adoption.

Read more
Introducing OpsLevel AI: Finding Your Lost Engineering Knowledge
Blog
Introducing OpsLevel AI: Finding Your Lost Engineering Knowledge

Read more
Product
Software catalogMaturityIntegrationsSelf-serviceKnowledge CenterBook a meeting
Company
About usCareersContact usCustomersPartnersSecurity
Resources
DocsEventsBlogPricingDemoGuide to Internal Developer PortalsGuide to Production Readiness
Comparisons
OpsLevel vs BackstageOpsLevel vs CortexOpsLevel vs Atlassian CompassOpsLevel vs Port
Subscribe
Join our newsletter to stay up to date on features and releases.
By subscribing you agree to with our Privacy Policy and provide consent to receive updates from our company.
SOC 2AICPA SOC
© 2024 J/K Labs Inc. All rights reserved.
Terms of Use
Privacy Policy
Responsible Disclosure
By using this website, you agree to the storing of cookies on your device to enhance site navigation, analyze site usage, and assist in our marketing efforts. View our Data Processing Agreement for more information.
Okay!