OpsLevel Logo
Product
Developer portal
Software catalog
Understand your entire architecture at a glance
Standards
Your guide to safe, reliable software
Developer self-service
Empower developers to move faster, risk-free
Integrations
Connect your most powerful third-party tools
Use Cases
Ownership
Build accountability and clarity into your catalog
Standardization
Set and rollout best practices for your software
Developer Experience
Free up your team to focus on high-impact work
Customers
Resources
All Resources
Our full library of resources
Pricing
Flexible and designed for your unique needs
Podcast
Podcast
Conversations with technical leaders
Blog
Blog
DevOps resources, tips, and best practices
Demo
Demo
Videos of our product and features
Tech talk
Tech talk
Our POV on technical topics
Guide
Guide
Practical resources to roll out new programs and features
DocsLog In
Talk to usTry for free
No items found.
Share this
Table of contents
 
Resources
Blog

DogOps: The Inventory

Product
Integration
Visibility
Automation
Platform engineer
Catalog
Service Detection
Life at OpsLevel
DogOps: The Inventory
Kyle Rockman
|
September 13, 2022
DogOps: The Inventory

As we mentioned in the first post of this blog series for #DogOps, when registering services to Circular, your initial task is to build your service catalog.  After you have inventoried your services and their metadata, you can begin to locate any existing holes and areas for improvement.

Today we'll cover the different ways in which we register services to Circular, and the reasons why we choose these methods.

By Hand

We registered our first service in Circular by hand using the Rails App. We filled out all the applicable metadata by hand.  That metadata includes details like: the language and framework used, the responsible team (in this case, engineering), the aliases we use in our systems, and some quick links to tools like our Datadog dashboard and PagerDuty rotation.

When starting out with OpsLevel, we recommend registering your first few services by hand, in order to familiarize yourself with the possible metadata.

I can already hear you screaming, "WHY WOULD YOU DO THAT??!!!"

In our opinion, by-hand registration encourages you to consider the available locations for collecting data from within your existing infrastructure and tools. As your understanding of OpsLevel grows, you will more easily understand how our automated collection methods can map data to a service.

Often, customers who use Kubernetes will jump right into the Kubernetes sync. They end up having a hard time because their resources don't have the necessary metadata to derive an owner or lifecycle for a service. Instead, they might have been better served using our CLI and a bash for loop.

opslevel.yml

As our company grew, so did our number of engineering teams. We had a larger amount of things we wanted to register in Circular and track.

In our last post, we listed the various things we register as a service to Circular. The main reason driving the increase was a need for tracking ownership and maturity of each thing individually.

We choose to register things that do not have a long-lived process component via opslevel.yml. For example, our CLI and our Terraform provider will never run in our infrastructure, but we still want to track and improve them. Therefore, they count as services. We will delve into more detail about how we do this in a future post focused on the maturity rubric and campaigns.

We also treat the “main” branch as the current source of truth and cut our releases from that branch when we are ready. This means that we can easily change any of the metadata with a simple pull request, which gives us git history on the changes to that metadata.

Kubernetes Sync

At OpsLevel we run almost everything in Kubernetes clusters. Those clusters include applications like Nginx ingress controller, cert-manager, the rails application and sidekiq workers.

We use Flux - the gitops toolkit - to apply our Kubernetes manifests to all our different clusters. This process leaves us with the question: should we just drop an opslevel.yml file in there too? Or is there a better way?

In this case, we opted to use our Kubernetes sync for registering and reconciling deployment resources that appear in our cluster. This is beneficial in that when someone goes around the gitops flow and manually creates a deployment in one of our clusters, it shows up in Circular. A regular audit of "unowned" services in OpsLevel now helps us head off the question of "what is this thing" 6 months later.

This process also helped us discover an observability problem within our product. How is one notified of new services being created? We took this feature request back to the product team as a user experience improvement.

There is a downside to using the Kubernetes sync. We have to populate the metadata that would normally be in the opslevel.yml directly onto the deployment resources as annotations. Inspecting the data works well when all you have is kubectl. But this process also moves the data out of the main services code repository into our flux repository, where all our Kubernetes manifests live. We deemed this trade-off to be acceptable for the time being and we might re-evaluate this in the future.

We are currently experimenting with using a hybrid approach where some fields like description, language and framework come from opslevel.yml and tags & tools come from Kubernetes. The side where this shines is when you deploy something you don't own the code repository for such as cert-manager. Now we can just adjust the annotations on the manifests we pulled from the cert-manager project.

In the end, the Kubernetes sync is a great way to bootstrap a large number of services, even if it often lacks the necessary metadata for a good service catalog. You can double down on Kubernetes annotations and even bake them into a template, so that all future services get automatically registered and all the necessary metadata is collected. It’s is a larger effort, but it pays off in the long run as you democratize the creation of new services.

Terraform

Lastly, we use Terraform to provision our Kubernetes clusters. We have recently taken to using our Terraform provider to register each cluster as a service, so we can track relevant metadata and PagerDuty alerts about our clusters health.

In the long term, we envision having many more Kubernetes clusters across different regions. The prospect of these additional clusters triggered us to begin leveraging the Terraform provider in a preemptive step to set us up for success as we scale and build a global customer base that requires many more Kubernetes cluster.

In our case, Terraform was the best place to get and set the metadata for each cluster, and our Terraform provider easily facilitated this process. Because we use a Terraform module for Kubernetes cluster creation, any time we spin out a new cluster it is automatically registered correctly with all the metadata.

Bonus - Tags

As you can see, we use a variety of methods to register services. As such, allow us to make one key recommendation for long-term success as your catalog grows. Use tags! Ensure that you use a standard tag key to hold a value which denotes the importance of the service. That way, in the future, you'll know exactly where to look to change any of the metadata.

If you saw this tag on a service you would instantly know that to change the lifecycle field I need to go dig around in the Terraform configuration. This learning did lead us to give product feedback that this kind of functionality be built into the productive natively so keep your eyes peeled for this small feature popping up in the near future.

Until Next Time

We look forward to continuing this series and we hope you enjoy and find the content useful as you continue your service maturity journey at your company.

More resources

Blog
September 19, 2023
by
Fernando Villalba
The OpsLevel Developer Experience (DevEx) series. Part 1: What is DevEx?

Great developer experience (DevEx) is what you get when developers can easily achieve and maintain flow state at work. This article begins a series where we tackle all of the areas that affect flow state and impair your developer experience at your company and provide example metrics and suggestion to help you operate like a potential future unicorn.

Blog
August 31, 2023
by
OpsLevel
August 2023 release notes

This month included an update to our Service Maturity features—to give you even more flexibility—plus more sorting and syncing improvements. Read on to learn more!

Blog
May 31, 2023
by
Haley Hnatiw
May 2023 release notes

See what we’ve shipped in the month of May.

OpsLevel Logo
Subscribe
Join our newsletter to stay up to date on features and releases.
By subscribing you agree to with our Privacy Policy and provide consent to receive updates from our company.
SOC 2AICPA SOC
Product
Software CatalogMaturityIntegrationsSelf-serviceRequest a demo
Company
About usCareersContact UsCustomersPartnersSecurity
Resources
Docs
Blog
Demo
© 1999 J/K Labs Inc. All rights reserved.
Cookie Preferences
Terms of Use
Privacy Policy
Responsible Disclosure
By using this website, you agree to the storing of cookies on your device to enhance site navigation, analyze site usage, and assist in our marketing efforts. View our Data Processing Agreement for more information.
Okay!