Three boring fixes made OpsLevel 40% faster

Over the last few weeks our team made OpsLevel a lot faster!

Requests to our GraphQL API are about 40% faster. Our check evaluation pipeline went from keeping ~9.8 Sidekiq threads busy around the clock to ~1.8.

There wasn't any major architectural change or rewrite in Rust (sorry DHH). The mundane truth is that we (with some help from AI) found work we didn't need to do, and we stopped doing it.

We ran bcrypt on almost every API request

This one is almost annoying.

Over 90% of OpsLevel's traffic comes from API clients, mostly integrations. For example, our Kubernetes integration regularly sends us cluster data from our customers' systems. Our check integrations regularly receive long outputs from security and linting tools running in CI.

Each of these API requests authenticates with a token, and we were verifying that token against a bcrypt digest.

When we first implemented API tokens in 2019, we picked bcrypt for ease. We had simply extended our existing user authentication mechanism that used bcrypt. The implementation was correct, we had other things to build, and we never looked at what it cost.

But bcrypt is slow. Deliberately.

For user passwords, that slowness is a feature. Humans pick guessable passwords, and if the hashes in our database ever leak, tools like hashcat can try millions of common words and variations per second against them. A slow hash makes every one of those guesses expensive to compute.

However, our API tokens are long random alphanumeric strings we generate ourselves. They have about 214 bits of entropy, so brute-forcing them isn't practical. The slowness of bcrypt added no real protection, but it did cost us ~62ms per request. What's worse, tokens created before October 2023 used an even higher bcrypt cost and took ~240ms (!!). Altogether bcrypt computation ate about 37% of our web CPU.

So for API auth, we switched to a cheaper hash of the token (HMAC-SHA256), keyed with a server-side pepper. We migrated tokens on first use with a one-time bcrypt fallback, so no customer had to rotate anything themselves.

And just like that, our GraphQL endpoint got about 40% faster overnight:

• p50: 150ms → 60ms
• Mean: 250ms → 150ms
• p90: 500ms → 300ms

It freed a lot of compute too. In the graph below, the light blue line is time spent in middleware before the GraphQL controller executes, which is basically bcrypt. It goes to near zero, and our baseline web CPU usage drops from around 20 cores to under 10 (about half).

Mixpanel was costing us 15%

The second fix was smaller, but still impactful.

We send product analytics to Mixpanel from both web requests and background jobs, and the mixpanel-ruby gem was adding more overhead than we thought.

In September we upgraded mixpanel-ruby from 2.3 to 3.3. The new version eagerly loads another gem, json-logic-rb, to support local feature flags. We don't use that feature, but the gem loaded in every Rails process anyway.

json-logic-rb uses Ruby refinements to redefine ==, <, >, <=, >= and ! on core classes like Integer, String, Array and Hash. Defining those refinements is enough for CRuby to assume its basic operators might have changed, so it turns off its fast paths for them across the whole process. Every comparison in our app got a little slower.

We stopped json-logic-rb from loading. In local benchmarks, a GraphQL query over 100 services ran 1.3x faster, and GraphQL introspection ran 2.3x faster. In production, Sidekiq jobs (which the bcrypt fix didn't touch) got about 15% faster across the board. The jobs all behave exactly the same way they did before.

We reported the issue upstream, and the json-logic-rb maintainer already has a fix in progress. Once it ships, we'll remove our workaround.

You can see the improvement in performance around the 18:00 mark.

Millions of checks on a fifth of the threads

OpsLevel runs millions of checks a day for our customers. A check verifies something about your software: a repo contains a required file or config stanza, infrastructure is set up the right way, service metadata is filled in correctly.

Checks are generally implemented as Sidekiq jobs and one common class of checks was decomposed into three workers: Kickoff, Loader and Evaluate. These fanned work out into lots of small jobs. That design repeated a lot of expensive work. One S3 payload could fan out and get fetched thousands of times. Every job compiled its own jq and Liquid expressions. Every check result was its own database write.

So we changed three things:

• One iterable worker now walks a whole batch, fetching the S3 payload and compiling jq and Liquid once per batch.
• check_results saves are batched.
• latest_check_results saves are batched.

The check pipeline used to keep ~9.8 Sidekiq threads busy on average. Now it's ~1.8.

How we found these

We're using agents more than ever at OpsLevel, across every part of how we build software.

Much of the conversation about AI coding is about writing code, but the bigger change for us has been in finding the work in the first place. None of these three fixes needed a clever algorithm. They needed someone to read a trace, follow the code path and ask why we were doing that at all, which used to mean an engineer picking one hunch and spending a day investigating it (sometimes for no return).

Agents have greatly lowered the cost of that investigative work. We can point them at a profile and the codebase and have them chase multiple leads at once. They're especially good at turning up latent performance and security issues that have been sitting in plain sight for years.

I'm reminded of ESR's maxim that "given enough eyeballs, all bugs are shallow".

Humans are still very much in the loop though. Our engineers decide which opportunities are worth pursuing and review every fix before it ships.

For example, Claude Code wrote the original bcrypt merge request, but Greptile (our AI code reviewer) caught a bug in the token backfill. A human (Evan) fixed it, and another human (Jason) tested and approved it before it went out.

All of this adds up to a faster, more efficient and more secure OpsLevel for our customers.

If you call our API from CI, your own scripts or your own agents, we hope you've noticed things are a bit snappier. A typical GraphQL call now comes back in about 60ms instead of ~150ms. We also now have plenty of extra headroom as you add more components and checks.

We'll keep pointing agents at OpsLevel and sharing what they find. If something still feels slow to you, let us know, because that's exactly where we want to look next.