Agentic Load Testing on K8s: Gap and Build

Why Now

Load testing has always had the same problem: the engines are good at generating load and leave everything around it to you. Someone has to write the scenario, someone has to decide what “normal” looks like, and someone has to stare at a latency chart and work out why p95 tripled halfway through the ramp. The German Testing Board’s 2024 survey found a quarter of respondents do not automate load and performance testing at all, and a third want more training in it.

Two things changed in 2025-2026. LLMs got good enough at reading an OpenAPI spec to draft a k6 scenario a human can review in minutes, and tool-calling agents got reliable enough to run a bounded investigation loop: pull summary stats, slice the timeseries, cluster errors, check Prometheus, and write down a ranked hypothesis with evidence. The incumbents noticed. Grafana shipped natural-language k6 script authoring in May 2026, Gatling added an AI run summary in January 2026, and Tricentis launched agentic analysis for NeoLoad in March 2026. Meanwhile the traffic being tested is changing: Imperva reports automated traffic passed human traffic in 2024, and Wikimedia traced a 50% bandwidth jump to AI scrapers.

The Gap

The market map shows where the AI work is going: into each vendor’s own engine and cloud. Grafana’s assistant writes k6 scripts, Gatling’s summarizes Gatling runs, NeoLoad’s analysis lives inside NeoLoad. What nobody offers is the analysis step as an open-source, vendor-independent layer that you can point at your existing k6 results, run fully offline with a local model, and trust because the detection itself is deterministic code.

That is the wedge for agentic load testing as an independent product: keep the industry-standard engine, add an agent that drafts the test and explains the result in a form like “throughput flattened while latency doubled at the knee, errors cluster on connection timeouts, check the pool size before adding nodes”. It does not replace k6 or your thresholds. It replaces the hour of squinting at dashboards that teams skip, and it does it without moving you onto a vendor’s platform.

Reference Architecture

Shamal v0.1.0 is the open-source reference build for this teardown. Shamal v0.1 supports k6 as its only engine, behind an adapter interface meant for Locust or Gatling later. Today it is a CLI that runs anywhere k6 runs: a laptop, a CI runner, or a Pod. The Kubernetes architecture below is how we would run it in a cluster, with the scale-out part provided by Grafana’s open-source k6-operator, not by Shamal itself. Distributed load workers are an explicit non-goal of Shamal v1.

The CI gate from the Shamal repo shows the most important design choice: the LLM is never in the pass/fail path. shamal run exits on k6 thresholds alone, and investigation runs afterwards under if: always():

- name: Run the committed scenario
  run: shamal run ./perf/checkout.k6.js --results results.json

- name: Investigate and report (runs even when the gate fails)
  if: always()
  env:
    SHAMAL_MODEL: claude-sonnet-5
    ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
  run: |
    shamal investigate --results results.json
    shamal report --results results.json

How We Would Take It to Market

The buyer is not a performance engineer. It is the platform or backend lead at a 20-200 person SaaS company who knows they should load test before a launch and never has the time. They already have k6 or nothing, and they will not adopt a new engine.

That points to three packaging options for a startup built on this pattern:

• Open-core CLI plus a hosted investigation history: the CLI stays free, teams pay for trend storage across runs, comparison between releases, and shared reports. This is the most common devtools model and the easiest to start.
• CI marketplace app: a GitHub or GitLab app that comments on PRs with the investigation, priced per repository or per seat.
• Pre-launch readiness service: a fixed-scope engagement where the tool does the heavy lifting and an engineer signs off. Lower scale, but it pays from day one and produces the case studies the product needs.

The first ten customers are most likely in places where the pain is visible: teams posting “why does my API fall over at N users” in r/devops and r/kubernetes, k6 users who already have scripts but no analysis, and companies preparing a launch or a Black Friday.

What We Learned Building It

A few decisions from building Shamal v0.1.0 that apply to anyone building in this space:

• Keep generated tests as plain engine code. Shamal’s plan output is ordinary k6 JavaScript with a provenance header. Delete Shamal and the tests still run. Lock-in is the first objection any team raises about an AI layer over their tests, and this answers it in one sentence.
• Constrain the generator. The LLM fills journeys, data, ramp stages and thresholds into a fixed scaffold rather than writing freeform code. That bounds hallucination and guarantees the output parses.
• Make the knee detection deterministic, and let the model only interpret it. Shamal’s saturation check is plain code: take the median p95 of the first 20% of the run as a baseline, then look for the first point where p95 is at least 2x that baseline while virtual users grew 15% or more over the last three samples and requests per second grew 10% or less. It needs at least 8 data points and returns an evidence window, never a cause. The agent’s job is to explain that window, not to find it.
• Budget the agent. The investigation loop is capped at 15 model calls by default and must end with a structured finding: symptom, cited evidence windows, ranked hypotheses with high, medium or low confidence, and next steps. An unbounded agent is a cost line nobody can forecast.
• Air-gap is a feature, not an afterthought. Shamal supports local models through Ollama from day one, makes no telemetry calls, and only connects to the endpoints you configure. Regulated teams that will never send production metrics to a hosted API can still use the whole pipeline.

At v0.1.0 the repo has 122 automated tests passing. What it does not have yet is published benchmark data from real client systems, which is why this teardown shows no performance numbers. We will add them as build logs when we have results we can share.