When you’re building a digital surgery platform, performance isn’t a nice-to-have — it’s a safety requirement. Devices in operating rooms need to sync reliably, workflows need to complete within strict time windows, and the system needs to handle scale without degradation.
I was tasked with building an end-to-end performance testing framework from scratch. Here’s how it went.
The Problem
The existing performance testing setup had three critical gaps:
- No realistic workload — tests used synthetic data that didn’t represent actual device behavior. API calls were fired in isolation without the sequencing and dependencies that real workflows have.
- No concurrency — tests ran devices sequentially. In production, thousands of devices operate simultaneously, creating contention patterns that sequential tests can’t reveal.
- No observability — when a test failed or showed degradation, there was no way to pinpoint where the bottleneck was. Results were pass/fail with no latency breakdown.
The result was a performance testing system that gave false confidence. Tests passed, but nobody knew if the system would hold under real load.
The Solution
I rebuilt the framework around three core principles:
Realistic SQLite Workload Generation
Instead of firing raw API calls, the framework generates realistic SQLite databases that mirror actual device data. Each simulated device has a complete data profile — configuration, surgical records, sync history, and metadata. This means the system under test processes the same data shapes and volumes it sees in production.
Adaptive Concurrency with Parallel Device Workers
The framework runs parallel device workers that simulate concurrent device operations. Concurrency levels are configurable and can ramp up dynamically to stress specific subsystems. Each worker operates independently, maintaining its own state and sync cycle, just like real devices.
DB-Primary Polling with API Fallback
One of the biggest wins was rethinking how the framework checks workflow progress. Instead of polling the API for status updates (which itself creates load), the framework queries the database directly for state changes and falls back to API calls only when necessary. This reduced API calls during test execution by 95%, meaning the framework measures the system’s actual performance without inflating it with monitoring overhead.
What We Measure
Every test run produces detailed metrics:
- Per-stage latency distributions — p65, p95, and p99 latencies for each stage of the device workflow. This shows not just the average case, but the tail latencies that affect real users.
- Throughput — devices processed per minute at each concurrency level, with breakdown by workflow stage.
- Error rates — categorized by error type, stage, and concurrency level. Not just “5% errors” but “5% timeout errors at the sync stage when concurrency exceeds 500 devices.”
The Result
The framework scaled the platform’s validated capacity from 30 devices to 75,000+ devices. Key outcomes:
- Scale validation — we can now prove the system handles 75K+ concurrent devices with quantified latency guarantees.
- Automated reporting — every test run generates HTML, Excel, and JSON reports automatically. No manual data collection, no spreadsheet wrangling.
- Jenkins integration — the framework runs as part of the CI/CD pipeline, providing performance regression detection on every build.
The framework didn’t just test performance — it became the system of record for capacity planning decisions.
“The best performance test is one that actually simulates production. Everything else is guessing.”