The Part of the System I Worked On
I worked on Kubescape Kubevuln, an open-source tool that scans container images for security vulnerabilities (CVEs). Over a few weeks, I found and fixed 7 bugs — all in the same part of the system.
These weren't simple typo fixes. They were bugs that only show up after real, repeated use: when the same image is scanned again and again, when outside data comes from other tools, and when specific edge cases line up. Finding bugs like this needs more than reading code — it needs understanding how the whole system is supposed to behave, and noticing where it quietly breaks that promise.
When Kubevuln scans an image, it finds CVEs and needs to record a decision for each one: is this CVE dangerous, or has someone decided it's safe to ignore? This record is stored in a format called VEX (Vulnerability Exploitability eXchange). Think of VEX as a decision log: 'This CVE was checked, and here is why it's marked safe or not safe.'
A CVE can get marked 'safe to ignore' in two different ways:
- 1. A person creates a rule (called a SecurityException) saying 'ignore this CVE.'
- 2. An outside trusted source (like Red Hat or Debian advisories) already says 'this CVE doesn't apply here.'
The bugs I found all came from one repeated mistake in the code: it assumed every ignored CVE came from source #1, without actually checking. Once you see that pattern, you start finding it everywhere — and that's exactly what happened.
Bug 1 & Bug 2: Data Loss & False Safe Grouping
Bug 1: A Real, Unsolved CVE Was Being Hidden — A CVE marked 'under investigation' (meaning: not yet known if it's safe) was being treated the same as a CVE that was confirmed safe. It got hidden from scan results. Hiding an unresolved risk is worse than showing a false alarm. A security tool should never make an open question disappear. I traced the logic from CVE status to UI presentation and skipped grouping 'under investigation' with confirmed safe entries.
Bug 2: Ignored CVEs Disappeared Instead of Being Recorded — When someone ignored a CVE using a rule, the CVE vanished completely from the VEX report — no record it ever existed. In security, 'we chose to ignore this' and 'we forgot this exists' must never look the same. If someone audits the system later, they need proof a decision was made. I updated the system to retain the CVE in the report, marked explicitly as 'not affected, ignored on purpose, here's why.'
Bug 3 & Bug 4: Data Ownership & Untagged Records
Bug 3: The Tool Was Overwriting Outside Data It Didn't Own — Every time a scan ran, the code reset all VEX records — including ones that came from outside trusted sources, not from Kubevuln itself. So trusted external data was being silently destroyed and replaced. This is a data-ownership bug: the code had no way to tell 'data I created' apart from 'data someone else gave me.'
My fix: I gave every record Kubevuln creates a clear, unique tag (a namespaced ID). Now the reset logic only touches records with that tag — anything else is left untouched. I didn't build an over-engineered subsystem; I used an ID that the data already needed and made it self-describing.
Bug 4: A Follow-Up Bug the First Fix Didn't Catch — After fixing Bug 3, I found a related case: some outside records had no ID at all. The old code guessed 'no ID means it's probably ours,' which was wrong. I made the rule strict: something is only treated as 'ours' if it has our exact tag. No tag means 'leave it alone.'
The trade-off: A few legacy records would now be duplicated instead of updated. I chose this on purpose. A duplicate is easy to clean up later. Silently overwriting someone else's data is catastrophic. When in doubt, pick the recoverable failure mode.
Bug 5 & Bug 6: Inaccurate Reasons & Cached Scan Reversion
Bug 5: The Wrong Reason Was Being Shown for Ignored CVEs — Every ignored CVE showed the same message: 'ignored by a SecurityException' — even when that wasn't true. Some were ignored by outside data, not a rule someone created. I made the code inspect provenance and display the authentic justification.
Bug 6: Cached Scans Un-Ignoring CVEs That Should Stay Ignored — If the same image was scanned twice, a CVE that was correctly ignored (from outside data) would sometimes come back as 'active' again on the second scan — because the code mistook it for a different kind of record. I tightened the validation to only treat records as exceptions when they contain exact exception markers.
Being honest about the fix's limits: The true root cause of this bug lived in a separate upstream repository. Rather than hiding this, I clearly documented the boundary in the PR, shipped the safe partial fix for immediate protection, and provided the upstream roadmap for full resolution.
Bug 7: Inconsistent Fingerprints From the Same Data (Non-Deterministic Hashing)
What was wrong: The system builds a fingerprint (a SHA-256 hash) for each VEX document to check if anything changed. But the exact same document was producing different fingerprints across runs.
I uncovered four separate causes of non-determinism:
- 1. Go map iteration order is randomized by design in the Go runtime.
- 2. Two fields were concatenated with no delimiter, allowing collisions (e.g., 'CVE-1' + '23' vs 'CVE-12' + '3').
- 3. Sort comparisons lacked tie-breaker criteria for identical primary keys, leaving order undefined.
- 4. The sort function mutated slice elements in place, creating subtle side-effects in caller routines.
How I fixed each one: Sorted map keys deterministically; added unambiguous boundary delimiters; introduced composite multi-key tie-breakers; and executed sorting exclusively on defensive slice copies.
Performance optimization: Full deep comparisons are expensive. I added a fast-path prefix comparison: most CVEs have distinct names and resolve in O(1), reserving multi-field tie-breaking for true collisions.
What This Project Shows & Senior Engineering Decision Framework
Looking back at all 7 fixes together, they share one theme: the code was making assumptions instead of checking facts — assuming who owns data, assuming things are always ordered the same way, assuming a missing value means one thing when it could mean another.
Finding and fixing these bugs needed root-cause thinking, careful trade-off decisions, rigorous test discipline, and transparent communication with maintainers.
