How We Scan Our Entire Infrastructure for CVEs
Building a unified vulnerability monitoring system across Kubernetes, virtual machines, and containers
One of the challenges of managing infrastructure is answering a seemingly simple question:
“Is any software running in our infrastructure vulnerable?”
The question sounds straightforward, but answering it becomes complicated when your infrastructure consists of multiple Kubernetes clusters, virtual machines, operating system packages, and standalone containers.
Traditional vulnerability scanners often generate thousands of findings, many of which are related to packages that are installed but never actually used. At the same time, vulnerabilities in running workloads can go unnoticed if images are only scanned during deployment.
We wanted a system that would continuously monitor our infrastructure, prioritize vulnerabilities based on real-world risk, and clearly distinguish between issues we can fix ourselves and those that depend on upstream vendors.
Instead of introducing an entirely new security platform, we built a unified vulnerability monitoring solution using open-source tools already present in our infrastructure.
Here is how it works.
1. Scanning Kubernetes Workloads with Trivy Operator
For our Kubernetes environments, we use Trivy Operator, deployed independently in each cluster.
Trivy Operator continuously discovers workloads and scans their container images for known vulnerabilities.
An important part of our configuration is daily rescanning.
A container image that was considered secure yesterday might contain a known vulnerability today, even if the image itself hasn't changed. New CVEs are published continuously, and vulnerability databases are updated accordingly.
By rescanning running workloads every day, we can detect newly disclosed vulnerabilities without waiting for a new deployment.
To avoid maintaining separate vulnerability databases in every scanning job, we use a shared Trivy Server.
The scanning jobs communicate with this server, which maintains the vulnerability database and performs the analysis using the image data provided by the clients.
This architecture centralizes database management and reduces unnecessary duplication across our clusters.
2. Scanning Virtual Machines and Podman Containers
Kubernetes is only part of our infrastructure.
We also operate traditional Linux servers and virtual machines, some of which run services such as Vault, Grafana, and Prometheus inside Podman containers.
To cover these environments, we deploy a daily Trivy scanning job on every host, scheduled through a systemd timer.
Each execution scans two areas:
- Operating system packages installed on the host.
- Container images associated with currently running Podman containers.
However, scanning every installed package creates a significant amount of noise.
For example, a Linux server might have hundreds of installed packages that are never used by any running service. Reporting vulnerabilities in all of them can make it difficult to identify the issues that actually matter.
To address this, we apply an additional filtering step to host package findings.
We retain vulnerabilities only when the affected package meets at least one of these conditions:
- It is loaded or used by a currently running process.
- It belongs to the currently running Linux kernel.
- It provides files with the setuid permission.
Everything else is excluded from our operational vulnerability reports.
This approach significantly reduces unnecessary alerts and helps us focus on software that is actively involved in the operation of our infrastructure.
There is an important trade-off: runtime-based filtering is a prioritization strategy, not a guarantee that unused packages are harmless. Installed software may still become relevant later, and certain vulnerabilities may be exploitable without a package appearing in the current process inventory.
For that reason, we treat this filtered view as our operational risk dashboard rather than a complete inventory of every potentially vulnerable installed component.
3. Prioritizing Real-World Threats with CISA KEV
Not all CVEs represent the same level of risk.
A vulnerability with a Critical severity rating is important, but a vulnerability that attackers are already exploiting in the wild may require even more urgent attention.
To make this distinction, we integrate the CISA Known Exploited Vulnerabilities (KEV) Catalog into our monitoring pipeline.
Our system refreshes the KEV feed every six hours.
This allows us to identify CVEs with documented evidence of active exploitation and prioritize them accordingly.
Rather than relying exclusively on CVSS severity scores, we combine vulnerability severity with exploitation intelligence.
This gives us a more practical basis for deciding what needs to be fixed first.
4. Centralizing Vulnerability Metrics in Prometheus
One of our main architectural goals was to avoid building separate monitoring systems for Kubernetes and traditional Linux hosts.
We already use Prometheus extensively, so we decided to make it the central collection point for vulnerability information.
Trivy Operator exposes vulnerability-related metrics for Kubernetes workloads.
For Linux hosts, our daily scanning jobs convert their results into Prometheus metrics and publish them through the node exporter textfile collector.
We normalize these metrics to follow the same structure and labeling conventions as the Kubernetes vulnerability metrics.
As a result, Prometheus can process findings from both environments using a common set of queries and alerting rules.
This gives us a unified view across:
- Kubernetes workloads
- Virtual machines and physical Linux hosts
- Podman containers
- Operating system packages
The main benefit is simplicity.
Instead of maintaining different alerting logic for each environment, we have one consistent vulnerability monitoring pipeline.
5. Actionable Alerts Instead of Endless CVE Notifications
Finding vulnerabilities is only useful if the findings lead to action.
A common problem with vulnerability management systems is that they generate large numbers of alerts without clearly indicating who should act or how urgently the issue needs to be resolved.
We wanted our alerts to answer three questions:
How urgent is this vulnerability? Can we fix it ourselves? What is the expected remediation deadline?
We use Prometheus alerting rules and Alertmanager to classify and route findings according to our remediation policy.
| Vulnerability category | Remediation target |
|---|---|
| Known exploited vulnerability (CISA KEV) | Within 48 hours |
| Critical vulnerability with an available fix | Within 7 days |
| High-severity vulnerability | Within 30 days |
| Vulnerability in a third-party image | Separate upstream warning |
These deadlines represent our internal operational targets rather than universal security standards.
The distinction between internally maintained and third-party images is particularly important.
When a vulnerability affects an image we build ourselves, we can usually update the base image, rebuild the application, and deploy a patched version.
However, when a vulnerability affects a third-party image, the remediation process may depend on the vendor publishing an updated release.
We classify these findings separately as upstream warnings.
This helps prevent vendor-dependent issues from being mixed with vulnerabilities that our own engineering team can immediately resolve.
Of course, an upstream dependency does not eliminate the risk. For serious issues, mitigation may still require configuration changes, isolation, disabling affected functionality, or temporarily replacing the component.
6. Monitoring the Vulnerability Scanners Themselves
One of the most important lessons in building this system was that a missing vulnerability alert does not necessarily mean the infrastructure is secure.
It might simply mean the scanner stopped working.
A vulnerability monitoring system needs to monitor its own health.
We therefore added alerting for several failure scenarios:
- A scheduled scan fails.
- Scan results become stale.
- A running image has never been scanned.
- Expected vulnerability metrics stop appearing.
These checks are essential because a failed scanner can otherwise create a false sense of security.
For example, if a host stops reporting scan results, its last known vulnerability count might remain unchanged or disappear from monitoring entirely.
Without freshness and coverage checks, this could incorrectly appear as a healthy system.
We explicitly distinguish between “no vulnerabilities detected” and “no valid scan results available.”
That distinction is fundamental to trustworthy security monitoring.
7. Automating Vulnerability Remediation
Detection is only half of vulnerability management.
The other half is reducing the time between discovering a vulnerability and deploying a fix.
Where possible, we automate that process.
Operating System Updates
Our Linux hosts run a weekly maintenance job using:
apt full-upgrade
This keeps operating system packages updated and automatically applies available package upgrades.
Depending on the update, additional actions such as service restarts or system reboots may still be necessary.
Container Image Rebuilds
For database container images that we maintain ourselves, we use CI pipelines to automate the rebuild process.
A rebuild can be triggered when:
- The underlying base image changes.
- A fix becomes available for a vulnerability affecting the image.
The pipeline builds an updated image and opens a Pull Request for review.
This allows us to automate repetitive maintenance tasks while keeping changes visible and reviewable before deployment.
It also creates a clear connection between vulnerability detection and remediation.
8. What This Architecture Gives Us
The biggest improvement is not simply that we can detect more CVEs.
It is that we can make better decisions about the vulnerabilities we detect.
By combining Trivy Operator, host-level scanning, CISA KEV, Prometheus, Alertmanager, and CI automation, we have built a consistent process for vulnerability management across different infrastructure environments.
The system provides several practical benefits.
- Continuous visibility: Running Kubernetes workloads and host environments are scanned regularly, even when the deployed software hasn't changed.
- Reduced alert noise: Host package findings are filtered according to runtime relevance, helping us focus on the most actionable vulnerabilities.
- Risk-based prioritization: Known exploited vulnerabilities receive higher priority than findings based solely on severity.
- Unified monitoring: Kubernetes and non-Kubernetes environments use the same metrics and alerting infrastructure.
- Scanner reliability: Failed, stale, or missing scans generate alerts rather than silently disappearing.
- Clear remediation ownership: Vulnerabilities we can fix ourselves are distinguished from issues that depend on upstream vendors.
- Automation: Routine package upgrades and container image rebuilds reduce manual effort and shorten the remediation process.
Final Thoughts
Vulnerability management is not just about finding as many CVEs as possible.
A scanner that reports thousands of vulnerabilities without meaningful prioritization can easily overwhelm an engineering team.
Our goal was to build something more practical: a system that continuously monitors the software we operate, identifies vulnerabilities that deserve attention, and helps us understand what action to take.
We also learned that vulnerability monitoring needs to account for its own reliability. An incomplete scan, an outdated report, or a missing workload can be just as important to investigate as an actual CVE finding.
There is no single scanner that can guarantee complete security coverage, and runtime-focused filtering does not replace broader vulnerability assessments.
But combining continuous scanning, exploitation intelligence, actionable alerting, and automated remediation gives us a much clearer picture of the risks across our infrastructure.
Ultimately, the goal is not to collect more vulnerability reports. It is to identify meaningful risks, understand who can fix them, and reduce the time they remain exposed.