NVIDIA and CrowdStrike collaborated to evaluate an agentic cybersecurity system. This system integrates Nemotron open models, Falcon telemetry, and specialized agent harnesses. The evaluation focused on an offensive-defensive agentic system operating within an isolated environment modeled on NVIDIA accelerated computing infrastructure.
Key Points
- The defensive harness incorporated six mechanisms: a schema knowledge base, telemetry grounding, specialized detection authoring with a customized Nemotron 3 Super, artifact linting, detection replay, and independent review.
- In the evaluated configuration, Nemotron 3 Ultra handled defensive orchestration, while a post-trained Nemotron 3 Super served as a bounded expert for detection generation and repair.
- Training for the Nemotron 3 Super used NVIDIA NeMo Gym and NVIDIA NeMo RL for reinforcement learning with verifiable rewards.
- Backtesting showed the optimized open-model pipeline achieved a 41.9% mean detection rate across six sessions, representing a 2.5x improvement over the default harness with Nemotron 3 Ultra alone.
- Live-fire testing against eight unseen attacks demonstrated that 45% of open-model detections generalized, compared to 29% for the frontier system.
- Three open detections achieved gold status and covered all eight attacks, while no frontier detections reached gold status.
Context
According to the NVIDIA Developer Blog, the evaluation aimed to address the challenge of identifying defensive gaps and converting them into reliable coverage. This requires repeatedly testing adaptive attacks and validating candidate detections against normal enterprise activity. The continuous offense-defense testing creates a feedback loop, where controlled attacks produce telemetry and ground truth for defensive agents to improve coverage.
The system operates as a closed loop, where each cycle strengthens defensive coverage while forcing the offense to find harder paths. The evaluation used attack paths and observable milestones to measure progress from action traces and sensor telemetry, rather than relying on the agent's self-reported claims. NVIDIA security experts reviewed the environment and threat paths for realism, and the same reviewed environment supported every attack run, detection test, and evaluation metric for consistent comparisons.
Why It Matters
This evaluation demonstrates a method for automating and accelerating the continuous testing and improvement of cybersecurity defenses. For builders, it highlights the potential of agentic systems and open models to identify and address security vulnerabilities more efficiently than traditional manual processes, potentially leading to more robust and adaptive security postures.
What To Do
- Explore NVIDIA Nemotron 3 to customize and evaluate specialized agents for bounded domains.
- Review NeMo Megatron Bridge for model integration capabilities.
- Try NeMo Gym to validate and execute generated queries in realistic environments.
- Use NeMo RL to update models based on verifiable results through reinforcement learning.