CALDERA, or Cyber Adversary Language and Decision Engine for Red Team Automation, is a web agent that runs on Chrome. Perhaps the most important distinguishing feature between CALDERA and the other two platforms we’re examining is that CALDERA is not agentless. To that point, 54ndc47 (that’s leetspeak for “Sandcat,” if you didn’t catch it) is the default agent that receives and executes instructions from CALDERA.
Simply put, you connect the machines you want to test to CALDERA by way of Sandcat. From there you can instruct CALDERA to run a variety of tests on the group of machines you’ve connected to it. Each individual ATT&CK technique in CALDERA is called an “ability,” and groups of abilities are called “adversaries.” In this way, CALDERA is a useful framework both for testing detection coverage and as an educational resource for defensive security professionals who want to learn more about threat detection.
CALDERA certainly offers the ability to run tests (like Atomic Red Team), but the platform is also a fully fledged testing suite that teams can customize to meet a long list of needs. For example, it allows users to natively map adversary emulation efforts back to MITRE ATT&CK—and can help facilitate many other types of adversarial research, for example:
- CALDERA supports other MITRE tools like Compass, which creates visualizations, like the ATT&CK coverage map I included earlier
- You can build your own agents instead of using Sandcat
- A somewhat outdated “Atomic tool” that you can use to merge Atomic Red Team tests
- Stockpile is a component that collects and lists TTPs for CALDERA
Those elements and more are available in the web interface when you install Caldera.
While Atomic Red Team’s Invoke-Atomic framework lets you simply test your detection capabilities for very specific “atomic” tests (or combinations thereof) on Windows, macOS, and Linux, Caldera offers you the starting point for much more involved adversary emulation. That said, digging deeper with CALDERA is a lot of work, and, in practice, it may be more than you’re looking for. Either way, take a look at the overview of documentation to learn more about the numerous possibilities available in CALDERA.
If you like the idea of having a framework to replay adversarial actions and want more reporting than Atomic Red Team offers out of the box, but feel like a lot of the above is more than you need, then Mordor may be exactly the middle ground that you’re looking for. Plus, it is a young project that welcomes contributors to develop more datasets.
Roberto Rodriguez created Mordor to help analysts who might not have extensive testing or red team experience simulate adversary behaviors to test their ability to detect or prevent. Mordor has really great documentation and excellent artwork—both solid additions. However, among the platforms analyzed here, Mordor is maybe the most different. While Atomic Red Team and CALDERA offer you the ability to test your security tooling or detection coverage with simulated attacks, Mordor focuses on the telemetry generated by simulated and real-world attacks.
From GitHub, in the maintainers’ own words:
Roberto and his brother Jose have deeply incorporated reporting via Jupyter Notebooks into their adversarial simulations—to create a very different approach to the same “atomic” type data, but with real-time reporting with which many of you in the community may already be familiar. They have also compiled all of this into the beautifully documented Threat-Hunter’s Playbook.
Roberto was recently on “The Brakeing Down Security” podcast, where in a two-part interview he explained the inspiration behind Mordor and some of its use cases. He explained that he especially wanted to ensure coverage of common privilege escalation and lateral movement techniques in Windows, which is why those are among the first and only tests currently available. In the second part of the interview, he dives deep into the data science aspects of Mordor and explains the benefits of using Jupyter Notebooks to share documentation.
We deliberately decided not to include Praetorian’s Purple Team Attack Automation tool in this comparison because it is entirely reliant on Metasploit. That said, it offers a great level of coverage and is certainly worth consideration.
Another noteworthy exemption from this comparison is Endgame’s Red Team Automation (RTA). It doesn’t seem to have been updated in some time, but it’s still a good collection of 50 or so tests for techniques like file timestomping, process injection, and beacon simulation.
The bottom line
One of the great strengths of each of these tools is that they are organized within the taxonomy of ATT&CK, thereby providing a common language and framework that has already been adopted by many organizations and other tools.
There is some overlap between Atomic Red Team and CALDERA, but the benefit of Atomic Red Team is clear: it’s fairly simple to set up and use, especially with the Invoke-Atomic and its detailed wiki page.
Caldera also offers a clear core strength too: it is extensible, and—given the sufficient time, attention, and resources—offers a free alternative to commercial testing and simulation platforms. Mordor, on the other hand, is entirely complementary to either CALDERA or Atomic Red Team.
Mordor offers unique reporting and the ability to examine how malicious materializes in different kinds of security telemetry. Further, the project is new and offers security professionals a great opportunity to contribute to an open source project that, over time, promises to become increasingly popular and helpful for security teams that are looking to improve their detection capabilities.
Many great open source projects are available to the red team and broader security community, and they don’t need to exist in mutual exclusivity. For example, there’s an awesome opportunity for someone to input Threat Hunter’s Playbook’s Jupyter Notebook approach into Atomic Red Team to create a new feature that adds in that same real-time reporting. Similarly, we can build tools for automatically running atomic tests in Invoke-Atomic or Caldera, and we can contribute the output of these tests back into Mordor.
No substitution for a skilled red team
Ultimately, these frameworks are no substitution for an experienced red team. The most skilled adversaries find exploitable gaps in tech or creative ways of using that tech to conduct attacks in ways that aren’t yet documented within frameworks like ATT&CK and Atomic Red Team.
A tool like Atomic Red Team will help validate that everything is working as it is supposed to, but a human red team will come up with clever attacks designed to subvert your organization’s specific defensive controls. In other words, a human red team ensures that you have coverage in between the frames of even the best frameworks.
Red teams also should work closely with trusted agents in a company to execute not only creative and innovative testing—to find (and help remediate) security gaps—but also to highlight specific areas of testing that have been predetermined to represent weaknesses in a company’s defenses. So in this latter sense, there may be some overlap between using framework-based automated testing; especially if it is being actively maintained by a large, cross-enterprise group of talented individuals.
A good starting point to read up on red teams is Tim MalcomVetter’s blog. I’ll highlight some other teams in upcoming weeks on my own blog as well.
JB (@Cherokeejb_) works in incident management, detection, and monitoring for a global organization. JB contributes as much as possible to the open source community, including the Atomic Red Team project, Internet Storm Center, and the Brakeing Down Security podcast. He has a passion for music, family, and nature; and his favorite working areas of research are currently macOS, forensics, and hunting for new, creative, and advanced defense tactics.