SOARCA is currently in its first release, with ongoing development aimed at expanding its capabilities, improving integration, and enhancing its functionalities. You can track our progress and upcoming milestones on GitHub.
We warmly welcome contributions to our repository. You can find the guidelines for contributing here.
SOARCA, an open-source SOAR (Security Orchestration, Automation and Response) tool developed by TNO, is designed be vendor-agnostic, allowing it to orchestrate various security actuators and systems. It is the first open-source SOAR that aims to be compliant with the CACAO v2.0 standard.
SOARCA enables cyber defenders to coordinate and automate their cyber operations, by using executable CACAO playbooks, and aims to achieve the following goals:
Standard Compliance: Adhering to the latest standards, including CACAO v2.0 and OpenC2, allows for interoperability with a wide range of technologies.
Extensibility with Open Interfaces: Enjoy the flexibility of an extensible tool featuring open and well-defined interfaces, promoting adaptability, customization, and experimentation.
Open-Source: Embrace an open-source model that not only offers cost-effective solutions but also supports unrestricted use and adaptation for research purposes.
You can use the following commands to execute the example playbooks via the terminal while SOARCA is running assuming on localhost. Alternatively you can go to http://localhost:8080/swagger/index.html and use the trigger/playbook endpoint.
Example playbooks:
# make sure an ssh server is running on adres 192.168.0.10curl -X POST -H "Content-Type: application/json" -d @./example/ssh-playbook.json localhost:8080/trigger/playbook
curl -X POST -H "Content-Type: application/json" -d @./example/http-playbook.json localhost:8080/trigger/playbook
curl -X POST -H "Content-Type: application/json" -d @./example/openc2-playbook.json localhost:8080/trigger/playbook
Configuration
SOARCA reads its configuration from the environment variables or a .env file. An example of a .env is given below:
git clone https://github.com/COSSAS/SOARCA.git
make build
cp .env.example .env
./build/soarca
Configuring SOARCA
variable
content
description
PORT
port
Set the exposed port of SOARCA the default is 8080
DATABASE
true | false
Set if you want to run with external database default is false
MONGODB_URI
uri
Set the Mongo DB uri default is mongodb://localhost:27017
DATABASE_NAME
name
Set the Mongo DB database name when using docker default is soarca
DB_USERNAME
user
Set the Mongo DB database user when using docker default is root
DB_PASSWORD
password
Set the Mongo DB database users password when using docker default is rootpassword. IT IS RECOMMENDED TO CHANGE THIS IN PRODUCTION!
MAX_REPORTERS
number
Set the maximum number of downstream reporters default is 5
LOG_GLOBAL_LEVEL
[Log levels]
One of the specified log levels. Defaults to info
LOG_MODE
development | production
If production is chosen the LOG_GLOBAL_LEVEL is used for all modules defaults to production
LOG_FILE_PATH
filepath
Path to the logfile you want to use for all logging. Defaults to "" (empty string)
LOG_FORMAT
text | json
The logging can be in plain text format or in JSON format. Defaults to json
MQTT_BROKER
dns name or ip
The broker address for SOARCA to connect to, for communication with fins default is localhost
MQTT_PORT
port
The broker address for SOARCA to connect to, for communication with fins default is 1883
ENABLE_FINS
true | false
Enable fins in SOARCA defaults to false
VALIDATION_SCHEMA_URL
url
Set a custom validation schema to be used to validate playbooks defaul is "" to use internal. NOTE: changing this heavily impacts performance.
2 - Vision & Concepts
The what and why of SOARCA
Context and Background
Security Orchestrator for Advanced Response to Cyber Attacks - SOARCA
Organisations are increasingly automating threat and incident response through playbook driven security workflow orchestration. The essence of this concept is that specific security events trigger a predefined series of response actions that are executed with no or only limited human intervention. These automated workflows are captured in machine-readable security playbooks, which are typically executed by a so called Security Orchestration, Automation and Response (SOAR) tool. The market for SOAR solutions has matured significantly over the past years and present day products support sophisticated automation workflows and a wide array of integrations with external security tools and data resources. Typically, however, the technology employed is proprietary and not easily adaptable for research and experimentation purposes. SOARCA aims to offer an open-source alternative for such solutions that is free of vendor dependencies and supports standardized formats and technologies where applicable.
SOARCA, TNO’s open-source SOAR, was developed for research and innovation purposes and allows SOC, CERT and CTI professionals to experiment with the concept of playbook driven security automation. It is open and extensible and its interfaces are well-defined and elaborately documented. It also offers native support for two emerging technology standards, both developed and maintained by OASIS Open:
CACAOv2. The Collaborative Automated Course of Action Operations (CACAO) standard provides a common framework and machine-processable schema for security playbooks that are natively interoperable and can be shared and executed across technological and organizational boundaries.
OpenC2. A standardized language for the command and control of cyber defense technologies. In essence it provides a vendor agnostic language and interface through which so called security actuators (e.g. firewalls or IAM solutions) can be reconfigured automatically.
SOARCA is available through TNO’s community platform COSSAS (Community for Open Source Security Automation Software) under the Apache 2.0 license. With its release, TNO aims to drive both the adoption and further development of novel technologies for cyber security automation forward. Here we note that open and accessible SOAR functionality is not only relevant for automation in threat and incident response but also for attack & defense simulations, cyber ranges, digital twinning and other emerging innovations that require orchestration of complex (security oriented) workflows.
At present, SOARCA is in an Alpha release phase and is intended for Proof of Concepts (PoCs) and research purposes, serving as a platform for demonstrations. The objective of the SOARCA team is to evolve SOARCA into a more mature SOAR orchestration tool suitable for operational environments. For potential applications of SOARCA, please refer to the ‘Use-Cases’ section of our documentation.
Core Concepts
Several concepts within SOARCA might be important to know.
Course of Action
A course of action (CoA) refers to a planned sequence of steps or activities taken to achieve a specific cyber security goal. These steps are often collected into “playbooks”. Usually in the form of prose in PDFs, internal wikis, or even scattered throughout emails.
The CACAO Security Playbooks Version 2.0 specification provides a standard for writing executable playbooks. These playbooks are stored in a machine-readable form, allowing them to be (semi-)automatically executed by an orchestration tool.
A CACAO playbook is a structured document that outlines a series of orchestrated actions to address specific security events, incidents, or other security-related activities. These playbooks allow for the automation of security steps.
Examples of repetitive tasks that might be automated using a CACAO Playbook might be:
Investigate the cause of security events.
Mitigate threats effectively.
Remediate vulnerabilities.
By following CACAO playbook specifications, organizations can enhance their automated response capabilities, foster collaboration, and ensure consistency of playbooks across diverse technological solutions.
SOARCA can be extended with custom extensions or rather so-called FIN (inspired by the majestic orca). A fin can be integrated within the SOARCA core. Technical descriptions of the components can be found here. Fins communicate with the SOARCA core using a pre-defined MQTT protocol.
Join the SOARCA Community
The SOARCA team invites cybersecurity professionals, researchers, and enthusiasts to join our community. Explore, adapt, and contribute to SOARCA’s growth. Let’s fortify cyber defenses together! See our contribution guidelines on how to make contributions. 🛡️🌐
Key Details
Project Name: SOARCA (Security Orchestrator for Advanced Response to Cyber Attacks)
SOARCA v1.0 only implements a part of the CACAO v2 spec. Only start, end, and action steps are supported at this time.
A CACAO playbook is a structured document that outlines a series of orchestrated actions to address specific security events, incidents, or other security-related activities. These playbooks allow for the automation of security steps.
SOARCA is a security orchestrator that reads the steps defined in a CACAO playbook and performs the necessary actions to execute the commands they contain. This makes a CACAO document an executable playbook.
SOARCA’s development is ongoing, and at this time, it only partly supports the entire CACAO specification. On this page we’ll go over the general concepts in a CACAO playbooks and the parts of the standard that are supported by SOARCA.
A CACAO playbook
Here, we have an example of a relatively simple CACAO playbook that demonstrates SOARCA’s capabilities. The flow of steps is depicted in the following image:
As you can see, this playbook contains a mix of logical steps (if-condition and while-condition) and action steps that perform commands on a target system.
Agents and targets
In CACAO playbooks, entities that execute commands are called agents, and the entities against which the commands are executed are called targets (see Agents and Targets in the spec).
Every action step in a CACAO playbook must have a single agent, and one or more targets. Both agents and targets are defined on the playbook level, in the agent_definitions and target_definitions properties. SOARCA will execute action steps that have an agent of the type soarca. The capability that will be selected to execute the step is determined by the name property of the agent. For more information, read the documentation on components.
Every CACAO playbook should start with a start step. From there, each step can define which step should be executed after the current step finishes. Depending on whether the current step has executed successfully, the next step is defined by the on_completion property. A non-successful step execution may instead trigger the workflow_exception step, specified in the playbook properties. Alternatively to on_completion, a step can specify on_success and on_failure, which allow a finer-grain control over the execution flow.
What constitutes a successful step execution and what is a failure depends on the specific capability executing the step.
An if-condition step allows executing different branches depending on a specified condition. The step must specify an on_true property, which references the start of a branch of steps that should be executed if the condition evaluates to true. Optionally, the if-condition step can define an on_false property that defines an alternative branch that is executed if the condition evaluates to false. In each case, the specified branch keeps executing until it encounters an end step.
The condition property contains a string that specifies a STIX Pattern. Currently, SOARCA only supports a very small subsection of the STIX Patterning specification. We support string based equality ('a' = 'a') and inequality ('a' != 'b') comparison. Example:
"if-condition--4b95eaa4-944a-4a9d-88d4-1374a70dbacd": {
"name": "If it is not new years",
"description": "Checks if it is 01-01-2025",
"on_completion": "end--db937fc8-3a42-41cc-b828-ec2db212f425",
"type": "if-condition",
"condition": "__soarca_ssh_result__:value != '01-01-2025'",
"on_true": "action--7fe08053-3685-4d8c-bc0a-40efce75113e"}
SOARCA supports variable interpolation, which means that variables can be used inside the condition property, as seen in the example above.
Similarly, CACAO specifies while-condition steps, whose on_true branch will be repeatedly executed until the condition evaluates to false.
The parallel step allows executing multiple branches (in parallel) specified in the next_steps property. At this time, the steps in next_steps are executed sequentially. Parallel execution is scheduled for a later release in SOARCA.
Next, we explain variables in CACAO and SOARCA.
Variables
The CACAO specification allows defining variables on the playbook level, as well as on the step level. Playbook variables are available throughout the playbook. In SOARCA, variables defined on the step level are available in that step, and in any step that executes in a sub-branch of an if-condition, while-condition, or parallel step.
According to the CACAO spec, variable names should start and end with double underscores (__). The CACAO spec allows defining multiple types of variables (strings, ip-addresses, numbers), but at this time SOARCA will interpret every value as a string. The constant and external properties are ignored.
SOARCA supports the interpolation of variables in different strings. The specific string-based properties that support interpolation depend on the capability. In general, string interpolation is supported in the properties of agents, targets, authentication information, and command properties.
Variable interpolation happens at the last possible moment, which means that step-dependant variables can be used in agent and target definitions.
Substitution is performed by replacing any occurrence of [variable_name]:value with the string value of that variable. Undefined variables are not replaced.
Action steps
Within CACAO playbooks, action steps can define commands that are executed by an agent against one or more targets. The agent and targets are referenced by ID. SOARCA selects the internal capability for handling the step by looking at the type and name of the agent. After selecting the proper capability, SOARCA will sequentially perform every command in the commands property for every target specified in targets. If any command fails to execute successfully, further execution is halted and the step is considered to have failed.
Action steps may return variables. On the successful execution of an action step, any variables returned are added to the globally available playbook variables. If the out_args property is specified and non-empty, only the variables listed in out_args will be added to the global playbook variables. The in_args property from the CACAO spec is ignored. Any variable defined on the playbook level, in parent-steps and within the step itself are available for interpolation.
In the case an action step ends in a failure, any variables returned from the step are ignored.
The example below shows how to run an ssh command on a single target system:
This page contains the Swagger documentation of the SOARCA REST API.
4 - Extensions & Capabilities
Extending SOARCA is done by developing a SOARCA-Fin.
Warning
SOARCA V.1.0.X implements currently the following native capalities: HTTP capability, OpenC2 capability, and SSH capability. Other core capabilities are part of our milestones which can be found here.
SOARCA features a set of native capabilities. The HTTP, OpenC2 HTTP, and SSH transport mechanisms are supported by the first release of SOARCA. SOARCA’s capabilities can be extended with custom implementations, which is further discussed on this page.
Extending the native capabilities
The native capabilities supported by SOARCA can be extended through a mechanism we named Fins. Your capability can be integrated with SOARCA by implementing the Fin protocol. This protocol regulates communication between SOARCA and the extension capabilities over an MQTT bus.
MQTT is a lightweight messaging protocol with libraries written in various programming languages. To integrate with SOARCA, you can write your own implementation of the Fin protocol, or use our python or golang libraries for easier integration.
Fin protocol
The underlying protocol for the SOARCA fins can be found here.
4.1 - Fin Python Library
Documentation of the Python Fin library
For the documentation about the Fin protocol we refer to documention page of SOARCA Fin Protocol.
Quick Start
To include the SOARCA Fin library, you can use the following command to install it via pip:
pip install soarca-fin-library
Example
An example on how to use the library is given below.
For more examples and the source code, we will refer to the Github page of the SOARCA-Fin-python-library, where we provide /examples folder.
The main object of the application is the SoarcaFin object, which is responsible for configuring and creating and controlling the capabilities.
The SoarcaFin creates MQTTClients for each capability registered, plus one for registering, unregistering and controlling the fi itself.
MQTTClients each have their own connection to the MQTT Broker and own Parser and Executor objects.
The Parser object parsers the raw MQTT messages and tries to convert them to one of the objects in src/models.
The Executor runs in their own thread and handles the actual execution of the messages.
The Executor polls a thread-safe queue for new messages and performs IO operations, such as sending messages to the MQTT broker and calling capability callbacks.
Setup SOARCA Capabilities
To register a fin to SOARCA, first create a SoarcaFin object and pass the fin_id in the constructor. The SOARCA fin_id must be in the format of: sourca-fin-<capability>-<uuid4>.
Call set_config_MQTT_server() to set the required configurations for the fin to connect to the MQTT broker.
For each capability to be registered, call create_fin_capability(). The capability callback funtion should return an object of type ResultStructure.
When all capabilities are initialized, call start_fin() for the SOARCA Fin to connect to the MQTT broker and register itself to SOARCA.
An example is given in this project in the file [examples/pong_example.py]
Want to contribute to this project? It is possible to contribute here.
Have you found a bug or want to request a feature? Please create an issue here.
4.2 - Fin protocol
Specification of the SOARCA Fin protocol
Goals
The goal of the protocol is to provide a simple and robust way to communicate between the SOARCA orchestrator and the capabilities (Fins) that can provide extra functions.
MQTT
To allow for dynamic communication MQTT is used to provide the backbone for the fin communication. SOARCA can be configured using the environment to use MQTT or just run stand-alone.
The Fin will use the protocol to register itself to SOARCA via the register message. Once register, it will communicate over the channel new channel designated by the fin UUID.
Commands to a specific capability will be communicated of the capability UUID channel.
Messages
Messages defined in the protocol
ack
nack
register
unregister
command
pause
resume
stop
legend
|field |content |type |description
|field name have the (optional) key if the field is not required |content indication |type of the value could be string, int etc. |A description for the field to provide extra information and context
@startuml
participant "SOARCA" as soarca
participant Capability as fin
soarca -> soarca : create [soarca] topic
fin -> fin : create [fin UUID] topic
soarca <- fin : [soarca] register
soarca --> fin : [fin UUID] ack
@enduml
Sending command
@startuml
participant "SOARCA" as soarca
participant Capability as fin
soarca -> fin : [capability UUID] command
soarca <-- fin : [capability UUID] ack
.... processing ....
soarca <- fin : [capability UUID] result
soarca --> fin: ack
@enduml
Unregistering a capability
@startuml
participant "SOARCA" as soarca
participant Capability as fin
participant "Second capability" as fin2
... SOARCA initiate unregistering one fin ...
soarca -> fin : [SOARCA] unregister fin-id
soarca <-- fin : [SOARCA] ack
note right fin2
This capability does not respond to this message
end note
... Fin initiate unregistering ...
soarca <- fin : [SOARCA] unregister fin-id
soarca --> fin : [SOARCA] ack
note right fin2
This capability does not respond to this message
end note
... SOARCA unregister all ...
soarca -> fin : [SOARCA] unregister all == true
soarca <-- fin : [SOARCA] ack
soarca <-- fin2 : [SOARCA] ack
note over soarca, fin2
soarca will go down after this command
end note
@enduml
Control
@startuml
participant "SOARCA" as soarca
participant Capability as fin
soarca -> fin : [fin UUID] control message
soarca <-- fin : [fin UUID] status
@enduml
4.3 - Native capabilities
Capabilities and transport mechanisms baked right into SOARCA
This page contains a list of capabilities that are natively implemented in SOARCA see details here. For MQTT-message-based capabilities, check here.
OpenC2 capability
The OpenC2 HTTP capability uses the http(s) transport layer as specified in OpenC2 HTTPS. It allows executing actions on an OpenC2-compatible security actuator.
SOARCA Core: This is the heart of SOARCA, represented in green.
SOARCA Native Capabilities: These are the functionalities explicitly defined in the Cacao v2 specification and are integral to the core. They are also represented in green.
Fins: These are the extension capabilities, also known as Fins. They enhance the functionality and integration of SOARCA and are depicted in orange. These are not (yet) part of this repository, but may be implemented by partners or TNO in the future.
Core component overview
SOARCA interacts with many components, the following diagram shows the intended setup for SOARCA and its components. Currently, not all parts are fully implemented. Keep track of the release notes to see what’s implemented and what needs some work.
5.1 - Application design
Details of the application architecture for SOARCA
Design decisions and core dependencies
To allow for fast execution and type-safe development SOARCA is developed in go. The application application can be deployed in Docker. Further dependencies are MQTT for the module system and go-gin for the REST API.
The overview on this page is aimed to guide you through the SOARCA architecture and components as well as the main flow.
Components
Components of SOARCA are displayed in the component diagram.
Green is implemented
Orange has limited functionality
Red is not started but will be added in future releases
@startuml
set separator ::
protocol /playbook #lightgreen
protocol /trigger #lightgreen
protocol /step #red
protocol /trusted/variables #red
protocol /status #red
class controller #lightgreen
class database #lightgreen
class log #lightgreen
class core::decomposer #lightgreen
class core::executor #lightgreen
class endpoints::playbook #lightgreen
class endpoints::trigger #lightgreen
class core::modules::http #lightgreen
class core::modules::ssh #lightgreen
class core::modules::openC2 #orange
class core::modules::fin #orange
class endpoints::step #red
class endpoints::variables #red
class endpoints::status #red
"/step" *-- endpoints::step
"/playbook" *-- endpoints::playbook
"/trigger" *-- endpoints::trigger
"/status" *-- endpoints::status
"/trusted/variables" *-- endpoints::variables
endpoints *-down- controller
controller -* database
log *- controller
controller -down-* core::decomposer
core::decomposer -down-> core::executor
core::executor --> core::modules::openC2
core::executor --> core::modules::fin
core::executor --> core::modules::http
core::executor --> core::modules::ssh
@enduml
Classes
This diagram consists of the class structure used by SOARCA
@startuml
interface IPlaybook
interface IStatus
interface ITrigger
Interface IPlaybookDatabase
Interface IDatabase
Interface ICapability
Interface IDecomposer
Interface IExecuter
class Controller
class Decomposer
class PlaybookDatabase
class Status
class Mongo
class Capability
Class Executer
IPlaybook <|.. Playbook
ITrigger <|.. Trigger
IStatus <|.. Status
ICapability <|.. Capability
IExecuter <|.. Executer
Trigger -> IPlaybookDatabase
IPlaybookDatabase <- Playbook
IPlaybookDatabase <|.. PlaybookDatabase
IDatabase <-up- PlaybookDatabase
IDatabase <|.. Mongo
IDecomposer <- Trigger
IDecomposer <|.. Decomposer
IExecuter <- Decomposer
ICapability <- Executer
@enduml
Controller
The SOARCA controller will create all classed needed by SOARCA. The controller glues the endpoints and decomposer together. Each run will instantiate a new decomposer.
These sequences will show a simplified overview of how the SOARCA components interact.
The main flow of the application is the following. Execution will start by processing the JSON formatted CACAO playbook if successful the playbook is handed over to the Decomposer. This is where the playbook is decomposed into its parts and passed step by step to the executor. These operations will block the API until execution is finished. For now, no variables are exposed via the API to the caller.
Actor Caller
Caller -> Api
Api -> Trigger : /trigger
Trigger -> Decomposer : Trigger playbook as ad-hoc execution
loop for each step
Decomposer -> Executor : Send step to executor
Executor -> Executor : select capability (ssh selected)
Executor -> Ssh : Command
Executor <-- Ssh : return
Decomposer <-- Executor
else execution failure (break loop)
Executor <-- Ssh : error
Decomposer <-- Executor: error
Decomposer -> Decomposer : stop execution
end
Trigger <-- Decomposer : execution details
Api <-- Trigger : execution details
Caller <-- Api
@startuml
protocol UiEndpoint {
GET /playbook
GET /playbook/meta
POST /playbook
GET /playbook/playbook-id
PUT /playbook/playbook-id
DELETE /playbook/playbook-id
POST /trigger/playbook
POST /trigger/playbook/id
GET /step
GET /status
GET /status/playbook
GET /status/playbook/id
GET /status/history
}
@enduml
General messages
Error
When an error occurs a 400 status is returned with the following JSON payload, the original call can be omitted in production for security reasons.
See if SOARCA is up this will only return if all SOARCA services are ready
Call payload
None
Response
200/OK
pong
Usage example flow
Stand alone
@startuml
participant "SWAGGER" as gui
control "SOARCA API" as api
control "controller" as controller
control "Executor" as exe
control "SSH-module" as ssh
gui -> api : /trigger/playbook with playbook body
api -> controller : execute playbook playload
controller -> exe : execute playbook
exe -> ssh : get url from log
exe <-- ssh : return result
controller <-- exe : results
api <-- controller: results
@enduml
Database load and execution
@startuml
participant "SWAGGER" as gui
control "SOARCA API" as api
control "controller" as controller
database "Mongo" as db
control "Executor" as exe
control "SSH-module" as ssh
gui -> api : /trigger/playbook/playbook--91220064-3c6f-4b58-99e9-196e64f9bde7
api -> controller : load playbook from database
controller -> db: retreive playbook
controller <-- db: playbook json
controller -> controller: validate playbook
controller -> exe : execute playbook
exe -> ssh : get url from log
exe <-- ssh : return result
controller <-- exe : results
api <-- controller: results
@enduml
A natural language explanation of the current status or related info
executed_by
entity-identifier
string
The entity executed the workflow step. This can be an organization, a team, a role, a defence component, etc.
commands_b64
list of base64
list of string
A list of Base64 encodings of the commands that were invoked during the execution of a workflow step, including any values stemming from variables. These are the actual commands executed.
error
error
string
Error raised along the execution of the step
variables
cacao variables
dictionary
Map of cacao variables handled in the step (both in and out) with current values and definitions
automated_execution
boolean
string
This property identifies if the workflow step was executed manually or automatically. It is either true or false.
The workflow step was executed successfully (completed).
failed
The workflow step failed.
ongoing
The workflow step is in progress.
server_side_error
A server-side error occurred.
client_side_error
A client-side error occurred.
timeout_error
A timeout error occurred. The timeout of a CACAO workflow step is specified in the “timeout” property.
exception_condition_error
A exception condition error ocurred. A CACAO playbook can incorporate an exception condition at the playbook level and, in particular, with the “workflow_exception” property.
If the execution has completed and no further steps need to be executed
200/OK
with payload:
@startjson
[
{
"type" : "execution-status",
"id" : "<execution-id>",
"execution_id" : "<execution-id>",
"playbook_id" : "<playbook-id>",
"started" : "<time-string>",
"ended" : "<time-string>",
"status" : "<status-enum-value>",
"status_text": "<status description>",
"errors" : ["error1", "..."],
"step_results" : {
"<step-id-1>" : {
"execution_id": "<execution-id>",
"step_id" : "<step-id>",
"started" : "<time-string>",
"ended" : "<time-string>",
"status" : "<status-enum-value>",
"status_text": "<status description>",
"errors" : ["error1", "..."],
"variables": {
"<variable-name-1>" : {
"type": "<type>",
"name": "<variable-name>",
"description": "<description>",
"value": "<value>",
"constant": "<true/false>",
"external": "<true/false>"
}
}
}
},
"request_interval" : "<n-seconds>"
}
]
@endjson
The payload will include all information that the finished execution has created.
If the execution is still ongoing:
206/Partial Content
with payload equal to the 200 response, but impliclty not including all information from the execution, since the execution is still ongoing.
The step results object will list the steps that have been executed until the report request, and those that are being executed at the moment of the report request.
The "request_interval" suggests the polling interval for the next request (default suggested is 5 seconds).
#### Error
400/BAD REQUEST with payload:
General error
404/NOT FOUND
No execution with the specified ID was found.
5.4 - Decomposer
Playbook deconstructor architecture
Decomposer structure
The decomposer will parse playbook objects to individual steps. This allows it to schedule new executor tasks.
Each incoming playbook will executed individually. Decomposing is done up to the step level.
Warning
SOARCA 1.0.x will only support steps of type action
Interface for interfacing with the Executor this will in turn select and execute the command on the right module or fin.
Execution details
The struct contains the details of the execution (execution id which is created for every execution) and the playbook id. The combination of these is unique.
Decomposition of playbook
participant caller
participant "Playbook decomposer" as decomposer
participant "Playbook state" as queue
participant Executor as exe
caller -> decomposer: Execute
caller <-- decomposer: ExecutionStatus
decomposer -> queue: store state
decomposer <-- queue:
loop for all playbook steps
decomposer -> queue: load state
decomposer <-- queue:
decomposer -> decomposer: parse step
decomposer -> decomposer: parse command
decomposer -> exe: execute command
note over exe: correct executer is selected
... Time has passed ...
decomposer <-- exe
end loop
5.5 - Executer
Design of the SOARCA step executer
Components
The executor consists of the following components.
Action executor
Playbook action executor
if-condition executor
while-condition executor
parallel executor
The decomposer interacts with every executor type. They all have separate interfaces to handle new step types in the future without changing the current interfaces.
package action{
interface IExecutor {
..., err Execute(...)
}
}
package playbookaction{
interface IExecutor {
..., err Execute(...)
}
}
package ifcondition{
interface IExecutor {
..., err Execute(...)
}
}
package whilecondition{
interface IExecutor {
..., err Execute(...)
}
}
package parallel{
interface IExecutor {
..., err Execute(...)
}
}
interface ICapability{
variables, error Execute(Metadata, command, variable[], target, agent)
string GetModuleName()
}
class "Decomposer" as decomposer
class "Action Executor" as Executor
class "Playbook Executor" as playbook
class "Parallel Executor" as parallelexecutor
class "While Executor" as while
class "If condition Executor" as condition
class "Ssh" as ssh
class "OpenC2" as openc2
class "HttpApi" as api
class "Fin" as fin
action.IExecutor <|.. Executor
ICapability <-up- Executor
ICapability <|.. ssh
ICapability <|.. openc2
ICapability <|.. api
ICapability <|.. fin
playbookaction.IExecutor <|.. playbook
ifcondition.IExecutor <|.. condition
whilecondition.IExecutor <|.. while
parallel.IExecutor <|.. parallelexecutor
decomposer -down-> playbookaction.IExecutor
decomposer -down-> ifcondition.IExecutor
decomposer -down-> whilecondition.IExecutor
decomposer -down-> parallel.IExecutor
decomposer -down-> action.IExecutor
Action executor
The action executor consist of the following components
The capability selector
Native capabilities (command executors)
MQTT capability to interact with: Fin capabilities (third-party executors)
The capability selector will select the implementation which is capable of executing the incoming command. There are native capabilities based on the CACAO command-type-ov:
Currently implemented
ssh
http-api
openc2-http
powershell
Coming soon
manual
Future (potentially)
bash
caldera-cmd
elastic
jupyter
kestrel
sigma
yara
Native capabilities
The executor will select a module that is capable of executing the command and pass the details to it. The capability selection is performed based on the agent type (see Agent and Target Common Properties in the CACAO 2.0 spec). The convention is that the agent type must equal soarca-<capability identifier>, e.g. soarca-ssh or soarca-openc2-http.
The result of the step execution will be returned to the decomposer. A result can be either output variables or error status.
MQTT executor -> Fin capabilities
The Executor will put the command on the MQTT topic that is offered by the module. How a module handles this is described in the link:modules.adoc[module documentation]
Component overview
package "Controller" {
component Decomposer as parser
}
package "Executor" {
component SSH as exe2
component "HTTP-API" as exe1
component MQTT as exe3
}
package "Fins" {
component "VirusTotal" as virustotal
component "E-mail Sender" as email
}
parser -- Executor
exe3 -- Fins : " MQTT topics"
Sequences
Example execution for SSH commands with SOARCA native capability.
@startuml
participant Decomposer as decomposer
participant "Capability selector" as selector
participant "SSH executor" as ssh
decomposer -> selector : Execute(...)
alt capability in SOARCA
selector -> ssh : execute ssh command
ssh -> ssh :
selector <-- ssh : results
decomposer <-- selector : OnCompletionCallback
else capability not available
decomposer <-- selector : Execution failure
note right: No capability can handle command \nor capability crashed etc..
end
Playbook action executor
The playbook executor handles execution of playbook action steps. The variables from the top level playbook are injected into the be executed playbook.
It could happen that in the downstream playbook the variables collide with the top level playbook. In this case the top level playbook variables are NOT transferred to the downstream playbook. Agents and Targets cannot be transferred between playbooks at this time. Playbooks are only loaded in the executor and then a new Decomposer is created to execute the playbook.
The result of the step execution will be returned to the decomposer. A result can be either output variables or error status.
package playbookaction{
interface IExecutor {
Variables, err Execute(meta, step, variables)
}
}
class "Decomposer" as decomposer
class "Action Executor" as exe
interface "IPlaybookController" as controller
interface "IDatabaseController" as database
playbookaction.IExecutor <|.. exe
decomposer -> playbookaction.IExecutor
exe -> controller
database <- exe
If condition executor
The if-condition executor will process a cacao if-condition step and determine it’s output.
The result of the step comparison will be returned to the decomposer. A result can be either a next step id and/or error status.
While condition executor
The if-condition executor will process a cacao while-condition step and determine it’s output.
The result of the step comparison will be returned to the decomposer. A result can be either a next step id and/or error status. Only STIX comparison expressions are implemented at this time.
The parallel executor will execute the parallel step. This wil be done in sequence to simplify implementation. As parallel steps must not be depended on each other sequential execution is possible. Later this will be changed.
5.6 - Executer Modules
Native executer modules
Executer modules are part of the SOARCA core. Executer modules perform the actual commands in CACAO playbook steps.
Native modules in SOARCA
The following capability modules are currently defined in SOARCA:
ssh
http-api
openc2-http
powershell
The capability will be selected based on the agent in the CACAO playbook step. The agent should be of type soarca and have a name corresponding to soarca-[capability name].
SSH capability
This capability executes SSH Commands on the specified targets.
This capability support User Authentication using the user-auth type. For SSH authentication username/password is authentication supported.
Success and failure
The SSH step is considered successful if a proper connection to each target can be initialized, the supplied command executes without error, and returns with zero exit status.
In every other circumstance the step is considered to have failed.
Variables
This module does not define specific variables as input, but variable interpolation is supported in the command and target definitions. It has the following output variables:
{
"__soarca_ssh_result__": {
"type": "string",
"value": "<stdout of the last command>" }
}
The command is considered to have successfully completed if a successful HTTP response is returned from each target. An HTTP response is successful if it’s response code is in the range 200-299.
Variables
This capability supports variable interpolation in the command, port, authentication info, and target definitions.
The result of the step is stored in the following output variables:
It supports the same authentication mechanisms as the HTTP-API capability.
Success and failure
Any successful HTTP response from an OpenC2 compliant endpoint (with a status code in the range 200-299) is considered a success. Connection failures and HTTP responses outside the 200-299 range are considered a failure.
Variables
It supports variable interpolation in the command, headers, and target definitions.
The result of the step is stored in the following output variables:
It supports the username, password authentication mechanism.
Success and failure
Any successful command will have a __soarca_powershell_result__. If an error occurs on the target a __soarca_powershell_error__ populated will be returned and Error will be set.
Variables
It supports variable interpolation in the command, headers, and target definitions.
The result of the step is stored in the following output variables:
participant Controller as controller
participant "Playbook Database" as playbook
database Database as db
controller -> playbook : get(id)
playbook -> db : read(playbookId)
note right
When the create fails an error will be thrown
end note
playbook <-- db : "playbook JSON"
controller <-- playbook: "CacaoPlaybook Object"
Writing playbook data
participant Controller as controller
participant "Playbook Database" as playbook
database Database as db
controller -> playbook : set(CacaoPlaybook Object)
playbook -> db : create(playbook JSON)
note right
When the create fails an error will be thrown
end note
Update playbook data
participant Controller as controller
participant "Playbook Database" as playbook
database Database as db
controller -> playbook : update(CacaoPlaybook Object)
playbook -> db : update(playbook id,playbook JSON)
note right
When the create fails an error will be thrown
end note
playbook <-- db : true
controller <-- playbook: true
Delete playbook data
participant Controller as controller
participant "Playbook Database" as playbook
database Database as db
controller -> playbook : remove(playbook id)
playbook -> db : remove(playbook id)
note right
When the create fails an error will be thrown
end note
Handling an error
participant Controller as controller
participant "Playbook Database" as playbook
database Database as db
controller -> playbook : remove(playbook id)
playbook -> db : remove(playbook id)
playbook <-- db: error
note right
playbook does not exists
end note
controller <-- playbook: error
5.8 - Reporter
Reporting of Playbook worfklow information and steps execution
SOARCA utilizes push-based reporting to provide information on the instantiation of a CACAO workflow, and information on the execution of workflow steps.
General Reporting Architecture
For the execution of a playbook, a Decomposer and invoked Executors are injected with a Reporter. The Reporter maintains the reporting logic that reports execution information to a set of specified and available reporting targets.
A reporting target can be internal to SOARCA, such as a Cache. A reporting target can also be a third-party tool, such as an external SOAR/ SIEM, or incident case management system (see connectors).
Upon execution trigger for a playbook, information about the chain of playbook steps to be executed will be pushed to the targets via dedicated reporting classes.
Along the execution of the workflow steps, the reporting classes will dynamically update the steps execution information such as output variables, and step execution success or failure.
The reporting modules enables the population and updating of views and data concerning workflow composition and its dynamic execution results. This data can be transmitted to SOARCA internal reporting components such as a cache, as well as to third-party tools (see connectors).
The schema below represents the architecture concept.
Reporting functionalities are triggered asynchronously by means of go routines, such that reporting logic does not affect the execution timings of the CACAO playbooks. Note that this also implies that there might be small inconsistencies between the actual status of an execution, and what can be found in the reporting. For instance, reporting on the status of a step N, might take longer than reporting on the status of step N+1. Typically, the actual playbook execution may be slightly ahead of the reporting - and generally within a seconds-wide window. Execution timings, though, are always reported correctly, since we generated them within the workflow execution itself, and not within the reporting module.
The flow diagram below highlights the asynchronous mechanisms of reporting.
@startuml
participant "Decomposer" as decomposer
participant "Executor" as executor
participant "Reporter" as reporter
participant "DonwstreamReporter" as ds_reporter
decomposer -> reporter : ReportWorkflowStart()
reporter -->> ds_reporter : go ReportWorkflowStart()
decomposer -> executor : Execute step
activate executor
executor -> reporter : ReportStepStart()
reporter -->> ds_reporter : go ReportStepStart()
executor -> reporter : ReportStepEnd()
reporter -->> ds_reporter : go ReportStepEnd()
decomposer <- executor : return (variables, errors)
deactivate executor
decomposer -> reporter : ReportWorkflowEnd()
reporter -->> ds_reporter : go ReportWorkflowEnd()
@enduml
Note that the main reporter module is invoked synchronously. In turn, the main reporter module calls all downstream reporters, which implement the actual reporting logic, asynchronously. Also note any eventual reporting error never stops an execution, and only logs a warning.
Interfaces
The reporting logic and extensibility is implemented in the SOARCA architecture by means of reporting interfaces. At this stage, we implement an IWorkflowReporter to push information about the entire workflow to be executed, and an IStepReporter to push step-specific information as the steps of the workflow are executed.
A high level Reporter component will implement both interfaces, and maintain the list of DownStreamRepporters activated for the SOARCA instance. The Reporter class will invoke all reporting functions for each active reporter. The Executer and Decomposer components will be injected each with the Reporter though, as interface of respectively workflow reporter, and step reporter, to keep the reporting scope separated.
The DownStream reporters will implement push-based reporting functions specific for the reporting target, as shown in the IDownStreamReporter interface. Internal components to SOARCA, and third-party tool reporters, will thus implement the IDownStreamReporter interface.
Native Reporters
SOARCA implements internally reporting modules to handle database and caches reporting.
Cache reporter
The Cache reporter mediates between decomposer and executors, database, and reporting APIs. As DownStreamReporter, the Cache stores workflow and step reports in-memory for an ongoing execution. As IExecutionInformant, the Cache provides information to the reporting API. The schema below shows how it is positioned in the SOARCA architecture.
The Cache thus reports the execution information downstream both in the database, and in memory. Upon execution information requests from the /reporter API, the cache can provide information fetching either from memory, or querying the database.
Connectors and 3P tools reporting
Reporting towards a 3rd Party tool is implemented in SOARCA by means of two components.
a 3PTool DownStreamReporter that organizes the execution information according to the data formats of the 3rd Party tool, and
a Connector module that receives data in 3PTool-specific formats, and handles the connectivity calls towards the 3P tool
At the current stage, the following 3rd Party tool DownStreamReporters and Connectors will be implemented in SOARCA.
TheHive
The TheHiveReporter reports on the execution of a playbook to an instance of The Hive platform. In order to be used, The Hive’s address and api key need to be configured in the environment variables of the SOARCA instance, either in the .env file for a source-code-built SOARCA instance, or in the SOARCA docker-compose for Docker builds. SOARCA will negotiate the authorization with the The Hive instance upon SOARCA initialization.
The TheHiveConnector receives execution information from the TheHiveReporter, and performs the actual network calls to report such infromation to the The Hive instance.
At this stage, third-party tools integrations may be built in SOARCA via packages implementing reporting logic for the specific tools. Alternatively, third-party tools may implement pull-based mechanisms (via the API) to get information from the execution of a playbook via SOARCA.
In the near future, we will (also) make available a SOARCA Report API that can establish a WebSocket connection to a third-party tool. As such, this will thus allow SOARCA to push execution updates as they come to third-party tools, without external tools having to poll SOARCA.
Furthermore, we will define more Connector donwstream reporters which will implement workflow execution reporting for specific platforms and products.
5.9 - Logging
SOARCA support extensive logging. Logging is based on the logrus framework.
SOARCA supports extensive logging. Logging is based on the logrus framework.
Format
Logging can be done in different formats suitable for your application. The following formats are available:
JSON default
Plain text
Destination
std::out default (terminal)
To file (to log file path)
Later:
syslog (NOT YET IMPLEMENTED)
Log levels
SOARCA supports the following log levels. Also is indicated how they are used.
PANIC (non fixable error system crash)
FATAL (non fixable error, restart would fix)
ERROR (operation went wrong but can be caught by other higher component)
WARNING (let the user know some operation might not have the expected result but execution can continue on normal path)
INFOdefault (let the user know that a major event has occurred)
DEBUG (add some extra detail to normal execution paths)
TRACE (get some fine grained detail from the logging)
Types of logging
SOARCA will log different information, these will be combined in the same output.
Runtime logging
Runtime logging wil include the running state of SOARCA, errors encountered when registering modules etc.
Security event logging
Will log the status of the execution of an playbook, database updates of playbooks
Using the logger (developer)
To use SOARCA logging you can add the following to your module.
type YourModule struct {
}
var component = reflect.TypeOf(YourModule{}).PkgPath()
var log *logger.Log
funcinit() {
log = logger.Logger(component, logger.Info, "", logger.Json)
}
Changing log level
To change logging for your SOARCA instance you can use the following environment variables
variable
content
description
LOG_GLOBAL_LEVEL
[Log levels]
One of the specified log levels. Defaults to info
LOG_MODE
development | production
If production is chosen the LOG_GLOBAL_LEVEL is used for all modules defaults to production
LOG_FILE_PATH
filepath
Path to the logfile you want to use for all logging. Defaults to "" (empty string)
LOG_FORMAT
text | json
The logging can be in plain text format or in JSON format. Defaults to json
This can be set as environment variables or loaded through the .env
Finally, the first release of the alpha release of SOARCA is here! In the first release, we emphasize laying the initial foundation and design of SOARCA to enable proof-of-concept demonstrations. Included in this release:
Execution of CACAO action steps via:
SSH
HTTP API
OpenC2
Logging capabilities
Efficient storage for playbooks
Design fin-protocol: the protocol description between SOARCA and the TO BE ADDED
7 - About
About COSSAS
SOARCA finds its home within the Community for Open Source Security Automation Software (COSSAS). COSSAS offers a continuously expanding base of novel software components for cyber security automation that SOC, CERT and CTI professionals can deploy and trial in their operational environments. COSSAS is an initiative of TNO.
About TNO
SOARCA is a project initiated and supported by TNO, the Netherlands Organisation for Applied Scientific Research. TNO, an independent research, development, and consultancy organization in the Netherlands, is dedicated to driving innovation for a safer, healthier, and more sustainable life. TNO focuses on areas such as sustainability, health, safety, and digital transformation. Its mission is to make knowledge serve the common good, connecting expertise to create impactful innovations. TNO actively engages with society, aiming for a positive impact through its work and insights. Learn more about TNO on their official website.
Core Team
Maarten de Kruijf (TNO)
Jan-Paul Konijn (TNO)
Hidde-Jan Jongsma (TNO)
Luca Morgese (TNO)
Richard Kerkdijk (TNO)
Frank Fransen (TNO)
Shari Finner (TNO)
Special Thanks
A special thanks to:
Sebastiaan Huskins for laying the initial and technical foundation for SOARCA.
Myla Fransen for the logo designs.
David Roefs as part of the AInception team
About SOARCA
This work has received funding from the European Union’s Horizon Europe Energy Research and Innovation programme under Grant Agreement No. 101075665 (eFORT project) and the European Defence Fund (EDF) under Grant Agreement No. 101103385 (AInception project). Views and opinions expressed are those of the authors only and do not necessarily reflect those of the European Union or European Climate, Infrastructure and Environment Executive Agency (CINEA). Neither the European Union nor the granting authority can be held responsible for them.
8 - Contribution Guidelines
How to contribute to SOARCA
SOARCA is an open-source project written in Golang and we love getting patches and contributions, and feature suggestions to make SOARCA and its docs even better. We welcome participation from anyone, regardless of their affiliation with OASIS. We invite constructive contributions and feedback from all contributors, following the standard practices for participation in GitHub public repository projects.
We expect everyone to follow our Code of Conduct, the licenses for each repository, and agree to our Contributor License Agreement when you make your first contribution.
Thank you for contributing to our project! Your efforts make a difference.
Before making contributions to the project repositories, please follow these general steps for GitHub contribution.
I found a bug / Creating issues
If there’s something you’d like to see in SOARCA (or if you’ve found something that isn’t working the way you’d expect), but you’re not sure how to fix it yourself, please create an issue. Make sure to adhere to the structure of an issue submission. Fully comprehend the problem at hand and provide comprehensive details in your issue description.
Security issues
For security issues, we kindly request that you refrain from reporting them using the issue tracker. Instead, please contact us directly: slack
Every submission, including those from project members, must undergo review and approval from at least one core maintainer. GitHub pull requests are utilized for this process. Consult GitHub Help for more
information on using pull requests.
Branch naming
The CI is configured to only allow for certain branch naming namely:
master
development
feature/
feature/docs/
bugfix/
release/x.x
hotfix/
Coding style
The project has opted to select the go style guide with some exceptions:
Would you like to enhance our documentation? Our documentation is built using the Hugo framework along with the Docsy theme template.
Quick start with Hugo and Docsy
Install Hugo; the installation guide can be found here.
Clone our repository, and if you make changes, fork our repository. Use the following command to clone: git clone <repository_url>.
All the documentation for the GitHub Pages lives under /documentation. Use the cd documentation && hugo serve command to preview the documentation locally. Open http://localhost:1313 in your web browser to view the documentation. In most cases, docsy will automatically reload the site to reflect any changes to the documentation or the code. Changes to some parts of the docsy code may require manually reloading the page or restarting the container.
Continue with the usual GitHub workflow to edit files, commit them, push the changes up to your fork, and create a pull request.
Updating a single page
If you’ve just spotted something you’d like to change while using the docs, Docsy has a shortcut for you:
Click Edit this page in the top right-hand corner of the page.
If you don’t already have an up-to-date fork of the project repo, you are prompted to get one - click Fork this repository and propose changes or Update your Fork to get an up-to-date version of the project to edit. The appropriate page in your fork is displayed in edit mode.
License
The project is licensed under the Apache License 2.0. See full license here.
8.1 - Contributor Covenant Code of Conduct
Our Pledge
We, as contributors and maintainers, pledge to make participation in our project and community a harassment-free experience for everyone, regardless of age, body size, disability, ethnicity, gender identity and expression, level of experience, nationality, personal appearance, race, religion, or sexual identity and orientation.
Our Standards
Examples of behavior that contributes to creating a positive environment include:
Using welcoming and inclusive language
Respecting differing viewpoints and experiences
Gracefully accepting constructive criticism
Focusing on what is best for the community
Showing empathy towards other community members
Examples of unacceptable behavior by participants include:
The use of sexualized language or imagery and unwelcome sexual attention or advances
Trolling, insulting/derogatory comments, and personal or political attacks
Public or private harassment
Publishing others’ private information, such as a physical or electronic address, without explicit permission
Any other conduct that could reasonably be considered inappropriate in a professional setting
Our Responsibilities
Project maintainers are responsible for clarifying the standards of acceptable behavior and are expected to take appropriate and fair corrective action in response to any instances of unacceptable behavior.
Project maintainers have the right and responsibility to remove, edit, or reject comments, commits, code, wiki edits, issues, and other contributions that are not aligned to this Code of Conduct, or to ban temporarily or permanently any contributor for other behaviors that they deem inappropriate, threatening, offensive, or harmful.
Scope
This Code of Conduct applies both within project spaces and in public spaces when an individual is representing the project or its community. Examples of representing a project or community include using an official project e-mail address, posting via an official social media account, or acting as an appointed representative at an online or offline event.
Enforcement
Instances of abusive, harassing, or otherwise unacceptable behavior may be reported by contacting the project team via Github or join our Slack channel. All complaints will be reviewed and investigated and will result in a response that is deemed necessary and appropriate to the circumstances. The project team is obligated to maintain confidentiality with regard to the reporter of an incident. Further details of specific enforcement policies may be posted separately.
Project maintainers who do not follow or enforce the Code of Conduct in good faith may face temporary or permanent repercussions as determined by other members of the project’s leadership.
Attribution
This Code of Conduct is adapted from the Contributor Covenant, version 2.1, available at here
For answers to common questions about this code of conduct, see
9 - Frequently asked questions
General questions
Does SOARCA have a GUI?
SOARCA currently does not feature a GUI for tracking the progress of playbook execution. To edit CACAO v2.0 playbooks, consider using the CACAO Roaster project.
