Design

• 1: Application design
• 2: API Description
• 3: Reporter API Description
• 4: Decomposer
• 5: Executer
• 6: Executer Modules
• 7: Database
• 8: Reporter
• 9: Logging

1 - Application design

Design decisions and core dependencies

To allow for fast execution and type-safe development SOARCA is developed in go. The application application can be deployed in Docker. Further dependencies are MQTT for the module system and go-gin for the REST API.

The overview on this page is aimed to guide you through the SOARCA architecture and components as well as the main flow.

Components

Components of SOARCA are displayed in the component diagram.

• Green is implemented
• Orange has limited functionality
• Red is not started but will be added in future releases

@startuml
set separator ::

protocol /playbook #lightgreen
protocol /trigger #lightgreen
protocol /step #red
protocol /trusted/variables #red
protocol /status #red

class controller  #lightgreen
class database  #lightgreen
class log  #lightgreen
class core::decomposer #lightgreen
class core::executor  #lightgreen
class endpoints::playbook #lightgreen
class endpoints::trigger #lightgreen
class core::modules::http #lightgreen
class core::modules::ssh #lightgreen
class core::modules::openC2 #orange
class core::modules::fin #orange

class endpoints::step #red
class endpoints::variables #red
class endpoints::status #red


"/step" *-- endpoints::step 
"/playbook" *-- endpoints::playbook
"/trigger" *-- endpoints::trigger
"/status" *-- endpoints::status
"/trusted/variables" *-- endpoints::variables

endpoints *-down- controller
controller -* database
log *- controller
controller -down-* core::decomposer
core::decomposer -down-> core::executor
core::executor --> core::modules::openC2
core::executor --> core::modules::fin
core::executor --> core::modules::http
core::executor --> core::modules::ssh
@enduml

Classes

This diagram consists of the class structure used by SOARCA

@startuml

interface IPlaybook
interface IStatus
interface ITrigger
Interface IPlaybookDatabase
Interface IDatabase
Interface ICapability
Interface IDecomposer
Interface IExecuter

class Controller
class Decomposer
class PlaybookDatabase
class Status
class Mongo
class Capability
Class Executer


IPlaybook <|.. Playbook
ITrigger <|.. Trigger
IStatus <|.. Status
ICapability <|.. Capability
IExecuter <|.. Executer
Trigger -> IPlaybookDatabase
IPlaybookDatabase <- Playbook
IPlaybookDatabase <|.. PlaybookDatabase
IDatabase <-up- PlaybookDatabase
IDatabase <|.. Mongo
IDecomposer <- Trigger
IDecomposer <|.. Decomposer
IExecuter <- Decomposer
ICapability <- Executer
@enduml

Controller

The SOARCA controller will create all classed needed by SOARCA. The controller glues the endpoints and decomposer together. Each run will instantiate a new decomposer.

interface IPlaybook{
    void Get()
    void Get(PlaybookId id)
    void Add(Playbook playbook)
    void Update(Playbook playbook)
    void Remove(Playbook playbook)
}

interface IStatus{

}

interface ITrigger{
    void TriggerById(PlaybookId id)
    void Trigger(Playbook playbook)
}

Interface IPlaybookDatabase

Interface IDecomposer
Interface IExecuter

class Trigger
class Controller
class Decomposer

IPlaybook <|.. Playbook
ITrigger <|.. Trigger
IStatus <|.. Status


Trigger -> IPlaybookDatabase
IPlaybookDatabase <- Playbook
IPlaybookDatabase <|.. PlaybookDatabase


IDecomposer <- Trigger
IDecomposer <|.. Decomposer
IExecuter -> Decomposer

Main application flow

These sequences will show a simplified overview of how the SOARCA components interact.

The main flow of the application is the following. Execution will start by processing the JSON formatted CACAO playbook if successful the playbook is handed over to the Decomposer. This is where the playbook is decomposed into its parts and passed step by step to the executor. These operations will block the API until execution is finished. For now, no variables are exposed via the API to the caller.

Actor Caller
Caller -> Api
Api -> Trigger : /trigger
Trigger -> Decomposer : Trigger playbook as ad-hoc execution
loop for each step 
Decomposer -> Executor : Send step to executor
Executor -> Executor : select capability (ssh selected)
Executor -> Ssh : Command
Executor <-- Ssh : return
Decomposer <-- Executor
else execution failure (break loop)
Executor <-- Ssh : error
Decomposer <-- Executor: error
Decomposer -> Decomposer : stop execution

end 
Trigger <-- Decomposer : execution details
Api <-- Trigger : execution details
Caller <-- Api

2 - API Description

Endpoint description

We will use HTTP status codes https://en.wikipedia.org/wiki/List_of_HTTP_status_codes

@startuml
protocol UiEndpoint {
    GET     /playbook
    GET     /playbook/meta
    POST    /playbook
    GET     /playbook/playbook-id
    PUT     /playbook/playbook-id
    DELETE  /playbook/playbook-id


    POST    /trigger/playbook
    POST    /trigger/playbook/id

    GET     /step

    GET     /status
    GET     /status/playbook
    GET     /status/playbook/id
    GET     /status/history

}
@enduml

General messages

Error

When an error occurs a 400 status is returned with the following JSON payload, the original call can be omitted in production for security reasons.

responses: 400/Bad request

@startjson
{
    "status": "400",
    "message": "What went wrong.",
    "original-call": "<optional> Request JSON data",
    "downstream-call" : "<optional> downstream call JSON"
}
@endjson

Unauthorized

When the caller does not have valid authentication 401/unauthorized will be returned.

cacao playbook JSON

@startjson
{
            "type": "playbook",
            "spec_version": "cacao-2.0",
            "id": "playbook--91220064-3c6f-4b58-99e9-196e64f9bde7",
            "name": "coa flow",
            "description": "This playbook will trigger a specific coa",
            "playbook_types": ["notification"],
            "created_by": "identity--06d8f218-f4e9-4f9f-9108-501de03d419f",
            "created": "2020-03-04T15:56:00.123456Z",
            "modified": "2020-03-04T15:56:00.123456Z",
            "revoked": false,
            "valid_from": "2020-03-04T15:56:00.123456Z",
            "valid_until": "2020-07-31T23:59:59.999999Z",
            "derived_from": [],
            "priority": 1,
            "severity": 1,
            "impact": 1,
            "industry_sectors": ["information-communications-technology", "research", "non-profit"],
            "labels": ["soarca"],
            "external_references": [
                {
                    "name": "TNO SOARCA",
                    "description": "SOARCA Homepage",
                    "source": "TNO - COSSAS - HxxPS://LINK-TO-CODE-REPO.TLD",
                    "url": "HxxPS://LINK-TO-CODE-REPO.TLD",
                    "hash": "00000000000000000000000000000000000000000000000000000000000",
                    "external_id": "TNO/SOARCA 2023.01"
                }
            ],
            "features": {
                "if_logic": true,
                "data_markings": false
            },
            "markings": [],
            "playbook_variables": {
                "$$flow_data_location$$": {
                    "type": "string",
                    "value": "<mongodb_location>",
                    "description": "location of event and flow data",
                    "constant": true
                },
                "$$event_type$$": {
                    "type" : "string",
                    "value": "<event_type_string>",
                    "description": "type of incomming event / trigger",
                    "constant": true	
                }
            },
            "workflow_start": "step--d737c35f-595e-4abf-83ef-d0b6793556b9",
            "workflow_exception": "step--40131926-89e9-44df-a018-5f92f2df7914",
            "workflow": {
                "step--5ea28f63-ac32-4e5e-bd0c-757a50a3a0d7":{
                    "type": "single",
                    "name": "BI for CoAs",
                    "delay": 0,
                    "timeout": 30000,
                    "command": {
                        "type": "http-api",
                        "command": "hxxps://our.bi/key=VALUE"
                    },
                    "on_success": "step--71b15428-275a-49b5-9f09-3944972a0054",
                    "on_failure": "step--71b15428-275a-49b5-9f09-3944972a0054"
                },
                "step--71b15428-275a-49b5-9f09-3944972a0054": {
                    "type": "end",
                    "name": "End Playbook SOARCA Main Flow"
                }
            },
            "targets": { 

            },
            "extension_definitions": { }
        }
@endjson

/playbook

The playbook endpoints are used to create playbooks in SOARCA, new playbooks can be added, and current ones edited and deleted.

GET /playbook

Get all playbook ids that are currently stored in SOARCA.

Call payload

None

Response

200/OK with payload:

@startjson
[
    {
        "type": "playbook",
        "etc" : "etc..."   
    }
]
@endjson

Error

400/BAD REQUEST with payload: General error

GET /playbook/meta

Get all playbook ids that are currently stored in SOARCA.

Call payload

None

Response

200/OK with payload:

@startjson

[
    {
        "id": "<playbook id>",
        "name": "<playbook name>",
        "description": "<playbook description>",
        "created": "<creation data time>",
        "valid_from": "<valid from date time>",
        "valid_until": "<valid until date time>",
        "labels": ["label 1","label 2"]
    }
]

@endjson

Error

400/BAD REQUEST with payload: General error

POST /playbook

Create a new playbook and store it in SOARCA. The format is

Payload

@startjson
{
    "type": "playbook",
    "etc" : "etc..."   
}
@endjson

Response

201/CREATED

@startjson
{
    "type": "playbook",
    "etc" : "etc..."   
}
@endjson

Error

400/BAD REQUEST with payload: General error, 409/CONFLICT if the entry already exists

GET /playbook/{playbook-id}

Get playbook details

Call payload

None

Response

200/OK with payload:

@startjson
{
    "<cacao-playbook> (json)"
}
@endjson

Error

400/BAD REQUEST

PUT `/playbook/{playbook-id}``

An existing playbook can be updated with PUT.

Call payload

A playbook like cacao playbook JSON

Response

200/OK with the edited playbook cacao playbook JSON

Error

400/BAD REQUEST for malformed request

When updated it will return 200/OK or General error in case of an error.

DELETE /playbook/{playbook-id}

An existing playbook can be deleted with DELETE. When removed it will return 200/OK or general error in case of an error.

Call payload

None

Response

200/OK if deleted

Error

400/BAD REQUEST if the resource does not exist

POST /trigger/playbook/xxxxxxxx-xxxx-Mxxx-Nxxx-xxxxxxxxxxxx

Execute playbook with a specific id

Call payload

None

Response

Will return 200/OK when finished with playbook playbook.

@startjson
{
    "execution-id": "xxxxxxxx-xxxx-Mxxx-Nxxx-xxxxxxxxxxxx",
    "playbook-id": "xxxxxxxx-xxxx-Mxxx-Nxxx-xxxxxxxxxxxx"
}
@endjson

Error

400/BAD REQUEST general error on error.

POST /trigger/playbook

Execute an ad-hoc playbook

Call payload

A playbook like cacao playbook JSON

Response

Will return 200/OK when finished with the playbook.

@startjson
{
    "execution-id": "xxxxxxxx-xxxx-Mxxx-Nxxx-xxxxxxxxxxxx",
    "playbook-id": "xxxxxxxx-xxxx-Mxxx-Nxxx-xxxxxxxxxxxx"
}
@endjson

Error

400/BAD REQUEST general error on error.

/step [NOT in SOARCA V1.0]

Get capable steps for SOARCA to allow a coa builder to generate or build valid coa’s

GET /step

Get all available steps for SOARCA.

Call payload

None

Response

200/OK

@startjson
{
    
    "steps": [{
        "module": "executor-module",
        "category" : "analyses",
        "context" : "external",
        "step--5ea28f63-ac32-4e5e-bd0c-757a50a3a0d7":{
                    "type": "single",
                    "name": "BI for CoAs",
                    "delay": 0,
                    "timeout": 30000,
                    "command": {
                        "type": "http-api",
                        "command": "hxxps://our.bi/key=VALUE"
                    },
                    "on_success": "step--71b15428-275a-49b5-9f09-3944972a0054",
                    "on_failure": "step--71b15428-275a-49b5-9f09-3944972a0054"
                }}]
}
@endjson

Module is the executing module name that will do the executer call.

Category defines what kind of step is executed:

@startuml
enum workflowType {
    analyses
    action
    asset-look-up
    etc...
}
@enduml

Context will define whether the call is internal or external:

@startuml
enum workflowType {
    internal
    external
}
@enduml

Error

400/BAD REQUEST general error on error.

/status

The status endpoints are used to get various statuses.

GET /status

Call this endpoint to see if SOARCA is up and ready. This call has no payload body.

Call payload

None

Response

200/OK

@startjson
{
    "version": "1.0.0",
    "runtime": "docker/windows/linux/macos/other",
    "mode" : "development/production",
    "time" : "2020-03-04T15:56:00.123456Z",
    "uptime": {
        "since": "2020-03-04T15:56:00.123456Z",
        "milis": "uptime in miliseconds"
    }
}
@endjson

Error

5XX/Internal error, 500/503/504 message.

GET /status/fins | not implemented

Call this endpoint to see if SOARCA Fins are up and ready. This call has no payload body.

Call payload

None

Response

200/OK

@startjson
{
    "fins": [
        {
            "name": "Fin name",
            "status": "ready/running/failed/stopped/...",
            "id": "The fin UUID",
            "version": "semver verison: 1.0.0"
        }
    ]
}
@endjson

Error

5XX/Internal error, 500/503/504 message.

GET /status/reporters | not implemented

Call this endpoint to see which SOARCA reportes are used. This call has no payload body.

Call payload

None

Response

200/OK

@startjson
{
    "reporters": [
        {
            "name": "Reporter name"
        }
    ]
}
@endjson

Error

5XX/Internal error, 500/503/504 message.

GET /status/ping

See if SOARCA is up this will only return if all SOARCA services are ready

Call payload

None

Response

200/OK

pong

Usage example flow

Stand alone

@startuml
participant "SWAGGER" as gui
control "SOARCA API" as api
control "controller" as controller
control "Executor" as exe
control "SSH-module" as ssh


gui -> api : /trigger/playbook with playbook body
api -> controller : execute playbook playload
controller -> exe : execute playbook
exe -> ssh : get url from log
exe <-- ssh : return result
controller <-- exe : results
api <-- controller: results

@enduml

Database load and execution

@startuml
participant "SWAGGER" as gui
control "SOARCA API" as api
control "controller" as controller
database "Mongo" as db
control "Executor" as exe
control "SSH-module" as ssh


gui -> api : /trigger/playbook/playbook--91220064-3c6f-4b58-99e9-196e64f9bde7
api -> controller : load playbook from database
controller -> db: retreive playbook
controller <-- db: playbook json
controller -> controller: validate playbook
controller -> exe : execute playbook
exe -> ssh : get url from log
exe <-- ssh : return result
controller <-- exe : results
api <-- controller: results

@enduml

3 - Reporter API Description

Endpoint description

We will use HTTP status codes https://en.wikipedia.org/wiki/List_of_HTTP_status_codes

@startuml
protocol Reporter {
    GET     /reporter
    GET     /reporter/{execution-id}
}
@enduml

/reporter

The reporter endpoint is used to fetch information about ongoing playbook executions in SOARCA

GET /reporter

Get all execution IDs of currently ongoing executions.

Call payload

None

Response

200/OK with payload:

@startjson
[
    {
        "executions": [
            {"execution_id" : "1", "playbook_id" : "a", "started" : "<timestamp>", "..." : "..."}, 
            "..."]
    }
]
@endjson

Error

400/BAD REQUEST with payload: General error

GET /reporter/{execution-id}

Get information about ongoing execution

Call payload

None

Response

Response data model:

Step execution data

Execution stataus

Table from Cyentific RNI workflow Status Vocabulary Name: execution-status-enum

If the execution has completed and no further steps need to be executed

200/OK with payload:

@startjson
[
    {
        "type" :        "execution-status",
        "id" :          "<execution-id>",
        "execution_id" : "<execution-id>",
        "playbook_id" :  "<playbook-id>",
        "started" :     "<time-string>",
        "ended" :       "<time-string>",
        "status" :      "<status-enum-value>",
        "status_text":  "<status description>",
        "errors" :      ["error1", "..."],
        "step_results" : {
            "<step-id-1>" : {
                "execution_id": "<execution-id>",
                "step_id" :     "<step-id>",
                "started" :     "<time-string>",
                "ended" :       "<time-string>",
                "status" :      "<status-enum-value>",
                "status_text":  "<status description>",
                "errors" :      ["error1", "..."],
                "variables":    {
                    "<variable-name-1>" : {
                        "type":         "<type>",
                        "name":         "<variable-name>",
                        "description":  "<description>",
                        "value":        "<value>",
                        "constant":     "<true/false>",
                        "external":     "<true/false>"
                    }
                }
            }
        },
        "request_interval" : "<n-seconds>"
    }
]
@endjson

The payload will include all information that the finished execution has created.


If the execution is still ongoing:

206/Partial Content
with payload equal to the 200 response, but impliclty not including all information from the execution, since the execution is still ongoing.

The step results object will list the steps that have been executed until the report request, and those that are being executed at the moment of the report request.

The "request_interval" suggests the polling interval for the next request (default suggested is 5 seconds).

#### Error
400/BAD REQUEST with payload:
General error

404/NOT FOUND
No execution with the specified ID was found.

4 - Decomposer

Decomposer structure

The decomposer will parse playbook objects to individual steps. This allows it to schedule new executor tasks.

Each incoming playbook will executed individually. Decomposing is done up to the step level.

struct ExecutionDetails{
    uuid executionId 
    uuid playbookId
}

Interface IDecomposer{
ExecutionDetails, error Execute(cacao playbook)
error getStatus(uuid playbookId)    
}
Interface IExecutor

class Controller
class Decomposer

IDecomposer <- Controller
IDecomposer <|.. Decomposer
IExecutor <- Decomposer

IExecutor

Interface for interfacing with the Executor this will in turn select and execute the command on the right module or fin.

Execution details

The struct contains the details of the execution (execution id which is created for every execution) and the playbook id. The combination of these is unique.

Decomposition of playbook

participant caller 
participant "Playbook decomposer" as decomposer
participant "Playbook state" as queue
participant Executor as exe



caller -> decomposer: Execute
caller <-- decomposer: ExecutionStatus

decomposer -> queue: store state
decomposer <-- queue:

loop for all playbook steps
    decomposer -> queue: load state
    decomposer <-- queue: 
    decomposer -> decomposer: parse step
    decomposer -> decomposer: parse command
   
    decomposer -> exe: execute command
    note over exe: correct executer is selected
    ... Time has passed ...
    decomposer <-- exe
end loop

5 - Executer

Components

The executor consists of the following components.

• Action executor
• Playbook action executor
• if-condition executor
• while-condition executor
• parallel executor

The decomposer interacts with every executor type. They all have separate interfaces to handle new step types in the future without changing the current interfaces.

package action{
    interface IExecutor {
        ..., err Execute(...)
    }
}

package playbookaction{
    interface IExecutor {
        ..., err Execute(...)
    }
}

package ifcondition{
    interface IExecutor {
        ..., err Execute(...)
    }
}

package whilecondition{
    interface IExecutor {
        ..., err Execute(...)
    }
}

package parallel{
    interface IExecutor {
        ..., err Execute(...)
    }
}


interface ICapability{
    variables, error Execute(Metadata, command, variable[], target, agent)
    string GetModuleName()
}

class "Decomposer" as decomposer
class "Action Executor" as Executor 
class "Playbook Executor" as playbook
class "Parallel Executor" as parallelexecutor
class "While Executor" as while
class "If condition Executor" as condition

class "Ssh" as ssh
class "OpenC2" as openc2
class "HttpApi" as api
class "Fin" as fin


action.IExecutor <|.. Executor
ICapability <-up- Executor
ICapability <|.. ssh
ICapability <|.. openc2
ICapability <|.. api
ICapability <|.. fin

playbookaction.IExecutor <|.. playbook
ifcondition.IExecutor <|.. condition
whilecondition.IExecutor <|.. while
parallel.IExecutor <|.. parallelexecutor

decomposer -down-> playbookaction.IExecutor
decomposer -down-> ifcondition.IExecutor
decomposer -down-> whilecondition.IExecutor
decomposer -down-> parallel.IExecutor
decomposer -down-> action.IExecutor

Action executor

The action executor consist of the following components

• The capability selector
• Native capabilities (command executors)
• MQTT capability to interact with: Fin capabilities (third-party executors)

The capability selector will select the implementation which is capable of executing the incoming command. There are native capabilities based on the CACAO command-type-ov:

• Currently implemented
  • ssh
  • http-api
  • openc2-http
  • powershell
• Coming soon
  • manual
• Future (potentially)
  • bash
  • caldera-cmd
  • elastic
  • jupyter
  • kestrel
  • sigma
  • yara

Native capabilities

The executor will select a module that is capable of executing the command and pass the details to it. The capability selection is performed based on the agent type (see Agent and Target Common Properties in the CACAO 2.0 spec). The convention is that the agent type must equal soarca-<capability identifier>, e.g. soarca-ssh or soarca-openc2-http.

The result of the step execution will be returned to the decomposer. A result can be either output variables or error status.

MQTT executor -> Fin capabilities

The Executor will put the command on the MQTT topic that is offered by the module. How a module handles this is described in the link:modules.adoc[module documentation]

Component overview

package "Controller" {
component Decomposer as parser

}
package "Executor" {
    component SSH as exe2
    component "HTTP-API" as exe1
    component MQTT as exe3
}

package "Fins" {
    component "VirusTotal" as virustotal
    component "E-mail Sender" as email
}

parser -- Executor
exe3 -- Fins : " MQTT topics"

Sequences

Example execution for SSH commands with SOARCA native capability.

@startuml

participant Decomposer as decomposer
participant "Capability selector" as selector
participant "SSH executor" as ssh

decomposer -> selector : Execute(...)
alt capability in SOARCA
    selector -> ssh : execute ssh command
    ssh -> ssh : 
    selector <-- ssh : results
    decomposer <-- selector : OnCompletionCallback
else capability not available 
    decomposer <-- selector : Execution failure
    note right: No capability can handle command \nor capability crashed etc..
end

Playbook action executor

The playbook executor handles execution of playbook action steps. The variables from the top level playbook are injected into the be executed playbook. It could happen that in the downstream playbook the variables collide with the top level playbook. In this case the top level playbook variables are NOT transferred to the downstream playbook. Agents and Targets cannot be transferred between playbooks at this time. Playbooks are only loaded in the executor and then a new Decomposer is created to execute the playbook.

The result of the step execution will be returned to the decomposer. A result can be either output variables or error status.

package playbookaction{
    interface IExecutor {
        Variables, err Execute(meta, step, variables)
    }
}
class "Decomposer" as decomposer
class "Action Executor" as exe 
interface "IPlaybookController" as controller
interface "IDatabaseController" as database
  
playbookaction.IExecutor <|.. exe
decomposer -> playbookaction.IExecutor
exe -> controller
database <- exe

If condition executor

The if-condition executor will process a cacao if-condition step and determine it’s output.

The result of the step comparison will be returned to the decomposer. A result can be either a next step id and/or error status.

While condition executor

The if-condition executor will process a cacao while-condition step and determine it’s output.

The result of the step comparison will be returned to the decomposer. A result can be either a next step id and/or error status. Only STIX comparison expressions are implemented at this time.

Parallel step executor

The parallel executor will execute the parallel step. This wil be done in sequence to simplify implementation. As parallel steps must not be depended on each other sequential execution is possible. Later this will be changed.

6 - Executer Modules

Executer modules are part of the SOARCA core. Executer modules perform the actual commands in CACAO playbook steps.

Native modules in SOARCA

The following capability modules are currently defined in SOARCA:

• ssh
• http-api
• openc2-http
• powershell

The capability will be selected based on the agent in the CACAO playbook step. The agent should be of type soarca and have a name corresponding to soarca-[capability name].

SSH capability

This capability executes SSH Commands on the specified targets.

This capability support User Authentication using the user-auth type. For SSH authentication username/password is authentication supported.

Success and failure

The SSH step is considered successful if a proper connection to each target can be initialized, the supplied command executes without error, and returns with zero exit status.

In every other circumstance the step is considered to have failed.

Variables

This module does not define specific variables as input, but variable interpolation is supported in the command and target definitions. It has the following output variables:

{
    "__soarca_ssh_result__": {
        "type": "string",
        "value": "<stdout of the last command>"
    }
}

Example

{
    "workflow": {
        "action--7777c6b6-e275-434e-9e0b-d68f72e691c1": {
            "type": "action",
            "agent": "soarca--00010001-1000-1000-a000-000100010001",
            "targets": ["linux--c7e6af1b-9e5a-4055-adeb-26b97e1c4db7"],
            "commands": [
                {
                    "type": "ssh",
                    "command": "ls -la"
                }
            ]
        }
    },
    "agent_definitions": {
        "soarca--00010001-1000-1000-a000-000100010001": {
            "type": "soarca",
            "name": "soarca-ssh"
        }
    },
    "target_definitions": {
        "linux--c7e6af1b-9e5a-4055-adeb-26b97e1c4db7": {
            "type": "linux",
            "name": "target",
            "address": { "ipv4": ["10.0.0.1"] }
        }
    }
}

HTTP-API capability

This capability implements the HTTP API Command.

Both HTTP Basic Authentication with user_id/password and token based OAuth2 Authentication are supported.

At this time, redirects are not supported.

Success and failure

The command is considered to have successfully completed if a successful HTTP response is returned from each target. An HTTP response is successful if it’s response code is in the range 200-299.

Variables

This capability supports variable interpolation in the command, port, authentication info, and target definitions.

The result of the step is stored in the following output variables:

{
    "__soarca_http_api_result__": {
        "type": "string",
        "value": "<http response body>"
    }
}

Example

{
    "workflow": {
        "action--8baa7c78-751b-4de9-81d4-775806cee0fb": {
            "type": "action",
            "agent": "soarca--00020001-1000-1000-a000-000100010001",
            "targets": ["http-api--4ebae9c3-9454-4e28-b25b-0f43cd97f9e0"],
            "commands": [
                {
                    "type": "http-api",
                    "command": "GET /overview HTTP/1.1",
                    "port": "8080"
                }
            ]
        }
    },
    "agent_definitions": {
        "soarca--00020001-1000-1000-a000-000100010001": {
            "type": "soarca",
            "name": "soarca-http-api"
        }
    },
    "target_definitions": {
        "http-api--4ebae9c3-9454-4e28-b25b-0f43cd97f9e0": {
            "type": "http-api",
            "name": "target",
            "address": { "dname": ["my.server.com"] }
        }
    }
}

OpenC2 capability

This capability implements the OpenC2 HTTP Command, by sending OpenC2 messages using the HTTPS transport method.

It supports the same authentication mechanisms as the HTTP-API capability.

Success and failure

Any successful HTTP response from an OpenC2 compliant endpoint (with a status code in the range 200-299) is considered a success. Connection failures and HTTP responses outside the 200-299 range are considered a failure.

Variables

It supports variable interpolation in the command, headers, and target definitions.

The result of the step is stored in the following output variables:

{
    "__soarca_openc2_http_result__": {
        "type": "string",
        "value": "<openc2-http response body>"
    }
}

Example

{
    "workflow": {
        "action--aa1470d8-57cc-4164-ae07-05745bef24f4": {
            "type": "action",
            "agent": "soarca--00030001-1000-1000-a000-000100010001",
            "targets": ["http-api--5a274b6d-dc65-41f7-987e-9717a7941876"],
            "commands": [{
                "type": "openc2-http",
                "command": "POST /openc2-api/ HTTP/1.1",
                "content_b64": "ewogICJoZWFkZXJzIjogewogICAgInJlcXVlc3RfaWQiOiAiZDFhYzA0ODktZWQ1MS00MzQ1LTkxNzUtZjMwNzhmMzBhZmU1IiwKICAgICJjcmVhdGVkIjogMTU0NTI1NzcwMDAwMCwKICAgICJmcm9tIjogInNvYXJjYS5ydW5uZXIubmV0IiwKICAgICJ0byI6IFsKICAgICAgImZpcmV3YWxsLmFwaS5jb20iCiAgICBdCiAgfSwKICAiYm9keSI6IHsKICAgICJvcGVuYzIiOiB7CiAgICAgICJyZXF1ZXN0IjogewogICAgICAgICJhY3Rpb24iOiAiZGVueSIsCiAgICAgICAgInRhcmdldCI6IHsKICAgICAgICAgICJmaWxlIjogewogICAgICAgICAgICAiaGFzaGVzIjogewogICAgICAgICAgICAgICJzaGEyNTYiOiAiMjJmZTcyYTM0ZjAwNmVhNjdkMjZiYjcwMDRlMmI2OTQxYjVjMzk1M2Q0M2FlN2VjMjRkNDFiMWE5MjhhNjk3MyIKICAgICAgICAgICAgfQogICAgICAgICAgfQogICAgICAgIH0KICAgICAgfQogICAgfQogIH0KfQ==",
                "headers": {
                    "Content-Type": ["application/openc2+json;version=1.0"]
                }
            }]
        }
    },
    "agent_definitions": {
        "soarca--00030001-1000-1000-a000-000100010001": {
            "type": "soarca",
            "name": "soarca-openc2-http"
        }
    },
    "target_definitions": {
        "http-api--5a274b6d-dc65-41f7-987e-9717a7941876": {
            "type": "http-api",
            "name": "openc2-compliant actuator",
            "address": { "ipv4": ["187.0.2.12"] }
        }
    }
}

PowerShell capability

This capability implements the PowerShell Command, by sending PowerShell commands using the WinRM transport method.

It supports the username, password authentication mechanism.

Success and failure

Any successful command will have a __soarca_powershell_result__. If an error occurs on the target a __soarca_powershell_error__ populated will be returned and Error will be set.

Variables

It supports variable interpolation in the command, headers, and target definitions.

The result of the step is stored in the following output variables:

{
    "__soarca_powershell_result__": {
        "type": "string",
        "value": "<raw powershell output>"
    }, 
    "__soarca_powershell_error__": {
        "type": "string",
        "value": "<raw powershell error output>"
    },
}

Example

{
    "workflow": {
        "action--aa1470d8-57cc-4164-ae07-05745bef24f4": {
            "type": "action",
            "agent": "soarca--00040001-1000-1000-a000-000100010001",
            "targets": ["net-address--d42d6731-791d-41af-8fa4-7b5699dfe402"],
            "commands": [{
                "type": "powershell",
                "command": "pwd"
            }]
        }
    },
    "agent_definitions": {
        "soarca--00040001-1000-1000-a000-000100010001": {
            "type": "soarca",
            "name": "soarca-powershell"
        }
    },
    "target_definitions": {
        "net-address--d42d6731-791d-41af-8fa4-7b5699dfe402": {
            "type": "net-address",
            "name": "Windows Server or Client with WinRM enabled",
            "address": { "ipv4": ["187.0.2.12"] }
        }
    }
}

MQTT fin module

This module is used by SOARCA to communicate with fins (capabilities) see fin documentation for more information

7 - Database

OARCA Database architecture, SOARCA makes use of MongoDB It is used to store and retrieve playbooks. Later it will also store individual steps.

Mongo

SOARCA employs separate collections in Mongo, utilizing a dedicated database object for each of them:

• Playbook
• Step

Interface IDatabase{
    void create(JsonData playbook)
    JsonData read(Id playbookId)
    void update(Id playbookId, JsonData playbook)
    void remove(Id playbookId)
}
Interface IPlaybookDatabase
Interface IStepDatabase

class Controller
class PlaybookDatabase

class Mongo


Controller -> IPlaybookDatabase
IStepDatabase <- Controller
IPlaybookDatabase <|.. PlaybookDatabase
IStepDatabase <|.. StepDatabase
StepDatabase -> IDatabase
IDatabase <- PlaybookDatabase

IDatabase <|.. Mongo

Getting data

Getting playbook data

participant Controller as controller
participant "Playbook Database" as playbook
database Database as db

controller -> playbook : get(id)
playbook -> db : read(playbookId)
note right
    When the create fails an error will be thrown
end note
playbook <-- db : "playbook JSON"
controller <-- playbook: "CacaoPlaybook Object"

Writing playbook data

participant Controller as controller
participant "Playbook Database" as playbook
database Database as db

controller -> playbook : set(CacaoPlaybook Object)
playbook -> db : create(playbook JSON)
note right
    When the create fails an error will be thrown
end note

Update playbook data

participant Controller as controller
participant "Playbook Database" as playbook
database Database as db

controller -> playbook : update(CacaoPlaybook Object)
playbook -> db : update(playbook id,playbook JSON)
note right
    When the create fails an error will be thrown
end note
playbook <-- db : true
controller <-- playbook: true

Delete playbook data

participant Controller as controller
participant "Playbook Database" as playbook
database Database as db

controller -> playbook : remove(playbook id)
playbook -> db : remove(playbook id)
note right
    When the create fails an error will be thrown
end note

Handling an error

participant Controller as controller
participant "Playbook Database" as playbook
database Database as db

controller -> playbook : remove(playbook id)
playbook -> db : remove(playbook id)
playbook <-- db: error 
note right
    playbook does not exists
end note
controller <-- playbook: error 

8 - Reporter

SOARCA utilizes push-based reporting to provide information on the instantiation of a CACAO workflow, and information on the execution of workflow steps.

General Reporting Architecture

For the execution of a playbook, a Decomposer and invoked Executors are injected with a Reporter. The Reporter maintains the reporting logic that reports execution information to a set of specified and available reporting targets.

A reporting target can be internal to SOARCA, such as a Cache. A reporting target can also be a third-party tool, such as an external SOAR/ SIEM, or incident case management system (see connectors).

Upon execution trigger for a playbook, information about the chain of playbook steps to be executed will be pushed to the targets via dedicated reporting classes.

Along the execution of the workflow steps, the reporting classes will dynamically update the steps execution information such as output variables, and step execution success or failure.

The reporting modules enables the population and updating of views and data concerning workflow composition and its dynamic execution results. This data can be transmitted to SOARCA internal reporting components such as a cache, as well as to third-party tools (see connectors).

The schema below represents the architecture concept.

@startuml
set separator ::

interface IStepReporter{
   ReportStepStart() error
   ReportStepEnd() error
}

interface IWorkflowReporter{
    ReportWorkflowStart() error
    ReportWorkflowEnd() error
}


interface IDownStreamReporter {
    ReportWorkflowStart() error
    ReportWorkflowEnd() error
	ReportStepStart() error
	ReportStepEnd() error
}

class Reporter {
    reporters []IDownStreamReporter
    RegisterReporters() error
    ReportWorkflowStart()
    ReportWorkflowEnd()
    ReportStepStart()
    ReportStepEnd()
}

class Cache
class TheHiveReporter
class 3PToolReporter

class Decomposer
class Executor

Decomposer -right-> IWorkflowReporter
Executor -left-> IStepReporter

Reporter .up.|> IStepReporter
Reporter .up.|> IWorkflowReporter
Reporter -right-> IDownStreamReporter

Cache .up.|> IDownStreamReporter
TheHiveReporter .up.|> IDownStreamReporter
3PToolReporter .up.|> IDownStreamReporter

Reporting is Asynchronous

Reporting functionalities are triggered asynchronously by means of go routines, such that reporting logic does not affect the execution timings of the CACAO playbooks. Note that this also implies that there might be small inconsistencies between the actual status of an execution, and what can be found in the reporting. For instance, reporting on the status of a step N, might take longer than reporting on the status of step N+1. Typically, the actual playbook execution may be slightly ahead of the reporting - and generally within a seconds-wide window. Execution timings, though, are always reported correctly, since we generated them within the workflow execution itself, and not within the reporting module.

The flow diagram below highlights the asynchronous mechanisms of reporting.

@startuml

participant "Decomposer" as decomposer
participant "Executor" as executor
participant "Reporter" as reporter
participant "DonwstreamReporter" as ds_reporter

decomposer -> reporter : ReportWorkflowStart()
reporter -->> ds_reporter : go ReportWorkflowStart()
decomposer -> executor : Execute step
activate executor
executor -> reporter : ReportStepStart()
reporter -->> ds_reporter : go ReportStepStart()
executor -> reporter : ReportStepEnd()
reporter -->> ds_reporter : go ReportStepEnd()
decomposer <- executor : return (variables, errors)
deactivate executor
decomposer -> reporter : ReportWorkflowEnd()
reporter -->> ds_reporter : go ReportWorkflowEnd()

@enduml

Note that the main reporter module is invoked synchronously. In turn, the main reporter module calls all downstream reporters, which implement the actual reporting logic, asynchronously. Also note any eventual reporting error never stops an execution, and only logs a warning.

Interfaces

The reporting logic and extensibility is implemented in the SOARCA architecture by means of reporting interfaces. At this stage, we implement an IWorkflowReporter to push information about the entire workflow to be executed, and an IStepReporter to push step-specific information as the steps of the workflow are executed.

A high level Reporter component will implement both interfaces, and maintain the list of DownStreamRepporters activated for the SOARCA instance. The Reporter class will invoke all reporting functions for each active reporter. The Executer and Decomposer components will be injected each with the Reporter though, as interface of respectively workflow reporter, and step reporter, to keep the reporting scope separated.

The DownStream reporters will implement push-based reporting functions specific for the reporting target, as shown in the IDownStreamReporter interface. Internal components to SOARCA, and third-party tool reporters, will thus implement the IDownStreamReporter interface.

Native Reporters

SOARCA implements internally reporting modules to handle database and caches reporting.

Cache reporter

The Cache reporter mediates between decomposer and executors, database, and reporting APIs. As DownStreamReporter, the Cache stores workflow and step reports in-memory for an ongoing execution. As IExecutionInformant, the Cache provides information to the reporting API. The schema below shows how it is positioned in the SOARCA architecture.

@startuml

protocol /reporter

interface IDownStreamReporter {
    ReportWorkflow() error
	ReportStep() error
}
interface IDatabase

interface IExecutionInformer

class ReporterApi
class Reporter
class Cache {
    cache []ExecutionEntry
}


"/reporter" -right-> ReporterApi

Reporter -> IDownStreamReporter
Cache -left-> IDatabase

Cache .up.|> IDownStreamReporter
Cache .up.|> IExecutionInformer
ReporterApi -down-> IExecutionInformer

The Cache thus reports the execution information downstream both in the database, and in memory. Upon execution information requests from the /reporter API, the cache can provide information fetching either from memory, or querying the database.

Connectors and 3P tools reporting

Reporting towards a 3rd Party tool is implemented in SOARCA by means of two components.

1. a 3PTool DownStreamReporter that organizes the execution information according to the data formats of the 3rd Party tool, and
2. a Connector module that receives data in 3PTool-specific formats, and handles the connectivity calls towards the 3P tool

At the current stage, the following 3rd Party tool DownStreamReporters and Connectors will be implemented in SOARCA.

TheHive

The TheHiveReporter reports on the execution of a playbook to an instance of The Hive platform. In order to be used, The Hive’s address and api key need to be configured in the environment variables of the SOARCA instance, either in the .env file for a source-code-built SOARCA instance, or in the SOARCA docker-compose for Docker builds. SOARCA will negotiate the authorization with the The Hive instance upon SOARCA initialization.

The TheHiveConnector receives execution information from the TheHiveReporter, and performs the actual network calls to report such infromation to the The Hive instance.

@startuml
set separator ::

interface IDownStreamReporter {
    ReportWorkflowStart() error
    ReportWorkflowEnd() error
	ReportStepStart() error
	ReportStepEnd() error
}

interface ITheHiveConnector {
}


class TheHiveReporter
class TheHiveConnector

TheHiveReporter .up.|> IDownStreamReporter
TheHiveReporter -right-> ITheHiveConnector
TheHiveConnector .up.|> ITheHiveConnector

Future plans

At this stage, third-party tools integrations may be built in SOARCA via packages implementing reporting logic for the specific tools. Alternatively, third-party tools may implement pull-based mechanisms (via the API) to get information from the execution of a playbook via SOARCA.

In the near future, we will (also) make available a SOARCA Report API that can establish a WebSocket connection to a third-party tool. As such, this will thus allow SOARCA to push execution updates as they come to third-party tools, without external tools having to poll SOARCA.

Furthermore, we will define more Connector donwstream reporters which will implement workflow execution reporting for specific platforms and products.

9 - Logging

SOARCA supports extensive logging. Logging is based on the logrus framework.

Format

Logging can be done in different formats suitable for your application. The following formats are available:

• JSON default
• Plain text

Destination

• std::out default (terminal)
• To file (to log file path)

Later:

• syslog (NOT YET IMPLEMENTED)

Log levels

SOARCA supports the following log levels. Also is indicated how they are used.

• PANIC (non fixable error system crash)
• FATAL (non fixable error, restart would fix)
• ERROR (operation went wrong but can be caught by other higher component)
• WARNING (let the user know some operation might not have the expected result but execution can continue on normal path)
• INFO default (let the user know that a major event has occurred)
• DEBUG (add some extra detail to normal execution paths)
• TRACE (get some fine grained detail from the logging)

Types of logging

SOARCA will log different information, these will be combined in the same output.

Runtime logging

Runtime logging wil include the running state of SOARCA, errors encountered when registering modules etc.

Security event logging

Will log the status of the execution of an playbook, database updates of playbooks

Using the logger (developer)

To use SOARCA logging you can add the following to your module.

type YourModule struct {
}

var component = reflect.TypeOf(YourModule{}).PkgPath()
var log *logger.Log

func init() {
	log = logger.Logger(component, logger.Info, "", logger.Json)
}

Changing log level

To change logging for your SOARCA instance you can use the following environment variables

This can be set as environment variables or loaded through the .env

LOG_GLOBAL_LEVEL: "info"
LOG_MODE: "production"
LOG_FILE_PATH: ""
LOG_FORMAT: "json"