Overview

TWIGS is an acronym for ThreatWatch Information Gathering Script. It is a python-based package that can be installed using “pip” i.e. python package manager. Twigs can help discover various classes of assets like cloud instances, servers, source code, containers and more. Also twigs can help identify hard-coded secrets in source code as well.

You can read more about twigs here.

Note though most examples of twigs commands in this guide are depicted on Linux platform, twigs supports Windows platform as well.

Installing twigs

Twigs is a python-based package and can be installed using ‘pip’ as below:

$ sudo pip install twigs

Initial setup of twigs

We recommend that you perform the following steps one-time after installing twigs to simplify usage of twigs subsequently:

  • Configure your environment setup (via something like .bashrc for Bash Shell) to store typical parameters required with twigs.
  • These parameters are as follows:
    • TW_HANDLE —> User login to be used for asset ingestion. Note this user becomes the owner of the ingested assets.
    • TW_TOKEN —> API token for the specific user. You can find the value by following below steps:
      1. Login into ThreatWatch console
      2. Click on “Profile” in top menu
      3. Click on “Key Management” in left menu
      4. If you have not generated an API Key as yet, then click on “Generate New Key”, else click on “Copy to clipboard”. Note if you re-generate a new API key, then earlier key is disabled by default.
    • TW_INSTANCE —> Specify the specific TW instance provisioned for your organization.
  • Save these as follows in your profile script (like .bashrc):

    export TW_HANDLE=<your login>

    export TW_TOKEN=<your API key>

    export TW_INSTANCE=<your TW instance>

  • Next time you login, these variables will be automatically set and will help reduce the clutter in your twigs command-line.

Current Platform/OS support for discovery

Twigs can discover various classes of assets as below:

  • Cloud assets from AWS, Azure and GCP
  • Docker containers
  • Import assets from a file (PDF or JSON)
  • Servers, desktops, laptops etc.
  • Source code repository
  • Ingest existing assets details from CMDB like ServiceNow, etc.

Twigs supports top platforms/OS as below:

  • RedHat
  • CentOS
  • Ubuntu
  • Debian
  • Amazon Linux
  • Windows

TWIGS common options

Twigs provides a bunch of common options i.e. irrespective of what type of assets are being discovered. These common options are optional as seen below:

twigs [-h] [-v] [–handle HANDLE] [–token TOKEN] [–instance INSTANCE]

             [–tag_critical] [–tag TAG] [–apply_policy APPLY_POLICY]

             [–out OUT] [–no_scan] [–email_report] [–quiet]

Here is a quick description of these options:

  • tag_critical – Mark the asset(s) as business critical
  • tag – You can use this option multiple times on the command line to add multiple tags to the asset(s)
  • apply_policy – You can use this option to specify a policy JSON file when you use twigs in your CI/CD pipeline to make policy based decisions like fail the build if any “DoNow” priority vulnerability impacts are discovered or any strong copyleft violations are found
  • out – Use this option to specify path to a JSON file to save the asset(s)
  • no_scan – Indicate that you don’t wish to start a vulnerability assessment for the discovered asset(s)
  • email_report – Once the vulnerability assessment is done, then you will automatically be emailed a copy of the vulnerability assessment report.
  • quiet – Do not display informational messages on the console during twigs run

TWIGS Cloud Discovery

Twigs supports all 3 major cloud providers – AWS, Azure and GCP. This section covers discovery all these major cloud providers.

AWS

Overview

Twigs supports cloud-native discovery for AWS i.e. twigs can ingest asset inventory gathered by AWS Systems Manager.

Pre-requisites

One needs to configure AWS Systems Manager to report asset inventory which is subsequently ingested by twigs. For more details on how to configure AWS Systems Manager, please refer to the links below:

Steps involved

After you have configured AWS Systems Manager to gather inventory, then you can run twigs to ingest this collected inventory into your ThreatWatch instance by following the below mentioned steps:

  1. Open a new shell / terminal
  2. Check that twigs is installed and running properly by running below command:

twigs aws -h

  1. Keep following AWS details handy to run the command:
    • AWS Account Identifier (AWS_ACCOUNT)
    • AWS Access Key (AWS_ACCESS_KEY)
    • AWS Secret Key (AWS_SECRET_KEY)
    • AWS Region (AWS_REGION)
    • AWS S3 Bucket (AWS_S3_BUCKET)
  1. Run the command below:

      twigs aws –aws_account AWS_ACCOUNT

                –aws_access_key AWS_ACCESS_KEY

                –aws_secret_key AWS_SECRET_KEY

                –aws_region AWS_REGION

                –aws_s3_bucket AWS_S3_BUCKET

                [–enable_tracking_tags]

  1. It is suggested that you enable_tracking_tags, which allows you to easily identify AWS cloud instances in ThreatWatch
  2. Note AWS cloud discovery may require some time depending on the number of EC2 instances in your AWS cloud setup.
  3. After discovery is complete, you can login into ThreatWatch Console to view the newly discovered assets.

Azure

Overview

Twigs supports cloud-native discovery for Azure i.e. twigs can ingest asset inventory gathered by Azure in your Log Analytics Workspace.

Pre-requisites

Setting up a Azure Monitor in your Azure subscription requires some steps, you can refer to the documentation below:

Steps involved

After you have configured Azure Monitor to collect Azure VMs data in a Log Analytics Workspace, you can run twigs to ingest this collected inventory into your ThreatWatch instance by following the below mentioned steps:

  1. Open a new shell / terminal
  2. Check that twigs is installed and running properly by running below command:

twigs azure -h

  1. You need the following information to run twigs command:
    • Azure Tenant Identifier (AZURE_TENANT_ID)
    • Azure Application Identifier (AZURE_APPLICATION_ID)
    • Azure Application Key (AZURE_APPLICATION_KEY)
    • Azure Subscription (AZURE_SUBSCRIPTION)
    • Azure Resource Group (AZURE_RESOURCE_GROUP)
    • Azure Log Analytics Workspace (AZURE_WORKSPACE)
  1. You can get these details from Azure Portal.
  2. If you do not know values for (AZURE_SUBSCRIPTION, AZURE_RESOURCE_GROUP, AZURE_WORKSPACE), then simply run twigs with no values for those and twigs will list out possible values (as shown below) by querying your Azure subscription. You can then select the right value.

$ twigs azure –azure_tenant_id “MY_TENANT_ID” –azure_application_id “MY_APPLICATION_ID” –azure_application_key “MY_APPLICATION_KEY”

INFO     Started new run…

INFO     Using handle specified in “TW_HANDLE” environment variable…

INFO     Using token specified in “TW_TOKEN” environment variable…

INFO     Using instance specified in “TW_INSTANCE” environment variable…

INFO     Getting access token…

Missing details for subscription/resource group/workspace….

Available subscriptions with resource group and workspace details as below:

Subscription: MY_SUBSCRIPTION

** Resource group: MY_RESOURCE_GROUP1

** Resource group: MY_RESOURCE_GROUP2

** Resource group: MY_RESOURCE_GROUP3

** Workspace: MY_LOG_ANALYTICS_WORKSPACE

Please re-run twigs with appropriate values for subscription, resource group and workspace.

  1. Run the command as shown below:

$ twigs azure –azure_tenant_id AZURE_TENANT_ID

                   –azure_application_id AZURE_APPLICATION_ID

                   –azure_application_key AZURE_APPLICATION_KEY

                   –azure_subscription AZURE_SUBSCRIPTION

                   –azure_resource_group AZURE_RESOURCE_GROUP

                   –azure_workspace AZURE_WORKSPACE

                   [—enable_tracking_tags]

  1. It is suggested that you enable_tracking_tags, which allows you to easily identify Azure cloud instances in ThreatWatch.
  2. Note Azure cloud discovery may require some time depending on the number of VM instances in your Azure cloud setup.
  3. After discovery is complete, you can login into ThreatWatch Console to view the newly discovered assets.

GCP

Overview

Google Cloud Platform (GCP) does not provide a good way to get complete and reliable inventory of VMs (unlike AWS and Azure). However, twigs provides a robust way to discover local and remotes hosts.

For GCP, ThreatWatch recommends the use of twigs in remote discovery mode for hosts and this section covers the approach using remote discovery. You will typically install twigs on the jump host or bastion host in your GCP environment for remote host discovery (since twigs needs to relay the asset information to your ThreatWatch instance).

If you would like to install twigs on all your GCP instances and then perform local host discovery on each of these, please refer here.

Pre-requisites

In the absence of a good inventory mechanism from GCP,  it is recommended that you use remote hosts discovery provided by twigs.

Please follow the steps mentioned here to create a CSV file with details of your GCP environment.

Steps involved

Once you have the CSV file ready, you can follow the steps mentioned here to discover all VMs in your GCP environment.

Host discovery

Overview

Twigs can discover hosts in two ways as below:

  • Discover the current host where is twigs is running. This is called as local host discovery and covered in more detail here.
  • Discover multiple hosts remotely. This is called as remote host discovery and covered in more detail her.

Local Host Discovery

Overview

Host discovery (local) is a fairly straightforward process. It needs twigs to be installed on the required host.

Pre-requisites

Twigs should be installed on the required host.

Steps involved

Once you have twigs installed on the required host, then you can follow the steps below for discovery local host as an asset in ThreatWatch:

  1. Open a new shell / terminal
  2. Check that twigs is installed and running properly by running below command:

twigs host -h

  1. You can run the command as below:

$ twigs host [–assetid ASSETID] [–assetname ASSETNAME]

  1. After discovery is complete, you can login into ThreatWatch Console to view the newly discovery asset.

Remote hosts discovery

Overview

Twigs can help discover multiple hosts easily using remote hosts discovery.

Pre-requisites

Twigs remote discovery for hosts uses a CSV (comma-separate values) file which provides details about the hosts to be discovered. The CSV format has support for specifying individual remote hosts via hostname or IP address and you can specify a CIDR (Classless Inter-Domain Routing) or subnet range to discover hosts in your GCP cloud. You can read more details about the format of the CSV file here.

It is recommended that you secure the credentials shared in the CSV file using the ‘—secure’ option provided by twigs. This can done by following the steps below:

  1. Assume that you have created remote_hosts.csv which contains credentials in clear text.
  2. Run the following command to secure the file:

$ twigs host –host_list remote_hosts.csv –secure

  1. Open the remote_hosts.csv file to confirm that the credentials are secured if you want.

Steps involved

You can follow the steps below for remote hosts discovery:

  1. Open a new shell / terminal
  2. Check that twigs is installed and running properly by running below command:

twigs host -h

  1. You need the following information to run twigs command:
    • remote_hosts.csv file created earlier as mentioned in pre-requisites section
  1. Run the twigs command as below:

$ twigs host –remote_hosts_csv <<PATH_TO_REMOTE_HOSTS_CSV>> [–password PASSWORD]

  1. The discovery process may take some time depending on the number of hosts to be discovered.
  2. After discovery is complete, you can login into ThreatWatch Console to view the newly discovered assets.

Container discovery

Overview

Twigs supports discovering docker container images as an asset in ThreatWatch.

Pre-requisites

None.

Steps involved

You can follow the steps below to discover your docker container images/instances as assets in ThreatWatch:

  1. Open a new shell / terminal
  2. Check that twigs is installed and running properly by running below command:

twigs docker -h

  1. You can run the command below:

$ twigs docker –image IMAGE [–assetid ASSETID] [–assetname ASSETNAME]

where IMAGE has format (repo:tag) and if tag is not specified, then “latest” is assumed.

  1. After discovery is complete, you can login into ThreatWatch console to view the newly discovered asset.

File-based discovery

Overview

File-based discovery mode in twigs allows you to ingest assets specified in the file to ThreatWatch. Currently supported file formats are as follows:

  • PDF (Portable Document Format)
  • JSON (JavaScript Object Notation) – Assets are represented in standard JSON format. Note the “–out” switch in twigs command-line outputs the asset as a JSON file. The same file can be passed for discovery subsequently.

Pre-requisites

You need to have asset information in either PDF or JSON format.

Steps involved

The steps involved to discover assets from PDF or JSON file are as below:

  1. Open a new shell / terminal.
  2. Check that twigs is installed and running properly by running below command:

$ twigs file -h

  1. You can run the command as below:

$ twigs file [-h] –in <<INPUT_FILE_OR_DIRECTORY>>

                      [–assetid ASSETID]

                      [–assetname ASSETNAME]

                      [–type {repo}]

Notes:

    1. You can specify single PDF or JSON file for ingestion or if you have multiple JSON files in a directory, then specify path to that directory.
    2. For single PDF file, you can specify optional parameters (asset ID , asset name and repository type) on the command-line. For JSON file(s), these values are used from the JSON file itself.
  1. After discovery is complete, you can login into ThreatWatch Console to view the newly discovery assets.

nmap-based discovery

Overview

Twigs supports discovering assets from your environment using nmap.

Pre-requisites

You need to have nmap installed on your host (where you will be running twigs).

Steps involved

The steps involved to discover assets using nmap in your environment are as below:

  1. Open a new shell / terminal.
  2. Check that twigs is installed and running properly by running below command:

$ twigs nmap -h

  1. You can run the command below:

$ twigs nmap [-h] –hosts HOSTS

where HOSTS can be hostname, IP address or CIDR

  1. After discovery is complete, you can login into ThreatWatch Console to view the newly discovery assets.

Discover source code as an asset

Overview

Twigs can discover your source code as an asset. For a quick primer, read this.

Supported technologies for source code discovery are:

  • Python [pip]
  • Java Script [npm, yarn]
  • Ruby
  • Java [maven, gradle]
  • .NET/C# [nuget]
  • DLL (* for vulnerability assessment only)

There are multiple functionalities provided as below:

  • Identify vulnerabilities – This helps you identify any vulnerabilities in 3rd party libraries / packages used in your source code project. You can indicate to twigs whether you are interested in tracking vulnerabilities in direct (shallow level) or indirect (deep level) dependencies.
  • License compliance – You need to know how licensing of the open source components (libraries/packages) used in your software project impact you. For example – one cannot release a commercial software product built using open source components with a restrictive license
  • Code secrets – Twigs can be used to identify any secrets that are inadvertently embedded in your source code. Twigs can identify secrets using any or all of the three approach mentioned below:
    • Entropy – Detect secrets by automatically identifying high entropy strings in your source code.
    • Regular Expressions – twigs provides support for regular expressions for identifying standard secrets (like OAuth tokens, JWT tokens, etc.). You can specify your own custom regular expressions in a file if needed.
    • Common Passwords – twigs provides support for identifying common passwords from a top 500 common passwords list. You can provide your own common passwords file, if needed.

Pre-requisites

None.

Steps involved

The steps involved to discover your source code as an asset are as below:

  1. Open a new shell / terminal.
  2. Check that twigs is installed and running properly by running below command:

$ twigs repo -h

  1. You can run the command as below:

$ twigs repo  –repo REPO

               [–type {pip,ruby,yarn,nuget,npm,maven,gradle,dll}]

               [–level {shallow,deep}]

               [–assetid ASSETID]

               [–assetname ASSETNAME]

               [–secrets_scan]

               [–enable_entropy]

               [–regex_rules_file REGEX_RULES_FILE]

               [–check_common_passwords]

               [–common_passwords_file COMMON_PASSWORDS_FILE]

               [–include_patterns INCLUDE_PATTERNS]

               [–include_patterns_file INCLUDE_PATTERNS_FILE]

               [–exclude_patterns EXCLUDE_PATTERNS]

               [–exclude_patterns_file EXCLUDE_PATTERNS_FILE]

               [–mask_secret]

               [–no_code]

  1. After discovery is complete, you can login into ThreatWatch Console to view the newly discovery asset from your source code.

Discover assets from ServiceNow

Overview

You can point twigs to your ServiceNow (CMDB) to ingest existing asset information to ThreatWatch.

Pre-requisites

None.

Steps involved

You can follow the steps below to ingest your existing asset inventory from ServiceNow to ThreatWatch:

  1. Open a new shell / terminal.
  2. Check that twigs is installed and running properly by running below command:

$ twigs servicenow -h

  1. Keep following ServiceNow instance details handy to run the command:
    • ServiceNow User (SNOW_USER)
    • ServiceNow User Password (SNOW_USER_PWD)
    • ServiceNow Instance (SNOW_INSTANCE)
  1. You can run the command below:

     $ twigs servicenow  –snow_user SNOW_USER

                         –snow_user_pwd SNOW_USER_PWD

                         –snow_instance SNOW_INSTANCE

                        [–enable_tracking_tags]

  1. It is suggested that you enable_tracking_tags, which allows you to easily identify assets ingested from ServiceNow in ThreatWatch
  2. After discovery is complete, you can login into ThreatWatch Console to view the newly discovery asset from your source code.

DAST checks on web applications as assets

Overview

Using this feature you can run DAST checks on your web applications and track them in ThreatWatch. This allows you to track your web applications as assets in ThreatWatch

Pre-requisites

A supported DAST tool like skipfish, arachni etc. is required. This needs to be installed on the host where twigs is running.

Steps involved

You can follow the steps below to discover your web application as an asset and run DAST checks on it

  1. Open a new shell / terminal.
  2. Check that twigs is installed and running properly by running below command:

$ twigs dast -h

  1. Make sure you have a supported DAST plugin installed. Supported tool currently is skipfish. You can download and install skipfish from https://code.google.com/archive/p/skipfish/
  2. Note your web application url and make sure it is accessible from the host where twigs is running.
  3. You can run the command below:

     $ twigs dast  –url WEB_APPLICATION_URL

  –assetid UNIQUE_ASSET_ID

                        [–args EXTRA_ARGS_FOR_SKIPFISH ]

                        [–assetname NAME_LABEL_FOR_ASSET ]

  1. After discovery is complete, you can login into ThreatWatch Console to view the newly discovery web application asset as well as results of the DAST tests.
  2. These tests could be automated as part of a CI/CD pipeline to get your web application regularly tested.
  3. Twigs will automatically mark/resolve any fixed issues that were discovered as part of a previous run.

Run and track docker-bench results

Overview

Using this feature you can run docker bench CIS checks on your docker hosts and containers and track them in ThreatWatch. This allows you to track your docker host as a regular linux host as well as running docker-bench tests on it.

Pre-requisites

docker-bench is required. This can be downloaded from https://github.com/docker/docker-bench-security.

Steps involved

  1. Open a new shell / terminal.
  2. Check that twigs is installed and running properly by running below command:

$ twigs docker_cis -h

  1. Make sure you have docker-bench downloaded
  2. Note the location of docker-bench-security.sh which is available in the downloaded copy of docker-bench-security
  3. You can run the command below:

     $ twigs docker_cis  [–assetid UNIQUE_ASSET_ID]

                       [–assetname NAME_LABEL_FOR_ASSET ]

  1. Asset id and name are optional.
  2. After discovery is complete, you can login into ThreatWatch Console to view the newly discovery host as well as results of the docker bench tests.
  3. Twigs will automatically mark/resolve any fixed issues that were discovered as part of a previous run.

Run and track CIS benchmarks for AWS

Overview

Using this feature you can run CIS benchmark tests (v1.2.0) for your AWS cloud. This includes CIS level 1 and level 2 checks for AWS as specified here: https://www.cisecurity.org/benchmark/amazon_web_services/

Pre-requisites

Prowler is a tool that allows you to run CIS benchmarks for AWS. This tool is free and can be downloaded from https://github.com/toniblyx/prowlerPlease install any dependencies required by prowler tool.

Steps involved

  1. Open a new shell / terminal.
  2. Check that twigs is installed and running properly by running below command:

$ twigs aws_cis -h

  1. Make sure you have Prowler downloaded
  2. Note the location of prowler executable which is available in the downloaded copy of prowler.
  3. You can run the command below:

     $ twigs aws_cis  –aws_access_key AWS_ACCESS_KEY

                      –aws_secret_key AWS_SECRET_KEY

                      –assetid UNIQUE_ASSET_ID

                      –assetname NAME_LABEL_FOR_ASSET

                      –prowler_home HOME_DIRECTORY_FOR_PROWLER

  1. Asset id is not optional. Use a unique identifier for your AWS cloud instance as an asset.
  2. After discovery is complete, you can login into ThreatWatch Console to view the newly discovered AWS instance as an asset as well as results of the CIS benchmark tests.
  3. Twigs will automatically mark/resolve any fixed issues that were discovered as part of a previous run.

Run and track CIS benchmarks for Azure

Overview

Using this feature you can run CIS benchmark tests (v1.0.0) for your Azure cloud. This includes CIS level 1 and level 2 checks for Azure as specified here: https://www.cisecurity.org/benchmark/azure/

Pre-requisites

Azure CLI is required, please install it by following the mentioned here for your Operating System.

Steps involved

  1. Open a new shell / terminal.
  2. Check that twigs is installed and running properly by running below command:

$ twigs azure_cis -h

  1. Sign in into your Azure instance using Azure CLI as described here on the box where you will be running twigs.
  2. You can run the command below:

     $ twigs azure_cis –assetid UNIQUE_ASSET_ID

                       –assetname NAME_LABEL_FOR_ASSET

  1. Asset id is not optional. Use a unique identifier for your Azure cloud instance as an asset.
  2. After discovery is complete, you can login into ThreatWatch Console to view the newly discovered Azure instance as an asset as well as results of the CIS benchmark tests.
  3. Twigs will automatically mark/resolve any fixed issues that were discovered as part of a previous run.

Run and track CIS benchmarks for Google Cloud Platform

Overview

Using this feature you can run CIS benchmark tests (v1.1.0) for your Google Cloud Platform. This includes CIS level 1 and level 2 checks for Google Cloud Platform as specified here: https://www.cisecurity.org/benchmark/google_cloud_computing_platform/.

Pre-requisites

Google Cloud SDK is required, please install it by following instructions mentioned here for your Operating System. The SDK provides tools (like gcloud, gsutil, bq, etc.) which are used.

Steps involved

  1. Open a new shell / terminal.
  2. Check that twigs is installed and running properly by running below command:

$ twigs gcp_cis -h

  1. Sign in into your Google Cloud Platform instance using gcloud CLI as described here on the box where you will be running twigs.
  2. You can run the command below:

     $ twigs gcp_cis –assetid UNIQUE_ASSET_ID

                     –assetname NAME_LABEL_FOR_ASSET

  1. Asset id is not optional. Use a unique identifier for your Google Cloud Platform instance as an asset.
  2. After discovery is complete, you can login into ThreatWatch Console to view the newly discovered Google Cloud Platform instance as an asset as well as results of the CIS benchmark tests.
  3. Twigs will automatically mark/resolve any fixed issues that were discovered as part of a previous run.

Appendix

Troubleshooting discovery issues

Here are some common troubleshooting tips for twigs:

  • Twigs generates a log file called twigs.log in the current working directory. You can look at this file for any error or warning messages.
  • Twigs can export all discovered assets to specified JSON file using “–out” switch. You can view/edit this file if needed.
  • If the host running twigs has no connectivity to the internet, then twigs will be unable to automatically push the discovered assets to your ThreatWatch instance. Restore internet connectivity and then run twigs again. Note you can also import the generated JSON file if needed.

If you observe any issues with twigs, please write to us at support@threatwatch.io and we would be happy to help.

Twigs command-line usage

Please refer to twigs documentation for command-line usage

Twigs source code

Twigs is open source. If you are interested in going through the source code of twigs, you can find it here.

Leave a Reply

Your email address will not be published. Required fields are marked *