Overview

TWIGS is an acronym for ThreatWatch Information Gathering Script. It is a python-based package that can be installed using “pip” i.e. python package manager. Twigs can help discover various classes of assets like cloud instances, servers, source code, containers and more. Also twigs can help identify hard-coded secrets in source code as well.

You can read more about twigs here.

Note though most examples of twigs commands in this guide are depicted on Linux platform, twigs supports Windows platform as well.

Installing twigs

Twigs is a python-based package and can be installed using ‘pip’ as below:

$ sudo pip install twigs

Initial setup of twigs

We recommend that you perform the following steps one-time after installing twigs to simplify usage of twigs subsequently:

  • Configure your environment setup (via something like .bashrc for Bash Shell) to store typical parameters required with twigs.
  • These parameters are as follows:
    • TW_HANDLE —> User login to be used for asset ingestion. Note this user becomes the owner of the ingested assets.
    • TW_TOKEN —> API token for the specific user. You can find the value by following below steps:
      1. Login into ThreatWatch console
      2. Click on “Profile” in top menu
      3. Click on “Key Management” in left menu
      4. If you have not generated an API Key as yet, then click on “Generate New Key”, else click on “Copy to clipboard”. Note if you re-generate a new API key, then earlier key is disabled by default.
    • TW_INSTANCE —> Specify the specific TW instance provisioned for your organization.
  • Save these as follows in your profile script (like .bashrc):

    export TW_HANDLE=<your login>

    export TW_TOKEN=<your API key>

    export TW_INSTANCE=<your TW instance>

  • Next time you login, these variables will be automatically set and will help reduce the clutter in your twigs command-line.

Current Platform/OS support for discovery

Twigs can discover various classes of assets as below:

  • Cloud assets from AWS, Azure and GCP
  • Docker containers
  • Import assets from a file (PDF or CSV)
  • Servers, desktops, laptops etc.
  • Source code repository
  • Ingest existing assets details from CMDB like ServiceNow, etc.

Twigs supports top platforms/OS as below:

  • RedHat
  • CentOS
  • Ubuntu
  • Debian
  • Amazon Linux
  • Windows

TWIGS Cloud Discovery

Twigs supports all 3 major cloud providers – AWS, Azure and GCP. This section covers discovery all these major cloud providers.

AWS

Overview

Twigs supports cloud-native discovery for AWS i.e. twigs can ingest asset inventory gathered by AWS Systems Manager.

Pre-requisites

One needs to configure AWS Systems Manager to report asset inventory which is subsequently ingested by twigs. For more details on how to configure AWS Systems Manager, please refer to the links below:

Steps involved

After you have configured AWS Systems Manager to gather inventory, then you can run twigs to ingest this collected inventory into your ThreatWatch instance by following the below mentioned steps:

  1. Open a new shell / terminal
  2. Check that twigs is installed and running properly by running below command:

twigs aws -h

  1. Keep following AWS details handy to run the command:

  • AWS Account Identifier (AWS_ACCOUNT)
  • AWS Access Key (AWS_ACCESS_KEY)
  • AWS Secret Key (AWS_SECRET_KEY)
  • AWS Region (AWS_REGION)
  • AWS S3 Bucket (AWS_S3_BUCKET)

  1. Run the command below:

      twigs aws --aws_account AWS_ACCOUNT

                --aws_access_key AWS_ACCESS_KEY

                --aws_secret_key AWS_SECRET_KEY

                --aws_region AWS_REGION

                --aws_s3_bucket AWS_S3_BUCKET

                [--enable_tracking_tags]

  1. It is suggested that you enable_tracking_tags, which allows you to easily identify AWS cloud instances in ThreatWatch
  2. Note AWS cloud discovery may require some time depending on the number of EC2 instances in your AWS cloud setup.
  3. After discovery is complete, you can login into ThreatWatch Console to view the newly discovered assets.

Azure

Overview

Twigs supports cloud-native discovery for Azure i.e. twigs can ingest asset inventory gathered by Azure in your Log Analytics Workspace.

Pre-requisites

Setting up a Azure Monitor in your Azure subscription requires some steps, you can refer to the documentation below:

Steps involved

  1. Open a new shell / terminal
  2. Check that twigs is installed and running properly by running below command:

After you have configured Azure Monitor to collect Azure VMs data in a Log Analytics Workspace, you can run twigs to ingest this collected inventory into your ThreatWatch instance by following the below mentioned steps:

	twigs azure -h

  1. You need the following information to run twigs command:

  • Azure Tenant Identifier (AZURE_TENANT_ID)
  • Azure Application Identifier (AZURE_APPLICATION_ID)
  • Azure Application Key (AZURE_APPLICATION_KEY)
  • Azure Subscription (AZURE_SUBSCRIPTION)
  • Azure Resource Group (AZURE_RESOURCE_GROUP)
  • Azure Log Analytics Workspace (AZURE_WORKSPACE)

  1. You can get these details from Azure Portal.
  2. If you do not know values for (AZURE_SUBSCRIPTION, AZURE_RESOURCE_GROUP, AZURE_WORKSPACE), then simply run twigs with no values for those and twigs will list out possible values (as shown below) by querying your Azure subscription. You can then select the right value.

$ twigs azure –azure_tenant_id “MY_TENANT_ID” –azure_application_id “MY_APPLICATION_ID” –azure_application_key “MY_APPLICATION_KEY”

INFO     Started new run…

INFO     Using handle specified in “TW_HANDLE” environment variable…

INFO     Using token specified in “TW_TOKEN” environment variable…

INFO     Using instance specified in “TW_INSTANCE” environment variable…

INFO     Getting access token…

Missing details for subscription/resource group/workspace….

Available subscriptions with resource group and workspace details as below:

Subscription: MY_SUBSCRIPTION

** Resource group: MY_RESOURCE_GROUP1

** Resource group: MY_RESOURCE_GROUP2

** Resource group: MY_RESOURCE_GROUP3

** Workspace: MY_LOG_ANALYTICS_WORKSPACE

Please re-run twigs with appropriate values for subscription, resource group and workspace.

  1. Run the command as shown below:

$ twigs azure –azure_tenant_id AZURE_TENANT_ID

–azure_application_id AZURE_APPLICATION_ID

–azure_application_key AZURE_APPLICATION_KEY

–azure_subscription AZURE_SUBSCRIPTION

–azure_resource_group AZURE_RESOURCE_GROUP

–azure_workspace AZURE_WORKSPACE

[—enable_tracking_tags]

  1. It is suggested that you enable_tracking_tags, which allows you to easily identify Azure cloud instances in ThreatWatch.
  2. Note Azure cloud discovery may require some time depending on the number of VM instances in your Azure cloud setup.
  3. After discovery is complete, you can login into ThreatWatch Console to view the newly discovered assets.

GCP

Overview

Google Cloud Platform (GCP) does not provide a good way to get complete and reliable inventory of VMs (unlike AWS and Azure). However, twigs provides a robust way to discover local and remotes hosts.

For GCP, ThreatWatch recommends the use of twigs in remote discovery mode for hosts and this section covers the approach using remote discovery. You will typically install twigs on the jump host or bastion host in your GCP environment for remote host discovery (since twigs needs to relay the asset information to your ThreatWatch instance).

If you would like to install twigs on all your GCP instances and then perform local host discovery on each of these, please refer here.

Pre-requisites

In the absence of a good inventory mechanism from GCP,  it is recommended that you use remote hosts discovery provided by twigs.

Please follow the steps mentioned here to create a CSV file with details of your GCP environment.

Steps involved

Once you have the CSV file ready, you can follow the steps mentioned here to discover all VMs in your GCP environment.

Host discovery

Overview

Twigs can discover hosts in two ways as below:

  • Discover the current host where is twigs is running. This is called as local host discovery and covered in more detail here.
  • Discover multiple hosts remotely. This is called as remote host discovery and covered in more detail her.

Local Host Discovery

Overview

Host discovery (local) is a fairly straightforward process. It needs twigs to be installed on the required host.

Pre-requisites

Twigs should be installed on the required host.

Steps involved

Once you have twigs installed on the required host, then you can follow the steps below for discovery local host as an asset in ThreatWatch:

  1. Open a new shell / terminal
  2. Check that twigs is installed and running properly by running below command:

	twigs host -h

  1. You can run the command as below:

	$ twigs host [--assetid ASSETID] [--assetname ASSETNAME]

  1. After discovery is complete, you can login into ThreatWatch Console to view the newly discovery asset.

Remote hosts discovery

Overview

Twigs can help discover multiple hosts easily using remote hosts discovery.

Pre-requisites

Twigs remote discovery for hosts uses a CSV (comma-separate values) file which provides details about the hosts to be discovered. The CSV format has support for specifying individual remote hosts via hostname or IP address and you can specify a CIDR (Classless Inter-Domain Routing) or subnet range to discover hosts in your GCP cloud. You can read more details about the format of the CSV file here.

It is recommended that you secure the credentials shared in the CSV file using the ‘—secure’ option provided by twigs. This can done by following the steps below:

  1. Assume that you have created remote_hosts.csv which contains credentials in clear text.
  2. Run the following command to secure the file:

$ twigs host --host_list remote_hosts.csv --secure

  1. Open the remote_hosts.csv file to confirm that the credentials are secured if you want.

Steps involved

You can follow the steps below for remote hosts discovery:

  1. Open a new shell / terminal
  2. Check that twigs is installed and running properly by running below command:

twigs host -h

  1. You need the following information to run twigs command:

  • remote_hosts.csv file created earlier as mentioned in pre-requisites section

  1. Run the twigs command as below:

	$ twigs host --remote_hosts_csv <<PATH_TO_REMOTE_HOSTS_CSV>> [--password PASSWORD]

  1. The discovery process may take some time depending on the number of hosts to be discovered.
  2. After discovery is complete, you can login into ThreatWatch Console to view the newly discovered assets.

Container discovery

Overview

Twigs supports discovering docker container images as an asset in ThreatWatch.

Pre-requisites

None.

Steps involved

You can follow the steps below to discover your docker container images/instances as assets in ThreatWatch:

  1. Open a new shell / terminal
  2. Check that twigs is installed and running properly by running below command:

twigs docker -h

  1. You can run the command below:

	$ twigs docker --image IMAGE [--assetid ASSETID] [--assetname ASSETNAME]

where IMAGE has format (repo:tag) and if tag is not specified, then “latest” is assumed.

  1. After discovery is complete, you can login into ThreatWatch console to view the newly discovered asset.

File-based discovery

Overview

File-based discovery mode in twigs allows you to ingest assets specified in the file to ThreatWatch. Currently supported file formats are as follows:

  • PDF (Portable Document Format)
  • CSV (Comma-Separated Value) – This is standard CSV format supported by ThreatWatch and is  as documented here.

Pre-requisites

You need to have asset information in either PDF or CSV format.

Steps involved

The steps involved to discover assets from PDF or CSV file are as below:

  1. Open a new shell / terminal.
  2. Check that twigs is installed and running properly by running below command:

	$ twigs file -h

  1. You can run the command as below:

	$ twigs file [-h] --in <<INPUT_FILE_OR_DIRECTORY>>

                      [--assetid ASSETID]

                      [--assetname ASSETNAME]

                      [--type {repo}]

Notes:

  1. You can specify single PDF or CSV file for ingestion or if you have multiple CSV files in a directory, then specify path to that directory.
  2. For single PDF file, you can specify optional parameters (asset ID , asset name and repository type) on the command-line. For CSV file(s), these values are used from the CSV file itself.

  1. After discovery is complete, you can login into ThreatWatch Console to view the newly discovery assets.

Nmap-based discovery

Overview

Twigs supports discovering assets from your environment using nmap.

Pre-requisites

You need to have nmap installed on your host (where you will be running twigs).

Steps involved

The steps involved to discover assets using nmap in your environment are as below:

  1. Open a new shell / terminal.
  2. Check that twigs is installed and running properly by running below command:

	$ twigs nmap -h

  1. You can run the command below:

	$ twigs nmap [-h] --hosts HOSTS

where HOSTS can be hostname, IP address or CIDR

  1. After discovery is complete, you can login into ThreatWatch Console to view the newly discovery assets.

Discover source code as an asset

Overview

Twigs can discover your source code as an asset. For a quick primer, read this.

Supported technologies for source code discovery are:

  • Python [pip]
  • Java Script [npm, yarn]
  • Ruby
  • Java [maven, gradle]
  • .NET/C# [nuget]
  • DLL (* for vulnerability assessment only)

There are multiple functionalities provided as below:

  • Identify vulnerabilities – This helps you identify any vulnerabilities in 3rd party libraries / packages used in your source code project. You can indicate to twigs whether you are interested in tracking vulnerabilities in direct (shallow level) or indirect (deep level) dependencies.
  • License compliance – You need to know how licensing of the open source components (libraries/packages) used in your software project impact you. For example – one cannot release a commercial software product built using open source components with a restrictive license
  • Code secrets – Twigs can be used to identify any secrets that are inadvertently embedded in your source code. Twigs can identify secrets using any or all of the three approach mentioned below:
    • Entropy – Detect secrets by automatically identifying high entropy strings in your source code.
    • Regular Expressions – twigs provides support for regular expressions for identifying standard secrets (like OAuth tokens, JWT tokens, etc.). You can specify your own custom regular expressions in a file if needed.
    • Common Passwords – twigs provides support for identifying common passwords from a top 500 common passwords list. You can provide your own common passwords file, if needed.

Pre-requisites

None.

Steps involved

The steps involved to discover your source code as an asset are as below:

  1. Open a new shell / terminal.
  2. Check that twigs is installed and running properly by running below command:

	$ twigs repo -h

  1. You can run the command as below:

	$ twigs repo  --repo REPO

               [--type {pip,ruby,yarn,nuget,npm,maven,gradle,dll}]

               [--level {shallow,deep}]

               [--assetid ASSETID]

               [--assetname ASSETNAME]

               [--secrets_scan]

               [--enable_entropy]

               [--regex_rules_file REGEX_RULES_FILE]

               [--check_common_passwords]

               [--common_passwords_file COMMON_PASSWORDS_FILE]

               [--include_patterns INCLUDE_PATTERNS]

               [--include_patterns_file INCLUDE_PATTERNS_FILE]

               [--exclude_patterns EXCLUDE_PATTERNS]

               [--exclude_patterns_file EXCLUDE_PATTERNS_FILE]

               [--mask_secret]

               [--no_code]

  1. After discovery is complete, you can login into ThreatWatch Console to view the newly discovery asset from your source code.

Discover assets from ServiceNow

Overview

You can point twigs to your ServiceNow (CMDB) to ingest existing asset information to ThreatWatch.

Pre-requisites

None.

Steps involved

You can follow the steps below to ingest your existing asset inventory from ServiceNow to ThreatWatch:

  1. Open a new shell / terminal.
  2. Check that twigs is installed and running properly by running below command:

	$ twigs servicenow -h

  1. Keep following ServiceNow instance details handy to run the command:

  • ServiceNow User (SNOW_USER)
  • ServiceNow User Password (SNOW_USER_PWD)
  • ServiceNow Instance (SNOW_INSTANCE)

  1. You can run the command below:

     $ twigs servicenow  --snow_user SNOW_USER

                         --snow_user_pwd SNOW_USER_PWD

                         --snow_instance SNOW_INSTANCE

                        [--enable_tracking_tags]

  1. It is suggested that you enable_tracking_tags, which allows you to easily identify assets ingested from ServiceNow in ThreatWatch
  2. After discovery is complete, you can login into ThreatWatch Console to view the newly discovery asset from your source code.

Appendix

Troubleshooting discovery issues

Here are some common troubleshooting tips for twigs:

  • Twigs generates a log file called twigs.log in the current working directory. You can look at this file for any error or warning messages.
  • Twigs by default exports all discovered assets to out.csv in the current working directory. You can view/edit this file if needed.
  • If the host running twigs has no connectivity to the internet, then twigs will be unable to automatically push the discovered assets to your ThreatWatch instance. Restore internet connectivity and then run twigs again. Note you can also import the generated CSV file if needed.

If you observe any issues with twigs, please write to us at support@threatwatch.io and we would be happy to help.

Twigs command-line usage

Please refer to twigs documentation for command-line usage

ThreatWatch CSV file format

The standard CSV format supported by ThreatWatch for assets is as described below:

  • There should be no column headers in the CSV file
  • Each line in the CSV file describes one asset
  • Format of each line is as follows (without square brackets): [asset_id], [asset_name], [asset_type], [product1], [product2] … [productN]
  • [asset_name] field is optional and can be empty.
  • [asset_type] can be one of the supported asset types, or ‘Other’ if asset type is not known.
  • Note the “productN” entries can be prefixed with special tags as mentioned below:
    • :OWNER: – This indicates that the value is the owner of the asset. For example – “:OWNER:demo@threatwatch.io
    • :TAG: – This indicates that the value is a tag to the applied to the asset. For example – “:TAG:Amazon Linux”

Twigs source code

Twigs is open source. If you are interested in going through the source code of twigs, you can find it here.

Leave a Reply

Your email address will not be published. Required fields are marked *