Getting started with ‘twigs’

by Ketan Nilangekar

Introduction

The bedrock of asset discovery in ThreatWatch is ‘twigs’ (short for ThreatWatch Inventory Gathering Script). twigs is a python based open source utility script maintained by ThreatWatch. twigs aims to provide a simple extensible interface to all types of assets and asset management systems in order to discover the metadata required by ThreatWatch to perform effective no-scan vulnerability management.

Please note that twigs is only designed to discover and inventory the metadata required for no-scan vulnerability assessments using ThreatWatch and it does not do any vulnerability assessment by itself. As of this writing, twigs can inventory hosts, AWS/Azure cloud assets, container images, open source dependencies from code repositories and ServiceNow CMDB. In the future, twigs will continue to extend to cover more types of assets and asset databases.

Requirements and installation

twigs can run on most Linux based systems (Redhat, CentOS, Ubuntu, Debian, Amazon Linux AMI) with Python 2.7. The github page for twigs is here and the latest online documentation is available here.

To install twigs simply run

$ sudo pip install twigs

on any suitable Linux system that needs to be discovered or which can be used to reach other systems that need to be discovered. Although twigs has very few python dependencies, you may also use python virtual environment to reduce dependency conflicts. A virtual environment for twigs for python 2.7 can be setup as follows:

$ python -m virtualenv --python=/usr/bin/python2.7 twigs_env_2_7

Basic CLI Options

The basic usage of twigs is as follows:

twigs [-h] [--handle HANDLE] [--token TOKEN] [--instance INSTANCE]
             [--out OUT] [--scan {quick,regular,full}] [--email_report]
             [--purge_assets]
             {aws,azure,servicenow,opensource,host,docker} ...

In most cases you will need the ‘–handle’ which is the ThreatWatch user/email, the ‘–token’ which is the API token issued for the user and the ‘–instance’ which is the name of the ThreatWatch instance where the discovery needs to be reported to. The API token can be obtained by logging into your ThreatWatch instance, navigating to ‘Profile->Key Management’ and generating an API token for yourself. In most cases ‘–instance’ will be the hostname of your dedicated ThreatWatch instance e.g. acme.threatwatch.io. ‘–instance’ defaults to threatwatch.io unless specified.

The ‘–handle’, ‘–token’ and ‘–instance’ can also be specified in the environment variables TW_HANDLE, TW_TOKEN and TW_INSTANCE respectively to keep the sensitive information from being recorded on command line history or being visible in process status as well as making the twigs CLI more convenient.

Discovery modes

twigs supports the following asset types/sources as modes for discovery:

host – Discover hosts locally or remotely using twigs as a lightweight agent for Red Hat, CentOS, Ubuntu, Debian, Amazon Linux AMI

aws – Use AWS inventory services to discover assets from an AWS cloud subscription. Additional steps are required to configure the AWS inventory service which are documented here.

azure – Use Azure inventory services to discover assets from an Azure cloud subscription. Additional steps are required to configure the Azure inventory service which are documented here.

docker – Discover and inventory container images locally or from a container repository.

opensource – Point twigs to your local/remote code repository/directory to inventory open source dependencies.

servicenow – Discover assets from an existing ServiceNow CMDB instance.

Each one of these modes may require specific inputs other than the basic options e.g. AWS subscription details, ServiceNow instance details etc. Specific help on this can be obtained by giving ‘-h’ after any mode in the CLI. For e.g.

$ twigs aws -h

Basic local host discovery

To do a simple local host discovery, install twigs on the Linux host of your choice (Red Hat, CentOS, Ubuntu, Debian, Amazon Linux AMI) and run the host discovery mode (replace handle, token and instance with appropriate values for you).

$ twigs --handle roadrunner@acme.com --token c8dddddd-eeee-eeee-eeee-aaca617649cc --instance acme.threatwatch.io host


You can schedule these twigs command using a suitable scheduler (cron) to have discovery run at a regular interval. This way twigs can be used as a lightweight agent for host discovery which can keep track of configuration drift on your host.

Trigger baseline scanning and reporting

twigs offers options to run a baseline scan for the discovered assets so you don’t have to login to the Web UI. The ‘–scan’ option provides for a ‘quick’, ‘regular’ or ‘full’ type of scan. For e.g.

$ twigs --handle roadrunner@acme.com --token c8dddddd-eeee-eeee-eeee-aaca617649cc --instance acme.threatwatch.io host --scan quick

You can also have the scan report emailed back to the ‘–handle’ email id once it is completed.

$ twigs --handle roadrunner@acme.com --token c8dddddd-eeee-eeee-eeee-aaca617649cc --instance acme.threatwatch.io host --scan quick --email_report

Security, privacy and anonymity of asset metadata

twigs offers a number of features to ensure the security of your asset inventory during discover and transit over the wire to your ThreatWatch instance. Because twigs is open source, its users can review and audit it to make sure it complies with their internal security toolset requirements. All communication between twigs and your ThreatWatch instance is outbound over secure http protocol only.

If you don’t want the asset inventory to be reported to a ThreatWatch instance but instead have it recorded locally in a CSV file, you can use the ‘–out’ option. This is useful in cases where users need to edit or anonymize the discovered assets before they are reported to ThreatWatch. In any case all assets discovered by twigs will be recorded in a file in the working directory named ‘out.csv’ even if ‘–out’ option is not given.

$ twigs --handle roadrunner@acme.com --out acmehost1.csv host

Also, if users want to edit, redact or anonymize the asset inventory they can do so using the ‘–out’ option before they upload it to their ThreatWatch instance. The anonymized asset CSV can then be uploaded to the ThreatWatch instance from the Web UI.

In addition to this, some discovery modes e.g. host, also allow overriding the asset ID or names using ‘–assetid’ and ‘–assetname’ options. For e.g.

$ twigs --handle roadrunner@acme.com --token c8dddddd-eeee-eeee-eeee-aaca617649cc --instance acme.threatwatch.io host --assetid myasset01 --assetname myasset

Finally, if users do not want the ThreatWatch instance to preserve the asset information beyond the baseline scan then the ‘–purge_assets’ option will remove the assets once the scan is complete and the report is emailed to the user. For e.g.

$ twigs --handle roadrunner@acme.com --token c8dddddd-eeee-eeee-eeee-aaca617649cc --instance acme.threatwatch.io host --scan quick --email_report --purge_assets

Windows discovery

The github repo for twigs has included a PowerShell script for discovering Windows hosts. This script can be run as a lightweight agent on the host itself or can be run remotely using PowerShell Remoting which is enabled by default on Windows Server 2012 systems. For more information check here.

The easy way to run windows_discovery.ps1 on a windows systems is,

C:\twigs\twigs\windows_discovery.ps1 --handle “roadrunner@acme.com” --token “c8dddddd-eeee-eeee-eeee-aaca617649cc” --instance acme.threatwatch.io

To run remotely,

Invoke-Command -ComputerName Server01, Server02 -FilePath C:\twigs\twigs\windows_discovery.ps1 --handle “roadrunner@acme.com” --token “c8dddddd-eeee-eeee-eeee-aaca617649cc” --instance acme.threatwatch.io

A future version of twigs will have windows discovery integrated with the twigs CLI itself.

 

Leave a Reply

Your email address will not be published. Required fields are marked *