OSINT Collection and Reconnaissance Tool – SpiderFoot

Introduction

SpiderFoot is an OSINT (Open Source Intelligence) automation and reconnaissance tool, which goal is to automate the process of gathering intelligence about a given target (IP address, domain name, hostname, network subnet, ASN, etc.).

SpiderFoot: The Most Complete OSINT Collection and Reconnaissance Tool

SpiderFoot is an OSINT automation tool, which you can use for black-box pentesting to gather information about any target, such as: DNS, Whois, Web pages, passive DNS, spam blacklists, file meta data, threat intelligence lists as well as services like SHODAN, HaveIBeenPwned, etc, but you can also try this tools against your own network to see what information you are giving away.

Features:

• Open Source: it’s written in Python, and it’s hosted on Github. Since it’s open source, anyone can contribute and make this tool even better.
• Cross-platform: it can be run on both Linux and Windows operating systems.
• Web interface: you can run it from a CLI (Command Line Interface), but it also supports a web interface.
• Module support: it works including more than 100+ modules, so you can run almost any kind of test against the target network. SpiderFoot modules were programmed to interact with each other, allowing all related modules to share the same data about the target.
• Documentation: unlike other OSINT tools, it’s well-documented, which allows you to discover, read, and understand how everything works including installation process, usage, modules, etc.
• SpiderFoot HX: you can also choose to run it from its own self-hosted platform, which includes advanced features.

Modules:

• abuse.ch – Check if a host/domain, IP or netblock is malicious according to abuse.ch.
• AlienVault OTX – Obtain information from AlienVault Open Threat Exchange (OTX)
• Base64 – Identify Base64-encoded strings in any content and URLs, often revealing interesting hidden information.
• Blockchain – Queries blockchain.info to find the balance of identified bitcoin wallet addresses.
• cybercrime-tracker.net – Check if a host/domain or IP is malicious according to cybercrime-tracker.net.
• DNS Brute-force – Attempts to identify hostnames through brute-forcing common names.
• DNS Raw Records – Retrieves raw DNS records such as MX, TXT and others.
  
• Google Maps – Identifies potential physical addresses and latitude/longitude coordinates.
• HackerTarget.com – Search HackerTarget.com for hosts sharing the same IP.
  
• Honeypot Checker – Query the projecthoneypot.org database for entries.
  
• malwaredomainlist.com – Check if a host/domain, IP or netblock is malicious according to malwaredomainlist.com.
• and many many more.

To see full list of available module, visit SpiderFoot Docu Page.

Prerequisites:

• Linux/BSD/Solaris: Python 2.7, all from reqirements.txt (lxml, netaddr, M2Crypto, CherryPy, bs4, requests and Mako modules).
• Windows: All dependencies are packaged within executable file.

Install

Linux/BSD/Solaris

First, you’ll need to clone it from a GitHub repo:

$ git clone https://github.com/smicallef/spiderfoot.git

Navigate to the working directory and install prerequisites using pip:

$cd spiderfoot/
$ pip install -r requirements.txt

or:

$ pip install lxml netaddr M2Crypto cherrypy mako requests bs4

Now, you can run it as follows:

$ python ./sf.py

Install via Docker

SpiderFoot ships with a Dockerfile. First, clone SpiderFoot GitHub repo:

$ git clone https://github.com/smicallef/spiderfoot.git

Navigate to the working directory, and build the docker image with the following:

$ cd spiderfoot/
$ cat Dockerfile
$ docker build -t spiderfoot

In order to run it, we need to map a port on the server to run the container to that port using -p argument:

$ docker run -p 5009:5001 -d spiderfoot

With command above, we are mapping port 5009 on the local server to port 5001, which is the port SpiderFoot is listening on within the container.

To start using tool, run the following:

$ python ./sfcli.py -s http://localhost:5009

Usage

  _________      .__    .___          ___________            __
 /   _____/_____ |__| __| _/__________\_   _____/___   _____/  |_
 \_____  \\____ \|  |/ __ |/ __ \_  __ \    __)/  _ \ /  _ \   __\
 /        \  |_> >  / /_/ \  ___/|  | \/     \(  <_> |  <_> )  |
/_______  /   __/|__\____ |\___  >__|  \___  / \____/ \____/|__|
        \/|__|           \/    \/          \/
                Open Source Intelligence Automation.
                by Steve Micallef | @binarypool

[*] Version 2.12.
[*] Server http://127.0.0.1:5001 responding.
[*] Type 'help' or '?'.
sf>

To list all available commands, use help:

sf> help

Command          Description                              
----------------+-------------------------------------------------------
help [command]  | This help output.
debug           | Enable/Disable debug output.
clear           | Clear the screen.
history         | Enable/Disable/List command history.
spool           | Enable/Disable spooling output.
shell           | Execute a shell command.
exit            | Exit the SpiderFoot CLI (won't impact running scans).
ping            | Test connectivity to the SpiderFoot server.
modules         | List available modules.
types           | List available data types.
set             | Set variables and configuration settings.
scans           | List all scans that have been run or are running.
start           | Start a new scan.
stop            | Stop a scan.
delete          | Delete a scan.
scaninfo        | Scan information.
data            | Show data from a scan's results.
summary         | Scan result summary.
find            | Search for data within scan results.
query           | Run SQL against the SpiderFoot SQLite database.
logs            | View/watch logs from a scan.

---

---