DNSweeper – Asynchronous Public DNS Auditing Tool

Introduction

DNSweeper is DNS auditing tool which uses asynchronous libraries for querying and extracting DNS data from public resolvers around the globe. Written in Python, it’s designed to extend information about domains and subdomains (works best if you feed it with already scraped subdomains).

DNSweeper: Asynchronous Public DNS Auditing Tool

As Petr Javorik mentioned, the idea for this tool came from MassDNS (another DNS bruteforce tool). Initially he wanted just to select reliable public resolvers and feed them to MassDNS:

“You have to feed it with reliable public resolvers otherwise you will spend up significant time by cleaning up results. As time went by I got an idea that I could integrate results from vertical DNS enumeration into other scripting tools without using awk, sed, grep etc. So I made all results from DNSweeper available as json.”

As we’re all aware, some open resolvers (frequently related to Level3 DNS) provide results even when the domain is non-existent (a marketing trick to utilize your traffic). So, the main advantage of DNSweeper is that it doesn’t provide “false” positives:

“No subdomains pointing to empty A records are returned. Results from censoring resolvers and other “bad” resolvers are filtered out.

It resolves enumerated domain (only ONE) by TRUSTED resolvers. We get set of trusted A records. Now it queries all public resolvers for enumerated domain and extracts A records from each of them. Then it compares each public resolver’s result if returned A record(s) returned from particular public resolver belong to results from trusted resolvers. If so then public resolver is moved for next filtering phase. “

Filtering

“Next filtering phase consists of querying random subdomain for enumerated domain (njvwlsjdngbitkep.test-domain.com). If resolver returns valid A record then it’s discarded.”

With this approach Petr was able to extract ~10000 public resolvers for subsequent bruteforcing and other commands. In later testing phases he noticed that it was still not enough, bruteforce still gave few false positives. He solved it by final fine filtering:

“All extracted subdomains are finally queried by trusted resolvers only, if no A records returned then subdomain is discarded”

Filtering resolvers is ongoing research, and he’s very much interested in hearing other ideas on how to improve it, so feel free to contribute. Tool was tested on OS X and Linux, and as Javorik mentioned, Windows testers are more then welcome.

Two stage Brute-force command

Another neat feature of this tool is two stage bruteforce command, where DNSweeper first bruteforces given domain (with huge wordlist) and then it recursively bruteforces previously discovered subdomains (with smaller wordlist, defined with --bruteforce-recursive option). In case “--bruteforce-recursive FILE” flag is omitted, no recursive bruteforce is performed.

Petr also mentioned that it’s always better to feed DNSweeper with file that includes already scraped subdomains. Those subdomains can be hosted on IPs and Subnets that don’t have to be discovered via bruteforcing. By not including them we can miss some ASN records and related subnets. Reverse ASN lookup might not be effective with domains hosted on cloud services, but can otherwise provide substantial amount of subdomains. It’s definitely worth a try.

Install

Clone DNSweeper repo:

$ git clone https://github.com/MMquant/DNSweeper.git

Install python headers (needed for pycares):

$ apt install python3.7-dev

As for the requirements, you’ll need aiodns & requests. Install modules from requirements.txt:

$ pip3 install -r requirements.txt

DNSweeper Usage

Use --help to list available options:

python3.6 DNSweeper.py --help
 usage: DNSweeper.py [-h]
                     {enumerate,resolvers,bruteforce,forward_lookup,asn_reverse_lookup,update_resolvers}
                     …
 DNSsweeper - Asynchronous public DNS auditing tool
 optional arguments:
   -h, --help            show this help message and exit
 commands:
   {enumerate,resolvers,bruteforce,forward_lookup,asn_reverse_lookup,update_resolvers}
                         Description
     enumerate           Perform complete enumeration
     resolvers           Output filtered resolvers
     bruteforce          Bruteforce subdomains
     forward_lookup      Perform forward lookup
     asn_reverse_lookup  Perform ASN reverse lookup
     update_resolvers    Update raw resolver list

Sub-options:

python3.6 DNSweeper.py enumerate --help
 usage: DNSweeper.py enumerate [-h] (-f FILE | -d DOMAIN) [-p FILE] [-r REGEX]
                               [-o DIR_PATH] [--use-cache] [--fast-sweep]
                               [--no-bruteforce] [--bruteforce-recursive FILE]
                               [--exclude FILE] [-v]
 enumerate command prepares filtered public resolvers for given input domain.
 It performs bruteforce subdomain discovery. Then it forward-lookup each
 subdomain in all filtered resolvers and extracts corresponding ASN netblocks.
 Next it performs reverse-lookup in all IPs from discovered ASN netblocks and
 optionally filters output with given REGEX.
 optional arguments:
   -h, --help            show this help message and exit
   -f FILE               Path to file with (scraped) subdomains
   -d DOMAIN             Domain (ie. test_domain.com)
   -p FILE               Path to file with bruteforce payload
   -r REGEX              Reverse lookup regex matching pattern
   -o DIR_PATH           Path to directory with results
   --use-cache           Use cached resolvers
   --fast-sweep          Don't sweep all resolvers. For every subdomain return
                         just first valid resolver answer.
   --no-bruteforce       Don't use bruteforce command.
   --bruteforce-recursive FILE
                         Enable recursive bruteforce. Path to payload file must
                         be specified. Use smaller wordlists.
   --exclude FILE        File with subdomains which not to enumerate. (improves
                         speed)
   -v                    Verbosity, -v, -vv, -vvv

Result structure

The results/ directory is created in current working directory which contains files with given command output.

results/
 ├── asn_reverse_lookup_all_ptr.json
 ├── asn_reverse_lookup_asn.json
 ├── asn_reverse_lookup_regex_ptr.json
 ├── bruteforce_result.json
 ├── filtered_resolvers_result.json
 ├── enumerate_unique_subdomains.json
 ├── forward_lookup_result.json
 └── forward_lookup_unique_ips.json

Performance and Examples

Latest commits include some performance optimization tips & tricks which require root or admin privileges so every user should perform them manually before using DNSweeper.

Requests per seconds can be seen in stdout if -vv or -vvv verbosity is selected. There are basically just 2 speeds in DNSweeper ‘--fast-sweep‘ and not ‘--fast-sweep‘.

“Resolving speed has high variance and it strongly depends on ISP, NIC and overall network traffic. In my lab I can achieve 120-240 req/s in no –fast-sweep mode and 500-1200 req/s in –fast-sweep mode.”

We’ll try “newyorktimes.com” (dom_r1). Without any optimizations, on our test server (2 CPU, 8GB, 250Mbps):

$ time python3.6 DNSweeper.py enumerate -f dom_r1 -vv --no_bruteforce --use-cache

• Reverse lookup for 256 IPs / Chunk Size: 24 / Chunk Size List: 11
• Forward Lookup: 63 req/s
• ASN Reverse Lookup: between 5 and 130 req/s
• Total time: 49s

results/forward_lookup_result.json:

[
   {
     "name":"newyorktimes.com",
     "A":[
       "192.0.79.32",
       "192.0.79.33"
     ]
   }
 ]

results/forward_lookup_unique_ips.json:

[
   "192.0.79.32",
   "192.0.79.33"
]

results/enumerate_unique_subdomains.json

[]

results/asn_reverse_lookup_asn.json:

[
    {
      "AS":"2635",
      "IP":"192.0.79.33",
      "BGP Prefix":"192.0.79.0/24",
      "CC":"US",
      "Registry":"arin",
      "Allocated":"2012-11-20",
      "Info":"AUTOMATTIC - Automattic, Inc, US"
    }
]

Another quick test, “blic.rs”, robtex shows a vast number of subdomains (subdomains):

• baneri.blic.rs
• english.blic.rs
• gameplanet.blic.rs
• lillydrogerie.blic.rs
• mail.blic.rs
• sport.blic.rs
• trak-analytics.blic.rs
• vremeplov.blic.rs
• www.blic.rs
• zena.blic.rs

$ time python3.6 DNSweeper.py enumerate -f subdomains -vv --use-cache

• Subdomains payload: 100k [data/bitquark-subdomains-top100K.txt.ascii]
• Calculated chunk_size list: 4167
• Average speed: 6 req/s (8000 processed req / 20 min)

Optimization

In order to get DNSweeper working properly you should tune-up your OS so that it can handle thousands of outgoing TCP connections reliably. Usual system defaults are:

net.ipv4.ip_local_port_range = 32768   61000
net.ipv4.tcp_fin_timeout = 60

This basically means your system cannot consistently guarantee more than (61000 – 32768) / 60 = 470 sockets per second.

Reduce time to hold socket (reference) :

$ sysctl net.ipv4.tcp_fin_timeout=30

Increase local port range that is used by TCP and UDP to choose the local port:

$ sysctl net.ipv4.ip_local_port_range="15000 61000"

With that you should see over 1500 outbound connections per second.

Default Sysctl values on a typical Linux box for tcp_tw_reuse would be

net.ipv4.tcp_tw_reuse=0

This doesn not allow a connection from a “used” socket (in wait state) and force the sockets to last the complete time_wait cycle. Recommended:

sysctl net.ipv4.tcp_tw_reuse=1 

This allows fast cycling of sockets in time_wait state and re-using them.

Edit /etc/security/limits.conf file by adding

 <your_username>          soft        nofile      65535
 <your_username>          hard        nofile      65535

You might need restart your system after making these changes. To check if you can run enough concurrent TCP connections run

$ ulimit -Hn

which should be >=25000.

Conclusion

Reliability DNSweeper offers comes with a high cost: speed. It carefully works its way through DNS waters, so if you need to scan a large number of domains/subdomain fast, this probably isn’t the tool for you.