DNS Host Enumeration Tool Bake-Off

[UPDATED – 10/2013 with blacksheepwall content]

The Scenario

In my line-o-work, we need to not only discover hosts on the network to assess but also the domain names of the websites hosted on those servers. See, there are these things called “Virtual Hosts” (http://httpd.apache.org/docs/2.2/vhosts/) where a server administrator, to save time/money/effort, can configure a single IP address to respond to multiple domain names. They then listen on that IP address for a request for one of their domains and that server will serve the pages. Cool! One IP with multiple domain names. For example: IP address 192.168.1.100 could resolve to http://www.example.com, mail.example.com and videos.example.com. When a browser requested http://www.example.com, the server would have the web application configured to listen for that request, serve up some content.

Now, let’s say that you are given a subnet of hosts (192.168.1.0/24 in CIDR notation or a class C in Ye Olde Fashioned lingo) to assess during a pentest. You discover that there is a host (192.168.1.100) that has TCP port 80 open and is running a web server on it. Awesome! You open up your web browser, type in http://192.168.1.100 and get a weird error about not getting any content or a default page was not set up and, well, no website! It may be that your system is using virtual hosts and you need to visit the site using its virtual host name and not the IP.

This post about some tools that you can use to find those host names within DNS servers and on the Internet. This class of tools is sometimes referred to as DNS Brute Forcers because they submit *.example.com where “*” is substituted by a word list. So if your word list had apple, admin, email, and test in it, these tools can send apple.example.com, admin.example.com, email.example.com and test.example.com as it iterates through your list of words. I’d be doing a disservice to these tools and their creators if I implied that that was all these scripts/applications do. I urge you to read up on each and decide which is right for you and your needs.

The Tools

I choose several free, stand-alone tools to try out and try to do an apples-to-apples test to see which worked best for host discovery. All were installed on a Kali Linux VM. I know that some commercial suites and frameworks have this capability but for my bake-off, I chose the following tools:
Now we need targets. Let’s pick three domains:
  1. apple.com (large, international)
  2. dhs.gov (US Gov’t <– always of interest!)
  3. sans.org (a smaller domain)

Word Lists

Now, we need to use the same word list across all tools. Since the DNSRecon tool is in front of me now, we’ll use its https://github.com/darkoperator/dnsrecon/blob/master/namelist.txt file for all of the tools. With this, we should expect to see all the tools’ brute-force the same number of hosts. To be fair to the developers, we’ll also use each tool’s own word list to see if one list is “better” than another. Below is some of the output from the “analysis” of the default word lists. Fierce had the biggest list and theHarvester had the smallest.

root@kali:~/dns# ll
total 32
-rw-r–r– 1 root root 11957 Aug  5 22:21 dnsrecon_namelist.txt
-rw-r–r– 1 root root 13191 Aug  5 22:21 fierce_hosts.txt
-rw-r–r– 1 root root  3370 Aug  5 22:45 harvest-dns-names.txt

root@kali:~/dns# wc *
 1907  1907 11957 dnsrecon_namelist.txt
 2062  2062 13191 fierce_hosts.txt
  563   563  3370 harvest-dns-names.txt

Each tool has other switches to control threads and output and so many other things. I’m not touching any of that.

DNSRecon

Setup

Carlos Perez’s (@Carlos_Perez) tool is actively developed and full of DNS-reaping features. git clone https://github.com/darkoperator/dnsrecon.git to grab the tool. I just got version 0.8.7 and chose to run it with:

# ./dnsrecon.py -D  ~/dns/dnsrecon_namelist.txt -d domainname] -t brt -c outfile.csv

where -D namelist.txt is the default brute-forcing word list that ships with the tool. This is the file I’ll use on the other tools too. The -t brt tells the tool to use the word list and try to brute-force the DNS. 


Results

Domain Records Found
using DNSRecon Word list		 Records Found
using Own List
apple.com
142
 –
dhs.gov
48
 –
sans.org
22
 –

Comments

Couple of comments here. DNSRecon discovered DNS6 (IPv6) entries in the DNS system, which was really cool. DNSRecon completed very quickly. It also tagged each entry with the type of DNS record. One issue to watch for was that it marked a bunch of sans.org domains as valid but the IP they “resolved to” was my DNS server. A bunch of false positives. See the CSV snippet of output below:

     Type,Name,Address,Target,Port,String
     CNAME,tw.apple.com.pub.vitrue.com,,deejay.cloud.vitrue.com,
     A,deejay.cloud.vitrue.com,50.16.57.100
     A,deejay.cloud.vitrue.com,50.16.7.33
     A,uk.apple.com,17.172.224.31
     A,uk.apple.com,17.149.160.31
     CNAME,webcast.apple.com,,salesguide.apple.com,
     A,salesguide.apple.com,17.254.20.77
     CNAME,www.apple.com,,www.isg-apple.com.akadns.net,
     CNAME,www.isg-apple.com.akadns.net,,www.apple.com.edgekey.net,

fierce2

Setup

The first hurdle you need to jump for fierce is the installation. It was real simple on a Linux system using the instructions http://trac.assembla.com/fierce/wiki/Installing. I used version 2.0-r420 for this test. Last commits to the tool seemed to be from 12/20/2012.

Let’s go ahead and kick off a scan:

# ./fierce -dns [domainname] -prefix ~/dns/dnsrecon_namelist.txt -all

where the -dns [domainname] is the target and -prefix ~/namelist.txt is the list (in our case from DNSRecon) with the subdomains you’d like to brute-force The -all tells the tool to run all the tests. Fierce does come with its own word list (hosts.txt). 


Results

Domain Records Found
using DNSRecon Word list		 Records Found
using Own List
apple.com
358
 358
dhs.gov
22
 22
sans.org
50
 50

Comments

This tool took a while to do the enumeration of entries. Again, I did not boost the number of threads or anything here. Judging from the numbers though, what this lacked in speed it made up for in lots of content. The default output looks like:

    17.254.3.59   gcrmbzp22.apple.com   59.3.254.17.in-addr.arpa
    17.254.3.60   appleorder.apple.com  60.3.254.17.in-addr.arpa
    17.149.160.26 nwk-redirect.apple.com 26.160.149.17.in-addr.arpa
    17.149.160.27 livepage.apple.com     27.160.149.17.in-addr.arpa
    17.149.160.30 asia-red.apple.com     30.160.149.17.in-addr.arpa

I’d like to point out here that because this went so slow, I got to watch the domain names scroll by. Apple has some very interesting ones:

  • Spices
    • Found Node! (17.254.17.106 / salt.apple.com)
    • Found Node! (17.254.17.107 / pepper.apple.com)
    • Found Node! (17.254.17.108 / cumin.apple.com)
  • Sports
    • Found Node! (17.254.17.176 / rugby.apple.com)
    • Found Node! (17.254.17.177 / rugger.apple.com)
    • Found Node! (17.254.17.178 / soccer.apple.com)
    • Found Node! (17.254.17.179 / kicker.apple.com)

theHarvester


Setup

After a quick SVN CO of this source (v2.2a used for this test), you are up and running on Christian Martorella’s tool. Looks like the last commits to the repository were from February 2013. The scan looks like:

     # ./theHarvester.py -d [domainname] -c -b all

where the -c tells the tool to brute-force and -b all tells it to use all sources. Yes, someone will probably point out that this means it is not only using DNS but web searches to discover hosts. I want to find the most and this is my test so I can choose what I want. 

Since we cannot specify the file name where our word list is, for the first part of the test, I’ve moved the DNSRecon namelist.txt file into this tool’s directory instead of the dns-names.txt default file.


Results

Domain Records Found
using DNSRecon Word list		 Records Found
using Own List
apple.com
72
 72
dhs.gov
93
 88
sans.org
57
 53

Comments

The tool ran very quickly and organized the results well. I liked how this tool ID’d the virtual hosts. Unfortunately apple.com results had several IPs that were shared by other customers of the web hosting provider and lots of false positives came out in the results. They were all flagged in the Virtual hosts section and were easy to remove. Other tools found the hosts but didn’t flag/separate them in the output. Default output snippet is below:
     [+] Hosts found in search engines:
     ————————————
     66.35.59.202:www.sans.org
     66.35.59.249:isc.sans.org
     66.35.59.8:dns21b.sans.org

     [+] Virtual hosts:
     ==================
     66.35.59.202 http://www.sans.org
     66.35.59.202 www2.sans.org
     66.35.59.249 isc.sans.edu
     66.35.59.249 iscold.sans.edu

blacksheepwall [updated 10/2013]

Setup

Tom Steele’s (@_tomsteele) tool was brought to my attention recently and so I thought I’d add it to the evaluation. It uses node.js which will need to be installed on your system prior to use. Installation is written out in the README.md file in the git repository. Get the tool with:

git clone https://github.com/tomsteele/blacksheepwall.git. 

I chose to run it with:
# ./cmd.js -d ~/dns/dnsrecon_namelist.txt -t [domainname]
NOTE – If your system uses OpenDNS or some other DNS service that redirects invalid DNS lookups to itself or a search page, you will need to use a different DNS sever such as Google’s at 8.8.8.8. 

Results

Domain Records Found
using DNSRecon Word list		 Records Found
using Own List
apple.com
70
 –
dhs.gov
22
 –
sans.org
21
 –

Comments

blacksheepwall has several output formats. I liked the “clean” format that grouped the domain names by the IP address that they resolved to. This easily showed which hosts resolved to the same server. Check out the sans.org output in the clean format below:
root@kali:/pentest/blacksheepwall/bin# ./cmd.js -b -d words.txt -t sans.org –clean
bsw started at Mon Oct 07 2013 18:35:31 GMT-0400 (EDT)
bsw finished at Mon Oct 07 2013 18:36:38 GMT-0400 (EDT)
204.51.94.43:
  blogs.sans.org
127.0.0.1:
  nt4.sans.org
  bsd.sans.org
  linux1.sans.org
  solaris.sans.org
  localhost.sans.org
  linux2.sans.org
10.10.10.10:
  oracle.sans.org
204.51.94.201:
  portal.sans.org
204.51.94.215:
  admin.sans.org
204.51.94.41:
  search.sans.org
204.51.94.173:
  files.sans.org
66.35.59.202:
  www2.sans.org
204.51.94.202:
  www3.sans.org
66.35.59.44:
  mail.sans.org
  mail2.sans.org
66.35.59.50:
  spam.sans.org
204.51.94.233:
  labs.sans.org
208.255.174.6:
  defiant.sans.org
204.51.94.40:
  lists.sans.org

Conclusions

Best Word List

Using the default word list from Fierce or DNSRecon should yield you adequate results when used with any tool.

Best Tool

If we are just looking at a “this tool consistently found more hosts than all others” metric, then theHarvester would be the winner. But as most of us know, each tool has a character of its own. DNSRecon was REALLY fast and pulled IPv6 DNS records. theHarvester had easily-readable default output and good sorting of entries. It also ran fast and did find more records than the others. The newest entry, blacksheepwall, leveraged Bing, performed reverse lookups and had some good-looking output options.


Results

Domain Best DNSRecon Run Best Fierce2 Run Best theHarvester Run Best blacksheepwall Run
apple.com
142
 358 72 70
dhs.gov
48
 22 93 22
sans.org
22
 50 57 21

I did not check to see which tools found which hosts (i.e., tool X and Y found this host but Z did not). That was too deep for this comparison but would be interesting.

So which tool will I use in the future? All of them. They all were solid and produced results. With the domain names you have, perhaps one tool finds more hosts than others. Try all three and make your own decision.

One big thank you to Philip Hartlieb (@pjhartlieb) for the request for this blog post. Appreciate it!

Comments are closed.

Up ↑