When we conduct network penetration tests, we often find that the majority of accessible services are web applications. Often times, these web applications have administrative interfaces with weak/default passwords or have vulnerabilities that allow us to access sensitive data or even allow arbitrary execution of code. Trying to find interesting web applications among a list of hundreds or thousands of servers can be a daunting task. For each web server, we typically take a screenshot, brute force common directory names, and look for vulnerable configuration settings. The only sane way to do this at scale is through automation.
We have released a set of basic scripts (https://github.com/AppSecConsulting/Pentest-Tools) that can be used to help with enumerating web applications at scale.
- msfweb.sh – Extracts web servers from Metasploit console using the services command
- gobuster.sh – Bruteforce common files on a web server using Gobuster
- nikto.sh – Scan each web server for common vulnerabilities using Nikto
- wkhtml.sh – Take a screenshot of the web server using wkhtmltoimage
Each of the scanning scripts (gobuster.sh, nikto.sh, and wkhtml.sh) expect a newline delimited file with each line describing a web server in the format of:
scheme host port
msfweb.sh
During engagements, we typically use msfconsole to import Nmap and Nessus results into the Metasploit database so we can then search for targets that match specific criteria using the services command. The msfweb.sh script uses the services command to extract a list of web servers from the Metasploit database and then puts the list of servers into the “scheme host port” format required by the scanning scripts.
# ./msfweb.sh
Saving extracted web servers to servers.txt
# cat servers.txt
http 192.168.0.1 80
http 192.168.0.2 443
http 192.168.0.2 80
http 192.168.0.3 80
http 192.168.0.4 443
http 192.168.0.4 80
gobuster.sh
The gobuster.sh script will run Gobuster against each web server in the input file. The script includes a WORDLIST variable and an OPTS variable, which can be modified to meet your needs. The gobuster.sh script expects an input file and an output directory, where all of the output files will be stored. The output for each web server will be stored in a separate file with a naming convention of “gobuster-[HOST]-[PORT].txt”.
To bruteforce common filenames on each web server with an output folder of “enum” and an input file of “servers.txt”, execute the following command:
# ./gobuster.sh enum servers.txt
nikto.sh and wkhtml.sh
The nikto.sh and wkhtml.sh scripts work in the same way as the gobuster.sh script. Each script has an OPTS variable that can be modified to meet your needs and each one stores its output in a separate file within the given directory. The naming convention for the nikto.sh script is “nikto-[HOST]-[PORT].txt” and the naming convention for the wkhtml.sh script is “wkhtml-[HOST]-[PORT].png”.
To run a Nikto scan against each server with an output folder of “enum” and an input file of “servers.txt”, execute the following command:
# ./nikto.sh enum servers.txt
To take a screenshot of each server with an output folder of “enum” and an input file of “servers.txt”, execute the following command:
# ./wkhtml.sh enum servers.txt
Each script has a set of default options that have worked well for us. You can see the default options for each script in the screenshot below:
Figure 1 – Screenshot of default options in each scanning script.
These web enumeration scripts were designed to run in Kali Linux and should include most, if not all, of the prerequisites. If they aren’t installed for some reason, you can install the prerequisites with apt:
# apt install gobuster, secliststs, nikto, csvkit, wkhtmltopdf
We’d love to get your feedback on web enumeration. What other scripts would you add? Are there better tools to use? Better default options? Let us know about it here: https://github.com/AppSecConsulting/Pentest-Tools.
