Fingerprinting / Crawling

Banner Grabbing

curl -I inlanefreight.com

Wafw00f

wafw00f inlanefreight.com

Nikto

nikto -h inlanefreight.com -Tuning b

The -Tuning b flag tells Nikto to only run the Software Identification modules.

GHunt

ghunt spiderdal -u domain.com

Cloud

Crawling / Scrapping

URLFinder

GitHub - projectdiscovery/urlfinder: A high-speed tool for passively gathering URLs, optimized for efficient and comprehensive web asset discovery without active scanning.GitHub

Katana

GitHub - projectdiscovery/katana: A next-generation crawling and spidering framework.GitHub

katana -u https://example.com -d 5 waybackarchive, commoncrawl, alienvault -kf -jc -fx -ef woff,css,png,svg,jpg,woff2,jpeg,gif,svg > results.txt

Gourlex

gourlex -t domain.com

GitHub - trap-bytes/gourlex: Gourlex is a simple tool that can be used to extract URLs and paths from web pages.GitHub

Hakrawler

GitHub - hakluke/hakrawler: Simple, fast web crawler designed for easy, quick discovery of endpoints and assets within a web applicationGitHub

LazyEgg

GitHub - schooldropout1337/lazyeggGitHub

Reconspider

reconspider | Kali Linux ToolsKali Linux

GitHub - 0xSs0rZ/Recon: reconGitHub

pip3 install scrapy

$ wget https://academy.hackthebox.com/storage/modules/279/ReconSpider.zip
$ unzip ReconSpider.zip 

python3 ReconSpider.py http://inlanefreight.com

cat results.json

creepyCrawler

GitHub - chm0dx/creepyCrawler: OSINT tool to crawl a site and extract useful recon info.GitHub

SecretOpt1c

GitHub - blackhatethicalhacking/SecretOpt1c: SecretOpt1c is a Red Team tool that helps uncover sensitive information in websites using ACTIVE and PASSIVE Techniques for Superior Accuracy!GitHub

Photon

GitHub - s0md3v/Photon: Incredibly fast crawler designed for OSINT.GitHub