How to: Web Asset Discovery
https://github.com/tomnomnom/waybackurls
# Accept line-delimited domains on stdin, fetch known URLs from the Wayback Machine for *.domain and output them on stdout.
```cat subdomains.txt | waybackurls > waybackurls.txt```

https://github.com/tomnomnom/hacks
# Hacky one-off scripts, tests etc.
```cat waybackurls.txt | go run /root/Tools/hacks/anti-burl/main.go | tee waybackurls_valid.txt```

https://github.com/lc/gau
# getallurls (gau) fetches known URLs from AlienVault's Open Threat Exchange, the Wayback Machine, and Common Crawl for any given domain.
```cat domains.txt | gau --threads 5```

https://github.com/xnl-h4ck3r/waymore
# The idea behind waymore is to find even more links from the Wayback Machine than other existing tools.
```waymore -n -mode U -p 5 -i https://example.com```

https://github.com/projectdiscovery/urlfinder
# A high-speed tool for passively gathering URLs, optimized for efficient web asset discovery without active scanning.
```urlfinder -duc -silent -d example.com```

https://github.com/jaeles-project/gospider
# Fast web spider written in Go.
```gospider -s "https://example.com/" -o output -c 20 -d 10```

https://github.com/xnl-h4ck3r/xnLinkFinder
# This is a tool used to discover endpoints (and potential parameters) for a given target.

https://github.com/hakluke/hakrawler
# Fast golang web crawler for gathering URLs and JavaScript file locations. This is basically a simple implementation of the awesome Gocolly library.
```echo https://example.com | hakrawler```

https://github.com/projectdiscovery/katana
# A next-generation crawling and spidering framework.
```katana -u https://example.com```

https://geotargetly.com/geo-browse
# Geo Browse is a tool designed to capture screenshots of your website from different countries.

https://commoncrawl.org/
# We build and maintain an open repository of web crawl data that can be accessed and analyzed by anyone.

https://github.com/bitquark/shortscan
# Shortscan is designed to quickly determine which files with short filenames exist on an IIS webserver. Once a short filename has been identified the tool will try to automatically identify the full filename.
```shortscan https://example.com/```