Robots.txt file
Summary: PortSwigger Research explains that the robots.txt file instructs web robots such as search engine crawlers about which locations on a site they may or may not crawl and index. The file is not itself a security vulnerability, but it often identifies restricted or private areas, and if an application relies on it for access control without enforcing proper access control, that presents a serious vulnerability.
Bottom line: Attackers are assumed to read robots.txt closely and can map out site contents from it, especially locations not linked from elsewhere, so any restricted area listed there is effectively disclosed to anyone who looks.
Read the original at PortSwigger Research ↗
- Topics
- Vulnerabilities & Patching
- Related
- Attackers Exploit UnrealIRCd Flaw That Enables Server-Level Command Injection Cyber Risks · Nov 14, 2026U.S. CISA adds Check Point and UnrealIRCd flaws to its Known Exploited Vulnerabilities catalog Cyber Risks · Nov 14, 2026PostgreSQL Patches 52 Flaws, Including 22-Year-Old Bug Enabling Authenticated RCE Cyber Risks · Nov 13, 2026