The Advanced section for a web application identifies advanced options.
Select crawling hints to instruct the scanning engine to adhere to existing configurations when scanning the web application.
Robots.txt is a convention to prevent cooperating web spiders and other web robots from accessing all or part of a website which is otherwise publicly viewable. Select Use Robots.txt to adhere to a robots.txt file if present in the web application. Select Enforce directives to adhere to the directives in the robots.txt file.
Sitemap.xml is an XML file that lists URLs for a site to inform search engines about URLs that are available for crawling. Select Use Sitemap.xml to adhere to a sitemap.xml file if present in the web application.
This section identifies headers that need to be injected by the scanning engine to scan the web application. This option is intended to be used for situations where a workaround is needed for complex authentication schemes or to impersonate a web browser.
Enter header information in the field provided. A maximum of 2048 characters may be entered.
Enter each header in the format: <header>: <text>
Multiple headers may be entered. Each header must be separated by a new line.
To bypass a complex login form (for example, for multi-step authentication or CAPTCHA), where mwf_login is the session identifier for the application:
Cookie: mwf_login=2-e3b930b2cf6549d0351346d3cf56e9ae
To bypass a complex login form (for example, for multi-step authentication or CAPTCHA), where ASPSESSIONIDAARTTCBQ is the session identifier for the application:
Cookie: ASPSESSIONIDAARTTCBQ=BGHDNEICDKJBGJFMOIAOPLAG
To use a personalized user agent:
User-Agent: Mozilla/5.0 (iPhone; U; CPU like Mac OS X; en) AppleWebKit/420+ (KHTML, like Gecko) Version/3.0 Mobile/1A543a Safari/419.3
Some web applications display different information for different user agents. For instance a web application accessed by a mobile device will display light content containing different functionality, links, forms and underlying HTML code. For this reason, the scanning engine may find different vulnerabilities.
To bypass basic authentication:
Authorization: Basic bXl1c2VyOm15cGFzc3dvcmQ=
When a header such as the above is provided, the header basic authentication overrides any authentication record with basic authentication defined.