URLScan.io¶
https://urlscan.io is a service to scan and analyse websites. When a URL is submitted to urlscan.io, an automated process will browse to the URL like a regular user and record the activity that this page navigation creates. This includes the domains and IPs contacted, the resources (JavaScript, CSS, etc) requested from those domains, as well as additional information about the page itself. urlscan.io will take a screenshot of the page, record the DOM content, JavaScript global variables, cookies created by the page, and a myriad of other observations.
This integration is a Resilient function that can be called from workflows, to submit a URL for analysis by urlscan.io. It returns the report metadata, report URL, and base64-encoded screenshot that is attached to the incident.
Table of Contents¶
Release Notes¶
Version |
Date |
Description |
|---|---|---|
1.1.6 |
04/2025 |
Converted example workflows to python3 |
1.1.5 |
06/2021 |
Bug fix error for non existing url |
1.1.4 |
09/2020 |
Proxy support added |
1.1.3 |
05/2020 |
Compatibility with older versions of resilient-circuits |
1.1.2 |
04/2020 |
Support added for App Host |
1.1.1 |
04/2020 |
Pinned version of resilient-lib to work with write_file_attachment |
1.1.0 |
03/2020 |
Removed workflow dependency on fn_utilities |
1.0.0 |
12/2018 |
Initial Release |
Installation¶
App Host Setup¶
All the components for running this integration in a container already exist when using the App Host app.
To install,
Navigate to Administrative Settings and then the Apps tab.
Click the Install button and select the downloaded file: app-fn_urlscanio-x.x.x.zip.
Go to the Configuration tab and edit the app.config file, editing the API key for URLScanIO and making any additional setting changes.
Integration Server Setup¶
Unzip file from AppExchange:
unzip app-fn_urlscanio-1.1.5.zip
The resulting .tar.gz file can be installed using
pip install fn_urlscanio-x.x.x.tar.gz
After installation, before running, you must import the customizations into your Resilient platform,
resilient-circuits customize -l fn-urlscanio
App Configuration¶
The following block is automatically added to your app.config file when running resilient-circuits config -u -l fn-urlscanio.
You will need to add your API key and have the flexibility to adjust the URL parameters if required.
Config |
Required |
Example |
Description |
|---|---|---|---|
urlscanio_report_url |
Yes |
|
URL to retrieve scan reports |
urlscanio_screenshot_url |
Yes |
|
URL for website screenshots |
urlscanio_api_key |
Yes |
1790000-0000-0000-0000-a1b2c3d4597 / Provide your URLScanIO API key |
|
timeout |
No |
300 / Seconds to timeout reports which take a long time to complete. Default 5 minutes |
|
http_proxy |
No |
|
Optional http proxy URL |
https_proxy |
No |
|
Optional https proxy URL |
Pre-Processing Script¶
# This is an artifact workflow;
# The URL to scan is the artifact value
inputs.urlscanio_url = artifact.value
# Set the incident id
inputs.incident_id = incident.id
Post-Processing Script¶
No action is performed after the workflow is complete, so we simply outline the result format in the results for ease of use.
# The result contains,
# {
# "png_url": the URL of the screenshot image
# "png_base64content": the base64-encoded screenshot (PNG)
# "report_url": the URL of the JSON report_url
# "report": the JSON report, which will contain lots of detail of the page analysis (see urlscan.io for details).
# }
#
# In this case, the file is already attached to the incident. Nothing to do here.
workflow.addProperty('convert_json_to_rich_text', {
"version": 1.0,
"header": "Artifact scan results for {}".format(artifact.value),
"padding": 10,
"separator": u"<br>",
"sort": True,
"json": results,
"json_omit_list": ['png_base64content'],
"incident_field": None
})
Other notes¶
To regenerate the customization blob,
resilient-sdk codegen -p fn_urlscanio -m urlscanio --workflow example_urlscanio --rule "Example: urlscan.io"