Web Inventory tool, takes screenshots of webpages using Pyppeteer (headless Chrome/Chromium) and provides some extra bells & whistles to make life easier.

Overview

WitnessMe

WitnessMe

WitnessMe is primarily a Web Inventory tool inspired by Eyewitness, its also written to be extensible allowing you to create custom functionality that can take advantage of the headless browser it drives in the back-end.

WitnessMe uses the Pyppeteer library to drive Headless Chromium.

Sponsors

Table of Contents

Motivation

Are there are a bunch of other tools that do this? Absolutely. See the following projects for alternatives (I'm sure there are more, these are the ones I've personally tried):

The reason why I wrote WitnessMe was that none of these projects had all of the features I wanted/needed in order for them to work well within my workflow. Additionally, some of them are prone to a decent amount of installation/dependency hell.

Here are some of the main features that make WitnessMe "stand out":

  • Written in Python 3.7+
  • Ability to parse extremely large Nessus and NMap XML files
  • Docker compatible
  • No installation/dependency hell
  • Full test suite! Everything is less prone to bugs
  • CSV & HTML reporting
  • HTTP Proxy Support
  • Provides a RESTful API! Scan stuff remotely!
  • CLI interface to view and search scan results without having to view the reports.
  • Signature scanning (Signatures use YAML files)
  • Preview screenshots directly in the terminal (On MacOSX/ITerm2 and some Nix terminals)
  • Extensibly written, allowing you to add functionality that can take advantage of headless chromium.
  • Built to be deployed to the Clouds (e.g. GCP Cloud Run , AWS ElasticBeanstalk etc...)

Official Discord Channel

Come hang out on Discord!

Porchetta Industries

Installation

Docker

Running WitnessMe from a Docker container is fully supported and is the easiest/recommended way of using the tool.

Note: it is highly recommended to give the Docker container at least 4GB of RAM during large scans as Chromium can be a resource hog. If you keep running into "Page Crash" errors, it's because your container does not have enough memory. On Mac/Windows you can change this by clicking the Docker Task Bar Icon -> Preferences -> Resources. For Linux, refer to Docker's documentation

Pull the image from Docker Hub:

docker pull byt3bl33d3r/witnessme

You can then spin up a docker container, run it like the main witnessme script and pass it the same arguments:

docker run --rm -ti $IMAGE_ID screenshot https://google.com 192.168.0.1/24

Alternatively, you can drop into a shell within the container and run the tools that way. This also allows you to execute the wmdb and wmapi scripts.

docker run --rm -ti --entrypoint=/bin/sh $IMAGE_ID

Python Package

WitnessMe is also available as a Python package (Python 3.7 or above is required). If you do install it this way it is extremely recommended to use pipx as it takes care of installing everything in isolated environments for you in a seamless manner.

Run the following commands:

python3 -m pip install --user pipx
pipx install witnessme

All of the WitnessMe scripts should now be in your PATH and ready to go.

Development Install

You really should only install WitnessMe this way if you intend to hack on the source code. You're going to Python 3.7+ and Poetry: please refer to the Poetry installation documentation in order to install it.

git clone https://github.com/byt3bl33d3r/WitnessMe && cd WitnessMe
poetry install

Quick Starts

Finding F5 Load Balancers Vulnerable to CVE-2020-5902

Install WitnessMe using Docker:

docker pull byt3bl33d3r/witnessme

Get the $IMAGE_ID from the docker images command output, then run the following command to drop into a shell inside the container. Additionally, specify the -v flag to mount the current directory inside the container at the path /transfer in order to copy the scan results back to your host machine (if so desired):

docker run -it --entrypoint=/bin/sh -v $(pwd):/transfer $IMAGE_ID

Scan your network using WitnessMe, it can accept multiple .Nessus files, Nmap XMLs, IP ranges/CIDRs. Example:

witnessme screenshot 10.0.1.0/24 192.168.0.1-20 ~/my_nessus_scan.nessus ~/my_nmap_scan.xml

After the scan is finished, a folder will have been created in the current directory with the results. Access the results using the wmdb command line utility:

wmdb scan_2020_$TIME/

To quickly identify F5 load balancers, first perform a signature scan using the scan command. Then search for "BIG-IP" or "F5" using the servers command (this will search for the "BIG-IP" and "F5" string in the signature name, page title and server header):

image

Additionally, you can generate an HTML or CSV report using the following commands:

WMDB ≫ generate_report html
WMDB ≫ generate_report csv

You can then copy the entire scan folder which will contain all of the reports and results to your host machine by copying it to the /transfer folder.

Scraping Javascript Heavy Webpages

As of v1.5.0, WitnessMe has a grab command which allows you to quickly scrape Javascript heavy webpages by rendering the page first with Headless Chromium and then parsing the resulting HTML using the specified XPath (see here for an XPath cheatsheet).

Below are a few examples to get your started.

This grabs a list of all advertised domains on the 144.161.160.0/23 subnet from Hurricane Electric's BGP Toolkit:

witnessme -d grab -x '//div[@id="dns"]/table//tr/td[2]/a/text()' https://bgp.he.net/net/144.161.160.0/23#_dns

RESTful API

As of version 1.0, WitnessMe has a RESTful API which allows you to interact with the tool remotely.

Note: Currently, the API does not implement any authentication mechanisms. Make sure to allow/deny access at the transport level

To start the RESTful API for testing/development purposes run :

wmapi

The API documentation will then be available at http://127.0.0.1:8000/docs

Uvicorn should be used to enable SSL and run the API in production. See this dockerfile for an example.

Deploying to the Cloud (™)

Since WitnessMe has a RESTful API now, you can deploy it to the magical cloud and perform scanning from there. This would have a number of benefits, including giving you a fresh external IP on every scan (More OPSEC safe when assessing attack surface on Red Teams).

There are a number of ways of doing this, you can obviously do it the traditional way (e.g. spin up a machine, install docker etc..).

Recently cloud service providers started offering ways of running Docker containers directly in a fully managed environment. Think of it as serverless functions (e.g. AWS Lambdas) only with Docker containers.

This would technically allow you to really quickly deploy and run WitnessMe (or really anything in a Docker container) without having to worry about underlying infrastructure and removes a lot of the security concerns that come with that.

Below are some of the ones I've tried along with the steps necessary to get it going and any issues I encountered.

GCP Cloud Run

Unfortunately, it seems like Cloud Run doesn't allow outbound internet access to containers, if anybody knows of a way to get around this please get in touch

Cloud Run is by far the easiest of these services to work with.

This repository includes the cloudbuild.yaml file necessary to get this setup and running.

From the repositories root folder (after you authenticated and setup a project), these two commands will automatically build the Docker image, publish it to the Gcloud Container Registry and deploy a working container to Cloud Run:

gcloud builds submit --config cloudbuild.yaml
gcloud run deploy --image gcr.io/$PROJECT_ID/witnessme --platform managed

The output will give you a HTTPS url to invoke the WitnessMe RESTful API from :)

When you're done:

gcloud run services delete witnessme
gcloud container images delete gcr.io/$PROJECT_ID/witnessme

AWS ElasticBeanstalk

TO DO

Usage

There are 3 main utilities:

  • witnessme: is the main CLI interface.
  • wmdb: allows you to browse the database (created on each scan) to view results and generate reports.
  • wmapi: provides a RESTful API to schedule, start, stop and monitor scans.

Modes of Operations

As of v1.5.0 there are two main modes (commands) that the witnessme utility Supports:

  • The screenshot command, you guessed it, screenshots webpages. This is the main functionality.
  • The grab command allows you to scrape pages and quickly grab server headers.
usage: witnessme [-h] [--threads THREADS] [--timeout TIMEOUT] [-d] [-v] {screenshot,grab} ...

WitnessMe!

positional arguments:
  {screenshot,grab}

optional arguments:
  -h, --help         show this help message and exit
  --threads THREADS  Number of concurrent browser tab(s) to open
                     [WARNING: This can cause huge RAM consumption if set to high values] (default: 15)
  --timeout TIMEOUT  Timeout for each connection attempt in seconds (default: 15)
  -d, --debug        Enable debug output (default: False)
  -v, --version      show program's version number and exit

Screenshot Mode

$ witnessme screenshot --help
usage: witnessme screenshot [-h] [-p PORTS [PORTS ...]] target [target ...]

positional arguments:
  target                The target IP(s), range(s), CIDR(s) or hostname(s), NMap XML file(s), .Nessus file(s)

optional arguments:
  -h, --help            show this help message and exit
  -p PORTS [PORTS ...], --ports PORTS [PORTS ...]
                        Ports to scan if IP Range/CIDR is provided

Can accept a mix of .Nessus file(s), Nmap XML file(s), files containing URLs and/or IPs, IP addresses/ranges/CIDRs and URLs or alternatively read from stdin.

Note: WitnessMe detects .Nessus and NMap files by their extension so make sure Nessus files have a .nessus extension and NMap scans have a .xml extension

Long story short, should be able to handle anything you throw at it:

witnessme screenshot 192.168.1.0/24 192.168.1.10-20 https://bing.com ~/my_nessus_scan.nessus ~/my_nmap_scan.xml ~/myfilewithURLSandIPs
$ cat my_domain_list.txt | witnessme screenshot -

If an IP address/range/CIDR is specified as a target, WitnessMe will attempt to screenshot HTTP & HTTPS pages on ports 80, 8080, 443, 8443 by default. This is customizable with the --port argument.

Once a scan is completed, a folder with all the screenshots and a database will be in the current directory, point wmdb to the folder in order to see the results.

wmdb scan_2019_11_05_021237/

Grab Mode

$ witnessme grab --help
usage: witnessme grab [-h] [-x XPATH | -l] target [target ...]

positional arguments:
  target                The target IP(s), range(s), CIDR(s) or hostname(s), NMap XML file(s), .Nessus file(s)

optional arguments:
  -h, --help            show this help message and exit
  -x XPATH, --xpath XPATH
                        XPath to use
  -l, --links           Get all links

The grab subcommand allows you to render Javascript heavy webpages and scrape their content using XPaths. See this section for some examples.

Interacting with the Scan Database

Once a scan is completed (using the screenshot mode), a folder with all the screenshots and a database will be in the current directory, point wmdb to the folder in order to see the results.

wmdb scan_2019_11_05_021237/

This will drop you into the WMDB CLI menu.

Pressing tab will show you the available commands and a help menu:

Tab

The servers and hosts commands in the wmdb CLI accept 1 argument. WMCLI is smart enough to know what you're trying to do with that argument

Server Command

No arguments will show all discovered servers. Passing it an argument will search the title and server columns for that pattern (it's case insensitive).

For example if you wanted to search for all discovered Apache Tomcat servers:

  • servers tomcat or servers 'apache tomcat'

Similarly if you wanted to find servers with a 'login' in the title:

  • servers login

Hosts Command

No arguments will show all discovered hosts. Passing it an argument will search the IP and Hostname columns for that pattern (it's case insensitive). If the value corresponds to a Host ID it will show you the host information and all of the servers discovered on that host which is extremely useful for reporting purposes and/or when targeting specific hosts.

Signature Scan

You can perform a signature scan on all discovered services using the scan command.

Generating Reports

You can use the generate_report command in the wmdb cli to generate reports in HTML or CSV format. To generate a HTML report simply run generate_report without any arguments. Here's an example of what it'll look like:

image

To generate a CSV report:

WMDB ≫ generate_report csv

The reports will then be available in the scan folder.

Proxying

As of v1.5 WitnessMe supports proxying all of its traffic through an HTTP proxy. Specify a HTTP_PROXY environment variable to force the underlying headless browser to proxy its traffic through the desired host:

HTTP_PROXY=http://127.0.0.1:8080 witnessme screenshot ~/my_targets.txt
HTTP_PROXY=http://127.0.0.1:8080 witnessme grab https://www.google.com

Preview Screenshots Directly in the Terminal

Note: this feature will only work if you're on MacOSX and using ITerm2

You can preview screenshots directly in the terminal using the show command:

ScreenPreview

Writing Signatures

If you run into a new webapp write a signature for it! It's beyond simple and they're all in YAML!

Don't believe me? Here's the AirOS signature (you can find them all in the signatures directory):

credentials:
- password: ubnt
  username: ubnt
name: AirOS
signatures:
- airos_logo.png
- form enctype="multipart/form-data" id="loginform" method="post"
- align="center" class="loginsubtable"
- function onLangChange()
# AirOS ubnt/ubnt

Yup that's it. Just plop it in the signatures folder and POW! Done.

Issues
  • Error taking screenshot: expected string or bytes-like object

    Error taking screenshot: expected string or bytes-like object

    I'll started WitnessMe with a .nessus file, but I receive the error

    Error taking screenshot: expected string or bytes-like object

    Screenshot: https://monosnap.com/file/OofiFk9CJ4JSLOr8dJX17x8rFon9kY

    Moar info required tovarish 
    opened by BlueCanary-DM 14
  • Connection to Chrome/Chromium times out after 1k-ish screenshots?

    Connection to Chrome/Chromium times out after 1k-ish screenshots?

    So apparently, there's yet another websocket connection bug in pyppeteer. Seems that after 400ish screenshots the connection to chrome just dies. :( https://github.com/miyakogi/pyppeteer/issues/149

    Tried some of the proposed fixes but none of them worked for me.

    [E:pyppeteer.connection] connection unexpectedly closed
    2019-11-10 23:19:09,304 [ERROR] - base_events.py: default_exception_handler - Task exception was never retrieved
    future: <Task finished name='Task-7697' coro=<Connection._async_send() done, defined at /home/bhis/.local/share/virtualenvs/WitnessMe-QFMsedki/lib/python3.8/site-packages/pyppeteer/connection.py:69> exception=InvalidStateError('invalid state')>
    Traceback (most recent call last):
      File "/home/bhis/.local/share/virtualenvs/WitnessMe-QFMsedki/lib/python3.8/site-packages/websockets/protocol.py", line 827, in transfer_data
        message = await self.read_message()
      File "/home/bhis/.local/share/virtualenvs/WitnessMe-QFMsedki/lib/python3.8/site-packages/websockets/protocol.py", line 895, in read_message
        frame = await self.read_data_frame(max_size=self.max_size)
      File "/home/bhis/.local/share/virtualenvs/WitnessMe-QFMsedki/lib/python3.8/site-packages/websockets/protocol.py", line 971, in read_data_frame
        frame = await self.read_frame(max_size)
      File "/home/bhis/.local/share/virtualenvs/WitnessMe-QFMsedki/lib/python3.8/site-packages/websockets/protocol.py", line 1047, in read_frame
        frame = await Frame.read(
      File "/home/bhis/.local/share/virtualenvs/WitnessMe-QFMsedki/lib/python3.8/site-packages/websockets/framing.py", line 105, in read
        data = await reader(2)
      File "/home/bhis/.pyenv/versions/3.8.0/lib/python3.8/asyncio/streams.py", line 738, in readexactly
        raise exceptions.IncompleteReadError(incomplete, n)
    asyncio.exceptions.IncompleteReadError: 0 bytes read on a total of 2 expected bytes
    
    The above exception was the direct cause of the following exception:
    
    Traceback (most recent call last):
      File "/home/bhis/.local/share/virtualenvs/WitnessMe-QFMsedki/lib/python3.8/site-packages/pyppeteer/connection.py", line 73, in _async_send
        await self.connection.send(msg)
      File "/home/bhis/.local/share/virtualenvs/WitnessMe-QFMsedki/lib/python3.8/site-packages/websockets/protocol.py", line 555, in send
        await self.ensure_open()
      File "/home/bhis/.local/share/virtualenvs/WitnessMe-QFMsedki/lib/python3.8/site-packages/websockets/protocol.py", line 803, in ensure_open
        raise self.connection_closed_exc()
    websockets.exceptions.ConnectionClosedError: code = 1006 (connection closed abnormally [internal]), no reason
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "/home/bhis/.local/share/virtualenvs/WitnessMe-QFMsedki/lib/python3.8/site-packages/pyppeteer/connection.py", line 79, in _async_send
        await self.dispose()
      File "/home/bhis/.local/share/virtualenvs/WitnessMe-QFMsedki/lib/python3.8/site-packages/pyppeteer/connection.py", line 170, in dispose
        await self._on_close()
      File "/home/bhis/.local/share/virtualenvs/WitnessMe-QFMsedki/lib/python3.8/site-packages/pyppeteer/connection.py", line 151, in _on_close
        cb.set_exception(_rewriteError(
    asyncio.exceptions.InvalidStateError: invalid state
    
    bug 
    opened by byt3bl33d3r 7
  • Error calling command 'scan': module 'yaml' has no attribute 'CLoader'

    Error calling command 'scan': module 'yaml' has no attribute 'CLoader'

    The working environment is Mac OS.

    WMDB ≫ scan 1 Traceback (most recent call last): File "wmdb.py", line 239, in cmdloop await self.scan() File "wmdb.py", line 197, in scan self.signatures.load() File "/Users/[redacted]/opt/WitnessMe/witnessme/signatures.py", line 15, in load self.signatures.append(yaml.load(sig, Loader=yaml.CLoader)) AttributeError: module 'yaml' has no attribute 'CLoader' Error calling command 'scan': module 'yaml' has no attribute 'CLoader'

    Moar info required tovarish 
    opened by sdcampbell 7
  • TypeError when parsing nmap file

    TypeError when parsing nmap file

    The following error is thrown when an nmap XML is given as input :

      File "/home/neo/.local/pipx/venvs/witnessme/lib/python3.8/site-packages/witnessme/parsers.py", line 109, in parser_callback
        if port["@protocol"] == "tcp" and port["state"]["@state"] == "open":
    TypeError: string indices must be integers
    
    bug 
    opened by pwnfoo 6
  • Errors during nmap XML parsing

    Errors during nmap XML parsing

    I was attempting to run WitnessMe with a large nmap XML file for the top 1000 ports in a certain range. When passing this to WitnessMe (using both pipx and docker) I received the following error:

    # docker run -it --entrypoint=/bin/sh -v $(pwd):/transfer a45ee02a44c1
    $ witnessme screenshot /transfer/top1000.xml
    [witnessme.screenshot] INFO - Starting scan 593de326-9a88-4ee1-9701-305af0bfb66b
    Task exception was never retrieved
    future: <Task finished name='Task-3' coro=<HeadlessChromium.target_producer() done, defined at /home/appuser/.local/lib/python3.8/site-packages/witnessme/headlessbrowser.py:64> exception=KeyError('port')>
    Traceback (most recent call last):
      File "/home/appuser/.local/lib/python3.8/site-packages/witnessme/headlessbrowser.py", line 66, in target_producer
        for url in generated_targets:
      File "/home/appuser/.local/lib/python3.8/site-packages/witnessme/parsers.py", line 198, in generate
        for url in generated_urls:
      File "/home/appuser/.local/lib/python3.8/site-packages/witnessme/parsers.py", line 85, in __enter__
        xmltodict.parse(
      File "/home/appuser/.local/lib/python3.8/site-packages/xmltodict.py", line 325, in parse
        parser.ParseFile(xml_input)
      File "/usr/src/python/Modules/pyexpat.c", line 461, in EndElement
      File "/home/appuser/.local/lib/python3.8/site-packages/xmltodict.py", line 126, in endElement
        should_continue = self.item_callback(self.path, item)
      File "/home/appuser/.local/lib/python3.8/site-packages/witnessme/parsers.py", line 108, in parser_callback
        ports = item["ports"]["port"]
    KeyError: 'port'
    

    I was able to use gowitness and it parsed the file normally, so I believe this has something to do with how WitnessMe handles nmap XML parsing.

    bug 
    opened by sebrink 5
  • Adding --no-sandbox param to the pyppeteer

    Adding --no-sandbox param to the pyppeteer

    When you try to run witness me with root user, pyppeteer will be ignoring to initiate chrome process. In order to avoid from that, we gotta pass args=['--no-sandbox'] to launch() function.

    https://github.com/byt3bl33d3r/WitnessMe/blob/c8086041a418026b1b4b13afeb253ee6cdcef188/witnessme.py#L139

    I do propose following fix. I've tested it on locally.

    browser = await pyppeteer.launch(headless=True, ignoreHTTPSErrors=True, args=['--no-sandbox'])
    
    duplicate 
    opened by mdisec 4
  • pyppeteer.errors.BrowserError: Browser closed unexpectedly:

    pyppeteer.errors.BrowserError: Browser closed unexpectedly:

    pyppeteer.errors.BrowserError: Browser closed unexpectedly:Running as root without --no-sandbox is not supported.

    bug Moar info required tovarish 
    opened by korang 3
  • websocket installation fails during pipenv install --three

    websocket installation fails during pipenv install --three

    I am trying to add WitnessMe to the SamuraiWTF build. We are getting a failure during the set up from pipenv failing to find a matching version of websockets.

    Basically the error is:

    "ERROR: Could not find a version that satisfies the requirement websocket==8.1" It then lists "from versions" and that only goes to 7.0. This is a Debian system from SamuraiWTF

    opened by secureideas 3
  • Doesn't recognize file with urls

    Doesn't recognize file with urls

    Hi, I'm trying to feed it a plain file with urls but it appears it's trying to resolve the filename instead of the actual urls in the file.

    [email protected] ~/tools/witnessme docker run --rm -ti 0c2b4fbab0b3 /tmp/domains
    [witnessme.utils] DEBUG - Patching pyppeteer...
    [witnessme.scan] DEBUG - Waiting for queue to populate...
    [witnessme.parsers] DEBUG - Detected IP Address/Range/CIDR, hostname or URL as a target
    [witnessme.scan] INFO - Starting headless browser
    [witnessme.scan] INFO - Using 8 worker thread(s)
    [witnessme.scan] ERROR - Error taking screenshot: net::ERR_NAME_NOT_RESOLVED at http:///tmp/domains:8080
    [witnessme.scan] ERROR - Error taking screenshot: net::ERR_NAME_NOT_RESOLVED at http:///tmp/domains:443
    [witnessme.scan] ERROR - Error taking screenshot: net::ERR_NAME_NOT_RESOLVED at http:///tmp/domains:8443
    [witnessme.scan] ERROR - Error taking screenshot: net::ERR_NAME_NOT_RESOLVED at https:///tmp/domains:80
    [witnessme.scan] ERROR - Error taking screenshot: net::ERR_NAME_NOT_RESOLVED at https:///tmp/domains:8080
    [witnessme.scan] ERROR - Error taking screenshot: net::ERR_NAME_NOT_RESOLVED at https:///tmp/domains:443
    [witnessme.scan] ERROR - Error taking screenshot: net::ERR_NAME_NOT_RESOLVED at http:///tmp/domains:80
    [witnessme.scan] ERROR - Error taking screenshot: net::ERR_NAME_NOT_RESOLVED at https:///tmp/domains:8443
    [witnessme.scan] INFO - Killing headless browser
    
    Moar info required tovarish 
    opened by bluecanarybe 3
  • Ability to output to CSV or XML --feature request

    Ability to output to CSV or XML --feature request

    The tool makes a very useful graph output. Would it be possible to leverage it in another format that is shareable to people that don't have WitnessMe installed?

    Specifically, I was hoping for a CSV or XML output.

    enhancement 
    opened by nullenc0de 2
  • /transfer running as root and cannot transfer files to host

    /transfer running as root and cannot transfer files to host

    Using the docker and -v to mount current volume to /transfer directory, I can see that /transfer is owned by root and hence cannot transfer any files from docker into the /transfer folder to take the files outside to the host machine

    opened by virgilcj 0
  • Nmap/Nessus file upload through api

    Nmap/Nessus file upload through api

    First off, very cool project :grin: I enjoy most of the stuff you do :partying_face: so thank you for doing this! :pray:

    I was curious if there was a reason for not exposing ( from what I can see in the fastapi docs page ) uploading a nmap/nessus file?

    I noticed you have the functionality already there from the CLI perspective, but is there a limitation for exposing that through the API?

    opened by elreydetoda 0
  • Allow to specify a different folder name location

    Allow to specify a different folder name location

    It would be nice if you can specify a different folder name for output.

    witnessme -out /screenshots instead of automatic (pwd)/scan_datetimestamp

    opened by greckko 0
  • Bump pydantic from 1.6.1 to 1.6.2

    Bump pydantic from 1.6.1 to 1.6.2

    Bumps pydantic from 1.6.1 to 1.6.2.

    Release notes

    Sourced from pydantic's releases.

    v1.6.2 (2021-05-11)

    Security fix: Fix date and datetime parsing so passing either 'infinity' or float('inf') (or their negative values) does not cause an infinite loop, see security advisory CVE-2021-29510.

    Changelog

    Sourced from pydantic's changelog.

    v1.6.2 (2021-05-11)

    • Security fix: Fix date and datetime parsing so passing either 'infinity' or float('inf') (or their negative values) does not cause an infinite loop, See security advisory CVE-2021-29510
    Commits

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    • @dependabot use these labels will set the current labels as the default for future PRs for this repo and language
    • @dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language
    • @dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language
    • @dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

    You can disable automated security fix PRs for this repo from the Security Alerts page.

    dependencies 
    opened by dependabot[bot] 0
  • Bump py from 1.9.0 to 1.10.0

    Bump py from 1.9.0 to 1.10.0

    Bumps py from 1.9.0 to 1.10.0.

    Changelog

    Sourced from py's changelog.

    1.10.0 (2020-12-12)

    • Fix a regular expression DoS vulnerability in the py.path.svnwc SVN blame functionality (CVE-2020-29651)
    • Update vendored apipkg: 1.4 => 1.5
    • Update vendored iniconfig: 1.0.0 => 1.1.1
    Commits
    • e5ff378 Update CHANGELOG for 1.10.0
    • 94cf44f Update vendored libs
    • 5e8ded5 testing: comment out an assert which fails on Python 3.9 for now
    • afdffcc Rename HOWTORELEASE.rst to RELEASING.rst
    • 2de53a6 Merge pull request #266 from nicoddemus/gh-actions
    • fa1b32e Merge pull request #264 from hugovk/patch-2
    • 887d6b8 Skip test_samefile_symlink on pypy3 on Windows
    • e94e670 Fix test_comments() in test_source
    • fef9a32 Adapt test
    • 4a694b0 Add GitHub Actions badge to README
    • Additional commits viewable in compare view

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    • @dependabot use these labels will set the current labels as the default for future PRs for this repo and language
    • @dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language
    • @dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language
    • @dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

    You can disable automated security fix PRs for this repo from the Security Alerts page.

    dependencies 
    opened by dependabot[bot] 0
  • Bump lxml from 4.5.2 to 4.6.3

    Bump lxml from 4.5.2 to 4.6.3

    Bumps lxml from 4.5.2 to 4.6.3.

    Changelog

    Sourced from lxml's changelog.

    4.6.3 (2021-03-21)

    Bugs fixed

    • A vulnerability (CVE-2021-28957) was discovered in the HTML Cleaner by Kevin Chung, which allowed JavaScript to pass through. The cleaner now removes the HTML5 formaction attribute.

    4.6.2 (2020-11-26)

    Bugs fixed

    • A vulnerability (CVE-2020-27783) was discovered in the HTML Cleaner by Yaniv Nizry, which allowed JavaScript to pass through. The cleaner now removes more sneaky "style" content.

    4.6.1 (2020-10-18)

    Bugs fixed

    • A vulnerability was discovered in the HTML Cleaner by Yaniv Nizry, which allowed JavaScript to pass through. The cleaner now removes more sneaky "style" content.

    4.6.0 (2020-10-17)

    Features added

    • GH#310: lxml.html.InputGetter supports __len__() to count the number of input fields. Patch by Aidan Woolley.

    • lxml.html.InputGetter has a new .items() method to ease processing all input fields.

    • lxml.html.InputGetter.keys() now returns the field names in document order.

    • GH-309: The API documentation is now generated using sphinx-apidoc. Patch by Chris Mayo.

    Bugs fixed

    ... (truncated)

    Commits
    • a5f9cb5 Prepare release of lxml 4.6.3.
    • 2d01a1b Add HTML-5 "formaction" attribute to "defs.link_attrs" (GH-316)
    • e986a9c Fix reference in docs.
    • 4cb5736 Work around Py2's lack of "re.ASCII".
    • c30106f Prepare release of 4.6.2.
    • a105ab8 Prevent combinations of <math/svg> and <style> to sneak JavaScript through th...
    • c053dc1 Add a recipe for a look-ahead generator to allow modifications during tree it...
    • b083124 lxml actually works in Py3.9.
    • 0f80590 lxml actually works in Py3.9.
    • fd8893c Add a doc note that the .find() methods are usually faster than one might exp...
    • Additional commits viewable in compare view

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    • @dependabot use these labels will set the current labels as the default for future PRs for this repo and language
    • @dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language
    • @dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language
    • @dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

    You can disable automated security fix PRs for this repo from the Security Alerts page.

    dependencies 
    opened by dependabot[bot] 0
  • Bump pyyaml from 5.3.1 to 5.4

    Bump pyyaml from 5.3.1 to 5.4

    Bumps pyyaml from 5.3.1 to 5.4.

    Changelog

    Sourced from pyyaml's changelog.

    5.4 (2021-01-19)

    Commits
    • 58d0cb7 5.4 release
    • a60f7a1 Fix compatibility with Jython
    • ee98abd Run CI on PR base branch changes
    • ddf2033 constructor.timezone: _copy & deepcopy
    • fc914d5 Avoid repeatedly appending to yaml_implicit_resolvers
    • a001f27 Fix for CVE-2020-14343
    • fe15062 Add 3.9 to appveyor file for completeness sake
    • 1e1c7fb Add a newline character to end of pyproject.toml
    • 0b6b7d6 Start sentences and phrases for capital letters
    • c976915 Shell code improvements
    • Additional commits viewable in compare view

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    • @dependabot use these labels will set the current labels as the default for future PRs for this repo and language
    • @dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language
    • @dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language
    • @dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

    You can disable automated security fix PRs for this repo from the Security Alerts page.

    dependencies 
    opened by dependabot[bot] 0
  • Bump jinja2 from 2.11.2 to 2.11.3

    Bump jinja2 from 2.11.2 to 2.11.3

    Bumps jinja2 from 2.11.2 to 2.11.3.

    Release notes

    Sourced from jinja2's releases.

    2.11.3

    This contains a fix for a speed issue with the urlize filter. urlize is likely to be called on untrusted user input. For certain inputs some of the regular expressions used to parse the text could take a very long time due to backtracking. As part of the fix, the email matching became slightly stricter. The various speedups apply to urlize in general, not just the specific input cases.

    Changelog

    Sourced from jinja2's changelog.

    Version 2.11.3

    Released 2021-01-31

    • Improve the speed of the urlize filter by reducing regex backtracking. Email matching requires a word character at the start of the domain part, and only word characters in the TLD. :pr:1343
    Commits

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    • @dependabot use these labels will set the current labels as the default for future PRs for this repo and language
    • @dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language
    • @dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language
    • @dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

    You can disable automated security fix PRs for this repo from the Security Alerts page.

    dependencies 
    opened by dependabot[bot] 0
  • Signature Based on Favicon

    Signature Based on Favicon

    There's been a few times on engagements where I only see a blank page, and even X-Powered-By and Server headers are suppressed, but the favicon for the product deployed to the server is still available. I was wondering what you think about adding the ability to signature based on favicons?

    I'm thinking it could either be done by a quick MD5 or maybe even CRC32 to help keep performance up. I wanted to get your input on it before I try to code anything up and submit a PR

    enhancement 
    opened by decidedlygray 4
  • Unsupported screenshot mime type: image/vnd.mozilla.apng

    Unsupported screenshot mime type: image/vnd.mozilla.apng

    Recently updated WitnessMe and ran the following command:

    witnessme screenshot textFile.txt - the text file has one url per line as 'https://something.com'

    It seems the valid urls (I am able to manually verify them) are providing the following error:

    [witnessme.headlessbrowser] ERROR - Error navigating to url https://SOMETHING.COM: Unsupported screenshot mime type: image/vnd.mozilla.apng

    Thank you in advance for your help.

    opened by weuroi578 2
Owner
byt3bl33d3r
T H I C C M A L W A R E
byt3bl33d3r
All of the ad-hoc things you're doing to manage incidents today, done for you, and much more!

About What's Dispatch? Put simply, Dispatch is: All of the ad-hoc things you’re doing to manage incidents today, done for you, and a bunch of other th

Netflix, Inc. 3.1k Oct 15, 2021
Web Version of avatarify to democratize even further

Web-avatarify for image animations This is the code base for this website and its backend. This aims to bring technology closer to everyone, just by a

Carlos Andrés Álvarez Restrepo 59 Sep 9, 2021
Monitor Python applications using Spring Boot Admin

Pyctuator Monitor Python web apps using Spring Boot Admin. Pyctuator supports Flask, FastAPI, aiohttp and Tornado. Django support is planned as well.

SolarEdge Technologies 97 Oct 22, 2021
A Prometheus Python client library for asyncio-based applications

aioprometheus aioprometheus is a Prometheus Python client library for asyncio-based applications. It provides metrics collection and serving capabilit

null 90 Oct 20, 2021
ReST based network device broker

The Open API Platform for Network Devices netpalm makes it easy to push and pull state from your apps to your network by providing multiple southbound

null 316 Sep 28, 2021
Backend logic implementation for realworld with awesome FastAPI

Backend logic implementation for realworld with awesome FastAPI

Nik 1.4k Oct 23, 2021
FastAPI Admin Dashboard based on FastAPI and Tortoise ORM.

FastAPI ADMIN 中文文档 Introduction FastAPI-Admin is a admin dashboard based on fastapi and tortoise-orm. FastAPI-Admin provide crud feature out-of-the-bo

long2ice 929 Oct 23, 2021
python fastapi example connection to mysql

Quickstart Then run the following commands to bootstrap your environment with poetry: git clone https://github.com/xiaozl/fastapi-realworld-example-ap

null 31 Oct 12, 2021
api versioning for fastapi web applications

fastapi-versioning api versioning for fastapi web applications Installation pip install fastapi-versioning Examples from fastapi import FastAPI from f

Dean Way 303 Oct 21, 2021
row level security for FastAPI framework

Row Level Permissions for FastAPI While trying out the excellent FastApi framework there was one peace missing for me: an easy, declarative way to def

Holger Frey 187 Oct 17, 2021
TODO aplication made with Python's FastAPI framework and Hexagonal Architecture

FastAPI Todolist Description Todolist aplication made with Python's FastAPI framework and Hexagonal Architecture. This is a test repository for the pu

Giovanni Armane 40 Oct 14, 2021
a lightweight web framework based on fastapi

start-fastapi Version 2021, based on FastAPI, an easy-to-use web app developed upon Starlette Framework Version 2020 中文文档 Requirements python 3.6+ (fo

HiKari 61 Sep 29, 2021
Backend, modern REST API for obtaining match and odds data crawled from multiple sites. Using FastAPI, MongoDB as database, Motor as async MongoDB client, Scrapy as crawler and Docker.

Introduction Apiestas is a project composed of a backend powered by the awesome framework FastAPI and a crawler powered by Scrapy. This project has fo

Fran Lozano 43 Oct 9, 2021
Online Repo Browser

MSYS2 Web Interface A simple web interface for browsing the MSYS2 repos. Rebuild CSS/JS (optional): cd frontend npm install npm run build Run for Dev

MSYS2 44 Oct 19, 2021
Reusable utilities for FastAPI

Reusable utilities for FastAPI Documentation: https://fastapi-utils.davidmontague.xyz Source Code: https://github.com/dmontagu/fastapi-utils FastAPI i

David Montague 868 Oct 22, 2021
The template for building scalable web APIs based on FastAPI, Tortoise ORM and other.

FastAPI and Tortoise ORM. Powerful but simple template for web APIs w/ FastAPI (as web framework) and Tortoise-ORM (for working via database without h

prostomarkeloff 72 Oct 15, 2021
The base to start an openapi project featuring: SQLModel, Typer, FastAPI, JWT Token Auth, Interactive Shell, Management Commands.

The base to start an openapi project featuring: SQLModel, Typer, FastAPI, JWT Token Auth, Interactive Shell, Management Commands.

Bruno Rocha 56 Oct 17, 2021
Publish Xarray Datasets via a REST API.

Xpublish Publish Xarray Datasets via a REST API. Serverside: Publish a Xarray Dataset through a rest API ds.rest.serve(host="0.0.0.0", port=9000) Clie

xarray-contrib 70 Oct 18, 2021
sample web application built with FastAPI + uvicorn

SPARKY Sample web application built with FastAPI & Python 3.8 shows simple Flask-like structure with a Bootstrap template index.html also has a backgr

mrx 19 Jun 16, 2021