Python Testing Crawler
🐍
🩺
🕷️
A crawler for automated functional testing of a web application
Crawling a server-side-rendered web application is a low cost way to get low quality test coverage of your JavaScript-light web application.
If you have only partial test coverage of your routes, but still want to protect against silly mistakes, then this is for you.
Features:
- Selectively spider pages and resources, or just request them
- Submit forms, and control what values to send
- Ignore links by source using CSS selectors
- Fail fast or collect many errors
- Configurable using straightforward rules
Works with the test clients for Flask (inc Flask-WebTest), Django and WebTest.
Why should I use this?
Here's an example: Flaskr, the Flask tutorial application has 166 lines of test code to achieve 100% test coverage.
Using Python Testing Crawler in a similar way to the Usage example below, we can hit 73% with very little effort. Disclaimer: Of course! It's not the same quality or utility of testing! But it is better than no tests, a complement to hand-written unit or functional tests and a useful stopgap.
Installation
$ pip install python-testing-crawler
Usage
Create a crawler using your framework's existing test client, tell it where to start and what rules to obey, then set it off:
from python_testing_crawler import Crawler
from python_testing_crawler import Rule, Request, Ignore, Allow
def test_crawl_all():
client = ## ... existing testing client
## ... any setup ...
crawler = Crawler(
client=my_testing_client,
initial_paths=['/'],
rules=[
Rule("a", '/.*', "GET", Request()),
]
)
crawler.crawl()
This will crawl all anchor links to relative addresses beginning "/". Any exceptions encountered will be collected and presented at the end of the crawl. For more power see the Rules section below.
If you need to authorise the client's session, e.g. login, then you should that before creating the Crawler.
It is also a good idea to create enough data, via fixtures or otherwise, to expose enough endpoints.
How do I setup a test client?
It depends on your framework:
- Flask: https://flask.palletsprojects.com/en/1.1.x/testing/
- Django: https://docs.djangoproject.com/en/3.0/topics/testing/tools/
Crawler Options
| Param | Description |
|---|---|
initial_paths |
list of paths/URLs to start from |
rules |
list of Rules to control the crawler; see below |
path_attrs |
list of attribute names to extract paths/URLs from; defaults to "href" -- include "src" if you want to check e.g.
, |
909 Dec 15, 2022
1 Nov 16, 2021
2.1k Dec 31, 2022
2.1k Jan 1, 2023
6 Jan 28, 2022
1 Nov 19, 2021
15 Feb 1, 2022
1.1k Dec 30, 2022
4 Sep 22, 2022
3k Jan 4, 2023
1.6k Jan 6, 2023
1.2k Jan 3, 2023
122 Nov 22, 2022
4.1k Nov 30, 2022
56 Dec 23, 2022
325 Dec 30, 2022
4 Jan 30, 2022
3 Mar 3, 2022