Data Extractor
Combine XPath, CSS Selectors and JSONPath for Web data extracting.
Quickstarts
Installation
Install the stable version from PYPI.
pip install "data-extractor[jsonpath-extractor]" # for extracting JSON data
pip install "data-extractor[lxml]" # for extracting HTML data
Or install the latest version from Github.
pip install "data-extractor[jsonpath-extractor] @ git+https://github.com/linw1995/data_extractor.git@master"
Extract JSON data
Currently supports to extract JSON data with below optional dependencies
install one dependency of them to extract JSON data.
Extract HTML(XML) data
Currently supports to extract HTML(XML) data with below optional dependencies
- lxml for using XPath
- cssselect for using CSS-Selectors
Usage
from data_extractor import Field, Item, JSONExtractor
class Count(Item):
followings = Field(JSONExtractor("countFollowings"))
fans = Field(JSONExtractor("countFans"))
class User(Item):
name_ = Field(JSONExtractor("name"), name="name")
age = Field(JSONExtractor("age"), default=17)
count = Count()
assert User(JSONExtractor("data.users[*]"), is_many=True).extract(
{
"data": {
"users": [
{
"name": "john",
"age": 19,
"countFollowings": 14,
"countFans": 212,
},
{
"name": "jack",
"description": "",
"countFollowings": 54,
"countFans": 312,
},
]
}
}
) == [
{"name": "john", "age": 19, "count": {"followings": 14, "fans": 212}},
{"name": "jack", "age": 17, "count": {"followings": 54, "fans": 312}},
]
Changelog
Unreleased
Feature
- Generic extractor with convertor (#83)
- mypy plugin for type annotation of extracting result (#83)
v0.10.2
Build
- upgrade jsonpath-extractor to v0.8.0
Contributing
Environment Setup
Clone the source codes from Github.
git clone https://github.com/linw1995/data_extractor.git
cd data_extractor
Setup the development environment. Please make sure you install the pdm, pre-commit and nox CLIs in your environment.
make init
make PYTHON=3.7 init # for specific python version
Linting
Use pre-commit for installing linters to ensure a good code style.
make pre-commit
Run linters. Some linters run via CLI nox, so make sure you install it.
make check-all
Testing
Run quick tests.
make
Run quick tests with verbose.
make vtest
Run tests with coverage. Testing in multiple Python environments is powered by CLI nox.
make cov
3.3k Jan 5, 2023
7 Jan 16, 2022
59 Nov 13, 2021
49 Dec 20, 2022
12.3k Jan 1, 2023
3k Jan 3, 2023
72 Aug 18, 2022
710 Jan 4, 2023
8 Oct 13, 2022
8 Dec 18, 2022
3 Sep 23, 2021
3 Jul 8, 2022
176 Jan 1, 2023
1 Nov 27, 2021
38 Nov 14, 2022
1 Dec 15, 2021
31 Aug 9, 2022
26 May 26, 2022
11 Sep 25, 2022
108 Jan 8, 2023