Introduction
A dead simple crawler to get books information from Douban.
Pre-requesites
- Python 3
- Install dependencies from
requirements.txt - (Optional) Install Anaconda to handle environment
Usage
On Local Machine
- Run
get_tagsto fetch all the trending tags.
# This will generate a file tags.csv
python app.py get_tags -o /your-output-dir
- Run
crawl_booksto start crawling the books by the tags from the previous step.
python app.py crawl_books -i /some-where/tags.csv -o /your-output-dir
Certainly, you can create the tags.csv without using the
get_tagsscript. You might want to make sure the tags you specified can lead to any actual result of books.
Docker Compose
You'll need to install Docker and the docker-compose command before proceeding.
docker-compose build --no-cache && docker-compose up -d --force-recreate
You can omit --force-recreate if you want to keep the container when the configuration or the image hasn't changed.
License
MIT © mogita
4 Dec 3, 2022
2 Jan 24, 2022
1 Oct 5, 2021
2 Apr 29, 2022
17 Jan 24, 2022
3 Oct 4, 2022
1 Nov 14, 2021
13 Dec 27, 2022
46 Dec 16, 2022
1 Dec 3, 2022
2 Feb 17, 2022
15 May 17, 2022
1 Jan 10, 2022
15.7k Jan 4, 2023
2.9k Jan 3, 2023
9.3k Jan 2, 2023
8 Sep 20, 2021
10 Jul 3, 2022
2 Mar 12, 2022