A scrapy pipeline that provides an easy way to store files and images using various folder structures.

Panagiotis Simakis

Last update: Oct 23, 2022

Related tags

Web Crawling python3 scrapy scrapy-extension folder-structure folder-tree scrapy-pipeline

Overview

scrapy-folder-tree

This is a scrapy pipeline that provides an easy way to store files and images using various folder structures.

Supported folder structures:

Given this scraped file: 05b40af07cb3284506acbf395452e0e93bfc94c8.jpg, you can choose the following folder structures:

Using file name

full
├── 0
.   ├── 5
.   .   ├── b
.   .   .   ├── 05b40af07cb3284506acbf395452e0e93bfc94c8.jpg

Using crawling time

full
├── 0
.   ├── 11
.   .   ├── 48
.   .   .   ├── 05b40af07cb3284506acbf395452e0e93bfc94c8.jpg

Using crawling date

full
├── 2022
.   ├── 1
.   .   ├── 24
.   .   .   ├── 05b40af07cb3284506acbf395452e0e93bfc94c8.jpg

Installation

pip install scrapy_folder_tree

Usage

Use the following settings in your project:

ITEM_PIPELINES = {
    'scrapy_folder_tree.FilesHashTreePipeline': 300
}

FOLDER_TREE_DEPTH = 3

Visual scraping for Scrapy

Portia Portia is a tool that allows you to visually scrape websites without any programming knowledge required. With Portia you can annotate a web pag

8.7k Jan 5, 2023

Scrapy, a fast high-level web crawling & scraping framework for Python.

Scrapy Overview Scrapy is a fast high-level web crawling and web scraping framework, used to crawl websites and extract structured data from their pag

45.5k Jan 7, 2023

An experiment to deploy a serverless infrastructure for a scrapy project.

Serverless Scrapy project This project aims to evaluate the feasibility of an architecture based on serverless technology for a web crawler using scra

5 Jul 8, 2022

Snowflake database loading utility with Scrapy integration

Snowflake Stage Exporter Snowflake database loading utility with Scrapy integration. Meant for streaming ingestion of JSON serializable objects into S

0 Dec 6, 2021

Scraping news from Ucsal portal with Scrapy.

NewsScraping Esse é um projeto de raspagem das últimas noticias, de 2021, do portal da universidade Ucsal http://noosfero.ucsal.br/institucional Tecno

0 Sep 30, 2021

Fundamentus scrapy

Fundamentus_scrapy Baixa informacões que os outros scrapys do fundamentus não realizam. Para iniciar (python main.py), sera criado um arquivo chamado

1 Oct 24, 2021

Crawler do site Fundamentus.com com o uso do framework scrapy, tanto da aba detalhada como a de resumo.

Crawler do site Fundamentus.com com o uso do framework scrapy, tanto da aba detalhada como a de resumo. (Todas as infomações)

3 Oct 4, 2022

Scrapy-based cyber security news finder

Cyber-Security-News-Scraper Scrapy-based cyber security news finder Goal To keep up to date on the constant barrage of information within the field of

2 Nov 1, 2021

Iptvcrawl - A scrapy project for crawl IPTV playlist

iptvcrawl a scrapy project for crawl IPTV playlist. Dependency Python3 pip insta

18 May 5, 2022

Comments

Introduce `FOLDER_TREE_LENGTH` config parameter on `scrapy_folder_tree.FilesHashTreePipeline`

Given this filename: abcdefg.png and FOLDER_TREE_LENGTH=2

Currently, when using the scrapy_folder_tree.FilesHashTreePipeline pipeline, the file path will look like a/b/abcdefg.png.

An extra config value (FOLDER_TREE_LENGTH) should be added similar to FOLDER_TREE_LENGTH in order to allow the user to specify the folder length. For example, if FOLDER_TREE_LENGTH is equal to 2, the file path should look like ab/cd/abcdefg.png etc.

In order to support this functionality, the extra config should be added here and the filename should be split, based on the FOLDER_TREE_LENGTH here.
enhancement good first issue first-timers-only hacktoberfest

opened by sp1thas 12
Run unit-tests on Python 3.11
Hi @sp1thas - all have done :) You can check how it works in my fork: https://github.com/AlexNDRmac/scrapy-folder-tree/actions/runs/3249069844

Added:

Extended build matrix for testing job with Python 3.11

Changed:

Upgraded lxml from 4.7.1 to 4.9.1 for compatibility with Python 3.11

Updated poetry.lock

Updated checkout and setup-python actions

Closes #2
opened by AlexNDRmac 0