Feapder的管道扩展

Overview

FEAPDER 管道扩展

简介

此模块为feapderpipelines扩展,感谢广大开发者对feapder的贡献

随着feapder支持的pipelines越来越多,为减少feapder的体积,特将pipelines提出,使用者可按需安装

管道

PostgreSQL

贡献者:沈瑞祥

联系方式:[email protected]

安装

pip install feapder_pipelines[pgsql]

使用

feapder项目的setting.py中使用如下配置

# PostgreSQL
PGSQL_IP = 
PGSQL_PORT = 
PGSQL_DB = 
PGSQL_USER_NAME = 
PGSQL_USER_PASS = 

ITEM_PIPELINES = [
    "feapder_pipelines.pipelines.pgsql_pipeline.PgsqlPipeline"
]
细节 注:入库时 ON CONFLICT(key) 默认为id或通过如下sql查出来的第一个值
select column_names from(
        select
            t.relname as table_name,
            i.relname as index_name,
            array_to_string(array_agg(a.attname), ', ') as column_names
        from
            pg_class t,
            pg_class i,
            pg_index ix,
            pg_attribute a
        where
            t.oid = ix.indrelid
            and i.oid = ix.indexrelid
            and a.attrelid = t.oid
            and a.attnum = ANY(ix.indkey)
            and t.relkind = 'r'
            and t.relname like '%'
        group by
            t.relname,
            i.relname
        order by
            t.relname,
            i.relname) as res
    where table_name = 'table_name';
You might also like...
Releases(v1.0.3)
Owner
boris
爬虫工程师、数据工程师
boris