Makes google's political ad database actually useful

Last update: Apr 28, 2022

Overview

Making Google's political ad transparency library suck less

This is a series of scripts that takes Google's political ad transparency data and makes the ad content searchable as, ironically, the world's most powerful search company does not make their ad data searchable.

It can also takes the ad targeting information and map it to electorates, but this only works for postcodes at the moment so isn't in the main group of scripts yet.

It is aimed at Australian content, but most of the scripts could be applied to all ad content if you'd like to use it elsewhere.

Get the data

The current output is a work-in-progress, but you can find the latest file here as gzipped csv or json

What it does:

Gets the text content from text ads
Gets the YouTube title for YouTube ads
Gets the YouTube transcript for YouTube ads if it is available
Gets the image URL for image ads

Still to do:

OCR images and put the text in the database
Figure out if there's a good way to get text from animated html ads
Run the non-YouTube video ads through speech-to-text and put the text in the database
Archive ad content to S3 as Google removes it entirely when an ad is removed (eg UAP ads)
Archive ad database daily to S3 with a timestamp
Add electorates for Australian ads

You might also like...

A Persistent Embedded Graph Database for Python

Cog - Embedded Graph Database for Python cogdb.io New release: 2.0.5! Installing Cog pip install cogdb Cog is a persistent embedded graph database im

214 Dec 30, 2022

A Painless Simple Way To Create Schema and Do Database Operations Quickly In Python

PainlessDB - Taking Your Pain away to the moon 🚀 Contribute · Community · Documentation 🎫 Introduction : PainlessDB is a Python-based free and open-

3 Jul 15, 2022

HTTP graph database built in Python 3

KiwiDB HTTP graph database built in Python 3. Reference Format References are strings in the format: {refIDENTIFIER@GROUP} Authentication Currently, t

1 Dec 17, 2021

A NoSQL database made in python.

CookieDB A NoSQL database made in python.

1 Nov 30, 2022

Tiny local JSON database for Python.

Pylowdb Simple to use local JSON database 🦉 # This is pure python, not specific to pylowdb ;) db.data['posts'] = ({ 'id': 1, 'title': 'pylowdb is awe

3 Jan 26, 2022

PathfinderMonsterDatabase - A database of all monsters in Pathfinder 1e, created by parsing aonprd.com

PathfinderMonsterDatabase A database of all monsters in Pathfinder 1e, created by parsing aonprd.com Setup Run the following line to install all requi

11 Jun 12, 2022

ClutterDB - Extremely simple JSON database made for infrequent changes which behaves like a dict

extremely simple JSON database made for infrequent changes which behaves like a dict this was made for ClutterBot

1 Jan 12, 2022

Shelf DB is a tiny document database for Python to stores documents or JSON-like data

Shelf DB Introduction Shelf DB is a tiny document database for Python to stores documents or JSON-like data. Get it $ pip install shelfdb shelfquery S

35 Nov 3, 2022

A very simple document database

DockieDb A simple in-memory document database. Installation Build the Wheel Fork or clone this repository and run python setup.py bdist_wheel in the r

1 Jan 16, 2022

Comments

added parsing for html images and all supporting code

What does this change?

This adds support for html image parsing which has been tested with png images but should work with all image formats. Changes have been integrated across necessary scripts imageParser .py and ocrImages. A full functionality test still needs to be run.

Unfortunately Tesseract is not picking up text from the parsed html images,. This may be because they have transparent backgrounds and a possible solution could involve adding a white background.

opened by foroveralls 0
Missing col in sqlite table?
Looks like scraperwiki is looking for a "video_type" col - which doesn't automatically exist in the scraperwiki.sqlite file (at least not mine)

(manually adding that field allows script to run btw)

sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) no such column: video_type [SQL: select * from aus_ads where Ad_Type='Video' AND video_type IS NULL]

Cols I do automatically have:

Ad_ID Ad_URL Ad_Type Regions Advertiser_ID Advertiser_Name Ad_Campaigns_List Date_Range_Start Date_Range_End Num_of_Days Impressions Spend_USD First_Served_Timestamp Last_Served_Timestamp Age_Targeting Gender_Targeting Geo_Targeting_Included Geo_Targeting_Excluded Spend_Range_Min_USD Spend_Range_Max_USD Spend_Range_Min_AUD Spend_Range_Max_AUD
opened by dylanjmcconnell 0
requirements

Need to install some non-standard (at least for me) libraries

• youtube_transcript_api • scraperwiki

(...others I can see include pandas, simplejson, requests, pytesseract, PIL, cv2)

opened by dylanjmcconnell 0

Owner

The Guardian

The source code of the world's leading liberal voice

GitHub

TinyDB is a lightweight document oriented database optimized for your happiness :)

Quick Links Example Code Supported Python Versions Documentation Changelog Extensions Contributing Introduction TinyDB is a lightweight document orien

5.6k Dec 30, 2022

Python object-oriented database

ZODB, a Python object-oriented database ZODB provides an object-oriented database for Python that provides a high-degree of transparency. ZODB runs on

574 Dec 31, 2022

This is a simple graph database in SQLite, inspired by

This is a simple graph database in SQLite, inspired by "SQLite as a document database".

1.2k Jan 3, 2023

Elara DB is an easy to use, lightweight NoSQL database that can also be used as a fast in-memory cache.

Elara DB is an easy to use, lightweight NoSQL database written for python that can also be used as a fast in-memory cache for JSON-serializable data. Includes various methods and features to manipulate data structures in-memory, protect database files and export data.

101 Jan 4, 2023

Python function to extract all the rows from a SQLite database file while iterating over its bytes, such as while downloading it

16 Nov 9, 2022