Download Web-10K data by querying Bing Image Search

Overview

gpv2-web10k

This repository contains the script to download images from the Web-10K dataset. The script takes in a list of queries, queries Bing Image Search, and downloads the returned thumbnail images to an Amazon S3 bucket the user specifies. To use this script, you will need a Bing Image Search API key.

Setup

python3 -mvenv venv
source venv/bin/activate
pip install -r requirements.txt

Adding the Bing Search API Key and Amazon S3 Bucket name

Add your API key to get_api_key() in tasks.py on Line 45.

Add the bucket name to tasks.py on Line 21. The images will be downloaded to this bucket.

Running the script

invoke query query_sample.json  # to query Bing Image Search with the queries listed in query_sample.json
invoke print-query-results "mt. everest"  # to print the results of a specific query
invoke generate-html  # to generate an HTML containing the returned images
invoke download-images  # to download the images to an Amazon S3 bucket

Useful links:

Bing Image Search API Pricing (for ~40K queries using an S3-tier instance, we paid about $160)

Bing Image Search API v7 query parameters (to change the returned response content)

Bing Image Search APIs v7 response objects (to understand the returned objects)

You might also like...
You Can download any video/image in all social medias very easy and High Speed.
You Can download any video/image in all social medias very easy and High Speed.

All-Downloader You Can download any video/image in all social medias very easy and High Speed. also you can easily download videos from web browsers s

Download history data from binance and save to dataframe or csv file

Binance history data downloader Download history data from binance and save to dataframe or csv file

Download India Stocks Historical Data
Download India Stocks Historical Data

Kite Helper - Download Stock Market Data 🌎 Website Simple Application to Download any stock market data in .csv format using Kite 🏃‍♂️ Running Serve

👻🟡  Download all Snapchat video & photo memories from a data export.
👻🟡 Download all Snapchat video & photo memories from a data export.

Snapchat "Memories" Fetcher In compliance with the California Consumer Privacy Act of 2018 (“CCPA”), businesses which collect and store user data must

Download candlestick data fast & easy for analysis
Download candlestick data fast & easy for analysis

crypto-candlesticks 📈 The goal behind this project is to facilitate downloading cryptocurrency candlestick data fast & simple. Currently only the Bit

A Celery application to collect data, download media and extract information from social media APIs

Project IBEX A Celery application to collect data, download media and extract information from social media APIs. Requirements You must have a Redis D

A modern CLI to download animes automatically from Twist
A modern CLI to download animes automatically from Twist

Kurby Kurby is a nice and simple CLI that use Twist website, and their huge collection to download animes for free and automatically Animes from Twist

Download courses from khanacademy.org

khan-dl A python script to download courses from Khan Academy using youtube-dl and beautifulsoup4.

A tool written in Python to download all Snapmaps content from a specific location.
A tool written in Python to download all Snapmaps content from a specific location.

snapmap-archiver A tool written in Python to download all Snapmaps content from a specific location.

Comments
  • Dataset open-source available

    Dataset open-source available

    Thank you for this interesting work and making the code available.

    The code for downloading query images is provided using Bing Image Search. I was wondering if the dataset can be made available via google drive etc, everyone would not need to use query scripts to download separately? Is that feasible?

    Thanks, Nirat

    opened by nirat1606 1
Owner
AI2
AI2
Python library to download bulk of images from Bing.com

Python library to download bulk of images form Bing.com. This package uses async url, which makes it very fast while downloading.

Guru Prasad Singh 105 Dec 14, 2022
Download and save Bing wallpapers and set as background for GNOME desktop

Save Bing wallpapers and set as background for GNOME desktop This script downloads the Bing wallpaper and sets it in the background of your gnome desk

manikamran 2 Nov 6, 2021
A simple python script to fetch Bing daily images and set them randomly using hsetroot

Daily bing walls A script to set the daily bing image as the wallpaper. It can also set from a random set of previously fetched wallpapers. usage: dai

Harinarayanan K S 5 Nov 23, 2022
A collection of modules I have created to programmatically search for/download imagery from live cam feeds across the state of California.

A collection of modules that I have created to programmatically search for/download imagery from all publicly available live cam feeds across the state of California. In no way am I affiliated with any of these organizations and these modules/methods of gathering imagery are completely unofficial.

Chad Groom 5 Nov 21, 2022
Python script designed to search and fetch direct download links from nxbrew.com

SwitchGamesDownloader Only for windows nxbrew.com is a website, accessible only using a proxy, where the majority of games for the Nintendo Switch are

Backend 91 Dec 28, 2022
Python based Telegram bot. Search and download YouTube video or audio.

Python-Telegram-Youtube-Media-Bot Python based Telegram bot. Search and download YouTube video or audio. Just change settings.py and start TelegramBot

Ahmet Bohur 2 Oct 2, 2022
Search the gallerys by tag and download pictures to the local

booruDownloader Search the gallerys by tag and download pictures to the local

null 6 Jun 30, 2022
Search & download music from a certain streaming service

Search & download music from a certain streaming service

mat 2 Mar 11, 2022
This is a python based web scraping bot for windows to download all ACCEPTED submissions of any user on Codeforces

CODEFORCES DOWNLOADER This is a python based web scraping bot for windows to download all ACCEPTED submissions of any user on Codeforces Requirements

Mohak 6 Dec 29, 2022