gpv2-web10k
This repository contains the script to download images from the Web-10K dataset. The script takes in a list of queries, queries Bing Image Search, and downloads the returned thumbnail images to an Amazon S3 bucket the user specifies. To use this script, you will need a Bing Image Search API key.
Setup
python3 -mvenv venv
source venv/bin/activate
pip install -r requirements.txt
Adding the Bing Search API Key and Amazon S3 Bucket name
Add your API key to get_api_key() in tasks.py on Line 45.
Add the bucket name to tasks.py on Line 21. The images will be downloaded to this bucket.
Running the script
invoke query query_sample.json # to query Bing Image Search with the queries listed in query_sample.json
invoke print-query-results "mt. everest" # to print the results of a specific query
invoke generate-html # to generate an HTML containing the returned images
invoke download-images # to download the images to an Amazon S3 bucket
Useful links:
Bing Image Search API Pricing (for ~40K queries using an S3-tier instance, we paid about $160)
Bing Image Search API v7 query parameters (to change the returned response content)
Bing Image Search APIs v7 response objects (to understand the returned objects)
14 Dec 16, 2022
10 Dec 2, 2022
12 Dec 6, 2022
18 Dec 26, 2022
31 Dec 11, 2022
4 Dec 15, 2022
48 Dec 22, 2022
806 Jan 3, 2023
46 Dec 9, 2022
105 Dec 14, 2022
2 Nov 6, 2021
5 Nov 23, 2022
13 Aug 27, 2022
5 Nov 21, 2022
91 Dec 28, 2022
2 Oct 2, 2022
6 Jun 30, 2022
2 Mar 11, 2022
6 Dec 29, 2022