Pixel art search engine for opengameart

Overview

Pixel Art Reverse Image Search for OpenGameArt

What does the final search look like?

The final search with an example can be found here.

It looks like this: OpenGameArt Search

Why did I want a reverse image search for OpenGameArt?

I wanted to build a reverse image search for OpenGameArt as Google Image Search and TinEye don't give good results for it. I had previously generated a huge tile map to give an overview of similar images on OpenGameArt, but it wasn't very resource friendly on the web or image browser and had to be split into smaller files, plus it's not searchable in any way, just scrollable. So I wanted a way for people to explore what kind of art is available on OpenGameArt, and landed on using similarity search to browse the image space.

How did I do the crawling?

The first thing I had to do was retrieve the search results for the query I was interested in on OpenGameArt, mostly the 2D art. Then I had to retrieve each HTML page which was in the search results index and parse the HTML for links to files. OpenGameArt contains a lot of archive files like zip and rar files, so I then had to unpack them to get to the images.

For example here is a snippet showing how to parse the content page and get file links:

responseBody = await Common.ReadURIOrCache(blob, Common.BaseURI + page, client);

var htmlDoc = new HtmlDocument();
htmlDoc.LoadHtml(responseBody);
var htmlBody = htmlDoc.DocumentNode.SelectSingleNode("//body");

foreach (var nNode in htmlBody.Descendants("a"))
{
    if (nNode.NodeType == HtmlNodeType.Element &&
        nNode.Attributes["href"] != null &&
        nNode.Attributes["href"].Value.Contains("/default/files/"))
    {
        msg.Add(HttpUtility.HtmlDecode(nNode.Attributes["href"].Value.Replace(Common.FileURI, "")));
    }
}

Which technology did I use for the crawling and how much did it cost?

I used Azure Functions to do the crawling steps, with some back and forth manual intervention to correct things as needed. Each step had its own queue and then put the job for the next step on the next queue. In the end the invocations cost around 50 USD on Azure, for let's say 10-20 million Function invocations if I remember correctly.

Which alternatives did I investigate?

I tried to use the open source Milvus database, but it crashed on my DigitalOcean server because I didn't have enough memory on it. Then I accidentally and luckily discovered the link to Pinecone in a Hacker News comment section and decided to use that instead, as the trial was free and I didn't have to expand my server memory to use Milvus. In the end I expanded my server anyway, but I didn't try Milvus again (at least not yet).

What data do you need on each image to create a reverse image search?

I used VGG16 feature extraction in my script for this. See the article for more information, but in essence it's 4096 32-bit floating point numbers for each image, which describe various features of the image, say for instance in a very simplified way how many stripes or squares it has or how green it is. But these features are based on neurons in the neural network for VGG16 (which is usually used for image classification), so the features could be more complicated than what is described by simple feature tags. And the reason we need these vectors is that it's easy to use Euclidean distance or cosine similarity or another measure on two vectors to see if they are similar, and then consequently the images are similar. Furthermore there is search technology on these vectors that enable quick search on a large amount of them.

Here's a simplified python script to show how to do the feature extraction:

#!/usr/bin/env python3
# -*- coding: utf-8 -*-
# vim: ft=python ts=4 sw=4 sts=4 et fenc=utf-8

from tensorflow.keras.applications.vgg16 import VGG16
from tensorflow.keras.preprocessing import image
from tensorflow.keras.applications.vgg16 import decode_predictions, preprocess_input
from tensorflow.keras.models import Model
from tensorflow.compiler import xla
import numpy as np
import time
import os
import sys
import PIL
import json
import math
import multiprocessing
from glob import glob
from PIL import Image
from io import BytesIO

model = VGG16(weights='imagenet', include_top=True)
feat_extractor = Model(inputs=model.input, outputs=model.get_layer("fc2").output)

def prepImage(img):
    x = np.array(img.resize((224, 224)).convert('RGB'))
    x = np.expand_dims(x, axis=0)
    x = preprocess_input(x)
    return x

def main():
    'entry point'
    fname = 'demo.jpg'
    dt = Image.open(fname)
    pimg = prepImage(dt)

    print("Computing feature vector", fname)
    features = feat_extractor.predict(pimg)
    print(features)

if __name__ == '__main__':
    main()

Here's the output of the script:

emh@frostpunk ~/public_html/ogasearch 0% ./test.py                                                                                                                                                                                                                                                                                                                         (git)-[gh-pages]
2021-04-07 18:48:03.158023: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory
2021-04-07 18:48:03.158082: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
2021-04-07 18:48:07.783109: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2021-04-07 18:48:07.783485: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory
2021-04-07 18:48:07.783530: W tensorflow/stream_executor/cuda/cuda_driver.cc:326] failed call to cuInit: UNKNOWN ERROR (303)
2021-04-07 18:48:07.783580: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (frostpunk): /proc/driver/nvidia/version does not exist
2021-04-07 18:48:07.784058: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2021-04-07 18:48:07.784513: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2021-04-07 18:48:08.599925: W tensorflow/core/framework/cpu_allocator_impl.cc:80] Allocation of 411041792 exceeds 10% of free system memory.
2021-04-07 18:48:09.194634: W tensorflow/core/framework/cpu_allocator_impl.cc:80] Allocation of 411041792 exceeds 10% of free system memory.
2021-04-07 18:48:09.385612: W tensorflow/core/framework/cpu_allocator_impl.cc:80] Allocation of 411041792 exceeds 10% of free system memory.
2021-04-07 18:48:13.033066: W tensorflow/core/framework/cpu_allocator_impl.cc:80] Allocation of 411041792 exceeds 10% of free system memory.
Computing feature vector demo.jpg
2021-04-07 18:48:13.706621: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2)
2021-04-07 18:48:13.717564: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2199995000 Hz
[[0.        3.1128967 1.5611947 ... 1.2625191 0.7709812 0.       ]]
./test.py  12.20s user 4.66s system 132% cpu 12.731 total

How did I maintain a link between the image URLs and the vector datebase feature vectors?

I also wanted to put all the image URLs into a SQL database in the end, and have a flag for whether I had made the VGG16 feature extraction and whether it was added to the vector database (Milvus or Pinecone. It's essential to be able to map back and forth between an integer primary key, which is used in Pineone, and the URL and perhaps other metadata that belongs to the image, as [Pinecone](https://www.pinecone.io/ doesn't store other metadata than the primary key. In the end I dumbed the SQL database to a tab separated text file and loaded it on query server startup.

How long time did it take?

I think I spent a week in total to run all the code to finish, each step taking on the order of a day or two, crawl, computing feature vectors. I don't remember how much time it took to insert the vectors into the Pinecone database, but I think it was not the most time-consuming step.

Two ways of searching: Words and images

  • There are two ways of searching, either you can put a keyword, which just plainly (and a bit slowly at O(n)) iterates linearly through the URLs looking for a string match. I stuck with linear search since it's simple to implement and all the URLs are kept in memory anyway so it's not that slow. I dumped all the URLs to a text file and loading it to memory on query server load instead of querying the SQL server each time.
  • The other way of searching is that you put an image URL, which will run feature extraction on your image (on my server) and then query Pinecone for similar vectors, which will map to primary keys, which in turn I look up in the list of URLs.
  • I also maintain a "reverse database" text file in order to link back to the OpenGameArt site for images found (there are some bugs with this I haven't fixed yet, in which case it just links to OpenGameArt main page). This file is also loaded on query server startup. Finally there is also a link under each image to search for similar images, which implicitly uses the second kind of query by image.

What are some problems I encountered?

At the end I also added a quick fix to remove near-duplicate image results which had an identical score. I ran into some troubles on the search page with "double" URL encoding, because I had stored the files using URL encoding in the file system, but I worked around it with some detection code on the frontend for when the browser double-encoded the URL-encoded file names. I recommend storing the crawled files without URL encoding. I regret that my scripts are not so high quality or polished, for example there are multiple steps in scripts and I change things by editing the script instead of taking command line arguments. I don't feel like posting snippets from the scripts and explaining as they are a bit messy. Additionally I moved the files from Azure storage to my DigitalOcean server mid-way, before processing the feature extraction, so there's some inconsistent data location handling.

What are the final takeaways?

  • I recommend doing the crawl perhaps on a cheaper substrate than Azure Functions and Azure Storage to save some money, for example your own server or a fixed price Cloud server. Well it just cost 50 USD but I could have done it for free on my DigitalOcean server, so that's why.
  • I recommend building a more robust crawler, idempotent and restartable at any point which it may terminate at or require some manual intervention (for example I exceeded the Azure Function maximum run time of 5 minutes on extracting some of the large zip files, so I extracted them running the Functions locally in VS Code).
  • One thing I regret that I didn't get done this time was to extract all the tiles from tile sheets to individual images for searching. That would have made the search even more useful. On the other hand it could have cluttered the similarity search with too many nearly identical images.

Conclusion and final remarks

  • It might also be useful to prototype the system using a bit of content, then run the whole pipeline end to end on all the content once you get it working, instead of completing the first step in crawl, then doing all feature extraction and then doing all database insertion as I did.
  • In conclusion what I made was a bit of a hack, not so robust scripting for updating for new content, but it worked fine as a prototype and gave decent image search results (not always so spot on, but I blame it on that the feature extraction is not really targeted at tiny pixel art (though resized/upscaled before feature extraction)).
  • It could be interesting to see whether Milvus could also deliver similar results, a side by side comparison of some kind, on speed and quality, but I found it much easier to use Pinecone since it's already up and running as a service so that I didn't have to run my own vector database.

Script Locations

You might also like...
Isearch (OSINT) 🔎 Face recognition reverse image search on Instagram profile feed photos.
Isearch (OSINT) 🔎 Face recognition reverse image search on Instagram profile feed photos.

isearch is an OSINT tool on Instagram. Offers a face recognition reverse image search on Instagram profile feed photos.

Deep Text Search is an AI-powered multilingual text search and recommendation engine with state-of-the-art transformer-based multilingual text embedding (50+ languages).
Deep Text Search is an AI-powered multilingual text search and recommendation engine with state-of-the-art transformer-based multilingual text embedding (50+ languages).

Deep Text Search - AI Based Text Search & Recommendation System Deep Text Search is an AI-powered multilingual text search and recommendation engine w

Deep Image Search is an AI-based image search engine that includes deep transfor learning features Extraction and tree-based vectorized search.
Deep Image Search is an AI-based image search engine that includes deep transfor learning features Extraction and tree-based vectorized search.

Deep Image Search - AI-Based Image Search Engine Deep Image Search is an AI-based image search engine that includes deep transfer learning features Ex

A Python-based application demonstrating various search algorithms, namely Depth-First Search (DFS), Breadth-First Search (BFS), and A* Search (Manhattan Distance Heuristic)
A Python-based application demonstrating various search algorithms, namely Depth-First Search (DFS), Breadth-First Search (BFS), and A* Search (Manhattan Distance Heuristic)

A Python-based application demonstrating various search algorithms, namely Depth-First Search (DFS), Breadth-First Search (BFS), and the A* Search (using the Manhattan Distance Heuristic)

A fast, dataset-agnostic, deep visual search engine for digital art history

imgs.ai imgs.ai is a fast, dataset-agnostic, deep visual search engine for digital art history based on neural network embeddings. It utilizes modern

Pixel art as well as various sets for hand crafting
Pixel art as well as various sets for hand crafting

Pixel art as well as various sets for hand crafting

👾 Python project to help you convert any image into a pixel art.
👾 Python project to help you convert any image into a pixel art.

👾 Pixel Art Generator Python project to help you convert any image into a pixel art. ⚙️ Developer's Guide Things you need to get started with this co

Pixelarticons - Pixel Art Icons made simple for Flutter, powered by pixelarticons and fontify
Pixelarticons - Pixel Art Icons made simple for Flutter, powered by pixelarticons and fontify

Pixelarticons - Pixel Art Icons made simple for Flutter, powered by pixelarticons and fontify

This repo contains the code and data used in the paper
This repo contains the code and data used in the paper "Wizard of Search Engine: Access to Information Through Conversations with Search Engines"

Wizard of Search Engine: Access to Information Through Conversations with Search Engines by Pengjie Ren, Zhongkun Liu, Xiaomeng Song, Hongtao Tian, Zh

 Deep Image Search - AI-Based Image Search Engine
Deep Image Search - AI-Based Image Search Engine

Deep Image Search is an AI-based image search engine that includes deep transfer learning features Extraction and tree-based vectorized search technique.

Connectionist Temporal Classification (CTC) decoding algorithms: best path, beam search, lexicon search, prefix search, and token passing. Implemented in Python.

CTC Decoding Algorithms Update 2021: installable Python package Python implementation of some common Connectionist Temporal Classification (CTC) decod

Open Source Discord bot with many cool features like Weather, Balance, Avatar, User, Server, RP-commands, Gif search, YouTube search, VK post search etc.

Сокобот Дискорд бот с открытым исходным кодом. Содержит в себе экономику, полезные команды (!аватар, !юзер, !сервер и тд.), рп-команды (!обнять, !глад

Generative Art Synthesizer - a python program that generates python programs that generates generative art
Generative Art Synthesizer - a python program that generates python programs that generates generative art

GAS - Generative Art Synthesizer Generative Art Synthesizer - a python program that generates python programs that generates generative art. Examples

Img-to-ascii-art - Converter of image to ascii art

img-to-ascii-art Converter of image to ascii art Latest Features. Intoducing Col

Ascify-Art - An easy to use, GUI based and user-friendly colored ASCII art generator from images!
Ascify-Art - An easy to use, GUI based and user-friendly colored ASCII art generator from images!

Ascify-Art This is a python based colored ASCII art generator for free! How to Install? You can download and use the python version if you want, modul

An awesome list of AI for art and design - resources, and popular datasets and how we may apply computer vision tasks to art and design.
An awesome list of AI for art and design - resources, and popular datasets and how we may apply computer vision tasks to art and design.

Awesome AI for Art & Design An awesome list of AI for art and design - resources, and popular datasets and how we may apply computer vision tasks to a

A 3D engine powered by ASCII art
A 3D engine powered by ASCII art

3D engine powered by ASCII art

:mag: End-to-End Framework for building natural language search interfaces to data by utilizing Transformers and the State-of-the-Art of NLP. Supporting DPR, Elasticsearch, HuggingFace’s Modelhub and much more!
:mag: End-to-End Framework for building natural language search interfaces to data by utilizing Transformers and the State-of-the-Art of NLP. Supporting DPR, Elasticsearch, HuggingFace’s Modelhub and much more!

Haystack is an end-to-end framework that enables you to build powerful and production-ready pipelines for different search use cases. Whether you want

Comments
  • Bump SharpCompress from 0.28.1 to 0.29.0

    Bump SharpCompress from 0.28.1 to 0.29.0

    Bumps SharpCompress from 0.28.1 to 0.29.0.

    Release notes

    Sourced from SharpCompress's releases.

    0.29.0 - minor behavior change and fixes

    Two minor fixes from contributors and a behavior fix for extraction. Now the destination directory has to exist before extracting will be allowed.

    More good fixes!

    Minor bug fixes

    Two issues found and squashed

    Commits
    • ea5c8dc Merge pull request #614 from adamhathcock/ensure-dest-dir-exists
    • 0209d00 Minor updates and prep for 0.29
    • a8d065d Ensure destination directory exists
    • 7bd9711 Merge pull request #610 from cyr/master
    • 61802ea Merge branch 'adamhathcock:master' into master
    • b425659 Merge pull request #611 from Thunderstr1k3/fix-zipheader-seeking
    • 3e32e3d Allowing to seek empty zip files
    • 1b661c9 Fixed bug where large (int32+ file size) adds an additional 512 bytes of padd...
    • 54fc26b Update build and mark for 0.28.3
    • 161f99b Merge pull request #601 from salvois/master
    • Additional commits viewable in compare view

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    • @dependabot use these labels will set the current labels as the default for future PRs for this repo and language
    • @dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language
    • @dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language
    • @dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

    You can disable automated security fix PRs for this repo from the Security Alerts page.

    dependencies 
    opened by dependabot[bot] 0
Owner
Eivind Magnus Hvidevold
Eivind Magnus Hvidevold
Ocular is a state-of-the-art historical OCR system.

Ocular Ocular is a state-of-the-art historical OCR system. Its primary features are: Unsupervised learning of unknown fonts: requires only document im

null 228 Dec 30, 2022
Tesseract Open Source OCR Engine (main repository)

Tesseract OCR About This package contains an OCR engine - libtesseract and a command line program - tesseract. Tesseract 4 adds a new neural net (LSTM

null 48.4k Jan 9, 2023
Line based ATR Engine based on OCRopy

OCR Engine based on OCRopy and Kraken using python3. It is designed to both be easy to use from the command line but also be modular to be integrated

null 948 Dec 23, 2022
A Tensorflow model for text recognition (CNN + seq2seq with visual attention) available as a Python package and compatible with Google Cloud ML Engine.

Attention-based OCR Visual attention-based OCR model for image recognition with additional tools for creating TFRecords datasets and exporting the tra

Ed Medvedev 933 Dec 29, 2022
OCR engine for all the languages

Description kraken is a turn-key OCR system optimized for historical and non-Latin script material. kraken's main features are: Fully trainable layout

null 431 Jan 4, 2023
It is a image ocr tool using the Tesseract-OCR engine with the pytesseract package and has a GUI.

OCR-Tool It is a image ocr tool made in Python using the Tesseract-OCR engine with the pytesseract package and has a GUI. This is my second ever pytho

Khant Htet Aung 4 Jul 11, 2022
Convert PDF/Image to TXT using EasyOcr - the best OCR engine available!

PDFImage2TXT - DOWNLOAD INSTALLER HERE What can you do with it? Convert scanned PDFs to TXT. Convert scanned Documents to TXT. No coding required!! In

Hans Alemão 2 Feb 22, 2022
nofacedb/faceprocessor is a face recognition engine for NoFaceDB program complex.

faceprocessor nofacedb/faceprocessor is a face recognition engine for NoFaceDB program complex. Tech faceprocessor uses a number of open source projec

NoFaceDB 3 Sep 6, 2021
This pyhton script converts a pdf to Image then using tesseract as OCR engine converts Image to Text

Script_Convertir_PDF_IMG_TXT Este script de pyhton convierte un pdf en Imagen luego utilizando tesseract como motor OCR convierte la Imagen a Texto. p

alebogado 1 Jan 27, 2022
Code for AAAI 2021 paper: Sequential End-to-end Network for Efficient Person Search

This repository hosts the source code of our paper: [AAAI 2021]Sequential End-to-end Network for Efficient Person Search. SeqNet achieves the state-of

Zj Li 218 Dec 31, 2022