An easier way to build neural search on the cloud
Jina is a deep learning-powered search framework for building cross-/multi-modal search systems (e.g. text, images, video, audio) on the cloud.
Docs โข Hello World โข Quick Start โข Learn โข Examples โข Contribute โข Jobs โข Website โข Slack
Installation
|
x86/64,arm/v6,v7,v8 (Apple M1) |
On Linux/macOS & Python 3.7/3.8/3.9 | Docker Users |
|---|---|---|
| Standard | pip install -U jina |
docker run jinaai/jina:latest |
| Daemon | pip install -U "jina[daemon]" |
docker run --network=host jinaai/jina:latest-daemon |
| With Extras | pip install -U "jina[devel]" |
docker run jinaai/jina:latest-devel |
| Dev/Pre-Release | pip install --pre jina |
docker run jinaai/jina:master |
Version identifiers are explained here. To install Jina with extra dependencies please refer to the docs. Jina can run on Windows Subsystem for Linux. We welcome the community to help us with native Windows support.
YAML Completion in IDE
Developing Jina app often means writing YAML configs. We provide a JSON Schema for your IDE to enable code completion, syntax validation, members listing and displaying help text. Here is a video tutorial to walk you through the setup.
 |
PyCharm
|
 |
VSCode
"yaml.schemas": {
"https://api.jina.ai/schemas/latest.json": ["/*.jina.yml", "/*.jaml"],
}
|
Jina "Hello, World!"
๐
๐
Just starting out? Try Jina's "Hello, World" - jina hello --help
๐
Fashion Image Search
A simple image neural search demo for Fashion-MNIST. No extra dependencies needed, simply run:
jina hello fashion # more options in --help
...or even easier for Docker users, no install required:
docker run -v "$(pwd)/j:/j" jinaai/jina hello fashion --workdir /j && open j/hello-world.html
# replace "open" with "xdg-open" on Linux
Click here to see console output
๐ค
Covid-19 Chatbot
For NLP engineers, we provide a simple chatbot demo for answering Covid-19 questions. To run that:
pip install "jina[chatbot]"
jina hello chatbot
This downloads CovidQA dataset and tells Jina to index 418 question-answer pairs with DistilBERT. The index process takes about 1 minute on CPU. Then it opens a web page where you can input questions and ask Jina.
๐ช Multimodal Document Search
A multimodal-document contains multiple data types, e.g. a PDF document often contains figures and text. Jina lets you build a multimodal search solution in just minutes. To run our minimum multimodal document search demo:
pip install "jina[multimodal]"
jina hello multimodal
This downloads people image dataset and tells Jina to index 2,000 image-caption pairs with MobileNet and DistilBERT. The index process takes about 3 minute on CPU. Then it opens a web page where you can query multimodal documents. We have prepared a YouTube tutorial to walk you through this demo.
Get Started
|
|
CRUD Functions โข Document โข Flow |
|
|
Feed Data โข Fetch Result โข Add Logic โข Inter & Intra Parallelism โข Decentralize โข Asynchronous |
|
|
Customize Encoder โข Test Encoder โข Parallelism & Batching โข Add Data Indexer โข Compose Flow from YAML โข Search โข Evaluation โข REST Interface |
๐ฅ
Fundamentals
CRUD Functions
First we look at basic CRUD operations. In Jina, CRUD corresponds to four functions: index (create), search (read), update, and delete. With Documents below as an example:
import numpy as np
from jina import Document
docs = [Document(id='๐ฒ', embedding=np.array([0, 0]), tags={'guardian': 'Azure Dragon', 'position': 'East'}),
Document(id='๐ฆ', embedding=np.array([1, 0]), tags={'guardian': 'Vermilion Bird', 'position': 'South'}),
Document(id='๐ข', embedding=np.array([0, 1]), tags={'guardian': 'Black Tortoise', 'position': 'North'}),
Document(id='๐ฏ', embedding=np.array([1, 1]), tags={'guardian': 'White Tiger', 'position': 'West'})]
Let's build a Flow with a simple indexer:
from jina import Flow
f = Flow().add(uses='_index')
Document and Flow are basic concepts in Jina, which will be explained later. _index is a built-in embedding + structured storage that you can use out of the box.
| Index |
# save four docs (both embedding and structured info) into storage
with f:
f.index(docs, on_done=print)
|
| Search |
# retrieve top-3 neighbours of ๐ฒ, this print ๐ฒ๐ฆ๐ข with score 0, 1, 1 respectively
with f:
f.search(docs[0], top_k=3, on_done=lambda x: print(x.docs[0].matches))
{"id": "๐ฒ", "tags": {"guardian": "Azure Dragon", "position": "East"}, "embedding": {"dense": {"buffer": "AAAAAAAAAAAAAAAAAAAAAA==", "shape": [2], "dtype": "<i8"}}, "score": {"opName": "NumpyIndexer", "refId": "๐ฒ"}, "adjacency": 1}
{"id": "๐ฆ", "tags": {"position": "South", "guardian": "Vermilion Bird"}, "embedding": {"dense": {"buffer": "AQAAAAAAAAAAAAAAAAAAAA==", "shape": [2], "dtype": "<i8"}}, "score": {"value": 1.0, "opName": "NumpyIndexer", "refId": "๐ฒ"}, "adjacency": 1}
{"id": "๐ข", "tags": {"guardian": "Black Tortoise", "position": "North"}, "embedding": {"dense": {"buffer": "AAAAAAAAAAABAAAAAAAAAA==", "shape": [2], "dtype": "<i8"}}, "score": {"value": 1.0, "opName": "NumpyIndexer", "refId": "๐ฒ"}, "adjacency": 1}
|
| Update |
# update ๐ฒ embedding in the storage
docs[0].embedding = np.array([1, 1])
with f:
f.update(docs[0])
|
| Delete |
# remove ๐ฆ๐ฒ Documents from the storage
with f:
f.delete(['๐ฆ', '๐ฒ'])
|
For further details about CRUD functionality, checkout docs.jina.ai.
Document
Document is Jina's primitive data type. It can contain text, image, array, embedding, URI, and be accompanied by rich meta information. To construct a Document, you can use:
import numpy
from jina import Document
doc1 = Document(content=text_from_file, mime_type='text/x-python') # a text document contains python code
doc2 = Document(content=numpy.random.random([10, 10])) # a ndarray document
A Document can be recursed both vertically and horizontally to have nested Documents and matched Documents. To better see the Document's recursive structure, you can use .plot() function. If you are using JupyterLab/Notebook, all Document objects will be auto-rendered.
import numpy
from jina import Document
d0 = Document(id='๐ฒ', embedding=np.array([0, 0]))
d1 = Document(id='๐ฆ', embedding=np.array([1, 0]))
d2 = Document(id='๐ข', embedding=np.array([0, 1]))
d3 = Document(id='๐ฏ', embedding=np.array([1, 1]))
d0.chunks.append(d1)
d0.chunks[0].chunks.append(d2)
d0.matches.append(d3)
d0.plot() # simply `d0` on JupyterLab
|
 |
Click here to see more about MultimodalDocument
MultimodalDocument
A MultimodalDocument is a document composed of multiple Document from different modalities (e.g. text, image, audio).
Jina provides multiple ways to build a multimodal Document. For example, you can provide the modality names and the content in a dict:
from jina import MultimodalDocument
document = MultimodalDocument(modality_content_map={
'title': 'my holiday picture',
'description': 'the family having fun on the beach',
'image': PIL.Image.open('path/to/image.jpg')
})
One can also compose a MultimodalDocument from multiple Document directly:
from jina.types import Document, MultimodalDocument
doc_title = Document(content='my holiday picture', modality='title')
doc_desc = Document(content='the family having fun on the beach', modality='description')
doc_img = Document(content=PIL.Image.open('path/to/image.jpg'), modality='image')
doc_img.tags['date'] = '10/08/2019'
document = MultimodalDocument(chunks=[doc_title, doc_description, doc_img])
Fusion Embeddings from Different Modalities
To extract fusion embeddings from different modalities Jina provides BaseMultiModalEncoder abstract class, which has a unique encode interface.
def encode(self, *data: 'numpy.ndarray', **kwargs) -> 'numpy.ndarray':
...
MultimodalDriver provides data to the MultimodalDocument in the correct expected order. In this example below, image embedding is passed to the encoder as the first argument, and text as the second.
jtype: MyMultimodalEncoder
with:
positional_modality: ['image', 'text']
requests:
on:
[IndexRequest, SearchRequest]:
- jtype: MultiModalDriver {}
Interested readers can refer to jina-ai/example: how to build a multimodal search engine for image retrieval using TIRG (Composing Text and Image for Image Retrieval) for the usage of MultimodalDriver and BaseMultiModalEncoder in practice.
Flow
Jina provides a high-level Flow API to simplify building CRUD workflows. To create a new Flow:
from jina import Flow
f = Flow().add()
This creates a simple Flow with one Pod. You can chain multiple .add()s in a single Flow.
To visualize the Flow, simply chain it with .plot('my-flow.svg'). If you are using a Jupyter notebook, the Flow object will be displayed inline without plot.
Gateway is the entrypoint of the Flow.
Get the vibe? Now we're talking! Let's learn more about the basic concepts and features of Jina:
|
|
CRUD Functions โข Document โข Flow |
|
|
Feed Data โข Fetch Result โข Add Logic โข Inter & Intra Parallelism โข Decentralize โข Asynchronous |
|
|
Customize Encoder โข Test Encoder โข Parallelism & Batching โข Add Data Indexer โข Compose Flow from YAML โข Search โข Evaluation โข REST Interface |
๐ฃ
Basic
Feed Data
To use a Flow, open it via with context manager, like you would open a file in Python. Now let's create some empty Documents and index them:
from jina import Document
with Flow().add() as f:
f.index((Document() for _ in range(10)))
Flow supports CRUD operations: index, search, update, delete. In addition, it also provides sugary syntax on ndarray, csv, ndjson and arbitrary files.
| Input | Example of index/search |
Explain |
numpy.ndarray |
with f:
f.index_ndarray(numpy.random.random([4,2]))
|
Input four |
| CSV |
with f, open('index.csv') as fp:
f.index_csv(fp, field_resolver={'pic_url': 'uri'})
|
Each line in |
JSON Lines/ndjson/LDJSON |
with f, open('index.ndjson') as fp:
f.index_ndjson(fp, field_resolver={'question_id': 'id'})
|
Each line in |
| Files with wildcards |
with f:
f.index_files(['/tmp/*.mp4', '/tmp/*.pdf'])
|
Each file captured is constructed as a |
Fetch Result
Once a request is done, callback functions are fired. Jina Flow implements a Promise-like interface: You can add callback functions on_done, on_error, on_always to hook different events. In the example below, our Flow passes the message then prints the result when successful. If something goes wrong, it beeps. Finally, the result is written to output.txt.
def beep(*args):
# make a beep sound
import os
os.system('echo -n "\a";')
with Flow().add() as f, open('output.txt', 'w') as fp:
f.index(numpy.random.random([4, 5, 2]),
on_done=print, on_error=beep, on_always=lambda x: fp.write(x.json()))
Add Logic
To add logic to the Flow, use the uses parameter to attach a Pod with an Executor. uses accepts multiple value types including class name, Docker image, (inline) YAML or built-in shortcut.
f = (Flow().add(uses=MyBertEncoder) # the class of a Jina Executor
.add(uses='docker://jinahub/pod.encoder.dummy_mwu_encoder:0.0.6-0.9.3') # the image name
.add(uses='myencoder.yml') # YAML serialization of a Jina Executor
.add(uses='!WaveletTransformer | {freq: 20}') # inline YAML config
.add(uses='_pass') # built-in shortcut executor
.add(uses={'__cls': 'MyBertEncoder', 'with': {'param': 1.23}})) # dict config object with __cls keyword
The power of Jina lies in its decentralized architecture: Each add creates a new Pod, and these Pods can be run as a local thread/process, a remote process, inside a Docker container, or even inside a remote Docker container.
Inter & Intra Parallelism
Chaining .add()s creates a sequential Flow. For parallelism, use the needs parameter:
f = (Flow().add(name='p1', needs='gateway')
.add(name='p2', needs='gateway')
.add(name='p3', needs='gateway')
.needs(['p1','p2', 'p3'], name='r1').plot())
p1, p2, p3 now subscribe to Gateway and conduct their work in parallel. The last .needs() blocks all Pods until they finish their work. Note: parallelism can also be performed inside a Pod using parallel:
f = (Flow().add(name='p1', needs='gateway')
.add(name='p2', needs='gateway')
.add(name='p3', parallel=3)
.needs(['p1','p3'], name='r1').plot())
Decentralized Flow
A Flow does not have to be local-only: You can put any Pod to remote(s). In the example below, with the host keyword gpu-pod, is put to a remote machine for parallelization, whereas other Pods stay local. Extra file dependencies that need to be uploaded are specified via the upload_files keyword.
| 123.456.78.9 |
# have docker installed
docker run --name=jinad --network=host -v /var/run/docker.sock:/var/run/docker.sock jinaai/jina:latest-daemon --port-expose 8000
# to stop it
docker rm -f jinad
|
| Local |
import numpy as np
from jina import Flow
f = (Flow()
.add()
.add(name='gpu_pod',
uses='mwu_encoder.yml',
host='123.456.78.9:8000',
parallel=2,
upload_files=['mwu_encoder.py'])
.add())
with f:
f.index_ndarray(np.random.random([10, 100]), output=print)
|
We provide a demo server on cloud.jina.ai:8000, give the following snippet a try!
from jina import Flow
with Flow().add().add(host='cloud.jina.ai:8000') as f:
f.index(['hello', 'world'])
Asynchronous Flow
While synchronous from outside, Jina runs asynchronously under the hood: it manages the eventloop(s) for scheduling the jobs. If the user wants more control over the eventloop, then AsyncFlow can be used.
Unlike Flow, the CRUD of AsyncFlow accepts input and output functions as async generators. This is useful when your data sources involve other asynchronous libraries (e.g. motor for MongoDB):
from jina import AsyncFlow
async def input_function():
for _ in range(10):
yield Document()
await asyncio.sleep(0.1)
with AsyncFlow().add() as f:
async for resp in f.index(input_function):
print(resp)
AsyncFlow is particularly useful when Jina is using as part of integration, where another heavy-lifting job is running concurrently:
async def run_async_flow_5s(): # WaitDriver pause 5s makes total roundtrip ~5s
with AsyncFlow().add(uses='- !WaitDriver {}') as f:
async for resp in f.index_ndarray(numpy.random.random([5, 4])):
print(resp)
async def heavylifting(): # total roundtrip takes ~5s
print('heavylifting other io-bound jobs, e.g. download, upload, file io')
await asyncio.sleep(5)
print('heavylifting done after 5s')
async def concurrent_main(): # about 5s; but some dispatch cost, can't be just 5s, usually at <7s
await asyncio.gather(run_async_flow_5s(), heavylifting())
if __name__ == '__main__':
asyncio.run(concurrent_main())
AsyncFlow is very useful when using Jina inside a Jupyter Notebook. As Jupyter/ipython already manages an eventloop and thanks to autoawait, AsyncFlow can run out-of-the-box in Jupyter.
That's all you need to know for understanding the magic behind hello-world. Now let's dive deeper into it!
|
|
CRUD Functions โข Document โข Flow |
|
|
Feed Data โข Fetch Result โข Add Logic โข Inter & Intra Parallelism โข Decentralize โข Asynchronous |
|
|
Customize Encoder โข Test Encoder โข Parallelism & Batching โข Add Data Indexer โข Compose Flow from YAML โข Search โข Evaluation โข REST Interface |
๐ฅ
Breakdown of hello-world
Customize Encoder
Let's first build a naive image encoder that embeds images into vectors using an orthogonal projection. To do this, we simply inherit from BaseImageEncoder: a base class from the jina.executors.encoders module. We then override its __init__() and encode() methods.
import numpy as np
from jina.executors.encoders import BaseImageEncoder
class MyEncoder(BaseImageEncoder):
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
np.random.seed(1337)
H = np.random.rand(784, 64)
u, s, vh = np.linalg.svd(H, full_matrices=False)
self.oth_mat = u @ vh
def encode(self, data: 'np.ndarray', *args, **kwargs):
return (data.reshape([-1, 784]) / 255) @ self.oth_mat
Jina provides a family of Executor classes, which summarize frequently-used algorithmic components in neural search. This family consists of encoders, indexers, crafters, evaluators, and classifiers, each with a well-designed interface. You can find the list of all 107 built-in executors here. If they don't meet your needs, inheriting from one of them is the easiest way to bootstrap your own Executor. Simply use our Jina Hub CLI:
pip install jina[hub] && jina hub new
Test Encoder in Flow
Let's test our encoder in the Flow with some synthetic data:
def validate(req):
assert len(req.docs) == 100
assert NdArray(req.docs[0].embedding).value.shape == (64,)
f = Flow().add(uses='MyEncoder')
with f:
f.index_ndarray(numpy.random.random([100, 28, 28]), on_done=validate)
All good! Now our validate function confirms that all one hundred 28x28 synthetic images have been embedded into 100x64 vectors.
Parallelism & Batching
By setting a larger input, you can play with batch_size and parallel:
f = Flow().add(uses='MyEncoder', parallel=10)
with f:
f.index_ndarray(numpy.random.random([60000, 28, 28]), batch_size=1024)
Add Data Indexer
Now we need to add an indexer to store all the embeddings and the image for later retrieval. Jina provides a simple numpy-powered vector indexer NumpyIndexer, and a key-value indexer BinaryPbIndexer. We can combine them in a single YAML file:
jtype: CompoundIndexer
components:
- jtype: NumpyIndexer
with:
index_filename: vec.gz
- jtype: BinaryPbIndexer
with:
index_filename: chunk.gz
metas:
workspace: ./
jtype:defines the class name of the structure;with:defines arguments for initializing this class object.
Essentially, the above YAML config is equivalent to the following Python code:
from jina.executors.indexers.vector import NumpyIndexer
from jina.executors.indexers.keyvalue import BinaryPbIndexer
from jina.executors.indexers import CompoundIndexer
a = NumpyIndexer(index_filename='vec.gz')
b = BinaryPbIndexer(index_filename='vec.gz')
c = CompoundIndexer()
c.components = lambda: [a, b]
Compose Flow from YAML
Now let's add our indexer YAML file to the Flow with .add(uses=). Let's also add two shards to the indexer to improve its scalability:
f = Flow().add(uses='MyEncoder', parallel=2).add(uses='myindexer.yml', shards=2).plot()
When you have many arguments, constructing a Flow in Python can get cumbersome. In that case, you can simply move all arguments into one flow.yml:
jtype: Flow
version: '1.0'
pods:
- name: encode
uses: MyEncoder
parallel: 2
- name:index
uses: myindexer.yml
shards: 2
And then load it in Python:
f = Flow.load_config('flow.yml')
Search
Querying a Flow is similar to what we did with indexing. Simply load the query Flow and switch from f.index to f.search. Say you want to retrieve the top 50 documents that are similar to your query and then plot them in HTML:
f = Flow.load_config('flows/query.yml')
with f:
f.search_ndarray(numpy.random.random([10, 28, 28]), shuffle=True, on_done=plot_in_html, top_k=50)
Evaluation
To compute precision recall on the retrieved result, you can add _eval_pr, a built-in evaluator for computing precision & recall.
f = (Flow().add(...)
.add(uses='_eval_pr'))
You can construct an iterator of query and groundtruth pairs and feed to the flow f, via:
from jina import Document
def query_generator():
for _ in range(10):
q = Document()
# now construct expect matches as groundtruth
gt = Document(q, copy=True) # make sure 'gt' is identical to 'q'
gt.matches.append(...)
yield q, gt
f.search(query_iterator, ...)
REST Interface
In practice, the query Flow and the client (i.e. data sender) are often physically separated. Moreover, the client may prefer to use a REST API rather than gRPC when querying. You can set port_expose to a public port and turn on REST support with restful=True:
f = Flow(port_expose=45678, restful=True)
with f:
f.block()
That is the essence behind jina hello fashion. It is merely a taste of what Jina can do. Weโre really excited to see what you do with Jina! You can easily create a Jina project from templates with one terminal command:
pip install jina[hub] && jina hub new --type app
This creates a Python entrypoint, YAML configs and a Dockerfile. You can start from there.
Learn
 |
 |
 |
Examples (View all)
Example code to build your own projects
|
NLP Semantic Wikipedia Search with Transformers and DistilBERTBrand new to neural search? See a simple text-search example to understand how Jina works |
|
Add Incremental Indexing to Wikipedia SearchIndex more effectively by adding incremental indexing to your Wikipedia search |
|
Search Lyrics with Transformers and PyTorchGet a better understanding of chunks by searching a lyrics database. Now with shiny front-end! |
|
Google's Big Transfer Model in (Pokรฉ-)ProductionUse SOTA visual representation for searching Pokรฉmon! |
|
Search YouTube audio data with VggishA demo of neural search for audio data based Vggish model. |
|
Search Tumblr GIFs with KerasEncoderUse prefetching and sharding to improve the performance of your index and query Flow when searching animated GIFs. |
Please check our examples repo for advanced and community-submitted examples.
Want to read more? Check our Founder Han Xiao's blog and our official blog.
Documentation
Apart from the learning resources above, We highly recommended you go through our documentation to master Jina.
Our docs are built on every push, merge, and release of Jina's master branch. Documentation for older versions is archived here.
Are you a "Doc"-star? Join us! We welcome all kinds of improvements on the documentation.
Contributing
We welcome all kinds of contributions from the open-source community, individuals and partners. We owe our success to your active involvement.
Contributors
โจ
Community
- Code of conduct - play nicely with the Jina community
- Slack workspace - join #general on our Slack to meet the team and ask questions
- YouTube channel - subscribe to the latest video tutorials, release demos, webinars and presentations.
- LinkedIn - get to know Jina AI as a company and find job opportunities
- follow and interact with us using hashtag
#JinaSearch- Company - know more about our company and how we are fully committed to open-source.
Open Governance
As part of our open governance model, we host Jina's Engineering All Hands in public. This Zoom meeting recurs monthly on the second Tuesday of each month, at 14:00-15:30 (CET). Everyone can join in via the following calendar invite.
The meeting will also be live-streamed and later published to our YouTube channel.
Join Us
Jina is an open-source project. We are hiring full-stack developers, evangelists, and PMs to build the next neural search ecosystem in open source.
License
Copyright (c) 2020-2021 Jina AI Limited. All rights reserved.
Jina is licensed under the Apache License, Version 2.0. See LICENSE for the full license text.
![[Advise: The API call failed because the CUDA driver and runtime could not be initialized. ]](https://avatars.githubusercontent.com/u/8979086?v=4)
139 Jan 1, 2023
0 Aug 9, 2022
291 Nov 18, 2022
75 Nov 24, 2022
95 Dec 18, 2022
892 Dec 28, 2022
214 Jan 8, 2023
13 Dec 8, 2022
3.2k Dec 31, 2022
146 Dec 28, 2022
19 Oct 27, 2022
114 Dec 29, 2022
3.8k Dec 29, 2022
6 Jul 7, 2022
1 Nov 15, 2021
243 Jan 7, 2023
4.3k Jan 8, 2023