An easier way to build neural search on the cloud

Overview

Jina banner

An easier way to build neural search on the cloud

Python 3.7 3.8 3.9 PyPI Docker Image Version (latest semver) CI CD codecov

Jina is a deep learning-powered search framework for building cross-/multi-modal search systems (e.g. text, images, video, audio) on the cloud.

⏱️ Time Saver - The design pattern of neural search systems, from zero to a production-ready system in minutes.

🍱 Full-Stack Ownership - Keep an end-to-end stack ownership of your solution, avoid the integration pitfalls with fragmented, multi-vendor, generic legacy tools.

🌌 Universal Search - Large-scale indexing and querying of unstructured data: video, image, long/short text, music, source code, etc.

🧠 First-Class AI Models - First-class support for state-of-the-art AI models, easily usable and extendable with a Pythonic interface.

🌩️ Fast & Cloud Ready - Decentralized architecture from day one. Scalable & cloud-native by design: enjoy containerizing, distributing, sharding, async, REST/gRPC/WebSocket.

❤️ Made with Love - Never compromise on quality, actively maintained by a passionate full-time, venture-backed team.


DocsHello WorldQuick StartLearnExamplesContributeJobsWebsiteSlack

Installation

📦
x86/64,arm/v6,v7,v8 (Apple M1)
On Linux/macOS & Python 3.7/3.8/3.9 Docker Users
Standard pip install -U jina docker run jinaai/jina:latest
Daemon pip install -U "jina[daemon]" docker run --network=host jinaai/jina:latest-daemon
With Extras pip install -U "jina[devel]" docker run jinaai/jina:latest-devel
Dev/Pre-Release pip install --pre jina docker run jinaai/jina:master

Version identifiers are explained here. To install Jina with extra dependencies please refer to the docs. Jina can run on Windows Subsystem for Linux. We welcome the community to help us with native Windows support.

YAML Completion in IDE

Developing Jina app often means writing YAML configs. We provide a JSON Schema for your IDE to enable code completion, syntax validation, members listing and displaying help text. Here is a video tutorial to walk you through the setup.

PyCharm

  1. Click menu Preferences -> JSON Schema mappings;
  2. Add a new schema, in the Schema File or URL write https://api.jina.ai/schemas/latest.json; select JSON Schema Version 7;
  3. Add a file path pattern and link it to *.jaml and *.jina.yml.

VSCode

  1. Install the extension: YAML Language Support by Red Hat;
  2. In IDE-level settings.json add:
"yaml.schemas": {
    "https://api.jina.ai/schemas/latest.json": ["/*.jina.yml", "/*.jaml"],
}

Jina "Hello, World!" 👋 🌍

Just starting out? Try Jina's "Hello, World" - jina hello --help

👗 Fashion Image Search

A simple image neural search demo for Fashion-MNIST. No extra dependencies needed, simply run:

jina hello fashion  # more options in --help

...or even easier for Docker users, no install required:

docker run -v "$(pwd)/j:/j" jinaai/jina hello fashion --workdir /j && open j/hello-world.html
# replace "open" with "xdg-open" on Linux
Click here to see console output

hello world console output

This downloads the Fashion-MNIST training and test dataset and tells Jina to index 60,000 images from the training set. Then it randomly samples images from the test set as queries and asks Jina to retrieve relevant results. The whole process takes about 1 minute.

🤖 Covid-19 Chatbot

For NLP engineers, we provide a simple chatbot demo for answering Covid-19 questions. To run that:

pip install "jina[chatbot]"

jina hello chatbot

This downloads CovidQA dataset and tells Jina to index 418 question-answer pairs with DistilBERT. The index process takes about 1 minute on CPU. Then it opens a web page where you can input questions and ask Jina.



🪆 Multimodal Document Search

A multimodal-document contains multiple data types, e.g. a PDF document often contains figures and text. Jina lets you build a multimodal search solution in just minutes. To run our minimum multimodal document search demo:

pip install "jina[multimodal]"

jina hello multimodal

This downloads people image dataset and tells Jina to index 2,000 image-caption pairs with MobileNet and DistilBERT. The index process takes about 3 minute on CPU. Then it opens a web page where you can query multimodal documents. We have prepared a YouTube tutorial to walk you through this demo.




Get Started

🥚 CRUD FunctionsDocumentFlow
🐣 Feed DataFetch ResultAdd LogicInter & Intra ParallelismDecentralizeAsynchronous
🐥 Customize EncoderTest EncoderParallelism & BatchingAdd Data IndexerCompose Flow from YAMLSearchEvaluationREST Interface

🥚 Fundamentals

CRUD Functions

First we look at basic CRUD operations. In Jina, CRUD corresponds to four functions: index (create), search (read), update, and delete. With Documents below as an example:

import numpy as np
from jina import Document
docs = [Document(id='🐲', embedding=np.array([0, 0]), tags={'guardian': 'Azure Dragon', 'position': 'East'}),
        Document(id='🐦', embedding=np.array([1, 0]), tags={'guardian': 'Vermilion Bird', 'position': 'South'}),
        Document(id='🐢', embedding=np.array([0, 1]), tags={'guardian': 'Black Tortoise', 'position': 'North'}),
        Document(id='🐯', embedding=np.array([1, 1]), tags={'guardian': 'White Tiger', 'position': 'West'})]

Let's build a Flow with a simple indexer:

from jina import Flow
f = Flow().add(uses='_index')

Document and Flow are basic concepts in Jina, which will be explained later. _index is a built-in embedding + structured storage that you can use out of the box.

Index
# save four docs (both embedding and structured info) into storage
with f:
    f.index(docs, on_done=print)
Search
# retrieve top-3 neighbours of 🐲, this print 🐲🐦🐢 with score 0, 1, 1 respectively
with f:
    f.search(docs[0], top_k=3, on_done=lambda x: print(x.docs[0].matches))
{"id": "🐲", "tags": {"guardian": "Azure Dragon", "position": "East"}, "embedding": {"dense": {"buffer": "AAAAAAAAAAAAAAAAAAAAAA==", "shape": [2], "dtype": "<i8"}}, "score": {"opName": "NumpyIndexer", "refId": "🐲"}, "adjacency": 1}
{"id": "🐦", "tags": {"position": "South", "guardian": "Vermilion Bird"}, "embedding": {"dense": {"buffer": "AQAAAAAAAAAAAAAAAAAAAA==", "shape": [2], "dtype": "<i8"}}, "score": {"value": 1.0, "opName": "NumpyIndexer", "refId": "🐲"}, "adjacency": 1}
{"id": "🐢", "tags": {"guardian": "Black Tortoise", "position": "North"}, "embedding": {"dense": {"buffer": "AAAAAAAAAAABAAAAAAAAAA==", "shape": [2], "dtype": "<i8"}}, "score": {"value": 1.0, "opName": "NumpyIndexer", "refId": "🐲"}, "adjacency": 1}
Update
# update 🐲 embedding in the storage
docs[0].embedding = np.array([1, 1])
with f:
    f.update(docs[0])
Delete
# remove 🐦🐲 Documents from the storage
with f:
    f.delete(['🐦', '🐲'])

For further details about CRUD functionality, checkout docs.jina.ai.

Document

Document is Jina's primitive data type. It can contain text, image, array, embedding, URI, and be accompanied by rich meta information. To construct a Document, you can use:

import numpy
from jina import Document

doc1 = Document(content=text_from_file, mime_type='text/x-python')  # a text document contains python code
doc2 = Document(content=numpy.random.random([10, 10]))  # a ndarray document

A Document can be recursed both vertically and horizontally to have nested Documents and matched Documents. To better see the Document's recursive structure, you can use .plot() function. If you are using JupyterLab/Notebook, all Document objects will be auto-rendered.

import numpy
from jina import Document

d0 = Document(id='🐲', embedding=np.array([0, 0]))
d1 = Document(id='🐦', embedding=np.array([1, 0]))
d2 = Document(id='🐢', embedding=np.array([0, 1]))
d3 = Document(id='🐯', embedding=np.array([1, 1]))

d0.chunks.append(d1)
d0.chunks[0].chunks.append(d2)
d0.matches.append(d3)

d0.plot()  # simply `d0` on JupyterLab
Click here to see more about MultimodalDocument

MultimodalDocument

A MultimodalDocument is a document composed of multiple Document from different modalities (e.g. text, image, audio).

Jina provides multiple ways to build a multimodal Document. For example, you can provide the modality names and the content in a dict:

from jina import MultimodalDocument
document = MultimodalDocument(modality_content_map={
    'title': 'my holiday picture',
    'description': 'the family having fun on the beach',
    'image': PIL.Image.open('path/to/image.jpg')
})

One can also compose a MultimodalDocument from multiple Document directly:

from jina.types import Document, MultimodalDocument

doc_title = Document(content='my holiday picture', modality='title')
doc_desc = Document(content='the family having fun on the beach', modality='description')
doc_img = Document(content=PIL.Image.open('path/to/image.jpg'), modality='image')
doc_img.tags['date'] = '10/08/2019'

document = MultimodalDocument(chunks=[doc_title, doc_description, doc_img])
Fusion Embeddings from Different Modalities

To extract fusion embeddings from different modalities Jina provides BaseMultiModalEncoder abstract class, which has a unique encode interface.

def encode(self, *data: 'numpy.ndarray', **kwargs) -> 'numpy.ndarray':
    ...

MultimodalDriver provides data to the MultimodalDocument in the correct expected order. In this example below, image embedding is passed to the encoder as the first argument, and text as the second.

jtype: MyMultimodalEncoder
with:
  positional_modality: ['image', 'text']
requests:
  on:
    [IndexRequest, SearchRequest]:
      - jtype: MultiModalDriver {}

Interested readers can refer to jina-ai/example: how to build a multimodal search engine for image retrieval using TIRG (Composing Text and Image for Image Retrieval) for the usage of MultimodalDriver and BaseMultiModalEncoder in practice.

Flow

Jina provides a high-level Flow API to simplify building CRUD workflows. To create a new Flow:

from jina import Flow
f = Flow().add()

This creates a simple Flow with one Pod. You can chain multiple .add()s in a single Flow.

To visualize the Flow, simply chain it with .plot('my-flow.svg'). If you are using a Jupyter notebook, the Flow object will be displayed inline without plot.

Gateway is the entrypoint of the Flow.

Get the vibe? Now we're talking! Let's learn more about the basic concepts and features of Jina:


🥚 CRUD FunctionsDocumentFlow
🐣 Feed DataFetch ResultAdd LogicInter & Intra ParallelismDecentralizeAsynchronous
🐥 Customize EncoderTest EncoderParallelism & BatchingAdd Data IndexerCompose Flow from YAMLSearchEvaluationREST Interface

🐣 Basic

Feed Data

To use a Flow, open it via with context manager, like you would open a file in Python. Now let's create some empty Documents and index them:

from jina import Document

with Flow().add() as f:
    f.index((Document() for _ in range(10)))

Flow supports CRUD operations: index, search, update, delete. In addition, it also provides sugary syntax on ndarray, csv, ndjson and arbitrary files.

Input Example of index/search Explain
numpy.ndarray
with f:
  f.index_ndarray(numpy.random.random([4,2]))

Input four Documents, each document.blob is an ndarray([2])

CSV
with f, open('index.csv') as fp:
  f.index_csv(fp, field_resolver={'pic_url': 'uri'})

Each line in index.csv is constructed as a Document, CSV field pic_url mapped to document.uri.

JSON Lines/ndjson/LDJSON
with f, open('index.ndjson') as fp:
  f.index_ndjson(fp, field_resolver={'question_id': 'id'})

Each line in index.ndjson is constructed as a Document, JSON field question_id mapped to document.id.

Files with wildcards
with f:
  f.index_files(['/tmp/*.mp4', '/tmp/*.pdf'])

Each file captured is constructed as a Document, and Document content (text, blob, buffer) is auto-guessed & filled.

Fetch Result

Once a request is done, callback functions are fired. Jina Flow implements a Promise-like interface: You can add callback functions on_done, on_error, on_always to hook different events. In the example below, our Flow passes the message then prints the result when successful. If something goes wrong, it beeps. Finally, the result is written to output.txt.

def beep(*args):
    # make a beep sound
    import os
    os.system('echo -n "\a";')

with Flow().add() as f, open('output.txt', 'w') as fp:
    f.index(numpy.random.random([4, 5, 2]),
            on_done=print, on_error=beep, on_always=lambda x: fp.write(x.json()))

Add Logic

To add logic to the Flow, use the uses parameter to attach a Pod with an Executor. uses accepts multiple value types including class name, Docker image, (inline) YAML or built-in shortcut.

f = (Flow().add(uses=MyBertEncoder)  # the class of a Jina Executor
           .add(uses='docker://jinahub/pod.encoder.dummy_mwu_encoder:0.0.6-0.9.3')  # the image name
           .add(uses='myencoder.yml')  # YAML serialization of a Jina Executor
           .add(uses='!WaveletTransformer | {freq: 20}')  # inline YAML config
           .add(uses='_pass')  # built-in shortcut executor
           .add(uses={'__cls': 'MyBertEncoder', 'with': {'param': 1.23}}))  # dict config object with __cls keyword

The power of Jina lies in its decentralized architecture: Each add creates a new Pod, and these Pods can be run as a local thread/process, a remote process, inside a Docker container, or even inside a remote Docker container.

Inter & Intra Parallelism

Chaining .add()s creates a sequential Flow. For parallelism, use the needs parameter:

f = (Flow().add(name='p1', needs='gateway')
           .add(name='p2', needs='gateway')
           .add(name='p3', needs='gateway')
           .needs(['p1','p2', 'p3'], name='r1').plot())

p1, p2, p3 now subscribe to Gateway and conduct their work in parallel. The last .needs() blocks all Pods until they finish their work. Note: parallelism can also be performed inside a Pod using parallel:

f = (Flow().add(name='p1', needs='gateway')
           .add(name='p2', needs='gateway')
           .add(name='p3', parallel=3)
           .needs(['p1','p3'], name='r1').plot())

Decentralized Flow

A Flow does not have to be local-only: You can put any Pod to remote(s). In the example below, with the host keyword gpu-pod, is put to a remote machine for parallelization, whereas other Pods stay local. Extra file dependencies that need to be uploaded are specified via the upload_files keyword.

123.456.78.9
# have docker installed
docker run --name=jinad --network=host -v /var/run/docker.sock:/var/run/docker.sock jinaai/jina:latest-daemon --port-expose 8000
# to stop it
docker rm -f jinad
Local
import numpy as np
from jina import Flow

f = (Flow()
     .add()
     .add(name='gpu_pod',
          uses='mwu_encoder.yml',
          host='123.456.78.9:8000',
          parallel=2,
          upload_files=['mwu_encoder.py'])
     .add())

with f:
    f.index_ndarray(np.random.random([10, 100]), output=print)

We provide a demo server on cloud.jina.ai:8000, give the following snippet a try!

from jina import Flow

with Flow().add().add(host='cloud.jina.ai:8000') as f:
    f.index(['hello', 'world'])

Asynchronous Flow

While synchronous from outside, Jina runs asynchronously under the hood: it manages the eventloop(s) for scheduling the jobs. If the user wants more control over the eventloop, then AsyncFlow can be used.

Unlike Flow, the CRUD of AsyncFlow accepts input and output functions as async generators. This is useful when your data sources involve other asynchronous libraries (e.g. motor for MongoDB):

from jina import AsyncFlow

async def input_function():
    for _ in range(10):
        yield Document()
        await asyncio.sleep(0.1)

with AsyncFlow().add() as f:
    async for resp in f.index(input_function):
        print(resp)

AsyncFlow is particularly useful when Jina is using as part of integration, where another heavy-lifting job is running concurrently:

async def run_async_flow_5s():  # WaitDriver pause 5s makes total roundtrip ~5s
    with AsyncFlow().add(uses='- !WaitDriver {}') as f:
        async for resp in f.index_ndarray(numpy.random.random([5, 4])):
            print(resp)

async def heavylifting():  # total roundtrip takes ~5s
    print('heavylifting other io-bound jobs, e.g. download, upload, file io')
    await asyncio.sleep(5)
    print('heavylifting done after 5s')

async def concurrent_main():  # about 5s; but some dispatch cost, can't be just 5s, usually at <7s
    await asyncio.gather(run_async_flow_5s(), heavylifting())

if __name__ == '__main__':
    asyncio.run(concurrent_main())

AsyncFlow is very useful when using Jina inside a Jupyter Notebook. As Jupyter/ipython already manages an eventloop and thanks to autoawait, AsyncFlow can run out-of-the-box in Jupyter.

That's all you need to know for understanding the magic behind hello-world. Now let's dive deeper into it!


🥚 CRUD FunctionsDocumentFlow
🐣 Feed DataFetch ResultAdd LogicInter & Intra ParallelismDecentralizeAsynchronous
🐥 Customize EncoderTest EncoderParallelism & BatchingAdd Data IndexerCompose Flow from YAMLSearchEvaluationREST Interface

🐥 Breakdown of hello-world

Customize Encoder

Let's first build a naive image encoder that embeds images into vectors using an orthogonal projection. To do this, we simply inherit from BaseImageEncoder: a base class from the jina.executors.encoders module. We then override its __init__() and encode() methods.

import numpy as np
from jina.executors.encoders import BaseImageEncoder

class MyEncoder(BaseImageEncoder):

    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        np.random.seed(1337)
        H = np.random.rand(784, 64)
        u, s, vh = np.linalg.svd(H, full_matrices=False)
        self.oth_mat = u @ vh

    def encode(self, data: 'np.ndarray', *args, **kwargs):
        return (data.reshape([-1, 784]) / 255) @ self.oth_mat

Jina provides a family of Executor classes, which summarize frequently-used algorithmic components in neural search. This family consists of encoders, indexers, crafters, evaluators, and classifiers, each with a well-designed interface. You can find the list of all 107 built-in executors here. If they don't meet your needs, inheriting from one of them is the easiest way to bootstrap your own Executor. Simply use our Jina Hub CLI:

pip install jina[hub] && jina hub new

Test Encoder in Flow

Let's test our encoder in the Flow with some synthetic data:

def validate(req):
    assert len(req.docs) == 100
    assert NdArray(req.docs[0].embedding).value.shape == (64,)

f = Flow().add(uses='MyEncoder')

with f:
    f.index_ndarray(numpy.random.random([100, 28, 28]), on_done=validate)

All good! Now our validate function confirms that all one hundred 28x28 synthetic images have been embedded into 100x64 vectors.

Parallelism & Batching

By setting a larger input, you can play with batch_size and parallel:

f = Flow().add(uses='MyEncoder', parallel=10)

with f:
    f.index_ndarray(numpy.random.random([60000, 28, 28]), batch_size=1024)

Add Data Indexer

Now we need to add an indexer to store all the embeddings and the image for later retrieval. Jina provides a simple numpy-powered vector indexer NumpyIndexer, and a key-value indexer BinaryPbIndexer. We can combine them in a single YAML file:

jtype: CompoundIndexer
components:
  - jtype: NumpyIndexer
    with:
      index_filename: vec.gz
  - jtype: BinaryPbIndexer
    with:
      index_filename: chunk.gz
metas:
  workspace: ./
  • jtype: defines the class name of the structure;
  • with: defines arguments for initializing this class object.

💡 Config your IDE to enable autocompletion on YAML

Essentially, the above YAML config is equivalent to the following Python code:

from jina.executors.indexers.vector import NumpyIndexer
from jina.executors.indexers.keyvalue import BinaryPbIndexer
from jina.executors.indexers import CompoundIndexer

a = NumpyIndexer(index_filename='vec.gz')
b = BinaryPbIndexer(index_filename='vec.gz')
c = CompoundIndexer()
c.components = lambda: [a, b]

Compose Flow from YAML

Now let's add our indexer YAML file to the Flow with .add(uses=). Let's also add two shards to the indexer to improve its scalability:

f = Flow().add(uses='MyEncoder', parallel=2).add(uses='myindexer.yml', shards=2).plot()

When you have many arguments, constructing a Flow in Python can get cumbersome. In that case, you can simply move all arguments into one flow.yml:

jtype: Flow
version: '1.0'
pods:
  - name: encode
    uses: MyEncoder
    parallel: 2
  - name:index
    uses: myindexer.yml
    shards: 2

And then load it in Python:

f = Flow.load_config('flow.yml')

Search

Querying a Flow is similar to what we did with indexing. Simply load the query Flow and switch from f.index to f.search. Say you want to retrieve the top 50 documents that are similar to your query and then plot them in HTML:

f = Flow.load_config('flows/query.yml')
with f:
    f.search_ndarray(numpy.random.random([10, 28, 28]), shuffle=True, on_done=plot_in_html, top_k=50)

Evaluation

To compute precision recall on the retrieved result, you can add _eval_pr, a built-in evaluator for computing precision & recall.

f = (Flow().add(...)
           .add(uses='_eval_pr'))

You can construct an iterator of query and groundtruth pairs and feed to the flow f, via:

from jina import Document

def query_generator():
    for _ in range(10):
        q = Document()
        # now construct expect matches as groundtruth
        gt = Document(q, copy=True)  # make sure 'gt' is identical to 'q'
        gt.matches.append(...)
        yield q, gt

f.search(query_iterator, ...)

REST Interface

In practice, the query Flow and the client (i.e. data sender) are often physically separated. Moreover, the client may prefer to use a REST API rather than gRPC when querying. You can set port_expose to a public port and turn on REST support with restful=True:

f = Flow(port_expose=45678, restful=True)

with f:
    f.block()

That is the essence behind jina hello fashion. It is merely a taste of what Jina can do. We’re really excited to see what you do with Jina! You can easily create a Jina project from templates with one terminal command:

pip install jina[hub] && jina hub new --type app

This creates a Python entrypoint, YAML configs and a Dockerfile. You can start from there.

Learn

Examples (View all)

Example code to build your own projects

📄

NLP Semantic Wikipedia Search with Transformers and DistilBERT

Brand new to neural search? See a simple text-search example to understand how Jina works

📄

Add Incremental Indexing to Wikipedia Search

Index more effectively by adding incremental indexing to your Wikipedia search

📄

Search Lyrics with Transformers and PyTorch

Get a better understanding of chunks by searching a lyrics database. Now with shiny front-end!

🖼️

Google's Big Transfer Model in (Poké-)Production

Use SOTA visual representation for searching Pokémon!

🎧

Search YouTube audio data with Vggish

A demo of neural search for audio data based Vggish model.

🎞️

Search Tumblr GIFs with KerasEncoder

Use prefetching and sharding to improve the performance of your index and query Flow when searching animated GIFs.

Please check our examples repo for advanced and community-submitted examples.

Want to read more? Check our Founder Han Xiao's blog and our official blog.

Documentation

Apart from the learning resources above, We highly recommended you go through our documentation to master Jina.

Our docs are built on every push, merge, and release of Jina's master branch. Documentation for older versions is archived here.

Are you a "Doc"-star? Join us! We welcome all kinds of improvements on the documentation.

Contributing

We welcome all kinds of contributions from the open-source community, individuals and partners. We owe our success to your active involvement.

Contributors

All Contributors

Community

  • Code of conduct - play nicely with the Jina community
  • Slack workspace - join #general on our Slack to meet the team and ask questions
  • YouTube channel - subscribe to the latest video tutorials, release demos, webinars and presentations.
  • LinkedIn - get to know Jina AI as a company and find job opportunities
  • Twitter Follow - follow and interact with us using hashtag #JinaSearch
  • Company - know more about our company and how we are fully committed to open-source.

Open Governance

As part of our open governance model, we host Jina's Engineering All Hands in public. This Zoom meeting recurs monthly on the second Tuesday of each month, at 14:00-15:30 (CET). Everyone can join in via the following calendar invite.

The meeting will also be live-streamed and later published to our YouTube channel.

Join Us

Jina is an open-source project. We are hiring full-stack developers, evangelists, and PMs to build the next neural search ecosystem in open source.

License

Copyright (c) 2020-2021 Jina AI Limited. All rights reserved.

Jina is licensed under the Apache License, Version 2.0. See LICENSE for the full license text.

Issues
  • [Advise: The API call failed because the CUDA driver and runtime could not be initialized. ]

    [Advise: The API call failed because the CUDA driver and runtime could not be initialized. ]

    Describe your proposal/problem

    Dear Jina Team,

    Recently, I tried nlp-simple example and replaced TransformerTorchEncoder with TextPaddlehubEncoder. However, I got the following error.

    Traceback (most recent call last):
      File "/home/work/renyuanhang/my_anaconda3/envs/jina_rocket2/lib/python3.8/site-packages/jina/peapods/runtimes/zmq/zed.py", line 73, in _load_executor
        self._executor = BaseExecutor.load_config(
      File "/home/work/renyuanhang/my_anaconda3/envs/jina_rocket2/lib/python3.8/site-packages/jina/jaml/__init__.py", line 531, in load_config
        return JAML.load(tag_yml, substitute=False)
      File "/home/work/renyuanhang/my_anaconda3/envs/jina_rocket2/lib/python3.8/site-packages/jina/jaml/__init__.py", line 89, in load
        r = yaml.load(stream, Loader=JinaLoader)
      File "/home/work/renyuanhang/my_anaconda3/envs/jina_rocket2/lib/python3.8/site-packages/yaml/__init__.py", line 114, in load
        return loader.get_single_data()
      File "/home/work/renyuanhang/my_anaconda3/envs/jina_rocket2/lib/python3.8/site-packages/yaml/constructor.py", line 51, in get_single_data
        return self.construct_document(node)
      File "/home/work/renyuanhang/my_anaconda3/envs/jina_rocket2/lib/python3.8/site-packages/yaml/constructor.py", line 55, in construct_document
        data = self.construct_object(node)
      File "/home/work/renyuanhang/my_anaconda3/envs/jina_rocket2/lib/python3.8/site-packages/yaml/constructor.py", line 100, in construct_object
        data = constructor(self, node)
      File "/home/work/renyuanhang/my_anaconda3/envs/jina_rocket2/lib/python3.8/site-packages/jina/jaml/__init__.py", line 422, in _from_yaml
        return get_parser(cls, version=data.get('version', None)).parse(cls, data)
      File "/home/work/renyuanhang/my_anaconda3/envs/jina_rocket2/lib/python3.8/site-packages/jina/jaml/parsers/executor/legacy.py", line 130, in parse
        obj = cls(
      File "/home/work/renyuanhang/my_anaconda3/envs/jina_rocket2/lib/python3.8/site-packages/jina/executors/__init__.py", line 82, in __call__
        getattr(obj, '_post_init_wrapper', lambda *x: None)(m, r)
      File "/home/work/renyuanhang/my_anaconda3/envs/jina_rocket2/lib/python3.8/site-packages/jina/executors/__init__.py", line 174, in _post_init_wrapper
        self.post_init()
      File "/home/work/renyuanhang/my_anaconda3/envs/jina_rocket2/lib/python3.8/site-packages/jina/hub/encoders/nlp/TextPaddlehubEncoder/__init__.py", line 38, in post_init
        self.model = hub.Module(name=self.model_name)
      File "/home/work/renyuanhang/my_anaconda3/envs/jina_rocket2/lib/python3.8/site-packages/paddlehub/module/module.py", line 171, in __new__
        module = cls.init_with_name(
      File "/home/work/renyuanhang/my_anaconda3/envs/jina_rocket2/lib/python3.8/site-packages/paddlehub/module/module.py", line 263, in init_with_name
        user_module_cls = manager.install(name=name, version=version, source=source, update=update, branch=branch)
      File "/home/work/renyuanhang/my_anaconda3/envs/jina_rocket2/lib/python3.8/site-packages/paddlehub/module/manager.py", line 188, in install
        return self._install_from_name(name, version)
      File "/home/work/renyuanhang/my_anaconda3/envs/jina_rocket2/lib/python3.8/site-packages/paddlehub/module/manager.py", line 263, in _install_from_name
        return self._install_from_url(item['url'])
      File "/home/work/renyuanhang/my_anaconda3/envs/jina_rocket2/lib/python3.8/site-packages/paddlehub/module/manager.py", line 256, in _install_from_url
        return self._install_from_archive(file)
      File "/home/work/renyuanhang/my_anaconda3/envs/jina_rocket2/lib/python3.8/site-packages/paddlehub/module/manager.py", line 361, in _install_from_archive
        return self._install_from_directory(directory)
      File "/home/work/renyuanhang/my_anaconda3/envs/jina_rocket2/lib/python3.8/site-packages/paddlehub/module/manager.py", line 345, in _install_from_directory
        hub_module_cls = HubModule.load(self._get_normalized_path(module_info.name))
      File "/home/work/renyuanhang/my_anaconda3/envs/jina_rocket2/lib/python3.8/site-packages/paddlehub/module/module.py", line 219, in load
        paddle.set_device(place)
      File "/home/work/renyuanhang/my_anaconda3/envs/jina_rocket2/lib/python3.8/site-packages/paddle/device.py", line 166, in set_device
        framework._set_expected_place(place)
      File "/home/work/renyuanhang/my_anaconda3/envs/jina_rocket2/lib/python3.8/site-packages/paddle/fluid/framework.py", line 317, in _set_expected_place
        _set_dygraph_tracer_expected_place(place)
      File "/home/work/renyuanhang/my_anaconda3/envs/jina_rocket2/lib/python3.8/site-packages/paddle/fluid/framework.py", line 311, in _set_dygraph_tracer_expected_place
        _dygraph_tracer_._expected_place = place
    OSError: (External)  Cuda error(3), initialization error.
      [Advise: The API call failed because the CUDA driver and runtime could not be initialized. ] (at /paddle/paddle/fluid/platform/gpu_info.cc:229)
    

    It seems to be related with https://github.com/PaddlePaddle/Paddle/issues/25185.

    However, I have no idea how to fix this. Could you help me to figure it out? Thank you very much!


    Environment

    paddlehub==2.0.0 paddlenlp==2.0.1 paddlepaddle-gpu==2.0.2.post90 jina==1.3.0

    Screenshots

    image

    area/core type/bug 
    opened by ryh95 47
  • Permanently delete documents from storage

    Permanently delete documents from storage

    Hey Jina team!

    As described here documents persist the deletion and are only marked as inaccessible.

    My question: Is there a possibility to delete documents from storage without reindexing everything again? Deletion performance isn't an issue for us, but reindexing everything is very inconvenient in our use case.

    Thanks! I really appreciate your kind support.

    Sub-tasks

    • [x] Delete from NpIndexer. @ #2046
    • [x] Delete from BinaryPbIndexer. @ https://github.com/jina-ai/jina/pull/2102
    area/core priority/important-soon type/feature-request status/done 
    opened by ace-kay-law-neo 39
  • No documents are found with BinaryPbIndexer in 1.1.2

    No documents are found with BinaryPbIndexer in 1.1.2

    Versions 1.0.11: everything works as expected 1.1.2: BinaryPbIndexer doesn't find any documents

    Detailed description He Jina team, when I upgrade to 1.1.2 and reindex my dataset, the BinaryPbIndexer no longer works. It doesn't raise any errors, but I just can't find any documents. Just as a remark, I use custom ids which always consist of 33 characters. Where there any changes for the BinaryPbIndexer, the KVIndexDriver or the KVSearchDriver?

    area/core priority/critical type/bug status/done 
    opened by janandreschweiger 39
  • File seems to be closed when trying to index

    File seems to be closed when trying to index

    Yes @JoanFM, using the prefetch argument in the flow-config solved the original problem. I tried to pass the batch_size and the request_size like you, but I still get the same error. I have some additional logs, which might indicate what causes the problem. As I said, the first few batches are indexed successfully, but at some point I get this error:

            [email protected][I]:no update since 2021-04-10 07:38:17, will not save. If you really want to save it, call "touch()" before "save()" to force saving
            [email protected][I]:#sent: 3 #recv: 4 sent_size: 13.8 MB recv_size: 1.2 MB
          [email protected][I]:recv IndexRequest  from gateway▸crafter/ZEDRuntime▸encoder/ZEDRuntime▸chunk_idx/ZEDRuntime▸⚐
            [email protected][I]:recv IndexRequest  from gateway▸crafter/ZEDRuntime▸encoder/ZEDRuntime▸⚐
       [email protected][S]:artifacts of this executor (wrap_idx) is persisted to /var/huforce/fs/workspace/wrap_idx-0/wrap_idx.bin
    [email protected][S]:artifacts of this executor (chunk_idx) is persisted to /var/huforce/fs/workspace/chunk_idx-0/chunk_idx.bin
          [email protected][I]:no update since 2021-04-10 07:38:17, will not save. If you really want to save it, call "touch()" before "save()" to force saving
          [email protected][I]:#sent: 3 #recv: 4 sent_size: 1.9 MB recv_size: 15.6 MB
            [email protected][I]:recv IndexRequest  from gateway▸crafter/ZEDRuntime▸encoder/ZEDRuntime▸chunk_idx/ZEDRuntime▸doc_idx/ZEDRuntime▸⚐
    DEBUG | Message: Successfully indexed 4 documents!
            [email protected][S]:artifacts of this executor (doc_idx_file) is persisted to /var/huforce/fs/workspace/doc_idx_file-0/doc_idx_file.bin
            [email protected][I]:#sent: 3 #recv: 4 sent_size: 6.3 KB recv_size: 2.2 MB
           | 📃     16 ⏱️ 262.9s 🐎 0.1/s      4   requests        [email protected][I]:send: 5 recv: 4 pending: 1
            [email protected][I]:recv IndexRequest  from gateway▸crafter/ZEDRuntime▸⚐
    DEBUG | Message: File-Info: research_admission.txt
    DEBUG | Message: File-Info: cloudcomputing.pptx
    DEBUG | Message: File-Info: WS_2012_White_Paper_Hyper-V.pdf
    DEBUG | Message: File-Info: writing-sql-queries.docx
            [email protected][I]:no update since 2021-04-10 07:38:17, will not save. If you really want to save it, call "touch()" before "save()" to force saving
            [email protected][I]:#sent: 5 #recv: 6 sent_size: 1.2 MB recv_size: 5.2 KB
            [email protected][I]:#sent: 4 #recv: 5 sent_size: 15.6 MB recv_size: 1.2 MB
          [email protected][I]:recv IndexRequest  from gateway▸crafter/ZEDRuntime▸encoder/ZEDRuntime▸chunk_idx/ZEDRuntime▸⚐
            [email protected][I]:recv IndexRequest  from gateway▸crafter/ZEDRuntime▸encoder/ZEDRuntime▸⚐
          [email protected][I]:#sent: 4 #recv: 5 sent_size: 2.2 MB recv_size: 16.1 MB
    DEBUG | Message: Encoding 552 paragraphs
          [email protected][E]:ValueError('write to closed file')
     add "--quiet-error" to suppress the exception details
    Traceback (most recent call last):
      File "/usr/local/lib/python3.8/site-packages/jina/peapods/runtimes/zmq/zed.py", line 198, in _msg_callback
        self._zmqlet.send_message(self._callback(msg))
      File "/usr/local/lib/python3.8/site-packages/jina/peapods/runtimes/zmq/zed.py", line 184, in _callback
        self._pre_hook(msg)._handle(msg)._post_hook(msg)
      File "/usr/local/lib/python3.8/site-packages/jina/peapods/runtimes/zmq/zed.py", line 177, in _handle
        self._executor(self.request_type)
      File "/usr/local/lib/python3.8/site-packages/jina/executors/__init__.py", line 595, in __call__
        d()
      File "/usr/local/lib/python3.8/site-packages/jina/drivers/__init__.py", line 381, in __call__
        self._apply_all(documents, *args, **kwargs)
      File "/usr/local/lib/python3.8/site-packages/jina/drivers/index.py", line 58, in _apply_all
        self.exec_fn(keys, values)
      File "/usr/local/lib/python3.8/site-packages/jina/executors/decorators.py", line 44, in arg_wrapper
        f = func(self, *args, **kwargs)
      File "/usr/local/lib/python3.8/site-packages/jina/executors/indexers/keyvalue.py", line 178, in add
        self.write_handler.header.write(
    ValueError: write to closed file
            [email protected][I]:recv IndexRequest  from gateway▸crafter/ZEDRuntime▸encoder/ZEDRuntime▸chunk_idx/ZEDRuntime✖▸doc_idx/ZEDRuntime▸⚐
    dev_ai            | DEBUG | Message: Successfully indexed 4 documents!
    

    What's also strange is, that the failed documents for the chunk-idx are passed on to the doc-idx, as you can see in the logs.

    My flow for indexing:

    !AsyncFlow
    with:
      logserver: false
      show-exc-info: true
      prefetch: 2
      # request-size: 2
      # batch_size: 2
    pods:
      crafter:
        uses: pods/craft.yml
      encoder:
        uses: pods/encode.yml
        timeout_ready: 600000
      chunk_idx:
        uses: pods/chunk.yml
      doc_idx:
        uses: pods/doc.yml
    

    Originally posted by @nero-nazok in https://github.com/jina-ai/jina/issues/2298#issuecomment-817084516

    area/core type/bug 
    opened by JoanFM 32
  • Unable to run Jina on Raspberry Pi 4

    Unable to run Jina on Raspberry Pi 4

    I installed Jina using the installation instructions in the docs and I was unable to run even the Jina command without the error

    free(): invalid pointer
    Aborted
    

    I have tried installing Jina from master branch, 1.2.4, and from pip. Yet, the problem persists


    Environment Can't actually run this command.

    Latest version of Raspbian OS (aarm64), on a RPi4. Python 3.7.3

    Screenshots

    type/bug 
    opened by navanchauhan 25
  • Stress Test in CICD with big datasets

    Stress Test in CICD with big datasets

    Describe your problem We are building a semantic text search system, using NumpyIndexer and cosine distance metric. We have been testing Jina with smaller sets for the moment, running in single instances, but our final goal is to run Jira in a distributed system over datasets with billions of documents.

    As a first step, and to validate our configuration when querying smaller datasets, we have indexed around 30 million text documents (simple sentences), using the following settings:

    • Prefetch: 200
    • Batch size: 1000
    • Shards: 10
    • Parallel: 10

    (29889733 docs :white_check_mark: done in :stopwatch: 6595.7s :racehorse: 4531.8/s) So the indexing part is fine.

    But we are having problems while performing queries over this data. Any query causes an out of memory error at OS level.

    We tried running queries on differents environments:

    • Machines with up to 240 GB of RAM.
    • Decreasing settings down to:
    • Prefetch: 10
    • Batch size: 50
    • Shards: 1
    • Parallel: 1

    But we are still having the same memory problems.

    We would like to get some feedback about the problems we are having.

    Dataset details: Regarding the input data (around 150GB of plain text files), it is spread into multiple files in the ./source/ folder. Each input file has the following structure: file1

    {"text": "my sample text", "vector": [ 0.1234567, -0.2345678, 0.3456789...]}
    {"text": "my other sample text", "vector": [ 0.4567890, 0.5678901, -0.6789012...]}
    ...
    

    file2

    {"text": "my sample text", "vector": [ 0.1234567, -0.2345678, 0.3456789...]}
    {"text": "my other sample text", "vector": [ 0.4567890, 0.5678901, -0.6789012...]}
    ...
    

    Basically, multiline json files, where each line is a json document. And each document is formed by two fields:

    • text: The text string (to be added as meta-information during indexing).
    • vector: A pre-generated [512,] float32 array generated from the text field by using the universal sentence encoder ("https://tfhub.dev/google/universal-sentence-encoder/4")

    Once the data is indexed, the resulting compressed dataset is around 56GB.

    Here you can find the setup files tar.gz

    What is your guess? We guess there must be something wrong in our configuration, as we didn’t expect to run into memory problems, given the dataset size and the instances we have used for testing.

    Environment We have tried with differents environments, but the common one is this:

    (jina) [[email protected] ~]$ jina --version-full
    jina                          0.4.1
    jina-proto                    0.0.55
    jina-vcs-tag                  (unset)
    libzmq                        4.3.2
    pyzmq                         1.18.5
    protobuf                      3.13.0
    proto-backend                 cpp
    grpcio                        1.31.0
    ruamel.yaml                   0.16.10
    python                        3.8.5
    platform                      Linux
    platform-release              4.14.146-93.123.amzn1.x86_64
    platform-version              #1 SMP Tue Sep 24 00:45:23 UTC 2019
    architecture                  x86_64
    processor                     x86_64
    jina-resources                /home/ec2-user/.pyenv/versions/3.8.5/envs/jina/lib/python3.8/site-packages/jina/resources
    JINA_ARRAY_QUANT              (unset)
    JINA_CONTRIB_MODULE           (unset)
    JINA_CONTRIB_MODULE_IS_LOADING(unset)
    JINA_CONTROL_PORT             (unset)
    JINA_DEFAULT_HOST             (unset)
    JINA_EXECUTOR_WORKDIR         (unset)
    JINA_FULL_CLI                 (unset)
    JINA_IPC_SOCK_TMP             (unset)
    JINA_LOG_FILE                 (unset)
    JINA_LOG_LONG                 (unset)
    JINA_LOG_NO_COLOR             (unset)
    JINA_LOG_PROFILING            (unset)
    JINA_LOG_SSE                  (unset)
    JINA_LOG_VERBOSITY            (unset)
    JINA_POD_NAME                 (unset)
    JINA_PROFILING                (unset)
    JINA_SOCKET_HWM               (unset)
    JINA_STACK_CONFIG             (unset)
    JINA_TEST_CONTAINER           (unset)
    JINA_TEST_GPU                 (unset)
    JINA_TEST_PRETRAINED          (unset)
    JINA_VCS_VERSION              (unset)
    JINA_VERSION                  (unset)
    JINA_WARN_UNNAMED             (unset)
    JINA_BINARY_DELIMITER         (unset)
    JINA_DISABLE_UVLOOP           (unset)
    
    (jina) [[email protected] ~]$ cat /etc/issue
    Amazon Linux AMI release 2018.03
    Kernel \r on an \m
    

    Screenshots

    Extracted from logs:

    ...
    Unable to allocate 5.70 GiB for an array with shape (2988733, 512) and data type float32
    ...
    

    ┆Issue is synchronized with this Jira Task by Unito

    priority/critical kind/problem type/bug 
    opened by jmferrete 22
  • the design of `_traverse_apply` makes all BaseRecursiveDriver low-efficient

    the design of `_traverse_apply` makes all BaseRecursiveDriver low-efficient

    Describe the bug

    Highest priority, must solve now.

    The current design combination of _apply_all and _traverse_apply in BaseRecursiveDriver results in a low efficient recursion, in particular when each document has only a small number of chunks. The _apply_all will only apply on that small documents, making the CPU/GPU extremely data-hungry.

    This is inefficient. The only reason that this problem was not paid enough attention is that the problem only becomes obvious when you have multiple granularities. Single granularity, i.e. directly working on root-level won't reveal it.

    This inefficient design also introduces further hacky implementations of the driver. One example is the Cache in EncodeDriver which is unnecessary if apply_all can work on all defined granularity in one-shot. Note that this temporally solves the data-hungry problem for EncodeDriver but all other drivers facing the same problem.

    Describe how you solve it

    • redesign _traverse_apply, move it to Document
    • _apply_all iterates over the concatenation of iterators in one shot
    • remove hacky implementation in EncodeDriver

    Environment

    Screenshots

    area/core priority/critical type/bug status/done 
    opened by hanxiao 22
  • fix: custom long ids for rankers

    fix: custom long ids for rankers

    fixes #1553

    area/core size/S area/testing component/executor component/driver executor/indexer executor/ranker component/type 
    opened by janandreschweiger 21
  • feat: add typing in some functions with corresponding test

    feat: add typing in some functions with corresponding test

    add typing in some files, also created a test for jina.executors.encoders.helper class (only for reduce_mean function,since the return value in other functions in helper are the same)

    opened by roccia 19
  • TypeError(

    TypeError("get_embedding() got an unexpected keyword argument 'texts'")

    Describe the bug

    Can not load encoder model. The encoder.yml is as below:

    !TextPaddlehubEncoder
    with:
      model_name: chinese-electra-small
    requests:
      on:
        ControlRequest:
          - !ControlReqDriver {}
        [SearchRequest, TrainRequest, IndexRequest]:
          - !EncodeDriver
            with:
              traversal_paths: ['c']
    

    Showed errors: image

    Describe how you solve it


    Environment

    jina 0.9.16

    Screenshots

    Stale type/bug 
    opened by yiouyou 18
  • refactor: terminate by signals

    refactor: terminate by signals

    null

    size/L area/core area/testing area/network component/peapod component/flow 
    opened by JoanFM 2
  • fix: fix rm from exitstack

    fix: fix rm from exitstack

    null

    size/M area/core area/network component/peapod 
    opened by JoanFM 2
  • rolling update lets context managers unexited, leaking BasePea

    rolling update lets context managers unexited, leaking BasePea

    Problem description Our stack of context managers is growing indifinetly

    priority/important-soon 
    opened by JoanFM 0
  • refactor color_axis to channel_axis

    refactor color_axis to channel_axis

    In a typical image search task, we send image with dimensionality of B, C, H, W into a deep learning model, where:

    • B refers to batch_size.
    • C refers to channels, e.g. RGB is 3 channel image while grey-scale image only have 1 channel
    • H is the image height
    • W is the image width.

    By default Jina executors and Finetuner use an argument channel_axis which allow user to move the channel axis to a specific axis. For instance channel_axis=-1 means the image has the shape of B, H, W, C.

    However, in Jina core, we have a different name color_axis. We want to make these two arguments aligned, so we expect someone up us refactor Jina, changing color_axis to channel_axis, and adjust associated unit tests.

    good first issue 
    opened by bwanglzu 4
  • fix(k8s): run all k8s tests in ci

    fix(k8s): run all k8s tests in ci

    WIP for testing k8s test execution in CI

    area/housekeeping area/cicd size/XS 
    opened by jacobowitz 2
  • send_message interface of grpclet should be sync

    send_message interface of grpclet should be sync

    The send_message() interface in the Grpclet is async at the moment and the GrpcDataRuntime uses this to create tasks for the calls. This is a bit weird because all the asyncio functionality is actually capsulated by the Grpclet (it creates and manages asyncio task) and there is no need to expose this.

    So we should make the send_message interface sync and all of their uses. At the moment this is quite a bit of work because we also have to unify it with the zmqlet and some test mocks. This includes also uses in ZEDRuntime and the Gateway runtimes.

    We should wait with this one until star routing is done so that we have to change less.

    priority/backlog focus/ease-of-use 
    opened by jacobowitz 0
  • Add executor scalability with replicas section

    Add executor scalability with replicas section

    Enrich/minimize/segment the code with some text and put into flow/topology section to prove that “by setting replicas, one can solve the scalability problem; and the flow enjoys a nonblocking behavior"

    area/docs 
    opened by CatStark 1
  • docs: add executor scalability with parallel section

    docs: add executor scalability with parallel section

    Add section that describes how to avoid bottlenecks with parallel

    size/S area/docs 
    opened by alaeddine-13 2
  •  typo: univeral->universal

    typo: univeral->universal

    null

    opened by sarvesh4396 1
  • Logstreaming with JinaD broken for Flows/Peas/Pods

    Logstreaming with JinaD broken for Flows/Peas/Pods

    Steps to reproduce:

    • Start a Flow with JinaD.
    • Connect to websocket to stream logs via ws://<hostname>:8000/logstream/<flow-id>.
    • Client successfully connects, but no logs are streamed.
    • Server drops client connection after 60 secs with the error waited for 60 secs for /tmp/jinad/jworkspace-a263205c-f6ee-45a4-841c-5f8dce9f5b8c/logs/jflow-4a4795c3-0c37-4338-a725-036fd9b765a2/logging.log to be ready, exiting

    It is broken for Flows, Peas & Pods, but works during Workspace creation.

    priority/backlog component/logging type/bug area/jinaD 
    opened by deepankarm 0
Releases(v2.1.10)
  • v2.1.10(Oct 20, 2021)

    Release Note (2.1.10)

    Release time: 2021-10-20 09:38:15

    🙇 We'd like to thank all contributors for this new release! In particular, Han Xiao, Jina Dev Bot, 🙇

    🐞 Bug fixes

    • [37210f8d] - requirements: add pillow to standard (Han Xiao)

    📗 Documentation

    • [e9474f6c] - add finetuner links to ecosystem (Han Xiao)

    🍹 Other Improvements

    • [bceeeef5] - fix pillow dependencies (Han Xiao)
    • [101794f8] - version: the next version will be 2.1.10 (Jina Dev Bot)
    Source code(tar.gz)
    Source code(zip)
  • v2.1.9(Oct 20, 2021)

    Release Note (2.1.9)

    Release time: 2021-10-20 07:30:32

    🙇 We'd like to thank all contributors for this new release! In particular, felix-wang, Wang Bo, Han Xiao, Jina Dev Bot, Florian Hönicke, Deepankar Mahapatro, Joan Fontanals, Gautham Suresh, Tobias Jacobowitz, Maximilian Werk, 🙇

    💥 Breaking changes

    • [1f4e16c5] - disable prefetch by default, remove on-recv :boom: (#3590) (Deepankar Mahapatro)

    🆕 New Features

    • [86a6b987] - types: add to_datauri in from_files generator (#3708) (Han Xiao)
    • [249e191c] - document: add support for hugging face datasets :champagne: (#3613) (Gautham Suresh)
    • [d3f7330e] - request parameters return StructView (#3702) (Joan Fontanals)
    • [f91dce40] - terminate process in case of cancel runtime failure (#3701) (Joan Fontanals)
    • [5745c341] - runtimes: Support SIGTERM properly (#3672) (Tobias Jacobowitz)
    • [03fab2ae] - runtime: return dict will pass to request.parameters (#3692) (Han Xiao)
    • [633fb287] - type: add fp32 storage to dense ndarray pb (#3690) (Han Xiao)
    • [05e93547] - concurrent rolling updates of shards (#3681) (Joan Fontanals)

    🐞 Bug fixes

    • [c9a86306] - requirments not exist (#3715) (felix-wang)
    • [0aa6c3d9] - fix windows py38 tornado (#3679) (Joan Fontanals)
    • [cf6705f4] - pass shards args in k8s (#3689) (Joan Fontanals)
    • [fcbb98e7] - convert uri to datauri without overwriting buffer (#3697) (felix-wang)
    • [48fe7245] - time is not unlimited (#3696) (Maximilian Werk)
    • [54d916e3] - helper: fix keyboard interrupt propagate (#3694) (Han Xiao)
    • [b64c127c] - hub: check if deps satisfied in advance (#3691) (Han Xiao)

    📗 Documentation

    • [e5faa896] - add warn in notebook environ (#3714) (Wang Bo)
    • [319bf22b] - add finetuner links to ecosystem (Han Xiao)
    • [db1dbaf8] - complete k8s documentation (#3700) (Florian Hönicke)

    🏁 Unit Test and CICD

    • [9e510fb1] - stream: separate asserts (#3710) (Deepankar Mahapatro)
    • [a078c07e] - rescue old invalid test (#3707) (Joan Fontanals)
    • [f7e0f9fa] - k8s: proper k8s pod test (#3618) (Wang Bo)

    🍹 Other Improvements

    • [618419c9] - update readme (Han Xiao)
    • [0c683fb8] - contributor: update contributors (Jina Dev Bot)
    • [d5e7731a] - docs: fix structure on docs and readme (Han Xiao)
    • [fd71c9b5] - docs: update TOC (Jina Dev Bot)
    • [9172f2af] - version: the next version will be 2.1.9 (Jina Dev Bot)
    Source code(tar.gz)
    Source code(zip)
  • v2.1.8(Oct 15, 2021)

    Release Note (2.1.8)

    Release time: 2021-10-15 20:19:04

    🙇 We'd like to thank all contributors for this new release! In particular, Tobias Jacobowitz, Han Xiao, Zhaofeng Miao, Joan Fontanals, Jina Dev Bot, Florian Hönicke, M4rtin König, 🙇

    🆕 New Features

    • [bd6e5626] - helper: add eta on progressbar for known-length inputs (#3688) (Han Xiao)
    • [18c32e78] - k8s pod rolling update (#3631) (Joan Fontanals)
    • [8a44d1b3] - k8s update policy (#3669) (Florian Hönicke)

    🐞 Bug fixes

    • [dca6d016] - docker: args processing in container runtime (#3684) (Tobias Jacobowitz)
    • [7d9e6fd3] - dockerfile: use right environment when build core and perf (#3673) (Zhaofeng Miao)
    • [af9c2749] - fix ambiguity uses after and before (#3683) (Joan Fontanals)
    • [2b08e230] - gateway uvicorn needed (#3674) (Florian Hönicke)
    • [76285be3] - args: allow disabling k8s_connection_pooling (#3532) (#3667) (M4rtin König)
    • [b341b9d6] - compatibility with pyyaml 6.0 (#3675) (Joan Fontanals)
    • [e969ce29] - set debug log level to k8spod (#3660) (Joan Fontanals)
    • [cf7c215f] - fix plot document (#3659) (Joan Fontanals)

    🧼 Code Refactoring

    • [b014f0c8] - go lang warning (#3671) (Florian Hönicke)

    📗 Documentation

    • [4a090144] - refactor shard and replica docs (#3630) (Tobias Jacobowitz)

    🍹 Other Improvements

    • [f30ecd89] - contributor: update contributors (Jina Dev Bot)
    • [deee1719] - update codeowners (#3677) (Han Xiao)
    • [b0746c4d] - docs: update TOC (Jina Dev Bot)
    • [0221034c] - version: the next version will be 2.1.8 (Jina Dev Bot)
    Source code(tar.gz)
    Source code(zip)
  • v2.1.7(Oct 13, 2021)

    Release Note (2.1.7)

    Release time: 2021-10-13 17:17:56

    🙇 We'd like to thank all contributors for this new release! In particular, Jina Dev Bot, Deepankar Mahapatro, Han Xiao, Joan Fontanals, Florian Hönicke, Wang Bo, CallmeMehdi, Tobias Jacobowitz, 🙇

    🆕 New Features

    • [1e54f679] - k8s hubble no cache (#3627) (Florian Hönicke)
    • [e42f6a49] - pass uses_with to rolling_update (#3608) (Joan Fontanals)
    • [6763577f] - k8s perf (#3628) (Florian Hönicke)
    • [6cd417b0] - k8s: add environment variable configurations :champagne: (#3502) (Wang Bo)
    • [693cf475] - sharding: align shard and replica concept with k8s (#3535) (Tobias Jacobowitz)

    🐞 Bug fixes

    • [82d3eace] - client: websocket client streaming (#3600) (Deepankar Mahapatro)
    • [f40e63c1] - match: fix type of batch_size and limit in match function (#3639) (Han Xiao)
    • [28a6689c] - fix darray sort tie (#3637) (Joan Fontanals)
    • [411482ca] - validate limit input in match (#3633) (Joan Fontanals)
    • [ba9a6510] - move portforward to cicd tag (#3629) (Joan Fontanals)
    • [6d409199] - fix reference before assignment (#3615) (Joan Fontanals)

    🧼 Code Refactoring

    • [03b47634] - avoid shards clearance (#3624) (Joan Fontanals)
    • [aed448aa] - remove Optionality of DocumentArray 🍾 (#3553) (CallmeMehdi)

    📗 Documentation

    • [553b8f6f] - add env to docs (#3626) (Wang Bo)

    🍹 Other Improvements

    • [12b3a639] - contributor: update contributors (Jina Dev Bot)
    • [4504ee54] - docs: update TOC (Jina Dev Bot)
    • [e641a531] - version: the next version will be 2.1.7 (Jina Dev Bot)
    Source code(tar.gz)
    Source code(zip)
  • v2.1.6(Oct 9, 2021)

    Release Note (2.1.6)

    Release time: 2021-10-09 18:30:30

    🙇 We'd like to thank all contributors for this new release! In particular, Tobias Jacobowitz, Han Xiao, Jina Dev Bot, Winston Wong, gitcommitshow, Joan Fontanals, Maximilian Werk, cristian, Florian Hönicke, George Mastrapas, Deepankar Mahapatro, GeopJr, Nan Wang, Wang Bo, Tadej Svetina, 🙇

    🆕 New Features

    • [e86a86ad] - added more progress bar options (#3605) (Maximilian Werk)
    • [31a69680] - documentarray: heap based top k sorting in DocumentArray (#3467) (#3570) (George Mastrapas)
    • [3353cc83] - start portforwarding when flow.post in K8s (#3484) (Florian Hönicke)
    • [2c51fa7e] - client: add prefetch to http client (#3562) (Deepankar Mahapatro)
    • [ea0957fc] - k8s: kubernetes completion, wait for ready and close (#3509) (Joan Fontanals)
    • [30517494] - docsbot: enable custom backend (#3573) (Deepankar Mahapatro)
    • [8bd5c39b] - add buffers getter setter (#3550) (Wang Bo)

    ⚡ Performance Improvements

    • [ded0a61e] - types: optimize memory in match when traversal path is root :champagne: (#3481) (Winston Wong)

    🐞 Bug fixes

    • [4e7536e0] - daemon: workaround failing jinad ci (#3623) (Tobias Jacobowitz)
    • [5604b9a0] - runtime: add random_name to container to solve conflict naming (#3621) (Han Xiao)
    • [4eb1d987] - fix fastapi version (#3612) (Joan Fontanals)
    • [3ac26038] - scipy sparse import (#3583) (cristian)
    • [c77ed135] - docs: performance and a11y (#3565) (GeopJr)
    • [23a6426e] - conda requirements (#3549) (Tadej Svetina)
    • [9d758a0a] - kubernetes: add kubernetes dep to perf (#3542) (Tobias Jacobowitz)
    • [4abdc0d8] - use test-pip in k8stest (#3528) (Joan Fontanals)

    📗 Documentation

    • [6ee0b241] - added video explanation of jina concepts (#3616) (gitcommitshow)
    • [632f0615] - add link to flow parallelization (#3597) (cristian)
    • [1b621851] - k8s credentials (#3602) (Florian Hönicke)
    • [0fb8ee1e] - expose http endpoints with flow yaml (#3589) (Deepankar Mahapatro)
    • [1780eceb] - fix a typo (#3576) (Nan Wang)
    • [e70b460c] - update link to indexer (#3501) (Maximilian Werk)

    🏁 Unit Test and CICD

    • [add58f5a] - add tests proving grpc data with inspect (#3595) (Joan Fontanals)
    • [5be2d49e] - k8s: skip k8s from success-all-test (#3592) (Deepankar Mahapatro)
    • [f7e306a6] - try to return results (#3581) (Joan Fontanals)
    • [da60ff0e] - fix cla bugs (#3578) (Nan Wang)
    • [fcada5b6] - fix cla (#3577) (Nan Wang)
    • [ce1f4225] - fix cla (#3575) (Nan Wang)
    • [d3622020] - fix cd (#3572) (Wang Bo)
    • [e29e5f49] - flow: move slow async flow into independent test (#3551) (Wang Bo)

    🍹 Other Improvements

    • [b8927b6a] - contributor: update contributors (Jina Dev Bot)
    • [232ec471] - ci: disable CLA bot (Han Xiao)
    • [f2aa3dc8] - cd: fix missing checkout in cd (Han Xiao)
    • [a76a875b] - docs: update TOC (Jina Dev Bot)
    • [03ad5807] - version: the next version will be 2.1.6 (Jina Dev Bot)
    Source code(tar.gz)
    Source code(zip)
  • v2.1.5(Sep 30, 2021)

    Release Note (2.1.5)

    Release time: 2021-09-30 14:18:19

    🙇 We'd like to thank all contributors for this new release! In particular, Joan Fontanals, Han Xiao, Wang Bo, Jina Dev Bot, AlaeddineAbdessalem, Roshan Jossy, Deepankar Mahapatro, Nan Wang, felix-wang, cristian, sauravgarg540, Florian Hönicke, Maximilian Werk, Tobias Jacobowitz, 🙇

    🆕 New Features

    • [dd453551] - daemon: allow gpus arg for remote Executors (#3510) (Deepankar Mahapatro)
    • [190f83d0] - add parser to jina hub new (#3448) (sauravgarg540)
    • [abd8e6c8] - k8s: add K8s connection pooling for grpc (#3275) (Tobias Jacobowitz)

    🐞 Bug fixes

    • [2ae33c71] - fix dam (#3530) (Joan Fontanals)
    • [4286a86f] - helper: fix progressbar print to flush (#3529) (Han Xiao)
    • [91f2cbd1] - fix test_da_csv_write on windows (#3516) (AlaeddineAbdessalem)
    • [3a8b1992] - fix plot, naming and kubernetes (#3445) (Joan Fontanals)
    • [86f653aa] - hubble: enforce download specific tag (#3472) (felix-wang)
    • [329f7979] - progress bar overwrite other logs (#3500) (Maximilian Werk)
    • [a1faf8f9] - fix skip code injection (#3497) (Joan Fontanals)

    🧼 Code Refactoring

    • [8870a95b] - refactor image blob to uri (#3523) (Wang Bo)
    • [ad78e88b] - RAII for pods (#3474) (Joan Fontanals)

    📗 Documentation

    • [ac6fb800] - docbot: position chatbox to be always on bottom (#3515) (Roshan Jossy)
    • [6d90faab] - remote: add gpu arg (#3511) (Deepankar Mahapatro)
    • [3f5ee8e3] - replace UUID with NAME (#3488) (Nan Wang)
    • [cbff5437] - fix reference to fill_embedding (#3506) (cristian)
    • [c21d32c9] - k8s console output (#3503) (Florian Hönicke)
    • [5e69dbc7] - minor typos in indexers (#3490) (cristian)

    🏁 Unit Test and CICD

    • [044fe4a8] - fix cd redundant needs (#3534) (Joan Fontanals)
    • [6f563785] - try to fix cd yml (#3526) (Joan Fontanals)
    • [fb3259f6] - xfail only parameter with x-python (#3517) (AlaeddineAbdessalem)
    • [82bf8294] - remove unexisting needs in cd.yml (#3512) (Joan Fontanals)
    • [4251a4df] - wait k8s test to finish (#3498) (Joan Fontanals)
    • [36b1eb5a] - skip code injection for forks (#3494) (Deepankar Mahapatro)

    🍹 Other Improvements

    • [4769b336] - ci: remove extra needs (Han Xiao)
    • [384aa4a0] - fix readme (Han Xiao)
    • [48c53232] - cd: fix job dependency in cd (Han Xiao)
    • [b2ec1648] - contributor: update contributors (Jina Dev Bot)
    • [e6051094] - cli: improve the help text remove redundant code (#3519) (Han Xiao)
    • [a5041bea] - clean document array abstract class (#3513) (AlaeddineAbdessalem)
    • [1420bdbe] - docs: update svg for hub user journey (Han Xiao)
    • [95bd814d] - update announcement in readme and docs (Jina Dev Bot)
    • [5e8a6f3e] - ci: fix announcement cd (Han Xiao)
    • [0e58aaee] - docs: update TOC (Jina Dev Bot)
    • [51ffcd5f] - version: the next version will be 2.1.5 (Jina Dev Bot)
    Source code(tar.gz)
    Source code(zip)
  • v2.1.4(Sep 27, 2021)

    Release Note (2.1.4)

    Release time: 2021-09-27 15:23:23

    🙇 We'd like to thank all contributors for this new release! In particular, cristian, Deepankar Mahapatro, Ce Gao, Jina Dev Bot, felix-wang, Han Xiao, Joan Fontanals, 🙇

    🆕 New Features

    • [e6da08f7] - daemon: custom dockerfile (#3482) (Deepankar Mahapatro)
    • [29099339] - windows: beta support (#3290) (Deepankar Mahapatro)
    • [39242dd8] - k8s uses native flag (#3478) (Joan Fontanals)

    ⚡ Performance Improvements

    • [4647b16e] - daemon: reload files across flows/executors (#3477) (Deepankar Mahapatro)

    🐞 Bug fixes

    • [618d7d2b] - fix service/deployment log (#3483) (Ce Gao)
    • [2b873963] - hubble: disk cache compatible across python versions (#3463) (felix-wang)
    • [31728be4] - fix get address table k8s (#3479) (Joan Fontanals)

    📗 Documentation

    • [e9d9a437] - add explanation on crud usage (#3486) (cristian)

    🍹 Other Improvements

    • [dcf3e7d4] - contributor: update contributors (Jina Dev Bot)
    • [195ebb55] - docs: add david and tadej avatars (Han Xiao)
    • [d4233844] - docs: fix windows support doc (Han Xiao)
    • [1e301204] - docs: fix announcement style (Han Xiao)
    • [a64f763f] - update announcement in readme and docs (Jina Dev Bot)
    • [c5da6474] - update announcement script (Han Xiao)
    • [c26acaf3] - update readme (Han Xiao)
    • [cdde1afe] - docs: fix sidebar links (Han Xiao)
    • [60b424b2] - docs: update TOC (Jina Dev Bot)
    • [c016b19b] - version: the next version will be 2.1.4 (Jina Dev Bot)
    Source code(tar.gz)
    Source code(zip)
  • v2.1.3(Sep 24, 2021)

    Release Note (2.1.3)

    Release time: 2021-09-24 16:14:48

    🙇 We'd like to thank all contributors for this new release! In particular, Deepankar Mahapatro, Han Xiao, Jina Dev Bot, cristian, Alex Cureton-Griffiths, Maximilian Werk, Wang Bo, Florian Hönicke, Tadej Svetina, 🙇

    🆕 New Features

    • [406b53ea] - docsbot: add thumbup/down to docs bot ui (#3465) (Han Xiao)
    • [f5f1a77f] - types: documentarray supports save and load with csv (#3460) (Han Xiao)
    • [ebdc442c] - expose path as getter for finetuner (#3453) (Wang Bo)
    • [b8f2641d] - k8s custom port expose (#3449) (Florian Hönicke)
    • [cb461923] - gateway: https support to uvicorn based servers (#3444) (Deepankar Mahapatro)
    • [4a5eee9f] - add setters and getters for da and dam (#3418) (Tadej Svetina)

    🐞 Bug fixes

    • [48285d12] - daemon: remote flow with parallel>1 (#3469) (Deepankar Mahapatro)
    • [707b7aac] - fix pod default naming in flow (#3473) (Han Xiao)
    • [3f35d3fd] - types: .match fn exclude_self should maintain num results (#3458) (Han Xiao)
    • [951ffc47] - batching for multi-valued blobs and embeddings (#3451) (Tadej Svetina)
    • [19f31c0a] - types: revert 3437 and fix dam in match (#3447) (Han Xiao)

    🧼 Code Refactoring

    • [51d962ec] - remove multimodal document (#3454) (Wang Bo)

    📗 Documentation

    • [008fca77] - more explicit naming (#3461) (cristian)
    • [9ff52c12] - readme: small polishing (#3457) (Alex Cureton-Griffiths)
    • [45620bec] - first indexer cheat sheet version (#3433) (Maximilian Werk)

    🏁 Unit Test and CICD

    • [c1e6d9a1] - docs: deploy to netlify on pr (#3459) (Deepankar Mahapatro)
    • [41ac8549] - docs: deploy docs from any branch (#3456) (Deepankar Mahapatro)

    🍹 Other Improvements

    • [c77521b8] - contributor: update contributors (Jina Dev Bot)
    • [9f0ff3a6] - fix pod default naming in flow (Han Xiao)
    • [eacff19e] - docs: rearrange experimental section (Han Xiao)
    • [f28e156d] - docs: update TOC (Jina Dev Bot)
    • [c97d99f4] - docs: update docs pypi deps (Han Xiao)
    • [907cf8e1] - docs: add manifest yaml explaination (Han Xiao)
    • [51b5cd41] - docs: fix section name (Han Xiao)
    • [89eba6ae] - version: the next version will be 2.1.3 (Jina Dev Bot)
    Source code(tar.gz)
    Source code(zip)
  • v2.1.2(Sep 20, 2021)

    Release Note (2.1.2)

    Release time: 2021-09-20 12:41:49

    🙇 We'd like to thank all contributors for this new release! In particular, AlaeddineAbdessalem, Han Xiao, Jina Dev Bot, 🙇

    🆕 New Features

    • [c9f2e651] - new image force (#3441) (AlaeddineAbdessalem)
    • [d2c1db8c] - add doc bot to doc website (#3440) (Han Xiao)
    • [053cc520] - types: add batch method to DA and DAM (#3435) (Han Xiao)
    • [9de19939] - helper: progressbar now has a spinner (#3434) (Han Xiao)

    🐞 Bug fixes

    • [b25945eb] - types: make _flatten of da and dam consistent (#3437) (Han Xiao)
    • [1d214f7f] - helper: fix spinner timer update (#3436) (Han Xiao)

    🍹 Other Improvements

    • [e51f6062] - docs: add powered by to docbot (Han Xiao)
    • [964e7174] - contributor: update contributors (Jina Dev Bot)
    • [86c842f5] - docs: add clean code on pop (Han Xiao)
    • [1616d567] - docs: fix typos in multiple docs (Han Xiao)
    • [20da7feb] - docs: update TOC (Jina Dev Bot)
    • [0684129e] - add pytest-lazy-fixture to extra-requirements.txt (Han Xiao)
    • [569ccafe] - version: the next version will be 2.1.2 (Jina Dev Bot)
    Source code(tar.gz)
    Source code(zip)
  • v2.1.1(Sep 18, 2021)

    Release Note (2.1.1)

    Release time: 2021-09-18 06:46:23

    🙇 We'd like to thank all contributors for this new release! In particular, Han Xiao, felix-wang, Deepankar Mahapatro, Jina Dev Bot, Tadej Svetina, Ce Gao, Tobias Jacobowitz, Raghav Prabhakar, Florian Hönicke, Anudeep Gunukula, cristian, Zhaofeng Miao, AlaeddineAbdessalem, 🙇

    🆕 New Features

    • [1619bef4] - implement blobs getter and setter (#3417) (Tadej Svetina)
    • [785d8f96] - hubio: do not pull latest version by default (#3424) (Han Xiao)
    • [e8dccccd] - add clear method (#3142) (Anudeep Gunukula)
    • [50a7e2d3] - flow polls ready status from every pod 🍾 (#3410) (Han Xiao)
    • [fbd0ec23] - types: allow match on traversal paths (#3405) (Han Xiao)

    ⚡ Performance Improvements

    • [3ba5ec18] - make slow pkg_resources import inline (#3409) (Han Xiao)

    🐞 Bug fixes

    • [bc35b414] - throw exception for positional arguments (#3310) (felix-wang)
    • [11916dc1] - hubio: only print single usage at a time (#3430) (Han Xiao)
    • [e7d569c6] - k8s: Do not hard code the FQDN (#3422) (Ce Gao)
    • [df59b5f5] - fix the wrong namespace in the example (#3429) (Ce Gao)
    • [feb33517] - daemon: improve error logging (#3423) (Tobias Jacobowitz)
    • [c7779895] - flow: raise error when using grpc data runtime with shards (#3364) (Raghav Prabhakar)
    • [d8c7be19] - hubio: make hub pulling progressbar smoother (Han Xiao)
    • [df462df0] - hubio: make hub pulling progressbar smoother (#3414) (Han Xiao)
    • [f7e45f74] - spawn: reregister jaml classes (#3407) (Deepankar Mahapatro)
    • [3bfcc517] - spawn: uses executor with quotes (#3404) (Deepankar Mahapatro)
    • [c939bf32] - hub: replace docker hub url with jina hub url, and it is clickable (#3403) (Zhaofeng Miao)
    • [cebb1e06] - hub: use name instead of uuid after hub pulling (#3402) (Zhaofeng Miao)
    • [9c98f60e] - remove unused class (#3374) (Ce Gao)

    🧼 Code Refactoring

    • [212b3bae] - types: fix traversal path type check (#3416) (Han Xiao)
    • [d37e6ab1] - helloworld: remove unnecessary code in hello world (#3395) (Han Xiao)

    📗 Documentation

    • [25810171] - daemon: refactor remote executors (#3432) (Deepankar Mahapatro)
    • [7c67bc9f] - add note on compound executor (#3406) (cristian)
    • [7a0a29b2] - add introduction of jina hub website (#3411) (Zhaofeng Miao)
    • [6004d571] - k8s: improve k8s documentations (#3412) (Han Xiao)
    • [660cb3e8] - add chatbot tutorial (#3382) (AlaeddineAbdessalem)
    • [1a6e894d] - add executor GPU tutorial (#3394) (Tadej Svetina)

    🏁 Unit Test and CICD

    • [eae4693d] - pre commit hook (#3426) (Florian Hönicke)

    🍹 Other Improvements

    • [fdcc0a3d] - fix glitches in readme (Han Xiao)
    • [0b5f4618] - contributor: update contributors (Jina Dev Bot)
    • [01ad9d85] - docs: fix broken link of the compound executor (Han Xiao)
    • [00d7a5fc] - move graph document to experimental section (Han Xiao)
    • [33f91b5c] - pods: Print logs with JinaLogger and add logs for k8s namespace creation (#3421) (Ce Gao)
    • [21671d69] - restructure doc tutorials orders (Han Xiao)
    • [9900b2b9] - remove eah banner (Han Xiao)
    • [53bfcedd] - update readme slack count (Han Xiao)
    • [73d074b4] - update docs link (Han Xiao)
    • [40589cd9] - docs: update TOC (Jina Dev Bot)
    • [327b52ee] - add _path to memmap to ease copy (#3392) (Han Xiao)
    • [ce061940] - fix doc style in jinad (Han Xiao)
    • [e2eb8d66] - fix doc style (Han Xiao)
    • [64b64155] - version: the next version will be 2.1.1 (Jina Dev Bot)
    Source code(tar.gz)
    Source code(zip)
  • v2.1.0(Sep 12, 2021)

    Release Note (2.1.0)

    Release time: 2021-09-12 12:07:58

    🙇 We'd like to thank all contributors for this new release! In particular, Han Xiao, Jina Dev Bot, 🙇

    🐞 Bug fixes

    • [59b67fa8] - executor: improve checking for executor init and @requests func (#3391) (Han Xiao)

    🍹 Other Improvements

    • [55954e69] - bump version to 2.1 (Han Xiao)
    • [33f0ded3] - version: the next version will be 2.0.25 (Jina Dev Bot)
    Source code(tar.gz)
    Source code(zip)
  • v2.0.24(Sep 12, 2021)

    Release Note (2.0.24)

    Release time: 2021-09-12 06:45:12

    🙇 We'd like to thank all contributors for this new release! In particular, Jina Dev Bot, Roshan Jossy, Bharat Raghunathan, Han Xiao, cristian, Aakash Jhawar, Tadej Svetina, Tobias Jacobowitz, Florian Hönicke, Sebastian Lettner, dependabot[bot], Winston Wong, felix-wang, AlaeddineAbdessalem, Maximilian Werk, Alan Zhisheng Niu, Wang Bo, David Buchaca Prats, Zhaofeng Miao, Nishil, Maksudur Rahman Maateen, 🙇

    🆕 New Features

    • [8a47de91] - flow: add install_requirments args when using jinahub (#3324) (felix-wang)
    • [b33e6693] - match: add exclude_self to match func (#3352) (Han Xiao)
    • [cbe578dc] - k8s parallel for shards (#3350) (Florian Hönicke)
    • [7fe4c3e2] - k8s: client access from within the cluster (#3344) (Tobias Jacobowitz)

    ⚡ Performance Improvements

    • [ad3961a2] - type: document fast init (#3339) (AlaeddineAbdessalem)
    • [b7797d7e] - embeddings (#3338) (David Buchaca Prats)

    🐞 Bug fixes

    • [3e8c35e0] - empty DA match should be just nop (#3385) (Han Xiao)
    • [2cf15df5] - uses=Executor should work with spawn start method (#3380) (Tadej Svetina)
    • [b0e46a30] - types: allow da extend with None (#3379) (Tobias Jacobowitz)
    • [71ef9711] - clean-up and re-ordering of jina hub new table (#3369) (cristian)
    • [63c04bb7] - k8s needs (#3372) (Florian Hönicke)
    • [d78ff672] - routing: fix static routing table args (#3363) (Tobias Jacobowitz)
    • [c0b04ffa] - type: fix check on unique content field (#3366) (Han Xiao)
    • [972e718e] - name can be None in args (#3362) (cristian)
    • [6a272d93] - #3351 (#3353) (cristian)
    • [f185a2c1] - k8s: uses with (#3354) (Florian Hönicke)
    • [fc340722] - dam can be returned by an executor (#3348) (Maximilian Werk)
    • [1feb09fe] - small changes to conda script (#3336) (Tadej Svetina)

    🧼 Code Refactoring

    • [dcd3c8eb] - protobuf: use .uri as an independent attribute (#3347) (felix-wang)
    • [77eee702] - unnecessary raise as builtin sample already raise (#3355) (Han Xiao)
    • [d4b7834b] - remove redundant code (#3318) (Florian Hönicke)
    • [40416c21] - hub: change alias to name (#3335) (Zhaofeng Miao)
    • [b2d3577f] - make operators private (#3325) (Wang Bo)

    📗 Documentation

    • [e5ca2e08] - readme: remove deprecated link(s) (#3388) (Bharat Raghunathan)
    • [504614e5] - add tutorial on building executor (#3371) (cristian)
    • [43215cd0] - add note on indexers (#3375) (cristian)
    • [0a366371] - kubernetes cook book (#3357) (Florian Hönicke)
    • [8dcbc328] - type: restructure document type (Han Xiao)
    • [673178a6] - fix css styling and typos (Han Xiao)
    • [5f32945b] - type: restructure document type (#3361) (Han Xiao)
    • [eab95eba] - add note to warn users about int data type in doc.tags and parameters (#3349) (Winston Wong)
    • [97833376] - use port replace port expose in client (#3340) (Wang Bo)
    • [7c61a227] - add conda install instructions (#3337) (Tadej Svetina)
    • [f76326d4] - fix typo (Han Xiao)
    • [3bf0a77f] - add recommend way to perform heavy lift jobs (#3330) (Wang Bo)
    • [e645d87b] - fix document array creation (#3320) (Wang Bo)
    • [58367d5b] - document: remove typo and small rewrite of match explanation (#3319) (David Buchaca Prats)

    🏁 Unit Test and CICD

    • [7a3f706e] - docs: fix broken styles in 404 page (#3376) (Roshan Jossy)
    • [7a1d6abd] - k8s: deployment utils (#3367) (Sebastian Lettner)
    • [501a9e85] - k8s: unit test tools (#3359) (Sebastian Lettner)
    • [a11ea9cf] - pods k8s parse args unit test (#3358) (Sebastian Lettner)
    • [9a8c4317] - classify flow unit test (#3346) (Alan Zhisheng Niu)
    • [747b2cd8] - enable workflows for external contributors (#3204) (Maksudur Rahman Maateen)

    🍹 Other Improvements

    • [42703057] - contributor: update contributors (Jina Dev Bot)
    • [61b56e62] - fix typo on daemon dockerize (#3383) (Aakash Jhawar)
    • [2c05ab76] - update docs for index executors (Han Xiao)
    • [817843a5] - deps: bump pillow in /scripts/latency-tracking (#3365) (dependabot[bot])
    • [700b137a] - fix readme typo (Han Xiao)
    • [442a54d2] - update readme banner (Han Xiao)
    • [97d0dedf] - fix readme (Han Xiao)
    • [3da47f52] - add eah title (#3329) (Nishil)
    • [3c263658] - docs: update TOC (Jina Dev Bot)
    • [22456093] - version: the next version will be 2.0.24 (Jina Dev Bot)
    Source code(tar.gz)
    Source code(zip)
  • v2.0.23(Sep 1, 2021)

    Release Note (2.0.23)

    Release time: 2021-09-01 12:10:20

    🙇 We'd like to thank all contributors for this new release! In particular, Sebastian Lettner, Bastin, Deepankar Mahapatro, Han Xiao, AlaeddineAbdessalem, Jina Dev Bot, 🙇

    🆕 New Features

    • [033dc123] - k8s template folder lean (#3309) (Sebastian Lettner)
    • [c1ce3eb1] - disable button and search animation (#3313) (Bastin)

    🐞 Bug fixes

    • [20d5c138] - rolling update with container runtime (#3312) (Deepankar Mahapatro)

    📗 Documentation

    • [cf176ab7] - logo: change to static logo (Han Xiao)
    • [c9194319] - hub: fix figure in the hub docs (#3316) (Han Xiao)
    • [12c2672f] - hub: add hub executor best practices (#3315) (AlaeddineAbdessalem)

    🍹 Other Improvements

    • [e41f81ac] - fix typo in readme (Han Xiao)
    • [ad5241b2] - docs: update TOC (Jina Dev Bot)
    • [a6f09eac] - contributor: update contributors (Jina Dev Bot)
    • [1eca0631] - remove content from all cookbooks (Han Xiao)
    • [b7f62413] - version: the next version will be 2.0.23 (Jina Dev Bot)
    Source code(tar.gz)
    Source code(zip)
  • v2.0.22(Aug 31, 2021)

    Release Note (2.0.22)

    Release time: 2021-08-31 22:08:52

    🙇 We'd like to thank all contributors for this new release! In particular, Han Xiao, AlaeddineAbdessalem, Maksudur Rahman Maateen, Deepankar Mahapatro, Jina Dev Bot, Florian Hönicke, David Buchaca Prats, 🙇

    🆕 New Features

    • [b8a61b61] - kubernetes mvp (#2939) (Florian Hönicke)

    ⚡ Performance Improvements

    • [8ac9bc50] - add cache nodes (#3295) (David Buchaca Prats)

    🐞 Bug fixes

    • [a2e9ebbf] - ci: fixed benchmark CI (#3311) (Maksudur Rahman Maateen)
    • [4221b258] - daemon: port mapping for compound pods (#3304) (Deepankar Mahapatro)
    • [ca7375aa] - daemon: manage ports for remote flows (#3266) (Deepankar Mahapatro)

    📗 Documentation

    • [370ff14d] - update hub documentations (Han Xiao)
    • [973a2fbd] - migrate & rewrite cookbooks to the doc website (#3282) (AlaeddineAbdessalem)

    🍹 Other Improvements

    • [bec21e75] - fix release note script (Han Xiao)
    • [06e50113] - fix typo in readme (Han Xiao)
    • [0acf86ad] - move cookbook links to new docs website (Han Xiao)
    • [c818f9d9] - docs: fix inline-tab dependencies (Han Xiao)
    • [1cc72fd5] - contributor: update contributors (Jina Dev Bot)
    • [800b85a9] - Refactor don't resolve host (#3296) (Florian Hönicke)
    • [52cd3074] - update slack counter (Han Xiao)
    • [dae5f656] - docs: update TOC (Jina Dev Bot)
    • [9cde0711] - version: the next version will be 2.0.22 (Jina Dev Bot)
    Source code(tar.gz)
    Source code(zip)
  • v2.0.21(Aug 28, 2021)

    Release Note (2.0.21)

    Release time: 2021-08-28 10:19:06

    🙇 We'd like to thank all contributors for this new release! In particular, Han Xiao, David Buchaca Prats, felix-wang, Tadej Svetina, Maksudur Rahman Maateen, Jina Dev Bot, Joan Fontanals, Maximilian Werk, Paul-Louis NECH, Yongxuan Zhang, Deepankar Mahapatro, Tobias Jacobowitz, 🙇

    🆕 New Features

    • [329e3d46] - add conda recipe script (#3174) (Tadej Svetina)
    • [0a66a7c0] - add edges and nodes in batches (#3259) (Joan Fontanals)
    • [e46d974c] - make docs behavior consistent :boom: (#3227) (Maximilian Werk)
    • [384e91bd] - ci: enabled automated benchmarking (#3248) (Maksudur Rahman Maateen)
    • [d8d80708] - daemon: support envvars (#3251) (Deepankar Mahapatro)
    • [3fa30c37] - helper: add update desc & details for progressbar (#3255) (Han Xiao)

    ⚡ Performance Improvements

    • [53f961d4] - reuse loop to add edge features (#3274) (David Buchaca Prats)

    🐞 Bug fixes

    • [0147ad76] - script: fix port_expose in jinad (Han Xiao)
    • [cb16849f] - dam: warning truncate doc id (#3264) (felix-wang)
    • [fa444fa9] - ci: fixed benchmark ci (#3285) (Maksudur Rahman Maateen)
    • [07f7636d] - fix multiple import problem (#3276) (Joan Fontanals)
    • [ff2f9a3f] - check runtime class executor native (#3273) (Joan Fontanals)
    • [0e038ea2] - helper: fix on done rendering for progressbar (#3269) (Han Xiao)
    • [dc605e6d] - ci: fixed benchmark ci error (#3268) (Maksudur Rahman Maateen)
    • [caf132db] - fix document update (#3263) (Joan Fontanals)
    • [dbb599b1] - ci: fixed benchmark workflow syntax (#3262) (Maksudur Rahman Maateen)
    • [8a67a40e] - with args in flow.save_config (#3256) (Deepankar Mahapatro)
    • [7dd9db88] - grpc: enable grpc runtime for http and websocket gateway (#3244) (Tobias Jacobowitz)
    • [c1c8d396] - helper: fix small progressbar rendering problem (#3246) (Han Xiao)

    🧼 Code Refactoring

    • [03df9f48] - port renaming :boom: (#3254) (Maximilian Werk)

    📗 Documentation

    • [701729de] - readme: update console output to match (#3100) (Paul-Louis NECH)

    🏁 Unit Test and CICD

    • [791bb567] - fix test on mac (#3249) (Yongxuan Zhang)

    🍹 Other Improvements

    • [d7a85f58] - comment out twine upload (Han Xiao)
    • [b4385d28] - contributor: update contributors (Jina Dev Bot)
    • [d68d532d] - ci: fix doc build job (Han Xiao)
    • [553d2116] - docs: update TOC (Jina Dev Bot)
    • [243475f7] - helper: remove redundant progress info (Han Xiao)
    • [4967d6a8] - fix doc gen job (Han Xiao)
    • [ef71b991] - update doc website theme (Han Xiao)
    • [a7cb0921] - version: the next version will be 2.0.21 (Jina Dev Bot)
    Source code(tar.gz)
    Source code(zip)
  • v2.0.20(Aug 22, 2021)

    Release Note (2.0.20)

    Release time: 2021-08-22 12:31:55

    🙇 We'd like to thank all contributors for this new release! In particular, Han Xiao, Jina Dev Bot, Deepankar Mahapatro, David Buchaca Prats, Maximilian Werk, Tobias Jacobowitz, Tadej Svetina, Roshan Jossy, Piper, Joan Fontanals, Zhaofeng Miao, Nan Wang, 🙇

    🆕 New Features

    • [fcf17a9c] - daemon: add timeout to client (#3236) (Deepankar Mahapatro)
    • [b4526c97] - routing: add static routing table feature (#3222) (Tobias Jacobowitz)
    • [83f60053] - add structuring executor repositories best practices (#3156) (Tadej Svetina)
    • [a41205be] - hubio: modify fields of manifest produced by jina hub new (#3215) (Zhaofeng Miao)

    ⚡ Performance Improvements

    • [c659685a] - helloworld: optimize the performance of hello fashion (#3239) (Han Xiao)
    • [de6d3429] - remove environment variable reading from init (#3235) (David Buchaca Prats)

    🐞 Bug fixes

    • [8e13ad2c] - helper: fix bad warning message on unknown cli (#3237) (Han Xiao)
    • [0649cce1] - no warning for metas in executor (#3219) (Maximilian Werk)
    • [4d86d781] - types: fix split swallow the category at float zero (#3226) (Han Xiao)
    • [b31c4c2d] - fix graph plot (#3220) (Joan Fontanals)
    • [664bf1ad] - add jina-bot to the cla whitelist (#3216) (Nan Wang)
    • [21a3dfa4] - fix listview insert, append (#3212) (Joan Fontanals)
    • [1517ce8c] - license should be included in distribution (#3213) (Tadej Svetina)

    🧼 Code Refactoring

    • [91be0f76] - helper: make progressbar more useful and modern (#3238) (Han Xiao)
    • [706b41cc] - except: rename Flow version except (#3224) (Han Xiao)

    📗 Documentation

    • [fd2c6500] - readme: update links for tutorials and blogs (#3214) (Roshan Jossy)
    • [957fbee9] - update the number of slack members (#3228) (Piper)

    🍹 Other Improvements

    • [31273bd8] - contributor: update contributors (Jina Dev Bot)
    • [53f49e33] - docs: update TOC (Jina Dev Bot)
    • [8d9ee246] - changed pull_request_target to pull_request (#3189) (Deepankar Mahapatro)
    • [55c49447] - comment out walkthroughs until links fixed (Han Xiao)
    • [88d8a6fb] - version: the next version will be 2.0.20 (Jina Dev Bot)
    Source code(tar.gz)
    Source code(zip)
  • v2.0.19(Aug 18, 2021)

    Release Note (2.0.19)

    Release time: 2021-08-18 11:49:12

    🙇 We'd like to thank all contributors for this new release! In particular, Han Xiao, Joan Fontanals, Maximilian Werk, Jina Dev Bot, Maksudur Rahman Maateen, David Buchaca Prats, Wang Bo, AlaeddineAbdessalem, Tadej Svetina, Deepankar Mahapatro, Tobias Jacobowitz, cristian, 🙇

    💥 Breaking changes

    • [c2eca9f6] - cli: 💥 change zed_runtime to executor --native (#3196) (Han Xiao)

    🆕 New Features

    • [53e4607c] - pass native in container runtime docker (#3200) (Joan Fontanals)
    • [2939170b] - blobs property (#3198) (Joan Fontanals)
    • [befb7cd3] - unknown args loggin (#3172) (Maximilian Werk)
    • [b32a0a56] - changed pull_request_target to pull_request (#3189) (Maksudur Rahman Maateen)
    • [21a62afc] - add setter to embeddings (#3190) (David Buchaca Prats)
    • [7a700e5e] - add split prepare for trainer (#3167) (Wang Bo)
    • [6cb52827] - grpc for requests (#3129) (Tobias Jacobowitz)
    • [2f1563d5] - add efficient access to embeddings (#3157) (David Buchaca Prats)

    ⚡ Performance Improvements

    • [f44e6c90] - types: remove warning logic in binary doc construction (#3207) (Han Xiao)
    • [52ef3d64] - use embeddings property in cache (#3184) (Wang Bo)
    • [44ea0269] - cache get attribute embedding (#3130) (Wang Bo)

    🐞 Bug fixes

    • [945adb8d] - setup: add core deps to perf deps (#3203) (Han Xiao)
    • [6403c4b1] - types: fix document id type hint (#3183) (Han Xiao)
    • [cb809d81] - traverse chunk match (#3179) (AlaeddineAbdessalem)
    • [9146979f] - math: normalization after topk with original minmax value (#3176) (Han Xiao)
    • [2a3ac6b1] - daemon: hub cache env var (#3175) (Deepankar Mahapatro)
    • [4c4f69b1] - math: bound cosine and minmax norm and fix divide-zero (#3173) (Han Xiao)
    • [25bb9030] - executor point to pea again (#3144) (Joan Fontanals)
    • [3f7e96ba] - errorlog from partiald (#3164) (Deepankar Mahapatro)
    • [1cca4af0] - fix error message test (#3153) (Joan Fontanals)
    • [284872f5] - catch RuntimeError on signal handler set (#3152) (Tobias Jacobowitz)
    • [630a7353] - allow more versions of dgl (#3150) (David Buchaca Prats)

    🧼 Code Refactoring

    • [8c68f75b] - remove imports (#3177) (Joan Fontanals)
    • [152e2e95] - remove unused imports (#3158) (Joan Fontanals)
    • [47249582] - remove dict keys creation (#3151) (David Buchaca Prats)
    • [ab16cee0] - no copy on append (#3148) (Joan Fontanals)

    📗 Documentation

    • [97cd0592] - add links to tutorials in readme (#3137) (cristian)

    🏁 Unit Test and CICD

    • [f52ba2d9] - inject Document overload during CI (#3162) (AlaeddineAbdessalem)
    • [f949975a] - increase hub test coverage (#3147) (AlaeddineAbdessalem)

    🍹 Other Improvements

    • [b525b1d9] - contributor: update contributors (Jina Dev Bot)
    • [397de240] - merge demo dependencies (#3180) (Tadej Svetina)
    • [8f3891f0] - merge latency tracking repo (#3047) (Maksudur Rahman Maateen)
    • [58866a9a] - docs: update TOC (Jina Dev Bot)
    • [caa019a7] - update readme (Han Xiao)
    • [30157b4a] - docs: fix readme typo (Han Xiao)
    • [91161fbe] - version: the next version will be 2.0.19 (Jina Dev Bot)
    Source code(tar.gz)
    Source code(zip)
  • v2.0.18(Aug 11, 2021)

    Release Note (2.0.18)

    Release time: 2021-08-11 07:04:34

    🙇 We'd like to thank all contributors for this new release! In particular, Han Xiao, Jina Dev Bot, Deepankar Mahapatro, Zhaofeng Miao, AlaeddineAbdessalem, 🙇

    🧼 Code Refactoring

    • [2b79c067] - setup: make standard deps as the default deps (#3145) (Han Xiao)

    📗 Documentation

    • [73589246] - daemon in readme (#3122) (Deepankar Mahapatro)
    • [b7a1cf7f] - hub: add description about the tag of hub (#3125) (Zhaofeng Miao)
    • [0eb06a94] - update dam chapter and add uses_with , uses_metas , uses_requests to Flow.md (#3146) (AlaeddineAbdessalem)

    🍹 Other Improvements

    • [2e22f88d] - docs: fix layout of jinad markdown (Han Xiao)
    • [6c9704c2] - docs: update TOC (Jina Dev Bot)
    • [2cc4fb0b] - contributor: update contributors (Jina Dev Bot)
    • [994a61c0] - docs: fix layout of jinad (Han Xiao)
    • [92613c32] - version: the next version will be 2.0.18 (Jina Dev Bot)
    Source code(tar.gz)
    Source code(zip)
  • v2.0.17(Aug 10, 2021)

    Release Note (2.0.17)

    Release time: 2021-08-10 12:17:55

    🙇 We'd like to thank all contributors for this new release! In particular, Paul-Louis NECH, Joan Fontanals, Jina Dev Bot, felix-wang, AlaeddineAbdessalem, Roshan Jossy, Han Xiao, David Buchaca Prats, Zhaofeng Miao, Maximilian Werk, Wang Bo, cristian, Deepankar Mahapatro, Sebastian Lettner, 🙇

    🆕 New Features

    • [9745d8d6] - unify da dam interface (#3002) (AlaeddineAbdessalem)
    • [668cca7c] - match with batching capability (#3113) (David Buchaca Prats)
    • [1f03fefe] - hubble: push/pull rich logging (#3104) (felix-wang)
    • [dbe17886] - supporting jina hub push --tag and --docker-file (#3116) (Zhaofeng Miao)
    • [fa7c4770] - add data request handler (#3105) (Joan Fontanals)
    • [4e14b19e] - daemon: rolling update via client (#3111) (Deepankar Mahapatro)
    • [3329576f] - change default entrypoint and allow runtimes in main process (#2985) (Joan Fontanals)
    • [990a51d6] - executor: add uses-requests cli to override requests in Executor (#3092) (Han Xiao)
    • [3ff5fe98] - connect to pred config (#3051) (Maximilian Werk)

    🐞 Bug fixes

    • [845e8805] - hubble: docker pull progressbar (#3136) (felix-wang)
    • [c7a2f60c] - hubio: more meaningful exception when docker not running (#3131) (Han Xiao)
    • [01564f94] - http: tag update (#3108) (Deepankar Mahapatro)
    • [55df38f7] - delete and list workspaces (#3102) (Deepankar Mahapatro)
    • [44054eea] - add sparse support (#3097) (David Buchaca Prats)

    🧼 Code Refactoring

    • [2aee8360] - improve document update method (#3112) (Wang Bo)
    • [c69d97bf] - remove attribute (#3117) (Joan Fontanals)
    • [ed3ff21e] - change bind to all inbound (#3069) (Sebastian Lettner)

    📗 Documentation

    • [b099fc30] - Executor.md: Minor typos (#3101) (Paul-Louis NECH)
    • [9e20540d] - update cookbooks #138 (#3098) (AlaeddineAbdessalem)
    • [0263d30a] - add black logo (#3135) (Roshan Jossy)
    • [c6999e13] - add black logo (#3133) (Roshan Jossy)
    • [f9755fd7] - added add method configs (#3066) (Maximilian Werk)
    • [6a58fdef] - fix guy - person (#3115) (cristian)

    🏁 Unit Test and CICD

    • [e7098354] - refactor test (#3132) (Joan Fontanals)

    🍹 Other Improvements

    • [ab7e6a5b] - docs: update TOC (Jina Dev Bot)
    • [0d8544fb] - contributor: update contributors (Jina Dev Bot)
    • [7394523a] - revert add black logo (#3133)" (#3134) (Han Xiao)
    • [066c2938] - version: the next version will be 2.0.17 (Jina Dev Bot)
    Source code(tar.gz)
    Source code(zip)
  • v2.0.16(Aug 3, 2021)

    Release Note (2.0.16)

    Release time: 2021-08-03 09:18:53

    🙇 We'd like to thank all contributors for this new release! In particular, Han Xiao, Jina Dev Bot, Joan Fontanals, 🙇

    ⚡ Performance Improvements

    • [bac641d9] - types: remove unnecessary DA building in get_attributes (#3064) (Han Xiao)

    🐞 Bug fixes

    • [183b76f5] - init: bring back default fork mp start method on macos (#3084) (Han Xiao)
    • [4951b3d4] - setup: fix missing wheel dependencies in setup.py (#3078) (Han Xiao)
    • [50373245] - fix plot (#3074) (Joan Fontanals)
    • [671a310b] - fix 0 sparse array (#3073) (Joan Fontanals)
    • [27d93c25] - math: fix minmax normalize to row-wise normalization (#3065) (Han Xiao)

    🍹 Other Improvements

    • [046356fd] - docs: update TOC (Jina Dev Bot)
    • [11c78e0f] - contributor: update contributors (Jina Dev Bot)
    • [a71ee9a7] - version: the next version will be 2.0.16 (Jina Dev Bot)
    Source code(tar.gz)
    Source code(zip)
  • v2.0.15(Aug 2, 2021)

    Release Note (2.0.15)

    Release time: 2021-08-02 02:30:24

    🙇 We'd like to thank all contributors for this new release! In particular, Han Xiao, Jina Dev Bot, 🙇

    🆕 New Features

    • [868d87b6] - types: add tsne algorithm to visualization (#3063) (Han Xiao)
    • [999b4fa9] - add env JINA_MP_START_METHOD for changing mp start_method (#3062) (Han Xiao)
    • [dfc474e6] - types: improve usability of visualize function (#3059) (Han Xiao)

    ⚡ Performance Improvements

    • [4543fcaf] - types: remove cache_invalidator in DA for better performance (#3060) (Han Xiao)

    🍹 Other Improvements

    • [3d33a39f] - docs: update TOC (Jina Dev Bot)
    • [001f4c61] - contributor: update contributors (Jina Dev Bot)
    • [f59930d5] - update readme and contributing guide (Han Xiao)
    • [f9255586] - remove google colab button (Han Xiao)
    • [e420e22b] - version: the next version will be 2.0.15 (Jina Dev Bot)
    Source code(tar.gz)
    Source code(zip)
  • v2.0.14(Jul 31, 2021)

    Release Note (2.0.14)

    Release time: 2021-07-31 14:46:52

    🙇 We'd like to thank all contributors for this new release! In particular, Han Xiao, David Buchaca Prats, Wang Bo, Jina Dev Bot, Deepankar Mahapatro, Zhaofeng Miao, 🙇

    🆕 New Features

    • [63d59edc] - pca algorithm (#3015) (David Buchaca Prats)

    🐞 Bug fixes

    • [b2292d56] - helloworld: fix doc content to 1d array (#3050) (Han Xiao)
    • [c687fac6] - http: fix pydantic models (#3046) (Han Xiao)
    • [f40d7964] - match with same documents (#3040) (Wang Bo)
    • [8f01fcec] - hub: fix typo base image name (#3039) (Zhaofeng Miao)
    • [635435b7] - add Dockerfile to resources (Han Xiao)
    • [6564e91e] - fashion fixh writting as black and white image (#3032) (David Buchaca Prats)

    🚧 Code Refactoring

    • [390ff0ab] - proto: remove content_hash from builtin proto (#3058) (Han Xiao)
    • [74aaa7e8] - document: do not hash content if copy is true (#3038) (Wang Bo)

    🏁 Unit Test and CICD

    • [85f6d60d] - disable cancel prev runs (Han Xiao)
    • [4694b76f] - default token for cancel workflow (#3042) (Deepankar Mahapatro)
    • [ace7f5b9] - remove unnecessary dev bot token (#3035) (Han Xiao)
    • [4b2f79a4] - daemon: py module (#3033) (Deepankar Mahapatro)

    🍹 Other Improvements

    • [9f71c889] - update readme (Han Xiao)
    • [b205a199] - types: remove redundant copy in match function (#3056) (Han Xiao)
    • [ab657d98] - speed up update content hash (#3045) (Wang Bo)
    • [78d80f97] - contributor: update contributors (Jina Dev Bot)
    • [b0b50bef] - docs: update TOC (Jina Dev Bot)
    • [e7e08852] - update cookbook (Han Xiao)
    • [76cfb37b] - version: the next version will be 2.0.14 (Jina Dev Bot)
    Source code(tar.gz)
    Source code(zip)
  • v2.0.12(Jul 28, 2021)

    Release Note (2.0.12)

    Release time: 2021-07-28 13:31:04

    🙇 We'd like to thank all contributors for this new release! In particular, Han Xiao, Wang Bo, Deepankar Mahapatro, Maksudur Rahman Maateen, Anirudh Anup, Jina Dev Bot, 🙇

    🆕 New Features

    • [28a38186] - hub: add new command to jina hub CLI (#3031) (Han Xiao)
    • [1582623e] - enabled benchmarking into ci (#3001) (Maksudur Rahman Maateen)
    • [221c00bc] - daemon: python client (#2994) (Deepankar Mahapatro)
    • [2302e456] - add shuffle to documentarray (#3021) (Wang Bo)

    🐞 Bug fixes

    • [ec73b170] - latency tracking long running issue (#3020) (Maksudur Rahman Maateen)

    🚧 Code Refactoring

    • [fcaeb1ae] - parser: clarify gpu argument description (#3023) (Wang Bo)

    📗 Documentation

    • [36fe25b1] - remote executor usage (#3028) (Deepankar Mahapatro)

    🏁 Unit Test and CICD

    • [5f150f18] - update ubuntu base image to latest (Han Xiao)

    🍹 Other Improvements

    • [2e7fafce] - update readme (Han Xiao)
    • [3b137f9e] - updated README.md (#3008) (Anirudh Anup)
    • [be7d39d3] - docs: update TOC (Jina Dev Bot)
    • [c62a8558] - contributor: update contributors (Jina Dev Bot)
    • [a6055cd4] - version: the next version will be 2.0.12 (Jina Dev Bot)
    Source code(tar.gz)
    Source code(zip)
  • v2.0.13(Jul 28, 2021)

    Release Note (2.0.13)

    Release time: 2021-07-28 13:31:04

    🙇 We'd like to thank all contributors for this new release! In particular, Han Xiao, Jina Dev Bot, 🙇

    🍹 Other Improvements

    • [ed264bcc] - remove benchmark core in tag (Han Xiao)
    • [2ccab7d6] - version: the next version will be 2.0.13 (Jina Dev Bot)
    Source code(tar.gz)
    Source code(zip)
  • v2.0.11(Jul 27, 2021)

    Release Note (2.0.11)

    Release time: 2021-07-27 07:41:40

    🙇 We'd like to thank all contributors for this new release! In particular, felix-wang, Han Xiao, Jina Dev Bot, Wang Bo, Deepankar Mahapatro, David Buchaca Prats, Joan Fontanals, AlaeddineAbdessalem, Maksudur Rahman Maateen, Alex Cureton-Griffiths, Kelly, 🙇

    🆕 New Features

    • [d7fae97b] - types: allow scipy metric in match and add minmax normalization (#3019) (Han Xiao)
    • [ba2e5370] - add sample method to document array (#3003) (Wang Bo)
    • [46ddd13f] - flow: add uses on flow plot (#3006) (Han Xiao)
    • [64591c97] - find method for DocumentArray (#2997) (David Buchaca Prats)
    • [f3fe76e9] - changed latency tracking Dockerfile location (#2969) (Maksudur Rahman Maateen)

    🐞 Bug fixes

    • [75c28fc7] - importer: add custom sys meta path finder (#2953) (felix-wang)
    • [5ba5c3d1] - cli: fix lookup on hidden args (#3018) (Han Xiao)
    • [e60ec064] - hubble: more informed logging for fetch_meta (#3009) (felix-wang)
    • [7c9c91a5] - types: clear matches before adding (#3005) (Han Xiao)
    • [a7e53849] - add missing tests (#3004) (David Buchaca Prats)
    • [f02f2e85] - apply islice on iterator object of data in batch iterator (#2993) (AlaeddineAbdessalem)

    🚧 Code Refactoring

    • [aea7c9e4] - types: move common math function to jina.math module (#3016) (Han Xiao)

    📗 Documentation

    • [04138141] - add match explanation in cookbook (#2995) (David Buchaca Prats)

    🏁 Unit Test and CICD

    • [8802287b] - cancel prev runs for tests (#3007) (Deepankar Mahapatro)
    • [a493cf03] - add test rolling update docker (#2999) (Joan Fontanals)
    • [50e14df4] - test remote pod with parallel (#2998) (Joan Fontanals)

    🍹 Other Improvements

    • [598613a1] - docs: update TOC (Jina Dev Bot)
    • [00286d64] - contributor: update contributors (Jina Dev Bot)
    • [9435a47c] - update banner in readme (Han Xiao)
    • [ead6c842] - add meetup banner (#2996) (Alex Cureton-Griffiths)
    • [e08db428] - update gif animator in readme (Han Xiao)
    • [3d413364] - adding new readme logo iteration animation (#2988) (Kelly)
    • [8598c077] - fix typo in readme (Han Xiao)
    • [64829ea7] - version: the next version will be 2.0.11 (Jina Dev Bot)
    Source code(tar.gz)
    Source code(zip)
  • v2.0.10(Jul 22, 2021)

    Release Note (2.0.10)

    Release time: 2021-07-22 06:18:39

    🙇 We'd like to thank all contributors for this new release! In particular, Han Xiao, Jina Dev Bot, 🙇

    🆕 New Features

    • [723c7d91] - cli: add jina help cli for quick arg lookup (Han Xiao)
    • [0ca4f7a3] - cli: add jina help cli for quick arg lookup (#2989) (Han Xiao)

    🚧 Code Refactoring

    • [2e402c9d] - types: improving style and code quality of .match method (#2987) (Han Xiao)

    🍹 Other Improvements

    • [04790803] - docs: update TOC (Jina Dev Bot)
    • [15983b9d] - contributor: update contributors (Jina Dev Bot)
    • [da23c8d4] - version: the next version will be 2.0.10 (Jina Dev Bot)
    Source code(tar.gz)
    Source code(zip)
  • v2.0.9(Jul 21, 2021)

    Release Note (2.0.9)

    Release time: 2021-07-21 14:38:46

    🙇 We'd like to thank all contributors for this new release! In particular, David Buchaca Prats, Wang Bo, Han Xiao, AlaeddineAbdessalem, Joan Fontanals, Jina Dev Bot, Deepankar Mahapatro, 🙇

    🆕 New Features

    • [9a231c40] - add match function (#2942) (David Buchaca Prats)

    🐞 Bug fixes

    • [5217ea97] - cli: parse known args only and warn unknown (#2986) (Han Xiao)
    • [133c90ba] - seek body file when computing _start (#2980) (AlaeddineAbdessalem)
    • [32dc3dc3] - fix setting extra host_connect_in (#2973) (Joan Fontanals)
    • [10117c14] - ci: change latency tracking path (#2981) (Joan Fontanals)
    • [a6afc9b8] - types: allow setter Document.tags for StructView (#2976) (Han Xiao)
    • [6bea7f24] - daemon: dockersock mounting for mac (#2971) (Deepankar Mahapatro)

    🚧 Code Refactoring

    • [50cc25df] - update hello examples (#2984) (David Buchaca Prats)

    📗 Documentation

    • [8d0fe594] - cookbook: add document creation clean code (#2983) (Wang Bo)

    🍹 Other Improvements

    • [99479dc1] - contributor: update contributors (Jina Dev Bot)
    • [8d2c7d37] - docs: update TOC (Jina Dev Bot)
    • [9fc9a726] - version: the next version will be 2.0.9 (Jina Dev Bot)
    Source code(tar.gz)
    Source code(zip)
  • v2.0.8(Jul 19, 2021)

    Release Note (2.0.8)

    Release time: 2021-07-19 03:06:17

    🙇 We'd like to thank all contributors for this new release! In particular, Jina Dev Bot, Wang Bo, Han Xiao, felix-wang, Zhaofeng Miao, Deepankar Mahapatro, Joan Fontanals, Alex Cureton-Griffiths, Maksudur Rahman Maateen, Nan Wang, David Buchaca Prats, AlaeddineAbdessalem, Kelton Zhang, Tobias Jacobowitz, 🙇

    🆕 New Features

    • [06d2f2e7] - hubble: upgrade the local executor if necessary (#2920) (felix-wang)
    • [5082c49e] - daemon: async interaction with children (#2948) (Deepankar Mahapatro)
    • [12fff436] - shipped latency tracking code in scripts (#2933) (Maksudur Rahman Maateen)
    • [4e72338a] - support offline executors (#2935) (AlaeddineAbdessalem)

    🐞 Bug fixes

    • [a7dd9c82] - ci: head commit ahead base commit (#2968) (Wang Bo)
    • [1c6958ba] - fix pydantic typo example (#2965) (Han Xiao)
    • [bab63236] - hubble: logging error message refering to jinahub uri (#2945) (felix-wang)
    • [e0893758] - fix early pea shutdown logic (#2947) (Joan Fontanals)
    • [41a90a8d] - add lz4, uvloop, filelock to daemon (#2946) (Kelton Zhang)
    • [d28b279b] - core: dont respond to cancel and idle (#2940) (Tobias Jacobowitz)
    • [e0e3aa17] - fix arguments passed (#2928) (Joan Fontanals)

    🚧 Code Refactoring

    • [3e1ad70c] - cli: rename override_metas/with to uses_metas/with (#2964) (Han Xiao)
    • [46cf10bf] - small nicer refactor (#2941) (Joan Fontanals)

    📗 Documentation

    • [0c21d687] - cookbook: add mac user hostname attention (#2960) (Zhaofeng Miao)
    • [0e8546a6] - update readme from jina hello examples (#2905) (David Buchaca Prats)
    • [abb27763] - warn against simply run "jinad" (#2932) (Kelton Zhang)

    🏁 Unit Test and CICD

    • [de8e01ef] - show external pod started with exec (#2952) (Joan Fontanals)
    • [d81cd363] - add tests for termination (#2924) (Joan Fontanals)
    • [b9064104] - add git to jinad docker image (#2938) (Deepankar Mahapatro)
    • [36773424] - adding rich to daemon reqs (#2929) (Deepankar Mahapatro)
    • [473be875] - fix fluentd with jinad image (#2931) (Deepankar Mahapatro)

    🍹 Other Improvements

    • [e5b10c38] - contributor: update contributors (Jina Dev Bot)
    • [547d5d4e] - importer: improve styling and error message in importer (#2967) (Han Xiao)
    • [894a4a71] - docs: update TOC (Jina Dev Bot)
    • [a1f74e6b] - chore remove event banner (#2949) (Alex Cureton-Griffiths)
    • [99c542e9] - remove the meetup banner (#2951) (Nan Wang)
    • [fde37372] - version: the next version will be 2.0.8 (Jina Dev Bot)
    Source code(tar.gz)
    Source code(zip)
  • v2.0.7(Jul 12, 2021)

    Release Note (2.0.7)

    Release time: 2021-07-12 14:36:57

    🙇 We'd like to thank all contributors for this new release! In particular, Alex Cureton-Griffiths, cristian, Jina Dev Bot, 🙇

    🆕 New Features

    • [ac16f790] - auto create workspace if not there (#2926) (cristian)

    🍹 Other Improvements

    • [b748b36a] - docs readme event banner (#2925) (Alex Cureton-Griffiths)
    • [7f581ebf] - version: the next version will be 2.0.7 (Jina Dev Bot)
    Source code(tar.gz)
    Source code(zip)
  • v2.0.6(Jul 12, 2021)

    Release Note (2.0.6)

    Release time: 2021-07-12 12:22:08

    🙇 We'd like to thank all contributors for this new release! In particular, Wang Bo, Joan Fontanals, felix-wang, Winston Wong, Jina Dev Bot, Nan Wang, Gikiman, 🙇

    🆕 New Features

    • [f9c50b73] - add gpu support to container runtime (#2914) (Wang Bo)
    • [e74e7b4e] - add debug log about flushed messages (#2908) (Joan Fontanals)
    • [6c5a64c2] - hubble: add filelock while installing executor from hubble (#2866) (felix-wang)

    🐞 Bug fixes

    • [9cf21c32] - remove unused import (#2921) (Joan Fontanals)
    • [57918806] - wait for server to stop in cancel (#2902) (Joan Fontanals)
    • [fc481fb3] - fix #2910 (#2911) (Nan Wang)

    🚧 Code Refactoring

    • [bd34aae6] - small refactors (#2922) (Joan Fontanals)

    📗 Documentation

    • [632315fb] - declare cors in Get Started 's FLow (#2907) (Gikiman)

    🏁 Unit Test and CICD

    • [9917aabc] - timeout control adjusting for ci (#2916) (Joan Fontanals)
    • [4a8abd44] - test flow.load_config with override_with on protocol (#2901) (Winston Wong)

    🍹 Other Improvements

    • [320f14ac] - contributor: update contributors (Jina Dev Bot)
    • [e7e0426d] - remove the latency track (#2915) (Nan Wang)
    • [a1b7fe47] - docs: update TOC (Jina Dev Bot)
    • [d4c6af5b] - version: the next version will be 2.0.6 (Jina Dev Bot)
    Source code(tar.gz)
    Source code(zip)
Owner
Jina AI
A Neural Search Company. We provide the cloud-native neural search solution powered by state-of-the-art AI technology.
Jina AI
An easier way to build neural search on the cloud

An easier way to build neural search on the cloud Jina is a deep learning-powered search framework for building cross-/multi-modal search systems (e.g

Jina AI 11.6k Oct 16, 2021
PyTorch implementation of neural style transfer algorithm

neural-style-pt This is a PyTorch implementation of the paper A Neural Algorithm of Artistic Style by Leon A. Gatys, Alexander S. Ecker, and Matthias

null 594 Oct 23, 2021
A comprehensive list of published machine learning applications to cosmology

ml-in-cosmology This github attempts to maintain a comprehensive list of published machine learning applications to cosmology, organized by subject ma

George Stein 211 Oct 15, 2021
A curated list of neural rendering resources.

Awesome-of-Neural-Rendering A curated list of neural rendering and related resources. Please feel free to pull requests or open an issue to add papers

Zhiwei ZHANG 25 Oct 4, 2021
Official Implementation of Neural Splines

Neural Splines: Fitting 3D Surfaces with Inifinitely-Wide Neural Networks This repository contains the official implementation of the CVPR 2021 (Oral)

Francis Williams 21 Oct 15, 2021
GNN4Traffic - This is the repository for the collection of Graph Neural Network for Traffic Forecasting

GNN4Traffic - This is the repository for the collection of Graph Neural Network for Traffic Forecasting

null 215 Oct 18, 2021
ManiSkill-Learn is a framework for training agents on SAPIEN Open-Source Manipulation Skill Challenge (ManiSkill Challenge), a large-scale learning-from-demonstrations benchmark for object manipulation.

ManiSkill-Learn ManiSkill-Learn is a framework for training agents on SAPIEN Open-Source Manipulation Skill Challenge, a large-scale learning-from-dem

Hao Su's Lab, UCSD 31 Oct 24, 2021
MIM: MIM Installs OpenMMLab Packages

MIM provides a unified API for launching and installing OpenMMLab projects and their extensions, and managing the OpenMMLab model zoo.

OpenMMLab 143 Oct 18, 2021
code for paper "Does Unsupervised Architecture Representation Learning Help Neural Architecture Search?"

Does Unsupervised Architecture Representation Learning Help Neural Architecture Search? Code for paper: Does Unsupervised Architecture Representation

null 32 Sep 23, 2021
A curated list of neural network pruning resources.

A curated list of neural network pruning and related resources. Inspired by awesome-deep-vision, awesome-adversarial-machine-learning, awesome-deep-learning-papers and Awesome-NAS.

Yang He 1.3k Oct 15, 2021
An implementation for Neural Architecture Search with Random Labels (CVPR 2021 poster) on Pytorch.

Neural Architecture Search with Random Labels(RLNAS) Introduction This project provides an implementation for Neural Architecture Search with Random L

null 15 Aug 26, 2021
PyTorch implementation of DeepDream algorithm

neural-dream This is a PyTorch implementation of DeepDream. The code is based on neural-style-pt. Here we DeepDream a photograph of the Golden Gate Br

null 96 Oct 15, 2021
Vector AI — A platform for building vector based applications. Encode, query and analyse data using vectors.

Vector AI is a framework designed to make the process of building production grade vector based applications as quickly and easily as possible. Create

Vector AI 231 Oct 21, 2021
All course materials for the Zero to Mastery Deep Learning with TensorFlow course.

All course materials for the Zero to Mastery Deep Learning with TensorFlow course.

Daniel Bourke 1.7k Oct 24, 2021
Densely Connected Search Space for More Flexible Neural Architecture Search (CVPR2020)

DenseNAS The code of the CVPR2020 paper Densely Connected Search Space for More Flexible Neural Architecture Search. Neural architecture search (NAS)

Jamin Fong 281 Oct 21, 2021
Vision Transformer and MLP-Mixer Architectures

Vision Transformer and MLP-Mixer Architectures Update (2.7.2021): Added the "When Vision Transformers Outperform ResNets..." paper, and SAM (Sharpness

Google Research 3.7k Oct 24, 2021
Tensors and neural networks in Haskell

Hasktorch Hasktorch is a library for tensors and neural networks in Haskell. It is an independent open source community project which leverages the co

hasktorch 816 Oct 15, 2021
Deep Learning for 3D Point Clouds: A Survey (IEEE TPAMI, 2020)

??Deep Learning for 3D Point Clouds (IEEE TPAMI, 2020)

Qingyong 1.1k Oct 15, 2021
Must-read Papers on Physics-Informed Neural Networks.

PINNpapers Contributed by IDRL lab. Introduction Physics-Informed Neural Network (PINN) has achieved great success in scientific computing since 2017.

IDRL 30 Oct 15, 2021