Cải thiện Elasticsearch trong bài toán semantic search sử dụng phương pháp Sentence Embeddings

Vo Van Phuc

Last update: Nov 25, 2022

Related tags

Text Data & NLP elastic_simCSE

Overview

Cải thiện Elasticsearch trong bài toán semantic search sử dụng phương pháp Sentence Embeddings

Trong bài viết này mình sẽ sử dụng pretrain model SimCSE_Vietnamese để cải thiện elastic search trong bài toán semantic search.

Mọi người có thể xem bài viết hướng dẫn đầy đủ tại đây.

Cài đặt:

git clone https://github.com/vovanphuc/elastic_simCSE.git
cd elastic_simCSE
pip install -r requirements.txt

Đánh index cho toàn bộ data

python3 index_elastic.py

Search keyword và so sánh giữa BM25 (elasticsearch thường) và simCSE:

streamlit run main.py

Kết quả tìm kiếm:

Kết quả khi sử dụng elasticsearch bình thường.

Kết quả khi sử dụng SimCSE_VietNamese.

Contact:

Email: [email protected]

Facebook: facebook.com/vovanphucc

Linkedin: linkedin.com/in/vovanphuc

You might also like...

:mag: End-to-End Framework for building natural language search interfaces to data by utilizing Transformers and the State-of-the-Art of NLP. Supporting DPR, Elasticsearch, HuggingFace’s Modelhub and much more!

Haystack is an end-to-end framework that enables you to build powerful and production-ready pipelines for different search use cases. Whether you want

1.4k Feb 18, 2021

Connectionist Temporal Classification (CTC) decoding algorithms: best path, beam search, lexicon search, prefix search, and token passing. Implemented in Python.

CTC Decoding Algorithms Update 2021: installable Python package Python implementation of some common Connectionist Temporal Classification (CTC) decod

736 Jan 3, 2023

The following links explain a bit the idea of semantic search and how search mechanisms work by doing retrieve and rerank

Main Idea The following links explain a bit the idea of semantic search and how search mechanisms work by doing retrieve and rerank Semantic Search Re

2 Jan 28, 2022

Top2Vec is an algorithm for topic modeling and semantic search.

Top2Vec is an algorithm for topic modeling and semantic search. It automatically detects topics present in text and generates jointly embedded topic, document and word vectors.

2.4k Jan 6, 2023

Blue Brain text mining toolbox for semantic search and structured information extraction

Blue Brain Search Source Code DOI Data & Models DOI Documentation Latest Release Python Versions License Build Status Static Typing Code Style Securit

29 Dec 1, 2022

txtai: Build AI-powered semantic search applications in Go

txtai: Build AI-powered semantic search applications in Go txtai executes machine-learning workflows to transform data and build AI-powered semantic s

49 Dec 6, 2022

Create a semantic search engine with a neural network (i.e. BERT) whose knowledge base can be updated

Create a semantic search engine with a neural network (i.e. BERT) whose knowledge base can be updated. This engine can later be used for downstream tasks in NLP such as Q&A, summarization, generation, and natural language understanding (NLU).

1 Mar 20, 2022

A sentence aligner for comparable corpora

About Yalign is a tool for extracting parallel sentences from comparable corpora. Statistical Machine Translation relies on parallel corpora (eg.. eur

128 Aug 24, 2022

Extract Keywords from sentence or Replace keywords in sentences.

FlashText This module can be used to replace keywords in sentences or extract keywords from sentences. It is based on the FlashText algorithm. Install

5.3k Jan 1, 2023

Owner

Vo Van Phuc

KEEP IT SIMPLE STUPID!

GitHub

:mag: Transformers at scale for question answering & neural search. Using NLP via a modular Retriever-Reader-Pipeline. Supporting DPR, Elasticsearch, HuggingFace's Modelhub...

Haystack is an end-to-end framework for Question Answering & Neural search that enables you to ... ... ask questions in natural language and find gran

6.4k Jan 9, 2023

Cải thiện Elasticsearch trong bài toán semantic search sử dụng phương pháp Sentence Embeddings

Related tags

Overview

Cải thiện Elasticsearch trong bài toán semantic search sử dụng phương pháp Sentence Embeddings

Cài đặt:

Đánh index cho toàn bộ data

Search keyword và so sánh giữa BM25 (elasticsearch thường) và simCSE:

Kết quả tìm kiếm:

Contact:

You might also like...

:mag: End-to-End Framework for building natural language search interfaces to data by utilizing Transformers and the State-of-the-Art of NLP. Supporting DPR, Elasticsearch, HuggingFace’s Modelhub and much more!

Connectionist Temporal Classification (CTC) decoding algorithms: best path, beam search, lexicon search, prefix search, and token passing. Implemented in Python.

The following links explain a bit the idea of semantic search and how search mechanisms work by doing retrieve and rerank

Top2Vec is an algorithm for topic modeling and semantic search.

Blue Brain text mining toolbox for semantic search and structured information extraction

txtai: Build AI-powered semantic search applications in Go

Create a semantic search engine with a neural network (i.e. BERT) whose knowledge base can be updated

A sentence aligner for comparable corpora

Extract Keywords from sentence or Replace keywords in sentences.

Owner

Vo Van Phuc

🐍💯pySBD (Python Sentence Boundary Disambiguation) is a rule-based sentence boundary detection that works out-of-the-box.

REST API for sentence tokenization and embedding using Multilingual Universal Sentence Encoder.

Sentence Embeddings with BERT & XLNet

Sentence Embeddings with BERT & XLNet

SimCSE: Simple Contrastive Learning of Sentence Embeddings

InferSent sentence embeddings

Shared code for training sentence embeddings with Flax / JAX

Korean Simple Contrastive Learning of Sentence Embeddings using SKT KoBERT and kakaobrain KorNLU dataset

NAACL 2022: MCSE: Multimodal Contrastive Learning of Sentence Embeddings

:mag: Transformers at scale for question answering & neural search. Using NLP via a modular Retriever-Reader-Pipeline. Supporting DPR, Elasticsearch, HuggingFace's Modelhub...