Convert a collection of features to a fixed-dimensional matrix using the hashing trick.

Jina AI

Last update: Mar 15, 2022

Related tags

Security related resources executor-featurehasher

Overview

FeatureHasher

Convert a collection of features to a fixed-dimensional matrix using the hashing trick.

Note, this requires Jina>=2.2.4.

Example

Here I use FeatureHasher to hash each sentence of Pride and Prejudice into a 128-dim vector, and then use .match to find top-K similar sentences.

from jina import Document, DocumentArray, Flow

# load 
   
d = Document(uri='https://www.gutenberg.org/files/1342/1342-0.txt').convert_uri_to_text()

# cut into non-empty sentences store in a DA
da = DocumentArray(Document(text=s.strip()) for s in d.text.split('\n') if s.strip())

# use FeatureHasher in a Flow
f = Flow().add(uses='jinahub://FeatureHasher')

embed_da = DocumentArray()
with f:
    f.post('/', da, on_done=lambda req: embed_da.extend(req.docs), show_progress=True)

print('self-matching...')
embed_da.match(embed_da, exclude_self=True, limit=5, normalization=(1, 0))
print('total sentences: ', len(embed_da))
for d in embed_da:
    print(d.text)
    for m in d.matches:
        print(m.scores['cosine'], m.text)
    input()

           Flow@17400[I]:🎉 Flow is ready to use!
	🔗 Protocol: 		GRPC
	🏠 Local access:	0.0.0.0:52628
	🔒 Private network:	192.168.178.31:52628
	🌐 Public address:	217.70.138.123:52628
⠹       DONE ━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0:00:01 100% ETA: 0 seconds 40 steps done in 1 second
total sentences:  12153
The Project Gutenberg eBook of Pride and Prejudice, by Jane Austen

   
     *** END OF THE PROJECT GUTENBERG EBOOK PRIDE AND PREJUDICE ***

    
      *** START OF THE PROJECT GUTENBERG EBOOK PRIDE AND PREJUDICE ***

     
       production, promotion and distribution of Project Gutenberg-tm

      
        Pride and Prejudice

       
         By Jane Austen This eBook is for the use of anyone anywhere in the United States and 
        
          This eBook is for the use of anyone anywhere in the United States and 
         
           by the awkwardness of the application, and at length wholly 
          
            Elizabeth passed the chief of the night in her sister’s room, and 
           
             the happiest memories in the world. Nothing of the past was 
            
              charities and charitable donations in all 50 states of the United

In practice, you can implement matching and storing via an indexer inside Flow. This example is only for demo purpose so any non-feature hashing related ops are implemented outside the Flow to avoid distraction.

You might also like...

A collection of write-ups and solutions for Cyber FastTrack Spring 2021.

IMPORTANT: Please contact us before you use any styling or content shown here! Cyber FastTrack Spring 2021 / National Cyber Scholarship Competition -

48 Aug 28, 2022

Kunyu, more efficient corporate asset collection

Kunyu(坤舆) - More efficient corporate asset collection English | 中文文档 0x00 Introduce Tool introduction Kunyu (kunyu), whose name is taken from , is act

772 Jan 5, 2023

WinRemoteEnum is a module-based collection of operations achievable by a low-privileged domain user.

WinRemoteEnum WinRemoteEnum is a module-based collection of operations achievable by a low-privileged domain user, sharing the goal of remotely gather

9 Nov 9, 2022

Vulnerability Exploitation Code Collection Repository

Introduction expbox is an exploit code collection repository List CVE-2021-41349 Exchange XSS PoC = Exchange 2013 update 23 = Exchange 2016 update 2

263 Feb 14, 2022

This collection of tools that makes it easy to secure and/or obfuscate messages, files, and data.

Scrambler App This collection of tools that makes it easy to secure and/or obfuscate messages, files, and data. It leverages encryption tools such as

2 Aug 31, 2022

A collection of intelligence about Log4Shell and its exploitation activity

Log4Shell-IOCs Members of the Curated Intelligence Trust Group have compiled a list of IOC feeds and threat reports focused on the recent Log4Shell ex

172 Nov 17, 2022

HatSploit collection of generic payloads designed to provide a wide range of attacks without having to spend time writing new ones.

5 May 10, 2022

Osint-Tool - Information collection tool in python

Osint-Tool Herramienta para la recolección de información Pronto más opciones In

3 Apr 9, 2022

Create a secure tunnel from a custom domain to localhost using Fly and WireGuard.

Fly Dev Tunnel Developers commonly use apps like ngrok, localtunnel, or cloudflared to expose a local web service at a publicly-accessible URL. This i

170 Dec 11, 2022

Convert a collection of features to a fixed-dimensional matrix using the hashing trick.

Related tags

Overview

FeatureHasher

Example

You might also like...

A collection of write-ups and solutions for Cyber FastTrack Spring 2021.

Kunyu, more efficient corporate asset collection

WinRemoteEnum is a module-based collection of operations achievable by a low-privileged domain user.

Vulnerability Exploitation Code Collection Repository

This collection of tools that makes it easy to secure and/or obfuscate messages, files, and data.

A collection of intelligence about Log4Shell and its exploitation activity

HatSploit collection of generic payloads designed to provide a wide range of attacks without having to spend time writing new ones.

Osint-Tool - Information collection tool in python

Create a secure tunnel from a custom domain to localhost using Fly and WireGuard.

Owner

Jina AI

A tool used to obfuscate python scripts, bind obfuscated scripts to fixed machine or expire obfuscated scripts.

Collection Of Discord Hacking Tools / Fun Stuff / Exploits That Is Completely Made Using Python.

Anti-Nuke capabilities, powerful moderation features, auto punishments, captcha-verification and more.

A windows post exploitation tool that contains a lot of features for information gathering and more.

Safe Policy Optimization with Local Features

An experimental script to perform bulk parsing of arbitrary file features with YARA and console logging.

A collection of over 5.1 million sub-domains and assets belonging to public bug bounty programs, compiled into a repo, for performing bulk operations.

SpiderFoot automates OSINT collection so that you can focus on analysis.

A forensic collection tool written in Python.