Import, visualize, and analyze SpiderFoot OSINT data in Neo4j, a graph database

Overview

SpiderFoot Neo4j Tools

Import, visualize, and analyze SpiderFoot OSINT data in Neo4j, a graph database

A big graph

Step 1: Installation

NOTE: This installs the sfgraph command-line utility

$ pip install spiderfoot-neo4j

Step 2: Start Neo4j

NOTE: Docker must first be installed

$ docker run --rm --name sfgraph -v "$(pwd)/neo4j_database:/data" -e 'NEO4J_AUTH=neo4j/CHANGETHISIFYOURENOTZUCK' -e 'NEO4JLABS_PLUGINS=["apoc", "graph-data-science"]' -e 'NEO4J_dbms_security_procedures_unrestricted=apoc.*,gds.*' -p "7474:7474" -p "7687:7687" neo4j

Step 3: Import Scans

Spiderfoot scan ID in web browser

$ sfgraph path_to/spiderfoot.db -s   ...

Step 4: Browse Spiderfoot Data in Neo4j

Visit http://127.0.0.1:7474 and log in with neo4j/CHANGETHISIFYOURENOTZUCK Spiderfoot data in Neo4j

Step 5 (Optional): Use cool algorithms to find new targets

The --suggest option will rank nodes based on their connectedness in the graph. This is perfect for finding closely-related affiliates (child companies, etc.) to scan and add to the graph. By default, Harmonic Centrality is used, but others such as PageRank can be specified with --closeness-algorithm

$ sfgraph --suggest DOMAIN_NAME

Closeness scores

Example CYPHER Queries

() RETURN p # shortest path to all INTERNET_NAMEs from seed domain MATCH p=shortestPath((d:DOMAIN_NAME {data:"evilcorp.com"})-[*]-(n:INTERNET_NAME)) RETURN p # match only primary targets (non-affiliates) MATCH (n {scanned: true}) return n # match only affiliates MATCH (n {affiliate: true}) return n ">
# match all INTERNET_NAMEs
MATCH (n:INTERNET_NAME) RETURN n

# match multiple event types
MATCH (n) WHERE n:INTERNET_NAME OR n:DOMAIN_NAME OR n:EMAILADDR RETURN n

# match by attribute
MATCH (n {data: "evilcorp.com"}) RETURN n

# match by spiderfoot module (relationship)
MATCH p=()-[r:WHOIS]->() RETURN p

# shortest path to all INTERNET_NAMEs from seed domain
MATCH p=shortestPath((d:DOMAIN_NAME {data:"evilcorp.com"})-[*]-(n:INTERNET_NAME)) RETURN p

# match only primary targets (non-affiliates)
MATCH (n {scanned: true}) return n

# match only affiliates
MATCH (n {affiliate: true}) return n

CLI Help

sfgraph [-h] [-db SQLITEDB] [-s SCANS [SCANS ...]] [--uri URI] [-u USERNAME] [-p PASSWORD] [--clear] [--suggest SUGGEST]
               [--closeness-algorithm {pageRank,articleRank,closenessCentrality,harmonicCentrality,betweennessCentrality,eigenvectorCentrality}] [-v]

optional arguments:
  -h, --help            show this help message and exit
  -db SQLITEDB, --sqlitedb SQLITEDB
                        Spiderfoot sqlite database
  -s SCANS [SCANS ...], --scans SCANS [SCANS ...]
                        scan IDs to import
  --uri URI             Neo4j database URI (default: bolt://127.0.0.1:7687)
  -u USERNAME, --username USERNAME
                        Neo4j username (default: neo4j)
  -p PASSWORD, --password PASSWORD
                        Neo4j password
  --clear               Wipe the Neo4j database
  --suggest SUGGEST     Suggest targets of this type (e.g. DOMAIN_NAME) based on their connectedness in the graph
  --closeness-algorithm {pageRank,articleRank,closenessCentrality,harmonicCentrality,betweennessCentrality,eigenvectorCentrality}
                        Algorithm to use when suggesting targets
  -v, -d, --debug       Verbose / debug
You might also like...
Python package to Create, Read, Write, Edit, and Visualize GSFLOW models
Python package to Create, Read, Write, Edit, and Visualize GSFLOW models

pygsflow pyGSFLOW is a python package to Create, Read, Write, Edit, and Visualize GSFLOW models API Documentation pyGSFLOW API documentation can be fo

A small collection of tools made by me, that you can use to visualize atomic orbitals in both 2D and 3D in different aspects.
A small collection of tools made by me, that you can use to visualize atomic orbitals in both 2D and 3D in different aspects.

Orbitals in Python A small collection of tools made by me, that you can use to visualize atomic orbitals in both 2D and 3D in different aspects, and o

 Epagneul is a tool to visualize and investigate windows event logs
Epagneul is a tool to visualize and investigate windows event logs

epagneul Epagneul is a tool to visualize and investigate windows event logs. Dep

This project is an Algorithm Visualizer where a user can visualize algorithms like Bubble Sort, Merge Sort, Quick Sort, Selection Sort, Linear Search and Binary Search.
This project is an Algorithm Visualizer where a user can visualize algorithms like Bubble Sort, Merge Sort, Quick Sort, Selection Sort, Linear Search and Binary Search.

Algo_Visualizer This project is an Algorithm Visualizer where a user can visualize common algorithms like "Bubble Sort", "Merge Sort", "Quick Sort", "

WebApp served by OAK PoE device to visualize various streams, metadata and AI results
WebApp served by OAK PoE device to visualize various streams, metadata and AI results

DepthAI PoE WebApp | Bootstrap 4 & Vue.js SPA Dashboard Based on dashmin (https:

Domain Connectivity Analysis Tools to analyze aggregate connectivity patterns across a set of domains during security investigations
Domain Connectivity Analysis Tools to analyze aggregate connectivity patterns across a set of domains during security investigations

DomainCAT (Domain Connectivity Analysis Tool) Domain Connectivity Analysis Tool is used to analyze aggregate connectivity patterns across a set of dom

Automatically Visualize any dataset, any size with a single line of code.  Created by Ram Seshadri. Collaborators Welcome. Permission Granted upon Request.
Automatically Visualize any dataset, any size with a single line of code. Created by Ram Seshadri. Collaborators Welcome. Permission Granted upon Request.

AutoViz Automatically Visualize any dataset, any size with a single line of code. AutoViz performs automatic visualization of any dataset with one lin

Automatically Visualize any dataset, any size with a single line of code.  Created by Ram Seshadri. Collaborators Welcome. Permission Granted upon Request.
Automatically Visualize any dataset, any size with a single line of code. Created by Ram Seshadri. Collaborators Welcome. Permission Granted upon Request.

AutoViz Automatically Visualize any dataset, any size with a single line of code. AutoViz performs automatic visualization of any dataset with one lin

Visualize tensors in a plain Python REPL using Sparklines
Visualize tensors in a plain Python REPL using Sparklines

Visualize tensors in a plain Python REPL using Sparklines

Comments
  • Compatable with spdierfoot v4.0.0?

    Compatable with spdierfoot v4.0.0?

    Can't seem to find a spiderfoot.db to import on either my linux install or on a windows 10 install of spiderfoot. Could it be because of spiderfoot 4.0.0 being unsupported? Thanks

    opened by onnionn 0
Owner
Black Lantern Security
Security Organization
Black Lantern Security
In-memory Graph Database and Knowledge Graph with Natural Language Interface, compatible with Pandas

CogniPy for Pandas - In-memory Graph Database and Knowledge Graph with Natural Language Interface Whats in the box Reasoning, exploration of RDF/OWL,

Cognitum Octopus 34 Dec 13, 2022
Extract data from ThousandEyes REST API and visualize it on your customized Grafana Dashboard.

ThousandEyes Grafana Dashboard Extract data from the ThousandEyes REST API and visualize it on your customized Grafana Dashboard. Deploy Grafana, Infl

Flo Pachinger 16 Nov 26, 2022
Visualize your pandas data with one-line code

PandasEcharts 简介 基于pandas和pyecharts的可视化工具 安装 pip 安装 $ pip install pandasecharts 源码安装 $ git clone https://github.com/gamersover/pandasecharts $ cd pand

陈华杰 2 Apr 13, 2022
Visualize data of Vietnam's regions with interactive maps.

Plotting Vietnam Development Map This is my personal project that I use plotly to analyse and visualize data of Vietnam's regions with interactive map

null 1 Jun 26, 2022
Small binja plugin to import header file to types

binja-import-header (v1.0.0) Author: matteyeux Import header file to Binary Ninja types view Description: Binary Ninja plugin to import types from C h

matteyeux 15 Dec 10, 2022
Visualize and compare datasets, target values and associations, with one line of code.

In-depth EDA (target analysis, comparison, feature analysis, correlation) in two lines of code! Sweetviz is an open-source Python library that generat

Francois Bertrand 2.3k Jan 5, 2023
Visualize and compare datasets, target values and associations, with one line of code.

In-depth EDA (target analysis, comparison, feature analysis, correlation) in two lines of code! Sweetviz is an open-source Python library that generat

Francois Bertrand 1.2k Feb 18, 2021
Generate graphs with NetworkX, natively visualize with D3.js and pywebview

webview_d3 This is some PoC code to render graphs created with NetworkX natively using D3.js and pywebview. The main benifit of this approac

byt3bl33d3r 68 Aug 18, 2022
Extract and visualize information from Gurobi log files

GRBlogtools Extract information from Gurobi log files and generate pandas DataFrames or Excel worksheets for further processing. Also includes a wrapp

Gurobi Optimization 56 Nov 17, 2022
This is a web application to visualize various famous technical indicators and stocks tickers from user

Visualizing Technical Indicators Using Python and Plotly. Currently facing issues hosting the application on heroku. As soon as I am able to I'll like

null 4 Aug 4, 2022