SpiderFoot automates OSINT collection so that you can focus on analysis.

Debian - Parrot Security OS (latest version) Dell laptop 64bit

python3 sf.py Traceback (most recent call last): File "sf.py", line 21, in import cherrypy ModuleNotFoundError: No module named 'cherrypy'

I just pulled spiderfoot tried running it but ERROR

How can I can run it? I tried running it root home and su home etc.

also cd /root/.spiderfoot/spiderfoot.db is not found too bash: cd: /root/.spiderfoot: No such file or directory

ERROR: spiderfoot.db file exists in .
SpiderFoot no longer supports loading the spiderfoot.db database from the application directory.
The database is now loaded from your home directory: /root/.spiderfoot/spiderfoot.db
This message will go away once you move or remove spiderfoot.db from .

Please provide a description of the issue and any relevant error messages.

If you can provide an application stack trace that is even better.

is there any tutorial for this one? What version of Python are you using? Python 3.8.6 What version of SpiderFoot are you using (stable release or Git master branch)? Git master branch You may also wish to check if your issue has been posted previously:

• https://github.com/smicallef/spiderfoot/issues

Please provide a description of the issue and any relevant error messages. When python3 ./sf.py -l 127.0.0.1:5001 is executed , i get the below error.

I have installed gcd and also added PATH to .profile

Stack Trace

Traceback (most recent call last): File "/home/user1/Desktop/user2/OSINT/spiderfoot-3.2.1/./sf.py", line 30, in from sflib import SpiderFoot File "/home/user1/Desktop/user2/OSINT/spiderfoot-3.2.1/sflib.py", line 40, in from networkx import nx File "/home/user1/.local/lib/python3.9/site-packages/networkx/init.py", line 114, in import networkx.generators File "/home/user1/.local/lib/python3.9/site-packages/networkx/generators/init.py", line 14, in from networkx.generators.intersection import * File "/home/user1/.local/lib/python3.9/site-packages/networkx/generators/intersection.py", line 13, in from networkx.algorithms import bipartite File "/home/user1/.local/lib/python3.9/site-packages/networkx/algorithms/init.py", line 16, in from networkx.algorithms.dag import * File "/home/user1/.local/lib/python3.9/site-packages/networkx/algorithms/dag.py", line 23, in from fractions import gcd ImportError: cannot import name 'gcd' from 'fractions' (/usr/lib/python3.9/fractions.py)

What version of Python are you using?

Python 3.9

What version of SpiderFoot are you using (stable release or Git master branch)?

Stable

While scanning, I received the following error message:

Unhandled exception (error) encountered during scan. Please report this as a bug: Traceback (most recent call last): File "/home/spiderfoot/sfscan.py", line 265, in startScan psMod.notifyListeners(firstEvent) File "/home/spiderfoot/sflib.py", line 1527, in notifyListeners listener.handleEvent(sfEvent) File "/home/spiderfoot/modules/sfp_dnsresolve.py", line 222, in handleEvent self.processHost(addr, parentEvent, False) File "/home/spiderfoot/modules/sfp_dnsresolve.py", line 338, in processHost self.processDomain(dom, evt) File "/home/spiderfoot/modules/sfp_dnsresolve.py", line 357, in processDomain self.notifyListeners(domevt) File "/home/spiderfoot/sflib.py", line 1527, in notifyListeners listener.handleEvent(sfEvent) File "/home/spiderfoot/modules/sfp_dnsbrute.py", line 131, in handleEvent self.notifyListeners(evt) File "/home/spiderfoot/sflib.py", line 1527, in notifyListeners listener.handleEvent(sfEvent) File "/home/spiderfoot/modules/sfp_dnsresolve.py", line 222, in handleEvent self.processHost(addr, parentEvent, False) File "/home/spiderfoot/modules/sfp_dnsresolve.py", line 334, in processHost self.notifyListeners(evt) File "/home/spiderfoot/sflib.py", line 1527, in notifyListeners listener.handleEvent(sfEvent) File "/home/spiderfoot/modules/sfp_robtex.py", line 167, in handleEvent self.notifyListeners(evt) File "/home/spiderfoot/sflib.py", line 1527, in notifyListeners listener.handleEvent(sfEvent) File "/home/spiderfoot/modules/sfp_badipscom.py", line 246, in handleEvent url = self.lookupItem(cid, typeId, eventData) File "/home/spiderfoot/modules/sfp_badipscom.py", line 197, in lookupItem return self.resourceList(cid, target, itemType) File "/home/spiderfoot/modules/sfp_badipscom.py", line 183, in resourceList re.match(rxTgt, line, re.IGNORECASE): File "/usr/lib/python2.7/re.py", line 141, in match return _compile(pattern, flags).match(string) File "/usr/lib/python2.7/re.py", line 251, in _compile raise error, v # invalid expression error: nothing to repeat

What could be the cause of this error message?

Unhandled exception (KeyError) encountered during scan. Please report this as a bug: ['Traceback (most recent call last):\n', ' File "/root/Tools/spiderfoot-2.12/sfscan.py", line 265, in startScan\n psMod.notifyListeners(firstEvent)\n', ' File "/root/Tools/spiderfoot-2.12/sflib.py", line 1527, in notifyListeners\n listener.handleEvent(sfEvent)\n', ' File "/root/Tools/spiderfoot-2.12/modules/sfp_pastebin.py", line 123, in handleEvent\n self.notifyListeners(evt2)\n', ' File "/root/Tools/spiderfoot-2.12/sflib.py", line 1527, in notifyListeners\n listener.handleEvent(sfEvent)\n', ' File "/root/Tools/spiderfoot-2.12/modules/sfp_email.py", line 98, in handleEvent\n self.notifyListeners(evt)\n', ' File "/root/Tools/spiderfoot-2.12/sflib.py", line 1527, in notifyListeners\n listener.handleEvent(sfEvent)\n', ' File "/root/Tools/spiderfoot-2.12/modules/sfp_fullcontact.py", line 151, in handleEvent\n e = SpiderFootEvent("HUMAN_NAME", data['fullName'], self.name, event)\n', "KeyError: 'fullName'\n"]

stack trace: Unhandled exception (MemoryError) encountered during scan. Please report this as a bug: ['Traceback (most recent call last):\n', ' File "/home/gianz/spiderfoot/sfscan.py", line 195, in startScan\n module.start()\n', ' File "/home/gianz/spiderfoot/modules/sfp_bingsearch.py", line 81, in start\n self.notifyListeners(evt)\n', ' File "/home/gianz/spiderfoot/sflib.py", line 1050, in notifyListeners\n listener.handleEvent(sfEvent)\n', ' File "/home/gianz/spiderfoot/modules/sfp_dns.py", line 120, in handleEvent\n self.processHost(match, parentEvent)\n', ' File "/home/gianz/spiderfoot/modules/sfp_dns.py", line 271, in processHost\n self.notifyListeners(evt)\n', ' File "/home/gianz/spiderfoot/sflib.py", line 1050, in notifyListeners\n listener.handleEvent(sfEvent)\n', ' File "/home/gianz/spiderfoot/modules/sfp_spider.py", line 250, in handleEvent\n return self.spiderFrom(spiderTarget)\n', ' File "/home/gianz/spiderfoot/modules/sfp_spider.py", line 274, in spiderFrom\n links = self.processUrl(startingPoint) # fetch first page\n', ' File "/home/gianz/spiderfoot/modules/sfp_spider.py", line 105, in processUrl\n self.contentNotify(url, fetched, self.urlEvents[url])\n', ' File "/home/gianz/spiderfoot/modules/sfp_spider.py", line 189, in contentNotify\n self.notifyListeners(event)\n', ' File "/home/gianz/spiderfoot/sflib.py", line 1050, in notifyListeners\n listener.handleEvent(sfEvent)\n', ' File "/home/gianz/spiderfoot/modules/sfp_dns.py", line 120, in handleEvent\n self.processHost(match, parentEvent)\n', ' File "/home/gianz/spiderfoot/modules/sfp_dns.py", line 271, in processHost\n self.notifyListeners(evt)\n', ' File "/home/gianz/spiderfoot/sflib.py", line 1050, in notifyListeners\n listener.handleEvent(sfEvent)\n', ' File "/home/gianz/spiderfoot/modules/sfp_spider.py", line 250, in handleEvent\n return self.spiderFrom(spiderTarget)\n', ' File "/home/gianz/spiderfoot/modules/sfp_spider.py", line 300, in spiderFrom\n freshLinks = self.processUrl(link)\n', ' File "/home/gianz/spiderfoot/modules/sfp_spider.py", line 128, in processUrl\n self.urlEvents[link] = self.linkNotify(link, self.urlEvents[url])\n', ' File "/home/gianz/spiderfoot/modules/sfp_spider.py", line 181, in linkNotify\n self.notifyListeners(event)\n', ' File "/home/gianz/spiderfoot/sflib.py", line 1050, in notifyListeners\n listener.handleEvent(sfEvent)\n', ' File "/home/gianz/spiderfoot/modules/sfp_dns.py", line 120, in handleEvent\n self.processHost(match, parentEvent)\n', ' File "/home/gianz/spiderfoot/modules/sfp_dns.py", line 271, in processHost\n self.notifyListeners(evt)\n', ' File "/home/gianz/spiderfoot/sflib.py", line 1050, in notifyListeners\n listener.handleEvent(sfEvent)\n', ' File "/home/gianz/spiderfoot/modules/sfp_spider.py", line 250, in handleEvent\n return self.spiderFrom(spiderTarget)\n', ' File "/home/gianz/spiderfoot/modules/sfp_spider.py", line 274, in spiderFrom\n links = self.processUrl(startingPoint) # fetch first page\n', ' File "/home/gianz/spiderfoot/modules/sfp_spider.py", line 105, in processUrl\n self.contentNotify(url, fetched, self.urlEvents[url])\n', ' File "/home/gianz/spiderfoot/modules/sfp_spider.py", line 189, in contentNotify\n self.notifyListeners(event)\n', ' File "/home/gianz/spiderfoot/sflib.py", line 1050, in notifyListeners\n listener.handleEvent(sfEvent)\n', ' File "/home/gianz/spiderfoot/modules/sfp_dns.py", line 120, in handleEvent\n self.processHost(match, parentEvent)\n', ' File "/home/gianz/spiderfoot/modules/sfp_dns.py", line 271, in processHost\n self.notifyListeners(evt)\n', ' File "/home/gianz/spiderfoot/sflib.py", line 1050, in notifyListeners\n listener.handleEvent(sfEvent)\n', ' File "/home/gianz/spiderfoot/modules/sfp_spider.py", line 250, in handleEvent\n return self.spiderFrom(spiderTarget)\n', ' File "/home/gianz/spiderfoot/modules/sfp_spider.py", line 300, in spiderFrom\n freshLinks = self.processUrl(link)\n', ' File "/home/gianz/spiderfoot/modules/sfp_spider.py", line 105, in processUrl\n self.contentNotify(url, fetched, self.urlEvents[url])\n', ' File "/home/gianz/spiderfoot/modules/sfp_spider.py", line 189, in contentNotify\n self.notifyListeners(event)\n', ' File "/home/gianz/spiderfoot/sflib.py", line 1050, in notifyListeners\n listener.handleEvent(sfEvent)\n', ' File "/home/gianz/spiderfoot/modules/sfp_dns.py", line 120, in handleEvent\n self.processHost(match, parentEvent)\n', ' File "/home/gianz/spiderfoot/modules/sfp_dns.py", line 271, in processHost\n self.notifyListeners(evt)\n', ' File "/home/gianz/spiderfoot/sflib.py", line 1050, in notifyListeners\n listener.handleEvent(sfEvent)\n', ' File "/home/gianz/spiderfoot/modules/sfp_spider.py", line 250, in handleEvent\n return self.spiderFrom(spiderTarget)\n', ' File "/home/gianz/spiderfoot/modules/sfp_spider.py", line 300, in spiderFrom\n freshLinks = self.processUrl(link)\n', ' File "/home/gianz/spiderfoot/modules/sfp_spider.py", line 105, in processUrl\n self.contentNotify(url, fetched, self.urlEvents[url])\n', ' File "/home/gianz/spiderfoot/modules/sfp_spider.py", line 189, in contentNotify\n self.notifyListeners(event)\n', ' File "/home/gianz/spiderfoot/sflib.py", line 1050, in notifyListeners\n listener.handleEvent(sfEvent)\n', ' File "/home/gianz/spiderfoot/modules/sfp_dns.py", line 120, in handleEvent\n self.processHost(match, parentEvent)\n', ' File "/home/gianz/spiderfoot/modules/sfp_dns.py", line 271, in processHost\n self.notifyListeners(evt)\n', ' File "/home/gianz/spiderfoot/sflib.py", line 1050, in notifyListeners\n listener.handleEvent(sfEvent)\n', ' File "/home/gianz/spiderfoot/modules/sfp_spider.py", line 250, in handleEvent\n return self.spiderFrom(spiderTarget)\n', ' File "/home/gianz/spiderfoot/modules/sfp_spider.py", line 300, in spiderFrom\n freshLinks = self.processUrl(link)\n', ' File "/home/gianz/spiderfoot/modules/sfp_spider.py", line 105, in processUrl\n self.contentNotify(url, fetched, self.urlEvents[url])\n', ' File "/home/gianz/spiderfoot/modules/sfp_spider.py", line 189, in contentNotify\n self.notifyListeners(event)\n', ' File "/home/gianz/spiderfoot/sflib.py", line 1050, in notifyListeners\n listener.handleEvent(sfEvent)\n', ' File "/home/gianz/spiderfoot/modules/sfp_dns.py", line 120, in handleEvent\n self.processHost(match, parentEvent)\n', ' File "/home/gianz/spiderfoot/modules/sfp_dns.py", line 271, in processHost\n self.notifyListeners(evt)\n', ' File "/home/gianz/spiderfoot/sflib.py", line 1050, in notifyListeners\n listener.handleEvent(sfEvent)\n', ' File "/home/gianz/spiderfoot/modules/sfp_spider.py", line 250, in handleEvent\n return self.spiderFrom(spiderTarget)\n', ' File "/home/gianz/spiderfoot/modules/sfp_spider.py", line 274, in spiderFrom\n links = self.processUrl(startingPoint) # fetch first page\n', ' File "/home/gianz/spiderfoot/modules/sfp_spider.py", line 105, in processUrl\n self.contentNotify(url, fetched, self.urlEvents[url])\n', ' File "/home/gianz/spiderfoot/modules/sfp_spider.py", line 189, in contentNotify\n self.notifyListeners(event)\n', ' File "/home/gianz/spiderfoot/sflib.py", line 1050, in notifyListeners\n listener.handleEvent(sfEvent)\n', ' File "/home/gianz/spiderfoot/modules/sfp_dns.py", line 120, in handleEvent\n self.processHost(match, parentEvent)\n', ' File "/home/gianz/spiderfoot/modules/sfp_dns.py", line 271, in processHost\n self.notifyListeners(evt)\n', ' File "/home/gianz/spiderfoot/sflib.py", line 1050, in notifyListeners\n listener.handleEvent(sfEvent)\n', ' File "/home/gianz/spiderfoot/modules/sfp_spider.py", line 250, in handleEvent\n return self.spiderFrom(spiderTarget)\n', ' File "/home/gianz/spiderfoot/modules/sfp_spider.py", line 300, in spiderFrom\n freshLinks = self.processUrl(link)\n', ' File "/home/gianz/spiderfoot/modules/sfp_spider.py", line 128, in processUrl\n self.urlEvents[link] = self.linkNotify(link, self.urlEvents[url])\n', ' File "/home/gianz/spiderfoot/modules/sfp_spider.py", line 181, in linkNotify\n self.notifyListeners(event)\n', ' File "/home/gianz/spiderfoot/sflib.py", line 1050, in notifyListeners\n listener.handleEvent(sfEvent)\n', ' File "/home/gianz/spiderfoot/modules/sfp_dns.py", line 120, in handleEvent\n self.processHost(match, parentEvent)\n', ' File "/home/gianz/spiderfoot/modules/sfp_dns.py", line 271, in processHost\n self.notifyListeners(evt)\n', ' File "/home/gianz/spiderfoot/sflib.py", line 1050, in notifyListeners\n listener.handleEvent(sfEvent)\n', ' File "/home/gianz/spiderfoot/modules/sfp_spider.py", line 250, in handleEvent\n return self.spiderFrom(spiderTarget)\n', ' File "/home/gianz/spiderfoot/modules/sfp_spider.py", line 300, in spiderFrom\n freshLinks = self.processUrl(link)\n', ' File "/home/gianz/spiderfoot/modules/sfp_spider.py", line 105, in processUrl\n self.contentNotify(url, fetched, self.urlEvents[url])\n', ' File "/home/gianz/spiderfoot/modules/sfp_spider.py", line 189, in contentNotify\n self.notifyListeners(event)\n', ' File "/home/gianz/spiderfoot/sflib.py", line 1050, in notifyListeners\n listener.handleEvent(sfEvent)\n', ' File "/home/gianz/spiderfoot/modules/sfp_dns.py", line 120, in handleEvent\n self.processHost(match, parentEvent)\n', ' File "/home/gianz/spiderfoot/modules/sfp_dns.py", line 271, in processHost\n self.notifyListeners(evt)\n', ' File "/home/gianz/spiderfoot/sflib.py", line 1050, in notifyListeners\n listener.handleEvent(sfEvent)\n', ' File "/home/gianz/spiderfoot/modules/sfp_spider.py", line 250, in handleEvent\n return self.spiderFrom(spiderTarget)\n', ' File "/home/gianz/spiderfoot/modules/sfp_spider.py", line 300, in spiderFrom\n freshLinks = self.processUrl(link)\n', ' File "/home/gianz/spiderfoot/modules/sfp_spider.py", line 117, in processUrl\n links = sf.parseLinks(url, fetched['content'], self.baseDomain)\n', ' File "/home/gianz/spiderfoot/sflib.py", line 578, in parseLinks\n self.error("Error applying regex2 to: " + data)\n', 'MemoryError\n']

Hello! Faced with a problem: I ran 2 scans of 2 sites, after a while I noticed that it was completed, although I understood that this could not be the case (not all goals were met). I logged into the linux server and saw this error in the console: (2022-05-28 01:05:22,064 [ERROR] threadpool : Error in thread worker sharedThreadPool_worker_2: Traceback (most recent call last) File "/root/spiderfoot-4.0/spiderfoot/threadpool.py", line 256, in run' result = callback(×args, ×*kuargs) File "/root/spiderfoot-4.0/modules/sfp_spider.py", line 313, in handleEvent return self.spider from(spider Target) File "/root/spiderfoot-4.0/modules/sfp_spider.py", line 364, insider from freshLinks = self.processUrl(link) File "/root/spiderfoot-4.0/modules/sfp_spider.py", line 112, in process Url if self.opts['use cookies'] and fetch['headers'] is not None: TypeError: 'NoneType' object is not subscriptable 2022-05-28 02:04:46,205 [INFO] sf lib : Running 37 correlation rules. 2022-05-28 02:28:34,125 [INFO] sf lib : Scan [089C2113] completed. 2022-05-28 02:30:13,018 [INFO] sf lib : Scan [0E601609] completed.) Can you tell me what could cause such errors and what are the solutions to the problem? I leave information about my system and software: Python version used: 3.10.4 System used: Debian 11 Spiderfoot version used: 4.0 Free disk space: 100 GB Cores/RAM: 8/12 Thanks!

I'm really sorry my question is probably stupid, but how i find information from a website? I mean, how can i add a class or id in the search? Thanks in advance

Just got stable 3.3 from the download tgz. Unzipped on debian 10 and installed requirements.

networkx is stated as >=2.5 and currently installs 2.6.2. Launching sf fails on

File "/opt/spiderfoot-3.3/sflib.py", line 42, in <module>
    from networkx import nx
ImportError: cannot import name 'nx' from 'networkx'

Correcting the requirement to ==2.5 solves the problem.

Same with secure that installs 0.3.0.

Traceback (most recent call last):
  File "sf.py", line 32, in <module>
    from sfwebui import SpiderFootWebUi
  File "/opt/spiderfoot-3.3/sfwebui.py", line 27, in <module>
    from secure import SecureHeaders
ImportError: cannot import name 'SecureHeaders' from 'secure'

Using secure==0.2.1 makes it work again.

First, I've tried to insert an bitcoin adress containing 40 characteres but I got this error: Invalid target type. Could not recognize it as a target SpiderFoot supports.

Second, there are some plugins not availables like Bicoin Finder who does web scrapping for a specific address. How do I add a module to the tool?

Hello.

After installing pip and Spiderfoot in Ubuntu, when introducing the command "python sf.py" in the terminal, this is the result:

File "sf.py", line 356 print ("[", end=' ') ^ SyntaxError: invalid syntax

And I cannot launch Spiderfoot. Thanks for any help. C.

Please provide a description of the issue and any relevant error messages. Cloning Spiderfoot If you can provide an application stack trace that is even better.

What version of Python are you using? Python 3 What version of SpiderFoot are you using (stable release or Git master branch)? SpiderFoots CLI You may also wish to check if your issue has been posted previously: Yeah ~$ git clone * https://github.com/smicallef/spiderfoot.git

I am evaluating the CLI and finding it ignores the type filters. Dropped back to using the Web UI on a fresh pull from git and it seems to also ignore type filters. Anyone have any ideas what might be going on? Do these work for others? -q -x -f -s domain.com -t IP_ADDRESS,INTERNET_NAME,EMAILADDR for example works well if calling sf.py, but when issued against the CLI scans everything.

https://www.criminalip.io/ - Criminal IP is a specialized Cyber Threat Intelligence (CTI) search engine that allows users to search for various security-related information such as malicious IP addresses, domains, banners, etc. It has an asset search function which can assess IP usage information and domain search, which works to assess DNS attached to the IP.

I am using SF 4.0.0 in conjunction with Python 3.10.2.

While performing a passive search on the target example.com I encountered the following error: [INFO] sfwebui : Waiting for the scan to initialize... [INFO] sflib : Downloading configuration data from: https://publicsuffix.org/list/effective_tld_names.dat [ERROR] sflib : Unable to open option URL, https://publicsuffix.org/list/effective_tld_names.dat: HTTPSConnectionPool(host='publicsuffix.org', port=443): Max retries exceeded with url: /list/effective_tld_names.dat (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:997)'))) [INFO] sflib : Scan [4DADA4C8] failed: Could not update TLD list Process Process-1: Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap self.run() File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/process.py", line 108, in run self._target(*self._args, **self._kwargs) File "/Users/myusrename/Documents/myProjectFolder/spiderfoot/sfscan.py", line 28, in startSpiderFootScanner return SpiderFootScanner(*args, **kwargs) File "/Users/myusrename/Documents/myProjectFolder/spiderfoot/sfscan.py", line 206, in init raise ValueError("Could not update TLD list") ValueError: Could not update TLD list

It would be great if I could find a sample config for NGINX in the repo on how to run spiderfoot behind a ReverseProxy.

Attached an working example:

# /etc/nginx/conf.d/spider.conf
server {
        server_name spider.example.com;
        listen 443 ssl http2;
        listen [::]:443 http2;

        location / {
        # Reverse Proxy
                proxy_pass http://127.0.0.1:5001;
                proxy_buffering off;
                proxy_set_header Host $host;
                proxy_set_header X-Real-IP $remote_addr;

        # Basic Auth
                auth_basic_user_file /var/www/.htpasswd;
                auth_basic "Administrator Area";
        }

        # SSL Certificate Path
        ssl_certificate /etc/letsencrypt/live/spider.example.com/fullchain.pem;
        ssl_certificate_key /etc/letsencrypt/live/spider.example.com/privkey.pem;   
}

When a scan is done, and I try to export the data, to csv, it works, but to excel, it just opens a new browser tab. Or do I need to export to excel via the cli?

I'm using spiderfoot 4.0.0 running on debian 11