Line 169 errors out if you run photon against an IP. Easiest fix might be to just add a try/except, but there is prob a more elgant solution.

I'm pretty sure this was working before.

root@kali:/opt/Photon# python /opt/Photon/photon.py -u http://192.168.0.213:80
      ____  __          __
     / __ \/ /_  ____  / /_____  ____
    / /_/ / __ \/ __ \/ __/ __ \/ __ \
   / ____/ / / / /_/ / /_/ /_/ / / / /
  /_/   /_/ /_/\____/\__/\____/_/ /_/ v1.1.1

Traceback (most recent call last):
  File "/opt/Photon/photon.py", line 169, in <module>
    domain = get_fld(host, fix_protocol=True) # Extracts top level domain out of the host
  File "/usr/local/lib/python2.7/dist-packages/tld/utils.py", line 387, in get_fld
    search_private=search_private
  File "/usr/local/lib/python2.7/dist-packages/tld/utils.py", line 339, in process_url
    raise TldDomainNotFound(domain_name=domain_name)
tld.exceptions.TldDomainNotFound: Domain 192.168.0.213 didn't match any existing TLD name!

Added option to skip crawling of URLs that match a regex pattern, changed handling of seeds, and removed double spaces.

I am assuming this is what you meant in the ideas column tell me if I'm way off though :)

Awesome tool. I've been looking for something like this for a while to integrate with something I am building! I added two new options for you to consider:

1. -o/--output option that allows the user to specify an output directory (overriding the default).

Command: python photon.py -u http://10.10.10.102:80 -l 2 -t100 -o /pentest/photontest

In this case, all of the output files will be written to /pentest/photontest:

root@htbeu:/pentest/photontest# ls -ltr
total 24
-rw-r--r-- 1 root root    0 Jul 25 11:11 scripts.txt
-rw-r--r-- 1 root root 3260 Jul 25 11:11 robots.txt
-rw-r--r-- 1 root root 3260 Jul 25 11:11 links.txt
-rw-r--r-- 1 root root   17 Jul 25 11:11 intel.txt
-rw-r--r-- 1 root root  437 Jul 25 11:11 fuzzable.txt
-rw-r--r-- 1 root root  146 Jul 25 11:11 files.txt
-rw-r--r-- 1 root root    0 Jul 25 11:11 failed.txt
-rw-r--r-- 1 root root   96 Jul 25 11:11 external.txt
-rw-r--r-- 1 root root    0 Jul 25 11:11 endpoints.txt
-rw-r--r-- 1 root root    0 Jul 25 11:11 custom.txt

2. A --stdout option that allow user to print everything to stdout so they can pipe it into another tool or redirect all output to an output file with an operating system redirector.

Command: root@htbeu:/opt/dev/Photon# python photon.py -u http://10.10.10.9:80 -l 2 -t100 --stdout

Output:

      ____  __          __
     / __ \/ /_  ____  / /_____  ____
    / /_/ / __ \/ __ \/ __/ __ \/ __ \
   / ____/ / / / /_/ / /_/ /_/ / / / /
  /_/   /_/ /_/\____/\__/\____/_/ /_/

[+] URLs retrieved from robots.txt: 68
[~] Level 1: 69 URLs
[!] Progress: 69/69
[~] Level 2: 9 URLs
[!] Progress: 9/9
[~] Crawling 0 JavaScript files

--------------------------------------------------
[+] URLs: 78
[+] Intel: 1
[+] Files: 1
[+] Endpoints: 0
[+] Fuzzable URLs: 9
[+] Custom strings: 0
[+] JavaScript Files: 0
[+] External References: 3
--------------------------------------------------
[!] Total time taken: 0:32
[!] Average request time: 0.40
[+] Results saved in 10.10.10.9:80 directory

All Results:

http://10.10.10.9:80/themes/*.gif
http://10.10.10.9:80/modules/*.png
http://10.10.10.9:80/INSTALL.mysql.txt
http://10.10.10.9:80/install.php
http://10.10.10.9:80/scripts/
http://10.10.10.9:80/node/add/
http://10.10.10.9:80/?q=admin/
http://10.10.10.9:80/themes/*.png
http://10.10.10.9:80/modules/*.gif
http://10.10.10.9:80
http://10.10.10.9:80/includes/
http://10.10.10.9:80/?q=user/password/
http://10.10.10.9:80/INSTALL.txt
http://10.10.10.9:80/profiles/
http://10.10.10.9:80/themes/bartik/css/ie6.css?on28x3
http://10.10.10.9:80/MAINTAINERS.txt
http://10.10.10.9:80/themes/bartik/css/ie.css?on28x3
http://10.10.10.9:80/modules/*.jpeg
http://10.10.10.9:80/misc/*.gif

The last thing i changed is the way you were wiping the directory each time you ran the tool so that you would get clean output. If you accept the -o option which allows the user to specify the directory, you can't just blindly delete the directory anymore (can't trust user input ;)). So i think i added a cleaner way to just overwrite each file (replacing the w+ with w), that should accomplish the same thing without needing to delete directories.

Changes:

• Added the -v/--verbose option for verbose output.
• Fixed 2 logical errors in detecting bad JS scripts. - With this fix, number of bad js files have almost gone to zero.

[!] Progress: 1/1
[~] Level 2: 387 URLs
[!] Progress: 387/387
[~] Level 3: 18078 URLs
[!] Progress: 18078/18078
[~] Level 4: 90143 URLs
[!] Progress: 39750/90143^C
[~] Crawling 0 JavaScript files

Traceback (most recent call last):
  File "photon.py", line 454, in <module>
    for url in external:
RuntimeError: Set changed size during iteration

SSL issues

You're ignoring SSL verification:

/usr/local/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py:843: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecureRequestWarning)

You can fix this by adding the following to the top of the file (or wherever you feel like putting it):

import warnings
warnings.filterwarnings("ignore")

Now this is bad practice, for obvious reasons, however, it makes sense why you're doing it. The above solution is the quickest and simplest solution to solve the problem and keep the errors from being annoying.

An example with and without:

Without:

With:

Dammit Unicode

If I supply a URL that is in, lets say Russian and try to extract all the data:

      ____  __          __
     / __ \/ /_  ____  / /_____  ____
    / /_/ / __ \/ __ \/ __/ __ \/ __ \
   / ____/ / / / /_/ / /_/ /_/ / / / /
  /_/   /_/ /_/\____/\__/\____/_/ /_/ 

 URLs retrieved from robots.txt: 5
 Level 1: 6 URLs
 Progress: 6/6
 Level 2: 35 URLs
 Progress: 35/35
 Level 3: 7 URLs
 Progress: 7/7
 Crawling 7 JavaScript files
 Progress: 7/7
Traceback (most recent call last):
  File "photon.py", line 429, in <module>
    f.write(x + '\n')
UnicodeEncodeError: 'ascii' codec can't encode character u'\u200b' in position 7: ordinal not in range(128)

Easiest solution would be to just ignore things like this and continue, with a warning to the user that it was ignored.

Current Issue: Broken links for which server returns 404 status (or similar error) are not reported as failed links, instead these erroneous pages are parsed for text content.

The intels are searched only inside the page plain text to avoid retrieving tokens are garbage javascript code. Better regular expressions could allow searching inside javascript code for intels though.

v1.3.0

• Added more intels (GENERIC_URL, BRACKET_URL, BACKSLASH_URL, HEXENCODED_URL, URLENCODED_URL, B64ENCODED_URL, IPV4, IPV6, EMAIL, MD5, SHA1, SHA256, SHA512, YARA_PARSE, CREDIT_CARD)
• Intel search only applied to text (not inside javascript or html tags)
• proxy support with -p, --proxy option (http proxy only)
• minor fixes and pep8 format

Tested on

os: Linux Mint 19.1
python: 3.6.7

Hi After using the option --update i get this error:

Traceback (most recent call last): File "photon.py", line 187, in domain = topLevel(main_url) File "photon.py", line 182, in topLevel toplevel = tld.get_fld(host, fix_protocol=True) AttributeError: 'module' object has no attribute 'get_fld'

Any clue? I installed also on another system and still the same change target: same delete totally and clone again: same

OS Kali3 Thanks

Exception in thread Thread-4694: Traceback (most recent call last): File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner self.run() File "/usr/lib/python2.7/threading.py", line 754, in run self.__target(*self.__args, **self.__kwargs) File "photon.py", line 211, in extractor if is_link(link, processed, files): File "/usr/share/Photon/core/utils.py", line 41, in is_link is_file = url.endswith(BAD_TYPES) TypeError: endswith first arg must be str, unicode, or tuple, not list

Add .gitignore and requirements file just to improve development experience.

Doing this:

• You won't need to remove unnecessary python compiled files or either other not needed files.
• You just "pip install -r requirements.txt", and you'll have all necessary third-party libraries to execute script.

Under [core/requester.py] line 48 of code.

response = SESSION.get

Certificate verification is disabled by setting verify to False in get. This may lead to Man-in-the-middle attacks.

I see that this project examines robots.txt and sitemap.xml. I was wondering if you could add some of the other .well-known files like ads.txt, security.txt and others found https://well-known.dev/resources/

So i cant get an dns map only creates when getting from google and example.com.

Im using python 3 and command: python3 photon.py -u "http://example.com" --dns