A Static Analysis Tool for Detecting Security Vulnerabilities in Python Web Applications

Issue: #127 There is a few steps for completing this PR. Now we can get all ".py" files in directory and exclude some files with "-x" option.

As we can see on CodeClimate https://codeclimate.com/github/python-security/pyt/coverage/5935971dbf92ed000102998b there is pretty low test coverage of main, I understand why this is but adding some tests for it would increase our test coverage percentage and 75% isn't satisfying.

If you have any trouble with this I can help, I am going to label this issue as Easy so new comers see it.

If you look at https://github.com/trailofbits/manticore/blob/master/README.md you can see a nice link at the top to the docs. I'll write the docs once the layout is there, please see https://www.slideshare.net/mobile/JohnCosta/how-to-readthedocs

(So the [easy] issues are good for new people who want to start contributing to look at.)

Right now sinks seem to be considered during vulnerability analysis only in case of "module scope imports". E.g. vulnerabilities w.r.t. sink subprocess.call( are only detected in case the production code imports module scope wise:

import subprocess

subprocess.call(

In case the production code introduces the sink via module import the vulnerability won't be detected.

from subprocess import call

call(

Would be great to get a new release published on pypi soon.

We have over 190 commits added, though I'm not sure of the specific criteria for bumping the version.

This should resolve #128. The change is so straight forward and any potential tests would be awkward, so I'm not sure we want to include specific tests for this (there were none before for interactive mode anyway).

I'm open to suggestions though.

You can manually test this change by using this sample code:

import scrypt


image_name = request.args.get('image_name')
if not image_name:
    image_name = 'foo'
foo = scrypt.outer(image_name) # Any call after ControlFlowNode caused the problem
foo = scrypt.hash(foo, 'salt')
foo = scrypt.encrypt(os.urandom(datalength), foo)
send_file(foo)

Then python -m pyt sample.py -m bb.txt -i. You can see how it does as many as you want until you answer s.

So both detect-secrets and Bandit have the concept of whitelisting a line by putting a comment at the end, similar to how you've probably seen people do # noqa: F401 or whatever, with pylint.

Let us steal once again, from Bandit, since they are most similar to us, here are the relevant lines, but we shall change lineno + 1 for to enumerate(lines, start=1) because it is more pythonic.

They also have the --ignore-nosec do not skip lines with # nosec comments command line optionso we shall pass in the set of lines to the 2 calls tofind_vulnerabilities` in __main__,

Hi!

For some weird reason when cloning the repo on a mac (tested with 10.11 and 10.13) the file pyt/trigger_definitions/flask_trigger_words.pyt won't be written.

here's an example:

} /tmp$ git clone https://github.com/python-security/pyt.git
Cloning into 'pyt'...
remote: Counting objects: 5740, done.
remote: Total 5740 (delta 0), reused 0 (delta 0), pack-reused 5740
Receiving objects: 100% (5740/5740), 2.62 MiB | 3.75 MiB/s, done.
Resolving deltas: 100% (3916/3916), done.
Checking connectivity... done.
} /tmp$ cd pyt/
} /tmp/pyt$ git status
On branch master
Your branch is up-to-date with 'origin/master'.
Changes not staged for commit:
  (use "git add/rm <file>..." to update what will be committed)
  (use "git checkout -- <file>..." to discard changes in working directory)

	deleted:    pyt/trigger_definitions/flask_trigger_words.pyt

no changes added to commit (use "git add" and/or "git commit -a")

Even copy pasting the content in a file result in the file not existing. Tried with default terminal, iterm2 and intellij's terminal, all the same so musn't be the terminal.

Doing some try/fail we suspect that the faulty line is subprocess.call( but doing a hexdump of the file (on a xenial box) doesn't show much...

root@web1:~/pyt/pyt/trigger_definitions# cat flask_trigger_words.pyt | hexdump -C
00000000  73 6f 75 72 63 65 73 3a  0a 67 65 74 28 0a 2e 64  |sources:.get(..d|
00000010  61 74 61 0a 66 6f 72 6d  5b 0a 66 6f 72 6d 28 0a  |ata.form[.form(.|
00000020  4d 61 72 6b 75 70 28 0a  63 6f 6f 6b 69 65 73 5b  |Markup(.cookies[|
00000030  0a 66 69 6c 65 73 5b 0a  53 51 4c 41 6c 63 68 65  |.files[.SQLAlche|
00000040  6d 79 0a 0a 73 69 6e 6b  73 3a 0a 72 65 70 6c 61  |my..sinks:.repla|
00000050  63 65 28 20 2d 3e 20 65  73 63 61 70 65 0a 73 65  |ce( -> escape.se|
00000060  6e 64 5f 66 69 6c 65 28  20 2d 3e 20 27 2e 2e 27  |nd_file( -> '..'|
00000070  2c 20 27 2e 2e 27 20 69  6e 0a 65 78 65 63 75 74  |, '..' in.execut|
00000080  65 28 0a 73 79 73 74 65  6d 28 0a 66 69 6c 74 65  |e(.system(.filte|
00000090  72 28 0a 73 75 62 70 72  6f 63 65 73 73 2e 63 61  |r(.subprocess.ca|
000000a0  6c 6c 28 0a 72 65 6e 64  65 72 5f 74 65 6d 70 6c  |ll(.render_templ|
000000b0  61 74 65 28 0a 73 65 74  5f 63 6f 6f 6b 69 65 28  |ate(.set_cookie(|
000000c0  0a 72 65 64 69 72 65 63  74 28 0a 75 72 6c 5f 66  |.redirect(.url_f|
000000d0  6f 72 28 0a 66 6c 61 73  68 28 0a 6a 73 6f 6e 69  |or(.flash(.jsoni|
000000e0  66 79 28                                          |fy(|
000000e3

The result of this is the tool can't seem to run on mac since this file is not available, fails with

Traceback (most recent call last):
  File ".../bin/pyt", line 11, in <module>
    load_entry_point('pyt==1.0.0a20', 'console_scripts', 'pyt')()
  File ".../lib/python3.5/site-packages/pyt-1.0.0a20-py3.5.egg/pyt/__main__.py", line 247, in main
    args.trim_reassigned_in)
  File ".../lib/python3.5/site-packages/pyt-1.0.0a20-py3.5.egg/pyt/vulnerabilities.py", line 394, in find_vulnerabilities
    definitions = parse(trigger_word_file)
  File ".../lib/python3.5/site-packages/pyt-1.0.0a20-py3.5.egg/pyt/trigger_definitions_parser.py", line 48, in parse
    with open(trigger_word_file, 'r') as fd:
FileNotFoundError: [Errno 2] No such file or directory: '.../lib/python3.5/site-packages/pyt-1.0.0a20-py3.5.egg/pyt/trigger_definitions/flask_trigger_words.pyt'

Does that ring any bell?

~~I'll try to work on this relatively soon, but~~ to think out loud..

In interprocedural_cfg.py, we have

def return_handler(self, node, function_nodes):
    """Handle the return from a function during a function call."""
    call_node = None
    for n in function_nodes:
        if isinstance(n, ConnectToExitNode):
            LHS = CALL_IDENTIFIER + 'call_' + str(self.function_index)
            previous_node = self.nodes[-1]
            if not call_node:
                RHS = 'ret_' + get_call_names_as_string(node.func)
                r = RestoreNode(LHS + ' = ' + RHS, LHS, [RHS],
                                line_number=node.lineno,
                                path=self.filenames[-1])
                call_node = self.append_node(r)
                previous_node.connect(call_node)
        else:
            # lave rigtig kobling
            pass

which cleaned is

def return_handler(self, call_node, function_nodes):
    """Handle the return from a function during a function call.

    Args:
        call_node(ast.Call) : The node that calls the definition.
        function_nodes(list[Node]): List of nodes of the function being called.
    """
    for node in function_nodes:
        # Only Return's and Raise's can be of type ConnectToExitNode
        if isinstance(node, ConnectToExitNode):                
            # Create e.g. ¤call_1 = ret_func_foo RestoreNode
            LHS = CALL_IDENTIFIER + 'call_' + str(self.function_call_index)
            RHS = 'ret_' + get_call_names_as_string(call_node.func)
            return_node = RestoreNode(LHS + ' = ' + RHS,
                                      LHS,
                                      [RHS],
                                      line_number=call_node.lineno,
                                      path=self.filenames[-1])
            self.nodes[-1].connect(return_node)
            self.nodes.append(return_node)
            return 

Firstly, the for loop and the if statement seem to just serve the purpose of "Is there a node of type Return or Raise in the function?" But I think all functions should have at least one return node, right? I'm not sure if I understand the original intention that well e.g. what was going to be in the else?

Secondly, here is an example to illustrate the problem/need to handle multiple returns:

TODO

So let us steal, once again (a 3rd, or 4th time), from Bandit.

https://github.com/openstack/bandit/blob/master/bandit/cli/main.py#L157-L160

https://github.com/openstack/bandit/blob/8f09d8b208f037b7d49ed6bc88f2ac200e7cc06c/bandit/core/manager.py#L171-L219

This will enable a user to just give -r /path/to/files instead of -f file one at a time.

This is less straightforward, you should also add

    parser.add_argument(
        'targets', metavar='targets', type=str, nargs='*',
        help='source file(s) or directory(s) to be tested'
    )

    parser.add_argument(
        '-x', '--exclude', dest='excluded_paths', action='store',
        default='', help='comma-separated list of paths to exclude from scan '
                         '(note that these are in addition to the excluded '
                         'paths provided in the config file)'
    )

too.

This will help us out a great deal, as I'm working on documentation, and then after docs I'll return to the CFG work (the expr_star_handler stuff). So without you I wouldn't work on this for a while.

Hi team,

Thanks for your project. We added Pyt to our security tools section. While reviewing we noticed we could only extract tags from the GitHub repo, as no releases are used. Now tags are sorted by alphabet, meaning that the tagged version 'finalfinal' always is listed as the highest release number. Not sure if it is possible, but you could re-tag it 'final' and 'finalfinal'. Then the community can pick up future released and promote your project. Otherwise, it gets stuck on a very old version ;-)

Excuse me, why is there no argument in the variable args in the picture below? cmd_result = subprocess.Popen(cmd,shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, stdin=subprocess.DEVNULL).stdout.read() Sorry, I'm not a native speaker. I hope I'm making myself clear

Traceback (most recent call last):
  File "c:\users\matth\appdata\local\programs\python\python38\lib\runpy.py", line 193, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "c:\users\matth\appdata\local\programs\python\python38\lib\runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "C:\Users\matth\.local\bin\pyt.exe\__main__.py", line 7, in <module>
  File "c:\users\matth\.local\pipx\venvs\python-taint\lib\site-packages\pyt\__main__.py", line 92, in main
    nosec_lines[path] = retrieve_nosec_lines(path)
  File "c:\users\matth\.local\pipx\venvs\python-taint\lib\site-packages\pyt\__main__.py", line 57, in retrieve_nosec_lines
    lines = file.readlines()
  File "c:\users\matth\appdata\local\programs\python\python38\lib\encodings\cp1252.py", line 23, in decode
    return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position 2105: character maps to <undefined>

sometimes this helps

export PYTHONIOENCODING=utf-8
export LC_ALL=en_US.UTF-8
export LANG=en_US.UTF-8

but today it didn't so I'm about ready to stop using pyt... I'm somewhat worried I've been using it wrong for a few years because of the various tools I use, pyt never complained about anything, (i.e. found no vulnerabilities or bugs, neither positive or false)

If anyone ever takes over this project, then all the file open() calls should either specify utf-8 (a better "guess") or use chardet to make a really good guess.

There is a small typo in README.rst.

Should read traversal rather than traveral.

Semi-automated pull request generated by https://github.com/timgates42/meticulous/blob/master/docs/NOTE.md

Looks like pyt took a dependency on a private attribute that has disappeared. Yes. I saw the tool is no longer maintained, but it used to at least run on 3.8.x

Traceback (most recent call last):
  File "/root/.local/bin/pyt", line 8, in <module>
    sys.exit(main())
  File "/root/.local/pipx/venvs/python-taint/lib/python3.9/site-packages/pyt/__main__.py", line 101, in main
    cfg = make_cfg(
  File "/root/.local/pipx/venvs/python-taint/lib/python3.9/site-packages/pyt/cfg/make_cfg.py", line 36, in make_cfg
    visitor = ExprVisitor(
  File "/root/.local/pipx/venvs/python-taint/lib/python3.9/site-packages/pyt/cfg/expr_visitor.py", line 69, in __init__
    self.init_cfg(node)
  File "/root/.local/pipx/venvs/python-taint/lib/python3.9/site-packages/pyt/cfg/expr_visitor.py", line 76, in init_cfg
    module_statements = self.visit(node)
  File "/usr/local/lib/python3.9/ast.py", line 407, in visit
    return visitor(node)
  File "/root/.local/pipx/venvs/python-taint/lib/python3.9/site-packages/pyt/cfg/stmt_visitor.py", line 67, in visit_Module
    return self.stmt_star_handler(node.body)
  File "/root/.local/pipx/venvs/python-taint/lib/python3.9/site-packages/pyt/cfg/stmt_visitor.py", line 88, in stmt_star_handler
    node = self.visit(stmt)
  File "/usr/local/lib/python3.9/ast.py", line 407, in visit
    return visitor(node)
  File "/root/.local/pipx/venvs/python-taint/lib/python3.9/site-packages/pyt/cfg/stmt_visitor.py", line 1069, in visit_ImportFrom
    return self.add_module(
  File "/root/.local/pipx/venvs/python-taint/lib/python3.9/site-packages/pyt/cfg/stmt_visitor.py", line 807, in add_module
    self.visit(tree)
  File "/usr/local/lib/python3.9/ast.py", line 407, in visit
    return visitor(node)
  File "/root/.local/pipx/venvs/python-taint/lib/python3.9/site-packages/pyt/cfg/stmt_visitor.py", line 67, in visit_Module
    return self.stmt_star_handler(node.body)
  File "/root/.local/pipx/venvs/python-taint/lib/python3.9/site-packages/pyt/cfg/stmt_visitor.py", line 88, in stmt_star_handler
    node = self.visit(stmt)
  File "/usr/local/lib/python3.9/ast.py", line 407, in visit
    return visitor(node)
  File "/root/.local/pipx/venvs/python-taint/lib/python3.9/site-packages/pyt/cfg/stmt_visitor.py", line 460, in visit_Assign
    label.visit(node)
  File "/usr/local/lib/python3.9/ast.py", line 407, in visit
    return visitor(node)
  File "/root/.local/pipx/venvs/python-taint/lib/python3.9/site-packages/pyt/helper_visitors/label_visitor.py", line 56, in visit_Assign
    self.visit(node.value)
  File "/usr/local/lib/python3.9/ast.py", line 407, in visit
    return visitor(node)
  File "/root/.local/pipx/venvs/python-taint/lib/python3.9/site-packages/pyt/helper_visitors/label_visitor.py", line 334, in visit_IfExp
    self.visit(node.orelse)
  File "/usr/local/lib/python3.9/ast.py", line 407, in visit
    return visitor(node)
  File "/root/.local/pipx/venvs/python-taint/lib/python3.9/site-packages/pyt/helper_visitors/label_visitor.py", line 173, in visit_Subscript
    self.slicev(node.slice)
  File "/root/.local/pipx/venvs/python-taint/lib/python3.9/site-packages/pyt/helper_visitors/label_visitor.py", line 190, in slicev
    self.visit(node.value)
  File "/usr/local/lib/python3.9/ast.py", line 407, in visit
    return visitor(node)
  File "/usr/local/lib/python3.9/ast.py", line 411, in generic_visit
    for field, value in iter_fields(node):
  File "/usr/local/lib/python3.9/ast.py", line 249, in iter_fields
    for field in node._fields:
AttributeError: 'str' object has no attribute '_fields'

Hi, I am trying pyt to work through callbacks. It taints the function arguments but not callbacks. If anyone has any idea about this , please let me know.