Comments being baked into the prefix member of nodes is unintuitive and awkward to use for parsing I think. It would make much more sense to have a Comment class.

(This has come up while I'm working on mutmut).

Python 2.6 and 3.3 are EOL and no longer receiving security updates (or any updates) from the core Python team.

They're also little used.

Here's the pip installs for parso from PyPI for last month:

| python_version | percent | download_count | | -------------- | ------: | -------------: | | 3.6 | 55.88% | 188,129 | | 3.5 | 23.47% | 79,005 | | 3.4 | 16.04% | 53,990 | | 2.7 | 4.01% | 13,504 | | 3.7 | 0.37% | 1,259 | | 3.3 | 0.23% | 773 | | 2.6 | 0.00% | 13 | | 3.8 | 0.00% | 6 |

Source: pypinfo --start-date -39 --end-date -10 --percent --pip --markdown parso pyversion

Running from a pipenv managed virtualenv (Python 3.6.5).

I'm using Neovim v0.3.1 with deoplete-jedi. Trying to autocomplete after typing any keyword throws this exception:

[deoplete] Traceback (most recent call last):
  File "/home/sett/.vim/plugged/deoplete.nvim/rplugin/python3/deoplete/child.py", line 179, in _gather_results
    result = self._get_result(context, source)
  File "/home/sett/.vim/plugged/deoplete.nvim/rplugin/python3/deoplete/child.py", line 234, in _get_result
    ctx['candidates'] = source.gather_candidates(ctx)
  File "/home/sett/.vim/plugged/deoplete-jedi/rplugin/python3/deoplete/sources/deoplete_jedi/profiler.py", line 37, in wrapper
    return func(self, *args, **kwargs)
  File "/home/sett/.vim/plugged/deoplete-jedi/rplugin/python3/deoplete/sources/deoplete_jedi.py", line 167, in gather_candidates
    environment=self._env)
  File "/home/sett/.vim/plugged/deoplete-jedi/rplugin/python3/deoplete/sources/deoplete_jedi/profiler.py", line 37, in wrapper
    return func(self, *args, **kwargs)
  File "/home/sett/.vim/plugged/deoplete-jedi/rplugin/python3/deoplete/sources/deoplete_jedi.py", line 119, in get_script
    return jedi.Script(source, line, col, filename, environment=self._env)
  File "/home/sett/.vim/plugged/deoplete-jedi/rplugin/python3/deoplete/vendored/jedi/jedi/api/__init__.py", line 118, in __init__
    cache_path=settings.cache_directory,
  File "/home/sett/.vim/plugged/deoplete-jedi/rplugin/python3/deoplete/vendored/jedi/jedi/evaluate/__init__.py", line 388, in parse_and_get_code
    return self.grammar.parse(code=code, path=path, **kwargs), code
  File "/home/sett/.vim/plugged/deoplete-jedi/rplugin/python3/deoplete/vendored/parso/parso/grammar.py", line 76, in parse
    return self._parse(code=code, **kwargs)
  File "/home/sett/.vim/plugged/deoplete-jedi/rplugin/python3/deoplete/vendored/parso/parso/grammar.py", line 126, in _parse
    new_lines=lines
  File "/home/sett/.vim/plugged/deoplete-jedi/rplugin/python3/deoplete/vendored/parso/parso/python/diff.py", line 236, in update
    self._copy_from_old_parser(line_offset, i2, j2)
  File "/home/sett/.vim/plugged/deoplete-jedi/rplugin/python3/deoplete/vendored/parso/parso/python/diff.py", line 289, in _copy_from_old_parser
    line_offset
  File "/home/sett/.vim/plugged/deoplete-jedi/rplugin/python3/deoplete/vendored/parso/parso/python/diff.py", line 598, in copy_nodes
    self.prefix,
  File "/home/sett/.vim/plugged/deoplete-jedi/rplugin/python3/deoplete/vendored/parso/parso/python/diff.py", line 686, in _copy_nodes
    assert last_line_offset_leaf == ':'
AssertionError
Error from jedi: AssertionError().  Use :messages / see above for error details.

Simply removing/commenting offending line (line 686, diff.py) makes it work normally.

in various places in https://github.com/davidhalter/parso/blob/master/parso/python/diff.py, logging is done using the root logger instead of creating a separate logger using logging.getLogger. this makes it difficult to customize the logging behavior of parso separately from other loggers in one's app.

example:

https://github.com/davidhalter/parso/blob/0ec02e1d7feb8e79a238f79a06aa1749f02794c4/parso/python/diff.py#L119

The * in foo * bar produces produces an operator AST node with value *, same as the * in foo(*bar) even though these are radically different things. The same goes for **. It would be much nicer if these were differentiated in the AST clearly.

This was found when working on mutmut.

:eyes: Some source code analysis tools can help to find opportunities for improving software components. :thought_balloon: I propose to increase the usage of augmented assignment statements accordingly.

Would you like to integrate anything from a transformation result which can be generated by a command like the following? (:point_right: Please check also for questionable change suggestions because of an evolving search pattern.)

[Markus_Elfring@fedora lokal]$ perl -p -i.orig -0777 -e 's/^(?<indentation>\s+)(?<target>\S+)\s*=\s*\k<target>[ \t]*(?<operator>[+\-%&|^@]|\*\*?|\/\/?|<<|>>)/$+{indentation}$+{target} $+{operator}=/gm' $(find ~/Projekte/parso/lokal -name '*.py')

How do you think about to improve six source code places? :thinking:

:crystal_ball: How would a corresponding refactoring be achieved by the means of the software “parso”?

Given the following code:

def bullet(text: str) -> str:
    return f'\N{BULLET} {text}'


def error(text: str) -> str:
    return f'\N{NO ENTRY} {text}'

A syntax error is marked for ENTRY but it is valid Python code.

I'm currently trying to get rid of a lot of old code. The next release will not feature the old Pythons anymore :tada: !

IMO we should also drop Python 2 parsing. Do you agree @isidentical ? (Also don't review yet, a lot is going to change)

I guess after merging this branch anyone can feel free to make various Python 3 improvements.

Cache files that weren't accessed in the last 30 days will be automatically garbage collected. This collection happens when the save_module is called via a lock system that would make it happen only one time per day. Resolves #120

Currently people using parso might be using a lot of disk space after a while. ~/.cache/parso might get quite big. (or whatever it is on other OS's, see parso/cache.py.

IMO we should probably clean up in parso.cache.save_module. That function should just check "once a day" if files are older than a month. In that case they should be deleted. It should be fine to delete stuff that we might parse again, since the cache is not extremely important and just there to speed up some things.

As a small additional thing, we should probably use scandir, see also https://github.com/davidhalter/jedi/blob/7fd5c8af8ff04711599fea10ca8babe51b280464/jedi/_compatibility.py#L29-L47 for Python 2.7/3.5. However we could also just remove Python 2.7/3.5 support for this feature. I don't plan on maintaining it for a long time.

Parso includes calls to hashlib.sha256 with an argument.

The documentation doesn't mention arguments. In my Windows Python 3.8 it does take arguments, but here is a log from a package build in Fedora, which failed with:

parser = <class 'parso.python.parser.Parser'>
diff_parser = <class 'parso.python.diff.DiffParser'>
    def __init__(self, text, tokenizer, parser=BaseParser, diff_parser=None):
        self._pgen_grammar = generate_grammar(
            text,
            token_namespace=self._get_token_namespace()
        )
        self._parser = parser
        self._tokenizer = tokenizer
        self._diff_parser = diff_parser
>       self._hashed = hashlib.sha256(text.encode("utf-8")).hexdigest()
E       TypeError: sha256() takes no arguments
/usr/lib/python3.8/site-packages/parso/grammar.py:39: TypeError

This is probably simple to fix, see:

>>> import jedi
>>> x = jedi.Script('f(a := 1, b)\n').get_syntax_errors()
>>> x[0].get_message()
'SyntaxError: positional argument follows keyword argument'
>>> jedi.Script('f(a := 1)\n').get_syntax_errors()
[]

I've tried to apply mutmut to PyPDF2 and failed. Initially I thought the issue was with mutmut ( https://github.com/boxed/mutmut/issues/252 ) , but I guess it's a parso bug.

Minimal example

import parso

parso.parse('stream.write(b_(rf"\{ord_(c):0>3o}"))', error_recovery=False)

Traceback

Traceback (most recent call last):
  File "/home/moose/.pyenv/versions/3.10.2/lib/python3.10/site-packages/parso/parser.py", line 180, in _add_token
    plan = stack[-1].dfa.transitions[transition]
KeyError: ReservedString(})

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/moose/.pyenv/versions/3.10.2/lib/python3.10/site-packages/parso/__init__.py", line 58, in parse
    return grammar.parse(code, **kwargs)
  File "/home/moose/.pyenv/versions/3.10.2/lib/python3.10/site-packages/parso/grammar.py", line 156, in parse
    root_node = p.parse(tokens=tokens)
  File "/home/moose/.pyenv/versions/3.10.2/lib/python3.10/site-packages/parso/python/parser.py", line 79, in parse
    return super().parse(tokens)
  File "/home/moose/.pyenv/versions/3.10.2/lib/python3.10/site-packages/parso/parser.py", line 129, in parse
    self._add_token(token)
  File "/home/moose/.pyenv/versions/3.10.2/lib/python3.10/site-packages/parso/parser.py", line 186, in _add_token
    self.error_recovery(token)
  File "/home/moose/.pyenv/versions/3.10.2/lib/python3.10/site-packages/parso/python/parser.py", line 140, in error_recovery
    return super().error_recovery(token)
  File "/home/moose/.pyenv/versions/3.10.2/lib/python3.10/site-packages/parso/parser.py", line 152, in error_recovery
    raise ParserSyntaxError('SyntaxError: invalid syntax', error_leaf)
parso.parser.ParserSyntaxError: ('SyntaxError: invalid syntax', <ErrorLeaf: PythonTokenTypes.OP:'}', (1, 33)>)

Environment

$ python --version     
Python 3.10.2

>>> parso.__version__
'0.8.3'

Besides match case, python 3.9 alpha6 / 3.10 also added parenthesized imports and context-managers. Parenthesized imports works no problem in jedi, but parenthesized context managers lead to a syntax error:

>>> import jedi
>>> source = '''
... with (
...     open('/dev/stdout', 'w') as a,
...     open('/dev/stdout', 'w') as b,
... ):
...     a.write('a')
...     b.write('b')
... '''
>>> script = jedi.Script(source, path='example.py')
>>> [print(x, x.get_message()) for x in script.get_syntax_errors()]
<SyntaxError from=(3, 29) to=(3, 31)> SyntaxError: invalid syntax
<SyntaxError from=(4, 29) to=(4, 31)> SyntaxError: invalid syntax
<SyntaxError from=(5, 0) to=(5, 1)> SyntaxError: invalid syntax
<SyntaxError from=(6, 0) to=(6, 4)> IndentationError: unexpected indent

See:

• https://docs.python.org/3/whatsnew/3.10.html#parenthesized-context-managers

Probably relates to:

• davidhalter/jedi#1830

Code snippet:

im.convert('1').save(rf'script\temp\{ord(c)}.png')
                                           ^ SyntaxError: invalid syntax *jedi*

rf - a convenient way of invoiding anomalous backslash in string, isn't it?

I'm starting to look for a library to do source transformations and I think parso could handle it via its parso.Grammar.refactor() method, but there's not docs on how to use it.