Hello,

Little unconsistency when parsing comment at start of file.

If it starts with a newline, we alternate endl/comment:

parse('\n#foo\n#bar\n')
Out[7]: 
[{'formatting': [], 'indent': '', 'type': 'endl', 'value': '\n'},
 {'formatting': [], 'type': 'comment', 'value': '#foo'},
 {'formatting': [], 'indent': '', 'type': 'endl', 'value': '\n'},
 {'formatting': [], 'type': 'comment', 'value': '#bar'},
 {'formatting': [], 'indent': '', 'type': 'endl', 'value': '\n'}]

If it starts with a comment, we get a endl that includes the comment as formatting.

parse('#foo\n#bar\n')
[{'formatting': [{'formatting': [], 'type': 'comment', 'value': '#foo'}],
  'indent': '',
  'type': 'endl',
  'value': '\n'},
 {'formatting': [], 'type': 'comment', 'value': '#bar'},
 {'formatting': [], 'indent': '', 'type': 'endl', 'value': '\n'}]

Not too sure what the proper behaviour is supposed to be, but I guess it should be something similar in both cases.

Thanks for your work on this project! A question/suggestion: would you consider granting commit rights to more people to help the project move forward? There are quite a few PRs for Python 3 support that have been open for ~9 months without activity (#108, #113, #114 #115, #116, #118, #119, #120, #126) and it would be great to have support for some of these features in a release.

@odcinek seems like a good person for the job if he's willing, seeing how most of the PRs are his...

/ping @Psycojoker [With apologies to psychoJoker who I pinged by accident before editing...]

Resolves https://github.com/PyCQA/baron/issues/127

Please let me know what you think.

This should allow for typed parameters like def foo(x: int) and def foo(x: List[int]). I didn't try testing when parameters like *args or **kwargs are typed. I think that would be good to add in the future but it just wasn't as necessary for me at the moment.

This does not add support for typed variables or function return types, but I hope to add support for the later in the near future assuming this is accepted.

If another process writes the cache file while we were generating the grammar, we'll fail to run os.open(cache_file, os.O_RDWR | os.O_CREAT | os.O_EXCL, 0o0600) because the file (now) exists.

This allows baron to recover in case the cache file has invalid JSON by deleting and recreating it. Before, it would fail with an exception on initialization.

In Python, two adjacent strings get concatenated implicitly. Missing commas in multi-line string lists is a common source of bugs causing unwanted string concatenation. In this case, it is clear that this comma is missing by mistake and there should not be a concatenation.

Hi guys, I ran into a problem where Baron interprets the "-" in the "->" of a function's return annotation as a minus instead of as part of an arrow.

def foo() -> int: return 3

baron.parser.ParsingError: Error, got an unexpected token MINUS here: 1 def foo() -<---- here The token MINUS should be one of those: COLON

Very cool project by the way; would be nice to use something like RB in combination with support for annotations to enable some proper static analysis of functions, maybe even hijack some ideas from Haskell.

Python 3 has a wonderful feature called f-strings, which allows in-string interpolation. E.g.:

from baron import parse
parse("f'{a}'")

These are not currently parsed by baron.

As per their blog post of the 27th April ‘Securing subdomains’:

Starting today, Read the Docs will start hosting projects from subdomains on the domain readthedocs.io, instead of on readthedocs.org. This change addresses some security concerns around site cookies while hosting user generated data on the same domain as our dashboard.

Test Plan: Manually visited all the links I’ve modified.

Baron fails to see an error here:

>>> a = 1 = 2
  File "<stdin>", line 1
SyntaxError: can't assign to literal

>>> RedBaron('a = 1 = 2')
0   a = 1 = 2

I know baron has a relaxed syntax, so I don't know if this is really an error. But it should at least be documented.

example:

core_topic_set = {
                *flattenUnhashableList(
                    [value for key, value in getMetatopic[dog_or_cat].items()]
                )
            }

example:

the unicode below are just attributes inside a class.

orders=[
        BSP.all.order.最多点击,
        BSP.all.order.最多收藏,
        BSP.all.order.最新发布,
        BSP.all.order.最多弹幕,
        BSP.all.order.综合排序,
    ]

exception:

Traceback (most recent call last):
  File "/media/root/parrot/pyjom/tests/skipexception_code_and_continue_resurrection_time_travel_debugging/codemod_redbaron/redbaron_test.py", line 6, in <module>
    r=redbaron.RedBaron(t)
  File "/usr/local/lib/python3.9/dist-packages/redbaron/redbaron.py", line 36, in __init__
    self.node_list = base_nodes.NodeList.from_fst(baron.parse(source_code), parent=self, on_attribute="root")
  File "/usr/local/lib/python3.9/dist-packages/baron/baron.py", line 49, in parse
    tokens = tokenize(source_code, False)
  File "/usr/local/lib/python3.9/dist-packages/baron/baron.py", line 70, in tokenize
    splitted = split(pouet)
  File "/usr/local/lib/python3.9/dist-packages/baron/spliter.py", line 6, in split
    return list(split_generator(sequence))
  File "/usr/local/lib/python3.9/dist-packages/baron/spliter.py", line 76, in split_generator
    raise UntreatedError("Untreated elements: %s" % iterator.rest_of_the_sequence().__repr__()[:50])
baron.spliter.UntreatedError: Untreated elements: '最多点击,\n        BSP.all.order.最多收藏,\n        BSP.a

The following code...

import redbaron
rb = redbaron.RedBaron('func = print if True else print')
print(rb.dumps())

...produces the following output:

func = printif True else print

Note that the space between the print and the if has been removed.

This bug does not occur if the the word print is replaced with foo.

The following code fails with a ParsingError:

from redbaron import RedBaron

text = '''
if True:
    pass
elif True: pass

else:
    pass
'''

RedBaron(text)

The python interpreter parses text without any syntax error.

Deleting the blank line between the elif and the else fixes the ParsingError.

Traceback (most recent call last):
  File ".../site-packages/baron/baron.py", line 20, in _parse
    return parser(tokens)
  File ".../site-packages/baron/grammator.py", line 836, in parse
    return parser.parse(iter(tokens))
  File ".../site-packages/baron/parser.py", line 165, in parse
    raise ParsingError(debug_output)
baron.parser.ParsingError: Error, got an unexpected token ELSE here:

   1
   2 if True:
   3     pass
   4 elif True: pass
   5
   6 else<---- here

The token ELSE should be one of those: $end, ASSERT, AT, BACKQUOTE, BINARY, BINARY_RAW_STRING, BINARY_STRING, BREAK, CLASS, COMMENT, COMPLEX, CONTINUE, DEF, DEL, ELLIPSIS, ENDL, ENDMARKER, EXEC, FLOAT, FLOAT_EXPONANT, FLOAT_EXPONANT_COMPLEX, FOR, FROM, GLOBAL, HEXA, IF, IMPORT, INT, INTERPOLATED_RAW_STRING, INTERPOLATED_STRING, LAMBDA, LEFT_BRACKET, LEFT_PARENTHESIS, LEFT_SQUARE_BRACKET, LONG, MINUS, NAME, NONLOCAL, NOT, OCTA, PASS, PLUS, PRINT, RAISE, RAW_STRING, RETURN, STAR, STRING, TILDE, TRY, UNICODE_RAW_STRING, UNICODE_STRING, WHILE, WITH, YIELD

Baron has failed to parse this input. If this is valid python code (and by that I mean that the python binary successfully parse this code without any syntax error) (also consider that baron does not yet parse python 3 code integrally) it would be kind if you can extract a snippet of your code that make Baron fails and open a bug here: https://github.com/PyCQA/baron/issues

Sorry for the inconvenience.

PEP-570 adds support for positional-only arguments. Baron seem to support keyword-only arguments, as shown in test_case_1 below, which is represented by a kwargs_only_marker node. Following this convention i propose creating a posargs_only_marker node for the test_case_2 example.

Testcase:

from redbaron import RedBaron
import ast, json

test_case_1 = "def foo(a, *, b, c): pass"  # kwarg only
test_case_2 = "def foo(a, /, b, c): pass"  # pos only

print(ast.dump(ast.parse(test_case_1), indent=4))        # dump python ast
print("=" * 60)
print(json.dumps(RedBaron(test_case_1).fst(), indent=4)) # dump redbaron ast
print("=" * 60)
print(ast.dump(ast.parse(test_case_2), indent=4))        # dump python ast
print("=" * 60)
print(json.dumps(RedBaron(test_case_2).fst(), indent=4)) # dump redbaron ast

Module(
    body=[
        FunctionDef(
            name='foo',
            args=arguments(
                posonlyargs=[],
                args=[
                    arg(arg='a')],
                kwonlyargs=[
                    arg(arg='b'),
                    arg(arg='c')],
                kw_defaults=[
                    None,
                    None],
                defaults=[]),
            body=[
                Pass()],
            decorator_list=[])],
    type_ignores=[])
============================================================
[
    {
        "type": "def",
        "async": false,
        "name": "foo",
        "decorators": [],
        "async_formatting": [],
        "first_formatting": [
            {
                "type": "space",
                "value": " "
            }
        ],
        "second_formatting": [],
        "third_formatting": [],
        "arguments": [
            {
                "type": "def_argument",
                "annotation_first_formatting": [],
                "annotation_second_formatting": [],
                "first_formatting": [],
                "second_formatting": [],
                "target": {
                    "type": "name",
                    "value": "a"
                },
                "annotation": {},
                "value": {}
            },
            {
                "type": "comma",
                "first_formatting": [],
                "second_formatting": [
                    {
                        "type": "space",
                        "value": " "
                    }
                ]
            },
            {
                "type": "kwargs_only_marker",
                "formatting": []
            },
            {
                "type": "comma",
                "first_formatting": [],
                "second_formatting": [
                    {
                        "type": "space",
                        "value": " "
                    }
                ]
            },
            {
                "type": "def_argument",
                "annotation_first_formatting": [],
                "annotation_second_formatting": [],
                "first_formatting": [],
                "second_formatting": [],
                "target": {
                    "type": "name",
                    "value": "b"
                },
                "annotation": {},
                "value": {}
            },
            {
                "type": "comma",
                "first_formatting": [],
                "second_formatting": [
                    {
                        "type": "space",
                        "value": " "
                    }
                ]
            },
            {
                "type": "def_argument",
                "annotation_first_formatting": [],
                "annotation_second_formatting": [],
                "first_formatting": [],
                "second_formatting": [],
                "target": {
                    "type": "name",
                    "value": "c"
                },
                "annotation": {},
                "value": {}
            }
        ],
        "fourth_formatting": [],
        "return_annotation_first_formatting": [],
        "return_annotation_second_formatting": [],
        "fifth_formatting": [],
        "sixth_formatting": [
            {
                "type": "space",
                "value": " "
            }
        ],
        "value": [
            {
                "type": "pass"
            },
            {
                "type": "endl",
                "value": "\n",
                "indent": "",
                "formatting": []
            }
        ],
        "return_annotation": {}
    }
]
============================================================
Module(
    body=[
        FunctionDef(
            name='foo',
            args=arguments(
                posonlyargs=[
                    arg(arg='a')],
                args=[
                    arg(arg='b'),
                    arg(arg='c')],
                kwonlyargs=[],
                kw_defaults=[],
                defaults=[]),
            body=[
                Pass()],
            decorator_list=[])],
    type_ignores=[])
============================================================
Traceback (most recent call last):
  File "/home/pbsds/.cache/pypoetry/virtualenvs/fix-my-functions-Moge_7ev-py3.9/lib/python3.9/site-packages/baron/baron.py", line 20, in _parse
    return parser(tokens)
  File "/home/pbsds/.cache/pypoetry/virtualenvs/fix-my-functions-Moge_7ev-py3.9/lib/python3.9/site-packages/baron/grammator.py", line 836, in parse
    return parser.parse(iter(tokens))
  File "/home/pbsds/.cache/pypoetry/virtualenvs/fix-my-functions-Moge_7ev-py3.9/lib/python3.9/site-packages/baron/parser.py", line 165, in parse
    raise ParsingError(debug_output)
baron.parser.ParsingError: Error, got an unexpected token SLASH here:

   1 def foo(a, /<---- here

The token SLASH should be one of those: COMMA, DOUBLE_STAR, LEFT_PARENTHESIS, NAME, RIGHT_PARENTHESIS, STAR