Bottom-up approach to refactoring in python

Overview

Introduction

Build Status Latest Version Supported Python versions Development Status Wheel Status Download format License Backers on Open Collective Sponsors on Open Collective

RedBaron is a python library and tool powerful enough to be used into IPython solely that intent to make the process of writing code that modify source code as easy and as simple as possible. That include writing custom refactoring, generic refactoring, tools, IDE or directly modifying you source code into IPython with a higher and more powerful abstraction than the advanced texts modification tools that you find in advanced text editors and IDE.

RedBaron guaranteed you that it will only modify your code where you ask him to. To achieve this, it is based on Baron a lossless AST for Python that guarantees the operation ast_to_code(code_to_ast(source_code)) == source_code. (Baron's AST is called an FST, a Full Syntax Tree).

RedBaron API and feel is heavily inspired by BeautifulSoup. It tries to be simple and intuitive and that once you've get the basics principles, you are good without reading the doc for 80% of your operations.

For now, RedBaron can be considered in alpha, the core is quite stable but it is not battle tested yet and is still a bit rough. Feedback and contribution are very welcome.

The public documented API on the other side is guaranteed to be retro-compatible and won't break until 2.0 (if breaking is needed at that point). There might be the only exception that if you directly call specific nodes constructors with FST that this API change, but this is not documented and simply horribly unpracticable, so I'm expecting no one to do that.

Support

RedBaron is support python python 2 and up to python 3.7 grammar.

Roadmap

Current roadmap is as boring as needed:

  • bug fixes
  • new small features (walker pattern, maybe code generation) and performance improvement.

Installation

pip install redbaron[pygments]

Or if you don't want to have syntax highlight in your shell or don't need it:

pip install redbaron

Running tests

pip install pytest
py.test tests

Community

You can reach us on irc.freenode.net#baron or irc.freenode.net##python-code-quality.

Financial support

Baron and RedBaron are a very advanced piece of engineering that requires a lot of time of concentration to work on. Until the end of 2018, the development has been a full volunteer work mostly done by Bram, but now, to reach the next level and bring those projects to the stability and quality you expect, we need your support.

You can join our contributors and sponsors on our transparent OpenCollective, every contribution will count and will be mainly used to work on the projects stability and quality but also on continuing, on the side, the R&D side of those projects.

Backers

Thank you to all our backers! šŸ™ [Become a backer]

Sponsors

Support this project by becoming a sponsor. Your logo will show up here with a link to your website. [Become a sponsor]

Contributors

This project exists thanks to all the people who contribute.

Code of Conduct

As a member of PyCQA, RedBaron follows its Code of Conduct.

Links

RedBaron is fully documented, be sure to check the tutorial and documentation.

Comments
  • Regression with if or def statements and trailing comments:

    Regression with if or def statements and trailing comments:

    Hello,

    I am having a severe red baron regression parsing these kinds of codes:

        if python_version >= 340 and False: # TODO: Temporarily reverted:
            tmp_class = class_creation_function.allocateTempVariable(
                temp_scope = None,
                name       = "__class__"
            )
    
    def displayTree(tree): # pragma: no cover
        # Import only locally so the Qt4 dependency doesn't normally come into play
        # when it's not strictly needed.
        from .gui import TreeDisplay
    
    

    Seems the comments trailing the ":" are now considered illegal. But I swear to you, they are.

    The version I am using from PyPI, is 0.6.3, I think 0.6.2 didn't do this at all, but I might be wrong.

    Yours, Kay

    opened by kayhayen 13
  • Unsynchronized tree: __repr__ recursion fix for #119, #121

    Unsynchronized tree: __repr__ recursion fix for #119, #121

    Possible fix #119 and fix #121 . (Copied from my email version to @Psycojoker ) After debugging, I found some solutions or hacks:

    Let me explain on excerpts from the code.

    • This part of the code for the version of 0.6.1 that there is in pip: https://github.com/PyCQA/redbaron/blob/0.6.1/redbaron/base_nodes.py#L1591
    • This part of the code for the version of master branch in git: https://github.com/PyCQA/redbaron/blob/master/redbaron/base_nodes.py#L1642

    In master branch problem does not appear, but the bug is not solved. It's not appearing because last changes have double-negative(problem's not in lazy or unlazy logging function):

    • if RedBaron run under debug then in __repr__ method of Node class condition if in_a_shell() is True and dataflow is going down into __str__().

    That's true for the python shell and ipython shell too. And that's why a bug is not occurring in the shell.

    • If RedBaron is run without debug mode and not in shells then bug doesn't occur because in log function redbaron.DEBUG parameter is False. And repr function is not called. String representation of the object wouldn't be called.

    This right for the master branch. But for 0.6.1 it's not true.

    Look carefully at this code in 0.6.1 and in master branch: the difference is in one comma in call of log function. And that's why in 0.6.1 python tries to get repr of the object before a call of a log function. The cost of this bug is in one comma you think=) but it is actually still deeper, check.

    This problem occurs in 0.6.1 because in https://github.com/PyCQA/redbaron/blob/0.6.1/redbaron/base_nodes.py#L1591 python tries to get a string before it calls a log function. It calls a __repr__ function implicitly. In __repr__ function in_a_shell() returns False for Pycharm or something else that's not a python shell or ipython shell.

    Next, it's trying to get a path : https://github.com/PyCQA/redbaron/blob/0.6.1/redbaron/base_nodes.py#L951, and after a few intermediate calls stops here: https://github.com/PyCQA/redbaron/blob/master/redbaron/base_nodes.py#L121 , to get an index of the node in the parent, but a tree has not synchronized yet with last changes: https://github.com/PyCQA/redbaron/blob/0.6.1/redbaron/base_nodes.py#L1333. Look, here we have already done insertion of the code. Node.data includes insertion, but node.node_list does not. Parent points to node_list, but node_list doesn't have a new insertion. Index method raises the exception - ValueError(https://docs.python.org/2/tutorial/datastructures.html), because there is no such item.

    Excerpt from https://github.com/PyCQA/redbaron/blob/master/redbaron/base_nodes.py#L121: python pos = parent.index(node.node_list if isinstance(node, ProxyList) else node) for parent.index(node) raises a ValueError, message of this error calls implicitly __repr__ again to get a representation of the object, I mean, this call is being called again and would try to get path and index again. Because the Tree is not being synchronized before a log call. Yeap, Pycharm(py.test) tells about it =)

    • First my solution: write something like this into https://github.com/PyCQA/redbaron/blob/master/redbaron/utils.py#L33:
        if os.getenv('PYCHARM_HOSTED'):
            if os.getenv('IPYTHONENABLE'):
                print("IPYTHONENABLE %s" % os.getenv('IPYTHONENABLE'))
            print("PYCHARM_HOSTED : {0}".format(os.getenv('PYCHARM_HOSTED')))
        if not os.getenv('PYCHARM_HOSTED'):
            if __IPYTHON__:
                print("IPYTHOSHELL %s" % __IPYTHON__)
    

    I tested this code and that's why it looks how it looks to avoid exceptions and misunderstanding. But it is a hack. And maybe is not useful.

    • Second solution: add try, except block into https://github.com/PyCQA/redbaron/blob/master/redbaron/base_nodes.py#L121:
    if isinstance(parent, NodeList):
         try:
                pos = parent.index(node.node_list if isinstance(node, ProxyList) else node)
         except Exception:
                return 0
                return pos
    

    In this part, we catch exception for ValueError in UserList or another unknown exception(UserList.index raises ValueError) This is the hack too and code is not clean and clear(IMHO). And this ain't cool.

    • Third solution: Delete all log calls in https://github.com/PyCQA/redbaron/blob/master/redbaron/base_nodes.py#L1642. But error occurs in another place. or rewrite log function... But it has already done because (you)@Psycojoker changed a log function call that's satisfied declaration of this method. I described it above.
    • Fourth solution(this one, I think, fixes a problem and it is not a hack):
    #  first test, test_insert(DEBUG=FALSE) failed, changes have not synchronized yet
    
    def _synchronise(self):
        # TODO(alekum): log method calls __repr__ implicitly to get a self.data representation
        log("Before synchronise, self.data = '%s' + '%s'" % (self.first_blank_lines, self.data))
        #log("Before synchronise, self.data = '%s' + '%s'", self.first_blank_lines, self.data)
        super(LineProxyList, self)._synchronise()
        #log("After synchronise, self.data = '%s' + '%s'" % (self.first_blank_lines, self.data))
        log("After synchronise, self.data = '%s' + '%s'", self.first_blank_lines, self.data)
    
    # second test, test_insert(DEBUG=FALSE) passed, changes have synchronized already
    
    def _synchronise(self):
        # TODO(alekum): log method calls __repr__ implicitly to get a self.data representation
        #log("Before synchronise, self.data = '%s' + '%s'" % (self.first_blank_lines, self.data))
        log("Before synchronise, self.data = '%s' + '%s'", self.first_blank_lines, self.data)
        super(LineProxyList, self)._synchronise()
        log("After synchronise, self.data = '%s' + '%s'" % (self.first_blank_lines, self.data))
        #log("After synchronise, self.data = '%s' + '%s'", self.first_blank_lines, self.data)
    
    # The solution(DEBUG=FALSE), that I've described: actual state of a node is in node_list.
    
    def _synchronise(self):
        # TODO(alekum): log method calls __repr__ implicitly to get a self.data representation
        log("Before synchronise, self.data = '%s' + '%s'" % (self.first_blank_lines, self.node_list))
        #log("Before synchronise, self.data = '%s' + '%s'", self.first_blank_lines, self.data)
        super(LineProxyList, self)._synchronise()
        log("After synchronise, self.data = '%s' + '%s'" % (self.first_blank_lines, self.node_list))
        #log("After synchronise, self.data = '%s' + '%s'", self.first_blank_lines, self.data)
    

    I've tested it with master and 0.6.1, all is good. but I am not sure that it is a right and idiomatic way. It doesn't matter that in master I've changed this call. Do you know any cases where this solution could raise an AttributeError because self doesn't have attribute node_list? I couldn't find.

    opened by rojaster 10
  • Can't use redbaron in IPython notebook

    Can't use redbaron in IPython notebook

    redbaron calls fileno() on sys.stdout which raises unhandled UnsupportedOperation.

    ---------------------------------------------------------------------------
    UnsupportedOperation                      Traceback (most recent call last)
    <ipython-input-10-7a726da5bd0a> in <module>()
    ----> 1 str(tree)
    
    C:\Python2.7\lib\site-packages\redbaron.pyc in __str__(self)
       1270         to_return = ""
       1271         for number, value in enumerate(self.data):
    -> 1272             to_return += ("%-3s " % number) + "\n    ".join(value.__repr__().split("\n"))
       1273             to_return += "\n"
       1274         return to_return
    
    C:\Python2.7\lib\site-packages\redbaron.pyc in __repr__(self)
        852     def __repr__(self):
        853         # the isinstance here is for building sphinx doc
    --> 854         if isinstance(sys.stdout, StringIO) or os.isatty(sys.stdout.fileno()):
        855             return self.__str__()
        856 
    
    C:\Python2.7\lib\site-packages\IPython\kernel\zmq\iostream.pyc in fileno(self)
        192 
        193     def fileno(self):
    --> 194         raise UnsupportedOperation("IOStream has no fileno.")
        195 
        196     def write(self, string):
    
    UnsupportedOperation: IOStream has no fileno.
    

    http://nbviewer.ipython.org/gist/remram44/449041cd67ad390037b8

    opened by remram44 9
  • Append call argument breaks help()

    Append call argument breaks help()

    Test:

    code = 'a(b)'
    red = RedBaron(code)[0]
    red.call.append('c')
    red.help(True)
    
    
    

    Result:

    /anaconda3/lib/python3.5/site-packages/redbaron/base_nodes.py in help(self, deep, with_formatting)
        906     def help(self, deep=2, with_formatting=False):
        907         if runned_from_ipython():
    --> 908             sys.stdout.write(help_highlight(self.__help__(deep=deep, with_formatting=with_formatting) + "\n"))
        909         else:
        910             sys.stdout.write(self.__help__(deep=deep, with_formatting=with_formatting) + "\n")
    
    /anaconda3/lib/python3.5/site-packages/redbaron/base_nodes.py in __help__(self, deep, with_formatting)
        930                 to_join.append(("%s ->" % key))
        931                 for i in getattr(self, key):
    --> 932                     to_join.append("  * " + indent(i.__help__(deep=new_deep, with_formatting=with_formatting), "      ").lstrip())
        933 
        934         if deep and with_formatting:
    
    /anaconda3/lib/python3.5/site-packages/redbaron/base_nodes.py in __help__(self, deep, with_formatting)
        930                 to_join.append(("%s ->" % key))
        931                 for i in getattr(self, key):
    --> 932                     to_join.append("  * " + indent(i.__help__(deep=new_deep, with_formatting=with_formatting), "      ").lstrip())
        933 
        934         if deep and with_formatting:
    
    /anaconda3/lib/python3.5/site-packages/redbaron/base_nodes.py in __help__(self, deep, with_formatting)
        919         else:
        920             to_join.append("# identifiers: %s" % ", ".join(self.generate_identifiers()))
    --> 921             if self._get_helpers():
        922                 to_join.append("# helpers: %s" % ", ".join(self._get_helpers()))
        923             if self._default_test_value != "value":
    
    /anaconda3/lib/python3.5/site-packages/redbaron/base_nodes.py in _get_helpers(self)
        882             'generate_identifiers',
        883         ])
    --> 884         return [x for x in dir(self) if not x.startswith("_") and x not in not_helpers and inspect.ismethod(getattr(self, x))]
        885 
        886     def fst(self):
    
    /anaconda3/lib/python3.5/site-packages/redbaron/base_nodes.py in <listcomp>(.0)
        882             'generate_identifiers',
        883         ])
    --> 884         return [x for x in dir(self) if not x.startswith("_") and x not in not_helpers and inspect.ismethod(getattr(self, x))]
        885 
        886     def fst(self):
    
    /anaconda3/lib/python3.5/site-packages/redbaron/base_nodes.py in __getattr__(self, key)
        678 
        679         if key not in redbaron.ALL_IDENTIFIERS:
    --> 680             raise AttributeError("%s instance has no attribute '%s' and '%s' is not a valid identifier of another node" % (self.__class__.__name__, key, key))
        681 
        682         return self.find(key)
    
    AttributeError: CallArgumentNode instance has no attribute 'next' and 'next' is not a valid identifier of another node
    

    Note that the new argument itself is added correctly:

    In [216]: red
    Out[216]: a(b, c)
    
    opened by gasparka 8
  • redbaron.base_nodes.NodeList.insert() raises ValueError (on in_a_shell() is False)

    redbaron.base_nodes.NodeList.insert() raises ValueError (on in_a_shell() is False)

    For Example.

    # this is OK (because in_a_shell() returns True, so __str__() is called on __repr__())
    $ python myscript.py
    # but this is NG. (because in_a_shel() returns False)
    $ python myscript.py > output.py
    

    Actually, after changing such as below. some tests are failed.

    diff --git a/redbaron/utils.py b/redbaron/utils.py
    index 451675b..91c60b7 100644
    --- a/redbaron/utils.py
    +++ b/redbaron/utils.py
    @@ -31,6 +31,7 @@ def log(string, *args):
    
    
     def in_a_shell():
    +    return False
         # the isinstance here is for building sphinx doc
         if redbaron.DEBUG or isinstance(sys.stdout, StringIO):
             return True
    
    $ py.test tests
    # ... snip
    ======================================= 62 failed, 1346 passed in 42.57 seconds =======================================
    
    opened by podhmo 7
  • 'utils.in_a_shell()' : stack overflow.

    'utils.in_a_shell()' : stack overflow.

    Happens on Pycharm 2016.2.2, works fine on console. Version: 0.6.1 (latest from pip) Example:

    from redbaron import RedBaron
    import textwrap
    code = textwrap.dedent("""\
        def a():
            pass""")
    red = RedBaron(code)[0]
    red.value.insert(0, 'a') # <- problem here
    

    Result:

    Fatal Python error: Cannot recover from stack overflow.
    
    Current thread 0x00007fcd31630700 (most recent call first):
      File "python3.5/site-packages/redbaron/utils.py", line 38 in in_a_shell
      File "python3.5/site-packages/redbaron/base_nodes.py", line 946 in __repr__
      File "python3.5/collections/__init__.py", line 1081 in index
      File "python3.5/site-packages/redbaron/base_nodes.py", line 119 in get_holder_on_attribute
      File "python3.5/site-packages/redbaron/base_nodes.py", line 50 in __init__
      File "python3.5/site-packages/redbaron/base_nodes.py", line 840 in path
      File "python3.5/site-packages/redbaron/base_nodes.py", line 951 in __repr__
      File "python3.5/collections/__init__.py", line 1081 in index
      File "python3.5/site-packages/redbaron/base_nodes.py", line 119 in get_holder_on_attribute
      File "python3.5/site-packages/redbaron/base_nodes.py", line 50 in __init__
      File "python3.5/site-packages/redbaron/base_nodes.py", line 840 in path
      File "python3.5/site-packages/redbaron/base_nodes.py", line 951 in __repr__
    

    I have currently set 'utils.in_a_shell()' default return to True instead of False. This fixes the problem but i am not sure if this is a correct way to do it.

    Thanks for the good work :)

    opened by gasparka 7
  • Cannot modify whitespace from function declarations

    Cannot modify whitespace from function declarations

    Hello,

    having a function:

    def f( a, b ): pass

    I would like to set third_formatting and fourth_formatting on it, but it says "unhandled case", when I set this to "". My intent is to write code, which automatically formats function definition arguments to contain new-lines as necessary only, and no leading or trailing space for arguments.

    Can you please add support for these.

    opened by kayhayen 5
  • Cannot modify spaces after trailing comma

    Cannot modify spaces after trailing comma

    The following code:

    (self, )

    Note the trailing space after the comma. I was writing a modifier that removes trailing and leading spaces. And came up with this:

    if "\n" not in tuple_node.dumps():
        tuple_node.second_formatting = ""
        tuple_node.third_formatting = ""
    
        for argument in tuple_node.value:
            if argument.type in ("string", "binary_string"):
                argument.second_formatting = ""
    

    Note, that I find it strange, that I need to drop leading spaces from contained strings (same for call nodes, strings seem to attract whitespace for no good reason, where I would naturally attribute it to the outside). I tried setting all four formattings to "x#", and this resulted a string containing them, plus still a space after the comma. Where is it hiding? The name typed argument doesn't indicate anything in its dict.

    opened by kayhayen 5
  • Can't find CommentNode via `find` method.

    Can't find CommentNode via `find` method.

    Env:

    Python 3.5.2 RedBaron 0.6.1

    Example:

    r = RedBaron("""
    class X:
        # test
        def __init__(self): 
            pass
    """)
    r.find('comment') # returns None
    r.find_all('comment') # returns NodeList with len == 1
    

    Expected behavior:

    r.find('comment') - should return node with comment

    opened by gtors 4
  • [Question, feature request?] After modifying the tree, how to print elements back ?

    [Question, feature request?] After modifying the tree, how to print elements back ?

    Hi, So I'm using RedBaron to help me edit code, that's pretty exciting. My difficulty now is to re-write the tree IĀ modified, sometimes just nodes. It doesn't seem to be a method on redbaron objects for that.

    For instance, let's take the code

    def foo(self, key=[], second): pass
    

    I parse it, I sort the arguments and IĀ want to print them back. It's easy with simple arguments that have no value, but with "key=[]" IĀ don't find its textual representation, I find its "first_formatting" that is a python list.

    {'first_formatting': [], 'type': 'def_argument', 'target': {'type': 'name', 'value': 'key'}, 'value': {'type': 'name', 'value': 'val'}, 'second_formatting': []}
    

    So IĀ could treat this "[]" as a special case and return the string "[]", but how will IĀ handle a list with something in it, or a crazy argument with nested lists ?

    Hope IĀ was clear; thanks !

    opened by vindarel 4
  • Can't get return in assignments

    Can't get return in assignments

    Here is code"

    a = \
        2
    

    It is impossible to get return. Here is what I am doing:

    red = RedBaron(_source_)
    assignments = red.find_all('AssignmentNode')
    for ind, body_line in enumerate(assignments.data):
        if body_line.second_formatting.data[0].value == '\n':
            print True
    

    It doesn't print True. Is it bug or I am wrong? How can I detect that assignment is defined in a separated lines?

    opened by b5y 4
  • Unsupported positional-only argument syntax

    Unsupported positional-only argument syntax

    This syntax works fine in python:

    def foo(a, /, b): pass
    

    but redbaron fails to parse it:

    >>> redbaron.RedBaron("def foo(a, /, b): pass")
    Traceback (most recent call last):
      File "/usr/lib/python3/dist-packages/baron/baron.py", line 20, in _parse
        return parser(tokens)
      File "/usr/lib/python3/dist-packages/baron/grammator.py", line 836, in parse
        return parser.parse(iter(tokens))
      File "/usr/lib/python3/dist-packages/baron/parser.py", line 165, in parse
        raise ParsingError(debug_output)
    baron.parser.ParsingError: Error, got an unexpected token SLASH here:
    
       1 def foo(a, /<---- here
    
    The token SLASH should be one of those: COMMA, DOUBLE_STAR, LEFT_PARENTHESIS, NAME, RIGHT_PARENTHESIS, STAR
    
    Baron has failed to parse this input. If this is valid python code (and by that I mean that the python binary successfully parse this code without any syntax error) (also consider that baron does not yet parse python 3 code integrally) it would be kind if you can extract a snippet of your code that make Baron fails and open a bug here: https://github.com/PyCQA/baron/issues
    
    Sorry for the inconvenience.
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
      File "/usr/lib/python3/dist-packages/redbaron/redbaron.py", line 36, in __init__
        self.node_list = base_nodes.NodeList.from_fst(baron.parse(source_code), parent=self, on_attribute="root")
      File "/usr/lib/python3/dist-packages/baron/baron.py", line 57, in parse
        to_return = _parse(tokens, print_function)
      File "/usr/lib/python3/dist-packages/baron/baron.py", line 24, in _parse
        return parser(tokens)
      File "/usr/lib/python3/dist-packages/baron/grammator.py", line 836, in parse
        return parser.parse(iter(tokens))
      File "/usr/lib/python3/dist-packages/baron/parser.py", line 165, in parse
        raise ParsingError(debug_output)
    baron.parser.ParsingError: Error, got an unexpected token SLASH here:
    
       1 def foo(a, /<---- here
    
    The token SLASH should be one of those: COMMA, DOUBLE_STAR, LEFT_PARENTHESIS, NAME, RIGHT_PARENTHESIS, STAR
    
    Baron has failed to parse this input. If this is valid python code (and by that I mean that the python binary successfully parse this code without any syntax error) (also consider that baron does not yet parse python 3 code integrally) it would be kind if you can extract a snippet of your code that make Baron fails and open a bug here: https://github.com/PyCQA/baron/issues
    
    Sorry for the inconvenience.
    >>> 
    
    opened by hydrargyrum 0
  • support for context manager tuple

    support for context manager tuple

    with (context1 as c1, context2 as c2) does not work.

    >>> import redbaron
    >>> redbaron.RedBaron('''
    ... with (open("a.txt", "w") as a, open("b.txt", "w") as b):
    ...     print(1)
    ... ''')
    Traceback (most recent call last):
      File "~/.asdf/installs/python/3.10.7/lib/python3.10/site-packages/baron/baron.py", line 20, in _parse
        return parser(tokens)
      File "~/.asdf/installs/python/3.10.7/lib/python3.10/site-packages/baron/grammator.py", line 836, in parse
        return parser.parse(iter(tokens))
      File "~/.asdf/installs/python/3.10.7/lib/python3.10/site-packages/baron/parser.py", line 165, in parse
        raise ParsingError(debug_output)
    baron.parser.ParsingError: Error, got an unexpected token AS here:
    
       1
       2 with (open("a.txt", "w") as <---- here
    
    The token AS should be one of those: COMMA, FOR, RIGHT_PARENTHESIS
    
    Baron has failed to parse this input. If this is valid python code (and by that I mean that the python binary successfully parse this code without any syntax error) (also consider that baron does not yet parse python 3 code integrally) it would be kind if you can extract a snippet of your code that make Baron fails and open a bug here: https://github.com/PyCQA/baron/issues
    
    Sorry for the inconvenience.
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
      File "~/.asdf/installs/python/3.10.7/lib/python3.10/site-packages/redbaron/redbaron.py", line 36, in __init__
        self.node_list = base_nodes.NodeList.from_fst(baron.parse(source_code), parent=self, on_attribute="root")
      File "~/.asdf/installs/python/3.10.7/lib/python3.10/site-packages/baron/baron.py", line 66, in parse
        return _parse(tokens, print_function)
      File "~/.asdf/installs/python/3.10.7/lib/python3.10/site-packages/baron/baron.py", line 24, in _parse
        return parser(tokens)
      File "~/.asdf/installs/python/3.10.7/lib/python3.10/site-packages/baron/grammator.py", line 836, in parse
        return parser.parse(iter(tokens))
      File "~/.asdf/installs/python/3.10.7/lib/python3.10/site-packages/baron/parser.py", line 165, in parse
        raise ParsingError(debug_output)
    baron.parser.ParsingError: Error, got an unexpected token AS here:
    
       1
       2 with (open("a.txt", "w") as <---- here
    
    The token AS should be one of those: COMMA, FOR, RIGHT_PARENTHESIS
    
    Baron has failed to parse this input. If this is valid python code (and by that I mean that the python binary successfully parse this code without any syntax error) (also consider that baron does not yet parse python 3 code integrally) it would be kind if you can extract a snippet of your code that make Baron fails and open a bug here: https://github.com/PyCQA/baron/issues
    
    Sorry for the inconvenience.
    
    opened by elijahr 0
  • support for walrus operator

    support for walrus operator

    The walrus operator which was added in Python 3.8 (:=) is not supported:

    >>> import redbaron
    >>> redbaron.RedBaron('''
    ... if foo := True:
    ...     print(foo)
    ... ''')
    Traceback (most recent call last):
      File "~/.asdf/installs/python/3.10.7/lib/python3.10/site-packages/baron/baron.py", line 20, in _parse
        return parser(tokens)
      File "~/.asdf/installs/python/3.10.7/lib/python3.10/site-packages/baron/grammator.py", line 836, in parse
        return parser.parse(iter(tokens))
      File "~/.asdf/installs/python/3.10.7/lib/python3.10/site-packages/baron/parser.py", line 165, in parse
        raise ParsingError(debug_output)
    baron.parser.ParsingError: Error, got an unexpected token EQUAL here:
    
       1
       2 if foo := <---- here
    
    The token EQUAL should be one of those: ASSERT, BACKQUOTE, BINARY, BINARY_RAW_STRING, BINARY_STRING, BREAK, COMMENT, COMPLEX, CONTINUE, DEL, ELLIPSIS, ENDL, EXEC, FLOAT, FLOAT_EXPONANT, FLOAT_EXPONANT_COMPLEX, FROM, GLOBAL, HEXA, IMPORT, INT, INTERPOLATED_RAW_STRING, INTERPOLATED_STRING, LAMBDA, LEFT_BRACKET, LEFT_PARENTHESIS, LEFT_SQUARE_BRACKET, LONG, MINUS, NAME, NONLOCAL, NOT, OCTA, PASS, PLUS, PRINT, RAISE, RAW_STRING, RETURN, STAR, STRING, TILDE, UNICODE_RAW_STRING, UNICODE_STRING, YIELD
    
    Baron has failed to parse this input. If this is valid python code (and by that I mean that the python binary successfully parse this code without any syntax error) (also consider that baron does not yet parse python 3 code integrally) it would be kind if you can extract a snippet of your code that make Baron fails and open a bug here: https://github.com/PyCQA/baron/issues
    
    Sorry for the inconvenience.
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
      File "~/.asdf/installs/python/3.10.7/lib/python3.10/site-packages/redbaron/redbaron.py", line 36, in __init__
        self.node_list = base_nodes.NodeList.from_fst(baron.parse(source_code), parent=self, on_attribute="root")
      File "~/.asdf/installs/python/3.10.7/lib/python3.10/site-packages/baron/baron.py", line 66, in parse
        return _parse(tokens, print_function)
      File "~/.asdf/installs/python/3.10.7/lib/python3.10/site-packages/baron/baron.py", line 24, in _parse
        return parser(tokens)
      File "~/.asdf/installs/python/3.10.7/lib/python3.10/site-packages/baron/grammator.py", line 836, in parse
        return parser.parse(iter(tokens))
      File "~/.asdf/installs/python/3.10.7/lib/python3.10/site-packages/baron/parser.py", line 165, in parse
        raise ParsingError(debug_output)
    baron.parser.ParsingError: Error, got an unexpected token EQUAL here:
    
       1
       2 if foo := <---- here
    
    The token EQUAL should be one of those: ASSERT, BACKQUOTE, BINARY, BINARY_RAW_STRING, BINARY_STRING, BREAK, COMMENT, COMPLEX, CONTINUE, DEL, ELLIPSIS, ENDL, FLOAT, FLOAT_EXPONANT, FLOAT_EXPONANT_COMPLEX, FROM, GLOBAL, HEXA, IMPORT, INT, INTERPOLATED_RAW_STRING, INTERPOLATED_STRING, LAMBDA, LEFT_BRACKET, LEFT_PARENTHESIS, LEFT_SQUARE_BRACKET, LONG, MINUS, NAME, NONLOCAL, NOT, OCTA, PASS, PLUS, RAISE, RAW_STRING, RETURN, STAR, STRING, TILDE, UNICODE_RAW_STRING, UNICODE_STRING, YIELD
    
    Baron has failed to parse this input. If this is valid python code (and by that I mean that the python binary successfully parse this code without any syntax error) (also consider that baron does not yet parse python 3 code integrally) it would be kind if you can extract a snippet of your code that make Baron fails and open a bug here: https://github.com/PyCQA/baron/issues
    
    Sorry for the inconvenience.
    
    opened by elijahr 0
  • bug: Unpacking operator `*` used in list gives an error

    bug: Unpacking operator `*` used in list gives an error

    Version: 0.9.2

    Test Case:

    from redbaron import RedBaron
    RedBaron("[*[1,2,3]]")]
    

    gives the following error:

    ParsingError: Error, got an unexpected token STAR here:
    
       1 [*<---- here
    
    The token STAR should be one of those: BACKQUOTE, BINARY, BINARY_RAW_STRING, BINARY_STRING, COMPLEX, ELLIPSIS, FLOAT, FLOAT_EXPONANT, FLOAT_EXPONANT_COMPLEX, HEXA, INT, INTERPOLATED_RAW_STRING, INTERPOLATED_STRING, LAMBDA, LEFT_BRACKET, LEFT_PARENTHESIS, LEFT_SQUARE_BRACKET, LONG, MINUS, NAME, NOT, OCTA, PLUS, RAW_STRING, RIGHT_SQUARE_BRACKET, STRING, TILDE, UNICODE_RAW_STRING, UNICODE_STRING
    
    Baron has failed to parse this input. If this is valid python code (and by that I mean that the python binary successfully parse this code without any syntax error) (also consider that baron does not yet parse python 3 code integrally) it would be kind if you can extract a snippet of your code that make Baron fails and open a bug here: https://github.com/PyCQA/baron/issues
    
    Sorry for the inconvenience.
    
    opened by GindaChen 0
  • py.test tests failing

    py.test tests failing

    $ git clone ...
    
    $ py.test tests
    ================================================================================ test session starts =================================================================================
    platform linux -- Python 3.9.5, pytest-7.1.2, pluggy-0.13.0
    rootdir: /mnt/C/project/contrib/redbaron
    collected 1431 items / 1 error                                                                                                                                                       
    
    ======================================================================================= ERRORS =======================================================================================
    ____________________________________________________________________ ERROR collecting tests/test_bounding_box.py _____________________________________________________________________
    Fixture "red" called directly. Fixtures are not meant to be called directly,
    but are created automatically when test functions request them as parameters.
    See https://docs.pytest.org/en/stable/explanation/fixtures.html for more information about fixtures, and
    https://docs.pytest.org/en/stable/deprecations.html#calling-fixtures-directly about how to update your code.
    ============================================================================== short test summary info ===============================================================================
    ERROR tests/test_bounding_box.py
    !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
    ================================================================================== 1 error in 0.40s ==================================================================================
    
    
    opened by yilabs 0
  • RedBaron confuse relative path to ellipsis in from_imports

    RedBaron confuse relative path to ellipsis in from_imports

    Hi !

    The following code:

    import redbaron
    import json
    print(json.dumps(redbaron.RedBaron('from ...util import properties').fst(), indent=4))
    

    returns:

    [
        {
            "type": "from_import",
            "first_formatting": [
                {
                    "type": "space",
                    "value": " "
                }
            ],
            "value": [
                {
                    "type": "ellipsis",
                    "first_formatting": [],
                    "second_formatting": []
                },
                {
                    "type": "name",
                    "value": "util"
                }
            ],
            "second_formatting": [
                {
                    "type": "space",
                    "value": " "
                }
            ],
            "third_formatting": [
                {
                    "type": "space",
                    "value": " "
                }
            ],
            "targets": [
                {
                    "type": "name_as_name",
                    "value": "properties",
                    "target": "",
                    "first_formatting": [],
                    "second_formatting": []
                }
            ]
        }
    ]
    

    Where we can clearly see the ellipsis being parsed from the from_import line.

    To me, this seems to be a misparse.

    Note that increasing the dot number will still yield an ellispsis. To get N ellispsis, provide N*3 dots.

    opened by Aluriak 0
Releases(0.9.2)
  • 0.9.2(Mar 17, 2019)

  • 0.9.1(Feb 25, 2019)

  • 0.9(Feb 25, 2019)

    • full python 3.7 syntax support

    • BREAKING CHANGE: annotations are now member of {Def,List,Dict}Argument to flatten the data structure, TypedNameNode doesn't exist anymore

    • add support for return annotation by bram

    • add support for variable annotation https://github.com/PyCQA/redbaron/pull/177 by scottbelden and additional work by bram

    • add support for raise from by bram

    Source code(tar.gz)
    Source code(zip)
  • 0.8(Feb 25, 2019)

    • add support to typed function arguments https://github.com/PyCQA/redbaron/pull/168 Scott Belden and additional work by bram
    • add support to set async attributes to with/for nodes, was missing in previous release
    Source code(tar.gz)
    Source code(zip)
  • 0.7.1(Feb 25, 2019)

    • fix to_python() when strings are indented by stripping whitespace before evaluating by duncf https://github.com/PyCQA/redbaron/pull/140

    Python 3 support (based on work in Baron):

    • support ellipsis
    • support matrix operator
    • support f-strings
    • support numeric literals
    • support nonlocal statement
    • support keyword only markers
    • support yield from statement
    • support async/await statements
    Source code(tar.gz)
    Source code(zip)
  • 0.7(Aug 21, 2018)

    • fix to_python() when strings are indented by stripping whitespace before evaluating by duncf https://github.com/PyCQA/redbaron/pull/140

    Python 3 support (based on work in Baron):

    • support ellipsis
    • support matrix operator
    • support f-strings
    • support numeric literals
    • support nonlocal statement
    • support keyword only markers
    • support yield from statement
    • support async/await statements
    Source code(tar.gz)
    Source code(zip)
Owner
Python Code Quality Authority
Organization for code quality tools (and plugins) for the Python programming language
Python Code Quality Authority
a python refactoring library

rope, a python refactoring library ... Overview Rope is a python refactoring library. Notes Nick Smith <[email protected]> takes over maintaining rope

null 1.5k Dec 30, 2022
Safe code refactoring for modern Python.

Safe code refactoring for modern Python projects. Overview Bowler is a refactoring tool for manipulating Python at the syntax tree level. It enables s

Facebook Incubator 1.4k Jan 4, 2023
AST based refactoring tool for Python.

breakfast AST based refactoring tool. (Very early days, not usable yet.) Why 'breakfast'? I don't know about the most important, but it's a good meal.

eric casteleijn 0 Feb 22, 2022
Refactoring Python Applications for Simplicity

Python Refactoring Refactoring Python Applications for Simplicity. You can open and read project files or use this summary ?? Concatenate String my_st

Mohammad Dori 3 Jul 15, 2022
IDE allow you to refactor code, Baron allows you to write refactoring code.

Introduction Baron is a Full Syntax Tree (FST) library for Python. By opposition to an AST which drops some syntax information in the process of its c

Python Code Quality Authority 278 Dec 29, 2022
Simple, hassle-free, dependency-free, AST based source code refactoring toolkit.

refactor is an end-to-end refactoring framework that is built on top of the 'simple but effective refactorings' assumption. It is much easier to write a simple script with it rather than trying to figure out what sort of a regex you need in order to replace a pattern (if it is even matchable with regexes).

Batuhan Taskaya 385 Jan 6, 2023
A simple Python bytecode framework in pure Python

A simple Python bytecode framework in pure Python

null 3 Jan 23, 2022
Find dead Python code

Vulture - Find dead code Vulture finds unused code in Python programs. This is useful for cleaning up and finding errors in large code bases. If you r

Jendrik Seipp 2.4k Dec 27, 2022
A system for Python that generates static type annotations by collecting runtime types

MonkeyType MonkeyType collects runtime types of function arguments and return values, and can automatically generate stub files or even add draft type

Instagram 4.1k Dec 28, 2022
Programmatically edit text files with Python. Useful for source to source transformations.

massedit formerly known as Python Mass Editor Implements a python mass editor to process text files using Python code. The modification(s) is (are) sh

null 106 Dec 17, 2022
Tool for translation type comments to type annotations in Python

com2ann Tool for translation of type comments to type annotations in Python. The tool requires Python 3.8 to run. But the supported target code versio

Ivan Levkivskyi 123 Nov 12, 2022
Removes commented-out code from Python files

eradicate eradicate removes commented-out code from Python files. Introduction With modern revision control available, there is no reason to save comm

Steven Myint 146 Dec 13, 2022
A library that modifies python source code to conform to pep8.

Pep8ify: Clean your code with ease Pep8ify is a library that modifies python source code to conform to pep8. Installation This library currently works

Steve Pulec 117 Jan 3, 2023
Code generation and code search for Python and Javascript.

Codeon Code generation and code search for Python and Javascript. Similar to GitHub Copilot with one major difference: Code search is leveraged to mak

null 51 Dec 8, 2022
Leap is an experimental package written to enable the utilization of C-like goto statements in Python functions

Leap is an experimental package written to enable the utilization of C-like goto statements in Python functions

null 6 Dec 26, 2022
Turn your C++/Java code into a Python-like format for extra style points and to make everyone hates you

Turn your C++/Java code into a Python-like format for extra style points and to make everyone hates you

TĆ“ Đį»©c (Watson) 4 Feb 7, 2022
Simple Pose: Rethinking and Improving a Bottom-up Approach for Multi-Person Pose Estimation

SimplePose Code and pre-trained models for our paper, ā€œSimple Pose: Rethinking and Improving a Bottom-up Approach for Multi-Person Pose Estimationā€, a

Jia Li 256 Dec 24, 2022
An attempt at the implementation of Glom, Geoffrey Hinton's new idea that integrates neural fields, predictive coding, top-down-bottom-up, and attention (consensus between columns)

GLOM - Pytorch (wip) An attempt at the implementation of Glom, Geoffrey Hinton's new idea that integrates neural fields, predictive coding,

Phil Wang 173 Dec 14, 2022
Bottom-up Human Pose Estimation

Introduction This is the official code of Rethinking the Heatmap Regression for Bottom-up Human Pose Estimation. This paper has been accepted to CVPR2

null 108 Dec 1, 2022
This is an official implementation of our CVPR 2021 paper "Bottom-Up Human Pose Estimation Via Disentangled Keypoint Regression" (https://arxiv.org/abs/2104.02300)

Bottom-Up Human Pose Estimation Via Disentangled Keypoint Regression Introduction In this paper, we are interested in the bottom-up paradigm of estima

HRNet 367 Dec 27, 2022