Latest minimal example: compile('A.\u018a\\ ', '<string>', 'single')

Hi! We are using hypothesmith to test our wemake-python-styleguide linter. And today I got this failure:

self = <hypothesmith.syntactic.GrammarStrategy object at 0x7f702a236ee0>
data = ConjectureData(VALID, 1628 bytes, frozen)
symbol = NonTerminal('simple_stmt')
draw_state = DrawState(result=['\r\n \t\t\t \t \t\n #²\x97\x9c\n\t\r\n \t \t#\U000cf1a7`\x1e£Í\x0b#Ã\U0009460b\U00041f90\U000b0ffc§...b6¯\n#r×í@d¿\x08\r\n  \t\t\t\t\t\t \t\t\t  \t\t \t\t\t\t#\U000dca7e\U000cad5c?\U0005caae¤ÂÁ0Z\U000aa6eap#\U0010c8a6{'])

    def draw_symbol(self, data, symbol, draw_state):  # type: ignore
        count = len(draw_state.result)
        super().draw_symbol(data, symbol, draw_state)
        if symbol.name in COMPILE_MODES:
            try:
                compile(
                    source="".join(draw_state.result[count:]),
                    filename="<string>",
                    mode=COMPILE_MODES[symbol.name],
                )
            except SystemError as err:  # pragma: no cover
                # Extra output to help track down a possible upstream issue
                # https://github.com/Zac-HD/stdlib-property-tests/issues/14
                source_code = "".join(draw_state.result[count:])
>               raise Exception(
                    "unexpected error while attempting to compile "
                    f"{ascii(source_code)!r} in mode={COMPILE_MODES[symbol.name]}"
                ) from err
E               Exception: unexpected error while attempting to compile "'import\\u012c\\u0155u\\u011f\\u9782\\u0117\\xc4\\xd8.\\xcd\\u0179\\u0152\\u0139\\u010b\\xdb.\\xd0\\U0002852aD\\u017b\\xd6\\\\\\t \\t\\r\\nas#\\U0009a4a1\\u0178T\\u5408\\U00029fe8\\xcb\\u0141\\U0002b2b9\\u013c\\xf34D\\xd1p\\xf1\\t\\x0c \\x0c\\x0c \\x0c\\t\\x0c \\x0c\\t \\x0c\\x0c\\x0c \\\\\\x0c\\t\\n;pass\\r\\n\\t \\t\\t  \\r\\n\\r\\n\\t\\t\\t \\t\\t#\\x8b\\n\\r\\n  \\t\\t#\\xe6k\\x81E\\r\\n\\t  \\t \\t#\\U0001ed6b\\xd1#^\\r\\n \\r\\n  \\t\\r\\n  \\t\\t\\t #G\\x08\\x08#\\xe3^6\\xf1\\x87\\U000d8254\\U00070572 oQ}\\x10\\xce\\xc1\\U000563b6\\xaf\\n#r\\xd7\\xed@d\\xbf\\x08\\r\\n  \\t\\t\\t\\t\\t\\t \\t\\t\\t  \\t\\t \\t\\t\\t\\t#\\U000dca7e\\U000cad5c?\\U0005caae\\xa4\\xc2\\xc10Z\\U000aa6eap#\\U0010c8a6{'" in mode=single

.venv/lib/python3.9/site-packages/hypothesmith/syntactic.py:119: Exception
---------------------------------- Hypothesis ----------------------------------
You can add @seed(11935010665446469984514748066733166299) to this test or run pytest with --hypothesis-seed=11935010665446469984514748066733166299 to reproduce this failure.

Test file: https://github.com/wemake-services/wemake-python-styleguide/blob/master/tests/test_checker/test_hypothesis.py Version used: 0.1.5

Hi! My CI crashed with something rather interesting:

Link: https://github.com/wemake-services/wemake-python-styleguide/runs/4175506726?check_suite_focus=true

 =================================== FAILURES ===================================
______________________________ test_no_exceptions ______________________________

self = <hypothesmith.syntactic.GrammarStrategy object at 0x7f4930fc3e80>
data = ConjectureData(VALID, 28 bytes, frozen)
symbol = NonTerminal('simple_stmt')
draw_state = DrawState(result=['global', 'A', '\\ \n', '#\n'])

    def draw_symbol(self, data, symbol, draw_state):  # type: ignore
        count = len(draw_state.result)
        super().draw_symbol(data, symbol, draw_state)
        if symbol.name in COMPILE_MODES:
            try:
>               compile(
                    source="".join(draw_state.result[count:]),
                    filename="<string>",
                    mode=COMPILE_MODES[symbol.name],
                )
E               SystemError: Negative size passed to PyUnicode_New

.venv/lib/python3.9/site-packages/hypothesmith/syntactic.py:110: SystemError

The above exception was the direct cause of the following exception:

default_options = options(min_name_length=2, max_name_length=45, i_control_code=True, i_dont_control_code=True, max_noqa_comments=10, ne...gnitive_average=8, max_call_level=3, max_annotation_complexity=3, max_import_from_members=8, max_tuple_unpack_length=4)
parse_ast_tree = <function parse_ast_tree.<locals>.factory at 0x7f479cd01430>
parse_tokens = <function parse_tokens.<locals>.factory at 0x7f479ce863a0>

>   ???

tests/test_checker/test_hypothesis.py:40: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
.venv/lib/python3.9/site-packages/hypothesmith/syntactic.py:89: in do_draw
    result = super().do_draw(data)
.venv/lib/python3.9/site-packages/hypothesis/extra/lark.py:153: in do_draw
    self.draw_symbol(data, start, state)
.venv/lib/python3.9/site-packages/hypothesmith/syntactic.py:107: in draw_symbol
    super().draw_symbol(data, symbol, draw_state)
.venv/lib/python3.9/site-packages/hypothesis/extra/lark.py:181: in draw_symbol
    self.draw_symbol(data, e, draw_state)
.venv/lib/python3.9/site-packages/hypothesmith/syntactic.py:107: in draw_symbol
    super().draw_symbol(data, symbol, draw_state)
.venv/lib/python3.9/site-packages/hypothesis/extra/lark.py:181: in draw_symbol
    self.draw_symbol(data, e, draw_state)
.venv/lib/python3.9/site-packages/hypothesmith/syntactic.py:107: in draw_symbol
    super().draw_symbol(data, symbol, draw_state)
.venv/lib/python3.9/site-packages/hypothesis/extra/lark.py:181: in draw_symbol
    self.draw_symbol(data, e, draw_state)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <hypothesmith.syntactic.GrammarStrategy object at 0x7f4930fc3e80>
data = ConjectureData(VALID, 28 bytes, frozen)
symbol = NonTerminal('simple_stmt')
draw_state = DrawState(result=['global', 'A', '\\ \n', '#\n'])

    def draw_symbol(self, data, symbol, draw_state):  # type: ignore
        count = len(draw_state.result)
        super().draw_symbol(data, symbol, draw_state)
        if symbol.name in COMPILE_MODES:
            try:
                compile(
                    source="".join(draw_state.result[count:]),
                    filename="<string>",
                    mode=COMPILE_MODES[symbol.name],
                )
            except SyntaxError:
                # Python's grammar doesn't actually fully describe the behaviour of the
                # CPython parser and AST-post-processor, so we just filter out errors.
                assume(False)
            except Exception as err:  # pragma: no cover
                # Attempting to compile almost-valid strings has triggered a wide range
                # of bizzare errors in CPython, especially with the new PEG parser,
                # and so we maintain this extra clause to ensure that we get a decent
                # error message out of it.
                if isinstance(err, SystemError) and sys.version_info[:3] == (3, 9, 0):
                    # We've triggered https://bugs.python.org/issue42218 - it's been
                    # fixed upstream, so we'll treat it as if it were a SyntaxError.
                    assume(False)
                source_code = ascii("".join(draw_state.result[count:]))
>               raise type(err)(
                    f"compile({source_code}, '<string>', "
                    f"{COMPILE_MODES[symbol.name]!r}) "
                    f"raised {type(err).__name__}: {str(err)}"
                ) from err
E               SystemError: compile('globalA\\ \n#\n', '<string>', 'single') raised SystemError: Negative size passed to PyUnicode_New

.venv/lib/python3.9/site-packages/hypothesmith/syntactic.py:129: SystemError
---------------------------------- Hypothesis ----------------------------------
Highest target scores:
               6  (label='(hypothesmith) number of unique ast node types')
               9  (label='(hypothesmith) instructions in bytecode')
              13  (label='(hypothesmith) total number of ast nodes')


You can reproduce this example by temporarily adding @reproduce_failure('6.24.2', b'AXicY2RkYGBlABIgwMwIYgAJCB8AAUAAEQ==') as a decorator on your test case

Hi!

Thanks a lot for writing this, today I have stumbled upon this package in https://github.com/gforcada/flake8-builtins/pull/46 and then decided to implement the same check in my own project: https://github.com/wemake-services/wemake-python-styleguide/issues/1080

It is really useful for linters and code quality tools!

I am not sure if that's actually a bug or not, but it looks like the generated source code is not a valid python:

E         File "<unknown>", line 1
E           pass\;#
E                 ^
E       SyntaxError: unexpected character after line continuation character

I can get around this problem by rejecting code that is not valid:

try:
    compile(code_to_parse, '<filename>', 'exec') 
except SyntaxError:
    reject()

Repro:

1. create a clean virtualenv
2. pip install hypothesmith
3. open python prompt (3.10.8)
4. any of the following doesn't work:
   • from hypothesmith import from_grammar
   • from hypothesmith import from_node

Error:

$ python          
Python 3.10.8 (main, Nov  1 2022, 14:18:21) [GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from hypothesmith import grammar
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/tmp/test_hypothesmith/.virtualenv/test_hypothesmith-dgeh/lib/python3.10/site-packages/hypothesmith/__init__.py", line 3, in <module>
    from hypothesmith.cst import from_node
  File "/tmp/test_hypothesmith/.virtualenv/test_hypothesmith-dgeh/lib/python3.10/site-packages/hypothesmith/cst.py", line 24, in <module>
    from hypothesmith.syntactic import identifiers
  File "/tmp/test_hypothesmith/.virtualenv/test_hypothesmith-dgeh/lib/python3.10/site-packages/hypothesmith/syntactic.py", line 21, in <module>
    LARK_GRAMMAR = read_text("hypothesmith", "python.lark")
  File "/usr/lib/python3.10/importlib/resources.py", line 103, in read_text
    with open_text(package, resource, encoding, errors) as fp:
  File "/usr/lib/python3.10/importlib/resources.py", line 82, in open_text
    open_binary(package, resource), encoding=encoding, errors=errors
  File "/usr/lib/python3.10/importlib/resources.py", line 46, in open_binary
    return reader.open_resource(resource)
  File "/usr/lib/python3.10/importlib/abc.py", line 433, in open_resource
    return self.files().joinpath(resource).open('rb')
  File "/usr/lib/python3.10/pathlib.py", line 1119, in open
    return self._accessor.open(self, mode, buffering, encoding, errors,
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/test_hypothesmith/.virtualenv/test_hypothesmith-dgeh/lib/python3.10/site-packages/hypothesmith/python.lark'

I assume it's related to the recent lark/lark-parser snafus

I am trying to reduce the usage of linting programs like black or isort in nixpkgs to make programs less likely to break when they are updated and they trigger fewer rebuilds.

According to https://pypi.org/project/hypothesmith/ latest version is 0.1.8 however there is no in git repo version tags. Is it possible to add version tag for last version?

Commit https://github.com/HypothesisWorks/hypothesis/commit/2b7ddefbedfe1d20975576f9bc947c056f936399#diff-a6a0e1e84af3282e1fc162b4ebec8fdf introduced a new required parameter to LarkStrategy in hypothesis, but hypothesmith seems to still initialize without it. A marginally useful stack trace from LibCST is as follows:

Traceback (most recent call last):
  File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/usr/lib/python3.6/unittest/__main__.py", line 18, in <module>
    main(module=None)
  File "/usr/lib/python3.6/unittest/main.py", line 94, in __init__
    self.parseArgs(argv)
  File "/usr/lib/python3.6/unittest/main.py", line 141, in parseArgs
    self.createTests()
  File "/usr/lib/python3.6/unittest/main.py", line 148, in createTests
    self.module)
  File "/usr/lib/python3.6/unittest/loader.py", line 219, in loadTestsFromNames
    suites = [self.loadTestsFromName(name, module) for name in names]
  File "/usr/lib/python3.6/unittest/loader.py", line 219, in <listcomp>
    suites = [self.loadTestsFromName(name, module) for name in names]
  File "/usr/lib/python3.6/unittest/loader.py", line 153, in loadTestsFromName
    module = __import__(module_name)
  File "/home/dragonminded/LibCST/libcst/tests/test_fuzz.py", line 49, in <module>
    class FuzzTest(unittest.TestCase):
  File "/home/dragonminded/LibCST/libcst/tests/test_fuzz.py", line 55, in FuzzTest
    @hypothesis.given(source_code=from_grammar(start="file_input"))
  File "/home/dragonminded/LibCST/.tox/fuzz/lib/python3.6/site-packages/hypothesmith/syntactic.py", line 106, in from_grammar
    return GrammarStrategy(grammar, start, explicit_strategies).map(check_and_fix)
  File "/home/dragonminded/LibCST/.tox/fuzz/lib/python3.6/site-packages/hypothesmith/syntactic.py", line 55, in __init__
    LarkStrategy.__init__(self, grammar, start=start)  # type: ignore
TypeError: __init__() missing 1 required positional argument: 'explicit'

Pinning to hypothesis 4.32.3 works around the problem.

Grammar-based generation works, and gives us syntactically valid source code.

The next step is to get semantically valid source code! The clear best approach for this is to generate a syntax tree, and "unparse" it into source code. Based on experiments at the PyCon Australia sprints the best AST to use is probably from lib2to3 - and that will give us the unparsing for free via black.

After that, I'd like to go to a concrete syntax tree where we draw formatting information at the same time as the node. This would massively improve our usefulness for black, but it's a lot of extra work.

Does it seem correct? I've simply taken the code in nonempty_seq and removed min_size=1. Maybe we could factorize and create a similar function for possibly empty sequences.

I've packaged hypothesmith 0.2.0 and ran the self tests, and I got:

_____________________________________________________________________ test_source_code_from_libcst_node_type[MatchSingleton] _____________________________________________________________________
tests/test_cst.py:23: in test_source_code_from_libcst_node_type
    @given(data=st.data())
E   hypothesis.errors.Unsatisfiable: Unable to satisfy assumptions of test_source_code_from_libcst_node_type
------------------------------------------------------------------------------------------- Hypothesis -------------------------------------------------------------------------------------------
You can add @seed(77441592302281572436730079892928709310) to this test or run pytest with --hypothesis-seed=77441592302281572436730079892928709310 to reproduce this failure.

The installed dependencies are libcst-0.4.1, lark-parser-0.12.0, and hypothesis 6.36.1, with python 3.10.2; on NetBSD/amd64 in case it matters.

For distribution purposes we need the license to be shipped with the sourcecode. The tests would allow us to verify our python stack against the hypotesmith and see if we break up something. Alternative option would be if you could add tags here so we would fetch the tag tarball from github.

See this comment on Reddit and this blog post:

Be warned that Python always applies NFKC normalization to characters. Therefore, two distinct characters may actually produce the same variable name. For example:

>>> ª = 1 # FEMININE ORDINAL INDICATOR
>>> a # LATIN SMALL LETTER A (i.e., ASCII lowercase 'a')
1

Hypothesmith should deliberately violate this rule, to expose tools which compare identifiers as strings without correctly normalizing them first.

I must be missing something simple. I run this program:

from hypothesis import given
import hypothesmith as hs

@given(hs.from_grammar("eval_input"))
@settings(max_examples=50)
def test(expr):
    print(repr(expr))

test()

This prints examples like these:

'A'
'A\n'
'A,A,'
'A,A'
'A,'
'A,A,\n'
'A\n'
'AorA,A'
'AorA,'
'A,\n'
'lambda:A,\n'

I am pretty sure it meant A or A, not AorA. (I saw more similar example in other runs and variations of the program.)

It also occasionally prints a traceback and this error:

hypothesis.errors.FailedHealthCheck: It looks like your strategy is filtering out a lot of data. Health check fo
und 50 filtered examples but only 6 good ones. This will make your tests much slower, and also will probably dis
tort the data generation quite a lot. You should adapt your strategy to filter less. This can also be caused by 
a low max_leaves parameter in recursive() calls

This is Python 3.9.2 on Windows.

hypothesis          6.13.0
hypothesmith        0.1.8

I figure I'm doing something wrong or not understanding something?

When generating eval input I consistently get the following error:

hypothesis.errors.FailedHealthCheck: It looks like your strategy is filtering out a lot of data. Health check found 50 filtered examples but only 3 good ones. This will make your tests much slower, and also will probably distort the data generation quite a lot. You should adapt your strategy to filter less. This can also be caused by a low max_leaves parameter in recursive() calls
See https://hypothesis.readthedocs.io/en/latest/healthchecks.html for more information about this. If you want to disable just this health check, add HealthCheck.filter_too_much to the suppress_health_check settings for this test.

The test is really straightforward:

@hypothesis.given(hypothesmith.from_grammar(start="eval_input"))
def test_expressions(self, expression: str):
    expression = expression.strip()
    print(expression)
    self.assertTrue(expression)

Note: the error appears in subsequent runs. The first time I get no error but is quite slow, but whenever I re-run the tests the errors is triggered and the tests are run much faster.

Every project with aspirations to greatness needs a logo, and hypothesmith is no exception. Are you the generous designer who can help?

• Hypothesmith is, as the name suggests, built on Hypothesis. You may therefore want to draw on that project's logo and brand, though it's not required.
• The other major inspiration is CSmith. I wouldn't copy them too closely, but the blacksmith theme is pretty obvious. Perhaps other kinds of smithing (silver, gold, etc.) would look cool?
• Hypothesmith creates Python code, so you could also work in the Python snakes somehow.
• Once hypothesmithhas a logo I like, I'll be printing it on stickers - and will send you some wherever you are if you would like some. The logo need not include the project name, but it would be nice to have a sticker design that does for easier recognition.

Ideas or sketches are welcome, not just finished proposals :grin: