Describe the bug

A space character right after a link destination scheme causes an IndexError.

[Contact](http:// mail.com)
[Contact](mailto: [email protected])

Reproduce the bug

from markdown_it import MarkdownIt
MarkdownIt().parse("[Contact](mailto: [email protected])")

Error

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Library/Caches/pypoetry/virtualenvs/webapp-gmQ5g8dx-py3.9/lib/python3.9/site-packages/markdown_it/main.py", line 260, in parse
    self.core.process(state)
  File "/Library/Caches/pypoetry/virtualenvs/webapp-gmQ5g8dx-py3.9/lib/python3.9/site-packages/markdown_it/parser_core.py", line 33, in process
    rule(state)
  File "/Library/Caches/pypoetry/virtualenvs/webapp-gmQ5g8dx-py3.9/lib/python3.9/site-packages/markdown_it/rules_core/inline.py", line 10, in inline
    state.md.inline.parse(token.content, state.md, state.env, token.children)
  File "/Library/Caches/pypoetry/virtualenvs/webapp-gmQ5g8dx-py3.9/lib/python3.9/site-packages/markdown_it/parser_inline.py", line 120, in parse
    self.tokenize(state)
  File "/Library/Caches/pypoetry/virtualenvs/webapp-gmQ5g8dx-py3.9/lib/python3.9/site-packages/markdown_it/parser_inline.py", line 102, in tokenize
    ok = rule(state, False)
  File "/Library/Caches/pypoetry/virtualenvs/webapp-gmQ5g8dx-py3.9/lib/python3.9/site-packages/markdown_it/rules_inline/link.py", line 54, in link
    href = state.md.normalizeLink(res.str)
  File "/Library/Caches/pypoetry/virtualenvs/webapp-gmQ5g8dx-py3.9/lib/python3.9/site-packages/markdown_it/main.py", line 331, in normalizeLink
    return normalize_url.normalizeLink(url)
  File "/Library/Caches/pypoetry/virtualenvs/webapp-gmQ5g8dx-py3.9/lib/python3.9/site-packages/markdown_it/common/normalize_url.py", line 21, in normalizeLink
    parsed = mdurl.parse(url, slashes_denote_host=True)
  File "/Library/Caches/pypoetry/virtualenvs/webapp-gmQ5g8dx-py3.9/lib/python3.9/site-packages/mdurl/_parse.py", line 300, in url_parse
    u.parse(url, slashes_denote_host)
  File "/Library/Caches/pypoetry/virtualenvs/webapp-gmQ5g8dx-py3.9/lib/python3.9/site-packages/mdurl/_parse.py", line 204, in parse
    if rest[host_end - 1] == ":":
IndexError: string index out of range

List your environment

Python 3.9 markdown-it-py 2.0.1

Describe the bug

If there are more than one block quotes in a myst.md file then jupytext (via markdown-it-py) causes

 File "anaconda3/envs/qe-lectures/lib/python3.8/site-packages/markdown_it/rules_block/state_block.py", line 134, in skipEmptyLines
    if (self.bMarks[from_pos] + self.tShift[from_pos]) < self.eMarks[from_pos]:
IndexError: list index out of range

To Reproduce

A minimal Example

---
jupytext:
  text_representation:
    extension: .md
    format_name: myst
kernelspec:
  display_name: Python 3
  language: python
  name: python3
---

# Simple Example

> A Block Quote

some text

> Another Block Quote

and then run jupytext f<ile.md> --to ipynb

Expected behavior

Notebook output with a single cell containing markdown and two block-quotes

Environment

Jupyter Book: 0.8.3 MyST-NB: 0.10.1 Sphinx Book Theme: 0.0.38 MyST-Parser: 0.12.10 Jupyter-Cache: 0.4.1 NbClient: 0.5.0

Hi!

I was looking at performance and noticed that the charCodeAt function was called a lot with some redundancy.

We very often compare ord() codes, and I think it's justified to store them in an attribute for StateCore, StateBlock and StateInline. Eg:

class StateCore(StateBase):
    def __init__(self, src: str, md, env, tokens=None):
        self.src = src
        self.ords = [ord(c) for c in src]
        ...

Then we just replace each variant of: charCodeAt(state.src, pos)

by

state.ords[pos]

Furthermore, StateCore and StateBlock can share a significant part of their ord codes. So we can add an optional parameter to the StateBlock constructor to copy the StateCore ord codes:

class StateBlock(StateBase):
    def __init__(self, src: str, md, env, tokens: List[Token], ords: List[int] = None):
        self.src = src

        if ords is not None:
            self.ords = ords
        else:
            self.ords = [ord(c) for c in src]

Here are some benchmark numbers (100 iterations with benchmark.py):

markdown-it-py (0.4.9): 18.66 s => Original markdown-it-py (0.4.9): 17.93 s => store ord codes as attributes to remove charCodeAt where possible markdown-it-py (0.4.9): 16.43 s => share StateCore ord codes with StateBlock

This is a ~10% performance boost.

However, these changes do not strictly copy the behavior of charCodeAt since it bypasses its try/except clause.

def charCodeAt(src: str, pos: int):
    try:
        return ord(src[pos])
    except IndexError:
        return None

Tests are OK but I wonder if this can have an impact on illformed markdown ? I could create a specific structure to alleviate this issue (a defaultlist that returns None when there is an IndexError, as is done in charCodeAt), but there would be a small downside in code readability, so I'd like to have your input on this.

Let me know what you think!

+ /usr/bin/python3 -Bm pytest -ra
=========================================================================== test session starts ============================================================================
platform linux -- Python 3.8.9, pytest-6.2.4, py-1.10.0, pluggy-0.13.1
rootdir: /home/tkloczko/rpmbuild/BUILD/markdown-it-py-1.1.0
plugins: forked-1.3.0, shutil-1.7.0, virtualenv-1.7.0, expect-1.1.0, cov-2.11.1, httpbin-1.0.0, xdist-2.2.1, flake8-1.0.7, timeout-1.4.2, betamax-0.8.1, pyfakefs-4.4.0, freezegun-0.4.2, cases-3.4.6, case-1.5.3, isort-1.3.0, aspectlib-1.5.2, mock-3.6.0, asyncio-0.15.1, toolbox-0.5, xprocess-0.17.1, flaky-3.7.0, requests-mock-1.9.2, aiohttp-0.3.0, checkdocs-2.7.0, hypothesis-6.12.1
collected 863 items
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/usr/lib/python3.8/site-packages/_pytest/main.py", line 269, in wrap_session
INTERNALERROR>     session.exitstatus = doit(config, session) or 0
INTERNALERROR>   File "/usr/lib/python3.8/site-packages/_pytest/main.py", line 322, in _main
INTERNALERROR>     config.hook.pytest_collection(session=session)
INTERNALERROR>   File "/usr/lib/python3.8/site-packages/pluggy/hooks.py", line 286, in __call__
INTERNALERROR>     return self._hookexec(self, self.get_hookimpls(), kwargs)
INTERNALERROR>   File "/usr/lib/python3.8/site-packages/pluggy/manager.py", line 93, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR>   File "/usr/lib/python3.8/site-packages/pluggy/manager.py", line 84, in <lambda>
INTERNALERROR>     self._inner_hookexec = lambda hook, methods, kwargs: hook.multicall(
INTERNALERROR>   File "/usr/lib/python3.8/site-packages/pluggy/callers.py", line 208, in _multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>   File "/usr/lib/python3.8/site-packages/pluggy/callers.py", line 80, in get_result
INTERNALERROR>     raise ex[1].with_traceback(ex[2])
INTERNALERROR>   File "/usr/lib/python3.8/site-packages/pluggy/callers.py", line 187, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "/usr/lib/python3.8/site-packages/_pytest/main.py", line 333, in pytest_collection
INTERNALERROR>     session.perform_collect()
INTERNALERROR>   File "/usr/lib/python3.8/site-packages/_pytest/main.py", line 636, in perform_collect
INTERNALERROR>     self.config.pluginmanager.check_pending()
INTERNALERROR>   File "/usr/lib/python3.8/site-packages/pluggy/manager.py", line 274, in check_pending
INTERNALERROR>     raise PluginValidationError(
INTERNALERROR> pluggy.manager.PluginValidationError: unknown hook 'pytest_benchmark_update_machine_info' in plugin <module 'conftest' from '/home/tkloczko/rpmbuild/BUILD/markdown-it-py-1.1.0/benchmarking/conftest.py'>

========================================================================== no tests ran in 0.29s ===========================================================================

Issue

#5

Summary

Implemented replacements (Simple typographic replacements) rule.

Change

• Added a test to make sure that even a single capital letter is matched correctly.

def normalize(state: StateCore) -> None:

    # Normalize newlines
    string = NEWLINES_RE.sub("\n", state.src)

    # Replace NULL characters
    string = NULL_RE.sub("\uFFFD", string)

    state.src = string

When updating state.src, state.srcCharCode should be updated synchronously.

Describe the bug

The presence of three empty lines at the end of a particular file causes the build to break.

To Reproduce

Steps to reproduce the behavior:

1. Clone https://github.com/poldrack/psych-open-science-guide
2. "jb build guide" should work properly
3. Add two additional line feeds to the end of guide/4_reproducibleanalysis.md
4. "jb build guide" should now fail with an error.

Expected behavior

When the extra lines are added, the follow Exception occurs:

Environment

• Python 3.8.3

• output of jupyter-book --version: Jupyter Book: 0.7.3 MyST-NB: 0.8.4 Sphinx Book Theme: 0.0.33 MyST-Parser: 0.9.0 Jupyter-Cache: 0.2.2

• Operating System: Mac OS X

Describe the bug

Consider a notebook that contains the following cell:

> test
>

or a markdown file with the above. The key point is that the cell/markdown content must end with a blockquote that has a blank line at the end, still part of the block quote. The python short traceback is (I'll attach the long one):

Exception occurred:
  File "/Users/fperez/local/conda/lib/python3.9/site-packages/markdown_it/rules_block/blockquote.py", line 158, in blockquote
    if state.srcCharCode[pos] == 0x20:  # /* space */
IndexError: tuple index out of range

Reproduce the bug

Running a build with a notebook with the above (or an md file) should suffice.

List your environment

> jupyter-book --version
Jupyter Book      : 0.11.3
External ToC      : 0.2.3
MyST-Parser       : 0.13.7
MyST-NB           : 0.12.3
Sphinx Book Theme : 0.1.10
Jupyter-Cache     : 0.4.3
NbClient          : 0.5.10

Describe the bug

Image tokens always have an empty alt attribute, and the option store_labels doesn't work for inline links (only reference ones).

To Reproduce

Steps to reproduce the behavior:

Python 3.8.3 (default, Jun  7 2020, 18:20:38) 
Type 'copyright', 'credits' or 'license' for more information
IPython 7.19.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: from markdown_it import MarkdownIt                                                                     

In [2]: md = MarkdownIt()                                                                                      

In [3]: src = 'this is a ![picture](test.png "so nice") of a thing'                                            

In [4]: for t in md.parse(src): 
   ...:     if t.type == 'inline': 
   ...:         for c in t.children: 
   ...:             print(c) 
   ...:                                                                                                        
Token(type='text', tag='', nesting=0, attrs=None, map=None, level=0, children=None, content='this is a ', markup='', info='', meta={}, block=False, hidden=False)
Token(type='image', tag='img', nesting=0, attrs=[['src', 'test.png'], ['alt', ''], ['title', 'so nice']], map=None, level=0, children=[Token(type='text', tag='', nesting=0, attrs=None, map=None, level=0, children=None, content='picture', markup='', info='', meta={}, block=False, hidden=False)], content='', markup='', info='', meta={}, block=False, hidden=False)
Token(type='text', tag='', nesting=0, attrs=None, map=None, level=0, children=None, content=' of a thing', markup='', info='', meta={}, block=False, hidden=False)

In [5]: md.options['store_labels'] = True                                                                      

In [6]: for t in md.parse(src): 
   ...:     if t.type == 'inline': 
   ...:         for c in t.children: 
   ...:             print(c) 
   ...:                                                                                                        
Token(type='text', tag='', nesting=0, attrs=None, map=None, level=0, children=None, content='this is a ', markup='', info='', meta={}, block=False, hidden=False)
Token(type='image', tag='img', nesting=0, attrs=[['src', 'test.png'], ['alt', ''], ['title', 'so nice']], map=None, level=0, children=[Token(type='text', tag='', nesting=0, attrs=None, map=None, level=0, children=None, content='picture', markup='', info='', meta={}, block=False, hidden=False)], content='', markup='', info='', meta={}, block=False, hidden=False)
Token(type='text', tag='', nesting=0, attrs=None, map=None, level=0, children=None, content=' of a thing', markup='', info='', meta={}, block=False, hidden=False)

Expected behavior

Image tokens should have their alt attribute set here: https://github.com/executablebooks/markdown-it-py/blob/16b85370f6ef538b50d4da15fa883e05be258c75/markdown_it/rules_inline/image.py#L140

And the store_labels option should add labels to inline links/images that aren't declared as references.

Bonus: does the label still need to be upper-cased in Python? https://github.com/executablebooks/markdown-it-py/blob/16b85370f6ef538b50d4da15fa883e05be258c75/markdown_it/common/utils.py#L337

These are the final tests that need to be fixed; one that is directly applicable to CommonMark compliance, and the others are for additional compliance with markdown-it. They are all to do with how \ escapes are treated for some of the more obscure cases in link titles / code fence languages, e.g. [](<\&quot;> "\&amp;\&ouml;")

• [x] https://github.com/ExecutableBookProject/markdown-it-py/blob/bc799d47e219fac146dd7f38f30641fb295a16fd/tests/test_cmark_spec/test_spec.py#L23
• [ ] https://github.com/ExecutableBookProject/markdown-it-py/blob/bc799d47e219fac146dd7f38f30641fb295a16fd/tests/test_port/test_fixtures.py#L28
• [ ] https://github.com/ExecutableBookProject/markdown-it-py/blob/bc799d47e219fac146dd7f38f30641fb295a16fd/tests/test_port/test_fixtures.py#L53
• [ ] https://github.com/executablebooks/markdown-it-py/pull/140/commits/f0c96ae5dbbbf0d27ed44e2f6ac237662a57001d (from https://github.com/markdown-it/markdown-it/commit/f156ed1bffec0e02b2353abf248c5b10c6101524)

A continuation on https://github.com/executablebooks/markdown-it-py/pull/109 (based on the branch there). Adds the changes in https://github.com/markdown-it/markdown-it/compare/11.0.1...12.0.4 on top

https://github.com/executablebooks/markdown-it-py/pull/109 can be merged first and then this one (to make review easier), or this can be merged directly.