2.9:

>>> jinja2.Template('{% set a = -1 %}{% for x in range(5) %}[{{ a }}:{% set a = x %}{{ a }}] {% endfor %}{{ a }}').render()
u'[:0] [:1] [:2] [:3] [:4] -1'

2.8:

>>> jinja2.Template('{% set a = -1 %}{% for x in range(5) %}[{{ a }}:{% set a = x %}{{ a }}] {% endfor %}{{ a }}').render()
u'[-1:0] [0:1] [1:2] [2:3] [3:4] -1'

Originally reported on IRC:

A change in jinja2 scoping appears to affect me, and I'm unsure of the correct fix. Specifically the problem is the assignment of year here: https://github.com/kennethlove/alex-gaynor-blog-design/blob/551172/templates/archive.html#L13-L24

This works by having an alternate CodeGenerator that avoids doing to_string after the yield statement and a new version of concat that handles the returned generator with a bit more "intelligence".

Related to https://github.com/ansible/ansible/pull/23943

We use jinja heavily in the ansible project. Although it seems to target a text based destination for the renderers, our users have a desire to preserve the types of their templated vars. We also do a lot of internal intercept and post-processing to preserve those types but it's hit or miss and never obvious to the end user what will work and what won't. Therefore, I'm trying to extend jinja beyond what it's original usecase might have been.

This is a first pass and I hope to drive discussion with it and shape it into something the rest of the jinja community is happy with.

This PR adds a few more type-related tests.

• boolean Testing if an object is a boolean required 2 tests.

    {% if result.value is boolean %}
  

• false Make this similar to testing none value, not requiring sameas

    {% if result.value is false %}
  

• true Make this similar to testing none value, not requiring sameas

    {% if result.value is true %}
  

• integer The existing 'number' test does not make a distinction between integer, float or even booleans

    {% if result.value is integer %}
  

• float The existing 'number' test does not make a distinction between integer, float or even booleans

    {% if result.value is float %}
  

• ~list~ ~The existing 'sequence' or 'iterable test does not make a distinction between strings, lists or mappings. Even 'iterable' does not help here.~

    {% if result.value is list %}
  

This brings the same convenience as:

• none

    {% if result.value is none %}
  

• string

    {% if result.value is string %}
  

• mapping

    {% if result.value is mapping %}
  

For the original Jinja2 use-case where values eventually turn into strings anyway, type-checking is less of an issue, however in Ansible or other projects that use Jinja2 for more than web-templating this is more important.

We see people turning booleans into strings to compare with 'True' or 'False'. Or doing very weird things to determine what is a string, or a list. If you're not careful you end up with a character, instead of the first element.

You can see the effect of testing different types in Jinja including these new tests: https://github.com/dagwieers/ansible/blob/jinja2-type-tests/test/integration/targets/jinja2_tests/tasks/main.yml

PS This PR also has a workaround for a small doc issue breaking Travis testing.

Fixes #178

Blocks now support a new syntax {%* ... %} that aligns the indentation of multiline string with the block statement itself. This is especially useful when templating YAML or other languages where indentation matters. Example:

labels.j2:

tla: webtool
env: {{ env }}

deployment.yaml.j2:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  labels:
    {% include 'labels.j2' %}
  name: webtool
spec:
  selector:
    matchLabels:
      {% include 'labels.j2' %}
  strategy:
    type: Recreate

...renders to broken YAML:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  labels:
    tla: webtool
env: qa
  name: webtool
spec:
  selector:
    matchLabels:
      tla: webtool
env: qa
  strategy:
    type: Recreate

deployment_new_syntax.yaml.j2:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  labels:
    {%* include 'labels.j2' %}
  name: webtool
spec:
  selector:
    matchLabels:
      {%* include 'labels.j2' %}
  strategy:
    type: Recreate

...renders correctly:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  labels:
    tla: webtool
    env: qa
  name: webtool
spec:
  selector:
    matchLabels:
      tla: webtool
      env: qa
  strategy:
    type: Recreate

Currently, jinja2 expects either _path defined or get_filenames: https://github.com/pallets/jinja/blob/master/src/jinja2/loaders.py#L262-L281

The pytest assertion rewriter defines neither (see https://github.com/pytest-dev/pytest/blob/master/src/_pytest/assertion/rewrite.py#L48), and as such running a test suite on a source code that has the following global:

from jinja2 import PackageLoader
LOADER = PackageLoader(__name__, "templates")

will fail with:

    raise ValueError(
E   ValueError: The 'xxx' package was not installed in a way that PackageLoader understands.

Not sure if here jinja2 needs to support more ways to get the template rooot, or pytest loader is missing some methods.

What do you think about add name and filename parameters in Environment.from_string method ? https://github.com/pallets/jinja/blob/master/jinja2/environment.py#L849

Something like that:

    def from_string(self, source, name=None, filename=None, globals=None, template_class=None):
        """Load a template from a string.  This parses the source given and
        returns a :class:`Template` object.
        """
        globals = self.make_globals(globals)
        cls = template_class or self.template_class
        return cls.from_code(self, self.compile(source, name=name, filename=filename), globals, None)

Best regards, Stéphane

Description

Markup has release a new version and now Jinja throws an error.

File "/opt/hostedtoolcache/Python/3.8.12/x64/lib/python3.8/site-packages/dbt/utils.py", line 9, in <module>
    import jinja2
  File "/opt/hostedtoolcache/Python/3.8.12/x64/lib/python3.8/site-packages/jinja2/__init__.py", line 12, in <module>
    from .environment import Environment
  File "/opt/hostedtoolcache/Python/3.8.12/x64/lib/python3.8/site-packages/jinja2/environment.py", line [25](https://github.com/cloudfactory/edw-dbt/runs/5243038097?check_suite_focus=true#step:7:25), in <module>
    from .defaults import BLOCK_END_STRING
  File "/opt/hostedtoolcache/Python/3.8.12/x64/lib/python3.8/site-packages/jinja2/defaults.py", line 3, in <module>
    from .filters import FILTERS as DEFAULT_FILTERS  # noqa: F401
  File "/opt/hostedtoolcache/Python/3.8.12/x64/lib/python3.8/site-packages/jinja2/filters.py", line 13, in <module>
    from markupsafe import soft_unicode
ImportError: cannot import name 'soft_unicode' from 'markupsafe' (/opt/hostedtoolcache/Python/3.8.12/x64/lib/python3.8/site-packages/markupsafe/__init__.py)
Error: Process completed with exit code 1.

Bug Replication

Install Jinja with MarkupSafe 2.1.0

Expected behavior

No errors when importing jinja.

Environment

• Python version: 3.8.12
• Jinja version: 2.11.3
• MarkupSafe: 2.1.0

Currently the method of choice to iterate over dictionaries is foo.items() or foo|dictsort. The former requires that the item provided does in fact have an .items method which is sensible to assume in general. The dictsort filter does the same but also sorts.

In MiniJinja the issue came up that there is no .items implemented there as I did not intend on implementing this method there. In MiniJinja I would prefer to have a |pairs filter. The reason for this is twofold: on the one hand is .items() a Python-ism that shines through, on the other hand does it create this challenge that the same syntax does different things (method access vs item lookup). In Jinja2 this problem has been addressed by changing the order of priority between attribute and item access which requires users to use foo["items"] if they do not want to refer to the method. This has been a source of bugs in the past and as a result I was considering not making the same mistake again.

At the same time I do not really wish to create some new patterns in MiniJinja if they do not exist in Jinja2 itself.

My proposal is now to either implement .items() in MiniJinja regardless or to implement |pairs / |items in both. It would effectively just call .items() on the passed value.

The main benefit would be that it's easier to write templates in a common subset of Jinja2 that works also in languages that are not backed by Python objects.

Refs https://github.com/mitsuhiko/minijinja/issues/32

Once this is merged into master, I'll probably release a 3.0.0a1 so people can start testing early.

• Remove _compat module and other compat code.
• Run pyupgrade, use f-strings everywhere, replace x and y or z with y if x else z.
• Remove deprecated code from Jinja 2.x.
• Stop building universal wheel.
• Bump bytecode cache version.
• ~I18N doesn't look for ugettext.~
• Remove old docs.

Still have some things left that I'll either hit later or get help with at the PyCascades 2020 sprint:

• Inline asyncsupport instead of patching where possible. Keep in mind performance reasons for patching.
• Refactor compiler to extract some helper functions so things like "async for" if self.is_async else "for", Markup if ctx.volatile else str, etc. aren't written by hand all over the place.
• Use super instead of BaseClass.method where appropriate.
• Apply pyupgrade and code style fixes to docs.
• Apply pyupgrade and code style fixes to generated code.
• Add tests for using Babel for translations.

Expected Behavior

When handling a template containing the NEL character, I would expect the rendered result to still contain the character. Maybe I just missed to set an option when creating the template object.

Actual Behavior

The NEL character is stripped from the rendered string.

Minimum Example to Reproduce the Behavior

from jinja2 import Template

input = '\x85'

template = Template(input)
output = template.render()

print(input)
print(input.encode('raw_unicode_escape'))
print(output)
print(output.encode('raw_unicode_escape'))

this produces

…
b'\x85'

b''

Your Environment

• Python version: 3.7.2rc1
• Jinja version: 2.10

Continuing discussion from https://github.com/pallets/meta/issues/10#issuecomment-209980352

The naming is inconsistent:

• Github repo is jinja
• Pypi package name is jinja2
• Pallets project calls it "Jinja": https://www.palletsprojects.com/p/jinja/
• RTD namespace is jinja2.readthedocs.io
• Pocoo docs (currently the official ones) are "Jinja": http://jinja.pocoo.org/docs/2.9/
• file extensions are sometimes .jinja, .j2, .jinja2... Ansible project currently uses .j2

We should pick either "Jinja" or "Jinja2" and use it everywhere for consistency.

I am open to either, "Jinja" is simpler and shorter, but "Jinja2" has a more distinctive ring to it and less likely to get confused with any other projects.

• fixes #

Checklist:

• [ ] Add tests that demonstrate the correct behavior of the change. Tests should fail without the change.
• [ ] Add or update relevant docs, in the docs folder and in code.
• [ ] Add an entry in CHANGES.rst summarizing the change and linking to the issue.
• [ ] Add .. versionchanged:: entries in any relevant code docs.
• [ ] Run pre-commit hooks and fix any issues.
• [ ] Run pytest and tox, no tests failed.

unique() filter doesn't work chained after a filter that return an async generator.

This introduces a async variant of the filter transform the async generator into a list to be able to apply the unique filter.

Fixes #1781

Checklist:

• [X] Add tests that demonstrate the correct behavior of the change. Tests should fail without the change.
• [X] Add or update relevant docs, in the docs folder and in code.
• [X] Add an entry in CHANGES.rst summarizing the change and linking to the issue.
• [X] Add .. versionchanged:: entries in any relevant code docs.
• [X] Run pre-commit hooks and fix any issues.
• [X] Run pytest and tox, no tests failed.

unique() filter does not work when chain with a filter that returns async generator

Environment:

• Python version: 3.11
• Jinja version: 3.1.2

The template:

{%- set commit_authors = commits | rejectattr("author", "eq", "mergify[bot]") | unique(false, "email_author")| list -%}

The backtrace:

  File "xxxxxx.py", line 2314, in render_template
    return await env.from_string(template).render_async(**infos)
  File "jinja2/environment.py", line 1324, in render_async
    return self.environment.handle_exception()
  File "jinja2/environment.py", line 936, in handle_exception
    raise rewrite_traceback_stack(source=source)
  File "jinja2/environment.py", line 1321, in <listcomp>
    [n async for n in self.root_render_func(ctx)]  # type: ignore
  File "<template>", line 7, in top-level template code
  File "jinja2/async_utils.py", line 65, in auto_await
    return await t.cast("t.Awaitable[V]", value)
  File "jinja2/filters.py", line 1329, in do_list
    return await auto_to_list(value)
  File "jinja2/async_utils.py", line 84, in auto_to_list
    return [x async for x in auto_aiter(value)]
  File "jinja2/async_utils.py", line 84, in <listcomp>
    return [x async for x in auto_aiter(value)]
  File "jinja2/async_utils.py", line 77, in auto_aiter
    for item in t.cast("t.Iterable[V]", iterable):
  File "jinja2/filters.py", line 437, in do_unique
    for item in value:
TypeError: 'async_generator' object is not iterable

Add Environment.extract_parsed_names to support tracking dynamic inheritance or inclusion.

Compared to jinja2.meta.find_referenced_templates(), it:

a. works on dynamic inheritance and includes b. does not work unless and until you actually render the template

Many buildsystems are unable to support (b), but some do e.g. [1], the point being that if the target file does not exist, dependency information is not needed since the target file must be built anyway. In such cases, you may prefer this function due to (a).

[1] https://make.mad-scientist.net/papers/advanced-auto-dependency-generation/

• fixes #1775

Checklist:

• [X] Add tests that demonstrate the correct behavior of the change. Tests should fail without the change.
• [X] Add or update relevant docs, in the docs folder and in code.
• [x] Add an entry in CHANGES.rst summarizing the change and linking to the issue.
• [X] Add .. versionchanged:: entries in any relevant code docs.
• [x] Run pre-commit hooks and fix any issues.
• [x] Run pytest and tox, no tests failed.