What do you think about having a Cython implementation of toolz that can be used as a regular C extension in CPython, or be cimport-ed by other Cython code?

I've been messing around with Cython lately, and I became curious how much performance could be gained by implementing toolz in Cython. I am almost finished with a first-pass implementation (it goes quickly when one doesn't try to fine-tune everything), and just have half of itertoolz left to do.

Performance increases of x2-x4 are common. Some perform even better (like x10), and a few are virtually the same. There is also less overhead when calling functions defined in Cython, which at times can be significant regardless of how things scale.

However, performance when called from Python isn't the only consideration. A common strategy used by the scientific, mobile, and game communities to increase performance of their applications is to convert Python code that is frequently run to Cython. Developing in Cython also tends to be very imperative. A Cython version of toolz will allow fast implementations to be used in other Cython code (via cimport) while facilitating a more functional style of programming.

Looking ahead, cython.parallel exposes OpenMP at a low level, which should allow for more efficient parallel processing.

Thoughts? Any ideas for a name? I am thinking coolz, because ctoolz and cytoolz sound like they are utilities for C or Cython code. I can push what I currently have to a repo once it has a name. Should this be part of pytoolz?

Here is a semi-streaming Join function, analagous to SQL Join

Join two sequences on common attributes

This is a semi-streaming operation. The LEFT sequence is fully evaluated and placed into memory. The RIGHT side is evaluated lazily and so can be arbitrarily large.

The following example joins quantities of sold fruit to the name of the quantity.

>>> names = [(1, 'one'), (2, 'two'), (3, 'three')]
>>> fruit = [('apple', 1), ('banana', 2), ('coconut', 2), ('orange', 1)]

>>> result = join(first, second, names, fruit, apply=lambda x, y: x + y)
>>> for row in result:
...     print(row)
(1, 'one', 'apple', 1)
(2, 'two', 'banana', 2)
(2, 'two', 'coconut', 2)
(1, 'one', 'orange', 1)

I found some logical predicate functions useful recently, so I added them to toolz to complement...complement. Does it seem reasonable? The implementation and testing here isn't necessary the final result, just a proof-of-concept.

Also, I wasn't sure whether to call the new functions the imaginary verbs conjunct and disjunct (they're only nouns and adjectives in my dictionary) as per standard functional style or the clearer but longer conjunction and disjunction. Went with the latter for now.

This builds upon the discussion and feedback from #166, which also has a faster (but harder to understand) implementation.

EqualityHashKey creates a hash key that uses equality comparisons between items, which may be used to create hash keys for otherwise unhashable types. The trade-offs for using this are discussed in the docstring. Additional usage cases would qualify as compelling reasons to promote EqualityHashKey out of the sandbox (imho).

@asmeurer, do you have any suggestions or additional cases where this would be applicable?

Hey guys! I would expect the following behavior from toolz.curry:

>>> @toolz.curry
>>> def kwonly_sum(a, *, b=10):
           return a + b
>>> b_is_five = kwonly_sum(b=5) # actually raise exception here
>>> b_is_five(5)
10 # what I want

The exception gives a suggested solution:

TypeError                                 Traceback (most recent call last)
/home/mtartre/.conda/envs/std/lib/python3.4/site-packages/toolz/functoolz.py in __call__(self, *args, **kwargs)
    218         try:
--> 219             return self._partial(*args, **kwargs)
    220         except TypeError:

TypeError: kwonly_sum() missing 1 required positional argument: 'a'

During handling of the above exception, another exception occurred:
ValueError                                Traceback (most recent call last)
<ipython-input-31-fd1daf64ecb7> in <module>()
----> 1 kwonly_sum(b=5)(5)

/home/mtartre/.conda/envs/std/lib/python3.4/site-packages/toolz/functoolz.py in __call__(self, *args, **kwargs)
    220         except TypeError:
    221             # If there was a genuine TypeError
--> 222             required_args = _num_required_args(self.func)
    223             if (required_args is not None and
    224                     len(args) + len(self.args) >= required_args):

/home/mtartre/.conda/envs/std/lib/python3.4/site-packages/toolz/functoolz.py in _num_required_args(func)
    116         return known_numargs[func]
    117     try:
--> 118         spec = inspect.getargspec(func)
    119         if spec.varargs:
    120             return None

/home/mtartre/.conda/envs/std/lib/python3.4/inspect.py in getargspec(func)
    934         getfullargspec(func)
    935     if kwonlyargs or ann:
--> 936         raise ValueError("Function has keyword-only arguments or annotations"
    937                          ", use getfullargspec() API which can support them")
    938     return ArgSpec(args, varargs, varkw, defaults)

ValueError: Function has keyword-only arguments or annotations, use getfullargspec() API which can support them

The issue is exactly the same in cytoolz. Apologies I don't have the fix in a pull request, my firm requires legal approval for that.

I like the Dicttoolz package, but for many of my use cases I need the deterministic behaviour of OrderedDict. Adapting Dicttoolz to return OrderedDict if all of its inputs are one is relatively straightforward. Is this something you would consider merging?

Woah!

I was curious what it would be like to trace the input and output of toolz functions and user-defined functions. As a proof-of-concept, I created this branch:

https://github.com/eriknw/toolz/tree/trace_with_q

Simply do from toolz.traced import * and viola! In another termal, watch the output real-time via tail -f /tmp/toolz.

To trace a user function use trace as a decorator or function.

The results are astounding. I would paste example traces here, but I think you guys have got to try this out yourself.

q was copied from https://github.com/zestyping/q and was slightly modified to output to "/tmp/toolz" instead of "/tmp/q".

As I said above, this was meant as a proof-of-concept. It begs the question, though, whether such functionality should be added to toolz, how it should behave, etc. Tracing can be very handy for debugging and as an educational tool for new users.

If you encounter any bugs in the above branch, please post here.

Thoughts and reactions?

The idea of this is to use exception based api functions alongside your normal functional code.

for example:

map(itemgetter('key'), seq) -> map(excepts(itemgetter('key'), KeyError), seq)

This helps us get around the fact that I cannot put an except clause in a lambda. I have found this to be very useful in my own code.

Most of this code is for fresh __name__ and __doc__ attributes.

Issue #178 impressed upon me just how costly attribute resolution can be. In this case, groupby was made faster by avoiding resolving the attribute list.append.

This implementation is also more memory efficient than the current version that uses a defaultdict that gets cast to a dict. While casting a defaultdict d to a dict as dict(d) is fast, it is still a fast copy.

Honorable mention goes to the following implementation:

def groupby_alt(func, seq):
    d = collections.defaultdict(lambda: [].append)
    for item in seq:
        d[func(item)](item)
    rv = {}
    for k, v in iteritems(d):
        rv[k] = v.__self__
    return rv

This alternative implementation can at times be very impressive. You should play with it!

Using update_wrapper and wraps would be preferable, however they both cause errors -- pickling issues in curry and attribute errors for memoizing a partial in Python 2. However, more than just __name__ and __doc__ should be transferred: __module__, if present and __qualname__ and __annotations__ in Python 3. Updating with func.__dict__ isn't possible in curry (source of pickling problems), but should be done in memoize.

conflicted with kwargs['func']

example.py:

from toolz import curry
@curry
def foo(x, y, func=int, bar=str):
    return str(func(x*y))
foo(bar=float)(4.2, 3.8, func=round)
foo(func=int)(4.2, 3.8, bar=str)

The last line would throw TypeError: __init__() got multiple values for keyword argument 'func' because curry.__init__(self, func, *args, **kwargs) names its first positional argument "func" This effectively prevented creating a curry object with such a kwarg. I didn't find other functions with the same problem.

> /home/digenis/src/toolz/toolz/example.py(1)<module>()
-> from toolz import curry
(pdb) c
Traceback (most recent call last):
 ...
  File "/home/digenis/src/toolz/toolz/functoolz.py", line 224, in __call__
    return curry(self._partial, *args, **kwargs)
TypeError: __init__() got multiple values for keyword argument 'func'
...
> /home/digenis/src/toolz/toolz/functoolz.py(224)__call__()
-> return curry(self._partial, *args, **kwargs)
(pdb) self._partial  # __init__ will receive this as the positional argument "func"
<functools.partial object at 0x7f16874ee0a8>
(pdb) pp args
()
(pdb) pp kwargs  # but it will also receive a kwarg named "func"
{'func': <type 'int'>}


Post mortem debugger finished. The example.py will be restarted
...

I abbreviated some line ranges with "..."

Bumps pypa/gh-action-pypi-publish from 1.5.0 to 1.6.4.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Hi! I often encounter the same pattern in my code:

def to_power(array, powers):
    result = []
    for entry in array:
        power = powers.get(entry)
        if power is not None and power >= 0:
            result.append(entry ** power)

    return result


def reverse_mapping(expensive_func, values):
    result = {}
    for entry in values:
        value = expensive_func(entry)
        if value is not None:
            result[value] = entry

    return result

The examples are somewhat simplistic, but you get the idea:

1. create a container
2. iterate some data, do some branching, etc and fill the container
3. return the container

I came up with several decorators that reduce this to:

@collect
def to_power(array, powers):
    for entry in array:
        power = powers.get(entry)
        if power is not None and power >= 0:
            yield entry ** power


@composed(dict)
def reverse_mapping(expensive_func, values):
    for entry in values:
        value = expensive_func(entry)
        if value is not None:
            yield value, entry

composed(func) simply applies func to the result of the decorated function, which effectively gathers the generator. And collect is just a shorter version of composed(list)

I can create a PR with my implementation, if you are interested in adding it to toolz.

It would be really nice to be able to iterate all the funcs in compose, without having to combine the first and funcs properties. It would let you immediately use Compose objects as iterables in the itertoolz functions.

For an example, consider the simple logging strategy outlined in my gist here: https://gist.github.com/ZeroBomb/8ac470b1d4b02c11f2873c5d4e0512a1

As written, I need to define this somewhat extraneous function

def get_funcs(composition):
    return (composition.first,)+composition.funcs

in order to map over those functions and re-compose:

@curry
def interleave_map(func, items):
    # [1,2,3] -> [func(1), 1, func(2), 2, func(3), 3]
    return interleave([map(func, items), items])

# define a debug function that interleaves logging funcs inbetween each func in an existing composition
debug = compose_left(get_funcs, interleave_map(passthru_log), star(compose_left))

if the Compose class were iterable, I could completely eliminate the get_funcs function, and comfortably feed the compose object directly into interleave:

def debug(composition):
    return compose_left(*interleave_map(passthru_log, composition))

I have added

• A basic config file for pyright
• A CI job to run pyright
• comments to ignore errors that pyright detects in existing code.

This is to type-check any type hints that are added to toolz, as suggested in #496. These can be added incrementally.

Convert these loops:

for item in seq:
    yield item

To the more modern and slightly more efficient

yield from seq

A quick benchmark (a is a list of 30 sorted lists of 60 random integers)

# old
In [8]: %timeit list(merge_sorted(*a))
815 µs ± 31.3 µs per loop (mean ± std. dev. of 7 runs, 1,000 loops each)

# new
In [7]: %timeit list(merge_sorted(*a))
766 µs ± 26.2 µs per loop (mean ± std. dev. of 7 runs, 1,000 loops each)

It is currently tempting to test get_in like this:

get_in('x', {'x':5}) # returns 5

and conclude that this will also work:

get_in('test', {'test':5}) # actually returns None

It does not work, because 'test' is treated as ['t','e','s','t']. In complex dictionaries, you may actually get a value, like

get_in('xy', {'x': {'y': 5}} ) # returns 5

The documentation should probably call this out explicitly, if this is the intended behavior, perhaps by giving the 'xy' example above.

I, for one, wouldn't mind an implementation where get_in('test', {'test':5}) returns 5, but I wouldn't go so far as to say that is the right approach. I'm imagining it would facilitate doing something like this:

juxt(*map(curry(get_in), ['str1', ['str2', 'str3'], 'etc']))