PR 'Add the ability to pipe args and kwargs.' -> #23

I added the functionality to pipe *args and **kwargs to a function. And now you don't need to pipe a tuple with the first argument as a function and the second argument as a parameter Now you can pass *args and **kwargs to a function using pipe '|'. Now prepare_function_for_pipe knows how to handle keyword-only arguments. And or knows how to handle next_func as *args and **kwargs to self.func.

    # Automatic partial with *args
    range_args: tuple[int, int, int] = (1, 20, 2)
    # Using pipe
    my_range: Callable = pipe | range | range_args
    # Using tuple
    my_range: Callable = pipe | (range, range_args)
    # list(my_range()) == [1, 3, 5, 7, 9, 11, 13, 15, 17, 19]

    # Automatic partial with **kwargs
    dataclass_kwargs: dict[str, bool] = {'frozen': True, 'kw_only': True, 'slots': True}
    # Using pipe
    my_dataclass: Callable = pipe | dataclass | dataclass_kwargs
    # Using tuple
    my_dataclass: Callable = pipe | (dataclass, dataclass_kwargs)
    @my_dataclass
    class Bla:
        foo: int
        bar: str

    # Bla(5, 'bbb') -> Raises TypeError: takes 1 positional argument but 3 were given
    # Bla(foo=5, bar='bbb').foo == 5

This patch added support for:

2 ** (64 - X)   # __rpow__, __rsub__, and all other __rFOOBAR__ methods
X >> 8          # __rshift__ and __rrshift__ (There were already X > 8 and X < 8, so I guess X >> 8 does not indicate a pipe redirection either)
1 << (64 - X)   # __lshift__ and __rlshift__
X @ np.eye(10)  # matrix multiplication since Python 3.5
X & 1
X ^ 1
+X              # I wonder if anyone would ever use this unary operator...

This patch also fixed a tiny repr problem:

>>> X // 42
X / 42          # should have been X // 42, even in Python 2.7

This patch did not add support for:

index in X      # the opposite of X._in_(index); __contains__ did not work and always returned True?
len(X)          # it would be confusing and inconsistent if built-in functions worked, but custom functions did not...

Magic method references: Python 2, Python 3

Hi everyone, Just wanted to throw an idea out here - would pipetools be a good fit for stream-based/reactive programming? Specifically what I'm thinking of is piping streams of data together, rather than piping functions, where it's not necessarily the case that a stream will a emit a datum every time it consumes one, in the way that a piped function will always pass its result to the next function in the sequence.

Would it be enough to override the "compose" method of the Pipe class to adapt to such a use-case?

issue

I encounter an UnicodeDecodeError when trying to install pipetools on windows 7 (pip 10, python 3.6.5). Figure out that this is caused by file encoding issue when trying to open README.rst sth. like

UnicodeDecodeError: 'gbk' codec can't decode byte 0x80 in position 205: illegal multibyte sequence

what this fix does

Use io.open for python 2/3 compatibility, set the encoding paramater of io.open to utf-8 fixs the issue.

Not totally sure about anything here (this is my first time leaving a bug) but it seems like I've found a situation where these two seemingly-the-same statements do different things. I've included a minimal example:

import numpy as np
from pipetools import pipe

def scaleDiagonal(matrix):

	m = np.copy(matrix)
	size = matrix.shape[0]

	smallest = min([ m[i,i] for i in range(size) ])

	for i in range(size):
		m[i,i] = m[i,i] / smallest

	return m

x = np.array([1,2,3,4,5,6,7,8,9]).reshape(3,3)

y = (pipe | scaleDiagonal)(x)         # Works
z = x > pipe | scaleDiagonal          # Errors

According alert from DeprecationWarning we have two issues:

pipetools/main.py:1: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3, and in 3.9 it will stop working
    from collections import Iterable

and

pipetools/utils.py:2: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3, and in 3.9 it will stop working
    from collections import Mapping

Docs: https://docs.pytest.org/en/stable/warnings.html

I don't know if it's possible to change > to >> for the input method? IMO >> is less used, especially in numerical / science-related code. I looked around the source code but didn't find where to make the change (I don't see __gt__ method in Pipe class).

Thank you.

I am trying to run this example

additives_df.loc[:, ["pallets"]] > pipe | (lambda df: df.head())

but i get the exceptions

TypeError: '>' not supported between instances of 'float' and 'function'

Is there a proper way to pipe pandas dataframes

Works not included in this PR but you'd probably want:

• [x] Run publish_docs.py to update README.rst
• [x] Bump the version number and update changelog.rst
• [x] Upload to PyPI

Signed-off-by: Asaf Taller [email protected] https://github.com/0101/pipetools/issues/24

I added the functionality to pipe *args and **kwargs to a function. And now you don't need to pipe a tuple with the first argument as a function and the second argument as a parameter Now you can pass *args and **kwargs to a function using pipe '|'. Now prepare_function_for_pipe knows how to handle keyword-only arguments. And or knows how to handle next_func as *args and **kwargs to self.func.

    # Automatic partial with *args
    range_args: tuple[int, int, int] = (1, 20, 2)
    # Using pipe
    my_range: Callable = pipe | range | range_args
    # Using tuple
    my_range: Callable = pipe | (range, range_args)
    # list(my_range()) == [1, 3, 5, 7, 9, 11, 13, 15, 17, 19]

    # Automatic partial with **kwargs
    dataclass_kwargs: dict[str, bool] = {'frozen': True, 'kw_only': True, 'slots': True}
    # Using pipe
    my_dataclass: Callable = pipe | dataclass | dataclass_kwargs
    # Using tuple
    my_dataclass: Callable = pipe | (dataclass, dataclass_kwargs)
    @my_dataclass
    class Bla:
        foo: int
        bar: str

    # Bla(5, 'bbb') -> Raises TypeError: takes 1 positional argument but 3 were given
    # Bla(foo=5, bar='bbb').foo == 5

Let me know if you have any questions about this PR. @0101

I'd like to propose a new util: foreach_i.

Motivation

You know how JS's Array.map also passes the element index as a 2nd parameter to the function?

> ['a', 'b', 'c'].map((x, i) => `Element ${i} is ${x}`)
[ 'Element 0 is a', 'Element 1 is b', 'Element 2 is c' ]

That's exactly what I'm trying to do here. The only difference is that the index would be passed as the 1st param:

def test_foreach_i():
    
    r = ['a', 'b', 'c'] > (pipe
                          | foreach_i(lambda i, x: f'Element {i} is {x}')
                          | list
                          )
    
    assert r == [ 'Element 0 is a'
                , 'Element 1 is b'
                , 'Element 2 is c'    
                ]

(Naïve) Implementation

from pipetools.utils import foreach, as_args
from typing import Callable, TypeVar


A = TypeVar('A')
B = TypeVar('B')

def foreach_i(f: Callable[[int, A], B]):
    
    return enumerate | foreach(as_args(f))

The same could be done for foreach_do.

What is a good practice or recommendation for writing docstrings that are simple and clean to write and work well with documentation tools.

Related Stackoverflow quesion

There is some overhead to create pipes. For some use cases it may be advantageous to cache pipes or even partial pipes. Would it be possible to cache pipes automatically? ... or by some switch, etc.?

Here you can see the "penalty" associated with creating pipes.

$ ipython3
Python 3.7.5 (default, Oct 17 2019, 12:21:00) 
Type 'copyright', 'credits' or 'license' for more information
IPython 7.18.1 -- An enhanced Interactive Python. Type '?' for help.

In [1]: from pipetools import pipe,X,foreach

In [2]: def my_func(count=10000, predef = False):
   ...:     if predef == False:
   ...:         for k in range(count):
   ...:             a = range(10) > pipe | foreach(X**2) | sum
   ...:     else:
   ...:         my_pipe = pipe | foreach(X**2) | sum
   ...:         for k in range(count):
   ...:             a = range(10) > my_pipe
   ...:     return a
   ...: 

In [3]: %timeit my_func()
202 ms ± 8.8 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

In [4]: %timeit my_func(predef=True)
59.5 ms ± 1.67 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)

In [5]: %timeit for k in range(10000): a=sum([x**2 for x in range(10)])
29.9 ms ± 962 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)

I'd like to propose a new join util.

The idea is simple: it calls foreach(str) before calling str.join. So, whenever you have something like

| foreach(str)
| ', '.join

you can replace it with

| join(', ')

You can customize the string conversion by passing either a function or a format string as the second parameter, e.g.:

| join(', ', lambda x: '-{}-'.format(x))        # function

| join(', ', '-{}-')                            # fmt string

Implementation

def join(delim, formatter=str):
    '''
    join(' ')
    join(' ', fmtFn)
    join(' ', fmtString)
    '''
    
    return foreach(formatter) | delim.join

Tests

def test_join(self):
    
    r = [1, 2, 3] > (pipe
                    | join(', ')
                    )
    
    self.assertEquals(r, '1, 2, 3')
    
    
def test_join_with_formatter(self):
    
    r = [1, 2, 3] > (pipe
                    | join(', ', lambda x: '-{}-'.format(x))
                    )
    
    self.assertEquals(r, '-1-, -2-, -3-')
    
    
def test_join_with_fmtString(self):
    
    r = [1, 2, 3] > (pipe
                    | join(', ', '-{}-')
                    )
    
    self.assertEquals(r, '-1-, -2-, -3-')