Hi, when trying "pip install tuplex", i am getting the following errors

ERROR: Could not find a version that satisfies the requirement tuplex (from versions: none) ERROR: No matching distribution found for tuplex

Thanks,

I downloaded a 311 requests dataset which contains 6 columns (one of them is "Incident Zip") in order to run the 311 benchmark (tuplex/benchmarks/311/runtuplex.py)

When the following code is executed:

tstart = time.time()
df = ctx.csv(
    ",".join(perf_paths),
    null_values=["Unspecified", "NO CLUE", "NA", "N/A", "0", ""],
    type_hints={0: typing.Optional[str],
                1: typing.Optional[str],
                2: typing.Optional[str],
                3: typing.Optional[str],
                4: typing.Optional[str],
                5: typing.Optional[str],
    },
)
# Do the pipeline
df = df.mapColumn("IncidentZip", fix_zip_codes).unique()
# Output to csv
df.tocsv(output_path)
job_time = time.time() - tstart
print(json.dumps({"startupTime": startup_time, "jobTime": job_time}))

I receive the following error: [2022-04-26 16:57:10.185] [global] [error] decoding of other types not yet supported...

Initially I suspected that the issue might be that Incident Zip column is inferred as f64 so I explicitly set the type of all columns to be str, as seen in the code snippet above.

However, the error still persists. Any ideas?

PS: I believe that error is on unique() method because when I remove the call it works fine.

Thanks in advance.

The python binding for aggregateByKey (in dataset.py) currently crashes due to the deprecated function get_lambda_source - update to the new function get_udf_source.

The tuplex crashes from a corrupted exception partition when the pipeline is executed twice. This could be an issue from invalidating partition.

Example pipeline when it crashes:

ds = c.parallelize([(1, "A"),(2, "a"),(3, 2)]).filter(lambda a, b: a > 1)
ds.collect()
ds.collect()

Hello!

I tried building Weld for Python 3 but had no success. I managed, however, to get pyweld and pygrizzly installed using python2 pip. Python 2 is deprecated and I would like to build pygrizzly and pyweld for Python 3.

Did you manage to build pyweld and pygrizzly for Python 3?

Thanks in advance!

Tuplex crashes when the the is keyword is used with a non-boolean or non-none type argument.

Example:

c = Context()
c.parallelize([1, 2, 3]).filter(lambda x: x is 2).collect()

Hello :)

Instead of opening a new issue I thought I would piggyback onto this one since it's very related. (Do let me know if you'd rather have me open a new one and I will do so.)

I'm following the Ubuntu 20.04 build options specified in the readme and I can't get past the python3 setup.py install step (I'm using Python 3.8). The error I get is the following:

    ERROR: Command errored out with exit status 1:
     command: /home/gorka/dev/tuplex/venv38/bin/python -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/home/gorka/dev/tuplex/setup.py'"'"'; __file__='"'"'/home/gorka/dev/tuplex/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' develop --no-deps
         cwd: /home/gorka/dev/tuplex/
    Complete output (62 lines):
    running develop
    running egg_info
    writing tuplex/python/tuplex.egg-info/PKG-INFO
    writing dependency_links to tuplex/python/tuplex.egg-info/dependency_links.txt
    writing requirements to tuplex/python/tuplex.egg-info/requires.txt
    writing top-level names to tuplex/python/tuplex.egg-info/top_level.txt
    reading manifest file 'tuplex/python/tuplex.egg-info/SOURCES.txt'
    reading manifest template 'MANIFEST.in'
    warning: no files found matching '*' under directory 'tuplex/libexec'
    warning: no files found matching '*' under directory 'tuplex/tuplex/libexec'
    warning: no files found matching '*.so'
    warning: no files found matching '*.dll'
    adding license file 'LICENSE'
    writing manifest file 'tuplex/python/tuplex.egg-info/SOURCES.txt'
    running build_ext
    -- Building dev version
    CMake Error at CMakeLists.txt:9 (project):
      Running
    
       '/tmp/pip-build-env-8hwvbpaz/overlay/bin/ninja' '--version'
    
      failed with:
    
       No such file or directory
    
    
    -- Configuring incomplete, errors occurred!
    See also "/home/gorka/dev/tuplex/build/temp.linux-x86_64-3.8/CMakeFiles/CMakeOutput.log".
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/home/gorka/dev/tuplex/setup.py", line 308, in <module>
        setup(name="tuplex",
      File "/tmp/pip-build-env-wpou7eah/overlay/lib/python3.8/site-packages/setuptools/__init__.py", line 153, in setup
        return distutils.core.setup(**attrs)
      File "/usr/lib/python3.8/distutils/core.py", line 148, in setup
        dist.run_commands()
      File "/usr/lib/python3.8/distutils/dist.py", line 966, in run_commands
        self.run_command(cmd)
      File "/usr/lib/python3.8/distutils/dist.py", line 985, in run_command
        cmd_obj.run()
      File "/tmp/pip-build-env-wpou7eah/overlay/lib/python3.8/site-packages/setuptools/command/develop.py", line 34, in run
        self.install_for_development()
      File "/tmp/pip-build-env-wpou7eah/overlay/lib/python3.8/site-packages/setuptools/command/develop.py", line 114, in install_for_development
        self.run_command('build_ext')
      File "/usr/lib/python3.8/distutils/cmd.py", line 313, in run_command
        self.distribution.run_command(command)
      File "/usr/lib/python3.8/distutils/dist.py", line 985, in run_command
        cmd_obj.run()
      File "/tmp/pip-build-env-wpou7eah/overlay/lib/python3.8/site-packages/setuptools/command/build_ext.py", line 79, in run
        _build_ext.run(self)
      File "/usr/lib/python3.8/distutils/command/build_ext.py", line 340, in run
        self.build_extensions()
      File "/usr/lib/python3.8/distutils/command/build_ext.py", line 449, in build_extensions
        self._build_extensions_serial()
      File "/usr/lib/python3.8/distutils/command/build_ext.py", line 474, in _build_extensions_serial
        self.build_extension(ext)
      File "/home/gorka/dev/tuplex/setup.py", line 205, in build_extension
        subprocess.check_call(
      File "/usr/lib/python3.8/subprocess.py", line 364, in check_call
        raise CalledProcessError(retcode, cmd)
    subprocess.CalledProcessError: Command '['cmake', '/home/gorka/dev/tuplex/tuplex', '-DPYTHON_EXECUTABLE=/home/gorka/dev/tuplex/venv38/bin/python', '-DCMAKE_BUILD_TYPE=Release', '-DPYTHON3_VERSION=3.8', '-GNinja']' returned non-zero exit status 1.
    configuring cmake with: cmake /home/gorka/dev/tuplex/tuplex -DPYTHON_EXECUTABLE=/home/gorka/dev/tuplex/venv38/bin/python -DCMAKE_BUILD_TYPE=Release -DPYTHON3_VERSION=3.8 -GNinja
    ----------------------------------------
ERROR: Command errored out with exit status 1: /home/gorka/dev/tuplex/venv38/bin/python -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/home/gorka/dev/tuplex/setup.py'"'"'; __file__='"'"'/home/gorka/dev/tuplex/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' develop --no-deps Check the logs for full command output.

CMakeFiles/CMakeOutput.log contains the following:

The system is: Linux - 5.4.0-89-generic - x86_64

Thank you in advance.

Cheers, Gorka.

Originally posted by @gorkaerana in https://github.com/tuplex/tuplex/issues/8#issuecomment-948719143

1. Add the following built-in functions in UDF: iterable can be a list, homogenous tuple, string, range or iterator object. (i) iter(iterable) : Returns an iterator. (ii) reversed(seq) : seq can be a list, homogenous tuple, string or range object. (iii) enumerate(iterable[, start]) : Start=0 by default if not provided. Returns an iterator with yieldType=(I64, iter(iterable).yieldType). (iv) zip(*iterables): Currently at least one iterable must be provided. Returns an iterator with yieldType=(iter(iterable_1).yieldType, iter(iterable_2).yieldType, ..., iter(iterable_N).yieldType). (v) next(iterator[, default]): Returns the next item from iterator. If iterator is exhausted, returns default if default is provided, otherwise raise StopIteration.

   Reference: Built-in Functions -- Python 3.9.7 documentation

2. Using unsupported types in the functions above (dictionary as iterable or seq, non-homogenous tuple as iterable or seq, default type in next call different from iterator.yieldType) or returning an iterator from UDF is resolved through fallback mode. Error handling codes are adapted from ASTHelper and now refactoring into IFailable.

3. Add support for using iterators (generated from iter, enumerate or zip) as testlist in for loops (i.e. for i in iteratorType: ...).

This is added support for the WebUI. It supports previous functionality of a much older version of Tuplex as well as new support for splitting the plan into stages, tracking stage dependencies, join operators, aggregate operators, and showing UDF's for filter operators.

It would be great if you could do a code review and tell me what I should change.

After installing in a venv with:

python -m pip install --upgrade --upgrade-strategy eager tuplex

... and attempting the first example:

python
from tuplex import *
Welcome to

  _____            _
 |_   _|   _ _ __ | | _____  __
   | || | | | '_ \| |/ _ \ \/ /
   | || |_| | |_) | |  __/>  <
   |_| \__,_| .__/|_|\___/_/\_\ 0.3.0
            |_|
    
using Python 3.8.10 (default, May  4 2021, 00:00:00) 
[GCC 10.2.1 20201125 (Red Hat 10.2.1-9)] on linux
 Interactive Shell mode
>>> c = Context()
Segmentation fault (core dumped)

Currently, if a resolver has a schema mismatch to its corresponding operator the job fails. However, this should be allowed - despite it basically means triggering the fallback path!

Bug discovered due to the random shuffling failure on CI, where basically tracing within the resolver triggered a schema mismatch.

Patching CVE-2007-4559

Hi, we are security researchers from the Advanced Research Center at Trellix. We have began a campaign to patch a widespread bug named CVE-2007-4559. CVE-2007-4559 is a 15 year old bug in the Python tarfile package. By using extract() or extractall() on a tarfile object without sanitizing input, a maliciously crafted .tar file could perform a directory path traversal attack. We found at least one unsantized extractall() in your codebase and are providing a patch for you via pull request. The patch essentially checks to see if all tarfile members will be extracted safely and throws an exception otherwise. We encourage you to use this patch or your own solution to secure against CVE-2007-4559. Further technical information about the vulnerability can be found in this blog.

If you have further questions you may contact us through this projects lead researcher Kasimir Schulz.

Aggregates should be improved (perhaps after struct type is enabled) and push down selects from both combine/aggregate UDFs. Also error handling has to be added and maybe higher level functions like count, mean, std, min, max, ... providing standard aggregates.