added: backwards compatible way to check if a ValueTarget exceeds a given length. This can help prevent nefarious clients exhausting server memory by uploading large amounts of data in form value fields.

added: test for max_length exceeded

• Makefile: add more targets, declare all dependencies, add help target
• update travis
• setup flake8 to ignore venv subdirectory
• remove annotation.html from git

Hi, sorry to intrude, but there may be some issues regarding recent evolution made by the pypa and the choice of poetry as the "packaging backend" (no offense). Information may be worth something on your side of other users of this package, don't know.

In an environment with pip==18.1, setuptools==40.8.0, gcc and some other libs, pip install streaming-form-data works without any issue, everything's fine. Well, you gotta have more than just pip and setuptools, but that happens.

In an environment with pip==20.1.1 (and higher) and setuptools==46.4.0, even with the gcc and libs, I got the following

root@cdb4ae3c174c:/app# pip install streaming-form-data
Collecting streaming-form-data
  Downloading streaming-form-data-1.7.0.tar.gz (92 kB)
     |████████████████████████████████| 92 kB 1.1 MB/s 
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Building wheels for collected packages: streaming-form-data
  Building wheel for streaming-form-data (PEP 517) ... error
  ERROR: Command errored out with exit status 1:
   command: /usr/local/bin/python /usr/local/lib/python3.8/site-packages/pip/_vendor/pep517/_in_process.py build_wheel /tmp/tmp5lcaherm
       cwd: /tmp/pip-install-zbvymr76/streaming-form-data
  Complete output (25 lines):
  A setup.py file already exists. Using it.
  Traceback (most recent call last):
    File "/tmp/pip-install-zbvymr76/streaming-form-data/setup.py", line 2, in <module>
      from setuptools import setup
  ModuleNotFoundError: No module named 'setuptools'
[...]
 ERROR: Failed building wheel for streaming-form-data
Failed to build streaming-form-data
ERROR: Could not build wheels for streaming-form-data which use PEP 517 and cannot be installed directly

At first I didn't paid too much attention, thought it was about the wheel and the pep 517, and tried things like --no-binary :all: and --no-binary streaming-form-data, also switching the use of PEP 517, to no avail alas. Before realizing it was not the wheel itself, but the direct install too.

Long story short, if you dig around the pypa discussions on github and so forth, it seems there is some issue with some packaging (especially PEP 517/poetry linked) and finding packages in build environments. Hence the "no setuptools found" issue at the beginning of the stacktrace. Plus there is quite some instability/uncertainty in the current state of Python packaging (once again, cf. pypa and recent evolution).

So for now my solution is

pip install poetry
pip --no-build-isolation streaming-form-data

But that's not so satisfying. Is there something that could be done streaming-form-data side ? I don't think so but it's worth a shot. Maybe there other way to configure pip so as to not have this issue (maybe something else than no binary or pep 517 related elements).

Hi, I found the speed of the parser to be rather low.

My desktop i7 CPU is parsing only approx 30 MB/s. And my intranet network is 1 Gbit. So I want this parser to work at least at 100 MB/s.

I will do better benchmark of current solution later.

I investigated the code and I'm sure I can do 100+ MB/s by changing cython code to be more C-like. So it will not read single bytes from Python objects, but will use raw C pointers instead.

Before starting work on pull request I want to ensure

1. speed improvements are important for you
2. rewriting cython code to more dangerous C-style code without bounds checking for every memory read operation is acceptable for this project in general

P.S. By the way, internal algorithm of matching boundaries in the stream is well done. It is very effective.

Hi!

I'm having trouble parsing a form streamed through uwsgi. I get the error below. I tried to track it down, but the closest I could get was that the byte was not a hyphen. The traceback seems to show it erroring there, but I verified the character being passed in was indeed a hyphen (45). Tried to print debugging info from within the .pyx, but couldn't get messages to show up in the terminal. Never used cython, so not sure of the internals.

Thanks in advance for your help.

~Sean

  File "/usr/local/lib/python2.7/dist-packages/upload/upload.py", line 388, in upload
    parser.data_received(b)
  File "/usr/local/lib/python2.7/dist-packages/streaming_form_data/parser.py", line 53, in data_received
    self._parser.data_received(data)
  File "streaming_form_data/_parser.pyx", line 150, in streaming_form_data._parser._Parser.data_received (streaming_form_data/_parser.c:3929)
  File "streaming_form_data/_parser.pyx", line 172, in streaming_form_data._parser._Parser.data_received (streaming_form_data/_parser.c:3875)
  File "streaming_form_data/_parser.pyx", line 184, in streaming_form_data._parser._Parser._parse (streaming_form_data/_parser.c:4148)
streaming_form_data._parser._Failed

$ python3 upload-test.py
Traceback (most recent call last):
  File "upload-test.py", line 10, in <module>
    from streaming_form_data import StreamingFormDataParser
  File "/usr/local/lib/python3.7/dist-packages/streaming_form_data/__init__.py", line 1, in <module>
    from streaming_form_data.parser import (  # NOQA
  File "/usr/local/lib/python3.7/dist-packages/streaming_form_data/parser.py", line 4, in <module>
    from streaming_form_data._parser import ErrorGroup, _Parser  # type: ignore
ImportError: /usr/local/lib/python3.7/dist-packages/streaming_form_data/_parser.so: undefined symbol: _Py_ZeroStruct

I used this: https://github.com/siddhantgoel/streaming-form-data/blob/master/examples/flask/upload-test.py

From line 200 of _parser.c there is a enum definition:

#ifdef SIZEOF_VOID_P
    enum { __pyx_check_sizeof_voidp = 1 / (int)(SIZEOF_VOID_P == sizeof(void*)) };
  #endif

which, when SIZEOF_VOID_P != sizeof(void*)), will raise an error. I compile your code on windows mingw-w64 using python setup.py build --compiler=mingw32 and get this error, which means SIZEOF_VOID_P is defined and SIZEOF_VOID_P != sizeof(void*). Is that a deliberate setting?

Command to reproduce:

docker run --entrypoint=pip3 python:3.6 install "streaming-form-data==1.5.1"

Error message / stack trace:

  ERROR: Command errored out with exit status 1:
   command: /usr/local/bin/python /usr/local/lib/python3.6/site-packages/pip/_vendor/pep517/_in_process.py get_requires_for_build_wheel /tmp/tmpv61zx81r
       cwd: /tmp/pip-wheel-j1vdg9g0/streaming-form-data
  Complete output (14 lines):
  Traceback (most recent call last):
    File "/usr/local/lib/python3.6/site-packages/pip/_vendor/pep517/_in_process.py", line 257, in <module>
      main()
    File "/usr/local/lib/python3.6/site-packages/pip/_vendor/pep517/_in_process.py", line 240, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
    File "/usr/local/lib/python3.6/site-packages/pip/_vendor/pep517/_in_process.py", line 91, in get_requires_for_build_wheel
      return hook(config_settings)
    File "/tmp/pip-build-env-ztxe98dq/overlay/lib/python3.6/site-packages/poetry/masonry/api.py", line 25, in get_requires_for_build_wheel
      poetry = Factory().create_poetry(Path("."))
    File "/tmp/pip-build-env-ztxe98dq/overlay/lib/python3.6/site-packages/poetry/factory.py", line 54, in create_poetry
      raise RuntimeError("The Poetry configuration is invalid:\n" + message)
  RuntimeError: The Poetry configuration is invalid:
    - Additional properties are not allowed ('url' was unexpected)

Hi,

I integrated the flask example of upload-test.py to airflow webserver UI as a uploading plugin, it allows user to upload a csv file within airflow webserver UI and save the file to server directory(''/usr/local/airflow/uploads/'), however the parser fails to get registered with any header information and the chunked data won't be written to file through the parser.

1. I have verified that the upload-test.py worked well on local flask host with @app.route.
2. I have verified the airflow plugin interface below functioned well when I using request.files and .save(path_to_save).

Here is the flask app(@expose) under airflow plugin:

class PipelineLauncher(AppBuilderBaseView): # from flask_appbuilder import BaseView as AppBuilderBaseView
    @expose('/', methods=('GET', 'POST'))
    def list(self):
        if request.method == 'POST':
            path_to_save = '/usr/local/airflow/uploads/temp.csv'   #path mounted with airflow
            
            file_ = FileTarget(path_to_save)
            parser = StreamingFormDataParser(headers=request.headers)
            parser.register('file', file_)

            while True:
                chunk = request.stream.read(8192)
                if not chunk:
                    break
                parser.data_received(chunk)
            
            #df = pd.read_csv(path_to_save) this will throw error 'pandas.errors.EmptyDataError: No columns to parse from file'
            #rows = df.shape[0]

            return self.render_template("debug.html", 
                                     path_to_save=path_to_save,
                                     file_object=file_,
                                     header=request.headers,
                                     filename=file_.multipart_filename,
                                     content_type=file_.multipart_content_type)
        return self.render_template("index.html")

# debug.html
# path_to_save: {{ path_to_save }}
# file_object:  {{ file_object }}
# header: {{ header }}
# filename: {{ filename }}
# content_type: {{ content_type }}
          
bp = Blueprint(
    "pipeline", __name__,
    template_folder='templates',
    static_folder='static',
    static_url_path='/static/pipeline_launcher')

class AirflowCustomLauncher(AirflowPlugin):
    name = "pipeline"
    pipeline_launcher = PipelineLauncher()
    pipeline_launcher_package = {
        "name": "Manual Upload Plugin",
        "category": "Launch Pipeline",
        "view": pipeline_launcher
    }
    appbuilder_views = [pipeline_launcher_package]
    admin_views = [pipeline_launcher_package]
    flask_blueprints = [bp]

index.html

{% include "airflow/master.html" %}
{% block body %} 
<title>Upload XLS/XLSX/CSV files to InfluxDB.</title>
<form method="post" class="admin-form form-horizontal" enctype="multipart/form-data" role='form'>
  <div class="col-md-12 text-center">
    <h3>Manual Upload for InfluxDB</h3>
    <br/>
    <p> This plugin currently only supports .csv, .xls, and .xlsx files. Larger files and .xlsx files will take longer than usual to process. Upon submitting a file, you will be taken to a page to preview your file as well as configure upload parameters. </p>
  </div>
  {% if csrf_token %}
  <input type="hidden" name="csrf_token" value="{{ csrf_token() }}" />
  {% endif %}
  <div class="form-group"> --> 
      <!-- You can take parameters from the user using the form elements and pass them to backend -->
    <label class="col-md-4 control-label">File: </label>
    <div class="col-md-6">
      <input class="form-control" type="file" name="file" />
    </div>
  </div>
  <div class="col-md-offset-4 col-md-10 submit-row">
    <button type="submit" class="btn btn-primary">Process File</button>
  </div>
  <div class="container">
    {% for message in get_flashed_messages() %}
      <div class="alert alert-warning">
        {{ message }}
      </div>
    {% endfor %}
  </div>
</form>
{% endblock %} 

The plugin allows me to choose a file to upload, and after I selected a csv file, here is the output from debug.html page:

path_to_save: /usr/local/airflow/uploads/temp.csv

file_object: <streaming_form_data.targets.FileTarget object at 0x7f7be38fb550>

header: Host: localhost:8080 Accept: text/html,application/xhtml+xml,application/xml;q=0.9,/;q=0.8 Accept-Encoding: gzip, deflate Accept-Language: en-us Content-Type: multipart/form-data; boundary=----WebKitFormBoundaryDSj0i1GXH4P0ITsx Origin: http://localhost:8080 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.1.1 Safari/605.1.15 Connection: keep-alive Upgrade-Insecure-Requests: 1 Referer: http://localhost:8080/pipelinelauncher/ Content-Length: 328842 Cookie: session=.eJwlj0tuQyEMRffCOAP-tt9mngzYbVSaVMAbRd17iSqPru5Hxy9z6pD5aQ7lPuVmznszhwnQFAGJrGvFh4aOxbP1PiOpT14qVScgBUKELATYgFJRpSSQhKu3pFmEi5Kr6kVjzM0iWqLkss9SiEKstlllZIGguGvIO6wCxWyQHxnf_JDHMsca10arc-i5nl_y2ISsEZLDrLW6nGy2GPYxSAQBTrYgqrftvdT445yL1zVPvfcl413vfTv9WbnLlnvyZq4p4_99Z37_AH8MU-Q.YTuY0A.t-_l07dcNPe_RN6CWI_Pg5cZ3vo

filename: None

content_type: None

Any help would be appreciated. Thank you.

When trying to install via pip3 on Ubuntu 20.04, which doesn't bundle Python2 headers by default, the compilation of _parser.c fails, as it's looking for Python.h in a Python2.7 directory. It can be worked around with apt install python2.7-dev. Not sure where the explicit Python2 reference is coming from - I couldn't spot it at first sight in this repo.

$ pip3 install --user streaming-form-data
Collecting streaming-form-data
  Using cached streaming-form-data-1.6.0.tar.gz (91 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Building wheels for collected packages: streaming-form-data
  Building wheel for streaming-form-data (PEP 517) ... error
  ERROR: Command errored out with exit status 1:
   command: /usr/bin/python3 /tmp/tmp0ldlmrgn build_wheel /tmp/tmpwhxvefb1                                                                                                                  
       cwd: /tmp/pip-install-nf4hxqoq/streaming-form-data                                                                                                                                   
  Complete output (53 lines):                                                                                                                                                               
  Traceback (most recent call last):                                                                                                                                                        
    File "/tmp/pip-build-env-uyrjh8h7/overlay/lib/python3.8/site-packages/poetry/utils/env.py", line 889, in _run                                                                           
      output = subprocess.check_output(                                                                                                                                                     
    File "/usr/lib/python3.8/subprocess.py", line 411, in check_output                                                                                                                      
      return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,                                                                                                                      
    File "/tmp/pip-build-env-uyrjh8h7/overlay/lib/python3.8/site-packages/poetry/utils/_compat.py", line 205, in run                                                                        
      raise CalledProcessError(                                                                                                                                                             
  poetry.utils._compat.CalledProcessError: Command '['/usr/bin/python', 'setup.py', 'build', '-b', 'build']' returned non-zero exit status 1.                                               
                                                                                                                                                                                            
  During handling of the above exception, another exception occurred:                                                                                                                       
                                                                                                                                                                                            
  Traceback (most recent call last):                                                                                                                                                        
    File "/tmp/tmp0ldlmrgn", line 280, in <module>                                                                                                                                          
      main()                                                                                                                                                                                
    File "/tmp/tmp0ldlmrgn", line 263, in main                                                                                                                                              
      json_out['return_val'] = hook(**hook_input['kwargs'])                                                                                                                                 
    File "/tmp/tmp0ldlmrgn", line 204, in build_wheel                                                                                                                                       
      return _build_backend().build_wheel(wheel_directory, config_settings,                                                                                                                 
    File "/tmp/pip-build-env-uyrjh8h7/overlay/lib/python3.8/site-packages/poetry/masonry/api.py", line 62, in build_wheel                                                                   
      WheelBuilder.make_in(                                                                                                                                                                 
    File "/tmp/pip-build-env-uyrjh8h7/overlay/lib/python3.8/site-packages/poetry/masonry/builders/wheel.py", line 55, in make_in                                                            
      wb.build()                                                                                                                                                                            
    File "/tmp/pip-build-env-uyrjh8h7/overlay/lib/python3.8/site-packages/poetry/masonry/builders/wheel.py", line 81, in build                                                              
      self._build(zip_file)                                                                                                                                                                 
    File "/tmp/pip-build-env-uyrjh8h7/overlay/lib/python3.8/site-packages/poetry/masonry/builders/wheel.py", line 103, in _build                                                            
      self._env.run(                                                                                                                                                                        
    File "/tmp/pip-build-env-uyrjh8h7/overlay/lib/python3.8/site-packages/poetry/utils/env.py", line 856, in run                                                                            
      return self._run(cmd, **kwargs)                                                                                                                                                       
    File "/tmp/pip-build-env-uyrjh8h7/overlay/lib/python3.8/site-packages/poetry/utils/env.py", line 893, in _run                                                                           
      raise EnvCommandError(e, input=input_)                                                                                                                                                
  poetry.utils.env.EnvCommandError: Command ['/usr/bin/python', 'setup.py', 'build', '-b', 'build'] errored with the following return code 1, and output:                                   
  running build                                                                                                                                                                             
  running build_py                                                                                                                                                                          
  creating build                                                                                                                                                                            
  creating build/lib.linux-x86_64-2.7                                                                                                                                                       
  creating build/lib.linux-x86_64-2.7/streaming_form_data                                                                                                                                   
  copying streaming_form_data/validators.py -> build/lib.linux-x86_64-2.7/streaming_form_data                                                                                               
  copying streaming_form_data/targets.py -> build/lib.linux-x86_64-2.7/streaming_form_data                                                                                                  
  copying streaming_form_data/parser.py -> build/lib.linux-x86_64-2.7/streaming_form_data                                                                                                   
  copying streaming_form_data/__init__.py -> build/lib.linux-x86_64-2.7/streaming_form_data                                                                                                 
  copying streaming_form_data/_parser.pyx -> build/lib.linux-x86_64-2.7/streaming_form_data                                                                                                 
  copying streaming_form_data/_parser.c -> build/lib.linux-x86_64-2.7/streaming_form_data                                                                                                   
  running build_ext                                                                                                                                                                         
  building 'streaming_form_data._parser' extension                                                                                                                                          
  creating build/temp.linux-x86_64-2.7                                                                                                                                                      
  creating build/temp.linux-x86_64-2.7/streaming_form_data                                                                                                                                  
  x86_64-linux-gnu-gcc -pthread -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fdebug-prefix-map=/build/python2.7-1x6jhf/python2.7-2.7.18~rc1=. -fstack-protector-strong -Wformat -Werror=format-security -fPIC -I/usr/include/python2.7 -c streaming_form_data/_parser.c -o build/temp.linux-x86_64-2.7/streaming_form_data/_parser.o                                         
  streaming_form_data/_parser.c:4:10: fatal error: Python.h: Datei oder Verzeichnis nicht gefunden                                                                                          
      4 | #include "Python.h"                                                                                                                                                               
        |          ^~~~~~~~~~                                                                                                                                                               
  compilation terminated.                                                                                                                                                                   
  error: command 'x86_64-linux-gnu-gcc' failed with exit status 1                                                                                                                           
                                                                                                                                                                                            
  ----------------------------------------                                                                                                                                                  
  ERROR: Failed building wheel for streaming-form-data
Failed to build streaming-form-data
ERROR: Could not build wheels for streaming-form-data which use PEP 517 and cannot be installed directly

Hey @siddhantgoel, thanks for the great library! This PR contains a fix to address an issue with the content-type parsing affecting the content-type field in the ENDING_HEADER part.

Python's string.lstrip function is not doing what is expected at the specific position, resulting in the fact that content-type: text/plain is modified to xt/plain instead of text/plain. The test is modified to check for different content-types as image/png is by chance not affected by the issue. Using replace instead of lstrip yields the correct result.

You can test the incorrect behavior by starting a Python console and executing 'content-type: text/plain'.lstrip('content-type: ') which returns xt/plain. 'content-type: text/plain'.replace('content-type: ', '') on the other hand returns text/plain.

I think It would be great if you move the c declarations to a .pxd file to be available to other cython modules as in my case I don't need to use it as a python module to minimize the python interactions as possible.

How does this library handle multi-valued fields? For example, if the input form allows for the entry of an arbitrary number of "people", each of which has a field height, weight, etc?

If I understand the source code correctly, as data comes in it is simply appended to an array (for ValueTarget()), which is joined to a single bytes string when value is called, which would leave me with something like 14022036 if there were three weight values submitted (140,220,36), with no way to know where to split the values.

I could make a custom subclass that doesn't join the array, but that would only work if I could be sure each "chunk" was a full, discrete, value, which I do not believe is the case...

Building async support into this package would make a lot of sense. One use case, for example, would be to do networked IO in parallel while file chunks are being uploaded. This is required for S3 uploads (or basically uploads to any other file hosting service).

The API for how this should look like is not yet clear. Async code (at least how Python implements it) tends to "split" the codebase into sync and async parts, so we would need to figure that one out first.

There's a discussion and some ideas in https://github.com/siddhantgoel/streaming-form-data/pull/29 .

An S3Target class would be useful to stream file contents directly from the request to an S3 bucket. This actually shouldn't be too hard to write using boto3.