Meinheld is a high performance asynchronous WSGI Web Server (based on picoev)

Not sure whether/how meinheld is related but I'll post here to get a comment. I have been using meinheld happily for years for my home alarm system running on a raspberry pi. Some time ago I ported to Arch on a RPi2. Today I saw a core dump which I believe may be related to the relatively recent upgrade on Arch from python 3.4 to python 3.5:

pi@pi3:~ coredumpctl --no-pager info 
           PID: 220 (python)
           UID: 1000 (pi)
           GID: 1000 (pi)
        Signal: 11 (SEGV)
     Timestamp: Wed 2015-12-02 05:10:32 AEST (6h ago)
  Command Line: env/bin/python pialarm
    Executable: /usr/bin/python3.5
 Control Group: /user.slice/user-1000.slice/[email protected]/pialarm.service
          Unit: [email protected]
         Slice: user-1000.slice
     Owner UID: 1000 (pi)
       Boot ID: bcca24ad226540f8a6a397a9dc33ff85
    Machine ID: 663f6cee40174da8ab551172a29af2b2
      Hostname: pi3
      Coredump: /var/lib/systemd/coredump/core.python.1000.bcca24ad226540f8a6a397a9dc33ff85.220.1448997032000000.lz4
       Message: Process 220 (python) of user 1000 dumped core.

                Stack trace of thread 220:
                #0  0x0000000075428ec8 parse_http_request (server.cpython-35m-arm-linux-gnueabihf.so)
                #1  0x0000000075429094 read_callback (server.cpython-35m-arm-linux-gnueabihf.so)
                #2  0x00000000754347c8 picoev_poll_once_internal (server.cpython-35m-arm-linux-gnueabihf.so)
                #3  0x000000007542b4e4 picoev_loop_once (server.cpython-35m-arm-linux-gnueabihf.so)
                #4  0x0000000076d70068 PyCFunction_Call (libpython3.5m.so.1.0)

Presumably this may be due to greenlet? I see that meinheld requires specific greenlet version 0.4.5 although 0.4.9 is on PyPI. Any reason it can't use the most update to date package?

I am trying to use gunicorn with meinheld on a pypy3 flask interpreter.

1. would you recommend this architecture?
2. am facing this issue - could you help me what i have to do?

File "/opt/pypy/site-packages/meinheld/__init__.py", line 1, in <module>
    from meinheld.server import *
ImportError: /opt/pypy/site-packages/meinheld/server.pypy37-pp73-x86_64-linux-gnu.so: undefined symbol: PyFile_FromFd

I have installed the following packages

apt-get update && apt-get install -y \
apt-utils \
gcc \
nginx \
supervisor \
python3-pip \
build-essential \
curl \
zlib1g-dev \
libbz2-dev \
python3-dev \
python3.7-dev \
python-dev \ 
libffi-dev \ 
libevent-dev \
libssl-dev \ 
libgc-dev \
liblzma-dev \
python-gevent \ 
apt-file \
&& rm -rf /var/lib/apt/lists/*```

Some frameworks (e.g. Bottle and Flask) and maybe applications depend on threading.local for context-local storage. meinheld.patch.patch_all() does not provide a patch for threading.local and therefor is not enough to get these frameworks to work properly.

See https://github.com/bottlepy/bottle/issues/896 for an example.

Hey there, great job on this WSGI framework, thanks so much for sharing it 😄

I've discovered a minor issue when running Gunicorn with Meinheld workers in the foreground and hitting Ctrl+C to exit; it seems that KeyboardInterrupt isn't handled so we see an exception.

(venv) [centos@myserver my-benchmarks]$ gunicorn -w 1 --worker-class=meinheld.gmeinheld.MeinheldWorker api_falcon:app
[2018-05-16 12:39:07 +0000] [7341] [INFO] Starting gunicorn 19.8.1
[2018-05-16 12:39:07 +0000] [7341] [INFO] Listening at: http://127.0.0.1:8000 (7341)
[2018-05-16 12:39:07 +0000] [7341] [INFO] Using worker: meinheld.gmeinheld.MeinheldWorker
[2018-05-16 12:39:07 +0000] [7344] [INFO] Booting worker with pid: 7344
^C[2018-05-16 12:40:07 +0000] [7341] [INFO] Handling signal: int
[2018-05-16 12:40:07 +0000] [7344] [ERROR] Exception in worker process
Traceback (most recent call last):
  File "/home/centos/venv/lib64/python3.6/site-packages/gunicorn/arbiter.py", line 583, in spawn_worker
    worker.init_process()
  File "/home/centos/venv/lib64/python3.6/site-packages/gunicorn/workers/base.py", line 134, in init_process
    self.run()
  File "/home/centos/venv/lib64/python3.6/site-packages/meinheld/gmeinheld.py", line 46, in run
    server.run(self.wsgi)
KeyboardInterrupt
[2018-05-16 12:40:07 +0000] [7344] [INFO] Worker exiting (pid: 7344)
[2018-05-16 12:40:07 +0000] [7341] [INFO] Shutting down: Master
(venv) [centos@myserver my-benchmarks]$ 

I'm on Python 3.6.5 with the latest gunicorn and Meinheld from PyPI.

Here's my little api_falcon.py if it helps:

import falcon


class HelloResource(object):
    def on_get(self, req, resp):
        resp.status = falcon.HTTP_200
        resp.body = 'Hello world!'


app = falcon.API()
app.add_route('/', HelloResource())

Any help would be greatly appreciated 😄

Thanks Fotis

I have tested all the scenarios when my django application run without meinheld, and there is no exception. But exceptions occurs if application is run by meinheld.

Senario 1

The uploaded file is just a text file. It is wired that I uploaded a file of size 200kb successfully. Then I copy the content of this file and paste to double its size. The exception occurred when the second file is uploaded.

I tried to debug and found that function 'meinheld.input.read' returns a str which cannot concat to bytes.

The call stack is:

Traceback (most recent call last):
  File "/home/dev/virtualenv/python3/lib/python3.4/site-packages/django/core/handlers/base.py", line 132, in get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/home/dev/virtualenv/python3/lib/python3.4/site-packages/django/contrib/admin/options.py", line 616, in wrapper
    return self.admin_site.admin_view(view)(*args, **kwargs)
  File "/home/dev/virtualenv/python3/lib/python3.4/site-packages/django/views/decorators/cache.py", line 57, in _wrapped_view_func
    response = view_func(request, *args, **kwargs)
  File "/home/dev/virtualenv/python3/lib/python3.4/site-packages/django/contrib/admin/sites.py", line 233, in inner
    return view(request, *args, **kwargs)
  File "/home/dev/virtualenv/python3/lib/python3.4/site-packages/django/views/decorators/csrf.py", line 58, in wrapped_view
    return view_func(*args, **kwargs)
  File "/home/dev/d5_django_server/apps/product/admin.py", line 129, in change_view
    return self.changeform_view(request, object_id, form_url, extra_context)
  File "/home/dev/virtualenv/python3/lib/python3.4/site-packages/django/utils/decorators.py", line 34, in _wrapper
    return bound_func(*args, **kwargs)
  File "/home/dev/virtualenv/python3/lib/python3.4/site-packages/django/utils/decorators.py", line 106, in _wrapped_view
    result = middleware.process_view(request, view_func, args, kwargs)
  File "/home/dev/virtualenv/python3/lib/python3.4/site-packages/django/middleware/csrf.py", line 174, in process_view
    request_csrf_token = request.POST.get('csrfmiddlewaretoken', '')
  File "/home/dev/virtualenv/python3/lib/python3.4/site-packages/django/core/handlers/wsgi.py", line 137, in _get_post
    self._load_post_and_files()
  File "/home/dev/virtualenv/python3/lib/python3.4/site-packages/django/http/request.py", line 260, in _load_post_and_files
    self._post, self._files = self.parse_file_upload(self.META, data)
  File "/home/dev/virtualenv/python3/lib/python3.4/site-packages/django/http/request.py", line 225, in parse_file_upload
    return parser.parse()
  File "/home/dev/virtualenv/python3/lib/python3.4/site-packages/django/http/multipartparser.py", line 149, in parse
    for item_type, meta_data, field_stream in Parser(stream, self._boundary):
  File "/home/dev/virtualenv/python3/lib/python3.4/site-packages/django/http/multipartparser.py", line 626, in __iter__
    for sub_stream in boundarystream:
  File "/home/dev/virtualenv/python3/lib/python3.4/site-packages/django/http/multipartparser.py", line 439, in __next__
    return LazyStream(BoundaryIter(self._stream, self._boundary))
  File "/home/dev/virtualenv/python3/lib/python3.4/site-packages/django/http/multipartparser.py", line 466, in __init__
    unused_char = self._stream.read(1)
  File "/home/dev/virtualenv/python3/lib/python3.4/site-packages/django/http/multipartparser.py", line 337, in read
    out = b''.join(parts())
  File "/home/dev/virtualenv/python3/lib/python3.4/site-packages/django/http/multipartparser.py", line 330, in parts
    chunk = next(self)
  File "/home/dev/virtualenv/python3/lib/python3.4/site-packages/django/http/multipartparser.py", line 352, in __next__
    output = next(self._producer)
  File "/home/dev/virtualenv/python3/lib/python3.4/site-packages/django/http/multipartparser.py", line 414, in __next__
    data = self.flo.read(self.chunk_size)
  File "/home/dev/virtualenv/python3/lib/python3.4/site-packages/django/http/request.py", line 292, in read
    return self._stream.read(*args, **kwargs)
  File "/home/dev/virtualenv/python3/lib/python3.4/site-packages/django/core/handlers/wsgi.py", line 57, in read
    result = self.buffer + self._read_limited(size - len(self.buffer))
TypeError: can't concat bytes to str

Senario 2

When a jpg is uploaded, the exception is different.

The call stack is:

Traceback (most recent call last):
  File "/home/dev/virtualenv/python3/lib/python3.4/site-packages/django/core/handlers/base.py", line 132, in get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/home/dev/virtualenv/python3/lib/python3.4/site-packages/django/contrib/admin/options.py", line 616, in wrapper
    return self.admin_site.admin_view(view)(*args, **kwargs)
  File "/home/dev/virtualenv/python3/lib/python3.4/site-packages/django/views/decorators/cache.py", line 57, in _wrapped_view_func
    response = view_func(request, *args, **kwargs)
  File "/home/dev/virtualenv/python3/lib/python3.4/site-packages/django/contrib/admin/sites.py", line 233, in inner
    return view(request, *args, **kwargs)
  File "/home/dev/virtualenv/python3/lib/python3.4/site-packages/django/views/decorators/csrf.py", line 58, in wrapped_view
    return view_func(*args, **kwargs)
  File "/home/dev/d5_django_server/apps/product/admin.py", line 129, in change_view
    return self.changeform_view(request, object_id, form_url, extra_context)
  File "/home/dev/virtualenv/python3/lib/python3.4/site-packages/django/utils/decorators.py", line 34, in _wrapper
    return bound_func(*args, **kwargs)
  File "/home/dev/virtualenv/python3/lib/python3.4/site-packages/django/utils/decorators.py", line 106, in _wrapped_view
    result = middleware.process_view(request, view_func, args, kwargs)
  File "/home/dev/virtualenv/python3/lib/python3.4/site-packages/django/middleware/csrf.py", line 174, in process_view
    request_csrf_token = request.POST.get('csrfmiddlewaretoken', '')
  File "/home/dev/virtualenv/python3/lib/python3.4/site-packages/django/core/handlers/wsgi.py", line 137, in _get_post
    self._load_post_and_files()
  File "/home/dev/virtualenv/python3/lib/python3.4/site-packages/django/http/request.py", line 260, in _load_post_and_files
    self._post, self._files = self.parse_file_upload(self.META, data)
  File "/home/dev/virtualenv/python3/lib/python3.4/site-packages/django/http/request.py", line 225, in parse_file_upload
    return parser.parse()
  File "/home/dev/virtualenv/python3/lib/python3.4/site-packages/django/http/multipartparser.py", line 149, in parse
    for item_type, meta_data, field_stream in Parser(stream, self._boundary):
  File "/home/dev/virtualenv/python3/lib/python3.4/site-packages/django/http/multipartparser.py", line 626, in __iter__
    for sub_stream in boundarystream:
  File "/home/dev/virtualenv/python3/lib/python3.4/site-packages/django/http/multipartparser.py", line 439, in __next__
    return LazyStream(BoundaryIter(self._stream, self._boundary))
  File "/home/dev/virtualenv/python3/lib/python3.4/site-packages/django/http/multipartparser.py", line 466, in __init__
    unused_char = self._stream.read(1)
  File "/home/dev/virtualenv/python3/lib/python3.4/site-packages/django/http/multipartparser.py", line 337, in read
    out = b''.join(parts())
  File "/home/dev/virtualenv/python3/lib/python3.4/site-packages/django/http/multipartparser.py", line 330, in parts
    chunk = next(self)
  File "/home/dev/virtualenv/python3/lib/python3.4/site-packages/django/http/multipartparser.py", line 352, in __next__
    output = next(self._producer)
  File "/home/dev/virtualenv/python3/lib/python3.4/site-packages/django/http/multipartparser.py", line 414, in __next__
    data = self.flo.read(self.chunk_size)
  File "/home/dev/virtualenv/python3/lib/python3.4/site-packages/django/http/request.py", line 292, in read
    return self._stream.read(*args, **kwargs)
  File "/home/dev/virtualenv/python3/lib/python3.4/site-packages/django/core/handlers/wsgi.py", line 57, in read
    result = self.buffer + self._read_limited(size - len(self.buffer))
  File "/home/dev/virtualenv/python3/lib/python3.4/site-packages/django/core/handlers/wsgi.py", line 45, in _read_limited
    result = self.stream.read(size)
  File "/home/dev/virtualenv/python3/lib/python3.4/codecs.py", line 313, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 1412: invalid start byte

I am using Meinheld as Gunicorn worker for Django project. Setup is working on Python 3.5 under Ubuntu 16.04. Nginx is running as reverse proxy for Gunicorn.

Django tries to encode urls from utf-8 to latin-1 (ISO_8859_1) and fails.

I have tried to reproduce this error with Django's devserver and with Gunicorn default worker. Both doesn't have described problem.

Package versions:

meinheld (0.6.0)
gunicorn (19.6.0)
Django (1.10.2)

He was performing a test with meinheld, I was trying to test whether it is possible to address several requests at once. I see that the server listens only 1 to 1.

Is there any way to change this? I would like the requests received have the same context and not try a fork process.

I tried to use gunicorn + meinheld, now if I can handle several at a time, but do not share memory for me is a problem. I tried to put --threads 10, but to indicate that parameter only serves 1 to 1

from bottle import run, get
import time


@get("/test_yield")
def test_yield():
    print("init test_yield")
    for x in range(10):
        time.sleep(1)
        yield(str(x))
    print("end test_yield")

if __name__ == "__main__":
    run(server="meinheld", port="9876", host="0.0.0.0")

init test_yield
end test_yield
"127.0.0.1 - - [30/Jun/2016:11:32:59 +0000] "GET /test_yield HTTP/1.1" 200 10 "-" "Mozilla/5.0 (pc-x86_64-linux-gnu) Siege/3.0.5"
init test_yield
end test_yield
"127.0.0.1 - - [30/Jun/2016:11:33:09 +0000] "GET /test_yield HTTP/1.1" 200 10 "-" "Mozilla/5.0 (pc-x86_64-linux-gnu) Siege/3.0.5"

There's a pyscogreen module for installing a greenlet compatible function into pyscopg2 to get DB queries to run asynchronously. Because it's gevent (and libevent) it seems that meinheld would need it's own async monkey-patcher for pyscopg2. Just bringing it up.

We have this problem with kitsune (the project for support.mozilla.org). The stack is:

• Django 1.8.15
• Gunicorn 19.7.1
• Greenlet 0.4.12
• Meinheld 0.6.1 (as Gunicorn worker)

The problem seems to be that Django's built-in syndication framework treats the HttpResponse object like a writable file descriptor (which it's meant to be) which means the response content is actually a list of strings in the order in which they were written. Calling response.content is a property method that joins the list into a string. This response object works fine in all other WSGI servers I tested but Meinheld seems to lose the file descriptor, send a broken pipe error, and the response ends up being 0 length.

We've solved this for now by manually squashing the list of strings in the response to a single string, but this is merely a work around and not a real fix. I decided to file an issue in case anyone else has this problem and is as confused by it as we were. I'm not sure where the issue is in Meinheld, but perhaps some WSGI expert here can steer us in the right direction. I'm happy to file a pull-request but so far I've not been able to find the bug in the code.

Hey guys,

Had an ELB -> NGINX -> Gunicorn + Meinheld and kept getting a net::ERR_CONTENT_LENGTH_MISMATCH when I disabled cache via chrome console. Once I removed Meinheld everything worked. Not sure what the issue was, but here's the NGINX config and the run command.

Run command:

gunicorn -w $(($(nproc) * 2 + 1)) -t 60 --worker-class="egg:meinheld#gunicorn_worker" --bind 0.0.0.0:5000 app:app

Proxy config:

    map $http_upgrade $connection_upgrade {
        default        "upgrade";
        ""            "";
    }
    
    server {
        listen 80;

    	gzip on;
	    gzip_comp_level 4;
	    gzip_types text/html text/plain text/css application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript;

        if ($time_iso8601 ~ "^(\d{4})-(\d{2})-(\d{2})T(\d{2})") {
            set $year $1;
            set $month $2;
            set $day $3;
            set $hour $4;
        }
        access_log /var/log/nginx/healthd/application.log.$year-$month-$day-$hour healthd;

        access_log    /var/log/nginx/access.log;
    
        location / {
            proxy_pass            http://docker;
            proxy_http_version    1.1;
    
            proxy_set_header    Connection            $connection_upgrade;
            proxy_set_header    Upgrade                $http_upgrade;
            proxy_set_header    Host                $host;
            proxy_set_header    X-Real-IP            $remote_addr;
            proxy_set_header    X-Forwarded-For        $proxy_add_x_forwarded_for;
        }
    }

and NGINX.conf

user  nginx;
worker_processes  auto;

error_log  /var/log/nginx/error.log;

pid        /var/run/nginx.pid;

events {
    worker_connections  1024;
}

http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    access_log    /var/log/nginx/access.log;

    log_format  healthd '$msec"$uri"$status"$request_time"$upstream_response_time"$http_x_forwarded_for';

    include       /etc/nginx/conf.d/*.conf;
    include       /etc/nginx/sites-enabled/*;
}

Hi,

I've been using meinheld as a backend with chaussette to run my webapp. I've run into a problem that while trying to make a post request with the requests library I get the following exception:

SSLError: [Errno 2] _ssl.c:504: The operation did not complete (read)

But when I change the backend to waitress (for example) the problem is gone. So I thought it had something to do with meinheld.

Thanks

I'm facing a compatibility issue with the latest eventlet version 0.32.0 and above, 0.31.1 works fine. Other dependencies are gunicorn==20.1.0 meinheld==1.0.2 falcon==2.0.0 Below error, we are getting after each API call to the falcon server. Where workers get killed due to below error and then restarts itself.

2021-11-30 19:58:19 [66677] [INFO] worker received SIGABRT signal
Traceback (most recent call last):
  File "project_path/.tox/py39/lib/python3.9/site-packages/eventlet/hubs/hub.py", line 365, in run
    self.wait(sleep_time)
  File "project_path/.tox/py39/lib/python3.9/site-packages/eventlet/hubs/kqueue.py", line 96, in wait
    time.sleep(seconds)
  File "project_path/.tox/py39/lib/python3.9/site-packages/gunicorn/workers/base.py", line 203, in handle_abort
    sys.exit(1)
SystemExit: 1

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "project_path/.tox/py39/lib/python3.9/site-packages/meinheld/mlogging.py", line 187, in _access
    'h': environ.get('REMOTE_ADDR', '-'),
SystemError: <method 'get' of 'dict' objects> returned a result with an error set

Meinheld provides a wsgi.file_wrapper, however, it seems that the provided implementation does not conform to PEP 3333 (see the relevant chapter: Optional Platform-Specific File Handling).

If the provided file-like object implements .read(), but not .fileno(), Meinheld raises a TypeError, and disconnects the client. This seems to happen because PyObject_AsFileDescriptor() is used here indiscriminately, even if the provided file-like object doesn't have one: https://github.com/mopemope/meinheld/blob/6ddb22b52783d22ba15c6cdfbcbc9ed05b5c0c46/meinheld/server/response.c#L1267

The specification reads:

Of course, platform-specific file transmission APIs don't usually accept arbitrary "file-like" objects. Therefore, a wsgi.file_wrapper has to introspect the supplied object for things such as a fileno() (Unix-like OSes) or a java.nio.FileChannel (under Jython) in order to determine if the file-like object is suitable for use with the platform-specific API it supports.

Note that even if the object is not suitable for the platform API, the wsgi.file_wrapper must still return an iterable that wraps read() and close(), so that applications using file wrappers are portable across platforms.

I am using Gunicorn + Flask + Meinheld + NGINX with egg:meinheld#gunicorn_worker setting but when I try to upload more than 10 MB server fails with 502 Bad Gateway. I think Meinheld causes the issue because I can upload the same files with sync workers.

Exact error is:

[error] 4338#4338: *1 sendfile() failed (32: Broken pipe) while sending request to upstream, client: 127.0.0.1, server: localhost, request: "POST /******** HTTP/1.1", upstream: "http://127.0.0.1:8000/*********", host: "localhost", referrer: "http://localhost/******/"

Edit: When I disable Gunicorn and use only Meinheld, it works without an error.

Any ideas?

When using meinheld as gunicorn worker and using loguru as logger, memory is adding as requests increase. I have updated the gunicorn config by setting max_requests, but it does not work. And the gunicorn workers don't restart, could you help me?

greenlet==0.4.17 gunicorn==20.0.4 meinheld==1.0.2 loguru==0.5.3

Follow up on issue: https://github.com/mopemope/meinheld/issues/92 Can we disable the asyncIO feature from Meinheld conditionally? e.g by simply removing patch.patch_all()?