Developer-friendly asynchrony for Django

Related tags

Django channels
Overview

Django Channels

https://github.com/django/channels/workflows/Tests/badge.svg?branch=master https://readthedocs.org/projects/channels/badge/?version=latest

Channels augments Django to bring WebSocket, long-poll HTTP, task offloading and other async support to your code, using familiar Django design patterns and a flexible underlying framework that lets you not only customize behaviours but also write support for your own protocols and needs.

Documentation, installation and getting started instructions are at https://channels.readthedocs.io

Channels is an official Django Project and as such has a deprecation policy. Details about what's deprecated or pending deprecation for each release is in the release notes.

Support can be obtained through several locations - see our support docs for more.

You can install channels from PyPI as the channels package. See our installation and tutorial docs for more.

Dependencies

All Channels projects currently support Python 3.6 and up. channels is compatible with Django 2.2, 3.0, and 3.1.

Contributing

To learn more about contributing, please read our contributing docs.

Maintenance and Security

To report security issues, please contact [email protected]. For GPG signatures and more security process information, see https://docs.djangoproject.com/en/dev/internals/security/.

To report bugs or request new features, please open a new GitHub issue. For larger discussions, please post to the django-developers mailing list.

Maintenance is overseen by Carlton Gibson with help from others. It is a best-effort basis - we unfortunately can only dedicate guaranteed time to fixing security holes.

If you are interested in joining the maintenance team, please read more about contributing and get in touch!

Other Projects

The Channels project is made up of several packages; the others are:

  • Daphne, the HTTP and Websocket termination server
  • channels_redis, the Redis channel backend
  • asgiref, the base ASGI library/memory backend
Issues
  • Idle database connections not reaped

    Idle database connections not reaped

    Initially reported in https://github.com/django/daphne/issues/163

    I suspect this is due to the lack of calling close_old_connections, as seen in Channels 1 here: https://github.com/django/channels/blob/1.x/channels/signals.py#L10

    bug exp/beginner 
    opened by andrewgodwin 54
  • Add ChannelLiveServerTestCase

    Add ChannelLiveServerTestCase

    As we discussed earlier in #494 we need live server test case for example to run selenium against it. This is work in progress PR, but I want some feedback already.

    Few questions we need to answer before merging it.

    1. Channels documentation mention test_channel_aliases as ability to test complex layers setup. I don't know what to do in this case for live server test. It is possible to spawn multiple live server instances. But its not clear to me what user API we should provide.

    2. Regular live server thread contains connection override logic. I don't understand right now what problem does it solve. It will be awesome if any one can describe the reason it present there or provide test case for it.

    3. How we can validate that twisted bind port successfully? For now I select first port from the possible range without any validation.

    4. apply_routes setup test instance with overriding setUp and tearDown methods. Live server spawns its threads in the test class setup (as original one do). So apply routes hook works after we start live server threads and overrides not available in those threads. For now I change apply_routes decorator to work on class basis. But we also can consider running live server for each individual test method however this will slow things down and inconsistent with django behavior.

    5. We need to write docs.

    cc @bittner

    opened by proofit404 54
  • Seeing HTTP/WS send decode error after upgrading to channels 1.1.3

    Seeing HTTP/WS send decode error after upgrading to channels 1.1.3

    I just upgraded Django channels to 1.1.3 as well as the following related libraries:

    channels==1.1.3 daphne==1.2.0 asgi-redis==1.3.0 asgiref==1.1.1

    Everything ran fine in my dev environment. But once I pushed to production, installed the new libraries, and completely restarted all services, I immediately started seeing the following error:

    ERROR: HTTP/WS send decode error: Cannot dispatch message on channel u'daphne.response.dUTtOgoynk!bNiNpvqevU'
    

    The client was showing the following error:

    WebSocket connection to 'wss://<url>.com/' failed: Error during WebSocket handshake: Unexpected response code: 403
    

    When I downgraded to the previous library versions, everything worked fine again.

    It appears the issue is during the websocket connect / initial handshake?

    I'm using the enforce_ordering decorator on all my websocket consumers as well as the asgi redis back-end.

    Any thoughts on what's going on?

    bug blocked/needs-investigation exp/intermediate 
    opened by sachinrekhi 53
  • ChannelsLiveServerTestCase ReactorNotRestartable exception

    ChannelsLiveServerTestCase ReactorNotRestartable exception

    Yes, I updated all channels related stuff this morning :-)

    Traceback (most recent call last):
      File "/home/brian/.pyenv/versions/3.6.4/envs/myrocc/lib/python3.6/site-packages/nose/suite.py", line 210, in run
        self.setUp()
      File "/home/brian/.pyenv/versions/3.6.4/envs/myrocc/lib/python3.6/site-packages/nose/suite.py", line 293, in setUp
        self.setupContext(ancestor)
      File "/home/brian/.pyenv/versions/3.6.4/envs/myrocc/lib/python3.6/site-packages/nose/suite.py", line 316, in setupContext
        try_run(context, names)
      File "/home/brian/.pyenv/versions/3.6.4/envs/myrocc/lib/python3.6/site-packages/nose/util.py", line 471, in try_run
        return func()
      File "/home/brian/.pyenv/versions/3.6.4/envs/myrocc/lib/python3.6/site-packages/django/test/testcases.py", line 1352, in setUpClass
        raise cls.server_thread.error
      File "/home/brian/.pyenv/versions/3.6.4/envs/myrocc/lib/python3.6/site-packages/channels/testing/live.py", line 28, in run
        self.daphne.run()
      File "/home/brian/.pyenv/versions/3.6.4/envs/myrocc/lib/python3.6/site-packages/daphne/server.py", line 132, in run
        reactor.run(installSignalHandlers=self.signal_handlers)
      File "/home/brian/.pyenv/versions/3.6.4/envs/myrocc/lib/python3.6/site-packages/twisted/internet/asyncioreactor.py", line 266, in run
        self.startRunning(installSignalHandlers=installSignalHandlers)
      File "/home/brian/.pyenv/versions/3.6.4/envs/myrocc/lib/python3.6/site-packages/twisted/internet/base.py", line 1222, in startRunning
        ReactorBase.startRunning(self)
      File "/home/brian/.pyenv/versions/3.6.4/envs/myrocc/lib/python3.6/site-packages/twisted/internet/base.py", line 730, in startRunning
        raise error.ReactorNotRestartable()
    twisted.internet.error.ReactorNotRestartable: 
    
    bug exp/advanced 
    opened by brianmay 47
  • Multiple requests to synchronous views are handled sequentially, not concurrently. Django 3.1.3, Channels 3.0.2, Daphne 3.0.1.

    Multiple requests to synchronous views are handled sequentially, not concurrently. Django 3.1.3, Channels 3.0.2, Daphne 3.0.1.

    Pipfile:

    
    [packages]
    Django = "==3.1.3"
    asgiref = "==3.2.10"
    channels = "==3.0.2"
    daphne = "==3.0.1"
    

    (Note: asgiref 3.3.10 was also tried)

    Code / Minimal example

    I started a completely new Django + Django channels project to verify I can still see the same behaviour.

    views.py:

    
    def view_1(request):
        for i in range(10):
            time.sleep(1)
            print(request, i)
    
        return HttpResponse(b'Hello World #1!')
    
    
    def view_2(request):
        return HttpResponse(b'Hello World #2!')
    

    asgi.py:

    
    import os
    
    from channels.routing import ProtocolTypeRouter, URLRouter
    from django.core.asgi import get_asgi_application
    from django.urls import re_path
    
    os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'channels_test.settings')
    
    application = ProtocolTypeRouter({
        'http': URLRouter([
            re_path('', get_asgi_application())
        ])
    })
    

    urls.py:

    
    urlpatterns = [
        path('admin/', admin.site.urls),
        path('view1/', views.view_1),
        path('view2/', views.view_2),
    ]
    

    I have two Django views. View1 and View2. Both are synchronous. As you can see, View1 sleeps for 10 seconds before returning a response. Calls to View2 respond instantaneously.

    How you're running Channels (runserver? daphne/runworker? Nginx/Apache in front?)

    Issue is seen in both runserver and also when running a single daphne worker. Nginx is serving the requests.

    What Happened

    If I open up a browser and go to localhost/view1, I can see in the console the printouts of the sleep for 10 seconds. If, while view1 is still processing, I go to localhost/view2, the page does not respond until view1 has finished processing.

    What I expected

    I should not expect View2 to have to wait for View1 to finish before returning a response, the requests should run concurrently. If I revert back to channels 2.4.0 with the following pip setup:

    Pipfile:

    
    [packages]
    Django = "==3.1.3"
    asgiref = "==3.2.10"
    channels = "==2.4.0"
    daphne = "==2.5.0"
    

    and the following asgi.py:

    
    import os
    
    from channels.routing import ProtocolTypeRouter, URLRouter
    from channels.routing import AsgiHandler
    from django.urls import re_path
    
    os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'channels_test.settings')
    
    import django
    django.setup()
    application = ProtocolTypeRouter({
        'http': URLRouter([
            re_path('', AsgiHandler)
        ])
    })
    
    

    I no longer see the behaviour of the ticket. As I did not expect a regression in behaviour for synchronous views after upgrading django / channels, I expected to see this same older behaviour after upgrading channels.

    Workarounds

    1. Convert the synchronous views to asynchronous views. This is not feasible for me as my entire project uses django rest framework which does not support asynchronous views (as Django does not provide async generic views).

    2. Reverting back to Channels 2.X / Daphne 2.X.

    Investigation / Notes

    • While I know Django introduced async views, I did not expect a regression in behaviour for existing uses of synchronous views with ASGI. I did not see documentation indicating this behaviour should be expected.

    • For my project, all APIs sent are now processed sequentially resulting in a very unresponsive front-end.

    • I am certain this is down to the use of Django's ASGIHandler and its use of thread_sensitive=True in sync_to_async. Synchronous views are all placed in the same thread with thread_sensitive=True, so I'm assuming this is the cause of the issue - and wasn't an issue with channel's own version of ASGIHandler. If I access django.core.handlers.base and modify thread_sensitive of _get_response_async to False, the view responds instantly, like normal:

    Line 228 of django.core.handlers.base

    
                if not asyncio.iscoroutinefunction(wrapped_callback):
                    wrapped_callback = sync_to_async(wrapped_callback, thread_sensitive=False)
    
    
    • I wasn't sure whether to report as a Channels or a Django issue. As using Django's get_asgi_application is part of the recommended way of upgrading channels from 2.X to 3.X, and this is all part of the Django ecosystem, it felt more appropriate here.
    opened by StefanCardnell 42
  • Cannot call AsyncToSync twice in one sync context for channels_redis

    Cannot call AsyncToSync twice in one sync context for channels_redis

    I am using a SyncConsumer which adds itself to a group on websocket.connect with group_add method of channel_layer.

    class TaskConsumer(JsonWebsocketConsumer):
        def connect(self):
            self.accept()
            AsyncToSync(self.channel_layer.group_add)('task-i-1', self.channel_name)
    
        def task_message(self, event):
            self.send_json(event["text"])
    

    If I try to send message to this group using group_send method that is wrapped with AsyncToSync, first message is succeed but further messages throw this exception:

    >>> c = get_channel_layer() >>> AsyncToSync(c.group_send)('task-i-1', {'type': 'task.message', 'text':{}}) >>> AsyncToSync(c.group_send)('task-i-1', {'type': 'task.message', 'text':{}}) Connection <RedisConnection [db:0]> has pending commands, closing it. Traceback (most recent call last): File "/home/ahmet/websocket_channel/env/lib64/python3.6/site-packages/channel s_redis/core.py", line 316, in group_send await connection.zremrangebyscore(key, min=0, max=int(time.time()) - self.group_expiry) RuntimeError: Task <Task pending coro=<AsyncToSync.main_wrap() running at /home/ahmet/webso cket_channel/env/lib64/python3.6/site-packages/asgiref/sync.py:57> cb=[_run_until_complete_cb() at /usr/lib64/python3.6/asyncio/base_events.py:176]> got Future attached to a different loop

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last): File "", line 1, in File "/home/ahmet/websocket_channel/env/lib64/python3.6/site-packages/asgiref /sync.py", line 49, in call call_result.result() File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 425, in result return self.__get_result() File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result raise self._exception File "/home/ahmet/websocket_channel/env/lib64/python3.6/site-packages/asgiref /sync.py", line 57, in main_wrap result = await self.awaitable(*args, **kwargs) File "/home/ahmet/websocket_channel/env/lib64/python3.6/site-packages/channel s_redis/core.py", line 320, in group_send await connection.zrange(key, 0, -1) File "/home/ahmet/websocket_channel/env/lib64/python3.6/site-packages/aioredi s/commands/init.py", line 152, in exit self._release_callback(conn) File "/home/ahmet/websocket_channel/env/lib64/python3.6/site-packages/aioredi s/pool.py", line 361, in release conn.close() File "/home/ahmet/websocket_channel/env/lib64/python3.6/site-packages/aioredi s/connection.py", line 352, in close self._do_close(ConnectionForcedCloseError()) File "/home/ahmet/websocket_channel/env/lib64/python3.6/site-packages/aioredi s/connection.py", line 359, in _do_close self._writer.transport.close() File "/usr/lib64/python3.6/asyncio/selector_events.py", line 621, in close self._loop.call_soon(self._call_connection_lost, None) File "/usr/lib64/python3.6/asyncio/base_events.py", line 574, in call_soon self._check_closed() File "/usr/lib64/python3.6/asyncio/base_events.py", line 357, in _check_closed raise RuntimeError('Event loop is closed') RuntimeError: Event loop is closed

    • OpenSuse Leap(42.2)
    • python[3.6.4], django[2.0.2], channels[2.0.0], daphne[2.0.2], channels-redis[2.0.2]. asgiref[2.1.3]
    • Django running with runserver
    bug exp/intermediate 
    opened by ahaltindis 41
  • Install channels without daphne

    Install channels without daphne

    Currently channels has daphne as dependency. So it is not possible to install channels on one machine without installing daphne with its dependencies.

    The only place in code of channels, which needs daphne, is the runserver command, which kind of a development command that should not be used in production. So there is no need to require daphne as a requirment for a production setup.

    Could it be possible to create a package, that you can install without installing daphne as well?

    Our background is as follows: We have a django-channels app, that has be used also on windows from low-tech-users. But it is quite hard to install a current version of twisted on windows. So we build geiss as an replacement for daphne, which does not need twisted. But with the current package of channels, it is not possible to get rid of twisted, even when we don't want to use daphne.

    But I think, that there are other use cases where someone wants to install channels without daphne. For example if you want to separate the worker on different containers/hardware then the protocol server.

    enhancement 
    opened by ostcar 39
  • Testing with pytest-asyncio raises RuntimeError: Event loop is closed

    Testing with pytest-asyncio raises RuntimeError: Event loop is closed

    Hi, this is a duplicate of this question.

    I'm trying to test new channels 2.0 with pytest-asyncio (0.8.0). If I place different assertions in the same function like:

    import json
    import pytest
    from concurrent.futures._base import TimeoutError
    from channels.testing import WebsocketCommunicator
    from someapp.consumers import MyConsumer
    
    
    @pytest.mark.django_db
    @pytest.mark.asyncio
    async def setup_database_and_websocket():
        path = 'foo'
        communicator = WebsocketCommunicator(MyConsumer, path)
        connected, subprotocol = await communicator.connect()
        assert connected
        return communicator
    
    
    @pytest.mark.django_db
    @pytest.mark.asyncio
    async def test_1_and_2():
        communicator = await setup_database_and_websocket()
        sent = {"message": 'abc'}
        await communicator.send_json_to(sent)
        with pytest.raises(TimeoutError):
            await communicator.receive_from()
        await communicator.send_input({
            "type": "websocket.disconnect",
            "code": 1000,
        })
    
        communicator = await setup_database_and_websocket()
        sent = {"message": 1}
        await communicator.send_json_to(sent)
        with pytest.raises(TimeoutError):
            await communicator.receive_from()
        await communicator.send_input({
            "type": "websocket.disconnect",
            "code": 1000,
        })
    

    then I'm not getting an error. But if I separate test cases like:

    @pytest.mark.django_db
    @pytest.mark.asyncio
    async def test_1():
        communicator = await setup_database_and_websocket()
        sent = {"message": 'abc'}
        await communicator.send_json_to(sent)
        with pytest.raises(TimeoutError):
            await communicator.receive_from()
        await communicator.send_input({
            "type": "websocket.disconnect",
            "code": 1000,
        })
    
    
    @pytest.mark.django_db
    @pytest.mark.asyncio
    async def test_2():
        communicator = await setup_database_and_websocket()
        sent = {"message": 1}
        await communicator.send_json_to(sent)
        with pytest.raises(TimeoutError):
            await communicator.receive_from()
        await communicator.send_input({
            "type": "websocket.disconnect",
            "code": 1000,
        })
    

    then I'm getting following error upon the second receive_form call:

    with pytest.raises(TimeoutError):
    >           await communicator.receive_from()
    
    someapp/tests/test_consumers_async.py:106: 
    _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
    ../../../.virtualenvs/some_env/lib/python3.6/site-packages/channels/testing/websocket.py:71: in receive_from
    response = await self.receive_output(timeout)
    ../../../.virtualenvs/some_env/lib/python3.6/site-packages/asgiref/testing.py:66: in receive_output
    self.future.result()
    ../../../.virtualenvs/some_env/lib/python3.6/site-packages/channels/consumer.py:54: in __call__
    await await_many_dispatch([receive, self.channel_receive], self.dispatch)
    ../../../.virtualenvs/some_env/lib/python3.6/site-packages/channels/utils.py:48: in await_many_dispatch
    await dispatch(result)
    ../../../.virtualenvs/some_env/lib/python3.6/site-packages/asgiref/sync.py:95: in __call__
    return await asyncio.wait_for(future, timeout=None)
    /usr/lib/python3.6/asyncio/tasks.py:339: in wait_for
    return (yield from fut)
    /usr/lib/python3.6/concurrent/futures/thread.py:56: in run
    result = self.fn(*self.args, **self.kwargs)
    ../../../.virtualenvs/some_env/lib/python3.6/site-packages/channels/db.py:13: in thread_handler
    return super().thread_handler(loop, *args, **kwargs)
    ../../../.virtualenvs/some_env/lib/python3.6/site-packages/asgiref/sync.py:110: in thread_handler
    return self.func(*args, **kwargs)
    ../../../.virtualenvs/some_env/lib/python3.6/site-packages/channels/consumer.py:99: in dispatch
    handler(message)
    ../../../.virtualenvs/some_env/lib/python3.6/site-packages/channels/generic/websocket.py:19: in websocket_connect
    self.connect()
    someapp/consumers.py:22: in connect
    self.group_name, self.channel_name)
    ../../../.virtualenvs/some_env/lib/python3.6/site-packages/asgiref/sync.py:49: in __call__
    return call_result.result()
    /usr/lib/python3.6/concurrent/futures/_base.py:432: in result
    return self.__get_result()
    /usr/lib/python3.6/concurrent/futures/_base.py:384: in __get_result
    raise self._exception
    ../../../.virtualenvs/some_env/lib/python3.6/site-packages/asgiref/sync.py:63: in main_wrap
    result = await self.awaitable(*args, **kwargs)
    ../../../.virtualenvs/some_env/lib/python3.6/site-packages/channels_redis/core.py:290: in group_add
    await connection.expire(group_key, self.group_expiry)
    ../../../.virtualenvs/some_env/lib/python3.6/site-packages/aioredis/commands/__init__.py:152: in __exit__
    self._release_callback(conn)
    ../../../.virtualenvs/some_env/lib/python3.6/site-packages/aioredis/pool.py:361: in release
    conn.close()
    ../../../.virtualenvs/some_env/lib/python3.6/site-packages/aioredis/connection.py:352: in close
    self._do_close(ConnectionForcedCloseError())
    ../../../.virtualenvs/some_env/lib/python3.6/site-packages/aioredis/connection.py:359: in _do_close
    self._writer.transport.close()
    /usr/lib/python3.6/asyncio/selector_events.py:621: in close
    self._loop.call_soon(self._call_connection_lost, None)
    /usr/lib/python3.6/asyncio/base_events.py:574: in call_soon
    self._check_closed()
    _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
    
    self = <_UnixSelectorEventLoop running=False closed=True debug=False>
    
    def _check_closed(self):
    if self._closed:
    >           raise RuntimeError('Event loop is closed')
    E           RuntimeError: Event loop is closed
    
    /usr/lib/python3.6/asyncio/base_events.py:357: RuntimeError
    

    Also if I do (as in https://channels.readthedocs.io/en/latest/topics/testing.html):

    await communicator.disconnect()
    

    instead of:

    await communicator.send_input({
        "type": "websocket.disconnect",
        "code": 1000,
    })
    

    then the following error is raised:

    >       await communicator.disconnect()
    
    someapp/tests/test_consumers_async.py:96: 
    _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
    ../../../.virtualenvs/some_env/lib/python3.6/site-packages/channels/testing/websocket.py:100: in disconnect
        await self.future
    ../../../.virtualenvs/some_env/lib/python3.6/site-packages/channels/consumer.py:54: in __call__
        await await_many_dispatch([receive, self.channel_receive], self.dispatch)
    ../../../.virtualenvs/some_env/lib/python3.6/site-packages/channels/utils.py:48: in await_many_dispatch
        await dispatch(result)
    ../../../.virtualenvs/some_env/lib/python3.6/site-packages/asgiref/sync.py:95: in __call__
        return await asyncio.wait_for(future, timeout=None)
    /usr/lib/python3.6/asyncio/tasks.py:339: in wait_for
        return (yield from fut)
    /usr/lib/python3.6/concurrent/futures/thread.py:56: in run
        result = self.fn(*self.args, **self.kwargs)
    ../../../.virtualenvs/some_env/lib/python3.6/site-packages/channels/db.py:13: in thread_handler
        return super().thread_handler(loop, *args, **kwargs)
    ../../../.virtualenvs/some_env/lib/python3.6/site-packages/asgiref/sync.py:110: in thread_handler
        return self.func(*args, **kwargs)
    ../../../.virtualenvs/some_env/lib/python3.6/site-packages/channels/consumer.py:99: in dispatch
        handler(message)
    _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
    
    self = <someapp.consumers.ChatConsumer object at 0x7f38fcc55240>
    message = {'code': 1000, 'type': 'websocket.disconnect'}
    
        def websocket_disconnect(self, message):
            """
                Called when a WebSocket connection is closed. Base level so you don't
                need to call super() all the time.
                """
            # TODO: group leaving
    >       self.disconnect(message["code"])
    E       TypeError: disconnect() takes 1 positional argument but 2 were given
    

    What should I do to separate those test cases in the respective individual test functions?

    bug exp/advanced 
    opened by kradem 38
  • Daphne / ASGI won't use the DJANGO_SETTINGS_MODULE I specify

    Daphne / ASGI won't use the DJANGO_SETTINGS_MODULE I specify

    Hello,

    I currently have a problem deploying my Django application with django-channels. I'm trying to setup daphne with nginx and for that, I made this ASGI file:

    import os
    from channels.asgi import get_channel_layer
    
    os.environ.setdefault("DJANGO_SETTINGS_MODULE", "myapp.settings_production")
    
    channel_layer = get_channel_layer()
    

    But somehow, it keeps using myapp.settings (and thus, fails at the database connection)

    I use this command to launch daphne: daphne myapp.asgi:channel_layer -v 2

    Python version: 2.7.11+ Django version: 1.9.7 django-channels version: 0.15.0

    Am I doing something wrong?

    opened by DarthLabonne 33
  • ASGIRef 3.4.1 + Channels 3.0.3 causes non-deterministic 500 errors serving static files

    ASGIRef 3.4.1 + Channels 3.0.3 causes non-deterministic 500 errors serving static files

    So this was the very first scenario in which the Single thread executor error was found and that lead to me opening django/asgiref#275

    While trying to get a simple repro-case for it, we figured out a way to trigger an error related to it in a very simple way and this was fixed with https://github.com/django/asgiref/releases/tag/3.4.1

    But testing the new 3.4.1 version against our code-base still yielded the same 500 errors while serving static files (at least) in the dev environment.

    I've updated https://github.com/rdmrocha/asgiref-thread-bug with this new repro-case, by loading a crapload of JS files (1500) but that can be changed in the views.py file.

    It doesn't ALWAYS happen (so you might need a hard-refresh or two) but when it does, you'll be greeted with something like this: Screenshot 2021-07-02 at 16 30 50 Screenshot 2021-07-02 at 16 33 17

    I believe this is still related https://github.com/django/asgiref/commit/13d0b82a505a753ef116e11b62a6dfcae6a80987 as reverting to v3.3.4 via requirements.txt makes the error go away.

    Looking at the offending code inside channels/http.py it looks like this might be a thread exhaustion issue but this is pure speculation. image

    since the handle is decorated as sync_to_async: Screenshot 2021-07-02 at 17 01 40 This is forcing the send to become sync and we're waiting on it like this: await self.handle(scope, async_to_sync(send), body_stream). If there's no more threads available, I speculate that they might end up in a deadlock waiting for the unwrap of this await async_to_sync(async_to_sync) call, eventually triggering the protection introduced in https://github.com/django/asgiref/commit/13d0b82a505a753ef116e11b62a6dfcae6a80987

    But take this last part with a grain of salt as this is pure speculation without diving into the code and debugging it. Hope it helps

    opened by rdmrocha 31
  • Lifespan support

    Lifespan support

    Lifespan is a new addition to the ASGI specification that provides events pertaining to the main event loop. You can find its documentation here: https://asgi.readthedocs.io/en/latest/specs/lifespan.html

    Is there any interest in implementing this in Django Channels?

    enhancement exp/beginner 
    opened by ferndot 31
  • Planning Channels v4.0

    Planning Channels v4.0

    I've begun work on the next major version, which will be v4.0. I'm looking to release early-September 2022.

    The main highlights will be:

    • Removal of the old AsgiHandler, and related code. All of this now lives in Django, and you should be using that.
    • Removal of support for ASGI v2 double-callables. Your apps should all be ASGI v3 single-callables by now.
    • A move of the runserver command to Daphne, so that folks using other ASGI servers don't have that import overhead to use Channels.
    • And then various tidy-ups and other fixes that we've got time for.

    The hope is this leaves Channels nice and tight as the Consumers, Channel Layer stuff, and not too much else. The use-case is: "I want to do Websockets, or lower-level ASGI with my Django app" It should be that we can evolve through point-releases for a while from now after this. 🤞

    Matching updates to Daphne and channels_redis will come at the same time.

    See the recent additions to Closed Pull Requests and the WIP CHANGELOG until I can draft the full release announcements for the details.

    I've begun with the code changes. You'll need to install the repos from main in order to test (e.g. the runserver move) Any feedback appreciated.

    Next up, I want to go through the docs and tutorials and get those into place, before I'll look at updating the packaging, and we'll see where we're at.

    opened by carltongibson 0
  • 502, ProtocolTypeRouter took too long to shut down and was killed.

    502, ProtocolTypeRouter took too long to shut down and was killed.

    i have gunicorn that accepts all http request, and daphne with asgi for websockets connections only. on my frontend i have reconnect script so i see that it start connection than from 5 to 10 times connection fails with error 502 and than connects. From log i see this.

     WARNING  Application instance <Task pending name='Task-30589' coro=<ProtocolTypeRouter.__call__() running at /usr/local/lib/python3.10/site-packages/channels/routing.py:71> wait_for=<Future pending cb=[_chain_future.<locals>._call_check_cancel() at /usr/local/lib/python3.10/asyncio/futures.py:384, Task.task_wakeup()]>> for connection <WebSocketProtocol client=['172.6.66.8', 0] path=b'/ws/profiles/'> took too long to shut down and was killed.
     WARNING  Application instance <Task pending name='Task-30555' coro=<ProtocolTypeRouter.__call__() running at /usr/local/lib/python3.10/site-packages/channels/routing.py:71> wait_for=<Future pending cb=[_chain_future.<locals>._call_check_cancel() at /usr/local/lib/python3.10/asyncio/futures.py:384, Task.task_wakeup()]>> for connection <WebSocketProtocol client=['172.6.66.8', 0] path=b'/ws/profiles/'> took too long to shut down and was killed.
     WARNING  Application instance <Task pending name='Task-30599' coro=<ProtocolTypeRouter.__call__() running at /usr/local/lib/python3.10/site-packages/channels/routing.py:71> wait_for=<Future pending cb=[_chain_future.<locals>._call_check_cancel() at /usr/local/lib/python3.10/asyncio/futures.py:384, Task.task_wakeup()]>> for connection <WebSocketProtocol client=['172.6.66.8', 0] path=b'/ws/profiles/'> took too long to shut down and was killed.
    

    Django==4.0.6 daphne==3.0.2 channels==3.0.5 channels-redis==3.4.1 asgiref==3.5.2

    blocked/user-response 
    opened by kojibhy 2
  • Middleware support in nested URLRouter or consumers

    Middleware support in nested URLRouter or consumers

    While working on auth with django channels today, I stumbled upon a problem where there is no straight forward way of adding auth for nested router or per-consumer level router

    Django rest framework provides a way to do this via a class member of the targetter class/view/consumer in this case. So after reading a few SO questions online and a few parts of codebase, I wrote a middleware something like this, which allows a global and per-consumer level auth classes.

    class MultiAuthMiddleware:
        def __init__(
            self,
            app,
            *,
            default_auth_handler: Callable[[dict], bool] = None
        ):
            self.app = app
            self.default_auth_handler = default_auth_handler
    
        def _get_matching_callback(self, routes, path):
            path = path.lstrip("/")
            for route in routes:
                match = route_pattern_match(route, path)
                if not match:
                    continue
    
                if isinstance(route.callback, URLRouter):
                    new_path, args, kwargs = match
                    new_routes = route.callback.routes
                    return self._get_matching_callback(new_routes, new_path)
                else:
                    return route.callback
    
        def _get_auth_handler_from_callback(self, callback):
            if callback is None:
                return self.default_auth_handler
            return getattr(
                callback.consumer_class,
                'auth_handler',
                self.default_auth_handler
            )
    
        async def __call__(self, scope, receive, send):
            path = scope['path']
            callback = self._get_matching_callback(self.app.routes, path)
            auth_handler = self._get_auth_handler_from_callback(callback)
    
            if auth_handler is None:
                # No auth required, so just pass through
                await self.app(scope, receive, send)
    
            is_authenticated, scope = await auth_handler.authenticate(scope)
            if not is_authenticated:
                # auth failed, deny the connection
                denier = WebsocketDenier()
                return await denier(scope, receive, send)
    
            # auth succeeded, pass through
            return await self.app(scope, receive, send)
    

    Now this is clearly something I wrote with what I could understand would be the right pattern but I can immediately identify some anti-patterns when compared to rest of your codebase. But this allowed me to solve my problem

    So my proposal is to either (in order of preference)

    1. Somehow allow adding middlewares inside each consumer or in any top level or nested URLRouter. Now what I have used as auth_handler is not exactly a middleware (it does not get app instance in init and does not return a new asgi app as callable) but that should be doable
    2. Add this functionality somehow to URLRouter (not sure how that'll work with existing Auth stack middleware)
    3. Add a middleware in package which supports this
    opened by sourabhv 1
  • Added MQTTAsgi project to community projects.

    Added MQTTAsgi project to community projects.

    mqttasgi is an ASGI protocol server that implements a complete interface for MQTT for the Django development framework. Built following daphne protocol server as a guide.

    opened by sivulich 0
  • Feature request: Implement name=

    Feature request: Implement name="foo" to routes and {% route 'foo' %} template tag.

    I will try to get this working and create a new "Pull request", but I encourage anyone to implement it if they can. I think it would be a good feature for the project.

    opened by saarrooiii 0
Django friendly finite state machine support

Django friendly finite state machine support django-fsm adds simple declarative state management for django models. If you need parallel task executio

Viewflow 2k Aug 11, 2022
Django URL Shortener is a Django app to to include URL Shortening feature in your Django Project

Django URL Shortener Django URL Shortener is a Django app to to include URL Shortening feature in your Django Project Install this package to your Dja

Rishav Sinha 4 Nov 18, 2021
The friendly PIL fork (Python Imaging Library)

Pillow Python Imaging Library (Fork) Pillow is the friendly PIL fork by Alex Clark and Contributors. PIL is the Python Imaging Library by Fredrik Lund

Pillow 10k Aug 11, 2022
Meta package to combine turbo-django and stimulus-django

Hotwire + Django This repository aims to help you integrate Hotwire with Django ?? Inspiration might be taken from @hotwired/hotwire-rails. We are sti

Hotwire for Django 31 Aug 9, 2022
django-reversion is an extension to the Django web framework that provides version control for model instances.

django-reversion django-reversion is an extension to the Django web framework that provides version control for model instances. Requirements Python 3

Dave Hall 2.8k Aug 4, 2022
Django-environ allows you to utilize 12factor inspired environment variables to configure your Django application.

Django-environ django-environ allows you to use Twelve-factor methodology to configure your Django application with environment variables. import envi

Daniele Faraglia 2.6k Aug 8, 2022
Rosetta is a Django application that eases the translation process of your Django projects

Rosetta Rosetta is a Django application that facilitates the translation process of your Django projects. Because it doesn't export any models, Rosett

Marco Bonetti 891 Jul 24, 2022
Cookiecutter Django is a framework for jumpstarting production-ready Django projects quickly.

Cookiecutter Django Powered by Cookiecutter, Cookiecutter Django is a framework for jumpstarting production-ready Django projects quickly. Documentati

Daniel Feldroy 9.6k Aug 13, 2022
Django project starter on steroids: quickly create a Django app AND generate source code for data models + REST/GraphQL APIs (the generated code is auto-linted and has 100% test coverage).

Create Django App ?? We're a Django project starter on steroids! One-line command to create a Django app with all the dependencies auto-installed AND

imagine.ai 68 Jun 23, 2022
django-quill-editor makes Quill.js easy to use on Django Forms and admin sites

django-quill-editor django-quill-editor makes Quill.js easy to use on Django Forms and admin sites No configuration required for static files! The ent

lhy 126 Jul 25, 2022
A Django chatbot that is capable of doing math and searching Chinese poet online. Developed with django, channels, celery and redis.

Django Channels Websocket Chatbot A Django chatbot that is capable of doing math and searching Chinese poet online. Developed with django, channels, c

Yunbo Shi 9 May 20, 2022
A handy tool for generating Django-based backend projects without coding. On the other hand, it is a code generator of the Django framework.

Django Sage Painless The django-sage-painless is a valuable package based on Django Web Framework & Django Rest Framework for high-level and rapid web

sageteam 50 Jul 7, 2022
A beginner django project and also my first Django project which involves shortening of a longer URL into a short one using a unique id.

Django-URL-Shortener A beginner django project and also my first Django project which involves shortening of a longer URL into a short one using a uni

Rohini Rao 3 Aug 8, 2021
Dockerizing Django with Postgres, Gunicorn, Nginx and Certbot. A fully Django starter project.

Dockerizing Django with Postgres, Gunicorn, Nginx and Certbot ?? Features A Django stater project with fully basic requirements for a production-ready

null 8 Jun 27, 2022
pytest-django allows you to test your Django project/applications with the pytest testing tool.

pytest-django allows you to test your Django project/applications with the pytest testing tool.

pytest-dev 1.1k Aug 7, 2022
APIs for a Chat app. Written with Django Rest framework and Django channels.

ChatAPI APIs for a Chat app. Written with Django Rest framework and Django channels. The documentation for the http end points can be found here This

Victor Aderibigbe 16 Jul 22, 2022
django-dashing is a customisable, modular dashboard application framework for Django to visualize interesting data about your project. Inspired in the dashboard framework Dashing

django-dashing django-dashing is a customisable, modular dashboard application framework for Django to visualize interesting data about your project.

talPor Solutions 692 Aug 5, 2022
Django-MySQL extends Django's built-in MySQL and MariaDB support their specific features not available on other databases.

Django-MySQL The dolphin-pony - proof that cute + cute = double cute. Django-MySQL extends Django's built-in MySQL and MariaDB support their specific

Adam Johnson 480 Jul 25, 2022
Django-Audiofield is a simple app that allows Audio files upload, management and conversion to different audio format (mp3, wav & ogg), which also makes it easy to play audio files into your Django application.

Django-Audiofield Description: Django Audio Management Tools Maintainer: Areski Contributors: list of contributors Django-Audiofield is a simple app t

Areski Belaid 164 Jul 21, 2022