There are small typos in:

• doc/leak.md
• src/netius/common/http2.py

Fixes:

• Should read these rather than tese.
• Should read resetting rather than reseting.

Semi-automated pull request generated by https://github.com/timgates42/meticulous/blob/master/docs/NOTE.md

Description

For some reason the Forward HTTP proxy is working too slowly under Microsoft Edge and Firefox browsers. It seems as if the connections are getting into a stalled state for some reason.

Note that this only happens for HTTP based connections and not the ones with CONNECT.

Description

While using lugardajoia.com website and HTTP2 under Chrome for IPad some resources fail to load for an unknown reason. This may be related with #13.

Notes

This issue has yet to be verified under other iOS devices like iPhone. Ths issue does not occur under Safari for iOS.

• Initial version of the compatible protocols support
• Major refactor in the base netius event loop structure
• HTTP client support for protocol
• DNS client support for protocol

Description

It's important to implement a proper way of "timeouting" an HTTP request so that the control flow is retuned to the caller on an error callback (just like normal HTTP error eg: 500).

This should be a regression from the major event loop refactor taken some weeks ago.

Description

Sometimes while playing multiple videos from the same source at the same time the Chrome browser gets stuck like if no DATA frames are being sent (stream blocked). This must be blamed on the proxy server.

To reproduce this problem try to play at the same time multiple videos from:

https://builds.stage.hive.pt/videos/

Images

Description

TFTP is still used under a lot of scenarios to allow simple transfer of files. One of the most well known scenarios is the PXE boot infra-structure.

For that a simple server under the netius infra-structure must be created.

Links

Wikipedia RFC 1350 Client for Windows PXELINUX

Hi!

At first I want to thank you for great working library, which is easy to use and play nice with use cases in our projects. When file is sent to server (using Cordova File Transfer plugin) from our hybrid app, content of environment['wsgi.input'].getvalue() is:

--+++++
Content-Disposition: form-data; name="file"; filename="image.jpg"
Content-Type: application/octet-stream

(Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.
--+++++--

And when I need to get text content of uploaded file, I have to manually slice environment['wsgi.input'] to get content (to get dispose of boundaries, Content info etc..):

uploadedFileContent = inputWithFile.getvalue()
uploadedFile=StringIO(uploadedFileContent[uploadedFileContent.index(START_OF_FILE_CONTENT):len(uploadedFileContent)])
uploadedFile=StringIO(uploadedFile.getvalue([0:uploadedFile.getvalue().index(BEGINNING_OF_ADDITIONAL_INFO_AT_END_OF_FILE)])
content = str(uploadedFile.getvalue())

Is there a way in your library to get content of uploaded file out of the box, without the need to slice environment['wsgi.input'] maually?

Thank you in advance for reply, Radek from Techmetria.

Description

There's currently no support for multiple files in the Bit torrent Server/Client infra-structure, this support must be created.

Reference

BitTorrent Specification

Description

HTTP/1.1 is slow and old and a new standard must be implemented. It's going to be challenging mostly because of the required implementation of the headers compression HPACK (RFC7541).

Notice that there's already a lot of browsers supporting this specification as seen here.

Inspiration

Currently the best inspiration is to use the nghttp2 library that provides support fot Python through its binding libraries.

There's also the node-http2 (node.js implementation) quite interesting for its sinplicity.

Estimation

2-5 days

References

• RFC7540
• RFC7541
• Draft Spec
• Tools for debugging testing and using HTTP 2
• Hyper Python Http2 client

Attachements

rfc7540.pdf

Description

Sometimes it's important to have a virtual hosting based serving with a different SSL certificate per each host (proxy usage). For this situations the solution is to user SNI extension to SSL that provides a way to handle this scenarios. SNI is available for Python since 3.3 but only 3.4 handles server side SNI correctly.

Implementation

The corre implementation implies the compatability of netius with 3.4 and the usage of the ssl.SSLContext.set_servername_callback method.

Reference

http://en.wikipedia.org/wiki/Server_Name_Indication https://docs.python.org/3/library/ssl.html#ssl.SSLContext.set_servername_callback

Rationale

The world of HTTP is now constantly migrating and although the evolution from HTTP 1.1 to 2.0 took a lot of time the migration from 2.0 to 3.0 was quite fast.

Netius should follow this migration path and add support for HTTP 3.0 including the implementation of the QUIC transport protocol on top of UDP.

References

• https://http3-explained.haxx.se/en/quic.html

Design changes

• Merge of the server and client base infrastructure under common.py (server(), socket_tcp(), socket_udp())
• More server related methods in the compact.py file

Current Status

• Echo UDP server already working
• EchoProtocol is the one used for the testing

Testing

To run the echo server Protocol implementation using netius run:

PYTHONPATH=. python3 netius/servers/echo.py 

To use the compa version meaning that an asyncio like interface will be used underneath the hoods use:

COMPAT=1 PYTHONPATH=. python3 netius/servers/echo.py

This PR is related to #23

Description

Since the inception of the asyncio library in Python 3.5, the Python network world has changed, and a lot of "scalability" emerged from that.

As Netius was created before the arrival of asyncio, the tech approach to network diverged and although overall both strategies represent nonblocking asyncio event-loop based solutions, they have different approaches and abstractions for server and client implementation.

The objective of this issue is to merge (via compatibility layer) both approaches, making it possible the following scenarios:

• A netius HTTP should be able to run on the asyncio event loop
• An asyncio service should be able to run in the netius event loop

Implementation

Architecture

• Transport - Connection vs Connectionless (Datagram)
• Protocol - Equivalent to part of the current Connection and also part of the current Server/Client (protocol logic)
• Server - Equivalent to the newly created Service that spawns multiple connections (that should be converted into transports)
• Parsers - Already exists and should remain pretty much the same
• Event Loop - Must be improved to distance itself from the server vs client approach, meaning that a event loop is something simpler that only handles the polling strategy adapting details
• Servers/Clients (Agents) - Must be simple Abstract API entry-points that facilitate some operations (no logic), this abstract logic should be re-use as much as possible

Change ideas

Command based client

Must rethink very well what's a command-based client (eg: SMTP, IMAP, etc.) as their session is currently very hard-coded, it must be much more similar to command-based. Kind of like a bot-based server (more similar to the FTP server). And their state machine should be more flexible.

Migration strategy

This is the overall idea of how to migrate a client

HTTPConnection -> HTTPProtocol

Principles

• Retro-compatibility should always be a priority
• Concepts like auto_close and auto_pause no longer make sense
• Bi-directional compatibility between netius and asyncio
  • Protocols developed for asyncio should be able to run without problems in netius
  • Netius server implementations should be able to run without problem in the base asyncio event loop

Reference

Transports and protocols (callback-based API) aiohttp server

Description

It's important to be able to handle a fallback process when a back-end server is not available. Our current proxy solutions does not handle the unavailability of the back-end server on a proper way.

Solution

Proper solution would imply

• Constant "ping" of the back-end server
• Marking the back-end server as not available
• Not redirecting request to the unavailable services

Description

The current proxy infra-structure does not support the proxying of Websocket connections, this is relevant for scenarios where the connection is mixed (eg: http://rancher.hive:1001 or https://guacamole.stage.hive.pt).

Solution

Proper attention to the upgrade header must be done to properly handle these connections.

References

• NGINX Websocket proxying