Replace lambdas with functions. Remove a use of old_round (total removal depends on Tribler/tribler#5024 and Tribler/anydex-core#31). ~~Other minor refractoring: removing the redundant u prefix from strings; remove cast_to_* functions when it is immediately clear how to manipulate the data.~~

Fixes #651.

Fixes #369

Added a new endpoint for retrieving the latest TrustChain block via the DHT mechanism. Added a new callback in the TrustChainCommunity class which is called whenever a block is being added in the DB, irrelevant of the block's type. It is now possible to retrieve the latest TrustChain block of some peer via HTTP requests. Created a new test set for the newly added block retrieval mechanism. In addition to this, made a few changes to the REST API test suite in terms of the communication modules, such as moving classes from one module to another and creating new modules which host logic for HTTP requests and communication for specific endpoints.

We should use Swagger for our REST endpoints. See below for the original exploration of this idea.

Old

Nobody wants to write documentation. Instead, we should automatically generate this. Also, nobody likes to do precondition and postcondition checks. We can do both as follows.

First we can scan all the distinct endpoints available by instantiating a RootEndpoint and inspecting its children. We can then automatically generate documentation from a .restapi variable tied to the REST API handler functions (render_POST, render_GET, etc.).

We can attach and generate the .restapi variable to these functions statically by using annotations/wrapper functions. Specifically we should expose:

• RESTInput, which takes an option, a jsontype and a description argument.
• RESTOutput, which takes a conditional_lambda and a output_format argument.

RESTInput

The first input a RESTInput takes is the option. This option specifies what option in the request URI is being described. The jsontype then specifies the JSON datatype, which can be a combination of:

• number/boolean/null
• string with specified encoding, for example STR_TYPE["UTF-8"] or STR_TYPE["ASCII"]
• array/object
• tuple of (jsontype, description) which adds additional documentation to an inner construction

Lastly, each input should have a description. This is not very useful when using RESTInput for runtime precondition checking, but mostly for humans reading generated documentation.

RESTOutput

The RESTOutput takes a conditional_lambda, which is a lambda function taking the input and returning True if its output format should be applied. I have created a function for converting lambda bytecode to human readable form already, so this is a multi-purpose argument (both for checking postconditions programmatically and for generating documentation). Lastly the output_format specifies what the output matches to (or should match to). The output_format follows the same rules as the jsontype for the RESTInput. The description is once again to explain in human terms what the output is used for.

Example

class NetworkEndpoint(resource.Resource):
    """
    This endpoint is responsible for handing all requests regarding the state of the network.
    """

    def __init__(self, session):
        resource.Resource.__init__(self)
        self.session = session

    # Takes no RESTInput
    @RESTOutput(lambda input: input is None,
                {
                    "peers": {
                        (STR_TYPE["BASE64"], "The sha1 of the peer's public key."): {
                            "ip": STR_TYPE["BASE64"],
                            "port": NUMBER_TYPE,
                            "public_key": STR_TYPE["BASE64"],
                            "services": [(STR_TYPE["BASE64"], "The sha1 of the Community's public key.")]
                        }
                    }
                },
                "All of the known peers and their services.")
    def render_GET(self, request):
        return json.dumps({"peers": self.render_peers()})

Would generate the following documentation:

NetworkEndpoint

Input

Output

This documentation should include the following.

Configuring Peer discovery:

• A pointer to the basics of DiscoveryStrategies.
• How the RandomWalk works and when to use it.
• How the EdgeWalk works and when to use it.
• How the RandomChurn works and when to use it.
• How to parameterize the walk sizes. The default node degree (20-30) makes sense for most use cases, see for example The Role of Network Topology for Distributed Machine Learning by Giovanni Neglia, Gianmarco Calbi, Don Towsley and Gayane Vardoyan. Add notes on creating a new Network() when mixing with communities of unequal parameters.
• Caution on combining node removal strategies.

Convergence of the connected Peers:

• ~~How the Network class actually works and when peers are added, hooking into the peer discovery basics documentation.~~ EDIT: Out of scope. Add a note on sharing peers between communities.
• ~~How the DiscoveryCommunity consolidates peers and how blocking introduction requests and responses balances peers over communities.~~ EDIT: Out of scope.

Request for dataclass-based message payloads.

Since message payloads are used very frequently I suggest we simplify the creation and usage of Payloads. Python data classes might be a promising direction.

Here is an example:

The payload is a wrapper:

class LazierPayload:
    serializer = default_serializer

    @staticmethod
    def _type_map(t: Type) -> str:
        if t == int:
            return "Q"
        elif t == bytes:
            return "varlenH"
        elif "Tuple" in str(t) or "List" in str(t) or "Set" in str(t):
            return (
                "varlenH-list"
                if "int" in str(t) or "bytes" in str(t)
                else [typing.get_args(t)[0]]
            )
        elif hasattr(t, "format_list"):
            return t
        else:
            raise NotImplementedError(t, " unknown")

    @classmethod
    def init_class(cls):
        # Copy all methods of VariablePayload except init
        d = {
            k: v
            for k, v in VariablePayload.__dict__.items()
            if not str(k).startswith("__")
            and str(k) != "names"
            and str(k) != "format_list"
        }

        for (name, method) in d.items():
            setattr(cls, name, method)
        # Populate names and format list
        fields = get_cls_fields(cls)

        for f, t in fields.items():
            cls.names.append(f)
            cls.format_list.append(cls._type_map(t))
        return cls

    def is_optimal_size(self):
        return check_size_limit(self.to_bytes())

    def to_bytes(self) -> bytes:
        return self.serializer.pack_serializable(self)

    @classmethod
    def from_bytes(cls, pack: bytes) -> "BamiPayload":
        return cls.serializer.unpack_serializable(cls, pack)[0]

def payload(cls):
    d = {k: v for k, v in LazierPayload.__dict__.items() if not str(k).startswith("__")}
    for k, v in d.items():
        setattr(cls, k, v)
    cls.names = list()
    cls.format_list = list()

    # Populate all by mro
    added_classes = set()
    new_mro = []
    has_imp_ser = False

    for superclass in cls.__mro__:
        if superclass == ImpSer:
            has_imp_ser = True
        if hasattr(superclass, "names"):
            cls.names.extend(superclass.names)
            cls.format_list.extend(superclass.format_list)

    new_mro.append(ImpSer)
    if ImpSer not in added_classes:
        added_classes.add(ImpSer)

    if not has_imp_ser:
        new_vals = tuple([ImpSer] + list(cls.__bases__))
        new_cls = type(cls.__name__, new_vals, dict(cls.__dict__))
    else:
        new_cls = cls

    return new_cls.init_class()

From the developer point of view the dataclass would act as a VariablePayload.

Downside: we have to write map from Python types to ipv8 struct types

IPv8 should function as an Android app.

All of the dependencies have been compiled before for the Tribler project, these should/could be reused for the py-to-app compiler.

This PR adds the (highly experimental) latency community. The latency community is not enabled by default. To work with the latency community:

1. Create a new Network class (very important) for the LatencyCommunity.
2. Load the LatencyCommunity with only the LatencyEdgeWalk strategy into IPv8.
3. Fetch your partners from the community.accepted_proposals set.

LatencyEdgeWalk

The LatencyEdgeWalk strategy will collect a set number of peers from the bootstrap nodes/trackers, 30 by default. Next, the strategy will attempt to get successive network-unique ping-unique (0.05 seconds similarity) peers from each of these 30 edge roots, up until a given edge length (6 by default). As the edges are being grown, the strategy also provides ping measurements if needed.

After each step in the discovery process the LatencyEdgeWalk strategy will update its registered overlay by setting the overlay's possible_peers attribute to the current list of peers included in the disconnected graph of edges (the ancestry datastructure).

Every so often (by default 30 seconds), the strategy performs a cleanup of all open connections to peers which are not included in the ancestry.

At the end of each cycle we distinguish two types of peers:

• Open connections which cannot be included in a latency-unique edge, we will dub this the total peers.
• Open connections which are part of a latency-unique edge, which we dub the possible peers.

LatencyCommunity

The LatencyCommunity is in charge of the matching of peers. The Community predominantly uses the PeerSelector to advise any matching with others. The overlay itself periodically (by default every 5 seconds) (a) sends out proposals and keeps a window (30 by default) of optimal matches or (b) breaks the current worst match it has if the amount of matches is equal to the required amount (60 by default). Asynchronously the LatencyCommunity may decide to accept or reject incoming proposals.

The LatencyCommunity provides the last class of peers:

• Peers which have bilaterraly agreed to form a matching, we dub these matched peers.

PeerSelector

This class uses kerning to decide the error of including a peer into a given existing set of choices, according to a specified target distribution. The idea is that a programmer gives both (1) a continuous function (lambda x: preferred_count / x by default) and (2) a list of x-coordinates (bins) to evaluate the function (0.001 through 2.001 with steps of 0.05 by default). This function need to be scaled precisely for the kerning, this is handled automatically by generate_reference for a given preferred peer count and the current binning distribution. We use Gaussian kernel density estimation.

Gumby & Jenkins

My personal Jenkins experiment for this PR can be found here: https://jenkins-ci.tribler.org/job/pers/job/latency_noodle_qstokkink/. The source code for the experiment can be found here: https://github.com/qstokkink/gumby/tree/latency_overlay/experiments/latency_overlay.

Future work

Even though the basic concept works, the CPU utilization should be improved and the matching algorithm will require some further love.

…e various REST requests, a set of dummy peers which can be used for testing, and an interface for different types of inter-peer communication which should facilitate decoupling between the implementation of the communication, and its usage. Currently, the interface has been implemented in the test suite for the HTTP type requests.

UPnP provides a way for an application/device to request a port forwarding through the network's NAT from a UPnP-enabled router. This is another technique to enable better NAT traversal. UPnP-IGD uses broadcast HTTP requests over UDP (and is already enabled in Libtorrent). We can copy the logic for announcing the port from there.

While working on the rust-ipv8 implementation (with @ichorid) we noticed that the bloom filter (originally from dispersy) is absolutely never used in py-ipv8. Therefore this pull request removes it from py-ipv8.

However this isn't the end of the story. tribler/Tribler does use the bloom filter. We will make a pull request ASAP to add it to tribler/Tribler itself.

This is a screenshot from Tribler running 30 seconds using the current master HEAD (f4dccad6e50e07b01a55b05b9fe0c09ada9deb58):

There are no peers at all, perhaps the IPv8 tracker is down?

Eventually after +- 2 minutes it starts finding peers though:

Fixes #1110

This PR:

• Adds an __all__ definition for base.py to avoid exporting newly added module-level variables.
• Adds TestBase support for Python 3.11 using unittest.IsolatedAsyncioTestCase.
• Fixes RandomChurn using random.sample on a set (only list is supported as of Python 3.11).

See original issue for implementation substantiation. Other notes:

• TestBase now inherits from unittest.IsolatedAsyncioTestCase for all Python versions >= 3.8 by default. However, asynctest is still supported up to Python 3.10 and TestBase subclasses can switch to asynctest by setting __asynctest_compatibility_mode__ to True.
• loop has a class scope in asynctest, an instance scope in unittest, and it needs to be retrieved upon invocation (a property) for Python 3.11.
• _callSetUp and _callTearDown are overwritten to support the old TestBase behavior of allowing both async def and def setup and teardowns, including its own. For (only) the teardown of TestBase, the asyncTearDown is still explicitly needed.

The IPv8 unit tests fail on Python 3.11 due to the asynctest dependency.

Traceback (most recent call last):
  File "C:\py-ipv8\run_all_tests.py", line 166, in <module>
    test_class_names = find_all_test_class_names()
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\py-ipv8\run_all_tests.py", line 139, in find_all_test_class_names
    test_class_names.extend(derive_test_class_names(found_test))
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\py-ipv8\run_all_tests.py", line 126, in derive_test_class_names
    module_instance = importlib.import_module(module_name)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Python311\Lib\importlib\__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1206, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1178, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1149, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 940, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "C:\py-ipv8\ipv8\test\test_community.py", line 1, in <module>
    from .base import TestBase
  File "C:\py-ipv8\ipv8\test\base.py", line 13, in <module>
    import asynctest
  File "C:\Python311\Lib\site-packages\asynctest\__init__.py", line 22, in <module>
    from .case import *
  File "C:\Python311\Lib\site-packages\asynctest\case.py", line 54, in <module>
    import asynctest.selector
  File "C:\Python311\Lib\site-packages\asynctest\selector.py", line 29, in <module>
    from . import mock
  File "C:\Python311\Lib\site-packages\asynctest\mock.py", line 428, in <module>
    class _AwaitEvent:
  File "C:\Python311\Lib\site-packages\asynctest\mock.py", line 433, in _AwaitEvent
    @asyncio.coroutine
     ^^^^^^^^^^^^^^^^^
AttributeError: module 'asyncio' has no attribute 'coroutine'. Did you mean: 'coroutines'?

This issue is a different approach to core problem of #1091.

To reiterate: GitHub only sees PR approvals from users with write access as "real" approvals and the security model of this repository seeks to use as little write access as possible. This leads to the situation where nobody can offer formal PR approval, even though members of Tribler/core should be able to offer "real" approvals.

My new proposal is to use a secondary mechanism that:

1. Listens for new approvals on a PR.
2. Checks if the approving user is a member of Tribler/core and separately approves the PR as well.

To make this secondary approval valid/formal/"real" we can use the @tribler-ci account, the only account that will always have write access.

~~These are potential GitHub actions we could use for this: https://github.com/marketplace/actions/auto-approve, https://github.com/marketplace/actions/approve-pull-request~~ EDIT: This approach failed.

Currently, the type of _ez_unpack_auth unpacked payload is not understood correctly by PyCharm:

You can see that PyCharm can only see the base Payload methods and not the methods of IntroductionRequestPayload.

In the following PR, I suggest a fix that provides the correct result type:

Also, while fixing that issue, I noticed that currently, DataclassPayload defined as

DataclassPayload = typing.TypeVar('DataclassPayload')
...
AnyPayload = typing.Union[Payload, DataclassPayload]
AnyPayloadType = typing.Union[typing.Type[Payload], typing.Type[DataclassPayload]]

In my opinion, it is not correct to use TypeVar here. Type variables are for generic functions when the function result type should be derived from the function arguments'. Here, DataclassPayload is not used for generics and does not specify that the class is related to data classes. So, as a part of a fix, I'm replacing it with a protocol that checks that the class is actually a data class.

The DHTCommunity (and its subclass the DHTDiscoveryCommunity) have very useful functionality to store data and to connect to peers based on their public key. This functionality allows IPv8 to serve as both a distributed file system and allows programmers to both create neighborhood persistence and superpeer networks.

We may want to leave IPv8-as-a-file-system functionality out of the documentation to avoid competing with Tribler's solution. However, it would be useful to give an example of storing known peers on custom Community.unload() and attempting to reconnect when initializing (i.e., neighborhood persistence). This one example would also cover "find-your-friend"/superpeer functionality.

Our requirements.txt file mostly does not specify what the minimum version is we support for each dependency. We should make this more explicit to avoid preventable failures.

I ran an automated test to check which of the available versions of our dependencies managed to (1) install on my Windows 10 machine and (2) pass our unit tests. The results are as follows (green passed, red failed either step 1 or 2):