Python implementation of the Session open group server

The user configures in sogs.ini a list of alphabets to reject by default:

alphabet_filters = [ arabic, cyrillic, persian ]

An empty list means no filtering is desired.

If there are rooms in which these alphabets should be allowed, a list of room ids per alphabet may be specified for whitelisting.

alphabet_whitelist_arabic = [ 8 ] alphabet_whitelist_cyrillic = [ 5, 13 ] alphabet_whitelist_persian = [ 18 ]

A list of valid room ids can be obtained with the following SQL:

$ sudo sqlite3 /var/lib/session-open-group-server/sogs.db 'SELECT * FROM rooms;'

The implementation can easily be extended to more alphabets.

This pull-request supersedes #128.

In its current state, the filter is of limited use:

• The filter allows only a single group to be whitelisted from filtering. This makes it unusable if Farsi is an allowed language in more than one group on the server.

• No error other than the failure to send is reported to the client.

The purpose of this PR is to spark discussion and pave the way for the implementation of a multi-lingual, per group language filter.

-- Codepoint reference: https://github.com/mirhmousavi/Regex.Persian.Language

I want to use this program to create a group and push messages from other channels to the group at the same time. My code is here: https://github.com/liqsliu/session-pysogs/blob/7efa148dae9f9643a6da04e7e36989a6b0112ca5/sogs/mule.py#L99 At present, I can't see the messages I send with it in the Android client, but I can see it in the web browser.

Expanding IN clauses in bindings with SQLAlchemy requires a little more work (and the existing IN clauses were broken).

This expose the needed mechanics in the query() interface, used such as:

    db.query(
        "SELECT * FROM blah WHERE id IN :ids",
        ids=[1,2,3],
        bind_expanding=['ids']
    )

After enabling id blinding on sog.caliban.org, I observe that messages posted prior to blinding are still downloaded by Session with the original unblinded id.

This allows the association of blinded ids with their unblinded counterparts for any given display name.

Should id blinding not be retroactively applied to all messages downloaded from the server?

Adds support for server-side reactions handling for open groups.

Reactions are now returned as an extra field when messages are retrieved, such as this:

        "reactions": {
          "👍": {
            "index": 1,
            "total": 150,
            "reactors": [
              "058328640343b91c03d393b8dc6ce15c8f93b191ff78054b6ae2c8030e2680b4d6",
              "05d782a81448199aea940f496cc70dc644c269e5979e472536ea03a067bf46d54a",
              "05233432db6edcd320a06b550d4dcf1a06cc486c58f8017a565f6f7dc0fc8b9512",
              "05ca519a397a1aa8a7a1515ff433dd2784cbcbb31c358b1a6e97fb758ceab44dca"
            ],
            "you": true
          },
          "👎": {
            "index": 0,
            "total": 27,
            "reactors": [
              "05ca519a397a1aa8a7a1515ff433dd2784cbcbb31c358b1a6e97fb758ceab44dca",
              "055297c3dc032bef33a21dcb1b76265c8319e53ed0867accd3c274c25ae33c1731",
              "05986c4e7fec0ac1a3a544d84367b2e995f85a6356be8d082724781a9ff699ac7a",
              "0537f1727302b70be32fc371523097bcdbc3e681f947a68ef5e3ea1365a689666a"
            ]
          },
          "🍆": {
            "index": 2,
            "total": 1,
            "reactors": [
              "051234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef"
            ],
            "you": true
          },

When polling for new messages (using the /since endpoint) you also get updates for any posts that have had new reactions added or removed from the post (but not otherwise changed); these look vaguely similar to messages that you'd normally get back from /since, except they are missing most of the info other than id, seqno, and the reactions:

{ "id" 1234, "seqno": 56768, "reactions": { /* as above */ } }

so that, when polling, you get notified (in bulk) of reaction changes in one go per message.

Since this might break existing clients, I've feature-guarded this by adding a new URL parameter t=r to the /since endpoint, so that you'd fetch: /room/ROOMID/messages/since/56768?t=r to tell SOGS that you want to opt-in to the new reactions rows.

Reactions update the room's sequence number, so clients should update their current seqno value when they get such a reaction row (so that they will only get the reactions again if there are changes to reactions).

The reactors shows the first N session IDs that reacted; by default this is 4, but can be anywhere from 0 to 20 by adding a reactors=N query string parameter. (As a special case, 0 omits the "reactors" key entirely).

Adding/removing a reaction is done via endpoints: Submitting reactions is done via:

PUT /room/ROOM/reaction/MSGID/👍 (body is an empty json dict {}) DELETE /room/ROOM/reaction/MSGID/👍

(I want to say that the 👍 should be URL-encoded utf-8 bytes (e.g. %f0%9f%91%8d for the encoded 👍) when making a direct HTTP request (e.g. over lokinet), but from my testing uwsgi and sogs seem perfectly happy with a raw UTF-8 value there as well. Onion requests should be perfectly fine with raw utf8. I haven't yet tested whether it survives cleanly through an nginx proxy in the direct HTTP case yet).

Moderator endpoints are:

DELETE /room/ROOMID/reactions/MSGID/🍆

to delete all of a single reaction from a message, and:

DELETE /room/ROOMID/reactions/MSGID

to delete all reactions of any type from a message.

There is also an endpoint to get all reactors (not limited to the first N) of a message, including the timestamp when they reacted (this is probably not immediately useful for Session, but would be quite useful for, say, a bot that wants to check for all reactions on startup):

GET /room/ROOMID/reactors/MSGID/🍆

which returns lists of pairs of session ID + timestamp, sorted by timestamp, of when reactions were added. For example:

[
    ["05ca519a397a1aa8a7a1515ff433dd2784cbcbb31c358b1a6e97fb758ceab44dca", 1656631467.6396575],
    ["055297c3dc032bef33a21dcb1b76265c8319e53ed0867accd3c274c25ae33c1731", 1656631480.2537503]
]

i'm getting this error when attempting to install the oxen proxy package:

Package python3-oxenmq is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source

E: Package 'python3-oxenmq' has no installation candidate

any workarounds? running ubuntu 22.04

I run a status page and would love to have some sort of way to tie it into the open group, maybe like a webhook of some sort? So it can send a message to the group when one of my services goes down, for example.

Not sure if this is possible, but I guess asking won't hurt!

• Cross

This adds an oxenmq as a mule that listens on :22028 by default (using curve encryption with the same x25519 key as the server). This oxenmq now handles db cleanup as well (instead of scheduling it through a regular front-end worker).

You can kick the mule by sending it signals through the signal.send_signal function so that it can be triggered when things happen from model; a couple stubs are included here for message creation and message deletion.

(NB: requires pyoxenmq built against the last oxenmq (1.2.9), such as the updated python3-oxenmq 1.0.0-3 package, or installed with pip3 with an updated liboxenmq-dev installed).

Steps: * Follow the steps to install for Ubuntu

Error happens at step: * sudo apt install sogs-standalone

Output:

sudo apt install sogs-standalone
Reading package lists... Done
Building dependency tree
Reading state information... Done
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
 sogs-standalone : Depends: python3-sogs (= 0.3.0-2) but it is not going to be installed
Unable to correct problems, you have held broken packages.

I am able to verify the repo is added correctly:

Hit:1 https://packages.grafana.com/oss/deb stable InRelease
Hit:2 https://deb.oxen.io focal InRelease
Hit:3 http://iad-ad-2.clouds.ports.ubuntu.com/ubuntu-ports focal InRelease
Hit:4 http://ports.ubuntu.com/ubuntu-ports focal-security InRelease
Hit:5 http://iad-ad-2.clouds.ports.ubuntu.com/ubuntu-ports focal-updates InRelease
Hit:6 http://iad-ad-2.clouds.ports.ubuntu.com/ubuntu-ports focal-backports InRelease

We have an updated python3-nacl packaged in our repo, but the deb is just depending on python3-nacl, not >= 1.4.0, and so you can upgrade sogs by itself and not have the updated nacl module installed.

I enabled id blinding for sog.caliban.org on 2023-01-01 at 00.00 UTC. The server runs PySOGS 0.3.6.

Immediately, I noticed that I had lost hidden global admin privileges in all of my groups. The extra menu options no longer appeared for me.

These were restored by executing the following:

$ sudo sogs --add-moderators <my_unblinded_session_id> --rooms=+ --hidden --admin

Other group-level admins are reporting that privileged features are still displayed in Session's menus, but no longer effective.

Hello,

profanity list phrases are currently not effective against fancy text https://textfancy.com/ and one have to add for example: purple 𝑃𝑈𝑅𝑃𝐿𝐸

Can You please disable fancy text in Session or convert it to normal/regular text (or do it on SOGS or on profanity filtering level) so the regular text block really everything and fancy one is not a problem for admin ?

https://github.com/nitanmarcel/fancy_text https://github.com/Secret-chest/fancify-text#readme

I suspect that the profanity filtering does not support longer phrases like: privacy?public_key=118df8c6c471ac0468c7c77e1cdc12f24a139ee8a07c6e3bf4e7855640dad821 or aaaaaaaaaaadaaaaaaa118df8c6c471ac0468c7c77e1cdc12f24a139ee8a07c6e3bf4e7855640dad821

When i add it and ran "sudo systemctl restart sogs"

Job for sogs-proxied.service failed. See "systemctl status sogs-proxied.service" and "journalctl -xe" for details.

$ sogs --version PySOGS 0.3.5

I am on Linux Debian 11, .deb package

Can you reproduce it and fix it please?

After creating a group and viewing the group from the web page, the QR code is a broken image. When I view the QR code URL I get the error: Internal Server Error The server encountered an internal error and was unable to complete your request. Either the server is overloaded or there is an error in the application.

and logs show the errors below: Dec 9 06:52:12 test uwsgi[44550]: #033[32m2022-12-09 06:52:12,979#033[0m #033[35mtest#033[0m #033[34msogs.web[44550]#033[0m #033[1;30mERROR#033[0m #033[31mException on /r/test/invite.png [GET]#033[0m Dec 9 06:52:12 test uwsgi[44550]: Traceback (most recent call last): Dec 9 06:52:12 test uwsgi[44550]: File "/usr/lib/python3/dist-packages/flask/app.py", line 2070, in wsgi_app Dec 9 06:52:12 test uwsgi[44550]: response = self.full_dispatch_request() Dec 9 06:52:12 test uwsgi[44550]: File "/usr/lib/python3/dist-packages/flask/app.py", line 1515, in full_dispatch_request Dec 9 06:52:12 test uwsgi[44550]: rv = self.handle_user_exception(e) Dec 9 06:52:12 test uwsgi[44550]: File "/usr/lib/python3/dist-packages/flask/app.py", line 1513, in full_dispatch_request Dec 9 06:52:12 test uwsgi[44550]: rv = self.dispatch_request() Dec 9 06:52:12 test uwsgi[44550]: File "/usr/lib/python3/dist-packages/flask/app.py", line 1499, in dispatch_request Dec 9 06:52:12 test uwsgi[44550]: return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Dec 9 06:52:12 test uwsgi[44550]: File "/usr/lib/python3/dist-packages/sogs/routes/views.py", line 64, in serve_invite_qr Dec 9 06:52:12 test uwsgi[44550]: img = qrencode.encode(room.url) Dec 9 06:52:12 test uwsgi[44550]: File "/usr/lib/python3/dist-packages/qrencode/init.py", line 47, in encode Dec 9 06:52:12 test uwsgi[44550]: version, size, data = _encode(data, version, level, hint, True) Dec 9 06:52:12 test uwsgi[44550]: SystemError: PY_SSIZE_T_CLEAN macro must be defined for '#' formats

OS is a fresh Ubunto 20.04 Python version: 3.10.6

Followed install steps here for sogs-standalone: https://docs.oxen.io/products-built-on-oxen/session/guides/open-group-setup

Any ideas? thx

Hello, in sogs-proxied 0.3.5 on Debian 11 (apt package) with nginx "tail /var/log/nginx/error.log" shows many:

22:31:57 [error] redacted: *545 open() "/var/www/html/oxen/v4/lsrpc" failed (2: No such file or directory), client: redacted, server: 127.0.0.1, request: "POST /oxen/v4/lsrpc HTTP/1.1", host: "redacted" 22:31:58 [error] redacted: *548 open() "/var/www/html/oxen/v4/lsrpc" failed (2: No such file or directory), client: redacted, server: 127.0.0.1, request: "POST /oxen/v4/lsrpc HTTP/1.1", host: "redacted" 22:31:58 [error] redacted: *549 open() "/var/www/html/oxen/v4/lsrpc" failed (2: No such file or directory), client: redacted, server: 127.0.0.1, request: "POST /oxen/v4/lsrpc HTTP/1.1", host: "redacted"

I am unable to find any directory/file like that: find / -name lsrpc;find / -name oxen|grep v4

My nginx.conf is here.

Which command to run, what to modify to prevent this please?