I'm not sure what's involved to allow for this, but it would be nice to be able to whitelist some of the more common community packages (e.g. numpy, pandas, etc.) for use inside of the snekbox.

Thoughts?

@MarkKoz's edit:

If you have suggestions for package you can write them in this issue or just open a PR to add them in. All you need to do is add them like you would any other dependency to pipenv. Just make sure you pin the version and that the lock file gets updated. A short justification for your suggestions would be appreciated.

Relevant Issues

Resolves #63 python-discord/organisation#138

Description

New hooks were added for pre-commit and they will run in CI too. The pipenv run lint script will now run all the new hooks, including flake8.

Hooks added

A couple of these hooks automatically apply fixes. However, they still report failure and leave any changes they make uncommitted. Therefore, the user has to commit the automatic fixes.

• check-merge-conflict - Check for files that contain merge conflict strings.
• check-toml - Attempts to load all toml files to verify syntax.
• check-yaml - Attempts to load all yaml files to verify syntax.
• end-of-file-fixer - Makes sure files end in a newline and only a newline.
• mixed-line-ending - Replaces mixed line endings with LF.
• trailing-whitespace - Trims trailing whitespace.
• python-check-blanket-noqa - Enforce that noqa annotations always occur with specific codes

Reasoning

• Dev dependencies were updated and re-pinned because why not? They were getting old enough. I'm just following along with the other repos.
• No cache used for pre-commit because it runs in a Docker container. I suppose it could be copied to the container from the cache on the host, but that doesn't seem worthwhile.

Description

This is a patchset to utilise snekbox to run Unix commands (namely GNU/Linux Debian stable).

Implementation

It implements a new API endpoint /unixcmd, the usage of which is analogue to the already existing /eval endpoint - but instead of evaluating the input parameter value as Python code, it is executed in a real Linux environment in a bash shell. As with /eval, it returns output with the combined output of stdout and stderr and whatever return code bash returned, which is usually the return code of the last command executed.

Currently, this is implemented using a Debian root filesystem. When building the venv Docker image, unless the variable NO_LINUXFS is set, debootstrap is used to bootstrap a Debian stable image into /snekbox/linuxfs. When /unixcmd is called, snekbox instructs NsJail to bind this directory as / (as read-only) and run its /bin/bash.

It should be noted this environment is technically and practically different than the one snekbox evaluates Python code in. The main differences are that the maximum PID (process) count is increased to 10, which is a design requirement (bash would be unable to run anything otherwise), and that this environment has a full filesystem hierarchy as opposed to just the files that snekbox needs to run Python. As such, I feel comfortable only letting a higher rank such as Helpers+ run this command, but that is a bot interface implementation detail rather than snekbox's.

Rationale

I've noticed that there are a lot of times in the #unix channel when someone asks for help with something which could easily be demonstrated on a "real" Unix system, but harder to explain as a concept. This feature would be useful in such cases.

Example usage

Request:

POST /unixcmd HTTP/1.1
Host: localhost:8060
Content-Type: application/json
Content-Length: 24

{"input":"id; uname -a"}

Response:

HTTP/1.1 200 OK
Server: gunicorn/19.10.0
Date: Mon, 30 Mar 2020 16:01:11 GMT
Connection: close
content-type: application/json
content-length: 185

{"stdout": "uid=65534(nobody) gid=65534(nogroup) groups=65534(nogroup)\nLinux snekbox 5.5.11-arch1-1 #1 SMP PREEMPT Sun, 22 Mar 2020 16:33:15 +0000 x86_64 GNU/Linux\n", "returncode": 0}

To-Do

• [x] fix broken test test_subprocess_resource_unavailable (do not pass PID count as nsjail argument)
• [x] write tests for new code
• [ ] Bash prints "Resource temporarily unavailable" three times before exiting when NsJail limits are hit (time limit, pid limit). Need to find a way to display this in a user-friendly manner
• [x] write logic for when /unixcmd is called when no linuxfs is set up. Currently, NsJail returns blank output and return code -1 (255)
• [ ] create PR for bot interface implementation
• [ ] UnixCmdResource and EvalResource are the same apart from the nsjail function being called. Find a way to remove duplication
• [x] fix logparser logic
• [ ] update documentation

Comments, ideas, criticism, etc all very welcome.

This PR bumps the snekbox version to the latest stable python version. Closes #152, closes #149

• [x] Bump version in dockerfile
• [x] Bump version in documentation
• [x] Relock dependency versions
• [x] ~~Possibly look into bumping dependency versions?~~
• [x] Upgrade and test runtime packages

I'll upgrade where possible to avoid the image having to compile a bunch of packages itself, to speed up builds, however we might have to revisit in the future as I imagine a lot of packages don't have wheels yet.

Since snekbox does not run with a tty, stdout is technically raw bytes, and thus incomplete surrogate pairs can be printed without the client application erroring, and instead fail within _consume_stdout when we attempt to decode it to a str.

This commit sets the PYTHONIOENCODING environment variable to inform python to open the pipe in utf-8 mode.

However, clever use of execl and os.unsetenv() can unset this environment variable, so we add a safety check to _consume_stdout to fail out of parsing output if it contains invalid unicode. This should only happen in deliberate cases, or significant bugs in python or a c library where output is printed to stdout ignoring the python stdout encoding.

#88 left tests in a broken state. This wasn't noticed since CI had a bug that prevented the step from failing despite tests failing. This PR fixes both of these issues.

Network mode is currently set to "host". That's considered pretty insecure; instead, we should just connect port 8060 to the host. I tried that way and it worked for me, is there a different reason we should be using host mode?

At the moment, the only way to use snekbox is to follow the instructions in the README. It would be nice if there were also instructions on how to quickly get a working container up and running, even if it's just a script, although it'd be better if there were instructions on how to use the image on the Docker hub.

This is further to the discussions in #meta and the 2019 Hacktoberfest channel, and https://github.com/python-discord/site/issues/210

It would be excellent if Snekbox supported a kind of "session mode". Such a mode would allow one to run a persistent Python process in order to serve up a Python REPL or other long-running interactive application. There are some hang-ups to this approach, however..

Shared Processes

Having Python processes that are shared between multiple sessions is probably not possible for a code-based REPL, as it's pretty easy to escape the code session and mess with the process or other sessions running in the process.

This means that running a process per-REPL with some kind of time limit or max process limit or something is probably the best approach, but this could get heavy on resources if we're not careful

lemon suggested looking at PEP 0554 on Discord, which could be very useful for an eventual implementation when it comes to shared processes.

Worker Routing

The current design of Snekbox makes routing to specific workers impossible, and they don't have a mechanism of communicating with each other in order to route to the correct Python process. Additionally, Gunicorn does not have any way to route requests to specific workers - it's a fundamental part of how Gunicorn works that prevents us from doing this.

That said, there are a few approaches I think we could take.

1. A separate daemon could be run that supports asynchronous access and dispatches to persistent processes running under nsjail as required
2. Falcon could be dropped and replaced with something that is designed with asynchronous programming in mind and is intended to be run as a single worker
3. REPL scope could be serialised and stored in the database somehow, and reloaded on each request - but this could make operations very slow indeed

I'm entirely unsure what the best approach to this would be, however. The nature of webapps means that you usually distribute load between workers in order to keep things fast, but doing that means you lose a way to route to the same worker for any set of requests.

Does anyone have any comments on this?

As you may know, NsJail complicates things when testing locally because it requires root to run. The simplest approach is to basically offer nothing, but I don't think people are too keen on installing NsJail themselves and running as root.

I was thinking of running tests in a Docker container. Currently, getting files from the container is only needed for working with coverage.py. There are two options to getting the outputs of coverage:

1. Mount the local snekbox repository into the container using a bind mount.
2. No bind mount or volumes. Just use docker cp to copy files

Either way, the files will be owned by root. However, this can easily be fixed with chown. I already had a lot of stuff figured out for testing in containers it works quite well. Though file permissions seem easy to fix, the issue made me start reconsidering the entire container approach.

I'm not particularly sure if this is the proper place to ask but I'm having issues running snekbox. It works fine on my local PC (Windows) but when I try and run it on my VPS (Linux, Debian) I get a PermissionError while trying to run code. It seems to have something to do with cgroup but I'm not particularly experienced with that so I can't figure out how to fix it. Traceback that happens each time I try and run code on it:

Traceback (most recent call last):
  File "/snekbox/snekbox/api/resources/eval.py", line 87, in on_post
    result = self.nsjail.python3(code, py_args=args)
  File "/snekbox/snekbox/nsjail.py", line 190, in python3
    cgroup = self._create_dynamic_cgroups()
  File "/snekbox/snekbox/nsjail.py", line 96, in _create_dynamic_cgroups
    (mem / "memory.limit_in_bytes").write_text(mem_max, encoding="utf-8")
  File "/usr/local/lib/python3.9/pathlib.py", line 1275, in write_text
    with self.open(mode='w', encoding=encoding, errors=errors) as f:
  File "/usr/local/lib/python3.9/pathlib.py", line 1242, in open
    return io.open(self, mode, buffering, encoding, errors, newline,
  File "/usr/local/lib/python3.9/pathlib.py", line 1110, in _opener
    return self._accessor.open(self, flags, mode)
PermissionError: [Errno 13] Permission denied: '/sys/fs/cgroup/memory/snekbox-6a523f3a-3eaf-4da0-8e8f-dcfd0e0f425b/memory.limit_in_bytes'

Initially /sys/fs/cgroup/memory didn't exist, after searching around to no avail I just made the folder with mkdir, after which it got populated with various other stuff but not any folders under snekbox.

If I could get any help on this from people with more experience than me, it'd be greatly appreciated. And in the case that it's just a distro issue (which I don't think it is), I'm more than happy to just swap distros on my server.

Closes #114, #103

Implementing a memory-based file system, running written python files, and sending binary files from snekbox to requester.

Permission specifications

1. We will create a container directory at /snekbox/memfs, which will be mounted as a tmpfs memory directory. The memfs folder is execute-only for user code with no read/write permissions.
2. Each snekbox run instance will have a new mounted tmpfs on the docker system created with a unique ID, a subfolder of this (/home) is then mounted as R/W folders onto the NSJail instance.
3. No instance should be able to read other instance filesystem mounts as they are not mounted to NSJail.

# Docker host
|- /memfs/ab043b4e-5955-4734-b8e2-998c0f6c7131/home/main.py

# Visible to instance user
|- /home/main.py

4. This UUID directory is read/execute only for the user code
5. There is also a home folder, which has read/write permission for nsjail, and is mounted into nsjail as a R/W folder at /home
6. The $HOME env var has also been set to point to this mounted folder in the nsjail. (Required for libraries like matplotlib to work)

Usage

The user has a cwd in /home which they can read/write in. In this directory there will be a output folder. Any files written within this folder will be uploaded.

The file system per instance is default to be limited at 48MiB, can be changed in the cfg file.

Current plan

File system

• [x] 1. Implement a unique tempdir generator backed by dev/shm
• [x] 2. Change runtime config to mount an unique tempdir to each snekbox execution
• [x] 3. Make this the working directory for the python file
• [x] 4. Use a written python file for better compatibility with file usage (__file__) and other usages such as inspect.getsource

Sending binary files via API

• [x] 1. Scan for created files after snekbox execution completes
• [x] 2. Serialize binary file into JSON, with data and name (possible base64 encoding and / or compression)
• [x] 3. Add API route
• [x] 4. Add a user-code side method for files to be marked sent, or some standard (like naming the file "out.*")
• [x] 5. Formalize API route specifications and docs

API Request changes

Backwards-compatible usage of input is retained. If input is supplied, args defaults to -c. Either one of input or args must be supplied.

Compatible mode with input

{
    "input": "[i for i in range(1000)]",
    "args": ["-m", "timeit"]
}

New args without input

{
    "args": ["-m", "timeit", "[i for i in range(1000)]"]
}

Sending files to be run

{
    "args": ["main.py"],
    "files": [
        {
            "name": "main.py",
            "content": "print(1)"
        }
    ]
}

API response additions

(Previous)

{
	"stdout": "Hello\n",
	"returncode": 0
}

(New) New dictionary key files of type list[dict], holding dictionaries representing attachments Sub-dictionaries will have type dict[str, str | int] Attachment dict keys:

• "name" (str)
• "size" (int) Content size in bytes (before compression)
• "content" (str) Base64 encoded bytes

{
	"stdout": "Hello\n",
	"returncode": 0,
	"files": [
		{
			"name": "output.png",
			"size": 57344,
			"content": "SGVsbG8...=",
		},
	]
}

Demo Images

Description

Install multiple python versions within the container, and add an option in the API to select between them.

Rationale

Currently, the bot supports evaluation of both 3.10 and 3.11 code, something I think should be kept and expanded in the future (to include major/important python versions). This is achieved by using the latest 3.10 image published here. This is fine for now, but it makes it harder to provide feature parity and avoid issues when users switch between modes, and it makes it difficult to patch and fix security issues.

Implementation

The way I see us doing this is by building all the versions of python we want to support directly into the image, then aiming the jail at the right executable. This would simplify implementation by having one server, with one codebase, while allowing us to evaluate user code on any version.

Specifics

Some more specific implementation details that can be up for debate:

1. The docker image will simply install whatever versions of python you pass to it in the build arg, to well defined locations
2. We'll build two images in CI, one with all versions we want supported, and one lightweight for dev
3. We can expose an endpoint to get all supported versions
4. We should investigate the best way to install all the versions to the docker image. I imagine the best solution is to just start from an image with all the required build-tools, install all the python versions, and use that as the base for the image. I found this docker image which has all the versions from 2.3-3.9 installed, and it comes out to a whooping... 600 MBs! I don't think this specific project would work for us since it hasn't been updated since 3.9, but it does make the idea seem feasiable.
5. Are the exposed packages going to be an issue, or can we simply install all the same versions to all python interpreters.

I'm not happy with the current state of documentation for the API, as it requires looking through docstrings in the source code. I've been considering using an OpenAPI specification and generating a website with documentation using SwaggerUI or ReDoc.

Because snekbox is intended to be hosted by the users, I believe users would find the documentation more convenient and accessible if hosted on GitHub pages. The alternative would be to add a route to Falcon, but this would require users to be able to access their own snekbox instance during development, or temporarily create one.

There are three main approaches to this:

1. Manually write an OpenAPI spec and manually ensure the Falcon implementation is compliant.
2. Generate an OpenAPI spec from the code.
   • spectree comes with a plugin for Falcon.
   • apispec has a third-party plugin for Falcon.
3. Generate Falcon code from an OpenAPI spec.
   • falcon-openapi

I'm not commited to using OpenAPI. If there's an alternative for generating nice REST API documentation, I'm open to suggestions.

Approach 1

The upside is that there are no additional dependencies and no code changes requires in snekbox. The downside is that there are 2 things to maintain instead of one. I don't mind that too much though. The bigger concern is making sure they are in-sync. Since the code and spec are in separate files, it seems more easy for them to go out of sync. If there is a tool that can automatically verify an implementation is compliant with a spec, I would be more inclined to go for this option.

Approach 2

The main concern is how to access the OpenAPI spec from the GH Pages repo. The spec could either live on this repo or on the GH pages repo; not sure which makes more sense. CI could generate the spec and then create a commit if it detects a change, but that feels hacky. Alternatively, it could just check if a spec needs to be regenerated without actually doing it, but it would be a bit tedious for the PR author to have to generate it.

The other concern is that it feels backwards to generate a spec from code. Usually it's the other way around: code is written based on a specification. However, I think the OpenAPI spec is more of a means to an end to get some nice documentation. I don't think there is value in the spec itself e.g. for someone to implement the same API in another language.

spectree

It's designed with the intent of documentation being added as a route to the existing API. However, it is possible to avoid adding the route and instead exporting the spec for use elsewhere. It is heavier on dependencies since it mandates the use of pydantic, though it would replace jsonschema since spectree supports validation too. Maybe it evens out.

falcon-apispec

Its intent is to directly generate the spec. It is lighter on dependencies, but I don't like its approach of relying on YAML in docstring.

It doesn't support Falcon 3 and seems unmaintained, but there is a PR that supposedly fixes this. Snekbox technically doesn't need any features of Falcon 3.0, but I don't want to downgrade to an unmaintained version. Furthermore, I don't want to use a library that's unmaintained.

Approach 3

This avoids the problem of generating a spec, since it will be manually written. However, generating code from the spec does feel a bit hacky, and I'm not sure how robust falcon-openapi is. It doesn't seem maintained. Unclear if Falcon 3 is supported (if it is, it probably uses things that are deprecated). This statement from the library also doesn't instil confidence:

I am unsure if operationId will make it into the final version. I may change this to only check for the x-falcon property. I plan on doing more research to determine if this an appropriate way to use the operationId property.

For those of us running our own snekbox instances, an informational endpoint would be useful. This could list the api version, the current version of snekbox, and perhaps the installed packages.

Several users have expressed a desire to interact with files with their executed code. I believe such feature is in scope and that it would be a great feature. Supporting it would require a way to mount an isolated, size-capped r/w directory for each NsJail instance. Implementation details need to be investigated to determine the feasability of this feature.