An extension for Arma 3 that lets you write extensions in Python 3

Overview

Pythia logo

Pythia

An Arma 3 extension that lets you to write python extensions for Arma 3. And it's really simple and straightforward to use!

Pythia's Discord server.

TL;DR:

Having the following function defined:

def my_function(some_string, number, the_truth):
    return ["awesome", 42, True, (1, 3.5)]

And calling this SQF code:

["MyAwesomeModule.my_function", ["arg1", 3.14256, False]] call py3_fnc_callExtension

Will return this to your SQF code:

["awesome", 42, True, [1, 3.5]]

Features:

  • Full type conversion both ways: pass in an SQF array of ints, get a python array of ints. Return a tuple of a float, a bool and a string, get an SQF array containing the float, bool and the string
  • Embedded python installation (you don't have to install anything; just run the mod)
  • Python code stored inside @YourOwnMod directory
  • Python modules can call and reuse each other, even between separate Arma mods
  • Background Python threads
  • Cython and other compiled python extensions "just work" (C-like speed for performance-critical modules)
  • Extendable python environment through pip
  • Proven to work with libraries such as numpy, scipy, matplotlib, PyQt5, etc...
  • Automatic python code reloader for easier development
  • Calling SQF back from Python (experimental, using asyncio syntax)
  • Allows returning more than 10240 characters from the extension transparently
  • Annoys sloppy SQF developers with correct code indentation since Day One ;)

Potential features

These features could be implemented quite easily but for some reason have never been done. Want them in or want to help developing them? Contact the developers!

  • Mods contained inside single .pbo files
  • Calling functions in the background and polling for them

Example mods:

The following are mods that use Pythia to accomplish their goal.

Frontline

Dynamic Frontline in action

Frontline is like Squad but done in Arma. Like Project Reality: Arma 3 but better. With a Dynamic Frontline feature that moves as you conquer terrain (and a bunch of other features).

The frontline computation is done in Python, with the use of numpy, scipy, matplotlib and custom Cython code.

ObjectPlacementTBH

ObjectPlacementTBH

It's just a tool for object placement, to be honest... ;)

Pythia is used for loading in xml files, file IO, writing images using PIL, loading layers.cfg (using Armaclass). The newest version is using PyQt5 to display Qt widgets over the Arma window.

Status

Current status: Finishing touches before 1.0. You can use it right now - it's stable. Yes, really.

If you are serious about using Pythia, see the issues page and especially this one. You can contact me to ask for planned changes, on Pythia's Discord server. I don't bite :).

Example usage

Your directory structure:

@MyAwesomeMod/
├── Addons/  # (...)
└── python_code/  # Can be named however you want; you can have more than one
    ├── $PYTHIA$  # Contains the name of your python package, for example: MyAwesomeModule
    ├── __init__.py
    ├── module.py
    ├── cython_module.cp37-win_amd64.pyd  # Compiled Cython code, because we can!
    └── cython_module.cp37-win32.pyd      # Same code but for 32-bit Arma

__init__.py:

def my_function(my, arguments):
    return ["awesome", 42, True, (1, 2)]

module.py:

from .cython_module import stuff  # You can import code from other modules, obviously

def call_stuff():
    return stuff()

Now run Arma 3 with -mod=@Pythia;@MyAwesomeMod and execute the following:


Console:

["MyAwesomeModule.my_function", [3.14256, False]] call py3_fnc_callExtension

Result:

["awesome", 42, True, [1, 2]]

Console:

["MyAwesomeModule.module.call_stuff"] call py3_fnc_callExtension

Result:

["Hello world from a Cython module!"]

Note: MyAwesomeModule is the string that was in the $PYTHIA$ file. You can use any string you want here, obviously.

Performance:

The code is written with performance in mind, meaning that function lookups are cached to limit the number of getattrs, for example. However, keep in mind that the accessibility requirements (SQF <=> Python type conversion) and the general overhead caused by BIs design choice of allowing passing only strings to callExtension must take its toll. I'm open to implementing an alternate set of restricted commands that swap convenience for speed, if required, though...

As such, it is suggested to limit the number of python calls in each frame. It is still faster to call one function with two sets of arguments than two functions, one right after the other.

Concrete numbers:

The test below was executed by calling pythia.ping with different types of arguments, meaning that each list of arguments had to be converted to python and the return value had to be converted back to SQF. Each test was conducted 3 times and the lowest value was written down.

The exact arguments for each test can be found on the scratchpad.

# Type of arguments 10 arguments 100 arguments
1 Integers 0.0198 ms 0.0858 ms
2 Floats 0.0225 ms 0.1091 ms
3 Booleans 0.0155 ms 0.0580 ms
4 Strings 0.0161 ms 0.0580 ms
5 Arrays with ints 0.0318 ms 0.2086 ms
6 Empty arrays 0.0153 ms 0.0555 ms

Note that you will probably usually pass a number of arguments lower than 10 (and if you don't, your function will most probably be slower than the (de)serializing overhead) so you can assume that each Pythia call takes around 0.02 ms on a recent computer.

This allows for under 150 Pythia calls per frame if you want to stay under the 3ms arbitrary limit (arbitrary because the callExtension calls are not really limited by the engine, they just block execution until finished).

Note: You may get away with calls that take even 100 ms server-side, if they happen rarely enough.

Note: If your code is REALLY big and slow consider using a background Python thread. See below.

Your own mod development

Code reloader

Note: The reloader currently only works for native python code! If your code uses Cython or custom C extensions (dll/pyd files) you will get errors when reloading.

To turn the reloader on, simply call:

["pythia.enable_reloader", [True]] call py3_fnc_callExtension

The reloader will then watch all the imported Pythia modules for changes and if any source file is updated, every loaded module will be reloaded on the next Pythia call. Builtin python modules and modules installed with pip are exempt from reloading.

Important: objects in the global namespace will stay as they are. If such an object points to data inside a module, that data will point to the old (not reloaded) module instance, even after the reload! (the module will not be garbage-collected as long as there is something pointing to it).

This can lead to situations where you have two instances of the same module loaded in-memory. The recommended way is to create pre_reload/post_reload hooks for your module and reinitialize your new module and global variables with the data from the old module, on reload. See below.

Keeping your module data between reloads

To prevent losing data during reload your module needs to have __pre_reload__ and __post_reload__ hooks declared. __post_reload__(state) will be called with the exact same arguments as have been returned by __pre_reload__() before reloading the module.

The reloader works as follows: (simplified pseudo-python code)

def reload_all_modules():
    for module in pythia_modules:
        modules_state[module.__name__] = module.__pre_reload__()

    reload_modules(pythia_modules)

    for module in pythia_modules:
        module.__post_reload__(modules_state[module.__name__])

Creating those functions is purely optional.

Note that using the reloader causes a time penalty for each call and shouldn't be used in production.

Threads

TODO

Calling SQF back from Python

TODO

Note that you need to be comfortable with using asyncio-type code to use this feature!

Installing

  • Build the mod yourself or get a prebuilt version.
  • Copy @Pythia to Arma 3 directory.
  • (Optional for development) Create a python directory in Arma 3 directory. Put all your python functions there.
  • If any of your pythia mods contains a requirements.txt file, simply drag it onto @Pythia\install_requirements.bat to install all the requirements to both python installations before running the game.

Pythia development

Building requirements

  • Python 3.7 64-bit and 32-bit (you need to build both the 32-bit and 64-bit extensions)
  • Visual Studio Community 2017

Building

  • File -> Open -> CMake: CmakeLists.txt, make sure that Release configuration is selected (instead of Debug) and CMake -> Build All. Remember to build for both x86 and x86_64.
  • Run python tools\make_pbos.py to build the required PBO files.
  • Run python tools\create_embedded_python.py @Pythia to download and prepare x86 and x86_64 python installations.

Contributing

All contributions are welcome! Is something in the documentation unclear? Feel free to submit a PR or drop a note on Pythia's Discord server.

Comments
  • Build the python embedded executable from sources

    Build the python embedded executable from sources

    https://stackoverflow.com/a/52930398

    In case I need to rebuild Python 3.7 myself, in light of the latest CVE (the official download page doesn't have binary builds for the last Python 3.7 release, only sources, as they want people to move to 3.8)

    opened by overfl0 5
  • Lower level interfacing

    Lower level interfacing

    Hello, If I am missing the point completely here I apologize, but is it possible to, for example create a group or unit in python directly? or just return primitive datatypes? Is this possible currently, will it be possible, can I help make it possible? and is it possible to hook into events such as pre init, post init and event handlers such as hit and killed? Cheers. Lonja

    opened by Tirpitz93 5
  • Port to Python 3.9

    Port to Python 3.9

    • [x] Update create_embedded_python.py to be able to be parametrized from github actions
    • [x] Python 3.8 - https://docs.python.org/3/whatsnew/3.8.html
    • [x] Python 3.9 - https://docs.python.org/3/whatsnew/3.9.html
    • [x] Python 3.10 (optionally) - https://docs.python.org/3/whatsnew/3.10.html
    opened by overfl0 4
  • Python thread may not be fully working

    Python thread may not be fully working

    Context: http://www.indelible.org/ink/python-reloading/

    Preparation: pip install reloader Write the file locally: https://github.com/jparise/python-reloader/blob/master/monitor.py

    Test:

    import reloader
    from .monitor import Reloader
    reloader.enable(blacklist=['itertools', 'logging', 'math', 'matplotlib', 'numpy', 'scipy', 'sys', 'skimage'])
    r = Reloader()
    
    ...
    
    r.poll() # In functions calls
    

    A strange thing is happening: after adding logger.critical() messages inside ModuleMonitor::_scan it looks like the Monitor thread is only running when a foreground function is called. As soon as that function is finished, the background thread looks to be frozen, without any activity.

    opened by overfl0 4
  • Ensure additional paths are added when using `pywin32`

    Ensure additional paths are added when using `pywin32`

    This is probably caused by the site module being enabled, but we need to make sure we can enable site while not leaking the environment through user site directories, etc...

    opened by overfl0 3
  • Getting Error while iterating iterator. Report a bug!!!

    Getting Error while iterating iterator. Report a bug!!!

    When running the following code:

    async def connectToTwitch(channelName, oAuthKey, nickName):
        try:
            nickname = nickName
            token = oAuthKey
            channel = channelName
            client = Client()
            await client.login_oauth(token, nickname)
        except Exception as e:
            return str(e)
    

    I get: image This piece of code is run in game using the following:

    _channelName = twitchBot_chat_channelName; 
    _oAuthKey = twitchBot_chat_oAuth; 
    _nickName = twitchBot_chat_nickName;
    ["CCB.connectToTwitch", [_channelName, _oAuthKey, _nickName]] call py3_fnc_callExtension;
    

    The client object comes from pytmi

    opened by JessevGool 3
  • Move to python 3.7

    Move to python 3.7

    • [x] Change the installing script used by AppVeyor
    • [x] Replace instances of python35 in the source code
    • [x] Replace instances of python35 in the project include dirs and linker dirs
    • [x] Check the ._pth file is read correctly
    • [x] Ensure the user_site is not added to the path
    • [x] Have a separate directory per python version
    • [x] Install pip correctly
    • [x] Check if it works fine with all the other currently used mods
    • [x] Check the default paths passed in the registry

    https://docs.python.org/3.6/using/windows.html#finding-modules https://pymotw.com/2/site/

    low priority 
    opened by overfl0 3
  • Create a self-contained python distribution with Pythia

    Create a self-contained python distribution with Pythia

    This is almost done, but does not seem to be working for some people.

    Some investigation is required. Probably logging will need to be fixed for this to work, as well. #23

    opened by overfl0 3
  • Not joining python threads prevents Arma from terminating the process

    Not joining python threads prevents Arma from terminating the process

    See #11 for a reproduction.

    When a Python thread is created but not terminated (joined). It will continue running after Arma is shut down, keeping the arma3.exe process running.

    It is unclear what is causing this, but at first sight, it should have received a DLL_PROCESS_DETACH. So either it doesn't receive it, or a running python ignores the Py_Finalize(); call we're running in the destructor.

    opened by overfl0 3
  • Use parseSimpleArray - benchmark and consider implementing

    Use parseSimpleArray - benchmark and consider implementing

    Use parseSimpleArray instead of call compile.

    Check if we are generating anything more than what is reported to be parsable by parseSimpleArray.

    We've gotten rid of SQF from Python so, in theory, this should not matter anyway, now.

    • [x] Check python outputs
    • [x] Check what happens when the extension throws error messages
    • [ ] Extend the unit tests for overflows, etc...
    • [x] Replace the call and test
    • [x] Benchmark
    opened by overfl0 2
  • Bump numpy from 1.21.0 to 1.22.0 in /tests/@CythonNumpyMod

    Bump numpy from 1.21.0 to 1.22.0 in /tests/@CythonNumpyMod

    Bumps numpy from 1.21.0 to 1.22.0.

    Release notes

    Sourced from numpy's releases.

    v1.22.0

    NumPy 1.22.0 Release Notes

    NumPy 1.22.0 is a big release featuring the work of 153 contributors spread over 609 pull requests. There have been many improvements, highlights are:

    • Annotations of the main namespace are essentially complete. Upstream is a moving target, so there will likely be further improvements, but the major work is done. This is probably the most user visible enhancement in this release.
    • A preliminary version of the proposed Array-API is provided. This is a step in creating a standard collection of functions that can be used across application such as CuPy and JAX.
    • NumPy now has a DLPack backend. DLPack provides a common interchange format for array (tensor) data.
    • New methods for quantile, percentile, and related functions. The new methods provide a complete set of the methods commonly found in the literature.
    • A new configurable allocator for use by downstream projects.

    These are in addition to the ongoing work to provide SIMD support for commonly used functions, improvements to F2PY, and better documentation.

    The Python versions supported in this release are 3.8-3.10, Python 3.7 has been dropped. Note that 32 bit wheels are only provided for Python 3.8 and 3.9 on Windows, all other wheels are 64 bits on account of Ubuntu, Fedora, and other Linux distributions dropping 32 bit support. All 64 bit wheels are also linked with 64 bit integer OpenBLAS, which should fix the occasional problems encountered by folks using truly huge arrays.

    Expired deprecations

    Deprecated numeric style dtype strings have been removed

    Using the strings "Bytes0", "Datetime64", "Str0", "Uint32", and "Uint64" as a dtype will now raise a TypeError.

    (gh-19539)

    Expired deprecations for loads, ndfromtxt, and mafromtxt in npyio

    numpy.loads was deprecated in v1.15, with the recommendation that users use pickle.loads instead. ndfromtxt and mafromtxt were both deprecated in v1.17 - users should use numpy.genfromtxt instead with the appropriate value for the usemask parameter.

    (gh-19615)

    ... (truncated)

    Commits

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    • @dependabot use these labels will set the current labels as the default for future PRs for this repo and language
    • @dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language
    • @dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language
    • @dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

    You can disable automated security fix PRs for this repo from the Security Alerts page.

    dependencies 
    opened by dependabot[bot] 2
  • CVE-2007-4559 Patch

    CVE-2007-4559 Patch

    Patching CVE-2007-4559

    Hi, we are security researchers from the Advanced Research Center at Trellix. We have began a campaign to patch a widespread bug named CVE-2007-4559. CVE-2007-4559 is a 15 year old bug in the Python tarfile package. By using extract() or extractall() on a tarfile object without sanitizing input, a maliciously crafted .tar file could perform a directory path traversal attack. We found at least one unsantized extractall() in your codebase and are providing a patch for you via pull request. The patch essentially checks to see if all tarfile members will be extracted safely and throws an exception otherwise. We encourage you to use this patch or your own solution to secure against CVE-2007-4559. Further technical information about the vulnerability can be found in this blog.

    If you have further questions you may contact us through this projects lead researcher Kasimir Schulz.

    opened by TrellixVulnTeam 0
  • Add Python 3.11 support

    Add Python 3.11 support

    We're NOT planning to switch to Python 3.11 anytime soon (because there are no wheels released for 3.11 yet) but it would be good to be ready to do so in the future.

    https://docs.python.org/3.11/whatsnew/3.11.html

    opened by overfl0 0
  • Alternative callExtension syntax - benchmark and consider implementing

    Alternative callExtension syntax - benchmark and consider implementing

    • [ ] Replace SQF side with error handling
    • [ ] C++ side handling with argument list
    • [ ] Check what happens on timeouts
    • [ ] Check if everything works
    • [ ] Benchmark
    opened by overfl0 0
  • Ability to include other files from adapter.py

    Ability to include other files from adapter.py

    This enhancement may be blocked by #7.

    Allow the storage of multiple .py files that could then be included by adapter.py so that all the code is not in one big file.

    This won't impact performance as once a module is loaded to memory, import something has no effect so it will be loaded only once.

    low priority 
    opened by overfl0 3
Releases(1.0.0)
Owner
Lukasz Taczuk
C/C++/Python developer with a penchant for IT Security and Reverse-Engineering.
Lukasz Taczuk
A site that went kinda viral that lets you put Bernie Sanders in places

Bernie In Places An app that accidentally went viral! Read the story in WIRED here Install First, create a python virtual environment, and install all

null 310 Aug 22, 2022
uMap lets you create maps with OpenStreetMap layers in a minute and embed them in your site.

uMap project About uMap lets you create maps with OpenStreetMap layers in a minute and embed them in your site. Because we think that the more OSM wil

null 771 Dec 29, 2022
An addin for Autodesk Fusion 360 that lets you view your design in a Looking Glass Portrait 3D display

An addin for Autodesk Fusion 360 that lets you view your design in a Looking Glass Portrait 3D display

Brian Peiris 12 Nov 2, 2022
A program that lets you use your tablet's tilting to emulate an actual joystick on a Linux computer.

Tablet Tilt Joystick A program that lets you use your tablet's tilting to emulate an actual joystick on a Linux computer. It's called tablet tilt joys

null 1 Feb 7, 2022
sawa (ꦱꦮ) is an open source programming language, an interpreter to be precise, where you can write python code using javanese character.

ꦱꦮ sawa (ꦱꦮ) is an open source programming language, an interpreter to be precise, where you can write python code using javanese character. sawa iku

Rony Lantip 307 Jan 7, 2023
Create VSCode Extensions with python

About Create vscode extensions with python. Installation Stable version: pip install vscode-ext Why use this? Why should you use this for building VSc

Swas.py 134 Jan 7, 2023
Why write code when you can import it directly from GitHub Copilot?

Copilot Importer Why write code when you can import it directly from GitHub Copilot? What is Copilot Importer? The copilot python module will dynamica

Mythic 41 Jan 4, 2023
lets learn Python language with basic examples. highly recommended for beginners who just start coding.

Lets Learn Python ?? Learn python from basic programs. learn python from scratch. 1.Online python compiler: https://www.onlinegdb.com/online_python_co

Subhranshu Choudhury 1 Jan 18, 2022
Write complicated anonymous functions other than lambdas in Python.

lambdex allows you to write multi-line anonymous function expression (called a lambdex) in an idiomatic manner.

Xie Jingyi 71 May 19, 2022
💘 Write any Python with 9 Characters: e,x,c,h,r,(,+,1,)

?? PyFuck exchr(+1) PyFuck is a strange playful code. It uses only nine different characters to write Python3 code. Inspired by aemkei/jsfuck Example

Satoki 10 Dec 25, 2022
Svg-turtle - Use the Python turtle to write SVG files

SaVaGe Turtle Use the Python turtle to write SVG files If you're using the Pytho

Don Kirkby 7 Dec 21, 2022
You can easily send campaigns, e-marketing have actually account using cash will thank you for using our tools, and you can support our Vodafone Cash +201090788026

*** Welcome User Sorry I Mean Hello Brother ✓ Devolper and Design : Mokhtar Abdelkreem ========================================== You Can Follow Us O

Mo Code 1 Nov 3, 2021
This is a modified variation of abhiTronix's vidgear. In this variation, it is possible to write the output file anywhere regardless the permissions.

Info In order to download this package: Windows 10: Press Windows+S, Type PowerShell (cmd in older versions) and hit enter, Type pip install vidgear_n

Ege Akman 3 Jan 30, 2022
Generate Openbox Menus from a easy to write configuration file.

openbox-menu-generator Generate Openbox Menus from a easy to write configuration file. Example Configuration: ('#' indicate comments but not implement

null 3 Jul 14, 2022
A simple way to read and write LAPS passwords from linux.

A simple way to read and write LAPS passwords from linux. This script is a python setter/getter for property ms-Mcs-AdmPwd used by LAPS inspired by @s

Podalirius 36 Dec 9, 2022
Write a program that works out whether if a given year is a leap year

Leap Year ?? This is a Difficult Challenge ?? Instructions Write a program that works out whether if a given year is a leap year. A normal year has 36

Rodrigo Santos 0 Jun 22, 2022
Write Streamlit apps using Notion! (Prototype)

Streamlit + Notion test app Write Streamlit apps using Notion! ☠️ IMPORTANT: This is just a little prototype I made to play with some ideas. Not meant

Thiago Teixeira 22 Sep 8, 2022
This is a far more in-depth and advanced version of "Write user interface to a file API Sample"

Fusion360-Write-UserInterface This is a far more in-depth and advanced version of "Write user interface to a file API Sample" from https://help.autode

null 4 Mar 18, 2022