pyinfra automates infrastructure super fast at massive scale. It can be used for ad-hoc command execution, service deployment, configuration management and more.

Nick Barrett

Last update: Dec 29, 2022

Related tags

DevOps Tools python infrastructure high-performance configuration-management remote-execution cloud-management pyinfra

Overview

pyinfra automates/provisions/manages/deploys infrastructure super fast at massive scale. It can be used for ad-hoc command execution, service deployment, configuration management and more. Core design features include:

🚀 Super fast execution over thousands of hosts with predictable performance.
🚨 Instant debugging with stdout & stderr output on error or as required (-v|-vv|-vvv).
📦 Extendable with any Python package as configured & written in standard Python.
💻 Agentless execution against SSH/Docker/subprocess/winrm hosts.
❗️ Two stage process that enables --dry runs before executing any changes.
🔌 Integrated with Docker, Vagrant/Mech & Ansible out of the box.

When you run pyinfra you'll see something like (non animated version):

Quickstart

pyinfra can be installed via pip:

pip install pyinfra

Now you can execute commands & operations over SSH:

# Execute an arbitrary shell command
pyinfra my-server.net exec -- echo "hello world"

# Install iftop apt package if not present
pyinfra my-server.net apt.packages iftop sudo=true update=true

These can then be saved to a deploy file, let's call it deploy.py:

from pyinfra.operations import apt

apt.packages(
    name='Ensure iftop is installed',
    packages=['iftop'],
    sudo=True,
    update=True,
)

And executed with:

pyinfra my-server.net deploy.py

pyinfra @docker/ubuntu deploy.py

Documentation

Comments

WIP: Winrm

Please leave this PR open.

I'll need your help with a few things.

But, got this working:

$ pyinfra 192.168.2.246 --winrm_username vagrant --winrm_password vagrant --winrm_port 5985 exec -- echo hello
--> Loading config...
--> Loading inventory...

--> Connecting to hosts...
    [192.168.2.246] Connected

--> Proposed changes:
    Ungrouped:
    [192.168.2.246]   Operations: 1   Commands: 1

--> Beginning operation run...
--> Starting operation: Server/Shell ('echo hello',)
[192.168.2.246] >>> 'echo hello'
    [192.168.2.246] Success

--> Results:
    Ungrouped:
    [192.168.2.246]   Successful: 1   Errors: 0   Commands: 1/1

:-D

opened by mkinney 34

sudo password

Is your feature request related to a problem? Please describe. So far I can't find examples/documentation if users with password for sudo can be used

Describe the solution you'd like I'd like to enter a password to use for sudo just like other cfmgt tools
Operations Connectors

opened by pathcl 29
Fix support `use_sudo_password` in facts
Describe the bug If I do have user present it does not continue because the user already exist.

To Reproduce server.user( {'Ensure users is present'}, 'someuser', present=True, )

Expected behavior To not error and continue.

Meta

Include output of pyinfra --support. --> Support information:

If you are having issues with pyinfra or wish to make feature requests, please check out the GitHub issues at https://github.com/Fizzadar/pyinfra/issues . When adding an issue, be sure to include the following:

System: Linux Platform: Linux-5.6.19-300.fc32.x86_64-x86_64-with-glibc2.2.5 Release: 5.6.19-300.fc32.x86_64 Machine: x86_64 pyinfra: v0.16 Executable: /home/nardusg/.local/bin/pyinfra Python: 3.8.3 (CPython, GCC 10.1.1 20200507 (Red Hat 10.1.1-1))

How was pyinfra installed (source/pip)? pip

Include pyinfra-debug.log (if one was created)

Consider including output with -vv and --debug. --> Starting operation: Ensure users is present [pyinfra.api.operations] Starting operation Ensure users is present on someserver [pyinfra.api.connectors.ssh] Running command on someserver: (pty=None) env SUDO_ASKPASS=/tmp/pyinfra-pyinfra-sudo-askpass *** sudo -H -A -k sh -c 'useradd -d /home/someuser someuser' [narrem.narra.co.za] >>> env SUDO_ASKPASS=/tmp/pyinfra-pyinfra-sudo-askpass *** sudo -H -A -k sh -c 'useradd -d /home/someuser someuser' [someserver] useradd: user 'someuser' already exists [pyinfra.api.connectors.ssh] Waiting for exit status... [pyinfra.api.connectors.ssh] Command exit status: 9 [narrem.narra.co.za] Error [pyinfra.api.state] Failing hosts: someserver

Bug
opened by nardusg 22

pyinfra error: No such group: file

Describe the bug

Simple ad-hoc command failed with error. I didn't create inventory file, just run this ad-hoc command from bash shell in any directory.

$ pyinfra --debug --user=root 192.168.0.115 exec -- echo
--> Loading config...
--> Loading inventory...
    [pyinfra_cli.inventory] Creating fake inventory...

--> Connecting to hosts...
    [pyinfra.api.connectors.ssh] Connecting to: 192.168.0.115 ({'allow_agent': True, 'look_for_keys': True, 'username': 'root', 'timeout': 10})
    [192.168.0.115] Connected
    [pyinfra.api.state] Activating host: 192.168.0.115
    Use of Inventory.<group_name> is deprecated, please use `Inventory.get_group` instead.
--> pyinfra error: No such group: __file__

Meta

Include output of pyinfra --support.

OS: macOS 10.15.5

$ pyinfra --support
--> Support information:

    If you are having issues with pyinfra or wish to make feature requests, please
    check out the GitHub issues at https://github.com/Fizzadar/pyinfra/issues .
    When adding an issue, be sure to include the following:

    System: Darwin
      Platform: Darwin-19.5.0-x86_64-i386-64bit
      Release: 19.5.0
      Machine: x86_64
    pyinfra: v0.14.5
    Executable: /usr/local/bin/pyinfra
    Python: 3.7.3 (CPython, Clang 11.0.3 (clang-1103.0.32.62))

How was pyinfra installed (source/pip)?

pip.

Bug

opened by iredmail 22

Obfuscate Secrets

Is your feature request related to a problem? Please describe.

Like to obfuscate secrets for commands that use them. For example, in the operation postgresql.role({"create role"}, "deploy", present=True, password=DATABASE_PASSWORD, postgresql_user="postgres")

This will output the value of DATABASE_PASSWORD which is a leak of information to the console or if pyinfra is in a CI system it will come out in the logs.

Describe the solution you'd like

I'd like to see the secret obfuscated to something like three stars ***. Or anything really, length of obfuscation shouldn't match actual string length. When obfuscation is needed should be known so the api shouldn't need to change (I think).

opened by meantheory 15
Add more patterns/examples for deploys
I would like to see more patterns/examples for deploys.

Ansible's docs usually contain a useable snippet as a starting point. Could the pyinfra docs be expanded to include examples? For example, the apt.packages could have an example of what a typical code snippet would look like.

Would like to see examples on using the "when" or a conditional task, so as to make deployments quicker/idempotent.

For example, the "if CentOS" in deploys.rst could show how to also use the "when" format.

Another example would be to check for the existence of a file then use that as a "when" condition.

Yet another example, would be to run a remote shell command and use the output from that as a "when" condition in a task. (a good example of this would be check if docker apt key is installed before needing to download that key)
opened by mkinney 15
Add type annotations

First draft of some type annotations. This will make Mypy complain a lot, but it's a first step on making the cobase leverage type annotations benefits. I'll keep this as a draft for now. One of the decisions to make is whether to add lots of Any for variables with unknown types (or multiple type possibilities), or just to not add the annotation for now.

Ran all tests (both unit and end-to-end with ssh and docker) and everything seems fine (I'm on a macOS with Python 3.10). I also could add tox with the other Python versions to increase test reliability, I don't think I used anything version specific (such as | or TypedDicts and such), so this should be safe.

opened by lowercase00 14

Incorrect su usage for FreeBSD

Using su results in hanging processes running: su root -s /bin/sh -c stat /etc/installurl 1> /dev/null 2> /dev/null && (stat -c 'user=%U…

This is most likely because -s does not mean something like "shell", it means "Set the MAC label to the user's default label as part of the user credential setup" (so does not take an argument), so we end up with sh launched:

ec2-user@freebsd:~ $ su root -s /bin/sh -c stat /etc/installurl
root@freebsd:/home/ec2-user #

While this works:

ec2-user@freebsd:~ $ su root -s -c stat /etc/installurl
1895890688 94 crw--w---- 1 ec2-user tty 94 0 "Jul 23 12:36:27 2020" "Jul 23 12:36:27 2020" "Jul 23 12:36:27 2020" "Dec 31 23:59:59 1969" 4096 0 0 /dev/pts/0
ec2-user@freebsd:~ $ su root -c stat /etc/installurl
1895890688 94 crw--w---- 1 ec2-user tty 94 0 "Jul 23 12:36:27 2020" "Jul 23 12:36:27 2020" "Jul 23 12:36:27 2020" "Dec 31 23:59:59 1969" 4096 0 0 /dev/pts/0

Bug

opened by valpackett 14

Using su_user with use_su_login yields an error with a macOS target
Describe the bug Using su_user with use_su_login yields an error with a macOS target. Using use_su_login alone works without specifying the user, but it does not seem to load and shell profiles (nor .zshrc or .bash_profile):

-> Proposed changes: Groups: mac_vms [vm] Operations: 1 Commands: 1 --> Beginning operation run... --> Starting operation: Test ruby ("commands=['which ruby']",) [pyinfra\api\operations] Starting operation Test ruby on vm [pyinfra\api\connectors\ssh] Running command on vm: (pty=None) su -l the_user -s `which sh` -c 'which ruby' [vm] >>> su -l the_user -s `which sh` -c 'which ruby' [vm] su: Sorry [pyinfra\api\connectors\ssh] Waiting for exit status... [pyinfra\api\connectors\ssh] Command exit status: 1 [vm] Error [pyinfra\api\state] Failing hosts: vm No hosts remaining! --> ←[31m←[1mpyinfra error←[0m:

To bring some context: I have a target macOS machine with chruby set up using a specific version of ruby, under .rubies built with ruby-install. I want to install some gems on the target machine with pyinfra, but this requires it to have chruby run beforehand so that it selects the correct gem version, instead of the one pre-installed with macOS. The commands to do this are in the .zshrc, and when I login to an SSH shell it works as intended. However, even when running with use_su_login = True, it returns the incorrect version of ruby:

server.shell( name = 'Test ruby' ,use_su_login = True ,commands = [ 'which ruby' ] ) # Results in: [vm] >>> sh -c 'which ruby' [vm] /usr/bin/ruby

From an SSH login shell:

> which ruby /Users/USER/.rubies/ruby-2.7.1/bin/ruby

To Reproduce With a macOS target (my target is Catalina 10.15.6) such an operation with both su_user and use_su_login specified:

server.shell( name = 'Test ruby' ,su_user = host.data.user ,use_su_login = True ,commands = [ 'which ruby' ] )

Expected behavior No errors to occur

Meta

Include output of pyinfra --support. Note that I am on 1.0.2 as I ran into some other issues with since 1.0.3. I plan to report the issues as well as soon as I resolve this so I can continue with the other ops.

System: Windows Platform: Windows-10-10.0.19041-SP0 Release: 10 Machine: AMD64 pyinfra: v1.0.2 Executable: C:\Users\the_user\AppData\Local\Programs\Python\Python38-32\Scripts\pyinfra.exe Python: 3.8.3 (CPython, MSC v.1925 32 bit (Intel))

How was pyinfra installed (source/pip)? pip

Thanks!
Bug
opened by harold-b 13
winrm connector is broken
Describe the bug

The winrm connector appears to be broken, I can not run any commands via the connector. It appears make_windows_command() is getting unexpected arguments via **command_kwargs from run_shell_command() in the winrm connector.

To Reproduce

Try to run any command via winrm and you will see: TypeError: make_win_command() got an unexpected keyword argument 'use_sudo_password' 2021-03-16T12:42:44Z <Greenlet at 0x7fb062873290: _run_server_op(<pyinfra.api.state.State object at 0x7fb0628423d0>, Host(@winrm/10.10.10.10), '784a97bf1955d5f7a2b9dd6c1e371e17b73c42bc')> failed with TypeError

This is happening on the latest v1.3.8 release as well as the several prior releases.

Expected behavior

The command completes successfully and returns valid output. The fix may be as easy as enumerating the additional kwargs in run_shell_command and ignoring them.

https://github.com/Fizzadar/pyinfra/blob/master/pyinfra/api/connectors/winrm.py#L105

Meta

Include output of pyinfra --support.

How was pyinfra installed (source/pip)?

Include pyinfra-debug.log (if one was created)

Consider including output with -vv and --debug.

Bug
opened by mfrg 11
yum/dnf packages are split on the last dash, breaking packages with dashes *in the name*
Describe the bug When trying to remove the packages 'vim-enhanced' or 'vim-common' using dnf the removal does not happen.

To Reproduce

dnf.packages( packages=["git","vim-common", "vim-enhanced"], present=False, sudo=True, )

There is also a failing test in the PR https://github.com/Fizzadar/pyinfra/pull/380 .

Expected behavior Packages 'vim-enhanced' and 'vim-common' are removed

Expected output:

sudo -H -n sh -c 'dnf remove -y git vim-common vim-enhanced'

Actual output:

sudo -H -n sh -c 'dnf remove -y git'

Meta Discussion and a test : https://github.com/Fizzadar/pyinfra/pull/380
Bug Operations
opened by alexandrunastase 11

files.put not updating (on r/w filesystem)

Kind of like #878, here's another files.put silently failing to update the file. Before:

bang% ls -l /tmp/net_traffic.aarch64
-rwxrwxr-x 1 drewp drewp 8361192 Dec 30 15:22 /tmp/net_traffic.aarch64

pipe% % ls -l /opt/net_traffic/  
total 8132
-rwxr-xr-x 1 root root 8326736 Dec  2 10:11 net_traffic.aarch64

bang% sudo ssh root@pipe touch /opt/net_traffic/not_a_readonly_fs

pipe% ls -l /opt/net_traffic/
total 8132
-rwxr-xr-x 1 root root 8326736 Dec  2 10:11 net_traffic.aarch64
-rw-r--r-- 1 root root       0 Dec 30 15:48 not_a_readonly_fs

deploy.py follows:

from pyinfra.operations import files, systemd

target = '/opt/net_traffic'

files.directory(target)
files.put(src="/tmp/net_traffic.aarch64", dest=target, mode="a+rx")

files.put(src='./net_traffic.service',
          dest='/etc/systemd/system/net_traffic.service')
systemd.service(service='net_traffic.service',
                daemon_reload=True,
                enabled=True,
                running=True,
                restarted=True)

I ran pyinfra -vvv pipe deploy.py.

pyinfra logs:

      6 --> Connecting to hosts...
      7     [pipe] Connected
      8 
      9 --> Preparing Operations...
     10     Loading: deploy.py
     11 [pipe] >>> sh -c '! (test -e /opt/net_traffic || test -L /opt/net_traffic ) || ( stat -c '"'"'user=%U group=%G mode=%A atime=%X mtime=%Y ctime=%Z size=%s %N'"'"' /opt/net_traffic 2> /dev/null || stat -f '"'"'user=%Su group=%Sg mode=%Sp atime=%a mtime=%m ctime=%c s     11 ize=%z %N%SY'"'"' /opt/net_traffic )'
     12 [pipe] user=drewp group=drewp mode=drwxrwxr-x atime=1672441894 mtime=1670004715 ctime=1670004715 size=4096 '/opt/net_traffic'
     13     [pipe] Loaded fact files.Directory (path=/opt/net_traffic)
     14     [pipe] noop: directory /opt/net_traffic already exists
     15 [pipe] >>> sh -c '! (test -e /opt/net_traffic || test -L /opt/net_traffic ) || ( stat -c '"'"'user=%U group=%G mode=%A atime=%X mtime=%Y ctime=%Z size=%s %N'"'"' /opt/net_traffic 2> /dev/null || stat -f '"'"'user=%Su group=%Sg mode=%Sp atime=%a mtime=%m ctime=%c s     15 ize=%z %N%SY'"'"' /opt/net_traffic )'
     16 [pipe] user=drewp group=drewp mode=drwxrwxr-x atime=1672441894 mtime=1670004715 ctime=1670004715 size=4096 '/opt/net_traffic'
     17     [pipe] Loaded fact files.File (path=/opt/net_traffic)
     18 [pipe] >>> sh -c '! (test -e /opt/net_traffic/net_traffic.aarch64 || test -L /opt/net_traffic/net_traffic.aarch64 ) || ( stat -c '"'"'user=%U group=%G mode=%A atime=%X mtime=%Y ctime=%Z size=%s %N'"'"' /opt/net_traffic/net_traffic.aarch64 2> /dev/null || stat -f '     18 "'"'user=%Su group=%Sg mode=%Sp atime=%a mtime=%m ctime=%c size=%z %N%SY'"'"' /opt/net_traffic/net_traffic.aarch64 )'
     19 [pipe] user=root group=root mode=-rwxr-xr-x atime=1672442515 mtime=1670004716 ctime=1672442515 size=8326736 '/opt/net_traffic/net_traffic.aarch64'
     20     [pipe] Loaded fact files.File (path=/opt/net_traffic/net_traffic.aarch64)
     21 [pipe] >>> sh -c '! (test -e /etc/systemd/system/net_traffic.service || test -L /etc/systemd/system/net_traffic.service ) || ( stat -c '"'"'user=%U group=%G mode=%A atime=%X mtime=%Y ctime=%Z size=%s %N'"'"' /etc/systemd/system/net_traffic.service 2> /dev/null ||      21 stat -f '"'"'user=%Su group=%Sg mode=%Sp atime=%a mtime=%m ctime=%c size=%z %N%SY'"'"' /etc/systemd/system/net_traffic.service )'
     22 [pipe] user=root group=root mode=-rw-r--r-- atime=1672438834 mtime=1670004649 ctime=1670004649 size=273 '/etc/systemd/system/net_traffic.service'
     23     [pipe] Loaded fact files.File (path=/etc/systemd/system/net_traffic.service)
     24 [pipe] >>> sh -c '! (test -e /etc/systemd/system || test -L /etc/systemd/system ) || ( stat -c '"'"'user=%U group=%G mode=%A atime=%X mtime=%Y ctime=%Z size=%s %N'"'"' /etc/systemd/system 2> /dev/null || stat -f '"'"'user=%Su group=%Sg mode=%Sp atime=%a mtime=%m c     24 time=%c size=%z %N%SY'"'"' /etc/systemd/system )'
     25 [pipe] user=root group=root mode=drwxr-xr-x atime=1672438834 mtime=1671657241 ctime=1671657241 size=4096 '/etc/systemd/system'
     26     [pipe] Loaded fact files.Directory (path=/etc/systemd/system)
     27 [pipe] >>> sh -c 'test -e /etc/systemd/system/net_traffic.service && ( sha1sum /etc/systemd/system/net_traffic.service 2> /dev/null || shasum /etc/systemd/system/net_traffic.service 2> /dev/null || sha1 /etc/systemd/system/net_traffic.service ) || true'
     28 [pipe] ba56cf7b6203fb99310c395dd490bd86c090090f  /etc/systemd/system/net_traffic.service
     29     [pipe] Loaded fact files.Sha1File (path=/etc/systemd/system/net_traffic.service)
     30     [pipe] noop: file /etc/systemd/system/net_traffic.service is already uploaded
     31 [pipe] >>> sh -c '! command -v systemctl >/dev/null || systemctl show --all --property Id --property SubState '"'"'*'"'"''

(service lines removed)

   1989     [pipe] Loaded fact systemd.SystemdEnabled (machine=None, user_mode=False, user_name=None)
   1990     [pipe] Ready: deploy.py
   1991 
   1992 --> Proposed changes:
   1993     Ungrouped:
   1994     [pipe]   Operations: 4   Change: 2   No change: 2   
   1995 
   1996 
   1997 --> Beginning operation run...
   1998 --> Starting operation: Files/Directory (/opt/net_traffic)
   1999     [pipe] No changes
   2000 
   2001 --> Starting operation: Files/Put (src=/tmp/net_traffic.aarch64, dest=/opt/net_traffic, mode=a+rx, force=True)
   2002     Failed to upload file, retrying: Failure
   2003 [pipe] file uploaded: /opt/net_traffic/net_traffic.aarch64
   2004 [pipe] >>> sh -c 'chmod a+rx /opt/net_traffic/net_traffic.aarch64'
   2005     [pipe] Success
   2006 
   2007 --> Starting operation: Files/Put (src=./net_traffic.service, dest=/etc/systemd/system/net_traffic.service)
   2008     [pipe] No changes
   2009 
   2010 --> Starting operation: Systemd/Service (service=net_traffic.service, daemon_reload=True, enabled=True, running=True, restarted=True)
   2011 [pipe] >>> sh -c 'systemctl daemon-reload'
   2012 [pipe] >>> sh -c 'systemctl restart net_traffic.service'
   2013     [pipe] Success
   2014 
   2015 
   2016 --> Results:
   2017     Ungrouped:
   2018     [pipe]   Changed: 2   No change: 2   Errors: 0

2003,2005 suggest success, but the new file content has not actually been put:

% ls -l /opt/net_traffic/
total 8132
-rwxr-xr-x 1 root root 8326736 Dec  2 10:11 net_traffic.aarch64
-rw-r--r-- 1 root root       0 Dec 30 15:48 not_a_readonly_fs

Retrying with put(...,force=True) didn't help.

pyinfra 2.6

Bug

opened by drewp 4

iptables.rule make duplicated rules

Describe the bug

iptables.rule make duplicated rules

To Reproduce

from pyinfra import host, logger
from pyinfra.operations import iptables, server, python
from pyinfra.facts.iptables import IptablesRules

facts = host.get_fact(IptablesRules, table="filter")
logger.info(facts) # previously added rules are well collected in the fact

iptables.rule(
    chain="INPUT",
    jump="ACCEPT",
    destination_port=22,
    protocol="TCP",
)
ipt = server.shell(["iptables -S"])


def cb(*args):
    logger.info(ipt.stdout)


python.call(function=cb) # we see duplication of the rule, one more at every run

Expected behavior

do not duplicate rules

Describe the bug

example code for ssh.command is invalid python

ssh.command(
    name="Create file by running echo from host one to host two",
    hostname="two.example.com",
    command="echo "one was here" > /tmp/one.txt",
    user="vagrant",
)

To Reproduce

read https://docs.pyinfra.com/en/2.x/operations/ssh.html#ssh-command

Describe the bug

Please consider the following code snippet. When run multiple times, installing the packages via pip.packages is reported as "changed" every run.

To Reproduce

from pyinfra.api import deploy
from pyinfra.operations import pip

@deploy("Venv")
def venv():
    pip.venv(
        python="python3",
        path="/tmp/clientvenv",
    )

    pip.packages(
        name="Install packages",
        packages=["boto3==1.26.27", "lucidity==1.6.0"],
        virtualenv="/tmp/clientvenv",
    )

venv()

Run via

pyinfra inventory_client.py deploy_client.py

All involved systems are Ubuntu 22.04.

Expected behavior

The packages are found in the given venv and not reported as changed every time. Please tell me I am just not seeing something basic here... I feel dumb, this should be easy :D

Meta

    System: Linux
      Platform: Linux-5.15.0-56-generic-x86_64-with-glibc2.35
      Release: 5.15.0-56-generic
      Machine: x86_64
    pyinfra: v2.5.2
    Executable: /home/grndgdnke/lidl_render_backend/lidl_render_backend_venv/bin/pyinfra
    Python: 3.10.6 (CPython, GCC 11.3.0)

How was pyinfra installed (source/pip)?

venv + pip install

Bug

opened by sebastianelsner 2

Releases(v2.6.1)

v2.6.1(Dec 31, 2022)
Fix reuse of temporary file names in server.script operation (@link2xt)

Fix retry handling on SFTP errors

Source code(tar.gz)
Source code(zip)
v2.6(Dec 19, 2022)
Added:

Use SSH connector parameters with files.rsync operation (@StevenKGER)

Add auto_remove arguments to apt.upgrade operation (@mcataford)

Make it possible to call any function/op/deploy from the CLI

Fixed:

Fix handling of ALL/GRANT OPTION permissions in mysql.privileges operation (@gchazot)

Fix mysql.load operation with spaces in filenames (@gchazot)

Fix fact apk.ApkPackages for packages with numbers in the name (@dchauviere)

Fix fact openrc.OpenrcStatus for services with start times (@dchauviere)

Fix files.put for files containing spaces in local connector (@uggedal)

Fix performance of fact phase when calling functions/operations directly in CLI

Source code(tar.gz)
Source code(zip)
v2.5.3(Nov 9, 2022)
Fix handling of facts with no arguments or with global arguments

Fix mutable default breaking Host.loop position tracking

Cleanup exception handling within operation code

Source code(tar.gz)
Source code(zip)
v2.5.2(Nov 3, 2022)
Fix/make safer check for sysvinit in server.service operation

Fix parsing of sticky/setgid/setuid permission bits in files.* facts

Respect TMPDIR when asking for sudo password (@jaysoffian)

Fix old windows fact names (@simon04)

Fix consistency of facts called in vs. out of operation context

Fix a bunch of pylint issues (@marksmayo)

Fix docstrings on python.* operations

Source code(tar.gz)
Source code(zip)
v2.5.1(Oct 24, 2022)
Fix bug in pre/post condition handling that would print non-fatal errors

Source code(tar.gz)
Source code(zip)
v2.5(Oct 24, 2022)
Added:

Add selinux.boolean, selinux.file_context, selinux.file_context_mapping & selinux.port operations (@morrison12)

Add selinux.SEBoolean, selinux.FileContextMapping, selinux.SEPorts, selinux.SEPort facts (@morrison12)

Add snap.package operation & snap.SnapPackage, snap.SnapPackages facts (@pabloxio)

Add beta files.block operation implementation (@morrison12)

Fixed:

Include all systemd units in systemd.SystemdStatus fact (@mariusmuja)

Handle installed RPM packages in rpm.RpmPackage fact (@mariusmuja)

Fix host loop cycle errors with new host.loop method

Always use ISO format date in server.Date fact, should resolve any outstanding parse errors

Other changes:

Fix a whole load of documentation typos (@simonw)

Generic typing stub for operation decorator (@mariusmuja)

First pass at type annotations for the API (@lowercase00)

Add type checking CI job

Decomposition of many internal API functions & cleanup (@lowercase00)

Use macos-latest GitHub runner (@morrison12)

Fix documentation URL (@blaisep)

Source code(tar.gz)
Source code(zip)
v2.4(Aug 13, 2022)
Delayed getting this out, lots of little improvements.

Added:

Add server.user_authorized_keys operation

Add global _continue_on_error argument

Add dir_mode argument to files.sync operation (@filips123)

Copy local permissions when mode=True in files.put operation

Add headers and insecure arguments to files.download operation

Fixed:

Get facts with host & state context (@jaysoffian)

Fix short facts with arguments (@jaysoffian)

Fix hang on launch of container in lxd.container operation (@zachwaite)

Run operations with host context

Fix idempotency with uploads to a directory in files.put operation

Other changes:

Fix multiple doc typos (@timgates42)

Fix variable typo (@bouke-sf)

Fix CLI shell autocomplete doc (@jaysoffian)

Implement idempotency in git.bare_repo operation

Add typing to fact classes

Start testing files operations with pathlib objects

Source code(tar.gz)
Source code(zip)
v2.3(Jul 16, 2022)
Relatively small quick release with two additions and a bunch of fixes.

Added:

Add create_home argument to server.user operation

Separate no change/change in proposed changes & results output

Support IO-like objects as stdin

Fixed:

Fix short fact gathering

Fix handling of IO-like objects when assume_exists=True in files.put operation

Don't fail to ensure user home dir that already exists as a link

Rename file utils to avoid clashes/confusion with operations

Internal:

Check operation type stubs during CI

Source code(tar.gz)
Source code(zip)
v2.2(Jun 2, 2022)
The main feature of 2.2 is the switch to using a DAG to generate operation order. This mostly replaces line-number ordering (still used to tie-break) and means hacks such as state.preserve_loop_order are no longer required!

The second highlight feature is the inclusion of type stub files for operations that include all of the global arguments. Thank you to @StefanBRas for implementing this.

Other changes:

Use home directory fact for default in server.user operation (@yunzheng)

Fix matching replace as a whole line in files.line operation

Fix bug in mysql.privileges invalid argument requesting MysqlUserGrants fact

Source code(tar.gz)
Source code(zip)
v2.1(May 3, 2022)
First 2.x point release! Major feature: nested operations (at last!).

Based on the changes to operations in 2.x nested operations make it possible to generate & execute operations on the fly at execution time, rather than using the low-level connector API. This unlocks all kinds of complex deploys that were previously impossible or complex to implement. Let's look at an example:

from pyinfra import logger from pyinfra.operations import python, server def callback(): result = server.shell(commands=["echo output"]) logger.info(f"Got result: {result.stdout}") python.call( name="Execute callback function", function=callback, )

Other new stuff:

Add host.reload_fact(...) - bypasses the fact cache to force reloading of fact data

Add deb.DebArch fact

Add ssh_paramiko_connect_kwargs host data used in the @ssh connector

Bugfixes:

Fix: Remove state/host arguments from apt.dist_upgrade operation (@pabloxio)

Fix files.put hashing local file that doesn't exist when assume_exists=True

Fix parsing of link targets in RHEL 6 systems

Prefer zypper over apt when both present in server.packages operation

Internal changes:

Fix license link (@Lab-Brat)

Run black and isort across the codebase, now part of CI

Source code(tar.gz)
Source code(zip)
v2.0.2(Apr 28, 2022)
Fix for config.SUDO/etc handling for operation argument defaults

Source code(tar.gz)
Source code(zip)
v2.0.1(Apr 17, 2022)
Rewrite & fix/speedup systemd facts using systemctl show

Support passing IO-like objects into files.template operation

Support accept-new SSH config for StrictHostKeyChecking

Fix hashing of facts with non-keyword arguments

Fix connect to non-connected hosts before collecting facts

Fix config.REQUIRE_PYINFRA_VERSION & config.REQUIRE_PACKAGES handling

Many little docs improvements

Source code(tar.gz)
Source code(zip)
v2.0(Apr 9, 2022)
pyinfra automates/provisions/manages/deploys infrastructure. It can be used for ad-hoc command execution, service deployment, configuration management and more. See the readme for more information.

The first 2.x release! Like v0 -> v1 this release mostly removes legacy APIs and methods which show warnings in v1. Major changes:

Breaking: Python 2.7 (finally!), 3.5 support dropped, Python 3.6 is now the minimum required version.

Breaking: the "deploy directory" concept has been removed - everything now executes from the current working directory which removes the ambiguous magic v1 used to pick a deploy directory. A new --chdir CLI flag has been added to set the working directory before pyinfra executes.

This may affect scripts or CI workflows currently setup, for example:

# Old v1, deploy directory becomes deploys/elasticsearch/ pyinfra deploys/elasticsearch/inventories/production.py deploys/elasticsearch/deploy.py # New v2, explicit chdir required pyinfra --chdir deploys/elasticsearch/ inventories/production.py deploy.py

Parallel operation generation & facts rewrite - this is a huge improvement to how pyinfra generates commands to run on target hosts. This is now run in parallel across all hosts. Facts are now collected by individual host rather than across all hosts which may yield significant speedups in certain situations.

This change also brings support for all of the execution global arguments to facts, and hugely simplifies the facts implementation. Global arguments will now be read from host data in exactly the same way they are for operations, which was often a confusing gotcha in v1. This also means that the arguments can have different values for each host and this will not cause issues.

Other breaking changes (warnings shown in v1 for most):

Non-existent host data raises an AttributeError when accessed via host.data.X

Change default branch argument to None in git.repo operation

present argument removed from mysql.privileges operation

Config variables must now be set on the global config object

Old style host.fact.fact_name access has been removed

The legacy init.* operations have been removed

Stop lowercasing package names in facts & operations

Remove --facts and --operations CLI flags

Remove --debug-data CLI flag

Remove Windows prefix on all Windows facts

Rename name argument to path in windows_files.* operations

Remove support for jinja2 template parsing in string arguments

Remove old pyinfra.modules module/import

Remove config.MIN_PYINFRA_VERSION

Remove branch and create_branch arguments in git.worktree operation

Remove touch_periodic argument in apt.update operation (never used)

pyinfra.api.connectors module moved to pyinfra.connectors

Deprecated (showing warnings, to be removed in v3):

state and host arguments no longer need to be passed into operation or deploy functions

postgresql_* arguments renamed to psql_* in postgresql.* operations & facts

Source code(tar.gz)
Source code(zip)
v1.7.3(Apr 9, 2022)
Basename files before appending to backup directory

Fix operation ordering when executing multiple files

Source code(tar.gz)
Source code(zip)
v2.0rc1(Mar 30, 2022)

Source code(tar.gz)
Source code(zip)
v1.7.2(Mar 27, 2022)
Revert legend flag change in Systemd* facts

Source code(tar.gz)
Source code(zip)
v1.7.1(Mar 26, 2022)
Escape exported env variables (@lun-4)

Expand units Systemd* facts pick up (@jmpolom)

Deprecate match, replace with text in files.replace operation

Show warnings for removal of present argument in mysql.privileges operation in v2

Import top level os to avoid confusion in files operation module

Source code(tar.gz)
Source code(zip)
v1.7(Mar 6, 2022)
Add --user and --machine flags to systemd.* operations (@jmpolom)

Fix treat Zypper package names as case-sensitive (@sysadmin75)

Fix CLI flags with click<7.0 (@gchazot)

Fix automatic sudo check when get_pty=True

Fix respect config.TEMP_DIR setting for sudo file up/downloads

Source code(tar.gz)
Source code(zip)
v1.6.3(Feb 19, 2022)
Enable passing success_exit_codes into host.get_fact calls (@mohsinhijazee)

Fix bug triggered when no SSH config file exists

Source code(tar.gz)
Source code(zip)
v1.6.2(Feb 18, 2022)
SSH connector config fixes:

Implement & fix StrictHostKeyChecking matching how OpenSSH works (@bauen1)

Fix / use SSH config for ProxyJump hosts (@artizirk)

Add ways to disable SSH agent / looking for SSH keys (host.data.ssh_allow_agent & host.data.ssh_look_for_keys)

Fix Match exec SSH config

Operation/fact fixes:

Set LC_TIME in server.Date fact

Use a temporary file + move in files.download operation

Misc:

Workaround fix for host.data.env not being a dictionary

Workaround fix for host.data.use_sudo_password being different for each host

Fix bash autocomplete documentation (@oz123)

Fix incorrect handling of internal errors when hosts fail

Refreshed logo/readme/docs/website for upcoming v2

Source code(tar.gz)
Source code(zip)
v1.6.1(Jan 8, 2022)
Fix typo bug in the SSH client preventing connection

Source code(tar.gz)
Source code(zip)
v1.6(Jan 8, 2022)
A small release with a few features and fixes. Future development will utilise smaller, more regular releases to avoid long gaps waiting on feature X to be complete.

This release also changes the git branching model to use current for anything non-breaking, and the next branch now targets the next major version. This should avoid unncessary confusion for the wonderful contributors to pyinfra.

Operations:

Make it possible to load relative (to CWD) templates (includes/bases/etc) (@themanifold)

Add apt.dist_update operation (@bauen1)

Fix files.Link fact & files.link operation where path pre-exists and is not a link

Connectors:

Support setting shell_executable=None to pass raw commands to the target (@KuxaBeast)

Fix handling of SSH proxy jump config

Only load SSH host keys once

Make it possible to specify a SSH config file via host.data.ssh_config_file

Source code(tar.gz)
Source code(zip)
v1.5(Nov 14, 2021)
This release includes a huge number of improvements, major highlights:

Automatic sudo password prompting - at last! pyinfra can now detect when a sudo password is required and will prompt the user.

Global config object - there is now a pyinfra.config object that should be used to set config variables:

from pyinfra import config config.SUDO = True

This replaces the old style of setting SUDO = True at the top of files which was extracted via the AST. This new object means config variables can be set to anything (including the results of functions) and work across multiple deploy files.

Operation & fact updates:

Add pkgin.packages & pkgin.upgrade operations (@lun-4)

Add pkgin.PkginPackages fact (@lun-4)

Add selinux.FileContext & selinux.SEBoolean facts (@benridley)

Add openrc.service operation

Add openrc.OpenrcStatus & openrc.OpenrcEnabled facts

Add last login time to server.Users fact (@sysadmin75)

Fetch held deb package versions in deb.DebPackages fact (@GerardoGR)

Add force, force_backup and force_backup_dir arguments to files.files, files.directory & files.link operations

Add resource and SSL arguments to mysql.user operation

Add with_grant_option argument to mysql.privileges operation

Add available argument to apk.packages operation (@lun-4)

Connector updates:

Transfer ownership of files uploaded with su/sudo (@benridley)

Add ssh_forward_agent group data variable overriding SSH config

Other bits:

Add doas & doas_user global arugments

Use host data for global arguments (host.data.sudo, etc)

Add --data key=value CLI flag

Propagate return value from @deploy decorator (@karlicoss)

Large expansion of tested idempotent operations

Loosen distro requirement

Source code(tar.gz)
Source code(zip)
v1.4.19(Nov 13, 2021)
Use uname -m in server.Arch fact (@weakish)

Fix user check in server.user operation (@glassbeads)

Fix setting user primary group in server.user operation (@glassbeads)

Always add CWD to sys.path (@glassbeads)

Fix issues with regex parsing for deb.DebPackage & deb.DebPackages facts

Fix parsing apt repository lines with [] characters

Source code(tar.gz)
Source code(zip)
v1.4.18(Oct 30, 2021)
Fix exception collecting Docker.* facts on objects that don't exist

Support variable interpolation in files.FindInFile fact

Add debug-inventory example to the help doc

Show a warning when a --limit option doesn't match any hosts

Warn only if SSH keys fail to load

Include actual error when we encounter authentication errors

Source code(tar.gz)
Source code(zip)
v1.4.17(Oct 15, 2021)
Fix excluding multiple directories in files.sync operation (@gchazot)

Cleanup filesystem mocking in operation tests (@gchazot)

Fix bug in command generation when config sudo user is set

Stop removing sudo/su arguments in the Docker connector

Source code(tar.gz)
Source code(zip)
v1.4.16(Oct 3, 2021)
Fix loading of system SSH host keys

Fix/show warning when no SSH host key present on connect

Source code(tar.gz)
Source code(zip)
v1.4.15(Oct 3, 2021)
Don't fail if dest in files.put is a directory (@oz123)

Improve internal unix path joining

Source code(tar.gz)
Source code(zip)
v1.4.14(Sep 2, 2021)
Add quote_path=True argument to files.[FindFiles|FindLinks|FindDirectories] facts

Fix enable/disable service in sysvinit.service operation using un-quoted path

Fix path join handling in certain cases in files.sync operation

Always normalise paths in files.[file|link|directory] operations

Source code(tar.gz)
Source code(zip)
v1.4.13(Aug 31, 2021)
Fix exclude_dir in files.sync operation (@gchazot)

Fix multiple nested imports by properly resetting current exec filename

Improve error when there are no Vagrant instances and a name is provided

Source code(tar.gz)
Source code(zip)