Maintainer edits

Overall TODO generated by huge arse discussion thread.

• [x] Add new concepts doc on configuration, including a clearly marked hierarchy definition, and a big WARNING about how multiple files in same location w/ diff extensions are not merged.
• [x] Make sure to clarify .yaml, not .yml.
• [x] ~~Probably also needs an update to the CLI docs section re: flags and env vars (new subsection)~~ punting on user-driven config updates that aren't env vars; env vars have their own section in config so no need to duplicate
• [x] ~~Re: env vars, need to figure out best way for user-driven config settings to work too, eventually~~ punting
• [x] ~~Brief mention of concepts doc in the tutorial~~ Actually don't see an elegant way to work it in - feels best just as a standalone regular concepts doc for now
• [x] Make sure API in docs gets updated to be consistent & 'modern' - e.g. obj['foo.bar'] should turn into obj['foo']['bar'] or ideally with the option of obj.foo.bar.
• [x] Update concepts/contexts to adapt, possibly paring it way down or removing entirely, tl;dr it's just about explaining how contexts perform shared state.
  • The 'nested collection configs get merged' explanation could still live here but probably lives better in the new config page, or the namespacing page if it exists.
• [x] Vendorize etcaetera
• [x] Use its Config object in place of Context's existing inner dicts, get tests passing again referencing it (it's dict-like).
  • Decide whether the current run/general/etc splits become sub-config objects or just nested namespaces within the primary config object
  • Leaning towards the latter so people can override them everywhere
• [x] Add new CLI flags for setting config options (primarily -f or whatnot for "specific config file")
• [x] Implement hierarchy re: multiple conf files + env vars + ~~cli flags~~ + in-module configs
• [x] Conf file loading of different extensions at the same 'location' is exclusive - no merging of eg /etc/invoke.yaml and /etc/invoke.json.
• [x] Debugging of conf option overrides must be easy
  • either via a no-op CLI flag showing what the tree would be for a given task module (or task invocation?), or
  • via regular debug output (ideally + a no-op option)
  • really, those go to the same place
• [x] If not covered in the previous point: it must be easy to troubleshoot stupid typos in config file paths.
  • e.g. if you accidentally created a /etc/invok.yaml, or fatfingered the arg to invoke -f <path> - it should be clear when running with -d which config files it attempted to load and which ones failed to load.
  • Main problem is that for the first 3-4 levels we have 3x file paths we end up searching for, so it will be somewhat verbose :( but no obvious solution offhand, better to add it to debug output (or some other level?) than try to be clever.
• [x] Update invocations to use this new setup instead of dotted names - they still technically work but are invalid for eg env var translation.
• [ ] Context objects (& subclasses like Fabric2's Connection) need a clean API for this configuration stuff - either Context directly grows all the config related methods (perhaps becoming a etcaetera.Config subclass itself?) or it accepts some config object (again perhaps etc.Config) as an instance member.
  • The current driving need here is actually testing related, I need to inject a config value override or two in Connection tests.
• [ ] Fill in any remaining skipped tests related to this feature set.

Original ticket description from sophacles

It would be nice to be able to pass context variables on the command line to invoke scripts. Something like:

> inv -X var1=val1 -Xvar2=val2 target --target1opt=val

And perhaps even allow targeting variables to specific namespaces, ala:

inv -X ns1.var1=val1 -Xvar2=val2 ns1.target  target2

for those cases where target2 should use the default for var1, but ns.target1 needs the overridden value.

There are of course concerns about how command line set options override things in a call to Collection.configure... take for instance this script:

@ctask
def task1(ctx):
      run("echo {val1}".format(**ctx.config))

@ctask
def task2(ctx):
    run("echo {var1} and {var2}".format(**ctx.config))

ns1 = Collection(task1)
ns1.configure({"var1":"ns1var1"})
ns = Collection(ns, task2)
ns.configure({"var1":"globalvar1","var2":"globalvar2"})

Which is the right output:

option 1:

# inv -X ns1.var1=clival1 -X var2=clivar2 ns1.task1 task2
clival1
globalvar1 and clivar2

(note - because generally it's considered cli options override anything in-program)

option 2:

# inv -X ns1.var1=clival1 -X var2=clivar2 ns1.task1 task2
globalvar1
globalvar1 and clivar2

(note because of the way overrides work in invoke already)

I think everyone would agree that:

# inv -X var1=clival1 ns1.task task.2
clival1
clival1 and globalvar2

is correct.

In Windows, command output is not displayed in terminal, even with the simple task below.

@task
def test():
    """output test"""
    run('echo hello invoke')

The --echo command line option also doesn't work in Windows. Things are fine in Linux though.

Windows 7 32bit, Python 2.7.9, Invoke 0.9.0

Defining a task that uses the new Python3 keyword-only or annotation argument syntax and invoking causes a ValueError: Function has keyword-only arguments or annotations, use getfullargspec() API which can support them and halts.

Per the docstring for Python3 inspect, getfullargspec() could be used, but that function is itself deprecated in favor of inspect.signature() Unfortunately, neither of those functions exists in Python 2.6, so some level of cascading will be necessary.

For example, neither of these can be invoked:

from invoke import task

@task
def func(ctx, *args, keyonlyarg=42):
    print(keyonlyarg)

@task
def func2(ctx, x:int=3):
    print(x)

Unicode handling is nasty, it is quite hard to make it right, but this PR makes Invoke reliable in non-ascii world.

Old Description

I've had a session with POSIX locale on Linux and a program that I've run pushed non-ASCII (UTF-8 in my case) output anyway. This revealed a bug when user-requested/locale.getpreferredencoding() is UTF-8, but sys.stdout is ASCII.

See also: Fabric #69

Problem

Need to allow parameterization of tasks as invoked via the CLI (tasks invoked via Python are simply passing regular args/kwargs or their list/dict counterparts.) Sub-problem: translating strings/flag existence into Python types such as lists, booleans etc.

Current implementation in Fabric is fab taskname:positional_arg_value,keyword_arg=kwarg_value,.... Some other tools use "regular" CLI flags, e.g. paver taskname -F value.

"Custom"/"attached" args

Pluses:

• Works great for multi-task invokes, zero ambiguity about which param values go to which tasks
• Supports positional arguments well

Minuses:

• Requires custom parsing, which may lead to bugs and at the very least increases code maintenance burden
• Slight documentation burden -- requires users to learn how that custom parsing works, re: need for escaping (esp. how that meshes with normal shell escapes) and etc.

"Regular" flag args

Pluses:

• Can (probably) just use argparse
• Many users already familiar with it

Minuses:

• Doesn't mesh well with multi-task invokes unless we:
  • enforce BSD-style "order matters" (not even sure argparse can do this? EDIT: perhaps with subcommands, if >1 subcommand can be invoked at a time)
  • or use an ugly "task delimiter" syntax like invoke task1 --arg1 --next task2 --arg2 (again, can argparse do this?)
  • or hope/pray that all simultaneously-invoked tasks have zero overlap in their kwargs and no positional args are ever needed.
    • How common is this? If we assume flags nix any possibility of positional args and that this is OK, it could be reasonable to assume two or three tasks called together would all have different names for their kwargs. Maybe.
• Namespace clashes with core (non task related) flags such as -c/--collection
  • This also requires order-matters, or simply limiting the user-allowed namespace
  • Limiting the namespace also presents backwards compatibility problems -- if version 1.1 lacks a --foo flag and a user declares mytask(foo='bar'), and then we unknowingly add --foo to core in 1.2, that user's code will break.
• May require more setup on the user side, re: explicitly declaring each task's args/kwargs via decorators or whatnot
  • Though we could introspect the function signature and automatically create some, if needed
• Doesn't allow for positional arguments unless we go the delimiter route.

Potential solutions

• Continue using Fabric's "custom parsing" solution pretty much as-is
  • See above re: pluses/minuses
• Try coming up with a "better" custom parsing solution/syntax, if there are any
  • What do other tools like Make, Rake, Thor etc use?
    • Thor uses regular flags + positional arguments, seems to simply ignore the "multiple tasks vs one task + positional args" problem, and uses "pass all flags to all tasks" to handle multi-task invocation.
      • Don't like this very much -- it "works" but has the "all tasks invoked must have different args" requirement mentioned above.
    • make uses shell environment variables, which is similar to how Thor treats flags -- every task is going to get the same values for any overlapping names.
    • Rake uses custom parsing like we do, but uses taskname[args, here] instead of taskname:args,here. I find this actually worse than the colon we use, because some shells like zsh treat square-brackets specially and need extra escaping/quoting.
• Allow custom parsing everywhere, but also allow use of flag args in single-task invocations
  • Plus: gives flag lovers what they want
  • Minus: "more than one way to do it" and its problems (newbie confusion, more code/bugs, more docs to write.)
• Use flags 100% of the time, relying on ordering or delimiting flags to handle multi-task invocations
  • See above re: pluses/minuses
  • Possibly use argparse subcommands, but AFAIK those are again targeted at using a single subcommand per invoke.

I.e. instead of just inv --list spitting out the current default flat list, allow things like inv --list nested for a nested/indented display, inv --list=foo to just list the tasks within namespace foo, etc ad infinitum.

LATER EDITS: so a lot of strongly related tickets have popped up and there seem to be two major currents:

• Formatting, i.e. flat with dots vs nested vs machine-readable
• Content, i.e. all tasks vs just namespaces vs just tasks within a given namespace

Seems unwise/impossible to shoehorn all of that under a single optional value to --list so I'm sure we'll grow some additional flags either way, with the main question being which (if any) such option deserves to live as the optional --listvalue.

Brainstorm:

• Format could be a single flag with string option, e.g. --list-format (flat|nested|machine) (what term do other tools use for "machine-readable" anyways? Can't remember offhand...LATER EDIT: actually these days we often just use JSON for that and just call it eg --list-format=json) or it could be a handful of booleans like --list-nested, --list-machine.
  • Former feels easier to implement, harder to fuck up by giving >1 at a time, and easier for users to grasp (i.e. fewer options to think about) so it seems like a no-brainer.
• Content needs levers for 'root' (though this needs to not clash with the related-but-distinct --root flag which changes where tasks files are sought from; but perhaps we should be, well, namespacing those as well so they are renamed to eg --discovery-root) and 'depth` and I think that's it, since those together can enable just about anything:
  • "Show me everything": root is the top of the namespace, depth is None
  • "Show me just the namespaces": root is top of namespace, depth is 1
  • "Show me just tasks under namespace foo"
• Might actually need another lever for the confluence of those two, i.e. "I am listing some subset of tasks and I want to tweak whether they are shown as full paths or sub-paths"?
  • Though at this point I wonder if users who want a non-full-path display (i.e. inv --namespace foo --list spitting out task names like bar and baz instead of foo.bar and foo.baz) should be relying on "each Python module can be its own standalone namespace" and doing things like inv --collection=namespace --list" instead (see Collection loading.)
    • Since, if one truly wants to 'focus' on a given sub-namespace, giving it to --collection automatically ensures that everything - display, invocation, etc - instantly works as you'd expect.
  • The main argument I can think of why one would not want to go that route is if one wanted to do nontrivial slice n dice of namespace contents within the root tasks file (such that the namespace does not reflect Python module or package structure.)
  • However it's not necessarily out of the running, we could either implement things to be depth-aware and default to stripping out parent levels on display/invocation – or (probably better) just make it possible to say "hey I know you loaded my root tasks namespace, but I'd like you to act as if I gave you foo.bar as an argument to --collection - even though, between you and me, foo.bar isn't visible on the Python import paths."
  • Rereading this later, my gut says "ugh we're making this too complex, just add another flag if people REALLY want to see non-full paths for some stupid reason". And besides, what happened to the 'nested' idea? That's another way to show the 'short' names while clearly presenting depth/parentage.
• Minor quirk: do we bother showing the root namespace when we are being asked to "show me the namespaces"? I'm leaning towards no, but might be nice to be explicit...? e.g. "your namespaces are: (root), foo and bar"

Background

This is a follow-on from #289 and also encompasses some or all of #241, #274, #262, #332, #242, #321, and #338. My goal is to make this general story better, ASAP, while leaving remaining work clearly defined (regardless of who ends up taking a swing at it).

I need to doublecheck some of those linked issues that I've not spent a lot of time on, but the bulk of discussion is in #289 and #274, much of it driven by @pfmoore who (as a Python 3 on Windows user) serves as a very competent & useful counterweight to myself (a default-Python-2 on Unix developer). Any solution that works for both of us is likely one that serves most users.

The basics that fall out of what's been said there so far:

• Where & how bytes/strings are decoded/encoded, needs to be better defined/documented/contracted;
• Encoding/decoding should be pushed as far out as possible, to the "edges" of the code, to limit how often it happens & becomes an issue;
  • But also centralized further - the code handling it needs to be clearly labeled & no faffing about should happen outside that one area.
• The default behavior should encompass as many use cases as is reasonable to do, and should degrade as gracefully as it can otherwise (re: exceptions, vs replacements, vs ???);
  • But users should also be given the ability to work around that default cleanly & fully whenever it doesn't function for their use case.
  • Ideally, we'll balance these two things well instead of using "the user can twiddle the knobs we provided" as an excuse to let the default behavior suffer. But this is hard and time is short, so...

TODO

• [x] Triple check existing locations where we explicitly or implicitly encode/decode, and move them all somewhere centralized & obvious (without otherwise altering them).
  • Right now the only explicit mentions of (encode|decode)\( are a bare handful in runner.py, so there's likely others that are implicit instead.
  • Not sure what the "most obvious" place is, but given most of this is directly tied to run() and friends, it may still just live in the Runner class or runner.py generally.
• [x] Identify & implement additional tests (integration or regular) for use cases not represented by the funky_characters_in_stdout integration tests
  • Those tests should have caught some of the basic e.g. cat tree -> kaboom bugs that snuck in during 0.11.x, but did not.
  • I identified the basic reason why & have updated it so they fail again, in a branch.
  • But they only represent a single slice of the problem - look at the discussion in #274, as well as the other top level issues above, and see what other tests can be added that currently fail.
  • Ideally at least some of these would go in the regular, not integration, suite.
• [x] Get those tests passing, either based off of #274 or with something new
• [x] If not implied in above: probably makes sense to add more encoding config toggles (maybe up to "one for each stream") so users can override them granularly; the single encoding is nice and all, but won't work in cases where e.g. one's invoke-invoking terminal is Encoding A but the environment the subprocess is operating in is using Encoding B.
  • Made this #353
• [x] Handle, or surface into this or other tickets, any TODOs added in the branch left undone
• [ ] See if @pfmoore has time for immediate feedback to those changes (sadly I wasn't able to give him a thumbs up for his offer of making more PRs on his end, back in Feb) just in case.
• [x] Re: comments below, update read_proc_output so we always expect bytes:
  • [x] Enhance docstring(s) to make it clear that any file-like objects given explicitly, need to yield bytes
  • [x] Remove the "if" statement
  • [x] Update our test suites to use BytesIO instead of StringIO (& any related string literal changes, or maybe a wrapper if lazy).
• [x] Update changelog entry for any fully-closed tickets & their authors

Interrupting a running task with Ctrl+C results in this traceback:

^CTraceback (most recent call last):
  File "/Users/peter/Envs/are/bin/inv", line 9, in <module>
    load_entry_point('invoke==0.8.2', 'console_scripts', 'inv')()
  File "/Users/peter/Envs/are/lib/python2.7/site-packages/invoke/cli.py", line 295, in main
    dispatch(sys.argv)
  File "/Users/peter/Envs/are/lib/python2.7/site-packages/invoke/cli.py", line 288, in dispatch
    return executor.execute(*tasks, dedupe=dedupe)
  File "/Users/peter/Envs/are/lib/python2.7/site-packages/invoke/executor.py", line 89, in execute
    task=task, name=name, args=args, kwargs=kwargs
  File "/Users/peter/Envs/are/lib/python2.7/site-packages/invoke/executor.py", line 128, in _execute
    return task(*args, **kwargs)
  File "/Users/peter/Envs/are/lib/python2.7/site-packages/invoke/tasks.py", line 108, in __call__
    result = self.body(*args, **kwargs)
  File "/Users/peter/Code/are-platform/API/tasks.py", line 56, in start
    run_env('python are/servers/dev_server.py', shell_env, verbose)
  File "/Users/peter/Code/are-platform/API/tasks.py", line 27, in run_env
    run('%s %s' % (env_vars, cmd))
  File "/Users/peter/Envs/are/lib/python2.7/site-packages/invoke/runner.py", line 143, in run
    stdout, stderr, exited, exception = func(command, warn, hide)
  File "/Users/peter/Envs/are/lib/python2.7/site-packages/invoke/runner.py", line 65, in run
    stdout, stderr = process.communicate()
  File "/usr/local/Cellar/python/2.7.6/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 798, in communicate
    return self._communicate(input)
  File "/Users/peter/Envs/are/lib/python2.7/site-packages/invoke/monkey.py", line 58, in _communicate
    rlist, wlist, xlist = select.select(read_set, write_set, [])
KeyboardInterrupt

I'm migrating from Fabric which handles SIGINT nicely by exiting with sys.exit(1) and printing Stopped.. It also properly forwards the interrupt to any processes started with fabric.api.local(). How can I get this behaviour in Invoke?

Willing to contribute a fix for this if I'm pointed in the right direction!

This represents porting Fabric 1.x's limited ability to autorespond to sudo prompts, to Invoke, and making it Better™.

For now, I've gone with a basic "when you see X, always respond with Y" functionality that can be easily configured via a {pattern: response} dict.

This isn't the most powerful implementation; e.g. one might want "only respond to patterns once", "respond to X pattern the first 3 times only", "respond to pattern X but only if you've already responded to pattern Y", etc etc.

However it's arguably more flexible than using e.g. pexpect (which requires you to anticipate exactly what prompts will happen, in what order; Invoke/Fabric use cases tend to have prompts appear arbitrarily) and I expect it will be enhanced in the future, perhaps by ripping out the responder aspects into their own class or something.

The functionality is already pretty much complete after a ~week of hacking, fucking up, getting confused, becoming less confused, and more hacking. I'm making this ticket just so I have something to link in the changelog & a checklist for the random minor tasks left to perform.

Left to do:

• [x] Deal with recently-discovered funkiness in interactive stdin use case
• [x] Make sure the basic functionality works for Fabric 2 as well as with local/subprocess
• [x] Reinstate the Windows support for handle_input, which mirrors real stdin to the subprocess' stdin.
• [x] Doublecheck whether we actually need Windows getch as per https://github.com/pyinvoke/invoke/pull/304#issuecomment-174005880
• [x] Figure out whether to retain the sleep call in handle_input; git-blame the Fab 1 line and doublecheck the rationale. Probably do want it back tho, probably chews more CPU than it really needs to as-is.
• [x] Reinstate the "print read stdin data to stdout when pty=False" behavior from fab 1 - it is necessary
• [x] Ditto the "halt stdin mirroring while the local codebase is explicitly trying to prompt for shit it needs" functionality from Fab 1.
  • ...does this mean Fab 1 literally can't deal with users manually executing things like raw_input and getpass? Cuz if it can then that implies said halting/locking might not have been necessary? Or is a corner case?
  • See notes in desc of #294 - probably not going to go down the same road as in Fab 1 which means we wouldn't have to worry about this wrinkle.
• [x] Centralize & normalize the encoding related noise, as per comments by @pfmoore in comments below.
  • Quite possibly via work/discussion being done over on #274 and its linked tickets
• [ ] Try to replicate fabric/fabric#1339 (both in Fabric 1, and here, and in Fabric 2)
• [x] Figure out if it's worth implementing a(n easily disabled and well documented!) convenience "add \n to response values if not present" feature.
  • Ye olde "helpfulness can be evil"...
  • but in MY OWN TESTING I've fucked up and assumed an implicit \n multiple times already, so there's no way tons of users wouldn't make the same assumption.
  • Spun off as #368
• [x] Determine what's up with swallowed exceptions in the out/err/in threads when the main thread gets an eg KeyboardException. Feels like a thing we need to at least partially port from fab v1.
• [x] Python 3 test pass so Travis doesn't get mad at me
• [x] Ascertain why the test suite got slower again and how much of it's due to this branch
• [x] Double-check the doc situation; may want to make a "feature list!" section on index page?
  • #394
• [x] Changelog entry

Description

Splitting this ticket from #170 at the request of @bitprophet

It would be nice to easily create scripts that I can place in my $PATH that internally use invoke, but are named according to script functionality. Some rational (copied from referenced ticket discussion):

I could probably train myself always to do:

inv -c /path/to/source/tree/utils/$TASK ....

But I kind of like having them:

1. as a script available in my $PATH
2. as tools named cognitively, just like when I'm using busybox... i just link (e.g) ls and cp rather than calling busybox ls ... or busybox cp file1 file2
3. in a consistent place (depending on what I'm doing, the source tree where the task files live can vary. It's in one place on my dev machine, and different places in deployment depending on what OS, nfs details, and so on. We're still working out our "one true deployment strategy" so this changes as stuff comes up).

Also noteworthy, several of my targets are most commonly called from places in the system that are not the directory where invoke lives.

Currently I do this via the code in this gist: https://gist.github.com/sophacles/c17ce33a14d582dfa268 and creating scripts entry points in a setup.py.

To-do list

[Note: this part of the ticket is being maintained by @bitprophet]

• [x] Parameterization of version number by itself (not name+version as we currently do)
• [x] Parameterization of tool/package name, defaulting to the name seen in argv[0]
  • [x] Capitalized in --version output, e.g. Invoke 1.2
  • [x] Allowance for arbitrary additional lines of version info, e.g. for Fabric wanting to display its + Paramiko's, and even e.g. other data like Python version?
  • [x] Left as-is in --help output, e.g. inv[oke] --core-opts ...
  • [x] Is there an automated way to determine inv vs invoke vs etc? We don't have access to setuptools' data at runtime, so we'll probably have to provide this manually. (Maybe stored in similar fashion as version - reused in both spots?)
• [x] Ability to update global --help to include elements of --list, i.e. for tools distributing with built-in tasks as subcommands (which would be...most of them...Fabric is actually unusual in this regard).
  • I.e. probably sticking --list type output above the core flags, which is what most real subcommand based CLI apps tend to do (see git, pip etc)
• [x] Fix up per-task help to honor the dynamic program name (it's hardcoded right now, as above)
• [x] Ability to override which flags are presented and/or their default values

Cleanup:

• [x] Check all the task modules in _support to remove any no longer in use (& consolidate some if possible)
• [x] Check all methods in _utils.py for whether they've gone out of use
• [x] Ensure docs build & the docstrings recently added look good rendered

I.e.:

@task
def pretask():
    pass

@task(pre='pretask')
def realtask(arg):
    pass

This will fail because at the bottom of Executor.execute() we build a task list and then give all of them the CLI arguments. I.e. pre/post tasks are given the same input as the "main" task.

The API we would need to fix this is probably:

@task
def pretask():
    pass

@task(pre={'pretask': []})
def realtask(arg):
    pass

Or, with a pretask that has an interesting signature:

@task
def pretask(myarg):
    pass

@task(pre={'pretask': ['foo']})
def realtask(arg):
    pass

Would be equivalent to calling pretask('foo') prior to calling realtask().

Allowing a dict for keyword args would also make sense; however the full implementation would want both an args list and a kwargs dict, which would then both need wrapping in a tuple or list. That gets pretty fugly pretty fast:

@task(pre={'pretask': ([], {'kwarg': 'value'}})

Another possibility (this is actually what nate-pycon on IRC originally thought of) is to reuse the parser and say e.g.:

@task(pre=['pretask --kwarg=value'])

However I think even the most complex pure-Python version above is preferable, because:

• it's not sufficient to be the ONLY way of doing this, because forcing a serialize/deserialize step when pure Python is much faster/easier;
• if we make it just another alternative, that's almost definitely crossing my "too many ways to do it -> confusing and overly complex code" threshold

Fixes #486

The process stays alive after the sudo fail attempt (because it'll ask for the password again, so the process will wait for another stdin), so we must terminate it. Any subsequent use of subprocess.PIPE for stdin will be blocked otherwise.

Hi, a usage question as the docs are quite lite on this topic.

If I want to make 5 context.run() calls concurrently but I don't want to complete execution of my function until they have all finished, is there something more elegant than this:

class AggregateError(Exception): 
    pass

running = [context.run(foo, async=True), context.run(bar, async=True), context.run(baz, async=True)]

while running:
    for promise, index in enumerate(running[:]):
        ....

We're using invoke internally and it seems to be the last dependency incompatible with Python 3.11.

Error we're getting is pasted below, probably not the last one.

Traceback (most recent call last):
  File ".venv/bin/invoke", line 8, in <module>
    sys.exit(program.run())
             ^^^^^^^^^^^^^
  File ".venv/lib/python3.11/site-packages/invoke/program.py", line 373, in run
    self.parse_collection()
  File ".venv/lib/python3.11/site-packages/invoke/program.py", line 465, in parse_collection
    self.load_collection()
  File ".venv/lib/python3.11/site-packages/invoke/program.py", line 699, in load_collection
    module, parent = loader.load(coll_name)
                     ^^^^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.11/site-packages/invoke/loader.py", line 76, in load
    module = imp.load_module(name, fd, path, desc)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/imp.py", line 235, in load_module
    return load_source(name, filename, file)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/imp.py", line 172, in load_source
    module = _load(spec)
             ^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 721, in _load
  File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 940, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "src/tasks.py", line 29, in <module>
    @task(name="migrate")
     ^^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.11/site-packages/invoke/tasks.py", line 357, in inner
    obj = klass(
          ^^^^^^
  File ".venv/lib/python3.11/site-packages/invoke/tasks.py", line 76, in __init__
    self.positional = self.fill_implicit_positionals(positional)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.11/site-packages/invoke/tasks.py", line 167, in fill_implicit_positionals
    args, spec_dict = self.argspec(self.body)
                      ^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.11/site-packages/invoke/tasks.py", line 153, in argspec
    spec = inspect.getargspec(func)
           ^^^^^^^^^^^^^^^^^^
AttributeError: module 'inspect' has no attribute 'getargspec'. Did you mean: 'getargs'?

I would like to add python interpreter options, such as -OO during the invoke call. However, I did not find a way other then patch the invoke (and inv) scripts in venv manually. Am I missing something?

Currently, _task_with_merged_config returns a single level merged dict. Howerver, if the config contains nested dicts, only the first level gets merged, removing eventual fields non defined in origin config.

This aims to recursively merge config when retrieving the task config, and to unify how collection configs are merged.