See https://github.com/pantsbuild/pants/pull/13962 for an example and https://github.com/pantsbuild/pants/pull/13963 for prework. Blocked on https://github.com/pantsbuild/pex/issues/1402.

We have to figure out if we want to keep Poetry for now for generating tool lockfiles, but deprecated. Poetry is broken in that [python-repos] does not work, and we also leak the cache globally. So there's motivation to eagerly switch. But, the resolver will change which could break some users, and the lockfile contents have a different format (new lockfile is more useful).

User lockfiles should always use Pex because that's a new feature. It's only tool lockfiles where we need to worry about backwards compatibility.

pants fmt failure on master as of master plus just some typing changes in https://github.com/pantsbuild/pants/pull/10647 : env: python3.7: No such file or directory

OS is macOS 10.15.6. which python3 reveals interpreter in a venv that is v3.8.5.

$ ./pants fmt src/python/pants::
   Compiling engine v0.0.1 (XXX)
    Finished release [optimized + debuginfo] target(s) in 2m 49s
12:11:49 [INFO] initializing pantsd...
12:12:00 [INFO] pantsd initialized.
12:12:08.46 [INFO] Completed: Building black.pex with 2 requirements: black==19.10b0, setuptools
12:12:08 [WARN] /Users/tdyas/TC/pants/src/python/pants/base/exception_sink.py:359: DeprecationWarning: PY_SSIZE_T_CLEAN will be required for '#' formats
  process_title=setproctitle.getproctitle(),

12:12:08 [ERROR] 1 Exception encountered:

Engine traceback:
  in select
  in `fmt` goal
  in pants.backend.python.lint.python_fmt.format_python_target
  in Format with Black
  in pants.engine.process.fallible_to_exec_result_or_raise
Traceback (most recent call last):
  File "XXX/pants/src/python/pants/engine/process.py", line 229, in fallible_to_exec_result_or_raise
    raise ProcessExecutionFailure(
pants.engine.process.ProcessExecutionFailure: Process 'Run Black on 413 files.' failed with exit code 127.
stdout:

stderr:
env: python3.7: No such file or directory

Traceback (most recent call last):
  File "XXX/pants/src/python/pants/bin/local_pants_runner.py", line 255, in run
    engine_result = self._run_v2()
  File "XXX/pants/src/python/pants/bin/local_pants_runner.py", line 166, in _run_v2
    return self._maybe_run_v2_body(goals, poll=False)
  File "XXX/pants/src/python/pants/bin/local_pants_runner.py", line 183, in _maybe_run_v2_body
    return self.graph_session.run_goal_rules(
  File "XXX/pants/src/python/pants/init/engine_initializer.py", line 117, in run_goal_rules
    exit_code = self.scheduler_session.run_goal_rule(
  File "XXX/pants/src/python/pants/engine/internals/scheduler.py", line 552, in run_goal_rule
    self._raise_on_error([t for _, t in throws])
  File "XXX/pants/src/python/pants/engine/internals/scheduler.py", line 511, in _raise_on_error
    raise ExecutionError(
pants.engine.internals.scheduler.ExecutionError: 1 Exception encountered:

Engine traceback:
  in select
  in `fmt` goal
  in pants.backend.python.lint.python_fmt.format_python_target
  in Format with Black
  in pants.engine.process.fallible_to_exec_result_or_raise
Traceback (most recent call last):
  File "XXX/pants/src/python/pants/engine/process.py", line 229, in fallible_to_exec_result_or_raise
    raise ProcessExecutionFailure(
pants.engine.process.ProcessExecutionFailure: Process 'Run Black on 413 files.' failed with exit code 127.
stdout:

stderr:
env: python3.7: No such file or directory

If I compile twice the same target, first time it is a success, second time it fails as if the 3rd party dependency is not in the classpath anymore.

I've set up a minimal example here. The steps to reproduce the error is to do ./pants compile test-app/src/main/scala twice.

The coursier jar used is created with this script .

@cosmicexplorer reproduced the error using pants version 1.14.0.dev0 even when he removed the overrides to coursier

Protobuf code generation is currently failing when run in remote execution against Buildfarm even after the fix in https://github.com/pantsbuild/pants/pull/13077 for archive extraction in remote execution.

Using the example-python repository for the test, the error that occurs when running ./pants test helloworld:: (with appropriate options to enable remote execution) is:

Exception message: 1 Exception encountered:

  ProcessExecutionFailure: Process 'Generating Python sources from helloworld/util/proto/config.proto.' failed with exit code 1.
stdout:

stderr:
/tmp/worker/remote-execution/operations/28150c45-177d-4894-9fcb-7b88fc57f9c7/.cache/pex_root/venvs/8ccfdd776b06121b883ca8d3331e429b87fe042d/ddab8011daaee380698ac2fb9701af18c90c03f6/bin/protoc-gen-mypy: 2: exec: /tmp/worker/remote-execution/operations/c79cf097-80ab-4212-b27f-b999f573cfc7/.cache/pex_root/venvs/short/02a3c5ef/bin/python3.8: not found
--mypy_out: protoc-gen-mypy: Plugin failed with status code 127.

Note that the path to protoc-gen-mypy and the path to python3.8 are different: 28150c45-177d-4894-9fcb-7b88fc57f9c7 vs c79cf097-80ab-4212-b27f-b999f573cfc7. The shebang for the python3.8 in the venv pex has the path that was valid in a prior remote execution sandbox.

This occurs on the current Pants main. It is unclear when it was introduced since the bug was obscured by the archive extraction bug fixed by https://github.com/pantsbuild/pants/pull/13077.

we spoke about this in the pants chat but here are the details:

i am currently a user of pants for a jvm + python + js monorepo. we use the node_module to call webpack to create some javascript assets. we specify and output_directory for the node_module. a jvm_binary target then depends on the output of this node_module.

currently, pants will cache the npm dependencies for the node module but not the files generated in the output_dir. since these files are used by a jvm_binary, it does not make sense for these output assets to NOT be cached. each compile we do requires re-running webpack even though no javascript files have changed.

in short, i would like to be able to cache the files in output_dir from node_module.

The current state of the "target" API that is consumed by @rules is very scattered, mostly due to a need for backward compatibility, and the incremental introduction of v2 to the rest of the codebase.

Now that @console_rules have begun to be used more widely, and this API (that was originally only designed to act as adaptors to allow pantsd to consume @rules for BUILD file parsing) is beginning to be used by end users writing @rules, it's time for another round of design.

A previous bit of design was done in https://docs.google.com/document/d/102EFbwk6cpM9-_4ZSYhMQYA0zL1UKbXzW-DCd8KsFeg/edit?usp=sharing, but some of the considerations there are stale: see comments on this ticket, and example usages of these APIs in the v2 python test runner and in v2 list for more information.

A quick explanation of the status quo as of ~~March 2018~~ ~~April 2019~~ August 2019 ~~February 2020:

None of these APIs are public yet, and they're all going to need to change... they came out of the engine experiment, and they support more features than we need. So we need to figure out which bits to prune.

Currently the hierarchy of construction (bottom to top) is:

• HydratedStruct - A Struct that has been deserialized from a BUILD file. Currently this type supports inlining other Structs via address references: see this example ...where because the configurations field is expecting to receive Structs, the inline address reference there is expanded and inlined into
• TargetAdaptor - A Struct subclass representing a target. So in the case of expanding a legacy build graph, the above step operates on concrete subclasses of TargetAdaptor. A TargetAdaptor still has sources represented as a PathGlobs object in a SourcesField wrapper: ie, the source globs haven't been expanded.
• HydratedTarget - An object containing enough information to actually construct the legacy BuildGraph interface (with an EagerFilesetWithSpec object representing fully expanded and fingerprinted globs). As described on #4769 and in the document linked above, this is not the interface we want to expose to anyone, as it is too granular to avoid expanding sources if the usecase doesn't require them.
• LegacyHydratedTarget - a HydratedTarget but with a BuildFileAddress rather than an Address, which is needed for V1 to work properly.

Test-cases were added test test simple_code_gen, and integration tests for protobuf and wire gen were updated to accomodate the modified file paths due to the isolation strategy.

This would be for formatters that make semantic changes, like Pyupgrade and Autoflake.

The open question is whether we think it's useful to have formatting-only tools like Black and Isort run at the same time as semantic changers like Pyupgrade. Is it worth the convenience of having a single goal, or it's more useful to split out these two separate but related activities?

Either way, these fixer tools should run with lint in check-only mode. This only impacts how you have the rules actually write their changes to disk.

Currently, determine_bootstrap_environment is called to compute a "default" EnvironmentName before the rule graph is entered, such that all @goal_rules have one in scope automatically.

While this was useful for getting things going and proving out our cross-compilation capabilities (demonstrated in #11148 and #13682), it is error prone. The environment design has been trending toward (eventually) giving all targets an environment= field (even if the vast majority of the time it is only set via __defaults__). And in a world where ~all targets have an environment= field, ever accidentally using the default would definitely represent an issue.

Instead, we'd like to put the onus on each @goal_rule to select the relevant environment (usually via targets). Additionally, to resolve the open question in Appendix A in the design doc, we will pin effectively all calculations in graph.py to the __local__ environment (to eventually potentially be made configurable). This will mean that graph-introspection goals won't need to choose environments, and other @goal_rules won't need to select an environment while computing the graph.

Is your feature request related to a problem? Please describe.

In a monorepo you will like have N services (likely containing one-or-more underlying packages) as well as M "common" packages. Logically speaking, any piece of code should be able to dip it's hand into at most 2 top-level packages: it's own and the common code. Having code from service X depend on code from service Y would be a mistake and shouldn't be mechanically allowed.

Describe the solution you'd like Some way of specifying visibility in a BUILD file (with the default as "public" to maintain backwards-compatibility).

Initial implementation can be scoped to only supporting 2 values: "public" and "package".

For simplicity, if visibility was inherited from the nearest-declared-ancestor-dir that would limit the amount of visibility declarations to N BUILD files (one for each service, the M common packages can remain with the public default).

Additionally, this could be hidden behind a plugin to make it truly opt-in.

Describe alternatives you've considered A plugin which enforces this at test/lint/check-time instead of visibility declarations in BUILD files.

Additional context N/A

now that we're adequately entangled in platform-specific deps that are often times resolving (or failing to resolve.. or build..) against pypi, it would probably make sense for us to go ahead and start building and releasing a binary pex of pants along side (or in place of) sdists. this should eliminate most customer issues related to resolves and/or resolve-time builds of native deps at pants install/bootstrap time and ensure a consistent and tested shared set of backing wheels for running pants in most cases.

at Twitter, we deploy pants only via pex using a custom build and release process. having a formally released OSS pex that we can directly consume will help us stay closer to shared releases vs direct sha consumption.

Instead of returning any thrown exceptions for the execution request directly, support catching them in the rules during rule graph execution so recoverable/non critical errors may be handled.

This PR revives previous work by @cosmicexplorer in #10954 bringing it up-to-date with current main -- with some minor adjustments.

The primary difference is in the error handling; tracebacks rely more on the Exception.__cause__ attribute of the errors and rather than adding an additional layer of result wrapping for the nodes, we propagate all the Failures to the rules instead, so any exceptions thrown during rule execution is picked up as a Failure::Throw{} as before, but now it is forwarded to the parent rule to potentially catch. So, use raise .. from e in order to preserve the tracebacks of the previous error when desired.

Example:

@rule
async def catch_merge_digests_error(file_input: FileInput) -> MergedOutput:
    # Create two separate digests writing different contents to the same file path.
    input_1 = CreateDigest((FileContent(path=file_input.filename, content=b"yes"),))
    input_2 = CreateDigest((FileContent(path=file_input.filename, content=b"no"),))
    digests = await MultiGet(Get(Digest, CreateDigest, input_1), Get(Digest, CreateDigest, input_2))
    try:
        merged = await Get(Digest, MergeDigests(digests))
    except Exception as e:
        raise Exception(f"error merging digests for input {file_input}: {e}")
    return MergedOutput(merged)

Describe the bug Since Python 3.7, any StopIteration exceptions raised from within a coroutine is now translated to a RuntimeError.

https://docs.python.org/3/library/exceptions.html?highlight=stopiteration#StopIteration

Pants version main

OS Any

Additional info

#17901 fixed this for the pants.testutil.RuleRunner.run_rule_with_mocks() method.

Affects pants.engine.internals.selectors.native_engine_generator_send() method:

https://github.com/pantsbuild/pants/blob/df1c5b2418d9c62622b6039bb0fcc91b08f1dee4/src/python/pants/engine/internals/selectors.py#L606-L611

It seems to me that the e.args may have been an implementation detail, as it is not mentioned in the docs for StopIteration.

This fixes ./cargo doc: before this PR, it would fail with several errors (and some warnings):

• Standardise on a single version of bollard-stubs. Having two resulted in the warning below. (Having one version also seems like it likely cuts total build time down a little bit since bollard-stubs takes a fair time (10s) to compile on my M1 Mac.)

  warning: output filename collision.
  The lib target `bollard-stubs` in package `bollard-stubs v1.42.0-rc.4 (https://github.com/fussybeaver/bollard.git?rev=2d66d11b44aeff0373ece3d64a44b243e5152973#2d66d11b)` has the same output filename as the lib target `bollard-stubs` in package `bollard-stubs v1.42.0-rc.3`.
  Colliding filename is: /Users/huon/projects/pantsbuild/pants/src/rust/engine/target/doc/bollard_stubs/index.html
  The targets should have unique names.
  This is a known bug where multiple crates with the same name use
  the same path; see <https://github.com/rust-lang/cargo/issues/6313>.
  

• Use <https://...> for links, instead of just plain https://... because rustdoc complains, e.g.:

  error: this URL is not a hyperlink
     --> task_executor/src/lib.rs:149:7
      |
  149 |   /// https://docs.rs/tokio/0.2.20/tokio/task/struct.JoinHandle.html
      |       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ help: use an automatic link instead: `<https://docs.rs/tokio/0.2.20/tokio/task/struct.JoinHandle.html>`
      |
      = note: bare URLs are not automatically turned into clickable links
  

• Add backticks around some text that's mean to be literal code, but is almost HTML (Option<DirectoryDigest>).
• Disable the automatic generation of doc comments for some specific protobuf fields, because they contain invalid HTML:

  warning: unclosed HTML tag `ip`
     --> /Users/huon/projects/pantsbuild/pants/src/rust/engine/target/debug/build/protos-eab05d27eca6b7b0/out/google.rpc.rs:268:32
      |
  268 |     /// For example, "clientip:<ip address of client>" or "project:<Google
      |                                ^^^
      |
      = note: `#[warn(rustdoc::invalid_html_tags)]` on by default
  
  ...
  warning: unclosed HTML tag `owner`
     --> /Users/huon/projects/pantsbuild/pants/src/rust/engine/target/debug/build/protos-eab05d27eca6b7b0/out/google.rpc.rs:367:26
      |
  367 |   /// For example, "user:<owner email>" or "project:<Google developer project
      |                          ^^^^^^
  

This PR also adds a ./cargo doc invocation to CI, to stop this regressing in future. This seems to take ~1.5 minutes. I don't know if it is worth the time: it may just be better to fix this on demand, if/when it breaks.

Background: I've been poking at the Rust core little bit, and have found it a bit hard to understand what's going on with certain parts (e.g. the generated protobuf wrappers). Having access to this documentation locally (./cargo doc --open) was helpful.

This capability already existed in Pex, this PR plumbs it through in Pants.

This will allow, e.g., running a single gunicorn PEX with different args for different services, without needing a shim file.