The Pants Build System

Overview

Pants Build System

Pants is a scalable build system for monorepos: codebases containing multiple projects, often using multiple programming languages and frameworks, in a single unified code repository.

Some noteworthy features include:

  • Explicit dependency modeling.
  • Fine-grained invalidation.
  • Shared result caching.
  • Concurrent execution.
  • Remote execution.
  • Unified interface for multiple tools and languages.
  • Extensibility and customizability via a plugin API.

Documentation: www.pantsbuild.org.

We release to PyPI version license

We use Travis CI to verify the build Build Status.

Requirements

To run Pants, you need:

  • Linux or macOS.
  • Python 3.7+ discoverable on your PATH.
  • A C compiler, system headers and Python headers (to compile native Python modules).
  • Internet access (so that Pants can fully bootstrap itself).
Comments
  • Switch to PEX for lockfile generation

    Switch to PEX for lockfile generation

    See https://github.com/pantsbuild/pants/pull/13962 for an example and https://github.com/pantsbuild/pants/pull/13963 for prework. Blocked on https://github.com/pantsbuild/pex/issues/1402.

    We have to figure out if we want to keep Poetry for now for generating tool lockfiles, but deprecated. Poetry is broken in that [python-repos] does not work, and we also leak the cache globally. So there's motivation to eagerly switch. But, the resolver will change which could break some users, and the lockfile contents have a different format (new lockfile is more useful).

    User lockfiles should always use Pex because that's a new feature. It's only tool lockfiles where we need to worry about backwards compatibility.

    opened by Eric-Arellano 52
  • `--use-first-matching-interpreter` shebang doesn't play nicely with macOS Python 3.7 install

    `--use-first-matching-interpreter` shebang doesn't play nicely with macOS Python 3.7 install

    pants fmt failure on master as of master plus just some typing changes in https://github.com/pantsbuild/pants/pull/10647 : env: python3.7: No such file or directory

    OS is macOS 10.15.6. which python3 reveals interpreter in a venv that is v3.8.5.

    $ ./pants fmt src/python/pants::
       Compiling engine v0.0.1 (XXX)
        Finished release [optimized + debuginfo] target(s) in 2m 49s
    12:11:49 [INFO] initializing pantsd...
    12:12:00 [INFO] pantsd initialized.
    12:12:08.46 [INFO] Completed: Building black.pex with 2 requirements: black==19.10b0, setuptools
    12:12:08 [WARN] /Users/tdyas/TC/pants/src/python/pants/base/exception_sink.py:359: DeprecationWarning: PY_SSIZE_T_CLEAN will be required for '#' formats
      process_title=setproctitle.getproctitle(),
    
    12:12:08 [ERROR] 1 Exception encountered:
    
    Engine traceback:
      in select
      in `fmt` goal
      in pants.backend.python.lint.python_fmt.format_python_target
      in Format with Black
      in pants.engine.process.fallible_to_exec_result_or_raise
    Traceback (most recent call last):
      File "XXX/pants/src/python/pants/engine/process.py", line 229, in fallible_to_exec_result_or_raise
        raise ProcessExecutionFailure(
    pants.engine.process.ProcessExecutionFailure: Process 'Run Black on 413 files.' failed with exit code 127.
    stdout:
    
    stderr:
    env: python3.7: No such file or directory
    
    Traceback (most recent call last):
      File "XXX/pants/src/python/pants/bin/local_pants_runner.py", line 255, in run
        engine_result = self._run_v2()
      File "XXX/pants/src/python/pants/bin/local_pants_runner.py", line 166, in _run_v2
        return self._maybe_run_v2_body(goals, poll=False)
      File "XXX/pants/src/python/pants/bin/local_pants_runner.py", line 183, in _maybe_run_v2_body
        return self.graph_session.run_goal_rules(
      File "XXX/pants/src/python/pants/init/engine_initializer.py", line 117, in run_goal_rules
        exit_code = self.scheduler_session.run_goal_rule(
      File "XXX/pants/src/python/pants/engine/internals/scheduler.py", line 552, in run_goal_rule
        self._raise_on_error([t for _, t in throws])
      File "XXX/pants/src/python/pants/engine/internals/scheduler.py", line 511, in _raise_on_error
        raise ExecutionError(
    pants.engine.internals.scheduler.ExecutionError: 1 Exception encountered:
    
    Engine traceback:
      in select
      in `fmt` goal
      in pants.backend.python.lint.python_fmt.format_python_target
      in Format with Black
      in pants.engine.process.fallible_to_exec_result_or_raise
    Traceback (most recent call last):
      File "XXX/pants/src/python/pants/engine/process.py", line 229, in fallible_to_exec_result_or_raise
        raise ProcessExecutionFailure(
    pants.engine.process.ProcessExecutionFailure: Process 'Run Black on 413 files.' failed with exit code 127.
    stdout:
    
    stderr:
    env: python3.7: No such file or directory
    
    bug 
    opened by tdyas 40
  • Coursier break second and nexts compilations

    Coursier break second and nexts compilations

    If I compile twice the same target, first time it is a success, second time it fails as if the 3rd party dependency is not in the classpath anymore.

    I've set up a minimal example here. The steps to reproduce the error is to do ./pants compile test-app/src/main/scala twice.

    The coursier jar used is created with this script .

    @cosmicexplorer reproduced the error using pants version 1.14.0.dev0 even when he removed the overrides to coursier

    bug 
    opened by lgirault 34
  • pex python interpreter path contains absolute path from previous remote execution sandbox

    pex python interpreter path contains absolute path from previous remote execution sandbox

    Protobuf code generation is currently failing when run in remote execution against Buildfarm even after the fix in https://github.com/pantsbuild/pants/pull/13077 for archive extraction in remote execution.

    Using the example-python repository for the test, the error that occurs when running ./pants test helloworld:: (with appropriate options to enable remote execution) is:

    Exception message: 1 Exception encountered:
    
      ProcessExecutionFailure: Process 'Generating Python sources from helloworld/util/proto/config.proto.' failed with exit code 1.
    stdout:
    
    stderr:
    /tmp/worker/remote-execution/operations/28150c45-177d-4894-9fcb-7b88fc57f9c7/.cache/pex_root/venvs/8ccfdd776b06121b883ca8d3331e429b87fe042d/ddab8011daaee380698ac2fb9701af18c90c03f6/bin/protoc-gen-mypy: 2: exec: /tmp/worker/remote-execution/operations/c79cf097-80ab-4212-b27f-b999f573cfc7/.cache/pex_root/venvs/short/02a3c5ef/bin/python3.8: not found
    --mypy_out: protoc-gen-mypy: Plugin failed with status code 127.
    

    Note that the path to protoc-gen-mypy and the path to python3.8 are different: 28150c45-177d-4894-9fcb-7b88fc57f9c7 vs c79cf097-80ab-4212-b27f-b999f573cfc7. The shebang for the python3.8 in the venv pex has the path that was valid in a prior remote execution sandbox.

    This occurs on the current Pants main. It is unclear when it was introduced since the bug was obscured by the archive extraction bug fixed by https://github.com/pantsbuild/pants/pull/13077.

    bug 
    opened by tdyas 33
  • add cacheing support for output of node_module

    add cacheing support for output of node_module

    we spoke about this in the pants chat but here are the details:

    i am currently a user of pants for a jvm + python + js monorepo. we use the node_module to call webpack to create some javascript assets. we specify and output_directory for the node_module. a jvm_binary target then depends on the output of this node_module.

    currently, pants will cache the npm dependencies for the node module but not the files generated in the output_dir. since these files are used by a jvm_binary, it does not make sense for these output assets to NOT be cached. each compile we do requires re-running webpack even though no javascript files have changed.

    in short, i would like to be able to cache the files in output_dir from node_module.

    opened by LarryFinn 32
  • Design v2 Target API

    Design v2 Target API

    The current state of the "target" API that is consumed by @rules is very scattered, mostly due to a need for backward compatibility, and the incremental introduction of v2 to the rest of the codebase.

    Now that @console_rules have begun to be used more widely, and this API (that was originally only designed to act as adaptors to allow pantsd to consume @rules for BUILD file parsing) is beginning to be used by end users writing @rules, it's time for another round of design.

    A previous bit of design was done in https://docs.google.com/document/d/102EFbwk6cpM9-_4ZSYhMQYA0zL1UKbXzW-DCd8KsFeg/edit?usp=sharing, but some of the considerations there are stale: see comments on this ticket, and example usages of these APIs in the v2 python test runner and in v2 list for more information.


    A quick explanation of the status quo as of ~~March 2018~~ ~~April 2019~~ August 2019 ~~February 2020:

    None of these APIs are public yet, and they're all going to need to change... they came out of the engine experiment, and they support more features than we need. So we need to figure out which bits to prune.

    Currently the hierarchy of construction (bottom to top) is:

    • HydratedStruct - A Struct that has been deserialized from a BUILD file. Currently this type supports inlining other Structs via address references: see this example ...where because the configurations field is expecting to receive Structs, the inline address reference there is expanded and inlined into
    • TargetAdaptor - A Struct subclass representing a target. So in the case of expanding a legacy build graph, the above step operates on concrete subclasses of TargetAdaptor. A TargetAdaptor still has sources represented as a PathGlobs object in a SourcesField wrapper: ie, the source globs haven't been expanded.
    • HydratedTarget - An object containing enough information to actually construct the legacy BuildGraph interface (with an EagerFilesetWithSpec object representing fully expanded and fingerprinted globs). As described on #4769 and in the document linked above, this is not the interface we want to expose to anyone, as it is too granular to avoid expanding sources if the usecase doesn't require them.
    • LegacyHydratedTarget - a HydratedTarget but with a BuildFileAddress rather than an Address, which is needed for V1 to work properly.
    opened by stuhood 32
  • Implemented isolated code-generation strategy for simple_codegen_task.

    Implemented isolated code-generation strategy for simple_codegen_task.

    Test-cases were added test test simple_code_gen, and integration tests for protobuf and wire gen were updated to accomodate the modified file paths due to the isolation strategy.

    opened by gmalmquist 32
  • Should we have a `fix` goal (vs. `fmt`)?

    Should we have a `fix` goal (vs. `fmt`)?

    This would be for formatters that make semantic changes, like Pyupgrade and Autoflake.

    The open question is whether we think it's useful to have formatting-only tools like Black and Isort run at the same time as semantic changers like Pyupgrade. Is it worth the convenience of having a single goal, or it's more useful to split out these two separate but related activities?

    Either way, these fixer tools should run with lint in check-only mode. This only impacts how you have the rules actually write their changes to disk.

    opened by Eric-Arellano 31
  • Require goals to explicitly locate/choose an environment, and pin `graph` calculations to `__local__`

    Require goals to explicitly locate/choose an environment, and pin `graph` calculations to `__local__`

    Currently, determine_bootstrap_environment is called to compute a "default" EnvironmentName before the rule graph is entered, such that all @goal_rules have one in scope automatically.

    While this was useful for getting things going and proving out our cross-compilation capabilities (demonstrated in #11148 and #13682), it is error prone. The environment design has been trending toward (eventually) giving all targets an environment= field (even if the vast majority of the time it is only set via __defaults__). And in a world where ~all targets have an environment= field, ever accidentally using the default would definitely represent an issue.

    Instead, we'd like to put the onus on each @goal_rule to select the relevant environment (usually via targets). Additionally, to resolve the open question in Appendix A in the design doc, we will pin effectively all calculations in graph.py to the __local__ environment (to eventually potentially be made configurable). This will mean that graph-introspection goals won't need to choose environments, and other @goal_rules won't need to select an environment while computing the graph.

    estimate: ~2W 
    opened by stuhood 29
  • "Visibility" support (akin to Bazel's)

    Is your feature request related to a problem? Please describe.

    In a monorepo you will like have N services (likely containing one-or-more underlying packages) as well as M "common" packages. Logically speaking, any piece of code should be able to dip it's hand into at most 2 top-level packages: it's own and the common code. Having code from service X depend on code from service Y would be a mistake and shouldn't be mechanically allowed.

    Describe the solution you'd like Some way of specifying visibility in a BUILD file (with the default as "public" to maintain backwards-compatibility).

    Initial implementation can be scoped to only supporting 2 values: "public" and "package".

    For simplicity, if visibility was inherited from the nearest-declared-ancestor-dir that would limit the amount of visibility declarations to N BUILD files (one for each service, the M common packages can remain with the public default).

    Additionally, this could be hidden behind a plugin to make it truly opt-in.

    Describe alternatives you've considered A plugin which enforces this at test/lint/check-time instead of visibility declarations in BUILD files.

    Additional context N/A

    enhancement 
    opened by joshua-cannon-techlabs 29
  • release pants as a pex

    release pants as a pex

    now that we're adequately entangled in platform-specific deps that are often times resolving (or failing to resolve.. or build..) against pypi, it would probably make sense for us to go ahead and start building and releasing a binary pex of pants along side (or in place of) sdists. this should eliminate most customer issues related to resolves and/or resolve-time builds of native deps at pants install/bootstrap time and ensure a consistent and tested shared set of backing wheels for running pants in most cases.

    at Twitter, we deploy pants only via pex using a custom build and release process. having a formally released OSS pex that we can directly consume will help us stay closer to shared releases vs direct sha consumption.

    opened by kwlzn 29
  • Support catching `@rule` errors

    Support catching `@rule` errors

    Instead of returning any thrown exceptions for the execution request directly, support catching them in the rules during rule graph execution so recoverable/non critical errors may be handled.

    This PR revives previous work by @cosmicexplorer in #10954 bringing it up-to-date with current main -- with some minor adjustments.

    The primary difference is in the error handling; tracebacks rely more on the Exception.__cause__ attribute of the errors and rather than adding an additional layer of result wrapping for the nodes, we propagate all the Failures to the rules instead, so any exceptions thrown during rule execution is picked up as a Failure::Throw{} as before, but now it is forwarded to the parent rule to potentially catch. So, use raise .. from e in order to preserve the tracebacks of the previous error when desired.

    Example:

    @rule
    async def catch_merge_digests_error(file_input: FileInput) -> MergedOutput:
        # Create two separate digests writing different contents to the same file path.
        input_1 = CreateDigest((FileContent(path=file_input.filename, content=b"yes"),))
        input_2 = CreateDigest((FileContent(path=file_input.filename, content=b"no"),))
        digests = await MultiGet(Get(Digest, CreateDigest, input_1), Get(Digest, CreateDigest, input_2))
        try:
            merged = await Get(Digest, MergeDigests(digests))
        except Exception as e:
            raise Exception(f"error merging digests for input {file_input}: {e}")
        return MergedOutput(merged)
    
    category:new feature 
    opened by kaos 0
  • Coroutines `StopIteration` is always `return`

    Coroutines `StopIteration` is always `return`

    Describe the bug Since Python 3.7, any StopIteration exceptions raised from within a coroutine is now translated to a RuntimeError.

    https://docs.python.org/3/library/exceptions.html?highlight=stopiteration#StopIteration

    Pants version main

    OS Any

    Additional info

    #17901 fixed this for the pants.testutil.RuleRunner.run_rule_with_mocks() method.

    Affects pants.engine.internals.selectors.native_engine_generator_send() method:

    https://github.com/pantsbuild/pants/blob/df1c5b2418d9c62622b6039bb0fcc91b08f1dee4/src/python/pants/engine/internals/selectors.py#L606-L611

    It seems to me that the e.args may have been an implementation detail, as it is not mentioned in the docs for StopIteration.

    bug 
    opened by kaos 0
  • Fix errors and warnings in `./cargo doc`, and run in CI

    Fix errors and warnings in `./cargo doc`, and run in CI

    This fixes ./cargo doc: before this PR, it would fail with several errors (and some warnings):

    • Standardise on a single version of bollard-stubs. Having two resulted in the warning below. (Having one version also seems like it likely cuts total build time down a little bit since bollard-stubs takes a fair time (10s) to compile on my M1 Mac.)
      warning: output filename collision.
      The lib target `bollard-stubs` in package `bollard-stubs v1.42.0-rc.4 (https://github.com/fussybeaver/bollard.git?rev=2d66d11b44aeff0373ece3d64a44b243e5152973#2d66d11b)` has the same output filename as the lib target `bollard-stubs` in package `bollard-stubs v1.42.0-rc.3`.
      Colliding filename is: /Users/huon/projects/pantsbuild/pants/src/rust/engine/target/doc/bollard_stubs/index.html
      The targets should have unique names.
      This is a known bug where multiple crates with the same name use
      the same path; see <https://github.com/rust-lang/cargo/issues/6313>.
      
    • Use <https://...> for links, instead of just plain https://... because rustdoc complains, e.g.:
      error: this URL is not a hyperlink
         --> task_executor/src/lib.rs:149:7
          |
      149 |   /// https://docs.rs/tokio/0.2.20/tokio/task/struct.JoinHandle.html
          |       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ help: use an automatic link instead: `<https://docs.rs/tokio/0.2.20/tokio/task/struct.JoinHandle.html>`
          |
          = note: bare URLs are not automatically turned into clickable links
      
    • Add backticks around some text that's mean to be literal code, but is almost HTML (Option<DirectoryDigest>).
    • Disable the automatic generation of doc comments for some specific protobuf fields, because they contain invalid HTML:
      warning: unclosed HTML tag `ip`
         --> /Users/huon/projects/pantsbuild/pants/src/rust/engine/target/debug/build/protos-eab05d27eca6b7b0/out/google.rpc.rs:268:32
          |
      268 |     /// For example, "clientip:<ip address of client>" or "project:<Google
          |                                ^^^
          |
          = note: `#[warn(rustdoc::invalid_html_tags)]` on by default
      
      ...
      warning: unclosed HTML tag `owner`
         --> /Users/huon/projects/pantsbuild/pants/src/rust/engine/target/debug/build/protos-eab05d27eca6b7b0/out/google.rpc.rs:367:26
          |
      367 |   /// For example, "user:<owner email>" or "project:<Google developer project
          |                          ^^^^^^
      

    This PR also adds a ./cargo doc invocation to CI, to stop this regressing in future. This seems to take ~1.5 minutes. I don't know if it is worth the time: it may just be better to fix this on demand, if/when it breaks.

    Background: I've been poking at the Rust core little bit, and have found it a bit hard to understand what's going on with certain parts (e.g. the generated protobuf wrappers). Having access to this documentation locally (./cargo doc --open) was helpful.

    opened by huonw 0
  • Support freezing command line args and env vars into pex binaries.

    Support freezing command line args and env vars into pex binaries.

    This capability already existed in Pex, this PR plumbs it through in Pants.

    This will allow, e.g., running a single gunicorn PEX with different args for different services, without needing a shim file.

    category:new feature 
    opened by benjyw 2
Releases(release_2.16.0.dev4)
Owner
Pants Build
Pants Build System
Pants Build
The Meson Build System

Meson® is a project to create the best possible next-generation build system. Status Dependencies Python (version 3.6 or newer) Ninja (version 1.8.2 o

The Meson Build System 4.4k Jan 2, 2023
Clang-based cross platform build system written in Python

Clang-build Find the full documentation at https://clang-build.readthedocs.io First steps Customisations Multiple targets Multiple projects Defaults M

Trick 17 9 Jun 29, 2022
Software build automation tool for Python.

PyBuilder — an easy-to-use build automation tool for Python PyBuilder is a software build tool written in 100% pure Python, mainly targeting Python ap

PyBuilder 1.5k Jan 4, 2023
A pynt of Python build.

A pynt of Python build. Raghunandan Rao Features Easy to learn. Build tasks are just python funtions. Manages dependencies between tasks. Automaticall

Raghunandan Rao 154 Jan 4, 2023
🔨🐍Make-like build automation tool for Python projects with extensive DSL features.

Pyke (WIP, Beta Release) Make-like build automation tool for Python projects with extensive DSL features. Features: Users can specify tasks, subtasks,

Ire 17 Jul 5, 2022
A Star Trek Online build tool in Python

SETS - STO Equipment and Trait Selector A Star Trek Online build tool in Python Description Pre-alpha version of build tool for STO Getting Started De

Star Trek Online Community Developers 7 Nov 12, 2022
Ubuntu env build; Nginx build; DB build;

Deploy 介绍 Deploy related scripts bitnami Dependencies Ubuntu openssl envsubst docker v18.06.3 docker-compose init base env upload https://gitlab-runn

Colin(liuji) 10 Dec 1, 2021
:package: :fire: Python project management. Manage packages: convert between formats, lock, install, resolve, isolate, test, build graph, show outdated, audit. Manage venvs, build package, bump version.

THE PROJECT IS ARCHIVED Forks: https://github.com/orsinium/forks DepHell -- project management for Python. Why it is better than all other tools: Form

DepHell 1.7k Dec 30, 2022
An easy way to build PyTorch datasets. Modularly build datasets and automatically cache processed results

EasyDatas An easy way to build PyTorch datasets. Modularly build datasets and automatically cache processed results Installation pip install git+https

Ximing Yang 4 Dec 14, 2021
Official project repository for the Setuptools build system

See the Installation Instructions in the Python Packaging User's Guide for instructions on installing, upgrading, and uninstalling Setuptools. Questio

Python Packaging Authority 1.9k Jan 8, 2023
The Meson Build System

Meson® is a project to create the best possible next-generation build system. Status Dependencies Python (version 3.6 or newer) Ninja (version 1.8.2 o

The Meson Build System 4.4k Jan 2, 2023
open-information-extraction-system, build open-knowledge-graph(SPO, subject-predicate-object) by pyltp(version==3.4.0)

中文开放信息抽取系统, open-information-extraction-system, build open-knowledge-graph(SPO, subject-predicate-object) by pyltp(version==3.4.0)

null 7 Nov 2, 2022
Ninja is a small build system with a focus on speed.

Ninja Python Distributions Ninja is a small build system with a focus on speed. The latest Ninja python wheels provide ninja 1.10.2.g51db2.kitware.job

null 33 Dec 19, 2022
Clang-based cross platform build system written in Python

Clang-build Find the full documentation at https://clang-build.readthedocs.io First steps Customisations Multiple targets Multiple projects Defaults M

Trick 17 9 Jun 29, 2022
Build AGNOS, the operating system for your comma three

agnos-builder This is the tool to build AGNOS, our Ubuntu based OS. AGNOS runs on the comma three devkit. NOTE: the edk2_tici and agnos-firmare submod

comma.ai 21 Dec 24, 2022
A dynamic multi-STL, multi-process OpenSCAD build system with autoplating support

scad-build This is a multi-STL OpenSCAD build system based around GNU make. It supports dynamic build targets, intelligent previews with user-defined

Jordan Mulcahey 1 Dec 21, 2021
Build a medical knowledge graph based on Unified Language Medical System (UMLS)

UMLS-Graph Build a medical knowledge graph based on Unified Language Medical System (UMLS) Requisite Install MySQL Server 5.6 and import UMLS data int

Donghua Chen 6 Dec 25, 2022
OptiPLANT is a cloud-based based system that empowers professional and non-professional data scientists to build high-quality predictive models

OptiPLANT OptiPLANT is a cloud-based based system that empowers professional and non-professional data scientists to build high-quality predictive mod

Intellia ICT 1 Jan 26, 2022
AoT is a system for automatically generating off-target test harness by using build information.

AoT: Auto off-Target Automatically generating off-target test harness by using build information. Brought to you by the Mobile Security Team at Samsun

Samsung 10 Oct 19, 2022
A RESTful API for creating and monitoring resource components of a hypothetical build system. Built with FastAPI and pydantic. Complete with testing and CI.

diskspace-monitor-CRUD Background The build system is part of a large environment with a multitude of different components. Many of the components hav

Nick Hopewell 67 Dec 14, 2022