Clang-based cross platform build system written in Python

Overview

Clang-build

PyPI version CI Code quality Coverage

Demonstration

Find the full documentation at https://clang-build.readthedocs.io

Motivation:

  • Building as much as possible from source eases dependency management and ensures stability and reproducibility
  • Meta build systems are inherently the wrong way to go, either the build system or the compiler should be platform-agnostic (ideally both).
  • Trying to cover all use-cases is the wrong way to go - there is no need to let people do it the wrong way
  • CMake is cumbersome, unnecessarily generic and verbose and people should not need a programming/scripting language whose only purpose is to build C++
  • With Clang, finally a properly cross-platform compiler exists

Goals:

  • One compiler (Clang), one build system (written in Python)
  • Automatic integration with language servers (compile_commands.json)
  • Simple projects should be simple to build
  • Build process for reasonable project structures should still be easy
  • Adding third-party dependencies should be manageable

What it's not designed to do:

  • Build anything aside from C language dialects
  • Be able to adapt to any project structure in the world - certain standards are encouraged
  • Work smoothly with or locate pre-built libraries and libraries installed by system package managers

Related resources:

Usage

In order to run clang-build, you only need Clang and Python3. Install via pip install clang-build (add the --user flag if you don't have admin rights).

Running clang-build will try to build the current directory. The command-line options include

  • -d path/to/dir to build a different directory
  • -p to show a progress bar
  • -V to print some additional info
  • --debug to print the called clang commands

The given directory will be searched for a clang-build.toml file, which you can use to configure your build targets, if necessary. However, if you only want to build an executable, you will likely not even need a build file.

clang-build tries to use sane defaults, designed to make most projects very easy to configure and even complex projects far easier than with common build or meta-build systems.

Real-World Examples

Examples of real-world used and tested projects, which can be easily be integrated into your project using clang-build:

General Ideas

Note: not all of these are implemented, yet.

What should be trivial

This would be things that require only the invocation of clang-build and no build file.

  • build any hello world program or other MWE, given a reasonable folder structure (i.e anything with a main and without non-std dependencies)
  • include anything that can be found by sane default search
  • using command line arguments:
    • specify root/source and build directories
    • set build type (last used should be cached/remembered)
    • set verbosity

Sane defaults and default behaviour:

  • platform-independence
  • build into a "build/" directory, not into toplevel
  • for multiple targets build each into its own "build/targetname"
  • default search paths for different platforms, including also e.g. "./include", "./lib", "./build/lib", "/usr/local/...", ...

What should be easy

This would be things that only require a minimal TOML project file

  • add dependency / external project from source folder or remote (e.g. github)
  • header-only should be trivial
  • for a library with a good folder structure, it should be easy to write a build config
  • create a library from one subfolder, an executable from another and link them
  • setting target-specific (note: defaults should be sane!)
    • source file extensions
    • source directories
    • compile and link flags
    • optional version
    • dependencies (which may include non-targets, e.g. configuration steps)
    • properties (required c++ version, definitions/#defines, ...)
  • access to flag "lists" such as flags for
    • coverage
    • cuda
    • openmp
  • set target-specific flags, include folders, etc. which should not be propagated to dependency parents as "private"

What should be possible

Steps that would involve more effort from the user, including possibly some python code

  • a Target configuration step before building (e.g. for more involved version numbering)
  • through the configuration step, inclusion of e.g. CMake-project should be possible
  • packaging: any target may be packaged, meaning it's dependencies are handled and if built, binaries may be bundled
  • external package dependencies
  • binaries on a server
  • source on a server (fallback from binaries)
  • binaries on disk, try to determine version from path and file names
  • source on disk, try to determine version from path and file names

Project File By Example

A single target

Note:

  • by default, the root and folders, as well as "include" and "src" subdirectories will be searched for ".hpp", ".hxx", ".h" and ".cpp", ".cxx" and ".c" files
  • a target without target_type, but with source files will be an executable
  • output_name should not contain pre- or suffixes such as lib, .exe, .so, as they are added automatically
  • if we don't care about the output name, in this case we could skip the project file entirely
# Top-level brackets indicate a target
[hello]
output_name = "runHello"

Two targets with linking

# Build a library
[mylib]
target_type = "shared library"

# Build an executable and link the library
[myexe]
output_name = "runExe"
target_type = "executable"
dependencies = ["mylib"]
[myexe.flags]
link = ["-DMYEXE_SOME_DEFINE"]

Adding external dependencies

Note:

  • external targets will be copied/downloaded into "build/targetname/external_sources"
  • you can specify a subdirectory, if the thirdparty code has an unusual structure
  • further granularity is given by include_directories and sources
  • sources, headers_exclude and sources_exclude expect a list of globbing patterns or files (not folders!)
[mylib]
url = "https://github.com/trick-17/mylib"
version = 1.1 # will try to `git checkout 1.1`
directory = "sources"           # will point to "build/mylib/external_sources/sources"
include_directories = ["mylib/include"] # will point to "build/mylib/external_sources/sources/mylib/include"
sources = ["mylib/src/*"]     # will list everything inside "build/mylib/external_sources/sources/mylib/src"
# Maybe we need to deactivate annoying warnings coming from the library
[mylib.flags]
compile = ["-Wno-deprecated-declarations", "-Wno-self-assign"]

# Build an executable and link the library
[myexe]
dependencies = ["mylib"]
Comments
  • Update documentation

    Update documentation

    On PR #23, a lot of changes were made and the documentation needs to be updated accordingly.

    • [x] update Readme with better example of small config file and a full config file
    • [x] update Readme and make it more compact
    • [x] add docs folder and readthedocs
    • [x] add step by step incremental guide to documentation in docs folder
    • [x] add note that boost filesystem can be built with this, as a non-trivial example of a real-world application
    documentation 
    opened by GPMueller 13
  • Allow explicit listing of sources and deactivation of glob

    Allow explicit listing of sources and deactivation of glob

    Specifying folders instead of files is more sensible, but sometimes a complete glob is too much. Therefore it would be good to have the options to

    • manually specify additional source files outside of the given folders
    • deactivate globbing of folders (per target)
    discussion 
    opened by GPMueller 10
  • Compile flag order mixup

    Compile flag order mixup

    I am not sure if this has been an issue so far, but I just noticed that in target.py the compile flags order is scrambled. I think the intent was to get rid of duplicates, but the current implementation could change the order:

    https://github.com/Trick-17/clang-build/blob/c5d5c1bba495566e7a2936e84593fed04f0b136f/clang_build/target.py#L148

    bug 
    opened by NOhs 9
  • A clang-build tooling API

    A clang-build tooling API

    So as discussed previously, I think it would make sense to not only have clang-build as a script/app that you can run, but something, you can also integrate into your project/workflow. The clang-build script itself should then use that API to provide the current functionality. I think we should play around with a simple first version, something like that:

    Project discovery

    • get_targets(folder_name) -> tree of targets

    Project creation

    • create_target?

    Project inspection

    • target class should have public fields:
      • Name
      • Type
      • List of targets it depends on
      • Attributes set in clang toml
      • Output name
      • Output path

    Should this be read/write? Then we could potentially have a gui to set toml files.

    Target compilation

    • compile_targets(*targets), should have some way to report progress to the outside world, maybe we could even remove the hard-coded progress output in the actual target class definitions etc.

    CLI interface

    • Since some projects might not use Python, it might make sense to mirror these functions to the command line so that other programs can just parse the command line output, e.g. we could have something like:
    clang-build --function get_targets "/home/my_project"
    

    would print

    target_0_name
        dependency_1_name
        dependency_2_name
            dependency_3_name
            dependency_3_name
    target_1_name
    

    with the names from the toml files etc, as usual.

    and for all API functions this could automatically get wrapped (though we have to make sure that the documentation is still good).

    So it is obvious that we have most of the stuff anyways in some form in our project, but I think cleaning it up to such a clean interface might help in the long run. Please feel free to edit as needed (I might have forgotten some things).

    discussion 
    opened by NOhs 8
  • Filter output

    Filter output

    Currently you are using print statements everywhere. However, if you would use functions like the python logging stuff, you could set different output levels. So people who want to integrate your code into their toolchain could silence your output or select how much output they want.

    enhancement 
    opened by NOhs 7
  • More warnings

    More warnings

    I think the following flags should be added to the defaults:

    -fsanitize=address
    

    see https://clang.llvm.org/docs/AddressSanitizer.html

    and also

    -Wshadow
    

    seem useful flags for writing clean code. The first one can help identify illegal access mistakes and the second one can help finding shadowing bugs like in e.g. nested loops where the loop variables are shadowed in one of the inner loops.

    enhancement discussion 
    opened by NOhs 6
  • Checking the readme.rst

    Checking the readme.rst

    Since pypi is supposed to nicely render the *.rst file, we have to make sure it is bug-free. For this several tools can be put in the CI scripts, see also this discussion: https://stackoverflow.com/questions/16367770/my-rst-readme-is-not-formatted-on-pypi-python-org

    documentation tests 
    opened by NOhs 6
  • Typo in example

    Typo in example

    In the readme it says

    # Include an external package/target (i.e. not from this toml file)
    [somelib]
    external = true
    path = "/path/to/sources"
    
    # Build an executable and link the library
    [myexe]
    [myexe.sources]
    include_directories = ["include", "mylib.sources.include_directories"]
    source_directories = ["src"]
    
    [myexe.link]
    dependencies = ["somelib"]
    

    Shouldn't it read: somelib.sources.include_directories?

    Also I am not sure I understand the example. So we have the linker dependency on somelib. Why do we have to explicitly give it the source files again? This seems redundant and error prone (one might forget one or the other and not directly notice it, because there is an entry already that states that there is a dependency).

    question documentation 
    opened by NOhs 6
  • Link-Command: distinguish between C and C++

    Link-Command: distinguish between C and C++

    In single_source.py the file extension will decide which compiler to use. For the linker, there is no such thing (see target.py), but I am not sure if the clangpp and clang linker are different.

    discussion 
    opened by NOhs 5
  • Removed list mangling

    Removed list mangling

    Regarding #74, this merge request removes most occurences of set where we might want to preserve the order. Compile flags for sure need the order preserved. For the include paths I think it shouldn't be advertised, but for debugging purposes it is better if the include paths are in a somewhat reasonable order.

    opened by NOhs 5
  • Comment and code contradict each other

    Comment and code contradict each other

    Looking at the following code:

    https://github.com/Trick-17/clang-build/blob/c5d5c1bba495566e7a2936e84593fed04f0b136f/clang_build/target.py#L76-L80

    We see that while the comment says that header only targets' include directories are always public, this is not realized in the actual code, hence requiring this unnecessary check to then also include the "non-public" include directories of header only targets.

    discussion 
    opened by NOhs 5
  • Autogenerate docusaurus GH pages

    Autogenerate docusaurus GH pages

    A GH page is easy to setup and we now have a test page at https://trick-17.github.io/clang-build which is published from https://github.com/Trick-17/clang-build/tree/gh-pages

    The goals would be

    1. tagged commits build a new version
    2. untagged master commits build and overwrite the latest version
    3. other commits could build and overwrite the branch-name version

    Note: point (3) would be intended for us to be able to test new documentation pages, see generated docs before releasing, etc. - so for this to work as intended, we would have to remove the corresponding docs page when merging a branch, which I don't think we can reasonably automate. It might be easier for us to not publish branch docs and build & test them locally instead.

    documentation CI 
    opened by GPMueller 0
  • Improve toolchain API: defaults

    Improve toolchain API: defaults

    This is a follow-up to issue #123, which was implemented on PR #129. An open point that wasn't addressed:

    refactor the way default flags etc. work to make custom toolchains safer and easier to create

    Currently, the default clang toolchain specifies

    DEFAULT_COMPILE_FLAGS = {
        BuildType.Default: [],
        BuildType.Release: [],
        BuildType.RelWithDebInfo: [],
        BuildType.Debug: [],
        BuildType.Coverage: [],
    }
    
    DEFAULT_LINK_FLAGS = {
        BuildType.Default: [],
        BuildType.Release: [],
        BuildType.RelWithDebInfo: [],
        BuildType.Debug: [],
        BuildType.Coverage: [],
    }
    
    PLATFORM_DEFAULTS = {
        "linux": {
            "PLATFORM": "linux",
            "EXECUTABLE_PREFIX": "",
            "EXECUTABLE_SUFFIX": "",
            "SHARED_LIBRARY_PREFIX": "lib",
    ...
    

    which, in a sense, specifies an implicit API through these dictionaries being used across the codebase. We should make this a more explicit API and make it easier to override parts of it, without having to copy-paste entire dictionaries from our sources.

    enhancement 
    opened by GPMueller 0
  • Get compilation-database entries from clang instead of writing them ourselves?

    Get compilation-database entries from clang instead of writing them ourselves?

    This is a continuation of issue #109.

    I discovered that clang, since version 5.0, contains a -MJ flag which outputs compilation database entries:

    -MJ<arg> Write a compilation database entry per input

    It is not immediately clear to me how this could/should be used in clang-build and whether that would make sense currently. It seems we would have to use it for each file in the project, and then merge the outputs into a JSON-formatted compilation database, see also https://github.com/Sarcasm/notes/blob/master/dev/compilation-database.rst#clang


    It seems that for example gcc does not have this command line option, so maybe it would not be beneficial to let clang do it - we should ideally be able to generate a database for any toolchain.

    discussion 
    opened by GPMueller 0
  • pbr versioning seems incorrect, leading to missing PyPI deploy

    pbr versioning seems incorrect, leading to missing PyPI deploy

    See this action for the commit after the 0.1.0 tag: https://github.com/Trick-17/clang-build/runs/3195258480?check_suite_focus=true#step:5:10 The 0.1.0 tagged build has the correct version: https://github.com/Trick-17/clang-build/runs/3176181314?check_suite_focus=true#step:5:17

    bug CI 
    opened by GPMueller 0
  • Add project- and target-properties to deactivate default flags

    Add project- and target-properties to deactivate default flags

    While our defaults are opinionated, we cannot claim that our flags are always the right choice. Therefore, I would like to add a switch on both project- and target-level to deactivate them:

    [myexe]
      default_compile_flags = false
      default_link_flags = false
    

    The feature is already implemented in BuildFlags, it just needs to be exposed.

    enhancement 
    opened by GPMueller 0
  • IO module now behaves according to issue #121.

    IO module now behaves according to issue #121.

    Apart from the changes in behaviour as described in issue #121, the new implementation

    • gets rid of a lot of Path <-> str back and forth conversions,
    • it removes duplicates by consistently using sets instead of manually cleaning up
    • it tries to more closely use terminology used throughout the project
    • it tries to split up the code into reasonable helper functions
    • it tries to improve the expressiveness of the "main" function
    opened by NOhs 1
Releases(0.1.0)
  • 0.1.0(Jul 27, 2021)

    Implemented enhancements:

    • Inconsistency between parameter names for flags and include directories #113
    • Make dependencies "private" by default #101
    • Allow platform-dependence in targets? #68
    • More granular inheritance of flags #66
    • Improve the build stages #64
    • Recursive cloning of git submodules #62
    • Allow building C code #58
    • Insufficient warning when trying to link an executable into another target #55
    • More warnings #39
    • Allow sub-projects (modularisation) #36
    • depfiles should not always be re-generated #35
    • Remove all sys.exit(1) from within the code #27
    • Improve detection of supported standards #14
    • Filter output #13
    • Improve build folder structure #7
    • Stop build at first failure #5

    Fixed bugs:

    • Improve parsing of flags with parameters (i.e. non-switch flags) #97
    • Compile flag order mixup #74
    • Logging setup mixed into main function #60
    • Target-name and folder name #52
    • Travis OSX clang++ executable is not found #38
    • Travis OSX is still running python2.7 #37
    • Test status in readme shows appveyor two times instead of also showing travis #34
    • PyPi release on every branch will lead to issues with pbr and its naming scheme #30
    • Test for multiple targets, linking shared library #25

    Closed issues:

    • Link-Command: distinguish between C and C++ #94
    • Build Type #92
    • project identifier vs name #91
    • A clang-build tooling API #90
    • Flag parsing more complicated than necessary? #89
    • Solve scripting: a project creation API #87
    • Potential bug in flags order #81
    • First steps documentation #79
    • Comment and code contradict each other #77
    • Tests: try to add a test building a python module with pybind11 #43
    • External keyword might be unnecessary #42
    • Get rid of Eigen copy in repository #40
    • Add clang-build create feature? #33
    • Coverage, which vendor to choose? #29
    • Checking the readme.rst #28
    • Update documentation #26
    • Add ability to call external scripts at certain places #22
    • Mix up of paths in readme #20
    • Inconsistency between with and without toml configuration file scenario #19
    • Renaming myexe.link #17
    • Wrong link in Readme #16
    • Ability to only (re-)build a specific set of Targets? #9
    • Allow explicit listing of sources and deactivation of glob #8
    • Coverage build type? #3
    Source code(tar.gz)
    Source code(zip)
Owner
Trick 17
MWEs and useful projects and examples
Trick 17
The Meson Build System

Meson® is a project to create the best possible next-generation build system. Status Dependencies Python (version 3.6 or newer) Ninja (version 1.8.2 o

The Meson Build System 4.4k Jan 2, 2023
The Pants Build System

Pants Build System Pants is a scalable build system for monorepos: codebases containing multiple projects, often using multiple programming languages

Pants Build 2.5k Jan 7, 2023
Software build automation tool for Python.

PyBuilder — an easy-to-use build automation tool for Python PyBuilder is a software build tool written in 100% pure Python, mainly targeting Python ap

PyBuilder 1.5k Jan 4, 2023
A pynt of Python build.

A pynt of Python build. Raghunandan Rao Features Easy to learn. Build tasks are just python funtions. Manages dependencies between tasks. Automaticall

Raghunandan Rao 154 Jan 4, 2023
🔨🐍Make-like build automation tool for Python projects with extensive DSL features.

Pyke (WIP, Beta Release) Make-like build automation tool for Python projects with extensive DSL features. Features: Users can specify tasks, subtasks,

Ire 17 Jul 5, 2022
A Star Trek Online build tool in Python

SETS - STO Equipment and Trait Selector A Star Trek Online build tool in Python Description Pre-alpha version of build tool for STO Getting Started De

Star Trek Online Community Developers 7 Nov 12, 2022
PlatformIO is a professional collaborative platform for embedded development :alien: A place where Developers and Teams have true Freedom! No more vendor lock-in!

PlatformIO Quick Links: Web | PlatformIO IDE | Project Examples | Docs | Donate | Contact Us Social: LinkedIn | Twitter | Facebook | Community Forums

PlatformIO 6.5k Jan 8, 2023
Buildout is a deployment automation tool written in and extended with Python

Buildout Buildout is a project designed to solve 2 problems: Application-centric assembly and deployment Assembly runs the gamut from stitching togeth

buildout 552 Nov 26, 2022
Python-based continuous integration testing framework; your pull requests are more than welcome!

Buildbot The Continuous Integration Framework Buildbot is based on original work from Brian Warner, and currently maintained by the Botherders. Visit

Buildbot 5k Jan 5, 2023
Python-based project scripting.

Paver - Easy Scripting for Software Projects Web: https://pythonhosted.org/Paver/ Download: https://pypi.python.org/pypi/Paver/ Source: https://github

Paver community 452 Dec 9, 2022
A small clone of GNU Make based on file checksums

Pyke This weekend project is a small clone (most of the code is in a single file of just about 200LoC) of GNU Make with the twist that it rebuilds a t

Antonio De Lucreziis 3 Nov 24, 2021
The official binary distribution format for Python

wheel This library is the reference implementation of the Python wheel packaging standard, as defined in PEP 427. It has two different roles: A setupt

Python Packaging Authority 368 Dec 23, 2022
Simplified packaging of Python modules

Flit is a simple way to put Python packages and modules on PyPI. It tries to require less thought about packaging and help you avoid common mistakes.

Thomas Kluyver 1.9k Jan 5, 2023
Python program for installing many tools automatically

Tool Installer is a script made with python which help user in installing tools automatically

Spider Anongreyhat 6 Mar 13, 2022
PyPacker: a dumb little script for turning Python apps into standalone executable packages on Windows

PyPacker: a dumb little script for turning Python apps into standalone executable packages on Windows PyPacker is my attempt at creating a way to make

Serdar Yegulalp 9 Mar 15, 2022
This is a python helper package for Telebirr H5 Web payment integration helper.

Telebirr This is a python helper package for Telebirr H5 Web payment integration helper. The use case for this interface is payment via web. Third par

null 24 Dec 13, 2022
Python package used on Hardfight projects to make building, testing and deploying easy.

Hardfight Devtools Build, test and deploy Hardfight projects easly ?? What is it Devtools is a Python tool to make building, testing and deploying int

Hardfight 1 Dec 5, 2021
Turn clang-tidy warnings and fixes to comments in your pull request

clang-tidy pull request comments A GitHub Action to post clang-tidy warnings and suggestions as review comments on your pull request. What platisd/cla

Dimitris Platis 30 Dec 13, 2022
Integrate clang-format with Sublime Text

Sublime Text Clang Format Plugin This is a minimal plugin integrating clang-format with Sublime Text, with emphasis on the word minimal. It is not rea

Jon Palmisciano 1 Dec 17, 2021
Ubuntu env build; Nginx build; DB build;

Deploy 介绍 Deploy related scripts bitnami Dependencies Ubuntu openssl envsubst docker v18.06.3 docker-compose init base env upload https://gitlab-runn

Colin(liuji) 10 Dec 1, 2021