One-stop-shop for docs and test coverage of dbt projects.

Overview

dbt-coverage

PyPI Downloads GitHub last commit PyPI - Python Version PyPI - Format License: MIT FOSSA Status

One-stop-shop for docs and test coverage of dbt projects.

Why do I need something like this?

dbt-coverage is to dbt what coverage.py and interrogate are to Python.

It is a single CLI tool which checks your dbt project for missing documentation and tests.

Keeping documentation and tests close to the actual SQL code that generates the final model is one of the best design choices of dbt. It ensures documentation is actually useful and tests are actually used. But how do you make adding those a habit in your dbt project?

That is exactly where dbt-coverage comes in. It will

  • Give you a better sense of the level of documentation and test coverage in your project;
  • Help your CI/CD pipeline make sure new changes include documentation and tests;
  • Let you quickly assess the documentation and tests of a new dbt project you get your hands on.

Still not convinced? Here are some more features:

  • zero-config: just install it and run it, there is nothing to set up
  • 🏁 minimal dependences: the only dependencies are click (already installed with dbt) and typer
  • 📦 very small: at ~480 SLOC, you can easily validate it works as advertised

Installation

pip install dbt-coverage

Usage

dbt-coverage comes with two basic commands: compute and compare. The documentation for the individual commands can be shown by using the --help option.

Compute

Compute coverage from target/catalog.json and target/manifest.json files found in a dbt project, e.g. jaffle_shop.

To choose between documentation and test coverage, pass doc or test as the CLI argument.

$ cd jaffle_shop
$ dbt run  # Materialize models
$ dbt docs generate  # Generate catalog.json and manifest.json
$ dbt-coverage compute doc --cov-report coverage-doc.json  # Compute doc coverage, print it and write it to coverage-doc.json file

Coverage report
=====================================================================
jaffle_shop.customers                                  6/7      85.7%
jaffle_shop.orders                                     9/9     100.0%
jaffle_shop.raw_customers                              0/3       0.0%
jaffle_shop.raw_orders                                 0/4       0.0%
jaffle_shop.raw_payments                               0/4       0.0%
jaffle_shop.stg_customers                              0/3       0.0%
jaffle_shop.stg_orders                                 0/4       0.0%
jaffle_shop.stg_payments                               0/4       0.0%
=====================================================================
Total                                                 15/38     39.5%

$ dbt-coverage compute test --cov-report coverage-test.json  # Compute test coverage, print it and write it to coverage-test.json file

Coverage report
=====================================================================
jaffle_shop.customers                                  1/7      14.3%
jaffle_shop.orders                                     8/9      88.9%
jaffle_shop.raw_customers                              0/3       0.0%
jaffle_shop.raw_orders                                 0/4       0.0%
jaffle_shop.raw_payments                               0/4       0.0%
jaffle_shop.stg_customers                              1/3      33.3%
jaffle_shop.stg_orders                                 2/4      50.0%
jaffle_shop.stg_payments                               2/4      50.0%
=====================================================================
Total                                                 14/38     36.8%

Compare

Compare two coverage.json files generated by the compute command. This is useful to ensure that the coverage does not drop while making changes to the project.

$ dbt-coverage compare coverage-after.json coverage-before.json

# Coverage delta summary
              before     after            +/-
=============================================
Coverage      39.47%    38.46%         -1.01%
=============================================
Tables             8         8          +0/+0
Columns           38        39          +1/+0
=============================================
Hits              15        15          +0/+0
Misses            23        24          +1/+0
=============================================

# New misses
=========================================================================
Catalog                         15/38   (39.47%)  ->    15/39   (38.46%) 
=========================================================================
- jaffle_shop.customers          6/7    (85.71%)  ->     6/8    (75.00%) 
-- new_col                       -/-       (-)    ->     0/1     (0.00%) 
=========================================================================

Combined use-case

$ cd my-dbt-project

$ dbt run  # Materialize models
$ dbt docs generate  # Generate catalog.json and manifest.json
$ dbt-coverage compute doc --cov-report before.json --cov-fail-under 0.5  # Fail if coverage is lower than 50%

# Make changes to the dbt project, e.g. add some columns to the DWH, document some columns, etc.

$ dbt run  # Materialize the changed models
$ dbt docs generate  # Generate catalog.json and manifest.json
$ dbt-coverage compute doc --cov-report after.json --cov-fail-compare before.json  # Fail if the current coverage is lower than coverage in before.json
$ dbt-coverage compare after.json before.json  # Generate a detailed coverage delta report

Related packages

License

Licensed under the MIT license (see LICENSE.md file for more details).

FOSSA Status

Comments
  • Allow model filtering

    Allow model filtering

    Issue

    • Closes https://github.com/slidoapp/dbt-coverage/issues/24:

    "Please add options in the CLI to include and exclude models to filter out the checks in some of the models or a path."

    Summary

    I added the ability to perform compute commands on only a subset of tables by adding a --model-path-filter option. This means that a subset of models can be selected based upon their original_file_path value (taken from the manifest.json artifact).

    This functionality means that dbt-coverage can now be used in monolithic dbt projects which contain sub-projects owned by different teams. Before adding model selection functionality, using dbt-coverage would not have been useful/advisable in such a structure because another, unrelated team may decrease the overall coverage, which can then block PR merging (should dbt-coverage have been integrated as part of a CI/CD pipeline for example).

    • See example of added functionality from updated README.md below:
    $ cd jaffle_shop
    $ dbt run  # Materialize models
    $ dbt docs generate  # Generate catalog.json and manifest.json
    $ dbt-coverage compute doc --cov-report coverage-doc.json --model-path-filter models/staging/  # Compute doc coverage for a subset of tables, print it and write it to coverage-doc.json file
    
    Coverage report
    ======================================================
    jaffle_shop.stg_customers              0/3       0.0%
    jaffle_shop.stg_orders                 0/4       0.0%
    jaffle_shop.stg_payments               0/4       0.0%
    ======================================================
    Total                                  0/11      0.0%
    
    $ dbt-coverage compute doc --cov-report coverage-doc.json --model-path-filter models/orders.sql --model-path-filter models/staging/  # Compute doc coverage for a subset of tables, print it and write it to coverage-doc.json file
    
    Coverage report
    ======================================================
    jaffle_shop.orders                     0/9       0.0%
    jaffle_shop.stg_customers              0/3       0.0%
    jaffle_shop.stg_orders                 0/4       0.0%
    jaffle_shop.stg_payments               0/4       0.0%
    ======================================================
    Total                                  0/20      0.0%
    

    Note: this is a relatively 'rough' solution and there are likely many improvements that could be made to my code / far more elegant implementations that would achieve the same functionality. Please, feel free to suggest changes!

    Testing

    • I have tested these changes on dbt's jaffle_shop 'testing project' and have not encountered issues so far.
    opened by followingell 5
  • Custom run artifacts path

    Custom run artifacts path

    Goal

    Give more flexibility to user who have their manifest.json & catalog.json files not stored in /target, which is mainly the case for user using dbt cloud (need to retrieve the file by api call), adding a new parameter to use a custom path allow user not to create an unnecessary targetdir in their cicd pipeline

    Description

    • Add run_artifacts_dir : Path parameter to load_manifest and load_catalog
    • Add optional run_artifacts_dir parameter to cli
    opened by Fszta 5
  • Support multiple coverage doc/test save formats

    Support multiple coverage doc/test save formats

    From my understanding we can only save .json files as a result of dbt-coverage compute.

    It would be nice if we could pick other formats, for example .csv That way we can put those files as seeds and write to the db.

    opened by rloredo 5
  • dbt 1.0

    dbt 1.0

    • Support dbt 1.0 and drop support for dbt<1
    • Warn when using with unsupported manifest.json version

    @mrshu, I acknowledged the changes in the new v4 manifest.json format introduced by dbt 1.0 and by that the support for v3 dropped. Since the difference between the formats is only tiny, do you think it would make sense to support also v3 manifest.json (used in dbt<1) by having 2 methods for parsing the manifest, one for v3 and one for v4?

    Closes #26.

    opened by sweco 3
  • Tabulate doc coverage output using markdown

    Tabulate doc coverage output using markdown

    Resolves #46

    What has changed?

    Introduces a new argument called cov_format that accepts string (default) or markdown for the compute doc command.

    Outputs the results to the terminal in either the default string format, or in a markdown friendly format

    Example

    dbt-coverage compute doc --cov-format markdown
    

    image

    results in:

    image

    opened by vvvito 2
  • Make error more informative if target/manifest.json doesn't exist

    Make error more informative if target/manifest.json doesn't exist

    Planning to submit a PR, just wanted to tee this up first to document the need

    Steps to reproduce:

    pip install dbt-coverage==0.2.2
    git clone https://github.com/dbt-labs/jaffle_shop
    git checkout b0b77aa
    dbt-coverage compute doc
    

    Line of code throwing the error: https://github.com/slidoapp/dbt-coverage/blob/9f97e6c8fc24e83babc51c2b0584afc88e45308c/dbt_coverage/init.py#L507

    Error message:

    INFO:root:Loading catalog and manifest files from project dir: .
    Traceback (most recent call last):
      File "/Users/brentbrewington/.pyenv/versions/3.9.0/bin/dbt-coverage", line 8, in <module>
        sys.exit(app())
      File "/Users/brentbrewington/.pyenv/versions/3.9.0/lib/python3.9/site-packages/typer/main.py", line 214, in __call__
        return get_command(self)(*args, **kwargs)
      File "/Users/brentbrewington/.pyenv/versions/3.9.0/lib/python3.9/site-packages/click/core.py", line 1130, in __call__
        return self.main(*args, **kwargs)
      File "/Users/brentbrewington/.pyenv/versions/3.9.0/lib/python3.9/site-packages/click/core.py", line 1055, in main
        rv = self.invoke(ctx)
      File "/Users/brentbrewington/.pyenv/versions/3.9.0/lib/python3.9/site-packages/click/core.py", line 1657, in invoke
        return _process_result(sub_ctx.command.invoke(sub_ctx))
      File "/Users/brentbrewington/.pyenv/versions/3.9.0/lib/python3.9/site-packages/click/core.py", line 1404, in invoke
        return ctx.invoke(self.callback, **ctx.params)
      File "/Users/brentbrewington/.pyenv/versions/3.9.0/lib/python3.9/site-packages/click/core.py", line 760, in invoke
        return __callback(*args, **kwargs)
      File "/Users/brentbrewington/.pyenv/versions/3.9.0/lib/python3.9/site-packages/typer/main.py", line 532, in wrapper
        return callback(**use_params)  # type: ignore
      File "/Users/brentbrewington/.pyenv/versions/3.9.0/lib/python3.9/site-packages/dbt_coverage/__init__.py", line 660, in compute
        return do_compute(project_dir, cov_report, cov_type, cov_fail_under, cov_fail_compare)
      File "/Users/brentbrewington/.pyenv/versions/3.9.0/lib/python3.9/site-packages/dbt_coverage/__init__.py", line 616, in do_compute
        catalog = load_files(project_dir)
      File "/Users/brentbrewington/.pyenv/versions/3.9.0/lib/python3.9/site-packages/dbt_coverage/__init__.py", line 533, in load_files
        catalog = load_catalog(project_dir)
      File "/Users/brentbrewington/.pyenv/versions/3.9.0/lib/python3.9/site-packages/dbt_coverage/__init__.py", line 507, in load_catalog
        with open(project_dir / 'target/catalog.json') as f:
    FileNotFoundError: [Errno 2] No such file or directory: 'target/catalog.json'
    
    opened by bbrewington 2
  • Add support for Manifest v6

    Add support for Manifest v6

    opened by reegs 2
  • Take into account seeds

    Take into account seeds

    It seems that seeds are not considered while reading manifest.json. As a result, the following error pops while running dbt-coverage compute doc

    (venv_dbt) ➜  myCodeFolder git:(master) dbt-coverage compute doc                                
    INFO:root:Loading catalog and manifest files from project dir: .
    INFO:root:Successfully loaded 200 tables from catalog
    Traceback (most recent call last):
      File "/Users/victor/myCodeFolder/venv_dbt/bin/dbt-coverage", line 8, in <module>
        sys.exit(app())
      File "/Users/victor/myCodeFolder/venv_dbt/lib/python3.8/site-packages/typer/main.py", line 214, in __call__
        return get_command(self)(*args, **kwargs)
      File "/Users/victor/myCodeFolder/venv_dbt/lib/python3.8/site-packages/click/core.py", line 1137, in __call__
        return self.main(*args, **kwargs)
      File "/Users/victor/myCodeFolder/venv_dbt/lib/python3.8/site-packages/click/core.py", line 1062, in main
        rv = self.invoke(ctx)
      File "/Users/victor/myCodeFolder/venv_dbt/lib/python3.8/site-packages/click/core.py", line 1668, in invoke
        return _process_result(sub_ctx.command.invoke(sub_ctx))
      File "/Users/victor/myCodeFolder/venv_dbt/lib/python3.8/site-packages/click/core.py", line 1404, in invoke
        return ctx.invoke(self.callback, **ctx.params)
      File "/Users/victor/myCodeFolder/venv_dbt/lib/python3.8/site-packages/click/core.py", line 763, in invoke
        return __callback(*args, **kwargs)
      File "/Users/victor/myCodeFolder/venv_dbt/lib/python3.8/site-packages/typer/main.py", line 500, in wrapper
        return callback(**use_params)  # type: ignore
      File "/Users/victor/myCodeFolder/venv_dbt/lib/python3.8/site-packages/dbt_coverage/__init__.py", line 604, in compute
        do_compute(project_dir, cov_report, cov_type, cov_fail_under, cov_fail_compare)
      File "/Users/victor/myCodeFolder/venv_dbt/lib/python3.8/site-packages/dbt_coverage/__init__.py", line 562, in do_compute
        catalog = load_files(project_dir)
      File "/Users/victor/myCodeFolder/venv_dbt/lib/python3.8/site-packages/dbt_coverage/__init__.py", line 486, in load_files
        manifest = load_manifest(project_dir)
      File "/Users/victor/myCodeFolder/venv_dbt/lib/python3.8/site-packages/dbt_coverage/__init__.py", line 477, in load_manifest
        manifest = Manifest.from_nodes(manifest_nodes)
      File "/Users/victor/myCodeFolder/venv_dbt/lib/python3.8/site-packages/dbt_coverage/__init__.py", line 121, in from_nodes
        table_name = id_to_table_name[table_id]
    KeyError: 'seed.dbt_project.browsers'
    

    Versions:

    dbt-core==0.21.0
    dbt-coverage==0.1.7
    

    Python 3.8

    opened by vdeb 2
  • KeyError: 'column_name' ?

    KeyError: 'column_name' ?

    # dbt build
    ...
    Done. PASS=23 WARN=0 ERROR=1 SKIP=1 TOTAL=25
    # dbt docs generate
    Running with dbt=0.21.1-rc1
    ...
    $ pip install dbt-coverage
    Successfully installed dbt-coverage-0.1.7 typer-0.4.0
    $ dbt-coverage compute doc 
    INFO:root:Loading catalog and manifest files from project dir: .
    INFO:root:Successfully loaded 17 tables from catalog
    
        catalog = load_files(project_dir)
      File "/root/.local/share/virtualenvs/data-pipelines-kwiw1yve/lib/python3.8/site-packages/dbt_coverage/__init__.py", line 486, in load_files
        manifest = load_manifest(project_dir)
      File "/root/.local/share/virtualenvs/data-pipelines-kwiw1yve/lib/python3.8/site-packages/dbt_coverage/__init__.py", line 477, in load_manifest
        manifest = Manifest.from_nodes(manifest_nodes)
      File "/root/.local/share/virtualenvs/data-pipelines-kwiw1yve/lib/python3.8/site-packages/dbt_coverage/__init__.py", line 122, in from_nodes
        column_name = node['column_name'] or node['test_metadata']['kwargs']['column_name'] \
    KeyError: 'column_name'
    
    opened by chapmanjacobd 2
  • Fix matching of entities between manifest and catalog

    Fix matching of entities between manifest and catalog

    This fixes two issues:

    1. Some systems (e.g. Snowflake) uppercase all table and column names. In accordance with the dbt-docs official source code, we lowercase all table and column names before matching them.
    2. There can be multiple tables with the same name but in different schemas. We therefore use schema_name.table_name to distinguish tables.
    opened by sweco 1
  • Add contents

    Add contents

    What was done

    • Copy contents of dbt_docs_coverage.py verbatim to __init__.py.
    • Add README with basic documentation.
    • Create Python package using poetry.
    • Register dbt_coverage.__init__:app as a CLI script with name dbt-coverage.

    Review objectives

    • Please review README.
    • Please review the poetry files.
    • Ignore __init__.py, it was copied verbatim.
    opened by sweco 1
  • Display coverage doc/test result on gitlab badges

    Display coverage doc/test result on gitlab badges

    Hi team,

    I am wondering whether it is already possible for me to display the coverage doc/test result as a percentage on gitlab badges. Please see similar example for python unit test below image

    Thank you

    opened by kelvinheng92 3
  • Clean up `len(model_path_filter) >= 1` once a Typer issue gets resolved

    Clean up `len(model_path_filter) >= 1` once a Typer issue gets resolved

    Originally I had used is not None as you say, however, this doesn't work due to a known issue with typer: https://github.com/tiangolo/typer/issues/410 whereby [an] "Empty tuple [is] returned for Option instead of expected None when no values passed". As such, I have to use len() since until the issue is resolved model_path_filter will never be None.

    ~~Since you asked the question it's obvious that I could make the reason for my choice clearer, WDYT to the below suggestion?~~

    Originally posted by @followingell in https://github.com/slidoapp/dbt-coverage/pull/45#discussion_r1012694530

    good first issue 
    opened by mrshu 2
  • Support name / identifier format within sources

    Support name / identifier format within sources

    The script does not currently recognize tests applied to a source that is set-up using an identifier rather than just the raw as being associated with the source.

    I think, but am not certain, that this might be related to #30

    opened by reegs 1
  • could you add ability produce a coverage report in cobertura format

    could you add ability produce a coverage report in cobertura format

    To be able to retrieve dbt-coverage output in gitlab, a cobertura format is needed.

    See: https://docs.gitlab.com/ee/ci/yaml/artifacts_reports.html#artifactsreportscoverage_report

    Could you add a command line argument to choose output format ? (json or xml-cobertura)

    opened by cedric-orange 1
  • support generic tests from dbt_utils when calculating coverage

    support generic tests from dbt_utils when calculating coverage

    Our project is relying on tests defined in dbt_utils - is there a way of adding support for these as well?

    https://github.com/dbt-labs/dbt-utils#unique_combination_of_columns-source

    opened by aa-matthias 5
Releases(v0.3.2)
Owner
Slido
Slido is an audience interaction platform for meetings and events. Public repositories
Slido
A command-line utility that creates projects from cookiecutters (project templates), e.g. Python package projects, VueJS projects.

Cookiecutter A command-line utility that creates projects from cookiecutters (project templates), e.g. creating a Python package project from a Python

null 18.6k Jan 2, 2023
A data engineering project with Kafka, Spark Streaming, dbt, Docker, Airflow, Terraform, GCP and much more!

Streamify A data pipeline with Kafka, Spark Streaming, dbt, Docker, Airflow, Terraform, GCP and much more! Description Objective The project will stre

Ankur Chavda 206 Dec 30, 2022
dbt adapter for Firebolt

dbt-firebolt dbt adapter for Firebolt dbt-firebolt supports dbt 0.21 and newer Installation First, download the JDBC driver and place it wherever you'

null 23 Dec 14, 2022
An Airflow operator to call the main function from the dbt-core Python package

airflow-dbt-python An Airflow operator to call the main function from the dbt-core Python package Motivation Airflow running in a managed environment

Tomás Farías Santana 93 Jan 8, 2023
Model synchronization from dbt to Metabase.

dbt-metabase Model synchronization from dbt to Metabase. If dbt is your source of truth for database schemas and you use Metabase as your analytics to

Mike Gouline 270 Jan 8, 2023
dbt (data build tool) adapter for Oracle Autonomous Database

dbt-oracle version 1.0.0 dbt (data build tool) adapter for the Oracle database. dbt "adapters" are responsible for adapting dbt's functionality to a g

Oracle 22 Nov 15, 2022
Start and stop your NiceHash miners using this script.

NiceHash Mining Scheduler Use this script to schedule your NiceHash Miner(s). Electricity costs between 4-9pm are high in my area and I want NiceHash

SeaRoth 2 Sep 30, 2022
Stopmagic gives you the power of creating amazing Stop Motion animations faster and easier than ever before.

Stopmagic gives you the power of creating amazing Stop Motion animations faster and easier than ever before. This project is maintained by Aldrin Mathew.

Aldrin's Art Factory 67 Dec 31, 2022
Beginner Projects A couple of beginner projects here

Beginner Projects A couple of beginner projects here, listed from easiest to hardest :) selector.py: simply a random selector to tell me who to faceti

Kylie 272 Jan 7, 2023
An AI-powered device to stop people from stealing my packages.

Package Theft Prevention Device An AI-powered device to stop people from stealing my packages. Installation To install on a raspberry pi, clone the re

rydercalmdown 157 Nov 24, 2022
Vaccine for STOP/DJVU ransomware, prevents encryption

STOP/DJVU Ransomware Vaccine Prevents STOP/DJVU Ransomware from encrypting your files. This tool does not prevent the infection itself. STOP ransomwar

Karsten Hahn 16 May 31, 2022
Stop ask your soraka to ult you, just ult yourself

Lollo's auto-ultimate script Are you tired of your low elo friend who can't ult you with soraka when you ask for it? Use Useless Support and just ult

null 9 Oct 20, 2022
A place where one-off ideas/partial projects can live comfortably

A place to post ideas, partial projects, or anything else that doesn't necessarily warrant its own repo, from my mind to the web.

Carson Scott 2 Feb 25, 2022
Ahmed Hossam 12 Oct 17, 2022
Inspect the resources of your android projects and understand which ones are not being used and could potentially be removed.

Android Resources Checker What This program will inspect the resources of your app and help you understand which ones are not being used and could pot

Fábio Carballo 39 Feb 8, 2022
Data Structures and Algorithms Python - Practice data structures and algorithms in python with few small projects

Data Structures and Algorithms All the essential resources and template code nee

Hesham 13 Dec 1, 2022
A python script to run any executable and pass test cases to it's stdin and compare stdout with correct output.

quera_testcase_checker A python script to run any executable and pass test cases to it's stdin and compare stdout with correct output. proper way to u

k3y1 1 Nov 15, 2021
Repo created for the purpose of adding any kind of programs and projects

Programs and Project Repository A repository for adding programs and projects of any kind starting from beginners level to expert ones Contributing to

Unicorn Dev Community 3 Nov 2, 2022