Write maintainable, production-ready pipelines using Jupyter or your favorite text editor. Develop locally, deploy to the cloud. ☁️

Describe your changes

make font size dynamic based on width of the the tasks/circles in the dag plot

Issue ticket number and link

Closes #903

Checklist before requesting a review

• [x] I have performed a self-review of my code
• [x] I have added thorough tests (when necessary).
• [x] I have added the right documentation (when needed). Product update? If yes, write one line about this update.

ploomber report --> gives me an error , ValueError: Error when using d3 backend: expected a path with extension .html, but got: '/tmp/tmpmojk06x4.png', please change the extension

Describe your changes

added and tested did you mean feature to ploomber examples issue #805

Issue ticket number and link

Closes #x

Checklist before requesting a review

• [x] I have performed a self-review of my code
• [x] I have added thorough tests (when necessary).
• [x] I have added the right documentation (when needed). Product update? If yes, write one line about this update.

papermill shows a logging message if black is not installed, which is causing confusion for ploomber users.

https://github.com/nteract/papermill/blob/aecd0ac16d34dd41a1a8e4ca3351844932af3d94/papermill/translators.py#L195

we should suppress the message

I see the following error when I do ploomber build -p train, but do not get the error when running ploomber build. My guess it that it relates to a joblib.dump. I tried to add custom un/serializers after seeing the error, but that did not fix it. Sorry, the pipeline is a bit complicated at this point and the error is hard to generate a easy to reproduce example for testing.

Traceback (most recent call last):
  File "/home/hopkir/anaconda3/envs/hydrates/lib/python3.8/site-packages/ploomber/cli/io.py", line 34, in wrapper
    fn(**kwargs)
  File "/home/hopkir/anaconda3/envs/hydrates/lib/python3.8/site-packages/ploomber/cli/build.py", line 49, in main
    report = dag.build_partially(args.partially,
  File "/home/hopkir/anaconda3/envs/hydrates/lib/python3.8/site-packages/ploomber/dag/dag.py", line 667, in build_partially
    dag_copy = deepcopy(self)
  File "/home/hopkir/anaconda3/envs/hydrates/lib/python3.8/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/home/hopkir/anaconda3/envs/hydrates/lib/python3.8/copy.py", line 270, in _reconstruct
    state = deepcopy(state, memo)
  File "/home/hopkir/anaconda3/envs/hydrates/lib/python3.8/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/home/hopkir/anaconda3/envs/hydrates/lib/python3.8/copy.py", line 230, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/home/hopkir/anaconda3/envs/hydrates/lib/python3.8/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/home/hopkir/anaconda3/envs/hydrates/lib/python3.8/copy.py", line 270, in _reconstruct
    state = deepcopy(state, memo)
  File "/home/hopkir/anaconda3/envs/hydrates/lib/python3.8/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/home/hopkir/anaconda3/envs/hydrates/lib/python3.8/copy.py", line 230, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/home/hopkir/anaconda3/envs/hydrates/lib/python3.8/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/home/hopkir/anaconda3/envs/hydrates/lib/python3.8/copy.py", line 230, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/home/hopkir/anaconda3/envs/hydrates/lib/python3.8/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/home/hopkir/anaconda3/envs/hydrates/lib/python3.8/copy.py", line 230, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/home/hopkir/anaconda3/envs/hydrates/lib/python3.8/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/home/hopkir/anaconda3/envs/hydrates/lib/python3.8/copy.py", line 270, in _reconstruct
    state = deepcopy(state, memo)
  File "/home/hopkir/anaconda3/envs/hydrates/lib/python3.8/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/home/hopkir/anaconda3/envs/hydrates/lib/python3.8/copy.py", line 230, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/home/hopkir/anaconda3/envs/hydrates/lib/python3.8/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/home/hopkir/anaconda3/envs/hydrates/lib/python3.8/copy.py", line 270, in _reconstruct
    state = deepcopy(state, memo)
  File "/home/hopkir/anaconda3/envs/hydrates/lib/python3.8/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/home/hopkir/anaconda3/envs/hydrates/lib/python3.8/copy.py", line 230, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/home/hopkir/anaconda3/envs/hydrates/lib/python3.8/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/home/hopkir/anaconda3/envs/hydrates/lib/python3.8/copy.py", line 270, in _reconstruct
    state = deepcopy(state, memo)
  File "/home/hopkir/anaconda3/envs/hydrates/lib/python3.8/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/home/hopkir/anaconda3/envs/hydrates/lib/python3.8/copy.py", line 230, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/home/hopkir/anaconda3/envs/hydrates/lib/python3.8/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/home/hopkir/anaconda3/envs/hydrates/lib/python3.8/copy.py", line 270, in _reconstruct
    state = deepcopy(state, memo)
  File "/home/hopkir/anaconda3/envs/hydrates/lib/python3.8/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/home/hopkir/anaconda3/envs/hydrates/lib/python3.8/copy.py", line 230, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/home/hopkir/anaconda3/envs/hydrates/lib/python3.8/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/home/hopkir/anaconda3/envs/hydrates/lib/python3.8/copy.py", line 270, in _reconstruct
    state = deepcopy(state, memo)
  File "/home/hopkir/anaconda3/envs/hydrates/lib/python3.8/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/home/hopkir/anaconda3/envs/hydrates/lib/python3.8/copy.py", line 230, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/home/hopkir/anaconda3/envs/hydrates/lib/python3.8/copy.py", line 161, in deepcopy
    rv = reductor(4)
TypeError: cannot pickle 'module' object

Documentation page: community/index

Use case

I work in a business where we have an abundance of data but not necessarily the best data for data science efforts. In may cases we have millions of time series records for feature building but poor records of the target data we'd like to predict. Further complicating efforts are our many many different database servers of various type (oracle, sql, teradata, aws, azure).

Because of this I spend a lot of time helping our business become 'data science ready' by developing data pipelines to build useful datasets. I'm currently using python to integrate and transform disparate data sources and outputting to table

Request

I would like to be able to pass unique values from one .sql task to another as parameters. Something like this:

tasks:
  - name: get-data-1
    source: sql/first_data.sql
    product: '{{data}}/raw/first_data.parquet'
    client: src.clients.src1
    chunksize: null
  
  - name: get-data-2
    source: sql/second-data.sql
    product: '{{data}}/raw/second-data.parquet'
    client: src.clients.src2
    params:
      startYear: '{{startYear}}'
      endYear: '{{endYear}}'      
    upstream: get-data-1
    chunksize: null

second-data-2.sql example

select *
from schema.table
where column in (upstream[params])
and year(date) between {{startYear}} and {{endYear}}

Alternatively, could I output the params via python as an intermediate .yaml and use as inputs like this?

import pandas as pd
import yaml

df = pd.read_parquet(upstream['first_data.parquet''])
vals = df['column'].unique()

d = dict({'val':vals})

with open("sample.yaml", "w") as f:
  yaml.dump(d, f)
  f.close()

I have a notebook folder in ploomber project and ploomber tries to run something so I can't even load the notebook (which BTW is not referenced in pipeline.yaml in any way)

What happened? How is it possible that ploomber breaks jupyter?

Error message

[I 11:59:22.671 NotebookApp] [Ploomber] Requested model: work/wwf/notebooks/Enrich_Test.ipynb. Looking for DAG with root dir: /home/kuba/Projects                            
[E 11:59:22.786 NotebookApp] Uncaught exception GET /api/contents/work/wwf/notebooks/Enrich_Test.ipynb?type=notebook&_=1656064762442 (127.0.0.1)                             
    HTTPServerRequest(protocol='http', host='localhost:8888', method='GET', uri='/api/contents/work/wwf/notebooks/Enrich_Test.ipynb?type=notebook&_=1656064762442', version='
HTTP/1.1', remote_ip='127.0.0.1')                                                                                                                                            
    Traceback (most recent call last):                                                                                                                                       
      File "/home/kuba/.local/lib/python3.8/site-packages/tornado/web.py", line 1704, in _execute                                                                            
        result = await result                                                                                                                                                
      File "/home/kuba/.local/lib/python3.8/site-packages/tornado/gen.py", line 234, in wrapper                                                                              
        yielded = ctx_run(next, result)                                                                                                                                      
      File "/home/kuba/.local/lib/python3.8/site-packages/notebook/services/contents/handlers.py", line 118, in get                                                          
        model = yield maybe_future(self.contents_manager.get(                  
      File "/home/kuba/.local/lib/python3.8/site-packages/ploomber/jupyter/manager.py", line 267, in get                                                                     
        self.load_dag(                                                                                                                                                       
      File "/home/kuba/.local/lib/python3.8/site-packages/ploomber/jupyter/manager.py", line 193, in load_dag                                                                
        pairs = [(resolve_path(                                                       
      File "/home/kuba/.local/lib/python3.8/site-packages/ploomber/jupyter/manager.py", line 196, in <listcomp>                                                                      if t.source.loc is not None]                                                  
      File "/home/kuba/.local/lib/python3.8/site-packages/ploomber/sources/pythoncallablesource.py", line 130, in loc                                                        
        self._loc = self._callable_loader.get_loc()                                                                                                                          
      File "/home/kuba/.local/lib/python3.8/site-packages/ploomber/sources/pythoncallablesource.py", line 58, in get_loc                                                     
        loc, _ = lazily_locate_dotted_path(self._primitive)                                                                                                                        File "/home/kuba/.local/lib/python3.8/site-packages/ploomber/util/dotted_path.py", line 343, in lazily_locate_dotted_path                                              
        raise ModuleNotFoundError('Error processing dotted '
    ModuleNotFoundError: Error processing dotted path 'deepsense_wwf.data_utils.copy_zipped_data': 'deepsense_wwf' appears to be a namespace package, which are not supported
[W 11:59:22.790 NotebookApp] Unhandled error          
[E 11:59:22.791 NotebookApp] {                                                                                                                                               
      "Host": "localhost:8888",                                                                                                                                              
      "Accept": "application/json, text/javascript, */*; q=0.01",                                                                                                            
      "Referer": "http://localhost:8888/notebooks/work/wwf/notebooks/Enrich_Test.ipynb",                                                                                     
      "User-Agent": "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:101.0) Gecko/20100101 Firefox/101.0"                                                                         
    }                                                          

We have a new cookbook section: https://github.com/ploomber/projects/tree/master/cookbook

where we add minimal examples for certain use cases, we should add links to each one on the relevant sections in the docs

Describe your changes

updated documentation to reflect python 3.6 not being supported removed workarounds for python 3.6

Issue ticket number and link

Closes #819

Checklist before requesting a review

• [x] I have performed a self-review of my code
• [x] I have added thorough tests (when necessary).
• [x] I have added the right documentation (when needed). Product update? If yes, write one line about this update.

When opening in a large monitor, the docs display a right bar to move between sections (example):

However, on tablets and phones, we remove that bar. It'd be better to add a button to the top bar to toggle it, just like we do with the left bar (see right button):

source code: https://github.com/ploomber/ploomber/blob/76b2abf78092e6696d9e25e47cefdc1e1589529f/doc/_templates/layout.html#L41

check doc/contributing.md for instructions

We won't be supporting Python 3.6 beginning in Ploomber 0.20

There's a PR that removes everything related to it: https://github.com/ploomber/ploomber/pull/876

But we haven't sent any warnings to users

we should add one here: https://github.com/ploomber/ploomber/blob/master/src/ploomber/init.py

essentially, if we detect they're running 3.6, we should display a message

that should be released in the next 0.19 release. on 0.20, we merge https://github.com/ploomber/ploomber/pull/876 and remove the warning

pygments 2.14 is breaking our CI. I added a quickfix since I verified that this failure is not affecting anything critical (it's just terminal output formatting). I think this is best than pinning pygments since pining the version is worst as it'll increase the complexity of installing the package.

This is some ASCII code stuff, and I'm unsure why the pygments update is behaving in this way, here's the error:

===================================================================================== FAILURES ======================================================================================
________________________________________________________________ test_code_highlight[with markup and code_highlight] ________________________________________________________________

has_markup = True, code_highlight = True, expected = '{kw}assert{hl-reset} {number}0{hl-reset}\n'
color_mapping = <class 'io_mod.test_terminalwriter.color_mapping.<locals>.ColorMapping'>

    @pytest.mark.parametrize(
        ("has_markup", "code_highlight", "expected"),
        [
            pytest.param(
                True,
                True,
                "{kw}assert{hl-reset} {number}0{hl-reset}\n",
                id="with markup and code_highlight",
            ),
            pytest.param(
                True,
                False,
                "assert 0\n",
                id="with markup but no code_highlight",
            ),
            pytest.param(
                False,
                True,
                "assert 0\n",
                id="without markup but with code_highlight",
            ),
            pytest.param(
                False,
                False,
                "assert 0\n",
                id="neither markup nor code_highlight",
            ),
        ],
    )
    def test_code_highlight(has_markup, code_highlight, expected, color_mapping):
        f = io.StringIO()
        tw = terminalwriter.TerminalWriter(f)
        tw.hasmarkup = has_markup
        tw.code_highlight = code_highlight
        tw._write_source(["assert 0"], lexer='py')

>       assert f.getvalue().splitlines(keepends=True) == color_mapping.format(
            [expected])
E       AssertionError: assert ['\x1b[94mass...[39;49;00m\n'] == ['\x1b[94mass...[39;49;00m\n']
E         At index 0 diff: '\x1b[94massert\x1b[39;49;00m \x1b[94m0\x1b[39;49;00m\x1b[90m\x1b[39;49;00m\n' != '\x1b[94massert\x1b[39;49;00m \x1b[94m0\x1b[39;49;00m\n'
E         Use -v to get more diff

Describe your changes

Issue ticket number and link

Closes #701

Checklist before requesting a review

• [x] I have performed a self-review of my code
• [x] I have added thorough tests (when necessary).
• [x] I have added the right documentation (when needed). Product update? If yes, write one line about this update.

I got a report from a cloud user that they had a "malformed API key error", somewhere in our docs, we have the following instructions:

ploomber cloud set-key {your-key}

And the user was setting it with the {} characters. We should detect if the passed key begins with { and ends with }. If so, ignore those characters (or maybe show an error telling them to remove them?)

Context: Currently, we will send the user a welcome message when user joins the Ploomber Slack Workspace. We may consider to provide the list of our current links.

Example - Ideal After:

Example - Currently Implementation:

Action Item:

• [ ] Discuss if providing those links will help users to find the resource easier
  • What to include in the list? Currently we can have Ploomber.io, Github link, Doc link, Blog link
• [ ] Modify the bot message with the new adding section

Hi!

I am a recent adopter of Ploomber and enjoying it. Thanks a lot for making this framework.

I am using grid in my pipeline.yaml and it works great. The only downside is that when visualizing the DAG with pipeline plot all nodes are included (and if there are enough nodes the html show nothing).

However, since nodes made by a grid step are related I would like them to appear as a single node in the DAG. Would it be possible to make an option for pipeline plot such that the user can choose whether or not to collapse grid nodes?