Meltano: ELT for the DataOps era. Meltano is open source, self-hosted, CLI-first, debuggable, and extensible.

Related tags

Data Analysis meltano
Overview

Meltano: ELT for the DataOps era

pipeline status

Meltano is open source, self-hosted, CLI-first, debuggable, and extensible.

Pipelines are code, ready to be version controlled, containerized, and deployed continuously. Develop and test locally, then deploy in production along with the built-in Airflow integration, or inside your orchestrator of choice.

Meltano embraces the Singer standard and its community-maintained library of open source extractors and loaders, and leverages dbt for transformation.

Documentation

Check out the "Getting Started" guide or find the full documentation at https://www.meltano.com/docs/.

Contributing to Meltano

We welcome contributions and improvements, please see the contribution guidelines

Responsible Disclosure Policy

Please refer to the responsible disclosure policy on our website.

License

This code is distributed under the MIT license, see the LICENSE file.

Comments
  • Python-based Plugin Architecture

    Python-based Plugin Architecture

    I think we have a path forward on python plugin architecture (https://gitlab.com/groups/meltano/-/epics/134#note_981098666) which does not require modifying Meltano itself (at least not in the first iterations).

    Since pip_url for any repo can include multiple libraries, we can extend any plugins' capabilities by adding a meltano-managed python tool to any plugin that needs capabilities. Then those capabilities just need to registered as CLI-executable and be mapped to commands and capabilities in the Meltano plugin definition.

    Example: registering dbt packages in packages.yml

    Today we have custom code in Meltano that adds entries to packages.yml for dbt during meltano add transform .... We could replace it as follows:

    1. Create a new python tool called meltano-dbt-helpers with a registered CLI command meltano-dbt-util add package.
    2. Add that package to the pipurl for all dbt plugins. E.g. pip_url: dbt-snowflake meltano-dbt-helpers.
    3. Register commands in the package that leverage the helper. E.g. 
      commands:
 add-dep:
   command: meltano-dbt-util
   args: add package

    Then from the CLI as a user:

    meltano run dbt:add-dep dbt-util

    Would add dbt-util to the package.

    For "smarter" integrations, we just need a contract that pairs capabilities like service with expected commands, like ui, up, and down.

    Comparison with other approaches

    The best part of this approach is that it doesn't require any code changes in Meltano - we and community members could implement this today with existing features.

    roadmap kind/Feature valuestream/Meltano 
    opened by aaronsteers 33
  • feat: Add systemdb support for MS SQL Server

    feat: Add systemdb support for MS SQL Server

    Relates to issue #3238.

    The main improvement is limiting all String types in the alembic migrations. This allows the possibility to use both MSSQL and MYSQL as a Meltano backend database.

    Both databases were manually tested. It might be valuable to add automatic tests to the Github actions.

    To use MYSQL run: pip install mysqlclient To use MSSQL run: pip install pyodbc

    Community-Contributed PR 
    opened by JulesHuisman 32
  • fix: symbolic link creation for meltano.exe when on windows

    fix: symbolic link creation for meltano.exe when on windows

    This fixes the bug #6467 where on Windows meltano upgrade stops because MeltanoInvoker.invoke() is calling subprocess.run() which requires a executable target. Since the symbolic link .meltano\run\bin is targeting meltano not meltano.exe a failure occurs. I updated the Project.activate() function to target meltano.exe when Windows is detected.

    This does not fix any previously created symbolic links that might exist in projects.

    Close #6467

    Community-Contributed PR Windows kind/Bug valuestream/Meltano 
    opened by BuzzCutNorman 19
  • Create `cron` utility extension

    Create `cron` utility extension

    Migrated from GitLab: https://gitlab.com/meltano/meltano/-/issues/3522

    Originally created by @alxthm on 2022-05-24 14:54:47

    For simple meltano deployments, it would be great to have a cron-based orchestrator using the meltano schedule configuration, without having to set-up an Airflow instance with a db, users, etc.

    The meltano UI already makes it possible to manually run meltano schedules, so this would be a nice complement :smile:

    kind/Feature valuestream/Meltano migrated from gitlab 
    opened by MeltyBot 19
  • Disallow calling a `mapper` by name in `meltano run` and alert to use a `mapping` name

    Disallow calling a `mapper` by name in `meltano run` and alert to use a `mapping` name

    The document misses a step for mapper configuration. Property mapping_name has to be added to the mappers extras, and its value need to match with the name under mappings. It doesn't seem been documented anywhere.

    plugins:
  mappers:
  - name: transform-field
    variant: transferwise
    pip_url: pipelinewise-transform-field
    executable: transform-field
    mapping_name: transform-field-hash
    mappings:
    - name: transform-field-hash
      config:
        transformations:
          - field_id: Name
            tap_stream_name: dbo-Test
            type: MASK-HIDDEN
    Documentation Accepting Pull Requests User Support kind/Bug valuestream/Meltano plugin/Singer 
    opened by lidong-zhao 18
  • bug: Meltano Upgrade on Windows stops with error

    bug: Meltano Upgrade on Windows stops with error "is not a valid Win32 application"

    Meltano Version

    2.3.0

    Python Version

    3.9

    Bug scope

    CLI (options, error messages, logging, etc.)

    Operating System

    Windows Server 2022 Standard

    Description

    I have noticed on Windows if the file(symbolic link) .meltano/run/bin is present in your project when you run the upgrade via meltano upgrade or the melatno ui "Upgrade Button" you will get the following WinError.

    [WinError 193] %1 is not a valid Win32 application

    if you remove the file and run meltano upgrade the upgrade will complete. I am not sure what the long term ramification of deleting the file is.

    image

    Code

    No response

    User Support Windows kind/Bug valuestream/Meltano 
    opened by BuzzCutNorman 17
  • New process and guidelines for integration tests (replaces `demo-project` and `homepage_sample_code` tests in GitLab CI

    New process and guidelines for integration tests (replaces `demo-project` and `homepage_sample_code` tests in GitLab CI

    From a call with @tayloramurphy and @DouweM, we started thinking of replacing demo-project with a set of sample Meltano projects in meltano/meltano that can serve as integration tests, perhaps along with specific scripts that would run - such as the invoking of specific plugins.

    I'm sure there are better and more sophisticated approaches, but one approach would be a convention have a meltano.yml, a script.sh, and a meltano.after.yml which should match to the results. Another approach would be a set of pytest tests that perform a series of operations against (a copy of?) the source meltano.yml file and confirm the results (or errors), possibly including specific telemetry events that are expected to be sent over the course of those script commands.

    kind/Tech Debt valuestream/Meltano 
    opened by aaronsteers 17
  • Restore `project_id` from `analytics.json` & improve `analytics.json` handling

    Restore `project_id` from `analytics.json` & improve `analytics.json` handling

    • Restore project_id from analytics.json if it is not found in meltano.yml.
    • Overwrite analytics.json with a new one if it is malformed.
    • Handle more errors when dealing with analytics.json (e.g. JSONDecodeError).
    • Add additional debug logging.
    • Do not update meltano.yml from ProjectContext (n.b. this would cause the project_id key to end up inside of the config of the active environment instead of the top-level config).
    opened by WillDaSilva 17
  • ci: add `pytest` support on windows; change: add `meltano elt` guidance for windows users

    ci: add `pytest` support on windows; change: add `meltano elt` guidance for windows users

    Closes https://github.com/meltano/meltano/issues/5940

    Old MR from Gitlab https://gitlab.com/meltano/meltano/-/merge_requests/2639/diffs

    (Note this has been rebased since this MR)

    Pipeline is running here https://gitlab.com/vischous/meltano/-/merge_requests/3/pipelines

    opened by visch 17
  • bug: Projects without `environments` nor `default_environment` see error where `dev` environment is not found

    bug: Projects without `environments` nor `default_environment` see error where `dev` environment is not found

    Meltano Version

    2.5.0

    Python Version

    NA

    Bug scope

    Configuration (settings parsing, validation, etc.)

    Operating System

    NA

    Description

    Users with projects created from scratch (not using meltano init) or started before Environments where introduced may not be using default_environment nor environments in their meltano.yml.

    This causes an error because in 2.5.0 we made default_environment a project setting that defaults to dev. Any project without a dev environment, as those exemplified above, will see an error because Meltano will be looking for dev.

    Code

    $ meltano install
Environment 'dev' was not found

    This is reproducible with the following meltano.yml:

    plugins:
  loaders:
  - name: target-jsonl
    variant: andyh1203
    pip_url: target-jsonl

    Workaround

    From AJ:

    There should be two ways to work around this issue.

    Running meltano environment add dev as a one-time fix will resolve the issue, even if no other changes are made and no environment-specific settings are added.

    Adding the --no-environment CLI arg will override the default dev environment name.

    Accepting Pull Requests kind/Bug urgency/High valuestream/Meltano 
    opened by edgarrmondragon 16
  • Give `http://melta.no` troubleshooting help link when plugin fails, with CTA to Slack in web copy

    Give `http://melta.no` troubleshooting help link when plugin fails, with CTA to Slack in web copy

    Spec (edited by AJ):

    When Meltano fails, we'll print a standard error message:

    [Error description text.]

    To contact the Meltano team and access troubleshooting help and resources, visit http://melta.no

    Orig. issue description (Douwe)

    When meltano invoke or meltano run fails, a user is more likely to manage to solve the issue and stick with Meltano if they join Slack than if they don't, as evidenced by the number of people who responded to the survey saying the docs weren't always helpful, but the Slack community was. I imagine a good amount of people that try us out and churn would not have if they had known about Slack. That also lets us reach out to them to learn more about their use case and experience, even if they do end up churning.

    So I think we should recommend joining Slack in the product wherever that makes sense.

    @tayloramurphy Wdyt? Seems like an easy change with potentially large impact.

    cli needs refinement OKR: New User Experience kind/Feature valuestream/Meltano 
    opened by DouweM 16
  • bug: Unable to get `--help` for `meltano config ... set` without a project dir context

    bug: Unable to get `--help` for `meltano config ... set` without a project dir context

    Meltano Version

    2.12.0

    Python Version

    3.10

    Bug scope

    API

    Operating System

    MacOS

    Description

    Two problems.

    Problem 1: It doesn't seem to be possible to get --help on the config set operation.

    Problem 2: The error message ('is_default_environment') doesn't help at all.

    It may not be possible to solve problem 1 without refactoring that command, but problem 2 should be addressable with improved exception/error handling.

    Code

    aj@AJs-MacBook-Pro ~ % pipx run meltano config set --help
⚠️  meltano is already on your PATH and installed at
    /Users/aj/.local/bin/meltano. Downloading and running anyway.
Need help fixing this problem? Visit http://melta.no/ for troubleshooting steps, or to
join our friendly Slack community.

'is_default_environment'

aj@AJs-MacBook-Pro ~ % pipx run meltano config asdf set --help
⚠️  meltano is already on your PATH and installed at
    /Users/aj/.local/bin/meltano. Downloading and running anyway.
Need help fixing this problem? Visit http://melta.no/ for troubleshooting steps, or to
join our friendly Slack community.

'is_default_environment'

    Running without project context

    The above was executed without a project context, meaning the working directory was not a valid Meltano project. This could be part of the issue, but regardless, I think it could be expected that --help will work even without an active project folder.

    Related

    • https://github.com/meltano/meltano/discussions/7137
    cli OKR: New User Experience ux kind/Bug valuestream/Meltano 
    opened by aaronsteers 0
  • bug: `meltano config <tap> test` returns error with selection criteria

    bug: `meltano config test` returns error with selection criteria

    Meltano Version

    2.9.0

    Python Version

    3.9

    Bug scope

    CLI (options, error messages, logging, etc.)

    Operating System

    Codespace

    Description

    I was testing a tap-github connector in GitHub Codespace. When I had a selection criteria to filter to just issues I would get this error:

    root@codespaces-f3cd48:/workspaces/meltano-codespace-ready# meltano config tap-github test
2022-12-23T21:46:26.678163Z [info     ] The default environment 'codespace' will be ignored for `meltano config`. To configure a specific environment, please use the option `--environment=<environment name>`.
Need help fixing this problem? Visit http://melta.no/ for troubleshooting steps, or to
join our friendly Slack community.

'ascii' codec can't decode byte 0xe2 in position 3232: ordinal not in range(128)

    Removing the selection criteria resolved it.

    The Fix

    testing configuration should still work even with selection criteria present!

    Code

    No response

    kind/Bug valuestream/Meltano 
    opened by tayloramurphy 0
  • bug: nested configs are cleared when using space as delimiter and when using `--interactive`

    bug: nested configs are cleared when using space as delimiter and when using `--interactive`

    Update (@aaronsteers)

    Per the issue thread below, we've identified that the issue is present in the behavior of the CLI syntax with space as delimiter but not when dot is used. The proposal is to make these two function identically:

    meltano config testy-mctest set file_configs.format json       # the expected behavior (does not reset sibling config)
meltano config testy-mctest set file_configs format json       # bug: clears sibling nested config

    Original Issue description (@pnadolny13)

    When using --interactive to update nested settings it overwrites the entry config set. For example:

      - name: tap-file
    config:
      file_configs:
        dataset_name: test_file
        format: csv
        path: /my/path/

    Then I run meltano config tap-file set --interactive, navigating to the setting file_configs.format (in dot notation) as defined in the plugin definition lock file, updating to json, meltano overwrites the whole nested config with only the new value:

      - name: tap-file
    config:
      file_configs:
        format: json

    Slack thread with video showing bug https://meltano.slack.com/archives/C03GKHWS0HM/p1671746236351709

    meltano, version 2.12.0

    kind/Bug valuestream/Meltano 
    opened by pnadolny13 4
  • bug: Error: Setting value set via multiple environment Azure connection string (state backend).

    bug: Error: Setting value set via multiple environment Azure connection string (state backend).

    Meltano Version

    2.12.0

    Python Version

    3.10

    Bug scope

    Configuration (settings parsing, validation, etc.)

    Operating System

    Linux - Ubuntu 20.04

    Description

    I am running into the same issue as #7059, but with the Azure connection string.

    I solved the issue when running the Meltano cli. However, the issue came back when I tried to run the Dagster utility.

    Code

    Error: Setting value set via multiple environment variables: ['MELTANO_STATE_BACKEND_AZURE_CONNECTION_STRING', 'AZURE_STORAGE_CONNECTION_STRING']
    kind/Bug valuestream/Meltano 
    opened by JulesHuisman 2
Releases(v2.12.0)

  • v2.12.0(Dec 19, 2022)

  • v2.11.1(Dec 9, 2022)

  • v2.11.0(Dec 8, 2022)

    v2.11.0 (2022-12-08)

    ✨ New

    • #7019 Support env vars in plugin pip_url
    • #6995 meltano install --force

    🐛 Fixes

    • #6873 Correct migration downgrade for 2.7.1 release
    • #7026 Permit env under plugins in meltano.yml schema
    • #7022 Pin setuptools==57.5.0 in plugin venvs
    • #7017 Remove premature deserialization of settings in the CLI
    • #7010 Re-create plugin venv if Python executable is missing
    • #6997 Correct meltano config behavior when explicit environment is provided
    • #6986 List strings instead of enum objects as CLI options

    📚 Documentation Improvements

    • #7046 Fix internal links and meltano config example code block in state backends page
    • #7041 Expand UI deprecation description -- Thanks @qbatten!
    • #7044 Add star donation blurb to readme
    • #7043 Add december campaign banner
    • #7030 Env var expansion within pip_url
    • #7035 Update custom extractor Poetry instructions -- Thanks @EMCP!
    • #7014 Add more project links to PyPI page
    • #7012 Complete list of commands that run with environment flag
    • #7002 Add docs for state backend migration
    Source code(tar.gz)
    Source code(zip)
    meltano-2.11.0-py3-none-any.whl(5.30 MB)

  • v2.10.0(Nov 16, 2022)

  • v2.9.0(Nov 8, 2022)

    v2.9.0 (2022-11-08)

    ✨ New

    • #6964 Add --cwd CLI option to override the current working directory.
    • #6785 Add support for NO_COLOR env var to disable colored output.
    • #6628 Make .meltano folder path configurable via new MELTANO_SYS_DIR_ROOT environment variable. -- Thanks @rawwar!

    🐛 Fixes

    • #6938 Fix is_redacted error when a referenced setting is undefined.
    • #6468 Fix symbolic link creation when running meltano upgrade on Windows -- Thanks @BuzzCutNorman!

    📚 Documentation Improvements

    • #6961 Fix typo "differe" -> "different" -- Thanks @kayakr!
    • #6951 Reindent code block -- Thanks @jx2lee!
    Source code(tar.gz)
    Source code(zip)
    meltano-2.9.0-py3-none-any.whl(5.28 MB)

  • v2.8.0(Oct 18, 2022)

    v2.8.0 (2022-10-17)

    ✨ New

    🐛 Fixes

    • #6751 Resolve issue causing Meltano UI to crash when connections to PyPI are blocked. -- Thanks @dingobar!
    • #6862 The default_environment setting now only applies to commands which execute within the environment: run, invoke, and elt.
    • #6877 Reverts the default value for the default_environment setting to null instead of 'dev', resolving the related breaking change from v2.5.0.
    • #6891 Resolve unhandled exception when running interactive config on plugins with no settings.

    ⚙️ Changes

    • #6862 The meltano run command will now fail if the Meltano environment is not set and no default environment is configured.

    📚 Documentation Improvements

    Source code(tar.gz)
    Source code(zip)
    meltano-2.8.0-py3-none-any.whl(3.61 MB)

  • v2.7.2(Oct 3, 2022)

    v2.7.2 (2022-10-03)

    🐛 Fixes

    • https://github.com/meltano/meltano/issues/6819 Fix error 'ProjectSettingsService' object has no attribute 'plugin' when running meltano config meltano.
    • https://github.com/meltano/meltano/issues/6818 Fix issue causing SQLAlchemy deprecation warnings.

    📚 Documentation Improvements

    • https://github.com/meltano/meltano/issues/6800 Simplified installation quick start, with advanced info moved to the detailed install guide.
    • https://github.com/meltano/meltano/issues/6781 Add parts 2-4 to the tutorial, with quick access from https://docs.meltano.com/.
    Source code(tar.gz)
    Source code(zip)
    meltano-2.7.2-py3-none-any.whl(3.61 MB)

  • v2.7.1(Sep 27, 2022)

  • v2.7.0(Sep 23, 2022)

    v2.7.0 (2022-09-23)

    ✨ New

    🚀 Changes

    • #6756 Place project_id before environments in meltano.yml in new projects.

    🐛 Fixes

    • #6783 Remove no-op "disable telemetry" toast from the UI.
    • #6779 Use v1.0.1 of the exit_event Snowplow telemetry schema.

    ⚙️ Under the Hood

    • #6742 Refactor to add support for state backends. Meltano will now store state to a new internal state table within systemdb, with support for other state backends coming soon.
    • #6745 Use random UUID as default project_id when project ID is missing.
    • #6748 Define MeltanoError exception class and recommend to contributors for improved CLI error messaging.

    📚 Documentation Improvements

    • #6764 Added Part 1 of an ELT tutorial (Link)
    • #6769 Added a guide for migrating existing dbt projects to Meltano (Link).
    • #6737 Added a Meltano at a Glance page to the Getting Started section (Link).
    • #6753 Added a handy list of video tutorials and demos (Link).
    • #6764 Move installation guide to Getting Started section (Link).
    • #6709 Added a tutorial for using Meltano with DataHub (Link).
    • #6739 Add a tutorial for using Meltano with Jupyter (Link).
    • #6743 Describe how project_id is hashed (Link) .
    Source code(tar.gz)
    Source code(zip)
    meltano-2.7.0-py3-none-any.whl(3.54 MB)

  • v2.6.0(Sep 12, 2022)

    v2.6.0 (2022-09-08)

    ✨ New

    🐛 Fixes

    • #6694 Resolves a bug in meltano remove where error states that plugin is missing the settings_with_extras attribute.
    • #6652 Better handling of permission errors when creating a new project -- Thanks @jakegut!
    • #6676 Prevent warnings/errors when running meltano config <plugin> test -- Thanks @ReubenFrankel!

    📚 Documentation Improvements

    • #6706 Add troubleshooting page -- Thanks @jared-rimmer!
    • #6688 Add docs about environment variable expansion.
    • #6671 Add example meltano projects.
    • #6674 Add docs for schedule env key.
    • #6680 Fix feature flag name for strict env var mode.
    • #6672 Add tutorial on how to debug a custom extractor -- Thanks @jared-rimmer!
    Source code(tar.gz)
    Source code(zip)
    meltano-2.6.0-py3-none-any.whl(3.61 MB)

  • v2.5.0(Aug 24, 2022)

    v2.5.0 (2022-08-24)

    ✨ New

    • #6578 Add --no-install option to meltano add -- Thanks, @ReubenFrankel!
    • #6596 Support glob paths in file plugin update extra -- Thanks, @ReubenFrankel!
    • #6581 Add new settings for hub_api_root and hub_url_auth -- Thanks, @DanielPDWalker!

    ⚡ Performance Improvements

    • #6644 Improve performance when handling large yaml files.

    🚀 Changes

    • #6592 Add validation for CRON expressions in schedules.
    • #6582 Make default_environment a top-level setting, with default value 'dev'.
      • NOTE: This was retroactively noted as a breaking change, since it added a requirement that a dev environment name must exist if default_environment was not overridden by the project. This requirement is now removed in the v2.7.3 Release, reverting the breaking change behavior.

    🐛 Fixes

    • #6627 Raise exception when meltano.yml is empty -- Thanks, @rawwar!
    • #6543 Register legacy DiscoveryFile object in Ruamel YAML instance -- Thanks, @z3z1ma!
    • #6656 Fixed issue where STRICT_ENV_VAR_MODE feature flag would not raise an error as expected.
    • #6643 Resolve issue related to alter_column warning for MSSQL.
    • #6636 Fix help links pointing to docs.meltano.com instead of meltano.com/docs.
    • #6601 Raise an error if DB migrations fail during upgrade.
    • #6609 Fix broken --pattern CLI arg in meltano state list.
    • #6584 Avoid pyproject.toml Poetry parsing issue.

    ⚙️ Under the Hood

    • #6621 Make ProjectInitService.init generate/set a project_id.
    • #6611 Use closing for DB sessions & add docstrings to meltano.core.db.
    • #6610 Replace StaleJobFailer class with fail_stale_jobs function.

    📚 Documentation Improvements

    Source code(tar.gz)
    Source code(zip)
    meltano-2.5.0-py3-none-any.whl(3.61 MB)

  • v2.4.0(Aug 5, 2022)

    v2.4.0 (2022-08-05)

    ✨ New

    • #5983 Add support for environment variables inheritance across configuration layers.
    • #6167 Add support for MS SQL Server as a systemdb backend -- Thanks @JulesHuisman!

    🚀 Changes

    • #6116 Always use a predictable store location when the default --store=auto setting is applied.

    🐛 Fixes

    • #6532 Resolve issue where Snowplow requests would hang if the Snowplow endpoint was not available.
    • #6360 meltano lock now updates and locks plugins using the Hub definition if no prior lock file exists.
    • #6383 Resolve missing systemdb error by initializing backend during meltano install -- Thanks @visch!
    • #6189 Meltano now validates mapper and mappings names with alerting as necessary -- Thanks @BuzzCutNorman!
    • #6240 Resolve a locking issue when the enable_uvicorn feature flag is enabled -- Thanks @BuzzCutNorman!

    ⚙️ Under the hood

    • #6469 Pin werkzeug library to <2.2.0.

    📚 Documentation Improvements

    • #6107 New reference page for Meltano plugin definition YAML syntax, including plugin containerization options and optional Hub metadata fields.
    • #6456 Add guidance for using custom python interpreters during pipx install -- Thanks @seajhawk!
    • #6456 Add Powershell syntax example for setting Meltano environment variables. -- Thanks @seajhawk!
    Source code(tar.gz)
    Source code(zip)
    meltano-2.4.0-py3-none-any.whl(3.61 MB)

  • v2.3.0(Jul 22, 2022)

    ✨ New

    🚀 Changes

    • #6150 Meltano now surfaces full stderr output for catalog discovery failures.
    • #3419 Change default meltano config behavior to --no-environment.

    🐛 Fixes

    • #6301 Resolve failures with message object has no attribute 'add_related' when adding new plugins via Meltano UI. -- Thanks, @BuzzCutNorman!
    • #6188 Resolve meltano run failures when plugins are invoked without providing a subcommand.
    • #6359 Resolve meltano lock failures by not attempting to lock plugins declared using inherit_from syntax.
    • #6234 Emit RuntimeWarning if a Meltano setting is provided which does not match expected setting name.
    • #6380, #3322 Complex extras such as metadata are now honored in inherited plugins and environment inheritance.

    📚 Documentation Improvements

    Full Changelog: https://github.com/meltano/meltano/compare/v2.2.0...v2.3.0

    Source code(tar.gz)
    Source code(zip)
    meltano-2.3.0-py3-none-any.whl(3.53 MB)

  • v2.2.0(Jul 1, 2022)

    ✨ New

    🐛 Fixes

    • #5985 Address warning in Airflow plugin version check by using recommended packaging.version.Version.

    📝 Documentation Improvements

    • #3174 We now recommend installing with pipx over manually created virtual environments.

    ⚙️ Under the Hood

    • #3367 Windows now covered in CI tests for basic use cases. - Thanks, Derek Visch!
    • #2035 Rename job table and job_id column in backend systemdb.
    Source code(tar.gz)
    Source code(zip)
    meltano-2.2.0-py3-none-any.whl(3.53 MB)

  • v2.2.0rc0(Jul 1, 2022)

    New

    • #3452 Add copy and move commands to meltano state

    Changes

    • #3174 We now recommend pipx over manually created virtual environments.
    • #3450 Windows instllation instructions added
    • #3367 Windows now covered in CI for some basic cases
    • #3135 Formalize settings alias behavior and disallow conflicting values between setting name and aliases
    • #2035 Rename job table and job_id column in backend db

    Fixes

    • #5985 - Address warning in Airflow plugin version check by using recommended packaging.version.Version.
    Source code(tar.gz)
    Source code(zip)
    meltano-2.2.0rc0-py3-none-any.whl(3.53 MB)

  • v2.1.0(Jun 17, 2022)

    New

    • https://github.com/meltano/meltano/pull/6114 Add a new meltano lock command to lock specific plugin definitions and optionally update existing lock files.

    Fixes

    • https://github.com/meltano/meltano/pull/6227 Resolve a bug that would cause intermittent errors when calling meltano config airflow.
    • https://github.com/meltano/meltano/pull/6216 Fix bug in Meltano UI that would prevent schedules from being executed in the web interface.

    Under the hood

    • Continued improvements to anonymized usage stats.

    Full Changelog: https://github.com/meltano/meltano/compare/v2.0.3...v2.1.0

    Source code(tar.gz)
    Source code(zip)

  • v2.0.3(Jun 15, 2022)

    What's Changed

    Changes

    • https://github.com/meltano/meltano/pull/6186 Improve CLI startup performance.

    Under the Hood

    • https://github.com/meltano/meltano/pull/6187, https://github.com/meltano/meltano/pull/6212 Various telemetry improvements.

    Full Changelog: https://github.com/meltano/meltano/compare/v2.0.2...v2.0.3

    Source code(tar.gz)
    Source code(zip)

  • v2.0.2(Jun 15, 2022)

  • v2.0.1(Jun 9, 2022)

  • v2.0.0(Jun 8, 2022)

    2.0.0 - (2022-06-08)

    Meltano 2.0 is a major update with multiple new features and some breaking changes. See our 2.0 Migration Guide for more information.

    New

    • #3283 Support meltano add directly from Hub.
    • #2924 Initial meltano job support via new meltano job command and support for referencing jobs in meltano run commands.
    • #5923 Add support for jobs to schedules and improves general schedule cli UX.
    • #3348 Add env_var_strict_mode feature flag to raise an exception if unset environment variables are referenced in settings.
    • #6060 Generate lockfiles when running meltano add to install new plugins (now enabled by default).
    • #3355 New paradigm of explicit plugin dependencies using requires:, replaces implicitly "related" plugins.

    Changes

    • #5957 Remove the auto-add functionality that applied to transform plugins and plugins with matching namespace.
    • #6043 Let Meltano settings be stored under environments.

    Breaks

    • #3427 Drop support for the legacy "Dashboards" and "Explore" UX in Meltano UI.
    • #3427 Drop support for legacy model and dashboard plugin types.
    • #3048 Drop support for env_aliases feature and remove its references in discovery.yml.
    • #5957 Drop support for the --include-related CLI flag.

    Fixes

    • #5979 Fixes meltano remove bug where lock files were not removed if they include a variant in the file name.
    • #6015 Fixes a bug where meltano config meltano set would not always set a value.

    Under the Hood

    • Improved telemetry coverage across the product.
    Source code(tar.gz)
    Source code(zip)

  • v1.105.0(May 30, 2022)

    1.105.0 - (2022-05-19)

    New

    • #3298 Add adapter-specific dbt plugins for Redshift, Postgres, and BigQuery.

    • #3482 Expand project-set environment variables in Environment-level env:.

    Changes

    • #3174 - We now recommend pipx over manually created virtual environments.

    • #3484 Use SettingsService.feature_flag convention for lockfiles.

    Fixes

    • #3483 Fixes a bug where pipx installed meltano can not find executable(.exe) meltano dependency packages. Thanks Dan Norman!

    • #3500 Fixes bug where lock file was added but not used.

    Source code(tar.gz)
    Source code(zip)

  • v1.98.1(May 30, 2022)

  • v1.98.0(May 30, 2022)

    1.98.0 - (2022-03-10)

    Changes

    • #3316 Improve the meltano init experience with beautified text and cleaner logging
    • #3317 Improve Snowplow documentation and simplify telemetry notification during meltano init
    • #3319 Limit Snowplow tracker logs to the command line to ERROR level by default.
    • #3217 Allow user to configure Sendgrid asm_group_id. Thanks, @davesgonechina!
    • #3321 Switch Snowplow tracker endpoint to https.

    Fixes

    • #3306 Fix Environments bug where custom configuration values that were not defined either as a setting or custom configuration in the primary plugin definition were not passed to the plugin in the Environment context.
    Source code(tar.gz)
    Source code(zip)

  • v1.97.0(May 30, 2022)

    1.97.0 - (2022-03-03)

    New

    • #2527, #2528 Add CRON interval mechanism and modal to /pipelines page.
    • #3017 Add Snowplow Tracking to Meltano.
    • #3130 Add support for incremental jobs to meltano run.

    Changes

    • #3282 Expand meltano elt failure information to help new users find debug logging. Thanks, Derek Visch!

    Fixes

    • #3289 In target-bigquery default variant adswerve, set the default of the merge_state_messages setting to false so state objects are not merged by the target.
    • #3301 Fix a typo that causes requests to the /install/batch endpoint to fail.
    Source code(tar.gz)
    Source code(zip)

  • v1.13.0(May 30, 2022)

    1.13.0 - (2019-12-23)

    New

    • #1269 Add kind: file so single file uploads can be used with extractors (tap-google-analytics's key_file_location is the first user)
    • #1494 Add LIKE options to Analyze Filter UI so users better understand what filtering patterns are available

    Changes

    • #1399 Log Modal now has a prompt to explain potential factors in required time for pipelines to complete
    • #1433 Remove /orchestrate route and thus the Airflow iframe as this is overkill for our current target users

    Fixes

    • #1434 Fix Analyze CTAs to only enable if at least one related pipeline has succeeded
    • #1447 Various fixes around loading and reloading reports to mitigate false positive sqlErrorMessage conditions
    • #1509 Allow plugin profile config to be set through meltano.yml
    Source code(tar.gz)
    Source code(zip)

  • v1.12.0(May 30, 2022)

    1.12.0 - (2019-12-16)

    New

    • #1374 Add role revocation for users and roles in the meltano permissions command
    • #1377 Document cleanup steps after MeltanoData testing
    • #1438 Add documentation for DNS spoofing error
    • #1436 Add video walkthrough on how to setup Google Analytics so that the Meltano Extractor can be able to access the Google APIs and the Google Analytics data.

    Changes

    • #1350 Switch to all lower case for Snowflake permission comparisons in the meltano permissions command
    • #1449 Hide the Marketo Extractor form Meltano UI
    • #1397 Optimize workflow for MeltanoData setup
    • #1423 Update sidebar and docs to include Ansible
    Source code(tar.gz)
    Source code(zip)

  • v1.7.1(May 30, 2022)

Owner
Meltano
ELT for the DataOps era
Meltano
GitHub https://meltano.com
Full ELT process on GCP environment.

Rent Houses Germany - GCP Pipeline Project: The goal of the project is to extract data about house rentals in Germany, store, process and analyze it u

Felipe Demenech Vasconcelos 2 Jan 20, 2022
Elementary is an open-source data reliability framework for modern data teams. The first module of the framework is data lineage.

Data lineage made simple, reliable, and automated. Effortlessly track the flow of data, understand dependencies and analyze impact. Features Visualiza

null 898 Jan 9, 2023
In this project, ETL pipeline is build on data warehouse hosted on AWS Redshift.

ETL Pipeline for AWS Project Description In this project, ETL pipeline is build on data warehouse hosted on AWS Redshift. The data is loaded from S3 t

Mobeen Ahmed 1 Nov 1, 2021
Visions provides an extensible suite of tools to support common data analysis operations

Visions And these visions of data types, they kept us up past the dawn. Visions provides an extensible suite of tools to support common data analysis

null 168 Dec 28, 2022
First and foremost, we want dbt documentation to retain a DRY principle. Every time we repeat ourselves, we waste our time. Second, we want to understand column level lineage and automate impact analysis.

dbt-osmosis First and foremost, we want dbt documentation to retain a DRY principle. Every time we repeat ourselves, we waste our time. Second, we wan

Alexander Butler 150 Jan 6, 2023
Data Analysis for First Year Laboratory at Imperial College, London.

Data Analysis for First Year Laboratory at Imperial College, London. For personal reference only, and to reference in lab reports and lab books.

Martin He 0 Aug 29, 2022
My first Python project is a simple Mad Libs program.

Python CLI Mad Libs Game My first Python project is a simple Mad Libs program. Mad Libs is a phrasal template word game created by Leonard Stern and R

Carson Johnson 1 Dec 10, 2021
VevestaX is an open source Python package for ML Engineers and Data Scientists.

VevestaX Track failed and successful experiments as well as features. VevestaX is an open source Python package for ML Engineers and Data Scientists.

Vevesta 24 Dec 14, 2022
CleanX is an open source python library for exploring, cleaning and augmenting large datasets of X-rays, or certain other types of radiological images.

cleanX CleanX is an open source python library for exploring, cleaning and augmenting large datasets of X-rays, or certain other types of radiological

Candace Makeda Moore, MD 20 Jan 5, 2023
yt is an open-source, permissively-licensed Python library for analyzing and visualizing volumetric data.

The yt Project yt is an open-source, permissively-licensed Python library for analyzing and visualizing volumetric data. yt supports structured, varia

The yt project 367 Dec 25, 2022
PCAfold is an open-source Python library for generating, analyzing and improving low-dimensional manifolds obtained via Principal Component Analysis (PCA).

PCAfold is an open-source Python library for generating, analyzing and improving low-dimensional manifolds obtained via Principal Component Analysis (PCA).

Burn Research 4 Oct 13, 2022
Karate Club: An API Oriented Open-source Python Framework for Unsupervised Learning on Graphs (CIKM 2020)

Karate Club is an unsupervised machine learning extension library for NetworkX. Please look at the Documentation, relevant Paper, Promo Video, and Ext

Benedek Rozemberczki 1.8k Jan 9, 2023
Hue Editor: Open source SQL Query Assistant for Databases/Warehouses

Hue Editor: Open source SQL Query Assistant for Databases/Warehouses

Cloudera 759 Jan 7, 2023
HyperSpy is an open source Python library for the interactive analysis of multidimensional datasets

HyperSpy is an open source Python library for the interactive analysis of multidimensional datasets that can be described as multidimensional arrays o

HyperSpy 411 Dec 27, 2022
Open-source Laplacian Eigenmaps for dimensionality reduction of large data in python.

Fast Laplacian Eigenmaps in python Open-source Laplacian Eigenmaps for dimensionality reduction of large data in python. Comes with an wrapper for NMS

null 17 Jul 9, 2022
Open source platform for Data Science Management automation

Hydrosphere examples This repo contains demo scenarios and pre-trained models to show Hydrosphere capabilities. Data and artifacts management Some mod

hydrosphere.io 6 Aug 10, 2021
Spaghetti: an open-source Python library for the analysis of network-based spatial data

pysal/spaghetti SPAtial GrapHs: nETworks, Topology, & Inference Spaghetti is an open-source Python library for the analysis of network-based spatial d

Python Spatial Analysis Library 203 Jan 3, 2023
OpenARB is an open source program aiming to emulate a free market while encouraging players to participate in arbitrage in order to increase working capital.

Overview OpenARB is an open source program aiming to emulate a free market while encouraging players to participate in arbitrage in order to increase

Tom 3 Feb 12, 2022
Snakemake workflow for converting FASTQ files to self-contained CRAM files with maximum lossless compression.

Snakemake workflow: <name> A Snakemake workflow for <description> Usage The usage of this workflow is described in the Snakemake Workflow Catalog. If

Algorithms for reproducible bioinformatics (Koesterlab) 1 Dec 16, 2021