I was able to successfully load catalog using dbcat but I'm geting the following error when I tried to parse queries using a file in json format(I also tried the given test file)

File "~/Python/3.8/lib/python/site-packages/data_lineage/parser/init.py", line 124, in parse name = str(hash(sql)) TypeError: unhashable type: 'dict'

Here's line 124: https://github.com/tokern/data-lineage/blob/f347484c43f8cb9b97c44086dd2557e3b40904ab/data_lineage/parser/init.py#L124

Code executed:

from dbcat import catalog_connection
from data_lineage.parser import parse_queries, visit_dml_query
import json

with open("queries2.json", "r") as file:
    queries = json.load(file)

catalog_conf = """
catalog:
  user:test
  password: t@st
  host: 127.0.0.1
  port: 5432
  database: postgres
"""
catalog = catalog_connection(catalog_conf)

parsed = parse_queries(queries)

visited = visit_dml_query(catalog, parsed)

I'm adding a snowflake source as follows.. where sf_db_name is my database name e.g. snowfoo (verified in debugger)...

 source = catalog.add_source(name=f"sf1_{time.time_ns()}", source_type="snowflake", database=sf_db_name, username=sf_username, password=sf_password, account=sf_account, role=sf_role, warehouse=sf_warehouse)

... but when it goes to scan, it looks like the code thinks my database name is 'prod':

tokern-data-lineage | sqlalchemy.exc.ProgrammingError: (snowflake.connector.errors.ProgrammingError) 002003 (02000): SQL compilation error:
tokern-data-lineage | Database 'PROD' does not exist or not authorized.
tokern-data-lineage | [SQL:
tokern-data-lineage |     SELECT
tokern-data-lineage |         lower(c.column_name) AS col_name,
tokern-data-lineage |         c.comment AS col_description,
tokern-data-lineage |         lower(c.data_type) AS col_type,
tokern-data-lineage |         lower(c.ordinal_position) AS col_sort_order,
tokern-data-lineage |         lower(c.table_catalog) AS database,
tokern-data-lineage |         lower(c.table_catalog) AS cluster,
tokern-data-lineage |         lower(c.table_schema) AS schema,
tokern-data-lineage |         lower(c.table_name) AS name,
tokern-data-lineage |         t.comment AS description,
tokern-data-lineage |         decode(lower(t.table_type), 'view', 'true', 'false') AS is_view
tokern-data-lineage |     FROM
tokern-data-lineage |         prod.INFORMATION_SCHEMA.COLUMNS AS c
tokern-data-lineage |     LEFT JOIN
tokern-data-lineage |         prod.INFORMATION_SCHEMA.TABLES t
tokern-data-lineage |             ON c.TABLE_NAME = t.TABLE_NAME
tokern-data-lineage |             AND c.TABLE_SCHEMA = t.TABLE_SCHEMA
tokern-data-lineage |      ;
tokern-data-lineage |     ]
tokern-data-lineage | (Background on this error at: http://sqlalche.me/e/13/f405)

.. I'm trying to look through the tokern code repos to see where the disconnect might be happening, but not sure yet...

When I add my snowflake DB for scanning, using this bit of code (with the values replaced as per my snowflake database):

from data_lineage import Catalog

catalog = Catalog(docker_address)

# Register wikimedia datawarehouse with data-lineage app.

source = catalog.add_source(name="wikimedia", source_type="postgresql", **wikimedia_db)

# Scan the wikimedia data warehouse and register all schemata, tables and columns.

catalog.scan_source(source)

... I get

tokern-data-lineage-visualizer | 2021/10/08 21:51:40 [error] 34#34: *1 upstream prematurely closed connection while reading response header from upstream, client: 10.10.0.1, server: , request: "POST /api/v1/catalog/scanner HTTP/1.1", upstream: "http://10.10.0.3:4142/api/v1/catalog/scanner", host: "127.0.0.1:8000"

... I think it's because snowflake isn't returning fast enough... but I'm not sure. Tried updating the warehouse size to large to make the scan faster, but getting the same thing. Seems like it times out pretty fast... at least for my large database. Any ideas?

Python 3.8.0 in an isolated venv, 0.8.3 data-lineage. Thanks for this package!

Currently it doesn't appear that the dml_visitor will walk through the common table expressions to build the lineage. Am I interpreting this wrong? Within vistor.py line 45 and 61 both visit the "with clause". There doesn't seem to be any functionality for handling the commontableexpr or ctes within the parsed statments. This causes any statements with ctes to throw an error when calling parse_queries, as no table is found when attempting to bind a CTE in a FROM clause.

Hi, I'm trying to install the 0.8 version on a docker that runs on Debian Buster and when it runs the pip command to install it prints the following warning/error:

#12 9.444 Collecting data-lineage==0.8.0 (from -r /project/requirements.txt (line 25))
#12 9.466   Could not find a version that satisfies the requirement data-lineage==0.8.0 (from -r /project/requirements.txt (line 25)) (from versions: 0.1.2, 0.2.0, 0.3.0, 0.5.1, 0.5.2, 0.6.0, 0.7.0)
#12 9.541 No matching distribution found for data-lineage==0.8.0 (from -r /project/requirements.txt (line 25))

Is this normal behavior? Do I have to add something before trying to install?

Hello! i use this docs https://tokern.io/docs/data-lineage/installation

1 curl https://raw.githubusercontent.com/tokern/data-lineage/master/install-manifests/docker-compose-demodb/docker-compose.yml -o docker-compose.yml 2 docker-compose up -d 3 get error ERROR: In file './docker-compose.yml', the services name 404 must be a quoted string, i.e. '404'.

Hi, I am trying to parse query history from Snowflake on Jupyter notebook.

data lineage version 0.3.0

!pip install snowflake-connector-python[secure-local-storage,pandas] data-lineage

import datetime
end_time = datetime.datetime.now()
start_time = end_time - datetime.timedelta(days=7)

query = f"""
SELECT query_text
FROM table(information_schema.query_history(
    end_time_range_start=>to_timestamp_ltz('{start_time.isoformat()}'),
    end_time_range_end=>to_timestamp_ltz('{end_time.isoformat()}')));
"""

cursors = conn.execute_string(
    sql_text=query
)

queries = []
for cursor in cursors:
  for row in cursor:
    print(row[0])
    queries.append(row[0])

from data_lineage.parser import parse_queries, visit_dml_queries

# Parse all queries
parsed = parse_queries(queries)

# Visit the parse trees to extract source and target queries
visited = visit_dml_queries(catalog, parsed)

# Create a graph and visualize it

from data_lineage.parser import create_graph
graph = create_graph(catalog, visited)

import plotly
plotly.offline.iplot(graph.fig())

Then I got this error. Would you help me find the root cause?

---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-33-151c67ea977c> in <module>
----> 1 from data_lineage.parser import parse_queries, visit_dml_queries
      2 
      3 # Parse all queries
      4 parsed = parse_queries(queries)
      5 

ImportError: cannot import name 'parse_queries' from 'data_lineage.parser' (/opt/conda/lib/python3.8/site-packages/data_lineage/parser/__init__.py)

Opening this issue to let everyone know till it gets fixed...the installation for pglast fails requiring a xxhash.h file. Here's the link to issue and how to resolve it: https://github.com/lelit/pglast/issues/82

Please feel free to close if you think its inappropriate

I am trying to use this example: https://tokern.io/docs/data-lineage/queries ... first issue... this bit of code looks like it's just going to fetch a single row from the query history from snowflake:

queries = []
with connection.get_cursor() as cursor:
  cursor.execute(query)
  row = cursor.fetchone()

  while row is not None:
    queries.append(row[0])

... is this intended? Note that it's using .fetchone()

Then.. second issue... when I go back to the example here: https://tokern.io/docs/data-lineage/example

I see this bit of code...

analyze = Analyze(docker_address)

for query in queries:
    print(query)
    analyze.analyze(**query, source=source, start_time=datetime.now(), end_time=datetime.now())

... what does the queries array look like? Or better yet, what does the single query item look like? Above it, in the example, it looks to be a JSON payload....

with open("queries.json", "r") as file:
    queries = json.load(file)

.... but I've no idea what the payload is supposed to look like.

I've tried 8 different ways of passing this **query variable into analyze(...) - using the results from the snowflake example on https://tokern.io/docs/data-lineage/queries - but I can never seem to get it right. Either I get an error saying that ** expects a mapping when I use strings or tuples (which is fine, but what's the mapping the function expects?) - or I get an error in the API console itself like

tokern-data-lineage |     raise ValueError('Bad argument, expected a ast.Node instance or a tuple')
tokern-data-lineage | ValueError: Bad argument, expected a ast.Node instance or a tuple

.. could we get a more concrete snowflake example, or at the bare minimum please indicate what the query variable is supposed to look like?

Note that I am also trying to inspect the unit tests and use those as examples, but still not getting very far.

Thanks for this package!

calling

analyze.analyze(**{"query":query}, source=dl_source, start_time=datetime.now(), end_time=datetime.now())

with a large query, ​I get a "request too long" - seems that even though it is POSTing, it's still appending the query to the URL, thus the request fails.. e.g.

tokern-data-lineage-visualizer | 10.10.0.1 - - [14/Oct/2021:14:39:00 +0000] "POST /api/v1/analyze?query=ANY_REALLY_LONG_QUERY_HERE

amundsen-databuilder which is one of the package dependencies for dbcat requires flask 1.0.2 whereas data-lineage requires flask 1.1

Please feel free to close if its not a valid issue.

Bumps certifi from 2021.5.30 to 2022.12.7.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Trying out a demo, I saw the scan (see also https://github.com/tokern/data-lineage/issues/106) command fail, with the server looking for port 6379 on localhost. Sure enough, starting local redis removed that problem. Can this be documented ? It looks like the docker compose file includes it, just the instructions don't.

Trying out a demo, I tried to run catalog.scan_source(source). But that does not exist. After some digging, it looks like this works:

from data_lineage import Scan

Scan('http://127.0.0.1:8000').start(source)

Please fix the demo pages.

$ data_lineage --catalog-user xxx --catalog-password yyy
Traceback (most recent call last):
  File "/opt/homebrew/bin/data_lineage", line 5, in <module>
    from data_lineage.__main__ import main
  File "/opt/homebrew/lib/python3.9/site-packages/data_lineage/__main__.py", line 7, in <module>
    from data_lineage.server import create_server
  File "/opt/homebrew/lib/python3.9/site-packages/data_lineage/server.py", line 5, in <module>
    import flask_restless
  File "/opt/homebrew/lib/python3.9/site-packages/flask_restless/__init__.py", line 22, in <module>
    from .manager import APIManager  # noqa
  File "/opt/homebrew/lib/python3.9/site-packages/flask_restless/manager.py", line 24, in <module>
    from flask import Blueprint
  File "/opt/homebrew/lib/python3.9/site-packages/flask/__init__.py", line 14, in <module>
    from jinja2 import escape
  File "/opt/homebrew/lib/python3.9/site-packages/jinja2/__init__.py", line 12, in <module>
    from .environment import Environment
  File "/opt/homebrew/lib/python3.9/site-packages/jinja2/environment.py", line 25, in <module>
    from .defaults import BLOCK_END_STRING
  File "/opt/homebrew/lib/python3.9/site-packages/jinja2/defaults.py", line 3, in <module>
    from .filters import FILTERS as DEFAULT_FILTERS  # noqa: F401
  File "/opt/homebrew/lib/python3.9/site-packages/jinja2/filters.py", line 13, in <module>
    from markupsafe import soft_unicode
ImportError: cannot import name 'soft_unicode' from 'markupsafe' (/opt/homebrew/lib/python3.9/site-packages/markupsafe/__init__.py)

Looks like that was removed in 2.1.0. You may want to specify markupsafe==2.0.1.

This is probably due to SQLAlchemy's requirement of mysqlclient, but when doing

pip install data-lineage

The following is seen

Collecting mysqlclient<3,>=1.3.6
  Using cached mysqlclient-2.1.1.tar.gz (88 kB)
  Preparing metadata (setup.py) ... error
  error: subprocess-exited-with-error
  
  × python setup.py egg_info did not run successfully.
  │ exit code: 1
  ╰─> [16 lines of output]
      /bin/sh: mysql_config: command not found
      /bin/sh: mariadb_config: command not found
      /bin/sh: mysql_config: command not found
      Traceback (most recent call last):
        File "<string>", line 2, in <module>
        File "<pip-setuptools-caller>", line 34, in <module>
        File "/private/var/folders/th/yz4tb0ss5t3_4df1xnfrkg3r0000gn/T/pip-install-auypdvbk/mysqlclient_42a825d5ee084d6686c16912ef8320cc/setup.py", line 15, in <module>
          metadata, options = get_config()
        File "/private/var/folders/th/yz4tb0ss5t3_4df1xnfrkg3r0000gn/T/pip-install-auypdvbk/mysqlclient_42a825d5ee084d6686c16912ef8320cc/setup_posix.py", line 70, in get_config
          libs = mysql_config("libs")
        File "/private/var/folders/th/yz4tb0ss5t3_4df1xnfrkg3r0000gn/T/pip-install-auypdvbk/mysqlclient_42a825d5ee084d6686c16912ef8320cc/setup_posix.py", line 31, in mysql_config
          raise OSError("{} not found".format(_mysql_config_path))
      OSError: mysql_config not found
      mysql_config --version
      mariadb_config --version
      mysql_config --libs
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

Installing mysql client fixes it.

Since you are using SQLAlchemy, this is out of your hands but this issue is to suggest maybe add a note to that effect in the docs?

changed the CATALOG_PASSWORD,CATALOG_USER, CATALOG_DB, CATALOG_HOST accordingly and ran this command docker-compose -f tokern-lineage-engine.yml up. Throwing me an error
return self.dbapi.connect(*cargs, **cparams) tokern-data-lineage | File "/opt/pysetup/.venv/lib/python3.8/site-packages/psycopg2/init.py", line 122, in connect tokern-data-lineage | conn = _connect(dsn, connection_factory=connection_factory, **kwasync) tokern-data-lineage | sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) could not translate host name "-xxxxxxx.amazonaws.com" to address: Temporary failure in name resolution