Overview

As a data engineer, there are quite a few properties that we can extract programmatically that are currently not supported by amundsen as first class properties in the UI. For some of these properties that are likely widely useful, I can understand creating issues to ingest them so that they appear in the panel on the right. However, some of these properties are very company specific and wouldn't be needed by other companies. Therefore, I think that it would be useful to allow users to ingest structured data without needing to make changes to amundsen infrastructure.

What do we do now? And why it won't work in long run?

Currently, we get around this by programmatically updating the table description with prepared markdown, however in the long run we also want users to be able to edit table and column descriptions through amundsen which will put us in a bind. We no longer would able to update programmatically without a lot of added complexity concerning reconciliation and merging of changes.

Proposal

My proposal is that there are two types of descriptions for tables and columns

One would be programmatic and cannot be modified manually. The other would be the current description. By default, the programmatic description would not appear on the page unless it is populated.

Note about column level

This is true on the column level as well, where we may want to include company specific attributes, but I can understand why column level programmatic descriptions maybe too much to ask for in terms of cluttering the UI.

Summary of Changes

Upgrade feast extractor with new Feast architecture

Tests

• Update old tests and remove deprecated ones

CheckList

Make sure you have checked all steps below to ensure a timely review.

• [x] PR title addresses the issue accurately and concisely. Example: "Updates the version of Flask to v1.0.2"
• [x] PR includes a summary of changes.
• [x] PR adds unit tests, updates existing unit tests, OR documents why no test additions or modifications are needed.
• [x] In case of new functionality, my PR adds documentation that describes how to use it.
  • All the public functions and the classes in the PR contain docstrings that explain what it does

I'm trying to implement Google OIDC authentication on an existing amundsen. All the micro services (search, metadata, frontend) are deployed with helm on AWS EKS and, without authentication, it works. But when I enable OIDC, as you'll see below, session information does not persist and frontend starts showing a "Something went wrong..." message.

Here's the related logs from frontend. (sensitive info replaced with arbitrary number of *'s + I put the 'test_data' in session for debugging purposes)

2021-08-13T17:23:20+09:00 /usr/local/lib/python3.7/site-packages/flask/json/__init__.py:179: DeprecationWarning: Importing 'itsdangerous.json' is deprecated and will be removed in ItsDangerous 2.1. Use Python's 'json' module instead.
2021-08-13T17:23:20+09:00   rv = _json.dumps(obj, **kwargs)
2021-08-13T17:23:20+09:00 /usr/local/lib/python3.7/site-packages/flask/json/__init__.py:205: DeprecationWarning: Importing 'itsdangerous.json' is deprecated and will be removed in ItsDangerous 2.1. Use Python's 'json' module instead.
2021-08-13T17:23:20+09:00   return _json.loads(s, **kwargs)
2021-08-13T17:23:20+09:00 2021-08-13T08:23:20+0000.168 [DEBUG] __init__._before_request:23 (11:MainThread) - Whitelisted Endpoint: status,healthcheck,health,logout
2021-08-13T17:23:20+09:00 2021-08-13T08:23:20+0000.169 [ERROR] models._fetch_token:88 (11:MainThread) - Calling _fetch_token(name=google)...
2021-08-13T17:23:20+09:00 2021-08-13T08:23:20+0000.169 [ERROR] models._fetch_token:92 (11:MainThread) - <SecureCookieSession {'test_data': 'test data', 'user': {'__id': '*****@*************.com', 'at_hash': '**************', 'aud': '***********************', 'azp': '*****************', 'display_name': '******************', 'email': '*****************', 'email_verified': True, 'exp': 1628846069, 'family_name': '******', 'given_name': '***********', 'hd': '***************, 'iat': 1628842469, 'iss': 'https://accounts.google.com', 'locale': 'en', 'name': '************', 'nonce': '*************', 'picture': '****************', 'profile_url': '', 'sub': '**********', 'user_id': '*****************'}}>
2021-08-13T17:23:32+09:00 /usr/local/lib/python3.7/site-packages/flask/json/__init__.py:179: DeprecationWarning: Importing 'itsdangerous.json' is deprecated and will be removed in ItsDangerous 2.1. Use Python's 'json' module instead.
2021-08-13T17:23:32+09:00   rv = _json.dumps(obj, **kwargs)
2021-08-13T17:23:32+09:00 /usr/local/lib/python3.7/site-packages/flask/json/__init__.py:179: DeprecationWarning: Importing 'itsdangerous.json' is deprecated and will be removed in ItsDangerous 2.1. Use Python's 'json' module instead.
2021-08-13T17:23:32+09:00   rv = _json.dumps(obj, **kwargs)
2021-08-13T17:23:32+09:00 /usr/local/lib/python3.7/site-packages/flask/json/__init__.py:179: DeprecationWarning: Importing 'itsdangerous.json' is deprecated and will be removed in ItsDangerous 2.1. Use Python's 'json' module instead.
2021-08-13T17:23:32+09:00   rv = _json.dumps(obj, **kwargs)
2021-08-13T17:23:32+09:00 2021-08-13T08:23:32+0000.720 [DEBUG] __init__._before_request:23 (10:MainThread) - Whitelisted Endpoint: status,healthcheck,health,logout
2021-08-13T17:23:32+09:00 2021-08-13T08:23:32+0000.720 [ERROR] models._fetch_token:88 (10:MainThread) - Calling _fetch_token(name=google)...
2021-08-13T17:23:32+09:00 2021-08-13T08:23:32+0000.720 [ERROR] models._fetch_token:92 (10:MainThread) - <SecureCookieSession {'test_data': 'test data'}>
2021-08-13T17:23:32+09:00 2021-08-13T08:23:32+0000.721 [ERROR] models._fetch_token:100 (10:MainThread) - 'user'
2021-08-13T17:23:32+09:00 2021-08-13T08:23:32+0000.721 [ERROR] __init__._before_request:59 (10:MainThread) - User not logged in, redirecting to auth
2021-08-13T17:23:32+09:00 Traceback (most recent call last):
2021-08-13T17:23:32+09:00   File "/usr/local/lib/python3.7/site-packages/flaskoidc/models.py", line 94, in _fetch_token
2021-08-13T17:23:32+09:00     user_id=session["user"]["__id"],
2021-08-13T17:23:32+09:00   File "/usr/local/lib/python3.7/site-packages/flask/sessions.py", line 83, in __getitem__
2021-08-13T17:23:32+09:00     return super(SecureCookieSession, self).__getitem__(key)
2021-08-13T17:23:32+09:00 KeyError: 'user'

My suspicions are:

1. Google OIDC has different specs, for example, in my setup, OAuth2Token (flaskoidc) saves access token, refresh token and such, instead of name or user_id. So I tweaked the flaskoidc lib to authenticate the current session user using some google apis, but still the error persists.
2. Metadata also produces suspicious logs. Except the healthcheck signals, it always returns 302. Is it meant to work like this? I'm not sure.

Expected Behavior

meant to work out of the box?

Current Behavior

frontend shows 'something went wrong' messages after login when hooked with google oidc

Possible Solution

Steps to Reproduce

1. deploy amundsen using the helm chart in gith repo using frontend-oidc: 3.11.1, metadata-oidc: 3.5.0, search:2.5.1 with flask OIDC env variables for Google oidc

I'm trying to install Amundsen on docker running on Windows 10. I'm getting an error while run a docker-compose using atlas.

Expected Behavior

Install success

Current Behavior

Getting erro when execute: docker-compose -f docker-amundsen-atlas.yml up

Steps to Reproduce

1. Clone the repo
2. Execute: docker-compose -f docker-amundsen-atlas.yml up (get an error in this step)

• Error: error: subprocess-exited-with-error

  × python setup.py egg_info did not run successfully. │ exit code: 1 ╰─> [10 lines of output] /bin/sh: 1: npm: not found ERROR:root:npm must be available /bin/sh: 1: npm: not found /app/setup.py:30: DeprecationWarning: The 'warn' function is deprecated, use 'warning' instead logging.warn('Installation of npm dependencies failed') WARNING:root:Installation of npm dependencies failed /app/setup.py:31: DeprecationWarning: The 'warn' function is deprecated, use 'warning' instead logging.warn(str(e)) WARNING:root:Command '['npm install']' returned non-zero exit status 127. error in amundsen-frontend setup command: 'extras_require' must be a dictionary whose values are strings or lists of strings containing valid project/version requirement specifiers. [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip. error: metadata-generation-failed

× Encountered error while generating package metadata. ╰─> See above for output.

note: This is an issue with the package mentioned above, not pip. hint: See above for details. 1 error occurred: * Status: The command '/bin/sh -c pip3 install -e .' returned a non-zero code: 1, Code: 1

Screenshots (if appropriate)

Context

Your Environment

• Amunsen version used: 6.7.4
• Data warehouse stores: none yet
• Deployment (k8s or native): native
• Link to your fork or repository:

Expected Behavior or Use Case

In the docs, it suggests to give USER_DETAIL_METHOD = get_user_details within the metadata_service config. After defining the function given in the docs:

def get_user_details(user_id):
    user_info = {
        'email': 'email',
        'user_id': user_id,
        'first_name': 'Firstname',
        'last_name': 'Lastname',
        'full_name': 'Firstname Lastname',
    }
    return user_info

The only info attached to the user is the email and the users are created with names Firstname Lastname. We're able to properly authenticate the user based on their email with Okta, but we are unable to create the user with their proper names.

Possible Implementation

In order for more info to be available for the function, the function header needs to change and accept other arguments like a json of the header. Something like:

def get_user_details(data):
    user_info = {
        'email': data['email'],
        'user_id': data['user_id'],
        'first_name': data['first_name'],
        'last_name': data['last_name'],
        'full_name': data['first_name']+" "+data['last_name'],
    }
    return user_info

where data is a json object with the request header info.

If this is implemented, then all of the function calls will need to be changed so that the correct info is passed along. We would need to fork the metadata submodule, build our own image and test our changes. After finishing testing, we'd push back upstream.

Context

This is needed to properly create users with their correct names.

I was following #852 but opened a new issue to provide more context and its current state.

Currently ES Proxy in Search service allows for simple user-password pair, non SSL setup. Whereas, in our usage we would like to use AWS Elasticsearch Service which requires specific ES client initialisation, which currently requires code modification and injection of another Elasticsearch client.

Expected Behavior or Use Case

User can setup in config using env variables:

PROXY_CLIENT=ELASTICSEARCH_AWS
CREDENTIALS_PROXY_USER=aws_access_key
CREDENTIALS_PROXY_PASSWORD=aws_secret_key

Service or Ingestion ETL

amundsen-search service.

Possible Implementation

The change would be mostly to the initialisation of ES Proxy, eg.

# Current ES Proxy would require ES client, without any other changes to the class business logic
class ElasticsearchProxy(BaseProxy):
    """
    ElasticSearch connection handler
    """

    def __init__(self, *,
                 client: Elasticsearch = None,
                 page_size: int = 10
                 ) -> None:
        """
        Constructs Elasticsearch client for interactions with the cluster.
        Allows caller to pass a fully constructed Elasticsearch client, {elasticsearch_client}.

        :param elasticsearch_client: Elasticsearch client to use, if provided
        :param  page_size: Number of search results to return per request
        """
        self.elasticsearch = client

        self.page_size = page_size
...

# Old implementation would only be for creating simple ES client but most of the logic is still in ElasticsearchProxy; 
# can be setup as before by env variable PROXY_CLIENT=ELASTICSEARCH
class SimpleElasticsearchProxy(ElasticsearchProxy):
    """
    ElasticSearch connection handler
    """

    def __init__(self, *,
                 host: str = None,
                 user: str = '',
                 password: str = '',
                 page_size: int = 10
                 ) -> None:
        """
        Constructs simple Elasticsearch client from the parameters provided.

        :param host: Elasticsearch host we should connect to
        :param user: user name to use for authentication
        :param password: user password to use for authentication
        :param  page_size: Number of search results to return per request
        """
        http_auth = (user, password) if user else None
        client = Elasticsearch(host, http_auth=http_auth)

        super().__init__(client=client, page_size=page_size)

# AWS ES Proxy connector can be setup via env variable PROXY_CLIENT=ELASTICSEARCH_AWS
class AwsElasticsearchProxy(ElasticsearchProxy):
    """
    ElasticSearch connection handler
    """

    def __init__(self, *,
                 host: str = None,
                 user: str = '',
                 password: str = '',
                 page_size: int = 10
                 ) -> None:
        """
        Constructs simple Elasticsearch client from the parameters provided.

        :param host: Elasticsearch host we should connect to
        :param user: AWS access key
        :param password: AWS secret key
        :param  page_size: Number of search results to return per request
        """
        region = os.environ.get('AWS_REGION')
        awsauth = AWS4Auth(user, password, region, 'es')

        client = Elasticsearch(
            hosts=[{'host': host, 'port': 443}],
            http_auth=awsauth,
            use_ssl=True,
            verify_certs=True,
            connection_class=RequestsHttpConnection
        )

        super().__init__(client=client, page_size=page_size)

Context

This would allow to have different implementation of ES and other proxy client, which can be selected via configuration and would not require witting new code and manual docker image building.

AC

• there will be scripts provided that allow amundsen neo4j data to be backed up (on a schedule) to cloud provider blob storage. aws s3 makes the most sense, and if others need other providers (e.g. azure), then they can provide an extension to this functionality
• once these scripts are established, we should extended them to the k8s setup as well

Please add points on what you expect from such a guide in a comment below. I will then try to consolidate input and draft up an outline in this comment.

The guide can end up as ~~/docs/deployment.md~~ is /docs/owners_manual.md better?

Initial outline:

• [ ] Basic install of services (in different environments)

  • [x] Docker-compose “vanilla”, but with Gunicorn (WIP #109) ~~data in volumes etc.~~
  • [ ] AWS ECS. original PR: https://github.com/lyft/amundsenfrontendlibrary/pull/216 (or EC2): https://github.com/lyft/amundsenfrontendlibrary/issues/186
  • [x] Kubernetes helm chart install ~~(convert from Compose using https://kompose.io?)~~ (upcoming PR see https://github.com/lyft/amundsen/issues/53#issuecomment-538575978 below)

• [ ] Setting up ingest (with or without Airflow, see https://github.com/lyft/amundsen/issues/53#issuecomment-617370073)

  Figure out which parts of this belongs with Architecture.md and which in Databuilder repo?

  • [ ] Compared to Quickstart ingest (https://github.com/lyft/amundsen/issues/75)
  • [ ] Then mention source by source; Extractor(s), Model, Metadata - Table Metadata: - Users - Table Usage: (How it works and why in https://github.com/lyft/amundsen/issues/381#issuecomment-613387814) - ...

• [ ] Configuration - custom build of frontend (to not have to maintain a fork we need to get https://github.com/lyft/amundsen/issues/408 transmogrified into proper documentation/tooling)

  • [x] Small tweaks to turn on/off features, adding logo etc. (mostly Done) https://github.com/lyft/amundsenfrontendlibrary/commit/c256115f7d64da121de4ea36ea9c55592c11f9d5 in PR https://github.com/lyft/amundsenfrontendlibrary/pull/255
  • [x] Config of email notification/feedback Done in PR https://github.com/lyft/amundsenfrontendlibrary/pull/291
  • [x] Data preview (integration to SuperSet) - https://github.com/lyft/amundsen/issues/27#issuecomment-517477074 has some draft contextual lead in and reasoning and a link to example setup. But ultimately what ticks off the box for this is Taos Guide in https://github.com/lyft/amundsen/blob/master/docs/tutorials/data-preview-with-superset.md (or on the https://lyft.github.io/amundsen/ site, search for SuperSet!)

• [ ] Security

  • [ ] Auth - passwords etc.
  • [ ] secure communication
  • [ ] production grade docker as per Production-ready Docker images (via https://www.youtube.com/watch?v=cDzFm68aMao)

• [x] Backup - initial WiP in https://github.com/lyft/amundsen/issues/53#issuecomment-516159598 below ... current result in https://github.com/lyft/amundsen/issues/381#issuecomment-614534794 - and restore (on K8s) implemented in https://github.com/lyft/amundsen/pull/394

• [ ] Monitoring (statsd etc.?)

• [ ] Handling upgrades

• [ ] ....

Hi Team, I’m wondering if there’s a plan to apply multiprocessing on the publishers. We have a large amount of metadata in our production, which ended up running 3 million queries on neo4j . It takes about 90 minutes to finish.

To investigate the bottleneck, I looked into the code and logged the time elapsed for each step in a single iteration in the _publish_node function. This is the result

• Neo4j query: 0.1ms
• Create statement: 1ms
• Others: super fast, doesn’t matter

Surprisingly, the bottleneck is not the db query, it’s the statement creation. The process is basically

1. loop each row in csv
2. parse the row into a dictionary
3. loop through each key value pair in the dictionary to get the props
4. fill the statement Jinjia template with the props
5. execute the query with the statement

I’m thinking that instead of read a row => create a node in graph db one by one, maybe we could use multiprocessing to speed up the process. I believe there will be no dependency issue as long as we publish all the nodes before publishing relations, which is already handled in the current codebase. I’m planning on implementing multiprocessing for this, is there any potential problem? Like dependency, graph db load, etc..

Expected Behavior or Use Case

Speed up the performance of the publisher. Currently, a 90 min sync is not acceptable for our use case 😢

Service or Ingestion ETL

Ingestion ETL, publisher

Possible Implementation

Thanks to @dkunitsk 's idea, I think there are three possible implementations

1. Multiprocessing on call side
2. Multiprocessing on Neo4j publisher
3. Neo4j UNWIND (Batch processing)

class HiveParallelIndexer:
    # Shim for adding all node labels to the NEO4J_DEADLOCK_NODE_LABELS config
    # which enables retries for those node labels. This is important for parallel writing
    # since we see intermittent Neo4j deadlock errors relatively often.
    class ContainsAllList(list):
        def __contains__(self, item):
            return True

    def __init__(self, publish_tag: str, parallelism: int):
        self.publish_tag = publish_tag
        self.parallelism = parallelism

    def __call__(self, worker_index: int):
        # Sharding:
        #   - take the md5 hash of the schema.table_name
        #   - convert the first 3 characters of the hash to decimal (3 chosen arbitrarily)
        #   - mod by total number of processes
        where_clause_suffix = """
            WHERE MOD(CONV(LEFT(MD5(CONCAT(d.NAME, '.', t.TBL_NAME)), 3), 16, 10), {total_parallelism}) = {worker_index}
            AND t.TBL_TYPE IN ('EXTERNAL_TABLE', 'MANAGED_TABLE', 'VIRTUAL_VIEW')
            AND (t.VIEW_EXPANDED_TEXT != '/* Presto View */' OR t.VIEW_EXPANDED_TEXT is NULL)
        """.format(total_parallelism=self.parallelism,
            worker_index=worker_index)

        # configs relevant for multiprocessing
        job_config = ConfigFactory.from_dict({
            'extractor.hive_table_metadata.{}'.format(HiveTableMetadataExtractor.WHERE_CLAUSE_SUFFIX_KEY):
                where_clause_suffix,
            # keeping this relatively low, in our experience, reduces neo4j deadlocks
            'publisher.neo4j.{}'.format(neo4j_csv_publisher.NEO4J_TRANSACTION_SIZE):
                100,
            'publisher.neo4j.{}'.format(neo4j_csv_publisher.NEO4J_DEADLOCK_NODE_LABELS):
                HiveParallelIndexer.ContainsAllList(),
        })
        job = DefaultJob(conf=job_config,
                         task=DefaultTask(
                             extractor=HiveTableMetadataExtractor(),
                             loader=FsNeo4jCSVLoader()),
                         publisher=Neo4jCsvPublisher())
        job.launch()


parallelism = 16
indexer = HiveParallelIndexer(
    publish_tag='2021-12-03'
    parallelism=parallelism)

with multiprocessing.Pool(processes=parallelism) as pool:
    def callback(_):
        # fast fail in case of exception in any process
        print('terminating due to exception')
        pool.terminate()
    res = pool.map_async(indexer, [i for i in range(parallelism)], error_callback=callback)
    res.get()

Screenshots of Slack Discussion

Users of Amundsen occasionally want to display table level attributes on the table detail page that are specific to their business or data source. Unlike programmatic descriptions, they also want to be able to edit this information directly in the UI as these attributes are typically human-generated.

Some examples include:

• retention policy
• data usage policy

Expected Behavior or Use Case

Users can display and edit additional custom attributes in the table detail page.

Service or Ingestion ETL

frontend and metadata services

Possible Implementation

Define additional custom table attributes via configuration in the frontend. The custom table attributes are then displayed in the table detail page using the EditableText component. Custom attributes are persisted to the graph using the metadata service and a new PATCH table API endpoint.

Example Screenshots (if appropriate):

Context

This would allow users to add human-generated business-specific metadata to Amundsen and maintain it directly in the UI.

I have uploaded information about my postgresql tables. When I try to see Information about tables I see error "Something went wrong... "

Expected Behavior

I want to see metadata information about tables.

Current Behavior

I can see that upload was succesfull because information about tables availible in neo4j store.

I have used docker-amundsen.yml script for deloyment. In amundsenfrontend container logs recieve this error

2021-03-14T13:11:33+0000.463 [ERROR] v0._get_table_metadata:149 (11:MainThread) - Encountered exception: {'columns': {0: {'col_type': ['Field may not be null.']}, 1: {'col_type': ['Field may not be null.']}}} Traceback (most recent call last): File "/app/amundsen_application/api/metadata/v0.py", line 143, in _get_table_metadata results_dict['tableData'] = marshall_table_full(table_data_raw) File "/app/amundsen_application/api/utils/metadata_utils.py", line 109, in marshall_table_full table: Table = schema.load(table_dict).data File "/usr/local/lib/python3.7/site-packages/marshmallow/schema.py", line 588, in load result, errors = self._do_load(data, many, partial=partial, postprocess=True) File "/usr/local/lib/python3.7/site-packages/marshmallow/schema.py", line 711, in _do_load raise exc marshmallow.exceptions.ValidationError: {'columns': {0: {'col_type': ['Field may not be null.']}, 1: {'col_type': ['Field may not be null.']}}} 2021-03-14T13:11:33+0000.463 [DEBUG] action_log_callback.on_post_execution:70 (11:MainThread) - Calling callbacks: [<function logging_action_log at 0x7f26506bf8c0>] 2021-03-14T13:11:33+0000.464 [DEBUG] action_log_callback.logging_action_log:85 (11:MainThread) - logging_action_log: ActionLogParams(command='_get_table_metadata', start_epoch_ms=1615727493322, end_epoch_ms=1615727493463, user='[email protected]', host_name='ee9f995f6b9d', pos_args_json='[]', keyword_args_json='{"table_key": "postgres://ioekgftt.public/demo", "index": "0", "source": "search_results"}', output='{"tableData": {}, "msg": "Encountered exception: {\'columns\': {0: {\'col_type\': [\'Field may not be null.\']}, 1: {\'col_type\': [\'Field may not be null.\']}}}", "status_code": 500}', error=None)

Steps to Reproduce

1. Deploy amundsen on docker
2. Use sample_postgres_loader.py to upload data

Your Environment

• Amunsen version used: amundsen-frontend:3.1.0, amundsen-search:2.4.1, amundsen-metadata:3.3.0
• Data warehouse stores: postgresql
• Deployment (k8s or native): native(docker)

Summary of Changes

The change is related to issue #2072. It is about supporting table/column lineage for mysql backend end to end.

• databuilder: updated table_lineage model for table/column lineage record iterator and the corresponding sample_data_loader_mysql.py
• metadata_service: updated mysql_proxy.py to add lineage related endpoint.

Tests

Added unit tests in both databuilder and metadata_service for lineage related change.

Documentation

N/A

CheckList

Make sure you have checked all steps below to ensure a timely review.

• [X] PR title addresses the issue accurately and concisely. Example: "Updates the version of Flask to v1.0.2"
  • In case you are adding a dependency, check if the license complies with the ASF 3rd Party License Policy.
• [X] PR includes a summary of changes.
• [X] PR adds unit tests, updates existing unit tests, OR documents why no test additions or modifications are needed.
• [X] In case of new functionality, my PR adds documentation that describes how to use it.
  • All the public functions and the classes in the PR contain docstrings that explain what it does

Summary of Changes

For certain deployments the graph view can get unreadable when it displays nodes at too great a depth. The app hardcodes the table lineage graph depth to 5 currently. This PR makes that value configurable using the config-types pattern that the rest of the front end uses.

There is some discussion about the feature in a previous PR that I botched: https://github.com/amundsen-io/amundsen/pull/2069

Tests

Added a unit test for the function that gets the default graph depth

Documentation

CheckList

Make sure you have checked all steps below to ensure a timely review.

• [x] PR title addresses the issue accurately and concisely. Example: "Updates the version of Flask to v1.0.2"
  • In case you are adding a dependency, check if the license complies with the ASF 3rd Party License Policy.
• [x] PR includes a summary of changes.
• [x] PR adds unit tests, updates existing unit tests, OR documents why no test additions or modifications are needed.
• [ ] In case of new functionality, my PR adds documentation that describes how to use it.
  • All the public functions and the classes in the PR contain docstrings that explain what it does

fix : gunicorn start command contains same port in all three places. So changed into three different ports which we configured in ports section.

Summary of Changes

Changed into three different ports

Tests

Documentation

CheckList

Make sure you have checked all steps below to ensure a timely review.

• [x] PR title addresses the issue accurately and concisely. Example: "Updates the version of Flask to v1.0.2"
  • In case you are adding a dependency, check if the license complies with the ASF 3rd Party License Policy.
• [x] PR includes a summary of changes.
• [x] PR adds unit tests, updates existing unit tests, OR documents why no test additions or modifications are needed.
• [x] In case of new functionality, my PR adds documentation that describes how to use it.
  • All the public functions and the classes in the PR contain docstrings that explain what it does

Summary of Changes

updating the mssql databuilder connection string, since existing way of writng sql connection string with sqlalchemy is not working with the latest version of sql server, After spending few hours i found this way to build the connection string for mssql. I have mainly tested it with Azure MSSQL with SQL Server authentication and able to index the schema into neo4j.

Changes made in this file https://github.com/amundsen-io/amundsen/blob/main/databuilder/example/scripts/sample_mssql_metadata.py

Summary of Changes

Tests

Documentation

CheckList

Make sure you have checked all steps below to ensure a timely review.

• [ ] PR title addresses the issue accurately and concisely. Example: "Updates the version of Flask to v1.0.2"
  • In case you are adding a dependency, check if the license complies with the ASF 3rd Party License Policy.
• [ ] PR includes a summary of changes.
• [ ] PR adds unit tests, updates existing unit tests, OR documents why no test additions or modifications are needed.
• [ ] In case of new functionality, my PR adds documentation that describes how to use it.
  • All the public functions and the classes in the PR contain docstrings that explain what it does

fixes: #2066

Summary of Changes

Changes made to: https://github.com/amundsen-io/amundsen/blob/main/databuilder/example/scripts/sample_mysql_loader.py

import pymysql
pymysql.install_as_MySQLdb()

def connection_string():
    user = 'root'
    password='root'
    host = 'localhost'
    port = '3307'
    db = 'test_db'
    return "mysql+pymysql://%s:%s@%s:%s/%s" % (user,password, host, port, db)

job_config = ConfigFactory.from_dict({
        f'extractor.mysql_metadata.{MysqlMetadataExtractor.WHERE_CLAUSE_SUFFIX_KEY}': where_clause_suffix,
        f'extractor.mysql_metadata.{MysqlMetadataExtractor.USE_CATALOG_AS_CLUSTER_NAME}': True,
        f'extractor.mysql_metadata.extractor.sqlalchemy.{SQLAlchemyExtractor.CONN_STRING}': connection_string(),
        f'loader.filesystem_csv_neo4j.{FsNeo4jCSVLoader.NODE_DIR_PATH}': node_files_folder,
        f'loader.filesystem_csv_neo4j.{FsNeo4jCSVLoader.RELATION_DIR_PATH}': relationship_files_folder,
        f'publisher.neo4j.{neo4j_csv_publisher.NODE_FILES_DIR}': node_files_folder,
        f'publisher.neo4j.{neo4j_csv_publisher.RELATION_FILES_DIR}': relationship_files_folder,
        f'publisher.neo4j.{neo4j_csv_publisher.NEO4J_END_POINT_KEY}': neo4j_endpoint,
        f'publisher.neo4j.{neo4j_csv_publisher.NEO4J_USER}': neo4j_user,
        f'publisher.neo4j.{neo4j_csv_publisher.NEO4J_PASSWORD}': neo4j_password,
        f'publisher.neo4j.{neo4j_csv_publisher.NEO4J_ENCRYPTED}': False,
        f'publisher.neo4j.{neo4j_csv_publisher.JOB_PUBLISH_TAG}': 'unique_tag',  # should use unique tag here like {ds}
    })

Changes made to : https://github.com/amundsen-io/amundsen/blob/main/databuilder/setup.py

requirements_path = os.path.join(os.path.dirname(os.path.realpath(__file__)),
                                 '../requirements.txt')

CheckList

Make sure you have checked all steps below to ensure a timely review.

• [x] PR title addresses the issue accurately and concisely. Example: "Updates the version of Flask to v1.0.2"
  • In case you are adding a dependency, check if the license complies with the ASF 3rd Party License Policy.
• [x] PR includes a summary of changes.
• [ ] PR adds unit tests, updates existing unit tests, OR documents why no test additions or modifications are needed.
• [ ] In case of new functionality, my PR adds documentation that describes how to use it.
  • All the public functions and the classes in the PR contain docstrings that explain what it does

Expected Behavior

Changed the connection string in https://github.com/amundsen-io/amundsen/blob/main/databuilder/example/scripts/sample_mysql_loader.py to load locally hosted MySQL data into Amundsen . Changes made in above file:

import pymysql
pymysql.install_as_MySQLdb()

def connection_string():
    user = 'root'
    password='root'
    host = 'localhost'
    port = '3307'
    db = 'test_db'
    return "mysql+pymysql://%s:%s@%s:%s/%s" % (user,password, host, port, db)

Current Behavior

While running the python file, I get the following error: ERROR:neo4j:Failed to write data to connection IPv4Address(('127.0.0.1', 7687)) (IPv4Address(('127.0.0.1', 7687))).

I tried loading the sample data by running the https://github.com/amundsen-io/amundsen/blob/main/databuilder/example/scripts/sample_data_loader.py file and it worked.

Possible Solution

fix: Adding the below code to job_config in sample_mysql_loader.py fixed the issue. f'publisher.neo4j.{neo4j_csv_publisher.NEO4J_ENCRYPTED}': False So now the code looks like this:

job_config = ConfigFactory.from_dict({
        f'extractor.mysql_metadata.{MysqlMetadataExtractor.WHERE_CLAUSE_SUFFIX_KEY}': where_clause_suffix,
        f'extractor.mysql_metadata.{MysqlMetadataExtractor.USE_CATALOG_AS_CLUSTER_NAME}': True,
        f'extractor.mysql_metadata.extractor.sqlalchemy.{SQLAlchemyExtractor.CONN_STRING}': connection_string(),
        f'loader.filesystem_csv_neo4j.{FsNeo4jCSVLoader.NODE_DIR_PATH}': node_files_folder,
        f'loader.filesystem_csv_neo4j.{FsNeo4jCSVLoader.RELATION_DIR_PATH}': relationship_files_folder,
        f'publisher.neo4j.{neo4j_csv_publisher.NODE_FILES_DIR}': node_files_folder,
        f'publisher.neo4j.{neo4j_csv_publisher.RELATION_FILES_DIR}': relationship_files_folder,
        f'publisher.neo4j.{neo4j_csv_publisher.NEO4J_END_POINT_KEY}': neo4j_endpoint,
        f'publisher.neo4j.{neo4j_csv_publisher.NEO4J_USER}': neo4j_user,
        f'publisher.neo4j.{neo4j_csv_publisher.NEO4J_PASSWORD}': neo4j_password,
        f'publisher.neo4j.{neo4j_csv_publisher.NEO4J_ENCRYPTED}': False,
        f'publisher.neo4j.{neo4j_csv_publisher.JOB_PUBLISH_TAG}': 'unique_tag',  # should use unique tag here like {ds}
    })

Your Environment

• Amundsen databuilder version used: 7.4.3
• Deployment (k8s or native): native
• Link to your fork or repository: (https://github.com/MalavikaN1/amundsen)