The issue is relatively subtle and obvious at the same time:

The current state of the world is read from the endpoints, not from the models. Plus, we cannot update an already creating/updating endpoint. So, when you tag a new version for production while its destination endpoint is creating/updating:

• Bridge sees the new version is missing from the current state of the world and creates a model for it in SageMaker
• Bridge sees it cannot update the state of the endpoints and skips
• On the next iteration, the state of the world has not changed, so Bridge attempts the exact same control loop actions. But now a model exists. So you get an error. Root cause: model version creation a) not idempotent and b) not done in a 'transaction' with endpoint updates. Solved A instead of B.

Testing revealed a similar bug with deletion:

• Start with endpoint in S1.
• Update to S2, update still in progress.
• Update to S3. S2 is no longer active/needed and is deleted but the endpoint state still has S2. So, on the next control loop run Bridge tries to delete s2 again. Solution is the same, idempotency

I added language around "syncing model versions tagged as production" - not sure if this is accurate, but something like it felt necessary to add context around how MLFlow tags are consumed and updated to Sagemaker.

This configures s3fs, but also provides a good framework for configuring JupyterLab (via config overrides and startup scripts baked into the container image).

I've gone ahead and installed gator, but its probably not completely set up right.

I ran into some strange issues with AWS rejecting queries for some buckets in regions other than us-east-1, but not others. Just to be safe, I recommend that users use us-east-1.

Ticket Link

Description:

• What is this PR about: The description box presented content in what looked like a cut off container. The issue was that the content received scroll bars by default even when no scrolling was required.
• What are the changes that were made: Set overflow to auto instead of scroll
• Any issues you ran into and how it affected your approach: The main issue was the fact that in local development I had no access to the page with the display component. I ended up copying this component into one I did have access to to troubleshoot and test.
• Screenshots if applicable:

Before: Now:

Testing/Reproduction:

How to test this: This should be tested with a PR that:

• has not description (no scroll should appear)
• has a short description (no scroll should appear)
• has a long description (should be able to scroll)

Verified that we can create the reports we need can be generated in Mixpanel. They actually have a pretty cool UI.

Design decisions:

• Plan for future deploy targets by adding a kind and id as properties of the DeployTarget class.
• Route everything through a central analytics module
  • Allow analytics to be turned on/off centrally
  • Allow async event transmission in the future
• Per-event tracking helper methods
  • Encapsulate event formatting/definitions to one file where they can be centrally managed
  • Expose strongly typed API to callers
• Event-oriented design (capture version and routing creation/deletion as individual events not in one state bundle)
• Easier analytics - track how many stage changes are happening as a direct count of model_routing_updated events
• Only way to do it... we can't actually infer much from the current and desired routing maps. Either need to calculate count-based routing updates in the main app or delegate to the deploy target classes that do actually make this calculation.

TODO:

• Document the opt-out environment var in docs + logs
• Bake the environment variable into the docker image in quay as a Docker arg (to avoid the api key getting scraped from the open source repo)
• Report package version in events so that we can measure version skew

Ticket Link

Description:

• What is this PR about: We wanted to simplify the FE template code structure, and decided to get rid of the packages/ui folder that made the build more cumbersome.
• What are the changes that were made: -Components from packages/ui copied over to roots src directory. -packages/ui folder was deleted with all its contents. -Some files in root were merged with correlating files from packages/ui. -package.json in root updated -storybook adjusted

Successfully rebuilt the code env locally from this branch before opening this PR.

Id love to get any feedback on this

Makes the local MLflow into a one step process. I decided to do this by wrapping up the example model training code into a docker image that is run as a service in docker compose. Using docker gets past the nasty python environment issue which would likely trip up many users

Is introducing seeding into compose smart?

Keeping it in compose has the major advantage that all the credentials config and networking is in one file. Vs a script that would need to be run in a configured environment to have access to the minio bucket etc.

We could still decouple by removing the seeding from the CMD of the image and running it later with docker-compose run. But I think in > 95% of cases where the local mlflow is used, people will want a seeded registry to try one of our tools. So I think just doing it all at once is actually the best approach.

Chrome has decided to have a magic list of ports that we cannot use on localhost. This includes 6000. We should move the quickstart to use a different port that isn't on the magic list.

https://superuser.com/questions/188058/which-ports-are-considered-unsafe-by-chrome

Killing the container requires repeated ctrl-c signals - looks like the first one kills the main loop in Bridge and the next is caught by a python thread

This looks like the error

Traceback (most recent call last):
  File "/opt/conda/lib/python3.9/site-packages/flask/app.py", line 2070, in wsgi_app
    response = self.full_dispatch_request()
  File "/opt/conda/lib/python3.9/site-packages/flask/app.py", line 1516, in full_dispatch_request
    return self.finalize_request(rv)
  File "/opt/conda/lib/python3.9/site-packages/flask/app.py", line 1535, in finalize_request
    response = self.make_response(rv)
  File "/opt/conda/lib/python3.9/site-packages/flask/app.py", line 1727, in make_response
    raise TypeError(
TypeError: The view function did not return a valid response. The return type must be a string, dict, tuple, Response instance, or WSGI callable, but it was a int.

See if we can pull the S3 endpoint that the registry is using so that we can avoid duplicating this config across the registry and Bridge itself when we use a non-s3 backend

Replication:

1. Add a couple model versions to registry
2. Init bridge in a fresh account
3. Bridge creates an endpoint/model
4. Before bridge is done creating the endpoint, add and tag a new version destined for the endpoint
5. Bridge skips the update.
6. After some time (perhaps when endpoint becomes updateable) bridge attempts update and then fails due to existing model.