Splitgraph
Overview
Splitgraph is a tool for building, versioning and querying reproducible datasets. It's inspired by Docker and Git, so it feels familiar. And it's powered by PostgreSQL, so it works seamlessly with existing tools in the Postgres ecosystem. Use Splitgraph to package your data into self-contained data images that you can share with other Splitgraph instances.
Splitgraph.com, or Splitgraph Cloud, is a public Splitgraph instance where you can share and discover data. It's a Splitgraph peer powered by the Splitgraph Core code in this repository, adding proprietary features like a data catalog, multitenancy, and a distributed SQL proxy.
You can explore 40k+ open datasets in the catalog. You can also connect directly to the Data Delivery Network and query any of the datasets, without installing anything.
To install sgr (the command line client) or a local Splitgraph Engine, see the Installation section of this readme.
Build and Query Versioned, Reproducible Datasets
Splitfiles give you a declarative language, inspired by Dockerfiles, for expressing data transformations in ordinary SQL familiar to any researcher or business analyst. You can reference other images, or even other databases, with a simple JOIN.
When you build data with Splitfiles, you get provenance tracking of the resulting data: it's possible to find out what sources went into every dataset and know when to rebuild it if the sources ever change. You can easily integrate Splitgraph into your existing CI pipelines, to keep your data up-to-date and stay on top of changes to upstream sources.
Splitgraph images are also version-controlled, and you can manipulate them with Git-like operations through a CLI. You can check out any image into a PostgreSQL schema and interact with it using any PostgreSQL client. Splitgraph will capture your changes to the data, and then you can commit them as delta-compressed changesets that you can package into new images.
Splitgraph supports PostgreSQL foreign data wrappers. We call this feature mounting. With mounting, you can query other databases (like PostgreSQL/MongoDB/MySQL) or open data providers (like Socrata) from your Splitgraph instance with plain SQL. You can even snapshot the results or use them in Splitfiles.
Why Splitgraph?
Splitgraph isn't opinionated and doesn't break existing abstractions. To any existing PostgreSQL application, Splitgraph images are just another database. We have carefully designed Splitgraph to not break the abstraction of a PostgreSQL table and wire protocol, because doing otherwise would mean throwing away a vast existing ecosystem of applications, users, libraries and extensions. This means that a lot of tools that work with PostgreSQL work with Splitgraph out of the box.
Components
The code in this repository, known as Splitgraph Core, contains:
sgrcommand line client:sgris the main command line tool used to work with Splitgraph "images" (data snapshots). Use it to ingest data, work with splitfiles, and push data to Splitgraph.com.- Splitgraph Engine: a Docker image of the latest Postgres with Splitgraph and other required extensions pre-installed.
- Splitgraph Python library: All Splitgraph functionality is available in the Python API, offering first-class support for data science workflows including Jupyter notebooks and Pandas dataframes.
Docs
Documentation is available at https://www.splitgraph.com/docs, specifically:
We also recommend reading our Blog, including some of our favorite posts:
- Supercharging
dbtwith Splitgraph: versioning, sharing, cross-DB joins - Querying 40,000+ datasets with SQL
- Foreign data wrappers: PostgreSQL's secret weapon?
Installation
Pre-requisites:
- Docker is required to run the Splitgraph Engine.
sgrmust have access to Docker. You either need to install Docker locally or have access to a remote Docker socket.
For Linux and OSX, once Docker is running, install Splitgraph with a single script:
$ bash -c "$(curl -sL https://github.com/splitgraph/splitgraph/releases/latest/download/install.sh)"
This will download the sgr binary and set up the Splitgraph Engine Docker container.
Alternatively, you can get the sgr single binary from the releases page and run sgr engine add to create an engine.
See the installation guide for more installation methods.
Quick start guide
You can follow the quick start guide that will guide you through the basics of using Splitgraph with public and private data.
Alternatively, Splitgraph comes with plenty of examples to get you started.
If you're stuck or have any questions, check out the documentation or join our Discord channel!
Contributing
Setting up a development environment
- Splitgraph requires Python 3.6 or later.
- Install Poetry:
curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py | pythonto manage dependencies - Install pre-commit hooks (we use Black to format code)
git clone --recurse-submodules https://github.com/splitgraph/splitgraph.gitpoetry install- To build the engine Docker image:
cd engine && make
Running tests
The test suite requires docker-compose. You will also need to add these lines to your /etc/hosts or equivalent:
127.0.0.1 local_engine
127.0.0.1 remote_engine
127.0.0.1 objectstorage
To run the core test suite, do
docker-compose -f test/architecture/docker-compose.core.yml up -d
poetry run pytest -m "not mounting and not example"
To run the test suite related to "mounting" and importing data from other databases (PostgreSQL, MySQL, Mongo), do
docker-compose -f test/architecture/docker-compose.core.yml -f test/architecture/docker-compose.mounting.yml up -d
poetry run pytest -m mounting
Finally, to test the example projects, do
# Example projects spin up their own engines
docker-compose -f test/architecture/docker-compose.core.yml -f test/architecture/docker-compose.core.yml down -v
poetry run pytest -m example
All of these tests run in CI.
17 Dec 11, 2022
3 Feb 14, 2022
25.4k Dec 30, 2022
79 Dec 2, 2022
8 Jan 9, 2023
10 Dec 24, 2022
14.5k Jan 3, 2023
37 Dec 15, 2022
5.8k Jan 3, 2023
1.3k Dec 28, 2022
112 Nov 5, 2022
36 Dec 15, 2022
808 Jan 4, 2023
108 Dec 26, 2022
4 Dec 9, 2022
16 Sep 25, 2022
22 Dec 5, 2022
5 Dec 27, 2022