Deploy a STAC API and a dynamic mosaic tiler API using AWS CDK.

Development Seed

Last update: Oct 30, 2022

Related tags

Third-party APIs Wrappers eoAPI

Overview

Earth Observation API

Deploy a STAC API and a dynamic mosaic tiler API using AWS CDK.

Source Code: https://github.com/developmentseed/eoAPI

Deployment

The stack is deployed by the AWS CDK utility. Under the hood, CDK will create the deployment packages required for AWS Lambda, upload it to AWS, and handle the creation of the Lambda and API Gateway resources.

Install CDK and connect to your AWS account. This step is only necessary once per AWS account.

# Download titiler repo
$ git clone https://github.com/developmentseed/eoapi.git

# install cdk dependencies
$ pip install -r deployment/requirements.txt
$ npm install

$ npm run cdk bootstrap # Deploys the CDK toolkit stack into an AWS environment

# in specific region
$ npm run cdk bootstrap aws://${AWS_ACCOUNT_ID}/eu-central-1

Pre-Generate CFN template

$ npm run cdk synth  # Synthesizes and prints the CloudFormation template for this stack

Update settings

Using environment variable or editing deployment/cdk/config.py

Deploy

$ EOAPI_STAGE=staging npm run cdk deploy eoapi-staging --profile {my-aws-profile}

# Deploy in specific region
$ AWS_DEFAULT_REGION=eu-central-1 AWS_REGION=eu-central-1 npm run cdk deploy eoapi-production --profile {my-aws-profile}

STAC + PgSTAC + TiTiler

The dyanic tiler deployed within eoAPI is built on top of titiler-pgstac and pgstac. It enables large scale mosaic based on results of STAC searches queries:

Local

You can launch the APIs locally using docker. This will start 3 services: database, eoapi.stac, eoapi.raster

$ git clone https://github.com/developmentseed/eoAPI.git
$ cd eoAPI
$ docker-compose build
$ docker-compose up

Project structure

 ├──demo/                  - items/collections and notebook
 ├──deployment/            - AWS CDK code
 └──src/eoapi/
    ├── raster/            - eoAPI raster Application package
    ├── stac/              - eoAPI stac Application package
    └── vector/            - eoAPI vector Application package [EXPERIMENTAL]

Contribution & Development

See CONTRIBUTING.md

License

See LICENSE

Authors

Created by Development Seed

See contributors for a listing of individual contributors.

Changes

See CHANGES.md.

Comments

mosaic table in PG database

Enable retrieving mosaic by name instead of mosaicid

name| mosaicid | minzoom | maxzoom | bounds | default assets | available assets -- | -- | -- | -- | -- | -- | -- "vincent.mosaic" | "ada12e921qduashdas" | 0 | 24 | [-180, -90, 180, 90] | ["cog"] | ["cog", "thumbnail", "raw"]

opened by vincentsarago 11

Tilejson endpoint redirects to the page which returns 400 Bad Request

$ curl -L -v "http://localhost:8081/collections/openaerialmap/items/6136495327f4170005130793/tilejson.json?tile_scale=1"
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 8081 (#0)
> GET /collections/openaerialmap/items/6136495327f4170005130793/tilejson.json?tile_scale=1 HTTP/1.1
> Host: localhost:8081
> User-Agent: Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0)
> Accept: */*
> Referer: 
> 
< HTTP/1.1 307 Temporary Redirect
< date: Thu, 09 Sep 2021 09:25:00 GMT
< server: uvicorn
< location: http://0.0.0.0:8082/stac/tilejson.json?url=stac%3A%2F%2FeyJpZCI6ICI2MTM2NDk1MzI3ZjQxNzAwMDUxMzA3OTMiLCAiYmJveCI6IFstODAuMzUwNDg2LCAyNy40MTgyODEsIC04MC4zNDYxNzQsIDI3LjQyNTg4NF0sICJ0eXBlIjogIkZlYXR1cmUiLCAibGlua3MiOiBbeyJyZWwiOiAiY29sbGVjdGlvbiIsICJocmVmIjogIm9wZW5hZXJpYWxtYXAiLCAidHlwZSI6ICJhcHBsaWNhdGlvbi9qc29uIn1dLCAiYXNzZXRzIjogeyJpbWFnZSI6IHsiaHJlZiI6ICJodHRwczovL29pbi1ob3Rvc20uczMuYW1hem9uYXdzLmNvbS82MTM2NDhjODI3ZjQxNzAwMDUxMzA3OTEvMC82MTM2NDhjODI3ZjQxNzAwMDUxMzA3OTIudGlmIiwgInR5cGUiOiAiaW1hZ2UvdGlmZjsgYXBwbGljYXRpb249Z2VvdGlmZjsgcHJvZmlsZT1jbG91ZC1vcHRpbWl6ZWQifX0sICJnZW9tZXRyeSI6IHsiYmJveCI6IFstODAuMzUwNDg2LCAyNy40MTgyODEsIC04MC4zNDYxNzQsIDI3LjQyNTg4NF0sICJ0eXBlIjogIk11bHRpUG9seWdvbiIsICJjb29yZGluYXRlcyI6IFtbW1stODAuMzUwNDQyLCAyNy40MjU4ODRdLCBbLTgwLjM1MDQ4NiwgMjcuNDE4MzAxXSwgWy04MC4zNDYyMTgsIDI3LjQxODI4MV0sIFstODAuMzQ2MTc0LCAyNy40MjU4NjRdLCBbLTgwLjM1MDQ0MiwgMjcuNDI1ODg0XV1dXX0sICJjb2xsZWN0aW9uIjogIm9wZW5hZXJpYWxtYXAiLCAicHJvcGVydGllcyI6IHsidGl0bGUiOiAiV2FsZ3JlZW5zIC0gMjAyMTowOTowNiAxMDoxNToxMSIsICJjb250YWN0IjogIldlcyBNYWd5YXIsd2VzbWFneWFyQGdtYWlsLmNvbSIsICJkYXRldGltZSI6IG51bGwsICJwbGF0Zm9ybSI6ICJ1YXYiLCAicHJvdmlkZXIiOiAiV2ViT0RNIiwgImVuZF9kYXRldGltZSI6ICIyMDIxLTA5LTA2VDA2OjE1OjExWiIsICJzdGFydF9kYXRldGltZSI6ICIyMDIxLTA5LTA2VDA1OjE1OjExWiJ9LCAic3RhY192ZXJzaW9uIjogIjEuMC4wIiwgInN0YWNfZXh0ZW5zaW9ucyI6IFtdfQ%3D%3D
< transfer-encoding: chunked
< 
* Ignoring the response-body
* Connection #0 to host localhost left intact
* Issue another request to this URL: 'http://0.0.0.0:8082/stac/tilejson.json?url=stac%3A%2F%2FeyJpZCI6ICI2MTM2NDk1MzI3ZjQxNzAwMDUxMzA3OTMiLCAiYmJveCI6IFstODAuMzUwNDg2LCAyNy40MTgyODEsIC04MC4zNDYxNzQsIDI3LjQyNTg4NF0sICJ0eXBlIjogIkZlYXR1cmUiLCAibGlua3MiOiBbeyJyZWwiOiAiY29sbGVjdGlvbiIsICJocmVmIjogIm9wZW5hZXJpYWxtYXAiLCAidHlwZSI6ICJhcHBsaWNhdGlvbi9qc29uIn1dLCAiYXNzZXRzIjogeyJpbWFnZSI6IHsiaHJlZiI6ICJodHRwczovL29pbi1ob3Rvc20uczMuYW1hem9uYXdzLmNvbS82MTM2NDhjODI3ZjQxNzAwMDUxMzA3OTEvMC82MTM2NDhjODI3ZjQxNzAwMDUxMzA3OTIudGlmIiwgInR5cGUiOiAiaW1hZ2UvdGlmZjsgYXBwbGljYXRpb249Z2VvdGlmZjsgcHJvZmlsZT1jbG91ZC1vcHRpbWl6ZWQifX0sICJnZW9tZXRyeSI6IHsiYmJveCI6IFstODAuMzUwNDg2LCAyNy40MTgyODEsIC04MC4zNDYxNzQsIDI3LjQyNTg4NF0sICJ0eXBlIjogIk11bHRpUG9seWdvbiIsICJjb29yZGluYXRlcyI6IFtbW1stODAuMzUwNDQyLCAyNy40MjU4ODRdLCBbLTgwLjM1MDQ4NiwgMjcuNDE4MzAxXSwgWy04MC4zNDYyMTgsIDI3LjQxODI4MV0sIFstODAuMzQ2MTc0LCAyNy40MjU4NjRdLCBbLTgwLjM1MDQ0MiwgMjcuNDI1ODg0XV1dXX0sICJjb2xsZWN0aW9uIjogIm9wZW5hZXJpYWxtYXAiLCAicHJvcGVydGllcyI6IHsidGl0bGUiOiAiV2FsZ3JlZW5zIC0gMjAyMTowOTowNiAxMDoxNToxMSIsICJjb250YWN0IjogIldlcyBNYWd5YXIsd2VzbWFneWFyQGdtYWlsLmNvbSIsICJkYXRldGltZSI6IG51bGwsICJwbGF0Zm9ybSI6ICJ1YXYiLCAicHJvdmlkZXIiOiAiV2ViT0RNIiwgImVuZF9kYXRldGltZSI6ICIyMDIxLTA5LTA2VDA2OjE1OjExWiIsICJzdGFydF9kYXRldGltZSI6ICIyMDIxLTA5LTA2VDA1OjE1OjExWiJ9LCAic3RhY192ZXJzaW9uIjogIjEuMC4wIiwgInN0YWNfZXh0ZW5zaW9ucyI6IFtdfQ%3D%3D'
*   Trying 0.0.0.0...
* TCP_NODELAY set
* Connected to 0.0.0.0 (127.0.0.1) port 8082 (#1)
> GET /stac/tilejson.json?url=stac%3A%2F%2FeyJpZCI6ICI2MTM2NDk1MzI3ZjQxNzAwMDUxMzA3OTMiLCAiYmJveCI6IFstODAuMzUwNDg2LCAyNy40MTgyODEsIC04MC4zNDYxNzQsIDI3LjQyNTg4NF0sICJ0eXBlIjogIkZlYXR1cmUiLCAibGlua3MiOiBbeyJyZWwiOiAiY29sbGVjdGlvbiIsICJocmVmIjogIm9wZW5hZXJpYWxtYXAiLCAidHlwZSI6ICJhcHBsaWNhdGlvbi9qc29uIn1dLCAiYXNzZXRzIjogeyJpbWFnZSI6IHsiaHJlZiI6ICJodHRwczovL29pbi1ob3Rvc20uczMuYW1hem9uYXdzLmNvbS82MTM2NDhjODI3ZjQxNzAwMDUxMzA3OTEvMC82MTM2NDhjODI3ZjQxNzAwMDUxMzA3OTIudGlmIiwgInR5cGUiOiAiaW1hZ2UvdGlmZjsgYXBwbGljYXRpb249Z2VvdGlmZjsgcHJvZmlsZT1jbG91ZC1vcHRpbWl6ZWQifX0sICJnZW9tZXRyeSI6IHsiYmJveCI6IFstODAuMzUwNDg2LCAyNy40MTgyODEsIC04MC4zNDYxNzQsIDI3LjQyNTg4NF0sICJ0eXBlIjogIk11bHRpUG9seWdvbiIsICJjb29yZGluYXRlcyI6IFtbW1stODAuMzUwNDQyLCAyNy40MjU4ODRdLCBbLTgwLjM1MDQ4NiwgMjcuNDE4MzAxXSwgWy04MC4zNDYyMTgsIDI3LjQxODI4MV0sIFstODAuMzQ2MTc0LCAyNy40MjU4NjRdLCBbLTgwLjM1MDQ0MiwgMjcuNDI1ODg0XV1dXX0sICJjb2xsZWN0aW9uIjogIm9wZW5hZXJpYWxtYXAiLCAicHJvcGVydGllcyI6IHsidGl0bGUiOiAiV2FsZ3JlZW5zIC0gMjAyMTowOTowNiAxMDoxNToxMSIsICJjb250YWN0IjogIldlcyBNYWd5YXIsd2VzbWFneWFyQGdtYWlsLmNvbSIsICJkYXRldGltZSI6IG51bGwsICJwbGF0Zm9ybSI6ICJ1YXYiLCAicHJvdmlkZXIiOiAiV2ViT0RNIiwgImVuZF9kYXRldGltZSI6ICIyMDIxLTA5LTA2VDA2OjE1OjExWiIsICJzdGFydF9kYXRldGltZSI6ICIyMDIxLTA5LTA2VDA1OjE1OjExWiJ9LCAic3RhY192ZXJzaW9uIjogIjEuMC4wIiwgInN0YWNfZXh0ZW5zaW9ucyI6IFtdfQ%3D%3D HTTP/1.1
> Host: 0.0.0.0:8082
> User-Agent: Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0)
> Accept: */*
> Referer: http://localhost:8081/collections/openaerialmap/items/6136495327f4170005130793/tilejson.json?tile_scale=1
> 
< HTTP/1.1 400 Bad Request
< date: Thu, 09 Sep 2021 09:25:00 GMT
< server: uvicorn
< content-length: 76
< content-type: application/json
< 
* Connection #1 to host 0.0.0.0 left intact
{"detail":"assets must be defined either via expression or assets options."}

opened by drnextgis 6

Added a publicly available basemap (token is not required)

By now in order to have a basemap for the search viewer user have to provide a mapbox token via environment variable. This PR adds a default basemap (Dark Matter basemaps from Carto) which doesn't require any preconfigurations.

opened by drnextgis 3
Use "black" profile when running isort (commit hooks)

There are some incompatibilities between the standard configurations for isort and black (ref.

To avoid issues of black and isort continuously reformatting the same file, add the --profile black option to isort

opened by leothomas 2

Error on demo data ingestion

Hi! I'm trying to upload some data to the database according to this manual:

$ pypgstac version --dsn postgresql://username:password@localhost:5432/postgis
0.3.4

$ pypgstac load collections oam_collection.json --dsn postgresql://username:password@localhost:5432/postgis --method insert
loading oam_collection.json into collections using insert
Traceback (most recent call last):
  File "/usr/local/bin/pypgstac", line 5, in <module>
    app()
  File "/usr/local/lib/python3.7/dist-packages/typer/main.py", line 214, in __call__
    return get_command(self)(*args, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/typer/main.py", line 497, in wrapper
    return callback(**use_params)  # type: ignore
  File "/opt/src/pypgstac/pypgstac/pypgstac.py", line 43, in load
    typer.echo(asyncio.run(load_ndjson(file=file, table=table, dsn=dsn, method=method)))
  File "/usr/lib/python3.7/asyncio/runners.py", line 43, in run
    return loop.run_until_complete(main)
  File "/usr/lib/python3.7/asyncio/base_events.py", line 584, in run_until_complete
    return future.result()
  File "/opt/src/pypgstac/pypgstac/load.py", line 246, in load_ndjson
    await load_iterator(f, table, conn, method)
  File "/opt/src/pypgstac/pypgstac/load.py", line 230, in load_iterator
    await copy(iter, table, conn)
  File "/opt/src/pypgstac/pypgstac/load.py", line 125, in copy
    delimiter=chr(31),
  File "/usr/local/lib/python3.7/dist-packages/asyncpg/connection.py", line 859, in copy_to_table
    return await self._copy_in(copy_stmt, source, timeout)
  File "/usr/local/lib/python3.7/dist-packages/asyncpg/connection.py", line 1042, in _copy_in
    copy_stmt, reader, data, None, None, timeout)
  File "asyncpg/protocol/protocol.pyx", line 506, in copy_in
asyncpg.exceptions.InvalidTextRepresentationError: invalid input syntax for type json
DETAIL:  The input string ended unexpectedly.

opened by drnextgis 2

Update README.md and Dockerfile.stac pip install issue

this updates the readme so the code box can be copied and run in the terminal without error due to the prompt symbol included in the code box $

it also addresses the issues listed in https://github.com/developmentseed/eoAPI/issues/49

opened by rbavery 1
set `db_max_conn_size` to 1 when deployed in AWS Lambda

lambda doesn't need to open a pool of connection because we have a 1:1 request/lambda behaviour

stac-fastapi: db_min_conn_size / db_max_conn_size timvt: db_min_conn_size / db_max_conn_size tifeatures: db_min_conn_size / db_max_conn_size titiler-pgstac: db_min_conn_size / db_max_conn_size

opened by vincentsarago 1
Fix tuple destructuring overriding class id member.
What I am changing

The variable name used in tuple de-structuring is overriding the class id member resulting in incorrect resource names within the stack. For a simplified example

id = "an_id" print(id) gateway_endpoints = [("S3", "s3_service")] for (id, service) in gateway_endpoints: print(service) print(id)

How I did it

Changed the name of the de-structuring variable to key

How you can test it

Deploy the stack.
opened by sharkinsspatial 1
add mosaic search + other updates
This PR adds a mosaic/list endpoint to list/search mosaic registered in PgSTAC database

ref:

https://github.com/developmentseed/eoAPI/issues/22

https://github.com/stac-utils/titiler-pgstac/pull/45

Other fix/additions

Pin mangum to >0.14 (ref https://github.com/developmentseed/eoAPI/issues/34)

update PgSTAC version (https://github.com/developmentseed/eoAPI/issues/35)

Update titiler-pgstac version

breaking changes

add CDK_ prefix to deployment related environment variables
opened by vincentsarago 1

Pin markupsafe version

https://github.com/pallets/markupsafe/issues/286

>>> import jinja2
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.9/site-packages/jinja2/__init__.py", line 12, in <module>
    from .environment import Environment
  File "/usr/local/lib/python3.9/site-packages/jinja2/environment.py", line 25, in <module>
    from .defaults import BLOCK_END_STRING
  File "/usr/local/lib/python3.9/site-packages/jinja2/defaults.py", line 3, in <module>
    from .filters import FILTERS as DEFAULT_FILTERS  # noqa: F401
  File "/usr/local/lib/python3.9/site-packages/jinja2/filters.py", line 13, in <module>
    from markupsafe import soft_unicode
ImportError: cannot import name 'soft_unicode' from 'markupsafe' (/usr/local/lib/python3.9/site-packages/markupsafe/__init__.py)

opened by drnextgis 1

Context STAC extension

Context extension (to be more precise its matched property) is disabled by default in PGStac. I think we should find a way to make it configurable.

It seems that it can be easily added to db_handler but what about database service in the docker-compose?

opened by drnextgis 1
replace vector and feature service by Tipg
When we started we added a simple OGC feature service which is almost a 1:1 with the stac service so it's not really useful. The vector service tried to create MVT from stac searches which could be a nice feature but maybe not scale.

Let's remove those two service and replace them by a OGC Feature/Tile service based on https://github.com/developmentseed/tipg (but not connected to the pgstac database (for now)

Services

pgstac db

stac (stac-fastapi) -> pgstac schema in db

raster (titiler-pgstac) -> pgstac schema in db

vector (tipg) -> public schema in db (we could add the possibility to connect to pgstac schema too)
opened by vincentsarago 0
[CDK] raster <-> stac API connection

using STAC item URL (from eoAPI.stac service) as input of the eoAPI.raster service result in timeout meaning the raster service can't access the STAC service 🤷‍♂️

opened by vincentsarago 0
Issue 1

Goal: Create a one command line deployment stack with stac-fastapi and titiler How: Custom stac-fastapi and titiler code + full CDK app Why: Ease setup of stac-fastapi and titiler

opened by vincentsarago 0

Deploy a STAC API and a dynamic mosaic tiler API using AWS CDK.

Related tags

Overview

Earth Observation API

Deployment

STAC + PgSTAC + TiTiler

Local

Project structure

Contribution & Development

License

Authors

Changes

Comments

What I am changing

How I did it

How you can test it

Services

Owner

Development Seed

CloudFormation template and CDK stack that contains a CustomResource with Lambda function to allow the setting of the targetAccountIds attribute of the EC2 Image Builder AMI distribution settings which is not currently supported (as of October 2021) in CloudFormation or CDK.

Cdk-python-crud-app - CDK Python CRUD App

Project template for using aws-cdk, Chalice and React in concert, including RDS Postgresql and AWS Cognito

AWS Blog post code for running feature-extraction on images using AWS Batch and Cloud Development Kit (CDK).

DIAL(Did I Alert Lambda?) is a centralised security misconfiguration detection framework which completely runs on AWS Managed services like AWS API Gateway, AWS Event Bridge & AWS Lambda

Recommended AWS CDK project structure for Python applications

Criando Lambda Functions para Ingerir Dados de APIs com AWS CDK

This repository contains code written in the AWS Cloud Development Kit (CDK)

Implement backup and recovery with AWS Backup across your AWS Organizations using a CI/CD pipeline (AWS CodePipeline).

Automated AWS account hardening with AWS Control Tower and AWS Step Functions

Multi-Branch CI/CD Pipeline using CDK Pipelines.

This repository contains ready to deploy automations on AWS

Create CDK projects with projen

Aws-lambda-requests-wrapper - Request/Response wrapper for AWS Lambda with API Gateway

AWS Auto Inventory allows you to quickly and easily generate inventory reports of your AWS resources.

A suite of utilities for AWS Lambda Functions that makes tracing with AWS X-Ray, structured logging and creating custom metrics asynchronously easier

Unauthenticated enumeration of services, roles, and users in an AWS account or in every AWS account in existence.

Automatically compile an AWS Service Control Policy that ONLY allows AWS services that are compliant with your preferred compliance frameworks.