Django models and endpoints for working with large images -- tile serving

Last update: Dec 17, 2022

Comments

EPSG and its effect on COG and Tiles Opacity

Great work with the library.

I have an issue with opacity/clarity of tiles, thumbnails and overall Geotiffs/COGs as well as COGs not getting tiled.

Opacity Issue: When using the swagger API to get a thumbnail of an existing COG, if I include "EPSG:3857" on the request, I get a washed out image

However, if I omit the EPSG Code, I get a clear image thumbnail. I have tested with multiple COGs and raw GeoTiffs

In Django Admin, the same Geotiffs and COGs are also washed out

I have also used a separate Leaflet html page to access the /api/maps/{id}/{z}/{x}/{y}.png and I get the same washed out images

<html>
  <head>
    <meta name="viewport" content="initial-scale=1.0, user-scalable=no" />
    <meta charset="utf-8" />
    <style>
      html,
      body {
        height: 100%;
        margin: 0;
        padding: 0;
      }
      #map {
        height: 100%;
      }
    </style>
    <link
      rel="stylesheet"
      href="https://unpkg.com/[email protected]/dist/leaflet.css"
      integrity="sha512-hoalWLoI8r4UszCkZ5kL8vayOGVae1oxXe/2A4AO6J9+580uKHDO3JdHb7NzwwzK5xr/Fs0W40kiNHxM9vyTtQ=="
      crossorigin=""
    />
    <script
      src="https://unpkg.com/[email protected]/dist/leaflet.js"
      integrity="sha512-BB3hKbKWOc9Ez/TAwyWxNXeoV9c1v6FIeYiBieIWkpLjauysF18NzgR1MBNBXf8/KABdlkX68nAhlwcDFLGPCQ=="
      crossorigin=""
    ></script>
  </head>
  <body>
    <div id="map"></div>
    <script>
      var map = L.map("map").setView(
        [-0.24511062205544606, 34.87011481076246],
        16
      );
      L.tileLayer("https://{s}.tile.openstreetmap.org/{z}/{x}/{y}.png", {
        attribution:
          '&copy; <a href="https://www.openstreetmap.org/copyright">OpenStreetMap</a> contributors',
        maxZoom: 18,
      }).addTo(map);
      L.tileLayer(
        "http://localhost:8000/api/maps/1/tiles/{z}/{x}/{y}.png?projection=EPSG%3A3857",
        {
          attribution:
            '&copy; <a href="https://www.openstreetmap.org/copyright">OpenStreetMap</a> contributors',
          maxZoom: 23,
          opacity: 1,
        }
      ).addTo(map);

    </script>
  </body>
</html>

However, in the above example, if I do not append the EPSG code on the url in html, then I get an error (no map rendered) and a 400 bad request

As for individual tiles, if I include the EPSG Code in the request body in swagger UI, i get a washed out tile

However, if I omit the EPSG code, I get a bad request error

An example COG metadata is

{
  "geospatial": true,
  "levels": 8,
  "sizeX": 32465,
  "sizeY": 28586,
  "sourceLevels": 8,
  "sourceSizeX": 32465,
  "sourceSizeY": 28586,
  "tileWidth": 256,
  "tileHeight": 256,
  "bounds": {
    "ll": {
      "x": 34.85392278135806,
      "y": -0.23501171685313998
    },
    "ul": {
      "x": 34.85392278135806,
      "y": -0.2273338589976692
    },
    "lr": {
      "x": 34.86258419062559,
      "y": -0.23501171685313998
    },
    "ur": {
      "x": 34.86258419062559,
      "y": -0.2273338589976692
    },
    "srs": "EPSG:4326",
    "xmin": 34.85392278135806,
    "xmax": 34.86258419062559,
    "ymin": -0.23501171685313998,
    "ymax": -0.2273338589976692
  },
  "sourceBounds": {
    "ll": {
      "x": 34.85392278135806,
      "y": -0.23501171685313998
    },
    "ul": {
      "x": 34.85392278135806,
      "y": -0.2273338589976692
    },
    "lr": {
      "x": 34.86258419062559,
      "y": -0.23501171685313998
    },
    "ur": {
      "x": 34.86258419062559,
      "y": -0.2273338589976692
    },
    "srs": "+proj=longlat +ellps=WGS84 +towgs84=0,0,0,0,0,0,0 +no_defs",
    "xmin": 34.85392278135806,
    "xmax": 34.86258419062559,
    "ymin": -0.23501171685313998,
    "ymax": -0.2273338589976692
  },
  "bands": {
    "1": {
      "min": 55,
      "max": 255,
      "mean": 193.05740298087738,
      "stdev": 72.13546456636203,
      "units": "metre",
      "interpretation": "red",
      "maskband": 4
    },
    "2": {
      "min": 63,
      "max": 255,
      "mean": 194.14802446569178,
      "stdev": 70.6158617103163,
      "units": "metre",
      "interpretation": "green",
      "maskband": 4
    },
    "3": {
      "min": 41,
      "max": 255,
      "mean": 182.4484673790776,
      "stdev": 84.39825325395805,
      "units": "metre",
      "interpretation": "blue",
      "maskband": 4
    },
    "4": {
      "min": 0,
      "max": 255,
      "mean": 109.17366774465692,
      "stdev": 126.17604981162934,
      "units": "metre",
      "interpretation": "alpha"
    }
  },
  "magnification": null,
  "mm_x": 2.969893339423529,
  "mm_y": 2.969893339423529,
  "proj4": [
    "+proj=longlat +ellps=WGS84 +towgs84=0,0,0,0,0,0,0 +no_defs"
  ],
  "frames": false
}

bug

opened by geoffreynyaga 7

Add back Base64 styling support

This reverts commit 3c74b7f37a5d91f68048fe5a907eedb2ead734f0.

Take the following style object:

{
    "bands": [
        {
            "frame": 0,
            "palette": "#ffffff"
        },
        {
            "frame": 1,
            "palette": "#ffffff"
        },
        {
            "frame": 2,
            "palette": "#ffffff"
        },
        {
            "frame": 3,
            "palette": "#ffffff"
        },
        {
            "frame": 4,
            "palette": "#ffffff"
        },
        {
            "frame": 5,
            "palette": "#ffffff"
        },
        {
            "frame": 6,
            "palette": "#ffffff"
        },
        {
            "frame": 7,
            "palette": "#ffffff"
        },
        {
            "frame": 8,
            "palette": "#ffffff"
        },
        {
            "frame": 9,
            "palette": "#ffffff"
        },
        {
            "frame": 10,
            "palette": "#ffffff"
        },
        {
            "frame": 11,
            "palette": "#ffffff"
        }
    ]
}

Base64 encoding of this object is smaller than the URI encoded version (and is so in all similar examples I have tried)

JSON Stringify (length 397):

{"bands":[{"frame":0,"palette":"#ffffff"},{"frame":1,"palette":"#ffffff"},{"frame":2,"palette":"#ffffff"},{"frame":3,"palette":"#ffffff"},{"frame":4,"palette":"#ffffff"},{"frame":5,"palette":"#ffffff"},{"frame":6,"palette":"#ffffff"},{"frame":7,"palette":"#ffffff"},{"frame":8,"palette":"#ffffff"},{"frame":9,"palette":"#ffffff"},{"frame":10,"palette":"#ffffff"},{"frame":11,"palette":"#ffffff"}]}

Base64 (length 532):

eyJiYW5kcyI6W3siZnJhbWUiOjAsInBhbGV0dGUiOiIjZmZmZmZmIn0seyJmcmFtZSI6MSwicGFsZXR0ZSI6IiNmZmZmZmYifSx7ImZyYW1lIjoyLCJwYWxldHRlIjoiI2ZmZmZmZiJ9LHsiZnJhbWUiOjMsInBhbGV0dGUiOiIjZmZmZmZmIn0seyJmcmFtZSI6NCwicGFsZXR0ZSI6IiNmZmZmZmYifSx7ImZyYW1lIjo1LCJwYWxldHRlIjoiI2ZmZmZmZiJ9LHsiZnJhbWUiOjYsInBhbGV0dGUiOiIjZmZmZmZmIn0seyJmcmFtZSI6NywicGFsZXR0ZSI6IiNmZmZmZmYifSx7ImZyYW1lIjo4LCJwYWxldHRlIjoiI2ZmZmZmZiJ9LHsiZnJhbWUiOjksInBhbGV0dGUiOiIjZmZmZmZmIn0seyJmcmFtZSI6MTAsInBhbGV0dGUiOiIjZmZmZmZmIn0seyJmcmFtZSI6MTEsInBhbGV0dGUiOiIjZmZmZmZmIn1dfQ==

encodeURIComponent (length 721):

%7B%22bands%22%3A%5B%7B%22frame%22%3A0%2C%22palette%22%3A%22%23ffffff%22%7D%2C%7B%22frame%22%3A1%2C%22palette%22%3A%22%23ffffff%22%7D%2C%7B%22frame%22%3A2%2C%22palette%22%3A%22%23ffffff%22%7D%2C%7B%22frame%22%3A3%2C%22palette%22%3A%22%23ffffff%22%7D%2C%7B%22frame%22%3A4%2C%22palette%22%3A%22%23ffffff%22%7D%2C%7B%22frame%22%3A5%2C%22palette%22%3A%22%23ffffff%22%7D%2C%7B%22frame%22%3A6%2C%22palette%22%3A%22%23ffffff%22%7D%2C%7B%22frame%22%3A7%2C%22palette%22%3A%22%23ffffff%22%7D%2C%7B%22frame%22%3A8%2C%22palette%22%3A%22%23ffffff%22%7D%2C%7B%22frame%22%3A9%2C%22palette%22%3A%22%23ffffff%22%7D%2C%7B%22frame%22%3A10%2C%22palette%22%3A%22%23ffffff%22%7D%2C%7B%22frame%22%3A11%2C%22palette%22%3A%22%23ffffff%22%7D%5D%7D

If syles get more complex, they may quickly approach the common limit of 1024 characters for query strings. Having support for Base64 will give users a little more wiggle room to make sure their styles don't exceed limits set by their web server.

Related: https://github.com/atlascope/atlascope/pull/139#pullrequestreview-946184928

opened by banesullivan 2

Use django cache with large-image
requires https://github.com/girder/large_image/pull/876

resolves #2

To do:

[x] add some sort of test that will validate that large-image is really using the Django cache
opened by banesullivan 1
Use custom format handler and regroup endpoints

Resolves #22

~This is still a work in progress as these changes highlight an issue in drf-yasg I need to fully document/capture~ not addressing this

This introduces breaking changes to the API endpoints:

Endpoints are now grouped into info, data, and tiles subgroups.

I could implement a deprecated version of the API, but considering we're still in the early days and most projects are pinning versions. This shouldn't matter too much and I will do the migration manually

opened by banesullivan 1
Add specification of color to UI palette choice

The palette UI currently only allows for the selection of an MPL colormap - modify this to support hex colors (and/or everything large-image can support)

opened by banesullivan 1
Can we embed warnings/error in tile response body?
Sometimes we get warnings like:

Tiff image is missing many lower resolution levels (3). It will be inefficient to read lower resolution tiles.

ref https://sentry.io/organizations/django-large-image/issues/3131476233/?project=6272210&query=is%3Aunresolved

and in the future, we may warn if an image is not a properly tiled COG or pyramidal tiff (see #4).

It would be great if we could embed these warnings in the tile and/or metadata endpoint responses. While in many cases, these warnings will go unnoticed, in some of our custom applications we can monitor these and even create handlers to provide a toast to the user that says something along the lines of "Performance warning: -insert warning-"
opened by banesullivan 1
Performance issues

I'm seeing significant poor performance on the production demo. This could be due to the images not being proper COGs or something odd with how files are being checked out, causing requests to not be parallel

opened by banesullivan 1
FileExistsError with os.symlink
https://github.com/girder/django-large-image/blob/43c9ef62d83c4d6f33d3b49cc41f2c678f296584/django_large_image/utilities.py#L137

FileExistsError: [Errno 17] File exists: '/home/geoff/LargeImage/media/data/test.tif' -> '/tmp/django-large-image/file_cache/ImageFile-1/test.tif'
bug
opened by banesullivan 0
Add renderers and format extension handling
Can we add renderers (see details at bottom) and remove the duplicate _png, _jpeg, and _tif variants of many of the endpoints so that they are one endpoint that accepts different formats? I know this is possible, but I haven't been able to get it working with DRF 😕

For example, we have the thumbnail endpoint split as two endpoints:

/image-file/{id}/thumbnail.png

/image-file/{id}/thumbnail.jpeg

Can we combine these to: /image-file/{id}/thumbnail.{format}

Thus reducing the method shims for these endpoints like

https://github.com/girder/django-large-image/blob/09ef6e823d78de9a304cebbba453cb4a46b53ac2/django_large_image/rest/data.py#L38

and

https://github.com/girder/django-large-image/blob/09ef6e823d78de9a304cebbba453cb4a46b53ac2/django_large_image/rest/data.py#L48

So that we are only using

https://github.com/girder/django-large-image/blob/09ef6e823d78de9a304cebbba453cb4a46b53ac2/django_large_image/rest/data.py#L25

with the format argument.

So far I have been unsuccessful in getting this to work. What's annoying is that the default .json and .api "formats" work for these endpoints, but I cannot add any custom formats/renderers like .png, .jpeg, or .tif

from rest_framework.renderers import BaseRenderer as RFBaseRenderer class BaseRenderer(RFBaseRenderer): render_style = 'binary' charset = None def render(self, data, media_type=None, renderer_context=None): return data class PNGRenderer(BaseRenderer): media_type = 'image/png' format = 'png' class JPEGRenderer(BaseRenderer): media_type = 'image/jpeg' format = 'jpeg' class JPGRenderer(BaseRenderer): media_type = 'image/jpeg' format = 'jpg' image_renderers = [PNGRenderer, JPEGRenderer, JPGRenderer] class TifRenderer(BaseRenderer): media_type = 'image/jpeg' format = 'tif' class TiffRenderer(BaseRenderer): media_type = 'image/jpeg' format = 'tiff' image_data_renderers = image_renderers + [TifRenderer, TiffRenderer]
opened by banesullivan 0
Change VSI failed logging to warning

https://github.com/ResonantGeoData/django-large-image/blob/87b6175cb08aff01367ff264815c5891fe9c5411/django_large_image/rest/core.py#L86

This is showing up as an error on Sentry. I should change this to a warning and perhaps more gracefully handle it to ensure only GDAL is used with this - might come with #11

opened by banesullivan 0
Style min/max value of "" is not valid; using "auto"

https://sentry.io/organizations/django-large-image/issues/3126500235/?project=6272210&query=is%3Aunresolved

Fix handling of empty min/max values in query parameters to avoid this warning

opened by banesullivan 0
Add COG validation endpoint

Large-image's GDAL source has a validateCOG method which should be wrapped into an endpoint to validate if an image is a COG

https://girder.github.io/large_image/_build/large_image_source_gdal/large_image_source_gdal.html#large_image_source_gdal.GDALFileTileSource.validateCOG

This will not be available/valid for all tile sources, so we will need a clever way to limit it to GDAL sources and error out when not a GDAL source

opened by banesullivan 0
Set min/max help text to value range
The min/max fields in the UI do not show the min/max range for each band and it can be tough to decipher what the value range is.

It would be great if we could do two things:

Set the help text to include the actual min/max values for the selected band

When initially hitting the up/down arrows, it is incremented from the actual min/max values
opened by banesullivan 0
`frames` in `/info/metadata` but not in `/tiles/metadata`

See https://github.com/atlascope/atlascope/pull/173#issuecomment-1163453575

Need to make sure the tiles_metadata endpoint is using the metadata wrapper in the tilesource module and not grabbing that directly from large_image

opened by banesullivan 0

Releases(0.8.0)

0.8.0(Jun 17, 2022)
New Features

Notable

Major UI overhaul

Copy metadata button

Modern UI styling

Advanced multi-band compositing with custom colors

Connects large-image's caching mechanisms to utilize the Django cache framework

Specify the named cache as the LARGE_IMAGE_CACHE_NAME, otherwise uses the 'default' cache

New tiffdump endpoint utilizing tifftools to extract information from Tiff files

Support for URI encoded styles in request URL parameters (documented in README)

Improved internal mechanisms for opening tile sources for easier overrides downstream

User can specify the source class to use through the source URL parameter

Better format handing for endpoints that serve image data (single endpoint where PNG, JPEG, or TIFF are specified for the desired format)

Additional

Support max_width and max_height specification for generating thumbnails

Improvements to SSR template overriding

mypy type annotations throughout API

New TileMetadataSerializer for the tiles/metadata endpoint

Improved API error handling

Better OpenAPI RESTful arguments documentation

Demos

Standalone Docker image demo app: https://github.com/girder/django-large-image/pkgs/container/django-large-image-demo

Usage with django-raster: https://github.com/ResonantGeoData/django-raster-demo

Noted support for GeoDjango's GDALRaster in README

Document how to convert Pyramidal Tiffs (COGs) with Celery tasks

Deprecations & Changes

internal_metadata -> metadata_internal

Metadata endpoints are now under the info/ prefix

e.g., metadata -> info/metadata

Data endpoints are now under the data/ prefix

e.g., thumbnail.png -> data/thumbnail.(?P<fmt>png|jpg|jpeg)

Source code(tar.gz)
Source code(zip)
0.4.0(Apr 10, 2022)
This release has breaking changes

There are now 4 ViewSet mixin classes for use with both detail and non-detail viewsets in DRF.

The following are the provided mixin classes and their use case:

LargeImageMixin: for use with a standard, non-detail ViewSet. Users must implement get_path()

LargeImageDetailMixin: for use with a detail viewset like GenericViewSet. Users must implement get_path()

LargeImageFileDetailMixin: (most commonly used) for use with a detail viewset like GenericViewSet where the associated model has a FileField storing the image data.

LargeImageVSIFileDetailMixin: (geospatial) for use with a detail viewset like GenericViewSet where the associated model has a FileField storing the image data that is intended to be read with GDAL. This will access the data over GDAL's Virtual File System interface (a VSI path).

Most users will want to use LargeImageFileDetailMixin

Other changes:

Style parameters can now be included as a JSON blob in the request body following the format specified by large-image

The thumbnail, tile, and region endpoints now each have .png, .jpeg, and .tif variants where appropriate

DRF ValidationErrors are raised for user-caused API errors (return 400 status on endpoint)

More usage examples

Source code(tar.gz)
Source code(zip)
0.2.4(Apr 2, 2022)
django-large-image is here to solve your image tile serving needs in Django! Take a look at the README for some example implementations and give it a try in your project with image data.

Highlights

Easy to use mixin interface

Rich set of RESTful endpoints to extract information from large image formats

Serve image tiles in the slippy maps standard (z/x/y.png)

Produce thumbnails of large images on the fly

Works with FileField interfaces

Works with URL files (through GDAL's VFS/VSI layer)

Included viewer for admin interface or server-rendered templates

OpenAPI documentation included

Source code(tar.gz)
Source code(zip)

Owner

Resonant GeoData

Geospatial data cataloging with Django

GitHub

Tools to easily create permissioned CRUD endpoints in graphene-django.

graphene-django-plus Tools to easily create permissioned CRUD endpoints in graphene-django. Install pip install graphene-django-plus To make use of ev

74 Aug 9, 2022

Radically simplified static file serving for Python web apps

WhiteNoise Radically simplified static file serving for Python web apps With a couple of lines of config WhiteNoise allows your web app to serve its o

2.1k Dec 15, 2022

django Filer is a file management application for django that makes handling of files and images a breeze.

1.6k Jan 6, 2023

Utility for working with recurring dates in Django.

django-recurrence django-recurrence is a utility for working with recurring dates in Django. Documentation is available at https://django-recurrence.r

408 Jan 6, 2023

A Django app for working with BTCPayServer

btcpay-django A Django app for working with BTCPayServer Installation pip install btcpay-django Developers Release To cut a release, run bumpversion,

3 Nov 20, 2022

Django project starter on steroids: quickly create a Django app AND generate source code for data models + REST/GraphQL APIs (the generated code is auto-linted and has 100% test coverage).

Create Django App ?? We're a Django project starter on steroids! One-line command to create a Django app with all the dependencies auto-installed AND

68 Oct 19, 2022

Django URL Shortener is a Django app to to include URL Shortening feature in your Django Project

Django URL Shortener Django URL Shortener is a Django app to to include URL Shortening feature in your Django Project Install this package to your Dja

4 Nov 18, 2021

A drop-in replacement for django's ImageField that provides a flexible, intuitive and easily-extensible interface for quickly creating new images from the one assigned to the field.

django-versatileimagefield A drop-in replacement for django's ImageField that provides a flexible, intuitive and easily-extensible interface for creat

490 Dec 13, 2022

Helps working with singletons - things like global settings that you want to edit from the admin site.

726 Jan 8, 2023

A package to handle images in django

Django Image Tools Django Image Tools is a small app that will allow you to manage your project's images without worrying much about image sizes, how

42 Jun 2, 2022

A Django chatbot that is capable of doing math and searching Chinese poet online. Developed with django, channels, celery and redis.

Django Channels Websocket Chatbot A Django chatbot that is capable of doing math and searching Chinese poet online. Developed with django, channels, c

8 Oct 28, 2022

Blog focused on skills enhancement and knowledge sharing. Tech Stack's: Vue.js, Django and Django-Ninja

2 Sep 21, 2022

Meta package to combine turbo-django and stimulus-django

Hotwire + Django This repository aims to help you integrate Hotwire with Django ?? Inspiration might be taken from @hotwired/hotwire-rails. We are sti

31 Aug 9, 2022

django-quill-editor makes Quill.js easy to use on Django Forms and admin sites

django-quill-editor django-quill-editor makes Quill.js easy to use on Django Forms and admin sites No configuration required for static files! The ent

139 Dec 5, 2022

A beginner django project and also my first Django project which involves shortening of a longer URL into a short one using a unique id.

Django-URL-Shortener A beginner django project and also my first Django project which involves shortening of a longer URL into a short one using a uni

3 Aug 8, 2021

Dockerizing Django with Postgres, Gunicorn, Nginx and Certbot. A fully Django starter project.

Dockerizing Django with Postgres, Gunicorn, Nginx and Certbot ?? Features A Django stater project with fully basic requirements for a production-ready

8 Jun 27, 2022

APIs for a Chat app. Written with Django Rest framework and Django channels.

ChatAPI APIs for a Chat app. Written with Django Rest framework and Django channels. The documentation for the http end points can be found here This

18 Sep 9, 2022

Django-MySQL extends Django's built-in MySQL and MariaDB support their specific features not available on other databases.

Django-MySQL The dolphin-pony - proof that cute + cute = double cute. Django-MySQL extends Django's built-in MySQL and MariaDB support their specific

504 Jan 4, 2023

Django-Audiofield is a simple app that allows Audio files upload, management and conversion to different audio format (mp3, wav & ogg), which also makes it easy to play audio files into your Django application.

Django-Audiofield Description: Django Audio Management Tools Maintainer: Areski Contributors: list of contributors Django-Audiofield is a simple app t

167 Nov 10, 2022

Django models and endpoints for working with large images -- tile serving

Related tags

Overview

Django Large Image