Vector tile server for the Wildfire Predictive Services Unit

Overview

wps-tileserver

Vector tile server for the Wildfire Predictive Services Unit

Lifecycle:Experimental

Overview

The intention of this project is to:

  • provide tools to easily spin up a vector tile server in openshift, in a project agnostic manner.
  • provide tools to manually pull data from an esri arc server into a postgis database.
  • provide tools that periodically synchronize data from an esri arc server into postgis.

Components

  • postgis database server. (it is assumed you have a working postgis database server)
  • pg_tileserv - serves up vector tiles from postgis server.
  • proxy server (varnish?) - caches responses.
  • sync cronjob - updates database periodically.

Reference

https://blog.crunchydata.com/blog/production-postgis-vector-tiles-caching https://github.com/CrunchyData/pg_tileserv

Local development

Assumptions

  • postgresql server with postgis running locally

Configure pg_tile server

Download the latest pg_tileserver, unzip and start.

mkdir pg_tileserv
cd pg_tileserv
wget https://postgisftw.s3.amazonaws.com/pg_tileserv_latest_linux.zip
unzip pg_tileserv
export DATABASE_URL=postgresql://tileserv:tileserv@localhost/tileserv
./pg_tileserv

Install binary requirements

Install gdal

Ubuntu

sudo apt install gdal-bin

Install python requirements

This step only required if you're going to be using the python scripts in this repo to load data. If you're loading directly from shapefiles, then skip this step.

Assumptions

  • appropriate python version is install
  • python poetry is installed

Install python requirements

poetry install

Loading data

Create a user and database for your tileserver

create user tileserv with password 'tileserv';
create database tileserv with owner tileserv;
\c tileserv
CREATE EXTENSION IF NOT EXISTS postgis;

Using an arcserver rest endpoint

Given some arcserver layer endpoint, e.g.: https://maps.gov.bc.ca/arcserver/rest/services/whse/bcgw_pub_whse_legal_admin_boundaries/MapServer/8

poetry run python fetch_feature_layer.py https://maps.gov.bc.ca/arcserver/rest/services/whse/bcgw_pub_whse_legal_admin_boundaries/MapServer/8

Using a shapefile

ogr2ogr -f "PostgreSQL" PG:"dbname=tileserv host=localhost user=tileserv password=tileserv" "my_shapefile.shp" -lco precision=NO -nln fire_area_thessian_polygons

Deploy

Assumptions

  • You have the oc command line installed and you're logged in.
  • You have docker installed locally.
  • You have a postgres database in your target openshift environment that can be accessed by pg_tileserv (you made need to add additional rules to allow your tile server to communicate with your database.)

Instructions

Prepare your openshift environment

# we have docker limits, so pull the images local - then put them in openshift

# pull local
docker pull eeacms/varnish
docker pull pramsey/pg_tileserv

# tag for upload
docker tag eeacms/varnish image-registry.apps.silver.devops.gov.bc.ca/e1e498-tools/varnish:latest
docker tag pramsey/pg_tileserv image-registry.apps.silver.devops.gov.bc.ca/e1e498-tools/pg_tileserv:latest

# log in to openshift docker
docker login -u developer -p $(oc whoami -t) image-registry.apps.silver.devops.gov.bc.ca

# push it
docker push image-registry.apps.silver.devops.gov.bc.ca/e1e498-tools/varnish:latest
docker push image-registry.apps.silver.devops.gov.bc.ca/e1e498-tools/pg_tileserv:latest

Deploy pg_tilserver

# deploy pg_tileserv
oc -n e1e498-dev process -f tileserv.yaml | oc -n e1e498-dev apply -f -

Manually Loading data into your openshift hosted postgis database

The easiest way to achieve this, is to tunnel to your database server and then run the import scripts as if your database was local.

oc port-forward patroni-wps-mapserver-prototype-1 5432:5432
Comments
  • Geometry labels

    Geometry labels

    Create a separate table for each layer, that contains only the centroid of the source layer. This allows for a separate layer that only contains labels, not entire geometries and thus avoiding duplication of labels that may occur when serving up geometries that span multiple tiles.

    opened by Sybrand 1
  • Custom feature import

    Custom feature import

    Replace ogr2ogr in order to account for:

    • bc gov column standards (create_date and update_date)
    • specify geom type (ogr2ogr has param -nlt which does the same though)
    • updating features (if id matches, will to an update instead of an insert - again ogr2ogr probably supports this with -update)
    • remove gdal-bin as a binary requirement (does however then introduce psycopg2, GeoAlchemy2 and Shapely as python requirements)
    opened by Sybrand 1
  • It's Been a While Since This Repository has Been Updated

    It's Been a While Since This Repository has Been Updated

    This issue is a kind reminder that your repository has been inactive for 181 days. Some repositories are maintained in accordance with business requirements that infrequently change thus appearing inactive, and some repositories are inactive because they are unmaintained.

    To help differentiate products that are unmaintained from products that do not require frequent maintenance, repomountie will open an issue whenever a repository has not been updated in 180 days.

    • If this product is being actively maintained, please close this issue.
    • If this repository isn't being actively maintained anymore, please archive this repository. Also, for bonus points, please add a dormant or retired life cycle badge.

    Thank you for your help ensuring effective governance of our open-source ecosystem!

    opened by repo-mountie[bot] 0
  • Add missing compliance audit file

    Add missing compliance audit file

    TL;DR 🏎️

    Your repo is missing a compliance audit file so I've created this PR with a template that you can update with the correct PIA and STRA status (status options in the table below). If you'd like me to do this for you, skip to the commands section below.

    Compliance

    Projects in our organization (bcgov) need to complete a Privacy Impact Assessment (PIA) and Security Threat & Risk Assessment (STRA) before they go live in production. Since every ministry has their own way of doing both the STRA and PIA we don't enforce that projects do them, only that they report on the current status.

    To help with reporting, I've added a compliance audit file as part of this pull request. Please checkout this branch and edit update status as needed. Here is a table of possible states:

    | Status | Description | | ----------- | :----------------------------------------------------------------------------------------------------- | | TBD | If you're surprised by this news, use this state. I'll let you talk to your MISO and check back later. | | in-progress | Use this state when your assessment(s) are underway. | | completed | Use this state when your assessment(s) are completed. 🙌 🎉 | | not-required | You have consulted with your MISO or Privacy Officer and they agree that no PIA or STRA is required.|

    Here is what a completed audit file might look like:

    name: compliance
    description: |
      This document is used to track a projects PIA and STRA
      compliance.
    spec:
      - name: PIA
        status: in-progress
        last-updated: '2019-11-22T00:03:52.138Z'
      - name: STRA
        status: completed
        last-updated: '2019-11-22T00:03:52.138Z'
    

    For more information check out the BC Policy Framework for GitHub.

    Pro Tip 🤓

    • If you're not sure what to do add a comment below with the command @repo-mountie help in it; a real-live-person will reply back to help you out.
    • Leverage this PR to document the history of your PIA and STRA thus far by adding a comment or two.

    Commands 🤖

    I can update the status of the PIA and STRA for you; you'll just need to merge the PR when I'm done. You can find the available status values in the table above. Below are some commands I understand:

    | Command | Description | | :------------------------------- | :------------------------------------------------ | | @repo-mountie help | You're freaking out and want to talk to a person. | | @repo-mountie update-pia STATUS | You want me to update the PIA status. | | @repo-mountie update-stra STATUS | You want me to update the STRA status. |

    Examples

    @repo-mountie update-pia completed
    @repo-mountie update-stra in-progress
    
    opened by repo-mountie[bot] 0
  • Add project lifecycle badge

    Add project lifecycle badge

    No Project Lifecycle Badge found in your readme!

    Hello! I scanned your readme and could not find a project lifecycle badge. A project lifecycle badge will provide contributors to your project as well as other stakeholders (platform services, executive) insight into the lifecycle of your repository.

    What is a Project Lifecycle Badge?

    It is a simple image that neatly describes your project's stage in its lifecycle. More information can be found in the project lifecycle badges documentation.

    What do I need to do?

    I suggest you make a PR into your README.md and add a project lifecycle badge near the top where it is easy for your users to pick it up :). Once it is merged feel free to close this issue. I will not open up a new one :)

    opened by repo-mountie[bot] 0
  • Add missing topics

    Add missing topics

    TL;DR

    Topics greatly improve the discoverability of repos; please add the short code from the table below to the topics of your repo so that ministries can use GitHub's search to find out what repos belong to them and other visitors can find useful content (and reuse it!).

    Why Topic

    In short order we'll add our 800th repo. This large number clearly demonstrates the success of using GitHub and our Open Source initiative. This huge success means it's critical that we work to make our content as discoverable as possible. Through discoverability, we promote code reuse across a large decentralized organization like the Government of British Columbia as well as allow ministries to find the repos they own.

    What to do

    Below is a table of abbreviation a.k.a short codes for each ministry; they're the ones used in all @gov.bc.ca email addresses. Please add the short codes of the ministry or organization that "owns" this repo as a topic.

    add a topic

    That's it, you're done!!!

    How to use

    Once topics are added, you can use them in GitHub's search. For example, enter something like org:bcgov topic:citz to find all the repos that belong to Citizens' Services. You can refine this search by adding key words specific to a subject you're interested in. To learn more about searching through repos check out GitHub's doc on searching.

    Pro Tip 🤓

    • If your org is not in the list below, or the table contains errors, please create an issue here.

    • While you're doing this, add additional topics that would help someone searching for "something". These can be the language used javascript or R; something like opendata or data for data only repos; or any other key words that are useful.

    • Add a meaningful description to your repo. This is hugely valuable to people looking through our repositories.

    • If your application is live, add the production URL.

    Ministry Short Codes

    | Short Code | Organization Name | | :--------- | :------------ | | AEST | Advanced Education, Skills & Training | | AGRI | Agriculture | | ALC | Agriculture Land Commission | | AG | Attorney General | | MCF | Children & Family Development | | CITZ | Citizens' Services | | DBC | Destination BC | | EMBC | Emergency Management BC | | EAO | Environmental Assessment Office | | EDUC | Education | | EMPR | Energy, Mines & Petroleum Resources | | ENV | Environment & Climate Change Strategy | | FIN | Finance | | FLNR | Forests, Lands, Natural Resource Operations & Rural Development | | HLTH | Health | | IRR | Indigenous Relations & Reconciliation | | JEDC | Jobs, Economic Development & Competitiveness | | LBR | Labour Policy & Legislation | | LDB | BC Liquor Distribution Branch | | MMHA | Mental Health & Addictions | | MAH | Municipal Affairs & Housing | | BCPC | Pension Corporation | | PSA | Public Service Agency | | PSSG | Public Safety and Solicitor General | | SDPR | Social Development & Poverty Reduction | | TCA | Tourism, Arts & Culture | | TRAN | Transportation & Infrastructure |

    NOTE See an error or omission? Please create an issue here to get it remedied.

    opened by repo-mountie[bot] 0
  • Lets use common phrasing

    Lets use common phrasing

    TL;DR 🏎️

    Teams are encouraged to favour modern inclusive phrasing both in their communication as well as in any source checked into their repositories. You'll find a table at the end of this text with preferred phrasing to socialize with your team.

    Words Matter

    We're aligning our development community to favour inclusive phrasing for common technical expressions. There is a table below that outlines the phrases that are being retired along with the preferred alternatives.

    During your team scrum, technical meetings, documentation, the code you write, etc. use the inclusive phrasing from the table below. That's it - it really is that easy.

    For the curious mind, the Public Service Agency (PSA) has published a guide describing how Words Matter in our daily communication. Its an insightful read and a good reminder to be curious and open minded.

    What about the master branch?

    The word "master" is not inherently bad or non-inclusive. For example people get a masters degree; become a master of their craft; or master a skill. It's generally when the word "master" is used along side the word "slave" that it becomes non-inclusive.

    Some teams choose to use the word main for the default branch of a repo as opposed to the more commonly used master branch. While it's not required or recommended, your team is empowered to do what works for them. If you do rename the master branch consider using main so that we have consistency among the repos within our organization.

    Preferred Phrasing

    | Non-Inclusive | | Inclusive | | :------------- |:--:| :-------- | | Whitelist | => | Allowlist | | Blacklist | => | Denylist | | Master / Slave | => | Leader / Follower; Primary / Standby; etc | | Grandfathered | => | Legacy status | | Sanity check | => | Quick check; Confidence check; etc | | Dummy value | => | Placeholder value; Sample value; etc |

    Pro Tip 🤓

    This list is not comprehensive. If you're aware of other outdated nomenclature please create an issue (PR preferred) with your suggestion.

    opened by repo-mountie[bot] 0
Owner
Province of British Columbia
Where Ideas Work
Province of British Columbia
GA SEI Unit 4 project backend for Bloom.

Grow Your OpportunitiesTM Background Watch the Bloom Intro Video At Bloom, we believe every job seeker deserves an opportunity to find meaningful work

Jonathan Herman 3 Sep 20, 2021
Placeholders is a single-unit storage solution for your Frontend.

Placeholder Placeholders is a single-unit file storage solution for your Frontend. Why Placeholder? Generally, when a website/service requests for fil

Tanmoy Sen Gupta 1 Nov 9, 2021
A docker container (Docker Desktop) for a simple python Web app few unit tested

Short web app using Flask, tested with unittest on making massive requests, responses of the website, containerized

Omar 1 Dec 13, 2021
Passenger Car Unit (PCU) Calculator

This is a streamlit web application which can be used to calculate Passenger Car Unit (PCU) values for a selected road section.

Dineth Dhananjaya 1 Apr 26, 2022
Two predictive attributes (Speed and Angle) and one attribute target (Power)

Two predictive attributes (Speed and Angle) and one attribute target (Power). A container crane has the function of transporting containers from one point to another point. The difficulty of this task lies in the fact that the container is connected to the bridge crane by cables causing an opening angle while the container is being transported, interfering with the operation at high speeds due to oscillation that occurs at the end point, which could cause accidents.

Astitva Veer Garg 1 Jan 11, 2022
Werkzeug has a debug console that requires a pin. It's possible to bypass this with an LFI vulnerability or use it as a local privilege escalation vector.

Werkzeug Debug Console Pin Bypass Werkzeug has a debug console that requires a pin by default. It's possible to bypass this with an LFI vulnerability

Wyatt Dahlenburg 23 Dec 17, 2022
Transform a Google Drive server into a VFX pipeline ready server

Google Drive VFX Server VFX Pipeline About The Project Quick tutorial to setup a Google Drive Server for multiple machines access, and VFX Pipeline on

Valentin Beaumont 17 Jun 27, 2022
Certipy is a Python tool to enumerate and abuse misconfigurations in Active Directory Certificate Services (AD CS).

Certipy Certipy is a Python tool to enumerate and abuse misconfigurations in Active Directory Certificate Services (AD CS). Based on the C# variant Ce

ollypwn 1.3k Jan 1, 2023
Python client SDK designed to simplify integrations by automating key generation and certificate enrollment using Venafi machine identity services.

This open source project is community-supported. To report a problem or share an idea, use Issues; and if you have a suggestion for fixing the issue,

Venafi, Inc. 13 Sep 27, 2022
Programmatic interface to Synapse services for Python

A Python client for Sage Bionetworks' Synapse, a collaborative, open-source research platform that allows teams to share data, track analyses, and collaborate

Sage Bionetworks 54 Dec 23, 2022
Run-Your-Own Firefox Sync Server

Run-Your-Own Firefox Sync Server This is an all-in-one package for running a self-hosted Firefox Sync server. It bundles the "tokenserver" project for

Mozilla Services 1.7k Dec 30, 2022
YunoHost is an operating system aiming to simplify as much as possible the administration of a server.

YunoHost is an operating system aiming to simplify as much as possible the administration of a server. This repository corresponds to the core code, written mostly in Python and Bash.

YunoHost 1.5k Jan 9, 2023
A simple service that allows you to run commands on the server using text

Server Text A simple flask service that allows you to run commands on the server/computer over sms. Think of it as a shell where you run commands over

MT Devs 49 Nov 9, 2021
An open source server for Super Mario Bros. 35

SMB35 A custom server for Super Mario Bros. 35 This server is highly experimental. Do not expect it to work without flaws.

Yannik Marchand 162 Dec 7, 2022
Remote execution of a simple function on the server

FunFetch Remote execution of a simple function on the server All types of Python support objects.

Decave 4 Jun 30, 2022
No more support server flooding with questions about unsupported hosting.

No more support server flooding with questions about unsupported hosting.

null 3 Aug 9, 2021
tagls is a language server based on gtags.

tagls tagls is a language server based on gtags. Why I wrote it? Almost all modern editors have great support to LSP, but language servers based on se

daquexian 31 Dec 1, 2022
Experimental Brawl Stars v36.218 server emulator written in Python.

Brawl Stars v36 Experimental Brawl Stars v36.218 server emulator written in Python. Requirements: Python 3.7 or higher colorama Running the server In

null 8 Oct 31, 2021
Ultimate Score Server for RealistikOsu

USSR Ultimate Score Server for RealistikOsu (well not just us but it makes the acronym work.) Also I wonder how long this name will last. What is this

RealistikOsu! 15 Dec 14, 2022