PYLE38
iwpnd.pw · Report Bug · Request Feature
Table of Contents
About The Project
This is an asynchonous Python client for Tile38 that allows for fast and easy interaction with the worlds fastest in-memory geodatabase Tile38.
Example
import asyncio
from pyle38 import Tile38
async def main():
tile38 = Tile38(url="redis://localhost:9851", follower_url="redis://localhost:9851")
await tile38.set("fleet", "truck").point(52.25,13.37).exec()
response = await tile38.follower()
.within("fleet")
.circle(52.25, 13.37, 1000)
.asObjects()
assert response.ok
print(response.dict())
asyncio.run(main())
> {
"ok": True,
"elapsed": "48.8µs",
"objects": [
{
"object": {
"type": "Point",
"coordinates": [
13.37,
52.25
]
},
"id": "truck"
}
],
"count": 1,
"cursor": 0
}
Example IPython
In [1]: %autoawait asyncio
In [2]: from pyle38 import Tile38
In [3]: tile38 = Tile38(url='redis://localhost:9851', follower_url='redis://localhost:9852')
In [4]: await tile38.set("fleet", "truck").point(52.25,13.37).exec()
Out[4]: JSONResponse(ok=True, elapsed='51.9µs', err=None)
In [5]: response = await tile38.within("fleet")
...: .circle(52.25, 13.37, 1000)
...: .asObjects()
In [6]: print(response.dict())
{
"ok": True,
"elapsed": "46.3µs",
"objects": [
{
"object": {
"type": "Point",
"coordinates": [
13.37,
52.25
]
},
"id": "truck"
}
],
"count": 1,
"cursor": 0
}
Features
- fully typed using mypy and pydantic
- lazy client
- optional build-in leader/follower logic
- easy to use and integrate
- next to no external dependencies
Built With
Getting Started
Requirements
Python==^3.8.2
Installation
pip install pyle38
Or using Poetry
poetry add pyle38
Now start your Tile38 instance(s) either locally using Docker and docker-compose.
docker-compose up
Or follow the installation instruction on tile38.com to start your install Tile38 and start a server locally.
If you already have a Tile38 instance running somewhere read on.
Import
from pyle38 import Tile38
tile38 = Tile38('leader:9851')
Leader / Follower
When it comes to replication, Tile38 follows a leader-follower model. The leader receives all commands that SET data, a follower on the other hand is read-only and can only query data. For more on replication in Tile38 refer to the official documentation.
This client is not meant to setup a replication, because this should happen in your infrastructure. But it helps you to specifically execute commands on leader or follower. This way you can make sure that the leader always has enough resources to execute SETs and fire geofence events on webhooks.
For now you can set one follower url to bet set alongside the leader url.
from pyle38.tile38 import Tile38
tile38 = Tile38('leader:9851', 'follower:9851')
Once the client is instantiated with a follower, commands can be explicitly send to the follower, but adding .follower() to your command chaining.
await tile38.follower().get('fleet', 'truck1').asObjects()
Pagination
Tile38 has hidden limits set for the amount of objects that can be returned in one request. For most queries/searches this limit is set to 100. This client gives you the option to either paginate the results yourself by add .cursor() and .limit() to your queries, or it abstracts pagination away from the user by adding .all().
Let's say your fleet in Berlin extends 100 vehicles, then
await tile38.within('fleet').get('cities', 'Berlin').asObjects()
will only return 100 vehicle objects. Now you can either get the rest by setting the limit to the amount of vehicles you have in the city and get them all.
await tile38.within('fleet').limit(10000).get('cities', 'Berlin').asObjects()
Or, you can paginate the results in multiple concurrent requests to fit your requirements.
await tile38.within('fleet')
.cursor(0)
.limit(100)
.get('cities', 'Berlin')
.asObjects()
await tile38.within('fleet')
.cursor(100)
.limit(100)
.get('cities', 'Berlin')
.asObjects()
await tile38.within('fleet')
.cursor(200)
.limit(100)
.get('cities', 'Berlin')
.asObjects();
Responses
For now, every Tile38 commands response is parsed into a pydantic object for validation and type safety.
response = await tile38.set('fleet','truck1')
.point(52.25,13.37)
.exec()
print(response.dict())
> {'ok': True, 'elapsed': '40.7µs'}
response = await tile38.get('fleet', 'truck1')
print(response.ok)
> True
print(response.object)
> {
'type': 'Point',
'coordinates': [13.37, 52.25]
}
print(response.dict())
> {
'ok': True,
'elapsed': '29.3µs',
'object': {
'type': 'Point',
'coordinates': [13.37, 52.25]
}
}
Commands
Keys
SET
Set the value of an id. If a value is already associated to that key/id, it'll be overwritten.
await tile38.set('fleet', 'truck1')
.fields({ "maxSpeed": 90, "milage": 90000 })
.point(33.5123, -112.2693)
.exec()
await tile38.set('fleet', 'truck1:driver').string('John Denton').exec()
Options
.fields() |
Optional additional fields. MUST BE numerical |
.ex(value) |
Set the specified expire time, in seconds. |
.nx() |
Only set the object if it does not already exist. |
.xx() |
Only set the object if it already exist. |
Input
.point(lat, lon) |
Set a simple point in latitude, longitude |
.bounds(minlat, minlon, maxlat, maxlon) |
Set as minimum bounding rectangle |
.object(feature) |
Set as feature |
.hash(geohash) |
Set as geohash |
.string(value) |
Set as string. To retrieve string values you can use .get(), scan() or .search() |
FSET
Set the value for one or more fields of an object. Fields must be double precision floating points. Returns the integer count of how many fields changed their values.
await tile38.fset('fleet', 'truck1', { "maxSpeed": 90, "milage": 90000 })
Options
.xx() |
FSET returns error if fields are set on non-existing ids. xx() options changes this behaviour and instead returns 0 if id does not exist. If key does not exist FSET still returns error |
GET
Get the object of an id.
await tile38.get('fleet', 'truck1')
Options
.withfields() |
will also return the fields that belong to the object. Field values that are equal to zero are omitted. |
Output
.asObject() |
(default) get as object |
.asBounds() |
get as minimum bounding rectangle |
.asHash(precision) |
get as hash |
.asPoint() |
get as point |
DEL
Remove a specific object by key and id.
await tile38.del('fleet', 'truck1')
PDEL
Remove objects that match a given pattern.
await tile38.pDel('fleet', 'truck*')
DROP
Remove all objects in a given key.
await tile38.drop('fleet')
BOUNDS
Get the minimum bounding rectangle for all objects in a given key
await tile38.bounds('fleet')
EXPIRE
Set an expiration/time to live in seconds of an object.
await tile38.expire('fleet', 'truck1', 10)
TTL
Get the expiration/time to live in seconds of an object.
await tile38.ttl('fleet', 'truck1', 10)
PERSIST
Remove an existing expiration/time to live of an object.
await tile38.persist('fleet', 'truck1')
KEYS
Get all keys matching a glob-style-pattern. Pattern defaults to '*'
await tile38.keys()
STATS
Return stats for one or more keys. The returned stats array contains one or more entries, depending on the number of keys in the request.
await tile38.stats('fleet1', 'fleet2')
Returns
in_memory_size |
estimated memory size in bytes |
num_objects |
objects in the given key |
num_points |
number of geographical objects in the given key |
JSET/JSET/JDEL
Set a value in a JSON document. JGET, JSET, and JDEL allow for working with JSON strings
await tile38.jset('user', 901, 'name', 'Tom')
await tile38.jget('user', 901)
> {'name': 'Tom'}
await tile38.jset('user', 901, 'name.first', 'Tom')
await tile38.jset('user', 901, 'name.first', 'Anderson')
await tile38.jget('user', 901)
> {'name': { 'first': 'Tom', 'last': 'Anderson' }}
await tile38.jdel('user', 901, 'name.last')
await tile38.jget('user', 901);
> {'name': { 'first': 'Tom' }}
RENAME
Renames a collection key to newkey.
Options
.nx() |
Default: false. Changes behavior on how renaming acts if newkey already exists |
If the newkey already exists it will be deleted prior to renaming.
await tile38.rename('fleet', 'newfleet', false)
If the newkey already exists it will do nothing.
await tile38.rename('fleet', 'newfleet', true)
Search
Searches are Tile38 bread and butter. They are what make Tile38 a ultra-fast, serious and cheap alternative to PostGIS for a lot of use-cases.
WITHIN
WITHIN searches a key for objects that are fully contained within a given bounding area.
await tile38.within('fleet').bounds(33.462, -112.268, 33.491, -112.245)
> {
"ok":true,
"objects":[
{
"id":"1",
"object":{
"type":"Feature",
"geometry":{
"type":"Point",
"coordinates":[
-112.2693,
33.5123
]
},
"properties":{
}
}
}
],
"count":1,
"cursor":1,
"elapsed":"72.527µs"
}
await tile38.within('fleet').nofields().asCount()
> {
"ok":true,
"count":205,
"cursor":0,
"elapsed":"2.078168µs"
}
await tile38.within('fleet').get('warehouses', 'Berlin').asCount();
> {
"ok":true,
"count":50,
"cursor":0,
"elapsed":"2.078168µs"
}
Options
.cursor(value) |
used to iterate through your search results. Defaults to 0 if not set explicitly |
.limit(value) |
limit the number of returned objects. Defaults to 100 if not set explicitly |
.nofields() |
if not set and one of the objects in the key has fields attached, fields will be returned. Use this to suppress this behavior and don't return fields. |
.match(pattern) |
Match can be used to filtered objects considered in the search with a glob pattern. .match('truck*') e.g. will only consider ids that start with truck within your key. |
.sparse(value) |
caution seems bugged since Tile38 1.22.6. Accepts values between 1 and 8. Can be used to distribute the results of a search evenly across the requested area. |
Outputs
.asObjects() |
return as array of objects |
.asBounds() |
return as array of minimum bounding rectangles: {"id": str,"bounds":{"sw":{"lat": float,"lon": float},"ne":{"lat": float,"lon": float}}} |
.asCount() |
returns a count of the objects in the search |
.asHashes(precision) |
returns an array of {"id": str,"hash": str} |
.asIds() |
returns an array of ids |
.asPoints() |
returns objects as points: {"id": str,"point":{"lat": float,"lon": float}. If the searched key is a collection of Polygon objects, the returned points are the centroids. |
Query
.get(key, id) |
Search a given stored item in a collection. |
.circle(lat, lon, radius) |
Search a given circle of latitude, longitude and radius. |
.bounds(minlat, minlon, maxlat, maxlon) |
Search a given bounding box. |
.hash(value) |
Search a given geohash. |
.quadkey(value) |
Search a given quadkey |
.tile(x, y, z) |
Search a given tile |
.object(value) |
Search a given GeoJSON polygon feature. |
INTERSECTS
Intersects searches a key for objects that are fully contained within a given bounding area, but also returns those that intersect the requested area. When used to search a collection of keys consisting of Point objects (e.g. vehicle movement data) it works like a .within() search as Points cannot intersect. When used to search a collection of keys consisting of Polygon or LineString it also returns objects, that only partly overlap the requested area.
await tile38.intersects('warehouses').hash('u33d').asObjects()
await tile38.intersects('fleet').get('warehouses', 'Berlin').asIds()
> {
"ok":true,
"ids":[
"truck1"
],
"count":1,
"cursor":0,
"elapsed":"2.078168ms"
}
Options
.clip() |
Tells Tile38 to clip returned objects at the bounding box of the requested area. Works with .bounds(), .hash(), .tile() and .quadkey() |
.cursor(value) |
used to iterate through your search results. Defaults to 0 if not set explicitly |
.limit(value) |
limit the number of returned objects. Defaults to 100 if not set explicitly |
.nofields() |
if not set and one of the objects in the key has fields attached, fields will be returned. Use this to suppress this behavior and don't return fields. |
.match(pattern) |
Match can be used to filtered objects considered in the search with a glob pattern. .match('warehouse*') e.g. will only consider ids that start with warehouse within your key. |
.sparse(value) |
caution seems bugged since Tile38 1.22.6. Accepts values between 1 and 8. Can be used to distribute the results of a search evenly across the requested area. |
Outputs
.asObjects() |
return as array of objects |
.asBounds() |
return as array of minimum bounding rectangles: {"id": str,"bounds":{"sw":{"lat": float,"lon": float},"ne":{"lat": float,"lon": float}}} |
.asCount() |
returns a count of the objects in the search |
.asHashes(precision) |
returns an array of {"id": str,"hash": str} |
.asIds() |
returns an array of ids |
.asPoints() |
returns objects as points: {"id": str,"point":{"lat": float,"lon": float}. If the searched key is a collection of Polygon objects, the returned points are the centroids. |
Query
.get(key, id) |
Search a given stored item in a collection. |
.circle(lat, lon, radius) |
Search a given circle of latitude, longitude and radius. |
.bounds(minlat, minlon, maxlat, maxlon) |
Search a given bounding box. |
.hash(value) |
Search a given geohash. |
.quadkey(value) |
Search a given quadkey |
.tile(x, y, z) |
Search a given tile |
.object(value) |
Search a given GeoJSON polygon feature. |
Nearby
await tile38.set('fleet', 'truck1')
.point(33.5123, -112.2693)
.exec()
await tile38.nearby('fleet').point(33.5124, -112.2694)
> {
"ok":true,
"count":1,
"cursor":0,
"elapsed":"42.8µs"
}
await tile38.nearby('fleet').point(33.5124, -112.2694, 10)
// because truck1 is further away than 10m
> {
"ok":true,
"count":0,
"cursor":0,
"elapsed":"36µs"
}
Options
.distance() |
Returns the distance in meters to the object from the query .point() |
.cursor(value) |
used to iterate through your search results. Defaults to 0 if not set explicitly |
.limit(value) |
limit the number of returned objects. Defaults to 100 if not set explicitly |
.nofields() |
if not set and one of the objects in the key has fields attached, fields will be returned. Use this to suppress this behavior and don't return fields. |
.match(pattern) |
Match can be used to filtered objects considered in the search with a glob pattern. .match('warehouse*') e.g. will only consider ids that start with warehouse within your key. |
.sparse(value) |
caution seems bugged since Tile38 1.22.6. Accepts values between 1 and 8. Can be used to distribute the results of a search evenly across the requested area. |
Outputs
.asObjects() |
return as array of objects |
.asBounds() |
return as array of minimum bounding rectangles: {"id": str,"bounds":{"sw":{"lat": float,"lon": float},"ne":{"lat": float,"lon": float}}} |
.asCount() |
returns a count of the objects in the search |
.asHashes(precision) |
returns an array of {"id": str,"hash": str} |
.asIds() |
returns an array of ids |
.asPoints() |
returns objects as points: {"id": str,"point":{"lat": float,"lon": float}. If the searched key is a collection of Polygon objects, the returned points are the centroids. |
Query
.point(lat, lon, radius: Optional[int]) |
Search nearby a given of latitude, longitude. If radius is set, searches nearby the given radius. |
Scan
Incrementally iterate through a given collection key.
await tile38.scan('fleet')
Options
.asc() |
Values are returned in ascending order. Default if not set. |
.desc() |
Values are returned in descending order. |
.cursor(value) |
used to iterate through your search results. Defaults to 0 if not set explicitly |
.limit(value) |
limit the number of returned objects. Defaults to 100 if not set explicitly |
.nofields() |
if not set and one of the objects in the key has fields attached, fields will be returned. Use this to suppress this behavior and don't return fields. |
.match(pattern) |
Match can be used to filtered objects considered in the search with a glob pattern. .match('warehouse*') e.g. will only consider ids that start with warehouse within your key. |
Outputs
.asObjects() |
return as array of objects |
.asBounds() |
return as array of minimum bounding rectangles: {"id": str,"bounds":{"sw":{"lat": float,"lon": float},"ne":{"lat": float,"lon": float}}} |
.asCount() |
returns a count of the objects in the search |
.asHashes(precision) |
returns an array of {"id": str,"hash": str} |
.asIds() |
returns an array of ids |
.asPoints() |
returns objects as points: {"id": str,"point":{"lat": float,"lon": float}. If the searched key is a collection of Polygon objects, the returned points are the centroids. |
Search
Used to iterate through a keys string values.
await tile38.set('fleet', 'truck1:driver').string('John').exec()
await tile38.search('fleet').match('J*').asStringObjects()
> {
"ok":true,
"objects":[
{
"id":"truck1:driver",
"object":"John"
}
],
"count":1,
"cursor":0,
"elapsed":"59.9µs"
}
Options
.asc() |
Values are returned in ascending order. Default if not set. |
.desc() |
Values are returned in descending order. |
.cursor(value) |
used to iterate through your search results. Defaults to 0 if not set explicitly |
.limit(value) |
limit the number of returned objects. Defaults to 100 if not set explicitly |
.nofields() |
if not set and one of the objects in the key has fields attached, fields will be returned. Use this to suppress this behavior and don't return fields. |
.match(pattern) |
Match can be used to filtered objects considered in the search with a glob pattern. .match('J*') e.g. will only consider string values objects that have a string value starting with J |
Outputs
.asStringObjects() |
return as array of objects |
.asCount() |
returns a count of the objects in the search |
.asIds() |
returns an array of ids |
Server / Connection
CONFIG GET / REWRITE / SET
While .config_get() fetches the requested configuration, .config_set() can be used to change the configuration.
Important, changes made with .set() will only take affect after .config_rewrite() is used.
Options
requirepass |
Set a password and make server password-protected, if not set defaults to "" (no password required). |
leaderauth |
If leader is password-protected, followers have to authenticate before they are allowed to follow. Set leaderauth to password of the leader, prior to .follow(). |
protected-mode |
Tile38 only allows authenticated clients or connections from localhost. Defaults to: "yes" |
maxmemory |
Set max memory in bytes. Get max memory in bytes/kb/mb/gb. |
autogc |
Set auto garbage collection to time in seconds where server performs a garbage collection. Defaults to: 0 (no garbage collection) |
keep_alive |
Time server keeps client connections alive. Defaults to: 300 (seconds) |
await tile38.config_get('keepalive')
> {
"ok":true,
"properties":{
"keepalive":"300"
},
"elapsed":"54.6µs"
}
await tile38.config_set('keepalive', 400)
> {"ok":true,"elapsed":"36.9µs"}
await tile38.config_rewrite()
> {"ok":true,"elapsed":"363µs"}
await tile38.config_get('keepalive')
> {
"ok":true,
"properties":{
"keepalive":"400"
},
"elapsed":"33.8µs"
}
Advanced options Advanced configuration can not be set with commands, but has to be set in a config file in your data directory. Options above, as well as advanced options can be set and are loaded on start-up.
follow_host |
URI of Leader to follow |
follow_port |
PORT of Leader to follow |
follow_pos |
? |
follow_id |
ID of Leader |
server_id |
Server ID of the current instance |
read_only |
Make Tile38 read-only |
FLUSHDB
Delete all keys and hooks.
await tile38.flushDb()
GC
Instructs the server to perform a garbage collection.
await tile38.gc()
READONLY
Sets Tile38 into read-only mode. Commands such as.set() and .del() will fail.
await tile38.readonly(True)
SERVER
Get Tile38 statistics.
await tile38.server()
Or get extended statistics:
await tile38.server_extended()
Geofencing
A geofence is a virtual boundary that can detect when an object enters or exits the area. This boundary can be a radius or any search area format, such as a bounding box, GeoJSON object, etc. Tile38 can turn any standard search into a geofence monitor by adding the FENCE keyword to the search.
Geofence events can be:
inside(object in specified area),outside(object outside specified area),enter(object enters specified area),exit(object exits specified area),crosses(object that was not in specified area, has enter/exit it).
Geofence events can be send on upon commands:
setwhich sends an event when an object is.set()delwhich sends a last event when the object that resides in the geosearch is deleted via.del()dropwhich sends a message when the entire collection is dropped
SETHOOK
Creates a webhook that points to a geosearch (NEARBY/WITHIN/INTERSECTS). Whenever a commands creates/deletes/drops an object that fulfills the geosearch, an event is send to the specified endpoint.
# sends event to endpoint, when object in 'fleet'
# enters the area of a 500m radius around
# latitude 33.5123 and longitude -112.2693
await tile38.sethook('warehouse', 'http://10.0.20.78/endpoint')
.nearby('fleet')
.point(33.5123, -112.2693, 500)
.activate()
await tile38.set('fleet', 'bus').point(33.5123001, -112.2693001).exec()
# results in event =
> {
"command": "set",
"group": "5c5203ccf5ec4e4f349fd038",
"detect": "inside",
"hook": "warehouse",
"key": "fleet",
"time": "2021-03-22T13:06:36.769273-07:00",
"id": "bus",
"meta": {},
"object": { "type": "Point", "coordinates": [-112.2693001, 33.5123001] }
}
Geosearch
.nearby(name, endpoint) |
|
.within(name, endpoint) |
|
.intersects(name, endpoint) |
Options
.meta(meta) |
Optional add additional meta information that are send in the geofence event. |
.ex(value) |
Optional TTL in seconds |
.commands(which[]) |
Select on which command a hook should send an event. Defaults to: ['set', 'del', 'drop'] |
.detect(what[]) |
Select what events should be detected. Defaults to: ['enter', 'exit', 'crosses', 'inside', 'outside'] |
Endpoints
| HTTP/HTTPS | http:// https:// send messages over HTTP/S. For options see link. |
| gRPC | grpc:// send messages over gRPC. For options see link. |
| Redis | redis:// send messages to Redis. For options see link |
| Disque | disque:// send messages to Disque. For options see link. |
| Kafka | kafka:// send messages to a Kafka topic. For options see link. |
| AMQP | amqp:// send messages to RabbitMQ. For options see link. |
| MQTT | mqtt:// send messages to an MQTT broker. For options see link. |
| SQS | sqs:// send messages to an Amazon SQS queue. For options see link. |
| NATS | nats:// send messages to a NATS topic. For options see link. |
SETCHAN / SUBSCRIBE / PSUBSCRIBE
Similar to sethook(), but opens a PUB/SUB channel.
# Start a channel that sends event, when object in 'fleet'
# enters the area of a 500m radius around
# latitude 33.5123 and longitude -112.2693
await tile38.setchan('warehouse', 'http://10.0.20.78/endpoint')
.nearby('fleet')
.point(33.5123, -112.2693, 500)
.activate()
Given a proper setup of a pubsub channel, every set .set() results in:
await tile38.set('fleet', 'bus')
.point(33.5123001, -112.2693001)
.exec();
# event =
> {
"command": "set",
"group": "5c5203ccf5ec4e4f349fd038",
"detect": "inside",
"hook": "warehouse",
"key": "fleet",
"time": "2021-03-22T13:06:36.769273-07:00",
"id": "bus",
"meta": {},
"object": { "type": "Point", "coordinates": [-112.2693001, 33.5123001] }
}
Geosearch
.nearby(name, endpoint) |
|
.within(name, endpoint) |
|
.intersects(name, endpoint) |
Options
.meta(meta) |
Optional addition meta information that a send in the geofence event. |
.ex(value) |
Optional TTL in seconds |
.commands(which[]) |
Select on which command a hook should send an event. Defaults to: ['set', 'del', 'drop'] |
.detect(what[]) |
Select what events should be detected. Defaults to: ['enter', 'exit', 'crosses', 'inside', 'outside'] |
Addition Information
For more information, please refer to:
Roadmap
See the open issues for a list of proposed features (and known issues).
Contributing
Contributions are what make the open source community such an amazing place to be learn, inspire, and create. Any contributions you make are greatly appreciated.
- Fork the Project
- Create your Feature Branch (
git checkout -b feature/AmazingFeature) - Commit your Changes (
git commit -m 'Add some AmazingFeature') - Push to the Branch (
git push origin feature/AmazingFeature) - Open a Pull Request
License
MIT
Maintainer
Benjamin Ramser - @iwpnd
Project Link: https://github.com/iwpnd/pyle38
Acknowledgements
Josh Baker - maintainer of Tile38
11 Mar 2, 2022
3 Mar 22, 2022
3 May 6, 2022
225 Dec 8, 2022
7 Oct 26, 2022
1 Feb 9, 2022
104 Dec 30, 2022
8 Aug 7, 2022
1 Jan 27, 2022
4 Apr 11, 2022
4 Dec 26, 2021
33 Dec 27, 2022
5 Dec 8, 2021
6 Sep 1, 2022
5 Oct 4, 2022
1 Feb 17, 2022
31 Dec 2, 2022
6 Mar 31, 2022
36 Dec 21, 2022
9 Dec 24, 2022