Platform for building statistical models of cities and regions

Related tags

Geolocation urbansim
Overview

UrbanSim

Latest Version Build Status Test Coverage

UrbanSim is a platform for building statistical models of cities and regions. These models help forecast long-range patterns in real estate development, demographics, and related outcomes, under various policy scenarios.

This urbansim Python library is a core component. It contains tools for statistical estimation and simulation; domain-specific logic about housing markets, household relocation, and other processes; and frameworks and utilities for assembling a model.

How it works

Operational UrbanSim models begin with detailed data about a particular region, and then estimate and validate a system of interconnected model components. Full models draw on a number of libraries: not just urbansim, but also Orca for task orchestration, Synthpop for population synthesis, Pandana for network analysis, and so on. Collectively, these make up the Urban Data Science Toolkit (UDST).

UrbanSim models are used by public agencies, consultancies, and researchers in dozens of cities around the U.S. and world. The core platform is open source, but many operational models make use of additional cloud-hosted model building and visualization tools provided by UrbanSim Inc.

Learn More

Installation

  • pip install urbansim
  • conda install urbansim --channel conda-forge

Technical documentation

Comments
  • High Level Interface

    High Level Interface

    I'm starting to put down some ideas for the high level interface to urbansim, you can see some sketches here: https://gist.github.com/jiffyclub/0bb253757547fbc14add. There isn't a huge difference between the three sketches there now, but I'll keep thinking about things and probably our dialogue will spur some ideas all around.

    Let me know what looks good, what's missing, what's on your wishlist for this, etc.

    opened by jiffyclub 20
  • Allow sampling of alternatives during prediction

    Allow sampling of alternatives during prediction

    It's kind of limited sampling, not something you could use during a location choice model. Alternatives may show up as available for more than one chooser and alternatives may show up more than once for a single chooser.

    opened by jiffyclub 18
  • Installing troubles

    Installing troubles

    I cant get conda to install urbansim > 1.3, Win 10, Py 2.7 64 bit

    IPythonNotebookScratch>conda config --add channels synthicity
    Skipping channels: synthicity, item already exists
    IPythonNotebookScratch>conda install urbansim
    Using Anaconda Cloud api site https://api.anaconda.org
    Fetching package metadata: ......
    Solving package specifications: ................................
    Package plan for installation in environment C:\Anaconda2:
    
    The following NEW packages will be INSTALLED:
    
        urbansim: 1.3-py27_0
    
    Proceed ([y]/n)?n
    IPythonNotebookScratch>conda install -c https://conda.anaconda.org/synthicity urbansim
    Using Anaconda Cloud api site https://api.anaconda.org
    Fetching package metadata: ........
    Solving package specifications: ................................
    Package plan for installation in environment C:\Anaconda2:
    
    The following NEW packages will be INSTALLED:
    
        urbansim: 1.3-py27_0
    
    Proceed ([y]/n)?n
    

    What am I doing wrong here?

    opened by Eh2406 16
  • Supply and Demand Corrections

    Supply and Demand Corrections

    @fscottfoti, this implements the supply/demand comparison and adjustment we'd specced out. Very simple, straight comparison of (probabilities * len(choosers)).groupby(alternatives[col].values).sum() / alternatives[col].value_counts() with adjustments done across 5 iterations, capped at 0.75-1.25 fractional change per iteration.

    opened by jiffyclub 16
  • transition model updates

    transition model updates

    Made some changes to the transition model. These include:

    1. Update _updated_linked_table method to use pandas merge:

    I was hitting a performance bottle-neck when adding a large number of agents (and therefore linked agents). I was adding ~70k agents and 194K linked agents and this was taking around ~34 seconds. Changing the code to use the merge reduced this to ~4 seconds.

    1. Add ability to control to a count provided by an 'accounting' column:

    This was a feature that MAG had added to the old opus code. For example, we may want to sample from households but the control we want to hit exactly is household population. I added a new sampling file to the utilities folder to handle most of the heavy lifting w/ the idea that maybe this functionality could be of use outside the transition model as well. Changes were then made to add and remove rows methods and each of the transition classes to accommodate this.

    I've tested this, and everything seems to be working fine. But I get that this is addressing pretty core functionality so if you are not comfortable bringing this in right now I'm cool w/ mirroring the transition file (something like mag_transition?). We could also leave in this in our specific urbansim implementation but I thought it might be of use to other agencies.

    opened by bridwell 16
  • Fix sample rows no replace

    Fix sample rows no replace

    This addresses feedback when the accounting total is not met (#178) and also providing probability distributions to influence the sample (#149).

    To obtain feedback, set the return_status argument to True, and this will return both the sampled rows and the status. If False, then only the sampled rows will be returned. This is still the default, so as to not break any existing calls.

    To provide sampling probabilities, set the prob_column argument to the name of the column in the data frame containing the weights or probabilities. These values will be normalized so they sum to 1, if they do not already.

    The logic for sampling with accounting but without replacement has been changed to hopefully be more robust.

    opened by bridwell 15
  • No feedback if iteration does not reach exact result in utils.sampling.sample_rows

    No feedback if iteration does not reach exact result in utils.sampling.sample_rows

    Function utils.sampling.sample rows includes a loop which shall converge to an exact pick of samples. In some cases, this loop cannot converge. Example:

    import pandas as pd
    import urbansim.utils.sampling
    df = pd.DataFrame({'tot': [2,2,2]})
    sample = urbansim.utils.sampling.sample_rows(total=3, data=df, replace=False, accounting_columns='tot')
    sample.sum()  # = 2 and not = 3 as expected
    

    Even if an exact pick is possible, the random permutation l.51 causes a further problem. If the values needed for an exact pick are at the head of the randomized list sample_idx, they are not available in the loop to adjust for an inexact pick, the loop cannot converge. This aspect depends on the random seed, so it can make integration tests fail randomly.

    Suggested solution: there should be some feedback from the sample_rows function whenever convergence to an exact pick is not achieved. The caller may use this information if he wishes. In this way a more robust test suite can be obtained.

    opened by moritz-kirmse-forcity 14
  • Cache scope

    Cache scope

    This adds a cache_scope option to the table, column, and injectable decorators that allow results to be automatically cleared from the cache at set points. The options are 'step', 'iter', and 'forever'. Items with scope 'step' are cleared after every model is run, items with scope 'iter' are cleared at the end of every model year, and things with scope 'forever' are never automatically cleared.

    This removes the need for table_source so that has been removed. The equivalent is now to use cache_scope='forever'.

    I waffled a little bit on using 'forever' or 'simulation' for the long-lived scope. With a scope of 'simulation' it'd make sense to clear it at the end of the simulation, with 'forever' it'd be best to never clear it. I'm up for either one, anyone have an opinion?

    opened by jiffyclub 14
  • Injectable Memoization

    Injectable Memoization

    This enables something like this:

    @sim.injectable(autocall=False, memoize=True)
    def my_utility_func(x, y, z):
        return x + y + z
    
    @sim.column()
    def my_col(another_col, my_utility_func):
        result = my_utility_func(1, 2, 3)
        return another_col * result
    

    And if you call my_utility_func again with the input of 1, 2, 3 you'll get back the cached result instead of re-doing the calculation.

    Memoization requires autocall=False and that function inputs always be hashable. The memoization cache follows the same rules as other caching, so you can use cache_scope=, sim.clear_cache, sim.disable_cache, and sim.enable_cache as usual.

    opened by jiffyclub 12
  • Account class

    Account class

    For tracking money. Right now it has .add_transaction(amount, subaccount, metadata), .add_transactions(transactions), and .to_frame() methods. You can access the balance or list of transactions via the .balance and .transactions attributes, respectively. What else does it need?

    opened by jiffyclub 10
  • use `pd.Int64Index()`

    use `pd.Int64Index()`

    • return pd.DataFrame with pd.Int64Index() index in remove_rows()
    • add assert_empty_int64index() to use with test_tgrtransition_with_accounting()

    I tried changing as little code as possible.

    Note that the failed build is due to changes in pandas (possibly 0.18 (or >=0.17.1)), as commented here.

    Alternatively, for the test_*remove*all() tests at least, the check_index_type option in pdt.assert_frame_equal() can be set to False. This isn't guaranteed to fix everything.

    opened by juanshishido 9
  • Deprecation in yaml conversion

    Deprecation in yaml conversion

    In Pandas 1.2+, pandas.Index.to_native_types() is deprecated, raising warnings like the following:

    Screen_Shot_2021-02-09_at_11 52 05_AM1

    This comes up in code that serializes data to yaml for storage and later reloading.

    urbansim/utils/yamlio.py#L48

    The replacement suggested in the message doesn't sound as general-purpose, but maybe it would work if Pandas is able to convert string representations of ints and floats back to the appropriate data type. Another option could be to use to_json().

    pandas.Index.to_native_types() pandas.Series.astype() pandas.Series.to_json()

    opened by smmaurer 0
  • Functionality of this repo

    Functionality of this repo

    Hey guys, I'm just going through this code and noticing, it doesn't seem to have the same functionality as what's been implemented in the UrbanSim Cloud Platform. Where can I find the files being used in the UrbanSim Cloud Platform? Are they also open source?

    Also, I have found an old version of the SqFtProForma.py file and it seems to have more functionality than the file in this repo. Was this file not working? Is this urbansim repo old?

    Old file was here https://github.com/UDST/developer/blob/master/developer/sqftproforma.py

    opened by eweyftw 1
  • No pandas tour

    No pandas tour

    Description of the bug

    Wes McKinney's intro to pandas is no more available http://udst.github.io/urbansim/gettingstarted.html#pandas

    Vimeo link points to nowhere https://vimeo.com/59324550

    opened by abitrolly 0
  • Update tests to support PyTest 4.0 +

    Update tests to support PyTest 4.0 +

    Some of our unit test syntax has been deprecated and removed in recent versions of PyTest.

    For example, tests that directly called the df() fixture defined in test_mnl.py#L82 were raising errors when I worked on PR #222.

    This shouldn't be too hard to fix, but will require going through and setting up some of the test data in alternative ways. It doesn't seem especially high-priority, though.

    For now, I've pinned pytest at v3.10 in the Travis and AppVeyor scripts.

    Type: maintenance 
    opened by smmaurer 0
  • Transition model, preserve index name?

    Transition model, preserve index name?

    When running transition model, the resulting data frame loses the index name (I'm guessing because of the concat). Is this worth preserving? This would be a simple fix, but not sure if it would break existing implementations.

    Type: bug 
    opened by bridwell 2
  • Transition model: option for relaxing conditions when filtering agents to be sampled from

    Transition model: option for relaxing conditions when filtering agents to be sampled from

    The transition model will not work well if the subset of agents to be sampled from is very small or even empty. Thus, an option of relaxing the filtering conditions (as present in Opus) would be very useful (e.g. if there are not enough households of the given characteristics within the given city, sample from the corresponding county or the whole region).

    If nothing else, an option to use a user-defined callback function would help, together with letting the model to access the model configuration. It would replace/extend the filter_table call: https://github.com/UDST/urbansim/blob/79f815a6503e109f50be270cee92d0f4a34f49ef/urbansim/models/transition.py#L305

    opened by hanase 2
Releases(v3.2)
  • v3.2(May 15, 2020)

    • Improved installation and compatibility
    • Support for Pandas 1.0
    • Various improvements and bug fixes
    • Note that active development of certain UrbanSim components has moved to stand-alone libraries in UDST: Developer, Choicemodels, UrbanSim Templates
    Source code(tar.gz)
    Source code(zip)
  • v3.1.1(May 9, 2017)

  • v3.1.0(May 9, 2017)

Owner
Urban Data Science Toolkit
Open source projects supporting urban spatial analysis, simulation, and visualization
Urban Data Science Toolkit
Hapi is a Python library for building Conceptual Distributed Model using HBV96 lumped model & Muskingum routing method

Current build status All platforms: Current release info Name Downloads Version Platforms Hapi - Hydrological library for Python Hapi is an open-sourc

Mostafa Farrag 15 Dec 26, 2022
GeoNode is an open source platform that facilitates the creation, sharing, and collaborative use of geospatial data.

Table of Contents What is GeoNode? Try out GeoNode Install Learn GeoNode Development Contributing Roadmap Showcase Most useful links Licensing What is

GeoNode Development Team 1.2k Dec 26, 2022
A Django application that provides country choices for use with forms, flag icons static files, and a country field for models.

Django Countries A Django application that provides country choices for use with forms, flag icons static files, and a country field for models. Insta

Chris Beaven 1.2k Jan 3, 2023
Helping data scientists better understand their datasets and models in text classification. With love from ServiceNow.

Azimuth, an open-source dataset and error analysis tool for text classification, with love from ServiceNow. Overview Azimuth is an open source applica

ServiceNow 145 Dec 23, 2022
Creates 3D geometries from 2D vector graphics, for use in geodynamic models

geomIO - creating 3D geometries from 2D input This is the Julia and Python version of geomIO, a free open source software to generate 3D volumes and s

null 3 Feb 1, 2022
OSMnx: Python for street networks. Retrieve, model, analyze, and visualize street networks and other spatial data from OpenStreetMap.

OSMnx OSMnx is a Python package that lets you download geospatial data from OpenStreetMap and model, project, visualize, and analyze real-world street

Geoff Boeing 4k Jan 8, 2023
Read and write rasters in parallel using Rasterio and Dask

dask-rasterio dask-rasterio provides some methods for reading and writing rasters in parallel using Rasterio and Dask arrays. Usage Read a multiband r

Dymaxion Labs 85 Aug 30, 2022
Location field and widget for Django. It supports Google Maps, OpenStreetMap and Mapbox

django-location-field Let users pick locations using a map widget and store its latitude and longitude. Stable version: django-location-field==2.1.0 D

Caio Ariede 481 Dec 29, 2022
Deal with Bing Maps Tiles and Pixels / WGS 84 coordinates conversions, and generate grid Shapefiles

PyBingTiles This is a small toolkit in order to deal with Bing Tiles, used i.e. by Facebook for their Data for Good datasets. Install Clone this repos

Shoichi 1 Dec 8, 2021
A short term landscape evolution using a path sampling method to solve water and sediment flow continuity equations and model mass flows over complex topographies.

r.sim.terrain A short-term landscape evolution model that simulates topographic change for both steady state and dynamic flow regimes across a range o

Brendan Harmon 7 Oct 21, 2022
Using Global fishing watch's data to build a machine learning model that can identify illegal fishing and poaching activities through satellite and geo-location data.

Using Global fishing watch's data to build a machine learning model that can identify illegal fishing and poaching activities through satellite and geo-location data.

Ayush Mishra 3 May 6, 2022
This repository contains the scripts to derivate the ENU and ECEF coordinates from the longitude, latitude, and altitude values encoded in the NAD83 coordinates.

This repository contains the scripts to derivate the ENU and ECEF coordinates from the longitude, latitude, and altitude values encoded in the NAD83 coordinates.

Luigi Cruz 1 Feb 7, 2022
Python bindings and utilities for GeoJSON

geojson This Python library contains: Functions for encoding and decoding GeoJSON formatted data Classes for all GeoJSON Objects An implementation of

Jazzband 765 Jan 6, 2023
Manipulation and analysis of geometric objects

Shapely Manipulation and analysis of geometric objects in the Cartesian plane. Shapely is a BSD-licensed Python package for manipulation and analysis

null 3.1k Jan 3, 2023
Python interface to PROJ (cartographic projections and coordinate transformations library)

pyproj Python interface to PROJ (cartographic projections and coordinate transformations library). Documentation Stable: http://pyproj4.github.io/pypr

null 832 Dec 31, 2022
Rasterio reads and writes geospatial raster datasets

Rasterio Rasterio reads and writes geospatial raster data. Geographic information systems use GeoTIFF and other formats to organize and store gridded,

Mapbox 1.9k Jan 7, 2023
Fiona reads and writes geographic data files

Fiona Fiona reads and writes geographic data files and thereby helps Python programmers integrate geographic information systems with other computer s

null 987 Jan 4, 2023
Python bindings and utilities for GeoJSON

geojson This Python library contains: Functions for encoding and decoding GeoJSON formatted data Classes for all GeoJSON Objects An implementation of

Jazzband 763 Dec 26, 2022
Documentation and samples for ArcGIS API for Python

ArcGIS API for Python ArcGIS API for Python is a Python library for working with maps and geospatial data, powered by web GIS. It provides simple and

Esri 1.4k Dec 30, 2022