📊📈 Serves up Pandas dataframes via the Django REST Framework for use in client-side (i.e. d3.js) visualizations and offline analysis (e.g. Excel)

wq framework

Last update: Jan 1, 2023

Related tags

Data Visualization chart django csv dataviz rest-api excel django-rest-framework spreadsheet pandas

Overview

Django REST Pandas

Django REST Framework + pandas = A Model-driven Visualization API

Django REST Pandas (DRP) provides a simple way to generate and serve pandas DataFrames via the Django REST Framework. The resulting API can serve up CSV (and a number of other formats) for consumption by a client-side visualization tool like d3.js.

The design philosophy of DRP enforces a strict separation between data and presentation. This keeps the implementation simple, but also has the nice side effect of making it trivial to provide the source data for your visualizations. This capability can often be leveraged by sending users to the same URL that your visualization code uses internally to load the data.

DRP does not include any JavaScript code, leaving the implementation of interactive visualizations as an exercise for the implementer. That said, DRP is commonly used in conjunction with the wq.app library, which provides wq/chart.js and wq/pandas.js, a collection of chart functions and data loaders that work well with CSV served by DRP.

Live Demo

The climata-viewer project uses Django REST Pandas and wq/chart.js to provide interactive visualizations and spreadsheet downloads.

Related Work

The field of Python-powered data analysis and visualization is growing, and there are a number of similar solutions that may fit your needs better.

Django Pandas provides a custom ORM model manager with pandas support. By contrast, Django REST Pandas works at the view level, by integrating pandas via custom Django REST Framework serializers and renderers.
DRF-CSV provides straightforward CSV renderers for use with Django REST Framework. It may be useful if you just want a CSV API and don't have a need for the pandas DataFrame functionality.
mpld3 provides a direct bridge from matplotlib to d3.js, complete with seamless IPython integration. It is restricted to the (large) matplotlib chart vocabularly but should be sufficient for many use cases.
Bokeh is a complete client-server visualization platform. It does not leverage d3 or Django, but is notable as a comprehensive, forward-looking approach to addressing similar use cases.

The goal of Django REST Pandas is to provide a generic REST API for serving up pandas dataframes. In this sense, it is similar to the Plot Server in Bokeh, but more generic in that it does not assume any particular visualization format or technology. Further, DRP is optimized for integration with public-facing Django-powered websites (unlike mpld3 which is primarily intended for use within IPython).

In summary, DRP is designed for use cases where:

You want to support live spreadsheet downloads as well as interactive visualizations, and/or
You want full control over the client visualization stack in order to integrate it with the rest of your website and/or build process. This usually means writing JavaScript code by hand. mpld3 may be a better choice for data exploration if you are more comfortable with (I)Python and need something that can generate interactive visualizations out of the box.

Supported Formats

The following output formats are provided by default. These are provided as renderer classes in order to leverage the content type negotiation built into Django REST Framework. This means clients can specify a format via:

an HTTP "Accept" header (Accept: text/csv),
a format parameter (/path?format=csv), or
a format extension (/path.csv)

The HTTP header and format parameter are enabled by default on every pandas view. Using the extension requires a custom URL configuration (see below).

Format	Content Type	pandas DataFrame Function	Notes
HTML	`text/html`	`to_html()`	See notes on HTML output
CSV	`text/csv`	`to_csv()`
TXT	`text/plain`	`to_csv()`	Useful for testing, as most browsers will download a CSV file instead of displaying it
JSON	`application/json`	`to_json()`	`date_format` and `orient` can be provided in URL (e.g. `/path.json?orient=columns`)
XLSX	`application/vnd.openxml...sheet`	`to_excel()`
XLS	`application/vnd.ms-excel`	`to_excel()`
PNG	`image/png`	`plot()`	Currently not very customizable, but a simple way to view the data as an image.
SVG	`image/svg`	`plot()`	Eventually these could become a fallback for clients that can't handle d3.js

The underlying implementation is a set of serializers that take the normal serializer result and put it into a dataframe. Then, the included renderers generate the output using the built in pandas functionality.

Usage

Getting Started

# Recommended: create virtual environment
# python3 -m venv venv
# . venv/bin/activate
pip install rest-pandas

NOTE: Django REST Pandas relies on pandas, which itself relies on NumPy and other scientific Python libraries written in C. This is usually fine, since pip can use Python Wheels to install precompiled versions. If you are having trouble installing DRP due to dependency issues, you may need to pre-install pandas using apt or conda.

Usage Examples

No Model

The example below allows you to create a simple API for an existing Pandas DataFrame, e.g. generated from an existing file.

# views.py
from rest_pandas import PandasSimpleView
import pandas as pd

class TimeSeriesView(PandasSimpleView):
    def get_data(self, request, *args, **kwargs):
        return pd.read_csv('data.csv')

Model-Backed

The example below assumes you already have a Django project set up with a single TimeSeries model.

# views.py
from rest_pandas import PandasView
from .models import TimeSeries
from .serializers import TimeSeriesSerializer

# Short version (leverages default DRP settings):
class TimeSeriesView(PandasView):
    queryset = TimeSeries.objects.all()
    serializer_class = TimeSeriesSerializer
    # That's it!  The view will be able to export the model dataset to any of
    # the included formats listed above.  No further customization is needed to
    # leverage the defaults.

# Long Version and step-by-step explanation
class TimeSeriesView(PandasView):
    # Assign a default model queryset to the view
    queryset = TimeSeries.objects.all()

    # Step 1. In response to get(), the underlying Django REST Framework view
    # will load the queryset and then pass it to the following function.
    def filter_queryset(self, qs): 
        # At this point, you can filter queryset based on self.request or other
        # settings (useful for limiting memory usage).  This function can be
        # omitted if you are using a filter backend or do not need filtering.
        return qs
        
    # Step 2. A Django REST Framework serializer class should serialize each
    # row in the queryset into a simple dict format.  A simple ModelSerializer
    # should be sufficient for most cases.
    serializer_class = TimeSeriesSerializer  # extends ModelSerializer

    # Step 3.  The included PandasSerializer will load all of the row dicts
    # into array and convert the array into a pandas DataFrame.  The DataFrame
    # is essentially an intermediate format between Step 2 (dict) and Step 4
    # (output format).  The default DataFrame simply maps each model field to a
    # column heading, and will be sufficient in many cases.  If you do not need
    # to transform the dataframe, you can skip to step 4.
    
    # If you would like to transform the dataframe (e.g. to pivot or add
    # columns), you can do so in one of two ways:

    # A. Create a subclass of PandasSerializer, define a function called
    # transform_dataframe(self, dataframe) on the subclass, and assign it to
    # pandas_serializer_class on the view.  You can also use one of the three
    # provided pivoting serializers (see Advanced Usage below).
    #
    # class MyCustomPandasSerializer(PandasSerializer):
    #     def transform_dataframe(self, dataframe):
    #         dataframe.some_pivot_function(in_place=True)
    #         return dataframe
    #
    pandas_serializer_class = MyCustomPandasSerializer

    # B. Alternatively, you can create a custom transform_dataframe function
    # directly on the view.  Again, if no custom transformations are needed,
    # this function does not need to be defined.
    def transform_dataframe(self, dataframe):
        dataframe.some_pivot_function(in_place=True)
        return dataframe
    
    # NOTE: As the name implies, the primary purpose of transform_dataframe()
    # is to apply a transformation to an existing dataframe.  In PandasView,
    # this dataframe is created by serializing data queried from a Django
    # model.  If you would like to supply your own custom DataFrame from the
    # start (without using a Django model), you can do so with PandasSimpleView
    # as shown in the first example.

    # Step 4. Finally, the provided renderer classes will convert the DataFrame
    # to any of the supported output formats (see above).  By default, all of
    # the formats above are enabled.  To restrict output to only the formats
    # you are interested in, you can define renderer_classes on the view:
    renderer_classes = [PandasCSVRenderer, PandasExcelRenderer]
    # You can also set the default renderers for all of your pandas views by
    # defining the PANDAS_RENDERERS in your settings.py.

    # Step 5 (Optional).  The default filename may not be particularly useful
    # for your users.  To override, define get_pandas_filename() on your view.
    # If a filename is returned, rest_pandas will include the following header:
    # 'Content-Disposition: attachment; filename="Data Export.xlsx"'
    def get_pandas_filename(self, request, format):
        if format in ('xls', 'xlsx'):
            # Use custom filename and Content-Disposition header
            return "Data Export"  # Extension will be appended automatically
        else:
            # Default filename from URL (no Content-Disposition header)
            return None

Integrating with Existing Views

If you have an existing viewset, or are otherwise unable to directly subclass one of PandasSimpleView, PandasView, PandasViewSet, or PandasMixin, you can also integrate the renderers and serializer directly. The most important thing is to ensure that your serializer class has Meta.list_serializer_class set to PandasSerializer or a subclass. Then, make sure that the Pandas renderers are included in your renderer options. See #32 and #36 for examples.

Django Pandas Integration

You can also let Django Pandas handle querying and generating the dataframe, and only use Django REST Pandas for the rendering:

# models.py
from django_pandas.managers import DataFrameManager

class TimeSeries(models.Model):
    # ...
    objects = DataFrameManager()

# views.py
from rest_pandas import PandasSimpleView
from .models import TimeSeries

class TimeSeriesView(PandasSimpleView):
    def get_data(self, request, *args, **kwargs):
        return TimeSeries.objects.to_timeseries(
            index='date',
        )

Registering URLs

# urls.py
from django.conf.urls import patterns, include, url

from .views import TimeSeriesView
urlpatterns = patterns('',
    url(r'^data', TimeSeriesView.as_view()),
)

# This is only required to support extension-style formats (e.g. /data.csv)
from rest_framework.urlpatterns import format_suffix_patterns
urlpatterns = format_suffix_patterns(urlpatterns)

The default PandasView will serve up all of the available data from the provided model in a simple tabular form. You can also use a PandasViewSet if you are using Django REST Framework's ViewSets and Routers.

Customizing Renderers

You can override the default renderers by setting PANDAS_RENDERERS in your settings.py, or by overriding renderer_classes in your individual view(s). PANDAS_RENDERERS is defined separately from Django REST Framework's own DEFAULT_RENDERER_CLASSES setting, in case you want to have DRP-enabled views intermingled with regular DRF views.

You can also include DRP renderers in DEFAULT_RENDERER_CLASSES. In that case, extend PandasMixin or set list_serializer_class on your serializer. Otherwise, you may get an error saying the serializer output is not a DataFrame. In short, there are three paths to getting DRP renderers working with your views:

Extend PandasView, PandasSimpleView, or PandasViewSet, and use the PANDAS_RENDERERS setting (which defaults to the list above).
Extend PandasMixin and customize REST_FRAMEWORK['DEFAULT_RENDERER_CLASSES'] to add one or more rest_pandas renderers.
Set renderer_classes explicitly on the view, and set Serializer.Meta.list_serializer_class to PandasSerializer or a subclass. (See #32 and #36 for examples.)

class TimeSeriesView(PandasView):
    # renderer_classes default to PANDAS_RENDERERS
    ...

class TimeSeriesView(PandasMixin, ListAPIView):
    # renderer_classes default to REST_FRAMEWORK['DEFAULT_RENDERER_CLASSES']
    ...

Date Formatting

By default, Django REST Framework will serialize dates as strings before they are processed by the renderer classes. In many cases, you may want to preserve the dates as datetime objects and let Pandas handle the rendering. To do this, define an explicit DateTimeField or DateField on your DRF serializer and set format=None:

# serializers.py
class TimeSeriesSerializer(serializers.ModelSerializer):
    date = serializers.DateField(format=None)
    class Meta:
        model = TimeSeries
        fields = '__all__'

Alternately, you can disable date serialization globally by setting DATETIME_FORMAT and/or DATE_FORMAT to None in your settings.py:

# settings.py
DATE_FORMAT = None

HTML Output

The HTML renderer provides the ability to create an interactive view that shares the same URL as your data API. The dataframe is processed by to_html(), then passed to TemplateHTMLRenderer with the following context:

context variable description

table

Output

to_html()
     
       from   
      
        
        name 
        View name 
        
        
        description 
        View description 
        
        
        url 
        Current URL Path (without parameters) 
        
        
        url_params 
        URL parameters 
        
        
        available_formats 
        Array of allowed extensions (e.g. 'csv', 'json', 'xlsx') 
        
        
        wq_chart_type 
        Recommended chart type (for use with wq/chartapp.js, see below) 
        
       
     
 As with TemplateHTMLRenderer, the template name is controlled by the view. If you are using DRP together with the wq framework, you can leverage the default mustache/rest_pandas.html template, which is designed for use with the wq/chartapp.js plugin. Otherwise, you will probably want to provide a custom template and/or set template_name on the view.
 If you need to do a lot of customization, and/or you don't really need the entire dataframe rendered in a 
PandasView
      
      Note: For backwards compatibility, PandasHTMLRenderer is only included in the default PANDAS_RENDERERS if rest_pandas is listed in your installed apps. 
     
 
       Building Interactive Charts
In addition to use as a data export tool, DRP is well-suited for creating data API backends for interactive charts. In particular, DRP can be used with d3.js, wq/pandas.js, and wq/chart.js, to create interactive time series, scatter, and box plot charts - as well as any of the infinite other charting possibilities d3.js provides.
To facilitate data API building, the CSV renderer is the default in Django REST Pandas. While the pandas JSON serializer is improving, the primary reason for making CSV the default is the compactness it provides over JSON when serializing time series data. The default CSV output from DRP will have single row of column headers, making it suitable as-is for use with e.g. d3.csv(). However, DRP is often used with the custom serializers below to produce a dataframe with nested multi-row column headers. This is harder to parse with d3.csv() but can be easily processed by wq/pandas.js, an extension to d3.js.
      
      // mychart.js
define(['d3', 'wq/pandas', 'wq/chart'], function(d3, pandas, chart) {

// Unpivoted data (single-row header)
d3.csv("/data.csv", render);

// Pivoted data (multi-row header)
pandas.get('/data.csv', render);

function render(error, data) {
    d3.select('svg')
       .selectAll('rect')
       .data(data)
       // ...
}

}); 
     
DRP includes three custom serializers with transform_dataframe() functions that address common use cases. These serializer classes can be leveraged by assigning them to pandas_serializer_class on your view. If you are using the wq framework, these serializers can automatically leverage DRP's default HTML template together with wq/chartapp.js to provide interactive charts. If you are not using the full wq framework, you can still use wq/pandas.js and wq/chart.js directly with the CSV output of these serializers.
For documentation purposes, the examples below assume the following dataset:
     
      , you can always create another view for the interface and make the  only handle the API.        
     
      
       
        
        Location 
        Measurement 
        Date 
        Value 
        
       
       
        
        site1 
        temperature 
        2016-01-01 
        3 
        
        
        site1 
        humidity 
        2016-01-01 
        30 
        
        
        site2 
        temperature 
        2016-01-01 
        4 
        
        
        site2 
        temperature 
        2016-01-02 
        5 
        
       
     
  
       PandasUnstackedSerializer
 PandasUnstackedSerializer unstacks the dataframe so a few key attributes are listed in a multi-row column header. This makes it easier to include metadata about e.g. a time series without repeating the same values on every data row.
 To specify which attributes to use in column headers, define the attribute pandas_unstacked_header on your ModelSerializer subclass. You will generally also want to define pandas_index, which is a list of metadata fields unique to each row (e.g. the timestamp). 
      
      # serializers.py
from rest_framework import serializers
from .models import TimeSeries

class TimeSeriesSerializer(serializers.ModelSerializer):
    class Meta:
        model = MultiTimeSeries
        fields = ['date', 'location', 'measurement', 'value']
        pandas_index = ['date']
        pandas_unstacked_header = ['location', 'measurement']

# views.py
from rest_pandas import PandasView, PandasUnstackedSerializer
from .models import TimeSeries
from .serializers import TimeSeriesSerializer

class TimeSeriesView(PandasView):
    queryset = TimeSeries.objects.all()
    serializer_class = TimeSeriesSerializer
    pandas_serializer_class = PandasUnstackedSerializer 
     
 With the above example data, this configuration would output a CSV file with the following layout: 
      
       
        
          
        Value 
        Value 
        Value 
        
       
       
        
        Location 
        site1 
        site1 
        site2 
        
        
        Measurement 
        temperature 
        humidity 
        temperature 
        
        
        Date 
         
         
          
        
        
        2016-01-01 
        3 
        30 
        4 
        
        
        2016-01-02 
         
         
        5 
        
       
     
 This could then be processed by wq/pandas.js into the following structure: 
      
      [
    {
        "location": "site1",
        "measurement": "temperature",
        "data": [
            {"date": "2016-01-01", "value": 3}
        ]
    },
    {
        "location": "site1",
        "measurement": "humidity",
        "data": [
            {"date": "2016-01-01", "value": 30}
        ]
    },
    {
        "location": "site2",
        "measurement": "temperature",
        "data": [
            {"date": "2016-01-01", "value": 4},
            {"date": "2016-01-02", "value": 5}
        ]
    }
] 
     
 The output of PandasUnstackedSerializer can be used with the timeSeries() chart provided by wq/chart.js: 
      
      define(['d3', 'wq/pandas', 'wq/chart'], function(d3, pandas, chart) {

var svg = d3.select('svg');
var plot = chart.timeSeries();
pandas.get('/data/timeseries.csv', function(data) {
    svg.datum(data).call(plot);
});

}); 
     
  
       PandasScatterSerializer
 PandasScatterSerializer unstacks the dataframe and also combines selected attributes to make it easier to plot two measurements against each other in an x-y scatterplot.
 To specify which attributes to use for the coordinate names, define the attribute pandas_scatter_coord on your ModelSerializer subclass. You can also specify additional metadata attributes to include in the header with pandas_scatter_header. You will generally also want to define pandas_index, which is a list of metadata fields unique to each row (e.g. the timestamp). 
      
      # serializers.py
from rest_framework import serializers
from .models import TimeSeries

class TimeSeriesSerializer(serializers.ModelSerializer):
    class Meta:
        model = MultiTimeSeries
        fields = ['date', 'location', 'measurement', 'value']
        pandas_index = ['date']
        pandas_scatter_coord = ['measurement']
        pandas_scatter_header = ['location']

# views.py
from rest_pandas import PandasView, PandasScatterSerializer
from .models import TimeSeries
from .serializers import TimeSeriesSerializer

class TimeSeriesView(PandasView):
    queryset = TimeSeries.objects.all()
    serializer_class = TimeSeriesSerializer
    pandas_serializer_class = PandasScatterSerializer 
     
 With the above example data, this configuration would output a CSV file with the following layout: 
      
       
        
          
        temperature-value 
        humidity-value 
        temperature-value 
        
       
       
        
        Location 
        site1 
        site1 
        site2 
        
        
        Date 
         
         
          
        
        
        2014-01-01 
        3 
        30 
        4 
        
        
        2014-01-02 
         
         
        5 
        
       
     
 This could then be processed by wq/pandas.js into the following structure: 
      
      [
    {
        "location": "site1",
        "data": [
            {
                "date": "2016-01-01",
                "temperature-value": 3,
                "humidity-value": 30
            }
        ]
    },
    {
        "location": "site2",
        "data": [
            {
                "date": "2016-01-01",
                "temperature-value": 4
            },
            {
                "date": "2016-01-02",
                "temperature-value": 5
            }
        ]
    }
] 
     
 The output of PandasScatterSerializer can be used with the scatter() chart provided by wq/chart.js: 
      
      define(['d3', 'wq/pandas', 'wq/chart'], function(d3, pandas, chart) {

var svg = d3.select('svg');
var plot = chart.scatter()
    .xvalue(function(d) {
        return d['temperature-value'];
    })
    .yvalue(function(d) {
        return d['humidity-value'];
    });

pandas.get('/data/scatter.csv', function(data) {
    svg.datum(data).call(plot);
});

}); 
     
  
       PandasBoxplotSerializer
 PandasBoxplotSerializer computes boxplot statistics (via matplotlib's boxplot_stats) and pushes the results out via an unstacked dataframe. The statistics can be aggregated for a specified group column as well as by date.
 To specify which attribute to use for the group column, define the attribute pandas_boxplot_group on your ModelSerializer subclass. To specify an attribute to use for date-based grouping, define pandas_boxplot_date. You will generally also want to define pandas_boxplot_header, which will unstack any metadata columns and exclude them from statistics. 
      
      # serializers.py
from rest_framework import serializers
from .models import TimeSeries

class TimeSeriesSerializer(serializers.ModelSerializer):
    class Meta:
        model = MultiTimeSeries
        fields = ['date', 'location', 'measurement', 'value']
        pandas_boxplot_group = 'site'
        pandas_boxplot_date = 'date'
        pandas_boxplot_header = ['measurement']

# views.py
from rest_pandas import PandasView, PandasBoxplotSerializer
from .models import TimeSeries
from .serializers import TimeSeriesSerializer

class TimeSeriesView(PandasView):
    queryset = TimeSeries.objects.all()
    serializer_class = TimeSeriesSerializer
    pandas_serializer_class = PandasBoxplotSerializer 
     
 With the above example data, this configuration will output a CSV file with the same general structure as PandasUnstackedSerializer, but with the value spread across multiple boxplot statistics columns (value-mean, value-q1,value-whishi, etc.). An optional group` parameter can be added to the query string to switch between various groupings: 
      
       
        
        name 
        purpose 
        
       
       
        
        ?group=series 
        Group by series (pandas_boxplot_group) 
        
        
        ?group=series-year 
        Group by series, then by year 
        
        
        ?group=series-month 
        Group by series, then by month 
        
        
        ?group=year 
        Summarize all data by year 
        
        
        ?group=month 
        Summarize all data by month 
        
       
     
 The output of PandasBoxplotSerializer can be used with the boxplot() chart provided by wq/chart.js: 
      
      define(['d3', 'wq/pandas', 'wq/chart'], function(d3, pandas, chart) {

var svg = d3.select('svg');
var plot = chart.boxplot();
pandas.get('/data/boxplot.csv?group=year', function(data) {
    svg.datum(data).call(plot);
});

});

Comments

PandasView Upload File - Example Request

Hi,

Currently I am working on receiving files form users (csv, xlsx) and pass this files to rest-pandas so its content can be imported to your models or any custom python object.

Before going further I would like to know if the scope of this project includes importing data from files.

If this is part of scope or if project leaders agree to include this part I will be posting my progress.

Have a nice day!

opened by klahnen 12

WindowsError: [error 32]

I am experiencing this error on Windows 7.

WindowsError: [error 32] The process cannot access the file because it is being used by another    process.

When this code is executed:

def transform_dataframe(self, dataframe):
    from pandas.io.excel import ExcelWriter
    writer = ExcelWriter('output.xlsx')
    dataframe.to_excel(writer, 'Page1')
    return dataframe

Output.xlsx is being generated on the root folder of the project.

opened by klahnen 9

Using a renderer in DRF settings for Excel output option
I'm trying to create an option on all of my endpoints to output XLS files when the appropriate header is set. I may not be understanding correctly, but here's what I've tried to do:

REST_FRAMEWORK = { ... 'DEFAULT_RENDERER_CLASSES': ( 'rest_framework.renderers.JSONRenderer', 'rest_framework.renderers.BrowsableAPIRenderer', 'rest_pandas.renderers.PandasExcelRenderer', ), }

I haven't included it in INSTALLED_APPS as I've read about the problems that causes. However, when trying this method, Django's runserver is giving me this error:

ImportError: Could not import 'rest_pandas.renderers.PandasExcelRenderer' for API setting 'DEFAULT_RENDERER_CLASSES'. ImportError: cannot import name 'APIView'.

When I try this, runserver comes up but the OPTIONS only shows application/json and text/html under the renders section:

from rest_pandas.renderers import PandasExcelRenderer REST_FRAMEWORK = { ... 'DEFAULT_RENDERER_CLASSES': ( 'rest_framework.renderers.JSONRenderer', 'rest_framework.renderers.BrowsableAPIRenderer', PandasExcelRenderer, ), }

I'm sure I'm missing something simple, and I'll be happy to issue a how-to PR on the README if someone can help me get this working! :)
opened by FlipperPA 5
models with ID columns not named ID cause exception
With a model that has an ID column not named ID, and the following code:

class TestView(PandasView): queryset = WeirdIDModel.objects.all() serializer_class = WeirdIDModelSerializer

when trying to query the class, you get an error KeyError, "id".

Reading that stack trace and the source I see that this is because by default we run "set_index('id') on the df, and this can be fixed in three obvious ways:

set pandas_index on the meta of the serializer

override get_index_fields on the serializer

override get_dataframe

setting the index seems to be needless duplication (I've already written once what the index of this model is)

an override to get_index_fields I think would just create behaviour that should be default: WeirdIDModelSerializer._meta.pk.name gives the ID, for my version of Django at least

so something like

try: id_name = (self.<field for model>)._meta.pk.name except: id_name = 'id' default_fields = [id_name]

Thinking about it more, and reading the source, it seems odd to even set an index, especially without preserving the original column - I think that DRF classes that we're mimicking would by default pass the index through to the client, and since I'm trying to drop-in replace some existing code which might rely on that, I'll have to replicate that manually.
opened by FalseProtagonist 3

rest_pandas in INSTALLED_APPS breaks in Django 1.9

I use: Python 2.7 Django 1.9 DRF 3.2.2 Pandas 0.17.1 And i have this problem:

Traceback (most recent call last):
  File "manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/usr/local/lib/python2.7/dist-packages/django/core/management/__init__.py", line 350, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python2.7/dist-packages/django/core/management/__init__.py", line 324, in execute
    django.setup()
  File "/usr/local/lib/python2.7/dist-packages/django/__init__.py", line 18, in setup
    apps.populate(settings.INSTALLED_APPS)
  File "/usr/local/lib/python2.7/dist-packages/django/apps/registry.py", line 85, in populate
    app_config = AppConfig.create(entry)
  File "/usr/local/lib/python2.7/dist-packages/django/apps/config.py", line 90, in create
    module = import_module(entry)
  File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module
    __import__(name)
  File "build/bdist.linux-x86_64/egg/rest_pandas/__init__.py", line 1, in <module>
  File "build/bdist.linux-x86_64/egg/rest_pandas/views.py", line 1, in <module>
  File "/usr/local/lib/python2.7/dist-packages/rest_framework/views.py", line 98, in <module>
    class APIView(View):
  File "/usr/local/lib/python2.7/dist-packages/rest_framework/views.py", line 103, in APIView
    authentication_classes = api_settings.DEFAULT_AUTHENTICATION_CLASSES
  File "/usr/local/lib/python2.7/dist-packages/rest_framework/settings.py", line 203, in __getattr__
    val = perform_import(val, attr)
  File "/usr/local/lib/python2.7/dist-packages/rest_framework/settings.py", line 148, in perform_import
    return [import_from_string(item, setting_name) for item in val]
  File "/usr/local/lib/python2.7/dist-packages/rest_framework/settings.py", line 160, in import_from_string
    module = importlib.import_module(module_path)
  File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module
    __import__(name)
  File "/usr/local/lib/python2.7/dist-packages/rest_framework/authentication.py", line 13, in <module>
    from rest_framework.authtoken.models import Token
  File "/usr/local/lib/python2.7/dist-packages/rest_framework/authtoken/models.py", line 16, in <module>
    class Token(models.Model):
  File "/usr/local/lib/python2.7/dist-packages/django/db/models/base.py", line 94, in __new__
    app_config = apps.get_containing_app_config(module)
  File "/usr/local/lib/python2.7/dist-packages/django/apps/registry.py", line 239, in get_containing_app_config
    self.check_apps_ready()
  File "/usr/local/lib/python2.7/dist-packages/django/apps/registry.py", line 124, in check_apps_ready
    raise AppRegistryNotReady("Apps aren't loaded yet.")
django.core.exceptions.AppRegistryNotReady: Apps aren't loaded yet.

opened by Dipolle 3

date_format not used when creating csv

To modify the behavior of the csv renderer, I have subclassed PandasCSVRenderer

class PandasCSVNoColNoIndexRenderer(PandasCSVRenderer):
    def get_pandas_kwargs(self, data, renderer_context):
        output = super().get_pandas_kwargs(data, renderer_context)
        output['header'] = False
        output['index'] = False
        # Remark on padding zeros: 
        #   removing padding zeros is platform dependent so we better leave it this way (%-d in linux, %#d in windos)
        #   https://stackoverflow.com/questions/28894172/why-does-d-or-e-remove-the-leading-space-or-zero
        output['sep'] = ';'
        output['date_format'] = '%d-%m-%Y %H:%M'  

        return output

All options work out fine, except for the 'date_format' option. I have set in settings.py

and indeed when inside get_pandas_kwargs

renderer_context['response'].data.iloc[10,3]

returns

Timestamp('1985-07-01 00:00:00')

The string is also correct as

renderer_context['response'].data.iloc[10,3].strftime('%d-%m-%Y %H:%M')

yields the desired output (30-11-1999 01:02)

Do you have any clue why this is happening?

opened by martinwk 2

Response data is a ReturnList, not a DataFrame! Did you extend PandasMixin?

I really would love to use this nice package but I'm getting this error no matter what I do.

I see that no one is really maintaining this package anymore but I really hope someone may clarify this for me.

Added 'rest_pandas' in installed app BEFORE 'rest_framework'
Added 'rest_pandas.renderers.PandasExcelRenderer' in DEFAULT_RENDERED_CLASS
I see an xlsx selectable option in the api django rest framework view
Added PandasMixin to my BaseView
Response data is a ReturnList, not a DataFrame! Did you extend PandasMixin?

Following my baseApi view:

class BaseApi(mixins.RetrieveModelMixin,
              mixins.ListModelMixin,
              mixins.UpdateModelMixin,
              mixins.DestroyModelMixin,
              PandasMixin,
              BulkCreateModelMixin,
              BulkUpdateModelMixin,
              generics.GenericAPIView):

    filter_backends = (OrderingFilter, SearchFilter, DjangoFilterBackend)
    pagination_class = StandardResultsSetPagination

    def get_list(self, request, *args, **kwargs):

        queryset = self.filter_queryset(self.get_queryset())

        page = self.paginate_queryset(queryset)

        if page is not None:
            serializer = self.get_serializer(page, many=True)
            response = self.get_paginated_response(serializer.data)
        else:
            serializer = self.get_serializer(queryset, many=True)
            response = Response(serializer.data)

        query = {
            "ordering_fields": [],
            "filter_fields": [],
            "search_fields": [],
            "element_count": len(response.data),
            "element_total_count": queryset.count(),
            "previous_page": self.paginator.get_previous_link(),
            "next_page": self.paginator.get_next_link(),
            "page_count": self.paginator.page.paginator.num_pages,
            "current_page_size": self.paginator.get_page_size(request),
            "current_page": self.paginator.page.number,
            "max_page_size": self.paginator.max_page_size,
        }

        for backend in list(self.filter_backends):
            if isinstance(backend(), OrderingFilter):
                query["ordering_fields"] = backend().get_ordering(
                    request=self.request,
                    queryset=self.queryset.all(),
                    view=self
                ) or []
            elif isinstance(backend(), SearchFilter):
                query["search_fields"] = backend().get_search_terms(request)
            elif isinstance(backend(), DjangoFilterBackend):
                for f in self.request.query_params.keys():
                    if hasattr(self, 'filter_fields'):
                        if f in self.filter_fields:
                            query["filter_fields"].append({f: self.request.query_params[f]})
                    elif hasattr(self, 'filter_class'):
                        if f in self.filter_class.Meta.fields:
                            query["filter_fields"].append({f: self.request.query_params[f]})

        for k, v in query.items():
            response[k.replace("_", " ").title().replace(" ", "-")] = json.dumps(v)

        if "debug" in self.request.query_params.keys() and self.request.query_params["debug"] == "true":
            response.data = {
                "query": query,
                "data": response.data,
            }

        return response

    # API BASE VIEWS ###########################
    def get(self, request, *args, **kwargs):
        if 'pk' in kwargs.keys():
            return self.retrieve(request, *args, **kwargs)
        else:
            return self.get_list(request, *args, **kwargs)

    def put(self, request, *args, **kwargs):
        if 'pk' in kwargs.keys():
            return self.update(request, partial=True)
        else:
            return self.partial_bulk_update(request, *args, **kwargs)

    def delete(self, request, *args, **kwargs):
        return self.destroy(request, *args, **kwargs)

    def post(self, request, *args, **kwargs):
        return self.create(request, *args, **kwargs)
    # END API BASE VIEWS #######################

opened by pinkynrg 2

Calling `PandasBaseRenderer().render(data)` throws `TypeError`
For personal reasons, I wanted to call the PandasCSVRenderer on a dataframe outside of a view.

Because renderer_context is defaulted to None, the following code will throw TypeError: argument of type 'NoneType' is not iterable:

from rest_pandas import renderers data_frame = pandas.DataFrame(some_data) print renderers.PandasCSVRenderer().render(data_frame) >>>TypeError: argument of type 'NoneType' is not iterable

I suggest two options to fix the following code: https://github.com/wq/django-rest-pandas/blob/edd3225acb21fcb0e36d8889d7bdfa5dedc5d67b/rest_pandas/renderers.py#L30

Change the if statement:

if renderer_context and 'response' in renderer_context:

Add the following at the top of the function:

if renderer_context is None: renreder_context = {} ...
opened by arthurio 2

Permission_class is ignored

As PandasView inherits from APIListView I would expect that the attribute permission_classes can be used to set permission on this view. However, the following view can be accessed when logged out

class DownloadFileViewAPI(PandasView):
    queryset = Well.objects.all()
    permission_classes = (MayUploadWriteOrDownloadRead,)
    filter_class = WellFilter
   
    serializer_class        = MetaDataSerializer # extends Serializer
    pandas_serializer_class = FlatteningPandasSerializer
    renderer_classes        = [PandasExcelNoColNoIndexRenderer,
                               PandasCSVNoColNoIndexRenderer,
                               PandasJSONRenderer]

While the following view can only be accessed when logged in

class WellList(generics.ListAPIView):
    queryset = Well.objects.all()
    permission_classes = (MayUploadWriteOrDownloadRead,)
    serializer_class = WellSerializer
    filter_class = WellFilter

The only difference is the serializer class and the View they inherit from so I would say this is something with PandasView, but I cannot guess why. Am I missing something here?

Adding these mixin's restricts the view to logged in users again.

class DownloadFileViewAPI(LoginRequiredMixin, PandasView):

So this is my workaround for now

opened by martinwk 2

Change Json format ?
Hello, I need to construct my json like :

{ "x": [ 5.0, 3.0, 8.0, 5.0] ], "y": [ 7.0, 2.0, 6.0, 4.0] }

I have this, which is exactly what to_json gives: [{"x":5,"y":7},{"x":3,"y":2},{"x":8,"y":6},{"x":5,"y":4}]

Is it possible to change this behavior ? It might be a pandas question.

Thanks for your efforts ! I understand better how REST works now :) Thomas
opened by slamer59 2

Excel file generation Exception Response Data is not a Dataframe.

Basically I am trying to generate XLSX File and stream it for download.

My tests are like this:

response = self.client.get('/grades/export?format=csv')
print response
self.assertEqual(response.status_code, status.HTTP_200_OK)
print '===================================================='
response = self.client.get('/grades/export?format=xlsx')
print response
self.assertEqual(response.status_code, status.HTTP_200_OK)

And my Output is this:

ERROR: test_export_grades_to_excel (functional_tests.tests.GradesExportTestCase)
Traceback (most recent call last):
File "C:\Users\lhernandezm\Documents\Git\alvin\functional_tests\tests.py", line 528, in test_export_grades_to_excel
response = self.client.get('/grades/export?format=xlsx')
File "C:\Users\path\lib\site-packages\rest_framework\test.py", line 162, in get
response = super(APIClient, self).get(path, data=data, **extra)
File "C:\Users\path\lib\site-packages\rest_framework\test.py", line 88, in get
return self.generic('GET', path, **r)
File "C:\Users\path\lib\site-packages\django\test\client.py", line 379, in generic
return self.request(**r)
File "C:\Users\path\lib\site-packages\rest_framework\test.py", line 159, in request
return super(APIClient, self).request(**kwargs)
File "C:\Users\path\lib\site-packages\rest_framework\test.py", line 111, in request
request = super(APIRequestFactory, self).request(**kwargs)
File "C:\Users\path\lib\site-packages\django\test\client.py", line 466, in request
six.reraise(*exc_info)
File "C:\Users\path\lib\site-packages\django\core\handlers\base.py", line 164, in get_response
response = response.render()
File "C:\Users\path\lib\site-packages\django\template\response.py", line 158, in render
self.content = self.rendered_content
File "C:\Users\path\lib\site-packages\rest_framework\response.py", line 71, in rendered_content
ret = renderer.render(self.data, media_type, context)
File "C:\Users\path\lib\site-packages\rest_pandas\renderers.py", line 30, in render
"Response data is a %s, not a DataFrame!" % type(data)
Exception: Response data is a <type 'NoneType'>, not a DataFrame!

CSV Test Succeeds! But XLSX gives me the error above.

My current pip freeze is: colorama==0.3.3 coverage==3.7.1 decorator==4.0.4 diff-match-patch==20121119 Django==1.8.2 django-appconf==1.0.1 django-audit-log==0.8.0 django-autoslug==1.8.0 django-cities-light==3.1.2 django-compressor==1.5 django-cors-headers==1.1.0 django-email-as-username==1.6.7 django-extensions==1.5.5 django-filter==0.10.0 django-guardian==1.3 django-import-export==0.2.8 django-rest-swagger==0.3.4 djangorestframework==3.2.4 djangorestframework-jwt==1.7.2 factory-boy==2.5.2 fake-factory==0.5.2 flake8==2.4.1 ipdb==0.8.1 ipython==4.0.0 ipython-genutils==0.1.0 jdcal==1.0 Markdown==2.6.2 mccabe==0.3.1 numpy==1.10.0 openpyxl==1.8.6 pandas==0.16.2 path.py==8.1.1 pep8==1.5.7 pickleshare==0.5 Pillow==2.8.1 psycopg2==2.6.1 pyflakes==0.8.1 Pygments==2.0.2 PyJWT==1.3.0 python-dateutil==2.4.2 pytz==2015.6 PyYAML==3.11 rest-pandas==0.3.2 selenium==2.47.1 simplegeneric==0.8.1 six==1.9.0 tablib==0.10.0 traitlets==4.0.0 Unidecode==0.4.18 Unipath==1.1 Werkzeug==0.10.4 wheel==0.24.0 XlsxWriter==0.7.6 xlwt==1.0.0

I know this is beta so Docs are for next development phase, please let me know if I am doing something wrong or if it is a bug.

Also I would like to collaborate with Docs.

opened by klahnen 2

Pypi release?

There has not been a release on pypi since 2019, are there any plans to release? There are features on the main branch that I would like to use, but without tracking main.

opened by bradydean 0
docs: Fix a few typos
There are small typos in:

docs/releases/django-rest-pandas-0.3.2.md

docs/releases/django-rest-pandas-0.5.0.md

docs/renderers/csv.md

Fixes:

Should read tabular rather than tablular.

Should read compatibility rather than compatiblity.

Should read browsable rather than browseable.

Semi-automated pull request generated by https://github.com/timgates42/meticulous/blob/master/docs/NOTE.md
opened by timgates42 0
Possible to Access ExcelRenderer from PandasSimpleView?

Apologies, this is probably not the best place to ask this question, but it might lend itself to some useful enhancements to others and I wasn't sure if there was a better forum for asking questions.

I am extending PandasExcelRenderer so that I can 1) use a permanent file (with a filename based on URL parameters passed to renderer_context) stored on my server instead of a temp file and 2) set the sheetname of the file based on the filename of the permanent file. The filename and sheetname are defined in get_pandas_kwargs based on renderer_context and sheet_name is specified in the return value of get_pandas_kwargs.

Is there any way to access this information in PandasSimpleView? Currently I am duplicating the code form get_pandas_kwargs (to determine filename) in PandasSimpleView.get_pandas_filename(), but was hoping for something cleaner and more DRY.

Any guidance or suggestions would be greatly appreciated. Again, apologies if this is the wrong place to post this question, a redirect to a better forum would be appreciated as well if applicable.

opened by geoffrey-eisenbarth 0

how to return a json Response when query_params with 'format=xlsx'

If there isn't a 'form' param in request ,I want return an error, but raised an error.

def list(self, request: Request, *args: Any, **kwargs: Any) -> Response:
    if self.request.query_params.get('form') and self.request.query_params.get('format') == 'xlsx':
        return Response({'error' : 'no form'})

error:

Response data is a dict, not a DataFrame!

opened by wgf4242 2

Releases(v1.1.0)

v1.1.0(Mar 28, 2019)
Django REST Pandas 1.1.0 includes a new filename option (#31), confirmed support for Django 2, and a couple of minor fixes.

New Functionality

Added a get_pandas_filename() view method, for cases where you have users downloading files through the API (#31). For example:

class TimeSeriesView(PandasView): # If a filename is returned, rest_pandas will include the following header: # 'Content-Disposition: attachment; filename="Data Export.xlsx"' def get_pandas_filename(self, request, format): if format in ('xls', 'xlsx'): # Use custom filename and Content-Disposition header return "Data Export" # Extension will be appended automatically else: # Default filename from URL (no Content-Disposition header) return None

Bug Fixes

Don't crash if renderer_context is missing (#34)

PandasBoxplotSerializer: handle non-numeric columns and duplicate rows (abeb57680f4138f8ac5f92591cc83c5c414bff85)

Documentation Improvements

Test Django 2 support (#35), add wheel and LICENSE for PyPI (#33)

Using rest_pandas with an existing view (#32, #36)

Source code(tar.gz)
Source code(zip)
v1.0.0(Sep 13, 2017)
Django REST Pandas 1.0.0 brings a number of API improvements that make it easier to integrate with existing DRF projects.

New Functionality

Support mixing with DRP renderers with regular DRF renderers, including in REST_FRAMEWORK["DEFAULT_RENDERER_CLASSES"] (#28)

Better detection of default index field name (#13, #29)

Include index field(s) in default JSON output (#29). The orient parameter now defaults to a DRP-specific "records-index", which is like "records" but calls reset_index() before rendering.

Bug Fixes

Don't crash if "id" is not a serializer field (#13)

Fix null value handling in PandasScatterSerializer (2636cc4)

Documentation Improvements

Supported URL parameters for JSON output (#26)

DateTimeField serialization tips (#27)

django-pandas integration (#11)

HTML output and integration with wq/chartapp.js (#2)

Source code(tar.gz)
Source code(zip)
v0.5.0(Nov 8, 2016)

Django REST Pandas 0.5.0 introduces a simple PandasHTMLRenderer for use in a browseable visualization API (#2). To enable it by default, you just need to add rest_pandas to your INSTALLED_APPS. You will need a template called rest_pandas.html, or you can install django-mustache to use the provided mustache template (which is optimized for integration with a wq-powered application).

This release also includes updates for pandas 0.19 and drops support for Django REST Framework 2.4 (#23).
Source code(tar.gz)
Source code(zip)
v0.4.1(Aug 25, 2016)
Django REST Pandas 0.4.1 brings a few minor bug fixes:

Fix issues with image renderer (c0c8b5cd5d1a107191c3793447bc956e4da09f19)

Fix file error on Windows (#19)

Update tests for Django 1.10

Code style (#25)

Source code(tar.gz)
Source code(zip)
v0.4.0(Nov 26, 2015)
Django REST Pandas 0.4.0 brings a few new features as well as confirmed compatibility with Django 1.9.

#15 - Ability to override JSON format via URL keyword (via @kevinmickey)

#17 - Unstacked, Scatter, and Boxplot serializers for common charting uses, generalized from wq.db.contrib.chart.

#14 - Better documentation and tests for "simple" use case

Source code(tar.gz)
Source code(zip)
v0.3.2(Apr 20, 2015)
This release is just to verify compatiblity with Django 1.8 and pandas 0.16.0. Older versions should still work, though note that Django 1.6 is no longer being tested against.

The only actual code change is https://github.com/wq/django-rest-pandas/commit/5faa4ec4d32466dac89ef117392faa87428a801f, which switches the JSON renderer from a default of orient="index" to orient="records" ~~to get around a breaking test~~ because it's a more reasonable default. You can restore the old behavior by subclassing PandasJSONRenderer and overriding get_pandas_kwargs(), but:

As is noted in the README, the CSV renderer is the one you probably want to be using anyway. You can use wq/pandas.js to convert CSV to JSON after you've loaded it on the client.

If you really want JSON output, you're probably already using the vanilla DRF JSONRenderer anyway.

Source code(tar.gz)
Source code(zip)
v0.3.1(Mar 19, 2015)

This release of DRP adds a small patch (1a81adcb8be3a65ae6c2c4b9d15a7661df5dd4d9) to ensure that no pagination class will be used for DRP views when running in Django REST Framework 3.
Source code(tar.gz)
Source code(zip)
v0.3.0(Mar 12, 2015)
Django REST pandas 0.3.0 adds support for Django REST Framework 3 (#10) and a better separation between model serialization and DataFrame creation - the latter now happening in a separate serializer (#8).

The new code is mostly backwards compatible (the tests haven't changed) but there are a couple of implementation changes that may affect you if you've customized or extended DRP.

PandasBaseSerializer has been renamed to PandasSerializer and replaces the former PandasSerializer. When using DRP with DRF 3, the new PandasSerializer is a ListSerializer and can only be used as such.

PandasSimpleSerializer has been renamed to SimpleSerializer and no longer has any pandas-related functionality (since that's now handled in a separate step).

For the DRP views, a new PandasMixin class encapsulates the functionality needed to add Pandas capabilities to a serialized result. This is accomplished via the new with_pandas_serializer(cls) method that customizes any existing serializer_class to enable Pandas capabilities as needed. The Pandas serializer can be overridden by specifying pandas_serializer_class.

Source code(tar.gz)
Source code(zip)
v0.2.1(Nov 17, 2014)
Port wq/pandas.js to Python for testing (9880befd2ae4c4a37b5440375b6370a8c36a1529)

Fix name of pandas library in code & docs (#9)

Minor bug fixes (35c767d7d3e12955e04c10c9b897684a5ede4508, 14aeb0144e2ae1f57bcdedec4a8154878658dabb)

Source code(tar.gz)
Source code(zip)
v0.2.0(Aug 12, 2014)
Test on Python 3 (#3) and Django 1.7 RC2

Source code(tar.gz)
Source code(zip)
v0.1.1(Aug 12, 2014)
pass Django REST Framework charset through to pandas encoding (default is "utf-8")

option to replace None values to avoid fragile indexes

Source code(tar.gz)
Source code(zip)
v0.1.0(Apr 17, 2014)

Initial Release
Source code(tar.gz)
Source code(zip)