iAWE is a wonderful dataset for those of us who work on Non-Intrusive Load Monitoring (NILM) algorithms.

Overview


Ax

Description

iAWE is a wonderful dataset for those of us who work on Non-Intrusive Load Monitoring (NILM) algorithms. You can find its main page and description via this link. If you are familiar with NILM-TK API, you probably know that you can work with iAWE hdf5 data file in NILM-TK. However I faced some problems that convinced me to Not use NILM-TK and iAWE hdf5 datafile. Instead, I decided to use the iAWE appliance consumption CSV files and preprocess them myself. So if you have problems with NILM-TK API and iAWE hdf5 data file too, this piece of code may help you to prepare 11 appliance consumption data for your NILM algorithm.

Installation

  • First, download the iAWE dataset using this link (also available on iAWE page!).
  • Download the electricity.tar.gz file.


Ax

  • Download the repo and all its folders.
  • Unzip the electricity.tar.gz and copy all 12 CSV file (plus the labels file into the electricity folder of the downloaded repo.
  • Now everythng is ready for you to start the data preprocessing using the main.py file. But before running the code let me show you what kind of problems we had with the original iAWE hdf5 file.

What problems did we solve?

Well, to be honest NILM-TK documentation is not very clear! If you try to use the hdf5 datafile of the datasets that works with NILM-TK, soon you will admit it. Sometimes you find the the similiar questions on stack overflow but when you try them, they simply don't work due to some updates in NILM-TK (undocumented maybe!?). So, having full control on the data was my main incentive to redo the data preprocessing by my self. You see 12 CSV files in your downloaded files. They belong to:

  • main meter (1)
  • main meter (2)
  • fridge
  • air conditioner (1)
  • air conditioner (2)
  • washing machine
  • laptop
  • iron
  • kitchen outlets
  • television
  • water filter
  • water motor The publisher of iAWE dataset has recommended to ignore the water motor CSV file as it is not accurate (so did we!). Each CSV file consists of timestamp, W, VAR, VA, f, V, PF and A columns. timestamp can be read and converted to read time and date by Python libraries. The publisher of dataset have collected time stamps to reduce the size of final data files which means there is no sampling when the appliances are not consuming power. On the other hand the start time of different appliances measurement is not the same so the length, start and end of most csv files are different. When you plot it in NILM-TK it is fine becuase it reads the timestamps and ignores the NA time steps. However when you want to feed this data into your algorithm it will be a problem which needs data preprocessing. To better understand the problem when using the raw data in iAWE dataset, I've plotted W (active power) of the air conditioner which is CSV file number 4.


AC

As you see, when youplot it in Python the NA timestamp will be plotted as a direct line between last available data and the next available one. It is neither human readable (to some extents!) nor NILM algorithm readable. In fact what your NILM algorithm will be fed with is the series of these values because your algorithm has nothing to do with timestamps! See this is what NILM algorithm sees as the AC power consumption:


AC WO

Now to make it both human readable and NILM algorithm readable, I did as below: (I've commented the code so you can see what is happening in every part of the code)

  • Loaded all CSV files in a dictionary of Dataframes with CSV file orders
  • Measured the lowes and highest timestamp in order to know the length of the measurement period (they have different lengthes!)
  • Created a big dataframe of zeros with from lowest timestamp to the highest one as its index
  • Used the update method on dataframes to transfer the values of dataframes to the big dataframes of zeros (Now all of them have the same length)
  • Putting all dfs into a dictionary of dataframes
  • Casting all the dataframes into the efficient period of sampling (Because now we know which part of sampling is useless)
  • Removing NAN values
  • Dropping unwanted columns
  • Filling NA values with last available value in dataframes
  • Saving all the dataframes as CSV files in the prepared data folder
  • Done!


AC WO

Conclusion

Basically, what we have here after running this code is 11 CSV files of W, VAR, VA, f, V, PF and A for 11 different meters. Prepared CSV file are all of the same length without NAN or NA values which are ready to be fed to any NILM algorithm. Despite the fact that I've done these changes to iAWE dataset, I'm sure the publishers of this dataset have much better solution via NILM-TK to have such an output. However due to lack of documentation or changes in their code I prefered to do this data preprocessing myself. Hope you enjoy it!

You might also like...
This is the code repository for 40 Algorithms Every Programmer Should Know , published by Packt.
This is the code repository for 40 Algorithms Every Programmer Should Know , published by Packt.

40 Algorithms Every Programmer Should Know, published by Packt

Solving a card game with three search algorithms: BFS, IDS, and A*

Search Algorithms Overview In this project, we want to solve a card game with three search algorithms. In this card game, we have to sort our cards by

🧬 Performant Evolutionary Algorithms For Python with Ray support

🧬 Performant Evolutionary Algorithms For Python with Ray support

Nature-inspired algorithms are a very popular tool for solving optimization problems.
Nature-inspired algorithms are a very popular tool for solving optimization problems.

Nature-inspired algorithms are a very popular tool for solving optimization problems. Numerous variants of nature-inspired algorithms have been develo

All algorithms implemented in Python for education

The Algorithms - Python All algorithms implemented in Python - for education Implementations are for learning purposes only. As they may be less effic

Implementation of Apriori algorithms via Python

Installing run bellow command for installing all packages pip install -r requirements.txt Data Put csv data under this directory "infrastructure/data

A simple python application to visualize sorting algorithms.
A simple python application to visualize sorting algorithms.

Visualize sorting algorithms A simple python application to visualize sorting algorithms. Sort Algorithms Name Function Name O( ) Bubble Sort bubble_s

Programming Foundations Algorithms With Python
Programming Foundations Algorithms With Python

Programming-Foundations-Algorithms Algorithms purpose to solve a specific proplem with a sequential sets of steps for instance : if you need to add di

Planning Algorithms in AI and Robotics. MSc course at Skoltech Data Science program

Planning Algorithms in AI and Robotics course T2 2021-22 The Planning Algorithms in AI and Robotics course at Skoltech, MS in Data Science, during T2,

Owner
Mozaffar Etezadifar
NILM and RL researcher @ Polytechnique Montreal
Mozaffar Etezadifar
Genetic algorithms are heuristic search algorithms inspired by the process that supports the evolution of life.

Genetic algorithms are heuristic search algorithms inspired by the process that supports the evolution of life. The algorithm is designed to replicate the natural selection process to carry generation, i.e. survival of the fittest of beings.

Mahdi Hassanzadeh 4 Dec 24, 2022
Using Bayesian, KNN, Logistic Regression to classify spam and non-spam.

Make Sure the dataset file "spamData.mat" is in the folder <spam\src> Environment: Python --version >= 3.7 Third Party: numpy, matplotlib, math, scipy

null 0 Dec 26, 2021
Minimal examples of data structures and algorithms in Python

Pythonic Data Structures and Algorithms Minimal and clean example implementations of data structures and algorithms in Python 3. Contributing Thanks f

Keon 22k Jan 9, 2023
Repository for data structure and algorithms in Python for coding interviews

Python Data Structures and Algorithms This repository contains questions requiring implementation of data structures and algorithms concepts. It is us

Prabhu Pant 1.9k Jan 1, 2023
All Algorithms implemented in Python

The Algorithms - Python All algorithms implemented in Python (for education) These implementations are for learning purposes only. Therefore they may

The Algorithms 150.6k Jan 3, 2023
:computer: Data Structures and Algorithms in Python

Algorithms in Python Implementations of a few algorithms and datastructures for fun and profit! Completed Karatsuba Multiplication Basic Sorting Rabin

Prakhar Srivastav 2.9k Jan 1, 2023
Algorithms implemented in Python

Python Algorithms Library Laurent Luce Description The purpose of this library is to help you with common algorithms like: A* path finding. String Mat

Laurent Luce 264 Dec 6, 2022
Algorithms and data structures for educational, demonstrational and experimental purposes.

Algorithms and Data Structures (ands) Introduction This project was created for personal use mostly while studying for an exam (starting in the month

null 50 Dec 6, 2022
A command line tool for memorizing algorithms in Python by typing them.

Algo Drills A command line tool for memorizing algorithms in Python by typing them. In alpha and things will change. How it works Type out an algorith

Travis Jungroth 43 Dec 2, 2022
Python sample codes for robotics algorithms.

PythonRobotics Python codes for robotics algorithm. Table of Contents What is this? Requirements Documentation How to use Localization Extended Kalman

Atsushi Sakai 17.2k Jan 1, 2023