As of https://github.com/christofferaakre/japanese-ebook-analysis/pull/10, we now have access to frequency information from several different frequency lists, as well as an overall frequency that takes into account all the frequency lists. We can use this information to show histograms of the frequency distribution, and then we can also show some metrics regarding the frequency. However, I am not sure which metrics best sum up the overall frequency distribution.

I created a class to store all of the information we get from a mecab parse and changed the parsing method to parse() in order to avoid error with random bytes in the beginning of "surface".

Use frequency lists to display the frequency rating of a word (1 being most common, 10000 being 10000th most common) in addition to the number of occurences in the book. Another good idea is to use frequency lists for several different domains (e.g. Slice of Life, Shounen anime, novels, etc)

Credit to mods at r/learnjapanese for suggestion

We can now calculate how many of the words in the book the user knows given a word-list. Currently, the word-list path is hardcoded to be 'word-list.txt' is the root directory, see the analyse_known_words function defined in analysis.py. This information has also been added to the books.html page to display it to the user. An example word-list can be found in data/jlpt-word-list.txt.

This pull request adds support for frequency lists. Frequency lists are put in the frequency-lists folder, and must have the same format as the ones that are currently there. The, when the user uploads an ebook, we find the frequency of every word in the book according to each frequency lists, and we also compute an overall frequency that takes into account all of them - Details about this can be found in the get_overall_frequency function defined in frequency_lists.js

Using the IDs from jmdict entries found in tagainijisho (CSV files) and jmdict, it is possible to create a mapping from words to JLPT levels. This allows us to show the distribution of words by their JLPT level.

I've already done such a mapping for German words in the jmdict, so I could provide a file with mappings or a script to create the mappings with.

Currently, we are only analysing individual words. If we also break the book up into sentences, we get access to some useful metrics like average sentence length etc. Two options seem feasible to me:

1. Reconstruct the sentences by stringing together individual words until we hit a sentence-ending character like 。
2. Maybe mecab has a thing that lets you break text up into sentences rather than words

I think option 1) should be sufficient, as I can't really think of too many edge cases.

Currently, we use furigana4epub to remove furigana from .epub files, but we don't remove furigana from .txt files. I have been unable to find a suitbale library/tool to do this, so I tried to implement something myself: https://github.com/christofferaakre/japanese-ebook-analysis/commit/90707b1da313a5b95a3caf3bc9e2c0402c8399d1 Unfortunately, it doesn't seem to quite work.

Deploy the server somewhere on the web so that the user doesn't need to clone the repository, install dependencies, and then start the server themselves.

Currently, the service looks quite bad, and could look a lot better with some polish on the css/js/etc. The relevant files to look at are:

• templates/books.html
• templates/header.html
• templates/upload_file.html
• static/css/style.css
• static/css/books.css
• static/css/upload_file.css