Fava - web interface for Beancount

Beancount itself is a double-book accounting tool, and does not know of the concept of "budgets".

But the more I use fava I wish there would be a way to display information if a certain account is over budget. I do practice setting budgets for various "accounts" myself to enforce certain spending behavior, and it would be nice to integrate this into the system.

In the "budgeting"-branch I drafted a first version of how this might work and look:

• There is a new file which contains the budget-rules. It is a simple plain-text-file and has content like this:

  Expenses:Food:Groceries
      2013-W52       100.00 USD
      2016-W01       120.00 USD
      2016-W02       110.00 USD
      2016-W06       150.00 USD
  
  Expenses:Food:Alcohol
      2016-W01        20.00 USD
      2016-W02        10.00 USD
      2016-W06         5.00 USD
  
  ...
  

  The format is very simple: First, the account name (Expenses:Food:Groceries). Then, lines with the week of the year where this budget applies to (and starts) (2016-W01), and finally the amount and currency (20.00 USD). I will call these lines "Budget entries".

• These budget entries represent the budget for the specified week of the year, and all weeks thereafter, until a new rule is set for a week after that. So you can set a budget of 10.00 USD at the very first week of 2015 (2015-W01), and this budget applies to all weeks until a new entry starting in week 5 of that same year (2015-W05) appears and sets the new budget to 20.00 USD.

• A budget for a specific account, currency, start-date and end-date is then calculated by stepping through each day in that period, and adding the budget for the week of that day (divided by 7) to the total sum, until the end-date is reached.

I included a test-file in fava/util/test-budget.budget, which contains the rules above for Groceries and Alcohol (accounts in the file generated by bean-example).

You can test it by starting fava with a bean-example-generated file, and navigating to http://localhost:5000/account/Expenses:Food/monthly_changes/ (for example). It should look like this:

Things to do:

• Discussing if this is a good idea or not
• Discussing if it is to be included in fava or not
• Discussing if the core functionality of this should be factored out into it's own package or should stay within fava
• Writing tests for the "custom language parser"
• The budget is currently "generated" and displayed in the account.html-template, but should ultimately be either part of the API or in a second API (BudgetAPI?).

What do you think @yagebu @corani @redstreet @jbms @xentac @blais

Implementation of #421

• [x] Basic UI
• [x] Mobile UI
• [x] Helpers for UI (autocomplete, formatting, adding new lines, etc.)
• [x] Data for autocomplete
• [x] Writing the resulting transaction to the beancount file
• [x] ~~~Translations (ru, de)~~~ (shouldn't have to be part of feature-PR)
• [x] Tests
• [ ] Documentation

This is a very big feature to implement, so any help is welcome!

As discussed in #436 and #503, it would be great to have intelligent suggestions for account names when doing imports. (I hope you don't mind that I am bringing this old issue up again?).

Implementation originally was originally planned in #503:
intelligent account suggestions for fava's import process were discussed by @aumayr in #503, but the merge request was closed and the feature thus postponed.

@aumayr commented on Feb 20, 2017 in #503: Updated TODO-List:

• [ ] Support for bean-file
• [ ] "Intelligent" way to suggest accounts, etc. (like discussed in #436)
• [x] Support for more types of entries (balance and note seem important)
• [ ] Tests

@aumayr commented on Apr 6, 2017 in #503: Only four points remain, which can be implemented after the PR is merged:

• Figure out (and integrate) the duplicate-entry-detection-mechanism by bean-extract. For me, it never recognises an entry as duplicate. If duplicates are recognised, mark them in the Ingest-UI as "duplicate".
• Add bean-file functionality.
• The Ingest-UI is slow for >50 entries.
• Add example code for how to implement an importer and configure Fava to use it. This should also contain a blueprint for a best-practice importer.

Links to related implementations:
Some more infos and links to existing implementations in other tools (that can maybe be reused?) can be found in #436:

@johannesjh commented on Jan 16, 2017 in #436: Are you aware of existing implementations? There even exists one tool written in Python.

• quentinsf/icsv2ledger (written in Python)
• cantino/reckon (written in Ruby)
• tomszilagyi/banks2ledger (written in Clojure)

in the meantime, I found these additional tools, also see this list: http://plaintextaccounting.org/#data-importconversion

• jbms/beancount-import (written in Python)
• PaNaVTEC/csv2beancount (written in Closure)

Technical design:
Does the current implementation of the import process save enough data so that a machine learning algorithm can learn from previous imports? Conceptually, this data would consist of mappings from original source data (e.g., a line from CSV file) to output (corresponding beancount transactions).

What would be a suitable place to store this data? I found the recommendation that importers should generate a __source__ metadata field for each transaction. But the data could also be stored in a separate file.

Interaction design, to be discussed:
Given a user started an import process by clicking "extract" next to a file that has been identified (or in the future: uploaded), when the user then looks at the extracted data... Where in the UI should the intelligent account suggestions be displayed? Some ideas:

• fava could automatically suggest postings that balance all unbalanced transactions generated by the importer. (unless importers are expected to only generate balanced transactions?)
• the dropdown list for manually selecting an account could be sorted by relevance based on the smart suggestions.

[email protected] (via Mailing list):

When displaying the balance sheet, for Assets and Liabilities, I typically always compare my current balance to what the bank/credit-card/financial institution currently shows. Once they compare correctly, I put in a 'balance' directive for that account.

It would be immensely helpful if account names were optionally color coded in the balance sheet display to help with this process, like so:

Show account in:

• green: if the latest posting is a balance check that passed (i.e., known-good)
• red: if the latest posting is a balance check that failed (i.e., known-bad)
• yellow: if the latest posting is not a balance check (i.e., unknown)

@blais (via Mailing list):

That's a cool idea, and I would add maybe a "gray" if the account hasn't been updated in a while (as compared to the last available date in the file).

I completely destroyed #393 when trying to rebase it, so I created a fresh branch to continue working. There are some important discussion comments there.

Related issue with discussion: #386

To try out the plugin add this to your beancount file:

plugin "fava.plugins.statements_from_metadata"

2016-10-30 * "" "Test"
  statement: "2016-10-30 test.pdf"
  Expenses:Other                                             10.00 EUR
  Assets:Cash
    statement: "2016-10-30 test 2.pdf"

Things to do related to the plugin:

• [x] Link the transaction and the generated Document entry
• [x] Support multiple statements per transaction (currently metadata is a dict), like statements, statements-2, statements-3, ...
• [x] Add description of the feature to help pages, including which paths are searched
• [x] Testcases

General things to do:

• [x] Support "relative" paths suggested by @corani (https://github.com/beancount/fava/issues/386#issuecomment-256267212) when downloading files (in the Journal)
• [x] Drag'n'drop: Add support for multiple document_root
• [x] Drag'n'drop: Add support for dropping on a Journal transaction or posting (upload; add the corresponding metadata entry)

There is a common use-case for me and for business: Attach a receipt/invoice to a transaction.

How I do it: With metadata, like this

2016-10-20 * "Fancy Restaurant" "Business Dinner with Bill Gates" #project-a
    statement: "invoices/Expenses/BusinessDinners/2016-10-20 Dinner Receipt Gates.pdf"
    Expenses:BusinessDinners                      150.00 USD
    Assets:Cash

This differs from the current document upload, and the document feature in beancount, as it does not list this document explicitly in the Journal (different folder than the documents-folder), but adds the PDF as metadata to the transaction and can be viewed directly with the transaction in the Journal.

It would be a very handy feature if we extend the documents drag'n'drop to the Journal, where, when dropped on a transaction, the document is uploaded and the metadata entry is added. There would have to be a separate fava-option for the path to the "invoices"-folder, as it differs from the "documents"-folder (so that the receipts/invoices do not show up as documents in the Journal).

@yagebu Do you think it is possible to add the metadata-entry to the Beancount-file when dragging-dropping the document on a transaction in the Journal?

Hi, I was testing fava and beancount out (along with hledger and hledger-web against my current gnucash system), and while I love the data display with fava, the transaction entry is really only suitable for computer use (where I'd presumably prefer to use vim/emacs anyway).

hledger-web has a nice form-based transaction input which eases transaction input especially from mobile devices. Please consider something similar. For mobile transaction entry I think the user would just want to enter the description, amount(s), and select accounts from a list (or have search results like in hledger-web).

fava has seen a lot of changes since the last release, and I would like to bake a new release soon to make these changes accessible for people who do not run on master.

@yagebu There is a new "Project" (the new Github feature) where I added all the open issues to discuss which ones to include and track the progress. Please move them around as you see fit, especially those who will probably not make it into this release.

Do you have any other thoughts on this release? 0.4.0 or 1.0.0?

One of the distinctive features of Beancount is the ability to write SQL like queries to create custom reports. For example, today I needed to get an overview of my expenses per month over the last year:

SELECT
  MONTH(date) AS month,
  SUM(COST(position)) AS balance
WHERE
   account ~ 'Expenses:' AND
   currency = 'CNY' AND
   YEAR(date) = 2015
GROUP BY 1
ORDER BY 1;

Instead of creating these queries from scratch every time (or having to save them elsewhere), it would be useful if it were possible to save them, right from the Custom Query page.

UI wise, I imagine a select box, Save and Delete button next to the existing Submit Query button. The select box would contain a list of saved queries and a Save as new Query option.

This PR adds initial ingest-functionality described in #436 to Fava:

• Upload a (CSV-) file
• Parse the file and display multiple transaction forms
• Save the transactions to the beancount file

This PR includes a sample bank CSV importer, which works with the following files:

• CSV: testfile.csv.zip (This is similar to what one of my banks (sorry for the German language in the file) let's me download on their website)
• Beancount: example.beancount.zip

You can enable the feature by declaring "ingesters" as a Fava option like in the example beancount file:

2016-04-14 custom "fava-option" "ingesters" "fava.ingest.sample_bank_csv_ingester"

This is what the UI looks like (with the files from above, 2 transactions):

Feedback (and help) is more than welcome!

[The first commit updates our code to use the new shiny conversion utilities in beancount.core.convert. @blais, could we get a new release of Beancount so that we can depend on this?, edit: this has been cherry-picked into master]

More importantly, this allows us to offer a toggle to show market values instead of cost values (ref #464).

This should also allow us to convert everything to a single currency (later, ref #242).

Fava version 1.23.1 While Fava is trying to find custom Extensions and import them if there is a problem (an exception) in an extension Fava fails to report the error. It swallow the details. Its not clear what is the cause of failure.

The only way to know what is causing the extension to fail is to put a break point here and check the ImportError.

Its very helpful if Fava can report the exact error details of an extension in case it happens during import.

Currently extensions that include javascript in script tags in the html have an issue where the scripts are only run some of the time (script doesn't run on a partial load but does run on a full refresh). This fixes makes the scripts run on both partial and full loads.

What do you think about implementing an option to fill single-bar charts with two colors based on their value?
In this case, I have manually filled in green the bars above 0, and in red the negative ones:

By default they are all blue:

<rect fill="rgb(126, 174, 253)" width="29.51824817518248" x="0" y="135.8056037554339" height="61.58523968170877"></rect>

The coloring method is related to #843

I found some issues with how Fava uses prices provided with price directive depending on the view. Let's consider the following journal with pricing provided for 4 consecutive days.

option "operating_currency" "USD"

2022-01-01 open Assets:Commodity COM
2022-01-01 open Assets:Cash USD
2022-01-01 open Income:All USD

2022-01-01 * "Transfer"
  Assets:Cash                                           100 USD
  Income:All                                           -100 USD

2022-01-01 * "Buy"
  Assets:Cash                                          -100 USD
  Assets:Commodity                                        1 COM {{100 USD}} 
  
2022-01-01 price COM                                    101 USD
2022-01-02 price COM                                    102 USD
2022-01-03 price COM                                    103 USD
2022-01-04 price COM                                    104 USD

In the balance sheet view (at market value) prices seem to be offset 1 day forward. In that case price dated 2022-01-04 (104 USD) is taken to calculate asset values dated 2022-01-05. Also, there is no data point for the purchase date at all.

In the account view (balances daily) the offset seems to be in the opposite direction, 1 day backward. Purchase day balance (2022-01-01) is calculated with pricing from the following day (2022-01-02). Pricing from the last available day is used twice (for 2022-01-03 and 2022-01-04).

Is it a fava bug, or is there something I am missing in terms of how pricing data is being used?

I am using beancount 2.3.5 and fava 1.23.