The lexer updates a set with all the currencies seen during parsing. This set is then assigned to the automatic 'currencies' option. Despite what is reported in a comment, this information is not used by beancount.core.getters.get_commodity_map(). Besides tests, the only user is in the beancount.plugins.check_commodity plugin. The plugin can however be rewritten in a simpler form relaying exclusively on the information returned by get_commodity_map().

It may be argued that the set in the options dictionary is more efficient than recomputing the set with get_commodity_map(), however, the only user of the options calls this function anyway, so keeping the set around is actually more work than not doing it.

smart_importer has an apply hook mechanism which allows you to apply functions to all entries during import.

I found this very useful in another project I was working on (a rules-based importer) and I now also use it to apply a function I wrote that contains business logic for the organization.

Since there's nothing smart_importer-specific about the hook mechanism, could this be moved into beancount itself?

Copying @tarioch from smart_importer.

@blais this is the port of the lexer to RE/flex. I updated the classical build (ie not the Bazel one) to compile the code and I haven't tried to integrate it into the Bazel build. Therefore the CI will fail miserably.

I tested the code against v2 and it works just the same as the Flex one. Let me know if you have any question. You need RE/flex version 2.1.5 or later.

I have a couple more patches on top that rework the input handling. I was hoping to get support for encodings other than UTF-8, but that is implemented in RE/flex only for FILE* type input sources and not for std::istream ones (see Genivia/RE-flex#94).

This contains also the commits in #580: #580 should be merged into v2 and this into master.

Original report by Alen Šiljak (Bitbucket: Alen Siljak, GitHub: MisterY).

Since ledger supports ISO date format, perhaps changing the default date format to ISO would be convenient? Help push the standardization. :slight_smile:

Ash a workaround, is there a way to customize the date format for bean-report ledgeroutput?

Original report by Jeremy Maitin-Shepard (Bitbucket: jbms, GitHub: jbms).

Python 3 includes a much faster C implementation of the decimal library, but this is not always installed by default. For example, on Arch Linux it is only available if the mpdecimal package is installed, and I didn't happen to have that package installed on my computer.

This dramatically improves the performance of Beancount, and therefore it might be worth suggesting in the documentation to be sure it is available, or checking automatically in the code and issuing a warning if it is not found. It can be tested by trying to:

import _decimal

This check might have to be updated eventually in a later version of Python, but since it would just be a warning it is not as big a deal if it breaks.

As specified in docstring located here: beancount.core.amount.Amount.new

def __new__(cls, number, currency):
    """Constructor from a number and currency.

    Args:
      number: A string or Decimal instance. Will get converted automatically.
      currency: A string, the currency symbol to use.
    """

closes #532

Original report by Jason Chu (Bitbucket: xentac, GitHub: xentac).

If my price source looks like

1841-01-01 commodity CAD
  export: "CASH"
  name: "CA Dollar"
  price: "USD:yahoo/^CAD=X" 

or

1841-01-01 commodity CAD
  export: "CASH"
  name: "CA Dollar"
  price: "USD:oanda/^USD_CAD"

bean-price gives me this output:

Traceback (most recent call last):
  File "/usr/lib/python3.6/shelve.py", line 111, in __getitem__
    value = self.cache[key]
KeyError: '36af117ec1697ec0980fa6e2b9309fbf'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/bin/bean-price", line 4, in <module>
    from beancount.prices.price import main; main()
  File "/usr/lib/python3.6/site-packages/beancount/prices/price.py", line 349, in main
    price_entries = sorted(price_entries, key=lambda e: e.currency)
  File "/usr/lib/python3.6/concurrent/futures/_base.py", line 586, in result_iterator
    yield fs.pop().result()
  File "/usr/lib/python3.6/concurrent/futures/_base.py", line 425, in result
    return self.__get_result()
  File "/usr/lib/python3.6/concurrent/futures/_base.py", line 384, in __get_result
    raise self._exception
  File "/usr/lib/python3.6/concurrent/futures/thread.py", line 56, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/lib/python3.6/site-packages/beancount/prices/price.py", line 139, in fetch_price
    srcprice = fetch_cached_price(source, psource.symbol, dprice.date)
  File "/usr/lib/python3.6/site-packages/beancount/prices/price.py", line 84, in fetch_cached_price
    timestamp_created, result = _CACHE[key]
  File "/usr/lib/python3.6/shelve.py", line 114, in __getitem__
    value = Unpickler(f).load()
AttributeError: Can't get attribute 'tzfile' on <module 'dateutil.zoneinfo' from '/usr/lib/python3.6/site-packages/dateutil/zoneinfo/__init__.py'>

If I call bean-price with --no-cache, everything works awesome. From what I can gather of the error, it looks like pickling some timezones isn't supported?

Original report by Martin Michlmayr (Bitbucket: tbm13, GitHub: tbm).

If a price pair MR/USD exists as well as USD/EUR, why can't beancount convert from MR to EUR directly? i.e. why are prices not transitive? (They are in ledger)

Example:

beancount> SELECT CONVERT(position, "USD") WHERE account ~ 'Test' AND currency ~ 'MR'
convert_pos
-----------
 7.5000 USD
-7.5000 USD
beancount> SELECT CONVERT(position, "EUR") WHERE account ~ 'Test' AND currency ~ 'USD'
       convert_position_c_        
----------------------------------
 8.130081300813008130081300813 EUR
-8.130081300813008130081300813 EUR
beancount> SELECT CONVERT(position, "EUR") WHERE account ~ 'Test' AND currency ~ 'MR'
convert_
--------
 1000 MR
-1000 MR
beancount> SELECT CONVERT(CONVERT(position, "USD"), "EUR") WHERE account ~ 'Test' AND currency ~ 'MR'
  convert_convert_position_c__c_  
----------------------------------
 6.097560975609756097560975610 EUR
-6.097560975609756097560975610 EUR
beancount> 

Test case:

2014-01-01 open Assets:Test

2018-03-20 * "Test"
  Assets:Test                1000 MR
  Assets:Test

2018-03-20 * "Test"
  Assets:Test                10 USD
  Assets:Test

2018-03-25 price EUR 1.23 USD
2018-03-25 price MR 0.0075 USD

looks like it's blocked by #536, workflow shoule work fine if #536 is fixed, see https://test.pypi.org/project/beancount/#files

You need to add a github actions secrets named PYPI_TOKEN

This PR also add a workflow to test if it compile fine on windows https://github.com/beancount/beancount/pull/537/files#diff-6d1ed6f2f15040f1022bb9655a581b0d

close #154

Original report by Martin Michlmayr (Bitbucket: tbm13, GitHub: tbm).

I’m wondering whether it would be possible to make metadata (both on transactions and postings) into an OrderedDict.

I have written an importer which basically does something like:

            meta = data.new_metadata(f.name, index)
            meta['donor_name'] = get_name(row)
            meta['donor_email'] = row['Email']

It makes sense to have name first, email second, but currently beancounts sorts the keys and email comes out first.

Can you think of any disadvantages of using an OrderedDict? If not, maybe I can try to see if I can contribute a patch.

‌

Original report by Martin Blais (Bitbucket: blais, GitHub: blais).

Try installing this with MingW32 instead of Cygwin. It's a bit crazy, but some people want to run it that way.

I am using the forecast plugin to set down some income, and when I try adding pad/balance at the end of the year, it's not generating the correct amount to match the given balance.

versions: beancount 2.3.5 python 3.10.4

Here is the sample beancount file for testing:

plugin "beancount.plugins.forecast"

2022-01-01 open Assets:Income
2022-01-01 open Income:Fulltime
2022-01-01 open Equity:Opening-Balances

2022-01-01 # "Income [MONTHLY UNTIL 2022-03-01]"
    Income:Fulltime               -1000 AUD
    Assets:Income                  1000 AUD

2022-12-24 pad Assets:Income Equity:Opening-Balances
2022-12-25 balance Assets:Income   3000 AUD

results for bean-check

Balance failed for 'Assets:Income': expected 3000 AUD != accumulated 5000 AUD (2000 too much)

   2022-12-25 balance Assets:Income                                   3000 AUD

the generated pad transaction:

** Balances before transaction --------------------------------

  Assets:Income                                                     3000 AUD

  Equity:Opening-Balances                                                   


** Unbooked Transaction --------------------------------

2022-12-24 P "(Padding inserted for Balance of 3000 AUD for difference 2000 AUD)"
  Assets:Income             2000 AUD
  Equity:Opening-Balances  -2000 AUD


** Transaction --------------------------------

2022-12-24 P "(Padding inserted for Balance of 3000 AUD for difference 2000 AUD)"
  Assets:Income             2000 AUD
  Equity:Opening-Balances  -2000 AUD


** Residual and Tolerances --------------------------------



** Balances after transaction --------------------------------

* Assets:Income                                                     5000 AUD

* Equity:Opening-Balances                                          -2000 AUD

I expect a unused pad entry in above case, which it should report:

Unused Pad entry
  2022-12-24 pad Assets:Income Equity:Opening-Balances

I recently upgraded an installation from beancount 2.0rc1 on 2.3.5 and several of my files started getting syntax error, unexpected ASTERISK from bean-check test.bean. (With python 3.9 on debian.)

Line 2 gets the error, it is just an asterisk on a line by itself.

Line 6 does NOT get the error, it is an asterisk followed by a space.

test.bean:

* this is a comment
*
* The line above is part of a commented-series of lines, it does not have a
* trailing space. The line below has a trailing space.
* 

2022-01-01 open Assets:Cash USD
2022-01-01 open Expenses:Misc USD

2022-01-02 * "blah"
    Assets:Cash   -1.00 USD
    Expenses:Misc  1.00 USD

The workaround appears to be to replace a lone asterisk on a line with asterisk+space (:%s/^\*$/* /) -- or delete these lines.

I created a python 3.9 venv to test, and it appears that this change occurred between 2.3.2 and 2.3.3.

csv.Importer() calls normalize_config() in file_date() and extract(). After skipping the requested number of lines, normalize_config() calls csv.Sniffer().has_header():

https://github.com/beancount/beancount/blob/79da2f90d7c8d969b59359ede66c2b1cced8cffd/beancount/ingest/importers/csv.py#L397

I have a CSV file for which has_header() returns True when it should return False; here's the entire file:

01/05/2022,E-TRANSFER ***b3y   ,,23.00,25.00

beancount can't do anything about the fact that has_header() is returning the wrong result. But! But it can, in this case, avoid calling has_header() in the first place.

Here's the description of normalize_config():

Using the header line, convert the configuration field name lookups to int indexes.

For this CSV file, the config field name lookups are already int indexes when normalize_config() is called:

           config = {
                csv.Col.DATE: 0,
                csv.Col.PAYEE: 1,
                csv.Col.AMOUNT_DEBIT: 2,
                csv.Col.AMOUNT_CREDIT: 3,
                csv.Col.BALANCE: 4,
            }

When the field name lookups are already all int indexes normalize_config() should, I think, bail early.

(When has_header() returns False normalize_config() asserts that all indexes are ints. Basically, it should perform that check before sniffing the file. If all indexes are ints normalize_config() can return config, False and avoid has_header() entirely.)

Hello,

I wanted to give a try running beancount 2.3.5 with latest Pypy version and get the following error at startup

└─[$] <> python -V
Python 3.9.12 (05fbe3aa5b0845e6c37239768aa455451aa5faba, Mar 29 2022, 08:15:34)
[PyPy 7.3.9 with GCC 10.2.1 20210130 (Red Hat 10.2.1-11)]

└─[$] <> bean-check bean_files/dge-header.bean      
Traceback (most recent call last):
  File "/home/ang-st/.pyenv/versions/pypy3.9-7.3.9/bin/bean-check", line 33, in <module>
    sys.exit(load_entry_point('beancount==2.3.5', 'console_scripts', 'bean-check')())
  File "/home/ang-st/.pyenv/versions/pypy3.9-7.3.9/bin/bean-check", line 25, in importlib_load_entry_point
    return next(matches).load()
  File "/home/ang-st/.pyenv/versions/pypy3.9-7.3.9/lib/pypy3.9/importlib/metadata.py", line 86, in load
    module = import_module(match.group('module'))
  File "/home/ang-st/.pyenv/versions/pypy3.9-7.3.9/lib/pypy3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<builtin>/frozen importlib._bootstrap_external", line 863, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/home/ang-st/.pyenv/versions/pypy3.9-7.3.9/lib/pypy3.9/site-packages/beancount/scripts/check.py", line 11, in <module>
    from beancount import loader
  File "<frozen importlib._bootstrap>", line 1058, in _handle_fromlist
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/home/ang-st/.pyenv/versions/pypy3.9-7.3.9/lib/pypy3.9/site-packages/beancount/loader.py", line 26, in <module>
    from beancount.parser import parser
  File "/home/ang-st/.pyenv/versions/pypy3.9-7.3.9/lib/pypy3.9/site-packages/beancount/parser/parser.py", line 119, in <module>
    from beancount.parser import _parser
  File "<frozen importlib._bootstrap>", line 1058, in _handle_fromlist
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
ImportError: /home/ang-st/.pyenv/versions/pypy3.9-7.3.9/lib/pypy3.9/site-packages/beancount/parser/_parser.pypy39-pp73-x86_64-linux-gnu.so: undefined symbol: PyExceptionClass_Name

I tried with an earlier version and I still get the same error. Any chance I could make beancount running on Pypy ?

Thank you

Would be helpful to add a bazel module to https://bazel.build/build/bzlmod via https://github.com/bazelbuild/bazel-central-registry. So that beancount3 can be experimented on as bazel module. Would need a tagged v3 release and some configuration setup in the bazel-central-registry.

I'm running the following beancount file

option "operating_currency" "USD"
option "operating_currency" "GBP"

plugin "beancount.plugins.implicit_prices"

2020-01-01 open Expenses:Things
2020-01-01 open Assets:Current
2020-01-01 open Equity:Conversions:Current

2020-06-01 * "Bought a widget online"
  Expenses:Things   50.00 USD @@ 100.00 GBP
  Assets:Current                -100.00 GBP

2020-07-01 * "Bought a widget online"
  Expenses:Things   50.00 USD @@ 110.00 GBP
  Assets:Current                -110.00 GBP

2020-08-01 * "Bought a widget online (in GBP)"
  Expenses:Things                 90.00 GBP
  Assets:Current                 -90.00 GBP

Below is the generated balance sheet for that file, with a new account created -> Equity:Conversions:Current

However I don't see the same set of Equity accounts under the Trial Balance. I even tried to be explicit and write an open directive. The missing conversions account is frustrating because I can't seem to make the TB balance