Hello,

Since we might add some features for fuzzy search, especially when the currency is not found (see https://github.com/scrapinghub/price-parser/issues/28#issuecomment-1274568097), we think it'd be nice to have some evaluation script and dataset that we can use to compare the quality of the extractions (thanks for the idea @lopuhin !).

The purpose of the evaluation is to compare the currency extraction quality of Price.fromstring. If the functionality of this function is extended via PR (e.g. we perform a fuzzy search if the currency is not found), we should be able to see an improvement in the metrics.

I decided not to add it as a test because the evaluation metrics can be interpreted in several ways.

For now, the metrics are simply global accuracy and accuracy per symbol (ground truth) for quality evaluation, and total extraction time and extraction time per sample for performance evaluation. These metrics can be easily extended, as the script is quite simple (evaluation/evaluate_currency_extraction.py).

If in the future we add other evaluation metrics or datasets, this script can be improved and generalized (e.g. adding argparse). For now I decided to keep it simple.

The dataset (dataset_eval.json) can also be easily extended to include more cases.

Here's a sample of the dataset_eval.json:

{
    "string": "14.00 SGD / Each",
    "currency": "SGD"
}

The evaluation script will extract the currency from string using Price.fromstring and compare it with currency. A correct extraction happens when both are equal.

With the current dataset and price_parser versions, this is the output of the evaluation:

./evaluation/evaluate_currency_extraction.py 
-----------------------------------
symbol (target)      acc      support   
-----------------------------------
           None      ‎1.0     13        
              $      ‎1.0     12        
             $U      ‎1.0     1         
             $b      ‎0.0     1         
           .د.ب      ‎0.0     1         
            AED      ‎1.0     1         
           AU $      ‎0.0     1         
            AU$      ‎1.0     1         
            AUD      ‎1.0     1         
             Ar      ‎0.0     1         
            B/.      ‎1.0     1         
             BD      ‎1.0     1         
            BZ$      ‎1.0     1         
             Br      ‎1.0     2         
             Bs      ‎1.0     1         
             C$      ‎1.0     2         
            CA$      ‎1.0     1         
           CAD$      ‎0.0     1         
           CAN$      ‎0.0     1         
            CFA      ‎1.0     1         
           CUC$      ‎1.0     1         
              D      ‎0.0     1         
            DKK      ‎1.0     1         
             DT      ‎1.0     1         
             Db      ‎1.0     1         
              E      ‎0.0     1         
            FBu      ‎1.0     1         
           FCFA      ‎1.0     1         
             FG      ‎1.0     1         
            FRw      ‎0.0     1         
            Fdj      ‎1.0     1         
             Ft      ‎1.0     1         
              G      ‎0.0     1         
            GEL      ‎1.0     1         
             Gs      ‎0.0     1         
             J$      ‎1.0     1         
              K      ‎0.0     1         
             KD      ‎1.0     1         
             KM      ‎1.0     1         
            KMF      ‎1.0     1         
            KSh      ‎0.0     1         
             Kr      ‎0.0     1         
             Kč      ‎1.0     1         
              L      ‎0.0     2         
             LD      ‎1.0     1         
            LEI      ‎1.0     1         
            Lek      ‎1.0     1         
             MK      ‎1.0     1         
           MOP$      ‎1.0     1         
             MT      ‎0.0     1         
            NT$      ‎1.0     1         
            Nfk      ‎1.0     1         
            Nu.      ‎1.0     1         
              P      ‎0.0     1         
              Q      ‎0.0     1         
              R      ‎0.0     1         
             R$      ‎1.0     2         
            RD$      ‎1.0     1         
             RF      ‎0.0     1         
             RM      ‎1.0     1         
             Rf      ‎1.0     1         
             Rp      ‎1.0     1         
             R₣      ‎0.0     1         
            S/.      ‎1.0     1         
            SGD      ‎1.0     1         
            SPL      ‎0.0     1         
             T$      ‎1.0     1         
             TL      ‎1.0     1         
            TSh      ‎1.0     1         
            TT$      ‎1.0     1         
             UM      ‎1.0     1         
            US$      ‎1.0     1         
            USD      ‎1.0     1         
            USh      ‎1.0     1         
             VT      ‎1.0     1         
             Z$      ‎1.0     1         
             ZK      ‎1.0     1         
          franc      ‎0.0     1         
         francs      ‎0.0     1         
             kn      ‎1.0     1         
             kr      ‎1.0     1         
            lei      ‎1.0     1         
           null      ‎0.0     1         
              q      ‎0.0     1         
            yen      ‎0.0     1         
             zl      ‎0.0     1         
             zł      ‎1.0     1         
              ¢      ‎0.0     1         
              £      ‎1.0     2         
              ƒ      ‎1.0     2         
             ЅM      ‎0.0     1         
            Дин      ‎0.0     1         
            ден      ‎0.0     1         
             лв      ‎1.0     1         
           руб.      ‎1.0     1         
              ֏      ‎1.0     1         
              ؋      ‎1.0     1         
           ج.س.      ‎0.0     1         
             د.      ‎0.0     1         
            د.إ      ‎0.0     2         
            د.ت      ‎0.0     1         
            د.ك      ‎0.0     1         
             دج      ‎0.0     1         
            ع.د      ‎0.0     1         
            ل.د      ‎0.0     1         
          نافكا      ‎0.0     1         
              ৳      ‎1.0     1         
              ლ      ‎0.0     1         
             ብር      ‎0.0     1         
            ናቕፋ      ‎0.0     1         
              ៛      ‎1.0     1         
              ₡      ‎1.0     1         
              ₣      ‎1.0     1         
              ₦      ‎1.0     1         
              ₨      ‎1.0     1         
              ₪      ‎1.0     1         
              ₫      ‎1.0     1         
              €      ‎1.0     5         
              ₭      ‎1.0     1         
              ₮      ‎1.0     1         
              ₱      ‎1.0     1         
              ₴      ‎1.0     1         
              ₹      ‎1.0     1         
              ₺      ‎1.0     1         
              ₼      ‎1.0     1         
              ₽      ‎1.0     1         
              ₿      ‎1.0     1         
              元      ‎0.0     1         
              ﷼      ‎1.0     1         

Global accuracy: 0.7117


Total processing time: 3.03 ms
Processing time per sample: 0.018604 ms

I'm open to suggestions

These examples:

Price.fromstring('180', currency_hint='$${product.price.currency}')
Price.fromstring('180', currency_hint='qwerty $')
Price.fromstring('180', currency_hint='qwerty $ asdfg')

All of them return Price(amount=Decimal('180'), currency='$').

From what I understand in the code, this happens in _search_safe_currency:

_search_unsafe_currency('asd $ qwe')
# <re.Match object; match='$'>

Would it be in the scope of the library to match the currency more strictly? This would cover the use-case when we are pretty sure that the currency hint is exact, not fuzzy.

I wanted to contribute to this repo. Tried to install it manually using the python setup.py develop and python setup.py commands but it gave me errors.

PR on top of https://github.com/scrapinghub/price-parser/pull/53, fixes issue https://github.com/scrapinghub/price-parser/issues/52

The Github Actions yml files were heavily inspired in the ones of https://github.com/scrapy/parsel (thanks @Gallaecio for the examples!)

I have not tested the publish.yml though. When we make a version release it should get triggered and upload the version to pypi. I also added the PYPI_TOKEN to gh actions secrets (it was in .travis.yml before).

Currently the price-parser library call only extract prices and store them with it's currency tag. I would like to propose the ability of converting one price to another. If this proposal is approved I would start working on the idea.

I don't think there will be any noticeable difference in the real world, but it's still the right way to write regex. https://docs.python.org/3/howto/regex.html#greedy-versus-non-greedy Greedy

%timeit re.search(r"""(\d[\d\s.,]*)\s*?(?:[^%\d]|$)""", "90 728.00 руб 103 100.00 руб", re.VERBOSE).group(1)
1.97 µs ± 200 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)

Non-greedy

%timeit re.search(r"""(\d[\d\s.,]*)\s*(?:[^%\d]|$)""", "90 728.00 руб 103 100.00 руб", re.VERBOSE).group(1)
1.86 µs ± 103 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)

Sometimes values like 140.000 can mean 140, not 140K; this happens when e.g. website authors put price in semantic markup with 3 digit precision. It looks like the only way to fix it is to allow customizing price extraction; e.g. it can be fixed by having "decimal_separator_hint" argument for price parsing functions, which one can pass for these "bad" websites.

Korean, and Japanese uses each specific money character like '₩', '￥'. But that country also using their text based money character to represent currency. Korean uses '원', Japanese uses '円'. but this parser can't parse that character.

Thank you.

When a test fails, there are no clear indication of the input strings used for the test. This makes identifying which test case is failing very difficult.

Current:

    def test_parsing(example: Example):
        parsed = Price.fromstring(example.price_raw, example.currency_raw)
>       assert parsed == example
E       AssertionError: assert Price(amount=None, currency='EUR') == Example(amount=None, currency=None)

After this PR:

    def test_parsing(example: Example):
        parsed = Price.fromstring(example.price_raw, example.currency_raw)
>       assert parsed == example, f"Failed scenario: price={example.price_raw}, currency_hint={example.currency_raw}"
E       AssertionError: Failed scenario: price=SOMETHING EUROPE SOMETHING, currency_hint=None
E       assert Price(amount=None, currency='EUR') == Example(amount=None, currency=None)

as @lopuhin noticed, we shouldn't limit amount of digits after a decimal separator - when we don't detect decimal separator properly it is just stripped out, causing price to be incorrect.

A follow-up to #7 :)

When type checking with MyPy, an error is thrown when using from price_parser import parse_price, namely:

Module "price_parser" does not explicitly export attribute "parse_price"; implicit reexport disabled

I suspect this is due to the entities not being defined in __all__

First of all, thank you for creating it. it scales very well. overall nice library.

I came across this issue when currency value has E notation. Ex:

from price_parser import Price
Price.fromstring("3.891506499E8")
> Price(amount=Decimal('3.891506499'), currency=None)

from price_parser import Price
print(Price.fromstring('3.350').amount_float)
print(Price.fromstring('3.3500').amount_float)
print(Price.fromstring('3.35').amount_float)
print(Price.fromstring('3.355').amount_float)

Results:

3350.0
3.35
3.35
3355.0

• python 3.8.6
• price-parser 0.3.4

When I type: Price.fromstring("Today I buy 3 coats with 300.000 VND") Your library returned: Price(amount=Decimal('3'), currency='VND') but the result I want is: Price(amount=Decimal('300.000'), currency='VND')

eg:

from price_parser import Price

# ok
price = Price.fromstring("¥36,000")
print(price)
# Price(amount=Decimal('36000'), currency='¥')

# not ok
price = Price.fromstring("36元人民币")
print(price)
# Price(amount=Decimal('36'), currency=None)