Hi, thanks for your great works!

After reading your paper, I found the baseline of this dataset on all attribtues has achieved 98.34 on F1. Does it means that this dataset is too simple as a benchmark in attribute value extraction task?

Hi I downloaded All_Amazon_Meta.json and when I run clean_amazon_product_metadata_main.sh, I get the error below. Not sure what I'm doing wrong.

Thank you

Traceback (most recent call last): File "apache_beam/runners/common.py", line 1198, in apache_beam.runners.common.DoFnRunner.process File "apache_beam/runners/common.py", line 537, in apache_beam.runners.common.SimpleInvoker.invoke_process File "/Users/[email protected]/opt/anaconda3/lib/python3.8/site-packages/apache_beam/transforms/core.py", line 1635, in wrapper = lambda x, *args, **kwargs: [fn(x, *args, **kwargs)] File "/Users/[email protected]/opt/anaconda3/lib/python3.8/json/init.py", line 357, in loads return _default_decoder.decode(s) File "/Users/[email protected]/opt/anaconda3/lib/python3.8/json/decoder.py", line 337, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/Users/[email protected]/opt/anaconda3/lib/python3.8/json/decoder.py", line 355, in raw_decode raise JSONDecodeError("Expecting value", s, err.value) from None json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Add a README.md containing descriptions of the dataset and the instructions to create the full version of the dataset by using the Amazon metadataset.

Hi thanks for the repo / data / paper, great work! I am creating this Issue to understand how exactly evaluation is done, since I am using autoregressive formulation of attribute extraction, the attribute extraction is done through free-form text generation, and no attribute type is provided as input.

1. For positive samples (product paragraphs contain at least one attribute value), for the following example ("target_attribute_vals" contain annotated [attribute value](attribute type) that appears in "text", while "predicted_attribute_vals" contain prediction from a model)

{
    "text": "Nymph Womens's Chffion Polka Dot Maxi Halter Dress Extra Long",
    "target_attribute_vals": "Nymph Womens's Chffion [Polka Dot](Pattern) [Maxi](Length) [Halter](Neckline) Dress [Extra Long](Length)",
    "predicted_attribute_vals": "Nymph Womens's Chffion [Polka Dot](Pattern) [Maxi](Pattern) [Halter](Neckline) [Dress](Type) Extra Long"
}

The flattened tuples of target and prediction attribute values would be:

{
    "target_tuples": [("Pattern", "Polka Dot"), ("Length", "Maxi"), ("Neckline", "Halter"), ("Length", "Extra Long")], 
    "predicted_tuples": [("Pattern", "Polka Dot"), ("Pattern", "Maxi"), ("Neckline", "Halter"), ("Type", "Dress")]
}

• Then following Section 5.3 of the paper, No value (VN), Correct values (VC), Wrong values (VW) for the above would be

No value (VN): 2 # "Length", "Maxi") and ("Length", "Extra Long") missing
Correct values (VC): 2 # ("Pattern", "Polka Dot") and ("Neckline", "Halter") correct
Wrong values (VW): 1 # ("Pattern", "Maxi") is not matching ("Pattern", "Polka Dot")

The above value counts sum up to 5 attribute value pairs, but the "target_tuples" only had 4 attribute value pairs. Is this expected?

2. For negative samples (Here I used "target_attributes_as_in_file" instead of "target_attribute_vals" format as in earlier positive examples, and I also added category as shown in the file)

[
{
    "text": "Taylor Dresses Women's High Low Lace Shirt Dress", 
    "target_attributes_as_in_file": [{'key': 'Pattern', 'evidences': []}], 
    "predicted_attribute_vals": "[Taylor](Brand) [Dresses](Type) Women's [High](Size) [Low Lace](Type) Shirt Dress", 
    "category": "Dresses"
}, 
{
    "text": "Taylor Dresses Women's High Low Lace Shirt Dress Nice", 
    "target_attributes_as_in_file": [{'key': 'Neckline', 'evidences': []}, {'key': 'Pattern', 'evidences': []}, , {'key': 'Type', 'evidences': []}], 
    "predicted_attribute_vals": "[Taylor](Brand) [Dresses](Type) Women's [High](Size) [Low Lace](Type) Shirt Dress",
    "category": "Dresses"
}
]

Then following Section 5.3 of the paper,

• No value (NN), some incorrect Value (NV) for the first sample would be

No value (NN): 1 # No Pattern in "predicted_attribute_vals"
some incorrect Value (NV): 0 # No Pattern in "predicted_attribute_vals", thus cannot be incorrect

• No value (NN), some incorrect Value (NV) for the second sample (which is very similar to the first sample) would be

No value (NN): 2 # No Neckline, Pattern in "predicted_attribute_vals"
some incorrect Value (NV): 1 # [Dresses](Type) in "predicted_attribute_vals"

It seems that some incorrect Value (NV) can arbitrarily change depending on "target_attributes_as_in_file" which is not consistent across the same category (every sample under category="Dresses" has different "target_attributes_as_in_file"), is this expected?

Hi, I found there are multiple values for one attribute in one paragraph in this dataset. But in your paper, the model only "seeks the best answer span in the product context". Can this model extracts multiple spans in the product context for one attribute?

data with multiple spans

{
  "id": "8198319301",
  "category": "Coats & Jackets",
  "paragraphs": [
    {
      "text": "HTOOHTOOH Women's Plus-size Casual Turn Down Collar Mid Length Jean Jacket",
      "source": "title"
    },
    ...
  ],
  "attributes": [
    {
      "key": "Style",
      "evidences": [
        {
          "value": "Casual",
          "pid": 0,
          "begin": 28,
          "end": 34
        },
        {
          "value": "Jean Jacket",
          "pid": 0,
          "begin": 63,
          "end": 74
        }
      ]
    }
  ]
}

{
  "id": "B00002N7X0",
  "category": "Aprons",
  "paragraphs": [
    {
      "text": "McGuire Nicholas C9 4 Pocket Utility Bib Apron in Natural Cotton",
      "source": "title"
    },
    {
      "text": "Constructed of heavy duty but lightweight cotton and ideal for a variety of jobs. The large waist pockets help to store tools or brushes. Reinforced at stress points for added durability. 2 large waist pockets 1 medium bib pocket 1 small bib pocket Extra reinforcement at stress points Canvas loop neck & waist tie Cotton canvas.",
      "source": "description"
    },
    ...
  ],
  "attributes": [
    {
      "key": "Style",
      "evidences": [
        {
          "value": "Bib",
          "pid": 0,
          "begin": 37,
          "end": 40
        },
        {
          "value": "bib",
          "pid": 1,
          "begin": 219,
          "end": 222
        },
        {
          "value": "bib",
          "pid": 1,
          "begin": 238,
          "end": 241
        },
        {
          "value": "neck",
          "pid": 1,
          "begin": 298,
          "end": 302
        },
        ...
      ]
    }
  ]
}