I'm trying to replicate the results (to ensure I'm setting everything up properly) using the WikiSQL dataset. I am using "tapas_wikisql_sqa_masklm_small_reset.zip" model. However when I try running:

!python tapas/tapas/run_task_main.py \
  --task="WIKISQL" \
  --input_dir="data/" \
  --output_dir="results/wsql/input_data" \
  --bert_vocab_file="tapas_model/bert_config.json" \
  --mode="create_data"

I encounter the following error:

I1005 02:55:27.819439 139825599944576 sqa_utils.py:102] Total	Valid	Failed	File
56355	55775	580	train.tsv
8421	8421	0	dev.tsv
15878	15878	0	test.tsv
Creating TF examples ...
I1005 02:55:36.143442 139825599944576 run_task_main.py:152] Creating TF examples ...
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tapas/scripts/prediction_utils.py:48: tf_record_iterator (from tensorflow.python.lib.io.tf_record) is deprecated and will be removed in a future version.
Instructions for updating:
Use eager execution and: 
`tf.data.TFRecordDataset(path)`
W1005 02:55:36.144381 139825599944576 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/tapas/scripts/prediction_utils.py:48: tf_record_iterator (from tensorflow.python.lib.io.tf_record) is deprecated and will be removed in a future version.
Instructions for updating:
Use eager execution and: 
`tf.data.TFRecordDataset(path)`
I1005 02:55:36.159258 139825599944576 number_annotation_utils.py:149] Can't consolidate types: (None, text: "Current slogan"
) {0: [float_value: 2013.0
, date {
  year: 2013
}
], 1: [], 2: [], 3: [], 4: [], 5: [], 6: []} 1
Traceback (most recent call last):
  File "tapas/tapas/run_task_main.py", line 782, in <module>
    app.run(main)
  File "/usr/local/lib/python3.6/dist-packages/absl/app.py", line 300, in run
    _run_main(main, args)
  File "/usr/local/lib/python3.6/dist-packages/absl/app.py", line 251, in _run_main
    sys.exit(main(argv))
  File "tapas/tapas/run_task_main.py", line 743, in main
    output_dir=output_dir)
  File "tapas/tapas/run_task_main.py", line 178, in _create_all_examples
    test_mode=test_mode)
  File "tapas/tapas/run_task_main.py", line 231, in _create_examples
    examples.append(converter.convert(interaction, i))
  File "/usr/local/lib/python3.6/dist-packages/tapas/utils/tf_example_utils.py", line 1096, in convert
    drop_rows_to_fit=self._drop_rows_to_fit)
  File "/usr/local/lib/python3.6/dist-packages/tapas/utils/tf_example_utils.py", line 1018, in _to_trimmed_features
    serialized_example.tokens, feature_dict, table=table, question=question)
  File "/usr/local/lib/python3.6/dist-packages/tapas/utils/tf_example_utils.py", line 673, in _to_features
    input_ids = self._to_token_ids(tokens)
  File "/usr/local/lib/python3.6/dist-packages/tapas/utils/tf_example_utils.py", line 656, in _to_token_ids
    return self._tokenizer.convert_tokens_to_ids(_get_pieces(tokens))
  File "/usr/local/lib/python3.6/dist-packages/tapas/utils/tf_example_utils.py", line 332, in convert_tokens_to_ids
    return self._wp_tokenizer.convert_tokens_to_ids(word_pieces)
  File "/usr/local/lib/python3.6/dist-packages/official/nlp/bert/tokenization.py", line 190, in convert_tokens_to_ids
    return convert_by_vocab(self.vocab, tokens)
  File "/usr/local/lib/python3.6/dist-packages/official/nlp/bert/tokenization.py", line 150, in convert_by_vocab
    output.append(vocab[item])
KeyError: '[CLS]'

Is this an issue with the WikiSQL data itself, or is it the .py utility within TAPAS?

Hi,

Thanks for releasing this in open source. Wonderful concept and very different for other seq2seq or ln2sql like approaches.

I am facing performance issues when trying with sqa prediction notebook (using SQA Large). It takes more than 60 seconds on a dual gpu machine for evaluating the model and giving response to a query. Is this normal? How can we improve the prediction time?

Thanks, Manish

Hello,

I am trying to run this codebase on a single machine with eight GPUs. I have installed all through requirements.txt and prepped the data. When I run, I am only able to use train_batch_size=8 and notice that only one of the eight GPUs is utilized (the other 7 show ~300MB of data on device while the first GPU shows ~15GB). Additionally, while I can see this usage of the GPU(s) by the run script, I get an output message in the train log of: I0514 19:25:27.752172 139628342118208 tpu_estimator.py:2965] Running train on CPU, though I have been ignoring this for now. So I am trying to get the other seven GPUs in the loop so that I can train with train_batch_size=64.

I initially tried wrapping the optimization code in:

strategy = tf.distribute.MirroredStrategy()
with strategy.scope():
   # rest here from bert/optimization.py

and I notice that the model is properly replicated across the eight GPUs, however I cannot expand my train_batch_size to any multiple larger than 8. I tried wrapping the dataset object, at the end of input_fn before returning, in strategy.experimental_distribute_dataset(ds) to see if it was a matter of not sending batches to each device. However, I ran into deeper errors that I am unfamiliar with when pursuing this route (if this is a preferable way to enable multi-GPU I could update this issue with stack traces I got after running with the aforementioned changes).

Before debugging further in this direction, I tried to step back to the outer run_task_main.py after reading that you can instead pass MirroredStrategy or CentralStorageStrategy objects directly into the RunConfig that goes into an Estimator. So I undid the aforementioned changes that I manually made in the lower levels (e.g. reset repo back to master) and added to:

run_config = tf.estimator.tpu.RunConfig(
  ...
  train_distribute=strategy,
  ...

However, I now run into the error:

Traceback (most recent call last):
  File "tapas/run_task_main.py", line 777, in <module>
    app.run(main)
  File "/mydata/repos/tapas2/venv/lib/python3.7/site-packages/absl/app.py", line 299, in run
    _run_main(main, args)
  File "/mydata/repos/tapas2/venv/lib/python3.7/site-packages/absl/app.py", line 250, in _run_main
    sys.exit(main(argv))
  File "tapas/run_task_main.py", line 762, in main
    loop_predict=FLAGS.loop_predict,
  File "tapas/run_task_main.py", line 440, in _train_and_predict
    max_steps=tapas_config.num_train_steps,
  File "/mydata/repos/tapas2/venv/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/tpu/tpu_estimator.py", line 2876, in train
    rendezvous.raise_errors()
  File "/mydata/repos/tapas2/venv/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/tpu/error_handling.py", line 131, in raise_errors
    six.reraise(typ, value, traceback)
  File "/mydata/repos/tapas2/venv/lib/python3.7/site-packages/six.py", line 703, in reraise
    raise value
  File "/mydata/repos/tapas2/venv/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/tpu/tpu_estimator.py", line 2871, in train
    saving_listeners=saving_listeners)
  File "/mydata/repos/tapas2/venv/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 367, in train
    loss = self._train_model(input_fn, hooks, saving_listeners)
  File "/mydata/repos/tapas2/venv/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1156, in _train_model
    return self._train_model_distributed(input_fn, hooks, saving_listeners)
  File "/mydata/repos/tapas2/venv/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1219, in _train_model_distributed
    self._config._train_distribute, input_fn, hooks, saving_listeners)
  File "/mydata/repos/tapas2/venv/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1255, in _actual_train_model_distributed
    input_fn, ModeKeys.TRAIN, strategy)
  File "/mydata/repos/tapas2/venv/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1009, in _get_iterator_from_input_fn
    lambda input_context: self._call_input_fn(input_fn, mode,
  File "/mydata/repos/tapas2/venv/lib/python3.7/site-packages/tensorflow/python/distribute/distribute_lib.py", line 774, in make_input_fn_iterator
    input_fn, replication_mode)
  File "/mydata/repos/tapas2/venv/lib/python3.7/site-packages/tensorflow/python/distribute/distribute_lib.py", line 406, in make_input_fn_iterator
    input_fn, replication_mode=replication_mode)
  File "/mydata/repos/tapas2/venv/lib/python3.7/site-packages/tensorflow/python/distribute/parameter_server_strategy.py", line 318, in _make_input_fn_iterator
    self._container_strategy())
  File "/mydata/repos/tapas2/venv/lib/python3.7/site-packages/tensorflow/python/distribute/input_lib.py", line 550, in __init__
    result = input_fn(ctx)
  File "/mydata/repos/tapas2/venv/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1010, in <lambda>
    input_context))
TypeError: _call_input_fn() takes 3 positional arguments but 4 were given

which I suspect may have to do with the functools.partial wrap around input_fn, but I am having trouble understanding this or determining next steps (I am generally unfamiliar with Tensorflow as a library).

If anybody can help me with this it would be greatly appreciated. Thanks so much for the work and time!

I'm trying to produce the reported numbers in the README in Colab for WikiSQL (supervised and weakly supervised) and SQA.

For WikiSQL I am using tapas_wikisql_sqa_masklm_small_reset. For the weakly supervised I tried:

!python tapas/tapas/run_task_main.py \
  --task="WIKISQL" \
  --output_dir="gs://gs-colab-bucket/data" \
  --bert_config_file="data/wikisql/model/bert_config.json" \
  --bert_vocab_file="data/wikisql/model/vocab.txt" \
  --tpu_name='grpc://10.11.116.106:8470' \
  --use_tpu="true" \
  --mode="predict_and_evaluate"

But results were nowhere close:

I1006 22:51:43.239191 140375104935808 calc_metrics_utils.py:414] denotation_accuracy=0.004988
dev denotation accuracy: 0.0050
I1006 22:51:43.456792 140375104935808 run_task_main.py:152] dev denotation accuracy: 0.0050
I1006 22:52:38.437489 140375104935808 calc_metrics_utils.py:414] denotation_accuracy=0.006361
test denotation accuracy: 0.0064
I1006 22:52:38.888067 140375104935808 run_task_main.py:152] test denotation accuracy: 0.0064

Trying WikiSQL supervised:

!python tapas/tapas/run_task_main.py \
  --task="WIKISQL_SUPERVISED" \
  --output_dir="gs://gs-colab-bucket/data_supervised" \
  --bert_config_file="data_supervised/wikisql_supervised/model/bert_config.json" \
  --bert_vocab_file="data_supervised/wikisql_supervised/model/vocab.txt" \
  --tpu_name='grpc://10.11.116.106:8470' \
  --use_tpu="true" \
  --mode="predict_and_evaluate"

Resulted in much better numbers, but still not matching the README:

`tf.data.TFRecordDataset(path)`
I1010 04:55:43.283555 140631071238016 calc_metrics_utils.py:414] denotation_accuracy=0.749673
dev denotation accuracy: 0.7497
I1010 04:55:43.428153 140631071238016 run_task_main.py:152] dev denotation accuracy: 0.7497
I1010 04:56:35.534448 140631071238016 calc_metrics_utils.py:414] denotation_accuracy=0.736554
test denotation accuracy: 0.7366
I1010 04:56:35.813148 140631071238016 run_task_main.py:152] test denotation accuracy: 0.7366

Next I tried SQA using tapas_sqa_masklm_large_reset:

!python tapas/tapas/run_task_main.py \
  --task="SQA" \
  --output_dir="gs://gs-colab-bucket/data_sqa" \
  --bert_config_file="gs://gs-colab-bucket/data_sqa/sqa/model/bert_config.json" \
  --bert_vocab_file="gs://gs-colab-bucket/data_sqa/sqa/model/vocab.txt" \
  --loop_predict="false" \
  --tpu_name='grpc://10.116.216.194:8470' \
  --use_tpu="true" \
  --mode="predict_and_evaluate"

But I again get results 5% lower than README:

I1010 20:17:49.353913 140657230886784 calc_metrics_utils.py:414] denotation_accuracy=0.642384
dev denotation accuracy: 0.6424
I1010 20:17:49.409619 140657230886784 run_task_main.py:152] dev denotation accuracy: 0.6424
Warning: Can't evaluate for dev_seq because gs://gs-colab-bucket/data_sqa/sqa/model/random-split-1-dev_sequence.tsv doesn't exist.
W1010 20:17:49.546101 140657230886784 run_task_main.py:157] Can't evaluate for dev_seq because gs://gs-colab-bucket/data_sqa/sqa/model/random-split-1-dev_sequence.tsv doesn't exist.
I1010 20:18:00.948407 140657230886784 calc_metrics_utils.py:414] denotation_accuracy=0.652722
test denotation accuracy: 0.6527
I1010 20:18:01.029566 140657230886784 run_task_main.py:152] test denotation accuracy: 0.6527
Warning: Can't evaluate for test_seq because gs://gs-colab-bucket/data_sqa/sqa/model/test_sequence.tsv doesn't exist.

I see the warnings in above regarding sequence, but that seems to have to do with the TPU:

Warning: Skipping SQA sequence evaluation because eval is running on TPU.

Update: also tried the WTQ dataset and tapas_wtq_wikisql_sqa_masklm_large_reset model:

!python tapas/tapas/run_task_main.py \
  --task="WTQ" \
  --output_dir="gs://gs-colab-bucket/data_wtq" \
  --bert_config_file="gs://gs-colab-bucket/data_wtq/wtq/model/bert_config.json" \
  --bert_vocab_file="gs://gs-colab-bucket/data_wtq/wtq/model/vocab.txt" \
  --loop_predict="false" \
  --tpu_name='grpc://10.92.172.218:8470' \
  --use_tpu="true" \
  --mode="predict_and_evaluate"

Getting results 10% lower than the README:

`tf.data.TFRecordDataset(path)`
I1011 04:48:55.989768 140620040472448 calc_metrics_utils.py:414] denotation_accuracy=0.387544
dev denotation accuracy: 0.3875
I1011 04:48:56.042782 140620040472448 run_task_main.py:152] dev denotation accuracy: 0.3875
I1011 04:49:14.033151 140620040472448 calc_metrics_utils.py:414] denotation_accuracy=0.390654
test denotation accuracy: 0.3907
I1011 04:49:14.111141 140620040472448 run_task_main.py:152] test denotation accuracy: 0.3907

I'm wondering if anyone has been able to successfully replicate the reported numbers?

Hi, I am stuck at result path in sample code it throwing following error [Errno 2] No such file or directory: 'results/sqa/model/test_sequence.tsv' I even checked in result folder i did not find the test_sequence.tsv file

Hi all,

When trying to run the Colab notebook I get:

FileNotFoundError: [Errno 2] No such file or directory: 'results/sqa/model/test_sequence.tsv'

Also, when I tried to run the run_task_main.py on my local device I get ModuleNotFoundError: No module named 'tapas'

I've checked and all the folders contains init.py file and not sure how to run it Can you please advise?

I tried to run the prediction on a table of size 1.8M X 60 (using the sqa large model) in the example colab file. The following are the steps that I had taken:

1. Changed the following line in the bert_config.json max_position_embeddings: 2048
2. Changed the max_seq_length max_seq_length = 2048

However, it just returns the table without any predictions.

Could you please provide the changes that one should make to accommodate a table of such size. Thank you.

Hi, can you share the trained model on WikiTableQuestion please?

Because I tried all base models that you shared without doing any fine-tuning and got error like this tensorflow.python.framework.errors_impl.NotFoundError: Key column_output_bias not found in checkpoint. I guess the base model are not trained using WikiTableQuestions right?

Hello,

I am new to this topic and I'm currently trying to use the pruning/filtering method for long tables in the WTQ notebook. I tried using the flag --prune_columns in the prediction function, but it still gives me "Can't convert interaction: error: Sequence too long". What are the necessary steps to filter/prune long tables during prediction?

Thank you in advance.

Hello,

While model data creation using task WTQ, I got AttributeError: 'GFile' object has no attribute 'readable'

COMMAND USED: ! python tapas/tapas/run_task_main.py
--task="WTQ"
--input_dir="{input_dir}"
--output_dir="{output_dir}"
--bert_vocab_file="/content/tapas_model/vocab.txt"
--mode="create_data"

ERROR :

`WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/compat/v2_compat.py:96: disable_resource_variables (from tensorflow.python.ops.variable_scope) is deprecated and will be removed in a future version.
Instructions for updating:
non-resource variables are not supported in the long term
Creating interactions ...
I0928 05:29:55.827632 140189694424960 run_task_main.py:152] Creating interactions ...
I0928 05:29:55.827936 140189694424960 wtq_utils.py:152] Converting data from: training.tsv...
Traceback (most recent call last):
  File "tapas/tapas/run_task_main.py", line 782, in <module>
    app.run(main)
  File "/usr/local/lib/python3.6/dist-packages/absl/app.py", line 300, in run
    _run_main(main, args)
  File "/usr/local/lib/python3.6/dist-packages/absl/app.py", line 251, in _run_main
    sys.exit(main(argv))
  File "tapas/tapas/run_task_main.py", line 736, in main
    task_utils.create_interactions(task, FLAGS.input_dir, output_dir)
  File "/usr/local/lib/python3.6/dist-packages/tapas/utils/task_utils.py", line 110, in create_interactions
    wtq_utils.convert(input_dir, output_dir)
  File "/usr/local/lib/python3.6/dist-packages/tapas/utils/wtq_utils.py", line 241, in convert
    _convert_data(table_cache, input_dir, output_dir, train_file, version)
  File "/usr/local/lib/python3.6/dist-packages/tapas/utils/wtq_utils.py", line 168, in _convert_data
    table = _read_wtq_table(input_dir, wtq_table_id)
  File "/usr/local/lib/python3.6/dist-packages/tapas/utils/wtq_utils.py", line 96, in _read_wtq_table
    dtype='str',
  File "/usr/local/lib/python3.6/dist-packages/pandas/io/parsers.py", line 676, in parser_f
    return _read(filepath_or_buffer, kwds)
  File "/usr/local/lib/python3.6/dist-packages/pandas/io/parsers.py", line 448, in _read
    parser = TextFileReader(fp_or_buf, **kwds)
  File "/usr/local/lib/python3.6/dist-packages/pandas/io/parsers.py", line 880, in __init__
    self._make_engine(self.engine)
  File "/usr/local/lib/python3.6/dist-packages/pandas/io/parsers.py", line 1114, in _make_engine
    self._engine = CParserWrapper(self.f, **self.options)
  File "/usr/local/lib/python3.6/dist-packages/pandas/io/parsers.py", line 1880, in __init__
    src = TextIOWrapper(src, encoding=encoding, newline="")
AttributeError: 'GFile' object has no attribute 'readable'`

@muelletm @ebursztein Can you release the tfrecords data for pre-training? I want to develop a 6 layers pretrained checkpoint for TAPAS.

I am currently running the pre-training data generation in my single CPU machine and it has been running for 3 days. Does it take so long on CPU? Should it create any temporary files? Because I cannot see train.tfrecords and test.tfrecords file even though it has been running for 3 days.

Can we run pre-training data on GPU? I tried to allocate the gpu but the existing code is not using any?

Basically im trying to use HuggingFace's Tapas transfer to make an open-domain Q&A bot, but I need to tokenize the strings and table. I've been trying to use Deep Java Library to run it on Android, but i don't think I could do that without rewriting Huggingface's tokenizer in Java from scratch. Does anyone know if there's an Tapas model that runs on Android/Java?

Hello Maintainer of Github repo google-research/tapas!

Thank you for your work on google-research/tapas. This GitHub project is interesting, and we think that it would be a great addition to make this work instantly discoverable & available as an API for all your users, to quickly try and use it in their applications.

The list of model card(s) covered by this PR are:

• https://console.tiyaro.ai/explore/google-tapas-base-finetuned-wtq-3434648
• https://console.tiyaro.ai/explore/google-tapas-large-finetuned-wtq-9758148

On Tiyaro, every model in google-research/tapas will get its own:

• Dedicated model card (e.g. https://console.tiyaro.ai/explore/google-tapas-base-finetuned-wtq-3434648
• Model demo (e.g. https://console.tiyaro.ai/explore/google-tapas-base-finetuned-wtq-3434648/demo)
• Unique Inference API (e.g. https://api.tiyaro.ai/explore/huggingface/1//google/tapas-base-finetuned-wtq)
• Sample code snippets and swagger spec for the API

Users will also be able to compare your model with other models of similar types on various parameters using Tiyaro Experiments (https://tiyaro.ai/blog/ocr/)

—- I am from Tiyaro.ai (https://tiyaro.ai/). We are working on enabling developers to instantly evaluate, use and customize the world’s best AI. We are constantly working on adding new features to Tiyaro EasyTrain, EasyServe & Experiments, to make the best use of your ML model, and making AI more accessible for anyone.

Sincerely, I-Jong Lin

Hi,

For below table I search "total profit for delhi?" Delhi is not in the table but it's calculating sum of profit for all the rows.

It seems to be happen for each table if the search keyword is not in the table. Please suggest how to proceed?

I tried on hugging face with similar query?

Split the cell with "clone and install" and "run import" code. And added a note to remind the user to restart the runtime.

Because execution under one single cell does not give the user a chance to restart the runtime which eventually results in ModuleNotFoundError for imports of tapas.utils fucntions.

I am trying to fine-tune Tapas following the instructions here: https://huggingface.co/transformers/v4.3.0/model_doc/tapas.html#usage-fine-tuning , Weak supervision for aggregation (WTQ) using the https://www.microsoft.com/en-us/download/details.aspx?id=54253 , which follow the required format of dataset in the SQA format, tsv files with most of the named columns. But, there is no float_answer column. And as mentioned,

float_answer: the float answer to the question, if there is one (np.nan if there isn’t). Only required in case of weak supervision for aggregation (such as WTQ and WikiSQL)

Since I am using WTQ, I need the float_answer column. I tried populating float_answer based on answer_text as suggested here, using https://github.com/google-research/tapas/blob/master/tapas/utils/interaction_utils_parser.py 's parse_question(table, question, mode) function. However, I am getting errors.

I copied everything from here and put these args: .

But, I get this error: TypeError: Parameter to CopyFrom() must be instance of same class: expected language.tapas.Question got str.

1) Can you, please help understand what args should I Use or how else can I populate float_answer?

I am using table_csv and the question, answer to which is in the table given:

2) Also we have tried to simply add float_answer column and make all the values np.nan. Crashed, too.

encoding["float_answer"] = torch.tensor(float("nan"))

Is there tutorial for WTQ fine-tuning? Thanx!