Lingvo is a framework for building neural networks in Tensorflow, particularly sequence models.

Last update: Jan 5, 2023

Overview

Lingvo

What is it?

Lingvo is a framework for building neural networks in Tensorflow, particularly sequence models.

A list of publications using Lingvo can be found here.

Releases
- Major breaking changes
Quick start
Models
References
License

Releases

PyPI Version	Commit
0.10.0	075fd1d88fa6f92681f58a2383264337d0e737ee
0.9.1	c1124c5aa7af13d2dd2b6d43293c8ca6d022b008
0.9.0	f826e99803d1b51dccbbbed1ef857ba48a2bbefe

Older releases

PyPI Version	Commit
0.8.2	93e123c6788e934e6b7b1fd85770371becf1e92e
0.7.2	b05642fe386ee79e0d88aa083565c9a93428519e

Details for older releases are unavailable.

Major breaking changes

NOTE: this is not a comprehensive list. Lingvo releases do not offer any guarantees regarding backwards compatibility.

HEAD

Nothing here.

0.10.0

General
- The theta_fn arg to CreateVariable() has been removed.

0.9.1

General
- Python 3.9 is now supported.
- ops.beam_search_step now takes and returns an additional arg beam_done.
- The namedtuple beam_search_helper.BeamSearchDecodeOutput now removes the field done_hyps.

0.9.0

General
- Tensorflow 2.5 is now the required version.
- Python 3.5 support has been removed.
- py_utils.AddGlobalVN and py_utils.AddPerStepVN have been combined into py_utils.AddVN.
- BaseSchedule().Value() no longer takes a step arg.
- Classes deriving from BaseSchedule should implement Value() not FProp().
- theta.global_step has been removed in favor of py_utils.GetGlobalStep().
- py_utils.GenerateStepSeedPair() no longer takes a global_step arg.
- PostTrainingStepUpdate() no longer takes a global_step arg.
- The fatal_errors argument to custom input ops now takes error message substrings rather than integer error codes.

Older releases

0.8.2

General
- NestedMap Flatten/Pack/Transform/Filter etc now expand descendent dicts as well.
- Subclasses of BaseLayer extending from abc.ABCMeta should now extend base_layer.ABCLayerMeta instead.
- Trying to call self.CreateChild outside of __init__ now raises an error.
- base_layer.initializer has been removed. Subclasses no longer need to decorate their __init__ function.
- Trying to call self.CreateVariable outside of __init__ or _CreateLayerVariables now raises an error.
- It is no longer possible to access self.vars or self.theta inside of __init__. Refactor by moving the variable creation and access to _CreateLayerVariables. The variable scope is set automatically according to the layer name in _CreateLayerVariables.

Details for older releases are unavailable.

Quick start

Installation

There are two ways to set up Lingvo: installing a fixed version through pip, or cloning the repository and building it with bazel. Docker configurations are provided for each case.

If you would just like to use the framework as-is, it is easiest to just install it through pip. This makes it possible to develop and train custom models using a frozen version of the Lingvo framework. However, it is difficult to modify the framework code or implement new custom ops.

If you would like to develop the framework further and potentially contribute pull requests, you should avoid using pip and clone the repository instead.

pip:

The Lingvo pip package can be installed with pip3 install lingvo.

See the codelab for how to get started with the pip package.

From sources:

The prerequisites are:

a TensorFlow 2.6 installation,
a C++ compiler (only g++ 7.3 is officially supported), and
the bazel build system.

Refer to docker/dev.dockerfile for a set of working requirements.

git clone the repository, then use bazel to build and run targets directly. The python -m module commands in the codelab need to be mapped onto bazel run commands.

docker:

Docker configurations are available for both situations. Instructions can be found in the comments on the top of each file.

lib.dockerfile has the Lingvo pip package preinstalled.
dev.dockerfile can be used to build Lingvo from sources.

How to install docker.

Running the MNIST image model

Preparing the input data

pip:

mkdir -p /tmp/mnist
python3 -m lingvo.tools.keras2ckpt --dataset=mnist

bazel:

mkdir -p /tmp/mnist
bazel run -c opt //lingvo/tools:keras2ckpt -- --dataset=mnist

The following files will be created in /tmp/mnist:

mnist.data-00000-of-00001: 53MB.
mnist.index: 241 bytes.

Running the model

pip:

cd /tmp/mnist
curl -O https://raw.githubusercontent.com/tensorflow/lingvo/master/lingvo/tasks/image/params/mnist.py
python3 -m lingvo.trainer --run_locally=cpu --mode=sync --model=mnist.LeNet5 --logdir=/tmp/mnist/log

bazel:

(cpu) bazel build -c opt //lingvo:trainer
(gpu) bazel build -c opt --config=cuda //lingvo:trainer
bazel-bin/lingvo/trainer --run_locally=cpu --mode=sync --model=image.mnist.LeNet5 --logdir=/tmp/mnist/log --logtostderr

After about 20 seconds, the loss should drop below 0.3 and a checkpoint will be saved, like below. Kill the trainer with Ctrl+C.

trainer.py:518] step:   205, steps/sec: 11.64 ... loss:0.25747201 ...
checkpointer.py:115] Save checkpoint
checkpointer.py:117] Save checkpoint done: /tmp/mnist/log/train/ckpt-00000205

Some artifacts will be produced in /tmp/mnist/log/control:

params.txt: hyper-parameters.
model_analysis.txt: model sizes for each layer.
train.pbtxt: the training tf.GraphDef.
events.*: a tensorboard events file.

As well as in /tmp/mnist/log/train:

checkpoint: a text file containing information about the checkpoint files.
ckpt-*: the checkpoint files.

Now, let's evaluate the model on the "Test" dataset. In the normal training setup the trainer and evaler should be run at the same time as two separate processes.

pip:

python3 -m lingvo.trainer --job=evaler_test --run_locally=cpu --mode=sync --model=mnist.LeNet5 --logdir=/tmp/mnist/log

bazel:

bazel-bin/lingvo/trainer --job=evaler_test --run_locally=cpu --mode=sync --model=image.mnist.LeNet5 --logdir=/tmp/mnist/log --logtostderr

Kill the job with Ctrl+C when it starts waiting for a new checkpoint.

base_runner.py:177] No new check point is found: /tmp/mnist/log/train/ckpt-00000205

The evaluation accuracy can be found slightly earlier in the logs.

base_runner.py:111] eval_test: step:   205, acc5: 0.99775392, accuracy: 0.94150388, ..., loss: 0.20770954, ...

Running the machine translation model

To run a more elaborate model, you'll need a cluster with GPUs. Please refer to third_party/py/lingvo/tasks/mt/README.md for more information.

Running the GShard transformer based giant language model

To train a GShard language model with one trillion parameters on GCP using CloudTPUs v3-512 using 512-way model parallelism, please refer to third_party/py/lingvo/tasks/lm/README.md for more information.

Running the 3d object detection model

To run the StarNet model using CloudTPUs on GCP, please refer to third_party/py/lingvo/tasks/car/README.md.

Models

Automatic Speech Recognition

Listen, Attend and Spell.
William Chan, Navdeep Jaitly, Quoc V. Le, and Oriol Vinyals. ICASSP 2016.

End-to-end Continuous Speech Recognition using Attention-based Recurrent NN: First Results.
Jan Chorowski, Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. arXiv 2014.
- asr.librispeech.Librispeech960Grapheme
- asr.librispeech.Librispeech960Wpm

Car

StarNet: Targeted Computation for Object Detection in Point Clouds.
Jiquan Ngiam, Benjamin Caine, Wei Han, Brandon Yang, Yuning Chai, Pei Sun, Yin Zhou, Xi Yi, Ouais Alsharif, Patrick Nguyen, Zhifeng Chen, Jonathon Shlens, and Vijay Vasudevan. arXiv 2019.

Image

Gradient-based learning applied to document recognition.
Yann LeCun, Leon Bottou, Yoshua Bengio, and Patrick Haffner. IEEE 1998.
- image.mnist.LeNet5

Language Modelling

mt.wmt14_en_de_xendec.WmtEnDeXEnDec⁸
Exploring the Limits of Language Modeling.
Rafal Jozefowicz, Oriol Vinyals, Mike Schuster, Noam Shazeer, and Yonghui Wu. arXiv, 2016.
- lm.one_billion_wds.WordLevelOneBwdsSimpleSampledSoftmax
GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding.
Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer and Zhifeng Chen arXiv, 2020.
- lm.synthetic_packed_input.DenseLm1T16x16

Machine Translation

The Best of Both Worlds: Combining Recent Advances in Neural Machine Translation.
Mia X. Chen, Orhan Firat, Ankur Bapna, Melvin Johnson, Wolfgang Macherey, George Foster, Llion Jones, Mike Schuster, Noam Shazeer, Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Zhifeng Chen, Yonghui Wu, and Macduff Hughes. ACL 2018.
Self-supervised and Supervised Joint Training for Resource-rich Neural Machine Translation.
Yong Cheng, Wei Wang, Lu Jiang, and Wolfgang Macherey. ICML 2021.
- mt.wmt14_en_de_xendec.WmtEnDeXEnDec

References

Please cite this paper when referencing Lingvo.

@misc{shen2019lingvo,
    title={Lingvo: a Modular and Scalable Framework for Sequence-to-Sequence Modeling},
    author={Jonathan Shen and Patrick Nguyen and Yonghui Wu and Zhifeng Chen and others},
    year={2019},
    eprint={1902.08295},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}

License

Apache License 2.0

Comments

Undefined reference to google::protobuf::FileDescriptor::DebugString()

I just installed lingvo and it looks like I face the issue with protobuf linking. The nightly-tf seems to be up-to-date

mironov@70e0b410070b:~/lingvo$ python -c "import tensorflow as tf;print(tf.__version__)"
1.14.1-dev20190305

The exact error from bazel build is

mironov@70e0b410070b:~/lingvo$ bazel build -c opt //lingvo:trainer
INFO: Analysed target //lingvo:trainer (22 packages loaded).
INFO: Found 1 target...
ERROR: /workspace/lingvo/lingvo/tools/BUILD:98:1: Linking of rule '//lingvo/tools:generate_proto_def' failed (Exit 1)
bazel-out/host/bin/lingvo/tools/_objs/generate_proto_def/generate_proto_def.o:generate_proto_def.cc:function (anonymous namespace)::WriteDotProto(google::protobuf::FileDescriptor const*, char const*): error: undefined reference to 'google::protobuf::FileDescriptor::DebugString() const'
collect2: error: ld returned 1 exit status
Target //lingvo:trainer failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 3.746s, Critical Path: 2.20s
INFO: 3 processes: 3 processwrapper-sandbox.
FAILED: Build did NOT complete successfully

Could you please check?

bug stat:awaiting response

opened by grwlf 23

list index out of range when running model

when I am running tf.app.run(trainer.main,argv=argv) in model training part I am getting list index out of range error. pls tell me how to get rid out of this...

I am not getting to which element its trying to access
bug stat:awaiting response

opened by nim456 18
How to use tf_debug in Predictor to see the tensor value each step in RNN while inference?
Hello, thank you for your prompt and enthusiastic answer each time~

Recently, I am trying to add LM fusion in ASR task. And I want to check whether I am right by step by step debugging while inference. However, no matter how I use TensorBoardDebugWrapperSession or LocalCLIDebugWrapperSession it will report an error.

I change this line:

https://github.com/tensorflow/lingvo/blob/0d7828b0bbba2615a7581c79f532beb4251adfe8/lingvo/core/predictor.py#L208

to

## When try TensorBoardDebugWrapperSession return self._RunWithValidSession( tf_debug.TensorBoardDebugWrapperSession.run, fetches, feed_dict=feeds, options=run_options) ## When try LocalCLIDebugWrapperSession return self._RunWithDebugSession( tf_debug.LocalCLIDebugWrapperSession.run, fetches, feed_dict=feeds, options=run_options)

I change this line:

https://github.com/tensorflow/lingvo/blob/0d7828b0bbba2615a7581c79f532beb4251adfe8/lingvo/core/predictor.py#L160

to

## When try TensorBoardDebugWrapperSession self._sess = tf_debug.TensorBoardDebugWrapperSession(self._sess, "dm-System-Product-Name:6008") return fn(self._sess, *args, **kwargs)

However, it occurs:

I remove self._sess from fn's args:

## When try TensorBoardDebugWrapperSession self._sess = tf_debug.TensorBoardDebugWrapperSession(self._sess, "dm-System-Product-Name:6008") return fn(*args, **kwargs)

It also occurs an error.

Then, I keep the following line unchanged, and change this line: https://github.com/tensorflow/lingvo/blob/0d7828b0bbba2615a7581c79f532beb4251adfe8/lingvo/core/predictor.py#L142

to

self._sess = sess self._sess = tf_debug.TensorBoardDebugWrapperSession(self._sess, 'localhost:6008')

However, it occurs:

I have tried many other ways. However, none of them succeed. So, would you mind tell me how to use tf_debug in Predictor to see how RNN runs while inference? Or, if you have other method to debug step by step, could you share me your code about it? Thank you a lot!!

(I want to be able to see the results of each step of the variable at runtime like the following picture)
opened by iamxiaoyubei 14

local build error

ERROR: error loading package 'lingvo': Encountered error while reading extension file 'subpar.bzl': no such package '@subpar//': Traceback (most recent call last):
        File "/home/luban/.cache/bazel/_bazel_luban/b5ef85f1c360696308ba7ab9000cfd03/external/bazel_tools/tools/build_defs/repo/git.bzl", line 166
                _clone_or_update(ctx)
        File "/home/luban/.cache/bazel/_bazel_luban/b5ef85f1c360696308ba7ab9000cfd03/external/bazel_tools/tools/build_defs/repo/git.bzl", line 72, in _clone_or_update
                fail(("error cloning %s:\n%s" % (ctx....)))
error cloning subpar:

bug

opened by zh794390558 14

How to call another model while a graph is executed in a tf.Session

I trained two models: ASR Wpm model and Language model. When I passed a intermediate value of ASR model to Language model in order to predict the intermediate result, I got an error. Because tensorflow is not able to pass tensor into feed_dict. But I can only get the value which passed to language model while the ASR model is running. So this value can only be saved in the form of tensor. So, could you tell me how can I call language model in ASR model? I really hope that you can help me. Thank you!!

opened by iamxiaoyubei 11

Large Model - GPipe Transformer - OOM Issue

I am trying to run the large model described in the comment section of https://github.com/tensorflow/lingvo/blob/master/lingvo/tasks/lm/params/one_billion_wds.py#L179

However, I keep getting OOM error.

Following are the variations I have tried: #4 GPU mode: SPLITS = [2 + 11 * i for i in range(GPU-1)] LAYERS = SPLIT[-1] BATCH = 2 or 4 or 8 MICROBATCH =2 or 4 or 8 (respectively)

#8 GPU mode : EMBEDDING_DIM = 1024 GPUS = 8 # For example, on 8 accelerator, each of which has 16G mem. SPLITS = [10 + 12 * i for i in range(GPUS - 1)] LAYERS = SPLITS[-1] #4.9B params BATCH_SIZE = 2 NUM_MICRO_BATCHES = 2

...
019-06-08 01:46:36.299559: I tensorflow/core/common_runtime/bfc_allocator.cc:824] Stats:
Limit:                 15750955008
InUse:                 15584222976
MaxInUse:              15584223232
NumAllocs:                    2361
MaxAllocSize:           3250061312

2019-06-08 01:46:36.299616: W tensorflow/core/common_runtime/bfc_allocator.cc:319] ****************************************************************************************************
2019-06-08 01:46:36.299642: W tensorflow/core/framework/op_kernel.cc:1502] OP_REQUIRES failed at matmul_op.cc:480 : Resource exhausted: OOM when allocating tensor with shape[169,793472] and type float on /job:local/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
2019-06-08 01:46:36.919956: I lingvo/core/ops/record_yielder.cc:317] 0x7fa6a682e060Basic record yielder exit
I0608 01:46:41.227104 140355034937088 base_runner.py:239] trainer done (fatal error).
I0608 01:46:41.227474 140355034937088 base_runner.py:118] trainer exception: From /job:local/replica:0/task:0:
[_Derived_]OOM when allocating tensor with shape[1024,49592] and type float on /job:local/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
         [[{{node Add_28}}]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

         [[gradients/fprop/1bwds_wpm_level_lm/tower_0_0/Sum_435_grad/Tile_G4412]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.


E0608 01:46:41.230962 140355034937088 base_runner.py:246] Traceback (most recent call last):
E0608 01:46:41.231157 140355034937088 base_runner.py:246]   File "/tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/base_runner.py", line 199, in _RunLoop
E0608 01:46:41.231286 140355034937088 base_runner.py:246]     loop_func(*loop_args)
E0608 01:46:41.231404 140355034937088 base_runner.py:246]   File "/tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/trainer.py", line 538, in _Loop
E0608 01:46:41.231527 140355034937088 base_runner.py:246]     model_task.per_example_tensors,
E0608 01:46:41.231688 140355034937088 base_runner.py:246]   File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 950, in run
E0608 01:46:41.231817 140355034937088 base_runner.py:246]     run_metadata_ptr)
E0608 01:46:41.231914 140355034937088 base_runner.py:246]   File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1173, in _run
E0608 01:46:41.231992 140355034937088 base_runner.py:246]     feed_dict_tensor, options, run_metadata)
E0608 01:46:41.232054 140355034937088 base_runner.py:246]   File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1350, in _do_run
E0608 01:46:41.232116 140355034937088 base_runner.py:246]     run_metadata)
E0608 01:46:41.232177 140355034937088 base_runner.py:246]   File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1370, in _do_call
E0608 01:46:41.232239 140355034937088 base_runner.py:246]     raise type(e)(node_def, op, message)
E0608 01:46:41.232300 140355034937088 base_runner.py:246] ResourceExhaustedError: From /job:local/replica:0/task:0:
E0608 01:46:41.232366 140355034937088 base_runner.py:246] [_Derived_]OOM when allocating tensor with shape[1024,49592] and type float on /job:local/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
E0608 01:46:41.232431 140355034937088 base_runner.py:246]        [[{{node Add_28}}]]
E0608 01:46:41.232492 140355034937088 base_runner.py:246] Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
E0608 01:46:41.232563 140355034937088 base_runner.py:246]
E0608 01:46:41.232625 140355034937088 base_runner.py:246]        [[gradients/fprop/1bwds_wpm_level_lm/tower_0_0/Sum_435_grad/Tile_G4412]]
E0608 01:46:41.232690 140355034937088 base_runner.py:246] Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
E0608 01:46:41.232752 140355034937088 base_runner.py:246]
E0608 01:46:41.232812 140355034937088 base_runner.py:246]

opened by msharmavikram 11

Getting segmentation fault while trying to run Gpipe example code

Hi, as you mentioned in #48 about changing OneBWdsGPipeTransformer hparams and then try to run on 8 GPU's and gave the command to run. I did not understand what are those parameters, can I get help which parameters fit for my system. I am using machine consisting of 4 GPU. What ever the parameters I change I am facing segmentation fault core dumped. I am also attaching my system info(GPU).

command : bazel-bin/lingvo/trainer --run_locally=gpu --mode=sync --model=lm.one_billion_wds.OneBWdsGPipeTransformer --logdir=/tmp/mnist/log --logtostderr --worker_split_size=4 segmentation fault.txt

system info: GPU: sys_info.txt

opened by Raviteja1996 10

Is asr task job can run correctly on gpu?

I use tf-nightly-gpu==1.13.0-dev20181116, but when I run asr task, I got the error below.

bazel-bin/lingvo/trainer --run_locally=gpu --mode=sync --model=asr.librispeech.Librispeech960Base --logdir=/tmp/librispeech/log --logtostderr

INFO:tensorflow:Retry: caught exception: _WaitTillInit while running FailedPreconditionError: Attempting to use uninitialized value global_step
	 [[{{node _send_global_step_0}}]]
. Call failed at (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 774, in __bootstrap
    self.__bootstrap_inner()
  File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/trainer.py", line 401, in Start
    self._RunLoop('trainer', self._Loop)
  File "/tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/core/retry.py", line 50, in wrapper
    return func(*args, **kwargs)
  File "/tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/base_runner.py", line 173, in _RunLoop
    loop_func(*args)
Traceback for above exception (most recent call last):
  File "/tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/core/retry.py", line 50, in wrapper
    return func(*args, **kwargs)
  File "/tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/trainer.py", line 416, in _WaitTillInit
    global_step = sess.run(self._model.global_step)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 929, in run
    run_metadata_ptr)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1152, in _run
    feed_dict_tensor, options, run_metadata)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1328, in _do_run
    run_metadata)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1348, in _do_call
    raise type(e)(node_def, op, message)
Waiting for 0.11 seconds before retrying.

opened by fanlu 10

Reduce memory requirement Starnet on Waymo dataset

I have been unable to get Starnet working on TPUs so am now trying on 8 x v100 GPUs. However, I am not running into memory issues.

Are there any way to reduce the memory required of the model?

I have tried setting p.batch_size = 1 in tasks/car/params/waymo.py but still run out of memory.

I have tried many set of training parameters but the ones used for the error below was:

bazel-bin/lingvo/trainer --logtostderr --model=car.waymo.StarNetVehicle --mode=async --logdir=gs://waymo-challange/logs/starnet.waymo.3d_object.v3 --run_locally=gpu --worker_gpus=7 --worker_split_size=7 --worker_replicas=7 --controller_gpus=1

Error:

I0512 13:49:24.586677 139606888535808 base_runner.py:111] step:     0
WARNING:tensorflow:Issue encountered when serializing __batch_norm_update_dict.
Type is unsupported, or the types of the items don't match field type in CollectionDef. Note this is a warning and probably safe to ignore.
'dict' object has no attribute 'name'
W0512 13:49:30.131858 139606896928512 meta_graph.py:436] Issue encountered when serializing __batch_norm_update_dict.
Type is unsupported, or the types of the items don't match field type in CollectionDef. Note this is a warning and probably safe to ignore.
'dict' object has no attribute 'name'
2020-05-12 13:49:30.990706: I ./lingvo/core/ops/input_common.h:74] Create RecordProcessor
2020-05-12 13:49:30.994158: I lingvo/core/ops/input_common.cc:34] Input source weights are empty, fall back to legacy behavior.
2020-05-12 13:49:30.994610: I lingvo/core/ops/record_yielder.cc:376] 0x7ef8a34d9f20 Record yielder start
2020-05-12 13:49:30.994631: I lingvo/core/ops/record_yielder.cc:378] Randomly seed RecordYielder.
2020-05-12 13:49:30.994655: I ./lingvo/core/ops/input_common.h:80] Create batcher
2020-05-12 13:49:30.994678: I lingvo/core/ops/record_yielder.cc:485] Epoch 1 gs://waymo_open_dataset_v_1_0_0_tf_example_lingvo/v.1.0.0/train.tfr-*-of-01000
I0512 13:49:31.538286 139606896928512 checkpointer.py:118] Save checkpoint done: gs://waymo-challange/logs/starnet.waymo.3d_object.v3/train/ckpt-00000000
2020-05-12 13:49:33.201902: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2020-05-12 13:50:02.702309: W tensorflow/core/common_runtime/bfc_allocator.cc:424] Allocator (GPU_0_bfc) ran out of memory trying to allocate 512.00MiB (rounded to 536870912).  Current allocation summary follows.
2020-05-12 13:50:02.702388: I tensorflow/core/common_runtime/bfc_allocator.cc:894] BFCAllocator dump for GPU_0_bfc
2020-05-12 13:50:02.702401: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (256):   Total Chunks: 176, Chunks in use: 176. 44.0KiB allocated for chunks. 44.0KiB in use in bin. 651B client-requested in use in bin.
2020-05-12 13:50:02.702414: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (512):   Total Chunks: 32, Chunks in use: 31. 16.2KiB allocated for chunks. 15.8KiB in use in bin. 15.5KiB client-requested in use in bin.
2020-05-12 13:50:02.702425: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (1024):  Total Chunks: 32, Chunks in use: 29. 37.5KiB allocated for chunks. 33.2KiB in use in bin. 30.0KiB client-requested in use in bin.
2020-05-12 13:50:02.702435: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (2048):  Total Chunks: 1, Chunks in use: 0. 2.2KiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-05-12 13:50:02.702445: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (4096):  Total Chunks: 3, Chunks in use: 1. 14.8KiB allocated for chunks. 4.0KiB in use in bin. 4.0KiB client-requested in use in bin.
2020-05-12 13:50:02.702455: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (8192):  Total Chunks: 2, Chunks in use: 0. 26.5KiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-05-12 13:50:02.702465: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (16384):         Total Chunks: 1, Chunks in use: 0. 17.0KiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-05-12 13:50:02.702474: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (32768):         Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-05-12 13:50:02.702485: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (65536):         Total Chunks: 12, Chunks in use: 9. 771.2KiB allocated for chunks. 576.2KiB in use in bin. 576.0KiB client-requested in use in bin.
2020-05-12 13:50:02.702496: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (131072):        Total Chunks: 14, Chunks in use: 10. 1.89MiB allocated for chunks. 1.25MiB in use in bin. 1.25MiB client-requested in use in bin.
2020-05-12 13:50:02.702506: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (262144):        Total Chunks: 16, Chunks in use: 14. 4.43MiB allocated for chunks. 3.78MiB in use in bin. 3.78MiB client-requested in use in bin.
2020-05-12 13:50:02.702517: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (524288):        Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-05-12 13:50:02.702530: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (1048576):       Total Chunks: 6, Chunks in use: 5. 9.38MiB allocated for chunks. 8.20MiB in use in bin. 8.20MiB client-requested in use in bin.
2020-05-12 13:50:02.702541: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (2097152):       Total Chunks: 14, Chunks in use: 12. 29.92MiB allocated for chunks. 24.59MiB in use in bin. 24.34MiB client-requested in use in bin.
2020-05-12 13:50:02.702551: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (4194304):       Total Chunks: 3, Chunks in use: 0. 14.07MiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-05-12 13:50:02.702561: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (8388608):       Total Chunks: 2, Chunks in use: 0. 25.71MiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-05-12 13:50:02.702571: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (16777216):      Total Chunks: 1, Chunks in use: 0. 28.00MiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-05-12 13:50:02.702581: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (33554432):      Total Chunks: 1, Chunks in use: 0. 32.00MiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-05-12 13:50:02.702590: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (67108864):      Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-05-12 13:50:02.702600: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (134217728):     Total Chunks: 20, Chunks in use: 19. 2.50GiB allocated for chunks. 2.38GiB in use in bin. 2.38GiB client-requested in use in bin.
2020-05-12 13:50:02.702611: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (268435456):     Total Chunks: 31, Chunks in use: 30. 11.98GiB allocated for chunks. 11.50GiB in use in bin. 11.50GiB client-requested in use in bin.
2020-05-12 13:50:02.702622: I tensorflow/core/common_runtime/bfc_allocator.cc:917] Bin for 512.00MiB was 256.00MiB, Chunk State: 
2020-05-12 13:50:02.702638: I tensorflow/core/common_runtime/bfc_allocator.cc:923]   Size: 492.95MiB | Requested Size: 256.0KiB | in_use: 0 | bin_num: 20, prev:   Size: 512.00MiB | Requested Size: 512.00MiB | in_use: 1 | bin_num: -1
2020-05-12 13:50:02.702648: I tensorflow/core/common_runtime/bfc_allocator.cc:930] Next region of size 15702704128
2020-05-12 13:50:02.702660: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ec000000 of size 1280 next 1
2020-05-12 13:50:02.702668: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ec000500 of size 256 next 2
2020-05-12 13:50:02.702677: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ec000600 of size 256 next 3
2020-05-12 13:50:02.702686: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ec000700 of size 256 next 4
2020-05-12 13:50:02.702694: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ec000800 of size 256 next 5
2020-05-12 13:50:02.702703: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ec000900 of size 256 next 6
2020-05-12 13:50:02.702711: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ec000a00 of size 256 next 7
2020-05-12 13:50:02.702719: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ec000b00 of size 256 next 8
2020-05-12 13:50:02.702727: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ec000c00 of size 256 next 9
2020-05-12 13:50:02.702736: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ec000d00 of size 262144 next 10
2020-05-12 13:50:02.702744: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ec040d00 of size 262144 next 11
2020-05-12 13:50:02.702753: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ec080d00 of size 262144 next 12
2020-05-12 13:50:02.702761: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ec0c0d00 of size 262144 next 13
2020-05-12 13:50:02.702769: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ec100d00 of size 262144 next 14
2020-05-12 13:50:02.702777: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ec140d00 of size 262144 next 15
2020-05-12 13:50:02.702792: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ec180d00 of size 262144 next 16
2020-05-12 13:50:02.702801: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ec1c0d00 of size 262144 next 17
2020-05-12 13:50:02.702809: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ec200d00 of size 262144 next 18
2020-05-12 13:50:02.702818: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ec240d00 of size 262144 next 19
2020-05-12 13:50:02.702827: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ec280d00 of size 262144 next 20
2020-05-12 13:50:02.702835: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ec2c0d00 of size 2048000 next 21
2020-05-12 13:50:02.702843: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ec4b4d00 of size 2048000 next 22
2020-05-12 13:50:02.702852: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ec6a8d00 of size 2457600 next 23
2020-05-12 13:50:02.702861: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ec900d00 of size 1228800 next 24
2020-05-12 13:50:02.702869: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4eca2cd00 of size 1228800 next 25
2020-05-12 13:50:02.702878: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ecb58d00 of size 409600 next 26
2020-05-12 13:50:02.702886: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ecbbcd00 of size 409600 next 27
2020-05-12 13:50:02.702894: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ecc20d00 of size 256 next 28
2020-05-12 13:50:02.702903: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ecc20e00 of size 256 next 29
2020-05-12 13:50:02.702912: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ecc20f00 of size 256 next 30
2020-05-12 13:50:02.702920: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ecc21000 of size 256 next 31
2020-05-12 13:50:02.702929: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ecc21100 of size 256 next 32
2020-05-12 13:50:02.702938: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ecc21200 of size 256 next 33
2020-05-12 13:50:02.702953: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ecc21300 of size 256 next 34
2020-05-12 13:50:02.702967: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ecc21400 of size 256 next 35
2020-05-12 13:50:02.702980: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ecc21500 of size 256 next 36
2020-05-12 13:50:02.702991: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ecc21600 of size 256 next 37
2020-05-12 13:50:02.703003: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ecc21700 of size 256 next 38
2020-05-12 13:50:02.703013: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ecc21800 of size 256 next 39
2020-05-12 13:50:02.703023: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ecc21900 of size 2048000 next 40
2020-05-12 13:50:02.703035: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ece15900 of size 256 next 41
2020-05-12 13:50:02.703047: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ece15a00 of size 256 next 42
2020-05-12 13:50:02.703059: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ece15b00 of size 256 next 43
2020-05-12 13:50:02.703070: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ece15c00 of size 256 next 44
2020-05-12 13:50:02.703081: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ece15d00 of size 262144 next 45
2020-05-12 13:50:02.703093: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ece55d00 of size 256 next 46
2020-05-12 13:50:02.703106: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ece55e00 of size 256 next 47
2020-05-12 13:50:02.703118: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ece55f00 of size 512 next 48
2020-05-12 13:50:02.703128: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ece56100 of size 2097152 next 49
2020-05-12 13:50:02.703140: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed056100 of size 1024 next 50
2020-05-12 13:50:02.703151: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed056500 of size 256 next 51
2020-05-12 13:50:02.703160: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed056600 of size 1536 next 52
2020-05-12 13:50:02.703172: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed056c00 of size 256 next 53
2020-05-12 13:50:02.703184: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed056d00 of size 256 next 54
2020-05-12 13:50:02.703194: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed056e00 of size 4096 next 55
2020-05-12 13:50:02.703202: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed057e00 of size 256 next 56
2020-05-12 13:50:02.703211: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed057f00 of size 256 next 57
2020-05-12 13:50:02.703219: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed058000 of size 256 next 58
2020-05-12 13:50:02.703231: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed058100 of size 256 next 59
2020-05-12 13:50:02.703243: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed058200 of size 256 next 60
2020-05-12 13:50:02.703255: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed058300 of size 256 next 61
2020-05-12 13:50:02.703266: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed058400 of size 256 next 62
2020-05-12 13:50:02.703275: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed058500 of size 256 next 63
2020-05-12 13:50:02.703285: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed058600 of size 256 next 64
2020-05-12 13:50:02.703293: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed058700 of size 256 next 65
2020-05-12 13:50:02.703299: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed058800 of size 256 next 66
2020-05-12 13:50:02.703311: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed058900 of size 256 next 67
2020-05-12 13:50:02.703322: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed058a00 of size 256 next 68
2020-05-12 13:50:02.703331: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed058b00 of size 256 next 69
2020-05-12 13:50:02.703342: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed058c00 of size 256 next 70
2020-05-12 13:50:02.703351: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed058d00 of size 256 next 71
2020-05-12 13:50:02.703360: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed058e00 of size 256 next 72
2020-05-12 13:50:02.703368: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed058f00 of size 256 next 73
2020-05-12 13:50:02.703377: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed059000 of size 256 next 74
2020-05-12 13:50:02.703386: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed059100 of size 256 next 75
2020-05-12 13:50:02.703396: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed059200 of size 256 next 76
2020-05-12 13:50:02.703406: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed059300 of size 256 next 77
2020-05-12 13:50:02.703416: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed059400 of size 256 next 78
2020-05-12 13:50:02.703426: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed059500 of size 256 next 79
2020-05-12 13:50:02.703435: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed059600 of size 256 next 80
2020-05-12 13:50:02.703445: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed059700 of size 256 next 81
2020-05-12 13:50:02.703456: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed059800 of size 256 next 82
2020-05-12 13:50:02.703466: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed059900 of size 256 next 83
2020-05-12 13:50:02.703474: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed059a00 of size 256 next 84
2020-05-12 13:50:02.703483: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed059b00 of size 256 next 85
2020-05-12 13:50:02.703493: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed059c00 of size 256 next 86
2020-05-12 13:50:02.703502: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed059d00 of size 256 next 87
2020-05-12 13:50:02.703512: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed059e00 of size 256 next 88
2020-05-12 13:50:02.703522: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed059f00 of size 256 next 89
2020-05-12 13:50:02.703533: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05a000 of size 256 next 90
2020-05-12 13:50:02.703543: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05a100 of size 256 next 91
2020-05-12 13:50:02.703552: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05a200 of size 256 next 92
2020-05-12 13:50:02.703562: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05a300 of size 256 next 93
2020-05-12 13:50:02.703572: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05a400 of size 256 next 94
2020-05-12 13:50:02.703582: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05a500 of size 256 next 95
2020-05-12 13:50:02.703591: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05a600 of size 256 next 96
2020-05-12 13:50:02.703599: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05a700 of size 256 next 97
2020-05-12 13:50:02.703609: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05a800 of size 256 next 98
2020-05-12 13:50:02.703618: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05a900 of size 256 next 99
2020-05-12 13:50:02.703629: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05aa00 of size 256 next 100
2020-05-12 13:50:02.703641: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05ab00 of size 256 next 101
2020-05-12 13:50:02.703652: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05ac00 of size 256 next 102
2020-05-12 13:50:02.703662: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05ad00 of size 256 next 103
2020-05-12 13:50:02.703673: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05ae00 of size 256 next 104
2020-05-12 13:50:02.703681: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05af00 of size 256 next 105
2020-05-12 13:50:02.703690: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05b000 of size 256 next 106
2020-05-12 13:50:02.703698: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05b100 of size 256 next 107
2020-05-12 13:50:02.703707: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05b200 of size 256 next 108
2020-05-12 13:50:02.703714: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05b300 of size 256 next 109
2020-05-12 13:50:02.703722: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05b400 of size 256 next 110
2020-05-12 13:50:02.703730: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05b500 of size 256 next 111
2020-05-12 13:50:02.703742: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05b600 of size 256 next 112
2020-05-12 13:50:02.703750: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05b700 of size 256 next 113
2020-05-12 13:50:02.703758: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05b800 of size 256 next 114
2020-05-12 13:50:02.703767: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05b900 of size 256 next 115
2020-05-12 13:50:02.703775: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05ba00 of size 256 next 116
2020-05-12 13:50:02.703783: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05bb00 of size 256 next 117
2020-05-12 13:50:02.703792: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05bc00 of size 256 next 118
2020-05-12 13:50:02.703800: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05bd00 of size 256 next 119
2020-05-12 13:50:02.703810: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05be00 of size 256 next 120
2020-05-12 13:50:02.703817: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05bf00 of size 256 next 121
2020-05-12 13:50:02.703825: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05c000 of size 256 next 122
2020-05-12 13:50:02.703832: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05c100 of size 256 next 123
2020-05-12 13:50:02.703840: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05c200 of size 256 next 124
2020-05-12 13:50:02.703848: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05c300 of size 256 next 125
2020-05-12 13:50:02.703856: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05c400 of size 256 next 126
2020-05-12 13:50:02.703862: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05c500 of size 256 next 127
2020-05-12 13:50:02.703871: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05c600 of size 256 next 128
2020-05-12 13:50:02.703879: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05c700 of size 256 next 129
2020-05-12 13:50:02.703887: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05c800 of size 256 next 130
2020-05-12 13:50:02.703894: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05c900 of size 256 next 131
2020-05-12 13:50:02.703902: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05ca00 of size 256 next 132
2020-05-12 13:50:02.703910: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05cb00 of size 256 next 133
2020-05-12 13:50:02.703918: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05cc00 of size 256 next 134
2020-05-12 13:50:02.703926: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05cd00 of size 256 next 135
2020-05-12 13:50:02.703935: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05ce00 of size 256 next 136
2020-05-12 13:50:02.703942: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05cf00 of size 256 next 137
2020-05-12 13:50:02.703950: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05d000 of size 256 next 138
2020-05-12 13:50:02.703959: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05d100 of size 256 next 139
2020-05-12 13:50:02.703965: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05d200 of size 256 next 140
2020-05-12 13:50:02.703972: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05d300 of size 256 next 141
2020-05-12 13:50:02.703978: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05d400 of size 256 next 142
2020-05-12 13:50:02.703986: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05d500 of size 256 next 143
2020-05-12 13:50:02.703995: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05d600 of size 256 next 144
2020-05-12 13:50:02.704001: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05d700 of size 256 next 145
2020-05-12 13:50:02.704010: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05d800 of size 256 next 146
2020-05-12 13:50:02.704020: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05d900 of size 256 next 147
2020-05-12 13:50:02.704032: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05da00 of size 256 next 148
2020-05-12 13:50:02.704045: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05db00 of size 256 next 149
2020-05-12 13:50:02.704053: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05dc00 of size 256 next 150
2020-05-12 13:50:02.704063: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05dd00 of size 256 next 151
2020-05-12 13:50:02.704074: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05de00 of size 256 next 152
2020-05-12 13:50:02.704085: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05df00 of size 256 next 153
2020-05-12 13:50:02.704102: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05e000 of size 256 next 154
2020-05-12 13:50:02.704115: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05e100 of size 256 next 155
2020-05-12 13:50:02.704125: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05e200 of size 256 next 156
2020-05-12 13:50:02.704135: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05e300 of size 256 next 157
2020-05-12 13:50:02.704147: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05e400 of size 256 next 158
2020-05-12 13:50:02.704157: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05e500 of size 256 next 159
2020-05-12 13:50:02.704167: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05e600 of size 256 next 160
2020-05-12 13:50:02.704178: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05e700 of size 256 next 161
2020-05-12 13:50:02.704190: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05e800 of size 256 next 162
2020-05-12 13:50:02.704199: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05e900 of size 256 next 163
2020-05-12 13:50:02.704208: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05ea00 of size 256 next 164
2020-05-12 13:50:02.704216: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05eb00 of size 256 next 165
2020-05-12 13:50:02.704227: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05ec00 of size 256 next 166
2020-05-12 13:50:02.704235: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05ed00 of size 256 next 167
2020-05-12 13:50:02.704243: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05ee00 of size 256 next 168
2020-05-12 13:50:02.704251: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05ef00 of size 256 next 169
2020-05-12 13:50:02.704260: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05f000 of size 256 next 170
2020-05-12 13:50:02.704271: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05f100 of size 256 next 171
2020-05-12 13:50:02.704282: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05f200 of size 256 next 172
2020-05-12 13:50:02.704293: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05f300 of size 256 next 173
2020-05-12 13:50:02.704304: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05f400 of size 256 next 174
2020-05-12 13:50:02.704315: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05f500 of size 256 next 175
2020-05-12 13:50:02.704326: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05f600 of size 256 next 176
2020-05-12 13:50:02.704337: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05f700 of size 256 next 177
2020-05-12 13:50:02.704345: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05f800 of size 256 next 178
2020-05-12 13:50:02.704354: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05f900 of size 256 next 179
2020-05-12 13:50:02.704362: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05fa00 of size 256 next 180
2020-05-12 13:50:02.704370: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05fb00 of size 256 next 181
2020-05-12 13:50:02.704379: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05fc00 of size 134217728 next 182
2020-05-12 13:50:02.704387: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4f505fc00 of size 134217728 next 183
2020-05-12 13:50:02.704400: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4fd05fc00 of size 134217728 next 184
2020-05-12 13:50:02.704410: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef50505fc00 of size 134217728 next 185
2020-05-12 13:50:02.704421: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef50d05fc00 of size 134217728 next 186
2020-05-12 13:50:02.704432: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef51505fc00 of size 256 next 187
2020-05-12 13:50:02.704444: I tensorflow/core/common_runtime/bfc_allocator.cc:950] Free  at 7ef51505fd00 of size 131072 next 188
2020-05-12 13:50:02.704460: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef51507fd00 of size 1024 next 192
2020-05-12 13:50:02.704472: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef515080100 of size 256 next 193
2020-05-12 13:50:02.704482: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef515080200 of size 512 next 194
2020-05-12 13:50:02.704493: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef515080400 of size 65792 next 196
2020-05-12 13:50:02.704504: I tensorflow/core/common_runtime/bfc_allocator.cc:950] Free  at 7ef515090500 of size 12800 next 199
2020-05-12 13:50:02.704515: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef515093700 of size 1024 next 200
2020-05-12 13:50:02.704527: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef515093b00 of size 1024 next 201
2020-05-12 13:50:02.704537: I tensorflow/core/common_runtime/bfc_allocator.cc:950] Free  at 7ef515093f00 of size 211968 next 207
2020-05-12 13:50:02.704546: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef5150c7b00 of size 131072 next 208
2020-05-12 13:50:02.704554: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef5150e7b00 of size 131072 next 209
2020-05-12 13:50:02.704562: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef515107b00 of size 512 next 210
2020-05-12 13:50:02.704570: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef515107d00 of size 512 next 212
2020-05-12 13:50:02.704578: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef515107f00 of size 1280 next 213
2020-05-12 13:50:02.704588: I tensorflow/core/common_runtime/bfc_allocator.cc:950] Free  at 7ef515108400 of size 4916224 next 215
2020-05-12 13:50:02.704596: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef5155b8800 of size 512 next 216
2020-05-12 13:50:02.704604: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef5155b8a00 of size 1024 next 217
2020-05-12 13:50:02.704612: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef5155b8e00 of size 512 next 218
2020-05-12 13:50:02.704620: I tensorflow/core/common_runtime/bfc_allocator.cc:950] Free  at 7ef5155b9000 of size 6144 next 223
2020-05-12 13:50:02.704630: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef5155ba800 of size 1024 next 224
2020-05-12 13:50:02.704643: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef5155bac00 of size 1024 next 225
2020-05-12 13:50:02.704656: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef5155bb000 of size 512 next 226
2020-05-12 13:50:02.704667: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef5155bb200 of size 1024 next 304
2020-05-12 13:50:02.704676: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef5155bb600 of size 1024 next 319
2020-05-12 13:50:02.704686: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef5155bba00 of size 1024 next 283
2020-05-12 13:50:02.704694: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef5155bbe00 of size 1024 next 249
2020-05-12 13:50:02.704703: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef5155bc200 of size 1024 next 268
2020-05-12 13:50:02.704713: I tensorflow/core/common_runtime/bfc_allocator.cc:950] Free  at 7ef5155bc600 of size 1536 next 236
2020-05-12 13:50:02.704722: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef5155bcc00 of size 512 next 237
2020-05-12 13:50:02.704731: I tensorflow/core/common_runtime/bfc_allocator.cc:950] Free  at 7ef5155bce00 of size 10380800 next 259
2020-05-12 13:50:02.704740: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef515fa3400 of size 512 next 260
2020-05-12 13:50:02.704748: I tensorflow/core/common_runtime/bfc_allocator.cc:950] Free  at 7ef515fa3600 of size 398592 next 271
2020-05-12 13:50:02.704764: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef516004b00 of size 1024 next 272
2020-05-12 13:50:02.704773: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef516004f00 of size 1280 next 274
2020-05-12 13:50:02.704782: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef516005400 of size 512 next 372
2020-05-12 13:50:02.704792: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef516005600 of size 512 next 275
2020-05-12 13:50:02.704803: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef516005800 of size 512 next 277
2020-05-12 13:50:02.704812: I tensorflow/core/common_runtime/bfc_allocator.cc:950] Free  at 7ef516005a00 of size 197632 next 280
2020-05-12 13:50:02.704824: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef516035e00 of size 512 next 281
2020-05-12 13:50:02.704833: I tensorflow/core/common_runtime/bfc_allocator.cc:950] Free  at 7ef516036000 of size 280320 next 295
2020-05-12 13:50:02.704844: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef51607a700 of size 65536 next 296
2020-05-12 13:50:02.704854: I tensorflow/core/common_runtime/bfc_allocator.cc:950] Free  at 7ef51608a700 of size 135424 next 306
2020-05-12 13:50:02.704865: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef5160ab800 of size 512 next 284
2020-05-12 13:50:02.704874: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef5160aba00 of size 512 next 307
2020-05-12 13:50:02.704884: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef5160abc00 of size 65536 next 308
2020-05-12 13:50:02.704893: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef5160bbc00 of size 1024 next 324
2020-05-12 13:50:02.704904: I tensorflow/core/common_runtime/bfc_allocator.cc:950] Free  at 7ef5160bc000 of size 1024 next 343
2020-05-12 13:50:02.704914: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef5160bc400 of size 256 next 312
2020-05-12 13:50:02.704927: I tensorflow/core/common_runtime/bfc_allocator.cc:950] Free  at 7ef5160bc500 of size 1792 next 297
2020-05-12 13:50:02.704936: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef5160bcc00 of size 256 next 222
2020-05-12 13:50:02.704945: I tensorflow/core/common_runtime/bfc_allocator.cc:950] Free  at 7ef5160bcd00 of size 2304 next 278
2020-05-12 13:50:02.704954: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef5160bd600 of size 256 next 240
2020-05-12 13:50:02.704961: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef5160bd700 of size 256 next 286
2020-05-12 13:50:02.704970: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef5160bd800 of size 256 next 287
2020-05-12 13:50:02.704978: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef5160bd900 of size 256 next 393
2020-05-12 13:50:02.704986: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef5160bda00 of size 256 next 339
2020-05-12 13:50:02.704995: I tensorflow/core/common_runtime/bfc_allocator.cc:950] Free  at 7ef5160bdb00 of size 4864 next 314
2020-05-12 13:50:02.705003: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef5160bee00 of size 131072 next 315
2020-05-12 13:50:02.705012: I tensorflow/core/common_runtime/bfc_allocator.cc:950] Free  at 7ef5160dee00 of size 14336 next 316
2020-05-12 13:50:02.705020: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef5160e2600 of size 256 next 317
2020-05-12 13:50:02.705029: I tensorflow/core/common_runtime/bfc_allocator.cc:950] Free  at 7ef5160e2700 of size 17408 next 328
2020-05-12 13:50:02.705038: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef5160e6b00 of size 65536 next 329
2020-05-12 13:50:02.705047: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef5160f6b00 of size 65536 next 330
2020-05-12 13:50:02.705063: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef516106b00 of size 131072 next 331
2020-05-12 13:50:02.705077: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef516126b00 of size 512 next 333
2020-05-12 13:50:02.705088: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef516126d00 of size 256 next 232
2020-05-12 13:50:02.705098: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef516126e00 of size 256 next 334
2020-05-12 13:50:02.705108: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef516126f00 of size 256 next 205
2020-05-12 13:50:02.705119: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef516127000 of size 256 next 189
2020-05-12 13:50:02.705128: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef516127100 of size 256 next 279
2020-05-12 13:50:02.705137: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef516127200 of size 256 next 252
2020-05-12 13:50:02.705150: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef516127300 of size 256 next 335
2020-05-12 13:50:02.705161: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef516127400 of size 131072 next 336
2020-05-12 13:50:02.705171: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef516147400 of size 256 next 337
2020-05-12 13:50:02.705182: I tensorflow/core/common_runtime/bfc_allocator.cc:950] Free  at 7ef516147500 of size 4926976 next 345
2020-05-12 13:50:02.705191: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef5165fa300 of size 65536 next 346
2020-05-12 13:50:02.705199: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef51660a300 of size 512 next 347
2020-05-12 13:50:02.705210: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef51660a500 of size 1280 next 349
2020-05-12 13:50:02.705220: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef51660aa00 of size 131072 next 350
2020-05-12 13:50:02.705233: I tensorflow/core/common_runtime/bfc_allocator.cc:950] Free  at 7ef51662aa00 of size 65536 next 351
2020-05-12 13:50:02.705242: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef51663aa00 of size 512 next 353
2020-05-12 13:50:02.705255: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef51663ac00 of size 512 next 354
2020-05-12 13:50:02.705266: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef51663ae00 of size 512 next 377
2020-05-12 13:50:02.705277: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef51663b000 of size 512 next 355
2020-05-12 13:50:02.705287: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef51663b200 of size 1792 next 362
2020-05-12 13:50:02.705296: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef51663b900 of size 1024 next 363
2020-05-12 13:50:02.705306: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef51663bd00 of size 512 next 364
2020-05-12 13:50:02.705317: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef51663bf00 of size 1024 next 365
2020-05-12 13:50:02.705326: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef51663c300 of size 65536 next 366
2020-05-12 13:50:02.705337: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef51664c300 of size 512 next 367
2020-05-12 13:50:02.705347: I tensorflow/core/common_runtime/bfc_allocator.cc:950] Free  at 7ef51664c500 of size 68096 next 378
2020-05-12 13:50:02.705358: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef51665cf00 of size 65536 next 379
2020-05-12 13:50:02.705367: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef51666cf00 of size 131072 next 380
2020-05-12 13:50:02.705375: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef51668cf00 of size 1024 next 381
2020-05-12 13:50:02.705383: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef51668d300 of size 512 next 382
2020-05-12 13:50:02.705392: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef51668d500 of size 1024 next 383
2020-05-12 13:50:02.705401: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef51668d900 of size 512 next 290
2020-05-12 13:50:02.705416: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef51668db00 of size 768 next 387
2020-05-12 13:50:02.705428: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef51668de00 of size 512 next 388
2020-05-12 13:50:02.705437: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef51668e000 of size 1792 next 395
2020-05-12 13:50:02.705449: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef51668e700 of size 512 next 396
2020-05-12 13:50:02.705458: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef51668e900 of size 1024 next 397
2020-05-12 13:50:02.705466: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef51668ed00 of size 65536 next 398
2020-05-12 13:50:02.705474: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef51669ed00 of size 131072 next 399
2020-05-12 13:50:02.705486: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef5166bed00 of size 256 next 400
2020-05-12 13:50:02.705500: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef5166bee00 of size 512 next 401
2020-05-12 13:50:02.705510: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef5166bf000 of size 256 next 298
2020-05-12 13:50:02.705519: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef5166bf100 of size 256 next 198
2020-05-12 13:50:02.705528: I tensorflow/core/common_runtime/bfc_allocator.cc:950] Free  at 7ef5166bf200 of size 512 next 402
2020-05-12 13:50:02.705538: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef5166bf400 of size 1792 next 409
2020-05-12 13:50:02.705546: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef5166bfb00 of size 131072 next 410
2020-05-12 13:50:02.705555: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef5166dfb00 of size 1024 next 411
2020-05-12 13:50:02.705563: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef5166dff00 of size 512 next 412
2020-05-12 13:50:02.705572: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef5166e0100 of size 1536 next 413
2020-05-12 13:50:02.705581: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef5166e0700 of size 512 next 414
2020-05-12 13:50:02.705597: I tensorflow/core/common_runtime/bfc_allocator.cc:950] Free  at 7ef5166e0900 of size 66048 next 416
2020-05-12 13:50:02.705606: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef5166f0b00 of size 131072 next 417
2020-05-12 13:50:02.705614: I tensorflow/core/common_runtime/bfc_allocator.cc:950] Free  at 7ef516710b00 of size 16580608 next 253
2020-05-12 13:50:02.705623: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef5176e0b00 of size 2097152 next 256
2020-05-12 13:50:02.705634: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef5178e0b00 of size 2097152 next 250
2020-05-12 13:50:02.705644: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef517ae0b00 of size 2097152 next 348
2020-05-12 13:50:02.705654: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef517ce0b00 of size 2097152 next 230
2020-05-12 13:50:02.705664: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef517ee0b00 of size 2097152 next 390
2020-05-12 13:50:02.705674: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef5180e0b00 of size 2097152 next 326
2020-05-12 13:50:02.705682: I tensorflow/core/common_runtime/bfc_allocator.cc:950] Free  at 7ef5182e0b00 of size 3129344 next 276
2020-05-12 13:50:02.705692: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef5185dcb00 of size 2359296 next 288
2020-05-12 13:50:02.705703: I tensorflow/core/common_runtime/bfc_allocator.cc:950] Free  at 7ef51881cb00 of size 1228800 next 415
2020-05-12 13:50:02.705714: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef518948b00 of size 2097152 next 263
2020-05-12 13:50:02.705724: I tensorflow/core/common_runtime/bfc_allocator.cc:950] Free  at 7ef518b48b00 of size 2457600 next 384
2020-05-12 13:50:02.705735: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef518da0b00 of size 134217728 next 231
2020-05-12 13:50:02.705745: I tensorflow/core/common_runtime/bfc_allocator.cc:950] Free  at 7ef520da0b00 of size 4915200 next 221
2020-05-12 13:50:02.705756: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef521250b00 of size 2097152 next 332
2020-05-12 13:50:02.705767: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef521450b00 of size 2097152 next 195
2020-05-12 13:50:02.705779: I tensorflow/core/common_runtime/bfc_allocator.cc:950] Free  at 7ef521650b00 of size 29360128 next 227
2020-05-12 13:50:02.705790: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef523250b00 of size 134217728 next 293
2020-05-12 13:50:02.705800: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef52b250b00 of size 134217728 next 294
2020-05-12 13:50:02.705811: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef533250b00 of size 268435456 next 257
2020-05-12 13:50:02.705822: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef543250b00 of size 268435456 next 289
2020-05-12 13:50:02.705832: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef553250b00 of size 268435456 next 299
2020-05-12 13:50:02.705844: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef563250b00 of size 536870912 next 318
2020-05-12 13:50:02.705853: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef583250b00 of size 536870912 next 251
2020-05-12 13:50:02.705864: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef5a3250b00 of size 536870912 next 261
2020-05-12 13:50:02.705873: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef5c3250b00 of size 268435456 next 291
2020-05-12 13:50:02.705886: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef5d3250b00 of size 268435456 next 247
2020-05-12 13:50:02.705895: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef5e3250b00 of size 536870912 next 220
2020-05-12 13:50:02.705906: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef603250b00 of size 536870912 next 235
2020-05-12 13:50:02.705915: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef623250b00 of size 536870912 next 369
2020-05-12 13:50:02.705923: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef643250b00 of size 134217728 next 320
2020-05-12 13:50:02.705932: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef64b250b00 of size 134217728 next 321
2020-05-12 13:50:02.705940: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef653250b00 of size 134217728 next 309
2020-05-12 13:50:02.705952: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef65b250b00 of size 268435456 next 403
2020-05-12 13:50:02.705961: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef66b250b00 of size 268435456 next 404
2020-05-12 13:50:02.705970: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef67b250b00 of size 268435456 next 389
2020-05-12 13:50:02.705978: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef68b250b00 of size 536870912 next 373
2020-05-12 13:50:02.705986: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef6ab250b00 of size 536870912 next 356
2020-05-12 13:50:02.705996: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef6cb250b00 of size 536870912 next 357
2020-05-12 13:50:02.706005: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef6eb250b00 of size 134217728 next 371
2020-05-12 13:50:02.706015: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef6f3250b00 of size 134217728 next 303
2020-05-12 13:50:02.706024: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef6fb250b00 of size 268435456 next 386
2020-05-12 13:50:02.706033: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef70b250b00 of size 268435456 next 370
2020-05-12 13:50:02.706045: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef71b250b00 of size 268435456 next 352
2020-05-12 13:50:02.706054: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef72b250b00 of size 536870912 next 385
2020-05-12 13:50:02.706060: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef74b250b00 of size 536870912 next 302
2020-05-12 13:50:02.706071: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef76b250b00 of size 536870912 next 267
2020-05-12 13:50:02.706081: I tensorflow/core/common_runtime/bfc_allocator.cc:950] Free  at 7ef78b250b00 of size 33554432 next 203
2020-05-12 13:50:02.706090: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef78d250b00 of size 134217728 next 340
2020-05-12 13:50:02.706099: I tensorflow/core/common_runtime/bfc_allocator.cc:950] Free  at 7ef795250b00 of size 134217728 next 344
2020-05-12 13:50:02.706109: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef79d250b00 of size 268435456 next 202
2020-05-12 13:50:02.706117: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef7ad250b00 of size 268435456 next 191
2020-05-12 13:50:02.706127: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef7bd250b00 of size 268435456 next 408
2020-05-12 13:50:02.706136: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef7cd250b00 of size 536870912 next 305
2020-05-12 13:50:02.706148: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef7ed250b00 of size 536870912 next 394
2020-05-12 13:50:02.706158: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef80d250b00 of size 536870912 next 361
2020-05-12 13:50:02.706169: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef82d250b00 of size 134217728 next 197
2020-05-12 13:50:02.706226: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef835250b00 of size 134217728 next 376
2020-05-12 13:50:02.706239: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef83d250b00 of size 134217728 next 338
2020-05-12 13:50:02.706251: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef845250b00 of size 134217728 next 325
2020-05-12 13:50:02.706264: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef84d250b00 of size 134217728 next 368
2020-05-12 13:50:02.706281: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef855250b00 of size 536870912 next 238
2020-05-12 13:50:02.706296: I tensorflow/core/common_runtime/bfc_allocator.cc:950] Free  at 7ef875250b00 of size 516896000 next 18446744073709551615
2020-05-12 13:50:02.706307: I tensorflow/core/common_runtime/bfc_allocator.cc:955]      Summary of in-use Chunks by size: 
2020-05-12 13:50:02.706322: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 176 Chunks of size 256 totalling 44.0KiB
2020-05-12 13:50:02.706335: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 30 Chunks of size 512 totalling 15.0KiB
2020-05-12 13:50:02.706349: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 1 Chunks of size 768 totalling 768B
2020-05-12 13:50:02.706363: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 20 Chunks of size 1024 totalling 20.0KiB
2020-05-12 13:50:02.706373: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 4 Chunks of size 1280 totalling 5.0KiB
2020-05-12 13:50:02.706382: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 2 Chunks of size 1536 totalling 3.0KiB
2020-05-12 13:50:02.706390: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 3 Chunks of size 1792 totalling 5.2KiB
2020-05-12 13:50:02.706399: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 1 Chunks of size 4096 totalling 4.0KiB
2020-05-12 13:50:02.706408: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 8 Chunks of size 65536 totalling 512.0KiB
2020-05-12 13:50:02.706416: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 1 Chunks of size 65792 totalling 64.2KiB
2020-05-12 13:50:02.706427: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 10 Chunks of size 131072 totalling 1.25MiB
2020-05-12 13:50:02.706435: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 12 Chunks of size 262144 totalling 3.00MiB
2020-05-12 13:50:02.706444: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 2 Chunks of size 409600 totalling 800.0KiB
2020-05-12 13:50:02.706452: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 2 Chunks of size 1228800 totalling 2.34MiB
2020-05-12 13:50:02.706461: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 3 Chunks of size 2048000 totalling 5.86MiB
2020-05-12 13:50:02.706471: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 10 Chunks of size 2097152 totalling 20.00MiB
2020-05-12 13:50:02.706479: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 1 Chunks of size 2359296 totalling 2.25MiB
2020-05-12 13:50:02.706488: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 1 Chunks of size 2457600 totalling 2.34MiB
2020-05-12 13:50:02.706502: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 19 Chunks of size 134217728 totalling 2.38GiB
2020-05-12 13:50:02.706509: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 14 Chunks of size 268435456 totalling 3.50GiB
2020-05-12 13:50:02.706522: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 16 Chunks of size 536870912 totalling 8.00GiB
2020-05-12 13:50:02.706533: I tensorflow/core/common_runtime/bfc_allocator.cc:962] Sum Total of in-use chunks: 13.91GiB
2020-05-12 13:50:02.706542: I tensorflow/core/common_runtime/bfc_allocator.cc:964] total_region_allocated_bytes_: 15702704128 memory_limit_: 15702704128 available bytes: 0 curr_region_allocation_bytes_: 31405408256
2020-05-12 13:50:02.706560: I tensorflow/core/common_runtime/bfc_allocator.cc:970] Stats: 
Limit:                 15702704128
InUse:                 14938522880
MaxInUse:              15074972928
NumAllocs:                     946
MaxAllocSize:            536870912

2020-05-12 13:50:02.706595: W tensorflow/core/common_runtime/bfc_allocator.cc:429] *************************************************************************************************___
2020-05-12 13:50:02.706629: W tensorflow/core/framework/op_kernel.cc:1655] OP_REQUIRES failed at cwise_ops_common.h:263 : Resource exhausted: OOM when allocating tensor with shape[1,1024,512,256] and type float on /job:local/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
2020-05-12 13:50:10.736175: I lingvo/core/ops/record_yielder.cc:407] 0x7ef8a34d9f20Basic record yielder exit
E0512 13:50:11.172914 139606888535808 base_runner.py:244] trainer done (fatal error): <class 'tensorflow.python.framework.errors_impl.ResourceExhaustedError'>
I0512 13:50:11.175342 139606888535808 base_runner.py:111] trainer exception: From /job:local/replica:0/task:0:
OOM when allocating tensor with shape[1,1024,512,256] and type float on /job:local/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
         [[node gradients/fprop/starnet/tower_0_0/feat/validated_gin/gin/gin_concat/seq/gin_concat/seq/gin_concat/seq/gin_concat/seq/gin_concat/seq/validated_gin_intermediate/gin_intermediate/validated_mlp/mlp/validated_validated_l001/validated_l001/l001/key_seq/bn/bn/sub_2_grad/Neg (defined at tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/core/py_utils.py:1923) ]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.


Original stack trace for 'gradients/fprop/starnet/tower_0_0/feat/validated_gin/gin/gin_concat/seq/gin_concat/seq/gin_concat/seq/gin_concat/seq/gin_concat/seq/validated_gin_intermediate/gin_intermediate/validated_mlp/mlp/validated_validated_l001/validated_l001/l001/key_seq/bn/bn/sub_2_grad/Neg':
  File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/trainer.py", line 1874, in <module>
    tf.app.run(main)
  File "usr/local/lib/python3.6/dist-packages/tensorflow_core/python/platform/app.py", line 40, in run
    _run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef)
  File "usr/local/lib/python3.6/dist-packages/absl/app.py", line 299, in run
    _run_main(main, args)
  File "usr/local/lib/python3.6/dist-packages/absl/app.py", line 250, in _run_main
    sys.exit(main(argv))
  File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/trainer.py", line 1865, in main
    RunnerManager(FLAGS.model).Start()
  File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/trainer.py", line 1861, in Start
    self.StartRunners(self.CreateRunners(FLAGS.job.split(','), FLAGS.logdir))
  File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/trainer.py", line 1609, in CreateRunners
    trial)
  File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/trainer.py", line 1562, in _CreateRunner
    return self.Trainer(cfg, *common_args)
  File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/trainer.py", line 398, in __init__
    self._model.ConstructFPropBPropGraph()
  File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/core/base_model.py", line 1197, in ConstructFPropBPropGraph
    self._task.BProp()
  File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/core/base_model.py", line 538, in BProp
    self._BPropForVariables(self.vars)
  File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/core/base_model.py", line 568, in _BPropForVariables
    gradient_adjuster=self.AdjustGradients)
  File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/core/learner.py", line 189, in Apply
    skip_zero_gradients=p.skip_zero_gradients)
  File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/core/optimizer.py", line 54, in ComputeGradients
    return py_utils.ComputeGradients(loss, vmap, *args, **kwargs)
  File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/core/py_utils.py", line 2143, in ComputeGradients
    colocate_gradients_with_ops, gate_gradients)
  File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/core/py_utils.py", line 1923, in ComputeGradientsSimple
    gate_gradients=gate_gradients)
  File "usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/gradients_impl.py", line 158, in gradients
    unconnected_gradients)
  File "usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/gradients_util.py", line 669, in _GradientsHelper
    lambda: grad_fn(op, *out_grads))
  File "usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/gradients_util.py", line 336, in _MaybeCompile
    return grad_fn()  # Exit early
  File "usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/gradients_util.py", line 669, in <lambda>
    lambda: grad_fn(op, *out_grads))
  File "usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/math_grad.py", line 1201, in _SubGrad
    gy = array_ops.reshape(math_ops.reduce_sum(-grad, ry), sy)
  File "usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/gen_math_ops.py", line 6304, in neg
    "Neg", x=x, name=name)
  File "usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/op_def_library.py", line 742, in _apply_op_helper
    attrs=attr_protos, op_def=op_def)
  File "usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/ops.py", line 3322, in _create_op_internal
    op_def=op_def)
  File "usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/ops.py", line 1756, in __init__
    self._traceback = tf_stack.extract_stack()

...which was originally created as op 'fprop/starnet/tower_0_0/feat/validated_gin/gin/gin_concat/seq/gin_concat/seq/gin_concat/seq/gin_concat/seq/gin_concat/seq/validated_gin_intermediate/gin_intermediate/validated_mlp/mlp/validated_validated_l001/validated_l001/l001/key_seq/bn/bn/sub_2', defined at:
  File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/trainer.py", line 1874, in <module>
    tf.app.run(main)
[elided 7 identical lines from previous traceback]
  File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/trainer.py", line 398, in __init__
    self._model.ConstructFPropBPropGraph()
  File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/core/base_model.py", line 1196, in ConstructFPropBPropGraph
    self._task.FPropDefaultTheta()
  File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/core/base_model.py", line 524, in FPropDefaultTheta
    return self.FProp(self.theta, input_batch)
  File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/core/base_model.py", line 445, in FProp
    metrics, per_example = self._FPropSplitInputBatch(theta, input_batch)
  File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/core/base_model.py", line 491, in _FPropSplitInputBatch
    metrics, per_example = self.FPropTower(theta_local, batch)
  File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/core/base_model.py", line 413, in FPropTower
    predictions = self.ComputePredictions(theta, input_batch)
  File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/tasks/car/starnet.py", line 856, in ComputePredictions
    featurized_cell = self._CellFeaturizer(theta, input_batch)
  File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/tasks/car/starnet.py", line 831, in _CellFeaturizer
    point_input)
  File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/core/builder_layers.py", line 422, in FProp
    args = ch.FProp(th, *args)
  File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/core/builder_layers.py", line 422, in FProp
    args = ch.FProp(th, *args)
  File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/core/builder_layers.py", line 422, in FProp
    args = ch.FProp(th, *args)
  File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/core/builder_layers.py", line 938, in FProp
    out = ch.FProp(th, *args)
  File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/core/builder_layers.py", line 422, in FProp
    args = ch.FProp(th, *args)
  File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/core/builder_layers.py", line 938, in FProp
    out = ch.FProp(th, *args)
  File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/core/builder_layers.py", line 422, in FProp
    args = ch.FProp(th, *args)
  File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/core/builder_layers.py", line 938, in FProp
    out = ch.FProp(th, *args)
  File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/core/builder_layers.py", line 422, in FProp
    args = ch.FProp(th, *args)


E0512 13:50:11.177675 139606888535808 base_runner.py:251] Traceback (most recent call last):
E0512 13:50:11.177758 139606888535808 base_runner.py:251]   File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py", line 1367, in _do_call
E0512 13:50:11.177819 139606888535808 base_runner.py:251]     return fn(*args)
E0512 13:50:11.177872 139606888535808 base_runner.py:251]   File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py", line 1352, in _run_fn
E0512 13:50:11.177922 139606888535808 base_runner.py:251]     target_list, run_metadata)
E0512 13:50:11.177970 139606888535808 base_runner.py:251]   File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py", line 1445, in _call_tf_sessionrun
E0512 13:50:11.178018 139606888535808 base_runner.py:251]     run_metadata)
E0512 13:50:11.178066 139606888535808 base_runner.py:251] tensorflow.python.framework.errors_impl.ResourceExhaustedError: From /job:local/replica:0/task:0:
E0512 13:50:11.178113 139606888535808 base_runner.py:251] OOM when allocating tensor with shape[1,1024,512,256] and type float on /job:local/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
E0512 13:50:11.178161 139606888535808 base_runner.py:251]        [[{{node gradients/fprop/starnet/tower_0_0/feat/validated_gin/gin/gin_concat/seq/gin_concat/seq/gin_concat/seq/gin_concat/seq/gin_concat/seq/validated_gin_intermediate/gin_intermediate/validated_mlp/mlp/validated_validated_l001/validated_l001/l001/key_seq/bn/bn/sub_2_grad/Neg}}]]
E0512 13:50:11.178250 139606888535808 base_runner.py:251] Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
E0512 13:50:11.178305 139606888535808 base_runner.py:251] 
E0512 13:50:11.178353 139606888535808 base_runner.py:251] 
E0512 13:50:11.178400 139606888535808 base_runner.py:251] During handling of the above exception, another exception occurred:
E0512 13:50:11.178450 139606888535808 base_runner.py:251] 
E0512 13:50:11.178497 139606888535808 base_runner.py:251] Traceback (most recent call last):
E0512 13:50:11.178543 139606888535808 base_runner.py:251]   File "/tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/base_runner.py", line 193, in _RunLoop
E0512 13:50:11.178590 139606888535808 base_runner.py:251]     loop_func(*loop_args)
E0512 13:50:11.178636 139606888535808 base_runner.py:251]   File "/tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/trainer.py", line 504, in _Loop
E0512 13:50:11.178682 139606888535808 base_runner.py:251]     model_task.per_example_tensors,
E0512 13:50:11.178728 139606888535808 base_runner.py:251]   File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py", line 960, in run
E0512 13:50:11.178775 139606888535808 base_runner.py:251]     run_metadata_ptr)
E0512 13:50:11.178820 139606888535808 base_runner.py:251]   File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py", line 1183, in _run
E0512 13:50:11.178866 139606888535808 base_runner.py:251]     feed_dict_tensor, options, run_metadata)
E0512 13:50:11.178911 139606888535808 base_runner.py:251]   File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py", line 1361, in _do_run
E0512 13:50:11.178961 139606888535808 base_runner.py:251]     run_metadata)
E0512 13:50:11.179010 139606888535808 base_runner.py:251]   File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py", line 1386, in _do_call
E0512 13:50:11.179056 139606888535808 base_runner.py:251]     raise type(e)(node_def, op, message)
E0512 13:50:11.179102 139606888535808 base_runner.py:251] tensorflow.python.framework.errors_impl.ResourceExhaustedError: From /job:local/replica:0/task:0:
E0512 13:50:11.179148 139606888535808 base_runner.py:251] OOM when allocating tensor with shape[1,1024,512,256] and type float on /job:local/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
E0512 13:50:11.179193 139606888535808 base_runner.py:251]        [[node gradients/fprop/starnet/tower_0_0/feat/validated_gin/gin/gin_concat/seq/gin_concat/seq/gin_concat/seq/gin_concat/seq/gin_concat/seq/validated_gin_intermediate/gin_intermediate/validated_mlp/mlp/validated_validated_l001/validated_l001/l001/key_seq/bn/bn/sub_2_grad/Neg (defined at tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/core/py_utils.py:1923) ]]
E0512 13:50:11.179267 139606888535808 base_runner.py:251] Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
E0512 13:50:11.179316 139606888535808 base_runner.py:251] 
E0512 13:50:11.179363 139606888535808 base_runner.py:251] 
E0512 13:50:11.179409 139606888535808 base_runner.py:251] Original stack trace for 'gradients/fprop/starnet/tower_0_0/feat/validated_gin/gin/gin_concat/seq/gin_concat/seq/gin_concat/seq/gin_concat/seq/gin_concat/seq/validated_gin_intermediate/gin_intermediate/validated_mlp/mlp/validated_validated_l001/validated_l001/l001/key_seq/bn/bn/sub_2_grad/Neg':
E0512 13:50:11.179456 139606888535808 base_runner.py:251]   File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/trainer.py", line 1874, in <module>
E0512 13:50:11.179502 139606888535808 base_runner.py:251]     tf.app.run(main)
E0512 13:50:11.179549 139606888535808 base_runner.py:251]   File "usr/local/lib/python3.6/dist-packages/tensorflow_core/python/platform/app.py", line 40, in run
E0512 13:50:11.179595 139606888535808 base_runner.py:251]     _run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef)
E0512 13:50:11.179647 139606888535808 base_runner.py:251]   File "usr/local/lib/python3.6/dist-packages/absl/app.py", line 299, in run
E0512 13:50:11.179695 139606888535808 base_runner.py:251]     _run_main(main, args)
E0512 13:50:11.179741 139606888535808 base_runner.py:251]   File "usr/local/lib/python3.6/dist-packages/absl/app.py", line 250, in _run_main
E0512 13:50:11.179788 139606888535808 base_runner.py:251]     sys.exit(main(argv))
E0512 13:50:11.179834 139606888535808 base_runner.py:251]   File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/trainer.py", line 1865, in main
E0512 13:50:11.179880 139606888535808 base_runner.py:251]     RunnerManager(FLAGS.model).Start()
E0512 13:50:11.179926 139606888535808 base_runner.py:251]   File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/trainer.py", line 1861, in Start
E0512 13:50:11.179972 139606888535808 base_runner.py:251]     self.StartRunners(self.CreateRunners(FLAGS.job.split(','), FLAGS.logdir))
E0512 13:50:11.180018 139606888535808 base_runner.py:251]   File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/trainer.py", line 1609, in CreateRunners
E0512 13:50:11.180064 139606888535808 base_runner.py:251]     trial)
E0512 13:50:11.180110 139606888535808 base_runner.py:251]   File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/trainer.py", line 1562, in _CreateRunner
E0512 13:50:11.180155 139606888535808 base_runner.py:251]     return self.Trainer(cfg, *common_args)
E0512 13:50:11.180201 139606888535808 base_runner.py:251]   File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/trainer.py", line 398, in __init__
E0512 13:50:11.180247 139606888535808 base_runner.py:251]     self._model.ConstructFPropBPropGraph()
E0512 13:50:11.180293 139606888535808 base_runner.py:251]   File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/core/base_model.py", line 1197, in ConstructFPropBPropGraph
E0512 13:50:11.180339 139606888535808 base_runner.py:251]     self._task.BProp()
E0512 13:50:11.180384 139606888535808 base_runner.py:251]   File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/core/base_model.py", line 538, in BProp
E0512 13:50:11.180430 139606888535808 base_runner.py:251]     self._BPropForVariables(self.vars)
E0512 13:50:11.180479 139606888535808 base_runner.py:251]   File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/core/base_model.py", line 568, in _BPropForVariables
E0512 13:50:11.180526 139606888535808 base_runner.py:251]     gradient_adjuster=self.AdjustGradients)
E0512 13:50:11.180572 139606888535808 base_runner.py:251]   File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/core/learner.py", line 189, in Apply
E0512 13:50:11.180617 139606888535808 base_runner.py:251]     skip_zero_gradients=p.skip_zero_gradients)
E0512 13:50:11.180663 139606888535808 base_runner.py:251]   File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/core/optimizer.py", line 54, in ComputeGradients
E0512 13:50:11.180709 139606888535808 base_runner.py:251]     return py_utils.ComputeGradients(loss, vmap, *args, **kwargs)
E0512 13:50:11.180755 139606888535808 base_runner.py:251]   File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/core/py_utils.py", line 2143, in ComputeGradients
E0512 13:50:11.180800 139606888535808 base_runner.py:251]     colocate_gradients_with_ops, gate_gradients)
E0512 13:50:11.180846 139606888535808 base_runner.py:251]   File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/core/py_utils.py", line 1923, in ComputeGradientsSimple
E0512 13:50:11.180892 139606888535808 base_runner.py:251]     gate_gradients=gate_gradients)
E0512 13:50:11.180938 139606888535808 base_runner.py:251]   File "usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/gradients_impl.py", line 158, in gradients
E0512 13:50:11.180983 139606888535808 base_runner.py:251]     unconnected_gradients)
E0512 13:50:11.181029 139606888535808 base_runner.py:251]   File "usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/gradients_util.py", line 669, in _GradientsHelper
E0512 13:50:11.181074 139606888535808 base_runner.py:251]     lambda: grad_fn(op, *out_grads))
E0512 13:50:11.181120 139606888535808 base_runner.py:251]   File "usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/gradients_util.py", line 336, in _MaybeCompile
E0512 13:50:11.181165 139606888535808 base_runner.py:251]     return grad_fn()  # Exit early
E0512 13:50:11.181211 139606888535808 base_runner.py:251]   File "usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/gradients_util.py", line 669, in <lambda>
E0512 13:50:11.181257 139606888535808 base_runner.py:251]     lambda: grad_fn(op, *out_grads))
E0512 13:50:11.181302 139606888535808 base_runner.py:251]   File "usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/math_grad.py", line 1201, in _SubGrad
E0512 13:50:11.181348 139606888535808 base_runner.py:251]     gy = array_ops.reshape(math_ops.reduce_sum(-grad, ry), sy)
E0512 13:50:11.181395 139606888535808 base_runner.py:251]   File "usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/gen_math_ops.py", line 6304, in neg
E0512 13:50:11.181441 139606888535808 base_runner.py:251]     "Neg", x=x, name=name)
E0512 13:50:11.181487 139606888535808 base_runner.py:251]   File "usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/op_def_library.py", line 742, in _apply_op_helper
E0512 13:50:11.181534 139606888535808 base_runner.py:251]     attrs=attr_protos, op_def=op_def)
E0512 13:50:11.181580 139606888535808 base_runner.py:251]   File "usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/ops.py", line 3322, in _create_op_internal
E0512 13:50:11.181626 139606888535808 base_runner.py:251]     op_def=op_def)
E0512 13:50:11.181672 139606888535808 base_runner.py:251]   File "usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/ops.py", line 1756, in __init__
E0512 13:50:11.181717 139606888535808 base_runner.py:251]     self._traceback = tf_stack.extract_stack()
E0512 13:50:11.181763 139606888535808 base_runner.py:251] 
E0512 13:50:11.181808 139606888535808 base_runner.py:251] ...which was originally created as op 'fprop/starnet/tower_0_0/feat/validated_gin/gin/gin_concat/seq/gin_concat/seq/gin_concat/seq/gin_concat/seq/gin_concat/seq/validated_gin_intermediate/gin_intermediate/validated_mlp/mlp/validated_validated_l001/validated_l001/l001/key_seq/bn/bn/sub_2', defined at:
E0512 13:50:11.181856 139606888535808 base_runner.py:251]   File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/trainer.py", line 1874, in <module>
E0512 13:50:11.181903 139606888535808 base_runner.py:251]     tf.app.run(main)
E0512 13:50:11.181948 139606888535808 base_runner.py:251] [elided 7 identical lines from previous traceback]
E0512 13:50:11.181994 139606888535808 base_runner.py:251]   File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/trainer.py", line 398, in __init__
E0512 13:50:11.182041 139606888535808 base_runner.py:251]     self._model.ConstructFPropBPropGraph()
E0512 13:50:11.182086 139606888535808 base_runner.py:251]   File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/core/base_model.py", line 1196, in ConstructFPropBPropGraph
E0512 13:50:11.182132 139606888535808 base_runner.py:251]     self._task.FPropDefaultTheta()
E0512 13:50:11.182210 139606888535808 base_runner.py:251]   File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/core/base_model.py", line 524, in FPropDefaultTheta
E0512 13:50:11.182261 139606888535808 base_runner.py:251]     return self.FProp(self.theta, input_batch)
E0512 13:50:11.182308 139606888535808 base_runner.py:251]   File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/core/base_model.py", line 445, in FProp
E0512 13:50:11.182354 139606888535808 base_runner.py:251]     metrics, per_example = self._FPropSplitInputBatch(theta, input_batch)
E0512 13:50:11.182400 139606888535808 base_runner.py:251]   File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/core/base_model.py", line 491, in _FPropSplitInputBatch
E0512 13:50:11.182446 139606888535808 base_runner.py:251]     metrics, per_example = self.FPropTower(theta_local, batch)
E0512 13:50:11.182492 139606888535808 base_runner.py:251]   File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/core/base_model.py", line 413, in FPropTower
E0512 13:50:11.182537 139606888535808 base_runner.py:251]     predictions = self.ComputePredictions(theta, input_batch)
E0512 13:50:11.182582 139606888535808 base_runner.py:251]   File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/tasks/car/starnet.py", line 856, in ComputePredictions
E0512 13:50:11.182628 139606888535808 base_runner.py:251]     featurized_cell = self._CellFeaturizer(theta, input_batch)
E0512 13:50:11.182672 139606888535808 base_runner.py:251]   File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/tasks/car/starnet.py", line 831, in _CellFeaturizer
E0512 13:50:11.182718 139606888535808 base_runner.py:251]     point_input)
E0512 13:50:11.182763 139606888535808 base_runner.py:251]   File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/core/builder_layers.py", line 422, in FProp
E0512 13:50:11.182809 139606888535808 base_runner.py:251]     args = ch.FProp(th, *args)
E0512 13:50:11.182855 139606888535808 base_runner.py:251]   File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/core/builder_layers.py", line 422, in FProp
E0512 13:50:11.182900 139606888535808 base_runner.py:251]     args = ch.FProp(th, *args)
E0512 13:50:11.182945 139606888535808 base_runner.py:251]   File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/core/builder_layers.py", line 422, in FProp
E0512 13:50:11.182990 139606888535808 base_runner.py:251]     args = ch.FProp(th, *args)
E0512 13:50:11.183035 139606888535808 base_runner.py:251]   File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/core/builder_layers.py", line 938, in FProp
E0512 13:50:11.183080 139606888535808 base_runner.py:251]     out = ch.FProp(th, *args)
E0512 13:50:11.183126 139606888535808 base_runner.py:251]   File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/core/builder_layers.py", line 422, in FProp
E0512 13:50:11.183171 139606888535808 base_runner.py:251]     args = ch.FProp(th, *args)
E0512 13:50:11.183217 139606888535808 base_runner.py:251]   File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/core/builder_layers.py", line 938, in FProp
E0512 13:50:11.183262 139606888535808 base_runner.py:251]     out = ch.FProp(th, *args)
E0512 13:50:11.183307 139606888535808 base_runner.py:251]   File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/core/builder_layers.py", line 422, in FProp
E0512 13:50:11.183352 139606888535808 base_runner.py:251]     args = ch.FProp(th, *args)
E0512 13:50:11.183397 139606888535808 base_runner.py:251]   File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/core/builder_layers.py", line 938, in FProp
E0512 13:50:11.183443 139606888535808 base_runner.py:251]     out = ch.FProp(th, *args)
E0512 13:50:11.183488 139606888535808 base_runner.py:251]   File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/core/builder_layers.py", line 422, in FProp
E0512 13:50:11.183534 139606888535808 base_runner.py:251]     args = ch.FProp(th, *args)

Any pointers much appreciated.

Thanks!

opened by JWHennessey 9

generate_waymo_tf.py hangs

Hi,

Thanks for open sourcing such a great library.

I am trying to convert some of the Waymo data tfrecord data into the tf-example format used my lingvo. The data hosted at waymo_open_dataset_v_1_0_0_tf_example_lingvo/v.1.0.0/ doesn't include the data for the domain adaption subset as far as I can tell.

I've tried running the generate_waymo_tf.py script both with the DataflowRunner and DirectRunner. Both seem to end up hanging and never generating any outputs.

Are there any suggestions on how to debug this?

Here is how I run the script which should be trying to process only a single segment.

python3 /tmp/lingvo/lingvo/tasks/car/waymo/tools/generate_waymo_tf.py \
  --input_file_pattern=gs://{bucket}/data/domain_adaptation/validation/segment-10302420246752500683_5520_000_5540_000* \
  --output_filebase=gs://{bucket}/tf_example_lingvo/domain_adaptation/validation/output -- \
  --project='{project}' \
  --temp_location=gs://{bucket}/tmp/ \
  --region 'europe-west4' \
  --zone 'europe-west4-c' \
  --num_workers 2 \
  --setup_file /tmp/lingvo/lingvo/tasks/car/waymo/tools/setup.py

The outputs:

2020-05-08 16:06:49.830048: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libnvinfer.so.6'; dlerror: libnvinfer.so.6: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/nvidia/lib:/usr/local/nvidia/lib64
2020-05-08 16:06:49.830188: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libnvinfer_plugin.so.6'; dlerror: libnvinfer_plugin.so.6: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/nvidia/lib:/usr/local/nvidia/lib64
2020-05-08 16:06:49.830222: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:30] Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
WARNING:absl:Lingvo does not support eager execution yet. Please disable eager execution with tf.compat.v1.disable_eager_execution() or proceed at your own risk.
I0508 16:06:50.560759 139933516105536 auth.py:108] Setting socket default timeout to 60 seconds.
I0508 16:06:50.560907 139933516105536 auth.py:111] socket default timeout is 60.0 seconds.
I0508 16:06:50.567577 139933516105536 gcsio.py:480] Starting the size estimation of the input
I0508 16:06:50.568466 139933516105536 transport.py:151] Attempting refresh to obtain initial access_token
I0508 16:06:50.639814 139933516105536 gcsio.py:496] Finished listing 1 files in 0.07223939895629883 seconds.
I0508 16:06:50.640702 139933516105536 pipeline.py:180] Missing pipeline option (runner). Executing pipeline using the default runner: DirectRunner.
I0508 16:06:51.375959 139933516105536 fn_api_runner_transforms.py:548] ==================== <function annotate_downstream_side_inputs at 0x7f445eff9048> ====================
I0508 16:06:51.376795 139933516105536 fn_api_runner_transforms.py:548] ==================== <function fix_side_input_pcoll_coders at 0x7f445eff9158> ====================
I0508 16:06:51.377469 139933516105536 fn_api_runner_transforms.py:548] ==================== <function lift_combiners at 0x7f445eff91e0> ====================
I0508 16:06:51.377765 139933516105536 fn_api_runner_transforms.py:548] ==================== <function expand_sdf at 0x7f445eff9268> ====================
I0508 16:06:51.378669 139933516105536 fn_api_runner_transforms.py:548] ==================== <function expand_gbk at 0x7f445eff92f0> ====================
I0508 16:06:51.379250 139933516105536 fn_api_runner_transforms.py:548] ==================== <function sink_flattens at 0x7f445eff9400> ====================
I0508 16:06:51.379594 139933516105536 fn_api_runner_transforms.py:548] ==================== <function greedily_fuse at 0x7f445eff9488> ====================
I0508 16:06:51.381523 139933516105536 fn_api_runner_transforms.py:548] ==================== <function read_to_impulse at 0x7f445eff9510> ====================
I0508 16:06:51.381770 139933516105536 fn_api_runner_transforms.py:548] ==================== <function impulse_to_input at 0x7f445eff9598> ====================
I0508 16:06:51.382106 139933516105536 fn_api_runner_transforms.py:548] ==================== <function inject_timer_pcollections at 0x7f445eff9730> ====================
I0508 16:06:51.382540 139933516105536 fn_api_runner_transforms.py:548] ==================== <function sort_stages at 0x7f445eff97b8> ====================
I0508 16:06:51.382743 139933516105536 fn_api_runner_transforms.py:548] ==================== <function window_pcollection_coders at 0x7f445eff9840> ====================
I0508 16:06:51.384798 139933516105536 statecache.py:154] Creating state cache with size 100
I0508 16:06:51.386486 139933516105536 fn_api_runner.py:2011] Created Worker handler <apache_beam.runners.portability.fn_api_runner.EmbeddedWorkerHandler object at 0x7f445ea61898> for environment urn: "beam:env:embedded_python:v1"

I0508 16:06:51.386961 139933516105536 fn_api_runner.py:974] Running (((ref_AppliedPTransform_Read/Read/_SDFBoundedSourceWrapper/Impulse_5)+(Read/Read/_SDFBoundedSourceWrapper/ParDo(SDFBoundedSourceDoFn)/PairWithRestriction))+(Read/Read/_SDFBoundedSourceWrapper/ParDo(SDFBoundedSourceDoFn)/SplitAndSizeRestriction))+(ref_PCollection_PCollection_1_split/Write)
I0508 16:06:51.425244 139933369054976 gcsio.py:480] Starting the size estimation of the input
I0508 16:06:51.478516 139933369054976 gcsio.py:496] Finished listing 1 files in 0.05326652526855469 seconds.
I0508 16:06:51.480220 139933369054976 gcsio.py:480] Starting the size estimation of the input
I0508 16:06:51.527962 139933369054976 gcsio.py:496] Finished listing 1 files in 0.04774069786071777 seconds.
I0508 16:06:51.528657 139933369054976 gcsio.py:480] Starting the size estimation of the input
I0508 16:06:51.575519 139933369054976 gcsio.py:496] Finished listing 1 files in 0.04685568809509277 seconds.
I0508 16:06:51.604114 139933516105536 fn_api_runner.py:974] Running (((((ref_AppliedPTransform_Write/Write/WriteImpl/DoOnce/Impulse_12)+(ref_AppliedPTransform_Write/Write/WriteImpl/DoOnce/FlatMap(<lambda at core.py:2639>)_13))+(ref_AppliedPTransform_Write/Write/WriteImpl/DoOnce/Map(decode)_15))+(ref_AppliedPTransform_Write/Write/WriteImpl/InitializeWrite_16))+(ref_PCollection_PCollection_6/Write))+(ref_PCollection_PCollection_7/Write)
I0508 16:06:51.626923 139933516105536 fn_api_runner.py:974] Running ((((((ref_PCollection_PCollection_1_split/Read)+(Read/Read/_SDFBoundedSourceWrapper/ParDo(SDFBoundedSourceDoFn)/Process))+(ref_AppliedPTransform_ConvertToTFExample_7))+(ref_AppliedPTransform_Write/Write/WriteImpl/WriteBundles_17))+(ref_AppliedPTransform_Write/Write/WriteImpl/Pair_18))+(ref_AppliedPTransform_Write/Write/WriteImpl/WindowInto(WindowIntoFn)_19))+(Write/Write/WriteImpl/GroupByKey/Write)
W0508 16:06:52.151952 139933369054976 tfrecordio.py:63] Couldn't find python-snappy so the implementation of _TFRecordUtil._masked_crc32c is not as fast as it could be.
2020-05-08 16:06:53.771899: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
2020-05-08 16:06:54.350836: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-08 16:06:54.351292: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1555] Found device 0 with properties: 
pciBusID: 0000:00:04.0 name: Tesla V100-SXM2-16GB computeCapability: 7.0
coreClock: 1.53GHz coreCount: 80 deviceMemorySize: 15.75GiB deviceMemoryBandwidth: 836.37GiB/s
2020-05-08 16:06:54.351615: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcudart.so.10.1'; dlerror: libcudart.so.10.1: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/nvidia/lib:/usr/local/nvidia/lib64
2020-05-08 16:06:54.351855: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcublas.so.10'; dlerror: libcublas.so.10: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/nvidia/lib:/usr/local/nvidia/lib64
2020-05-08 16:06:54.352015: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcufft.so.10'; dlerror: libcufft.so.10: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/nvidia/lib:/usr/local/nvidia/lib64
2020-05-08 16:06:54.352181: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcurand.so.10'; dlerror: libcurand.so.10: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/nvidia/lib:/usr/local/nvidia/lib64
2020-05-08 16:06:54.352364: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcusolver.so.10'; dlerror: libcusolver.so.10: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/nvidia/lib:/usr/local/nvidia/lib64
2020-05-08 16:06:54.352548: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcusparse.so.10'; dlerror: libcusparse.so.10: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/nvidia/lib:/usr/local/nvidia/lib64
2020-05-08 16:06:54.356941: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-05-08 16:06:54.356972: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1592] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...
2020-05-08 16:06:54.357462: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 AVX512F FMA
2020-05-08 16:06:54.363119: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2000175000 Hz
2020-05-08 16:06:54.363433: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f4450fa39b0 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-05-08 16:06:54.363463: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
2020-05-08 16:06:54.460026: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-08 16:06:54.460613: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f445101f5a0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2020-05-08 16:06:54.460666: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Tesla V100-SXM2-16GB, Compute Capability 7.0
2020-05-08 16:06:54.460823: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1096] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-05-08 16:06:54.460847: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102]

Any help much appreciated. Thanks!

System Config

-----------------------------------------------------------------------------+
| NVIDIA-SMI 410.104      Driver Version: 410.104      CUDA Version: 10.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla V100-SXM2...  Off  | 00000000:00:04.0 Off |                    0 |
| N/A   36C    P0    38W / 300W |    417MiB / 16130MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
+-----------------------------------------------------------------------------+

opened by JWHennessey 9

Could you mind tell me which papers are you referring to to write ASR Task?

I am currently looking at the code of your ASR Task. But I found that many of the codes inside are difficult to understand, especially the decoder part. So I hope to find some references to help understand and learn. Could you mind tell me which papers are you referring to to write ASR Task? Thank you so much!!

opened by iamxiaoyubei 9
Bump certifi from 2022.5.18.1 to 2022.12.7 in /docker
Bumps certifi from 2022.5.18.1 to 2022.12.7.

Commits

9e9e840 2022.12.07

b81bdb2 2022.09.24

939a28f 2022.09.14

aca828a 2022.06.15.2

de0eae1 Only use importlib.resources's new files() / Traversable API on Python ≥3.11 ...

b8eb5e9 2022.06.15.1

47fb7ab Fix deprecation warning on Python 3.11 (#199)

b0b48e0 fixes #198 -- update link in license

9d514b4 2022.06.15

4151e88 Add py.typed to MANIFEST.in to package in sdist (#196)

See full diff in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

@dependabot rebase will rebase this PR

@dependabot recreate will recreate this PR, overwriting any edits that have been made to it

@dependabot merge will merge this PR after your CI passes on it

@dependabot squash and merge will squash and merge this PR after your CI passes on it

@dependabot cancel merge will cancel a previously requested merge and block automerging

@dependabot reopen will reopen this PR if it is closed

@dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually

@dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

@dependabot use these labels will set the current labels as the default for future PRs for this repo and language

@dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language

@dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language

@dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

You can disable automated security fix PRs for this repo from the Security Alerts page.

dependencies
opened by dependabot[bot] 0
Bump pillow from 9.1.1 to 9.3.0 in /docker
Bumps pillow from 9.1.1 to 9.3.0.

Release notes

Sourced from pillow's releases.

9.3.0

https://pillow.readthedocs.io/en/stable/releasenotes/9.3.0.html

Changes

Initialize libtiff buffer when saving #6699 [@radarhere]

Limit SAMPLESPERPIXEL to avoid runtime DOS #6700 [@wiredfool]

Inline fname2char to fix memory leak #6329 [@nulano]

Fix memory leaks related to text features #6330 [@nulano]

Use double quotes for version check on old CPython on Windows #6695 [@hugovk]

GHA: replace deprecated set-output command with GITHUB_OUTPUT file #6697 [@nulano]

Remove backup implementation of Round for Windows platforms #6693 [@cgohlke]

Upload fribidi.dll to GitHub Actions #6532 [@nulano]

Fixed set_variation_by_name offset #6445 [@radarhere]

Windows build improvements #6562 [@nulano]

Fix malloc in _imagingft.c:font_setvaraxes #6690 [@cgohlke]

Only use ASCII characters in C source file #6691 [@cgohlke]

Release Python GIL when converting images using matrix operations #6418 [@hmaarrfk]

Added ExifTags enums #6630 [@radarhere]

Do not modify previous frame when calculating delta in PNG #6683 [@radarhere]

Added support for reading BMP images with RLE4 compression #6674 [@npjg]

Decode JPEG compressed BLP1 data in original mode #6678 [@radarhere]

pylint warnings #6659 [@marksmayo]

Added GPS TIFF tag info #6661 [@radarhere]

Added conversion between RGB/RGBA/RGBX and LAB #6647 [@radarhere]

Do not attempt normalization if mode is already normal #6644 [@radarhere]

Fixed seeking to an L frame in a GIF #6576 [@radarhere]

Consider all frames when selecting mode for PNG save_all #6610 [@radarhere]

Don't reassign crc on ChunkStream close #6627 [@radarhere]

Raise a warning if NumPy failed to raise an error during conversion #6594 [@radarhere]

Only read a maximum of 100 bytes at a time in IMT header #6623 [@radarhere]

Show all frames in ImageShow #6611 [@radarhere]

Allow FLI palette chunk to not be first #6626 [@radarhere]

If first GIF frame has transparency for RGB_ALWAYS loading strategy, use RGBA mode #6592 [@radarhere]

Round box position to integer when pasting embedded color #6517 [@radarhere]

Removed EXIF prefix when saving WebP #6582 [@radarhere]

Pad IM palette to 768 bytes when saving #6579 [@radarhere]

Added DDS BC6H reading #6449 [@ShadelessFox]

Added support for opening WhiteIsZero 16-bit integer TIFF images #6642 [@JayWiz]

Raise an error when allocating translucent color to RGB palette #6654 [@jsbueno]

Moved mode check outside of loops #6650 [@radarhere]

Added reading of TIFF child images #6569 [@radarhere]

Improved ImageOps palette handling #6596 [@PososikTeam]

Defer parsing of palette into colors #6567 [@radarhere]

Apply transparency to P images in ImageTk.PhotoImage #6559 [@radarhere]

Use rounding in ImageOps contain() and pad() #6522 [@bibinhashley]

Fixed GIF remapping to palette with duplicate entries #6548 [@radarhere]

Allow remap_palette() to return an image with less than 256 palette entries #6543 [@radarhere]

Corrected BMP and TGA palette size when saving #6500 [@radarhere]

... (truncated)

Changelog

Sourced from pillow's changelog.

9.3.0 (2022-10-29)

Limit SAMPLESPERPIXEL to avoid runtime DOS #6700 [wiredfool]

Initialize libtiff buffer when saving #6699 [radarhere]

Inline fname2char to fix memory leak #6329 [nulano]

Fix memory leaks related to text features #6330 [nulano]

Use double quotes for version check on old CPython on Windows #6695 [hugovk]

Remove backup implementation of Round for Windows platforms #6693 [cgohlke]

Fixed set_variation_by_name offset #6445 [radarhere]

Fix malloc in _imagingft.c:font_setvaraxes #6690 [cgohlke]

Release Python GIL when converting images using matrix operations #6418 [hmaarrfk]

Added ExifTags enums #6630 [radarhere]

Do not modify previous frame when calculating delta in PNG #6683 [radarhere]

Added support for reading BMP images with RLE4 compression #6674 [npjg, radarhere]

Decode JPEG compressed BLP1 data in original mode #6678 [radarhere]

Added GPS TIFF tag info #6661 [radarhere]

Added conversion between RGB/RGBA/RGBX and LAB #6647 [radarhere]

Do not attempt normalization if mode is already normal #6644 [radarhere]

... (truncated)

Commits

d594f4c Update CHANGES.rst [ci skip]

909dc64 9.3.0 version bump

1a51ce7 Merge pull request #6699 from hugovk/security-libtiff_buffer

2444cdd Merge pull request #6700 from hugovk/security-samples_per_pixel-sec

744f455 Added release notes

0846bfa Add to release notes

799a6a0 Fix linting

00b25fd Hide UserWarning in logs

05b175e Tighter test case

13f2c5a Prevent DOS with large SAMPLESPERPIXEL in Tiff IFD

Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

@dependabot rebase will rebase this PR

@dependabot recreate will recreate this PR, overwriting any edits that have been made to it

@dependabot merge will merge this PR after your CI passes on it

@dependabot squash and merge will squash and merge this PR after your CI passes on it

@dependabot cancel merge will cancel a previously requested merge and block automerging

@dependabot reopen will reopen this PR if it is closed

@dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually

@dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

@dependabot use these labels will set the current labels as the default for future PRs for this repo and language

@dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language

@dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language

@dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

You can disable automated security fix PRs for this repo from the Security Alerts page.

dependencies
opened by dependabot[bot] 0
Bump tensorflow from 2.9.1 to 2.9.3 in /docker
Bumps tensorflow from 2.9.1 to 2.9.3.

Release notes

Sourced from tensorflow's releases.

TensorFlow 2.9.3

Release 2.9.3

This release introduces several vulnerability fixes:

Fixes an overflow in tf.keras.losses.poisson (CVE-2022-41887)

Fixes a heap OOB failure in ThreadUnsafeUnigramCandidateSampler caused by missing validation (CVE-2022-41880)

Fixes a segfault in ndarray_tensor_bridge (CVE-2022-41884)

Fixes an overflow in FusedResizeAndPadConv2D (CVE-2022-41885)

Fixes a overflow in ImageProjectiveTransformV2 (CVE-2022-41886)

Fixes an FPE in tf.image.generate_bounding_box_proposals on GPU (CVE-2022-41888)

Fixes a segfault in pywrap_tfe_src caused by invalid attributes (CVE-2022-41889)

Fixes a CHECK fail in BCast (CVE-2022-41890)

Fixes a segfault in TensorListConcat (CVE-2022-41891)

Fixes a CHECK_EQ fail in TensorListResize (CVE-2022-41893)

Fixes an overflow in CONV_3D_TRANSPOSE on TFLite (CVE-2022-41894)

Fixes a heap OOB in MirrorPadGrad (CVE-2022-41895)

Fixes a crash in Mfcc (CVE-2022-41896)

Fixes a heap OOB in FractionalMaxPoolGrad (CVE-2022-41897)

Fixes a CHECK fail in SparseFillEmptyRowsGrad (CVE-2022-41898)

Fixes a CHECK fail in SdcaOptimizer (CVE-2022-41899)

Fixes a heap OOB in FractionalAvgPool and FractionalMaxPool(CVE-2022-41900)

Fixes a CHECK_EQ in SparseMatrixNNZ (CVE-2022-41901)

Fixes an OOB write in grappler (CVE-2022-41902)

Fixes a overflow in ResizeNearestNeighborGrad (CVE-2022-41907)

Fixes a CHECK fail in PyFunc (CVE-2022-41908)

Fixes a segfault in CompositeTensorVariantToComponents (CVE-2022-41909)

Fixes a invalid char to bool conversion in printing a tensor (CVE-2022-41911)

Fixes a heap overflow in QuantizeAndDequantizeV2 (CVE-2022-41910)

Fixes a CHECK failure in SobolSample via missing validation (CVE-2022-35935)

Fixes a CHECK fail in TensorListScatter and TensorListScatterV2 in eager mode (CVE-2022-35935)

TensorFlow 2.9.2

Release 2.9.2

This releases introduces several vulnerability fixes:

Fixes a CHECK failure in tf.reshape caused by overflows (CVE-2022-35934)

Fixes a CHECK failure in SobolSample caused by missing validation (CVE-2022-35935)

Fixes an OOB read in Gather_nd op in TF Lite (CVE-2022-35937)

Fixes a CHECK failure in TensorListReserve caused by missing validation (CVE-2022-35960)

Fixes an OOB write in Scatter_nd op in TF Lite (CVE-2022-35939)

Fixes an integer overflow in RaggedRangeOp (CVE-2022-35940)

Fixes a CHECK failure in AvgPoolOp (CVE-2022-35941)

Fixes a CHECK failures in UnbatchGradOp (CVE-2022-35952)

Fixes a segfault TFLite converter on per-channel quantized transposed convolutions (CVE-2022-36027)

Fixes a CHECK failures in AvgPool3DGrad (CVE-2022-35959)

Fixes a CHECK failures in FractionalAvgPoolGrad (CVE-2022-35963)

Fixes a segfault in BlockLSTMGradV2 (CVE-2022-35964)

Fixes a segfault in LowerBound and UpperBound (CVE-2022-35965)

... (truncated)

Changelog

Sourced from tensorflow's changelog.

Release 2.9.3

This release introduces several vulnerability fixes:

Fixes an overflow in tf.keras.losses.poisson (CVE-2022-41887)

Fixes a heap OOB failure in ThreadUnsafeUnigramCandidateSampler caused by missing validation (CVE-2022-41880)

Fixes a segfault in ndarray_tensor_bridge (CVE-2022-41884)

Fixes an overflow in FusedResizeAndPadConv2D (CVE-2022-41885)

Fixes a overflow in ImageProjectiveTransformV2 (CVE-2022-41886)

Fixes an FPE in tf.image.generate_bounding_box_proposals on GPU (CVE-2022-41888)

Fixes a segfault in pywrap_tfe_src caused by invalid attributes (CVE-2022-41889)

Fixes a CHECK fail in BCast (CVE-2022-41890)

Fixes a segfault in TensorListConcat (CVE-2022-41891)

Fixes a CHECK_EQ fail in TensorListResize (CVE-2022-41893)

Fixes an overflow in CONV_3D_TRANSPOSE on TFLite (CVE-2022-41894)

Fixes a heap OOB in MirrorPadGrad (CVE-2022-41895)

Fixes a crash in Mfcc (CVE-2022-41896)

Fixes a heap OOB in FractionalMaxPoolGrad (CVE-2022-41897)

Fixes a CHECK fail in SparseFillEmptyRowsGrad (CVE-2022-41898)

Fixes a CHECK fail in SdcaOptimizer (CVE-2022-41899)

Fixes a heap OOB in FractionalAvgPool and FractionalMaxPool(CVE-2022-41900)

Fixes a CHECK_EQ in SparseMatrixNNZ (CVE-2022-41901)

Fixes an OOB write in grappler (CVE-2022-41902)

Fixes a overflow in ResizeNearestNeighborGrad (CVE-2022-41907)

Fixes a CHECK fail in PyFunc (CVE-2022-41908)

Fixes a segfault in CompositeTensorVariantToComponents (CVE-2022-41909)

Fixes a invalid char to bool conversion in printing a tensor (CVE-2022-41911)

Fixes a heap overflow in QuantizeAndDequantizeV2 (CVE-2022-41910)

Fixes a CHECK failure in SobolSample via missing validation (CVE-2022-35935)

Fixes a CHECK fail in TensorListScatter and TensorListScatterV2 in eager mode (CVE-2022-35935)

Release 2.8.4

This release introduces several vulnerability fixes:

Fixes a heap OOB failure in ThreadUnsafeUnigramCandidateSampler caused by missing validation (CVE-2022-41880)

Fixes a segfault in ndarray_tensor_bridge (CVE-2022-41884)

Fixes an overflow in FusedResizeAndPadConv2D (CVE-2022-41885)

Fixes a overflow in ImageProjectiveTransformV2 (CVE-2022-41886)

Fixes an FPE in tf.image.generate_bounding_box_proposals on GPU (CVE-2022-41888)

Fixes a segfault in pywrap_tfe_src caused by invalid attributes (CVE-2022-41889)

Fixes a CHECK fail in BCast (CVE-2022-41890)

Fixes a segfault in TensorListConcat (CVE-2022-41891)

Fixes a CHECK_EQ fail in TensorListResize (CVE-2022-41893)

Fixes an overflow in CONV_3D_TRANSPOSE on TFLite (CVE-2022-41894)

Fixes a heap OOB in MirrorPadGrad (CVE-2022-41895)

Fixes a crash in Mfcc (CVE-2022-41896)

Fixes a heap OOB in FractionalMaxPoolGrad (CVE-2022-41897)

Fixes a CHECK fail in SparseFillEmptyRowsGrad (CVE-2022-41898)

Fixes a CHECK fail in SdcaOptimizer (CVE-2022-41899)

... (truncated)

Commits

a5ed5f3 Merge pull request #58584 from tensorflow/vinila21-patch-2

258f9a1 Update py_func.cc

cd27cfb Merge pull request #58580 from tensorflow-jenkins/version-numbers-2.9.3-24474

3e75385 Update version numbers to 2.9.3

bc72c39 Merge pull request #58482 from tensorflow-jenkins/relnotes-2.9.3-25695

3506c90 Update RELEASE.md

8dcb48e Update RELEASE.md

4f34ec8 Merge pull request #58576 from pak-laura/c2.99f03a9d3bafe902c1e6beb105b2f2417...

6fc67e4 Replace CHECK with returning an InternalError on failing to create python tuple

5dbe90a Merge pull request #58570 from tensorflow/r2.9-7b174a0f2e4

Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

@dependabot rebase will rebase this PR

@dependabot recreate will recreate this PR, overwriting any edits that have been made to it

@dependabot merge will merge this PR after your CI passes on it

@dependabot squash and merge will squash and merge this PR after your CI passes on it

@dependabot cancel merge will cancel a previously requested merge and block automerging

@dependabot reopen will reopen this PR if it is closed

@dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually

@dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

@dependabot use these labels will set the current labels as the default for future PRs for this repo and language

@dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language

@dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language

@dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

You can disable automated security fix PRs for this repo from the Security Alerts page.

dependencies
opened by dependabot[bot] 0
images

How do you project the point cloud onto the image and keep only the foreground points and remove the background points, can you tell me how to do this and the corresponding code

opened by anlent 0
Bump protobuf from 3.19.4 to 3.19.5 in /docker
Bumps protobuf from 3.19.4 to 3.19.5.

Release notes

Sourced from protobuf's releases.

Protocol Buffers v3.19.5

C++

Reduce memory consumption of MessageSet parsing

This release addresses a Security Advisory for C++ and Python users

Commits

b464cfb Updating changelog

40859fb Updating version.json and repo version numbers to: 19.5

3b175f1 Merge pull request #10543 from deannagarcia/3.19.x

c05b5f3 Add missing includes

0299c03 Apply patch

0a722f1 Update version.json with "lts": true (#10533)

d5eb60a Merge pull request #10530 from protocolbuffers/deannagarcia-patch-6

6cf1f78 Update version.json

97fc844 Merge pull request #10504 from deannagarcia/3.19.x

29d60a2 Add version file

Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

@dependabot rebase will rebase this PR

@dependabot recreate will recreate this PR, overwriting any edits that have been made to it

@dependabot merge will merge this PR after your CI passes on it

@dependabot squash and merge will squash and merge this PR after your CI passes on it

@dependabot cancel merge will cancel a previously requested merge and block automerging

@dependabot reopen will reopen this PR if it is closed

@dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually

@dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

@dependabot use these labels will set the current labels as the default for future PRs for this repo and language

@dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language

@dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language

@dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

You can disable automated security fix PRs for this repo from the Security Alerts page.

dependencies
opened by dependabot[bot] 0