Lingvo is a framework for building neural networks in Tensorflow, particularly sequence models.

Overview

Lingvo

PyPI Python

Documentation

License

What is it?

Lingvo is a framework for building neural networks in Tensorflow, particularly sequence models.

A list of publications using Lingvo can be found here.

Table of Contents

Releases

PyPI Version Commit
0.10.0 075fd1d88fa6f92681f58a2383264337d0e737ee
0.9.1 c1124c5aa7af13d2dd2b6d43293c8ca6d022b008
0.9.0 f826e99803d1b51dccbbbed1ef857ba48a2bbefe
Older releases

PyPI Version Commit
0.8.2 93e123c6788e934e6b7b1fd85770371becf1e92e
0.7.2 b05642fe386ee79e0d88aa083565c9a93428519e

Details for older releases are unavailable.

Major breaking changes

NOTE: this is not a comprehensive list. Lingvo releases do not offer any guarantees regarding backwards compatibility.

HEAD

Nothing here.

0.10.0

  • General
    • The theta_fn arg to CreateVariable() has been removed.

0.9.1

  • General
    • Python 3.9 is now supported.
    • ops.beam_search_step now takes and returns an additional arg beam_done.
    • The namedtuple beam_search_helper.BeamSearchDecodeOutput now removes the field done_hyps.

0.9.0

  • General
    • Tensorflow 2.5 is now the required version.
    • Python 3.5 support has been removed.
    • py_utils.AddGlobalVN and py_utils.AddPerStepVN have been combined into py_utils.AddVN.
    • BaseSchedule().Value() no longer takes a step arg.
    • Classes deriving from BaseSchedule should implement Value() not FProp().
    • theta.global_step has been removed in favor of py_utils.GetGlobalStep().
    • py_utils.GenerateStepSeedPair() no longer takes a global_step arg.
    • PostTrainingStepUpdate() no longer takes a global_step arg.
    • The fatal_errors argument to custom input ops now takes error message substrings rather than integer error codes.
Older releases

0.8.2

  • General
    • NestedMap Flatten/Pack/Transform/Filter etc now expand descendent dicts as well.
    • Subclasses of BaseLayer extending from abc.ABCMeta should now extend base_layer.ABCLayerMeta instead.
    • Trying to call self.CreateChild outside of __init__ now raises an error.
    • base_layer.initializer has been removed. Subclasses no longer need to decorate their __init__ function.
    • Trying to call self.CreateVariable outside of __init__ or _CreateLayerVariables now raises an error.
    • It is no longer possible to access self.vars or self.theta inside of __init__. Refactor by moving the variable creation and access to _CreateLayerVariables. The variable scope is set automatically according to the layer name in _CreateLayerVariables.

Details for older releases are unavailable.

Quick start

Installation

There are two ways to set up Lingvo: installing a fixed version through pip, or cloning the repository and building it with bazel. Docker configurations are provided for each case.

If you would just like to use the framework as-is, it is easiest to just install it through pip. This makes it possible to develop and train custom models using a frozen version of the Lingvo framework. However, it is difficult to modify the framework code or implement new custom ops.

If you would like to develop the framework further and potentially contribute pull requests, you should avoid using pip and clone the repository instead.

pip:

The Lingvo pip package can be installed with pip3 install lingvo.

See the codelab for how to get started with the pip package.

From sources:

The prerequisites are:

  • a TensorFlow 2.6 installation,
  • a C++ compiler (only g++ 7.3 is officially supported), and
  • the bazel build system.

Refer to docker/dev.dockerfile for a set of working requirements.

git clone the repository, then use bazel to build and run targets directly. The python -m module commands in the codelab need to be mapped onto bazel run commands.

docker:

Docker configurations are available for both situations. Instructions can be found in the comments on the top of each file.

How to install docker.

Running the MNIST image model

Preparing the input data

pip:

mkdir -p /tmp/mnist
python3 -m lingvo.tools.keras2ckpt --dataset=mnist

bazel:

mkdir -p /tmp/mnist
bazel run -c opt //lingvo/tools:keras2ckpt -- --dataset=mnist

The following files will be created in /tmp/mnist:

  • mnist.data-00000-of-00001: 53MB.
  • mnist.index: 241 bytes.

Running the model

pip:

cd /tmp/mnist
curl -O https://raw.githubusercontent.com/tensorflow/lingvo/master/lingvo/tasks/image/params/mnist.py
python3 -m lingvo.trainer --run_locally=cpu --mode=sync --model=mnist.LeNet5 --logdir=/tmp/mnist/log

bazel:

(cpu) bazel build -c opt //lingvo:trainer
(gpu) bazel build -c opt --config=cuda //lingvo:trainer
bazel-bin/lingvo/trainer --run_locally=cpu --mode=sync --model=image.mnist.LeNet5 --logdir=/tmp/mnist/log --logtostderr

After about 20 seconds, the loss should drop below 0.3 and a checkpoint will be saved, like below. Kill the trainer with Ctrl+C.

trainer.py:518] step:   205, steps/sec: 11.64 ... loss:0.25747201 ...
checkpointer.py:115] Save checkpoint
checkpointer.py:117] Save checkpoint done: /tmp/mnist/log/train/ckpt-00000205

Some artifacts will be produced in /tmp/mnist/log/control:

  • params.txt: hyper-parameters.
  • model_analysis.txt: model sizes for each layer.
  • train.pbtxt: the training tf.GraphDef.
  • events.*: a tensorboard events file.

As well as in /tmp/mnist/log/train:

  • checkpoint: a text file containing information about the checkpoint files.
  • ckpt-*: the checkpoint files.

Now, let's evaluate the model on the "Test" dataset. In the normal training setup the trainer and evaler should be run at the same time as two separate processes.

pip:

python3 -m lingvo.trainer --job=evaler_test --run_locally=cpu --mode=sync --model=mnist.LeNet5 --logdir=/tmp/mnist/log

bazel:

bazel-bin/lingvo/trainer --job=evaler_test --run_locally=cpu --mode=sync --model=image.mnist.LeNet5 --logdir=/tmp/mnist/log --logtostderr

Kill the job with Ctrl+C when it starts waiting for a new checkpoint.

base_runner.py:177] No new check point is found: /tmp/mnist/log/train/ckpt-00000205

The evaluation accuracy can be found slightly earlier in the logs.

base_runner.py:111] eval_test: step:   205, acc5: 0.99775392, accuracy: 0.94150388, ..., loss: 0.20770954, ...

Running the machine translation model

To run a more elaborate model, you'll need a cluster with GPUs. Please refer to third_party/py/lingvo/tasks/mt/README.md for more information.

Running the GShard transformer based giant language model

To train a GShard language model with one trillion parameters on GCP using CloudTPUs v3-512 using 512-way model parallelism, please refer to third_party/py/lingvo/tasks/lm/README.md for more information.

Running the 3d object detection model

To run the StarNet model using CloudTPUs on GCP, please refer to third_party/py/lingvo/tasks/car/README.md.

Models

Automatic Speech Recognition

Car

Image

Language Modelling

Machine Translation

References

Please cite this paper when referencing Lingvo.

@misc{shen2019lingvo,
    title={Lingvo: a Modular and Scalable Framework for Sequence-to-Sequence Modeling},
    author={Jonathan Shen and Patrick Nguyen and Yonghui Wu and Zhifeng Chen and others},
    year={2019},
    eprint={1902.08295},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}

License

Apache License 2.0

Comments
  • Undefined reference to google::protobuf::FileDescriptor::DebugString()

    Undefined reference to google::protobuf::FileDescriptor::DebugString()

    I just installed lingvo and it looks like I face the issue with protobuf linking. The nightly-tf seems to be up-to-date

    mironov@70e0b410070b:~/lingvo$ python -c "import tensorflow as tf;print(tf.__version__)"
    1.14.1-dev20190305
    

    The exact error from bazel build is

    mironov@70e0b410070b:~/lingvo$ bazel build -c opt //lingvo:trainer
    INFO: Analysed target //lingvo:trainer (22 packages loaded).
    INFO: Found 1 target...
    ERROR: /workspace/lingvo/lingvo/tools/BUILD:98:1: Linking of rule '//lingvo/tools:generate_proto_def' failed (Exit 1)
    bazel-out/host/bin/lingvo/tools/_objs/generate_proto_def/generate_proto_def.o:generate_proto_def.cc:function (anonymous namespace)::WriteDotProto(google::protobuf::FileDescriptor const*, char const*): error: undefined reference to 'google::protobuf::FileDescriptor::DebugString() const'
    collect2: error: ld returned 1 exit status
    Target //lingvo:trainer failed to build
    Use --verbose_failures to see the command lines of failed build steps.
    INFO: Elapsed time: 3.746s, Critical Path: 2.20s
    INFO: 3 processes: 3 processwrapper-sandbox.
    FAILED: Build did NOT complete successfully
    

    Could you please check?

    bug stat:awaiting response 
    opened by grwlf 23
  • list index out of range when running model

    list index out of range when running model

    when I am running tf.app.run(trainer.main,argv=argv) in model training part I am getting list index out of range error. pls tell me how to get rid out of this...

    I am not getting to which element its trying to access

    bug stat:awaiting response 
    opened by nim456 18
  • How to use tf_debug in Predictor to see the tensor value each step in RNN while inference?

    How to use tf_debug in Predictor to see the tensor value each step in RNN while inference?

    Hello, thank you for your prompt and enthusiastic answer each time~

    Recently, I am trying to add LM fusion in ASR task. And I want to check whether I am right by step by step debugging while inference. However, no matter how I use TensorBoardDebugWrapperSession or LocalCLIDebugWrapperSession it will report an error.

    I change this line:

    https://github.com/tensorflow/lingvo/blob/0d7828b0bbba2615a7581c79f532beb4251adfe8/lingvo/core/predictor.py#L208

    to

    ## When try TensorBoardDebugWrapperSession
        return self._RunWithValidSession(
            tf_debug.TensorBoardDebugWrapperSession.run, fetches, feed_dict=feeds, options=run_options)
    ## When try LocalCLIDebugWrapperSession
        return self._RunWithDebugSession(
            tf_debug.LocalCLIDebugWrapperSession.run, fetches, feed_dict=feeds, options=run_options)
    

    I change this line:

    https://github.com/tensorflow/lingvo/blob/0d7828b0bbba2615a7581c79f532beb4251adfe8/lingvo/core/predictor.py#L160

    to

    ## When try TensorBoardDebugWrapperSession
          self._sess = tf_debug.TensorBoardDebugWrapperSession(self._sess, "dm-System-Product-Name:6008")
          return fn(self._sess, *args, **kwargs)
    

    However, it occurs: TIM图片20190604213913

    I remove self._sess from fn's args:

    ## When try TensorBoardDebugWrapperSession
          self._sess = tf_debug.TensorBoardDebugWrapperSession(self._sess, "dm-System-Product-Name:6008")
          return fn(*args, **kwargs)
    

    It also occurs an error.

    Then, I keep the following line unchanged, and change this line: https://github.com/tensorflow/lingvo/blob/0d7828b0bbba2615a7581c79f532beb4251adfe8/lingvo/core/predictor.py#L142

    to

        self._sess = sess
        self._sess = tf_debug.TensorBoardDebugWrapperSession(self._sess, 'localhost:6008')
    

    However, it occurs: TIM图片20190604213938

    I have tried many other ways. However, none of them succeed. So, would you mind tell me how to use tf_debug in Predictor to see how RNN runs while inference? Or, if you have other method to debug step by step, could you share me your code about it? Thank you a lot!!

    (I want to be able to see the results of each step of the variable at runtime like the following picture) mmexport1559657226740

    opened by iamxiaoyubei 14
  • local build error

    local build error

    ERROR: error loading package 'lingvo': Encountered error while reading extension file 'subpar.bzl': no such package '@subpar//': Traceback (most recent call last):
            File "/home/luban/.cache/bazel/_bazel_luban/b5ef85f1c360696308ba7ab9000cfd03/external/bazel_tools/tools/build_defs/repo/git.bzl", line 166
                    _clone_or_update(ctx)
            File "/home/luban/.cache/bazel/_bazel_luban/b5ef85f1c360696308ba7ab9000cfd03/external/bazel_tools/tools/build_defs/repo/git.bzl", line 72, in _clone_or_update
                    fail(("error cloning %s:\n%s" % (ctx....)))
    error cloning subpar:
    
    bug 
    opened by zh794390558 14
  • How to call another model while a graph is executed in a tf.Session

    How to call another model while a graph is executed in a tf.Session

    I trained two models: ASR Wpm model and Language model. When I passed a intermediate value of ASR model to Language model in order to predict the intermediate result, I got an error. Because tensorflow is not able to pass tensor into feed_dict. But I can only get the value which passed to language model while the ASR model is running. So this value can only be saved in the form of tensor. So, could you tell me how can I call language model in ASR model? I really hope that you can help me. Thank you!!

    opened by iamxiaoyubei 11
  • Large Model - GPipe Transformer - OOM Issue

    Large Model - GPipe Transformer - OOM Issue

    I am trying to run the large model described in the comment section of https://github.com/tensorflow/lingvo/blob/master/lingvo/tasks/lm/params/one_billion_wds.py#L179

    However, I keep getting OOM error.

    Following are the variations I have tried: #4 GPU mode: SPLITS = [2 + 11 * i for i in range(GPU-1)] LAYERS = SPLIT[-1] BATCH = 2 or 4 or 8 MICROBATCH =2 or 4 or 8 (respectively)

    #8 GPU mode : EMBEDDING_DIM = 1024 GPUS = 8 # For example, on 8 accelerator, each of which has 16G mem. SPLITS = [10 + 12 * i for i in range(GPUS - 1)] LAYERS = SPLITS[-1] #4.9B params BATCH_SIZE = 2 NUM_MICRO_BATCHES = 2

    ...
    019-06-08 01:46:36.299559: I tensorflow/core/common_runtime/bfc_allocator.cc:824] Stats:
    Limit:                 15750955008
    InUse:                 15584222976
    MaxInUse:              15584223232
    NumAllocs:                    2361
    MaxAllocSize:           3250061312
    
    2019-06-08 01:46:36.299616: W tensorflow/core/common_runtime/bfc_allocator.cc:319] ****************************************************************************************************
    2019-06-08 01:46:36.299642: W tensorflow/core/framework/op_kernel.cc:1502] OP_REQUIRES failed at matmul_op.cc:480 : Resource exhausted: OOM when allocating tensor with shape[169,793472] and type float on /job:local/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
    2019-06-08 01:46:36.919956: I lingvo/core/ops/record_yielder.cc:317] 0x7fa6a682e060Basic record yielder exit
    I0608 01:46:41.227104 140355034937088 base_runner.py:239] trainer done (fatal error).
    I0608 01:46:41.227474 140355034937088 base_runner.py:118] trainer exception: From /job:local/replica:0/task:0:
    [_Derived_]OOM when allocating tensor with shape[1024,49592] and type float on /job:local/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
             [[{{node Add_28}}]]
    Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
    
             [[gradients/fprop/1bwds_wpm_level_lm/tower_0_0/Sum_435_grad/Tile_G4412]]
    Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
    
    
    E0608 01:46:41.230962 140355034937088 base_runner.py:246] Traceback (most recent call last):
    E0608 01:46:41.231157 140355034937088 base_runner.py:246]   File "/tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/base_runner.py", line 199, in _RunLoop
    E0608 01:46:41.231286 140355034937088 base_runner.py:246]     loop_func(*loop_args)
    E0608 01:46:41.231404 140355034937088 base_runner.py:246]   File "/tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/trainer.py", line 538, in _Loop
    E0608 01:46:41.231527 140355034937088 base_runner.py:246]     model_task.per_example_tensors,
    E0608 01:46:41.231688 140355034937088 base_runner.py:246]   File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 950, in run
    E0608 01:46:41.231817 140355034937088 base_runner.py:246]     run_metadata_ptr)
    E0608 01:46:41.231914 140355034937088 base_runner.py:246]   File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1173, in _run
    E0608 01:46:41.231992 140355034937088 base_runner.py:246]     feed_dict_tensor, options, run_metadata)
    E0608 01:46:41.232054 140355034937088 base_runner.py:246]   File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1350, in _do_run
    E0608 01:46:41.232116 140355034937088 base_runner.py:246]     run_metadata)
    E0608 01:46:41.232177 140355034937088 base_runner.py:246]   File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1370, in _do_call
    E0608 01:46:41.232239 140355034937088 base_runner.py:246]     raise type(e)(node_def, op, message)
    E0608 01:46:41.232300 140355034937088 base_runner.py:246] ResourceExhaustedError: From /job:local/replica:0/task:0:
    E0608 01:46:41.232366 140355034937088 base_runner.py:246] [_Derived_]OOM when allocating tensor with shape[1024,49592] and type float on /job:local/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
    E0608 01:46:41.232431 140355034937088 base_runner.py:246]        [[{{node Add_28}}]]
    E0608 01:46:41.232492 140355034937088 base_runner.py:246] Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
    E0608 01:46:41.232563 140355034937088 base_runner.py:246]
    E0608 01:46:41.232625 140355034937088 base_runner.py:246]        [[gradients/fprop/1bwds_wpm_level_lm/tower_0_0/Sum_435_grad/Tile_G4412]]
    E0608 01:46:41.232690 140355034937088 base_runner.py:246] Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
    E0608 01:46:41.232752 140355034937088 base_runner.py:246]
    E0608 01:46:41.232812 140355034937088 base_runner.py:246]
    
    opened by msharmavikram 11
  • Getting segmentation fault while trying to run Gpipe example code

    Getting segmentation fault while trying to run Gpipe example code

    Hi, as you mentioned in #48 about changing OneBWdsGPipeTransformer hparams and then try to run on 8 GPU's and gave the command to run. I did not understand what are those parameters, can I get help which parameters fit for my system. I am using machine consisting of 4 GPU. What ever the parameters I change I am facing segmentation fault core dumped. I am also attaching my system info(GPU).

    command : bazel-bin/lingvo/trainer --run_locally=gpu --mode=sync --model=lm.one_billion_wds.OneBWdsGPipeTransformer --logdir=/tmp/mnist/log --logtostderr --worker_split_size=4 segmentation fault.txt

    system info: GPU: sys_info.txt

    opened by Raviteja1996 10
  • Is asr task job can run correctly on gpu?

    Is asr task job can run correctly on gpu?

    I use tf-nightly-gpu==1.13.0-dev20181116, but when I run asr task, I got the error below.

    bazel-bin/lingvo/trainer --run_locally=gpu --mode=sync --model=asr.librispeech.Librispeech960Base --logdir=/tmp/librispeech/log --logtostderr
    
    INFO:tensorflow:Retry: caught exception: _WaitTillInit while running FailedPreconditionError: Attempting to use uninitialized value global_step
    	 [[{{node _send_global_step_0}}]]
    . Call failed at (most recent call last):
      File "/usr/lib/python2.7/threading.py", line 774, in __bootstrap
        self.__bootstrap_inner()
      File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
        self.run()
      File "/usr/lib/python2.7/threading.py", line 754, in run
        self.__target(*self.__args, **self.__kwargs)
      File "/tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/trainer.py", line 401, in Start
        self._RunLoop('trainer', self._Loop)
      File "/tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/core/retry.py", line 50, in wrapper
        return func(*args, **kwargs)
      File "/tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/base_runner.py", line 173, in _RunLoop
        loop_func(*args)
    Traceback for above exception (most recent call last):
      File "/tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/core/retry.py", line 50, in wrapper
        return func(*args, **kwargs)
      File "/tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/trainer.py", line 416, in _WaitTillInit
        global_step = sess.run(self._model.global_step)
      File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 929, in run
        run_metadata_ptr)
      File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1152, in _run
        feed_dict_tensor, options, run_metadata)
      File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1328, in _do_run
        run_metadata)
      File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1348, in _do_call
        raise type(e)(node_def, op, message)
    Waiting for 0.11 seconds before retrying.
    
    opened by fanlu 10
  • Reduce memory requirement Starnet on Waymo dataset

    Reduce memory requirement Starnet on Waymo dataset

    I have been unable to get Starnet working on TPUs so am now trying on 8 x v100 GPUs. However, I am not running into memory issues.

    Are there any way to reduce the memory required of the model?

    I have tried setting p.batch_size = 1 in tasks/car/params/waymo.py but still run out of memory.

    I have tried many set of training parameters but the ones used for the error below was:

    bazel-bin/lingvo/trainer --logtostderr --model=car.waymo.StarNetVehicle --mode=async --logdir=gs://waymo-challange/logs/starnet.waymo.3d_object.v3 --run_locally=gpu --worker_gpus=7 --worker_split_size=7 --worker_replicas=7 --controller_gpus=1

    Error:

    I0512 13:49:24.586677 139606888535808 base_runner.py:111] step:     0
    WARNING:tensorflow:Issue encountered when serializing __batch_norm_update_dict.
    Type is unsupported, or the types of the items don't match field type in CollectionDef. Note this is a warning and probably safe to ignore.
    'dict' object has no attribute 'name'
    W0512 13:49:30.131858 139606896928512 meta_graph.py:436] Issue encountered when serializing __batch_norm_update_dict.
    Type is unsupported, or the types of the items don't match field type in CollectionDef. Note this is a warning and probably safe to ignore.
    'dict' object has no attribute 'name'
    2020-05-12 13:49:30.990706: I ./lingvo/core/ops/input_common.h:74] Create RecordProcessor
    2020-05-12 13:49:30.994158: I lingvo/core/ops/input_common.cc:34] Input source weights are empty, fall back to legacy behavior.
    2020-05-12 13:49:30.994610: I lingvo/core/ops/record_yielder.cc:376] 0x7ef8a34d9f20 Record yielder start
    2020-05-12 13:49:30.994631: I lingvo/core/ops/record_yielder.cc:378] Randomly seed RecordYielder.
    2020-05-12 13:49:30.994655: I ./lingvo/core/ops/input_common.h:80] Create batcher
    2020-05-12 13:49:30.994678: I lingvo/core/ops/record_yielder.cc:485] Epoch 1 gs://waymo_open_dataset_v_1_0_0_tf_example_lingvo/v.1.0.0/train.tfr-*-of-01000
    I0512 13:49:31.538286 139606896928512 checkpointer.py:118] Save checkpoint done: gs://waymo-challange/logs/starnet.waymo.3d_object.v3/train/ckpt-00000000
    2020-05-12 13:49:33.201902: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
    2020-05-12 13:50:02.702309: W tensorflow/core/common_runtime/bfc_allocator.cc:424] Allocator (GPU_0_bfc) ran out of memory trying to allocate 512.00MiB (rounded to 536870912).  Current allocation summary follows.
    2020-05-12 13:50:02.702388: I tensorflow/core/common_runtime/bfc_allocator.cc:894] BFCAllocator dump for GPU_0_bfc
    2020-05-12 13:50:02.702401: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (256):   Total Chunks: 176, Chunks in use: 176. 44.0KiB allocated for chunks. 44.0KiB in use in bin. 651B client-requested in use in bin.
    2020-05-12 13:50:02.702414: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (512):   Total Chunks: 32, Chunks in use: 31. 16.2KiB allocated for chunks. 15.8KiB in use in bin. 15.5KiB client-requested in use in bin.
    2020-05-12 13:50:02.702425: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (1024):  Total Chunks: 32, Chunks in use: 29. 37.5KiB allocated for chunks. 33.2KiB in use in bin. 30.0KiB client-requested in use in bin.
    2020-05-12 13:50:02.702435: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (2048):  Total Chunks: 1, Chunks in use: 0. 2.2KiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
    2020-05-12 13:50:02.702445: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (4096):  Total Chunks: 3, Chunks in use: 1. 14.8KiB allocated for chunks. 4.0KiB in use in bin. 4.0KiB client-requested in use in bin.
    2020-05-12 13:50:02.702455: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (8192):  Total Chunks: 2, Chunks in use: 0. 26.5KiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
    2020-05-12 13:50:02.702465: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (16384):         Total Chunks: 1, Chunks in use: 0. 17.0KiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
    2020-05-12 13:50:02.702474: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (32768):         Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
    2020-05-12 13:50:02.702485: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (65536):         Total Chunks: 12, Chunks in use: 9. 771.2KiB allocated for chunks. 576.2KiB in use in bin. 576.0KiB client-requested in use in bin.
    2020-05-12 13:50:02.702496: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (131072):        Total Chunks: 14, Chunks in use: 10. 1.89MiB allocated for chunks. 1.25MiB in use in bin. 1.25MiB client-requested in use in bin.
    2020-05-12 13:50:02.702506: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (262144):        Total Chunks: 16, Chunks in use: 14. 4.43MiB allocated for chunks. 3.78MiB in use in bin. 3.78MiB client-requested in use in bin.
    2020-05-12 13:50:02.702517: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (524288):        Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
    2020-05-12 13:50:02.702530: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (1048576):       Total Chunks: 6, Chunks in use: 5. 9.38MiB allocated for chunks. 8.20MiB in use in bin. 8.20MiB client-requested in use in bin.
    2020-05-12 13:50:02.702541: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (2097152):       Total Chunks: 14, Chunks in use: 12. 29.92MiB allocated for chunks. 24.59MiB in use in bin. 24.34MiB client-requested in use in bin.
    2020-05-12 13:50:02.702551: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (4194304):       Total Chunks: 3, Chunks in use: 0. 14.07MiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
    2020-05-12 13:50:02.702561: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (8388608):       Total Chunks: 2, Chunks in use: 0. 25.71MiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
    2020-05-12 13:50:02.702571: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (16777216):      Total Chunks: 1, Chunks in use: 0. 28.00MiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
    2020-05-12 13:50:02.702581: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (33554432):      Total Chunks: 1, Chunks in use: 0. 32.00MiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
    2020-05-12 13:50:02.702590: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (67108864):      Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
    2020-05-12 13:50:02.702600: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (134217728):     Total Chunks: 20, Chunks in use: 19. 2.50GiB allocated for chunks. 2.38GiB in use in bin. 2.38GiB client-requested in use in bin.
    2020-05-12 13:50:02.702611: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (268435456):     Total Chunks: 31, Chunks in use: 30. 11.98GiB allocated for chunks. 11.50GiB in use in bin. 11.50GiB client-requested in use in bin.
    2020-05-12 13:50:02.702622: I tensorflow/core/common_runtime/bfc_allocator.cc:917] Bin for 512.00MiB was 256.00MiB, Chunk State: 
    2020-05-12 13:50:02.702638: I tensorflow/core/common_runtime/bfc_allocator.cc:923]   Size: 492.95MiB | Requested Size: 256.0KiB | in_use: 0 | bin_num: 20, prev:   Size: 512.00MiB | Requested Size: 512.00MiB | in_use: 1 | bin_num: -1
    2020-05-12 13:50:02.702648: I tensorflow/core/common_runtime/bfc_allocator.cc:930] Next region of size 15702704128
    2020-05-12 13:50:02.702660: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ec000000 of size 1280 next 1
    2020-05-12 13:50:02.702668: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ec000500 of size 256 next 2
    2020-05-12 13:50:02.702677: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ec000600 of size 256 next 3
    2020-05-12 13:50:02.702686: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ec000700 of size 256 next 4
    2020-05-12 13:50:02.702694: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ec000800 of size 256 next 5
    2020-05-12 13:50:02.702703: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ec000900 of size 256 next 6
    2020-05-12 13:50:02.702711: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ec000a00 of size 256 next 7
    2020-05-12 13:50:02.702719: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ec000b00 of size 256 next 8
    2020-05-12 13:50:02.702727: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ec000c00 of size 256 next 9
    2020-05-12 13:50:02.702736: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ec000d00 of size 262144 next 10
    2020-05-12 13:50:02.702744: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ec040d00 of size 262144 next 11
    2020-05-12 13:50:02.702753: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ec080d00 of size 262144 next 12
    2020-05-12 13:50:02.702761: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ec0c0d00 of size 262144 next 13
    2020-05-12 13:50:02.702769: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ec100d00 of size 262144 next 14
    2020-05-12 13:50:02.702777: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ec140d00 of size 262144 next 15
    2020-05-12 13:50:02.702792: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ec180d00 of size 262144 next 16
    2020-05-12 13:50:02.702801: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ec1c0d00 of size 262144 next 17
    2020-05-12 13:50:02.702809: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ec200d00 of size 262144 next 18
    2020-05-12 13:50:02.702818: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ec240d00 of size 262144 next 19
    2020-05-12 13:50:02.702827: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ec280d00 of size 262144 next 20
    2020-05-12 13:50:02.702835: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ec2c0d00 of size 2048000 next 21
    2020-05-12 13:50:02.702843: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ec4b4d00 of size 2048000 next 22
    2020-05-12 13:50:02.702852: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ec6a8d00 of size 2457600 next 23
    2020-05-12 13:50:02.702861: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ec900d00 of size 1228800 next 24
    2020-05-12 13:50:02.702869: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4eca2cd00 of size 1228800 next 25
    2020-05-12 13:50:02.702878: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ecb58d00 of size 409600 next 26
    2020-05-12 13:50:02.702886: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ecbbcd00 of size 409600 next 27
    2020-05-12 13:50:02.702894: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ecc20d00 of size 256 next 28
    2020-05-12 13:50:02.702903: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ecc20e00 of size 256 next 29
    2020-05-12 13:50:02.702912: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ecc20f00 of size 256 next 30
    2020-05-12 13:50:02.702920: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ecc21000 of size 256 next 31
    2020-05-12 13:50:02.702929: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ecc21100 of size 256 next 32
    2020-05-12 13:50:02.702938: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ecc21200 of size 256 next 33
    2020-05-12 13:50:02.702953: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ecc21300 of size 256 next 34
    2020-05-12 13:50:02.702967: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ecc21400 of size 256 next 35
    2020-05-12 13:50:02.702980: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ecc21500 of size 256 next 36
    2020-05-12 13:50:02.702991: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ecc21600 of size 256 next 37
    2020-05-12 13:50:02.703003: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ecc21700 of size 256 next 38
    2020-05-12 13:50:02.703013: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ecc21800 of size 256 next 39
    2020-05-12 13:50:02.703023: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ecc21900 of size 2048000 next 40
    2020-05-12 13:50:02.703035: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ece15900 of size 256 next 41
    2020-05-12 13:50:02.703047: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ece15a00 of size 256 next 42
    2020-05-12 13:50:02.703059: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ece15b00 of size 256 next 43
    2020-05-12 13:50:02.703070: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ece15c00 of size 256 next 44
    2020-05-12 13:50:02.703081: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ece15d00 of size 262144 next 45
    2020-05-12 13:50:02.703093: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ece55d00 of size 256 next 46
    2020-05-12 13:50:02.703106: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ece55e00 of size 256 next 47
    2020-05-12 13:50:02.703118: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ece55f00 of size 512 next 48
    2020-05-12 13:50:02.703128: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ece56100 of size 2097152 next 49
    2020-05-12 13:50:02.703140: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed056100 of size 1024 next 50
    2020-05-12 13:50:02.703151: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed056500 of size 256 next 51
    2020-05-12 13:50:02.703160: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed056600 of size 1536 next 52
    2020-05-12 13:50:02.703172: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed056c00 of size 256 next 53
    2020-05-12 13:50:02.703184: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed056d00 of size 256 next 54
    2020-05-12 13:50:02.703194: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed056e00 of size 4096 next 55
    2020-05-12 13:50:02.703202: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed057e00 of size 256 next 56
    2020-05-12 13:50:02.703211: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed057f00 of size 256 next 57
    2020-05-12 13:50:02.703219: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed058000 of size 256 next 58
    2020-05-12 13:50:02.703231: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed058100 of size 256 next 59
    2020-05-12 13:50:02.703243: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed058200 of size 256 next 60
    2020-05-12 13:50:02.703255: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed058300 of size 256 next 61
    2020-05-12 13:50:02.703266: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed058400 of size 256 next 62
    2020-05-12 13:50:02.703275: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed058500 of size 256 next 63
    2020-05-12 13:50:02.703285: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed058600 of size 256 next 64
    2020-05-12 13:50:02.703293: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed058700 of size 256 next 65
    2020-05-12 13:50:02.703299: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed058800 of size 256 next 66
    2020-05-12 13:50:02.703311: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed058900 of size 256 next 67
    2020-05-12 13:50:02.703322: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed058a00 of size 256 next 68
    2020-05-12 13:50:02.703331: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed058b00 of size 256 next 69
    2020-05-12 13:50:02.703342: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed058c00 of size 256 next 70
    2020-05-12 13:50:02.703351: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed058d00 of size 256 next 71
    2020-05-12 13:50:02.703360: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed058e00 of size 256 next 72
    2020-05-12 13:50:02.703368: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed058f00 of size 256 next 73
    2020-05-12 13:50:02.703377: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed059000 of size 256 next 74
    2020-05-12 13:50:02.703386: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed059100 of size 256 next 75
    2020-05-12 13:50:02.703396: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed059200 of size 256 next 76
    2020-05-12 13:50:02.703406: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed059300 of size 256 next 77
    2020-05-12 13:50:02.703416: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed059400 of size 256 next 78
    2020-05-12 13:50:02.703426: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed059500 of size 256 next 79
    2020-05-12 13:50:02.703435: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed059600 of size 256 next 80
    2020-05-12 13:50:02.703445: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed059700 of size 256 next 81
    2020-05-12 13:50:02.703456: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed059800 of size 256 next 82
    2020-05-12 13:50:02.703466: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed059900 of size 256 next 83
    2020-05-12 13:50:02.703474: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed059a00 of size 256 next 84
    2020-05-12 13:50:02.703483: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed059b00 of size 256 next 85
    2020-05-12 13:50:02.703493: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed059c00 of size 256 next 86
    2020-05-12 13:50:02.703502: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed059d00 of size 256 next 87
    2020-05-12 13:50:02.703512: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed059e00 of size 256 next 88
    2020-05-12 13:50:02.703522: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed059f00 of size 256 next 89
    2020-05-12 13:50:02.703533: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05a000 of size 256 next 90
    2020-05-12 13:50:02.703543: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05a100 of size 256 next 91
    2020-05-12 13:50:02.703552: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05a200 of size 256 next 92
    2020-05-12 13:50:02.703562: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05a300 of size 256 next 93
    2020-05-12 13:50:02.703572: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05a400 of size 256 next 94
    2020-05-12 13:50:02.703582: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05a500 of size 256 next 95
    2020-05-12 13:50:02.703591: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05a600 of size 256 next 96
    2020-05-12 13:50:02.703599: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05a700 of size 256 next 97
    2020-05-12 13:50:02.703609: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05a800 of size 256 next 98
    2020-05-12 13:50:02.703618: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05a900 of size 256 next 99
    2020-05-12 13:50:02.703629: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05aa00 of size 256 next 100
    2020-05-12 13:50:02.703641: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05ab00 of size 256 next 101
    2020-05-12 13:50:02.703652: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05ac00 of size 256 next 102
    2020-05-12 13:50:02.703662: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05ad00 of size 256 next 103
    2020-05-12 13:50:02.703673: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05ae00 of size 256 next 104
    2020-05-12 13:50:02.703681: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05af00 of size 256 next 105
    2020-05-12 13:50:02.703690: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05b000 of size 256 next 106
    2020-05-12 13:50:02.703698: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05b100 of size 256 next 107
    2020-05-12 13:50:02.703707: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05b200 of size 256 next 108
    2020-05-12 13:50:02.703714: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05b300 of size 256 next 109
    2020-05-12 13:50:02.703722: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05b400 of size 256 next 110
    2020-05-12 13:50:02.703730: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05b500 of size 256 next 111
    2020-05-12 13:50:02.703742: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05b600 of size 256 next 112
    2020-05-12 13:50:02.703750: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05b700 of size 256 next 113
    2020-05-12 13:50:02.703758: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05b800 of size 256 next 114
    2020-05-12 13:50:02.703767: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05b900 of size 256 next 115
    2020-05-12 13:50:02.703775: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05ba00 of size 256 next 116
    2020-05-12 13:50:02.703783: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05bb00 of size 256 next 117
    2020-05-12 13:50:02.703792: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05bc00 of size 256 next 118
    2020-05-12 13:50:02.703800: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05bd00 of size 256 next 119
    2020-05-12 13:50:02.703810: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05be00 of size 256 next 120
    2020-05-12 13:50:02.703817: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05bf00 of size 256 next 121
    2020-05-12 13:50:02.703825: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05c000 of size 256 next 122
    2020-05-12 13:50:02.703832: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05c100 of size 256 next 123
    2020-05-12 13:50:02.703840: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05c200 of size 256 next 124
    2020-05-12 13:50:02.703848: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05c300 of size 256 next 125
    2020-05-12 13:50:02.703856: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05c400 of size 256 next 126
    2020-05-12 13:50:02.703862: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05c500 of size 256 next 127
    2020-05-12 13:50:02.703871: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05c600 of size 256 next 128
    2020-05-12 13:50:02.703879: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05c700 of size 256 next 129
    2020-05-12 13:50:02.703887: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05c800 of size 256 next 130
    2020-05-12 13:50:02.703894: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05c900 of size 256 next 131
    2020-05-12 13:50:02.703902: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05ca00 of size 256 next 132
    2020-05-12 13:50:02.703910: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05cb00 of size 256 next 133
    2020-05-12 13:50:02.703918: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05cc00 of size 256 next 134
    2020-05-12 13:50:02.703926: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05cd00 of size 256 next 135
    2020-05-12 13:50:02.703935: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05ce00 of size 256 next 136
    2020-05-12 13:50:02.703942: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05cf00 of size 256 next 137
    2020-05-12 13:50:02.703950: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05d000 of size 256 next 138
    2020-05-12 13:50:02.703959: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05d100 of size 256 next 139
    2020-05-12 13:50:02.703965: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05d200 of size 256 next 140
    2020-05-12 13:50:02.703972: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05d300 of size 256 next 141
    2020-05-12 13:50:02.703978: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05d400 of size 256 next 142
    2020-05-12 13:50:02.703986: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05d500 of size 256 next 143
    2020-05-12 13:50:02.703995: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05d600 of size 256 next 144
    2020-05-12 13:50:02.704001: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05d700 of size 256 next 145
    2020-05-12 13:50:02.704010: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05d800 of size 256 next 146
    2020-05-12 13:50:02.704020: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05d900 of size 256 next 147
    2020-05-12 13:50:02.704032: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05da00 of size 256 next 148
    2020-05-12 13:50:02.704045: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05db00 of size 256 next 149
    2020-05-12 13:50:02.704053: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05dc00 of size 256 next 150
    2020-05-12 13:50:02.704063: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05dd00 of size 256 next 151
    2020-05-12 13:50:02.704074: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05de00 of size 256 next 152
    2020-05-12 13:50:02.704085: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05df00 of size 256 next 153
    2020-05-12 13:50:02.704102: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05e000 of size 256 next 154
    2020-05-12 13:50:02.704115: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05e100 of size 256 next 155
    2020-05-12 13:50:02.704125: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05e200 of size 256 next 156
    2020-05-12 13:50:02.704135: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05e300 of size 256 next 157
    2020-05-12 13:50:02.704147: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05e400 of size 256 next 158
    2020-05-12 13:50:02.704157: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05e500 of size 256 next 159
    2020-05-12 13:50:02.704167: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05e600 of size 256 next 160
    2020-05-12 13:50:02.704178: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05e700 of size 256 next 161
    2020-05-12 13:50:02.704190: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05e800 of size 256 next 162
    2020-05-12 13:50:02.704199: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05e900 of size 256 next 163
    2020-05-12 13:50:02.704208: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05ea00 of size 256 next 164
    2020-05-12 13:50:02.704216: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05eb00 of size 256 next 165
    2020-05-12 13:50:02.704227: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05ec00 of size 256 next 166
    2020-05-12 13:50:02.704235: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05ed00 of size 256 next 167
    2020-05-12 13:50:02.704243: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05ee00 of size 256 next 168
    2020-05-12 13:50:02.704251: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05ef00 of size 256 next 169
    2020-05-12 13:50:02.704260: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05f000 of size 256 next 170
    2020-05-12 13:50:02.704271: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05f100 of size 256 next 171
    2020-05-12 13:50:02.704282: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05f200 of size 256 next 172
    2020-05-12 13:50:02.704293: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05f300 of size 256 next 173
    2020-05-12 13:50:02.704304: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05f400 of size 256 next 174
    2020-05-12 13:50:02.704315: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05f500 of size 256 next 175
    2020-05-12 13:50:02.704326: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05f600 of size 256 next 176
    2020-05-12 13:50:02.704337: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05f700 of size 256 next 177
    2020-05-12 13:50:02.704345: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05f800 of size 256 next 178
    2020-05-12 13:50:02.704354: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05f900 of size 256 next 179
    2020-05-12 13:50:02.704362: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05fa00 of size 256 next 180
    2020-05-12 13:50:02.704370: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05fb00 of size 256 next 181
    2020-05-12 13:50:02.704379: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05fc00 of size 134217728 next 182
    2020-05-12 13:50:02.704387: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4f505fc00 of size 134217728 next 183
    2020-05-12 13:50:02.704400: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4fd05fc00 of size 134217728 next 184
    2020-05-12 13:50:02.704410: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef50505fc00 of size 134217728 next 185
    2020-05-12 13:50:02.704421: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef50d05fc00 of size 134217728 next 186
    2020-05-12 13:50:02.704432: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef51505fc00 of size 256 next 187
    2020-05-12 13:50:02.704444: I tensorflow/core/common_runtime/bfc_allocator.cc:950] Free  at 7ef51505fd00 of size 131072 next 188
    2020-05-12 13:50:02.704460: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef51507fd00 of size 1024 next 192
    2020-05-12 13:50:02.704472: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef515080100 of size 256 next 193
    2020-05-12 13:50:02.704482: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef515080200 of size 512 next 194
    2020-05-12 13:50:02.704493: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef515080400 of size 65792 next 196
    2020-05-12 13:50:02.704504: I tensorflow/core/common_runtime/bfc_allocator.cc:950] Free  at 7ef515090500 of size 12800 next 199
    2020-05-12 13:50:02.704515: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef515093700 of size 1024 next 200
    2020-05-12 13:50:02.704527: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef515093b00 of size 1024 next 201
    2020-05-12 13:50:02.704537: I tensorflow/core/common_runtime/bfc_allocator.cc:950] Free  at 7ef515093f00 of size 211968 next 207
    2020-05-12 13:50:02.704546: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef5150c7b00 of size 131072 next 208
    2020-05-12 13:50:02.704554: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef5150e7b00 of size 131072 next 209
    2020-05-12 13:50:02.704562: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef515107b00 of size 512 next 210
    2020-05-12 13:50:02.704570: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef515107d00 of size 512 next 212
    2020-05-12 13:50:02.704578: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef515107f00 of size 1280 next 213
    2020-05-12 13:50:02.704588: I tensorflow/core/common_runtime/bfc_allocator.cc:950] Free  at 7ef515108400 of size 4916224 next 215
    2020-05-12 13:50:02.704596: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef5155b8800 of size 512 next 216
    2020-05-12 13:50:02.704604: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef5155b8a00 of size 1024 next 217
    2020-05-12 13:50:02.704612: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef5155b8e00 of size 512 next 218
    2020-05-12 13:50:02.704620: I tensorflow/core/common_runtime/bfc_allocator.cc:950] Free  at 7ef5155b9000 of size 6144 next 223
    2020-05-12 13:50:02.704630: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef5155ba800 of size 1024 next 224
    2020-05-12 13:50:02.704643: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef5155bac00 of size 1024 next 225
    2020-05-12 13:50:02.704656: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef5155bb000 of size 512 next 226
    2020-05-12 13:50:02.704667: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef5155bb200 of size 1024 next 304
    2020-05-12 13:50:02.704676: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef5155bb600 of size 1024 next 319
    2020-05-12 13:50:02.704686: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef5155bba00 of size 1024 next 283
    2020-05-12 13:50:02.704694: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef5155bbe00 of size 1024 next 249
    2020-05-12 13:50:02.704703: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef5155bc200 of size 1024 next 268
    2020-05-12 13:50:02.704713: I tensorflow/core/common_runtime/bfc_allocator.cc:950] Free  at 7ef5155bc600 of size 1536 next 236
    2020-05-12 13:50:02.704722: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef5155bcc00 of size 512 next 237
    2020-05-12 13:50:02.704731: I tensorflow/core/common_runtime/bfc_allocator.cc:950] Free  at 7ef5155bce00 of size 10380800 next 259
    2020-05-12 13:50:02.704740: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef515fa3400 of size 512 next 260
    2020-05-12 13:50:02.704748: I tensorflow/core/common_runtime/bfc_allocator.cc:950] Free  at 7ef515fa3600 of size 398592 next 271
    2020-05-12 13:50:02.704764: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef516004b00 of size 1024 next 272
    2020-05-12 13:50:02.704773: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef516004f00 of size 1280 next 274
    2020-05-12 13:50:02.704782: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef516005400 of size 512 next 372
    2020-05-12 13:50:02.704792: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef516005600 of size 512 next 275
    2020-05-12 13:50:02.704803: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef516005800 of size 512 next 277
    2020-05-12 13:50:02.704812: I tensorflow/core/common_runtime/bfc_allocator.cc:950] Free  at 7ef516005a00 of size 197632 next 280
    2020-05-12 13:50:02.704824: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef516035e00 of size 512 next 281
    2020-05-12 13:50:02.704833: I tensorflow/core/common_runtime/bfc_allocator.cc:950] Free  at 7ef516036000 of size 280320 next 295
    2020-05-12 13:50:02.704844: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef51607a700 of size 65536 next 296
    2020-05-12 13:50:02.704854: I tensorflow/core/common_runtime/bfc_allocator.cc:950] Free  at 7ef51608a700 of size 135424 next 306
    2020-05-12 13:50:02.704865: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef5160ab800 of size 512 next 284
    2020-05-12 13:50:02.704874: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef5160aba00 of size 512 next 307
    2020-05-12 13:50:02.704884: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef5160abc00 of size 65536 next 308
    2020-05-12 13:50:02.704893: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef5160bbc00 of size 1024 next 324
    2020-05-12 13:50:02.704904: I tensorflow/core/common_runtime/bfc_allocator.cc:950] Free  at 7ef5160bc000 of size 1024 next 343
    2020-05-12 13:50:02.704914: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef5160bc400 of size 256 next 312
    2020-05-12 13:50:02.704927: I tensorflow/core/common_runtime/bfc_allocator.cc:950] Free  at 7ef5160bc500 of size 1792 next 297
    2020-05-12 13:50:02.704936: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef5160bcc00 of size 256 next 222
    2020-05-12 13:50:02.704945: I tensorflow/core/common_runtime/bfc_allocator.cc:950] Free  at 7ef5160bcd00 of size 2304 next 278
    2020-05-12 13:50:02.704954: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef5160bd600 of size 256 next 240
    2020-05-12 13:50:02.704961: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef5160bd700 of size 256 next 286
    2020-05-12 13:50:02.704970: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef5160bd800 of size 256 next 287
    2020-05-12 13:50:02.704978: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef5160bd900 of size 256 next 393
    2020-05-12 13:50:02.704986: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef5160bda00 of size 256 next 339
    2020-05-12 13:50:02.704995: I tensorflow/core/common_runtime/bfc_allocator.cc:950] Free  at 7ef5160bdb00 of size 4864 next 314
    2020-05-12 13:50:02.705003: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef5160bee00 of size 131072 next 315
    2020-05-12 13:50:02.705012: I tensorflow/core/common_runtime/bfc_allocator.cc:950] Free  at 7ef5160dee00 of size 14336 next 316
    2020-05-12 13:50:02.705020: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef5160e2600 of size 256 next 317
    2020-05-12 13:50:02.705029: I tensorflow/core/common_runtime/bfc_allocator.cc:950] Free  at 7ef5160e2700 of size 17408 next 328
    2020-05-12 13:50:02.705038: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef5160e6b00 of size 65536 next 329
    2020-05-12 13:50:02.705047: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef5160f6b00 of size 65536 next 330
    2020-05-12 13:50:02.705063: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef516106b00 of size 131072 next 331
    2020-05-12 13:50:02.705077: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef516126b00 of size 512 next 333
    2020-05-12 13:50:02.705088: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef516126d00 of size 256 next 232
    2020-05-12 13:50:02.705098: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef516126e00 of size 256 next 334
    2020-05-12 13:50:02.705108: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef516126f00 of size 256 next 205
    2020-05-12 13:50:02.705119: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef516127000 of size 256 next 189
    2020-05-12 13:50:02.705128: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef516127100 of size 256 next 279
    2020-05-12 13:50:02.705137: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef516127200 of size 256 next 252
    2020-05-12 13:50:02.705150: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef516127300 of size 256 next 335
    2020-05-12 13:50:02.705161: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef516127400 of size 131072 next 336
    2020-05-12 13:50:02.705171: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef516147400 of size 256 next 337
    2020-05-12 13:50:02.705182: I tensorflow/core/common_runtime/bfc_allocator.cc:950] Free  at 7ef516147500 of size 4926976 next 345
    2020-05-12 13:50:02.705191: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef5165fa300 of size 65536 next 346
    2020-05-12 13:50:02.705199: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef51660a300 of size 512 next 347
    2020-05-12 13:50:02.705210: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef51660a500 of size 1280 next 349
    2020-05-12 13:50:02.705220: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef51660aa00 of size 131072 next 350
    2020-05-12 13:50:02.705233: I tensorflow/core/common_runtime/bfc_allocator.cc:950] Free  at 7ef51662aa00 of size 65536 next 351
    2020-05-12 13:50:02.705242: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef51663aa00 of size 512 next 353
    2020-05-12 13:50:02.705255: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef51663ac00 of size 512 next 354
    2020-05-12 13:50:02.705266: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef51663ae00 of size 512 next 377
    2020-05-12 13:50:02.705277: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef51663b000 of size 512 next 355
    2020-05-12 13:50:02.705287: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef51663b200 of size 1792 next 362
    2020-05-12 13:50:02.705296: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef51663b900 of size 1024 next 363
    2020-05-12 13:50:02.705306: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef51663bd00 of size 512 next 364
    2020-05-12 13:50:02.705317: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef51663bf00 of size 1024 next 365
    2020-05-12 13:50:02.705326: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef51663c300 of size 65536 next 366
    2020-05-12 13:50:02.705337: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef51664c300 of size 512 next 367
    2020-05-12 13:50:02.705347: I tensorflow/core/common_runtime/bfc_allocator.cc:950] Free  at 7ef51664c500 of size 68096 next 378
    2020-05-12 13:50:02.705358: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef51665cf00 of size 65536 next 379
    2020-05-12 13:50:02.705367: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef51666cf00 of size 131072 next 380
    2020-05-12 13:50:02.705375: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef51668cf00 of size 1024 next 381
    2020-05-12 13:50:02.705383: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef51668d300 of size 512 next 382
    2020-05-12 13:50:02.705392: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef51668d500 of size 1024 next 383
    2020-05-12 13:50:02.705401: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef51668d900 of size 512 next 290
    2020-05-12 13:50:02.705416: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef51668db00 of size 768 next 387
    2020-05-12 13:50:02.705428: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef51668de00 of size 512 next 388
    2020-05-12 13:50:02.705437: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef51668e000 of size 1792 next 395
    2020-05-12 13:50:02.705449: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef51668e700 of size 512 next 396
    2020-05-12 13:50:02.705458: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef51668e900 of size 1024 next 397
    2020-05-12 13:50:02.705466: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef51668ed00 of size 65536 next 398
    2020-05-12 13:50:02.705474: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef51669ed00 of size 131072 next 399
    2020-05-12 13:50:02.705486: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef5166bed00 of size 256 next 400
    2020-05-12 13:50:02.705500: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef5166bee00 of size 512 next 401
    2020-05-12 13:50:02.705510: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef5166bf000 of size 256 next 298
    2020-05-12 13:50:02.705519: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef5166bf100 of size 256 next 198
    2020-05-12 13:50:02.705528: I tensorflow/core/common_runtime/bfc_allocator.cc:950] Free  at 7ef5166bf200 of size 512 next 402
    2020-05-12 13:50:02.705538: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef5166bf400 of size 1792 next 409
    2020-05-12 13:50:02.705546: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef5166bfb00 of size 131072 next 410
    2020-05-12 13:50:02.705555: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef5166dfb00 of size 1024 next 411
    2020-05-12 13:50:02.705563: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef5166dff00 of size 512 next 412
    2020-05-12 13:50:02.705572: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef5166e0100 of size 1536 next 413
    2020-05-12 13:50:02.705581: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef5166e0700 of size 512 next 414
    2020-05-12 13:50:02.705597: I tensorflow/core/common_runtime/bfc_allocator.cc:950] Free  at 7ef5166e0900 of size 66048 next 416
    2020-05-12 13:50:02.705606: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef5166f0b00 of size 131072 next 417
    2020-05-12 13:50:02.705614: I tensorflow/core/common_runtime/bfc_allocator.cc:950] Free  at 7ef516710b00 of size 16580608 next 253
    2020-05-12 13:50:02.705623: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef5176e0b00 of size 2097152 next 256
    2020-05-12 13:50:02.705634: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef5178e0b00 of size 2097152 next 250
    2020-05-12 13:50:02.705644: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef517ae0b00 of size 2097152 next 348
    2020-05-12 13:50:02.705654: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef517ce0b00 of size 2097152 next 230
    2020-05-12 13:50:02.705664: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef517ee0b00 of size 2097152 next 390
    2020-05-12 13:50:02.705674: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef5180e0b00 of size 2097152 next 326
    2020-05-12 13:50:02.705682: I tensorflow/core/common_runtime/bfc_allocator.cc:950] Free  at 7ef5182e0b00 of size 3129344 next 276
    2020-05-12 13:50:02.705692: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef5185dcb00 of size 2359296 next 288
    2020-05-12 13:50:02.705703: I tensorflow/core/common_runtime/bfc_allocator.cc:950] Free  at 7ef51881cb00 of size 1228800 next 415
    2020-05-12 13:50:02.705714: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef518948b00 of size 2097152 next 263
    2020-05-12 13:50:02.705724: I tensorflow/core/common_runtime/bfc_allocator.cc:950] Free  at 7ef518b48b00 of size 2457600 next 384
    2020-05-12 13:50:02.705735: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef518da0b00 of size 134217728 next 231
    2020-05-12 13:50:02.705745: I tensorflow/core/common_runtime/bfc_allocator.cc:950] Free  at 7ef520da0b00 of size 4915200 next 221
    2020-05-12 13:50:02.705756: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef521250b00 of size 2097152 next 332
    2020-05-12 13:50:02.705767: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef521450b00 of size 2097152 next 195
    2020-05-12 13:50:02.705779: I tensorflow/core/common_runtime/bfc_allocator.cc:950] Free  at 7ef521650b00 of size 29360128 next 227
    2020-05-12 13:50:02.705790: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef523250b00 of size 134217728 next 293
    2020-05-12 13:50:02.705800: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef52b250b00 of size 134217728 next 294
    2020-05-12 13:50:02.705811: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef533250b00 of size 268435456 next 257
    2020-05-12 13:50:02.705822: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef543250b00 of size 268435456 next 289
    2020-05-12 13:50:02.705832: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef553250b00 of size 268435456 next 299
    2020-05-12 13:50:02.705844: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef563250b00 of size 536870912 next 318
    2020-05-12 13:50:02.705853: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef583250b00 of size 536870912 next 251
    2020-05-12 13:50:02.705864: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef5a3250b00 of size 536870912 next 261
    2020-05-12 13:50:02.705873: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef5c3250b00 of size 268435456 next 291
    2020-05-12 13:50:02.705886: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef5d3250b00 of size 268435456 next 247
    2020-05-12 13:50:02.705895: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef5e3250b00 of size 536870912 next 220
    2020-05-12 13:50:02.705906: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef603250b00 of size 536870912 next 235
    2020-05-12 13:50:02.705915: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef623250b00 of size 536870912 next 369
    2020-05-12 13:50:02.705923: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef643250b00 of size 134217728 next 320
    2020-05-12 13:50:02.705932: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef64b250b00 of size 134217728 next 321
    2020-05-12 13:50:02.705940: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef653250b00 of size 134217728 next 309
    2020-05-12 13:50:02.705952: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef65b250b00 of size 268435456 next 403
    2020-05-12 13:50:02.705961: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef66b250b00 of size 268435456 next 404
    2020-05-12 13:50:02.705970: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef67b250b00 of size 268435456 next 389
    2020-05-12 13:50:02.705978: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef68b250b00 of size 536870912 next 373
    2020-05-12 13:50:02.705986: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef6ab250b00 of size 536870912 next 356
    2020-05-12 13:50:02.705996: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef6cb250b00 of size 536870912 next 357
    2020-05-12 13:50:02.706005: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef6eb250b00 of size 134217728 next 371
    2020-05-12 13:50:02.706015: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef6f3250b00 of size 134217728 next 303
    2020-05-12 13:50:02.706024: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef6fb250b00 of size 268435456 next 386
    2020-05-12 13:50:02.706033: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef70b250b00 of size 268435456 next 370
    2020-05-12 13:50:02.706045: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef71b250b00 of size 268435456 next 352
    2020-05-12 13:50:02.706054: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef72b250b00 of size 536870912 next 385
    2020-05-12 13:50:02.706060: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef74b250b00 of size 536870912 next 302
    2020-05-12 13:50:02.706071: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef76b250b00 of size 536870912 next 267
    2020-05-12 13:50:02.706081: I tensorflow/core/common_runtime/bfc_allocator.cc:950] Free  at 7ef78b250b00 of size 33554432 next 203
    2020-05-12 13:50:02.706090: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef78d250b00 of size 134217728 next 340
    2020-05-12 13:50:02.706099: I tensorflow/core/common_runtime/bfc_allocator.cc:950] Free  at 7ef795250b00 of size 134217728 next 344
    2020-05-12 13:50:02.706109: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef79d250b00 of size 268435456 next 202
    2020-05-12 13:50:02.706117: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef7ad250b00 of size 268435456 next 191
    2020-05-12 13:50:02.706127: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef7bd250b00 of size 268435456 next 408
    2020-05-12 13:50:02.706136: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef7cd250b00 of size 536870912 next 305
    2020-05-12 13:50:02.706148: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef7ed250b00 of size 536870912 next 394
    2020-05-12 13:50:02.706158: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef80d250b00 of size 536870912 next 361
    2020-05-12 13:50:02.706169: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef82d250b00 of size 134217728 next 197
    2020-05-12 13:50:02.706226: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef835250b00 of size 134217728 next 376
    2020-05-12 13:50:02.706239: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef83d250b00 of size 134217728 next 338
    2020-05-12 13:50:02.706251: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef845250b00 of size 134217728 next 325
    2020-05-12 13:50:02.706264: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef84d250b00 of size 134217728 next 368
    2020-05-12 13:50:02.706281: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef855250b00 of size 536870912 next 238
    2020-05-12 13:50:02.706296: I tensorflow/core/common_runtime/bfc_allocator.cc:950] Free  at 7ef875250b00 of size 516896000 next 18446744073709551615
    2020-05-12 13:50:02.706307: I tensorflow/core/common_runtime/bfc_allocator.cc:955]      Summary of in-use Chunks by size: 
    2020-05-12 13:50:02.706322: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 176 Chunks of size 256 totalling 44.0KiB
    2020-05-12 13:50:02.706335: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 30 Chunks of size 512 totalling 15.0KiB
    2020-05-12 13:50:02.706349: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 1 Chunks of size 768 totalling 768B
    2020-05-12 13:50:02.706363: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 20 Chunks of size 1024 totalling 20.0KiB
    2020-05-12 13:50:02.706373: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 4 Chunks of size 1280 totalling 5.0KiB
    2020-05-12 13:50:02.706382: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 2 Chunks of size 1536 totalling 3.0KiB
    2020-05-12 13:50:02.706390: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 3 Chunks of size 1792 totalling 5.2KiB
    2020-05-12 13:50:02.706399: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 1 Chunks of size 4096 totalling 4.0KiB
    2020-05-12 13:50:02.706408: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 8 Chunks of size 65536 totalling 512.0KiB
    2020-05-12 13:50:02.706416: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 1 Chunks of size 65792 totalling 64.2KiB
    2020-05-12 13:50:02.706427: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 10 Chunks of size 131072 totalling 1.25MiB
    2020-05-12 13:50:02.706435: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 12 Chunks of size 262144 totalling 3.00MiB
    2020-05-12 13:50:02.706444: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 2 Chunks of size 409600 totalling 800.0KiB
    2020-05-12 13:50:02.706452: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 2 Chunks of size 1228800 totalling 2.34MiB
    2020-05-12 13:50:02.706461: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 3 Chunks of size 2048000 totalling 5.86MiB
    2020-05-12 13:50:02.706471: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 10 Chunks of size 2097152 totalling 20.00MiB
    2020-05-12 13:50:02.706479: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 1 Chunks of size 2359296 totalling 2.25MiB
    2020-05-12 13:50:02.706488: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 1 Chunks of size 2457600 totalling 2.34MiB
    2020-05-12 13:50:02.706502: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 19 Chunks of size 134217728 totalling 2.38GiB
    2020-05-12 13:50:02.706509: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 14 Chunks of size 268435456 totalling 3.50GiB
    2020-05-12 13:50:02.706522: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 16 Chunks of size 536870912 totalling 8.00GiB
    2020-05-12 13:50:02.706533: I tensorflow/core/common_runtime/bfc_allocator.cc:962] Sum Total of in-use chunks: 13.91GiB
    2020-05-12 13:50:02.706542: I tensorflow/core/common_runtime/bfc_allocator.cc:964] total_region_allocated_bytes_: 15702704128 memory_limit_: 15702704128 available bytes: 0 curr_region_allocation_bytes_: 31405408256
    2020-05-12 13:50:02.706560: I tensorflow/core/common_runtime/bfc_allocator.cc:970] Stats: 
    Limit:                 15702704128
    InUse:                 14938522880
    MaxInUse:              15074972928
    NumAllocs:                     946
    MaxAllocSize:            536870912
    
    2020-05-12 13:50:02.706595: W tensorflow/core/common_runtime/bfc_allocator.cc:429] *************************************************************************************************___
    2020-05-12 13:50:02.706629: W tensorflow/core/framework/op_kernel.cc:1655] OP_REQUIRES failed at cwise_ops_common.h:263 : Resource exhausted: OOM when allocating tensor with shape[1,1024,512,256] and type float on /job:local/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
    2020-05-12 13:50:10.736175: I lingvo/core/ops/record_yielder.cc:407] 0x7ef8a34d9f20Basic record yielder exit
    E0512 13:50:11.172914 139606888535808 base_runner.py:244] trainer done (fatal error): <class 'tensorflow.python.framework.errors_impl.ResourceExhaustedError'>
    I0512 13:50:11.175342 139606888535808 base_runner.py:111] trainer exception: From /job:local/replica:0/task:0:
    OOM when allocating tensor with shape[1,1024,512,256] and type float on /job:local/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
             [[node gradients/fprop/starnet/tower_0_0/feat/validated_gin/gin/gin_concat/seq/gin_concat/seq/gin_concat/seq/gin_concat/seq/gin_concat/seq/validated_gin_intermediate/gin_intermediate/validated_mlp/mlp/validated_validated_l001/validated_l001/l001/key_seq/bn/bn/sub_2_grad/Neg (defined at tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/core/py_utils.py:1923) ]]
    Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
    
    
    Original stack trace for 'gradients/fprop/starnet/tower_0_0/feat/validated_gin/gin/gin_concat/seq/gin_concat/seq/gin_concat/seq/gin_concat/seq/gin_concat/seq/validated_gin_intermediate/gin_intermediate/validated_mlp/mlp/validated_validated_l001/validated_l001/l001/key_seq/bn/bn/sub_2_grad/Neg':
      File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/trainer.py", line 1874, in <module>
        tf.app.run(main)
      File "usr/local/lib/python3.6/dist-packages/tensorflow_core/python/platform/app.py", line 40, in run
        _run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef)
      File "usr/local/lib/python3.6/dist-packages/absl/app.py", line 299, in run
        _run_main(main, args)
      File "usr/local/lib/python3.6/dist-packages/absl/app.py", line 250, in _run_main
        sys.exit(main(argv))
      File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/trainer.py", line 1865, in main
        RunnerManager(FLAGS.model).Start()
      File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/trainer.py", line 1861, in Start
        self.StartRunners(self.CreateRunners(FLAGS.job.split(','), FLAGS.logdir))
      File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/trainer.py", line 1609, in CreateRunners
        trial)
      File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/trainer.py", line 1562, in _CreateRunner
        return self.Trainer(cfg, *common_args)
      File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/trainer.py", line 398, in __init__
        self._model.ConstructFPropBPropGraph()
      File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/core/base_model.py", line 1197, in ConstructFPropBPropGraph
        self._task.BProp()
      File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/core/base_model.py", line 538, in BProp
        self._BPropForVariables(self.vars)
      File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/core/base_model.py", line 568, in _BPropForVariables
        gradient_adjuster=self.AdjustGradients)
      File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/core/learner.py", line 189, in Apply
        skip_zero_gradients=p.skip_zero_gradients)
      File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/core/optimizer.py", line 54, in ComputeGradients
        return py_utils.ComputeGradients(loss, vmap, *args, **kwargs)
      File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/core/py_utils.py", line 2143, in ComputeGradients
        colocate_gradients_with_ops, gate_gradients)
      File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/core/py_utils.py", line 1923, in ComputeGradientsSimple
        gate_gradients=gate_gradients)
      File "usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/gradients_impl.py", line 158, in gradients
        unconnected_gradients)
      File "usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/gradients_util.py", line 669, in _GradientsHelper
        lambda: grad_fn(op, *out_grads))
      File "usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/gradients_util.py", line 336, in _MaybeCompile
        return grad_fn()  # Exit early
      File "usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/gradients_util.py", line 669, in <lambda>
        lambda: grad_fn(op, *out_grads))
      File "usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/math_grad.py", line 1201, in _SubGrad
        gy = array_ops.reshape(math_ops.reduce_sum(-grad, ry), sy)
      File "usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/gen_math_ops.py", line 6304, in neg
        "Neg", x=x, name=name)
      File "usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/op_def_library.py", line 742, in _apply_op_helper
        attrs=attr_protos, op_def=op_def)
      File "usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/ops.py", line 3322, in _create_op_internal
        op_def=op_def)
      File "usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/ops.py", line 1756, in __init__
        self._traceback = tf_stack.extract_stack()
    
    ...which was originally created as op 'fprop/starnet/tower_0_0/feat/validated_gin/gin/gin_concat/seq/gin_concat/seq/gin_concat/seq/gin_concat/seq/gin_concat/seq/validated_gin_intermediate/gin_intermediate/validated_mlp/mlp/validated_validated_l001/validated_l001/l001/key_seq/bn/bn/sub_2', defined at:
      File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/trainer.py", line 1874, in <module>
        tf.app.run(main)
    [elided 7 identical lines from previous traceback]
      File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/trainer.py", line 398, in __init__
        self._model.ConstructFPropBPropGraph()
      File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/core/base_model.py", line 1196, in ConstructFPropBPropGraph
        self._task.FPropDefaultTheta()
      File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/core/base_model.py", line 524, in FPropDefaultTheta
        return self.FProp(self.theta, input_batch)
      File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/core/base_model.py", line 445, in FProp
        metrics, per_example = self._FPropSplitInputBatch(theta, input_batch)
      File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/core/base_model.py", line 491, in _FPropSplitInputBatch
        metrics, per_example = self.FPropTower(theta_local, batch)
      File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/core/base_model.py", line 413, in FPropTower
        predictions = self.ComputePredictions(theta, input_batch)
      File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/tasks/car/starnet.py", line 856, in ComputePredictions
        featurized_cell = self._CellFeaturizer(theta, input_batch)
      File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/tasks/car/starnet.py", line 831, in _CellFeaturizer
        point_input)
      File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/core/builder_layers.py", line 422, in FProp
        args = ch.FProp(th, *args)
      File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/core/builder_layers.py", line 422, in FProp
        args = ch.FProp(th, *args)
      File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/core/builder_layers.py", line 422, in FProp
        args = ch.FProp(th, *args)
      File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/core/builder_layers.py", line 938, in FProp
        out = ch.FProp(th, *args)
      File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/core/builder_layers.py", line 422, in FProp
        args = ch.FProp(th, *args)
      File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/core/builder_layers.py", line 938, in FProp
        out = ch.FProp(th, *args)
      File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/core/builder_layers.py", line 422, in FProp
        args = ch.FProp(th, *args)
      File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/core/builder_layers.py", line 938, in FProp
        out = ch.FProp(th, *args)
      File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/core/builder_layers.py", line 422, in FProp
        args = ch.FProp(th, *args)
    
    
    E0512 13:50:11.177675 139606888535808 base_runner.py:251] Traceback (most recent call last):
    E0512 13:50:11.177758 139606888535808 base_runner.py:251]   File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py", line 1367, in _do_call
    E0512 13:50:11.177819 139606888535808 base_runner.py:251]     return fn(*args)
    E0512 13:50:11.177872 139606888535808 base_runner.py:251]   File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py", line 1352, in _run_fn
    E0512 13:50:11.177922 139606888535808 base_runner.py:251]     target_list, run_metadata)
    E0512 13:50:11.177970 139606888535808 base_runner.py:251]   File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py", line 1445, in _call_tf_sessionrun
    E0512 13:50:11.178018 139606888535808 base_runner.py:251]     run_metadata)
    E0512 13:50:11.178066 139606888535808 base_runner.py:251] tensorflow.python.framework.errors_impl.ResourceExhaustedError: From /job:local/replica:0/task:0:
    E0512 13:50:11.178113 139606888535808 base_runner.py:251] OOM when allocating tensor with shape[1,1024,512,256] and type float on /job:local/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
    E0512 13:50:11.178161 139606888535808 base_runner.py:251]        [[{{node gradients/fprop/starnet/tower_0_0/feat/validated_gin/gin/gin_concat/seq/gin_concat/seq/gin_concat/seq/gin_concat/seq/gin_concat/seq/validated_gin_intermediate/gin_intermediate/validated_mlp/mlp/validated_validated_l001/validated_l001/l001/key_seq/bn/bn/sub_2_grad/Neg}}]]
    E0512 13:50:11.178250 139606888535808 base_runner.py:251] Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
    E0512 13:50:11.178305 139606888535808 base_runner.py:251] 
    E0512 13:50:11.178353 139606888535808 base_runner.py:251] 
    E0512 13:50:11.178400 139606888535808 base_runner.py:251] During handling of the above exception, another exception occurred:
    E0512 13:50:11.178450 139606888535808 base_runner.py:251] 
    E0512 13:50:11.178497 139606888535808 base_runner.py:251] Traceback (most recent call last):
    E0512 13:50:11.178543 139606888535808 base_runner.py:251]   File "/tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/base_runner.py", line 193, in _RunLoop
    E0512 13:50:11.178590 139606888535808 base_runner.py:251]     loop_func(*loop_args)
    E0512 13:50:11.178636 139606888535808 base_runner.py:251]   File "/tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/trainer.py", line 504, in _Loop
    E0512 13:50:11.178682 139606888535808 base_runner.py:251]     model_task.per_example_tensors,
    E0512 13:50:11.178728 139606888535808 base_runner.py:251]   File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py", line 960, in run
    E0512 13:50:11.178775 139606888535808 base_runner.py:251]     run_metadata_ptr)
    E0512 13:50:11.178820 139606888535808 base_runner.py:251]   File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py", line 1183, in _run
    E0512 13:50:11.178866 139606888535808 base_runner.py:251]     feed_dict_tensor, options, run_metadata)
    E0512 13:50:11.178911 139606888535808 base_runner.py:251]   File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py", line 1361, in _do_run
    E0512 13:50:11.178961 139606888535808 base_runner.py:251]     run_metadata)
    E0512 13:50:11.179010 139606888535808 base_runner.py:251]   File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py", line 1386, in _do_call
    E0512 13:50:11.179056 139606888535808 base_runner.py:251]     raise type(e)(node_def, op, message)
    E0512 13:50:11.179102 139606888535808 base_runner.py:251] tensorflow.python.framework.errors_impl.ResourceExhaustedError: From /job:local/replica:0/task:0:
    E0512 13:50:11.179148 139606888535808 base_runner.py:251] OOM when allocating tensor with shape[1,1024,512,256] and type float on /job:local/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
    E0512 13:50:11.179193 139606888535808 base_runner.py:251]        [[node gradients/fprop/starnet/tower_0_0/feat/validated_gin/gin/gin_concat/seq/gin_concat/seq/gin_concat/seq/gin_concat/seq/gin_concat/seq/validated_gin_intermediate/gin_intermediate/validated_mlp/mlp/validated_validated_l001/validated_l001/l001/key_seq/bn/bn/sub_2_grad/Neg (defined at tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/core/py_utils.py:1923) ]]
    E0512 13:50:11.179267 139606888535808 base_runner.py:251] Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
    E0512 13:50:11.179316 139606888535808 base_runner.py:251] 
    E0512 13:50:11.179363 139606888535808 base_runner.py:251] 
    E0512 13:50:11.179409 139606888535808 base_runner.py:251] Original stack trace for 'gradients/fprop/starnet/tower_0_0/feat/validated_gin/gin/gin_concat/seq/gin_concat/seq/gin_concat/seq/gin_concat/seq/gin_concat/seq/validated_gin_intermediate/gin_intermediate/validated_mlp/mlp/validated_validated_l001/validated_l001/l001/key_seq/bn/bn/sub_2_grad/Neg':
    E0512 13:50:11.179456 139606888535808 base_runner.py:251]   File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/trainer.py", line 1874, in <module>
    E0512 13:50:11.179502 139606888535808 base_runner.py:251]     tf.app.run(main)
    E0512 13:50:11.179549 139606888535808 base_runner.py:251]   File "usr/local/lib/python3.6/dist-packages/tensorflow_core/python/platform/app.py", line 40, in run
    E0512 13:50:11.179595 139606888535808 base_runner.py:251]     _run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef)
    E0512 13:50:11.179647 139606888535808 base_runner.py:251]   File "usr/local/lib/python3.6/dist-packages/absl/app.py", line 299, in run
    E0512 13:50:11.179695 139606888535808 base_runner.py:251]     _run_main(main, args)
    E0512 13:50:11.179741 139606888535808 base_runner.py:251]   File "usr/local/lib/python3.6/dist-packages/absl/app.py", line 250, in _run_main
    E0512 13:50:11.179788 139606888535808 base_runner.py:251]     sys.exit(main(argv))
    E0512 13:50:11.179834 139606888535808 base_runner.py:251]   File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/trainer.py", line 1865, in main
    E0512 13:50:11.179880 139606888535808 base_runner.py:251]     RunnerManager(FLAGS.model).Start()
    E0512 13:50:11.179926 139606888535808 base_runner.py:251]   File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/trainer.py", line 1861, in Start
    E0512 13:50:11.179972 139606888535808 base_runner.py:251]     self.StartRunners(self.CreateRunners(FLAGS.job.split(','), FLAGS.logdir))
    E0512 13:50:11.180018 139606888535808 base_runner.py:251]   File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/trainer.py", line 1609, in CreateRunners
    E0512 13:50:11.180064 139606888535808 base_runner.py:251]     trial)
    E0512 13:50:11.180110 139606888535808 base_runner.py:251]   File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/trainer.py", line 1562, in _CreateRunner
    E0512 13:50:11.180155 139606888535808 base_runner.py:251]     return self.Trainer(cfg, *common_args)
    E0512 13:50:11.180201 139606888535808 base_runner.py:251]   File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/trainer.py", line 398, in __init__
    E0512 13:50:11.180247 139606888535808 base_runner.py:251]     self._model.ConstructFPropBPropGraph()
    E0512 13:50:11.180293 139606888535808 base_runner.py:251]   File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/core/base_model.py", line 1197, in ConstructFPropBPropGraph
    E0512 13:50:11.180339 139606888535808 base_runner.py:251]     self._task.BProp()
    E0512 13:50:11.180384 139606888535808 base_runner.py:251]   File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/core/base_model.py", line 538, in BProp
    E0512 13:50:11.180430 139606888535808 base_runner.py:251]     self._BPropForVariables(self.vars)
    E0512 13:50:11.180479 139606888535808 base_runner.py:251]   File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/core/base_model.py", line 568, in _BPropForVariables
    E0512 13:50:11.180526 139606888535808 base_runner.py:251]     gradient_adjuster=self.AdjustGradients)
    E0512 13:50:11.180572 139606888535808 base_runner.py:251]   File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/core/learner.py", line 189, in Apply
    E0512 13:50:11.180617 139606888535808 base_runner.py:251]     skip_zero_gradients=p.skip_zero_gradients)
    E0512 13:50:11.180663 139606888535808 base_runner.py:251]   File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/core/optimizer.py", line 54, in ComputeGradients
    E0512 13:50:11.180709 139606888535808 base_runner.py:251]     return py_utils.ComputeGradients(loss, vmap, *args, **kwargs)
    E0512 13:50:11.180755 139606888535808 base_runner.py:251]   File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/core/py_utils.py", line 2143, in ComputeGradients
    E0512 13:50:11.180800 139606888535808 base_runner.py:251]     colocate_gradients_with_ops, gate_gradients)
    E0512 13:50:11.180846 139606888535808 base_runner.py:251]   File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/core/py_utils.py", line 1923, in ComputeGradientsSimple
    E0512 13:50:11.180892 139606888535808 base_runner.py:251]     gate_gradients=gate_gradients)
    E0512 13:50:11.180938 139606888535808 base_runner.py:251]   File "usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/gradients_impl.py", line 158, in gradients
    E0512 13:50:11.180983 139606888535808 base_runner.py:251]     unconnected_gradients)
    E0512 13:50:11.181029 139606888535808 base_runner.py:251]   File "usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/gradients_util.py", line 669, in _GradientsHelper
    E0512 13:50:11.181074 139606888535808 base_runner.py:251]     lambda: grad_fn(op, *out_grads))
    E0512 13:50:11.181120 139606888535808 base_runner.py:251]   File "usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/gradients_util.py", line 336, in _MaybeCompile
    E0512 13:50:11.181165 139606888535808 base_runner.py:251]     return grad_fn()  # Exit early
    E0512 13:50:11.181211 139606888535808 base_runner.py:251]   File "usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/gradients_util.py", line 669, in <lambda>
    E0512 13:50:11.181257 139606888535808 base_runner.py:251]     lambda: grad_fn(op, *out_grads))
    E0512 13:50:11.181302 139606888535808 base_runner.py:251]   File "usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/math_grad.py", line 1201, in _SubGrad
    E0512 13:50:11.181348 139606888535808 base_runner.py:251]     gy = array_ops.reshape(math_ops.reduce_sum(-grad, ry), sy)
    E0512 13:50:11.181395 139606888535808 base_runner.py:251]   File "usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/gen_math_ops.py", line 6304, in neg
    E0512 13:50:11.181441 139606888535808 base_runner.py:251]     "Neg", x=x, name=name)
    E0512 13:50:11.181487 139606888535808 base_runner.py:251]   File "usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/op_def_library.py", line 742, in _apply_op_helper
    E0512 13:50:11.181534 139606888535808 base_runner.py:251]     attrs=attr_protos, op_def=op_def)
    E0512 13:50:11.181580 139606888535808 base_runner.py:251]   File "usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/ops.py", line 3322, in _create_op_internal
    E0512 13:50:11.181626 139606888535808 base_runner.py:251]     op_def=op_def)
    E0512 13:50:11.181672 139606888535808 base_runner.py:251]   File "usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/ops.py", line 1756, in __init__
    E0512 13:50:11.181717 139606888535808 base_runner.py:251]     self._traceback = tf_stack.extract_stack()
    E0512 13:50:11.181763 139606888535808 base_runner.py:251] 
    E0512 13:50:11.181808 139606888535808 base_runner.py:251] ...which was originally created as op 'fprop/starnet/tower_0_0/feat/validated_gin/gin/gin_concat/seq/gin_concat/seq/gin_concat/seq/gin_concat/seq/gin_concat/seq/validated_gin_intermediate/gin_intermediate/validated_mlp/mlp/validated_validated_l001/validated_l001/l001/key_seq/bn/bn/sub_2', defined at:
    E0512 13:50:11.181856 139606888535808 base_runner.py:251]   File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/trainer.py", line 1874, in <module>
    E0512 13:50:11.181903 139606888535808 base_runner.py:251]     tf.app.run(main)
    E0512 13:50:11.181948 139606888535808 base_runner.py:251] [elided 7 identical lines from previous traceback]
    E0512 13:50:11.181994 139606888535808 base_runner.py:251]   File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/trainer.py", line 398, in __init__
    E0512 13:50:11.182041 139606888535808 base_runner.py:251]     self._model.ConstructFPropBPropGraph()
    E0512 13:50:11.182086 139606888535808 base_runner.py:251]   File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/core/base_model.py", line 1196, in ConstructFPropBPropGraph
    E0512 13:50:11.182132 139606888535808 base_runner.py:251]     self._task.FPropDefaultTheta()
    E0512 13:50:11.182210 139606888535808 base_runner.py:251]   File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/core/base_model.py", line 524, in FPropDefaultTheta
    E0512 13:50:11.182261 139606888535808 base_runner.py:251]     return self.FProp(self.theta, input_batch)
    E0512 13:50:11.182308 139606888535808 base_runner.py:251]   File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/core/base_model.py", line 445, in FProp
    E0512 13:50:11.182354 139606888535808 base_runner.py:251]     metrics, per_example = self._FPropSplitInputBatch(theta, input_batch)
    E0512 13:50:11.182400 139606888535808 base_runner.py:251]   File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/core/base_model.py", line 491, in _FPropSplitInputBatch
    E0512 13:50:11.182446 139606888535808 base_runner.py:251]     metrics, per_example = self.FPropTower(theta_local, batch)
    E0512 13:50:11.182492 139606888535808 base_runner.py:251]   File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/core/base_model.py", line 413, in FPropTower
    E0512 13:50:11.182537 139606888535808 base_runner.py:251]     predictions = self.ComputePredictions(theta, input_batch)
    E0512 13:50:11.182582 139606888535808 base_runner.py:251]   File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/tasks/car/starnet.py", line 856, in ComputePredictions
    E0512 13:50:11.182628 139606888535808 base_runner.py:251]     featurized_cell = self._CellFeaturizer(theta, input_batch)
    E0512 13:50:11.182672 139606888535808 base_runner.py:251]   File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/tasks/car/starnet.py", line 831, in _CellFeaturizer
    E0512 13:50:11.182718 139606888535808 base_runner.py:251]     point_input)
    E0512 13:50:11.182763 139606888535808 base_runner.py:251]   File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/core/builder_layers.py", line 422, in FProp
    E0512 13:50:11.182809 139606888535808 base_runner.py:251]     args = ch.FProp(th, *args)
    E0512 13:50:11.182855 139606888535808 base_runner.py:251]   File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/core/builder_layers.py", line 422, in FProp
    E0512 13:50:11.182900 139606888535808 base_runner.py:251]     args = ch.FProp(th, *args)
    E0512 13:50:11.182945 139606888535808 base_runner.py:251]   File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/core/builder_layers.py", line 422, in FProp
    E0512 13:50:11.182990 139606888535808 base_runner.py:251]     args = ch.FProp(th, *args)
    E0512 13:50:11.183035 139606888535808 base_runner.py:251]   File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/core/builder_layers.py", line 938, in FProp
    E0512 13:50:11.183080 139606888535808 base_runner.py:251]     out = ch.FProp(th, *args)
    E0512 13:50:11.183126 139606888535808 base_runner.py:251]   File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/core/builder_layers.py", line 422, in FProp
    E0512 13:50:11.183171 139606888535808 base_runner.py:251]     args = ch.FProp(th, *args)
    E0512 13:50:11.183217 139606888535808 base_runner.py:251]   File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/core/builder_layers.py", line 938, in FProp
    E0512 13:50:11.183262 139606888535808 base_runner.py:251]     out = ch.FProp(th, *args)
    E0512 13:50:11.183307 139606888535808 base_runner.py:251]   File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/core/builder_layers.py", line 422, in FProp
    E0512 13:50:11.183352 139606888535808 base_runner.py:251]     args = ch.FProp(th, *args)
    E0512 13:50:11.183397 139606888535808 base_runner.py:251]   File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/core/builder_layers.py", line 938, in FProp
    E0512 13:50:11.183443 139606888535808 base_runner.py:251]     out = ch.FProp(th, *args)
    E0512 13:50:11.183488 139606888535808 base_runner.py:251]   File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/core/builder_layers.py", line 422, in FProp
    E0512 13:50:11.183534 139606888535808 base_runner.py:251]     args = ch.FProp(th, *args)
    

    Any pointers much appreciated.

    Thanks!

    opened by JWHennessey 9
  • generate_waymo_tf.py hangs

    generate_waymo_tf.py hangs

    Hi,

    Thanks for open sourcing such a great library.

    I am trying to convert some of the Waymo data tfrecord data into the tf-example format used my lingvo. The data hosted at waymo_open_dataset_v_1_0_0_tf_example_lingvo/v.1.0.0/ doesn't include the data for the domain adaption subset as far as I can tell.

    I've tried running the generate_waymo_tf.py script both with the DataflowRunner and DirectRunner. Both seem to end up hanging and never generating any outputs.

    Are there any suggestions on how to debug this?

    Here is how I run the script which should be trying to process only a single segment.

    python3 /tmp/lingvo/lingvo/tasks/car/waymo/tools/generate_waymo_tf.py \
      --input_file_pattern=gs://{bucket}/data/domain_adaptation/validation/segment-10302420246752500683_5520_000_5540_000* \
      --output_filebase=gs://{bucket}/tf_example_lingvo/domain_adaptation/validation/output -- \
      --project='{project}' \
      --temp_location=gs://{bucket}/tmp/ \
      --region 'europe-west4' \
      --zone 'europe-west4-c' \
      --num_workers 2 \
      --setup_file /tmp/lingvo/lingvo/tasks/car/waymo/tools/setup.py  
    

    The outputs:

    2020-05-08 16:06:49.830048: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libnvinfer.so.6'; dlerror: libnvinfer.so.6: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/nvidia/lib:/usr/local/nvidia/lib64
    2020-05-08 16:06:49.830188: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libnvinfer_plugin.so.6'; dlerror: libnvinfer_plugin.so.6: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/nvidia/lib:/usr/local/nvidia/lib64
    2020-05-08 16:06:49.830222: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:30] Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
    WARNING:absl:Lingvo does not support eager execution yet. Please disable eager execution with tf.compat.v1.disable_eager_execution() or proceed at your own risk.
    I0508 16:06:50.560759 139933516105536 auth.py:108] Setting socket default timeout to 60 seconds.
    I0508 16:06:50.560907 139933516105536 auth.py:111] socket default timeout is 60.0 seconds.
    I0508 16:06:50.567577 139933516105536 gcsio.py:480] Starting the size estimation of the input
    I0508 16:06:50.568466 139933516105536 transport.py:151] Attempting refresh to obtain initial access_token
    I0508 16:06:50.639814 139933516105536 gcsio.py:496] Finished listing 1 files in 0.07223939895629883 seconds.
    I0508 16:06:50.640702 139933516105536 pipeline.py:180] Missing pipeline option (runner). Executing pipeline using the default runner: DirectRunner.
    I0508 16:06:51.375959 139933516105536 fn_api_runner_transforms.py:548] ==================== <function annotate_downstream_side_inputs at 0x7f445eff9048> ====================
    I0508 16:06:51.376795 139933516105536 fn_api_runner_transforms.py:548] ==================== <function fix_side_input_pcoll_coders at 0x7f445eff9158> ====================
    I0508 16:06:51.377469 139933516105536 fn_api_runner_transforms.py:548] ==================== <function lift_combiners at 0x7f445eff91e0> ====================
    I0508 16:06:51.377765 139933516105536 fn_api_runner_transforms.py:548] ==================== <function expand_sdf at 0x7f445eff9268> ====================
    I0508 16:06:51.378669 139933516105536 fn_api_runner_transforms.py:548] ==================== <function expand_gbk at 0x7f445eff92f0> ====================
    I0508 16:06:51.379250 139933516105536 fn_api_runner_transforms.py:548] ==================== <function sink_flattens at 0x7f445eff9400> ====================
    I0508 16:06:51.379594 139933516105536 fn_api_runner_transforms.py:548] ==================== <function greedily_fuse at 0x7f445eff9488> ====================
    I0508 16:06:51.381523 139933516105536 fn_api_runner_transforms.py:548] ==================== <function read_to_impulse at 0x7f445eff9510> ====================
    I0508 16:06:51.381770 139933516105536 fn_api_runner_transforms.py:548] ==================== <function impulse_to_input at 0x7f445eff9598> ====================
    I0508 16:06:51.382106 139933516105536 fn_api_runner_transforms.py:548] ==================== <function inject_timer_pcollections at 0x7f445eff9730> ====================
    I0508 16:06:51.382540 139933516105536 fn_api_runner_transforms.py:548] ==================== <function sort_stages at 0x7f445eff97b8> ====================
    I0508 16:06:51.382743 139933516105536 fn_api_runner_transforms.py:548] ==================== <function window_pcollection_coders at 0x7f445eff9840> ====================
    I0508 16:06:51.384798 139933516105536 statecache.py:154] Creating state cache with size 100
    I0508 16:06:51.386486 139933516105536 fn_api_runner.py:2011] Created Worker handler <apache_beam.runners.portability.fn_api_runner.EmbeddedWorkerHandler object at 0x7f445ea61898> for environment urn: "beam:env:embedded_python:v1"
    
    I0508 16:06:51.386961 139933516105536 fn_api_runner.py:974] Running (((ref_AppliedPTransform_Read/Read/_SDFBoundedSourceWrapper/Impulse_5)+(Read/Read/_SDFBoundedSourceWrapper/ParDo(SDFBoundedSourceDoFn)/PairWithRestriction))+(Read/Read/_SDFBoundedSourceWrapper/ParDo(SDFBoundedSourceDoFn)/SplitAndSizeRestriction))+(ref_PCollection_PCollection_1_split/Write)
    I0508 16:06:51.425244 139933369054976 gcsio.py:480] Starting the size estimation of the input
    I0508 16:06:51.478516 139933369054976 gcsio.py:496] Finished listing 1 files in 0.05326652526855469 seconds.
    I0508 16:06:51.480220 139933369054976 gcsio.py:480] Starting the size estimation of the input
    I0508 16:06:51.527962 139933369054976 gcsio.py:496] Finished listing 1 files in 0.04774069786071777 seconds.
    I0508 16:06:51.528657 139933369054976 gcsio.py:480] Starting the size estimation of the input
    I0508 16:06:51.575519 139933369054976 gcsio.py:496] Finished listing 1 files in 0.04685568809509277 seconds.
    I0508 16:06:51.604114 139933516105536 fn_api_runner.py:974] Running (((((ref_AppliedPTransform_Write/Write/WriteImpl/DoOnce/Impulse_12)+(ref_AppliedPTransform_Write/Write/WriteImpl/DoOnce/FlatMap(<lambda at core.py:2639>)_13))+(ref_AppliedPTransform_Write/Write/WriteImpl/DoOnce/Map(decode)_15))+(ref_AppliedPTransform_Write/Write/WriteImpl/InitializeWrite_16))+(ref_PCollection_PCollection_6/Write))+(ref_PCollection_PCollection_7/Write)
    I0508 16:06:51.626923 139933516105536 fn_api_runner.py:974] Running ((((((ref_PCollection_PCollection_1_split/Read)+(Read/Read/_SDFBoundedSourceWrapper/ParDo(SDFBoundedSourceDoFn)/Process))+(ref_AppliedPTransform_ConvertToTFExample_7))+(ref_AppliedPTransform_Write/Write/WriteImpl/WriteBundles_17))+(ref_AppliedPTransform_Write/Write/WriteImpl/Pair_18))+(ref_AppliedPTransform_Write/Write/WriteImpl/WindowInto(WindowIntoFn)_19))+(Write/Write/WriteImpl/GroupByKey/Write)
    W0508 16:06:52.151952 139933369054976 tfrecordio.py:63] Couldn't find python-snappy so the implementation of _TFRecordUtil._masked_crc32c is not as fast as it could be.
    2020-05-08 16:06:53.771899: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
    2020-05-08 16:06:54.350836: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
    2020-05-08 16:06:54.351292: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1555] Found device 0 with properties: 
    pciBusID: 0000:00:04.0 name: Tesla V100-SXM2-16GB computeCapability: 7.0
    coreClock: 1.53GHz coreCount: 80 deviceMemorySize: 15.75GiB deviceMemoryBandwidth: 836.37GiB/s
    2020-05-08 16:06:54.351615: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcudart.so.10.1'; dlerror: libcudart.so.10.1: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/nvidia/lib:/usr/local/nvidia/lib64
    2020-05-08 16:06:54.351855: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcublas.so.10'; dlerror: libcublas.so.10: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/nvidia/lib:/usr/local/nvidia/lib64
    2020-05-08 16:06:54.352015: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcufft.so.10'; dlerror: libcufft.so.10: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/nvidia/lib:/usr/local/nvidia/lib64
    2020-05-08 16:06:54.352181: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcurand.so.10'; dlerror: libcurand.so.10: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/nvidia/lib:/usr/local/nvidia/lib64
    2020-05-08 16:06:54.352364: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcusolver.so.10'; dlerror: libcusolver.so.10: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/nvidia/lib:/usr/local/nvidia/lib64
    2020-05-08 16:06:54.352548: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcusparse.so.10'; dlerror: libcusparse.so.10: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/nvidia/lib:/usr/local/nvidia/lib64
    2020-05-08 16:06:54.356941: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
    2020-05-08 16:06:54.356972: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1592] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
    Skipping registering GPU devices...
    2020-05-08 16:06:54.357462: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 AVX512F FMA
    2020-05-08 16:06:54.363119: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2000175000 Hz
    2020-05-08 16:06:54.363433: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f4450fa39b0 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
    2020-05-08 16:06:54.363463: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
    2020-05-08 16:06:54.460026: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
    2020-05-08 16:06:54.460613: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f445101f5a0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
    2020-05-08 16:06:54.460666: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Tesla V100-SXM2-16GB, Compute Capability 7.0
    2020-05-08 16:06:54.460823: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1096] Device interconnect StreamExecutor with strength 1 edge matrix:
    2020-05-08 16:06:54.460847: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102]
    

    Any help much appreciated. Thanks!

    System Config

    -----------------------------------------------------------------------------+
    | NVIDIA-SMI 410.104      Driver Version: 410.104      CUDA Version: 10.0     |
    |-------------------------------+----------------------+----------------------+
    | GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
    | Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
    |===============================+======================+======================|
    |   0  Tesla V100-SXM2...  Off  | 00000000:00:04.0 Off |                    0 |
    | N/A   36C    P0    38W / 300W |    417MiB / 16130MiB |      0%      Default |
    +-------------------------------+----------------------+----------------------+
                                                                                   
    +-----------------------------------------------------------------------------+
    | Processes:                                                       GPU Memory |
    |  GPU       PID   Type   Process name                             Usage      |
    |=============================================================================|
    +-----------------------------------------------------------------------------+
    
    opened by JWHennessey 9
  • Could you mind tell me which papers are you referring to to write ASR Task?

    Could you mind tell me which papers are you referring to to write ASR Task?

    I am currently looking at the code of your ASR Task. But I found that many of the codes inside are difficult to understand, especially the decoder part. So I hope to find some references to help understand and learn. Could you mind tell me which papers are you referring to to write ASR Task? Thank you so much!!

    opened by iamxiaoyubei 9
  • Bump certifi from 2022.5.18.1 to 2022.12.7 in /docker

    Bump certifi from 2022.5.18.1 to 2022.12.7 in /docker

    Bumps certifi from 2022.5.18.1 to 2022.12.7.

    Commits

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    • @dependabot use these labels will set the current labels as the default for future PRs for this repo and language
    • @dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language
    • @dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language
    • @dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

    You can disable automated security fix PRs for this repo from the Security Alerts page.

    dependencies 
    opened by dependabot[bot] 0
  • Bump pillow from 9.1.1 to 9.3.0 in /docker

    Bump pillow from 9.1.1 to 9.3.0 in /docker

    Bumps pillow from 9.1.1 to 9.3.0.

    Release notes

    Sourced from pillow's releases.

    9.3.0

    https://pillow.readthedocs.io/en/stable/releasenotes/9.3.0.html

    Changes

    ... (truncated)

    Changelog

    Sourced from pillow's changelog.

    9.3.0 (2022-10-29)

    • Limit SAMPLESPERPIXEL to avoid runtime DOS #6700 [wiredfool]

    • Initialize libtiff buffer when saving #6699 [radarhere]

    • Inline fname2char to fix memory leak #6329 [nulano]

    • Fix memory leaks related to text features #6330 [nulano]

    • Use double quotes for version check on old CPython on Windows #6695 [hugovk]

    • Remove backup implementation of Round for Windows platforms #6693 [cgohlke]

    • Fixed set_variation_by_name offset #6445 [radarhere]

    • Fix malloc in _imagingft.c:font_setvaraxes #6690 [cgohlke]

    • Release Python GIL when converting images using matrix operations #6418 [hmaarrfk]

    • Added ExifTags enums #6630 [radarhere]

    • Do not modify previous frame when calculating delta in PNG #6683 [radarhere]

    • Added support for reading BMP images with RLE4 compression #6674 [npjg, radarhere]

    • Decode JPEG compressed BLP1 data in original mode #6678 [radarhere]

    • Added GPS TIFF tag info #6661 [radarhere]

    • Added conversion between RGB/RGBA/RGBX and LAB #6647 [radarhere]

    • Do not attempt normalization if mode is already normal #6644 [radarhere]

    ... (truncated)

    Commits

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    • @dependabot use these labels will set the current labels as the default for future PRs for this repo and language
    • @dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language
    • @dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language
    • @dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

    You can disable automated security fix PRs for this repo from the Security Alerts page.

    dependencies 
    opened by dependabot[bot] 0
  • Bump tensorflow from 2.9.1 to 2.9.3 in /docker

    Bump tensorflow from 2.9.1 to 2.9.3 in /docker

    Bumps tensorflow from 2.9.1 to 2.9.3.

    Release notes

    Sourced from tensorflow's releases.

    TensorFlow 2.9.3

    Release 2.9.3

    This release introduces several vulnerability fixes:

    TensorFlow 2.9.2

    Release 2.9.2

    This releases introduces several vulnerability fixes:

    ... (truncated)

    Changelog

    Sourced from tensorflow's changelog.

    Release 2.9.3

    This release introduces several vulnerability fixes:

    Release 2.8.4

    This release introduces several vulnerability fixes:

    ... (truncated)

    Commits
    • a5ed5f3 Merge pull request #58584 from tensorflow/vinila21-patch-2
    • 258f9a1 Update py_func.cc
    • cd27cfb Merge pull request #58580 from tensorflow-jenkins/version-numbers-2.9.3-24474
    • 3e75385 Update version numbers to 2.9.3
    • bc72c39 Merge pull request #58482 from tensorflow-jenkins/relnotes-2.9.3-25695
    • 3506c90 Update RELEASE.md
    • 8dcb48e Update RELEASE.md
    • 4f34ec8 Merge pull request #58576 from pak-laura/c2.99f03a9d3bafe902c1e6beb105b2f2417...
    • 6fc67e4 Replace CHECK with returning an InternalError on failing to create python tuple
    • 5dbe90a Merge pull request #58570 from tensorflow/r2.9-7b174a0f2e4
    • Additional commits viewable in compare view

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    • @dependabot use these labels will set the current labels as the default for future PRs for this repo and language
    • @dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language
    • @dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language
    • @dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

    You can disable automated security fix PRs for this repo from the Security Alerts page.

    dependencies 
    opened by dependabot[bot] 0
  • images

    images

    image

    How do you project the point cloud onto the image and keep only the foreground points and remove the background points, can you tell me how to do this and the corresponding code

    opened by anlent 0
  • Bump protobuf from 3.19.4 to 3.19.5 in /docker

    Bump protobuf from 3.19.4 to 3.19.5 in /docker

    Bumps protobuf from 3.19.4 to 3.19.5.

    Release notes

    Sourced from protobuf's releases.

    Protocol Buffers v3.19.5

    C++

    Commits

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    • @dependabot use these labels will set the current labels as the default for future PRs for this repo and language
    • @dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language
    • @dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language
    • @dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

    You can disable automated security fix PRs for this repo from the Security Alerts page.

    dependencies 
    opened by dependabot[bot] 0
Owner
null
Pervasive Attention: 2D Convolutional Networks for Sequence-to-Sequence Prediction

This is a fork of Fairseq(-py) with implementations of the following models: Pervasive Attention - 2D Convolutional Neural Networks for Sequence-to-Se

Maha 490 Dec 15, 2022
Sequence to Sequence Models with PyTorch

Sequence to Sequence models with PyTorch This repository contains implementations of Sequence to Sequence (Seq2Seq) models in PyTorch At present it ha

Sandeep Subramanian 708 Dec 19, 2022
Official repository of OFA. Paper: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework

Paper | Blog OFA is a unified multimodal pretrained model that unifies modalities (i.e., cross-modality, vision, language) and tasks (e.g., image gene

OFA Sys 1.4k Jan 8, 2023
An implementation of a sequence to sequence neural network using an encoder-decoder

Keras implementation of a sequence to sequence model for time series prediction using an encoder-decoder architecture. I created this post to share a

Luke Tonin 195 Dec 17, 2022
NeuPy is a Tensorflow based python library for prototyping and building neural networks

NeuPy v0.8.2 NeuPy is a python library for prototyping and building neural networks. NeuPy uses Tensorflow as a computational backend for deep learnin

Yurii Shevchuk 729 Jan 3, 2023
Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers

Segmentation Transformer Implementation of Segmentation Transformer in PyTorch, a new model to achieve SOTA in semantic segmentation while using trans

Abhay Gupta 161 Dec 8, 2022
Implementation of SETR model, Original paper: Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers.

SETR - Pytorch Since the original paper (Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers.) has no official

zhaohu xing 112 Dec 16, 2022
Understanding and Improving Encoder Layer Fusion in Sequence-to-Sequence Learning (ICLR 2021)

Understanding and Improving Encoder Layer Fusion in Sequence-to-Sequence Learning (ICLR 2021) Citation Please cite as: @inproceedings{liu2020understan

Sunbow Liu 22 Nov 25, 2022
[CVPR 2021] Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers

[CVPR 2021] Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers

Fudan Zhang Vision Group 897 Jan 5, 2023
Sequence-to-Sequence learning using PyTorch

Seq2Seq in PyTorch This is a complete suite for training sequence-to-sequence models in PyTorch. It consists of several models and code to both train

Elad Hoffer 514 Nov 17, 2022
Sequence lineage information extracted from RKI sequence data repo

Pango lineage information for German SARS-CoV-2 sequences This repository contains a join of the metadata and pango lineage tables of all German SARS-

Cornelius Roemer 24 Oct 26, 2022
API for RL algorithm design & testing of BCA (Building Control Agent) HVAC on EnergyPlus building energy simulator by wrapping their EMS Python API

RL - EmsPy (work In Progress...) The EmsPy Python package was made to facilitate Reinforcement Learning (RL) algorithm research for developing and tes

null 20 Jan 5, 2023
A PyTorch Implementation of Gated Graph Sequence Neural Networks (GGNN)

A PyTorch Implementation of GGNN This is a PyTorch implementation of the Gated Graph Sequence Neural Networks (GGNN) as described in the paper Gated G

Ching-Yao Chuang 427 Dec 13, 2022
Selene is a Python library and command line interface for training deep neural networks from biological sequence data such as genomes.

Selene is a Python library and command line interface for training deep neural networks from biological sequence data such as genomes.

Troyanskaya Laboratory 323 Jan 1, 2023
A PyTorch Implementation of Gated Graph Sequence Neural Networks (GGNN)

A PyTorch Implementation of GGNN This is a PyTorch implementation of the Gated Graph Sequence Neural Networks (GGNN) as described in the paper Gated G

Ching-Yao Chuang 427 Dec 13, 2022
A modular, research-friendly framework for high-performance and inference of sequence models at many scales

T5X T5X is a modular, composable, research-friendly framework for high-performance, configurable, self-service training, evaluation, and inference of

Google Research 1.1k Jan 8, 2023
Learning and Building Convolutional Neural Networks using PyTorch

Image Classification Using Deep Learning Learning and Building Convolutional Neural Networks using PyTorch. Models, selected are based on number of ci

Mayur 126 Dec 22, 2022
Pytorch implementation of "Forward Thinking: Building and Training Neural Networks One Layer at a Time"

forward-thinking-pytorch Pytorch implementation of Forward Thinking: Building and Training Neural Networks One Layer at a Time Requirements Python 2.7

Kim Heecheol 65 Oct 6, 2022