I have been unable to get Starnet working on TPUs so am now trying on 8 x v100 GPUs. However, I am not running into memory issues.
Are there any way to reduce the memory required of the model?
I have tried setting p.batch_size = 1
in tasks/car/params/waymo.py
but still run out of memory.
I have tried many set of training parameters but the ones used for the error below was:
bazel-bin/lingvo/trainer --logtostderr --model=car.waymo.StarNetVehicle --mode=async --logdir=gs://waymo-challange/logs/starnet.waymo.3d_object.v3 --run_locally=gpu --worker_gpus=7 --worker_split_size=7 --worker_replicas=7 --controller_gpus=1
Error:
I0512 13:49:24.586677 139606888535808 base_runner.py:111] step: 0
WARNING:tensorflow:Issue encountered when serializing __batch_norm_update_dict.
Type is unsupported, or the types of the items don't match field type in CollectionDef. Note this is a warning and probably safe to ignore.
'dict' object has no attribute 'name'
W0512 13:49:30.131858 139606896928512 meta_graph.py:436] Issue encountered when serializing __batch_norm_update_dict.
Type is unsupported, or the types of the items don't match field type in CollectionDef. Note this is a warning and probably safe to ignore.
'dict' object has no attribute 'name'
2020-05-12 13:49:30.990706: I ./lingvo/core/ops/input_common.h:74] Create RecordProcessor
2020-05-12 13:49:30.994158: I lingvo/core/ops/input_common.cc:34] Input source weights are empty, fall back to legacy behavior.
2020-05-12 13:49:30.994610: I lingvo/core/ops/record_yielder.cc:376] 0x7ef8a34d9f20 Record yielder start
2020-05-12 13:49:30.994631: I lingvo/core/ops/record_yielder.cc:378] Randomly seed RecordYielder.
2020-05-12 13:49:30.994655: I ./lingvo/core/ops/input_common.h:80] Create batcher
2020-05-12 13:49:30.994678: I lingvo/core/ops/record_yielder.cc:485] Epoch 1 gs://waymo_open_dataset_v_1_0_0_tf_example_lingvo/v.1.0.0/train.tfr-*-of-01000
I0512 13:49:31.538286 139606896928512 checkpointer.py:118] Save checkpoint done: gs://waymo-challange/logs/starnet.waymo.3d_object.v3/train/ckpt-00000000
2020-05-12 13:49:33.201902: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2020-05-12 13:50:02.702309: W tensorflow/core/common_runtime/bfc_allocator.cc:424] Allocator (GPU_0_bfc) ran out of memory trying to allocate 512.00MiB (rounded to 536870912). Current allocation summary follows.
2020-05-12 13:50:02.702388: I tensorflow/core/common_runtime/bfc_allocator.cc:894] BFCAllocator dump for GPU_0_bfc
2020-05-12 13:50:02.702401: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (256): Total Chunks: 176, Chunks in use: 176. 44.0KiB allocated for chunks. 44.0KiB in use in bin. 651B client-requested in use in bin.
2020-05-12 13:50:02.702414: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (512): Total Chunks: 32, Chunks in use: 31. 16.2KiB allocated for chunks. 15.8KiB in use in bin. 15.5KiB client-requested in use in bin.
2020-05-12 13:50:02.702425: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (1024): Total Chunks: 32, Chunks in use: 29. 37.5KiB allocated for chunks. 33.2KiB in use in bin. 30.0KiB client-requested in use in bin.
2020-05-12 13:50:02.702435: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (2048): Total Chunks: 1, Chunks in use: 0. 2.2KiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-05-12 13:50:02.702445: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (4096): Total Chunks: 3, Chunks in use: 1. 14.8KiB allocated for chunks. 4.0KiB in use in bin. 4.0KiB client-requested in use in bin.
2020-05-12 13:50:02.702455: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (8192): Total Chunks: 2, Chunks in use: 0. 26.5KiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-05-12 13:50:02.702465: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (16384): Total Chunks: 1, Chunks in use: 0. 17.0KiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-05-12 13:50:02.702474: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (32768): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-05-12 13:50:02.702485: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (65536): Total Chunks: 12, Chunks in use: 9. 771.2KiB allocated for chunks. 576.2KiB in use in bin. 576.0KiB client-requested in use in bin.
2020-05-12 13:50:02.702496: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (131072): Total Chunks: 14, Chunks in use: 10. 1.89MiB allocated for chunks. 1.25MiB in use in bin. 1.25MiB client-requested in use in bin.
2020-05-12 13:50:02.702506: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (262144): Total Chunks: 16, Chunks in use: 14. 4.43MiB allocated for chunks. 3.78MiB in use in bin. 3.78MiB client-requested in use in bin.
2020-05-12 13:50:02.702517: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (524288): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-05-12 13:50:02.702530: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (1048576): Total Chunks: 6, Chunks in use: 5. 9.38MiB allocated for chunks. 8.20MiB in use in bin. 8.20MiB client-requested in use in bin.
2020-05-12 13:50:02.702541: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (2097152): Total Chunks: 14, Chunks in use: 12. 29.92MiB allocated for chunks. 24.59MiB in use in bin. 24.34MiB client-requested in use in bin.
2020-05-12 13:50:02.702551: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (4194304): Total Chunks: 3, Chunks in use: 0. 14.07MiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-05-12 13:50:02.702561: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (8388608): Total Chunks: 2, Chunks in use: 0. 25.71MiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-05-12 13:50:02.702571: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (16777216): Total Chunks: 1, Chunks in use: 0. 28.00MiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-05-12 13:50:02.702581: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (33554432): Total Chunks: 1, Chunks in use: 0. 32.00MiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-05-12 13:50:02.702590: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (67108864): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-05-12 13:50:02.702600: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (134217728): Total Chunks: 20, Chunks in use: 19. 2.50GiB allocated for chunks. 2.38GiB in use in bin. 2.38GiB client-requested in use in bin.
2020-05-12 13:50:02.702611: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (268435456): Total Chunks: 31, Chunks in use: 30. 11.98GiB allocated for chunks. 11.50GiB in use in bin. 11.50GiB client-requested in use in bin.
2020-05-12 13:50:02.702622: I tensorflow/core/common_runtime/bfc_allocator.cc:917] Bin for 512.00MiB was 256.00MiB, Chunk State:
2020-05-12 13:50:02.702638: I tensorflow/core/common_runtime/bfc_allocator.cc:923] Size: 492.95MiB | Requested Size: 256.0KiB | in_use: 0 | bin_num: 20, prev: Size: 512.00MiB | Requested Size: 512.00MiB | in_use: 1 | bin_num: -1
2020-05-12 13:50:02.702648: I tensorflow/core/common_runtime/bfc_allocator.cc:930] Next region of size 15702704128
2020-05-12 13:50:02.702660: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ec000000 of size 1280 next 1
2020-05-12 13:50:02.702668: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ec000500 of size 256 next 2
2020-05-12 13:50:02.702677: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ec000600 of size 256 next 3
2020-05-12 13:50:02.702686: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ec000700 of size 256 next 4
2020-05-12 13:50:02.702694: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ec000800 of size 256 next 5
2020-05-12 13:50:02.702703: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ec000900 of size 256 next 6
2020-05-12 13:50:02.702711: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ec000a00 of size 256 next 7
2020-05-12 13:50:02.702719: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ec000b00 of size 256 next 8
2020-05-12 13:50:02.702727: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ec000c00 of size 256 next 9
2020-05-12 13:50:02.702736: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ec000d00 of size 262144 next 10
2020-05-12 13:50:02.702744: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ec040d00 of size 262144 next 11
2020-05-12 13:50:02.702753: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ec080d00 of size 262144 next 12
2020-05-12 13:50:02.702761: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ec0c0d00 of size 262144 next 13
2020-05-12 13:50:02.702769: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ec100d00 of size 262144 next 14
2020-05-12 13:50:02.702777: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ec140d00 of size 262144 next 15
2020-05-12 13:50:02.702792: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ec180d00 of size 262144 next 16
2020-05-12 13:50:02.702801: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ec1c0d00 of size 262144 next 17
2020-05-12 13:50:02.702809: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ec200d00 of size 262144 next 18
2020-05-12 13:50:02.702818: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ec240d00 of size 262144 next 19
2020-05-12 13:50:02.702827: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ec280d00 of size 262144 next 20
2020-05-12 13:50:02.702835: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ec2c0d00 of size 2048000 next 21
2020-05-12 13:50:02.702843: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ec4b4d00 of size 2048000 next 22
2020-05-12 13:50:02.702852: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ec6a8d00 of size 2457600 next 23
2020-05-12 13:50:02.702861: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ec900d00 of size 1228800 next 24
2020-05-12 13:50:02.702869: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4eca2cd00 of size 1228800 next 25
2020-05-12 13:50:02.702878: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ecb58d00 of size 409600 next 26
2020-05-12 13:50:02.702886: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ecbbcd00 of size 409600 next 27
2020-05-12 13:50:02.702894: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ecc20d00 of size 256 next 28
2020-05-12 13:50:02.702903: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ecc20e00 of size 256 next 29
2020-05-12 13:50:02.702912: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ecc20f00 of size 256 next 30
2020-05-12 13:50:02.702920: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ecc21000 of size 256 next 31
2020-05-12 13:50:02.702929: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ecc21100 of size 256 next 32
2020-05-12 13:50:02.702938: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ecc21200 of size 256 next 33
2020-05-12 13:50:02.702953: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ecc21300 of size 256 next 34
2020-05-12 13:50:02.702967: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ecc21400 of size 256 next 35
2020-05-12 13:50:02.702980: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ecc21500 of size 256 next 36
2020-05-12 13:50:02.702991: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ecc21600 of size 256 next 37
2020-05-12 13:50:02.703003: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ecc21700 of size 256 next 38
2020-05-12 13:50:02.703013: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ecc21800 of size 256 next 39
2020-05-12 13:50:02.703023: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ecc21900 of size 2048000 next 40
2020-05-12 13:50:02.703035: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ece15900 of size 256 next 41
2020-05-12 13:50:02.703047: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ece15a00 of size 256 next 42
2020-05-12 13:50:02.703059: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ece15b00 of size 256 next 43
2020-05-12 13:50:02.703070: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ece15c00 of size 256 next 44
2020-05-12 13:50:02.703081: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ece15d00 of size 262144 next 45
2020-05-12 13:50:02.703093: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ece55d00 of size 256 next 46
2020-05-12 13:50:02.703106: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ece55e00 of size 256 next 47
2020-05-12 13:50:02.703118: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ece55f00 of size 512 next 48
2020-05-12 13:50:02.703128: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ece56100 of size 2097152 next 49
2020-05-12 13:50:02.703140: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed056100 of size 1024 next 50
2020-05-12 13:50:02.703151: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed056500 of size 256 next 51
2020-05-12 13:50:02.703160: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed056600 of size 1536 next 52
2020-05-12 13:50:02.703172: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed056c00 of size 256 next 53
2020-05-12 13:50:02.703184: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed056d00 of size 256 next 54
2020-05-12 13:50:02.703194: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed056e00 of size 4096 next 55
2020-05-12 13:50:02.703202: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed057e00 of size 256 next 56
2020-05-12 13:50:02.703211: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed057f00 of size 256 next 57
2020-05-12 13:50:02.703219: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed058000 of size 256 next 58
2020-05-12 13:50:02.703231: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed058100 of size 256 next 59
2020-05-12 13:50:02.703243: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed058200 of size 256 next 60
2020-05-12 13:50:02.703255: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed058300 of size 256 next 61
2020-05-12 13:50:02.703266: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed058400 of size 256 next 62
2020-05-12 13:50:02.703275: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed058500 of size 256 next 63
2020-05-12 13:50:02.703285: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed058600 of size 256 next 64
2020-05-12 13:50:02.703293: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed058700 of size 256 next 65
2020-05-12 13:50:02.703299: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed058800 of size 256 next 66
2020-05-12 13:50:02.703311: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed058900 of size 256 next 67
2020-05-12 13:50:02.703322: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed058a00 of size 256 next 68
2020-05-12 13:50:02.703331: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed058b00 of size 256 next 69
2020-05-12 13:50:02.703342: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed058c00 of size 256 next 70
2020-05-12 13:50:02.703351: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed058d00 of size 256 next 71
2020-05-12 13:50:02.703360: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed058e00 of size 256 next 72
2020-05-12 13:50:02.703368: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed058f00 of size 256 next 73
2020-05-12 13:50:02.703377: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed059000 of size 256 next 74
2020-05-12 13:50:02.703386: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed059100 of size 256 next 75
2020-05-12 13:50:02.703396: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed059200 of size 256 next 76
2020-05-12 13:50:02.703406: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed059300 of size 256 next 77
2020-05-12 13:50:02.703416: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed059400 of size 256 next 78
2020-05-12 13:50:02.703426: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed059500 of size 256 next 79
2020-05-12 13:50:02.703435: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed059600 of size 256 next 80
2020-05-12 13:50:02.703445: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed059700 of size 256 next 81
2020-05-12 13:50:02.703456: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed059800 of size 256 next 82
2020-05-12 13:50:02.703466: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed059900 of size 256 next 83
2020-05-12 13:50:02.703474: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed059a00 of size 256 next 84
2020-05-12 13:50:02.703483: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed059b00 of size 256 next 85
2020-05-12 13:50:02.703493: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed059c00 of size 256 next 86
2020-05-12 13:50:02.703502: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed059d00 of size 256 next 87
2020-05-12 13:50:02.703512: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed059e00 of size 256 next 88
2020-05-12 13:50:02.703522: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed059f00 of size 256 next 89
2020-05-12 13:50:02.703533: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05a000 of size 256 next 90
2020-05-12 13:50:02.703543: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05a100 of size 256 next 91
2020-05-12 13:50:02.703552: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05a200 of size 256 next 92
2020-05-12 13:50:02.703562: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05a300 of size 256 next 93
2020-05-12 13:50:02.703572: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05a400 of size 256 next 94
2020-05-12 13:50:02.703582: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05a500 of size 256 next 95
2020-05-12 13:50:02.703591: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05a600 of size 256 next 96
2020-05-12 13:50:02.703599: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05a700 of size 256 next 97
2020-05-12 13:50:02.703609: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05a800 of size 256 next 98
2020-05-12 13:50:02.703618: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05a900 of size 256 next 99
2020-05-12 13:50:02.703629: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05aa00 of size 256 next 100
2020-05-12 13:50:02.703641: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05ab00 of size 256 next 101
2020-05-12 13:50:02.703652: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05ac00 of size 256 next 102
2020-05-12 13:50:02.703662: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05ad00 of size 256 next 103
2020-05-12 13:50:02.703673: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05ae00 of size 256 next 104
2020-05-12 13:50:02.703681: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05af00 of size 256 next 105
2020-05-12 13:50:02.703690: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05b000 of size 256 next 106
2020-05-12 13:50:02.703698: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05b100 of size 256 next 107
2020-05-12 13:50:02.703707: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05b200 of size 256 next 108
2020-05-12 13:50:02.703714: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05b300 of size 256 next 109
2020-05-12 13:50:02.703722: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05b400 of size 256 next 110
2020-05-12 13:50:02.703730: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05b500 of size 256 next 111
2020-05-12 13:50:02.703742: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05b600 of size 256 next 112
2020-05-12 13:50:02.703750: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05b700 of size 256 next 113
2020-05-12 13:50:02.703758: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05b800 of size 256 next 114
2020-05-12 13:50:02.703767: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05b900 of size 256 next 115
2020-05-12 13:50:02.703775: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05ba00 of size 256 next 116
2020-05-12 13:50:02.703783: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05bb00 of size 256 next 117
2020-05-12 13:50:02.703792: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05bc00 of size 256 next 118
2020-05-12 13:50:02.703800: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05bd00 of size 256 next 119
2020-05-12 13:50:02.703810: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05be00 of size 256 next 120
2020-05-12 13:50:02.703817: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05bf00 of size 256 next 121
2020-05-12 13:50:02.703825: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05c000 of size 256 next 122
2020-05-12 13:50:02.703832: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05c100 of size 256 next 123
2020-05-12 13:50:02.703840: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05c200 of size 256 next 124
2020-05-12 13:50:02.703848: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05c300 of size 256 next 125
2020-05-12 13:50:02.703856: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05c400 of size 256 next 126
2020-05-12 13:50:02.703862: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05c500 of size 256 next 127
2020-05-12 13:50:02.703871: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05c600 of size 256 next 128
2020-05-12 13:50:02.703879: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05c700 of size 256 next 129
2020-05-12 13:50:02.703887: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05c800 of size 256 next 130
2020-05-12 13:50:02.703894: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05c900 of size 256 next 131
2020-05-12 13:50:02.703902: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05ca00 of size 256 next 132
2020-05-12 13:50:02.703910: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05cb00 of size 256 next 133
2020-05-12 13:50:02.703918: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05cc00 of size 256 next 134
2020-05-12 13:50:02.703926: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05cd00 of size 256 next 135
2020-05-12 13:50:02.703935: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05ce00 of size 256 next 136
2020-05-12 13:50:02.703942: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05cf00 of size 256 next 137
2020-05-12 13:50:02.703950: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05d000 of size 256 next 138
2020-05-12 13:50:02.703959: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05d100 of size 256 next 139
2020-05-12 13:50:02.703965: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05d200 of size 256 next 140
2020-05-12 13:50:02.703972: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05d300 of size 256 next 141
2020-05-12 13:50:02.703978: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05d400 of size 256 next 142
2020-05-12 13:50:02.703986: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05d500 of size 256 next 143
2020-05-12 13:50:02.703995: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05d600 of size 256 next 144
2020-05-12 13:50:02.704001: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05d700 of size 256 next 145
2020-05-12 13:50:02.704010: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05d800 of size 256 next 146
2020-05-12 13:50:02.704020: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05d900 of size 256 next 147
2020-05-12 13:50:02.704032: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05da00 of size 256 next 148
2020-05-12 13:50:02.704045: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05db00 of size 256 next 149
2020-05-12 13:50:02.704053: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05dc00 of size 256 next 150
2020-05-12 13:50:02.704063: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05dd00 of size 256 next 151
2020-05-12 13:50:02.704074: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05de00 of size 256 next 152
2020-05-12 13:50:02.704085: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05df00 of size 256 next 153
2020-05-12 13:50:02.704102: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05e000 of size 256 next 154
2020-05-12 13:50:02.704115: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05e100 of size 256 next 155
2020-05-12 13:50:02.704125: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05e200 of size 256 next 156
2020-05-12 13:50:02.704135: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05e300 of size 256 next 157
2020-05-12 13:50:02.704147: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05e400 of size 256 next 158
2020-05-12 13:50:02.704157: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05e500 of size 256 next 159
2020-05-12 13:50:02.704167: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05e600 of size 256 next 160
2020-05-12 13:50:02.704178: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05e700 of size 256 next 161
2020-05-12 13:50:02.704190: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05e800 of size 256 next 162
2020-05-12 13:50:02.704199: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05e900 of size 256 next 163
2020-05-12 13:50:02.704208: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05ea00 of size 256 next 164
2020-05-12 13:50:02.704216: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05eb00 of size 256 next 165
2020-05-12 13:50:02.704227: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05ec00 of size 256 next 166
2020-05-12 13:50:02.704235: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05ed00 of size 256 next 167
2020-05-12 13:50:02.704243: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05ee00 of size 256 next 168
2020-05-12 13:50:02.704251: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05ef00 of size 256 next 169
2020-05-12 13:50:02.704260: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05f000 of size 256 next 170
2020-05-12 13:50:02.704271: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05f100 of size 256 next 171
2020-05-12 13:50:02.704282: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05f200 of size 256 next 172
2020-05-12 13:50:02.704293: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05f300 of size 256 next 173
2020-05-12 13:50:02.704304: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05f400 of size 256 next 174
2020-05-12 13:50:02.704315: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05f500 of size 256 next 175
2020-05-12 13:50:02.704326: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05f600 of size 256 next 176
2020-05-12 13:50:02.704337: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05f700 of size 256 next 177
2020-05-12 13:50:02.704345: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05f800 of size 256 next 178
2020-05-12 13:50:02.704354: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05f900 of size 256 next 179
2020-05-12 13:50:02.704362: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05fa00 of size 256 next 180
2020-05-12 13:50:02.704370: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05fb00 of size 256 next 181
2020-05-12 13:50:02.704379: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4ed05fc00 of size 134217728 next 182
2020-05-12 13:50:02.704387: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4f505fc00 of size 134217728 next 183
2020-05-12 13:50:02.704400: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef4fd05fc00 of size 134217728 next 184
2020-05-12 13:50:02.704410: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef50505fc00 of size 134217728 next 185
2020-05-12 13:50:02.704421: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef50d05fc00 of size 134217728 next 186
2020-05-12 13:50:02.704432: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef51505fc00 of size 256 next 187
2020-05-12 13:50:02.704444: I tensorflow/core/common_runtime/bfc_allocator.cc:950] Free at 7ef51505fd00 of size 131072 next 188
2020-05-12 13:50:02.704460: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef51507fd00 of size 1024 next 192
2020-05-12 13:50:02.704472: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef515080100 of size 256 next 193
2020-05-12 13:50:02.704482: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef515080200 of size 512 next 194
2020-05-12 13:50:02.704493: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef515080400 of size 65792 next 196
2020-05-12 13:50:02.704504: I tensorflow/core/common_runtime/bfc_allocator.cc:950] Free at 7ef515090500 of size 12800 next 199
2020-05-12 13:50:02.704515: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef515093700 of size 1024 next 200
2020-05-12 13:50:02.704527: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef515093b00 of size 1024 next 201
2020-05-12 13:50:02.704537: I tensorflow/core/common_runtime/bfc_allocator.cc:950] Free at 7ef515093f00 of size 211968 next 207
2020-05-12 13:50:02.704546: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef5150c7b00 of size 131072 next 208
2020-05-12 13:50:02.704554: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef5150e7b00 of size 131072 next 209
2020-05-12 13:50:02.704562: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef515107b00 of size 512 next 210
2020-05-12 13:50:02.704570: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef515107d00 of size 512 next 212
2020-05-12 13:50:02.704578: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef515107f00 of size 1280 next 213
2020-05-12 13:50:02.704588: I tensorflow/core/common_runtime/bfc_allocator.cc:950] Free at 7ef515108400 of size 4916224 next 215
2020-05-12 13:50:02.704596: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef5155b8800 of size 512 next 216
2020-05-12 13:50:02.704604: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef5155b8a00 of size 1024 next 217
2020-05-12 13:50:02.704612: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef5155b8e00 of size 512 next 218
2020-05-12 13:50:02.704620: I tensorflow/core/common_runtime/bfc_allocator.cc:950] Free at 7ef5155b9000 of size 6144 next 223
2020-05-12 13:50:02.704630: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef5155ba800 of size 1024 next 224
2020-05-12 13:50:02.704643: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef5155bac00 of size 1024 next 225
2020-05-12 13:50:02.704656: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef5155bb000 of size 512 next 226
2020-05-12 13:50:02.704667: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef5155bb200 of size 1024 next 304
2020-05-12 13:50:02.704676: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef5155bb600 of size 1024 next 319
2020-05-12 13:50:02.704686: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef5155bba00 of size 1024 next 283
2020-05-12 13:50:02.704694: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef5155bbe00 of size 1024 next 249
2020-05-12 13:50:02.704703: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef5155bc200 of size 1024 next 268
2020-05-12 13:50:02.704713: I tensorflow/core/common_runtime/bfc_allocator.cc:950] Free at 7ef5155bc600 of size 1536 next 236
2020-05-12 13:50:02.704722: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef5155bcc00 of size 512 next 237
2020-05-12 13:50:02.704731: I tensorflow/core/common_runtime/bfc_allocator.cc:950] Free at 7ef5155bce00 of size 10380800 next 259
2020-05-12 13:50:02.704740: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef515fa3400 of size 512 next 260
2020-05-12 13:50:02.704748: I tensorflow/core/common_runtime/bfc_allocator.cc:950] Free at 7ef515fa3600 of size 398592 next 271
2020-05-12 13:50:02.704764: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef516004b00 of size 1024 next 272
2020-05-12 13:50:02.704773: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef516004f00 of size 1280 next 274
2020-05-12 13:50:02.704782: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef516005400 of size 512 next 372
2020-05-12 13:50:02.704792: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef516005600 of size 512 next 275
2020-05-12 13:50:02.704803: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef516005800 of size 512 next 277
2020-05-12 13:50:02.704812: I tensorflow/core/common_runtime/bfc_allocator.cc:950] Free at 7ef516005a00 of size 197632 next 280
2020-05-12 13:50:02.704824: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef516035e00 of size 512 next 281
2020-05-12 13:50:02.704833: I tensorflow/core/common_runtime/bfc_allocator.cc:950] Free at 7ef516036000 of size 280320 next 295
2020-05-12 13:50:02.704844: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef51607a700 of size 65536 next 296
2020-05-12 13:50:02.704854: I tensorflow/core/common_runtime/bfc_allocator.cc:950] Free at 7ef51608a700 of size 135424 next 306
2020-05-12 13:50:02.704865: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef5160ab800 of size 512 next 284
2020-05-12 13:50:02.704874: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef5160aba00 of size 512 next 307
2020-05-12 13:50:02.704884: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef5160abc00 of size 65536 next 308
2020-05-12 13:50:02.704893: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef5160bbc00 of size 1024 next 324
2020-05-12 13:50:02.704904: I tensorflow/core/common_runtime/bfc_allocator.cc:950] Free at 7ef5160bc000 of size 1024 next 343
2020-05-12 13:50:02.704914: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef5160bc400 of size 256 next 312
2020-05-12 13:50:02.704927: I tensorflow/core/common_runtime/bfc_allocator.cc:950] Free at 7ef5160bc500 of size 1792 next 297
2020-05-12 13:50:02.704936: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef5160bcc00 of size 256 next 222
2020-05-12 13:50:02.704945: I tensorflow/core/common_runtime/bfc_allocator.cc:950] Free at 7ef5160bcd00 of size 2304 next 278
2020-05-12 13:50:02.704954: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef5160bd600 of size 256 next 240
2020-05-12 13:50:02.704961: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef5160bd700 of size 256 next 286
2020-05-12 13:50:02.704970: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef5160bd800 of size 256 next 287
2020-05-12 13:50:02.704978: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef5160bd900 of size 256 next 393
2020-05-12 13:50:02.704986: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef5160bda00 of size 256 next 339
2020-05-12 13:50:02.704995: I tensorflow/core/common_runtime/bfc_allocator.cc:950] Free at 7ef5160bdb00 of size 4864 next 314
2020-05-12 13:50:02.705003: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef5160bee00 of size 131072 next 315
2020-05-12 13:50:02.705012: I tensorflow/core/common_runtime/bfc_allocator.cc:950] Free at 7ef5160dee00 of size 14336 next 316
2020-05-12 13:50:02.705020: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef5160e2600 of size 256 next 317
2020-05-12 13:50:02.705029: I tensorflow/core/common_runtime/bfc_allocator.cc:950] Free at 7ef5160e2700 of size 17408 next 328
2020-05-12 13:50:02.705038: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef5160e6b00 of size 65536 next 329
2020-05-12 13:50:02.705047: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef5160f6b00 of size 65536 next 330
2020-05-12 13:50:02.705063: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef516106b00 of size 131072 next 331
2020-05-12 13:50:02.705077: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef516126b00 of size 512 next 333
2020-05-12 13:50:02.705088: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef516126d00 of size 256 next 232
2020-05-12 13:50:02.705098: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef516126e00 of size 256 next 334
2020-05-12 13:50:02.705108: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef516126f00 of size 256 next 205
2020-05-12 13:50:02.705119: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef516127000 of size 256 next 189
2020-05-12 13:50:02.705128: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef516127100 of size 256 next 279
2020-05-12 13:50:02.705137: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef516127200 of size 256 next 252
2020-05-12 13:50:02.705150: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef516127300 of size 256 next 335
2020-05-12 13:50:02.705161: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef516127400 of size 131072 next 336
2020-05-12 13:50:02.705171: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef516147400 of size 256 next 337
2020-05-12 13:50:02.705182: I tensorflow/core/common_runtime/bfc_allocator.cc:950] Free at 7ef516147500 of size 4926976 next 345
2020-05-12 13:50:02.705191: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef5165fa300 of size 65536 next 346
2020-05-12 13:50:02.705199: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef51660a300 of size 512 next 347
2020-05-12 13:50:02.705210: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef51660a500 of size 1280 next 349
2020-05-12 13:50:02.705220: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef51660aa00 of size 131072 next 350
2020-05-12 13:50:02.705233: I tensorflow/core/common_runtime/bfc_allocator.cc:950] Free at 7ef51662aa00 of size 65536 next 351
2020-05-12 13:50:02.705242: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef51663aa00 of size 512 next 353
2020-05-12 13:50:02.705255: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef51663ac00 of size 512 next 354
2020-05-12 13:50:02.705266: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef51663ae00 of size 512 next 377
2020-05-12 13:50:02.705277: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef51663b000 of size 512 next 355
2020-05-12 13:50:02.705287: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef51663b200 of size 1792 next 362
2020-05-12 13:50:02.705296: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef51663b900 of size 1024 next 363
2020-05-12 13:50:02.705306: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef51663bd00 of size 512 next 364
2020-05-12 13:50:02.705317: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef51663bf00 of size 1024 next 365
2020-05-12 13:50:02.705326: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef51663c300 of size 65536 next 366
2020-05-12 13:50:02.705337: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef51664c300 of size 512 next 367
2020-05-12 13:50:02.705347: I tensorflow/core/common_runtime/bfc_allocator.cc:950] Free at 7ef51664c500 of size 68096 next 378
2020-05-12 13:50:02.705358: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef51665cf00 of size 65536 next 379
2020-05-12 13:50:02.705367: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef51666cf00 of size 131072 next 380
2020-05-12 13:50:02.705375: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef51668cf00 of size 1024 next 381
2020-05-12 13:50:02.705383: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef51668d300 of size 512 next 382
2020-05-12 13:50:02.705392: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef51668d500 of size 1024 next 383
2020-05-12 13:50:02.705401: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef51668d900 of size 512 next 290
2020-05-12 13:50:02.705416: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef51668db00 of size 768 next 387
2020-05-12 13:50:02.705428: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef51668de00 of size 512 next 388
2020-05-12 13:50:02.705437: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef51668e000 of size 1792 next 395
2020-05-12 13:50:02.705449: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef51668e700 of size 512 next 396
2020-05-12 13:50:02.705458: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef51668e900 of size 1024 next 397
2020-05-12 13:50:02.705466: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef51668ed00 of size 65536 next 398
2020-05-12 13:50:02.705474: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef51669ed00 of size 131072 next 399
2020-05-12 13:50:02.705486: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef5166bed00 of size 256 next 400
2020-05-12 13:50:02.705500: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef5166bee00 of size 512 next 401
2020-05-12 13:50:02.705510: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef5166bf000 of size 256 next 298
2020-05-12 13:50:02.705519: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef5166bf100 of size 256 next 198
2020-05-12 13:50:02.705528: I tensorflow/core/common_runtime/bfc_allocator.cc:950] Free at 7ef5166bf200 of size 512 next 402
2020-05-12 13:50:02.705538: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef5166bf400 of size 1792 next 409
2020-05-12 13:50:02.705546: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef5166bfb00 of size 131072 next 410
2020-05-12 13:50:02.705555: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef5166dfb00 of size 1024 next 411
2020-05-12 13:50:02.705563: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef5166dff00 of size 512 next 412
2020-05-12 13:50:02.705572: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef5166e0100 of size 1536 next 413
2020-05-12 13:50:02.705581: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef5166e0700 of size 512 next 414
2020-05-12 13:50:02.705597: I tensorflow/core/common_runtime/bfc_allocator.cc:950] Free at 7ef5166e0900 of size 66048 next 416
2020-05-12 13:50:02.705606: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef5166f0b00 of size 131072 next 417
2020-05-12 13:50:02.705614: I tensorflow/core/common_runtime/bfc_allocator.cc:950] Free at 7ef516710b00 of size 16580608 next 253
2020-05-12 13:50:02.705623: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef5176e0b00 of size 2097152 next 256
2020-05-12 13:50:02.705634: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef5178e0b00 of size 2097152 next 250
2020-05-12 13:50:02.705644: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef517ae0b00 of size 2097152 next 348
2020-05-12 13:50:02.705654: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef517ce0b00 of size 2097152 next 230
2020-05-12 13:50:02.705664: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef517ee0b00 of size 2097152 next 390
2020-05-12 13:50:02.705674: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef5180e0b00 of size 2097152 next 326
2020-05-12 13:50:02.705682: I tensorflow/core/common_runtime/bfc_allocator.cc:950] Free at 7ef5182e0b00 of size 3129344 next 276
2020-05-12 13:50:02.705692: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef5185dcb00 of size 2359296 next 288
2020-05-12 13:50:02.705703: I tensorflow/core/common_runtime/bfc_allocator.cc:950] Free at 7ef51881cb00 of size 1228800 next 415
2020-05-12 13:50:02.705714: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef518948b00 of size 2097152 next 263
2020-05-12 13:50:02.705724: I tensorflow/core/common_runtime/bfc_allocator.cc:950] Free at 7ef518b48b00 of size 2457600 next 384
2020-05-12 13:50:02.705735: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef518da0b00 of size 134217728 next 231
2020-05-12 13:50:02.705745: I tensorflow/core/common_runtime/bfc_allocator.cc:950] Free at 7ef520da0b00 of size 4915200 next 221
2020-05-12 13:50:02.705756: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef521250b00 of size 2097152 next 332
2020-05-12 13:50:02.705767: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef521450b00 of size 2097152 next 195
2020-05-12 13:50:02.705779: I tensorflow/core/common_runtime/bfc_allocator.cc:950] Free at 7ef521650b00 of size 29360128 next 227
2020-05-12 13:50:02.705790: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef523250b00 of size 134217728 next 293
2020-05-12 13:50:02.705800: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef52b250b00 of size 134217728 next 294
2020-05-12 13:50:02.705811: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef533250b00 of size 268435456 next 257
2020-05-12 13:50:02.705822: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef543250b00 of size 268435456 next 289
2020-05-12 13:50:02.705832: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef553250b00 of size 268435456 next 299
2020-05-12 13:50:02.705844: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef563250b00 of size 536870912 next 318
2020-05-12 13:50:02.705853: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef583250b00 of size 536870912 next 251
2020-05-12 13:50:02.705864: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef5a3250b00 of size 536870912 next 261
2020-05-12 13:50:02.705873: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef5c3250b00 of size 268435456 next 291
2020-05-12 13:50:02.705886: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef5d3250b00 of size 268435456 next 247
2020-05-12 13:50:02.705895: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef5e3250b00 of size 536870912 next 220
2020-05-12 13:50:02.705906: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef603250b00 of size 536870912 next 235
2020-05-12 13:50:02.705915: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef623250b00 of size 536870912 next 369
2020-05-12 13:50:02.705923: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef643250b00 of size 134217728 next 320
2020-05-12 13:50:02.705932: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef64b250b00 of size 134217728 next 321
2020-05-12 13:50:02.705940: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef653250b00 of size 134217728 next 309
2020-05-12 13:50:02.705952: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef65b250b00 of size 268435456 next 403
2020-05-12 13:50:02.705961: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef66b250b00 of size 268435456 next 404
2020-05-12 13:50:02.705970: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef67b250b00 of size 268435456 next 389
2020-05-12 13:50:02.705978: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef68b250b00 of size 536870912 next 373
2020-05-12 13:50:02.705986: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef6ab250b00 of size 536870912 next 356
2020-05-12 13:50:02.705996: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef6cb250b00 of size 536870912 next 357
2020-05-12 13:50:02.706005: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef6eb250b00 of size 134217728 next 371
2020-05-12 13:50:02.706015: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef6f3250b00 of size 134217728 next 303
2020-05-12 13:50:02.706024: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef6fb250b00 of size 268435456 next 386
2020-05-12 13:50:02.706033: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef70b250b00 of size 268435456 next 370
2020-05-12 13:50:02.706045: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef71b250b00 of size 268435456 next 352
2020-05-12 13:50:02.706054: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef72b250b00 of size 536870912 next 385
2020-05-12 13:50:02.706060: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef74b250b00 of size 536870912 next 302
2020-05-12 13:50:02.706071: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef76b250b00 of size 536870912 next 267
2020-05-12 13:50:02.706081: I tensorflow/core/common_runtime/bfc_allocator.cc:950] Free at 7ef78b250b00 of size 33554432 next 203
2020-05-12 13:50:02.706090: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef78d250b00 of size 134217728 next 340
2020-05-12 13:50:02.706099: I tensorflow/core/common_runtime/bfc_allocator.cc:950] Free at 7ef795250b00 of size 134217728 next 344
2020-05-12 13:50:02.706109: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef79d250b00 of size 268435456 next 202
2020-05-12 13:50:02.706117: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef7ad250b00 of size 268435456 next 191
2020-05-12 13:50:02.706127: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef7bd250b00 of size 268435456 next 408
2020-05-12 13:50:02.706136: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef7cd250b00 of size 536870912 next 305
2020-05-12 13:50:02.706148: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef7ed250b00 of size 536870912 next 394
2020-05-12 13:50:02.706158: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef80d250b00 of size 536870912 next 361
2020-05-12 13:50:02.706169: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef82d250b00 of size 134217728 next 197
2020-05-12 13:50:02.706226: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef835250b00 of size 134217728 next 376
2020-05-12 13:50:02.706239: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef83d250b00 of size 134217728 next 338
2020-05-12 13:50:02.706251: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef845250b00 of size 134217728 next 325
2020-05-12 13:50:02.706264: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef84d250b00 of size 134217728 next 368
2020-05-12 13:50:02.706281: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7ef855250b00 of size 536870912 next 238
2020-05-12 13:50:02.706296: I tensorflow/core/common_runtime/bfc_allocator.cc:950] Free at 7ef875250b00 of size 516896000 next 18446744073709551615
2020-05-12 13:50:02.706307: I tensorflow/core/common_runtime/bfc_allocator.cc:955] Summary of in-use Chunks by size:
2020-05-12 13:50:02.706322: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 176 Chunks of size 256 totalling 44.0KiB
2020-05-12 13:50:02.706335: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 30 Chunks of size 512 totalling 15.0KiB
2020-05-12 13:50:02.706349: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 1 Chunks of size 768 totalling 768B
2020-05-12 13:50:02.706363: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 20 Chunks of size 1024 totalling 20.0KiB
2020-05-12 13:50:02.706373: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 4 Chunks of size 1280 totalling 5.0KiB
2020-05-12 13:50:02.706382: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 2 Chunks of size 1536 totalling 3.0KiB
2020-05-12 13:50:02.706390: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 3 Chunks of size 1792 totalling 5.2KiB
2020-05-12 13:50:02.706399: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 1 Chunks of size 4096 totalling 4.0KiB
2020-05-12 13:50:02.706408: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 8 Chunks of size 65536 totalling 512.0KiB
2020-05-12 13:50:02.706416: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 1 Chunks of size 65792 totalling 64.2KiB
2020-05-12 13:50:02.706427: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 10 Chunks of size 131072 totalling 1.25MiB
2020-05-12 13:50:02.706435: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 12 Chunks of size 262144 totalling 3.00MiB
2020-05-12 13:50:02.706444: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 2 Chunks of size 409600 totalling 800.0KiB
2020-05-12 13:50:02.706452: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 2 Chunks of size 1228800 totalling 2.34MiB
2020-05-12 13:50:02.706461: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 3 Chunks of size 2048000 totalling 5.86MiB
2020-05-12 13:50:02.706471: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 10 Chunks of size 2097152 totalling 20.00MiB
2020-05-12 13:50:02.706479: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 1 Chunks of size 2359296 totalling 2.25MiB
2020-05-12 13:50:02.706488: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 1 Chunks of size 2457600 totalling 2.34MiB
2020-05-12 13:50:02.706502: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 19 Chunks of size 134217728 totalling 2.38GiB
2020-05-12 13:50:02.706509: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 14 Chunks of size 268435456 totalling 3.50GiB
2020-05-12 13:50:02.706522: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 16 Chunks of size 536870912 totalling 8.00GiB
2020-05-12 13:50:02.706533: I tensorflow/core/common_runtime/bfc_allocator.cc:962] Sum Total of in-use chunks: 13.91GiB
2020-05-12 13:50:02.706542: I tensorflow/core/common_runtime/bfc_allocator.cc:964] total_region_allocated_bytes_: 15702704128 memory_limit_: 15702704128 available bytes: 0 curr_region_allocation_bytes_: 31405408256
2020-05-12 13:50:02.706560: I tensorflow/core/common_runtime/bfc_allocator.cc:970] Stats:
Limit: 15702704128
InUse: 14938522880
MaxInUse: 15074972928
NumAllocs: 946
MaxAllocSize: 536870912
2020-05-12 13:50:02.706595: W tensorflow/core/common_runtime/bfc_allocator.cc:429] *************************************************************************************************___
2020-05-12 13:50:02.706629: W tensorflow/core/framework/op_kernel.cc:1655] OP_REQUIRES failed at cwise_ops_common.h:263 : Resource exhausted: OOM when allocating tensor with shape[1,1024,512,256] and type float on /job:local/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
2020-05-12 13:50:10.736175: I lingvo/core/ops/record_yielder.cc:407] 0x7ef8a34d9f20Basic record yielder exit
E0512 13:50:11.172914 139606888535808 base_runner.py:244] trainer done (fatal error): <class 'tensorflow.python.framework.errors_impl.ResourceExhaustedError'>
I0512 13:50:11.175342 139606888535808 base_runner.py:111] trainer exception: From /job:local/replica:0/task:0:
OOM when allocating tensor with shape[1,1024,512,256] and type float on /job:local/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
[[node gradients/fprop/starnet/tower_0_0/feat/validated_gin/gin/gin_concat/seq/gin_concat/seq/gin_concat/seq/gin_concat/seq/gin_concat/seq/validated_gin_intermediate/gin_intermediate/validated_mlp/mlp/validated_validated_l001/validated_l001/l001/key_seq/bn/bn/sub_2_grad/Neg (defined at tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/core/py_utils.py:1923) ]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
Original stack trace for 'gradients/fprop/starnet/tower_0_0/feat/validated_gin/gin/gin_concat/seq/gin_concat/seq/gin_concat/seq/gin_concat/seq/gin_concat/seq/validated_gin_intermediate/gin_intermediate/validated_mlp/mlp/validated_validated_l001/validated_l001/l001/key_seq/bn/bn/sub_2_grad/Neg':
File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/trainer.py", line 1874, in <module>
tf.app.run(main)
File "usr/local/lib/python3.6/dist-packages/tensorflow_core/python/platform/app.py", line 40, in run
_run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef)
File "usr/local/lib/python3.6/dist-packages/absl/app.py", line 299, in run
_run_main(main, args)
File "usr/local/lib/python3.6/dist-packages/absl/app.py", line 250, in _run_main
sys.exit(main(argv))
File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/trainer.py", line 1865, in main
RunnerManager(FLAGS.model).Start()
File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/trainer.py", line 1861, in Start
self.StartRunners(self.CreateRunners(FLAGS.job.split(','), FLAGS.logdir))
File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/trainer.py", line 1609, in CreateRunners
trial)
File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/trainer.py", line 1562, in _CreateRunner
return self.Trainer(cfg, *common_args)
File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/trainer.py", line 398, in __init__
self._model.ConstructFPropBPropGraph()
File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/core/base_model.py", line 1197, in ConstructFPropBPropGraph
self._task.BProp()
File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/core/base_model.py", line 538, in BProp
self._BPropForVariables(self.vars)
File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/core/base_model.py", line 568, in _BPropForVariables
gradient_adjuster=self.AdjustGradients)
File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/core/learner.py", line 189, in Apply
skip_zero_gradients=p.skip_zero_gradients)
File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/core/optimizer.py", line 54, in ComputeGradients
return py_utils.ComputeGradients(loss, vmap, *args, **kwargs)
File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/core/py_utils.py", line 2143, in ComputeGradients
colocate_gradients_with_ops, gate_gradients)
File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/core/py_utils.py", line 1923, in ComputeGradientsSimple
gate_gradients=gate_gradients)
File "usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/gradients_impl.py", line 158, in gradients
unconnected_gradients)
File "usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/gradients_util.py", line 669, in _GradientsHelper
lambda: grad_fn(op, *out_grads))
File "usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/gradients_util.py", line 336, in _MaybeCompile
return grad_fn() # Exit early
File "usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/gradients_util.py", line 669, in <lambda>
lambda: grad_fn(op, *out_grads))
File "usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/math_grad.py", line 1201, in _SubGrad
gy = array_ops.reshape(math_ops.reduce_sum(-grad, ry), sy)
File "usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/gen_math_ops.py", line 6304, in neg
"Neg", x=x, name=name)
File "usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/op_def_library.py", line 742, in _apply_op_helper
attrs=attr_protos, op_def=op_def)
File "usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/ops.py", line 3322, in _create_op_internal
op_def=op_def)
File "usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/ops.py", line 1756, in __init__
self._traceback = tf_stack.extract_stack()
...which was originally created as op 'fprop/starnet/tower_0_0/feat/validated_gin/gin/gin_concat/seq/gin_concat/seq/gin_concat/seq/gin_concat/seq/gin_concat/seq/validated_gin_intermediate/gin_intermediate/validated_mlp/mlp/validated_validated_l001/validated_l001/l001/key_seq/bn/bn/sub_2', defined at:
File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/trainer.py", line 1874, in <module>
tf.app.run(main)
[elided 7 identical lines from previous traceback]
File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/trainer.py", line 398, in __init__
self._model.ConstructFPropBPropGraph()
File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/core/base_model.py", line 1196, in ConstructFPropBPropGraph
self._task.FPropDefaultTheta()
File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/core/base_model.py", line 524, in FPropDefaultTheta
return self.FProp(self.theta, input_batch)
File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/core/base_model.py", line 445, in FProp
metrics, per_example = self._FPropSplitInputBatch(theta, input_batch)
File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/core/base_model.py", line 491, in _FPropSplitInputBatch
metrics, per_example = self.FPropTower(theta_local, batch)
File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/core/base_model.py", line 413, in FPropTower
predictions = self.ComputePredictions(theta, input_batch)
File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/tasks/car/starnet.py", line 856, in ComputePredictions
featurized_cell = self._CellFeaturizer(theta, input_batch)
File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/tasks/car/starnet.py", line 831, in _CellFeaturizer
point_input)
File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/core/builder_layers.py", line 422, in FProp
args = ch.FProp(th, *args)
File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/core/builder_layers.py", line 422, in FProp
args = ch.FProp(th, *args)
File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/core/builder_layers.py", line 422, in FProp
args = ch.FProp(th, *args)
File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/core/builder_layers.py", line 938, in FProp
out = ch.FProp(th, *args)
File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/core/builder_layers.py", line 422, in FProp
args = ch.FProp(th, *args)
File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/core/builder_layers.py", line 938, in FProp
out = ch.FProp(th, *args)
File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/core/builder_layers.py", line 422, in FProp
args = ch.FProp(th, *args)
File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/core/builder_layers.py", line 938, in FProp
out = ch.FProp(th, *args)
File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/core/builder_layers.py", line 422, in FProp
args = ch.FProp(th, *args)
E0512 13:50:11.177675 139606888535808 base_runner.py:251] Traceback (most recent call last):
E0512 13:50:11.177758 139606888535808 base_runner.py:251] File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py", line 1367, in _do_call
E0512 13:50:11.177819 139606888535808 base_runner.py:251] return fn(*args)
E0512 13:50:11.177872 139606888535808 base_runner.py:251] File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py", line 1352, in _run_fn
E0512 13:50:11.177922 139606888535808 base_runner.py:251] target_list, run_metadata)
E0512 13:50:11.177970 139606888535808 base_runner.py:251] File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py", line 1445, in _call_tf_sessionrun
E0512 13:50:11.178018 139606888535808 base_runner.py:251] run_metadata)
E0512 13:50:11.178066 139606888535808 base_runner.py:251] tensorflow.python.framework.errors_impl.ResourceExhaustedError: From /job:local/replica:0/task:0:
E0512 13:50:11.178113 139606888535808 base_runner.py:251] OOM when allocating tensor with shape[1,1024,512,256] and type float on /job:local/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
E0512 13:50:11.178161 139606888535808 base_runner.py:251] [[{{node gradients/fprop/starnet/tower_0_0/feat/validated_gin/gin/gin_concat/seq/gin_concat/seq/gin_concat/seq/gin_concat/seq/gin_concat/seq/validated_gin_intermediate/gin_intermediate/validated_mlp/mlp/validated_validated_l001/validated_l001/l001/key_seq/bn/bn/sub_2_grad/Neg}}]]
E0512 13:50:11.178250 139606888535808 base_runner.py:251] Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
E0512 13:50:11.178305 139606888535808 base_runner.py:251]
E0512 13:50:11.178353 139606888535808 base_runner.py:251]
E0512 13:50:11.178400 139606888535808 base_runner.py:251] During handling of the above exception, another exception occurred:
E0512 13:50:11.178450 139606888535808 base_runner.py:251]
E0512 13:50:11.178497 139606888535808 base_runner.py:251] Traceback (most recent call last):
E0512 13:50:11.178543 139606888535808 base_runner.py:251] File "/tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/base_runner.py", line 193, in _RunLoop
E0512 13:50:11.178590 139606888535808 base_runner.py:251] loop_func(*loop_args)
E0512 13:50:11.178636 139606888535808 base_runner.py:251] File "/tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/trainer.py", line 504, in _Loop
E0512 13:50:11.178682 139606888535808 base_runner.py:251] model_task.per_example_tensors,
E0512 13:50:11.178728 139606888535808 base_runner.py:251] File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py", line 960, in run
E0512 13:50:11.178775 139606888535808 base_runner.py:251] run_metadata_ptr)
E0512 13:50:11.178820 139606888535808 base_runner.py:251] File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py", line 1183, in _run
E0512 13:50:11.178866 139606888535808 base_runner.py:251] feed_dict_tensor, options, run_metadata)
E0512 13:50:11.178911 139606888535808 base_runner.py:251] File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py", line 1361, in _do_run
E0512 13:50:11.178961 139606888535808 base_runner.py:251] run_metadata)
E0512 13:50:11.179010 139606888535808 base_runner.py:251] File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py", line 1386, in _do_call
E0512 13:50:11.179056 139606888535808 base_runner.py:251] raise type(e)(node_def, op, message)
E0512 13:50:11.179102 139606888535808 base_runner.py:251] tensorflow.python.framework.errors_impl.ResourceExhaustedError: From /job:local/replica:0/task:0:
E0512 13:50:11.179148 139606888535808 base_runner.py:251] OOM when allocating tensor with shape[1,1024,512,256] and type float on /job:local/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
E0512 13:50:11.179193 139606888535808 base_runner.py:251] [[node gradients/fprop/starnet/tower_0_0/feat/validated_gin/gin/gin_concat/seq/gin_concat/seq/gin_concat/seq/gin_concat/seq/gin_concat/seq/validated_gin_intermediate/gin_intermediate/validated_mlp/mlp/validated_validated_l001/validated_l001/l001/key_seq/bn/bn/sub_2_grad/Neg (defined at tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/core/py_utils.py:1923) ]]
E0512 13:50:11.179267 139606888535808 base_runner.py:251] Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
E0512 13:50:11.179316 139606888535808 base_runner.py:251]
E0512 13:50:11.179363 139606888535808 base_runner.py:251]
E0512 13:50:11.179409 139606888535808 base_runner.py:251] Original stack trace for 'gradients/fprop/starnet/tower_0_0/feat/validated_gin/gin/gin_concat/seq/gin_concat/seq/gin_concat/seq/gin_concat/seq/gin_concat/seq/validated_gin_intermediate/gin_intermediate/validated_mlp/mlp/validated_validated_l001/validated_l001/l001/key_seq/bn/bn/sub_2_grad/Neg':
E0512 13:50:11.179456 139606888535808 base_runner.py:251] File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/trainer.py", line 1874, in <module>
E0512 13:50:11.179502 139606888535808 base_runner.py:251] tf.app.run(main)
E0512 13:50:11.179549 139606888535808 base_runner.py:251] File "usr/local/lib/python3.6/dist-packages/tensorflow_core/python/platform/app.py", line 40, in run
E0512 13:50:11.179595 139606888535808 base_runner.py:251] _run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef)
E0512 13:50:11.179647 139606888535808 base_runner.py:251] File "usr/local/lib/python3.6/dist-packages/absl/app.py", line 299, in run
E0512 13:50:11.179695 139606888535808 base_runner.py:251] _run_main(main, args)
E0512 13:50:11.179741 139606888535808 base_runner.py:251] File "usr/local/lib/python3.6/dist-packages/absl/app.py", line 250, in _run_main
E0512 13:50:11.179788 139606888535808 base_runner.py:251] sys.exit(main(argv))
E0512 13:50:11.179834 139606888535808 base_runner.py:251] File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/trainer.py", line 1865, in main
E0512 13:50:11.179880 139606888535808 base_runner.py:251] RunnerManager(FLAGS.model).Start()
E0512 13:50:11.179926 139606888535808 base_runner.py:251] File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/trainer.py", line 1861, in Start
E0512 13:50:11.179972 139606888535808 base_runner.py:251] self.StartRunners(self.CreateRunners(FLAGS.job.split(','), FLAGS.logdir))
E0512 13:50:11.180018 139606888535808 base_runner.py:251] File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/trainer.py", line 1609, in CreateRunners
E0512 13:50:11.180064 139606888535808 base_runner.py:251] trial)
E0512 13:50:11.180110 139606888535808 base_runner.py:251] File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/trainer.py", line 1562, in _CreateRunner
E0512 13:50:11.180155 139606888535808 base_runner.py:251] return self.Trainer(cfg, *common_args)
E0512 13:50:11.180201 139606888535808 base_runner.py:251] File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/trainer.py", line 398, in __init__
E0512 13:50:11.180247 139606888535808 base_runner.py:251] self._model.ConstructFPropBPropGraph()
E0512 13:50:11.180293 139606888535808 base_runner.py:251] File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/core/base_model.py", line 1197, in ConstructFPropBPropGraph
E0512 13:50:11.180339 139606888535808 base_runner.py:251] self._task.BProp()
E0512 13:50:11.180384 139606888535808 base_runner.py:251] File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/core/base_model.py", line 538, in BProp
E0512 13:50:11.180430 139606888535808 base_runner.py:251] self._BPropForVariables(self.vars)
E0512 13:50:11.180479 139606888535808 base_runner.py:251] File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/core/base_model.py", line 568, in _BPropForVariables
E0512 13:50:11.180526 139606888535808 base_runner.py:251] gradient_adjuster=self.AdjustGradients)
E0512 13:50:11.180572 139606888535808 base_runner.py:251] File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/core/learner.py", line 189, in Apply
E0512 13:50:11.180617 139606888535808 base_runner.py:251] skip_zero_gradients=p.skip_zero_gradients)
E0512 13:50:11.180663 139606888535808 base_runner.py:251] File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/core/optimizer.py", line 54, in ComputeGradients
E0512 13:50:11.180709 139606888535808 base_runner.py:251] return py_utils.ComputeGradients(loss, vmap, *args, **kwargs)
E0512 13:50:11.180755 139606888535808 base_runner.py:251] File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/core/py_utils.py", line 2143, in ComputeGradients
E0512 13:50:11.180800 139606888535808 base_runner.py:251] colocate_gradients_with_ops, gate_gradients)
E0512 13:50:11.180846 139606888535808 base_runner.py:251] File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/core/py_utils.py", line 1923, in ComputeGradientsSimple
E0512 13:50:11.180892 139606888535808 base_runner.py:251] gate_gradients=gate_gradients)
E0512 13:50:11.180938 139606888535808 base_runner.py:251] File "usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/gradients_impl.py", line 158, in gradients
E0512 13:50:11.180983 139606888535808 base_runner.py:251] unconnected_gradients)
E0512 13:50:11.181029 139606888535808 base_runner.py:251] File "usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/gradients_util.py", line 669, in _GradientsHelper
E0512 13:50:11.181074 139606888535808 base_runner.py:251] lambda: grad_fn(op, *out_grads))
E0512 13:50:11.181120 139606888535808 base_runner.py:251] File "usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/gradients_util.py", line 336, in _MaybeCompile
E0512 13:50:11.181165 139606888535808 base_runner.py:251] return grad_fn() # Exit early
E0512 13:50:11.181211 139606888535808 base_runner.py:251] File "usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/gradients_util.py", line 669, in <lambda>
E0512 13:50:11.181257 139606888535808 base_runner.py:251] lambda: grad_fn(op, *out_grads))
E0512 13:50:11.181302 139606888535808 base_runner.py:251] File "usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/math_grad.py", line 1201, in _SubGrad
E0512 13:50:11.181348 139606888535808 base_runner.py:251] gy = array_ops.reshape(math_ops.reduce_sum(-grad, ry), sy)
E0512 13:50:11.181395 139606888535808 base_runner.py:251] File "usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/gen_math_ops.py", line 6304, in neg
E0512 13:50:11.181441 139606888535808 base_runner.py:251] "Neg", x=x, name=name)
E0512 13:50:11.181487 139606888535808 base_runner.py:251] File "usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/op_def_library.py", line 742, in _apply_op_helper
E0512 13:50:11.181534 139606888535808 base_runner.py:251] attrs=attr_protos, op_def=op_def)
E0512 13:50:11.181580 139606888535808 base_runner.py:251] File "usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/ops.py", line 3322, in _create_op_internal
E0512 13:50:11.181626 139606888535808 base_runner.py:251] op_def=op_def)
E0512 13:50:11.181672 139606888535808 base_runner.py:251] File "usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/ops.py", line 1756, in __init__
E0512 13:50:11.181717 139606888535808 base_runner.py:251] self._traceback = tf_stack.extract_stack()
E0512 13:50:11.181763 139606888535808 base_runner.py:251]
E0512 13:50:11.181808 139606888535808 base_runner.py:251] ...which was originally created as op 'fprop/starnet/tower_0_0/feat/validated_gin/gin/gin_concat/seq/gin_concat/seq/gin_concat/seq/gin_concat/seq/gin_concat/seq/validated_gin_intermediate/gin_intermediate/validated_mlp/mlp/validated_validated_l001/validated_l001/l001/key_seq/bn/bn/sub_2', defined at:
E0512 13:50:11.181856 139606888535808 base_runner.py:251] File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/trainer.py", line 1874, in <module>
E0512 13:50:11.181903 139606888535808 base_runner.py:251] tf.app.run(main)
E0512 13:50:11.181948 139606888535808 base_runner.py:251] [elided 7 identical lines from previous traceback]
E0512 13:50:11.181994 139606888535808 base_runner.py:251] File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/trainer.py", line 398, in __init__
E0512 13:50:11.182041 139606888535808 base_runner.py:251] self._model.ConstructFPropBPropGraph()
E0512 13:50:11.182086 139606888535808 base_runner.py:251] File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/core/base_model.py", line 1196, in ConstructFPropBPropGraph
E0512 13:50:11.182132 139606888535808 base_runner.py:251] self._task.FPropDefaultTheta()
E0512 13:50:11.182210 139606888535808 base_runner.py:251] File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/core/base_model.py", line 524, in FPropDefaultTheta
E0512 13:50:11.182261 139606888535808 base_runner.py:251] return self.FProp(self.theta, input_batch)
E0512 13:50:11.182308 139606888535808 base_runner.py:251] File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/core/base_model.py", line 445, in FProp
E0512 13:50:11.182354 139606888535808 base_runner.py:251] metrics, per_example = self._FPropSplitInputBatch(theta, input_batch)
E0512 13:50:11.182400 139606888535808 base_runner.py:251] File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/core/base_model.py", line 491, in _FPropSplitInputBatch
E0512 13:50:11.182446 139606888535808 base_runner.py:251] metrics, per_example = self.FPropTower(theta_local, batch)
E0512 13:50:11.182492 139606888535808 base_runner.py:251] File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/core/base_model.py", line 413, in FPropTower
E0512 13:50:11.182537 139606888535808 base_runner.py:251] predictions = self.ComputePredictions(theta, input_batch)
E0512 13:50:11.182582 139606888535808 base_runner.py:251] File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/tasks/car/starnet.py", line 856, in ComputePredictions
E0512 13:50:11.182628 139606888535808 base_runner.py:251] featurized_cell = self._CellFeaturizer(theta, input_batch)
E0512 13:50:11.182672 139606888535808 base_runner.py:251] File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/tasks/car/starnet.py", line 831, in _CellFeaturizer
E0512 13:50:11.182718 139606888535808 base_runner.py:251] point_input)
E0512 13:50:11.182763 139606888535808 base_runner.py:251] File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/core/builder_layers.py", line 422, in FProp
E0512 13:50:11.182809 139606888535808 base_runner.py:251] args = ch.FProp(th, *args)
E0512 13:50:11.182855 139606888535808 base_runner.py:251] File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/core/builder_layers.py", line 422, in FProp
E0512 13:50:11.182900 139606888535808 base_runner.py:251] args = ch.FProp(th, *args)
E0512 13:50:11.182945 139606888535808 base_runner.py:251] File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/core/builder_layers.py", line 422, in FProp
E0512 13:50:11.182990 139606888535808 base_runner.py:251] args = ch.FProp(th, *args)
E0512 13:50:11.183035 139606888535808 base_runner.py:251] File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/core/builder_layers.py", line 938, in FProp
E0512 13:50:11.183080 139606888535808 base_runner.py:251] out = ch.FProp(th, *args)
E0512 13:50:11.183126 139606888535808 base_runner.py:251] File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/core/builder_layers.py", line 422, in FProp
E0512 13:50:11.183171 139606888535808 base_runner.py:251] args = ch.FProp(th, *args)
E0512 13:50:11.183217 139606888535808 base_runner.py:251] File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/core/builder_layers.py", line 938, in FProp
E0512 13:50:11.183262 139606888535808 base_runner.py:251] out = ch.FProp(th, *args)
E0512 13:50:11.183307 139606888535808 base_runner.py:251] File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/core/builder_layers.py", line 422, in FProp
E0512 13:50:11.183352 139606888535808 base_runner.py:251] args = ch.FProp(th, *args)
E0512 13:50:11.183397 139606888535808 base_runner.py:251] File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/core/builder_layers.py", line 938, in FProp
E0512 13:50:11.183443 139606888535808 base_runner.py:251] out = ch.FProp(th, *args)
E0512 13:50:11.183488 139606888535808 base_runner.py:251] File "tmp/lingvo/bazel-bin/lingvo/trainer.runfiles/__main__/lingvo/core/builder_layers.py", line 422, in FProp
E0512 13:50:11.183534 139606888535808 base_runner.py:251] args = ch.FProp(th, *args)
Any pointers much appreciated.
Thanks!