For the BF16 graph, we usually find a concat+cast pattern from the feature column to DNN part. The optimization is for concat(FP32 -> FP32) + cast(FP32 > BF16) and concat(BF16 -> BF16) + cast(BF16 > FP32) to fuse one op.

Please make sure that this is a build/installation issue. As per our GitHub Policy, we only address code/doc bugs, performance issues, feature requests and build/installation issues on GitHub. tag:build_template

System information

• OS Platform and Distribution (e.g., Linux Ubuntu 16.04):
• Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device:
• TensorFlow installed from (source or binary):
• TensorFlow version:1.15
• Python version:2.7
• Installed using virtualenv? pip? conda?:
• Bazel version (if compiling from source):
• GCC/Compiler version (if compiling from source): g++ 7.5
• CUDA/cuDNN version:
• GPU model and memory:

Describe the problem

ERROR: /DeepRec/tensorflow/BUILD:893:1: Executing genrule //tensorflow:tf_python_api_gen_v1 failed (Exit 1)
Traceback (most recent call last):
  File "/root/.cache/bazel/_bazel_root/ac437ea991a64a55acdfc27c9ef15814/execroot/org_tensorflow/bazel-out/host/bin/tensorflow/create_tensorflow.python_api_1_tf_python_api_gen_v1.runfiles/org_tensorflow/tensorflow/python/tools/api/generator/create_python_api.py", line 27, in <module>
    from tensorflow.python.tools.api.generator import doc_srcs
  File "/root/.cache/bazel/_bazel_root/ac437ea991a64a55acdfc27c9ef15814/execroot/org_tensorflow/bazel-out/host/bin/tensorflow/create_tensorflow.python_api_1_tf_python_api_gen_v1.runfiles/org_tensorflow/tensorflow/python/__init__.py", line 73, in <module>
    from tensorflow.python.ops.standard_ops import *
  File "/root/.cache/bazel/_bazel_root/ac437ea991a64a55acdfc27c9ef15814/execroot/org_tensorflow/bazel-out/host/bin/tensorflow/create_tensorflow.python_api_1_tf_python_api_gen_v1.runfiles/org_tensorflow/tensorflow/python/ops/standard_ops.py", line 25, in <module>
    from tensorflow.python import autograph
  File "/root/.cache/bazel/_bazel_root/ac437ea991a64a55acdfc27c9ef15814/execroot/org_tensorflow/bazel-out/host/bin/tensorflow/create_tensorflow.python_api_1_tf_python_api_gen_v1.runfiles/org_tensorflow/tensorflow/python/autograph/__init__.py", line 35, in <module>
    from tensorflow.python.autograph import operators
  File "/root/.cache/bazel/_bazel_root/ac437ea991a64a55acdfc27c9ef15814/execroot/org_tensorflow/bazel-out/host/bin/tensorflow/create_tensorflow.python_api_1_tf_python_api_gen_v1.runfiles/org_tensorflow/tensorflow/python/autograph/operators/__init__.py", line 40, in <module>
    from tensorflow.python.autograph.operators.control_flow import for_stmt
  File "/root/.cache/bazel/_bazel_root/ac437ea991a64a55acdfc27c9ef15814/execroot/org_tensorflow/bazel-out/host/bin/tensorflow/create_tensorflow.python_api_1_tf_python_api_gen_v1.runfiles/org_tensorflow/tensorflow/python/autograph/operators/control_flow.py", line 65, in <module>
    from tensorflow.python.autograph.operators import py_builtins
  File "/root/.cache/bazel/_bazel_root/ac437ea991a64a55acdfc27c9ef15814/execroot/org_tensorflow/bazel-out/host/bin/tensorflow/create_tensorflow.python_api_1_tf_python_api_gen_v1.runfiles/org_tensorflow/tensorflow/python/autograph/operators/py_builtins.py", line 30, in <module>
    from tensorflow.python.data.ops import dataset_ops
  File "/root/.cache/bazel/_bazel_root/ac437ea991a64a55acdfc27c9ef15814/execroot/org_tensorflow/bazel-out/host/bin/tensorflow/create_tensorflow.python_api_1_tf_python_api_gen_v1.runfiles/org_tensorflow/tensorflow/python/data/__init__.py", line 25, in <module>
    from tensorflow.python.data import experimental
  File "/root/.cache/bazel/_bazel_root/ac437ea991a64a55acdfc27c9ef15814/execroot/org_tensorflow/bazel-out/host/bin/tensorflow/create_tensorflow.python_api_1_tf_python_api_gen_v1.runfiles/org_tensorflow/tensorflow/python/data/experimental/__init__.py", line 89, in <module>
    from tensorflow.python.data.experimental.ops.batching import dense_to_sparse_batch
  File "/root/.cache/bazel/_bazel_root/ac437ea991a64a55acdfc27c9ef15814/execroot/org_tensorflow/bazel-out/host/bin/tensorflow/create_tensorflow.python_api_1_tf_python_api_gen_v1.runfiles/org_tensorflow/tensorflow/python/data/experimental/ops/batching.py", line 20, in <module>
    from tensorflow.python.data.ops import dataset_ops
  File "/root/.cache/bazel/_bazel_root/ac437ea991a64a55acdfc27c9ef15814/execroot/org_tensorflow/bazel-out/host/bin/tensorflow/create_tensorflow.python_api_1_tf_python_api_gen_v1.runfiles/org_tensorflow/tensorflow/python/data/ops/dataset_ops.py", line 40, in <module>
    from tensorflow.python.data.ops import iterator_ops
  File "/root/.cache/bazel/_bazel_root/ac437ea991a64a55acdfc27c9ef15814/execroot/org_tensorflow/bazel-out/host/bin/tensorflow/create_tensorflow.python_api_1_tf_python_api_gen_v1.runfiles/org_tensorflow/tensorflow/python/data/ops/iterator_ops.py", line 35, in <module>
    from tensorflow.python.training.saver import BaseSaverBuilder
  File "/root/.cache/bazel/_bazel_root/ac437ea991a64a55acdfc27c9ef15814/execroot/org_tensorflow/bazel-out/host/bin/tensorflow/create_tensorflow.python_api_1_tf_python_api_gen_v1.runfiles/org_tensorflow/tensorflow/python/training/saver.py", line 57, in <module>
    from tensorflow.python.training.saving import saveable_object_util
  File "/root/.cache/bazel/_bazel_root/ac437ea991a64a55acdfc27c9ef15814/execroot/org_tensorflow/bazel-out/host/bin/tensorflow/create_tensorflow.python_api_1_tf_python_api_gen_v1.runfiles/org_tensorflow/tensorflow/python/training/saving/saveable_object_util.py", line 33, in <module>
    from tensorflow.python.training import saver
ImportError: cannot import name saver
Target //tensorflow/tools/pip_package:build_pip_package failed to build

Provide the exact sequence of commands / steps that you executed before running into the problem

bazel build --cxxopt="-D_GLIBCXX_USE_CXX11_ABI=1" --host_cxxopt="-D_GLIBCXX_USE_CXX11_ABI=1" -c opt --config=v1 --config=opt --config=mkl_threadpool --define build_with_mkl_dnn_v1_only=true //tensorflow/tools/pip_package:build_pip_package

Any other info / logs Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached.

When I use the latest commit to build a PMEM memkind environment and execute the launch script, the following error will appear.

1. The commit code version I used

2.The build option I used

bazel build --cxxopt="-D_GLIBCXX_USE_CXX11_ABI=0" --host_cxxopt="-D_GLIBCXX_USE_CXX11_ABI=0" -c opt --copt="-L/usr/local/lib" --copt="-lpmem" --copt="-lmemkind" --config=opt //tensorflow/tools/pip_package:build_pip_package

3. The scprit I used numactl -N 1 ./launch.sh --batch_size=1280 --dim_size=512 --max_mock_id_amplify=1800 --num_steps=2000 --ev_storage=pmem_memkind

4. error logs

INFO:tensorflow:Running local_init_op. INFO:tensorflow:Done running local_init_op. Traceback (most recent call last): File "/home/pai/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 1365, in _do_call return fn(*args) File "/home/pai/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 1350, in _run_fn target_list, run_metadata) File "/home/pai/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 1443, in _call_tf_sessionrun run_metadata) tensorflow.python.framework.errors_impl.InvalidArgumentError: From /job:ps/replica:0/task:0: MultiLevel EV's Cache size -1 should large than IDs in batch 1280 [[{{node fm/embedding_lookup_36}}]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "./benchmark.py", line 228, in tf.app.run() File "/home/pai/lib/python3.6/site-packages/tensorflow_core/python/platform/app.py", line 40, in run _run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef) File "/home/pai/lib/python3.6/site-packages/absl/app.py", line 312, in run _run_main(main, args) File "/home/pai/lib/python3.6/site-packages/absl/app.py", line 258, in _run_main sys.exit(main(argv)) File "./benchmark.py", line 203, in main sess.run(train_op) File "/home/pai/lib/python3.6/site-packages/tensorflow_core/python/training/monitored_session.py", line 804, in run run_metadata=run_metadata) File "/home/pai/lib/python3.6/site-packages/tensorflow_core/python/training/monitored_session.py", line 1309, in run run_metadata=run_metadata) File "/home/pai/lib/python3.6/site-packages/tensorflow_core/python/training/monitored_session.py", line 1410, in run raise six.reraise(*original_exc_info) File "/home/pai/lib/python3.6/site-packages/six.py", line 719, in reraise raise value File "/home/pai/lib/python3.6/site-packages/tensorflow_core/python/training/monitored_session.py", line 1395, in run return self._sess.run(*args, **kwargs) File "/home/pai/lib/python3.6/site-packages/tensorflow_core/python/training/monitored_session.py", line 1468, in run run_metadata=run_metadata) File "/home/pai/lib/python3.6/site-packages/tensorflow_core/python/training/monitored_session.py", line 1226, in run return self._sess.run(*args, **kwargs) File "/home/pai/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 956, in run run_metadata_ptr) File "/home/pai/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 1180, in _run feed_dict_tensor, options, run_metadata) File "/home/pai/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 1359, in _do_run run_metadata) File "/home/pai/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 1384, in _do_call raise type(e)(node_def, op, message) tensorflow.python.framework.errors_impl.InvalidArgumentError: From /job:ps/replica:0/task:0: MultiLevel EV's Cache size -1 should large than IDs in batch 1280 [[node fm/embedding_lookup_36 (defined at /home/pai/lib/python3.6/site-packages/tensorflow_core/python/framework/ops.py:1748) ]]

Original stack trace for 'fm/embedding_lookup_36': File "./benchmark.py", line 228, in tf.app.run() File "/home/pai/lib/python3.6/site-packages/tensorflow_core/python/platform/app.py", line 40, in run _run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef) File "/home/pai/lib/python3.6/site-packages/absl/app.py", line 312, in run _run_main(main, args) File "/home/pai/lib/python3.6/site-packages/absl/app.py", line 258, in _run_main sys.exit(main(argv)) File "./benchmark.py", line 121, in main tf.nn.embedding_lookup(fm_w, batch['col{}'.format(sidx)])) File "/home/pai/lib/python3.6/site-packages/tensorflow_core/python/ops/embedding_ops.py", line 418, in embedding_lookup counts=counts) File "/home/pai/lib/python3.6/site-packages/tensorflow_core/python/ops/embedding_ops.py", line 184, in _embedding_lookup_and_transform counts=counts), File "/home/pai/lib/python3.6/site-packages/tensorflow_core/python/util/dispatch.py", line 180, in wrapper return target(*args, **kwargs) File "/home/pai/lib/python3.6/site-packages/tensorflow_core/python/ops/array_ops.py", line 3958, in gather counts=counts) File "/home/pai/lib/python3.6/site-packages/tensorflow_core/python/ops/kv_variable_ops.py", line 749, in sparse_read name=name) File "/home/pai/lib/python3.6/site-packages/tensorflow_core/python/ops/gen_kv_variable_ops.py", line 647, in kv_resource_gather validate_indices=validate_indices, name=name) File "/home/pai/lib/python3.6/site-packages/tensorflow_core/python/framework/op_def_library.py", line 794, in _apply_op_helper op_def=op_def) File "/home/pai/lib/python3.6/site-packages/tensorflow_core/python/util/deprecation.py", line 507, in new_func return func(*args, **kwargs) File "/home/pai/lib/python3.6/site-packages/tensorflow_core/python/framework/ops.py", line 3357, in create_op attrs, op_def, compute_device) File "/home/pai/lib/python3.6/site-packages/tensorflow_core/python/framework/ops.py", line 3426, in _create_op_internal op_def=op_def) File "/home/pai/lib/python3.6/site-packages/tensorflow_core/python/framework/ops.py", line 1748, in init self._traceback = tf_stack.extract_stack()

Try replacing the TextLineDataset with a tensor slice dataset (see train.py), but MonitoredTrainingSession is much slower than the original. It takes roughly 100-110 seconds to create. The TextLineDataSet takes only 7 seconds. If I set checkpoint_dir to None, it can save 70 seconds.

Do you have good advice for this? Whether checkpoint_dir can be improved?

Please make sure that this is a build/installation issue. As per our GitHub Policy, we only address code/doc bugs, performance issues, feature requests and build/installation issues on GitHub. tag:build_template

System information

• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Centos 7
• Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device: no
• TensorFlow installed from (source or binary): source
• TensorFlow version: r1.15.5-deeprec2204u1
• Python version:
• Installed using virtualenv? pip? conda?:
• Bazel version (if compiling from source):
• GCC/Compiler version (if compiling from source): gcc version 8.3.1 20190311 (Red Hat 8.3.1-3) (GCC)
• CUDA/cuDNN version: cuda11.4
• GPU model and memory:

Describe the problem

Build deeprec fail when we use gcc 8.3.1. It triggers gcc 8.3.1 compiler bug. The error is as follows:

unique_ali_op_ut.h:498:77: internal compiler error: in is_normal_capture_proxy, at cp/lambda.c:292

Provide the exact sequence of commands / steps that you executed before running into the problem

Any other info / logs Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached.

When we save checkpoint, the error F ./tensorflow/core/framework/embedding/value_ptr.h:256] Unsupport GlobalStep in subclass of ValuePtrBase occurs. Because I find that the checkpoint is a temporary file best_checkpoint/best.data-00000-of-00001.tempstate11898667549733680686.

Modelzoo perf Test based on [Release] Update DeepRec release version to 1.15.5+deeprec2201. (#43). Test machines: Alibaba Cloud ECS general purpose instance family with high clock speeds - ecs.hfg7.2xlarge.

Test perf result:

Gstep | WDL | WDL | DLRM | DLRM | DeepFM | DeepFM | DSSM | DSSM | DIEN | DIEN | DIN | DIN -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- / | value | percent | value | percent | value | percent | value | percent | value | percent | value | percent Commuty TF | 31.92626 | baseline | 82.09168 | baseline | 37.20978 | baseline | 18.54726 | baseline | 14.62987 | baseline | 18.57746 | baseline DeepRec FP32 | 34.69318 | 108.67% | 105.4547 | 128.46% | 43.31713 | 116.41% | 21.64175 | 116.68% | 13.27125 | 90.71% | 17.6932 | 95.24% DeepRec BF16 | 49.38222 | 154.68% | 114.2221 | 139.14% | 47.34401 | 127.24% | 23.13698 | 124.75% | 13.0392 | 89.13% | 17.20525 | 92.61%

Test AUC result:

AUC | WDL | WDL | DLRM | DLRM | DeepFM | DeepFM | DSSM | DSSM | DIEN | DIEN | DIN | DIN -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- / | value | percent | value | percent | value | percent | value | percent | value | percent | value | percent Commuty TF | 0.775168 | baseline | 0.768852 | baseline | 0.744794 | baseline | 0.504404 | baseline | 0.8443 | baseline | 0.7887 | baseline DeepRec FP32 | 0.775515 | 100.04% | 0.771128 | 100.30% | 0.746055 | 100.17% | 0.503653 | 99.85% | 0.8472 | 100.34% | 0.7913 | 100.33% DeepRec BF16 | 0.77604 | 100.11% | 0.772185 | 100.43% | 0.741192 | 99.52% | 0.492327 | 97.61% | 0.8358 | 98.99% | 0.7883 | 99.95%

PS: DSSM dataset is small, so its ACC and AUC is limited.

测试环境 性能对比

[1] Invalid argument: Trying to access resource linear/linear_model/C1/weights/part_0 located in device /job:localhost/replica:0/task:0/device:CPU:0 from device /job:localhost/replica:0/task:0/device:GPU:0 [2] 2022-06-07 09:49:01.768708: W tensorflow/core/framework/op_kernel.cc:1651] OP_REQUIRES failed at resource_variable_ops.cc:400 : Invalid argument: Trying to access resource linear/linear_model/C12/weights/part_0 located in device /job:localhost/replica:0/task:0/device:CPU:0 from device /job:localhost/replica:0/task:0/device:GPU:0

Parallelize UnsortedSegmentSum on CPU deivce.

Under the same condition, we can see the “parallel” way is more effective. Op | Row | Col | S_id | T_nums -- | -- | -- | -- | -- UnsortedSegmentSum | 4096 | 1024 | 128 | 1 UnsortedSegmentSum | 4096 | 1024 | 128 | 2 UnsortedSegmentSum | 4096 | 1024 | 128 | 4 UnsortedSegmentSum | 4096 | 1024 | 128 | 8 UnsortedSegmentSum | 4096 | 1024 | 128 | 16

Please make sure that this is a bug. As per our GitHub Policy, we only address code/doc bugs, performance issues, feature requests and build/installation issues on GitHub. tag:bug_template

System information

• Have I written custom code (as opposed to using a stock example script provided in TensorFlow): YES
• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Linux Ubuntu 18.04 in Docker
• Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device:
• TensorFlow installed from (source or binary): source
• TensorFlow version (use command below): r1.15.5-deeprec2204-39-g0527d0b2ad8 1.15.5
• Python version: Python 3.6.9
• Bazel version (if compiling from source): Bazelisk version: v1.11.0 Build label: 0.24.1
• GCC/Compiler version (if compiling from source): gcc version 7.5.0 (Ubuntu 7.5.0-3ubuntu1~18.04)
• CUDA/cuDNN version: CUDA=11.4, V11.4.152, cuDNN 8
• GPU model and memory: NVIDIA-SMI 470.82.01 Driver Version: 470.82.01 CUDA Version: 11.4, Tesla P100 * 4, 16280MiB

You can collect some of this information using our environment capture script You can also obtain the TensorFlow version with: 1. TF 1.0: python -c "import tensorflow as tf; print(tf.GIT_VERSION, tf.VERSION)" 2. TF 2.0: python -c "import tensorflow as tf; print(tf.version.GIT_VERSION, tf.version.VERSION)"

Describe the current behavior In some kind of GPU instance in aliyun, I build DeepRec from source following this docs: https://github.com/alibaba/DeepRec#how-to-build, I confirm I enabled GPU, but in this machine, I notice my code only run on CPU, and GPU-Util is always zero and with low GPU Memory-Usage, here is a runtime capture

But on other machines, the same building and execute behavior works normally.

Here is the CPU info which works fine:

# cat /proc/cpuinfo
processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 85
model name      : Intel(R) Xeon(R) Platinum 8163 CPU @ 2.50GHz
stepping        : 4
microcode       : 0x1
cpu MHz         : 2499.998
cache size      : 33792 KB
physical id     : 0
siblings        : 16
core id         : 0
cpu cores       : 8
apicid          : 0
initial apicid  : 0
fpu             : yes
fpu_exception   : yes
cpuid level     : 22
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc eagerfpu pni pclmulqdq monitor ssse3 fma cx16 pcid sse
4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsav
eopt xsavec xgetbv1 arat
bogomips        : 4999.99
clflush size    : 64
cache_alignment : 64
address sizes   : 46 bits physical, 48 bits virtual
power management:

Here is the CPU info which works with low GPU util:

$ cat /proc/cpuinfo
processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 79
model name      : Intel(R) Xeon(R) CPU E5-2682 v4 @ 2.50GHz
stepping        : 1
microcode       : 0x1
cpu MHz         : 2499.996
cache size      : 40960 KB
physical id     : 0
siblings        : 32
core id         : 0
cpu cores       : 16
apicid          : 0
initial apicid  : 0
fpu             : yes
fpu_exception   : yes
cpuid level     : 20
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc eagerfpu pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic
movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single ibrs ibpb stibp fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm rdseed adx smap xsaveopt arat spec_ctrl intel_stibp
bogomips        : 4999.99
clflush size    : 64
cache_alignment : 64
address sizes   : 46 bits physical, 48 bits virtual
power management:

Describe the expected behavior

Code to reproduce the issue Provide a reproducible test case that is the bare minimum necessary to generate the problem.

Other info / logs Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached.

This PR mainly change the _MklFuedMatMul layout type, It deleted those unnecessary tensor format changes and reduce framework overhead.

• Before applying this PR.
• After applying this PR..

Performance changing |_MklFusedMatMul performance|Time(ms)|percent| |:--:|:--:|:--:| |DeepRec FP32 - Before|8.862|baseline| |DeepRec FP32 - After|8.689|101%|

error log:

Traceback (most recent call last):
  File "train.py", line 832, in <module>
    main()
  File "train.py", line 544, in main
    iterator = tf.data.Iterator.from_structure(train_dataset.output_types,
AttributeError: 'PrefetchDataset' object has no attribute 'output_types'

modify :/root/DeepRec/modelzoo/dlrm

diff --git a/modelzoo/dlrm/train.py b/modelzoo/dlrm/train.py
index 1cd0e7915e..5fbc5ee4f2 100644
--- a/modelzoo/dlrm/train.py
+++ b/modelzoo/dlrm/train.py
@@ -24,6 +24,7 @@ from tensorflow.python.client import timeline
 import json

 from tensorflow.python.ops import partitioned_variables
+from tensorflow.python.data.experimental.ops import parquet_dataset_ops

 # Set to INFO for tracking training, default is WARN. ERROR for least messages
 tf.logging.set_verbosity(tf.logging.INFO)
@@ -300,6 +301,22 @@ def build_model_input(filename, batch_size, num_epochs):
         features = all_columns
         return features, labels

+    def parse_parquet(value):
+        cont_defaults = [[0.0] for i in range(1, 14)]
+        cate_defaults = [[' '] for i in range(1, 27)]
+        label_defaults = [[0]]
+        column_headers = TRAIN_DATA_COLUMNS
+        record_defaults = label_defaults + cont_defaults + cate_defaults
+        columns = value
+        vs = []
+        for k,v in columns.items():
+            vs.append(v)
+        all_columns = collections.OrderedDict(zip(column_headers, vs))
+        labels = all_columns.pop(LABEL_COLUMN[0])
+        features = all_columns
+        return features, labels
+
+
     '''Work Queue Feature'''
     if args.workqueue and not args.tf:
         from tensorflow.python.ops.work_queue import WorkQueue
@@ -311,12 +328,8 @@ def build_model_input(filename, batch_size, num_epochs):

Please make sure that this is a bug. As per our GitHub Policy, we only address code/doc bugs, performance issues, feature requests and build/installation issues on GitHub. tag:bug_template

System information

• Have I written custom code (as opposed to using a stock example script provided in TensorFlow):
• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Ubuntu 16.04
• Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device: None
• TensorFlow installed from (source or binary): source
• TensorFlow version (use command below): 1.15.5+deeprec2208
• Python version: python3.6
• Bazel version (if compiling from source): 0.26.1
• GCC/Compiler version (if compiling from source): gcc version 7.5.0
• CUDA/cuDNN version: None
• GPU model and memory: None

You can collect some of this information using our environment capture script You can also obtain the TensorFlow version with: 1. TF 1.0: python -c "import tensorflow as tf; print(tf.GIT_VERSION, tf.VERSION)" 2. TF 2.0: python -c "import tensorflow as tf; print(tf.version.GIT_VERSION, tf.version.VERSION)"

Describe the current behavior I use python generate parquet files, when read parquet files use ParquetDataset and set drop_remainder=True, it return a dynamic shape Tensor.

Describe the expected behavior when use TFRecordDataset and set drop_remainder=True, it return a static shape Tensor. it should be a static shape Tensor when drop_remainder=True.

Code to reproduce the issue Provide a reproducible test case that is the bare minimum necessary to generate the problem. generate parquet:

import pandas as pd
import pyarrow as pa
import pyarrow.parquet as pq
 
schema = pa.schema([
    ('f1', pa.int64()),
    ('f2', pa.int64()),
    ('f3', pa.int64()),
    ('f4', pa.int64()),
    ('label', pa.float32())
])
 
f1 = pa.array([1, 2, 3, 4, 5], type = pa.int64())
f2 = pa.array([1, 2, 3, 4, 5], type = pa.int64())
f3 = pa.array([1, 2, 3, 4, 5], type = pa.int64())
f4 = pa.array([1, 2, 3, 4, 5], type = pa.int64())
label = pa.array([0.1, 0.2, 0.3, 0.4, 0.5], pa.float32())
 
batch = pa.RecordBatch.from_arrays(
    [f1, f2, f3, f4, label],
    schema = schema
)
table = pa.Table.from_batches([batch])
 
pq.write_table(table, 'feature.parquet')

read parquet:

import os

import tensorflow as tf
from tensorflow.python.data.experimental.ops.dataframe import DataFrame
from tensorflow.python.data.experimental.ops.parquet_dataset_ops import ParquetDataset
from tensorflow.python.data.ops import dataset_ops


def make_initializable_iterator(ds):
    r"""Wrapper of make_initializable_iterator."""
    if hasattr(dataset_ops, "make_initializable_iterator"):
        return dataset_ops.make_initializable_iterator(ds)
    return ds.make_initializable_iterator()


def parquet_map(record):
    label = record.pop("label")
    return record, label


filename = """feature.parquet"""

ds = ParquetDataset(
    filename,
    batch_size=2,
    fields=[
        DataFrame.Field("f1", tf.int64),
        DataFrame.Field("f2", tf.int64),
        DataFrame.Field("f3", tf.int64),
        DataFrame.Field("f4", tf.int64),
        DataFrame.Field("label", tf.float32),
    ],
    num_parallel_reads=8,
    drop_remainder=True,
).map(parquet_map)
ds = ds.prefetch(4)

iterator = make_initializable_iterator(ds)
features, labels = iterator.get_next()
print("f1 type is:")
print(type(features['f1']))
print('f1 shape is:')
print(features['f1'].shape)

sess_config = tf.ConfigProto(allow_soft_placement=True, log_device_placement=False)

with tf.Session(config=sess_config) as sess:
    sess.run(iterator.initializer)
    for i in range(1):
        feature, label = sess.run([features, labels])
        print(feature)
        print("Label: ")
        print(label)

Other info / logs Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached.

f1 type is:
<class 'tensorflow.python.framework.ops.Tensor'>
f1 shape is:
(?,)
2023-01-03 13:23:36.623328: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2600000000 Hz
2023-01-03 13:23:36.629326: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x4ea9280 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2023-01-03 13:23:36.629356: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
{'f1': array([1, 2]), 'f2': array([1, 2]), 'f3': array([1, 2]), 'f4': array([1, 2])}
Label: 
[0.1 0.2]

add tfrecord code: write tfrecord:

import tensorflow as tf
tf.enable_eager_execution()

# All raw values should be converted to a type compatible with tf.Example. Use
# the following functions to do these convertions.
def _bytes_feature(value):
    """Returns a bytes_list from a string / byte."""
    return tf.train.Feature(bytes_list=tf.train.BytesList(value=value))


def _float_feature(value):
    """Returns a float_list from a float / double."""
    return tf.train.Feature(float_list=tf.train.FloatList(value=value))


def _int64_feature(value):
    """Returns an int64_list from a bool / enum / int / uint."""
    return tf.train.Feature(int64_list=tf.train.Int64List(value=value))

def write_record():
    f1 = [1, 2, 3, 4, 5]
    label = [1, 2, 3, 4, 5]

    feature = {
        'label': _int64_feature(label),
        'f1': _int64_feature(f1),
    }
    
    # Create a `example` from the feature dict.
    tf_example = tf.train.Example(features=tf.train.Features(feature=feature))
  
    # Write the serialized example to a record file.
    with tf.python_io.TFRecordWriter('feature.tfrecords') as writer:
        writer.write(tf_example.SerializeToString())

if __name__ == "__main__":
    write_record()

read tfrecord:

import os

import tensorflow as tf
from tensorflow.python.data.experimental.ops.dataframe import DataFrame
from tensorflow.python.data.experimental.ops.parquet_dataset_ops import ParquetDataset
from tensorflow.python.data.ops import dataset_ops


def make_initializable_iterator(ds):
    r"""Wrapper of make_initializable_iterator."""
    if hasattr(dataset_ops, "make_initializable_iterator"):
        return dataset_ops.make_initializable_iterator(ds)
    return ds.make_initializable_iterator()


def tfrecord_map(example_proto):
    features = {}
    features['f1'] = tf.FixedLenFeature(shape=(1,), dtype=tf.int64)
    features['label'] = tf.FixedLenFeature(shape=(1,), dtype=tf.int64)
    parsed_example = tf.parse_example(example_proto, features)
    f1 = parsed_example['f1']
    label = parsed_example['label']
    features = {'f1': f1}
    labels = {'label': label}
    return features, labels


filename = """feature.tfrecords"""

dataset = tf.data.TFRecordDataset(filename)
dataset = dataset.batch(2, drop_remainder=True)
dataset = dataset.map(lambda example_proto: tfrecord_map(example_proto))
dataset = dataset.prefetch(2)
iterator = make_initializable_iterator(dataset)
features, labels = iterator.get_next()
print("f1 type is:")
print(type(features['f1']))
print('f1 shape is:')
print(features['f1'].shape)

result:

f1 type is:
<class 'tensorflow.python.framework.ops.Tensor'>
f1 shape is:
(2, 1)

Please make sure that this is a bug. As per our GitHub Policy, we only address code/doc bugs, performance issues, feature requests and build/installation issues on GitHub. tag:bug_template

System information

• Have I written custom code (as opposed to using a stock example script provided in TensorFlow): Yes
• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Ubuntu 18.04
• Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device: None
• TensorFlow installed from (source or binary): source
• TensorFlow version (use command below): r1.15.5-deeprec2210-25-ga27850bf1de 1.15.5
• Python version: Python 3.6.9
• Bazel version (if compiling from source): 0.26.1
• GCC/Compiler version (if compiling from source): gcc version 7.5.0 (Ubuntu 7.5.0-3ubuntu1~18.04)
• CUDA/cuDNN version: cuda:11.7.0-cudnn8
• GPU model and memory: NVIDIA TITAN V 12288MiB

You can collect some of this information using our environment capture script You can also obtain the TensorFlow version with: 1. TF 1.0: python -c "import tensorflow as tf; print(tf.GIT_VERSION, tf.VERSION)" 2. TF 2.0: python -c "import tensorflow as tf; print(tf.version.GIT_VERSION, tf.version.VERSION)"

Describe the current behavior

I use apache iceberg to generate parquet files, when parquet files compressed by zstd, ParquetDataset crashed with reading int64 type data. I notice DeepRec use arrow=5.0, but the arrow supports DELTA_BINARY_PACKED encoding begins at version 7.0, so I think we need to upgrade arrow version, and that won't affect other user's compatibility.

Describe the expected behavior Works expected with DELTA_BINARY_PACKED encoding.

Code to reproduce the issue Provide a reproducible test case that is the bare minimum necessary to generate the problem. part.zstd.parquet: https://drive.google.com/file/d/1CoumvsuL47trnFi4Bn6haRIsgTy9frSE/view?usp=share_link part.gz.parquet: https://drive.google.com/file/d/1V_cOrjIVTVZ5y7Q4KbHa085ay6GeaZH-/view?usp=share_link

import os

import tensorflow as tf
from tensorflow.python.data.experimental.ops.dataframe import DataFrame
from tensorflow.python.data.experimental.ops.parquet_dataset_ops import ParquetDataset
from tensorflow.python.data.ops import dataset_ops



def make_initializable_iterator(ds):
    r"""Wrapper of make_initializable_iterator."""
    if hasattr(dataset_ops, "make_initializable_iterator"):
        return dataset_ops.make_initializable_iterator(ds)
    return ds.make_initializable_iterator()


def parquet_map(record):
    label = record.pop("label")
    return record, label


filename = """part.zstd.parquet"""
# filename = 'part.gz.parquet'

# Read from a parquet file.
ds = ParquetDataset(
    filename,
    batch_size=4,
    fields=[
        DataFrame.Field("f_2672", tf.int64),
        DataFrame.Field("f_2671", tf.int64, ragged_rank=0),
        DataFrame.Field("f_2673", tf.int64, ragged_rank=0),
        DataFrame.Field("f_5196", tf.float32, ragged_rank=0),
        DataFrame.Field("f_8436", tf.float32, ragged_rank=0),
        DataFrame.Field("label", tf.int32),
    ],
    num_parallel_reads=8,
).map(parquet_map)
ds = ds.prefetch(4)

iterator = make_initializable_iterator(ds)
features, labels = iterator.get_next()

sess_config = tf.ConfigProto(allow_soft_placement=True, log_device_placement=False)

with tf.Session(config=sess_config) as sess:
    sess.run(iterator.initializer)
    for i in range(1):
        feature, label = sess.run([features, labels])
        print(feature)
        print("Label: ")
        print(label)

Other info / logs Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached.

Please make sure that this is a bug. As per our GitHub Policy, we only address code/doc bugs, performance issues, feature requests and build/installation issues on GitHub. tag:bug_template

System information

• Have I written custom code (as opposed to using a stock example script provided in TensorFlow):
• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Ubuntu 20.04.2 LTS
• Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device:
• TensorFlow installed from (source or binary):
• TensorFlow version (use command below): 1.15
• Python version: 3.6
• Bazel version (if compiling from source): 0.26.1
• GCC/Compiler version (if compiling from source):
• CUDA/cuDNN version: 11.4
• GPU model and memory: T4

Describe the current behavior I set sess_config.graph_options.optimizer_options.micro_batch_num = 2, and it occurs that

  File "/usr/local/lib/python3.8/dist-packages/tensorflow_core/python/client/session.py", line 1365, in _do_call
    return fn(*args)
  File "/usr/local/lib/python3.8/dist-packages/tensorflow_core/python/client/session.py", line 1349, in _run_fn
    return self._call_tf_sessionrun(options, feed_dict, fetch_list,
  File "/usr/local/lib/python3.8/dist-packages/tensorflow_core/python/client/session.py", line 1441, in _call_tf_sessionrun
    return tf_session.TF_SessionRun_wrapper(self._session, options, feed_dict,
tensorflow.python.framework.errors_impl.FailedPreconditionError: 2 root error(s) found.
  (0) Failed precondition: GetNext() failed because the iterator has not been initialized. Ensure that you have run the initializer operation for this iterator before getting the next element.
         [[{{node cond/IteratorGetNext_1/dup0}}]]
         [[metrics_1/ROC_cvr2_cpd_second_stay_act_metric/assert_greater_equal/Assert/AssertGuard/Assert/data_1/_22249]]
  (1) Failed precondition: GetNext() failed because the iterator has not been initialized. Ensure that you have run the initializer operation for this iterator before getting the next element.
         [[{{node cond/IteratorGetNext_1/dup0}}]]

But if I disable micro_batch, the code is running normally.

Code to reproduce the issue

_train_input_fn = tf.compat.v1.data.make_initializable_iterator(_train_input_fn)
_eval_input_fn = tf.compat.v1.data.make_initializable_iterator(_eval_input_fn)
features, labels = tf.cond(is_training, true_fn=lambda: _train_input_fn.get_next(),
                           false_fn=lambda: _eval_input_fn.get_next())
nitializer = [tf.compat.v1.global_variables_initializer(),
                                tf.compat.v1.local_variables_initializer(),
                                tf.compat.v1.tables_initializer(),
                                _train_input_fn.initializer,
                                _eval_input_fn.initializer]
sess_config = tf.compat.v1.ConfigProto(allow_soft_placement=True, log_device_placement=log_device_placement)
sess_config.gpu_options.allow_growth = True
sess_config.graph_options.optimizer_options.micro_batch_num = 2

sess_config.intra_op_parallelism_threads = intra_threads
sess_config.inter_op_parallelism_threads = inter_threads
session = tf.compat.v1.Session(config=sess_config)
with session:
    session.run(initializer)

Provide a reproducible test case that is the bare minimum necessary to generate the problem.

Other info / logs Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached.

Problem: 使用MapReduce生成对应的Parquet存储，但是使用DeepRec的ParquetDataset 抛异常了，异常如下： No supported fields found in parquet file。 对应的parquet的schema格式如下： message example {required int32 id;required binary email;} 使用的MR任务参考：https://github.com/whale2/iow-hadoop-streaming 原因： 走读代码发现没法扫描footer里面的schema，导致抛异常了。为什么没法获取schema呢？

排查步骤： (1) 自己用python代码是可以顺利读取对应的schema的。 (2) 放弃mr，使用python代码生成类似的结构的内容，ParquetDataset 是可以顺利读出来的。 (3) 对比MR生成的parquest和python本地生成的parquet格式发现，一个是bytes，一个是string。 (4) 因此怀疑是MR生成的parquet是bytes格式，导致DeepRec没法识别。

最终将schema定义成如下，问题解决： message example {required int32 id;required binary email(UTF-8);}