Bayesian Deep Learning for Manufacturing 2.0 (dlmfg)
Object Shape Error Response (OSER)
Digital Lifecycle Management - In Process Quality Improvement (IPQI)
Overview
The open source Bayesian Deep Learning for Manufacturing (dlmfg) Library is built using a TensorFlow, TensorFlow Probablity and Keras back end to build:
-
Bayesian deep learning models such as Bayesian 3D Convolutional Neural Network and Bayesian 3D U-net to enable root cause analysis in Manufacturing Systems.
-
Deep reinforcement learning models such as Deep Deterministic Policy Gradients to enable control and correction in Manufacturing Systems.
The library can be used across various domains such as assembly systems, stamping, additive manufacturing and milling where the key problem is Object Shape Error Detection and Estimation. The library is build using Object Oriented Programming to enable extension and contribution from other related disciplines within the artificial intelligence community as well as the manufacturing community.
Video
A Video for the work can be found here
Dataset
Published Work
The published works can be found below:
-
A novel hybrid shell element formulation (QUAD+ and TRIA+): A benchmarking and comparative study
-
3D convolutional neural networks to estimate assembly process parameters using 3D point-clouds
-
Deep learning enhanced digital twin for closed loop in-process quality improvement
Two follow up papers are currently under review are expected by April 2021
Documentation
The complete documentation and other ongoing research can be found here: Documentation and Research.
Highlights and New Additions
- Bayesian 3D U-Net model integrating Bayesian layers and attention blocks for uncertainty quantification and superior decoder performance leveraging the where to look capability with multi-task capabilities to estimate bot real-valued(regression) and categorical(classification) based values. The Decoder is used to obtain real-valued segmentation maps
- Deep Reinforcement Learning using deep deterministic policy gradient (DDPG) and a custom made multi physics manufacturing environment to build agents to correct manufacturing systems
- Closed Loop Sampling for faster model training and convergence using epistemic uncertainty of the Bayesian CNN models
- Matlab Python Integration to enable low latency connection between multi-physics manufacturing environments (Matlab) and TensorFlow based DDPG agents
- Multi-Physics Manufacturing System Simulations to generate custom datasets for various fault scenarios using Variation Response Method (VRM) kernel
- Uncertainty guided continual learning to enable life long/incremental training for multiple case studies
- Automated Model Selection using Keras Tuner that enables hyperparameter optimization and benchmarking for various deep learning architectures
- Exploratory notebooks for various case studies
- 3D Gradient-weighted Class Activation Maps for interpretability of deep learning models
- Datasets for Industrial multi-station case studies for training and benchmarking deep learning models
Installation
The library can be cloned using:
Git clone https://github.com/sumitsinha/Deep_Learning_for_Manufacturing
Dataset Download
The datasets can be download by running the download_data.py file within the downloads file. The specifics of the download can be specified in the download\config.py file.
The library consists of the following two key datasets:
- 3D Cloud of Point data with node deviations and process parameters for Single Part Car Halo Reinforcement – Obtained due to variations in the Measurement Station locators and Stamping Process
- 3D Cloud of Point data with node deviations and process parameters for Two part assembly for Car Door Inner and Hinge Reinforcement – Obtained due to variations in the Assembly System locators and joining tools.
Bayesian 3D CNN Model Architecture
Motivated by the recent development of Bayesian Deep Neural Networks Bayesian models considering parameters to be distributions have been build using TensorFlow Probability. The Aleatoric uncertainty have been modelled using Multi-variate normal distributions as outputs while the epistemic distributions have been modelled using distributions on model parameters by using Flip-out layers.
The Bayesian 3D CNN model for single station system has the following layers: 
> negloglik = lambda y, rv_y: -rv_y.log_prob(y)
> model = tf.keras.Sequential([
tf.keras.layers.InputLayer(input_shape=(voxel_dim,voxel_dim,voxel_dim,deviation_channels)), tfp.layers.Convolution3DFlipout(32, kernel_size=(5,5,5),strides=(2,2,2),activation=tf.nn.relu),
tfp.layers.Convolution3DFlipout(32, kernel_size=(4,4,4),strides=(2,2,2),activation=tf.nn.relu),
tfp.layers.Convolution3DFlipout(32, kernel_size=(3,3,3),strides=(1,1,1),activation=tf.nn.relu),
tf.keras.layers.MaxPooling3D(pool_size=[2, 2, 2]),
tf.keras.layers.Flatten(),
tfp.layers.DenseFlipout(128,activation=tf.nn.relu),
tfp.layers.DenseFlipout(64,activation=tf.nn.relu),
tfp.layers.DenseFlipout(self.output_dimension),
tfp.layers.DistributionLambda(lambda t: tfd.MultivariateNormalDiag(loc=t[..., :self.output_dimension], scale_diag=aleatoric_tensor)),])
model.compile(optimizer=tf.keras.optimizers.Adam(),loss=negloglik,metrics=[tf.keras.metrics.MeanAbsoluteError()])
Bayesian 3D U-Net Model Architecture
For scaling to multi-station systems consisting of both categorical and continuous process parameters and prediction of point-clouds (object shape error) in previous stations a 3D - Net Attention based architecture is leveraged.
Down-sampling Kernel 
Attention based Up-Sampling Kernel 
def attention_block(x, g, inter_channel):
theta_x = Conv(inter_channel, [1,1,1], strides=[1,1,1])(x)
phi_g = Conv(inter_channel, [1,1,1], strides=[1,1,1])(g)
f = Activation('relu')(add([theta_x, phi_g]))
psi_f = Conv(1, [1,1,1], strides=[1,1,1])(f)
rate = Activation('sigmoid')(psi_f)
att_x = multiply([x, rate])
return att_x
input_size=(voxel_dim,voxel_dim,voxel_dim,deviation_channels)
inputs = Input(input_size)
x = inputs
# Down sampling
for i in range(depth):
out_channel = 2**i * filter_root
# Residual/Skip connection
res = tfp.layers.Convolution3DFlipout(out_channel, kernel_size=1, kernel_divergence_fn=kl_divergence_function,padding='same', name="Identity{}_1".format(i))(x)
# First Conv Block with Conv, BN and activation
conv1 = tfp.layers.Convolution3DFlipout(out_channel, kernel_size=3, kernel_divergence_fn=kl_divergence_function,padding='same', name="Conv{}_1".format(i))(x)
#if batch_norm:
#conv1 = BatchNormalization(name="BN{}_1".format(i))(conv1)
act1 = Activation(activation, name="Act{}_1".format(i))(conv1)
# Second Conv block with Conv and BN only
conv2 = tfp.layers.Convolution3DFlipout(out_channel, kernel_size=3, padding='same', kernel_divergence_fn=kl_divergence_function,name="Conv{}_2".format(i))(act1)
#if batch_norm:
#conv2 = BatchNormalization(name="BN{}_2".format(i))(conv2)
resconnection = Add(name="Add{}_1".format(i))([res, conv2])
act2 = Activation(activation, name="Act{}_2".format(i))(resconnection)
# Max pooling
if i < depth - 1:
long_connection_store[str(i)] = act2
x = MaxPooling(padding='same', name="MaxPooling{}_1".format(i))(act2)
else:
x = act2
feature_vector_reg=Conv(reg_kccs, 1, padding='same', activation=final_activation, name='Process_Parameter_Reg_output')(x)
process_parameter_reg=GlobalAveragePooling3D()(feature_vector_reg)
feature_vector_cla=Conv(categorical_kccs, 1, padding='same', activation=final_activation, name='Process_Parameter_Cla_output')(x)
process_parameter_cla=GlobalAveragePooling3D()(feature_vector_cla)
#feature_categorical=Flatten()(feature_vector)
#reg_output=tfp.layers.DenseFlipout(output_dimension,kernel_divergence_fn=kl_divergence_function)(process_parameter)
#Process Parameter Outputs
reg_distrbution=tfp.layers.DistributionLambda(lambda t:tfd.MultivariateNormalDiag(loc=t[..., :reg_kccs], scale_diag=aleatoric_tensor),name="regression_outputs")(process_parameter_reg)
cla_distrbution=Activation('sigmoid', name="classification_outputs")(process_parameter_cla)
#cla_distrbution=tfp.layers.DenseFlipout(categorical_kccs, kernel_divergence_fn=kl_divergence_function,activation=tf.nn.sigmoid,name="classification_outputs")(process_parameter_cla)
# Upsampling
for i in range(depth - 2, -1, -1):
out_channel = 2**(i) * filter_root
# long connection from down sampling path.
long_connection = long_connection_store[str(i)]
up1 = UpSampling(name="UpSampling{}_1".format(i))(x)
up_conv1 = Conv(out_channel, 2, activation='relu', padding='same', name="upConvSam{}_1".format(i))(up1)
attention_layer = attention_block(x=long_connection, g=up_conv1, inter_channel=out_channel // 4)
# Concatenate.
#up_conc = Concatenate(axis=-1, name="upConcatenate{}_1".format(i))([up_conv1, long_connection])
up_conc = Concatenate(axis=-1, name="upConcatenate{}_1".format(i))([up_conv1, attention_layer])
# Convolutions
up_conv2 = Conv(out_channel, 3, padding='same', name="upConv{}_1".format(i))(up_conc)
up_act1 = Activation(activation, name="upAct{}_1".format(i))(up_conv2)
up_conv2 = Conv(out_channel, 3, padding='same', name="upConv{}_2".format(i))(up_act1)
# Residual/Skip connection
res = Conv(out_channel, kernel_size=1, padding='same', use_bias=False, name="upIdentity{}_1".format(i))(up_conc)
resconnection = Add(name="upAdd{}_1".format(i))([res, up_conv2])
x = Activation(activation, name="upAct{}_2".format(i))(resconnection)
Verification and Validation
Details of verification and validation of the model on an actual system can be found here: Real System Implementation
Benchmarking
Benchmarking of the model is done against various deep learning and machine learning approaches to highlight superiority.
Decoder Outputs
The segmentations outputs for the Bayesian 3D U-Net model enables estimation of dimensional quality of products in between stages and stations of the process.
Model Interpretability
3D Grad-weighted Class Activation Maps (3D Grad-CAMS) for each level of the encoder provides insights into the working of the model and integrate a measure of trust in all estimates.
Please cite work as:
Sinha, S., Glorieux, E., Franciosa, P., & Ceglarek, D. (2019, June). 3D convolutional neural networks to estimate assembly process parameters using 3D point-clouds. In Multimodal Sensing: Technologies and Applications (Vol. 11059, p. 110590B). International Society for Optics and Photonics.
@inproceedings{Sinha2019, author = {Sinha, Sumit and Glorieux, Emile and Franciosa, Pasquale and Ceglarek, Dariusz}, booktitle = {Multimodal Sensing: Technologies and Applications}, doi = {10.1117/12.2526062}, month = {jun}, pages = {10}, publisher = {SPIE}, title = {{3D convolutional neural networks to estimate assembly process parameters using 3D point-clouds}}, year = {2019} }
Data generation has been done using a Multi-fidelity CAE simulation software known as VRM :
Franciosa, P., Palit, A., Gerbino, S., & Ceglarek, D. (2019). A novel hybrid shell element formulation (QUAD+ and TRIA+): A benchmarking and comparative study. Finite Elements in Analysis and Design, 166, 103319.
Collaboration:
Please contact Sumit Sinha, Dr Pasquale Franciosa or Prof Darek Ceglarek in case of any clarifications or collaborative work with the Digital Lifecycle Management group at WMG, University of Warwick
1 Nov 15, 2021
47 Nov 22, 2022
4 Nov 1, 2022
210 Jan 4, 2023
2 Dec 7, 2021
527 Dec 4, 2022
25.1k Jan 2, 2023
20 Dec 2, 2022
20 Aug 21, 2022
107 Dec 20, 2022
24 Jun 22, 2022
7 Oct 30, 2022
463 Dec 23, 2022
8 Nov 7, 2022
4 Apr 15, 2022
81 Dec 15, 2022
35 Oct 31, 2022
197 Nov 26, 2022
305 Dec 16, 2022