hello，I want to rapeat your work and rewrite it by ｐｙtorch.　can you tell me more about the detail about training，＂A base learning rate 5e-5 was used for pretraining＂you mean pretrain in the same dataset(Koniq-10k and livec)?

def create_triq_model(n_quality_levels,
                      input_shape=(None, None, 3),
                      backbone='resnet50',
                      transformer_params=(2, 32, 8, 64),
                      maximum_position_encoding=193,
                      vis=False):
    chanDim = -1
    # define the model input
    inputs = Input(shape=input_shape)
    filters = (32, 64, 128)
    # loop over the number of filters
    for (i, f) in enumerate(filters):
        # if this is the first CONV layer then set the input
        # appropriately
        if i == 0:
            x = Rescaling(1./255)(inputs)

        # CONV => RELU => BN => POOL
        x = Conv2D(f, (3, 3), padding="same")(x)
        x = Activation("relu")(x)
        x = BatchNormalization(axis=chanDim)(x)
        x = MaxPooling2D(pool_size=(2, 2))(x)
    
    x = Conv2D(256, (3, 3), padding="same")(x)
    x = Activation("relu")(x)
    x = BatchNormalization(axis=chanDim)(x)
    x = MaxPooling2D(pool_size=(2, 2))(x)
    
    x = ZeroPadding2D(padding=(1, 1))(x)
    x = Conv2D(2048, (3, 3), padding="same")(x)
    x = Activation("relu")(x)
    x = BatchNormalization(axis=chanDim)(x)
    x = MaxPooling2D(pool_size=(2, 2))(x)
    dropout_rate = 0.1
    
    transformer = TriQImageQualityTransformer(
        num_layers=transformer_params[0],
        d_model=transformer_params[1],
        num_heads=transformer_params[2],
        mlp_dim=transformer_params[3],
        dropout=dropout_rate,
        n_quality_levels=n_quality_levels,
        maximum_position_encoding=maximum_position_encoding,
        vis=vis
    )
    outputs = transformer(x)
  
    model = Model(inputs=inputs, outputs=outputs)
    model.summary()
    return model

gpus = tf.config.experimental.list_physical_devices('GPU')
tf.config.experimental.set_visible_devices(gpus[0], 'GPU')
input_shape = (564, 504, 3)
#model = create_triq_model(n_quality_levels=5, input_shape=input_shape, backbone='vgg16')
model = create_triq_model(n_quality_levels=1, input_shape=input_shape, backbone='resnet50')

from tensorflow.keras.optimizers import Adam
opt = Adam(learning_rate=0.001, decay=1e-3 / 200)
model.compile(loss="mean_squared_error", optimizer=opt)
model.fit(trainImagesX, trainY, validation_data=(valImagesX, valY),
          epochs=108, batch_size=16)

In the above code, I have modified the create_triq_model function in such a way that it uses a custom CNN model instead of the RSNET or VGGNet. The custom CNN model is such that its output shape is (18, 16, 2048). This output is fed to TriqImageQualityTransformer.

The issue is that after training the model predicts the same value for every input. I have experimented with various hyperparameters. It might output different values for different hyperparameter settings but for a particular setting, for every image as input, it outputs the same output. One more thing to note is that if I do not use a transformer but instead use an Artificial Neural Network, then the network trains well.

Ca you please suggest what am I doing wrong here?

thanks for your work. it is very cool. I test jpg image with size 1919 × 1440. it will show me that:

tensorflow.python.framework.errors_impl.InvalidArgumentError:  Incompatible shapes: [1,661,32] vs. [1,193,32]
	 [[node model/tri_q_image_quality_transformer/add_1 (defined at /data2/zhx3/triq/src/models/transformer_iqa.py:198) ]] [Op:__inference_predict_function_9663]

Errors may have originated from an input operation.
Input Source operations connected to node model/tri_q_image_quality_transformer/add_1:
 model/tri_q_image_quality_transformer/concat (defined at /data2/zhx3/triq/src/models/transformer_iqa.py:194)

Function call stack:
predict_function

Hello, I have been reproducing this project recently, and it feels great. Now I encounter a problem. I want to input two pictures at a time (or input one, and then enter the model and then segment it), I have read it for a long time, but I didn't find where to modify it. For the input-shape of None type, I can do nothing. Looking forward to your comments and guidance, thank you very much. - a beginner

Hello! I followed all the instructions for training and prepared the data & labels accordingly. When I ran the training script it runs for a few steps say 170/2135 and then it stops throwing exception errors.

I then changed return np.array(images_aug), np.array(y_scores) to return np.array(images_aug, dtype='object'), np.array(y_scores, dtype='object'), but now script is just stuck and doesn't consume much GPU memory after a while(700MB/16GB). I even tried training from scratch(not loaded ImageNet pretrained weights) but still no luck.

My conda env details: tensorflow-gpu==2.1.0 tensorflow_addons==0.8.3 h5py==2.10.0

Hello, I have a question, the data shape of koniq-10k dataset is not consistent. Some is (224,224), otherwise some is(224,224,3)。but I do not find the process about the difference. Can you tell me more about the detail? thanks a lot.

Hello, I would like to ask what is the value of PLCC of the training set you get, when the epoch of training is 120? I think the result I get is a bit wrong.

hello, i have a question. I want to predict all the pictures of koniq using the trained model. So, I used a loop to process all the pictures in the folder, but there are some problem like this, can you help me?

tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[64,386,514] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [[node model_10/pool1_pad/Pad (defined at E:/Graudate/Code/triq-master-play/src/examples/image_quality_prediction.py:21) ]] Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info. [Op:__inference_predict_function_106843]

Hi,

The requirement.txt file said that the TensorFlow version used in this project is 2.2.0. However, when I tried to run the train_triq.py file, the error happened, which said that "Tensorflow Addons supports using Python ops for all Tensorflow versions above or equal to 2.3.0 and strictly below 2.5.0 (nightly versions are not supported)". It seems like the function tensorflow_addons.activations.gelu does not support the TensorFlow 2.2.0.

I'm not familiar with TensorFlow. Therefore, I want to check the TensorFlow version and discuss why this error happened.

hello，I can't open this website http://database.mmsp-kn.de/koniq-10k-database.html to download koniq-10k dataset, do you have any method to download the data set, thanks.

Hi,

Could you please tell how you handled different image sizes as input during the training phase? Lets say we have three images of size (1080x1080), (1608x1608) and (2000x2000). If we give these images as an input to the network during training, how was this taken care of? Were the images padded with zeros to the image resolution of maximum size? Thanks.