Hi,

I've loaded the saved model trained on the custom dataset using model.load. When I'm predicting test set with model.predict, Instead of labels, I'm getting the sequence of numbers, which looks like it starts from the length of sequences passed to predict. Following is a screenshot for your reference.

Thank you in advance.

The way you estimate the transition_bias parameter seems illogic and incorrect to me. The parameter is calculated like this in uisrnn.utils.resize_sequence:

transit_num = 0
for entry in range(len(cluster_id) - 1):
  transit_num += (cluster_id[entry] != cluster_id[entry + 1])
bias_denominator = len(cluster_id)
bias = (transit_num + 1) / bias_denominator

So you look at how many speaker changes there are in the concatenated sequence and divide it by the number of segments. Apart from the fact that you should divide by the number of transitions between segments (len(cluster_id) - 1) and not by the number of segments, it seems to me that your code also counts as speaker change the transition between speakers that belong to different sequences. This error arises from the fact that you concatenate the sequences and later calculate transition_bias, while, actually, you should do the opposite, I reckon.

To make it clearer, here is an example. Given: seq1 = spk1 | spk2 | spk3 seq2 = spk2 | spk3| spk3

How it should be: n_speaker_changes = 3 n_transitions = 4 transition_bias = 3/4

Your calculation: concatenated_sequence = spk1 | spk2 | spk3 | spk2 | spk3| spk3 n_speaker_changes = 4 n_segments = 6 transition_bias = 2/3

This is not only a problem with toy examples. Imagine you have 1000 short sequences as train data, at the numerator you add a very big number n_sequences - 1 = 999. This can, in my opinion, hurt performance in some particular cases.

So the estimation you provide in the code is different from the one in the paper. Not considering enforce_cluster_id_uniqueness your estimation is something like: (speaker_changes + n_sequences - 1) / n_segments According to the paper, it should look more like: spaker_changes / (n_segments - n_sequences)

Also, there is no constraint on when you calculate it. It can be computed before calling fit_concatenated in the fit function. In the unlikely case that fit_concatenated is called straight from the user there is no way to calculate it, then maybe there could be a check to require the user to calculate transition_bias beforehand.

Sorry if I misunderstood something or got it all wrong. I'm here to help.

P.S.1: why don't you estimate crp_alpha the same way you do with transition_bias? Just because it is not possible in fit_concatenated? Estimating both parameters in the fit function seems reasonable. P.S.2: It feels a bit inelegant and misleading to calculate transition_bias inside a function that is called resize_sequence

Describe the question

Documentation states that one can only apply prediction one sequence at a time.

On my machine (with GPU), it takes more than 10s to process one sequence with 100 samples. Would be nice to support batch prediction to make processing large collection of sequences faster. Beam search probably makes it impossible though.

Anyway, thanks for open-sourcing this implementation. This is really appreciated!

My background

Have I read the README.md file?

• yes

Have I searched for similar questions from closed issues?

• yes

Have I tried to find the answers in the paper Fully Supervised Speaker Diarization?

• yes

Have I tried to find the answers in the reference Speaker Diarization with LSTM?

• not applicable

Have I tried to find the answers in the reference Generalized End-to-End Loss for Speaker Verification?

• not applicable

Hi, I run this demo.py twice. The first time it works well,and its accurancy is 1.But when i delete the model and try again it works but its result is only about 0.8.I'm sure i didn't change the program.I have tried to deleted the whole program and git clone it again. Its result is still about 0.8.Then I run run_test.sh and got a error,

====================================================================== FAIL: test_four_clusters (main.TestIntegration) Four clusters on vertices of a square.

Traceback (most recent call last): File "tests/integration_test.py", line 99, in test_four_clusters self.assertEqual(1.0, accuracy) AssertionError: 1.0 != 0.9

Ran 1 test in 17.543s

FAILED (failures=1)

There must be something strange happens.Could anyone tell me why could lead to this happen? Thanks.

Are input d-vectors for training assumed L2-normalized?

In Generalized End-to-End Loss for Speaker Verification they are defined as L2-normalized in eq. 4.

In sample toy_training_data.npz, they are also L2-normalized:

cd uis-rnn
python3 -c 'import numpy; train_data = numpy.load("./data/toy_training_data.npz", allow_pickle=True); print((tr
ain_data["train_sequence"] ** 2).sum(axis = 1))'
# [1. 1. 1. ... 1. 1. 1.]

But eq. 11 from Fully Supervised Speaker Diarization models segment speaker embeddings as normally-distributed vectors and does not assume unit-length explicitly (if it did, maybe a von Mises–Fisher distribution would be a better distribution).

Thank you!

I've noticed that the training args accepts a given value of crp_alpha, and there were issues about adding support for estimation of crp_alpha.

I've added a script, which accepts the train_sequence and train_cluster_id loaded from './data/toy_training_data.npz', iterate through a searching range, and gives the best crp_alpha value estimated from training data.

The script is pretty simply actually. The steps were as follows:

1. Iterate through a range of alpha
2. Iterate through all training samples
3. Calculate p(y|z) for each sample
4. Return the alpha with highest p(y|z)

I hope this script will help some people.

I want to know whether the loss below is normal or not，I set the batchsize=10 then ,no matter how I change dataset, the converge loss is about 900。

My background

Have I read the README.md file?

• yes

Have I searched for similar questions from closed issues?

• yes

Have I tried to find the answers in the paper Fully Supervised Speaker Diarization?

• yes

Have I tried to find the answers in the reference Speaker Diarization with LSTM?

• yes

Have I tried to find the answers in the reference Generalized End-to-End Loss for Speaker Verification?

• yes

Describe the question

Hi, I have been working on this issue for almost a month. I finally manage to get a good EER on the training of LSTM, and now training on UIS-RNN.

I have a question about the d-vector.

so here the specification for your training is: sampling rate: 16K for mel-transform: [25ms window, 10ms hop length] training LSTM [140-180 frame], lets say we fix to 160 frames training UIS-RNN: 400ms segment-level d-vector

so here 25ms audio will become 1 frame of mel-spectrum, 160 frame is roughly 1.6s(with hop) means for training the LSTM, we are acutally feeding 1.6s audio into LSTM.

However, for UIS-RNN, you mention we need to use VAD to segment the audio into maximum 400ms segments.

so a 400ms segmented audio after [25ms window, 10ms hop length] mel-transform will only gives you around 40 frames, and how can we generate multiple d-vectors from this 40 frame data in order to get a segment-level d-vector? (because you mention we need to L2 norm and average them).

I really appreciate that you can give me some instruction on this point.

My background

Have I read the README.md file?

• yes

Have I searched for similar questions from closed issues?

• yes

Have I tried to find the answers in the paper Fully Supervised Speaker Diarization?

• yes, I read the paper again and again

Have I tried to find the answers in the reference Speaker Diarization with LSTM?

• yes, I read the paper again and again

Have I tried to find the answers in the reference Generalized End-to-End Loss for Speaker Verification?

• yes, I read the paper again and again

I spent some time trying to figure out what the GRU really does. My understanding is that it is used to estimate the running mean (mu_t in the paper) of each cluster.

I can see the benefit of a RNN for this (it can learn to not take some noisy samples into account) but I am wondering whether you had the chance to compare to an actual running mean.

@wq2012 The overlap rate seems strongly influenced the number of speakers. Since when overlap size is larger, the speaker embedding will change more smoothly, and the change points will hard to detect, it's apt to generate fewer speakers. And the size of sliding window also affects a lot, although this problem is caused by the speaker embedding algorithm. This is my project integrates with the vgg-speaker-recognition algorithm: Speaker-Diarization Thanks a lot.

This pull request fixes issue #55.

What is in this pull request:

• new function uisrnn.utils.estimate_transition_bias that calculates transition_bias correctly
• removed transition_bias estimation snippet from uisrnn.utils.resize_sequence. Now resize_sequence only retuns sub_sequences and seq_lengths
• moved transition_bias update (for multiple calls to fit) from fit_concatenated to fit
• added warning message if train_sequences is already concatenated and transition_bias has not been passed as argument, saying that the estimation can be incorrect
• edited few test scripts to adapt to the new shape of the data returned from uisrnn.utils.resize_sequence

I also wanted to add _ to fit_concatenated to make it private so that the user is discouraged to call it, but I didn't because maybe it has some other dependency to some internal code of yours.

Describe the question

if i enable gpu args.enable_cuda=True by default it use 'cuda:0' but i have multiple gpu on my pc so i think we can make an argument or environment variable CUDA_VISIBLE_DEVICES to chose manually.

My background

Have I read the README.md file?

• yes

Have I searched for similar questions from closed issues?

• yes

Have I tried to find the answers in the paper Fully Supervised Speaker Diarization?

• yes

Have I tried to find the answers in the reference Speaker Diarization with LSTM?

• yes

Have I tried to find the answers in the reference Generalized End-to-End Loss for Speaker Verification?

• yes

Describe the bug

Making a prediction on CPU after training on GPU, getting running time error.

To Reproduce

Train UIS-RNN on GPU Predict on CPU only machine

Commands and arguments

Default arguments.

Logs

Traceback (most recent call last):
  File ".\speakerDiarization.py", line 207, in <module>
    main(r'wavs/rmdmy.wav', embedding_per_second=1.2, overlap_rate=0.4)
  File ".\speakerDiarization.py", line 158, in main
    uisrnnModel.load(SAVED_MODEL_NAME)
  File "C:\Users\Prasanth\Desktop\VScode\stage\Speaker-Diarization\uisrnn\uisrnn.py", line 151, in load
    var_dict = torch.load(filepath)
  File "C:\Users\Prasanth\AppData\Local\Programs\Python\Python37\lib\site-packages\torch\serialization.py", line 607, in load
    return _load(opened_zipfile, map_location, pickle_module, **pickle_load_args)
  File "C:\Users\Prasanth\AppData\Local\Programs\Python\Python37\lib\site-packages\torch\serialization.py", line 882, in _load
    result = unpickler.load()
  File "C:\Users\Prasanth\AppData\Local\Programs\Python\Python37\lib\site-packages\torch\serialization.py", line 857, in persistent_load
    load_tensor(data_type, size, key, _maybe_decode_ascii(location))
  File "C:\Users\Prasanth\AppData\Local\Programs\Python\Python37\lib\site-packages\torch\serialization.py", line 846, in load_tensor
    loaded_storages[key] = restore_location(storage, location)
  File "C:\Users\Prasanth\AppData\Local\Programs\Python\Python37\lib\site-packages\torch\serialization.py", line 175, in default_restore_location
    result = fn(storage, location)
  File "C:\Users\Prasanth\AppData\Local\Programs\Python\Python37\lib\site-packages\torch\serialization.py", line 151, in _cuda_deserialize
    device = validate_cuda_device(location)
  File "C:\Users\Prasanth\AppData\Local\Programs\Python\Python37\lib\site-packages\torch\serialization.py", line 135, in validate_cuda_device
    raise RuntimeError('Attempting to deserialize object on a CUDA '
RuntimeError: Attempting to deserialize object on a CUDA device but torch.cuda.is_available() is False. If you are running on a CPU-only machine, please use torch.load with map_location=torch.device('cpu') to map your storages to the CPU.

Versions

• uis-rnn git HEAD: 0.1.0
• numpy: 1.18.5
• scipy: 1.7.3
• torch: 1.10.2

I have a speaker diarization dataset in Vietnamese, where, in every audio file, segments of speakers are already annotated. How should I prepare and process data to be able to train UIS-RNN on my custom data ?

I see that previously you answered that "for big amount of data you can fit model several times"(#8). But I didn't work with pytorch before and don't know how it is must work: how pass info about losses and gradients for different parts of dataset. That's why I want to ask if your library has ability to fit with custom data generator (like fit_generator in keras). Or maybe you can tell me where I can see example for such case.

This is what my class for data looks like(prevoiusly I save different parts of data in "data.npz"):

from torch.utils.data import Dataset
class Data(Dataset):
    def __init__(self, set, batch_size=32, shuffle=True):
        self.data = np.load("data.npz", allow_pickle=True)
        self.set = set
        self.batch_size = batch_size
        self.shuffle = shuffle
        self.on_epoch_end()

    def __len__(self):
        return int(np.floor(len(self.indexes) / self.batch_size))

    def __getitem__(self, index):
        temp_indexes = self.indexes[index * self.batch_size:(index + 1) * self.batch_size]

        sequences_batch, clusters_batch = self.__data_generation(temp_indexes)

        return sequences_batch, clusters_batch

    def on_epoch_end(self):
        self.indexes = np.arange(self.data[f'{self.set}_sequence'].shape[0])
        if self.shuffle:
            np.random.shuffle(self.indexes)

    def __data_generation(self, temp_indexes):
        sequences = self.data[f'{self.set}_sequence'][temp_indexes]
        clusters = self.data[f'{self.set}_cluster_id'][temp_indexes]
        sequences = [seq.astype(float) + 0.00001 for seq in sequences]
        clusters = [np.array(cid).astype(str) for cid in clusters]

        return sequences, clusters

And this is how I create generator:

train_set = Data("train", 32, True)
train_generator = DataLoader(train_set)

P.S.: I'll be happy to receive any help, because I don't even sure that I go in the right direction.

Describe the bug

After training the UIS-RNN algorithm, I call the predict() method with the test_sequences. In this case, the test sequence is a 2-d numpy array, not a list. Somehow the method gets stuck and never finishes. I checked the input shape and according to the method doc it should work fine.

Maybe I am doing something wrong?

To Reproduce

Commands and arguments

import numpy as np
import uisrnn

SAVED_MODEL_NAME = 'saved_model.uisrnn'


model_args, training_args, inference_args = uisrnn.parse_arguments()
model = uisrnn.UISRNN(model_args)
model.load(SAVED_MODEL_NAME)
test_sequence = np.load('./data/test_sequence.npy', allow_pickle=True)
predicted_cluster_id = model.predict(test_sequence, inference_args)

Logs

Nothing printed.

Serialized models

https://drive.google.com/open?id=1N4JwQQv27Xap-UlvHCNARTiKGT3356AX

Data samples

Data
https://www.kaggle.com/mfekadu/darpa-timit-acousticphonetic-continuous-speech
Embeddings
https://drive.google.com/open?id=1N4JwQQv27Xap-UlvHCNARTiKGT3356AX

Versions

• uis-rnn: 0.1.0
• numpy: 1.18.0
• scipy: 1.4.1
• torch: 1.3.1

Additional context

Debugging the code, I see that everything goes well until this point: https://github.com/google/uis-rnn/blob/cb3a9c6764b3ce40e1327d524eea5e1568e884b7/uisrnn/uisrnn.py#L549 The statement after the for loop is finished is not hit. Using this implementation to generate the embeddings: https://github.com/HarryVolek/PyTorch_Speaker_Verification

Describe the question

In Diarization task, i train on AMI train-dev set and ICSI corpus , i test on AMI test set. Both datasets include audios of 3-5 speakers in 50-70 minutes. My d embedding trains on Voxceleb1,2 with EER = 4.55%. I train uirnn with window size .24ms, overlap 50%, segment size .4ms. The result is poor on both train and test set. I also read all your code about uirnn, i don't understand 1> why do you split up the original utterances and concatenate them by speaker and then use that input for training? 2> Why doese the input ignore which audio the utterance belongs to, just merge all utterances in 1 single audio? .This process seems completely different to inference process and also reduce the capacity of using batch size if one speaker talk too much. For 1 hour audio, the output has 20-30 speakers instead of 3-5 speakers no matter the smaller of crp_alpha is.

My background

Have I read the README.md file?

• yes

Have I searched for similar questions from closed issues?

• yes

Have I tried to find the answers in the paper Fully Supervised Speaker Diarization?

• yes

Have I tried to find the answers in the reference Speaker Diarization with LSTM?

• yes

Have I tried to find the answers in the reference Generalized End-to-End Loss for Speaker Verification?

• yes