I'm currently working on a implementation where I occasionally receive raw byte data from a TCP socket. I want to pass this data into the pipeline but the current AudioSource:s seem to be limited to microphone input and audio files. Does the current version support what I'm trying to implement or do I have to write it myself?

Hi @juanmc2005 ! it's me again, sorry for bothering you... I have several questions

1. I have already fine-tuned a segmentation model with other dataset (not AIM, VoxConverse) and store in ".ckpt" model. I sure that it can be loaded by using ckpt_path = "path/where/store/ckpt/epoch=9_best.ckpt" Model.from_pretrained(ckpt_path, strict=False) i want to put replace this model in your system for some testing. But when i change here by segmentation: PipelineModel = Model.from_pretrained('path/where/store/ckpt/epoch=9_best.ckpt'). When using "python -m diart.stream", the system is still using "pyannote/segmentation". could you give me some hints?
2. Is there any useful augmentation method for training or fine-tuning? I know that can add back noise or something. Where should i add them?
3. this system can use for real-time right (streaming)? If yes, how to confirm "real-time"? like real-time factor or something else?

Thanks for your awesome project, it helps me a lot!! Looking forward to your reply~~ Best Regards

Hi,

The following example on top level README doesn't work:

from sources import MicrophoneAudioSource
from functional import FrameWiseModel, ChunkWiseModel, OverlappedSpeechPenalty, EmbeddingNormalization

mic = MicrophoneAudioSource(sample_rate=16000)

# Initialize independent modules
segmentation = FrameWiseModel("pyannote/segmentation")
embedding = ChunkWiseModel("pyannote/embedding")
osp = OverlappedSpeechPenalty(gamma=3, beta=10)
normalization = EmbeddingNormalization(norm=1)

# Branch the microphone stream to calculate segmentation
segmentation_stream = mic.stream.pipe(ops.map(segmentation))
# Join audio and segmentation stream to calculate speaker embeddings
embedding_stream = rx.zip(mic.stream, segmentation_stream).pipe(
    ops.starmap(lambda wave, seg: (wave, osp(seg))),
    ops.starmap(embedding),
    ops.map(normalization)
)

embedding_stream.suscribe(on_next=lambda emb: print(emb.shape))

mic.read()

First of all there is a type in declaring variable ops (it's been declared as osp).

Secondly, in line segmentation_stream = mic.stream.pipe(ops.map(segmentation)), there is no method named map in variable ops.

Thanks!

Hi, @juanmc2005

Is there a chance that diart can run on windows since PortAudio library is only supported on Linux ?

Also, if its possible to import Literal from typing_extensions(py3.7) instead of Typing(py3.8)

I get this error while running this demo script :

import rx
import rx.operators as ops
import diart.operators as myops
from diart.sources import MicrophoneAudioSource
import diart.functional as fn

sample_rate = 16000
mic = MicrophoneAudioSource(sample_rate)

error : ImportError: cannot import name 'Literal' from 'typing' (/usr/lib/python3.7/typing.py)

I try to use the following command test the model: python src/main.py path/to/voxconverse_test_wav/aepyx.wav --latency=0.5 --tau=0.576 --rho=0.915 --delta=0.648 --output results/vox_test_0.5s But I compared my model output with 'expected_outputs/online/0.5s/VoxConverse.rttm', and found that my output ended early for example: The last line of my output is SPEAKER aepyx 1 149.953 1.989 <NA> <NA> speaker0 <NA> <NA> But regarding the last line of the 'expected_outputs/online/0.5s/VoxConverse.rttm' of the aepyx.wav file is SPEAKER aepyx 1 168.063 0.507 <NA> <NA> F <NA> <NA> They ended at different times. I don’t know if the command I entered is wrong, or if it is due to other reasons

Hi! It's me again, sorry for bothering you. I have several questions... Q1. I try to reproduce the results of the paper by using the following hyper-parameters. SO~ I test them on the AMI test dataset and VoxConverse test dataset, but the result seems different. In the AMI dataset, there are 24 wav files. I wrote a script containing several python codes to do the testing. For each wav file, i use the command provided below. python -m diart.demo $wavpath --tau=0.507 --rho=0.006 --delta=1.057 --output ./output/AMI_dia_fintetune/ After obtain each rttm files, i calculate the DER for each wav files, like "der = metric(reference, hypothesis)" The reference rttm is from "AMI/MixHeadset.test.rttm" them I calculate "sum of DER"/total file(24files in this case), i got 0.3408511916216123 (which means 34.085% DER). Do i do something wrong....? I could provide the rttm or the DER for each wav file. The VoxConverse dataset is still processing. I'm afraid I misunderstood something, so I ask about the problem first...

BTW, I use the pyannote v1.1 to do the same things, them i got 0.48516973769490174 as final DER. # v1.1 import torch pipeline = torch.hub.load("pyannote/pyannote-audio", "dia") diarization = pipeline({"audio": "xxx.wav"}) So o'm afraid that i did something wrong....

Q2 At the same time, I have another question. Here show that you try lots of methods with different latency. Does python -m diart.demo using the "5.0 latency" way which has the greatest result in the paper? If the answer is yes, how to change different model for other latency for inference? And how to train this part?

Again, thanks for your awesome project!! Sorry for all those stupid questions... Looking forward to your reply...

hi, @juanmc2005 sorry for bothering you! as title, I want to use optuna to tuning parameters, and the segmentation model used in pipeline is my own trained follow training_a_model.ipynb. And i have already saved in "ckpt" format when i followed the "Custom models" part, try to define own segmentation model like

class MySegmentationModel(SegmentationModel):
   def __init__(self):
        super().__init__()
        self.my_pretrained_model = torch.load("./epoch=0-step=69.ckpt")  **<- put my own ckpt here**
    def __call__(
        self,
        waveform: torch.Tensor,
        weights: Optional[torch.Tensor] = None
    ) -> torch.Tensor:
        return self.my_pretrained_model(waveform, weights)

then, i redefine the config like config = PipelineConfig(segmentation=MySegmentationModel()) optimizer = Optimizer(benchmark, config, hparams, p) it comes out the error shown as below

then i trace back, and it seems that it can detect duration, sample rate, or sth could you tell me how to fix these problems or I can't use my segmentation model in optuna?

I want to do this because i guess the parameter will be affected in different model, so i want to give it a try

Thanks for your awesome work and help!!! expected your response!

I have an issue with the audio access with the example code from the README, probably due to my audio setup:

>>> mic.read()
torch.Size([4, 512])
python: src/hostapi/alsa/pa_linux_alsa.c:3641: PaAlsaStreamComponent_BeginPolling: Assertion `ret == self->nfds' failed.
Aborted (core dumped)

This is strange because I'm running this in a Docker container and I also use the sounddevice library without problems in another project in the same container.

Originally posted by @KannebTo in https://github.com/juanmc2005/StreamingSpeakerDiarization/issues/8#issuecomment-949453753

Hi Team,

Thanks for the great work with this project. I tried the model with few audio samples and it seems to work great!

I was wondering if there is any interest in developing a streaming websocket-based API endpoint to serve the model output. I went through the code and while I am not familiar with the reactive programming paradigm (RxPY), it looked like with some effort, it should be possible to develop a streaming solution.

Let me know what you all think.

Thanks and keep up the great work!

Hello!! Thanks for your sharing !! It's a masterpiece work! I am just confused about what is "oracle segmentation" in the Table.1 I can't get it~I try to survey it but failed to get the answer. thanks!

Hi, First of all thank you for this repo.

I was wondering if its possible to use the diarization pipeline to fetch the audio segments of discrete speakers. It would immensely help if there's a way around it.

Hello,

I'm trying to use the websocket server for diarization. I implemented a client as it is not provided. I get an output for the diarization but it is always speaker0.

Here is the js code for my client:

// Set up the start and stop buttons.
document.getElementById('start-button').onclick = start;
document.getElementById('stop-button').onclick = stop;

// Global variables to hold the audio stream and the WebSocket connection.
var stream;
var socket;

function b64encode(chunk) {
    // Convert the chunk array to a Float32Array
    const bytes = new Float32Array(chunk).buffer;

    // Encode the bytes as a base64 string
    let encoded = btoa(String.fromCharCode.apply(null, new Uint8Array(bytes)));

    // Return the encoded string as a UTF-8 encoded string
    return decodeURIComponent(encoded);
}


function start() {
    // Disable the start button.
    document.getElementById('start-button').disabled = true;
    // Enable the stop button.
    document.getElementById('stop-button').disabled = false;

    // Get access to the microphone.
    navigator.mediaDevices.getUserMedia({ audio: true }).then(function (s) {
        stream = s;

        // Create a new WebSocket connection.
        socket = new WebSocket('ws://localhost:7007');

        // When the WebSocket connection is open, start sending the audio data.
        socket.onopen = function () {
            var audioContext = new AudioContext();
            var source = audioContext.createMediaStreamSource(stream);
            var processor = audioContext.createScriptProcessor(1024, 1, 1);
            source.connect(processor);
            processor.connect(audioContext.destination);

            processor.onaudioprocess = function (event) {
                var data = event.inputBuffer.getChannelData(0);
                // var dataString = String.fromCharCode.apply(null, new Uint16Array(data));
                console.log("sending")
                socket.send(b64encode(data));
            };
        };

        socket.onmessage = function (e) {
            console.log(e);
        }
    }).catch(function (error) {
        console.error('Error getting microphone input:', error);
    });
}

function stop() {
    // Disable the stop button.
    document.getElementById('stop-button').disabled = true;
    // Enable the start button.
    document.getElementById('start-button').disabled = false;

    // Close the WebSocket connection.
    socket.close();
    // Stop the audio stream.
    stream.getTracks().forEach(function (track) { track.stop(); });
}

Do you have the code for the client you used ? I would like to make sure that I got that right before looking anywhere else.

Best,

Problem

The implementation of the CLI is a bit messy and mixed with the python API.

Idea

Use jsonargparse to group diart.stream, diart.tune and diart.benchmark into a single diart.py with automatic CLI documentation (i.e. removing argdoc.py). The new API would remove the dot (diart stream instead of diart.stream), --help would be generated from the docstring, and a class Diart could contain the implementation of all three scripts.

I'm relatively inexperienced when it comes to Python

when I make changes to the source code, none of the changes take effect when I run the script.

I've been trying to find solutions for days and I have had no luck.

Is it something specific to this project that is preventing changes from being made?

For instance, I simply tried to change the output of the print statement for the reports that are being printed in the console after the plot window is closed and even the simplest change is not taking effect.

Any help would be GREATLY appreciated, Thank You!

The gif in the README is not up to date with the newest changes in the API. It may be better to wait a bit for the API to stabilize before doing this.

Problem

When writing custom models or pipelines, one may want to react to specific inference events, for example before/after benchmarking on a file.

Idea

Add classes RealTimeInferenceHook and BenchmarkHook to define listener interfaces. This can also be used to implement other behavior like progress bars, writing results, etc.

Example

class MyHook(BenchmarkHook):
    def on_finished(results):
        print("Finished!")

benchmark = Benchmark(..., hooks=[MyHook()])