Soxan
در زبان پارسی به نام سخن
This repository consists of models, scripts, and notebooks that help you to use all the benefits of Wav2Vec 2.0 in your research. In the following, I'll show you how to train speech tasks in your dataset and how to use the pretrained models.
How to train
I'm just at the beginning of all the possible speech tasks. To start, we continue the training script with the speech emotion recognition problem.
Training - Notebook
| Task | Notebook |
|---|---|
| Speech Emotion Recognition (Wav2Vec 2.0) |  |
| Speech Emotion Recognition (Hubert) | |
| Audio Classification (Wav2Vec 2.0) | |
Training - CMD
python3 run_wav2vec_clf.py \
--pooling_mode="mean" \
--model_name_or_path="lighteternal/wav2vec2-large-xlsr-53-greek" \
--model_mode="wav2vec2" \ # or you can use hubert
--output_dir=/path/to/output \
--cache_dir=/path/to/cache/ \
--train_file=/path/to/train.csv \
--validation_file=/path/to/dev.csv \
--test_file=/path/to/test.csv \
--per_device_train_batch_size=4 \
--per_device_eval_batch_size=4 \
--gradient_accumulation_steps=2 \
--learning_rate=1e-4 \
--num_train_epochs=5.0 \
--evaluation_strategy="steps"\
--save_steps=100 \
--eval_steps=100 \
--logging_steps=100 \
--save_total_limit=2 \
--do_eval \
--do_train \
--fp16 \
--freeze_feature_extractor
Prediction
import torch
import torch.nn as nn
import torch.nn.functional as F
import torchaudio
from transformers import AutoConfig, Wav2Vec2FeatureExtractor
from src.models import Wav2Vec2ForSpeechClassification, HubertForSpeechClassification
model_name_or_path = "path/to/your-pretrained-model"
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
config = AutoConfig.from_pretrained(model_name_or_path)
feature_extractor = Wav2Vec2FeatureExtractor.from_pretrained(model_name_or_path)
sampling_rate = feature_extractor.sampling_rate
# for wav2vec
model = Wav2Vec2ForSpeechClassification.from_pretrained(model_name_or_path).to(device)
# for hubert
model = HubertForSpeechClassification.from_pretrained(model_name_or_path).to(device)
def speech_file_to_array_fn(path, sampling_rate):
speech_array, _sampling_rate = torchaudio.load(path)
resampler = torchaudio.transforms.Resample(_sampling_rate, sampling_rate)
speech = resampler(speech_array).squeeze().numpy()
return speech
def predict(path, sampling_rate):
speech = speech_file_to_array_fn(path, sampling_rate)
inputs = feature_extractor(speech, sampling_rate=sampling_rate, return_tensors="pt", padding=True)
inputs = {key: inputs[key].to(device) for key in inputs}
with torch.no_grad():
logits = model(**inputs).logits
scores = F.softmax(logits, dim=1).detach().cpu().numpy()[0]
outputs = [{"Emotion": config.id2label[i], "Score": f"{round(score * 100, 3):.1f}%"} for i, score in
enumerate(scores)]
return outputs
path = "/path/to/disgust.wav"
outputs = predict(path, sampling_rate)
Output:
[
{'Emotion': 'anger', 'Score': '0.0%'},
{'Emotion': 'disgust', 'Score': '99.2%'},
{'Emotion': 'fear', 'Score': '0.1%'},
{'Emotion': 'happiness', 'Score': '0.3%'},
{'Emotion': 'sadness', 'Score': '0.5%'}
]
Demos
| Demo | Link |
|---|---|
| Speech To Text With Emotion Recognition (Persian) - soon | huggingface.co/spaces/m3hrdadfi/speech-text-emotion |
114 Dec 12, 2022
85 Dec 1, 2022
292 Dec 25, 2022
87 Dec 21, 2022
43 Nov 27, 2022
145 Dec 30, 2022
3 Oct 11, 2022
6 Oct 4, 2022
3 Jan 4, 2023
14 Dec 2, 2022
565 Jan 4, 2023
23 Jan 25, 2022
24 Nov 24, 2022
67 Dec 1, 2022
2k Jan 4, 2023
5 Nov 11, 2022
1 Feb 13, 2022
255 Dec 29, 2022