new WER script

[harvestingmoon]

WER testing based off speaker 6097 of the HiTTS Dataset. Audio carried is ~ 10mb and contains dozens of short 10 second audio. WER_Scripting.py would then calculate the WER via DP.

[foldl]

Are those Java class files are redundant?

[harvestingmoon]

Ah yes I think those gradle files are redundant, pls ignore them

Only use the files that are in wer_testing

[foldl]

This file is not used.

[harvestingmoon]

Fixed under new script

[harvestingmoon]

have also added some preview to cli which would be iterated at every audio loop:

../build/bin/whisper-cli -m ../models/ggml-base.en.bin -t 4 -p 1 -f ./6097_5_mins/audio/presentpictureofnsw_02_mann_0083.wav
Word transcribed is : ['and all boats to be moored within the hospital waff and hulk.']
Actual word is: and all boats to be moored within the hospital wharf and hulk
wer for audio/presentpictureofnsw_02_mann_0083.wav is 0.17

[ggerganov]

• Don't push changes to the bindings
• The audio files should not be committed in the repo, but should be downloaded instead
• The script should not parse stdout - use existing file output options
• Should handle short audio inputs (i.e. less than 1s)

[foldl]

• The audio files should not be committed in the repo, but should be downloaded instead

Downloadable datasets are huge with several hundreds of hours of audio. While I think this work item is to create some light-weight tests on WER performance which can be integrated into github workflows.

If there is a tiny dataset contained in repo, then WER benchmarking will work just out of the box and on-the-fly. A selected tiny dataset can also cover English/non-English, clean or noisy, which will be handy & useful. For example, if some sort of noise cancellation is added, then some noisy audio files can be added to the dataset and get benchmark easily & quickly.

Anyway, I suggested to add audio files to this repo. My apologies to @harvestingmoon.

[harvestingmoon]

• Should handle short audio inputs (i.e. less than 1s)

I have tried short audio inputs with the Google Command Dataset which contains audio input files approx 1s each. However, the problem with this is that whisper.cpp is unable to capture any words at all (I believe it is because the audio inputs are just too short) so there is difficulty in calculating WER. Hence, I switched over to the Hifi-TTS dataset.

No worries @foldl ! am glad to try to help / contribute 😄

I can continue slowly developing the script if given the green-light 👍🏼