Flash + language support (ref #2)

- Achieved big performance improvement + memory usage reduction
- Can now translate / transcribe different languages

Author: ggerganov

Makefile

@@ -30,11 +30,16 @@ samples:
 # runs it on all samples in the folder "./samples":
 
 .PHONY: tiny.en
+.PHONY: tiny
 .PHONY: base.en
-.PHONY: medium.en
+.PHONY: base
 .PHONY: small.en
+.PHONY: small
+.PHONY: medium.en
+.PHONY: medium
+.PHONY: large
 
-tiny.en base.en medium.en small.en: main
+tiny.en tiny base.en base small.en small medium.en medium large: main
 	bash ./download-ggml-model.sh $@
 	@echo ""
 	@echo "==============================================="

README.md

@@ -4,7 +4,8 @@ C/C++ port of [OpenAI's Whisper](https://github.com/openai/whisper) speech-to-te
 
 - Plain C/C++ implementation without dependencies
 - ARM_NEON and AVX intrinsics support
-- F16 support
+- Mixed F16 / F32 support
+- Low memory usage (Flash Attention + Flash Forward)
 
 ## Usage
 
@@ -27,9 +28,33 @@ For a quick demo, simply run `make base.en`:
 ```bash
 $ make base.en
 
-Downloading base.en (142 MB just once)
-mkdir -p models
-models/ggml-base.en.bin      100%[=================================>] 141.11M  7.50MB/s    in 19s
+gcc -pthread -O3 -mavx -mavx2 -mfma -mf16c -c ggml.c
+g++ -pthread -O3 -std=c++11 -c main.cpp
+g++ -o main ggml.o main.o
+./main -h
+
+usage: ./main [options]
+
+options:
+  -h,       --help           show this help message and exit
+  -s SEED,  --seed SEED      RNG seed (default: -1)
+  -t N,     --threads N      number of threads to use during computation (default: 4)
+  -T N,     --tokens N       maximum number of tokens to generate per iteration (default: 64)
+  -v,       --verbose        verbose output
+            --translate      translate from source language to english
+  -ps,      --print_special  print special tokens
+  -l LANG,  --language LANG  spoken language (default: en)
+  -m FNAME, --model FNAME    model path (default: models/ggml-base.en.bin)
+  -f FNAME, --file FNAME     input WAV file path (default: samples/jfk.wav)
+
+bash ./download-ggml-model.sh base.en
+Downloading ggml model base.en ...
+models/ggml-base.en.bin         100%[=====================================>] 141.11M  7.84MB/s    in 18s
+Done! Model 'base.en' saved in 'models/ggml-base.en.bin'
+You can now use it like this:
+
+  $ ./main -m models/ggml-base.en.bin -f samples/jfk.wav
+
 
 ===============================================
 Running base.en on all samples in ./samples ...
@@ -52,23 +77,24 @@ whisper_model_load: n_text_layer  = 6
 whisper_model_load: n_mels        = 80
 whisper_model_load: f16           = 1
 whisper_model_load: type          = 2
-whisper_model_load: mem_required  = 782.00 MB
+whisper_model_load: mem_required  = 611.00 MB
 whisper_model_load: adding 1607 extra tokens
-whisper_model_load: ggml ctx size = 186.26 MB
-whisper_model_load: memory size =    45.66 MB
+whisper_model_load: ggml ctx size = 163.43 MB
+whisper_model_load: memory size =    22.83 MB
 whisper_model_load: model size  =   140.54 MB
 log_mel_spectrogram: n_sample = 176000, n_len = 1100
 log_mel_spectrogram: recording length: 11.000000 s
 
- And so my fellow Americans ask not what your country can do for you. Ask what you can do for your country.
+main: processing 176000 samples (11.0 sec), 4 threads, lang = english, task = transcribe ...
 
-main:     load time =    60.62 ms
-main:      mel time =    38.69 ms
-main:   sample time =     2.36 ms
-main:   encode time =   875.63 ms / 145.94 ms per layer
-main:   decode time =   103.17 ms
-main:    total time =  1081.13 ms
+ And so my fellow Americans ask not what your country can do for you. Ask what you can do for your country.
 
+main:     load time =    71.89 ms
+main:      mel time =    36.95 ms
+main:   sample time =     2.10 ms
+main:   encode time =   700.94 ms / 116.82 ms per layer
+main:   decode time =    86.14 ms
+main:    total time =   898.72 ms
 ```
 
 The command downloads the `base.en` model converted to custom `ggml` format and runs the inference on all `.wav` samples in the folder `samples`.
@@ -81,13 +107,18 @@ make samples
 
 This will download a few more audio files from Wikipedia and convert them to 16-bit WAV format via `ffmpeg`.
 
-You can download and run the other `.en` models as follows:
+You can download and run the other models as follows:
 
 ```
 make tiny.en
+make tiny
 make base.en
+make base
 make small.en
+make small
 make medium.en
+make medium
+make large
 ```
 
 For detailed usage instructions, run: `./main -h`
@@ -101,10 +132,8 @@ ffmpeg -i input.mp3 -ar 16000 -ac 1 -c:a pcm_s16le output.wav
 
 ## Limitations
 
-- Only `.en` models are supported
 - Very basic greedy sampling scheme - always pick up the top token
 - No timestamps
-- English only
 - Inference only
 - Runs on the CPU
 - Only mono-channel 16-bit WAV is supported
@@ -113,10 +142,11 @@ ffmpeg -i input.mp3 -ar 16000 -ac 1 -c:a pcm_s16le output.wav
 
 | Model | Disk | Mem |
 | ---   | --- | --- |
-| tiny.en | 75 MB | ~600 MB |
-| base.en | 142 MB | ~800 MB |
-| small.en | 466 MB | ~1.6 GB |
-| medium.en | 1.5 GB | ~3.5 GB |
+| tiny | 75 MB | ~460 MB |
+| base | 142 MB | ~620 MB |
+| small | 466 MB | ~1.3 GB |
+| medium | 1.5 GB | ~2.8 GB |
+| large | 2.9 GB | ~4.9 GB |
 
 ## ggml format
 

download-ggml-model.sh

@@ -6,7 +6,7 @@
 ggml_path=$(dirname $(realpath $0))
 
 # Whisper models
-models=( "tiny.en" "base.en" "small.en" "medium.en" )
+models=( "tiny.en" "tiny" "base.en" "base" "small.en" "small" "medium.en" "medium" "large" )
 
 # list available models
 function list_models {