Add WebAssembly example for VAD + Moonshine models. (#1535)

Fangjun Kuang · GitHub
Commit 8436ba834c9131a1e10f031174e2644fc85f90ad 8436ba83 1 parent 3f777b3f
README.md
scripts/wasm/generate-vad-asr.py
wasm/vad-asr/app-vad-asr.js
--- a/README.md
查看文件 @8436ba8
+++ b/README.md
查看文件 @8436ba8
@@ -109,6 +109,7 @@ We also have spaces built using WebAssembly. They are listed below:
 |Real-time speech recognition (English) |[Click me][wasm-hf-streaming-asr-en-zipformer]    |[地址][wasm-ms-streaming-asr-en-zipformer]|
 |VAD + speech recognition (Chinese + English + Korean + Japanese + Cantonese) with [SenseVoice][SenseVoice]|[Click me][wasm-hf-vad-asr-zh-en-ko-ja-yue-sense-voice]| [地址][wasm-ms-vad-asr-zh-en-ko-ja-yue-sense-voice]|
 |VAD + speech recognition (English) with [Whisper][Whisper] tiny.en|[Click me][wasm-hf-vad-asr-en-whisper-tiny-en]| [地址][wasm-ms-vad-asr-en-whisper-tiny-en]|
+|VAD + speech recognition (English) with [Moonshine tiny][Moonshine tiny]|[Click me][wasm-hf-vad-asr-en-moonshine-tiny-en]| [地址][wasm-ms-vad-asr-en-moonshine-tiny-en]|
 |VAD + speech recognition (English) with Zipformer trained with [GigaSpeech][GigaSpeech]    |[Click me][wasm-hf-vad-asr-en-zipformer-gigaspeech]| [地址][wasm-ms-vad-asr-en-zipformer-gigaspeech]|
 |VAD + speech recognition (Chinese) with Zipformer trained with [WenetSpeech][WenetSpeech]  |[Click me][wasm-hf-vad-asr-zh-zipformer-wenetspeech]| [地址][wasm-ms-vad-asr-zh-zipformer-wenetspeech]|
 |VAD + speech recognition (Japanese) with Zipformer trained with [ReazonSpeech][ReazonSpeech]|[Click me][wasm-hf-vad-asr-ja-zipformer-reazonspeech]| [地址][wasm-ms-vad-asr-ja-zipformer-reazonspeech]|
@@ -240,7 +241,7 @@ for more models. The following table lists only **SOME** of them.
 |Name | Supported Languages| Description|
 |-----|-----|----|
 |[Whisper tiny.en](https://github.com/k2-fsa/sherpa-onnx/releases/download/asr-models/sherpa-onnx-whisper-tiny.en.tar.bz2)|English| See [also](https://k2-fsa.github.io/sherpa/onnx/pretrained_models/whisper/tiny.en.html)|
-|[Moonshine tiny](https://github.com/k2-fsa/sherpa-onnx/releases/download/asr-models/sherpa-onnx-moonshine-tiny-en-int8.tar.bz2)|English|See [also](https://github.com/usefulsensors/moonshine)|
+|[Moonshine tiny][Moonshine tiny]|English|See [also](https://github.com/usefulsensors/moonshine)|
 |[sherpa-onnx-sense-voice-zh-en-ja-ko-yue-2024-07-17][sherpa-onnx-sense-voice-zh-en-ja-ko-yue-2024-07-17]|Chinese, Cantonese, English, Korean, Japanese| 支持多种中文方言. See [also](https://k2-fsa.github.io/sherpa/onnx/sense-voice/index.html)|
 |[sherpa-onnx-paraformer-zh-2024-03-09][sherpa-onnx-paraformer-zh-2024-03-09]|Chinese, English| 也支持多种中文方言. See [also](https://k2-fsa.github.io/sherpa/onnx/pretrained_models/offline-paraformer/paraformer-models.html#csukuangfj-sherpa-onnx-paraformer-zh-2024-03-09-chinese-english)|
 |[sherpa-onnx-zipformer-ja-reazonspeech-2024-08-01][sherpa-onnx-zipformer-ja-reazonspeech-2024-08-01]|Japanese|See [also](https://k2-fsa.github.io/sherpa/onnx/pretrained_models/offline-transducer/zipformer-transducer-models.html#sherpa-onnx-zipformer-ja-reazonspeech-2024-08-01-japanese)|
@@ -320,6 +321,8 @@ Video demo in Chinese: [辷ｺｼ∫き逾樊蕗菴蠑謇灘ｭ玲撃∫悄豁｣蠖ｱ蜩崎邇
 [wasm-ms-vad-asr-zh-en-ko-ja-yue-sense-voice]: https://www.modelscope.cn/studios/csukuangfj/web-assembly-vad-asr-sherpa-onnx-zh-en-jp-ko-cantonese-sense-voice
 [wasm-hf-vad-asr-en-whisper-tiny-en]: https://huggingface.co/spaces/k2-fsa/web-assembly-vad-asr-sherpa-onnx-en-whisper-tiny
 [wasm-ms-vad-asr-en-whisper-tiny-en]: https://www.modelscope.cn/studios/csukuangfj/web-assembly-vad-asr-sherpa-onnx-en-whisper-tiny
+[wasm-hf-vad-asr-en-moonshine-tiny-en]: https://huggingface.co/spaces/k2-fsa/web-assembly-vad-asr-sherpa-onnx-en-moonshine-tiny
+[wasm-ms-vad-asr-en-moonshine-tiny-en]: https://www.modelscope.cn/studios/csukuangfj/web-assembly-vad-asr-sherpa-onnx-en-moonshine-tiny
 [wasm-hf-vad-asr-en-zipformer-gigaspeech]: https://huggingface.co/spaces/k2-fsa/web-assembly-vad-asr-sherpa-onnx-en-zipformer-gigaspeech
 [wasm-ms-vad-asr-en-zipformer-gigaspeech]: https://www.modelscope.cn/studios/k2-fsa/web-assembly-vad-asr-sherpa-onnx-en-zipformer-gigaspeech
 [wasm-hf-vad-asr-zh-zipformer-wenetspeech]: https://huggingface.co/spaces/k2-fsa/web-assembly-vad-asr-sherpa-onnx-zh-zipformer-wenetspeech
@@ -405,3 +408,4 @@ Video demo in Chinese: [辷ｺｼ∫き逾樊蕗菴蠑謇灘ｭ玲撃∫悄豁｣蠖ｱ蜩崎邇
 [sherpa-onnx-telespeech-ctc-int8-zh-2024-06-04]: https://github.com/k2-fsa/sherpa-onnx/releases/download/asr-models/sherpa-onnx-telespeech-ctc-int8-zh-2024-06-04.tar.bz2
 [sherpa-onnx-sense-voice-zh-en-ja-ko-yue-2024-07-17]: https://github.com/k2-fsa/sherpa-onnx/releases/download/asr-models/sherpa-onnx-sense-voice-zh-en-ja-ko-yue-2024-07-17.tar.bz2
 [sherpa-onnx-streaming-zipformer-fr-2023-04-14]: https://github.com/k2-fsa/sherpa-onnx/releases/download/asr-models/sherpa-onnx-streaming-zipformer-fr-2023-04-14.tar.bz2
+[Moonshine tiny]: https://github.com/k2-fsa/sherpa-onnx/releases/download/asr-models/sherpa-onnx-moonshine-tiny-en-int8.tar.bz2
--- a/scripts/wasm/generate-vad-asr.py
查看文件 @8436ba8
+++ b/scripts/wasm/generate-vad-asr.py
查看文件 @8436ba8
@@ -52,6 +52,24 @@ def get_models():
             """,
         ),
         Model(
+            model_name="sherpa-onnx-moonshine-tiny-en-int8",
+            hf="k2-fsa/web-assembly-vad-asr-sherpa-onnx-en-moonshine-tiny",
+            ms="csukuangfj/web-assembly-vad-asr-sherpa-onnx-en-moonshine-tiny",
+            short_name="vad-asr-en-moonshine_tiny",
+            cmd="""
+            pushd $model_name
+            mv -v preprocess.onnx ../moonshine-preprocessor.onnx
+            mv -v encode.int8.onnx ../moonshine-encoder.onnx
+            mv -v uncached_decode.int8.onnx ../moonshine-uncached-decoder.onnx
+            mv -v cached_decode.int8.onnx ../moonshine-cached-decoder.onnx
+            mv -v tokens.txt ../
+            popd
+            rm -rf $model_name
+            sed -i.bak 's/Zipformer/Moonshine tiny supporting English 英文/g' ../index.html
+            git diff
+            """,
+        ),
+        Model(
             model_name="sherpa-onnx-sense-voice-zh-en-ja-ko-yue-2024-07-17",
             hf="k2-fsa/web-assembly-vad-asr-sherpa-onnx-zh-en-ja-ko-cantonese-sense-voice",
             ms="csukuangfj/web-assembly-vad-asr-sherpa-onnx-zh-en-jp-ko-cantonese-sense-voice",
--- a/wasm/vad-asr/app-vad-asr.js
查看文件 @8436ba8
+++ b/wasm/vad-asr/app-vad-asr.js
查看文件 @8436ba8
@@ -111,6 +111,13 @@ function initOfflineRecognizer() {
     };
   } else if (fileExists('telespeech.onnx')) {
     config.modelConfig.telespeechCtc = './telespeech.onnx';
+  } else if (fileExists('moonshine-preprocessor.onnx')) {
+    config.modelConfig.moonshine = {
+      preprocessor: './moonshine-preprocessor.onnx',
+      encoder: './moonshine-encoder.onnx',
+      uncachedDecoder: './moonshine-uncached-decoder.onnx',
+      cachedDecoder: './moonshine-cached-decoder.onnx'
+    };
   } else {
     console.log('Please specify a model.');
     alert('Please specify a model.');