Add "load_by_name" API at wasi-nn #4267

HongxiaWangSSSS · 2025-05-12T03:10:38Z

When using WASI-NN ,we want to reduce copying the AI model from the host to the WASM by using "load_by_name" API.
We also want to use it to improve the performance, keep the safety and WASI-NN also supports this method on different backends.

I test with 3 tflite models.(x86_64, Ubuntu 22.04)

Both coco_ssd_mobilenet_v1 and coco_ssd_mobilenet_v3 are for detection, its file size and input tensor size is different.
mobilenet_v2 is for classification and size is more bigger.

The time consumed does not show a linear growth as the file size increases.
For most cases, load_by_name will more faster whether the load tflite or the entire inference process (load+ set input +compute +get output).

…y_name

yamt · 2025-05-12T10:03:27Z

core/iwasm/libraries/wasi-nn/include/wasi_nn.h

@@ -30,7 +30,7 @@ load(graph_builder_array *builder, graph_encoding encoding,
    __attribute__((import_module("wasi_nn")));

 wasi_nn_error
-load_by_name(const char *name, graph *g)
+load_by_name(char *name, uint32_t name_len, graph *g)


is this a bug fix?

wasi_nn.h is a header for WebAssembly applications written in the C language. Is there a specific reason that we need to change it?

yes, its a bugfix,

Since we have two sets of APIs for historical reasons, we might remove one in another PR. For now, let's ensure both are functional.

I suggest we use WASM_ENABLE_WASI_EPHEMERAL_NN on the wasm side.

With this flag, we declare two sets of APIs in wasi_nn.h. Please align with the content of native_symbols_wasi_nn in wasi_nn.c

lum1n0us · 2025-05-12T12:34:26Z

core/iwasm/libraries/wasi-nn/src/wasi_nn.c

@@ -697,6 +697,7 @@ static NativeSymbol native_symbols_wasi_nn[] = {
    REG_NATIVE_FUNC(get_output, "(ii*i*)i"),
 #else  /* WASM_ENABLE_WASI_EPHEMERAL_NN == 0 */
    REG_NATIVE_FUNC(load, "(*ii*)i"),
+    REG_NATIVE_FUNC(load_by_name, "(*i*)i"),


Why not use -DWASM_ENABLE_WASI_EPHEMERAL_NN=1?

at first, i just built with -DWASM_ENABLE_WASI_NN=1.

If use -DWAMR_BUILD_WASI_EPHEMERAL_NN=1 during compilation, you will be able to use the set of APIs, including load_by_name(). There is no need to change this line.

If -DWAMR_BUILD_WASI_EPHEMERAL_NN=1 must be added , I think there is no need to add this line.
Currently, it is possible to use the default wasi-nn instead of wasi_ephemeral_nn.

Yes. Please do it.

lum1n0us · 2025-05-12T12:39:53Z

core/iwasm/libraries/wasi-nn/include/wasi_nn.h

@@ -30,7 +30,7 @@ load(graph_builder_array *builder, graph_encoding encoding,
    __attribute__((import_module("wasi_nn")));

 wasi_nn_error
-load_by_name(const char *name, graph *g)
+load_by_name(char *name, uint32_t name_len, graph *g)


wasi_nn.h is a header for WebAssembly applications written in the C language. Is there a specific reason that we need to change it?

lum1n0us · 2025-05-12T12:40:59Z

core/iwasm/libraries/wasi-nn/src/wasi_nn_tensorflowlite.cpp

@@ -85,14 +85,11 @@ is_valid_graph(TFLiteContext *tfl_ctx, graph g)
        NN_ERR_PRINTF("Invalid graph: %d >= %d.", g, MAX_GRAPHS_PER_INST);
        return runtime_error;
    }
-    if (tfl_ctx->models[g].model_pointer == NULL) {
+    if (tfl_ctx->models[g].model_pointer == NULL
+        && tfl_ctx->models[g].model == NULL) {


The original version can output different information based on various invalid argument cases. Is there a specific reason we need to merge them?

If call load_by_name(), there is no need to save the tflite buf to model_pointer.
https://github.com/HongxiaWangSSSS/wasm-micro-runtime/blob/c5414fd28baf973e3c95db1318de4d26f88007d3/core/iwasm/libraries/wasi-nn/src/wasi_nn_tensorflowlite.cpp#L141C49-L141C68

I'm not sure why not free model_point after below operation in load()
https://github.com/HongxiaWangSSSS/wasm-micro-runtime/blob/c5414fd28baf973e3c95db1318de4d26f88007d3/core/iwasm/libraries/wasi-nn/src/wasi_nn_tensorflowlite.cpp#L151

It is required to validate TFLitesContext.models[g] for both cases, using load() and load_by_name(). It will not be acceptable if the change disables one of these cases.

Whether it is load or load_by_name, the check of models[g].model_pointerdoes not seem to be necessary, just make sure the models[g].model is not NULL maybe is enough.
Do you have any idea?

does not seem to be necessary.

Why is that?

after this operation, the models[g].model_pointer's connect has been saved in models[g].model.

lum1n0us · 2025-05-13T23:36:42Z

core/iwasm/libraries/wasi-nn/src/wasi_nn_tensorflowlite.cpp

@@ -85,14 +85,11 @@ is_valid_graph(TFLiteContext *tfl_ctx, graph g)
        NN_ERR_PRINTF("Invalid graph: %d >= %d.", g, MAX_GRAPHS_PER_INST);
        return runtime_error;
    }
-    if (tfl_ctx->models[g].model_pointer == NULL) {
+    if (tfl_ctx->models[g].model_pointer == NULL
+        && tfl_ctx->models[g].model == NULL) {


does not seem to be necessary.

Why is that?

lum1n0us · 2025-05-14T01:02:22Z

core/iwasm/libraries/wasi-nn/src/wasi_nn.c

@@ -697,6 +697,7 @@ static NativeSymbol native_symbols_wasi_nn[] = {
    REG_NATIVE_FUNC(get_output, "(ii*i*)i"),
 #else  /* WASM_ENABLE_WASI_EPHEMERAL_NN == 0 */
    REG_NATIVE_FUNC(load, "(*ii*)i"),
+    REG_NATIVE_FUNC(load_by_name, "(*i*)i"),


Yes. Please do it.

lum1n0us · 2025-05-14T01:07:54Z

core/iwasm/libraries/wasi-nn/test/utils.c

@@ -108,7 +109,12 @@ run_inference(execution_target target, float *input, uint32_t *input_size,
              uint32_t num_output_tensors)
 {
    graph graph;
+
+#if USE_WASM_LOAD_BY_NAME == 0


Once we have two API set in wasi_nn.h separated by WASM_ENABLE_WASI_EPHEMERAL_NN:

Wrap wasm_load_by_name() with WASM_ENABLE_WASI_EPHEMERAL_NN.

There is no need to introduce USE_WASM_LOAD_BY_NAME.

Use WASM_ENABLE_WASI_EPHEMERAL_NN to decide whether to use wasm_load() or wasm_load_by_name().

lum1n0us · 2025-05-14T01:24:27Z

core/iwasm/libraries/wasi-nn/include/wasi_nn.h

@@ -30,7 +30,7 @@ load(graph_builder_array *builder, graph_encoding encoding,
    __attribute__((import_module("wasi_nn")));

 wasi_nn_error
-load_by_name(const char *name, graph *g)
+load_by_name(char *name, uint32_t name_len, graph *g)


Since we have two sets of APIs for historical reasons, we might remove one in another PR. For now, let's ensure both are functional.

I suggest we use WASM_ENABLE_WASI_EPHEMERAL_NN on the wasm side.

With this flag, we declare two sets of APIs in wasi_nn.h. Please align with the content of native_symbols_wasi_nn in wasi_nn.c

add wasm load by name for WASI-NN

1df9b7c

HongxiaWangSSSS marked this pull request as ready for review May 12, 2025 05:24

HongxiaWangSSSS requested review from loganek, lum1n0us, no1wudi, TianlongLiang, wenyongh, xujuntwt95329 and yamt as code owners May 12, 2025 05:24

Merge branch 'bytecodealliance:main' into feature/support_wasm_load_b…

c5414fd

…y_name

yamt reviewed May 12, 2025

View reviewed changes

lum1n0us reviewed May 12, 2025

View reviewed changes

lum1n0us reviewed May 14, 2025

View reviewed changes

remove wasm_load_by_name from old wasi-nn

ff4ed2f

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add "load_by_name" API at wasi-nn #4267

Add "load_by_name" API at wasi-nn #4267

HongxiaWangSSSS commented May 12, 2025

yamt May 12, 2025

lum1n0us May 12, 2025

HongxiaWangSSSS May 13, 2025

lum1n0us May 14, 2025

lum1n0us May 12, 2025

HongxiaWangSSSS May 13, 2025

lum1n0us May 13, 2025

HongxiaWangSSSS May 13, 2025

lum1n0us May 14, 2025

lum1n0us May 12, 2025

lum1n0us May 12, 2025

HongxiaWangSSSS May 13, 2025

HongxiaWangSSSS May 13, 2025

lum1n0us May 13, 2025

HongxiaWangSSSS May 13, 2025

lum1n0us May 13, 2025

HongxiaWangSSSS May 19, 2025

lum1n0us May 13, 2025

lum1n0us May 14, 2025

lum1n0us May 14, 2025

HongxiaWangSSSS May 19, 2025

lum1n0us May 14, 2025

Add "load_by_name" API at wasi-nn #4267

Are you sure you want to change the base?

Add "load_by_name" API at wasi-nn #4267

Conversation

HongxiaWangSSSS commented May 12, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment