# ohos_model_benchmark

**Repository Path**: ybf521/ohos_model_benchmark

## Basic Information

- **Project Name**: ohos_model_benchmark
- **Description**: ohos ohos_model_benchmark run test.
- **Primary Language**: Unknown
- **License**: Apache-2.0
- **Default Branch**: master
- **Homepage**: None
- **GVP Project**: No

## Statistics

- **Stars**: 1
- **Forks**: 0
- **Created**: 2026-05-28
- **Last Updated**: 2026-07-02

## Categories & Tags

**Categories**: Uncategorized

**Tags**: None

## README

# AI Model Benchmark for HarmonyOS

A cross-framework AI model benchmarking tool that compares inference performance and output accuracy across six backends — HIAI (CANN NPU), MindSpore Lite, MNN, ONNXRuntime, NCNN, and TFLite — on HarmonyOS devices.

## Features

- Six inference backends: **HIAI (CANN NPU)**, **MindSpore Lite**, **MNN**, **ONNXRuntime (ORT)**, **NCNN**, **TFLite**
- CPU / NPU device switching (HIAI: NPU only, MSLite: CPU or NPU, all others: CPU only)
- Multi-run benchmarking (1~100 iterations) with min / avg / max latency statistics per stage
- **On-device accuracy comparison**: compute cosine similarity, RMSE, MAE, and max absolute error against ONNXRuntime reference output during inference, with automatic quality labels (excellent / good / acceptable / poor)
- **Output Dump**: export inference output data to device files for offline analysis
- **Reference data input**: load input reference data from rawfile by model input tensor name, falling back to constant 0.5 fill
- **NCNN dual-file support**: automatically loads `.ncnn.param` + `.ncnn.bin` pair and concatenates into a unified buffer
- Async execution with progress animation, non-blocking UI
- **AutoRun batch testing**: one-tap benchmark of all models in `models/` (backend auto-detected by extension; HIAI on NPU, MSLite on both CPU and NPU, others on CPU), exporting performance data to an Excel report (via libxlsxwriter)
- Python accuracy comparison tools: 2-way and 3-way comparison based on ONNXRuntime reference output, generating KDE / Scatter / Violin / Q-Q / Bland-Altman plots

## Architecture Overview

```
┌──────────────────── ArkTS UI Layer ─────────────────────┐
│  Index.ets  →  Config (Backend / Device / Model / Runs) │
│               Toggles (Dump / AccuracyCMP)               │
│  →  RunFullBenchmarkAsync() via libentry.so NAPI        │
└──────────────────────────┬──────────────────────────────┘
                           │ NAPI bridge
┌──────────────────────────▼──────────────────────────────┐
│                  C++ Native Layer                        │
│  napi_init.cpp                                           │
│    ├── RunFullBenchmarkAsync (async, multi-run aggregate)│
│    │     Params: rawfile path, ResourceManager, backend, │
│    │           enableNpu, enableDump, enableAccuracyCmp, │
│    │           runCount, callback                        │
│    └──────────→ ModelFactory::Create(config)             │
│    └──────────→ BenchmarkTool::ComputeAccuracyMetrics() │
│    └──────────→ GetInputNames → load reference data     │
└──────────────────────────┬──────────────────────────────┘
                           │ Factory pattern
    ┌──────────┬──────────┼──────────┬──────────┬──────────┐
    ▼          ▼          ▼          ▼          ▼          ▼
┌────────┐ ┌────────┐ ┌────────┐ ┌────────┐ ┌────────┐ ┌────────┐
│  HIAI  │ │ MSLite │ │  MNN   │ │  ORT   │ │  NCNN  │ │TFLite  │
│ (.om)  │ │.ms/.mi │ │ (.mnn) │ │(.onnx) │ │.ncnn.* │ │.tflite │
│  NPU   │ │CPU/NNRT│ │  CPU   │ │  CPU   │ │  CPU   │ │  CPU   │
│CANNKit │ │MSLite  │ │MNN Lib │ │ORT Lib │ │NCNN Lib│ │TF Lite │
└────────┘ └────────┘ └────────┘ └────────┘ └────────┘ └────────┘
        All implement ModelManagerBase interface:
        LoadModelFromBuffer / LoadModelFromFile → FillInputData
        → RunModel → GetOutputData → UnloadModel / DumpModelOutput
        + GetInputCount / GetInputElementCounts / GetInputNames / GetOutputNames
```

> AutoRun mode reuses the `ExecuteBenchmark` flow above via `RunAutoBenchmarkAsync` to batch-run all models in `models/` and export an Excel report. See [AutoRun Batch Testing & Excel Export](#autorun-batch-testing--excel-export) below.

## Supported Model Formats

| Backend  | Model Format                     | Device            |
|----------|----------------------------------|--------------------|
| HIAI     | `.om`                            | NPU                |
| MSLite   | `.ms` / `.mindir`                | CPU / NPU (NNRT)   |
| MNN      | `.mnn`                           | CPU                |
| ORT      | `.onnx`                          | CPU                |
| NCNN     | `.ncnn.param` + `.ncnn.bin`      | CPU                |
| TFLite   | `.tflite`                        | CPU                |

> **NCNN note**: The NCNN backend requires two files (`.ncnn.param` + `.ncnn.bin`). The NAPI layer automatically concatenates them: an 8-byte header storing the param file size, followed by param data, then bin data, passed as a unified buffer to `LoadModelFromBuffer`.

## Benchmark Stages

Each iteration executes the following sequence with individual timing:

| Stage         | Description                                    |
|---------------|------------------------------------------------|
| LoadModel     | Load model data from rawfile into memory        |
| InitTensors   | Fill input data (reference data or constant 0.5)|
| RunModel      | Execute one inference pass                      |
| GetResult     | Retrieve output data                            |
| Unload        | Release model resources                         |
| **Sum**       | Total of all stages above                       |

## Accuracy Comparison

### On-Device Accuracy Comparison

When the **AccuracyCMP** toggle is enabled, the app performs the following on the last inference iteration:

1. Load ONNXRuntime reference output from the `reference_data/` directory in rawfile
2. Call `BenchmarkTool::ComputeAccuracyMetrics()` to compute:
   - **Cosine Similarity**
   - **RMSE** (Root Mean Square Error)
   - **MAE** (Mean Absolute Error)
   - **Max Absolute Error**
3. Automatically assign a quality label: `excellent` (>0.99), `good` (>0.95), `acceptable` (>0.90), `poor` (≤0.90)
4. The accuracy panel displays color-coded quality labels and metrics for each output tensor

### Input Data Source

Input data files are named by model input tensor name, placed in the `reference_data/` directory in rawfile (shared with output reference data, distinguished by tensor name). Naming rules:

- Path format: `reference_data/<sanitized_tensor_name>.txt`
- `sanitized_tensor_name` = original tensor name with all non-alphanumeric characters replaced by `_`

Examples:
- Tensor named `x` → `reference_data/x.txt`
- Tensor named `input_images:0` → `reference_data/input_images_0.txt`
- Tensor named `data` → `reference_data/data.txt`

During inference, the app automatically searches for matching files by tensor name:
- If a matching file is found, its data is loaded as the input tensor fill values
- If not found, constant 0.5 fill is used

### Reference Output Search

Reference output files are searched based on the model filename, with automatic suffix stripping (`_quant`, `_fp16`, `_int8`, `_qat`, `_sym`) for matching.

Search order:
1. Primary: `reference_data/<model_basename>_<sanitized_output_name>.txt`
2. Fallback: `reference_data/<sanitized_output_name>.txt`

## AutoRun Batch Testing & Excel Export

**AutoRun** benchmarks every model in `rawfile/models/` in one tap — no need to manually pick a backend and model each time.

### Workflow

1. Tap **AutoRun All Models**
2. The native layer scans `models/` and detects the backend by extension (`.om→HIAI`, `.ms/.mindir→MSLite`, `.mnn→MNN`, `.onnx→ORT`, `.ncnn.param→NCNN`, `.tflite→TFLite`; `.ncnn.bin` is loaded with its `.param`)
3. Each model reuses the single-model benchmark flow (`ExecuteBenchmark`), runs `runCount` iterations, and collects per-stage min/avg/max plus accuracy
4. Device: HIAI runs on NPU only, MSLite runs on both CPU and NPU (one result row per device), all others on CPU only; **Dump / AccuracyCMP follow the current UI toggles**
5. When done, `tools/excel_writer` (libxlsxwriter) writes the results to an Excel report

### Excel Report

- Output path: `<app sandbox filesDir>/benchmark_<yyyyMMdd_HHmmss>.xlsx` (the full path is shown in the result banner)
- One row per model:

| Column | Description |
|--------|-------------|
| Backend / Model / RunCount / Device | backend, model filename, iterations, device (NPU/CPU) |
| LoadModel / InitTensors / RunModel / GetResult / Unload / Sum min/avg/max (ms) | per-stage latency stats |
| Status / Error | OK / FAIL and failure reason |
| AvgCosine / WorstQuality / AccuracyDetails | accuracy summary (filled only when AccuracyCMP is on) |
| InputShapes / OutputShapes | per input/output tensor shape (e.g. `[1,3,224,224]`, multiple tensors `;`-separated) |

> libxlsxwriter assembles xlsx in a temp dir by default; since `/tmp` may not be writable in the OHOS sandbox, `tools/excel_writer.cpp` sets `tmpdir` to the output directory via `workbook_new_opt`.
>
> **Build note**: `libxlsxwriter.a` is a static lib — to link into `entry.so` (a shared lib) it must be compiled for arm64-v8a with `-fPIC`; `libz.so.1` is its dynamic dependency, provided by `libs/arm64-v8a/zlib/` and packaged into the hap.

## Configuration (ModelConfig)

| Field               | Type   | Default                      | Description             |
|---------------------|--------|------------------------------|-------------------------|
| `backend_type`      | enum   | HIAI                         | Inference backend type  |
| `enable_npu`        | bool   | false                        | Enable NPU              |
| `enable_dump_output`| bool   | false                        | Export inference output  |
| `enable_accuracy_cmp`| bool  | false                        | Enable accuracy comparison |
| `enable_fp16`       | bool   | false                        | Enable FP16 inference   |
| `thread_num`        | int32  | 1                            | Inference thread count  |
| `enable_npu_cache`  | bool   | false                        | Enable NPU model cache  |
| `cache_dir`         | string | `/data/storage/el2/base/cache/` | Cache directory path |
| `cache_version`     | string | `"1"`                        | Cache version number    |
| `cache_model_tag`   | string | `"model_tag"`                | Cache model tag         |
| `band_mode`         | string | `"normal"`                   | Execution mode          |
| `execute_device`    | string | `"npu"`                      | Execution device        |
| `mnn_forward_type`  | int32  | 0                            | MNN forward compute type|

## Requirements

- **DevEco Studio** (HarmonyOS IDE)
- **HarmonyOS SDK**: API 6.0.0(20)
- **BiSheng compiler**: for C++ native layer compilation
- **Target device**: HarmonyOS phone/tablet (arm64-v8a)
- **Python 3.x** (only for accuracy comparison scripts)

## Build & Run

### 1. Build the Project

1. Open the project in **DevEco Studio**
2. The Hvigor build system will automatically:
   - Compile ArkTS UI code
   - Build C++ native layer via CMake + BiSheng compiler (C++17 standard, `c++_static` STL)
   - Link MNN, MindSpore Lite NDK, CANN/HiAI, ONNXRuntime, NCNN, OpenMP, TFLite, libxlsxwriter, zlib prebuilt libraries
   - Package rawfile model resources and reference data
   - Sign with the debug certificate from `build-profile.json5`
3. Build output is a `.hap` (HarmonyOS Ability Package)

> **TFLite build note**: CMakeLists.txt requires `--allow-multiple-definition` (resolves duplicate symbols in TFLite static libs) and `--start-group/--end-group` (resolves circular dependencies among TFLite static libs).

### 2. Run a Benchmark

1. Deploy the `.hap` to a HarmonyOS device or emulator
2. The app auto-scans the `models/` directory in `rawfile`, filtering models by the selected backend's file extension
3. Select: **Backend** → **Device** → **Model file** → **Run count** (1~100)
4. Optionally enable: **Dump** (export output) or **AccuracyCMP** (accuracy comparison)
5. Tap **Run Benchmark** — the async NAPI call runs multi-iteration inference on a background thread
6. The results panel shows min / avg / max latency (ms) for each stage
7. If AccuracyCMP is enabled, the accuracy panel shows per-tensor cosine similarity, RMSE, MAE, max absolute error, and quality labels
8. Alternatively tap **AutoRun All Models** to benchmark every model in `models/` in one go (Dump / AccuracyCMP follow the current toggles; HIAI on NPU, MSLite on both CPU and NPU, others on CPU)
9. When AutoRun finishes, performance data is exported to an Excel report `<app sandbox filesDir>/benchmark_<timestamp>.xlsx`; the UI shows the path and a per-model per-device `Sum` average-latency summary

### 3. Accuracy Comparison (Python Scripts)

```bash
# Generate ONNXRuntime reference output (load input data by tensor name)
python python/scripts/benchmark_file_input.py --model model.onnx --input_dir ./input_data --output_dir ./output_data

# Generate random float32 input data (customizable shape and output filename)
python python/scripts/gen_input.py

# 2-way accuracy comparison (validation directory)
python python/valid/torch_ohos_accuracy_cmp.py

# 3-way accuracy comparison (e.g. NPU vs NPU-quantized vs ONNXRuntime)
python python/scripts/torch_ohos_accuracy_cmp3.py
```

The Python accuracy scripts output cosine similarity matrices, RMSE, MAE, max absolute error, and generate KDE / Scatter / Violin / Q-Q / Bland-Altman plots.

## Project Structure

```
ohos_model_benchmark/
├── entry/
│   ├── src/main/
│   │   ├── ets/
│   │   │   ├── pages/Index.ets            # Main UI (config / run / results / accuracy)
│   │   │   ├── entryability/EntryAbility.ets  # App lifecycle
│   │   │   ├── entrybackupability/EntryBackupAbility.ets  # Data backup extension
│   │   │   └── utils/FileUtils.ets        # File I/O utilities
│   │   ├── cpp/
│   │   │   ├── napi_init.cpp              # NAPI bridge (async API + NCNN dual-file concat)
│   │   │   ├── engine/
│   │   │   │   ├── model_manager_base.h   # Abstract interface + ModelConfig + BackendType
│   │   │   │   ├── model_factory.h        # Factory pattern for backend creation
│   │   │   │   └── backend/
│   │   │   │       ├── cann/              # HIAI/CANN NPU backend
│   │   │   │       ├── mslite/            # MindSpore Lite backend
│   │   │   │       ├── mnn/               # MNN backend
│   │   │   │       ├── ort/               # ONNXRuntime backend
│   │   │   │       ├── ncnn/              # NCNN backend
│   │   │   │       └── tflite/            # TFLite backend (+ tflite_compat.cpp)
│   │   │   ├── tools/
│   │   │   │   ├── benchmark_tool.h       # AccuracyResult + accuracy computation / data loading
│   │   │   │   ├── benchmark_tool.cpp     # ComputeAccuracyMetrics / LoadDataFromBuffer impl
│   │   │   │   ├── excel_writer.h         # Excel report data structures + writer API
│   │   │   │   └── excel_writer.cpp       # writes perf/accuracy report via libxlsxwriter
│   │   │   ├── third_party/               # Third-party headers (mnn/ort/ncnn/openmp/tensorflow/flatbuffers/libxlswriter)
│   │   │   └── CMakeLists.txt             # C++17 + c++_static + TFLite linking special handling
│   │   ├── resources/rawfile/
│   │   │   ├── models/                    # All backend models in one dir, distinguished by extension
│   │   │   │   ├── *.om                   # HIAI models
│   │   │   │   ├── *.ms                   # MSLite models
│   │   │   │   ├── *.mnn                  # MNN models
│   │   │   │   ├── *.onnx                 # ORT models
│   │   │   │   ├── *.ncnn.param + *.ncnn.bin # NCNN models (dual files)
│   │   │   │   └── *.tflite               # TFLite models
│   │   │   └── reference_data/<tensor_name>.txt  # Input/output reference data (by tensor name)
│   │   └── module.json5
│   ├── libs/arm64-v8a/
│   │   ├── mnn/                           # MNN prebuilt libs (libMNN.so, libMNN_Express.so)
│   │   ├── ort/                           # ONNXRuntime prebuilt libs (libonnxruntime.so.1)
│   │   ├── ncnn/                          # NCNN prebuilt libs (libncnnd.so.1)
│   │   ├── openmp/                        # OpenMP prebuilt libs (libomp.so)
│   │   ├── tflite/                        # TFLite static libs (~110 .a files)
│   │   ├── xlsxwriter/                    # libxlsxwriter static lib (libxlsxwriter.a)
│   │   └── zlib/                          # zlib dynamic lib (libz.so.1, libxlsxwriter dependency)
│   ├── oh-package.json5
│   └── build-profile.json5
├── python/
│   ├── scripts/
│   │   ├── benchmark_file_input.py        # ONNXRuntime reference output (load input by tensor name)
│   │   ├── gen_input.py                   # Generate random float32 input data
│   │   └── torch_ohos_accuracy_cmp3.py    # 3-way accuracy comparison
│   └── valid/
│       ├── torch_ohos_accuracy_cmp.py     # 2-way accuracy comparison
│       └── feature.txt                    # Validation reference data
├── build-profile.json5
├── hvigorfile.ts
└── hvigor/
```

## Dependencies

### Native C++ Libraries

| Library                     | Purpose                              |
|----------------------------|--------------------------------------|
| `libace_napi.z.so`         | C++-ArkTS bridge (NAPI)              |
| `libhilog_ndk.z.so`        | HarmonyOS logging                    |
| `mindspore_lite_ndk.so`    | MindSpore Lite inference SDK         |
| `librawfile.z.so`          | Read rawfile resources               |
| `hiai_foundation`          | HiAI / CANN foundation library       |
| `libneural_network_core.so`| Neural Network Runtime core          |
| `libMNN.so` + `libMNN_Express.so` | MNN inference libs (prebuilt) |
| `libonnxruntime.so.1`      | ONNXRuntime inference (prebuilt)     |
| `libncnnd.so.1`            | NCNN inference (prebuilt)            |
| `libomp.so`                | OpenMP threading (prebuilt)          |
| `libtensorflow-lite.a` + deps | TFLite inference (static-linked, prebuilt) |
| `libxlsxwriter.a`      | Excel report generation (static-linked, prebuilt) |
| `libz.so.1`            | zlib compression lib, libxlsxwriter dependency (prebuilt) |

### ArkTS Dependencies

| Package                    | Purpose                        |
|----------------------------|--------------------------------|
| `@kit.AbilityKit`          | UIAbility / ResourceManager    |
| `@kit.PerformanceAnalysisKit` | Hilog                      |
| `@kit.ArkUI`               | Window management              |
| `@kit.CoreFileKit`         | File I/O / Backup extension    |

## License

Apache License 2.0