# DA3-RK3588

**Repository Path**: gitesin/DA3-RK3588

## Basic Information

- **Project Name**: DA3-RK3588
- **Description**: No description available
- **Primary Language**: Unknown
- **License**: Not specified
- **Default Branch**: master
- **Homepage**: None
- **GVP Project**: No

## Statistics

- **Stars**: 0
- **Forks**: 0
- **Created**: 2026-06-03
- **Last Updated**: 2026-06-03

## Categories & Tags

**Categories**: Uncategorized

**Tags**: None

## README

# DA3 RK3588 Visual Odometry

This repository is evolving into a complete DA3-based visual odometry pipeline for RK3588:

- DA3 model conversion (`ONNX -> RKNN`)
- RKNN runtime inference on RK3588 NPU
- VO processor (`RGB pair -> depth + pose + tracked trajectory`)
- gRPC streaming for camera input and remote debug visualization

## Current status

- Implemented: RKNN conversion tools and runtime smoke test (`RKNN/`)
- Implemented: VO processing pipeline with optical-flow keyframe logic (`RKNN_DA3_VO/`)
- Implemented: gRPC frame streaming + DA3 split debug stream (`grpc/`)
- In progress: stabilization, calibration integration, and accuracy/performance tuning

https://github.com/user-attachments/assets/4ce615e0-1da2-40b0-8e37-dc3017fb2004


## Repository layout

```text
.
├── RKNN/                    # ONNX patching + RKNN conversion + runtime inference test
├── RKNN_DA3_VO/             # Main VO pipeline
├── grpc/                    # gRPC proto, server/client, web client
├── scripts/                 # Helper scripts for run/config
├── assets/                  # Demo images
├── data/                    # Models, calibration, datasets (local working dir)
└── README.md
```

## Quick start

### 1) Install dependencies

```bash
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
```

Base Python dependencies are listed in `requirements.txt`.
RKNN toolkit packages are installed separately (see `RKNN/README.md`).

### 2) (Optional) Configure USB camera mode on RK3588

```bash
./scripts/configure_v4l2_camera_mode.sh \
  --device /dev/video20 \
  --width 1280 \
  --height 720 \
  --fps 30 \
  --pixfmt MJPG
```

### 3) Run DA3 VO processor

```bash
python3 RKNN_DA3_VO/main_processor.py \
  --rknn data/da3_small_2_392x224_fp32.rknn \
  --camera 20 \
  --core-mask auto \
  --optical-threshold mediate \
  --log-level INFO \
  --debug-grpc-host 0.0.0.0 \
  --debug-grpc-port 50061
```

Notes:

- `--debug-grpc-port 0` disables debug stream.
- RKNN filename should contain input size, e.g. `..._504x280...rknn`, so preprocessing can parse expected model resolution.

### 4) View debug stream remotely

view with web:

```bash
./scripts/run_web_client.sh 192.168.19.214:50061 127.0.0.1:8081
```

view with python:

```bash
python3 grpc/grpc_video_client.py \
  --host <rk3588-ip> \
  --port 50061 \
  --stream da3_split \
  --fps 15 \
  --show \
  --window-name "DA3 VO Debug Stream"
```

`da3_split` carries two logical streams:

- `StreamDa3Rgb`: RGB JPEG + timestamp
- `StreamDa3DepthPose`: depth raw f32 + timestamp + pose

The client renders a side-by-side RGB/depth view and overlays pose/timestamps.

## RKNN workflow

Use `RKNN/README.md` for full conversion details. Typical flow:

1. Prepare/optimize ONNX model.
2. Convert to RKNN (`fp32` or `int8` quantized).
3. Validate runtime on RK3588 with `RKNN/da3_rknn_runtime.py`.

Example runtime test:

```bash
python3 RKNN/da3_rknn_runtime.py \
  --rknn data/da3_small_2_504x280.rknn \
  --image0 assets/00301.jpg \
  --image1 assets/00321.jpg \
  --loops 20 \
  --core-mask auto \
  --save-dir rknn_outputs
```

## gRPC camera streamer (standalone)

This is useful independent of VO for camera bring-up and transport checks.

Start streamer:

```bash
python3 grpc/grpc_video_streamer.py --camera 1 --port 50051
```

Or via wrapper:

```bash
./scripts/run_grpc_streamer.sh --camera 1 --port 50051
```

Subscribe:

```bash
python3 grpc/grpc_video_client.py --host 127.0.0.1 --port 50051 --stream video --show
```

Web client docs:

- `grpc/web_client/README.md`

## Key CLI options

### VO processor (`RKNN_DA3_VO/main_processor.py`)

```text
--rknn                  Path to DA3 .rknn model (required)
--camera                USB camera index (default: 0)
--width/--height        Capture resolution hint (default: 1920x1080)
--fps                   Capture FPS hint (default: 30)
--core-mask             auto|0|1|2|0_1_2
--optical-threshold     tight|mediate|medium|moderate|loose
--debug-grpc-port       Debug gRPC port, 0 disables stream
--log-level             DEBUG|INFO|WARNING|ERROR
```

### gRPC streamer (`grpc/grpc_video_streamer.py`)

```text
--camera                USB camera index
--width/--height        Capture resolution
--fps                   Capture FPS
--pixfmt                FOURCC (MJPG/YUYV/...)
--jpeg-quality          JPEG quality [1-100]
--host/--port           Bind address
```

### gRPC client (`grpc/grpc_video_client.py`)

```text
--stream                auto|video|da3_split
--show                  OpenCV live view
--save-every            Save every Nth frame
--max-frames            Stop after N frames
```

## Development roadmap

- Integrate camera intrinsics/extrinsics calibration loading into VO path.
- Add trajectory logging/export (`TUM`, `KITTI` format) and evaluation scripts.
- Add deterministic replay mode from recorded image pairs/video.
- Add performance profiling and latency/throughput dashboards for RK3588.
- Add CI checks for proto generation, lint, and basic pipeline smoke tests.

## Related docs

- `RKNN/README.md`: model conversion and runtime
- `RKNN_DA3_VO/README.md`: VO module usage
- `grpc/web_client/README.md`: browser client for gRPC stream