| System View | Third-person View | |
|---|---|---|
| Object Navigation |
[.webm](https://github.com/user-attachments/assets/e7b6eaff-9951-4a0b-8438-2bbaeb1c27d5) | [.webm](https://github.com/user-attachments/assets/5d636dd9-3074-420a-926c-3152df65b4c9) |
| Find the vacuum_cleaner. | ||
| Self-attribute Condition |
[.webm](https://github.com/user-attachments/assets/e92a65df-7934-46ac-9c9b-d1994ae09fa8) | [.webm](https://github.com/user-attachments/assets/9b8a0077-a2b4-4253-a656-d8973bbca930) |
| Find the chair with one person sitting on it. | ||
| Spatial Condition |
[.webm](https://github.com/user-attachments/assets/cfbca2dd-7d0a-49d1-90b3-92a1efcd05f9) | [.webm](https://github.com/user-attachments/assets/6a4f7a93-35a9-44b6-88a1-d638d7d8ca2f) |
| Find the person sitting on the sofa/bench. | ||
More demos on our project page.
## Platform This branch targets the **Unitree G1** humanoid. For other platforms, switch to the corresponding branch: - [`main`](https://github.com/zwandering/VLM_ROS) — wheeled robot + Unity simulation - [`unitree_go2`](https://github.com/zwandering/VLM_ROS/tree/unitree_go2) — Unitree Go2 quadruped ### Hardware - Unitree G1 humanoid, controlled via WebRTC - Livox Mid-360 lidar + Ricoh Theta Z1 360-degree camera - Asus NUC 14 Pro (Intel Core Ultra 5) as the onboard computer - Desktop workstation / Laptop with NVIDIA RTX 4090 for the semantic mapping and VLM reasoning - Wired / WiFi network shared between robot, NUC, and desktop Sensor calibration is handled by the [`360_camera`](https://github.com/jizhang-cmu/360_camera/tree/jazzy) repository (see [360 Camera Driver](#360-camera-driver)). ## Contents - [Demo](#demo) - [Platform](#platform) - [Hardware](#hardware) - [Installation](#installation) - [Dependencies](#1-dependencies) - [Submodules and Python Packages](#2-submodules-and-python-packages) - [SLAM Dependencies](#3-slam-dependencies) - [Mid-360 Lidar Driver](#4-mid-360-lidar-driver) - [Compile](#5-compile) - [Real-robot Setup](#real-robot-setup) - [System Setup](#system-setup) - [360 Camera Driver](#360-camera-driver) - [System Usage](#system-usage) - [Credits](#credits) - [Citation](#citation) - [License](#license) ## Installation The system has been tested on **Ubuntu 24.04** with **ROS2 Jazzy**. ### 1) Dependencies Install [ROS2 Jazzy](https://docs.ros.org/en/jazzy/Installation.html), then: ```bash echo "source /opt/ros/jazzy/setup.bash" >> ~/.bashrc source ~/.bashrc ``` Install system dependencies: ```bash sudo apt update sudo apt install ros-jazzy-desktop-full ros-jazzy-pcl-ros libpcl-dev git sudo apt install -y nlohmann-json3-dev sudo apt install ros-jazzy-backward-ros sudo apt install -y python3-pip portaudio19-dev ``` ### 2) Submodules and Python Packages ```bash git submodule update --init --recursive pip install -r requirement.txt --break-system-package # Unitree WebRTC pip install unitree_webrtc_connect --break-system-package # detectron2 python -m pip install 'git+https://github.com/facebookresearch/detectron2.git' --break-system-package # pytorch3d pip install "git+https://github.com/facebookresearch/pytorch3d.git" --no-build-isolation --break-system-package # sam2 cd src/semantic_mapping/semantic_mapping/external/sam2 pip install -e . --break-system-package cd checkpoints && ./download_ckpts.sh && cd ../.. # spacy python -m spacy download en_core_web_sm --break-system-package # CLIP pip install git+https://github.com/ultralytics/CLIP.git --break-system-package # YOLO models python set_yolo_e.py python set_yolo_world.py ``` ### 3) SLAM Dependencies Install **Sophus** (from `src/slam/dependency/Sophus`): ```bash mkdir build && cd build cmake .. -DBUILD_TESTS=OFF make && sudo make install ``` Install **Ceres Solver** (from `src/slam/dependency/ceres-solver`): ```bash mkdir build && cd build cmake .. make -j6 && sudo make install ``` Install **GTSAM** (from `src/slam/dependency/gtsam`): ```bash mkdir build && cd build cmake .. -DGTSAM_USE_SYSTEM_EIGEN=ON -DGTSAM_BUILD_WITH_MARCH_NATIVE=OFF make -j6 && sudo make install sudo /sbin/ldconfig -v ``` ### 4) Mid-360 Lidar Driver Install **Livox-SDK2** (from `src/utilities/livox_ros_driver2/Livox-SDK2`): ```bash mkdir build && cd build cmake .. make && sudo make install ``` Configure the lidar IP in `src/utilities/livox_ros_driver2/config/MID360_config.json` — set the IP to `192.168.1.1xx` where `xx` are the last two digits of the lidar serial number. Compile the driver: ```bash colcon build --symlink-install --cmake-args -DCMAKE_BUILD_TYPE=Release --packages-select livox_ros_driver2 ``` ### 5) Compile **For simulation** (skips SLAM and lidar driver): ```bash colcon build --symlink-install --cmake-args -DCMAKE_BUILD_TYPE=Release --packages-skip arise_slam_mid360 arise_slam_mid360_msgs livox_ros_driver2 ``` **For real robot** (full build, requires steps 3-4): ```bash colcon build --symlink-install --cmake-args -DCMAKE_BUILD_TYPE=Release ``` ### VLM API Key The VLM node supports two providers via the OpenAI-compatible interface. Set **one** of the following: **Gemini** (default) — get a key from [Google AI Studio](https://aistudio.google.com/app/api-keys): ```bash export GEMINI_API_KEY="your-api-key-here" ``` **Qwen (DashScope)** — get a key from [Alibaba Cloud DashScope](https://dashscope.console.aliyun.com/): ```bash export DASHSCOPE_API_KEY="your-api-key-here" ``` If both keys are set, Gemini is used by default; override with `export VLM_PROVIDER=qwen`. Optionally override Qwen model names with `QWEN_MODEL` / `QWEN_MODEL_LITE`. Add the line(s) to `~/.bashrc` so they persist across terminal sessions. ## Real-robot Setup ### System Setup Install [Ubuntu 24.04](https://releases.ubuntu.com/noble) and [ROS2 Jazzy](https://docs.ros.org/en/jazzy/Installation.html) on both the onboard NUC and the desktop workstation: ```bash echo "source /opt/ros/jazzy/setup.bash" >> ~/.bashrc source ~/.bashrc ``` Follow the [Installation](#installation) section on both machines to install all dependencies and compile the full repository. Configure the WebRTC connection to the G1 in `src/unitree_webrtc_ros/config/unitree_params.yaml`. By default the system uses `LocalAP` mode at `192.168.12.1` (connecting to the robot's own WiFi access point). Other supported modes are `LocalSTA` (robot on the same LAN — set `robot_ip` accordingly) and `Remote` (via Unitree cloud). ### 360 Camera Driver The system uses a Ricoh Theta Z1 360-degree camera. The camera driver and lidar-to-camera calibration tools are maintained in a separate repository — clone it alongside this repo and follow its README to build and configure: [https://github.com/jizhang-cmu/360_camera/tree/jazzy](https://github.com/jizhang-cmu/360_camera/tree/jazzy) ### System Usage The system runs across two machines: **On the onboard computer (NUC):** ```bash # Terminal 1 — 360 camera driver (in the 360_camera repo) ./360_camera_sensorpod.sh # Terminal 2 — domain bridge source install/setup.bash ros2 launch domain_bridge domain_bridge.launch # Terminal 3 — navigation system ./system_real_robot_with_exploration_planner_g1.sh ``` **On the desktop computer (4090):** ```bash # Terminal 1 — republish compressed camera images export ROS_DOMAIN_ID=1 ros2 run image_transport republish \ --ros-args \ -p in_transport:=compressed \ -p out_transport:=raw \ --remap in/compressed:=/camera/image/compressed \ --remap out:=/camera/image # Terminal 2 — exploration planner ./system_real_robot_with_exploration_planner_4090.sh ```
Exploration