Overview
After an earthquake, phone networks and internet often go down, and cloud-based AI stops working exactly when it is needed. This project is a payload that works without any connection: it mounts on a ROS 2 robot, searches for victims, and keeps working when the network does not.
It combines thermal and visual detection, a radar sensor, gas and flame sensing, a voice assistant that speaks with the victim, and a long-range radio link for alerts. Everything runs on a small NVIDIA Jetson TX2 computer carried by the robot.
The problem
In the 2023 Kahramanmaraş earthquakes, cellular and internet networks failed within minutes. Most AI-assisted rescue tools depend on the cloud, so they stop working in exactly that situation. They also tend to rely on a single camera, and they rarely talk to the trapped person.
The idea
A self-contained payload that bolts onto a ROS 2 robot as an external “AI brain”. It finds people, talks to them, watches for hazards and reports back, while running entirely on the robot.
What it does
- Finds victims by combining a thermal camera, a visual detector and a radar sensor.
- Talks to them through an offline voice assistant that gives calm, supportive guidance until help arrives.
- Watches for danger with gas, flame and temperature sensors, and keeps the robot away from hazards.
- Reports back over Wi-Fi for maps and video, and over a long-range LoRa link when Wi-Fi drops.
Seeing victims
A low-power thermal camera watches continuously and only wakes the heavier visual detector when it sees body heat. This saves energy and cuts false alarms. The detector was trained on images we collected ourselves in the lab, showing people lying on the floor, partly hidden or in dim light, because public datasets mostly show people standing.
Sensing through barriers
A 24 GHz radar sensor can pick up the tiny chest movement of breathing, even when the person is hidden. It was tested in the lab with a person lying still and breathing normally, at 8 set distances with 10 repeats each, so 80 trials per case.
- Without a barrier: chest motion was detected in 54 of 80 trials (67.5%), with an average distance error of about 0.16 m and a useful range of about 6 m.
- Behind a 3 cm wooden barrier: 39 of 80 trials (48.75%), an average distance error of about 0.18 m and a range of about 5 m.
Because of these limits, the radar is used only as a confirmation sensor after a visual or audio trigger, never as a stand-alone search tool, and the figures describe the lab setup, not rubble or reinforced concrete.
Talking to victims
Once a victim is confirmed, the robot switches to voice mode. Speech is converted to text, a small language model prepares a short supportive reply, and a text-to-speech voice speaks it. All three steps run locally on the robot: whisper.cpp with a small English model turns speech into text in about 2.3 seconds, Gemma 3 1B writes the reply in about 3.4 seconds, and Piper speaks it in about 0.6 seconds. The robot therefore starts answering about 6.3 seconds after the victim stops speaking. The two heavy models cannot fit in memory together, so a small state machine switches between them quickly.
Staying safe
A separate microcontroller reads gas, flame and temperature sensors on a fixed schedule, so sensing is never blocked by the AI workload. When a limit is crossed, a hazard marker appears on the map and the robot plans a route around the area.
Running on very little power
Measured directly on the Jetson TX2, the whole platform stayed under 12 W at its peak. It draws about 2 W while idle or while only the sensors and radio are active, and rises to about 7 W when the visual detector is running.
Staying connected
Wi-Fi carries maps and video when coverage exists. A small LoRa radio carries compact status messages (robot position, victim status and hazards) when Wi-Fi is lost. Alerts and hazards show up on a command-centre screen.
Two different robots
To show that the payload is not tied to one robot, it was mounted on two different bases: a TurtleBot3 Waffle Pi and a TurtleBot2 Kobuki, both working through the same ROS 2 interface.
From simulation to the real robot
Navigation and mapping were first tried in a Gazebo simulation with collapsed obstacles, narrow corridors and simulated victims, then moved to the physical robots. The robot builds a map as it moves, heads towards a confirmed victim, and avoids zones flagged as dangerous.
Honest limits
These are lab results on indoor research robots. The detector was trained on simulated disaster scenes, the radar was only tested in the lab and works as a confirmation sensor, and the voice assistant needs about six seconds to start replying. Testing in real rubble and on rugged robot bodies is the next step.
Credits
Built at the RAI Laboratory, Mechatronics Engineering Department, Fırat University, with Munef El Muhammed, Anas Mahyoub Naji Saeed Alqadhi and Aysegul Ucar.
