Mohammed Bajhaw
← Home

Published · Journal Article

PV-STAM: Velocity-Aware Attention for Mapless Deep Reinforcement Learning Navigation in Dynamic Environments

A compact attention module that lets a robot with only a 2D LiDAR tell approaching obstacles from static ones, trained with deep reinforcement learning and validated in 130 physical TurtleBot3 trials.

Applied Sciences (MDPI), vol. 16, no. 18, article 90832026DOI: 10.3390/app16189083

Abstract

Mapless deep reinforcement learning (DRL) navigation in dynamic indoor environments is difficult under single-frame 2D LiDAR, which reports where an obstacle is but not whether it is approaching. We introduce the Positional-Velocity Spatio-Temporal Attention Module (PV-STAM), a compact perception block (19,968 trainable parameters, under 3% of network capacity) that combines a per-sector scan-difference channel with a two-head self-attention mechanism and a 384→48 compression bottleneck. Policies are trained with Soft Actor-Critic across a seven-phase progressive curriculum with bidirectional demotion, scaling from static goal-seeking to fifteen simultaneously moving obstacles at 0.18 m/s, with a Hardware-Calibrated Training Mode applied in the final phase. Seven configurations were evaluated on three zero-shot benchmark arenas over 300 episodes each across three random seeds, and key control variants were validated in 130 valid physical trials on a TurtleBot3 Waffle Pi across two matched corridor scenarios. The simulation and physical evaluations give different orderings, and this dissociation is the paper’s principal result. In simulation, SAC-PV-STAM, SAC-R-PV-STAM and SAC-MLP-FS lie within 4.6 percentage points of one another on two of three benchmarks and are not separable; on hardware, they separate with large margins. SAC-R-PV-STAM reached the goal in 19 of 20 static trials and 20 of 20 trials with a moving obstacle, with no threshold violations across 40 trials, against SAC-MLP-FS (stratified p = 0.00068) and SAC-PV-STAM (stratified p = 1.1 × 10−7). A non-recurrent variant stalled with a clear floor ahead—SAC-PV-STAM commanded no forward velocity above 0.005 m/s in any of the 2746 samples of the final 91.3 s of a 122.3 s stall while the LiDAR reported 3.50 m directly ahead—and a prediction stated before the experiment, that introducing a moving obstacle would remove the stall, was confirmed (timeouts 9/10 to 0/20, p = 7 × 10−7). We further report a scan-difference contamination analysis showing up to 12.0× more spurious spikes during self-rotation on hardware, a width-matched comparison separating the effect of Huber critic loss from that of critic width, and a recurrent LSTM baseline that remains significantly inferior across all three benchmarks and unstable across seeds.

In short

A robot with a single 2D LiDAR scan can see where obstacles are, but not whether they are coming towards it. PV-STAM is a small add-on to the robot's neural network that compares consecutive scans sector by sector and uses attention to pick out approaching obstacles. The robot learns to navigate with deep reinforcement learning, first in simulation and then on a real TurtleBot3, without any map.

Main contributions

  • PV-STAM module: a compact attention block that adds only 19,968 parameters, under 3% of the network.
  • Seven-phase curriculum: training that grows from static goal-seeking to fifteen moving obstacles.
  • Simulation versus reality: the leading variants look alike in simulation but separate clearly on the real robot.
  • A stall failure mode: a non-recurrent policy can freeze with clear floor ahead, and a prediction that a moving obstacle would remove the stall was confirmed.
  • Critic analysis: a width-matched comparison showing the Huber-loss effect cannot be separated from critic width.

How the system works

PV-STAM system pipeline from LiDAR preprocessing to simulation and real-world validation
System pipeline: LiDAR preprocessing, the PV-STAM attention embedding with the SAC policy, and validation in Gazebo and on real hardware.

Real-robot evaluation

Key variants were tested in 130 valid physical trials on a TurtleBot3 Waffle Pi in a 3.3 x 7.6 m corridor, with static cardboard obstacles and with a moving obstacle.

TurtleBot3 Waffle Pi hardware platform with component annotations
The TurtleBot3 platform: LDS-01 2D LiDAR, NVIDIA Jetson AGX Orin, OpenCR 1.0 controller and power bank.
The physical evaluation corridor and its cardboard obstacle configuration
The physical evaluation corridor with the TurtleBot3 (left) and the cardboard obstacle layout (right).

Key results

Physical trial outcomes by variant for the static corridor and moving obstacle scenarios
Physical trial outcomes by variant: goal reached, timeout or collision.
  • Best variant: SAC-R-PV-STAM reached the goal in 19 of 20 static trials and 20 of 20 moving-obstacle trials, with no threshold violations across 40 trials.
  • Simulation versus hardware: in simulation three variants lie within 4.6 percentage points of each other on two of three benchmarks; on hardware they separate by large margins.
  • Stall: SAC-PV-STAM stalled with 3.50 m of clear floor ahead, and adding a moving obstacle removed the stall (timeouts fell from 9 of 10 to 0 of 20).
  • Scan contamination: naive scan differencing showed up to 12.0 times more spurious spikes while the robot rotated on hardware.

Read the full paper

The complete method, benchmarks, tables and references are in the paper. Use the Open PDF button on this page to read it. The article is open access under a CC BY 4.0 licence.

Keywords

Mapless NavigationDeep Reinforcement LearningSpatio-Temporal AttentionLiDARDynamic Obstacle AvoidanceSim-to-Real Transfer