Under Review

H-Flow: Self-supervised Human Scene Flow via Physics-inspired Joint Multi-modal Learning

Zhanbo Huang 1 Xiaoming Liu 1,2 Yu Kong 1
1 Michigan State University 2 University of North Carolina, Chapel Hill
arXiv

Abstract

Parametric human models capture global pose but cannot represent the non-rigid surface dynamics of clothing and soft tissue. Generic scene flow estimates dense motion but breaks down on articulated bodies, where pixel-level supervision is also intractable to acquire. We introduce H-Flow, a dense human scene flow that captures both skeletal kinematics and surface deformation.

A unified multi-head transformer estimates flow from monocular video, jointly predicting pose and depth as companion outputs. The challenge lies in the lack of supervision. In place of unattainable labels, we anchor the network in the physics of human motion, encoding geometric, structural, and biomechanical priors as cross-modal training objectives.

We further introduce DynAct4D, a high-fidelity synthetic benchmark providing dense flow annotations across diverse subjects, garments, and motions. On standard benchmarks, H-Flow outperforms scene-flow and parametric baselines, and generalizes zero-shot to in-the-wild video. Code, models, and the DynAct4D benchmark will be released upon publication.

Dense Human Scene Flow

From a single monocular video, H-Flow recovers dense per-point human motion that generic scene flow misses.

Input videoInput video
Input video monocular
H-Flow (Ours) dense human scene flow
ZeroMSF generic scene flow

Method

H-Flow architecture
H-Flow architecture. A unified multi-head transformer estimates dense scene flow from monocular video, jointly predicting pose and depth as companion outputs.
Physics-inspired self-supervision
Physics-inspired self-supervision. With no dense flow labels available, H-Flow is anchored by geometric, structural, and biomechanical priors, encoded as cross-modal training objectives.

DynAct4D Benchmark

A high-fidelity synthetic benchmark providing dense scene-flow ground truth across diverse subjects, garments, and motions.

DynAct4D benchmark samples
DynAct4D spans varied identities, clothing, and actions with per-point flow annotations, enabling supervised evaluation where real-world dense labels are unattainable.

Qualitative Results

Orbit views of the recovered 3D scene flow. H-Flow reconstructs coherent human motion close to ground truth, where generic scene flow distorts the articulated body.

Input videoInput video
Input video
Ground Truth
H-Flow (Ours)
ZeroMSF

BibTeX 

@article{huang2026hflow,
  title   = {H-Flow: Self-supervised Human Scene Flow via Physics-inspired Joint Multi-modal Learning},
  author  = {Huang, Zhanbo and Liu, Xiaoming and Kong, Yu},
  journal = {arXiv preprint arXiv:2605.22629},
  year    = {2026},
}