Learning Human-centric Motion Representation for Action Analysis

Michigan State University

Abstract

We introduce H-MoRe, an innovative pipeline designed to learn precise, human-centered motion representations.

Our approach dynamically retains essential human motion features while filtering out background noise. Unlike traditional methods that rely on fully supervised learning with synthetic data, H-MoRe employs a self-supervised learning paradigm directly from real-world scenarios, incorporating both human pose and body shape information.

Drawing inspiration from kinematics, H-MoRe encodes absolute and relative movements of body points into a matrix representation, termed world-local flows, to capture subtle motion details. This method provides a detailed understanding of human motion, making it highly adaptable to various action-based applications.

Dense Human-centric Flow

H-MoRe represents motion as human-centric world-local flows, locking onto the moving body where generic optical flow is distracted by the background.

RGB input video

H-MoRe (Ours) human-centric flow

SEA-RAFT optical flow

Approach

Motivation: humans decompose motion to analyze it effortlessly — Motivation. Where a machine is unsure, humans recognize motion effortlessly by leveraging prior knowledge — decomposing it into world motion (a system relative to its surroundings) and local motion (a part relative to the system). Computers need the same trick.

From human intuition to a joint constraint learning framework — From human intuition to computation. H-MoRe transfers this decomposition trick into a Joint Constraint Learning Framework, predicting world-local flows from two frames under boundary ("same shape") and skeleton ("same color") constraints — so a computer can analyze human motion the way we analyze simple objects.

Single-inference architecture via vector decomposition — Architecture. Rather than inferring world and local flow with two separate models, H-MoRe runs a single world-motion estimate (Φ) and a lightweight estimate of the body's centroid motion (Ψ). Local flow then follows directly by vector decomposition — world minus centroid motion — with no second flow network.

Self-supervised physical-prior constraints — Self-supervised objectives. With no flow labels, H-MoRe is anchored by physical priors — angular and intensity constraints align flow with the skeleton ("same color"), while boundary and centroid constraints match the flow edge to the human silhouette ("same shape").

Qualitative Results

Against State-of-the-Art Flow

Flow visualizations generated by our H-MoRe and seven optical flow estimation algorithms.

Frame-by-Frame

Drag the slider to inspect the H-MoRe inference result for each frame.

9 / 46

BibTeX

@inproceedings{huang2025hmore,
  title     = {Learning Human-centric Motion Representation for Action Analysis},
  author    = {Huang, Zhanbo and Liu, Xiaoming and Kong, Yu},
  booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  year      = {2025},
}