Human Reconstruction and Motion Capture using a Single Flying Camera


Fig. 1 Realtime 3D human motion capture by a flying camera.

Abstract

With the emergence of consumer RGB-D camera, interdisciplinary research in computer vision, graphics and robotics experienced huge growth in recent years. Aiming at the intelligent human body 3D reconstruction and motion capture, we adopt the aerial robot that employed with RGB-D camera as the flying camera, and present two novel works in this webpage: iHuman3D and FlyFusion for automated, adaptive, and real-time human body 3D reconstruction and motion capture.

Specifically, for static human full body 3D reconstruction, a real-time and active view planning system iHuman3D is proposed based on a highly efficient ray casting algorithm in GPU and a novel information gain formulation directly in Truncated Signed Distance Function (TSDF). Human body reconstruction module revises the traditional volumetric fusion pipeline with a compactly-designed non-rigid deformation for slight motion of the human target. Both the active view planning and human body reconstruction are unified in the same TSDF volume-based representation. On the other hand, for dynamic human motion capture, following the active reconstruction clue, a Geometry And Motion Energy (GAME) metric for guiding the viewpoint optimization in the volumetric space, proposed FlyFusion succeeds to enable active viewpoint selection based on the immediate dynamic reconstruction geometry and predicted human motion.

Quantitative and qualitative experiments are conducted to validate that the proposed systems effectively remove the constraints of fixed capture volume and extra manual labor, enabling real-time and intelligent human body 3D reconstruction and motion capture. Given above distinctiveness, we believe our work bridges the gap in robotics and human modelling, and will further promote both robotics, computer vision and graphics communities in interactive 3D human reconstruction.


iHuman3D: Intelligent Human Body 3D Reconstruction using a Single Flying Camera

Project Page

Code (Simulation)

Code (Reconstruction) coming soon!

Video

Paper

FlyFusion: Realtime Dynamic Scene Reconstruction Using a Flying Depth Camera

Project Page

Code (Simulation)

Code (Motion Capture) coming soon!

Video

Paper

Fly Camera Design Manual (Coming Soon!)