1 Motion Capture CSE 3541 Matt Boggus

2 MoCap Technologies Instrument the talent to facilitate tracking feature points on the human figure Need some kind of sense-able markers from which positional and possibly rotational status can be recorded

3 “Traditional” MoCap TechnologiesOptical – uses video capture passive – markers just reflect light active – markers emit light Magnetic – active sensors sense their position and orientation in magnetic field Electro-Mechanical – rotors connected to limb-aligned rods record their status – for hands, optical sensors used sometimes

4 Optical - Passive

5 Newer MoCap TechnologiesInertial systems (similar to Wii technology) Make-up (fluorescent powder) Semi-passive imperceptible markers (emit light and observe changes ; Kinect uses infrared) Markerless systems (video tracking)

6 Markerless Organic Motion www.organicmotion.com/Image Metrics (facial)

7 Kinect – skeleton tracking, depth, audioUsing the Kinect in Unity

8 Passive Optical

9 Motion capture lab Multiple markers - e.g., 20-40Multiple cameras - e.g., 8-14 high-res, high-speed Constrained, conditioned space - 20x20 non-reflective Multiple lights - synced w/ cameras Vicon

10 Motion capture lab

11 Processing the Images PROCESSING STEPS Extract markers from videoTrack markers over time in video Marker cleanup 3D marker position reconstruction Joint position reconstruction Joint angle reconstruction

12 Extract markers from videoBasic image processing aided by constrained environment: High contrast markers Special illumination Non-reflective environment

13 Practice problem - algorithm to extract markers from imageImage source: https://www.engadget.com/2014/07/14/motion-capture-explainer/

14 Marker tracking Given frames each with recognized markersAssociate markers over multiple frames Temporal coherence using: Position Frame rate Velocity

15 Marker Clean-up

16

17

18 Camera calibration Project known 3-space points to camera’s imageSix degrees of freedom - use that many known point-pairs

19 Projecting from 2D image out to 3-space

20 3D – image plane projectionProjecting marker onto image plane

21 Reconstructing a 3D marker

22 Working with Mocap Sample files https://sites.google.com/a/cgspeed.com/cgspeed/motion-capture Tools MotionBuilder Maya 3DSMax

23 Example BVH file See ACCAD files

24 Constructing a hierarchical character for a BVH fileExample on board

25 Manipulating the mocap dataNext steps: Manipulating the mocap data Frequency deconstruction & manipulation Transitioning between two motions – Blending How to map a motion onto a figure with different geometry – Retargeting Finding motion clips to create behavior – motion graphs

26 Additional and Cut slides

27 Mocap hardware tech

28 Optical - Active

30 Ex: Metamotion Electro-Mechanical

31 Reconstructing the data

32 Fitting to the skeletonLocate joints relative to markers Markers Joint

33 Reconstructing angles