1University of Trento2LIGM, Ecole des Ponts, IP Paris, Univ Gustave Eiffel, CNRS
Abstract
Recent advances in 3D human motion and language integration have primarily focused on text-to-motion generation, leaving the task of motion understanding relatively unexplored.
We introduce Dense Motion Captioning, a novel task that aims to temporally localize and caption actions within 3D human motion sequences.
Current datasets fall short in providing detailed temporal annotations and predominantly consist of short sequences featuring few actions.
To overcome these limitations, we present the Complex Motion dataset (CompMo), the first large-scale dataset featuring richly annotated, complex motion sequences with precise temporal boundaries.
Built through a carefully designed data generation pipeline, CompMo includes 60,000 motion sequences, each composed of multiple actions ranging from at least two to ten, accurately annotated with their temporal extents.
We further present DEMO, a model that integrates a large language model with a simple motion adapter, trained to generate dense, temporally grounded captions.
Our experiments show that DEMO substantially outperforms existing methods on CompMo as well as on adapted benchmarks, establishing a robust baseline for future research in 3D motion understanding and captioning.
Gallery of CompMo dataset
In this gallery, we show some samples from our proposed CompMo dataset.
Different colors in the motion correspond to different actions.
The bottom displayed is the dense captions of the data in the format of 'start - description'.
00:00:00 - walks forward with a big right kick
00:06:55 - pretends to be a t-rex
00:16:55 - swing his left leg behind him
00:23:85 - lofts their left hand to their face
00:27:75 - walks forward at mid pace
00:00:00 - walks down steps with their left hand on a railing
00:04:70 - jump forward on the floor
00:09:25 - punches forward with his left arm
00:11:95 - takes off their t-shirt with one arm
00:15:54 - looking at the damage to his left leg
00:25:85 - takes a left handed golf swing
00:29:65 - 3 jumping jacks in place
00:37:00 - lifts both hands into the air
00:00:00 - looks at their right hand
00:08:61 - walks in a clockwise semi-circle
00:12:26 - waves their left hand over their head in a friendly gesture
00:19:81 - walks backwards swiftly
00:29:03 - hops on their left foot then their right
00:33:34 - walking wobbly as if drunk or dizzy
00:38:64 - waves with the left hand
00:46:84 - lays on the ground and does a swimming motion with their arms and legs
00:55:49 - jumping rope
00:65:44 - putting on a garment
00:00:00 - kicks at something with their left leg
00:02:55 - kicks and punches in a manner consistant with martial arts
00:11:35 - sat down in a curtsy position
00:17:45 - walking backwards, towards the left
00:23:30 - bends down, the places their left hand on their stomach
00:33:30 - walking forward with a low kick to the left
00:38:25 - stretching his right leg
00:42:55 - sit ups
00:00:00 - stands up from being on one knee
00:08:95 - walked forward and a medium pace
00:12:89 - scratching their head in confusion
00:17:50 - walks towards the right side
00:22:60 - jumps several times
00:26:00 - hops on his left leg
00:28:45 - walking in a random pattern
00:00:00 - executes a handstand with his feet up in the air and returns to his feet in a standing position
00:06:60 - walks and stumbles to his right
00:09:39 - kicks an object with their right foot
00:13:34 - ran in a diagonal line from one side to the other
00:15:24 - shrugs his arms
00:17:95 - walking like a zombie in a half circle to the right
00:00:00 - walks down several stairs
00:05:65 - walking sideways to the left, then diagonally to right and back to center
00:15:45 - jumping on one leg first on the left leg then next on the right leg
00:00:00 - walks in a tight line (on a beam or string)
00:08:40 - walking forward like a seesaw
00:15:35 - taking something off the shelf
00:19:75 - walking in an unbalanced and wobbly way
00:21:70 - bowing over
00:30:25 - bends his left leg in many directions above the ground
00:40:35 - kicks an object on the ground first with his left foot and then his right
00:44:65 - pushes an object into the ground with his left hand
00:49:76 - walks forwards casually
00:00:00 - jumps over something and lands on feet
00:03:35 - doing multiple cart wheels
00:12:25 - picks up an object on the left side with both hands
00:21:10 - walk slowly while in a crouching position
00:29:35 - takes four steps forward, down a ramp, toward the viewer
00:34:75 - hits their leg once
00:44:75 - walks in a counterclockwise circle and stops
00:00:00 - reaches with their right arm 5 different times in 5 different locations
00:10:00 - stretches first their left arm across their body, then their right
00:14:70 - stirring with their right arm
00:00:00 - standing straight with their arms out front
00:04:00 - walked forward in left and right direction and after return
00:13:90 - doing push ups on the ground
00:23:65 - jumps up with his whole body
00:27:25 - walks forward in a clockwise semi circle
00:31:00 - push recovery stand
00:34:70 - jogs for a second and stops
00:40:40 - stretches arms over head
00:44:55 - raises their right hand to their mouth
00:48:25 - wiping the floor
00:00:00 - opening a door
00:04:00 - moving in a strange circle motion away from body
00:14:00 - dancing with himself
00:24:80 - walks forward and slightly to the right
00:30:48 - stumbles backwards with left leg first
00:34:28 - pushing arms in and out
00:44:28 - walks casually in a coutercounterclockwise circle
00:51:43 - walking in an erratic pattern
00:61:33 - losing footing and does a slight bounce to the right
00:63:98 - walks on a treadmill at a slow pace
00:00:00 - takes a slight bow forward
00:04:80 - kicks with left foot
00:09:00 - making a simple crouch down
00:17:25 - wipes down a window with their right hand
00:26:46 - walked forward in left and after right direction
00:36:46 - standing to the right side
00:46:28 - spins around in place
00:00:00 - squats multiple times
00:08:80 - playing the guitar
00:16:25 - puts their hand up to their mouth as if yawning
00:22:35 - took one step to the left
00:27:40 - stretches arms up together above head
00:31:90 - crawling to the right side then the left side
00:41:05 - stands up to a prone position
00:45:85 - swings on tree branches like a monkey
00:55:85 - walking backwards in a circle
00:00:00 - walks at a right bend
00:03:19 - put his hands on his knees
00:11:24 - acting like a cat
00:21:89 - kicks the air with his right foot
00:26:04 - stepped over the clean spots
00:30:49 - slides to the left
00:39:50 - walks forward in a straight line carefully
00:47:88 - throws and catches something
00:00:00 - stepping to the left and then back to the right
00:07:15 - shoveling in yard
00:16:25 - moves the right leg in a small circular motion
00:23:10 - walks at a moderate pace forward and to the left
00:28:65 - walking very fast around in a circle
00:37:80 - swinging a tennis racket
00:41:95 - doing a dance like a ballerina
00:51:74 - walking in a random pattern then back
Main Results
Here we show DEMO's results on our proposed CompMo dataset.
The bottom displays GT dense captions and DEMO generated captions in sequence.
GT:
00:00:00 - shuffles 2 steps to the left, 3 steps to the right and 1 step back to center,
00:05:75 - adjusts something on the ground,
00:13:01 - picks something up off the grond with his right hand,
00:17:56 - walks backwards, his hands down by his side,
00:23:34 - jumps in place four times,
00:26:70 - takes one large step to their right,
00:31:20 - walks to his left then to his right and then to his lefts again,
00:43:20 - walks quickly five steps forward,
00:48:60 - swimming the free style stroke,
00:60:20 - walks briskly in counterclockwise circles.
DEMO:
00:00:00 - side-steps quickly, like a basketball shuffle -- first to the right and then to the left,
00:05:60 - adjusts something on the ground,
00:12:71 - picks an object up off the floor with their left hand,
00:17:93 - walks backwards after a second or two,
00:23:33 - takes a large step diagonally with their left leg,
00:27:88 - takes a couple steps to the left then a couple steps to the right and back to the left again,
00:37:48 - steps back two steps and lowers to a crouch position,
00:46:58 - swimming on his back slowly,
00:56:98 - walks in a counterclockwise circle and returns to his original position.
GT:
00:00:00 - walks at a normal pace in a clockwise circle,
00:06:25 - approaches and squats,
00:09:55 - raises both arms into the air above their head,
00:11:15 - moves an object from the right to the left,
00:21:15 - reaches forwards with their right hand then does a rowing motion with both hands,
00:30:50 - squats up and down two times,
00:38:40 - extends their left foot forward,
00:39:10 - climbing up a pole,
00:49:15 - looks around behind left and right,
00:55:95 - sneaking sideways.
DEMO:
00:00:00 - walks in a circle,
00:06:50 - squats with their arms above their head,
00:08:50 - raises both arms into the air above their head,
00:10:75 - moves an object from the right to the left,
00:20:75 - shooting a free throw,
00:26:55 - squats with their arms out in front of them,
00:31:65 - squats up and down with their hands out in front of them,
00:42:90 - bending down multiple times,
00:53:35 - limping with his right leg.
GT:
00:00:00 - walks straightforward, one foot in front of the other, like they are on a tight rope,
00:09:35 - flaps his arms like a bird,
00:19:04 - moves something with both hands diagnoly,
00:24:54 - runs forward at moderate pace,
00:28:70 - looks at their right arm,
00:32:20 - holds his head with both hands and kicks with his right leg,
00:41:75 - holding on to a stool to get down,
00:46:50 - doing karate kicks,
00:56:30 - took a step to the left.
DEMO:
00:00:00 - walks with their arms straight out and legs stiff, in a u shape,
00:10:00 - puts both arms forward then to his sides10.0,
00:20:40 - checks their watch on their left hand,
00:23:95 - jogs forward at an average pace,
00:26:20 - checks the watch on their left arm,
00:30:20 - holds his head with both hands and kicks with his right leg,
00:38:65 - gets down onto their knees,
00:44:65 - kicks with their right leg 4 times while their hands are in front of their face,
00:53:50 - kicks something with their right foot,
00:57:80 - takes a large step to their left.
GT:
00:00:00 - squats very slightly,
00:01:40 - acting like a gorilla,
00:12:55 - throwing something with his right hand,
00:16:80 - thinking in a crouched over position,
00:21:35 - balances on the right leg,
00:26:85 - moves his left leg up,
00:35:55 - running forward at a fast pace,
00:36:94 - walks in a circle towards their right,
00:46:30 - walks forward at speedy pace,
00:48:69 - doing karate kicks.
DEMO:
00:00:00 - shakes shoulders fast,
00:03:84 - acting like a human dinosaur,
00:12:69 - throws something with his right arm,
00:17:04 - crouched with knees bent and arms bent,
00:22:18 - kicks their left foot,
00:32:94 - runs forward and then to the left,
00:36:29 - walks in a circle in a clockwise direction,
00:45:69 - kicks with their right leg 4 times while their hands are in front of their face.
GT:
00:00:00 - walks forward then right arm and shoulder stumble to the right hand side two times,
00:05:39 - walks in a curved path forward and to the right,
00:09:48 - rests their hands on their legs.
DEMO:
00:00:00 - walks forward then right arm and shoulder stumble to the right hand side two times,
00:05:39 - walks forward and to the right in a semicircle,
00:09:84 - sitting down with their hands on their thighs.
GT:
00:00:00 - squats multiple times,
00:08:80 - playing the guitar,
00:16:25 - puts their hand up to their mouth as if yawning,
00:22:35 - took one step to the left,
00:27:40 - stretches arms up together above head,
00:31:90 - crawling to the right side then the left side,
00:41:05 - stands up to a prone position,
00:45:85 - swings on tree branches like a monkey,
00:55:85 - walking backwards in a circle.
DEMO:
00:00:00 - squats up and down two times,
00:08:55 - strums a guitar with his left arm,
00:14:60 - moves their left arm in a circular motion in front of their stomach area,
00:18:59 - takes a large step to their left over an object,
00:23:34 - stretches their arms out in circles,
00:33:34 - crawls around on the ground,
00:42:59 - throws an object over-hand,
00:51:89 - throws something large with both arms,
00:61:89 - walking backwards in a strait line.
GT:
00:00:00 - walks in a tight line (on a beam or string),
00:08:40 - walking forward like a seesaw # 8.40 # 15.35,
00:15:35 - taking something off the shelf # 15.35 # 19.75,
00:19:75 - walking in an unbalanced and wobbly way,
00:21:70 - bowing over,
00:30:25 - bends his left leg in many directions above the ground,
00:40:35 - kicks an object on the ground first with his left foot and then his right,
00:44:65 - pushes an object into the ground with his left hand,
00:49:76 - walks forwards casually.
DEMO:
00:00:00 - walks straightforward, one foot in front of the other, like they are on a tight rope,
00:08:40 - walks forward and stops in the center,
00:16:50 - puts their hands on something in the order left-left-top-bottom,
00:23:39 - leans to his right,
00:28:59 - balancing on their legs with arms out,
00:42:49 - kicks something with their left foot,
00:47:56 - puts a golf ball on a tee,
00:55:33 - walks forward with his right hand on a railing,
00:59:75 - takes a diagonal step to the left.
GT:
00:00:00 - hops around on their right leg,
00:10:00 - lifts his right leg.
DEMO:
00:00:00 - hops on one leg in a circle,
00:10:00 - kicks their left leg forward one time.
GT:
00:00:00: - jumps over something and lands on feet,
00:03:35 - doing multiple cart wheels,
00:12:25 - picks up an object on the left side with both hands,
00:21:10 - walk slowly while in a crouching position,
00:29:35 - takes four steps forward, down a ramp, toward the viewer,
00:34:75 - hits their leg once,
00:44:75 - walks in a counterclockwise circle and stops.
DEMO:
00:00:00 - jumps forward with his arms at his sides,
00:03:15 - doing a cartwheel forward then back again,
00:11:75 - moves object from right to left,
00:17:95 - creeps forward with their knees and arms slightly bent,
00:25:95 - walks forward with his arms at his side slowly,
00:33:90 - sits down on something and the stands up again,
00:41:65 - raised hands while bend on the knee,
00:43:65 - walks in a circle in a counterclockwise direction,
00:51:50 - steps sideways to their left.
GT:
00:00:00 - staggers around slowly in an clockwise movement,
00:09:10 - walks in a circle with their right hand up to their face,
00:18:54 - wipes their hands on their upper legs,
00:27:83 - runs and stops,
00:30:98 - kicking their left leg into the air,
00:42:78 - falls to his right,
00:48:12 - stretches arms up together above head,
00:51:58 - moving both hands in an up and down motion,
00:60:48 - moves back and right with the left foot first.
DEMO:
00:00:00: - walks in a circle in an anticlockwise direction and stops at original position,
00:08:75 - walks in a circle with their left hand up to their face,
00:17:30 - stands still while squatted and t-poses at the end,
00:24:75 - runs forward and stops,
00:29:55 - moving the right leg and like dancing,
00:41:20 - falls to his left,
00:46:20 - lifts both his arms up and stretchs,
00:50:20 - waves their hands together in front of them.
GT:
00:00:00 - takes a slight bow forward,
00:04:80 - kicks with left foot,
00:09:00 - making a simple crouch down,
00:17:25 - wipes down a window with their right hand,
00:26:46 - walked forward in left and after right direction,
00:36:46 - standing to the right side,
00:46:28 - spins around in place.
DEMO:
00:00:00 - takes a bow quickly,
00:04:90 - kicks something with their right foot,
00:09:30 - making a simple crouch down,
00:15:80 - doing a range of motion exercise from their wrist to their shoulders on both their right & left sides,
00:25:50 - walks in a bend shape to the left,
00:30:55 - walks forward and to the left in an arc shape,
00:35:45 - gets in the swimming pool and does the breast stroke,
00:44:30 - lifts their arms and spins slowly,
00:49:90 - throws something with his left hand.
GT:
00:00:00 - walks forward in a side to side pattern,
00:09:26 - takes a couple steps to the left then back again to the right and repeats,
00:19:81 - punches with their right arm,
00:22:36 - steps to the side with their right foot.
DEMO:
00:00:00 - walks in a s shaped pattern back and forth across the square area,
00:08:55 - stepping to the left and then back to the right,
00:17:30 - punches with their left hand then their right hand,
00:21:30 - takes a step forward and to his left.
GT:
00:00:00 - walks at a right bend,
00:03:19 - put his hands on his knees,
00:11:24 - acting like a cat,
00:21:89 - kicks the air with his right foot,
00:26:04 - stepped over the clean spots,
00:30:49 - slides to the left,
00:39:50 - walks forward in a straight line carefully,
00:47:88 - throws and catches something.
DEMO:
00:00:00 - walks in a quarter circle to their left with the right leg stiffer than the other,
00:04:33 - put his hands on his knees,
00:13:53 - acting like a human dinosaur,
00:21:08 - kicks something with their right leg,
00:23:88 - takes a big step forward and to the right,
00:27:88 - takes a step forward and three steps more to his left,
00:33:28 - takes a step backwards, right leg first,
00:38:39 - walks forward in a straight line carefully,
00:46:71 - catches a ball.
GT:
00:00:00 - stepping to the left and then back to the right,
00:07:15 - shoveling in yard,
00:16:25 - moves the right leg in a small circular motion,
00:23:10 - walks at a moderate pace forward and to the left,
00:28:65 - walking very fast around in a circle,
00:37:80 - swinging a tennis racket,
00:41:95 - doing a dance like a ballerina,
00:51:74 - walking in a random pattern then back.
DEMO:
00:00:00 - shuffles to the left, then to the right, then back to the left,
00:07:40 - digging with a shovel,
00:16:55 - kicks their right leg out,
00:19:30 - kicks their right leg forward one time,
00:22:80 - walks in a bend shape to the left,
00:27:55 - walks in a circle in an anticlockwise manner,
00:35:20 - playing tennis with his right arm,
00:41:40 - doing ballet leaps,
00:46:01 - jumping jacks then two forward leg jumps,
00:55:41 - walks in a bend shape to the left.
GT:
00:00:00 - standing completely stale,
00:01:20 - doing some kind of left arm with,
00:09:34 - walking backwards in a straight line,
00:13:99 - walking in a square on uneven ground,
00:23:99 - jogs forward 2 steps,
00:29:99 - walks sideways and hops,
00:33:69 - claps their hands in front of them,
00:37:69 - walks forward at a brisk pace.
DEMO:
00:00:00 - moving around both arms,
00:10:00 - walks backwards with his hands down at his sides,
00:15:55 - walks forward and upward,
00:23:30 - takes a step forward and three steps more to his left,
00:28:55 - jumps up and down 2 times,
00:33:05 - claps his hands 5 times,
00:36:65 - walks forward and to the left in an arc.
GT:
00:00:00 - bouncing to do boxing,
00:08:35 - took four steps forward.
DEMO:
00:00:00 - boxing with his left hand and then does 3 upper cut punches,
00:08:55 - walks forward with a slight arc to the left.
GT:
00:00:00 - opening a door,
00:04:00 - moving in a strange circle motion away from body,
00:14:00 - dancing with himself,
00:24:80 - walks forward and slightly to the right,
00:30:48 - stumbles backwards with left leg first,
00:34:28 - pushing arms in and out,
00:44:28 - walks casually in a coutercounterclockwise circle,
00:51:43 - walking in an erratic pattern,
00:61:33 - losing footing and does a slight bounce to the right,
00:63:98 - walks on a treadmill at a slow pace.
DEMO:
00:00:00 - raises their right arm in a salute,
00:04:60 - waving with both of their arms,
00:14:60 - dances sporadically around the space,
00:22:65 - walks forward with his hands stiff at his sides,
00:28:25 - takes a quick step backwards and to their left,
00:32:50 - pushing arms in and out,
00:42:50 - walks in a circle in an anticlockwise manner,
00:48:90 - walks in a s shaped pattern back and forth across the square area,
00:59:50 - walks forward and stumbles to his right,
00:61:50 - walks in a circle with exaggerated hip and arm movements, either drunk or flirty.
Comparison Result with UniMotion
Here we show comparison results with UniMotion on CompMo and H3D+BABEL dataset.
The bottom displays GT dense captions, DEMO, and UniMotion generated captions in sequence.
CompMo
GT:
00:00:00 - walks at a normal pace in a clockwise circle,
00:06:25 - approaches and squats,
00:09:55 - raises both arms into the air above their head,
00:11:15 - moves an object from the right to the left,
00:21:15 - reaches forwards with their right hand then does a rowing motion with both hands,
00:30:50 - squats up and down two times,
00:38:40 - extends their left foot forward,
00:39:10 - climbing up a pole,
00:49:15 - looks around behind left and right,
00:55:95 - sneaking sideways.
DEMO:
00:00:00 - walks in a circle,
00:06:50 - squats with their arms above their head,
00:08:50 - raises both arms into the air above their head,
00:10:75 - moves an object from the right to the left,
00:20:75 - shooting a free throw,
00:26:55 - squats with their arms out in front of them,
00:31:65 - squats up and down with their hands out in front of them,
00:42:90 - bending down multiple times,
00:53:35 - limping with his right leg.
UniMotion
00:00:00 - moving in a right angle triangular path,
00:03:80 - turn on flashlight.
GT:
00:00:00 - hops around on their right leg,
00:10:00 - lifts his right leg.
DEMO:
00:00:00 - hops on one leg in a circle,
00:10:00 - kicks their left leg forward one time.
UniMotion
00:00:00 - moving in a right angle triangular path,
00:02:15 - smooth out object on the wall,
00:02:25 - moving in a right angle triangular path,
00:02:30 - smooth out object on the wall,
00:02:35 - moving in a right angle triangular path,
00:10:05 - smooth out object on the wall.
H3D+BABEL
GT:
00:00:000 - stand,
00:01:398 - walk forward,
00:03:898 - walk up stairs,
00:07:273 - stand.
GT:
00:00:00 - shuffles 2 steps to the left, 3 steps to the right and 1 step back to center,
00:05:75 - adjusts something on the ground,
00:13:01 - picks something up off the grond with his right hand,
00:17:56 - walks backwards, his hands down by his side,
00:23:34 - jumps in place four times,
00:26:70 - takes one large step to their right,
00:31:20 - walks to his left then to his right and then to his lefts again,
00:43:20 - walks quickly five steps forward,
00:48:60 - swimming the free style stroke,
00:60:20 - walks briskly in counterclockwise circles.
DEMO:
00:00:00 - side-steps quickly, like a basketball shuffle -- first to the right and then to the left,
00:05:60 - adjusts something on the ground,
00:12:71 - picks an object up off the floor with their left hand,
00:17:93 - walks backwards after a second or two,
00:23:33 - takes a large step diagonally with their left leg,
00:27:88 - takes a couple steps to the left then a couple steps to the right and back to the left again,
00:37:48 - steps back two steps and lowers to a crouch position,
00:46:58 - swimming on his back slowly,
00:56:98 - walks in a counterclockwise circle and returns to his original position.
UniMotion
00:00:00 - moving in a right angle triangular path,
00:58:55 - turn on flashlight,
00:58:70 - moving in a right angle triangular path,
00:60:75 - turn on flashlight.
GT:
00:00:00 - executes a handstand with his feet up in the air and returns to his feet in a standing position
00:06:60 - walks and stumbles to his right
00:09:39 - kicks an object with their right foot
00:13:34 - ran in a diagonal line from one side to the other
00:15:24 - shrugs his arms
00:17:95 - walking like a zombie in a half circle to the right
DEMO:
00:00:00 - steps forward and does a handstand,
00:07:35 - walks forward at a fast pace,
00:11:95 - runs forward and does a summersalt and lands perfectly,
00:15:45 - walks around like a zombie,
00:25:45 - walks with his left hand in the air.
UniMotion
00:00:00 - smooth out object on the wall.
