Co²Skill: Whole-Body Control via Skill Composition for Long-Horizon Human-Environment Interaction

Jeonghwan Kim · Hyeonwoo Kim · Hanbyul Joo
Seoul National University

TL;DR One policy for a physics-simulated humanoid that composes whole-body scene interaction and dexterous manipulation, running skills sequentially or simultaneously.

Paper ↗ Method · Individual skills · Skill composition · Citation

The character picks up a wineglass and keeps it in hand through a sequence of scene interactions.

Everyday tasks mix moving through a scene with handling objects, like picking up a glass and then sitting down with it. Physics-based controllers usually learn these two separately. Co²Skill learns both in one policy.

Method

Sparse target goals

Every task is given as a sparse target goal, a few points marking where to interact, such as the top of the stairs or a seat. One policy reads the character, the nearby scene, and this goal, and outputs joint commands that run in physics simulation.

Goal conditions: two red target points, one at the foot and one at the top of a staircase, with the character's pose ghosted.
Goal conditions
Unified policyπ(a | s, g)
Simulated output

Stages 1–2: Motion prior

A whole-body tracking policy is trained on motion capture, then distilled into a motion prior and an action decoder.

Stage 1 trains a whole-body tracking policy from reference trajectories. Stage 2 distills it into a reference encoder, a character state prior, and an action decoder.

Stage 3: Multi-task adaptation

With the prior and decoder frozen, a shared task encoder reads the task observations and predicts a correction in the prior’s latent space.

Stage 3: the shared task encoder reads state and task tokens, with inactive task tokens masked, and adds a latent correction to the frozen prior before the action decoder. Task observations for reach, climb, sit, and manipulation mark each goal with a sparse target.

Task-token masking

Masked task tokens turn on only the inputs the current task and phase need.

Reference-guided exploration

References guide close-range contact. Varied starts teach reaching the goal from new positions.

Reset-free chaining

Lift → sit, climb, or reach is trained without resets, so the grasp carries over.

Individual skills

Scene interaction

At test time the policy gets only a goal, such as a seat or the top of the stairs. No reference trajectory.

Sit on a chair

Climb the stairs

Manipulation

Given the object’s target pose and grasp conditions, the policy runs pick → move → place.

Cup

Water bottle

Mug

Camera

Skill composition

Sequential execution

Scene goals are issued one after another, without resetting the character.

Chair → platform → stairs

Simultaneous interaction

After the lift, a scene goal starts while the object stays in hand.

Carry upstairs

Hold and sit

Hold and climb

Citation

@misc{kim2026co2skill,
  title={Co${}^{2}$Skill: Whole-Body Control via Skill Composition for Long-Horizon Human-Environment Interaction},
  author={Jeonghwan Kim and Hyeonwoo Kim and Hanbyul Joo},
  year={2026},
  eprint={2610.09291},
  archivePrefix={arXiv},
  primaryClass={cs.RO},
  url={https://arxiv.org/abs/2610.09291},
}