Skill-Adaptive Ghost Instructors

Abstract

Ghost-hand guidance is widely used in VR motor-skill learning, but static transparency schemes often lead to over-reliance — learners follow the ghost passively rather than internalizing the skill. We present Skill-Adaptive Ghost Instructors, a VR piano learning system that dynamically adjusts ghost-hand transparency based on the learner's real-time performance.

Through a formal user study on Meta Quest 3, we compared a static baseline condition against our skill-adaptive system across measures of note-retention, over-reliance, task load (NASA-TLX), and perceived usefulness. Results show that adaptive guidance significantly reduces over-reliance while maintaining or improving post-practice retention — suggesting that fading assistance tied to performance milestones supports more effective skill internalization.

Key Contributions

Designed a skill-adaptive transparency control loop that evaluates per-note accuracy in real time and adjusts ghost opacity accordingly.
Implemented full hand-tracking and MIDI-event logging on Meta Quest 3 in Unity / C#.
Conducted a within-subjects user study (n=24) with pre/post retention tests, NASA-TLX, and over-reliance metrics.
Demonstrated that adaptive guidance reduces passive mirroring behaviour without increasing cognitive workload.

System Design

The system consists of three layers: (1) a hand-tracking module capturing finger joint positions at 60 Hz via Meta's Hand Tracking SDK, (2) a performance evaluator comparing pressed keys against the target sequence and maintaining a rolling accuracy score, and (3) a ghost renderer that maps accuracy to ghost opacity through a piecewise sigmoid function, providing high visibility when the learner struggles and fading to near-invisible when performance is strong.

Technologies

Meta Quest 3Unity / C#Hand Tracking SDK XR Interaction ToolkitMIDINASA-TLX User Study (n=24)Adaptive AI

BibTeX

@inproceedings{hsieh2026skill,
  title={Skill-Adaptive Ghost Instructors: Enhancing Retention and Reducing Over-Reliance in VR Piano Learning},
  author={Hsieh, Tzu-Hsin and Michelle Stefanie Visser, Cassandra and Eisemann, Elmar and Marroquim, Ricardo},
  booktitle={Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems},
  pages={1--17},
  year={2026}
}