Music-JEPA: Learning a World Model of Sound from Action
New York University · Courant Institute of Mathematical Sciences
Developing an action-conditioned world model that predicts future audio representations from musical performance states and actions for embodied piano transcription.
I am extending piano-roll actions with fingering and hand-motion information, and studying forward and inverse prediction to improve transcription accuracy and physical plausibility.
Principal investigator: Yann LeCunMentor: Ziyu Wang
Human- and AI-Assisted Audiovisual Creation
Duke Kunshan University
Designing controlled audiovisual experiments to compare how people respond to human-created and AI-assisted work.
Online participant studies measure audiovisual coherence, emotional response, and perceived authenticity, with analysis of differences between the two conditions.
Supervisor: Juan Carlos Vasquez
Multimodal Temporal Modeling with ECG and Chest X-rays
Duke University · Center for Computational Evolutionary Intelligence
Developing a JEPA-style framework that uses a prior chest X-ray and an intervening ECG signal to predict the representation of a future chest X-ray.
My work explores cross-attention, residual latent prediction, and relative-time conditioning with BioViL-T and ECG-JEPA representations to understand when physiological signals add useful information.
Principal investigator: Yiran ChenMentor: Qijia Huang
