Looking Alive
An interactive, perceptive Leonardo da Vinci.
A character that doesn't just talk at you. It notices you, and has the restraint to know when not to.
…strong model implementations of the deep neural network and GMM-HMM, with clear optimizations.
The build, in five chapters
One character across four bodies, plus the idea that ties them together. Two chapters have work you can read today: The Mind is built and running, and The Face is underway. The last two are mapped. Open any chapter below.
Chapter one decides who to notice and what to say. This chapter makes that judgment visible — a face that turns toward the person it chose, warms its expression, and speaks the line. It runs on screen today. Putting it on a physical bust is what's still ahead.
Thirty seconds of the character running its authored beats: observe, notice, engage, speak, settle. The engagement state and the spoken line are burned into the frame, because the animation is driven by the same logged decision the audit reads — the face and the record cannot disagree.
The identity had to be his, and his documented likeness is a red-chalk drawing in three-quarter view. Every automatic face animator needs a detectable frontal face, and no detector can find one in chalk. That turned a vague goal — “make it look like Leonardo” — into a pass/fail gate: does MediaPipe detect a face in the output? If not, it cannot animate, so it does not ship.
The first solution generated a photoreal frontal from his documented features. It worked mechanically and failed on the thing that mattered. Kept here rather than quietly dropped, because the path from generic-photoreal to self-portrait-grounded cartoon is the result.
For restyling a specific face while keeping its identity, the literature favours IP-Adapter — which decouples identity from surface style — over InstantID, which entangles them. So IP-Adapter was the correct first choice. It needs a ~2.5 GB image encoder, and two download runs died at the shell timeout with no output. The pivot turned out to be both simpler and more literally grounded: image-to-image straight from the self-portrait uses his actual pixels rather than a CLIP embedding of them, and needs no download at all.
None of this face is a learned model. Every channel is a deterministic, hand-designed function of the logged decision and the spoken line — which is the point. It makes the face auditable by construction: the same decision always produces the same frame, with no black box between the reason and the expression.
A mouth timeline written from the text
Each vowel in the spoken line opens the mouth toward 0.85, each consonant closes it toward 0.25, punctuation rests it at zero — then the whole track is smoothed so it doesn't strobe. No lip-sync model, and the same line always animates identically.
Presets that scale with the engagement score
Each decision state carries a set of ARKit weights: relaxed while observing, brows raised and inviting on a bid, warm on engage. The eye-widen and smile weights scale with the logged score, so a person who is closer and more attentive gets a warmer face than a marginal one.
Geometry, straight from the target's position
Yaw and pitch are a direct function of where the engaged person sits in frame, capped at 28° and 12°, and damped across held targets so the head doesn't jitter. It turns toward the person it chose — not toward the camera.
478 landmarks and a thin-plate spline
MediaPipe finds 478 face landmarks on the still, the weights above move the control points, and a thin-plate spline deforms the image per frame — using per-landmark depth for a pseudo-3D turn. A classical interpolation, not a learned renderer.
With no learned parameters, tuning is dialling control weights — two global scales, expression intensity and head-turn range, move the whole feel. Because the same weights feed the warp, a FLAME rig, and a MetaHuman, what gets dialled in here transfers to any renderer.
A thin-plate spline can only push pixels around a flat still. It turns the head, widens the eyes, and opens the mouth on a timeline — but it cannot build the inside of a mouth or round a cheek into a real smile. Generating a separate image per expression proved the point rather than solving it: to get genuinely new expressions from one identity, you need a 3D model of the head.
GAGAvatar reconstructs a controllable 3D Gaussian head from a single image, driven by FLAME parameters. It is not a pip install: a Linux and CUDA research stack, so the build ran in WSL2 on the same 3080. A conda environment pinned to PyTorch 2.4.1 / CUDA 12.1 / pytorch3d 0.7.8, and — the crux — compiling their custom 32-channel Gaussian rasterizer from source, which meant matching nvcc 12.1 against gcc-12 by hand because Ubuntu's gcc-15 is too new, and supplying the CUDA C++ core headers. The value here was diagnosis: reading the build scripts and isolating each failure until the renderer compiled.
GAGAvatar is trained on real human faces, so a stylised cartoon is out of distribution — and it shows in the mouth interior. The source has a closed mouth, so the model never sees its teeth and can only hallucinate a smear when the jaw opens. Running the same pipeline on a realistic frontal source reconstructs a plausible row of teeth, which confirms the cause is the source rather than a bug. The obvious fix was to regenerate the cartoon with a gentle open-mouth smile — four strengths across four seeds, sixteen candidates. It did not work. His long moustache structurally covers the mouth, so no low-strength pass ever parted the lips. The call was to keep the character and manage the limit at the driving stage instead: drive gentle, mostly closed-mouth expressions — a good fit for a dignified Renaissance sage — and accept the rare smear rather than change the face. A mouth-only inpaint is parked for if open-mouth speech is ever needed.
The 3D frames above are cropped to the source and the reconstruction. The full reenactment figures include a driver panel showing an identifiable real person from the tool's demo footage, which was never consented for publication — so the driver is not shown here, on the same standing rule that de-identified the live frames in chapter one. Re-rendering against a consented or synthetic driver is the clean fix, and it is on the list.
The first run of the generator produced zero output and exited. No error, no traceback, nothing to search for. Isolating it with stepwise flushed prints showed it died the instant diffusers was imported — but only when MediaPipe had already loaded. The cause was a native OpenMP/DLL clash between the two, order-dependent, and a hard abort at the C level, which is why Python never got to raise anything. The fix was to import torch and diffusers before MediaPipe. The inline smoke test had passed the whole time because it happened to import in the safe order — the exact shape of bug that reads as “works on my machine” until you pin the real cause.
Everything below is pretrained and run for inference. Nothing in the face path was trained here — the only network trained in this project is the expression reader in chapter one, and it reads a visitor's face, never da Vinci's. It drives none of this.
This face is a generated stylised approximation grounded in Leonardo's self-portrait — not a documented likeness. That label travels with it wherever it is shown.
Take the face off the screen.
The digital face proves the control system: the same logged decision that chooses a person also aims the gaze, sets the expression, and drives the mouth. Those controls are renderer-agnostic by design. The remaining work is making them move servos instead of pixels — a physical Leonardo bust whose eyes lock onto the chosen guest, whose brow and mouth react in persona, and which speaks with a real voice rather than a text-derived timeline.
A guest steps sideways and the eyes follow the person, not the camera.
Articulated head rig: eyes, brow, and jaw in hardware
Gaze actuation driven by the attention signal that already exists
Real TTS voice, with Audio2Face lip-sync replacing the text rule
Affect read → expression mapping on physical actuators
The PhD arc is built to bring four researchers' strengths together: a believable, deployable character that perceives and reasons about people, explainably, then steps off the screen into a physical, reactive robot.
Markus Gross
Interactive digital characters and the technology that makes them feel present, including projection into physical space.
Joseph Campbell
Theory of mind, anticipating human intent, and interpretable interaction, the backbone of the “explain every decision” principle.
Heni Ben Amor
Reactive control and robot learning: characters and robots that respond to people in the moment, the engine behind the robotic phase.
Stelian Coros
Physics-based, expressive character and robot motion, how a believable performance transfers to a body that obeys physics.
The single most repeatable bit of theme-park magic is a character who makes a guest feel seen. Today that depends on a gifted human performer. This builds it as a real-time, repeatable, explainable system: a character that notices the specific guest in front of it, reacts in persona, plays to a crowd, and eventually steps off the screen into the room. Da Vinci is the first host; the perception and decision engine is the product.
- [1]Wampfler, R., et al. (2025). A Platform for Interactive AI Character Experiences (Digital Einstein). SIGGRAPH Conf. Papers '25.
- [2]Campbell, J. & Ben Amor, H. (2017). Bayesian Interaction Primitives: A SLAM Approach to Human-Robot Interaction. CoRL, PMLR 78.
- [3]Campbell, J., Stepputtis, S. & Ben Amor, H. (2019). Probabilistic Multimodal Modeling for Human-Robot Interaction Tasks. RSS. arXiv:1908.04955.
- [4]Oguntola, I., Campbell, J., Stepputtis, S. & Sycara, K. (2023). Theory of Mind as Intrinsic Motivation for Multi-Agent RL. ICML Workshop. arXiv:2307.01158.
- [5]Zhang, X.-J., et al. (2025). Model-Agnostic Policy Explanations with Large Language Models. COLM. arXiv:2504.05625.
- [6]Serifi, A., et al. (2024). Robot Motion Diffusion Model (RobotMDM): Motion Generation for Robotic Characters. SIGGRAPH Asia.
- [7]Coros, S., et al. (2013). Computational Design of Mechanical Characters. ACM TOG 32(4), SIGGRAPH.
- [8]Bates, J. (1994). The Role of Emotion in Believable Agents. Communications of the ACM 37(7).
- [9]Cheng, T., Song, L., Ge, Y., et al. (2024). YOLO-World: Real-Time Open-Vocabulary Object Detection. CVPR. arXiv:2401.17270.
- [10]Jocher, G., et al. (2024). Ultralytics YOLO11 (software).
- [11]Zhang, Y., Sun, P., Jiang, Y., et al. (2022). ByteTrack: Multi-Object Tracking by Associating Every Detection Box. ECCV. arXiv:2110.06864.
- [12]Abdelrahman, A. A., et al. (2022). L2CS-Net: Fine-Grained Gaze Estimation.
- [13]Lin, T.-Y., Maire, M., Belongie, S., et al. (2014). Microsoft COCO: Common Objects in Context. ECCV.
- [14]Glas, D. F., Shiomi, M., Kanda, T., et al. (2017). Personal Greetings: Personalizing Robot Utterances Based on Novelty of Observed Behavior. Int. J. of Social Robotics.