Most digital characters run a script. The naive "reactive" upgrade is worse in a real space: it greets whoever is nearest, which is a nuisance. The character I'm building does the harder, more human thing — it chooses. It reads who is genuinely present with it, engages the one person for whom it's actually the right moment, and, just as often, chooses to wait. The bet behind the whole project is that engagement feels alive when it's earned and intentional, not constant.
The hard part isn't the face
It's the judgment, and the restraint. Out of everyone in view, who is genuinely open to a moment, is now the right time — or is the honest answer to wait? A great host reads a room: they engage the right person at the right moment and leave everyone else in peace. That's the behavior I'm trying to make repeatable.
The perception layer runs at about 26 FPS on a single consumer GPU, fusing person detection and multi-person tracking (YOLO11 + ByteTrack) with graded gaze and head orientation (L2CS-Net) into one real-time read of the space.
Deciding who is open — and when to hold back
Once the scene is parsed, every person gets a score for how open they are to a moment, right now. Gaze carries the most weight: a graded, directional sense of whether they're looking at the character or past it. Then whether their torso is actually facing in, how close they are, and how long they've lingered.
But a high score isn't enough on its own. The character commits only when two independent signals are both present — a genuine relevance signal and a social-license signal — and otherwise it waits.
def openness_score(person, frame):
# higher = more open to a moment, right now
gaze = looking_at_camera(person.head_pose) # graded 0..1, not a binary
facing = 1.0 if person.torso_facing_camera else 0.4
proximity = inverse_distance(person.bbox, frame)
dwell = normalized_dwell(person.track_id) # have they lingered?
return 0.45 * gaze + 0.25 * facing + 0.15 * proximity + 0.15 * dwell
target = max(people, key=lambda p: openness_score(p, frame))
# restraint: engage at most one, and only when a relevance signal
# AND a social-license signal are both present. otherwise, wait.
if relevance(target) >= REL_GATE and social_license(target) >= LIC_GATE:
behavior = "engage" # commit to this one person, in character
else:
behavior = "wait" # the honest call is to hold back
The weights are the interesting part, and the part I keep tuning. Gaze dominates because presence is mostly about attention, not proximity. Someone standing right next to the screen but looking away is less open than someone a few feet back, locked in.
Restraint is the point
The temptation is to engage more. It's the wrong instinct. A character that talks at whoever is nearest is a greeter; the valuable, human behavior is holding back — engaging the one person for whom it's genuinely the right moment and respecting everyone else. So I measure the policy against an indiscriminate baseline that engages the nearest person every chance it gets, the "salesman," because restraint only means something if you can show what the greedy version would have done instead.
When it does engage, what it says is drawn from authored, fact-checked lines grounded in Leonardo's own notebooks and the historical record — always about the shared craft, and never a remark about the person's appearance, face, or gaze.
Every decision on the record
A character staged for guests has to be trustworthy and tunable, so nothing is a black box. The system logs every choice — who it engaged, who it deliberately passed over, for how long, and exactly which factors drove the score. When it holds back, the log says why. That makes the behavior debuggable for me and explainable to the people deciding whether to put it in front of a crowd. "Why not her?" has an answer.
What's next
The engine that does the judging is built and running today. Next is the measured selectivity result — the restraint policy versus the greedy baseline, on consented or synthetic footage — plus an expressive talking face, a richer in-character voice, and the affect read that lets the character sense how a guest is responding and adapt. The persona is the first host; the earned-engagement engine is the product.