5.9 KiB
5.9 KiB
Masterarbeit UDP Multiplayer System: Structure & Roles Overview
This document explains how this custom Unity multiplayer system works. It covers real-time avatar, audio, and experiment synchronization, and goes through all the main multiplayer components.
1. High-Level Architecture
The system enables multiple VR clients (users, experimenters, or bots) to:
- Synchronize avatars (pose, facial blendshapes)
- Stream live audio (mic/NPC speech)
- React to central experiment control (tasks, object activation)
- Discover peers automatically (no manual IP entry)
- Debug and inspect all network state in real time
All this communication happens over a single shared UDP port (from NetworkConfig), allowing both unicast and broadcast, and avoiding race conditions/conflicts in Unity's component-based design.
2. Key Modules & Their Roles
A. Avatar Synchronization
Scripts:
UDPAvatarBroadcaster.cs,UDPAvatarBroadcasterAgent.csUDPAvatarReceiver.cs,UDPAvatarReceiverAgent.cs- Location: Assets/Scripts (main code), "Agent" variants support ReadyPlayerMe/CC4 naming
Function:
- Broadcaster: Gathers local avatar transform (root, bones, blendshapes) and sends packets at a steady rate using UDP (binary, compact protocol)
- Receiver: Applies the most recent received pose/facial data to the local duplicate avatar
- Mode: Optimized (priority bones only) or Full (all bones/blends)
Public API:
SetPlayerID(byte id)(filter by remote peer if needed)- Inspector toggles (start/stop broadcast, debug info)
B. Audio Streaming
Scripts:
ConvaiSimpleUDPAudioSender/Receiver.cs(Player-to-player, live mic, PTT)ConvaiUDPSpeechSender/Receiver.cs(NPC/AI speech across network)- Location: Assets/Scripts/Multiplayer
Function:
- SimpleUDPAudioSender: Captures mic, chunks to short UDP packets, broadcasts to peer
- SimpleUDPAudioReceiver: Buffers incoming packets, simulates mic input on target NPC
- UDPSpeechSender/Receiver: Dedicated for Convai/ReadyPlayerMe/NPC speech playback, includes transcript metadata and allows for sequential clip transmission (not just live voice)
Public Events:
OnAudioReceiving,OnSpeechReceiving(Unity Actions for state/busy status)
C. Experiment State Control
Script:
VRExperimentController.cs- Location: Assets/Scripts
Function:
- Listens to incoming UDP (JSON messages) for commands from a supervisor or experimenter
- Controls what avatars/objects/world elements are active based on experimental tasks/blocks/trials
- Ensures logic for practice rounds, condition changes, object resets
Key API/Bindings:
- Maps all avatars/objects via inspector
- Handles: start/stop experiment, practice block, activate/deactivate particular items
- Scene state queries (is object/condition active, etc)
D. UDP Port Sharing & Packet Routing
Script:
SharedUDPListener.cs- Location: Assets/Scripts/Multiplayer
Function:
- Singleton MonoBehaviour that owns the UDP socket
- Dispatches incoming packets to all subscribers based on event:
OnPacketReceived(byte[] packet, IPEndPoint from) - Ensures only one script binds to each UDP port (solves race/conflict problem in Unity)
Usage:
- All networked scripts subscribe/unsubscribe as needed
E. Peer Discovery & Dynamic Targeting
Script:
UDPPeerDiscovery.cs- Location: Assets/Scripts/Multiplayer
Function:
- Broadcasts regular 'hello' packets with player ID and waits for similar packets from peers
- On discovery, updates all Broadcaster/Audio scripts to direct unicast, not just broadcast
- Handles peer disconnects/loss, provides connection state events
Public Events:
OnPeerDiscovered,OnPeerLost, and status properties (connection state, peer IP, local player ID)
F. In-Game Network Debugging
Script:
NetworkDebugUI.cs- Location: Assets/Scripts/Multiplayer
Function:
- Provides a world-space (VR compatible) or 2D overlay UI showing:
- Peer discovery status
- Audio/pose sender/receiver status (packets, port, active/inactive)
- Experiment state, currently active objects
- Packet throughput/logs
- Uses Input System for toggle, supports VR and editor
Public API:
- Toggle/show/hide programmatically
- Extendable for additional waveform, voice, or custom logs
4. Script/Module Summary Table
| Script | Role/Function | Key API / Events |
|---|---|---|
| UDPAvatarBroadcaster(.Agent) | Broadcast live pose/facial state as binary UDP | SetPlayerID, Inspector toggles |
| UDPAvatarReceiver(.Agent) | Receive/apply remote pose/facial updates | SetPlayerID |
| ConvaiSimpleUDPAudioSender/Rcv | Mic/PTT live audio stream; player voice | OnAudioReceiving |
| ConvaiUDPSpeechSender/Receiver | NPC/AI speech transfer with transcript | OnSpeechReceiving |
| VRExperimentController | Orchestrate scene, respond to experiment JSON, control object state | Condition/object methods |
| SharedUDPListener | UDP socket singleton, routes all packets | OnPacketReceived event |
| UDPPeerDiscovery | Peer IP autodiscovery, targeting updates | OnPeerDiscovered/Lost |
| NetworkDebugUI | VR debug/status in-game overlay | Show/Hide |
| NetworkConfig | ScriptableObject for IP/port/config | Static instance |