From 761f6b1bfe6c7c0533138f8795585e180f0de867 Mon Sep 17 00:00:00 2001 From: "tom.hempel" Date: Thu, 30 Oct 2025 12:18:47 +0100 Subject: [PATCH] added comprehensive overview document --- .../Assets/Scripts/script-overview.md | 141 ++++++++++++++++++ 1 file changed, 141 insertions(+) create mode 100644 Unity-Master/Assets/Scripts/script-overview.md diff --git a/Unity-Master/Assets/Scripts/script-overview.md b/Unity-Master/Assets/Scripts/script-overview.md new file mode 100644 index 0000000..20883b2 --- /dev/null +++ b/Unity-Master/Assets/Scripts/script-overview.md @@ -0,0 +1,141 @@ +# Masterarbeit UDP Multiplayer System: Structure & Roles Overview + +This document explains how this custom Unity multiplayer system works. It covers real-time avatar, audio, and experiment synchronization, and goes through all the main multiplayer components. + +--- + +## 1. High-Level Architecture + +The system enables multiple VR clients (users, experimenters, or bots) to: +- Synchronize avatars (pose, facial blendshapes) +- Stream live audio (mic/NPC speech) +- React to central experiment control (tasks, object activation) +- Discover peers automatically (no manual IP entry) +- Debug and inspect all network state in real time + +All this communication happens over a _single shared UDP port_ (from `NetworkConfig`), allowing both unicast and broadcast, and avoiding race conditions/conflicts in Unity's component-based design. + +--- + +## 2. Key Modules & Their Roles + +### **A. Avatar Synchronization** + +**Scripts:** +- `UDPAvatarBroadcaster.cs`, `UDPAvatarBroadcasterAgent.cs` +- `UDPAvatarReceiver.cs`, `UDPAvatarReceiverAgent.cs` +- _Location_: Assets/Scripts (main code), "Agent" variants support ReadyPlayerMe/CC4 naming + +**Function:** +- **Broadcaster**: Gathers local avatar transform (root, bones, blendshapes) and sends packets at a steady rate using UDP (binary, compact protocol) +- **Receiver**: Applies the most recent received pose/facial data to the local duplicate avatar +- **Mode:** Optimized (priority bones only) or Full (all bones/blends) + +**Public API:** +- `SetPlayerID(byte id)` (filter by remote peer if needed) +- Inspector toggles (start/stop broadcast, debug info) + +--- + +### **B. Audio Streaming** + +**Scripts:** +- `ConvaiSimpleUDPAudioSender/Receiver.cs` (Player-to-player, live mic, PTT) +- `ConvaiUDPSpeechSender/Receiver.cs` (NPC/AI speech across network) +- _Location_: Assets/Scripts/Multiplayer + +**Function:** +- **SimpleUDPAudioSender**: Captures mic, chunks to short UDP packets, broadcasts to peer +- **SimpleUDPAudioReceiver**: Buffers incoming packets, simulates mic input on target NPC +- **UDPSpeechSender/Receiver**: Dedicated for Convai/ReadyPlayerMe/NPC speech playback, includes transcript metadata and allows for sequential clip transmission (not just live voice) + +**Public Events:** +- `OnAudioReceiving`, `OnSpeechReceiving` (Unity Actions for state/busy status) + +--- + +### **C. Experiment State Control** + +**Script:** +- `VRExperimentController.cs` +- _Location_: Assets/Scripts + +**Function:** +- Listens to incoming UDP (JSON messages) for commands from a supervisor or experimenter +- Controls what avatars/objects/world elements are active based on experimental tasks/blocks/trials +- Ensures logic for practice rounds, condition changes, object resets + +**Key API/Bindings:** +- Maps all avatars/objects via inspector +- Handles: start/stop experiment, practice block, activate/deactivate particular items +- Scene state queries (is object/condition active, etc) + +--- + +### **D. UDP Port Sharing & Packet Routing** + +**Script:** +- `SharedUDPListener.cs` +- _Location_: Assets/Scripts/Multiplayer + +**Function:** +- **Singleton** MonoBehaviour that _owns the UDP socket_ +- Dispatches incoming packets to all subscribers based on event: `OnPacketReceived(byte[] packet, IPEndPoint from)` +- Ensures only one script binds to each UDP port (solves race/conflict problem in Unity) + +**Usage:** +- All networked scripts _subscribe/unsubscribe_ as needed + +--- + +### **E. Peer Discovery & Dynamic Targeting** + +**Script:** +- `UDPPeerDiscovery.cs` +- _Location_: Assets/Scripts/Multiplayer + +**Function:** +- Broadcasts regular 'hello' packets with player ID and waits for similar packets from peers +- On discovery, updates all Broadcaster/Audio scripts to direct unicast, not just broadcast +- Handles peer disconnects/loss, provides connection state events + +**Public Events:** +- `OnPeerDiscovered`, `OnPeerLost`, and status properties (connection state, peer IP, local player ID) + +--- + +### **F. In-Game Network Debugging** + +**Script:** +- `NetworkDebugUI.cs` +- _Location_: Assets/Scripts/Multiplayer + +**Function:** +- Provides a world-space (VR compatible) or 2D overlay UI showing: + - Peer discovery status + - Audio/pose sender/receiver status (packets, port, active/inactive) + - Experiment state, currently active objects + - Packet throughput/logs +- Uses Input System for toggle, supports VR and editor + +**Public API:** +- Toggle/show/hide programmatically +- Extendable for additional waveform, voice, or custom logs + +--- + +## 4. Script/Module Summary Table + +| Script | Role/Function | Key API / Events | +|-----------------------------------|----------------------------------------------------------------------|---------------------------------| +| UDPAvatarBroadcaster(.Agent) | Broadcast live pose/facial state as binary UDP | SetPlayerID, Inspector toggles | +| UDPAvatarReceiver(.Agent) | Receive/apply remote pose/facial updates | SetPlayerID | +| ConvaiSimpleUDPAudioSender/Rcv | Mic/PTT live audio stream; player voice | OnAudioReceiving | +| ConvaiUDPSpeechSender/Receiver | NPC/AI speech transfer with transcript | OnSpeechReceiving | +| VRExperimentController | Orchestrate scene, respond to experiment JSON, control object state | Condition/object methods | +| SharedUDPListener | UDP socket singleton, routes all packets | OnPacketReceived event | +| UDPPeerDiscovery | Peer IP autodiscovery, targeting updates | OnPeerDiscovered/Lost | +| NetworkDebugUI | VR debug/status in-game overlay | Show/Hide | +| NetworkConfig | ScriptableObject for IP/port/config | Static instance | + +---