added comprehensive overview document
This commit is contained in:
tom.hempel
141
Unity-Master/Assets/Scripts/script-overview.md
Normal file
141
Unity-Master/Assets/Scripts/script-overview.md
Normal file
@ -0,0 +1,141 @@
|
||||
# Masterarbeit UDP Multiplayer System: Structure & Roles Overview
|
||||
|
||||
This document explains how this custom Unity multiplayer system works. It covers real-time avatar, audio, and experiment synchronization, and goes through all the main multiplayer components.
|
||||
|
||||
---
|
||||
|
||||
## 1. High-Level Architecture
|
||||
|
||||
The system enables multiple VR clients (users, experimenters, or bots) to:
|
||||
- Synchronize avatars (pose, facial blendshapes)
|
||||
- Stream live audio (mic/NPC speech)
|
||||
- React to central experiment control (tasks, object activation)
|
||||
- Discover peers automatically (no manual IP entry)
|
||||
- Debug and inspect all network state in real time
|
||||
|
||||
All this communication happens over a _single shared UDP port_ (from `NetworkConfig`), allowing both unicast and broadcast, and avoiding race conditions/conflicts in Unity's component-based design.
|
||||
|
||||
---
|
||||
|
||||
## 2. Key Modules & Their Roles
|
||||
|
||||
### **A. Avatar Synchronization**
|
||||
|
||||
**Scripts:**
|
||||
- `UDPAvatarBroadcaster.cs`, `UDPAvatarBroadcasterAgent.cs`
|
||||
- `UDPAvatarReceiver.cs`, `UDPAvatarReceiverAgent.cs`
|
||||
- _Location_: Assets/Scripts (main code), "Agent" variants support ReadyPlayerMe/CC4 naming
|
||||
|
||||
**Function:**
|
||||
- **Broadcaster**: Gathers local avatar transform (root, bones, blendshapes) and sends packets at a steady rate using UDP (binary, compact protocol)
|
||||
- **Receiver**: Applies the most recent received pose/facial data to the local duplicate avatar
|
||||
- **Mode:** Optimized (priority bones only) or Full (all bones/blends)
|
||||
|
||||
**Public API:**
|
||||
- `SetPlayerID(byte id)` (filter by remote peer if needed)
|
||||
- Inspector toggles (start/stop broadcast, debug info)
|
||||
|
||||
---
|
||||
|
||||
### **B. Audio Streaming**
|
||||
|
||||
**Scripts:**
|
||||
- `ConvaiSimpleUDPAudioSender/Receiver.cs` (Player-to-player, live mic, PTT)
|
||||
- `ConvaiUDPSpeechSender/Receiver.cs` (NPC/AI speech across network)
|
||||
- _Location_: Assets/Scripts/Multiplayer
|
||||
|
||||
**Function:**
|
||||
- **SimpleUDPAudioSender**: Captures mic, chunks to short UDP packets, broadcasts to peer
|
||||
- **SimpleUDPAudioReceiver**: Buffers incoming packets, simulates mic input on target NPC
|
||||
- **UDPSpeechSender/Receiver**: Dedicated for Convai/ReadyPlayerMe/NPC speech playback, includes transcript metadata and allows for sequential clip transmission (not just live voice)
|
||||
|
||||
**Public Events:**
|
||||
- `OnAudioReceiving`, `OnSpeechReceiving` (Unity Actions for state/busy status)
|
||||
|
||||
---
|
||||
|
||||
### **C. Experiment State Control**
|
||||
|
||||
**Script:**
|
||||
- `VRExperimentController.cs`
|
||||
- _Location_: Assets/Scripts
|
||||
|
||||
**Function:**
|
||||
- Listens to incoming UDP (JSON messages) for commands from a supervisor or experimenter
|
||||
- Controls what avatars/objects/world elements are active based on experimental tasks/blocks/trials
|
||||
- Ensures logic for practice rounds, condition changes, object resets
|
||||
|
||||
**Key API/Bindings:**
|
||||
- Maps all avatars/objects via inspector
|
||||
- Handles: start/stop experiment, practice block, activate/deactivate particular items
|
||||
- Scene state queries (is object/condition active, etc)
|
||||
|
||||
---
|
||||
|
||||
### **D. UDP Port Sharing & Packet Routing**
|
||||
|
||||
**Script:**
|
||||
- `SharedUDPListener.cs`
|
||||
- _Location_: Assets/Scripts/Multiplayer
|
||||
|
||||
**Function:**
|
||||
- **Singleton** MonoBehaviour that _owns the UDP socket_
|
||||
- Dispatches incoming packets to all subscribers based on event: `OnPacketReceived(byte[] packet, IPEndPoint from)`
|
||||
- Ensures only one script binds to each UDP port (solves race/conflict problem in Unity)
|
||||
|
||||
**Usage:**
|
||||
- All networked scripts _subscribe/unsubscribe_ as needed
|
||||
|
||||
---
|
||||
|
||||
### **E. Peer Discovery & Dynamic Targeting**
|
||||
|
||||
**Script:**
|
||||
- `UDPPeerDiscovery.cs`
|
||||
- _Location_: Assets/Scripts/Multiplayer
|
||||
|
||||
**Function:**
|
||||
- Broadcasts regular 'hello' packets with player ID and waits for similar packets from peers
|
||||
- On discovery, updates all Broadcaster/Audio scripts to direct unicast, not just broadcast
|
||||
- Handles peer disconnects/loss, provides connection state events
|
||||
|
||||
**Public Events:**
|
||||
- `OnPeerDiscovered`, `OnPeerLost`, and status properties (connection state, peer IP, local player ID)
|
||||
|
||||
---
|
||||
|
||||
### **F. In-Game Network Debugging**
|
||||
|
||||
**Script:**
|
||||
- `NetworkDebugUI.cs`
|
||||
- _Location_: Assets/Scripts/Multiplayer
|
||||
|
||||
**Function:**
|
||||
- Provides a world-space (VR compatible) or 2D overlay UI showing:
|
||||
- Peer discovery status
|
||||
- Audio/pose sender/receiver status (packets, port, active/inactive)
|
||||
- Experiment state, currently active objects
|
||||
- Packet throughput/logs
|
||||
- Uses Input System for toggle, supports VR and editor
|
||||
|
||||
**Public API:**
|
||||
- Toggle/show/hide programmatically
|
||||
- Extendable for additional waveform, voice, or custom logs
|
||||
|
||||
---
|
||||
|
||||
## 4. Script/Module Summary Table
|
||||
|
||||
| Script | Role/Function | Key API / Events |
|
||||
|-----------------------------------|----------------------------------------------------------------------|---------------------------------|
|
||||
| UDPAvatarBroadcaster(.Agent) | Broadcast live pose/facial state as binary UDP | SetPlayerID, Inspector toggles |
|
||||
| UDPAvatarReceiver(.Agent) | Receive/apply remote pose/facial updates | SetPlayerID |
|
||||
| ConvaiSimpleUDPAudioSender/Rcv | Mic/PTT live audio stream; player voice | OnAudioReceiving |
|
||||
| ConvaiUDPSpeechSender/Receiver | NPC/AI speech transfer with transcript | OnSpeechReceiving |
|
||||
| VRExperimentController | Orchestrate scene, respond to experiment JSON, control object state | Condition/object methods |
|
||||
| SharedUDPListener | UDP socket singleton, routes all packets | OnPacketReceived event |
|
||||
| UDPPeerDiscovery | Peer IP autodiscovery, targeting updates | OnPeerDiscovered/Lost |
|
||||
| NetworkDebugUI | VR debug/status in-game overlay | Show/Hide |
|
||||
| NetworkConfig | ScriptableObject for IP/port/config | Static instance |
|
||||
|
||||
---
|
||||
Reference in New Issue
Block a user