# Masterarbeit UDP Multiplayer System: Structure & Roles Overview

This document explains how this custom Unity multiplayer system works. It covers real-time avatar, audio, and experiment synchronization, and goes through all the main multiplayer components.

---

## 1. High-Level Architecture

The system enables multiple VR clients (users, experimenters, or bots) to:
- Synchronize avatars (pose, facial blendshapes)
- Stream live audio (mic/NPC speech)
- React to central experiment control (tasks, object activation)
- Discover peers automatically (no manual IP entry)
- Debug and inspect all network state in real time

All this communication happens over a _single shared UDP port_ (from `NetworkConfig`), allowing both unicast and broadcast, and avoiding race conditions/conflicts in Unity's component-based design.

---

## 2. Key Modules & Their Roles

### **A. Avatar Synchronization**

**Scripts:**
- `UDPAvatarBroadcaster.cs`, `UDPAvatarBroadcasterAgent.cs`
- `UDPAvatarReceiver.cs`, `UDPAvatarReceiverAgent.cs`
- _Location_: Assets/Scripts (main code), "Agent" variants support ReadyPlayerMe/CC4 naming

**Function:**
- **Broadcaster**: Gathers local avatar transform (root, bones, blendshapes) and sends packets at a steady rate using UDP (binary, compact protocol)
- **Receiver**: Applies the most recent received pose/facial data to the local duplicate avatar
- **Mode:** Optimized (priority bones only) or Full (all bones/blends)

**Public API:**
- `SetPlayerID(byte id)` (filter by remote peer if needed)
- Inspector toggles (start/stop broadcast, debug info)

---

### **B. Audio Streaming**

**Scripts:**
- `ConvaiSimpleUDPAudioSender/Receiver.cs`  (Player-to-player, live mic, PTT)
- `ConvaiUDPSpeechSender/Receiver.cs`       (NPC/AI speech across network)
- _Location_: Assets/Scripts/Multiplayer

**Function:**
- **SimpleUDPAudioSender**: Captures mic, chunks to short UDP packets, broadcasts to peer
- **SimpleUDPAudioReceiver**: Buffers incoming packets, simulates mic input on target NPC
- **UDPSpeechSender/Receiver**: Dedicated for Convai/ReadyPlayerMe/NPC speech playback, includes transcript metadata and allows for sequential clip transmission (not just live voice)

**Public Events:**
- `OnAudioReceiving`, `OnSpeechReceiving` (Unity Actions for state/busy status)

---

### **C. Experiment State Control**

**Script:**
- `VRExperimentController.cs`
- _Location_: Assets/Scripts

**Function:**
- Listens to incoming UDP (JSON messages) for commands from a supervisor or experimenter
- Controls what avatars/objects/world elements are active based on experimental tasks/blocks/trials
- Ensures logic for practice rounds, condition changes, object resets

**Key API/Bindings:**
- Maps all avatars/objects via inspector
- Handles: start/stop experiment, practice block, activate/deactivate particular items
- Scene state queries (is object/condition active, etc)

---

### **D. UDP Port Sharing & Packet Routing**

**Script:**
- `SharedUDPListener.cs`
- _Location_: Assets/Scripts/Multiplayer

**Function:**
- **Singleton** MonoBehaviour that _owns the UDP socket_
- Dispatches incoming packets to all subscribers based on event: `OnPacketReceived(byte[] packet, IPEndPoint from)`
- Ensures only one script binds to each UDP port (solves race/conflict problem in Unity)

**Usage:**
- All networked scripts _subscribe/unsubscribe_ as needed

---

### **E. Peer Discovery & Dynamic Targeting**

**Script:**
- `UDPPeerDiscovery.cs`
- _Location_: Assets/Scripts/Multiplayer

**Function:**
- Broadcasts regular 'hello' packets with player ID and waits for similar packets from peers
- On discovery, updates all Broadcaster/Audio scripts to direct unicast, not just broadcast
- Handles peer disconnects/loss, provides connection state events

**Public Events:**
- `OnPeerDiscovered`, `OnPeerLost`, and status properties (connection state, peer IP, local player ID)

---

### **F. In-Game Network Debugging**

**Script:**
- `NetworkDebugUI.cs`
- _Location_: Assets/Scripts/Multiplayer

**Function:**
- Provides a world-space (VR compatible) or 2D overlay UI showing:
    - Peer discovery status
    - Audio/pose sender/receiver status (packets, port, active/inactive)
    - Experiment state, currently active objects
    - Packet throughput/logs
- Uses Input System for toggle, supports VR and editor

**Public API:**
- Toggle/show/hide programmatically
- Extendable for additional waveform, voice, or custom logs

---

## 4. Script/Module Summary Table

| Script                           | Role/Function                                                         | Key API / Events                |
|-----------------------------------|----------------------------------------------------------------------|---------------------------------|
| UDPAvatarBroadcaster(.Agent)      | Broadcast live pose/facial state as binary UDP                       | SetPlayerID, Inspector toggles  |
| UDPAvatarReceiver(.Agent)         | Receive/apply remote pose/facial updates                             | SetPlayerID                     |
| ConvaiSimpleUDPAudioSender/Rcv    | Mic/PTT live audio stream; player voice                              | OnAudioReceiving                |
| ConvaiUDPSpeechSender/Receiver    | NPC/AI speech transfer with transcript                               | OnSpeechReceiving               |
| VRExperimentController            | Orchestrate scene, respond to experiment JSON, control object state  | Condition/object methods        |
| SharedUDPListener                 | UDP socket singleton, routes all packets                             | OnPacketReceived event          |
| UDPPeerDiscovery                  | Peer IP autodiscovery, targeting updates                             | OnPeerDiscovered/Lost           |
| NetworkDebugUI                    | VR debug/status in-game overlay                                      | Show/Hide                       |
| NetworkConfig                     | ScriptableObject for IP/port/config                                  | Static instance                 |

---