charade_experiment/README.md

# VR Charades Research Project

## Structure

- `app.apk` - VR Charades application
- `experiment-scripts/` - Control scripts and web interface
- `data-analysis/` - Analysis notebook and face/hand tracking data
- `videos/` - Recorded experiment footage
- `vrcharades-unity/` - Unity source code
- `VR_Charades_Paper.pdf` - Research paper
- `Einverstaendniserklaerungen.pdf` - Consent forms

## Installing the VR Application

### Sideloading via SideQuest

Before running experiments, you need to install `app.apk` on your Quest Pro headsets using SideQuest:

#### Prerequisites
1. **Enable Developer Mode** on your Quest Pro:
   - Create a Meta Developer account at https://developer.oculus.com/
   - Install the Meta Quest mobile app on your phone
   - Add your Quest Pro to your account
   - Go to Settings → Developer Mode and enable it

2. **Download and Install SideQuest**:
   - Download SideQuest from https://sidequestvr.com/
   - Install SideQuest on your computer (Windows/Mac/Linux)

#### Sideloading Steps
1. **Connect Quest Pro to computer** via USB-C cable
2. **Put on the headset** and allow USB debugging when prompted
3. **Open SideQuest** on your computer
4. **Verify connection** - SideQuest should show your Quest Pro as connected (green dot)
5. **Install the APK**:
   - Click the "Install APK from folder" button in SideQuest
   - Navigate to and select `app.apk`
   - Wait for installation to complete
6. **Launch the app**:
   - In VR: Go to App Library → Unknown Sources → VR Charades
   - Or use SideQuest's "Currently Installed Apps" section to launch it

#### Repeat for All Headsets
- Install the APK on all Quest Pro headsets that will participate in the experiment
- Ensure all headsets are connected to the same Wi-Fi network

## Running Experiments

### Prerequisites

1. Create and activate a virtual environment:
```bash
# Windows
cd experiment-scripts
python -m venv venv
venv\Scripts\activate

# Linux/Mac
cd experiment-scripts
python3 -m venv venv
source venv/bin/activate
```

2. Install dependencies:
```bash
pip install -r requirements.txt
```

### Experiment Scripts Overview

- **app.py**: Web interface for controlling the experiment and word list
- **server.py**: UDP relay server for communication between VR clients
- **control.py**: Command-line tool for setting VR client modes and server IP
- **index.html**: Web UI served by app.py
- **word-list.txt**: Default list of charade words

### Setup Instructions

#### 1. Network Setup
- Connect all VR headsets to the same network as the server
- Note the IP addresses of the VR headsets (you'll need these for configuration)
- **Important**: Update `experiment-scripts/server.py` lines 19-20 with your actual VR headset IPs
  ```python
  # Replace these example IPs with your actual headset IPs
  DEVICE1_ADDR = ("YOUR_PLAYER1_IP", 5001)  # e.g., ("192.168.1.100", 5001)
  DEVICE2_ADDR = ("YOUR_PLAYER2_IP", 5001)  # e.g., ("192.168.1.101", 5001)
  ```

#### 2. Start the Relay Server
```bash
cd experiment-scripts
python server.py
```

#### 3. Start the Web Interface
```bash
cd experiment-scripts
python -m fastapi dev app.py
```
Then navigate to `http://localhost:8000`

#### 4. Configure VR Clients

Tell clients which server to connect to:
```bash
# Windows PowerShell (replace with your actual VR headset IPs)
cd experiment-scripts
$env:TARGET_IP="YOUR_PLAYER1_IP,YOUR_PLAYER2_IP" ; python control.py "IP:127.0.0.1"

# Linux/Mac (replace with your actual VR headset IPs)
cd experiment-scripts
TARGET_IP="YOUR_PLAYER1_IP,YOUR_PLAYER2_IP" python3 control.py "IP:127.0.0.1"
```

#### 5. Set Experiment Condition

Choose one of these modes:
```bash
# Dynamic Face only
$env:TARGET_IP="YOUR_PLAYER1_IP,YOUR_PLAYER2_IP" ; python control.py "MODE:1;1;1;0"

# Dynamic Hands only  
$env:TARGET_IP="YOUR_PLAYER1_IP,YOUR_PLAYER2_IP" ; python control.py "MODE:0;0;0;1"

# Dynamic Hands + Face
$env:TARGET_IP="YOUR_PLAYER1_IP,YOUR_PLAYER2_IP" ; python control.py "MODE:1;1;1;1"

# Static Face only
$env:TARGET_IP="YOUR_PLAYER1_IP,YOUR_PLAYER2_IP" ; python control.py "MODE:1;0;0;0"

# Static Hands only (requires controllers)
$env:TARGET_IP="YOUR_PLAYER1_IP,YOUR_PLAYER2_IP" ; python control.py "MODE:0;0;0;1"

# Static Hands + Face (requires controllers for hands)
$env:TARGET_IP="YOUR_PLAYER1_IP,YOUR_PLAYER2_IP" ; python control.py "MODE:1;0;0;1"
```

#### 6. Prepare Word List

You can shuffle the word list directly in the web interface (see Web Interface Usage section below).

### Web Interface Usage

Once you have the web interface running at `http://localhost:8000`:

#### Setting Up a Charades Session

1. **Configure Player IPs**
   - Enter the IP addresses of Player 1 and Player 2 VR headsets
   - These should match the IPs you used in the `control.py` commands

2. **Prepare Word List**
   - Copy your word list from `word-list.txt` and paste it into the large text area on the right side
   - Click the **"Shuffle"** button to randomize the word order
   - Click the **"Modify"** button to generate interactive word items

3. **Set Game Parameters**
   - **Current Player For Acting**: Select which player will be acting out the words
   - **Time (s)**: Set the time limit for each word (e.g., 30 seconds)
   - **Last Word Status**: Set to "None" for the first word

#### Running the Experiment

**Manual Mode (Individual Words)**
1. Enter a word in the "Word" field
2. Select the acting player (Player 1 or Player 2)
3. Set the time limit
4. Click **"Send"** to transmit the word to the VR headsets

**Automatic Mode (Word List)**
1. After clicking "Modify" with your word list, the interface shows all words with timers
2. The system automatically starts with the first word and counts down
3. **During the session**:
   - **Mark words correct**: Hover over a word and check the checkbox if guessed correctly
   - **Visual indicators**: ▶ = Current word, ✅ = Correct, ❌ = Time expired
4. **Stop the session**: Click **"Stop"** to end early

#### Exporting Results

1. After completing the word list, click **"Save as CSV"**
2. This downloads a CSV file with word name, correctness, and time remaining

### Mode Details

Mode format: `<show_head>;<show_facial_expression>;<show_eye_rotation>;<show_hands>`

- **Dynamic modes**: Real-time face/hand tracking
- **Static modes**: Participants use controllers instead of natural movement

## Unity Project Details

### Unity Version
- **Unity Editor**: 6000.0.49f1 (2024.1.19f1)

### Key Dependencies
- **Meta XR SDK Core**: 76.0.1
- **Meta XR SDK Interaction**: 76.0.1
- **Meta XR SDK Movement**: 76.0.1 (local file dependency)
- **Unity XR Interaction Toolkit**: 3.1.1
- **Unity XR Hands**: 1.5.0
- **Unity XR Management**: 4.5.1
- **Unity OpenXR**: 1.14.1
- **Universal Render Pipeline**: 17.0.4
- **Unity Input System**: 1.14.0
- **Unity Timeline**: 1.8.7

### Build Requirements
- Unity 6000.0.49f1 or compatible
- Meta XR SDK 76.0.1
- Android Build Support module
- OpenXR support enabled
- Quest Pro development setup