VR Charades Research Project

Structure

• app.apk - VR Charades application
• experiment-scripts/ - Control scripts and web interface
• data-analysis/ - Analysis notebook and face/hand tracking data
• videos/ - Recorded experiment footage
• vrcharades-unity/ - Unity source code
• VR_Charades_Paper.pdf - Research paper
• Einverstaendniserklaerungen.pdf - Consent forms

Installing the VR Application

Sideloading via SideQuest

Before running experiments, you need to install app.apk on your Quest Pro headsets using SideQuest:

Prerequisites

1. Enable Developer Mode on your Quest Pro:

   • Create a Meta Developer account at https://developer.oculus.com/
   • Install the Meta Quest mobile app on your phone
   • Add your Quest Pro to your account
   • Go to Settings → Developer Mode and enable it

2. Download and Install SideQuest:

   • Download SideQuest from https://sidequestvr.com/
   • Install SideQuest on your computer (Windows/Mac/Linux)

Sideloading Steps

1. Connect Quest Pro to computer via USB-C cable
2. Put on the headset and allow USB debugging when prompted
3. Open SideQuest on your computer
4. Verify connection - SideQuest should show your Quest Pro as connected (green dot)
5. Install the APK:
   • Click the "Install APK from folder" button in SideQuest
   • Navigate to and select app.apk
   • Wait for installation to complete
6. Launch the app:
   • In VR: Go to App Library → Unknown Sources → VR Charades
   • Or use SideQuest's "Currently Installed Apps" section to launch it

Repeat for All Headsets

• Install the APK on all Quest Pro headsets that will participate in the experiment
• Ensure all headsets are connected to the same Wi-Fi network

Running Experiments

Prerequisites

1. Create and activate a virtual environment:

# Windows
cd experiment-scripts
python -m venv venv
venv\Scripts\activate

# Linux/Mac
cd experiment-scripts
python3 -m venv venv
source venv/bin/activate

2. Install dependencies:

pip install -r requirements.txt

Experiment Scripts Overview

• app.py: Web interface for controlling the experiment and word list
• server.py: UDP relay server for communication between VR clients
• control.py: Command-line tool for setting VR client modes and server IP
• index.html: Web UI served by app.py
• word-list.txt: Default list of charade words

Setup Instructions

1. Network Setup

• Connect all VR headsets to the same network as the server
• Note the IP addresses of the VR headsets (you'll need these for configuration)
• Important: Update experiment-scripts/server.py lines 19-20 with your actual VR headset IPs

  # Replace these example IPs with your actual headset IPs
  DEVICE1_ADDR = ("YOUR_PLAYER1_IP", 5001)  # e.g., ("192.168.1.100", 5001)
  DEVICE2_ADDR = ("YOUR_PLAYER2_IP", 5001)  # e.g., ("192.168.1.101", 5001)
  

2. Start the Relay Server

cd experiment-scripts
python server.py

3. Start the Web Interface

cd experiment-scripts
python -m fastapi dev app.py

Then navigate to http://localhost:8000

4. Configure VR Clients

Tell clients which server to connect to:

# Windows PowerShell (replace with your actual VR headset IPs)
cd experiment-scripts
$env:TARGET_IP="YOUR_PLAYER1_IP,YOUR_PLAYER2_IP" ; python control.py "IP:127.0.0.1"

# Linux/Mac (replace with your actual VR headset IPs)
cd experiment-scripts
TARGET_IP="YOUR_PLAYER1_IP,YOUR_PLAYER2_IP" python3 control.py "IP:127.0.0.1"

5. Set Experiment Condition

Choose one of these modes:

# Dynamic Face only
$env:TARGET_IP="YOUR_PLAYER1_IP,YOUR_PLAYER2_IP" ; python control.py "MODE:1;1;1;0"

# Dynamic Hands only  
$env:TARGET_IP="YOUR_PLAYER1_IP,YOUR_PLAYER2_IP" ; python control.py "MODE:0;0;0;1"

# Dynamic Hands + Face
$env:TARGET_IP="YOUR_PLAYER1_IP,YOUR_PLAYER2_IP" ; python control.py "MODE:1;1;1;1"

# Static Face only
$env:TARGET_IP="YOUR_PLAYER1_IP,YOUR_PLAYER2_IP" ; python control.py "MODE:1;0;0;0"

# Static Hands only (requires controllers)
$env:TARGET_IP="YOUR_PLAYER1_IP,YOUR_PLAYER2_IP" ; python control.py "MODE:0;0;0;1"

# Static Hands + Face (requires controllers for hands)
$env:TARGET_IP="YOUR_PLAYER1_IP,YOUR_PLAYER2_IP" ; python control.py "MODE:1;0;0;1"

6. Prepare Word List

Shuffle the word list for randomization:

# Windows PowerShell
cd experiment-scripts
Get-Content word-list.txt | Sort-Object {Get-Random} | Out-File word-list-shuffled.txt

# Linux/Mac
cd experiment-scripts
shuf word-list.txt > word-list-shuffled.txt

Web Interface Usage

Once you have the web interface running at http://localhost:8000:

Setting Up a Charades Session

1. Configure Player IPs

   • Enter the IP addresses of Player 1 and Player 2 VR headsets
   • These should match the IPs you used in the control.py commands

2. Prepare Word List

   • Copy your shuffled word list from word-list-shuffled.txt
   • Paste it into the large text area on the right side
   • Click the "Modify" button to generate interactive word items

3. Set Game Parameters

   • Current Player For Acting: Select which player will be acting out the words
   • Time (s): Set the time limit for each word (e.g., 30 seconds)
   • Last Word Status: Set to "None" for the first word

Running the Experiment

Manual Mode (Individual Words)

1. Enter a word in the "Word" field
2. Select the acting player (Player 1 or Player 2)
3. Set the time limit
4. Click "Send" to transmit the word to the VR headsets

Automatic Mode (Word List)

1. After clicking "Modify" with your word list, the interface shows all words with timers
2. The system automatically starts with the first word and counts down
3. During the session:
   • Mark words correct: Hover over a word and check the checkbox if guessed correctly
   • Visual indicators: ▶ = Current word, ✅ = Correct, ❌ = Time expired
4. Stop the session: Click "Stop" to end early

Exporting Results

1. After completing the word list, click "Save as CSV"
2. This downloads a CSV file with word name, correctness, and time remaining

Mode Details

Mode format: <show_head>;<show_facial_expression>;<show_eye_rotation>;<show_hands>

• Dynamic modes: Real-time face/hand tracking
• Static modes: Participants use controllers instead of natural movement

Unity Project Details

Unity Version

• Unity Editor: 6000.0.49f1 (2024.1.19f1)

Key Dependencies

• Meta XR SDK Core: 76.0.1
• Meta XR SDK Interaction: 76.0.1
• Meta XR SDK Movement: 76.0.1 (local file dependency)
• Unity XR Interaction Toolkit: 3.1.1
• Unity XR Hands: 1.5.0
• Unity XR Management: 4.5.1
• Unity OpenXR: 1.14.1
• Universal Render Pipeline: 17.0.4
• Unity Input System: 1.14.0
• Unity Timeline: 1.8.7

Build Requirements

• Unity 6000.0.49f1 or compatible
• Meta XR SDK 76.0.1
• Android Build Support module
• OpenXR support enabled
• Quest Pro development setup