AeThex-OS/temp-connect-extract/AeThex-Connect-main/PHASE4-CALLS.md

PHASE 4: VOICE & VIDEO CALLS - DOCUMENTATION

Overview

Phase 4 implements WebRTC-based voice and video calling with support for:

• 1-on-1 audio and video calls
• Group calls with up to 20 participants
• Screen sharing
• TURN/STUN servers for NAT traversal
• Real-time media controls (mute, video toggle)
• Connection quality monitoring
• Call recording support (infrastructure)

Architecture

WebRTC Topology

1-on-1 Calls: Mesh topology with direct peer-to-peer connections Group Calls: SFU (Selective Forwarding Unit) using Mediasoup (placeholder for future implementation)

Components

┌─────────────┐         ┌─────────────┐         ┌─────────────┐
│   Client A  │◄───────►│   Server    │◄───────►│   Client B  │
│  (Browser)  │  WebRTC │  Socket.io  │  WebRTC │  (Browser)  │
│             │ Signals │  Signaling  │ Signals │             │
└─────────────┘         └─────────────┘         └─────────────┘
       ▲                       │                        ▲
       │                       │                        │
       │            ┌──────────▼──────────┐            │
       └───────────►│   TURN/STUN Server  │◄───────────┘
                    │  (NAT Traversal)    │
                    └─────────────────────┘

Database Schema

Calls Table

CREATE TABLE calls (
  id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  conversation_id UUID NOT NULL REFERENCES conversations(id) ON DELETE CASCADE,
  type VARCHAR(20) NOT NULL DEFAULT 'audio', -- 'audio', 'video', 'screen'
  status VARCHAR(20) NOT NULL DEFAULT 'initiated',
  -- Status: 'initiated', 'ringing', 'active', 'ended', 'missed', 'rejected', 'failed'
  initiated_by UUID NOT NULL REFERENCES users(id) ON DELETE CASCADE,
  started_at TIMESTAMPTZ,
  ended_at TIMESTAMPTZ,
  duration_seconds INTEGER,
  end_reason VARCHAR(50),
  sfu_room_id VARCHAR(255),
  recording_url TEXT,
  quality_stats JSONB,
  created_at TIMESTAMPTZ DEFAULT NOW(),
  updated_at TIMESTAMPTZ DEFAULT NOW()
);

Call Participants Table

CREATE TABLE call_participants (
  id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  call_id UUID NOT NULL REFERENCES calls(id) ON DELETE CASCADE,
  user_id UUID NOT NULL REFERENCES users(id) ON DELETE CASCADE,
  status VARCHAR(20) NOT NULL DEFAULT 'invited',
  -- Status: 'invited', 'ringing', 'joined', 'left', 'rejected', 'missed'
  joined_at TIMESTAMPTZ,
  left_at TIMESTAMPTZ,
  ice_candidates JSONB,
  media_state JSONB DEFAULT '{"audioEnabled": true, "videoEnabled": true, "screenSharing": false}',
  media_stats JSONB,
  connection_quality VARCHAR(20),
  created_at TIMESTAMPTZ DEFAULT NOW(),
  updated_at TIMESTAMPTZ DEFAULT NOW()
);

TURN Credentials Table

CREATE TABLE turn_credentials (
  id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  user_id UUID NOT NULL REFERENCES users(id) ON DELETE CASCADE,
  username VARCHAR(255) NOT NULL,
  credential VARCHAR(255) NOT NULL,
  expires_at TIMESTAMPTZ NOT NULL,
  created_at TIMESTAMPTZ DEFAULT NOW()
);

-- Auto-cleanup function
CREATE OR REPLACE FUNCTION cleanup_expired_turn_credentials()
RETURNS void AS $$
BEGIN
  DELETE FROM turn_credentials WHERE expires_at < NOW();
END;
$$ LANGUAGE plpgsql;

API Endpoints

1. POST /api/calls/initiate

Initiate a new call.

Request:

{
  "conversationId": "uuid",
  "type": "video", // or "audio"
  "participantIds": ["uuid1", "uuid2"]
}

Response:

{
  "callId": "uuid",
  "status": "initiated",
  "participants": [
    {
      "userId": "uuid",
      "userName": "John Doe",
      "userIdentifier": "john@example.com",
      "status": "invited"
    }
  ]
}

2. POST /api/calls/:callId/answer

Answer an incoming call.

Response:

{
  "callId": "uuid",
  "status": "active",
  "startedAt": "2025-01-10T14:30:00Z"
}

3. POST /api/calls/:callId/reject

Reject an incoming call.

Response:

{
  "callId": "uuid",
  "status": "rejected"
}

4. POST /api/calls/:callId/end

End an active call.

Response:

{
  "callId": "uuid",
  "status": "ended",
  "duration": 120,
  "endReason": "ended-by-user"
}

5. PATCH /api/calls/:callId/media

Update media state (mute/unmute, video on/off).

Request:

{
  "audioEnabled": true,
  "videoEnabled": false,
  "screenSharing": false
}

Response:

{
  "success": true,
  "mediaState": {
    "audioEnabled": true,
    "videoEnabled": false,
    "screenSharing": false
  }
}

6. GET /api/calls/turn-credentials

Get temporary TURN server credentials.

Response:

{
  "credentials": {
    "urls": ["turn:turn.example.com:3000"],
    "username": "1736517600:username",
    "credential": "hmac-sha1-hash"
  },
  "expiresAt": "2025-01-11T14:00:00Z"
}

7. GET /api/calls/:callId

Get call details.

Response:

{
  "call": {
    "id": "uuid",
    "conversationId": "uuid",
    "type": "video",
    "status": "active",
    "initiatedBy": "uuid",
    "startedAt": "2025-01-10T14:30:00Z",
    "participants": [...]
  }
}

WebSocket Events

Client → Server

call:offer

Send WebRTC offer to peer.

socket.emit('call:offer', {
  callId: 'uuid',
  targetUserId: 'uuid',
  offer: RTCSessionDescription
});

call:answer

Send WebRTC answer to peer.

socket.emit('call:answer', {
  callId: 'uuid',
  targetUserId: 'uuid',
  answer: RTCSessionDescription
});

call:ice-candidate

Send ICE candidate to peer.

socket.emit('call:ice-candidate', {
  callId: 'uuid',
  targetUserId: 'uuid',
  candidate: RTCIceCandidate
});

Server → Client

call:incoming

Notify user of incoming call.

socket.on('call:incoming', (data) => {
  // data: { callId, conversationId, type, initiatedBy, participants }
});

call:offer

Receive WebRTC offer from peer.

socket.on('call:offer', (data) => {
  // data: { callId, fromUserId, offer }
});

call:answer

Receive WebRTC answer from peer.

socket.on('call:answer', (data) => {
  // data: { callId, fromUserId, answer }
});

call:ice-candidate

Receive ICE candidate from peer.

socket.on('call:ice-candidate', (data) => {
  // data: { callId, fromUserId, candidate }
});

call:ended

Notify that call has ended.

socket.on('call:ended', (data) => {
  // data: { callId, reason, endedBy }
});

call:participant-joined

Notify that participant joined group call.

socket.on('call:participant-joined', (data) => {
  // data: { callId, userId, userName, userIdentifier }
});

call:participant-left

Notify that participant left group call.

socket.on('call:participant-left', (data) => {
  // data: { callId, userId }
});

call:media-state-changed

Notify that participant's media state changed.

socket.on('call:media-state-changed', (data) => {
  // data: { callId, userId, mediaState }
});

Frontend Integration

WebRTC Manager Usage

import WebRTCManager from './utils/webrtc';

// Initialize
const webrtcManager = new WebRTCManager(socket);

// Set TURN credentials
const turnCreds = await fetch('/api/calls/turn-credentials');
await webrtcManager.setTurnCredentials(turnCreds.credentials);

// Get local media stream
const localStream = await webrtcManager.initializeLocalStream(true, true);
localVideoRef.current.srcObject = localStream;

// Setup event handlers
webrtcManager.onRemoteStream = (userId, stream) => {
  remoteVideoRef.current.srcObject = stream;
};

// Initiate call
webrtcManager.currentCallId = callId;
webrtcManager.isInitiator = true;
await webrtcManager.initiateCallToUser(targetUserId);

// Toggle audio/video
webrtcManager.toggleAudio(false); // mute
webrtcManager.toggleVideo(false); // video off

// Screen sharing
await webrtcManager.startScreenShare();
webrtcManager.stopScreenShare();

// Cleanup
webrtcManager.cleanup();

Call Component Usage

import Call from './components/Call';

function App() {
  const [showCall, setShowCall] = useState(false);
  
  return (
    <div>
      {showCall && (
        <Call
          socket={socket}
          conversationId="uuid"
          participants={[
            { userId: 'uuid', userName: 'John Doe' }
          ]}
          onCallEnd={(data) => {
            console.log('Call ended:', data);
            setShowCall(false);
          }}
        />
      )}
    </div>
  );
}

TURN Server Setup (Coturn)

Installation

# Ubuntu/Debian
sudo apt-get update
sudo apt-get install coturn

# Enable service
sudo systemctl enable coturn

Configuration

Edit /etc/turnserver.conf:

# Listening port
listening-port=3000
tls-listening-port=5349

# External IP (replace with your server IP)
external-ip=YOUR_SERVER_IP

# Relay IPs
relay-ip=YOUR_SERVER_IP

# Realm
realm=turn.yourdomain.com

# Authentication
use-auth-secret
static-auth-secret=YOUR_TURN_SECRET

# Logging
verbose
log-file=/var/log/turnserver.log

# Security
no-multicast-peers
no-cli
no-loopback-peers
no-tlsv1
no-tlsv1_1

# Quotas
max-bps=1000000
user-quota=12
total-quota=1200

Environment Variables

Add to .env:

# TURN Server Configuration
TURN_SERVER_HOST=turn.yourdomain.com
TURN_SERVER_PORT=3000
TURN_SECRET=your-turn-secret-key
TURN_TTL=86400

Firewall Rules

# Allow TURN ports
sudo ufw allow 3000/tcp
sudo ufw allow 3000/udp
sudo ufw allow 5349/tcp
sudo ufw allow 5349/udp

# Allow UDP relay ports
sudo ufw allow 49152:65535/udp

Start Service

sudo systemctl start coturn
sudo systemctl status coturn

Testing TURN Server

Use the Trickle ICE test page:

1. Add your TURN server URL: turn:YOUR_SERVER_IP:3000
2. Generate TURN credentials using the HMAC method
3. Click "Gather candidates"
4. Verify relay candidates appear

Media Codecs

Audio

• Codec: Opus
• Sample Rate: 48kHz
• Bitrate: 32-128 kbps (adaptive)
• Features: Echo cancellation, noise suppression, auto gain control

Video

• Codecs: VP8, VP9, H.264 (fallback)
• Clock Rate: 90kHz
• Resolutions:
  • 1280x720 (HD) - default
  • 640x480 (SD) - low bandwidth
  • 320x240 (LD) - very low bandwidth
• Frame Rate: 30 fps (ideal), 15-60 fps range
• Bitrate: 500kbps-2Mbps (adaptive)

Connection Quality Monitoring

The system monitors connection quality based on:

1. Round Trip Time (RTT)

   • Good: < 100ms
   • Fair: 100-300ms
   • Poor: > 300ms

2. Packet Loss

   • Good: < 2%
   • Fair: 2-5%
   • Poor: > 5%

3. Available Bitrate

   • Good: > 500kbps
   • Fair: 200-500kbps
   • Poor: < 200kbps

Quality is checked every 3 seconds and displayed to users.

Error Handling

Common Errors

1. Media Access Denied

   Failed to access camera/microphone: NotAllowedError
   

   • User denied browser permission
   • Solution: Request permission again, show help dialog

2. ICE Connection Failed

   Connection failed with user: ICE connection failed
   

   • NAT/firewall blocking connection
   • Solution: Ensure TURN server is configured and reachable

3. Peer Connection Closed

   Connection closed with user: Connection lost
   

   • Network interruption or user disconnected
   • Solution: Notify user, attempt reconnection

4. Turn Credentials Expired

   TURN credentials expired
   

   • Credentials have 24-hour TTL
   • Solution: Fetch new credentials automatically

Security Considerations

1. TURN Authentication: Time-limited credentials using HMAC-SHA1
2. DTLS: WebRTC encrypts all media streams with DTLS-SRTP
3. JWT Auth: All API calls require valid JWT token
4. Rate Limiting: Protect against DoS attacks
5. User Verification: Verify users are in conversation before allowing calls

Testing Checklist

• 1-on-1 audio call works
• 1-on-1 video call works
• Mute/unmute audio works
• Toggle video on/off works
• Screen sharing works
• Call can be answered
• Call can be rejected
• Call can be ended
• Connection quality indicator updates
• Call duration displays correctly
• Multiple participants can join (group call)
• Participant joins/leaves notifications work
• Media state changes propagate
• TURN server fallback works (test behind NAT)
• Call persists after page refresh (reconnection)
• Missed call notifications work
• Call history is recorded

Performance Optimization

Bandwidth Usage

Audio Only (per participant):

• Opus @ 32kbps: ~15 MB/hour
• Opus @ 64kbps: ~30 MB/hour

Video + Audio (per participant):

• 480p @ 500kbps: ~225 MB/hour
• 720p @ 1Mbps: ~450 MB/hour
• 1080p @ 2Mbps: ~900 MB/hour

Recommendations

1. Start with audio only for low bandwidth users
2. Use VP9 if supported (better compression than VP8)
3. Enable simulcast for group calls (SFU)
4. Adaptive bitrate based on network conditions
5. Limit group calls to 20 participants max

Future Enhancements

• Mediasoup SFU: Implement actual SFU for efficient group calls
• Call Recording: Record and store calls in cloud storage
• Background Blur: Virtual backgrounds using ML
• Noise Cancellation: Advanced audio processing
• Grid/Speaker View: Different layouts for group calls
• Reactions: Emoji reactions during calls
• Hand Raise: Signal to speak in large calls
• Breakout Rooms: Split large calls into smaller groups
• Call Scheduling: Schedule calls in advance
• Call Analytics: Detailed quality metrics and reports

Troubleshooting

No Audio/Video

1. Check browser permissions
2. Verify camera/microphone is not used by another app
3. Test with navigator.mediaDevices.enumerateDevices()
4. Check browser console for errors

Connection Fails

1. Test TURN server with Trickle ICE
2. Verify firewall allows UDP ports 49152-65535
3. Check TURN credentials are not expired
4. Ensure both users are online

Poor Quality

1. Check network bandwidth
2. Monitor packet loss and RTT
3. Reduce video resolution
4. Switch to audio-only mode

Echo/Feedback

1. Ensure echoCancellation: true in audio constraints
2. Use headphones instead of speakers
3. Reduce microphone gain
4. Check for multiple audio sources

Support

For issues or questions:

• Check logs in browser console
• Review /var/log/turnserver.log for TURN issues
• Monitor backend logs for signaling errors
• Test with multiple browsers (Chrome, Firefox, Safari)

---

Phase 4 Complete ✓