# Behavioral Tracking

Source: https://www.potatoannotator.com/docs/features/behavioral-tracking

Potato's behavioral tracking system captures detailed interaction data during annotation sessions, enabling researchers to analyze annotator behavior, timing patterns, AI assistance usage, and decision-making processes.

## Overview

The behavioral tracking system captures:

- **Every annotation action**: Label selections, span annotations, text inputs
- **Precise timestamps**: Server and client-side timestamps
- **AI assistance usage**: When suggestions were shown and whether they were accepted
- **Focus and timing data**: Time spent on each element, scroll depth
- **Navigation history**: Complete path through instances

## What Gets Tracked

### Interaction Events

Every user interaction with the annotation interface is captured:

| Event Type | Description | Example Target |
|------------|-------------|----------------|
| `click` | Mouse clicks on elements | `label:positive`, `nav:next` |
| `focus_in` | Element receives focus | `textbox:explanation` |
| `focus_out` | Element loses focus | `label:negative` |
| `keypress` | Keyboard shortcuts | `key:1`, `nav:ArrowRight` |
| `navigation` | Instance navigation | `next`, `prev`, `instance_load` |
| `save` | Annotation save events | `instance:123` |
| `annotation_change` | Label modifications | `schema:sentiment` |

### AI Assistance Usage

Complete lifecycle tracking for AI-assisted annotation:

```json
{
  "request_timestamp": 1706500010.0,
  "response_timestamp": 1706500012.5,
  "schema_name": "sentiment",
  "suggestions_shown": ["positive", "neutral"],
  "suggestion_accepted": "positive",
  "time_to_decision_ms": 3500
}
```

### Annotation Changes

Detailed change history for all annotations:

```json
{
  "timestamp": 1706500002.5,
  "schema_name": "sentiment",
  "label_name": "positive",
  "action": "select",
  "old_value": null,
  "new_value": true,
  "source": "user"
}
```

**Source types:**
- `user` - Direct user interaction
- `ai_accept` - User accepted AI suggestion
- `keyboard` - Keyboard shortcut used
- `prefill` - Pre-filled from configuration

## Data Format

Each annotation instance includes a `behavioral_data` object:

```json
{
  "id": "instance_123",
  "annotations": {
    "sentiment": {"positive": true}
  },
  "behavioral_data": {
    "instance_id": "instance_123",
    "session_start": 1706500000.0,
    "session_end": 1706500045.0,
    "total_time_ms": 45000,
    "interactions": [...],
    "ai_usage": [...],
    "annotation_changes": [...],
    "navigation_history": [...],
    "focus_time_by_element": {
      "label:positive": 2500,
      "textbox:explanation": 8000
    },
    "scroll_depth_max": 75.5
  }
}
```

## Configuration

Behavioral tracking is enabled by default. No additional configuration is required.

### Frontend Debug Mode

To enable debug logging for the interaction tracker:

```javascript
// In browser console
window.interactionTracker.setDebugMode(true);
```

## Analysis Examples

### Loading Behavioral Data

```python
import json
from pathlib import Path

def load_behavioral_data(annotation_dir: str) -> dict:
    data = {}
    for user_dir in Path(annotation_dir).iterdir():
        if not user_dir.is_dir():
            continue
        state_file = user_dir / 'user_state.json'
        if state_file.exists():
            with open(state_file) as f:
                user_state = json.load(f)
            user_id = user_state.get('user_id')
            behavioral = user_state.get('instance_id_to_behavioral_data', {})
            data[user_id] = behavioral
    return data
```

### Analyzing Annotation Time

```python
def analyze_annotation_time(behavioral_data: dict) -> dict:
    stats = {}
    for user_id, instances in behavioral_data.items():
        times = []
        for instance_id, bd in instances.items():
            if 'total_time_ms' in bd:
                times.append(bd['total_time_ms'] / 1000)
        if times:
            stats[user_id] = {
                'mean_time': sum(times) / len(times),
                'min_time': min(times),
                'max_time': max(times),
                'total_instances': len(times)
            }
    return stats
```

### Detecting Suspicious Behavior

```python
def detect_suspicious_annotators(behavioral_data: dict,
                                  min_time_threshold: float = 2.0) -> list:
    suspicious = []
    for user_id, instances in behavioral_data.items():
        fast_count = 0
        for instance_id, bd in instances.items():
            time_sec = bd.get('total_time_ms', 0) / 1000
            if time_sec < min_time_threshold:
                fast_count += 1

        total = len(instances)
        if total > 0:
            fast_rate = fast_count / total
            if fast_rate > 0.5:
                suspicious.append({
                    'user_id': user_id,
                    'fast_rate': fast_rate,
                    'total_instances': total
                })
    return suspicious
```

## Admin Dashboard Integration

The Admin Dashboard includes a **Behavioral Analytics** tab with:

1. **User Interaction Heatmap**: Visual representation of interaction patterns
2. **AI Assistance Metrics**: Accept/reject rates, decision times
3. **Timing Distribution**: Histogram of annotation times
4. **Suspicious Activity Alerts**: Flagged annotators requiring review

## API Endpoints

### Track Interactions

```http
POST /api/track_interactions
Content-Type: application/json

{
  "instance_id": "instance_123",
  "events": [...],
  "focus_time": {"element": ms},
  "scroll_depth": 75.5
}
```

### Get Behavioral Data

```http
GET /api/behavioral_data/<instance_id>
```

Returns the complete behavioral data for an instance.

## Best Practices

### For Researchers

1. **Baseline Establishment**: Collect behavioral data from known-good annotators to establish baselines
2. **Quality Metrics**: Use behavioral data alongside annotation agreement for quality assessment
3. **Training Evaluation**: Compare pre- and post-training behavioral patterns
4. **AI Impact Analysis**: Measure how AI assistance affects annotation quality and speed

### For Annotation Projects

1. **Monitor in Real-Time**: Use the admin dashboard to spot issues early
2. **Set Thresholds**: Define acceptable ranges for timing and interaction metrics
3. **Provide Feedback**: Use behavioral insights to provide targeted annotator feedback

## Troubleshooting

### No Behavioral Data Being Collected

1. Verify `interaction_tracker.js` is loaded (check browser Network tab)
2. Check browser console for JavaScript errors
3. Verify API endpoints are accessible (`/api/track_interactions`)

### Data Not Persisting

1. Check that user state is being saved (look for `user_state.json`)
2. Ensure the annotation output directory is writable

## Further Reading

- [Admin Dashboard](/docs/features/admin-dashboard) - Real-time monitoring
- [Annotation History](/docs/features/annotation-history) - Detailed change tracking
- [Quality Control](/docs/features/quality-control) - Automated quality checks

For implementation details, see the [source documentation](https://github.com/davidjurgens/potato/blob/main/docs/behavioral_tracking.md).