Skip to content

Geofence Status Update Investigation

Problem Summary

Trackers have geofence events that don't match their current_status:

  • Tracker 2827: Has ENTRY event at delivery location, but status shows IN_STORAGE
  • Tracker 2645: Has TRANSIT events after storage entry, but status shows IN_STORAGE

Code Analysis

Status Determination Flow

The unified_geofence_service follows this flow:

1. _process_single_report()
   ↓
2. _process_status_and_geofence()
   ↓
3. _determine_status_with_hysteresis()
   ↓
4. If status changed: _handle_status_change() → _update_tracker_status_atomically()
   If status same: _handle_no_status_change() → _create_geofence_event_only()

Status Change Logic

In _update_tracker_status_atomically():

# Updates tracker status
tracker.current_status = new_status
tracker.current_state_start = location_report.timestamp

# Creates status history
status_history.create(db, obj_in=status_create)

# Creates geofence event
event_type = self._determine_event_type(old_status, new_status)
self._create_geofence_event(...)

# Commits atomically
db.commit()

No Status Change Logic

In _create_geofence_event_only():

# Determine event type based on whether we're in a geofence
if delivery_location_id or storage_location_id:
    # Staying in same geofence (DWELL)
    event_type = GeofenceEventType.DWELL
else:
    # Staying in transit (TRANSIT)
    event_type = GeofenceEventType.TRANSIT

Key Finding: TRANSIT Event Creation

The service DOES create TRANSIT events correctly! Looking at the code:

  1. When tracker is not in any geofence → status should be IN_TRANSIT
  2. If current_status is already IN_TRANSIT → no status change
  3. Service creates TRANSIT event via _create_geofence_event_only()

This is correct behavior! The TRANSIT events are meant to be created.

Root Cause: Status Never Changed Initially

The problem is NOT with TRANSIT event creation. The problem is that the status was never updated when it should have been.

Scenario for Tracker 2827

  1. 2025-11-24: Tracker entered storage → Should update to IN_STORAGE ✅ (worked)
  2. 2025-11-29: Tracker entered delivery location 90 → Should update to DELIVERED ❌ (FAILED)
  3. Geofence ENTRY event was created (id 395424) ✅
  4. But tracker status was NOT updated ❌

Scenario for Tracker 2645

  1. 2025-11-26: Tracker entered storage 85 → Should update to IN_STORAGE ✅ (worked)
  2. 2025-12-01 onwards: Tracker left all geofences → Should update to IN_TRANSIT ❌ (FAILED)
  3. TRANSIT events were created correctly ✅
  4. But tracker status was NOT updated ❌

Potential Causes

1. Status Update Failed But Event Creation Succeeded

Looking at _update_tracker_status_atomically():

try:
    # Update tracker
    tracker.current_status = new_status

    # Create status history
    status_history.create(db, obj_in=status_create)

    # Create geofence event
    self._create_geofence_event(...)

    # Commit ALL changes atomically
    db.commit()
    return True
except Exception as e:
    db.rollback()
    return False

This should be atomic - either all succeed or all fail. But the geofence event exists, which means the commit succeeded. So why didn't the status update?

2. Race Condition with Concurrent Processing

The service has this code:

# CRITICAL: Refresh tracker from database to get latest status
# This prevents race conditions when multiple reports are processed concurrently
db.refresh(tracker)
current_status = tracker.current_status or TrackerStatus.CREATED

Possible race condition:

  1. Thread A: Reads tracker status as IN_STORAGE
  2. Thread B: Reads tracker status as IN_STORAGE
  3. Thread A: Determines new status is DELIVERED, starts update
  4. Thread B: Determines new status is IN_TRANSIT, starts update
  5. One update overwrites the other

But wait - the atomic update uses db.commit() which should handle this. Unless...

3. Status Update Logic Bug

Looking more carefully at _handle_status_change():

updated = self._update_tracker_status_atomically(
    db, tracker, new_status, location_report, delivery_id, storage_id
)

result["status_updated"] = updated

if updated:
    self.status_updates += 1
    self.geofence_events_created += 1
else:
    self.logger.error(
        f"Tracker {tracker.id}: Status update FAILED"
    )

The function returns True/False. If it returns False, the error is logged but the transaction is rolled back. So we shouldn't see the geofence event if the update failed.

4. Separate Geofence Event Creation

CRITICAL FINDING! Looking more carefully:

For tracker 2827, the geofence event (id 395424) was created on 2025-11-29, but there might have been a DIFFERENT code path that created the event without updating status.

Wait, let me check if there's any code that creates geofence events outside of the status update flow...

Looking at the code, there are TWO ways geofence events are created:

  1. _update_tracker_status_atomically() - Creates event WITH status update (atomic)
  2. _create_geofence_event_only() - Creates event WITHOUT status update (when status doesn't change)

The bug could be: The service determined the status SHOULD change, but then something prevented the update, and it fell back to creating an event only?

No, that doesn't match the code flow...

5. Service Was Not Running During Location Report

Most Likely Cause: The unified_geofence_service was not running when these location reports came in!

Evidence:

  • Geofence events exist and are timestamped correctly
  • But they might have been created much later (check created_at vs timestamp)
  • Tracker 2827 event: timestamp: 2025-11-29 01:27:44, created_at: 2025-11-29 09:23:04 → 7.5 hour delay!

This suggests:

  1. Location report came in at 01:27:44
  2. Service was not running or failed to process it
  3. Event was created retroactively at 09:23:04 (possibly by a backfill job)
  4. The retroactive event creation didn't update the tracker status

6. Looking for Backfill/Batch Processing Code

The service has a process_pending_location_reports() function that processes unprocessed reports. This might run periodically and create geofence events for old reports, but it might not update tracker status correctly if the tracker has moved on.

The Real Issue

The unified_geofence_service correctly processes location reports and updates status when it runs. But:

  1. Service downtime: If the service isn't running when location reports arrive, they're not processed in real-time
  2. Backfill creates events but not status updates: When the service catches up later, it creates geofence events but may not update tracker status if the tracker has moved to a different location

Recommendations

Immediate Fix

The SQL script we created correctly syncs status from events, which fixes the current data.

Long-term Fixes

  1. Add monitoring to detect when unified_geofence_service is not running
  2. Add alerting for gaps between location report timestamps and geofence event creation
  3. Fix backfill logic to properly update status even for old location reports
  4. Consider event sourcing: Status should be derived from events, not updated separately

Investigation Next Steps

  1. Check service logs for 2025-11-29 01:27:44 (tracker 2827 entry time)
  2. Check if unified_geofence_service was running at that time
  3. Look for error logs around that time
  4. Check how the geofence event was created 7.5 hours later