Skip to content

Unified Geofence Service Fixes

Overview

Fixed critical issues preventing geofence event detection and status updates in the unified geofence service.

Problems Identified

1. Missing Integration (CRITICAL)

Issue: Tracker fetcher service stored location reports but never called the unified geofence service to process them.

Symptoms:

  • Location reports had nearest_city = NULL (not geocoded)
  • No geofence events generated
  • Tracker statuses not updated despite location changes

Example: Location reports 791903 and 791902 met all quality thresholds but were never processed.

2. EXIT Events Never Generated

Issue: Geofence events were only created during status changes, and only inside _update_tracker_status_atomically().

Problem Flow:

if new_status != current_status:
    # Creates events HERE - but only when status changes!
    _update_tracker_status_atomically(...)
else:
    # NO EVENT CREATED - tracker could exit geofence with no event!
    _update_tracking_fields(tracker)

Result: A tracker moving slightly outside a geofence (but not far enough to trigger IN_TRANSIT due to hysteresis) would generate NO EXIT event.

3. Excessive Hysteresis

Issue: Hysteresis buffer was too large, making exit detection very slow.

Old Values:

  • MAX_HYSTERESIS_BUFFER_METERS = 100 (100 meters!)
  • HYSTERESIS_PERCENTAGE = 0.2 (20%)

Example with 100m geofence radius:

  • Entry: 100m + 85m accuracy = 185m effective radius
  • Exit: 100m + 85m accuracy + 100m hysteresis = 285m effective radius

A tracker needed to move 285 meters away before being detected as IN_TRANSIT!

Solutions Implemented

Phase 1: Add Integration Between Services

File: services/tracker_fetcher/taskiq_service.py

Changes:

  1. Modified _store_location_reports() to track new report IDs
  2. Changed _store_single_location_report() to return report ID (instead of bool)
  3. Added _process_reports_through_geofence_service() method
  4. Integrated geofence processing after storing each batch of reports

Key Code:

def _store_location_reports(self, hashed_adv_key: str, reports: List) -> int:
    new_reports_count = 0
    new_report_ids = []  # Track new reports

    with get_db_context() as db:
        for report in reports:
            report_id = self._store_single_location_report(db, hashed_adv_key, report)
            if report_id:
                new_reports_count += 1
                new_report_ids.append(report_id)

    # NEW: Process through geofence service
    if new_report_ids:
        self._process_reports_through_geofence_service(new_report_ids)

    return new_reports_count

Benefits:

  • Location reports now automatically processed
  • Geocoding happens immediately
  • Status updates occur in real-time
  • Geofence events generated for each report

Phase 2: Fix EXIT Event Generation

File: services/unified_geofence_service/service.py

Changes:

  1. Added _get_previous_geofence_state() to track last geofence
  2. Added _generate_geofence_events() to create events independently of status
  3. Modified process_location_report() to generate events before status update
  4. Added skip_event_creation parameter to prevent duplicate events

Key Logic:

# Get previous geofence state
previous_geofence = self._get_previous_geofence_state(tracker.id)

# Determine current geofence
new_status, delivery_id, storage_id = self._determine_status_with_hysteresis(...)

# Generate events BEFORE status update
events_created = self._generate_geofence_events(
    tracker=tracker,
    location_report=location_report,
    previous_geofence=previous_geofence,
    current_delivery_id=delivery_id,
    current_storage_id=storage_id,
)

# Then update status (skip event creation to avoid duplicates)
if new_status != current_status:
    self._update_tracker_status_atomically(..., skip_event_creation=True)

Event Generation Rules:

  • EXIT event: Generated when previous_geofence exists but current_geofence is None
  • ENTRY event: Generated when entering a new geofence or switching between geofences
  • Events created regardless of status change

Benefits:

  • EXIT events now properly generated
  • Complete audit trail of geofence crossings
  • Events track actual position changes, not just status changes

Phase 3: Tune Hysteresis

File: services/unified_geofence_service/service.py

Changes:

# OLD VALUES
MAX_HYSTERESIS_BUFFER_METERS = 100  # Too large!
HYSTERESIS_PERCENTAGE = 0.2  # 20% - too aggressive!

# NEW VALUES
MAX_HYSTERESIS_BUFFER_METERS = 30  # Reduced to 30m
HYSTERESIS_PERCENTAGE = 0.15  # Reduced to 15%

Impact with 100m geofence radius:

Scenario Old New
Entry radius 185m 185m (unchanged)
Exit radius 285m 215m (70m more responsive!)

Benefits:

  • Faster detection of exits
  • More accurate status transitions
  • Still maintains stability against GPS drift

Expected Behavior After Fixes

Normal Flow

  1. Tracker fetcher retrieves location reports from Apple FindMy
  2. Reports stored in location_reports table with RETURNING id
  3. NEW: Report IDs passed to unified geofence service
  4. Unified geofence service:
  5. Geocodes location (populates nearest_city)
  6. Checks quality thresholds
  7. Gets previous geofence state
  8. Determines current geofence presence
  9. Generates EXIT event if left geofence
  10. Generates ENTRY event if entered geofence
  11. Updates tracker status if changed
  12. Creates status history

Example: Tracker Movement

Scenario: Tracker is DELIVERED, moves away from delivery location

Old Behavior:

  • Location reports stored
  • No processing by geofence service
  • No EXIT event generated
  • Status remains DELIVERED indefinitely

New Behavior:

  1. Location report stored with ID=791903
  2. Geofence service processes report 791903
  3. Geocodes location → nearest_city = "Birmingham"
  4. Previous state: In delivery location (from last ENTRY event)
  5. Current state: Outside all geofences (>215m from delivery)
  6. EXIT event generated for delivery location
  7. Status updated: DELIVERED → IN_TRANSIT
  8. Status history created

Testing Recommendations

1. Check Recent Location Reports

SELECT id, hashed_adv_key, timestamp, nearest_city, confidence, horizontal_accuracy
FROM location_reports
WHERE timestamp > NOW() - INTERVAL '1 hour'
ORDER BY timestamp DESC
LIMIT 20;

Expected: nearest_city should be populated for new reports (not NULL)

2. Check Geofence Events

SELECT ge.id, ge.tracker_id, ge.event_type, ge.timestamp,
       t.name as tracker_name,
       dl.name as delivery_location,
       sl.name as storage_location
FROM geofence_events ge
JOIN trackers t ON ge.tracker_id = t.id
LEFT JOIN delivery_locations dl ON ge.delivery_location_id = dl.id
LEFT JOIN storage_locations sl ON ge.storage_location_id = sl.id
WHERE ge.timestamp > NOW() - INTERVAL '1 hour'
ORDER BY ge.timestamp DESC;

Expected: Should see both ENTRY and EXIT events

3. Check Status Transitions

SELECT sh.tracker_id, sh.timestamp, sh.status,
       t.name as tracker_name,
       dl.name as delivery_location
FROM status_history sh
JOIN trackers t ON sh.tracker_id = t.id
LEFT JOIN delivery_locations dl ON sh.delivery_location_id = dl.id
WHERE sh.timestamp > NOW() - INTERVAL '1 hour'
ORDER BY sh.timestamp DESC;

Expected: Status changes should occur more quickly

4. Monitor Logs

Tracker Fetcher Logs:

"Processed reports through geofence service"
report_count=X, updated=Y, skipped=Z

Unified Geofence Service Logs:

"Generated EXIT event for tracker {id}"
"Generated ENTRY event for tracker {id}"
"Status changed for tracker {id}: DELIVERED -> IN_TRANSIT"

Configuration Tuning

If exit detection is still too slow or too fast, adjust these constants in services/unified_geofence_service/service.py:

# Make exits FASTER (more sensitive to movement)
MAX_HYSTERESIS_BUFFER_METERS = 20  # Lower value
HYSTERESIS_PERCENTAGE = 0.10  # Lower percentage

# Make exits SLOWER (more stable against GPS drift)
MAX_HYSTERESIS_BUFFER_METERS = 50  # Higher value
HYSTERESIS_PERCENTAGE = 0.20  # Higher percentage

Recommended: Start with current values (30m, 15%) and monitor for 24-48 hours before adjusting.

Deployment Notes

Prerequisites

  • No database schema changes required
  • No new dependencies added
  • Backward compatible with existing data

Rollout Steps

  1. Deploy updated code to dev/staging environment
  2. Monitor logs for successful integration
  3. Verify geofence events are being created
  4. Check status updates are occurring
  5. Deploy to production

Rollback Plan

If issues occur:

  1. Revert tracker fetcher service (remove geofence integration)
  2. Location reports will still be stored
  3. Can process backlog later with batch processing task

Performance Considerations

  • Geofence processing adds ~100-200ms per location report
  • Geocoding API calls are cached
  • Database queries optimized with proper indexes
  • Processing happens asynchronously (doesn't block tracker fetching)

Monitoring

Key Metrics to Watch

  1. Location reports processed per hour
  2. Geofence events generated per hour
  3. Ratio of EXIT to ENTRY events (should be roughly equal over time)
  4. Status update latency (time from report to status change)
  5. Geocoding success rate

Health Checks

# Check tracker fetcher is calling geofence service
grep "Processed reports through geofence service" /logs/tracker_fetcher.log

# Check EXIT events are being generated
psql -c "SELECT COUNT(*) FROM geofence_events WHERE event_type = 'exit' AND timestamp > NOW() - INTERVAL '1 hour';"

# Check geocoding is working
psql -c "SELECT COUNT(*) FROM location_reports WHERE nearest_city IS NOT NULL AND timestamp > NOW() - INTERVAL '1 hour';"

Known Limitations

  1. Geocoding rate limits: Using OpenStreetMap Nominatim with 1 request per second limit
  2. Exit detection delay: Still requires tracker to move 215m away (with 100m geofence)
  3. Historical data: Existing location reports not processed (only new reports)

Future Enhancements

  1. Backfill processor: Process historical location reports to populate missing events
  2. Configurable hysteresis: Per-geofence hysteresis settings
  3. Geocoding optimization: Batch geocoding or alternative providers
  4. Real-time notifications: Webhook support for geofence events
  5. Analytics dashboard: Visualize geofence crossings and dwell times

References

  • Original issue: Missing EXIT events and slow status updates
  • Affected location reports: 791903, 791902 (examples)
  • Code changes: 2 files modified, ~200 lines added
  • Testing: Comprehensive behavioral tests needed