Sync Scheduler job trigger engine
epic-external-system-integration-configuration-backend-infrastructure-task-008 — Build the core job trigger engine that executes on a pg_cron or Supabase scheduled Edge Function invocation. The engine reads the due integrations from the cron config reader, creates export job records in the sync_run_log table with status 'pending', invokes the Integration Edge Function for each due integration, and updates job status to 'running', 'completed', or 'failed' based on the response.
Acceptance Criteria
Technical Requirements
Execution Context
Tier 3 - 413 tasks
Can start after Tier 2 completes
Handles integration between different epics or system components. Requires coordination across multiple development streams.
Implementation Notes
Implement as job-trigger-engine.ts with a single exported async function runScheduledJobs(supabaseClient, integrationEdgeFunctionUrl). Use a p-limit or manual concurrency pool (not raw Promise.all on large arrays) to cap parallelism at 5. For duplicate prevention, use Postgres advisory locks: SELECT pg_try_advisory_xact_lock(hashtext(org_id || integration_id)) inside a transaction before inserting the pending row. If the lock is not acquired, skip.
Status transitions: use UPDATE ... WHERE status = 'pending' RETURNING * to detect race conditions — if 0 rows updated, another instance won the race. For the structured summary, collect results from all Promise.allSettled results and build the summary before returning. Handle the case where the cron config reader returns 0 due integrations gracefully — return immediately with triggered: 0 without error.
Testing Requirements
Unit tests with mocked Supabase client and Integration Edge Function: (1) single due integration → inserts pending, updates to running, updates to completed; (2) Integration EF returns error → updates to failed with error captured; (3) already-running integration (within 30 min) → skipped, not duplicate-triggered; (4) two concurrent engine calls → advisory lock prevents duplicate runs; (5) parallel execution — 5 integrations triggered concurrently, all complete. Integration test: full end-to-end with Supabase local dev — seed organization_integrations with 3 due entries, invoke scheduler, assert sync_run_log contains 3 completed rows. Load test: 20 due integrations complete within 50s.
Supabase Edge Functions have cold start latency that can cause the first sync invocation after idle periods to fail or timeout when the external API has a short connection window, leading to missed scheduled syncs that go undetected.
Mitigation & Contingency
Mitigation: Configure Edge Function memory and implement a warm-up ping mechanism before heavy sync invocations. Set generous timeout values on the external API calls. Log all cold-start incidents for monitoring.
Contingency: If cold starts cause consistent sync failures, migrate the sync scheduler to a persistent Supabase cron job that pre-warms the function 30 seconds before the scheduled sync time.
The sync scheduler must execute jobs at predictable times for financial reporting accuracy. Drift in cron execution timing (due to Supabase infrastructure delays) could cause syncs to run at wrong times, leading to missing data in accounting exports or duplicate exports across reporting periods.
Mitigation & Contingency
Mitigation: Implement idempotency keys based on integration ID + scheduled period, so re-runs of a delayed sync cannot create duplicate exports. Log actual execution timestamps vs scheduled timestamps and alert on drift exceeding 5 minutes.
Contingency: If scheduler reliability is insufficient, integrate with a dedicated cron service (e.g., pg_cron on Supabase) for millisecond-precise scheduling, replacing the application-level scheduler.
Aggressive health monitoring ping frequency could trigger rate limiting on external APIs (especially Xledger and Dynamics), causing legitimate export calls to fail after the monitor exhausts the API's request quota.
Mitigation & Contingency
Mitigation: Use lightweight health check endpoints (HEAD requests or vendor-specific ping/status endpoints) rather than data requests. Set health check frequency to once per 15 minutes minimum. Implement exponential backoff after consecutive failures.
Contingency: If rate limiting occurs, disable active health monitoring for the affected integration type and switch to passive health detection (mark unhealthy only when a scheduled sync fails).