Performance test activity query with large datasets
epic-bufdir-reporting-export-processing-services-task-017 — Implement performance tests for the BufdirActivityQueryService simulating organizations with large activity volumes (1000+ activities, 100+ peer mentors, multiple chapters). Measure query execution time and deduplication processing time, identify bottlenecks, and optimize database queries with appropriate indexes. Target: full org export query completes in under 5 seconds.
Acceptance Criteria
Technical Requirements
Execution Context
Tier 5 - 253 tasks
Can start after Tier 4 completes
Implementation Notes
The primary performance bottleneck is almost certainly the join between activities, peer_mentors, and chapters tables filtered by organization_id. Indexes to consider: (1) CREATE INDEX idx_activities_org_period ON activities(organization_id, reporting_period_id); (2) CREATE INDEX idx_activities_peer_mentor ON activities(peer_mentor_id); (3) CREATE INDEX idx_peer_mentors_org ON peer_mentors(organization_id). Deduplication (removing double-counted activities where a peer mentor belongs to multiple chapters) should be a pure Dart operation on the fetched list — measure this separately from the DB query. Use Supabase's rpc() to call a stored procedure for EXPLAIN ANALYZE output, or connect via a raw pg connection in test-only code.
Seed data generation: use a DataFactory helper that generates realistic Norwegian activity records with proper date ranges within the reporting period. Commit index migrations as numbered SQL files in supabase/migrations/ following the project's existing migration naming convention.
Testing Requirements
Performance tests using flutter_test with Stopwatch for timing measurement. Structure as: (1) one-time setUp that inserts 1000 synthetic activities via batch Supabase inserts (use upsert in chunks of 100 to stay within rate limits); (2) timed measurement block using Stopwatch.start/stop around the BufdirActivityQueryService.fetchAllForOrganization() call; (3) assertion that elapsed milliseconds < 5000; (4) separate timed block for deduplication logic. Log Stopwatch results with print() for CI visibility. After initial run, run EXPLAIN ANALYZE via a raw SQL query through Supabase service role connection and log the plan output.
Apply identified indexes (e.g., on organization_id, reporting_period, peer_mentor_id) in a new migration file. Re-run and assert improvement. Test teardown deletes all seeded records.
NHF contacts can belong to up to five local chapters simultaneously. If the deduplication logic in the activity query service incorrectly attributes cross-chapter activities, organisations will either under-report or over-report to Bufdir, which could trigger grant clawback or compliance investigations.
Mitigation & Contingency
Mitigation: Implement deduplication using the existing multi-chapter membership service as the source of truth for chapter affiliation. Write test fixtures covering all known multi-chapter edge cases and validate outputs against manually prepared reference exports from NHF.
Contingency: If deduplication cannot be made deterministic for complex hierarchies before release, gate the export behind an org-level feature flag and require NHF to validate a preview export against their manual Excel before enabling in production.
Server-side Dart libraries for Excel generation are less mature than equivalents in Node.js or Python. The chosen library may lack support for Bufdir-required formatting features (merged cells, data validation, specific date formats), requiring significant workaround effort or a library switch mid-implementation.
Mitigation & Contingency
Mitigation: Evaluate the top two Dart xlsx libraries (excel, spreadsheet_decoder) against a Bufdir template sample file before committing. Identify all required formatting features and verify library support in a spike.
Contingency: If no Dart library meets requirements, implement the Excel generation as a Supabase Edge Function in TypeScript using the well-supported ExcelJS library, exposing it to the Dart backend via an internal RPC call.
The attachment bundler must retrieve documents from Supabase Storage that were uploaded by the document attachments feature. If storage paths, RLS policies, or signed URL expiry have not been standardised across features, the bundler may fail to retrieve attachments at export time.
Mitigation & Contingency
Mitigation: Audit the document attachments feature's storage schema and RLS policies before implementing the bundler. Agree on a stable internal service-account access pattern for cross-feature storage reads.
Contingency: If cross-feature storage access cannot be made reliable, implement the bundler to include only attachments that can be retrieved successfully and produce a manifest listing any attachments that could not be bundled, rather than failing the entire export.