Conversation
Replace repeated LEFT OUTER JOIN (SELECT callsign, MAX(lastupload) ...) derived-table joins with simpler LEFT JOIN lotw_users lotw across Distances_model, Logbook_model, Logbookadvanced_model, and Worked_all_britain_model. Also remove an unnecessary DISTINCT/inner station_profile join from an optimized Logbook_model subquery. These changes simplify SQL emitted by the CI query builder, improve compatibility and readability, and may improve performance; note that using direct joins can change which lastupload row is returned if multiple lotw_users rows exist for a callsign.
Improve query performance and memory usage by adding a migration to create several indexes and converting various upload/reporting flows to batched processing. Changes include: - Add migration 245 to create indexes: idx_mode, idx_sat_name, idx_continent, idx_qsl_stats (composite), idx_hrdlog_upload, idx_qrz_upload to speed common queries. - Bump migration version to 245 in config. - Hrdlog and Qrz controllers: switch to fetching QSOs in LIMIT/OFFSET batches, process updates in chunks, add logging for batch progress, and use break 2 to stop outer loops on invalid API keys to avoid continuing work. - Logbook_model: - get_hrdlog_qsos and get_qrz_qsos now accept limit/offset and use ORDER BY/LIMIT/OFFSET for batching. - Replaced a subquery with a JOIN for grid-based filtering to improve performance. - get_modes rewritten to use query builder (distinct/select/order_by). - Bulk DXCC updates use update_batch instead of per-row queries. - calls_without_station_id limited to 500 results to avoid UI hangs. Overall goal: prevent memory exhaustion with large datasets and improve database query performance.
Bump migration version and add a new migration (246) to create/drop a 4-char prefix index on COL_GRIDSQUARE to speed LIKE queries; migration 245 now also drops the prefix index if present. Add Stations::user_owns_station and use it in Logbookadvanced_model to avoid N+1 station lookups. Add caching to Logbooks_model::list_logbook_relationships to prevent redundant queries. Optimize Oqrs_model queries to select only needed columns for several methods. Add Stats::getUniqueCallsignsConsolidated to combine multiple unique-callsign groupings into a single set of queries for better performance. Database debug toggles added where appropriate.
Bump migration version to 247 and add a migration that creates performance indexes for LoTW workflows (idx_lotw_qslrdate, idx_lotw_qsos_to_upload, idx_lotw_confirmation_match) to speed common queries. Refactor Lotw controller to collect records in memory, build rows for rendering, and perform a single batch confirmation update via a new Logbook_model::lotw_update_batch() method instead of many individual updates. Implement lotw_update_batch() to look up matching QSOs, perform update_batch() calls, and separately update gridsquares/distances using the Qra library. Simplify LotwCert::toggle_archive_certificate() to toggle archived state with a single UPDATE and return the resulting status.
Replace the specific idx_lotw_confirmation_match index with a generic idx_confirmation_match and update related comments and local variable names. Comments now note the index benefits both LOTW and QRZ processing (import_check, lotw_update_batch, process_qrz_batch). The composite index columns remain (COL_TIME_ON, COL_CALL, COL_BAND, COL_STATION_CALLSIGN); existence checks and add/drop statements were adjusted to use the new name.
Add batch processing for eQSL imports/exports and introduce performance indexes. Bump migration_version to 248 and add Migration_add_eqsl_performance_indexes to create idx_eqsl_qslrdate and idx_eqsl_confirmation_match indexes to speed up eQSL queries and batch matching. Modify Eqsl controller to collect successful uploads and perform a single batch mark-as-sent update. Update EqslImporter to queue confirmations and run a single batch update at the end. Implement eqsl_update_batch() and eqsl_mark_sent_batch() in Eqslmethods_model to perform efficient batch DB operations and logging, reducing per-record queries for large eQSL operations.
Delete the conditional block that dropped the `idx_gridsquare_prefix` index from the table. Add `$this->db->db_debug = true;` and close out the migration method/class. This prevents the migration from removing the gridsquare prefix index and ensures DB debug mode is restored after the migration.
Replace malformed code with a proper existence check for the 'idx_confirmation_match' index, drop the index with ALTER TABLE if present, and re-enable db_debug. This fixes a syntax/logic error in the migration so the LOTW/QRZ/eQSL confirmation match index is removed safely during migration.
Add per-user/logbook file caching for dashboard statistics (15min TTL) and update Dashboard controller to use cached qso, countries, VUCC and QSL queries. Introduce clear_dashboard_cache() in Logbook_model and call it on QSO create/update/delete to invalidate affected users' cache entries. Add migration (249) to create composite indexes (idx_dxcc_stats, idx_vucc_gridsquare, idx_vucc_grids) to improve dashboard query performance, and bump migration_version to 249. Also include generated cache files under application/cache for current stats.
In Logbook_model.php, replace select('DISTINCT user_id') with $this->db->distinct(); $this->db->select('user_id') to use CI's distinct API and produce correct SQL. Also correct the where clause to use station_logbooks_relationship.station_location_id instead of station_id so shared-access users are filtered by the proper station location.
Update .gitignore to exclude all files under /application/cache while whitelisting /application/cache/index.html and /application/cache/.htaccess. This prevents committing transient cache files but preserves placeholder and access-control files needed to keep the directory and settings in the repo.
Add checks around the unsaved-changes logic to bypass the confirmation when the #callsign field is present but empty. The patch inserts a guard in the page unload catcher and before showing the confirmation so users creating new/blank callsign entries won't receive an unnecessary unsaved-change warning (checks element existence and uses trim()).
Introduce a Station Diary UI and backend endpoint to allow quick note creation via HTMX. Added Notes::quick_add in the controller with form validation, note creation, success/error responses and a small JS form reset. Updated view_log/index to show a Station Diary button (when user_show_notes=1) and include the modal form that posts to notes/quick_add and renders messages into the modal.
Display the saved creation timestamp (including time) across notes list, note view, and edit form. Adds a human-readable "Saved" line in application/views/notes/edit.php and application/views/notes/view.php, and updates application/views/notes/main.php to include the time using the format "M d, Y \a\t g:i A". The date input in the edit form still uses a YYYY-MM-DD value for the date picker, and entries with empty or zero dates show "N/A".
Add a Station Diary print feature: controller Notes.php now sets a has_diary_entries flag for the main notes view and adds a station_diary() action that loads Station Diary entries and renders a print-friendly view. The notes main view conditionally shows a "Station Diary" button that opens the print page in a new tab. A new view notes/station_diary_print.php provides a print-optimized layout, entry listing with date/time handling, total entries summary, a print button, and a friendly fallback when no entries are found.
Normalize indentation, spacing and PHP/HTML formatting in application/views/notes/main.php and application/views/notes/station_diary_print.php. Changes are purely cosmetic (whitespace, indentation, minor spacing around concatenation and tags, and newline adjustments) to improve readability and consistency; no functional logic was modified.
Add mappings for QMR-KWT-2 -> QMR-KWT-2_(RS95S) and Lobachevsky -> Lobachevsky_(RS83S), and introduce conditional APRS name mapping for BOTAN (when COL_MODE == 'PKT') and SONATE-2 (when COL_MODE == 'PKT'). These changes extend satellite name normalization in application/models/Logbook_model.php to handle these satellites and modes.
Allow LoTW batch updates to reference QSOs by COL_PRIMARY_KEY for reliable matching. Controller: include 'primary_key' in batch update payloads. Model: collect numeric primary keys, expand WHERE to include COL_PRIMARY_KEY IN (...) alongside the existing datetime/call/band/station match, add early exit/log when no usable keys provided, and build an id-based lookup (qso_map_by_id) to prefer primary-key matches while remaining backward-compatible with the composite key lookup. This reduces ambiguous matches and improves update accuracy.
Remove the per-row "LoTW Status" header and cells from the Lotw table output to avoid repeating the same status for each QSO. Instead, append a single "LoTW Status" summary div after the table using $lotw_status. This simplifies the table layout and centralizes the LoTW status display.
Append the LoTW status div to the $table variable before assigning it to $data['lotw_table'], rather than concatenating it afterward. This ensures the status is part of the table HTML output and avoids the separate post-assignment append.
Initialize and increment a $found_count while parsing ADIF records when import_check returns 'Found', then after batch update compute already-confirmed QSOs and expand the LoTW status message to include: Found, Updated, Already confirmed, and Gridsquares updated. Uses max(0, ...) to avoid negative already-confirmed values and updates the log output accordingly.
Mark certain dropdowns (Gridmaster, Other Export Options, Third Party Logbooks) with a new dropdown-submenu-left class in header.php so submenus can open to the left. Update general.css to style .dropdown-submenu and .dropdown-submenu-left positioning, refine .navbar .dropdown-menu appearance, and improve dropdown item spacing, hover effects and submenu chevron styling for better alignment and visual consistency (prevents overflow and improves UX).
Enable HTMX-driven certificate uploads and partial UI updates for LoTW. Updated Lotw controller to detect HTMX requests in do_cert_upload, return inline success/error alerts for modal uploads, and added cert_table_refresh endpoint to return the cert table partial. Extracted the certificate table into a new view (lotw_views/cert_table.php) and replaced the inline table in index.php with a container that loads that partial. Added a Bootstrap modal view (upload_cert_modal.php) with an HTMX multipart form, drag-and-drop file UI, JS handlers to manage upload state and refresh the certificate list, and modal reset behavior. Minor refactors to flash/success handling to support both full-page and HTMX flows.
Bump migration version to 250 and add Migration_tag_2_8_8. The new migration sets the stored app version to 2.8.8 and resets the user's version_dialog confirmation (option_value = 'false') to trigger the version info dialog. The down() method rolls the version back to 2.8.7.
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 5 potential issues.
Bugbot Autofix is OFF. To automatically fix reported issues with Cloud Agents, enable Autofix in the Cursor dashboard.
This is the final PR Bugbot will review for you during this billing cycle
Your free Bugbot reviews will reset on March 5
Details
You are on the Bugbot Free tier. On this plan, Bugbot will review limited PRs each billing cycle.
To receive Bugbot reviews on all of your PRs, visit the Cursor dashboard to activate Pro and start your 14-day free trial.
| if (!empty($successful_uploads)) { | ||
| $affected_rows = $this->eqslmethods_model->eqsl_mark_sent_batch($successful_uploads); | ||
| log_message('info', 'eQSL export: Marked ' . $affected_rows . ' QSOs as sent to eQSL'); | ||
| } |
There was a problem hiding this comment.
Batch eQSL marks lost when redirect fires mid-loop
High Severity
The batch update of $successful_uploads only runs after the entire foreach loop completes. However, uploadQso() calls redirect() directly on several error conditions (auth failure, 500, 400, format error). When a redirect fires mid-loop, PHP exits before reaching the batch eqsl_mark_sent_batch() call, so any QSOs already successfully uploaded in that session are never marked as sent in the database. Those QSOs will be re-uploaded on the next export, causing duplicates on eQSL. The original code called eqsl_mark_sent() immediately on each success, avoiding this problem.
Additional Locations (1)
| } else if ($data['COL_SAT_NAME'] == 'SONATE-2') { | ||
| if ($data['COL_MODE'] == 'PKT') { | ||
| $sat_name = 'SONATE-2 APRS'; | ||
| } |
There was a problem hiding this comment.
Unreachable duplicate SONATE-2 satellite branch
Medium Severity
The new else if ($data['COL_SAT_NAME'] == 'SONATE-2') conditional block added at line 1195 is dead code. The original unconditional SONATE-2 branch at line 1185 still exists earlier in the same else if chain and will always match first, so the new PKT-mode-only conditional is never reached. The intended behavior — only applying the APRS name for SONATE-2 in PKT mode (mirroring the BOTAN pattern) — is silently ignored, and all SONATE-2 contacts regardless of mode continue to receive the SONATE-2 APRS satellite name.
| $update_count++; | ||
| } | ||
| } | ||
| } |
There was a problem hiding this comment.
Per-QSO eQSL import status assigned by position not result
Low Severity
After the batch eQSL update, the code back-fills per-QSO status labels by iterating $qsos in array order: the first $updated entries tagged "Queued for batch update" are labeled "Updated", and the next $dupes are labeled "Already received...". However, the batch update in eqsl_update_batch() determines which specific records are updates vs duplicates based on their current DB state, not their position in the array. The positional assignment produces wrong per-QSO status labels in the import results whenever the updated and duplicate records are not ordered at the front of the $qsos array.
| if ($eqsl_qslrdate_index_exists > 0) { | ||
| $sql = "ALTER TABLE $table_name DROP INDEX `idx_eqsl_qslrdate`"; | ||
| $this->db->query($sql); | ||
| } |
There was a problem hiding this comment.
Migration 247 rollback drops index owned by migration 248
Low Severity
The down() method of migration 247 attempts to drop idx_eqsl_qslrdate, but that index is created by migration 248's up(), not by migration 247. If migration 247 is rolled back while migration 248 is still applied (e.g., targeting a specific migration version), it silently removes an index that belongs to 248, leaving the database in an inconsistent state where migration 248 is marked as applied but its index no longer exists.
| $has_more_qsos = false; | ||
| } else { | ||
| $offset += $batch_size; | ||
| log_message('info', 'HRDLog batch upload: Processed ' . $offset . ' QSOs so far for station_id: ' . $station_id); |
There was a problem hiding this comment.
Offset-based batch pagination skips QSOs mid-upload
High Severity
Both mass_upload_qsos functions mark QSOs as sent inside the inner foreach loop (via markqso() in HRDLog and mark_qrz_qsos_sent_batch() every 100 records in QRZ). This removes those rows from the unprocessed result set mid-batch. After each batch finishes, $offset += $batch_size is applied, but since ~1000 rows were removed from the WHERE clause result set, OFFSET 1000 now skips the next 1000 un-processed QSOs. In a large upload, every second batch of 1000 QSOs is silently skipped. The fix is to keep $offset at 0 across all fetches, since the WHERE filter already excludes sent QSOs.
Additional Locations (1)
1 participant
This pull request introduces several performance and usability improvements across multiple controllers, focusing on caching, batch processing, and enhanced support for modern frontend interactions. Key areas addressed include dashboard statistics caching, batch processing for eQSL and HRDLog uploads, optimized LoTW certificate handling with HTMX support, and efficient LoTW ADIF import updates.
Performance and caching enhancements:
Dashboard.phpto reduce database load and improve response times for QSO, country, VUCC, and QSL statistics. [1] [2]Eqsl.php. (F85a32ceL85R162, [1] [2] [3] [4] [5]Hrdlog.phpto process QSOs in batches, preventing memory exhaustion and enabling robust handling of large datasets. [1] [2]LoTW certificate and ADIF import improvements:
Lotw.phpto support HTMX requests for modal-based uploads, returning partial responses for improved frontend UX. [1] [2] [3] [4] [5]Lotw.phpfor batch updating confirmations and gridsquares, reducing database operations and providing detailed status reporting. [1] [2]Migration tracking:
migration.phpto reflect new database changes.Note
Medium Risk
Touches core logbook update paths and adds several schema/index migrations plus new caching invalidation, so mistakes could cause stale dashboard stats or missed/incorrect QSO status updates despite the changes being primarily performance-focused.
Overview
Performance-focused release: Adds file-based caching for dashboard statistics (QSO/country/VUCC/QSL cards) and introduces cache invalidation on QSO create/edit/delete to reduce repeated heavy queries.
Batching + DB optimizations: Converts LoTW and eQSL confirmation imports/exports to batch updates, fetches QRZ/HRDLog uploads in chunks to avoid memory exhaustion, and adds multiple migrations with new indexes (modes, gridsquare prefixes, LoTW/eQSL matching, dashboard aggregates) plus a
2.8.8version-tag migration.UX/HTMX enhancements: Implements HTMX modal LoTW certificate upload with partial table refresh, adds a quick-add/printable Station Diary flow in Notes, and includes small UI tweaks (submenu positioning/styles, safer unsaved-change warnings, minor query/select optimizations and join simplifications).
Written by Cursor Bugbot for commit fe38b96. This will update automatically on new commits. Configure here.