feat(test): Compose Preview Screenshot Testing with 164 tests#5098
Draft
jamesarich wants to merge 30 commits intomeshtastic:mainfrom
Draft
feat(test): Compose Preview Screenshot Testing with 164 tests#5098jamesarich wants to merge 30 commits intomeshtastic:mainfrom
jamesarich wants to merge 30 commits intomeshtastic:mainfrom
Conversation
github-actions
bot
added
enhancement
This commit implements comprehensive screenshot testing infrastructure for Meshtastic-Android: **Configuration (Phase 1)** - Enable experimental screenshot test flag in gradle.properties - Add screenshot plugin (com.android.compose.screenshot v0.0.1-alpha14) to libs.versions.toml - Configure app/build.gradle.kts with screenshot dependencies and experimental flag - Create ScreenshotTestingConventionPlugin for build logic reusability **CMP UI Previews (Phase 2)** - Create preview composables in core/ui/src/commonMain/kotlin/org/meshtastic/core/ui/preview/ - BasicComponentPreviews.kt: Button variants, text styles, icon buttons - ExtendedComponentPreviews.kt: Cards, input fields, checkboxes, dialogs, chips - MultiPreview annotation for automatic light/dark theme variations **Screenshot Tests (Phase 3)** - Create CoreComponentScreenshotTests.kt with 8 screenshot test methods - Each test marked with @previewTest annotation - Tests validate appearance across device configurations and themes **Documentation & Workflow** - SCREENSHOT_TESTING.md: Comprehensive 400+ line guide (quick start, patterns, best practices) - SCREENSHOT_TESTING_QUICK_REFERENCE.md: Quick command reference and troubleshooting - SCREENSHOT_TESTING_SETUP_COMPLETE.md: Implementation summary and next steps - .agent_plans/screenshot_testing_setup.md: Technical architecture decisions **Available Commands** - ./gradlew updateGoogleDebugScreenshotTest: Generate reference images - ./gradlew validateGoogleDebugScreenshotTest: Run validation and generate HTML report - IDE integration with Android Studio for visual diff inspection This enables automated visual regression testing for CMP UI components while maintaining the architectural boundaries between business logic (commonMain) and platform-specific testing. Closes #NONE
- Fix typo: headlinesMedium -> headlineMedium in BasicComponentPreviews.kt - Fix wrong package: material3.Row -> foundation.layout.Row (Row lives in foundation.layout) - Fix architecture violation: move preview files from core/ui/commonMain to app/src/screenshotTest (android-specific @Preview annotations with uiMode must not be in commonMain per AGENTS.md KMP rules) - Fix convention plugin: apply screenshot plugin directly in app/build.gradle.kts, convention plugin now validates plugin is applied and adds config/deps only - Remove unused import (configure) in ScreenshotTesting.kt - Update docs to reflect correct file locations
…i and feature modules Add comprehensive Compose Preview screenshot tests covering components from feature/node, feature/messaging, feature/connections, and core/ui. Fix prior session bugs: replace unavailable Icons.Filled.* with MeshtasticIcons, add missing @composable annotations to test methods, and fix MeshtasticTheme -> AppTheme. Consolidate redundant standalone docs into .skills/testing-ci/SKILL.md.
…re components Add previews and screenshot tests for: - NodeDataInfo: DistanceInfo, LastHeardInfo, HopsInfo, ChannelInfo, IconInfo, SnrRssi - Utility: TransportIcon, CopyIconButton, BluetoothSignalInfo, NodeKeyStatusIcon, AutoLinkText, SlidingSelector, InsetDivider, PlaceholderScreen, IAQScale, ConnectionsNavIcon - Feature: LogLine, NodeFilterTextField, ExpressiveSection, ThemePickerDialog, HomoglyphSetting, NotificationSection, WarningDialog, NodeActionButton, LoadingOverlay, MapReportingPreference Brings total screenshot test count to 80.
…dule feature components Add previews and screenshot tests for: - Core/UI: ElevationInfo, SignedIntegerEditTextPreference, AdaptiveTwoPane, BottomSheetDialog, MenuFAB (collapsed + expanded), SecurityIcon (all states) - Feature/Settings: DebugCustomFilterInput, RouterRoleConfirmationDialog - Feature/Node: FirmwareReleaseSheetContent - Feature/Messaging: QuickChatRow (enabled + disabled) - Feature/Connections: DeviceListItem (TCP, Mock, connecting states) Brings total screenshot test count to 91.
…age, add imageDifferenceThreshold - Replace @Preview(showBackground=true) with @MultiPreview on all 91 test methods across 15 screenshot test files, doubling coverage to light+dark theme variants per Google's recommended multi-preview pattern. - Add imageDifferenceThreshold (0.05%) to testOptions.screenshotTests in app/build.gradle.kts to prevent false positives from font rendering differences across machines. - Fix SKILL.md docs: MeshtasticTheme -> AppTheme, @Preview -> @MultiPreview in the 'Writing a Preview + Test' instructions.
…oogleDebug Generated via updateGoogleDebugScreenshotTest. Covers all 91 @previewTest methods with both Light and Dark theme variants from @MultiPreview. These baselines enable validateGoogleDebugScreenshotTest to detect visual regressions.
…of Random The minnieMouse node from NodePreviewParameterProvider uses Random.nextInt() for its num field, causing different chip colors on each render and failing screenshot validation. Use a fixed num (2024) to make the preview deterministic. Updates the 2 affected reference images (light + dark).
Add :app:validateGoogleDebugScreenshotTest to the shard-app matrix in reusable-check.yml so screenshot reference images are validated on every PR, catching visual regressions before merge.
…RegenerateDialog Add 2 new preview composables and screenshot tests covering: - DeviceListSection with TCP and Mock device entries - PrivateKeyRegenerateDialog showing key regeneration confirmation Total: 93 @previewTest methods, 186 reference images (light + dark).
Leverage Res.string.* accessors and direct domain model construction to unblock 18 components that were incorrectly marked as infeasible. New preview coverage: - Dialogs: ClickableTextField, MeshtasticResourceDialog, MeshtasticTextDialog, DeliveryInfo (success + error), ShutdownConfirmationDialog - Features: MessageActionsContent, MessageTopBar, Legend, LegendInfoDialog, EditDeviceProfileDialog, EditChannelDialog - Node/Contact: SignalInfo (detail), NodeDetailsSection, ContactItem, CurrentlyConnectedInfo - Messages: MessageItem, ReplySnippet, MessageStatusDialog Total: 112 @previewTest methods, 224 reference images (light + dark).
…s for CI compatibility Raise threshold from 0.05% to 2% to accommodate cross-platform font/emoji rendering differences (macOS vs Linux CI runners). Replace emoji unicode literals in MessageActionsContent preview with ASCII text to avoid platform-dependent glyph rendering.
… device form factor) Full cross-product coverage: - Theme: light + dark - Font scale: 1x + 2x (large font) - Device: phone + foldable (673dp) + tablet (1280dp) 112 test methods x 12 variants = 1,344 reference images
…eenshot tests - Screenshot shard now auto-records and pushes updated references on same-repo PRs (NiA pattern via stefanzweifel/git-auto-commit-action) - Fork PRs hard-fail with instructions to update locally - Merge queue / main hard-fail if references are stale - Separate 'screenshot-diffs' artifact for easier debugging - Skip Codecov upload for screenshot shard (no JUnit/Kover output) - Updated SKILL.md with @MultiPreview coverage, auto-update workflow, and revised debugging instructions
jamesarich
force-pushed
the
feat/screenshot-testing
branch
from
e2cb5c0 to
47a98d2
Compare
The reusable workflow cannot request more permissions than the caller grants. pull-request.yml has contents:read, so requesting contents:write caused a startup_failure on all jobs. The auto-commit step will silently no-op without write access. To enable it, the caller workflow must be updated to grant contents:write.
|
The ref override (github.head_ref) caused checkout to resolve against the base repo where the fork branch doesn't exist, breaking the composite action path. Removed auto-record/auto-commit steps since they require contents:write + ref override which aren't feasible for fork PRs. Retained: fork protection, diff artifact upload, failure summary.
…ovision, node detail, messaging, and settings - Fix NodePreviewParameterProvider determinism (replace Random/currentTime with fixed constants) - Add RTL locale variants (ar) to @MultiPreview annotation (now 14 variants per test) - Wire up Tier 1 public previews: AlertPreviews, SignalInfoSimple, LazyColumnDragAndDrop - Add wifi-provision screenshot tests (19 previews covering all provisioning phases) - Add node detail screenshot tests (10 previews: DeviceActions, TelemetricActions, NodeDetailContent) - Add messaging feature screenshot tests (5 previews: QuickChat, Reactions, MessageInput) - Add settings section screenshot tests (4 previews: AppInfo, Appearance, Persistence, Privacy) - Make feature module preview functions public for cross-module screenshot test access - Regenerate all 2,198 reference images (157 tests x 14 variants)
…PositionCard, NodeContextMenu, PacketResponseStateDialog, and DebugActiveFilters - QrDialog: channel sharing dialog with null painter - NotesSection: favorite node with/without notes - PositionCard: metric and imperial units, selected/unselected states - NodeContextMenu: expanded menu for regular and favorite nodes - PacketResponseStateDialog: loading, success, and error states - DebugActiveFilters: AND and OR filter modes with active filters - Total: 169 @previewTest methods, 2,366 reference images across 26 test files
…th nowSeconds in preview
Remove PositionCard (collapses to 1x1), NodeContextMenu (DropdownMenu Popup not captured), and PrivacySection (Android framework deps) preview tests and their 70 reference images. Revert PrivacySectionPreview back to private. Update SKILL.md metrics to 164 tests / 2,296 images.
Move the fork-fail and same-repo-fail steps (exit 1) after the artifact upload and GITHUB_STEP_SUMMARY steps. Previously the hard-fail ran first, causing the job to enter failure state and skip the summary step which lacked an always() guard.
Extract screenshot validation from the reusable-check.yml test-shards matrix into a dedicated screenshot-tests.yml workflow. This eliminates the ~15min lint-check dependency so screenshots start immediately on push, running fully in parallel with the main CI pipeline. Key changes: - New screenshot-tests.yml with dorny/paths-filter to skip when no UI files changed, and a status gate job for required check compatibility - Remove shard-screenshot from reusable-check.yml (3 shards remain) - Update SKILL.md CI documentation to reflect new architecture
# Conflicts: # gradle/libs.versions.toml
The 30-minute timeout was too tight for the full app compilation + 2,296 image render + comparison cycle on standard CI runners.
# Conflicts: # .skills/testing-ci/SKILL.md
…dark phone) Trimmed @MultiPreview from 14 to 2 variants to keep CI under 45min. Updated SKILL.md metrics to reflect new image count.
jamesarich
changed the title
- Add .github/scripts/screenshot-summary.sh: generates JPEG thumbnail grid for GitHub Step Summary (328 thumbs, ~280KB) and self-contained HTML gallery artifact (~31MB) with dark theme, search, filtering, lightbox, and side-by-side expected/actual/diff for failures - Update screenshot-tests.yml: replace inline summary with external script, add ImageMagick install step, upload gallery artifact (14-day retention), update diffs artifact to include rendered images - Add .github/scripts/** to paths-filter so script changes trigger CI - Apply spotless formatting to BasicComponentPreviews.kt KDoc
Replace with a simple text-based step summary that lists failed image names and links to the screenshot-diffs artifact. Remove ImageMagick dependency and gallery artifact upload.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
com.android.compose.screenshotplugin (v0.0.1-alpha14) with a build-logic convention plugin and Gradle configuration@PreviewTestmethods across 26 test classes and 27 preview files, generating 328 reference images via@MultiPreview(2 variants: Light Phone + Dark Phone)screenshot-tests.ymlCI workflow with path filtering for automatic visual regression detection on PRsNodePreviewParameterProvider) for reproducible renders.skills/testing-ci/SKILL.mdsection 5@MultiPreview Coverage (2 variants per test)
Component Coverage
Core UI (
core:ui)Buttons, text, icons, cards, inputs, checkboxes, chips, alert dialogs, preferences (8 types), battery info, signal info, satellite count, node chip, titled card, list items, telemetry displays (temperature, humidity, device, LoRa, IAQ), channel item, placeholder, security icons, elevation info, adaptive two-pane, bottom sheet dialog, menu FAB, auto-link text, sliding selector, copy icon button, connections nav icon, inset divider, and more
Feature Modules
feature:node— InfoCard, NodeStatusIcons, TimeFrameSelector, metrics, cooldown buttons, FirmwareReleaseSheetContent, NodeDetailsSection, Legend, LegendInfoDialog, NotesSection, SignalInfoDetail, NodeDataInfo (distance, lastHeard, hops, channel, SNR/RSSI)feature:messaging— MessageStatusIcon, DeleteMessageDialog, UnreadMessagesDivider, ActionModeTopBar, MessageActionsContent, MessageTopBar, MessageItem, ReplySnippet, MessageStatusDialog, DeliveryInfo, ContactItem, QuickChatRow, QuickChatItem, EditQuickChatDialog, ReactionItem, ReactionRow, MessageInputfeature:connections— EmptyStateContent, ConnectingDeviceInfo, ConnectionsSegmentedBar, DeviceListItem, DeviceListSection, CurrentlyConnectedInfofeature:settings— AppInfoSection, AppearanceSection, PersistenceSection, DebugCustomFilterInput, DebugActiveFilters, RouterRoleConfirmationDialog, ThemePickerDialog, HomoglyphSetting, NotificationSection, WarningDialog, ShutdownConfirmationDialog, EditDeviceProfileDialog, EditChannelDialog, PrivateKeyRegenerateDialog, PacketResponseStateDialog (loading/success/error)feature:wifi-provision— All 19 Wi-Fi provisioning UI states (scanning, device found, networks, provisioning, success/fail)Technical Details
app/src/screenshotTest/(Android source set) using AndroidX@PreviewannotationsAppThemewith@MultiPreviewannotation for 2-variant coverage (light/dark phone)imageDifferenceThresholdset to0.02f(2%) to absorb macOS/Linux font rendering differencesandroid.experimental.enableScreenshotTest=truein bothgradle.propertiesand convention plugin (both required)Node,Message,Contact, proto types) constructed inline — no test framework dependencies neededapp/src/screenshotTestGoogleDebug/reference/CI Architecture
screenshot-tests.yml) runs independently from the main CI pipeline — no lint-check dependencydorny/paths-filterto skip when no UI files changed; always runs onmerge_groupeventsscreenshot-diffsartifact and writesGITHUB_STEP_SUMMARYwith failed test namesKnown Limitations
PositionCard,NodeContextMenu, andPrivacySectionare excluded — they use framework features (Popup windows, permissions, LocalContext) incompatible with the screenshot test environmentValidation
spotlessCheck— PASSEDupdateGoogleDebugScreenshotTest— PASSED (328/328 images generated)