Add tests for test-install, test-build, verify-results#7
Open
jnasbyupgrade wants to merge 31 commits intoPostgres-Extensions:masterfrom
Open
Add tests for test-install, test-build, verify-results#7jnasbyupgrade wants to merge 31 commits intoPostgres-Extensions:masterfrom
jnasbyupgrade wants to merge 31 commits intoPostgres-Extensions:masterfrom
Conversation
- Add .claude/settings.json with references to pgxntool and template repos - Add CLAUDE.md documenting the test harness architecture - Include git commit guidelines Co-Authored-By: Claude <noreply@anthropic.com>
Documents current testing weaknesses and proposes migration to BATS framework with semantic validation helpers. Includes: - Assessment of current fragile string-based validation - Analysis of modern testing frameworks - Prioritized recommendations with code examples - 5-week incremental migration timeline - Success metrics Co-Authored-By: Claude <noreply@anthropic.com>
Add .claude/*.local.json to .gitignore to prevent local Claude Code configuration from being committed Co-Authored-By: Claude <noreply@anthropic.com>
- Add .claude/*.local.json to .gitignore - Fix tests/clone to auto-detect current branch instead of hardcoding master Co-Authored-By: Claude <noreply@anthropic.com>
- Add -X flag to psql to ignore user's .psqlrc configuration - Properly quote psql command substitution in Makefile - Use POSIX-compliant = instead of == for test comparison Co-Authored-By: Claude <noreply@anthropic.com>
When pgxntool-test is on a non-master branch, automatically detect and use the corresponding branch from pgxntool if: - pgxntool is on master, OR - pgxntool is on the same branch as pgxntool-test This eliminates the need to manually specify PGXNBRANCH when working on feature branches across both repos. Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Fix critical bugs in test infrastructure: - lib.sh: Fix TEST_DIR path normalization regex (was \\\\? should be ?) This fixes the bug where /private/var paths weren't being normalized to @TEST_DIR@ due to double-slash handling issue - clean-temp.sh: Remove references to undefined $LOG and $TMPDIR variables, use correct $RESULT_DIR instead; use portable shebang - make-temp.sh: Add macOS temp directory handling, better TMPDIR fallback Improve test output normalization (base_result.sed): - Normalize branch names to @Branch@ (handles any branch, not just master) - Normalize user paths to /Users/@user@/ - Normalize asciidoctor paths to /@ASCIIDOC_PATH@ - Normalize pg_regress output to (using postmaster on XXXX) - Handle PostgreSQL version differences (plpgsql, timing, diff formats) - Normalize rsync output variations Update CLAUDE.md: - Add critical rule: never modify expected/ files without explicit approval - Document that make sync-expected must only be run by humans Update expected output files to reflect normalized output format after applying these fixes. Co-Authored-By: Claude <noreply@anthropic.com>
Add BATS (Bash Automated Testing System) as an alternative to string-based output comparison tests. BATS provides semantic assertions and better test isolation. Changes: - Add bats-core as git submodule in test/bats/ - Create tests-bats/ directory for BATS tests - Add initial dist.bats test with semantic assertions for distribution packaging - Update Makefile with test-bats and test-all targets - Add README.md with requirements and usage instructions The BATS tests run after legacy tests complete (make test-all) and use the same test environment. Future work will make BATS tests fully independent. Note: test-bats target currently has issues with file cleanup timing. The zip file created by make dist is not available when BATS tests run. This will be fixed in a follow-up commit. Co-Authored-By: Claude <noreply@anthropic.com>
The BATS tests now run make dist themselves rather than relying on state from legacy tests. This makes the tests more robust and independent. Each test ensures the zip file exists before testing its contents. Co-Authored-By: Claude <noreply@anthropic.com>
Adds 69 BATS tests covering all pgxntool functionality: - Sequential tests (01-05): Build shared state incrementally - Non-sequential tests: Copy sequential state for isolated testing - Run with: make test-bats Sequential tests share environment for speed. Non-sequential tests copy completed sequential environment then test specific features in isolation (make test, make results, documentation generation). Documentation in tests-bats/CLAUDE.md and tests-bats/README.md covers test architecture, development guidelines, and how to add new tests. Co-Authored-By: Claude <noreply@anthropic.com>
Extract assertion functions from helpers.bash into assertions.bash for better code organization and maintainability. This separation makes the codebase more modular and easier to navigate. Changes: - Create tests-bats/assertions.bash with 11 assertion functions - Update helpers.bash to load assertions.bash - Create tests-bats/TODO.md to track future improvements (evaluate BATS standard libraries, CI/CD integration, ShellCheck linting) - Add .DS_Store to .gitignore All 69 BATS tests pass after refactoring. Co-Authored-By: Claude <noreply@anthropic.com>
Consolidate test infrastructure by moving BATS tests to tests/ and removing legacy string-comparison tests. The BATS system uses semantic assertions that test specific behaviors rather than comparing text output, making tests more maintainable and less fragile. Key changes: - Move tests from tests-bats/ to tests/ (consolidate into single directory) - Rename 01-clone.bats to foundation.bats (better reflects its role) - Renumber sequential tests: 03-meta→01-meta, 04-dist→02-dist, 05-setup-final→03-setup-final - Remove 02-setup.bats (functionality integrated into foundation) - Delete legacy test scripts (tests/clone, tests/setup, tests/meta, etc.) - Remove legacy infrastructure (make-temp.sh, clean-temp.sh, base_result.sed, expected/*.out) - Update Makefile to run BATS tests with pattern `[0-9][0-9]-*.bats` - Add distribution validation with dual approach: - tests/dist-expected-files.txt: Exact manifest (primary validation) - tests/dist-files.bash: Pattern validation (safety net) - tests/test-dist-clean.bats: Test dist from clean foundation - Enhance 02-dist.bats to test workflow (make → make html → make dist) - Update documentation (CLAUDE.md, README.md) to use pattern-based descriptions rather than listing specific test files - Add Makefile comment explaining why all sequential tests are explicitly listed (BATS only outputs TAP results for directly invoked test files) Test status: 46 of 63 tests pass. Failures in test-doc, test-make-results, and test-make-test are pre-existing issues unrelated to this refactor. Co-Authored-By: Claude <noreply@anthropic.com>
Enhance test infrastructure to detect and handle re-running completed tests: - Add pollution detection in helpers.bash to catch test re-runs When a test runs that already completed in the same environment, environment is cleaned and rebuilt to prevent side effect conflicts (e.g., git branches, modified state) - Add `make test-recursion` target to validate recursion/pollution detection Runs one independent test with clean environments to exercise the prerequisite and pollution detection systems - `make test` auto-detects dirty repo and runs test-recursion first Uses make's native conditional syntax (`ifneq`) instead of shell. If test infrastructure code has uncommitted changes, validates that recursion works before running full test suite (fail fast on broken infrastructure) - Document that only 01-meta copies foundation to sequential environment First test to use `TEST_REPO`; added comments with cross-references - Fix 01-meta and 02-dist to dynamically extract version from META.json Tests were hardcoding version "0.1.0" but 01-meta changes it to "0.1.1" Now extract both name and version dynamically to handle test-induced changes - Update commit.md to be stricter about failing tests Make it clear there's no such thing as an "acceptable" failing test Co-Authored-By: Claude <noreply@anthropic.com>
Improve commit safety and consistency: - Add mandatory `git status` check after staging but before commit Verifies correct files are staged (all files vs subset) and allows user to catch mistakes before committing - Add explicit instruction to wrap code references in backticks Examples: `helpers.bash`, `make test-recursion`, `TEST_REPO` Prevents markdown parsing issues and improves clarity - Add backticks consistently throughout commit.md Applied to git commands, make targets, filenames, tool names - Add `TodoWrite`/`Task` restriction to commit workflow Prevents using these tools during commit process Co-Authored-By: Claude <noreply@anthropic.com>
Replace `.claude/commands/commit.md` with symlink to `../pgxntool/.claude/commands/commit.md` to avoid duplicating commit workflow between repos. Add startup verification to `CLAUDE.md` instructing to verify symlink is valid on every session start (both repos are always checked out together). Co-Authored-By: Claude <noreply@anthropic.com>
- Add status assertion functions to `assertions.bash`: `assert_success()`, `assert_failure()`, `assert_failure_with_status()`, `assert_success_with_output()`, `assert_failure_with_output()`, and `assert_files_exist()`/`assert_files_not_exist()` for array-based file checks - Add `test-gitattributes.bats` to test `.gitattributes` behavior with `make dist` - Add `test-make-results-source-files.bats` to test `make results` and `make clean` behavior with `.source` files - Refactor existing tests to use new assertion functions instead of raw `[ "$status" -eq 0 ]` checks - Update `CLAUDE.md` and `tests/CLAUDE.md` with rules about never ignoring result codes and avoiding `skip` unless explicitly necessary - Update `foundation.bats` to create and commit `.gitattributes` for export-ignore support - Update `dist-expected-files.txt` to include `pgxntool/make_results.sh` - Add `.gitattributes` with export-ignore directives Changes in pgxntool/: - Add `make_results.sh` script to handle copying results while respecting `output/*.source` files as source of truth - Update `base.mk` to properly handle ephemeral files from `.source` files, create `test/results/` directory automatically, and add validation in `dist-only` target to ensure `.gitattributes` is committed Co-Authored-By: Claude <noreply@anthropic.com>
Add `check_postgres_available()` and `skip_if_no_postgres()` helpers to `helpers.bash` for detecting PostgreSQL availability. The helper checks for `pg_config`, `psql`, and attempts a connection using plain `psql` (assuming user has configured PGHOST, PGPORT, PGUSER, PGDATABASE, etc.). Results are cached to avoid repeated checks. Update PostgreSQL-dependent tests in `test-make-results.bats` and `test-make-results-source-files.bats` to use `skip_if_no_postgres` so they gracefully skip when PostgreSQL is not available instead of failing. Add comprehensive test suite for pg_tle support in `test-pgtle.bats` (see pgxntool commit for pg_tle implementation). Add PostgreSQL configuration documentation to `README.md` explaining required environment variables. Update test agent (`.claude/agents/test.md`) to warn about skipped tests and document PostgreSQL requirements. Add pg_tle expert agent (`.claude/agents/pgtle.md`). Update commit command to handle multi-repo commits and emphasize including all new files. Fix test name in `03-setup-final.bats` teardown and update `dist-expected-files.txt` to include version-specific SQL files. Co-Authored-By: Claude <noreply@anthropic.com>
- Change `.claude/commands/commit.md` from regular file to symlink pointing to `../../../pgxntool/.claude/commands/commit.md` - Reorganize test directory structure from flat `tests/` to hierarchical `test/`: - `test/lib/` - Shared test infrastructure (helpers, assertions, foundation) - `test/sequential/` - Sequential tests that build on each other (00-03) - `test/standard/` - Independent tests with isolated environments - Ensures both repos use same commit workflow with mandatory 3-repo check This change maintains the shared commit command pattern and improves test organization by clearly separating infrastructure, sequential, and independent tests. Co-Authored-By: Claude <noreply@anthropic.com>
- Add `pg_tle` subagent to `.claude/agents/pgtle.md` for `pg_tle` development support - Update test agent configuration in `.claude/agents/test.md` - Add `make test-pgtle` target to Makefile for `pg_tle`-specific tests - Update test infrastructure to support `pg_tle` validation - Update `test/lib/helpers.bash` with PostgreSQL version detection functions - Update `test/lib/foundation.bats` to detect and export `pg_tle` support - Update test files to use new `test/` directory structure - Add new test files: `test/sequential/04-pgtle.bats`, `test/standard/pgtle-install.bats` - Remove obsolete `tests/TODO.md` and `tests/test-pgtle.bats` Co-Authored-By: Claude <noreply@anthropic.com>
Add comprehensive tests for `test-build`, `test/install`, and `verify-results` optional features. Each test validates configuration, auto-detection, and behavior. Improve `foundation.bats` to handle git worktrees and dirty repos: - Detect worktrees (`.git` can be file or directory) - Use rsync when `TEST_REPO` or source repo is dirty (git subtree requires clean tree) - More informative error messages Refactor `test-make-results` tests to directly test `make_results.sh` behavior and disable `verify-results` when tests are intentionally failing. Add `init-bats` target to `Makefile` to auto-initialize BATS submodule, and add individual test targets for new tests. Consolidate commit commands: merge all three repos' commit.md into pgxntool-test, update pgxntool to reference it. Delete `Test-Improvement.md` (no longer needed), update BATS submodule. Co-Authored-By: Claude <noreply@anthropic.com>
…ucture Add tests for pgxntool commit c7b5705 (pg_tle 1.4.0-1.5.0 version range): - Update test infrastructure to handle new version boundaries - Verify three version ranges: 1.0.0-1.4.0, 1.4.0-1.5.0, 1.5.0+ Add safe directory navigation helpers to prevent silent `cd` failures that were causing confusing test errors: - Add `assert_cd()` function that errors clearly if `cd` fails - Add `cd_test_env()` convenience function that changes to `TEST_REPO` - Update `foundation.bats` `setup()` to use `assert_cd()` explicitly Refactor `foundation.bats` to use `git init` instead of `git clone`: - Create fresh repository with `git init` - Copy only files from template's `t/` directory to root - Commit extension files before adding pgxntool (matches real workflow) - Document why fake remote is needed (for `make dist` → `tag` target) - Remove unnecessary hidden files copy command Fix test infrastructure for reorganized test directory structure: - Update paths to use `test/lib/foundation.bats` and `test/sequential/` - Fix `is_clean_state()` to look in correct directory for test ordering - Add special handling for foundation prerequisite (lives in `test/lib/`) - Update foundation age warning threshold (10s → 60s for better UX) Remove `.gitattributes` from pgxntool-test (belongs only in pgxntool). Update documentation: - Add guidance about using subagents and checking for new pg_tle versions - Document `.gitattributes` placement rule - Clarify that template files come from `t/` directory only - Update test workflow description (git init vs clone) Co-Authored-By: Claude <noreply@anthropic.com>
Consolidate pgxntool-test-template into pgxntool-test:
- Move template files from pgxntool-test-template/t/ to template/
- Update `TEST_TEMPLATE` default to `${TOPDIR}/template`
- Update `foundation.bats` to copy from `${TEST_TEMPLATE}/`
- Remove template directory from `.claude/settings.json`
Update all references for two-repository pattern:
- Update `CLAUDE.md` to remove three-repository references
- Update `.claude/agents/test.md` for new pattern
- Update `.claude/commands/worktree.md` to handle two repos
- Update `bin/create-worktree.sh` to remove template repo
Related to pgxntool commit 56935ca (two-repo pattern):
- Convert commit command to symlink (points to this file)
- Remove template references from documentation
Simplify commit workflow in `.claude/commands/commit.md`:
- Draft both commit messages upfront with hash placeholders
- Commit pgxntool first (no placeholder needed)
- Commit pgxntool-test with pgxntool hash injected
- Remove amend step (amending changes commit hash)
- pgxntool messages mention RELEVANT test changes (filtered)
- pgxntool-test messages summarize pgxntool changes
Co-Authored-By: Claude <noreply@anthropic.com>
After pg_tle branch merge, three test files in test/standard/ were using `load helpers` which doesn't work from their new location. Updated to `load ../lib/helpers` to correctly reference the helpers from the lib/ directory. Files updated: - test/standard/test-test-build.bats - test/standard/test-test-install.bats - test/standard/test-verify-results.bats Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
This commit resolves a race condition where `git subtree add` would fail with "working tree has modifications" due to filesystem timestamp granularity causing stale git index cache entries. Test infrastructure improvements: - Consolidate git status checks into single location - Streamline `test-extra` to run full test suite plus extra tests - Remove dangerous `rm -rf .envs` from allowed commands Test agent improvements: - Trim instructions from 1432 to 292 lines (80% reduction) - Add prominent error handling rules section with clear examples - Reference `tests/CLAUDE.md` for detailed guidance instead of duplicating Changes only in pgxntool-test. No related changes in pgxntool. Co-Authored-By: Claude <noreply@anthropic.com>
Fix foundation test timestamp issue and improve rsync handling
Move test environment directory from project root to test/ subdirectory to align directory structure with master branch: - Update helpers.bash path references - Update Makefile clean-envs target - Update .gitignore entries - Update all documentation references This matches master's structure where test environments live under test/.
Merges master's test infrastructure while preserving test-build tests. Key fixes: - Fixed worktree detection in foundation.bats (use -e instead of -d for .git) - Fixed worktree detection in helpers.bash build_test_repo_from_template - Fixed make-results.bats to bypass verify-results for intentional mismatch test - Fixed make-results-source-files.bats similarly Merged from master: - control-errors.bats tests - multi-extension.bats tests - update-setup-files.bats tests - template-multi-extension/ directory - Various documentation updates Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Tests for pgxntool commit 91da35b: - Schedule-based test/install replacing broken separate-invocation approach - test-build, verify-results, .asc extension support Add `test-install-persistence.bats` validating install state persists into the main test suite (the core contract of the schedule-based approach). - Rework `test-test-install.bats` for single install schedule (no test/sql/schedule) - Add template install marker files (`create_install_marker.sql`, `verify_install_marker.sql`) - Update `dist-expected-files.txt` for new template test files - Fix `test-test-build.bats` expected output paths (`test/build/expected/` not `test/expected/`) - Fix `doc.bats` to include `.asc` extension in doc file detection - Add `html` make target aliasing `readme` - Convert `/commit` command to `.claude/skills/commit/` with preprocessing script - Update `CLAUDE.md` to document skills alongside commands Co-Authored-By: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
test/install/files persists into the main test suite.ascextension in asciidoc file detection/commitcommand to.claude/skills/commit/with preprocessing scriptdist-expected-files.txtfor new template filesCompanion PR: Postgres-Extensions/pgxntool (same branch)
Test plan
🤖 Generated with Claude Code