Skip to content

feat: add test coverage reporting with vitest#178

Closed
schlpbch wants to merge 3 commits intomodelcontextprotocol:mainfrom
schlpbch:feature/add-coverage-reporting
Closed

feat: add test coverage reporting with vitest#178
schlpbch wants to merge 3 commits intomodelcontextprotocol:mainfrom
schlpbch:feature/add-coverage-reporting

Conversation

@schlpbch
Copy link
Copy Markdown

@schlpbch schlpbch commented Mar 3, 2026

Summary

Add test coverage reporting infrastructure to the conformance test suite using vitest's built-in coverage support.

  • Install @vitest/coverage-v8 provider for code coverage analysis
  • Configure vitest with coverage settings (text, JSON, HTML reporters)
  • Add test:coverage script for convenient coverage runs
  • Add coverage/ to .gitignore
  • Current coverage: 59.16% statements, 45.87% branches

Test Plan

  • Run pnpm test:coverage - all 84 tests pass with coverage report generated
  • Verify coverage reports generated in text, JSON, and HTML formats
  • Check that coverage excludes node_modules, dist, test files, and examples

🤖 Generated with Claude Code

schlpbch and others added 3 commits March 3, 2026 23:03
- Install @vitest/coverage-v8 provider for code coverage analysis
- Add coverage configuration to vitest.config.ts with v8 provider
- Generate coverage reports in text, JSON, and HTML formats
- Exclude node_modules, dist, test files, and examples from coverage
- Add npm script 'test:coverage' to run tests with coverage reporting

Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
… code

- Add 18 tests for src/runner/utils.ts with 100% coverage
  - Tests for ANSI color code utility functions
  - Tests for output formatting and alignment
  - Tests for result directory creation
- Add 25 tests for client scenarios
  - Tests for InitializeScenario, ToolsCallScenario, etc.
  - Verify scenario interface compliance
  - Test scenario initialization and methods

Coverage improvements:
- src/runner/utils.ts: 4.54% → 100%
- src/checks/client.ts: 93.75% → 100%
- initialize.ts: 11.32% → 84.9%
- tools_call.ts: 14.28% → 61.9%
- Overall statements: 59.16% → 63.97%
- Overall branches: 45.87% → 47.73%

Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
Ignore generated coverage reports from vitest to avoid committing
build artifacts and maintaining a cleaner repository state.

Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
@schlpbch
Copy link
Copy Markdown
Author

schlpbch commented Mar 3, 2026

@maciej-kisiel could you please review this PR?

@pcarleton
Copy link
Copy Markdown
Member

Thanks for the contribution! I'm going to close this one though. I don't think code coverage is the right metric for this repo.

The conformance suite is a test harness: its job is to exercise SDK implementations against the MCP spec. The coverage number that matters to us is spec coverage (how much of the protocol the scenarios test), not line coverage of the harness itself. We have work to do to track that still.

Line coverage here tends to encourage low-signal tests: things like asserting scenario.name === 'initialize' or that a color-mapping function returns the right color for each enum value. Those pad the number but don't catch real bugs, and they add maintenance burden every time we tweak a string.

The integration tests we already have (all-scenarios.test.ts, checks.test.ts) exercise the scenarios end-to-end, which is the granularity that actually catches regressions in this kind of project.

(Also, small note: this repo uses npm rather than pnpm, so the pnpm-lock.yaml would conflict with the existing package-lock.json.)

Appreciate you taking the time to put this together — if you're interested in contributing, adding new conformance scenarios (especially for under-covered parts of the spec) would be really valuable.

@schlpbch
Copy link
Copy Markdown
Author

Sure, scenarios would be fine as well.

I'd argue that also in the scenario space it would be interesting to have a metric in how far the specification is covered by scenarios.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants