feat: add test coverage reporting with vitest#178
feat: add test coverage reporting with vitest#178schlpbch wants to merge 3 commits intomodelcontextprotocol:mainfrom
Conversation
- Install @vitest/coverage-v8 provider for code coverage analysis - Add coverage configuration to vitest.config.ts with v8 provider - Generate coverage reports in text, JSON, and HTML formats - Exclude node_modules, dist, test files, and examples from coverage - Add npm script 'test:coverage' to run tests with coverage reporting Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
… code - Add 18 tests for src/runner/utils.ts with 100% coverage - Tests for ANSI color code utility functions - Tests for output formatting and alignment - Tests for result directory creation - Add 25 tests for client scenarios - Tests for InitializeScenario, ToolsCallScenario, etc. - Verify scenario interface compliance - Test scenario initialization and methods Coverage improvements: - src/runner/utils.ts: 4.54% → 100% - src/checks/client.ts: 93.75% → 100% - initialize.ts: 11.32% → 84.9% - tools_call.ts: 14.28% → 61.9% - Overall statements: 59.16% → 63.97% - Overall branches: 45.87% → 47.73% Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
Ignore generated coverage reports from vitest to avoid committing build artifacts and maintaining a cleaner repository state. Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
|
@maciej-kisiel could you please review this PR? |
|
Thanks for the contribution! I'm going to close this one though. I don't think code coverage is the right metric for this repo. The conformance suite is a test harness: its job is to exercise SDK implementations against the MCP spec. The coverage number that matters to us is spec coverage (how much of the protocol the scenarios test), not line coverage of the harness itself. We have work to do to track that still. Line coverage here tends to encourage low-signal tests: things like asserting The integration tests we already have ( (Also, small note: this repo uses npm rather than pnpm, so the Appreciate you taking the time to put this together — if you're interested in contributing, adding new conformance scenarios (especially for under-covered parts of the spec) would be really valuable. |
|
Sure, scenarios would be fine as well. I'd argue that also in the scenario space it would be interesting to have a metric in how far the specification is covered by scenarios. |
Summary
Add test coverage reporting infrastructure to the conformance test suite using vitest's built-in coverage support.
@vitest/coverage-v8provider for code coverage analysistest:coveragescript for convenient coverage runscoverage/to.gitignoreTest Plan
pnpm test:coverage- all 84 tests pass with coverage report generated🤖 Generated with Claude Code