docs: fix SDK API references to match actual v0.2.8 code#146
Open
vamsimanohar wants to merge 3 commits intoopensearch-project:mainfrom
Open
docs: fix SDK API references to match actual v0.2.8 code#146vamsimanohar wants to merge 3 commits intoopensearch-project:mainfrom
vamsimanohar wants to merge 3 commits intoopensearch-project:mainfrom
Conversation
vamsimanohar
requested review from
anirudha,
goyamegh,
kylehounslow and
ps48
as code owners
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #146 +/- ##
=======================================
Coverage 18.51% 18.51%
=======================================
Files 3 3
Lines 54 54
Branches 19 18 -1
=======================================
Hits 10 10
Misses 44 44 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
Cross-checked all SDK documentation against the actual opensearch-genai-observability-sdk-py v0.2.8 source code and fixed multiple discrepancies: - Rename `Experiment` to `Benchmark` throughout (the SDK class is `Benchmark`, not `Experiment`) - Fix `ExperimentSummary` -> `BenchmarkSummary`, `CaseResult` -> `TestCaseResult` in result type references - Add missing exports to API overview table: `BenchmarkResult`, `BenchmarkSummary`, `TestCaseResult`, `ScoreSummary` - Remove non-existent individual extras (`[cohere]`, `[mistral]`, `[groq]`, `[ollama]`) from auto-instrumentation table - these are only available via `[otel-instrumentors]` bundle - Add missing `[google]` extra to installation section - Add missing env vars `OTEL_EXPORTER_OTLP_TRACES_PROTOCOL` and `OTEL_EXPORTER_OTLP_PROTOCOL` to environment variables table - Show actual default endpoint URL instead of "Data Prepper default" - Add `Benchmark` constructor parameters documentation - Add manual (non-context-manager) usage example for `Benchmark` - Fix evaluation-integrations.mdx variable names for consistency Signed-off-by: Vamsi Manohar <reddyvam@amazon.com>
vamsimanohar
force-pushed
the
docs/fix-sdk-api-references
branch
from
0395cd0 to
0da8a02
Compare
evaluation-integrations.mdx: - Add version caveat Aside for third-party framework APIs - Add install commands (pip install) for each section - DeepEval: add tip about deepeval.evaluate() alternative - RAGAS: add version note about v0.2+ API changes, fix iteration to use df.iterrows() instead of enumerate(itertuples()) - MLflow: fix to use DataFrame input, add mlflow.start_run() context, add version note about model_type deprecation in 2.12+ - pytest: move register() to conftest.py session-scoped fixture to avoid re-initialization across test files, add install command, rename scorer to avoid shadowing builtin - Fix related links: restore "Experiments" in link text to match actual page title at /docs/agent-health/evaluations/experiments/ integrations.mdx: - CrewAI: suggest [otel-instrumentors] extra which includes CrewAI auto-instrumentation, add note about what it provides - OpenAI Agents SDK: add import comment clarifying package name Signed-off-by: Vamsi Manohar <reddyvam@amazon.com>
kylehounslow
approved these changes
Mar 31, 2026
docs/starlight-docs/src/content/docs/ai-observability/evaluation-integrations.mdx
Outdated
Show resolved
Hide resolved
Address review comment — keep related links consistent with evaluation.mdx. Signed-off-by: Vamshi Vijay Nakkirtha <vamsimanohar@gmail.com> Signed-off-by: Vamsi Manohar <reddyvam@amazon.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Cross-checked all SDK documentation against the actual
opensearch-genai-observability-sdk-pyv0.2.8 source code and fixed multiple discrepancies:Experiment->Benchmark: The SDK class isBenchmark, notExperiment. Updated all references across 5 doc pages (python.mdx, evaluation.mdx, evaluation-integrations.mdx, getting-started.mdx, index.md)ExperimentSummary->BenchmarkSummary,CaseResult->TestCaseResultBenchmarkResult,BenchmarkSummary,TestCaseResult,ScoreSummary[cohere],[mistral],[groq],[ollama]) that are only available via[otel-instrumentors]bundle. Added missing[google]extra.OTEL_EXPORTER_OTLP_TRACES_PROTOCOLandOTEL_EXPORTER_OTLP_PROTOCOLBenchmarkdocs: Added constructor parameters table, manual (non-context-manager) usage exampleVerification
opensearch-genai-observability-sdk-pyv0.2.8pyproject.tomlfor available extrasTest plan
npm run buildindocs/starlight-docs/passes (validates all internal links)