LLM benchmarking framework with SystemDS, vLLM & OpenAI backends - LDE Project#2431

Open

kubraaksux wants to merge 76 commits intoapache:mainfrom

kubraaksux:llm-benchmark

Commits on Jan 19, 2026

Add LLM benchmarking framework to staging
kubraaksux
committed

Commits on Feb 15, 2026

Commits on Feb 16, 2026

Run SystemDS with Qwen 3B and Mistral 7B for direct vLLM comparison
kubraaksux
committed
Remove deprecated trust_remote_code from dataset loaders
kubraaksux
committed
Update README with actual benchmark results and SystemDS backend docs
kubraaksux
committed
Add gitignore rules for .env files, meeting notes, and local tool configs
kubraaksux
committed
Redesign benchmark report for clarity and minimal UI
kubraaksux
committed
Update benchmark runner with systemds backend and GPU comparison mode
kubraaksux
committed
Clean up report: remove dead code, unused CSS, and hardcoded model name
kubraaksux
committed
Add presentation-friendly summary tables to benchmark report
kubraaksux
committed
Increase worker startup timeout to 300s for larger models
kubraaksux
committed
Revert accidental changes to MatrixBlockDictionary.java
kubraaksux
committed
Add concurrency=4 benchmark results and fix json_extraction type check
kubraaksux
committed
Revert accidental changes to MatrixBlockDictionary.java
kubraaksux
committed
Regenerate benchmark report with SystemDS results
kubraaksux
committed
Add GPU batching to SystemDS JMLC backend with benchmark results
kubraaksux
committed
Add GPU batching support to JMLC LLM inference
kubraaksux
committed
Keep both sequential and batched inference modes for reproducibility
kubraaksux
committed
Keep both sequential and batched inference modes in PreparedScript
kubraaksux
committed
Add gitignore rules for .env files, meeting notes, and local tool config
kubraaksux
committed
Add llmPredict builtin, opcode and ParamBuiltinOp entries
kubraaksux
committed
Add llmPredict parser validation in ParameterizedBuiltinFunctionExpression
kubraaksux
committed
Wire llmPredict through hop, lop and instruction generation
kubraaksux
committed
Add llmPredict CP instruction with HTTP-based inference
kubraaksux
committed
Remove Py4J-based LLM inference from JMLC API
kubraaksux
committed
Rewrite LLM test to use llmPredict DML built-in
kubraaksux
committed
Add OpenAI-compatible HTTP inference server for HuggingFace models
kubraaksux
committed
Merge branch 'llm-api' into llm-benchmark
kubraaksux
committed
Update benchmark backend to use llmPredict DML built-in
kubraaksux
committed
Fix llmPredict code quality and clean up Py4J remnants
kubraaksux
committed
Fix llmPredict code quality issues
kubraaksux
committed
Add concurrency parameter to llmPredict built-in
kubraaksux
committed
Merge branch 'llm-api' into llm-benchmark
kubraaksux
committed
Remove old SystemDS results and clean up headers
kubraaksux
committed
Pass concurrency to llmPredict via SYSTEMDS_CONCURRENCY env var
kubraaksux
committed
Route SystemDS concurrency through Java instead of Python threads
kubraaksux
committed
Fix JVM incubator vector module for Py4J gateway
kubraaksux
committed
Fix JMLC frame binding: match DML variable names to registered inputs
kubraaksux
committed
Add SystemDS llmPredict benchmark results (c=1 and c=4)
kubraaksux
committed
Fix benchmark results accuracy and update documentation
kubraaksux
committed
Fix data accuracy across README, PR description, and HTML report
kubraaksux
committed
Rewrite README and PR description with accurate data and honest conclusions
kubraaksux
committed
Fix math extraction bug, add cost tables, cross-backend comparisons, and evaluation methodology
kubraaksux
committed
Fix Mistral math explanation: 20/31 wrong math, 10/31 extractor failures
kubraaksux
committed

Commits on Feb 25, 2026

Add dedicated LlmPredictCPInstruction with error handling, negative tests, and license headers
kubraaksux
committed