diff --git a/services-custom/dynamodb-enhanced/EC2_BENCHMARK_INSTRUCTIONS.md b/services-custom/dynamodb-enhanced/EC2_BENCHMARK_INSTRUCTIONS.md new file mode 100644 index 000000000000..61ff3a4b9f78 --- /dev/null +++ b/services-custom/dynamodb-enhanced/EC2_BENCHMARK_INSTRUCTIONS.md @@ -0,0 +1,275 @@ +# Enhanced Query Benchmark Instructions + +This document covers two ways to run the Enhanced Query (join and aggregation) benchmarks: **local DynamoDB** (no AWS required) and **EC2 + real DynamoDB** (production-like numbers). + +--- + +## Local DynamoDB benchmark (recommended for design doc) + +No AWS account or credentials required. Uses in-process DynamoDB Local, creates and seeds 1000 customers × 1000 orders, then runs five scenarios and prints latency stats (avgMs, p50Ms, p95Ms, rows). + +### How to run + +From the **repository root**: + +```bash +./services-custom/dynamodb-enhanced/run-enhanced-query-benchmark-local.sh +``` + +The script sets `USE_LOCAL_DYNAMODB=true` and invokes the benchmark runner. Results are printed to stdout. To save CSV results: + +```bash +BENCHMARK_OUTPUT_FILE=benchmark_local.csv ./services-custom/dynamodb-enhanced/run-enhanced-query-benchmark-local.sh +``` + +Optional env vars (set before running the script): `BENCHMARK_ITERATIONS` (default 5), `BENCHMARK_WARMUP` (default 2), `BENCHMARK_OUTPUT_FILE` (optional path for CSV). + +### Results + +Example output (environment and scenario lines): + +``` +Using in-process DynamoDB Local. +Creating tables and seeding data (1000 customers x 1000 orders)... +... +Environment: DynamoDB Local (in-process) CUSTOMERS_TABLE=customers_large ORDERS_TABLE=orders_large +Warmup=2 Iterations=5 +--- +baseOnly_keyCondition: avgMs=... p50Ms=... p95Ms=... rows=1 +joinInner_c1: avgMs=... p50Ms=... p95Ms=... rows=1000 +... +``` + +Use this output (or the CSV file) in the design document. See [COMPLEX_QUERIES_DESIGN.md](COMPLEX_QUERIES_DESIGN.md#benchmarking) for where to reference the benchmark and link to results. + +--- + +## EC2 + Real DynamoDB benchmark + +Use this for production-like latency (e.g. external claims or SLA discussions). Requires AWS account and EC2. + +### Prerequisites + +- AWS account with permissions to create DynamoDB tables and launch EC2 instances (or use existing EC2). +- AWS CLI configured (`aws configure`) or IAM role for EC2 with DynamoDB access. +- Java 8+ and Maven 3.6+ (on your machine for building; on EC2 for running). + +--- + +## Step 1: Create DynamoDB tables (AWS CLI) + +Create two tables in your chosen region (e.g. `us-east-1`) with the same schema as the functional tests. + +**Customers table** (partition key: `customerId` String): + +```bash +export AWS_REGION=us-east-1 +export CUSTOMERS_TABLE=customers_large +export ORDERS_TABLE=orders_large + +aws dynamodb create-table \ + --table-name $CUSTOMERS_TABLE \ + --attribute-definitions AttributeName=customerId,AttributeType=S \ + --key-schema AttributeName=customerId,KeyType=HASH \ + --billing-mode PAY_PER_REQUEST \ + --region $AWS_REGION +``` + +**Orders table** (partition key: `customerId` String, sort key: `orderId` String): + +```bash +aws dynamodb create-table \ + --table-name $ORDERS_TABLE \ + --attribute-definitions \ + AttributeName=customerId,AttributeType=S \ + AttributeName=orderId,AttributeType=S \ + --key-schema \ + AttributeName=customerId,KeyType=HASH \ + AttributeName=orderId,KeyType=RANGE \ + --billing-mode PAY_PER_REQUEST \ + --region $AWS_REGION +``` + +Wait until both tables are `ACTIVE`: + +```bash +aws dynamodb describe-table --table-name $CUSTOMERS_TABLE --query 'Table.TableStatus' +aws dynamodb describe-table --table-name $ORDERS_TABLE --query 'Table.TableStatus' +``` + +--- + +## Step 2: Seed the tables (optional: use benchmark runner with CREATE_AND_SEED) + +You can either seed from your **local machine** (or an EC2 instance) by running the benchmark runner once with `CREATE_AND_SEED=true`. This creates tables if they do not exist (skip if you already created them in Step 1) and seeds **1000 customers × 1000 orders** (1M orders). For tables you already created, use the same table names and set only the seed path. + +**Option A – Seed from local (or EC2) with the runner** + +From the **repo root**: + +```bash +export AWS_REGION=us-east-1 +export CUSTOMERS_TABLE=customers_large +export ORDERS_TABLE=orders_large +export CREATE_AND_SEED=true + +mvn test-compile exec:java -pl services-custom/dynamodb-enhanced \ + -Dexec.mainClass="software.amazon.awssdk.enhanced.dynamodb.functionaltests.EnhancedQueryBenchmarkRunner" \ + -Dexec.classpathScope=test +``` + +If the tables already exist, the initializer will skip creation and only seed data (idempotent). If you use **pay-per-request** billing, no capacity settings are needed. Seeding 1M items may take several minutes and incur write cost. + +**Option B – Create tables via runner (omit Step 1)** + +If you omit Step 1 and set `CREATE_AND_SEED=true`, the runner will try to create the tables. The SDK’s `createTable` uses **provisioned** throughput by default (50 RCU/WCU). For pay-per-request, create tables in Step 1 and only seed via the runner (run with `CREATE_AND_SEED=true` once; the initializer skips create if tables exist). + +--- + +## Step 3: Launch EC2 and install Java + Maven + +1. Launch an EC2 instance (e.g. Amazon Linux 2 or Ubuntu) in the **same region** as your DynamoDB tables. +2. Attach an **IAM role** to the instance with at least: + - `dynamodb:GetItem`, `dynamodb:PutItem`, `dynamodb:Query`, `dynamodb:Scan`, `dynamodb:BatchWriteItem`, `dynamodb:DescribeTable`, `dynamodb:CreateTable` (if you use CREATE_AND_SEED). +3. SSH into the instance and install Java and Maven: + +**Amazon Linux 2:** + +```bash +sudo yum install -y java-11-amazon-corretto maven +``` + +**Ubuntu:** + +```bash +sudo apt-get update && sudo apt-get install -y openjdk-11-jdk maven +``` + +4. Verify: + +```bash +java -version +mvn -version +``` + +--- + +## Step 4: Build and copy the project to EC2 + +**On your local machine** (from repo root): + +```bash +cd /path/to/aws-sdk-java-v2 +mvn clean package -pl services-custom/dynamodb-enhanced -DskipTests -q +``` + +Copy the module and its dependencies to EC2. Option A: copy the whole repo and build on EC2. Option B: copy the built JAR and dependency JARs. + +**Option A – Copy repo and build on EC2** + +```bash +scp -r . ec2-user@:~/aws-sdk-java-v2 +ssh ec2-user@ "cd ~/aws-sdk-java-v2 && mvn clean test-compile -pl services-custom/dynamodb-enhanced -DskipTests -q" +``` + +**Option B – Copy only the dynamodb-enhanced module and run with mvn exec:java on EC2** + +Copy the entire `aws-sdk-java-v2` repo (or at least the parent POMs and `services-custom/dynamodb-enhanced`) so that `mvn exec:java -pl services-custom/dynamodb-enhanced` can resolve the parent and run the benchmark. Building on EC2 is usually simpler: + +```bash +rsync -avz --exclude='.git' . ec2-user@:~/aws-sdk-java-v2 +``` + +Then on EC2: + +```bash +cd ~/aws-sdk-java-v2 +mvn test-compile -pl services-custom/dynamodb-enhanced -DskipTests -q +``` + +--- + +## Step 5: Run the benchmark on EC2 + +SSH to the EC2 instance and set environment variables, then run the benchmark. + +```bash +cd ~/aws-sdk-java-v2 + +export AWS_REGION=us-east-1 +export CUSTOMERS_TABLE=customers_large +export ORDERS_TABLE=orders_large +export BENCHMARK_ITERATIONS=5 +export BENCHMARK_WARMUP=2 +# Optional: append CSV results to a file +export BENCHMARK_OUTPUT_FILE=benchmark_results.csv + +# Do NOT set CREATE_AND_SEED unless you want to create/seed from this instance (tables should already exist and be seeded). + +mvn exec:java -pl services-custom/dynamodb-enhanced \ + -Dexec.mainClass="software.amazon.awssdk.enhanced.dynamodb.functionaltests.EnhancedQueryBenchmarkRunner" \ + -Dexec.classpathScope=test -q +``` + +Example output: + +``` +Environment: AWS_REGION=us-east-1 CUSTOMERS_TABLE=customers_large ORDERS_TABLE=orders_large +Warmup=2 Iterations=5 +--- +baseOnly_keyCondition: avgMs=45.20 p50Ms=42 p95Ms=58 rows=1 +joinInner_c1: avgMs=320.40 p50Ms=310 p95Ms=380 rows=1000 +aggregation_groupByCount_c1: avgMs=305.20 p50Ms=298 p95Ms=350 rows=1 +aggregation_groupBySum_c1: avgMs=318.60 p50Ms=312 p95Ms=355 rows=1 +joinLeft_c1_limit50: avgMs=89.40 p50Ms=85 p95Ms=102 rows=50 +``` + +--- + +## Step 6: Collect results + +- **Stdout**: Redirect to a file, e.g. `mvn exec:java ... > benchmark_stdout.txt 2>&1`. +- **CSV**: If `BENCHMARK_OUTPUT_FILE` is set, the runner appends one CSV line per scenario to the file. Copy the file from EC2: + + ```bash + scp ec2-user@:~/aws-sdk-java-v2/benchmark_results.csv . + ``` + +Use the output (avgMs, p50Ms, p95Ms, rows) in your design doc. Document in the doc: **region**, **EC2 instance type**, **table names**, **dataset size** (e.g. 1000 customers × 1000 orders), and **billing mode** (pay-per-request or provisioned). + +--- + +## Step 7: Cleanup (optional) + +To avoid ongoing cost, delete the DynamoDB tables and terminate the EC2 instance when done: + +```bash +aws dynamodb delete-table --table-name customers_large --region us-east-1 +aws dynamodb delete-table --table-name orders_large --region us-east-1 +# Terminate the EC2 instance from the AWS Console or CLI. +``` + +--- + +## Environment variable reference + +| Variable | Required | Default | Description | +|----------|----------|---------|-------------| +| `AWS_REGION` | No | default SDK region | DynamoDB region (e.g. `us-east-1`). | +| `CUSTOMERS_TABLE` | No | `customers_large` | Customers table name. | +| `ORDERS_TABLE` | No | `orders_large` | Orders table name. | +| `CREATE_AND_SEED` | No | (unset) | Set to `true` to create tables (if missing) and seed 1000×1000 data. Requires DynamoDB create/put permissions. | +| `BENCHMARK_ITERATIONS` | No | `5` | Number of measured runs per scenario. | +| `BENCHMARK_WARMUP` | No | `2` | Warm-up runs per scenario before measuring. | +| `BENCHMARK_OUTPUT_FILE` | No | (none) | If set, CSV results are appended to this path. | + +--- + +## Running locally against DynamoDB Local + +To run the same benchmark against **DynamoDB Local** (e.g. for CI or no-AWS runs): + +1. Start DynamoDB Local (e.g. `docker run -p 8000:8000 amazon/dynamodb-local` or the SDK’s embedded LocalDynamoDb). +2. Set `AWS_REGION` and point the SDK to the local endpoint (e.g. `DYNAMODB_ENDPOINT_OVERRIDE=http://localhost:8000` if your test setup supports it, or run the functional tests which use in-process LocalDynamoDb). + +The benchmark runner does **not** set an endpoint override by default; it uses the default DynamoDB endpoint for the given region. To run against Local, you would need to configure the client with an endpoint override (e.g. in a variant of the runner or via a system property your client builder reads). The functional tests and `run-enhanced-query-tests-and-print-timing.sh` already run against Local and produce timing output for the design doc. diff --git a/services-custom/dynamodb-enhanced/pom.xml b/services-custom/dynamodb-enhanced/pom.xml index c822f6118436..0f00b1cd5764 100644 --- a/services-custom/dynamodb-enhanced/pom.xml +++ b/services-custom/dynamodb-enhanced/pom.xml @@ -111,6 +111,13 @@ http-client-spi ${awsjavasdk.version} + + + software.amazon.awssdk + url-connection-client + ${awsjavasdk.version} + test + software.amazon.awssdk sdk-core diff --git a/services-custom/dynamodb-enhanced/src/main/java/software/amazon/awssdk/enhanced/dynamodb/DynamoDbEnhancedAsyncClient.java b/services-custom/dynamodb-enhanced/src/main/java/software/amazon/awssdk/enhanced/dynamodb/DynamoDbEnhancedAsyncClient.java index 9d9cbf100f28..e260248d2bbc 100644 --- a/services-custom/dynamodb-enhanced/src/main/java/software/amazon/awssdk/enhanced/dynamodb/DynamoDbEnhancedAsyncClient.java +++ b/services-custom/dynamodb-enhanced/src/main/java/software/amazon/awssdk/enhanced/dynamodb/DynamoDbEnhancedAsyncClient.java @@ -37,6 +37,8 @@ import software.amazon.awssdk.enhanced.dynamodb.model.TransactWriteItemsEnhancedRequest; import software.amazon.awssdk.enhanced.dynamodb.model.TransactWriteItemsEnhancedResponse; import software.amazon.awssdk.enhanced.dynamodb.model.UpdateItemEnhancedRequest; +import software.amazon.awssdk.enhanced.dynamodb.query.result.EnhancedQueryRow; +import software.amazon.awssdk.enhanced.dynamodb.query.spec.QueryExpressionSpec; import software.amazon.awssdk.services.dynamodb.DynamoDbAsyncClient; /** @@ -527,13 +529,22 @@ default CompletableFuture transactWriteItems throw new UnsupportedOperationException(); } + /** + * Executes an enhanced query (joins, aggregations, filters) described by the given spec, asynchronously. + * + * @param spec the query specification + * @return a publisher of result rows + */ + default SdkPublisher enhancedQuery(QueryExpressionSpec spec) { + throw new UnsupportedOperationException(); + } + /** * Returns the underlying low-level {@link DynamoDbAsyncClient} that this enhanced client uses to make API calls. *

* The returned client is the same instance that was provided during construction via - * {@link Builder#dynamoDbClient(DynamoDbAsyncClient)}, or the internally-created one if {@link #create()} was used. - * It is not a copy — any operations performed on it (including {@code close()}) will affect this - * enhanced client as well. + * {@link Builder#dynamoDbClient(DynamoDbAsyncClient)}, or the internally-created one if {@link #create()} was used. It is + * not a copy — any operations performed on it (including {@code close()}) will affect this enhanced client as well. * * @return the underlying {@link DynamoDbAsyncClient} * @throws UnsupportedOperationException if the implementation does not support this operation diff --git a/services-custom/dynamodb-enhanced/src/main/java/software/amazon/awssdk/enhanced/dynamodb/DynamoDbEnhancedClient.java b/services-custom/dynamodb-enhanced/src/main/java/software/amazon/awssdk/enhanced/dynamodb/DynamoDbEnhancedClient.java index 29121c077609..08aca0e36110 100644 --- a/services-custom/dynamodb-enhanced/src/main/java/software/amazon/awssdk/enhanced/dynamodb/DynamoDbEnhancedClient.java +++ b/services-custom/dynamodb-enhanced/src/main/java/software/amazon/awssdk/enhanced/dynamodb/DynamoDbEnhancedClient.java @@ -36,6 +36,9 @@ import software.amazon.awssdk.enhanced.dynamodb.model.TransactWriteItemsEnhancedRequest; import software.amazon.awssdk.enhanced.dynamodb.model.TransactWriteItemsEnhancedResponse; import software.amazon.awssdk.enhanced.dynamodb.model.UpdateItemEnhancedRequest; +import software.amazon.awssdk.enhanced.dynamodb.query.result.EnhancedQueryLatencyReport; +import software.amazon.awssdk.enhanced.dynamodb.query.result.EnhancedQueryResult; +import software.amazon.awssdk.enhanced.dynamodb.query.spec.QueryExpressionSpec; import software.amazon.awssdk.services.dynamodb.DynamoDbClient; import software.amazon.awssdk.services.dynamodb.model.BatchGetItemRequest; @@ -537,13 +540,34 @@ default TransactWriteItemsEnhancedResponse transactWriteItemsWithResponse( throw new UnsupportedOperationException(); } + /** + * Executes an enhanced query (joins, aggregations, filters) described by the given spec. + * + * @param spec the query specification + * @return iterable of result rows + */ + default EnhancedQueryResult enhancedQuery(QueryExpressionSpec spec) { + throw new UnsupportedOperationException(); + } + + /** + * Executes an enhanced query and optionally reports latency. + * + * @param spec the query specification + * @param reportConsumer optional consumer for the latency report; may be null + * @return iterable of result rows + */ + default EnhancedQueryResult enhancedQuery(QueryExpressionSpec spec, + Consumer reportConsumer) { + throw new UnsupportedOperationException(); + } + /** * Returns the underlying low-level {@link DynamoDbClient} that this enhanced client uses to make API calls. *

* The returned client is the same instance that was provided during construction via - * {@link Builder#dynamoDbClient(DynamoDbClient)}, or the internally-created one if {@link #create()} was used. - * It is not a copy — any operations performed on it (including {@code close()}) will affect this - * enhanced client as well. + * {@link Builder#dynamoDbClient(DynamoDbClient)}, or the internally-created one if {@link #create()} was used. It is + * not a copy — any operations performed on it (including {@code close()}) will affect this enhanced client as well. * * @return the underlying {@link DynamoDbClient} * @throws UnsupportedOperationException if the implementation does not support this operation diff --git a/services-custom/dynamodb-enhanced/src/main/java/software/amazon/awssdk/enhanced/dynamodb/internal/client/DefaultDynamoDbEnhancedAsyncClient.java b/services-custom/dynamodb-enhanced/src/main/java/software/amazon/awssdk/enhanced/dynamodb/internal/client/DefaultDynamoDbEnhancedAsyncClient.java index 4366a18b90e1..e18f359931ae 100644 --- a/services-custom/dynamodb-enhanced/src/main/java/software/amazon/awssdk/enhanced/dynamodb/internal/client/DefaultDynamoDbEnhancedAsyncClient.java +++ b/services-custom/dynamodb-enhanced/src/main/java/software/amazon/awssdk/enhanced/dynamodb/internal/client/DefaultDynamoDbEnhancedAsyncClient.java @@ -22,6 +22,7 @@ import java.util.function.Consumer; import software.amazon.awssdk.annotations.NotThreadSafe; import software.amazon.awssdk.annotations.SdkInternalApi; +import software.amazon.awssdk.core.async.SdkPublisher; import software.amazon.awssdk.enhanced.dynamodb.Document; import software.amazon.awssdk.enhanced.dynamodb.DynamoDbEnhancedAsyncClient; import software.amazon.awssdk.enhanced.dynamodb.DynamoDbEnhancedClientExtension; @@ -37,16 +38,22 @@ import software.amazon.awssdk.enhanced.dynamodb.model.TransactGetItemsEnhancedRequest; import software.amazon.awssdk.enhanced.dynamodb.model.TransactWriteItemsEnhancedRequest; import software.amazon.awssdk.enhanced.dynamodb.model.TransactWriteItemsEnhancedResponse; +import software.amazon.awssdk.enhanced.dynamodb.query.engine.DefaultQueryExpressionAsyncEngine; +import software.amazon.awssdk.enhanced.dynamodb.query.engine.QueryExpressionAsyncEngine; +import software.amazon.awssdk.enhanced.dynamodb.query.result.EnhancedQueryRow; +import software.amazon.awssdk.enhanced.dynamodb.query.spec.QueryExpressionSpec; import software.amazon.awssdk.services.dynamodb.DynamoDbAsyncClient; @SdkInternalApi public final class DefaultDynamoDbEnhancedAsyncClient implements DynamoDbEnhancedAsyncClient { private final DynamoDbAsyncClient dynamoDbClient; private final DynamoDbEnhancedClientExtension extension; + private final QueryExpressionAsyncEngine queryExpressionAsyncEngine; private DefaultDynamoDbEnhancedAsyncClient(Builder builder) { this.dynamoDbClient = builder.dynamoDbClient == null ? DynamoDbAsyncClient.create() : builder.dynamoDbClient; this.extension = ExtensionResolver.resolveExtensions(builder.dynamoDbEnhancedClientExtensions); + this.queryExpressionAsyncEngine = new DefaultQueryExpressionAsyncEngine(this); } public static Builder builder() { @@ -132,6 +139,11 @@ public CompletableFuture transactWriteItemsW return transactWriteItemsWithResponse(builder.build()); } + @Override + public SdkPublisher enhancedQuery(QueryExpressionSpec spec) { + return queryExpressionAsyncEngine.execute(spec); + } + @Override public DynamoDbAsyncClient dynamoDbAsyncClient() { return dynamoDbClient; diff --git a/services-custom/dynamodb-enhanced/src/main/java/software/amazon/awssdk/enhanced/dynamodb/internal/client/DefaultDynamoDbEnhancedClient.java b/services-custom/dynamodb-enhanced/src/main/java/software/amazon/awssdk/enhanced/dynamodb/internal/client/DefaultDynamoDbEnhancedClient.java index 3bb522247676..c65909c956d7 100644 --- a/services-custom/dynamodb-enhanced/src/main/java/software/amazon/awssdk/enhanced/dynamodb/internal/client/DefaultDynamoDbEnhancedClient.java +++ b/services-custom/dynamodb-enhanced/src/main/java/software/amazon/awssdk/enhanced/dynamodb/internal/client/DefaultDynamoDbEnhancedClient.java @@ -36,16 +36,23 @@ import software.amazon.awssdk.enhanced.dynamodb.model.TransactGetItemsEnhancedRequest; import software.amazon.awssdk.enhanced.dynamodb.model.TransactWriteItemsEnhancedRequest; import software.amazon.awssdk.enhanced.dynamodb.model.TransactWriteItemsEnhancedResponse; +import software.amazon.awssdk.enhanced.dynamodb.query.engine.DefaultQueryExpressionEngine; +import software.amazon.awssdk.enhanced.dynamodb.query.engine.QueryExpressionEngine; +import software.amazon.awssdk.enhanced.dynamodb.query.result.EnhancedQueryLatencyReport; +import software.amazon.awssdk.enhanced.dynamodb.query.result.EnhancedQueryResult; +import software.amazon.awssdk.enhanced.dynamodb.query.spec.QueryExpressionSpec; import software.amazon.awssdk.services.dynamodb.DynamoDbClient; @SdkInternalApi public final class DefaultDynamoDbEnhancedClient implements DynamoDbEnhancedClient { private final DynamoDbClient dynamoDbClient; private final DynamoDbEnhancedClientExtension extension; + private final QueryExpressionEngine queryExpressionEngine; private DefaultDynamoDbEnhancedClient(Builder builder) { this.dynamoDbClient = builder.dynamoDbClient == null ? DynamoDbClient.create() : builder.dynamoDbClient; this.extension = ExtensionResolver.resolveExtensions(builder.dynamoDbEnhancedClientExtensions); + this.queryExpressionEngine = new DefaultQueryExpressionEngine(this); } public static Builder builder() { @@ -126,6 +133,17 @@ public TransactWriteItemsEnhancedResponse transactWriteItemsWithResponse( return transactWriteItemsWithResponse(builder.build()); } + @Override + public EnhancedQueryResult enhancedQuery(QueryExpressionSpec spec) { + return queryExpressionEngine.execute(spec); + } + + @Override + public EnhancedQueryResult enhancedQuery(QueryExpressionSpec spec, + Consumer reportConsumer) { + return queryExpressionEngine.execute(spec, reportConsumer); + } + @Override public DynamoDbClient dynamoDbClient() { return dynamoDbClient; diff --git a/services-custom/dynamodb-enhanced/src/main/java/software/amazon/awssdk/enhanced/dynamodb/query/ENHANCED_QUERY_GUIDE.md b/services-custom/dynamodb-enhanced/src/main/java/software/amazon/awssdk/enhanced/dynamodb/query/ENHANCED_QUERY_GUIDE.md new file mode 100644 index 000000000000..5c5446a5d649 --- /dev/null +++ b/services-custom/dynamodb-enhanced/src/main/java/software/amazon/awssdk/enhanced/dynamodb/query/ENHANCED_QUERY_GUIDE.md @@ -0,0 +1,877 @@ +# Enhanced Query Playbook + +A comprehensive reference for the DynamoDB Enhanced Client's **Enhanced Query** API: how to build queries, how execution +actually works, class-by-class scope, condition trees, output examples, benchmark workflows, and diagrams. + +--- + +## Table of contents + +1. [Setup](#1-setup) +2. [Core concepts](#2-core-concepts) +3. [Class reference](#3-class-reference) +4. [Single-table query](#4-single-table-query) +5. [Full-table scan](#5-full-table-scan) +6. [Filtering](#6-filtering) +7. [Nested attribute filtering](#7-nested-attribute-filtering) +8. [Joins](#8-joins) +9. [Pre-join filters](#9-pre-join-filters) +10. [ExecutionMode deep-dive](#10-executionmode-deep-dive) +11. [Aggregations](#11-aggregations) +12. [Join plus aggregation](#12-join-plus-aggregation) +13. [Ordering](#13-ordering) +14. [Projection](#14-projection) +15. [Limit](#15-limit) +16. [Latency report](#16-latency-report) +17. [Async API](#17-async-api) +18. [Condition system deep-dive](#18-condition-system-deep-dive) +19. [Execution flow diagrams](#19-execution-flow-diagrams) +20. [Build validation](#20-build-validation) +21. [Performance tips](#21-performance-tips) +22. [Complete example](#22-complete-example) +23. [Benchmark guide](#23-benchmark-guide) + +--- + +## 1. Setup + +```xml + + + software.amazon.awssdk + dynamodb-enhanced + +``` + +```java +DynamoDbClient lowLevel = DynamoDbClient.create(); +DynamoDbEnhancedClient client = DynamoDbEnhancedClient.builder() + .dynamoDbClient(lowLevel) + .build(); + +DynamoDbTable customers = client.table("Customers", CUSTOMER_SCHEMA); +DynamoDbTable orders = client.table("Orders", ORDER_SCHEMA); +``` + +--- + +## 2. Core concepts + +| Concept | Description | +|-------------------------------|------------------------------------| +| `QueryExpressionBuilder` | Fluent API for query specification | +| `QueryExpressionSpec` | Immutable output of the builder | +| `enhancedQuery(spec)` | Sync execution API | +| `enhancedQuery(spec)` async | Publisher-based async API | +| `keyCondition` | Only server-side filter pushdown | +| `where` | Final in-memory filter | +| `filterBase` / `filterJoined` | In-memory pre-join filters | +| `ExecutionMode` | Strict key only or scan allowed | + +Important: only `keyCondition` is pushed to DynamoDB query planning. Join logic, non-key filtering, grouping, ordering, +and limit are in-memory engine operations. + +--- + +## 3. Class reference + +### Public query model classes + +| Class | Scope | What it owns | +|------------------------------|----------------------|-------------------------------------------------------------------------------| +| `QueryExpressionBuilder` | Builder API | Fluent query construction and validation | +| `QueryExpressionSpec` | Immutable spec | Full query plan data (tables, join, conditions, grouping, order, mode, limit) | +| `AggregateSpec` | Aggregation config | Function, input attribute, output alias | +| `OrderBySpec` | Sort config | Sort key and direction, attribute or aggregate | +| `EnhancedQueryRow` | Result row | `itemsByAlias` and `aggregates` | +| `EnhancedQueryResult` | Result stream (sync) | Iterable wrapper for rows | +| `EnhancedQueryLatencyReport` | Telemetry model | base/joined/in-memory/total timing values | + +### Condition and evaluation classes + +| Class | Scope | Notes | +|----------------------|--------------------------------------|------------------------------------------------------------------------------| +| `Condition` | Filter AST factories and combinators | `eq`, `gt`, `between`, `contains`, `beginsWith`, `and`, `or`, `not`, `group` | +| `ConditionEvaluator` | Recursive condition execution | Applies conditions over item maps, including dot-path lookups | + +### Enums + +| Enum | Values | Purpose | +|-----------------------|-------------------------------------|-----------------------| +| `ExecutionMode` | `STRICT_KEY_ONLY`, `ALLOW_SCAN` | Scan fallback policy | +| `JoinType` | `INNER`, `LEFT`, `RIGHT`, `FULL` | Join output semantics | +| `AggregationFunction` | `COUNT`, `SUM`, `AVG`, `MIN`, `MAX` | Aggregate operator | +| `SortDirection` | `ASC`, `DESC` | Ordering direction | + +### Internal engine classes (implementation detail) + +| Class | Scope | +|------------------------------------------------------------------------|----------------------------------------------| +| `QueryExpressionEngine` | Sync execution interface | +| `QueryExpressionAsyncEngine` | Async execution interface | +| `DefaultQueryExpressionEngine` | Sync execution implementation | +| `DefaultQueryExpressionAsyncEngine` | Async execution implementation | +| `JoinedTableObjectMapSyncFetcher` / `JoinedTableObjectMapAsyncFetcher` | Joined-side lookup strategy (PK/GSI/scan) | +| `QueryEngineSupport` | Shared aggregation/sort/group helpers | +| `JoinRowAliases` | Alias mapping for base/joined item maps | +| `AttributeValueConversion` | DynamoDB `AttributeValue` conversion helpers | + +--- + +## 4. Single-table query + +```java +QueryExpressionSpec spec = QueryExpressionBuilder.from(customers) + .keyCondition(QueryConditional.keyEqualTo(k -> k.partitionValue("c1"))) + .build(); + +for( +EnhancedQueryRow row :client. + +enhancedQuery(spec)){ + System.out. + +println(row.getItem("base"). + +get("customerId")); + } +``` + +**Output (example):** + +```text +c1 +``` + +Execution summary: base table uses DynamoDB `Query`; rows are emitted with alias `base`. + +--- + +## 5. Full-table scan + +```java +QueryExpressionSpec spec = QueryExpressionBuilder.from(customers) + .executionMode(ExecutionMode.ALLOW_SCAN) + .where(Condition.eq("region", "EU")) + .build(); +``` + +**Output (example):** + +```text +rows=500 +first.customerId=c1 +last.customerId=c999 +``` + +On seeded test data, EU is odd customer IDs, so there are 500 matches out of 1000. + +--- + +## 6. Filtering + +All `Condition` filters run in memory: + +```java +QueryExpressionSpec spec = QueryExpressionBuilder.from(orders) + .executionMode(ExecutionMode.ALLOW_SCAN) + .where( + Condition.eq("customerId", "c1") + .and(Condition.gte("amount", 50)) + .and(Condition.beginsWith("orderId", "c1-o")) + ) + .build(); +``` + +**Output (example):** + +```text +rows=997 +first.orderId=c1-o4 +first.amount=50 +``` + +--- + +## 7. Nested attribute filtering + +```java +QueryExpressionSpec spec = QueryExpressionBuilder.from(customers) + .executionMode(ExecutionMode.ALLOW_SCAN) + .where(Condition.eq("address.city", "Seattle")) + .build(); +``` + +**Output (example):** + +```text +rows=2 +customerIds=[c1, c3] +``` + +Dot-path (`address.city`) resolution traverses nested maps. Missing path components evaluate as non-match. + +--- + +## 8. Joins + +```java +QueryExpressionSpec spec = QueryExpressionBuilder.from(customers) + .join(orders, JoinType.INNER, "customerId", "customerId") + .keyCondition(QueryConditional.keyEqualTo(k -> k.partitionValue("c1"))) + .build(); +``` + +**Output (example):** + +```text +rows=1000 +sample: base.customerId=c1, joined.orderId=c1-o1 +``` + +### Join type behavior + +| Join type | Output semantics | +|-----------|-----------------------------------------------| +| `INNER` | Matched pairs only | +| `LEFT` | All base rows, empty joined map when no match | +| `RIGHT` | All joined rows, empty base map when no match | +| `FULL` | Union of LEFT and RIGHT behavior | + +**Output examples by type:** + +```text +INNER c1 -> rows=1000 +LEFT c1 -> rows=1000 +RIGHT c1 -> rows=1000 +FULL c1 -> rows=1000 +``` + +--- + +## 9. Pre-join filters + +```java +QueryExpressionSpec spec = QueryExpressionBuilder.from(customers) + .join(orders, JoinType.INNER, "customerId", "customerId") + .keyCondition(QueryConditional.keyEqualTo(k -> k.partitionValue("c1"))) + .filterBase(Condition.eq("region", "EU")) + .filterJoined(Condition.gte("amount", 50)) + .where(Condition.beginsWith("orderId", "c1-o")) + .build(); +``` + +Execution order: + +1. `filterBase` +2. `filterJoined` +3. `where` + +--- + +## 10. ExecutionMode deep-dive + +`ExecutionMode` controls scan fallback, not validation. + +### 10.1 STRICT_KEY_ONLY plus key condition + +```java +QueryExpressionSpec spec = QueryExpressionBuilder.from(customers) + .executionMode(ExecutionMode.STRICT_KEY_ONLY) + .keyCondition(QueryConditional.keyEqualTo(k -> k.partitionValue("c1"))) + .build(); +``` + +**Output:** `rows=1` (uses `Query`) + +### 10.2 STRICT_KEY_ONLY without key condition + +```java +QueryExpressionSpec spec = QueryExpressionBuilder.from(customers) + .executionMode(ExecutionMode.STRICT_KEY_ONLY) + .where(Condition.eq("region", "EU")) + .build(); +``` + +**Output:** `rows=0` (no scan fallback) + +### 10.3 ALLOW_SCAN without key condition + +```java +QueryExpressionSpec spec = QueryExpressionBuilder.from(customers) + .executionMode(ExecutionMode.ALLOW_SCAN) + .where(Condition.eq("region", "EU")) + .build(); +``` + +**Output:** `rows=500` (scan fallback allowed) + +### 10.4 ALLOW_SCAN with key condition + +```java +QueryExpressionSpec spec = QueryExpressionBuilder.from(customers) + .executionMode(ExecutionMode.ALLOW_SCAN) + .keyCondition(QueryConditional.keyEqualTo(k -> k.partitionValue("c1"))) + .build(); +``` + +**Output:** `rows=1` (still uses `Query`; scan is fallback only) + +### 10.5 Join with STRICT_KEY_ONLY and non-key joined path + +```java +QueryExpressionSpec spec = QueryExpressionBuilder.from(customers) + .join(orders, JoinType.LEFT, "region", "status") + .executionMode(ExecutionMode.STRICT_KEY_ONLY) + .keyCondition(QueryConditional.keyEqualTo(k -> k.partitionValue("c1"))) + .build(); +``` + +**Output:** base rows preserved by LEFT semantics; joined side can be empty where no key/index path exists. + +### 10.6 Join with ALLOW_SCAN and non-key joined path + +```java +QueryExpressionSpec spec = QueryExpressionBuilder.from(customers) + .join(orders, JoinType.LEFT, "region", "status") + .executionMode(ExecutionMode.ALLOW_SCAN) + .keyCondition(QueryConditional.keyEqualTo(k -> k.partitionValue("c1"))) + .build(); +``` + +**Output:** joined lookups may use scan fallback. + +--- + +## 11. Aggregations + +```java +QueryExpressionSpec spec = QueryExpressionBuilder.from(orders) + .executionMode(ExecutionMode.ALLOW_SCAN) + .groupBy("customerId") + .aggregate(AggregationFunction.COUNT, "orderId", "orderCount") + .aggregate(AggregationFunction.SUM, "amount", "totalAmount") + .aggregate(AggregationFunction.AVG, "amount", "avgAmount") + .build(); +``` + +**Output (example for `customerId=c1`):** + +```text +customerId=c1 +orderCount=1000 +totalAmount=510500 +avgAmount=510.5 +``` + +### Aggregation function types + +| Function | Meaning | Output type | +|----------|-----------------|--------------------------------------| +| `COUNT` | Count values | Long | +| `SUM` | Numeric sum | BigDecimal-compatible numeric output | +| `AVG` | Numeric average | BigDecimal-compatible numeric output | +| `MIN` | Minimum | Input-compatible comparable | +| `MAX` | Maximum | Input-compatible comparable | + +--- + +## 12. Join plus aggregation + +```java +QueryExpressionSpec spec = QueryExpressionBuilder.from(customers) + .join(orders, JoinType.INNER, "customerId", "customerId") + .keyCondition(QueryConditional.keyEqualTo(k -> k.partitionValue("c1"))) + .filterJoined(Condition.gte("amount", 50)) + .groupBy("customerId") + .aggregate(AggregationFunction.COUNT, "orderId", "orderCount") + .aggregate(AggregationFunction.SUM, "amount", "totalAmount") + .build(); +``` + +**Output (example):** + +```text +customerId=c1 +orderCount=997 +totalAmount=510440 +``` + +--- + +## 13. Ordering + +```java +.orderBy("name",SortDirection.ASC) +. + +orderByAggregate("totalAmount",SortDirection.DESC) +``` + +Ordering is in-memory and runs after filtering/aggregation. + +--- + +## 14. Projection + +```java +.project("customerId","name","address.city") +``` + +Projection is pushed to DynamoDB (`ProjectionExpression`) but does not replace in-memory filtering rules. If a filter +needs an attribute, ensure it is projected. + +--- + +## 15. Limit + +```java +.limit(100) +``` + +Limit is applied after ordering/aggregation and caps final emitted rows. + +--- + +## 16. Latency report + +```java +client.enhancedQuery(spec, report ->{ + System.out. + +println("Base query: "+report.baseQueryMs() +" ms"); + System.out. + +println("Joined lookups: "+report.joinedLookupsMs() +" ms"); + System.out. + +println("In-memory: "+report.inMemoryProcessingMs() +" ms"); + System.out. + +println("Total: "+report.totalMs() +" ms"); + }); +``` + +--- + +## 17. Async API + +```java +DynamoDbEnhancedAsyncClient asyncClient = DynamoDbEnhancedAsyncClient.builder() + .dynamoDbClient(DynamoDbAsyncClient.create()) + .build(); + +QueryExpressionSpec spec = QueryExpressionBuilder.from(asyncCustomers) + .join(asyncOrders, JoinType.INNER, "customerId", "customerId") + .keyCondition(QueryConditional.keyEqualTo(k -> k.partitionValue("c1"))) + .build(); + +asyncClient. + +enhancedQuery(spec). + +subscribe(row -> + System.out. + +println(row.getItem("base"). + +get("customerId"))); +``` + +--- + +## 18. Condition system deep-dive + +Conditions are represented as a tree and evaluated recursively against item maps. + +### 18.1 Leaf condition node types + +| Node type | Factory | Meaning | +|------------|--------------------------------|-----------------------| +| Comparator | `eq`, `gt`, `gte`, `lt`, `lte` | Binary comparison | +| Between | `between(attr, from, to)` | Inclusive range check | +| Function | `contains`, `beginsWith` | String/set helpers | + +### 18.2 Composite node types + +| Node type | Created by | Meaning | +|-----------|------------------------|---------------------| +| `And` | `.and(...)` | Both children true | +| `Or` | `.or(...)` | Either child true | +| `Not` | `.not()` | Logical negation | +| `Group` | `Condition.group(...)` | Precedence grouping | + +### 18.3 Condition tree example + +``` + Or + / \ + And Not + / \ \ +eq(region, EU) gt(age, 25) eq(status, INACTIVE) +``` + +Expression that builds this tree: + +```java +Condition.eq("region", "EU") + .and(Condition.gt("age", 25)) + .or(Condition.eq("status", "INACTIVE").not()); +``` + +Evaluation order: `And` evaluates left then right (short-circuits on false). `Or` evaluates left +then right (short-circuits on true). `Not` inverts its child. `Group` is transparent -- it only +affects precedence during tree construction, not evaluation. + +### 18.4 Operator semantics + +| Operator | Rule | +|--------------|-------------------------------------------------| +| `eq` | true if comparison returns zero | +| `gt` / `gte` | standard numeric/comparable ordering | +| `lt` / `lte` | standard numeric/comparable ordering | +| `between` | inclusive `[from, to]` | +| `contains` | substring for `String`, element match for `Set` | +| `beginsWith` | prefix check for `String` | + +Comparison behavior: + +- Number values are compared as `BigDecimal`. +- Comparable values use `compareTo` when type-compatible. +- If comparable cast fails, fallback uses `toString()` lexical comparison. +- Null handling is deterministic (`null` compared with non-null is ordered lower). + +### 18.5 Dot-path resolution + +Paths like `address.city` or `metadata.score.value` are resolved by map traversal at each path segment. If any segment +is missing or non-map, the condition does not match. + +### 18.6 Combined map behavior for join `where` + +When evaluating `where` in join flows, base and joined maps are seen as a combined view. Base alias values are preferred +when present; joined map values fill missing attributes. + +### 18.7 Input/output condition examples + +Input item: + +```text +{name=Alice, age=30, region=EU, address={city=Seattle}} +``` + +| Condition | Result | +|-------------------------------------------------------------------------------------------------------------|--------| +| `Condition.eq("name", "Alice")` | true | +| `Condition.gt("age", 25)` | true | +| `Condition.between("age", 20, 29)` | false | +| `Condition.contains("name", "li")` | true | +| `Condition.beginsWith("name", "Al")` | true | +| `Condition.eq("address.city", "Seattle")` | true | +| `Condition.eq("region", "EU").and(Condition.gt("age", 25))` | true | +| `Condition.eq("region", "EU").or(Condition.eq("region", "US"))` | true | +| `Condition.eq("region", "NA").not()` | true | +| `Condition.group(Condition.eq("region","EU").or(Condition.eq("region","US"))).and(Condition.gt("age", 40))` | false | + +--- + +## 19. Execution flow diagrams + +### 19.1 Single-table flow + +``` + ┌─────────────────────┐ + │ keyCondition set? │ + └──────────┬──────────┘ + yes ┌─────┴─────┐ no + │ │ + ┌────▼───┐ ┌────▼──────────────┐ + │ Query │ │ ExecutionMode? │ + └────┬───┘ └────┬──────────┬───┘ + │ ALLOW_SCAN STRICT_KEY_ONLY + │ │ │ + │ ┌────▼───┐ ┌──────▼──────┐ + │ │ Scan │ │ Empty result │ + │ └────┬───┘ └─────────────┘ + │ │ + ┌────▼───────▼───┐ + │ Apply where() │ + └───────┬───────┘ + ┌───────▼───────┐ + │ groupBy + agg │ + └───────┬───────┘ + ┌───────▼───────┐ + │ orderBy │ + └───────┬───────┘ + ┌───────▼───────┐ + │ limit │ + └───────┬───────┘ + ┌───────▼───────┐ + │ Return rows │ + └───────────────┘ +``` + +### 19.2 Join flow + +``` + ┌──────────────────┐ + │ Fetch base rows │ + └────────┬─────────┘ + ┌────────▼─────────┐ + │ Apply filterBase │ + └────────┬─────────┘ + ┌────────▼──────────────────┐ + │ Collect distinct join keys│ + └────────┬──────────────────┘ + ┌────────▼─────────────┐ + │ Lookup joined rows │ (see 19.3 for strategy) + └────────┬─────────────┘ + ┌────────▼───────────────┐ + │ Apply filterJoined │ + └────────┬───────────────┘ + ┌────────▼───────────────┐ + │ Combine by JoinType │ + └────────┬───────────────┘ + ┌────────▼───────────────┐ + │ Apply where on merged │ + └────────┬───────────────┘ + ┌────────▼───────────────┐ + │ groupBy + aggregate │ + └────────┬───────────────┘ + ┌────────▼───────────────┐ + │ orderBy + limit │ + └────────┬───────────────┘ + ┌────────▼───────────────┐ + │ Return rows │ + └────────────────────────┘ +``` + +### 19.3 Joined lookup strategy + +``` + ┌───────────────────────────────┐ + │ Join attr is joined table PK? │ + └──────────┬────────────────┬───┘ + yes no + │ │ + ┌─────────▼──────┐ ┌─────▼───────────────┐ + │ Query by PK │ │ Join attr has GSI? │ + └────────────────┘ └─────┬───────────┬────┘ + yes no + │ │ + ┌─────────▼──────┐ ┌──▼──────────────┐ + │ Query by GSI │ │ ExecutionMode? │ + └────────────────┘ └──┬──────────┬───┘ + ALLOW_SCAN STRICT_KEY_ONLY + │ │ + ┌──────▼──────┐ ┌─────▼──────────┐ + │ Scan joined │ │ No rows (empty)│ + └─────────────┘ └────────────────┘ +``` + +### 19.4 Async flow + +``` + ┌────────────────────────┐ + │ Async execute │ + └───────────┬────────────┘ + ┌───────────▼────────────┐ + │ Fetch base publisher │ + └───────────┬────────────┘ + ┌───────────▼────────────┐ + │ Simple base-only path? │ + └─────┬──────────────┬───┘ + yes no + │ │ + ┌─────▼───────────┐ ┌▼──────────────────────┐ + │ Stream rows via │ │ Drain publisher to list│ + │ publisher │ └───────────┬────────────┘ + └─────────────────┘ ┌───────────▼────────────┐ + │ Join + aggregation work │ + └───────────┬─────────────┘ + ┌───────────▼─────────────┐ + │ Publish result rows │ + └─────────────────────────┘ +``` + +--- + +## 20. Build validation + +`build()` rejects invalid query shapes with `IllegalStateException`. + +| Invalid shape | Typical validation message | +|----------------------------------------------|---------------------------------------------------------------| +| `groupBy()` without `aggregate()` | `groupBy() requires at least one aggregate()` | +| `filterBase()` without join | `filterBase() is only applicable when a join is configured` | +| `filterJoined()` without join | `filterJoined() is only applicable when a join is configured` | +| join table without join type (or vice-versa) | join configuration missing counterpart | +| missing join key names | left/right join key missing | + +--- + +## 21. Performance tips + +1. Use `keyCondition` whenever possible. +2. Prefer `filterBase` and `filterJoined` to reduce join cardinality early. +3. Design joined-table access so join key maps to PK/GSI. +4. Treat `ALLOW_SCAN` as opt-in for known workloads. +5. Use `project(...)` to reduce transfer volume. +6. Use `limit(...)` and ordering for top-N style results. +7. Monitor timing with `EnhancedQueryLatencyReport`. + +--- + +## 22. Complete example + +```java +QueryExpressionSpec spec = QueryExpressionBuilder.from(customers) + .join(orders, JoinType.INNER, "customerId", "customerId") + .keyCondition(QueryConditional.keyEqualTo(k -> k.partitionValue("c1"))) + .filterBase(Condition.eq("region", "EU")) + .filterJoined(Condition.gte("amount", 50)) + .groupBy("customerId") + .aggregate(AggregationFunction.SUM, "amount", "totalAmount") + .aggregate(AggregationFunction.COUNT, "orderId", "orderCount") + .aggregate(AggregationFunction.AVG, "amount", "avgOrder") + .orderByAggregate("totalAmount", SortDirection.DESC) + .limit(10) + .build(); + +EnhancedQueryResult result = client.enhancedQuery(spec, report -> { + System.out.printf("Timing: base=%dms joined=%dms memory=%dms total=%dms%n", + report.baseQueryMs(), report.joinedLookupsMs(), + report.inMemoryProcessingMs(), report.totalMs()); +}); + +for( +EnhancedQueryRow row :result){ + System.out. + +printf("%s total=%s count=%s avg=%s%n", + row.getItem("base"). + +get("customerId"), + row. + +getAggregate("totalAmount"), + row. + +getAggregate("orderCount"), + row. + +getAggregate("avgOrder")); + } +``` + +**Output (example):** + +```text +Timing: base=xxms joined=yyms memory=zzms total=ttms +c1 total=510440 count=997 avg=512.979... +``` + +--- + +## 23. Benchmark guide + +### 23.1 What is benchmarked + +The benchmark runner covers these scenarios: + +- `baseOnly_keyCondition` +- `joinInner_c1` +- `aggregation_groupByCount_c1` +- `aggregation_groupBySum_c1` +- `joinLeft_c1_limit50` + +### 23.2 Local benchmark script + +```bash +./services-custom/dynamodb-enhanced/run-enhanced-query-benchmark-local.sh +``` + +Optional CSV target: + +```bash +BENCHMARK_OUTPUT_FILE=benchmark_local.csv ./services-custom/dynamodb-enhanced/run-enhanced-query-benchmark-local.sh +``` + +What it does: + +1. Starts local/in-process benchmark environment. +2. Creates and seeds dataset (1000 customers x 1000 orders) when required. +3. Executes warmup iterations. +4. Executes measured iterations. +5. Prints formatted benchmark table. +6. Appends CSV lines if `BENCHMARK_OUTPUT_FILE` is configured. + +### 23.3 Tests plus timing script + +```bash +./services-custom/dynamodb-enhanced/run-enhanced-query-tests-and-print-timing.sh +``` + +Output includes: + +- Functional test pass/fail summary. +- Timing lines from test execution. +- Useful quick regression signal for query behavior and performance. + +### 23.4 Console output format + +Benchmark output prints a formatted table with: + +- `SCENARIO` +- `DDB OPERATION` +- `DESCRIPTION` +- `AVG(ms)` +- `P50(ms)` +- `P95(ms)` +- `ROWS` + +### 23.5 CSV output format + +Header: + +```text +scenario,description,ddbOperation,avgMs,p50Ms,p95Ms,rows,region,iterations +``` + +Sample row: + +```text +joinInner_c1,"INNER join for c1","Query base + Query joined",312.40,306,350,1000,local,5 +``` + +Field meanings: + +- `scenario`: scenario id +- `description`: human-readable description +- `ddbOperation`: underlying DynamoDB access pattern +- `avgMs`, `p50Ms`, `p95Ms`: latency distribution metrics +- `rows`: output row count +- `region`: `local` or AWS region name +- `iterations`: measured run count + +### 23.6 Environment variables + +| Variable | Default | Meaning | +|-------------------------|-------------------|------------------------------------| +| `BENCHMARK_ITERATIONS` | `5` | Measured iterations per scenario | +| `BENCHMARK_WARMUP` | `2` | Warmup iterations before measuring | +| `BENCHMARK_OUTPUT_FILE` | none | Optional CSV output path | +| `AWS_REGION` | SDK default | Region for non-local run | +| `CUSTOMERS_TABLE` | `customers_large` | Customers table name | +| `ORDERS_TABLE` | `orders_large` | Orders table name | +| `CREATE_AND_SEED` | unset | Create and seed dataset when true | + +### 23.7 EC2 benchmark flow + +For real DynamoDB latency numbers: + +1. Create target tables in AWS. +2. Seed dataset. +3. Run benchmark from EC2 in same region. +4. Capture console and CSV outputs. +5. Record region, table names, dataset size, and iteration count with results. + diff --git a/services-custom/dynamodb-enhanced/src/main/java/software/amazon/awssdk/enhanced/dynamodb/query/condition/Condition.java b/services-custom/dynamodb-enhanced/src/main/java/software/amazon/awssdk/enhanced/dynamodb/query/condition/Condition.java new file mode 100644 index 000000000000..27f2816602da --- /dev/null +++ b/services-custom/dynamodb-enhanced/src/main/java/software/amazon/awssdk/enhanced/dynamodb/query/condition/Condition.java @@ -0,0 +1,427 @@ +/* + * Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"). + * You may not use this file except in compliance with the License. + * A copy of the License is located at + * + * http://aws.amazon.com/apache2.0 + * + * or in the "license" file accompanying this file. This file is distributed + * on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either + * express or implied. See the License for the specific language governing + * permissions and limitations under the License. + */ + +package software.amazon.awssdk.enhanced.dynamodb.query.condition; + +import java.util.Objects; +import software.amazon.awssdk.annotations.SdkInternalApi; + +/** + * Represents a filter condition for enhanced queries. Conditions can be combined with {@link #and(Condition)}, + * {@link #or(Condition)}, and {@link #not()}, and grouped with {@link #group(Condition)}. + *

+ * Use the static factory methods to build conditions: {@link #eq(String, Object)}, {@link #gt(String, Object)}, + * {@link #gte(String, Object)}, {@link #lt(String, Object)}, {@link #lte(String, Object)}, + * {@link #between(String, Object, Object)}, {@link #contains(String, Object)}, {@link #beginsWith(String, String)}. + *

+ * Implementation classes are package-private inner types; external code should interact only through the {@code Condition} + * interface and its static/default methods. + */ +@SdkInternalApi +public interface Condition { + + /** + * Combines this condition with another using logical AND. + */ + default Condition and(Condition other) { + return new And(this, other); + } + + /** + * Combines this condition with another using logical OR. + */ + default Condition or(Condition other) { + return new Or(this, other); + } + + /** + * Negates this condition. + */ + default Condition not() { + return new Not(this); + } + + // ---- static factories ----------------------------------------------- + + /** + * Creates an equality condition: attribute = value. + */ + static Condition eq(String attribute, Object value) { + return new Comparator(attribute, "=", value); + } + + /** + * Creates a greater-than condition: attribute > value. + */ + static Condition gt(String attribute, Object value) { + return new Comparator(attribute, ">", value); + } + + /** + * Creates a greater-than-or-equal condition: attribute >= value. + */ + static Condition gte(String attribute, Object value) { + return new Comparator(attribute, ">=", value); + } + + /** + * Creates a less-than condition: attribute < value. + */ + static Condition lt(String attribute, Object value) { + return new Comparator(attribute, "<", value); + } + + /** + * Creates a less-than-or-equal condition: attribute <= value. + */ + static Condition lte(String attribute, Object value) { + return new Comparator(attribute, "<=", value); + } + + /** + * Creates a between condition: attribute BETWEEN from AND to. + */ + static Condition between(String attribute, Object from, Object to) { + return new Between(attribute, from, to); + } + + /** + * Creates a contains condition (for string or set attributes). + */ + static Condition contains(String attribute, Object value) { + return new Function(attribute, "contains", value); + } + + /** + * Creates a begins-with condition for string attributes. + */ + static Condition beginsWith(String attribute, String prefix) { + return new Function(attribute, "begins_with", prefix); + } + + /** + * Groups a condition in parentheses (for precedence in AND/OR trees). + */ + static Condition group(Condition inner) { + return new Group(inner); + } + + // ---- inner implementation classes ----------------------------------- + + /** + * Comparison condition ({@code =}, {@code >}, {@code >=}, {@code <}, {@code <=}). + */ + final class Comparator implements Condition { + + private final String attribute; + private final String operator; + private final Object value; + + Comparator(String attribute, String operator, Object value) { + this.attribute = attribute; + this.operator = operator; + this.value = value; + } + + String attribute() { + return attribute; + } + + String operator() { + return operator; + } + + Object value() { + return value; + } + + @Override + public boolean equals(Object o) { + if (this == o) { + return true; + } + if (o == null || getClass() != o.getClass()) { + return false; + } + Comparator that = (Comparator) o; + return Objects.equals(attribute, that.attribute) + && Objects.equals(operator, that.operator) + && Objects.equals(value, that.value); + } + + @Override + public int hashCode() { + int result = Objects.hashCode(attribute); + result = 31 * result + Objects.hashCode(operator); + result = 31 * result + Objects.hashCode(value); + return result; + } + } + + /** + * Range condition: attribute BETWEEN from AND to. + */ + final class Between implements Condition { + + private final String attribute; + private final Object from; + private final Object to; + + Between(String attribute, Object from, Object to) { + this.attribute = attribute; + this.from = from; + this.to = to; + } + + String attribute() { + return attribute; + } + + Object from() { + return from; + } + + Object to() { + return to; + } + + @Override + public boolean equals(Object o) { + if (this == o) { + return true; + } + if (o == null || getClass() != o.getClass()) { + return false; + } + Between that = (Between) o; + return Objects.equals(attribute, that.attribute) + && Objects.equals(from, that.from) + && Objects.equals(to, that.to); + } + + @Override + public int hashCode() { + int result = Objects.hashCode(attribute); + result = 31 * result + Objects.hashCode(from); + result = 31 * result + Objects.hashCode(to); + return result; + } + } + + /** + * Function-based condition ({@code contains}, {@code begins_with}). + */ + final class Function implements Condition { + + private final String attribute; + private final String function; + private final Object value; + + Function(String attribute, String function, Object value) { + this.attribute = attribute; + this.function = function; + this.value = value; + } + + String attribute() { + return attribute; + } + + String function() { + return function; + } + + Object value() { + return value; + } + + @Override + public boolean equals(Object o) { + if (this == o) { + return true; + } + if (o == null || getClass() != o.getClass()) { + return false; + } + Function that = (Function) o; + return Objects.equals(attribute, that.attribute) + && Objects.equals(function, that.function) + && Objects.equals(value, that.value); + } + + @Override + public int hashCode() { + int result = Objects.hashCode(attribute); + result = 31 * result + Objects.hashCode(function); + result = 31 * result + Objects.hashCode(value); + return result; + } + } + + /** + * Logical AND of two conditions. + */ + final class And implements Condition { + + private final Condition left; + private final Condition right; + + And(Condition left, Condition right) { + this.left = left; + this.right = right; + } + + Condition left() { + return left; + } + + Condition right() { + return right; + } + + @Override + public boolean equals(Object o) { + if (this == o) { + return true; + } + if (o == null || getClass() != o.getClass()) { + return false; + } + And that = (And) o; + return Objects.equals(left, that.left) + && Objects.equals(right, that.right); + } + + @Override + public int hashCode() { + int result = Objects.hashCode(left); + result = 31 * result + Objects.hashCode(right); + return result; + } + } + + /** + * Logical OR of two conditions. + */ + final class Or implements Condition { + + private final Condition left; + private final Condition right; + + Or(Condition left, Condition right) { + this.left = left; + this.right = right; + } + + Condition left() { + return left; + } + + Condition right() { + return right; + } + + @Override + public boolean equals(Object o) { + if (this == o) { + return true; + } + if (o == null || getClass() != o.getClass()) { + return false; + } + Or that = (Or) o; + return Objects.equals(left, that.left) + && Objects.equals(right, that.right); + } + + @Override + public int hashCode() { + int result = Objects.hashCode(left); + result = 31 * result + Objects.hashCode(right); + return result; + } + } + + /** + * Negation of a condition. Double negation cancels out. + */ + final class Not implements Condition { + + private final Condition inner; + + Not(Condition inner) { + this.inner = inner; + } + + Condition inner() { + return inner; + } + + @Override + public Condition not() { + return inner; + } + + @Override + public boolean equals(Object o) { + if (this == o) { + return true; + } + if (o == null || getClass() != o.getClass()) { + return false; + } + Not that = (Not) o; + return Objects.equals(inner, that.inner); + } + + @Override + public int hashCode() { + return Objects.hashCode(inner); + } + } + + /** + * Grouping condition for precedence in AND/OR trees. + */ + final class Group implements Condition { + + private final Condition inner; + + Group(Condition inner) { + this.inner = inner; + } + + Condition inner() { + return inner; + } + + @Override + public boolean equals(Object o) { + if (this == o) { + return true; + } + if (o == null || getClass() != o.getClass()) { + return false; + } + Group that = (Group) o; + return Objects.equals(inner, that.inner); + } + + @Override + public int hashCode() { + return Objects.hashCode(inner); + } + } +} diff --git a/services-custom/dynamodb-enhanced/src/main/java/software/amazon/awssdk/enhanced/dynamodb/query/condition/ConditionEvaluator.java b/services-custom/dynamodb-enhanced/src/main/java/software/amazon/awssdk/enhanced/dynamodb/query/condition/ConditionEvaluator.java new file mode 100644 index 000000000000..6ca0ebe6ff2c --- /dev/null +++ b/services-custom/dynamodb-enhanced/src/main/java/software/amazon/awssdk/enhanced/dynamodb/query/condition/ConditionEvaluator.java @@ -0,0 +1,237 @@ +/* + * Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"). + * You may not use this file except in compliance with the License. + * A copy of the License is located at + * + * http://aws.amazon.com/apache2.0 + * + * or in the "license" file accompanying this file. This file is distributed + * on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either + * express or implied. See the License for the specific language governing + * permissions and limitations under the License. + */ + +package software.amazon.awssdk.enhanced.dynamodb.query.condition; + +import java.util.AbstractMap; +import java.util.Map; +import java.util.Set; +import software.amazon.awssdk.annotations.SdkInternalApi; + +/** + * Evaluates a {@link Condition} against an item represented as a map of attribute names to values. Used by the query engine to + * apply in-memory filters. Values in the map are typically Java types (String, Number, Boolean, etc.); comparison uses natural + * ordering where applicable. + */ +@SdkInternalApi +public final class ConditionEvaluator { + + private ConditionEvaluator() { + } + + /** + * Returns true if the given condition is satisfied by the item. If condition is null, returns true. + * + * @param condition the condition to evaluate (may be null) + * @param item attribute name to value map (Java types) + * @return true if condition is null or satisfied + */ + public static boolean evaluate(Condition condition, Map item) { + if (condition == null) { + return true; + } + if (condition instanceof Condition.Comparator) { + return evaluateComparator((Condition.Comparator) condition, item); + } + if (condition instanceof Condition.Between) { + return evaluateBetween((Condition.Between) condition, item); + } + if (condition instanceof Condition.Function) { + return evaluateFunction((Condition.Function) condition, item); + } + if (condition instanceof Condition.And) { + Condition.And and = (Condition.And) condition; + return evaluate(and.left(), item) && evaluate(and.right(), item); + } + if (condition instanceof Condition.Or) { + Condition.Or or = (Condition.Or) condition; + return evaluate(or.left(), item) || evaluate(or.right(), item); + } + if (condition instanceof Condition.Not) { + return !evaluate(((Condition.Not) condition).inner(), item); + } + if (condition instanceof Condition.Group) { + return evaluate(((Condition.Group) condition).inner(), item); + } + return false; + } + + /** + * Two-map overload that avoids merging base and joined maps into a combined HashMap. Attribute lookups check {@code primary} + * first; if the value is null, {@code secondary} is checked. This matches the semantics of + * {@code Map combined = new HashMap<>(secondary); combined.putAll(primary)}. + */ + public static boolean evaluate(Condition condition, Map primary, Map secondary) { + if (condition == null) { + return true; + } + return evaluate(condition, new CombinedMapView(primary, secondary)); + } + + /** + * Looks up an attribute from two maps (primary wins). Supports dot-path traversal for nested attributes. + */ + public static Object lookupCombined(String key, Map primary, Map secondary) { + Object v = resolveAttribute(primary, key); + return v != null ? v : resolveAttribute(secondary, key); + } + + /** + * Resolves an attribute value from a map, supporting dot-separated paths for nested Map attributes. For example, + * {@code resolveAttribute(item, "address.city")} will look up {@code item.get("address")} and, if that value is a + * {@code Map}, look up {@code "city"} within it. + * + * @param item the item map + * @param attribute the attribute name, possibly dot-separated + * @return the resolved value, or null if not found or any intermediate value is not a Map + */ + @SuppressWarnings("unchecked") + public static Object resolveAttribute(Map item, String attribute) { + if (attribute == null || item == null) { + return null; + } + if (attribute.indexOf('.') < 0) { + return item.get(attribute); + } + String[] parts = attribute.split("\\."); + Object current = item; + for (String part : parts) { + if (!(current instanceof Map)) { + return null; + } + current = ((Map) current).get(part); + if (current == null) { + return null; + } + } + return current; + } + + private static boolean evaluateComparator(Condition.Comparator c, Map item) { + Object actual = resolveAttribute(item, c.attribute()); + Object expected = c.value(); + int cmp = compare(actual, expected); + switch (c.operator()) { + case "=": + return cmp == 0; + case ">": + return cmp > 0; + case ">=": + return cmp >= 0; + case "<": + return cmp < 0; + case "<=": + return cmp <= 0; + default: + return false; + } + } + + private static boolean evaluateBetween(Condition.Between c, Map item) { + Object actual = resolveAttribute(item, c.attribute()); + if (actual == null) { + return false; + } + int low = compare(actual, c.from()); + int high = compare(actual, c.to()); + return low >= 0 && high <= 0; + } + + @SuppressWarnings("unchecked") + private static boolean evaluateFunction(Condition.Function c, Map item) { + Object actual = resolveAttribute(item, c.attribute()); + Object val = c.value(); + switch (c.function()) { + case "contains": + if (actual == null) { + return false; + } + if (actual instanceof String && val instanceof String) { + return ((String) actual).contains((String) val); + } + if (actual instanceof java.util.Set) { + return ((java.util.Set) actual).contains(val); + } + return false; + case "begins_with": + if (actual instanceof String && val instanceof String) { + return ((String) actual).startsWith((String) val); + } + return false; + default: + return false; + } + } + + @SuppressWarnings({"unchecked", "rawtypes"}) + private static int compare(Object a, Object b) { + if (a == null && b == null) { + return 0; + } + if (a == null) { + return -1; + } + if (b == null) { + return 1; + } + if (a instanceof Number && b instanceof Number) { + java.math.BigDecimal left = (a instanceof java.math.BigDecimal) + ? (java.math.BigDecimal) a + : new java.math.BigDecimal(a.toString()); + java.math.BigDecimal right = (b instanceof java.math.BigDecimal) + ? (java.math.BigDecimal) b + : new java.math.BigDecimal(b.toString()); + return left.compareTo(right); + } + if (a instanceof Comparable && b instanceof Comparable) { + try { + return ((Comparable) a).compareTo(b); + } catch (ClassCastException e) { + return a.toString().compareTo(b.toString()); + } + } + return a.toString().compareTo(b.toString()); + } + + /** + * Lightweight read-only view over two maps. {@code get} checks primary first, then secondary. Only {@code get} and + * {@code containsKey} are supported; this is sufficient for condition evaluation. + */ + private static final class CombinedMapView extends AbstractMap { + private final Map primary; + private final Map secondary; + + CombinedMapView(Map primary, Map secondary) { + this.primary = primary; + this.secondary = secondary; + } + + @Override + public Object get(Object key) { + Object v = primary.get(key); + return v != null ? v : secondary.get(key); + } + + @Override + public boolean containsKey(Object key) { + return primary.containsKey(key) || secondary.containsKey(key); + } + + @Override + public Set> entrySet() { + throw new UnsupportedOperationException("CombinedMapView does not support entrySet"); + } + } +} diff --git a/services-custom/dynamodb-enhanced/src/main/java/software/amazon/awssdk/enhanced/dynamodb/query/engine/AttributeValueConversion.java b/services-custom/dynamodb-enhanced/src/main/java/software/amazon/awssdk/enhanced/dynamodb/query/engine/AttributeValueConversion.java new file mode 100644 index 000000000000..0c62f16e1aaf --- /dev/null +++ b/services-custom/dynamodb-enhanced/src/main/java/software/amazon/awssdk/enhanced/dynamodb/query/engine/AttributeValueConversion.java @@ -0,0 +1,107 @@ +/* + * Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"). + * You may not use this file except in compliance with the License. + * A copy of the License is located at + * + * http://aws.amazon.com/apache2.0 + * + * or in the "license" file accompanying this file. This file is distributed + * on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either + * express or implied. See the License for the specific language governing + * permissions and limitations under the License. + */ + +package software.amazon.awssdk.enhanced.dynamodb.query.engine; + +import java.math.BigDecimal; +import java.util.Collections; +import java.util.HashMap; +import java.util.Map; +import java.util.stream.Collectors; +import software.amazon.awssdk.annotations.SdkInternalApi; +import software.amazon.awssdk.services.dynamodb.model.AttributeValue; + +/** + * Converts DynamoDB {@link AttributeValue} and maps of them to Java types for condition evaluation and row building in the + * complex query engine. + */ +@SdkInternalApi +final class AttributeValueConversion { + + private AttributeValueConversion() { + } + + /** + * Converts a single AttributeValue to a Java object suitable for comparison and display. Numbers are returned as BigDecimal + * for consistent ordering. + */ + static Object toObject(AttributeValue av) { + if (av == null) { + return null; + } + if (av.nul() != null && av.nul()) { + return null; + } + if (av.s() != null) { + return av.s(); + } + if (av.n() != null) { + return new BigDecimal(av.n()); + } + if (av.bool() != null) { + return av.bool(); + } + if (av.b() != null) { + return av.b(); + } + if (av.m() != null) { + Map map = new HashMap<>(); + av.m().forEach((k, v) -> map.put(k, toObject(v))); + return map; + } + if (av.l() != null) { + return av.l().stream().map(AttributeValueConversion::toObject).collect(Collectors.toList()); + } + if (av.ss() != null) { + return av.ss(); + } + if (av.ns() != null) { + return av.ns().stream().map(BigDecimal::new).collect(Collectors.toSet()); + } + if (av.bs() != null) { + return av.bs(); + } + return null; + } + + /** + * Converts a map of attribute name to AttributeValue into a map of attribute name to Java object. + */ + static Map toObjectMap(Map attributeMap) { + if (attributeMap == null || attributeMap.isEmpty()) { + return Collections.emptyMap(); + } + Map result = new HashMap<>(); + attributeMap.forEach((k, v) -> result.put(k, toObject(v))); + return result; + } + + /** + * Converts a join key object (typically String or Number) into an AttributeValue suitable for use in low-level Query or Scan + * requests. + */ + static AttributeValue toKeyAttributeValue(Object key) { + if (key == null) { + return AttributeValue.builder().nul(true).build(); + } + if (key instanceof String) { + return AttributeValue.builder().s((String) key).build(); + } + if (key instanceof Number) { + return AttributeValue.builder().n(key.toString()).build(); + } + return AttributeValue.builder().s(key.toString()).build(); + } +} diff --git a/services-custom/dynamodb-enhanced/src/main/java/software/amazon/awssdk/enhanced/dynamodb/query/engine/DefaultQueryExpressionAsyncEngine.java b/services-custom/dynamodb-enhanced/src/main/java/software/amazon/awssdk/enhanced/dynamodb/query/engine/DefaultQueryExpressionAsyncEngine.java new file mode 100644 index 000000000000..7078632721ab --- /dev/null +++ b/services-custom/dynamodb-enhanced/src/main/java/software/amazon/awssdk/enhanced/dynamodb/query/engine/DefaultQueryExpressionAsyncEngine.java @@ -0,0 +1,777 @@ +/* + * Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"). + * You may not use this file except in compliance with the License. + * A copy of the License is located at + * + * http://aws.amazon.com/apache2.0 + * + * or in the "license" file accompanying this file. This file is distributed + * on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either + * express or implied. See the License for the specific language governing + * permissions and limitations under the License. + */ + +package software.amazon.awssdk.enhanced.dynamodb.query.engine; + +import java.util.AbstractMap; +import java.util.ArrayList; +import java.util.Collections; +import java.util.HashMap; +import java.util.HashSet; +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Map; +import java.util.Set; +import java.util.concurrent.Callable; +import java.util.concurrent.CompletableFuture; +import java.util.concurrent.ExecutionException; +import java.util.concurrent.ExecutorService; +import java.util.concurrent.Executors; +import java.util.concurrent.Future; +import java.util.stream.Collectors; +import software.amazon.awssdk.annotations.SdkInternalApi; +import software.amazon.awssdk.core.async.SdkPublisher; +import software.amazon.awssdk.enhanced.dynamodb.DynamoDbAsyncTable; +import software.amazon.awssdk.enhanced.dynamodb.DynamoDbEnhancedAsyncClient; +import software.amazon.awssdk.enhanced.dynamodb.DynamoDbTable; +import software.amazon.awssdk.enhanced.dynamodb.MappedTableResource; +import software.amazon.awssdk.enhanced.dynamodb.TableSchema; +import software.amazon.awssdk.enhanced.dynamodb.model.QueryEnhancedRequest; +import software.amazon.awssdk.enhanced.dynamodb.model.ScanEnhancedRequest; +import software.amazon.awssdk.enhanced.dynamodb.query.condition.ConditionEvaluator; +import software.amazon.awssdk.enhanced.dynamodb.query.enums.ExecutionMode; +import software.amazon.awssdk.enhanced.dynamodb.query.enums.JoinType; +import software.amazon.awssdk.enhanced.dynamodb.query.result.EnhancedQueryRow; +import software.amazon.awssdk.enhanced.dynamodb.query.spec.AggregateSpec; +import software.amazon.awssdk.enhanced.dynamodb.query.spec.QueryExpressionSpec; +import software.amazon.awssdk.services.dynamodb.DynamoDbAsyncClient; +import software.amazon.awssdk.services.dynamodb.model.AttributeValue; +import software.amazon.awssdk.services.dynamodb.model.QueryRequest; +import software.amazon.awssdk.services.dynamodb.model.QueryResponse; +import software.amazon.awssdk.services.dynamodb.model.ScanRequest; +import software.amazon.awssdk.services.dynamodb.model.ScanResponse; + +/** + * Default async implementation of {@link QueryExpressionAsyncEngine}. Executes base Query/Scan, optional join, and aggregation by + * draining async publishers where needed, then applying the same in-memory rules as the sync engine. + *

+ * Structure: + *

    + *
  • {@link #execute} → {@link #executeJoinAsync}, {@link #executeAggregationAsync} or {@link #executeBaseOnlyAsync}
    • + *
  • Simple join: {@link #collectJoinBasePairsForAsync}, {@link #materializeJoinRowsFromCacheAsync}, + * {@link #drainPublisherToList}; joined-side loads via {@link JoinedTableObjectMapAsyncFetcher}
    • + *
  • Join + aggregation: {@link #executeJoinWithAggregationAsync} with + * {@link #collectJoinAggregationBaseRowsAndEmptyBuckets} and {@link #processBaseRowsIntoAggBucketsAsync} or + * {@link #queryAndAggregateDirectAsync}
    • + *
  • Row-shape helpers: {@link JoinRowAliases}; aggregation/sort: {@link QueryEngineSupport}
    • + *
+ */ +@SdkInternalApi +public final class DefaultQueryExpressionAsyncEngine implements QueryExpressionAsyncEngine { + + private static final ExecutorService ASYNC_JOIN_EXECUTOR = Executors.newFixedThreadPool(200); + + private final DynamoDbEnhancedAsyncClient enhancedClient; + private final JoinedTableObjectMapAsyncFetcher joinFetcher; + + public DefaultQueryExpressionAsyncEngine(DynamoDbEnhancedAsyncClient enhancedClient) { + this.enhancedClient = enhancedClient; + this.joinFetcher = new JoinedTableObjectMapAsyncFetcher(enhancedClient.dynamoDbAsyncClient(), ASYNC_JOIN_EXECUTOR); + } + + @Override + @SuppressWarnings("unchecked") + public SdkPublisher execute(QueryExpressionSpec spec) { + if (spec.hasJoin()) { + return executeJoinAsync(spec); + } + if (!spec.aggregates().isEmpty() && !spec.groupByAttributes().isEmpty()) { + return executeAggregationAsync(spec); + } + return executeBaseOnlyAsync(spec); + } + + // ---- base-only -------------------------------------------------- + + @SuppressWarnings("unchecked") + private SdkPublisher executeBaseOnlyAsync(QueryExpressionSpec spec) { + MappedTableResource baseTable = spec.baseTable(); + TableSchema baseSchema = (TableSchema) baseTable.tableSchema(); + Integer limit = spec.limit(); + int maxItems = limit != null ? limit : Integer.MAX_VALUE; + + SdkPublisher baseItems; + if (spec.keyCondition() != null) { + QueryEnhancedRequest.Builder reqBuilder = QueryEnhancedRequest.builder() + .queryConditional(spec.keyCondition()); + if (limit != null) { + reqBuilder.limit(limit); + } + if (spec.projectAttributes() != null && !spec.projectAttributes().isEmpty()) { + reqBuilder.attributesToProject(spec.projectAttributes().toArray(new String[0])); + } + baseItems = baseTableItems(baseTable, reqBuilder.build(), null); + } else if (spec.executionMode() == ExecutionMode.ALLOW_SCAN) { + ScanEnhancedRequest.Builder scanBuilder = ScanEnhancedRequest.builder(); + if (limit != null) { + scanBuilder.limit(limit); + } + if (spec.projectAttributes() != null && !spec.projectAttributes().isEmpty()) { + scanBuilder.attributesToProject(spec.projectAttributes()); + } + baseItems = baseTableItems(baseTable, null, scanBuilder.build()); + } else { + return SdkPublisher.fromIterable(Collections.emptyList()); + } + return baseItems.flatMapIterable(item -> toBaseRows(item, baseSchema, spec)) + .limit(maxItems); + } + + @SuppressWarnings("unchecked") + private SdkPublisher baseTableItems(MappedTableResource baseTable, + QueryEnhancedRequest queryRequest, + ScanEnhancedRequest scanRequest) { + if (baseTable instanceof DynamoDbAsyncTable) { + DynamoDbAsyncTable asyncTable = (DynamoDbAsyncTable) baseTable; + if (queryRequest != null) { + return asyncTable.query(queryRequest).items(); + } + return asyncTable.scan(scanRequest).items(); + } + DynamoDbTable syncTable = (DynamoDbTable) baseTable; + if (queryRequest != null) { + return SdkPublisher.fromIterable(syncTable.query(queryRequest).items()); + } + return SdkPublisher.fromIterable(syncTable.scan(scanRequest).items()); + } + + private static List toBaseRows(Object item, TableSchema baseSchema, QueryExpressionSpec spec) { + Map objectMap = AttributeValueConversion.toObjectMap(baseSchema.itemToMap(item, false)); + if (!ConditionEvaluator.evaluate(spec.where(), objectMap)) { + return Collections.emptyList(); + } + Map> itemsByAlias = + Collections.singletonMap(QueryEngineSupport.BASE_ALIAS, objectMap); + return Collections.singletonList(EnhancedQueryRow.builder() + .itemsByAlias(itemsByAlias) + .build()); + } + + // ---- join (no aggregation) -------------------------------------- + + @SuppressWarnings("unchecked") + private SdkPublisher executeJoinAsync(QueryExpressionSpec spec) { + if (!spec.aggregates().isEmpty() && !spec.groupByAttributes().isEmpty()) { + return executeJoinWithAggregationAsync(spec); + } + + MappedTableResource baseTable = spec.baseTable(); + MappedTableResource joinedTable = spec.joinedTable(); + TableSchema baseSchema = (TableSchema) baseTable.tableSchema(); + String baseJoinAttr = spec.leftJoinKey(); + String joinedJoinAttr = spec.rightJoinKey(); + JoinType joinType = spec.joinType(); + Integer limit = spec.limit(); + + if (spec.keyCondition() == null && spec.executionMode() != ExecutionMode.ALLOW_SCAN) { + return SdkPublisher.fromIterable(Collections.emptyList()); + } + + QueryEnhancedRequest queryReq; + ScanEnhancedRequest scanReq; + if (spec.keyCondition() != null) { + int maxPage = QueryEngineSupport.MAX_BASE_PAGE_SIZE; + int pageLimit = limit != null ? Math.min(limit, maxPage) : maxPage; + QueryEnhancedRequest.Builder reqBuilder = QueryEnhancedRequest.builder() + .queryConditional(spec.keyCondition()) + .limit(pageLimit); + if (spec.projectAttributes() != null && !spec.projectAttributes().isEmpty()) { + reqBuilder.attributesToProject(spec.projectAttributes().toArray(new String[0])); + } + queryReq = reqBuilder.build(); + scanReq = null; + } else { + int maxPage = QueryEngineSupport.MAX_BASE_PAGE_SIZE; + int pageLimit = limit != null ? Math.min(limit, maxPage) : maxPage; + ScanEnhancedRequest.Builder scanBuilder = ScanEnhancedRequest.builder() + .limit(pageLimit); + if (spec.projectAttributes() != null && !spec.projectAttributes().isEmpty()) { + scanBuilder.attributesToProject(spec.projectAttributes()); + } + scanReq = scanBuilder.build(); + queryReq = null; + } + SdkPublisher baseItems = baseTableItems(baseTable, queryReq, scanReq); + List baseItemsList = drainPublisherToList(baseItems); + + List, Object>> baseRowsWithKeys = new ArrayList<>(); + List rows = new ArrayList<>(); + Set keysWithBase = new HashSet<>(); + collectJoinBasePairsForAsync(baseItemsList, baseSchema, baseJoinAttr, joinType, spec, limit, + baseRowsWithKeys, rows, keysWithBase); + + Set distinctKeys = baseRowsWithKeys.stream().map(Map.Entry::getValue).collect(Collectors.toSet()); + Map>> joinMap = + joinFetcher.resolveAndFetchJoinedObjectMaps(joinedTable, distinctKeys, joinedJoinAttr); + + materializeJoinRowsFromCacheAsync(baseRowsWithKeys, joinMap, joinType, spec, limit, rows); + + if (spec.keyCondition() == null && (joinType == JoinType.RIGHT || joinType == JoinType.FULL)) { + addRightSideOnlyRowsAsync(rows, joinedTable, joinedJoinAttr, keysWithBase, spec, limit); + } + int max = limit != null ? limit : Integer.MAX_VALUE; + List result = rows.size() > max ? new ArrayList<>(rows.subList(0, max)) : rows; + return SdkPublisher.fromIterable(result); + } + + /** + * Blocks until all base-table items from the async publisher are collected (same semantics as the previous inline + * {@code subscribe(...).get()}). + */ + private static List drainPublisherToList(SdkPublisher baseItems) { + List baseItemsList = new ArrayList<>(); + try { + baseItems.subscribe(baseItemsList::add).get(); + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + throw new RuntimeException(e); + } catch (ExecutionException e) { + throw new RuntimeException(e.getCause() != null ? e.getCause() : e); + } + return baseItemsList; + } + + /** + * Phase 1 of simple join: filter base rows, emit LEFT/FULL rows for null join keys, record distinct keys for fetch. + */ + private static void collectJoinBasePairsForAsync( + List baseItemsList, + TableSchema baseSchema, + String baseJoinAttr, + JoinType joinType, + QueryExpressionSpec spec, + Integer limit, + List, Object>> baseRowsWithKeys, + List rows, + Set keysWithBase) { + + for (Object baseItem : baseItemsList) { + if (limit != null && rows.size() >= limit) { + break; + } + Map baseMap = AttributeValueConversion.toObjectMap(baseSchema.itemToMap(baseItem, false)); + if (!ConditionEvaluator.evaluate(spec.filterBase(), baseMap)) { + continue; + } + Object joinKeyValue = baseMap.get(baseJoinAttr); + if (joinKeyValue != null) { + keysWithBase.add(joinKeyValue); + } + if (joinKeyValue == null) { + if (joinType == JoinType.LEFT || joinType == JoinType.FULL) { + rows.add(EnhancedQueryRow.builder() + .itemsByAlias(JoinRowAliases.leftOuterJoinRowWithEmptyJoined(baseMap)) + .build()); + } + continue; + } + baseRowsWithKeys.add(new AbstractMap.SimpleEntry<>(baseMap, joinKeyValue)); + } + } + + /** + * Phase 2: expand (base, join key) pairs using the pre-fetched joined-side cache into result rows. + */ + private static void materializeJoinRowsFromCacheAsync( + List, Object>> baseRowsWithKeys, + Map>> joinMap, + JoinType joinType, + QueryExpressionSpec spec, + Integer limit, + List rows) { + + for (Map.Entry, Object> e : baseRowsWithKeys) { + if (limit != null && rows.size() >= limit) { + break; + } + Map baseMap = e.getKey(); + Object joinKeyValue = e.getValue(); + List> joinedItems = joinMap.getOrDefault(joinKeyValue, Collections.emptyList()); + if (joinedItems.isEmpty()) { + if (joinType == JoinType.LEFT || joinType == JoinType.FULL) { + rows.add(EnhancedQueryRow.builder() + .itemsByAlias(JoinRowAliases.leftOuterJoinRowWithEmptyJoined(baseMap)) + .build()); + } + continue; + } + for (Map joinedMap : joinedItems) { + if (limit != null && rows.size() >= limit) { + break; + } + if (!ConditionEvaluator.evaluate(spec.filterJoined(), joinedMap)) { + continue; + } + if (!ConditionEvaluator.evaluate(spec.where(), joinedMap, baseMap)) { + continue; + } + rows.add(EnhancedQueryRow.builder() + .itemsByAlias(JoinRowAliases.innerJoinRow(baseMap, joinedMap)) + .build()); + } + } + } + + // ---- inline query + aggregation (for large key sets) --------------- + + /** + * Queries the joined table per-key and aggregates items inline as they arrive. Each per-key query task produces local + * aggregation buckets that are merged at the end. Avoids storing all joined items in memory (O(groups) instead of O(items)). + */ + @SuppressWarnings("unchecked") + private Map, Map> queryAndAggregateDirectAsync( + MappedTableResource joinedTable, + Set joinKeys, + String joinedJoinAttr, + Map>> baseRowsByJoinKey, + QueryExpressionSpec spec, + List groupByAttrs, + List aggregateSpecs, + JoinType joinType, + Integer limit) { + + DynamoDbAsyncClient asyncLowLevel = enhancedClient.dynamoDbAsyncClient(); + TableSchema joinedSchema = (TableSchema) joinedTable.tableSchema(); + String primaryPk = joinedSchema.tableMetadata().primaryPartitionKey(); + String indexName = primaryPk.equals(joinedJoinAttr) + ? null + : QueryEngineSupport.findIndexForAttribute(joinedSchema, joinedJoinAttr); + + List, Map>>> tasks = new ArrayList<>(); + List keyList = new ArrayList<>(joinKeys); + int cores = Math.max(1, Runtime.getRuntime().availableProcessors()); + int chunkSize = Math.max(1, (keyList.size() + cores - 1) / cores); + + for (int start = 0; start < keyList.size(); start += chunkSize) { + int chunkStart = start; + int chunkEnd = Math.min(start + chunkSize, keyList.size()); + tasks.add(() -> { + Map, Map> localBuckets = new LinkedHashMap<>(); + for (int ki = chunkStart; ki < chunkEnd; ki++) { + Object keyFinal = keyList.get(ki); + List> baseRows = baseRowsByJoinKey.getOrDefault( + keyFinal, Collections.emptyList()); + Map exclusiveStartKey = null; + do { + QueryRequest.Builder reqBuilder = + QueryRequest.builder() + .tableName(joinedTable.tableName()) + .keyConditionExpression("#k = :v") + .expressionAttributeNames(Collections.singletonMap("#k", joinedJoinAttr)) + .expressionAttributeValues(Collections.singletonMap( + ":v", AttributeValueConversion.toKeyAttributeValue(keyFinal))); + if (indexName != null) { + reqBuilder.indexName(indexName); + } + if (exclusiveStartKey != null) { + reqBuilder.exclusiveStartKey(exclusiveStartKey); + } + QueryResponse response; + try { + response = asyncLowLevel.query(reqBuilder.build()).get(); + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + throw new RuntimeException(e); + } catch (ExecutionException e) { + throw new RuntimeException(e.getCause() != null ? e.getCause() : e); + } + for (Map item : response.items()) { + Map joinedMap = AttributeValueConversion.toObjectMap(item); + if (!ConditionEvaluator.evaluate(spec.filterJoined(), joinedMap)) { + continue; + } + if (baseRows.isEmpty()) { + if (joinType == JoinType.LEFT || joinType == JoinType.FULL) { + List groupKey = QueryEngineSupport.buildGroupKey( + groupByAttrs, joinedMap, Collections.emptyMap()); + localBuckets.computeIfAbsent(groupKey, + k -> QueryEngineSupport.createEmptyBucket(aggregateSpecs)); + } + continue; + } + for (Map baseMap : baseRows) { + if (!ConditionEvaluator.evaluate(spec.where(), joinedMap, baseMap)) { + continue; + } + List groupKey = QueryEngineSupport.buildGroupKey( + groupByAttrs, joinedMap, baseMap); + Map bucket = localBuckets.computeIfAbsent( + groupKey, + k -> QueryEngineSupport.createEmptyBucket(aggregateSpecs)); + QueryEngineSupport.updateBucketTwoMap( + bucket, joinedMap, baseMap, aggregateSpecs); + } + } + exclusiveStartKey = response.lastEvaluatedKey().isEmpty() + ? null : response.lastEvaluatedKey(); + } while (exclusiveStartKey != null); + } + return localBuckets; + }); + } + + try { + List, Map>>> futures = ASYNC_JOIN_EXECUTOR.invokeAll(tasks); + Map, Map> merged = new LinkedHashMap<>(); + for (Future, Map>> f : futures) { + Map, Map> partial = f.get(); + for (Map.Entry, Map> e : partial.entrySet()) { + Map existing = merged.get(e.getKey()); + if (existing == null) { + merged.put(e.getKey(), e.getValue()); + } else { + QueryEngineSupport.mergeBucket(existing, e.getValue(), aggregateSpecs); + } + } + } + return merged; + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + throw new RuntimeException(e); + } catch (ExecutionException e) { + throw new RuntimeException(e.getCause() != null ? e.getCause() : e); + } + } + + // ---- join + aggregation ----------------------------------------- + + @SuppressWarnings("unchecked") + private SdkPublisher executeJoinWithAggregationAsync(QueryExpressionSpec spec) { + MappedTableResource baseTable = spec.baseTable(); + MappedTableResource joinedTable = spec.joinedTable(); + TableSchema baseSchema = (TableSchema) baseTable.tableSchema(); + String baseJoinAttr = spec.leftJoinKey(); + JoinType joinType = spec.joinType(); + Integer limit = spec.limit(); + List groupByAttrs = spec.groupByAttributes(); + List aggregateSpecs = spec.aggregates(); + + if (spec.keyCondition() == null && spec.executionMode() != ExecutionMode.ALLOW_SCAN) { + return SdkPublisher.fromIterable(Collections.emptyList()); + } + + // Phase 1: collect base items + QueryEnhancedRequest queryReq = null; + ScanEnhancedRequest scanReq = null; + int maxPage = QueryEngineSupport.MAX_BASE_PAGE_SIZE; + int pageLimit = limit != null ? Math.min(limit, maxPage) : maxPage; + if (spec.keyCondition() != null) { + queryReq = QueryEnhancedRequest.builder() + .queryConditional(spec.keyCondition()) + .limit(pageLimit) + .build(); + } else { + scanReq = ScanEnhancedRequest.builder() + .limit(pageLimit) + .build(); + } + SdkPublisher baseItems = baseTableItems(baseTable, queryReq, scanReq); + List baseItemsList = drainPublisherToList(baseItems); + + List, Object>> baseRowsWithKeys = new ArrayList<>(); + Set distinctJoinKeys = new HashSet<>(); + Map, Map> emptyBuckets = new LinkedHashMap<>(); + collectJoinAggregationBaseRowsAndEmptyBuckets( + baseItemsList, baseSchema, baseJoinAttr, joinType, spec, + groupByAttrs, aggregateSpecs, baseRowsWithKeys, distinctJoinKeys, emptyBuckets); + + // Phase 2+3: fetch joined items and aggregate + Integer bucketCreationLimit = spec.orderBy().isEmpty() ? limit : null; + Map, Map> buckets; + if (distinctJoinKeys.size() > QueryEngineSupport.INLINE_AGG_SCAN_THRESHOLD) { + Map>> baseRowsByJoinKey = new HashMap<>(); + for (Map.Entry, Object> entry : baseRowsWithKeys) { + baseRowsByJoinKey.computeIfAbsent(entry.getValue(), k -> new ArrayList<>()) + .add(entry.getKey()); + } + buckets = queryAndAggregateDirectAsync(joinedTable, distinctJoinKeys, spec.rightJoinKey(), + baseRowsByJoinKey, spec, groupByAttrs, aggregateSpecs, joinType, limit); + } else { + Map>> joinedObjectMaps = joinFetcher.resolveAndFetchJoinedObjectMaps( + joinedTable, distinctJoinKeys, spec.rightJoinKey()); + buckets = processBaseRowsIntoAggBucketsAsync( + baseRowsWithKeys, joinedObjectMaps, spec, groupByAttrs, aggregateSpecs, joinType, bucketCreationLimit); + } + + for (Map.Entry, Map> e : emptyBuckets.entrySet()) { + buckets.putIfAbsent(e.getKey(), e.getValue()); + } + + List rows = new ArrayList<>(); + if (spec.orderBy().isEmpty()) { + for (Map.Entry, Map> e : buckets.entrySet()) { + if (limit != null && rows.size() >= limit) { + break; + } + rows.add(QueryEngineSupport.aggregationRowFromBucket( + e.getValue(), aggregateSpecs, spec.projectAttributes())); + } + } else { + for (Map.Entry, Map> e : buckets.entrySet()) { + rows.add(QueryEngineSupport.aggregationRowFromBucket( + e.getValue(), aggregateSpecs, spec.projectAttributes())); + } + QueryEngineSupport.sortEnhancedQueryRows(rows, spec.orderBy()); + if (limit != null && rows.size() > limit) { + rows = new ArrayList<>(rows.subList(0, limit)); + } + } + return SdkPublisher.fromIterable(rows); + } + + /** + * Phase 1 for join+aggregation: qualifying base rows, distinct join keys, and LEFT/FULL buckets for null join keys. + */ + private static void collectJoinAggregationBaseRowsAndEmptyBuckets( + List baseItemsList, + TableSchema baseSchema, + String baseJoinAttr, + JoinType joinType, + QueryExpressionSpec spec, + List groupByAttrs, + List aggregateSpecs, + List, Object>> baseRowsWithKeys, + Set distinctJoinKeys, + Map, Map> emptyBuckets) { + + for (Object baseItem : baseItemsList) { + Map baseMap = AttributeValueConversion.toObjectMap(baseSchema.itemToMap(baseItem, false)); + if (!ConditionEvaluator.evaluate(spec.filterBase(), baseMap)) { + continue; + } + Object joinKeyValue = baseMap.get(baseJoinAttr); + if (joinKeyValue == null) { + if (joinType == JoinType.LEFT || joinType == JoinType.FULL) { + List groupKey = QueryEngineSupport.buildGroupKey( + groupByAttrs, baseMap, Collections.emptyMap()); + emptyBuckets.computeIfAbsent(groupKey, k -> { + Map b = QueryEngineSupport.createEmptyBucket(aggregateSpecs); + QueryEngineSupport.putRepresentativeBaseIfAbsent(b, baseMap); + return b; + }); + } + continue; + } + distinctJoinKeys.add(joinKeyValue); + baseRowsWithKeys.add(new AbstractMap.SimpleEntry<>(baseMap, joinKeyValue)); + } + } + + private static Map, Map> processBaseRowsIntoAggBucketsAsync( + List, Object>> baseRowsWithKeys, + Map>> joinedObjectMaps, + QueryExpressionSpec spec, + List groupByAttrs, + List aggregateSpecs, + JoinType joinType, + Integer limit) { + + int cores = Math.max(1, Runtime.getRuntime().availableProcessors()); + int totalRows = baseRowsWithKeys.size(); + if (totalRows == 0) { + return new LinkedHashMap<>(); + } + int chunkSize = Math.max(1, (totalRows + cores - 1) / cores); + + List, Map>>> tasks = new ArrayList<>(); + for (int start = 0; start < totalRows; start += chunkSize) { + int chunkStart = start; + int chunkEnd = Math.min(start + chunkSize, totalRows); + tasks.add(() -> { + Map, Map> localBuckets = new LinkedHashMap<>(); + for (int i = chunkStart; i < chunkEnd; i++) { + Map.Entry, Object> entry = baseRowsWithKeys.get(i); + Map baseMap = entry.getKey(); + Object joinKeyValue = entry.getValue(); + List> joinedMaps = joinedObjectMaps.getOrDefault(joinKeyValue, + Collections.emptyList()); + if (joinedMaps.isEmpty()) { + if (joinType == JoinType.LEFT || joinType == JoinType.FULL) { + List groupKey = QueryEngineSupport.buildGroupKey( + groupByAttrs, baseMap, Collections.emptyMap()); + localBuckets.computeIfAbsent(groupKey, k -> { + Map b = QueryEngineSupport.createEmptyBucket(aggregateSpecs); + QueryEngineSupport.putRepresentativeBaseIfAbsent(b, baseMap); + return b; + }); + } + continue; + } + for (Map joinedMap : joinedMaps) { + if (!ConditionEvaluator.evaluate(spec.filterJoined(), joinedMap)) { + continue; + } + if (!ConditionEvaluator.evaluate(spec.where(), joinedMap, baseMap)) { + continue; + } + List groupKey = QueryEngineSupport.buildGroupKey( + groupByAttrs, joinedMap, baseMap); + if (limit != null && !localBuckets.containsKey(groupKey) && localBuckets.size() >= limit) { + continue; + } + Map bucket = localBuckets.computeIfAbsent( + groupKey, + k -> QueryEngineSupport.createEmptyBucket(aggregateSpecs)); + QueryEngineSupport.updateBucketTwoMap(bucket, joinedMap, baseMap, aggregateSpecs); + } + } + return localBuckets; + }); + } + + try { + List, Map>>> futures = ASYNC_JOIN_EXECUTOR.invokeAll(tasks); + Map, Map> merged = new LinkedHashMap<>(); + for (Future, Map>> f : futures) { + Map, Map> partial = f.get(); + for (Map.Entry, Map> e : partial.entrySet()) { + Map existing = merged.get(e.getKey()); + if (existing == null) { + merged.put(e.getKey(), e.getValue()); + } else { + QueryEngineSupport.mergeBucket(existing, e.getValue(), aggregateSpecs); + } + } + } + return merged; + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + throw new RuntimeException(e); + } catch (ExecutionException e) { + throw new RuntimeException(e.getCause() != null ? e.getCause() : e); + } + } + + // ---- RIGHT/FULL orphans ----------------------------------------- + + private void addRightSideOnlyRowsAsync(List rows, + MappedTableResource joinedTable, + String joinedJoinAttr, + Set keysWithBase, + QueryExpressionSpec spec, + Integer limit) { + DynamoDbAsyncClient asyncLowLevel = enhancedClient.dynamoDbAsyncClient(); + String tableName = joinedTable.tableName(); + Map exclusiveStartKey = null; + do { + ScanRequest.Builder reqBuilder = + ScanRequest.builder() + .tableName(tableName); + if (exclusiveStartKey != null) { + reqBuilder.exclusiveStartKey(exclusiveStartKey); + } + ScanResponse response; + try { + response = asyncLowLevel.scan(reqBuilder.build()).get(); + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + throw new RuntimeException(e); + } catch (ExecutionException e) { + throw new RuntimeException(e.getCause() != null ? e.getCause() : e); + } + for (Map item : response.items()) { + if (limit != null && rows.size() >= limit) { + return; + } + Map joinedMap = AttributeValueConversion.toObjectMap(item); + Object joinKey = joinedMap.get(joinedJoinAttr); + if (joinKey != null && keysWithBase.contains(joinKey)) { + continue; + } + if (!ConditionEvaluator.evaluate(spec.filterJoined(), joinedMap)) { + continue; + } + if (!ConditionEvaluator.evaluate(spec.where(), joinedMap)) { + continue; + } + rows.add(EnhancedQueryRow.builder() + .itemsByAlias(JoinRowAliases.rightOuterJoinRowWithEmptyBase(joinedMap)) + .build()); + } + exclusiveStartKey = response.lastEvaluatedKey().isEmpty() + ? null : response.lastEvaluatedKey(); + } while (exclusiveStartKey != null); + } + + // ---- aggregation (no join) -------------------------------------- + + @SuppressWarnings("unchecked") + private SdkPublisher executeAggregationAsync(QueryExpressionSpec spec) { + List rows = new ArrayList<>(); + MappedTableResource baseTable = spec.baseTable(); + TableSchema baseSchema = (TableSchema) baseTable.tableSchema(); + + SdkPublisher baseItems; + if (spec.keyCondition() != null) { + QueryEnhancedRequest.Builder reqBuilder = QueryEnhancedRequest.builder() + .queryConditional(spec.keyCondition()); + if (spec.projectAttributes() != null && !spec.projectAttributes().isEmpty()) { + reqBuilder.attributesToProject(spec.projectAttributes().toArray(new String[0])); + } + baseItems = baseTableItems(baseTable, reqBuilder.build(), null); + } else if (spec.executionMode() == ExecutionMode.ALLOW_SCAN) { + ScanEnhancedRequest.Builder scanBuilder = ScanEnhancedRequest.builder(); + if (spec.projectAttributes() != null && !spec.projectAttributes().isEmpty()) { + scanBuilder.attributesToProject(spec.projectAttributes()); + } + baseItems = baseTableItems(baseTable, null, scanBuilder.build()); + } else { + return SdkPublisher.fromIterable(Collections.emptyList()); + } + + Map, Map> buckets = new LinkedHashMap<>(); + CompletableFuture done = baseItems.subscribe(item -> { + Map objectMap = AttributeValueConversion.toObjectMap(baseSchema.itemToMap(item, false)); + if (!ConditionEvaluator.evaluate(spec.where(), objectMap)) { + return; + } + List groupKey = QueryEngineSupport.buildGroupKey( + spec.groupByAttributes(), objectMap, Collections.emptyMap()); + Map bucket = buckets.computeIfAbsent( + groupKey, + k -> QueryEngineSupport.createEmptyBucket(spec.aggregates())); + QueryEngineSupport.updateBucket(bucket, objectMap, spec.aggregates()); + }); + try { + done.get(); + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + throw new RuntimeException(e); + } catch (ExecutionException e) { + throw new RuntimeException(e.getCause() != null ? e.getCause() : e); + } + if (spec.orderBy().isEmpty()) { + for (Map bucket : buckets.values()) { + rows.add(QueryEngineSupport.aggregationRowFromBucket( + bucket, spec.aggregates(), spec.projectAttributes())); + } + Integer limit = spec.limit(); + if (limit != null && rows.size() > limit) { + rows = new ArrayList<>(rows.subList(0, limit)); + } + } else { + for (Map bucket : buckets.values()) { + rows.add(QueryEngineSupport.aggregationRowFromBucket( + bucket, spec.aggregates(), spec.projectAttributes())); + } + QueryEngineSupport.sortEnhancedQueryRows(rows, spec.orderBy()); + Integer limit = spec.limit(); + if (limit != null && rows.size() > limit) { + rows = new ArrayList<>(rows.subList(0, limit)); + } + } + return SdkPublisher.fromIterable(rows); + } +} diff --git a/services-custom/dynamodb-enhanced/src/main/java/software/amazon/awssdk/enhanced/dynamodb/query/engine/DefaultQueryExpressionEngine.java b/services-custom/dynamodb-enhanced/src/main/java/software/amazon/awssdk/enhanced/dynamodb/query/engine/DefaultQueryExpressionEngine.java new file mode 100644 index 000000000000..8bf8481a59c0 --- /dev/null +++ b/services-custom/dynamodb-enhanced/src/main/java/software/amazon/awssdk/enhanced/dynamodb/query/engine/DefaultQueryExpressionEngine.java @@ -0,0 +1,775 @@ +/* + * Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"). + * You may not use this file except in compliance with the License. + * A copy of the License is located at + * + * http://aws.amazon.com/apache2.0 + * + * or in the "license" file accompanying this file. This file is distributed + * on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either + * express or implied. See the License for the specific language governing + * permissions and limitations under the License. + */ + +package software.amazon.awssdk.enhanced.dynamodb.query.engine; + +import java.util.AbstractMap; +import java.util.ArrayList; +import java.util.Collections; +import java.util.HashMap; +import java.util.HashSet; +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Map; +import java.util.Set; +import java.util.concurrent.Callable; +import java.util.concurrent.ExecutionException; +import java.util.concurrent.ExecutorService; +import java.util.concurrent.Executors; +import java.util.concurrent.Future; +import java.util.function.Function; +import java.util.stream.Collectors; +import software.amazon.awssdk.annotations.SdkInternalApi; +import software.amazon.awssdk.enhanced.dynamodb.DynamoDbEnhancedClient; +import software.amazon.awssdk.enhanced.dynamodb.DynamoDbTable; +import software.amazon.awssdk.enhanced.dynamodb.TableSchema; +import software.amazon.awssdk.enhanced.dynamodb.model.Page; +import software.amazon.awssdk.enhanced.dynamodb.model.QueryEnhancedRequest; +import software.amazon.awssdk.enhanced.dynamodb.model.ScanEnhancedRequest; +import software.amazon.awssdk.enhanced.dynamodb.query.condition.ConditionEvaluator; +import software.amazon.awssdk.enhanced.dynamodb.query.enums.ExecutionMode; +import software.amazon.awssdk.enhanced.dynamodb.query.enums.JoinType; +import software.amazon.awssdk.enhanced.dynamodb.query.result.DefaultEnhancedQueryResult; +import software.amazon.awssdk.enhanced.dynamodb.query.result.EnhancedQueryResult; +import software.amazon.awssdk.enhanced.dynamodb.query.result.EnhancedQueryRow; +import software.amazon.awssdk.enhanced.dynamodb.query.spec.AggregateSpec; +import software.amazon.awssdk.enhanced.dynamodb.query.spec.QueryExpressionSpec; +import software.amazon.awssdk.services.dynamodb.DynamoDbClient; +import software.amazon.awssdk.services.dynamodb.model.AttributeValue; +import software.amazon.awssdk.services.dynamodb.model.QueryRequest; +import software.amazon.awssdk.services.dynamodb.model.QueryResponse; +import software.amazon.awssdk.services.dynamodb.model.ScanRequest; +import software.amazon.awssdk.services.dynamodb.model.ScanResponse; + +/** + * Default implementation of {@link QueryExpressionEngine}. Interprets the spec, runs base Query (or Scan when allowed), optional + * join via Query on joined table, in-memory filter/aggregation, and produces {@link EnhancedQueryRow}s. + *

+ * Structure: + *

    + *
  • {@link #execute} → {@link #executeJoin}, {@link #executeAggregation}, or {@link #executeBaseOnly}
    • + *
  • Join without aggregation: {@link #executeJoin} uses {@link #mergeJoinCacheForDistinctKeys} and + * {@link #expandJoinPairsToRows} per page; joined-side data loads via {@link JoinedTableObjectMapSyncFetcher}
    • + *
  • Join + aggregation: {@link #executeJoinWithAggregation} (inline path {@link #queryAndAggregateDirect} or batched + * {@link #processBaseRowsIntoAggBuckets})
    • + *
  • Shared row-shape helpers: {@link JoinRowAliases}
    • + *
  • Aggregation/sort/limit: {@link QueryEngineSupport}
    • + *
+ */ +@SdkInternalApi +public final class DefaultQueryExpressionEngine implements QueryExpressionEngine { + + private static final ExecutorService JOIN_LOOKUP_EXECUTOR = Executors.newFixedThreadPool(200); + + private final DynamoDbEnhancedClient enhancedClient; + private final JoinedTableObjectMapSyncFetcher joinFetcher; + + public DefaultQueryExpressionEngine(DynamoDbEnhancedClient enhancedClient) { + this.enhancedClient = enhancedClient; + this.joinFetcher = new JoinedTableObjectMapSyncFetcher(enhancedClient.dynamoDbClient(), JOIN_LOOKUP_EXECUTOR); + } + + @Override + public EnhancedQueryResult execute(QueryExpressionSpec spec) { + if (spec.hasJoin()) { + return executeJoin(spec); + } + if (!spec.aggregates().isEmpty() && !spec.groupByAttributes().isEmpty()) { + return executeAggregation(spec); + } + return executeBaseOnly(spec); + } + + // ---- base-only -------------------------------------------------- + + @SuppressWarnings("unchecked") + private EnhancedQueryResult executeBaseOnly(QueryExpressionSpec spec) { + DynamoDbTable baseTable = (DynamoDbTable) spec.baseTable(); + TableSchema baseSchema = (TableSchema) baseTable.tableSchema(); + Integer limit = spec.limit(); + + if (spec.keyCondition() != null) { + QueryEnhancedRequest.Builder reqBuilder = QueryEnhancedRequest.builder() + .queryConditional(spec.keyCondition()); + if (limit != null) { + reqBuilder.limit(limit); + } + if (spec.projectAttributes() != null && !spec.projectAttributes().isEmpty()) { + reqBuilder.attributesToProject(spec.projectAttributes().toArray(new String[0])); + } + List rows = collectBaseRows(baseTable, baseSchema, reqBuilder.build(), spec, limit); + return new DefaultEnhancedQueryResult(rows); + } + + if (spec.executionMode() == ExecutionMode.ALLOW_SCAN) { + ScanEnhancedRequest.Builder scanBuilder = ScanEnhancedRequest.builder(); + if (limit != null) { + scanBuilder.limit(limit); + } + if (spec.projectAttributes() != null && !spec.projectAttributes().isEmpty()) { + scanBuilder.attributesToProject(spec.projectAttributes()); + } + List rows = collectBaseRowsFromScan(baseTable, baseSchema, scanBuilder.build(), spec, limit); + return new DefaultEnhancedQueryResult(rows); + } + + return new DefaultEnhancedQueryResult(Collections.emptyList()); + } + + @SuppressWarnings("unchecked") + private List collectBaseRows(DynamoDbTable baseTable, TableSchema baseSchema, + QueryEnhancedRequest request, QueryExpressionSpec spec, + Integer limit) { + List rows = new ArrayList<>(); + for (Page page : baseTable.query(request)) { + for (Object item : page.items()) { + if (limit != null && rows.size() >= limit) { + return rows; + } + Map objectMap = AttributeValueConversion.toObjectMap(baseSchema.itemToMap(item, false)); + if (!ConditionEvaluator.evaluate(spec.where(), objectMap)) { + continue; + } + rows.add(EnhancedQueryRow.builder() + .itemsByAlias(Collections.singletonMap(QueryEngineSupport.BASE_ALIAS, objectMap)) + .build()); + } + } + return rows; + } + + @SuppressWarnings("unchecked") + private List collectBaseRowsFromScan(DynamoDbTable baseTable, TableSchema baseSchema, + ScanEnhancedRequest request, QueryExpressionSpec spec, + Integer limit) { + List rows = new ArrayList<>(); + for (Page page : baseTable.scan(request)) { + for (Object item : page.items()) { + if (limit != null && rows.size() >= limit) { + return rows; + } + Map objectMap = AttributeValueConversion.toObjectMap(baseSchema.itemToMap(item, false)); + if (!ConditionEvaluator.evaluate(spec.where(), objectMap)) { + continue; + } + rows.add(EnhancedQueryRow.builder() + .itemsByAlias(Collections.singletonMap(QueryEngineSupport.BASE_ALIAS, objectMap)) + .build()); + } + } + return rows; + } + + // ---- join (no aggregation) -------------------------------------- + + @SuppressWarnings("unchecked") + private EnhancedQueryResult executeJoin(QueryExpressionSpec spec) { + if (!spec.aggregates().isEmpty() && !spec.groupByAttributes().isEmpty()) { + return executeJoinWithAggregation(spec); + } + + DynamoDbTable baseTable = (DynamoDbTable) spec.baseTable(); + DynamoDbTable joinedTable = (DynamoDbTable) spec.joinedTable(); + TableSchema baseSchema = (TableSchema) baseTable.tableSchema(); + String baseJoinAttr = spec.leftJoinKey(); + String joinedJoinAttr = spec.rightJoinKey(); + JoinType joinType = spec.joinType(); + Integer limit = spec.limit(); + + if (spec.keyCondition() == null && spec.executionMode() != ExecutionMode.ALLOW_SCAN) { + return new DefaultEnhancedQueryResult(Collections.emptyList()); + } + + List rows = new ArrayList<>(); + Set keysWithBase = new HashSet<>(); + Map>> globalJoinCache = new HashMap<>(); + Set alreadyFetchedKeys = new HashSet<>(); + + if (spec.keyCondition() != null) { + int maxPage = QueryEngineSupport.MAX_BASE_PAGE_SIZE; + int pageLimit = limit != null ? Math.min(limit, maxPage) : maxPage; + QueryEnhancedRequest.Builder reqBuilder = QueryEnhancedRequest.builder() + .queryConditional(spec.keyCondition()) + .limit(pageLimit); + if (spec.projectAttributes() != null && !spec.projectAttributes().isEmpty()) { + reqBuilder.attributesToProject(spec.projectAttributes().toArray(new String[0])); + } + QueryEnhancedRequest req = reqBuilder.build(); + for (Page page : baseTable.query(req)) { + List, Object>> baseRowsWithKeys = new ArrayList<>(); + for (Object baseItem : page.items()) { + Map baseMap = AttributeValueConversion.toObjectMap(baseSchema.itemToMap(baseItem, false)); + if (!ConditionEvaluator.evaluate(spec.filterBase(), baseMap)) { + continue; + } + Object joinKeyValue = baseMap.get(baseJoinAttr); + if (joinKeyValue != null) { + keysWithBase.add(joinKeyValue); + } + if (joinKeyValue == null) { + if (joinType == JoinType.LEFT || joinType == JoinType.FULL) { + rows.add(EnhancedQueryRow.builder() + .itemsByAlias(JoinRowAliases.leftOuterJoinRowWithEmptyJoined(baseMap)) + .build()); + } + continue; + } + baseRowsWithKeys.add(new AbstractMap.SimpleEntry<>(baseMap, joinKeyValue)); + } + if (limit != null && rows.size() >= limit) { + break; + } + mergeJoinCacheForDistinctKeys(baseRowsWithKeys, alreadyFetchedKeys, globalJoinCache, + joinedTable, joinedJoinAttr); + DefaultEnhancedQueryResult early = expandJoinPairsToRows( + baseRowsWithKeys, globalJoinCache, joinType, spec, limit, rows); + if (early != null) { + return early; + } + } + } else if (spec.executionMode() == ExecutionMode.ALLOW_SCAN) { + int maxPage = QueryEngineSupport.MAX_BASE_PAGE_SIZE; + int pageLimit = limit != null ? Math.min(limit, maxPage) : maxPage; + ScanEnhancedRequest.Builder scanBuilder = ScanEnhancedRequest.builder() + .limit(pageLimit); + if (spec.projectAttributes() != null && !spec.projectAttributes().isEmpty()) { + scanBuilder.attributesToProject(spec.projectAttributes()); + } + ScanEnhancedRequest scanReq = scanBuilder.build(); + for (Page page : baseTable.scan(scanReq)) { + List, Object>> baseRowsWithKeys = new ArrayList<>(); + for (Object baseItem : page.items()) { + Map baseMap = AttributeValueConversion.toObjectMap(baseSchema.itemToMap(baseItem, false)); + if (!ConditionEvaluator.evaluate(spec.filterBase(), baseMap)) { + continue; + } + Object joinKeyValue = baseMap.get(baseJoinAttr); + if (joinKeyValue != null) { + keysWithBase.add(joinKeyValue); + } + if (joinKeyValue == null) { + if (joinType == JoinType.LEFT || joinType == JoinType.FULL) { + rows.add(EnhancedQueryRow.builder() + .itemsByAlias(JoinRowAliases.leftOuterJoinRowWithEmptyJoined(baseMap)) + .build()); + } + continue; + } + baseRowsWithKeys.add(new AbstractMap.SimpleEntry<>(baseMap, joinKeyValue)); + } + if (limit != null && rows.size() >= limit) { + break; + } + mergeJoinCacheForDistinctKeys(baseRowsWithKeys, alreadyFetchedKeys, globalJoinCache, + joinedTable, joinedJoinAttr); + DefaultEnhancedQueryResult earlyScan = expandJoinPairsToRows( + baseRowsWithKeys, globalJoinCache, joinType, spec, limit, rows); + if (earlyScan != null) { + return earlyScan; + } + } + } + + if (spec.keyCondition() == null && (joinType == JoinType.RIGHT || joinType == JoinType.FULL)) { + addRightSideOnlyRows(rows, spec, joinedTable, joinedJoinAttr, keysWithBase, limit); + } + + return new DefaultEnhancedQueryResult(rows); + } + + /** + * For keys appearing in {@code baseRowsWithKeys} that have not been fetched yet, loads joined-side object maps and merges + * them into {@code globalJoinCache}. + */ + private void mergeJoinCacheForDistinctKeys( + List, Object>> baseRowsWithKeys, + Set alreadyFetchedKeys, + Map>> globalJoinCache, + DynamoDbTable joinedTable, + String joinedJoinAttr) { + + Set distinctKeys = baseRowsWithKeys.stream() + .map(Map.Entry::getValue).collect(Collectors.toSet()); + distinctKeys.removeAll(alreadyFetchedKeys); + if (!distinctKeys.isEmpty()) { + Map>> joinMap = + joinFetcher.resolveAndFetchJoinedObjectMaps(joinedTable, distinctKeys, joinedJoinAttr); + globalJoinCache.putAll(joinMap); + alreadyFetchedKeys.addAll(distinctKeys); + } + } + + /** + * Materializes join result rows for one base page. Returns a finished result if the row {@code limit} is hit; otherwise + * {@code null} so the caller continues paging. + */ + private DefaultEnhancedQueryResult expandJoinPairsToRows( + List, Object>> baseRowsWithKeys, + Map>> globalJoinCache, + JoinType joinType, + QueryExpressionSpec spec, + Integer limit, + List rows) { + + for (Map.Entry, Object> e : baseRowsWithKeys) { + if (limit != null && rows.size() >= limit) { + return new DefaultEnhancedQueryResult(rows); + } + Map baseMap = e.getKey(); + Object joinKeyValue = e.getValue(); + List> joinedItems = globalJoinCache.getOrDefault(joinKeyValue, Collections.emptyList()); + if (joinedItems.isEmpty()) { + if (joinType == JoinType.LEFT || joinType == JoinType.FULL) { + rows.add(EnhancedQueryRow.builder() + .itemsByAlias(JoinRowAliases.leftOuterJoinRowWithEmptyJoined(baseMap)) + .build()); + } + continue; + } + for (Map joinedMap : joinedItems) { + if (limit != null && rows.size() >= limit) { + return new DefaultEnhancedQueryResult(rows); + } + if (!ConditionEvaluator.evaluate(spec.filterJoined(), joinedMap)) { + continue; + } + if (!ConditionEvaluator.evaluate(spec.where(), joinedMap, baseMap)) { + continue; + } + rows.add(EnhancedQueryRow.builder() + .itemsByAlias(JoinRowAliases.innerJoinRow(baseMap, joinedMap)) + .build()); + } + } + return null; + } + + // ---- inline query + aggregation (for large key sets) --------------- + + /** + * Queries the joined table per-key and aggregates items inline as they arrive. Each per-key query task produces local + * aggregation buckets that are merged at the end. Avoids storing all joined items in memory (O(groups) instead of O(items)), + * eliminating GC pressure from millions of intermediate Map allocations. + */ + @SuppressWarnings("unchecked") + private Map, Map> queryAndAggregateDirect( + DynamoDbTable joinedTable, + Set joinKeys, + String joinedJoinAttr, + Map>> baseRowsByJoinKey, + QueryExpressionSpec spec, + List groupByAttrs, + List aggregateSpecs, + JoinType joinType, + Integer limit) { + + DynamoDbClient lowLevel = enhancedClient.dynamoDbClient(); + TableSchema joinedSchema = (TableSchema) joinedTable.tableSchema(); + String primaryPk = joinedSchema.tableMetadata().primaryPartitionKey(); + String indexName = primaryPk.equals(joinedJoinAttr) + ? null + : QueryEngineSupport.findIndexForAttribute(joinedSchema, joinedJoinAttr); + + List, Map>>> tasks = new ArrayList<>(); + List keyList = new ArrayList<>(joinKeys); + int cores = Math.max(1, Runtime.getRuntime().availableProcessors()); + int chunkSize = Math.max(1, (keyList.size() + cores - 1) / cores); + + for (int start = 0; start < keyList.size(); start += chunkSize) { + int chunkStart = start; + int chunkEnd = Math.min(start + chunkSize, keyList.size()); + tasks.add(() -> { + Map, Map> localBuckets = new LinkedHashMap<>(); + for (int ki = chunkStart; ki < chunkEnd; ki++) { + Object keyFinal = keyList.get(ki); + List> baseRows = baseRowsByJoinKey.getOrDefault( + keyFinal, Collections.emptyList()); + Map exclusiveStartKey = null; + do { + QueryRequest.Builder reqBuilder = + QueryRequest.builder() + .tableName(joinedTable.tableName()) + .keyConditionExpression("#k = :v") + .expressionAttributeNames(Collections.singletonMap("#k", joinedJoinAttr)) + .expressionAttributeValues(Collections.singletonMap( + ":v", AttributeValueConversion.toKeyAttributeValue(keyFinal))); + if (indexName != null) { + reqBuilder.indexName(indexName); + } + if (exclusiveStartKey != null) { + reqBuilder.exclusiveStartKey(exclusiveStartKey); + } + QueryResponse response = + lowLevel.query(reqBuilder.build()); + for (Map item : response.items()) { + Map joinedMap = AttributeValueConversion.toObjectMap(item); + if (!ConditionEvaluator.evaluate(spec.filterJoined(), joinedMap)) { + continue; + } + if (baseRows.isEmpty()) { + if (joinType == JoinType.LEFT || joinType == JoinType.FULL) { + List groupKey = QueryEngineSupport.buildGroupKey( + groupByAttrs, joinedMap, Collections.emptyMap()); + Function, Map> bucketFactory = + k -> QueryEngineSupport.createEmptyBucket(aggregateSpecs); + localBuckets.computeIfAbsent(groupKey, bucketFactory); + } + continue; + } + for (Map baseMap : baseRows) { + if (!ConditionEvaluator.evaluate(spec.where(), joinedMap, baseMap)) { + continue; + } + List groupKey = QueryEngineSupport.buildGroupKey( + groupByAttrs, joinedMap, baseMap); + Function, Map> bucketFactory = + k -> QueryEngineSupport.createEmptyBucket(aggregateSpecs); + Map bucket = localBuckets.computeIfAbsent(groupKey, bucketFactory); + QueryEngineSupport.updateBucketTwoMap(bucket, joinedMap, baseMap, aggregateSpecs); + } + } + exclusiveStartKey = response.lastEvaluatedKey().isEmpty() + ? null : response.lastEvaluatedKey(); + } while (exclusiveStartKey != null); + } + return localBuckets; + }); + } + + try { + List, Map>>> futures = JOIN_LOOKUP_EXECUTOR.invokeAll(tasks); + Map, Map> merged = new LinkedHashMap<>(); + for (Future, Map>> f : futures) { + Map, Map> partial = f.get(); + for (Map.Entry, Map> e : partial.entrySet()) { + Map existing = merged.get(e.getKey()); + if (existing == null) { + merged.put(e.getKey(), e.getValue()); + } else { + QueryEngineSupport.mergeBucket(existing, e.getValue(), aggregateSpecs); + } + } + } + return merged; + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + throw new RuntimeException(e); + } catch (ExecutionException e) { + throw new RuntimeException(e.getCause()); + } + } + + // ---- join + aggregation ----------------------------------------- + + @SuppressWarnings("unchecked") + private EnhancedQueryResult executeJoinWithAggregation(QueryExpressionSpec spec) { + DynamoDbTable baseTable = (DynamoDbTable) spec.baseTable(); + DynamoDbTable joinedTable = (DynamoDbTable) spec.joinedTable(); + TableSchema baseSchema = (TableSchema) baseTable.tableSchema(); + String baseJoinAttr = spec.leftJoinKey(); + JoinType joinType = spec.joinType(); + Integer limit = spec.limit(); + List groupByAttrs = spec.groupByAttributes(); + List aggregateSpecs = spec.aggregates(); + + if (spec.keyCondition() == null && spec.executionMode() != ExecutionMode.ALLOW_SCAN) { + return new DefaultEnhancedQueryResult(Collections.emptyList()); + } + + // Phase 1: collect all qualifying base rows and their join keys + List, Object>> baseRowsWithKeys = new ArrayList<>(); + Set distinctJoinKeys = new HashSet<>(); + Map, Map> emptyBuckets = new LinkedHashMap<>(); + + int maxPage = QueryEngineSupport.MAX_BASE_PAGE_SIZE; + int pageLimit = limit != null ? Math.min(limit, maxPage) : maxPage; + Iterable> basePages; + if (spec.keyCondition() != null) { + basePages = baseTable.query(QueryEnhancedRequest.builder() + .queryConditional(spec.keyCondition()) + .limit(pageLimit) + .build()); + } else { + basePages = baseTable.scan(ScanEnhancedRequest.builder() + .limit(pageLimit) + .build()); + } + + for (Page page : basePages) { + for (Object baseItem : page.items()) { + Map baseMap = AttributeValueConversion.toObjectMap(baseSchema.itemToMap(baseItem, false)); + if (!ConditionEvaluator.evaluate(spec.filterBase(), baseMap)) { + continue; + } + Object joinKeyValue = baseMap.get(baseJoinAttr); + if (joinKeyValue == null) { + if (joinType == JoinType.LEFT || joinType == JoinType.FULL) { + List groupKey = QueryEngineSupport.buildGroupKey(groupByAttrs, baseMap, Collections.emptyMap()); + emptyBuckets.computeIfAbsent(groupKey, k -> { + Map b = QueryEngineSupport.createEmptyBucket(aggregateSpecs); + QueryEngineSupport.putRepresentativeBaseIfAbsent(b, baseMap); + return b; + }); + } + continue; + } + distinctJoinKeys.add(joinKeyValue); + baseRowsWithKeys.add(new AbstractMap.SimpleEntry<>(baseMap, joinKeyValue)); + } + } + + // Phase 2+3: fetch joined items and aggregate + Integer bucketCreationLimit = spec.orderBy().isEmpty() ? limit : null; + Map, Map> buckets; + if (distinctJoinKeys.size() > QueryEngineSupport.INLINE_AGG_SCAN_THRESHOLD) { + Map>> baseRowsByJoinKey = new HashMap<>(); + for (Map.Entry, Object> entry : baseRowsWithKeys) { + baseRowsByJoinKey.computeIfAbsent(entry.getValue(), k -> new ArrayList<>()) + .add(entry.getKey()); + } + buckets = queryAndAggregateDirect(joinedTable, distinctJoinKeys, spec.rightJoinKey(), + baseRowsByJoinKey, spec, groupByAttrs, aggregateSpecs, joinType, limit); + } else { + Map>> joinedObjectMaps = joinFetcher.resolveAndFetchJoinedObjectMaps( + joinedTable, distinctJoinKeys, spec.rightJoinKey()); + buckets = processBaseRowsIntoAggBuckets( + baseRowsWithKeys, joinedObjectMaps, spec, groupByAttrs, aggregateSpecs, joinType, bucketCreationLimit); + } + + // Merge empty-key buckets (LEFT/FULL for null join keys) + for (Map.Entry, Map> e : emptyBuckets.entrySet()) { + buckets.putIfAbsent(e.getKey(), e.getValue()); + } + + List rows = new ArrayList<>(); + if (spec.orderBy().isEmpty()) { + for (Map.Entry, Map> e : buckets.entrySet()) { + if (limit != null && rows.size() >= limit) { + break; + } + rows.add(QueryEngineSupport.aggregationRowFromBucket( + e.getValue(), aggregateSpecs, spec.projectAttributes())); + } + } else { + for (Map.Entry, Map> e : buckets.entrySet()) { + rows.add(QueryEngineSupport.aggregationRowFromBucket( + e.getValue(), aggregateSpecs, spec.projectAttributes())); + } + QueryEngineSupport.sortEnhancedQueryRows(rows, spec.orderBy()); + if (limit != null && rows.size() > limit) { + rows = new ArrayList<>(rows.subList(0, limit)); + } + } + return new DefaultEnhancedQueryResult(rows); + } + + /** + * Splits base rows across available CPU cores, processes each chunk in parallel with its own local buckets, then merges the + * partial results. + */ + private static Map, Map> processBaseRowsIntoAggBuckets( + List, Object>> baseRowsWithKeys, + Map>> joinedObjectMaps, + QueryExpressionSpec spec, + List groupByAttrs, + List aggregateSpecs, + JoinType joinType, + Integer limit) { + + int cores = Math.max(1, Runtime.getRuntime().availableProcessors()); + int totalRows = baseRowsWithKeys.size(); + if (totalRows == 0) { + return new LinkedHashMap<>(); + } + int chunkSize = Math.max(1, (totalRows + cores - 1) / cores); + + List, Map>>> tasks = new ArrayList<>(); + for (int start = 0; start < totalRows; start += chunkSize) { + int chunkStart = start; + int chunkEnd = Math.min(start + chunkSize, totalRows); + tasks.add(() -> { + Map, Map> localBuckets = new LinkedHashMap<>(); + for (int i = chunkStart; i < chunkEnd; i++) { + Map.Entry, Object> entry = baseRowsWithKeys.get(i); + Map baseMap = entry.getKey(); + Object joinKeyValue = entry.getValue(); + List> joinedMaps = joinedObjectMaps.getOrDefault(joinKeyValue, + Collections.emptyList()); + if (joinedMaps.isEmpty()) { + if (joinType == JoinType.LEFT || joinType == JoinType.FULL) { + List groupKey = QueryEngineSupport.buildGroupKey( + groupByAttrs, baseMap, Collections.emptyMap()); + localBuckets.computeIfAbsent(groupKey, k -> { + Map b = QueryEngineSupport.createEmptyBucket(aggregateSpecs); + QueryEngineSupport.putRepresentativeBaseIfAbsent(b, baseMap); + return b; + }); + } + continue; + } + for (Map joinedMap : joinedMaps) { + if (!ConditionEvaluator.evaluate(spec.filterJoined(), joinedMap)) { + continue; + } + if (!ConditionEvaluator.evaluate(spec.where(), joinedMap, baseMap)) { + continue; + } + List groupKey = QueryEngineSupport.buildGroupKey( + groupByAttrs, joinedMap, baseMap); + if (limit != null && !localBuckets.containsKey(groupKey) && localBuckets.size() >= limit) { + continue; + } + Function, Map> bucketFactory = + k -> QueryEngineSupport.createEmptyBucket(aggregateSpecs); + Map bucket = localBuckets.computeIfAbsent(groupKey, bucketFactory); + QueryEngineSupport.updateBucketTwoMap(bucket, joinedMap, baseMap, aggregateSpecs); + } + } + return localBuckets; + }); + } + + try { + List, Map>>> futures = JOIN_LOOKUP_EXECUTOR.invokeAll(tasks); + Map, Map> merged = new LinkedHashMap<>(); + for (Future, Map>> f : futures) { + Map, Map> partial = f.get(); + for (Map.Entry, Map> e : partial.entrySet()) { + Map existing = merged.get(e.getKey()); + if (existing == null) { + merged.put(e.getKey(), e.getValue()); + } else { + QueryEngineSupport.mergeBucket(existing, e.getValue(), aggregateSpecs); + } + } + } + return merged; + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + throw new RuntimeException(e); + } catch (ExecutionException e) { + throw new RuntimeException(e.getCause()); + } + } + + // ---- RIGHT/FULL orphans ----------------------------------------- + + private void addRightSideOnlyRows(List rows, + QueryExpressionSpec spec, + DynamoDbTable joinedTable, + String joinedJoinAttr, + Set keysWithBase, + Integer limit) { + DynamoDbClient lowLevel = enhancedClient.dynamoDbClient(); + String tableName = joinedTable.tableName(); + Map exclusiveStartKey = null; + do { + ScanRequest.Builder reqBuilder = + ScanRequest.builder() + .tableName(tableName); + if (exclusiveStartKey != null) { + reqBuilder.exclusiveStartKey(exclusiveStartKey); + } + ScanResponse response = lowLevel.scan(reqBuilder.build()); + for (Map item : response.items()) { + if (limit != null && rows.size() >= limit) { + return; + } + Map joinedMap = AttributeValueConversion.toObjectMap(item); + Object joinKey = joinedMap.get(joinedJoinAttr); + if (joinKey != null && keysWithBase.contains(joinKey)) { + continue; + } + if (!ConditionEvaluator.evaluate(spec.filterJoined(), joinedMap)) { + continue; + } + if (!ConditionEvaluator.evaluate(spec.where(), joinedMap)) { + continue; + } + rows.add(EnhancedQueryRow.builder() + .itemsByAlias(JoinRowAliases.rightOuterJoinRowWithEmptyBase(joinedMap)) + .build()); + } + exclusiveStartKey = response.lastEvaluatedKey().isEmpty() + ? null : response.lastEvaluatedKey(); + } while (exclusiveStartKey != null); + } + + // ---- aggregation (no join) -------------------------------------- + + @SuppressWarnings("unchecked") + private EnhancedQueryResult executeAggregation(QueryExpressionSpec spec) { + DynamoDbTable baseTable = (DynamoDbTable) spec.baseTable(); + TableSchema baseSchema = (TableSchema) baseTable.tableSchema(); + List groupByAttrs = spec.groupByAttributes(); + List aggregateSpecs = spec.aggregates(); + Integer limit = spec.limit(); + + List items = new ArrayList<>(); + if (spec.keyCondition() != null) { + QueryEnhancedRequest.Builder reqBuilder = QueryEnhancedRequest.builder() + .queryConditional(spec.keyCondition()); + if (spec.projectAttributes() != null && !spec.projectAttributes().isEmpty()) { + reqBuilder.attributesToProject(spec.projectAttributes().toArray(new String[0])); + } + for (Page page : baseTable.query(reqBuilder.build())) { + items.addAll(page.items()); + } + } else if (spec.executionMode() == ExecutionMode.ALLOW_SCAN) { + ScanEnhancedRequest.Builder scanBuilder = ScanEnhancedRequest.builder(); + if (spec.projectAttributes() != null && !spec.projectAttributes().isEmpty()) { + scanBuilder.attributesToProject(spec.projectAttributes()); + } + for (Page page : baseTable.scan(scanBuilder.build())) { + items.addAll(page.items()); + } + } + + Map, Map> buckets = new LinkedHashMap<>(); + for (Object item : items) { + Map objectMap = AttributeValueConversion.toObjectMap(baseSchema.itemToMap(item, false)); + if (!ConditionEvaluator.evaluate(spec.where(), objectMap)) { + continue; + } + List groupKey = QueryEngineSupport.buildGroupKey( + groupByAttrs, objectMap, Collections.emptyMap()); + Function, Map> bucketFactory = + k -> QueryEngineSupport.createEmptyBucket(aggregateSpecs); + Map bucket = buckets.computeIfAbsent(groupKey, bucketFactory); + QueryEngineSupport.updateBucket(bucket, objectMap, aggregateSpecs); + } + + List rows = new ArrayList<>(); + if (spec.orderBy().isEmpty()) { + for (Map bucket : buckets.values()) { + if (limit != null && rows.size() >= limit) { + break; + } + rows.add(QueryEngineSupport.aggregationRowFromBucket( + bucket, aggregateSpecs, spec.projectAttributes())); + } + } else { + for (Map bucket : buckets.values()) { + rows.add(QueryEngineSupport.aggregationRowFromBucket( + bucket, aggregateSpecs, spec.projectAttributes())); + } + QueryEngineSupport.sortEnhancedQueryRows(rows, spec.orderBy()); + if (limit != null && rows.size() > limit) { + rows = new ArrayList<>(rows.subList(0, limit)); + } + } + return new DefaultEnhancedQueryResult(rows); + } +} diff --git a/services-custom/dynamodb-enhanced/src/main/java/software/amazon/awssdk/enhanced/dynamodb/query/engine/JoinRowAliases.java b/services-custom/dynamodb-enhanced/src/main/java/software/amazon/awssdk/enhanced/dynamodb/query/engine/JoinRowAliases.java new file mode 100644 index 000000000000..2b45e51b4dd8 --- /dev/null +++ b/services-custom/dynamodb-enhanced/src/main/java/software/amazon/awssdk/enhanced/dynamodb/query/engine/JoinRowAliases.java @@ -0,0 +1,64 @@ +/* + * Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"). + * You may not use this file except in compliance with the License. + * A copy of the License is located at + * + * http://aws.amazon.com/apache2.0 + * + * or in the "license" file accompanying this file. This file is distributed + * on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either + * express or implied. See the License for the specific language governing + * permissions and limitations under the License. + */ + +package software.amazon.awssdk.enhanced.dynamodb.query.engine; + +import java.util.Collections; +import java.util.HashMap; +import java.util.Map; +import software.amazon.awssdk.annotations.SdkInternalApi; +import software.amazon.awssdk.enhanced.dynamodb.query.result.EnhancedQueryRow; + +/** + * Builds the {@code itemsByAlias} map for {@link EnhancedQueryRow} in join + * queries. Centralizes base/joined alias keys so engines stay readable. + */ +@SdkInternalApi +public final class JoinRowAliases { + + private JoinRowAliases() { + } + + /** + * Row for LEFT/FULL when the join key is null on the base side, or when no joined rows exist for the key: base attributes + * plus an empty joined map. + */ + public static Map> leftOuterJoinRowWithEmptyJoined(Map base) { + Map> itemsByAlias = new HashMap<>(2); + itemsByAlias.put(QueryEngineSupport.BASE_ALIAS, base); + itemsByAlias.put(QueryEngineSupport.JOINED_ALIAS, Collections.emptyMap()); + return itemsByAlias; + } + + /** + * Row when both base and joined attribute maps are present (inner match, or outer join with at least one joined row). + */ + public static Map> innerJoinRow(Map base, Map joined) { + Map> itemsByAlias = new HashMap<>(2); + itemsByAlias.put(QueryEngineSupport.BASE_ALIAS, base); + itemsByAlias.put(QueryEngineSupport.JOINED_ALIAS, joined); + return itemsByAlias; + } + + /** + * Row for RIGHT/FULL when scanning the joined table for keys with no base match: empty base map and joined attributes. + */ + public static Map> rightOuterJoinRowWithEmptyBase(Map joined) { + Map> itemsByAlias = new HashMap<>(2); + itemsByAlias.put(QueryEngineSupport.BASE_ALIAS, Collections.emptyMap()); + itemsByAlias.put(QueryEngineSupport.JOINED_ALIAS, joined); + return itemsByAlias; + } +} diff --git a/services-custom/dynamodb-enhanced/src/main/java/software/amazon/awssdk/enhanced/dynamodb/query/engine/JoinedTableObjectMapAsyncFetcher.java b/services-custom/dynamodb-enhanced/src/main/java/software/amazon/awssdk/enhanced/dynamodb/query/engine/JoinedTableObjectMapAsyncFetcher.java new file mode 100644 index 000000000000..050e8213bcfb --- /dev/null +++ b/services-custom/dynamodb-enhanced/src/main/java/software/amazon/awssdk/enhanced/dynamodb/query/engine/JoinedTableObjectMapAsyncFetcher.java @@ -0,0 +1,280 @@ +/* + * Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"). + * You may not use this file except in compliance with the License. + * A copy of the License is located at + * + * http://aws.amazon.com/apache2.0 + * + * or in the "license" file accompanying this file. This file is distributed + * on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either + * express or implied. See the License for the specific language governing + * permissions and limitations under the License. + */ + +package software.amazon.awssdk.enhanced.dynamodb.query.engine; + +import java.util.AbstractMap; +import java.util.ArrayList; +import java.util.Collections; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import java.util.Set; +import java.util.concurrent.Callable; +import java.util.concurrent.ExecutionException; +import java.util.concurrent.ExecutorService; +import java.util.concurrent.Future; +import software.amazon.awssdk.annotations.SdkInternalApi; +import software.amazon.awssdk.enhanced.dynamodb.MappedTableResource; +import software.amazon.awssdk.enhanced.dynamodb.TableSchema; +import software.amazon.awssdk.services.dynamodb.DynamoDbAsyncClient; +import software.amazon.awssdk.services.dynamodb.model.AttributeValue; +import software.amazon.awssdk.services.dynamodb.model.QueryRequest; +import software.amazon.awssdk.services.dynamodb.model.QueryResponse; +import software.amazon.awssdk.services.dynamodb.model.ScanRequest; +import software.amazon.awssdk.services.dynamodb.model.ScanResponse; + +/** + * Async counterpart to {@link JoinedTableObjectMapSyncFetcher}: loads joined-table rows for join-key values using + * DynamoDbAsyncClient. Used by the async query engine. + */ +@SdkInternalApi +public final class JoinedTableObjectMapAsyncFetcher { + + private final DynamoDbAsyncClient asyncLowLevel; + private final ExecutorService joinExecutor; + + public JoinedTableObjectMapAsyncFetcher(DynamoDbAsyncClient asyncLowLevel, ExecutorService joinExecutor) { + this.asyncLowLevel = asyncLowLevel; + this.joinExecutor = joinExecutor; + } + + /** + * Three-tier routing mirroring the sync engine: PK match -> parallel per-key query, GSI match -> parallel per-key index + * query, no index -> low-level parallel scan via DynamoDbAsyncClient. + */ + @SuppressWarnings("unchecked") + public Map>> resolveAndFetchJoinedObjectMaps( + MappedTableResource joinedTable, Set joinKeys, String joinedJoinAttr) { + if (joinKeys.isEmpty()) { + return Collections.emptyMap(); + } + + TableSchema joinedSchema = (TableSchema) joinedTable.tableSchema(); + String primaryPk = joinedSchema.tableMetadata().primaryPartitionKey(); + + if (primaryPk.equals(joinedJoinAttr)) { + return lowLevelQueryByPk(joinedTable.tableName(), primaryPk, joinKeys); + } + + String matchedIndex = QueryEngineSupport.findIndexForAttribute(joinedSchema, joinedJoinAttr); + if (matchedIndex != null) { + return lowLevelQueryByGsi(joinedTable.tableName(), matchedIndex, joinedJoinAttr, joinKeys); + } + + return parallelScanFallback(joinedTable.tableName(), joinKeys, joinedJoinAttr); + } + + /** + * Low-level per-key Query path when the join attribute matches the table's primary key. Executes inline for a single key to + * avoid thread pool overhead. + */ + private Map>> lowLevelQueryByPk( + String tableName, String pkAttr, Set joinKeys) { + List>>>> tasks = new ArrayList<>(); + for (Object key : joinKeys) { + Object keyFinal = key; + tasks.add(() -> queryByPkForKey(asyncLowLevel, tableName, pkAttr, keyFinal)); + } + if (tasks.size() == 1) { + return executeInline(tasks.get(0)); + } + return executeAndCollect(tasks); + } + + private static Map.Entry>> queryByPkForKey( + DynamoDbAsyncClient client, String tableName, String pkAttr, Object keyFinal) { + List> items = new ArrayList<>(); + Map exclusiveStartKey = null; + do { + QueryRequest.Builder reqBuilder = + QueryRequest.builder() + .tableName(tableName) + .keyConditionExpression("#k = :v") + .expressionAttributeNames(Collections.singletonMap("#k", pkAttr)) + .expressionAttributeValues(Collections.singletonMap( + ":v", AttributeValueConversion.toKeyAttributeValue(keyFinal))); + if (exclusiveStartKey != null) { + reqBuilder.exclusiveStartKey(exclusiveStartKey); + } + QueryResponse response; + try { + response = client.query(reqBuilder.build()).get(); + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + throw new RuntimeException(e); + } catch (ExecutionException e) { + throw new RuntimeException(e.getCause() != null ? e.getCause() : e); + } + for (Map item : response.items()) { + items.add(AttributeValueConversion.toObjectMap(item)); + } + exclusiveStartKey = response.lastEvaluatedKey().isEmpty() + ? null : response.lastEvaluatedKey(); + } while (exclusiveStartKey != null); + return new AbstractMap.SimpleEntry<>(keyFinal, items); + } + + /** + * Low-level per-key Query path when the join attribute matches a GSI partition key. Executes inline for a single key to avoid + * thread pool overhead. + */ + private Map>> lowLevelQueryByGsi( + String tableName, String indexName, String attrName, Set joinKeys) { + List>>>> tasks = new ArrayList<>(); + for (Object key : joinKeys) { + Object keyFinal = key; + tasks.add(() -> queryByGsiForKey(asyncLowLevel, tableName, indexName, attrName, keyFinal)); + } + if (tasks.size() == 1) { + return executeInline(tasks.get(0)); + } + return executeAndCollect(tasks); + } + + private static Map.Entry>> queryByGsiForKey( + DynamoDbAsyncClient client, String tableName, String indexName, String attrName, Object keyFinal) { + List> items = new ArrayList<>(); + Map exclusiveStartKey = null; + do { + QueryRequest.Builder reqBuilder = + QueryRequest.builder() + .tableName(tableName) + .indexName(indexName) + .keyConditionExpression("#k = :v") + .expressionAttributeNames(Collections.singletonMap("#k", attrName)) + .expressionAttributeValues(Collections.singletonMap( + ":v", AttributeValueConversion.toKeyAttributeValue(keyFinal))); + if (exclusiveStartKey != null) { + reqBuilder.exclusiveStartKey(exclusiveStartKey); + } + QueryResponse response; + try { + response = client.query(reqBuilder.build()).get(); + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + throw new RuntimeException(e); + } catch (ExecutionException e) { + throw new RuntimeException(e.getCause() != null ? e.getCause() : e); + } + for (Map item : response.items()) { + items.add(AttributeValueConversion.toObjectMap(item)); + } + exclusiveStartKey = response.lastEvaluatedKey().isEmpty() + ? null : response.lastEvaluatedKey(); + } while (exclusiveStartKey != null); + return new AbstractMap.SimpleEntry<>(keyFinal, items); + } + + /** + * Low-level parallel scan using DynamoDbAsyncClient directly. Bypasses the enhanced-client bean round-trip and goes straight + * from AV map -> Object map (one conversion instead of three). + */ + private Map>> parallelScanFallback( + String tableName, Set neededKeys, String joinedJoinAttr) { + int totalSegments = QueryEngineSupport.PARALLEL_SCAN_SEGMENTS; + + List>>>> tasks = new ArrayList<>(); + for (int seg = 0; seg < totalSegments; seg++) { + int segment = seg; + tasks.add(() -> { + Map>> partial = new HashMap<>(); + Map exclusiveStartKey = null; + do { + ScanRequest.Builder reqBuilder = + ScanRequest.builder() + .tableName(tableName) + .segment(segment) + .totalSegments(totalSegments); + if (exclusiveStartKey != null) { + reqBuilder.exclusiveStartKey(exclusiveStartKey); + } + ScanResponse response; + try { + response = asyncLowLevel.scan(reqBuilder.build()).get(); + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + throw new RuntimeException(e); + } catch (ExecutionException e) { + throw new RuntimeException(e.getCause() != null ? e.getCause() : e); + } + for (Map item : response.items()) { + AttributeValue keyAv = item.get(joinedJoinAttr); + if (keyAv == null) { + continue; + } + Object keyObj = AttributeValueConversion.toObject(keyAv); + if (keyObj != null && neededKeys.contains(keyObj)) { + partial.computeIfAbsent(keyObj, k -> new ArrayList<>()) + .add(AttributeValueConversion.toObjectMap(item)); + } + } + exclusiveStartKey = response.lastEvaluatedKey().isEmpty() + ? null : response.lastEvaluatedKey(); + } while (exclusiveStartKey != null); + return partial; + }); + } + + try { + List>>>> futures = joinExecutor.invokeAll(tasks); + Map>> merged = new HashMap<>(); + for (Future>>> f : futures) { + for (Map.Entry>> e : f.get().entrySet()) { + List> list = merged.computeIfAbsent(e.getKey(), k -> new ArrayList<>()); + list.addAll(e.getValue()); + } + } + return merged; + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + throw new RuntimeException(e); + } catch (ExecutionException e) { + throw new RuntimeException(e.getCause() != null ? e.getCause() : e); + } + } + + private Map>> executeAndCollect( + List>>>> tasks) { + try { + List>>>> futures = joinExecutor.invokeAll(tasks); + Map>> result = new HashMap<>(); + for (Future>>> f : futures) { + Map.Entry>> e = f.get(); + result.put(e.getKey(), e.getValue()); + } + return result; + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + throw new RuntimeException(e); + } catch (ExecutionException e) { + throw new RuntimeException(e.getCause() != null ? e.getCause() : e); + } + } + + private static Map>> executeInline( + Callable>>> task) { + try { + Map.Entry>> entry = task.call(); + Map>> result = new HashMap<>(); + result.put(entry.getKey(), entry.getValue()); + return result; + } catch (RuntimeException e) { + throw e; + } catch (Exception e) { + throw new RuntimeException(e); + } + } +} diff --git a/services-custom/dynamodb-enhanced/src/main/java/software/amazon/awssdk/enhanced/dynamodb/query/engine/JoinedTableObjectMapSyncFetcher.java b/services-custom/dynamodb-enhanced/src/main/java/software/amazon/awssdk/enhanced/dynamodb/query/engine/JoinedTableObjectMapSyncFetcher.java new file mode 100644 index 000000000000..f3bfe61b85ac --- /dev/null +++ b/services-custom/dynamodb-enhanced/src/main/java/software/amazon/awssdk/enhanced/dynamodb/query/engine/JoinedTableObjectMapSyncFetcher.java @@ -0,0 +1,259 @@ +/* + * Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"). + * You may not use this file except in compliance with the License. + * A copy of the License is located at + * + * http://aws.amazon.com/apache2.0 + * + * or in the "license" file accompanying this file. This file is distributed + * on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either + * express or implied. See the License for the specific language governing + * permissions and limitations under the License. + */ + +package software.amazon.awssdk.enhanced.dynamodb.query.engine; + +import java.util.AbstractMap; +import java.util.ArrayList; +import java.util.Collections; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import java.util.Set; +import java.util.concurrent.Callable; +import java.util.concurrent.ExecutionException; +import java.util.concurrent.ExecutorService; +import java.util.concurrent.Future; +import software.amazon.awssdk.annotations.SdkInternalApi; +import software.amazon.awssdk.enhanced.dynamodb.DynamoDbTable; +import software.amazon.awssdk.enhanced.dynamodb.TableSchema; +import software.amazon.awssdk.services.dynamodb.DynamoDbClient; +import software.amazon.awssdk.services.dynamodb.model.AttributeValue; +import software.amazon.awssdk.services.dynamodb.model.QueryRequest; +import software.amazon.awssdk.services.dynamodb.model.QueryResponse; +import software.amazon.awssdk.services.dynamodb.model.ScanRequest; +import software.amazon.awssdk.services.dynamodb.model.ScanResponse; + +/** + * Resolves how to load joined-table rows for a set of join-key values (PK query, GSI query, or parallel scan) and returns + * attribute maps suitable for in-memory join/aggregation. Used by the sync query engine. + */ +@SdkInternalApi +public final class JoinedTableObjectMapSyncFetcher { + + private final DynamoDbClient lowLevel; + private final ExecutorService joinExecutor; + + public JoinedTableObjectMapSyncFetcher(DynamoDbClient lowLevel, ExecutorService joinExecutor) { + this.lowLevel = lowLevel; + this.joinExecutor = joinExecutor; + } + + /** + * Three-tier routing for fetching joined-table items pre-converted to object maps: + *
    + *
  1. If joinedJoinAttr matches the table's PK: parallel per-key query() on the base table
    2. + *
  2. If joinedJoinAttr matches a GSI's PK: parallel per-key query() on that index
    3. + *
  3. Otherwise: low-level parallel scan via DynamoDbClient (one AV-to-Object conversion, no bean round-trip)
    4. + *
+ */ + @SuppressWarnings("unchecked") + public Map>> resolveAndFetchJoinedObjectMaps( + DynamoDbTable joinedTable, Set joinKeys, String joinedJoinAttr) { + if (joinKeys.isEmpty()) { + return Collections.emptyMap(); + } + + TableSchema joinedSchema = (TableSchema) joinedTable.tableSchema(); + String primaryPk = joinedSchema.tableMetadata().primaryPartitionKey(); + + if (primaryPk.equals(joinedJoinAttr)) { + return lowLevelQueryByPk(joinedTable.tableName(), primaryPk, joinKeys); + } + + String matchedIndex = QueryEngineSupport.findIndexForAttribute(joinedSchema, joinedJoinAttr); + if (matchedIndex != null) { + return lowLevelQueryByGsi(joinedTable.tableName(), matchedIndex, joinedJoinAttr, joinKeys); + } + + return parallelScanFallback(joinedTable.tableName(), joinKeys, joinedJoinAttr); + } + + /** + * Low-level per-key Query path when the join attribute matches the table's primary key. Executes inline for a single key to + * avoid thread pool overhead. + */ + private Map>> lowLevelQueryByPk( + String tableName, String pkAttr, Set joinKeys) { + List>>>> tasks = new ArrayList<>(); + for (Object key : joinKeys) { + Object keyFinal = key; + tasks.add(() -> queryByPkForKey(lowLevel, tableName, pkAttr, keyFinal)); + } + if (tasks.size() == 1) { + return executeInline(tasks.get(0)); + } + return executeAndCollect(tasks); + } + + private static Map.Entry>> queryByPkForKey( + DynamoDbClient client, String tableName, String pkAttr, Object keyFinal) { + List> items = new ArrayList<>(); + Map exclusiveStartKey = null; + do { + QueryRequest.Builder reqBuilder = + QueryRequest.builder() + .tableName(tableName) + .keyConditionExpression("#k = :v") + .expressionAttributeNames(Collections.singletonMap("#k", pkAttr)) + .expressionAttributeValues(Collections.singletonMap( + ":v", AttributeValueConversion.toKeyAttributeValue(keyFinal))); + if (exclusiveStartKey != null) { + reqBuilder.exclusiveStartKey(exclusiveStartKey); + } + QueryResponse response = client.query(reqBuilder.build()); + for (Map item : response.items()) { + items.add(AttributeValueConversion.toObjectMap(item)); + } + exclusiveStartKey = response.lastEvaluatedKey().isEmpty() + ? null : response.lastEvaluatedKey(); + } while (exclusiveStartKey != null); + return new AbstractMap.SimpleEntry<>(keyFinal, items); + } + + /** + * Low-level per-key Query path when the join attribute matches a GSI partition key. Executes inline for a single key to avoid + * thread pool overhead. + */ + private Map>> lowLevelQueryByGsi( + String tableName, String indexName, String attrName, Set joinKeys) { + List>>>> tasks = new ArrayList<>(); + for (Object key : joinKeys) { + Object keyFinal = key; + tasks.add(() -> queryByGsiForKey(lowLevel, tableName, indexName, attrName, keyFinal)); + } + if (tasks.size() == 1) { + return executeInline(tasks.get(0)); + } + return executeAndCollect(tasks); + } + + private static Map.Entry>> queryByGsiForKey( + DynamoDbClient client, String tableName, String indexName, String attrName, Object keyFinal) { + List> items = new ArrayList<>(); + Map exclusiveStartKey = null; + do { + QueryRequest.Builder reqBuilder = + QueryRequest.builder() + .tableName(tableName) + .indexName(indexName) + .keyConditionExpression("#k = :v") + .expressionAttributeNames(Collections.singletonMap("#k", attrName)) + .expressionAttributeValues(Collections.singletonMap( + ":v", AttributeValueConversion.toKeyAttributeValue(keyFinal))); + if (exclusiveStartKey != null) { + reqBuilder.exclusiveStartKey(exclusiveStartKey); + } + QueryResponse response = client.query(reqBuilder.build()); + for (Map item : response.items()) { + items.add(AttributeValueConversion.toObjectMap(item)); + } + exclusiveStartKey = response.lastEvaluatedKey().isEmpty() + ? null : response.lastEvaluatedKey(); + } while (exclusiveStartKey != null); + return new AbstractMap.SimpleEntry<>(keyFinal, items); + } + + /** + * Low-level parallel scan using DynamoDbClient directly. Bypasses the enhanced-client bean round-trip (response -> bean -> AV + * map -> Object map) and goes straight from AV map -> Object map (one conversion). + */ + private Map>> parallelScanFallback( + String tableName, Set neededKeys, String joinedJoinAttr) { + int totalSegments = QueryEngineSupport.PARALLEL_SCAN_SEGMENTS; + + List>>>> tasks = new ArrayList<>(); + for (int seg = 0; seg < totalSegments; seg++) { + int segment = seg; + tasks.add(() -> { + Map>> partial = new HashMap<>(); + Map exclusiveStartKey = null; + do { + ScanRequest.Builder reqBuilder = + ScanRequest.builder() + .tableName(tableName) + .segment(segment) + .totalSegments(totalSegments); + if (exclusiveStartKey != null) { + reqBuilder.exclusiveStartKey(exclusiveStartKey); + } + ScanResponse response = lowLevel.scan(reqBuilder.build()); + for (Map item : response.items()) { + AttributeValue keyAv = item.get(joinedJoinAttr); + if (keyAv == null) { + continue; + } + Object key = AttributeValueConversion.toObject(keyAv); + if (key != null && neededKeys.contains(key)) { + partial.computeIfAbsent(key, k -> new ArrayList<>()) + .add(AttributeValueConversion.toObjectMap(item)); + } + } + exclusiveStartKey = response.lastEvaluatedKey().isEmpty() + ? null : response.lastEvaluatedKey(); + } while (exclusiveStartKey != null); + return partial; + }); + } + + try { + List>>>> futures = joinExecutor.invokeAll(tasks); + Map>> merged = new HashMap<>(); + for (Future>>> f : futures) { + for (Map.Entry>> e : f.get().entrySet()) { + merged.computeIfAbsent(e.getKey(), k -> new ArrayList<>()).addAll(e.getValue()); + } + } + return merged; + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + throw new RuntimeException(e); + } catch (ExecutionException e) { + throw new RuntimeException(e.getCause()); + } + } + + private Map>> executeAndCollect( + List>>>> tasks) { + try { + List>>>> futures = joinExecutor.invokeAll(tasks); + Map>> result = new HashMap<>(); + for (Future>>> f : futures) { + Map.Entry>> e = f.get(); + result.put(e.getKey(), e.getValue()); + } + return result; + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + throw new RuntimeException(e); + } catch (ExecutionException e) { + throw new RuntimeException(e.getCause()); + } + } + + private static Map>> executeInline( + Callable>>> task) { + try { + Map.Entry>> entry = task.call(); + Map>> result = new HashMap<>(); + result.put(entry.getKey(), entry.getValue()); + return result; + } catch (RuntimeException e) { + throw e; + } catch (Exception e) { + throw new RuntimeException(e); + } + } +} diff --git a/services-custom/dynamodb-enhanced/src/main/java/software/amazon/awssdk/enhanced/dynamodb/query/engine/QueryEngineSupport.java b/services-custom/dynamodb-enhanced/src/main/java/software/amazon/awssdk/enhanced/dynamodb/query/engine/QueryEngineSupport.java new file mode 100644 index 000000000000..478101be1aa0 --- /dev/null +++ b/services-custom/dynamodb-enhanced/src/main/java/software/amazon/awssdk/enhanced/dynamodb/query/engine/QueryEngineSupport.java @@ -0,0 +1,359 @@ +/* + * Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"). + * You may not use this file except in compliance with the License. + * A copy of the License is located at + * + * http://aws.amazon.com/apache2.0 + * + * or in the "license" file accompanying this file. This file is distributed + * on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either + * express or implied. See the License for the specific language governing + * permissions and limitations under the License. + */ + +package software.amazon.awssdk.enhanced.dynamodb.query.engine; + +import java.math.BigDecimal; +import java.util.ArrayList; +import java.util.Collection; +import java.util.Collections; +import java.util.HashMap; +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Map; +import java.util.Optional; +import software.amazon.awssdk.annotations.SdkInternalApi; +import software.amazon.awssdk.enhanced.dynamodb.IndexMetadata; +import software.amazon.awssdk.enhanced.dynamodb.KeyAttributeMetadata; +import software.amazon.awssdk.enhanced.dynamodb.TableSchema; +import software.amazon.awssdk.enhanced.dynamodb.query.condition.ConditionEvaluator; +import software.amazon.awssdk.enhanced.dynamodb.query.enums.AggregationFunction; +import software.amazon.awssdk.enhanced.dynamodb.query.enums.SortDirection; +import software.amazon.awssdk.enhanced.dynamodb.query.result.EnhancedQueryRow; +import software.amazon.awssdk.enhanced.dynamodb.query.spec.AggregateSpec; +import software.amazon.awssdk.enhanced.dynamodb.query.spec.OrderBySpec; + +/** + * Shared helpers for the sync and async query engines. Contains aggregation bucket management, group-key building, index + * detection, and common constants. + */ +@SdkInternalApi +public final class QueryEngineSupport { + + public static final String BASE_ALIAS = "base"; + public static final String JOINED_ALIAS = "joined"; + + public static final int MAX_BASE_PAGE_SIZE = 1000; + public static final int INLINE_AGG_SCAN_THRESHOLD = 100; + public static final int PARALLEL_SCAN_SEGMENTS = Math.min(Runtime.getRuntime().availableProcessors() * 2, 16); + + /** + * Internal bucket key for the first-seen base table row map (join + aggregation). Not an aggregate output. + */ + static final String REPRESENTATIVE_BASE_KEY = "__ddbEnhancedQueryRepresentativeBase"; + + private QueryEngineSupport() { + } + + // ---- group key --------------------------------------------------- + + public static List buildGroupKey(List groupByAttrs, + Map primary, + Map secondary) { + List key = new ArrayList<>(groupByAttrs.size()); + for (String attr : groupByAttrs) { + key.add(ConditionEvaluator.lookupCombined(attr, primary, secondary)); + } + return key; + } + + // ---- aggregation bucket management ------------------------------- + + public static Map createEmptyBucket(List aggregateSpecs) { + Map bucket = new HashMap<>(); + for (AggregateSpec agg : aggregateSpecs) { + switch (agg.function()) { + case COUNT: + bucket.put(agg.outputName(), 0L); + break; + case SUM: + case AVG: + bucket.put(agg.outputName() + "_sum", BigDecimal.ZERO); + bucket.put(agg.outputName() + "_count", 0L); + break; + case MIN: + case MAX: + bucket.put(agg.outputName(), null); + break; + default: + break; + } + } + return bucket; + } + + public static void updateBucket(Map bucket, + Map row, + List aggregateSpecs) { + for (AggregateSpec agg : aggregateSpecs) { + Object val = row.get(agg.attribute()); + applyAggregate(bucket, agg, val); + } + putRepresentativeBaseIfAbsent(bucket, row); + } + + public static void updateBucketTwoMap(Map bucket, + Map primary, + Map secondary, + List aggregateSpecs) { + for (AggregateSpec agg : aggregateSpecs) { + Object val = ConditionEvaluator.lookupCombined(agg.attribute(), primary, secondary); + applyAggregate(bucket, agg, val); + } + putRepresentativeBaseIfAbsent(bucket, secondary); + } + + /** + * Stores the first base table row for this aggregate bucket (join path: secondary map is base). + */ + @SuppressWarnings("unchecked") + public static void putRepresentativeBaseIfAbsent(Map bucket, + Map baseMap) { + if (baseMap == null || bucket.containsKey(REPRESENTATIVE_BASE_KEY)) { + return; + } + bucket.put(REPRESENTATIVE_BASE_KEY, new LinkedHashMap<>(baseMap)); + } + + @SuppressWarnings("unchecked") + public static void mergeBucket(Map target, + Map source, + List aggregateSpecs) { + for (AggregateSpec agg : aggregateSpecs) { + String out = agg.outputName(); + switch (agg.function()) { + case COUNT: + target.put(out, ((Number) target.get(out)).longValue() + + ((Number) source.get(out)).longValue()); + break; + case SUM: + case AVG: + BigDecimal ts = (BigDecimal) target.get(out + "_sum"); + BigDecimal ss = (BigDecimal) source.get(out + "_sum"); + target.put(out + "_sum", ts.add(ss)); + long tc = ((Number) target.get(out + "_count")).longValue(); + long sc = ((Number) source.get(out + "_count")).longValue(); + target.put(out + "_count", tc + sc); + break; + case MIN: + Object tMin = target.get(out); + Object sMin = source.get(out); + if (tMin == null + || (sMin != null && compareForAggregate(sMin, tMin) < 0)) { + target.put(out, sMin); + } + break; + case MAX: + Object tMax = target.get(out); + Object sMax = source.get(out); + if (tMax == null + || (sMax != null && compareForAggregate(sMax, tMax) > 0)) { + target.put(out, sMax); + } + break; + default: + break; + } + } + if (!target.containsKey(REPRESENTATIVE_BASE_KEY) && source.containsKey(REPRESENTATIVE_BASE_KEY)) { + target.put(REPRESENTATIVE_BASE_KEY, + new LinkedHashMap<>((Map) source.get(REPRESENTATIVE_BASE_KEY))); + } + } + + public static Map aggregatesFromBucket(Map bucket, + List aggregateSpecs) { + Map aggregates = new HashMap<>(); + for (AggregateSpec agg : aggregateSpecs) { + String out = agg.outputName(); + Object val; + if (agg.function() == AggregationFunction.AVG) { + BigDecimal sum = (BigDecimal) bucket.get(out + "_sum"); + Long count = bucket.get(out + "_count") != null + ? ((Number) bucket.get(out + "_count")).longValue() : null; + val = (sum != null && count != null && count > 0) + ? (sum.doubleValue() / count.doubleValue()) + : null; + } else if (agg.function() == AggregationFunction.SUM) { + val = bucket.get(out + "_sum"); + } else { + val = bucket.get(out); + } + if (val != null) { + aggregates.put(out, val); + } + } + return aggregates; + } + + /** + * Builds an {@link EnhancedQueryRow} from an aggregation bucket, including representative base attributes when present. + */ + @SuppressWarnings("unchecked") + public static EnhancedQueryRow aggregationRowFromBucket(Map bucket, + List aggregateSpecs, + List projectAttributes) { + Map aggregates = aggregatesFromBucket(bucket, aggregateSpecs); + Map rawBase = (Map) bucket.get(REPRESENTATIVE_BASE_KEY); + Map> itemsByAlias = Collections.emptyMap(); + if (rawBase != null) { + itemsByAlias = Collections.singletonMap(BASE_ALIAS, projectAttributeMap(rawBase, projectAttributes)); + } + return EnhancedQueryRow.builder() + .itemsByAlias(itemsByAlias) + .aggregates(aggregates) + .build(); + } + + private static Map projectAttributeMap(Map source, + List projectAttributes) { + if (projectAttributes == null || projectAttributes.isEmpty()) { + return new LinkedHashMap<>(source); + } + Map out = new LinkedHashMap<>(); + for (String attr : projectAttributes) { + if (source.containsKey(attr)) { + out.put(attr, source.get(attr)); + } + } + return out; + } + + /** + * Applies {@link OrderBySpec} ordering to result rows (stable multi-key sort). + */ + public static void sortEnhancedQueryRows(List rows, List orderBy) { + if (orderBy == null || orderBy.isEmpty()) { + return; + } + rows.sort((a, b) -> compareRowsWithOrderBy(a, b, orderBy)); + } + + private static int compareRowsWithOrderBy(EnhancedQueryRow a, + EnhancedQueryRow b, + List orderBy) { + for (OrderBySpec spec : orderBy) { + int c = compareRowBySpec(a, b, spec); + if (c != 0) { + return c; + } + } + return 0; + } + + private static int compareRowBySpec(EnhancedQueryRow a, EnhancedQueryRow b, OrderBySpec spec) { + Object va; + Object vb; + if (spec.isByAggregate()) { + va = a.getAggregate(spec.attributeOrAggregateName()); + vb = b.getAggregate(spec.attributeOrAggregateName()); + } else { + va = a.getItem(BASE_ALIAS).get(spec.attributeOrAggregateName()); + vb = b.getItem(BASE_ALIAS).get(spec.attributeOrAggregateName()); + } + int cmp = compareNullableSortValues(va, vb); + if (spec.direction() == SortDirection.DESC) { + cmp = -cmp; + } + return cmp; + } + + private static int compareNullableSortValues(Object a, Object b) { + if (a == null && b == null) { + return 0; + } + if (a == null) { + return 1; + } + if (b == null) { + return -1; + } + if (a instanceof Number && b instanceof Number) { + double da = ((Number) a).doubleValue(); + double db = ((Number) b).doubleValue(); + return Double.compare(da, db); + } + return compareForAggregate(a, b); + } + + // ---- index detection --------------------------------------------- + + public static String findIndexForAttribute(TableSchema schema, String attrName) { + Collection indices = schema.tableMetadata().indices(); + for (IndexMetadata idx : indices) { + Optional pk = idx.partitionKey(); + if (pk.isPresent() && pk.get().name().equals(attrName)) { + String idxName = idx.name(); + if (!"$PRIMARY_INDEX".equals(idxName)) { + return idxName; + } + } + } + return null; + } + + // ---- internal helpers -------------------------------------------- + + @SuppressWarnings("unchecked") + static void applyAggregate(Map bucket, AggregateSpec agg, Object val) { + String out = agg.outputName(); + switch (agg.function()) { + case COUNT: + bucket.put(out, ((Number) bucket.get(out)).longValue() + 1); + break; + case SUM: + case AVG: + Number n = val instanceof Number ? (Number) val + : (val != null ? new BigDecimal(val.toString()) : null); + if (n != null) { + BigDecimal sum = (BigDecimal) bucket.get(out + "_sum"); + long count = ((Number) bucket.get(out + "_count")).longValue(); + bucket.put(out + "_sum", sum.add(n instanceof BigDecimal + ? (BigDecimal) n + : new BigDecimal(n.toString()))); + bucket.put(out + "_count", count + 1); + } + break; + case MIN: + Object curMin = bucket.get(out); + if (curMin == null + || (val != null && compareForAggregate(val, curMin) < 0)) { + bucket.put(out, val); + } + break; + case MAX: + Object curMax = bucket.get(out); + if (curMax == null + || (val != null && compareForAggregate(val, curMax) > 0)) { + bucket.put(out, val); + } + break; + default: + break; + } + } + + @SuppressWarnings("unchecked") + static int compareForAggregate(Object a, Object b) { + if (a instanceof Comparable && b instanceof Comparable) { + try { + return ((Comparable) a).compareTo(b); + } catch (ClassCastException e) { + return a.toString().compareTo(b.toString()); + } + } + return a.toString().compareTo(b.toString()); + } +} diff --git a/services-custom/dynamodb-enhanced/src/main/java/software/amazon/awssdk/enhanced/dynamodb/query/engine/QueryExpressionAsyncEngine.java b/services-custom/dynamodb-enhanced/src/main/java/software/amazon/awssdk/enhanced/dynamodb/query/engine/QueryExpressionAsyncEngine.java new file mode 100644 index 000000000000..10f72a59cad4 --- /dev/null +++ b/services-custom/dynamodb-enhanced/src/main/java/software/amazon/awssdk/enhanced/dynamodb/query/engine/QueryExpressionAsyncEngine.java @@ -0,0 +1,34 @@ +/* + * Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"). + * You may not use this file except in compliance with the License. + * A copy of the License is located at + * + * http://aws.amazon.com/apache2.0 + * + * or in the "license" file accompanying this file. This file is distributed + * on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either + * express or implied. See the License for the specific language governing + * permissions and limitations under the License. + */ + +package software.amazon.awssdk.enhanced.dynamodb.query.engine; + +import software.amazon.awssdk.annotations.SdkInternalApi; +import software.amazon.awssdk.core.async.SdkPublisher; +import software.amazon.awssdk.enhanced.dynamodb.query.result.EnhancedQueryRow; +import software.amazon.awssdk.enhanced.dynamodb.query.spec.QueryExpressionSpec; + +/** + * Async engine that executes a {@link QueryExpressionSpec} and produces a stream of + * {@link EnhancedQueryRow}s. + */ +@SdkInternalApi +public interface QueryExpressionAsyncEngine { + + /** + * Execute the given spec and return a publisher of result rows. + */ + SdkPublisher execute(QueryExpressionSpec spec); +} diff --git a/services-custom/dynamodb-enhanced/src/main/java/software/amazon/awssdk/enhanced/dynamodb/query/engine/QueryExpressionBuilder.java b/services-custom/dynamodb-enhanced/src/main/java/software/amazon/awssdk/enhanced/dynamodb/query/engine/QueryExpressionBuilder.java new file mode 100644 index 000000000000..ca19416495b6 --- /dev/null +++ b/services-custom/dynamodb-enhanced/src/main/java/software/amazon/awssdk/enhanced/dynamodb/query/engine/QueryExpressionBuilder.java @@ -0,0 +1,449 @@ +/* + * Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"). + * You may not use this file except in compliance with the License. + * A copy of the License is located at + * + * http://aws.amazon.com/apache2.0 + * + * or in the "license" file accompanying this file. This file is distributed + * on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either + * express or implied. See the License for the specific language governing + * permissions and limitations under the License. + */ + +package software.amazon.awssdk.enhanced.dynamodb.query.engine; + +import java.util.ArrayList; +import java.util.Arrays; +import java.util.List; +import software.amazon.awssdk.annotations.SdkInternalApi; +import software.amazon.awssdk.enhanced.dynamodb.DynamoDbAsyncTable; +import software.amazon.awssdk.enhanced.dynamodb.DynamoDbTable; +import software.amazon.awssdk.enhanced.dynamodb.MappedTableResource; +import software.amazon.awssdk.enhanced.dynamodb.model.QueryConditional; +import software.amazon.awssdk.enhanced.dynamodb.query.condition.Condition; +import software.amazon.awssdk.enhanced.dynamodb.query.enums.AggregationFunction; +import software.amazon.awssdk.enhanced.dynamodb.query.enums.ExecutionMode; +import software.amazon.awssdk.enhanced.dynamodb.query.enums.JoinType; +import software.amazon.awssdk.enhanced.dynamodb.query.enums.SortDirection; +import software.amazon.awssdk.enhanced.dynamodb.query.result.EnhancedQueryRow; +import software.amazon.awssdk.enhanced.dynamodb.query.spec.AggregateSpec; +import software.amazon.awssdk.enhanced.dynamodb.query.spec.OrderBySpec; +import software.amazon.awssdk.enhanced.dynamodb.query.spec.QueryExpressionSpec; + +/** + * Fluent builder for {@link QueryExpressionSpec}. Provides a declarative API for constructing enhanced queries that support + * single-table scans, cross-table joins, in-memory aggregations, filtering, ordering, and projections -- all executed + * transparently by the Enhanced Client query engine. + * + *

Quick reference

+ * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + *
MethodPurposeApplies to
{@link #from}Set the base (left) tableAll
{@link #join}Add right table + keysJoin
{@link #keyCondition}DynamoDB key condition (server)All
{@link #where}In-memory filter on final rowsAll
{@link #filterBase}Pre-join filter on base rowsJoin
{@link #filterJoined}Pre-join filter on joined rowsJoin
{@link #groupBy}Group by attribute(s)Aggregation
{@link #aggregate}SUM / COUNT / AVG / MIN / MAXAggregation
{@link #orderBy}Sort by row attributeAll
{@link #orderByAggregate}Sort by aggregate outputAggregation
{@link #project}Projection pushdownAll
{@link #executionMode}STRICT_KEY_ONLY or ALLOW_SCANAll
{@link #limit}Cap result rows / bucketsAll
+ * + *

Usage patterns

+ * + * Single-table query with key condition: + * 
{@code
+ * QueryExpressionSpec spec = QueryExpressionBuilder.from(customersTable)
+ *     .keyCondition(QueryConditional.keyEqualTo(k -> k.partitionValue("c1")))
+ *     .where(Condition.eq("status", "ACTIVE"))
+ *     .build();
+ * }
+ * + * Join + aggregation: + * 
{@code
+ * QueryExpressionSpec spec = QueryExpressionBuilder.from(customersTable)
+ *     .join(ordersTable, JoinType.INNER, "customerId", "customerId")
+ *     .keyCondition(QueryConditional.keyEqualTo(k -> k.partitionValue("c1")))
+ *     .filterBase(Condition.eq("region", "EU"))
+ *     .filterJoined(Condition.gte("amount", 50))
+ *     .where(Condition.eq("status", "ACTIVE"))
+ *     .groupBy("customerId")
+ *     .aggregate(AggregationFunction.SUM, "amount", "totalAmount")
+ *     .orderByAggregate("totalAmount", SortDirection.DESC)
+ *     .limit(100)
+ *     .build();
+ * }
+ * + * Nested attribute filtering: + * 
{@code
+ * QueryExpressionSpec spec = QueryExpressionBuilder.from(customersTable)
+ *     .executionMode(ExecutionMode.ALLOW_SCAN)
+ *     .where(Condition.eq("address.city", "Seattle"))
+ *     .build();
+ * }
+ * + *

Validation

+ * {@link #build()} validates option compatibility and throws {@link IllegalStateException} with a descriptive + * message for invalid combinations (e.g. {@code groupBy} without {@code aggregate}, or {@code filterBase} + * without a join). + * + * @see QueryExpressionSpec + * @see Condition + * @see AggregationFunction + * @see JoinType + */ +@SdkInternalApi +public final class QueryExpressionBuilder { + private final MappedTableResource baseTable; + private MappedTableResource joinedTable; + private JoinType joinType; + private String leftJoinKey; + private String rightJoinKey; + private QueryConditional baseKeyCondition; + private Condition condition; + private Condition baseTableCondition; + private Condition joinedTableCondition; + private final List groupByAttributes; + private final List aggregates; + private final List orderBy; + private final List projectAttributes; + private ExecutionMode executionMode; + private Integer limit; + + private QueryExpressionBuilder(MappedTableResource baseTable) { + this.baseTable = baseTable; + this.groupByAttributes = new ArrayList<>(); + this.aggregates = new ArrayList<>(); + this.orderBy = new ArrayList<>(); + this.projectAttributes = new ArrayList<>(); + } + + /** + * Start building an enhanced query from the given base (left) table using the synchronous API. The base table is the primary + * data source; all queries start here. + * + * @param baseTable the sync DynamoDbTable to query + * @return a new builder instance + */ + public static QueryExpressionBuilder from(DynamoDbTable baseTable) { + return new QueryExpressionBuilder(baseTable); + } + + /** + * Start building an enhanced query from the given base (left) table using the asynchronous API. + * + * @param baseTable the async DynamoDbAsyncTable to query + * @return a new builder instance + */ + public static QueryExpressionBuilder from(DynamoDbAsyncTable baseTable) { + return new QueryExpressionBuilder(baseTable); + } + + /** + * Add a join with a second table. The engine fetches rows from the base table first, then looks up matching rows in the + * joined table by matching {@code leftJoinKey} (on the base table) to {@code rightJoinKey} (on the joined table). + * Semantically equivalent to SQL: {@code FROM base JOIN joined ON base.leftKey = joined.rightKey}. + * + *

Supported join types: {@link JoinType#INNER}, {@link JoinType#LEFT}, {@link JoinType#RIGHT}, + * {@link JoinType#FULL}. + * + * @param joinedTable the right-side table to join + * @param joinType INNER, LEFT, RIGHT, or FULL + * @param leftJoinKey attribute name on the base table used for matching + * @param rightJoinKey attribute name on the joined table used for matching + * @return this builder + */ + public QueryExpressionBuilder join(MappedTableResource joinedTable, JoinType joinType, + String leftJoinKey, String rightJoinKey) { + this.joinedTable = joinedTable; + this.joinType = joinType; + this.leftJoinKey = leftJoinKey; + this.rightJoinKey = rightJoinKey; + return this; + } + + /** + * Sets the DynamoDB key condition for the base table. This is the only server-side filter in the entire enhanced query + * -- it is pushed down to DynamoDB as the {@code KeyConditionExpression} in a {@code Query} API call. + * + *

When set, the engine calls DynamoDB {@code Query}; when absent and + * {@link ExecutionMode#ALLOW_SCAN} is enabled, the engine falls back to {@code Scan}. + * + *

Example: + * 

{@code
+     * .keyCondition(QueryConditional.keyEqualTo(k -> k.partitionValue("customerId-1")))
+     * }
+ * + * @param keyCondition the DynamoDB key condition (partition key, optionally with sort key range) + * @return this builder + */ + public QueryExpressionBuilder keyCondition(QueryConditional keyCondition) { + this.baseKeyCondition = keyCondition; + return this; + } + + /** + * @deprecated Use {@link #keyCondition(QueryConditional)}. + */ + @Deprecated + public QueryExpressionBuilder baseKeyCondition(QueryConditional baseKeyCondition) { + return keyCondition(baseKeyCondition); + } + + /** + * In-memory filter applied to the final row set, after all joins and before aggregation. + * + *
    + *
  • Without join: filters base-table items after fetch.
    • + *
  • With join: filters the combined (base + joined) view. Both base and joined attributes + * are accessible in the condition.
    • + *
+ * + *

Supports dot-path syntax for nested attributes: {@code Condition.eq("address.city", "Seattle")}. + * + * @param where the in-memory filter condition + * @return this builder + */ + public QueryExpressionBuilder where(Condition where) { + this.condition = where; + return this; + } + + /** + * @deprecated Use {@link #where(Condition)}. + */ + @Deprecated + public QueryExpressionBuilder withCondition(Condition condition) { + return where(condition); + } + + /** + * In-memory filter applied to base (left) items before the join. Rows that do not match are excluded before the engine + * looks up their counterparts in the joined table, reducing work. + * + *

Only valid when a {@link #join} is configured; calling this without a join causes + * {@link #build()} to throw {@link IllegalStateException}. For single-table filtering, use {@link #where(Condition)} + * instead. + * + * @param filterBase the pre-join filter for the base table + * @return this builder + */ + public QueryExpressionBuilder filterBase(Condition filterBase) { + this.baseTableCondition = filterBase; + return this; + } + + /** + * @deprecated Use {@link #filterBase(Condition)}. + */ + @Deprecated + public QueryExpressionBuilder withBaseTableCondition(Condition condition) { + return filterBase(condition); + } + + /** + * In-memory filter applied to joined (right) items before they are combined with base rows. Eliminates non-matching + * joined rows early, reducing memory and processing time. + * + *

Only valid when a {@link #join} is configured; calling this without a join causes + * {@link #build()} to throw {@link IllegalStateException}. + * + * @param filterJoined the pre-join filter for the joined table + * @return this builder + */ + public QueryExpressionBuilder filterJoined(Condition filterJoined) { + this.joinedTableCondition = filterJoined; + return this; + } + + /** + * @deprecated Use {@link #filterJoined(Condition)}. + */ + @Deprecated + public QueryExpressionBuilder withJoinedTableCondition(Condition condition) { + return filterJoined(condition); + } + + /** + * Group results by one or more attributes for aggregation. Each unique combination of the specified attributes becomes a + * bucket; aggregation functions are computed per bucket. Semantically equivalent to SQL {@code GROUP BY}. + * + *

Requires at least one {@link #aggregate} call; {@link #build()} throws {@link IllegalStateException} + * if {@code groupBy} is set without any aggregate. + * + * @param attributes one or more attribute names to group by + * @return this builder + */ + public QueryExpressionBuilder groupBy(String... attributes) { + this.groupByAttributes.addAll(Arrays.asList(attributes)); + return this; + } + + /** + * Define an aggregation function to compute over the result set (or per group when {@link #groupBy} is set). Multiple + * aggregates can be added; each produces an output accessible via {@link EnhancedQueryRow#getAggregate(String)}. + * + *

Supported functions: {@link AggregationFunction#COUNT}, {@link AggregationFunction#SUM}, + * {@link AggregationFunction#AVG}, {@link AggregationFunction#MIN}, {@link AggregationFunction#MAX}. + * + * @param function the aggregation function + * @param attribute the source attribute name to aggregate over + * @param outputName the alias for the result (used in {@code getAggregate(outputName)}) + * @return this builder + */ + public QueryExpressionBuilder aggregate(AggregationFunction function, String attribute, String outputName) { + this.aggregates.add(AggregateSpec.builder() + .function(function) + .attribute(attribute) + .outputName(outputName) + .build()); + return this; + } + + /** + * Sort the final result set by a row attribute. Sorting is performed in-memory after all filtering and joining. Multiple + * {@code orderBy} / {@code orderByAggregate} calls are applied in the order specified (first call is the primary sort key). + * + * @param attribute the attribute name to sort by + * @param direction {@link SortDirection#ASC} or {@link SortDirection#DESC} + * @return this builder + */ + public QueryExpressionBuilder orderBy(String attribute, SortDirection direction) { + this.orderBy.add(OrderBySpec.byAttribute(attribute, direction)); + return this; + } + + /** + * Sort the final result set by an aggregate output value. Only meaningful when at least one {@link #aggregate} is defined. + * The {@code aggregateOutputName} must match an alias specified in a prior {@code aggregate()} call. + * + * @param aggregateOutputName the aggregate alias to sort by + * @param direction {@link SortDirection#ASC} or {@link SortDirection#DESC} + * @return this builder + */ + public QueryExpressionBuilder orderByAggregate(String aggregateOutputName, SortDirection direction) { + this.orderBy.add(OrderBySpec.byAggregate(aggregateOutputName, direction)); + return this; + } + + /** + * Restrict returned attributes (projection pushdown). The specified attribute names are forwarded to the DynamoDB + * {@code ProjectionExpression}, reducing the data transferred from the service. Nested paths (e.g. {@code "address.city"}) + * are supported. + * + * @param attributes one or more attribute names to include in results + * @return this builder + */ + public QueryExpressionBuilder project(String... attributes) { + this.projectAttributes.addAll(Arrays.asList(attributes)); + return this; + } + + /** + * Set the execution mode. Defaults to {@link ExecutionMode#STRICT_KEY_ONLY} which only uses DynamoDB {@code Query} and + * {@code BatchGetItem} operations. Set to {@link ExecutionMode#ALLOW_SCAN} to permit full-table scans when no key condition + * is provided. + * + * @param mode {@link ExecutionMode#STRICT_KEY_ONLY} or {@link ExecutionMode#ALLOW_SCAN} + * @return this builder + */ + public QueryExpressionBuilder executionMode(ExecutionMode mode) { + this.executionMode = mode; + return this; + } + + /** + * Cap the number of result rows returned. When {@link #groupBy} is used, this limits the number of aggregation buckets. + * Applied after all filtering, joining, and aggregation. + * + * @param limit maximum number of rows or buckets + * @return this builder + */ + public QueryExpressionBuilder limit(int limit) { + this.limit = limit; + return this; + } + + /** + * Build the immutable spec. Validates that options are consistent: + *

    + *
  • {@code groupBy} requires at least one {@code aggregate}
    • + *
  • {@code filterBase} and {@code filterJoined} require a {@code join}
    • + *
  • {@code joinType}, {@code leftJoinKey}, {@code rightJoinKey} require a {@code joinedTable}
    • + *
  • {@code joinedTable} requires {@code joinType}, {@code leftJoinKey}, and {@code rightJoinKey}
    • + *
  • {@code baseTable} is required
    • + *
+ * + * @throws IllegalStateException if incompatible options are specified + */ + public QueryExpressionSpec build() { + validate(); + return QueryExpressionSpec.builder() + .baseTable(baseTable) + .joinedTable(joinedTable) + .joinType(joinType) + .leftJoinKey(leftJoinKey) + .rightJoinKey(rightJoinKey) + .keyCondition(baseKeyCondition) + .where(condition) + .filterBase(baseTableCondition) + .filterJoined(joinedTableCondition) + .groupByAttributes(new ArrayList<>(groupByAttributes)) + .aggregates(new ArrayList<>(aggregates)) + .orderBy(new ArrayList<>(orderBy)) + .projectAttributes(new ArrayList<>(projectAttributes)) + .executionMode(executionMode) + .limit(limit) + .build(); + } + + private void validate() { + if (baseTable == null) { + throw new IllegalStateException("baseTable is required. Use QueryExpressionBuilder.from(table) to set it."); + } + + boolean hasJoin = joinedTable != null; + + if (!hasJoin && joinType != null) { + throw new IllegalStateException( + "joinType is set but no joinedTable was provided. Call .join(table, joinType, leftKey, rightKey)."); + } + if (!hasJoin && (leftJoinKey != null || rightJoinKey != null)) { + throw new IllegalStateException( + "leftJoinKey/rightJoinKey are set but no joinedTable was provided. " + + "Call .join(table, joinType, leftKey, rightKey)."); + } + if (hasJoin && joinType == null) { + throw new IllegalStateException( + "joinedTable is set but joinType is missing. Call .join(table, joinType, leftKey, rightKey)."); + } + if (hasJoin && (leftJoinKey == null || rightJoinKey == null)) { + throw new IllegalStateException( + "joinedTable is set but leftJoinKey or rightJoinKey is missing. " + + "Call .join(table, joinType, leftKey, rightKey)."); + } + + if (!hasJoin && baseTableCondition != null) { + throw new IllegalStateException( + "filterBase() is only applicable when a join is configured. " + + "For single-table filtering, use where() instead."); + } + if (!hasJoin && joinedTableCondition != null) { + throw new IllegalStateException( + "filterJoined() is only applicable when a join is configured."); + } + + if (!groupByAttributes.isEmpty() && aggregates.isEmpty()) { + throw new IllegalStateException( + "groupBy() requires at least one aggregate(). " + + "Add .aggregate(function, attribute, outputName) to define an aggregation."); + } + } +} diff --git a/services-custom/dynamodb-enhanced/src/main/java/software/amazon/awssdk/enhanced/dynamodb/query/engine/QueryExpressionEngine.java b/services-custom/dynamodb-enhanced/src/main/java/software/amazon/awssdk/enhanced/dynamodb/query/engine/QueryExpressionEngine.java new file mode 100644 index 000000000000..ef00025754dd --- /dev/null +++ b/services-custom/dynamodb-enhanced/src/main/java/software/amazon/awssdk/enhanced/dynamodb/query/engine/QueryExpressionEngine.java @@ -0,0 +1,53 @@ +/* + * Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"). + * You may not use this file except in compliance with the License. + * A copy of the License is located at + * + * http://aws.amazon.com/apache2.0 + * + * or in the "license" file accompanying this file. This file is distributed + * on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either + * express or implied. See the License for the specific language governing + * permissions and limitations under the License. + */ + +package software.amazon.awssdk.enhanced.dynamodb.query.engine; + +import java.util.function.Consumer; +import software.amazon.awssdk.annotations.SdkInternalApi; +import software.amazon.awssdk.enhanced.dynamodb.query.result.EnhancedQueryLatencyReport; +import software.amazon.awssdk.enhanced.dynamodb.query.result.EnhancedQueryResult; +import software.amazon.awssdk.enhanced.dynamodb.query.spec.QueryExpressionSpec; + +/** + * Engine that executes a {@link QueryExpressionSpec} against DynamoDB, producing an + * {@link EnhancedQueryResult}. + */ +@SdkInternalApi +public interface QueryExpressionEngine { + + /** + * Execute the given spec and return an iterable of result rows. + */ + EnhancedQueryResult execute(QueryExpressionSpec spec); + + /** + * Execute the given spec and optionally report latency. + * + * @param spec the query specification + * @param reportConsumer optional consumer for the latency report; may be null + * @return iterable of result rows + */ + default EnhancedQueryResult execute(QueryExpressionSpec spec, + Consumer reportConsumer) { + long start = System.nanoTime(); + EnhancedQueryResult result = execute(spec); + if (reportConsumer != null) { + long totalMs = (System.nanoTime() - start) / 1_000_000; + reportConsumer.accept(new EnhancedQueryLatencyReport(0L, 0L, 0L, totalMs)); + } + return result; + } +} diff --git a/services-custom/dynamodb-enhanced/src/main/java/software/amazon/awssdk/enhanced/dynamodb/query/enums/AggregationFunction.java b/services-custom/dynamodb-enhanced/src/main/java/software/amazon/awssdk/enhanced/dynamodb/query/enums/AggregationFunction.java new file mode 100644 index 000000000000..689d9d736b56 --- /dev/null +++ b/services-custom/dynamodb-enhanced/src/main/java/software/amazon/awssdk/enhanced/dynamodb/query/enums/AggregationFunction.java @@ -0,0 +1,28 @@ +/* + * Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"). + * You may not use this file except in compliance with the License. + * A copy of the License is located at + * + * http://aws.amazon.com/apache2.0 + * + * or in the "license" file accompanying this file. This file is distributed + * on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either + * express or implied. See the License for the specific language governing + * permissions and limitations under the License. + */ + +package software.amazon.awssdk.enhanced.dynamodb.query.enums; + +/** + * Aggregation functions supported by the complex query API. + */ +public enum AggregationFunction { + COUNT, + SUM, + AVG, + MIN, + MAX +} + diff --git a/services-custom/dynamodb-enhanced/src/main/java/software/amazon/awssdk/enhanced/dynamodb/query/enums/ExecutionMode.java b/services-custom/dynamodb-enhanced/src/main/java/software/amazon/awssdk/enhanced/dynamodb/query/enums/ExecutionMode.java new file mode 100644 index 000000000000..44a9aa9b20db --- /dev/null +++ b/services-custom/dynamodb-enhanced/src/main/java/software/amazon/awssdk/enhanced/dynamodb/query/enums/ExecutionMode.java @@ -0,0 +1,40 @@ +/* + * Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"). + * You may not use this file except in compliance with the License. + * A copy of the License is located at + * + * http://aws.amazon.com/apache2.0 + * + * or in the "license" file accompanying this file. This file is distributed + * on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either + * express or implied. See the License for the specific language governing + * permissions and limitations under the License. + */ + +package software.amazon.awssdk.enhanced.dynamodb.query.enums; + +/** + * Execution strategy for complex queries. Controls whether the engine may use a full table or index Scan. + *

+ * STRICT_KEY_ONLY (default): Only key-based operations are allowed—{@code Query} (with partition key + * and optional sort condition) and {@code BatchGetItem}. If the request cannot be satisfied with keys (e.g. no partition key + * supplied, or join attribute is not a key on the joined table), the engine must fail or return empty; it must not perform a + * Scan. + *

+ * ALLOW_SCAN: Same as above, but the engine may fall back to a full table or index Scan when + * there is no usable key or index. Use when you explicitly accept Scan's cost and latency for that query. + */ +public enum ExecutionMode { + /** + * Only Query and BatchGetItem; no Scan. Default. + */ + STRICT_KEY_ONLY, + + /** + * Query and BatchGetItem, plus optional Scan when no key/index is available. + */ + ALLOW_SCAN +} + diff --git a/services-custom/dynamodb-enhanced/src/main/java/software/amazon/awssdk/enhanced/dynamodb/query/enums/JoinType.java b/services-custom/dynamodb-enhanced/src/main/java/software/amazon/awssdk/enhanced/dynamodb/query/enums/JoinType.java new file mode 100644 index 000000000000..ddfce66b4cbe --- /dev/null +++ b/services-custom/dynamodb-enhanced/src/main/java/software/amazon/awssdk/enhanced/dynamodb/query/enums/JoinType.java @@ -0,0 +1,58 @@ +/* + * Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"). + * You may not use this file except in compliance with the License. + * A copy of the License is located at + * + * http://aws.amazon.com/apache2.0 + * + * or in the "license" file accompanying this file. This file is distributed + * on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either + * express or implied. See the License for the specific language governing + * permissions and limitations under the License. + */ + +package software.amazon.awssdk.enhanced.dynamodb.query.enums; + +/** + * Join types supported by the complex query API when joining a base table to a joined table on a common attribute (e.g. + * customerId). + * + *

    + *
  • INNER: Only rows that have a matching pair (base row and joined row with the same join key) + * are emitted. Base rows with no matching joined row, and joined rows with no matching base row, + * are omitted.
    • + *
  • LEFT: Every base row is emitted. When there is at least one joined row with the same join key, + * one result row per (base, joined) pair is emitted. When there is no matching joined row (or the base + * join key is null), a single row is emitted with the base and an empty joined side.
    • + *
  • RIGHT: Every joined row is emitted. When there is at least one base row with the same join key, + * those pairs are emitted during the base-driven phase. Joined rows whose join key never appeared on the + * base side are emitted as rows with an empty base and that joined row.
    • + *
  • FULL: Union of LEFT and RIGHT. Each base row appears at least once (with empty joined when no + * match); each joined row appears at least once (with empty base when no match); matching pairs appear + * as (base, joined) rows.
    • + *
+ */ +public enum JoinType { + /** + * Only matching (base, joined) pairs; no base-only or joined-only rows. + */ + INNER, + + /** + * Every base row; joined side empty when there is no match. + */ + LEFT, + + /** + * Every joined row; base side empty when there is no match. + */ + RIGHT, + + /** + * Every base and every joined row; unmatched sides are empty. + */ + FULL +} + diff --git a/services-custom/dynamodb-enhanced/src/main/java/software/amazon/awssdk/enhanced/dynamodb/query/enums/SortDirection.java b/services-custom/dynamodb-enhanced/src/main/java/software/amazon/awssdk/enhanced/dynamodb/query/enums/SortDirection.java new file mode 100644 index 000000000000..99b2238288d7 --- /dev/null +++ b/services-custom/dynamodb-enhanced/src/main/java/software/amazon/awssdk/enhanced/dynamodb/query/enums/SortDirection.java @@ -0,0 +1,25 @@ +/* + * Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"). + * You may not use this file except in compliance with the License. + * A copy of the License is located at + * + * http://aws.amazon.com/apache2.0 + * + * or in the "license" file accompanying this file. This file is distributed + * on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either + * express or implied. See the License for the specific language governing + * permissions and limitations under the License. + */ + +package software.amazon.awssdk.enhanced.dynamodb.query.enums; + +/** + * Sort direction for ordering attributes or aggregates. + */ +public enum SortDirection { + ASC, + DESC +} + diff --git a/services-custom/dynamodb-enhanced/src/main/java/software/amazon/awssdk/enhanced/dynamodb/query/package-info.java b/services-custom/dynamodb-enhanced/src/main/java/software/amazon/awssdk/enhanced/dynamodb/query/package-info.java new file mode 100644 index 000000000000..fafb88a9150b --- /dev/null +++ b/services-custom/dynamodb-enhanced/src/main/java/software/amazon/awssdk/enhanced/dynamodb/query/package-info.java @@ -0,0 +1,105 @@ +/* + * Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"). + * You may not use this file except in compliance with the License. + * A copy of the License is located at + * + * http://aws.amazon.com/apache2.0 + * + * or in the "license" file accompanying this file. This file is distributed + * on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either + * express or implied. See the License for the specific language governing + * permissions and limitations under the License. + */ + +/** + * Enhanced query API for the DynamoDB Enhanced Client. + *

+ * Provides SQL-like join and aggregation capabilities on top of DynamoDB. Queries are described by a + * {@link software.amazon.awssdk.enhanced.dynamodb.query.spec.QueryExpressionSpec} (built via + * {@link software.amazon.awssdk.enhanced.dynamodb.query.engine.QueryExpressionBuilder}) and executed through + * {@link software.amazon.awssdk.enhanced.dynamodb.DynamoDbEnhancedClient#enhancedQuery}. + *

+ * Key types: + *

    + *
  • {@link software.amazon.awssdk.enhanced.dynamodb.query.engine.QueryExpressionBuilder} + * – fluent builder for query specifications
    • + *
  • {@link software.amazon.awssdk.enhanced.dynamodb.query.condition.Condition} + * – in-memory filter conditions with combinators
    • + *
  • {@link software.amazon.awssdk.enhanced.dynamodb.query.result.EnhancedQueryRow} + * – a single result row (join items or aggregate values)
    • + *
  • {@link software.amazon.awssdk.enhanced.dynamodb.query.result.EnhancedQueryResult} + * – iterable result set
    • + *
  • {@link software.amazon.awssdk.enhanced.dynamodb.query.spec.AggregateSpec} + * – aggregation function specification
    • + *
+ *

+ * Filter evaluation phases: + *

    + *
  • {@code filterBase} – applies to base (left) rows before join expansion
    • + *
  • {@code filterJoined} – applies to joined (right) rows before merge
    • + *
  • {@code where} – applies to final merged rows (base + joined view)
    • + *
+ *

+ * Execution behavior: + *

    + *
  • {@code keyCondition} is pushed to DynamoDB {@code Query} on the base table
    • + *
  • With no {@code keyCondition}, + * {@link software.amazon.awssdk.enhanced.dynamodb.query.enums.ExecutionMode#STRICT_KEY_ONLY} + * does not perform a full-table scan and returns an empty result for that query path (no exception for this case)
    • + *
  • {@link software.amazon.awssdk.enhanced.dynamodb.query.enums.ExecutionMode#ALLOW_SCAN} permits scan fallback when key + * conditions are absent
    • + *
  • {@code ExecutionMode} does not change builder validation rules; invalid query shapes (for example, join-only options + * used without a configured join) still fail fast at {@code build()} with {@link java.lang.IllegalStateException}
    • + *
+ * Example: + * 
{@code
+ * // STRICT_KEY_ONLY (default): no key condition means no scan fallback; result is empty for this path.
+ * QueryExpressionSpec strict = QueryExpressionBuilder.from(customersTable)
+ *     .where(Condition.eq("region", "EU"))
+ *     .build();
+ *
+ * // ALLOW_SCAN: same shape may scan when no key condition is provided.
+ * QueryExpressionSpec allowScan = QueryExpressionBuilder.from(customersTable)
+ *     .executionMode(ExecutionMode.ALLOW_SCAN)
+ *     .where(Condition.eq("region", "EU"))
+ *     .build();
+ * }
+ *

+ * Join and aggregation notes: + *

    + *
  • Supported join types: {@link software.amazon.awssdk.enhanced.dynamodb.query.enums.JoinType#INNER}, + * {@link software.amazon.awssdk.enhanced.dynamodb.query.enums.JoinType#LEFT}, + * {@link software.amazon.awssdk.enhanced.dynamodb.query.enums.JoinType#RIGHT}, + * {@link software.amazon.awssdk.enhanced.dynamodb.query.enums.JoinType#FULL}
    • + *
  • {@code groupBy} requires at least one aggregate function (for example, + * {@link software.amazon.awssdk.enhanced.dynamodb.query.enums.AggregationFunction#COUNT})
    • + *
+ *

+ * Result shape: + *

    + *
  • {@link software.amazon.awssdk.enhanced.dynamodb.query.result.EnhancedQueryRow} may represent joined entity data or + * aggregate output values, depending on the query specification
    • + *
  • {@link software.amazon.awssdk.enhanced.dynamodb.query.result.EnhancedQueryResult} provides iterable access to these + * rows
    • + *
+ *

+ * Local developer tooling: + *

    + *
  • {@code run-enhanced-query-benchmark-local.sh} runs the benchmark harness locally and writes CSV output + * (defaults to {@code enhanced-query-benchmark-local.csv}, configurable via {@code BENCHMARK_OUTPUT_FILE})
    • + *
  • {@code run-enhanced-query-tests-and-print-timing.sh} runs enhanced-query functional tests and prints timing + * information to standard output
    • + *
+ *

+ * Best practice: use side-specific predicates in {@code filterBase}/{@code filterJoined} for early pruning, and reserve + * {@code where} for predicates that need the merged row view. + *

+ * Implementation notes (internal): default engines live under {@code ...query.internal} ({@code DefaultQueryExpressionEngine}, + * {@code DefaultQueryExpressionAsyncEngine}) with shared helpers ({@code QueryEngineSupport}, {@code JoinRowAliases}, + * joined-table fetch types). + * + * @see software.amazon.awssdk.enhanced.dynamodb.query.engine.QueryExpressionBuilder + */ +package software.amazon.awssdk.enhanced.dynamodb.query; diff --git a/services-custom/dynamodb-enhanced/src/main/java/software/amazon/awssdk/enhanced/dynamodb/query/result/DefaultEnhancedQueryResult.java b/services-custom/dynamodb-enhanced/src/main/java/software/amazon/awssdk/enhanced/dynamodb/query/result/DefaultEnhancedQueryResult.java new file mode 100644 index 000000000000..e6ae5d54e9a8 --- /dev/null +++ b/services-custom/dynamodb-enhanced/src/main/java/software/amazon/awssdk/enhanced/dynamodb/query/result/DefaultEnhancedQueryResult.java @@ -0,0 +1,43 @@ +/* + * Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"). + * You may not use this file except in compliance with the License. + * A copy of the License is located at + * + * http://aws.amazon.com/apache2.0 + * + * or in the "license" file accompanying this file. This file is distributed + * on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either + * express or implied. See the License for the specific language governing + * permissions and limitations under the License. + */ + +package software.amazon.awssdk.enhanced.dynamodb.query.result; + +import java.util.Iterator; +import java.util.List; +import software.amazon.awssdk.annotations.SdkInternalApi; +import software.amazon.awssdk.core.pagination.sync.SdkIterable; + +/** + * Default implementation of {@link EnhancedQueryResult} that wraps an iterable of rows. + */ +@SdkInternalApi +public class DefaultEnhancedQueryResult implements EnhancedQueryResult { + + private final Iterable iterable; + + public DefaultEnhancedQueryResult(SdkIterable iterable) { + this.iterable = iterable; + } + + public DefaultEnhancedQueryResult(List rows) { + this.iterable = rows; + } + + @Override + public Iterator iterator() { + return iterable.iterator(); + } +} diff --git a/services-custom/dynamodb-enhanced/src/main/java/software/amazon/awssdk/enhanced/dynamodb/query/result/EnhancedQueryLatencyReport.java b/services-custom/dynamodb-enhanced/src/main/java/software/amazon/awssdk/enhanced/dynamodb/query/result/EnhancedQueryLatencyReport.java new file mode 100644 index 000000000000..99ace59ad63d --- /dev/null +++ b/services-custom/dynamodb-enhanced/src/main/java/software/amazon/awssdk/enhanced/dynamodb/query/result/EnhancedQueryLatencyReport.java @@ -0,0 +1,58 @@ +/* + * Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"). + * You may not use this file except in compliance with the License. + * A copy of the License is located at + * + * http://aws.amazon.com/apache2.0 + * + * or in the "license" file accompanying this file. This file is distributed + * on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either + * express or implied. See the License for the specific language governing + * permissions and limitations under the License. + */ + +package software.amazon.awssdk.enhanced.dynamodb.query.result; + +import software.amazon.awssdk.annotations.SdkInternalApi; + +/** + * Captures basic latency measurements for enhanced query execution. + *

+ * All values are expressed in milliseconds. + */ +@SdkInternalApi +public class EnhancedQueryLatencyReport { + + private final long baseQueryMs; + private final long joinedLookupsMs; + private final long inMemoryProcessingMs; + private final long totalMs; + + public EnhancedQueryLatencyReport(long baseQueryMs, + long joinedLookupsMs, + long inMemoryProcessingMs, + long totalMs) { + this.baseQueryMs = baseQueryMs; + this.joinedLookupsMs = joinedLookupsMs; + this.inMemoryProcessingMs = inMemoryProcessingMs; + this.totalMs = totalMs; + } + + public long baseQueryMs() { + return baseQueryMs; + } + + public long joinedLookupsMs() { + return joinedLookupsMs; + } + + public long inMemoryProcessingMs() { + return inMemoryProcessingMs; + } + + public long totalMs() { + return totalMs; + } +} diff --git a/services-custom/dynamodb-enhanced/src/main/java/software/amazon/awssdk/enhanced/dynamodb/query/result/EnhancedQueryResult.java b/services-custom/dynamodb-enhanced/src/main/java/software/amazon/awssdk/enhanced/dynamodb/query/result/EnhancedQueryResult.java new file mode 100644 index 000000000000..24f44c110652 --- /dev/null +++ b/services-custom/dynamodb-enhanced/src/main/java/software/amazon/awssdk/enhanced/dynamodb/query/result/EnhancedQueryResult.java @@ -0,0 +1,29 @@ +/* + * Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"). + * You may not use this file except in compliance with the License. + * A copy of the License is located at + * + * http://aws.amazon.com/apache2.0 + * + * or in the "license" file accompanying this file. This file is distributed + * on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either + * express or implied. See the License for the specific language governing + * permissions and limitations under the License. + */ + +package software.amazon.awssdk.enhanced.dynamodb.query.result; + +import software.amazon.awssdk.annotations.SdkInternalApi; +import software.amazon.awssdk.core.pagination.sync.SdkIterable; + +/** + * Result of executing an enhanced query. Provides an iterable of {@link EnhancedQueryRow}s. + *

+ * Implementations stream results where possible. + */ +@SdkInternalApi +public interface EnhancedQueryResult extends SdkIterable { +} + diff --git a/services-custom/dynamodb-enhanced/src/main/java/software/amazon/awssdk/enhanced/dynamodb/query/result/EnhancedQueryRow.java b/services-custom/dynamodb-enhanced/src/main/java/software/amazon/awssdk/enhanced/dynamodb/query/result/EnhancedQueryRow.java new file mode 100644 index 000000000000..167c816895ed --- /dev/null +++ b/services-custom/dynamodb-enhanced/src/main/java/software/amazon/awssdk/enhanced/dynamodb/query/result/EnhancedQueryRow.java @@ -0,0 +1,103 @@ +/* + * Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"). + * You may not use this file except in compliance with the License. + * A copy of the License is located at + * + * http://aws.amazon.com/apache2.0 + * + * or in the "license" file accompanying this file. This file is distributed + * on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either + * express or implied. See the License for the specific language governing + * permissions and limitations under the License. + */ + +package software.amazon.awssdk.enhanced.dynamodb.query.result; + +import java.util.Collections; +import java.util.Map; +import software.amazon.awssdk.annotations.SdkInternalApi; + +/** + * A single row from an enhanced query result. + *

+ * A row can represent: + *

    + *
  • A join result: items available by alias (e.g. {@code "base"}, {@code "joined"}).
    • + *
  • An aggregation result: aggregate values available by output name.
    • + *
+ */ +@SdkInternalApi +public class EnhancedQueryRow { + + private final Map> itemsByAlias; + private final Map aggregates; + + protected EnhancedQueryRow(Map> itemsByAlias, + Map aggregates) { + this.itemsByAlias = itemsByAlias == null + ? Collections.emptyMap() : Collections.unmodifiableMap(itemsByAlias); + this.aggregates = aggregates == null + ? Collections.emptyMap() : Collections.unmodifiableMap(aggregates); + } + + /** + * Returns the item attribute map for the given alias (e.g. "base" for base table, + * "joined" for joined table). + */ + public Map getItem(String alias) { + return itemsByAlias.getOrDefault(alias, Collections.emptyMap()); + } + + /** + * Returns all items by alias. Keys are table aliases; values are attribute name to value maps. + */ + public Map> itemsByAlias() { + return itemsByAlias; + } + + /** + * Returns the aggregate value for the given output name. + */ + public Object getAggregate(String outputName) { + return aggregates.get(outputName); + } + + /** + * Returns all aggregate values by output name. + */ + public Map aggregates() { + return aggregates; + } + + public static Builder builder() { + return new Builder(); + } + + /** + * Builder for {@link EnhancedQueryRow}. + */ + public static class Builder { + + private Map> itemsByAlias; + private Map aggregates; + + protected Builder() { + } + + public Builder itemsByAlias(Map> itemsByAlias) { + this.itemsByAlias = itemsByAlias; + return this; + } + + public Builder aggregates(Map aggregates) { + this.aggregates = aggregates; + return this; + } + + public EnhancedQueryRow build() { + return new EnhancedQueryRow(itemsByAlias, aggregates); + } + } +} diff --git a/services-custom/dynamodb-enhanced/src/main/java/software/amazon/awssdk/enhanced/dynamodb/query/run-enhanced-query-benchmark-local.sh b/services-custom/dynamodb-enhanced/src/main/java/software/amazon/awssdk/enhanced/dynamodb/query/run-enhanced-query-benchmark-local.sh new file mode 100755 index 000000000000..951bdd572377 --- /dev/null +++ b/services-custom/dynamodb-enhanced/src/main/java/software/amazon/awssdk/enhanced/dynamodb/query/run-enhanced-query-benchmark-local.sh @@ -0,0 +1,38 @@ +#!/usr/bin/env bash +# +# Runs the Enhanced Query (join and aggregation) benchmark against in-process DynamoDB Local. +# No AWS credentials or EC2 required. Creates and seeds 1000 customers x 1000 orders, then runs +# five scenarios and prints latency stats (avgMs, p50Ms, p95Ms, rows). +# +# Usage: from the repository root, run: +# ./services-custom/dynamodb-enhanced/run-enhanced-query-benchmark-local.sh +# +# Optional environment variables (before running the script): +# BENCHMARK_ITERATIONS – measured runs per scenario (default: 5) +# BENCHMARK_WARMUP – warm-up runs per scenario (default: 2) +# BENCHMARK_OUTPUT_FILE – if set, CSV results are appended to this path +# +# Example with custom iterations and saving results: +# BENCHMARK_ITERATIONS=10 BENCHMARK_OUTPUT_FILE=benchmark_local.csv \ +# ./services-custom/dynamodb-enhanced/run-enhanced-query-benchmark-local.sh +# + +set -e +REPO_ROOT="$(cd "$(dirname "$0")/../.." && pwd)" +cd "$REPO_ROOT" + +export USE_LOCAL_DYNAMODB=true +# Match surefire: avoid DynamoDB Local telemetry (Pinpoint) when not needed; also satisfied by url-connection-client on test classpath +export DDB_LOCAL_TELEMETRY=0 +# Optional: override iterations/warmup/output (defaults are in the runner) +# export BENCHMARK_ITERATIONS=5 +# export BENCHMARK_WARMUP=2 +# export BENCHMARK_OUTPUT_FILE=benchmark_local.csv + +echo "Running Enhanced Query benchmark (DynamoDB Local, 1000 customers x 1000 orders)..." +mvn test-compile exec:java -pl services-custom/dynamodb-enhanced \ + -Dexec.mainClass="software.amazon.awssdk.enhanced.dynamodb.functionaltests.EnhancedQueryBenchmarkRunner" \ + -Dexec.classpathScope=test \ + -Dspotbugs.skip=true + +echo "Done. Results are printed above." diff --git a/services-custom/dynamodb-enhanced/src/main/java/software/amazon/awssdk/enhanced/dynamodb/query/run-enhanced-query-tests-and-print-timing.sh b/services-custom/dynamodb-enhanced/src/main/java/software/amazon/awssdk/enhanced/dynamodb/query/run-enhanced-query-tests-and-print-timing.sh new file mode 100755 index 000000000000..10fae002f61c --- /dev/null +++ b/services-custom/dynamodb-enhanced/src/main/java/software/amazon/awssdk/enhanced/dynamodb/query/run-enhanced-query-tests-and-print-timing.sh @@ -0,0 +1,162 @@ +#!/usr/bin/env bash +# +# Runs the 6 enhanced-query functional test classes and prints a table: Test (class.method) | Time (ms). +# Then prints a report listing tests that exceeded 1 second. +# Run from repo root: ./services-custom/dynamodb-enhanced/run-enhanced-query-tests-and-print-timing.sh +# Requires: Maven, full SDK build (e.g. mvn install -DskipTests from root first). +# + +set -e +REPO_ROOT="$(cd "$(dirname "$0")/../.." && pwd)" +cd "$REPO_ROOT" +REPORT_DIR="services-custom/dynamodb-enhanced/target/surefire-reports" +CLASSES=( + "EnhancedQueryJoinSyncTest" + "EnhancedQueryJoinAsyncTest" + "EnhancedQueryAggregationSyncTest" + "EnhancedQueryAggregationAsyncTest" + "NestedAttributeFilteringTest" + "BuildValidationTest" +) +PACKAGE="software.amazon.awssdk.enhanced.dynamodb.functionaltests" +EXCEEDED_THRESHOLD_MS=5000 +EXCEEDED_FILE="${REPORT_DIR}/exceeded_1s.txt" + +get_operation() { + local class_method="$1" + local method="${class_method##*.}" + case "$method" in + aggregation_withFilter|aggregation_treeCondition|aggregation_orderByAggregate|aggregation_orderByKey|aggregation_limit|\ + executionMode_allowScan|aggregation_withJoinedTableCondition_allowScan_returnsFilteredCounts|\ + withCondition_returnsFilteredRows|treeCondition_returnsMatchingRows|limit_enforced|\ + filterByNestedState_scanMode|filterByNestedZip_between|filterByNestedCity_contains|filterByNestedCity_beginsWith|\ + filterByNestedAndFlatAttribute_combined|filterByNestedAttribute_orCombination) + echo "scan()" + ;; + executionMode_strictKeyOnly_withoutKey_returnsEmptyOrNoScan|\ + groupByWithoutAggregate_throwsWithMessage|filterBaseWithoutJoin_throwsWithMessage|filterJoinedWithoutJoin_throwsWithMessage|\ + validSingleTableQuery_doesNotThrow|validJoinQuery_doesNotThrow|validAggregationQuery_doesNotThrow|\ + validJoinWithAggregation_doesNotThrow|aggregateWithoutGroupBy_doesNotThrow) + echo "none" + ;; + *) + echo "query()" + ;; + esac +} + +METRICS_FILE="${REPORT_DIR}/query-metrics.txt" + +echo "Running enhanced-query functional test classes (EnhancedQuery*, NestedAttributeFiltering, BuildValidation)..." +# Remove stale metrics file so tests write fresh data +rm -f "$METRICS_FILE" +# Run each class in a separate Maven invocation to isolate local DynamoDB lifecycle per test class. +for class in "${CLASSES[@]}"; do + mvn test -pl services-custom/dynamodb-enhanced \ + -Dspotbugs.skip=true \ + -Dcheckstyle.skip=true \ + -Dtest="${PACKAGE}.${class}" +done + +printf "\n" +echo "Per-test execution time (milliseconds):" +echo "----------------------------------------" +printf "%-95s %10s\n" "TEST (class.method)" "TIME (ms)" +echo "----------------------------------------" + +rm -f "$EXCEEDED_FILE" +mkdir -p "$REPORT_DIR" +for class in "${CLASSES[@]}"; do + xml="${REPORT_DIR}/TEST-${PACKAGE}.${class}.xml" + if [[ -f "$xml" ]]; then + grep -oE ']+>' "$xml" | while read -r line; do + name=$(echo "$line" | sed -n 's/.* name="\([^"]*\)".*/\1/p') + label="${class}.${name}" + # Prefer query-execution time from the metrics file written by the test + metric_ms="" + if [[ -f "$METRICS_FILE" ]]; then + metric_line=$(grep "^${label} " "$METRICS_FILE" | tail -n1) + if [[ -n "$metric_line" ]]; then + metric_ms=$(echo "$metric_line" | awk '{print $2}') + fi + fi + if [[ -n "$metric_ms" ]]; then + ms="$metric_ms" + else + # Fallback: Surefire total test-method time (includes @Before/@After) + time=$(echo "$line" | sed -n 's/.* time="\([^"]*\)".*/\1/p') + ms=$(awk "BEGIN { printf \"%.0f\", ${time:-0} * 1000 }") + fi + printf "%-95s %10s\n" "$label" "$ms" + if [[ "$ms" =~ ^[0-9]+$ ]] && [[ "$ms" -gt "$EXCEEDED_THRESHOLD_MS" ]]; then + echo "${label} ${ms}" >> "$EXCEEDED_FILE" + fi + done + else + printf "%-95s %10s\n" "${class}.(no report - run failed?)" "-" + fi +done +echo "----------------------------------------" +printf "\n" +echo "Report — tests that exceeded ${EXCEEDED_THRESHOLD_MS} ms:" +echo "----------------------------------------" +if [[ -f "$EXCEEDED_FILE" ]] && [[ -s "$EXCEEDED_FILE" ]]; then + while read -r label ms; do + op=$(get_operation "$label") + printf " EXCEEDED %-80s %8s ms (%s)\n" "$label" "$ms" "$op" + done < "$EXCEEDED_FILE" +else + echo " (none)" +fi +echo "----------------------------------------" + +printf "\n" +echo "Summary table (all tests):" +echo "----------------------------------------" +printf "%-80s %12s %14s %10s\n" "TEST (class.method)" "TIME (ms)" "OPERATION" "ROWS" +echo "----------------------------------------" +ALL_RESULTS_FILE="${REPORT_DIR}/all_results.txt" +rm -f "$ALL_RESULTS_FILE" +for class in "${CLASSES[@]}"; do + xml="${REPORT_DIR}/TEST-${PACKAGE}.${class}.xml" + if [[ -f "$xml" ]]; then + grep -oE ']+>' "$xml" | while read -r line; do + name=$(echo "$line" | sed -n 's/.* name=\"\([^\"]*\)\".*/\1/p') + label="${class}.${name}" + op=$(get_operation "$label") + # Read query execution time and row count from the metrics file (written by tests) + ms="" + rows="-" + if [[ -f "$METRICS_FILE" ]]; then + metric_line=$(grep "^${label} " "$METRICS_FILE" | tail -n1) + if [[ -n "$metric_line" ]]; then + ms=$(echo "$metric_line" | awk '{print $2}') + rows=$(echo "$metric_line" | awk '{print $3}') + fi + fi + if [[ -z "$ms" ]]; then + # Fallback: Surefire total test-method time (includes @Before/@After) + time=$(echo "$line" | sed -n 's/.* time=\"\([^\"]*\)\".*/\1/p') + ms=$(awk "BEGIN { printf \"%.0f\", ${time:-0} * 1000 }") + fi + echo "${label} ${ms} ${op} ${rows}" >> "$ALL_RESULTS_FILE" + done + fi +done + +if [[ -f "$ALL_RESULTS_FILE" ]] && [[ -s "$ALL_RESULTS_FILE" ]]; then + GREEN_BG="\033[42;30m" + RED_BG="\033[41;37m" + RESET="\033[0m" + while read -r label ms op rows; do + if [[ "$ms" =~ ^[0-9]+$ ]] && [[ "$ms" -gt "$EXCEEDED_THRESHOLD_MS" ]]; then + color="$RED_BG" + else + color="$GREEN_BG" + fi + printf "${color}%-80s %12s %14s %10s${RESET}\n" "$label" "$ms" "$op" "$rows" + done < "$ALL_RESULTS_FILE" +else + echo " (no test results found)" +fi +echo "----------------------------------------" diff --git a/services-custom/dynamodb-enhanced/src/main/java/software/amazon/awssdk/enhanced/dynamodb/query/spec/AggregateSpec.java b/services-custom/dynamodb-enhanced/src/main/java/software/amazon/awssdk/enhanced/dynamodb/query/spec/AggregateSpec.java new file mode 100644 index 000000000000..37ccedfddd04 --- /dev/null +++ b/services-custom/dynamodb-enhanced/src/main/java/software/amazon/awssdk/enhanced/dynamodb/query/spec/AggregateSpec.java @@ -0,0 +1,80 @@ +/* + * Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"). + * You may not use this file except in compliance with the License. + * A copy of the License is located at + * + * http://aws.amazon.com/apache2.0 + * + * or in the "license" file accompanying this file. This file is distributed + * on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either + * express or implied. See the License for the specific language governing + * permissions and limitations under the License. + */ + +package software.amazon.awssdk.enhanced.dynamodb.query.spec; + +import java.util.Objects; +import software.amazon.awssdk.annotations.SdkInternalApi; +import software.amazon.awssdk.enhanced.dynamodb.query.enums.AggregationFunction; + +/** + * Specification for a single aggregation: function, source attribute, and output name. + */ +@SdkInternalApi +public final class AggregateSpec { + private final AggregationFunction function; + private final String attribute; + private final String outputName; + + private AggregateSpec(Builder b) { + this.function = Objects.requireNonNull(b.function, "function"); + this.attribute = Objects.requireNonNull(b.attribute, "attribute"); + this.outputName = Objects.requireNonNull(b.outputName, "outputName"); + } + + public AggregationFunction function() { + return function; + } + + public String attribute() { + return attribute; + } + + public String outputName() { + return outputName; + } + + public static Builder builder() { + return new Builder(); + } + + public static final class Builder { + private AggregationFunction function; + private String attribute; + private String outputName; + + private Builder() { + } + + public Builder function(AggregationFunction function) { + this.function = function; + return this; + } + + public Builder attribute(String attribute) { + this.attribute = attribute; + return this; + } + + public Builder outputName(String outputName) { + this.outputName = outputName; + return this; + } + + public AggregateSpec build() { + return new AggregateSpec(this); + } + } +} diff --git a/services-custom/dynamodb-enhanced/src/main/java/software/amazon/awssdk/enhanced/dynamodb/query/spec/OrderBySpec.java b/services-custom/dynamodb-enhanced/src/main/java/software/amazon/awssdk/enhanced/dynamodb/query/spec/OrderBySpec.java new file mode 100644 index 000000000000..a2f9b03175f7 --- /dev/null +++ b/services-custom/dynamodb-enhanced/src/main/java/software/amazon/awssdk/enhanced/dynamodb/query/spec/OrderBySpec.java @@ -0,0 +1,62 @@ +/* + * Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"). + * You may not use this file except in compliance with the License. + * A copy of the License is located at + * + * http://aws.amazon.com/apache2.0 + * + * or in the "license" file accompanying this file. This file is distributed + * on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either + * express or implied. See the License for the specific language governing + * permissions and limitations under the License. + */ + +package software.amazon.awssdk.enhanced.dynamodb.query.spec; + +import java.util.Objects; +import software.amazon.awssdk.annotations.SdkInternalApi; +import software.amazon.awssdk.enhanced.dynamodb.query.enums.SortDirection; + +/** + * Specification for ordering result rows by an attribute or by an aggregate output name. + */ +@SdkInternalApi +public final class OrderBySpec { + private final String attributeOrAggregateName; + private final SortDirection direction; + private final boolean byAggregate; + + private OrderBySpec(String attributeOrAggregateName, SortDirection direction, boolean byAggregate) { + this.attributeOrAggregateName = Objects.requireNonNull(attributeOrAggregateName, "attributeOrAggregateName"); + this.direction = Objects.requireNonNull(direction, "direction"); + this.byAggregate = byAggregate; + } + + /** + * Order by a row attribute (e.g. "customerId", "name"). + */ + public static OrderBySpec byAttribute(String attribute, SortDirection direction) { + return new OrderBySpec(attribute, direction, false); + } + + /** + * Order by an aggregate output name (e.g. "orderCount"). + */ + public static OrderBySpec byAggregate(String aggregateOutputName, SortDirection direction) { + return new OrderBySpec(aggregateOutputName, direction, true); + } + + public String attributeOrAggregateName() { + return attributeOrAggregateName; + } + + public SortDirection direction() { + return direction; + } + + public boolean isByAggregate() { + return byAggregate; + } +} diff --git a/services-custom/dynamodb-enhanced/src/main/java/software/amazon/awssdk/enhanced/dynamodb/query/spec/QueryExpressionSpec.java b/services-custom/dynamodb-enhanced/src/main/java/software/amazon/awssdk/enhanced/dynamodb/query/spec/QueryExpressionSpec.java new file mode 100644 index 000000000000..26f67bf24297 --- /dev/null +++ b/services-custom/dynamodb-enhanced/src/main/java/software/amazon/awssdk/enhanced/dynamodb/query/spec/QueryExpressionSpec.java @@ -0,0 +1,440 @@ +/* + * Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"). + * You may not use this file except in compliance with the License. + * A copy of the License is located at + * + * http://aws.amazon.com/apache2.0 + * + * or in the "license" file accompanying this file. This file is distributed + * on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either + * express or implied. See the License for the specific language governing + * permissions and limitations under the License. + */ + +package software.amazon.awssdk.enhanced.dynamodb.query.spec; + +import java.util.Collections; +import java.util.List; +import software.amazon.awssdk.annotations.SdkInternalApi; +import software.amazon.awssdk.enhanced.dynamodb.MappedTableResource; +import software.amazon.awssdk.enhanced.dynamodb.model.QueryConditional; +import software.amazon.awssdk.enhanced.dynamodb.query.condition.Condition; +import software.amazon.awssdk.enhanced.dynamodb.query.engine.QueryExpressionBuilder; +import software.amazon.awssdk.enhanced.dynamodb.query.enums.ExecutionMode; +import software.amazon.awssdk.enhanced.dynamodb.query.enums.JoinType; + +/** + * Immutable specification for an enhanced query. Contains all parameters the query engine needs to execute a single-table scan, + * cross-table join, or aggregation. + * + *

Prefer using {@link QueryExpressionBuilder} to construct instances -- it provides validation and a + * fluent API. This class is the internal data carrier passed to the engine. + * + *

Field reference

+ * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + *
FieldTypeApplies toDescription
{@link #baseTable()}MappedTableResourceAllThe primary (left) table to query.
{@link #joinedTable()}MappedTableResourceJoinThe secondary (right) table for joins. Null for single-table queries.
{@link #joinType()}JoinTypeJoinINNER, LEFT, RIGHT, or FULL. Defaults to INNER.
{@link #leftJoinKey()}StringJoinAttribute on the base table used for join matching (SQL: ON base.X = joined.Y).
{@link #rightJoinKey()}StringJoinAttribute on the joined table used for join matching.
{@link #keyCondition()}QueryConditionalAllServer-side DynamoDB key condition. Triggers Query instead of Scan.
{@link #where()}ConditionAllIn-memory filter on the final row set (post-join, pre-aggregation).
{@link #filterBase()}ConditionJoinIn-memory filter on base rows before join.
{@link #filterJoined()}ConditionJoinIn-memory filter on joined rows before combining.
{@link #groupByAttributes()}List<String>AggregationAttributes to group by (SQL GROUP BY).
{@link #aggregates()}List<AggregateSpec>AggregationAggregation functions (COUNT, SUM, AVG, MIN, MAX) with output aliases.
{@link #orderBy()}List<OrderBySpec>AllIn-memory sort specifications.
{@link #projectAttributes()}List<String>AllAttribute projection pushed down to DynamoDB.
{@link #executionMode()}ExecutionModeAllSTRICT_KEY_ONLY (default) or ALLOW_SCAN.
{@link #limit()}IntegerAllMaximum result rows / aggregation buckets.
+ * + * @see QueryExpressionBuilder + */ +@SdkInternalApi +public final class QueryExpressionSpec { + private final MappedTableResource baseTable; + private final MappedTableResource joinedTable; + private final JoinType joinType; + private final String leftJoinKey; + private final String rightJoinKey; + private final QueryConditional baseKeyCondition; + private final Condition condition; + private final Condition baseTableCondition; + private final Condition joinedTableCondition; + private final List groupByAttributes; + private final List aggregates; + private final List orderBy; + private final List projectAttributes; + private final ExecutionMode executionMode; + private final Integer limit; + + private QueryExpressionSpec(Builder b) { + this.baseTable = b.baseTable; + this.joinedTable = b.joinedTable; + this.joinType = b.joinType != null ? b.joinType : JoinType.INNER; + this.leftJoinKey = b.leftJoinKey; + this.rightJoinKey = b.rightJoinKey; + this.baseKeyCondition = b.baseKeyCondition; + this.condition = b.condition; + this.baseTableCondition = b.baseTableCondition; + this.joinedTableCondition = b.joinedTableCondition; + this.groupByAttributes = b.groupByAttributes == null ? Collections.emptyList() : + Collections.unmodifiableList(b.groupByAttributes); + this.aggregates = b.aggregates == null ? Collections.emptyList() : Collections.unmodifiableList(b.aggregates); + this.orderBy = b.orderBy == null ? Collections.emptyList() : Collections.unmodifiableList(b.orderBy); + this.projectAttributes = b.projectAttributes == null ? Collections.emptyList() : + Collections.unmodifiableList(b.projectAttributes); + this.executionMode = b.executionMode != null ? b.executionMode : ExecutionMode.STRICT_KEY_ONLY; + this.limit = b.limit; + } + + public static Builder builder() { + return new Builder(); + } + + /** + * The primary (left) table to query. Always non-null. + */ + public MappedTableResource baseTable() { + return baseTable; + } + + /** + * The secondary (right) table for joins, or null for single-table queries. + */ + public MappedTableResource joinedTable() { + return joinedTable; + } + + /** + * Returns true if a joined table is configured. + */ + public boolean hasJoin() { + return joinedTable != null; + } + + /** + * The join type (INNER, LEFT, RIGHT, FULL). Defaults to INNER when not explicitly set. + */ + public JoinType joinType() { + return joinType; + } + + /** + * Join key attribute name on the left/base table. Semantically equivalent to SQL {@code ON left. = right.}. + */ + public String leftJoinKey() { + return leftJoinKey; + } + + /** + * Join key attribute name on the right/joined table. Semantically equivalent to SQL {@code ON left. = right.}. + */ + public String rightJoinKey() { + return rightJoinKey; + } + + /** + * @deprecated Use {@link #leftJoinKey()}. + */ + @Deprecated + public String baseJoinAttribute() { + return leftJoinKey; + } + + /** + * @deprecated Use {@link #rightJoinKey()}. + */ + @Deprecated + public String joinedJoinAttribute() { + return rightJoinKey; + } + + /** + * Optional key condition for the base table. When set, the engine uses Query with this condition; when null and + * {@link #executionMode()} is {@link ExecutionMode#ALLOW_SCAN}, the engine may use Scan. + */ + public QueryConditional keyCondition() { + return baseKeyCondition; + } + + /** + * @deprecated Use {@link #keyCondition()}. + */ + @Deprecated + public QueryConditional baseKeyCondition() { + return baseKeyCondition; + } + + /** + * In-memory filter applied to the final row set. Without join: filters base items. With join: filters the combined (base + + * joined) view. Applied before aggregation. + */ + public Condition where() { + return condition; + } + + /** + * @deprecated Use {@link #where()}. + */ + @Deprecated + public Condition condition() { + return condition; + } + + /** + * In-memory filter applied to base (left) table rows before joining. Only meaningful when a join is configured. + */ + public Condition filterBase() { + return baseTableCondition; + } + + /** + * @deprecated Use {@link #filterBase()}. + */ + @Deprecated + public Condition baseTableCondition() { + return baseTableCondition; + } + + /** + * In-memory filter applied to joined (right) table rows before combining with base rows. Only meaningful when a join is + * configured. + */ + public Condition filterJoined() { + return joinedTableCondition; + } + + /** + * @deprecated Use {@link #filterJoined()}. + */ + @Deprecated + public Condition joinedTableCondition() { + return joinedTableCondition; + } + + /** + * Attributes to group by for aggregation (SQL GROUP BY). Empty list means no grouping. + */ + public List groupByAttributes() { + return groupByAttributes; + } + + /** + * Aggregation function definitions (COUNT, SUM, AVG, MIN, MAX). Empty list means no aggregation. + */ + public List aggregates() { + return aggregates; + } + + /** + * In-memory sort specifications. Applied after filtering and aggregation. + */ + public List orderBy() { + return orderBy; + } + + /** + * Attribute names for DynamoDB projection pushdown. Empty list means all attributes. + */ + public List projectAttributes() { + return projectAttributes; + } + + /** + * Execution mode: STRICT_KEY_ONLY (default) or ALLOW_SCAN. + */ + public ExecutionMode executionMode() { + return executionMode; + } + + /** + * Optional limit on number of result rows (or buckets when grouping). + */ + public Integer limit() { + return limit; + } + + public static final class Builder { + private MappedTableResource baseTable; + private MappedTableResource joinedTable; + private JoinType joinType; + private String leftJoinKey; + private String rightJoinKey; + private QueryConditional baseKeyCondition; + private Condition condition; + private Condition baseTableCondition; + private Condition joinedTableCondition; + private List groupByAttributes; + private List aggregates; + private List orderBy; + private List projectAttributes; + private ExecutionMode executionMode; + private Integer limit; + + private Builder() { + } + + public Builder baseTable(MappedTableResource baseTable) { + this.baseTable = baseTable; + return this; + } + + public Builder joinedTable(MappedTableResource joinedTable) { + this.joinedTable = joinedTable; + return this; + } + + public Builder joinType(JoinType joinType) { + this.joinType = joinType; + return this; + } + + public Builder leftJoinKey(String leftJoinKey) { + this.leftJoinKey = leftJoinKey; + return this; + } + + public Builder rightJoinKey(String rightJoinKey) { + this.rightJoinKey = rightJoinKey; + return this; + } + + /** + * @deprecated Use {@link #leftJoinKey(String)}. + */ + @Deprecated + public Builder baseJoinAttribute(String baseJoinAttribute) { + return leftJoinKey(baseJoinAttribute); + } + + /** + * @deprecated Use {@link #rightJoinKey(String)}. + */ + @Deprecated + public Builder joinedJoinAttribute(String joinedJoinAttribute) { + return rightJoinKey(joinedJoinAttribute); + } + + /** + * Set the key condition for the base table (DynamoDB {@code Query} key condition). When set, the engine uses + * {@code Query}; when null and {@link ExecutionMode#ALLOW_SCAN}, the engine may use {@code Scan}. + */ + public Builder keyCondition(QueryConditional keyCondition) { + this.baseKeyCondition = keyCondition; + return this; + } + + /** + * Filter applied to the final row set in-memory. + *

+ * With no join: applies to base items. With join: applies to the combined (base + joined) view. + */ + public Builder where(Condition where) { + this.condition = where; + return this; + } + + /** + * In-memory filter applied to base (left) items before joining. + */ + public Builder filterBase(Condition filterBase) { + this.baseTableCondition = filterBase; + return this; + } + + /** + * In-memory filter applied to joined (right) items before joining. + */ + public Builder filterJoined(Condition filterJoined) { + this.joinedTableCondition = filterJoined; + return this; + } + + /** + * @deprecated Use {@link #keyCondition(QueryConditional)}. + */ + @Deprecated + public Builder baseKeyCondition(QueryConditional baseKeyCondition) { + return keyCondition(baseKeyCondition); + } + + /** + * @deprecated Use {@link #where(Condition)}. + */ + @Deprecated + public Builder condition(Condition condition) { + return where(condition); + } + + /** + * @deprecated Use {@link #filterBase(Condition)}. + */ + @Deprecated + public Builder baseTableCondition(Condition baseTableCondition) { + return filterBase(baseTableCondition); + } + + /** + * @deprecated Use {@link #filterJoined(Condition)}. + */ + @Deprecated + public Builder joinedTableCondition(Condition joinedTableCondition) { + return filterJoined(joinedTableCondition); + } + + public Builder groupByAttributes(List groupByAttributes) { + this.groupByAttributes = groupByAttributes; + return this; + } + + public Builder aggregates(List aggregates) { + this.aggregates = aggregates; + return this; + } + + public Builder orderBy(List orderBy) { + this.orderBy = orderBy; + return this; + } + + public Builder projectAttributes(List projectAttributes) { + this.projectAttributes = projectAttributes; + return this; + } + + public Builder executionMode(ExecutionMode executionMode) { + this.executionMode = executionMode; + return this; + } + + public Builder limit(Integer limit) { + this.limit = limit; + return this; + } + + public QueryExpressionSpec build() { + return new QueryExpressionSpec(this); + } + } +} + diff --git a/services-custom/dynamodb-enhanced/src/test/java/software/amazon/awssdk/enhanced/dynamodb/functionaltests/BuildValidationTest.java b/services-custom/dynamodb-enhanced/src/test/java/software/amazon/awssdk/enhanced/dynamodb/functionaltests/BuildValidationTest.java new file mode 100644 index 000000000000..88b8e0646407 --- /dev/null +++ b/services-custom/dynamodb-enhanced/src/test/java/software/amazon/awssdk/enhanced/dynamodb/functionaltests/BuildValidationTest.java @@ -0,0 +1,141 @@ +/* + * Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"). + * You may not use this file except in compliance with the License. + * A copy of the License is located at + * + * http://aws.amazon.com/apache2.0 + * + * or in the "license" file accompanying this file. This file is distributed + * on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either + * express or implied. See the License for the specific language governing + * permissions and limitations under the License. + */ + +package software.amazon.awssdk.enhanced.dynamodb.functionaltests; + +import static org.assertj.core.api.Assertions.assertThatThrownBy; +import static software.amazon.awssdk.enhanced.dynamodb.mapper.StaticAttributeTags.primaryPartitionKey; + +import org.junit.Before; +import org.junit.Test; +import software.amazon.awssdk.enhanced.dynamodb.DynamoDbEnhancedClient; +import software.amazon.awssdk.enhanced.dynamodb.DynamoDbTable; +import software.amazon.awssdk.enhanced.dynamodb.TableSchema; +import software.amazon.awssdk.enhanced.dynamodb.mapper.StaticTableSchema; +import software.amazon.awssdk.enhanced.dynamodb.query.condition.Condition; +import software.amazon.awssdk.enhanced.dynamodb.query.engine.QueryExpressionBuilder; +import software.amazon.awssdk.enhanced.dynamodb.query.enums.AggregationFunction; +import software.amazon.awssdk.enhanced.dynamodb.query.enums.JoinType; + +/** + * Functional tests for build-time validation in {@link QueryExpressionBuilder#build()}. Verifies that incompatible option + * combinations throw {@link IllegalStateException} with clear messages. + */ +public class BuildValidationTest extends LocalDynamoDbTestBase { + + private static class SimpleRecord { + private String id; + + public String getId() { + return id; + } + + public void setId(String id) { + this.id = id; + } + } + + private static final TableSchema SCHEMA = + StaticTableSchema.builder(SimpleRecord.class) + .newItemSupplier(SimpleRecord::new) + .addAttribute(String.class, + a -> a.name("id") + .getter(SimpleRecord::getId) + .setter(SimpleRecord::setId) + .tags(primaryPartitionKey())) + .build(); + + private DynamoDbTable tableA; + private DynamoDbTable tableB; + + @Before + public void setUp() { + DynamoDbEnhancedClient client = DynamoDbEnhancedClient.builder() + .dynamoDbClient(localDynamoDb().createClient()) + .build(); + tableA = client.table(getConcreteTableName("table_a"), SCHEMA); + tableB = client.table(getConcreteTableName("table_b"), SCHEMA); + } + + @Test + public void groupByWithoutAggregate_throwsWithMessage() { + assertThatThrownBy(() -> + QueryExpressionBuilder.from(tableA) + .groupBy("id") + .build() + ).isInstanceOf(IllegalStateException.class) + .hasMessageContaining("groupBy() requires at least one aggregate()"); + } + + @Test + public void filterBaseWithoutJoin_throwsWithMessage() { + assertThatThrownBy(() -> + QueryExpressionBuilder.from(tableA) + .filterBase(Condition.eq("status", "ACTIVE")) + .build() + ).isInstanceOf(IllegalStateException.class) + .hasMessageContaining("filterBase() is only applicable when a join is configured"); + } + + @Test + public void filterJoinedWithoutJoin_throwsWithMessage() { + assertThatThrownBy(() -> + QueryExpressionBuilder.from(tableA) + .filterJoined(Condition.eq("amount", 100)) + .build() + ).isInstanceOf(IllegalStateException.class) + .hasMessageContaining("filterJoined() is only applicable when a join is configured"); + } + + @Test + public void validSingleTableQuery_doesNotThrow() { + QueryExpressionBuilder.from(tableA) + .where(Condition.eq("status", "ACTIVE")) + .build(); + } + + @Test + public void validJoinQuery_doesNotThrow() { + QueryExpressionBuilder.from(tableA) + .join(tableB, JoinType.INNER, "id", "id") + .filterBase(Condition.eq("status", "ACTIVE")) + .filterJoined(Condition.gt("amount", 0)) + .build(); + } + + @Test + public void validAggregationQuery_doesNotThrow() { + QueryExpressionBuilder.from(tableA) + .groupBy("status") + .aggregate(AggregationFunction.COUNT, "id", "total") + .build(); + } + + @Test + public void validJoinWithAggregation_doesNotThrow() { + QueryExpressionBuilder.from(tableA) + .join(tableB, JoinType.INNER, "id", "id") + .groupBy("id") + .aggregate(AggregationFunction.SUM, "amount", "totalAmount") + .build(); + } + + @Test + public void aggregateWithoutGroupBy_doesNotThrow() { + QueryExpressionBuilder.from(tableA) + .aggregate(AggregationFunction.COUNT, "id", "total") + .build(); + } +} diff --git a/services-custom/dynamodb-enhanced/src/test/java/software/amazon/awssdk/enhanced/dynamodb/functionaltests/LargeDatasetInitializer.java b/services-custom/dynamodb-enhanced/src/test/java/software/amazon/awssdk/enhanced/dynamodb/functionaltests/LargeDatasetInitializer.java new file mode 100644 index 000000000000..13edf4a2a55a --- /dev/null +++ b/services-custom/dynamodb-enhanced/src/test/java/software/amazon/awssdk/enhanced/dynamodb/functionaltests/LargeDatasetInitializer.java @@ -0,0 +1,318 @@ +/* + * Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"). + * You may not use this file except in compliance with the License. + * A copy of the License is located at + * + * http://aws.amazon.com/apache2.0 + * + * or in the "license" file accompanying this file. This file is distributed + * on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either + * express or implied. See the License for the specific language governing + * permissions and limitations under the License. + */ + +package software.amazon.awssdk.enhanced.dynamodb.functionaltests; + +import static software.amazon.awssdk.enhanced.dynamodb.mapper.StaticAttributeTags.primaryPartitionKey; +import static software.amazon.awssdk.enhanced.dynamodb.mapper.StaticAttributeTags.primarySortKey; + +import software.amazon.awssdk.enhanced.dynamodb.DynamoDbEnhancedClient; +import software.amazon.awssdk.enhanced.dynamodb.DynamoDbTable; +import software.amazon.awssdk.enhanced.dynamodb.TableSchema; +import software.amazon.awssdk.enhanced.dynamodb.mapper.StaticTableSchema; +import java.util.ArrayList; +import java.util.List; +import software.amazon.awssdk.enhanced.dynamodb.model.BatchWriteItemEnhancedRequest; +import software.amazon.awssdk.enhanced.dynamodb.model.WriteBatch; +import software.amazon.awssdk.services.dynamodb.DynamoDbClient; +import software.amazon.awssdk.services.dynamodb.model.ProvisionedThroughput; +import software.amazon.awssdk.services.dynamodb.model.ResourceInUseException; +import software.amazon.awssdk.services.dynamodb.model.ScanRequest; +import software.amazon.awssdk.services.dynamodb.model.Select; + +/** + * Idempotent utility to create and seed the Customers and Orders tables used by enhanced query integration tests. Intended to be + * run from {@code @BeforeClass} / {@code @BeforeAll} so that a large dataset exists before tests run. + *

+ * Idempotency: Safe to call multiple times. If the tables already exist, they are not + * recreated. If item counts already meet or exceed the requested {@code customerCount} and + * {@code customerCount * ordersPerCustomer}, seeding is skipped; otherwise only the missing items are written (by overwriting + * items with fixed IDs so that repeated runs do not duplicate). + *

+ * Table layout: Customers table has partition key {@code customerId} (String). Orders + * table has partition key {@code customerId} (String) and sort key {@code orderId} (String), plus an {@code amount} (Integer) + * attribute. This layout allows Query on Orders by customerId. + *

+ * Large-dataset tests: The enhanced-query test classes use {@link LocalDynamoDbLargeDatasetTestBase}, which + * uses the same in-process {@link LocalDynamoDb} as other functionaltests and calls {@link #initializeCustomersAndOrdersDataset} once in {@code @BeforeClass}. + * No external process or environment variables are required. {@link #main(String[])} is optional and runs an in-memory seed for + * quick verification of the initializer; it is not required for running the tests. + */ +public final class LargeDatasetInitializer { + + /** + * Table name used for the large-dataset Customers table (seed and join/aggregation tests). + */ + public static final String LARGE_CUSTOMERS_TABLE = "customers_large"; + /** + * Table name used for the large-dataset Orders table (seed and join/aggregation tests). + */ + public static final String LARGE_ORDERS_TABLE = "orders_large"; + + /** + * DynamoDB BatchWriteItem limit per request. + */ + private static final int BATCH_SIZE = 25; + + private static final ProvisionedThroughput DEFAULT_PROVISIONED_THROUGHPUT = + ProvisionedThroughput.builder() + .readCapacityUnits(50L) + .writeCapacityUnits(50L) + .build(); + + private LargeDatasetInitializer() { + } + + /** + * Ensures the Customers and Orders tables exist and contain at least {@code customerCount} customers and + * {@code customerCount * ordersPerCustomer} orders. Creates the tables if they do not exist (using the given provisioned + * throughput), then seeds data until counts are met. Safe to call multiple times; existing tables or sufficient counts cause + * creation or seeding to be skipped. + * + * @param dynamoDbClient low-level DynamoDB client (e.g. from + * {@link LocalDynamoDbLargeDatasetTestBase#getDynamoDbClient()}) + * @param customersTableName physical name of the Customers table + * @param ordersTableName physical name of the Orders table + * @param customerCount desired number of customer items + * @param ordersPerCustomer desired number of order items per customer (total orders = customerCount * ordersPerCustomer) + * @param provisionedThroughput throughput used when creating tables; ignored if tables already exist + */ + public static void initializeCustomersAndOrdersDataset( + DynamoDbClient dynamoDbClient, + String customersTableName, + String ordersTableName, + int customerCount, + int ordersPerCustomer, + ProvisionedThroughput provisionedThroughput + ) { + DynamoDbEnhancedClient enhancedClient = DynamoDbEnhancedClient.builder().dynamoDbClient(dynamoDbClient).build(); + DynamoDbTable customersTable = enhancedClient.table(customersTableName, CUSTOMER_SCHEMA); + DynamoDbTable ordersTable = enhancedClient.table(ordersTableName, ORDER_SCHEMA); + + ensureTableExists(customersTable, provisionedThroughput); + ensureTableExists(ordersTable, provisionedThroughput); + + // Idempotency: skip seeding if tables already have at least the requested counts. + int existingCustomers = dynamoDbClient.scan( + ScanRequest.builder().tableName(customersTableName).select(Select.COUNT).build()).count(); + if (existingCustomers >= customerCount) { + int existingOrders = dynamoDbClient.scan( + ScanRequest.builder().tableName(ordersTableName).select(Select.COUNT).build()).count(); + if (existingOrders >= (long) customerCount * ordersPerCustomer) { + System.out.println("LargeDatasetInitializer: skipping seed (customers=" + existingCustomers + + ", orders=" + existingOrders + " already meet or exceed requested counts)."); + return; + } + } + + long start = System.nanoTime(); + + seedCustomers(enhancedClient, customersTable, customerCount); + seedOrders(enhancedClient, ordersTable, customerCount, ordersPerCustomer); + + long elapsedMs = (System.nanoTime() - start) / 1_000_000; + System.out.println("Insertion of data took : " + elapsedMs / 1_000 + " seconds"); + } + + /** + * Optional entry point that runs an in-memory seed for verification. Starts an in-process {@link LocalDynamoDb}, creates the + * Customers and Orders tables, seeds 1_000 customers and 1_000×1_000 orders by default, then stops the server. Data is not + * persisted. For running the EnhancedQuery* tests, this is not required — the test base seeds in-process in + * {@code @BeforeClass}. + *

+ * Optional arguments: [customersTableName ordersTableName customerCount ordersPerCustomer]. Defaults: + * {@value #LARGE_CUSTOMERS_TABLE}, {@value #LARGE_ORDERS_TABLE}, 1000, 1000. + * + * @param args optional: customersTableName, ordersTableName, customerCount, ordersPerCustomer + */ + public static void main(String[] args) { + String customersTableName = args.length >= 1 ? args[0] : LARGE_CUSTOMERS_TABLE; + String ordersTableName = args.length >= 2 ? args[1] : LARGE_ORDERS_TABLE; + int customerCount = args.length >= 3 ? Integer.parseInt(args[2]) : 1000; + int ordersPerCustomer = args.length >= 4 ? Integer.parseInt(args[3]) : 1000; + + LocalDynamoDb local = new LocalDynamoDb(); + local.start(); + try { + try (DynamoDbClient client = local.createClient()) { + initializeCustomersAndOrdersDataset( + client, + customersTableName, + ordersTableName, + customerCount, + ordersPerCustomer, + DEFAULT_PROVISIONED_THROUGHPUT); + } + } finally { + local.stop(); + } + } + + /** + * Creates the table if it does not exist. If the table already exists (ResourceInUseException), the exception is ignored so + * that the call is idempotent. + */ + private static void ensureTableExists(DynamoDbTable table, ProvisionedThroughput throughput) { + try { + table.createTable(r -> r.provisionedThroughput(throughput)); + } catch (ResourceInUseException e) { + // Table already exists; idempotent. + } + } + + /** + * Writes customer items with IDs c1, c2, ... up to customerCount. Overwrites if already present. Builds batches of 25 and + * runs BatchWriteItem in parallel for speed. + */ + private static void seedCustomers(DynamoDbEnhancedClient enhancedClient, + DynamoDbTable table, + int customerCount) { + List batches = new ArrayList<>(); + for (int start = 1; start <= customerCount; start += BATCH_SIZE) { + int end = Math.min(start + BATCH_SIZE, customerCount + 1); + WriteBatch.Builder batchBuilder = + WriteBatch.builder(CustomerPojo.class).mappedTableResource(table); + for (int i = start; i < end; i++) { + CustomerPojo c = new CustomerPojo(); + c.setCustomerId("c" + i); + c.setName("Customer" + i); + c.setRegion(i % 2 == 1 ? "EU" : "NA"); + batchBuilder.addPutItem(r -> r.item(c)); + } + batches.add(BatchWriteItemEnhancedRequest.builder().writeBatches(batchBuilder.build()).build()); + } + batches.parallelStream().forEach(enhancedClient::batchWriteItem); + } + + /** + * Writes order items for each customer: c1-o1, c1-o2, ... so that each customer has ordersPerCustomer orders. Overwrites if + * already present. Builds batches of 25 and runs BatchWriteItem in parallel for speed. + */ + private static void seedOrders(DynamoDbEnhancedClient enhancedClient, + DynamoDbTable table, + int customerCount, + int ordersPerCustomer) { + List batches = new ArrayList<>(); + WriteBatch.Builder batchBuilder = WriteBatch.builder(OrderPojo.class).mappedTableResource(table); + int inBatch = 0; + for (int c = 1; c <= customerCount; c++) { + for (int o = 1; o <= ordersPerCustomer; o++) { + OrderPojo ord = new OrderPojo(); + ord.setCustomerId("c" + c); + ord.setOrderId("c" + c + "-o" + o); + ord.setAmount(10 * c + o); + batchBuilder.addPutItem(r -> r.item(ord)); + inBatch++; + if (inBatch >= BATCH_SIZE) { + batches.add(BatchWriteItemEnhancedRequest.builder().writeBatches(batchBuilder.build()).build()); + batchBuilder = WriteBatch.builder(OrderPojo.class).mappedTableResource(table); + inBatch = 0; + } + } + } + if (inBatch > 0) { + batches.add(BatchWriteItemEnhancedRequest.builder().writeBatches(batchBuilder.build()).build()); + } + batches.parallelStream().forEach(enhancedClient::batchWriteItem); + } + + /** + * Simple POJO for Customers table: customerId (PK), name, region. + */ + private static class CustomerPojo { + private String customerId; + private String name; + private String region; + + public String getCustomerId() { + return customerId; + } + + public void setCustomerId(String customerId) { + this.customerId = customerId; + } + + public String getName() { + return name; + } + + public void setName(String name) { + this.name = name; + } + + public String getRegion() { + return region; + } + + public void setRegion(String region) { + this.region = region; + } + } + + /** + * Simple POJO for Orders table: customerId (PK), orderId (SK), amount. + */ + private static class OrderPojo { + private String customerId; + private String orderId; + private Integer amount; + + public String getCustomerId() { + return customerId; + } + + public void setCustomerId(String customerId) { + this.customerId = customerId; + } + + public String getOrderId() { + return orderId; + } + + public void setOrderId(String orderId) { + this.orderId = orderId; + } + + public Integer getAmount() { + return amount; + } + + public void setAmount(Integer amount) { + this.amount = amount; + } + } + + private static final TableSchema CUSTOMER_SCHEMA = + StaticTableSchema.builder(CustomerPojo.class) + .newItemSupplier(CustomerPojo::new) + .addAttribute(String.class, + a -> a.name("customerId").getter(CustomerPojo::getCustomerId).setter(CustomerPojo::setCustomerId).tags(primaryPartitionKey())) + .addAttribute(String.class, + a -> a.name("name").getter(CustomerPojo::getName).setter(CustomerPojo::setName)) + .addAttribute(String.class, + a -> a.name("region").getter(CustomerPojo::getRegion).setter(CustomerPojo::setRegion)) + .build(); + + private static final TableSchema ORDER_SCHEMA = + StaticTableSchema.builder(OrderPojo.class) + .newItemSupplier(OrderPojo::new) + .addAttribute(String.class, + a -> a.name("customerId").getter(OrderPojo::getCustomerId).setter(OrderPojo::setCustomerId).tags(primaryPartitionKey())) + .addAttribute(String.class, + a -> a.name("orderId").getter(OrderPojo::getOrderId).setter(OrderPojo::setOrderId).tags(primarySortKey())) + .addAttribute(Integer.class, + a -> a.name("amount").getter(OrderPojo::getAmount).setter(OrderPojo::setAmount)) + .build(); +} diff --git a/services-custom/dynamodb-enhanced/src/test/java/software/amazon/awssdk/enhanced/dynamodb/functionaltests/LocalDynamoDbLargeDatasetTestBase.java b/services-custom/dynamodb-enhanced/src/test/java/software/amazon/awssdk/enhanced/dynamodb/functionaltests/LocalDynamoDbLargeDatasetTestBase.java new file mode 100644 index 000000000000..ecb411f1544f --- /dev/null +++ b/services-custom/dynamodb-enhanced/src/test/java/software/amazon/awssdk/enhanced/dynamodb/functionaltests/LocalDynamoDbLargeDatasetTestBase.java @@ -0,0 +1,151 @@ +/* + * Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"). + * You may not use this file except in compliance with the License. + * A copy of the License is located at + * + * http://aws.amazon.com/apache2.0 + * + * or in the "license" file accompanying this file. This file is distributed + * on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either + * express or implied. See the License for the specific language governing + * permissions and limitations under the License. + */ + +package software.amazon.awssdk.enhanced.dynamodb.functionaltests; + +import static org.hamcrest.MatcherAssert.assertThat; +import static org.hamcrest.Matchers.instanceOf; +import static org.hamcrest.Matchers.is; +import static org.hamcrest.Matchers.nullValue; + +import java.util.List; +import org.junit.AfterClass; +import org.junit.BeforeClass; +import software.amazon.awssdk.core.async.SdkPublisher; +import software.amazon.awssdk.services.dynamodb.DynamoDbAsyncClient; +import software.amazon.awssdk.services.dynamodb.DynamoDbClient; +import software.amazon.awssdk.services.dynamodb.model.ProvisionedThroughput; + +/** + * Test base for enhanced-query tests that use the large dataset. Uses the same in-process + * {@link LocalDynamoDb} as all other functionaltests (from {@link LocalDynamoDbTestBase}). + * Seeds the Customers and Orders tables once in {@code @BeforeClass} via + * {@link LargeDatasetInitializer#initializeCustomersAndOrdersDataset}. Does not stop the DB in + * {@code @AfterClass} so subsequent test classes in the same JVM reuse the same data. + *

+ * Dataset size is {@value #SEED_CUSTOMER_COUNT} customers and {@value #SEED_ORDERS_PER_CUSTOMER} + * orders per customer. + */ +public abstract class LocalDynamoDbLargeDatasetTestBase extends LocalDynamoDbTestBase { + + /** Number of customers seeded (c1 .. cN). */ + public static final int SEED_CUSTOMER_COUNT = 1000; + /** Number of orders per customer (cK-o1 .. cK-oM). Amount for cK-oO is 10*K + O. */ + public static final int SEED_ORDERS_PER_CUSTOMER = 1000; + /** For c1 with joined condition amount >= 50: orders 40..1000, count = 961. */ + public static final long EXPECTED_ORDER_COUNT_C1_AMOUNT_GE_50 = 961; + /** For c1 amount >= 50: min amount = 10 + 40 = 50. */ + public static final int EXPECTED_MIN_AMOUNT_C1_GE_50 = 50; + /** For c1 amount >= 50: max amount = 10 + 1000 = 1010. */ + public static final int EXPECTED_MAX_AMOUNT_C1_GE_50 = 1010; + /** For c1 amount >= 50: sum(50..1010) = 961*10 + sum(40..1000) = 9610 + 961*520 = 509330. */ + public static final long EXPECTED_TOTAL_AMOUNT_C1_GE_50 = 509330L; + /** For c1 amount >= 50: avg = 509330 / 961. */ + public static final double EXPECTED_AVG_AMOUNT_C1_GE_50 = (double) EXPECTED_TOTAL_AMOUNT_C1_GE_50 / EXPECTED_ORDER_COUNT_C1_AMOUNT_GE_50; + + /** For c1 without filter: all 1000 orders. */ + public static final long EXPECTED_ORDER_COUNT_C1 = SEED_ORDERS_PER_CUSTOMER; + /** For c1 without filter: min amount = 10*1 + 1 = 11. */ + public static final int EXPECTED_MIN_AMOUNT_C1 = 11; + /** For c1 without filter: max amount = 10*1 + 1000 = 1010. */ + public static final int EXPECTED_MAX_AMOUNT_C1 = 1010; + /** For c1 without filter: sum(11..1010) = 1000 * (11+1010)/2 = 510500. */ + public static final long EXPECTED_TOTAL_AMOUNT_C1 = 510500L; + /** For c1 without filter: avg = 510500 / 1000 = 510.5. */ + public static final double EXPECTED_AVG_AMOUNT_C1 = 510.5; + + /** Number of EU customers (odd IDs: c1, c3, c5 ... c999). */ + public static final int EXPECTED_EU_CUSTOMER_COUNT = 500; + /** Number of NA customers (even IDs: c2, c4, c6 ... c1000). */ + public static final int EXPECTED_NA_CUSTOMER_COUNT = 500; + + private static final ProvisionedThroughput DEFAULT_PROVISIONED_THROUGHPUT = + ProvisionedThroughput.builder() + .readCapacityUnits(50L) + .writeCapacityUnits(50L) + .build(); + + private static DynamoDbClient dynamoDbClient; + private static DynamoDbAsyncClient dynamoDbAsyncClient; + private static boolean datasetSeeded; + + @BeforeClass + public static void ensureLocalDynamoDbAndSeed() { + initializeLocalDynamoDb(); + if (datasetSeeded) { + return; + } + dynamoDbClient = localDynamoDb().createClient(); + dynamoDbAsyncClient = localDynamoDb().createAsyncClient(); + LargeDatasetInitializer.initializeCustomersAndOrdersDataset( + dynamoDbClient, + LargeDatasetInitializer.LARGE_CUSTOMERS_TABLE, + LargeDatasetInitializer.LARGE_ORDERS_TABLE, + SEED_CUSTOMER_COUNT, + SEED_ORDERS_PER_CUSTOMER, + DEFAULT_PROVISIONED_THROUGHPUT); + datasetSeeded = true; + } + + @AfterClass + public static void stopLocalDynamoDb() { + // Do not stop so other EnhancedQuery test classes in the same JVM reuse the DB and skip re-seeding. + } + + protected static DynamoDbClient getDynamoDbClient() { + return dynamoDbClient; + } + + protected static DynamoDbAsyncClient getDynamoDbAsyncClient() { + return dynamoDbAsyncClient; + } + + @Override + protected String getConcreteTableName(String logicalTableName) { + if ("customers".equals(logicalTableName)) { + return LargeDatasetInitializer.LARGE_CUSTOMERS_TABLE; + } + if ("orders".equals(logicalTableName)) { + return LargeDatasetInitializer.LARGE_ORDERS_TABLE; + } + return super.getConcreteTableName(logicalTableName); + } + + public static List drainPublisher(SdkPublisher publisher, int expectedNumberOfResults) { + BufferingSubscriber subscriber = new BufferingSubscriber<>(); + publisher.subscribe(subscriber); + subscriber.waitForCompletion(5000L); + + assertThat(subscriber.isCompleted(), is(true)); + assertThat(subscriber.bufferedError(), is(nullValue())); + assertThat(subscriber.bufferedItems().size(), is(expectedNumberOfResults)); + + return subscriber.bufferedItems(); + } + + public static List drainPublisherToError(SdkPublisher publisher, + int expectedNumberOfResults, + Class expectedError) { + BufferingSubscriber subscriber = new BufferingSubscriber<>(); + publisher.subscribe(subscriber); + subscriber.waitForCompletion(1000L); + + assertThat(subscriber.isCompleted(), is(false)); + assertThat(subscriber.bufferedError(), instanceOf(expectedError)); + assertThat(subscriber.bufferedItems().size(), is(expectedNumberOfResults)); + + return subscriber.bufferedItems(); + } +} diff --git a/services-custom/dynamodb-enhanced/src/test/java/software/amazon/awssdk/enhanced/dynamodb/functionaltests/LocalDynamoDbTestBase.java b/services-custom/dynamodb-enhanced/src/test/java/software/amazon/awssdk/enhanced/dynamodb/functionaltests/LocalDynamoDbTestBase.java index edce69d31825..79d610cd6123 100644 --- a/services-custom/dynamodb-enhanced/src/test/java/software/amazon/awssdk/enhanced/dynamodb/functionaltests/LocalDynamoDbTestBase.java +++ b/services-custom/dynamodb-enhanced/src/test/java/software/amazon/awssdk/enhanced/dynamodb/functionaltests/LocalDynamoDbTestBase.java @@ -15,7 +15,14 @@ package software.amazon.awssdk.enhanced.dynamodb.functionaltests; +import java.io.IOException; +import java.nio.file.Files; +import java.nio.file.Path; +import java.nio.file.Paths; +import java.nio.file.StandardOpenOption; +import java.util.Collections; import java.util.UUID; +import java.util.concurrent.atomic.AtomicBoolean; import org.junit.AfterClass; import org.junit.BeforeClass; import software.amazon.awssdk.services.dynamodb.model.ProvisionedThroughput; @@ -28,16 +35,22 @@ public class LocalDynamoDbTestBase { .writeCapacityUnits(50L) .build(); + private static final AtomicBoolean STARTED = new AtomicBoolean(false); + private String uniqueTableSuffix = UUID.randomUUID().toString(); @BeforeClass public static void initializeLocalDynamoDb() { - localDynamoDb.start(); + if (STARTED.compareAndSet(false, true)) { + localDynamoDb.start(); + } } @AfterClass public static void stopLocalDynamoDb() { - localDynamoDb.stop(); + if (STARTED.compareAndSet(true, false)) { + localDynamoDb.stop(); + } } protected static LocalDynamoDb localDynamoDb() { @@ -52,4 +65,21 @@ protected String getConcreteTableName(String logicalTableName) { protected ProvisionedThroughput getDefaultProvisionedThroughput() { return DEFAULT_PROVISIONED_THROUGHPUT; } + + private static final Path METRICS_FILE = Paths.get("target", "surefire-reports", "query-metrics.txt"); + + /** + * Appends a structured metric line for the script to read instead of parsing Surefire XML times or stdout logs. + * Format per line: {@code ClassName.testName } + */ + protected static synchronized void writeQueryMetric(String label, long queryMs, int rowCount) { + try { + Files.createDirectories(METRICS_FILE.getParent()); + Files.write(METRICS_FILE, + Collections.singletonList(label + " " + queryMs + " " + rowCount), + StandardOpenOption.CREATE, StandardOpenOption.APPEND); + } catch (IOException e) { + System.err.println("Failed to write query metric for " + label + ": " + e.getMessage()); + } + } } diff --git a/services-custom/dynamodb-enhanced/src/test/java/software/amazon/awssdk/enhanced/dynamodb/functionaltests/NestedAttributeFilteringTest.java b/services-custom/dynamodb-enhanced/src/test/java/software/amazon/awssdk/enhanced/dynamodb/functionaltests/NestedAttributeFilteringTest.java new file mode 100644 index 000000000000..1da41e8c95ed --- /dev/null +++ b/services-custom/dynamodb-enhanced/src/test/java/software/amazon/awssdk/enhanced/dynamodb/functionaltests/NestedAttributeFilteringTest.java @@ -0,0 +1,282 @@ +/* + * Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"). + * You may not use this file except in compliance with the License. + * A copy of the License is located at + * + * http://aws.amazon.com/apache2.0 + * + * or in the "license" file accompanying this file. This file is distributed + * on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either + * express or implied. See the License for the specific language governing + * permissions and limitations under the License. + */ + +package software.amazon.awssdk.enhanced.dynamodb.functionaltests; + +import static org.assertj.core.api.Assertions.assertThat; +import static software.amazon.awssdk.enhanced.dynamodb.mapper.StaticAttributeTags.primaryPartitionKey; + +import java.util.ArrayList; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import org.junit.After; +import org.junit.Before; +import org.junit.Test; +import software.amazon.awssdk.enhanced.dynamodb.DynamoDbEnhancedClient; +import software.amazon.awssdk.enhanced.dynamodb.DynamoDbTable; +import software.amazon.awssdk.enhanced.dynamodb.EnhancedType; +import software.amazon.awssdk.enhanced.dynamodb.TableSchema; +import software.amazon.awssdk.enhanced.dynamodb.mapper.StaticTableSchema; +import software.amazon.awssdk.enhanced.dynamodb.model.QueryConditional; +import software.amazon.awssdk.enhanced.dynamodb.query.condition.Condition; +import software.amazon.awssdk.enhanced.dynamodb.query.enums.ExecutionMode; +import software.amazon.awssdk.enhanced.dynamodb.query.result.EnhancedQueryResult; +import software.amazon.awssdk.enhanced.dynamodb.query.result.EnhancedQueryRow; +import software.amazon.awssdk.enhanced.dynamodb.query.engine.QueryExpressionBuilder; +import software.amazon.awssdk.enhanced.dynamodb.query.spec.QueryExpressionSpec; +import software.amazon.awssdk.services.dynamodb.model.DeleteTableRequest; + +/** + * Functional tests for nested attribute (dot-path) filtering in the enhanced query engine. + * Uses LocalDynamoDB with items that contain nested Map attributes (e.g. address.city). + */ +public class NestedAttributeFilteringTest extends LocalDynamoDbTestBase { + + private static final String TABLE_NAME = "nested_attr_test"; + + private static class CustomerWithAddress { + private String customerId; + private String name; + private Map address; + + public String getCustomerId() { + return customerId; + } + + public void setCustomerId(String customerId) { + this.customerId = customerId; + } + + public String getName() { + return name; + } + + public void setName(String name) { + this.name = name; + } + + public Map getAddress() { + return address; + } + + public void setAddress(Map address) { + this.address = address; + } + } + + private static final TableSchema SCHEMA = + StaticTableSchema.builder(CustomerWithAddress.class) + .newItemSupplier(CustomerWithAddress::new) + .addAttribute(String.class, + a -> a.name("customerId") + .getter(CustomerWithAddress::getCustomerId) + .setter(CustomerWithAddress::setCustomerId) + .tags(primaryPartitionKey())) + .addAttribute(String.class, + a -> a.name("name") + .getter(CustomerWithAddress::getName) + .setter(CustomerWithAddress::setName)) + .addAttribute(EnhancedType.mapOf(String.class, String.class), + a -> a.name("address") + .getter(CustomerWithAddress::getAddress) + .setter(CustomerWithAddress::setAddress)) + .build(); + + private DynamoDbEnhancedClient enhancedClient; + private DynamoDbTable table; + private String concreteTableName; + + @Before + public void setUp() { + concreteTableName = getConcreteTableName(TABLE_NAME); + enhancedClient = DynamoDbEnhancedClient.builder() + .dynamoDbClient(localDynamoDb().createClient()) + .build(); + table = enhancedClient.table(concreteTableName, SCHEMA); + table.createTable(r -> r.provisionedThroughput(getDefaultProvisionedThroughput())); + + putCustomer("c1", "Alice", "Seattle", "WA", "98101"); + putCustomer("c2", "Bob", "Portland", "OR", "97201"); + putCustomer("c3", "Charlie", "Seattle", "WA", "98102"); + putCustomer("c4", "Diana", "San Francisco", "CA", "94105"); + putCustomer("c5", "Eve", "New York", "NY", "10001"); + } + + @After + public void tearDown() { + try { + localDynamoDb().createClient().deleteTable(DeleteTableRequest.builder() + .tableName(concreteTableName) + .build()); + } catch (Exception ignored) { + // table may not exist + } + } + + private void putCustomer(String id, String name, String city, String state, String zip) { + CustomerWithAddress c = new CustomerWithAddress(); + c.setCustomerId(id); + c.setName(name); + Map address = new HashMap<>(); + address.put("city", city); + address.put("state", state); + address.put("zip", zip); + c.setAddress(address); + table.putItem(c); + } + + private String currentTestName() { + String thisClass = getClass().getName(); + for (StackTraceElement element : new Throwable().getStackTrace()) { + if (thisClass.equals(element.getClassName())) { + String method = element.getMethodName(); + if (!"currentTestName".equals(method) + && !"executeQuery".equals(method) + && !method.startsWith("lambda$") + && !method.startsWith("invoke")) { + return method; + } + } + } + return "unknownTest"; + } + + private List executeQuery(QueryExpressionSpec spec) { + String label = "NestedAttributeFilteringTest." + currentTestName(); + long start = System.nanoTime(); + EnhancedQueryResult result = enhancedClient.enhancedQuery(spec); + List rows = new ArrayList<>(); + result.forEach(rows::add); + long elapsedMs = (System.nanoTime() - start) / 1_000_000; + writeQueryMetric(label, elapsedMs, rows.size()); + return rows; + } + + @Test + public void filterByNestedCity_returnsMatchingRows() { + QueryExpressionSpec spec = QueryExpressionBuilder.from(table) + .keyCondition(QueryConditional.keyEqualTo(k -> k.partitionValue("c1"))) + .where(Condition.eq("address.city", "Seattle")) + .build(); + + List rows = executeQuery(spec); + assertThat(rows).hasSize(1); + assertThat(rows.get(0).getItem("base")).containsEntry("name", "Alice"); + } + + @Test + public void filterByNestedCity_noMatch() { + QueryExpressionSpec spec = QueryExpressionBuilder.from(table) + .keyCondition(QueryConditional.keyEqualTo(k -> k.partitionValue("c1"))) + .where(Condition.eq("address.city", "Portland")) + .build(); + + List rows = executeQuery(spec); + assertThat(rows).isEmpty(); + } + + @Test + public void filterByNestedState_scanMode() { + QueryExpressionSpec spec = QueryExpressionBuilder.from(table) + .executionMode(ExecutionMode.ALLOW_SCAN) + .where(Condition.eq("address.state", "WA")) + .build(); + + List rows = executeQuery(spec); + assertThat(rows).hasSize(2); + + List names = new ArrayList<>(); + for (EnhancedQueryRow row : rows) { + names.add((String) row.getItem("base").get("name")); + } + assertThat(names).containsExactlyInAnyOrder("Alice", "Charlie"); + } + + @Test + public void filterByNestedZip_between() { + QueryExpressionSpec spec = QueryExpressionBuilder.from(table) + .executionMode(ExecutionMode.ALLOW_SCAN) + .where(Condition.between("address.zip", "98000", "98999")) + .build(); + + List rows = executeQuery(spec); + assertThat(rows).hasSize(2); + } + + @Test + public void filterByNestedCity_contains() { + QueryExpressionSpec spec = QueryExpressionBuilder.from(table) + .executionMode(ExecutionMode.ALLOW_SCAN) + .where(Condition.contains("address.city", "land")) + .build(); + + List rows = executeQuery(spec); + assertThat(rows).hasSize(1); + assertThat(rows.get(0).getItem("base")).containsEntry("name", "Bob"); + } + + @Test + public void filterByNestedCity_beginsWith() { + QueryExpressionSpec spec = QueryExpressionBuilder.from(table) + .executionMode(ExecutionMode.ALLOW_SCAN) + .where(Condition.beginsWith("address.city", "San")) + .build(); + + List rows = executeQuery(spec); + assertThat(rows).hasSize(1); + assertThat(rows.get(0).getItem("base")).containsEntry("name", "Diana"); + } + + @Test + public void filterByNestedAndFlatAttribute_combined() { + QueryExpressionSpec spec = QueryExpressionBuilder.from(table) + .executionMode(ExecutionMode.ALLOW_SCAN) + .where(Condition.eq("address.state", "WA").and(Condition.eq("name", "Charlie"))) + .build(); + + List rows = executeQuery(spec); + assertThat(rows).hasSize(1); + assertThat(rows.get(0).getItem("base")).containsEntry("customerId", "c3"); + } + + @Test + public void filterByNestedAttribute_orCombination() { + QueryExpressionSpec spec = QueryExpressionBuilder.from(table) + .executionMode(ExecutionMode.ALLOW_SCAN) + .where(Condition.eq("address.state", "CA").or(Condition.eq("address.state", "NY"))) + .build(); + + List rows = executeQuery(spec); + assertThat(rows).hasSize(2); + + List names = new ArrayList<>(); + for (EnhancedQueryRow row : rows) { + names.add((String) row.getItem("base").get("name")); + } + assertThat(names).containsExactlyInAnyOrder("Diana", "Eve"); + } + + @Test + public void filterByMissingNestedPath_returnsNoMatch() { + QueryExpressionSpec spec = QueryExpressionBuilder.from(table) + .keyCondition(QueryConditional.keyEqualTo(k -> k.partitionValue("c1"))) + .where(Condition.eq("address.country", "US")) + .build(); + + List rows = executeQuery(spec); + assertThat(rows).isEmpty(); + } +} diff --git a/services-custom/dynamodb-enhanced/src/test/java/software/amazon/awssdk/enhanced/dynamodb/functionaltests/query/EnhancedQueryAggregationAsyncTest.java b/services-custom/dynamodb-enhanced/src/test/java/software/amazon/awssdk/enhanced/dynamodb/functionaltests/query/EnhancedQueryAggregationAsyncTest.java new file mode 100644 index 000000000000..05737cafb4c4 --- /dev/null +++ b/services-custom/dynamodb-enhanced/src/test/java/software/amazon/awssdk/enhanced/dynamodb/functionaltests/query/EnhancedQueryAggregationAsyncTest.java @@ -0,0 +1,646 @@ +/* + * Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"). + * You may not use this file except in compliance with the License. + * A copy of the License is located at + * + * http://aws.amazon.com/apache2.0 + * + * or in the "license" file accompanying this file. This file is distributed + * on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either + * express or implied. See the License for the specific language governing + * permissions and limitations under the License. + */ + +package software.amazon.awssdk.enhanced.dynamodb.functionaltests.query; + +import static org.assertj.core.api.Assertions.assertThat; +import static org.assertj.core.api.Assertions.entry; +import static software.amazon.awssdk.enhanced.dynamodb.mapper.StaticAttributeTags.primaryPartitionKey; +import static software.amazon.awssdk.enhanced.dynamodb.mapper.StaticAttributeTags.primarySortKey; + +import java.util.List; +import java.util.Map; +import java.util.Objects; +import org.junit.Before; +import org.junit.Test; +import software.amazon.awssdk.enhanced.dynamodb.DynamoDbAsyncTable; +import software.amazon.awssdk.enhanced.dynamodb.DynamoDbEnhancedAsyncClient; +import software.amazon.awssdk.enhanced.dynamodb.TableSchema; +import software.amazon.awssdk.enhanced.dynamodb.functionaltests.BufferingSubscriber; +import software.amazon.awssdk.enhanced.dynamodb.functionaltests.LargeDatasetInitializer; +import software.amazon.awssdk.enhanced.dynamodb.functionaltests.LocalDynamoDbLargeDatasetTestBase; +import software.amazon.awssdk.enhanced.dynamodb.internal.client.DefaultDynamoDbEnhancedAsyncClient; +import software.amazon.awssdk.enhanced.dynamodb.mapper.StaticTableSchema; +import software.amazon.awssdk.enhanced.dynamodb.model.QueryConditional; +import software.amazon.awssdk.enhanced.dynamodb.query.condition.Condition; +import software.amazon.awssdk.enhanced.dynamodb.query.enums.AggregationFunction; +import software.amazon.awssdk.enhanced.dynamodb.query.enums.ExecutionMode; +import software.amazon.awssdk.enhanced.dynamodb.query.enums.JoinType; +import software.amazon.awssdk.enhanced.dynamodb.query.enums.SortDirection; +import software.amazon.awssdk.enhanced.dynamodb.query.result.EnhancedQueryRow; +import software.amazon.awssdk.enhanced.dynamodb.query.engine.QueryExpressionBuilder; +import software.amazon.awssdk.enhanced.dynamodb.query.spec.QueryExpressionSpec; + +/** + * Aggregation-focused integration tests for the async enhanced query API. Assumes the large dataset has been seeded once via + * {@link LargeDatasetInitializer#main(String[])}. Each test drains the publisher, measures time, prints ms, fails if > 5 + * seconds. Covers GROUP BY, COUNT/SUM/MIN/MAX/AVG, filter, tree condition, ORDER BY aggregate/key, limit, ExecutionMode. + */ +public class EnhancedQueryAggregationAsyncTest extends LocalDynamoDbLargeDatasetTestBase { + + private static final long MAX_QUERY_MS = 5_000L; + private static final long DRAIN_TIMEOUT_MS = 10_000L; + + private static class CustomerRecord { + private String customerId; + private String name; + private String region; + + public String getCustomerId() { + return customerId; + } + + public void setCustomerId(String customerId) { + this.customerId = customerId; + } + + public String getName() { + return name; + } + + public void setName(String name) { + this.name = name; + } + + public String getRegion() { + return region; + } + + public void setRegion(String region) { + this.region = region; + } + + @Override + public boolean equals(Object o) { + if (this == o) { + return true; + } + if (o == null || getClass() != o.getClass()) { + return false; + } + CustomerRecord that = (CustomerRecord) o; + return Objects.equals(customerId, that.customerId) && Objects.equals(name, that.name) && Objects.equals(region, + that.region); + } + + @Override + public int hashCode() { + return Objects.hash(customerId, name, region); + } + } + + private static class OrderRecord { + private String customerId; + private String orderId; + private Integer amount; + + public String getCustomerId() { + return customerId; + } + + public void setCustomerId(String customerId) { + this.customerId = customerId; + } + + public String getOrderId() { + return orderId; + } + + public void setOrderId(String orderId) { + this.orderId = orderId; + } + + public Integer getAmount() { + return amount; + } + + public void setAmount(Integer amount) { + this.amount = amount; + } + + @Override + public boolean equals(Object o) { + if (this == o) { + return true; + } + if (o == null || getClass() != o.getClass()) { + return false; + } + OrderRecord that = (OrderRecord) o; + return Objects.equals(customerId, that.customerId) && Objects.equals(orderId, that.orderId) && Objects.equals(amount, that.amount); + } + + @Override + public int hashCode() { + return Objects.hash(customerId, orderId, amount); + } + } + + private static final TableSchema CUSTOMER_SCHEMA = + StaticTableSchema.builder(CustomerRecord.class) + .newItemSupplier(CustomerRecord::new) + .addAttribute(String.class, + a -> a.name("customerId").getter(CustomerRecord::getCustomerId).setter(CustomerRecord::setCustomerId).tags(primaryPartitionKey())) + .addAttribute(String.class, + a -> a.name("name").getter(CustomerRecord::getName).setter(CustomerRecord::setName)) + .addAttribute(String.class, + a -> a.name("region").getter(CustomerRecord::getRegion).setter(CustomerRecord::setRegion)) + .build(); + + private static final TableSchema ORDER_SCHEMA = + StaticTableSchema.builder(OrderRecord.class) + .newItemSupplier(OrderRecord::new) + .addAttribute(String.class, + a -> a.name("customerId").getter(OrderRecord::getCustomerId).setter(OrderRecord::setCustomerId).tags(primaryPartitionKey())) + .addAttribute(String.class, + a -> a.name("orderId").getter(OrderRecord::getOrderId).setter(OrderRecord::setOrderId).tags(primarySortKey())) + .addAttribute(Integer.class, + a -> a.name("amount").getter(OrderRecord::getAmount).setter(OrderRecord::setAmount)) + .build(); + + private DynamoDbEnhancedAsyncClient enhancedAsyncClient; + private DynamoDbAsyncTable customersTable; + private DynamoDbAsyncTable ordersTable; + + @Before + public void setUp() { + enhancedAsyncClient = DefaultDynamoDbEnhancedAsyncClient.builder() + .dynamoDbClient(getDynamoDbAsyncClient()) + .build(); + customersTable = enhancedAsyncClient.table(getConcreteTableName("customers"), CUSTOMER_SCHEMA); + ordersTable = enhancedAsyncClient.table(getConcreteTableName("orders"), ORDER_SCHEMA); + } + + private String currentTestName() { + String thisClass = getClass().getName(); + for (StackTraceElement element : new Throwable().getStackTrace()) { + if (thisClass.equals(element.getClassName())) { + String method = element.getMethodName(); + if (!"currentTestName".equals(method) + && !"runAndMeasure".equals(method) + && !method.startsWith("lambda$") + && !method.startsWith("invoke")) { + return method; + } + } + } + return "unknownTest"; + } + + private List runAndMeasure(QueryExpressionSpec spec) { + String testName = currentTestName(); + String label = "EnhancedQueryAggregationAsyncTest." + testName; + long start = System.nanoTime(); + BufferingSubscriber subscriber = new BufferingSubscriber<>(); + enhancedAsyncClient.enhancedQuery(spec).subscribe(subscriber); + subscriber.waitForCompletion(DRAIN_TIMEOUT_MS); + long elapsedMs = (System.nanoTime() - start) / 1_000_000; + List rows = subscriber.bufferedItems(); + System.out.println(label + " query took " + + elapsedMs + " ms, rows=" + rows.size()); + writeQueryMetric(label, elapsedMs, rows.size()); + assertThat(subscriber.bufferedError()).isNull(); + assertThat(elapsedMs).isLessThanOrEqualTo(MAX_QUERY_MS); + return rows; + } + + /** + * Base-only (no join, no aggregation): key condition and limit. Expected response: One row for c1 with customerId, name, + * region. DynamoDB operation: query() + */ + @Test + public void baseOnly_withKeyCondition() { + QueryExpressionSpec spec = QueryExpressionBuilder.from(customersTable) + .keyCondition(QueryConditional.keyEqualTo(k -> k.partitionValue( + "c1"))) + .limit(10) + .build(); + List rows = runAndMeasure(spec); + assertThat(rows).hasSize(1); + Map base = rows.get(0).getItem("base"); + assertThat(base).containsOnly( + entry("customerId", "c1"), + entry("name", "Customer1"), + entry("region", "EU")); + } + + /** + * GROUP BY customerId, COUNT(orderId) as orderCount for c1. Expected response: One row; orderCount = 1000L. DynamoDB + * operation: base=query(), join=query() + */ + @Test + public void aggregation_groupByCount() { + QueryExpressionSpec spec = QueryExpressionBuilder.from(customersTable) + .join(ordersTable, JoinType.INNER, "customerId", "customerId") + .groupBy("customerId") + .aggregate(AggregationFunction.COUNT, "orderId", "orderCount") + .keyCondition(QueryConditional.keyEqualTo(k -> k.partitionValue( + "c1"))) + .limit(10) + .build(); + List rows = runAndMeasure(spec); + assertThat(rows).hasSize(1); + assertThat(rows.get(0).getAggregate("orderCount")).isEqualTo(1000L); + assertThat(rows.get(0).aggregates()).containsKey("orderCount"); + Map base = rows.get(0).getItem("base"); + assertThat(base).containsEntry("customerId", "c1"); + assertThat(base).containsEntry("name", "Customer1"); + assertThat(base).containsEntry("region", "EU"); + } + + /** + * GROUP BY customerId, SUM(amount) as totalAmount for c1. Expected response: One row; totalAmount = 510500L. DynamoDB + * operation: base=query(), join=query() + */ + @Test + public void aggregation_groupBySum() { + QueryExpressionSpec spec = QueryExpressionBuilder.from(customersTable) + .join(ordersTable, JoinType.INNER, "customerId", "customerId") + .groupBy("customerId") + .aggregate(AggregationFunction.SUM, "amount", "totalAmount") + .keyCondition(QueryConditional.keyEqualTo(k -> k.partitionValue( + "c1"))) + .limit(10) + .build(); + List rows = runAndMeasure(spec); + assertThat(rows).hasSize(1); + assertThat(((Number) rows.get(0).getAggregate("totalAmount")).longValue()).isEqualTo(510500L); + assertThat(rows.get(0).aggregates()).containsKey("totalAmount"); + Map base = rows.get(0).getItem("base"); + assertThat(base).containsEntry("customerId", "c1"); + assertThat(base).containsEntry("name", "Customer1"); + assertThat(base).containsEntry("region", "EU"); + } + + /** + * GROUP BY + MIN and MAX. Asserts exact min/max for c1. DynamoDB operation: base=query(), join=query() + */ + @Test + public void aggregation_groupByMinMax() { + QueryExpressionSpec spec = QueryExpressionBuilder.from(customersTable) + .join(ordersTable, JoinType.INNER, "customerId", "customerId") + .groupBy("customerId") + .aggregate(AggregationFunction.MIN, "amount", "minAmount") + .aggregate(AggregationFunction.MAX, "amount", "maxAmount") + .keyCondition(QueryConditional.keyEqualTo(k -> k.partitionValue( + "c1"))) + .limit(10) + .build(); + List rows = runAndMeasure(spec); + assertThat(rows).hasSize(1); + assertThat(((Number) rows.get(0).getAggregate("minAmount")).intValue()).isEqualTo(11); + assertThat(((Number) rows.get(0).getAggregate("maxAmount")).intValue()).isEqualTo(1010); + assertThat(rows.get(0).aggregates()).containsKeys("minAmount", "maxAmount"); + } + + /** + * GROUP BY customerId, AVG(amount) for c1. Expected response: One row; avgAmount = 510.5. DynamoDB operation: base=query(), + * join=query() + */ + @Test + public void aggregation_groupByAvg() { + QueryExpressionSpec spec = QueryExpressionBuilder.from(customersTable) + .join(ordersTable, JoinType.INNER, "customerId", "customerId") + .groupBy("customerId") + .aggregate(AggregationFunction.AVG, "amount", "avgAmount") + .keyCondition(QueryConditional.keyEqualTo(k -> k.partitionValue( + "c1"))) + .limit(10) + .build(); + List rows = runAndMeasure(spec); + assertThat(rows).hasSize(1); + Object avg = rows.get(0).getAggregate("avgAmount"); + assertThat(avg).isNotNull(); + double avgVal = avg instanceof Number ? ((Number) avg).doubleValue() : Double.parseDouble(avg.toString()); + assertThat(avgVal).isEqualTo(510.5); + assertThat(rows.get(0).aggregates()).containsKey("avgAmount"); + } + + /** + * Aggregation with withBaseTableCondition(region=EU); ALLOW_SCAN. Expected response: One row per EU customer; orderCount = + * 1000L each. DynamoDB operation: base=scan(), join=query() + */ + @Test + public void aggregation_withFilter() { + QueryExpressionSpec spec = QueryExpressionBuilder.from(customersTable) + .join(ordersTable, JoinType.INNER, "customerId", "customerId") + .filterBase(Condition.eq("region", "EU")) + .groupBy("customerId") + .aggregate(AggregationFunction.COUNT, "orderId", "orderCount") + .executionMode(ExecutionMode.ALLOW_SCAN) + .limit(500) + .build(); + List rows = runAndMeasure(spec); + assertThat(rows).hasSize(EXPECTED_EU_CUSTOMER_COUNT); + for (EnhancedQueryRow row : rows) { + assertThat(row.getAggregate("orderCount")).isEqualTo(1000L); + assertThat(row.aggregates()).containsKey("orderCount"); + Map base = row.getItem("base"); + assertThat(base.get("region")).isEqualTo("EU"); + String customerId = (String) base.get("customerId"); + int num = Integer.parseInt(customerId.substring(1)); + assertThat(num % 2).as("EU customer should have odd ID: " + customerId).isEqualTo(1); + assertThat((String) base.get("name")).isEqualTo("Customer" + num); + } + } + + /** + * Aggregation with tree condition (region EU+name beginsWith C) OR region NA; ALLOW_SCAN. Expected response: One row per + * matching customer; orderCount = 1000L each. DynamoDB operation: base=scan(), join=query() + */ + @Test + public void aggregation_treeCondition() { + Condition tree = Condition.group( + Condition.eq("region", "EU").and(Condition.beginsWith("name", "C")) + ).or(Condition.eq("region", "NA")); + QueryExpressionSpec spec = QueryExpressionBuilder.from(customersTable) + .join(ordersTable, JoinType.INNER, "customerId", "customerId") + .filterBase(tree) + .groupBy("customerId") + .aggregate(AggregationFunction.COUNT, "orderId", "orderCount") + .executionMode(ExecutionMode.ALLOW_SCAN) + .limit(500) + .build(); + List rows = runAndMeasure(spec); + assertThat(rows).isNotEmpty(); + for (EnhancedQueryRow row : rows) { + assertThat(row.getAggregate("orderCount")).isEqualTo(1000L); + assertThat(row.aggregates()).containsKey("orderCount"); + Map base = row.getItem("base"); + String region = (String) base.get("region"); + String name = (String) base.get("name"); + boolean matchesCondition = + ("EU".equals(region) && name.startsWith("C")) || "NA".equals(region); + assertThat(matchesCondition) + .as("Row should match tree condition: region=%s, name=%s", region, name) + .isTrue(); + } + } + + /** + * ORDER BY aggregate DESC. DynamoDB operation: base=scan(), join=query() + */ + @Test + public void aggregation_orderByAggregate() { + QueryExpressionSpec spec = QueryExpressionBuilder.from(customersTable) + .join(ordersTable, JoinType.INNER, "customerId", "customerId") + .groupBy("customerId") + .aggregate(AggregationFunction.COUNT, "orderId", "orderCount") + .orderByAggregate("orderCount", SortDirection.DESC) + .executionMode(ExecutionMode.ALLOW_SCAN) + .limit(20) + .build(); + List rows = runAndMeasure(spec); + assertThat(rows).hasSizeLessThanOrEqualTo(20); + for (int i = 1; i < rows.size(); i++) { + Number prev = (Number) rows.get(i - 1).getAggregate("orderCount"); + Number curr = (Number) rows.get(i).getAggregate("orderCount"); + assertThat(prev.longValue()).isGreaterThanOrEqualTo(curr.longValue()); + } + for (EnhancedQueryRow row : rows) { + assertThat(row.getAggregate("orderCount")).isEqualTo(1000L); + } + } + + /** + * ORDER BY key customerId ASC; ALLOW_SCAN, limit 20. Expected response: Up to 20 rows; each orderCount = 1000L. DynamoDB + * operation: base=scan(), join=query() + */ + @Test + public void aggregation_orderByKey() { + QueryExpressionSpec spec = QueryExpressionBuilder.from(customersTable) + .join(ordersTable, JoinType.INNER, "customerId", "customerId") + .groupBy("customerId") + .aggregate(AggregationFunction.COUNT, "orderId", "orderCount") + .orderBy("customerId", SortDirection.ASC) + .executionMode(ExecutionMode.ALLOW_SCAN) + .limit(20) + .build(); + List rows = runAndMeasure(spec); + assertThat(rows).hasSizeLessThanOrEqualTo(20); + for (int i = 1; i < rows.size(); i++) { + String prev = (String) rows.get(i - 1).getItem("base").get("customerId"); + String curr = (String) rows.get(i).getItem("base").get("customerId"); + assertThat(prev.compareTo(curr)).as("customerIds should be ASC: %s <= %s", prev, curr) + .isLessThanOrEqualTo(0); + } + for (EnhancedQueryRow row : rows) { + assertThat(row.getAggregate("orderCount")).isEqualTo(1000L); + Map base = row.getItem("base"); + assertThat((String) base.get("customerId")).matches("c\\d+"); + assertThat((String) base.get("name")).startsWith("Customer"); + assertThat(base.get("region")).isIn("EU", "NA"); + } + } + + /** + * Limit on aggregation buckets; ALLOW_SCAN, limit 5. Expected response: At most 5 rows; each orderCount = 1000L. DynamoDB + * operation: base=scan(), join=query() + */ + @Test + public void aggregation_limit() { + QueryExpressionSpec spec = QueryExpressionBuilder.from(customersTable) + .join(ordersTable, JoinType.INNER, "customerId", "customerId") + .groupBy("customerId") + .aggregate(AggregationFunction.COUNT, "orderId", "orderCount") + .executionMode(ExecutionMode.ALLOW_SCAN) + .limit(5) + .build(); + List rows = runAndMeasure(spec); + assertThat(rows).hasSizeLessThanOrEqualTo(5); + for (EnhancedQueryRow row : rows) { + assertThat(row.getAggregate("orderCount")).isEqualTo(1000L); + } + } + + /** + * STRICT_KEY_ONLY with base key c1: aggregation runs. Expected response: One row; orderCount = 1000L. DynamoDB operation: + * base=query(), join=query() + */ + @Test + public void executionMode_strictKeyOnly_withKey() { + QueryExpressionSpec spec = QueryExpressionBuilder.from(customersTable) + .join(ordersTable, JoinType.INNER, "customerId", "customerId") + .groupBy("customerId") + .aggregate(AggregationFunction.COUNT, "orderId", "orderCount") + .executionMode(ExecutionMode.STRICT_KEY_ONLY) + .keyCondition(QueryConditional.keyEqualTo(k -> k.partitionValue( + "c1"))) + .limit(10) + .build(); + List rows = runAndMeasure(spec); + assertThat(rows).hasSize(1); + assertThat(rows.get(0).getAggregate("orderCount")).isEqualTo(1000L); + Map base = rows.get(0).getItem("base"); + assertThat(base).containsEntry("customerId", "c1"); + assertThat(base).containsEntry("name", "Customer1"); + assertThat(base).containsEntry("region", "EU"); + } + + /** + * ALLOW_SCAN aggregation over full scan; limit 100. Expected response: up to limit rows; each orderCount = 1000L. DynamoDB + * operation: base=scan(), join=query() + */ + @Test + public void executionMode_allowScan() { + QueryExpressionSpec spec = QueryExpressionBuilder.from(customersTable) + .join(ordersTable, JoinType.INNER, "customerId", "customerId") + .groupBy("customerId") + .aggregate(AggregationFunction.COUNT, "orderId", "orderCount") + .executionMode(ExecutionMode.ALLOW_SCAN) + .limit(100) + .build(); + List rows = runAndMeasure(spec); + assertThat(rows).hasSize(100); + for (EnhancedQueryRow row : rows) { + assertThat(row.getAggregate("orderCount")).isEqualTo(1000L); + assertThat(row.aggregates()).containsKey("orderCount"); + Map base = row.getItem("base"); + assertThat((String) base.get("customerId")).matches("c\\d+"); + assertThat((String) base.get("name")).startsWith("Customer"); + assertThat(base.get("region")).isIn("EU", "NA"); + } + } + + /** + * Join with withJoinedTableCondition(amount >= 50); groupBy customerId, COUNT(orderId) for c1. Expected response: One row; + * orderCount = 961 (EXPECTED_ORDER_COUNT_C1_AMOUNT_GE_50). DynamoDB operation: base=query(), join=query() + */ + @Test + public void aggregation_withJoinedTableCondition_returnsFilteredCount() { + QueryExpressionSpec spec = QueryExpressionBuilder.from(customersTable) + .join(ordersTable, JoinType.INNER, "customerId", "customerId") + .filterJoined(Condition.gte("amount", 50)) + .groupBy("customerId") + .aggregate(AggregationFunction.COUNT, "orderId", "orderCount") + .keyCondition(QueryConditional.keyEqualTo(k -> k.partitionValue("c1"))) + .limit(10) + .build(); + List rows = runAndMeasure(spec); + assertThat(rows).hasSize(1); + assertThat(rows.get(0).getAggregate("orderCount")).isEqualTo(EXPECTED_ORDER_COUNT_C1_AMOUNT_GE_50); + assertThat(rows.get(0).aggregates()).containsKey("orderCount"); + Map base = rows.get(0).getItem("base"); + assertThat(base).containsEntry("customerId", "c1"); + assertThat(base).containsEntry("name", "Customer1"); + assertThat(base).containsEntry("region", "EU"); + } + + /** + * Join with withJoinedTableCondition(amount >= 50); ALLOW_SCAN; one row per customer with filtered orderCount. Expected + * response: 100 rows (limited by spec); each row has orderCount present and <= SEED_ORDERS_PER_CUSTOMER. DynamoDB operation: + * base=scan(), join=query() + */ + @Test + public void aggregation_withJoinedTableCondition_allowScan_returnsFilteredCounts() { + QueryExpressionSpec spec = QueryExpressionBuilder.from(customersTable) + .join(ordersTable, JoinType.INNER, "customerId", "customerId") + .filterJoined(Condition.gte("amount", 50)) + .groupBy("customerId") + .aggregate(AggregationFunction.COUNT, "orderId", "orderCount") + .executionMode(ExecutionMode.ALLOW_SCAN) + .limit(100) + .build(); + List rows = runAndMeasure(spec); + assertThat(rows).hasSize(100); + for (EnhancedQueryRow row : rows) { + assertThat(row.getAggregate("orderCount")).isNotNull(); + long orderCount = ((Number) row.getAggregate("orderCount")).longValue(); + assertThat(orderCount).isGreaterThan(0); + assertThat(orderCount).isLessThanOrEqualTo(SEED_ORDERS_PER_CUSTOMER); + Map base = row.getItem("base"); + String customerId = (String) base.get("customerId"); + int k = Integer.parseInt(customerId.substring(1)); + long expectedCount = Math.max(0, SEED_ORDERS_PER_CUSTOMER - Math.max(0, 49 - 10 * k)); + assertThat(orderCount) + .as("orderCount for %s should match formula", customerId) + .isEqualTo(expectedCount); + } + } + + /** + * Exercises all aggregation builders: from, join, baseKeyCondition, withBaseTableCondition, withJoinedTableCondition, + * groupBy, aggregate (COUNT, SUM, MIN, MAX, AVG), orderBy, orderByAggregate, executionMode, limit. Expected response: One row + * (c1); orderCount=961, totalAmount/min/max/avg present and consistent. DynamoDB operation: base=query(), join=query() + */ + @Test + public void allBuilders_aggregationSpec_returnsExpectedAggregates() { + QueryExpressionSpec spec = QueryExpressionBuilder.from(customersTable) + .join(ordersTable, JoinType.INNER, "customerId", "customerId") + .keyCondition(QueryConditional.keyEqualTo(k -> k.partitionValue("c1"))) + .filterBase(Condition.eq("region", "EU")) + .filterJoined(Condition.gte("amount", 50)) + .groupBy("customerId") + .aggregate(AggregationFunction.COUNT, "orderId", "orderCount") + .aggregate(AggregationFunction.SUM, "amount", "totalAmount") + .aggregate(AggregationFunction.MIN, "amount", "minAmount") + .aggregate(AggregationFunction.MAX, "amount", "maxAmount") + .aggregate(AggregationFunction.AVG, "amount", "avgAmount") + .orderBy("customerId", SortDirection.ASC) + .orderByAggregate("orderCount", SortDirection.DESC) + .executionMode(ExecutionMode.STRICT_KEY_ONLY) + .limit(10) + .build(); + List rows = runAndMeasure(spec); + assertThat(rows).hasSize(1); + EnhancedQueryRow row = rows.get(0); + assertThat(row.getAggregate("orderCount")).isEqualTo(EXPECTED_ORDER_COUNT_C1_AMOUNT_GE_50); + assertThat(row.aggregates()).containsKeys("orderCount", "totalAmount", "minAmount", "maxAmount", "avgAmount"); + assertThat(((Number) row.getAggregate("minAmount")).intValue()).isEqualTo(EXPECTED_MIN_AMOUNT_C1_GE_50); + assertThat(((Number) row.getAggregate("maxAmount")).intValue()).isEqualTo(EXPECTED_MAX_AMOUNT_C1_GE_50); + assertThat(((Number) row.getAggregate("totalAmount")).longValue()).isEqualTo(EXPECTED_TOTAL_AMOUNT_C1_GE_50); + assertThat(((Number) row.getAggregate("avgAmount")).doubleValue()) + .isEqualTo(EXPECTED_AVG_AMOUNT_C1_GE_50); + long count = ((Number) row.getAggregate("orderCount")).longValue(); + double avg = ((Number) row.getAggregate("avgAmount")).doubleValue(); + long sum = ((Number) row.getAggregate("totalAmount")).longValue(); + assertThat(Math.abs(sum - count * avg)).as("sum should equal count * avg").isLessThan(1.0); + Map base = row.getItem("base"); + assertThat(base).containsEntry("customerId", "c1"); + assertThat(base).containsEntry("region", "EU"); + } + + /** + * STRICT_KEY_ONLY without base key condition: no scan; aggregation returns empty. Expected response: Empty list of rows. + * DynamoDB operation: none (returns empty) + */ + @Test + public void executionMode_strictKeyOnly_withoutKey_returnsEmptyOrNoScan() { + QueryExpressionSpec spec = QueryExpressionBuilder.from(customersTable) + .join(ordersTable, JoinType.INNER, "customerId", "customerId") + .groupBy("customerId") + .aggregate(AggregationFunction.COUNT, "orderId", "orderCount") + .executionMode(ExecutionMode.STRICT_KEY_ONLY) + .limit(10) + .build(); + List rows = runAndMeasure(spec); + assertThat(rows).isEmpty(); + } + + /** + * Base key condition for non-existent customer: no rows match. Expected response: Empty list of rows, no error. DynamoDB + * operation: query() + */ + @Test + public void emptyResult_returnsEmptyList() { + QueryExpressionSpec spec = QueryExpressionBuilder.from(customersTable) + .keyCondition(QueryConditional.keyEqualTo(k -> k.partitionValue( + "c99999"))) + .limit(10) + .build(); + List rows = runAndMeasure(spec); + assertThat(rows).isEmpty(); + } +} diff --git a/services-custom/dynamodb-enhanced/src/test/java/software/amazon/awssdk/enhanced/dynamodb/functionaltests/query/EnhancedQueryAggregationSyncTest.java b/services-custom/dynamodb-enhanced/src/test/java/software/amazon/awssdk/enhanced/dynamodb/functionaltests/query/EnhancedQueryAggregationSyncTest.java new file mode 100644 index 000000000000..a3cd2a736f5f --- /dev/null +++ b/services-custom/dynamodb-enhanced/src/test/java/software/amazon/awssdk/enhanced/dynamodb/functionaltests/query/EnhancedQueryAggregationSyncTest.java @@ -0,0 +1,681 @@ +/* + * Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"). + * You may not use this file except in compliance with the License. + * A copy of the License is located at + * + * http://aws.amazon.com/apache2.0 + * + * or in the "license" file accompanying this file. This file is distributed + * on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either + * express or implied. See the License for the specific language governing + * permissions and limitations under the License. + */ + +package software.amazon.awssdk.enhanced.dynamodb.functionaltests.query; + +import static org.assertj.core.api.Assertions.assertThat; +import static org.assertj.core.api.Assertions.entry; +import static software.amazon.awssdk.enhanced.dynamodb.mapper.StaticAttributeTags.primaryPartitionKey; +import static software.amazon.awssdk.enhanced.dynamodb.mapper.StaticAttributeTags.primarySortKey; + +import java.util.ArrayList; +import java.util.List; +import java.util.Map; +import java.util.Objects; +import org.junit.Before; +import org.junit.Test; +import software.amazon.awssdk.enhanced.dynamodb.DynamoDbEnhancedClient; +import software.amazon.awssdk.enhanced.dynamodb.DynamoDbTable; +import software.amazon.awssdk.enhanced.dynamodb.TableSchema; +import software.amazon.awssdk.enhanced.dynamodb.functionaltests.LargeDatasetInitializer; +import software.amazon.awssdk.enhanced.dynamodb.functionaltests.LocalDynamoDbLargeDatasetTestBase; +import software.amazon.awssdk.enhanced.dynamodb.mapper.StaticTableSchema; +import software.amazon.awssdk.enhanced.dynamodb.model.QueryConditional; +import software.amazon.awssdk.enhanced.dynamodb.query.condition.Condition; +import software.amazon.awssdk.enhanced.dynamodb.query.result.EnhancedQueryLatencyReport; +import software.amazon.awssdk.enhanced.dynamodb.query.result.EnhancedQueryResult; +import software.amazon.awssdk.enhanced.dynamodb.query.result.EnhancedQueryRow; +import software.amazon.awssdk.enhanced.dynamodb.query.engine.QueryExpressionBuilder; +import software.amazon.awssdk.enhanced.dynamodb.query.spec.QueryExpressionSpec; + +/** + * Aggregation-focused integration tests for the sync enhanced query API. Assumes the large dataset has been seeded once via + * {@link LargeDatasetInitializer#main(String[])}. No create/delete; each test measures query time, prints ms, fails if > 5 + * seconds. Covers GROUP BY, COUNT/SUM/MIN/MAX/AVG, filter, tree condition, ORDER BY aggregate/key, limit, ExecutionMode + * (base-only, ALLOW_SCAN, STRICT_KEY_ONLY). + */ +public class EnhancedQueryAggregationSyncTest extends LocalDynamoDbLargeDatasetTestBase { + + private static final long MAX_QUERY_MS = 5_000L; + + private static class CustomerRecord { + private String customerId; + private String name; + private String region; + + public String getCustomerId() { + return customerId; + } + + public void setCustomerId(String customerId) { + this.customerId = customerId; + } + + public String getName() { + return name; + } + + public void setName(String name) { + this.name = name; + } + + public String getRegion() { + return region; + } + + public void setRegion(String region) { + this.region = region; + } + + @Override + public boolean equals(Object o) { + if (this == o) { + return true; + } + if (o == null || getClass() != o.getClass()) { + return false; + } + CustomerRecord that = (CustomerRecord) o; + return Objects.equals(customerId, that.customerId) && Objects.equals(name, that.name) && Objects.equals(region, + that.region); + } + + @Override + public int hashCode() { + return Objects.hash(customerId, name, region); + } + } + + private static class OrderRecord { + private String customerId; + private String orderId; + private Integer amount; + + public String getCustomerId() { + return customerId; + } + + public void setCustomerId(String customerId) { + this.customerId = customerId; + } + + public String getOrderId() { + return orderId; + } + + public void setOrderId(String orderId) { + this.orderId = orderId; + } + + public Integer getAmount() { + return amount; + } + + public void setAmount(Integer amount) { + this.amount = amount; + } + + @Override + public boolean equals(Object o) { + if (this == o) { + return true; + } + if (o == null || getClass() != o.getClass()) { + return false; + } + OrderRecord that = (OrderRecord) o; + return Objects.equals(customerId, that.customerId) && Objects.equals(orderId, that.orderId) && Objects.equals(amount, that.amount); + } + + @Override + public int hashCode() { + return Objects.hash(customerId, orderId, amount); + } + } + + private static final TableSchema CUSTOMER_SCHEMA = + StaticTableSchema.builder(CustomerRecord.class) + .newItemSupplier(CustomerRecord::new) + .addAttribute(String.class, + a -> a.name("customerId").getter(CustomerRecord::getCustomerId).setter(CustomerRecord::setCustomerId).tags(primaryPartitionKey())) + .addAttribute(String.class, + a -> a.name("name").getter(CustomerRecord::getName).setter(CustomerRecord::setName)) + .addAttribute(String.class, + a -> a.name("region").getter(CustomerRecord::getRegion).setter(CustomerRecord::setRegion)) + .build(); + + private static final TableSchema ORDER_SCHEMA = + StaticTableSchema.builder(OrderRecord.class) + .newItemSupplier(OrderRecord::new) + .addAttribute(String.class, + a -> a.name("customerId").getter(OrderRecord::getCustomerId).setter(OrderRecord::setCustomerId).tags(primaryPartitionKey())) + .addAttribute(String.class, + a -> a.name("orderId").getter(OrderRecord::getOrderId).setter(OrderRecord::setOrderId).tags(primarySortKey())) + .addAttribute(Integer.class, + a -> a.name("amount").getter(OrderRecord::getAmount).setter(OrderRecord::setAmount)) + .build(); + + private DynamoDbEnhancedClient enhancedClient; + private DynamoDbTable customersTable; + private DynamoDbTable ordersTable; + + @Override + protected String getConcreteTableName(String logicalTableName) { + if ("customers".equals(logicalTableName)) { + return LargeDatasetInitializer.LARGE_CUSTOMERS_TABLE; + } + if ("orders".equals(logicalTableName)) { + return LargeDatasetInitializer.LARGE_ORDERS_TABLE; + } + return super.getConcreteTableName(logicalTableName); + } + + @Before + public void setUp() { + enhancedClient = DynamoDbEnhancedClient.builder().dynamoDbClient(getDynamoDbClient()).build(); + customersTable = enhancedClient.table(getConcreteTableName("customers"), CUSTOMER_SCHEMA); + ordersTable = enhancedClient.table(getConcreteTableName("orders"), ORDER_SCHEMA); + } + + private String currentTestName() { + String thisClass = getClass().getName(); + for (StackTraceElement element : new Throwable().getStackTrace()) { + if (thisClass.equals(element.getClassName())) { + String method = element.getMethodName(); + if (!"currentTestName".equals(method) + && !"runAndMeasure".equals(method) + && !method.startsWith("lambda$") + && !method.startsWith("invoke")) { + return method; + } + } + } + return "unknownTest"; + } + + private List runAndMeasure(QueryExpressionSpec spec) { + String testName = currentTestName(); + String label = "EnhancedQueryAggregationSyncTest." + testName; + EnhancedQueryLatencyReport[] reportHolder = new EnhancedQueryLatencyReport[1]; + EnhancedQueryResult result = enhancedClient.enhancedQuery(spec, r -> reportHolder[0] = r); + List rows = new ArrayList<>(); + result.forEach(rows::add); + EnhancedQueryLatencyReport report = reportHolder[0]; + long elapsedMs = report != null ? report.totalMs() : 0L; + if (report != null) { + System.out.println(label + + " EnhancedQueryLatencyReport: baseQueryMs=" + report.baseQueryMs() + + " joinedLookupsMs=" + report.joinedLookupsMs() + + " inMemoryProcessingMs=" + report.inMemoryProcessingMs() + + " totalMs=" + report.totalMs() + " rows=" + rows.size()); + } else { + System.out.println(label + " query took " + + elapsedMs + " ms, rows=" + rows.size()); + } + writeQueryMetric(label, elapsedMs, rows.size()); + assertThat(elapsedMs).isLessThanOrEqualTo(MAX_QUERY_MS); + return rows; + } + + /** + * Base-only (no join, no aggregation): key condition and limit. Expected response: One row for c1 with customerId, name, + * region. DynamoDB operation: query() + */ + @Test + public void baseOnly_withKeyCondition() { + QueryExpressionSpec spec = QueryExpressionBuilder.from(customersTable) + .keyCondition(QueryConditional.keyEqualTo(k -> k.partitionValue( + "c1"))) + .project("customerId", "name", "region", "orderId", "amount") + .limit(10) + .build(); + List rows = runAndMeasure(spec); + assertThat(rows).hasSize(1); + Map base = rows.get(0).getItem("base"); + assertThat(base).containsOnly( + entry("customerId", "c1"), + entry("name", "Customer1"), + entry("region", "EU")); + } + + /** + * GROUP BY customerId, COUNT(orderId) as orderCount for c1. Expected response: One row; orderCount = 1000L. DynamoDB + * operation: base=query(), join=query() + */ + @Test + public void aggregation_groupByCount() { + QueryExpressionSpec spec = QueryExpressionBuilder.from(customersTable) + .join(ordersTable, JoinType.INNER, "customerId", "customerId") + .groupBy("customerId") + .aggregate(AggregationFunction.COUNT, "orderId", "orderCount") + .keyCondition(QueryConditional.keyEqualTo(k -> k.partitionValue("c1"))) + .project("customerId", "name", "region", "orderId", "amount") + .limit(10) + .build(); + List rows = runAndMeasure(spec); + assertThat(rows).hasSize(1); + EnhancedQueryRow row = rows.get(0); + assertThat(row.getAggregate("orderCount")).isEqualTo(1000L); + assertThat(row.aggregates()).containsKey("orderCount"); + Map base = row.getItem("base"); + assertThat(base).containsEntry("customerId", "c1"); + assertThat(base).containsEntry("name", "Customer1"); + assertThat(base).containsEntry("region", "EU"); + } + + /** + * GROUP BY customerId, SUM(amount) as totalAmount for c1. Expected response: One row; totalAmount = 510500L. DynamoDB + * operation: base=query(), join=query() + */ + @Test + public void aggregation_groupBySum() { + QueryExpressionSpec spec = QueryExpressionBuilder.from(customersTable) + .join(ordersTable, JoinType.INNER, "customerId", "customerId") + .groupBy("customerId") + .aggregate(AggregationFunction.SUM, "amount", "totalAmount") + .keyCondition(QueryConditional.keyEqualTo(k -> k.partitionValue( + "c1"))) + .project("customerId", "name", "region", "orderId", "amount") + .limit(10) + .build(); + List rows = runAndMeasure(spec); + assertThat(rows).hasSize(1); + EnhancedQueryRow row = rows.get(0); + assertThat(((Number) row.getAggregate("totalAmount")).longValue()).isEqualTo(510500L); + assertThat(row.aggregates()).containsKey("totalAmount"); + Map base = row.getItem("base"); + assertThat(base).containsEntry("customerId", "c1"); + assertThat(base).containsEntry("name", "Customer1"); + assertThat(base).containsEntry("region", "EU"); + } + + /** + * GROUP BY customerId, MIN(amount) and MAX(amount) for c1. Expected response: One row; minAmount = 11, maxAmount = 110; + * aggregates contain both keys. DynamoDB operation: base=query(), join=query() + */ + @Test + public void aggregation_groupByMinMax() { + QueryExpressionSpec spec = QueryExpressionBuilder.from(customersTable) + .join(ordersTable, JoinType.INNER, "customerId", "customerId") + .groupBy("customerId") + .aggregate(AggregationFunction.MIN, "amount", "minAmount") + .aggregate(AggregationFunction.MAX, "amount", "maxAmount") + .keyCondition(QueryConditional.keyEqualTo(k -> k.partitionValue( + "c1"))) + .project("customerId", "name", "region", "orderId", "amount") + .limit(10) + .build(); + List rows = runAndMeasure(spec); + assertThat(rows).hasSize(1); + EnhancedQueryRow row = rows.get(0); + assertThat(((Number) row.getAggregate("minAmount")).intValue()).isEqualTo(11); + assertThat(((Number) row.getAggregate("maxAmount")).intValue()).isEqualTo(1010); + assertThat(row.aggregates()).containsKeys("minAmount", "maxAmount"); + } + + /** + * GROUP BY customerId, AVG(amount) for c1. Expected response: One row; avgAmount = 510.5. DynamoDB operation: base=query(), + * join=query() + */ + @Test + public void aggregation_groupByAvg() { + QueryExpressionSpec spec = QueryExpressionBuilder.from(customersTable) + .join(ordersTable, JoinType.INNER, "customerId", "customerId") + .groupBy("customerId") + .aggregate(AggregationFunction.AVG, "amount", "avgAmount") + .keyCondition(QueryConditional.keyEqualTo(k -> k.partitionValue( + "c1"))) + .project("customerId", "name", "region", "orderId", "amount") + .limit(10) + .build(); + List rows = runAndMeasure(spec); + assertThat(rows).hasSize(1); + Object avg = rows.get(0).getAggregate("avgAmount"); + assertThat(avg).isNotNull(); + double avgVal = avg instanceof Number ? ((Number) avg).doubleValue() : Double.parseDouble(avg.toString()); + assertThat(avgVal).isEqualTo(510.5); + assertThat(rows.get(0).aggregates()).containsKey("avgAmount"); + } + + /** + * Aggregation with withBaseTableCondition(region=EU); ALLOW_SCAN. Expected response: One row per EU customer; orderCount = + * 1000L each. DynamoDB operation: base=scan(), join=query() + */ + @Test + public void aggregation_withFilter() { + QueryExpressionSpec spec = QueryExpressionBuilder.from(customersTable) + .join(ordersTable, JoinType.INNER, "customerId", "customerId") + .filterBase(Condition.eq("region", "EU")) + .groupBy("customerId") + .aggregate(AggregationFunction.COUNT, "orderId", "orderCount") + .executionMode(ExecutionMode.ALLOW_SCAN) + .project("customerId", "name", "region", "orderId", "amount") + .limit(500) + .build(); + List rows = runAndMeasure(spec); + assertThat(rows).hasSize(EXPECTED_EU_CUSTOMER_COUNT); + for (EnhancedQueryRow row : rows) { + assertThat(row.getAggregate("orderCount")).isEqualTo(1000L); + assertThat(row.aggregates()).containsKey("orderCount"); + Map base = row.getItem("base"); + assertThat(base.get("region")).isEqualTo("EU"); + String customerId = (String) base.get("customerId"); + int num = Integer.parseInt(customerId.substring(1)); + assertThat(num % 2).as("EU customer should have odd ID: " + customerId).isEqualTo(1); + assertThat((String) base.get("name")).isEqualTo("Customer" + num); + } + } + + /** + * Aggregation with tree condition (region EU+name beginsWith C) OR region NA; ALLOW_SCAN. Expected response: One row per + * matching customer; orderCount = 1000L each. DynamoDB operation: base=scan(), join=query() + */ + @Test + public void aggregation_treeCondition() { + Condition tree = Condition.group( + Condition.eq("region", "EU").and(Condition.beginsWith("name", "C")) + ).or( + Condition.eq("region", "NA")); + + QueryExpressionSpec spec = QueryExpressionBuilder.from(customersTable) + .join(ordersTable, JoinType.INNER, "customerId", "customerId") + .filterBase(tree) + .groupBy("customerId") + .aggregate(AggregationFunction.COUNT, "orderId", "orderCount") + .executionMode(ExecutionMode.ALLOW_SCAN) + .project("customerId", "name", "region", "orderId", "amount") + .limit(500) + .build(); + List rows = runAndMeasure(spec); + assertThat(rows).isNotEmpty(); + for (EnhancedQueryRow row : rows) { + assertThat(row.getAggregate("orderCount")).isEqualTo(1000L); + assertThat(row.aggregates()).containsKey("orderCount"); + Map base = row.getItem("base"); + String region = (String) base.get("region"); + String name = (String) base.get("name"); + boolean matchesCondition = + ("EU".equals(region) && name.startsWith("C")) || "NA".equals(region); + assertThat(matchesCondition) + .as("Row should match tree condition: region=%s, name=%s", region, name) + .isTrue(); + } + } + + /** + * ORDER BY aggregate orderCount DESC; ALLOW_SCAN, limit 20. Expected response: Up to 20 rows; orderCount non-increasing; each + * orderCount = 1000L. DynamoDB operation: base=scan(), join=query() + */ + @Test + public void aggregation_orderByAggregate() { + QueryExpressionSpec spec = QueryExpressionBuilder.from(customersTable) + .join(ordersTable, JoinType.INNER, "customerId", "customerId") + .groupBy("customerId") + .aggregate(AggregationFunction.COUNT, "orderId", "orderCount") + .orderByAggregate("orderCount", SortDirection.DESC) + .executionMode(ExecutionMode.ALLOW_SCAN) + .project("customerId", "name", "region", "orderId", "amount") + .limit(20) + .build(); + List rows = runAndMeasure(spec); + assertThat(rows).hasSizeLessThanOrEqualTo(20); + for (int i = 1; i < rows.size(); i++) { + Number prev = (Number) rows.get(i - 1).getAggregate("orderCount"); + Number curr = (Number) rows.get(i).getAggregate("orderCount"); + assertThat(prev.longValue()).isGreaterThanOrEqualTo(curr.longValue()); + } + for (EnhancedQueryRow row : rows) { + assertThat(row.getAggregate("orderCount")).isEqualTo(1000L); + } + } + + /** + * ORDER BY key customerId ASC; ALLOW_SCAN, limit 20. Expected response: Up to 20 rows; each has orderCount = 1000L. DynamoDB + * operation: base=scan(), join=query() + */ + @Test + public void aggregation_orderByKey() { + QueryExpressionSpec spec = QueryExpressionBuilder.from(customersTable) + .join(ordersTable, JoinType.INNER, "customerId", "customerId") + .groupBy("customerId") + .aggregate(AggregationFunction.COUNT, "orderId", "orderCount") + .orderBy("customerId", SortDirection.ASC) + .executionMode(ExecutionMode.ALLOW_SCAN) + .project("customerId", "name", "region", "orderId", "amount") + .limit(20) + .build(); + List rows = runAndMeasure(spec); + assertThat(rows).hasSizeLessThanOrEqualTo(20); + for (int i = 1; i < rows.size(); i++) { + String prev = (String) rows.get(i - 1).getItem("base").get("customerId"); + String curr = (String) rows.get(i).getItem("base").get("customerId"); + assertThat(prev.compareTo(curr)).as("customerIds should be ASC: %s <= %s", prev, curr) + .isLessThanOrEqualTo(0); + } + for (EnhancedQueryRow row : rows) { + assertThat(row.getAggregate("orderCount")).isEqualTo(1000L); + Map base = row.getItem("base"); + assertThat((String) base.get("customerId")).matches("c\\d+"); + assertThat((String) base.get("name")).startsWith("Customer"); + assertThat(base.get("region")).isIn("EU", "NA"); + } + } + + /** + * Limit on aggregation buckets; ALLOW_SCAN, limit 5. Expected response: At most 5 rows; each orderCount = 1000L. DynamoDB + * operation: base=scan(), join=query() + */ + @Test + public void aggregation_limit() { + QueryExpressionSpec spec = QueryExpressionBuilder.from(customersTable) + .join(ordersTable, JoinType.INNER, "customerId", "customerId") + .groupBy("customerId") + .aggregate(AggregationFunction.COUNT, "orderId", "orderCount") + .executionMode(ExecutionMode.ALLOW_SCAN) + .project("customerId", "name", "region", "orderId", "amount") + .limit(5) + .build(); + List rows = runAndMeasure(spec); + assertThat(rows).hasSizeLessThanOrEqualTo(5); + for (EnhancedQueryRow row : rows) { + assertThat(row.getAggregate("orderCount")).isEqualTo(1000L); + } + } + + /** + * STRICT_KEY_ONLY with base key c1: aggregation runs. Expected response: One row; orderCount = 1000L. DynamoDB operation: + * base=query(), join=query() + */ + @Test + public void executionMode_strictKeyOnly_withKey() { + QueryExpressionSpec spec = QueryExpressionBuilder.from(customersTable) + .join(ordersTable, JoinType.INNER, "customerId", "customerId") + .groupBy("customerId") + .aggregate(AggregationFunction.COUNT, "orderId", "orderCount") + .executionMode(ExecutionMode.STRICT_KEY_ONLY) + .keyCondition(QueryConditional.keyEqualTo(k -> k.partitionValue( + "c1"))) + .project("customerId", "name", "region", "orderId", "amount") + .limit(10) + .build(); + List rows = runAndMeasure(spec); + assertThat(rows).hasSize(1); + assertThat(rows.get(0).getAggregate("orderCount")).isEqualTo(1000L); + Map base = rows.get(0).getItem("base"); + assertThat(base).containsEntry("customerId", "c1"); + assertThat(base).containsEntry("name", "Customer1"); + assertThat(base).containsEntry("region", "EU"); + } + + /** + * ALLOW_SCAN aggregation over full scan; limit 100. Expected response: up to limit rows; each orderCount = 1000L. DynamoDB + * operation: base=scan(), join=query() + */ + @Test + public void executionMode_allowScan() { + QueryExpressionSpec spec = QueryExpressionBuilder.from(customersTable) + .join(ordersTable, JoinType.INNER, "customerId", "customerId") + .groupBy("customerId") + .aggregate(AggregationFunction.COUNT, "orderId", "orderCount") + .executionMode(ExecutionMode.ALLOW_SCAN) + .project("customerId", "name", "region", "orderId", "amount") + .limit(100) + .build(); + List rows = runAndMeasure(spec); + assertThat(rows).hasSize(100); + for (EnhancedQueryRow row : rows) { + assertThat(row.getAggregate("orderCount")).isEqualTo(1000L); + assertThat(row.aggregates()).containsKey("orderCount"); + Map base = row.getItem("base"); + assertThat((String) base.get("customerId")).matches("c\\d+"); + assertThat((String) base.get("name")).startsWith("Customer"); + assertThat(base.get("region")).isIn("EU", "NA"); + } + } + + /** + * Join with withJoinedTableCondition(amount >= 50); groupBy customerId, COUNT(orderId) for c1. Expected response: One row; + * orderCount = 961 (EXPECTED_ORDER_COUNT_C1_AMOUNT_GE_50). DynamoDB operation: base=query(), join=query() + */ + @Test + public void aggregation_withJoinedTableCondition_returnsFilteredCount() { + QueryExpressionSpec spec = QueryExpressionBuilder.from(customersTable) + .join(ordersTable, JoinType.INNER, "customerId", "customerId") + .filterJoined(Condition.gte("amount", 50)) + .groupBy("customerId") + .aggregate(AggregationFunction.COUNT, "orderId", "orderCount") + .keyCondition(QueryConditional.keyEqualTo(k -> k.partitionValue("c1"))) + .project("customerId", "name", "region", "orderId", "amount") + .limit(10) + .build(); + List rows = runAndMeasure(spec); + assertThat(rows).hasSize(1); + assertThat(rows.get(0).getAggregate("orderCount")).isEqualTo(EXPECTED_ORDER_COUNT_C1_AMOUNT_GE_50); + assertThat(rows.get(0).aggregates()).containsKey("orderCount"); + Map base = rows.get(0).getItem("base"); + assertThat(base).containsEntry("customerId", "c1"); + assertThat(base).containsEntry("name", "Customer1"); + assertThat(base).containsEntry("region", "EU"); + } + + /** + * Join with withJoinedTableCondition(amount >= 50); ALLOW_SCAN; one row per customer with filtered orderCount. Expected + * response: 100 rows (limited by spec); each row has orderCount present and <= SEED_ORDERS_PER_CUSTOMER. DynamoDB operation: + * base=scan(), join=query() + */ + @Test + public void aggregation_withJoinedTableCondition_allowScan_returnsFilteredCounts() { + QueryExpressionSpec spec = QueryExpressionBuilder.from(customersTable) + .join(ordersTable, JoinType.INNER, "customerId", "customerId") + .filterJoined(Condition.gte("amount", 50)) + .groupBy("customerId") + .aggregate(AggregationFunction.COUNT, "orderId", "orderCount") + .executionMode(ExecutionMode.ALLOW_SCAN) + .project("customerId", "name", "region", "orderId", "amount") + .limit(100) + .build(); + List rows = runAndMeasure(spec); + assertThat(rows).hasSize(100); + for (EnhancedQueryRow row : rows) { + assertThat(row.getAggregate("orderCount")).isNotNull(); + long orderCount = ((Number) row.getAggregate("orderCount")).longValue(); + assertThat(orderCount).isGreaterThan(0); + assertThat(orderCount).isLessThanOrEqualTo(SEED_ORDERS_PER_CUSTOMER); + Map base = row.getItem("base"); + String customerId = (String) base.get("customerId"); + int k = Integer.parseInt(customerId.substring(1)); + long expectedCount = Math.max(0, SEED_ORDERS_PER_CUSTOMER - Math.max(0, 49 - 10 * k)); + assertThat(orderCount) + .as("orderCount for %s should match formula", customerId) + .isEqualTo(expectedCount); + } + } + + /** + * STRICT_KEY_ONLY without base key condition: no scan; aggregation returns empty. Expected response: Empty list of rows. + * DynamoDB operation: none (returns empty) + */ + @Test + public void executionMode_strictKeyOnly_withoutKey_returnsEmptyOrNoScan() { + QueryExpressionSpec spec = QueryExpressionBuilder.from(customersTable) + .join(ordersTable, JoinType.INNER, "customerId", "customerId") + .groupBy("customerId") + .aggregate(AggregationFunction.COUNT, "orderId", "orderCount") + .executionMode(ExecutionMode.STRICT_KEY_ONLY) + .project("customerId", "name", "region", "orderId", "amount") + .limit(10) + .build(); + List rows = runAndMeasure(spec); + assertThat(rows).isEmpty(); + } + + /** + * Exercises all aggregation builders: from, join, baseKeyCondition, withBaseTableCondition, withJoinedTableCondition, + * groupBy, aggregate (COUNT, SUM, MIN, MAX, AVG), orderBy, orderByAggregate, executionMode, limit. Expected response: One row + * (c1); orderCount=961, totalAmount/min/max/avg present and consistent. DynamoDB operation: base=query(), join=query() + */ + @Test + public void allBuilders_aggregationSpec_returnsExpectedAggregates() { + QueryExpressionSpec spec = QueryExpressionBuilder.from(customersTable) + .join(ordersTable, JoinType.INNER, "customerId", "customerId") + .keyCondition(QueryConditional.keyEqualTo(k -> k.partitionValue("c1"))) + .filterBase(Condition.eq("region", "EU")) + .filterJoined(Condition.gte("amount", 50)) + .groupBy("customerId") + .aggregate(AggregationFunction.COUNT, "orderId", "orderCount") + .aggregate(AggregationFunction.SUM, "amount", "totalAmount") + .aggregate(AggregationFunction.MIN, "amount", "minAmount") + .aggregate(AggregationFunction.MAX, "amount", "maxAmount") + .aggregate(AggregationFunction.AVG, "amount", "avgAmount") + .orderBy("customerId", SortDirection.ASC) + .orderByAggregate("orderCount", SortDirection.DESC) + .executionMode(ExecutionMode.STRICT_KEY_ONLY) + .project("customerId", "name", "region", "orderId", "amount") + .limit(10) + .build(); + List rows = runAndMeasure(spec); + assertThat(rows).hasSize(1); + EnhancedQueryRow row = rows.get(0); + assertThat(row.getAggregate("orderCount")).isEqualTo(EXPECTED_ORDER_COUNT_C1_AMOUNT_GE_50); + assertThat(row.aggregates()).containsKeys("orderCount", "totalAmount", "minAmount", "maxAmount", "avgAmount"); + assertThat(((Number) row.getAggregate("minAmount")).intValue()).isEqualTo(EXPECTED_MIN_AMOUNT_C1_GE_50); + assertThat(((Number) row.getAggregate("maxAmount")).intValue()).isEqualTo(EXPECTED_MAX_AMOUNT_C1_GE_50); + assertThat(((Number) row.getAggregate("totalAmount")).longValue()).isEqualTo(EXPECTED_TOTAL_AMOUNT_C1_GE_50); + assertThat(((Number) row.getAggregate("avgAmount")).doubleValue()) + .isEqualTo(EXPECTED_AVG_AMOUNT_C1_GE_50); + long count = ((Number) row.getAggregate("orderCount")).longValue(); + double avg = ((Number) row.getAggregate("avgAmount")).doubleValue(); + long sum = ((Number) row.getAggregate("totalAmount")).longValue(); + assertThat(Math.abs(sum - count * avg)).as("sum should equal count * avg").isLessThan(1.0); + Map base = row.getItem("base"); + assertThat(base).containsEntry("customerId", "c1"); + assertThat(base).containsEntry("region", "EU"); + } + + /** + * Base key condition for non-existent customer: no rows match. Expected response: Empty list of rows, no error. DynamoDB + * operation: query() + */ + @Test + public void emptyResult_returnsEmptyList() { + QueryExpressionSpec spec = QueryExpressionBuilder.from(customersTable) + .keyCondition(QueryConditional.keyEqualTo(k -> k.partitionValue( + "c99999"))) + .limit(10) + .build(); + List rows = runAndMeasure(spec); + assertThat(rows).isEmpty(); + } +} diff --git a/services-custom/dynamodb-enhanced/src/test/java/software/amazon/awssdk/enhanced/dynamodb/functionaltests/query/EnhancedQueryBenchmarkRunner.java b/services-custom/dynamodb-enhanced/src/test/java/software/amazon/awssdk/enhanced/dynamodb/functionaltests/query/EnhancedQueryBenchmarkRunner.java new file mode 100644 index 000000000000..e7a8d268f2ed --- /dev/null +++ b/services-custom/dynamodb-enhanced/src/test/java/software/amazon/awssdk/enhanced/dynamodb/functionaltests/query/EnhancedQueryBenchmarkRunner.java @@ -0,0 +1,685 @@ +/* + * Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"). + * You may not use this file except in compliance with the License. + * A copy of the License is located at + * + * http://aws.amazon.com/apache2.0 + * + * or in the "license" file accompanying this file. This file is distributed + * on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either + * express or implied. See the License for the specific language governing + * permissions and limitations under the License. + */ + +package software.amazon.awssdk.enhanced.dynamodb.functionaltests; + +import static software.amazon.awssdk.enhanced.dynamodb.mapper.StaticAttributeTags.primaryPartitionKey; +import static software.amazon.awssdk.enhanced.dynamodb.mapper.StaticAttributeTags.primarySortKey; + +import java.io.PrintStream; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.Collections; +import java.util.List; +import java.util.Locale; +import java.util.function.Supplier; +import software.amazon.awssdk.enhanced.dynamodb.DynamoDbEnhancedClient; +import software.amazon.awssdk.enhanced.dynamodb.DynamoDbTable; +import software.amazon.awssdk.enhanced.dynamodb.TableSchema; +import software.amazon.awssdk.enhanced.dynamodb.mapper.StaticTableSchema; +import software.amazon.awssdk.enhanced.dynamodb.model.QueryConditional; +import software.amazon.awssdk.enhanced.dynamodb.query.condition.Condition; +import software.amazon.awssdk.enhanced.dynamodb.query.engine.QueryExpressionBuilder; +import software.amazon.awssdk.enhanced.dynamodb.query.enums.AggregationFunction; +import software.amazon.awssdk.enhanced.dynamodb.query.enums.ExecutionMode; +import software.amazon.awssdk.enhanced.dynamodb.query.enums.JoinType; +import software.amazon.awssdk.enhanced.dynamodb.query.enums.SortDirection; +import software.amazon.awssdk.enhanced.dynamodb.query.result.EnhancedQueryLatencyReport; +import software.amazon.awssdk.enhanced.dynamodb.query.result.EnhancedQueryResult; +import software.amazon.awssdk.enhanced.dynamodb.query.result.EnhancedQueryRow; +import software.amazon.awssdk.enhanced.dynamodb.query.spec.QueryExpressionSpec; +import software.amazon.awssdk.regions.Region; +import software.amazon.awssdk.services.dynamodb.DynamoDbClient; +import software.amazon.awssdk.services.dynamodb.model.ProvisionedThroughput; + +/** + * Standalone benchmark runner for Enhanced Query (join and aggregation) scenarios. Connects to real DynamoDB (or DynamoDB Local) + * and runs a fixed set of query scenarios with warm-up and multiple iterations, then prints latency stats (avg, p50, p95) and row + * counts. + *

+ * Environment variables: + *

    + *
  • {@code AWS_REGION} – Optional. Region for DynamoDB (e.g. us-east-1). If unset, uses default region.
    • + *
  • {@code CUSTOMERS_TABLE} – Name of the Customers table (default: customers_large).
    • + *
  • {@code ORDERS_TABLE} – Name of the Orders table (default: orders_large).
    • + *
  • {@code CREATE_AND_SEED} – If "true", creates tables (if missing) and seeds 1000 customers x 1000 orders. + * Requires DynamoDB create/put permissions.
    • + *
  • {@code BENCHMARK_ITERATIONS} – Number of measured iterations per scenario (default: 5).
    • + *
  • {@code BENCHMARK_WARMUP} – Number of warm-up runs per scenario (default: 2).
    • + *
  • {@code BENCHMARK_OUTPUT_FILE} – Optional. If set, append CSV results to this file.
    • + *
  • {@code USE_LOCAL_DYNAMODB} – If "true", uses in-process DynamoDB Local: starts LocalDynamoDb, creates and seeds + * tables (1000 customers x 1000 orders), runs benchmarks, then stops. No AWS credentials required. Use + * {@code run-enhanced-query-benchmark-local.sh} to run this mode.
    • + *
+ *

+ * Run from repo root: + * 

+ * mvn test-compile exec:java -pl services-custom/dynamodb-enhanced \
+ *   -Dexec.mainClass="software.amazon.awssdk.enhanced.dynamodb.functionaltests.EnhancedQueryBenchmarkRunner" \
+ *   -Dexec.classpathScope=test
+ *
+ */ +public final class EnhancedQueryBenchmarkRunner { + + private static final String CUSTOMERS_TABLE_ENV = "CUSTOMERS_TABLE"; + private static final String ORDERS_TABLE_ENV = "ORDERS_TABLE"; + private static final String CREATE_AND_SEED_ENV = "CREATE_AND_SEED"; + private static final String BENCHMARK_ITERATIONS_ENV = "BENCHMARK_ITERATIONS"; + private static final String BENCHMARK_WARMUP_ENV = "BENCHMARK_WARMUP"; + private static final String BENCHMARK_OUTPUT_FILE_ENV = "BENCHMARK_OUTPUT_FILE"; + private static final String USE_LOCAL_DYNAMODB_ENV = "USE_LOCAL_DYNAMODB"; + + private static final String DEFAULT_CUSTOMERS_TABLE = "customers_large"; + private static final String DEFAULT_ORDERS_TABLE = "orders_large"; + private static final int DEFAULT_ITERATIONS = 5; + private static final int DEFAULT_WARMUP = 2; + + // Table column widths for aligned benchmark output + private static final int COL_SCENARIO = 38; + private static final int COL_DDB_OP = 26; + private static final int COL_DESCRIPTION = 62; + private static final int COL_AVG = 10; + private static final int COL_P50 = 10; + private static final int COL_P95 = 10; + private static final int COL_ROWS = 8; + + // Unicode box-drawing for table borders (easy to read) + private static final char BOX_H = '\u2500'; // ─ horizontal + private static final char BOX_V = '\u2502'; // │ vertical + private static final String BOX_TL = "\u250c"; // ┌ top-left + private static final String BOX_TC = "\u252c"; // ┬ top-center + private static final String BOX_TR = "\u2510"; // ┐ top-right + private static final String BOX_ML = "\u251c"; // ├ mid-left + private static final String BOX_MC = "\u253c"; // ┼ mid-cross + private static final String BOX_MR = "\u2524"; // ┤ mid-right + private static final String BOX_BL = "\u2514"; // └ bottom-left + private static final String BOX_BC = "\u2534"; // ┴ bottom-center + private static final String BOX_BR = "\u2518"; // ┘ bottom-right + + // ANSI color for latency columns (AVG, P50, P95) – cyan, reset for terminal + private static final String ANSI_CYAN = "\033[36m"; + private static final String ANSI_RESET = "\033[0m"; + + private static final ProvisionedThroughput PROVISIONED_THROUGHPUT = + ProvisionedThroughput.builder().readCapacityUnits(50L).writeCapacityUnits(50L).build(); + + // Minimal POJOs matching the table shape used by LargeDatasetInitializer and tests + private static class CustomerRecord { + private String customerId; + private String name; + private String region; + + public String getCustomerId() { + return customerId; + } + + public void setCustomerId(String customerId) { + this.customerId = customerId; + } + + public String getName() { + return name; + } + + public void setName(String name) { + this.name = name; + } + + public String getRegion() { + return region; + } + + public void setRegion(String region) { + this.region = region; + } + } + + private static class OrderRecord { + private String customerId; + private String orderId; + private Integer amount; + + public String getCustomerId() { + return customerId; + } + + public void setCustomerId(String customerId) { + this.customerId = customerId; + } + + public String getOrderId() { + return orderId; + } + + public void setOrderId(String orderId) { + this.orderId = orderId; + } + + public Integer getAmount() { + return amount; + } + + public void setAmount(Integer amount) { + this.amount = amount; + } + } + + private static final TableSchema CUSTOMER_SCHEMA = + StaticTableSchema.builder(CustomerRecord.class) + .newItemSupplier(CustomerRecord::new) + .addAttribute(String.class, + a -> a.name("customerId").getter(CustomerRecord::getCustomerId).setter(CustomerRecord::setCustomerId).tags(primaryPartitionKey())) + .addAttribute(String.class, + a -> a.name("name").getter(CustomerRecord::getName).setter(CustomerRecord::setName)) + .addAttribute(String.class, + a -> a.name("region").getter(CustomerRecord::getRegion).setter(CustomerRecord::setRegion)) + .build(); + + private static final TableSchema ORDER_SCHEMA = + StaticTableSchema.builder(OrderRecord.class) + .newItemSupplier(OrderRecord::new) + .addAttribute(String.class, + a -> a.name("customerId").getter(OrderRecord::getCustomerId).setter(OrderRecord::setCustomerId).tags(primaryPartitionKey())) + .addAttribute(String.class, + a -> a.name("orderId").getter(OrderRecord::getOrderId).setter(OrderRecord::setOrderId).tags(primarySortKey())) + .addAttribute(Integer.class, + a -> a.name("amount").getter(OrderRecord::getAmount).setter(OrderRecord::setAmount)) + .build(); + + public static void main(String[] args) { + String regionStr = System.getenv("AWS_REGION"); + String customersTable = System.getenv(CUSTOMERS_TABLE_ENV); + if (customersTable == null || customersTable.isEmpty()) { + customersTable = DEFAULT_CUSTOMERS_TABLE; + } + String ordersTable = System.getenv(ORDERS_TABLE_ENV); + if (ordersTable == null || ordersTable.isEmpty()) { + ordersTable = DEFAULT_ORDERS_TABLE; + } + boolean useLocalDynamoDb = "true".equalsIgnoreCase(System.getenv(USE_LOCAL_DYNAMODB_ENV)); + boolean createAndSeed = useLocalDynamoDb || "true".equalsIgnoreCase(System.getenv(CREATE_AND_SEED_ENV)); + int iterations = parseIntEnv(BENCHMARK_ITERATIONS_ENV, DEFAULT_ITERATIONS); + int warmup = parseIntEnv(BENCHMARK_WARMUP_ENV, DEFAULT_WARMUP); + String outputFile = System.getenv(BENCHMARK_OUTPUT_FILE_ENV); + + LocalDynamoDb localDynamoDb = null; + DynamoDbClient dynamoDbClient; + if (useLocalDynamoDb) { + localDynamoDb = new LocalDynamoDb(); + localDynamoDb.start(); + dynamoDbClient = localDynamoDb.createClient(); + System.out.println("Using in-process DynamoDB Local."); + } else if (regionStr != null && !regionStr.isEmpty()) { + dynamoDbClient = DynamoDbClient.builder().region(Region.of(regionStr)).build(); + } else { + dynamoDbClient = DynamoDbClient.create(); + } + + try { + if (createAndSeed) { + System.out.println("Creating tables and seeding data (1000 customers x 1000 orders)..."); + LargeDatasetInitializer.initializeCustomersAndOrdersDataset( + dynamoDbClient, + customersTable, + ordersTable, + 1000, + 1000, + PROVISIONED_THROUGHPUT); + System.out.println("Seed complete."); + } + + DynamoDbEnhancedClient enhancedClient = DynamoDbEnhancedClient.builder().dynamoDbClient(dynamoDbClient).build(); + DynamoDbTable customersTableRef = enhancedClient.table(customersTable, CUSTOMER_SCHEMA); + DynamoDbTable ordersTableRef = enhancedClient.table(ordersTable, ORDER_SCHEMA); + + List scenarios = Arrays.asList( + + // --- Base-only (no join, no aggregation) --- + + new Scenario("baseOnly_keyCondition", + "Get one customer by ID (c1). Uses partition key only; no join. DynamoDB: query() on Customers.", + "query()", + () -> QueryExpressionBuilder.from(customersTableRef) + .keyCondition(QueryConditional.keyEqualTo(k -> k.partitionValue("c1"))) + .project("customerId", "name", "region") + .limit(10) + .build()), + + new Scenario("baseOnly_scan_limit100", + "Read up to 100 customers without key condition (full table read). DynamoDB: scan() on Customers.", + "scan()", + () -> QueryExpressionBuilder.from(customersTableRef) + .executionMode(ExecutionMode.ALLOW_SCAN) + .project("customerId", "name", "region") + .limit(100) + .build()), + + new Scenario("baseOnly_scan_filterRegionEU", + "Scan Customers and filter in-memory to region=EU (~500 rows). DynamoDB: scan() on Customers.", + "scan()", + () -> QueryExpressionBuilder.from(customersTableRef) + .where(Condition.eq("region", "EU")) + .executionMode(ExecutionMode.ALLOW_SCAN) + .project("customerId", "name", "region") + .limit(600) + .build()), + + // --- Join (no aggregation) --- + + new Scenario("joinInner_c1", + "Customer c1 with all their orders (INNER join). Base by key, then orders by customerId. DynamoDB:" + + " query() + query().", + "base=query(), join=query()", + () -> QueryExpressionBuilder.from(customersTableRef) + .join(ordersTableRef, JoinType.INNER, "customerId", "customerId") + .keyCondition(QueryConditional.keyEqualTo(k -> k.partitionValue("c1"))) + .project("customerId", "name", "region", "orderId", "amount") + .limit(1100) + .build()), + + new Scenario("joinLeft_c1_limit50", + "Customer c1 LEFT-joined to orders, return first 50 rows (c1 plus up to 49 orders). DynamoDB: " + + "query() + query().", + "base=query(), join=query()", + () -> QueryExpressionBuilder.from(customersTableRef) + .join(ordersTableRef, JoinType.LEFT, "customerId", "customerId") + .keyCondition(QueryConditional.keyEqualTo(k -> k.partitionValue("c1"))) + .project("customerId", "name", "region", "orderId", "amount") + .limit(50) + .build()), + + new Scenario("joinRight_c1", + "All orders for customer c1, each with customer info (RIGHT join; 1000 rows). DynamoDB: query() + " + + "query().", + "base=query(), join=query()", + () -> QueryExpressionBuilder.from(customersTableRef) + .join(ordersTableRef, JoinType.RIGHT, "customerId", "customerId") + .keyCondition(QueryConditional.keyEqualTo(k -> k.partitionValue("c1"))) + .project("customerId", "name", "region", "orderId", "amount") + .limit(1100) + .build()), + + new Scenario("joinInner_c1_filterAmount50", + "Customer c1 INNER join orders, keep only orders with amount>=50. DynamoDB: query() + query(), " + + "filter in-memory.", + "base=query(), join=query()", + () -> QueryExpressionBuilder.from(customersTableRef) + .join(ordersTableRef, JoinType.INNER, "customerId", "customerId") + .keyCondition(QueryConditional.keyEqualTo(k -> k.partitionValue("c1"))) + .filterJoined(Condition.gte("amount", 50)) + .project("customerId", "name", "region", "orderId", "amount") + .limit(1000) + .build()), + + // --- Aggregation (join + GROUP BY) --- + + new Scenario("agg_count_c1", + "One row: customer c1 with COUNT(orders). Group by customerId. DynamoDB: query() on Customers + " + + "query() on Orders.", + "base=query(), join=query()", + () -> QueryExpressionBuilder.from(customersTableRef) + .join(ordersTableRef, JoinType.INNER, "customerId", "customerId") + .groupBy("customerId") + .aggregate(AggregationFunction.COUNT, "orderId", "orderCount") + .keyCondition(QueryConditional.keyEqualTo(k -> k.partitionValue("c1"))) + .project("customerId", "name", "region", "orderId", "amount") + .limit(10) + .build()), + + new Scenario("agg_sum_c1", + "One row: customer c1 with SUM(order amount). Group by customerId. DynamoDB: query() + query().", + "base=query(), join=query()", + () -> QueryExpressionBuilder.from(customersTableRef) + .join(ordersTableRef, JoinType.INNER, "customerId", "customerId") + .groupBy("customerId") + .aggregate(AggregationFunction.SUM, "amount", "totalAmount") + .keyCondition(QueryConditional.keyEqualTo(k -> k.partitionValue("c1"))) + .project("customerId", "name", "region", "orderId", "amount") + .limit(10) + .build()), + + new Scenario("agg_minMax_c1", + "One row: customer c1 with MIN(amount) and MAX(amount). Group by customerId. DynamoDB: query() + " + + "query().", + "base=query(), join=query()", + () -> QueryExpressionBuilder.from(customersTableRef) + .join(ordersTableRef, JoinType.INNER, "customerId", "customerId") + .groupBy("customerId") + .aggregate(AggregationFunction.MIN, "amount", "minAmount") + .aggregate(AggregationFunction.MAX, "amount", "maxAmount") + .keyCondition(QueryConditional.keyEqualTo(k -> k.partitionValue("c1"))) + .project("customerId", "name", "region", "orderId", "amount") + .limit(10) + .build()), + + new Scenario("agg_avg_c1", + "One row: customer c1 with AVG(order amount). Group by customerId. DynamoDB: query() + query().", + "base=query(), join=query()", + () -> QueryExpressionBuilder.from(customersTableRef) + .join(ordersTableRef, JoinType.INNER, "customerId", "customerId") + .groupBy("customerId") + .aggregate(AggregationFunction.AVG, "amount", "avgAmount") + .keyCondition(QueryConditional.keyEqualTo(k -> k.partitionValue("c1"))) + .project("customerId", "name", "region", "orderId", "amount") + .limit(10) + .build()), + + new Scenario("agg_allFunctions_c1_filterAmount50", + "c1: COUNT/SUM/MIN/MAX/AVG on orders with amount>=50, base filter region=EU. DynamoDB: query() + " + + "query().", + "base=query(), join=query()", + () -> QueryExpressionBuilder.from(customersTableRef) + .join(ordersTableRef, JoinType.INNER, "customerId", "customerId") + .keyCondition(QueryConditional.keyEqualTo(k -> k.partitionValue("c1"))) + .filterBase(Condition.eq("region", "EU")) + .filterJoined(Condition.gte("amount", 50)) + .groupBy("customerId") + .aggregate(AggregationFunction.COUNT, "orderId", "orderCount") + .aggregate(AggregationFunction.SUM, "amount", "totalAmount") + .aggregate(AggregationFunction.MIN, "amount", "minAmount") + .aggregate(AggregationFunction.MAX, "amount", "maxAmount") + .aggregate(AggregationFunction.AVG, "amount", "avgAmount") + .executionMode(ExecutionMode.STRICT_KEY_ONLY) + .project("customerId", "name", "region", "orderId", "amount") + .limit(10) + .build()), + + // --- Aggregation with scan --- + + new Scenario("agg_count_scanAll_limit20", + "COUNT(orders) per customer for first 20 customers. Base table read without key. DynamoDB: scan() " + + "+ query() per customer.", + "base=scan(), join=query()", + () -> QueryExpressionBuilder.from(customersTableRef) + .join(ordersTableRef, JoinType.INNER, "customerId", "customerId") + .groupBy("customerId") + .aggregate(AggregationFunction.COUNT, "orderId", "orderCount") + .executionMode(ExecutionMode.ALLOW_SCAN) + .project("customerId", "name", "region", "orderId", "amount") + .limit(20) + .build()), + + new Scenario("agg_count_scanFilterEU", + "COUNT(orders) per customer, only for customers with region=EU. Base read by scan + filter. " + + "DynamoDB: scan() + query().", + "base=scan(), join=query()", + () -> QueryExpressionBuilder.from(customersTableRef) + .join(ordersTableRef, JoinType.INNER, "customerId", "customerId") + .filterBase(Condition.eq("region", "EU")) + .groupBy("customerId") + .aggregate(AggregationFunction.COUNT, "orderId", "orderCount") + .executionMode(ExecutionMode.ALLOW_SCAN) + .project("customerId", "name", "region", "orderId", "amount") + .limit(500) + .build()), + + // --- Aggregation with ordering --- + + new Scenario("agg_count_orderByDesc_limit20", + "COUNT(orders) per customer, sort by count DESC, return top 20. Base by scan. DynamoDB: scan() + " + + "query(), sort in-memory.", + "base=scan(), join=query()", + () -> QueryExpressionBuilder.from(customersTableRef) + .join(ordersTableRef, JoinType.INNER, "customerId", "customerId") + .groupBy("customerId") + .aggregate(AggregationFunction.COUNT, "orderId", "orderCount") + .orderByAggregate("orderCount", SortDirection.DESC) + .executionMode(ExecutionMode.ALLOW_SCAN) + .project("customerId", "name", "region", "orderId", "amount") + .limit(20) + .build()) + ); + + PrintStream out = System.out; + StringBuilder csv = new StringBuilder(); + if (outputFile != null && !outputFile.isEmpty()) { + csv.append("scenario,description,ddbOperation,avgMs,p50Ms,p95Ms,rows,region,iterations\n"); + } + + out.println("Environment: " + (useLocalDynamoDb ? "DynamoDB Local (in-process)" : + "AWS_REGION=" + (regionStr != null ? regionStr : "default")) + + " CUSTOMERS_TABLE=" + customersTable + " ORDERS_TABLE=" + ordersTable); + out.println("Warmup=" + warmup + " Iterations=" + iterations); + out.println(); + out.println("DynamoDB operations:"); + out.println(" query() = read by partition (and optional sort) key; efficient, bounded by key."); + out.println(" scan() = full table (or index) read; no key condition; filters applied in-memory."); + out.println(" base=query(), join=query() = base table read by key, then joined table read by key per row."); + out.println(" base=scan(), join=query() = base table scanned, then joined table read by key per row."); + out.println(); + + String topBorder = tableBorder(BOX_TL, BOX_TC, BOX_TR); + String midBorder = tableBorder(BOX_ML, BOX_MC, BOX_MR); + String bottomBorder = tableBorder(BOX_BL, BOX_BC, BOX_BR); + out.println(topBorder); + out.println(tableDataRow( + padRight("SCENARIO", COL_SCENARIO), + padRight("DDB OPERATION", COL_DDB_OP), + padRight("DESCRIPTION", COL_DESCRIPTION), + ANSI_CYAN + padLeft("AVG(ms)", COL_AVG) + ANSI_RESET, + ANSI_CYAN + padLeft("P50(ms)", COL_P50) + ANSI_RESET, + ANSI_CYAN + padLeft("P95(ms)", COL_P95) + ANSI_RESET, + padLeft("ROWS", COL_ROWS))); + out.println(midBorder); + + for (int idx = 0; idx < scenarios.size(); idx++) { + Scenario scenario = scenarios.get(idx); + Result result = runScenario(enhancedClient, scenario, warmup, iterations); + List descLines = wrap(scenario.description, COL_DESCRIPTION); + String namePadded = padRight(truncate(scenario.name, COL_SCENARIO), COL_SCENARIO); + String ddbPadded = padRight(truncate(scenario.ddbOperation, COL_DDB_OP), COL_DDB_OP); + String avgStr = padLeft(String.format(Locale.US, "%.1f", result.avgMs), COL_AVG); + String p50Str = padLeft(String.valueOf(result.p50Ms), COL_P50); + String p95Str = padLeft(String.valueOf(result.p95Ms), COL_P95); + String rowsStr = padLeft(String.valueOf(result.rows), COL_ROWS); + String avgCol = ANSI_CYAN + avgStr + ANSI_RESET; + String p50Col = ANSI_CYAN + p50Str + ANSI_RESET; + String p95Col = ANSI_CYAN + p95Str + ANSI_RESET; + for (int i = 0; i < descLines.size(); i++) { + String descCell = padRight(descLines.get(i), COL_DESCRIPTION); + if (i == 0) { + out.println(tableDataRow(namePadded, ddbPadded, descCell, avgCol, p50Col, p95Col, rowsStr)); + } else { + out.println(tableDataRow( + repeat(' ', COL_SCENARIO), repeat(' ', COL_DDB_OP), descCell, + repeat(' ', COL_AVG), repeat(' ', COL_P50), repeat(' ', COL_P95), repeat(' ', COL_ROWS))); + } + } + if (csv.length() > 0) { + csv.append(String.format(Locale.US, "%s,\"%s\",\"%s\",%.2f,%d,%d,%d,%s,%d%n", + scenario.name, scenario.description, scenario.ddbOperation, + result.avgMs, result.p50Ms, result.p95Ms, result.rows, + useLocalDynamoDb ? "local" : (regionStr != null ? regionStr : "default"), + iterations)); + } + if (idx < scenarios.size() - 1) { + out.println(); + } + } + out.println(bottomBorder); + + if (outputFile != null && !outputFile.isEmpty() && csv.length() > 0) { + try { + java.nio.file.Files.write(java.nio.file.Paths.get(outputFile), + csv.toString().getBytes(java.nio.charset.StandardCharsets.UTF_8), + java.nio.file.StandardOpenOption.CREATE, java.nio.file.StandardOpenOption.APPEND); + out.println("Results appended to " + outputFile); + } catch (Exception e) { + System.err.println("Failed to write " + outputFile + ": " + e.getMessage()); + } + } + } finally { + dynamoDbClient.close(); + if (localDynamoDb != null) { + localDynamoDb.stop(); + } + } + } + + private static int parseIntEnv(String key, int defaultValue) { + String v = System.getenv(key); + if (v == null || v.isEmpty()) { + return defaultValue; + } + try { + return Integer.parseInt(v.trim()); + } catch (NumberFormatException e) { + return defaultValue; + } + } + + private static String truncate(String s, int maxLen) { + if (s == null) { + return ""; + } + if (s.length() <= maxLen) { + return s; + } + return s.substring(0, Math.max(0, maxLen - 3)) + "..."; + } + + /** + * Wraps text to multiple lines of at most maxLen characters, breaking at word boundaries when possible. + */ + private static List wrap(String s, int maxLen) { + List lines = new ArrayList<>(); + if (s == null || s.isEmpty()) { + lines.add(""); + return lines; + } + String rest = s.trim(); + while (!rest.isEmpty()) { + if (rest.length() <= maxLen) { + lines.add(rest); + break; + } + int breakAt = rest.lastIndexOf(' ', maxLen); + if (breakAt <= 0) { + breakAt = Math.min(maxLen, rest.length()); + } + lines.add(rest.substring(0, breakAt).trim()); + rest = rest.substring(breakAt).trim(); + } + return lines; + } + + private static String repeat(char ch, int count) { + StringBuilder sb = new StringBuilder(count); + for (int i = 0; i < count; i++) { + sb.append(ch); + } + return sb.toString(); + } + + private static String padRight(String s, int width) { + if (s == null) { + s = ""; + } + if (s.length() >= width) { + return s; + } + return s + repeat(' ', width - s.length()); + } + + private static String padLeft(String s, int width) { + if (s == null) { + s = ""; + } + if (s.length() >= width) { + return s; + } + return repeat(' ', width - s.length()) + s; + } + + /** + * Builds a horizontal table border (top, middle, or bottom) using box-drawing characters. + */ + private static String tableBorder(String left, String cross, String right) { + return left + repeat(BOX_H, COL_SCENARIO) + cross + repeat(BOX_H, COL_DDB_OP) + cross + + repeat(BOX_H, COL_DESCRIPTION) + cross + repeat(BOX_H, COL_AVG) + cross + + repeat(BOX_H, COL_P50) + cross + repeat(BOX_H, COL_P95) + cross + repeat(BOX_H, COL_ROWS) + right; + } + + /** + * Builds one row of table cells with vertical borders. + */ + private static String tableDataRow(String v1, String v2, String v3, String v4, String v5, String v6, String v7) { + return "" + BOX_V + v1 + BOX_V + v2 + BOX_V + v3 + BOX_V + v4 + BOX_V + v5 + BOX_V + v6 + BOX_V + v7 + BOX_V; + } + + private static class Scenario { + final String name; + final String description; + final String ddbOperation; + final Supplier specSupplier; + + Scenario(String name, String description, String ddbOperation, Supplier specSupplier) { + this.name = name; + this.description = description; + this.ddbOperation = ddbOperation; + this.specSupplier = specSupplier; + } + } + + private static class Result { + final double avgMs; + final long p50Ms; + final long p95Ms; + final int rows; + + Result(double avgMs, long p50Ms, long p95Ms, int rows) { + this.avgMs = avgMs; + this.p50Ms = p50Ms; + this.p95Ms = p95Ms; + this.rows = rows; + } + } + + private static Result runScenario(DynamoDbEnhancedClient enhancedClient, Scenario scenario, int warmup, int iterations) { + QueryExpressionSpec spec = scenario.specSupplier.get(); + for (int i = 0; i < warmup; i++) { + runOnce(enhancedClient, spec); + } + List times = new ArrayList<>(iterations); + int rows = 0; + for (int i = 0; i < iterations; i++) { + long[] msHolder = new long[1]; + int[] rowsHolder = new int[1]; + runOnce(enhancedClient, spec, msHolder, rowsHolder); + times.add(msHolder[0]); + rows = rowsHolder[0]; + } + Collections.sort(times); + long p50 = times.get((int) (iterations * 0.5)); + long p95 = times.get((int) Math.min(Math.ceil(iterations * 0.95) - 1, iterations - 1)); + double avg = times.stream().mapToLong(Long::longValue).average().orElse(0); + return new Result(avg, p50, p95, rows); + } + + private static void runOnce(DynamoDbEnhancedClient enhancedClient, QueryExpressionSpec spec, long[] outMs, int[] outRows) { + EnhancedQueryLatencyReport[] reportHolder = new EnhancedQueryLatencyReport[1]; + EnhancedQueryResult result = enhancedClient.enhancedQuery(spec, r -> reportHolder[0] = r); + int count = 0; + for (EnhancedQueryRow row : result) { + count++; + } + outMs[0] = reportHolder[0] != null ? reportHolder[0].totalMs() : 0L; + outRows[0] = count; + } + + private static void runOnce(DynamoDbEnhancedClient enhancedClient, QueryExpressionSpec spec) { + runOnce(enhancedClient, spec, new long[1], new int[1]); + } +} diff --git a/services-custom/dynamodb-enhanced/src/test/java/software/amazon/awssdk/enhanced/dynamodb/functionaltests/query/EnhancedQueryJoinAsyncTest.java b/services-custom/dynamodb-enhanced/src/test/java/software/amazon/awssdk/enhanced/dynamodb/functionaltests/query/EnhancedQueryJoinAsyncTest.java new file mode 100644 index 000000000000..336336bc8ab2 --- /dev/null +++ b/services-custom/dynamodb-enhanced/src/test/java/software/amazon/awssdk/enhanced/dynamodb/functionaltests/query/EnhancedQueryJoinAsyncTest.java @@ -0,0 +1,709 @@ +/* + * Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"). + * You may not use this file except in compliance with the License. + * A copy of the License is located at + * + * http://aws.amazon.com/apache2.0 + * + * or in the "license" file accompanying this file. This file is distributed + * on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either + * express or implied. See the License for the specific language governing + * permissions and limitations under the License. + */ + +package software.amazon.awssdk.enhanced.dynamodb.functionaltests.query; + +import static org.assertj.core.api.Assertions.assertThat; +import static org.assertj.core.api.Assertions.entry; +import static software.amazon.awssdk.enhanced.dynamodb.mapper.StaticAttributeTags.primaryPartitionKey; +import static software.amazon.awssdk.enhanced.dynamodb.mapper.StaticAttributeTags.primarySortKey; + +import java.util.HashSet; +import java.util.List; +import java.util.Map; +import java.util.Objects; +import java.util.Set; +import org.junit.Before; +import org.junit.Test; +import software.amazon.awssdk.enhanced.dynamodb.DynamoDbAsyncTable; +import software.amazon.awssdk.enhanced.dynamodb.DynamoDbEnhancedAsyncClient; +import software.amazon.awssdk.enhanced.dynamodb.TableSchema; +import software.amazon.awssdk.enhanced.dynamodb.functionaltests.BufferingSubscriber; +import software.amazon.awssdk.enhanced.dynamodb.functionaltests.LargeDatasetInitializer; +import software.amazon.awssdk.enhanced.dynamodb.functionaltests.LocalDynamoDbLargeDatasetTestBase; +import software.amazon.awssdk.enhanced.dynamodb.internal.client.DefaultDynamoDbEnhancedAsyncClient; +import software.amazon.awssdk.enhanced.dynamodb.mapper.StaticTableSchema; +import software.amazon.awssdk.enhanced.dynamodb.model.QueryConditional; +import software.amazon.awssdk.enhanced.dynamodb.query.condition.Condition; +import software.amazon.awssdk.enhanced.dynamodb.query.result.EnhancedQueryRow; +import software.amazon.awssdk.enhanced.dynamodb.query.engine.QueryExpressionBuilder; +import software.amazon.awssdk.enhanced.dynamodb.query.spec.QueryExpressionSpec; + +/** + * Join-focused integration tests for the async enhanced query API. Assumes the large dataset has been seeded once via + * {@link LargeDatasetInitializer#main(String[])}. Does not create or delete tables. Each test drains the publisher, measures + * execution time, prints ms, and fails if > 1 second. + */ +public class EnhancedQueryJoinAsyncTest extends LocalDynamoDbLargeDatasetTestBase { + + private static final long MAX_QUERY_MS = 1_000L; + private static final long DRAIN_TIMEOUT_MS = 10_000L; + + private static class CustomerRecord { + private String customerId; + private String name; + private String region; + + public String getCustomerId() { + return customerId; + } + + public void setCustomerId(String customerId) { + this.customerId = customerId; + } + + public String getName() { + return name; + } + + public void setName(String name) { + this.name = name; + } + + public String getRegion() { + return region; + } + + public void setRegion(String region) { + this.region = region; + } + + @Override + public boolean equals(Object o) { + if (this == o) { + return true; + } + if (o == null || getClass() != o.getClass()) { + return false; + } + CustomerRecord that = (CustomerRecord) o; + return Objects.equals(customerId, that.customerId) && Objects.equals(name, that.name) && Objects.equals(region, + that.region); + } + + @Override + public int hashCode() { + return Objects.hash(customerId, name, region); + } + } + + private static class OrderRecord { + private String customerId; + private String orderId; + private Integer amount; + + public String getCustomerId() { + return customerId; + } + + public void setCustomerId(String customerId) { + this.customerId = customerId; + } + + public String getOrderId() { + return orderId; + } + + public void setOrderId(String orderId) { + this.orderId = orderId; + } + + public Integer getAmount() { + return amount; + } + + public void setAmount(Integer amount) { + this.amount = amount; + } + + @Override + public boolean equals(Object o) { + if (this == o) { + return true; + } + if (o == null || getClass() != o.getClass()) { + return false; + } + OrderRecord that = (OrderRecord) o; + return Objects.equals(customerId, that.customerId) && Objects.equals(orderId, that.orderId) && Objects.equals(amount, that.amount); + } + + @Override + public int hashCode() { + return Objects.hash(customerId, orderId, amount); + } + } + + private static final TableSchema CUSTOMER_SCHEMA = + StaticTableSchema.builder(CustomerRecord.class) + .newItemSupplier(CustomerRecord::new) + .addAttribute(String.class, + a -> a.name("customerId").getter(CustomerRecord::getCustomerId).setter(CustomerRecord::setCustomerId).tags(primaryPartitionKey())) + .addAttribute(String.class, + a -> a.name("name").getter(CustomerRecord::getName).setter(CustomerRecord::setName)) + .addAttribute(String.class, + a -> a.name("region").getter(CustomerRecord::getRegion).setter(CustomerRecord::setRegion)) + .build(); + + private static final TableSchema ORDER_SCHEMA = + StaticTableSchema.builder(OrderRecord.class) + .newItemSupplier(OrderRecord::new) + .addAttribute(String.class, + a -> a.name("customerId").getter(OrderRecord::getCustomerId).setter(OrderRecord::setCustomerId).tags(primaryPartitionKey())) + .addAttribute(String.class, + a -> a.name("orderId").getter(OrderRecord::getOrderId).setter(OrderRecord::setOrderId).tags(primarySortKey())) + .addAttribute(Integer.class, + a -> a.name("amount").getter(OrderRecord::getAmount).setter(OrderRecord::setAmount)) + .build(); + + private DynamoDbEnhancedAsyncClient enhancedAsyncClient; + private DynamoDbAsyncTable customersTable; + private DynamoDbAsyncTable ordersTable; + + @Override + protected String getConcreteTableName(String logicalTableName) { + if ("customers".equals(logicalTableName)) { + return LargeDatasetInitializer.LARGE_CUSTOMERS_TABLE; + } + if ("orders".equals(logicalTableName)) { + return LargeDatasetInitializer.LARGE_ORDERS_TABLE; + } + return super.getConcreteTableName(logicalTableName); + } + + @Before + public void setUp() { + enhancedAsyncClient = DefaultDynamoDbEnhancedAsyncClient.builder() + .dynamoDbClient(getDynamoDbAsyncClient()) + .build(); + customersTable = enhancedAsyncClient.table(getConcreteTableName("customers"), CUSTOMER_SCHEMA); + ordersTable = enhancedAsyncClient.table(getConcreteTableName("orders"), ORDER_SCHEMA); + } + + private String currentTestName() { + String thisClass = getClass().getName(); + for (StackTraceElement element : new Throwable().getStackTrace()) { + if (thisClass.equals(element.getClassName())) { + String method = element.getMethodName(); + if (!"currentTestName".equals(method) + && !"runAndMeasure".equals(method) + && !method.startsWith("lambda$") + && !method.startsWith("invoke")) { + return method; + } + } + } + return "unknownTest"; + } + + private List runAndMeasure(QueryExpressionSpec spec) { + String testName = currentTestName(); + String label = "EnhancedQueryJoinAsyncTest." + testName; + long start = System.nanoTime(); + BufferingSubscriber subscriber = new BufferingSubscriber<>(); + enhancedAsyncClient.enhancedQuery(spec).subscribe(subscriber); + subscriber.waitForCompletion(DRAIN_TIMEOUT_MS); + long elapsedMs = (System.nanoTime() - start) / 1_000_000; + List rows = subscriber.bufferedItems(); + System.out.println(label + " query took " + + elapsedMs + " ms, rows=" + rows.size()); + writeQueryMetric(label, elapsedMs, rows.size()); + assertThat(subscriber.bufferedError()).isNull(); + assertThat(elapsedMs).isLessThanOrEqualTo(MAX_QUERY_MS); + return rows; + } + + /** + * Base-only: query customers with key condition and limit. Expected response: Single row for c1 with customerId, name, + * region; name=Customer1, region=EU. DynamoDB operation: query() + */ + @Test + public void baseOnly_withKeyConditionAndLimit() { + QueryExpressionSpec spec = QueryExpressionBuilder.from(customersTable) + .keyCondition(QueryConditional.keyEqualTo(k -> k.partitionValue( + "c1"))) + .project("customerId", "name", "region", "orderId", "amount") + .limit(10) + .build(); + List rows = runAndMeasure(spec); + assertThat(rows).hasSize(1); + Map base = rows.get(0).getItem("base"); + assertThat(base).containsOnly( + entry("customerId", "c1"), + entry("name", "Customer1"), + entry("region", "EU")); + } + + /** + * INNER join on customerId with key condition for c1. Expected response: One row per order of c1; each row has base and + * joined with customerId, orderId, amount; first order c1-o1, amount 11. DynamoDB operation: base=query(), join=query() + */ + @Test + public void joinInner_returnsMatchingPairs() { + QueryExpressionSpec spec = QueryExpressionBuilder.from(customersTable) + .join(ordersTable, JoinType.INNER, "customerId", "customerId") + .keyCondition(QueryConditional.keyEqualTo(k -> k.partitionValue( + "c1"))) + .project("customerId", "name", "region", "orderId", "amount") + .limit(SEED_ORDERS_PER_CUSTOMER + 100) + .build(); + List rows = runAndMeasure(spec); + assertThat(rows).hasSize(SEED_ORDERS_PER_CUSTOMER); + Set orderIds = new HashSet<>(); + for (EnhancedQueryRow row : rows) { + Map base = row.getItem("base"); + Map joined = row.getItem("joined"); + assertThat(base).containsOnly( + entry("customerId", "c1"), + entry("name", "Customer1"), + entry("region", "EU")); + assertThat(joined).containsOnlyKeys("customerId", "orderId", "amount"); + assertThat(joined.get("customerId")).isEqualTo("c1"); + String orderId = (String) joined.get("orderId"); + assertThat(orderId).matches("c1-o\\d+"); + assertThat(orderIds.add(orderId)).as("Order ID should be unique: " + orderId).isTrue(); + int orderNum = Integer.parseInt(orderId.substring(orderId.lastIndexOf('o') + 1)); + int amount = ((Number) joined.get("amount")).intValue(); + assertThat(amount).as("amount for %s should be 10+%d", orderId, orderNum).isEqualTo(10 + orderNum); + } + assertThat(orderIds).hasSize(SEED_ORDERS_PER_CUSTOMER); + assertThat(rows.get(0).getItem("joined").get("orderId")).isEqualTo("c1-o1"); + assertThat(((Number) rows.get(0).getItem("joined").get("amount")).intValue()).isEqualTo(11); + } + + /** + * LEFT join on customerId for c1: every base row with optional joined (limit 50). Expected response: 50 rows; each has base + * (c1) and joined (c1, orderId, amount). DynamoDB operation: base=query(), join=query() + */ + @Test + public void joinLeft_returnsBaseRowsWithOptionalJoined() { + QueryExpressionSpec spec = QueryExpressionBuilder.from(customersTable) + .join(ordersTable, JoinType.LEFT, "customerId", "customerId") + .keyCondition(QueryConditional.keyEqualTo(k -> k.partitionValue( + "c1"))) + .project("customerId", "name", "region", "orderId", "amount") + .limit(50) + .build(); + List rows = runAndMeasure(spec); + assertThat(rows).hasSize(50); + Set orderIds = new HashSet<>(); + for (EnhancedQueryRow row : rows) { + Map base = row.getItem("base"); + Map joined = row.getItem("joined"); + assertThat(base).containsOnly( + entry("customerId", "c1"), + entry("name", "Customer1"), + entry("region", "EU")); + assertThat(joined).containsOnlyKeys("customerId", "orderId", "amount"); + assertThat(joined.get("customerId")).isEqualTo("c1"); + String orderId = (String) joined.get("orderId"); + assertThat(orderIds.add(orderId)).as("Order ID should be unique: " + orderId).isTrue(); + int orderNum = Integer.parseInt(orderId.substring(orderId.lastIndexOf('o') + 1)); + int amount = ((Number) joined.get("amount")).intValue(); + assertThat(amount).isEqualTo(10 + orderNum); + } + assertThat(orderIds).hasSize(50); + } + + /** + * RIGHT join on customerId for c1: every joined row with optional base. Expected response: SEED_ORDERS_PER_CUSTOMER rows; + * each has joined (c1, orderId, amount) and base (c1). DynamoDB operation: base=query(), join=query() + */ + @Test + public void joinRight_returnsJoinedRowsWithOptionalBase() { + QueryExpressionSpec spec = QueryExpressionBuilder.from(customersTable) + .join(ordersTable, JoinType.RIGHT, "customerId", "customerId") + .keyCondition(QueryConditional.keyEqualTo(k -> k.partitionValue( + "c1"))) + .project("customerId", "name", "region", "orderId", "amount") + .limit(SEED_ORDERS_PER_CUSTOMER + 100) + .build(); + List rows = runAndMeasure(spec); + assertThat(rows).hasSize(SEED_ORDERS_PER_CUSTOMER); + Set orderIds = new HashSet<>(); + for (EnhancedQueryRow row : rows) { + Map base = row.getItem("base"); + Map joined = row.getItem("joined"); + assertThat(joined).containsOnlyKeys("customerId", "orderId", "amount"); + assertThat(joined.get("customerId")).isEqualTo("c1"); + String orderId = (String) joined.get("orderId"); + assertThat(orderIds.add(orderId)).as("Order ID should be unique: " + orderId).isTrue(); + int orderNum = Integer.parseInt(orderId.substring(orderId.lastIndexOf('o') + 1)); + int amount = ((Number) joined.get("amount")).intValue(); + assertThat(amount).isEqualTo(10 + orderNum); + assertThat(base).containsOnly( + entry("customerId", "c1"), + entry("name", "Customer1"), + entry("region", "EU")); + } + assertThat(orderIds).hasSize(SEED_ORDERS_PER_CUSTOMER); + } + + /** + * FULL join on customerId for c1: union of LEFT and RIGHT (limit 150). Expected response: 150 rows; rows may have empty base + * (right-only) or empty joined (left-only); when present, base is c1. DynamoDB operation: base=query(), join=query() + */ + @Test + public void joinFull_returnsUnionOfLeftAndRight() { + QueryExpressionSpec spec = QueryExpressionBuilder.from(customersTable) + .join(ordersTable, JoinType.FULL, "customerId", "customerId") + .keyCondition(QueryConditional.keyEqualTo(k -> k.partitionValue( + "c1"))) + .project("customerId", "name", "region", "orderId", "amount") + .limit(150) + .build(); + List rows = runAndMeasure(spec); + assertThat(rows).hasSize(150); + int bothPresent = 0; + for (EnhancedQueryRow row : rows) { + Map base = row.getItem("base"); + Map joined = row.getItem("joined"); + if (!base.isEmpty()) { + assertThat(base).containsOnlyKeys("customerId", "name", "region"); + assertThat(base.get("customerId")).isEqualTo("c1"); + assertThat(base.get("name")).isEqualTo("Customer1"); + assertThat(base.get("region")).isEqualTo("EU"); + } + if (!joined.isEmpty()) { + assertThat(joined).containsOnlyKeys("customerId", "orderId", "amount"); + assertThat(joined.get("customerId")).isEqualTo("c1"); + int orderNum = Integer.parseInt(((String) joined.get("orderId")).substring( + ((String) joined.get("orderId")).lastIndexOf('o') + 1)); + assertThat(((Number) joined.get("amount")).intValue()).isEqualTo(10 + orderNum); + } + assertThat(!base.isEmpty() || !joined.isEmpty()) + .as("At least one of base or joined must be present").isTrue(); + if (!base.isEmpty() && !joined.isEmpty()) { + assertThat(base.get("customerId")).isEqualTo(joined.get("customerId")); + bothPresent++; + } + } + assertThat(bothPresent).as("Most rows should have both base and joined").isGreaterThan(0); + } + + /** + * Filter base table with region=EU; ALLOW_SCAN with limit. Expected response: Only rows with region=EU; customerId and name + * present; name starts with "Customer". DynamoDB operation: scan() + */ + @Test + public void withCondition_returnsFilteredRows() { + QueryExpressionSpec spec = QueryExpressionBuilder.from(customersTable) + .where(Condition.eq("region", "EU")) + .executionMode(ExecutionMode.ALLOW_SCAN) + .project("customerId", "name", "region", "orderId", "amount") + .limit(600) + .build(); + List rows = runAndMeasure(spec); + assertThat(rows).hasSize(EXPECTED_EU_CUSTOMER_COUNT); + for (EnhancedQueryRow row : rows) { + Map base = row.getItem("base"); + assertThat(base).containsOnlyKeys("customerId", "name", "region"); + assertThat(base.get("region")).isEqualTo("EU"); + String customerId = (String) base.get("customerId"); + int num = Integer.parseInt(customerId.substring(1)); + assertThat(num % 2).as("EU customer should have odd ID: " + customerId).isEqualTo(1); + assertThat((String) base.get("name")).isEqualTo("Customer" + num); + } + } + + /** + * Tree condition: (region=EU AND name begins with "C") OR region=NA; ALLOW_SCAN. Expected response: Non-empty rows; each base + * has customerId, region (EU or NA). DynamoDB operation: scan() + */ + @Test + public void treeCondition_returnsMatchingRows() { + Condition tree = Condition.group( + Condition.eq("region", "EU").and(Condition.beginsWith("name", "C")) + ).or(Condition.eq("region", "NA")); + QueryExpressionSpec spec = QueryExpressionBuilder.from(customersTable) + .where(tree) + .executionMode(ExecutionMode.ALLOW_SCAN) + .project("customerId", "name", "region", "orderId", "amount") + .limit(1000) + .build(); + List rows = runAndMeasure(spec); + assertThat(rows).hasSize(SEED_CUSTOMER_COUNT); + for (EnhancedQueryRow row : rows) { + Map base = row.getItem("base"); + assertThat(base).containsOnlyKeys("customerId", "region", "name"); + String region = (String) base.get("region"); + String name = (String) base.get("name"); + boolean matchesCondition = + ("EU".equals(region) && name.startsWith("C")) || "NA".equals(region); + assertThat(matchesCondition) + .as("Row should match tree condition: region=%s, name=%s", region, name) + .isTrue(); + } + } + + /** + * Base-only scan with limit 5; no join. Expected response: Exactly 5 rows; each has base with customerId (cN), name, region. + * DynamoDB operation: scan() + */ + @Test + public void limit_enforced() { + QueryExpressionSpec spec = QueryExpressionBuilder.from(customersTable) + .executionMode(ExecutionMode.ALLOW_SCAN) + .project("customerId", "name", "region", "orderId", "amount") + .limit(5) + .build(); + List rows = runAndMeasure(spec); + assertThat(rows).hasSize(5); + for (EnhancedQueryRow row : rows) { + Map base = row.getItem("base"); + assertThat(base).containsOnlyKeys("customerId", "name", "region"); + assertThat((String) base.get("customerId")).matches("c\\d+"); + } + } + + /** + * STRICT_KEY_ONLY with base key condition for c1. Expected response: One row; base contains customerId c1, name Customer1, + * region EU. DynamoDB operation: query() + */ + @Test + public void executionMode_strictKeyOnly_withKey() { + QueryExpressionSpec spec = QueryExpressionBuilder.from(customersTable) + .executionMode(ExecutionMode.STRICT_KEY_ONLY) + .keyCondition(QueryConditional.keyEqualTo(k -> k.partitionValue( + "c1"))) + .project("customerId", "name", "region", "orderId", "amount") + .limit(10) + .build(); + List rows = runAndMeasure(spec); + assertThat(rows).hasSize(1); + Map base = rows.get(0).getItem("base"); + assertThat(base).containsOnly( + entry("customerId", "c1"), + entry("name", "Customer1"), + entry("region", "EU")); + } + + /** + * STRICT_KEY_ONLY without base key condition: no scan allowed. Expected response: Empty list of rows. DynamoDB operation: + * none (returns empty) + */ + @Test + public void executionMode_strictKeyOnly_withoutKey_returnsEmptyOrNoScan() { + QueryExpressionSpec spec = QueryExpressionBuilder.from(customersTable) + .executionMode(ExecutionMode.STRICT_KEY_ONLY) + .project("customerId", "name", "region", "orderId", "amount") + .limit(10) + .build(); + List rows = runAndMeasure(spec); + assertThat(rows).isEmpty(); + } + + /** + * ALLOW_SCAN with limit 100: full table scan. Expected response: up to limit rows; each has base customerId (cN), name, + * region (EU or NA). DynamoDB operation: scan() + */ + @Test + public void executionMode_allowScan() { + QueryExpressionSpec spec = QueryExpressionBuilder.from(customersTable) + .executionMode(ExecutionMode.ALLOW_SCAN) + .project("customerId", "name", "region", "orderId", "amount") + .limit(100) + .build(); + List rows = runAndMeasure(spec); + assertThat(rows).hasSize(100); + for (EnhancedQueryRow row : rows) { + Map base = row.getItem("base"); + assertThat(base).containsOnlyKeys("customerId", "name", "region"); + String customerId = (String) base.get("customerId"); + assertThat(customerId).matches("c\\d+"); + int num = Integer.parseInt(customerId.substring(1)); + assertThat((String) base.get("name")).isEqualTo("Customer" + num); + String region = (String) base.get("region"); + assertThat(region).isIn("EU", "NA"); + if (num % 2 == 1) { + assertThat(region).isEqualTo("EU"); + } else { + assertThat(region).isEqualTo("NA"); + } + } + } + + /** + * INNER join with joined-table condition amount >= 50; base key c1. Expected response: 50 rows; every joined row has amount + * >= 50. DynamoDB operation: base=query(), join=query() + */ + @Test + public void joinInner_withJoinedTableCondition_returnsFilteredJoinedRows() { + QueryExpressionSpec spec = QueryExpressionBuilder.from(customersTable) + .join(ordersTable, JoinType.INNER, "customerId", "customerId") + .keyCondition(QueryConditional.keyEqualTo(k -> k.partitionValue("c1"))) + .filterJoined(Condition.gte("amount", 50)) + .project("customerId", "name", "region", "orderId", "amount") + .limit(100) + .build(); + List rows = runAndMeasure(spec); + assertThat(rows).hasSize(100); + Set orderIds = new HashSet<>(); + for (EnhancedQueryRow row : rows) { + Map base = row.getItem("base"); + Map joined = row.getItem("joined"); + assertThat(base).containsOnly( + entry("customerId", "c1"), + entry("name", "Customer1"), + entry("region", "EU")); + assertThat(joined).containsOnlyKeys("customerId", "orderId", "amount"); + assertThat(joined.get("customerId")).isEqualTo("c1"); + int amount = ((Number) joined.get("amount")).intValue(); + assertThat(amount).isGreaterThanOrEqualTo(50); + assertThat(amount).isLessThanOrEqualTo(EXPECTED_MAX_AMOUNT_C1); + String orderId = (String) joined.get("orderId"); + assertThat(orderIds.add(orderId)).as("Order ID should be unique: " + orderId).isTrue(); + int orderNum = Integer.parseInt(orderId.substring(orderId.lastIndexOf('o') + 1)); + assertThat(amount).isEqualTo(10 + orderNum); + } + } + + /** + * LEFT join with joined-table condition amount >= 50; base key c1. Expected response: up to limit rows; all joined rows + * satisfy amount >= 50. DynamoDB operation: base=query(), join=query() + */ + @Test + public void joinLeft_withJoinedTableCondition_returnsFilteredJoinedRows() { + QueryExpressionSpec spec = QueryExpressionBuilder.from(customersTable) + .join(ordersTable, JoinType.LEFT, "customerId", "customerId") + .keyCondition(QueryConditional.keyEqualTo(k -> k.partitionValue("c1"))) + .filterJoined(Condition.gte("amount", 50)) + .project("customerId", "name", "region", "orderId", "amount") + .limit(100) + .build(); + List rows = runAndMeasure(spec); + assertThat(rows).hasSize(100); + for (EnhancedQueryRow row : rows) { + Map joined = row.getItem("joined"); + assertThat(((Number) joined.get("amount")).intValue()).isGreaterThanOrEqualTo(50); + } + } + + /** + * RIGHT join with joined-table condition amount >= 50; base key c1. Expected response: 100 rows; every row has joined data + * with amount >= 50; base may be empty (right-only) or match joined customerId. DynamoDB operation: base=query(), + * join=query() + */ + @Test + public void joinRight_withJoinedTableCondition_returnsFilteredJoinedRows() { + QueryExpressionSpec spec = QueryExpressionBuilder.from(customersTable) + .join(ordersTable, JoinType.RIGHT, "customerId", "customerId") + .keyCondition(QueryConditional.keyEqualTo(k -> k.partitionValue("c1"))) + .filterJoined(Condition.gte("amount", 50)) + .project("customerId", "name", "region", "orderId", "amount") + .limit(100) + .build(); + List rows = runAndMeasure(spec); + assertThat(rows).hasSize(100); + for (EnhancedQueryRow row : rows) { + Map joined = row.getItem("joined"); + Map base = row.getItem("base"); + assertThat(joined).containsOnlyKeys("customerId", "orderId", "amount"); + assertThat((Number) joined.get("amount")).isNotNull(); + assertThat(((Number) joined.get("amount")).intValue()).isGreaterThanOrEqualTo(50); + assertThat((String) joined.get("customerId")).matches("c\\d+"); + if (!base.isEmpty()) { + assertThat(base).containsOnlyKeys("customerId", "name", "region"); + assertThat(base.get("customerId")).isEqualTo(joined.get("customerId")); + } + } + } + + /** + * FULL join with joined-table condition amount >= 50; base key c1. Expected response: 150 rows; base may be empty; every row + * with joined has amount >= 50; when base present it is c1. DynamoDB operation: base=query(), join=query() + */ + @Test + public void joinFull_withJoinedTableCondition_returnsFilteredJoinedRows() { + QueryExpressionSpec spec = QueryExpressionBuilder.from(customersTable) + .join(ordersTable, JoinType.FULL, "customerId", "customerId") + .keyCondition(QueryConditional.keyEqualTo(k -> k.partitionValue("c1"))) + .filterJoined(Condition.gte("amount", 50)) + .project("customerId", "name", "region", "orderId", "amount") + .limit(150) + .build(); + List rows = runAndMeasure(spec); + assertThat(rows).hasSize(150); + for (EnhancedQueryRow row : rows) { + Map base = row.getItem("base"); + Map joined = row.getItem("joined"); + if (!base.isEmpty()) { + assertThat(base).containsOnly( + entry("customerId", "c1"), + entry("name", "Customer1"), + entry("region", "EU")); + } + if (!joined.isEmpty()) { + assertThat(joined).containsOnlyKeys("customerId", "orderId", "amount"); + assertThat(((Number) joined.get("amount")).intValue()).isGreaterThanOrEqualTo(50); + assertThat((String) joined.get("customerId")).matches("c\\d+"); + } + } + } + + /** + * Exercises all applicable join builders: from, join, baseKeyCondition, withBaseTableCondition, withJoinedTableCondition, + * executionMode, limit. Expected response: up to limit rows (c1, EU, amount >= 50). DynamoDB operation: base=query(), + * join=query() + */ + @Test + public void allBuilders_joinSpec_returnsExpectedRows() { + QueryExpressionSpec spec = QueryExpressionBuilder.from(customersTable) + .join(ordersTable, JoinType.INNER, "customerId", "customerId") + .keyCondition(QueryConditional.keyEqualTo(k -> k.partitionValue("c1"))) + .filterBase(Condition.eq("region", "EU")) + .filterJoined(Condition.gte("amount", 50)) + .executionMode(ExecutionMode.STRICT_KEY_ONLY) + .project("customerId", "name", "region", "orderId", "amount") + .limit(100) + .build(); + List rows = runAndMeasure(spec); + assertThat(rows).hasSize(100); + Set orderIds = new HashSet<>(); + for (EnhancedQueryRow row : rows) { + Map base = row.getItem("base"); + Map joined = row.getItem("joined"); + assertThat(base).containsOnly( + entry("customerId", "c1"), + entry("name", "Customer1"), + entry("region", "EU")); + assertThat(joined).containsOnlyKeys("customerId", "orderId", "amount"); + assertThat(joined.get("customerId")).isEqualTo("c1"); + int amount = ((Number) joined.get("amount")).intValue(); + assertThat(amount).isGreaterThanOrEqualTo(EXPECTED_MIN_AMOUNT_C1_GE_50); + assertThat(amount).isLessThanOrEqualTo(EXPECTED_MAX_AMOUNT_C1_GE_50); + String orderId = (String) joined.get("orderId"); + assertThat(orderIds.add(orderId)).as("Order ID should be unique: " + orderId).isTrue(); + int orderNum = Integer.parseInt(orderId.substring(orderId.lastIndexOf('o') + 1)); + assertThat(amount).isEqualTo(10 + orderNum); + } + assertThat(orderIds).hasSize(100); + } + + /** + * Base key condition for non-existent customer: no rows match. Expected response: Empty list of rows, no error. DynamoDB + * operation: query() + */ + @Test + public void emptyResult_returnsEmptyList() { + QueryExpressionSpec spec = QueryExpressionBuilder.from(customersTable) + .keyCondition(QueryConditional.keyEqualTo(k -> k.partitionValue( + "c99999"))) + .limit(10) + .build(); + List rows = runAndMeasure(spec); + assertThat(rows).isEmpty(); + } +} diff --git a/services-custom/dynamodb-enhanced/src/test/java/software/amazon/awssdk/enhanced/dynamodb/functionaltests/query/EnhancedQueryJoinSyncTest.java b/services-custom/dynamodb-enhanced/src/test/java/software/amazon/awssdk/enhanced/dynamodb/functionaltests/query/EnhancedQueryJoinSyncTest.java new file mode 100644 index 000000000000..7d22fba4afe5 --- /dev/null +++ b/services-custom/dynamodb-enhanced/src/test/java/software/amazon/awssdk/enhanced/dynamodb/functionaltests/query/EnhancedQueryJoinSyncTest.java @@ -0,0 +1,722 @@ +/* + * Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"). + * You may not use this file except in compliance with the License. + * A copy of the License is located at + * + * http://aws.amazon.com/apache2.0 + * + * or in the "license" file accompanying this file. This file is distributed + * on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either + * express or implied. See the License for the specific language governing + * permissions and limitations under the License. + */ + +package software.amazon.awssdk.enhanced.dynamodb.functionaltests.query; + +import static org.assertj.core.api.Assertions.assertThat; +import static org.assertj.core.api.Assertions.entry; +import static software.amazon.awssdk.enhanced.dynamodb.mapper.StaticAttributeTags.primaryPartitionKey; +import static software.amazon.awssdk.enhanced.dynamodb.mapper.StaticAttributeTags.primarySortKey; + +import java.util.ArrayList; +import java.util.HashSet; +import java.util.List; +import java.util.Map; +import java.util.Objects; +import java.util.Set; +import org.junit.Before; +import org.junit.Test; +import software.amazon.awssdk.enhanced.dynamodb.DynamoDbEnhancedClient; +import software.amazon.awssdk.enhanced.dynamodb.DynamoDbTable; +import software.amazon.awssdk.enhanced.dynamodb.TableSchema; +import software.amazon.awssdk.enhanced.dynamodb.functionaltests.LargeDatasetInitializer; +import software.amazon.awssdk.enhanced.dynamodb.functionaltests.LocalDynamoDbLargeDatasetTestBase; +import software.amazon.awssdk.enhanced.dynamodb.mapper.StaticTableSchema; +import software.amazon.awssdk.enhanced.dynamodb.model.QueryConditional; +import software.amazon.awssdk.enhanced.dynamodb.query.condition.Condition; +import software.amazon.awssdk.enhanced.dynamodb.query.result.EnhancedQueryLatencyReport; +import software.amazon.awssdk.enhanced.dynamodb.query.result.EnhancedQueryResult; +import software.amazon.awssdk.enhanced.dynamodb.query.result.EnhancedQueryRow; +import software.amazon.awssdk.enhanced.dynamodb.query.engine.QueryExpressionBuilder; +import software.amazon.awssdk.enhanced.dynamodb.query.spec.QueryExpressionSpec; + +/** + * Join-focused integration tests for the sync enhanced query API. Assumes the large dataset (1000 customers × 1000 orders) has + * been seeded once via {@link LargeDatasetInitializer#main(String[])}. Does not create or delete tables; only builds client and + * mapped tables in {@code @Before}. Each test measures query execution time (consuming all rows), prints duration in ms, and + * fails if > 1 second. + */ +public class EnhancedQueryJoinSyncTest extends LocalDynamoDbLargeDatasetTestBase { + + private static final long MAX_QUERY_MS = 1_000L; + + private static class CustomerRecord { + private String customerId; + private String name; + private String region; + + public String getCustomerId() { + return customerId; + } + + public void setCustomerId(String customerId) { + this.customerId = customerId; + } + + public String getName() { + return name; + } + + public void setName(String name) { + this.name = name; + } + + public String getRegion() { + return region; + } + + public void setRegion(String region) { + this.region = region; + } + + @Override + public boolean equals(Object o) { + if (this == o) { + return true; + } + if (o == null || getClass() != o.getClass()) { + return false; + } + CustomerRecord that = (CustomerRecord) o; + return Objects.equals(customerId, that.customerId) && Objects.equals(name, that.name) && Objects.equals(region, + that.region); + } + + @Override + public int hashCode() { + return Objects.hash(customerId, name, region); + } + } + + private static class OrderRecord { + private String customerId; + private String orderId; + private Integer amount; + + public String getCustomerId() { + return customerId; + } + + public void setCustomerId(String customerId) { + this.customerId = customerId; + } + + public String getOrderId() { + return orderId; + } + + public void setOrderId(String orderId) { + this.orderId = orderId; + } + + public Integer getAmount() { + return amount; + } + + public void setAmount(Integer amount) { + this.amount = amount; + } + + @Override + public boolean equals(Object o) { + if (this == o) { + return true; + } + if (o == null || getClass() != o.getClass()) { + return false; + } + OrderRecord that = (OrderRecord) o; + return Objects.equals(customerId, that.customerId) && Objects.equals(orderId, that.orderId) && Objects.equals(amount, that.amount); + } + + @Override + public int hashCode() { + return Objects.hash(customerId, orderId, amount); + } + } + + private static final TableSchema CUSTOMER_SCHEMA = + StaticTableSchema.builder(CustomerRecord.class) + .newItemSupplier(CustomerRecord::new) + .addAttribute(String.class, + a -> a.name("customerId").getter(CustomerRecord::getCustomerId).setter(CustomerRecord::setCustomerId).tags(primaryPartitionKey())) + .addAttribute(String.class, + a -> a.name("name").getter(CustomerRecord::getName).setter(CustomerRecord::setName)) + .addAttribute(String.class, + a -> a.name("region").getter(CustomerRecord::getRegion).setter(CustomerRecord::setRegion)) + .build(); + + private static final TableSchema ORDER_SCHEMA = + StaticTableSchema.builder(OrderRecord.class) + .newItemSupplier(OrderRecord::new) + .addAttribute(String.class, + a -> a.name("customerId").getter(OrderRecord::getCustomerId).setter(OrderRecord::setCustomerId).tags(primaryPartitionKey())) + .addAttribute(String.class, + a -> a.name("orderId").getter(OrderRecord::getOrderId).setter(OrderRecord::setOrderId).tags(primarySortKey())) + .addAttribute(Integer.class, + a -> a.name("amount").getter(OrderRecord::getAmount).setter(OrderRecord::setAmount)) + .build(); + + private DynamoDbEnhancedClient enhancedClient; + private DynamoDbTable customersTable; + private DynamoDbTable ordersTable; + + @Override + protected String getConcreteTableName(String logicalTableName) { + if ("customers".equals(logicalTableName)) { + return LargeDatasetInitializer.LARGE_CUSTOMERS_TABLE; + } + if ("orders".equals(logicalTableName)) { + return LargeDatasetInitializer.LARGE_ORDERS_TABLE; + } + return super.getConcreteTableName(logicalTableName); + } + + @Before + public void setUp() { + enhancedClient = DynamoDbEnhancedClient.builder().dynamoDbClient(getDynamoDbClient()).build(); + customersTable = enhancedClient.table(getConcreteTableName("customers"), CUSTOMER_SCHEMA); + ordersTable = enhancedClient.table(getConcreteTableName("orders"), ORDER_SCHEMA); + } + + private String currentTestName() { + String thisClass = getClass().getName(); + for (StackTraceElement element : new Throwable().getStackTrace()) { + if (thisClass.equals(element.getClassName())) { + String method = element.getMethodName(); + if (!"currentTestName".equals(method) + && !"runAndMeasure".equals(method) + && !method.startsWith("lambda$") + && !method.startsWith("invoke")) { + return method; + } + } + } + return "unknownTest"; + } + + private List runAndMeasure(QueryExpressionSpec spec) { + String testName = currentTestName(); + String label = "EnhancedQueryJoinSyncTest." + testName; + EnhancedQueryLatencyReport[] reportHolder = new EnhancedQueryLatencyReport[1]; + EnhancedQueryResult result = enhancedClient.enhancedQuery(spec, r -> reportHolder[0] = r); + List rows = new ArrayList<>(); + result.forEach(rows::add); + EnhancedQueryLatencyReport report = reportHolder[0]; + long elapsedMs = report != null ? report.totalMs() : 0L; + if (report != null) { + System.out.println(label + + " EnhancedQueryLatencyReport: baseQueryMs=" + report.baseQueryMs() + + " joinedLookupsMs=" + report.joinedLookupsMs() + + " inMemoryProcessingMs=" + report.inMemoryProcessingMs() + + " totalMs=" + report.totalMs() + " rows=" + rows.size()); + } else { + System.out.println(label + " query took " + + elapsedMs + " ms, rows=" + rows.size()); + } + writeQueryMetric(label, elapsedMs, rows.size()); + assertThat(elapsedMs).isLessThanOrEqualTo(MAX_QUERY_MS); + return rows; + } + + /** + * Base-only: query customers with key condition and limit. Expected response: Single row for c1 with customerId, name, + * region; name=Customer1, region=EU. DynamoDB operation: query() + */ + @Test + public void baseOnly_withKeyConditionAndLimit() { + QueryExpressionSpec spec = QueryExpressionBuilder.from(customersTable) + .keyCondition(QueryConditional.keyEqualTo(k -> k.partitionValue( + "c1"))) + .project("customerId", "name", "region", "orderId", "amount") + .limit(10) + .build(); + List rows = runAndMeasure(spec); + assertThat(rows).hasSize(1); + Map base = rows.get(0).getItem("base"); + assertThat(base).containsOnly( + entry("customerId", "c1"), + entry("name", "Customer1"), + entry("region", "EU")); + } + + /** + * INNER join on customerId with key condition for c1. Expected response: One row per order of c1; each row has base customer + * and joined order with customerId, orderId, amount; first order c1-o1 with amount 11; row count = SEED_ORDERS_PER_CUSTOMER. + * DynamoDB operation: base=query(), join=query() + */ + @Test + public void joinInner_returnsMatchingPairs() { + QueryExpressionSpec spec = QueryExpressionBuilder.from(customersTable) + .join(ordersTable, JoinType.INNER, "customerId", "customerId") + .keyCondition(QueryConditional.keyEqualTo(k -> k.partitionValue( + "c1"))) + .project("customerId", "name", "region", "orderId", "amount") + .limit(SEED_ORDERS_PER_CUSTOMER + 100) + .build(); + List rows = runAndMeasure(spec); + assertThat(rows).hasSize(SEED_ORDERS_PER_CUSTOMER); + Set orderIds = new HashSet<>(); + for (EnhancedQueryRow row : rows) { + Map base = row.getItem("base"); + Map joined = row.getItem("joined"); + assertThat(base).containsOnly( + entry("customerId", "c1"), + entry("name", "Customer1"), + entry("region", "EU")); + assertThat(joined).containsOnlyKeys("customerId", "orderId", "amount"); + assertThat(joined.get("customerId")).isEqualTo("c1"); + String orderId = (String) joined.get("orderId"); + assertThat(orderId).matches("c1-o\\d+"); + assertThat(orderIds.add(orderId)).as("Order ID should be unique: " + orderId).isTrue(); + int orderNum = Integer.parseInt(orderId.substring(orderId.lastIndexOf('o') + 1)); + int amount = ((Number) joined.get("amount")).intValue(); + assertThat(amount).as("amount for %s should be 10+%d", orderId, orderNum).isEqualTo(10 + orderNum); + } + assertThat(orderIds).hasSize(SEED_ORDERS_PER_CUSTOMER); + assertThat(rows.get(0).getItem("joined").get("orderId")).isEqualTo("c1-o1"); + assertThat(((Number) rows.get(0).getItem("joined").get("amount")).intValue()).isEqualTo(11); + } + + /** + * LEFT join on customerId for c1: every base row appears with optional joined row (limit 50). Expected response: 50 rows + * (limit); each has base (c1) and joined (c1, orderId, amount). DynamoDB operation: base=query(), join=query() + */ + @Test + public void joinLeft_returnsBaseRowsWithOptionalJoined() { + QueryExpressionSpec spec = QueryExpressionBuilder.from(customersTable) + .join(ordersTable, JoinType.LEFT, "customerId", "customerId") + .keyCondition(QueryConditional.keyEqualTo(k -> k.partitionValue( + "c1"))) + .project("customerId", "name", "region", "orderId", "amount") + .limit(50) + .build(); + List rows = runAndMeasure(spec); + assertThat(rows).hasSize(50); + Set orderIds = new HashSet<>(); + for (EnhancedQueryRow row : rows) { + Map base = row.getItem("base"); + Map joined = row.getItem("joined"); + assertThat(base).containsOnly( + entry("customerId", "c1"), + entry("name", "Customer1"), + entry("region", "EU")); + assertThat(joined).containsOnlyKeys("customerId", "orderId", "amount"); + assertThat(joined.get("customerId")).isEqualTo("c1"); + String orderId = (String) joined.get("orderId"); + assertThat(orderIds.add(orderId)).as("Order ID should be unique: " + orderId).isTrue(); + int orderNum = Integer.parseInt(orderId.substring(orderId.lastIndexOf('o') + 1)); + int amount = ((Number) joined.get("amount")).intValue(); + assertThat(amount).isEqualTo(10 + orderNum); + } + assertThat(orderIds).hasSize(50); + } + + /** + * RIGHT join on customerId for c1: every joined row appears with optional base. Expected response: SEED_ORDERS_PER_CUSTOMER + * rows; each has joined (c1, orderId, amount) and base (c1). DynamoDB operation: base=query(), join=query() + */ + @Test + public void joinRight_returnsJoinedRowsWithOptionalBase() { + QueryExpressionSpec spec = QueryExpressionBuilder.from(customersTable) + .join(ordersTable, JoinType.RIGHT, "customerId", "customerId") + .keyCondition(QueryConditional.keyEqualTo(k -> k.partitionValue( + "c1"))) + .project("customerId", "name", "region", "orderId", "amount") + .limit(SEED_ORDERS_PER_CUSTOMER + 100) + .build(); + List rows = runAndMeasure(spec); + assertThat(rows).hasSize(SEED_ORDERS_PER_CUSTOMER); + Set orderIds = new HashSet<>(); + for (EnhancedQueryRow row : rows) { + Map base = row.getItem("base"); + Map joined = row.getItem("joined"); + assertThat(joined).containsOnlyKeys("customerId", "orderId", "amount"); + assertThat(joined.get("customerId")).isEqualTo("c1"); + String orderId = (String) joined.get("orderId"); + assertThat(orderIds.add(orderId)).as("Order ID should be unique: " + orderId).isTrue(); + int orderNum = Integer.parseInt(orderId.substring(orderId.lastIndexOf('o') + 1)); + int amount = ((Number) joined.get("amount")).intValue(); + assertThat(amount).isEqualTo(10 + orderNum); + assertThat(base).containsOnly( + entry("customerId", "c1"), + entry("name", "Customer1"), + entry("region", "EU")); + } + assertThat(orderIds).hasSize(SEED_ORDERS_PER_CUSTOMER); + } + + /** + * FULL join on customerId for c1: union of LEFT and RIGHT (limit 150). Expected response: 150 rows; rows may have empty base + * (right-only) or empty joined (left-only); when present, base is c1. DynamoDB operation: base=query(), join=query() + */ + @Test + public void joinFull_returnsUnionOfLeftAndRight() { + QueryExpressionSpec spec = QueryExpressionBuilder.from(customersTable) + .join(ordersTable, JoinType.FULL, "customerId", "customerId") + .keyCondition(QueryConditional.keyEqualTo(k -> k.partitionValue( + "c1"))) + .project("customerId", "name", "region", "orderId", "amount") + .limit(150) + .build(); + List rows = runAndMeasure(spec); + assertThat(rows).hasSize(150); + int bothPresent = 0; + for (EnhancedQueryRow row : rows) { + Map base = row.getItem("base"); + Map joined = row.getItem("joined"); + if (!base.isEmpty()) { + assertThat(base).containsOnlyKeys("customerId", "name", "region"); + assertThat(base.get("customerId")).isEqualTo("c1"); + assertThat(base.get("name")).isEqualTo("Customer1"); + assertThat(base.get("region")).isEqualTo("EU"); + } + if (!joined.isEmpty()) { + assertThat(joined).containsOnlyKeys("customerId", "orderId", "amount"); + assertThat(joined.get("customerId")).isEqualTo("c1"); + int orderNum = Integer.parseInt(((String) joined.get("orderId")).substring( + ((String) joined.get("orderId")).lastIndexOf('o') + 1)); + assertThat(((Number) joined.get("amount")).intValue()).isEqualTo(10 + orderNum); + } + assertThat(!base.isEmpty() || !joined.isEmpty()) + .as("At least one of base or joined must be present").isTrue(); + if (!base.isEmpty() && !joined.isEmpty()) { + assertThat(base.get("customerId")).isEqualTo(joined.get("customerId")); + bothPresent++; + } + } + assertThat(bothPresent).as("Most rows should have both base and joined").isGreaterThan(0); + } + + /** + * Filter base table with region=EU; ALLOW_SCAN with limit. Expected response: Only rows with region=EU; customerId and name + * present; name starts with "Customer". DynamoDB operation: scan() + */ + @Test + public void withCondition_returnsFilteredRows() { + QueryExpressionSpec spec = QueryExpressionBuilder.from(customersTable) + .where(Condition.eq("region", "EU")) + .executionMode(ExecutionMode.ALLOW_SCAN) + .project("customerId", "name", "region", "orderId", "amount") + .limit(600) + .build(); + List rows = runAndMeasure(spec); + assertThat(rows).hasSize(EXPECTED_EU_CUSTOMER_COUNT); + for (EnhancedQueryRow row : rows) { + Map base = row.getItem("base"); + assertThat(base).containsOnlyKeys("customerId", "name", "region"); + assertThat(base.get("region")).isEqualTo("EU"); + String customerId = (String) base.get("customerId"); + int num = Integer.parseInt(customerId.substring(1)); + assertThat(num % 2).as("EU customer should have odd ID: " + customerId).isEqualTo(1); + assertThat((String) base.get("name")).isEqualTo("Customer" + num); + } + } + + /** + * Tree condition: (region=EU AND name begins with "C") OR region=NA; ALLOW_SCAN. Expected response: Non-empty rows; each base + * has customerId, region, name; region is EU or NA. DynamoDB operation: scan() + */ + @Test + public void treeCondition_returnsMatchingRows() { + Condition tree = Condition.group( + Condition.eq("region", "EU").and(Condition.beginsWith("name", "C")) + ).or( + Condition.eq("region", "NA")); + + QueryExpressionSpec spec = QueryExpressionBuilder.from(customersTable) + .where(tree) + .executionMode(ExecutionMode.ALLOW_SCAN) + .project("customerId", "name", "region", "orderId", "amount") + .limit(1000) + .build(); + List rows = runAndMeasure(spec); + assertThat(rows).hasSize(SEED_CUSTOMER_COUNT); + for (EnhancedQueryRow row : rows) { + Map base = row.getItem("base"); + assertThat(base).containsOnlyKeys("customerId", "region", "name"); + String region = (String) base.get("region"); + String name = (String) base.get("name"); + boolean matchesCondition = + ("EU".equals(region) && name.startsWith("C")) || "NA".equals(region); + assertThat(matchesCondition) + .as("Row should match tree condition: region=%s, name=%s", region, name) + .isTrue(); + } + } + + /** + * Base-only scan with limit 5; no join. Expected response: Exactly 5 rows; each has base with customerId (cN), name, region. + * DynamoDB operation: scan() + */ + @Test + public void limit_enforced() { + QueryExpressionSpec spec = QueryExpressionBuilder.from(customersTable) + .executionMode(ExecutionMode.ALLOW_SCAN) + .project("customerId", "name", "region", "orderId", "amount") + .limit(5) + .build(); + List rows = runAndMeasure(spec); + assertThat(rows).hasSize(5); + for (EnhancedQueryRow row : rows) { + Map base = row.getItem("base"); + assertThat(base).containsOnlyKeys("customerId", "name", "region"); + assertThat((String) base.get("customerId")).matches("c\\d+"); + assertThat((String) base.get("name")).startsWith("Customer"); + assertThat(base.get("region")).isIn("EU", "NA"); + } + } + + /** + * STRICT_KEY_ONLY with base key condition for c1. Expected response: One row; base contains customerId c1. DynamoDB + * operation: query() + */ + @Test + public void executionMode_strictKeyOnly_withKey() { + QueryExpressionSpec spec = QueryExpressionBuilder.from(customersTable) + .executionMode(ExecutionMode.STRICT_KEY_ONLY) + .keyCondition(QueryConditional.keyEqualTo(k -> k.partitionValue( + "c1"))) + .project("customerId", "name", "region", "orderId", "amount") + .limit(10) + .build(); + List rows = runAndMeasure(spec); + assertThat(rows).hasSize(1); + assertThat(rows.get(0).getItem("base")).containsOnly( + entry("customerId", "c1"), + entry("name", "Customer1"), + entry("region", "EU")); + } + + /** + * STRICT_KEY_ONLY without base key condition: no scan allowed. Expected response: Empty list of rows. DynamoDB operation: + * none (returns empty) + */ + @Test + public void executionMode_strictKeyOnly_withoutKey_returnsEmptyOrNoScan() { + QueryExpressionSpec spec = QueryExpressionBuilder.from(customersTable) + .executionMode(ExecutionMode.STRICT_KEY_ONLY) + .project("customerId", "name", "region", "orderId", "amount") + .limit(10) + .build(); + List rows = runAndMeasure(spec); + assertThat(rows).isEmpty(); + } + + /** + * ALLOW_SCAN with limit 100: full table scan. Expected response: up to limit rows; each has base customerId (cN), name, + * region (EU or NA). DynamoDB operation: scan() + */ + @Test + public void executionMode_allowScan() { + QueryExpressionSpec spec = QueryExpressionBuilder.from(customersTable) + .executionMode(ExecutionMode.ALLOW_SCAN) + .project("customerId", "name", "region", "orderId", "amount") + .limit(100) + .build(); + List rows = runAndMeasure(spec); + assertThat(rows).hasSize(100); + for (EnhancedQueryRow row : rows) { + Map base = row.getItem("base"); + assertThat(base).containsOnlyKeys("customerId", "name", "region"); + String customerId = (String) base.get("customerId"); + assertThat(customerId).matches("c\\d+"); + int num = Integer.parseInt(customerId.substring(1)); + assertThat((String) base.get("name")).isEqualTo("Customer" + num); + String region = (String) base.get("region"); + assertThat(region).isIn("EU", "NA"); + if (num % 2 == 1) { + assertThat(region).isEqualTo("EU"); + } else { + assertThat(region).isEqualTo("NA"); + } + } + } + + /** + * INNER join with joined-table condition amount >= 50; base key c1. Expected response: up to limit rows; every joined row has + * amount >= 50. DynamoDB operation: base=query(), join=query() + */ + @Test + public void joinInner_withJoinedTableCondition_returnsFilteredJoinedRows() { + QueryExpressionSpec spec = QueryExpressionBuilder.from(customersTable) + .join(ordersTable, JoinType.INNER, "customerId", "customerId") + .keyCondition(QueryConditional.keyEqualTo(k -> k.partitionValue("c1"))) + .filterJoined(Condition.gte("amount", 50)) + .project("customerId", "name", "region", "orderId", "amount") + .limit(100) + .build(); + List rows = runAndMeasure(spec); + assertThat(rows).hasSize(100); + Set orderIds = new HashSet<>(); + for (EnhancedQueryRow row : rows) { + Map base = row.getItem("base"); + Map joined = row.getItem("joined"); + assertThat(base).containsOnly( + entry("customerId", "c1"), + entry("name", "Customer1"), + entry("region", "EU")); + assertThat(joined).containsOnlyKeys("customerId", "orderId", "amount"); + assertThat(joined.get("customerId")).isEqualTo("c1"); + int amount = ((Number) joined.get("amount")).intValue(); + assertThat(amount).isGreaterThanOrEqualTo(50); + assertThat(amount).isLessThanOrEqualTo(EXPECTED_MAX_AMOUNT_C1); + String orderId = (String) joined.get("orderId"); + assertThat(orderIds.add(orderId)).as("Order ID should be unique: " + orderId).isTrue(); + int orderNum = Integer.parseInt(orderId.substring(orderId.lastIndexOf('o') + 1)); + assertThat(amount).isEqualTo(10 + orderNum); + } + } + + /** + * LEFT join with joined-table condition amount >= 50; base key c1. Expected response: up to limit rows; all joined rows + * satisfy amount >= 50. DynamoDB operation: base=query(), join=query() + */ + @Test + public void joinLeft_withJoinedTableCondition_returnsFilteredJoinedRows() { + QueryExpressionSpec spec = QueryExpressionBuilder.from(customersTable) + .join(ordersTable, JoinType.LEFT, "customerId", "customerId") + .keyCondition(QueryConditional.keyEqualTo(k -> k.partitionValue("c1"))) + .filterJoined(Condition.gte("amount", 50)) + .project("customerId", "name", "region", "orderId", "amount") + .limit(100) + .build(); + List rows = runAndMeasure(spec); + assertThat(rows).hasSize(100); + for (EnhancedQueryRow row : rows) { + Map base = row.getItem("base"); + Map joined = row.getItem("joined"); + assertThat(base).containsOnly( + entry("customerId", "c1"), + entry("name", "Customer1"), + entry("region", "EU")); + assertThat(joined).containsOnlyKeys("customerId", "orderId", "amount"); + assertThat(joined.get("customerId")).isEqualTo("c1"); + assertThat(((Number) joined.get("amount")).intValue()).isGreaterThanOrEqualTo(50); + } + } + + /** + * RIGHT join with joined-table condition amount >= 50; base key c1. Expected response: 100 rows; every row has joined data + * with amount >= 50; base may be empty (right-only) or match joined customerId. DynamoDB operation: base=query(), + * join=query() + */ + @Test + public void joinRight_withJoinedTableCondition_returnsFilteredJoinedRows() { + QueryExpressionSpec spec = QueryExpressionBuilder.from(customersTable) + .join(ordersTable, JoinType.RIGHT, "customerId", "customerId") + .keyCondition(QueryConditional.keyEqualTo(k -> k.partitionValue("c1"))) + .filterJoined(Condition.gte("amount", 50)) + .project("customerId", "name", "region", "orderId", "amount") + .limit(100) + .build(); + List rows = runAndMeasure(spec); + assertThat(rows).hasSize(100); + for (EnhancedQueryRow row : rows) { + Map joined = row.getItem("joined"); + Map base = row.getItem("base"); + assertThat(joined).containsKey("customerId").containsKey("orderId").containsKey("amount"); + assertThat(((Number) joined.get("amount")).intValue()).isGreaterThanOrEqualTo(50); + assertThat(((String) joined.get("customerId"))).matches("c\\d+"); + if (!base.isEmpty()) { + assertThat(base).containsOnlyKeys("customerId", "name", "region"); + assertThat(base.get("customerId")).isEqualTo(joined.get("customerId")); + } + } + } + + /** + * FULL join with joined-table condition amount >= 50; base key c1. Expected response: 150 rows; base may be empty; every row + * with joined has amount >= 50; when base present it is c1. DynamoDB operation: base=query(), join=query() + */ + @Test + public void joinFull_withJoinedTableCondition_returnsFilteredJoinedRows() { + QueryExpressionSpec spec = QueryExpressionBuilder.from(customersTable) + .join(ordersTable, JoinType.FULL, "customerId", "customerId") + .keyCondition(QueryConditional.keyEqualTo(k -> k.partitionValue("c1"))) + .filterJoined(Condition.gte("amount", 50)) + .project("customerId", "name", "region", "orderId", "amount") + .limit(150) + .build(); + List rows = runAndMeasure(spec); + assertThat(rows).hasSize(150); + for (EnhancedQueryRow row : rows) { + Map base = row.getItem("base"); + Map joined = row.getItem("joined"); + if (!base.isEmpty()) { + assertThat(base).containsEntry("customerId", "c1"); + } + if (!joined.isEmpty()) { + assertThat(joined).containsKey("customerId").containsKey("amount"); + assertThat(((Number) joined.get("amount")).intValue()).isGreaterThanOrEqualTo(50); + assertThat(((String) joined.get("customerId"))).matches("c\\d+"); + } + } + } + + /** + * Exercises all applicable join builders: from, join, baseKeyCondition, withBaseTableCondition, withJoinedTableCondition, + * executionMode, limit. Expected response: up to limit rows (c1, EU, amount >= 50). DynamoDB operation: base=query(), + * join=query() + */ + @Test + public void allBuilders_joinSpec_returnsExpectedRows() { + QueryExpressionSpec spec = QueryExpressionBuilder.from(customersTable) + .join(ordersTable, JoinType.INNER, "customerId", "customerId") + .keyCondition(QueryConditional.keyEqualTo(k -> k.partitionValue("c1"))) + .filterBase(Condition.eq("region", "EU")) + .filterJoined(Condition.gte("amount", 50)) + .executionMode(ExecutionMode.STRICT_KEY_ONLY) + .project("customerId", "name", "region", "orderId", "amount") + .limit(100) + .build(); + List rows = runAndMeasure(spec); + assertThat(rows).hasSize(100); + Set orderIds = new HashSet<>(); + for (EnhancedQueryRow row : rows) { + Map base = row.getItem("base"); + Map joined = row.getItem("joined"); + assertThat(base).containsOnly( + entry("customerId", "c1"), + entry("name", "Customer1"), + entry("region", "EU")); + assertThat(joined).containsOnlyKeys("customerId", "orderId", "amount"); + assertThat(joined.get("customerId")).isEqualTo("c1"); + int amount = ((Number) joined.get("amount")).intValue(); + assertThat(amount).isGreaterThanOrEqualTo(EXPECTED_MIN_AMOUNT_C1_GE_50); + assertThat(amount).isLessThanOrEqualTo(EXPECTED_MAX_AMOUNT_C1_GE_50); + String orderId = (String) joined.get("orderId"); + assertThat(orderIds.add(orderId)).as("Order ID should be unique: " + orderId).isTrue(); + int orderNum = Integer.parseInt(orderId.substring(orderId.lastIndexOf('o') + 1)); + assertThat(amount).isEqualTo(10 + orderNum); + } + assertThat(orderIds).hasSize(100); + } + + /** + * Base key condition for non-existent customer: no rows match. Expected response: Empty list of rows, no error. DynamoDB + * operation: query() + */ + @Test + public void emptyResult_returnsEmptyList() { + QueryExpressionSpec spec = QueryExpressionBuilder.from(customersTable) + .keyCondition(QueryConditional.keyEqualTo(k -> k.partitionValue( + "c99999"))) + .limit(10) + .build(); + List rows = runAndMeasure(spec); + assertThat(rows).isEmpty(); + } +} diff --git a/services-custom/dynamodb-enhanced/src/test/java/software/amazon/awssdk/enhanced/dynamodb/internal/operations/CreateTableOperationTest.java b/services-custom/dynamodb-enhanced/src/test/java/software/amazon/awssdk/enhanced/dynamodb/internal/operations/CreateTableOperationTest.java index b058314279c7..ba38623b0417 100644 --- a/services-custom/dynamodb-enhanced/src/test/java/software/amazon/awssdk/enhanced/dynamodb/internal/operations/CreateTableOperationTest.java +++ b/services-custom/dynamodb-enhanced/src/test/java/software/amazon/awssdk/enhanced/dynamodb/internal/operations/CreateTableOperationTest.java @@ -64,6 +64,7 @@ import software.amazon.awssdk.services.dynamodb.model.CreateTableResponse; import software.amazon.awssdk.services.dynamodb.model.GlobalSecondaryIndex; import software.amazon.awssdk.services.dynamodb.model.KeySchemaElement; +import software.amazon.awssdk.services.dynamodb.model.LocalSecondaryIndex; import software.amazon.awssdk.services.dynamodb.model.Projection; import software.amazon.awssdk.services.dynamodb.model.ProjectionType; import software.amazon.awssdk.services.dynamodb.model.ProvisionedThroughput; @@ -80,7 +81,7 @@ public class CreateTableOperationTest { private static final OperationContext GSI_1_CONTEXT = DefaultOperationContext.create(TABLE_NAME, "gsi_1"); - private static MatchedGsi matchesGsi(software.amazon.awssdk.services.dynamodb.model.GlobalSecondaryIndex other) { + private static MatchedGsi matchesGsi(GlobalSecondaryIndex other) { return new MatchedGsi(other); } @@ -88,16 +89,16 @@ private static MatchedGsi matchesGsi(software.amazon.awssdk.services.dynamodb.mo private DynamoDbClient mockDynamoDbClient; private static class MatchedGsi - extends TypeSafeMatcher { + extends TypeSafeMatcher { - private final software.amazon.awssdk.services.dynamodb.model.GlobalSecondaryIndex other; + private final GlobalSecondaryIndex other; - private MatchedGsi(software.amazon.awssdk.services.dynamodb.model.GlobalSecondaryIndex other) { + private MatchedGsi(GlobalSecondaryIndex other) { this.other = other; } @Override - protected boolean matchesSafely(software.amazon.awssdk.services.dynamodb.model.GlobalSecondaryIndex globalSecondaryIndex) { + protected boolean matchesSafely(GlobalSecondaryIndex globalSecondaryIndex) { if (!other.indexName().equals(globalSecondaryIndex.indexName())) { return false; } @@ -107,7 +108,7 @@ protected boolean matchesSafely(software.amazon.awssdk.services.dynamodb.model.G return false; } - return containsInAnyOrder(other.keySchema().toArray(new KeySchemaElement[]{})) + return containsInAnyOrder(other.keySchema().toArray(new KeySchemaElement[] {})) .matches(globalSecondaryIndex.keySchema()); } @@ -136,16 +137,16 @@ public void generateRequest_withLsiAndGsi() { List globalSecondaryIndexList = Arrays.asList( - EnhancedGlobalSecondaryIndex.builder() - .indexName("gsi_1") - .projection(projection1) - .provisionedThroughput(provisionedThroughput1) - .build(), - EnhancedGlobalSecondaryIndex.builder() - .indexName("gsi_2") - .projection(projection2) - .provisionedThroughput(provisionedThroughput2) - .build()); + EnhancedGlobalSecondaryIndex.builder() + .indexName("gsi_1") + .projection(projection1) + .provisionedThroughput(provisionedThroughput1) + .build(), + EnhancedGlobalSecondaryIndex.builder() + .indexName("gsi_2") + .projection(projection2) + .provisionedThroughput(provisionedThroughput2) + .build()); CreateTableOperation operation = CreateTableOperation.create(CreateTableEnhancedRequest.builder() @@ -159,7 +160,6 @@ public void generateRequest_withLsiAndGsi() { null); - assertThat(request.tableName(), is(TABLE_NAME)); assertThat(request.keySchema(), containsInAnyOrder(KeySchemaElement.builder() .attributeName("id") @@ -169,45 +169,45 @@ public void generateRequest_withLsiAndGsi() { .attributeName("sort") .keyType(RANGE) .build())); - software.amazon.awssdk.services.dynamodb.model.GlobalSecondaryIndex expectedGsi1 = - software.amazon.awssdk.services.dynamodb.model.GlobalSecondaryIndex.builder() - .indexName("gsi_1") - .keySchema(KeySchemaElement.builder() - .attributeName("gsi_id") - .keyType(HASH) - .build(), - KeySchemaElement.builder() - .attributeName("gsi_sort") - .keyType(RANGE) - .build()) - .projection(projection1) - .provisionedThroughput(provisionedThroughput1) - .build(); - software.amazon.awssdk.services.dynamodb.model.GlobalSecondaryIndex expectedGsi2 = - software.amazon.awssdk.services.dynamodb.model.GlobalSecondaryIndex.builder() - .indexName("gsi_2") - .keySchema(KeySchemaElement.builder() - .attributeName("gsi_id") - .keyType(HASH) - .build()) - .projection(projection2) - .provisionedThroughput(provisionedThroughput2) - .build(); + GlobalSecondaryIndex expectedGsi1 = + GlobalSecondaryIndex.builder() + .indexName("gsi_1") + .keySchema(KeySchemaElement.builder() + .attributeName("gsi_id") + .keyType(HASH) + .build(), + KeySchemaElement.builder() + .attributeName("gsi_sort") + .keyType(RANGE) + .build()) + .projection(projection1) + .provisionedThroughput(provisionedThroughput1) + .build(); + GlobalSecondaryIndex expectedGsi2 = + GlobalSecondaryIndex.builder() + .indexName("gsi_2") + .keySchema(KeySchemaElement.builder() + .attributeName("gsi_id") + .keyType(HASH) + .build()) + .projection(projection2) + .provisionedThroughput(provisionedThroughput2) + .build(); assertThat(request.globalSecondaryIndexes(), containsInAnyOrder(matchesGsi(expectedGsi1), matchesGsi(expectedGsi2))); - software.amazon.awssdk.services.dynamodb.model.LocalSecondaryIndex expectedLsi = - software.amazon.awssdk.services.dynamodb.model.LocalSecondaryIndex.builder() - .indexName("lsi_1") - .keySchema(KeySchemaElement.builder() - .attributeName("id") - .keyType(HASH) - .build(), - KeySchemaElement.builder() - .attributeName("lsi_sort") - .keyType(RANGE) - .build()) - .projection(projection3) - .build(); + LocalSecondaryIndex expectedLsi = + LocalSecondaryIndex.builder() + .indexName("lsi_1") + .keySchema(KeySchemaElement.builder() + .attributeName("id") + .keyType(HASH) + .build(), + KeySchemaElement.builder() + .attributeName("lsi_sort") + .keyType(RANGE) + .build()) + .projection(projection3) + .build(); assertThat(request.localSecondaryIndexes(), containsInAnyOrder(expectedLsi)); assertThat(request.attributeDefinitions(), containsInAnyOrder( AttributeDefinition.builder() @@ -240,11 +240,11 @@ public void generateRequest_invalidGsi() { .build(); List invalidGsiList = Collections.singletonList( - EnhancedGlobalSecondaryIndex.builder() - .indexName("invalid") - .projection(p -> p.projectionType(ProjectionType.ALL)) - .provisionedThroughput(provisionedThroughput) - .build()); + EnhancedGlobalSecondaryIndex.builder() + .indexName("invalid") + .projection(p -> p.projectionType(ProjectionType.ALL)) + .provisionedThroughput(provisionedThroughput) + .build()); CreateTableOperation operation = CreateTableOperation.create(CreateTableEnhancedRequest.builder().globalSecondaryIndices(invalidGsiList).build()); @@ -266,11 +266,11 @@ public void generateRequest_invalidGsiAsLsiReference() { @Test public void generateRequest_validLsiAsGsiReference() { List validLsiList = Collections.singletonList( - EnhancedGlobalSecondaryIndex.builder() - .indexName("lsi_1") - .projection(p -> p.projectionType(ProjectionType.ALL)) - .provisionedThroughput(p -> p.readCapacityUnits(1L).writeCapacityUnits(1L)) - .build()); + EnhancedGlobalSecondaryIndex.builder() + .indexName("lsi_1") + .projection(p -> p.projectionType(ProjectionType.ALL)) + .provisionedThroughput(p -> p.readCapacityUnits(1L).writeCapacityUnits(1L)) + .build()); CreateTableOperation operation = CreateTableOperation.create(CreateTableEnhancedRequest.builder().globalSecondaryIndices(validLsiList).build()); @@ -278,7 +278,7 @@ public void generateRequest_validLsiAsGsiReference() { CreateTableRequest request = operation.generateRequest(FakeItemWithIndices.getTableSchema(), PRIMARY_CONTEXT, null); assertThat(request.globalSecondaryIndexes().size(), is(1)); - software.amazon.awssdk.services.dynamodb.model.GlobalSecondaryIndex globalSecondaryIndex = + GlobalSecondaryIndex globalSecondaryIndex = request.globalSecondaryIndexes().get(0); assertThat(globalSecondaryIndex.indexName(), is("lsi_1")); @@ -286,7 +286,8 @@ public void generateRequest_validLsiAsGsiReference() { @Test public void generateRequest_nonReferencedIndicesDoNotCreateExtraAttributeDefinitions() { - CreateTableOperation operation = CreateTableOperation.create(CreateTableEnhancedRequest.builder().build()); + CreateTableOperation operation = + CreateTableOperation.create(CreateTableEnhancedRequest.builder().build()); CreateTableRequest request = operation.generateRequest(FakeItemWithIndices.getTableSchema(), PRIMARY_CONTEXT, null); @@ -316,10 +317,10 @@ public void generateRequest_invalidLsi() { @Test public void generateRequest_withProvisionedThroughput() { - ProvisionedThroughput provisionedThroughput = ProvisionedThroughput.builder() - .writeCapacityUnits(1L) - .readCapacityUnits(2L) - .build(); + ProvisionedThroughput provisionedThroughput = ProvisionedThroughput.builder() + .writeCapacityUnits(1L) + .readCapacityUnits(2L) + .build(); CreateTableOperation operation = CreateTableOperation.create( CreateTableEnhancedRequest.builder().provisionedThroughput(provisionedThroughput).build()); @@ -432,27 +433,28 @@ public void generateRequest_withBinaryKey() { @Test public void generateRequest_withByteBufferKey() { - CreateTableOperation operation = CreateTableOperation.create(CreateTableEnhancedRequest.builder() - .build()); + CreateTableOperation operation = + CreateTableOperation.create(CreateTableEnhancedRequest.builder() + .build()); CreateTableRequest request = operation.generateRequest(FakeItemWithByteBufferKey.getTableSchema(), - PRIMARY_CONTEXT, - null); + PRIMARY_CONTEXT, + null); assertThat(request.tableName(), is(TABLE_NAME)); assertThat(request.keySchema(), containsInAnyOrder(KeySchemaElement.builder() - .attributeName("id") - .keyType(HASH) - .build())); + .attributeName("id") + .keyType(HASH) + .build())); assertThat(request.globalSecondaryIndexes(), is(empty())); assertThat(request.localSecondaryIndexes(), is(empty())); assertThat(request.attributeDefinitions(), containsInAnyOrder( - AttributeDefinition.builder() - .attributeName("id") - .attributeType(ScalarAttributeType.B) - .build())); + AttributeDefinition.builder() + .attributeName("id") + .attributeType(ScalarAttributeType.B) + .build())); } @Test(expected = IllegalArgumentException.class) @@ -488,15 +490,15 @@ public void transformResults_doesNothing() { public void generateRequest_gsiWithSingleKeys_buildsCorrectly() { List gsiList = Collections.singletonList( EnhancedGlobalSecondaryIndex.builder() - .indexName("gsi_1") - .projection(p -> p.projectionType(ProjectionType.ALL)) - .provisionedThroughput(p -> p.readCapacityUnits(1L).writeCapacityUnits(1L)) - .build()); + .indexName("gsi_1") + .projection(p -> p.projectionType(ProjectionType.ALL)) + .provisionedThroughput(p -> p.readCapacityUnits(1L).writeCapacityUnits(1L)) + .build()); CreateTableOperation operation = CreateTableOperation.create(CreateTableEnhancedRequest.builder() - .globalSecondaryIndices(gsiList) - .build()); + .globalSecondaryIndices(gsiList) + .build()); CreateTableRequest request = operation.generateRequest(FakeItemWithIndices.getTableSchema(), PRIMARY_CONTEXT, null); @@ -511,15 +513,15 @@ public void generateRequest_gsiWithSingleKeys_buildsCorrectly() { public void generateRequest_gsiWithCompositeKeys() { List gsiList = Collections.singletonList( EnhancedGlobalSecondaryIndex.builder() - .indexName("composite_gsi") - .projection(p -> p.projectionType(ProjectionType.ALL)) - .provisionedThroughput(p -> p.readCapacityUnits(5L).writeCapacityUnits(5L)) - .build()); + .indexName("composite_gsi") + .projection(p -> p.projectionType(ProjectionType.ALL)) + .provisionedThroughput(p -> p.readCapacityUnits(5L).writeCapacityUnits(5L)) + .build()); CreateTableOperation operation = CreateTableOperation.create(CreateTableEnhancedRequest.builder() - .globalSecondaryIndices(gsiList) - .build()); + .globalSecondaryIndices(gsiList) + .build()); CreateTableRequest request = operation.generateRequest(FakeItemWithCompositeGsi.getTableSchema(), PRIMARY_CONTEXT, null); @@ -531,15 +533,15 @@ public void generateRequest_gsiWithCompositeKeys() { assertThat(gsi.keySchema().size(), is(4)); Set partitionKeyNames = gsi.keySchema().stream() - .filter(key -> key.keyType() == HASH) - .map(KeySchemaElement::attributeName) - .collect(Collectors.toSet()); + .filter(key -> key.keyType() == HASH) + .map(KeySchemaElement::attributeName) + .collect(Collectors.toSet()); assertThat(partitionKeyNames, containsInAnyOrder("gsi_pk1", "gsi_pk2")); Set sortKeyNames = gsi.keySchema().stream() - .filter(key -> key.keyType() == RANGE) - .map(KeySchemaElement::attributeName) - .collect(Collectors.toSet()); + .filter(key -> key.keyType() == RANGE) + .map(KeySchemaElement::attributeName) + .collect(Collectors.toSet()); assertThat(sortKeyNames, containsInAnyOrder("gsi_sk1", "gsi_sk2")); } @@ -547,15 +549,15 @@ public void generateRequest_gsiWithCompositeKeys() { public void generateRequest_gsiWithFlattenedPartitionKey() { List gsiList = Collections.singletonList( EnhancedGlobalSecondaryIndex.builder() - .indexName("flatten_partition_gsi") - .projection(p -> p.projectionType(ProjectionType.ALL)) - .provisionedThroughput(p -> p.readCapacityUnits(1L).writeCapacityUnits(1L)) - .build()); + .indexName("flatten_partition_gsi") + .projection(p -> p.projectionType(ProjectionType.ALL)) + .provisionedThroughput(p -> p.readCapacityUnits(1L).writeCapacityUnits(1L)) + .build()); CreateTableOperation operation = CreateTableOperation.create(CreateTableEnhancedRequest.builder() - .globalSecondaryIndices(gsiList) - .build()); + .globalSecondaryIndices(gsiList) + .build()); CreateTableRequest request = operation.generateRequest(FakeItemWithFlattenedGsi.getTableSchema(), PRIMARY_CONTEXT, null); @@ -572,15 +574,15 @@ public void generateRequest_gsiWithFlattenedPartitionKey() { public void generateRequest_gsiWithFlattenedSortKey() { List gsiList = Collections.singletonList( EnhancedGlobalSecondaryIndex.builder() - .indexName("flatten_sort_gsi") - .projection(p -> p.projectionType(ProjectionType.ALL)) - .provisionedThroughput(p -> p.readCapacityUnits(1L).writeCapacityUnits(1L)) - .build()); + .indexName("flatten_sort_gsi") + .projection(p -> p.projectionType(ProjectionType.ALL)) + .provisionedThroughput(p -> p.readCapacityUnits(1L).writeCapacityUnits(1L)) + .build()); CreateTableOperation operation = CreateTableOperation.create(CreateTableEnhancedRequest.builder() - .globalSecondaryIndices(gsiList) - .build()); + .globalSecondaryIndices(gsiList) + .build()); CreateTableRequest request = operation.generateRequest(FakeItemWithFlattenedGsi.getTableSchema(), PRIMARY_CONTEXT, null); @@ -599,15 +601,15 @@ public void generateRequest_gsiWithFlattenedSortKey() { public void generateRequest_gsiWithMixedFlattenedKeys() { List gsiList = Collections.singletonList( EnhancedGlobalSecondaryIndex.builder() - .indexName("flatten_mixed_gsi") - .projection(p -> p.projectionType(ProjectionType.ALL)) - .provisionedThroughput(p -> p.readCapacityUnits(1L).writeCapacityUnits(1L)) - .build()); + .indexName("flatten_mixed_gsi") + .projection(p -> p.projectionType(ProjectionType.ALL)) + .provisionedThroughput(p -> p.readCapacityUnits(1L).writeCapacityUnits(1L)) + .build()); CreateTableOperation operation = CreateTableOperation.create(CreateTableEnhancedRequest.builder() - .globalSecondaryIndices(gsiList) - .build()); + .globalSecondaryIndices(gsiList) + .build()); CreateTableRequest request = operation.generateRequest(FakeItemWithFlattenedGsi.getTableSchema(), PRIMARY_CONTEXT, null); @@ -616,17 +618,17 @@ public void generateRequest_gsiWithMixedFlattenedKeys() { GlobalSecondaryIndex gsi = request.globalSecondaryIndexes().get(0); assertThat(gsi.indexName(), is("flatten_mixed_gsi")); assertThat(gsi.keySchema().size(), is(2)); - + Set partitionKeyNames = gsi.keySchema().stream() - .filter(key -> key.keyType() == HASH) - .map(KeySchemaElement::attributeName) - .collect(Collectors.toSet()); + .filter(key -> key.keyType() == HASH) + .map(KeySchemaElement::attributeName) + .collect(Collectors.toSet()); assertThat(partitionKeyNames, containsInAnyOrder("gsiMixedPartitionKey")); Set sortKeyNames = gsi.keySchema().stream() - .filter(key -> key.keyType() == RANGE) - .map(KeySchemaElement::attributeName) - .collect(Collectors.toSet()); + .filter(key -> key.keyType() == RANGE) + .map(KeySchemaElement::attributeName) + .collect(Collectors.toSet()); assertThat(sortKeyNames, containsInAnyOrder("gsiMixedSortKey")); } @@ -634,15 +636,15 @@ public void generateRequest_gsiWithMixedFlattenedKeys() { public void generateRequest_gsiWithBothFlattenedKeys() { List gsiList = Collections.singletonList( EnhancedGlobalSecondaryIndex.builder() - .indexName("flatten_both_gsi") - .projection(p -> p.projectionType(ProjectionType.ALL)) - .provisionedThroughput(p -> p.readCapacityUnits(1L).writeCapacityUnits(1L)) - .build()); + .indexName("flatten_both_gsi") + .projection(p -> p.projectionType(ProjectionType.ALL)) + .provisionedThroughput(p -> p.readCapacityUnits(1L).writeCapacityUnits(1L)) + .build()); CreateTableOperation operation = CreateTableOperation.create(CreateTableEnhancedRequest.builder() - .globalSecondaryIndices(gsiList) - .build()); + .globalSecondaryIndices(gsiList) + .build()); CreateTableRequest request = operation.generateRequest(FakeItemWithFlattenedGsi.getTableSchema(), PRIMARY_CONTEXT, null); @@ -651,17 +653,17 @@ public void generateRequest_gsiWithBothFlattenedKeys() { GlobalSecondaryIndex gsi = request.globalSecondaryIndexes().get(0); assertThat(gsi.indexName(), is("flatten_both_gsi")); assertThat(gsi.keySchema().size(), is(2)); - + Set partitionKeyNames = gsi.keySchema().stream() - .filter(key -> key.keyType() == HASH) - .map(KeySchemaElement::attributeName) - .collect(Collectors.toSet()); + .filter(key -> key.keyType() == HASH) + .map(KeySchemaElement::attributeName) + .collect(Collectors.toSet()); assertThat(partitionKeyNames, containsInAnyOrder("gsiBothSortKey")); Set sortKeyNames = gsi.keySchema().stream() - .filter(key -> key.keyType() == RANGE) - .map(KeySchemaElement::attributeName) - .collect(Collectors.toSet()); + .filter(key -> key.keyType() == RANGE) + .map(KeySchemaElement::attributeName) + .collect(Collectors.toSet()); assertThat(sortKeyNames, containsInAnyOrder("gsiBothSortKey")); } @@ -669,15 +671,15 @@ public void generateRequest_gsiWithBothFlattenedKeys() { public void generateRequest_gsiWithMixedCompositePartitionKeys() { List gsiList = Collections.singletonList( EnhancedGlobalSecondaryIndex.builder() - .indexName("mixed_partition_gsi") - .projection(p -> p.projectionType(ProjectionType.ALL)) - .provisionedThroughput(p -> p.readCapacityUnits(1L).writeCapacityUnits(1L)) - .build()); + .indexName("mixed_partition_gsi") + .projection(p -> p.projectionType(ProjectionType.ALL)) + .provisionedThroughput(p -> p.readCapacityUnits(1L).writeCapacityUnits(1L)) + .build()); CreateTableOperation operation = CreateTableOperation.create(CreateTableEnhancedRequest.builder() - .globalSecondaryIndices(gsiList) - .build()); + .globalSecondaryIndices(gsiList) + .build()); CreateTableRequest request = operation.generateRequest(FakeItemWithMixedCompositeGsi.getTableSchema(), PRIMARY_CONTEXT, null); @@ -686,27 +688,28 @@ public void generateRequest_gsiWithMixedCompositePartitionKeys() { GlobalSecondaryIndex gsi = request.globalSecondaryIndexes().get(0); assertThat(gsi.indexName(), is("mixed_partition_gsi")); assertThat(gsi.keySchema().size(), is(4)); - + Set partitionKeyNames = gsi.keySchema().stream() - .filter(key -> key.keyType() == HASH) - .map(KeySchemaElement::attributeName) - .collect(Collectors.toSet()); - assertThat(partitionKeyNames, containsInAnyOrder("rootPartitionKey1", "rootPartitionKey2", "flattenedPartitionKey1", "flattenedPartitionKey2")); + .filter(key -> key.keyType() == HASH) + .map(KeySchemaElement::attributeName) + .collect(Collectors.toSet()); + assertThat(partitionKeyNames, containsInAnyOrder("rootPartitionKey1", "rootPartitionKey2", "flattenedPartitionKey1", + "flattenedPartitionKey2")); } @Test public void generateRequest_gsiWithMixedCompositeSortKeys() { List gsiList = Collections.singletonList( EnhancedGlobalSecondaryIndex.builder() - .indexName("mixed_sort_gsi") - .projection(p -> p.projectionType(ProjectionType.ALL)) - .provisionedThroughput(p -> p.readCapacityUnits(1L).writeCapacityUnits(1L)) - .build()); + .indexName("mixed_sort_gsi") + .projection(p -> p.projectionType(ProjectionType.ALL)) + .provisionedThroughput(p -> p.readCapacityUnits(1L).writeCapacityUnits(1L)) + .build()); CreateTableOperation operation = CreateTableOperation.create(CreateTableEnhancedRequest.builder() - .globalSecondaryIndices(gsiList) - .build()); + .globalSecondaryIndices(gsiList) + .build()); CreateTableRequest request = operation.generateRequest(FakeItemWithMixedCompositeGsi.getTableSchema(), PRIMARY_CONTEXT, null); @@ -717,15 +720,15 @@ public void generateRequest_gsiWithMixedCompositeSortKeys() { assertThat(gsi.keySchema().size(), is(6)); Set partitionKeyNames = gsi.keySchema().stream() - .filter(key -> key.keyType() == HASH) - .map(KeySchemaElement::attributeName) - .collect(Collectors.toSet()); + .filter(key -> key.keyType() == HASH) + .map(KeySchemaElement::attributeName) + .collect(Collectors.toSet()); assertThat(partitionKeyNames, containsInAnyOrder("rootPartitionKey1", "rootPartitionKey2")); Set sortKeyNames = gsi.keySchema().stream() - .filter(key -> key.keyType() == RANGE) - .map(KeySchemaElement::attributeName) - .collect(Collectors.toSet()); + .filter(key -> key.keyType() == RANGE) + .map(KeySchemaElement::attributeName) + .collect(Collectors.toSet()); assertThat(sortKeyNames, containsInAnyOrder("rootSortKey1", "rootSortKey2", "flattenedSortKey1", "flattenedSortKey2")); } @@ -733,15 +736,15 @@ public void generateRequest_gsiWithMixedCompositeSortKeys() { public void generateRequest_gsiWithFullMixedCompositeKeys() { List gsiList = Collections.singletonList( EnhancedGlobalSecondaryIndex.builder() - .indexName("full_mixed_gsi") - .projection(p -> p.projectionType(ProjectionType.ALL)) - .provisionedThroughput(p -> p.readCapacityUnits(1L).writeCapacityUnits(1L)) - .build()); + .indexName("full_mixed_gsi") + .projection(p -> p.projectionType(ProjectionType.ALL)) + .provisionedThroughput(p -> p.readCapacityUnits(1L).writeCapacityUnits(1L)) + .build()); CreateTableOperation operation = CreateTableOperation.create(CreateTableEnhancedRequest.builder() - .globalSecondaryIndices(gsiList) - .build()); + .globalSecondaryIndices(gsiList) + .build()); CreateTableRequest request = operation.generateRequest(FakeItemWithMixedCompositeGsi.getTableSchema(), PRIMARY_CONTEXT, null); @@ -750,17 +753,18 @@ public void generateRequest_gsiWithFullMixedCompositeKeys() { GlobalSecondaryIndex gsi = request.globalSecondaryIndexes().get(0); assertThat(gsi.indexName(), is("full_mixed_gsi")); assertThat(gsi.keySchema().size(), is(8)); - + Set partitionKeyNames = gsi.keySchema().stream() - .filter(key -> key.keyType() == HASH) - .map(KeySchemaElement::attributeName) - .collect(Collectors.toSet()); - assertThat(partitionKeyNames, containsInAnyOrder("rootPartitionKey1", "rootPartitionKey2", "flattenedPartitionKey1", "flattenedPartitionKey2")); + .filter(key -> key.keyType() == HASH) + .map(KeySchemaElement::attributeName) + .collect(Collectors.toSet()); + assertThat(partitionKeyNames, containsInAnyOrder("rootPartitionKey1", "rootPartitionKey2", "flattenedPartitionKey1", + "flattenedPartitionKey2")); Set sortKeyNames = gsi.keySchema().stream() - .filter(key -> key.keyType() == RANGE) - .map(KeySchemaElement::attributeName) - .collect(Collectors.toSet()); + .filter(key -> key.keyType() == RANGE) + .map(KeySchemaElement::attributeName) + .collect(Collectors.toSet()); assertThat(sortKeyNames, containsInAnyOrder("rootSortKey1", "rootSortKey2", "flattenedSortKey1", "flattenedSortKey2")); } @@ -768,15 +772,15 @@ public void generateRequest_gsiWithFullMixedCompositeKeys() { public void generateRequest_immutableGsiWithCompositeKeys() { List gsiList = Collections.singletonList( EnhancedGlobalSecondaryIndex.builder() - .indexName("gsi1") - .projection(p -> p.projectionType(ProjectionType.ALL)) - .provisionedThroughput(p -> p.readCapacityUnits(5L).writeCapacityUnits(5L)) - .build()); + .indexName("gsi1") + .projection(p -> p.projectionType(ProjectionType.ALL)) + .provisionedThroughput(p -> p.readCapacityUnits(5L).writeCapacityUnits(5L)) + .build()); CreateTableOperation operation = CreateTableOperation.create(CreateTableEnhancedRequest.builder() - .globalSecondaryIndices(gsiList) - .build()); + .globalSecondaryIndices(gsiList) + .build()); CreateTableRequest request = operation.generateRequest(ImmutableTableSchema.create(CompositeMetadataImmutable.class), PRIMARY_CONTEXT, null); @@ -787,15 +791,15 @@ public void generateRequest_immutableGsiWithCompositeKeys() { assertThat(gsi.keySchema().size(), is(4)); Set partitionKeyNames = gsi.keySchema().stream() - .filter(key -> key.keyType() == HASH) - .map(KeySchemaElement::attributeName) - .collect(Collectors.toSet()); + .filter(key -> key.keyType() == HASH) + .map(KeySchemaElement::attributeName) + .collect(Collectors.toSet()); assertThat(partitionKeyNames, containsInAnyOrder("gsiPk1", "gsiPk2")); Set sortKeyNames = gsi.keySchema().stream() - .filter(key -> key.keyType() == RANGE) - .map(KeySchemaElement::attributeName) - .collect(Collectors.toSet()); + .filter(key -> key.keyType() == RANGE) + .map(KeySchemaElement::attributeName) + .collect(Collectors.toSet()); assertThat(sortKeyNames, containsInAnyOrder("gsiSk1", "gsiSk2")); } @@ -803,38 +807,38 @@ public void generateRequest_immutableGsiWithCompositeKeys() { public void generateRequest_immutableGsiWithCrossIndexKeys() { List gsiList = Arrays.asList( EnhancedGlobalSecondaryIndex.builder() - .indexName("gsi1") - .projection(p -> p.projectionType(ProjectionType.ALL)) - .provisionedThroughput(p -> p.readCapacityUnits(1L).writeCapacityUnits(1L)) - .build(), + .indexName("gsi1") + .projection(p -> p.projectionType(ProjectionType.ALL)) + .provisionedThroughput(p -> p.readCapacityUnits(1L).writeCapacityUnits(1L)) + .build(), EnhancedGlobalSecondaryIndex.builder() - .indexName("gsi2") - .projection(p -> p.projectionType(ProjectionType.ALL)) - .provisionedThroughput(p -> p.readCapacityUnits(1L).writeCapacityUnits(1L)) - .build()); + .indexName("gsi2") + .projection(p -> p.projectionType(ProjectionType.ALL)) + .provisionedThroughput(p -> p.readCapacityUnits(1L).writeCapacityUnits(1L)) + .build()); CreateTableOperation operation = CreateTableOperation.create(CreateTableEnhancedRequest.builder() - .globalSecondaryIndices(gsiList) - .build()); + .globalSecondaryIndices(gsiList) + .build()); CreateTableRequest request = operation.generateRequest(ImmutableTableSchema.create(CrossIndexImmutable.class), PRIMARY_CONTEXT, null); assertThat(request.globalSecondaryIndexes().size(), is(2)); - + GlobalSecondaryIndex gsi1 = request.globalSecondaryIndexes().stream() - .filter(gsi -> "gsi1".equals(gsi.indexName())) - .findFirst().orElse(null); + .filter(gsi -> "gsi1".equals(gsi.indexName())) + .findFirst().orElse(null); assertThat(gsi1.keySchema().size(), is(2)); assertThat(gsi1.keySchema().get(0).attributeName(), is("attr1")); assertThat(gsi1.keySchema().get(0).keyType(), is(HASH)); assertThat(gsi1.keySchema().get(1).attributeName(), is("attr2")); assertThat(gsi1.keySchema().get(1).keyType(), is(HASH)); - + GlobalSecondaryIndex gsi2 = request.globalSecondaryIndexes().stream() - .filter(gsi -> "gsi2".equals(gsi.indexName())) - .findFirst().orElse(null); + .filter(gsi -> "gsi2".equals(gsi.indexName())) + .findFirst().orElse(null); assertThat(gsi2.keySchema().size(), is(2)); assertThat(gsi2.keySchema().get(0).attributeName(), is("attr3")); assertThat(gsi2.keySchema().get(0).keyType(), is(HASH)); @@ -846,15 +850,15 @@ public void generateRequest_immutableGsiWithCrossIndexKeys() { public void generateRequest_immutableGsiWithMixedFlattenedKeys() { List gsiList = Collections.singletonList( EnhancedGlobalSecondaryIndex.builder() - .indexName("mixed_gsi") - .projection(p -> p.projectionType(ProjectionType.ALL)) - .provisionedThroughput(p -> p.readCapacityUnits(1L).writeCapacityUnits(1L)) - .build()); + .indexName("mixed_gsi") + .projection(p -> p.projectionType(ProjectionType.ALL)) + .provisionedThroughput(p -> p.readCapacityUnits(1L).writeCapacityUnits(1L)) + .build()); CreateTableOperation operation = CreateTableOperation.create(CreateTableEnhancedRequest.builder() - .globalSecondaryIndices(gsiList) - .build()); + .globalSecondaryIndices(gsiList) + .build()); CreateTableRequest request = operation.generateRequest(ImmutableTableSchema.create(MixedFlattenedImmutable.class), PRIMARY_CONTEXT, null); @@ -865,15 +869,15 @@ public void generateRequest_immutableGsiWithMixedFlattenedKeys() { assertThat(gsi.keySchema().size(), is(4)); Set partitionKeyNames = gsi.keySchema().stream() - .filter(key -> key.keyType() == HASH) - .map(KeySchemaElement::attributeName) - .collect(Collectors.toSet()); + .filter(key -> key.keyType() == HASH) + .map(KeySchemaElement::attributeName) + .collect(Collectors.toSet()); assertThat(partitionKeyNames, containsInAnyOrder("rootKey1", "flatKey1")); Set sortKeyNames = gsi.keySchema().stream() - .filter(key -> key.keyType() == RANGE) - .map(KeySchemaElement::attributeName) - .collect(Collectors.toSet()); + .filter(key -> key.keyType() == RANGE) + .map(KeySchemaElement::attributeName) + .collect(Collectors.toSet()); assertThat(sortKeyNames, containsInAnyOrder("rootKey2", "flatKey2")); } } diff --git a/services-custom/dynamodb-enhanced/src/test/java/software/amazon/awssdk/enhanced/dynamodb/query/ConditionTest.java b/services-custom/dynamodb-enhanced/src/test/java/software/amazon/awssdk/enhanced/dynamodb/query/ConditionTest.java new file mode 100644 index 000000000000..0be04144910e --- /dev/null +++ b/services-custom/dynamodb-enhanced/src/test/java/software/amazon/awssdk/enhanced/dynamodb/query/ConditionTest.java @@ -0,0 +1,259 @@ +/* + * Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"). + * You may not use this file except in compliance with the License. + * A copy of the License is located at + * + * http://aws.amazon.com/apache2.0 + * + * or in the "license" file accompanying this file. This file is distributed + * on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either + * express or implied. See the License for the specific language governing + * permissions and limitations under the License. + */ + +package software.amazon.awssdk.enhanced.dynamodb.query; + +import static org.assertj.core.api.Assertions.assertThat; + +import java.util.HashMap; +import java.util.Map; +import org.junit.Test; +import software.amazon.awssdk.enhanced.dynamodb.query.condition.Condition; +import software.amazon.awssdk.enhanced.dynamodb.query.condition.ConditionEvaluator; + +/** + * Unit tests for the {@link Condition} DSL: primitive conditions (eq, gt, between, etc.), combinators (and, or, not), and tree + * conditions using {@link Condition#group(Condition)} for precedence. + *

+ * These tests do not use DynamoDB; they assert that condition objects are built correctly and that combined conditions are + * non-null and can be used in a spec. + */ +public class ConditionTest { + + /** + * Asserts that {@link Condition#eq(String, Object)} returns a non-null condition and that chaining + * {@link Condition#and(Condition)} produces a combined condition. + */ + @Test + public void eqAndAndCombination() { + Condition c = Condition.eq("status", "ACTIVE").and(Condition.gt("value", 0)); + assertThat(c).isNotNull(); + } + + /** + * Asserts that {@link Condition#or(Condition)} combines two conditions and returns non-null. + */ + @Test + public void orCombination() { + Condition c = Condition.eq("a", 1).or(Condition.eq("b", 2)); + assertThat(c).isNotNull(); + } + + /** + * Asserts that {@link Condition#not()} returns a negated condition (non-null). + */ + @Test + public void not() { + Condition c = Condition.eq("x", 1).not(); + assertThat(c).isNotNull(); + } + + /** + * Asserts that {@link Condition#group(Condition)} wraps a condition for precedence and that a tree expression (e.g. (a=1 AND + * b>0) OR c=2) can be built. Verifies the grouped condition is non-null and distinct from the inner condition. + */ + @Test + public void groupForPrecedence() { + Condition inner = Condition.eq("a", 1).and(Condition.gt("b", 0)); + Condition grouped = Condition.group(inner); + assertThat(grouped).isNotNull(); + Condition tree = grouped.or(Condition.eq("c", 2)); + assertThat(tree).isNotNull(); + } + + /** + * Asserts that all primitive factory methods return non-null conditions: eq, gt, gte, lt, lte, between, contains, + * beginsWith. + */ + @Test + public void primitiveConditions() { + assertThat(Condition.eq("k", "v")).isNotNull(); + assertThat(Condition.gt("k", 1)).isNotNull(); + assertThat(Condition.gte("k", 1)).isNotNull(); + assertThat(Condition.lt("k", 1)).isNotNull(); + assertThat(Condition.lte("k", 1)).isNotNull(); + assertThat(Condition.between("k", 0, 10)).isNotNull(); + assertThat(Condition.contains("k", "sub")).isNotNull(); + assertThat(Condition.beginsWith("k", "pre")).isNotNull(); + } + + /** + * Asserts that equality is consistent for the same condition built twice (eq with same attribute and value). + */ + @Test + public void eqConditionsEqual() { + Condition c1 = Condition.eq("id", 42); + Condition c2 = Condition.eq("id", 42); + assertThat(c1).isEqualTo(c2); + assertThat(c1.hashCode()).isEqualTo(c2.hashCode()); + } + + /** + * Asserts that and() and or() produce equal combinations when given the same operands. + */ + @Test + public void andOrEquality() { + Condition left = Condition.eq("a", 1); + Condition right = Condition.eq("b", 2); + Condition and1 = left.and(right); + Condition and2 = Condition.eq("a", 1).and(Condition.eq("b", 2)); + assertThat(and1).isEqualTo(and2); + Condition or1 = left.or(right); + Condition or2 = Condition.eq("a", 1).or(Condition.eq("b", 2)); + assertThat(or1).isEqualTo(or2); + } + + // --- Nested attribute (dot-path) evaluation tests --- + + private Map itemWithNestedAddress() { + Map address = new HashMap<>(); + address.put("city", "Seattle"); + address.put("zip", "98101"); + address.put("state", "WA"); + Map item = new HashMap<>(); + item.put("customerId", "c1"); + item.put("address", address); + return item; + } + + private Map itemWithDeepNesting() { + Map geo = new HashMap<>(); + geo.put("lat", 47.6); + geo.put("lng", -122.3); + Map address = new HashMap<>(); + address.put("city", "Seattle"); + address.put("geo", geo); + Map item = new HashMap<>(); + item.put("address", address); + return item; + } + + @Test + public void nestedAttribute_eqMatch() { + Map item = itemWithNestedAddress(); + assertThat(ConditionEvaluator.evaluate(Condition.eq("address.city", "Seattle"), item)).isTrue(); + } + + @Test + public void nestedAttribute_eqNoMatch() { + Map item = itemWithNestedAddress(); + assertThat(ConditionEvaluator.evaluate(Condition.eq("address.city", "Portland"), item)).isFalse(); + } + + @Test + public void nestedAttribute_gtOnString() { + Map item = itemWithNestedAddress(); + assertThat(ConditionEvaluator.evaluate(Condition.gt("address.state", "TX"), item)).isTrue(); + assertThat(ConditionEvaluator.evaluate(Condition.gt("address.state", "WA"), item)).isFalse(); + } + + @Test + public void nestedAttribute_betweenOnZip() { + Map item = itemWithNestedAddress(); + assertThat(ConditionEvaluator.evaluate(Condition.between("address.zip", "90000", "99999"), item)).isTrue(); + assertThat(ConditionEvaluator.evaluate(Condition.between("address.zip", "10000", "20000"), item)).isFalse(); + } + + @Test + public void nestedAttribute_contains() { + Map item = itemWithNestedAddress(); + assertThat(ConditionEvaluator.evaluate(Condition.contains("address.city", "att"), item)).isTrue(); + assertThat(ConditionEvaluator.evaluate(Condition.contains("address.city", "xyz"), item)).isFalse(); + } + + @Test + public void nestedAttribute_beginsWith() { + Map item = itemWithNestedAddress(); + assertThat(ConditionEvaluator.evaluate(Condition.beginsWith("address.city", "Sea"), item)).isTrue(); + assertThat(ConditionEvaluator.evaluate(Condition.beginsWith("address.city", "Por"), item)).isFalse(); + } + + @Test + public void nestedAttribute_deepPath() { + Map item = itemWithDeepNesting(); + assertThat(ConditionEvaluator.evaluate(Condition.gt("address.geo.lat", 40.0), item)).isTrue(); + assertThat(ConditionEvaluator.evaluate(Condition.lt("address.geo.lng", -100.0), item)).isTrue(); + } + + @Test + public void nestedAttribute_missingIntermediateReturnsNull() { + Map item = new HashMap<>(); + item.put("customerId", "c1"); + assertThat(ConditionEvaluator.evaluate(Condition.eq("address.city", "Seattle"), item)).isFalse(); + } + + @Test + public void nestedAttribute_intermediateNotMapReturnsNull() { + Map item = new HashMap<>(); + item.put("address", "plain-string-not-a-map"); + assertThat(ConditionEvaluator.evaluate(Condition.eq("address.city", "Seattle"), item)).isFalse(); + } + + @Test + public void nestedAttribute_andCombination() { + Map item = itemWithNestedAddress(); + Condition c = Condition.eq("address.city", "Seattle").and(Condition.eq("address.state", "WA")); + assertThat(ConditionEvaluator.evaluate(c, item)).isTrue(); + + Condition noMatch = Condition.eq("address.city", "Seattle").and(Condition.eq("address.state", "CA")); + assertThat(ConditionEvaluator.evaluate(noMatch, item)).isFalse(); + } + + @Test + public void nestedAttribute_orCombination() { + Map item = itemWithNestedAddress(); + Condition c = Condition.eq("address.city", "Portland").or(Condition.eq("address.state", "WA")); + assertThat(ConditionEvaluator.evaluate(c, item)).isTrue(); + } + + @Test + public void nestedAttribute_mixedWithFlatAttribute() { + Map item = itemWithNestedAddress(); + Condition c = Condition.eq("customerId", "c1").and(Condition.eq("address.city", "Seattle")); + assertThat(ConditionEvaluator.evaluate(c, item)).isTrue(); + } + + @Test + public void nestedAttribute_combinedMapView() { + Map primary = new HashMap<>(); + primary.put("customerId", "c1"); + + Map address = new HashMap<>(); + address.put("city", "Seattle"); + Map secondary = new HashMap<>(); + secondary.put("address", address); + + Condition c = Condition.eq("address.city", "Seattle"); + assertThat(ConditionEvaluator.evaluate(c, primary, secondary)).isTrue(); + } + + @Test + public void resolveAttribute_flatKeyUnchanged() { + Map item = new HashMap<>(); + item.put("name", "Alice"); + assertThat(ConditionEvaluator.resolveAttribute(item, "name")).isEqualTo("Alice"); + } + + @Test + public void resolveAttribute_nullItemReturnsNull() { + assertThat(ConditionEvaluator.resolveAttribute(null, "name")).isNull(); + } + + @Test + public void resolveAttribute_nullAttributeReturnsNull() { + Map item = new HashMap<>(); + assertThat(ConditionEvaluator.resolveAttribute(item, null)).isNull(); + } +} diff --git a/services-custom/dynamodb-enhanced/src/test/java/software/amazon/awssdk/enhanced/dynamodb/query/EnhancedQueryLatencyReportTest.java b/services-custom/dynamodb-enhanced/src/test/java/software/amazon/awssdk/enhanced/dynamodb/query/EnhancedQueryLatencyReportTest.java new file mode 100644 index 000000000000..7c92fc7a7646 --- /dev/null +++ b/services-custom/dynamodb-enhanced/src/test/java/software/amazon/awssdk/enhanced/dynamodb/query/EnhancedQueryLatencyReportTest.java @@ -0,0 +1,37 @@ +/* + * Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"). + * You may not use this file except in compliance with the License. + * A copy of the License is located at + * + * http://aws.amazon.com/apache2.0 + * + * or in the "license" file accompanying this file. This file is distributed + * on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either + * express or implied. See the License for the specific language governing + * permissions and limitations under the License. + */ + +package software.amazon.awssdk.enhanced.dynamodb.query; + +import static org.assertj.core.api.Assertions.assertThat; + +import org.junit.Test; +import software.amazon.awssdk.enhanced.dynamodb.query.result.EnhancedQueryLatencyReport; + +/** + * Unit tests for {@link EnhancedQueryLatencyReport}. + */ +public class EnhancedQueryLatencyReportTest { + + @Test + public void gettersReturnValuesPassedToConstructor() { + EnhancedQueryLatencyReport report = new EnhancedQueryLatencyReport(1L, 2L, 3L, 4L); + + assertThat(report.baseQueryMs()).isEqualTo(1L); + assertThat(report.joinedLookupsMs()).isEqualTo(2L); + assertThat(report.inMemoryProcessingMs()).isEqualTo(3L); + assertThat(report.totalMs()).isEqualTo(4L); + } +} diff --git a/services-custom/dynamodb-enhanced/src/test/java/software/amazon/awssdk/enhanced/dynamodb/query/EnhancedQueryResultTest.java b/services-custom/dynamodb-enhanced/src/test/java/software/amazon/awssdk/enhanced/dynamodb/query/EnhancedQueryResultTest.java new file mode 100644 index 000000000000..4c408f971e53 --- /dev/null +++ b/services-custom/dynamodb-enhanced/src/test/java/software/amazon/awssdk/enhanced/dynamodb/query/EnhancedQueryResultTest.java @@ -0,0 +1,71 @@ +/* + * Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"). + * You may not use this file except in compliance with the License. + * A copy of the License is located at + * + * http://aws.amazon.com/apache2.0 + * + * or in the "license" file accompanying this file. This file is distributed + * on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either + * express or implied. See the License for the specific language governing + * permissions and limitations under the License. + */ + +package software.amazon.awssdk.enhanced.dynamodb.query; + +import static org.assertj.core.api.Assertions.assertThat; + +import java.util.ArrayList; +import java.util.List; +import java.util.stream.Collectors; +import org.junit.Test; +import software.amazon.awssdk.enhanced.dynamodb.query.result.DefaultEnhancedQueryResult; +import software.amazon.awssdk.enhanced.dynamodb.query.result.EnhancedQueryResult; +import software.amazon.awssdk.enhanced.dynamodb.query.result.EnhancedQueryRow; + +/** + * Unit tests for {@link DefaultEnhancedQueryResult}. + */ +public class EnhancedQueryResultTest { + + @Test + public void fromList_iterationReturnsRowsInOrder() { + EnhancedQueryRow row1 = EnhancedQueryRow.builder().build(); + EnhancedQueryRow row2 = EnhancedQueryRow.builder().build(); + List list = new ArrayList<>(); + list.add(row1); + list.add(row2); + + EnhancedQueryResult result = new DefaultEnhancedQueryResult(list); + + List collected = new ArrayList<>(); + result.forEach(collected::add); + assertThat(collected).hasSize(2); + assertThat(collected.get(0)).isSameAs(row1); + assertThat(collected.get(1)).isSameAs(row2); + } + + @Test + public void stream_collectsSameRows() { + EnhancedQueryRow row1 = EnhancedQueryRow.builder().build(); + List list = new ArrayList<>(); + list.add(row1); + + EnhancedQueryResult result = new DefaultEnhancedQueryResult(list); + + List fromStream = result.stream().collect(Collectors.toList()); + assertThat(fromStream).hasSize(1); + assertThat(fromStream.get(0)).isSameAs(row1); + } + + @Test + public void emptyList_iterationReturnsNothing() { + EnhancedQueryResult result = new DefaultEnhancedQueryResult(new ArrayList<>()); + + List collected = new ArrayList<>(); + result.forEach(collected::add); + assertThat(collected).isEmpty(); + } +} diff --git a/services-custom/dynamodb-enhanced/src/test/java/software/amazon/awssdk/enhanced/dynamodb/query/EnhancedQueryRowTest.java b/services-custom/dynamodb-enhanced/src/test/java/software/amazon/awssdk/enhanced/dynamodb/query/EnhancedQueryRowTest.java new file mode 100644 index 000000000000..117eaf119e57 --- /dev/null +++ b/services-custom/dynamodb-enhanced/src/test/java/software/amazon/awssdk/enhanced/dynamodb/query/EnhancedQueryRowTest.java @@ -0,0 +1,96 @@ +/* + * Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"). + * You may not use this file except in compliance with the License. + * A copy of the License is located at + * + * http://aws.amazon.com/apache2.0 + * + * or in the "license" file accompanying this file. This file is distributed + * on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either + * express or implied. See the License for the specific language governing + * permissions and limitations under the License. + */ + +package software.amazon.awssdk.enhanced.dynamodb.query; + +import static org.assertj.core.api.Assertions.assertThat; + +import java.util.Collections; +import java.util.HashMap; +import java.util.Map; +import org.junit.Test; +import software.amazon.awssdk.enhanced.dynamodb.query.result.EnhancedQueryRow; + +/** + * Unit tests for {@link EnhancedQueryRow} and its builder. + */ +public class EnhancedQueryRowTest { + + @Test + public void emptyRow_returnsEmptyMapsAndNullAggregate() { + EnhancedQueryRow row = EnhancedQueryRow.builder().build(); + + assertThat(row.getItem("base")).isEmpty(); + assertThat(row.itemsByAlias()).isEmpty(); + assertThat(row.aggregates()).isEmpty(); + assertThat(row.getAggregate("any")).isNull(); + } + + @Test + public void itemsByAlias_getItemReturnsCorrectMap() { + Map baseMap = new HashMap<>(); + baseMap.put("customerId", "c1"); + baseMap.put("name", "Alice"); + Map> byAlias = new HashMap<>(); + byAlias.put("base", baseMap); + + EnhancedQueryRow row = EnhancedQueryRow.builder() + .itemsByAlias(byAlias) + .build(); + + assertThat(row.getItem("base")) + .containsEntry("customerId", "c1") + .containsEntry("name", "Alice"); + assertThat(row.itemsByAlias()).containsKey("base"); + assertThat(row.getItem("unknown")).isEmpty(); + } + + @Test + public void aggregates_getAggregateReturnsValue() { + Map aggs = new HashMap<>(); + aggs.put("orderCount", 5); + aggs.put("totalAmount", 100.50); + + EnhancedQueryRow row = EnhancedQueryRow.builder() + .aggregates(aggs) + .build(); + + assertThat(row.getAggregate("orderCount")).isEqualTo(5); + assertThat(row.getAggregate("totalAmount")).isEqualTo(100.50); + assertThat(row.aggregates()) + .containsEntry("orderCount", 5) + .containsEntry("totalAmount", 100.50); + assertThat(row.getAggregate("missing")).isNull(); + } + + @Test + public void rowWithItemsAndAggregates_returnsBoth() { + Map base = Collections.singletonMap("customerId", "c1"); + Map joined = Collections.singletonMap("orderId", "o1"); + Map> byAlias = new HashMap<>(); + byAlias.put("base", base); + byAlias.put("joined", joined); + Map aggs = Collections.singletonMap("orderCount", 3); + + EnhancedQueryRow row = EnhancedQueryRow.builder() + .itemsByAlias(byAlias) + .aggregates(aggs) + .build(); + + assertThat(row.getItem("base")).containsEntry("customerId", "c1"); + assertThat(row.getItem("joined")).containsEntry("orderId", "o1"); + assertThat(row.getAggregate("orderCount")).isEqualTo(3); + } +} diff --git a/services-custom/dynamodb-enhanced/src/test/java/software/amazon/awssdk/enhanced/dynamodb/query/QueryExpressionBuilderTest.java b/services-custom/dynamodb-enhanced/src/test/java/software/amazon/awssdk/enhanced/dynamodb/query/QueryExpressionBuilderTest.java new file mode 100644 index 000000000000..e131ec050a44 --- /dev/null +++ b/services-custom/dynamodb-enhanced/src/test/java/software/amazon/awssdk/enhanced/dynamodb/query/QueryExpressionBuilderTest.java @@ -0,0 +1,224 @@ +/* + * Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"). + * You may not use this file except in compliance with the License. + * A copy of the License is located at + * + * http://aws.amazon.com/apache2.0 + * + * or in the "license" file accompanying this file. This file is distributed + * on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either + * express or implied. See the License for the specific language governing + * permissions and limitations under the License. + */ + +package software.amazon.awssdk.enhanced.dynamodb.query; + +import static org.assertj.core.api.Assertions.assertThat; +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.when; + +import org.junit.Before; +import org.junit.Test; +import software.amazon.awssdk.enhanced.dynamodb.DynamoDbTable; +import software.amazon.awssdk.enhanced.dynamodb.query.enums.AggregationFunction; +import software.amazon.awssdk.enhanced.dynamodb.query.enums.ExecutionMode; +import software.amazon.awssdk.enhanced.dynamodb.query.enums.JoinType; +import software.amazon.awssdk.enhanced.dynamodb.query.enums.SortDirection; +import software.amazon.awssdk.enhanced.dynamodb.query.spec.AggregateSpec; +import software.amazon.awssdk.enhanced.dynamodb.query.condition.Condition; +import software.amazon.awssdk.enhanced.dynamodb.query.spec.OrderBySpec; +import software.amazon.awssdk.enhanced.dynamodb.query.engine.QueryExpressionBuilder; +import software.amazon.awssdk.enhanced.dynamodb.query.spec.QueryExpressionSpec; + +/** + * Unit tests for {@link QueryExpressionBuilder}. Verifies that the fluent builder produces a {@link QueryExpressionSpec} with the + * correct base table, join, conditions, group-by, aggregates, order-by, projection, execution mode, and limit. + *

+ * Uses a mock {@link DynamoDbTable} for the base (and optionally joined) table; no DynamoDB is used. + */ +@SuppressWarnings("unchecked") +public class QueryExpressionBuilderTest { + + private DynamoDbTable baseTable; + private DynamoDbTable joinedTable; + + /** + * Creates mock tables before each test so that {@link QueryExpressionBuilder#from(DynamoDbTable)} and + * {@link #join(DynamoDbTable, JoinType, String, String)} have valid table references. + */ + @Before + public void setUp() { + baseTable = mock(DynamoDbTable.class); + when(baseTable.tableName()).thenReturn("base-table"); + joinedTable = mock(DynamoDbTable.class); + when(joinedTable.tableName()).thenReturn("joined-table"); + } + + /** + * Asserts that {@link QueryExpressionBuilder#from(DynamoDbTable)} and {@link QueryExpressionBuilder#build()} produce a spec + * whose {@link QueryExpressionSpec#baseTable()} is the given table and {@link QueryExpressionSpec#hasJoin()} is false. + */ + @Test + public void fromAndBuild_setsBaseTableNoJoin() { + QueryExpressionSpec spec = QueryExpressionBuilder.from(baseTable).build(); + + assertThat(spec.baseTable()).isSameAs(baseTable); + assertThat(spec.hasJoin()).isFalse(); + assertThat(spec.joinedTable()).isNull(); + assertThat(spec.executionMode()).isEqualTo(ExecutionMode.STRICT_KEY_ONLY); + } + + /** + * Asserts that after {@link QueryExpressionBuilder#join(DynamoDbTable, JoinType, String, String)}, the built spec has the + * joined table, join type, and join attributes set. + */ + @Test + public void join_setsJoinedTableAndAttributes() { + QueryExpressionSpec spec = QueryExpressionBuilder.from(baseTable) + .join(joinedTable, JoinType.INNER, "customerId", "customerId") + .build(); + + assertThat(spec.hasJoin()).isTrue(); + assertThat(spec.joinedTable()).isSameAs(joinedTable); + assertThat(spec.joinType()).isEqualTo(JoinType.INNER); + assertThat(spec.leftJoinKey()).isEqualTo("customerId"); + assertThat(spec.rightJoinKey()).isEqualTo("customerId"); + } + + /** + * Asserts that {@link QueryExpressionBuilder#withCondition(Condition)} is stored in the spec as the single-table filter + * ({@link QueryExpressionSpec#condition()}). + */ + @Test + public void withCondition_setsCondition() { + Condition cond = Condition.eq("status", "ACTIVE"); + QueryExpressionSpec spec = QueryExpressionBuilder.from(baseTable).where(cond).build(); + + assertThat(spec.where()).isSameAs(cond); + } + + /** + * Asserts that {@link QueryExpressionBuilder#groupBy(String...)} adds attributes to + * {@link QueryExpressionSpec#groupByAttributes()}. + */ + @Test + public void groupBy_setsGroupByAttributes() { + QueryExpressionSpec spec = QueryExpressionBuilder.from(baseTable) + .groupBy("customerId", "category") + .build(); + + assertThat(spec.groupByAttributes()).containsExactly("customerId", "category"); + } + + /** + * Asserts that {@link QueryExpressionBuilder#aggregate(AggregationFunction, String, String)} adds an {@link AggregateSpec} + * with the given function, attribute, and output name. + */ + @Test + public void aggregate_addsAggregateSpec() { + QueryExpressionSpec spec = QueryExpressionBuilder.from(baseTable) + .aggregate(AggregationFunction.COUNT, "orderId", "orderCount") + .build(); + + assertThat(spec.aggregates()).hasSize(1); + assertThat(spec.aggregates().get(0).function()).isEqualTo(AggregationFunction.COUNT); + assertThat(spec.aggregates().get(0).attribute()).isEqualTo("orderId"); + assertThat(spec.aggregates().get(0).outputName()).isEqualTo("orderCount"); + } + + /** + * Asserts that {@link QueryExpressionBuilder#orderBy(String, SortDirection)} and + * {@link QueryExpressionBuilder#orderByAggregate(String, SortDirection)} add {@link OrderBySpec} entries to the spec. + */ + @Test + public void orderBy_andOrderByAggregate_addOrderBySpecs() { + QueryExpressionSpec spec = QueryExpressionBuilder.from(baseTable) + .orderBy("name", SortDirection.ASC) + .orderByAggregate("orderCount", SortDirection.DESC) + .build(); + + assertThat(spec.orderBy()).hasSize(2); + assertThat(spec.orderBy().get(0).attributeOrAggregateName()).isEqualTo("name"); + assertThat(spec.orderBy().get(0).direction()).isEqualTo(SortDirection.ASC); + assertThat(spec.orderBy().get(0).isByAggregate()).isFalse(); + assertThat(spec.orderBy().get(1).attributeOrAggregateName()).isEqualTo("orderCount"); + assertThat(spec.orderBy().get(1).direction()).isEqualTo(SortDirection.DESC); + assertThat(spec.orderBy().get(1).isByAggregate()).isTrue(); + } + + /** + * Asserts that {@link QueryExpressionBuilder#project(String...)} sets {@link QueryExpressionSpec#projectAttributes()}. + */ + @Test + public void project_setsProjectAttributes() { + QueryExpressionSpec spec = QueryExpressionBuilder.from(baseTable) + .project("customerId", "name", "orderId") + .build(); + + assertThat(spec.projectAttributes()).containsExactly("customerId", "name", "orderId"); + } + + /** + * Asserts that {@link QueryExpressionBuilder#executionMode(ExecutionMode)} sets the spec's execution mode (STRICT_KEY_ONLY or + * ALLOW_SCAN). + */ + @Test + public void executionMode_setsExecutionMode() { + QueryExpressionSpec spec = QueryExpressionBuilder.from(baseTable) + .executionMode(ExecutionMode.ALLOW_SCAN) + .build(); + + assertThat(spec.executionMode()).isEqualTo(ExecutionMode.ALLOW_SCAN); + } + + /** + * Asserts that when execution mode is not set, the default is {@link ExecutionMode#STRICT_KEY_ONLY}. + */ + @Test + public void defaultExecutionMode_isStrictKeyOnly() { + QueryExpressionSpec spec = QueryExpressionBuilder.from(baseTable).build(); + assertThat(spec.executionMode()).isEqualTo(ExecutionMode.STRICT_KEY_ONLY); + } + + /** + * Asserts that {@link QueryExpressionBuilder#limit(int)} sets {@link QueryExpressionSpec#limit()}. + */ + @Test + public void limit_setsLimit() { + QueryExpressionSpec spec = QueryExpressionBuilder.from(baseTable).limit(100).build(); + assertThat(spec.limit()).isEqualTo(100); + } + + /** + * Asserts that a full builder chain produces a spec with all components set. + */ + @Test + public void fullChain_producesCompleteSpec() { + Condition baseCond = Condition.eq("region", "EU"); + Condition joinedCond = Condition.gt("amount", 0); + + QueryExpressionSpec spec = QueryExpressionBuilder.from(baseTable) + .join(joinedTable, JoinType.LEFT, "customerId", "customerId") + .filterBase(baseCond) + .filterJoined(joinedCond) + .groupBy("customerId") + .aggregate(AggregationFunction.COUNT, "orderId", "cnt") + .orderByAggregate("cnt", SortDirection.DESC) + .project("customerId", "name") + .executionMode(ExecutionMode.STRICT_KEY_ONLY) + .limit(10) + .build(); + + assertThat(spec.baseTable()).isSameAs(baseTable); + assertThat(spec.hasJoin()).isTrue(); + assertThat(spec.filterBase()).isSameAs(baseCond); + assertThat(spec.filterJoined()).isSameAs(joinedCond); + assertThat(spec.groupByAttributes()).containsExactly("customerId"); + assertThat(spec.aggregates()).hasSize(1); + assertThat(spec.orderBy()).hasSize(1); + assertThat(spec.projectAttributes()).containsExactly("customerId", "name"); + assertThat(spec.limit()).isEqualTo(10); + } +}