-
Notifications
You must be signed in to change notification settings - Fork 540
Lambda #6123
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Closed
Closed
Lambda #6123
Changes from all commits
Commits
Show all changes
26 commits
Select commit
Hold shift + click to select a range
6bd4931
first stab
fulmicoton-dd 0eb6e28
second stab
fulmicoton-dd b439914
Summary of Changes
fulmicoton-dd 63818cd
File rationalization
fulmicoton-dd 8910340
blop
fulmicoton-dd 984b56c
lambda function validation
fulmicoton-dd fba9519
deployment
fulmicoton-dd 6c5fbf4
rationalizing code
fulmicoton-dd 494a144
blop
fulmicoton-dd 3ac52cd
better config
fulmicoton-dd aa3c16a
better config
fulmicoton-dd bcae2c7
blop
fulmicoton-dd 1e18cd5
blop
fulmicoton-dd aac7014
sever/vclient
fulmicoton-dd 7a4688b
bnlop
fulmicoton-dd 008a121
adding logs
fulmicoton-dd 888e853
blop
fulmicoton-dd 9284059
lambda by default
fulmicoton-dd 748f326
blop
fulmicoton-dd 4cd7cca
deploy versioning, invoke and added metrics.
fulmicoton-dd 93cd5b9
Added unit tests
fulmicoton-dd ff6fdfa
Moving lambda execution to the leaf.
fulmicoton-dd 257d90f
blop
fulmicoton-dd 14a7726
added doc
fulmicoton-dd 0d1afe7
blop
fulmicoton-dd c426937
Removed useless errors
fulmicoton-dd File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,78 @@ | ||
| # This workflow creates a new release for a quickwit search aws lambda. | ||
| # The artifact is a zip file containing a binary for ARM 64, | ||
| # ready to be deployed by the deployer. | ||
| # | ||
| # See quickwit-lambda-client/README.md | ||
| name: Release Lambda binary | ||
|
|
||
| on: | ||
| push: | ||
| tags: | ||
| - 'lambda-*' | ||
| workflow_dispatch: | ||
| inputs: | ||
| version: | ||
| description: 'Version tag (e.g., v0.8.0)' | ||
| required: false | ||
| default: 'dev' | ||
|
|
||
| permissions: | ||
| contents: read | ||
|
|
||
| jobs: | ||
| build-lambda: | ||
| name: Build Lambda ARM64 | ||
| runs-on: ubuntu-latest | ||
| permissions: | ||
| contents: write | ||
| actions: write | ||
| steps: | ||
| - uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 # v6.0.1 | ||
|
|
||
| - name: Set version | ||
| run: | | ||
| if [ "${{ github.ref_type }}" = "tag" ]; then | ||
| # Extract version from tag (e.g., lambda-v0.8.0 -> v0.8.0) | ||
| echo "ASSET_VERSION=${GITHUB_REF_NAME#lambda-}" >> $GITHUB_ENV | ||
| elif [ -n "${{ github.event.inputs.version }}" ] && [ "${{ github.event.inputs.version }}" != "dev" ]; then | ||
| echo "ASSET_VERSION=${{ github.event.inputs.version }}" >> $GITHUB_ENV | ||
| else | ||
| echo "ASSET_VERSION=dev-$(git rev-parse --short HEAD)" >> $GITHUB_ENV | ||
| fi | ||
|
|
||
| - name: Install rustup | ||
| run: curl https://sh.rustup.rs -sSf | sh -s -- --default-toolchain none -y | ||
|
|
||
| - name: Install cross | ||
| run: cargo install cross | ||
|
|
||
| - name: Retrieve and export commit date, hash, and tags | ||
| run: | | ||
| echo "QW_COMMIT_DATE=$(TZ=UTC0 git log -1 --format=%cd --date=format-local:%Y-%m-%dT%H:%M:%SZ)" >> $GITHUB_ENV | ||
| echo "QW_COMMIT_HASH=$(git rev-parse HEAD)" >> $GITHUB_ENV | ||
| echo "QW_COMMIT_TAGS=$(git tag --points-at HEAD | tr '\n' ',')" >> $GITHUB_ENV | ||
|
|
||
| - name: Build Lambda binary | ||
| run: cross build --release --features lambda-release --target aarch64-unknown-linux-gnu -p quickwit-lambda-server --bin quickwit-aws-lambda-leaf-search | ||
| env: | ||
| QW_COMMIT_DATE: ${{ env.QW_COMMIT_DATE }} | ||
| QW_COMMIT_HASH: ${{ env.QW_COMMIT_HASH }} | ||
| QW_COMMIT_TAGS: ${{ env.QW_COMMIT_TAGS }} | ||
| working-directory: ./quickwit | ||
|
|
||
| - name: Create Lambda zip | ||
| run: | | ||
| cd quickwit/target/aarch64-unknown-linux-gnu/release | ||
| cp quickwit-aws-lambda-leaf-search bootstrap | ||
| zip quickwit-aws-lambda-${{ env.ASSET_VERSION }}-aarch64.zip bootstrap | ||
| mv quickwit-aws-lambda-${{ env.ASSET_VERSION }}-aarch64.zip ../../../../ | ||
|
|
||
| - name: Upload to GitHub release | ||
| uses: quickwit-inc/upload-to-github-release@v1 | ||
| env: | ||
| GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} | ||
| with: | ||
| file: quickwit-aws-lambda-${{ env.ASSET_VERSION }}-aarch64.zip | ||
| overwrite: true | ||
| draft: true | ||
| tag_name: ${{ env.ASSET_VERSION }} | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,225 @@ | ||
| --- | ||
| title: Lambda configuration | ||
| sidebar_position: 6 | ||
| --- | ||
|
|
||
| Quickwit supports offloading leaf search operations to AWS Lambda for horizontal scaling. When the local search queue becomes saturated, overflow splits are automatically sent to Lambda functions for processing. | ||
|
|
||
| :::note | ||
| Lambda offloading is currently only supported on AWS. | ||
| ::: | ||
|
|
||
| ## How it works | ||
|
|
||
| Lambda offloading is **only active when a `lambda` configuration section is present** under `searcher` in your node configuration. When configured: | ||
|
|
||
| 1. Quickwit monitors the local search queue depth | ||
| 2. When pending searches exceed the `offload_threshold`, new splits are sent to Lambda instead of being queued locally | ||
| 3. Lambda returns per-split search results that are cached and merged with local results | ||
|
|
||
| This allows Quickwit to handle traffic spikes without provisioning additional searcher nodes. | ||
|
|
||
| ## Startup validation | ||
|
|
||
| When a `lambda` configuration is defined, Quickwit performs a **dry run invocation** at startup to verify that: | ||
| - The Lambda function exists | ||
| - The function version matches the embedded binary | ||
| - The invoker has permission to call the function | ||
|
|
||
| If this validation fails, **Quickwit will fail to start**. This ensures that Lambda offloading works correctly before the node begins serving traffic. | ||
|
|
||
| ## Configuration | ||
|
|
||
| Add a `lambda` section under `searcher` in your node configuration: | ||
|
|
||
| ```yaml | ||
| searcher: | ||
| lambda: | ||
| offload_threshold: 100 | ||
| auto_deploy: | ||
| execution_role_arn: arn:aws:iam::123456789012:role/quickwit-lambda-role | ||
| memory_size: 5 GiB | ||
| invocation_timeout_secs: 15 | ||
| ``` | ||
|
|
||
| ### Lambda configuration options | ||
|
|
||
| | Property | Description | Default value | | ||
| | --- | --- | --- | | ||
| | `function_name` | Name of the AWS Lambda function to invoke. | `quickwit-lambda-search` | | ||
| | `max_splits_per_invocation` | Maximum number of splits to send in a single Lambda invocation. Must be at least 1. | `10` | | ||
| | `offload_threshold` | Number of pending local searches before offloading to Lambda. A value of `0` offloads everything to Lambda. | `100` | | ||
| | `auto_deploy` | Auto-deployment configuration. If set, Quickwit automatically deploys or updates the Lambda function at startup. | (none) | | ||
|
|
||
| ### Auto-deploy configuration options | ||
|
|
||
| | Property | Description | Default value | | ||
| | --- | --- | --- | | ||
| | `execution_role_arn` | **Required.** IAM role ARN for the Lambda function's execution role. | | | ||
| | `memory_size` | Memory allocated to the Lambda function. More memory provides more CPU. | `5 GiB` | | ||
| | `invocation_timeout_secs` | Timeout for Lambda invocations in seconds. | `15` | | ||
|
|
||
| ## Deployment options | ||
|
|
||
| ### Automatic deployment (recommended) | ||
|
|
||
| With `auto_deploy` configured, Quickwit automatically: | ||
| 1. Creates the Lambda function if it doesn't exist | ||
| 2. Updates the function code if the embedded binary has changed | ||
| 3. Publishes a new version with a unique identifier | ||
| 4. Garbage collects old versions (keeps current + 5 most recent) | ||
|
|
||
| This is the recommended approach as it ensures the Lambda function always matches the Quickwit binary version. | ||
|
|
||
| ### Manual deployment | ||
|
|
||
| You can deploy the Lambda function manually without `auto_deploy`: | ||
| 1. Download the Lambda zip from [GitHub releases](https://github.com/quickwit-oss/quickwit/releases) | ||
| 2. Create or update the Lambda function using AWS CLI, Terraform, or the AWS Console | ||
| 3. Publish a version with description format `quickwit:{version}-{sha1}` (e.g., `quickwit:0_8_0-fa752891`) | ||
|
|
||
| The description must match the format Quickwit expects, or it won't find the function version. | ||
|
|
||
| ## IAM permissions | ||
|
|
||
| ### Permissions for the Quickwit node | ||
|
|
||
| The IAM role or user running Quickwit needs the following permissions to invoke Lambda: | ||
|
|
||
| ```json | ||
| { | ||
| "Version": "2012-10-17", | ||
| "Statement": [ | ||
| { | ||
| "Effect": "Allow", | ||
| "Action": [ | ||
| "lambda:InvokeFunction" | ||
| ], | ||
| "Resource": "arn:aws:lambda:*:*:function:quickwit-lambda-search:*" | ||
| } | ||
| ] | ||
| } | ||
| ``` | ||
|
|
||
| If using `auto_deploy`, additional permissions are required for deployment: | ||
|
|
||
| ```json | ||
| { | ||
| "Version": "2012-10-17", | ||
| "Statement": [ | ||
| { | ||
| "Effect": "Allow", | ||
| "Action": [ | ||
| "lambda:CreateFunction", | ||
| "lambda:GetFunction", | ||
| "lambda:UpdateFunctionCode", | ||
| "lambda:PublishVersion", | ||
| "lambda:ListVersionsByFunction", | ||
| "lambda:DeleteFunction" | ||
| ], | ||
| "Resource": "arn:aws:lambda:*:*:function:quickwit-lambda-search" | ||
| }, | ||
| { | ||
| "Effect": "Allow", | ||
| "Action": "iam:PassRole", | ||
| "Resource": "arn:aws:iam::*:role/quickwit-lambda-role", | ||
| "Condition": { | ||
| "StringEquals": { | ||
| "iam:PassedToService": "lambda.amazonaws.com" | ||
| } | ||
| } | ||
| } | ||
| ] | ||
| } | ||
| ``` | ||
|
|
||
| ### Lambda execution role | ||
|
|
||
| The Lambda function requires an execution role with S3 read access to your index data. CloudWatch logging permissions are not required. | ||
|
|
||
| Example policy: | ||
|
|
||
| ```json | ||
| { | ||
| "Version": "2012-10-17", | ||
| "Statement": [ | ||
| { | ||
| "Effect": "Allow", | ||
| "Action": "s3:GetObject", | ||
| "Resource": "arn:aws:s3:::your-index-bucket/*" | ||
| } | ||
| ] | ||
| } | ||
| ``` | ||
|
|
||
| The execution role must also have a trust policy allowing Lambda to assume it: | ||
|
|
||
| ```json | ||
| { | ||
| "Version": "2012-10-17", | ||
| "Statement": [ | ||
| { | ||
| "Effect": "Allow", | ||
| "Principal": { | ||
| "Service": "lambda.amazonaws.com" | ||
| }, | ||
| "Action": "sts:AssumeRole" | ||
| } | ||
| ] | ||
| } | ||
| ``` | ||
|
|
||
| ## Versioning | ||
|
|
||
| Quickwit uses content-based versioning for Lambda: | ||
| - A SHA1 hash of the Lambda binary is computed at build time | ||
| - This hash is embedded in the Lambda function description as `quickwit:{version}-{sha1_short}` | ||
| - When Quickwit starts, it searches for a version matching this description | ||
| - Different Quickwit builds with the same Lambda binary share the same Lambda version | ||
| - Updating the Lambda binary automatically triggers a new deployment | ||
|
|
||
| ## Example configuration | ||
|
|
||
| Minimal configuration (manual deployment): | ||
|
|
||
| ```yaml | ||
| searcher: | ||
| lambda: | ||
| function_name: {} | ||
| ``` | ||
|
|
||
|
|
||
| Minimal configuration (auto deployment): | ||
|
|
||
| ```yaml | ||
| searcher: | ||
| lambda: | ||
| auto_deploy: | ||
| execution_role_arn: arn:aws:iam::123456789012:role/quickwit-lambda-role | ||
| ``` | ||
|
|
||
|
|
||
| Full configuration (auto-deployment): | ||
|
|
||
| ```yaml | ||
| searcher: | ||
| lambda: | ||
| function_name: quickwit-lambda-search | ||
| max_splits_per_invocation: 10 | ||
| offload_threshold: 100 | ||
| auto_deploy: | ||
| execution_role_arn: arn:aws:iam::123456789012:role/quickwit-lambda-role | ||
| memory_size: 5 GiB | ||
| invocation_timeout_secs: 15 | ||
| ``` | ||
|
|
||
| Aggressive offloading (send everything to Lambda): | ||
|
|
||
| ```yaml | ||
| searcher: | ||
| lambda: | ||
| function_name: quickwit-lambda-search | ||
| offload_threshold: 0 | ||
| auto_deploy: | ||
| execution_role_arn: arn:aws:iam::123456789012:role/quickwit-lambda-role | ||
| ``` |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,16 @@ | ||
| # Quickwit Claude Guidelines | ||
|
|
||
| ## Code Formatting | ||
|
|
||
| Run `make fmt` to check and fix code formatting. This command performs three checks: | ||
|
|
||
| 1. **Rust formatting**: Ensures Rust code is properly formatted (via `cargo fmt`) | ||
| 2. **License headers**: Checks that files are prepended with the correct LICENSE header | ||
| 3. **Log format policy**: Checks that log statements follow our format rules: | ||
| - No trailing punctuation in log messages | ||
| - No uppercase for the first character of log messages | ||
| - See `scripts/check_log_format.sh` for details | ||
|
|
||
| ### Quick Fix | ||
|
|
||
| Use `/fmt` to automatically run format checks and see issues. |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Check warning
Code scanning / CodeQL
Unpinned tag for a non-immutable Action in workflow Medium