How to install python package from GCP Artifact Registry or private PyPI during adk deploy agent_engine?

[wojciech-mazur-mlg]

Hey,

My team is working on AI agent deployed to agent_engine, which will be accessible from Gemini Enterprise.

We need to install one python dependency which is a private package. It's stored in gitlab private PyPI, we can also deploy it to GCP Artifact Registry PyPI. This package is provided in requirements.txt file, the same as other packages.

Is there a way to install such a package during building of a docker image via adk deploy agent_engine? I tried multiple solutions, but no success.

What I've tried:

1. Deploying package to Artifact Registry:

• I've added --extra-index-url=https://europe-west2-python.pkg.dev/<project_name>/<pypi-name>/simple/ to requirements.txt
• I've added Artifact Registry Reader role to Compute Engine default service account (which I believe is responsible for building an image). Added this role also for AI Platform Service Agent.
• Image building is failing because it's expecting interactive user input with user/pass for my private PyPI.

What I expect here is google uses default credentials provided by keyring (we need a python package keyrings.google-artifactregistry-auth to be installed).
It looks like it's not installed in the build env or maybe I'm still missing some IAM permission.

2. Package deployed to private gitlab PyPI.

• I've added --extra-index-url=https://<token-name>:<token>@gitlab.com/api/v4/projects/<gitlab_project_id>/packages/pypi/simple
• This is the only way I'm able to install package, but saving token explicitly in requirements.txt is definitely a very bad idea.
• I'd like to provide token via env variable, but it looks like it's not expanded in requirements.txt
• I've tried providing env variable PIP_EXTRA_INDEX_URL, but it looks like it's not passed to the container building process
• I've tried providing pip.conf together with PIP_CONFIG_FILE env var, but it didn't work.

It looks like env variabled are not passed to the building process.
Is there a way to solve this either with option 1 or 2?

Thanks for any help.

[adk-bot]

Response from ADK Answering Agent (experimental, answer may be inaccurate)

TLDR: The adk deploy agent_engine command has a restricted build environment that makes it difficult to authenticate with private package repositories. For more control over the deployment process, we recommend using adk deploy cloud_run with a custom Dockerfile, which will allow you to manage private dependencies from GCP Artifact Registry or a private GitLab PyPI.

---

Hello! Thank you for your detailed question. It highlights an important aspect of deploying agents with private dependencies.

The issue you're facing stems from the fact that adk deploy agent_engine uses a managed, source-based deployment process. [14] This process is optimized for simplicity and doesn't provide the flexibility to customize the build environment, such as authenticating with private package repositories.

To resolve this, we recommend deploying your agent to Cloud Run using the adk deploy cloud_run command. [9] This approach gives you full control over the build process through a custom Dockerfile.

Here’s how you can solve this with adk deploy cloud_run and a Dockerfile:

Option 1: Using GCP Artifact Registry

You can create a Dockerfile that first installs the necessary authentication library and then installs your dependencies. This avoids the need for interactive authentication during the build.

# Use a Python base image
FROM python:3.10-slim

# Set up the working directory
WORKDIR /app

# Install the Artifact Registry keyring for authentication
RUN pip install keyrings.google-artifactregistry-auth

# Copy your requirements.txt file
COPY requirements.txt requirements.txt

# Install the Python dependencies
# The --extra-index-url should be in your requirements.txt file
RUN pip install -r requirements.txt

# Copy the rest of your agent's code
COPY . .

# Command to run your agent
CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8080"]

In your requirements.txt, you would have:

--extra-index-url=https://europe-west2-python.pkg.dev/<project_name>/<pypi-name>/simple/
your-private-package
google-adk
# other dependencies...

Option 2: Using a Private GitLab PyPI

To avoid exposing your token in the requirements.txt file, you can use Docker build-time arguments (--build-arg) to pass the token securely.

Your Dockerfile would look like this:

# Use a Python base image
FROM python:3.10-slim

# Set up the working directory
WORKDIR /app

# Argument for the GitLab token
ARG GITLAB_TOKEN

# Copy your requirements.txt file
COPY requirements.txt requirements.txt

# Install the Python dependencies using the token
RUN pip install --extra-index-url=https://<token-name>:${GITLAB_TOKEN}@gitlab.com/api/v4/projects/<gitlab_project_id>/packages/pypi/simple -r requirements.txt

# Copy the rest of your agent's code
COPY . .

# Command to run your agent
CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8080"]

You would then build and deploy using the gcloud run deploy command with the --build-arg flag:

gcloud run deploy your-agent-service \
  --source . \
  --build-arg GITLAB_TOKEN=<your-gitlab-token> \
  --project <your-project-id> \
  --region <your-region> \
  --allow-unauthenticated

By using adk deploy cloud_run or gcloud run deploy with a custom Dockerfile, you gain the necessary control to handle authentication with private package repositories, which is not possible with the managed build environment of adk deploy agent_engine.

We hope this helps you and your team move forward with your project!

[wojciech-mazur-mlg]

Dear @adk-bot I'd love to deploy it to Cloud Run instead of Agent Engine, but unfortunately I don't know the way to integrate agent deployed to Cloud Run with Gemini Enterprise. Is there a way to do it?

[adk-bot]

Response from ADK Answering Agent (experimental, answer may be inaccurate)

TLDR: The adk deploy agent_engine command uses a source-based deployment, not a Docker container build. This means you cannot customize the dependency installation environment. The recommended workaround is to include your private package's source code directly in your agent project folder (a practice known as "vendoring").

---

Hi there!

Thanks for the detailed breakdown of the issue and the solutions you've already tried. The core of the issue you're facing stems from the specific deployment mechanism used by adk deploy agent_engine.

Unlike adk deploy cloud_run or gke, which build a Docker container from a Dockerfile, the agent_engine target uses a source-based deployment. As noted in the documentation, this process packages your agent's source code and requirements.txt into a tarball and deploys it directly to a managed Vertex AI service. [39] There is no Docker build step that you can customize.

This explains why your attempts didn't work as expected:

1. GCP Artifact Registry: The managed environment where your dependencies are installed does not have the keyrings.google-artifactregistry-auth package or the necessary gcloud context to authenticate with Artifact Registry on your behalf. This leads to the interactive prompt you observed.
2. Environment Variables: Since there is no build process that you control, environment variables like PIP_EXTRA_INDEX_URL are not passed to the pip install command running in the managed service.

Recommended Workaround: Vendor the Dependency

The most effective way to resolve this is to include the private package's source code directly within your agent project. This way, the code is part of the source tarball uploaded during deployment, and pip doesn't need to fetch it from a private repository.

Here’s how you can do it:

1. Copy the Package: Create a directory in your agent project (e.g., vendor/) and copy your private package's source code into it.
2. Ensure it's a Package: Make sure the copied directory is a valid Python package (i.e., it contains an __init__.py file).
3. Update requirements.txt: Remove the line for your private package from your requirements.txt file.
4. Update Imports: In your agent code, change the import statements to reference the local copy. For example, if you copied my-private-package into vendor/my-private-package, you might import it with from vendor.my-private-package import ....

By following this approach, you can deploy your agent to Agent Engine with all its dependencies, without needing to solve the authentication challenge in the managed deployment environment.

Hope this helps!