Skip to content

Commit fbdcd32

Browse files
vikrantpuppalavikrantpuppala
authored
Migrate CI to protected runners and JFrog PyPI proxy (#770)
* Migrate CI to databricks-protected runners and route PyPI through JFrog Protected runners are required for Databricks OSS repos. Add a setup-jfrog composite action (OIDC-based, matching databricks-odbc) that sets PIP_INDEX_URL so all pip/poetry installs go through the JFrog PyPI proxy. Every workflow now runs on the databricks-protected-runner-group with id-token: write for the OIDC exchange. Co-authored-by: Isaac Signed-off-by: Vikrant Puppala <vikrant.puppala@databricks.com> * Add Poetry JFrog source configuration to all workflows The previous commit only set PIP_INDEX_URL, but Poetry uses its own resolver and needs explicit source configuration. Add a "Configure Poetry for JFrog" step after poetry install in every job that sets up the JFrog repository and credentials, then adds it as the primary source for the project. Co-authored-by: Isaac Signed-off-by: Vikrant Puppala <vikrant.puppala@databricks.com> * Fix step ordering: move JFrog setup after poetry install The snok/install-poetry action uses pip internally to install poetry. When PIP_INDEX_URL was set before this step, the installer tried to route through JFrog and failed with an SSL error. Move the JFrog OIDC token + PIP_INDEX_URL + poetry source configuration to run after Install Poetry but before poetry install. Co-authored-by: Isaac Signed-off-by: Vikrant Puppala <vikrant.puppala@databricks.com> * Replace snok/install-poetry with pip install through JFrog The hardened runners block direct access to install.python-poetry.org, causing snok/install-poetry to fail with SSL errors. Replace it with `pip install poetry==2.2.1` which routes through the JFrog PyPI proxy. New step ordering: checkout → setup-python → Setup JFrog (OIDC + PIP_INDEX_URL) → pip install poetry → Configure Poetry for JFrog → poetry install. Co-authored-by: Isaac Signed-off-by: Vikrant Puppala <vikrant.puppala@databricks.com> * Add poetry lock --no-update after source add to fix lock mismatch poetry source add modifies pyproject.toml, which makes poetry refuse to install from the existing lock file. Running poetry lock --no-update regenerates the lock file metadata without changing dependency versions. Co-authored-by: Isaac Signed-off-by: Vikrant Puppala <vikrant.puppala@databricks.com> * Fix poetry lock flag and YAML indentation Poetry 2.x doesn't have --no-update flag, use poetry lock instead. Also fix indentation of poetry lock in the arrow test job. Co-authored-by: Isaac Signed-off-by: Vikrant Puppala <vikrant.puppala@databricks.com> * Move JFrog setup before setup-python, matching sqlalchemy pattern Follow the proven pattern from databricks/databricks-sqlalchemy#59: checkout → Setup JFrog → setup-python → pip install poetry → poetry source add + poetry lock → poetry install. The hardened runners block pypi.org at the network level, so JFrog must be configured before actions/setup-python (which upgrades pip). Also simplified workflows by removing verbose section comments. Co-authored-by: Isaac Signed-off-by: Vikrant Puppala <vikrant.puppala@databricks.com> * Extract setup-poetry composite action to remove duplication Create .github/actions/setup-poetry that bundles JFrog setup, setup-python, poetry install via pip, JFrog source config, cache, and dependency install into a single reusable action with inputs for python-version, install-args, cache-path, and cache-suffix. All workflows now call setup-poetry instead of repeating these steps, matching the pattern from databricks/databricks-sqlalchemy#59. Co-authored-by: Isaac Signed-off-by: Vikrant Puppala <vikrant.puppala@databricks.com> --------- Signed-off-by: Vikrant Puppala <vikrant.puppala@databricks.com>
1 parent e056275 commit fbdcd32

File tree

7 files changed

+196
-394
lines changed

7 files changed

+196
-394
lines changed

‎.github/actions/setup-jfrog/action.yml‎

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,32 @@
1+
name: Setup JFrog OIDC
2+
description: Obtain a JFrog access token via GitHub OIDC and configure pip to use JFrog PyPI proxy
3+
4+
runs:
5+
using: composite
6+
steps:
7+
- name: Get JFrog OIDC token
8+
shell: bash
9+
run: |
10+
set -euo pipefail
11+
ID_TOKEN=$(curl -sLS \
12+
-H "User-Agent: actions/oidc-client" \
13+
-H "Authorization: Bearer $ACTIONS_ID_TOKEN_REQUEST_TOKEN" \
14+
"${ACTIONS_ID_TOKEN_REQUEST_URL}&audience=jfrog-github" | jq .value | tr -d '"')
15+
echo "::add-mask::${ID_TOKEN}"
16+
ACCESS_TOKEN=$(curl -sLS -XPOST -H "Content-Type: application/json" \
17+
"https://databricks.jfrog.io/access/api/v1/oidc/token" \
18+
-d "{\"grant_type\": \"urn:ietf:params:oauth:grant-type:token-exchange\", \"subject_token_type\":\"urn:ietf:params:oauth:token-type:id_token\", \"subject_token\": \"${ID_TOKEN}\", \"provider_name\": \"github-actions\"}" | jq .access_token | tr -d '"')
19+
echo "::add-mask::${ACCESS_TOKEN}"
20+
if [ -z "$ACCESS_TOKEN" ] || [ "$ACCESS_TOKEN" = "null" ]; then
21+
echo "FAIL: Could not extract JFrog access token"
22+
exit 1
23+
fi
24+
echo "JFROG_ACCESS_TOKEN=${ACCESS_TOKEN}" >> "$GITHUB_ENV"
25+
echo "JFrog OIDC token obtained successfully"
26+
27+
- name: Configure pip
28+
shell: bash
29+
run: |
30+
set -euo pipefail
31+
echo "PIP_INDEX_URL=https://gha-service-account:${JFROG_ACCESS_TOKEN}@databricks.jfrog.io/artifactory/api/pypi/db-pypi/simple" >> "$GITHUB_ENV"
32+
echo "pip configured to use JFrog registry"

‎.github/actions/setup-poetry/action.yml‎

Lines changed: 63 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,63 @@
1+
name: Setup Poetry with JFrog
2+
description: Install Poetry, configure JFrog as primary PyPI source, and install project dependencies
3+
4+
inputs:
5+
python-version:
6+
description: Python version to set up
7+
required: true
8+
install-args:
9+
description: Extra arguments for poetry install (e.g. --all-extras)
10+
required: false
11+
default: ""
12+
cache-path:
13+
description: Path to the virtualenv for caching (e.g. .venv or .venv-pyarrow)
14+
required: false
15+
default: ".venv"
16+
cache-suffix:
17+
description: Extra suffix for the cache key to avoid collisions across job variants
18+
required: false
19+
default: ""
20+
21+
runs:
22+
using: composite
23+
steps:
24+
- name: Setup JFrog
25+
uses: ./.github/actions/setup-jfrog
26+
27+
- name: Set up python ${{ inputs.python-version }}
28+
id: setup-python
29+
uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5
30+
with:
31+
python-version: ${{ inputs.python-version }}
32+
33+
- name: Install Poetry
34+
shell: bash
35+
run: |
36+
pip install poetry==2.2.1
37+
poetry config virtualenvs.create true
38+
poetry config virtualenvs.in-project true
39+
poetry config installer.parallel true
40+
41+
- name: Configure Poetry JFrog source
42+
shell: bash
43+
run: |
44+
poetry config repositories.jfrog https://databricks.jfrog.io/artifactory/api/pypi/db-pypi/simple
45+
poetry config http-basic.jfrog gha-service-account "${JFROG_ACCESS_TOKEN}"
46+
poetry source add --priority=primary jfrog https://databricks.jfrog.io/artifactory/api/pypi/db-pypi/simple
47+
poetry lock
48+
49+
- name: Load cached venv
50+
id: cached-poetry-dependencies
51+
uses: actions/cache@0057852bfaa89a56745cba8c7296529d2fc39830 # v4
52+
with:
53+
path: ${{ inputs.cache-path }}
54+
key: venv-${{ runner.os }}-${{ steps.setup-python.outputs.python-version }}-${{ inputs.cache-suffix }}${{ github.event.repository.name }}-${{ hashFiles('**/poetry.lock') }}
55+
56+
- name: Install dependencies
57+
if: steps.cached-poetry-dependencies.outputs.cache-hit != 'true'
58+
shell: bash
59+
run: poetry install --no-interaction --no-root
60+
61+
- name: Install library
62+
shell: bash
63+
run: poetry install --no-interaction ${{ inputs.install-args }}

‎.github/workflows/code-coverage.yml‎

Lines changed: 10 additions & 74 deletions
Original file line numberDiff line numberDiff line change
@@ -2,12 +2,15 @@ name: Code Coverage
22

33
permissions:
44
contents: read
5+
id-token: write
56

67
on: [pull_request, workflow_dispatch]
78

89
jobs:
910
test-with-coverage:
10-
runs-on: ubuntu-latest
11+
runs-on:
12+
group: databricks-protected-runner-group
13+
labels: linux-ubuntu-latest
1114
environment: azure-prod
1215
env:
1316
DATABRICKS_SERVER_HOSTNAME: ${{ secrets.DATABRICKS_HOST }}
@@ -16,63 +19,19 @@ jobs:
1619
DATABRICKS_CATALOG: peco
1720
DATABRICKS_USER: ${{ secrets.TEST_PECO_SP_ID }}
1821
steps:
19-
#----------------------------------------------
20-
# check-out repo and set-up python
21-
#----------------------------------------------
2222
- name: Check out repository
2323
uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
2424
with:
2525
fetch-depth: 0
26-
- name: Set up python
27-
id: setup-python
28-
uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5
29-
with:
30-
python-version: "3.10"
31-
#----------------------------------------------
32-
# ----- install system dependencies -----
33-
#----------------------------------------------
3426
- name: Install system dependencies
3527
run: |
3628
sudo apt-get update
3729
sudo apt-get install -y libkrb5-dev
38-
#----------------------------------------------
39-
# ----- install & configure poetry -----
40-
#----------------------------------------------
41-
- name: Install Poetry
42-
uses: snok/install-poetry@76e04a911780d5b312d89783f7b1cd627778900a # v1
30+
- name: Setup Poetry
31+
uses: ./.github/actions/setup-poetry
4332
with:
44-
version: "2.2.1"
45-
virtualenvs-create: true
46-
virtualenvs-in-project: true
47-
installer-parallel: true
48-
49-
#----------------------------------------------
50-
# load cached venv if cache exists
51-
#----------------------------------------------
52-
- name: Load cached venv
53-
id: cached-poetry-dependencies
54-
uses: actions/cache@0057852bfaa89a56745cba8c7296529d2fc39830 # v4
55-
with:
56-
path: .venv
57-
key: venv-${{ runner.os }}-${{ steps.setup-python.outputs.python-version }}-${{ github.event.repository.name }}-${{ hashFiles('**/poetry.lock') }}
58-
#----------------------------------------------
59-
# install dependencies if cache does not exist
60-
#----------------------------------------------
61-
- name: Install dependencies
62-
if: steps.cached-poetry-dependencies.outputs.cache-hit != 'true'
63-
run: poetry install --no-interaction --no-root
64-
#----------------------------------------------
65-
# install your root project, if required
66-
#----------------------------------------------
67-
- name: Install Kerberos system dependencies
68-
run: |
69-
sudo apt-get update
70-
sudo apt-get install -y libkrb5-dev
71-
- name: Install library
72-
run: poetry install --no-interaction --all-extras
73-
#----------------------------------------------
74-
# run parallel tests with coverage
75-
#----------------------------------------------
33+
python-version: "3.10"
34+
install-args: "--all-extras"
7635
- name: Run parallel tests with coverage
7736
continue-on-error: false
7837
run: |
@@ -83,24 +42,15 @@ jobs:
8342
--cov-report=xml \
8443
--cov-report=term \
8544
-v
86-
87-
#----------------------------------------------
88-
# run telemetry tests with coverage (isolated)
89-
#----------------------------------------------
9045
- name: Run telemetry tests with coverage (isolated)
9146
continue-on-error: false
9247
run: |
93-
# Run test_concurrent_telemetry.py separately for isolation
9448
poetry run pytest tests/e2e/test_concurrent_telemetry.py \
9549
--cov=src \
9650
--cov-append \
9751
--cov-report=xml \
9852
--cov-report=term \
9953
-v
100-
101-
#----------------------------------------------
102-
# check for coverage override
103-
#----------------------------------------------
10454
- name: Check for coverage override
10555
id: override
10656
env:
@@ -116,9 +66,6 @@ jobs:
11666
echo "override=false" >> $GITHUB_OUTPUT
11767
echo "No coverage override found"
11868
fi
119-
#----------------------------------------------
120-
# check coverage percentage
121-
#----------------------------------------------
12269
- name: Check coverage percentage
12370
if: steps.override.outputs.override == 'false'
12471
run: |
@@ -127,40 +74,29 @@ jobs:
12774
echo "ERROR: Coverage file not found at $COVERAGE_FILE"
12875
exit 1
12976
fi
130-
131-
# Install xmllint if not available
13277
if ! command -v xmllint &> /dev/null; then
13378
sudo apt-get update && sudo apt-get install -y libxml2-utils
13479
fi
135-
13680
COVERED=$(xmllint --xpath "string(//coverage/@lines-covered)" "$COVERAGE_FILE")
13781
TOTAL=$(xmllint --xpath "string(//coverage/@lines-valid)" "$COVERAGE_FILE")
13882
PERCENTAGE=$(python3 -c "covered=${COVERED}; total=${TOTAL}; print(round((covered/total)*100, 2))")
139-
14083
echo "Branch Coverage: $PERCENTAGE%"
14184
echo "Required Coverage: 85%"
142-
143-
# Use Python to compare the coverage with 85
14485
python3 -c "import sys; sys.exit(0 if float('$PERCENTAGE') >= 85 else 1)"
14586
if [ $? -eq 1 ]; then
14687
echo "ERROR: Coverage is $PERCENTAGE%, which is less than the required 85%"
14788
exit 1
14889
else
14990
echo "SUCCESS: Coverage is $PERCENTAGE%, which meets the required 85%"
15091
fi
151-
152-
#----------------------------------------------
153-
# coverage enforcement summary
154-
#----------------------------------------------
15592
- name: Coverage enforcement summary
15693
env:
15794
OVERRIDE: ${{ steps.override.outputs.override }}
15895
REASON: ${{ steps.override.outputs.reason }}
15996
run: |
16097
if [ "$OVERRIDE" == "true" ]; then
161-
echo "⚠️ Coverage checks bypassed: $REASON"
98+
echo "Coverage checks bypassed: $REASON"
16299
echo "Please ensure this override is justified and temporary"
163100
else
164-
echo "✅ Coverage checks enforced - minimum 85% required"
101+
echo "Coverage checks enforced - minimum 85% required"
165102
fi
166-

0 commit comments

Comments
 (0)