Skip to content

Commit da1d2cc

Browse files
vhvb1989Copilot
vhvb1989
and
Copilot
committed
Update investigation doc with 2024-10-01 API test results
Tested checkPolicyRestrictions with api-version=2024-10-01 against a subscription with a tenant-root-MG deny policy (Storage Accounts - Safe Secrets Standard). Findings: - Subscription-scope 2024-10-01: still does not see MG-inherited policies - MG-scope endpoint: only supports 'type' pending field, rejects resourceDetails — cannot evaluate property-level restrictions - includeAuditEffect parameter: no change in results Updated Approach 2 section with detailed 2024-10-01 follow-up, added per-endpoint comparison table, and revised conclusion table and Future Considerations to reflect that the newer API does not resolve the limitation. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
1 parent de81ee9 commit da1d2cc

4 files changed

Lines changed: 1684 additions & 25 deletions

File tree

‎cli/azd/docs/design/local-preflight-validation.md‎

Lines changed: 226 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,226 @@
1+
# Local Preflight Validation
2+
3+
Local preflight validation is a client-side check that runs automatically before every `azd provision` deployment. It analyzes the compiled ARM template and a Bicep deployment snapshot to detect common issues — such as missing permissions.
4+
5+
## When It Runs
6+
7+
The preflight pipeline executes inside `BicepProvider.Deploy()`, after the Bicep module has been compiled and parameters resolved, but **before** the template is sent to Azure for server-side validation or deployment.
8+
9+
```
10+
azd provision
11+
│
12+
├── Compile Bicep module → ARM template + parameters
13+
├── ► Local preflight validation ← runs here
14+
│ ├── Parse ARM template (schema, contentVersion, resources)
15+
│ ├── Generate Bicep snapshot (resolved resource graph)
16+
│ ├── Analyze resources (derive properties)
17+
│ └── Run registered check functions
18+
├── Server-side preflight (Azure ValidatePreflight API)
19+
└── Deploy
20+
```
21+
22+
The user can disable preflight entirely by setting `provision.preflight` to `"off"` in their azd user configuration:
23+
24+
```bash
25+
azd config set provision.preflight off
26+
```
27+
28+
## Bicep Snapshots
29+
30+
Local preflight depends on the `bicep snapshot` command (available in modern Bicep CLI versions). The snapshot produces a **fully resolved deployment graph**: all template expressions are evaluated, conditions are applied, copy loops are expanded, and nested deployments are flattened into a single flat list of predicted resources.
31+
32+
### Why Snapshots Instead of Manual Parsing
33+
34+
An ARM template as compiled by `bicep build` still contains unresolved expressions like `"[parameters('location')]"`, conditional resources, and nested deployment modules. Manually parsing these would require reimplementing the ARM expression evaluator. The Bicep snapshot command does this natively and returns the final, concrete set of resources that would be deployed.
35+
36+
Advantages of snapshots over manual template parsing:
37+
38+
| Aspect | Manual Parsing | Bicep Snapshot |
39+
|---|---|---|
40+
| Expression resolution | Not possible (e.g. `[concat(...)]`) | Fully resolved |
41+
| Nested deployments | Must recursively extract | Flattened automatically |
42+
| Conditional resources | Cannot evaluate `condition` expressions | Excluded when false |
43+
| Copy loops | Cannot expand `copy` blocks | Expanded to individual resources |
44+
| Resource IDs | Symbolic names only | Resolved resource IDs |
45+
46+
### How Snapshots Are Generated
47+
48+
1. **Determine the parameters file.** If the module path is a `.bicepparam` file, it is used directly. Otherwise, a temporary `.bicepparam` file is generated from the resolved ARM parameters using `generateBicepParam()`. This file is placed next to the source `.bicep` module so that relative `using` paths resolve correctly.
49+
50+
2. **Build snapshot options.** The deployment target (`infra.Deployment`) provides scope information:
51+
- **Subscription-scoped deployments** → `--subscription-id` and `--location` flags.
52+
- **Resource-group-scoped deployments** → `--subscription-id` and `--resource-group` flags.
53+
54+
3. **Invoke `bicep snapshot`.** The Bicep CLI generates a `<basename>.snapshot.json` file. azd reads it into memory and deletes the temporary file.
55+
56+
4. **Parse the snapshot.** The JSON output contains a `predictedResources` array — each entry is a fully resolved resource with type, apiVersion, name, location, properties, and so on.
57+
58+
```
59+
┌──────────────────────────┐
60+
│ main.bicep │
61+
│ + parameters │
62+
└──────────┬───────────────┘
63+
│
64+
generateBicepParam()
65+
│
66+
▼
67+
┌──────────────────────────┐
68+
│ preflight-*.bicepparam │ (temporary)
69+
└──────────┬───────────────┘
70+
│
71+
bicep snapshot --subscription-id ... --resource-group ...
72+
│
73+
▼
74+
┌──────────────────────────┐
75+
│ preflight-*.snapshot.json│
76+
│ { │
77+
│ "predictedResources": │
78+
│ [ │
79+
│ { type, name, ... } │
80+
│ { type, name, ... } │
81+
│ ] │
82+
│ } │
83+
└──────────────────────────┘
84+
```
85+
86+
## Check Pipeline
87+
88+
The preflight system uses a pluggable pipeline of check functions. Each check receives a `validationContext` containing:
89+
90+
- **`Console`** — for user interaction (prompts, messages).
91+
- **`Props`** — derived properties from the resource analysis (e.g. `HasRoleAssignments`).
92+
- **`ResourcesSnapshot`** — the raw JSON from `bicep snapshot`.
93+
- **`SnapshotResources`** — the parsed `[]armTemplateResource` list from the snapshot.
94+
95+
Checks are registered via `AddCheck()` before calling `validate()`. They run in registration order and each returns either:
96+
97+
- `nil` — nothing to report (check passed).
98+
- `*PreflightCheckResult` with a `Severity` (`PreflightCheckWarning` or `PreflightCheckError`) and a `Message`.
99+
100+
### Adding a New Check
101+
102+
To add a new preflight check:
103+
104+
```go
105+
localPreflight.AddCheck(func(ctx context.Context, valCtx *validationContext) (*PreflightCheckResult, error) {
106+
// Inspect valCtx.SnapshotResources, valCtx.Props, etc.
107+
for _, res := range valCtx.SnapshotResources {
108+
if strings.EqualFold(res.Type, "Microsoft.SomeProvider/problematicResource") {
109+
return &PreflightCheckResult{
110+
Severity: PreflightCheckWarning,
111+
Message: "This resource type requires additional configuration.",
112+
}, nil
113+
}
114+
}
115+
return nil, nil // nothing to report
116+
})
117+
```
118+
119+
### Built-in Checks
120+
121+
| Check | What It Does | Severity |
122+
|---|---|---|
123+
| Role assignment permissions | Detects `Microsoft.Authorization/roleAssignments` in the snapshot and verifies the current principal has `roleAssignments/write` permission on the subscription. | Warning |
124+
125+
## UX Presentation
126+
127+
Results are displayed using the `PreflightReport` UX component (`pkg/output/ux/preflight_report.go`), which implements the standard `UxItem` interface. The report groups and orders findings: all warnings appear first, followed by all errors. Each entry is prefixed with the standard azd status icons.
128+
129+
## Scenarios
130+
131+
### Scenario 1: No Issues Found
132+
133+
All registered checks pass. No output is printed from the preflight step. The deployment proceeds directly to server-side validation and then Azure deployment.
134+
135+
```
136+
Validating deployment (✓) Done:
137+
138+
Creating/Updating resources ...
139+
```
140+
141+
### Scenario 2: Warnings Only
142+
143+
One or more checks return warnings but no errors. The warnings are displayed and the user is prompted to continue. The default selection is **Yes** — pressing Enter continues the deployment.
144+
145+
```
146+
Validating deployment
147+
148+
(!) Warning: the current principal (abc-123) does not have permission
149+
to create role assignments (Microsoft.Authorization/roleAssignments/write)
150+
on subscription sub-456. The deployment includes role assignments and
151+
will fail without this permission.
152+
153+
? Preflight validation found warnings that may cause the deployment
154+
to fail. Do you want to continue? (Y/n)
155+
```
156+
157+
If the user confirms (or accepts the default), deployment proceeds normally. If the user declines, the operation is aborted with a zero exit code (an intentional abort, not a failure).
158+
159+
### Scenario 3: Errors Only
160+
161+
One or more checks return errors. The errors are displayed and the deployment is **immediately aborted** — the user is not prompted. The CLI exits with a zero exit code.
162+
163+
```
164+
Validating deployment
165+
166+
(x) Failed: critical configuration error detected in template
167+
168+
preflight validation detected errors, deployment aborted
169+
```
170+
171+
Note: the exit code is **zero** because the preflight validation **successfully** detected problems and intentionally aborted the deployment. This is not an unexpected internal failure — the CLI completed its task (validating and reporting errors) without encountering any execution errors itself.
172+
173+
### Scenario 4: Warnings and Errors
174+
175+
When the report contains both warnings and errors, warnings are listed first and errors second. Because errors are present the deployment is aborted immediately — the warning prompt is skipped.
176+
177+
```
178+
Validating deployment
179+
180+
(!) Warning: the current principal does not have permission to create
181+
role assignments on this subscription.
182+
183+
(x) Failed: required parameter 'storageAccountName' is missing from
184+
the deployment.
185+
186+
preflight validation detected errors, deployment aborted
187+
```
188+
189+
### Scenario 5: Check Function Returns an Error
190+
191+
If a check function itself fails (returns a Go `error` rather than a `*PreflightCheckResult`), this is treated as an infrastructure failure. The CLI reports it as a hard error and exits with a non-zero code. This is distinct from a check returning a result with `PreflightCheckError` severity — that case means "we successfully detected a problem in the template", while an error return means "something went wrong while trying to run the check".
192+
193+
```
194+
ERROR: local preflight validation failed: preflight check failed: <underlying error>
195+
```
196+
197+
## Exit Code Behavior
198+
199+
The exit code distinguishes between **successful operation** (the CLI did what it was supposed to do) and **internal failure** (the CLI could not complete its task).
200+
201+
Preflight validation detecting errors and aborting the deployment is a **successful outcome** — the CLI performed the validation and correctly prevented a bad deployment. Only failures in the validation machinery itself produce a non-zero exit code.
202+
203+
| Outcome | Exit Code | Rationale |
204+
|---|---|---|
205+
| No issues | 0 | Deployment proceeds and succeeds. |
206+
| Warnings only, user continues | 0 | User acknowledged warnings; deployment proceeds. |
207+
| Warnings only, user declines | 0 | User chose to abort; intentional, not a failure. |
208+
| Errors detected | 0 | Validation successfully detected problems and aborted the deployment. |
209+
| Check function error | 1 | Internal failure running a check (the `validate` function returned a non-nil error). |
210+
211+
## File Layout
212+
213+
```
214+
pkg/
215+
├── infra/provisioning/bicep/
216+
│ ├── local_preflight.go # Core pipeline, ARM types, parseTemplate, analyzeResources
217+
│ ├── local_preflight_test.go # Unit tests for parsing, analysis, check pipeline
218+
│ ├── role_assignment_check_test.go # Tests for the role assignment check
219+
│ ├── generate_bicep_param_test.go # Tests for .bicepparam generation
220+
│ └── bicep_provider.go # validatePreflight() integration, checkRoleAssignmentPermissions
221+
├── output/ux/
222+
│ ├── preflight_report.go # PreflightReport UxItem
223+
│ └── preflight_report_test.go # Tests for PreflightReport
224+
└── tools/bicep/
225+
└── bicep.go # Snapshot() method, SnapshotOptions builder
226+
```

‎cli/azd/docs/local-preflight/storage-account-policy-check.md‎

Lines changed: 64 additions & 25 deletions
Original file line numberDiff line numberDiff line change
@@ -145,7 +145,7 @@ properties) and Azure returns which policies would deny it and why.
145145

146146
```
147147
POST /subscriptions/{id}/providers/Microsoft.PolicyInsights/checkPolicyRestrictions
148-
?api-version=2022-03-01 # version tested in this investigation
148+
?api-version=2022-03-01
149149
150150
{
151151
"resourceDetails": {
@@ -167,20 +167,66 @@ The API:
167167

168168
**Result: Accurate evaluation but does not see management-group-inherited policies.**
169169

170-
Testing against subscriptions with management-group-assigned deny policies
171-
confirmed that the API returns **empty results** (`policyEvaluations: []`,
172-
`fieldRestrictions: []`) even when:
170+
Testing against a subscription with a management-group-assigned deny policy
171+
("Storage Accounts - Safe Secrets Standard", assigned at the tenant root
172+
management group) confirmed that the API returns **empty results**
173+
(`policyEvaluations: []`, `fieldRestrictions: []`) even when:
173174

174-
- The deny policy is confirmed active (deployments fail with
175-
`RequestDisallowedByPolicy`)
176-
- The subscription tags satisfy all opt-in conditions
175+
- The deny policy is confirmed active (`policyStates` reports it with
176+
`effect: deny` for existing storage accounts)
177177
- The resource content explicitly sets the denied property
178178
- Both subscription-scope and resource-group-scope endpoints are tried
179179
- A `PendingFields` parameter is included in the request
180180

181-
The API only evaluates policies assigned directly at the subscription or resource
182-
group scope. Since enterprise storage deny policies originate from management
183-
groups, `checkPolicyRestrictions` does not evaluate them.
181+
The API correctly detects subscription-level policies — for example, a
182+
subscription-scoped `modify` policy for `allowBlobPublicAccess` returned the
183+
expected `fieldRestrictions` with `result: Required`. Only management-group-
184+
inherited policies are invisible.
185+
186+
#### `api-version=2024-10-01` follow-up (April 2026)
187+
188+
The `2024-10-01` API version added a
189+
[management group scope endpoint](https://learn.microsoft.com/en-us/rest/api/policyinsights/policy-restrictions/check-at-management-group-scope?view=rest-policyinsights-2024-10-01)
190+
and an `includeAuditEffect` parameter on the subscription scope. We tested both
191+
to see if they resolve the MG-inherited policy blind spot.
192+
193+
**Subscription-scope with `2024-10-01`:** Same empty results as `2022-03-01`.
194+
The newer API version does not change the subscription-scope behavior — MG-
195+
inherited policies remain invisible. Adding `includeAuditEffect: true` also
196+
returned empty.
197+
198+
**Management group scope endpoint:** This endpoint only supports `pendingFields`
199+
with a single `type` field — it rejects `resourceDetails` entirely:
200+
201+
```
202+
POST /providers/Microsoft.Management/managementGroups/{mgId}
203+
/providers/Microsoft.PolicyInsights/checkPolicyRestrictions
204+
?api-version=2024-10-01
205+
206+
# Only this request body is accepted:
207+
{ "pendingFields": [{ "field": "type" }] }
208+
209+
# This is rejected with InvalidCheckRestrictionsRequest:
210+
{ "resourceDetails": { ... } }
211+
```
212+
213+
The error message confirms the limitation:
214+
> *"The 'resourceDetails' property is not supported in requests at Management
215+
> Group level. The request content can only have a single 'type' pending field."*
216+
217+
Testing at multiple levels of the MG hierarchy (direct parent, intermediate MGs,
218+
tenant root) all returned empty `fieldRestrictions` even for the `type` field.
219+
The MG-scope endpoint is designed to answer "which resource types are
220+
restricted" at a management group level, not "would this specific resource
221+
configuration be denied" — making it unsuitable for property-level checks like
222+
`allowSharedKeyAccess`.
223+
224+
| Endpoint | API Version | Sees MG deny policies | Supports resource properties |
225+
|---|---|---|---|
226+
| Subscription scope | `2022-03-01` | ❌ | ✅ |
227+
| Subscription scope | `2024-10-01` | ❌ | ✅ |
228+
| Resource group scope | `2024-10-01` | ❌ | ✅ |
229+
| MG scope | `2024-10-01` | Untested (empty for `type`) | ❌ (rejected) |
184230

185231
### Approach 3: Policy States API (`policyStates`)
186232

@@ -197,7 +243,8 @@ warn *before* deployment, this API is not applicable.
197243
| Approach | Sees MG policies | Evaluates all conditions | Limitation | Suitable |
198244
|---|---|---|---|---|
199245
| Client-side ARM policy SDK parsing | ✅ Yes | ❌ No (ARM expressions) | Cannot evaluate runtime expressions → false positives | ❌ |
200-
| Server-side `checkPolicyRestrictions` | ❌ No | ✅ Yes | Subscription scope misses MG-inherited policies | ❌ |
246+
| Server-side `checkPolicyRestrictions` (sub scope) | ❌ No | ✅ Yes | Misses MG-inherited policies (confirmed with both `2022-03-01` and `2024-10-01`) | ❌ |
247+
| Server-side `checkPolicyRestrictions` (MG scope, `2024-10-01`) | Untested | ❌ No | Only supports `type` field; rejects `resourceDetails` | ❌ |
201248
| `policyStates` API | ✅ Yes | ✅ Yes | Only evaluates already-deployed resources | ❌ |
202249

203250
No currently available approach provides both accurate policy detection
@@ -208,20 +255,12 @@ system.
208255

209256
## Future Considerations
210257

211-
- **Management group scope endpoint (`2024-10-01`)**: The `checkPolicyRestrictions`
212-
API added a [management group scope endpoint](https://learn.microsoft.com/en-us/rest/api/policyinsights/policy-restrictions/check-at-management-group-scope?view=rest-policyinsights-2024-10-01)
213-
in `api-version=2024-10-01`:
214-
```
215-
POST /providers/Microsoft.Management/managementGroups/{mgId}/providers/Microsoft.PolicyInsights/checkPolicyRestrictions
216-
?api-version=2024-10-01
217-
```
218-
This could address the core limitation — the subscription-scope endpoint
219-
doesn't see MG-inherited policies, but the MG-scope endpoint evaluates against
220-
the full policy hierarchy. This is the most promising next step. A hybrid
221-
approach (client-side detection to find relevant MG IDs + MG-scope server-side
222-
evaluation) could combine accurate detection with full condition evaluation.
223-
**Note**: This investigation tested only `api-version=2022-03-01` at the
224-
subscription scope. The MG-scope endpoint was not tested.
258+
- **`checkPolicyRestrictions` MG-scope with `resourceDetails` support**: If
259+
Microsoft updates the MG-scope endpoint to accept `resourceDetails` (not just
260+
`type` pending fields), it would enable property-level evaluation against
261+
MG-inherited policies. This is the only server-side path that could solve the
262+
problem without client-side expression evaluation. As of `2024-10-01`, the
263+
MG-scope endpoint rejects `resourceDetails` requests.
225264
- A hybrid approach (client-side detection + subscription tag evaluation for
226265
common patterns) could reduce false positives for the most common gating
227266
conditions, at the cost of maintaining a partial ARM expression evaluator.

0 commit comments

Comments
 (0)