feat(gcp): Add Hierarchical Namespace (HNS) support for GCS bucket by varunarya002 · Pull Request #3996 · apache/polaris

varunarya002 · 2026-03-13T20:06:03Z

Issue:
GCS buckets with HNS enabled require explicit folder creation and storage.folderAdmin permissions for managed folder operations. Without this, Iceberg table creation and Spark ingestion fail on HNS buckets.

Changes:

Add hierarchicalNamespace boolean field to GcpStorageConfigInfo (OpenAPI spec)
Add @nullable Boolean isHierarchicalNamespace() to GcpStorageConfigurationInfo
Generate storage.folderAdmin access boundary rule scoped to managedFolders/{path} for write locations when HNS is enabled
Plumb HNS flag through CatalogEntity model<->entity conversion
Add backward-compatible 3-param overload defaulting HNS to false

Tests:

HNS enabled single bucket (read + write + folder rules)
HNS with multiple buckets and partial writes
HNS without writes (no folder rules)
HNS with separate metadata and data buckets
HNS same bucket with separate metadata/data paths
Non-HNS vs HNS rule count comparison
Config serialization round-trip
CatalogEntity round-trip with HNS flag

Checklist

🛡️ Don't disclose security issues! (contact security@apache.org)
🔗 Clearly explained why the changes are needed, or linked related issues: Fixes #
🧪 Added/updated tests with good coverage, or manually tested (and explained how)
💡 Added comments for complex logic
🧾 Updated CHANGELOG.md (if needed)
📚 Updated documentation in site/content/in-dev/unreleased (if needed)

dimas-b

Thanks for your contribution, @varunarya002 !

Since this PR affects REST API parameters, please open a corresponding discussion on the dev ML, which is a usual practice in Polaris.

(this is not a complete review from my side 😅 ... just noting a couple of points for a start)

dimas-b · 2026-03-13T21:34:45Z

            gcsServiceAccount:
              type: string
              description: a Google cloud storage service account
+            hierarchicalNamespace:


nit: a similar field in Azure is called simply hierarchical

dimas-b · 2026-04-01T16:10:52Z

@varunarya002 : Do you have capacity to push this PR forward? If you do not have time, it's fine. We can build something based of your work and still give you attribution.

varunarya002 · 2026-04-01T17:33:04Z

Hi @dimas-b. I will be starting discussion about this change on dev ML.

dimas-b · 2026-04-06T16:26:41Z

+    JsonNode parsedRules = mapper.convertValue(credentialAccessBoundary, JsonNode.class);
+    JsonNode refRules = readResource(mapper, "gcp-testGenerateAccessBoundaryHnsEnabled.json");
+    assertThat(parsedRules)
+        .usingRecursiveComparison(


nit: does simple equals not work in this case?

dimas-b · 2026-04-08T21:57:34Z

+
+  public StorageLocationPreparer create(@Nonnull PolarisStorageConfigurationInfo storageConfig) {
+    if (storageConfig instanceof GcpStorageConfigurationInfo && storageConfiguration != null) {
+      return new GcsStorageLocationPreparer(storageConfiguration.gcpCredentialsSupplier(clock));


Could we use StorageAccessConfigProvider for storage credentials?

It's not feasible because of different trust levels (admin creds for HNS folder creation vs subscoped client tokens).

Hmmm... sorry, but I'm a bit confused now... Clients (e.g. Spark) should be able to create folders (for new data files with Iceberg partitioning) using the vended credentials, right?

dimas-b · 2026-04-08T21:59:02Z

+                () ->
+                    Optional.ofNullable(
+                        tableProperties.get(
+                            IcebergTableLikeEntity.USER_SPECIFIED_WRITE_METADATA_LOCATION_KEY)))


This does not look specific to GCS. Could we handle this part in a more general way?

Extracted AbstractStorageLocationPreparer base class with generic URI parsing, hierarchy building, bucket grouping.

dimas-b · 2026-04-08T22:01:04Z

+import java.util.Map;
+
+public interface StorageLocationPreparer {
+  void prepareTableLocation(String tableLocation, Map<String, String> tableProperties);


I wonder if we could simplify this SPI 🤔 I suppose we could simply request a certain set of folders to be created. Do you envision more sophisticated work to be done (in some future cases) for preparing table locations?

Changed to void prepareLocations(List) — no Iceberg concepts leak into the SPI.

dimas-b · 2026-04-08T22:53:07Z

+  }
+
+  public StorageLocationPreparer create(@Nonnull PolarisStorageConfigurationInfo storageConfig) {
+    if (storageConfig instanceof GcpStorageConfigurationInfo && storageConfiguration != null) {


Let's leverage CDI via storage type-based @Identifier annotations... similar to ServiceProducers.polarisAuthorizerFactory()

Factory uses Instance lookup by StorageType.name(). GCS preparer annotated with @Identifier("GCS").

Co-Authored-By: claude-flow <ruv@ruv.net>

dimas-b · 2026-04-16T18:55:46Z

Related dev thread: https://lists.apache.org/thread/jyx21cmbprttb5p6lr2q0q1ytn0jgv0w

dimas-b · 2026-04-20T18:48:45Z

+              CredentialAccessBoundary.AccessBoundaryRule.AvailabilityCondition.newBuilder()
+                  .setExpression(String.join(" || ", folderConditions))
+                  .build());
+          builder.setAvailablePermissions(List.of("inRole:roles/storage.folderAdmin"));


We add this unconditionally in this code, but it is required only for HNS, right?

I suppose it would be more robust to check resolveHnsStatus() here or have an HNS flag in the Storage Config.... I'm kind of leaning toward the flag, even though we can auto-detect this... WDYT?

My thinking is that users are generally aware of the the HNS flag in their GCS storage (or al least they should be) and it is not likely to change in runtime.

I agree with @dimas-b 's comment - is it a concern that we are introducing this permission for any GCS backed tables? That IMO goes against the principle of granting least-privileged access, so I think it would be helpful to review if that exposes any unnecessary actions for non HNS enabled tables.

sungwy

Hi @varunarya002 thank you so much for putting together this PR. The new StorageLocationPreparer abstraction is a highly reusable idea, and I think it is directionally correct. I took a first pass at reviewing this PR and left some comments.

Also, could we add a bit more context in the PR description about this model? We are effectively introducing a pre-createTable hook into the table creation flow, so it might be worth calling that out explicitly.

sungwy · 2026-04-20T23:10:38Z

+ * resolve folder hierarchies, group by bucket, and delegate to {@link
+ * #createFoldersForBucket(String, List)} for storage-specific operations.
+ */
+public abstract class AbstractStorageLocationPreparer implements StorageLocationPreparer {


I feel this name is a bit too abstract (pun intended 🙂). Would it make sense to give it a more descriptive name like HierarchicalFolderLocationPreparer or ObjectStoreFolderPreparer?

I also wonder whether introducing this abstraction is a bit premature. Do we expect these methods to be reusable across other cloud providers, or is this really tailored to the GCS HNS case for now?

If it is mainly the latter, would it make sense to fold this into GcsStorageLocationPreparer for now and only extract a shared abstraction once we have a second provider implementation and can validate the actual common ground?

sungwy · 2026-04-20T23:15:44Z

+      return NO_OP;
+    }
+    String key = storageConfig.getStorageType().name();
+    Instance<StorageLocationPreparer> selected = preparers.select(Identifier.Literal.of(key));


I like the StorageLocationPreparer as an abstraction, but instead of introducing it as a separate standalone selection path, would it make more sense for it to be exposed by PolarisStorageIntegration instead?

PolarisStorageIntegration is already selected from the storage configuration type, so grouping storage-specific behavior there seems cleaner and makes the config-driven selection more obvious and consistent.

I do think it still makes sense to model StorageLocationPreparer as a separate capability. I’m just wondering whether PolarisStorageIntegration should be single the interface that exposes that capability, rather than introducing a parallel factory and dispatch path for storage-specific behavior.

sungwy · 2026-04-20T23:16:57Z

+              CredentialAccessBoundary.AccessBoundaryRule.AvailabilityCondition.newBuilder()
+                  .setExpression(String.join(" || ", folderConditions))
+                  .build());
+          builder.setAvailablePermissions(List.of("inRole:roles/storage.folderAdmin"));


I agree with @dimas-b 's comment - is it a concern that we are introducing this permission for any GCS backed tables? That IMO goes against the principle of granting least-privileged access, so I think it would be helpful to review if that exposes any unnecessary actions for non HNS enabled tables.

github-project-automation Bot added this to Basic Kanban Board Mar 13, 2026

github-project-automation Bot moved this to PRs In Progress in Basic Kanban Board Mar 13, 2026

varunarya002 force-pushed the gcp_hns_bucket branch from 0bde95d to 119b101 Compare March 13, 2026 20:06

varunarya002 changed the title ~~feat(gcp): Add Hierarchical Namespace (HNS) support for GCS bucket cr…~~ feat(gcp): Add Hierarchical Namespace (HNS) support for GCS bucket Mar 13, 2026

dimas-b reviewed Mar 13, 2026

View reviewed changes

fivetran-caseykarst mentioned this pull request Apr 1, 2026

Improve GCS documentation formatting and fix grammatical issues #4106

Open

4 tasks

dimas-b reviewed Apr 3, 2026

View reviewed changes

Comment thread ...core/src/main/java/org/apache/polaris/core/storage/gcp/GcpCredentialsStorageIntegration.java

varunarya002 force-pushed the gcp_hns_bucket branch 5 times, most recently from 918e6d4 to f43e793 Compare April 5, 2026 08:32

dimas-b reviewed Apr 6, 2026

View reviewed changes

dimas-b reviewed Apr 8, 2026

View reviewed changes

varunarya002 and others added 4 commits April 12, 2026 20:41

Add Hierarchical Namespace (HNS) support for GCS bucket

d96ece5

Co-Authored-By: claude-flow <ruv@ruv.net>

CI fixes

8cb0a79

Fix test cases

7308a17

Address the comments

561f90d

varunarya002 force-pushed the gcp_hns_bucket branch from 1ccdfce to 561f90d Compare April 12, 2026 16:05

dimas-b requested a review from flyrain April 16, 2026 18:55

dimas-b reviewed Apr 20, 2026

View reviewed changes

sungwy reviewed Apr 20, 2026

View reviewed changes

Conversation

varunarya002 commented Mar 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Checklist

Uh oh!

dimas-b left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dimas-b commented Apr 1, 2026

Uh oh!

varunarya002 commented Apr 1, 2026

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dimas-b Apr 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dimas-b commented Apr 16, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sungwy left a comment

Choose a reason for hiding this comment

Uh oh!

sungwy Apr 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

varunarya002 commented Mar 13, 2026 •

edited

Loading

dimas-b Apr 8, 2026 •

edited

Loading

sungwy Apr 20, 2026 •

edited

Loading