Skip to content

GH-32123: [R] Expose azure blob filesystem#49553

Open
marberts wants to merge 40 commits intoapache:mainfrom
marberts:32123-expose-azure-blob-filesystem
Open

GH-32123: [R] Expose azure blob filesystem#49553
marberts wants to merge 40 commits intoapache:mainfrom
marberts:32123-expose-azure-blob-filesystem

Conversation

@marberts
Copy link
Contributor

@marberts marberts commented Mar 19, 2026

Rationale for this change

This PR adds support for Azure. The Arrow R package already has support for AWS and GCS, and the Arrow C++ library has had support for Azure for a couple years now. Support for Azure is already available in pyarrow.

This would close #32123.

What changes are included in this PR?

  • A new class AzureFileSystem that's analogous to S3FileSystem/GcsFileSystem, along with a helper function az_container() that's analogous to s3_bucket()/gcs_bucket().

  • Updates to src/filesystem.cpp to interact with the machinery in arrow/filesystem/azurefs.h.

  • Updates to the configuration and build scripts to support building with support for Azure.

  • Updates to the vignettes on cloud storage, installation, and developer setup.

Are these changes tested?

Yes. See tests/testthat/test-azure.R.

Are there any user-facing changes?

Yes. There is a new function az_container(), serving the analogous role to s3_bucket()/gcs_bucket(), along with an R6 class AzureFileSystem, again serving the same role as S3FileSystem and GcsFileSystem. There is also a function arrow_with_azure() to indicate if Arrow was built with support for Azure.

Collinbrown95 and others added 30 commits March 3, 2026 04:05
We'll want to use the proper build flag before submitting the PR but for now this successfully links the C++ binding to the azurefs_is_functional_test function.
…rts/arrow into 32123-expose-azure-blob-filesystem
Note: I think what's happening is that the build of Arrow I install doesn't have the ARROW_AZURE flag enabled. When I "force" the environment variable I get an error like "fs___AzureFileSystem__Make not found".
Note: this was causing a seg fault when anything that inherits from FileSystem ries to access the base_path property. Uncommenting the return line and rebuilding the R package resolved this issue.
At this point, I was able to call write_feather(example_data, fs$path("test/test.feather")) successfully against Azurite.

Committing progress before I make any further changes.
Switch over  to locally hosted azurite instead of local docker container azurite.
@github-actions
Copy link

Thanks for opening a pull request!

If this is not a minor PR. Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose

Opening GitHub issues ahead of time contributes to the Openness of the Apache Arrow project.

Then could you also rename the pull request title in the following format?

GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

or

MINOR: [${COMPONENT}] ${SUMMARY}

See also:

@marberts marberts changed the title 32123 expose azure blob filesystem GH-32123 [R]: Expose azure blob filesystem Mar 19, 2026
@marberts marberts changed the title GH-32123 [R]: Expose azure blob filesystem GH-32123: [R] Expose azure blob filesystem Mar 19, 2026
@github-actions
Copy link

⚠️ GitHub issue #32123 has been automatically assigned in GitHub to PR creator.

@marberts
Copy link
Contributor Author

marberts commented Mar 20, 2026

The CI is failing because the Azure C++ SDK depends on libxml2.

  • Update tools/nixlibs.R to check for libxml2 in with_cloud_support(), turning off the Azure feature flag if libxml2 isn't found.
    • What about configure and configure.win?
    • Is with_cloud_support() the only function that needs changing in tools/nixlibs.R?
  • Update the install.Rmd and developers/setup.Rmd vignettes to note that libxml2 is a dependency of Azure (same as with curl and openssl).
  • Update DESCRIPTION to include libxml2 in SystemRequirements.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[R] Expose Azure Blob Storage filesystem

2 participants