HDDS-14517. [Recon] Include all storage report fields in CSV report for Capacity Distribution

[priyeshkaratha]

What changes were proposed in this pull request?

This pull request simplifies how datanode storage metrics are downloaded. Instead of having multiple endpoints, the download logic is now handled by a single improved endpoint. The old endpoint that only supported pending deletion data has been removed. The new endpoint generates a detailed CSV with all relevant storage metrics, giving a more complete view of datanode health and usage.

• Refactored Datanode Metrics Download: The dedicated /download endpoint for pending deletion metrics in PendingDeletionEndpoint has been removed.
• Consolidated Storage Report: A new /download endpoint has been introduced in StorageDistributionEndpoint to provide a comprehensive CSV report.
• Expanded Report Fields: The new CSV report now includes all datanode storage fields such as Capacity, Used Space, Remaining Space, Committed Space, Reserved Space, Minimum Free Space, and Pending Block Size.
• Frontend URL Update: The frontend application (capacity.tsx) has been updated to use the new consolidated download URL.
• Updated Test Coverage: Corresponding unit tests have been removed from TestPendingDeletionEndpoint and new, comprehensive tests have been added in TestStorageDistributionEndpoint for the new download functionality.

What is the link to the Apache JIRA

HDDS-14517

How was this patch tested?

Tested using modified testcases.

[priyeshkaratha]

Hi @devabhishekpal @devmadhuu @ArafatKhan2198 can you please review the changes?

[devabhishekpal]

This change looks good to me, but it also needs frontend change.
The download button currently calls the pendingDeletion/download API endpoint.
However this changes it to storageDistribution/download so the download button won't work

[priyeshkaratha]

This change looks good to me, but it also needs frontend change. The download button currently calls the pendingDeletion/download API endpoint. However this changes it to storageDistribution/download so the download button won't work

Thanks @devabhishekpal for the review. calling storageDistribution/download is already part of the PR. Did I miss anything specific?

[devabhishekpal]

Thanks for the patch, lgtm @priyeshkaratha

[devmadhuu]

Thanks for the patch @priyeshkaratha . Some comments, pls check.

[devmadhuu]

Since we are adding new APIs, ReadMe also to be updated along with Swagger API doc
cc: @devabhishekpal

[devmadhuu]

1. collectDatanodeReports() calls nodeManager.getAllNodes() which iterates all datanodes
2. For each datanode, it calls nodeManager.getNodeStat() and nodeManager.getTotalFilesystemUsage()
3. This happens synchronously during the HTTP request

For 1000+ datanodes, this could be a bottleneck, can we use already-collected metrics from DataNodeMetricsService, because chances are that user may click to download csv as soon as he see the data in table and if this is not feasible, then can we add some kind of timeout protection ?

[devmadhuu]

When a datanode has pending deletion metrics but no storage report, the row is silently dropped from the CSV. This could mask operational issues.
Scenarios Where This Fails:

1. Datanode just registered but hasn't sent full report yet
2. Datanode in STALE state
3. Race condition between metrics collection and report generation
   Impact: Operators lose visibility into datanodes with pending deletions, which is exactly what they're trying to monitor.