This tutorial will give an introduction to the StorageGRID Webscale and S3 PowerShell Cmdlets.
Since PowerShell 3 all available Modules are automatically loaded. To explicitly load the StorageGRID-Webscale Module use
Import-Module -Name StorageGRID-WebscaleList all Cmdlets included in the StorageGRID-Webscale Module
Get-Command -Module StorageGRID-WebscaleList all Cmdlets included in the S3 Client (this command only works with PowerShell 6 and later)
Get-Command -Module S3-ClientShow the syntax of all Cmdlets from the StorageGRID-Webscale Module
Get-Command -Module StorageGRID-Webscale -SyntaxShow the syntax of all Cmdlets from the StorageGRID-Webscale Module (this command only works with PowerShell 6 and later)
Get-Command -Module S3-Client -SyntaxTo get detailed help including examples for a specific Cmdlet (e.g. for Connect-SgwServer) run
Get-Help Connect-SgwServer -DetailedIt is recommended to use profiles to manage the connections to StorageGRID. The profile contains the login information which will be saved under
Add-SgwProfile -ProfileName MyProfile -Name adminnode.example.com -Credential (Get-Credential)This section is dedicated to Tenant Users. If you are a Grid Administrator, check section Tutorial for Grid Administrators
For data retrieval a connection to the StorageGRID Management Server (also referred to as "Admin Node") is required. The Connect-SgwServer Cmdlet expects the hostname or IP (including port if different than 80 for HTTP or 443 for HTTPS) and the credentials for authentication
$Name = "admin-node.example.org"
$Credential = Get-Credential
$Server = Connect-SgwServer -Name $Name -Credential $CredentialIf the login fails, it is often due to an untrusted certificate of the StorageGRID Management Server. You can ignore the certificate check with the -SkipCertificateCheck option
$Server = Connect-SgwServer -Name $Name -Credential $Credential -SkipCertificateCheckBy default the connection to the StorageGRID Webscale Server is established through HTTPS. If that doesn't work, HTTP will be tried.
The Cmdlets allow to connect either as Grid Administrator or as Tenant User. To connect to the StorageGRID Management Server as a tenant user specify the AccountId of the tenant. Ask a Grid Administrator for the AccountId of the tenant if you don't know it.
Connect-SgwServer -Name $Name -Credential $Credential -AccountId $AccountIdTo create a new S3 tenant use the following steps.
$Credential = Get-Credential -UserName root
$Account = New-SgwAccount -Name "MyTenant" -Capabilities s3,management -Password $Credential.GetNetworkCredential().PasswordCheck that Account was created either by listing all accounts
Get-SgwAccountsor by checking for the individual account
$Account | Get-SgwAccountLog in with the root user of the new tenant
$Account | Connect-SgwServer -Name $Name -Credential $CredentialThen create S3 credentials and remember to store the access key and secret access key in a secure location
New-SgwS3AccessKeyCheck that S3 access keys have been created by listing all access key
Get-SgwS3AccessKeysor by checking for the individual account
$AccessKey | Get-SgwS3AccessKeyThe tenant can only be removed by a grid administrator. If you have stored the grid admin connection in the variable $Server as shown above, then you can use it here
$Account | Remove-SgwAccount -Server $ServerOtherwise connect as grid administrator again and then run
$Account | Remove-SgwAccountLocal groups and users can be used to grant full or restricted access to the resources of a tenant to e.g. a specific application.
Make sure that you are connected to the tenant as root user or user with privileges to manager users and groups (check the previous section for details).
$Account | Connect-SgwServer -Name $Name -Credential $CredentialIf you want to restrict S3 access by denying bucket creation, deletion, policy change and versioning change as well as object version deletion create the following policy
$GroupPolicy = New-AwsPolicy -Principal $null
$GroupPolicy = $GroupPolicy | Add-AwsPolicyStatement -Principal $null -Effect Deny -Action s3:CreateBucket,s3:DeleteBucket,s3:DeleteBucketPolicy,s3:PutBucketPolicy,s3:PutBucketVersioning,s3:DeleteObjectVersionNow create a new group with the previously created policy
$Group = New-SgwGroup -UniqueName "mygroup" -Type "local" -S3Policy $GroupPolicyUsers of this group will not be able to log into the tenant portal. If you want to allow Tenant Portal access specify either of the parameters -RootAccess,-ManageAllContainers,-ManageOwnS3Credentials.
If you want to allow full S3 access use the following to create the group
$Group = New-SgwGroup -UniqueName "mygroup" -Type "local" -S3FullAccessNow create a local user
$UserCredential = Get-Credential -UserName "myuser" -Message "Insert the password for the user"
$User = $Group | New-SgwUser -UniqueName "myuser" -Password $UserCredential.GetNetworkCredential().PasswordAs a tenant user, the Account usage can be retrieved with
Get-SgwAccountUsageA grid administrator can use this simple workflow to retrieve the S3 account usage data and export it as CSV to $HOME\Downloads\TenantAccounting.csv.
$TenantAccounting = foreach ($Account in (Get-SgwAccounts | Where-Object { $_.capabilities -match "s3" })) {
$Usage = $Account | Get-SgwAccountUsage
$Output = New-Object -TypeName PSCustomObject -Property @{Name=$Account.name;ID=$Account.id;"Calculation Time"=$Usage.calculationTime;"Object Count"=$Usage.objectCount;"Data Bytes used"=$Usage.dataBytes}
Write-Output $Output
}
$TenantAccounting | Export-Csv -Path $HOME\Downloads\TenantAccounting.csv -NoTypeInformationThe next workflow will retrieve the S3 Account usage per Bucket and export it as CSV to $HOME\Downloads\BucketAccounting.csv
$BucketAccounting = foreach ($Account in (Get-SgwAccounts | Where-Object { $_.capabilities -match "s3" })) {
$Usage = $Account | Get-SgwAccountUsage
foreach ($Bucket in $Usage.Buckets) {
$Output = New-Object -TypeName PSCustomObject -Property @{Name=$Account.name;ID=$Account.id;"Calculation Time"=$Usage.calculationTime;Bucket=$Bucket.Name;"Object Count"=$Bucket.objectCount;"Data Bytes used"=$Bucket.dataBytes}
Write-Output $Output
}
}
$BucketAccounting | Export-Csv -Path $HOME\Downloads\BucketAccounting.csv -NoTypeInformationThe following script expects that the S3-Client PowerShell Module is installed and that a connection to a server was established via Connect-SgwServer and the minimum API Version was set to 1 with Update-SgwConfigManagement -MinApiVersion 1.
$Accounts = Get-SgwAccounts -Capabilities s3
$TenantAccounting = @()
$BucketAccounting = @()
foreach ($Account in $Accounts) {
Write-Host "Checking account $($Account.Name)"
$AccountUsage = 0
$AccountObjectCount = 0
$Buckets = $Account | Get-S3Buckets
foreach ($Bucket in $Buckets) {
Write-Host "Checking bucket $($Bucket.BucketName)"
$BucketUsage = 0
$BucketObjectCount = 0
if ($Bucket) {
$Objects = $Bucket | Get-S3Objects
foreach ($Object in $Objects) {
$BucketObjectCount += 1
$ObjectMetadata = $Object | Get-SgwObjectMetadata
$ReplicaCount = ($ObjectMetadata.locations | ? { $_.type -eq "replicated" }).Count
$ErasureCodeDataCount = ($ObjectMetadata.locations.fragments | ? { $_.type -eq "data" }).Count
$ErasureCodeParityCount = ($ObjectMetadata.locations.fragments | ? { $_.type -eq "parity" }).Count
if ($ReplicaCount) {
$SizeFactor = $ReplicaCount
}
elseif ($ErasureCodeDataCount) {
$SizeFactor = ($ErasureCodeDataCount + $ErasureCodeParityCount) / $ErasureCodeDataCount
}
elseif ($ObjectMetadata.objectSizeBytes -gt 0) {
Write-Warning "No replicas and EC parts found for object ${Bucket.BucketName}/$($Object.Key)"
}
$SizeFactor = $ReplicaCount
if ($ObjectMetadata.diskSizeBytes) {
$BucketUsage += $SizeFactor * $ObjectMetadata.diskSizeBytes
}
else {
$BucketUsage += $SizeFactor * $ObjectMetadata.objectSizeBytes
}
}
$BucketAccounting += [PSCustomObject]@{AccountName=$Account.Name;AccountId=$Account.Id;BucketName=$Bucket.BucketName;BucketUsage=$BucketUsage;BucketObjectCount=$BucketObjectCount}
$AccountUsage += $BucketUsage
$AccountObjectCount += $BucketObjectCount
}
}
$TenantAccounting += [PSCustomObject]@{AccountName=$Account.Name;AccountId=$Account.Id;AccountUsage=$AccountUsage;AccountObjectCount=$AccountObjectCount}
}
$BucketAccounting | Export-Csv -Path $HOME\Downloads\BucketAccounting.csv -NoTypeInformation
$TenantAccounting | Export-Csv -Path $HOME\Downloads\TenantAccounting.csv -NoTypeInformationStorageGRID allows grid administrators to create reports for different metrics on grid, site and node level.
You can check the available metrics by using Command Completion. Just type the following command and hit the Tabulator key (->|)
Get-SgwReport -AttributeThe reports have a start and an end date. By default the attributes will be listed for the last hour. For attributes within the last 7 days StorageGRID will report the measure value in a specific interval. Attributes older than 7 days will be aggregated and StorageGRID will return average, minium and maximum value for each sample time. You can specify a time range using PowerShell DateTime objects.
Here are some examples:
Get-SgwReport -Attribute 'S3 Ingest - Rate (XSIR) [MB/s]'
Get-SgwReport -Attribute 'S3 Retrieval - Rate (XSRR) [MB/s]' -StartTime (Get-Date).AddDays(-1)
Get-SgwReport -Attribute 'Used Storage Capacity (XUSC) [Bytes]' -StartTime (Get-Date).AddDays(-10) -EndTime (Get-Date).AddDays(-1)By default the report is created for all nodes in the grid. To select a specific site specify the -site parameter
Get-SgwReport -Attribute 'S3 Ingest - Rate (XSIR) [MB/s]' -Site $SiteTo select a specific node specify the -node parameter
Get-SgwReport -Attribute 'S3 Ingest - Rate (XSIR) [MB/s]' -Node $NodeRetrieving individual metrics via REST API is possible since StorageGRID 11.0. To list the available metrics, run
Get-SgwMetricNamesStorageGRID uses Prometheus to collect the metrics and Prometheus Queries can be issued via e.g.
Get-SgwMetricQuery -Query 'storagegrid_s3_operations_successful'