APPLIED SHAREPOINT - USING PRIORITY CLEANUP (PREVIEW) FOR STORAGE RECOVERY
- 2 days ago
- 6 min read
Author: Jonathan Stuckey
Audience: SharePoint consultant; M365 support engineer; Information Management advisor
For an increasingly important topic in tenancy administration - storage recovery and data-management. This article provides some practical knowledge when deploying Purview policies for control for managing large file-types and raging storage consumption.
We cover the use of the Purview Priority Cleanup (Preview) for SharePoint Online storage capacity recovery. The article assumes that you are a power-user or administrator with the relevant knowledge and access to undertake the actions.
While you don't need to know all the in's and out's of the underlying platform, understanding of the implications of configuration and deploying Purview governance controls and (semi-)automation of services is critical to avoiding accidental (unrecoverable) data-loss.
Problem statement
Issues with capacity management or disposal bulk-processing, impacting storage usage. Issues typically are from one of the following:
highly active (transactional) files consuming excessive site-storage on heavily trafficked site(s),
stalled disposal of items in a site's Preservation Hold Library - not releasing space,
failed clean-up of OneDrive's under retention post off-boarding, and manager approval,
failed clean-up post release or removal of Retention policy on a site, or
no means to address transient items that need to be cleared when site should be restricted.
Purview Priority Cleanup
To address the problems above, Microsoft released (in Preview) the Purview Data-Lifecycle Management tool "Priority Cleanup". This is an administrative level control applied in Purview under Data-lifecycle management panel.
This tool allows you to setup a task that will ignore current Retention policy, labels and eDiscovery holds, to let you permanently deleted items on your site.
It does it using a combination of targeting with a hidden label, targeted site-selection, and Keyword Query Language (KQL) to identify objects to delete.
As of this article release Priority Cleanup is:
still in preview and therefore has no official operational support, and
will be attached to an E5 or Compliance add-on license - currently no extra cost.
Operational risks
Unfortunately, bypassing system behaviours has over-tones of circumventing potentially regulatory or legal processes which should not be undertaken by an admin without written confirmation and directive to perform the explicit action.
CAUTION: If in doubt, get it in writing before performing a task which has unrecoverable actions or destruction/disposal of data
Purview Retention processes can result in situations that need methodical root-cause analysis to resolve, this option should only be considered in the event of Retention policy or Retention labels:
not being processed correctly by system,
applied label or policy action having stalled and not feeding items through disposal to recycle bin, or
significant build of items in "Review" stage of disposal - without person to review.
Priority Cleanup should only be employed for a forced clean-up when the built-in automation, back-end tools or PowerShell options for cleaning-up site(s) fail to work as expected.
Recommended usage
Besides the suggested scenarios from Microsoft in the article: Microsoft Learn | Override holds to clean up files and reclaim storage we have recurring jobs based on the following scenarios:
Scenario: Disposal of system legacy digital source copies
Disposing of Digital Source Records, from a migrated system, tenancy or site. I.e. historical copies of source data which has subsequently been migrated and verified when moved.
With SharePoint this is often the case with system or site migrations and copies are left in the original system / site being decommissioned and the site is held under a Retention Policy or has applied labels.
Scenario: Globally applied Retention Policy
IT or Infrastructure partner enables a Global Retention Policy for SharePoint sites and / or one for OneDrive for Business.
Historic application of global policies with excessive duration (e.g. 25yrs+) and often using 'Retain forever' action. These will cause storage consumption to blow-out and may leave the organisation with significant operational exposure under legal or regulatory rules.
Pre-requisites
Additional Delegated Roles
Microsoft enforces additional role-requirements to run this process. On the plus-side this sets-up the business approval required to limit exposure from using this tool by requiring:
2-people (unique user IDs - logins) to interact and approve process to run, and
deliberate introduction of additional roles - outside of the Global Administrator remit.
Multi-stage approval
The workflow stems to capturing approval by another (delegated) operational role. The default in the setup wizard is using users in the eDiscovery administrator role. I.e. that role has to review and approve the creation and deployment of the Priority Cleanup policy.
IMPORTANT: there are checks to ensure that it is not the same user in both the initiator and approver roles
New roles
Microsoft has introduced a requirement for the following additional roles in this process:
Priority Cleanup Admin
Priority Cleanup Reviewer
Search and Purge
Best practice is to create a custom role-groups in Purview with the required list of delegated roles applied. It should be one which has the information mgmt and / or records management controls plus the new roles.
Referred approver is name user in the eDiscovery Administrator role. This is a delegated role hidden in the eDiscovery Manager role-group when you configure to add users to the two roles: eDiscovery Manager, eDiscovery Administrator
IMPORTANT: the user check on the user with eDiscovery Administrator role is not verified until the very end of the wizard. Check you have this correct BEFORE proceeding to create the policy.
KQL
The recommendation from Microsoft is to use the KQL query to target the action to a type of information or a specific library e.g. the Preservation Hold Library:
ParentLink:PreservationHoldLibraryThe configuration wizard in Purview provides the suggestions of:
ProgID:Media AND ProgID:Meeting...for addressing meeting recordings from Microsoft Teams meetings. No other usable options for the other scenarios identified in the online guidance.
Trial and error testing of general KQL options show that while support articles say applied wild-card options in the query works - it doesn't. Using a fully qualified reference for tokens work as expected. Applied options we've used include, but not limited to:
# path references using SiteURL work with full path, and no tailing '/'
SiteUrl=https://<tenancy>.sharepoint.com/sites/<siteurl>
# identification of all items of specific Content type works with full id
ContentTypeId:0x0101 # would affect all documents
# this would affect all items using the ContentType derivative of Document
ContentTypeId:0x010100D5C2F139516B419D801AC6C18942554D
# the following would target all PDFs
FileType:pdfThe combination of these in the query enables very specific targeting on site, or sites.
RECOMMENDATION: practice use of KQL and how to use Search schema mapping for RefinableStrings properties to indexed metadata and properties.
Assessment
Slow, but it works.
The benefit with the Priority Cleanup job is that it works on more than 1000 per day and is registered in the Disposition data. The drawbacks are you don't see the logging in Disposal report or UI - you need powershell to export reporting.
Internal assessment of this feature concurs with Aaron Smith's statement that the "Priority Cleanup", works but it's by no means 'priority' taking upwards of 7-working days to complete the process.
Wrap-up
There are several steps which can cause the process to fail or be overly challenging to report-on/manage:
The main one at the creation of the policy being the users participating have the correct delegated roles for their part in the approval process.
Inability to access reporting details in the Compliance logs, which can be challenging without the full (correct) Policy name and job ID number used to reference the hidden label that Microsoft apply.
Process stalls or fails to run. We have seen 1 instance of this in testing to be able to ensure reconciliation against partial/failed run keep a record list of the site and contents being processed.
RECOMMENDATION: prior to the application of the new policy generate a report which lists, with file-properties and metadata, files to be processed.
Resources
Microsoft Learn | Use priority cleanup to expedite the permanent deletion of sensitive information from mailboxes
A really-good, simple to read article by Aaron Smith
Disclaimer
Generative AI was used in the creation of concept images in this article. No AI was used for screenshots, reference information. Topic ideation for content and practical evaluation of the content presented was entirely the author's output. Template, structure and delivery are based on the Spoke support template.
Need help to develop your operational support? Give us a call
If you want to talk about troubleshooting with Microsoft 365 and SharePoint, or need help with the right-way to go about it, contact us at: hi@timewespoke.com
About the author: Jonathan Stuckey
Comments