Reporting Engine - Enable and configure Data Collectors

Avatar
Nerdio Product Support

This article guides you through the process of enabling and configuring Data Collectors as part of Nerdio Manager's Reporting Engine feature.

Data Collectors are background tasks that run on a scheduled or ad-hoc basis to collect and store data about configured tenants and accounts, which is then used as the basis for reports.

The following Data Collectors are available as part of the Reporting Engine:

  • Users: Data about users, including their details, numbers, roles, status, and licenses.

  • Groups: Data about Group types and assignments.

  • Tenants: Data about your managed customer environments, including domains, storage settings, regions, and subscriptions.

  • Secure Score: Data about your customer tenants' Microsoft Secure Score measurements.

  • Solution Baselines: Data about your customer tenants' status relative to your configured Solution Baselines.

  • Policy Baselines: Data about your customer tenants' status relative to your configured Policy Baselines.

  • Endpoints: Data about Intune-enrolled devices and their policy compliance status.

  • FSLogix Container Sizes: Data about FSLogix containers used for AVD deployments.

  • Desktop Login Activity: Data about user AVD logins.

  • OneDrive: User- and tenant-level data about OneDrive accounts, authentication, sharing settings, and data storage use.

  • SharePoint: Data relating to SharePoint sites in managed customer tenants, including active, orphaned, and archived sites.

  • Exchange Online: Data about Exchange Online mailboxes, including quotas, external forwarding, and tenant-level mail settings.

  • AVD Host Pool Assignments: Data about the users and groups assigned to each AVD host pool.

  • Azure Resource List: Data about Azure resources deployed in linked resource groups for managed customer accounts.

  • Vulnerabilities: Data relating to vulnerabilities identified by Microsoft Defender.

  • Conditional Access Policies: Data relating to conditional access policies applied in managed customer environments.

  • Purview: Data relating to compliance and governance management by Microsoft Purview.

  • Teams: Data about Microsoft Teams user activity (meetings, chats, and calls) over the last 30 days, for users with a Teams license.

  • M365 App Usage: Data about user activity in Outlook, Teams, OneDrive, and SharePoint over the last 30 days, used by the Microsoft 365 App Adoption section.

Note

  • The majority of tasks in this procedure are performed at the MSP level .

  • Per-account storage for reporting data and the account-level Data Collector settings (the CSOM certificate and the SharePoint, OneDrive, and Teams Data Collector switches) are configured at the customer account level.

Prerequisites

Role-based access control (RBAC) and permissions

The following matrix shows the Data Collector configuration permissions associated with built-in Nerdio Manager roles.

Built-in Role

MSP-Level Data Collectors

Account-Level Data Collectors

Account Admin
False
False

Account Help Desk
False
True

(Full permissions)

End User
False
False

MSP Admin
True

(Full permissions)
True

(Full permissions)

MSP Billing Admin
False
False

MSP Help Desk
True

(Full permissions)
True

(Full permissions)

MSP IT Admin
True

(Full permissions)
False

MSP Sales

Page, Read
False

Super Admin
True

(Full permissions)
True

(Full permissions)

Preparatory steps

Before configuring your Data Collectors, we recommend that you plan and review your data storage settings.

Unless otherwise configured, Reporting Engine uses your default MSP-level Azure storage account to store reporting data. Azure storage accounts offer an optimal balance of price and performance for report data compared to other resources like Nerdio's SQL Server, or more advanced storage options like Data Lakes.

If you prefer to use a dedicated storage account for reporting data, you can link an existing storage account or create a new storage account at the MSP level under System > System Settings > Integrations.

For customers who want to limit the amount of information that leaves their own tenants, Data Collectors can also be configured to store reporting data at the account level. Before you can configure account-level storage for reporting data, you must first link an Azure subscription and resource group to the account.

Note

You can change the storage account(s) associated with Reporting Engine at any time.

To change the MSP-level storage account (optional):

  1. At the MSP level, navigate to System > System Settings > Integrations, and scroll down to the Reporting Engine tile.

  2. Select Edit.

  3. Change the Source toggle from Default to Custom.

  4. In the Storage Option dropdown, choose whether to use an existing account or create a new one.

  5. Select the Resource Group to which your storage account belongs (or will belong).

  6. If creating a new storage account, select the Region where it will be created.

  7. If creating a new storage account, enter a name for the account in the Storage Account field.

  8. For the Mode setting, choose whether and how you want to migrate existing data:

    • Don’t migrate account data: All previously collected data will remain in the existing storage location, and all new data will move to the new location.

    • Migrate account data to the new storage account: Move any previously collected data from the original storage location to the new location.

    • Delete existing account data without migration: Delete all previously collected data.

    If you migrate or delete the data, you can also select Delete Old Storage Account to remove the previous storage account. This cannot be undone.

  9. Enter the text CONFIRM to continue, and select Save.

To use an account-level storage account for a customer's reporting data:

  1. At the account level, navigate to Settings > Integrations, and scroll down to the Reporting Engine tile.

  2. Next to the currently listed storage account name, select Edit.

  3. In the Configure storage location dialog, change the Source switch from Msp to Account.

  4. In the Storage Option dropdown, choose whether to use an existing account or create a new one.

  5. Select the Resource Group to which your storage account belongs (or will belong).

  6. If creating a new storage account, select the Region where it will be created.

  7. If creating a new storage account, enter a name for the account in the Storage Account field.

  8. For the Mode setting, choose whether and how you want to migrate existing data:

    • Don’t migrate account data: All previously collected data will remain in the existing storage location, and all new data will move to the new location.

    • Migrate account data to the new storage account: Move any previously collected data from the original storage location to the new location.

    • Delete existing account data without migration: Delete all previously collected data.

    If you migrate or delete the data, you can also select Delete Old Storage Account to remove the previous storage account. This cannot be undone.

  9. Enter the text CONFIRM to continue, and select Save.

Account-level prerequisites for Data Collectors

Some Data Collectors collect data for an account only when the account meets a prerequisite: a default Azure subscription, an enabled account feature, or a Data Collector setting in the account's Reporting Engine integration settings. If a prerequisite is missing, the Data Collector skips that account and records it in the task log, and its tile on the Data Collectors page shows the warning Some required features are missing for the selected accounts with a link to the account's settings. Accounts that do not meet a prerequisite cannot be selected when you configure the Data Collector.

Data Collector

Default Azure subscription required

Account-level feature or setting required

Users

No

None. MFA status and sign-in activity need an Entra ID P1 license and risk data needs Entra ID P2; mailbox sizes need the Exchange Online feature. Without them those fields are left empty and the task log records a warning.

Groups

No

None

Tenants

No

None

Secure Score

No

Secure score management feature

Solution Baselines

No

None

Policy Baselines

No

Modern Work feature

Endpoints

No

Modern Work feature; the tenant also needs an Intune license

FSLogix Container Sizes

Yes

None

Desktop Login Activity

Yes

None

OneDrive

No

OneDrive data collector status set to Enabled in the Reporting Engine tile, which requires a provisioned CSOM certificate

SharePoint

No

SharePoint data collector status set to Enabled in the Reporting Engine tile, which requires a provisioned CSOM certificate

Exchange Online

No

Exchange Online feature

AVD Host Pool Assignments

Yes

None

Azure Resource List

Yes

None

Vulnerabilities

No

Defender for Endpoint feature

Conditional Access Policies

No

None

Purview

No

Purview Management feature

Teams

No

Teams data collector status set to Enabled in the Reporting Engine tile

M365 App Usage

No

None

Account features are enabled at the account level under Settings > Integrations. The Data Collector settings are in the Reporting Engine tile on the same page, next to the storage settings described above:

  • CSOM certificate status: Required by the SharePoint and OneDrive Data Collectors. The status is Not Provisioned, Provisioned, or Provisioned (stored in MSP Key Vault). Accounts that already used those Data Collectors were migrated automatically. See the procedure below.

  • SharePoint data collector status and OneDrive data collector status: Select the status to switch it between Enabled and Disabled. Both are unavailable until the CSOM certificate is provisioned.

  • Teams data collector status: Select the status to enable or disable the Teams Data Collector for the account. It does not need the certificate. Enabling it adds the Reports.Read.All Microsoft Graph permission, which belongs to the Entra permission set, and it works with both the Classic and the granular permission model.

The OneDrive and SharePoint data collectors use a certificate to access the data of the account, and the certificate is stored in a Key Vault. Because it can be stored in the MSP Key Vault, Microsoft 365 customers that do not have an Azure subscription can also use these collectors. The Reporting Engine tile is shown for accounts without a subscription too.

To provision the CSOM certificate for an account:

  1. At the Account level, navigate to Settings > Integrations, and scroll down to the Reporting Engine tile.

  2. Next to CSOM certificate status, select Not Provisioned. The Provision CSOM certificate dialog opens.

  3. Choose where to store the access certificate. Select Store access certificate in MSP Key Vault to use the MSP Key Vault, or leave it cleared to use the Key Vault of the account.

    Note

    If the account has no Key Vault configured and the check box is cleared, the dialog warns Key Vault is not configured for this account. Either configure an account Key Vault or choose to use the MSP Key Vault to provision CSOM certificate. OK is unavailable until you resolve it.

  4. Select OK.

You need permission to update the report data collectors of the account to do this.

Enable, schedule, and run Data Collectors

Before you can report on the data associated with a particular Data Collector, you first have to enable the Data Collector and associate it with one or more schedules. You can also perform an immediate one-time run of the Data Collector.

The Data Collectors page (Insights > Reporting > Data Collectors) shows one expandable tile per Data Collector; use Expand All and Collapse All to open or close them. While a Data Collector runs, its tile shows Running... with the number of accounts completed out of the total (for example, 5/8), and afterwards Last run on with the date of the last run. The Data Collectors Tasks tab lists the data collection tasks and their logs. A tile shows a warning when an assigned account does not meet the Data Collector's prerequisites, or when previously collected data remains for a disabled Data Collector or account; select the link in the warning to clear the data.

A Data Collector run that fails for one account continues for the other accounts; the failure is recorded in the task log. Reports still run for all of their target accounts, with the data that was collected.

To enable a Data Collector:

  1. At the MSP level, navigate to Insights > Reporting > Data Collectors.

  2. Locate the tile for the Data Collector you want to configure, and switch Data Collector from Disabled to Enabled. The Configure Report Data Collector dialog opens.

  3. Optionally, change the Retention Period (Days). The value must be between 10 and 10,000 days.

    Tip

    The default setting is 365 days. Setting a shorter retention period will reduce storage overhead at the cost of reduced access to historical data and the ability to map trends over longer timespans.

  4. In the Accounts field, select the customer account(s) for which you want the Data Collector to obtain reporting data.

    Accounts that do not meet the Data Collector's prerequisites are unavailable in the list; their tooltip names the missing subscription or feature. See Account-level prerequisites for Data Collectors.

    Tip

    By default, Data Collectors are associated with all managed accounts. Selecting only specific accounts to obtain data for (for example, if you offer reporting as a premium service) can reduce both storage overhead and the performance impact on your Nerdio Manager instance, especially in large deployments.

  5. Either select a previously configured global schedule from the Schedule dropdown, or select Create new schedule to configure a new schedule specifically for this task.

    Tip

    Data collection jobs are RAM intensive, especially when running over large amounts of data and/or when multiple Data Collectors are run simultaneously, and can negatively impact the performance of your Nerdio Manager. To mitigate possible performance issues:

    • Run Data Collectors outside of regular business hours

    • Stagger data collection jobs so that only a single Data Collector is running at any one time.

  6. Select Confirm.

Note

  • A Data Collector can have up to 10 schedules. When the limit is reached, the dialog shows the message You have reached the maximum number of schedules (10). If you want to create a new schedule, please delete an existing one.

  • A Data Collector runs at most once per hour for an account. A run that starts less than 60 minutes after the previous run finished, or while a run is still in progress, is skipped, and the task log records that the previous run finished less than 60 minutes ago.

To add a Data Collector schedule:

  1. At the MSP level, navigate to Insights > Reporting > Data Collectors.

  2. In the tile for the Data Collector you want to configure, select Add New Schedule.

  3. Either select a previously configured global schedule from the Schedule dropdown, or select Create new schedule to configure a new schedule specifically for this task.

    Tip

    Data collection jobs are RAM intensive, especially when running over large amounts of data and/or when multiple Data Collectors are run simultaneously, and can negatively impact the performance of your Nerdio Manager. To mitigate possible performance issues:

    • Run Data Collectors outside of regular business hours

    • Stagger data collection jobs so that only a single Data Collector is running at any one time.

  4. Select Confirm.

To amend a Data Collector schedule:

  1. At the MSP level, navigate to Insights > Reporting > Data Collectors.

  2. Locate the tile for the Data Collector you want to configure. A list of the current schedules associated with the Data Collector is displayed.

  3. To amend an already-associated schedule, select the corresponding pencil pencil icon.png icon.

  4. In the configuration popup, you can amend the start date, time zone, start time, or repeat interval of the schedule. You can also use the Schedule dropdown to switch to editing another schedule that has already been associated with the Data Collector.

  5. Once you've made your changes, select Confirm.

To run a Data Collector immediately:

  1. At the MSP level, navigate to Insights > Reporting > Data Collectors.

  2. Locate the tile for the Data Collector you want to run, and select Run Now.

  3. When asked to confirm, select Run.

    The Data Collector will run immediately against all associated accounts.

Data Collector concurrency settings

Two MSP-level settings control how much data collection runs in parallel. At the MSP level, navigate to System > System Settings > Integrations, and in the Reporting Engine tile, under Data Collectors Configuration, set:

  • Max Concurrent Accounts: The maximum number of accounts that each Data Collector processes at the same time. The default is 10.

  • Max Concurrent Data Collectors: The maximum number of different Data Collectors that can run at the same time. The default is 2.

Both values must be between 1 and 100.

Need help?

Raise a support ticket for this item.

Was this article helpful?

1 out of 1 found this helpful
Have more questions? Submit a request

Related articles

Comments (0 comments)

Please sign in to leave a comment.