GetGraphUserStatisticsReport.PS1 Line 84 - Item has already been added. Key in dictionary about office365itpros HOT 13 CLOSED

BTW, I decided to go with hash tables to speed up performance. You can test things out with the V2 version of the script.

Turbocharging the Analysis of Office 365 Data with PowerShell Hash Tables

PowerShell hash tables are very efficient at retrieving data, which is just what’s needed when thousands of Office 365 accounts need processing. Our script to analyze usage data extracted from the Microsoft Graph was turbo-charged when we replaced list objects with hash tables, all of which makes it much easier to identify underused Office 365 accounts and save some money on licensing spend.

https://office365itpros.com/2020/09/14/speed-powershell-hash-tables-office365-data/

from office365itpros.

I have the same issue as @brandonbrownchenega
It looks like the script wants to add the first 999 users again.

from office365itpros.

Looks like the script has been updated with a fix, all working for me now!

from office365itpros.

I still have the same issue, the update didn't work for me, unfortunately.

I think I know what the problem is, but I don't know how to solve it.

The user signing data is fetched with the top 999, this works well. but when fetching the others (if there are any) is starts with the
$NextLink = $SignInData.'@Odata.NextLink'
In the @odata.nextlink part the &top999 is also added, so it starts adding them again.

Hopefully, this can be altered, because otherwise this is a great script to get insight in you tenant

from office365itpros.

Hmm, that's really odd because it's been working great for me. Are you getting the same error as V1 of the script, or a different one?

from office365itpros.

I'm getting the same error as with V1. it doesn't matter if I run it in ISE or Powershell (and VSCode). Also run as admin doesn't matter, I keep getting the error.
If I comment that part out, I will get a nice result. Only the login info is only available for 999 people, and the rest gets a N/B.

from office365itpros.

I'm running in to an issue now where the data is taking forever to process after being pulled down from Graph. Previously the script was taking just shy of an hour to run for ~3400 accounts and now it's looking it's going to take 2-3 days. Can't tell if that's me getting throttled on the Graph side or a local processing issue.

from office365itpros.

I'm seeing the "Item has already been added. Key in dictionary:" error when running the script on a machine that hasn't completed the first run of Internet Explorer (e.g. on a brand new virtual machine).

The Invoke-WebRequest command on Line 80 throws up an error immediately beforehand:
Invoke-WebRequest : The response content cannot be parsed because the Internet Explorer engine is not available, or Internet Explorer's first-launch configuration is not complete. Specify the UseBasicParsing parameter and try again.

A fix is to add the -UseBasicParsing parameter to the end of the Invoke-WebRequest command on Line 80, similar to what's on the Invoke-WebRequest command on Line 24.

I've added a Pull Request with that update.

from office365itpros.

Hi @SteveBurkettNZ ,

Thanks for your comment, this works for me!!

from office365itpros.

I'm running in to an issue now where the data is taking forever to process after being pulled down from Graph. Previously the script was taking just shy of an hour to run for ~3400 accounts and now it's looking it's going to take 2-3 days. Can't tell if that's me getting throttled on the Graph side or a local processing issue.

It's a local processing issue, where it's running the ForEach ($U in $Users) loop, repeatedly searching through the $Report array for relevant entries for that user. It's slow, taking just over 37 minutes per 1000 users for me on a 2 vCPU VM (Azure VM Standard D2s v3).

I've modified to use a DataTable which is much more efficient for searching larger datasets, and managed to get it down to 2 minutes 11 seconds per 1000 users. I've just found though that my dataview filter chokes if the user has an apostophe in their name (darn you Maricel D'Souza!) so I need to fix first. :(

from office365itpros.

I've committed V1.2.

I played around with replacing the List object used for $Report with a DataTable object, but the performance fix is really much simpler.

$Report contains all the records found from the Graph. The performance hit comes from scanning what can be a very large list to find records for a user, which happens six times (once per workload) in V1.1. I inserted a line to extract the data for the user being processed (usually six records):

$UserData = $Report | ? {$_.UPN -eq $U} # Extract records from list for the user

And now the six checks are performed against $UserData instead of the complete $Report list. This speeds things up quite a lot (it would be interesting if someone with a really big domain could test).

from office365itpros.

Thanks for this! All my problems with slow information processing have been solved now.

from office365itpros.

Yup, v2 working a treat:

Statistics for Graph Report Script V2.0

Time to fetch data from Microsoft Graph: 3:5
Time to prepare date for processing: 0:7
Time to create report from data: 5:12
Total time for script: 8:25
Total accounts processed: 26808
Accounts processsed per minute: 3184.71

from office365itpros.