Feedback from customer Take Away:
a. Summaries are good
b. Help me with Next Steps

Customer requested a list of commands (next steps suggestion) to find more information about their environment. For ex, how to find VTEP Table or ARP Table etc.

As the output of following commands are very dynamic, there is no value in dumping the output in the current Excel file. Instead, just guiding the customer with pre-populated commands is more useful as they don't have to figure out the commands and can just copy paste it to run.

List of commands for Hosts file:

• View VTEP Table
• View MAC Table
• View ARP Table
• Execute Ping
• May be do the same for pktcap-uw

To be able to troubleshoot issues and problems with install / modules etc, it'd be good to log the session in the same directory that is auto created when the tool is run. This way any errors are all recorded and can be shared.

When a issue happens and we want users to be able to troubleshoot or send us some information we should have a way for them to do that. In addition to #36 this can be a place for us to gather info like

PowerShell Version
PowerCLI Modules loaded
PowerOps Dependency modules loaded
Force reinstallation or check for updates on the modules.

For ex, after initial install, if powerNSX needs to be updated, currently the Install Dependencies gets greyed out and there is no way for user to force update.

avoids name collision if user is manually comparing across report directories.

Add new tab for Logical Switch (VXlan) in NSX Routing excel.

• show command output of: get-nsxlogicalswitch only name and vdnId
• show the list of VXlan with the total number of VMs connected to it

$vxlan = 0
$VMs = Get-VM | Select Name, @{N="PortGroups";E={Get-VirtualPortGroup -VM $_ | %{$_.Name}}}
$PGs = $VMs | %{$_.PortGroups} | Select -Unique
$myCol = @()
ForEach ($PG in $PGs)
{
  $myObj = "" | Select PGName, NumVMs
  $myObj.PGName = $PG
  $myObj.NumVMs = ($VMs | Where {$_.PortGroups -contains $PG} | Measure-Object).Count
  $myCol += $myObj
}
#$myCol | Sort NumVMs -Descending | Format-Table -AutoSize

$myCol | %{if ($_.PGName.StartsWith("vxw-dvs")){
$_
$VXlan ++}}
Write-Host "`nTotal number of VxLAN are: $vxlan"

Reported by @nmbradford

Remove-Item : Cannot remove the item at 'C:\users\nick\Documents\git\NSX-PowerOps' because it is in use.
At C:\users\nick\Documents\git\NSX-PowerOps\nsx-PowerOps.ps1:376 char:9


        Remove-Item ./$nsxMgrCommandRouteTable
        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~


CategoryInfo          : InvalidOperation: (:) [Remove-Item], PSInvalidOperationException
FullyQualifiedErrorId : InvalidOperation,Microsoft.PowerShell.Commands.RemoveItemCommand

appears related to DLR route table query. Suspect we could avoid the use of temporary files and build a hashtable instead. Also, spotted use of redirection in invokeNSxCLI -> should leverage set-content rather than relying on external functionality to PowerShell. Will look to refactor.

As a Developer: The VIB test runs against all the host in vCenter inventory. We should only be running it against the host in cluster that is prepared with NSX.

Currently, when the VTEP to VTEP test is run for single host, it asks the user:

Please provide the Host ID:

It gives the impression that you have to enter the MOID of the Host from vCenter. In reality it needs the name or IP used by vCenter to add the host to the vCenter. So if the ESXi host is added to vCenter with name for ex, 63esxcomp1.powerops.local then that needs to be the input. If the host is added to vCenter using the management IP of the host then that needs to be entered here.

We need to update the wording to something like:

Please provide the name/ip used to add the ESXi host to vCenter:

As a developer: change the health check 'NSX Replicator Service Test' under 'NSX Manager Test'. So, new logic first checks if NSX role is Primary or not. By executing following command:

Get-NsxManagerRole

If NSX role status is 'PRIMARY', check NSX Replicator Service is 'Running'. By executing following command:

$ComponentSummary = (Get-NsxManagerComponentSummary -connection $global:NsxConnection).componentsByGroup.entry.components.component

If NSX role status is anything other than PRIMARY, check NSX Replicator Service is 'STOPPED'.

Pass the test If above conditions matches otherwise throw an error.

Reported by @2cool2touch

We can add a new tab in the Component summary to list the ESXi hosts prepared with NSX overview. Ideally, this should have an output very similar to Host Installation tab in NSX :

Cluster Name, Host Name, ESXi Version, NSX Version, VDS prepared for NSX, Teaming Policy, # of VMks, CPU, Memory

PS C:\NSX-PowerOps-Nov1> get-vmhost | Select Name, Version, Build, PowerState, State, NumCpu, CpuTotalMhz, MemoryTotalGB | ft -auto

        Name                         Version Build   PowerState     State NumCpu CpuTotalMhz         MemoryTotalGB
        ----                         ------- -----   ----------     ----- ------ -----------         -------------
        esx01-s1.core.hypervizor.com 6.5.0   4887370  PoweredOn Connected      8       22400 21.999439239501953125
        esx02-s1.core.hypervizor.com 6.5.0   4887370  PoweredOn Connected      8       22400 21.999439239501953125
        esx03-s1.core.hypervizor.com 6.5.0   4887370  PoweredOn Connected      8       22400 11.999439239501953125
        esx04-s1.core.hypervizor.com 6.5.0   4887370  PoweredOn Connected      8       22400 11.999439239501953125

Currently the healthcheck outputs are saved as XML files. If saved and formatted as HTML, it will improve usability of the reports.

Ideally, add a new option to run all the non-interactive healthchecks with one click and generate one HTML report.

by Nick Bradford @nmbradford

The attempt to access $esxcli.network.vswitch.dvs.vmware.vxlan.list.invoke() fails due to the vxlan namespace not being present. This can occur on certain version of vSphere/NSX and requires a cycle of hostd to resolve ( was not available on host ssh either). This can lead to the documentation process to incorrectly claim that a host is not enabled for VXLAN when it actually is. An alternative would be to gather vmk information directly from network.ip.interface | ? { $_.netstack -match 'vxlan' } (vmk name, mtu and vds name are available in list.invoke(), ip and subnet in network.ip.interface.ipv4 and gateway is in network.ip.netstack)

Nick Bradford @nmbradford
simpler alternative to refactor is to warn in the catch that the host may just need hostd recycled.

when creating connection profiles, the name of the vc that is being connected to when prompting for creds should be visible.

menus that launch excel tasks or visio tasks, should be disabled when the resp dep is not installed.

As a developer: Current vTEp-to-vTEP ping test calls invoke-sshcommand to make the ping test. Change it to $esxcli network.diag.ping.Invoke() as customers might not have SSH enabled on Host(s).

This is due to HA-specifc route '169.254.1.0 255.255.255.252 169.254.1.1 C ' on DLR. This route is not pushed to the routing tables on ESXi hosts which causes the health check to fail.

The script needs to be updated to check if DLR HA is enabled and ignore this route when comparing routing tables.

If we look at the factors that need to be analyzed for NSX Microseg Deployments, especially scaled, they are all over the place and a real trip to find them thru traditional tools. When reviewing designs, these number aren't available and/hence people aren’t thinking about them either.

Some of the things that come to mind are

• number of VMs on NSX Prepd clusters (for VM/host ratio)
• number of vnics with DFW dvfilter (slot 2)
• number of vnics with non-DFW dvfilters (slot 4-7)
• number of universal sections
• number of universal rules
• number of global sections
• number of global rules
• number of Security Policies
• number of redirect rules
• number of Universal SGs
• number of Global SGs
• number of objects in each SG
• number of Universal Security Tags
• number of GLobal Security Tags
• number of VMs tagged with each ST
• number of tags per vm
• number of rules per vnic
• number of rules per host
• heap utilization per host
• Services deployed (GI, NI etc)
• Number of clusters prepared with services
• objects in services container set (GET /api/2.0/si/serviceprofile/serviceprofile-1/containerset)

I am not sure if this should go in the DFW2Excel or a separate menu item that gets called on demand. It can probably go under the Security Documentation sub-menu as Security Deployment Overview. Some of the APIs here may take a while to return in large environment. Keeping it separate will ensure regular DFW2Excel runs are not impacted.

We should review these and identify which can be done quickly and target them in this ticket. We can open new tickets and punt the rest to them.

Thoughts?

helthcheck output should be to report dir.

When implementing NSX DFW Security at large scale, the security group definition must be optimized in a way that they have least impact on NSX Manager and are optimized for performance. One of the common issues seen is where the Security Groups are created with Dynamic Criteria of 'Security Tag' 'equals' ''. This should be replaced with either Entity Belongs to --> Security Tag or a Static Include.

PowerNSX examples have a script to fix this issue. In PowerOps, we only want to perform the detection piece and point the user to PowerNSX script that they can run on their own to improve the design.

Convert2EntityBelongsTo.ps1
https://github.com/vmware/powernsx/blob/master/Examples/Convert2EntityBelongsTo.ps1

Running the script with no parameters will just read your configuration and see if it finds any instances of dynamic criteria configured with "Security Tag" & "Equals to".

C:\Users\Administrator\Documents> .\Convert2EntityBelongsTo.ps1
18:19:51 : Line(187) : Log file not found... creating a new one
18:19:51 : Line(187) : Logging to file .\Convert2EntityBelongsTo_2018_02_28_18_19_51.log
18:19:51 : Line(187) : Retrieving local NSX Security Groups
18:19:51 : Line(199) : Searching for Security Groups with 'Security Tag' & 'Equals' configured in dynamic criteria
18:19:51 : Line(299) : Script complete in 00:00:00.2184004
18:19:51 : Line(300) : Debug log saved to .\Convert2EntityBelongsTo_2018_02_28_18_19_51.log
C:\Users\Administrator\Documents>

Customize the code to setup an email account and automatically sent the email with all the documents (excel reports) to user after successfully running a scheduled task.

Reported by @nmbradford

You cannot call a method on a null-valued expression.
At C:\users\nick\Documents\git\NSX-PowerOps\nsx-PowerOps.ps1:317 char:12


        if($getDLRs.gettype().BaseType.Name -eq "Array"){
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~


CategoryInfo          : InvalidOperation: (:) [], RuntimeException
FullyQualifiedErrorId : InvokeMethodOnNull

Occurs because we don't check that $getDlrs exists before this which will occur if no DLR is configured.

• Check the same for Edges.

Based on initial conversation, @tonysangha - seems like I forgot to create an issue for it:

If we look at the factors that need to be analyzed for NSX Microseg Deployments, especially scaled, they take some time to collect and you need deeper understanding of NSX to do that too. A lot of times either this is not available or people aren’t thinking about them. If we can automate it, this can be real handy for NSX Customers. Some of the things that come to mind are

• Number of Clusters prepared with NSX
• Number of Hosts prepared for nSX
• number of VMs on NSX Prepd clusters
• number of vNics with dvfilter
• number of Universal rules
• number of Global rules
• number of Security Policies
• number of Redirect rules
• number of SGs
• number of objects in each SG
• number of STs
• number of VMs tagged with each ST
• number of Tags per vm
• number of Rules per vnic
• number of Rules per host
• heap utilization per host
• Number of Logical Switches

This would ideally go under Security Documentation for now. Later when we add more things to it, then it can be moved under Environment Documentation too.

VTEP-to-VTEP test excel should be saved in same place where other Excels are created.

Ideally, the DLR routing table on all hosts should be the same. This test should let customer run a test which asks

a. which DLR he wants to run the test for (show list of DLRs to choose from or something like that)
b. get the routing table from all the hosts
c. compare the routing table
d. show WARNING when the routing table of some hosts are different and identify the hosts
e. allow the export of test to a text file. I am hoping/assuming that spitting output to text would be much faster then Excel

Currently the version displayed is 0.1. As discussed can we change it to 2.0.

Under Connection Profiles, Add a way to disconnect from current connection and connect using a different profile

disconnect-viserver
disconnect-nsxserver

Currently the only way is to change the default profile to use different connections.

Currently we do not document Redirect rules. Feedback from field is that it would be very helpful to have that information documented.

@tonysangha

Allows 'offline' use of tool (demo). Could potentially support a demo flag if easier.

@thisispuneet -

Readme needs update

a. Releases & Major Branches

Current version: BETA . This should say 2.0

b. the Visio download text says Visio 2016 but the link posted is Visio 2013.

MS VISIO 2016 activated as Standard or Professional (optional - some feature wont work if not installed) [https://products.office.com/en-us/microsoft-visio-2013]

Can we remove the install details in the Readme and just point to http://www.vcdx248.com/2018/02/step-by-step-install-deploy-guide-of.html where we have step by step (including screenshots) of installation.

Add section names as per previous iteration of menu system, so menu items can have optional section name that they are grouped under.

We need an option like a 0-Run All Documentation. In demos and for customer's manual run, it becomes cumbersome to wait for one to finish before user has to do the next one. We should add an option say 0-Run All Documentation. Ask the user if they want DFW (small) or (Large) and just go about running them all.

Could anyone suggest me how to convert .xml helthcheck output to HTML format ?

Reported by @2cool2touch

Starting NSX 6.3.3 the name of the VIBs has changed from esx-vxlan and esx-vsip to esx-nsxv.

We need to add a check if the environment is running NSX Version <6.3.2 then check for esx-vxlan/esx-vsip and for 6.3.3 and later check for esx-nsxv.

Reference:
https://docs.vmware.com/en/VMware-NSX-for-vSphere/6.3/rn/releasenotes_nsx_vsphere_633.html

In NSX 6.3.3, NSX VIB names change on ESXi 6.0 and later. VIB names on ESXi 5.5 remain the same.
The esx-vxlan and esx-vsip VIBs are replaced with esx-nsxv if you have NSX 6.3.3 installed on ESXi 6.0 or later.

Currently the VTEP to VTEP test only runs against single site. It'd be great to extend this to support multi-site NSX setups.

Once you start the tool and if you chose not to auto connect or it failed for some reason. We need a way from within the tool to kickstart connection again.

In addition, if you are connected using a profile, but would like to connect using a different (non-default profile). There should be a way to switch the profile and connect using that without exiting.

May be we can add a #-4 in the Connection Profiles which says connect to selected profile.

This can also be useful if from within the tool you want to disconnect and then connect to a new profile that was just created.

May be there is a way but is not obvious.

show edge <edge-id> service highavailability

show edge edge-cf406e2d-15fe-4679-9041-b7bf335e9103 service highavailability
haIndex:              1
Highavailability Service:
Highavailability Status:             Active
Highavailability State since:        2017-02-14 01:59:15.995
Highavailability Unit Id:            1
Highavailability Unit State:         Up
Highavailability Admin State:        Up
Highavailability Running Nodes:      0, 1
Unit Poll Policy:
   Frequency:                       3.75  seconds
   Deadtime:                        15    seconds
Highavailability Services Status:
   Healthcheck Config Channel:      Up
   Healthcheck Status Channel:      Up
Highavailability Healthcheck Status:
   Peer unit [0]: Up   Active: 0
       Session via vNic_0: 169.254.1.14:169.254.1.13 Up
   This unit [1]: Up   Active: 1
Config Engine:
   HA Configuration:               Enabled
   HA Admin State:                 Up
   Config Engine Status:           Active
Highavailability Stateful Logical Status:
   File-Sync                       running
   Connection-Sync                 running
     xmit       xerr       rcv        rerr
     124797096  0          125645520  0

Often time when troubleshooting a VM related issue, it takes a lot of effort for customers to gather information. This troubleshooting menu will ask user to give either VM IP | Name | MOID and we will gather relevant information from vC and NSX about the VM. This includes VM deployment and also DFW Rules from the vNICs

That will keep host access consistent across all health check tests

when launching and a dep is not satisfied, the progress banner is not cleared.
when creating a connection profile, 'loading from config' progress banner is not cleared

Should prompt for user to connect to default or just launch without connection. Will need to add 'connect / disconnect' menu item.

As discussed on the emails and feedback from RackSpace, can we expose some filtering mechanism to the input of the script.

TenantID is something that can be filtered on already but if we can expose it as an input when someone runs the script, would be great.

Secondly, if we can do exact samething as tenantID to Edge Name as so far customer feedback is that the ESG names usually carry some meaning especially in multi-tenant environments.

Currently when VIB version test is run, it asks user to input the expected VIB version on all hosts. This can be improved by automatically detecting the expected VIB version for the running NSX Manager and then looking for that version across the hosts. It is expected that if all the hosts have not yet been upgraded, some errors will be thrown which is the also an intent of the tool to show where update is needed.

NSX-PowerOps uses PowerNSX for number of the tasks. In the recent version of PowerNSX some new features around use of uriPrefix have been introduced which require any NSX-PowerOps install to have PowerNSX 3.0.1091.

Currently, NSX-PowerOps does not enforce or check for PowerNSX version. Hence, even if the user has an older version of PowerNSX installed, our script wont ask the user to upgrade. This ticket is to see what is the best way to add a version check of PowerNSX to NSX-PowerOps.

When the tool is executed, if the required modules are not present, we disable all the menu options and only enable the INSTALL DEPENDENCIES option. We need a way for this trigger to happen when the minimum version of PowerNSX is not there.

Proposal:
Currently when the NSX-PowerOps script starts, it checks to see if the required modules (PowerNSX, Pester, POSH etc) are installed. We should explore if instead of just checking the existence of the module, we add to it a check of the version of PowerNSX, Pester and POSH and hard code a minimum version. For PowerNSX, currently it would be 3.0.1091. If that minimum is not met, we will declare that Modules are not installed which should trigger the option of Install Dependencies.

Basically, most of the mechanism is already there, we probably just need to add the version number check into the module check and the existing code should handle most of the rest.

currently healthchecks still prompt for creds