NSX-V Troubleshooting registration to vCenter

In the current NSX software release, the NSX Manager is tightly connected to the vCenter server in a 1:1 relationship.

During the process of coupling the NSX Manager to vCenter we have two different initial steps: the configuration of “Lookup Service” and “vCenter Server”.

1

Lookup Service:

Lookup Service allows to bind NSX role to SSO user or group. In other word this enable the “Role Based Access Control” authentication functionality in NSX and its optional configuration. Notice that without Lookup service configuration the functionality of NSX is not affected at all.

 

 VCenter Server:

This is a mandatory configuration. Registering the NSX Manager with vCenter injects a plugin into the vSphere Web Client for consumption of NSX functionalities within the Web management platform.

While trying to Register to vCenter or configuring the Lookup Service you might see this error:

“nested exception is java.net.UnknownHostException: vc-l-01a.corp.local( vc-l-01a.corp.local )”

2

Or when trying to setup the Lookup Service:

“nested exception is java.net.UnknownHostException: vc-l-01a.corp.local( vc-l-01a.corp.local )”

3

Or similar to this Error:

“NSX Management Service operation failed.( Initialization of Admin Registration Service Provider failed. Root Cause: Error occurred while registration of lookup service, com.vmware.vim.sso.admin.exception.InternalError: General failure. )”

 

Most of the problems to register NSX Manager to vCenter or configure the SSO Lookup service are:

  1. Connectivity problem between the NSX Managers and vCenter.
  2. Firewall blocking this connection.
  3. DNS not configured properly on NSX Manager or vCenter.
  4. Time is not synced between NSX Manager and vCenter.
  5. The user authenticated via SSO needs to have administrative rights.

 

TSHOT steps

Connectivity issue:

Verify connectivity from NSX Manager to vCenter. Ping from NSX Manager to vCenter using both the IP address and the Fully Qualified Domain Name (FQDN). Check for routing or static information or for the presence of a default route in NSX Manager:

nsxmgr-l-01a# show ip route

Codes: K – kernel route, C – connected, S – static,

> – selected route, * – FIB route

S>* 0.0.0.0/0 [1/0] via 192.168.110.2, mgmt

C>* 192.168.110.0/24 is directly connected, mgmt

 

DNS Issue:

Verify NSX Manager can successfully resolve the vCenter DNS name. Ping from NSX Manager to vCenter with FQDN:

nsxmgr-l-01a# ping vc-l-01a.corp.local

PING vc-l-01a.corp.local (192.168.110.22): 56 data bytes

64 bytes from 192.168.110.22: icmp_seq=0 ttl=64 time=0.576 ms

If this does not work verify the DNS configuration on the NSX Manager.

Go to Manage -> Network -> DNS Servers:

4

Firewall Issue:

If you have a firewall between NSX Manager and vCenter, verify it allows SSL communication on TCP/443 (also allow ping for connective checks).

A complete list of the communication ports and protocols used for VMware NSX for vSphere is available at the links below:

kb.vmware.com/kb/2079386

or

https://communities.vmware.com/docs/DOC-28142

 

NTP issue:

Verify that actual time is synced between vCenter and NSX Manager.

6

From NSX Manager CLI:

nsxmgr-l-01a# show clock
Tue Nov 18 06:51:34 UTC 2014

 

From vCenter CLI:

vc-l-01a:~ # date
Tue Nov 18 06:51:31 UTC 2014

Note: After configuration of Time settings, Appliance needs to be restarted.

 

User permission issue:

Registered user to vCenter or Lookup service must have administrative rights.
Try to work with default administrator user: administrator@vsphere.local

Now the official KB publish at 21/1/15:

KB-2102041

NSX Role Based Access Control

One of the most challenging problems in managing large networks is the complexity of security administration.

“Role-based access control (RBAC) is a method of regulating access to computer or network resources based on the roles of individual users within an enterprise. In this context, access is the ability of an individual user to perform a specific task, such as view, create, or modify a file. Roles are defined according to job competency, authority, and responsibility within the enterprise”

Within NSX we have four built in roles, We can map User or Group to one of the NSX Role. but i think Instead of assigning roles to individual users the preferred way is to assigning role to group.

Organizations create user groups for proper user management. After integration with SSO, NSX Manager can get the details of groups to which a user belongs to.

NSX Roles

Within NSX Manager we have four pre built RBAC roles cover different nsx permission and area in NSX environment.

The four NSX built in roles are: Auditor, Security Administrator, NSX administrator and Enterprise Administrator:

NSX RBAC Diagram

NSX RBAC Diagram

Configure the Lookup Service in NSX Manager

Whenever we want to assign role on NSX, we can assign role to SSO User or Group. When Lookup service is not configured then the group based role assignment would not work i.e the user from that group would not be able to login to NSX.

The reason is we cannot fetch any group information from the SSO server. The group based authentication provider is only available when Lookup service is configured. User login where the user is explicitly assigned role on NSX will not be affected. This means that the customer has to individually assign roles to the users and would not be able to take advantage of SSO groups.

For NSX, vCenter SSO server is one of the identity provider for authentication. For authentication on NSX, prerequisite is that the user / group has to be assigned role on NSX.

NSX Manager Lookup Service

NSX Manager Lookup Service

Note: NTP/DNS must configure on the NSX Manager for lookup service to work.

Note: The domain account must have AD read permission for all objects in the domain tree. 

Configure Active Directory Groups

In this blog i will use Microsoft Active directory  as user Identity source.  in “Active Directory Users and Computers” i created four different groups. The groups will have the same name is the NSX roles to make life easier:

Auditor, Security Administrator, NSX Administrator, Enterprise Administrator.

AD Groups

AD Groups

We create four A/D users and Add each user to different A/D group. for example nsxadmin user:

the user nsxadmin is associate with the group NSX Administrator. the association done by the Add button:

AD user

AD user

Same way i will associate a others users to A/D groups:

username:     groups:

auditor1      ->  Auditor

secadmin ->   Security Administrator

nsxadmin ->  NSX Administrator

entadmin ->  Enterprise Administrator

Connect Directory Domain to NSX Manager.

Go to “Network & Security” tab double click on the “NSX Manager”

map ad to nsx manager role 1

map ad to nsx manager role 1

Double click on “192.168.110.42” icon:

map ad to nsx manager role 2

Note: Configure Domain is not needed for RBAC, only if we want to use identity firewall rules base of user or group.

Go to “Manage -> “Domains” -> Click on the green Plus button:

map ad to nsx manager role 8

Fill Name and NetBIOS name fields with appropriate information of your Domain Name and NetBIOS name:

In My example my domain name is corp.local:

map ad to nsx manager role 9

Enter LDAP (i.e AD) IP address or hostname and domain account (username and password):

map ad to nsx manager role 10

Configuring LDAP option task  can be done via direct API call to bypass the Event Log Access described in the next steps).

Click on next.

Event Log Access:

In case we need to create NSX firewall rule with user identity based on AD groups. We will need to allow the NSX Manager read Active Directory “Security Event Log”. This logs contain Active Directory users logon/logoff from to domain. We use this information to bind the AD user  to an IP address.

NSX need access to “Event Log” provide dFW with user identity in one of the two case:

  1. The user logon to VM that doesn’t running VMtools.
  2. The user logon to the domain from PC located on physical environment.

BTW users login to to VM with VMtools up and running , we do not need the “Security Event Log” to bind the user to IP.

Permissions for the user to read logon/logoff events:

Windows 2008 or later domain servers:

Add the account to the Event Log Readers group. If you are using the on-device

User-ID agent, the account must also be a member of the Distributed COM Users Group.

 

 Windows 2003 domain servers:

Assign Manage Auditing and

Security Logs permissions through group policy

In both of this cases NSX will need to access the AD with read permissions for security event logs, the protocol using to read this information are CIFS or WMI.

During this process NSX  collecting  the following microsoft event ID:

For windows 2008/2012 – Event ID: 4624

For Windows 2003 – Event ID: 540

NSX will “Copy” this Event access log and from A/D and parse the data inside the nsx manager appliance.

map ad to nsx manager role 11

Click Next and Finish:

map ad to nsx manager role 12

Mapping Active Directory  Groups to NSX Managers Roles

Note: This step is must for NSX RBAC to work. 

Now we can map Active Directory groups to pre-built NSX Manager roles.

Go to “Manage -> “Users” -> Click on the green Plus button:

map ad to nsx manager role 3

Here we can select if we want to map specific A/D user to NSX Role or A/D Group to Role.

map ad to nsx manager role 4

In this blog i will use A/D group, we create A/D group called auditor. The format to input here is:

“group_name”@domain.name.  let’s start with auditor group, this group is “Read Only” permission:

map ad to nsx manager role 5

Select one of the NSX Role, for Auditor A/D group we chose Auditor

map ad to nsx manager role 6

We can limit the scope this group can work inside nsx manager object, for this example there is no limit:

map ad to nsx manager role 7

Same way Map all others A/D groups to NSX Roles:

Auditor@corp.local                           – >  Auditor

Security Administrator@corp.local        -> Security Administrator

NSX Administrator@corp.local               -> NSX Administrator

Enterprise Administrator@corp.local     -> Enterprise Administrator

Try our first login with user Auditor1:

Login1

 The login successfull but where is the “Network & Security” tab gone ?

Login2

So far we configure all NSX Manager part but we didnt take care of the vCenter Configuration permission for that group. are you confusing ?

vCenter has is own Role for each group. we need to configure roles to etch A/D group we configured. These settings determine what the user can make the in vCenter environment.

Configure vCenter Roles:

Let’s start by configure the Auditor Role for Auditor A/D group. we know this group is for “Read Only” in the NSX Manager, so it will make sense to give this group “Read Only” to all other vCenter environment.

Go to vCenter -> Manage -> Permissions and click the green button:

vCenter Roles 1

We need to choose Roles from the Assigned Role, if we select No-Access we will not be able login to vCenter. So we need to choose something from “Read-Only” to “Administrator”

For Auditor Role “Read Only” is the Minimum.

Select “Read Only” from the Assigned Role drop down list and click on the “Add” button from “User and Group”:

vCenter Roles 2

From the Domain Select your Domain name, in our lab the domain is “CORP”, choose your Active Directory group from the list (Auditor for this example) and click the “Add” button:

vCenter Roles 3

Click Ok and Ok for Next Step:

vCenter Roles 4

Same way we need to configure all other groups roles:

vCenter Roles 5

Now we can try to login with auditor1 user:

auditor1

As we can see auditor1 is in “Read Only” role:

auditor2

We can  verify that auditor1 can’t change any other vCenter configuration:

auditor3

Test secadmin user map to “NSX Security” role, this user cannot Change any NSX infrastructure related task like create new  add new NSX Controller Node:

secadmin1

But secadmin can create new firewall rule:

secadmin2

When logging with nsxadmin user map to NSX Administrator Role we can see that the user can add new Controller Node:

nsxadmin1

But nsxadmin user cannot change or see any firewall rules configure :

nsxadmin2

What if the user member of two A/D Group ?

The user will gain combined permission access of both of the groups.

For example: the user memberof “Auditor” group and “NSX Security”, the results will be user will have read only permission on all nsx infrastructure and also gain access to all security related area in NSX.

Summery

In this post we demonstrate the NSX manager different roles. We configure Microsoft Active Directory as External database source for user’s identity.

Improving NSX GUI user experience

Working with NSX on different environments I found that the vSphere Web client could work slowly. After hearing others users complains about those problems, I decided to write some tips for improving the overall user experience.

Fist try to login with local default user: administrator@vsphere.local; if this work faster than an LDAP user then please try to switching from using user@domain to domainuser.

  1. Increase the java memory limit for vCenter. The following VMware published KB article offers more information on how to do that:
    http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2021302
  2. Install the Client Interaction plugin. You should see the message on the vCenter login page:
client integration plugin

client integration plugin

3. Increase the local storage settings on the Flash Player will also speed up the web client.
Adobe have online tool to view and change the local storage setting:
http://www.macromedia.com/support/documentation/en/flashplayer/help/settings_manager07.html

Flash Storage Settings panel

Flash Storage Settings panel

 

Note: if the PC you are using to connect to vCenter does not have access to Internet this Adobe link will not work. Thanks to Micha Novak and Yaniv Yaakov (my team colleagues) for this tip.

When we load the Web Client in the blue Screen try to fast “right click” on your mouse, then click on the Settings button.

Flash Storage Seeting1

Swipe right the scroll bar:

Flash Storage Seeting2

4. Remove idle sessions from you vCenter

vCenter idle session

vCenter idle session