Unable to remove permissions in vCenter Appliance 6.0 U2, how to fix

Luciano PatrãoICT Senior Infraestructure  Engineer  
vExpert vSAN, NSX, Cloud Provider, Veeam Vanguard, Virtual Backups, and Storage design, and an active blogger.
This is an issue that we can get adding / removing permissions in the vCSA 6.0. We can also have issues searching for users / groups in the AD (using your identify sources). This is how one of the ways to handle this issues and fix it.
This issue happens in vCenter Appliance 6.0(VCSA).
First query users and Groups in the Domain (searching to add in the permissions) was very slow and in the Web Client sometimes we see the Domain with double entries, or could not find the users/groups.
Also when we look at the vCenter permissions(vCenter level, or just Folder/Cluster level) we see also the user/group with double entries(like domain.com\user and domain\user) and when try to remove any of these permissions one was always resident and will not be deleted. No errors, no warnings, just can't delete.
Note: vsphere.local users and groups did not show this behavior. So my focus needs to be in the AD and Identify Sources to troubleshoot.
Try to Stop/restart the Services to see if it fix the issue, but no luck. Rebooting the VCSA also did not work. While troubleshooting the issue, and knowing that there was double AD entries, somehow Domain alias and Domain Full Name was set in the system identity. Checking logs did not find anything related that could help(just some references to domain and domain.com), not the issue itself.
Googling the issue, did not find anything related to the same issue(exactly), but did find a VMware KB regarding some VCSA upgrade(that was not the case here, since is a new installation), but it shows a workaround to fix the issue regarding vCenter Server Appliance database to have the full domain name instead of an alias. Since we had both, it makes since to me that this could fix the issue.
How to fix the issue:
Before started, do a full backup of your VCSA, or at least do a Snapshot. In case of problems, you can always rollback.
So login to your VCSA using shell and root user.

Note: Don't forget that after you login you need to enable the BASH shell and run it. If you get: Shell is disabled, then you need to run the shell.set to enabled.
Using username "root".
                      VMware vCenter Server Appliance
                      Type: vCenter Server with an embedded Platform Services Controller
                      Last login: Thu Sep 22 22:09:53 UTC 2016 from xx.xx.xx.xx on ssh
                      Last login: Thu Sep 22 22:22:34 2016 from xx.xx.xx.xx
                      Connected to service
                          * List APIs: "help api list"
                          * List Plugins: "help pi list"
                          * Enable BASH access: "shell.set --enabled True"
                          * Launch BASH: "shell"
                      Command> shell.set --enabled True
                      Command> shell
                          ---------- !!!! WARNING WARNING WARNING !!!! ----------
                      Your use of "pi shell" has been logged!
                      The "pi shell" is intended for advanced troubleshooting operations and while
                      supported in this release, is a deprecated interface, and may be removed in a
                      future version of the product.  For alternative commands, exit the "pi shell"
                      and run the "help" command.
                      The "pi shell" command launches a root bash shell.  Commands within the shell
                      are not audited, and improper use of this command can severely harm the
                      Help us improve the product!  If your scenario requires "pi shell," please
                      submit a Service Request, or post your scenario to the
                      https://communities.vmware.com/community/vmtn/vcenter/vc forum and add
                      "appliance" tag.
                      hostname:~ #

Open in new window

After you are inside your VCSA console, you need to connect to the postgress database and set your DB to use Full Domain Name.

hostname:~ # cd /opt/vmware/vpostgres/current/bin
                      hostname:~ # ./psql -d VCDB -U postgres
                      psql.bin (9.3.9 (VMware Postgres release))
                      Type "help" for help.
                      VCDB=# update CIS_KV_KEYVALUE set KV_KEY = replace(kv_key,'DOMAIN%5C','DOMAIN.COM%5C');
                      VCDB=# update CIS_KV_KEYVALUE set KV_VALUE = replace(kv_value,'DOMAIN\','DOMAIN.COM\');

Open in new window

Note: Just replace the ‘DOMAIN’ for your alias Domain, and ‘DOMAIN.COM’ for your Domain Full Name.

Then just reboot your VCSA and the issue is fixed.

Final Note: What triggered this issue? In my case I was not able to identify the source of the issue, but I suspect that by changing our Identity Sources to a different LDAP(that is in a different location and in a different country) could have a different trust in the Forest and did trigger the issue. Since VMware informs that one of the source for this type of issues could be problems connecting to LDAP server and update the changes that we make in our permissions.

Hope this can help you.

This is the part of my "TIP Articles". So, please vote "Helpful" on this Article. And I encourage your comments and feedback.

Luciano Patrão.
Luciano PatrãoICT Senior Infraestructure  Engineer  
vExpert vSAN, NSX, Cloud Provider, Veeam Vanguard, Virtual Backups, and Storage design, and an active blogger.

Comments (0)

Have a question about something in this article? You can receive help directly from the article author. Sign up for a free trial to get started.