A while back, I was noticing some strange behaviors with some parts of the vSphere web client. Primarily, it was related to vSAN details.
Quickstart configuration wouldn’t load, status on a couple of the vSAN services would get stuck at loading, an error message about failed to extract requested data.
Turned out to be related to SSL certificates in some of the underpinnings.
Came across a Reddit thread:
https://old.reddit.com/r/vmware/comments/cxbk24/vsan_67u3_error_failed_to_extract_requested_data/
The solution was a python script for checking trust anchors.
https://web.vmware-labs.com/scripts/check-trust-anchors
Using the -cml switch to do a live check on machine certificates and colorize the output.
For some reason, one of the endpoint certificates being used was the original self-signed certificate. This should have been replaced, but it seems it had not.
[email protected] [ /tmp ]# ./check-trust-anchors -cml No 'lstool.txt' file found in this directory. Dumping service registrations to /tmp/lstool.txt... -----Endpoint Certificate 1----- Certificate Info: Issuer: CN=CA, DC=vsphere, DC=local, C=US, ST=California, O=vcenter.incendiary.local, OU=VMware Engineering Validity Not Before: Aug 11 22:57:30 2019 GMT Not After : Aug 5 22:57:29 2029 GMT Subject: CN=vcenter.incendiary.local, C=US SHA1 Fingerprint=51:EA:79:CE:81:69:CB:A6:0E:3B:47:42:4C:8D:28:68:94:3C:46:0D -------------------------------- -----Endpoint Certificate 2----- Certificate Info: Issuer: CN=incendiary-ROGUE-CA Validity Not Before: Oct 11 19:44:16 2019 GMT Not After : Oct 8 19:44:16 2029 GMT Subject: C=US, ST=Alaska, L=Anchorage, O=incendiary, CN=vcenter.incendiary.local SHA1 Fingerprint=4D:00:81:D9:F9:6E:06:38:57:38:ED:C1:31:78:BD:E0:F9:54:E8:51 -------------------------------- -----Machine SSL Certificate----- vcenter.incendiary.local Certificate Info: Issuer: CN=incendiary-ROGUE-CA Validity Not Before: Oct 11 19:44:16 2019 GMT Not After : Oct 8 19:44:16 2029 GMT Subject: C=US, ST=Alaska, L=Anchorage, O=incendiary, CN=vcenter.incendiary.local SHA1 Fingerprint=4D:00:81:D9:F9:6E:06:38:57:38:ED:C1:31:78:BD:E0:F9:54:E8:51
You can use the -e switch to display all the individual endpoints that are being used by each certificate. It’s a long list, so I won’t paste it here.
Then in conjunction with the -cml switch, use -f to fix.
It will prompt for the SSO password, and then ask which certificate fingerprint needs updated.
In my instance, it’s endpoint certificate 1.
51:EA:79:CE:81:69:CB:A6:0E:3B:47:42:4C:8D:28:68:94:3C:46:0D
So we provide the creds and that fingerprint.
[email protected] [ /tmp ]# ./check-trust-anchors -cml -f No 'lstool.txt' file found in this directory. Dumping service registrations to /tmp/lstool.txt... -----Endpoint Certificate 1----- Certificate Info: Issuer: CN=CA, DC=vsphere, DC=local, C=US, ST=California, O=vcenter.incendiary.local, OU=VMware Engineering Validity Not Before: Aug 11 22:57:30 2019 GMT Not After : Aug 5 22:57:29 2029 GMT Subject: CN=vcenter.incendiary.local, C=US SHA1 Fingerprint=51:EA:79:CE:81:69:CB:A6:0E:3B:47:42:4C:8D:28:68:94:3C:46:0D -------------------------------- -----Endpoint Certificate 2----- Certificate Info: Issuer: CN=incendiary-ROGUE-CA Validity Not Before: Oct 11 19:44:16 2019 GMT Not After : Oct 8 19:44:16 2029 GMT Subject: C=US, ST=Alaska, L=Anchorage, O=incendiary, CN=vcenter.incendiary.local SHA1 Fingerprint=4D:00:81:D9:F9:6E:06:38:57:38:ED:C1:31:78:BD:E0:F9:54:E8:51 -------------------------------- -----Machine SSL Certificate----- vcenter.incendiary.local Certificate Info: Issuer: CN=incendiary-ROGUE-CA Validity Not Before: Oct 11 19:44:16 2019 GMT Not After : Oct 8 19:44:16 2029 GMT Subject: C=US, ST=Alaska, L=Anchorage, O=incendiary, CN=vcenter.incendiary.local SHA1 Fingerprint=4D:00:81:D9:F9:6E:06:38:57:38:ED:C1:31:78:BD:E0:F9:54:E8:51 --------------------------------- {CYAN}SSL Trust Anchor Repair --------------------------------- Enter SSO admin [[email protected]]: Enter password for [email protected]: Enter fingerprint of trust anchor(s) to update: 51:EA:79:CE:81:69:CB:A6:0E:3B:47:42:4C:8D:28:68:94:3C:46:0D Enter the FQDN of the node to update: localhost Get site name Lookup all services Get service default-site:3a3b9e34-bc7c-4089-aee8-21f29c262ec7 Update service default-site:3a3b9e34-bc7c-4089-aee8-21f29c262ec7; spec: /tmp/svcspec_m5zme_pb Get service default-site:0ed7813d-b31a-47f0-9c30-b8c1d05ada25 Update service default-site:0ed7813d-b31a-47f0-9c30-b8c1d05ada25; spec: /tmp/svcspec_qof9h8zl Get service default-site:cdddfeca-7765-4ba2-acf4-b25e1bd2d26a Update service default-site:cdddfeca-7765-4ba2-acf4-b25e1bd2d26a; spec: /tmp/svcspec_i39mfvuc Get service f4335c81-ce40-4d03-8f83-d043bbb64e8e Don't update service f4335c81-ce40-4d03-8f83-d043bbb64e8e Get service be370880-e1e9-4bf6-9af2-4e7c5d756908 Don't update service be370880-e1e9-4bf6-9af2-4e7c5d756908 Get service 122bff4a-443d-4094-9035-be4e244286cd_com.vmware.vrops Don't update service 122bff4a-443d-4094-9035-be4e244286cd_com.vmware.vrops Get service 18360c0f-d11c-450e-967a-446beef6cbe0 Don't update service 18360c0f-d11c-450e-967a-446beef6cbe0 Get service 2fdd948a-fb14-49cc-a27d-7c4cae7e3e01 Don't update service 2fdd948a-fb14-49cc-a27d-7c4cae7e3e01 Get service ba3e37d4-26b9-47ab-9007-f3d8dc6a88cb Don't update service ba3e37d4-26b9-47ab-9007-f3d8dc6a88cb Get service 7a68d318-a9ca-4e2c-adbc-27873b7a7cb1 Don't update service 7a68d318-a9ca-4e2c-adbc-27873b7a7cb1 Get service 081405da-1dc1-43c9-adb0-885fabb1c325 Don't update service 081405da-1dc1-43c9-adb0-885fabb1c325 Get service 21504f6a-e383-4510-96e2-e2a814535cde Don't update service 21504f6a-e383-4510-96e2-e2a814535cde Get service 122bff4a-443d-4094-9035-be4e244286cd_com.vmware.nsx.ui.h5 Don't update service 122bff4a-443d-4094-9035-be4e244286cd_com.vmware.nsx.ui.h5 Get service 23994053-eef3-456b-b0ef-ae6d9f91d7a9 Don't update service 23994053-eef3-456b-b0ef-ae6d9f91d7a9 Get service 35a5d0cd-b2dd-46e7-81b3-59bc1da10f79 Don't update service 35a5d0cd-b2dd-46e7-81b3-59bc1da10f79 Get service 122bff4a-443d-4094-9035-be4e244286cd_com.vmware.vShieldManager Don't update service 122bff4a-443d-4094-9035-be4e244286cd_com.vmware.vShieldManager Get service 5bceba88-ee1a-46a2-a52a-2f42462ae3f3 Don't update service 5bceba88-ee1a-46a2-a52a-2f42462ae3f3 Get service da147e1a-09d6-4cc3-b784-9a762ca1694e Don't update service da147e1a-09d6-4cc3-b784-9a762ca1694e Get service 70f41eaa-5fbf-4c3e-ab4f-f7de3d0f3aa1 Don't update service 70f41eaa-5fbf-4c3e-ab4f-f7de3d0f3aa1 Get service f1d36406-546f-43e2-8ae3-b942c5439c54 Don't update service f1d36406-546f-43e2-8ae3-b942c5439c54 Get service 9fb71a11-ffd9-4872-8ce5-b4034f3d40a7 Don't update service 9fb71a11-ffd9-4872-8ce5-b4034f3d40a7 Get service abb20cc9-1261-467e-aafe-33b042775026 Don't update service abb20cc9-1261-467e-aafe-33b042775026 Get service d1f68783-4dd7-498e-8051-0080c746f254 Don't update service d1f68783-4dd7-498e-8051-0080c746f254 Get service a672b45e-7896-42cf-917b-cea8f75c0b71 Don't update service a672b45e-7896-42cf-917b-cea8f75c0b71 Get service 946bdda0-c2e3-4001-b7c3-244788312349 Don't update service 946bdda0-c2e3-4001-b7c3-244788312349 Get service bb159ac3-79df-4983-9556-290c945e1c9f Don't update service bb159ac3-79df-4983-9556-290c945e1c9f Get service 0c43747b-aac8-45cd-9428-56d06e800aef Don't update service 0c43747b-aac8-45cd-9428-56d06e800aef Get service a3cfb721-605f-4bf0-9e9a-d7ae0c605bac Don't update service a3cfb721-605f-4bf0-9e9a-d7ae0c605bac Get service f1d36406-546f-43e2-8ae3-b942c5439c54_kv Don't update service f1d36406-546f-43e2-8ae3-b942c5439c54_kv Get service 28aff66c-2429-425f-8b58-775b577d790b Don't update service 28aff66c-2429-425f-8b58-775b577d790b Get service 122bff4a-443d-4094-9035-be4e244286cd_com.vmware.vic Don't update service 122bff4a-443d-4094-9035-be4e244286cd_com.vmware.vic Get service 109d7960-21b1-4d99-a0bd-697e3550dbab Don't update service 109d7960-21b1-4d99-a0bd-697e3550dbab Get service 24a6452f-2c0b-4e56-a450-1fbee7243041 Don't update service 24a6452f-2c0b-4e56-a450-1fbee7243041 Get service bcdcb8b6-3782-419e-b826-ecece024d7c5 Don't update service bcdcb8b6-3782-419e-b826-ecece024d7c5 Get service 122bff4a-443d-4094-9035-be4e244286cd Don't update service 122bff4a-443d-4094-9035-be4e244286cd Get service 122bff4a-443d-4094-9035-be4e244286cd_com.vmware.vsphere.client Don't update service 122bff4a-443d-4094-9035-be4e244286cd_com.vmware.vsphere.client Get service 122bff4a-443d-4094-9035-be4e244286cd_com.vmware.vsan.dp Don't update service 122bff4a-443d-4094-9035-be4e244286cd_com.vmware.vsan.dp Get service f1d36406-546f-43e2-8ae3-b942c5439c54_authz Don't update service f1d36406-546f-43e2-8ae3-b942c5439c54_authz Get service b57f2a19-ce5b-40d2-aff7-dcad57518729 Don't update service b57f2a19-ce5b-40d2-aff7-dcad57518729 Get service 0f889511-99db-427b-983a-bdc8a308fcf5 Update service 0f889511-99db-427b-983a-bdc8a308fcf5; spec: /tmp/svcspec_hmbi6gma Updated 4 service(s)
We can see that it updated the 4 services that were using the wrong certificate, and then the strange behaviors I had were resolved.
Note that there is also a KB about a similar issue for vCenters that started life as vCenter 5.5, and issues began in vCenter 6.7. This was resolved in 6.7U3a.
https://kb.vmware.com/s/article/74731
Also note that VMware has a new tool called lsdoctor for “addressing issues in the PSC database, as well as data local to vCenter.”
This can also be used to correct certificate related issues.