Secure Boot VIB issue

The only way to be absolutely certain of non-native vibs is to compare with a build of ESXi that has been installed with native vibs.

Another method is to use the esxicli command...

esxcli software vib list

Select allOpen in new window

and this will list all the vibs VMware/VMW and VMware Certified are genuine.

but you may find other Vendors, QLC, INT, DELL, BCM, Avago, Qlogic...

it really depends if OEM or VMware ESXi was installed.

What do you mean if OEM or VMware ESXi was installed? We have a corporate account with VMware and download the ISO and patches from them.

But I need to ask again how I can determine if the list I provided can be safely deleted? That command doesn't tell me.

There are many versions of ESXi.

1. VMware version downloadable from VMware. Known as the vanilla VMware ESXi.
2. OEM versions of ESXi which include additional drivers (vibs) from OEMs.
3. Customisable versions of ESXi created by end users or admins.

Security and Critical updates are available from VMware and OEM Partners.

You will see vibs marked as VMware, and VMware Certified, and VMware and VMW as the vendor, but there could be other vibs supplied by other vendors.

If you can safely delete a vib will be based if your Hardware is using such a driver or vib.

Do you ha issues with the vibs you have listed ?

No these servers in vCenter have all been working. The problem comes from secure boot telling us that those vibs I listed are unsigned. This is due to upgrading ESXi via esxcli command and not through VUM or an ISO file as stated in other articles. So those vibs were not upgraded with the new digital certs.

https://blogs.vmware.com/vsphere/2018/06/prepping-an-esxi-6-7-host-for-secure-boot.html

As stated there check there are not in use and remove them, if they are in use check there is a native equivalent.

e.g. if you are not using this scsi-megaraid-sas because you do have the hardware installed it can be safely removed.

Do you happen to know how to check for a native equivilent or how to know if it will take over once the other vib is deleted?

This is an upgrade you are trying to complete ?

First you need to check your hardware,

e.g. does it include and are you using a scsi-megaraid controller, if you are not using safe to delete, you are not using i or dependant upon it.

a preifx vib is usually a non native driver e.g. net-, scsi- old legacy.

No I already did the upgrade, we're just trying to enable secure boot but we need to delete the VIBs specified since the upgrade to 6.5 (which we already did), did not upgrade the VIBs certificates.

I honestly don't know what megaraid controller is. I never heard of it. Theirs all kinds of VIBs that I listed that I never heard of and have no idea if ESXi or the hardware is dependent on them.

In that case I would not recommend an upgrade, re-install from scratch, this is now one of the biggest evils of an upgrade!!!

WE already upgraded. The VIBs are listed. If anyone happens to know how to tell if the VIBs are being used or if it's ok to get rid of them that would be great.

You can only get rid of them if you match the vibs to hardware devices in the server.

As I've said if you are not using the scsi-mega-driver, e.g. you do not have a device in your server you are safe to delete it.

If you cannot determine the hardware devices in your server, then best to complete a full install.

You could use the lspci command in the ESXi shell to determine the hardware devices. (e.g. network interfaces, storage controllers).

You can also use the esxcfg-info command to list what drivers are in use, nobody can tell you just to get rid of them, unless you do your homework to determine if they are in use.

In you are unsure how to proceed, I would escalate a SR to VMware Support, to work with an Engineer to assist you, to determine what you can delete safely.

From this forum, I cannot advise what is safe to delete without access to your server and logs, and diagnosis.

I do not even have a specification of what server you are using, and what hardware it contains, to make an informed decision as to what you can delete safely, and nobody else can either.

Oh I wasn't asking you to tell me if I can delete a specific driver. I was inquiring how to go about determining that on my own and I found the answer.

esxcli software vib list | grep blah
esxcli software vib get -n usc-tool-esxi
esxcli storage core adapter list (for scsi or storage vibs)
esxcli network nic list (list network drivers/vibs?)
esxcfg-info and open in notepad++ (search for vib name I think)

or if it's ok to get rid of them that would be great.

That suggests a question of if it's all right to delete them!

Those are all the tools that are required.

plus the logs, and lspci (check fr PCI devices).

esxcfg-info, you will see the interfaces in use. (you can use grep, for all those commands, no need to use notepad++)

It's called homework, or Server Administration, if you are uncomfortable, VMware Support, or someone can remotely connect to your server and check if in use and delete them.

I'm comfortable with a rebuild since the host is in a cluster, in maintenance mode with the guests moved off.  A rebuild is trivial so no need to call VMware support for a low stake situation.

I'm sorry it does not look like it from here... when I posted...

esxcli software vib list

Select allOpen in new window

esxcfg-info

Select allOpen in new window

and you don't know how to use grep.

Anyway select your solution, if you Answered it yourself.

Oh! I don't get paid from Answering Questions on Experts Exchange. I'm a volunteer, which also fyi is UNPAID

Call VMware Support if you need to delete the vibs, not re-install it, which I would highly recommend, not upgrade. This is one of he pitfalls of inplace upgrades, they bite you in the end.

I'll select a solution later, perhaps someone may add input.

I know the canned response is to call VMware for unsupported solutions but I've read enough comments online from those who've deleting their own VIBs without issue. Plus a rebuild shouldn't be a problem.

It's not a canned solution, if you are unsure of what to do, and what not to delete. (and it's complex!)

Again, without the hardware in front of me I cannot tell you which to delete, and you've not even given me a server spec, so detailed investigation is required.

logs - /var/log
dmesg
esxcli software vib
lspci
esxcfg-info

and then it's easy to get wrong, and most of the time this area is unsupported activity!

Any further help required or hand holding let me know. If you let me know your server make and model, I can check what vibs you should not be delete.

To continue were I left off. I have to ask. We're running a Cisco custom 6.5U3 ESXi image yet those vibs it installed aren't signed or whatever they need to be in order to enable secure boot as previously stated.

So I downloaded the 6.7U3 offline zip and copied the VIBs i.e. scsi_megaraid_sas in order to do an update instead of a deletion. So my question is can I update a 6.5 VIB with the 6.7 version?

Yes, you can, or you can upgrade straight from the internet.

The Update will fail and advise if it does not like your vibs.

see my EE Articles here for updating..

HOW TO: Update VMware ESXi 6.7 GA to ESXi 6.7U3(a) in 5 easy steps

HOW TO: Update VMware ESXi 6.7 GA to ESXi 6.7U3(a) direct from VMware.

Excellent articles Andrew, but in case you weren't tracking, we aren't upgrading ESXi, I'm just updating a single VIB file on our 6.5 hosts to a 6.7U3 VIB more specifically just the scsi_megaraid_sas firmware because a secure boot error says our current one is "unsigned". Example:

Secure boot cannot be enabled. Secure boot FAILED to verify signatures of the following vibs: scsi_megaraid_sas, FNIC etc etc

The vibs are version 6.5.xxxxxx and come from a custom CISCO 6.5 ISO I guess because the hosts are running on Cisco blades. We could probably delete it but the system may be using it.

esxcli storage core adapter list
vmhba0 unknown.vmhba0
vmhba1 FNIC
vmhba2 FNIC

I don't know why their unsigned, Not sure if I can delete it, not sure if updating it to 6.7U3 version will help etc.

Ah, so you want to take a vib from 6.7u3 and use it in 6.5 ?

that will not be supported, and you need to make sure it's the same version

mega_raid is a storage HBA controller, normally found in Dells, are you using a SCSI HBA for storage (local storage)

We also have FNIC which says CISCO vendor on vmhba1 which is also unsigned. If I force enable the secure boot, it will PSOD. Not sure how all these vibs got that way but if you're saying I can't update them, then how would I sign them? I can try to reinstall the original vibs I guess.
As far as megaraid-sas, we're using iSCSI for storage.

Is there a way to install the same vib but sign it within ESXi?

iSCSI will not use the mega_raid.sas driver.

You really are looking at rolling your own ESXi build here, using the vib authoring tool.

Ok I'll look into the authoring tool but I'll have to get secure boot working tomorrow so I'll try updating the Cisco FNIC firmware with the same version or perhaps delete and install and see what happens.

I'm trying to understand what you are trying to achieve, you've upgraded to the current situation, and now have a PSOD on the server.

So why not just install the ESXi server from scratch, would this not be a quicker fix ?

Sure I'll try and break it down.
Goal: Enable secure boot
Problem: unsigned vibs

1) If you turn on secure boot on an ESX host with unsigned vibs, the ESX host will not boot. Hence the name I guess
2) When I try and turn it on at the command prompt, it says secure boot failed, unsigned vibs. And it lists the vibs
3) Some of those unsigned drivers keeping me from enabling secure boot are vibs like FNIC, scsi-megaraid-sas etc
4) It looks like we're using some of these vibs i.e. FNIC when performing esxcli network core adapter list so I may not be able to delete them
5) Will try to reinstall an unsigned vib to see if that signs it. If that works, no need to rebuild all the hosts in site.

Yes, I appreciate that, so this was a working ESXi server, without Secure Boot Enabled, and worked perfectly.....

Turning on Secure Boot in the Server BIOS causes an PSOD, and that's because of Signing in the VIBs! where a handful of vibs have not been signed.

Correct ?

That's correct to my knowledge and a couple articles online confirm. We're running Cisco servers installed with custom ESXi ISO from VMware downloads. In any case, before I blow it up I wanted to download the individual vib zip file and reinstall it to test. I'll report back.

So, why not enable secure boot, and install a fresh ESXi ISO 6.5U3?

Is this embedded install on SD or install on HDD/SDD flash ?

rather than turning on a function that was not enabled when it was originally installed ?

and that should be a much quicker solution, than trying to retro-fit a solution around the current issue ?

(that's what the validation script does to warn you of just enabling secure boot in the BIOS, that could trash your ESXi host, for several reasons, VIBs not signed, boot loader missing.....)

The reason is because the hosts are in a remote state which I cannot KVM to until I firmware update/fix the OOB issue.  Therefore, i would not be able to do a rebuild right now making a vib reinstall easier at least in theory.

But I somewhat agree that a rebuild could be the way to go assuming the vib reinstall/sign doesn't work.

Oh to answer your other question, I think the ESX hosts were already reinstalled. That's what I was told.

Basically, what you are trying to do, is a not supported exercise, which is usually caught by the script, if script reports an issue...

complete re-install and I assume there is a reason now for enabling secure boot!

Quicker and easier to perform a supported activity and perform a new (re)install when secure-boot was enabled, rather than hack and slash job!!!!

(especially in a Production environment, which I assume this is, rather than a bedroom lab)

I'll have to disagree with you on multiple points here.

1) A simple vib reinstall is faster than a reinstall of the OS
2) A simple vib reinstall is not a hack job
3) I'm sure VMware supports vib reinstalls
4) A simple vib reinstall is the day and life of a high paid engineer

What do you think an engineer does when you update nic drivers for new HBA cards or other such troubleshooting? Is that also unsupported hacking?

Excellent Job. Document the changes and add to your change control record for your Change Programme Manager.

With all due respect, the uninvited advice and opinions comes off as snarky and trying to tell me how to do my job when you have no idea of the particular environment or politics here.  I didn't ask for advice on change control or need to be told that I'm performing unsupported hack jobs especially when not true.