Symptom: Your ONTAP 9.x system is giving errors about third party CA certificates expiring.
Source = mgwd
“mgmtgwd.certificate.expiring: A digital certificate with Fully Qualified Domain Name (FQDN) Class2PrimaryCA, Serial Number 85BD4BF3D8DAE369F694D75FC3A54423, Certificate Authority ‘Class 2 Primary CA’ and type server-ca for Vserver Netapp1 will expire in the next 4 day(s).”
“mgmtgwd.certificate.expiring: A digital certificate with Fully Qualified Domain Name (FQDN) DeutscheTelekomRootCA2, Serial Number 26, Certificate Authority ‘Deutsche Telekom Root CA 2’ and type server-ca for Vserver Netapp1 will expire in the next 7 day(s)”
“mgmtgwd.certificate.expiring: A digital certificate with Fully Qualified Domain Name (FQDN) UTN-USERFirst-Hardware, Serial Number 44BE0C8B500024B411D3362AFE650AFD, Certificate Authority ‘UTN-USERFirst-Hardware’ and type server-ca for Vserver Netapp1 will expire in the next 7 day(s).”
Symptom: The certificates are starting to expire in July 2019.
Symptom: The expiring certificates are type “server-ca”
Symptom: When you connect to your Netapp command prompt and type “security certificate show”, you see about 100 certificates, not 2-6.
Symptom: Can’t upgrade from ONTAP 9.1 to later versions
Symptom: You can’t create a new cluster on a brand new Netapp.
“Vserver Management .Error: Failed to add the Cserver record in RDB . The certificate has expired.”
Different issue: If you see expiration warnings about certificates that are type = server and are named the same as your Netapp, see this article instead.
How to fix Netapp expired self-signed certificate by creating a new one
Disclaimer: I’m just someone who admins Netapp SAN for my job. You should go to the Netapp website and contact their support for official guidance. This bug has the potential to cause a major outage. I’ve included links to relevant bug reports below.
Root cause of expiring server-ca certificates:
This is a bug in Netapp ONTAP 9.2 – 9.3 and later versions. The bug causing failures to upgrade or install new Netapp clusters is: BUG 1250500. (You will need a Netapp account to view it)
The general bug that addresses certificates expiring but you aren’t trying to upgrade or install a new Netapp is BUG 1245418 (You will need a Netapp account to view it).
The simple version is that Netapp included a bunch of third party certificates in their ONTAP 9.2+ releases for compatibility purposes. This brought the certificate count up from 2-6 to 100+. Each of those certificates has a different expiration date, with a few expiring in July 2019, several expiring in 2020, etc.
If you aren’t trying to upgrade your system or install a new Netapp, there is normally no impact from this bug. You just get a bunch of scary errors in the event log that repeat daily.
IF YOUR NETAPP IS AFFECTED, WAIT UNTIL THE NEW PATCH IS OUT (ETA JULY 15, 2019). I have been told that Netapp is working on an ONTAP update which includes a fix for the certificates and BUG 1250500. I would monitor this Netapp KB article for release timelines and for the exact versions of patches that have the fix.
How to fix the expiring server-ca certificates in Netapp
- Updated July 11, 2019: Netapp is releasing ONTAP 9.3 , 9.5, and 9.6 updates which fix the expiring certificates.
- Netapp Support does not want customers to try to fix this problem on their own.
- Put in a case with Netapp for a custom workaround or wait for the patch, due around July 15, 2019.
How to show the full list of certificates in Netapp:
These steps can be used to show the list of certificates installed on your Netapp.
- Looking at your event log, make a list of which certificates are expiring or expired. If you look at my examples above, you will see that I bolded the FQDN or “Common name” in the error messages. This is the information you need.
- Use a terminal emulator such as Putty. Connect using SSH port 22, to the IP of your Netapp Cluster. .
- When prompted, type your administrator username and password. admin is the default username and netapp!123 is the default password.
- type set -privilege advanced and type y to confirm if prompted.
- type security certificate show to see the whole list of certificates.
- Review the certificate list’s expiration date field to double-check which ones are expiring soon, or have expired.
- Make sure that the type = server-ca for the certificates you are considering. If an expiring certificate is type = server , you need to perform the steps in my other article to re-create the self-signed Netapp certificates.
- Type security certificate show -common-name Class2PrimaryCA Class2PrimaryCA is an example certificate common name. Put the common name of an expiring certificate here.
- You should see information about the expiring certificate display.
Selfish plug time!
Thanks for reading this article! I hope it helps you! If you have tips or feedback, please comment or send me an email so that others can benefit.
I am a consultant in the Maryland/DC area in the USA. My specialties are Windows migrations (to 2016 and to Office 365 / Azure), VMware migrations, Netapp and SAN, and high availability / disaster recovery planning. If you would like help with your complex project, or would like a architectural review to improve your availability, please reach out! More information and contact can be found on the About page. – Amira Armond
It’s a known bug, and we have a fix coming in a couple of days. If you absolutely can’t wait a couple of days for a patch, we have a workaround, and it’s different to this one, but you need to ask us for it.
The reason you need to ask is that we are tracking everyone we tell to run the workaround, and we will be testing systems with our workaround applied against future patches, and in case of any issues we can proactively notify. If you’ve modified the system in a different way, we don’t know about it, and it might break in fun and exciting ways.