Symptom: Your ONTAP 9.x system is giving errors about third party CA certificates expiring.
Source = mgwd
“mgmtgwd.certificate.expiring: A digital certificate with Fully Qualified Domain Name (FQDN) Class2PrimaryCA, Serial Number 85BD4BF3D8DAE369F694D75FC3A54423, Certificate Authority ‘Class 2 Primary CA’ and type server-ca for Vserver Netapp1 will expire in the next 4 day(s).”
“mgmtgwd.certificate.expiring: A digital certificate with Fully Qualified Domain Name (FQDN) DeutscheTelekomRootCA2, Serial Number 26, Certificate Authority ‘Deutsche Telekom Root CA 2’ and type server-ca for Vserver Netapp1 will expire in the next 7 day(s)”
“mgmtgwd.certificate.expiring: A digital certificate with Fully Qualified Domain Name (FQDN) UTN-USERFirst-Hardware, Serial Number 44BE0C8B500024B411D3362AFE650AFD, Certificate Authority ‘UTN-USERFirst-Hardware’ and type server-ca for Vserver Netapp1 will expire in the next 7 day(s).”
Symptom: The certificates are starting to expire in July 2019.
Symptom: The expiring certificates are type “server-ca”
Symptom: When you connect to your Netapp command prompt and type “security certificate show”, you see about 100 certificates, not 2-6.
Symptom: Can’t upgrade from ONTAP 9.1 to later versions
Symptom: You can’t create a new cluster on a brand new Netapp.
“Vserver Management .Error: Failed to add the Cserver record in RDB . The certificate has expired.”
Different issue: If you see expiration warnings about certificates that are type = server and are named the same as your Netapp, see this article instead.
Disclaimer: I’m just someone who admins Netapp SAN for my job. You should go to the Netapp website and contact their support for official guidance. This bug has the potential to cause a major outage. I’ve included links to relevant bug reports below.
Root cause of expiring server-ca certificates:
This is a bug in Netapp ONTAP 9.2 – 9.3 and later versions. The bug causing failures to upgrade or install new Netapp clusters is: BUG 1250500. (You will need a Netapp account to view it)
The general bug that addresses certificates expiring but you aren’t trying to upgrade or install a new Netapp is BUG 1245418 (You will need a Netapp account to view it).
The simple version is that Netapp included a bunch of third party certificates in their ONTAP 9.2+ releases for compatibility purposes. This brought the certificate count up from 2-6 to 100+. Each of those certificates has a different expiration date, with a few expiring in July 2019, several expiring in 2020, etc.
If you aren’t trying to upgrade your system or install a new Netapp, there is normally no impact from this bug. You just get a bunch of scary errors in the event log that repeat daily.
IF YOUR NETAPP IS AFFECTED, WAIT UNTIL THE NEW PATCH IS OUT (ETA JULY 15, 2019). I have been told that Netapp is working on an ONTAP update which includes a fix for the certificates and BUG 1250500. I would monitor this Netapp KB article for release timelines and for the exact versions of patches that have the fix.
How to fix the expiring server-ca certificates in Netapp
- Updated July 11, 2019: Netapp is releasing ONTAP 9.3 , 9.5, and 9.6 updates which fix the expiring certificates.
- Netapp Support does not want customers to try to fix this problem on their own.
- Put in a case with Netapp for a custom workaround or wait for the patch, due around July 15, 2019.
How to show the full list of certificates in Netapp:
These steps can be used to show the list of certificates installed on your Netapp.
- Looking at your event log, make a list of which certificates are expiring or expired. If you look at my examples above, you will see that I bolded the FQDN or “Common name” in the error messages. This is the information you need.
- Use a terminal emulator such as Putty. Connect using SSH port 22, to the IP of your Netapp Cluster. .
- When prompted, type your administrator username and password. admin is the default username and netapp!123 is the default password.
- type set -privilege advanced and type y to confirm if prompted.
- type security certificate show to see the whole list of certificates.
- Review the certificate list’s expiration date field to double-check which ones are expiring soon, or have expired.
- Make sure that the type = server-ca for the certificates you are considering. If an expiring certificate is type = server , you need to perform the steps in my other article to re-create the self-signed Netapp certificates.
- Type security certificate show -common-name Class2PrimaryCA Class2PrimaryCA is an example certificate common name. Put the common name of an expiring certificate here.
- You should see information about the expiring certificate display.
Selfish plug time!
Thanks for reading this article! I hope it helps you! If you have tips or feedback, please comment or send me an email so that others can benefit.
I am a consultant in the Maryland/DC area in the USA. My specialties are Windows migrations (to 2016 and to Office 365 / Azure), VMware migrations, Netapp and SAN, and high availability / disaster recovery planning. If you would like help with your complex project, or would like a architectural review to improve your availability, please reach out! More information and contact can be found on the About page. – Amira Armond