OEM SSL Cipher Hardening Reset After Securing OMS

By | September 12, 2019

I have recently been installing Oracle Enterprise Manager at several sites, and one of the key requirements has been to ensure that the installation isn’t using insecure HTTPS protocols. Securing the OMS and agents typically consists of two components – ensuring that only secure SSL ciphers are being used, and shutting down protocols that have known vulnerabilities. Thankfully, Oracle has documented the procedures in two separate MOS notes:

Doc ID 2138391.1 – 13c: How to Disable Weak SSLCipherSuites in Enterprise Manager 13c Cloud Control
Doc ID 2212006.1 – EM 13c: Enterprise Manager 13c Cloud Control Configuration to Support Transport Layer Security Protocol:TLSv1.2 only

This isn’t a post about how to perform the task – that is outlined pretty well in the MOS documents. The interesting piece is the behavior that I saw after I thought that the task was completed. We went through the entire setup of both notes and requested a security scan, and found that the agent upload port (4903) was reporting weak ciphers. That was strange, because we had updated all of the files as described in the MOS notes.

Currently, the OMS used in these examples has gone through the procedure detailed in MOS note #2138391.1. At this point, weak SSL ciphers have been disabled, but it has not been secured to require TLSv1.2. We can validate the SSL ciphers with nmap below (nmap output abbreviated to show only the SSL cipher output):

Andys-MacBook-Pro-3:~ acolvin$ sudo nmap -sV --script ssl-enum-ciphers -p 4903 enkpoemac1
Password:

Starting Nmap 7.40 ( https://nmap.org ) at 2019-08-22 09:17 CDT
---
| ssl-enum-ciphers:
| TLSv1.0:
| ciphers:
| TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA (secp256r1) - A
| TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA (secp256r1) - A
| compressors:
| NULL
| cipher preference: client
| TLSv1.1:
| ciphers:
| TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA (secp256r1) - A
| TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA (secp256r1) - A
| compressors:
| NULL
| cipher preference: client
| TLSv1.2:
| ciphers:
| TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA (secp256r1) - A
| TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA (secp256r1) - A
| TLS_RSA_WITH_AES_128_CBC_SHA256 (rsa 1024) - A
| TLS_RSA_WITH_AES_256_CBC_SHA256 (rsa 1024) - A
| compressors:
| NULL
| cipher preference: client
|_ least strength: A
---

As you can see, all of the ciphers that are allowed match an “A” score. Just to see, I checked the timestamp of the configuration files.

[oracle@enkpoemac1 gc_inst]$ find . -name httpd_em.conf -exec ls -l {} \;
-rw-r--r-- 1 oracle oinstall 5804 Aug 22 09:02 ./user_projects/domains/GCDomain/config/fmwconfig/components/OHS/ohs1/moduleconf/httpd_em.conf
-rw-r--r-- 1 oracle oinstall 5804 Aug 22 09:04 ./user_projects/domains/GCDomain/config/fmwconfig/components/OHS/instances/ohs1/moduleconf/httpd_em.conf

At this point, I run the procedure on MOS note #2212006.1. In order to lock down the TLS protocols, we run a couple of “emctl secure XXX” commands and restart the OMS:

[oracle@enkpoemac1 gc_inst]$ emctl secure oms -console -protocol "TLSv1.2"
Oracle Enterprise Manager Cloud Control 13c Release 3
Copyright (c) 1996, 2018 Oracle Corporation. All rights reserved.
Securing OMS... Started.
Enter Enterprise Manager Root (SYSMAN) Password :
Enter Agent Registration Password :
Securing OMS... Successful
Restart OMS

[oracle@enkpoemac1 gc_inst]$ emctl secure oms -protocol "TLSv1.2"
Oracle Enterprise Manager Cloud Control 13c Release 3
Copyright (c) 1996, 2018 Oracle Corporation. All rights reserved.
Securing OMS... Started.
Enter Enterprise Manager Root (SYSMAN) Password :
Enter Agent Registration Password :
Securing OMS... Successful
Restart OMS

[oracle@enkpoemac1 gc_inst]$ emctl stop oms -all
Oracle Enterprise Manager Cloud Control 13c Release 3
Copyright (c) 1996, 2018 Oracle Corporation. All rights reserved.
Stopping Oracle Management Server...
WebTier Successfully Stopped
Oracle Management Server Successfully Stopped
Oracle Management Server is Down
JVMD Engine is Down
Stopping BI Publisher Server...
BI Publisher Server Successfully Stopped
AdminServer Successfully Stopped
BI Publisher Server is Down

[oracle@enkpoemac1 gc_inst]$ emctl start oms
Oracle Enterprise Manager Cloud Control 13c Release 3
Copyright (c) 1996, 2018 Oracle Corporation. All rights reserved.
Starting Oracle Management Server...
WebTier Successfully Started
Oracle Management Server Successfully Started
Oracle Management Server is Up
JVMD Engine is Up
Starting BI Publisher Server ...
BI Publisher Server Successfully Started
BI Publisher Server is Up

After restarting the OMS, we run another scan, and see different results:

Andys-MacBook-Pro-3:~ acolvin$ sudo nmap -sV --script ssl-enum-ciphers -p 4903 enkpoemac1
Password:

Starting Nmap 7.40 ( https://nmap.org ) at 2019-08-22 09:50 CDT
---
| ssl-enum-ciphers:
| TLSv1.2:
| ciphers:
| TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA (secp256r1) - D
| TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA (secp256r1) - A
| TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA (secp256r1) - A
| TLS_ECDHE_RSA_WITH_RC4_128_SHA (secp256r1) - D
| TLS_RSA_WITH_3DES_EDE_CBC_SHA (rsa 1024) - D
| TLS_RSA_WITH_AES_128_CBC_SHA (rsa 1024) - A
| TLS_RSA_WITH_AES_128_CBC_SHA256 (rsa 1024) - A
| TLS_RSA_WITH_AES_128_GCM_SHA256 (rsa 1024) - A
| TLS_RSA_WITH_AES_256_CBC_SHA (rsa 1024) - A
| TLS_RSA_WITH_AES_256_CBC_SHA256 (rsa 1024) - A
| TLS_RSA_WITH_AES_256_GCM_SHA384 (rsa 1024) - A
| TLS_RSA_WITH_RC4_128_MD5 (rsa 1024) - D
| TLS_RSA_WITH_RC4_128_SHA (rsa 1024) - D
| compressors:
| NULL
| cipher preference: client
| warnings:
| 64-bit block cipher 3DES vulnerable to SWEET32 attack
| Broken cipher RC4 is deprecated by RFC 7465
| Ciphersuite uses MD5 for message integrity
|_ least strength: D
---

At this point, we decided to check the configuration files to see which files had been recently modified:

[oracle@enkpoemac1 config]$ find . -mmin -20 -type f -exec ls -l {} +
-rw-r----- 1 oracle oinstall 0 Aug 22 09:33 ./config.lok
-rw-r----- 1 oracle oinstall 60536 Aug 22 09:29 ./config.xml
-rw-r----- 1 oracle oinstall 2415 Aug 22 09:26 ./diagnostics/Module-FMWDFW-2818.xml
-rw-r--r-- 1 oracle oinstall 36868 Aug 22 09:29 ./fmwconfig/components/OHS/instances/ohs1/httpd.conf.emctl_secure
-rw-r----- 1 oracle oinstall 2920 Aug 22 09:26 ./fmwconfig/components/OHS/instances/ohs1/keystores/console/cwallet.sso
-rw-r----- 1 oracle oinstall 2843 Aug 22 09:26 ./fmwconfig/components/OHS/instances/ohs1/keystores/console/ewallet.p12
-rw-r----- 1 oracle oinstall 2920 Aug 22 09:29 ./fmwconfig/components/OHS/instances/ohs1/keystores/upload/cwallet.sso
-rw-r----- 1 oracle oinstall 2843 Aug 22 09:29 ./fmwconfig/components/OHS/instances/ohs1/keystores/upload/ewallet.p12
-rw-r--r-- 1 oracle oinstall 609 Aug 22 09:29 ./fmwconfig/components/OHS/instances/ohs1/moduleconf/agent_download.conf
-rw-r----- 1 oracle oinstall 609 Aug 22 09:26 ./fmwconfig/components/OHS/instances/ohs1/moduleconf/agent_download.conf.2019_08_22_09_26_45
-rw-r----- 1 oracle oinstall 609 Aug 22 09:29 ./fmwconfig/components/OHS/instances/ohs1/moduleconf/agent_download.conf.2019_08_22_09_29_50
-rw-r----- 1 oracle oinstall 1351 Aug 22 09:29 ./fmwconfig/components/OHS/instances/ohs1/moduleconf/httpd_bip.conf
-rw-r----- 1 oracle oinstall 1351 Aug 22 09:26 ./fmwconfig/components/OHS/instances/ohs1/moduleconf/httpd_bip.conf.2019_08_22_09_26_45
-rw-r----- 1 oracle oinstall 1351 Aug 22 09:29 ./fmwconfig/components/OHS/instances/ohs1/moduleconf/httpd_bip.conf.2019_08_22_09_29_50
-rw-r--r-- 1 oracle oinstall 5659 Aug 22 09:29 ./fmwconfig/components/OHS/instances/ohs1/moduleconf/httpd_em.conf
-rw-r----- 1 oracle oinstall 5347 Aug 22 09:26 ./fmwconfig/components/OHS/instances/ohs1/moduleconf/httpd_em.conf.2019_08_22_09_26_45
-rw-r----- 1 oracle oinstall 5347 Aug 22 09:29 ./fmwconfig/components/OHS/instances/ohs1/moduleconf/httpd_em.conf.2019_08_22_09_29_50
-rw-r----- 1 oracle oinstall 4051 Aug 22 09:29 ./fmwconfig/components/OHS/instances/ohs1/moduleconf/ssl_bip.conf
-rw-r----- 1 oracle oinstall 3972 Aug 22 09:26 ./fmwconfig/components/OHS/instances/ohs1/moduleconf/ssl_bip.conf.2019_08_22_09_26_45
-rw-r----- 1 oracle oinstall 3972 Aug 22 09:29 ./fmwconfig/components/OHS/instances/ohs1/moduleconf/ssl_bip.conf.2019_08_22_09_29_50
-rw-r----- 1 oracle oinstall 3981 Aug 22 09:29 ./fmwconfig/components/OHS/instances/ohs1/moduleconf/ssl_bip.conf.tmp
-rw-r----- 1 oracle oinstall 2105 Aug 22 09:29 ./fmwconfig/components/OHS/instances/ohs1/mod_wl_ohs.conf
-rw-r----- 1 oracle oinstall 2105 Aug 22 09:26 ./fmwconfig/components/OHS/instances/ohs1/mod_wl_ohs.conf.2019_08_22_09_26_29
-rw-r----- 1 oracle oinstall 2008 Aug 22 09:26 ./fmwconfig/components/OHS/instances/ohs1/mod_wl_ohs.conf.2019_08_22_09_26_45
-rw-r----- 1 oracle oinstall 2105 Aug 22 09:29 ./fmwconfig/components/OHS/instances/ohs1/mod_wl_ohs.conf.2019_08_22_09_29_38
-rw-r----- 1 oracle oinstall 2008 Aug 22 09:29 ./fmwconfig/components/OHS/instances/ohs1/mod_wl_ohs.conf.2019_08_22_09_29_50
-rw-r--r-- 1 oracle oinstall 2105 Aug 22 09:29 ./fmwconfig/components/OHS/instances/ohs1/mod_wl_ohs.conf.emctl_secure
-rw-r----- 1 oracle oinstall 3679 Aug 22 09:29 ./fmwconfig/components/OHS/instances/ohs1/ssl.conf
-rw-r----- 1 oracle oinstall 3682 Aug 22 09:26 ./fmwconfig/components/OHS/instances/ohs1/ssl.conf.2019_08_22_09_26_44
-rw-r----- 1 oracle oinstall 3679 Aug 22 09:26 ./fmwconfig/components/OHS/instances/ohs1/ssl.conf.2019_08_22_09_26_45
-rw-r----- 1 oracle oinstall 3679 Aug 22 09:29 ./fmwconfig/components/OHS/instances/ohs1/ssl.conf.2019_08_22_09_29_49
-rw-r----- 1 oracle oinstall 3679 Aug 22 09:29 ./fmwconfig/components/OHS/instances/ohs1/ssl.conf.2019_08_22_09_29_50
-rw-r--r-- 1 oracle oinstall 3679 Aug 22 09:29 ./fmwconfig/components/OHS/instances/ohs1/ssl.conf.emctl_secure
-rw-r--r-- 1 oracle oinstall 36868 Aug 22 09:29 ./fmwconfig/components/OHS/ohs1/httpd.conf.emctl_secure
-rw-r----- 1 oracle oinstall 2920 Aug 22 09:26 ./fmwconfig/components/OHS/ohs1/keystores/console/cwallet.sso
-rw-r--r-- 1 oracle oinstall 2843 Aug 22 09:26 ./fmwconfig/components/OHS/ohs1/keystores/console/ewallet.p12
-rw-r----- 1 oracle oinstall 2920 Aug 22 09:29 ./fmwconfig/components/OHS/ohs1/keystores/upload/cwallet.sso
-rw-r--r-- 1 oracle oinstall 2843 Aug 22 09:29 ./fmwconfig/components/OHS/ohs1/keystores/upload/ewallet.p12
-rw-r--r-- 1 oracle oinstall 609 Aug 22 09:29 ./fmwconfig/components/OHS/ohs1/moduleconf/agent_download.conf
-rw-r----- 1 oracle oinstall 609 Aug 22 09:26 ./fmwconfig/components/OHS/ohs1/moduleconf/agent_download.conf.2019_08_22_09_26_45
-rw-r----- 1 oracle oinstall 609 Aug 22 09:29 ./fmwconfig/components/OHS/ohs1/moduleconf/agent_download.conf.2019_08_22_09_29_50
-rw-r----- 1 oracle oinstall 1351 Aug 22 09:29 ./fmwconfig/components/OHS/ohs1/moduleconf/httpd_bip.conf
-rw-r----- 1 oracle oinstall 1351 Aug 22 09:26 ./fmwconfig/components/OHS/ohs1/moduleconf/httpd_bip.conf.2019_08_22_09_26_45
-rw-r----- 1 oracle oinstall 1351 Aug 22 09:29 ./fmwconfig/components/OHS/ohs1/moduleconf/httpd_bip.conf.2019_08_22_09_29_50
-rw-r--r-- 1 oracle oinstall 5659 Aug 22 09:29 ./fmwconfig/components/OHS/ohs1/moduleconf/httpd_em.conf
-rw-r----- 1 oracle oinstall 5347 Aug 22 09:26 ./fmwconfig/components/OHS/ohs1/moduleconf/httpd_em.conf.2019_08_22_09_26_45
-rw-r----- 1 oracle oinstall 5347 Aug 22 09:29 ./fmwconfig/components/OHS/ohs1/moduleconf/httpd_em.conf.2019_08_22_09_29_50
-rw-r----- 1 oracle oinstall 4051 Aug 22 09:29 ./fmwconfig/components/OHS/ohs1/moduleconf/ssl_bip.conf
-rw-r----- 1 oracle oinstall 3972 Aug 22 09:26 ./fmwconfig/components/OHS/ohs1/moduleconf/ssl_bip.conf.2019_08_22_09_26_45
-rw-r----- 1 oracle oinstall 3972 Aug 22 09:29 ./fmwconfig/components/OHS/ohs1/moduleconf/ssl_bip.conf.2019_08_22_09_29_50
-rw-r----- 1 oracle oinstall 3981 Aug 22 09:29 ./fmwconfig/components/OHS/ohs1/moduleconf/ssl_bip.conf.tmp
-rw-r----- 1 oracle oinstall 2105 Aug 22 09:29 ./fmwconfig/components/OHS/ohs1/mod_wl_ohs.conf
-rw-r----- 1 oracle oinstall 2105 Aug 22 09:26 ./fmwconfig/components/OHS/ohs1/mod_wl_ohs.conf.2019_08_22_09_26_29
-rw-r----- 1 oracle oinstall 2008 Aug 22 09:26 ./fmwconfig/components/OHS/ohs1/mod_wl_ohs.conf.2019_08_22_09_26_45
-rw-r----- 1 oracle oinstall 2105 Aug 22 09:29 ./fmwconfig/components/OHS/ohs1/mod_wl_ohs.conf.2019_08_22_09_29_38
-rw-r----- 1 oracle oinstall 2008 Aug 22 09:29 ./fmwconfig/components/OHS/ohs1/mod_wl_ohs.conf.2019_08_22_09_29_50
-rw-r--r-- 1 oracle oinstall 2105 Aug 22 09:29 ./fmwconfig/components/OHS/ohs1/mod_wl_ohs.conf.emctl_secure
-rw-r----- 1 oracle oinstall 3679 Aug 22 09:29 ./fmwconfig/components/OHS/ohs1/ssl.conf
-rw-r----- 1 oracle oinstall 3682 Aug 22 09:26 ./fmwconfig/components/OHS/ohs1/ssl.conf.2019_08_22_09_26_44
-rw-r----- 1 oracle oinstall 3679 Aug 22 09:26 ./fmwconfig/components/OHS/ohs1/ssl.conf.2019_08_22_09_26_45
-rw-r----- 1 oracle oinstall 3679 Aug 22 09:29 ./fmwconfig/components/OHS/ohs1/ssl.conf.2019_08_22_09_29_49
-rw-r----- 1 oracle oinstall 3679 Aug 22 09:29 ./fmwconfig/components/OHS/ohs1/ssl.conf.2019_08_22_09_29_50
-rw-r--r-- 1 oracle oinstall 3679 Aug 22 09:29 ./fmwconfig/components/OHS/ohs1/ssl.conf.emctl_secure
-rw-r----- 1 oracle oinstall 9513 Aug 22 09:33 ./fmwconfig/ovd/default/adapters.os_xml
-rw-r----- 1 oracle oinstall 3184 Aug 22 09:33 ./fmwconfig/ovd/default/server.os_xml
-rw-r----- 1 oracle oinstall 117 Aug 22 09:32 ./fmwconfig/servers/BIP/loggers.exclude
-rw-r----- 1 oracle oinstall 117 Aug 22 09:32 ./fmwconfig/servers/EMGC_ADMINSERVER/loggers.exclude
-rw-r----- 1 oracle oinstall 117 Aug 22 09:32 ./fmwconfig/servers/EMGC_OMS1/loggers.exclude

Sure enough, the following files were modified, and the following file was the culprit:

[oracle@enkpoemac1 config]$ grep Cipher ./fmwconfig/components/OHS/instances/ohs1/moduleconf/httpd_em.conf
#SSLCipherSuite HIGH
SSLCipherSuite SSL_RSA_WITH_RC4_128_MD5,SSL_RSA_WITH_RC4_128_SHA,SSL_RSA_WITH_AES_128_CBC_SHA,SSL_RSA_WITH_AES_256_CBC_SHA,RSA_WITH_AES_128_CBC_SHA256,RSA_WITH_AES_256_CBC_SHA256,RSA_WITH_AES_128_GCM_SHA256,RSA_WITH_AES_256_GCM_SHA384,ECDHE_ECDSA_WITH_AES_128_CBC_SHA,ECDHE_ECDSA_WITH_AES_256_CBC_SHA,ECDHE_ECDSA_WITH_AES_128_CBC_SHA256,ECDHE_ECDSA_WITH_AES_256_CBC_SHA384,ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,ECDHE_RSA_WITH_RC4_128_SHA,ECDHE_RSA_WITH_3DES_EDE_CBC_SHA,ECDHE_RSA_WITH_AES_128_CBC_SHA,ECDHE_RSA_WITH_AES_256_CBC_SHA

[oracle@enkpoemac1 config]$ ls -al ./fmwconfig/components/OHS/instances/ohs1/moduleconf/httpd_em.conf
-rw-r--r-- 1 oracle oinstall 5659 Aug 22 09:29 ./fmwconfig/components/OHS/instances/ohs1/moduleconf/httpd_em.conf

I corrected the httpd_em.cfg file to include the following setting for SSL ciphers:

SSLCipherSuite ECDHE_RSA_WITH_AES_128_CBC_SHA,ECDHE_RSA_WITH_AES_256_CBC_SHA,RSA_WITH_AES_128_CBC_SHA256,RSA_WITH_AES_256_CBC_SHA256

It appears as though the SSLCipherSuite option was reset back to the OEM default after running the “emctl secure oms” command. I modified the SSLCipherSuite entry again with the correct list, and restarted the OMS. We ran another check with nmap, and it now is back to the state of only negotiating with approved SSL ciphers:

Andys-MacBook-Pro-3:~ root# nmap -sV –script ssl-enum-ciphers -p 4903 enkpoemac1
Password:

Starting Nmap 7.40 ( https://nmap.org ) at 2019-08-22 10:39 CDT

| ssl-enum-ciphers:
| TLSv1.2:
| ciphers:
| TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA (secp256r1) – A
| TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA (secp256r1) – A
| TLS_RSA_WITH_AES_128_CBC_SHA256 (rsa 1024) – A
| TLS_RSA_WITH_AES_256_CBC_SHA256 (rsa 1024) – A
| compressors:
| NULL
| cipher preference: client
|_ least strength: A

Leave a Reply

Your email address will not be published. Required fields are marked *