I have recently been installing Oracle Enterprise Manager at several sites, and one of the key requirements has been to ensure that the installation isn’t using insecure HTTPS protocols. Securing the OMS and agents typically consists of two components - ensuring that only secure SSL ciphers are being used, and shutting down protocols that have known vulnerabilities. Thankfully, Oracle has documented the procedures in two separate MOS notes:
Doc ID 2138391.1 - 13c: How to Disable Weak SSLCipherSuites in Enterprise Manager 13c Cloud Control
Doc ID 2212006.1 - EM 13c: Enterprise Manager 13c Cloud Control Configuration to Support Transport Layer Security Protocol:TLSv1.2 only
This isn’t a post about how to perform the task - that is outlined pretty well in the MOS documents. The interesting piece is the behavior that I saw after I thought that the task was completed. We went through the entire setup of both notes and requested a security scan, and found that the agent upload port (4903) was reporting weak ciphers. That was strange, because we had updated all of the files as described in the MOS notes.
Currently, the OMS used in these examples has gone through the procedure detailed in MOS note #2138391.1. At this point, weak SSL ciphers have been disabled, but it has not been secured to require TLSv1.2. We can validate the SSL ciphers with nmap below (nmap output abbreviated to show only the SSL cipher output):
Andys-MacBook-Pro-3:~ acolvin$ sudo nmap -sV --script ssl-enum-ciphers -p 4903 enkpoemac1 Password: Starting Nmap 7.40 ( https://nmap.org ) at 2019-08-22 09:17 CDT --- | ssl-enum-ciphers: | TLSv1.0: | ciphers: | TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA (secp256r1) - A | TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA (secp256r1) - A | compressors: | NULL | cipher preference: client | TLSv1.1: | ciphers: | TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA (secp256r1) - A | TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA (secp256r1) - A | compressors: | NULL | cipher preference: client | TLSv1.2: | ciphers: | TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA (secp256r1) - A | TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA (secp256r1) - A | TLS_RSA_WITH_AES_128_CBC_SHA256 (rsa 1024) - A | TLS_RSA_WITH_AES_256_CBC_SHA256 (rsa 1024) - A | compressors: | NULL | cipher preference: client |_ least strength: A ---
As you can see, all of the ciphers that are allowed match an “A” score. Just to see, I checked the timestamp of the configuration files.
[oracle@enkpoemac1 gc_inst]$ find . -name httpd_em.conf -exec ls -l {} \;
-rw-r--r-- 1 oracle oinstall 5804 Aug 22 09:02 ./user_projects/domains/GCDomain/config/fmwconfig/components/OHS/ohs1/moduleconf/httpd_em.conf
-rw-r--r-- 1 oracle oinstall 5804 Aug 22 09:04 ./user_projects/domains/GCDomain/config/fmwconfig/components/OHS/instances/ohs1/moduleconf/httpd_em.conf
At this point, I run the procedure on MOS note #2212006.1. In order to lock down the TLS protocols, we run a couple of “emctl secure XXX” commands and restart the OMS: [oracle@enkpoemac1 gc_inst]$ emctl secure oms -console -protocol "TLSv1.2" Oracle Enterprise Manager Cloud Control 13c Release 3 Copyright (c) 1996, 2018 Oracle Corporation. All rights reserved. Securing OMS... Started. Enter Enterprise Manager Root (SYSMAN) Password : Enter Agent Registration Password : Securing OMS... Successful Restart OMS [oracle@enkpoemac1 gc_inst]$ emctl secure oms -protocol "TLSv1.2" Oracle Enterprise Manager Cloud Control 13c Release 3 Copyright (c) 1996, 2018 Oracle Corporation. All rights reserved. Securing OMS... Started. Enter Enterprise Manager Root (SYSMAN) Password : Enter Agent Registration Password : Securing OMS... Successful Restart OMS [oracle@enkpoemac1 gc_inst]$ emctl stop oms -all Oracle Enterprise Manager Cloud Control 13c Release 3 Copyright (c) 1996, 2018 Oracle Corporation. All rights reserved. Stopping Oracle Management Server... WebTier Successfully Stopped Oracle Management Server Successfully Stopped Oracle Management Server is Down JVMD Engine is Down Stopping BI Publisher Server... BI Publisher Server Successfully Stopped AdminServer Successfully Stopped BI Publisher Server is Down [oracle@enkpoemac1 gc_inst]$ emctl start oms Oracle Enterprise Manager Cloud Control 13c Release 3 Copyright (c) 1996, 2018 Oracle Corporation. All rights reserved. Starting Oracle Management Server... WebTier Successfully Started Oracle Management Server Successfully Started Oracle Management Server is Up JVMD Engine is Up Starting BI Publisher Server ... BI Publisher Server Successfully Started BI Publisher Server is Up
After restarting the OMS, we run another scan, and see different results:
Andys-MacBook-Pro-3:~ acolvin$ sudo nmap -sV --script ssl-enum-ciphers -p 4903 enkpoemac1 Password: Starting Nmap 7.40 ( https://nmap.org ) at 2019-08-22 09:50 CDT --- | ssl-enum-ciphers: | TLSv1.2: | ciphers: | TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA (secp256r1) - D | TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA (secp256r1) - A | TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA (secp256r1) - A | TLS_ECDHE_RSA_WITH_RC4_128_SHA (secp256r1) - D | TLS_RSA_WITH_3DES_EDE_CBC_SHA (rsa 1024) - D | TLS_RSA_WITH_AES_128_CBC_SHA (rsa 1024) - A | TLS_RSA_WITH_AES_128_CBC_SHA256 (rsa 1024) - A | TLS_RSA_WITH_AES_128_GCM_SHA256 (rsa 1024) - A | TLS_RSA_WITH_AES_256_CBC_SHA (rsa 1024) - A | TLS_RSA_WITH_AES_256_CBC_SHA256 (rsa 1024) - A | TLS_RSA_WITH_AES_256_GCM_SHA384 (rsa 1024) - A | TLS_RSA_WITH_RC4_128_MD5 (rsa 1024) - D | TLS_RSA_WITH_RC4_128_SHA (rsa 1024) - D | compressors: | NULL | cipher preference: client | warnings: | 64-bit block cipher 3DES vulnerable to SWEET32 attack | Broken cipher RC4 is deprecated by RFC 7465 | Ciphersuite uses MD5 for message integrity |_ least strength: D ---
At this point, we decided to check the configuration files to see which files had been recently modified:
[oracle@enkpoemac1 config]$ find . -mmin -20 -type f -exec ls -l {} +
-rw-r----- 1 oracle oinstall 0 Aug 22 09:33 ./config.lok
-rw-r----- 1 oracle oinstall 60536 Aug 22 09:29 ./config.xml
-rw-r----- 1 oracle oinstall 2415 Aug 22 09:26 ./diagnostics/Module-FMWDFW-2818.xml
-rw-r--r-- 1 oracle oinstall 36868 Aug 22 09:29 ./fmwconfig/components/OHS/instances/ohs1/httpd.conf.emctl_secure
-rw-r----- 1 oracle oinstall 2920 Aug 22 09:26 ./fmwconfig/components/OHS/instances/ohs1/keystores/console/cwallet.sso
-rw-r----- 1 oracle oinstall 2843 Aug 22 09:26 ./fmwconfig/components/OHS/instances/ohs1/keystores/console/ewallet.p12
-rw-r----- 1 oracle oinstall 2920 Aug 22 09:29 ./fmwconfig/components/OHS/instances/ohs1/keystores/upload/cwallet.sso
-rw-r----- 1 oracle oinstall 2843 Aug 22 09:29 ./fmwconfig/components/OHS/instances/ohs1/keystores/upload/ewallet.p12
-rw-r--r-- 1 oracle oinstall 609 Aug 22 09:29 ./fmwconfig/components/OHS/instances/ohs1/moduleconf/agent_download.conf
-rw-r----- 1 oracle oinstall 609 Aug 22 09:26 ./fmwconfig/components/OHS/instances/ohs1/moduleconf/agent_download.conf.2019_08_22_09_26_45
-rw-r----- 1 oracle oinstall 609 Aug 22 09:29 ./fmwconfig/components/OHS/instances/ohs1/moduleconf/agent_download.conf.2019_08_22_09_29_50
-rw-r----- 1 oracle oinstall 1351 Aug 22 09:29 ./fmwconfig/components/OHS/instances/ohs1/moduleconf/httpd_bip.conf
-rw-r----- 1 oracle oinstall 1351 Aug 22 09:26 ./fmwconfig/components/OHS/instances/ohs1/moduleconf/httpd_bip.conf.2019_08_22_09_26_45
-rw-r----- 1 oracle oinstall 1351 Aug 22 09:29 ./fmwconfig/components/OHS/instances/ohs1/moduleconf/httpd_bip.conf.2019_08_22_09_29_50
-rw-r--r-- 1 oracle oinstall 5659 Aug 22 09:29 ./fmwconfig/components/OHS/instances/ohs1/moduleconf/httpd_em.conf
-rw-r----- 1 oracle oinstall 5347 Aug 22 09:26 ./fmwconfig/components/OHS/instances/ohs1/moduleconf/httpd_em.conf.2019_08_22_09_26_45
-rw-r----- 1 oracle oinstall 5347 Aug 22 09:29 ./fmwconfig/components/OHS/instances/ohs1/moduleconf/httpd_em.conf.2019_08_22_09_29_50
-rw-r----- 1 oracle oinstall 4051 Aug 22 09:29 ./fmwconfig/components/OHS/instances/ohs1/moduleconf/ssl_bip.conf
-rw-r----- 1 oracle oinstall 3972 Aug 22 09:26 ./fmwconfig/components/OHS/instances/ohs1/moduleconf/ssl_bip.conf.2019_08_22_09_26_45
-rw-r----- 1 oracle oinstall 3972 Aug 22 09:29 ./fmwconfig/components/OHS/instances/ohs1/moduleconf/ssl_bip.conf.2019_08_22_09_29_50
-rw-r----- 1 oracle oinstall 3981 Aug 22 09:29 ./fmwconfig/components/OHS/instances/ohs1/moduleconf/ssl_bip.conf.tmp
-rw-r----- 1 oracle oinstall 2105 Aug 22 09:29 ./fmwconfig/components/OHS/instances/ohs1/mod_wl_ohs.conf
-rw-r----- 1 oracle oinstall 2105 Aug 22 09:26 ./fmwconfig/components/OHS/instances/ohs1/mod_wl_ohs.conf.2019_08_22_09_26_29
-rw-r----- 1 oracle oinstall 2008 Aug 22 09:26 ./fmwconfig/components/OHS/instances/ohs1/mod_wl_ohs.conf.2019_08_22_09_26_45
-rw-r----- 1 oracle oinstall 2105 Aug 22 09:29 ./fmwconfig/components/OHS/instances/ohs1/mod_wl_ohs.conf.2019_08_22_09_29_38
-rw-r----- 1 oracle oinstall 2008 Aug 22 09:29 ./fmwconfig/components/OHS/instances/ohs1/mod_wl_ohs.conf.2019_08_22_09_29_50
-rw-r--r-- 1 oracle oinstall 2105 Aug 22 09:29 ./fmwconfig/components/OHS/instances/ohs1/mod_wl_ohs.conf.emctl_secure
-rw-r----- 1 oracle oinstall 3679 Aug 22 09:29 ./fmwconfig/components/OHS/instances/ohs1/ssl.conf
-rw-r----- 1 oracle oinstall 3682 Aug 22 09:26 ./fmwconfig/components/OHS/instances/ohs1/ssl.conf.2019_08_22_09_26_44
-rw-r----- 1 oracle oinstall 3679 Aug 22 09:26 ./fmwconfig/components/OHS/instances/ohs1/ssl.conf.2019_08_22_09_26_45
-rw-r----- 1 oracle oinstall 3679 Aug 22 09:29 ./fmwconfig/components/OHS/instances/ohs1/ssl.conf.2019_08_22_09_29_49
-rw-r----- 1 oracle oinstall 3679 Aug 22 09:29 ./fmwconfig/components/OHS/instances/ohs1/ssl.conf.2019_08_22_09_29_50
-rw-r--r-- 1 oracle oinstall 3679 Aug 22 09:29 ./fmwconfig/components/OHS/instances/ohs1/ssl.conf.emctl_secure
-rw-r--r-- 1 oracle oinstall 36868 Aug 22 09:29 ./fmwconfig/components/OHS/ohs1/httpd.conf.emctl_secure
-rw-r----- 1 oracle oinstall 2920 Aug 22 09:26 ./fmwconfig/components/OHS/ohs1/keystores/console/cwallet.sso
-rw-r--r-- 1 oracle oinstall 2843 Aug 22 09:26 ./fmwconfig/components/OHS/ohs1/keystores/console/ewallet.p12
-rw-r----- 1 oracle oinstall 2920 Aug 22 09:29 ./fmwconfig/components/OHS/ohs1/keystores/upload/cwallet.sso
-rw-r--r-- 1 oracle oinstall 2843 Aug 22 09:29 ./fmwconfig/components/OHS/ohs1/keystores/upload/ewallet.p12
-rw-r--r-- 1 oracle oinstall 609 Aug 22 09:29 ./fmwconfig/components/OHS/ohs1/moduleconf/agent_download.conf
-rw-r----- 1 oracle oinstall 609 Aug 22 09:26 ./fmwconfig/components/OHS/ohs1/moduleconf/agent_download.conf.2019_08_22_09_26_45
-rw-r----- 1 oracle oinstall 609 Aug 22 09:29 ./fmwconfig/components/OHS/ohs1/moduleconf/agent_download.conf.2019_08_22_09_29_50
-rw-r----- 1 oracle oinstall 1351 Aug 22 09:29 ./fmwconfig/components/OHS/ohs1/moduleconf/httpd_bip.conf
-rw-r----- 1 oracle oinstall 1351 Aug 22 09:26 ./fmwconfig/components/OHS/ohs1/moduleconf/httpd_bip.conf.2019_08_22_09_26_45
-rw-r----- 1 oracle oinstall 1351 Aug 22 09:29 ./fmwconfig/components/OHS/ohs1/moduleconf/httpd_bip.conf.2019_08_22_09_29_50
-rw-r--r-- 1 oracle oinstall 5659 Aug 22 09:29 ./fmwconfig/components/OHS/ohs1/moduleconf/httpd_em.conf
-rw-r----- 1 oracle oinstall 5347 Aug 22 09:26 ./fmwconfig/components/OHS/ohs1/moduleconf/httpd_em.conf.2019_08_22_09_26_45
-rw-r----- 1 oracle oinstall 5347 Aug 22 09:29 ./fmwconfig/components/OHS/ohs1/moduleconf/httpd_em.conf.2019_08_22_09_29_50
-rw-r----- 1 oracle oinstall 4051 Aug 22 09:29 ./fmwconfig/components/OHS/ohs1/moduleconf/ssl_bip.conf
-rw-r----- 1 oracle oinstall 3972 Aug 22 09:26 ./fmwconfig/components/OHS/ohs1/moduleconf/ssl_bip.conf.2019_08_22_09_26_45
-rw-r----- 1 oracle oinstall 3972 Aug 22 09:29 ./fmwconfig/components/OHS/ohs1/moduleconf/ssl_bip.conf.2019_08_22_09_29_50
-rw-r----- 1 oracle oinstall 3981 Aug 22 09:29 ./fmwconfig/components/OHS/ohs1/moduleconf/ssl_bip.conf.tmp
-rw-r----- 1 oracle oinstall 2105 Aug 22 09:29 ./fmwconfig/components/OHS/ohs1/mod_wl_ohs.conf
-rw-r----- 1 oracle oinstall 2105 Aug 22 09:26 ./fmwconfig/components/OHS/ohs1/mod_wl_ohs.conf.2019_08_22_09_26_29
-rw-r----- 1 oracle oinstall 2008 Aug 22 09:26 ./fmwconfig/components/OHS/ohs1/mod_wl_ohs.conf.2019_08_22_09_26_45
-rw-r----- 1 oracle oinstall 2105 Aug 22 09:29 ./fmwconfig/components/OHS/ohs1/mod_wl_ohs.conf.2019_08_22_09_29_38
-rw-r----- 1 oracle oinstall 2008 Aug 22 09:29 ./fmwconfig/components/OHS/ohs1/mod_wl_ohs.conf.2019_08_22_09_29_50
-rw-r--r-- 1 oracle oinstall 2105 Aug 22 09:29 ./fmwconfig/components/OHS/ohs1/mod_wl_ohs.conf.emctl_secure
-rw-r----- 1 oracle oinstall 3679 Aug 22 09:29 ./fmwconfig/components/OHS/ohs1/ssl.conf
-rw-r----- 1 oracle oinstall 3682 Aug 22 09:26 ./fmwconfig/components/OHS/ohs1/ssl.conf.2019_08_22_09_26_44
-rw-r----- 1 oracle oinstall 3679 Aug 22 09:26 ./fmwconfig/components/OHS/ohs1/ssl.conf.2019_08_22_09_26_45
-rw-r----- 1 oracle oinstall 3679 Aug 22 09:29 ./fmwconfig/components/OHS/ohs1/ssl.conf.2019_08_22_09_29_49
-rw-r----- 1 oracle oinstall 3679 Aug 22 09:29 ./fmwconfig/components/OHS/ohs1/ssl.conf.2019_08_22_09_29_50
-rw-r--r-- 1 oracle oinstall 3679 Aug 22 09:29 ./fmwconfig/components/OHS/ohs1/ssl.conf.emctl_secure
-rw-r----- 1 oracle oinstall 9513 Aug 22 09:33 ./fmwconfig/ovd/default/adapters.os_xml
-rw-r----- 1 oracle oinstall 3184 Aug 22 09:33 ./fmwconfig/ovd/default/server.os_xml
-rw-r----- 1 oracle oinstall 117 Aug 22 09:32 ./fmwconfig/servers/BIP/loggers.exclude
-rw-r----- 1 oracle oinstall 117 Aug 22 09:32 ./fmwconfig/servers/EMGC_ADMINSERVER/loggers.exclude
-rw-r----- 1 oracle oinstall 117 Aug 22 09:32 ./fmwconfig/servers/EMGC_OMS1/loggers.exclude
Sure enough, the following files were modified, and the following file was the culprit: [oracle@enkpoemac1 config]$ grep Cipher ./fmwconfig/components/OHS/instances/ohs1/moduleconf/httpd_em.conf #SSLCipherSuite HIGH SSLCipherSuite SSL_RSA_WITH_RC4_128_MD5,SSL_RSA_WITH_RC4_128_SHA,SSL_RSA_WITH_AES_128_CBC_SHA,SSL_RSA_WITH_AES_256_CBC_SHA,RSA_WITH_AES_128_CBC_SHA256,RSA_WITH_AES_256_CBC_SHA256,RSA_WITH_AES_128_GCM_SHA256,RSA_WITH_AES_256_GCM_SHA384,ECDHE_ECDSA_WITH_AES_128_CBC_SHA,ECDHE_ECDSA_WITH_AES_256_CBC_SHA,ECDHE_ECDSA_WITH_AES_128_CBC_SHA256,ECDHE_ECDSA_WITH_AES_256_CBC_SHA384,ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,ECDHE_RSA_WITH_RC4_128_SHA,ECDHE_RSA_WITH_3DES_EDE_CBC_SHA,ECDHE_RSA_WITH_AES_128_CBC_SHA,ECDHE_RSA_WITH_AES_256_CBC_SHA [oracle@enkpoemac1 config]$ ls -al ./fmwconfig/components/OHS/instances/ohs1/moduleconf/httpd_em.conf -rw-r--r-- 1 oracle oinstall 5659 Aug 22 09:29 ./fmwconfig/components/OHS/instances/ohs1/moduleconf/httpd_em.conf
I corrected the httpd_em.cfg file to include the following setting for SSL ciphers:
SSLCipherSuite ECDHE_RSA_WITH_AES_128_CBC_SHA,ECDHE_RSA_WITH_AES_256_CBC_SHA,RSA_WITH_AES_128_CBC_SHA256,RSA_WITH_AES_256_CBC_SHA256
It appears as though the SSLCipherSuite option was reset back to the OEM default after running the “emctl secure oms” command. I modified the SSLCipherSuite entry again with the correct list, and restarted the OMS. We ran another check with nmap, and it now is back to the state of only negotiating with approved SSL ciphers:
Andys-MacBook-Pro-3:~ root# nmap -sV --script ssl-enum-ciphers -p 4903 enkpoemac1
Password:
Starting Nmap 7.40 ( https://nmap.org ) at 2019-08-22 10:39 CDT
---
| ssl-enum-ciphers:
| TLSv1.2:
| ciphers:
| TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA (secp256r1) - A
| TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA (secp256r1) - A
| TLS_RSA_WITH_AES_128_CBC_SHA256 (rsa 1024) - A
| TLS_RSA_WITH_AES_256_CBC_SHA256 (rsa 1024) - A
| compressors:
| NULL
| cipher preference: client
|_ least strength: A
---
Did you raise an Oracle bug for this?
Yes, I filed an SR. Oracle created bug #30469628, but I can’t see the current status for it.