Oracle-Ninja.com http://blog.oracle-ninja.com Andy Colvin's Oracle Blog Fri, 08 Mar 2013 04:06:35 +0000 en-US hourly 1 http://wordpress.org/?v=3.5.1 Exadata 11.2.3.2.1 NFS Issues – Ksplice Support for Exadata? http://blog.oracle-ninja.com/2013/03/exadata-11-2-3-2-1-nfs-issues-ksplice-support-for-exadata/?utm_source=rss&utm_medium=rss&utm_campaign=exadata-11-2-3-2-1-nfs-issues-ksplice-support-for-exadata http://blog.oracle-ninja.com/2013/03/exadata-11-2-3-2-1-nfs-issues-ksplice-support-for-exadata/#comments Fri, 08 Mar 2013 04:05:08 +0000 Andy Colvin http://blog.oracle-ninja.com/?p=600 When the 11.2.3.2.1 release of the Exadata Storage Server software was released, I was a little excited.  There were numerous oneoff patches for the previous release, 11.2.3.2.0, which was the first version to support the Exadata X3, writeback flashcache, run UEK on the X#-2 systems, etc.  With that many large changes introduced in one version, it was likely to see some bugs in the .0 release.  Fortunately, Oracle was quick to fix many of those issues, but it resulted in several separate patches to update the cellsrv software.

I was working with a colleague last week where we ready to apply this patch to a customer's Exadata system.  Everything went off without a hitch - upgrading from 11.2.2.4.2 straight to 11.2.3.2.1.  We even applied the patch to the customer's quarter rack in rolling mode, which took under 6 hours to complete.  After everything was back up and running, we took an archive log backup using RMAN.  For this customer, we back everything up to NFS because it won't fit within the FRA, and they don't want to leave backups inside the production system.  We were greeted with a strange error when we tried to kick off the backup job in RMAN:

?View Code NONE
RMAN> run {
2>   ALLOCATE CHANNEL DISK1 DEVICE TYPE DISK;
3>   BACKUP DATABASE FORMAT '/mnt/nfs/actest_%U';
4>   RELEASE CHANNEL DISK1;
5> }
 
using target database control file instead of recovery catalog
allocated channel: DISK1
channel DISK1: SID=397 instance=ACTEST1 device type=DISK
 
Starting backup at 13-02-28 21:38
channel DISK1: starting full datafile backup set
channel DISK1: specifying datafile(s) in backup set
input datafile file number=00007 name=+DATA/actest/datafile/tanel_bigfile.325.808412931
input datafile file number=00006 name=+DATA/actest/datafile/ts_data.380.779860027
input datafile file number=00001 name=+DATA/actest/datafile/system.367.779029515
input datafile file number=00002 name=+DATA/actest/datafile/sysaux.368.779029555
input datafile file number=00003 name=+DATA/actest/datafile/undotbs1.369.779029595
input datafile file number=00004 name=+DATA/actest/datafile/undotbs2.371.779029649
input datafile file number=00005 name=+DATA/actest/datafile/users.372.779029687
channel DISK1: starting piece 1 at 13-02-28 21:38
released channel: DISK1
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03009: failure of backup command on DISK1 channel at 02/28/2013 21:38:37
ORA-19504: failed to create file "/mnt/nfs/actest_1jo34pas_1_1"
ORA-27044: unable to write the header block of file
Linux-x86_64 Error: 12: Cannot allocate memory
Additional information: 3

It didn't matter what we were trying to back up, just that it was going to NFS.  This backup job had worked fine prior to the patch (we took a backup immediately preceding the maintenance window), but we had applied both a database bundle patch (this database was 11.2.0.2) and the latest storage server patch (11.2.3.2.1), which updates the Linux OS to OEL 5.8, as well as introduces the Oracle Unbreakable Enterprise Kernel into the mix.

We checked the mount options to make sure that everything was ok, and saw that it was:

?View Code NONE
[enkdb01:oracle:ACTEST1] /u01/app/oracle/product/11.2.0.3/dbhome_2/rdbms/lib 
> mount | grep "/mnt/nfs"
192.168.12.22:/export/nfs on /mnt/nfs type nfs (rw,bg,hard,nointr,rsize=32768,wsize=32768,tcp,nfsvers=3,timeo=600,actimeo=0,addr=192.168.12.22)

After poking around a bit, we opened a service request, which was answered pretty quickly by Oracle support.  It turns out that there is a known bug with the NFS driver included in the version of the UEK packaged with 11.2.3.2.1. Oracle provided 3 possible fixes, which I'll detail below. The fixes were:


Enable direct NFS


First, I'll cover enabling direct NFS.  I actually have a blog post in the works, but to give you a quick once-over on it, here goes.  Direct NFS is a custom-built NFS driver that does not interfere with the kernel's NFS driver.  Any operations that go through the database (RMAN, data pump, etc) will use this special driver that is optimized for database operations.  Processes that utilize direct NFS operate in user space (like FUSE), which has less overhead than kernel space.  Because direct NFS does not use the bad kernel NFS driver for backup operations, the bug is negated.  It goes without saying that if your databases are interacting with NFS, you should use direct NFS.  There is no penalty for doing so, and it's really easy to do.  Shut down your database instance, and relink for direct NFS:

?View Code NONE
[enkdb01:oracle:ACTEST1] /home/oracle
>  cd $ORACLE_HOME/rdbms/lib
[enkdb01:oracle:ACTEST1] /u01/app/oracle/product/11.2.0.3/dbhome_2/rdbms/lib 
>  make -f ins_rdbms.mk dnfs_on
rm -f /u01/app/oracle/product/11.2.0.3/dbhome_2/lib/libodm11.so; cp /u01/app/oracle/product/11.2.0.3/dbhome_2/lib/libnfsodm11.so /u01/app/oracle/product/11.2.0.3/dbhome_2/lib/libodm11.so

That's it.  If you see the following line in your alert log on instance startup, you're on your way:

?View Code NONE
Oracle instance running with ODM: Oracle Direct NFS ODM Library Version 3.0

Once you start to access the NFS devices, you can query v$dnfs_servers to ensure that the database is using direct NFS.

?View Code NONE
SYS:ACTEST1> select * from v$dnfs_servers;
 
         ID        SVRNAME        DIRNAME     MNTPORT     NFSPORT       WTMAX       RTMAX
----------- -------------- -------------- ----------- ----------- ----------- -----------
          1  192.168.12.22    /export/nfs         939        2049           0           0


Switch to non-UEK


The next option provided by Oracle support was to switch from the Unbreakable Enterprise Kernel included with 11.2.3.2.1 to the RedHat compatible kernel.  To do this, follow the instructions in the readme for the patch (patch #14522699).

First, remove the problematic UEK packages:

?View Code NONE
[enkdb01:root] /root 
> yum remove exadata-sun-computenode kernel-uek-devel kernel-uek-debuginfo kernel-uek-doc kernel-uek-debuginfo-common kernel-uek-headers
Loaded plugins: security
Setting up Remove Process
Resolving Dependencies
--> Running transaction check
---> Package exadata-sun-computenode.x86_64 0:11.2.3.2.1.130109-1 set to be erased
---> Package kernel-uek-debuginfo.x86_64 0:2.6.32-400.11.1.el5uek set to be erased
---> Package kernel-uek-debuginfo-common.x86_64 0:2.6.32-400.11.1.el5uek set to be erased
---> Package kernel-uek-devel.x86_64 0:2.6.32-400.11.1.el5uek set to be erased
---> Package kernel-uek-doc.noarch 0:2.6.32-400.11.1.el5uek set to be erased
---> Package kernel-uek-headers.x86_64 0:2.6.32-400.11.1.el5uek set to be erased
--> Processing Dependency: kernel-headers for package: glibc-headers
--> Processing Dependency: kernel-headers >= 2.2.1 for package: glibc-headers
--> Running transaction check
---> Package glibc-headers.x86_64 0:2.5-81.el5_8.7 set to be erased
--> Processing Dependency: glibc-headers for package: glibc-devel
--> Processing Dependency: glibc-headers = 2.5-81.el5_8.7 for package: glibc-devel
--> Processing Dependency: glibc-headers for package: glibc-devel
--> Processing Dependency: glibc-headers = 2.5-81.el5_8.7 for package: glibc-devel
--> Running transaction check
---> Package glibc-devel.i386 0:2.5-81.el5_8.7 set to be erased
--> Processing Dependency: glibc-devel >= 2.2.90-12 for package: gcc
--> Processing Dependency: glibc-devel >= 2.2.90-12 for package: compat-gcc-34
---> Package glibc-devel.x86_64 0:2.5-81.el5_8.7 set to be erased
--> Running transaction check
---> Package compat-gcc-34.x86_64 0:3.4.6-4 set to be erased
--> Processing Dependency: compat-gcc-34 = 3.4.6-4 for package: compat-gcc-34-c++
---> Package gcc.x86_64 0:4.1.2-52.el5_8.1 set to be erased
--> Processing Dependency: gcc = 4.1.2-52.el5_8.1 for package: gcc-c++
--> Running transaction check
---> Package compat-gcc-34-c++.x86_64 0:3.4.6-4 set to be erased
---> Package gcc-c++.x86_64 0:4.1.2-52.el5_8.1 set to be erased
--> Finished Dependency Resolution
 
Dependencies Resolved
 
========================================================================================================================================================
 Package                                        Arch                      Version                                    Repository                    Size
========================================================================================================================================================
Removing:
 exadata-sun-computenode                        x86_64                    11.2.3.2.1.130109-1                        installed                     95 k
 kernel-uek-debuginfo                           x86_64                    2.6.32-400.11.1.el5uek                     installed                    1.0 G
 kernel-uek-debuginfo-common                    x86_64                    2.6.32-400.11.1.el5uek                     installed                    163 M
 kernel-uek-devel                               x86_64                    2.6.32-400.11.1.el5uek                     installed                     45 M
 kernel-uek-doc                                 noarch                    2.6.32-400.11.1.el5uek                     installed                     33 M
 kernel-uek-headers                             x86_64                    2.6.32-400.11.1.el5uek                     installed                    2.2 M
Removing for dependencies:
 compat-gcc-34                                  x86_64                    3.4.6-4                                    installed                     12 M
 compat-gcc-34-c++                              x86_64                    3.4.6-4                                    installed                     84 M
 gcc                                            x86_64                    4.1.2-52.el5_8.1                           installed                    9.9 M
 gcc-c++                                        x86_64                    4.1.2-52.el5_8.1                           installed                    7.5 M
 glibc-devel                                    i386                      2.5-81.el5_8.7                             installed                    4.9 M
 glibc-devel                                    x86_64                    2.5-81.el5_8.7                             installed                    7.0 M
 glibc-headers                                  x86_64                    2.5-81.el5_8.7                             installed                    1.9 M
 
Transaction Summary
========================================================================================================================================================
Remove       13 Package(s)
Reinstall     0 Package(s)
Downgrade     0 Package(s)
 
Is this ok [y/N]: y
Downloading Packages:
Running rpm_check_debug
Running Transaction Test
Finished Transaction Test
Transaction Test Succeeded
Running Transaction
  Erasing        : kernel-uek-debuginfo                                                                                                            1/13 
  Erasing        : kernel-uek-debuginfo-common                                                                                                     2/13 
  Erasing        : gcc                                                                                                                             3/13 
  Erasing        : glibc-headers                                                                                                                   4/13 
  Erasing        : compat-gcc-34                                                                                                                   5/13 
  Erasing        : glibc-devel                                                                                                                     6/13 
  Erasing        : glibc-devel                                                                                                                     7/13 
  Erasing        : compat-gcc-34-c++                                                                                                               8/13 
  Erasing        : kernel-uek-headers                                                                                                              9/13 
  Erasing        : gcc-c++                                                                                                                        10/13 
  Erasing        : kernel-uek-devel                                                                                                               11/13 
  Erasing        : exadata-sun-computenode                                                                                                        12/13 
  Erasing        : kernel-uek-doc                                                                                                                 13/13 
 
Removed:
  exadata-sun-computenode.x86_64 0:11.2.3.2.1.130109-1                           kernel-uek-debuginfo.x86_64 0:2.6.32-400.11.1.el5uek                   
  kernel-uek-debuginfo-common.x86_64 0:2.6.32-400.11.1.el5uek                    kernel-uek-devel.x86_64 0:2.6.32-400.11.1.el5uek                       
  kernel-uek-doc.noarch 0:2.6.32-400.11.1.el5uek                                 kernel-uek-headers.x86_64 0:2.6.32-400.11.1.el5uek                     
 
Dependency Removed:
  compat-gcc-34.x86_64 0:3.4.6-4      compat-gcc-34-c++.x86_64 0:3.4.6-4    gcc.x86_64 0:4.1.2-52.el5_8.1           gcc-c++.x86_64 0:4.1.2-52.el5_8.1  
  glibc-devel.i386 0:2.5-81.el5_8.7   glibc-devel.x86_64 0:2.5-81.el5_8.7   glibc-headers.x86_64 0:2.5-81.el5_8.7  
 
Complete!

Next, modify the exclude line in /etc/yum.conf to include kernel-uek-headers

?View Code NONE
exclude=up2date kernel-uek-headers

Stop and disable CRS on the node

?View Code NONE
[enkdb01:root] /root 
> crsctl stop crs
[enkdb01:root] /root 
> crsctl disable crs
CRS-4621: Oracle High Availability Services autostart is disabled.

Install the exadata-sun-computenode-non-uek package from the yum repository.

?View Code NONE
[enkdb01:root] /root 
> yum --enablerepo=exadata_dbserver_11.2.3.2.1_x86_64_base install exadata-sun-computenode-non-uek
Loaded plugins: security
exadata_dbserver_11.2.3.2.1_x86_64_base                                                                                          | 1.9 kB     00:00     
Excluding Packages in global exclude list
Finished
Setting up Install Process
Resolving Dependencies
--> Running transaction check
---> Package exadata-sun-computenode-non-uek.x86_64 0:11.2.3.2.1.130109-1 set to be updated
exadata_dbserver_11.2.3.2.1_x86_64_base/filelists_db                                                                             | 680 kB     00:00     
--> Processing Dependency: gcc = 4.1.1-52.el5_8 for package: exadata-sun-computenode-non-uek
--> Processing Dependency: glibc-kernheaders = 3.0-46 for package: exadata-sun-computenode-non-uek
--> Processing Dependency: kernel-headers = 2.6.18-308.24.1.0.1.el5 for package: exadata-sun-computenode-non-uek
--> Processing Dependency: gcc = 4.1.2-52.el5_8.1 for package: exadata-sun-computenode-non-uek
--> Processing Dependency: ofa-2.6.18-308.24.1.0.1.el5 = 1.5.1-4.0.58 for package: exadata-sun-computenode-non-uek
--> Processing Dependency: glibc-devel = 2.5-81.el5_8.7 for package: exadata-sun-computenode-non-uek
--> Processing Dependency: glibc-headers = 2.5-81.el5_8.7 for package: exadata-sun-computenode-non-uek
--> Processing Dependency: gcc-c++ = 4.1.1-52.el5_8 for package: exadata-sun-computenode-non-uek
--> Processing Dependency: gcc-c++ = 4.1.2-52.el5_8.1 for package: exadata-sun-computenode-non-uek
--> Processing Dependency: megaraid_sas-2.6.18-308.24.1.0.1.el5 = v00.00.06.12-ORCL.2 for package: exadata-sun-computenode-non-uek
--> Running transaction check
---> Package gcc.x86_64 0:4.1.2-52.el5_8.1 set to be updated
---> Package gcc-c++.x86_64 0:4.1.2-52.el5_8.1 set to be updated
---> Package glibc-devel.x86_64 0:2.5-81.el5_8.7 set to be updated
---> Package glibc-headers.x86_64 0:2.5-81.el5_8.7 set to be updated
---> Package kernel-headers.x86_64 0:2.6.18-308.24.1.0.1.el5 set to be updated
---> Package megaraid_sas-2.6.18-308.24.1.0.1.el5.x86_64 0:v00.00.06.12-ORCL.2 set to be updated
---> Package ofa-2.6.18-308.24.1.0.1.el5.x86_64 0:1.5.1-4.0.58 set to be updated
--> Finished Dependency Resolution
 
Dependencies Resolved
 
========================================================================================================================================================
 Package                                       Arch            Version                           Repository                                        Size
========================================================================================================================================================
Installing:
 exadata-sun-computenode-non-uek               x86_64          11.2.3.2.1.130109-1               exadata_dbserver_11.2.3.2.1_x86_64_base           47 k
Installing for dependencies:
 gcc                                           x86_64          4.1.2-52.el5_8.1                  exadata_dbserver_11.2.3.2.1_x86_64_base          5.3 M
 gcc-c++                                       x86_64          4.1.2-52.el5_8.1                  exadata_dbserver_11.2.3.2.1_x86_64_base          3.8 M
 glibc-devel                                   x86_64          2.5-81.el5_8.7                    exadata_dbserver_11.2.3.2.1_x86_64_base          2.4 M
 glibc-headers                                 x86_64          2.5-81.el5_8.7                    exadata_dbserver_11.2.3.2.1_x86_64_base          597 k
 kernel-headers                                x86_64          2.6.18-308.24.1.0.1.el5           exadata_dbserver_11.2.3.2.1_x86_64_base          1.4 M
 megaraid_sas-2.6.18-308.24.1.0.1.el5          x86_64          v00.00.06.12-ORCL.2               exadata_dbserver_11.2.3.2.1_x86_64_base          287 k
 ofa-2.6.18-308.24.1.0.1.el5                   x86_64          1.5.1-4.0.58                      exadata_dbserver_11.2.3.2.1_x86_64_base          846 k
 
Transaction Summary
========================================================================================================================================================
Install       8 Package(s)
Upgrade       0 Package(s)
 
Total download size: 15 M
Is this ok [y/N]: y
Downloading Packages:
--------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                   4.5 GB/s |  15 MB     00:00     
Running rpm_check_debug
Running Transaction Test
Finished Transaction Test
Transaction Test Succeeded
Running Transaction
  Installing     : kernel-headers                                                                                                                   1/8 
  Installing     : glibc-headers                                                                                                                    2/8 
  Installing     : glibc-devel                                                                                                                      3/8 
  Installing     : ofa-2.6.18-308.24.1.0.1.el5                                                                                                      4/8 
  Installing     : megaraid_sas-2.6.18-308.24.1.0.1.el5                                                                                             5/8 
Current initrd saved in /boot/initrd-2.6.18-308.24.1.0.1.el5.img.bak.066-2200.11.2.3.2.1.130109
initrd image created: /boot/initrd-2.6.18-308.24.1.0.1.el5.img
  Installing     : gcc                                                                                                                              6/8 
  Installing     : gcc-c++                                                                                                                          7/8 
  Installing     : exadata-sun-computenode-non-uek                                                                                                  8/8 
 
Installed:
  exadata-sun-computenode-non-uek.x86_64 0:11.2.3.2.1.130109-1                                                                                              
 
Dependency Installed:
  gcc.x86_64 0:4.1.2-52.el5_8.1                                        gcc-c++.x86_64 0:4.1.2-52.el5_8.1                                                   
  glibc-devel.x86_64 0:2.5-81.el5_8.7                                  glibc-headers.x86_64 0:2.5-81.el5_8.7                                               
  kernel-headers.x86_64 0:2.6.18-308.24.1.0.1.el5                      megaraid_sas-2.6.18-308.24.1.0.1.el5.x86_64 0:v00.00.06.12-ORCL.2                   
  ofa-2.6.18-308.24.1.0.1.el5.x86_64 0:1.5.1-4.0.58                   
 
Complete!
 
[enkdb01:root] /root 
> 
Remote broadcast message (Thu Mar  7 22:00:43 2013):
 
Exadata post install steps started.
It may take up to 2 minutes.
The db node will be rebooted upon successful completion.

This will cause the compute node to reboot.  When it has come up, check to make sure that the server booted to the non-UEK version of the kernel:

?View Code NONE
[enkdb01:root] /root 
> uname -a
Linux enkdb01.enkitec.com 2.6.18-308.24.1.0.1.el5 #1 SMP Tue Dec 4 16:00:29 PST 2012 x86_64 x86_64 x86_64 GNU/Linux

Remove the unused UEK RPMs, and install the glibc RPM.

?View Code NONE
[enkdb01:root] /root 
> yum remove kernel-uek kernel-uek-firmware
Loaded plugins: security
Setting up Remove Process
Resolving Dependencies
--> Running transaction check
---> Package kernel-uek.x86_64 0:2.6.32-300.7.2.el5uek set to be erased
--> Processing Dependency: kernel-uek = 2.6.32-300.7.2.el5uek for package: megaraid_sas-2.6.32-300.7.2.el5uek
--> Processing Dependency: kernel-uek = 2.6.32-300.7.2.el5uek for package: ofa-2.6.32-300.7.2.el5uek
---> Package kernel-uek.x86_64 0:2.6.32-300.7.3.el5uek set to be erased
--> Processing Dependency: kernel-uek = 2.6.32-300.7.3.el5uek for package: ofa-2.6.32-300.7.3.el5uek
--> Processing Dependency: kernel-uek = 2.6.32-300.7.3.el5uek for package: megaraid_sas-2.6.32-300.7.3.el5uek
---> Package kernel-uek.x86_64 0:2.6.32-400.1.1.el5uek set to be erased
--> Processing Dependency: kernel-uek = 2.6.32-400.1.1.el5uek for package: ofa-2.6.32-400.1.1.el5uek
---> Package kernel-uek.x86_64 0:2.6.32-400.11.1.el5uek set to be erased
--> Processing Dependency: kernel-uek = 2.6.32-400.11.1.el5uek for package: ofa-2.6.32-400.11.1.el5uek
---> Package kernel-uek-firmware.noarch 0:2.6.32-400.11.1.el5uek set to be erased
--> Running transaction check
---> Package megaraid_sas-2.6.32-300.7.2.el5uek.x86_64 0:v00.00.06.12-ORCL.2 set to be erased
---> Package megaraid_sas-2.6.32-300.7.3.el5uek.x86_64 0:v00.00.06.12-ORCL.2 set to be erased
---> Package ofa-2.6.32-300.7.2.el5uek.x86_64 0:1.5.1-4.0.58 set to be erased
---> Package ofa-2.6.32-300.7.3.el5uek.x86_64 0:1.5.1-4.0.58 set to be erased
---> Package ofa-2.6.32-400.1.1.el5uek.x86_64 0:1.5.1-4.0.58 set to be erased
---> Package ofa-2.6.32-400.11.1.el5uek.x86_64 0:1.5.1-4.0.58 set to be erased
--> Finished Dependency Resolution
 
Dependencies Resolved
 
==========================================================================================================================================================================
 Package                                                  Arch                         Version                                      Repository                       Size
==========================================================================================================================================================================
Removing:
 kernel-uek                                               x86_64                       2.6.32-300.7.2.el5uek                        installed                        81 M
 kernel-uek                                               x86_64                       2.6.32-300.7.3.el5uek                        installed                        81 M
 kernel-uek                                               x86_64                       2.6.32-400.1.1.el5uek                        installed                        82 M
 kernel-uek                                               x86_64                       2.6.32-400.11.1.el5uek                       installed                        82 M
 kernel-uek-firmware                                      noarch                       2.6.32-400.11.1.el5uek                       installed                       5.3 M
Removing for dependencies:
 megaraid_sas-2.6.32-300.7.2.el5uek                       x86_64                       v00.00.06.12-ORCL.2                          installed                       1.1 M
 megaraid_sas-2.6.32-300.7.3.el5uek                       x86_64                       v00.00.06.12-ORCL.2                          installed                       1.1 M
 ofa-2.6.32-300.7.2.el5uek                                x86_64                       1.5.1-4.0.58                                 installed                       3.4 M
 ofa-2.6.32-300.7.3.el5uek                                x86_64                       1.5.1-4.0.58                                 installed                       3.4 M
 ofa-2.6.32-400.1.1.el5uek                                x86_64                       1.5.1-4.0.58                                 installed                       3.4 M
 ofa-2.6.32-400.11.1.el5uek                               x86_64                       1.5.1-4.0.58                                 installed                       3.4 M
 
Transaction Summary
==========================================================================================================================================================================
Remove       11 Package(s)
Reinstall     0 Package(s)
Downgrade     0 Package(s)
 
Is this ok [y/N]: y
Downloading Packages:
Running rpm_check_debug
Running Transaction Test
Finished Transaction Test
Transaction Test Succeeded
Running Transaction
  Erasing        : kernel-uek                                                                                                                                        1/11 
  Erasing        : ofa-2.6.32-400.1.1.el5uek                                                                                                                         2/11 
  Erasing        : kernel-uek                                                                                                                                        3/11 
  Erasing        : ofa-2.6.32-300.7.2.el5uek                                                                                                                         4/11 
Megaraid path valid. Proceeding.
  Erasing        : megaraid_sas-2.6.32-300.7.3.el5uek                                                                                                                5/11 
  Erasing        : ofa-2.6.32-300.7.3.el5uek                                                                                                                         6/11 
  Erasing        : ofa-2.6.32-400.11.1.el5uek                                                                                                                        7/11 
  Erasing        : kernel-uek                                                                                                                                        8/11 
  Erasing        : kernel-uek-firmware                                                                                                                               9/11 
Megaraid path valid. Proceeding.
Warning. /lib/modules/2.6.32-300.7.2.el5uek/kernel/drivers/scsi/megaraid/megaraid_sas.ko does not match file from the uninstalling rpm
Warning. Leave megaraid_sas driver as is
+ exit 0
  Erasing        : megaraid_sas-2.6.32-300.7.2.el5uek                                                                                                               10/11 
  Erasing        : kernel-uek                                                                                                                                       11/11 
 
Removed:
  kernel-uek.x86_64 0:2.6.32-300.7.2.el5uek            kernel-uek.x86_64 0:2.6.32-300.7.3.el5uek                     kernel-uek.x86_64 0:2.6.32-400.1.1.el5uek          
  kernel-uek.x86_64 0:2.6.32-400.11.1.el5uek           kernel-uek-firmware.noarch 0:2.6.32-400.11.1.el5uek          
 
Dependency Removed:
  megaraid_sas-2.6.32-300.7.2.el5uek.x86_64 0:v00.00.06.12-ORCL.2                     megaraid_sas-2.6.32-300.7.3.el5uek.x86_64 0:v00.00.06.12-ORCL.2                    
  ofa-2.6.32-300.7.2.el5uek.x86_64 0:1.5.1-4.0.58                                     ofa-2.6.32-300.7.3.el5uek.x86_64 0:1.5.1-4.0.58                                    
  ofa-2.6.32-400.1.1.el5uek.x86_64 0:1.5.1-4.0.58                                     ofa-2.6.32-400.11.1.el5uek.x86_64 0:1.5.1-4.0.58                                   
 
Complete!
 
[enkdb01:root] /root 
> yum --enablerepo=exadata_dbserver_11.2.3.2.1_x86_64_base install glibc-devel.i386
Loaded plugins: security
Excluding Packages in global exclude list
Finished
Setting up Install Process
Resolving Dependencies
--> Running transaction check
---> Package glibc-devel.i386 0:2.5-81.el5_8.7 set to be updated
--> Finished Dependency Resolution
 
Dependencies Resolved
 
==========================================================================================================================================================================
 Package                           Arch                       Version                                 Repository                                                     Size
==========================================================================================================================================================================
Installing:
 glibc-devel                       i386                       2.5-81.el5_8.7                          exadata_dbserver_11.2.3.2.1_x86_64_base                       2.1 M
 
Transaction Summary
==========================================================================================================================================================================
Install       1 Package(s)
Upgrade       0 Package(s)
 
Total download size: 2.1 M
Is this ok [y/N]: y
Downloading Packages:
Running rpm_check_debug
Running Transaction Test
Finished Transaction Test
Transaction Test Succeeded
Running Transaction
  Installing     : glibc-devel                                                                                                                                        1/1 
 
Installed:
  glibc-devel.i386 0:2.5-81.el5_8.7                                                                                                                                       
 
Complete!

Finally, relink the Oracle homes, restart CRS, and try backing up your database.

?View Code NONE
[enkdb01:root] /root 
> crsctl enable crs
CRS-4622: Oracle High Availability Services autostart is enabled.
 
[enkdb01:root] /root 
> crsctl start crs
CRS-4123: Oracle High Availability Services has been started.


Ksplice on the UEK


The last option is the most interesting, in my opinion - applying an online kernel patch using ksplice.  If you're not familiar with ksplice, it is a magical little piece of software acquired by Oracle which allows administrators to apply kernel patches to the kernel running in memory, eliminating the need for a reboot (generally).  Because ksplice isn't fully supported on Exadata, we have a few limitations on what can be done, but this is an interesting glimpse into what may be coming in future releases.  This patch (provided by Oracle Support) is a single ksplice patch for this bug.  To install, unzip the patch, copy the patch file to the uptrack directory, and apply the patch.  All of the following must be done as root.

First, make sure that there aren't any patches installed

?View Code NONE
[enkdb01:root] /tmp/ksplice_patch/uptrack
> ./ksplice-view
 
[enkdb01:root] /tmp/ksplice_patch/uptrack
>

Next, apply the patch - this takes about 3 seconds to apply:

?View Code NONE
[enkdb01:root] /tmp/ksplice_patch/uptrack
> ./ksplice-apply ksplice-9j3o3021.tar.gz
Done!

Check that the patch has applied successfully

?View Code NONE
[enkdb01:root] /tmp/ksplice_patch/uptrack
> ./ksplice-view
9j3o3021: Ksplice test suite: oracle-uek5-2.6.32-400.11.1.el5uek-amd64

That's all there is to it.  This will need to be applied on both nodes, and since it's not a full installation of ksplice, the patch will have to be reapplied upon each reboot of the compute node.  If you really want to, you could create a script to apply the patch when the server boots up.  On full installations of ksplice, the kernel will be "respliced" upon each reboot.

 

]]> http://blog.oracle-ninja.com/2013/03/exadata-11-2-3-2-1-nfs-issues-ksplice-support-for-exadata/feed/ 9 2013 Presentation Schedule http://blog.oracle-ninja.com/2013/02/2013-presentation-schedule/?utm_source=rss&utm_medium=rss&utm_campaign=2013-presentation-schedule http://blog.oracle-ninja.com/2013/02/2013-presentation-schedule/#comments Fri, 01 Feb 2013 16:33:49 +0000 Andy Colvin http://blog.oracle-ninja.com/?p=587 Well, it's been a really busy past few months, and I hate to admit it, but I've been neglecting this space more than anything. Despite having many different posts in the works, nothing is quite finished yet. I do have a little time to mention a few of my upcoming speaking events, though.

First, I'll be in Denver for the Rocky Mountain Oracle Users Group Training Days 2013, February 11-13.  Enkitec has more than a few sessions during the conference, ranging from Exadata to Big Data to APEX.  Check out the agenda for a full list.  Also, we'll have an Enkitec booth there, which is the place that I'll be hanging out when I'm not attending the numerous interesting sessions.  I have 2 presentations during the week:

I'm really looking forward to the RMAN session, as I'll finally be able to talk about some of the new stuff that's coming up in 12c.

I'm also going to be presenting a couple of Expert Seminars for Oracle University in the coming months.  I'll be presenting a "Getting Ready for Oracle Exadata" seminar over 2 half days in March (March 19-20) and May (May 28-29).  I'll be talking about topics like how to functionally manage your Exadata environment (backups, monitoring, patching, etc).  The sessions will be entirely online, and will have plenty of time for Q&A to ask any nagging questions.

Finally, I would like to mention that the call for papers is open for Enkitec's Engineered Systems conference, E4.  Once again, it will be held at the Four Seasons Las Colinas in August.  I know it's not the best time to come to the Dallas area, but they have great air conditioning, and it's nearly impossible to get this kind of access to some of the best Exadata professionals in the world.  The list of speakers from last year was incredible, and I'm sure that this year will be just as good.  Also, the attendees were great to talk to, since many of them were there to share their experiences running Exadata.  The call for papers is open, so if you have something you'd like to share, please submit your abstract!  We'll be covering more than just Exadata, with plans for several talks about Big Data, and possibly even an Exalogic or Exalytics session or two.

]]> http://blog.oracle-ninja.com/2013/02/2013-presentation-schedule/feed/ 0 Oracle Announces Exadata X3-2 and X3-8 http://blog.oracle-ninja.com/2012/10/oracle-announces-exadata-x3-2-and-x3-8/?utm_source=rss&utm_medium=rss&utm_campaign=oracle-announces-exadata-x3-2-and-x3-8 http://blog.oracle-ninja.com/2012/10/oracle-announces-exadata-x3-2-and-x3-8/#comments Mon, 01 Oct 2012 13:23:11 +0000 Andy Colvin http://blog.oracle-ninja.com/?p=570 Well, it's finally public, so we're able to openly talk about the new Exadata X3 systems.  Looking back on my pre-openworld predictions, I was pretty close on a few things.  I was correct on the database servers, which will have Xeon E5-2690 CPUs (8 core, 2.9GHz) with 128GB RAM upgradeable to 256GB.  It looks like we won't get active/active Infiniband for a while, since the cards in there are staying the same.  On the X3-8, the compute nodes are staying the same, for reasons detailed by Kevin Closson a few weeks ago.  I also previously blogged about the X3-2 eighth rack.  I think this will become one of the more popular options for customers, based on the quarter racks that we're seeing purchased.  I'm definitely interested to get my hands on one and see how half of the components have been disabled.  It's very cool that Oracle was able to still give the redundancy of a true Exadata in a smaller footprint.

One of the bigger improvements on the X3 series comes down at the storage level.  I was a little bit off on the CPUS, which will be E5-2630L (6 core, 2.0GHz) with an upgrade from 24GB to 64GB of RAM.  The biggest differences on the storage servers will come via the F40 flash cards, which increase storage 4x (400GB per card), meaning that you'll get 1.6TB of flash per cell.  Also, the version of the Exadata storage server software shipping with the X3 systems will be 11.2.3.2.0, which contains the famous "flash for all writes" cache.  Disk drives will stay the same (600GB or 3TB).

The new storage server software (11.2.3.2.0) should be released to the public some time this week, and it will include the flash write cache for previous systems.  I'm very interested to see what the performance of this feature will look like on the older X2 and V2 systems, where the flash cards are a little bit slower at writes than the new F40 cards.  It is worth noting that the write cache feature will be something that users can enable or disable, so if the performance is not what's expected, it can be disabled.  Rest assured that once the patch is released, it'll find its way onto one of Enkitec's Exadata shortly thereafter.

Also, this new storage server software release will introduce Oracle's Unbreakable Enterprise Kernel to the 2-socket Exadata crowd.  The UEK has been available for the X2-8 systems since their release, but Oracle had yet to run it on X2 systems.  This will change with the release of 11.2.3.2.0.  It is worth noting that it is still possible to go back to the RedHat compatible kernel if there is adverse performance on the UEK.

That's it for now, and as new things come up during the week, I'll try to post on here.

]]> http://blog.oracle-ninja.com/2012/10/oracle-announces-exadata-x3-2-and-x3-8/feed/ 0 Exadata X3-2 1/8th Rack http://blog.oracle-ninja.com/2012/09/exadata-x3-2-18th-rack/?utm_source=rss&utm_medium=rss&utm_campaign=exadata-x3-2-18th-rack http://blog.oracle-ninja.com/2012/09/exadata-x3-2-18th-rack/#comments Fri, 07 Sep 2012 20:16:31 +0000 Andy Colvin http://blog.oracle-ninja.com/?p=553 There have been a couple of posts we've seen lately about expectations of an Exadata X3-2 and X3-8 release at Oracle Open World 2012.  I mentioned in my previous post about the possible release of an X3-2 1/8th rack configuration.  I had guessed that this would be similar to the old V2 basic system that would include one compute node, one storage server, and one infiniband switch - all placed in your own rack.  It sounds like I was a little bit off from this original idea.

Oracle has stopped taking orders on X2-2 and X2-8 hardware, and we have had a handful of our customers let us know about emails that they have received from Oracle reps announcing an Exadata X3-2 1/8th rack for sale.  This configuration will work as "capacity on demand" (insert salesy buzz words).  The plan for the Exadata X3-2 1/8th rack is to contain all of the hardware that exists within a 1/4 rack configuration (2 compute nodes, 3 storage servers, 2 infiniband switches), but to disable half of the CPU cores, half of the flash cards, and half of the hard disks via software controls.

Here's what I would expect this to look like:

  • Compute Nodes
    • 8 CPU cores (16 threads)
    • 128GB RAM
  • Storage Servers
    • 6 or 8 CPU cores (12 or 16 threads)
    • 2 PCIe flash cards
    • 6 X 600GB SAS or 3TB SAS hard disks
  • 2 Infiniband Switches

This would leave you with either 10Tb or 54TB of raw disk space depending on whether high performance or high capacity drives were chosen.  The CPU cores and other hardware components would be disabled using software, probably similar to how unlicensed CPU cores in an ODA are disabled.  This would mean that the 1/8th rack configuration would still contain RAC (including RAC licenses), multiple storage servers (only half of the Exadata storage server licenses), and lots of flash cache.  The process from upgrading a 1/8 rack to a 1/4 rack system would simply be a matter of enabling the extra hardware, most likely through a license key.  Based on the increase in CPU/memory/flash that I'm expecting to see from the X2 --> X3 release, I would expect to see quite a few customers looking at Exadata as an option for many hardware refresh upgrades.  It will be really nice to actually test the improvements from the flash write cache that should be announced at Open World as well.

]]> http://blog.oracle-ninja.com/2012/09/exadata-x3-2-18th-rack/feed/ 4 Pre OpenWorld Predictions (Exadata X3-2?) http://blog.oracle-ninja.com/2012/08/pre-open-world-predictions-exadata-x3-2/?utm_source=rss&utm_medium=rss&utm_campaign=pre-open-world-predictions-exadata-x3-2 http://blog.oracle-ninja.com/2012/08/pre-open-world-predictions-exadata-x3-2/#comments Fri, 31 Aug 2012 20:13:36 +0000 Andy Colvin http://blog.oracle-ninja.com/?p=527 With only a month away from Larry Ellison's keynote at Oracle OpenWorld 2012, I thought that I would make a couple of wild guesses about new products that may or may not get announced this year.  I'll lump them into a few educated guesses and wild conjecture.  Insert standard blogging disclaimer (please read this part, Oracle lawyers):

Everything contained in this blog post is pulled from publicly available information and conclusions drawn from products that are currently available outside of Exadata.  None of this information comes from within Oracle - not that Oracle would be willing to give me any information otherwise.

I'll open with something that most people are actually expecting from Oracle - the introduction of the Oracle Exadata Database Machine X3-2 and most likely the X3-8.  Since the X2-2 is essentially 2 years old, I would expect Oracle to announce a new model.  To lend credence to this idea, I recently saw the following on the product search list when creating a new search box for knowledge articles in My Oracle Support:

Seeing an eighth rack Exadata could be interesting - possibly along the lines of the seldom-seen Exadata V2 basic system (One compute node, one storage server, one Infiniband switch).  This would be a big sell for customers that want to give Exadata a try, or very small dev/test systems.  Keep in mind that there isn't much redundancy in these systems, and the performance isn't quite as great as you'd see on a quarter rack (less spindles, less flash cache).  That basic system deserves a post of its own one of these days.  Read on for more OOW12 prognostications.

"Educated" Guesses

Any discussion of the Exadata X3-2 starts with hardware.  The Westmere CPUs that are in the X2-2 (Intel Xeon X5670 and X5675) are getting long in the tooth.  Intel's new Sandy Bridge E5-2600 CPUs are on their way in the X4170M3 and X4270M3 servers.  In fact, Oracle has renamed them the Sun Fire X3-2 (formerly X4170M3) and the Sun Fire X302L (formerly X4270M3).  Let's look at the specs on the servers that would constitute an X3-2 Exadata:

X3-2 Server (compute node) -  datasheet

Specifications:

  • 2 Socket Xeon E5-2600
  • 512GB RAM maximum
  • 4x 10 gigabit ethernet ports onboard
  • Four PCIe 3.0 slots (1 x16, 3 x8 slots)

My biggest questions are (1) how many CPU cores will Oracle place in these servers on Exadata, and (2) how much memory will be included?  I wouldn't be surprised to see Oracle include E5-2690 CPUs in here (2s16c32t@2.9GHz) with 128GB of RAM in a default configuration, increasing to 256GB or 512GB in a memory expansion pack.  Fortunately, we won't see the same issues in with the Sandy Bridge CPUs that we have on the current X2-2 systems (the memory frequency drops to 800MHz after the memory expansion pack is installed).  There was also a discussion on twitter last week between Alex Fatkulin and Greg Rahn about whether Oracle will enable active-active Infiniband connections (on the X2 and V2 systems, the PCI bus couldn't handle it).  In theory, compute nodes would be able to ingest the full speed of an active-active Infiniband link.  I've only heard of one real (non-laboratory) environment that actually pushed the limit of what a compute node can ingest, though - for some info on that, you'll have to talk to Karl Arao.

X3-2L Server (storage server) - datasheet

Specifications:

  • 2 Socket Xeon E5-2600
  • 256GB RAM maximum
  • 4x 10 gigabit ethernet ports onboard
  • Six PCIe 3.0 slots (1 x16, 5 x8 slots)

We'll have to wait and see what gets introduced on these systems, but I'm going to play it safe here.  Since the F20 flash cards in the X2 systems are approximately 3 years old - they're the same cards introduced in the Exadata V2back in 2009 - I would expect the flash to get an update.  This would most likely come in the way of the newer F40 flash adapter, which provides more capacity as well as faster I/O on both reads and writes.  I would expect the storage servers in the Exadata X3 to contain something along the lines of an E5-2660 (2s16c32t@2.2GHz) with 24GB or 48GB of memory.

It would also be nice to see Oracle utilize the 2 rear-access drive slots in the X3-2L for the operating system, removing the need for the DBFS_DG diskgroup, but we'll just have to wait and see on that one.  While I would really like to see it, I don't know how high it is on the list of things to do within the Exadata Storage Server operating system.

Mildly crazy ideas

I have a couple of other ideas about what Oracle may do with Exadata, but I'm not sure how far from the original configuration they'll go.  One addition that would be cool would be the possibility of using the X3-2L servers with 24 disk slots to create either an all flash (24 x 300GB SSD) would make the high performance version of Exadata really scream compared to the current 12 x 600GB hard disk option.  Even creating a model with 20 x 900GB 10,000RPM hard disks and 4 x 300GB SSD would be cool to see, although storage server licensing costs would go through the roof if they doubled the number of drives.

One more thing that I would expect to be announced during OOW that's not really X3 specific is the write-back cache  for flash writes.  I blogged about this previously, along with Andrew Mendelsohn's slide about up and coming items for Exadata.  Virtualization will be interesting to say the least.  I'm not entirely sure how I would feel about running Oracle VM on my Exadata, but we'll have to wait and see if that gets announced.

After the news that Oracle Database 12c will be released later this year, I fully expect a big announcement on that at OpenWorld.  I'd expect to hear some big features around "cloud" type things - replication, security, new enhancements to RAC, etc...but that would be a safe guess for anybody.  We'll just have to wait and see what gets announced in a month!

update 9/4/12

One more thing that I didn't mention previously that I would expect from Oracle is that all of the Exadata components (database and storage servers) will utilize the Oracle Unbreakable Enterprise Kernel with the next release.  It's been deployed on the latest versions of the Oracle Database Appliance (which will probably see a refresh as well), showing exciting performance gains.  It would be surprising to see Oracle continue down the road with the RedHat compatible kernel on their flagship product.

It would also be a safe bet that we'll see an 11.2.0.4 patchset release.  Each of the last 2 years we have seen a patchset release around the time of OpenWorld.  I would really like to see that include ACFS on Exadata instead of DBFS, but that's probably just a pipe dream for now.

]]> http://blog.oracle-ninja.com/2012/08/pre-open-world-predictions-exadata-x3-2/feed/ 9 Where in the World is Andy Colvin? http://blog.oracle-ninja.com/2012/08/where-in-the-world-is-andy-colvin/?utm_source=rss&utm_medium=rss&utm_campaign=where-in-the-world-is-andy-colvin http://blog.oracle-ninja.com/2012/08/where-in-the-world-is-andy-colvin/#comments Thu, 30 Aug 2012 21:20:05 +0000 Andy Colvin http://blog.oracle-ninja.com/?p=531 I've got a handful of presentations coming up in the latter part of the year, so I thought I'd add a quick post with where all I'm going to be.  Seems like I'm all over the map and I couldn't stop thinking of a game that I played way back when I was in elementary school.  Well, over the next few months, I'll be in a few places talking about Exadata, OEM, and other Oracle topics.  Here's a list of where I'll be, and what I'll be talking about.

Oracle Open World (San Francisco, CA - September 30 - October 4)

UK Oracle Users Group Conference (Birmingham, UK - December 3 - December 5)

  • Patching Exadata Demystified (December 4, 11:15AM)
  • Exadata Zero Downtime Migration (December 5, 11:15AM)

Of course, I'll be at Enkitec's booth (Moscone South, #421) at Open World as well, so feel free to stop by and say hi.  We may just have some goodies to give out as well.  I'm also teaching Enkitec's Exadata Administration course for a few sessions over the next 3 months.

]]> http://blog.oracle-ninja.com/2012/08/where-in-the-world-is-andy-colvin/feed/ 0 Exadata Flash Write-back – Sooner Than We Think? http://blog.oracle-ninja.com/2012/08/exadata-flash-write-back-sooner-than-we-think/?utm_source=rss&utm_medium=rss&utm_campaign=exadata-flash-write-back-sooner-than-we-think http://blog.oracle-ninja.com/2012/08/exadata-flash-write-back-sooner-than-we-think/#comments Sun, 26 Aug 2012 01:58:01 +0000 Andy Colvin http://blog.oracle-ninja.com/?p=518 If you missed Andy Mendelsohn's keynote at E4 last week, you may not have heard the hubbub that surrounded one of his last slides (tweeted by Frits Hooogland here).  The mention of the write-back enticed Kevin Closson to talk about the potential ramifications of such a feature.  There's a lot of information on that slide to digest (what's a pluggable database?  virtualization of database servers?), but I'm going to focus on the flash-based write-back cache.  Note that this is not mentioning the "Exadata Smart Flash Log" featured introduced last year with the 11.2.2.4.0 cell patch, discussed by Guy Harrison recently.  That feature sends writes to both flash and disk at the same time.  In my experience, the disk wins on > 90% of those writes.

This is something larger than just sending writes to flash...an issue that Oracle has likely been working on for a few years. Kevin had mentioned in his post that he expected it to be a feature in the 12.2 release, possibly 12.1 of the database. Because Mendelsohn mentioned that there was a 12-month timeframe for these items, I expected it would occur with the release of the new version of the Oracle database, 12c. I've been doing some poking around in the latest Exadata patch notes and saw a couple of interesting bugs around a write-back cache on Exadata using flash. Bug 14143451 "Enhancement for ASM write-back flash cache resilvering support" and bug 14132953 "Enhanacement to add Write-back flash cache resilvering support" have both been added to the August 2012 bundle patch for 11.2.0.3 (MOS note #1393410.1). If you look at these bugs, you will see that they are currently listed as fixed in 11.2.0.4. The fact that the enhancement has been added to 11.2.0.3 interests me. It looks similar to the introduction of the Exadata smart flash log feature, introduced in the 11.2.2.4.0 Exadata storage server version, released October 2011. If you look through the Exadata bundle patches for 11.2.0.2, you'll see that it was introduced into the database code in bundle patch 9 (MOS note #1314319.1). That bundle patch was released in July 2011. Sound familiar? I wouldn't put it past Oracle to include the write-back cache through a new version of the storage server software.

This sounds like the kind of feature that Larry Ellison would be very happy to announce at Open World in October. We'll just have to wait and see what gets announced. I'll have another post in the next week or so guessing about what may get announced a month from now in San Francisco.

]]> http://blog.oracle-ninja.com/2012/08/exadata-flash-write-back-sooner-than-we-think/feed/ 2 Upgrading the Oracle Database Appliance to Version 2.3.0.0.0 http://blog.oracle-ninja.com/2012/08/upgrading-the-oracle-database-appliance-t-version-2-3-0-0-0/?utm_source=rss&utm_medium=rss&utm_campaign=upgrading-the-oracle-database-appliance-t-version-2-3-0-0-0 http://blog.oracle-ninja.com/2012/08/upgrading-the-oracle-database-appliance-t-version-2-3-0-0-0/#comments Mon, 06 Aug 2012 04:50:53 +0000 Andy Colvin http://blog.oracle-ninja.com/?p=507 As you may have guessed, applying patches on the Oracle Database Appliance can be a little bit different from your standard Oracle environment.  Oracle releases a software version that covers all aspects of the ODA - firmware, operating system, and Oracle software stack (grid infrastructure, rdbms).  Versions are numbered like this (image courtesy MOS note #1397680.1:

 

The ODA was initially released with version 2.1.0.0.0, and has seen several releases over the last year:

Patch Features
2.1.0.3.0 CPU bugfix, 11.2.0.2.5 GI PSU5
2.1.0.3.1 OAK software updates
2.2.0.0.0 11.2.0.3 GI/RDBMS, OEL 5.8, UEK kernel
2.3.0.0.0 July 2012 PSU for 11.2.0.2/11.2.0.3, firmware upgrades, multiple database home support

In this post, we'll discuss upgrading an ODA running RAC or RAC one node to version 2.3.0.0.0.  Note that before going to 2.3, users must upgrade to 2.2 first.  This is because the 2.3 patch upgrade does not include some of the files used for the OEL 5.8 upgrade, among other things.

As with many functions on the ODA, patching is performed using the oakcli utility.  OAK stands for "Oracle Appliance Kit" and it is at the heart of the manageability enhancements provided by the ODA.  Check MOS note 888888.1 for links to all patches for the ODA.  Download patch 13982331 and place the zip file on each of the nodes in the ODA.  Using oakcli, unpack the patch file on each node:

?View Code NONE
[root@oda1 ~]# oakcli unpack -package p13982331_23000_Linux-x86-64.zip
Unpacking takes a while,  pls wait....
Successfully unpacked the files to repository.
?View Code NONE
[root@oda2 ~]# oakcli unpack -package p13982331_23000_Linux-x86-64.zip
Unpacking takes a while,  pls wait....
Successfully unpacked the files to repository.

Next, we will patch the infrastructure. This includes patches for the operating system, firmware, ASR, and the Oracle Appliance Kit. Run this only from the first node. The entire cluster will be shut down because the shared hard disks will have their firmware patched to a new version. The infrastructure is patched using "oakcli update -patch 2.3.0.0.0 --infra"

?View Code NONE
[root@oda1 ~]# oakcli update -patch 2.3.0.0.0 --infra
INFO: DB, ASM, Clusterware may be stopped during the patch if required
INFO: Both nodes may get rebooted automatically during the patch if required
Do you want to continue: [Y/N]?: y
INFO: User has confirmed the reboot
INFO: Patch bundle must be unpacked on the second node also before applying this patch
Did you unpack the patch bundle on the second node?: [Y/N]?: y
 
Please enter the 'root' user password: 
Please re-enter the 'root' user password: 
INFO: Setting up the SSH
............done
INFO: Running pre-install scripts
INFO: 2012-07-26 21:56:17: Running pre patch script for 2.3.0.0.0
INFO: 2012-07-26 21:56:17: Completed pre patch script for 2.3.0.0.0
INFO   : Running as root: /usr/bin/ssh -l root 192.168.16.25 /opt/oracle/oak/pkgrepos/System/2.3.0.0.0/bin/prepatch -v 2.3.0.0.0
 
INFO: 2012-07-26 21:56:20: ------------------Patching HMP-------------------------
SUCCESS: 2012-07-26 21:56:50: Successfully upgraded the HMP
 
INFO: 2012-07-26 21:56:51: ----------------------Patching OAK---------------------
SUCCESS: 2012-07-26 21:57:19: Succesfully upgraded OAK 
 
INFO: 2012-07-26 21:57:20: ----------------------Patching ASR---------------------
INFO: 2012-07-26 21:57:20: ASR is already upgraded or running with latest version 
 
INFO: 2012-07-26 21:57:20: ----------------------Patching IPMI---------------------
INFO: 2012-07-26 21:57:20: IPMI is already upgraded or running with latest version 
 
INFO: 2012-07-26 21:57:28: ----------------Patching the Storage-------------------
INFO: 2012-07-26 21:57:28: ....................Patching SSDs...............
INFO: 2012-07-26 21:57:28: Disk : d20  is already running with : ZeusIOPs G3 E125
INFO: 2012-07-26 21:57:28: Disk : d21  is already running with : ZeusIOPs G3 E125
INFO: 2012-07-26 21:57:28: Disk : d22  is already running with : ZeusIOPs G3 E125
INFO: 2012-07-26 21:57:28: Disk : d23  is already running with : ZeusIOPs G3 E125
INFO: 2012-07-26 21:57:29: ....................Patching shared HDDs...............
INFO: 2012-07-26 21:57:29: Updating the  Disk : d0 with the firmware : ST360057SSUN600G 0B25
INFO: 2012-07-26 21:57:31: Clusterware is running on one or more nodes of the cluster
INFO: 2012-07-26 21:57:31: Attempting to stop clusterware and its resources across the cluster
SUCCESS: 2012-07-26 21:59:38: Successfully stopped the clusterware
 
SUCCESS: 2012-07-26 21:59:59: Successfully updated the firmware on  Disk : d0 to ST360057SSUN600G 0B25
INFO: 2012-07-26 21:59:59: Updating the  Disk : d1 with the firmware : ST360057SSUN600G 0B25
SUCCESS: 2012-07-26 22:00:23: Successfully updated the firmware on  Disk : d1 to ST360057SSUN600G 0B25
INFO: 2012-07-26 22:00:23: Updating the  Disk : d2 with the firmware : ST360057SSUN600G 0B25
SUCCESS: 2012-07-26 22:00:44: Successfully updated the firmware on  Disk : d2 to ST360057SSUN600G 0B25
INFO: 2012-07-26 22:00:44: Updating the  Disk : d3 with the firmware : ST360057SSUN600G 0B25
SUCCESS: 2012-07-26 22:01:04: Successfully updated the firmware on  Disk : d3 to ST360057SSUN600G 0B25
INFO: 2012-07-26 22:01:05: Updating the  Disk : d4 with the firmware : ST360057SSUN600G 0B25
SUCCESS: 2012-07-26 22:01:26: Successfully updated the firmware on  Disk : d4 to ST360057SSUN600G 0B25
INFO: 2012-07-26 22:01:26: Updating the  Disk : d5 with the firmware : ST360057SSUN600G 0B25
SUCCESS: 2012-07-26 22:01:46: Successfully updated the firmware on  Disk : d5 to ST360057SSUN600G 0B25
INFO: 2012-07-26 22:01:47: Updating the  Disk : d6 with the firmware : ST360057SSUN600G 0B25
SUCCESS: 2012-07-26 22:02:09: Successfully updated the firmware on  Disk : d6 to ST360057SSUN600G 0B25
INFO: 2012-07-26 22:02:09: Updating the  Disk : d7 with the firmware : ST360057SSUN600G 0B25
SUCCESS: 2012-07-26 22:02:29: Successfully updated the firmware on  Disk : d7 to ST360057SSUN600G 0B25
INFO: 2012-07-26 22:02:29: Updating the  Disk : d8 with the firmware : ST360057SSUN600G 0B25
SUCCESS: 2012-07-26 22:02:48: Successfully updated the firmware on  Disk : d8 to ST360057SSUN600G 0B25
INFO: 2012-07-26 22:02:48: Updating the  Disk : d9 with the firmware : ST360057SSUN600G 0B25
SUCCESS: 2012-07-26 22:06:07: Successfully updated the firmware on  Disk : d9 to ST360057SSUN600G 0B25
INFO: 2012-07-26 22:06:07: Updating the  Disk : d10 with the firmware : ST360057SSUN600G 0B25
SUCCESS: 2012-07-26 22:06:27: Successfully updated the firmware on  Disk : d10 to ST360057SSUN600G 0B25
INFO: 2012-07-26 22:06:27: Updating the  Disk : d11 with the firmware : ST360057SSUN600G 0B25
SUCCESS: 2012-07-26 22:06:46: Successfully updated the firmware on  Disk : d11 to ST360057SSUN600G 0B25
INFO: 2012-07-26 22:06:46: Updating the  Disk : d12 with the firmware : ST360057SSUN600G 0B25
SUCCESS: 2012-07-26 22:07:07: Successfully updated the firmware on  Disk : d12 to ST360057SSUN600G 0B25
INFO: 2012-07-26 22:07:07: Updating the  Disk : d13 with the firmware : ST360057SSUN600G 0B25
SUCCESS: 2012-07-26 22:07:26: Successfully updated the firmware on  Disk : d13 to ST360057SSUN600G 0B25
INFO: 2012-07-26 22:07:26: Updating the  Disk : d14 with the firmware : ST360057SSUN600G 0B25
SUCCESS: 2012-07-26 22:07:46: Successfully updated the firmware on  Disk : d14 to ST360057SSUN600G 0B25
INFO: 2012-07-26 22:07:46: Updating the  Disk : d15 with the firmware : ST360057SSUN600G 0B25
SUCCESS: 2012-07-26 22:08:07: Successfully updated the firmware on  Disk : d15 to ST360057SSUN600G 0B25
INFO: 2012-07-26 22:08:07: Updating the  Disk : d16 with the firmware : ST360057SSUN600G 0B25
SUCCESS: 2012-07-26 22:08:27: Successfully updated the firmware on  Disk : d16 to ST360057SSUN600G 0B25
INFO: 2012-07-26 22:08:27: Updating the  Disk : d17 with the firmware : ST360057SSUN600G 0B25
SUCCESS: 2012-07-26 22:08:47: Successfully updated the firmware on  Disk : d17 to ST360057SSUN600G 0B25
INFO: 2012-07-26 22:08:47: Updating the  Disk : d18 with the firmware : ST360057SSUN600G 0B25
SUCCESS: 2012-07-26 22:09:07: Successfully updated the firmware on  Disk : d18 to ST360057SSUN600G 0B25
INFO: 2012-07-26 22:09:08: Updating the  Disk : d19 with the firmware : ST360057SSUN600G 0B25
SUCCESS: 2012-07-26 22:09:27: Successfully updated the firmware on  Disk : d19 to ST360057SSUN600G 0B25
INFO: 2012-07-26 22:09:27: ....................Patching local HDDs...............
INFO: 2012-07-26 22:09:27: Disk : c0d0  is already running with : ST95001N SA03
INFO: 2012-07-26 22:09:27: Disk : c0d1  is already running with : ST95001N SA03
INFO: 2012-07-26 22:09:27: ....................Patching Expanders...............
INFO: 2012-07-26 22:09:27: Expander : x0  is already running with : T4 Storage 0342
INFO: 2012-07-26 22:09:27: Expander : x1  is already running with : T4 Storage 0342
INFO: 2012-07-26 22:09:27: ....................Patching Controllers...............
INFO: 2012-07-26 22:09:27: No-update for the Controller: c0 
INFO: 2012-07-26 22:09:27: Controller : c1  is already running with : 0x0072 05.00.29.00
INFO: 2012-07-26 22:09:27: Controller : c2  is already running with : 0x0072 05.00.29.00
INFO: 2012-07-26 22:09:27: ------------Finished the storage Patching------------
 
INFO: 2012-07-26 22:09:29: -----------------Patching Ilom & Bios-----------------
INFO: 2012-07-26 22:09:29: Getting the ILOM Ip address
INFO: 2012-07-26 22:09:29: Updating the Ilom using LAN+ protocol
INFO: 2012-07-26 22:09:30: Updating the ILOM. It takes a while
INFO: 2012-07-26 22:14:25: Verifying the updated Ilom Version, it may take a while if ServiceProcessor is booting
INFO: 2012-07-26 22:14:27: Waiting for the service processor to be up
SUCCESS: 2012-07-26 22:18:16: Successfully updated the ILOM with the firmware 3.0.16.22 r73911
 
INFO: Patching the infrastructure on node: 192.168.16.25 , it may take upto 30 minutes. Please wait
...
INFO   : Running as root: /usr/bin/ssh -l root 192.168.16.25 /tmp/prepare_summary
INFO   : Running as root: /usr/bin/ssh -l root 192.168.16.25 /opt/oracle/oak/pkgrepos/System/2.3.0.0.0/bin/oakpatch -p infrastructure -f 2.3.0.0.0
 
............done
 
INFO: Infrastructure patching summary on node: 192.168.16.24
SUCCESS: 2012-07-26 22:29:19:  Successfully upgraded the HMP
SUCCESS: 2012-07-26 22:29:19:  Succesfully updated the OAK
INFO: 2012-07-26 22:29:19:  ASR is already upgraded
INFO: 2012-07-26 22:29:19:  IPMI is already upgraded
INFO: 2012-07-26 22:29:19:  Storage patching summary
SUCCESS: 2012-07-26 22:29:19:  No failures during storage upgrade
SUCCESS: 2012-07-26 22:29:19:  Successfully updated the ILOM & Bios
 
INFO: Infrastructure patching summary on node: 192.168.16.25
SUCCESS: 2012-07-26 22:29:19:  Successfully upgraded the HMP
SUCCESS: 2012-07-26 22:29:19:  Succesfully updated the OAK
INFO: 2012-07-26 22:29:19:  IPMI is already upgraded
INFO: 2012-07-26 22:29:19:  Storage patching summary
SUCCESS: 2012-07-26 22:29:19:  No failures during storage upgrade
SUCCESS: 2012-07-26 22:29:19:  Successfully updated the ILOM & Bios
 
INFO   : Running as root: /usr/bin/ssh -l root 192.168.16.25 /opt/oracle/oak/pkgrepos/System/2.3.0.0.0/bin/postpatch -v 2.3.0.0.0
INFO   : Running as root: /usr/bin/ssh -l root 192.168.16.24 /opt/oracle/oak/pkgrepos/System/2.3.0.0.0/bin/postpatch -v 2.3.0.0.0
INFO: Some of the patched components require node reboot. Rebooting the nodes
INFO   : Running as root: /usr/bin/ssh -l root 192.168.16.24 /tmp/pending_actions
INFO   : Running as root: /usr/bin/ssh -l root 192.168.16.25 /tmp/pending_actions
 
Broadcast message from root (Thu Jul 26 22:32:49 2012):
 
The system is going down for system halt NOW!

Patching the infrastructure took a little more than 30 minutes for both nodes. After the infrastructure has been patched, both nodes will reboot, and the cluster will come up. Before moving on, check to make sure that the serial number is visible on both nodes, and that they show the same number:

?View Code NONE
[root@oda1 ~]# dmidecode -t 1|grep 'Serial Number'
	Serial Number: **********
[root@oda2 ~]# dmidecode -t 1|grep 'Serial Number'
	Serial Number: **********

We are now ready to upgrade the Grid Infrastructure on the cluster. This portion is performed in a rolling fashion, patching the first node, followed by the second. The GI home is patched using "oakcli update -patch 2.3.0.0.0 --gi"

?View Code NONE
[root@oda1 ~]# oakcli update -patch 2.3.0.0.0 --gi
 
Please enter the 'root' user password: 
Please re-enter the 'root' user password: 
INFO: Setting up the SSH
............done
 
Please enter the 'grid' user password: 
Please re-enter the 'grid' user password: 
...
...
 
..........done
...
SUCCESS: All nodes in /opt/oracle/oak/temp_clunodes.txt are pingable and alive.
INFO: 2012-07-26 22:52:09: Setting up the ssh for grid user
..........done
...
SUCCESS: All nodes in /opt/oracle/oak/temp_clunodes.txt are pingable and alive.
INFO: 2012-07-26 22:52:29: Patching the GI home on node oda1
INFO: 2012-07-26 22:52:29: Updating the opatch
INFO: 2012-07-26 22:53:10: Performing the conflict checks
SUCCESS: 2012-07-26 22:53:25: Conflict checks passed for all the homes
INFO: 2012-07-26 22:53:25: Checking if the patch is already applied on any of the homes
INFO: 2012-07-26 22:53:31: No home is already up-to-date
SUCCESS: 2012-07-26 22:54:23: Successfully stopped the dbconsoles
INFO: 2012-07-26 22:54:28: Applying patch on the homes: /u01/app/11.2.0.3/grid
INFO: 2012-07-26 22:54:28: It may take upto 15 mins
SUCCESS: 2012-07-26 23:05:56: Successfully applied the patch on home: /u01/app/11.2.0.3/grid
SUCCESS: 2012-07-26 23:06:18: Successfully started the dbconsoles
INFO: 2012-07-26 23:06:18: Patching the GI home on node oda2
 
...
 
............done
 
INFO: GI patching summary on node: 192.168.16.24
SUCCESS: 2012-07-26 23:18:41:  Successfully applied the patch on home /u01/app/11.2.0.3/grid
 
INFO: GI patching summary on node: 192.168.16.25
SUCCESS: 2012-07-26 23:18:41:  Successfully applied the patch on home /u01/app/11.2.0.3/grid

This process took around 25-30 minutes to complete on our system, depending on the number of databases being started/stopped. Finally, we are ready to patch the database homes to the latest patch release. If your databases are running 11.2.0.3, they will be updated to 11.2.0.3.3, and 11.2.0.2 homes will be updated to 11.2.0.2.7. Note that the database patches are in-place, so your home will not change with this patch. Before patching the database homes, check that your databases are registered with the Oracle appliance manager:

?View Code NONE
[root@oda2 ~]# oakcli show databases
Database Name    Database Type   Database HomeName    Database HomeLocation                               Database Version                   
----------------  -----------     ----------------     ---------------------------------------             ---------------------              
SCRATCH          SINGLE          DEMO                 /u01/app/oracle/product/11.2.0/dbhome_1            11.2.0.2.5(13343424,13343447)      
oakdb            RAC             dbhome11203          /u01/app/oracle/product/11.2.0.3/dbhome_1          11.2.0.3.2(13696216,13696251)      
SLOB             SINGLE          dbhome11203          /u01/app/oracle/product/11.2.0.3/dbhome_1          11.2.0.3.2(13696216,13696251)      
DEMO             RAC             DEMO                 /u01/app/oracle/product/11.2.0/dbhome_1            11.2.0.2.5(13343424,13343447)      
CDH3             RAC             dbhome11203          /u01/app/oracle/product/11.2.0.3/dbhome_1          11.2.0.3.2(13696216,13696251)

You can see that we have both 11.2.0.3 and 11.2.0.2 databases installed on this system. Issue the "oakcli update -patch 2.3.0.0.0 --db" command to update all of your database homes. Once again, this portion of the patch is applied in a rolling fashion.

?View Code NONE
[root@oda1 ~]# oakcli update -patch 2.3.0.0.0 --database
 
Please enter the 'root' user password: 
Please re-enter the 'root' user password: 
INFO: Setting up the SSH
............done
 
Please enter the 'oracle' user password: 
Please re-enter the 'oracle' user password: 
...
...
 
..........done
...
SUCCESS: All nodes in /opt/oracle/oak/temp_clunodes.txt are pingable and alive.
INFO: 2012-07-26 23:26:42: Getting the possible database homes for patching
...
INFO: 2012-07-26 23:26:53: Patching 11.2.0.2 Database homes on node oda1
 
Found the following 11.2.0.2 homes possible for patching:
 
HOME_NAME                      HOME_LOCATION                                          
---------                      -------------                                          
DEMO                           /u01/app/oracle/product/11.2.0/dbhome_1                
 
[Please note that few of the above database homes may be already up-to-date. They will be automatically ignored]
 
Would you like to patch all the above homes: Y | N ? :y
INFO: 2012-07-26 23:27:06: Setting up ssh for the user oracle
..........done
...
SUCCESS: All nodes in /opt/oracle/oak/temp_clunodes.txt are pingable and alive.
INFO: 2012-07-26 23:27:26: Updating the opatch
INFO: 2012-07-26 23:27:55: Performing the conflict checks
SUCCESS: 2012-07-26 23:28:22: Conflict checks passed for all the homes
INFO: 2012-07-26 23:28:22: Checking if the patch is already applied on any of the homes
INFO: 2012-07-26 23:28:28: No home is already up-to-date
SUCCESS: 2012-07-26 23:28:59: Successfully stopped the dbconsoles
INFO: 2012-07-26 23:29:04: Applying patch on the homes: /u01/app/oracle/product/11.2.0/dbhome_1
INFO: 2012-07-26 23:29:04: It may take upto 15 mins
 
SUCCESS: 2012-07-26 23:42:27: Successfully applied the patch on home: /u01/app/oracle/product/11.2.0/dbhome_1
SUCCESS: 2012-07-26 23:42:46: Successfully started the dbconsoles
INFO: 2012-07-26 23:42:49: Patching 11.2.0.2 Database homes on node oda2
INFO: 2012-07-26 23:52:51: Running the catbundle.sql
INFO: 2012-07-26 23:52:54: Running catbundle.sql on the database DEMO
INFO: 2012-07-26 23:54:00: Running catbundle.sql on the database SCRATCH
...
INFO: 2012-07-26 23:54:43: Patching 11.2.0.3 Database homes on node oda1
 
Found the following 11.2.0.3 homes possible for patching:
 
HOME_NAME                      HOME_LOCATION                                          
---------                      -------------                                          
dbhome11203                    /u01/app/oracle/product/11.2.0.3/dbhome_1              
 
[Please note that few of the above database homes may be already up-to-date. They will be automatically ignored]
 
Would you like to patch all the above homes: Y | N ? :y
INFO: 2012-07-26 23:54:50: Updating the opatch
INFO: 2012-07-26 23:55:32: Performing the conflict checks
SUCCESS: 2012-07-26 23:55:47: Conflict checks passed for all the homes
INFO: 2012-07-26 23:55:47: Checking if the patch is already applied on any of the homes
INFO: 2012-07-26 23:55:53: No home is already up-to-date
SUCCESS: 2012-07-26 23:56:02: Successfully stopped the dbconsoles
INFO: 2012-07-26 23:56:07: Applying patch on the homes: /u01/app/oracle/product/11.2.0.3/dbhome_1
INFO: 2012-07-26 23:56:07: It may take upto 15 mins
SUCCESS: 2012-07-27 00:01:00: Successfully applied the patch on home: /u01/app/oracle/product/11.2.0.3/dbhome_1
SUCCESS: 2012-07-27 00:01:00: Successfully started the dbconsoles
INFO: 2012-07-27 00:01:04: Patching 11.2.0.3 Database homes on node oda2
INFO: 2012-07-27 00:05:14: Running the catbundle.sql
INFO: 2012-07-27 00:05:17: Running catbundle.sql on the database CDH3
INFO: 2012-07-27 00:05:48: Running catbundle.sql on the database oakdb
INFO: 2012-07-27 00:06:19: Running catbundle.sql on the database SLOB
 
............done
 
INFO: DB patching summary on node: 192.168.16.24
SUCCESS: 2012-07-27 00:07:08:  Successfully applied the patch on home /u01/app/oracle/product/11.2.0/dbhome_1
SUCCESS: 2012-07-27 00:07:08:  Successfully applied the patch on home /u01/app/oracle/product/11.2.0.3/dbhome_1
 
INFO: DB patching summary on node: 192.168.16.25
SUCCESS: 2012-07-27 00:07:08:  Successfully applied the patch on home /u01/app/oracle/product/11.2.0/dbhome_1
SUCCESS: 2012-07-27 00:07:08:  Successfully applied the patch on home /u01/app/oracle/product/11.2.0.3/dbhome_1

For our 5 databases, the patch application process took around 40 minutes. Nothing was required of the administrator, not even running the post-installation scripts. Once everything is done, run "oakcli show version -detail"

?View Code NONE
[root@oda1 ~]# oakcli show version -detail
Reading the metadata. It takes a while...
System Version  Component Name            Installed Version         Supported Version        
--------------  ---------------           ------------------        -----------------        
2.3.0.0.0                                                                                    
                Controller                05.00.29.00               Up-to-date               
                Expander                  0342                      Up-to-date               
                SSD_SHARED                E125                      Up-to-date               
                HDD_LOCAL                 SA03                      Up-to-date               
                HDD_SHARED                0B25                      Up-to-date               
                ILOM                      3.0.16.22 r73911          Up-to-date               
                BIOS                      12010309                  Up-to-date               
                IPMI                      1.8.10.4                  Up-to-date               
                HMP                       2.2.4                     Up-to-date               
                OAK                       2.3.0.0.0                 Up-to-date               
                OEL                       5.8                       Up-to-date               
                GI_HOME                   11.2.0.3.3(13923374,      Up-to-date               
                                          13919095)                                          
                DB_HOME                   11.2.0.3.3(13923374,      Up-to-date               
                                          13919095)

End to end, the process took a little over 2 hours to apply the patch.

]]> http://blog.oracle-ninja.com/2012/08/upgrading-the-oracle-database-appliance-t-version-2-3-0-0-0/feed/ 2 Enkitec Extreme Exadata Expo 2012 http://blog.oracle-ninja.com/2012/06/enkitec-extreme-exadata-expo-2012/?utm_source=rss&utm_medium=rss&utm_campaign=enkitec-extreme-exadata-expo-2012 http://blog.oracle-ninja.com/2012/06/enkitec-extreme-exadata-expo-2012/#comments Sun, 10 Jun 2012 03:00:20 +0000 Andy Colvin http://blog.oracle-ninja.com/?p=502 Enkitec will be hosting the first annual Enkitec Extreme Exadata Expo (E4) this August at the Four Seasons Hotel & Resort in Irving, TX.  While I would generally tell people to stay away from Texas in August, this event will definitely be worth it.  Enkitec has managed to stockpile some of the greatest minds in the world regarding Exadata and the Oracle database in general.  A quick list of some of the speakers:

They've even invited a guy named Andy Colvin to come talk about why you shouldn't be scared to patch your Exadata.  The list continues to grow, as more speakers should be announced soon.  The conference covers 2 days (August 13 - 14), and attendees will have the chance to enroll in one of Enkitec's 2 Exadata courses - Exadata performance optimization, or Exadata Administration (the latter is taught by your's truly).

What's really cool is that if you can't make it to Dallas, Enkitec is offering the chance to attend the conference virtually.  I definitely recommend being onsite, as attendees will have great access to the speakers that you won't get virtually.  Admission to the conference is $850 for onsite, $500 virtual.  I hope to see you there!

]]> http://blog.oracle-ninja.com/2012/06/enkitec-extreme-exadata-expo-2012/feed/ 0 Why I Don’t Like Role Separated Accounts – Part 1 http://blog.oracle-ninja.com/2012/05/why-i-dont-like-role-separated-accounts-part-1/?utm_source=rss&utm_medium=rss&utm_campaign=why-i-dont-like-role-separated-accounts-part-1 http://blog.oracle-ninja.com/2012/05/why-i-dont-like-role-separated-accounts-part-1/#comments Mon, 07 May 2012 17:04:19 +0000 Andy Colvin http://blog.oracle-ninja.com/?p=480 One of the things that I've touched on before that tends to bother me - role separated grid infrastructure installations - gave me another reason to show my dislike a few weeks ago.  While working on a system that was being upgraded from 11.2.0.2 to 11.2.0.3, we ran into a strange issue.  After upgrading from 11.2.0.2 to 11.2.0.3, we could no longer connect to our databases.  When we would attempt to connect remotely, we would get:

?View Code NONE
[acolvin@homer ~]$ sqlplus system@odademo
 
SQL*Plus: Release 11.2.0.3.0 Production on Sun May 20 11:58:23 2012
 
Copyright (c) 1982, 2011, Oracle.  All rights reserved.
 
Enter password:
ERROR:
ORA-12537: TNS:connection closed

We could connect to the database without issue internally. There were no network issues to report, everything appeared to be working fine, except we couldn't get in to the database.  The database we were connecting to (DEMO) was registered with the listener:

?View Code NONE
[grid@patty ~]$ lsnrctl stat LISTENER_SCAN1
 
LSNRCTL for Linux: Version 11.2.0.3.0 - Production on 20-MAY-2012 12:07:55
 
Copyright (c) 1991, 2011, Oracle.  All rights reserved.
 
Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN1)))
STATUS of the LISTENER
------------------------
Alias                     LISTENER_SCAN1
Version                   TNSLSNR for Linux: Version 11.2.0.3.0 - Production
Start Date                25-APR-2012 11:50:38
Uptime                    25 days 0 hr. 17 min. 17 sec
Trace Level               off
Security                  ON: Local OS Authentication
SNMP                      OFF
Listener Parameter File   /u01/app/11.2.0.3/grid/network/admin/listener.ora
Listener Log File         /u01/app/11.2.0.3/grid/log/diag/tnslsnr/patty/listener_scan1/alert/log.xml
Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER_SCAN1)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=****)(PORT=1521)))
Services Summary...
Service "DEMO" has 2 instance(s).
  Instance "DEMO1", status READY, has 2 handler(s) for this service...
  Instance "DEMO2", status READY, has 2 handler(s) for this service...
 
[grid@patty ~]$ lsnrctl stat
 
LSNRCTL for Linux: Version 11.2.0.3.0 - Production on 20-MAY-2012 12:05:51
 
Copyright (c) 1991, 2011, Oracle.  All rights reserved.
 
Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER)))
STATUS of the LISTENER
------------------------
Alias                     LISTENER
Version                   TNSLSNR for Linux: Version 11.2.0.3.0 - Production
Start Date                07-MAY-2012 15:21:43
Uptime                    12 days 20 hr. 44 min. 8 sec
Trace Level               off
Security                  ON: Local OS Authentication
SNMP                      OFF
Listener Parameter File   /u01/app/11.2.0.3/grid/network/admin/listener.ora
Listener Log File         /u01/app/grid/diag/tnslsnr/patty/listener/alert/log.xml
Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=****)(PORT=1521)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=****)(PORT=1521)))
Services Summary...
Service "+ASM" has 1 instance(s).
  Instance "+ASM1", status READY, has 1 handler(s) for this service...
Service "DEMO" has 1 instance(s).
  Instance "DEMO1", status READY, has 2 handler(s) for this service...

After poking around for a little bit, I came across MOS note #1069517.1, "ORA-12537 if Listener (including SCAN Listener) and Database are Owned by Different OS User." Hey, that looks familiar! Looking through the listener logs, we saw this error:

?View Code NONE
20-MAY-2012 12:10:48 * (CONNECT_DATA=(SID=DEMO1)(CID=(PROGRAM=perl@patty)(HOST=patty)(USER=oracle))) * (ADDRESS=(PROTOCOL=tcp)(HOST=192.168.10.219)(PORT=24042)) * establish * DEMO1 * 12518
TNS-12518: TNS:listener could not hand off client connection
 TNS-12547: TNS:lost contact
  TNS-12560: TNS:protocol adapter error
   TNS-00517: Lost contact
    Linux Error: 32: Broken pipe

Turns out this is an issue if the permissions on the oracle binary in the database $ORACLE_HOME/bin directory is missing the setuid bit.

?View Code NONE
[oracle@patty ~]$ ls -al /u01/app/oracle/product/11.2.0/dbhome_1/bin/oracle
-rwxr-x--x 1 oracle asmadmin 229009338 Jan 19 12:59 /u01/app/oracle/product/11.2.0/dbhome_1/bin/oracle

Once we reset the setuid bit, we were back in business:

?View Code NONE
[oracle@patty ~]$
chmod 6751 /u01/app/oracle/product/11.2.0/dbhome_1/bin/oracle
 
[oracle@patty ~]$ ls -al /u01/app/oracle/product/11.2.0/dbhome_1/bin/oracle
-rwsr-x--x 1 oracle asmadmin 229009338 Jan 19 12:59 /u01/app/oracle/product/11.2.0/dbhome_1/bin/oracle

While this isn't something that comes up often, it's still something that wouldn't happen under an environment owned entirely by the Oracle account.

]]> http://blog.oracle-ninja.com/2012/05/why-i-dont-like-role-separated-accounts-part-1/feed/ 3