As you may have guessed, applying patches on the Oracle Database Appliance can be a little bit different from your standard Oracle environment. Oracle releases a software version that covers all aspects of the ODA - firmware, operating system, and Oracle software stack (grid infrastructure, rdbms). Versions are numbered like this (image courtesy MOS note #1397680.1:
The ODA was initially released with version 188.8.131.52.0, and has seen several releases over the last year:
|184.108.40.206.0||CPU bugfix, 220.127.116.11.5 GI PSU5|
|18.104.22.168.1||OAK software updates|
|22.214.171.124.0||126.96.36.199 GI/RDBMS, OEL 5.8, UEK kernel|
|188.8.131.52.0||July 2012 PSU for 184.108.40.206/220.127.116.11, firmware upgrades, multiple database home support|
In this post, we'll discuss upgrading an ODA running RAC or RAC one node to version 18.104.22.168.0. Note that before going to 2.3, users must upgrade to 2.2 first. This is because the 2.3 patch upgrade does not include some of the files used for the OEL 5.8 upgrade, among other things.
Oracle has announced a new patching strategy for Exadata, starting with databases running 22.214.171.124. Oracle will be moving away from the monthly bundle patch philosophy, which was panned by many administrators as coming too often to keep up with, given the tight schedules held around most Exadata systems. Instead, Oracle will be releasing a Quarterly Database Patch for Exadata, or QDPE. The QDPE will most likely be released in conjunction with the standard critical patch updates (CPUs). Oracle will still release interim bundle patches, but recommends for customers to only install the QDPEs unless they have a specific need to install a bundle patch. Note that so far, the QDPEs are only being released for 126.96.36.199 - Linux x86_64, SPARC Solaris (supercluster), and Solaris x86_64.
In addition to the QDPE release, Oracle has announced a "full stack QDPE" - the Quarterly Full Stack Download Patch, or QFSDP. This "full stack" patch includes all of the latest software that can be found in MOS note #888828.1. The January 2012 QFSDP includes:
- Infrastructure Software
- Exadata Storage Server version 188.8.131.52.2
- Exadata Infiniband Switch version 1.3.3-2
- Exadata PDU firmware version 1.04
- 184.108.40.206 January 2012 QDPE
- Opatch 220.127.116.11.9
- Systems Management
- Patches for 11g OEM agents
- Management plugins for 11g OEM
- Patches for 11g OEM management server
No word on when Oracle will start including patches for the new OEM 12c. Keep in mind that these are just a collection of patches, they all still need to be installed as if they were downloaded separately. Oracle does not yet have a mechanism in place to apply the QDPE, storage server patches, Infiniband switch patches, etc in one swoop.
The current QDPE patch is January 2012 (patch #13513783), and the current DFSDP is January 2012 (patch #13551280).
Oracle has upgraded the supplemental note for 18.104.22.168.0 (1348647.1) with a handful of critical issues. First, there is an issue mentioned by Vishal Gupta here regarding 10 gigabit ethernet on the database servers. Currently, there is no workaround for this bug, and it is advised that customers utilizing 10 GbE on the database servers should stay on 22.214.171.124.5 for the minimal pack, while upgrading the storage servers to 126.96.36.199.0.
Additionally, there is a new issue that can be seen if the firmware on a disk fails to upgrade correctly during the patching process. If the firmware does not update correctly, it is possible that cellsrv will drop the celldisk (and corresponding griddisks), causing a loss of data on those disks. Before upgrading a cell to 188.8.131.52.0, check the status of your physical disks from cellcli with the following command:
cellcli -e 'list physicaldisk attributes luns where physicalInsertTime = null'
The command should return no output. If it does, Oracle recommends to reboot each cell that returns output from this command. After the reboot has completed, run the command again to verify that the disks are ready to be patched. Here is the output from one of Enkitec's quarter rack systems (note that there is no output given, so the disks are ok to be upgraded):
[enkcel01:root] /root > dcli -g cell_group -l root cellcli -e 'list physicaldisk attributes luns where physicalInsertTime = null' [enkcel01:root] /root >
Remember that these issues are not mentioned in the standard README files included in the 184.108.40.206.0 patch. Before applying any Exadata Storage Server patches, always consult the supplemental note that is referenced in the supported versions for Exadata note (#888828.1), as these notes are updated after a patch has been released.
Over the past few weeks, I've been working on some new (and older) installations of Exadata, and came across a few items that piqued my interest. Each of these things had been on my mind for a while, but it's nice to see them finally resolved.
The first is a small change to the installation tree of the Oracle homes on Exadata. With the release of 220.127.116.11, Oracle created a new "best practice" of performing all patch sets out of place into a new home. While this makes it really easy to roll back a patch, the default naming convention for Oracle homes on Exadata became a bit of a sticky situation. If your 18.104.22.168 Grid Infrastructure home was at /u01/app/11.2.0/grid, where would you put your 22.214.171.124 home when it's ready to come out? This was the topic of more than a few discussions around the Enkitec office. Do you extend the version out another digit to 126.96.36.199, or version the home (/u01/app/11.2.0/grid_188.8.131.52, etc). Well, Oracle has put this discussion to rest....Your new Oracle home directories on Exadata are:
Grid Infrastructure - /u01/app/184.108.40.206/grid Database - /u01/app/oracle/product/220.127.116.11/dbhome_1
read on for another change (it has to do with bundle patches)