Oracle-Ninja.com http://blog.oracle-ninja.com Andy Colvin's Oracle Blog Thu, 08 Mar 2012 23:44:33 +0000 en hourly 1 http://wordpress.org/?v=3.3.2 Yum Repository for Oracle Enterprise Linux http://blog.oracle-ninja.com/2012/03/yum-repository-for-oracle-enterprise-linux/?utm_source=rss&utm_medium=rss&utm_campaign=yum-repository-for-oracle-enterprise-linux http://blog.oracle-ninja.com/2012/03/yum-repository-for-oracle-enterprise-linux/#comments Thu, 08 Mar 2012 23:44:07 +0000 Andy Colvin http://blog.oracle-ninja.com/?p=443 One of the new requirements of the Exadata Storage Server patches (starting with 11.2.3.1.0) is that the compute nodes will be patched through yum.  Previously, the minimal pack was bundled with the storage server patch and applied directly to the database servers.  Instead of having the compute node directly receive updates from Oracle's yum repository (the Unbreakable Linux Network - ULN), it is recommended to configure a local yum repository that will download the RPMs from ULN.  After this is done, all local servers will connect to the yum repository to receive RPM patches, saving time and network bandwidth.  This post will describe how to configure the yum repository on Oracle Enterprise Linux (process borrowed from OTN).

Creating a yum repository does not require any additional license other than a ULN subscription (included with Exadata or any OEL support contract) unlike RedHat's Satellite product, which can be quite pricey.  First, you will need a server running Oracle Enterprise Linux.  If you don't have anything running OEL, check out Tim Hall's great site - oracle-base.com for a quick primer on installing OEL.  After you have a server up and running with OEL, you'll need to register it with the ULN.  This requires a CSI - included with an OEL license.  To do this, run:

rpm --import /usr/share/rhn/RPM-GPG-KEY
up2date --nox --register

 

1.  Accept the privacy policy

register

2. Enter the ULN credentials and OEL CSI

register

3.  Give the system a profile name and choose whether you want to upload the system information

register

4.  Choose whether to include the RPMs installed on the system in the ULN profile

register

5.  Click next

register

6.  Wait for the profile to be sent

register

7.  Click finish

register

After this is done, your system is registered in ULN.  Log in to http://linux.oracle.com and it should be listed under the systems tab.  In the picture below, we have 2 Exadata compute nodes and 1 server (homer.enkitec.com) that will be our yum server.

1. Click on the pencil next to the server that will host your yum repository to edit the configuration

register

2.  Check the "Yum Server" box and apply the changes

register

3.  Click on the system name to select the channels it will subscribe to

register

4.  Click "Manage Subscriptions"

register

5.  Select the channels that the yum repository will host.  For Exadata compute nodes, the minimum channels are Enterprise Linux 5 Add ons (x86_64), Exadata release 11.2.3.1.0 db server installation packages (x86_64), and Oracle Linux 5 Latest (x86_64).  Select those channels, move them to the "Subscribed Channels" list, and click "Save Subscriptions"

register

Now that you have a server configured as a yum repository in ULN with the appropriate channels, download the 167283.sh script from OTN and give it a more descriptive name.  I used yum_populate.sh.  It will create the yum repository in a directory you specify.  To specify the yum repository location, modify the REP_BASE entry in the script.  I used /archive2/software/linux/yum on my server:

?View Code NONE
# yum repository paths
REP_BASE=/archive2/software/linux/yum

After the script has been configured and put into place, run the script.  It will download the packages from the channels that the yum repository is subscribed to.  This could take a significant amount of time depending on your internet connection speed and the channels the server is subscribed to.  Also, you will have to allocate enough disk space for the channels you're subscribed to.  For example, the channels shown above take up ~26GB at the time of this writing (3/8/2012).

A web server is needed to utilize the yum repository.  I used the built in Apache web server.  First, I needed to make Apache see the directory containing the yum repository.  I added the following lines to the /etc/httpd/conf/httpd.conf file:

?View Code NONE
<Directory /archive2/software/linux/yum>
  Options Indexes FollowSymLinks
</Directory>
Alias /yum/ /archive2/software/linux/yum/

After doing this, I needed to set Apache to start at bootup, and start the web server:

?View Code NONE
[root@homer ~]# chkconfig httpd on
[root@homer ~]# service httpd restart
Stopping httpd:                                            [FAILED]
Starting httpd:                                            [  OK  ]

Now that Apache has started, check to ensure that you're able to view the yum repository from a web browser. Based on the Alias entry above, our site is at http://homer.enkitec.com/yum/ (note the trailing /):

register

Now that this is complete, you're ready to configure your clients to connect directly to the yum repository.  Note that the repository will only be updated when the yum_populate.sh script is run.  It's best to schedule this through cron to run a few times a week.  It's an incremental process, so subsequent runs will only download new RPM releases.

]]> http://blog.oracle-ninja.com/2012/03/yum-repository-for-oracle-enterprise-linux/feed/ 8 Exadata V2 Battery Replacement http://blog.oracle-ninja.com/2012/03/exadata-v2-battery-replacement/?utm_source=rss&utm_medium=rss&utm_campaign=exadata-v2-battery-replacement http://blog.oracle-ninja.com/2012/03/exadata-v2-battery-replacement/#comments Fri, 02 Mar 2012 07:40:26 +0000 Andy Colvin http://blog.oracle-ninja.com/?p=423 For some reason, I've been working on lots of Exadata V2 systems in the past few months.  One of the issues that I've been coming across for these clients is a failure in the battery that is used by the RAID controller.  It was originally expected for these batteries to last 2 years.  Unfortunately, there is a defect in the batteries where they reach their end of life after approximately 18 months.  The local Sun reps should have access to a schedule that says when the "regular maintenance" should occur.  For one client, it wasn't caught until the batteries had run down completely and the disks were in WriteThrough mode.  This can be seen by running MegaCLI64.  Here is the output to check the WriteBack/WriteThrough status for 2 different compute nodes (V2 is first, X2-2 is second):

?View Code NONE
[enkdb01:root] /root
> dmidecode -s system-product-name
SUN FIRE X4170 SERVER          
 
[enkdb01:root] /root
> /opt/MegaRAID/MegaCli/MegaCli64 -LDInfo -LALL -aALL | grep "Cache Policy"
Default Cache Policy: WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU
Current Cache Policy: WriteThrough, ReadAheadNone, Direct, No Write Cache if Bad BBU
Disk Cache Policy   : Disabled
?View Code NONE
[root@enkdb03 ~]# dmidecode -s system-product-name
SUN FIRE X4170 M2 SERVER
[root@enkdb03 ~]# /opt/MegaRAID/MegaCli/MegaCli64 -LDInfo -LALL -aALL | grep "Cache Policy"
Default Cache Policy: WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU
Current Cache Policy: WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU
Disk Cache Policy   : Disabled

If you have a V2 and you haven't replaced the batteries yet, it's worth running these commands to see what state your RAID controllers are in.  To find out what this means for you, read on after the break.

As you can see, on the V2 (X4170), the current cache policy is set to WriteThrough mode, while the X2 (X4170 M2) is still running in WriteBack mode.  What's the difference?  According to the interwebs (http://goo.gl/6LyVK):

RAID Features
Write Through Cache

With Write Through Cache the data is written to both the cache and drive once the data is retrieved. As the data is written to both places, should the information be required it can be retrieved from the cache for faster access. The downside of this method is that the time to carry out a Write operation is greater the time to do a Write to a non cache device. The total Write time is the time to write to the cache plus the time to Write the disk.

Write Back Cache

With Write Back Cache the write operation does not suffer from the Write time delay. The block of data is initially written to the cache, only when the cache is full or required is the data written to the disk.

The limitation of this method is that the storage device for a period of time does not contain the new or updated block of data. If the data in the cache is lost due to power failure the data cannot be recovered. When using Write Back Cache a battery backup module would prevent data loss in a RAID power failure. 

We obviously have a problem on the V2 system, since it's running in WriteThrough mode.  Why is this?  The first thing to check is the status of the battery.  From the Exadata setup/configuration best practices note (#1274318.1), we see the command that can be run to check the battery condition.  From the note:

 Proactive battery replacement should be performed within 60 days for any batteries that do not meet the following criteria:

1) "Full Charge Capacity" less than or equal to 800 mAh and "Max Error" less than 10%.

Immediately replace any batteries that do not meet the following criteria:

1) "Max Error" is 10% or greater (battery deemed unreliable regardless of "Full Charge Capacity" reading)

2) "Full Charge Capacity" less than 674 mAh regardless of "Max Error" reading

Let's run the command on our X2 and see how it looks:

?View Code NONE
[root@enkdb03 ~]# /opt/MegaRAID/MegaCli/MegaCli64 -AdpBbuCmd -a0 | grep "Full Charge" -A5 | sort | grep Full -A1
Full Charge Capacity: 1358 mAh
Max Error: 0 %

That looks great....just what we should expect. The "Full Charge Capacity" is well over 800 mAh threshold. Now, let's look at the system that's in WriteThrough mode:

?View Code NONE
[enkdb01:root] /root
> /opt/MegaRAID/MegaCli/MegaCli64 -AdpBbuCmd -a0 | grep "Full Charge" -A5 | sort | grep Full -A1
Full Charge Capacity: 562 mAh
Max Error: 2 %

That's not good. Full charge capacity is below the dreaded "immediate replacement" level.  Looks like we need to get the batteries replaced.  The process of replacing the batteries is very straightforward.  The node (cell or compute) has to be powered off, opened up, and the old battery is unplugged from the LSI disk controller, and the new battery is connected.  Repeat for each of your servers.  No outage is required...the batteries can be replaced in a rolling fashion.  After a few hours, the new battery is charged, and the disks return to WriteBack mode.  One caveat is that the LSI firmware image included in Exadata storage servers previous to 11.2.2.1.1 do not recognize the new batteries.  Most V2 Exadata systems shipped with version 11.2.1.3.1 or older.  This means that if you're running a V2 and haven't patched yet, it's time to start looking at getting the system up to date.  If you stay on an older version and choose to replace the batteries, you will most likely see no benefit.  All the more reason to keep you Exadata up to date.

One thing that I was curious about was how the battery degradation was missed.  Exadata systems run a periodic check of the battery that will force it down to no charge, and allow it to be charged up.  If you own an Exadata system, you have most likely seen the quarterly (formerly monthly) alerts that all drives are in WriteThrough mode.  After a few cycles, it's easy to become desensitized to these messages and just delete them.  It's imperative that you ensure that you also receive a message that the disks have returned to WriteBack mode.  If you don't receive this, then your batteries may need to be replaced.  Two message should be received for each storage server - one that the disks are in WriteThrough mode, and one that they've returned to WriteBack mode.  The messages should look like this:


Unfortunately, it's not quite good enough to wait for the all clear messages to come flooding your mailbox.  The compute nodes do not send these messages, which means that you could be in WriteThrough mode without being notified.  While it's not as critical as it is on the storage servers, running in WriteThrough mode will show some performance degradation when running operations against the local disks (trace files, logs, local batch jobs, etc).  To resolve this, I suggest running Exachk (MOS note #1070954.1) at least once a quarter.  It will help diagnose anything that may have gone sideways in your Exadata environment.

Happy replacing!

]]> http://blog.oracle-ninja.com/2012/03/exadata-v2-battery-replacement/feed/ 6 Speaking at Miracle Open World 2012 http://blog.oracle-ninja.com/2012/01/speaking-at-miracle-open-world-2012/?utm_source=rss&utm_medium=rss&utm_campaign=speaking-at-miracle-open-world-2012 http://blog.oracle-ninja.com/2012/01/speaking-at-miracle-open-world-2012/#comments Wed, 25 Jan 2012 01:09:25 +0000 Andy Colvin http://blog.oracle-ninja.com/?p=417 I've been invited to speak at Miracle Open World 2012 in Billund, Denmark!  My topic will be "Oracle Database Appliance Internals," which I'll try to make fun and exciting.  I'm definitely looking forward to the experience, and it'll be great to spend time with the other speakers and attendees.  My session is currently scheduled for Thursday afternoon at 4PM, before the dinner and beach party.  If you're coming to the conference, come by my session and say hi!

]]> http://blog.oracle-ninja.com/2012/01/speaking-at-miracle-open-world-2012/feed/ 1 New Exadata Full Stack Patches http://blog.oracle-ninja.com/2012/01/new-exadata-full-stack-patches/?utm_source=rss&utm_medium=rss&utm_campaign=new-exadata-full-stack-patches http://blog.oracle-ninja.com/2012/01/new-exadata-full-stack-patches/#comments Wed, 18 Jan 2012 16:06:30 +0000 Andy Colvin http://blog.oracle-ninja.com/?p=409 Oracle has announced a new patching strategy for Exadata, starting with databases running 11.2.0.3.  Oracle will be moving away from the monthly bundle patch philosophy, which was panned by many administrators as coming too often to keep up with, given the tight schedules held around most Exadata systems.  Instead, Oracle will be releasing a Quarterly Database Patch for Exadata, or QDPE.  The QDPE will most likely be released in conjunction with the standard critical patch updates (CPUs).  Oracle will still release interim bundle patches, but recommends for customers to only install the QDPEs unless they have a specific need to install a bundle patch.  Note that so far, the QDPEs are only being released for 11.2.0.3 - Linux x86_64, SPARC Solaris (supercluster), and Solaris x86_64.

In addition to the QDPE release, Oracle has announced a "full stack QDPE" - the Quarterly Full Stack Download Patch, or QFSDP.  This "full stack" patch includes all of the latest software that can be found in MOS note #888828.1.  The January 2012 QFSDP includes:

  • Infrastructure Software
    • Exadata Storage Server version 11.2.2.4.2
    • Exadata Infiniband Switch version 1.3.3-2
    • Exadata PDU firmware version 1.04
  • Database
    • 11.2.0.3 January 2012 QDPE
    • Opatch 11.2.0.1.9
    • OPlan
  • Systems Management
    • Patches for 11g OEM agents
    • Management plugins for 11g OEM
    • Patches for 11g OEM management server

No word on when Oracle will start including patches for the new OEM 12c.  Keep in mind that these are just a collection of patches, they all still need to be installed as if they were downloaded separately.  Oracle does not yet have a mechanism in place to apply the QDPE, storage server patches, Infiniband switch patches, etc in one swoop.

The current QDPE patch is January 2012 (patch #13513783), and the current DFSDP is January 2012 (patch #13551280).

]]> http://blog.oracle-ninja.com/2012/01/new-exadata-full-stack-patches/feed/ 0 Voting Disk Redundancy in ASM http://blog.oracle-ninja.com/2012/01/voting-disk-redundancy-in-asm/?utm_source=rss&utm_medium=rss&utm_campaign=voting-disk-redundancy-in-asm http://blog.oracle-ninja.com/2012/01/voting-disk-redundancy-in-asm/#comments Fri, 06 Jan 2012 19:15:09 +0000 Andy Colvin http://blog.oracle-ninja.com/?p=380 A recent discussion thread on the OTN Exadata forum made me want to test a feature of 11.2 RAC - voting disk redundancy.  There was one section of the Clusterware Adminsitration and Deployment Guide (http://goo.gl/eMrQM) that made me want to test it out to see how well it worked:

If voting disks are stored on Oracle ASM with normal or high redundancy, and the storage hardware in one failure group suffers a failure, then if there is another disk available in a disk group in an unaffected failure group, Oracle ASM recovers the voting disk in the unaffected failure group.

 
How resilient is it?  How quickly will the cluster recover the voting disk?  Do we have to wait for the ASM rebalance timer to tick down to zero before the voting disk gets recreated?

First, what is a voting disk?  The voting disks are used to determine which nodes are members of the cluster.  In a RAC configuration, there are either 1, 3, or 5 voting disks for the cluster.

Next, when voting disks are placed into an ASM diskgroup, which is the default in 11.2, the number of voting disks that are created depend on the redundancy level of the diskgroup.  For external redundancy, 1 voting disk is created.  When running ASM redundancy, voting disks have a higher requirement for the number of disks.  Normal redundancy creates 3 voting disks, and high redundancy creates 5.  Oracle recommends at least 300MB per voting disk file, and 300MB for each copy of the OCR.  On standard (non-Exadata) RAC builds, I prefer to place OCR and Voting disks into their own diskgroup, named GRID or SYSTEM.

Anyway, back to the fun stuff.  I'm testing this on Exadata, but the results should carry over to any other system running 11.2 RAC.  For starters, we have a diskgroup named DBFS_DG that is used to hold the OCR and voting disks.  We can see that by running the lsdg command in asmcmd. I've abbreviated the output for clarity.

?View Code NONE
[enkdb03:oracle:+ASM1] /u01/app/11.2.0.2/grid
> asmcmd
ASMCMD> lsdg
State    Type    Rebal  Total_MB   Free_MB  Usable_file_MB  Offline_disks  Voting_files  Name
MOUNTED  NORMAL  N      65261568  49688724        21877927              0             N  DATA/
MOUNTED  NORMAL  N       1192320    978292          434950              0             Y  DBFS_DG/
MOUNTED  HIGH    N      22935248  20570800         5466918              0             N  RECO/
ASMCMD>

The DBFS_DG diskgroup has 4 failgroups - ENKCEL04, ENKCEL05, ENKCEL06, and ENKCEL07:

?View Code NONE
SYS:+ASM1> select distinct g.name "Diskgroup",
  2    d.failgroup "Failgroup"
  3  from v$asm_diskgroup g,
  4    v$asm_disk d
  5  where g.group_number = d.group_number
  6  and g.NAME = 'DBFS_DG'
  7  /
 
Diskgroup		       Failgroup
------------------------------ ------------------------------
DBFS_DG 		       ENKCEL04
DBFS_DG 		       ENKCEL05
DBFS_DG 		       ENKCEL06
DBFS_DG 		       ENKCEL07
 
SYS:+ASM1>

Finally, our current voting disks reside in failgroups ENKCEL05, ENKCEL06, and ENKCEL07:

?View Code NONE
[enkdb03:oracle:+ASM1] /home/oracle
> sudo /u01/app/11.2.0.2/grid/bin/crsctl query css votedisk
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   43164b9cc7234fe1bff4eb968ec4a1dc (o/192.168.12.10/DBFS_DG_CD_02_enkcel06) [DBFS_DG]
 2. ONLINE   2e6db5ba5fd34fc8bfaa0ab8b9d0ddf5 (o/192.168.12.11/DBFS_DG_CD_03_enkcel07) [DBFS_DG]
 3. ONLINE   95eb38e5dfc34f3ebfd72127e7fe9c12 (o/192.168.12.9/DBFS_DG_CD_02_enkcel05) [DBFS_DG]
Located 3 voting disk(s).

Going back to the original questions, what does it take for CRS to notice that a voting disk is gone, and how quickly will it be replaced? Are they handled in the same way as normal disks, or some other way? When an ASM disk goes offline, ASM waits the amount of time listed in the disk_repair_time attribute for the diskgroup (default is 3.6 hours) before dropping the disk and performing a rebalance. Let's offline one of the failgroups that has a voting disk and find out.

?View Code NONE
SYS:+ASM1> !date
Fri Jan  6 12:57:18 CST 2012
 
SYS:+ASM1> alter diskgroup dbfs_dg offline disks in failgroup ENKCEL05;
 
Diskgroup altered.
 
SYS:+ASM1> !date
Fri Jan  6 12:57:46 CST 2012
 
SYS:+ASM1> !sudo /u01/app/11.2.0.2/grid/bin/crsctl query css votedisk
[sudo] password for oracle:
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   43164b9cc7234fe1bff4eb968ec4a1dc (o/192.168.12.10/DBFS_DG_CD_02_enkcel06) [DBFS_DG]
 2. ONLINE   2e6db5ba5fd34fc8bfaa0ab8b9d0ddf5 (o/192.168.12.11/DBFS_DG_CD_03_enkcel07) [DBFS_DG]
 3. ONLINE   ab06b75d54764f79bfd4ba032b317460 (o/192.168.12.8/DBFS_DG_CD_02_enkcel04) [DBFS_DG]
Located 3 voting disk(s).
 
SYS:+ASM1> !date
Fri Jan  6 12:58:03 CST 2012

That was quick! So from this exercise, we can see that ASM doesn't waste any time to recreate the voting disk. With files that are this critical to the stability of the cluster, it's not hard to see why. If we dig further, we can see that ocssd noticed that the voting disk on o/192.168.12.9/DBFS_DG_CD_02_enkcel05 was offline, and it proceeded to create a new file on disk o/192.168.12.8/DBFS_DG_CD_02_enkcel04:

?View Code NONE
2012-01-06 12:57:37.075
[cssd(10880)]CRS-1605:CSSD voting file is online: o/192.168.12.8/DBFS_DG_CD_02_enkcel04; details in /u01/app/11.2.0.2/grid/log/enkdb03/cssd/ocssd.log.
2012-01-06 12:57:37.075
[cssd(10880)]CRS-1604:CSSD voting file is offline: o/192.168.12.9/DBFS_DG_CD_02_enkcel05; details at (:CSSNM00069:) in /u01/app/11.2.0.2/grid/log/enkdb03/cssd/ocssd.log.

Finally, what happens when the failgroup comes back online? There are no changes made to the voting disks. The voting disk from the failgroup that was previously offline is still there, but will be overwritten if that space is needed:

?View Code NONE
SYS:+ASM1> !date
Fri Jan  6 13:05:09 CST 2012
 
SYS:+ASM1> alter diskgroup dbfs_dg online disks in failgroup ENKCEL05;
 
Diskgroup altered.
 
SYS:+ASM1> !date
Fri Jan  6 13:06:31 CST 2012
 
SYS:+ASM1> !sudo /u01/app/11.2.0.2/grid/bin/crsctl query css votedisk
[sudo] password for oracle:
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   43164b9cc7234fe1bff4eb968ec4a1dc (o/192.168.12.10/DBFS_DG_CD_02_enkcel06) [DBFS_DG]
 2. ONLINE   2e6db5ba5fd34fc8bfaa0ab8b9d0ddf5 (o/192.168.12.11/DBFS_DG_CD_03_enkcel07) [DBFS_DG]
 3. ONLINE   ab06b75d54764f79bfd4ba032b317460 (o/192.168.12.8/DBFS_DG_CD_02_enkcel04) [DBFS_DG]
Located 3 voting disk(s).

In previous versions, this automation did not exist. If a voting disk went offline, the DBA had to manually create a new voting disk. Keep in mind that this feature will only take effect if there must be another failgroup available to place the voting disk in. If you only have 3 failgroups for your normal redundancy (or 5 for your high redundancy) OCR/voting diskgroup, this automatic recreation will not occur.

]]> http://blog.oracle-ninja.com/2012/01/voting-disk-redundancy-in-asm/feed/ 5 Exadata Critical Patch for 11.2.2.3.x through 11.2.2.4.1 http://blog.oracle-ninja.com/2011/12/exadata-critical-patch-for-11-2-2-3-x-through-11-2-2-4-1/?utm_source=rss&utm_medium=rss&utm_campaign=exadata-critical-patch-for-11-2-2-3-x-through-11-2-2-4-1 http://blog.oracle-ninja.com/2011/12/exadata-critical-patch-for-11-2-2-3-x-through-11-2-2-4-1/#comments Thu, 29 Dec 2011 13:53:24 +0000 Andy Colvin http://blog.oracle-ninja.com/?p=364 Oracle has released a critical patch for storage server versions 11.2.2.3.x through 11.2.2.4.1.  While 11.2.2.4.1 was released last week, there were a few oneoff patches from 11.2.2.4.0 that didn't seem to make it in to the release.  Oracle has since released 11.2.2.4.2 (patch #13513611, supplemental note #1388400.1).  Similar to 11.2.2.4.1, this release looks to patch several outstanding issues.  Here's the list of bugs fixed from the readme for 11.2.2.4.2:

12764521        INFINIBAND DIAG COMMANDS (LIKE IBDIAGNET AND IBNETDISCOVER) ARE NOT WORKING
13083530        10 GB-E BONDED INTERFACES FAILING- EXADATA
13410353        AFTER UPGRADE TO 11.2.2.4 INFINIBAND CMDS IBDIAGNET, IBNETDISCOVER NOT WORKING
13489032        CHECKHWNFWPROFILE DOES NOT DETECT FAILED FLASH FDOM
13489445        ORA-600 [OSSMISC:OSSMISC_TIMER] WHEN NTPD DETECTED 6 MILLISECOND TIME DIFFERENCE
13512932        FIX INSTALLED WORKAROUND FOR NTP UPDATE BUG 13489445

As you can see, the previously mentioned bugs have been fixed.  There's another bug that was fixed in 11.2.2.4.1 that could be an issue for anybody running 11.2.2.3.x through 11.2.2.4.0.  This bug (13454147) can remove the flashcache from a cell that has an uptime of 6 months or greater.  Fortunately, Oracle has released a patch that includes these critical issues in the event that you can't quickly upgrade to 11.2.2.4.2 - I wouldn't advise running this version for at least a couple weeks...I always advise clients to wait that long for the early adopters to weed out any major issues.

Applying the critical patch only takes a minute, and doesn't take the storage servers or database instances offline.  After it's done, a restart of cellsrv needs to be scheduled, but that can be done in a rolling fashion.  Read on for an example of applying this patch.  As always, do not apply any patch to a production system before appropriately testing against a non-production system!

According to the documentation for patch 13517481, the following bugs are fixed:

Bug       Description
--------  -------------------------------------------------------------------
13454147  Flash cards go offline after 6 months of uptime
12886507  IDT switch in PCI riser resets causing missing flash cards
12626126  Temporary IO stall caused by drive medium errors causes cell reboot
13489445  CELLSRV crash if NTPD interrupted and time drifts back too far
13083530  10GbE network interfaces shutdown

The installation is very quick and easy.  Unpack the patch to a directory on the first database server (I used /u01/stage/patches/11.2.2.4.1_supplemental), and cd to the directory.

?View Code NONE
[enkdb01:root] /root
> cd /u01/stage/patches/11.2.2.4.1_supplemental/
 
[enkdb01:root] /u01/stage/patches/11.2.2.4.1_supplemental
> ls
p13517481_112100_Linux-x86-64.zip
 
[enkdb01:root] /u01/stage/patches/11.2.2.4.1_supplemental
> unzip p13517481_112100_Linux-x86-64.zip
Archive:  p13517481_112100_Linux-x86-64.zip
   creating: 13517481/
  inflating: 13517481/fixpciidt_12886507
  inflating: 13517481/10gig_rxusecs0
  inflating: 13517481/README.txt
  inflating: 13517481/install.sh
  inflating: 13517481/fix_flash_links.sh  
 
[enkdb01:root] /u01/stage/patches/11.2.2.4.1_supplemental
> cd 13517481/
 
[enkdb01:root] /u01/stage/patches/11.2.2.4.1_supplemental/13517481
> ls
10gig_rxusecs0  fix_flash_links.sh  fixpciidt_12886507  install.sh  README.txt

After this has been done, copy the all_group file from root's home directory, and verify that SSH equivalence works ok. If everything passes, run the patch check:

?View Code NONE
[enkdb01:root] /u01/stage/patches/11.2.2.4.1_supplemental/13517481
> cp ~/all_group .
 
[enkdb01:root] /u01/stage/patches/11.2.2.4.1_supplemental/13517481
> dcli -l root -g all_group hostname
enkdb01: enkdb01.enkitec.com
enkdb02: enkdb02.enkitec.com
enkcel01: enkcel01.enkitec.com
enkcel02: enkcel02.enkitec.com
enkcel03: enkcel03.enkitec.com
 
[enkdb01:root] /u01/stage/patches/11.2.2.4.1_supplemental/13517481
> ./install.sh -g all_group check
 
Additional details in patch13517481.log
 
Perform check using dcli on all systems in all_group: enkdb01 enkdb02 enkcel01 enkcel02 enkcel03 
 
Completed check on all systems.
 
Screen output captured from all systems and placed in patch13517481.log
 
Log files from all systems collected and placed in the current directory /u01/stage/patches/11.2.2.4.1_supplemental/13517481

Check the logs that were created (patch13517481.log for the summary, there is a log for each server), and if everything passes, then apply the patch. Note that the patch will only apply the required fixes based on the server type. On our V2 Exadata shown here, it does not apply the 10GbE patch...The V2 systems do not have 10GbE capability.

?View Code NONE
[enkdb01:root] /u01/stage/patches/11.2.2.4.1_supplemental/13517481
> ./install.sh -g all_group apply
 
Additional details in patch13517481.log
 
Perform apply using dcli on all systems in all_group: enkdb01 enkdb02 enkcel01 enkcel02 enkcel03 
 
Completed apply on all systems.
 
Screen output captured from all systems and placed in patch13517481.log
 
Log files from all systems collected and placed in the current directory /u01/stage/patches/11.2.2.4.1_supplemental/13517481

The log is appended, and you can check to make sure that the patches were applied successfully. Here are the relevant contents of our patch13517481.log file:

?View Code NONE
2011-12-29 07:13:22 CST main: ====================================================================
2011-12-29 07:20:49 CST main: Running ./install.sh with options ACTION=apply, BUGFIX=ALL, GROUPFILE=all_group
2011-12-29 07:20:49 CST main: Perform apply using dcli on all systems in all_group: enkdb01 enkdb02 enkcel01 enkcel02 enkcel03
2011-12-29 07:20:49 CST main: Verifying SSH setup and free space for all systems
2011-12-29 07:20:49 CST main: SSH validation for enkdb01 passed
2011-12-29 07:20:49 CST main: Free space validation for enkdb01 passed
2011-12-29 07:20:49 CST main: SSH validation for enkdb02 passed
2011-12-29 07:20:49 CST main: Free space validation for enkdb02 passed
2011-12-29 07:20:50 CST main: SSH validation for enkcel01 passed
2011-12-29 07:20:50 CST main: Free space validation for enkcel01 passed
2011-12-29 07:20:50 CST main: SSH validation for enkcel02 passed
2011-12-29 07:20:50 CST main: Free space validation for enkcel02 passed
2011-12-29 07:20:50 CST main: SSH validation for enkcel03 passed
2011-12-29 07:20:50 CST main: Free space validation for enkcel03 passed
2011-12-29 07:20:50 CST main: Create working directory /tmp/patch13517481_122911072049 on all systems
2011-12-29 07:20:50 CST main: Distribute patch files to all systems
2011-12-29 07:20:51 CST main: Execute './install.sh -b ALL apply' on all systems
2011-12-29 07:20:54 CST main: Completed apply on all systems.
2011-12-29 07:20:54 CST main: Screen output captured from all systems and placed in patch13517481.log
2011-12-29 07:20:54 CST main: enkdb01:
2011-12-29 07:20:54 CST main: enkdb01: Additional details in /tmp/patch13517481_122911072049/patch13517481.log
2011-12-29 07:20:54 CST main: enkdb01:
2011-12-29 07:20:54 CST main: enkdb01: Fix for bug 13454147 (NoFlash six months) - apply
2011-12-29 07:20:54 CST main: enkdb01: Fix not applicable to this system
2011-12-29 07:20:54 CST main: enkdb01:
2011-12-29 07:20:54 CST main: enkdb01: Fix for bug 12886507 (IDT switch reset) - apply
2011-12-29 07:20:54 CST main: enkdb01: Fix not applicable to this system
2011-12-29 07:20:54 CST main: enkdb01:
2011-12-29 07:20:54 CST main: enkdb01: Fix for bug 13489445 (NTPD CELLSRV crash) - apply
2011-12-29 07:20:54 CST main: enkdb01: Fix not applicable to this system
2011-12-29 07:20:54 CST main: enkdb01:
2011-12-29 07:20:54 CST main: enkdb01: Fix for bug 13083530 (10GbE shutdown) - apply
2011-12-29 07:20:54 CST main: enkdb01: Fix not applicable to this system
2011-12-29 07:20:54 CST main: enkdb01:
2011-12-29 07:20:54 CST main: enkdb01: Fix for bug 12626126 (IO stall cell reboot) - apply
2011-12-29 07:20:54 CST main: enkdb01: Fix not applicable to this system
2011-12-29 07:20:54 CST main: enkdb01:
2011-12-29 07:20:54 CST main: enkdb02:
2011-12-29 07:20:54 CST main: enkdb02: Additional details in /tmp/patch13517481_122911072049/patch13517481.log
2011-12-29 07:20:54 CST main: enkdb02:
2011-12-29 07:20:54 CST main: enkdb02: Fix for bug 13454147 (NoFlash six months) - apply
2011-12-29 07:20:54 CST main: enkdb02: Fix not applicable to this system
2011-12-29 07:20:54 CST main: enkdb02:
2011-12-29 07:20:54 CST main: enkdb02: Fix for bug 12886507 (IDT switch reset) - apply
2011-12-29 07:20:54 CST main: enkdb02: Fix not applicable to this system
2011-12-29 07:20:54 CST main: enkdb02:
2011-12-29 07:20:54 CST main: enkdb02: Fix for bug 13489445 (NTPD CELLSRV crash) - apply
2011-12-29 07:20:54 CST main: enkdb02: Fix not applicable to this system
2011-12-29 07:20:54 CST main: enkdb02:
2011-12-29 07:20:54 CST main: enkdb02: Fix for bug 13083530 (10GbE shutdown) - apply
2011-12-29 07:20:54 CST main: enkdb02: Fix not applicable to this system
2011-12-29 07:20:54 CST main: enkdb02:
2011-12-29 07:20:54 CST main: enkdb02: Fix for bug 12626126 (IO stall cell reboot) - apply
2011-12-29 07:20:54 CST main: enkdb02: Fix not applicable to this system
2011-12-29 07:20:54 CST main: enkdb02:
2011-12-29 07:20:54 CST main: enkcel01:
2011-12-29 07:20:54 CST main: enkcel01: Additional details in /tmp/patch13517481_122911072049/patch13517481.log
2011-12-29 07:20:54 CST main: enkcel01:
2011-12-29 07:20:54 CST main: enkcel01: Fix for bug 13454147 (NoFlash six months) - apply
2011-12-29 07:20:54 CST main: enkcel01: Fix for bug 13454147 - apply SUCCESS
2011-12-29 07:20:54 CST main: enkcel01:
2011-12-29 07:20:54 CST main: enkcel01: Fix for bug 12886507 (IDT switch reset) - apply
2011-12-29 07:20:54 CST main: enkcel01: Fix not needed for Exadata version 11.2.2.4.0.110929 - no action taken
2011-12-29 07:20:54 CST main: enkcel01:
2011-12-29 07:20:54 CST main: enkcel01: Fix for bug 13489445 (NTPD CELLSRV crash) - apply
2011-12-29 07:20:54 CST main: enkcel01: ACTION REQUIRED - Fix for bug 13489445 applied and will become active at next CELLSRV restart
2011-12-29 07:20:54 CST main: enkcel01: Fix for bug 13489445 - apply SUCCESS
2011-12-29 07:20:54 CST main: enkcel01:
2011-12-29 07:20:54 CST main: enkcel01: Fix for bug 13083530 (10GbE shutdown) - apply
2011-12-29 07:20:54 CST main: enkcel01: Fix not applicable to this system
2011-12-29 07:20:54 CST main: enkcel01:
2011-12-29 07:20:54 CST main: enkcel01: Fix for bug 12626126 (IO stall cell reboot) - apply
2011-12-29 07:20:54 CST main: enkcel01: Fix not needed for Exadata version 11.2.2.4.0.110929
2011-12-29 07:20:54 CST main: enkcel01:
2011-12-29 07:20:54 CST main: enkcel02:
2011-12-29 07:20:54 CST main: enkcel02: Additional details in /tmp/patch13517481_122911072049/patch13517481.log
2011-12-29 07:20:54 CST main: enkcel02:
2011-12-29 07:20:54 CST main: enkcel02: Fix for bug 13454147 (NoFlash six months) - apply
2011-12-29 07:20:54 CST main: enkcel02: Fix for bug 13454147 - apply SUCCESS
2011-12-29 07:20:54 CST main: enkcel02:
2011-12-29 07:20:54 CST main: enkcel02: Fix for bug 12886507 (IDT switch reset) - apply
2011-12-29 07:20:54 CST main: enkcel02: Fix not needed for Exadata version 11.2.2.4.0.110929 - no action taken
2011-12-29 07:20:54 CST main: enkcel02:
2011-12-29 07:20:54 CST main: enkcel02: Fix for bug 13489445 (NTPD CELLSRV crash) - apply
2011-12-29 07:20:54 CST main: enkcel02: ACTION REQUIRED - Fix for bug 13489445 applied and will become active at next CELLSRV restart
2011-12-29 07:20:54 CST main: enkcel02: Fix for bug 13489445 - apply SUCCESS
2011-12-29 07:20:54 CST main: enkcel02:
2011-12-29 07:20:54 CST main: enkcel02: Fix for bug 13083530 (10GbE shutdown) - apply
2011-12-29 07:20:54 CST main: enkcel02: Fix not applicable to this system
2011-12-29 07:20:54 CST main: enkcel02:
2011-12-29 07:20:54 CST main: enkcel02: Fix for bug 12626126 (IO stall cell reboot) - apply
2011-12-29 07:20:54 CST main: enkcel02: Fix not needed for Exadata version 11.2.2.4.0.110929
2011-12-29 07:20:54 CST main: enkcel02:
2011-12-29 07:20:54 CST main: enkcel03:
2011-12-29 07:20:54 CST main: enkcel03: Additional details in /tmp/patch13517481_122911072049/patch13517481.log
2011-12-29 07:20:54 CST main: enkcel03:
2011-12-29 07:20:54 CST main: enkcel03: Fix for bug 13454147 (NoFlash six months) - apply
2011-12-29 07:20:54 CST main: enkcel03: Fix for bug 13454147 - apply SUCCESS
2011-12-29 07:20:54 CST main: enkcel03:
2011-12-29 07:20:54 CST main: enkcel03: Fix for bug 12886507 (IDT switch reset) - apply
2011-12-29 07:20:54 CST main: enkcel03: Fix not needed for Exadata version 11.2.2.4.0.110929 - no action taken
2011-12-29 07:20:54 CST main: enkcel03:
2011-12-29 07:20:54 CST main: enkcel03: Fix for bug 13489445 (NTPD CELLSRV crash) - apply
2011-12-29 07:20:54 CST main: enkcel03: ACTION REQUIRED - Fix for bug 13489445 applied and will become active at next CELLSRV restart
2011-12-29 07:20:54 CST main: enkcel03: Fix for bug 13489445 - apply SUCCESS
2011-12-29 07:20:54 CST main: enkcel03:
2011-12-29 07:20:54 CST main: enkcel03: Fix for bug 13083530 (10GbE shutdown) - apply
2011-12-29 07:20:54 CST main: enkcel03: Fix not applicable to this system
2011-12-29 07:20:54 CST main: enkcel03:
2011-12-29 07:20:54 CST main: enkcel03: Fix for bug 12626126 (IO stall cell reboot) - apply
2011-12-29 07:20:54 CST main: enkcel03: Fix not needed for Exadata version 11.2.2.4.0.110929
2011-12-29 07:20:54 CST main: enkcel03:
2011-12-29 07:20:54 CST main: Log files from all systems collected and placed in the current directory /u01/stage/patches/11.2.2.4.1_supplemental/13517481
2011-12-29 07:20:54 CST main: Log file enkdb01_patch13517481.log from enkdb01
2011-12-29 07:20:54 CST main: Log file enkdb02_patch13517481.log from enkdb02
2011-12-29 07:20:54 CST main: Log file enkcel01_patch13517481.log from enkcel01
2011-12-29 07:20:54 CST main: Log file enkcel02_patch13517481.log from enkcel02
2011-12-29 07:20:54 CST main: Log file enkcel03_patch13517481.log from enkcel03
2011-12-29 07:20:54 CST main: Remove working directory /tmp/patch13517481_122911072049 on all systems
2011-12-29 07:20:54 CST main: Exiting
2011-12-29 07:20:54 CST main: ====================================================================

Note that the NTP fix will not be available until the next cellsrv restart.  This is another critical bug, so do not forget to bounce cellsrv sometime after applying the patch.  Remember that cellsrv bounces can be done in a rolling fashion, but should probably be scheduled for a window where the system activity is low.

]]> http://blog.oracle-ninja.com/2011/12/exadata-critical-patch-for-11-2-2-3-x-through-11-2-2-4-1/feed/ 0 Exadata Diskgroup Planning http://blog.oracle-ninja.com/2011/12/exadata-diskgroup-planning/?utm_source=rss&utm_medium=rss&utm_campaign=exadata-diskgroup-planning http://blog.oracle-ninja.com/2011/12/exadata-diskgroup-planning/#comments Wed, 21 Dec 2011 04:23:47 +0000 Andy Colvin http://blog.oracle-ninja.com/?p=334 As business has picked up since OpenWorld (didn't think that was possible, but that's another story for another day), we have been seeing more customers adopt or seriously look at Exadata as an option for new hardware implementations.  While many will complain that there isn't enough room for customization in the rigid process of configuring an Exadata system, there are still many possibilities to make your Exadata your own, whether it's during the initial configuration phase or shortly thereafter.  Of course, some of these modifications can be difficult to implement after the system is up and running with users logging in.  I'm planning on starting a series of posts regarding a couple of the hot-button topics with regard to Exadata configuration - ASM diskgroup layout (the topic for today), role separated vs standard authentication, and so on.  As these topics have no right answers, I'm more than open to a dialogue where you may disagree.  On to the good stuff!

A Quick Primer - The Exadata Storage Architecture

Ok...so we're looking at Exadata specifically in this post.  In the examples listed below, we'll discuss a quarter rack, since it's the easiest to diagram.  To expand to half or full racks, just adjust the number of cells (7, 14) and disks (84, 168) accordingly.  To see the relationship between the compute nodes (database servers), Infiniband switches, and storage servers refer to figure 1:

Figure 1 - Exadata Infiniband/Storage Connectivity

As you can see, each storage server has 12 physical disks that are carved up for the use of ASM.  Figure 2 shows the relationship between physicaldisks, celldisks, and griddisks.  What we're really interested in are the griddisks.

Figure 2 - Exadata Disk Relationship (Oracle's pretty diagram)

To expand upon the last piece of the diagram, we have a representation of what the celldisks look like when carved up according to the default installation on Exadata.  This shows the first 3 disks, followed by disk "n" which represents the remaining disks.

Figure 3 - Default Exadata Griddisk Layout (My ugly diagram)

On each cell, the first 2 disks have approximately 30GB carved off for the operating system.  This space is used on the remaining disks to create the DBFS_DG (formerly SYSTEMDG) griddisks to level out the remaining space.  After that, a griddisk for DATA is created (starting from the outside tracks for the best performance), followed by the RECO griddisk.  These griddisks are what translate to disks in the ASM instances.

What's Different with Exadata?

On a traditional SAN, the typical approach is to create multiple diskgroups, and  allocate a LUN for each diskgroup.  Each LUN would be made up of different disks.  Whether the decision to create multiple diskgroups is based on separating entire databases from each other or even ensuring that indexes and tables do not share spindles, it is up to the DBAs and storage administrators to design a storage strategy that best fits the specific environment.  Using this strategy, the I/O on one diskgroup would not affect the others.  Once again, all of this changes with Exadata.

As mentioned earlier, the default configuration on Exadata is to have 3 diskgroups - DATA, RECO, and DBFS_DG.  The diskgroups span all of the disks in the rack, and each physical disk is used by each of the diskgroups.  This is done in order to maximize performance by using as many spindles as you can.  Whether you have 1 diskgroup or 5 diskgroups on Exadata, they will still be sharing the same physical disks.  While it is possible to only use certain disks (0-5) for one disk group and others (6-11) for another diskgroup, this will result in a degradation of performance, due to only working on half of the available disks.

The other option for multiple diskgroups is to drop the default griddisks and create new ones based on a new layout of diskgroups.  The original diskgroups and griddisks (DATA, RECO) would be dropped, and new diskgroups would be created.  In the example below, I use DG1, DG2, DG3, DG4, and keep the original DBFS_DG.  As you can see, objects in DG1 should see better performance than DG4, due to the location on the disk.  The biggest issue that I see is the lack of flexibility after creating a layout like this.  It is my preference to spend as much time as possible with customers before building out the Exadata databases to find out how things should be laid out, especially the storage.

Multiple Exadata Diskgroups

Figure 4 - Multiple Exadata Diskgroups Layout

One of the things that I have learned from the various Exadata projects that I have been on is that once "the business" begins to see the performance gains from the original system that the Exadata was purchased for, other databases are sure to follow.  While you may have originally purchased that half rack for a Peoplesoft deployment, the BI folks start thinking about what Exadata can do for them, the data warehouse people begin to lobby to place their databases on the system, and there are twice as many databases running as what was originally planned.  What happens when you created diskgroups based on the 5 different Peoplesoft (not that there would only be 5 PSFT databases) or EBS databases that were originally envisioned to run on that Exadata?  With the storage segmented, it's hard to fit the new databases around the space left through the various diskgroups.  It's not as simple as adding a new LUN (short of purchasing more storage servers, although I'd always be up for that) for the reasons listed above.

Say your diskgroups originally followed a naming strategy where they were named after the database that resided in them.  What happens if you needed to add another database in there?  While it is possible to rename a diskgroup (see metalink note #948040.1), you would end up with the very sticky situation where the ABC griddisks belonged to the XYZ diskgroups.  Not something that I would want to have to remember.

Where Does This Leave Us?

This begs the question - what is gained by creating multiple diskgroups?  While I understand the argument that segmenting the disk usage keeps one database from gobbling up all of the available free space unexpectedly, I don't see this a a common occurrence in the Exadata world.  Certainly not as often as I see the other items above.  We are talking about terabytes of space here, after all.  Also, if the DBAs are going to take on the additional role of storage administrator on Exadata (as is usually the case), then it's time for us to be big boys (and girls), and own it.  It's not like managing the storage on Exadata is very tricky.  Once the storage is configured, there's very little to it.  There is no need to bother with zoning fibrechannel switches, configure storage groups, or worry about which LUNs contain which disks.  Yes, there are disk failures, but that's handled very well by ASM these days.  If you're feeling frisky, you can play around with ASM's "Intelligent Data Placement" to create "hot" and "cold" areas of the diskgroup to better manage performance.

Just like many other aspects of being a DBA on Exadata, the storage layout is different from a traditional system.  Based on the requirements and the architecture, administrators are being forced to think differently and look for new solutions.  As I said above, if you disagree, please feel free to drop a comment and defend your position.  As with most things Oracle, there isn't one right answer.

]]> http://blog.oracle-ninja.com/2011/12/exadata-diskgroup-planning/feed/ 2 Inside the Oracle Database Appliance – Part 2 http://blog.oracle-ninja.com/2011/12/inside-the-oracle-database-appliance-%e2%80%93-part-2/?utm_source=rss&utm_medium=rss&utm_campaign=inside-the-oracle-database-appliance-%25e2%2580%2593-part-2 http://blog.oracle-ninja.com/2011/12/inside-the-oracle-database-appliance-%e2%80%93-part-2/#comments Tue, 13 Dec 2011 03:09:09 +0000 Andy Colvin http://blog.oracle-ninja.com/?p=235 In part 1 of this series, we took a look inside the ODA to see what the OS was doing.  Here, we'll dig in a little further to the disk and storage architecture with regard to the hardware and ASM.

There have been a lot of questions about the storage layout of the shared disks.  We'll start at the lowest level and make our way to the disks as we move down the ladder.  First, there are 2 dual-ported LSI SAS controllers in each of the system controllers (SCs).  They are each connected to a SAS expander that is located on the system board.  Each of these SAS expanders connect to 12 of the hard disks on the front of the ODA.  The disks are dual-ported SAS, so that each disk is connected to an expander on each of the SCs.  Below is a diagram of the SAS connectivity on the ODA (Note:  all diagrams are collected from public ODA documentation, as well as various ODA-related support notes available on My Oracle Support).

From this, you can see the relationship between the SAS controllers, SAS expanders, and SAS drives on the front end.  If you look at the columns of disks, the first 2 columns are serviced by one expander, while the third and fourth columns are services by the other expander.  What the diagram refers to as "Controller-0" and "Controller-1" are actually the independent SCs in the X4370M2.  What this shows is that you can lose any of the following components in the diagram and your database will continue to run (assuming RAC is in use):

  • Hard Disk
  • SSD
  • SAS Expander
  • SAS Controller (RAID adapter)

Only the loss of an expander would cause the entire SC to come down.  In the event of an SSD or hard disk failure (even 2 hard disks), ASM redundancy would keep both database instances online.  All hard disks and solid state disks are hot-swappable, so there is no downtime to replace a disk.  You would have to power off the SC to replace a SAS controller, but the SC would still have a path to all disks until that maintenance window, leaving you the ability to take it down at your convenience.  The unaffected SC would continue to run without issue.  This also applies to other hardware failures local to a single SC.  The only components shared between the SCs are the SSD, hard disks, and power supplies (all of which are redundant).

How are the disks actually configured at the OS level?  Because all of the connections inside the SCs are redundant, the disk devices (and SSD) are configured to use device-mapper for multipathing.  The oak configurator automatically configures the /etc/multipath.conf file on each SC after the disks have been partitioned.  They are named based on the type of disk (HDD or SSD), expander, slot, and a hash value.  Below is an sample of the /etc/multipath.com file:

?View Code NONE
defaults {
user_friendly_names  yes
path_grouping_policy failover
failback manual
no_path_retry fail
flush_on_last_del yes
queue_without_daemon no
}
multipaths {
    multipath {
      wwid 35000c5003a446893
      alias HDD_E0_S01_977561747
      mode 660
      uid 1000
      gid 1006
}
    multipath {
      wwid 35000c5003a42ad5b
      alias HDD_E0_S00_977448283
      mode 660
      uid 1000
      gid 1006
}

This results in the following disks:

?View Code NONE
[root@patty ~]# ls -al /dev/mapper/*
crw------- 1 root root      10, 63 Dec  5 17:38 /dev/mapper/control
brw-rw---- 1 grid asmadmin 253, 16 Dec  6 09:47 /dev/mapper/HDD_E0_S00_977448283
brw-rw---- 1 grid asmadmin 253, 61 Dec 12 08:59 /dev/mapper/HDD_E0_S00_977448283p1
brw-rw---- 1 grid asmadmin 253, 62 Dec 12 08:59 /dev/mapper/HDD_E0_S00_977448283p2
brw-rw---- 1 grid asmadmin 253, 26 Dec  6 09:47 /dev/mapper/HDD_E0_S01_977561747
brw-rw---- 1 grid asmadmin 253, 29 Dec 12 08:59 /dev/mapper/HDD_E0_S01_977561747p1
brw-rw---- 1 grid asmadmin 253, 30 Dec 12 08:59 /dev/mapper/HDD_E0_S01_977561747p2
brw-rw---- 1 grid asmadmin 253, 17 Dec  6 09:48 /dev/mapper/HDD_E0_S04_975085971
brw-rw---- 1 grid asmadmin 253, 66 Dec 12 08:59 /dev/mapper/HDD_E0_S04_975085971p1
brw-rw---- 1 grid asmadmin 253, 67 Dec 12 08:59 /dev/mapper/HDD_E0_S04_975085971p2
brw-rw---- 1 grid asmadmin 253, 27 Dec  6 09:48 /dev/mapper/HDD_E0_S05_977762435
brw-rw---- 1 grid asmadmin 253, 37 Dec 12 08:59 /dev/mapper/HDD_E0_S05_977762435p1
brw-rw---- 1 grid asmadmin 253, 38 Dec 12 08:59 /dev/mapper/HDD_E0_S05_977762435p2
brw-rw---- 1 grid asmadmin 253, 18 Dec  6 09:48 /dev/mapper/HDD_E0_S08_975084479
brw-rw---- 1 grid asmadmin 253, 68 Dec 12 08:59 /dev/mapper/HDD_E0_S08_975084479p1
brw-rw---- 1 grid asmadmin 253, 69 Dec 12 08:48 /dev/mapper/HDD_E0_S08_975084479p2
brw-rw---- 1 grid asmadmin 253, 24 Dec  6 09:49 /dev/mapper/HDD_E0_S09_977562223
brw-rw---- 1 grid asmadmin 253, 33 Dec 12 08:59 /dev/mapper/HDD_E0_S09_977562223p1
brw-rw---- 1 grid asmadmin 253, 34 Dec 12 08:59 /dev/mapper/HDD_E0_S09_977562223p2
brw-rw---- 1 grid asmadmin 253, 19 Dec  6 09:49 /dev/mapper/HDD_E0_S12_975086927
brw-rw---- 1 grid asmadmin 253, 64 Dec 12 08:59 /dev/mapper/HDD_E0_S12_975086927p1
brw-rw---- 1 grid asmadmin 253, 65 Dec 12 08:34 /dev/mapper/HDD_E0_S12_975086927p2
brw-rw---- 1 grid asmadmin 253, 25 Dec  6 09:50 /dev/mapper/HDD_E0_S13_975069383
brw-rw---- 1 grid asmadmin 253, 35 Dec 12 08:59 /dev/mapper/HDD_E0_S13_975069383p1
brw-rw---- 1 grid asmadmin 253, 36 Dec 12 06:47 /dev/mapper/HDD_E0_S13_975069383p2
brw-rw---- 1 grid asmadmin 253, 20 Dec  6 09:49 /dev/mapper/HDD_E0_S16_977454443
brw-rw---- 1 grid asmadmin 253, 70 Dec 12 08:59 /dev/mapper/HDD_E0_S16_977454443p1
brw-rw---- 1 grid asmadmin 253, 71 Dec 12 08:48 /dev/mapper/HDD_E0_S16_977454443p2
brw-rw---- 1 grid asmadmin 253, 22 Dec  6 09:50 /dev/mapper/HDD_E0_S17_975084035
brw-rw---- 1 grid asmadmin 253, 31 Dec 12 08:59 /dev/mapper/HDD_E0_S17_975084035p1
brw-rw---- 1 grid asmadmin 253, 32 Dec 12 08:20 /dev/mapper/HDD_E0_S17_975084035p2
brw-rw---- 1 grid asmadmin 253,  4 Dec  6 09:50 /dev/mapper/HDD_E1_S02_979821467
brw-rw---- 1 grid asmadmin 253, 41 Dec 12 08:59 /dev/mapper/HDD_E1_S02_979821467p1
brw-rw---- 1 grid asmadmin 253, 42 Dec 12 08:59 /dev/mapper/HDD_E1_S02_979821467p2
brw-rw---- 1 grid asmadmin 253, 14 Dec  6 09:47 /dev/mapper/HDD_E1_S03_977580879
brw-rw---- 1 grid asmadmin 253, 39 Dec 12 08:59 /dev/mapper/HDD_E1_S03_977580879p1
brw-rw---- 1 grid asmadmin 253, 40 Dec 12 08:59 /dev/mapper/HDD_E1_S03_977580879p2
brw-rw---- 1 grid asmadmin 253,  5 Dec  6 09:48 /dev/mapper/HDD_E1_S06_977572915
brw-rw---- 1 grid asmadmin 253, 48 Dec 12 08:59 /dev/mapper/HDD_E1_S06_977572915p1
brw-rw---- 1 grid asmadmin 253, 49 Dec 12 08:00 /dev/mapper/HDD_E1_S06_977572915p2
brw-rw---- 1 grid asmadmin 253, 15 Dec  6 09:48 /dev/mapper/HDD_E1_S07_975059707
brw-rw---- 1 grid asmadmin 253, 59 Dec 12 08:59 /dev/mapper/HDD_E1_S07_975059707p1
brw-rw---- 1 grid asmadmin 253, 60 Dec 12 08:20 /dev/mapper/HDD_E1_S07_975059707p2
brw-rw---- 1 grid asmadmin 253,  6 Dec  6 09:48 /dev/mapper/HDD_E1_S10_977415523
brw-rw---- 1 grid asmadmin 253, 46 Dec 12 08:59 /dev/mapper/HDD_E1_S10_977415523p1
brw-rw---- 1 grid asmadmin 253, 47 Dec 12 08:15 /dev/mapper/HDD_E1_S10_977415523p2
brw-rw---- 1 grid asmadmin 253, 12 Dec  6 09:48 /dev/mapper/HDD_E1_S11_975067919
brw-rw---- 1 grid asmadmin 253, 55 Dec 12 08:59 /dev/mapper/HDD_E1_S11_975067919p1
brw-rw---- 1 grid asmadmin 253, 56 Dec 12 08:15 /dev/mapper/HDD_E1_S11_975067919p2
brw-rw---- 1 grid asmadmin 253,  7 Dec  6 09:49 /dev/mapper/HDD_E1_S14_979772987
brw-rw---- 1 grid asmadmin 253, 44 Dec 12 08:59 /dev/mapper/HDD_E1_S14_979772987p1
brw-rw---- 1 grid asmadmin 253, 45 Dec 12 07:17 /dev/mapper/HDD_E1_S14_979772987p2
brw-rw---- 1 grid asmadmin 253, 13 Dec  6 09:49 /dev/mapper/HDD_E1_S15_975079031
brw-rw---- 1 grid asmadmin 253, 57 Dec 12 08:59 /dev/mapper/HDD_E1_S15_975079031p1
brw-rw---- 1 grid asmadmin 253, 58 Dec 12 08:15 /dev/mapper/HDD_E1_S15_975079031p2
brw-rw---- 1 grid asmadmin 253,  8 Dec  6 09:50 /dev/mapper/HDD_E1_S18_975100927
brw-rw---- 1 grid asmadmin 253, 51 Dec 12 08:59 /dev/mapper/HDD_E1_S18_975100927p1
brw-rw---- 1 grid asmadmin 253, 52 Dec 12 07:04 /dev/mapper/HDD_E1_S18_975100927p2
brw-rw---- 1 grid asmadmin 253, 10 Dec  6 09:50 /dev/mapper/HDD_E1_S19_977425955
brw-rw---- 1 grid asmadmin 253, 53 Dec 12 08:59 /dev/mapper/HDD_E1_S19_977425955p1
brw-rw---- 1 grid asmadmin 253, 54 Dec 12 08:59 /dev/mapper/HDD_E1_S19_977425955p2
brw-rw---- 1 grid asmadmin 253, 21 Dec  6 09:49 /dev/mapper/SSD_E0_S20_805650385
brw-rw---- 1 grid asmadmin 253, 63 Dec 12 08:59 /dev/mapper/SSD_E0_S20_805650385p1
brw-rw---- 1 grid asmadmin 253, 23 Dec  6 09:49 /dev/mapper/SSD_E0_S21_805650582
brw-rw---- 1 grid asmadmin 253, 28 Dec 12 08:59 /dev/mapper/SSD_E0_S21_805650582p1
brw-rw---- 1 grid asmadmin 253,  9 Dec  6 09:50 /dev/mapper/SSD_E1_S22_805650581
brw-rw---- 1 grid asmadmin 253, 43 Dec 12 08:59 /dev/mapper/SSD_E1_S22_805650581p1
brw-rw---- 1 grid asmadmin 253, 11 Dec  6 09:50 /dev/mapper/SSD_E1_S23_805650633
brw-rw---- 1 grid asmadmin 253, 50 Dec 12 08:59 /dev/mapper/SSD_E1_S23_805650633p1
brw-rw---- 1 root disk     253,  2 Dec  5 17:38 /dev/mapper/VolGroupSys-LogVolOpt
brw-rw---- 1 root disk     253,  0 Dec  5 17:38 /dev/mapper/VolGroupSys-LogVolRoot
brw-rw---- 1 root disk     253,  3 Dec  5 17:38 /dev/mapper/VolGroupSys-LogVolSwap
brw-rw---- 1 root disk     253,  1 Dec  5 17:38 /dev/mapper/VolGroupSys-LogVolU01

As you can see, /dev/mapper/HDD_E1_S15_975079031 resides in expander 1, slot 15, and has 2 partitions (p1 and p2).  It is these disks that have been defined by device mapper that are added to ASM.  One thing that is gleaned from this is that the disks are actually partitioned, which is the opposite of how Oracle's other engineered database system (Exadata) handles disks.  While Exadata storage servers use the concept of physicaldisks, celldisks, and griddisks, the ODA goes back to the tried and true method of standard disk partitioning.  When configuring the ODA, you are given the choice of whether you would prefer backups to be internal or external to the ODA.  Naturally, choosing "internal" makes the RECO diskgroup larger, while the choice of "external" gives about an 85/15 split in favor of the DATA diskgroup.

We chose to install our ODA using the diskgroup sizing for external backups.  Here's what the installer created for us:

?View Code NONE
SQL&gt; @asm_diskgroups
 
Diskgroup  Sector Size AU Size (MB) State	Redundancy    Size (MB)    Free (MB)  Usable (MB)
---------- ----------- ------------ ----------- ---------- ------------ ------------ ------------
DATA		   512		  4 MOUNTED	HIGH	      9,830,400    9,630,088	3,210,029
RECO		   512		  4 MOUNTED	HIGH	      1,616,000    1,246,720	  415,573
REDO		   512		  4 MOUNTED	HIGH		280,016      254,844	   84,948

There was no choice in the redundancy....ODA uses high redundancy all the way. While it is possible to manually reconfigure the diskgroups to use normal redundancy, remember that losing failure groups in a normal redundancy configuration will take the diskgroup offline. Because Oracle does not include spare disks with the ODA, I recommend keeping high redundancy in the configuration. When we dig deeper into the ASM disk configuration we see that each disk is its own failgroup:

?View Code NONE
SQL&gt; @asm_disks
 
Diskgroup  Disk 				    Fail Group			   Size (MB)
---------- ---------------------------------------- ------------------------------ ---------
DATA	   /dev/mapper/HDD_E0_S00_977448283p1	    HDD_E0_S00_977448283P1	     491,520
DATA	   /dev/mapper/HDD_E0_S01_977561747p1	    HDD_E0_S01_977561747P1	     491,520
DATA	   /dev/mapper/HDD_E0_S04_975085971p1	    HDD_E0_S04_975085971P1	     491,520
DATA	   /dev/mapper/HDD_E0_S05_977762435p1	    HDD_E0_S05_977762435P1	     491,520
DATA	   /dev/mapper/HDD_E0_S08_975084479p1	    HDD_E0_S08_975084479P1	     491,520
DATA	   /dev/mapper/HDD_E0_S09_977562223p1	    HDD_E0_S09_977562223P1	     491,520
DATA	   /dev/mapper/HDD_E0_S12_975086927p1	    HDD_E0_S12_975086927P1	     491,520
DATA	   /dev/mapper/HDD_E0_S13_975069383p1	    HDD_E0_S13_975069383P1	     491,520
DATA	   /dev/mapper/HDD_E0_S16_977454443p1	    HDD_E0_S16_977454443P1	     491,520
DATA	   /dev/mapper/HDD_E0_S17_975084035p1	    HDD_E0_S17_975084035P1	     491,520
DATA	   /dev/mapper/HDD_E1_S02_979821467p1	    HDD_E1_S02_979821467P1	     491,520
DATA	   /dev/mapper/HDD_E1_S03_977580879p1	    HDD_E1_S03_977580879P1	     491,520
DATA	   /dev/mapper/HDD_E1_S06_977572915p1	    HDD_E1_S06_977572915P1	     491,520
DATA	   /dev/mapper/HDD_E1_S07_975059707p1	    HDD_E1_S07_975059707P1	     491,520
DATA	   /dev/mapper/HDD_E1_S10_977415523p1	    HDD_E1_S10_977415523P1	     491,520
DATA	   /dev/mapper/HDD_E1_S11_975067919p1	    HDD_E1_S11_975067919P1	     491,520
DATA	   /dev/mapper/HDD_E1_S14_979772987p1	    HDD_E1_S14_979772987P1	     491,520
DATA	   /dev/mapper/HDD_E1_S15_975079031p1	    HDD_E1_S15_975079031P1	     491,520
DATA	   /dev/mapper/HDD_E1_S18_975100927p1	    HDD_E1_S18_975100927P1	     491,520
DATA	   /dev/mapper/HDD_E1_S19_977425955p1	    HDD_E1_S19_977425955P1	     491,520
RECO	   /dev/mapper/HDD_E0_S00_977448283p2	    HDD_E0_S00_977448283P2	      80,800
RECO	   /dev/mapper/HDD_E0_S01_977561747p2	    HDD_E0_S01_977561747P2	      80,800
RECO	   /dev/mapper/HDD_E0_S04_975085971p2	    HDD_E0_S04_975085971P2	      80,800
RECO	   /dev/mapper/HDD_E0_S05_977762435p2	    HDD_E0_S05_977762435P2	      80,800
RECO	   /dev/mapper/HDD_E0_S08_975084479p2	    HDD_E0_S08_975084479P2	      80,800
RECO	   /dev/mapper/HDD_E0_S09_977562223p2	    HDD_E0_S09_977562223P2	      80,800
RECO	   /dev/mapper/HDD_E0_S12_975086927p2	    HDD_E0_S12_975086927P2	      80,800
RECO	   /dev/mapper/HDD_E0_S13_975069383p2	    HDD_E0_S13_975069383P2	      80,800
RECO	   /dev/mapper/HDD_E0_S16_977454443p2	    HDD_E0_S16_977454443P2	      80,800
RECO	   /dev/mapper/HDD_E0_S17_975084035p2	    HDD_E0_S17_975084035P2	      80,800
RECO	   /dev/mapper/HDD_E1_S02_979821467p2	    HDD_E1_S02_979821467P2	      80,800
RECO	   /dev/mapper/HDD_E1_S03_977580879p2	    HDD_E1_S03_977580879P2	      80,800
RECO	   /dev/mapper/HDD_E1_S06_977572915p2	    HDD_E1_S06_977572915P2	      80,800
RECO	   /dev/mapper/HDD_E1_S07_975059707p2	    HDD_E1_S07_975059707P2	      80,800
RECO	   /dev/mapper/HDD_E1_S10_977415523p2	    HDD_E1_S10_977415523P2	      80,800
RECO	   /dev/mapper/HDD_E1_S11_975067919p2	    HDD_E1_S11_975067919P2	      80,800
RECO	   /dev/mapper/HDD_E1_S14_979772987p2	    HDD_E1_S14_979772987P2	      80,800
RECO	   /dev/mapper/HDD_E1_S15_975079031p2	    HDD_E1_S15_975079031P2	      80,800
RECO	   /dev/mapper/HDD_E1_S18_975100927p2	    HDD_E1_S18_975100927P2	      80,800
RECO	   /dev/mapper/HDD_E1_S19_977425955p2	    HDD_E1_S19_977425955P2	      80,800
REDO	   /dev/mapper/SSD_E0_S20_805650385p1	    SSD_E0_S20_805650385P1	      70,004
REDO	   /dev/mapper/SSD_E0_S21_805650582p1	    SSD_E0_S21_805650582P1	      70,004
REDO	   /dev/mapper/SSD_E1_S22_805650581p1	    SSD_E1_S22_805650581P1	      70,004
REDO	   /dev/mapper/SSD_E1_S23_805650633p1	    SSD_E1_S23_805650633P1	      70,004
 
44 rows selected.

One other thing that I found surprising was that the sector size listed for the SSD was 512, not the 4096 that I expected. Normally, flash media has a 4k block size.  Looking at the attributes for the ASM diskgroups, there isn't anything special on the ODA:

?View Code NONE
SQL&gt; @asm_attributes
 
Diskgroup  Attribute			  Value
---------- ------------------------------ --------------------
DATA	   access_control.enabled	  FALSE
DATA	   access_control.umask 	  066
DATA	   au_size			  4194304
DATA	   cell.smart_scan_capable	  FALSE
DATA	   compatible.asm		  11.2.0.2.0
DATA	   compatible.rdbms		  11.2.0.2.0
DATA	   disk_repair_time		  3.6h
DATA	   idp.boundary 		  auto
DATA	   idp.type			  dynamic
DATA	   sector_size			  512
RECO	   access_control.enabled	  FALSE
RECO	   access_control.umask 	  066
RECO	   au_size			  4194304
RECO	   cell.smart_scan_capable	  FALSE
RECO	   compatible.advm		  11.2.0.0.0
RECO	   compatible.asm		  11.2.0.2.0
RECO	   compatible.rdbms		  11.2.0.2.0
RECO	   disk_repair_time		  3.6h
RECO	   idp.boundary 		  auto
RECO	   idp.type			  dynamic
RECO	   sector_size			  512
REDO	   access_control.enabled	  FALSE
REDO	   access_control.umask 	  066
REDO	   au_size			  4194304
REDO	   cell.smart_scan_capable	  FALSE
REDO	   compatible.asm		  11.2.0.2.0
REDO	   compatible.rdbms		  11.2.0.2.0
REDO	   disk_repair_time		  3.6h
REDO	   idp.boundary 		  auto
REDO	   idp.type			  dynamic
REDO	   sector_size			  512
 
31 rows selected.

The compatible parameters are set to 11.2.0.2, and cell.smart_scan_capable is set to false.  From this point, the ODA is just a standard RAC system that has direct-attached shared storage.

That's about it for now.  I'll be back soon with another post surrounding the usage of the new Oracle Appliance Kit (oakcli) that is used to manage storage and other components of the ODA.

]]> http://blog.oracle-ninja.com/2011/12/inside-the-oracle-database-appliance-%e2%80%93-part-2/feed/ 2 Exadata 11.2.2.4.0 10GbE Issue Resolved http://blog.oracle-ninja.com/2011/11/exadata-11-2-2-4-0-10gbe-issue-resolved/?utm_source=rss&utm_medium=rss&utm_campaign=exadata-11-2-2-4-0-10gbe-issue-resolved http://blog.oracle-ninja.com/2011/11/exadata-11-2-2-4-0-10gbe-issue-resolved/#comments Mon, 14 Nov 2011 22:56:23 +0000 Andy Colvin http://blog.oracle-ninja.com/?p=304 It appears that Oracle has resolved the issue with the 10GbE drivers that were introduced in version 11.2.2.4.0.  There is an updated note (1376664.1) that includes the patch to fix it. The issue was apparently related to TCP segmentation offloading, and can be fixed by installing the patch found in the note listed above. It does not require a reboot, and is similar to the fix for the IDT switch bug fixed in 11.2.2.4.0. Note that this bug only affected X2-2 systems utilizing 10 gigabit ethernet on the compute nodes. Oracle again recommends installing the 11.2.2.4.0 minimal pack on compute nodes.

After starting the service, users should see the following:

?View Code NONE
(root)# service disable10gigtso_13083530 start
Skipping igb interface eth0 using driver version 2.1.0-k2-1 - TSO disable unnecessary
Found ixgbe interface eth4 using driver version 2.0.84-k2 - Disabling TSO ... [SUCCESS]
Found ixgbe interface eth5 using driver version 2.0.84-k2 - Disabling TSO ... [SUCCESS]
]]> http://blog.oracle-ninja.com/2011/11/exadata-11-2-2-4-0-10gbe-issue-resolved/feed/ 1 Exadata Critical Issues with 11.2.2.4.0 Patch http://blog.oracle-ninja.com/2011/11/exadata-critical-issues-with-11-2-2-4-0-patch/?utm_source=rss&utm_medium=rss&utm_campaign=exadata-critical-issues-with-11-2-2-4-0-patch http://blog.oracle-ninja.com/2011/11/exadata-critical-issues-with-11-2-2-4-0-patch/#comments Tue, 08 Nov 2011 19:55:43 +0000 Andy Colvin http://blog.oracle-ninja.com/?p=287 Oracle has upgraded the supplemental note for 11.2.2.4.0 (1348647.1) with a handful of critical issues.  First, there is an issue mentioned by Vishal Gupta here regarding 10 gigabit ethernet on the database servers.  Currently, there is no workaround for this bug, and it is advised that customers utilizing 10 GbE on the database servers should stay on 11.2.2.3.5 for the minimal pack, while upgrading the storage servers to 11.2.2.4.0.

Additionally, there is a new issue that can be seen if the firmware on a disk fails to upgrade correctly during the patching process.  If the firmware does not update correctly, it is possible that cellsrv will drop the celldisk (and corresponding griddisks), causing a loss of data on those disks.  Before upgrading a cell to 11.2.2.4.0, check the status of your physical disks from cellcli with the following command:

?View Code NONE
cellcli -e 'list physicaldisk attributes luns where physicalInsertTime = null'

The command should return no output.  If it does, Oracle recommends to reboot each cell that returns output from this command.  After the reboot has completed, run the command again to verify that the disks are ready to be patched.  Here is the output from one of Enkitec's quarter rack systems (note that there is no output given, so the disks are ok to be upgraded):

?View Code NONE
[enkcel01:root] /root 
> dcli -g cell_group -l root cellcli -e 'list physicaldisk attributes luns where physicalInsertTime = null'
 
[enkcel01:root] /root 
>

Remember that these issues are not mentioned in the standard README files included in the 11.2.2.4.0 patch.  Before applying any Exadata Storage Server patches, always consult the supplemental note that is referenced in the supported versions for Exadata note (#888828.1), as these notes are updated after a patch has been released.

]]> http://blog.oracle-ninja.com/2011/11/exadata-critical-issues-with-11-2-2-4-0-patch/feed/ 1