I had the pleasure of rebuilding an Exadata rack for a customer a while back, and it provided a pretty good refresher in backup and recovery for me. As DBAs, we back up databases all the time, but the restores are performed much less frequently. In the case of this rack, there were several databases across multiple ASM diskgroups. One of the goals of the rebuild was to consolidate all of the databases into a single set of ASM diskgroups to better allow for future growth. In the example for this post, we'll use a database named TESTER that stores files in ASM diskgroups named +DATA2 and +RECO2. We'll take a backup, destroy the diskgroups entirely, and restore TESTER to diskgroups named +NEWDATA and +NEWRECO. Because this exercise was a planned operation, we used cold backups of the database to perform the restore. In most cases, you'll be performing point in time recovery on databases, and more information on that topic can be found in Oracle's documentation at http://docs.oracle.com/cd/E25054_01/backup.1111/e10642/rcmflash.htm.
In order to make matters more fun, let's say that the database has some datafiles that weren't created using Oracle's "Oracle Managed Files" (OMF) methodology. While definitely not the best way to do things, sometimes people forget that they're using ASM and want to create datafiles with static names. For the sake of argument, let's create a datafile named "BADIDEA" in +DATA2, and one named "REALLYBAD" in +RECO2.
SQL> select tablespace_name, file_name from dba_data_files order by 1;
TABLESPACE FILE_NAME
---------- --------------------------------------------------
SYSAUX +DATA2/tester/datafile/sysaux.260.867618419
SYSTEM +DATA2/tester/datafile/system.265.867618419
UNDOTBS1 +DATA2/tester/datafile/undotbs1.264.867618419
USERS +DATA2/tester/datafile/users.263.867618419
SQL> create tablespace BADIDEA datafile '+DATA2/tester/BADIDEA.DBF' size 500M autoextend on next 1M;
Tablespace created.
SQL> create tablespace REALLYBAD datafile '+RECO2/tester/REALLYBAD.DBF' size 500M autoextend on next 1M;
Tablespace created.
SQL> select tablespace_name, file_name from dba_data_files order by 1;
TABLESPACE FILE_NAME
---------- --------------------------------------------------
BADIDEA +DATA2/tester/badidea.dbf
REALLYBAD +RECO2/tester/reallybad.dbf
SYSAUX +DATA2/tester/datafile/sysaux.260.867618419
SYSTEM +DATA2/tester/datafile/system.265.867618419
UNDOTBS1 +DATA2/tester/datafile/undotbs1.264.867618419
USERS +DATA2/tester/datafile/users.263.867618419
6 rows selected.
While it may seem that the datafiles here aren't using OMF, if we look inside of ASM, we will see that they actually are, and the file names that I went out of my way to create are simply pointers to the OMF datafiles.
ASMCMD> ls -l data2/tester
Type Redund Striped Time Sys Name
N BADIDEA.DBF => +DATA2/TESTER/DATAFILE/BADIDEA.266.867619203
Y CONTROLFILE/
Y DATAFILE/
Y ONLINELOG/
Y PARAMETERFILE/
Y TEMPFILE/
N spfileTESTER.ora => +DATA2/TESTER/PARAMETERFILE/spfile.256.867618495
ASMCMD> ls -l data2/tester/datafile
Type Redund Striped Time Sys Name
DATAFILE UNPROT COARSE DEC 29 21:00:00 Y BADIDEA.266.867619203
DATAFILE UNPROT COARSE DEC 29 21:00:00 Y SYSAUX.260.867618419
DATAFILE UNPROT COARSE DEC 29 21:00:00 Y SYSTEM.265.867618419
DATAFILE UNPROT COARSE DEC 29 21:00:00 Y UNDOTBS1.264.867618419
DATAFILE UNPROT COARSE DEC 29 21:00:00 Y USERS.263.867618419
ASMCMD> ls -l reco2/tester
Type Redund Striped Time Sys Name
Y CONTROLFILE/
Y DATAFILE/
Y ONLINELOG/
N REALLYBAD.DBF => +RECO2/TESTER/DATAFILE/REALLYBAD.260.867619207
ASMCMD> ls -l reco2/tester/datafile
Type Redund Striped Time Sys Name
DATAFILE UNPROT COARSE DEC 29 21:00:00 Y REALLYBAD.260.867619207
Leandro Abite has an excellent script for looking at all files contained within ASM. If we run that here, we get the following output:
ASM File Name / Volume Name / Device Name Bytes Space File Type Creation Date Created?
--------------------------------------------------------------------------- ------------------ ------------------ ------------------ -------------------- --------
+DATA/ASM/ASMPARAMETERFILE/REGISTRY.253.866557445 1,536 4,194,304 ASMPARAMETERFILE 17-DEC-2014 14:24:04 Y
------------------ ------------------
1,536 4,194,304
+DATA2/TESTER/BADIDEA.DBF 524,296,192 526,385,152 DATAFILE 29-DEC-2014 21:20:02 N
+DATA2/TESTER/CONTROLFILE/Current.262.867618463 9,748,480 16,777,216 CONTROLFILE 29-DEC-2014 21:07:42 Y
+DATA2/TESTER/DATAFILE/BADIDEA.266.867619203 524,296,192 526,385,152 DATAFILE 29-DEC-2014 21:20:02 Y
+DATA2/TESTER/DATAFILE/SYSAUX.260.867618419 492,838,912 494,927,872 DATAFILE 29-DEC-2014 21:06:58 Y
+DATA2/TESTER/DATAFILE/SYSTEM.265.867618419 775,954,432 778,043,392 DATAFILE 29-DEC-2014 21:06:58 Y
+DATA2/TESTER/DATAFILE/UNDOTBS1.264.867618419 31,465,472 32,505,856 DATAFILE 29-DEC-2014 21:06:58 Y
+DATA2/TESTER/DATAFILE/USERS.263.867618419 5,251,072 6,291,456 DATAFILE 29-DEC-2014 21:06:58 Y
+DATA2/TESTER/ONLINELOG/group_1.261.867618465 52,429,312 53,477,376 ONLINELOG 29-DEC-2014 21:07:44 Y
+DATA2/TESTER/ONLINELOG/group_2.259.867618465 52,429,312 53,477,376 ONLINELOG 29-DEC-2014 21:07:44 Y
+DATA2/TESTER/ONLINELOG/group_3.258.867618465 52,429,312 53,477,376 ONLINELOG 29-DEC-2014 21:07:44 Y
+DATA2/TESTER/PARAMETERFILE/spfile.256.867618495 2,560 1,048,576 PARAMETERFILE 29-DEC-2014 21:08:15 Y
+DATA2/TESTER/TEMPFILE/TEMP.257.867618465 20,979,712 22,020,096 TEMPFILE 29-DEC-2014 21:07:45 Y
+DATA2/TESTER/spfileTESTER.ora 2,560 1,048,576 PARAMETERFILE 29-DEC-2014 21:08:15 N
------------------ ------------------
2,542,123,520 2,565,865,472
+RECO2/TESTER/CONTROLFILE/Current.256.867618463 9,748,480 16,777,216 CONTROLFILE 29-DEC-2014 21:07:42 Y
+RECO2/TESTER/DATAFILE/REALLYBAD.260.867619207 524,296,192 526,385,152 DATAFILE 29-DEC-2014 21:20:07 Y
+RECO2/TESTER/ONLINELOG/group_1.259.867618465 52,429,312 53,477,376 ONLINELOG 29-DEC-2014 21:07:44 Y
+RECO2/TESTER/ONLINELOG/group_2.258.867618465 52,429,312 53,477,376 ONLINELOG 29-DEC-2014 21:07:44 Y
+RECO2/TESTER/ONLINELOG/group_3.257.867618465 52,429,312 53,477,376 ONLINELOG 29-DEC-2014 21:07:44 Y
+RECO2/TESTER/REALLYBAD.DBF 524,296,192 526,385,152 DATAFILE 29-DEC-2014 21:20:07 N
------------------ ------------------
1,215,628,800 1,229,979,648
------------------ ------------------
Grand Total: 3,757,753,856 3,800,039,424
If you look closer at the non-OMF files that were created earlier, you will notice that they are listed twice in the output. The pointer is flagged as "N" in the final column (which represents automatically created files). Keep that in mind when looking at the space utilization...or just don't create non-OMF files on ASM. That's probably a better idea.
Anyway, on to the meat of the topic at hand. Because this was a planned backup/restore, I was smart enough to get backups of everything, including the password and parameter files. I created a pfile from the spfile that was in use, and copied that (along with the password file) over to my NFS share.
SQL> create pfile='/backup/tester/initTESTER.ora' from spfile;
File created.
SQL> !cp -ax $ORACLE_HOME/dbs/orapwTESTER /backup/tester/orapwTESTER
Next, it was time to take the database backup. I shut the database down, started it in a mounted state, and took a full backup including the controlfile. I'm using tee to make sure that everything is logged. This is important, because we will want to keep things like the DBID consistent after we restore the database. I'm including a "report schema" for good measure so that we can compare final results.
[oracle@localhost ~]$ rman target / | tee /backup/tester/backup_tester.log
Recovery Manager: Release 11.2.0.4.0 - Production on Mon Dec 29 21:41:40 2014
Copyright (c) 1982, 2011, Oracle and/or its affiliates. All rights reserved.
connected to target database: TESTER (DBID=3545964190)
RMAN> shutdown immediate;
using target database control file instead of recovery catalog
database closed
database dismounted
Oracle instance shut down
RMAN> startup mount;
connected to target database (not started)
Oracle instance started
database mounted
Total System Global Area 601272320 bytes
Fixed Size 2255592 bytes
Variable Size 230688024 bytes
Database Buffers 360710144 bytes
Redo Buffers 7618560 bytes
RMAN> run {
2> allocate channel d1 device type disk format '/backup/tester/tester_arch_%U';
3> backup archivelog all delete input;
4> release channel d1;
5> allocate channel d1 device type disk format '/backup/tester/tester_db_%U';
6> backup database include current controlfile;
7> release channel d1;
8> report schema;
9> }
allocated channel: d1
channel d1: SID=14 device type=DISK
Starting backup at 29-DEC-14
specification does not match any archived log in the repository
backup cancelled because there are no files to backup
Finished backup at 29-DEC-14
released channel: d1
allocated channel: d1
channel d1: SID=14 device type=DISK
Starting backup at 29-DEC-14
channel d1: starting full datafile backup set
channel d1: specifying datafile(s) in backup set
input datafile file number=00001 name=+DATA2/tester/datafile/system.265.867618419
input datafile file number=00005 name=+DATA2/tester/badidea.dbf
input datafile file number=00006 name=+RECO2/tester/reallybad.dbf
input datafile file number=00002 name=+DATA2/tester/datafile/sysaux.260.867618419
input datafile file number=00003 name=+DATA2/tester/datafile/undotbs1.264.867618419
input datafile file number=00004 name=+DATA2/tester/datafile/users.263.867618419
channel d1: starting piece 1 at 29-DEC-14
channel d1: finished piece 1 at 29-DEC-14
piece handle=/backup/tester/tester_db_01prdln0_1_1 tag=TAG20141229T214255 comment=NONE
channel d1: backup set complete, elapsed time: 00:00:15
channel d1: starting full datafile backup set
channel d1: specifying datafile(s) in backup set
including current control file in backup set
including current SPFILE in backup set
channel d1: starting piece 1 at 29-DEC-14
channel d1: finished piece 1 at 29-DEC-14
piece handle=/backup/tester/tester_db_02prdlnf_1_1 tag=TAG20141229T214255 comment=NONE
channel d1: backup set complete, elapsed time: 00:00:01
Finished backup at 29-DEC-14
released channel: d1
Report of database schema for database with db_unique_name TESTER
List of Permanent Datafiles
===========================
File Size(MB) Tablespace RB segs Datafile Name
---- -------- -------------------- ------- ------------------------
1 740 SYSTEM *** +DATA2/tester/datafile/system.265.867618419
2 470 SYSAUX *** +DATA2/tester/datafile/sysaux.260.867618419
3 30 UNDOTBS1 *** +DATA2/tester/datafile/undotbs1.264.867618419
4 5 USERS *** +DATA2/tester/datafile/users.263.867618419
5 500 BADIDEA *** +DATA2/tester/badidea.dbf
6 500 REALLYBAD *** +RECO2/tester/reallybad.dbf
List of Temporary Files
=======================
File Size(MB) Tablespace Maxsize(MB) Tempfile Name
---- -------- -------------------- ----------- --------------------
1 20 TEMP 32767 +DATA2/tester/tempfile/temp.257.867618465
Because this is a very small database with no actual contents, the backup script only uses one channel. The important things to know are that we have a consistent backup that includes the control file. Let's take a look at the backup sets that are now listed:
RMAN> list backupset;
using target database control file instead of recovery catalog
List of Backup Sets
===================
BS Key Type LV Size Device Type Elapsed Time Completion Time
------- ---- -- ---------- ----------- ------------ ---------------
1 Full 1016.39M DISK 00:00:09 29-DEC-14
BP Key: 1 Status: AVAILABLE Compressed: NO Tag: TAG20141229T214255
Piece Name: /backup/tester/tester_db_01prdln0_1_1
List of Datafiles in backup set 1
File LV Type Ckp SCN Ckp Time Name
---- -- ---- ---------- --------- ----
1 Full 933446 29-DEC-14 +DATA2/tester/datafile/system.265.867618419
2 Full 933446 29-DEC-14 +DATA2/tester/datafile/sysaux.260.867618419
3 Full 933446 29-DEC-14 +DATA2/tester/datafile/undotbs1.264.867618419
4 Full 933446 29-DEC-14 +DATA2/tester/datafile/users.263.867618419
5 Full 933446 29-DEC-14 +DATA2/tester/badidea.dbf
6 Full 933446 29-DEC-14 +RECO2/tester/reallybad.dbf
BS Key Type LV Size Device Type Elapsed Time Completion Time
------- ---- -- ---------- ----------- ------------ ---------------
2 Full 9.36M DISK 00:00:03 29-DEC-14
BP Key: 2 Status: AVAILABLE Compressed: NO Tag: TAG20141229T214255
Piece Name: /backup/tester/tester_db_02prdlnf_1_1
SPFILE Included: Modification time: 29-DEC-14
SPFILE db_unique_name: TESTER
Control File Included: Ckp SCN: 933446 Ckp time: 29-DEC-14
Backupset #1 contains all of the datafiles, and backupset #2 contains the control file. It's important to remember these piece names and have a backup listing when we perform the restore. Now that we have a good backup, it's time to trash the ASM diskgroups!
SQL> drop diskgroup DATA2 including contents;
Diskgroup dropped.
SQL> drop diskgroup RECO2 including contents;
Diskgroup dropped.
SQL> CREATE DISKGROUP NEWDATA
2 EXTERNAL REDUNDANCY
3 DISK '/dev/oracleasm/disks/ASM3'
4 ATTRIBUTE
5 'AU_SIZE'='1M',
6 'compatible.asm' = '11.2.0.2.0',
7 'compatible.rdbms' = '11.2.0.2.0'
8 /
Diskgroup created.
SQL> CREATE DISKGROUP NEWRECO
2 EXTERNAL REDUNDANCY
3 DISK '/dev/oracleasm/disks/ASM2'
4 ATTRIBUTE
5 'AU_SIZE'='1M',
6 'compatible.asm' = '11.2.0.2.0',
7 'compatible.rdbms' = '11.2.0.2.0'
8 /
Diskgroup created.
SQL> @asm_diskgroups
Diskgroup Sector Size AU Size (MB) State Redundancy Size (MB) Free (MB) Usable (MB)
---------- ----------- ------------ ----------- ---------- ------------ ------------ ------------
DATA 512 4 MOUNTED EXTERN 10,236 10,148
NEWDATA 512 1 MOUNTED EXTERN 10,236 10,184
NEWRECO 512 1 MOUNTED EXTERN 10,236 10,184
After dropping the diskgroups, I created new diskgroups, swapping the disks that were used (+DATA2 was on /dev/oracleasm/disks/ASM2 and +RECO2 was on /dev/oracleasm/disks/ASM3). We can see that there are now diskgroups named +NEWDATA and +NEWRECO.
At this point, we are ready to begin the database restore process. First, copy the password file back over to $ORACLE_HOME/dbs, and modify the control_files, db_create_file_dest, and db_recovery_file_dest parameters to reflect the new diskgroup names. Also, be sure to create the directory listed in audit_file_dest in case it was previously lost. Here's the full contents of my initTESTER.ora:
[oracle@localhost ~]$ cat /backup/tester/initTESTER.ora
TESTER.__db_cache_size=423624704
TESTER.__java_pool_size=4194304
TESTER.__large_pool_size=8388608
TESTER.__pga_aggregate_target=201326592
TESTER.__sga_target=603979776
TESTER.__shared_io_pool_size=0
TESTER.__shared_pool_size=155189248
TESTER.__streams_pool_size=0
*.audit_file_dest='/u01/app/oracle/admin/TESTER/adump'
*.audit_trail='db'
*.compatible='11.2.0.4.0'
*.control_files='+NEWDATA/tester/controlfile/current.262.867618463','+NEWRECO/tester/controlfile/current.256.867618463'
*.db_block_size=8192
*.db_create_file_dest='+NEWDATA'
*.db_domain=''
*.db_name='TESTER'
*.db_recovery_file_dest='+NEWRECO'
*.db_recovery_file_dest_size=4385144832
*.diagnostic_dest='/u01/app/oracle'
*.dispatchers='(PROTOCOL=TCP) (SERVICE=TESTERXDB)'
*.log_archive_format='%t_%s_%r.dbf'
*.open_cursors=300
*.pga_aggregate_target=201326592
*.processes=150
*.remote_login_passwordfile='EXCLUSIVE'
*.sga_target=603979776
*.undo_tablespace='UNDOTBS1'
I left the file names as-is in the control_files parameter setting and only changed the diskgroup names. When we restore the control file, it will generate new OMF file names.
Next, we can start the database in nomount state:
[oracle@localhost ~]$ rman target / | tee /backup/tester/restore1_tester.log
Recovery Manager: Release 11.2.0.4.0 - Production on Mon Dec 29 22:16:23 2014
Copyright (c) 1982, 2011, Oracle and/or its affiliates. All rights reserved.
connected to target database (not started)
RMAN> set dbid 3545964190;
executing command: SET DBID
RMAN> startup nomount pfile='/backup/tester/initTESTER.ora';
Oracle instance started
Total System Global Area 601272320 bytes
Fixed Size 2255592 bytes
Variable Size 230688024 bytes
Database Buffers 360710144 bytes
Redo Buffers 7618560 bytes
RMAN> restore controlfile from '/backup/tester/tester_db_02prdlnf_1_1';
Starting restore at 29-DEC-14
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=137 device type=DISK
channel ORA_DISK_1: restoring control file
channel ORA_DISK_1: restore complete, elapsed time: 00:00:01
output file name=+NEWDATA/tester/controlfile/current.256.867660489
output file name=+NEWRECO/tester/controlfile/current.256.867660489
Finished restore at 29-DEC-14
RMAN> exit
In the commands above, we set the DBID and start the database from the pfile that was created originally. Once the database was started in nomount state, the control file was restored to the +NEWDATA and +NEWRECO diskgroups. These values came from the pfile that was used to start the database. Next, take the names of the new control files and place those inside the pfile.
[oracle@localhost ~]$ grep control /backup/tester/initTESTER.ora
*.control_files='+NEWDATA/tester/controlfile/current.256.867660489','+NEWRECO/tester/controlfile/current.256.867660489'
Now that the correct control files are being used in the pfile, start the database again in a mounted state and restore the backups. Inside of the restore command, the "set newname for database" command is issued. This is the command that tells the database to use the new diskgroup names. First, we restart the database (mounted) using the modified pfile and issue the restore command.
[oracle@localhost ~]$ rman target / | tee /backup/tester/restore2_tester.log
Recovery Manager: Release 11.2.0.4.0 - Production on Tue Dec 30 08:55:56 2014
Copyright (c) 1982, 2011, Oracle and/or its affiliates. All rights reserved.
connected to target database: TESTER (not mounted)
RMAN> shutdown immediate;
using target database control file instead of recovery catalog
Oracle instance shut down
RMAN> startup mount pfile='/backup/tester/initTESTER.ora';
connected to target database (not started)
Oracle instance started
database mounted
Total System Global Area 601272320 bytes
Fixed Size 2255592 bytes
Variable Size 230688024 bytes
Database Buffers 360710144 bytes
Redo Buffers 7618560 bytes
RMAN> run {
2> set newname for database to '+NEWDATA';
3> restore database;
4> switch datafile all;
5> alter database open resetlogs;
6> }
executing command: SET NEWNAME
Starting restore at 30-DEC-14
Starting implicit crosscheck backup at 30-DEC-14
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=14 device type=DISK
Crosschecked 1 objects
Finished implicit crosscheck backup at 30-DEC-14
Starting implicit crosscheck copy at 30-DEC-14
using channel ORA_DISK_1
Finished implicit crosscheck copy at 30-DEC-14
searching for all files in the recovery area
cataloging files...
no files cataloged
using channel ORA_DISK_1
channel ORA_DISK_1: starting datafile backup set restore
channel ORA_DISK_1: specifying datafile(s) to restore from backup set
channel ORA_DISK_1: restoring datafile 00001 to +NEWDATA
channel ORA_DISK_1: restoring datafile 00002 to +NEWDATA
channel ORA_DISK_1: restoring datafile 00003 to +NEWDATA
channel ORA_DISK_1: restoring datafile 00004 to +NEWDATA
channel ORA_DISK_1: restoring datafile 00005 to +NEWDATA
channel ORA_DISK_1: restoring datafile 00006 to +NEWDATA
channel ORA_DISK_1: reading from backup piece /backup/tester/tester_db_01prdln0_1_1
channel ORA_DISK_1: piece handle=/backup/tester/tester_db_01prdln0_1_1 tag=TAG20141229T214255
channel ORA_DISK_1: restored backup piece 1
channel ORA_DISK_1: restore complete, elapsed time: 00:00:07
Finished restore at 30-DEC-14
datafile 1 switched to datafile copy
input datafile copy RECID=7 STAMP=867661005 file name=+NEWDATA/tester/datafile/system.257.867660999
datafile 2 switched to datafile copy
input datafile copy RECID=8 STAMP=867661005 file name=+NEWDATA/tester/datafile/sysaux.260.867660999
datafile 3 switched to datafile copy
input datafile copy RECID=9 STAMP=867661005 file name=+NEWDATA/tester/datafile/undotbs1.261.867660999
datafile 4 switched to datafile copy
input datafile copy RECID=10 STAMP=867661005 file name=+NEWDATA/tester/datafile/users.262.867660999
datafile 5 switched to datafile copy
input datafile copy RECID=11 STAMP=867661005 file name=+NEWDATA/tester/datafile/badidea.258.867660999
datafile 6 switched to datafile copy
input datafile copy RECID=12 STAMP=867661005 file name=+NEWDATA/tester/datafile/reallybad.259.867660999
database opened
RMAN>
About the alter database open resetlogs command - we have to open the database with resetlogs because the control files were restored from a backup. There is no other way to just open the database, even if a cold backup was taken. Further, the redo logs were lost when the diskgroups were destroyed. When the "alter database open resetlogs" command is issued, new redologs will be created in the desired location. Finally, create the spfile in ASM from the pfile that was originally used.
Final comment, please don't judge me for using ASMlib - it's a terrible habit that I'll shake from my VMs some day 🙂
“Final comment, please don’t judge me for using ASMlib” – you are not using it, you are using /dev/oracleasm/ paths instead of ORCL:* or maybe you wanted to say “don’t judge me for **not** using ASMLib”, either way you are getting some enemies 😉
Hi, Andy,
Nice writeup. Your example is using cold backup. I guess this is the reason you didn’t include “recover database” command in the recovery script. For large database, you might not have the luxury to do a cold backup in production. It might be a good idea to include this command, just to cover hot backup scenario.
Weidong