Point in time restore of an Oracle schema

I wanted to write on this topic in a previous post already. But then I stumbled upon an issue with read only tablspaces. So today I’ll cover the topic using R/W tablespaces only. The reason for writing this article is, that the majority of restore szenarios is not caused by infrastructure failures but by human errors. These happen during application updates, development procedures ore similar. That in turn means, that we have to cope with the content of the database rather than it’s phyiscal structure. Oracle RMAN offers methods to perform table point in time restores but often we need to restore a whole schema to a specific point in time due to constraints and depencies. RMAN has no method to this. But we can use tablspace point in time restore to perform something similar.

For this, I will use an auxiliary instance to perform the restore. That allows us to keep the original database and move data later on. Let’s walk throgh the process.

First, I prepare the auxiliary instance. For this method to work properly we need a connection to the original database and to the backups of this database.

[oracle@vm160 ~]$ export ORACLE_SID=aux
[oracle@vm160 ~]$ export ORACLE_HOME=/u01/app/oracle/product/db/19
[oracle@vm160 ~]$ vi $ORACLE_HOME/dbs/initaux.ora
[oracle@vm160 ~]$ cat $ORACLE_HOME/dbs/initaux.ora
db_name=aux
db_block_size=8192
db_create_file_dest='/u01/app/oracle/oradata'
db_recovery_file_dest='/u01/app/oracle/fra'
db_recovery_file_dest_size=10g
sga_target=1500M
enable_pluggable_database=true

Next, we start RMAN and prepare the auxilary (e.g. clone) database. We need a SPfile that we create from the init.ora that I just prepared.

oracle@vm160 ~]$ rman

Recovery Manager: Release 19.0.0.0.0 - Production on Mon May 13 13:40:19 2024
Version 19.18.0.0.0

Copyright (c) 1982, 2019, Oracle and/or its affiliates.  All rights reserved.

RMAN> connect target sys/R0b0tr0N##@vm160/orcl19.support.robotron.de

connected to target database: ORCL19 (DBID=422142402)

RMAN> connect auxiliary /

connected to auxiliary database (not started)

RMAN> startup clone nomount

Oracle instance started

Total System Global Area    1577055136 bytes

Fixed Size                     9163680 bytes
Variable Size                385875968 bytes
Database Buffers            1174405120 bytes
Redo Buffers                   7610368 bytes

RMAN> sql clone 'create spfile from pfile';

using target database control file instead of recovery catalog
sql statement: create spfile from pfile

RMAN> startup clone nomount force

Oracle instance started

Total System Global Area    1577055136 bytes

Fixed Size                     9163680 bytes
Variable Size                385875968 bytes
Database Buffers            1174405120 bytes
Redo Buffers                   7610368 bytes

What we will do now is a partial restore of the Pluggable Database that contains our schema to restore. But we skip all tablespaces, that do not have segments from the schema in question. Actually we do not do a restore but we use the DUPLICATE command which will perform all the required steps.

RMAN> duplicate database to aux pluggable database pdb2 skip tablespace pdb2:USERS until time "to_date('10.05.2024 08:00:00', 'dd.mm.yyyy hh24:mi:ss')";

Starting Duplicate Db at 2024-05-13 13:42:10
allocated channel: ORA_AUX_DISK_1
channel ORA_AUX_DISK_1: SID=181 device type=DISK
duplicating Online logs to Oracle Managed File (OMF) location
duplicating Datafiles to Oracle Managed File (OMF) location

contents of Memory Script:
{
   set until scn  49472124;
   sql clone "alter system set  control_files =
  ''/u01/app/oracle/oradata/AUX/controlfile/o1_mf_m43z4mm1_.ctl'', ''/u01/app/oracle/fra/AUX/controlfile/o1_mf_m43z4mm4_.ctl'' comment=
 ''Set by RMAN'' scope=spfile";
   sql clone "alter system set  db_name =
 ''ORCL19'' comment=
 ''Modified by RMAN duplicate'' scope=spfile";
   sql clone "alter system set  db_unique_name =
 ''aux'' comment=
 ''Modified by RMAN duplicate'' scope=spfile";
   shutdown clone immediate;
   startup clone force nomount
   restore clone primary controlfile;
   alter clone database mount;
}
executing Memory Script

executing command: SET until clause

sql statement: alter system set  control_files =   ''/u01/app/oracle/oradata/AUX/controlfile/o1_mf_m43z4mm1_.ctl'', ''/u01/app/oracle/fra/AUX/controlfile/o1_mf_m43z4mm4_.ctl'' comment= ''Set by RMAN'' scope=spfile

sql statement: alter system set  db_name =  ''ORCL19'' comment= ''Modified by RMAN duplicate'' scope=spfile

sql statement: alter system set  db_unique_name =  ''aux'' comment= ''Modified by RMAN duplicate'' scope=spfile

Oracle instance shut down

Oracle instance started

Total System Global Area    1577055136 bytes

Fixed Size                     9163680 bytes
Variable Size                385875968 bytes
Database Buffers            1174405120 bytes
Redo Buffers                   7610368 bytes

Starting restore at 2024-05-13 13:42:33
allocated channel: ORA_AUX_DISK_1
channel ORA_AUX_DISK_1: SID=4 device type=DISK

[...]


cataloged datafile copy
datafile copy file name=/u01/app/oracle/oradata/AUX/098BB26821426BADE063A024100AEA04/datafile/o1_mf_tbs_data_m43z6wbl_.dbf RECID=8 STAMP=1168868878
cataloged datafile copy
datafile copy file name=/u01/app/oracle/oradata/AUX/098BB26821426BADE063A024100AEA04/datafile/o1_mf_tbs_lob_m43z6wbh_.dbf RECID=9 STAMP=1168868878

datafile 31 switched to datafile copy
input datafile copy RECID=8 STAMP=1168868878 file name=/u01/app/oracle/oradata/AUX/098BB26821426BADE063A024100AEA04/datafile/o1_mf_tbs_data_m43z6wbl_.dbf

datafile 32 switched to datafile copy
input datafile copy RECID=9 STAMP=1168868878 file name=/u01/app/oracle/oradata/AUX/098BB26821426BADE063A024100AEA04/datafile/o1_mf_tbs_lob_m43z6wbh_.dbf

sql statement: alter tablespace  TBS_LOB online

sql statement: alter tablespace  TBS_DATA online
Dropping offline and skipped tablespaces
Executing: alter database default tablespace system
Executing: drop tablespace "USERS" including contents cascade constraints
Executing: drop tablespace "USERS" including contents cascade constraints
Finished Duplicate Db at 2024-05-13 13:48:22

Thah’s just it. We have a copy of the CDB as well as the PDB with only the tablespaces, that we need for our schema.

But since there is no possibility to rename the PDB, we need to be careful. When we do the duplication on the same host as the production database, we might get another instance registered for our PDB service.

[oracle@vm160 ~]$ lsnrctl status

[...]

Service "pdb2" has 2 instance(s).
  Instance "aux", status READY, has 1 handler(s) for this service...
  Instance "ORCL19INST", status READY, has 1 handler(s) for this service...

Such a situation is really dangereous because the listener will route new connections to one or the other instance. That means, we mix up the data from production and from the restored database.

One way to come around that, is to set the “db_domain” parameter for the auxiliary instance to something different then set in the originating database. Then we can distinguish between the production and the restored PDB.

[oracle@vm160 ~]$ lsnrctl status

[...]

Service "pdb2" has 1 instance(s).
  Instance "aux", status READY, has 1 handler(s) for this service...
Service "pdb2.support.robotron.de" has 1 instance(s).
  Instance "ORCL19INST", status READY, has 1 handler(s) for this service...

The last step would be to analyze the data and transport it back to production. This can be done by using Datapump for instance.

RMAN: Archivelogs, the UNTIL clause – expectations vs. reality

Sorry for the lengthy title, I just couldn’t put it shorter. So let’s start the story. Yesterday I was implementing a desaster recovery solution. The solution consists of a database copy, a process that transfers archivelogs from the original database and a process that applies the archivelogs to the copy. Dataguard would do that much better, but the system is running on Standard Edition 2, so no Dataguard at all. The solution worked quite well until I wanted to introduce an apply delay yesterday. That would mean, I still ship archivelogs on a regular basis but apply them only until a specific point in time. Assuming all archivelogs are present at the DR site, that is quite easy using RMAN:

RMAN> recover database until time 'sysdate-%s/1440';

The “%s” is filled with the desired apply delay in minutes. The fun starts with the housekeeping of the archivelogs. My asumption was, I can use the UNTIL clause to get rid of all the archivelogs, that are no longer required for recovery. Let’s first check the available archivelogs:

QL> select sequence#, first_time, next_time, first_change#, next_change# from v$archived_log where status='A' and first_change# >= 8582498175253 order by 1;

                 SEQUENCE# FIRST_TIME          NEXT_TIME                        FIRST_CHANGE#               NEXT_CHANGE#
-------------------------- ------------------- ------------------- -------------------------- --------------------------
                     16865 2024-04-08 10:26:16 2024-04-08 10:33:33              8582498176626              8582498177307

Just one archivelog. Let’s further asume, I already recovered until 10:30. So the archivelog is still needed for recovery since it ranges from 10:26 till 10:33. Now I run the RMAN DELETE command to remove the archivelogs, that are no longer needed. That should be none in this case. Of cause I use NOPROMPT FORCE because the command will be run by an automatic script.

MAN> delete  force noprompt archivelog until time "to_date('2024-04-08 10:30:00', 'yyyy-mm-dd hh24:mi:ss')";

allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=2 device type=DISK
archived log file name=/u01/app/oracle/standby/o1_mf_1_16865_m17byx6b_.arc thread=1 sequence=16865

But see, the archivelog was removed even though its next time is beyond my given timestamp. So I crosscheck that using a SCN in between. Of cause I made the archivelog available again for testing this testing purpose.

RMAN> delete  archivelog until scn 8582498177000;

allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=2 device type=DISK
archived log file name=/u01/app/oracle/standby/o1_mf_1_16865_m17byx6b_.arc thread=1 sequence=16865

And, surprise surprise, it also removed my archivelog. So what shall I do? Maybe reading the docs sheds some light on this issue.

Oracle Backup and Recovery Reference – archivelogRecordSpecifier

The table clearly states, that UNTIL SCN and UNTIL TIME both should use the FIRST_TIME to identify the archivelogs. If the FIRST_TIME “is less or equal” to the given timestamp, the archivelog falls into the range. So it is expected behaviour. The logic is good for restoring archivelogs because I will get all redo information that fall into the range. It is not as good for removing archivelogs because I will loose a little more redo information than specified.

What do we learn from that? Asumptions are a good starting point. Reading the docs is the next logical step. But if you want to make sure, things work as you want it to be, test it.

Update 2024-04-10

Changed the last paragraphs as I got really confused during testing and changed my understanding of the docs and the real behaviour several times. But that’s one thing, why writing is a good thing. You start thinking more deeply about things and do not stop thinking about it for some time.

Duplicating a PDB with Read Only tablespaces

I am a big fan of Oracle Recovery Manager (RMAN) since the very beginning. It evolved over time and makes the life of a DBA much easier, as long as you know the concepts. Todays blog post is about duplicating a database. Initially I wanted to perform a schema point in time restore which is not natively supported by RMAN. So let’s outline the system I have.

We are talking about an Oracle Database 19c, 19.18 to be precise. It is running the multitenant architecture and has 3 pluggable databases. I am using Oracle Managed Files (OMF) because this makes file handling easier for me. In one of these PDBs are several schemas spread over a couple of tablespaces. This is the tablespace layout:

SYS @ ORCL19:CDB$ROOT:>select con_id, tablespace_name, status from cdb_tablespaces;

    CON_ID TABLESPACE_NAME                STATUS
---------- ------------------------------ ---------
         3 SYSTEM                         ONLINE
         3 SYSAUX                         ONLINE
         3 UNDOTBS1                       ONLINE
         3 TEMP                           ONLINE
         3 USERS                          READ ONLY     <<<<<<<<
         3 TBS_DATA                       READ ONLY
         3 TBS_LOB                        READ ONLY
         3 TBS_ORG_UTPLSQL                ONLINE
         1 SYSTEM                         ONLINE
         1 SYSAUX                         ONLINE
         1 UNDOTBS1                       ONLINE
         1 TEMP                           ONLINE
         1 USERS                          ONLINE
         4 SYSTEM                         ONLINE
         4 SYSAUX                         ONLINE
         4 UNDOTBS1                       ONLINE
         4 TEMP                           ONLINE
         4 USERS                          ONLINE
         4 DATA                           ONLINE
         4 TBS_DATA                       READ ONLY
         4 TBS_LOB                        READ ONLY
         5 SYSTEM                         ONLINE
         5 SYSAUX                         ONLINE
         5 UNDOTBS1                       ONLINE
         5 TEMP                           ONLINE
         5 USERS                          ONLINE

I wanted to restore a specific tablespace, USERS in CON_ID 3, to some point in the past. Since I cannot use an automatic auxiliary instance for this, I created my own auxiliary instance. Not that complex, just create a parameterfile.

[oracle@vm160 ~]$ cat $ORACLE_HOME/dbs/initaux.ora
db_name=aux
db_block_size=8192
db_create_file_dest='/u01/app/oracle/oradata'
db_recovery_file_dest='/u01/app/oracle/fra'
db_recovery_file_dest_size=10g
sga_target=1500M
enable_pluggable_database=true

Having that, I started RMAN, connected to my target and my auxiliary database and started the duplicate. The DUPLICATE command allows us to include or exclude PDBs and/or skip specific tablespaces. Since I wanted to only restore one tablespace of one PDB, I combined these features. Furthermore I used the “NOOPEN” option to prevent the auxiliary instance from opening automatically after the duplication.

[oracle@vm160 ~]$ rman

Recovery Manager: Release 19.0.0.0.0 - Production on Thu Mar 21 08:21:41 2024
Version 19.18.0.0.0

Copyright (c) 1982, 2019, Oracle and/or its affiliates.  All rights reserved.

RMAN> connect target sys/********@vm160/orcl19.support.robotron.de

connected to target database: ORCL19 (DBID=422142402)

RMAN> connect auxiliary /

connected to auxiliary database (not started)

RMAN> startup clone nomount

Oracle instance started

Total System Global Area    1577055136 bytes

Fixed Size                     9163680 bytes
Variable Size                385875968 bytes
Database Buffers            1174405120 bytes
Redo Buffers                   7610368 bytes

RMAN> sql clone 'create spfile from pfile';

using target database control file instead of recovery catalog
sql statement: create spfile from pfile

RMAN> startup clone nomount force

Oracle instance started

Total System Global Area    1577055136 bytes

Fixed Size                     9163680 bytes
Variable Size                385875968 bytes
Database Buffers            1174405120 bytes
Redo Buffers                   7610368 bytes

RMAN> duplicate database to aux pluggable database pdb1 skip tablespace pdb1:TBS_DATA,pdb1:TBS_LOB,pdb1:TBS_ORG_UTPLSQL until time 'sysdate-1' noopen;

Starting Duplicate Db at 2024-03-21 08:22:30
allocated channel: ORA_AUX_DISK_1
channel ORA_AUX_DISK_1: SID=179 device type=DISK
duplicating Online logs to Oracle Managed File (OMF) location
duplicating Datafiles to Oracle Managed File (OMF) location

contents of Memory Script:
{
   set until scn  42779928;
   sql clone "alter system set  control_files =
  ''/u01/app/oracle/oradata/AUX/controlfile/o1_mf_lzqr1q5x_.ctl'', ''/u01/app/oracle/fra/AUX/controlfile/o1_mf_lzqr1q60_.ctl'' comment=
 ''Set by RMAN'' scope=spfile";
   sql clone "alter system set  db_name =
 ''ORCL19'' comment=
 ''Modified by RMAN duplicate'' scope=spfile";
   sql clone "alter system set  db_unique_name =
 ''aux'' comment=
 ''Modified by RMAN duplicate'' scope=spfile";
   shutdown clone immediate;
   startup clone force nomount
   restore clone primary controlfile;
   alter clone database mount;
}
executing Memory Script

executing command: SET until clause

sql statement: alter system set  control_files =   ''/u01/app/oracle/oradata/AUX/controlfile/o1_mf_lzqr1q5x_.ctl'', ''/u01/app/oracle/fra/AUX/controlfile/o1_mf_lzqr1q60_.ctl'' comment= ''Set by RMAN'' scope=spfile

sql statement: alter system set  db_name =  ''ORCL19'' comment= ''Modified by RMAN duplicate'' scope=spfile

sql statement: alter system set  db_unique_name =  ''aux'' comment= ''Modified by RMAN duplicate'' scope=spfile

Oracle instance shut down

Oracle instance started

Total System Global Area    1577055136 bytes

Fixed Size                     9163680 bytes
Variable Size                385875968 bytes
Database Buffers            1174405120 bytes
Redo Buffers                   7610368 bytes

Starting restore at 2024-03-21 08:22:53
allocated channel: ORA_AUX_DISK_1
channel ORA_AUX_DISK_1: SID=4 device type=DISK

channel ORA_AUX_DISK_1: starting datafile backup set restore
channel ORA_AUX_DISK_1: restoring control file
channel ORA_AUX_DISK_1: reading from backup piece /u01/app/oracle/fra/ORCL19/autobackup/2024_03_20/o1_mf_s_1164074408_lznf8rgr_.bkp
channel ORA_AUX_DISK_1: piece handle=/u01/app/oracle/fra/ORCL19/autobackup/2024_03_20/o1_mf_s_1164074408_lznf8rgr_.bkp tag=TAG20240320T020008
channel ORA_AUX_DISK_1: restored backup piece 1
channel ORA_AUX_DISK_1: restore complete, elapsed time: 00:00:01
output file name=/u01/app/oracle/oradata/AUX/controlfile/o1_mf_lzqr1q5x_.ctl
output file name=/u01/app/oracle/fra/AUX/controlfile/o1_mf_lzqr1q60_.ctl
Finished restore at 2024-03-21 08:22:55

database mounted
Skipping pluggable database ORCL19PDB2
Skipping pluggable database PDB2
Checking that duplicated tablespaces are self-contained
Automatically adding tablespace SYSTEM
Automatically adding tablespace SYSAUX
Automatically adding tablespace PDB$SEED:SYSTEM
Automatically adding tablespace PDB$SEED:SYSAUX
Automatically adding tablespace UNDOTBS1
Skipping tablespace USERS
Skipping tablespace PDB1:TBS_ORG_UTPLSQL
Skipping tablespace PDB1:TBS_LOB
Skipping tablespace PDB1:TBS_DATA

contents of Memory Script:
{
   set until scn  42779928;
   set newname for clone datafile  1 to new;
   set newname for clone datafile  3 to new;
   set newname for clone datafile  4 to new;
   set newname for clone datafile  5 to new;
   set newname for clone datafile  6 to new;
   set newname for clone datafile  8 to new;
   set newname for clone datafile  19 to new;
   set newname for clone datafile  20 to new;
   set newname for clone datafile  21 to new;
   set newname for clone datafile  22 to new;
   restore
   clone database
   skip forever tablespace  "USERS",
 "ORCL19PDB2":"USERS",
 "ORCL19PDB2":"UNDOTBS1",
 "ORCL19PDB2":"SYSTEM",
 "ORCL19PDB2":"SYSAUX",
 "PDB2":"USERS",
 "PDB2":"UNDOTBS1",
 "PDB2":"TBS_LOB",
 "PDB2":"TBS_DATA",
 "PDB2":"SYSTEM",
 "PDB2":"SYSAUX",
 "PDB2":"DATA",
 "PDB1":"TBS_ORG_UTPLSQL",
 "PDB1":"TBS_LOB",
 "PDB1":"TBS_DATA"   ;
}
executing Memory Script

executing command: SET until clause

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

Starting restore at 2024-03-21 08:23:13
using channel ORA_AUX_DISK_1

channel ORA_AUX_DISK_1: starting datafile backup set restore
channel ORA_AUX_DISK_1: specifying datafile(s) to restore from backup set
channel ORA_AUX_DISK_1: restoring datafile 00019 to /u01/app/oracle/oradata/AUX/ED7F830814134AADE053A024100A5D08/datafile/o1_mf_system_%u_.dbf
channel ORA_AUX_DISK_1: restoring datafile 00020 to /u01/app/oracle/oradata/AUX/ED7F830814134AADE053A024100A5D08/datafile/o1_mf_sysaux_%u_.dbf
channel ORA_AUX_DISK_1: restoring datafile 00021 to /u01/app/oracle/oradata/AUX/ED7F830814134AADE053A024100A5D08/datafile/o1_mf_undotbs1_%u_.dbf
channel ORA_AUX_DISK_1: restoring datafile 00022 to /u01/app/oracle/oradata/AUX/ED7F830814134AADE053A024100A5D08/datafile/o1_mf_users_%u_.dbf
channel ORA_AUX_DISK_1: reading from backup piece /u01/app/oracle/backup/orcl19/ORCL19_20240317_data_gh2lsmoo_1_1
channel ORA_AUX_DISK_1: piece handle=/u01/app/oracle/backup/orcl19/ORCL19_20240317_data_gh2lsmoo_1_1 tag=TAG20240317T010008
channel ORA_AUX_DISK_1: restored backup piece 1
channel ORA_AUX_DISK_1: restore complete, elapsed time: 00:00:15
channel ORA_AUX_DISK_1: starting datafile backup set restore
channel ORA_AUX_DISK_1: specifying datafile(s) to restore from backup set
channel ORA_AUX_DISK_1: restoring datafile 00001 to /u01/app/oracle/oradata/AUX/datafile/o1_mf_system_%u_.dbf
channel ORA_AUX_DISK_1: restoring datafile 00003 to /u01/app/oracle/oradata/AUX/datafile/o1_mf_sysaux_%u_.dbf
channel ORA_AUX_DISK_1: restoring datafile 00004 to /u01/app/oracle/oradata/AUX/datafile/o1_mf_undotbs1_%u_.dbf
channel ORA_AUX_DISK_1: reading from backup piece /u01/app/oracle/backup/orcl19/ORCL19_20240317_data_gj2lsmq0_1_1
channel ORA_AUX_DISK_1: piece handle=/u01/app/oracle/backup/orcl19/ORCL19_20240317_data_gj2lsmq0_1_1 tag=TAG20240317T010008
channel ORA_AUX_DISK_1: restored backup piece 1
channel ORA_AUX_DISK_1: restore complete, elapsed time: 00:00:15
channel ORA_AUX_DISK_1: starting datafile backup set restore
channel ORA_AUX_DISK_1: specifying datafile(s) to restore from backup set
channel ORA_AUX_DISK_1: restoring datafile 00005 to /u01/app/oracle/oradata/AUX/ED7F22D941443E9AE053A024100AD0FB/datafile/o1_mf_system_%u_.dbf
channel ORA_AUX_DISK_1: restoring datafile 00006 to /u01/app/oracle/oradata/AUX/ED7F22D941443E9AE053A024100AD0FB/datafile/o1_mf_sysaux_%u_.dbf
channel ORA_AUX_DISK_1: restoring datafile 00008 to /u01/app/oracle/oradata/AUX/ED7F22D941443E9AE053A024100AD0FB/datafile/o1_mf_undotbs1_%u_.dbf
channel ORA_AUX_DISK_1: reading from backup piece /u01/app/oracle/backup/orcl19/ORCL19_20240317_data_gl2lsmqn_1_1
channel ORA_AUX_DISK_1: piece handle=/u01/app/oracle/backup/orcl19/ORCL19_20240317_data_gl2lsmqn_1_1 tag=TAG20240317T010008
channel ORA_AUX_DISK_1: restored backup piece 1
channel ORA_AUX_DISK_1: restore complete, elapsed time: 00:00:07
Finished restore at 2024-03-21 08:23:51

contents of Memory Script:
{
   switch clone datafile all;
}
executing Memory Script

datafile 1 switched to datafile copy
input datafile copy RECID=14 STAMP=1164183831 file name=/u01/app/oracle/oradata/AUX/datafile/o1_mf_system_lzqr3jxq_.dbf
datafile 3 switched to datafile copy
input datafile copy RECID=15 STAMP=1164183831 file name=/u01/app/oracle/oradata/AUX/datafile/o1_mf_sysaux_lzqr3jwy_.dbf
datafile 4 switched to datafile copy
input datafile copy RECID=16 STAMP=1164183831 file name=/u01/app/oracle/oradata/AUX/datafile/o1_mf_undotbs1_lzqr3jy4_.dbf
datafile 5 switched to datafile copy
input datafile copy RECID=17 STAMP=1164183831 file name=/u01/app/oracle/oradata/AUX/ED7F22D941443E9AE053A024100AD0FB/datafile/o1_mf_system_lzqr400z_.dbf
datafile 6 switched to datafile copy
input datafile copy RECID=18 STAMP=1164183831 file name=/u01/app/oracle/oradata/AUX/ED7F22D941443E9AE053A024100AD0FB/datafile/o1_mf_sysaux_lzqr4014_.dbf
datafile 8 switched to datafile copy
input datafile copy RECID=19 STAMP=1164183831 file name=/u01/app/oracle/oradata/AUX/ED7F22D941443E9AE053A024100AD0FB/datafile/o1_mf_undotbs1_lzqr401g_.dbf
datafile 19 switched to datafile copy
input datafile copy RECID=20 STAMP=1164183831 file name=/u01/app/oracle/oradata/AUX/ED7F830814134AADE053A024100A5D08/datafile/o1_mf_system_lzqr31r4_.dbf
datafile 20 switched to datafile copy
input datafile copy RECID=21 STAMP=1164183831 file name=/u01/app/oracle/oradata/AUX/ED7F830814134AADE053A024100A5D08/datafile/o1_mf_sysaux_lzqr31rf_.dbf
datafile 21 switched to datafile copy
input datafile copy RECID=22 STAMP=1164183831 file name=/u01/app/oracle/oradata/AUX/ED7F830814134AADE053A024100A5D08/datafile/o1_mf_undotbs1_lzqr31rq_.dbf
datafile 22 switched to datafile copy
input datafile copy RECID=23 STAMP=1164183831 file name=/u01/app/oracle/oradata/AUX/ED7F830814134AADE053A024100A5D08/datafile/o1_mf_users_lzqr31r2_.dbf

contents of Memory Script:
{
   set until time  "to_date('2024/03/20 08:22:30', 'YYYY/MM/DD HH24:MI:SS')";
   recover
   clone database
   skip forever tablespace  "USERS",
 "ORCL19PDB2":"USERS",
 "ORCL19PDB2":"UNDOTBS1",
 "ORCL19PDB2":"SYSTEM",
 "ORCL19PDB2":"SYSAUX",
 "PDB2":"USERS",
 "PDB2":"UNDOTBS1",
 "PDB2":"TBS_LOB",
 "PDB2":"TBS_DATA",
 "PDB2":"SYSTEM",
 "PDB2":"SYSAUX",
 "PDB2":"DATA",
 "PDB1":"TBS_ORG_UTPLSQL",
 "PDB1":"TBS_LOB",
 "PDB1":"TBS_DATA"    delete archivelog
   ;
}
executing Memory Script

executing command: SET until clause

Starting recover at 2024-03-21 08:23:51
using channel ORA_AUX_DISK_1
datafile 22 not processed because file is read-only

Executing: alter database datafile 7 offline drop
Executing: alter database datafile 29, 30, 33 offline drop
Executing: alter database datafile 24, 25, 26, 27, 28, 31, 32 offline drop
Executing: alter database datafile 15, 16, 17, 23 offline drop
starting media recovery

channel ORA_AUX_DISK_1: starting archived log restore to default destination
channel ORA_AUX_DISK_1: restoring archived log
archived log thread=1 sequence=234
channel ORA_AUX_DISK_1: reading from backup piece /u01/app/oracle/backup/orcl19/ORCL19_20240317_arch_gm2lsmqq_1_1
channel ORA_AUX_DISK_1: piece handle=/u01/app/oracle/backup/orcl19/ORCL19_20240317_arch_gm2lsmqq_1_1 tag=TAG20240317T010114
channel ORA_AUX_DISK_1: restored backup piece 1
channel ORA_AUX_DISK_1: restore complete, elapsed time: 00:00:01
archived log file name=/u01/app/oracle/fra/AUX/archivelog/2024_03_21/o1_mf_1_234_lzqr4b89_.arc thread=1 sequence=234
channel clone_default: deleting archived log(s)
archived log file name=/u01/app/oracle/fra/AUX/archivelog/2024_03_21/o1_mf_1_234_lzqr4b89_.arc RECID=726 STAMP=1164183834
channel ORA_AUX_DISK_1: starting archived log restore to default destination
channel ORA_AUX_DISK_1: restoring archived log
archived log thread=1 sequence=235
channel ORA_AUX_DISK_1: reading from backup piece /u01/app/oracle/backup/orcl19/ORCL19_20240317_arch_go2lsq94_1_1
channel ORA_AUX_DISK_1: piece handle=/u01/app/oracle/backup/orcl19/ORCL19_20240317_arch_go2lsq94_1_1 tag=TAG20240317T020004
channel ORA_AUX_DISK_1: restored backup piece 1
channel ORA_AUX_DISK_1: restore complete, elapsed time: 00:00:01
archived log file name=/u01/app/oracle/fra/AUX/archivelog/2024_03_21/o1_mf_1_235_lzqr4ck8_.arc thread=1 sequence=235
channel clone_default: deleting archived log(s)
archived log file name=/u01/app/oracle/fra/AUX/archivelog/2024_03_21/o1_mf_1_235_lzqr4ck8_.arc RECID=727 STAMP=1164183835
channel ORA_AUX_DISK_1: starting archived log restore to default destination
channel ORA_AUX_DISK_1: restoring archived log
archived log thread=1 sequence=236
channel ORA_AUX_DISK_1: restoring archived log
archived log thread=1 sequence=237
channel ORA_AUX_DISK_1: restoring archived log
archived log thread=1 sequence=238
channel ORA_AUX_DISK_1: reading from backup piece /u01/app/oracle/backup/orcl19/ORCL19_20240318_arch_gq2lvel5_1_1
channel ORA_AUX_DISK_1: piece handle=/u01/app/oracle/backup/orcl19/ORCL19_20240318_arch_gq2lvel5_1_1 tag=TAG20240318T020005
channel ORA_AUX_DISK_1: restored backup piece 1
channel ORA_AUX_DISK_1: restore complete, elapsed time: 00:00:03
archived log file name=/u01/app/oracle/fra/AUX/archivelog/2024_03_21/o1_mf_1_236_lzqr4gz1_.arc thread=1 sequence=236
channel clone_default: deleting archived log(s)
archived log file name=/u01/app/oracle/fra/AUX/archivelog/2024_03_21/o1_mf_1_236_lzqr4gz1_.arc RECID=729 STAMP=1164183840
archived log file name=/u01/app/oracle/fra/AUX/archivelog/2024_03_21/o1_mf_1_237_lzqr4gys_.arc thread=1 sequence=237
channel clone_default: deleting archived log(s)
archived log file name=/u01/app/oracle/fra/AUX/archivelog/2024_03_21/o1_mf_1_237_lzqr4gys_.arc RECID=730 STAMP=1164183840
archived log file name=/u01/app/oracle/fra/AUX/archivelog/2024_03_21/o1_mf_1_238_lzqr4h00_.arc thread=1 sequence=238
channel clone_default: deleting archived log(s)
archived log file name=/u01/app/oracle/fra/AUX/archivelog/2024_03_21/o1_mf_1_238_lzqr4h00_.arc RECID=728 STAMP=1164183840
channel ORA_AUX_DISK_1: starting archived log restore to default destination
channel ORA_AUX_DISK_1: restoring archived log
archived log thread=1 sequence=239
channel ORA_AUX_DISK_1: restoring archived log
archived log thread=1 sequence=240
channel ORA_AUX_DISK_1: restoring archived log
archived log thread=1 sequence=241
channel ORA_AUX_DISK_1: restoring archived log
archived log thread=1 sequence=242
channel ORA_AUX_DISK_1: reading from backup piece /u01/app/oracle/backup/orcl19/ORCL19_20240319_arch_gs2m2315_1_1
channel ORA_AUX_DISK_1: piece handle=/u01/app/oracle/backup/orcl19/ORCL19_20240319_arch_gs2m2315_1_1 tag=TAG20240319T020005
channel ORA_AUX_DISK_1: restored backup piece 1
channel ORA_AUX_DISK_1: restore complete, elapsed time: 00:00:03
archived log file name=/u01/app/oracle/fra/AUX/archivelog/2024_03_21/o1_mf_1_239_lzqr55dj_.arc thread=1 sequence=239
channel clone_default: deleting archived log(s)
archived log file name=/u01/app/oracle/fra/AUX/archivelog/2024_03_21/o1_mf_1_239_lzqr55dj_.arc RECID=733 STAMP=1164183862
archived log file name=/u01/app/oracle/fra/AUX/archivelog/2024_03_21/o1_mf_1_240_lzqr55f6_.arc thread=1 sequence=240
channel clone_default: deleting archived log(s)
archived log file name=/u01/app/oracle/fra/AUX/archivelog/2024_03_21/o1_mf_1_240_lzqr55f6_.arc RECID=731 STAMP=1164183861
archived log file name=/u01/app/oracle/fra/AUX/archivelog/2024_03_21/o1_mf_1_241_lzqr55d4_.arc thread=1 sequence=241
channel clone_default: deleting archived log(s)
archived log file name=/u01/app/oracle/fra/AUX/archivelog/2024_03_21/o1_mf_1_241_lzqr55d4_.arc RECID=734 STAMP=1164183862
archived log file name=/u01/app/oracle/fra/AUX/archivelog/2024_03_21/o1_mf_1_242_lzqr55ds_.arc thread=1 sequence=242
channel clone_default: deleting archived log(s)
archived log file name=/u01/app/oracle/fra/AUX/archivelog/2024_03_21/o1_mf_1_242_lzqr55ds_.arc RECID=732 STAMP=1164183861
channel ORA_AUX_DISK_1: starting archived log restore to default destination
channel ORA_AUX_DISK_1: restoring archived log
archived log thread=1 sequence=243
channel ORA_AUX_DISK_1: restoring archived log
archived log thread=1 sequence=244
channel ORA_AUX_DISK_1: reading from backup piece /u01/app/oracle/backup/orcl19/ORCL19_20240320_arch_gu2m4nd7_1_1
channel ORA_AUX_DISK_1: piece handle=/u01/app/oracle/backup/orcl19/ORCL19_20240320_arch_gu2m4nd7_1_1 tag=TAG20240320T020007
channel ORA_AUX_DISK_1: restored backup piece 1
channel ORA_AUX_DISK_1: restore complete, elapsed time: 00:00:03
archived log file name=/u01/app/oracle/fra/AUX/archivelog/2024_03_21/o1_mf_1_243_lzqr5nvb_.arc thread=1 sequence=243
channel clone_default: deleting archived log(s)
archived log file name=/u01/app/oracle/fra/AUX/archivelog/2024_03_21/o1_mf_1_243_lzqr5nvb_.arc RECID=736 STAMP=1164183878
archived log file name=/u01/app/oracle/fra/AUX/archivelog/2024_03_21/o1_mf_1_244_lzqr5nvn_.arc thread=1 sequence=244
channel clone_default: deleting archived log(s)
archived log file name=/u01/app/oracle/fra/AUX/archivelog/2024_03_21/o1_mf_1_244_lzqr5nvn_.arc RECID=735 STAMP=1164183877
channel ORA_AUX_DISK_1: starting archived log restore to default destination
channel ORA_AUX_DISK_1: restoring archived log
archived log thread=1 sequence=245
channel ORA_AUX_DISK_1: reading from backup piece /u01/app/oracle/backup/orcl19/ORCL19_20240321_arch_h02m7bp5_1_1
channel ORA_AUX_DISK_1: piece handle=/u01/app/oracle/backup/orcl19/ORCL19_20240321_arch_h02m7bp5_1_1 tag=TAG20240321T020005
channel ORA_AUX_DISK_1: restored backup piece 1
channel ORA_AUX_DISK_1: restore complete, elapsed time: 00:00:01
archived log file name=/u01/app/oracle/fra/AUX/archivelog/2024_03_21/o1_mf_1_245_lzqr6354_.arc thread=1 sequence=245
channel clone_default: deleting archived log(s)
archived log file name=/u01/app/oracle/fra/AUX/archivelog/2024_03_21/o1_mf_1_245_lzqr6354_.arc RECID=737 STAMP=1164183891
media recovery complete, elapsed time: 00:00:02
Finished recover at 2024-03-21 08:24:55
Oracle instance started

Total System Global Area    1577055136 bytes

Fixed Size                     9163680 bytes
Variable Size                385875968 bytes
Database Buffers            1174405120 bytes
Redo Buffers                   7610368 bytes

contents of Memory Script:
{
   sql clone "alter system set  db_name =
 ''AUX'' comment=
 ''Reset to original value by RMAN'' scope=spfile";
   sql clone "alter system reset  db_unique_name scope=spfile";
}
executing Memory Script

sql statement: alter system set  db_name =  ''AUX'' comment= ''Reset to original value by RMAN'' scope=spfile

sql statement: alter system reset  db_unique_name scope=spfile
Oracle instance started

Total System Global Area    1577055136 bytes

Fixed Size                     9163680 bytes
Variable Size                385875968 bytes
Database Buffers            1174405120 bytes
Redo Buffers                   7610368 bytes
sql statement: CREATE CONTROLFILE REUSE SET DATABASE "AUX" RESETLOGS ARCHIVELOG
  MAXLOGFILES     16
  MAXLOGMEMBERS      3
  MAXDATAFILES     1024
  MAXINSTANCES     8
  MAXLOGHISTORY      292
 LOGFILE
  GROUP     1  SIZE 200 M ,
  GROUP     2  SIZE 200 M ,
  GROUP     3  SIZE 200 M
 DATAFILE
  '/u01/app/oracle/oradata/AUX/datafile/o1_mf_system_lzqr3jxq_.dbf',
  '/u01/app/oracle/oradata/AUX/ED7F22D941443E9AE053A024100AD0FB/datafile/o1_mf_system_lzqr400z_.dbf',
  '/u01/app/oracle/oradata/AUX/ED7F830814134AADE053A024100A5D08/datafile/o1_mf_system_lzqr31r4_.dbf'
 CHARACTER SET AL32UTF8


contents of Memory Script:
{
   set newname for clone tempfile  1 to new;
   set newname for clone tempfile  2 to new;
   set newname for clone tempfile  3 to new;
   switch clone tempfile all;
   catalog clone datafilecopy  "/u01/app/oracle/oradata/AUX/datafile/o1_mf_sysaux_lzqr3jwy_.dbf",
 "/u01/app/oracle/oradata/AUX/datafile/o1_mf_undotbs1_lzqr3jy4_.dbf",
 "/u01/app/oracle/oradata/AUX/ED7F22D941443E9AE053A024100AD0FB/datafile/o1_mf_sysaux_lzqr4014_.dbf",
 "/u01/app/oracle/oradata/AUX/ED7F22D941443E9AE053A024100AD0FB/datafile/o1_mf_undotbs1_lzqr401g_.dbf",
 "/u01/app/oracle/oradata/AUX/ED7F830814134AADE053A024100A5D08/datafile/o1_mf_sysaux_lzqr31rf_.dbf",
 "/u01/app/oracle/oradata/AUX/ED7F830814134AADE053A024100A5D08/datafile/o1_mf_undotbs1_lzqr31rq_.dbf";
   switch clone datafile all;
}
executing Memory Script

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

renamed tempfile 1 to /u01/app/oracle/oradata/AUX/datafile/o1_mf_temp_%u_.tmp in control file
renamed tempfile 2 to /u01/app/oracle/oradata/AUX/ED7F22D941443E9AE053A024100AD0FB/datafile/o1_mf_temp_%u_.tmp in control file
renamed tempfile 3 to /u01/app/oracle/oradata/AUX/ED7F830814134AADE053A024100A5D08/datafile/o1_mf_temp_%u_.tmp in control file

cataloged datafile copy
datafile copy file name=/u01/app/oracle/oradata/AUX/datafile/o1_mf_sysaux_lzqr3jwy_.dbf RECID=1 STAMP=1164183915
cataloged datafile copy
datafile copy file name=/u01/app/oracle/oradata/AUX/datafile/o1_mf_undotbs1_lzqr3jy4_.dbf RECID=2 STAMP=1164183915
cataloged datafile copy
datafile copy file name=/u01/app/oracle/oradata/AUX/ED7F22D941443E9AE053A024100AD0FB/datafile/o1_mf_sysaux_lzqr4014_.dbf RECID=3 STAMP=1164183915
cataloged datafile copy
datafile copy file name=/u01/app/oracle/oradata/AUX/ED7F22D941443E9AE053A024100AD0FB/datafile/o1_mf_undotbs1_lzqr401g_.dbf RECID=4 STAMP=1164183915
cataloged datafile copy
datafile copy file name=/u01/app/oracle/oradata/AUX/ED7F830814134AADE053A024100A5D08/datafile/o1_mf_sysaux_lzqr31rf_.dbf RECID=5 STAMP=1164183915
cataloged datafile copy
datafile copy file name=/u01/app/oracle/oradata/AUX/ED7F830814134AADE053A024100A5D08/datafile/o1_mf_undotbs1_lzqr31rq_.dbf RECID=6 STAMP=1164183915

datafile 3 switched to datafile copy
input datafile copy RECID=1 STAMP=1164183915 file name=/u01/app/oracle/oradata/AUX/datafile/o1_mf_sysaux_lzqr3jwy_.dbf
datafile 4 switched to datafile copy
input datafile copy RECID=2 STAMP=1164183915 file name=/u01/app/oracle/oradata/AUX/datafile/o1_mf_undotbs1_lzqr3jy4_.dbf
datafile 6 switched to datafile copy
input datafile copy RECID=3 STAMP=1164183915 file name=/u01/app/oracle/oradata/AUX/ED7F22D941443E9AE053A024100AD0FB/datafile/o1_mf_sysaux_lzqr4014_.dbf
datafile 8 switched to datafile copy
input datafile copy RECID=4 STAMP=1164183915 file name=/u01/app/oracle/oradata/AUX/ED7F22D941443E9AE053A024100AD0FB/datafile/o1_mf_undotbs1_lzqr401g_.dbf
datafile 20 switched to datafile copy
input datafile copy RECID=5 STAMP=1164183915 file name=/u01/app/oracle/oradata/AUX/ED7F830814134AADE053A024100A5D08/datafile/o1_mf_sysaux_lzqr31rf_.dbf
datafile 21 switched to datafile copy
input datafile copy RECID=6 STAMP=1164183915 file name=/u01/app/oracle/oradata/AUX/ED7F830814134AADE053A024100A5D08/datafile/o1_mf_undotbs1_lzqr31rq_.dbf
Leaving database unopened, as requested
Finished Duplicate Db at 2024-03-21 08:25:15

So far, so good. See line 180, my datafile got restored. It was also switched to after restoration, see line 229. But then, in line 260, it was skipped for the recovery part. That sounds not like a big deal since the file / the tablespace is read only. But at the end you see, that neither the CREATE CONTROLFILE nor the CATALOG and SWITCH part contains my datafile 22. Strange. Let’s see, what we have at this point.

SYS @ AUX:CDB$ROOT:>select file#, name from v$datafile;

     FILE# NAME
---------- ------------------------------------------------------------------------------------------------------------------------
         1 /u01/app/oracle/oradata/AUX/datafile/o1_mf_system_lzqr3jxq_.dbf
         3 /u01/app/oracle/oradata/AUX/datafile/o1_mf_sysaux_lzqr3jwy_.dbf
         4 /u01/app/oracle/oradata/AUX/datafile/o1_mf_undotbs1_lzqr3jy4_.dbf
         5 /u01/app/oracle/oradata/AUX/ED7F22D941443E9AE053A024100AD0FB/datafile/o1_mf_system_lzqr400z_.dbf
         6 /u01/app/oracle/oradata/AUX/ED7F22D941443E9AE053A024100AD0FB/datafile/o1_mf_sysaux_lzqr4014_.dbf
         8 /u01/app/oracle/oradata/AUX/ED7F22D941443E9AE053A024100AD0FB/datafile/o1_mf_undotbs1_lzqr401g_.dbf
        19 /u01/app/oracle/oradata/AUX/ED7F830814134AADE053A024100A5D08/datafile/o1_mf_system_lzqr31r4_.dbf
        20 /u01/app/oracle/oradata/AUX/ED7F830814134AADE053A024100A5D08/datafile/o1_mf_sysaux_lzqr31rf_.dbf
        21 /u01/app/oracle/oradata/AUX/ED7F830814134AADE053A024100A5D08/datafile/o1_mf_undotbs1_lzqr31rq_.dbf

9 rows selected.

SYS @ AUX:CDB$ROOT:>show pdbs

    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         2 PDB$SEED                       MOUNTED
         3 _###_UNKNOWN_PDB_#_3           MOUNTED
SYS @ AUX:CDB$ROOT:>select status from v$instance;

STATUS
------------
MOUNTED

As you can see, no file 22 is there. So let’s go ahead and open the database read only just to check things.

SYS @ AUX:CDB$ROOT:>alter database open read only;
alter database open read only
*
ERROR at line 1:
ORA-16008: indeterminate control file checkpoint

SYS @ AUX:CDB$ROOT:>!oerr ora 16008
16008, 00000, "indeterminate control file checkpoint"
// *Cause:  The control file for the database being opened for read-only access
//          was created using the CREATE CONTROLFILE statement. Therefore, a
//          control file checkpoint could not be calculated and the database
//          could not be opened for read-only access.
// *Action: First, open the database for read/write access. This creates
//          a valid control file checkpoint. Then, reopen the database for
//          read-only access.

Ok, that makes sense, RMAN did create a new controlfile. I’ll open the database read/write then and check again.

SYS @ AUX:CDB$ROOT:>alter database open resetlogs;

Database altered.

SYS @ AUX:CDB$ROOT:>show pdbs

    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         2 PDB$SEED                       READ ONLY  NO
         3 PDB1                           MOUNTED
         4 PDB2                           MOUNTED
         5 ORCL19PDB2                     MOUNTED
         
SYS @ AUX:CDB$ROOT:>select file#, name from v$datafile;

     FILE# NAME
---------- ------------------------------------------------------------------------------------------------------------------------
         1 /u01/app/oracle/oradata/AUX/datafile/o1_mf_system_lzqr3jxq_.dbf
         3 /u01/app/oracle/oradata/AUX/datafile/o1_mf_sysaux_lzqr3jwy_.dbf
         4 /u01/app/oracle/oradata/AUX/datafile/o1_mf_undotbs1_lzqr3jy4_.dbf
         5 /u01/app/oracle/oradata/AUX/ED7F22D941443E9AE053A024100AD0FB/datafile/o1_mf_system_lzqr400z_.dbf
         6 /u01/app/oracle/oradata/AUX/ED7F22D941443E9AE053A024100AD0FB/datafile/o1_mf_sysaux_lzqr4014_.dbf
         7 /u01/app/oracle/product/db/19/dbs/MISSING00007
         8 /u01/app/oracle/oradata/AUX/ED7F22D941443E9AE053A024100AD0FB/datafile/o1_mf_undotbs1_lzqr401g_.dbf
        19 /u01/app/oracle/oradata/AUX/ED7F830814134AADE053A024100A5D08/datafile/o1_mf_system_lzqr31r4_.dbf
        20 /u01/app/oracle/oradata/AUX/ED7F830814134AADE053A024100A5D08/datafile/o1_mf_sysaux_lzqr31rf_.dbf
        21 /u01/app/oracle/oradata/AUX/ED7F830814134AADE053A024100A5D08/datafile/o1_mf_undotbs1_lzqr31rq_.dbf
        22 /u01/app/oracle/product/db/19/dbs/MISSING00022
        29 /u01/app/oracle/product/db/19/dbs/MISSING00029
        30 /u01/app/oracle/product/db/19/dbs/MISSING00030
        33 /u01/app/oracle/product/db/19/dbs/MISSING00033

14 rows selected.

As you can see, we have much more information after opening the database. This is because we now have access to the data dictionary too, not only to the controlfile. So we can see, that the database knows about file 22, but not where to find it. But I do know, it was written in the RMAN output and can also be derived from the paths of the other datafiles.

SYS @ AUX:CDB$ROOT:>!ls -l /u01/app/oracle/oradata/AUX/ED7F830814134AADE053A024100A5D08/datafile
total 3490876
-rw-r-----. 1 oracle dba  555753472 Mar 21 08:29 o1_mf_sysaux_lzqr31rf_.dbf
-rw-r-----. 1 oracle dba  713039872 Mar 21 08:29 o1_mf_system_lzqr31r4_.dbf
-rw-r-----. 1 oracle dba  104865792 Mar 21 08:29 o1_mf_temp_lzqrh54t_.tmp
-rw-r-----. 1 oracle dba  157294592 Mar 21 08:29 o1_mf_undotbs1_lzqr31rq_.dbf
-rw-r-----. 1 oracle dba 1073750016 Mar 21 08:23 o1_mf_users_lzqr31r2_.dbf

The file is there, so let’s just use it. The database allows us to rename files, that should do the trick. The file itself is consistent since it was read/only in the source database.

SYS @ AUX:CDB$ROOT:>alter database rename file '/u01/app/oracle/product/db/19/dbs/MISSING00022' to '/u01/app/oracle/oradata/AUX/ED7F830814134AADE053A024100A5D08/datafile/o1_mf_users_lzqr31r2_.dbf';

Database altered.

SYS @ AUX:CDB$ROOT:>alter pluggable database pdb1 open ;

Pluggable database altered.

SYS @ AUX:CDB$ROOT:>alter session set container=pdb1;

Session altered.

SYS @ AUX:CDB$ROOT:>select * from hr.employees;
select * from hr.employees
                 *
ERROR at line 1:
ORA-00376: file 22 cannot be read at this time
ORA-01110: data file 22: '/u01/app/oracle/oradata/AUX/ED7F830814134AADE053A024100A5D08/datafile/o1_mf_users_lzqr31r2_.dbf'


SYS @ AUX:CDB$ROOT:>alter database datafile 22 online;
alter database datafile 22 online
*
ERROR at line 1:
ORA-01190: control file or data file 22 is from before the last RESETLOGS
ORA-01110: data file 22: '/u01/app/oracle/oradata/AUX/ED7F830814134AADE053A024100A5D08/datafile/o1_mf_users_lzqr31r2_.dbf'

Obviously I cannot use the restored file at all. But why is that? To figure this out, I dropped the auxiliary database and did another DUPLICATE, this time ommiting the NOOPEN option, just to see, what RMAN will do. I’ll only paste the relevant parts of this operation for the sake of clarity.

RMAN> duplicate database to aux pluggable database pdb1 skip tablespace pdb1:TBS_ORG_UTPLSQL until time 'sysdate-1' ;

Starting Duplicate Db at 2024-03-21 08:37:59

[...]

channel ORA_AUX_DISK_1: restoring datafile 00022 to /u01/app/oracle/oradata/AUX/ED7F830814134AADE053A024100A5D08/datafile/o1_mf_users_%u_.dbf

[...]

Starting recover at 2024-03-21 08:39:22
using channel ORA_AUX_DISK_1
datafile 22 not processed because file is read-only

[...]

sql statement: CREATE CONTROLFILE REUSE SET DATABASE "AUX" RESETLOGS ARCHIVELOG
  MAXLOGFILES     16
  MAXLOGMEMBERS      3
  MAXDATAFILES     1024
  MAXINSTANCES     8
  MAXLOGHISTORY      292
 LOGFILE
  GROUP     1  SIZE 200 M ,
  GROUP     2  SIZE 200 M ,
  GROUP     3  SIZE 200 M
 DATAFILE
  '/u01/app/oracle/oradata/AUX/datafile/o1_mf_system_lzqs0qjg_.dbf',
  '/u01/app/oracle/oradata/AUX/ED7F22D941443E9AE053A024100AD0FB/datafile/o1_mf_system_lzqs16m6_.dbf',
  '/u01/app/oracle/oradata/AUX/ED7F830814134AADE053A024100A5D08/datafile/o1_mf_system_lzqrzyg9_.dbf'
 CHARACTER SET AL32UTF8

[...]

contents of Memory Script:
{
   sql clone 'alter system set "_system_trig_enabled"=FALSE';
   Alter clone database open resetlogs;
   sql clone 'alter system reset "_system_trig_enabled"';
}
executing Memory Script

sql statement: alter system set "_system_trig_enabled"=FALSE

database opened

sql statement: alter system reset "_system_trig_enabled"
Executing: drop pluggable database "ORCL19PDB2" including datafiles
Executing: drop pluggable database "PDB2" including datafiles

contents of Memory Script:
{
   sql clone "alter pluggable database all open";
}
executing Memory Script

sql statement: alter pluggable database all open

contents of Memory Script:
{
   catalog clone datafilecopy  "/u01/app/oracle/oradata/AUX/ED7F830814134AADE053A024100A5D08/datafile/o1_mf_users_lzqrzyg7_.dbf",
 "/u01/app/oracle/oradata/AUX/ED7F830814134AADE053A024100A5D08/datafile/o1_mf_tbs_data_lzqrzygh_.dbf",
 "/u01/app/oracle/oradata/AUX/ED7F830814134AADE053A024100A5D08/datafile/o1_mf_tbs_lob_lzqrzyg4_.dbf";
   switch clone datafile  22 to datafilecopy
 "/u01/app/oracle/oradata/AUX/ED7F830814134AADE053A024100A5D08/datafile/o1_mf_users_lzqrzyg7_.dbf";
   switch clone datafile  29 to datafilecopy
 "/u01/app/oracle/oradata/AUX/ED7F830814134AADE053A024100A5D08/datafile/o1_mf_tbs_data_lzqrzygh_.dbf";
   switch clone datafile  30 to datafilecopy
 "/u01/app/oracle/oradata/AUX/ED7F830814134AADE053A024100A5D08/datafile/o1_mf_tbs_lob_lzqrzyg4_.dbf";
   #online the readonly tablespace
   sql clone 'PDB1' "alter tablespace
 USERS online";
   #online the readonly tablespace
   sql clone 'PDB1' "alter tablespace
 TBS_LOB online";
   #online the readonly tablespace
   sql clone 'PDB1' "alter tablespace
 TBS_DATA online";
}
executing Memory Script

cataloged datafile copy
datafile copy file name=/u01/app/oracle/oradata/AUX/ED7F830814134AADE053A024100A5D08/datafile/o1_mf_users_lzqrzyg7_.dbf RECID=7 STAMP=1164184865
cataloged datafile copy
datafile copy file name=/u01/app/oracle/oradata/AUX/ED7F830814134AADE053A024100A5D08/datafile/o1_mf_tbs_data_lzqrzygh_.dbf RECID=8 STAMP=1164184865
cataloged datafile copy
datafile copy file name=/u01/app/oracle/oradata/AUX/ED7F830814134AADE053A024100A5D08/datafile/o1_mf_tbs_lob_lzqrzyg4_.dbf RECID=9 STAMP=1164184865

datafile 22 switched to datafile copy
input datafile copy RECID=7 STAMP=1164184865 file name=/u01/app/oracle/oradata/AUX/ED7F830814134AADE053A024100A5D08/datafile/o1_mf_users_lzqrzyg7_.dbf

datafile 29 switched to datafile copy
input datafile copy RECID=8 STAMP=1164184865 file name=/u01/app/oracle/oradata/AUX/ED7F830814134AADE053A024100A5D08/datafile/o1_mf_tbs_data_lzqrzygh_.dbf

datafile 30 switched to datafile copy
input datafile copy RECID=9 STAMP=1164184865 file name=/u01/app/oracle/oradata/AUX/ED7F830814134AADE053A024100A5D08/datafile/o1_mf_tbs_lob_lzqrzyg4_.dbf

sql statement: alter tablespace  USERS online

sql statement: alter tablespace  TBS_LOB online

sql statement: alter tablespace  TBS_DATA online
Dropping offline and skipped tablespaces
Executing: alter database default tablespace system
Executing: drop tablespace "USERS" including contents cascade constraints
Executing: drop tablespace "TBS_ORG_UTPLSQL" including contents cascade constraints
Finished Duplicate Db at 2024-03-21 08:41:25

As you see, the restore and recover work the same way as with the NOOPEN option, no surprise here, I highlighted those lines. But then RMAN opens the database and after opening the database, it catalogs all the remaining files and switches the location of these files, see lines 61-64 and 82-83. The SWITCH command does something else than just renaming the file. And just to proof it works, here we go.

SYS @ AUX:CDB$ROOT:>select status from v$instance;

STATUS
------------
OPEN

SYS @ AUX:CDB$ROOT:>col name format a120
SYS @ AUX:CDB$ROOT:>set lines 200
SYS @ AUX:CDB$ROOT:>set pages 200
SYS @ AUX:CDB$ROOT:>select file#, name from v$datafile;

     FILE# NAME
---------- ------------------------------------------------------------------------------------------------------------------------
         1 /u01/app/oracle/oradata/AUX/datafile/o1_mf_system_lzqs0qjg_.dbf
         3 /u01/app/oracle/oradata/AUX/datafile/o1_mf_sysaux_lzqs0qjb_.dbf
         4 /u01/app/oracle/oradata/AUX/datafile/o1_mf_undotbs1_lzqs0qjk_.dbf
         5 /u01/app/oracle/oradata/AUX/ED7F22D941443E9AE053A024100AD0FB/datafile/o1_mf_system_lzqs16m6_.dbf
         6 /u01/app/oracle/oradata/AUX/ED7F22D941443E9AE053A024100AD0FB/datafile/o1_mf_sysaux_lzqs16mb_.dbf
         8 /u01/app/oracle/oradata/AUX/ED7F22D941443E9AE053A024100AD0FB/datafile/o1_mf_undotbs1_lzqs16mf_.dbf
        19 /u01/app/oracle/oradata/AUX/ED7F830814134AADE053A024100A5D08/datafile/o1_mf_system_lzqrzyg9_.dbf
        20 /u01/app/oracle/oradata/AUX/ED7F830814134AADE053A024100A5D08/datafile/o1_mf_sysaux_lzqrzygc_.dbf
        21 /u01/app/oracle/oradata/AUX/ED7F830814134AADE053A024100A5D08/datafile/o1_mf_undotbs1_lzqrzygf_.dbf
        22 /u01/app/oracle/oradata/AUX/ED7F830814134AADE053A024100A5D08/datafile/o1_mf_users_lzqrzyg7_.dbf
        29 /u01/app/oracle/oradata/AUX/ED7F830814134AADE053A024100A5D08/datafile/o1_mf_tbs_data_lzqrzygh_.dbf
        30 /u01/app/oracle/oradata/AUX/ED7F830814134AADE053A024100A5D08/datafile/o1_mf_tbs_lob_lzqrzyg4_.dbf

12 rows selected.

SYS @ AUX:CDB$ROOT:>alter session set container=pdb1;

Session altered.

SYS @ AUX:CDB$ROOT:>select count(*) from hr.employees;

  COUNT(*)
----------
       107

But can we use the SWITCH trick also for the NOOPEN case? I dropped the auxiliary database again and did another duplicate with the NOOPEN option as in the first case. After this duplication, the situation is again like this.

YS @ AUX:CDB$ROOT:>select file#, name from v$datafile;

     FILE# NAME
---------- ------------------------------------------------------------------------------------------------------------------------
         1 /u01/app/oracle/oradata/AUX/datafile/o1_mf_system_lzqskm2l_.dbf
         3 /u01/app/oracle/oradata/AUX/datafile/o1_mf_sysaux_lzqskm2f_.dbf
         4 /u01/app/oracle/oradata/AUX/datafile/o1_mf_undotbs1_lzqskm2p_.dbf
         5 /u01/app/oracle/oradata/AUX/ED7F22D941443E9AE053A024100AD0FB/datafile/o1_mf_system_lzqsl251_.dbf
         6 /u01/app/oracle/oradata/AUX/ED7F22D941443E9AE053A024100AD0FB/datafile/o1_mf_sysaux_lzqsl256_.dbf
         8 /u01/app/oracle/oradata/AUX/ED7F22D941443E9AE053A024100AD0FB/datafile/o1_mf_undotbs1_lzqsl259_.dbf
        19 /u01/app/oracle/oradata/AUX/ED7F830814134AADE053A024100A5D08/datafile/o1_mf_system_lzqsk40g_.dbf
        20 /u01/app/oracle/oradata/AUX/ED7F830814134AADE053A024100A5D08/datafile/o1_mf_sysaux_lzqsk40j_.dbf
        21 /u01/app/oracle/oradata/AUX/ED7F830814134AADE053A024100A5D08/datafile/o1_mf_undotbs1_lzqsk40k_.dbf

9 rows selected.

SYS @ AUX:CDB$ROOT:>!ls -l /u01/app/oracle/oradata/AUX/ED7F830814134AADE053A024100A5D08/datafile/
total 2441260
-rw-r-----. 1 oracle dba  555753472 Mar 21 08:49 o1_mf_sysaux_lzqsk40j_.dbf
-rw-r-----. 1 oracle dba  713039872 Mar 21 08:49 o1_mf_system_lzqsk40g_.dbf
-rw-r-----. 1 oracle dba  157294592 Mar 21 08:49 o1_mf_undotbs1_lzqsk40k_.dbf
-rw-r-----. 1 oracle dba 1073750016 Mar 21 08:47 o1_mf_users_lzqsk40b_.dbf

No file 22 in the views, but at least it got restored again. So let’s try to open the auxiliary database again and use the SWITCH command this time. Note, that the command needs to put into a RUN-block.

[oracle@vm160 ~]$ export ORACLE_SID=aux
[oracle@vm160 ~]$ rman target /

Recovery Manager: Release 19.0.0.0.0 - Production on Thu Mar 21 09:59:32 2024
Version 19.18.0.0.0

Copyright (c) 1982, 2019, Oracle and/or its affiliates.  All rights reserved.

connected to target database: AUX (DBID=2017441707, not open)

RMAN> alter database open resetlogs;

Statement processed

RMAN> run {
2> catalog datafilecopy  "/u01/app/oracle/oradata/AUX/ED7F830814134AADE053A024100A5D08/datafile/o1_mf_users_lzqsk40b_.dbf";
3> switch datafile 22 to datafilecopy "/u01/app/oracle/oradata/AUX/ED7F830814134AADE053A024100A5D08/datafile/o1_mf_users_lzqsk40b_.dbf";
4> }

cataloged datafile copy
datafile copy file name=/u01/app/oracle/oradata/AUX/ED7F830814134AADE053A024100A5D08/datafile/o1_mf_users_lzqsk40b_.dbf RECID=7 STAMP=1164189777

datafile 22 switched to datafile copy
input datafile copy RECID=7 STAMP=1164189777 file name=/u01/app/oracle/oradata/AUX/ED7F830814134AADE053A024100A5D08/datafile/o1_mf_users_lzqsk40b_.dbf

RMAN> sql "alter pluggable database pdb1 open";

sql statement: alter pluggable database pdb1 open

RMAN> sql 'PDB1' "alter tablespace users online";

sql statement: alter tablespace users online

Looks promising so far. But am I able to query some data from this file now?

[oracle@vm160 ~]$ sqlplus / as sysdba

SQL*Plus: Release 19.0.0.0.0 - Production on Thu Mar 21 10:04:55 2024
Version 19.18.0.0.0

Copyright (c) 1982, 2022, Oracle.  All rights reserved.


Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.18.0.0.0

SYS @ AUX:CDB$ROOT:>alter session set container=pdb1;

Session altered.

SYS @ AUX:CDB$ROOT:>select count(*) from hr.employees;

  COUNT(*)
----------
       107

Yes, I can! Finally! So what did I learn? RMAN duplication skips read only files during the recover operation which is totally fine. It adds those files *after* opening the database. If I open the database myself, I have to find the restored files myself and use the RMAN SWITCH command to make them part of the restored database, An “alter database rename file” is not sufficient. Good to know.

Oracle RMAN: connect / as sysbackup

This post is about RMAN connects and role separation. During an Oracle Database installtion, one can choose which OS group should map to which role for database administration. There is the well-known “dba” group which can do nearly everything to the database and maps to the SYSDBA role. And there are a few more groups for key management, dataguard administration and also for backup/restore. The backup/restore group is todays topic. A customer wants to perform the database backups by an OS user other than “oracle” which is the primary DBA user. That shouldn’t be a big thing, because we have these roles I mentioned. But wait, at first we need to check if the database software is configured for role separtion. In the file “$ORACLE_HOME/rdbms/lib/config.c” we can find this information.

[oracle@vm160 lib]$ grep -e "^\.L.*string" config.c
.Ldba_string:     .string "dba"
.Loper_string:    .string "oper"
.Lasm_string:     .string ""
.Lbkp_string:     .string "backupdba"
.Ldgd_string:     .string "dgdba"
.Lkmt_string:     .string "kmdba"
.Lrac_string:     .string "racdba"

In this case we are good to go. If we find the same OS group for the different roles, we need to recompile the software. The basic steps are described in MOS Note 558478.1 and are roughly these:

  • Stop everything that runs from this database home
  • make a backup of $ORACLE_HOME/rdbms/lib/config.o
  • change $ORACLE_HOME/rdbms/lib/config.c to map the roles to the desired OS groups
  • re-create config.o: make -f ins_rdbms.mk config.o
  • do a recompile/relink by running “make -f ins_rdbms.mk ioracle”
  • bring everything up again

Of cause the OS groups should be present at that time. Mine look like this:

[oracle@vm160 ~]$ tail -7 /etc/group
oinstall:x:54321:oracle
dba:x:54322:oracle
oper:x:54323:oracle
backupdba:x:54324:oracle
dgdba:x:54325:oracle
kmdba:x:54326:oracle
racdba:x:54330:oracle

What we now need is an OS user, that shall be used for performing database backups in the future. This user must be part of the “backupdba” group for being able to connect as SYSBACKUP to the database but must not be in the “dba” group because that would allow a connection as SYSDBA which we do not want. Beside that, the user must be part of the “oinstall” group which is the group that owns the database software installation.

[root@vm160 ~]# useradd -g oinstall-G backupdba,oinstall orarman
[root@vm160 ~]# id orarman
uid=54322(orarman) gid=54321(oinstall) groups=54321(oinstall),54324(backupdba)

I also defined a password for the new user, you can provide SSH keys as well to allow logins. So first I check with SQL*Plus if the user can connect to the database as SYSBACKUP but not as SYSDBA.

[orarman@vm160 ~]$ echo $ORACLE_SID
orcl19
[orarman@vm160 ~]$ sqlplus / as sysdba

SQL*Plus: Release 19.0.0.0.0 - Production on Thu Dec 21 07:25:07 2023
Version 19.18.0.0.0

Copyright (c) 1982, 2022, Oracle.  All rights reserved.

ERROR:
ORA-01017: invalid username/password; logon denied


Enter user-name:
[orarman@vm160 ~]$ sqlplus / as sysbackup

SQL*Plus: Release 19.0.0.0.0 - Production on Thu Dec 21 07:25:15 2023
Version 19.18.0.0.0

Copyright (c) 1982, 2022, Oracle.  All rights reserved.


Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.18.0.0.0

SQL>

That looks good so far and is exactly how I wanted it to behave. But in the end I need RMAN for doing all the backup things.

[orarman@vm160 ~]$ rman target /

Recovery Manager: Release 19.0.0.0.0 - Production on Thu Dec 21 07:28:31 2023
Version 19.18.0.0.0

Copyright (c) 1982, 2019, Oracle and/or its affiliates.  All rights reserved.

RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-00554: initialization of internal recovery manager package failed
RMAN-04005: error from target database:
ORA-01017: invalid username/password; logon denied

[orarman@vm160 ~]$ rman target / as sysbackup

Argument     Value          Description
-----------------------------------------------------------------------------
target       quoted-string  connect-string for target database
catalog      quoted-string  connect-string for recovery catalog
auxiliary    quoted-string  connect-string for auxiliary database
nocatalog    none           if specified, then no recovery catalog
cmdfile      quoted-string  name of input command file
log          quoted-string  name of output message log file
trace        quoted-string  name of output debugging message log file
append       none           if specified, log is opened in append mode
debug        optional-args  activate debugging
msgno        none           show RMAN-nnnn prefix for all messages
send         quoted-string  send a command to the media manager
pipe         string         building block for pipe names
script       string         name of catalog script to execute
using        list of args   arguments for rman variables
timeout      integer        number of seconds to wait for pipe input
checksyntax  none           check the command file for syntax errors
-----------------------------------------------------------------------------
Both single and double quotes (' or ") are accepted for a quoted-string.
Quotes are not required unless the string contains embedded white-space.

RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-00552: syntax error in command line arguments
RMAN-01009: syntax error: found "as": expecting one of: "append, auxiliary, catalog, checksyntax, cmdfile, log, msgno, nocatalog, pipe, script, send, target, timeout, using, @, ;"
RMAN-01007: at line 2 column 1 file: command line arguments

Ok, obviously not as straight-forward as I thought. Let’s check the docs how to connect from RMAN to a database: Oracle 19 Backup and Recovery Reference – connectStringSpec.

Reading the syntax diagram my connect strings are correct. Single quotes are optional and I can add “as sysbackup” to any connect identifier. Reading a bit further, there is this sentence:

If you do not specify a user ID or password when connecting to a target database, then a slash establishes a connection using the SYSDBA privilege by using operating system authentication (see Example 4-12).

Oracle 19 Backup and Recovery Reference: connectStringSpec

Does that mean, when I use the slash, it will always use the AS SYSDBA connection? I can’t believe this to be true. Let’s try RMAN without any connection and try to establish a connection at the RMAN prompt.

[orarman@vm160 ~]$ rman

Recovery Manager: Release 19.0.0.0.0 - Production on Thu Dec 21 06:59:21 2023
Version 19.18.0.0.0

Copyright (c) 1982, 2019, Oracle and/or its affiliates.  All rights reserved.

RMAN> connect target / as sysbackup

RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-00558: error encountered while parsing input commands
RMAN-01009: syntax error: found "as": expecting one of: "newline, ;"
RMAN-01007: at line 1 column 18 file: standard input

The error message is somewhat helpful. RMAN finds the word “as” when it expects a newline or semicolon. So maybe RMAN expects the connect string as a single value so let’s try with quotes.

RMAN> connect target '/ as sysbackup'

connected to target database: ORCL19 (DBID=422142402)

RMAN>

Ok, great. This works. Let’s give it another try using double-quotes. Not in the syntax diagram but who knows?

[orarman@vm160 ~]$ rman

Recovery Manager: Release 19.0.0.0.0 - Production on Thu Dec 21 06:59:41 2023
Version 19.18.0.0.0

Copyright (c) 1982, 2019, Oracle and/or its affiliates.  All rights reserved.

RMAN> connect target "/ as sysbackup"

connected to target database: ORCL19 (DBID=422142402)

RMAN>

That works too. Good to know. According to the docs, only single-quotes are allowed and are optional. That is obviously not true for all cases. Now finally I want to invoke RMAN directly from the command line including the connect string. Maybe these quotes will help to make the “/ as sysdba” a single parameter value rather than three.

[orarman@vm160 ~]$ rman target "'/ as sysbackup'"

Recovery Manager: Release 19.0.0.0.0 - Production on Thu Dec 21 07:47:41 2023
Version 19.18.0.0.0

Copyright (c) 1982, 2019, Oracle and/or its affiliates.  All rights reserved.

connected to target database: ORCL19 (DBID=422142402)

RMAN>

This does the trick. And it doesn’t matter, if we enclose the single-quoted string in double-quotes or vice versa. We need two quotes because the outer quotes are interpreted by the shell and tell the shell, that anyting inside these quotes shall be treated as a single value. The inner string with the final connect string is then passed to RMAN which handles it as shown above.

Conclusion: It is possible to establish a RMAN connection using the SYSBACKUP privilege without using a password, e.g. using OS authentication. But it is not as straight-forward as I thought and the docs are not quite clear about the connection string.

RMAN recovery window over redundancy

One of my favorite topics around the Oracle Database is backup and recovery because it is so essential for successfully operating a database landscape. The topic today is a basic setting in RMAN, that controls the lifetime of backups, the “retention policy”. Let’s get into it.

When planning a backup strategy, there are two major values, that should be taken into account. One is the Recovery Time Objective (RTO) which is the amount of time spent to bring a system back to life after an incident. The second is the Recovery Point Objective (RPO) which represents the point in time to which a system shall be restored to get back to operation. The latter one is mainly influencing our retention policy. Imagine the following situation, a major application update takes place which does a lot of DDL and DML to the database because there are so many cool new features in the new release of that application. But for some reason the update fails when it was halfway through. Of cause it cannot be rolled back easily because of the DDL that took place. So the DBA is asked to restore the database to the time just before the update began.

That is, where the fun part starts. Basically it should be an easy thing using RMAN. Set the point in time, restore the controlfile, restore the database, roll it forward until the point in time and then open the database. So let’s go ahead.

[oracle@vm160 ~]$ rman target /

Recovery Manager: Release 19.0.0.0.0 - Production on Tue Nov 7 08:33:41 2023
Version 19.18.0.0.0

Copyright (c) 1982, 2019, Oracle and/or its affiliates.  All rights reserved.

connected to target database: ORCL19 (DBID=422142402)

RMAN> startup nomount force

Oracle instance started

Total System Global Area    2147482320 bytes

Fixed Size                     9165520 bytes
Variable Size               1342177280 bytes
Database Buffers             788529152 bytes
Redo Buffers                   7610368 bytes

RMAN> set dbid=422142402

executing command: SET DBID

RMAN> run {
2> set until time "to_date('06.11.2023 04:00:00', 'dd.mm.yyyy hh24:mi:ss')";
3> restore controlfile from autobackup;
4> alter database mount;
5> restore database;
6> recover database;
7> }

executing command: SET until clause

Starting restore at 2023-11-07 08:35:08
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=237 device type=DISK

recovery area destination: /u01/app/oracle/fra
database name (or database unique name) used for search: ORCL19
channel ORA_DISK_1: no AUTOBACKUPS found in the recovery area
channel ORA_DISK_1: looking for AUTOBACKUP on day: 20231106
channel ORA_DISK_1: looking for AUTOBACKUP on day: 20231105
channel ORA_DISK_1: looking for AUTOBACKUP on day: 20231104
channel ORA_DISK_1: looking for AUTOBACKUP on day: 20231103
channel ORA_DISK_1: looking for AUTOBACKUP on day: 20231102
channel ORA_DISK_1: looking for AUTOBACKUP on day: 20231101
channel ORA_DISK_1: looking for AUTOBACKUP on day: 20231031
channel ORA_DISK_1: no AUTOBACKUP in 7 days found
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of restore command at 11/07/2023 08:35:11
RMAN-06172: no AUTOBACKUP found or specified handle is not a valid copy or piece

What? Why don’t I have any backup of my controlfile? Let’s check that.

RMAN> list backup of controlfile summary;


List of Backups
===============
Key     TY LV S Device Type Completion Time     #Pieces #Copies Compressed Tag
------- -- -- - ----------- ------------------- ------- ------- ---------- ---
68      B  F  A DISK        2023-11-07 08:44:01 1       1       NO         TAG20231107T084400

Ok, there is a controlfile backup, but just one. So what are my RMAN settings? Can I see something?

RMAN> show all;

using target database control file instead of recovery catalog
RMAN configuration parameters for database with db_unique_name ORCL19 are:
CONFIGURE RETENTION POLICY TO REDUNDANCY 1; # default
CONFIGURE BACKUP OPTIMIZATION OFF; # default
CONFIGURE DEFAULT DEVICE TYPE TO DISK; # default
CONFIGURE CONTROLFILE AUTOBACKUP ON; # default
CONFIGURE CONTROLFILE AUTOBACKUP FORMAT FOR DEVICE TYPE DISK TO '%F'; # default
CONFIGURE DEVICE TYPE DISK PARALLELISM 1 BACKUP TYPE TO BACKUPSET; # default
CONFIGURE DATAFILE BACKUP COPIES FOR DEVICE TYPE DISK TO 1; # default
CONFIGURE ARCHIVELOG BACKUP COPIES FOR DEVICE TYPE DISK TO 1; # default
CONFIGURE MAXSETSIZE TO UNLIMITED; # default
CONFIGURE ENCRYPTION FOR DATABASE OFF; # default
CONFIGURE ENCRYPTION ALGORITHM 'AES128'; # default
CONFIGURE COMPRESSION ALGORITHM 'BASIC' AS OF RELEASE 'DEFAULT' OPTIMIZE FOR LOAD TRUE ; # default
CONFIGURE RMAN OUTPUT TO KEEP FOR 7 DAYS; # default
CONFIGURE ARCHIVELOG DELETION POLICY TO NONE; # default
CONFIGURE SNAPSHOT CONTROLFILE NAME TO '/u01/app/oracle/product/db/19/dbs/snapcf_orcl19.f'; # default

The thing is the redundancy=1 as the retention policy. A redundancy of 1 (in words: one) means just what it says, RMAN keeps exactly one backup of each file it backs up. And often I see backup strategies relying highly on redundancy. Say I want to keep 2 weeks of backups for my database. A full backup takes place every sunday, so I set retention policy to “redundancy 2”. That way, RMAN keeps 2 full backups of my database which, according to the backup schedule, will be good for 2 weeks of backups. No, it is not. Why not? For a couple of reasons. First, as shown, the redundancy applies to all types of backups, also controlfile bakups etc. First, to be able to restore the database to the point 2 weeks ago, we’ll need a controlfile backup from that time. And there might be many of those backups. Second, if we change the backup schedule to backup the database twice a week, we change the retention to about one week implicitely. That’s not what I want it to be.

So how do I come around this? The answer is quite simple: Change the retention policy to be a recovery window rather than redundancy.

RMAN> CONFIGURE RETENTION POLICY TO RECOVERY WINDOW OF 14 DAYS;;

new RMAN configuration parameters:
CONFIGURE RETENTION POLICY TO RECOVERY WINDOW OF 14 DAYS;
new RMAN configuration parameters are successfully stored

With that setting the previous example would have run without any errors. There are many backups of the controlfile:

RMAN> list backup of controlfile summary;


List of Backups
===============
Key     TY LV S Device Type Completion Time     #Pieces #Copies Compressed Tag
------- -- -- - ----------- ------------------- ------- ------- ---------- ---
17      B  F  A DISK        2023-02-24 13:54:24 1       1       NO         TAG20230224T135422
24      B  F  A DISK        2023-10-27 09:13:43 1       1       NO         TAG20231027T091343
26      B  F  A DISK        2023-10-27 09:16:32 1       1       NO         TAG20231027T091632
28      B  F  A DISK        2023-10-28 02:00:06 1       1       NO         TAG20231028T020006
35      B  F  A DISK        2023-10-29 01:00:47 1       1       NO         TAG20231029T010046
37      B  F  A DISK        2023-10-29 02:00:05 1       1       NO         TAG20231029T020005
39      B  F  A DISK        2023-10-30 02:00:08 1       1       NO         TAG20231030T020008
41      B  F  A DISK        2023-10-31 02:00:05 1       1       NO         TAG20231031T020005
43      B  F  A DISK        2023-11-01 02:00:07 1       1       NO         TAG20231101T020007
45      B  F  A DISK        2023-11-02 02:00:06 1       1       NO         TAG20231102T020006
46      B  F  A DISK        2023-11-02 13:28:47 1       1       NO         TAG20231102T132847
47      B  F  A DISK        2023-11-02 13:44:19 1       1       NO         TAG20231102T134419
48      B  F  A DISK        2023-11-02 13:47:53 1       1       NO         TAG20231102T134753
49      B  F  A DISK        2023-11-02 13:57:21 1       1       NO         TAG20231102T135721
52      B  F  A DISK        2023-11-03 02:00:08 1       1       NO         TAG20231103T020008
54      B  F  A DISK        2023-11-04 02:00:07 1       1       NO         TAG20231104T020006
61      B  F  A DISK        2023-11-05 01:00:37 1       1       NO         TAG20231105T010037
63      B  F  A DISK        2023-11-05 02:00:06 1       1       NO         TAG20231105T020006
65      B  F  A DISK        2023-11-06 02:00:05 1       1       NO         TAG20231106T020005
66      B  F  A DISK        2023-11-07 08:24:25 1       1       NO         TAG20231107T082424
67      B  F  A DISK        2023-11-07 02:00:06 1       1       NO         TAG20231107T020006
68      B  F  A DISK        2023-11-07 08:44:01 1       1       NO         TAG20231107T084400

With all these backups I can easily restore the database to the given point in time.

[oracle@vm160 ~]$ rman target /

Recovery Manager: Release 19.0.0.0.0 - Production on Tue Nov 7 08:35:59 2023
Version 19.18.0.0.0

Copyright (c) 1982, 2019, Oracle and/or its affiliates.  All rights reserved.

connected to target database: ORCL19 (not mounted)

RMAN> set dbid=422142402

executing command: SET DBID

RMAN> run {
2> set until time "to_date('06.11.2023 04:00:00', 'dd.mm.yyyy hh24:mi:ss')";
3> restore controlfile from autobackup;
4> alter database mount;
5> restore database;
6> recover database;
7> }

executing command: SET until clause

Starting restore at 2023-11-07 08:36:51
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=20 device type=DISK

recovery area destination: /u01/app/oracle/fra
database name (or database unique name) used for search: ORCL19
channel ORA_DISK_1: AUTOBACKUP /u01/app/oracle/fra/ORCL19/autobackup/2023_11_06/o1_mf_s_1152151205_lnjgnooh_.bkp found in the recovery area
channel ORA_DISK_1: looking for AUTOBACKUP on day: 20231106
channel ORA_DISK_1: restoring control file from AUTOBACKUP /u01/app/oracle/fra/ORCL19/autobackup/2023_11_06/o1_mf_s_1152151205_lnjgnooh_.bkp
channel ORA_DISK_1: control file restore from AUTOBACKUP complete
output file name=/u01/app/oracle/oradata/ORCL19/controlfile/o1_mf_kq6lhlbv_.ctl
output file name=/u01/app/oracle/fra/ORCL19/controlfile/o1_mf_kq6lhlmd_.ctl
Finished restore at 2023-11-07 08:36:53

released channel: ORA_DISK_1
Statement processed

Starting restore at 2023-11-07 08:36:58
Starting implicit crosscheck backup at 2023-11-07 08:36:58
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=20 device type=DISK
Crosschecked 48 objects
Finished implicit crosscheck backup at 2023-11-07 08:36:59

Starting implicit crosscheck copy at 2023-11-07 08:36:59
using channel ORA_DISK_1
Finished implicit crosscheck copy at 2023-11-07 08:36:59

searching for all files in the recovery area
cataloging files...
cataloging done

List of Cataloged Files
=======================
File Name: /u01/app/oracle/fra/ORCL19/autobackup/2023_11_06/o1_mf_s_1152151205_lnjgnooh_.bkp
File Name: /u01/app/oracle/fra/ORCL19/autobackup/2023_11_07/o1_mf_s_1152260664_lnmsk92o_.bkp
File Name: /u01/app/oracle/fra/ORCL19/autobackup/2023_11_07/o1_mf_s_1152237606_lnm30pow_.bkp

using channel ORA_DISK_1

skipping datafile 5; already restored to file /u01/app/oracle/oradata/ORCL19/datafile/o1_mf_system_kq6m8z4c_.dbf
skipping datafile 6; already restored to file /u01/app/oracle/oradata/ORCL19/datafile/o1_mf_sysaux_kq6m8z7g_.dbf
skipping datafile 8; already restored to file /u01/app/oracle/oradata/ORCL19/datafile/o1_mf_undotbs1_kq6m8zgv_.dbf
channel ORA_DISK_1: starting datafile backup set restore
channel ORA_DISK_1: specifying datafile(s) to restore from backup set
channel ORA_DISK_1: restoring datafile 00001 to /u01/app/oracle/oradata/ORCL19/datafile/o1_mf_system_kq6l8vf4_.dbf
channel ORA_DISK_1: restoring datafile 00003 to /u01/app/oracle/oradata/ORCL19/datafile/o1_mf_sysaux_kq6lbyd0_.dbf
channel ORA_DISK_1: restoring datafile 00004 to /u01/app/oracle/oradata/ORCL19/datafile/o1_mf_undotbs1_kq6ld1qb_.dbf
channel ORA_DISK_1: restoring datafile 00007 to /u01/app/oracle/oradata/ORCL19/datafile/o1_mf_users_kq6ld4xm_.dbf
channel ORA_DISK_1: reading from backup piece /u01/app/oracle/backup/orcl19/ORCL19_20231105_data_1s2am3om_1_1
channel ORA_DISK_1: piece handle=/u01/app/oracle/backup/orcl19/ORCL19_20231105_data_1s2am3om_1_1 tag=TAG20231105T010006
channel ORA_DISK_1: restored backup piece 1
channel ORA_DISK_1: restore complete, elapsed time: 00:00:15
channel ORA_DISK_1: starting datafile backup set restore
channel ORA_DISK_1: specifying datafile(s) to restore from backup set
channel ORA_DISK_1: restoring datafile 00019 to /u01/app/oracle/oradata/ORCL19/ED7F830814134AADE053A024100A5D08/datafile/system_umbenannt.dbf
channel ORA_DISK_1: restoring datafile 00020 to /u01/app/oracle/oradata/ORCL19/ED7F830814134AADE053A024100A5D08/datafile/o1_mf_sysaux_kzgxy7n2_.dbf
channel ORA_DISK_1: restoring datafile 00021 to /u01/app/oracle/oradata/ORCL19/ED7F830814134AADE053A024100A5D08/datafile/o1_mf_undotbs1_kzgxy7n4_.dbf
channel ORA_DISK_1: restoring datafile 00022 to /u01/app/oracle/oradata/ORCL19/ED7F830814134AADE053A024100A5D08/datafile/o1_mf_users_kzgxy7n5_.dbf
channel ORA_DISK_1: reading from backup piece /u01/app/oracle/backup/orcl19/ORCL19_20231105_data_1t2am3p6_1_1
channel ORA_DISK_1: piece handle=/u01/app/oracle/backup/orcl19/ORCL19_20231105_data_1t2am3p6_1_1 tag=TAG20231105T010006
channel ORA_DISK_1: restored backup piece 1
channel ORA_DISK_1: restore complete, elapsed time: 00:00:07
channel ORA_DISK_1: starting datafile backup set restore
channel ORA_DISK_1: specifying datafile(s) to restore from backup set
channel ORA_DISK_1: restoring datafile 00015 to /u01/app/oracle/oradata/ORCL19/EF608A6519330511E053A024100A91E2/datafile/o1_mf_system_ks5pktn8_.dbf
channel ORA_DISK_1: restoring datafile 00016 to /u01/app/oracle/oradata/ORCL19/EF608A6519330511E053A024100A91E2/datafile/o1_mf_sysaux_ks5pktn9_.dbf
channel ORA_DISK_1: restoring datafile 00017 to /u01/app/oracle/oradata/ORCL19/EF608A6519330511E053A024100A91E2/datafile/o1_mf_undotbs1_ks5pktnb_.dbf
channel ORA_DISK_1: restoring datafile 00023 to /u01/app/oracle/oradata/ORCL19/EF608A6519330511E053A024100A91E2/datafile/o1_mf_users_ln761vy3_.dbf
channel ORA_DISK_1: reading from backup piece /u01/app/oracle/backup/orcl19/ORCL19_20231105_data_1u2am3pd_1_1
channel ORA_DISK_1: piece handle=/u01/app/oracle/backup/orcl19/ORCL19_20231105_data_1u2am3pd_1_1 tag=TAG20231105T010006
channel ORA_DISK_1: restored backup piece 1
channel ORA_DISK_1: restore complete, elapsed time: 00:00:07
Finished restore at 2023-11-07 08:37:29

Starting recover at 2023-11-07 08:37:29
using channel ORA_DISK_1

starting media recovery

archived log for thread 1 with sequence 7 is already on disk as file /u01/app/oracle/fra/ORCL19/onlinelog/o1_mf_1_kq6lkxbr_.log
archived log for thread 1 with sequence 8 is already on disk as file /u01/app/oracle/fra/ORCL19/onlinelog/o1_mf_2_kq6lk9px_.log
channel ORA_DISK_1: starting archived log restore to default destination
channel ORA_DISK_1: restoring archived log
archived log thread=1 sequence=4
channel ORA_DISK_1: reading from backup piece /u01/app/oracle/backup/orcl19/ORCL19_20231105_arch_202am3pk_1_1
channel ORA_DISK_1: piece handle=/u01/app/oracle/backup/orcl19/ORCL19_20231105_arch_202am3pk_1_1 tag=TAG20231105T010035
channel ORA_DISK_1: restored backup piece 1
channel ORA_DISK_1: restore complete, elapsed time: 00:00:01
archived log file name=/u01/app/oracle/fra/ORCL19/archivelog/2023_11_07/o1_mf_1_4_lnmt9trc_.arc thread=1 sequence=4
channel default: deleting archived log(s)
archived log file name=/u01/app/oracle/fra/ORCL19/archivelog/2023_11_07/o1_mf_1_4_lnmt9trc_.arc RECID=268 STAMP=1152261450
channel ORA_DISK_1: starting archived log restore to default destination
channel ORA_DISK_1: restoring archived log
archived log thread=1 sequence=5
channel ORA_DISK_1: reading from backup piece /u01/app/oracle/backup/orcl19/ORCL19_20231105_arch_222am794_1_1
channel ORA_DISK_1: piece handle=/u01/app/oracle/backup/orcl19/ORCL19_20231105_arch_222am794_1_1 tag=TAG20231105T020004
channel ORA_DISK_1: restored backup piece 1
channel ORA_DISK_1: restore complete, elapsed time: 00:00:01
archived log file name=/u01/app/oracle/fra/ORCL19/archivelog/2023_11_07/o1_mf_1_5_lnmt9w16_.arc thread=1 sequence=5
channel default: deleting archived log(s)
archived log file name=/u01/app/oracle/fra/ORCL19/archivelog/2023_11_07/o1_mf_1_5_lnmt9w16_.arc RECID=269 STAMP=1152261452
channel ORA_DISK_1: starting archived log restore to default destination
channel ORA_DISK_1: restoring archived log
archived log thread=1 sequence=6
channel ORA_DISK_1: reading from backup piece /u01/app/oracle/backup/orcl19/ORCL19_20231106_arch_242aorl4_1_1
channel ORA_DISK_1: piece handle=/u01/app/oracle/backup/orcl19/ORCL19_20231106_arch_242aorl4_1_1 tag=TAG20231106T020004
channel ORA_DISK_1: restored backup piece 1
channel ORA_DISK_1: restore complete, elapsed time: 00:00:01
archived log file name=/u01/app/oracle/fra/ORCL19/archivelog/2023_11_07/o1_mf_1_6_lnmt9yvg_.arc thread=1 sequence=6
channel default: deleting archived log(s)
archived log file name=/u01/app/oracle/fra/ORCL19/archivelog/2023_11_07/o1_mf_1_6_lnmt9yvg_.arc RECID=270 STAMP=1152261455
archived log file name=/u01/app/oracle/fra/ORCL19/onlinelog/o1_mf_1_kq6lkxbr_.log thread=1 sequence=7
archived log file name=/u01/app/oracle/fra/ORCL19/onlinelog/o1_mf_2_kq6lk9px_.log thread=1 sequence=8
media recovery complete, elapsed time: 00:00:07
Finished recover at 2023-11-07 08:37:42

Setting the RMAN retention policy to any redundancy is a really bad idea. It might work out for the backup of the datafiles, but it will definitely cause issues with the backups of controlfiles.

So typically we have a requirement for the RTO. We should always use the given value for setting the RMAN retention policy to a recovery windows so that RMAN can keep track of all the necessary backups and is able to keep them as long as they might be required.

Restore a PDB into another CDB

In my previous post I already wrote about a special restore scenario with the Multitenant Architecture. Today the story continues with another scenario that sounds quite simple in the first place. I will be using Oracle Database Version 19.18 for this topic.

So let’s get started. The goal is to restore a given PDB into another CDB at a specific point in time. Imagine some accidental data change happend which should be rolled back. Since only a small and rarely used table is affected, flashback or a complete Point-in-time Restore is not an option. The goal is to restore the PDB into another CDB at the given point in time to analyze the issue further. So let’s get started.

Instantiating a PDB in another CDB sounds like cloning at a first glance. But cloning always uses the running PDB as a source, there is no option to specify a point in time. So let’s try RMAN PDB duplication.

[oracle@odax5ha0 ~]$ rman

Recovery Manager: Release 19.0.0.0.0 - Production on Tue Mar 21 10:08:34 2023
Version 19.16.0.0.0

Copyright (c) 1982, 2019, Oracle and/or its affiliates.  All rights reserved.

RMAN> connect target sys/********@odax5ha0:1521/mpmcdb2x5.support.robotron.de

connected to target database: MPMCDB2 (DBID=889235828)

RMAN> connect auxiliary sys/********@odax5ha0:1521/mpmcdb1x5.support.robotron.de

connected to auxiliary database: MPMCDB1 (DBID=1547213674)

RMAN> duplicate pluggable database pdba as pdbclone  until time "to_date('21.03.2023 09:10:00')";

Starting Duplicate PDB at 2023-03-21 10:10:28
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of Duplicate PDB command at 03/21/2023 10:10:28
RMAN-05501: aborting duplication of target database
RMAN-05649: duplicate pluggable database does not allow the use of UNTIL SCN or TO RESTORE POINT clause

Oh yes, good start. And RMAN tells the truth, the docs tell us that.

As in the preivous post, I need an temporary auxiliary CDB instance to restore my PDB. Good news, this time the steps will be a little easier. I start with a Pfile derived from the target CDB.

SQL> create pfile='/tmp/initaux.ora' from spfile;

File created.

I remove the “control_files” parameter and set the “db_unique_name” to something unique. Having that I can startup my auxiliary instance and create a SPfile for it.

[oracle@odax5ha0 ~]$ export ORACLE_SID=CDBDUP

[oracle@odax5ha0 ~]$ sqlplus / as sysdba

SQL*Plus: Release 19.0.0.0.0 - Production on Tue Mar 21 10:14:22 2023
Version 19.16.0.0.0

Copyright (c) 1982, 2022, Oracle.  All rights reserved.

Connected to an idle instance.

SQL> startup nomount pfile=/tmp/initaux.ora
ORACLE instance started.

Total System Global Area 4294963760 bytes
Fixed Size                  9142832 bytes
Variable Size             805306368 bytes
Database Buffers         3456106496 bytes
Redo Buffers               24408064 bytes

SQL> create spfile from pfile='/tmp/initaux.ora';

File created.

SQL> startup nomount force
ORACLE instance started.

Total System Global Area 4287959600 bytes
Fixed Size                  9142832 bytes
Variable Size             838860800 bytes
Database Buffers         3422552064 bytes
Redo Buffers               17403904 bytes

SQL>

Now i can go ahead and try the duplication again, this time at CDB level because this allows me to specify a point in time.

[oracle@odax5ha0 ~]$ rman

Recovery Manager: Release 19.0.0.0.0 - Production on Tue Mar 21 10:28:43 2023
Version 19.16.0.0.0

Copyright (c) 1982, 2019, Oracle and/or its affiliates.  All rights reserved.

RMAN> connect target sys/********@odax5ha0:1521/mpmcdb1x5.support.robotron.de

connected to target database: MPMCDB1 (DBID=1547213674)

RMAN> connect auxiliary /

connected to auxiliary database (not started)

RMAN> duplicate database to CDBDUP pluggable database pdba until time "to_date('21.03.2023 09:10:00', 'dd.mm.yyyy hh24:mi:ss')";

Starting Duplicate Db at 2023-03-21 10:29:44
using target database control file instead of recovery catalog
allocated channel: ORA_AUX_DISK_1
channel ORA_AUX_DISK_1: SID=26 device type=DISK
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of Duplicate Db command at 03/21/2023 10:29:48
RMAN-05501: aborting duplication of target database
RMAN-06617: UNTIL TIME (2023-03-21 09:10:00) is ahead of last NEXT TIME in archived logs (2023-03-21 09:00:09)

Oh, look, another error. This is because there was not much DML happening and no log switch occured between the given point in time and right now. But this is easy to work around.

RMAN> alter system archive log current;

Statement processed

RMAN> duplicate database to CDBDUP pluggable database pdba until time "to_date('21.03.2023 09:10:00', 'dd.mm.yyyy hh24:mi:ss')";

Starting Duplicate Db at 2023-03-21 10:30:25
using channel ORA_AUX_DISK_1
duplicating Online logs to Oracle Managed File (OMF) location
duplicating Datafiles to Oracle Managed File (OMF) location

[...]

sql statement: alter pluggable database all open
Dropping offline and skipped tablespaces
Executing: alter database default tablespace system
Executing: drop tablespace "USERS" including contents cascade constraints
Finished Duplicate Db at 2023-03-21 10:34:50

RMAN>

The rest of the work is to unplug the PDB from the temporary auxiliary PDB, drop the auxiliary CDB and finally plug the PDB into the target CDB.

[oracle@odax5ha0 ~]$ sqlplus / as sysdba

SQL*Plus: Release 19.0.0.0.0 - Production on Tue Mar 21 10:41:30 2023
Version 19.16.0.0.0

Copyright (c) 1982, 2022, Oracle.  All rights reserved.


Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.16.0.0.0

SQL> select name from v$database;

NAME
---------
CDBDUP

SQL> show pdbs

    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         2 PDB$SEED                       READ ONLY  NO
         3 PDBA                           READ WRITE NO
SQL> alter pluggable database pdba close immediate;

Pluggable database altered.

SQL> alter pluggable database pdba unplug into '/tmp/pdba.xml';

Pluggable database altered.

SQL> drop pluggable database pdba keep datafiles;

Pluggable database dropped.

SQL> startup mount restrict exclusive force
ORACLE instance started.

Total System Global Area 4287959600 bytes
Fixed Size                  9142832 bytes
Variable Size             838860800 bytes
Database Buffers         3422552064 bytes
Redo Buffers               17403904 bytes
Database mounted.
SQL> drop database;

Database dropped.

Disconnected from Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.16.0.0.0

[oracle@odax5ha0 ~]$ export ORACLE_SID=MPMCDB2
[oracle@odax5ha0 ~]$ sqlplus / as sysdba

SQL*Plus: Release 19.0.0.0.0 - Production on Tue Mar 21 10:44:12 2023
Version 19.16.0.0.0

Copyright (c) 1982, 2022, Oracle.  All rights reserved.


Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.16.0.0.0

SQL> select name from v$database;

NAME
---------
MPMCDB2

SQL> create pluggable database PDBA using '/tmp/pdba.xml' copy tempfile reuse;

SQL> show pdbs

    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         2 PDB$SEED                       READ ONLY  NO
         3 PDBB                           READ WRITE NO
SQL> create pluggable database PDBA using '/tmp/pdba.xml' copy;

Pluggable database created.

SQL> alter pluggable database pdba open;

Pluggable database altered.

SQL> alter pluggable database pdba save state;

Pluggable database altered.

SQL> show pdbs

    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         2 PDB$SEED                       READ ONLY  NO
         3 PDBB                           READ WRITE NO
         4 PDBA                           READ WRITE NO

SQL>

And that’s just it. Easier than the other situation but not quite straight forward anyway. I think, this mght help in some situation.

Restore a dropped Pluggable Database

When reading the headline, the todays topic sounds not very complicated or special at all. But be warned, it definitely is. During a Migration-to-Multitenant project at a customers site we walked through quite a couple of restore scenarios. The most obvious scenario is dropping a PDB by accident. Of cause we have all the backups in place, that we will need to restore and recover the lost PDB.

Let’s start with what I did to the PDB:

[oracle@vm160 ~]$ sqlplus / as sysdba

SQL*Plus: Release 19.0.0.0.0 - Production on Thu Feb 23 13:54:28 2023
Version 19.18.0.0.0

Copyright (c) 1982, 2022, Oracle.  All rights reserved.

Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.18.0.0.0

SQL> show pdbs

    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         2 PDB$SEED                       READ ONLY  NO
         3 PDB1                           READ WRITE NO
         4 ORCL19PDB3                     READ WRITE NO
         5 ORCL19PDB2                     READ WRITE NO
SQL> alter session set container=pdb1;

Session altered.

SQL> select count(*) from hr.employees;

  COUNT(*)
----------
       108

SQL> alter session set container=cdb$root;

Session altered.

SQL> alter pluggable database pdb1 close immediate;

Pluggable database altered.

SQL> drop pluggable database pdb1 including datafiles;

Pluggable database dropped.

That was the easiest part of the story. You see, there were 3 PDBs, the dropped PDB had a table “EMPLOYEES” with 108 records. Now the simple goal ist to restore that PDB to right before the drop. To get that timestamp, we simply look it up in the alert.log:

[oracle@vm160 ~]$ tail /u01/app/oracle/diag/rdbms/orcl19/orcl19/trace/alert_orcl19.log
2023-02-23T13:55:01.355396+01:00
alter pluggable database pdb1 close immediate
2023-02-23T13:55:01.357622+01:00
PDB1(3):Pluggable database PDB1 closing
PDB1(3):JIT: pid 9707 requesting stop
PDB1(3):Closing sequence subsystem (3457133901505).
PDB1(3):Buffer Cache flush started: 3
2023-02-23T13:55:02.353510+01:00
PDB1(3):Buffer Cache flush finished: 3
Pluggable database PDB1 closed
Completed: alter pluggable database pdb1 close immediate
2023-02-23T13:55:14.816955+01:00
drop pluggable database pdb1 including datafiles
2023-02-23T13:55:17.338832+01:00
Deleted Oracle managed file /u01/app/oracle/oradata/ORCL19/ED7F830814134AADE053A024100A5D08/datafile/o1_mf_users_ksfwjspy_.dbf
Deleted Oracle managed file /u01/app/oracle/oradata/ORCL19/ED7F830814134AADE053A024100A5D08/datafile/o1_mf_temp_kq6nz8kv_.dbf
Deleted Oracle managed file /u01/app/oracle/oradata/ORCL19/ED7F830814134AADE053A024100A5D08/datafile/o1_mf_undotbs1_kq6nz8kv_.dbf
Deleted Oracle managed file /u01/app/oracle/oradata/ORCL19/ED7F830814134AADE053A024100A5D08/datafile/o1_mf_sysaux_kq6nz8kt_.dbf
Deleted Oracle managed file /u01/app/oracle/oradata/ORCL19/ED7F830814134AADE053A024100A5D08/datafile/o1_mf_system_kq6nz8kh_.dbf
2023-02-23T13:55:17.616908+01:00
Stopped service pdb1
Completed: drop pluggable database pdb1 including datafiles

Now that we identified the propper timestamp which is roughly 13:55, let’s start with RMAN:

[oracle@vm160 ~]$ rman target /

Recovery Manager: Release 19.0.0.0.0 - Production on Thu Feb 23 13:57:26 2023
Version 19.18.0.0.0

Copyright (c) 1982, 2019, Oracle and/or its affiliates.  All rights reserved.

connected to target database: ORCL19 (DBID=422142402)

RMAN> run {
2> set until time "to_date('23.02.2023 13:55:00', 'dd.mm.yyyy hh24:mi:ss')";
3> restore pluggable database pdb1;
4> recover pluggable database pdb1;
5> }

executing command: SET until clause

Starting restore at 2023-02-23 13:58:23
using target database control file instead of recovery catalog
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of restore command at 02/23/2023 13:58:24
ORA-01403: no data found

RMAN> restore pluggable database pdb1;

Starting restore at 2023-02-23 13:58:31
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=277 device type=DISK
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of restore command at 02/23/2023 13:58:31
RMAN-06813: could not translate pluggable database PDB1

Ok, neither can we do a point-in-time restore nor can we do a complete and current restore of the dropped PDB. This is because the controlfile is our single point of truth and it simply forgot everything about the PDB when I dropped it.

What we need to restore the PDB is an auxiliary instance to restore and recover the lost PDB. MOS Note “How to Restore – Dropped Pluggable database (PDB) in Multitenant (Doc ID 2034953.1)” is using DUPPLICATE for that, but this requires to have all neccessary backups in one location. So I chose a different approach. My auxiliary instance will be based on the original one, so I create a Pfile from it, that I will change according to my needs:

  • choose a new db_unique_name
  • remove “control_files” parameter
  • leave create-file-destinations unchanged
  • change memory parameters to minimum values
SQL> create pfile='/tmp/initaux.ora' from spfile;

File created.
*.audit_file_dest='/u01/app/oracle/admin/orcl19/adump'
*.audit_trail='db'
*.compatible='19.0.0'
*.container_data='ALL'
*.db_block_size=8192
*.db_create_file_dest='/u01/app/oracle/oradata'
*.db_domain='support.robotron.de'
*.db_name='orcl19'
*.db_unique_name='orcl19aux'
*.db_recovery_file_dest='/u01/app/oracle/fra'
*.db_recovery_file_dest_size=10494m
*.diagnostic_dest='/u01/app/oracle'
*.dispatchers='(PROTOCOL=TCP) (SERVICE=orcl19XDB)'
*.enable_pluggable_database=true
*.nls_language='AMERICAN'
*.nls_territory='AMERICA'
*.open_cursors=300
*.pga_aggregate_target=364m
*.processes=300
*.remote_login_passwordfile='EXCLUSIVE
*.sec_return_server_release_banner=TRUE
*.sga_target=2048m
*.undo_tablespace='UNDOTBS1'

Keeping the values for file destinations will restore the datafiles in their original paths in case of user-defined filenames. In case of OMF we will get new subdirectories according to the new db_unique_name. Either way the files will be created in the filesystem the original database is using.

With this Pfile I can now start my auxiliary instance and create a SPfile. Having a SPfile is handy because RMAN will update the “control_files” parameter accordingly when restoring the controlfiles. Of cause I have to set the ORACLE_SID to some value other than the original CDB.

[oracle@vm160 ~]$ export ORACLE_SID=aux
[oracle@vm160 ~]$ sqlplus / as sysdba

SQL*Plus: Release 19.0.0.0.0 - Production on Thu Feb 23 14:22:46 2023
Version 19.18.0.0.0

Copyright (c) 1982, 2022, Oracle.  All rights reserved.

Connected to an idle instance.

SQL> startup nomount pfile=/tmp/initaux.ora
ORACLE instance started.

Total System Global Area 2147482320 bytes
Fixed Size                  9165520 bytes
Variable Size             486539264 bytes
Database Buffers         1644167168 bytes
Redo Buffers                7610368 bytes

SQL> create spfile from pfile='/tmp/initaux.ora';

File created.

SQL> startup nomount force
ORACLE instance started.

Total System Global Area 2147482320 bytes
Fixed Size                  9165520 bytes
Variable Size             486539264 bytes
Database Buffers         1644167168 bytes
Redo Buffers                7610368 bytes

Having that, I can now restore and recover my PDB using RMAN. Note, since we only restore the one PDB, we have to skip all the other PDBs and the PDB$SEED during recovery.

[oracle@vm160 ~]$ rman target /

Recovery Manager: Release 19.0.0.0.0 - Production on Thu Feb 23 14:27:46 2023
Version 19.18.0.0.0

Copyright (c) 1982, 2019, Oracle and/or its affiliates.  All rights reserved.

connected to target database: ORCL19 (not mounted)

RMAN> set DBID 422142402

executing command: SET DBID

RMAN> run {
1      set until time "to_date('23.02.2023 13:55:00', 'dd.mm.yyyy hh24;mi:ss')";
2      set newname for database to new;
3      restore controlfile from '/u01/app/oracle/fra/ORCL19/autobackup/2023_02_23/o1_mf_s_1129557030_kzgr8qj7_.bkp';
4      alter database mount;
5      restore database root pluggable database pdb1;
6      switch datafile all;
7      switch tempfile all;
8      recover database skip forever tablespace
9      "ORCL19PDB3":"SYSTEM","ORCL19PDB3":"SYSAUX","ORCL19PDB3":"UNDOTBS1","ORCL19PDB2":"SYSTEM","ORCL19PDB2":"SYSAUX","ORCL19PDB2":"UNDOTBS1","PDB$SEED":"SYSTEM","PDB$SEED":"SYSAUX","PDB$SEED":"UNDOTBS1";
10 }

executing command: SET until clause

executing command: SET NEWNAME

Starting restore at 2023-02-23 14:51:50
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=256 device type=DISK

[...]

Starting recover at 2023-02-23 15:08:47
using channel ORA_DISK_1

Executing: alter database datafile 5, 6, 8 offline drop
Executing: alter database datafile 12, 13, 14 offline drop
Executing: alter database datafile 15, 16, 17 offline drop
starting media recovery

archived log for thread 1 with sequence 164 is already on disk as file /u01/app/oracle/fra/ORCL19/archivelog/2023_02_23/o1_mf_1_164_kzgr5rdp_.arc
archived log for thread 1 with sequence 165 is already on disk as file /u01/app/oracle/fra/ORCL19/onlinelog/o1_mf_3_kq6lk1ty_.log
archived log file name=/u01/app/oracle/fra/ORCL19/archivelog/2023_02_23/o1_mf_1_164_kzgr5rdp_.arc thread=1 sequence=164
archived log file name=/u01/app/oracle/fra/ORCL19/onlinelog/o1_mf_3_kq6lk1ty_.log thread=1 sequence=165
media recovery complete, elapsed time: 00:00:06
Finished recover at 2023-02-23 15:08:58

In case you have several PDBs to skip or if you do not like to type all the tablespaces by hand, you can use this handy SQL on the original database to generate the list of tablespaces to skip:

SQL> select listagg('"'||p.name||'":"'||t.tablespace_name||'"', ',')
  2  from v$pdbs p, cdb_tablespaces t
  3  where p.con_id=t.con_id
  4  and t.contents <> 'TEMPORARY'
SQL> /

LISTAGG('"'||P.NAME||'":"'||T.TABLESPACE_NAME||'"',',')
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
"ORCL19PDB3":"SYSTEM","ORCL19PDB3":"SYSAUX","ORCL19PDB3":"UNDOTBS1","ORCL19PDB2":"SYSTEM","ORCL19PDB2":"SYSAUX","ORCL19PDB2":"UNDOTBS1","PDB$SEED":"SYSTEM","PDB$SEED":"SYSAUX","PDB$SEED":"UNDOTBS1"

At this point we could open the auxiliary instance using the “open resetlogs” clause. But wait, we restored the controlfile from the original database so we have the same filenames for the redo members. And we are on the same host. Clearing those redo files might not be the best option. It would simply overwrite the redo files of our original database and we would end up doing another incomplete recovery… My solution is to create a new controlfile with modified redo members. As a basis I backup the controlfile to trace, change the db_name right away and restart my auxiliary instance to nomount.

SQL> alter database backup controlfile to trace as '/tmp/create_ctrl_aux.sql' resetlogs;

Database altered.

SQL> alter system set db_name="AUX" scope=spfile;

System altered.

SQL> startup nomount force
ORACLE instance started.

Total System Global Area 2147482320 bytes
Fixed Size                  9165520 bytes
Variable Size             486539264 bytes
Database Buffers         1644167168 bytes
Redo Buffers                7610368 bytes

Then I modified the “create controlfile” statement

  • set the new database name
  • remove all the files, that I didn’t restore
  • remove redo file names (OMF) – or – change redo file names

The final statement is this:

CREATE CONTROLFILE REUSE set DATABASE "AUX" RESETLOGS  ARCHIVELOG
    MAXLOGFILES 16
    MAXLOGMEMBERS 3
    MAXDATAFILES 1024
    MAXINSTANCES 8
    MAXLOGHISTORY 292
LOGFILE
  GROUP 1 SIZE 200M BLOCKSIZE 512,
  GROUP 2  SIZE 200M BLOCKSIZE 512,
  GROUP 3 SIZE 200M BLOCKSIZE 512
-- STANDBY LOGFILE
DATAFILE
  '/u01/app/oracle/oradata/ORCL19AUX/datafile/o1_mf_system_kzgvw29p_.dbf',
  '/u01/app/oracle/oradata/ORCL19AUX/datafile/o1_mf_sysaux_kzgvw291_.dbf',
  '/u01/app/oracle/oradata/ORCL19AUX/datafile/o1_mf_undotbs1_kzgvw2b9_.dbf',
  '/u01/app/oracle/oradata/ORCL19AUX/datafile/o1_mf_users_kzgvw2f8_.dbf',
  '/u01/app/oracle/oradata/ORCL19AUX/ED7F830814134AADE053A024100A5D08/datafile/o1_mf_system_kzgw69jf_.dbf',
  '/u01/app/oracle/oradata/ORCL19AUX/ED7F830814134AADE053A024100A5D08/datafile/o1_mf_sysaux_kzgw69jz_.dbf',
  '/u01/app/oracle/oradata/ORCL19AUX/ED7F830814134AADE053A024100A5D08/datafile/o1_mf_undotbs1_kzgw69kb_.dbf',
  '/u01/app/oracle/oradata/ORCL19AUX/ED7F830814134AADE053A024100A5D08/datafile/o1_mf_users_kzgw69h9_.dbf'
CHARACTER SET AL32UTF8
;

Finally, I create a new controlfile, open the auxiliary database and unplug the PDB keeping their datafiles.

SQL> CREATE CONTROLFILE REUSE set DATABASE "AUX" RESETLOGS  ARCHIVELOG
[...]

Control file created.

SQL> alter database open resetlogs;

Database altered.

SQL> show pdbs

    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         2 PDB$SEED                       MOUNTED
         3 PDB1                           MOUNTED
         4 ORCL19PDB3                     MOUNTED
         5 ORCL19PDB2                     MOUNTED
SQL> alter pluggable database PDB1 unplug into '/tmp/pdb1.xml';

Pluggable database altered.

SQL> drop pluggable database PDB1 keep datafiles;

Pluggable database dropped.

At this point we can transfer the restored PDB back to its original container. You can decide, if you want to use the restored datafiles right away or if you create another copy of them. Either way it is a simple one-liner.

SQL> create pluggable database PDB1 using '/tmp/pdb1.xml' copy;

Pluggable database created.

Or

SQL> create pluggable database PDB1 using '/tmp/pdb1.xml' nocopy tempfile reuse;

Pluggable database created.

Last step is to open the PDB again and check it’s contents.

SQL> alter  pluggable database PDB1 open;

Pluggable database altered.

SQL> alter  pluggable database PDB1 save state;

Pluggable database altered.

SQL> alter session set container=pdb1;

Session altered.

SQL> select count(*) from hr.employees;

  COUNT(*)
----------
       108

Et voila, missing accomplished. The very last step now is removing the auxiliary instance.

SQL> startup mount restrict exclusive force

SQL> drop database;

Whoever said, Multitenant will make DBA’s life easier did not do such a restore 🙂 … But to be fair, there are lot of scenarios where Multitenant is very handy, I must admit.

Insufficient SGA with RMAN automatic auxiliary instances

RMAN is the way to go when it comes to backup and restore. The tool was introduced with Release 8 of the Oracle Database, as far as I remember. And since then, it was heavily improved over time. One of the newer features is the native support for table and tablespace point-in-time recovery. It used to be a manual process in the past, we needed to restore a set of tablespaces to an auxiliary instance, recover it and then transfer the table or the tablespace in question to the source database using Datapump for instance.

This pocess was enhanced and now RMAN can handly the neccessary auxiliary instance on it’s own. It creates it, uses it for restore and recover and drops it afterwards. No magic here. But there is one thing to keep in mind. The auxiliary instance inherits some settings from the target database, the SGA_TARGET parameter for instance. I tested it with Release 19c and several RU’s, it was always the same but may change in the future. Let’s create a testcase, I have a database, create a new tablespace TEST_TS and put a table SOURCE_COPY in that tablespace, just to have it non-empty.

[odaadmin@odax5-base-00 ~]$ sqlplus / as sysdba

SQL*Plus: Release 19.0.0.0.0 - Production on Wed Jun 22 08:58:03 2022
Version 19.14.0.0.0

Copyright (c) 1982, 2021, Oracle.  All rights reserved.


Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.14.0.0.0

SQL> create tablespace test_ts datafile size 1g;

Tablespace created.

SQL> create table system.source_copy tablespace test_ts as select * from dba_source;

Table created.

Then I create a backup of cause. After creating that backup, I simply drop the tablespace TEST_TS and check the alert.log for the timestamp when the tablespace was dropped. That will be my Recovery Point Objective for the tablespace point-in-time restore.

SQL> drop tablespace test_ts including contents and datafiles;

Tablespace dropped.

SQL> exit

[odaadmin@odax5-base-00 ~]$ tail -f alert.log
2022-06-22 09:01:16.838000 +02:00
drop tablespace test_ts including contents and datafiles
2022-06-22 09:01:22.601000 +02:00
Deleted Oracle managed file /u02/app/oracle/oradata/MPMX5/MPMX5/datafile/o1_mf_test_ts_kc5hd4vf_.dbf
Completed: drop tablespace test_ts including contents and datafiles

Now I try to perform the RMAN tablespace point-in-time recovery. There are several options to that command, see the documentation, but for my case I just specify the location for the datafiles of my automatic auxiliary instance.

[odaadmin@odax5-base-00 ~]$ rman target /

Recovery Manager: Release 19.0.0.0.0 - Production on Wed Jun 22 09:05:53 2022
Version 19.14.0.0.0

Copyright (c) 1982, 2019, Oracle and/or its affiliates.  All rights reserved.

connected to target database: MPM (DBID=3407247611)

RMAN> recover tablespace test_ts
2> until time "to_date('22.06.2022 09:01:00', 'dd.mm.yyyy hh24:mi:ss')"
3> auxiliary destination '/u02/app/oracle/oradata/MPMX5/MPMX5/datafile';

Starting recover at 2022-06-22 09:05:56
using target database control file instead of recovery catalog
current log archived
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=216 instance=MPM1 device type=DISK
RMAN-05026: warning: presuming following set of tablespaces applies to specified point-in-time

List of tablespaces expected to have UNDO segments
Tablespace SYSTEM
Tablespace MPMPDB1:SYSTEM
Tablespace UNDOTBS1
Tablespace MPMPDB1:UNDOTBS1
Tablespace UNDOTBS2
Tablespace MPMPDB1:UNDO_2

Creating automatic instance, with SID='qAph'

initialization parameters used for automatic instance:
db_name=MPM
db_unique_name=qAph_pitr_MPM
compatible=19.0.0.0
db_block_size=8192
db_files=1024
diagnostic_dest=/u01/app/odaorabase/odaadmin
_system_trig_enabled=FALSE
sga_target=98074M
processes=200
db_create_file_dest=/u02/app/oracle/oradata/MPMX5/MPMX5/datafile
log_archive_dest_1='location=/u02/app/oracle/oradata/MPMX5/MPMX5/datafile'
enable_pluggable_database=true
_clone_one_pdb_recovery=true
#No auxiliary parameter file used


starting up automatic instance MPM

RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of recover command at 06/22/2022 09:06:35
RMAN-04014: startup failed: ORA-27106: system pages not available to allocate memory
Additional information: 6187
Additional information: 1

So the recovery fails, because the server lacks RAM to create the auxiliary instance. Let’s check the SGA of the target instance and the RAM.

[odaadmin@odax5-base-00 ~]$ sqlplus / as sysdba
Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.14.0.0.0

SQL> show parameter sga

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
allow_group_access_to_sga            boolean     FALSE
lock_sga                             boolean     FALSE
pre_page_sga                         boolean     TRUE
sga_max_size                         big integer 4G
sga_min_size                         big integer 0
sga_target                           big integer 96G
unified_audit_sga_queue_size         integer     1048576

SQL> !free -h
              total        used        free      shared  buff/cache   available
Mem:           251G        147G         77G        3.4G         27G         93G
Swap:           23G          0B         23G

So there’s just not enough memory available (or hugepages in my case) to startup the auxiliary instance. This is, because RMAN chooses the same SGA size for the auxiliary instance. This is documented behaviour, by the way. Beside the fact that Oracle states, it is usualy not neceessary to override the default, I stumbled quite a lot over this. Now, what can we do to override the defaults? There are two ways:

  1. Create a custom initialization parameter file for the auxiliary instance named “params_auxinst.ora” and place it in $ORACLE_HOME/rdbms/admin
  2. Create a custom initialization parameter file for the auxiliary instance and explicitly point RMAN to that file

For me, the latter one is the better approach because it is more obvious when logging the RMAN commands for documentation purposes for instance. Let’s see.

[odaadmin@odax5-base-00 ~]$ cat /tmp/initRMAN.ora
sga_target=4g

[odaadmin@odax5-base-00 ~]$ rman target /

Recovery Manager: Release 19.0.0.0.0 - Production on Wed Jun 22 09:05:53 2022
Version 19.14.0.0.0

Copyright (c) 1982, 2019, Oracle and/or its affiliates.  All rights reserved.

connected to target database: MPM (DBID=3407247611)

RMAN> set auxiliary instance parameter file to '/tmp/initRMAN.ora';

executing command: SET auxiliary parameter file
using target database control file instead of recovery catalog

RMAN> recover tablespace test_ts
2> until time "to_date('22.06.2022 09:01:00', 'dd.mm.yyyy hh24:mi:ss')"
3> auxiliary destination '/u02/app/oracle/oradata/MPMX5/MPMX5/datafile';

Starting recover at 2022-06-22 09:35:08
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=222 instance=MPM1 device type=DISK
RMAN-05026: warning: presuming following set of tablespaces applies to specified point-in-time

List of tablespaces expected to have UNDO segments
Tablespace SYSTEM
Tablespace MPMPDB1:SYSTEM
Tablespace UNDOTBS1
Tablespace MPMPDB1:UNDOTBS1
Tablespace UNDOTBS2
Tablespace MPMPDB1:UNDO_2

Creating automatic instance, with SID='pjtg'
using contents of file /tmp/initRMAN.ora

initialization parameters used for automatic instance:
db_name=MPM
db_unique_name=pjtg_pitr_MPM
compatible=19.0.0.0
db_block_size=8192
db_files=1024
diagnostic_dest=/u01/app/odaorabase/odaadmin
_system_trig_enabled=FALSE
sga_target=96G
processes=200
db_create_file_dest=/u02/app/oracle/oradata/MPMX5/MPMX5/datafile
log_archive_dest_1='location=/u02/app/oracle/oradata/MPMX5/MPMX5/datafile'
enable_pluggable_database=true
_clone_one_pdb_recovery=true
ifile=/tmp/initRMAN.ora


starting up automatic instance MPM

Oracle instance started

Total System Global Area    4294963752 bytes

Fixed Size                     9142824 bytes
Variable Size                855638016 bytes
Database Buffers            3388997632 bytes
Redo Buffers                  41185280 bytes
Automatic instance created


List of tablespaces that have been dropped from the target database:
Tablespace TEST_TS

contents of Memory Script:
{
# set requested point in time
set until  time "to_date('22.06.2022 09:01:00', 'dd.mm.yyyy hh24:mi:ss')";
# restore the controlfile
restore clone controlfile;

# mount the controlfile
sql clone 'alter database mount clone database';

[...]

Removing automatic instance
Automatic instance removed
auxiliary instance file /u02/app/oracle/oradata/MPMX5/MPMX5/datafile/MPMX5/datafile/o1_mf_temp_kc5knl39_.tmp deleted
auxiliary instance file /u02/app/oracle/oradata/MPMX5/MPMX5/datafile/MPMX5/onlinelog/o1_mf_4_kc5kn4z4_.log deleted
auxiliary instance file /u02/app/oracle/oradata/MPMX5/MPMX5/datafile/MPMX5/onlinelog/o1_mf_3_kc5kn4x5_.log deleted
auxiliary instance file /u02/app/oracle/oradata/MPMX5/MPMX5/datafile/MPMX5/onlinelog/o1_mf_2_kc5kn4vf_.log deleted
auxiliary instance file /u02/app/oracle/oradata/MPMX5/MPMX5/datafile/MPMX5/onlinelog/o1_mf_1_kc5kn4so_.log deleted
auxiliary instance file /u02/app/oracle/oradata/MPMX5/MPMX5/datafile/MPMX5/datafile/o1_mf_sysaux_kc5km3nb_.dbf deleted
auxiliary instance file /u02/app/oracle/oradata/MPMX5/MPMX5/datafile/MPMX5/E203D75664DE289CE0533E24100A3F14/datafile/o1_mf_undo_2_kc5kmdm9_.dbf deleted
auxiliary instance file /u02/app/oracle/oradata/MPMX5/MPMX5/datafile/MPMX5/datafile/o1_mf_undotbs2_kc5km3w5_.dbf deleted
auxiliary instance file /u02/app/oracle/oradata/MPMX5/MPMX5/datafile/MPMX5/E203D75664DE289CE0533E24100A3F14/datafile/o1_mf_undotbs1_kc5kmd8w_.dbf deleted
auxiliary instance file /u02/app/oracle/oradata/MPMX5/MPMX5/datafile/MPMX5/datafile/o1_mf_undotbs1_kc5km3v2_.dbf deleted
auxiliary instance file /u02/app/oracle/oradata/MPMX5/MPMX5/datafile/MPMX5/E203D75664DE289CE0533E24100A3F14/datafile/o1_mf_system_kc5kmcdv_.dbf deleted
auxiliary instance file /u02/app/oracle/oradata/MPMX5/MPMX5/datafile/MPMX5/datafile/o1_mf_system_kc5km3sg_.dbf deleted
auxiliary instance file /u02/app/oracle/oradata/MPMX5/MPMX5/datafile/MPMX5/controlfile/o1_mf_kc5klpq0_.ctl deleted
auxiliary instance file tspitr_pjtg_29120.dmp deleted
Finished recover at 2022-06-22 09:37:29

As we can see, the recovery went smooth this time. But we also see, that RMAN still uses the SGA_TARGET from the target database but now simply adds my custom parameter file with the IFILE option. The parameters from the IFILE then takes precedemce over the default values. There’s just one small thing in the docs, the diagram correctly says “AUXILIARY INSTANCE PARMETER FILE TO” whilst the description states “AUXILIARY INSTANCE PARMETERFILE TO” which is wrong. But now let’s crosscheck the memory usage during the recovery.

[odaadmin@odax5-base-00 ~]$ ps -ef |grep smon
root     10888     1  2 Jun10 ?        07:28:06 /u01/app/19.15.0.0/grid/bin/osysmond.bin
grid     13237     1  0 Jun10 ?        00:00:12 asm_smon_+ASM1
oracle   29865     1  0 Jun10 ?        00:00:16 ora_smon_mwe191
odaadmin 29930     1  0 Jun10 ?        00:00:15 ora_smon_sirac191
odaadmin 82423     1  0 08:56 ?        00:00:00 ora_smon_MPM1
odaadmin 86915     1  0 09:32 ?        00:00:00 ora_smon_FB2T1
odaadmin 92942     1  0 09:35 ?        00:00:00 ora_smon_pjtg
root     93881  7723  0 09:36 pts/0    00:00:00 grep --color=auto smon

[odaadmin@odax5-base-00 ~]$ export ORACLE_HOME=/u01/app/odaorabase/odaadmin
[odaadmin@odax5-base-00 ~]$ export ORACLE_SID=pjtg
[odaadmin@odax5-base-00 ~]$ sqlplus / as sysdba

SQL*Plus: Release 19.0.0.0.0 - Production on Wed Jun 22 09:36:24 2022
Version 19.14.0.0.0

Copyright (c) 1982, 2021, Oracle.  All rights reserved.


Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.14.0.0.0

SQL> show parameter sga_ta

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
sga_target                           big integer 4G

That’s just to prove, the auxiliary instance really did pick up the customized parameters. Ok, that’s all for today, folks 🙂

Rolling forward a Physical Standby

Today I am going to write about one of my favorite new features for Oracle Data Guard which was introduced with version 12.1. For this post I refer to primary and phyiscal standby database simply as primary and standby for the sake of readability. So normally a standby will receive the redo stream in some kind from the primary and immediatly applies the redo to it’s datafiles. That way the standby is always up-to-date as good as it can. But things happen, the standby might be out of sync with the primary. Either due to NOLOGGING operations which can be prevented by enabling the FORCE LOGGING mode or because the redo stream has gaps for some reason. In such a case a reinstate operation of the standby is required which is basically doing a complete new copy of the primary. This can be time consuming and resource intensive depending on the size of the database.

So what can be done to shorten this procedure? The answer is simple, create and apply an incremental backup. The idea behind that is this, we know the SCN at which the standby stopped applying redo. This SCN can be used as a starting point for an incremental backup. This incremental backup will then contain all the blocks that have a SCN higher than the one of the standby. In the next step this backup can be applied to the standby refreshing all the data blocks that have changed meanwhile. Maybe a controlfile refresh is necessary too in case some datafiles where added. This method was a manual process in 11.2.

Starting with 12.1 the whole thing became much more easier because the “restore/recover from service” option for RMAN was introduced. Let me walk through a simple example. In the example the standby was disabled in the Data Guard configuration. When re-enabling it, a gap in the redo stream was detected.

DGMGRL> show configuration

Configuration - mes19

  Protection Mode: MaxPerformance
  Members:
  mes19dc1 - Primary database
    mes19dc2 - Physical standby database (disabled)
      ORA-16749: The member was disabled manually.

Fast-Start Failover:  Disabled

Configuration Status:
SUCCESS   (status updated 42 seconds ago)

DGMGRL> enable database mes19dc2;
Enabled.
DGMGRL>  show configuration

Configuration - mes19

  Protection Mode: MaxPerformance
  Members:
  mes19dc1 - Primary database
    Error: ORA-16724: cannot resolve gap for one or more members

    mes19dc2 - Physical standby database
      Error: ORA-16810: multiple errors or warnings detected for the member

Fast-Start Failover:  Disabled

Configuration Status:
ERROR   (status updated 8 seconds ago)

Since this were RAC databases, I needed to stop the whole database and start with a single instance to perform the recovery.

$ srvctl stop database -db mes19dc2


SQL> startup nomount

Now the fun begins. At first I refreshed to controlfile using the new syntax. The only requirement is, that your SYS passwords are the same for target and source. But for a Data Guard setup this should be not an issue.

$ rman target /

Recovery Manager: Release 19.0.0.0.0 - Production on Tue Feb 18 15:44:42 2020
Version 19.5.0.0.0

Copyright (c) 1982, 2019, Oracle and/or its affiliates.  All rights reserved.

connected to target database: MES19 (not mounted)

RMAN> restore standby controlfile from service 'rboda0101x:1521/mes19dc1';

Starting restore at 18-FEB-20
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=206 instance=mes191 device type=DISK

channel ORA_DISK_1: starting datafile backup set restore
channel ORA_DISK_1: using network backup set from service rboda0101x:1521/mes19dc1.rb.de.conti.de
channel ORA_DISK_1: restoring control file
channel ORA_DISK_1: restore complete, elapsed time: 00:00:02
output file name=+REDO/MES19DC2/CONTROLFILE/current.279.1026811235
Finished restore at 18-FEB-20

Now I was able to mount the standby again. But since the controlfile is a restore from primary, all the paths to datafiles are now the ones that came from the primary. The solution is simply cataloging everything that was part of standby and switch datafiles.

RMAN> alter database mount;

released channel: ORA_DISK_1
Statement processed

RMAN> catalog start with '+DATA';

Starting implicit crosscheck backup at 18-FEB-20
allocated channel: ORA_DISK_1
allocated channel: ORA_DISK_2
allocated channel: ORA_DISK_3

...

RMAN> switch database to copy;

At this point I can start recovering the standby using the primary as the source for an incremental backup. The whole procedure, that I described earlier, is done automatically by RMAN using one single command.

RMAN> recover database from service 'rboda0101x:1521/mes19dc1';

This typically works straight away. Unfortunately I was using version 19.5 which obviously has a bug. It creates huge traces. And I mean really huge. These traces filled up my filesystem hosting the ADR. The traces contained something like this.

-- key in target file:
kcbtse_print_kod: printing kcbtekod structure 0x7fc77f1ae310
--kcbtekod_ena=0 kcbtekod_flag=0x0 kcbtekod_mkloc=0x0
--kcbtekod_key=0000000000000000000000000000000000000000000000000000000000000000
--kcbtekod_mkeyid=00000000000000000000000000000000
-- key in backup set:
kcbtse_print_kod: printing kcbtekod structure 0x7fc77fd61cf8
--kcbtekod_ena=2 kcbtekod_flag=0x6 kcbtekod_mkloc=0x0
--kcbtekod_key=ecc8c60f4de9fc8731ef019f080b17db00000000000000000000000000000000
--kcbtekod_mkeyid=9c5ac94c63844f08bf8cd818eb980ad7
-- key in target file:
kcbtse_print_kod: printing kcbtekod structure 0x7fc77f1ae310
--kcbtekod_ena=0 kcbtekod_flag=0x0 kcbtekod_mkloc=0x0
--kcbtekod_key=0000000000000000000000000000000000000000000000000000000000000000
--kcbtekod_mkeyid=00000000000000000000000000000000
-- key in backup set:
kcbtse_print_kod: printing kcbtekod structure 0x7fc77fd61cf8
--kcbtekod_ena=2 kcbtekod_flag=0x6 kcbtekod_mkloc=0x0
--kcbtekod_key=ecc8c60f4de9fc8731ef019f080b17db00000000000000000000000000000000
--kcbtekod_mkeyid=9c5ac94c63844f08bf8cd818eb980ad7

These lines were repeated over and over again. My simple, quick and dirty solution was to get rid of these traces once in a while.

mes191:root->oracle@rboda0201x:/u01/app/oracle/diag/rdbms/mes19dc2/mes191/trace
$ while true; do for i in $(ls mes191*trc); do echo ""> $i; done; sleep 60; done

Having that in place, my recover command in RMAN completed successfully.

channel ORA_DISK_1: starting incremental datafile backup set restore
channel ORA_DISK_1: using network backup set from service rboda0101x:1521/mes19dc1
destination for restore of datafile 00025: +DATA/MES19DC2/996EF5BF4924EC53E05328DED70AA5CE/DATAFILE/mes_data.314.1029229915
channel ORA_DISK_2: starting incremental datafile backup set restore
channel ORA_DISK_2: using network backup set from service rboda0101x:1521/mes19dc1
destination for restore of datafile 00026: +DATA/MES19DC2/996EF5BF4924EC53E05328DED70AA5CE/DATAFILE/mes_index.313.1029229915
channel ORA_DISK_2: restore complete, elapsed time: 01:01:25
channel ORA_DISK_1: restore complete, elapsed time: 01:22:26

starting media recovery

archived log for thread 1 with sequence 610 is already on disk as file +RECO/MES19DC2/ARCHIVELOG/2020_02_18/thread_1_seq_610.260.1032709543
archived log for thread 1 with sequence 611 is already on disk as file +RECO/MES19DC2/ARCHIVELOG/2020_02_18/thread_1_seq_611.381.1032713241
archived log for thread 1 with sequence 612 is already on disk as file +RECO/MES19DC2/ARCHIVELOG/2020_02_18/thread_1_seq_612.261.1032713243
archived log for thread 2 with sequence 741 is already on disk as file +RECO/MES19DC2/ARCHIVELOG/2020_02_18/thread_2_seq_741.380.1032709545
archived log for thread 2 with sequence 742 is already on disk as file +RECO/MES19DC2/ARCHIVELOG/2020_02_18/thread_2_seq_742.383.1032713241
archived log for thread 2 with sequence 743 is already on disk as file +RECO/MES19DC2/ARCHIVELOG/2020_02_18/thread_2_seq_743.257.1032713245
archived log file name=+RECO/MES19DC2/ARCHIVELOG/2020_02_18/thread_1_seq_610.260.1032709543 thread=1 sequence=610
archived log file name=+RECO/MES19DC2/ARCHIVELOG/2020_02_18/thread_2_seq_741.380.1032709545 thread=2 sequence=741
archived log file name=+RECO/MES19DC2/ARCHIVELOG/2020_02_18/thread_1_seq_611.381.1032713241 thread=1 sequence=611
archived log file name=+RECO/MES19DC2/ARCHIVELOG/2020_02_18/thread_2_seq_742.383.1032713241 thread=2 sequence=742
archived log file name=+RECO/MES19DC2/ARCHIVELOG/2020_02_18/thread_1_seq_612.261.1032713243 thread=1 sequence=612
archived log file name=+RECO/MES19DC2/ARCHIVELOG/2020_02_18/thread_2_seq_743.257.1032713245 thread=2 sequence=743
media recovery complete, elapsed time: 00:00:01
Finished recover at 18-FEB-20

There is one more thing to do at this point. All the Online and Standby Redo Logs inherited their names from the primary too. So I had to reset them to something that fits the standby. For non-OMF you would do an “alter database rename file”, for OMF I did it like this.

SQL> r
  1* select distinct  'alter database clear logfile group ' || group# || ';' from v$logfile

'ALTERDATABASECLEARLOGFILEGROUP'||GROUP#||';'
----------------------------------------------------------------------------
alter database clear logfile group 1;
alter database clear logfile group 21;
alter database clear logfile group 22;
alter database clear logfile group 23;
alter database clear logfile group 24;
alter database clear logfile group 25;
alter database clear logfile group 26;
alter database clear logfile group 27;
alter database clear logfile group 28;
alter database clear logfile group 29;
alter database clear logfile group 2;
alter database clear logfile group 30;
alter database clear logfile group 3;
alter database clear logfile group 4;
alter database clear logfile group 5;
alter database clear logfile group 6;
alter database clear logfile group 7;
alter database clear logfile group 8;

18 rows selected.

And then of cause execute all these commands. Finally I just enabled the Data Guard configuration again and checked the state.

DGMGRL> show configuration

Configuration - mes19

  Protection Mode: MaxPerformance
  Members:
  mes19dc1 - Primary database
    mes19dc2 - Physical standby database (disabled)
      ORA-16749: The member was disabled manually.

Fast-Start Failover:  Disabled

Configuration Status:
SUCCESS   (status updated 22 seconds ago)

DGMGRL> enable database mes19dc2;
Enabled.

DGMGRL> show configuration

Configuration - mes19

  Protection Mode: MaxPerformance
  Members:
  mes19dc1 - Primary database
    mes19dc2 - Physical standby database

Fast-Start Failover:  Disabled

Configuration Status:
SUCCESS   (status updated 30 seconds ago)

DGMGRL> show database mes19dc2

Database - mes19dc2

  Role:               PHYSICAL STANDBY
  Intended State:     APPLY-ON
  Transport Lag:      0 seconds (computed 1 second ago)
  Apply Lag:          0 seconds (computed 1 second ago)
  Average Apply Rate: 1.26 MByte/s
  Real Time Query:    OFF
  Instance(s):
    mes191 (apply instance)
    mes192

Database Status:
SUCCESS

You may wait up to a minute to get the current status reflect in the output of DGMGRL, so be patient.

Another thing which I did not show in this is example is the handling of new datafiles. For that you would just issue a “restore from service” before doing the actual recovery.

I really do like this new (or in the meantime not so new anymore) feature and I hope it’ll help you (and me) to speed up resynchronisation of your standby in the future.

More information about this can be found in the Docs or MOS Note 1987763.1.

Platform change and upgrade in one step – XTTS

Today I want to tell you something about cross platform transportable tablespaces (XTTS) and incrementally updated datafile copies. This is a very nice feature to move to another platform and change the Oracle version in one step. There are several MOS notes, whitepapers and of cause some Upgrade-Slides by Mike Dietrich covering this topic.

The basic steps in all cases and versions are these:

  1. set tablespace(s) read only
  2. export metadata for those tablespaces
  3. transport datafiles and dump to target system
  4. create necessary database users
  5. import dump into target database which makes the datafiles part of the target database
  6. set tablespace(s) read write
  7. optionally, transport other objects like views, PL/SQL etc.

But what if the tablespaces are too big and copy time would exceed the allowed downtime window? Then we can use incremental backups to shorten the downtime. The steps to do this, are quite similar.

  1. backup tablespace(s) for transport
  2. copy backup to target system
  3. restore and convert datafiles to target destination and platform
  4. create incremental backup for transported tablespace(s)
  5. recover transported datafiles using the incremental backup
  6. repeat steps 4 and 5 until final downtime window
  7. set tablespace(s) read only
  8. create final incremental backup for transported tablespace(s)
  9. apply incremental backup to transported datafiles
  10. continue with step 2 of the initial process above

Sounds a bit complicated, doesn’t it? So Oracle was so kind to put this whole process into scripts. The following MOS notes contain and describe these scripts:

  • 11G – Reduce Transportable Tablespace Downtime using Cross Platform Incremental Backup (Doc ID 1389592.1)
  • 12C – Reduce Transportable Tablespace Downtime using Cross Platform Incremental Backup (Doc ID 2005729.1)
  • V4 Reduce Transportable Tablespace Downtime using Cross Platform Incremental Backup (Doc ID 2471245.1)

During a migration project from Linux to SPARC (don’t ask) I wanted to use those scripts. But I experienced the following during “Phase 2”:

oracle@server2:~/xtts> $ORACLE_HOME/perl/bin/perl xttdriver.pl --backup
============================================================
trace file is /oracle/xtts/backup_Sep20_Thu_12_20_52_917//Sep20_Thu_12_20_52_917_.log
=============================================================

[…]

--------------------------------------------------------------------
Done with prepare phase
--------------------------------------------------------------------


--------------------------------------------------------------------
Find list of datafiles in system
--------------------------------------------------------------------

SP2-0027: Input is too long (> 2499 characters) - line ignored
SP2-0027: Input is too long (> 2499 characters) - line ignored
           from dual
           *
ERROR at line 19:
ORA-00936: missing expression

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Error:
------
Error in executing /oracle/xtts/backup_Sep20_Thu_12_27_07_889//diff.sql

Okay. The reason is, I have loads of tablespaces that need to be transferred. This list messed up the whole process. So I tried to understand the process and just do it myself. What I learned is, that it is quite easy to do it manually. And I share my findings with you by simply outlining the basic process in an upcoming blog post.