11
Restoring Fault Tolerance< /h1>

This chapter describes how to restore redundancy to your environment aft er a failure. It includes the following topics:

• Restoring Full Tolerance
• Res toring Failed Nodes or Instances in a RAC Cluster
• Restoring the Standby Database After a Failover
• Restoring Fault Tolerance after Secondary Site or Clusterwide Scheduled Outage
• Restoring Fault Tolerance after a Standby Database Data Failure
• Restoring Fault Tolerance After the Production Database Has Opened R esetlogs
• Restoring Fault Tolerance after Dua l Failures

Mounting the new standby database requires the following substeps:

    ALTER DATABASE FLASH
BACK OFF;

    ALTER DATABASE CREATE STANDBY CONTROLFILE AS controlfile_name;
    SHUTDOWN IMMEDIATE;

    STARTUP MOUNT;
 

    LSNRCTL STAT list_name;

Step 4P: A rchive to New Standby Database From New Production Database

Before t he new standby database was created, the current production remote standby archive destination probably stopped with an error and is no longer shipping files to the remote destination. To restart remote archiving, you may have to reenable the standby archive destina tion.

Query the V$ARCHIVE_DEST_STATUS view to see the current state of the arc hive destinations.

SELECT DEST_ID, DEST_NAME, STATUS, PROTECTION_MODE, DESTINATION, ERROR,
SRL
    FROM V$ARCHIVE_DEST_STATUS;

Enable the remote archive destination.

ALTER SYSTEM SET LOG_ARCHIVE_DEST_STATE_
n=ENABLE;

Switch a redo log file and verify that it was sent successfully.

ALTER SYSTEM SWITCH LOGFILE;
SELECT DEST_ID, DEST_NAME
, STATUS, PROTECTION_MODE, DESTINATION, ERROR, SRL
    FROM V$ARCHIVE_DEST_STATUS;

Shipping the archived redo log from the new production database notifies the standby database of the new production database incarnation number.

Step 5P: Start Managed Recovery

Start managed recovery or real-time apply managed recovery with one of the following statements:

ALTER DATABASE RECOVER MANAGED STANDBY DATABASE DISCONNECT;

or

ALTER DATABASE RECOVER MANAGED STANDBY DATABASE USING CURRENT LOGFILE 
DISCONNEC
T;

Ensure that recovery is applying the archived redo logs.

Step 6P: Restart MRP After It Encounters the End-of-Redo Marker

The managed recovery process (MRP) stops after it encounters the end-of-r edo marker that demarcates when the Data Guard failover was completed in the redo stream. This is not an error. Restart MRP, and it w ill continue with no problem.

Restoring a Logical Standby Database After a Failover

The following steps are required to restore a logical standby database after a failover:

• Step 1L: Retrieve END_PRIMARY_SCN
• Step 2L: Flash Back the Previous Production Database
• Step 3L: Open New Logical Standby Database and Start SQL Apply

Step 1 L: Retrieve END_PRIMARY_SCN

On the new production database, query fo r the SCN at which the previous standby database became the new production database.

SELECT
 VALUE AS TO_CHAR(STANDBY_BECAME_PRIMARY_SCN) FROM DBA_LOGSTDBY_PARAMETERS
WHERE NAME = 'END_PRIMARY_SCN';



Step 2L: F
lash Back the Previous Production Database

 
You can create a new logi
cal standby database by mounting the previous production database, flashing it back to STANDBY_BECAME_PRIMARY_SCN, and t
hen enabling the previous guard level setting.
SHUTDOWN IMMEDIATE;
ST
ARTUP MOUNT;
FLASHBACK DATABASE TO SCN standby_became_primary_scn;
ALTER DATABASE GUARD [
ALL | STANDBY | NONE];



Step 3L: Open New Logical Standby Database and Start SQL Apply

ALTER DATABASE OPEN RESETLOGS;
ALTER DATABASE START LOGICAL STANDBY APPLY NEW PRIMARY dblink;




You need to create a database link from the ne
w logical standby database to the new production database if it does not already exist. Use the following syntax:
CREATE PUBLIC DATABASE LINK dblink
  CONNECT TO system IDENTIFIED BY password
  USING 'service_name_of_new_p
rimary_database';



Restoring Fault Tolerance after Secondary Site or Clusterwide Scheduled Outage

 
The following steps are required to restore full fault tolerance after a scheduled secondary site or clus
terwide outage:

Step 1: Start th
e Standby Database
Step 2: Start Recovery
Step 3: Verify Log Transport Services on Produ
ction Database
Step 4: Verify that Recovery
is Progressing on Standby Database
Step 5: R
estore Production Database Protection Mode
 
Step 1: Start the Standby Database

 
You may have to restore the standby database from local backups, local tape backups, or from the primary site backups if t
he data in the secondary site has been damaged. Re-create the standby database from the new production database by following the step
s for creating a standby database in Oracle Data
Guard Concepts and Administration.
 
After the standby database has been reestablish
ed, start the standby database.
 

Step 2: St art Recovery

Step 3: Verify Log Transport Services on Production Database

You may have to reenable the production database remote archive destination. Quer y the V$ARCHIVE_DEST_STATUS view first to see the current state of the archive destinations:

SELECT DEST_ID, DEST_NAME, STATUS, PROTECTION_MODE, DESTINATION, ERROR, SRL 
FROM V$ARCHIVE_DEST
_STATUS;
ALTER SYSTEM SET LOG_ARCHIVE_DEST_STATE_n=ENABLE;
ALTER
SYSTEM SWITCH LOGFILE;

Verify log transport services between the production and standby databases by checking for errors. Query V$ARCHIVE_DEST and V$ARCHIVE_DEST_STATUS vie ws.

SELECT STATUS, TARGET, LOG_SEQUENCE, TYPE, PROCESS, REGISTER, ERROR 

    FROM V$ARCHIVE_DEST;
SELECT * FROM V$ARCHIVE_DEST_STATUS WHERE STATUS!='INACTIVE';

Step 4: Verify that Recove ry is Progressing on Standby Database

For a physical standby databas e, verify that there are no errors from the managed recovery process and that the recovery has applied the archived redo logs.

SELECT MAX(SEQUENCE#), THREAD# FROM V$LOG_HISTORY GROUP BY THREAD; SELECT P ROCESS, STATUS, THREAD#, SEQUENCE#, CLIENT_PROCESS FROM V$MANAGED_STANDBY; < a name="1008209">

For a logical standby database, verify that there are no errors from the logical standby process and that the recovery has applied the archived redo logs.

SELECT THREAD#, SEQUENCE# SEQ# 
<
a name="1008289">    FROM DBA_LOGSTDBY_LOG LOG, DBA_LOGSTDBY_PROGRESS PROG 
    WHERE PROG.APPLIED_SCN BETW
EEN LOG.FIRST_CHANGE# AND LOG.NEXT_CHANGE# 
    ORDER BY NEXT_CHANGE#;

Step 5: Restore Production Database Protection Mode

If you had to change the protection mode of the production dat abase from maximum protection to either maximum availability or maximum performance because of the standby database outage, then chan ge the production database protection mode back to maximum protection depending on your business requirements.

ALTER DATABASE SET STANDBY DATABASE TO MAXIMIZE [PROTECTION | AVAILABILITY];

Restoring Fault T olerance after a Standby Database Data Failure

Following an unplanne d outage of the standby database that requires a full or partial datafile restoration (such as data or media failure), full fault tol erance is compromised until the standby database is brought back into service. Full database protection should be restored as soon as possible. Note that using a Hardware Assisted Resilient Database configuration can prevent this type of problem.

The following steps are required to restore full fault tolerance after data failure of t he standby database:

• Step 1: Fix the Cause of the Outage
• Step 2: Restore the Backup of Affected Datafiles
• Step 3: Restor e Required Archived Redo Log Files
• Step 5: S tart Recovery or Apply
• Step 6: Verify Log Tr ansport Services On the Production Database
• Step 7: Verify that Recovery or Apply Is Progressing On the Standby Database
• < /a>Step 8: Restore Production Database Protection Mode

Step 1: Fix the Cause of the Outage

The root cause of the outage should be investigated and action taken to prevent the problem from occurring again.

Step 2: Restore the Backup of Affected Datafiles

Only the affected datafiles need to be restored on to the standby site.

Step 3: Restore Required Archived Redo Log Files

Archived redo log files may need to be restored to recover the restored data files up to the configured lag.

For physical standby databases:

• If the archived redo logs required for recovery are available on the standby system in a confi gured archive destination, then the managed recovery process automatically finds and applies them as needed. No restoration is necess ary.
• If the required archived redo logs have been deleted from the standby sys tem but are still available on the production system, then the fetch archive log (FAL) process is automatically invoked to transfer t hem to the standby system. No restoration is necessary.

For logical standby databases , initiate complete media recovery for the affected files. Consider the following:

• < a name="1008228">If the archived redo logs required for recovery are available on the standby system in a configured archive dest ination, then the recovery process automatically finds and applies them as needed. No restoration is necessary.
• If the required archived redo logs have been deleted from the standby system, then they must be res tored to the standby system. Complete media recovery for the affected files after the necessary archived redo logs are available.

Step 4 : Start the Standby Database

After the standby database has been ree stablished, start the standby database.

St ep 5: Start Recovery or Apply

Verify log transport services on the new production database by checking for errors when querying V$ARCHIVE_DEST and V$ARCHIVE_DEST_STATUS.

SELECT STATUS, TARGET, LOG_SEQUENCE, TYPE, PROCESS, REGISTER, ERROR FROM 
V$ARCHIVE_DEST;

SELECT * FROM V$ARCHIVE_DEST_STATUS WHERE STATUS != 'INACTIVE';

For a physical standby database, verify that ther e are no errors from the managed recovery process and that the recovery has applied archived redo logs.

SELECT MAX(SEQUENCE#), THREAD# FROM V$LOG_HISTORY GROUP BY THREAD;
SELECT PROCESS, STATUS, THREAD#
, SEQUENCE#, CLIENT_PROCESS
    FROM V$MANAGED_STANDBY;

SELECT THREAD#, SEQUENCE# SEQ# 
    FR
OM DBA_LOGSTDBY_LOG LOG, DBALOGSTDBY_PROGRESS PROG
    WHERE PROG.APPLIED_SCN BETWEEN LOG.FIRST_CHANGE# AND LOG
.NEXT_CHANGE#
    ORDER BY NEXT_CHANGE#;