| See Also:
|
table>
Define ASM Disk and Failure Groups Properly
ASM uses a concept calle
d failure groups to protect data against disk or component failures. When using failure groups, ASM optimizes file layout to reduce t
he unavailability of data due to the failure of a shared resource. Failure groups define disks that share components, so that if one
disk fails, then other disks sharing the component might also fail. An example of what might be defined as a failure group is a strin
g of SCSI disks on the same SCSI controller. Failure groups are used to determine which ASM disks to use for storing redundant data.
For example, if 2-way mirroring is specified for a file, then redundant copies of file extents will be stored in separate failure gro
ups.
Failure groups are used with storage that does not provide its own redundancy capabili
ty, such as disks that have not been configured according to RAID. The manner in which failure groups are defined for an ASM disk gro
up is site-specific because the definition depends on the configuration of the storage and how it is connected to the database system
s. ASM attempts to maintain three copies of its metadata, requiring a minimum of three failure groups for proper protection.
When using ASM with intelligent storage arrays, the storage array's protection features (such as ha
rdware RAID-1 mirroring) are typically used instead of ASM's redundancy capabilities, which are implemented by using the EXTERN
AL REDUNDANCY clause of the CREATE DISKGROUP statement. When using external redundancy, ASM failure groups are no
t used, because disks in an external redundancy disk group are presumed to be highly available.
Disks, as presented by the storage array to the operating system, should not be aggregated or subdivided by the array because it
can hide the physical disk boundaries needed for proper data placement and protection. If, however, physical disk boundaries are alw
ays hidden by the array, and if each logical device, as presented to the operating system, has the same size and performance characte
ristics, then simply place each logical device in the same ASM disk group (with redundancy defined as EXTERNAL REDUNDANCY),and place the database and flash recovery areas in that one ASM disk group. An alternative approach is to create two disk groups,
each concsisting of a single logical device, and place the database area in one disk group and the flash recovery in the other disk g
roup. This method provides additional protection against disk group metadata failure and corruption.
For example, suppose a storage array takes eight physical disks of size 36GB and configures them in a RAID 0+1 manner for p
erformance and protection, giving a total of 144GB of mirrored storage. Futhermore, this 144GB of storage is presented to the operati
ng system as two 72GB logical devices with each logical device being striped and mirrored across all eight drives. When configuring A
SM, place each 72GB logical device in the same ASM disk group, and place the database area and flash recovery areas on that disk grou
p.
If the two logical devices have different performance characteristics,( for example, one
corresponds to the inner half and the other to the outer half of the underlying physical drives) then the logical devices should be
placed in separate disk groups. Because the outer portion of a disk has a higher transfer rate, the outer half disk group should be u
sed for the database area; the inner half disk group should be used for the flash recovery area.
If multiple, distinct storage arrays are used with a database under ASM control, then multiple options are available. One optio
n is to create multiple ASM disk groups that do not share storage across multiple arrays and place the database and flash recovery ar
eas in separate disk groups, thus physically separating the database area from the flash recovery area. Another option is to create a
single disk group across arrays that consists of failure groups, where each failure group contains disks from just one array.These o
ptions provide protection against the failure of an entire storage array.
Use HARD-Compliant
Storage for the Greatest Protection Against Data Corruption
The Har
dware Assisted Resilient Data (HARD) initiative is a program designed to prevent data corruptions before they happen. Data corruption
s are very rare, but when they do occur, they can have a catastrophic effect on business. Under the HARD initiative, Oracle's storage
partners implement Oracle's data validation algorithms inside storage devices. This makes it possible to prevent corrupted data from
being written to permanent storage. The goal of HARD is to eliminate a class of failures that the computer industry has so far been
powerless to prevent. RAID has gained a wide following in the storage industry by ensuring the physical protection of data; HARD take
s data protection to the next level by going beyond protecting physical data to protecting business data.
In order to prevent corruptions before they happen, Oracle tightly integrates with advanced storage devices to create
a system that detects and eliminates corruptions before they happen. Oracle has worked with leading storage vendors to implement Orac
le's data validation and checking algorithms in the storage devices themselves. The classes of data corruptions that Oracle addresses
with HARD include:
- Writes that physically and logically corru
pt data file, control file and log file blocks
- Writes of Oracle blocks to inco
rrect locations
- Erroneous writes to Oracle data by programs other than Oracle<
/li>
- Writes of partial or incomplete blocks
End-to-end block validation is the technology employed by the operating system or storage subsystem to validate the Oracle d
ata block contents. By validating Oracle data in the storage devices, corruptions will be detected and eliminated before they can be
written to permanent storage. This goes beyond the current Oracle block validation features that do not detect a stray, lost, or corr
upted write until the next physical read.
Oracle vendors are given the opportunity to imple
ment validation checks based on a specification. A vendors' implementation may offer features specific to their storage technology. O
racle maintains a Web site that will show a comparison of each vendor's solution by product and Oracle version. For the most recent i
nformation, see http://otn.oracle.com/deploy/availability/
htdocs/HARD.html.
Storage Recommendation for RAC
The following recommendation applies to the "RAC only" architecture and MAA:
Protect the
Oracle Cluster Registry and Voting Disk From Media Failure
The share
d volumes created for the OCR and the voting disk should be configured using RAID to protect against media failure. This requires the
use of an external cluster volume manager, cluster file system, or storage hardware that provides RAID protection.
Recommendations for Configuring Server Hardware
The main server hardware components are the nodes for the database and application server farm and
the components within each node such as CPU, memory, interface boards (such as I/O and network), storage, and the cluster interconne
ct in a RAC environment.
This section includes the following topics:
Ser
ver Hardware Recommendations for All Architectures
The following rec
ommendations apply to all architectures:
Use Fewer, Faster, and Denser Components
Use fewer, faster CPUs instead of more, slower CPUs. Use fewer higher-density memory modules instead of more lower-densit
y memory modules. This reduces the number of components that can fail while providing the same service level. This needs to be balanc
ed with cost and redundancy.
Use Redundant Hardware Components
Using redundan
t hardware components enables the system to fail over to the working component while taking the failed component offline. Choose comp
onents (such as power supplies, cooling fans, and interface boards) that can be repaired or replaced while the system is running to p
revent unscheduled outages caused by the repair of hardware failures.
Use Systems That Can Detect and Isolate Failures
Use systems that can automatically detect failures and provide alternate paths around subsy
stems that have failed or isolate the subsystem. Choose a system that continues to run despite a component failure and automatically
works around the failed component without incurring a full outage. For example, find a system that can use an alternate path for I/O
requests if an adapter fails or can avoid a bad physical memory location if a memory board fails.
Protect the Boot Disk With a Backup Copy
Because mirroring does not protect against accidental removal of a file
or most corruptions of the boot disk, an online copy of the boot disk should be maintained so that the system can be quickly reboote
d using the same operating system image if a critical file is removed or corruption occurs on the primary boot image. Operating syste
m vendors often provide a mechanism to easily maintain multiple boot images.
Server Hardware Recommendations for RAC
Use a Support
ed Cluster System to Run RAC
RAC provides fast, automatic recovery f
rom node and instance failures. Using a properly supported configuration is necessary for a successful RAC implementation. A supporte
d hardware configuration may encompass server hardware, storage arrays, interconnect technology, and other hardware components.
<
a name="1011755">
Choose the Pro
per Cluster Interconnect
Select an interconnect that has redundancy,
high speed, low latency, low host resource consumption, and the ability to balance loads across multiple available paths. In two-nod
e configurations, it may be possible to use a direct-connect interconnect between the two nodes, but a switch should be used instead
to provide a degree of isolation between the network interface cards of the two nodes in the cluster. If you plan to have more than t
wo nodes in the future, then choose a switch-based interconnect solution instead of direct-connect to reduce the complexity of adding
additional nodes in the future. If you have more than two nodes, then a switch-based interconnect is highly recommended and, in many
cases, a requirement of the cluster solution being used.
Server Hardware Recommendations for Data Guard
The following recommendation applies to both the primary and secondary sites in "Data Guard only" and MAA
environments.
Use Identical Hardware for Every Machine at Both Sites
Using i
dentical hardware for machines at both sites provides a symmetric environment that is easier to administer and maintain. Such a symme
tric configuration mitigates failures or performance inconsistencies following a switchover or failover because of dissimilar hardwar
e.
Recomm
endations for Configuring Server Software
The recommendations for se
rversoftware apply to all nodes in a RAC environment and to both the primary and secondary sites in a Data Guard and MAA environment
because they contain the identical configuration.
This section includes the following topic
s:
Server Software Recommendations for All Architectures
The following recommendations apply to all architectures:
Use the Same OS Version, Patch Level, Single Patches, and Driver Versions
Use the same operating system version, patch level, single patches, and driver versions on all machines. Consistency wi
th operating system versions and patch levels reduces the likelihood of encountering incompatibilities or small inconsistencies betwe
en the software on all machines. In an open standards environment, it is impractical for any vendor to test each and every piece of n
ew software with every combination of software and hardware that has been previously released. Temporary differences can be tolerated
when using RAC or Data Guard as individual systems or groups of systems are upgraded or patched one at a time to minimize scheduled
outages, if the goal is that all machines will be upgraded to the same versions and patch levels.
Use an Operating System That is Fault-Tolera
nt to Hardware Failures
Use an operating system that, when coupled w
ith the proper hardware, supports the ability to automatically detect failures and provide alternate paths around subsystems that hav
e failed or isolate subsystems. Choose a system that can continue running if a component, such as a CPU or memory board, fails and au
tomatically provides paths around failed components, such as an alternate path for I/O requests in the event of an adapter failure.
p>
Configure
Swap Partititions Appropriately
Mirror disks containing swap partiti
ons so that if a disk that contains a swap partition fails, it will not result in an application or system failure.
Use disk-based swap instead of RAM-based swap. It is always good practice to make all system memory availabl
e for database and application use unless the amount available is sufficiently oversized to accommodate swap.
<
/a>
Do not use TMPFS (a Solaris implementation that stores files in virtual memory rather than on disk) or other file
systems that exclusively use virtual memory instead of disks for storage for the /tmp file system or other scratch file
systems. This prevents runaway programs that write to /tmp from having the potential to hang the system by exhausting al
l virtual memory.
Set Operating System Parameters to Enable Future Growth
An
example on UNIX is setting shared memory and semaphore kernel parameters high enough to enable future growth, thus preventing an outa
ge for reconfiguring the kernel and rebooting the system. Verify that using settings higher than required either presents no overhead
or incurs such small overhead on large systems that the effect is negligible.
Use Logging or Journal File Systems
Journal file systems and logging reduce or eliminate the number of file system checks required
following a system reboot, thereby facilitating faster restarting of the system.
<
h4 class="H3">Mirror Disks That Contain Oracle and Application Software
As with server hardware and software, the recommendations for Oracle so
ftware apply to both the primary and secondary sites because they contain identical configurations. Mirror disks containing Oracle an
d application software to prevent a disk failure from causing an unscheduled outage.
The following recommendations apply to a "RAC only" environment:
- Use Supported Clustering Software
- Use Network Time Protocol (NTP) On All Cluster Nodes
Use Suppor
ted Clustering Software
RAC provides fast, automatic recovery from n
ode and instance failures. Using a properly supported configuration is a key component of success. A supported software configuration
encompasses operating system versions and patch levels, clustering software versions, which possibly includes Oracle-supplied cluste
r software. On some platforms (such as Solaris, Linux, and Windows), Oracle supplies cluster software required for use with RAC.
Use Network Time Protocol (NTP) On All Cluster Nodes
Use NTP to synchronize the clocks on all nodes in the cluster to facilitate anal
ysis of tracing information based on timestamps.
Recommendations for Configuring the Network
This section includes the following topics:
Network Configuration Best Practices fo
r All Architectures
The following recommendations apply to all archi
tectures:
Ensure That All Network Components Are Redundant
Table 6-2 describes the necessary redundant network components that are illustrat
ed in Figure 6-1.
Table 6-2 Redundant Network Components <
/h5>
Figure 6-1 depicts a single site in an MAA environment, emphasizing the redundant network components.
Figure 6-1 Network Configuration
Text description of the illustration
maxav010.gif
Use Load Balancers to Distribute Incoming Requests
Applicat
ion layer load balancers sit logically in front of the application server farm and publish to the outside world a single IP address f
or the service running on a group of application servers. All requests are initially handled by the load balancer, which then distrib
utes them to a server within the application server farm. End users only need to address their requests to the published IP address;
the load balancer determines which server should handle the request.
If one of the middle-t
ier application servers cannot process a request, the load balancers route all subsequent requests to functioning servers. The load b
alancers should be redundant to avoid being a single point of failure because all client requests pass through a single hardware-base
d load balancer. Because failure of that piece of hardware is detrimental to the entire site, a backup load balancer is configured th
at has a heartbeat with the primary load balancer. With two load balancers, one is configured as standby and becomes active only if t
he primary load balancer becomes unavailable.
Network Configuration Best Practices for RAC
This recommendation applies to RAC environments.
Classify Network Interfaces Using the Oracle Interface Configuration Tool
Use the Oracle Interface Configuration (OIFCFG) tool to classify network
interfaces as public, cluster interconnect, or storage so that RAC properly selects a network interface for internode network traffic
.
<
font face="Arial, Helvetica, sans-serif" color="#330099">Network Configuration Best Practices for Data Guard
Configure
System TCP Parameters Appropriately
Configure system TCP parameters
that control the sending and receiving buffer sizes so that the bandwidth between sites can be fully utilized for log transport serv
ices. The proper buffer size is often governed by the bandwidth delay product (BDP) formula, particularly when using a high-speed, hi
gh-latency network.
Use WAN Traffic Managers to Provide Site Failover Capabilities
WAN traffic managers provide the initial access to the services located at the primary site. These managers are imple
mented at the primary and secondary sites to provide site failover capabilities when the primary site becomes completely unavailable.
Geographically separating the WAN traffic managers on separate networks reduces the impact of network failures or a natural disaster
that causes the primary site to become unavailable.