Oracle RAC, ASM, Exadata, Cloud, and More..: October 2011

ACFS Snapshots

ACFS snapshot is a online read point-in-time copy of an Oracle ACFS file system. It can be used to recover a deteted file, revert to a prior version. When the snapshot is created, a set of metadata including the directory structures, name of all files are created on the same volume.

+ASM1 > acfsutil registry -l
Device : /dev/asm/vol_share-356 : Mount Point : /share : Options : none : Nodes : all : Disk Group : DG_POC_ACFS : Volume : VOL_SHARE

Create ACFS snapshot image
+ASM1 > /sbin/acfsutil snap create snap_share001 /share
acfsutil snap create: Snapshot operation is complete.
/share/.ACFS/snaps/snap_share001 => is the snapshot file system

View File System
+ASM1 > /sbin/acfsutil info fs /share
    ACFS Version: 11.2.0.2.0
    flags:        MountPoint,Available
    mount time:   Wed Oct 19 14:05:09 2011
    volumes:      1
    total size:   140660178944
    total free:   138043518976
    primary volume: /dev/asm/vol_share-356
        label:                 DBA Share FS
        flags:                 Primary,Available,ADVM
        on-disk version:       39.0
        allocation unit:       4096
        major, minor:          252, 182273
        size:                  140660178944
        free:                  138043518976
        ADVM diskgroup         DG_POC_ACFS
        ADVM resize increment: 268435456
        ADVM redundancy:       unprotected
        ADVM stripe columns:   4
        ADVM stripe width:     131072
    number of snapshots: 1
    snapshot space usage: 49152

Resize ACFS
+ASM1 > acfsutil size +3g /share
acfsutil size: new file system size: 140660178944 (134144MB)

View detail information on each file system
acfsutil info fs /share
acfsutil info fs /share -s (statistics)

How to configure ACFS - ASM Cluster File System and EXT3 file system

In Oracle 11gR2 Grid Infrastructure, Oracle introduces the ASM Cluster file system, or ACFS. It provides shared, cluster-wide access to various type files like database binaries, user data, reports, configuration files, BFILEs, trace files and alerts.

Below are all the steps to setup ACFS file system or EXT3 file system via ACFS.
1) Pre-checking for ACFS configuration:

+ASM1 > cluvfy stage -pre acfscfg -n racapp1,racapp2-verbose

2) Create DG_POC_ACFS diskgroup

cat /apps/oracle/work/an/dg_acfs.xml to view contain

</dg>

+ASM1 > asmcmd mkdg /apps/oracle/work/an/dg_acfs.xml

3) Create ASM volume

ASMCMD> volcreate -G DG_POC_ACFS -s 128g vol_share

ASMCMD> volinfo -a

Diskgroup Name: DG_POC_ACFS

Volume Name: VOL_SHARE

Volume Device: /dev/asm/vol_share-356

State: ENABLED

Size (MB): 131072

Resize Unit (MB): 256

Redundancy: UNPROT

Stripe Columns: 4

Stripe Width (K): 128

Usage:

Mountpath:

4) Create the ACFS file system

as root

/sbin/mkfs -t acfs -n "DBA Share FS" /dev/asm/vol_share-356

mkfs.acfs: version = 11.2.0.2.0

mkfs.acfs: on-disk version = 39.0

mkfs.acfs: volume = /dev/asm/vol_share-356

mkfs.acfs: volume size = 137438953472

mkfs.acfs: Format complete.

5) Add acfs registry:

/sbin/acfsutil registry -f -a /dev/asm/vol_share-356 /share

acfsutil registry: mount point /share successfully added to Oracle Registry

6) Mount /share file system

/bin/mount -t acfs /dev/asm/vol_share-356 /share

Or you can also create it as an EXT3 file system

volcreate -G DG_POC_ACFS -s 128g vol_dba

mkfs -t ext3 /dev/asm/vol_dba-356

mount /dev/asm/vol_dba-356 /dba

7) Validate Post ACFS configuration

cluvfy stage -post acfscfg -n racapp1,racapp2-verbose

Shared Storage for Grid Infrastructure and Software

OCR and Voting can be stored in:

Automatic Storage Manager (ASM),
Oracle Cluster File System (OCFS2),
Redhat Global File System (GFS),
Certified Network File System (NFS).

Oracle Software can be stored in all of the above storage option and ASM Cluster File System (ACFS).

V$ASM_ATTRIBUTE and COMPATIBLE attributes

If the diskgroup attribute COMPATIBLE.ASM is set to 11.1 or higher, you can see all attributes by querying v$ASM_ATTRIBUTE and v$ASM_DISKGROUP. Also you can run lsattr via ASMCMD to some the same information for the diskgroup like below..

+ASM3 - oracle: asmcmd lsattr -lm -G OV

Group_Name Name                     Value       RO Sys
OV          access_control.enabled   FALSE       N   Y
OV          access_control.umask     066         N   Y
OV          au_size                  1048576     Y   Y
OV          cell.smart_scan_capable FALSE       N   N
OV          compatible.asm           11.2.0.0.0 N   Y
OV          compatible.rdbms         10.1.0.0.0 N   Y
OV          disk_repair_time         3.6h        N   Y
OV          sector_size              512         Y   Y

Compatibility Attributes: COMPATIBLE.RDBMS must be less than or equal to COMPATIBLE.ASM. To revert to previous value, you must create a new diskgroup with the old campatiblity attributes and then restore the database files that were on the diskgroup. If setting COMPATIBLE.RDBMS and COMPATIBLE.ASM to the same value like 10.1, it will allow the greatest flexibility regarding the ASM and databas software versions that can be use a diskgroup. However, ihis setting limits access to the 11g ASM features like fast mirror resync, allocation unit, preferrred read failure groups, etc..

ASM Metadata Backup md_backup and Restore md_restore

Most of the time, backup ASM disk group is not necessary since you can recreate it and restore its content. However, if you want to keep the user templates, aliases, and directories within ASM, you need to backup the metadata for the ASM objects. The new utility in 11g is AMBR (ASM metadata backup and restore).

Backup example:

asmcmd md_backup /apps/oracle/bkdgroup -G DATA

asmcmd md_backup /apps/oracle/bkdgroupall -G 'DATA,DG_FRA_PF501'

Disk group metadata to be backed up: DATA

Disk group metadata to be backed up: DG_FRA_PF501

Current alias directory path: rac-scan

Current alias directory path: rac-scan/ASMPARAMETERFILE

Current alias directory path: rac-scan/OCRFILE

Current alias directory path: ORATOOLS/ARCHIVELOG/2011_10_04

Current alias directory path: ORATOOLS/CONTROLFILE

Current alias directory path: ORATOOLS/ARCHIVELOG

Current alias directory path: ORATOOLS

Current alias directory path: ORATOOLS/ONLINELOG

Current alias directory path: ORATOOLS/ARCHIVELOG/2011_10_06

with backup mode md_backup, information about ASM disks and disk groups, configurations, attributes, etc.. are stored in a text file bkdgroup or bkdgroupall. It will be used later to restore the ASM diskgroup metadata definition. The information gathers during ASM backup contains the ASM diskgroup name, Redundancy Type, Allocation Unit Size, diskpath, alias directories, stripe, full path of alias entries, etc..

In the restore mode md_restore, it re-create the disk group based on the backup file with all user-defined templates with the exact configuration as the backuped disk group. There are several options when restore the disk group

full - re-create the disk group with the exact configuration

nodg - Restores metadata in an existing disk group provided as an input parameter

newdg - Change the configuration like failure group, disk group name, etc..

Restore example:

asmcmd md_restore bkdgroup --nodg -g DATA
asmcmd md_restore bkdgroupall --newdg -o 'DATA:DATA_NEW'

Troubleshooting Database Network Issue

Below are couple tips to investigate network traffic and database network connections.

netstat -ptc : Display the process ID and tcp connections. If a process with abnormally high in Send-Q, it means that it consumes too much network traffic.

ping and telnet - if you can't connect the remote servers, contact your network admin.

tnsping - check the remote listener and database to make sure they're up and running.

TNS_ADMIN - verify the tnsnames.ora file

Virtualization with Xen

Advantages: 1) Effective resource usage 2) Manageability 3) Security

Concepts:

Hypervisor is the manager of the Xen enviornment. It controls and provides acess to resource like CPU, memory, storage. It's started by the bootloader and acts as traffic controller for all virtualized OS. It control migration, starting, stopping, and puasing the virtualized OS. There are 2 types of virtualizations.

Paravirtualization: It's the native mode of Xen. Performance is the main advantage incurs no more than 5 % performnace impact v.s bare hardware

Full Virtualization: Xen provides a complete machine simulation to run OS which does not include Xen support. The extra operations make the OS beliving that it runs on bare hardware add overhead. Therefore, the performance impact is greated than paravirtualization.

Xen Domains: Xen virtual hosts. The first domain loaded is called Domain-0. Domian-0 is booted immediately after the hypervisor starts and provides the user interface at the console. It's used to install and manage other domain. Any virtual machine other than Domain-0 is called Domain-U. Dom-U has no visibility of the hypervisor or the physical hardware

ethtool, mii-tool

ethtool displays or changes detailed information about ethernet adapter.

ethtool eth3
Settings for eth3:
        Supported ports: [ TP ]
        Supported link modes:   10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Full
        Supports auto-negotiation: Yes
        Advertised link modes: 10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Full
        Advertised auto-negotiation: Yes
        Speed: 1000Mb/s
        Duplex: Full
        Port: Twisted Pair
        PHYAD: 1
        Transceiver: internal
        Auto-negotiation: on
        Supports Wake-on: g
        Wake-on: g
        Link detected: yes

mii-tool is another useful tool for quickly determining the state of an ethernet adapter
root > mii-tool -v eth3
eth3: negotiated, link ok
product info: vendor 00:50:ef, model 60 rev 8
basic mode:   autonegotiation enabled
basic status: autonegotiation complete, link ok
capabilities: 100baseTx-FD 100baseTx-HD 10baseT-FD 10baseT-HD
advertising: 100baseTx-FD 100baseTx-HD 10baseT-FD 10baseT-HD flow-control