Exadata storage objects

Hierarch of Exadata Storage Objects:

DISK-> LUN-> CELLDISK -> GRIDDISK-> ASM DISK
 
 
celladmin@cell ~]$ cellcli
CellCLI: Release 11.2.2.1.0 - Production on Thu Jan 10 15:43:39 PST 2013

Copyright (c) 2007, 2009, Oracle.  All rights reserved.
Cell Efficiency Ratio: 17M

CellCLI> list cell detail
         name:                   cell
         bmcType:                absent
         cellVersion:            OSS_MAIN_LINUX_101005
         cpuCount:               1
         fanCount:               1/1
         fanStatus:              normal
         id:                     3c4bc34b-9acd-4a98-a910-c660c9c76c03
         interconnectCount:      1
         interconnect1:          eth0
         iormBoost:              0.0
         ipaddress1:             192.168.10.101/24
         kernelVersion:          2.6.18-194.el5
         makeModel:              Fake hardware
         metricHistoryDays:      7
         offloadEfficiency:      17.3M
         powerCount:             1/1
         powerStatus:            normal
         status:                 online
         temperatureReading:     0.0
         temperatureStatus:      normal
         upTime:                 3 days, 6:22
         cellsrvStatus:          running
         msStatus:               running
         rsStatus:               running
CellCLI> list physicaldisk
         /opt/oracle/cell11.2.2.1.0_LINUX_101005/disks/raw/FLASH01       /opt/oracle/cell11.2.2.1.0_LINUX_101005/disks/raw/FLASH01       normal
         /opt/oracle/cell11.2.2.1.0_LINUX_101005/disks/raw/FLASH02       /opt/oracle/cell11.2.2.1.0_LINUX_101005/disks/raw/FLASH02       normal
         /opt/oracle/cell11.2.2.1.0_LINUX_101005/disks/raw/FLASH03       /opt/oracle/cell11.2.2.1.0_LINUX_101005/disks/raw/FLASH03       normal
         /opt/oracle/cell11.2.2.1.0_LINUX_101005/disks/raw/FLASH04       /opt/oracle/cell11.2.2.1.0_LINUX_101005/disks/raw/FLASH04       normal
         /opt/oracle/cell11.2.2.1.0_LINUX_101005/disks/raw/disk01        /opt/oracle/cell11.2.2.1.0_LINUX_101005/disks/raw/disk01        normal
         /opt/oracle/cell11.2.2.1.0_LINUX_101005/disks/raw/disk02        /opt/oracle/cell11.2.2.1.0_LINUX_101005/disks/raw/disk02        normal
         /opt/oracle/cell11.2.2.1.0_LINUX_101005/disks/raw/disk03        /opt/oracle/cell11.2.2.1.0_LINUX_101005/disks/raw/disk03        normal
         /opt/oracle/cell11.2.2.1.0_LINUX_101005/disks/raw/disk04        /opt/oracle/cell11.2.2.1.0_LINUX_101005/disks/raw/disk04        normal
         /opt/oracle/cell11.2.2.1.0_LINUX_101005/disks/raw/disk05        /opt/oracle/cell11.2.2.1.0_LINUX_101005/disks/raw/disk05        normal
         /opt/oracle/cell11.2.2.1.0_LINUX_101005/disks/raw/disk06        /opt/oracle/cell11.2.2.1.0_LINUX_101005/disks/raw/disk06        normal
         /opt/oracle/cell11.2.2.1.0_LINUX_101005/disks/raw/disk07        /opt/oracle/cell11.2.2.1.0_LINUX_101005/disks/raw/disk07        normal
         /opt/oracle/cell11.2.2.1.0_LINUX_101005/disks/raw/disk08        /opt/oracle/cell11.2.2.1.0_LINUX_101005/disks/raw/disk08        normal
         /opt/oracle/cell11.2.2.1.0_LINUX_101005/disks/raw/disk09        /opt/oracle/cell11.2.2.1.0_LINUX_101005/disks/raw/disk09        normal
         /opt/oracle/cell11.2.2.1.0_LINUX_101005/disks/raw/disk10        /opt/oracle/cell11.2.2.1.0_LINUX_101005/disks/raw/disk10        normal
         /opt/oracle/cell11.2.2.1.0_LINUX_101005/disks/raw/disk11        /opt/oracle/cell11.2.2.1.0_LINUX_101005/disks/raw/disk11        normal
         /opt/oracle/cell11.2.2.1.0_LINUX_101005/disks/raw/disk12        /opt/oracle/cell11.2.2.1.0_LINUX_101005/disks/raw/disk12        normal

CellCLI> list lun
         /opt/oracle/cell11.2.2.1.0_LINUX_101005/disks/raw/FLASH01       /opt/oracle/cell11.2.2.1.0_LINUX_101005/disks/raw/FLASH01       normal
         /opt/oracle/cell11.2.2.1.0_LINUX_101005/disks/raw/FLASH02       /opt/oracle/cell11.2.2.1.0_LINUX_101005/disks/raw/FLASH02       normal
         /opt/oracle/cell11.2.2.1.0_LINUX_101005/disks/raw/FLASH03       /opt/oracle/cell11.2.2.1.0_LINUX_101005/disks/raw/FLASH03       normal
         /opt/oracle/cell11.2.2.1.0_LINUX_101005/disks/raw/FLASH04       /opt/oracle/cell11.2.2.1.0_LINUX_101005/disks/raw/FLASH04       normal
         /opt/oracle/cell11.2.2.1.0_LINUX_101005/disks/raw/disk01        /opt/oracle/cell11.2.2.1.0_LINUX_101005/disks/raw/disk01        normal
         /opt/oracle/cell11.2.2.1.0_LINUX_101005/disks/raw/disk02        /opt/oracle/cell11.2.2.1.0_LINUX_101005/disks/raw/disk02        normal
         /opt/oracle/cell11.2.2.1.0_LINUX_101005/disks/raw/disk03        /opt/oracle/cell11.2.2.1.0_LINUX_101005/disks/raw/disk03        normal
         /opt/oracle/cell11.2.2.1.0_LINUX_101005/disks/raw/disk04        /opt/oracle/cell11.2.2.1.0_LINUX_101005/disks/raw/disk04        normal
         /opt/oracle/cell11.2.2.1.0_LINUX_101005/disks/raw/disk05        /opt/oracle/cell11.2.2.1.0_LINUX_101005/disks/raw/disk05        normal
         /opt/oracle/cell11.2.2.1.0_LINUX_101005/disks/raw/disk06        /opt/oracle/cell11.2.2.1.0_LINUX_101005/disks/raw/disk06        normal
         /opt/oracle/cell11.2.2.1.0_LINUX_101005/disks/raw/disk07        /opt/oracle/cell11.2.2.1.0_LINUX_101005/disks/raw/disk07        normal
         /opt/oracle/cell11.2.2.1.0_LINUX_101005/disks/raw/disk08        /opt/oracle/cell11.2.2.1.0_LINUX_101005/disks/raw/disk08        normal
         /opt/oracle/cell11.2.2.1.0_LINUX_101005/disks/raw/disk09        /opt/oracle/cell11.2.2.1.0_LINUX_101005/disks/raw/disk09        normal
         /opt/oracle/cell11.2.2.1.0_LINUX_101005/disks/raw/disk10        /opt/oracle/cell11.2.2.1.0_LINUX_101005/disks/raw/disk10        normal
         /opt/oracle/cell11.2.2.1.0_LINUX_101005/disks/raw/disk11        /opt/oracle/cell11.2.2.1.0_LINUX_101005/disks/raw/disk11        normal
         /opt/oracle/cell11.2.2.1.0_LINUX_101005/disks/raw/disk12        /opt/oracle/cell11.2.2.1.0_LINUX_101005/disks/raw/disk12        normal

CellCLI> list lun where disktype=harddisk
         /opt/oracle/cell11.2.2.1.0_LINUX_101005/disks/raw/disk01        /opt/oracle/cell11.2.2.1.0_LINUX_101005/disks/raw/disk01        normal
         /opt/oracle/cell11.2.2.1.0_LINUX_101005/disks/raw/disk02        /opt/oracle/cell11.2.2.1.0_LINUX_101005/disks/raw/disk02        normal
         /opt/oracle/cell11.2.2.1.0_LINUX_101005/disks/raw/disk03        /opt/oracle/cell11.2.2.1.0_LINUX_101005/disks/raw/disk03        normal
         /opt/oracle/cell11.2.2.1.0_LINUX_101005/disks/raw/disk04        /opt/oracle/cell11.2.2.1.0_LINUX_101005/disks/raw/disk04        normal
         /opt/oracle/cell11.2.2.1.0_LINUX_101005/disks/raw/disk05        /opt/oracle/cell11.2.2.1.0_LINUX_101005/disks/raw/disk05        normal
         /opt/oracle/cell11.2.2.1.0_LINUX_101005/disks/raw/disk06        /opt/oracle/cell11.2.2.1.0_LINUX_101005/disks/raw/disk06        normal
         /opt/oracle/cell11.2.2.1.0_LINUX_101005/disks/raw/disk07        /opt/oracle/cell11.2.2.1.0_LINUX_101005/disks/raw/disk07        normal
         /opt/oracle/cell11.2.2.1.0_LINUX_101005/disks/raw/disk08        /opt/oracle/cell11.2.2.1.0_LINUX_101005/disks/raw/disk08        normal
         /opt/oracle/cell11.2.2.1.0_LINUX_101005/disks/raw/disk09        /opt/oracle/cell11.2.2.1.0_LINUX_101005/disks/raw/disk09        normal
         /opt/oracle/cell11.2.2.1.0_LINUX_101005/disks/raw/disk10        /opt/oracle/cell11.2.2.1.0_LINUX_101005/disks/raw/disk10        normal
         /opt/oracle/cell11.2.2.1.0_LINUX_101005/disks/raw/disk11        /opt/oracle/cell11.2.2.1.0_LINUX_101005/disks/raw/disk11        normal
         /opt/oracle/cell11.2.2.1.0_LINUX_101005/disks/raw/disk12        /opt/oracle/cell11.2.2.1.0_LINUX_101005/disks/raw/disk12        normal


CellCLI> list lun where name like '.*disk01' detail
         name:                   /opt/oracle/cell11.2.2.1.0_LINUX_101005/disks/raw/disk01
         cellDisk:               CD_disk01_cell
         deviceName:             /opt/oracle/cell11.2.2.1.0_LINUX_101005/disks/raw/disk01
         diskType:               HardDisk
         id:                     /opt/oracle/cell11.2.2.1.0_LINUX_101005/disks/raw/disk01
         isSystemLun:            FALSE
         lunAutoCreate:          FALSE
         lunSize:                500M
         physicalDrives:         /opt/oracle/cell11.2.2.1.0_LINUX_101005/disks/raw/disk01
         raidLevel:              "RAID 0"
         status:                 normal

CellCLI> list celldisk CD_disk01_cell detail
         name:                   CD_disk01_cell
         comment:                
         creationTime:           2012-09-04T18:12:55-07:00
         deviceName:             /opt/oracle/cell11.2.2.1.0_LINUX_101005/disks/raw/disk01
         devicePartition:        /opt/oracle/cell11.2.2.1.0_LINUX_101005/disks/raw/disk01
         diskType:               HardDisk
         errorCount:             0
         freeSpace:              0
         id:                     00000139-93fc-a83d-0000-000000000000
         interleaving:           none
         lun:                    /opt/oracle/cell11.2.2.1.0_LINUX_101005/disks/raw/disk01
         raidLevel:              "RAID 0"
         size:                   496M
         status:                 normal

CellCLI> list griddisk where celldisk=CD_disk01_cell detail
         name:                   data2_CD_disk01_cell
         availableTo:            
         cellDisk:               CD_disk01_cell
         comment:                
         creationTime:           2012-09-04T23:46:14-07:00
         diskType:               HardDisk
         errorCount:             0
         id:                     00000139-952d-dd5d-0000-000000000000
         offset:                 48M
         size:                   448M
         status:                 active

SQL> select name, path, state, total_mb from v$asm_disk
  2  where name like '%DATA2_CD_DISK01_CELL%';

NAME                           PATH                                                             STATE      TOTAL_MB
------------------------------ ---------------------------------------------------------------- -------- ----------
DATA2_CD_DISK01_CELL           o/192.168.10.101/data2_CD_disk01_cell                            NORMAL          448

SQL> select a.name disk, b.name diskgroup
  2  from v$asm_disk a, v$asm_diskgroup b
  3  where b.group_number = a.group_number
  4  and a.name like '%DATA2_CD_DISK01_CELL%';

DISK                           DISKGROUP
------------------------------ ------------------------------
DATA2_CD_DISK01_CELL           DATA

CELLSRV, MS, RS

CELLSRV:  Is a multithreaded server.  The primary Exadata Storage software component and provides the majority of Exadata storage services.  It communicates with Oracle Database to serve simple block requests like database buffer cache reads and Sart Scan requests like tablespaces with projections and filters.  CELLSRV also implements I/ORM and collect statistics relating to its operation.

MS: provides Exadata cell management and configuration.  It works with CellCLI.  It also responsible for sending alerts and collects some statistics in addition to those collected by CELLSRV.

Restart Server (RS) is used to startup / shutdown the Cell Server (CELLSRV) and Management server (MS).  It also monitors these services to check whether they need to be restarted.
 

Configure RMAN to automatically purge archive logs after applying on the standby database


RMAN can automatically purge archivelogs from the FRA after archivelogs are applied to the standby database.

rman target /
RMAN > configure archivelog deletion poliyc to applied on all standby;

select a.thread#, a.sequence#, a.applied
from v$archived_log a, v$database d
where a.activation# = d.activation#
and a.applied='YES'
order by a.thread#

RMAN > show retention policy

RMAN > report obsolete;

When there is a space pressure in FRA, the archivelogs will be deleted and you can see it from alert_INSTANCE.log

Create Flash devices and Flash-based disk group


Exam your current flash-based cell disks

CellCLI> list celldisk attributes name, freeSpace where diskType=FlashDisk;
FD_00_cell  500M
FD_01_cell  500M
FD_02_cell  500M
FD_03_cell  500M

Create Smart Flash Cache with 1G

CellCLI> create flashcache all size=1024m;
Flash cache cell_FLASHCACHE successfully created


So, the available free space on all flash-based cell disks are:
CellCLI> list celldisk attributes name, freeSpace where diskType=FlashDisk;
FD_00_cell 192M
FD_01_cell 192M
FD_02_cell 192M
FD_03_cell 192M

CellCLI> create griddisk all flashdisk prefix=flash
GridDisk flash_FD_00_cell successfully created
GridDisk flash_FD_01_cell successfully created
GridDisk flash_FD_02_cell successfully created
GridDisk flash_FD_03_cell successfully created

Flash grid disks are created across the cells

CellCLI> list griddisk attributes name, size, ASMModeStatus where disktype=flashdisk;
flash_FD_00_cell 192M UNUSED
flash_FD_01_cell 192M UNUSED
flash_FD_02_cell 192M UNUSED
flash_FD_03_cell 192M UNUSED

SQL >  select path, header_status from v$asm_disk
  2* where path like '%flash%'
PATH HEADER_STATU
---------------------------------------------------------------- ------------
o/192.168.10.101/flash_FD_02_cell CANDIDATE
o/192.168.10.101/flash_FD_03_cell CANDIDATE
o/192.168.10.101/flash_FD_01_cell CANDIDATE
o/192.168.10.101/flash_FD_00_cell CANDIDATE

Create grid disks and ASM diskgroup

Prepare grid disks- Create set of griddisks on all available hard disks space cell disks

CellCLI> list griddisk
 reco_CD_disk07_cell  active
 reco_CD_disk08_cell  active
 reco_CD_disk09_cell  active
 reco_CD_disk10_cell  active
 reco_CD_disk11_cell  active
 reco_CD_disk12_cell  active

CellCLI> create griddisk all harddisk prefix=data2, size=100m;
Cell disks were skipped because they had no freespace for grid disks: CD_disk07_cell, CD_disk08_cell, CD_disk09_cell, CD_disk10_cell, CD_disk11_cell, CD_disk12_cell.
GridDisk data2_CD_disk01_cell successfully created
GridDisk data2_CD_disk02_cell successfully created
GridDisk data2_CD_disk03_cell successfully created
GridDisk data2_CD_disk04_cell successfully created
GridDisk data2_CD_disk05_cell successfully created
GridDisk data2_CD_disk06_cell successfully created

CellCLI> list griddisk;
 data2_CD_disk01_cell  active
 data2_CD_disk02_cell  active
 data2_CD_disk03_cell  active
 data2_CD_disk04_cell  active
 data2_CD_disk05_cell  active
 data2_CD_disk06_cell  active
 reco_CD_disk07_cell   active
 reco_CD_disk08_cell   active
 reco_CD_disk09_cell   active
 reco_CD_disk10_cell   active
 reco_CD_disk11_cell   active
 reco_CD_disk12_cell   active

CellCLI> list griddisk attributes name, size, ASMModeStatus
 data2_CD_disk01_cell  96M   UNUSED
 data2_CD_disk02_cell  96M   UNUSED
 data2_CD_disk03_cell  96M   UNUSED
 data2_CD_disk04_cell  96M   UNUSED
 data2_CD_disk05_cell  96M   UNUSED
 data2_CD_disk06_cell  96M   UNUSED
 reco_CD_disk07_cell   448M  ONLINE
 reco_CD_disk08_cell   448M  ONLINE
 reco_CD_disk09_cell   448M  ONLINE
 reco_CD_disk10_cell   448M  ONLINE
 reco_CD_disk11_cell   448M  ONLINE
 reco_CD_disk12_cell   448M  ONLINE

Create ASM diskgroup with AU_SIZE=4M as recommended value

SQL> create diskgroup DATA external redundancy disk 'o/*/data2*' attribute 'compatible.rdbms' = '11.2.0.0.0', 'compatible.asm' = '11.2.0.0.0', 'cell.smart_scan_capable' = 'TRUE', 'au_size' = '4M';

Diskgroup created.


SQL> select name, state from v$asm_diskgroup;

NAME     STATE
-------------------------------- -----------
DATA     MOUNTED
FRA     MOUNTED


CellCLI> list griddisk attributes name, size, ASMModeStatus
 data2_CD_disk01_cell  96M   ONLINE
 data2_CD_disk02_cell  96M   ONLINE
 data2_CD_disk03_cell  96M   ONLINE
 data2_CD_disk04_cell  96M   ONLINE
 data2_CD_disk05_cell  96M   ONLINE
 data2_CD_disk06_cell  96M   ONLINE
 reco_CD_disk07_cell   448M  ONLINE
 reco_CD_disk08_cell   448M  ONLINE
 reco_CD_disk09_cell   448M  ONLINE
 reco_CD_disk10_cell   448M  ONLINE
 reco_CD_disk11_cell   448M  ONLINE
 reco_CD_disk12_cell   448M  ONLINE

Exadata configuration tasks cell and storage provisioning

To run the performance test on the cell, you use Calibrate:
CellCLI> alter cell shutdown services cellsrv
Stopping CELLSRV services... 
The SHUTDOWN of CELLSRV services was successful.
CellCLI> calibrate;
Calibration will take a few minutes...
Aggregate random read throughput across all hard disk luns: 139 MBPS
Aggregate random read throughput across all flash disk luns: 2720.05 MBPS
Aggregate random read IOs per second (IOPS) across all hard disk luns: 1052
Aggregate random read IOs per second (IOPS) across all flash disk luns: 143248
Controller read throughput: 5477.08 MBPS
Calibrating hard disks (read only) ...
Calibrating flash disks (read only, note that writes will be significantly slower) ...
…….
CALIBRATE stress test is now running...
Calibration has finished.

CellCLI> alter cell validate configuration;
Cell cell successfully altered

CellCLI> list celldisk where freespace !=0
 CD_disk01      normal

CellCLI> create griddisk all harddisk prefix=datagri


CellCLI> drop celldisk CD_disk01_cell;
CellDisk CD_disk01_cell successfully dropped

CellCLI> list celldisk

CellCLI> create celldisk all;
CellDisk CD_disk01  successfully created

CellCLI> list griddisk attributes name where asmdeactivationoutcome !='Yes'

CellCLI> alter griddisk all inactive

CellCLI> list griddisk attributes name where asmdeactivationoutcome !='Yes'


CellCLI> alter griddisk all inactive

To power off Exadata Storage Servers

shutdown -h -y now


CellCLI> list griddisk

CellCLI> list griddisk attributes name, asmmodestatus

Cellcli > alter grid disk all active

CellCLI> list griddisk

CellCLI> alter griddisk all active
CellCLI> list griddisk attributes name, asmmodestatus


CellCLI> list griddisk attributes name, asmmodestatus

CellCLI> list griddisk attributes name, asmmodestatus
CellCLI> list griddisk attributes name, asmmodestatus

CellCLI> list griddisk


Apply Database PSU 11.2.0.3.2

Download the latest OPatch version Version: 11.2.0.3.0

opatch versionOPatch Version: 11.2.0.3.0

Create OCM configuration file $ <ORACLE_HOME>/OPatch/ocm/bin/emocmrsp

sudo opatch auto /apps/oracle/software/11.2.0.3/psu -oh /apps/oracle/product/11.2.0.3/RACDB -ocmrf /tmp/ocm.rsp

Note:  /apps/oracle/software/11.2.0.3/psu is the PSU software location.

[sudo] password for oracle:
Executing /usr/bin/perl /apps/oracle/product/11.2.0.3/RACDB/OPatch/crs/patch112.pl -patchdir /apps/oracle/software/11.2.0.3 -patchn psu -oh /apps/oracle/product/11.2.0.3/RACDB -ocmrf /tmp/ocm.rsp -paramfile /apps/11.2.0/grid/crs/install/crsconfig_params
opatch auto log file location is /apps/oracle/product/11.2.0.3/RACDB/OPatch/crs/../../cfgtoollogs/opatchauto2012-07-15_14-26-50.log
Detected Oracle Clusterware install
Using configuration parameter file: /apps/11.2.0/grid/crs/install/crsconfig_params
patch /apps/oracle/software/11.2.0.3/psu/13696251/custom/server/13696251  apply successful for home  /apps/oracle/product/11.2.0.3/RACDB
patch /apps/oracle/software/11.2.0.3/psu/13696216  apply successful for home  /apps/oracle/product/11.2.0.3/RACDB

Exadata X2-2 Summary


Quarter        Half       Full
Servers 2 4 8
Cores 24 48 96
Storage Servers 3 7 14
Number of disks 36 84 168
SAS (Small and Fast)  TB 21 50 100
SATA (Big and slower) TB 72 168 336
Infiniband 2 3 3
IOPS Flash  375000 750000 1500000

How to shutdown Exadata Storage Server

It’s necessary to shutdown the cell  when performaing maintenance on the Exadata Storage Server.  It’s also important to verify that taking Exadata Storage server offline will not impact Oracle ASM disk group and database availability depends on the ASM redundancy and mirror copies of data.  Below are the high level of how to shutdown Exadata Storage Server

CellCLI> list griddisk attributes name where asmdeactivationoutcome !='Yes'
If any grid disks are returned (like below), then it’s not safe to take the storage offline
                data_CD_1_ancell05
                data_CD_2_ancell05
                data_CD_3_ancell05
                data_CD_4_ancell05
                data_CD_5_ancell05
                data_CD_6_ancell05
                data_FD_00_ancell05
                data_FD_01_ancell05
                data_FD_02_ancell05A
                data_FD_03_ancell05
                reco_CD_7_ancell05
                reco_CD_8_ancell05
                reco_CD_9_ancell05
                reco_CD_disk10_ancell05
                reco_CD_disk11_ancell05
                reco_CD_disk12_ancell05


Inactivate the grid disks before bring Exadata Storage server offline
CellCLI> alter griddisk all inactive
GridDisk data_CD_1_ancell05 successfully altered
GridDisk data_CD_2_ancell05 successfully altered
GridDisk data_CD_3_ancell05 successfully altered
GridDisk data_CD_4_ancell05 successfully altered
….
….

Cellcli > list griddisk where status != ‘inactive’
If all grid disks are inactive, you can power down the exadata storage server

To power off Exadata Storage Servers
shutdown -h -y now

Apply 11.2.0.3.1 Grid Infrastructure Patch Set Update (GI PSU)


+ASM1 > $ORACLE_HOME/OPatch/ocm/bin/emocmrsp

Provide your email address to be informed of security issues, install and
initiate Oracle Configuration Manager. Easier for you if you use your My
Oracle Support Email address/User Name.
Visit
http://www.oracle.com/support/policies.html for details.
Email address/User Name:
ashley.nguyen@dbalab.com
Provide your My Oracle Support password to receive security updates via your My Oracle Support account.
Password (optional):          
The OCM configuration response file (ocm.rsp) was successfully created.

mv ocm.rsp /tmp

sudo -u root opatch auto /apps/oracle/software/psu.11.2.0.3.1 -oh /apps/11.2.0/grid -ocmrf /tmp/ocm.rsp

<===Failed ==>

If the opatch auto failed, you may need to re-run the command again.  See the example below

Re-run it and enter yes if this is not a shared home.

sudo -u root opatch auto /apps/oracle/software/psu.11.2.0.3.1 -oh /apps/11.2.0/grid -ocmrf /tmp/ocm.rsp

Executing /usr/bin/perl /apps/11.2.0/grid/OPatch/crs/patch112.pl -patchdir /apps/oracle/software -patchn psu.11.2.0.3.1 -oh /apps/11.2.0/grid -ocmrf /tmp/ocm.rsp -paramfile /apps/11.2.0/grid/crs/install/crsconfig_params
opatch auto log file location is /apps/11.2.0/grid/OPatch/crs/../../cfgtoollogs/opatchauto2012-06-04_17-41-29.log
Detected Oracle Clusterware install
Using configuration parameter file: /apps/11.2.0/grid/crs/install/crsconfig_params

Unable to determine if /apps/11.2.0/grid is shared oracle home
Enter 'yes' if this is not a shared home or if the prerequiste actions are performed to patch this shared home (yes/no):yes
Successfully unlock /apps/11.2.0/grid
patch /apps/oracle/software/psu.11.2.0.3.1/13348650  apply successful for home  /apps/11.2.0/grid
patch /apps/oracle/software/psu.11.2.0.3.1/13343438  apply successful for home  /apps/11.2.0/grid
ACFS-9300: ADVM/ACFS distribution files found.
ACFS-9312: Existing ADVM/ACFS installation detected.
ACFS-9314: Removing previous ADVM/ACFS installation.
ACFS-9315: Previous ADVM/ACFS components successfully removed.
ACFS-9307: Installing requested ADVM/ACFS software.
ACFS-9308: Loading installed ADVM/ACFS drivers.
ACFS-9321: Creating udev for ADVM/ACFS.
ACFS-9323: Creating module dependencies - this may take some time.
ACFS-9154: Loading 'oracleoks.ko' driver.
ACFS-9154: Loading 'oracleadvm.ko' driver.
ACFS-9154: Loading 'oracleacfs.ko' driver.
ACFS-9327: Verifying ADVM/ACFS devices.
ACFS-9156: Detecting control device '/dev/asm/.asm_ctl_spec'.
ACFS-9156: Detecting control device '/dev/ofsctl'.
ACFS-9309: ADVM/ACFS installation correctness verified.
CRS-4123: Oracle High Availability Services has been started.