Restore OCR and Vote after corrupted ASM CRS diskgroup

From the below alert logs, the CRS (OCR, VOTE in ASM diskgroup) was corrupted.

WARNNING: cache read a corrupted block group=1(GRID) fn=6 blk=2 from disk 1(APP_PV_DISK1)
Errors in file /apps/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_ora_8335.trc:
ORA-15196: invalid ASM block header [kfc.c:25316] [endian_kfbh] [6] [2] [0 != 1]
ORA-15196: invalid ASM block header [kfc.c:25165] [endian_kfbh] [6] [2] [0 != 1]
NOTE: a corrupted block from group GRID was dumped to /apps/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_ora_8335.trc
WARNNING: cache read(retry) a corrupted block group=1(GRID) fn=6 blk=2 from disk 1(APP_PV_DISK1)
Errors in file /apps/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_ora_8335.trc:
ORA-15196: invalid ASM block header [kfc.c:25316] [endian_kfbh] [6] [2] [0 != 1]
ORA-15196: invalid ASM block header [kfc.c:25165] [endian_kfbh] [6] [2] [0 != 1]

The fastest way to fix this problem is to recreate and recover the ocr and vote ASM diskgroup by following the below steps:

1) Stop crs from all nodes

                crsctl stop crs –f

2) To clear out ASM header of the corrupted ASM that contains OCR and vote files

sudo -u root dd if=/dev/zero of=/dev/emcpowerb9 bs=8192 count=12800

sudo -u root dd if=/dev/zero of=/dev/emcpowerc9 bs=8192 count=12800

sudo -u root dd if=/dev/zero of=/dev/emcpoweri9 bs=8192 count=12800

3)  Label the CRS disks

/etc/init.d/oracleasm deletedisk APP_PV_DISK1

/etc/init.d/oracleasm createdisk APP_PV_DISK1 /dev/emcpowerb9

/etc/init.d/oracleasm deletedisk APP_PV_DISK2

/etc/init.d/oracleasm createdisk APP_PV_DISK2 /dev/emcpowerc9

/etc/init.d/oracleasm deletedisk APP_PV_DISK3

/etc/init.d/oracleasm createdisk APP_PV_DISK3 /dev/emcpoweri9

4)  Rescan disks from all nodes as root

/usr/sbin/oracleasm scandisks

5)  Start CRS in exclusive mode to prevent ora.crsd to start up

[root@rac01bin]# ./crsctl start crs -excl -nocrs

6)  Create CRS diskgroup

SQL> create diskgroup GRID normal redundancy disk 'ORCL:APP_PV_DISK1','ORCL:APP_PV_DISK2','ORCL:APP_PV_DISK3' attribute=’11.2’;

Diskgroup created.

SQL> select name, state from v$asm_diskgroup;

NAME                           STATE

------------------------------ -----------

GRID                             MOUNTED

7)  Restore OCR from good / latest backup

cd /u01/grid/11.2.0/grid/cdata/appdev

sudo -u root /u01/grid/11.2.0/grid/bin/ocrconfig -restore backup00.ocr

8)  Recreate the vote disk

sudo -u root /u01/grid/11.2.0/grid/bin/crsctl replace votedisk +GRID

Successful addition of voting disk a47251fdbb914f2bbfe0abbfac752af0.

Successful addition of voting disk 9e34471ba0134f67bf90e922d1af2e3e.

Successful addition of voting disk 433dcfc8a0664f5bbf1a7b264ab108ce.

Successfully replaced voting disk group with +GRID.

CRS-4266: Voting file(s) successfully replaced

9)  Stop and restart crs

crsctl stop crs –f

crsctl start crs