OCR和voting disk损坏恢复

内容导航：

OCR和voting disk损坏恢复

如何恢复，迁移，添加，删除 Voting Disks

如何干净的清除并重建ocr和表决磁盘

如何删除asm磁盘卷组 oracleasm listdisks

一、OCR和voting disk损坏恢复

介绍

有时候服务器或存储的突发情况，导致数据库服务器重启同时出现OCR磁盘丢失报错，导致oracle数据库集群宕机。这时，就需要我们来恢复OCR磁盘和voting disk，从而恢复集群！整个过程中，数据库是无法使用的！

本文数据库版本11.2.0.4 RAC，使用dd命令来模拟ocr磁盘损坏，然后通过ocr的自动备份来恢复ocr信息，然后重建voting disk！

检查当前可用资源

[grid@rac1 ~]$ asmcmd

ASMCMD> lsdg

State Type Rebal Sector Block AU Total_MB Free_MB Req_mir_free_MB Usable_file_MB Offline_disks Voting_files Name

MOUNTED EXTERN N 512 4096 1048576 20473 18230 0 18230 0 N DBFILE/

MOUNTED EXTERN N 512 4096 1048576 13178 12448 0 12448 0 N FRA/

MOUNTED NORMAL N 512 4096 1048576 6141 5215 2047 1584 0 Y OCR/

检查ocr备份

[grid@rac1 ~]$ ocrconfig -showbackup

注意：因为我使用的是虚拟机，且不常使用这个环境，所以这个自动备份时间有点早，但是依然可以使用！

检查vote disk

[grid@rac1 ~]$ crsctl query css votedisk

图中显示的是当前votedisk相关信息！

创建pfile文件

创建一个ASM实例的pfile文件，后面会用到！！！ ---- 或者可以手动创建一个

SQL> create pfile= '/tmp/pfile' from spfile;

模拟OCR磁盘损坏

由于OCR磁盘组使用normal冗余模式，所以损坏掉了两个才无法启动！！！

使用DD命令将OCR的磁盘损坏！！！

dd if=/dev/zero of=/dev/mapper/mpathfp1 bs=1M count=100 ----损坏一块照常工作！！！

dd if=/dev/zero of=/dev/mapper/mpathgp1 bs=1M count=100

dd if=/dev/zero of=/dev/mapper/mpathhp1 bs=1M count=100

这里使用dd命令将磁盘头的信息清空！！

等一段时间后，grid下的alter日志会出现如下信息：

[cssd(2912)]CRS-1615:No I/O has completed after 50% of the maximum interval. Voting file /dev/oracleasm/disks/OCR01 will be considered not functional in 99370 milliseconds

[cssd(2957)]CRS-1614:No I/O has completed after 75% of the maximum interval. Voting file /dev/oracleasm/disks/OCR01 will be considered not functional in 49600 milliseconds

.......

[crsd(3534)]CRS-1006:The OCR location +OCR is inaccessible. Details in /u01/app/grid/log/rac1/crsd/crsd.log.

2021-03-18 13:14:33.229:

[/u01/app/grid/bin/oraagent.bin(3693)]CRS-5822:Agent '/u01/app/grid/bin/oraagent_grid' disconnected from server. Details at (:CRSAGF00117:) {0:3:6} in /u01/app/grid/log/rac1/agent/crsd/oraagent_grid/oraagent_grid.log.

如果需要更快出现报错，直接重启CRS！！！！

[root@rac1 bin]# ps -elf|grep smon

4 S root 2914 1 3 -40 - - 193678 hrtime 12:38 ? 00:01:21 /u01/app/grid/bin/osysmond.bin

0 S grid 3500 1 0 80 0 - 54961 semtim 12:39 ? 00:00:00 asm_smon_+ASM1

0 S root 9469 2759 0 80 0 - 25814 pipe_w 13:17 pts/0 00:00:00 grep smon

[root@rac1 bin]# ./crsctl stop crs

CRS-2796: The command may not proceed when Cluster Ready Services is not running

CRS-4687: Shutdown command has completed with errors.

CRS-4000: Command Stop failed, or completed with errors.

[root@rac1 bin]# ./crsctl check crs

CRS-4638: Oracle High Availability Services is online

CRS-4535: Cannot communicate with Cluster Ready Services

CRS-4529: Cluster Synchronization Services is online

CRS-4533: Event Manager is online

进程还在但是无法显示集群任何信息！！！！！

日志文件中会显示：

[crsd(9190)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/app/grid/log/rac1/crsd/crsd.log.

2021-03-18 13:15:11.080:

[crsd(9190)]CRS-0804:Cluster Ready Service aborted due to Oracle Cluster Registry error [PROC-26: Error while accessing the physical storage

]. Details at (:CRSD00111:) in /u01/app/grid/log/rac1/crsd/crsd.log.

2021-03-18 13:15:11.668:

[ohasd(2636)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac1'.

2021-03-18 13:15:13.158:

[crsd(9203)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/app/grid/log/rac1/crsd/crsd.log.

2021-03-18 13:15:13.170:

[crsd(9203)]CRS-0804:Cluster Ready Service aborted due to Oracle Cluster Registry error [PROC-26: Error while accessing the physical storage

]. Details at (:CRSD00111:) in /u01/app/grid/log/rac1/crsd/crsd.log.

2021-03-18 13:15:13.785:

[ohasd(2636)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac1'.

2021-03-18 13:15:15.274:

[crsd(9220)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/app/grid/log/rac1/crsd/crsd.log.

2021-03-18 13:15:15.294:

[crsd(9220)]CRS-0804:Cluster Ready Service aborted due to Oracle Cluster Registry error [PROC-26: Error while accessing the physical storage

]. Details at (:CRSD00111:) in /u01/app/grid/log/rac1/crsd/crsd.log.

2021-03-18 13:15:20.339:

[ohasd(2636)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac1'.

2021-03-18 13:15:20.340:

[ohasd(2636)]CRS-2771:Maximum restart attempts reached for resource 'ora.crsd'; will not restart.

2021-03-18 13:15:20.420:

[ohasd(2636)]CRS-2769:Unable to failover resource 'ora.crsd'.

........

[/u01/app/grid/bin/oraagent.bin(9910)]CRS-5019:All OCR locations are on ASM disk groups [OCR], and none of these disk groups are mounted. Details are at "(:CLSN00100:)" in "/u01/app/grid/log/rac1/agent/ohasd/oraagent_grid/oraagent_grid.log".

[client(12353)]CRS-10001:18-Mar-21 13:31 ACFS-9290: Waiting for ASM to shutdown.

[client(12406)]CRS-10001:18-Mar-21 13:31 ACFS-9290: Waiting for ASM to shutdown.

[client(12424)]CRS-10001:18-Mar-21 13:31 ACFS-9290: Waiting for ASM to shutdown.

由于OCR磁盘损坏了，在正常停止crs时，会一直等待 ASM关闭，所以这里直接手动abort掉！！！

强制停止crs

[root@rac1 bin]# ./crsctl stop crs -f

注意：使用-f 强制停止crs，把2个节点的crs都停止掉！

以独占模式启动crs

在节点1上：以独占模式启动crs

[root@rac1 bin]# ./crsctl start crs -excl -nocrs

参数说明：

-excl 该参数指定使用独占模式

-nocrs 该参数指定忽略查找crs及voting

11.2.0.1:

# $CRS_HOME/bin/crsctl start crs -excl

11.2.0.2 and above:

# $CRS_HOME/bin/crsctl start crs -excl -nocrs

使用创建好的pfile文件重启启动asm实例

这里使用最开始已经准备好的pfile文件(grid用户执行)

SQL>startup pfile='/tmp/pfile';

查询信息可以看到ASM磁盘信息为空：

SQL>select group_number, disk_number, mount_status, header_status, path from v$asm_disk;

创建OCR磁盘组

我们还是使用原来asm磁盘来重新创建一个OCR磁盘组，并且取同样的名称！(grid用户)

sqlplus / as sysasm

SQL>create diskgroup OCR normal redundancy disk '/dev/oracleasm/disks/OCR01','/dev/oracleasm/disks/OCR02', '/dev/oracleasm/disks/OCR03' attribute 'compatible.asm'='11.2.0.0.0';

发现OCR已经上线，但是磁盘组中没有任何文件！！！其他两个磁盘组没有损坏，所以文件依旧存在。

[grid@rac1 ~]$ asmcmd

ASMCMD> lsdg

State Type Rebal Sector Block AU Total_MB Free_MB Req_mir_free_MB Usable_file_MB Offline_disks Voting_files Name

MOUNTED EXTERN N 512 4096 1048576 20473 18230 0 18230 0 N DBFILE/

MOUNTED EXTERN N 512 4096 1048576 13178 12448 0 12448 0 N FRA/

MOUNTED NORMAL N 512 4096 1048576 6141 5982 2047 1967 0 N OCR/

创建ASM实例的spfile文件

SQL>create spfile='+OCR' from pfile='/tmp/pfile';

还原OCR文件

OCR文件会有自动备份功能：/u01/app/grid/cdata/raccluster

找一个最新的文件用来做还原！！！！

[root@rac1 ~]# cd /u01/app/grid/bin/

[root@rac1 bin]# ./ocrconfig -restore /u01/app/grid/cdata/raccluster/backup00.ocr

检查OCR：

恢复voting disk

如果遇到报错，需要将crs强制停止，然后重新以独占模式启动！！！！

[root@rac1 bin]# ./crsctl replace votedisk +OCR

[root@rac1 bin]# ./crsctl query css votedisk

检查集群环境

强制关闭crs,然后正常重启2个节点的crs！！！！ 查看日志， 启动需要一定时间！！！

[root@rac1 bin]# ./crsctl start crs

CRS-4123: Oracle High Availability Services has been started.

通过下面的命令检查：

crsctl status resource -t

ocrcheck

crsctl query css votedisk

crsctl check cluster -all

检查完成后，简单测试下数据库能否正常使用！

注意：直接重启服务器后，,有可能丢失磁盘标签，导致找不到磁盘，这时通过查看alter日志可以发现，报错信息是找不到OCR磁盘，这是因为使用dd命令有可能将磁盘上得asm磁盘标签给清理了，需要重新将标签补充上！！！！！

oracleasm renamedisk -f /dev/mapper/mpathfp1 ocr01

oracleasm renamedisk -f /dev/mapper/mpathgp1 ocr02

oracleasm renamedisk -f /dev/mapper/mpathhp1 ocr03

一、如何恢复，迁移，添加，删除 Voting Disks

恢复流程

在11gR2 之前，我们可以直接直接使用dd命令对voting disk进行备份。

DD示例

备份votedisk盘：

[root@raw1 bin]# dd if=/dev/raw/raw2 of=/home/oracle/voting_disk.bak

恢复votedisk盘：

[root@raw1 bin]# dd if=/home/oracle/voting_disk.bak of=/dev/raw/raw2

Oracle 11gR2中已经不支持使用dd命令来对voting disks 进行备份和恢复。

11gR2:

1. 如果当OCR 也损坏的情况，先恢复OCR,否则不需要。

2. 用root用户执行如下命令

用root用户，在其中一个节点启动Oracle Clusterware stack到exclusive 模式，这个模式不需要voting disk。

# crsctl start crs -excl

3. 检查当前定义的voting disk

执行如下命令检查当前的voting disk信息： (如果所有的voting disks都已经损坏，该命令就会返回空，也可能state被标记3或OFF.)

[root@node1 bin]# ./crsctl query css votedisk

## STATE File Universal Id File Name Disk group

-- ----- ----------------- --------- ---------

1. ONLINE 97b3037ba6684f0bbf04fa53aa7efb37 (ORCL:VOL1) [GRIDDG]

Located 1 voting disk(s).

4. 还原voting disk

4.1 如果voting disk 存储在Oracle ASM 上，那么运行如下SQL，将voting disk 转移我们指定的diskgroup中：

crsctl replace votedisk +asm_disk_group

4.2 如果voting disk没有存在Oracle ASM中，那么执行如下SQL，在执行之前需要先获取FUID。

$ crsctl delete css votedisk FUID

添加voting disk：

$ crsctl add css votedisk path_to_voting_disk

$ crsctl query css votedisk

5. 用root用户停止Oracle Clusterware Stack

# crsctl stop crs

然后使用root用户执行如下命令，将Clusterware stack启动到正常模式：

# crsctl start crs

迁移

查看当前每个voting disk的FUID 和路径 结果对应->disk的序列号，磁盘status，FUID，disk 路径，和存储voting disk的磁盘组名称。）

# crsctl query css votedisk2.1 如果想voting disk 从Oracle ASM 中迁移到其他的cluster file system中，迁移命令如下： （任意节点）

$ crsctl replace votedisk path_to_voting_disk

2 如果想把所有的voting disk从其他存储设备中迁移到oracle ASM diskgroup中，执行如下命令：（任意节点）

$ crsctl replace votedisk +asm_disk_group

3. 验证voting disk 的位置

$ crsctl query css votedisk

添加、删除

1（非asm环境，当使用Oracle ASM 来存储voting disk时，不能使用crsctl add|delete voteding 来修改voting disk）

$ crsctl add css votedisk path_to_voting_disk

$ crsctl delete css votedisk path_to_voting_disk

二、如何干净的清除并重建ocr和表决磁盘

您好，很高兴为您解答。

11gR2干净的清除并重建OCR和表决磁盘的方法

执行：$GRID_HOME/crs/install/rootcrs.pl -deconfig -force
如果是RAC的最后一个节点，那么执行：$GRID_HOME/crs/install/rootcrs.pl -deconfig -force -lastnode

1).如果在RAC中有任何节点是正常工作的，那么在问题节点执行以上命令来清除本节点的OCR和表决磁盘信息，然后重建，不必直接删除OCR和表决磁盘文件进行重建。
2).如果RAC所有节点都出现问题，需要所有节点重建OCR和表决磁盘，那么也不能直接将OCR和表决磁盘删除，重新执行root.sh进行重建，必须同样的执行以上命令，如果将OCR和表决磁盘文件之间删除，执行root.sh脚本会收到如下信息：

[root@niccl02 grid]# ./root.sh
Performing root user operation for Oracle 11g

The following environment variables are set as:
ORACLE_OWNER= grid
ORACLE_HOME= /u01/app/11.2.0/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /u01/app/11.2.0/grid/crs/install/crsconfig_params
User ignored Prerequisites during installation
Configure Oracle Grid Infrastructure for a Cluster ... succeeded

上面代码显示是成功的，实际上什么也没做。

如若满意，请点击右侧【采纳答案】，如若还有问题，请点击【追问】

希望我的回答对您有所帮助，望采纳！

~ O(∩_∩)O~

三、如何删除asm磁盘卷组 oracleasm listdisks

ASM磁盘组冗余的三种类型：external、normal、high,这里恢复的是normal状态,模拟OCR磁盘或votedisk不可用时，RAC会出现什么现象？给出故障定位的整个过程。在11.2.0.3中表决盘是放到了ocr中，所以OCR磁盘或votedisk不可用的两个实验一起做。在11.2.0.3中可　　ASM磁盘组冗余的三种类型：external、normal、high,这里恢复的是normal状态,模拟OCR磁盘或votedisk不可用时，RAC会出现什么现象？给出故障定位的整个过程。在11.2.0.3中表决盘是放到了ocr中，所以OCR磁盘或votedisk不可用的两个实验一起做。在11.2.0.3中可以手动备份OCR，但手动备份是无效的。　　ocrconfig-export/u01/ocr.exp检查OCR有哪些备份：　　[root@rac1~]#ocrconfig-showbackup　　rac12013/07/2205:39:51/u01/grid/crs/cdata/rac/backup00.ocr　　rac12013/07/2201:39:51/u01/grid/crs/cdata/rac/backup01.ocr　　rac12013/07/2121:39:50/u01/grid/crs/cdata/rac/backup02.ocr　　rac22013/07/2101:52:54/u01/grid/crs/cdata/rac/day.ocr　　rac22013/07/0901:52:25/u01/grid/crs/cdata/rac/week.ocr　　PROT-25:ManualbackupsfortheOracleClusterRegistryarenotavailable注意：orcle明确给出了手动备份是无效的！　　查看表决盘信息：　　[root@rac1~]#crsctlquerycssvotedisk　　##STATEFileUniversalIdFileNameDiskgroup　　------------------------------------------　　1.ONLINE745716af7e5b4faebfc8d948d096aa55(/dev/oracleasm/disks/OCR_VOT1)[OCR_VOT]　　2.ONLINE7092079f66c04f9dbf65974d0dcc611a(/dev/oracleasm/disks/OCR_VOT2)[OCR_VOT]　　3.ONLINE6510631353284f5fbf3d4c8839822dbd(/dev/oracleasm/disks/OCR_VOT3)[OCR_VOT]　　Located3votingdisk(s).　　#停库：　　[root@rac1~]#srvctlstopdatabase-dorcl-oimmediate　　#停集群：　　[root@rac1~]#crsctlstopcluster-all-f　　#破坏OCR和VOT：　　[root@rac1~]#ddif=/dev/zerof=/dev/mapper/mpathap1bs=1024Kcount=1　　记录了1+0的读入　　记录了1+0的写出　　1048576字节(1.0MB)已复制，0.0160613秒，65.3MB/秒　　[root@rac1~]#ddif=/dev/zerof=/dev/mapper/mpathap2bs=1024Kcount=1　　记录了1+0的读入　　记录了1+0的写出　　1048576字节(1.0MB)已复制，0.00800275秒，131MB/秒　　[root@rac1~]#ddif=/dev/zerof=/dev/mapper/mpathap3bs=1024Kcount=1　　记录了1+0的读入　　记录了1+0的写出　　1048576字节(1.0MB)已复制，0.00927389秒，113MB/秒注意：破坏后，各节点服务一切正常：　　[root@rac1~]#crs_stat-t　　NameTypeTargetStateHost　　------------------------------------------------------------　　ora.DATA.dgora.up.typeONLINEONLINErac1　　ora.FRA.dgora.up.typeONLINEONLINErac1　　ora.ER.lsnrora.er.typeONLINEONLINErac1　　ora.N1.lsnrora.er.typeONLINEONLINErac1　　ora.OCR_VOT.dgora.up.typeONLINEONLINErac1　　ora.asmora.asm.typeONLINEONLINErac1　　ora.orcl.dbora.se.typeONLINEONLINErac1　　ora.cvuora.cvu.typeONLINEONLINErac1　　ora.SM1.asmapplicationONLINEONLINErac1　　ora.C1.lsnrapplicationONLINEONLINErac1　　ora.ac1.gsdapplicationOFFLINEOFFLINE　　ora.ac1.onsapplicationONLINEONLINErac1　　ora.ac1.vipora.t1.typeONLINEONLINErac1　　ora.SM2.asmapplicationONLINEONLINErac2　　ora.C2.lsnrapplicationONLINEONLINErac2　　ora.ac2.gsdapplicationOFFLINEOFFLINE　　ora.ac2.onsapplicationONLINEONLINErac2　　ora.ac2.vipora.t1.typeONLINEONLINErac2　　ora.gsdora.gsd.typeOFFLINEOFFLINE　　ora.networkora.rk.typeONLINEONLINErac1　　ora.oc4jora.oc4j.typeONLINEONLINErac1　　ora.onsora.ons.typeONLINEONLINErac1　　ora.ry.acfsora.fs.typeONLINEONLINErac1　　ora.scan1.vipora.ip.typeONLINEONLINErac1所有节点重启操作系统后集群服务启不来了：　　[root@rac1~]#reboot如果只是停止集群服务，后面的重新创建ASM磁盘组会失败，但重启操作系统后，就可以创建成功。　　检查CRS：　　[grid@rac1~]$crsctlcheckcrs　　CRS-4638:OracleHighAvailabilityServicesisonline　　CRS-4535:CannotcommunicatewithClusterReadyServices　　CRS-4530:CommunicationsfailurecontactingClusterSynchronizationServicesdaemon　　CRS-4534:CannotcommunicatewithEventManager启动集群服务：　　[root@rac1~]#crsctlstartcluster-all　　CRS-2672:尝试启动'ora.cssdmonitor'(在'rac1'上)　　CRS-2672:尝试启动'ora.cssdmonitor'(在'rac2'上)　　CRS-2676:成功启动'ora.cssdmonitor'(在'rac1'上)　　CRS-2676:成功启动'ora.cssdmonitor'(在'rac2'上)　　CRS-2672:尝试启动'ora.cssd'(在'rac1'上)　　CRS-2672:尝试启动'ora.diskmon'(在'rac1'上)　　CRS-2672:尝试启动'ora.cssd'(在'rac2'上)　　CRS-2672:尝试启动'ora.diskmon'(在'rac2'上)　　CRS-2676:成功启动'ora.diskmon'(在'rac1'上)　　CRS-2676:成功启动'ora.diskmon'(在'rac2'上)　　#直停在这里其他终端使用其他命令启动集群服务：　　[root@rac1~]#crsctlstartcrs　　CRS-4640:OracleHighAvailabilityServicesisalreadyactive　　CRS-4000:CommandStartfailed,orcompletedwitherrors.操作系统及crs日志中没看到特别有用的信息：　　[root@rac1~]#vi/var/log/messages　　[grid@rac1~]#vi$ORACLE_HOME/log/rac1/crsd/crsd.logocss日志中提示：　　vi$ORACLE_HOME/log/rac1/cssd/ocssd.log　　2013-07-2121:15:08.550:[CSSD][1095031104]clssnmvFindInitialConfigs:Novotingfilesfound发现部分ASM磁盘没有了：　　[root@rac1~]#/etc/init.d/oracleasmscandisks　　ScanningthesystemforOracleASMLibdisks:[OK]　　[root@rac1~]#/etc/init.d/oracleasmlistdisks　　DATA　　FRA依照RAC安装文档重建ASM磁盘：　　[root@rac1~]#/etc/init.d/oracleasmcreatediskOCR_VOT1/dev/mapper/mpathap1　　Markingdisk"OCR_VOT1"asanASMdisk:[OK]　　[root@rac1~]#/etc/init.d/oracleasmcreatediskOCR_VOT2/dev/mapper/mpathap2　　Markingdisk"OCR_VOT2"asanASMdisk:[OK]　　[root@rac1~]#/etc/init.d/oracleasmcreatediskOCR_VOT3/dev/mapper/mpathap3　　Markingdisk"OCR_VOT3"asanASMdisk:[OK]停掉集群服务：　　要加-f，否则可能停止非常慢　　[root@rac1~]#crsctlstopcrs-f　　CRS-2791:StartingshutdownofOracleHighAvailabilityServices-managedresourceson'rac1'　　CRS-2673:Attemptingtostop'ora.mdnsd'on'rac1'　　CRS-2673:Attemptingtostop'ora.crf'on'rac1'　　CRS-2677:Stopof'ora.mdnsd'on'rac1'succeeded　　CRS-2677:Stopof'ora.crf'on'rac1'succeeded　　CRS-2673:Attemptingtostop'ora.gipcd'on'rac1'　　CRS-2677:Stopof'ora.gipcd'on'rac1'succeeded　　CRS-2673:Attemptingtostop'ora.gpnpd'on'rac1'　　CRS-2677:Stopof'ora.gpnpd'on'rac1'succeeded　　CRS-2793:ShutdownofOracleHighAvailabilityServices-managedresourceson'rac1'hascompleted　　CRS-4133:OracleHighAvailabilityServiceshasbeenstopped.以-excl-nocrs方式启动集群，这将启动ASM实例但不启动CRS　　[root@rac1~]#crsctlstartcrs-excl-nocrs　　CRS-4123:OracleHighAvailabilityServiceshasbeenstarted.　　CRS-2672:Attemptingtostart'ora.mdnsd'on'rac1'　　CRS-2676:Startof'ora.mdnsd'on'rac1'succeeded　　CRS-2672:Attemptingtostart'ora.gpnpd'on'rac1'　　CRS-2676:Startof'ora.gpnpd'on'rac1'succeeded　　CRS-2672:Attemptingtostart'ora.cssdmonitor'on'rac1'　　CRS-2672:Attemptingtostart'ora.gipcd'on'rac1'　　CRS-2676:Startof'ora.cssdmonitor'on'rac1'succeeded　　CRS-2676:Startof'ora.gipcd'on'rac1'succeeded　　CRS-2672:Attemptingtostart'ora.cssd'on'rac1'　　CRS-2672:Attemptingtostart'ora.diskmon'on'rac1'　　CRS-2676:Startof'ora.diskmon'on'rac1'succeeded　　CRS-2676:Startof'ora.cssd'on'rac1'succeeded　　CRS-2672:Attemptingtostart'ora.drivers.acfs'on'rac1'　　CRS-2679:Attemptingtoclean'ora.cluster_interconnect.haip'on'rac1'　　CRS-2672:Attemptingtostart'ora.ctssd'on'rac1'　　CRS-2681:Cleanof'ora.cluster_interconnect.haip'on'rac1'succeeded　　CRS-2672:Attemptingtostart'ora.cluster_interconnect.haip'on'rac1'　　CRS-2676:Startof'ora.drivers.acfs'on'rac1'succeeded　　CRS-2676:Startof'ora.ctssd'on'rac1'succeeded　　CRS-2676:Startof'ora.cluster_interconnect.haip'on'rac1'succeeded　　CRS-2672:Attemptingtostart'ora.asm'on'rac1'　　CRS-2676:Startof'ora.asm'on'rac1'succeeded此时crs仍然报错：　　[root@rac1~]#crs_stat-t　　CRS-0184:CannotcommunicatewiththeCRSdaemon.　　[root@rac1~]#crsctlcheckcrs　　CRS-4638:OracleHighAvailabilityServicesisonline　　CRS-4535:CannotcommunicatewithClusterReadyServices　　CRS-4530:CommunicationsfailurecontactingClusterSynchronizationServicesdaemon　　CRS-4534:CannotcommunicatewithEventManager重建原ocr和votedisk所在磁盘组：　　注意：这里是在grid用户下　　SQL>colpathfora50　　SQL>setlines300　　SQL>selectpath,header_statusfromv$asm_disk;　　SQL>creatediskgroupOCR_VOTnormalredundancydisk'/dev/oracleasm/disks/OCR_VOT1','/dev/　　oracleasm/disks/OCR_VOT2','/dev/oracleasm/disks/OCR_VOT3'　　attribute'compatible.rdbms'='11.2','compatible.asm'='11.2';ASM磁盘组冗余的三种类型：external、normal、high，我这里之前用的是normal。　　从ocrbackup中恢复OCR：　　在每个节点grid用户下：　　cd$ORACLE_HOME/cdata/rac　　ocrconfig-restore/u01/grid/crs/cdata/rac/backup00.ocr恢复表决盘的准备工作：　　showparameterasm_diskstring如果asm_diskstring没有值，表示ASM磁盘用的是默认ASM磁盘搜索路径。　　修改成实际的ASM磁盘搜索路径：　　altersystemsetasm_diskstring='/dev/oracleasm/disks/*';恢复表决盘：　　[root@rac1~]#crsctlreplacevotedisk+OCR_VOT　　Successfuladditionofvotingdisk4ad2b9cc0a754fffbf1515281199a78f.　　Successfuladditionofvotingdisk9f8dc1c013df4f39bfd85c64051a0bc1.　　Successfuladditionofvotingdiska4aea7a1aa434fb3bff161f6ea8ce102.　　Successfullyreplacedvotingdiskgroupwith+OCR_VOT.　　CRS-4266:Votingfile(s)successfullyreplacedocr和vot恢复后，crs等服务就会自动起来了。　　[root@rac1~]#crsctlcheckcrs　　CRS-4638:OracleHighAvailabilityServicesisonline　　CRS-4535:CannotcommunicatewithClusterReadyServices　　CRS-4529:ClusterSynchronizationServicesisonline　　CRS-4534:CannotcommunicatewithEventManager　　[root@rac1~]#crsctlquerycssvotedisk　　##STATEFileUniversalIdFileNameDiskgroup　　------------------------------------------　　1.ONLINE4ad2b9cc0a754fffbf1515281199a78f(/dev/oracleasm/disks/OCR_VOT1)[OCR_VOT]　　2.ONLINE9f8dc1c013df4f39bfd85c64051a0bc1(/dev/oracleasm/disks/OCR_VOT2)[OCR_VOT]　　3.ONLINEa4aea7a1aa434fb3bff161f6ea8ce102(/dev/oracleasm/disks/OCR_VOT3)[OCR_VOT]　　Located3votingdisk(s).重启集群服务，检查是否已经恢复正常：　　[root@rac1~]#crsctlstopcrs　　[root@rac1~]#crsctlstartcrs