11GR2RAC丢失OCR和CONTROLFILE重建恢复过程

今天准备用自己的rac虚拟机做点实验的，可是虚拟机拉起来时，频频报错，估计好久不用了（前一阵子有厂商来推销虚拟化全在那上做测试了），挨个检查发现是OCR出故障了，没办法重建下吧，顺便记录下重建的过程

专业领域包括网站设计制作、成都网站建设、商城网站定制开发、微信营销、系统平台开发， 与其他网站设计及系统开发公司不同，成都创新互联公司的整合解决方案结合了帮做网络品牌建设经验和互联网整合营销的理念，并将策略和执行紧密结合，为客户提供全网互联网整合方案。

1.使用orccheck命令报错

[root@rac1 ~]#ocrcheck

PROT-602: Failedto retrieve datafrom the cluster registry

PROC-26: Errorwhile accessing the physical storage

2.使用ocrconfig-restore恢复仍然报错

[root@rac1 rac-cluster]#ocrconfig-restore/u01/app/11.2.0/grid/cdata/rac-cluster/backup00.ocr

PROT-35: The configured OCR locations arenot accessible.

3.强制停掉两个节点的crs（两个节点都要执行）

[root@rac1 rac-cluster]#crsctl stop crs-f

CRS-2791:Starting shutdown of Oracle High Availability Services-managed resources on 'rac1'

CRS-2673:Attempting to stop 'ora.mDNSd' on 'rac1'

CRS-2673:Attempting to stop 'ora.ctssd' on 'rac1'

CRS-2673:Attempting to stop 'ora.evmd' on 'rac1'

CRS-2673:Attempting to stop 'ora.asm' on 'rac1'

CRS-2677:Stop of 'ora.evmd' on 'rac1' succeeded

CRS-2677:Stop of 'ora.mdnsd' on 'rac1' succeeded

CRS-2677:Stop of 'ora.ctssd' on 'rac1' succeeded

CRS-2677:Stop of 'ora.asm' on 'rac1' succeeded

CRS-2673:Attempting to stop 'ora.cluster_interconnect.haip' on 'rac1'

CRS-2677:Stop of 'ora.cluster_interconnect.haip' on 'rac1' succeeded

CRS-2673:Attempting to stop 'ora.cssd' on 'rac1'

CRS-2677:Stop of 'ora.cssd' on 'rac1' succeeded

CRS-2673:Attempting to stop 'ora.crf' on 'rac1'

CRS-2677:Stop of 'ora.crf' on 'rac1' succeeded

CRS-2673:Attempting to stop 'ora.gipcd' on 'rac1'

CRS-2677:Stop of 'ora.gipcd' on 'rac1' succeeded

CRS-2673:Attempting to stop 'ora.gpnpd' on 'rac1'

CRS-2677:Stop of 'ora.gpnpd' on 'rac1' succeeded

CRS-2793:Shutdown of Oracle High Availability Services-managed resources on 'rac1' has completed

CRS-4133:Oracle High Availability Services has been stopped.

[root@rac2 ~]#/u01/app/11.2.0/grid/bin/crsctl stop crs-f

CRS-2791:Starting shutdown of Oracle High Availability Services-managed resources on 'rac2'

CRS-2673:Attempting to stop 'ora.mdnsd' on 'rac2'

CRS-2673:Attempting to stop 'ora.ctssd' on 'rac2'

CRS-2673:Attempting to stop 'ora.evmd' on 'rac2'

CRS-2673:Attempting to stop 'ora.asm' on 'rac2'

CRS-2677:Stop of 'ora.evmd' on 'rac2' succeeded

CRS-2677:Stop of 'ora.mdnsd' on 'rac2' succeeded

CRS-2677:Stop of 'ora.ctssd' on 'rac2' succeeded

CRS-2677:Stop of 'ora.asm' on 'rac2' succeeded

CRS-2673:Attempting to stop 'ora.cluster_interconnect.haip' on 'rac2'

CRS-2677:Stop of 'ora.cluster_interconnect.haip' on 'rac2' succeeded

CRS-2673:Attempting to stop 'ora.cssd' on 'rac2'

CRS-2677:Stop of 'ora.cssd' on 'rac2' succeeded

CRS-2673:Attempting to stop 'ora.crf' on 'rac2'

CRS-2677:Stop of 'ora.crf' on 'rac2' succeeded

CRS-2673:Attempting to stop 'ora.gipcd' on 'rac2'

CRS-2677:Stop of 'ora.gipcd' on 'rac2' succeeded

CRS-2673:Attempting to stop 'ora.gpnpd' on 'rac2'

CRS-2677:Stop of 'ora.gpnpd' on 'rac2' succeeded

CRS-2793:Shutdown of Oracle High Availability Services-managed resources on 'rac2' has completed

CRS-4133:Oracle High Availability Services has been stopped.

4.在节点一上已独占方式启动(只启动ASM实例但不启动CRS)

[root@rac1 ~]#crsctl start crs-excl-nocrs

CRS-4123:Oracle High Availability Services has been started.

CRS-2672:Attempting to start 'ora.mdnsd' on 'rac1'

CRS-2676:Start of 'ora.mdnsd' on 'rac1' succeeded

CRS-2672:Attempting to start 'ora.gpnpd' on 'rac1'

CRS-2676:Start of 'ora.gpnpd' on 'rac1' succeeded

CRS-2672:Attempting to start 'ora.cssdmonitor' on 'rac1'

CRS-2672:Attempting to start 'ora.gipcd' on 'rac1'

CRS-2676:Start of 'ora.cssdmonitor' on 'rac1' succeeded

CRS-2676:Start of 'ora.gipcd' on 'rac1' succeeded

CRS-2672:Attempting to start 'ora.cssd' on 'rac1'

CRS-2672:Attempting to start 'ora.diskmon' on 'rac1'

CRS-2676:Start of 'ora.diskmon' on 'rac1' succeeded

CRS-2676:Start of 'ora.cssd' on 'rac1' succeeded

CRS-2679:Attempting to clean 'ora.cluster_interconnect.haip' on 'rac1'

CRS-2672:Attempting to start 'ora.ctssd' on 'rac1'

CRS-2681:Clean of 'ora.cluster_interconnect.haip' on 'rac1' succeeded

CRS-2672:Attempting to start 'ora.cluster_interconnect.haip' on 'rac1'

CRS-2676:Start of 'ora.ctssd' on 'rac1' succeeded

CRS-2676:Start of 'ora.cluster_interconnect.haip' on 'rac1' succeeded

CRS-2672:Attempting to start 'ora.asm' on 'rac1'

CRS-2676:Start of 'ora.asm' on 'rac1' succeeded
      

5.重建原votedisk所在的磁盘组

[root@rac1 ~]# su- grid

[grid@rac1 ~]$ sqlplus/as sysasm

SQL*Plus: Release11.2.0.4.0 Productionon Sun Feb2800:09:322016

Copyright (c)1982,2013, Oracle. All rights reserved.

Connectedto:

OracleDatabase 11g Enterprise Edition Release11.2.0.4.0- 64bit Production

With theReal Application Clustersand Automatic Storage Management options

SQL>create diskgroup systemdg normal redundancydisk'/dev/asm-diskb','/dev/asm-diskc','/dev/asm-diskd' ATTRIBUTE'compatible.rdbms'='11.2','compatible.asm'='11.2';

Diskgroup created.

6.使用最新的OCR备份进行恢复

[root@rac1 ~]#ocrconfig-restore/u01/app/11.2.0/grid/cdata/rac-cluster/backup00.ocr

[root@rac1 ~]#ocrcheck

Statusof Oracle Cluster Registryisas follows :

     Version                  :         3

     Totalspace (kbytes)     :    262120

     Usedspace (kbytes)      :      2900

     Availablespace (kbytes) :    259220

     ID                       :1207909490

     Device/File Name         : +SYSTEMDG

                                    Device/File integritycheck succeeded

                                    Device/Filenot configured

                                    Device/Filenot configured

                                    Device/Filenot configured

                                    Device/Filenot configured

     Cluster registry integritycheck succeeded

     Logical corruptioncheck succeeded

7.恢复votedisk

[root@rac1 ~]#crsctl query css votedisk

Located0 votingdisk(s).
[grid@rac1 ~]$ crsctl replace votedisk +SYSTEMDG

Successful additionof votingdisk a0e3b9c0c5934f79bf67aa783618b59d.

Successful additionof votingdisk 422a673bdbc94f97bff51f66f84a105e.

Successful additionof votingdisk a1b110709eb84f39bf351f1b01aa0c7d.

Successfully replaced votingdiskgroupwith+SYSTEMDG.

CRS-4266: Votingfile(s) successfully replaced

[root@rac1 rac-cluster]#crsctl query css votedisk

##  STATE   File Universal Id               File NameDiskgroup

--  -----    -----------------                --------- ---------

 1. ONLINE   03bd9e6851cf4fb4bf4c92e24a3d71cc (/dev/asm-diskb)[SYSTEMDG]

 2. ONLINE   362f2cfb5b3d4f29bf319c668e0efbe4 (/dev/asm-diskc)[SYSTEMDG]

 3. ONLINE   57659c53f4284fbdbfeabafb20c3fbdd (/dev/asm-diskd)[SYSTEMDG]

Notes：如果在恢复时遇到如下错误：

[grid@rac1 ~]$crsctlreplace votedisk +SYSTEMDG

CRS-4602: Failed27toadd votingfile a0e3b9c0c5934f79bf67aa783618b59d.

CRS-4602: Failed27toadd votingfile 422a673bdbc94f97bff51f66f84a105e.

CRS-4602: Failed27toadd votingfile a1b110709eb84f39bf351f1b01aa0c7d.

Failedtoreplace votingdiskgroupwith+SYSTEMDG.

CRS-4000: CommandReplace failed,or completedwith errors.

--登陆grid用户，检查并修改asm_diskstring参数

SQL>alter systemset asm_diskstring='/dev/asm*';

System altered.

--检查并修改spfile参数，并重启ASM

SQL>create spfilefrom memory;

File created.

8.重启crs

[root@rac1 ~]#crsctl stop crs

CRS-2791:Starting shutdown of Oracle High Availability Services-managed resources on 'rac1'

CRS-2673:Attempting to stop 'ora.ctssd' on 'rac1'

CRS-2673:Attempting to stop 'ora.asm' on 'rac1'

CRS-2673:Attempting to stop 'ora.mdnsd' on 'rac1'

CRS-2677:Stop of 'ora.mdnsd' on 'rac1' succeeded

CRS-2677:Stop of 'ora.asm' on 'rac1' succeeded

CRS-2673:Attempting to stop 'ora.cluster_interconnect.haip' on 'rac1'

CRS-2677:Stop of 'ora.cluster_interconnect.haip' on 'rac1' succeeded

CRS-2677:Stop of 'ora.ctssd' on 'rac1' succeeded

CRS-2673:Attempting to stop 'ora.cssd' on 'rac1'

CRS-2677:Stop of 'ora.cssd' on 'rac1' succeeded

CRS-2673:Attempting to stop 'ora.gipcd' on 'rac1'

CRS-2677:Stop of 'ora.gipcd' on 'rac1' succeeded

CRS-2673:Attempting to stop 'ora.gpnpd' on 'rac1'

CRS-2677:Stop of 'ora.gpnpd' on 'rac1' succeeded

CRS-2793:Shutdown of Oracle High Availability Services-managed resources on 'rac1' has completed

CRS-4133:Oracle High Availability Services has been stopped.

[root@rac1 ~]# crsctl start has

CRS-4123: Oracle High Availability Services has been started.

9.由于启动较慢，过几分钟查看状态

[root@rac1 ~]# crs_stat-t

Name           Type           Target    State     Host       

------------------------------------------------------------

ora.DATADG.dg  ora....up.type ONLINE    ONLINE    rac1       

ora....ER.lsnr ora....er.type ONLINE    ONLINE    rac1       

ora....N1.lsnr ora....er.type ONLINE    ONLINE    rac1       

ora....EMDG.dg ora....up.type ONLINE    ONLINE    rac1       

ora.asm        ora.asm.type   ONLINE    ONLINE    rac1       

ora.cvu        ora.cvu.type   ONLINE    ONLINE    rac1       

ora.gsd        ora.gsd.type   OFFLINE   OFFLINE              

ora....network ora....rk.type ONLINE    ONLINE    rac1       

ora.oc4j       ora.oc4j.type  ONLINE    OFFLINE              

ora.ons        ora.ons.type   ONLINE    ONLINE    rac1       

ora.orcl.db    ora....se.type ONLINE    OFFLINE              

ora....SM1.asm application    ONLINE    ONLINE    rac1       

ora....C1.lsnr application    ONLINE    ONLINE    rac1       

ora.rac1.gsd   application    OFFLINE   OFFLINE              

ora.rac1.ons   application    ONLINE    ONLINE    rac1       

ora.rac1.vip   ora....t1.type ONLINE    ONLINE    rac1       

ora.rac2.vip   ora....t1.type ONLINE    ONLINE    rac1       

ora.scan1.vip  ora....ip.type ONLINE    ONLINE    rac1       

[root@rac1 ~]#crsctlcheck crs

CRS-4638: Oracle High Availability Servicesis online

CRS-4537: Cluster Ready Servicesis online

CRS-4529: Cluster Synchronization Servicesis online

CRS-4533: Event Manageris online

10.但是在启动db的时候报错了。。。

PRCR-1079 : Failedto start resource ora.orcl.db

CRS-5017: The resource action "ora.orcl.db start" encountered the following error:

ORA-00205: errorin identifying controlfile,check alertlogfor more info

.For details referto "(:CLSN00107:)"in "/u01/app/11.2.0/grid/log/rac1/agent/crsd/oraagent_oracle/oraagent_oracle.log".

CRS-5017: The resource action "ora.orcl.db start" encountered the following error:

ORA-00205: errorin identifying controlfile,check alertlogfor more info

.For details referto "(:CLSN00107:)"in "/u01/app/11.2.0/grid/log/rac2/agent/crsd/oraagent_oracle/oraagent_oracle.log".

CRS-2674:Start of 'ora.orcl.db' on 'rac1' failed

CRS-2674:Start of 'ora.orcl.db' on 'rac2' failed

CRS-2632:There are no more servers to try to place resource 'ora.orcl.db' on that would satisfy its placement policy

--查看了下trace发现SYSTEMDG中的控制文件没了，装RAC的时候就没规划好，导致控制文件放到了votedisk里，但是SYSTEMDG刚刚被重建了，所以丢了

ALTER SYSTEMSET local_listener=' (ADDRESS=(PROTOCOL=TCP)(HOST=192.168.8.222)(PORT=1521))' SCOPE=MEMORY SID='orcl1';

ALTERDATABASE MOUNT/* db agent *//* {1:40511:558} */

This instance was firstto mount

NOTE: Loaded library: System

SUCCESS: diskgroup DATADG was mounted

SUCCESS: diskgroup SYSTEMDG was mounted

NOTE: dependencybetweendatabase orcland diskgroup resource ora.DATADG.dgis established

ORA-00210: cannotopen the specified controlfile

ORA-00202: controlfile:'+SYSTEMDG/orcl/controlfile/current.261.904952269'

ORA-17503: ksfdopn:2 Failedtoopenfile+SYSTEMDG/orcl/controlfile/current.261.904952269

ORA-15012: ASMfile'+SYSTEMDG/orcl/controlfile/current.261.904952269' doesnot exist

ORA-205 signalled during:ALTERDATABASE MOUNT/* db agent *//* {1:40511:558} */...

Sat Feb2723:48:142016

Shutting down instance (abort)

License high water mark=1

USER (ospid:21150): terminating the instance

Instance terminatedbyUSER, pid=21150

Sat Feb2723:48:142016

Instanceshutdown complete

11.将数据库启动到nomount阶段

[grid@rac1 trace]$srvctl startdatabase-d orcl-o nomount

12.在节点一上使用rman恢复一份控制文件到SYSTEMDG上

[oracle@rac1 ~]$ rman target/

Recovery Manager: Release11.2.0.4.0- Productionon Sat Feb2723:53:352016

Copyright (c)1982,2011, Oracleand/or its affiliates. All rights reserved.

connectedto targetdatabase: ORCL (not mounted)

RMAN>restore controlfileto'+SYSTEMDG'from'+DATADG/orcl/controlfile/current.260.901699963';

Startingrestore at2016/02/2723:53:43

using targetdatabase controlfile insteadof recovery catalog

allocated channel: ORA_DISK_1

channel ORA_DISK_1: SID=34 instance=orcl1 device type=DISK

channel ORA_DISK_1: copied controlfile copy

Finishedrestore at2016/02/2723:53:52

13.已恢复好的控制文件路径和名称

ASMCMD> pwd

+systemdg/orcl/CONTROLFILE

ASMCMD> ls

current.261.904953225

14.节点一上登陆db，修改control_files参数

[oracle@rac1 ~]$ sqlplus/as sysdba

SQL*Plus: Release11.2.0.4.0 Productionon Sat Feb2723:55:092016

Copyright (c)1982,2013, Oracle. All rights reserved.

Connectedto:

OracleDatabase 11g Enterprise Edition Release11.2.0.4.0- 64bit Production

With the Partitioning,Real Application Clusters, Automatic Storage Management, OLAP,

Data MiningandReal Application Testing options

SQL>alter systemset control_files='+DATADG/orcl/controlfile/current.260.901699963','+SYSTEMDG/orcl/CONTROLFILE/current.261.904953225' scope=spfile sid='*';

System altered.

15.重启数据库，并检查数据库状态

[grid@rac1 trace]$srvctl stopdatabase-d orcl

[grid@rac1 trace]$srvctl startdatabase-d orcl

[grid@rac1 trace]$srvctl statusdatabase-d orcl

Instance orcl1is runningon node rac1

Instance orcl2is runningon node rac2

16.保险起见，查看两个节点上的控制文件参数是否正确

[oracle@rac1 ~]$ sqlplus/as sysdba

SQL*Plus: Release11.2.0.4.0 Productionon Sat Feb2723:58:472016

Copyright (c)1982,2013, Oracle. All rights reserved.

Connectedto:

OracleDatabase 11g Enterprise Edition Release11.2.0.4.0- 64bit Production

With the Partitioning,Real Application Clusters, Automatic Storage Management, OLAP,

Data MiningandReal Application Testing options

SQL> show parameter control_file

NAME                                 TYPE          VALUE

------------------------------------ ----------- ------------------------------------------------------------------------------------------------

control_file_record_keep_time        integer      7

control_files                        string        +DATADG/orcl/controlfile/current.260.901699963,+SYSTEMDG/orcl/controlfile/current.261.904953225

[oracle@rac2 ~]$ sqlplus/as sysdba

SQL*Plus: Release11.2.0.4.0 Productionon Sat Feb2723:59:292016

Copyright (c)1982,2013, Oracle. All rights reserved.

Connectedto:

OracleDatabase 11g Enterprise Edition Release11.2.0.4.0- 64bit Production

With the Partitioning,Real Application Clusters, Automatic Storage Management, OLAP,

Data MiningandReal Application Testing options

SQL> show parameter control_file

NAME                                 TYPE           VALUE

------------------------------------ ----------- ------------------------------------------------------------------------------------------------

control_file_record_keep_time        integer      7

control_files                        string        +DATADG/orcl/controlfile/current.260.901699963,+SYSTEMDG/orcl/controlfile/current.261.904953225

OK，收工~~~~