19c rac adg其中一节点重启后crsd服务无法启动

系统环境：centos7.6

oracle版本：19c rac adg

adg其中一个节点重启后，使用 crsctl check cluster检查，服务异常。

使用crsctl status res -t -init检查发现crsd服务处于offine状态。

查看启动日志$ORACLE_BASE/diag/crs/rac01/crs/trace/ohasd_orarootagent_root.trc

有报错

Cluster Ready Service aborted due to Oracle Cluster Registry error [PROC-26: Error while accessing the physical storage Storage layer error [Insufficient quorum to open OCR devices] [0]]. Details at (:CRSD00111:) in /**/grid_base/diag/crs/wwdbdg01/crs/trace/crsd.trc.

查看crsd.trc日志文件，

2022-03-17 17:30:38.819*:kgfn.c@1784: kgfnRecordErrPriv: 1017 error=ORA-01017: invalid username/password; logon denied

2022-03-17 17:30:38.821 : default:1682702656: clsCredDomClose: Credctx deleted 0x56348f3224c0
2022-03-17 17:30:38.821 :  OCRRAW:1682702656: kgfnConnect2: failed to connect

2022-03-17 17:30:38.821*:kgfn.c@5461: kgfnConnect2: failed to connect
2022-03-17 17:30:38.821 :  OCRRAW:1682702656: kgfnConnect2Retry: failed to connect connect after 1 attempts, 136s elapsed

2022-03-17 17:30:38.821 :  OCRRAW:1682702656: kgfo_kge2slos error stack at kgfoAl06: ORA-01017: invalid username/password; logon denied


2022-03-17 17:30:38.821*:kgfo.c@1003: kgfo_kge2slos error stack at kgfoAl06: ORA-01017: invalid username/password; logon denied

2022-03-17 17:30:38.821 :  OCRRAW:1682702656: -- trace dump on error exit --

2022-03-17 17:30:38.821 :  OCRRAW:1682702656: Error [kgfoAl06] in [kgfokge] at kgfo.c:2165

2022-03-17 17:30:38.821 :  OCRRAW:1682702656: ORA-01017: invalid username/password; logon denied


2022-03-17 17:30:38.821 :  OCRRAW:1682702656: Category: 7

2022-03-17 17:30:38.821 :  OCRRAW:1682702656: DepInfo: 1017

2022-03-17 17:30:38.822 :  OCRRAW:1682702656: -- trace dump end --

2022-03-17 17:30:38.822 :  OCRASM:1682702656: proprasmo: Failed to open the file in DG [SYSTEMDG]
2022-03-17 17:30:38.822 :  OCRASM:1682702656: proprasmo: Error in open/create file in dg [SYSTEMDG]
  OCRASM:1682702656: SLOS : SLOS: cat=7, opn=kgfoAl06, dep=1017, loc=kgfokge

2022-03-17 17:30:38.822 :  OCRASM:1682702656: ASM Error Stack : ORA-01017: invalid username/password; logon denied

 default:1682702656: u_set_gbl_comp_error: comptype '108' : error '7'
2022-03-17 17:30:38.849 :  OCRRAW:1682702656: kgfnFindLocalNode03: kgfn_find_node_sid found no members

　　权限验证失败。通过手动启动asm实例，sqlplus / as sysasm      

然后再手动启动crsd   crsctl start crsd -init

参考文档&感谢：http://www.askmac.cn/archives/flex-asm-gi-failed-to-start.html

https://www.cnblogs.com/hxlasky/p/15049513.html