ORA-00600: [kqrpre_in_apply]查看

1.Environment
19.8的4节点rac+1个2节点dg+1个一节点的dg

2.Symptoms 3.Changes
在主库上创建一个pdb,然后发现名字不太规范，删除后，重新创建pdb。此时备库同步进程中断，报ORA-00600.

4.Cause
Bug 31686979 - PDB Creation is failing with ORA-00600: [kqrpre_in_apply] in DATAGUARD environment (Doc ID 31686979.8)

5.Solution
在ADG中会触发，在dg中不会触发。使用dg进行同步，然后再开启ADG。
后模拟，发现在主库创建pdb后，从库需要手动开启或者关闭数据库，其他的create和drop database是可以同步的。

6.References

相关日志

Recovery of Online Redo Log: Thread 2 Group 206 Seq 15804 Reading mem 0
Mem# 0: +DATA/DGORCL/ONLINELOG/group_206.566.1076721439
Mem# 1: +DATA/DGORCL/ONLINELOG/group_206.567.1076721441
2021-10-26T16:15:52.395890+08:00
Recovery created pluggable database TEST_PDB
*****************************************
WARNING: The converted filename '+DATA/dgorcl/cf3e15dc04be0c4be0531542870a8db4/datafile/system.170736.1086970551'
is an ASM fully qualified filename.
Changing the filename to '+DATA/MUST_RENAME_THIS_DATAFILE_0.4294967295.4294967295'.
Please rename it accordingly.
*****************************************
Recovery copied files for tablespace SYSTEM
Recovery successfully copied file +DATA/DGORCL/CF3E15DC04BE0C4BE0531542870A8DB4/DATAFILE/system.13501.1086970553 from +DATA/DGORCL/BA413D9EEDF14675E0531542870A0EDA/DATAFILE/system.266.1076624545
TEST_PDB(5):*****************************************
TEST_PDB(5):WARNING: The converted filename '+DATA/dgorcl/cf3e15dc04be0c4be0531542870a8db4/datafile/system.170736.1086970551'
TEST_PDB(5): is an ASM fully qualified filename.
TEST_PDB(5): Changing the filename to '+DATA/MUST_RENAME_THIS_DATAFILE_25548.4294967295.4294967295'.
TEST_PDB(5): Please rename it accordingly.
TEST_PDB(5):*****************************************
2021-10-26T16:15:53.562557+08:00
TEST_PDB(5):Successfully added datafile 499 to media recovery
TEST_PDB(5):Datafile #499: '+DATA/DGORCL/CF3E15DC04BE0C4BE0531542870A8DB4/DATAFILE/system.13501.1086970553'
*****************************************
WARNING: The converted filename '+DATA/dgorcl/cf3e15dc04be0c4be0531542870a8db4/datafile/sysaux.170733.1086970551'
is an ASM fully qualified filename.
Changing the filename to '+DATA/MUST_RENAME_THIS_DATAFILE_0.4294967295.4294967295'.
Please rename it accordingly.
*****************************************
Recovery copied files for tablespace SYSAUX
Recovery successfully copied file +DATA/DGORCL/CF3E15DC04BE0C4BE0531542870A8DB4/DATAFILE/sysaux.13502.1086970553 from +DATA/DGORCL/BA413D9EEDF14675E0531542870A0EDA/DATAFILE/sysaux.267.1076624545
TEST_PDB(5):*****************************************
TEST_PDB(5):WARNING: The converted filename '+DATA/dgorcl/cf3e15dc04be0c4be0531542870a8db4/datafile/sysaux.170733.1086970551'
TEST_PDB(5): is an ASM fully qualified filename.
TEST_PDB(5): Changing the filename to '+DATA/MUST_RENAME_THIS_DATAFILE_25548.4294967295.4294967295'.
TEST_PDB(5): Please rename it accordingly.
TEST_PDB(5):*****************************************
TEST_PDB(5):Successfully added datafile 500 to media recovery
TEST_PDB(5):Datafile #500: '+DATA/DGORCL/CF3E15DC04BE0C4BE0531542870A8DB4/DATAFILE/sysaux.13502.1086970553'
*****************************************
WARNING: The converted filename '+DATA/dgorcl/cf3e15dc04be0c4be0531542870a8db4/datafile/undotbs1.170738.1086970551'
is an ASM fully qualified filename.
Changing the filename to '+DATA/MUST_RENAME_THIS_DATAFILE_0.4294967295.4294967295'.
Please rename it accordingly.
*****************************************
2021-10-26T16:15:54.725268+08:00
Recovery copied files for tablespace UNDOTBS1
Recovery successfully copied file +DATA/DGORCL/CF3E15DC04BE0C4BE0531542870A8DB4/DATAFILE/undotbs1.13503.1086970555 from +DATA/DGORCL/BA413D9EEDF14675E0531542870A0EDA/DATAFILE/undotbs1.269.1076624547
TEST_PDB(5):*****************************************
TEST_PDB(5):WARNING: The converted filename '+DATA/dgorcl/cf3e15dc04be0c4be0531542870a8db4/datafile/undotbs1.170738.1086970551'
TEST_PDB(5): is an ASM fully qualified filename.
TEST_PDB(5): Changing the filename to '+DATA/MUST_RENAME_THIS_DATAFILE_25548.4294967295.4294967295'.
TEST_PDB(5): Please rename it accordingly.
TEST_PDB(5):*****************************************
TEST_PDB(5):Successfully added datafile 501 to media recovery
TEST_PDB(5):Datafile #501: '+DATA/DGORCL/CF3E15DC04BE0C4BE0531542870A8DB4/DATAFILE/undotbs1.13503.1086970555'
(5):Resize operation completed for file# 501, old size 194560K, new size 199680K
(5):Resize operation completed for file# 501, old size 199680K, new size 204800K
2021-10-26T16:15:55.949527+08:00
(5):Resize operation completed for file# 501, old size 204800K, new size 209920K
(5):Resize operation completed for file# 501, old size 209920K, new size 215040K
(5):Resize operation completed for file# 501, old size 215040K, new size 220160K
(5):Resize operation completed for file# 501, old size 220160K, new size 225280K
(5):Resize operation completed for file# 501, old size 225280K, new size 230400K
2021-10-26T16:15:57.041371+08:00
(5):Resize operation completed for file# 501, old size 230400K, new size 235520K
(5):Resize operation completed for file# 501, old size 235520K, new size 240640K
(5):Resize operation completed for file# 501, old size 240640K, new size 245760K
(5):Resize operation completed for file# 501, old size 245760K, new size 250880K
(5):Resize operation completed for file# 501, old size 250880K, new size 256000K
2021-10-26T16:15:58.121180+08:00
(5):Resize operation completed for file# 501, old size 256000K, new size 261120K
(5):Resize operation completed for file# 501, old size 261120K, new size 266240K
(5):Resize operation completed for file# 501, old size 266240K, new size 271360K
(5):Resize operation completed for file# 501, old size 271360K, new size 276480K
(5):Resize operation completed for file# 501, old size 276480K, new size 281600K
2021-10-26T16:15:59.205646+08:00
(5):Resize operation completed for file# 501, old size 281600K, new size 286720K
(5):Resize operation completed for file# 501, old size 286720K, new size 291840K
(5):Resize operation completed for file# 501, old size 291840K, new size 296960K
(5):Resize operation completed for file# 501, old size 296960K, new size 302080K
(5):Resize operation completed for file# 501, old size 302080K, new size 307200K
2021-10-26T16:16:00.244182+08:00
(5):Resize operation completed for file# 501, old size 307200K, new size 312320K
(5):Resize operation completed for file# 501, old size 312320K, new size 317440K
(5):Resize operation completed for file# 501, old size 317440K, new size 322560K
(5):Resize operation completed for file# 501, old size 322560K, new size 327680K
(5):Resize operation completed for file# 501, old size 327680K, new size 332800K
(5):Resize operation completed for file# 501, old size 332800K, new size 337920K
2021-10-26T16:16:01.268933+08:00
(5):Resize operation completed for file# 501, old size 337920K, new size 343040K
(5):Resize operation completed for file# 501, old size 343040K, new size 348160K
(5):Resize operation completed for file# 501, old size 348160K, new size 353280K
(5):Resize operation completed for file# 501, old size 353280K, new size 358400K
(5):Resize operation completed for file# 501, old size 358400K, new size 363520K
2021-10-26T16:16:02.354244+08:00
(5):Resize operation completed for file# 501, old size 363520K, new size 368640K
(5):Resize operation completed for file# 501, old size 368640K, new size 373760K
(5):Resize operation completed for file# 501, old size 373760K, new size 378880K
(5):Resize operation completed for file# 501, old size 378880K, new size 384000K
(5):Resize operation completed for file# 501, old size 384000K, new size 389120K
(5):Resize operation completed for file# 501, old size 389120K, new size 394240K
2021-10-26T16:16:03.456418+08:00
(5):Resize operation completed for file# 501, old size 394240K, new size 399360K
(5):Resize operation completed for file# 501, old size 399360K, new size 404480K
--More--(69%)
(5):Resize operation completed for file# 501, old size 445440K, new size 450560K
2021-10-26T16:27:29.746519+08:00
(5):Successfully added datafile 502 to media recovery
(5):Datafile #502: '+DATA/DGORCL/CF3E15DC04BE0C4BE0531542870A8DB4/DATAFILE/cbs_xxl_job_tbs.13504.1086971095'
(5):Resize operation completed for file# 500, old size 389120K, new size 399360K
2021-10-26T16:28:49.664215+08:00
TEST_PDB(5):Recovery deleting file #502:'+DATA/DGORCL/CF3E15DC04BE0C4BE0531542870A8DB4/DATAFILE/cbs_xxl_job_tbs.13504.1086971095' from controlfile.
2021-10-26T16:28:52.393452+08:00
TEST_PDB(5):Deleted Oracle managed file +DATA/DGORCL/CF3E15DC04BE0C4BE0531542870A8DB4/DATAFILE/cbs_xxl_job_tbs.13504.1086971095
TEST_PDB(5):Recovery dropped tablespace 'TEST_PDB_TBS'
TEST_PDB(5):Recovery dropped temporary tablespace 'TEMP'
TEST_PDB(5):Recovery deleting file #501:'+DATA/DGORCL/CF3E15DC04BE0C4BE0531542870A8DB4/DATAFILE/undotbs1.13503.1086970555' from controlfile.
TEST_PDB(5):Deleted Oracle managed file +DATA/DGORCL/CF3E15DC04BE0C4BE0531542870A8DB4/DATAFILE/undotbs1.13503.1086970555
TEST_PDB(5):Recovery dropped tablespace 'UNDOTBS1'
2021-10-26T16:28:53.750900+08:00
TEST_PDB(5):Recovery deleting file #500:'+DATA/DGORCL/CF3E15DC04BE0C4BE0531542870A8DB4/DATAFILE/sysaux.13502.1086970553' from controlfile.
TEST_PDB(5):Deleted Oracle managed file +DATA/DGORCL/CF3E15DC04BE0C4BE0531542870A8DB4/DATAFILE/sysaux.13502.1086970553
TEST_PDB(5):Recovery dropped tablespace 'SYSAUX'
TEST_PDB(5):Recovery deleting file #499:'+DATA/DGORCL/CF3E15DC04BE0C4BE0531542870A8DB4/DATAFILE/system.13501.1086970553' from controlfile.
TEST_PDB(5):Deleted Oracle managed file +DATA/DGORCL/CF3E15DC04BE0C4BE0531542870A8DB4/DATAFILE/system.13501.1086970553
TEST_PDB(5):Recovery dropped tablespace 'SYSTEM'
TEST_PDB(5):Recovery dropped pluggable database 'TEST_PDB'
Errors in file /picclife/app/oracle/diag/rdbms/dgorcl/dgorcl1/trace/dgorcl1_pr00_81289.trc (incident=1475473) (PDBNAME=CDB$ROOT):
ORA-00600: internal error code, arguments: [kqrpre_in_apply], [], [], [], [], [], [], [], [], [], [], []
Incident details in: /picclife/app/oracle/diag/rdbms/dgorcl/dgorcl1/incident/incdir_1475473/dgorcl1_pr00_81289_i1475473.trc
2021-10-26T16:28:57.098193+08:00
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
2021-10-26T16:28:57.099258+08:00
PR00 (PID:81289): MRP0: Background Media Recovery terminated with error 600
2021-10-26T16:28:57.100014+08:00
Errors in file /picclife/app/oracle/diag/rdbms/dgorcl/dgorcl1/trace/dgorcl1_pr00_81289.trc:
ORA-00600: internal error code, arguments: [kqrpre_in_apply], [], [], [], [], [], [], [], [], [], [], []
2021-10-26T16:28:57.115101+08:00
.... (PID:69088): Managed Standby Recovery not using Real Time Apply
2021-10-26T16:28:57.265916+08:00
--More--(70%)


Completed: alter database recover managed standby database using current logfile disconnect
2021-10-27T08:56:18.350370+08:00
PR00 (PID:58039): Media Recovery Log +DATA/DGORCL/ARCHIVELOG/2021_10_26/thread_3_seq_15153.13502.1086972013
2021-10-27T08:56:18.614220+08:00
PR00 (PID:58039): Media Recovery Log +DATA/DGORCL/ARCHIVELOG/2021_10_26/thread_4_seq_15022.13504.1086972013
2021-10-27T08:56:20.322207+08:00
Recovery created pluggable database TESTPDB
*****************************************
WARNING: The converted filename '+DATA/dgorcl/cf3e444341896596e0531542870a98b0/datafile/system.170730.1086971395'
is an ASM fully qualified filename.
Changing the filename to '+DATA/MUST_RENAME_THIS_DATAFILE_0.4294967295.4294967295'.
Please rename it accordingly.
*****************************************
Recovery copied files for tablespace SYSTEM
Recovery successfully copied file +DATA/DGORCL/CF3E444341896596E0531542870A98B0/DATAFILE/system.13642.1087030581 from +DATA/DGORCL/BA413D9EEDF14675E0531542870A0EDA/DATAFILE/system.266.1076624545
TESTPDB(6):*****************************************
TESTPDB(6):WARNING: The converted filename '+DATA/dgorcl/cf3e444341896596e0531542870a98b0/datafile/system.170730.1086971395'
TESTPDB(6): is an ASM fully qualified filename.
TESTPDB(6): Changing the filename to '+DATA/MUST_RENAME_THIS_DATAFILE_44860.4294967295.4294967295'.
TESTPDB(6): Please rename it accordingly.
TESTPDB(6):*****************************************
TESTPDB(6):Successfully added datafile 503 to media recovery
TESTPDB(6):Datafile #503: '+DATA/DGORCL/CF3E444341896596E0531542870A98B0/DATAFILE/system.13642.1087030581'
*****************************************
WARNING: The converted filename '+DATA/dgorcl/cf3e444341896596e0531542870a98b0/datafile/sysaux.170708.1086971395'
is an ASM fully qualified filename.
Changing the filename to '+DATA/MUST_RENAME_THIS_DATAFILE_0.4294967295.4294967295'.
Please rename it accordingly.
*****************************************
2021-10-27T08:56:21.786499+08:00
Recovery copied files for tablespace SYSAUX
Recovery successfully copied file +DATA/DGORCL/CF3E444341896596E0531542870A98B0/DATAFILE/sysaux.13643.1087030581 from +DATA/DGORCL/BA413D9EEDF14675E0531542870A0EDA/DATAFILE/sysaux.267.1076624545
TESTPDB(6):*****************************************
TESTPDB(6):WARNING: The converted filename '+DATA/dgorcl/cf3e444341896596e0531542870a98b0/datafile/sysaux.170708.1086971395'
TESTPDB(6): is an ASM fully qualified filename.
TESTPDB(6): Changing the filename to '+DATA/MUST_RENAME_THIS_DATAFILE_44860.4294967295.4294967295'.
TESTPDB(6): Please rename it accordingly.
TESTPDB(6):*****************************************
TESTPDB(6):Successfully added datafile 504 to media recovery
TESTPDB(6):Datafile #504: '+DATA/DGORCL/CF3E444341896596E0531542870A98B0/DATAFILE/sysaux.13643.1087030581'
*****************************************
WARNING: The converted filename '+DATA/dgorcl/cf3e444341896596e0531542870a98b0/datafile/undotbs1.170738.1086971395'
is an ASM fully qualified filename.
Changing the filename to '+DATA/MUST_RENAME_THIS_DATAFILE_0.4294967295.4294967295'.
Please rename it accordingly.
*****************************************
Recovery copied files for tablespace UNDOTBS1
Recovery successfully copied file +DATA/DGORCL/CF3E444341896596E0531542870A98B0/DATAFILE/undotbs1.13644.1087030583 from +DATA/DGORCL/BA413D9EEDF14675E0531542870A0EDA/DATAFILE/undotbs1.269.1076624547
TESTPDB(6):*****************************************
TESTPDB(6):WARNING: The converted filename '+DATA/dgorcl/cf3e444341896596e0531542870a98b0/datafile/undotbs1.170738.1086971395'
TESTPDB(6): is an ASM fully qualified filename.
TESTPDB(6): Changing the filename to '+DATA/MUST_RENAME_THIS_DATAFILE_44860.4294967295.4294967295'.
TESTPDB(6): Please rename it accordingly.
TESTPDB(6):*****************************************
2021-10-27T08:56:22.789421+08:00
TESTPDB(6):Successfully added datafile 505 to media recovery
TESTPDB(6):Datafile #505: '+DATA/DGORCL/CF3E444341896596E0531542870A98B0/DATAFILE/undotbs1.13644.1087030583'
(6):Resize operation completed for file# 505, old size 194560K, new size 199680K
(6):Resize operation completed for file# 505, old size 199680K, new size 204800K
(6):Resize operation completed for file# 505, old size 204800K, new size 209920K
(6):Resize operation completed for file# 505, old size 209920K, new size 215040K
(6):Resize operation completed for file# 505, old size 215040K, new size 220160K
(6):Resize operation completed for file# 505, old size 220160K, new size 225280K
2021-10-27T08:56:23.905568+08:00
(6):Resize operation completed for file# 505, old size 225280K, new size 230400K
(6):Resize operation completed for file# 505, old size 230400K, new size 235520K
(6):Resize operation completed for file# 505, old size 235520K, new size 240640K
(6):Resize operation completed for file# 505, old size 240640K, new size 245760K
(6):Resize operation completed for file# 505, old size 245760K, new size 250880K
(6):Resize operation completed for file# 505, old size 250880K, new size 256000K
(6):Resize operation completed for file# 505, old size 256000K, new size 261120K
(6):Resize operation completed for file# 505, old size 261120K, new size 266240K
2021-10-27T08:56:24.907013+08:00
(6):Resize operation completed for file# 505, old size 266240K, new size 271360K
(6):Resize operation completed for file# 505, old size 271360K, new size 276480K
(6):Resize operation completed for file# 505, old size 276480K, new size 281600K
(6):Resize operation completed for file# 505, old size 281600K, new size 286720K
(6):Resize operation completed for file# 505, old size 286720K, new size 291840K
(6):Resize operation completed for file# 505, old size 291840K, new size 296960K
(6):Resize operation completed for file# 505, old size 296960K, new size 302080K
(6):Resize operation completed for file# 505, old size 302080K, new size 307200K
2021-10-27T08:56:26.023176+08:00
(6):Resize operation completed for file# 505, old size 307200K, new size 312320K
(6):Resize operation completed for file# 505, old size 312320K, new size 317440K
(6):Resize operation completed for file# 505, old size 317440K, new size 322560K
(6):Resize operation completed for file# 505, old size 322560K, new size 327680K
(6):Resize operation completed for file# 505, old size 327680K, new size 332800K
(6):Resize operation completed for file# 505, old size 332800K, new size 337920K
(6):Resize operation completed for file# 505, old size 337920K, new size 343040K
(6):Resize operation completed for file# 505, old size 343040K, new size 348160K
2021-10-27T08:56:27.024099+08:00
(6):Resize operation completed for file# 505, old size 348160K, new size 353280K
(6):Resize operation completed for file# 505, old size 353280K, new size 358400K
(6):Resize operation completed for file# 505, old size 358400K, new size 363520K
(6):Resize operation completed for file# 505, old size 363520K, new size 368640K
(6):Resize operation completed for file# 505, old size 368640K, new size 373760K
(6):Resize operation completed for file# 505, old size 373760K, new size 378880K
(6):Resize operation completed for file# 505, old size 378880K, new size 384000K
(6):Resize operation completed for file# 505, old size 384000K, new size 389120K
2021-10-27T08:56:28.057865+08:00
(6):Resize operation completed for file# 505, old size 389120K, new size 394240K
(6):Resize operation completed for file# 505, old size 394240K, new size 399360K
(6):Resize operation completed for file# 505, old size 399360K, new size 404480K
(6):Resize operation completed for file# 505, old size 404480K, new size 409600K
(6):Resize operation completed for file# 505, old size 409600K, new size 414720K
(6):Resize operation completed for file# 505, old size 414720K, new size 419840K
(6):Resize operation completed for file# 505, old size 419840K, new size 424960K
(6):Resize operation completed for file# 505, old size 424960K, new size 430080K
2021-10-27T08:56:29.182631+08:00
(6):Resize operation completed for file# 505, old size 430080K, new size 435200K
(6):Resize operation completed for file# 505, old size 435200K, new size 440320K
(6):Resize operation completed for file# 505, old size 440320K, new size 445440K
(6):Resize operation completed for file# 505, old size 445440K, new size 450560K
2021-10-27T08:56:55.891278+08:00
Decreasing number of high priority LMS from 2 to 0
2021-10-27T08:59:15.914474+08:00
(6):Successfully added datafile 506 to media recovery
(6):Datafile #506: '+DATA/DGORCL/CF3E444341896596E0531542870A98B0/DATAFILE/cbsxxljob_tbs.13645.1087030593'
2021-10-27T08:59:17.933275+08:00
PR00 (PID:58039): Media Recovery Log +DATA/DGORCL/ARCHIVELOG/2021_10_26/thread_1_seq_23855.13505.1086973367
(6):Resize operation completed for file# 504, old size 389120K, new size 399360K
2021-10-27T08:59:20.300438+08:00
PR00 (PID:58039): Media Recovery Log +DATA/DGORCL/ARCHIVELOG/2021_10_26/thread_3_seq_15154.13507.1086973813
2021-10-27T08:59:20.531585+08:00
PR00 (PID:58039): Media Recovery Log +DATA/DGORCL/ARCHIVELOG/2021_10_26/thread_2_seq_15805.13508.1086973815
2021-10-27T08:59:20.761083+08:00
PR00 (PID:58039): Media Recovery Log +DATA/DGORCL/ARCHIVELOG/2021_10_26/thread_4_seq_15023.13506.1086973811
2021-10-27T08:59:22.148860+08:00
(6):Successfully added datafile 507 to media recovery
(6):Datafile #507: '+DATA/DGORCL/CF3E444341896596E0531542870A98B0/DATAFILE/undo_4.13646.1087030761'
2021-10-27T08:59:24.463618+08:00
(6):Successfully added datafile 508 to media recovery
(6):Datafile #508: '+DATA/DGORCL/CF3E444341896596E0531542870A98B0/DATAFILE/undo_5.13647.1087030763'
2021-10-27T08:59:26.646721+08:00
(6):Successfully added datafile 509 to media recovery
(6):Datafile #509: '+DATA/DGORCL/CF3E444341896596E0531542870A98B0/DATAFILE/undo_6.13648.1087030765'
2021-10-27T08:59:34.541827+08:00
PR00 (PID:58039): Media Recovery Log +DATA/DGORCL/ARCHIVELOG/2021_10_26/thread_1_seq_23856.13509.1086975167
2021-10-27T08:59:35.344467+08:00
PR00 (PID:58039): Media Recovery Log +DATA/DGORCL/ARCHIVELOG/2021_10_26/thread_4_seq_15024.13511.1086975613
2021-10-27T08:59:35.464424+08:00
PR00 (PID:58039): Media Recovery Log +DATA/DGORCL/ARCHIVELOG/2021_10_26/thread_3_seq_15155.13510.1086975613