FAILED TO READ MORROR SIDE 1 问题描述： 环境： RAC p740 p550 -bash-3.2$ srvctl status database -d kmis Instance kmis1 is running on node p550 Instance kmis2 is not running on node p740 AIX -bash-3.2$ oslevel -r 6100-08 数据库 oracle 11.2.2
FAILED TO READ MORROR SIDE 1
问题描述：
环境：
RAC p740 p550
-bash-3.2$ srvctl status database -d kmis
Instance kmis1 is running on node p550
Instance kmis2 is not running on node p740
AIX
-bash-3.2$ oslevel -r
6100-08
数据库 oracle 11.2.2
$ srvctl start database -d kmis
PRCR-1079 : Failed to start resource ora.kmis.db
CRS-5017: The resource action "ora.ARCHDG.dg start" encountered the following error:
ORA-15032: not all alterations performed
ORA-15017: diskgroup "ARCHDG" cannot be mounted
ORA-15063: ASM discovered an insufficient number of disks for diskgroup "ARCHDG"
ORA-15080: synchronous I/O operation to a disk failed
CRS-2674: Start of 'ora.ARCHDG.dg' on 'p740' failed
CRS-2632: There are no more servers to try to place resource 'ora.kmis.db' on that would satisfy its placement policy
$
$ srvctl start instance -d kmis -i kmis2 -o open;
PRCR-1013 : Failed to start resource ora.kmis.db
PRCR-1064 : Failed to start resource ora.kmis.db on node p740
CRS-5017: The resource action "ora.ARCHDG.dg start" encountered the following error:
ORA-15032: not all alterations performed
ORA-15017: diskgroup "ARCHDG" cannot be mounted
ORA-15063: ASM discovered an insufficient number of disks for diskgroup "ARCHDG"
ORA-15080: synchronous I/O operation to a disk failed
1
p550
ASMCMD> ls -lState Type Rebal NameMOUNTED EXTERN N ARCHDG/MOUNTED NORMAL N CRS/MOUNTED EXTERN N DATADG/ASMCMD> host
p740上
MOUNTED NORMAL N CRS/MOUNTED EXTERN N DATADG/挂载不上 ARCDG
一挂载就报错
odm：
|
|||||
 |
|||||
转到底部
|
| B - Defect | 12.1 | ||
| 2 - Severe Loss of Service | 11.2.0.1 | ||
| 90 - Closed, Verified by Filer | 212 - IBM AIX on POWER Systems (64-bit) | ||
| 2010-12-22 | |||
| 2014-3-19 | N/A | ||
| 11.2.0.1 | Generic | ||
| Oracle |
|
| Oracle Database Products | Oracle Database Suite | ||
| Oracle Database | 5 - Oracle Database - Enterprise Edition | ||
Hdr: 10422126 11.2.0.1 RDBMS 11.2.0.1 ASM PRODID-5 PORTID-212
Abstract: FAILED TO READ MORROR SIDE 1
*** 12/22/10 03:46 am ***
----
PROBLEM:
--------
Alert log spooling this error continously and filling up available space.
Here is the message.
-----------------------
WARNING: failed to read mirror side 1 of virtual extent 665 logical extent 0
of file 280 in group [1.445758315] from disk ASM_ESBQA001_DATA1_0005
allocation unit 75811 reason error; if possible,will try another mirror side
-----------------------
When the problem starts, errors spin in the alert log until disk space is
exausted. We have found that the process that is refrenced in the trace is
usually an orphaned ebiz form.
Recent Changes - Upgrade to 11.2.0.1
Over the weekend we have upgraded Production Database to 11.2.0.1. We've had
2 occurrence of the issue on 2 of the RAC nodes.
The issue had happened in QA and we have tried to drop the disk and add new
disk again, but it did not solve the problem.
One more thing to note is, when this happens, there was a "run-away" process
that also generates the same error message in a trace file. As long as that
session/process is killed, the error messages stops. Please investigate.
DIAGNOSTIC ANALYSIS:
--------------------
This looks closely related to bug.10051315 and bug.10055063
But customer has applied these patches already, even then we still see the
same warning messages.
We are still trying to get information from customer to understand how that
session is being killed(not sure could be from database end --or application
end).
Customer confirmed the following :
---------------
1.) Are they in-house scripts scheduled in database/os which are going to
kill the inactive sessions at database/os level ?
-- There are not any scripts scheduled to kill sessions. When this error
starts happening, we kill the process manually.
2.) Is this happening only during specific intervals.
-- I cannot see a pattern as to when it is happening. It seems random.
3.) Please confirm - the ASM instance is functional (one can connect to it,
run queries on the various v$asm% views, like: v$asm_diskgroup, v$asm_disk)
-- Yes, ASM is functional during the error. All other sessions continue to
run without errors.
---------------
WORKAROUND:
-----------
As long as that session/process is killed, the error messages stops.
RELATED BUGS:
-------------
bug.10051315 and bug.10055063
REPRODUCIBILITY:
----------------
Its random.
TEST CASE:
----------
NA
STACK TRACE:
------------
Function List (to Full stack) (to Summary stack)
ksedsts <- ksdxfstk <- ksdxcb <- sspuser <- f0
<- sdbgrfucf_close_file <- sdbgrfcf_close_file <- dbgrlWriteTextOnly
<- dbgrlWriteAlertDetail_int <- dbgrlWriteAlertDetail
<- dbgrlWriteAlertText <- dbkrlWriteAlertText <- dbkrlPril <- ksdfmw
<- ksdwrt
<- kfioCompleteIO <- kfioWaitIO <- kfioRequestPriv <- kfioRequest
<- ksfd_kfioRequest
<- ksfd_osmio <- ksfd_io <- ksfdread <- kcfrbd1 <- kcbzib
<- kcbgcur <- kturRollbackToSavepoint <- ktcrsp1 <- ksures <-
ksuxds
<- kssdel <- kssdch <- ksudlp <- opidcl <- opidrv
<- sou2o <- opimai_real <- ssthrdmain <- main
SUPPORTING INFORMATION:
-----------------------
Customer’s Management name: Al Fischer
Customer's Management 24x7 contact number: +1-616-648-6244
Customer’s Management email: al.fischer@amway.com
Primary Customer Email address: eric.lai
Secondary Contact Name: George Zhang
Secondary Contact Email: (EMAIL ADDRESS) george.zhang@alticor.com
24 HOUR CONTACT INFORMATION FOR P1 BUGS:
----------------------------------------
DIAL-IN INFORMATION:
--------------------
IMPACT DATE:
------------
" system at risk of crashing as alert logging is filling up; causing forms to
hang at Call Center"
*** 12/22/10 03:57 am ***
*** 12/22/10 04:20 am ***
*** 12/22/10 04:21 am ***
*** 12/22/10 04:21 am *** (CHG: Sta->16)
*** 12/22/10 04:33 am ***
*** 12/22/10 05:35 am ***
*** 12/22/10 06:08 am ***
*** 12/22/10 06:09 am ***
*** 12/22/10 06:10 am ***
*** 12/22/10 08:06 am ***
*** 12/22/10 11:25 am ***
*** 12/22/10 11:25 am ***
*** 12/22/10 01:47 pm *** (CHG: Sta->10)
*** 12/22/10 01:47 pm ***
*** 12/22/10 07:46 pm ***
*** 12/22/10 07:58 pm ***
File versions form GI & RDBMS homes.
Below is the info from GI_HOME.
[usax02w] $ ar -t -v -X32_64 $GI_HOME/lib/libserver11.a kfd.o
rw-rw-rw- 10000/1026 393236 Nov 15 13:29 2010 kfd.o
[usax02w] $ ar -t -v -X32_64 $GI_HOME/lib/libserver11.a kfdp.o
rw-rw-rw- 10000/1026 271741 Nov 15 13:29 2010 kfdp.o
[usax02w] $ ar -t -v -X32_64 $GI_HOME/lib/libserver11.a kfkio.o
rw-rw-rw- 10000/1026 103575 Nov 15 13:29 2010 kfkio.o
[usax02w] $ ar -t -v -X32_64 $GI_HOME/lib/libserver11.a kfksubs.o
rw-rw-rw- 10000/1026 166215 Nov 15 13:29 2010 kfksubs.o
Below is the info from RDBMS_HOME
[usax02w] $ echo $ORACLE_HOME
/u01/apps/oracle/product/11.2.0/db_1
[usax02w] $ ar -t -v -X32_64 $ORACLE_HOME/lib/libserver11.a kfd.o
rw-rw-rw- 10000/1026 393236 Nov 15 13:29 2010 kfd.o
[usax02w] $ ar -t -v -X32_64 $ORACLE_HOME/lib/libserver11.a kfdp.o
rw-rw-rw- 10000/1026 271741 Nov 15 13:29 2010 kfdp.o
[usax02w] $ ar -t -v -X32_64 $ORACLE_HOME/lib/libserver11.a kfkio.o
rw-rw-rw- 10000/1026 103575 Nov 15 13:29 2010 kfkio.o
[usax02w] $ ar -t -v -X32_64 $ORACLE_HOME/lib/libserver11.a kfksubs.o
rw-rw-rw- 10000/1026 166215 Nov 15 13:29 2010 kfksubs.o
Regards,
Prathap
*** 12/22/10 08:01 pm *** (CHG: Sta->16)
*** 12/22/10 08:25 pm ***
*** 12/23/10 01:48 am ***
*** 12/23/10 02:46 am ***
*** 12/23/10 06:25 am *** (CHG: Sta->11)
*** 12/23/10 06:25 am ***
*** 12/23/10 06:26 am ***
*** 12/23/10 06:26 am ***
*** 12/24/10 01:43 am ***
*** 12/24/10 01:43 am ***
*** 12/27/10 11:33 am ***
*** 12/27/10 11:45 am ***
*** 12/27/10 11:51 am ***
*** 12/27/10 12:22 pm *** ESCALATED
*** 12/27/10 12:22 pm ***
*** 12/27/10 12:29 pm ***
*** 12/28/10 12:42 am *** (CHG: Sta->30)
*** 12/28/10 12:42 am ***
*** 12/28/10 05:00 am *** (CHG: Sta->16)
*** 12/28/10 05:00 am ***
*** 12/28/10 05:08 am *** (CHG: Sta->11)
*** 12/28/10 09:37 am ***
*** 12/29/10 02:51 am ***
*** 12/31/10 03:05 am ***
*** 01/02/11 08:28 pm ***
*** 01/04/11 01:43 am *** (CHG: Sta->30)
*** 01/04/11 01:43 am ***
*** 01/04/11 01:54 am ***
*** 01/04/11 04:50 am *** (CHG: Sta->11)
*** 01/04/11 04:50 am ***
*** 01/05/11 01:26 am *** (CHG: Sta->30)
*** 01/05/11 01:26 am ***
*** 01/10/11 10:46 am ***
*** 01/10/11 10:46 am *** (CHG: Sta->11)
*** 01/11/11 03:14 am ***
*** 01/13/11 11:32 pm ***
RELEASE NOTES:
]] kfk SO is deleted, while there are still allocated sessions.
REDISCOVERY INFORMATION:
The following error, even with fix 10051315:
WARNING: IO Failed. group:1 disk(number.incarnation):5.0xe0d14bb7
disk_path:/dev/ASM_1457_4
AU:75811 disk_offset(bytes):79494299648 io_size:8192 operation:Read
type:synchronous
result:I/O error process_id:2338940
subsys:System iop:0x110939310 bufp:0x70000016f2ec000
osderr:0x434c5344 osderr1:0x0
WORKAROUND:
None
*** 01/13/11 11:44 pm ***
*** 01/13/11 11:44 pm ***
*** 01/13/11 11:44 pm *** (CHG: Sta->80)
*** 01/16/11 11:05 pm ***
*** 01/16/11 11:17 pm ***
*** 01/16/11 11:36 pm ***
*** 01/17/11 12:01 am ***
*** 01/17/11 07:09 am ***
*** 01/17/11 07:11 am ***
*** 01/17/11 09:25 am ***
*** 01/17/11 10:16 am ***
*** 01/17/11 11:14 pm ***
*** 01/18/11 05:06 am ***
*** 01/18/11 11:47 am ***
*** 01/18/11 11:52 pm ***
*** 01/19/11 02:05 am ***
*** 01/19/11 05:39 am ***
*** 01/19/11 05:45 am ***
*** 01/19/11 07:43 am ***
*** 01/19/11 08:44 am ***
*** 01/19/11 06:12 pm ***
*** 01/19/11 06:14 pm ***
*** 01/19/11 10:53 pm ***
*** 01/20/11 12:11 am ***
*** 01/20/11 04:04 am ***
*** 01/20/11 06:11 am ***
*** 01/20/11 06:59 am ***
*** 01/20/11 09:13 am ***
*** 01/20/11 10:44 am ***
*** 01/20/11 10:44 am ***
*** 01/20/11 11:00 am ***
*** 01/20/11 11:38 am ***
*** 01/20/11 04:58 pm ***
*** 01/21/11 04:05 am ***
*** 01/24/11 11:33 am ***
*** 01/25/11 02:17 am ***
*** 01/27/11 02:58 am ***
*** 01/27/11 03:16 am ***
*** 01/27/11 03:37 am ***
*** 01/27/11 03:49 am ***
*** 02/02/11 04:37 am ***
*** 02/04/11 08:25 pm ***
*** 02/05/11 02:46 am ***
*** 02/05/11 02:58 am ***
*** 02/05/11 02:59 am ***
*** 02/05/11 02:29 pm ***
*** 02/05/11 07:19 pm ***
*** 02/05/11 09:37 pm ***
*** 02/07/11 01:37 am ***
*** 02/07/11 06:14 am ***
*** 02/07/11 12:35 pm ***
*** 02/07/11 06:46 pm ***
*** 02/07/11 08:33 pm ***
*** 02/08/11 01:52 pm ***
*** 02/08/11 02:42 pm ***
*** 02/08/11 10:32 pm ***
*** 02/09/11 11:51 pm ***
*** 02/10/11 08:56 am ***
*** 02/10/11 08:59 am ***
*** 02/10/11 09:30 am ***
*** 02/11/11 11:52 pm ***
*** 02/12/11 05:10 pm ***
*** 02/15/11 02:55 am ***
*** 02/15/11 10:13 am ***
*** 02/16/11 01:07 am ***
*** 02/16/11 06:34 am ***
*** 02/18/11 01:39 am ***
*** 02/18/11 01:40 am ***
*** 02/24/11 07:20 am ***
*** 02/24/11 07:35 am ***
*** 02/25/11 12:34 am ***
*** 03/02/11 07:59 am ***
*** 03/02/11 08:25 am ***
*** 03/03/11 01:22 am ***
*** 03/03/11 03:34 pm ***
*** 03/07/11 11:54 am ***
*** 03/08/11 08:53 pm ***
*** 03/09/11 10:44 pm ***
*** 04/20/11 01:17 pm ***
*** 04/21/11 09:34 am ***
*** 04/21/11 01:46 pm ***
*** 04/22/11 03:47 pm ***
*** 04/29/11 12:33 pm ***
*** 04/29/11 01:21 pm ***
*** 05/02/11 07:53 am ***
*** 05/03/11 01:12 am ***
*** 05/03/11 01:37 am ***
*** 05/03/11 02:11 am ***
*** 05/11/11 03:00 am ***
*** 05/11/11 03:44 am ***
*** 05/16/11 08:07 am ***
*** 05/17/11 11:31 am ***
*** 05/17/11 11:34 am ***
*** 05/17/11 03:14 pm ***
*** 05/17/11 03:16 pm ***
*** 05/18/11 06:00 am ***
*** 05/18/11 06:11 am ***
*** 05/18/11 08:46 am ***
*** 06/09/11 12:44 pm ***
*** 06/10/11 10:21 am ***
*** 06/10/11 10:21 am ***
*** 06/11/11 04:44 am ***
*** 06/14/11 10:14 pm ***
*** 06/17/11 12:33 pm ***
*** 07/04/11 01:09 pm ***
*** 07/06/11 04:03 pm ***
*** 07/06/11 04:56 pm ***
*** 07/08/11 11:47 am ***
*** 07/08/11 12:16 pm ***
*** 07/12/11 07:28 am ***
*** 07/14/11 07:58 pm ***
*** 07/17/11 05:29 am ***
*** 07/17/11 05:30 am ***
*** 07/18/11 03:23 pm ***
*** 07/26/11 03:20 pm ***
*** 07/26/11 03:23 pm ***
*** 07/27/11 03:13 am ***
*** 07/27/11 03:15 am ***
*** 07/27/11 07:33 am ***
*** 08/02/11 01:34 pm ***
*** 08/03/11 01:03 am ***
*** 08/03/11 01:03 am ***
*** 08/03/11 01:15 am ***
*** 08/03/11 01:15 am ***
*** 08/05/11 08:00 am ***
*** 08/05/11 08:11 am ***
*** 08/05/11 10:42 am ***
*** 08/11/11 01:22 am ***
*** 08/12/11 04:00 am ***
*** 08/12/11 10:03 am ***
*** 08/18/11 12:46 pm ***
*** 08/22/11 12:14 pm ***
*** 09/07/11 04:49 am ***
*** 09/09/11 02:55 pm ***
*** 09/14/11 02:40 pm ***
*** 10/14/11 02:15 am ***
*** 10/14/11 02:15 am *** ESCALATION -> CLOSED
*** 10/14/11 02:15 am *** (CHG: Sta->90)
*** 10/14/11 02:21 am ***
*** 10/14/11 02:21 am ***
*** 10/14/11 09:56 am ***
*** 11/01/11 01:55 am ***
*** 11/01/11 10:49 am ***
*** 11/01/11 10:49 am ***
*** 11/27/11 01:15 am ***
*** 12/01/11 09:11 am ***
*** 01/17/12 03:29 pm ***
*** 01/23/12 08:26 pm ***
*** 01/23/12 08:26 pm ***
*** 03/14/12 01:07 am ***
*** 03/18/12 11:22 pm ***
*** 03/22/12 12:44 am ***
*** 04/02/12 06:10 am ***
*** 04/24/12 10:45 am ***
*** 01/26/13 08:03 am ***
*** 01/31/13 07:35 am ***
*** 02/24/13 11:27 pm ***
*** 04/16/13 02:14 am ***
*** 04/19/13 07:23 am ***
*** 05/12/13 09:12 pm ***
*** 05/20/13 06:28 am ***
*** 06/25/13 01:16 am ***
*** 07/08/13 09:41 pm ***
*** 07/09/13 09:44 pm ***
*** 07/09/13 10:24 pm ***
*** 07/17/13 03:02 am ***
*** 07/18/13 02:35 am ***
*** 08/12/13 03:54 am ***
*** 08/20/13 12:21 am ***
*** 09/30/13 12:26 pm ***
*** 10/23/13 05:08 pm ***
*** 10/29/13 08:22 am ***
*** 10/29/13 09:56 am ***
*** 10/29/13 09:56 am ***
*** 12/18/13 03:38 am ***
*** 01/23/14 09:24 pm ***
*** 02/11/14 09:58 am ***
*** 02/24/14 03:11 am ***
*** 02/24/14 06:02 am ***
*** 02/24/14 12:34 pm ***
*** 02/24/14 03:58 pm ***
*** 02/25/14 02:28 am ***
*** 03/18/14 04:34 am ***
*** 03/18/14 04:55 am ***
*** 03/18/14 05:09 am ***
*** 03/18/14 10:37 pm ***
*** 03/19/14 12:39 am ***
