前两天现场报流程无法下发,先看日志报错,是多线程造成的死锁:
<2016-1-27 下午03时08分59秒 GMT+08:00> <Error> <WebLogicServer> <BEA-000337> <[STUCK] ExecuteThread: ‘1’ for queue: ‘weblogic.kernel.Default (self-tuning)’ has been busy for “606” seconds working on the request “weblogic.servlet.internal.ServletRequestImpl@77a4a3a4[
POST /web/aaa/bbb/plan/common/bpms/pmsBpmsOperateAction.do HTTP/1.1
Connection: keep-alive
Content-Length: 7620
Accept: application/json, text/javascript, */*; q=0.01
Origin: http://10.150.22.1
X-Requested-With: XMLHttpRequest
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/38.0.2125.122 Safari/537.36 SE 2.X MetaSr 1.0
Content-Type: application/x-www-form-urlencoded; charset=UTF-8
Referer: http://10.150.22.1/web/aaa/bbb/plan/adjustplan/adjustplanaudit/audit/AdjustPlanSumAuditTodoList.jsp?listType=3
Accept-Encoding: gzip,deflate
Accept-Language: zh-CN,zh;q=0.8
Cookie: DWRSESSIONID=N3O6b25DRmadhrXc4E2$rYIT2al; JSESSIONIDMINI=DyMFWyqPrgJ3jnL1fpGLQt5qkQsBZc1nrVykqpv2xrtszDrN0Vqb!1710009500; BIGipServerpool_sys=152475146.53511.0000; ComtopSessionSID=<SNAID>DyMFWyqPrgJ3jnL1fpGLQt5qkQsBZc1nrVykqpv2xrtszDrN0Vqb!1710009500!1453877807674</SNAID>; cs_cStorage_=1; JSESSIONIDbbb=q0h5WyqCMZLk9LzVhGkLLKvRxvn2KtX2hT7LhVtHxW3LyGvpcR1c!10296795; BIGipServerpool_bbb=135697930.35603.0000
X-Forwarded-For: 10.122.8.83
]”, which is more than the configured time (StuckThreadMaxTime) of “600” seconds. Stack trace:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811)
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:969)
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1281)
java.util.concurrent.CountDownLatch.await(CountDownLatch.java:207)
com.comtop.aaa.bbb.plan.common.bpms.thread.ThreadBpmsUtil.execTread(ThreadBpmsUtil.java:170)
com.comtop.aaa.bbb.plan.common.bpms.thread.ThreadBpmsUtil.doBatchCommonFore(ThreadBpmsUtil.java:97)
com.comtop.aaa.bbb.plan.common.bpms.action.PmsBpmsOperateAction.doBatchCommonFore(PmsBpmsOperateAction.java:681)
com.comtop.aaa.bbb.plan.common.bpms.action.PmsBpmsOperateAction.handleBatchFore(PmsBpmsOperateAction.java:382)
查看数据库,可以看到堵塞的持有者是weblogic
SELECT bs.username “Blocking User”,
bs.username “DB User”,
bs.SID “SID”,
bs.serial# “Serial#”,
bs.sql_address “address”,
bs.sql_hash_value “Sql hash”,
bs.program “Blocking App”,
bs.machine “Blocking Machine”,
bs.osuser “Blocking OS User”,
bs.serial# “Serial#”,
ws.username “Waiting User”,
ws.SID “WSID”,
ws.program “Waiting App”,
ws.machine “Waiting Machine”,
ws.osuser “Waiting OS User”,
ws.serial# “WSerial#”,
wk.TYPE lock_type,
hk.lmode mode_held,
wk.request mode_requested,
TO_CHAR(hk.id1) lock_id1,
TO_CHAR(hk.id2) lock_id2,
hk.BLOCK blocking_others
FROM gv$lock hk, gv$session bs, gv$lock wk, gv$session ws
WHERE hk.BLOCK = 1
AND hk.lmode != 0
AND hk.lmode != 1
AND wk.request != 0
AND wk.TYPE(+) = hk.TYPE
AND wk.id1(+) = hk.id1
AND wk.id2(+) = hk.id2
AND hk.SID = bs.SID(+)
AND wk.SID = ws.SID(+)
AND (bs.username IS NOT NULL)
AND (bs.username <> ‘SYSTEM’)
AND (bs.username <> ‘SYS’)
ORDER BY 1;
在数据库报告中,也看到相应的SQL:
Elapsed Time (s) | Executions | Elapsed Time per Exec (s) | %Total | %CPU | %IO | SQL Id | SQL Module | SQL Text |
---|---|---|---|---|---|---|---|---|
6,781.34 | 7,180 | 0.94 | 7.72 | 0.06 | 0.93 | 40dnpz84608j2 | JDBC Thin Client | xxxxx |
4,589.00 | 322 | 14.25 | 5.22 | 63.95 | 0.07 | 5tx8x8vavda6h | JDBC Thin Client | xxxx. |
4,327.27 | 2 | 2,163.63 | 4.93 | 0.01 | 0.00 | apyz8h30zr2ph | JDBC Thin Client | UPDATE PLAN_BATCH SET PLAN_… |