FastJson导致的JVM Crash
凌晨六点Java进程crash了,生成了一个hs_error_pid.log文件。文件内容如下:
#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0x00007fe88ad98394, pid=4486, tid=140628449421056
#
# JRE version: Java(TM) SE Runtime Environment (8.0_60-b27) (build 1.8.0_60-b27)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.60-b23 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# J 17922 C2 com.alibaba.fastjson.serializer.MapSerializer.write(Lcom/alibaba/fastjson/serializer/JSONSerializer;Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/reflect/Type;IZ)V (1290 bytes) @ 0x00007fe88ad98394 [0x00007fe88ad97b40+0x854]
#
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# If you would like to submit a bug report, please visit:
# http://bugreport.java.com/bugreport/crash.jsp
#
--------------- T H R E A D ---------------
Current thread (0x00007fe3cc002000): JavaThread "vert.x-eventloop-thread-3" [_thread_in_Java, id=4677, stack(0x00007fe69cc1e000,0x00007fe69cc5f000)]
siginfo: si_signo: 11 (SIGSEGV), si_code: 1 (SEGV_MAPERR), si_addr: 0x0000000000000038
Registers:
RAX=0x0000000000000000, RBX=0x00000003aa391a38, RCX=0x00000000000a13c5, RDX=0x00000003aa391a38
RSP=0x00007fe69cc5c980, RBP=0x0000000000000001, RSI=0x00000003aa7cf1d0, RDI=0x0000000000000022
R8 =0x000000000000003a, R9 =0x00000003aa68ca40, R10=0x0000000000000030, R11=0x00000000000db1ee
R12=0x0000000000000000, R13=0x00000000000a13c6, R14=0x00000000754d1948, R15=0x00007fe3cc002000
RIP=0x00007fe88ad98394, EFLAGS=0x0000000000010246, CSGSFS=0x0000000000000033, ERR=0x0000000000000004
TRAPNO=0x000000000000000e
Top of Stack: (sp=0x00007fe69cc5c980)
0x00007fe69cc5c980: 00055bf800000000 00000003aa847008
0x00007fe69cc5c990: 00000003aa2dbd98 00000003aa391a38
0x00007fe69cc5c9a0: 00000005e9f3fef0 00000006a97cf600
0x00007fe69cc5c9b0: 0000000000000000 00000003aa2dbd98
0x00007fe69cc5c9c0: 00000003aa391a78 0000000580007568
0x00007fe69cc5c9d0: 000000058012aed8 000016d000000000
0x00007fe69cc5c9e0: 00000005000a13c1 00000006a7cc9d30
0x00007fe69cc5c9f0: 0000000000000030 00007fe3cc002000
0x00007fe69cc5ca00: 0000000000000030 00000006a7cc9d30
0x00007fe69cc5ca10: 000a13850000002c 000a130e000a138b
0x00007fe69cc5ca20: 00000001000a130e 0000001300000001
0x00007fe69cc5ca30: 00000006a7c6be18 0000000000000017
0x00007fe69cc5ca40: 00000000d52f9ec0 00007fe889f72498
0x00007fe69cc5ca50: 000853a075472347 0000003effffffff
0x00007fe69cc5ca60: 0000017faea8bc07 000000080001dea8
0x00007fe69cc5ca70: 00000000d52f9ec0 00007fe88ba499b4
0x00007fe69cc5ca80: 00000003aa391a78 0000000375472347
0x00007fe69cc5ca90: 00000003aa2dbd60 0001dea87545b7b3
0x00007fe69cc5caa0: 00000003aa391a38 000a1393000a138c
0x00007fe69cc5cab0: 00000003aa2dbd98 0000000000000000
0x00007fe69cc5cac0: 00000003aa17d928 00000005ea380150
0x00007fe69cc5cad0: 0000000000000001 00007fe88a707130
0x00007fe69cc5cae0: 00000006a7de7040 00007fe88a92ac14
0x00007fe69cc5caf0: 0000000000000009 0000001b00000000
0x00007fe69cc5cb00: 0000000000000001 00007fe88a1598dc
0x00007fe69cc5cb10: 0000000000000001 00007fe88a848d3c
0x00007fe69cc5cb20: 0000000000000000 00000003aa3851f8
0x00007fe69cc5cb30: 00000003aa391a78 00007fe88bab55c8
0x00007fe69cc5cb40: 00000003aa391a38 bd47002a7542fb25
0x00007fe69cc5cb50: 00000005ea380150 00000003aa391a78
0x00007fe69cc5cb60: 0000000300000000 0000000075472347
0x00007fe69cc5cb70: 00000003aa17d928 000a131400000000
Instructions: (pc=0x00007fe88ad98394)
0x00007fe88ad98374: 00 00 48 8b 74 24 18 48 8b 54 24 68 b9 3a 00 00
0x00007fe88ad98384: 00 66 90 e8 d4 8b 98 ff 4c 8b 94 24 80 00 00 00
0x00007fe88ad98394: 45 8b 52 08 49 bc 00 00 00 00 08 00 00 00 4d 03
0x00007fe88ad983a4: d4 4d 33 e4 4d 8b 52 68 4c 89 54 24 58 4c 3b 54
Register to memory mapping:
RAX=0x0000000000000000 is an unknown value
RBX=
[error occurred during error reporting (printing register info), id 0xb]
Stack: [0x00007fe69cc1e000,0x00007fe69cc5f000], sp=0x00007fe69cc5c980, free space=250k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
J 17922 C2 com.alibaba.fastjson.serializer.MapSerializer.write(Lcom/alibaba/fastjson/serializer/JSONSerializer;Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/reflect/Type;IZ)V (1290 bytes) @ 0x00007fe88ad98394 [0x00007fe88ad97b40+0x854]
J 4520 C2 com.alibaba.fastjson.serializer.MapSerializer.write(Lcom/alibaba/fastjson/serializer/JSONSerializer;Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/reflect/Type;I)V (13 bytes) @ 0x00007fe889f72498 [0x00007fe889f72460+0x38]
J 17029 C2 com.alibaba.fastjson.serializer.ASMSerializer_15_MonitorRecord.writeDirectNonContext(Lcom/alibaba/fastjson/serializer/JSONSerializer;Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/reflect/Type;I)V (669 bytes) @ 0x00007fe88ba499b4 [0x00007fe88ba49640+0x374]
J 12297 C2 com.alibaba.fastjson.serializer.ASMSerializer_15_MonitorRecord.write(Lcom/alibaba/fastjson/serializer/JSONSerializer;Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/reflect/Type;I)V (745 bytes) @ 0x00007fe88a707130 [0x00007fe88a7070c0+0x70]
J 16534 C2 com.alibaba.fastjson.serializer.ListSerializer.write(Lcom/alibaba/fastjson/serializer/JSONSerializer;Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/reflect/Type;I)V (642 bytes) @ 0x00007fe88bab55c8 [0x00007fe88bab4bc0+0xa08]
J 17922 C2 com.alibaba.fastjson.serializer.MapSerializer.write(Lcom/alibaba/fastjson/serializer/JSONSerializer;Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/reflect/Type;IZ)V (1290 bytes) @ 0x00007fe88ad98408 [0x00007fe88ad97b40+0x8c8]
J 4520 C2 com.alibaba.fastjson.serializer.MapSerializer.write(Lcom/alibaba/fastjson/serializer/JSONSerializer;Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/reflect/Type;I)V (13 bytes) @ 0x00007fe889f72498 [0x00007fe889f72460+0x38]
J 14196 C2 com.alibaba.fastjson.JSON.toJSONString(Ljava/lang/Object;[Lcom/alibaba/fastjson/serializer/SerializerFeature;)Ljava/lang/String; (9 bytes) @ 0x00007fe88a5249d4 [0x00007fe88a524600+0x3d4]
J 17123 C2 io.vertx.ext.web.impl.RoutingContextImplBase.iterateNext()Z (508 bytes) @ 0x00007fe88bb929d4 [0x00007fe88bb8e8e0+0x40f4]
J 11993 C1 io.vertx.ext.web.impl.MyRouter.handle(Lio/vertx/core/http/HttpServerRequest;)V (87 bytes) @ 0x00007fe88a580f74 [0x00007fe88a57fd00+0x1274]
J 11992 C1 io.vertx.ext.web.impl.MyRouter.handle(Ljava/lang/Object;)V (9 bytes) @ 0x00007fe88a3364dc [0x00007fe88a336340+0x19c]
J 11991 C1 io.vertx.core.http.impl.WebSocketRequestHandler.handle(Lio/vertx/core/http/HttpServerRequest;)V (56 bytes) @ 0x00007fe88970b5ac [0x00007fe88970b140+0x46c]
J 11990 C1 io.vertx.core.http.impl.WebSocketRequestHandler.handle(Ljava/lang/Object;)V (9 bytes) @ 0x00007fe88971b6dc [0x00007fe88971b540+0x19c]
J 11865 C1 io.vertx.core.http.impl.HttpServerRequestImpl.handleBegin()V (71 bytes) @ 0x00007fe88a39ed04 [0x00007fe88a39e520+0x7e4]
J 15599 C1 io.vertx.core.http.impl.Http1xServerConnection.handleMessage(Ljava/lang/Object;)V (116 bytes) @ 0x00007fe88a5dba5c [0x00007fe88a5db400+0x65c]
J 13951 C2 io.vertx.core.net.impl.VertxHandler$$Lambda$167.handle(Ljava/lang/Object;)V (9 bytes) @ 0x00007fe88ad93194 [0x00007fe88ad930e0+0xb4]
J 13950 C2 io.vertx.core.net.impl.VertxHandler.channelRead(Lio/netty/channel/ChannelHandlerContext;Ljava/lang/Object;)V (15 bytes) @ 0x00007fe88a56eb30 [0x00007fe88a56ea40+0xf0]
J 13901 C2 io.netty.handler.codec.http.websocketx.extensions.WebSocketServerExtensionHandler.channelRead(Lio/netty/channel/ChannelHandlerContext;Ljava/lang/Object;)V (210 bytes) @ 0x00007fe88ad2f31c [0x00007fe88ad2ef00+0x41c]
J 16709 C2 io.netty.handler.codec.ByteToMessageDecoder.channelRead(Lio/netty/channel/ChannelHandlerContext;Ljava/lang/Object;)V (317 bytes) @ 0x00007fe88b93c5dc [0x00007fe88b93bf20+0x6bc]
J 7867 C2 io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(Lio/netty/channel/ChannelHandlerContext;Ljava/lang/Object;)V (9 bytes) @ 0x00007fe88a6dbc4c [0x00007fe88a6dbaa0+0x1ac]
J 17014 C2 io.netty.channel.nio.NioEventLoop.processSelectedKey(Ljava/nio/channels/SelectionKey;Lio/netty/channel/nio/AbstractNioChannel;)V (142 bytes) @ 0x00007fe88bb723d4 [0x00007fe88bb71540+0xe94]
J 12995% C2 io.netty.channel.nio.NioEventLoop.run()V (221 bytes) @ 0x00007fe88b62644c [0x00007fe88b626080+0x3cc]
j io.netty.util.concurrent.SingleThreadEventExecutor$5.run()V+44
j io.netty.util.concurrent.FastThreadLocalRunnable.run()V+4
j java.lang.Thread.run()V+11
v ~StubRoutines::call_stub
V [libjvm.so+0x68bbe6] JavaCalls::call_helper(JavaValue*, methodHandle*, JavaCallArguments*, Thread*)+0x1056
V [libjvm.so+0x68c0f1] JavaCalls::call_virtual(JavaValue*, KlassHandle, Symbol*, Symbol*, JavaCallArguments*, Thread*)+0x321
V [libjvm.so+0x68c597] JavaCalls::call_virtual(JavaValue*, Handle, KlassHandle, Symbol*, Symbol*, Thread*)+0x47
V [libjvm.so+0x7232d0] thread_entry(JavaThread*, Thread*)+0xa0
V [libjvm.so+0xa68f3f] JavaThread::thread_main_inner()+0xdf
V [libjvm.so+0xa6906c] JavaThread::run()+0x11c
V [libjvm.so+0x91cb88] java_start(Thread*)+0x108
C [libpthread.so.0+0x7dd5] start_thread+0xc5
--------------- P R O C E S S ---------------
Java Threads: ( => current thread )
0x00007fe37c008800 JavaThread "Append-entries-vender_quotation_info_binlake_new_yf-0 - 1" [_thread_blocked, id=17691, stack(0x00007fe2358ad000,0x00007fe2358ee000)]
0x00007fe55008a000 JavaThread "FlushThread-vender_quotation_info_binlake_new_yf-0" [_thread_blocked, id=17690, stack(0x00007fe2358ee000,0x00007fe23592f000)]
。。。
0x00007fe8981bb800 JavaThread "Finalizer" daemon [_thread_blocked, id=4520, stack(0x00007fe888020000,0x00007fe888061000)]
0x00007fe8981b9800 JavaThread "Reference Handler" daemon [_thread_blocked, id=4519, stack(0x00007fe888061000,0x00007fe8880a2000)]
Other Threads:
0x00007fe8981b4800 VMThread [stack: 0x00007fe7ed82c000,0x00007fe7ed92c000] [id=4518]
0x00007fe89822e000 WatcherThread [stack: 0x00007fe7ec81c000,0x00007fe7ec91c000] [id=4539]
VM state:not at safepoint (normal execution)
VM Mutex/Monitor currently owned by a thread: None
Heap:
par new generation total 18874368K, used 13545531K [0x0000000080000000, 0x0000000580000000, 0x0000000580000000)
eden space 16777216K, 79% used [0x0000000080000000, 0x00000003aad6da50, 0x0000000480000000)
from space 2097152K, 12% used [0x0000000480000000, 0x000000048fea1328, 0x0000000500000000)
to space 2097152K, 0% used [0x0000000500000000, 0x0000000500000000, 0x0000000580000000)
concurrent mark-sweep generation total 10485760K, used 2692545K [0x0000000580000000, 0x0000000800000000, 0x0000000800000000)
Metaspace used 59770K, capacity 61640K, committed 61740K, reserved 1103872K
class space used 6525K, capacity 6900K, committed 6964K, reserved 1048576K
Card table byte_map: [0x00007fe8380fc000,0x00007fe83bcfd000] byte_map_base: 0x00007fe837cfc000
Marking Bits: (CMSBitMap*) 0x00007fe898153438
Bits: [0x00007fe819fff000, 0x00007fe823fff000)
Mod Union Table: (CMSBitMap*) 0x00007fe8981534f8
Bits: [0x00007fe831166000, 0x00007fe8313e6000)
Polling page: 0x00007fe89f0b6000
CodeCache: size=245760Kb used=45532Kb max_used=45665Kb free=200227Kb
bounds [0x00007fe889000000, 0x00007fe88bd60000, 0x00007fe898000000]
total_blobs=12956 nmethods=12285 adapters=579
compilation: enabled
Compilation events (10 events):
Event: 417655.889 Thread 0x00007fe898215800 17923 ! 3 org.apache.logging.log4j.core.impl.ThrowableProxyHelper::toCacheEntry (168 bytes)
Event: 417655.891 Thread 0x00007fe898215800 nmethod 17923 0x00007fe88bb28190 code [0x00007fe88bb284a0, 0x00007fe88bb2a1a8]
Event: 417661.083 Thread 0x00007fe89820c000 17924 4 com.alibaba.fastjson.serializer.ASMSerializer_35_Checkpoint::write (401 bytes)
Event: 417661.085 Thread 0x00007fe89820c000 nmethod 17924 0x00007fe88af7fad0 code [0x00007fe88af7fc40, 0x00007fe88af7fdc8]
Event: 417675.484 Thread 0x00007fe8981f7000 17925 ! 4 sun.reflect.GeneratedMethodAccessor165::invoke (68 bytes)
Event: 417675.485 Thread 0x00007fe8981f7000 nmethod 17925 0x00007fe88baa1250 code [0x00007fe88baa13c0, 0x00007fe88baa1538]
Event: 417689.765 Thread 0x00007fe898208000 17926 4 java.util.BitSet::set (58 bytes)
Event: 417689.767 Thread 0x00007fe898208000 nmethod 17926 0x00007fe88b9bb4d0 code [0x00007fe88b9bb620, 0x00007fe88b9bb7c8]
Event: 417708.633 Thread 0x00007fe8981f9800 17927 4 org.joyqueue.broker.monitor.converter.DefaultConverter$$Lambda$252/201558765::get$Lambda (15 bytes)
Event: 417708.635 Thread 0x00007fe8981f9800 nmethod 17927 0x00007fe88b83ffd0 code [0x00007fe88b840120, 0x00007fe88b840238]
GC Heap History (10 events):
Event: 415096.325 GC heap before
{Heap before GC invocations=4484 (full 4):
par new generation total 18874368K, used 17037775K [0x0000000080000000, 0x0000000580000000, 0x0000000580000000)
eden space 16777216K, 99% used [0x0000000080000000, 0x000000047f66b490, 0x0000000480000000)
from space 2097152K, 12% used [0x0000000500000000, 0x0000000510808a48, 0x0000000580000000)
to space 2097152K, 0% used [0x0000000480000000, 0x0000000480000000, 0x0000000500000000)
concurrent mark-sweep generation total 10485760K, used 2678306K [0x0000000580000000, 0x0000000800000000, 0x0000000800000000)
Metaspace used 59700K, capacity 61602K, committed 61740K, reserved 1103872K
class space used 6512K, capacity 6889K, committed 6964K, reserved 1048576K
Event: 415096.362 GC heap after
Heap after GC invocations=4485 (full 4):
par new generation total 18874368K, used 240316K [0x0000000080000000, 0x0000000580000000, 0x0000000580000000)
eden space 16777216K, 0% used [0x0000000080000000, 0x0000000080000000, 0x0000000480000000)
from space 2097152K, 11% used [0x0000000480000000, 0x000000048eaaf3b8, 0x0000000500000000)
to space 2097152K, 0% used [0x0000000500000000, 0x0000000500000000, 0x0000000580000000)
concurrent mark-sweep generation total 10485760K, used 2678315K [0x0000000580000000, 0x0000000800000000, 0x0000000800000000)
Metaspace used 59700K, capacity 61602K, committed 61740K, reserved 1103872K
class space used 6512K, capacity 6889K, committed 6964K, reserved 1048576K
}
Event: 415122.737 GC heap before
{Heap before GC invocations=4485 (full 4):
par new generation total 18874368K, used 17016560K [0x0000000080000000, 0x0000000580000000, 0x0000000580000000)
eden space 16777216K, 99% used [0x0000000080000000, 0x000000047ff0cff0, 0x0000000480000000)
from space 2097152K, 11% used [0x0000000480000000, 0x000000048eaaf3b8, 0x0000000500000000)
to space 2097152K, 0% used [0x0000000500000000, 0x0000000500000000, 0x0000000580000000)
concurrent mark-sweep generation total 10485760K, used 2678315K [0x0000000580000000, 0x0000000800000000, 0x0000000800000000)
Metaspace used 59700K, capacity 61602K, committed 61740K, reserved 1103872K
class space used 6512K, capacity 6889K, committed 6964K, reserved 1048576K
Event: 415122.788 GC heap after
Heap after GC invocations=4486 (full 4):
par new generation total 18874368K, used 262150K [0x0000000080000000, 0x0000000580000000, 0x0000000580000000)
eden space 16777216K, 0% used [0x0000000080000000, 0x0000000080000000, 0x0000000480000000)
from space 2097152K, 12% used [0x0000000500000000, 0x0000000510001878, 0x0000000580000000)
to space 2097152K, 0% used [0x0000000480000000, 0x0000000480000000, 0x0000000500000000)
concurrent mark-sweep generation total 10485760K, used 2678325K [0x0000000580000000, 0x0000000800000000, 0x0000000800000000)
Metaspace used 59700K, capacity 61602K, committed 61740K, reserved 1103872K
class space used 6512K, capacity 6889K, committed 6964K, reserved 1048576K
}
Event: 415571.656 GC heap before
{Heap before GC invocations=4486 (full 4):
par new generation total 18874368K, used 17034337K [0x0000000080000000, 0x0000000580000000, 0x0000000580000000)
eden space 16777216K, 99% used [0x0000000080000000, 0x000000047fb16c28, 0x0000000480000000)
from space 2097152K, 12% used [0x0000000500000000, 0x0000000510001878, 0x0000000580000000)
to space 2097152K, 0% used [0x0000000480000000, 0x0000000480000000, 0x0000000500000000)
concurrent mark-sweep generation total 10485760K, used 2678325K [0x0000000580000000, 0x0000000800000000, 0x0000000800000000)
Metaspace used 59701K, capacity 61602K, committed 61740K, reserved 1103872K
class space used 6512K, capacity 6889K, committed 6964K, reserved 1048576K
Event: 415571.711 GC heap after
Heap after GC invocations=4487 (full 4):
par new generation total 18874368K, used 256916K [0x0000000080000000, 0x0000000580000000, 0x0000000580000000)
eden space 16777216K, 0% used [0x0000000080000000, 0x0000000080000000, 0x0000000480000000)
from space 2097152K, 12% used [0x0000000480000000, 0x000000048fae5090, 0x0000000500000000)
to space 2097152K, 0% used [0x0000000500000000, 0x0000000500000000, 0x0000000580000000)
concurrent mark-sweep generation total 10485760K, used 2685009K [0x0000000580000000, 0x0000000800000000, 0x0000000800000000)
Metaspace used 59701K, capacity 61602K, committed 61740K, reserved 1103872K
class space used 6512K, capacity 6889K, committed 6964K, reserved 1048576K
}
Event: 415614.287 GC heap before
{Heap before GC invocations=4487 (full 4):
par new generation total 18874368K, used 17034132K [0x0000000080000000, 0x0000000580000000, 0x0000000580000000)
eden space 16777216K, 100% used [0x0000000080000000, 0x0000000480000000, 0x0000000480000000)
from space 2097152K, 12% used [0x0000000480000000, 0x000000048fae5090, 0x0000000500000000)
to space 2097152K, 0% used [0x0000000500000000, 0x0000000500000000, 0x0000000580000000)
concurrent mark-sweep generation total 10485760K, used 2685009K [0x0000000580000000, 0x0000000800000000, 0x0000000800000000)
Metaspace used 59704K, capacity 61602K, committed 61740K, reserved 1103872K
class space used 6512K, capacity 6889K, committed 6964K, reserved 1048576K
Event: 415614.336 GC heap after
Heap after GC invocations=4488 (full 4):
par new generation total 18874368K, used 236378K [0x0000000080000000, 0x0000000580000000, 0x0000000580000000)
eden space 16777216K, 0% used [0x0000000080000000, 0x0000000080000000, 0x0000000480000000)
from space 2097152K, 11% used [0x0000000500000000, 0x000000050e6d6a60, 0x0000000580000000)
to space 2097152K, 0% used [0x0000000480000000, 0x0000000480000000, 0x0000000500000000)
concurrent mark-sweep generation total 10485760K, used 2689885K [0x0000000580000000, 0x0000000800000000, 0x0000000800000000)
Metaspace used 59704K, capacity 61602K, committed 61740K, reserved 1103872K
class space used 6512K, capacity 6889K, committed 6964K, reserved 1048576K
}
Event: 416712.351 GC heap before
{Heap before GC invocations=4488 (full 4):
par new generation total 18874368K, used 17013594K [0x0000000080000000, 0x0000000580000000, 0x0000000580000000)
eden space 16777216K, 100% used [0x0000000080000000, 0x0000000480000000, 0x0000000480000000)
from space 2097152K, 11% used [0x0000000500000000, 0x000000050e6d6a60, 0x0000000580000000)
to space 2097152K, 0% used [0x0000000480000000, 0x0000000480000000, 0x0000000500000000)
concurrent mark-sweep generation total 10485760K, used 2689885K [0x0000000580000000, 0x0000000800000000, 0x0000000800000000)
Metaspace used 59758K, capacity 61630K, committed 61740K, reserved 1103872K
class space used 6525K, capacity 6899K, committed 6964K, reserved 1048576K
Event: 416712.407 GC heap after
Heap after GC invocations=4489 (full 4):
par new generation total 18874368K, used 260740K [0x0000000080000000, 0x0000000580000000, 0x0000000580000000)
eden space 16777216K, 0% used [0x0000000080000000, 0x0000000080000000, 0x0000000480000000)
from space 2097152K, 12% used [0x0000000480000000, 0x000000048fea1328, 0x0000000500000000)
to space 2097152K, 0% used [0x0000000500000000, 0x0000000500000000, 0x0000000580000000)
concurrent mark-sweep generation total 10485760K, used 2692545K [0x0000000580000000, 0x0000000800000000, 0x0000000800000000)
Metaspace used 59758K, capacity 61630K, committed 61740K, reserved 1103872K
class space used 6525K, capacity 6899K, committed 6964K, reserved 1048576K
}
Deoptimization events (10 events):
Event: 417683.267 Thread 0x00007fe3cc002000 Uncommon trap: reason=null_check action=none pc=0x00007fe88ae38d0c method=org.joyqueue.broker.manage.converter.StoreManageConverter.convert(Lorg/joyqueue/store/StoreManagementService$PartitionGroupMetric;)Lorg/joyqueue/manage/PartitionGroupMetric; @ 10
Event: 417686.304 Thread 0x00007fe3cc002000 Uncommon trap: reason=null_check action=none pc=0x00007fe88ae38d0c method=org.joyqueue.broker.manage.converter.StoreManageConverter.convert(Lorg/joyqueue/store/StoreManagementService$PartitionGroupMetric;)Lorg/joyqueue/manage/PartitionGroupMetric; @ 10
Event: 417690.602 Thread 0x00007fe617e54800 Uncommon trap: reason=range_check action=none pc=0x00007fe88a3704dc method=sun.reflect.generics.parser.SignatureParser.current()C @ 34
Event: 417690.602 Thread 0x00007fe617e54800 Uncommon trap: reason=range_check action=none pc=0x00007fe88a3704dc method=sun.reflect.generics.parser.SignatureParser.current()C @ 34
Event: 417690.602 Thread 0x00007fe617e54800 Uncommon trap: reason=range_check action=none pc=0x00007fe88a3704dc method=sun.reflect.generics.parser.SignatureParser.current()C @ 34
Event: 417690.602 Thread 0x00007fe617e54800 Uncommon trap: reason=range_check action=none pc=0x00007fe88a3704dc method=sun.reflect.generics.parser.SignatureParser.current()C @ 34
Event: 417700.612 Thread 0x00007fe617e54800 Uncommon trap: reason=range_check action=none pc=0x00007fe88a3704dc method=sun.reflect.generics.parser.SignatureParser.current()C @ 34
Event: 417700.613 Thread 0x00007fe617e54800 Uncommon trap: reason=range_check action=none pc=0x00007fe88a3704dc method=sun.reflect.generics.parser.SignatureParser.current()C @ 34
Event: 417700.613 Thread 0x00007fe617e54800 Uncommon trap: reason=range_check action=none pc=0x00007fe88a3704dc method=sun.reflect.generics.parser.SignatureParser.current()C @ 34
Event: 417700.613 Thread 0x00007fe617e54800 Uncommon trap: reason=range_check action=none pc=0x00007fe88a3704dc method=sun.reflect.generics.parser.SignatureParser.current()C @ 34
Internal exceptions (10 events):
Event: 417690.602 Thread 0x00007fe617e54800 Exception (0x00000003a5ac00c0) thrown at [/HUDSON/workspace/8-2-build-linux-amd64/jdk8u60/4407/hotspot/src/share/vm/interpreter/interpreterRuntime.cpp, line 366]
Event: 417690.602 Thread 0x00007fe617e54800 Exception (0x00000003a5ac1970) thrown at [/HUDSON/workspace/8-2-build-linux-amd64/jdk8u60/4407/hotspot/src/share/vm/interpreter/interpreterRuntime.cpp, line 366]
Event: 417690.602 Thread 0x00007fe617e54800 Exception (0x00000003a5ac3108) thrown at [/HUDSON/workspace/8-2-build-linux-amd64/jdk8u60/4407/hotspot/src/share/vm/interpreter/interpreterRuntime.cpp, line 366]
Event: 417690.602 Thread 0x00007fe617e54800 Exception (0x00000003a5ac4ca8) thrown at [/HUDSON/workspace/8-2-build-linux-amd64/jdk8u60/4407/hotspot/src/share/vm/interpreter/interpreterRuntime.cpp, line 366]
Event: 417700.612 Thread 0x00007fe617e54800 Exception (0x00000003a8365018) thrown at [/HUDSON/workspace/8-2-build-linux-amd64/jdk8u60/4407/hotspot/src/share/vm/classfile/systemDictionary.cpp, line 21
Event: 417700.613 Thread 0x00007fe617e54800 Exception (0x00000003a838e5c0) thrown at [/HUDSON/workspace/8-2-build-linux-amd64/jdk8u60/4407/hotspot/src/share/vm/interpreter/interpreterRuntime.cpp, line 366]
Event: 417700.613 Thread 0x00007fe617e54800 Exception (0x00000003a838fe70) thrown at [/HUDSON/workspace/8-2-build-linux-amd64/jdk8u60/4407/hotspot/src/share/vm/interpreter/interpreterRuntime.cpp, line 366]
Event: 417700.613 Thread 0x00007fe617e54800 Exception (0x00000003a8391608) thrown at [/HUDSON/workspace/8-2-build-linux-amd64/jdk8u60/4407/hotspot/src/share/vm/interpreter/interpreterRuntime.cpp, line 366]
Event: 417700.613 Thread 0x00007fe617e54800 Exception (0x00000003a83931a8) thrown at [/HUDSON/workspace/8-2-build-linux-amd64/jdk8u60/4407/hotspot/src/share/vm/interpreter/interpreterRuntime.cpp, line 366]
Event: 417708.641 Thread 0x00007fe3cc002000 Implicit null exception at 0x00007fe88ad98394 to 0x0000000000000000
Events (10 events):
Event: 417700.613 Thread 0x00007fe617e54800 DEOPT UNPACKING pc=0x00007fe889005229 sp=0x00007fe69eb8dc98 mode 2
Event: 417700.613 Thread 0x00007fe617e54800 Uncommon trap: trap_request=0xffffffe7 fr.pc=0x00007fe88a3704dc
Event: 417700.613 Thread 0x00007fe617e54800 DEOPT PACKING pc=0x00007fe88a3704dc sp=0x00007fe69eb8dee0
Event: 417700.613 Thread 0x00007fe617e54800 DEOPT UNPACKING pc=0x00007fe889005229 sp=0x00007fe69eb8dc98 mode 2
Event: 417700.613 Thread 0x00007fe617e54800 Uncommon trap: trap_request=0xffffffe7 fr.pc=0x00007fe88a3704dc
Event: 417700.613 Thread 0x00007fe617e54800 DEOPT PACKING pc=0x00007fe88a3704dc sp=0x00007fe69eb8dee0
Event: 417700.613 Thread 0x00007fe617e54800 DEOPT UNPACKING pc=0x00007fe889005229 sp=0x00007fe69eb8dc98 mode 2
Event: 417707.320 Executing VM operation: RevokeBias
Event: 417707.322 Executing VM operation: RevokeBias done
Event: 417708.635 Thread 0x00007fe8981f9800 flushing nmethod 0x00007fe8893921d0
Dynamic libraries:
00400000-00401000 r-xp 00000000 08:11 6442770556 /export/servers/jdk1.8.0_60/bin/java
00600000-00601000 rw-p 00000000 08:11 6442770556 /export/servers/jdk1.8.0_60/bin/java
00fa3000-00fc4000 rw-p 00000000 00:00 0 [heap]
。。。
FastJson采用了ASM字节码增强技术提升性能,我们的Java进程默认也是开启字节码的。
从hs_err_pid.log文件来看,vertx作为Server接受API请求时,调用JSON.toString,走了ASM字节码增强模式。凌晨6点,这个时候大概率是在处理一个周期性的监控请求(1分钟1次),这个监控有一个特别,拉取的数据特别大,大概2k多条监控数据(大概2MB),也就是JSON.toString处理的数据比较大。
从监控请求日志可以看到,宕机前3分钟拉取到的监控数据都为0。也就是说宕机前3分钟已经发生状况。持续3分钟才crash的。不排除JSON.toString在并发情况下产生异常。
从FastJson的git issues列表也没有相同的问题。初步考虑先把字节码增强关闭或者把这个接口进行改造,改成分批次拉取,控制每次转换的数据量。未来考虑把FastJson替换成其他工具,在可靠性要求很高的分布式系统中,用FastJson还是隐患很多。
多了解一些FastJson安全这方面的问题。
1.2.69版本之前版本存在安全漏洞。
AutoType安全隐患
AutoType是什么
Fastjson提供了autotype功能,允许用户在反序列化数据中通过“@type”指定反序列化的Class类型。
但是以下场景,可能导致安全隐患:
1. 服务端接收到的请求Json串中包含了指定恶意代码Class的@Type
2. 服务端调用JSON.parseObject()时触发了该Class中的构造函数、或者是getter、setter方法中的恶意代码
在 1.2.68 版本中,引入了 safeMode 的安全模式,无论白名单和黑名单,都不支持 AutoType,这样就可以彻底地杜绝攻击。
题外片:
fastjson为什么fast?
Java 对象序列化成 JSON 字符串,序列化SerializeWriter 类中包含了一个 char[] buf,为了避免每序列化一次就要做一次分配,使用了 ThreadLocal 来进行优化,这样就能够有效地减少对象的分配和垃圾回收,从而提升性能。使用 IdentityHashMap 而不是 HashMap,既可以避免多余的 equals 操作,又可以避免多线程并发情况下的死循环。