又是一篇关于姿势的文章,为什么是”又”呢?因为上个星期刚写完一篇关于Apache Commons Pool的正确使用姿势的文章,点击此处阅读。
Redis为我们提供便利的同时,我们也要善待里面的数据
Redis是我们数据的保管者,我们可以随时存随时取,大的小的,重要的不重要的,它都毫无怨言的帮我们保存着,甚至有些时候,我们变得很懒,存东西进去的时候顺便还贴张纸:“过了一个星期就帮我扔了吧”,对于这些,Redis也都默默的接受了(谁叫Antirez把redis设计的这么好呢)。
这次要写的就是关于这张留言纸的事。
Redis提供了一套“美好”的过期数据清理机制:
主动过期: Redis对数据是惰性过期,当一个key到了过期时间,Redis也不会马上清理,但如果这个key过期后被再次访问,Redis就会主动将它清理掉。
被动过期: 如果过期的Key一直没被访问,Redis也不会一直把它放那不管,它会每秒10次的执行以下的清理工作:
- 随机从所有带有过期时间的Key里取出20个
- 如果发现有过期的,就清理
- 如果这里有25%的Key都是过期的,就继续回到第一步再来一次
这套过期机制设计的很赞,可以这样理解:如果当前有超过1/4的Key是过期的话,就不停地清理,直到只剩下1/4不到的Key是要过期的为止,然后就慢慢地随机抽查着清理。
现在我们再说说对待Redis数据的姿势问题。
先看一个现象,某段程序每天24小时的跑着,但时不时地出现拿不到连接的错误:
redis.clients.jedis.exceptions.JedisConnectionException: Could not get a resource from the pool at redis.clients.util.Pool.getResource(Pool.java:50) at redis.clients.jedis.JedisPool.getResource(JedisPool.java:86) ... at sun.reflect.GeneratedMethodAccessor54.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:302) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:85) at com.dianping.cat.annotation.CatTransaction.doServerAround(CatTransaction.java:70) at sun.reflect.GeneratedMethodAccessor52.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621) at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610) at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:68) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:168) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:208) at com.sun.proxy.$Proxy102.getRecommendedMediaList(Unknown Source) at com.restful.MediaController.getMediaList(MediaController.java:139) at sun.reflect.GeneratedMethodAccessor53.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:222) at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:137) at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:110) at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:814) at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:737) at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85) at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:959) at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:893) at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970) at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:861) at javax.servlet.http.HttpServlet.service(HttpServlet.java:624) at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846) at javax.servlet.http.HttpServlet.service(HttpServlet.java:731) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at com.filter.HttpHeaderValidator.doFilter(HttpHeaderValidator.java:198) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at com.filter.CorrelationFilter.doFilter(CorrelationFilter.java:40) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:87) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:77) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:121) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:218) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:505) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:169) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103) at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:956) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:442) at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1082) at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:623) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1756) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1715) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.lang.Thread.run(Thread.java:745) Caused by: java.util.NoSuchElementException: Unable to validate object at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:506) at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:363) at redis.clients.util.Pool.getResource(Pool.java:48) ... 82 more
这个错误是因为JedisPool拿到一个连接后,调用 jedis.ping()
方法没有得到正确的返回”pong”消息,而出现这个问题时,并不是traffic的高峰期,redis的cpu不高,于是程序员很快就怀疑到网络上,难道有”丢包”?
但跑了几天,发现出错的时间固定在某几个时间点(如晚上9点~10点),莫非是扫地阿姨每天定时在机房打扫卫生,碰着网线了?(有时候不得不佩服程序员的想象力)
然而在一个夜黑风高的晚上,程序员的脑子在这个时候一般是最清醒的,这时告警短信又来了,打开redis监控,像在唯品会上逛街一样,对着redis的各项监控数据逛啊逛,突然,看到有几条Expires的数据,当时的表情只能用看到关注了几天的商品上标了大大”0.1折”时的表情来形容。
接下来的事情,想必大家也都猜到个七七八八了,当然是下单抢购支付了,脑子里一系列的运算,掐指一算: “这个时间点正好有一大批key过期”,而且都是大的Set集合,每个都有几十万的数据,再一算: “Redis里的大部分需要过期的key就是这些key”。
于是,答案有了,Redis进行被动过期清理时,发现怎么随机,都有超过1/4的Key都是过期的,就忙着不停的删啊删,再加上又都是大集合的删除,O(N)的操作复杂度,无奈,等redis删完时,客户端那边的命令都等超时了。
原因找到了,如何解决?看业务场景吧,对于我们的场景,做法是将过期时间设长一点,然后把这些可以删掉的Key标记一下,丢到一个后台线程那里分页删Set里的数据,这样就算redis再做过期操作,也不会用太多的时间来删除。
最后,稍稍总结下,对于大个的集合对象,放redis的时候需要额外注意,如果想依赖redis的过期,可能会造成过期时短暂的redis block。
所以,要善待redis数据,比如不用了就自己清理掉,不要等着redis来帮你。
http://neway6655.github.io/redis/2015/12/19/%E5%96%84%E5%BE%85Redis%E9%87%8C%E7%9A%84%E6%95%B0%E6%8D%AE.html