[Bug] SkyWalking Agent Fails to Support Lettuce Core 6.5.2+ ProtocolKeyword Changes

[CodingOX]

Search before asking

• I had searched in the issues and found no similar issues.

Apache SkyWalking Component

Java Agent (apache/skywalking-java)

What happened

In my Java project, I use Lettuce to connect to Redis. Recently, I upgraded the Lettuce core dependency from version 6.4.2.RELEASE to 6.5.2.RELEASE. After the upgrade, the host monitoring system alerted me that the disk space was growing rapidly. Upon investigation, I discovered that the SkyWalking agent logs were being printed excessively, filling up the disk space.

The log content looks like this:

ERROR 2025-01-15 11:00:44.763 http-nio-16000-exec-56 InstMethodsInter : class[class io.lettuce.core.protocol.DefaultEndpoint] before method[write] intercept failure 
java.lang.NoSuchMethodError: 'java.lang.String io.lettuce.core.protocol.ProtocolKeyword.name()'
        at org.apache.skywalking.apm.plugin.lettuce.v5.RedisChannelWriterInterceptor.beforeMethod(RedisChannelWriterInterceptor.java:74)
        at org.apache.skywalking.apm.agent.core.plugin.interceptor.enhance.InstMethodsInter.intercept(InstMethodsInter.java:76)
        at io.lettuce.core.protocol.DefaultEndpoint.write(DefaultEndpoint.java)
        at io.lettuce.core.protocol.CommandExpiryWriter.$sw$original$write$n6n29m0(CommandExpiryWriter.java:125)
        at io.lettuce.core.protocol.CommandExpiryWriter.$sw$original$write$n6n29m0$accessor$$sw$i30bfr2(CommandExpiryWriter.java)
        at io.lettuce.core.protocol.CommandExpiryWriter$$sw$auxiliary$cban2k2.call(Unknown Source)
        at org.apache.skywalking.apm.agent.core.plugin.interceptor.enhance.InstMethodsInter.intercept(InstMethodsInter.java:86)
        at io.lettuce.core.protocol.CommandExpiryWriter.write(CommandExpiryWriter.java)
        at io.lettuce.core.RedisChannelHandler.dispatch(RedisChannelHandler.java:203)
        at io.lettuce.core.StatefulRedisConnectionImpl.dispatch(StatefulRedisConnectionImpl.java:184)
        at io.lettuce.core.AbstractRedisAsyncCommands.dispatch(AbstractRedisAsyncCommands.java:740)
        at io.lettuce.core.AbstractRedisAsyncCommands.sadd(AbstractRedisAsyncCommands.java:2054)
        at org.springframework.data.redis.connection.lettuce.LettuceInvoker.lambda$just$3(LettuceInvoker.java:109)
        at org.springframework.data.redis.connection.lettuce.LettuceConnection.lambda$doInvoke$3(LettuceConnection.java:447)
        at org.springframework.data.redis.connection.lettuce.LettuceInvoker$Synchronizer.invoke(LettuceInvoker.java:665)
        at org.springframework.data.redis.connection.lettuce.LettuceInvoker.just(LettuceInvoker.java:109)
        at org.springframework.data.redis.connection.lettuce.LettuceSetCommands.sAdd(LettuceSetCommands.java:56)
        at org.springframework.data.redis.connection.DefaultedRedisConnection.sAdd(DefaultedRedisConnection.java:817)
        at org.springframework.data.redis.connection.DefaultStringRedisConnection.sAdd(DefaultStringRedisConnection.java:735)
        at org.springframework.data.redis.core.DefaultSetOperations.lambda$add$0(DefaultSetOperations.java:48)
        at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:411)
        at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:378)
        at org.springframework.data.redis.core.AbstractOperations.execute(AbstractOperations.java:117)
        at org.springframework.data.redis.core.DefaultSetOperations.add(DefaultSetOperations.java:48)
        at domain.xx.xx.bus.live.common.LiveRedisDao.insertToRedis(LiveRedisDao.java:46)
        at domain.xx.xx.bus.live.manager.LiveUserActionManager.enrol(LiveUserActionManager.java:109)
        at jdk.internal.reflect.GeneratedMethodAccessor179.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:569)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:359)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:196)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
        at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:380)
        at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184)
        at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:727)
        at domain.xx.xx.bus.live.manager.LiveUserActionManager$$SpringCGLIB$$0.enrol(<generated>)
        at domain.xx.xx.bus.live.manager.LiveCoreManager.replayHeart(LiveCoreManager.java:1139)
        at jdk.internal.reflect.GeneratedMethodAccessor121.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:569)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:359)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:196)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184)
        at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:727)
        at domain.xx.xx.bus.live.manager.LiveCoreManager$$SpringCGLIB$$0.replayHeart(<generated>)
        at domain.xx.xx.bus.live.controller.LiveReplayController.recordLiveReplayHeartForAPP(LiveReplayController.java:101)
        at jdk.internal.reflect.GeneratedMethodAccessor119.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:569)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:359)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:196)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
        at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:89)
        at domain.xx.xx.frame.validate.UeValidateAspect.valid(UeValidateAspect.kt:55)
        at jdk.internal.reflect.GeneratedMethodAccessor120.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:569)
        at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:638)
        at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:628)
        at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:71)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:173)
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184)
        at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:89)
        at domain.xx.xx.frame.monitor.SlowActionRecordAspect.valid(SlowActionRecordAspect.java:42)
        at jdk.internal.reflect.GeneratedMethodAccessor63.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:569)
        at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:638)
        at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:628)
        at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:71)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:173)
        at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:727)
        at domain.xx.xx.bus.live.controller.LiveReplayController$$SpringCGLIB$$0.recordLiveReplayHeartForAPP(<generated>)
        at jdk.internal.reflect.GeneratedMethodAccessor119.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:569)
        at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:257)
        at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:190)
        at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:118)
        at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:986)
        at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:891)
        at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)
        at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1088)
        at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:978)
        at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1014)
        at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:914)
        at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:590)
        at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:885)
        at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:658)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:195)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140)
        at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:51)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140)
        at com.github.xiaoymin.knife4j.spring.filter.JakartaProductionSecurityFilter.doFilter(JakartaProductionSecurityFilter.java:62)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140)
        at domain.xx.xx.frame.web.filter.GlobalWebFilter.doFilter(GlobalWebFilter.kt:62)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140)
        at org.springframework.web.servlet.resource.ResourceUrlEncodingFilter.doFilter(ResourceUrlEncodingFilter.java:66)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140)
        at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140)
        at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140)
        at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:167)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:90)
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:483)
        at org.apache.catalina.core.StandardHostValve.$sw$original$invoke$005sj03(StandardHostValve.java:115)
        at org.apache.catalina.core.StandardHostValve.$sw$original$invoke$005sj03$accessor$$sw$p8ebm33(StandardHostValve.java)
        at org.apache.catalina.core.StandardHostValve$$sw$auxiliary$hbfpgc3.call(Unknown Source)
        at org.apache.skywalking.apm.agent.core.plugin.interceptor.enhance.InstMethodsInter.intercept(InstMethodsInter.java:86)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:93)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:344)
        at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:397)
        at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:63)
        at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:905)
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1741)
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:52)
        at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1190)
        at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:63)
        at java.base/java.lang.Thread.run(Thread.java:840)

The relevant part of my pom.xml is as follows:

<dependency>
    <groupId>io.lettuce</groupId>
    <artifactId>lettuce-core</artifactId>
    <version>6.5.2.RELEASE</version>
</dependency>

Upon further analysis, I found that the io.lettuce.core.protocol.ProtocolKeyword interface was modified in version 6.5.2.

Before 6.5.2, the interface looked like this:

public interface ProtocolKeyword {

    /**
     * @return byte[] encoded representation.
     */
    byte[] getBytes();

    /**
     * @return name of the command.
     */
    String toString();
}

After 6.5.2, it was changed to:

/**
 * Interface for protocol keywords providing an encoded representation.
 *
 * @author Mark Paluch
 */
public interface ProtocolKeyword {

    /**
     * @return byte[] encoded representation.
     */
    byte[] getBytes();

    /**
     * @return name of the command.
     */
    String name();
}

This change seems to have caused a NoSuchMethodError in the SkyWalking agent, as it expects the toString() method but now encounters the name() method instead.

What you expected to happen

1. Support for Lettuce Core 6.5.2+:
   I expected the SkyWalking agent to be compatible with Lettuce Core 6.5.2 and later versions, especially since the ProtocolKeyword interface change is a breaking change that affects the agent's functionality.

2. Reasonable Disk Space Limits for Error Logs:
   I expected the SkyWalking agent to have a default configuration that limits the size of error logs to a reasonable amount, preventing excessive disk space usage in case of repeated errors.

How to reproduce

1. Set up a Spring Boot 3 project.
2. Use Lettuce to connect to Redis, with the Lettuce Core dependency version set to 6.5.2 or higher.
3. Run the application and observe the SkyWalking agent logs for error logging related to the NoSuchMethodError for io.lettuce.core.protocol.ProtocolKeyword.name().

Anything else

No response

Are you willing to submit a pull request to fix on your own?

• Yes I am willing to submit a pull request on my own!

Code of Conduct

• I agree to follow this project's Code of Conduct

[wu-sheng]

As it is a new version, you need to submit pull request and relative test scenario update to cover that.

[wu-sheng]

@CodingOX You will need a new plugin to pass the compiling.

[ktlcove]

I encountered same problem

# this is my cmdline
-javaagent:/path/to/skywalking-java-agent.jar -jar main.jar

i saw the situation like this.
if main.jar depend on lettuce 6.5.2+, skywalking will use that lettuce instead of itself(5.1.3)
so i have to rebuild skywalking-java-agent use same version with main.jar

so, does java has some general way slove this? (dependency conflict)

i am a op not develop and don not use java.

[wu-sheng]

This is the way. #12972 (comment) Need someone to contribute a new version compatible plugin.

[wu-sheng]

But as no one shows up, I think this is not resolved.

[ktlcove]

jetty.server need upgrade too. It is so hard for me.
maybe we shoud push users to don't use jar, but write skywalking-agent as a dependency in pom.
if don not do like this, two jar dependency conflict always appear sooner or later.

[wu-sheng]

That is not possible. You don't know this as you never wrote an agent.

[wu-sheng]

There is a patch(apache/skywalking-java#760) about this in 9.5.0