Skip to content

[CI] NettyAllocatorTests testTrashingCompositeByteBuf failing #126743

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
elasticsearchmachine opened this issue Apr 12, 2025 · 2 comments · Fixed by #126744
Closed

[CI] NettyAllocatorTests testTrashingCompositeByteBuf failing #126743

elasticsearchmachine opened this issue Apr 12, 2025 · 2 comments · Fixed by #126744
Assignees
@mhl-b
Labels
:Distributed Coordination/Network Http and internode communication implementations low-risk An open issue or test failure that is a low risk to future releases Team:Distributed Coordination Meta label for Distributed Coordination team >test-failure Triaged test failures from CI

Comments

@elasticsearchmachine
Copy link
Collaborator

elasticsearchmachine commented Apr 12, 2025 •
edited
Loading

Build Scans:

Reproduction Line:

./gradlew ":modules:transport-netty4:test" --tests "org.elasticsearch.transport.netty4.NettyAllocatorTests.testTrashingCompositeByteBuf" -Dtests.seed=5F2096B0C9B4D83D -Dtests.locale=ne-IN -Dtests.timezone=Australia/Melbourne -Druntime.java=24

Applicable branches:
main

Reproduces locally?:
N/A

Failure History:
See dashboard

Failure Message:

java.lang.AssertionError: 
Expected: an empty collection
     but: <[LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
Recent access records: 
#1:
	Hint: Caller of readInbound() will handle the message from this point
	io.netty.buffer.AdvancedLeakAwareCompositeByteBuf.touch(AdvancedLeakAwareCompositeByteBuf.java:36)
	io.netty.handler.codec.http.HttpObjectAggregator$AggregatedFullHttpMessage.touch(HttpObjectAggregator.java:366)
	io.netty.handler.codec.http.HttpObjectAggregator$AggregatedFullHttpRequest.touch(HttpObjectAggregator.java:445)
	io.netty.handler.codec.http.HttpObjectAggregator$AggregatedFullHttpRequest.touch(HttpObjectAggregator.java:396)
	io.netty.channel.embedded.EmbeddedChannel.readInbound(EmbeddedChannel.java:314)
	org.elasticsearch.http.netty4.Netty4HttpHeaderValidatorTests.testWithFlowControlAndAggregator(Netty4HttpHeaderValidatorTests.java:160)
	java.base/jdk.internal.ref
[truncated]

Issue Reasons:

  • [main] 3 consecutive failures in step ubuntu-2404-aarch64_checkpart1_platform-support-arm
  • [main] 2 consecutive failures in step ubuntu-2004-aarch64_checkpart1_platform-support-arm
  • [main] 2 consecutive failures in step concurrent-search-tests
  • [main] 2 consecutive failures in step ubuntu-2004_platform-support-unix
  • [main] 3 consecutive failures in step ubuntu-2404_platform-support-unix
  • [main] 5 consecutive failures in step rocky-9_platform-support-unix
  • [main] 2 consecutive failures in step rhel-9_platform-support-unix
  • [main] 5 consecutive failures in step amazonlinux-2023_platform-support-aws
  • [main] 7 consecutive failures in step oraclelinux-9_platform-support-unix
  • [main] 4 consecutive failures in step single-processor-node-tests
  • [main] 97 failures in test testTrashingCompositeByteBuf (15.3% fail rate in 636 executions)
  • [main] 3 failures in step almalinux-8-aarch64_checkpart1_platform-support-arm (33.3% fail rate in 9 executions)
  • [main] 7 failures in step ubuntu-2404-aarch64_checkpart1_platform-support-arm (77.8% fail rate in 9 executions)
  • [main] 7 failures in step ubuntu-2004-aarch64_checkpart1_platform-support-arm (77.8% fail rate in 9 executions)
  • [main] 4 failures in step ubuntu-2204_platform-support-unix (44.4% fail rate in 9 executions)

Note:
This issue was created using new test triage automation. Please report issues or feedback to es-delivery.
@elasticsearchmachine elasticsearchmachine added :Distributed Coordination/Network Http and internode communication implementations >test-failure Triaged test failures from CI needs:risk Requires assignment of a risk label (low, medium, blocker) Team:Distributed Coordination Meta label for Distributed Coordination team labels Apr 12, 2025
@elasticsearchmachine
Copy link
Collaborator Author

Pinging @elastic/es-distributed-coordination (Team:Distributed Coordination)

@mhl-b
Copy link
Contributor

mhl-b commented Apr 12, 2025

New test is leaking.

org.elasticsearch.http.netty4.Netty4HttpHeaderValidatorTests.testWithFlowControlAndAggregator

1> Created at: |  
-- | --
  | 1> 	io.netty.buffer.AbstractByteBufAllocator.compositeHeapBuffer(AbstractByteBufAllocator.java:214) |  
  | 1> 	io.netty.buffer.AbstractByteBufAllocator.compositeBuffer(AbstractByteBufAllocator.java:204) |  
  | 1> 	io.netty.handler.codec.MessageAggregator.decode(MessageAggregator.java:269) |  
  | 1> 	io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:91) |  
  | 1> 	org.elasticsearch.http.netty4.Netty4HttpAggregator.channelRead(Netty4HttpAggregator.java:51) |  
  | 1> 	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) |  
  | 1> 	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) |  
  | 1> 	io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) |  
  | 1> 	org.elasticsearch.http.netty4.Netty4HttpHeaderValidator.lambda$handleValidationResult$1(Netty4HttpHeaderValidator.java:111) |  
  | 1> 	io.netty.channel.embedded.EmbeddedEventLoop.runTasks(EmbeddedEventLoop.java:74) |  
  | 1> 	io.netty.channel.embedded.EmbeddedChannel.runPendingTasks(EmbeddedChannel.java:814) |  
  | 1> 	org.elasticsearch.http.netty4.Netty4HttpHeaderValidatorTests.testWithFlowControlAndAggregator(Netty4HttpHeaderValidatorTests.java:158)

@mhl-b mhl-b mentioned this issue Apr 12, 2025
Merged
@mhl-b mhl-b added low-risk An open issue or test failure that is a low risk to future releases and removed needs:risk Requires assignment of a risk label (low, medium, blocker) labels Apr 12, 2025
@mhl-b mhl-b self-assigned this Apr 12, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@mhl-b mhl-b

Labels
:Distributed Coordination/Network Http and internode communication implementations low-risk An open issue or test failure that is a low risk to future releases Team:Distributed Coordination Meta label for Distributed Coordination team >test-failure Triaged test failures from CI
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Release buffers in netty test mhl-b/elasticsearch
2 participants
@mhl-b @elasticsearchmachine