GC rate increases randomly, increasing the cpu to 100%, dropping indexing rate in v8.11.3

[nitesh8860]

Elasticsearch Version

8.11.3

Installed Plugins

No response

Java Version

bundled

OS Version

Linux oke-cbamoicz23a-ndvldsi765q-skex3d2ztgq-6 5.4.17-2136.305.5.3.el8uek.x86_64 #2 SMP Thu Mar 17 10:45:33 PDT 2022 x86_64 x86_64 x86_64 GNU/Linux

Problem Description

We were on version 8.7.0 from last few months and we saw no issues at all, cpu was always less then 30% overall.
our indexing rate stays around 25K/s Total, 12K/s primaries. search rate around 200/s. the cluster stores around 6TB of data at a time, 4500 shards.

We upgraded last week to 8.11.3, and the following issue started

• GC rate randomely starts increasing, and young GC reaches from 0 to 1, GC duration peaks from almost 0 to 50ms
• CPU percentage reaches 100% in next 30 mins
• indexing rate drops from 25K/s to 2K/s

We also saw an increase in bulk average size after upgrade

Steps to Reproduce

Upgrade ELK from 8.7.0 to 8.11.3

Logs (if relevant)

06:48:34.126
elasticsearch.server
[elasticsearch.server][INFO] {"@timestamp":"2024-01-02T01:18:34.126Z", "log.level": "INFO", "message":"[gc][55872] overhead, spent [267ms] collecting in the last [1s]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-es-ingest-data-hot-11][scheduler][T#1]","log.logger":"org.elasticsearch.monitor.jvm.JvmGcMonitorService","elasticsearch.cluster.uuid":"Z-shUT6qT0e8vWXtKaDfsQ","elasticsearch.node.id":"4vz0m_DyQ_--ihrJXZqMjA","elasticsearch.node.name":"elasticsearch-es-ingest-data-hot-11","elasticsearch.cluster.name":"elasticsearch"}
06:51:21.404
elasticsearch.server
[elasticsearch.server][INFO] {"@timestamp":"2024-01-02T01:21:21.404Z", "log.level": "INFO", "message":"[gc][56039] overhead, spent [261ms] collecting in the last [1s]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-es-ingest-data-hot-11][scheduler][T#1]","log.logger":"org.elasticsearch.monitor.jvm.JvmGcMonitorService","elasticsearch.cluster.uuid":"Z-shUT6qT0e8vWXtKaDfsQ","elasticsearch.node.id":"4vz0m_DyQ_--ihrJXZqMjA","elasticsearch.node.name":"elasticsearch-es-ingest-data-hot-11","elasticsearch.cluster.name":"elasticsearch"}
06:51:22.409
elasticsearch.server
[elasticsearch.server][INFO] {"@timestamp":"2024-01-02T01:21:22.409Z", "log.level": "INFO", "message":"[gc][56040] overhead, spent [275ms] collecting in the last [1s]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-es-ingest-data-hot-11][scheduler][T#1]","log.logger":"org.elasticsearch.monitor.jvm.JvmGcMonitorService","elasticsearch.cluster.uuid":"Z-shUT6qT0e8vWXtKaDfsQ","elasticsearch.node.id":"4vz0m_DyQ_--ihrJXZqMjA","elasticsearch.node.name":"elasticsearch-es-ingest-data-hot-11","elasticsearch.cluster.name":"elasticsearch"}
07:07:56.858
elasticsearch.server
[elasticsearch.server][WARN] {"@timestamp":"2024-01-02T01:37:56.858Z", "log.level": "WARN", "message":"[gc][57034] overhead, spent [538ms] collecting in the last [1s]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-es-ingest-data-hot-11][scheduler][T#1]","log.logger":"org.elasticsearch.monitor.jvm.JvmGcMonitorService","elasticsearch.cluster.uuid":"Z-shUT6qT0e8vWXtKaDfsQ","elasticsearch.node.id":"4vz0m_DyQ_--ihrJXZqMjA","elasticsearch.node.name":"elasticsearch-es-ingest-data-hot-11","elasticsearch.cluster.name":"elasticsearch"}
07:14:11.143
elasticsearch.server
[elasticsearch.server][INFO] {"@timestamp":"2024-01-02T01:44:11.143Z", "log.level": "INFO", "message":"[gc][57408] overhead, spent [511ms] collecting in the last [1s]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-es-ingest-data-hot-11][scheduler][T#1]","log.logger":"org.elasticsearch.monitor.jvm.JvmGcMonitorService","elasticsearch.cluster.uuid":"Z-shUT6qT0e8vWXtKaDfsQ","elasticsearch.node.id":"4vz0m_DyQ_--ihrJXZqMjA","elasticsearch.node.name":"elasticsearch-es-ingest-data-hot-11","elasticsearch.cluster.name":"elasticsearch"}
07:18:21.260
elasticsearch.server
[elasticsearch.server][INFO] {"@timestamp":"2024-01-02T01:48:21.260Z", "log.level": "INFO", "message":"[gc][57658] overhead, spent [260ms] collecting in the last [1s]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-es-ingest-data-hot-11][scheduler][T#1]","log.logger":"org.elasticsearch.monitor.jvm.JvmGcMonitorService","elasticsearch.cluster.uuid":"Z-shUT6qT0e8vWXtKaDfsQ","elasticsearch.node.id":"4vz0m_DyQ_--ihrJXZqMjA","elasticsearch.node.name":"elasticsearch-es-ingest-data-hot-11","elasticsearch.cluster.name":"elasticsearch"}
07:34:36.831
elasticsearch.server
[elasticsearch.server][INFO] {"@timestamp":"2024-01-02T02:04:36.831Z", "log.level": "INFO", "message":"[gc][58633] overhead, spent [383ms] collecting in the last [1s]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-es-ingest-data-hot-11][scheduler][T#1]","log.logger":"org.elasticsearch.monitor.jvm.JvmGcMonitorService","elasticsearch.cluster.uuid":"Z-shUT6qT0e8vWXtKaDfsQ","elasticsearch.node.id":"4vz0m_DyQ_--ihrJXZqMjA","elasticsearch.node.name":"elasticsearch-es-ingest-data-hot-11","elasticsearch.cluster.name":"elasticsearch"}
07:35:21.847
elasticsearch.server
[elasticsearch.server][INFO] {"@timestamp":"2024-01-02T02:05:21.847Z", "log.level": "INFO", "message":"[gc][58678] overhead, spent [409ms] collecting in the last [1s]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-es-ingest-data-hot-11][scheduler][T#1]","log.logger":"org.elasticsearch.monitor.jvm.JvmGcMonitorService","elasticsearch.cluster.uuid":"Z-shUT6qT0e8vWXtKaDfsQ","elasticsearch.node.id":"4vz0m_DyQ_--ihrJXZqMjA","elasticsearch.node.name":"elasticsearch-es-ingest-data-hot-11","elasticsearch.cluster.name":"elasticsearch"}
07:35:59.860
elasticsearch.server
[elasticsearch.server][INFO] {"@timestamp":"2024-01-02T02:05:59.860Z", "log.level": "INFO", "message":"[gc][58716] overhead, spent [396ms] collecting in the last [1s]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-es-ingest-data-hot-11][scheduler][T#1]","log.logger":"org.elasticsearch.monitor.jvm.JvmGcMonitorService","elasticsearch.cluster.uuid":"Z-shUT6qT0e8vWXtKaDfsQ","elasticsearch.node.id":"4vz0m_DyQ_--ihrJXZqMjA","elasticsearch.node.name":"elasticsearch-es-ingest-data-hot-11","elasticsearch.cluster.name":"elasticsearch"}
07:36:34.865
elasticsearch.server
[elasticsearch.server][WARN] {"@timestamp":"2024-01-02T02:06:34.865Z", "log.level": "WARN", "message":"[gc][58750] overhead, spent [1s] collecting in the last [1.9s]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-es-ingest-data-hot-11][scheduler][T#1]","log.logger":"org.elasticsearch.monitor.jvm.JvmGcMonitorService","elasticsearch.cluster.uuid":"Z-shUT6qT0e8vWXtKaDfsQ","elasticsearch.node.id":"4vz0m_DyQ_--ihrJXZqMjA","elasticsearch.node.name":"elasticsearch-es-ingest-data-hot-11","elasticsearch.cluster.name":"elasticsearch"}
07:37:14.144
elasticsearch.server
[elasticsearch.server][INFO] {"@timestamp":"2024-01-02T02:07:14.144Z", "log.level": "INFO", "message":"[gc][58789] overhead, spent [395ms] collecting in the last [1.2s]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-es-ingest-data-hot-11][scheduler][T#1]","log.logger":"org.elasticsearch.monitor.jvm.JvmGcMonitorService","elasticsearch.cluster.uuid":"Z-shUT6qT0e8vWXtKaDfsQ","elasticsearch.node.id":"4vz0m_DyQ_--ihrJXZqMjA","elasticsearch.node.name":"elasticsearch-es-ingest-data-hot-11","elasticsearch.cluster.name":"elasticsearch"}
07:37:52.209
elasticsearch.server
[elasticsearch.server][INFO] {"@timestamp":"2024-01-02T02:07:52.209Z", "log.level": "INFO", "message":"[gc][58827] overhead, spent [438ms] collecting in the last [1s]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-es-ingest-data-hot-11][scheduler][T#1]","log.logger":"org.elasticsearch.monitor.jvm.JvmGcMonitorService","elasticsearch.cluster.uuid":"Z-shUT6qT0e8vWXtKaDfsQ","elasticsearch.node.id":"4vz0m_DyQ_--ihrJXZqMjA","elasticsearch.node.name":"elasticsearch-es-ingest-data-hot-11","elasticsearch.cluster.name":"elasticsearch"}

[elasticsearchmachine]

Pinging @elastic/es-core-infra (Team:Core/Infra)

[thecoop]

The comment says it is 8.11.0, but the given version is 8.11.3. Which version is being used here?

[nitesh8860]

We actually tried both, similar behaviour, we are currently at 8.11.3, i have edited the description. thanks for pointing out.

[thecoop]

There are known issues with GC on JDK 20+, see #99592. We have merged improvements into 8.11.2 that improve matters, but (as you've noticed) don't fix the issue entirely.

The exact effect depends on your specific use & indexing patterns. A heap dump might help with determining if there's something else going on that is exacerbating the GC issues. The best bet is to run on JDK17, if possible, and keep watch on #99592

[nitesh8860]

We are running this cluster on k8s with ECK operator, as it really doesn't let us downgrade, could you please help in how can we use JDK20 or JDK17 with v8.11.3 ELK.

Error: cannot patch "elasticsearch" with kind Elasticsearch: admission webhook "elastic-es-validation-v1.k8s.elastic.co" denied the request: Elasticsearch.elasticsearch.k8s.elastic.co "elasticsearch" is invalid: spec.version: Invalid value: "8.7.0": Downgrades are not supported && cannot patch "esm" with kind Elasticsearch: admission webhook "elastic-es-validation-v1.k8s.elastic.co" denied the request: Elasticsearch.elasticsearch.k8s.elastic.co "esm" is invalid: spec.version: Invalid value: "8.7.0": Downgrades are not supported && cannot patch "kbm" with kind Kibana: admission webhook "elastic-kb-validation-v1.k8s.elastic.co" denied the request: Kibana.kibana.k8s.elastic.co "kbm" is invalid: spec.version: Forbidden: Version downgrades are not supported

I couldn't find anything around this in ECK docs

[nitesh8860]

Hello team, any help on above, or any config changes we can do to mitigate this issue ?
We reduced JVM heap size from 30G to 26G and it helped reducing frequency of this issue, but it is still happening once a day which is resolved only with a full reboot of the cluster data nodes.

[w6et]

can you share gc.log??the chart line broken cause by reboot ??

[nitesh8860]

yes, the broken chart line is caused by reboot, GC rate continuosly increases, here is snippet from when it was breaking down.

06:48:34.126
elasticsearch.server
[elasticsearch.server][INFO] {"@timestamp":"2024-01-02T01:18:34.126Z", "log.level": "INFO", "message":"[gc][55872] overhead, spent [267ms] collecting in the last [1s]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-es-ingest-data-hot-11][scheduler][T#1]","log.logger":"org.elasticsearch.monitor.jvm.JvmGcMonitorService","elasticsearch.cluster.uuid":"Z-shUT6qT0e8vWXtKaDfsQ","elasticsearch.node.id":"4vz0m_DyQ_--ihrJXZqMjA","elasticsearch.node.name":"elasticsearch-es-ingest-data-hot-11","elasticsearch.cluster.name":"elasticsearch"}
06:51:21.404
elasticsearch.server
[elasticsearch.server][INFO] {"@timestamp":"2024-01-02T01:21:21.404Z", "log.level": "INFO", "message":"[gc][56039] overhead, spent [261ms] collecting in the last [1s]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-es-ingest-data-hot-11][scheduler][T#1]","log.logger":"org.elasticsearch.monitor.jvm.JvmGcMonitorService","elasticsearch.cluster.uuid":"Z-shUT6qT0e8vWXtKaDfsQ","elasticsearch.node.id":"4vz0m_DyQ_--ihrJXZqMjA","elasticsearch.node.name":"elasticsearch-es-ingest-data-hot-11","elasticsearch.cluster.name":"elasticsearch"}
06:51:22.409
elasticsearch.server
[elasticsearch.server][INFO] {"@timestamp":"2024-01-02T01:21:22.409Z", "log.level": "INFO", "message":"[gc][56040] overhead, spent [275ms] collecting in the last [1s]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-es-ingest-data-hot-11][scheduler][T#1]","log.logger":"org.elasticsearch.monitor.jvm.JvmGcMonitorService","elasticsearch.cluster.uuid":"Z-shUT6qT0e8vWXtKaDfsQ","elasticsearch.node.id":"4vz0m_DyQ_--ihrJXZqMjA","elasticsearch.node.name":"elasticsearch-es-ingest-data-hot-11","elasticsearch.cluster.name":"elasticsearch"}
07:07:56.858
elasticsearch.server
[elasticsearch.server][WARN] {"@timestamp":"2024-01-02T01:37:56.858Z", "log.level": "WARN", "message":"[gc][57034] overhead, spent [538ms] collecting in the last [1s]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-es-ingest-data-hot-11][scheduler][T#1]","log.logger":"org.elasticsearch.monitor.jvm.JvmGcMonitorService","elasticsearch.cluster.uuid":"Z-shUT6qT0e8vWXtKaDfsQ","elasticsearch.node.id":"4vz0m_DyQ_--ihrJXZqMjA","elasticsearch.node.name":"elasticsearch-es-ingest-data-hot-11","elasticsearch.cluster.name":"elasticsearch"}
07:14:11.143
elasticsearch.server
[elasticsearch.server][INFO] {"@timestamp":"2024-01-02T01:44:11.143Z", "log.level": "INFO", "message":"[gc][57408] overhead, spent [511ms] collecting in the last [1s]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-es-ingest-data-hot-11][scheduler][T#1]","log.logger":"org.elasticsearch.monitor.jvm.JvmGcMonitorService","elasticsearch.cluster.uuid":"Z-shUT6qT0e8vWXtKaDfsQ","elasticsearch.node.id":"4vz0m_DyQ_--ihrJXZqMjA","elasticsearch.node.name":"elasticsearch-es-ingest-data-hot-11","elasticsearch.cluster.name":"elasticsearch"}
07:18:21.260
elasticsearch.server
[elasticsearch.server][INFO] {"@timestamp":"2024-01-02T01:48:21.260Z", "log.level": "INFO", "message":"[gc][57658] overhead, spent [260ms] collecting in the last [1s]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-es-ingest-data-hot-11][scheduler][T#1]","log.logger":"org.elasticsearch.monitor.jvm.JvmGcMonitorService","elasticsearch.cluster.uuid":"Z-shUT6qT0e8vWXtKaDfsQ","elasticsearch.node.id":"4vz0m_DyQ_--ihrJXZqMjA","elasticsearch.node.name":"elasticsearch-es-ingest-data-hot-11","elasticsearch.cluster.name":"elasticsearch"}
07:34:36.831
elasticsearch.server
[elasticsearch.server][INFO] {"@timestamp":"2024-01-02T02:04:36.831Z", "log.level": "INFO", "message":"[gc][58633] overhead, spent [383ms] collecting in the last [1s]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-es-ingest-data-hot-11][scheduler][T#1]","log.logger":"org.elasticsearch.monitor.jvm.JvmGcMonitorService","elasticsearch.cluster.uuid":"Z-shUT6qT0e8vWXtKaDfsQ","elasticsearch.node.id":"4vz0m_DyQ_--ihrJXZqMjA","elasticsearch.node.name":"elasticsearch-es-ingest-data-hot-11","elasticsearch.cluster.name":"elasticsearch"}
07:35:21.847
elasticsearch.server
[elasticsearch.server][INFO] {"@timestamp":"2024-01-02T02:05:21.847Z", "log.level": "INFO", "message":"[gc][58678] overhead, spent [409ms] collecting in the last [1s]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-es-ingest-data-hot-11][scheduler][T#1]","log.logger":"org.elasticsearch.monitor.jvm.JvmGcMonitorService","elasticsearch.cluster.uuid":"Z-shUT6qT0e8vWXtKaDfsQ","elasticsearch.node.id":"4vz0m_DyQ_--ihrJXZqMjA","elasticsearch.node.name":"elasticsearch-es-ingest-data-hot-11","elasticsearch.cluster.name":"elasticsearch"}
07:35:59.860
elasticsearch.server
[elasticsearch.server][INFO] {"@timestamp":"2024-01-02T02:05:59.860Z", "log.level": "INFO", "message":"[gc][58716] overhead, spent [396ms] collecting in the last [1s]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-es-ingest-data-hot-11][scheduler][T#1]","log.logger":"org.elasticsearch.monitor.jvm.JvmGcMonitorService","elasticsearch.cluster.uuid":"Z-shUT6qT0e8vWXtKaDfsQ","elasticsearch.node.id":"4vz0m_DyQ_--ihrJXZqMjA","elasticsearch.node.name":"elasticsearch-es-ingest-data-hot-11","elasticsearch.cluster.name":"elasticsearch"}
07:36:34.865
elasticsearch.server
[elasticsearch.server][WARN] {"@timestamp":"2024-01-02T02:06:34.865Z", "log.level": "WARN", "message":"[gc][58750] overhead, spent [1s] collecting in the last [1.9s]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-es-ingest-data-hot-11][scheduler][T#1]","log.logger":"org.elasticsearch.monitor.jvm.JvmGcMonitorService","elasticsearch.cluster.uuid":"Z-shUT6qT0e8vWXtKaDfsQ","elasticsearch.node.id":"4vz0m_DyQ_--ihrJXZqMjA","elasticsearch.node.name":"elasticsearch-es-ingest-data-hot-11","elasticsearch.cluster.name":"elasticsearch"}
07:37:14.144
elasticsearch.server
[elasticsearch.server][INFO] {"@timestamp":"2024-01-02T02:07:14.144Z", "log.level": "INFO", "message":"[gc][58789] overhead, spent [395ms] collecting in the last [1.2s]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-es-ingest-data-hot-11][scheduler][T#1]","log.logger":"org.elasticsearch.monitor.jvm.JvmGcMonitorService","elasticsearch.cluster.uuid":"Z-shUT6qT0e8vWXtKaDfsQ","elasticsearch.node.id":"4vz0m_DyQ_--ihrJXZqMjA","elasticsearch.node.name":"elasticsearch-es-ingest-data-hot-11","elasticsearch.cluster.name":"elasticsearch"}
07:37:52.209
elasticsearch.server
[elasticsearch.server][INFO] {"@timestamp":"2024-01-02T02:07:52.209Z", "log.level": "INFO", "message":"[gc][58827] overhead, spent [438ms] collecting in the last [1s]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-es-ingest-data-hot-11][scheduler][T#1]","log.logger":"org.elasticsearch.monitor.jvm.JvmGcMonitorService","elasticsearch.cluster.uuid":"Z-shUT6qT0e8vWXtKaDfsQ","elasticsearch.node.id":"4vz0m_DyQ_--ihrJXZqMjA","elasticsearch.node.name":"elasticsearch-es-ingest-data-hot-11","elasticsearch.cluster.name":"elasticsearch"}

[nitesh8860]

Hello Team, any help on above, or any config changes we can do to mitigate this issue ?

[nitesh8860]

Hello Team, any help on above, or any config changes we can do to mitigate this issue ?

[thecoop]

We have recently released 8.12, which includes various changes aimed at improving the behaviour of the GC on recent JDKs. I suggest upgrading to 8.12 to see what improvement that gives your particular use case.

[dougbyrne]

Hi, I don't see anything in the release notes that mentions JVM or JDK or GC, and there's no update on the issue #99592. Is there some other bug we should be following, or another list of relevant updates?

[thecoop]

A few of the tickets that have improved memory behaviour in 8.12 are #102511, #102462, #100793. We're improving the memory usage with every release over time, so upgrading to 8.12 should show some improvement in your use case.

[nitesh8860]

Hello @thecoop , we have upgraded to 8.12.0 now and I will update more on this ticket if we see improvements.
Thanks for helping on this.

[nitesh8860]

Hello @thecoop We have not seen any crashes since upgrade to 8.12.0, and it's been more than a week. ( while earlier it was crashing everyday ) i think this solves the problem. Thanks for your help :)