[CONNECTOR] Add experiment report for backup tool #1562
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
chia7712
reviewed
Mar 12, 2023
chia7712
reviewed
Mar 12, 2023
|
|
||
| 並且在本專案建立的 worker 中，預設的 heap size 會發生 `OutOfMemory` 的問題，在手動變更 heap 的 max size 可以避免這問題。 | ||
|
|
||
| 下圖為 hadoop 在寫入時段的寫入速度狀況，平均寫入速度大概會在 300~600 這區間波動 |
Contributor
There was a problem hiding this comment.
是否有機會可以更細部的討論一下這個數字？例如只看 hdfs 的吞吐量大概是多少？我們的設計加上去後有沒有善用 hdfs 的效能？
Collaborator
Author
There was a problem hiding this comment.
已新增透過 hadoop fs 指令持續向 hdfs 寫入資料的速度變化圖
基本上 exporter 寫入的速度跟使用兩台同時向 hdfs 寫入的速度是差不多的
5 tasks
chia7712
reviewed
Apr 18, 2023
Contributor
chia7712
left a comment
chia7712
left a comment
There was a problem hiding this comment.
@Haser0305 感謝更新，可否把連結添加到 connector 首頁
|
|
||
| ## 結論 | ||
|
|
||
| | 大小 | 類型 | 速度(MiB) | 時間 | |
Contributor
There was a problem hiding this comment.
結論可否簡述工具做了什麼以及效能表現（與直接寫hdfs相較只損失了多少）
Collaborator
Author
There was a problem hiding this comment.
好的，那我最近找時間再借一下機器做測試
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
#1491
現在有時實做出 exporter 的實驗，目前有在 100GB 與 500GB 的情境下跑過 exporter 工具，在 100GB 的情況下沒有問題正常結束，但是 500GB 在中後期會如實驗中所敘述，發生 heap out of memory 的情況，目前的解法是變更 worker 的 heap 大小就可以正常執行。