Skip to content

ORC-445: [C++] Code improvements in RLEV2Util. #346

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Dec 10, 2018

Conversation

fangzheng
Copy link
Contributor

No description provided.

} else {
return 64;
}
return ClosestFixedBitsMap[n];
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what if n > 64?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@xndai Hi Xiening, thanks for the catch. When computing patch width, it is possible for the input n to be larger than 64. I have added boundary check in this function and getClosestAlignedFixedBits().

@wgtmac
Copy link
Member

wgtmac commented Dec 4, 2018

Can you please provide some perf data to confirm the refactoring is effective?

@@ -23,85 +23,32 @@

namespace orc {
extern const uint32_t FBSToBitWidthMap[FixedBitSizes::SIZE];
extern const uint32_t ClosestFixedBitsMap[65];
extern const uint32_t ClosestAlignedFixedBitsMap[65];
extern const uint32_t BitWidthToFBSMap[65];

inline uint32_t decodeBitWidth(uint32_t n) {
return FBSToBitWidthMap[n];
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

make sure this won't be out of bound.

Copy link
Contributor Author

@fangzheng fangzheng Dec 5, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi Gang, I thought this over and don't feel we need to add boundary check here for two reasons:

  1. in current RLEv2 encoder and decoder code, all the callers of decodeBitWidth() pass in a integer that only has the lowest 5 bits set, so the input n is always less than FixedBitSizes::SIZE (32).
  2. Since this function is used to decode the 5-bit length code, it would be a programming error for a caller to pass in any value that is >= 32. In this case, returning 64 (as the original implementation does) is not helping.

@fangzheng
Copy link
Contributor Author

@wgtmac Hi Gang, thanks for the comments. I'll add performance measurements later this week.

@fangzheng
Copy link
Contributor Author

@wgtmac Hi Gang,
I put together a test program and some measurements here: https://github.com/fangzheng/RLEV2Util_performance

The new functions are 20 to 40% faster than the original ones.

I've also committed some new changes:

  1. change the lookup arrays from uint32_t to uint8_t to reduce their sizes and get better cache behavior.
  2. add a unit test TestRLEV2Util.cc to verify that the new implementation produces the same results as original one.
  3. simplify the code in encodeBitWidth() to avoid calling getClosestFixedBits().

Please let me know if you have further suggestions. Thanks.

Copy link
Member

@wgtmac wgtmac left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 LGTM. Thanks @fangzheng for the detail perf data!

@wgtmac wgtmac merged commit 9faf7f5 into apache:master Dec 10, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants