fix(backend): improve IPPrefixReconcileQuery performance #6249
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
github-actions
bot
added
the
group/backend
fatih-acar
force-pushed
the
fac-ipam-prefix-reconcile-perf-improvement
branch
from
3ebb39c to
e001b78
Compare
CodSpeed Performance ReportMerging #6249 will not alter performanceComparing Summary
|
dgarros
approved these changes
Apr 25, 2025
By directly filtering the node labels. Using WHERE in labels() did not trigger the NodeByLabelScan and thus processed all nodes. Also add a WHERE IN clause for the binary address lookup so that the binary address index is effectively used. Looking up for the binary address + prefix length tuple prevented the planner from using the index. Signed-off-by: Fatih Acar <[email protected]>
fatih-acar
force-pushed
the
fac-ipam-prefix-reconcile-perf-improvement
branch
from
e001b78 to
79a1610
Compare
Added one, feel free to merge if it's ok for you. |
ajtmccarty
approved these changes
Apr 25, 2025
ajtmccarty
reviewed
Apr 25, 2025
| self.params["possible_prefix_list"] = [] | ||
| for possible_prefix, max_length in possible_prefix_map.items(): | ||
| self.params["possible_prefix_and_length_list"].append([possible_prefix, max_length]) | ||
| self.params["possible_prefix_list"].append(possible_prefix) |
There was a problem hiding this comment.
I've also started using a dict instead of two lists for filtering like this, which saves on the number of variables you need to pass in with the query
in this case, the param would be something like possible_prefix_length_map and it would look like {possible_prefix: max_length}
then in the query it would look like
WHERE $possible_prefix_length_map[av.binary_address] IS NOT NULL
AND av.prefixlen <= $possible_prefix_length_map[av.binary_address]
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
By directly filtering the node labels. Using WHERE in labels() did not trigger the NodeByLabelScan and thus processed all nodes.
Also add a WHERE IN clause for the binary address lookup so that the binary address index is effectively used.
Looking up for the binary address + prefix length tuple prevented the planner from using the index.
This brings a 10% improvement according to the testing framework (using TestSite), with way lesser CPU usage.