fix: Fix logic in OnnxToOvNetworkBindings for stateful models #719
+12
−7
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This resolves the issue on ovep-develop where stateful models encounter a runtime error when using EPCtx-wrapped models with ORT GenAI.
The issue that I encountered when using ORT GenAI with EPCtx IR's was as follows:
Both the .onnx and the encapsulated IR have a 'beam_idx' tensor defined, so
matched_names
is set to true (line 67).The old logic was:
So immediately, it skips over this because
matched_named==true
. For beam_idx, it's something that ORT GenAI doesn't set, and hence we hit a runtime error duringBasicBacked::Infer
.I reworked the logic here to reflect the fact that both of these conditions should be true to hit this
continue
(skip io binding):And so, accomplished this with a nested if:
As that seemed more readable to me than
if ( session_context.enable_causallm && ( [.. the other conditions ...]) )
.