gh-127971: fix off-by-one read beyond the end of a string during search

[duaneg]

Fix cases where the string search algorithm reads one past the end of the character/byte array under certain conditions.

• Issue: ASan: heap-buffer-overflow in ucs2lib_default_find #127971

[efimov-mikhail]

I'm totally not sure that else branch is appropiate here for 'i == w2.
Even since we have i > w for this m this logic isn't clear for me.
What about simple replacement i <= w for loop condition with classical i < w?
It could be enough and cleaner.

[duaneg]

I think the else branch will be OK, since all it does is advance the index, and it is intended and expected that this could potentially advance it past the end of the string, in which case the for loop will terminate.

We can't just replace the the loop condition with i < w since the w here is the last valid index that the pattern could appear at, and needs to be checked. Otherwise we would miss valid matches, and indeed such a change breaks a large number of unit tests.

Note the conditionals that were changed are miss conditions, i.e. the algorithm has determined the character at the index cannot be part of the pattern at this location in the string. The conditionals modified are checking whether the following character could potentially be part of a pattern hit, so as to determine whether to skip it entirely by advancing the full length of the pattern or only as much as possible while still considering it as a valid potential hit. In the case where we are at the end of the buffer it doesn't actually matter which branch we take, since either way it will advance past it and terminate. We just need to avoid reading the invalid following character when it doesn't exist.

[efimov-mikhail]

Maybe it's better to rewrite condition as i+1 <= w? It seems to be more obvious way of checking for me.
It's very similar to "for" condition, but for another argument and before direct using of i+1 as index.

... the w here is the last valid index that the pattern could appear at

IMO, rewrited condition slightly reduces cognitive load.

[efimov-mikhail]

I'm not at all an expert in this code, so I've tried to apply your patch by myself.
It definitely fixes related issue.

But I suggest some changes.
Moreover, I can see very similar code at adaptive_find function.
IMO, we should investigate both cases and fix them in the same PR if needed.

[duaneg]

I'm not at all an expert in this code, so I've tried to apply your patch by myself. It definitely fixes related issue.

I have been hesitant to ping specific developers, as I don't want to annoy people or violate etiquette, but if you think it would be appropriate there are a couple of devs who have worked on this code frequently and/or recently I could tag in?

But I suggest some changes. Moreover, I can see very similar code at adaptive_find function. IMO, we should investigate both cases and fix them in the same PR if needed.

That is an excellent find, thanks! Agreed: I'll investigate, try to get a test that reproduces the issue there also, and fix it.

[efimov-mikhail]

I've just used git blame and find some names.
I hope that's appropriate.
@serhiy-storchaka @sweeneyde

[duaneg]

I can trigger the off-by-one reads in the adaptive algorithm easily enough, but I haven't been able to get it to cause an ASAN error as yet, so no luck so far on a test for that.

The third variant, default_rfind, already includes the equivalent guard:

            /* miss: check if previous character is part of pattern */          
            if (i > 0 && !STRINGLIB_BLOOM(mask, s[i-1])) {                      
                i = i - m;                                                      
            }                                                                   
            else {                                                              
                i = i - skip;                                                   
            }                                                                   
        }                                                                       
        else {                                                                  
            /* skip: check if previous character is part of pattern */          
            if (i > 0 && !STRINGLIB_BLOOM(mask, s[i-1])) {                      
                i = i - m;                                                      
            }                                                                   
        }                                                                       
    }                                                                           

[efimov-mikhail]

Does default_rfind have tests on it? If not, those worth to be added too.

IMO, we can add check and tests for adaptive_find even if there is no real crash under ASan.

[duaneg]

default_rfind does have tests, including ones that exercise the relevant code path. In fact any str.rfind with a non-matching pattern longer than one character (but shorter or equal to input length so it doesn't short-circuit) does so. adaptive_find also has tests, albeit none that trigger the code branches in question.

Looking closer at adaptive_find, on second thought I don't think the check is strictly required there. adaptive_find is only ever called directly from FASTSEARCH. For that function to use the adaptive_find algorithm the input length has to be > 2.5k length, and the pattern length has to be more than one character (actually 6+ or 100+ depending on input size). Given that, the character "one past the last possible position the pattern could start" will always be a valid character in the input string. As mentioned above, it doesn't actually matter which branch is taken, so it doesn't matter what its value is.

Presumably the extra clause in the conditional has some measurable performance impact, however slight, and given the input string minimum length requirements it will likely add up to much more than would be saved by avoiding one unnecessary character read. So, perhaps we shouldn't add it to adaptive_find, even though the current code seems safe only due to unchecked implementation details of when/under what circumstances it is called.

Nonetheless, I'll push out an update to this that adds an adaptive_find test, to ensure that code path is exercised, even though there is nothing we can check to verify it is doing what it is supposed to.

[efimov-mikhail]

I suggest to slightly rewrite i < w condition as i + 1 <= w for default_find and maybe i > 0 as i - 1 >= 0 for default_rfind.
Moreover, some short comment at adaptive_find code itself also seems valuable.

But those are only cosmetical changes, in other aspects LGTM.