Regex multiline support status

[RichardFevrier]

Hi 👋
I've been playing with regex on a project, so far so good up until I tried to use it in multiline context.
And I don't know if it's me or a current issue...

Here is an example to illustrate:

test := "\n\n\ntest\n// test\ntest"

rg, _ := regex.create_by_user(`/^test/m`)
defer regex.destroy_regex(rg)

capture, _ := regex.match_and_allocate_capture(rg, test)
defer regex.destroy(capture)

fmt.println(capture)

With this example nothing returns.
If I remove the ^ only the first one.
The goal here was, to get the first and the third occurences.
So it looks like global is implicit and multiline is ignored.
I've seen that you made changes around that @Feoramund recently and if you are willing to point me to the right direction I would be very thankful.

[Feoramund]

This pattern is working as it should with regards to how the VM sees the string you gave it.

AST before Optimizer: 「^⋅t⋅e⋅s⋅t」
AST after Optimizer: 「^⋅t⋅e⋅s⋅t」
            [PC: 0x0000] Save 0x00
            [PC: 0x0002] Assert_Start_Multiline
            [PC: 0x0003] Byte 't'
            [PC: 0x0005] Byte 'e'
            [PC: 0x0007] Byte 's'
            [PC: 0x0009] Byte 't'
            [PC: 0x000b] Save 0x01
            [PC: 0x000d] Match
### Adding initial thread.
Thread [PC:0x0000] thinking about Save
Thread [PC:0x0000] saving state: (slot 0 = 0)
Thread [PC:0x0002] thinking about Assert_Start_Multiline
Thread [PC:0x0003] thinking about Byte
*** New thread added [PC:0x0003]
### VM starting.
>>> Dispatching rune: '\n'
Thread [PC:0x0003] stepping on Byte
<<< Frame ended. (Threads: 0)
### VM finished.
Capture{pos = [], groups = []} false

The VM sees that the line is a valid start of a multiline string, proceeds, dispatches the \n rune to match against t, and fails.

.Multiline is not magical. The string given still must match the pattern, even across multiple lines, and when it does not, it will fail.

You will not be able to get any more captures than what you specify in your pattern, per run of the VM. If you want multiple captures of the same pattern, you must either manually run the VM on different strings (slices of the original string, perhaps) or use the iterator.

Note however that using the iterator with this pattern will not work either, as neither the VM nor the iterator will skip the string ahead upon a failed match. This is crucial to understand why this works the way it does.

I have considered changing the iterator to make your intended pattern work, and while I think it could be done for this narrow usecase, it complicates things. For instance, what do you do if someone tries a Multiline match with no ^$ anchors? Should it still advance to the next line start and try again? Why should the VM advance at all if it hits a failed match?

Here's an example that does what you want but is simpler:

package main

import "core:fmt"
import "core:strings"
import "core:text/regex"

main :: proc() {
	test := "\n\n\ntest\n// test\ntest"
	rg, err := regex.create_by_user(`/^test$/`)
	assert(err == nil)
	defer regex.destroy_regex(rg)

	caps := regex.preallocate_capture()
	defer regex.destroy(caps)

	line_num := 1
	for line in strings.split_lines_iterator(&test) {
		defer line_num += 1
		groups, ok := regex.match_with_preallocated_capture(rg, line, &caps)
		if !ok { continue }
		fmt.printfln("% 4i %q: %v %v", line_num, line, caps.groups[:groups], caps.pos[:groups])
	}
}

Running this results in:

   4 "test": ["test"] [[0, 4]]
   6 "test": ["test"] [[0, 4]]

You're able to get line numbers this way: critical information for any sensible parser.

I think core:text/regex is at a very good place in terms of feature support, and I'm not currently interested in extending it further. The fact that we support iterators should be more than enough for most people's valid use cases. RegEx is not meant to solve all text scanning problems, nor should it. It is best used in situations where you cannot predict the patterns but the strings are predictable. When you start wanting multiple captures from anywhere in a string without regard for where you start, getting into multiline patterns and pattern iteration, it may be a sign that it's time to write your own parser, and this is something I would prefer to encourage rather than adding more features to the package.