ES|QL random sampling

[jan-elastic]

No description provided.

[cla-checker-service]

💚 CLA has been signed

[elasticsearchmachine]

Hi @jan-elastic, I've created a changelog YAML for you.

[bpintea]

Can we not reuse the SurrogateExpression if'ace?
We could have SampledCount and SampledSum extending the existing corresponding aggs, which already implement that interface. These subclasses would take the sample as an argument and adjust the result of their superclasses in the way the current proposal does.
The rule swapping the Count and/or Sum nodes for their sampled- equivalents would need to be inserted above SubstituteSurrogates -- it'd only execute once; as PropagateSampleFrequencyToAggs does now.

[bpintea]

The new proposal works, but what stands out to me is the fact that the agg functions that need sampling corrections become aware of this, also when that's not needed, i.e. the correcting new agg function's implementation leaks into them.
An alternative would be make the ApplySampleCorrections rule be the "repository of knowledge" about which agg function needs correction and do the substitution there directly, w/o the use of an interface (HasSampleCorrection), in the lines of:

            if (plan instanceof Aggregate && sampleProbability.get() != null) {
                plan = plan.transformExpressionsOnly(e -> switch (e) {
                    case Count count -> new CountSampleCorrection(count.source(), count.field(), count.filter(), sampleProbability.get());
                    case Sum sum -> new SumSampleCorrection(sum.source(), sum.field(), sum.filter(), sampleProbability.get());
                    default -> e;
                });
                sampleProbability.set(null);
            }

[nik9000]

the correcting new agg function's implementation leaks into them.
An alternative would be make the ApplySampleCorrections rule be the "repository of knowledge" about which agg function needs correction and do the substitution there directly, w/o the use of an interface (HasSampleCorrection), in the lines of:

Aggs themselves are the best place to put "how to correct this agg for sampling", I think. I'm not picky on how we do it, but that's a per-agg decision if I've ever seen one.

[elasticsearchmachine]

Pinging @elastic/ml-core (Team:ML)

[bpintea]

LGTM so far, maybe safe for the to-be-decided seed handling.

[bpintea]

These should be replaced with non-SampleBreaking (and instance of UnaryPlan, though this check should maybe throw, even if in the future we might have Nary nodes that could allow Sample to slide by).

[jan-elastic]

fixed the first part; let's fix the second part once we have them

[jan-elastic]

After thinking about it more, we have concluded there are three types of commands:

• those that can be swapped (as propagate probability)
• those that only propagate probability
• those that break sampling

I have updated the code reflecting that, and removed the SampleBreaking interface in doing so. PTAL

[bpintea]

It'd be great to add a planning test if possible that iterates over all available commands, dynamically discovered (not sure if we have anything like that already), and checks that unless each command belongs to an allow-filter, it'll be swapped with SAMPLE; as simple as FROM .. | COMMAND .. | SAMPLE ... This would allow us to detect if a new command is added and the implementer forgot to add it to this list.