Skip to content

feat: Introduce cel ast rebuilder #4621

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 85 commits into
base: main
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
85 commits
Select commit Hold shift + click to select a range
d45f1dc
fix: Refactor NULL behavior with IN operator in CEL
skynetigor Apr 23, 2025
02bf401
fix: Update NULL handling in SQL test cases for consistency
skynetigor Apr 23, 2025
060bdd7
fix: Handle None comparison in SQL equality checks
skynetigor Apr 23, 2025
d8dfe14
fix: Update NULL handling in CEL to SQL test cases for IN operator
skynetigor Apr 23, 2025
cc4bce5
fix: Simplify NULL handling in SQL test cases and expressions
skynetigor Apr 23, 2025
7df9a04
fixes
skynetigor Apr 23, 2025
107554d
fix: Update NULL handling in SQL provider to use "NULL" string for co…
skynetigor Apr 23, 2025
b55dac3
fix mistake in properties metada providing to get_facet_options
skynetigor Apr 23, 2025
c59a1c4
Revert "fix mistake in properties metada providing to get_facet_options"
skynetigor Apr 23, 2025
145068b
fix: Bump version to 0.41.30 in pyproject.toml
skynetigor Apr 23, 2025
3181121
Merge branch 'main' into 4573-cant-query-incident-field-of-alerts
skynetigor Apr 25, 2025
9c2dcec
add debugging tips
skynetigor Apr 25, 2025
a524b87
introduce CelAstRebuilder
skynetigor Apr 25, 2025
39397a2
fix: Improve string representation
skynetigor Apr 25, 2025
8f215fa
refactor: Rename and enhance AST visitor methods for improved clarity…
skynetigor Apr 27, 2025
76ffbb1
add converting ConstantNode value based on data type of PropertyAcces…
skynetigor Apr 27, 2025
4a0b0ed
Add dialect-specific conversions
skynetigor Apr 27, 2025
17fd296
Make incident.id and id (alert id) be UUID for correct handling durin…
skynetigor Apr 27, 2025
e3246fb
Make incident id in incidents query be UUID
skynetigor Apr 27, 2025
68b6ab3
fix bug during data type extraction
skynetigor Apr 27, 2025
b2d4c59
Add data_type property for MultipleFieldsNode
skynetigor Apr 27, 2025
a42a27c
Add test cases for UUID conversion
skynetigor Apr 27, 2025
2020ea0
Remove unused import of Any from sqlite.py
skynetigor Apr 27, 2025
31a0543
Rename property_path from 'source' to 'providerType' in FacetDto
skynetigor Apr 27, 2025
40a0024
Add mapping for 'source' to 'alert.provider_type' in FieldMappingConf…
skynetigor Apr 27, 2025
6d8e0cc
Swap 'providerId' and 'source' mappings in FieldMappingConfiguration …
skynetigor Apr 27, 2025
8fab9fe
Rename property_path from 'providerType' to 'source' in FacetDto for …
skynetigor Apr 27, 2025
3a88e3e
Merge branch '4432-Incident-field-cant-be-queried' into 4573-introduc…
skynetigor Apr 27, 2025
432ae17
Refactor unary node handling and improve multiple fields processing i…
skynetigor Apr 27, 2025
d792977
Enhance multiple fields node handling by adding is_first and is_last …
skynetigor Apr 27, 2025
2779239
improve nodes debugging
skynetigor Apr 27, 2025
306ad69
get rid of MethodAccessNode
skynetigor Apr 27, 2025
a0b955b
get rid of MethodAccessNode code
skynetigor Apr 27, 2025
9b53358
use new comparison operators
skynetigor Apr 27, 2025
eda2c70
Remove MethodAccessNode references and related method calling logic
skynetigor Apr 27, 2025
e1d63f8
Merge branch 'main' into 4573-introduce-cel-ast-rebuilder
skynetigor Apr 28, 2025
f3d096b
Merge branch 'main' into 4573-introduce-cel-ast-rebuilder
skynetigor Apr 28, 2025
d2a2f4a
feat: Add field mapping for dismissed alerts in alert configurations
skynetigor Apr 28, 2025
b9f86d9
feat: Introduce CoalesceNode for handling coalesce operations in AST
skynetigor Apr 28, 2025
c326a85
feat: Enhance CelAstRebuilder to support CoalesceNode in logical comp…
skynetigor Apr 28, 2025
5613c90
feat: Implement CoalesceNode support in SQL providers for improved lo…
skynetigor Apr 28, 2025
a97c9a6
Introduce DataType enum in order to make CEL AST nodes more JSON frie…
skynetigor May 1, 2025
d36af25
feat: refactor AST node classes to use Pydantic for improved data val…
skynetigor May 1, 2025
c9bc3c7
replace old operators with enums and fix validation issues
skynetigor May 1, 2025
bb73dd1
feat: add CEL AST conversion to rules retrieval for enhanced data rep…
skynetigor May 1, 2025
63e4c28
Simplify contains/startsWith/endsWith CEL nodes
skynetigor May 1, 2025
5482799
Merge branch 'main' into 4662-labels-namespace-filtering-in-rules
skynetigor May 1, 2025
2991d73
feat: enhance AST node classes with Pydantic fields for improved data…
skynetigor May 1, 2025
1f1b1b1
Merge branch '4662-labels-namespace-filtering-in-rules' of https://gi…
skynetigor May 1, 2025
914b23f
fix: correct error message for unsupported type and update facet valu…
skynetigor May 1, 2025
bbe45c4
Refactor correlation table to use CelInput in readonly mode
skynetigor May 1, 2025
76461f7
refactor: rename 'cel_ast' to 'definition_cel_ast' for clarity
skynetigor May 1, 2025
8b8ddc7
refactor: rename cel_ast to definition_cel_ast for clarity
skynetigor May 1, 2025
bf857b9
Simplify PropertyAccessNode
skynetigor May 2, 2025
4df4b3c
refactor: enhance correlation sidebar with custom AST to query builde…
skynetigor May 3, 2025
11865fb
feat: implement conversion from CelAst to QueryBuilder AST
skynetigor May 3, 2025
21a36e3
fix incorrect value for ComparisonNodeOperator.NE enum
skynetigor May 3, 2025
818cf03
Add unit tests for convertCelAstToQueryBuilderAst function
skynetigor May 3, 2025
c6b28c3
refactor: remove unused FormattedQueryCell component and clean up imp…
skynetigor May 3, 2025
8151e9c
refactor: remove unused MethodAccessNode import from cel_ast_converte…
skynetigor May 3, 2025
508641a
refactor: update PropertyAccessNode assertions to use path attribute
skynetigor May 3, 2025
e74b651
Merge branch 'main' into 4662-labels-namespace-filtering-in-rules
skynetigor May 3, 2025
e5936b8
refactor: simplify query builder logic by extracting node visit funct…
skynetigor May 3, 2025
81838df
refactor: define node_type attribute directly in AST node classes and…
skynetigor May 3, 2025
3fb0aa0
refactor: replace alertsFound.length with totalAlertsFound in AlertsF…
skynetigor May 3, 2025
092ef24
Merge branch '4662-labels-namespace-filtering-in-rules' of https://gi…
skynetigor May 3, 2025
721f940
refactor: improve alert fetching and validation logic in CorrelationS…
skynetigor May 4, 2025
ae8bda8
refactor: enhance operand and comparison node handling in PropertiesM…
skynetigor May 4, 2025
4e0ddee
refactor: streamline property access node handling in PropertiesMapper
skynetigor May 4, 2025
c0555be
refactor: synchronize editor value with props and enhance comparison …
skynetigor May 4, 2025
0e85794
Merge branch 'main' into 4662-labels-namespace-filtering-in-rules
skynetigor May 4, 2025
95ed696
Merge branch 'main' into 4662-labels-namespace-filtering-in-rules
shahargl May 4, 2025
9a85a4d
Merge branch '4662-labels-namespace-filtering-in-rules' into 4573-int…
skynetigor May 4, 2025
19245d1
refactor: replace MethodAccessNode with operator constants and clean …
skynetigor May 4, 2025
21d8ee4
refactor: remove MemberAccessNode and update related references to Pr…
skynetigor May 4, 2025
0411f5e
refactor: update data type references and improve error handling in p…
skynetigor May 4, 2025
3bef83e
fix: update data type for incident ID mapping to use DataType.UUID
skynetigor May 4, 2025
8009ca4
Merge branch '4662-labels-namespace-filtering-in-rules' into 4573-int…
skynetigor May 4, 2025
6242a5e
refactor: update type casting in SQL filter to use from_type_to_data_…
skynetigor May 4, 2025
ebf46f7
refactor: streamline operand node handling in CelAstRebuilder and upd…
skynetigor May 4, 2025
e816836
refactor: optimize handling of null comparisons in multiple fields node
skynetigor May 4, 2025
d796219
fix debugging
skynetigor May 4, 2025
729b188
refactor booleans
skynetigor May 4, 2025
ed24b5b
refactor: enhance logical node processing by adding boolean flag for …
skynetigor May 4, 2025
b8bc09a
feat: revert back CoalesceNode for handling coalesce operations in AST
skynetigor May 4, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 6 additions & 4 deletions keep-ui/app/(keep)/rules/CorrelationSidebar/AlertsFoundBadge.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -3,12 +3,14 @@ import { AlertDto } from "@/entities/alerts/model";
import { DynamicImageProviderIcon } from "@/components/ui";

type AlertsFoundBadgeProps = {
totalAlertsFound: number;
alertsFound: AlertDto[];
isLoading: boolean;
role: "ruleCondition" | "correlationRuleConditions";
};

export const AlertsFoundBadge = ({
totalAlertsFound,
alertsFound,
isLoading,
role,
Expand All @@ -17,15 +19,15 @@ export const AlertsFoundBadge = ({
if (role === "ruleCondition") {
return (
<>
{alertsFound.length} alert{alertsFound.length > 1 ? "s" : ""} were
found matching this condition
{totalAlertsFound} alert{totalAlertsFound > 1 ? "s" : ""} were found
matching this condition
</>
);
}

return (
<>
{alertsFound.length} alert{alertsFound.length > 1 ? "s" : ""} were found
{totalAlertsFound} alert{totalAlertsFound > 1 ? "s" : ""} were found
matching correlation rule conditions
</>
);
Expand All @@ -39,7 +41,7 @@ export const AlertsFoundBadge = ({
return "No alerts were found with these correlation rule conditions. Please try something else.";
}

if (alertsFound.length === 0) {
if (totalAlertsFound === 0) {
return (
<Badge className="mt-3 w-full" color="gray">
{isLoading ? "Getting your alerts..." : getNotFoundText()}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -10,13 +10,13 @@ import { Link } from "@/components/ui";
import { ArrowUpRightIcon } from "@heroicons/react/24/outline";
import { useRules } from "utils/hooks/useRules";
import { useRouter, useSearchParams } from "next/navigation";
import { useSearchAlerts } from "utils/hooks/useSearchAlerts";
import { AlertsFoundBadge } from "./AlertsFoundBadge";
import { useApi } from "@/shared/lib/hooks/useApi";
import { useConfig } from "@/utils/hooks/useConfig";
import { showErrorToast } from "@/shared/ui";
import { CorrelationFormType } from "./types";
import { TIMEFRAME_UNITS_TO_SECONDS } from "./timeframe-constants";
import { useMatchingAlerts } from "./useMatchingAlerts";

type CorrelationSidebarBodyProps = {
toggle: VoidFunction;
Expand Down Expand Up @@ -46,10 +46,11 @@ export const CorrelationSidebarBody = ({
const searchParams = useSearchParams();
const selectedId = searchParams ? searchParams.get("id") : null;

const { data: alertsFound = [], isLoading } = useSearchAlerts({
query: methods.watch("query"),
timeframe: timeframeInSeconds,
});
const {
data: alertsFound = [],
totalCount: totalAlertsFound,
isLoading,
} = useMatchingAlerts(methods.watch("query"));

const [isCalloutShown, setIsCalloutShown] = useLocalStorage(
"correlation-callout",
Expand Down Expand Up @@ -85,7 +86,7 @@ export const CorrelationSidebarBody = ({
celQuery: formatQuery(query, "cel"),
timeframeInSeconds,
timeUnit: timeUnit,
groupingCriteria: alertsFound.length ? groupedAttributes : [],
groupingCriteria: totalAlertsFound ? groupedAttributes : [],
requireApprove: requireApprove,
resolveOn: resolveOn,
createOn: createOn,
Expand Down Expand Up @@ -169,8 +170,9 @@ export const CorrelationSidebarBody = ({
<CorrelationGroups />
</div>
<div className="flex flex-col border-t-2">
{alertsFound.length > 0 && (
{totalAlertsFound > 0 && (
<AlertsFoundBadge
totalAlertsFound={totalAlertsFound}
alertsFound={alertsFound}
isLoading={false}
role={"correlationRuleConditions"}
Expand Down
26 changes: 17 additions & 9 deletions keep-ui/app/(keep)/rules/CorrelationSidebar/RuleFields.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -18,11 +18,11 @@ import {
} from "react-querybuilder";
import { AlertsFoundBadge } from "./AlertsFoundBadge";
import { useFormContext } from "react-hook-form";
import { useSearchAlerts } from "utils/hooks/useSearchAlerts";
import { CorrelationFormType } from "./types";
import { TIMEFRAME_UNITS_TO_SECONDS } from "./timeframe-constants";
import { useDeduplicationFields } from "@/utils/hooks/useDeduplicationRules";
import { get } from "lodash";
import { useMatchingAlerts } from "./useMatchingAlerts";

const DEFAULT_OPERATORS = defaultOperators.filter((operator) =>
[
Expand Down Expand Up @@ -50,7 +50,7 @@ const OPERATORS_FORCE_TYPE_CAST = {
"<=": "number",
"<": "number",
">": "number",
}
};

const DEFAULT_FIELDS: QueryField[] = [
{ name: "source", label: "source", datatype: "text" },
Expand Down Expand Up @@ -117,9 +117,13 @@ const Field = ({
};

const castValueToOperationType = (value: string) => {
const castTo: string = get(OPERATORS_FORCE_TYPE_CAST, ruleField.operator, "text");
const castTo: string = get(
OPERATORS_FORCE_TYPE_CAST,
ruleField.operator,
"text"
);
return castTo === "number" ? Number(value) : value;
}
};

return (
<div key={ruleField.id}>
Expand Down Expand Up @@ -158,7 +162,9 @@ const Field = ({
{isValueEnabled && (
<div>
<TextInput
onValueChange={(newValue) => onFieldChange("value", castValueToOperationType(newValue))}
onValueChange={(newValue) =>
onFieldChange("value", castValueToOperationType(newValue))
}
defaultValue={ruleField.value}
required
error={!ruleField.value}
Expand Down Expand Up @@ -279,10 +285,11 @@ export const RuleFields = ({
? TIMEFRAME_UNITS_TO_SECONDS[watch("timeUnit")](+watch("timeAmount"))
: 0;

const { data: alertsFound = [], isLoading } = useSearchAlerts({
query: { combinator: "and", rules: ruleFields },
timeframe: timeframeInSeconds,
});
const {
data: alertsFound = [],
totalCount: totalAlertsFound,
isLoading,
} = useMatchingAlerts({ combinator: "and", rules: ruleFields });

return (
<div key={rule.id} className="bg-gray-100 px-4 py-3 rounded space-y-2">
Expand Down Expand Up @@ -346,6 +353,7 @@ export const RuleFields = ({
</div>

<AlertsFoundBadge
totalAlertsFound={totalAlertsFound}
alertsFound={alertsFound}
isLoading={isLoading}
role={"ruleCondition"}
Expand Down
Loading
Loading