Skip to content

Commit 13b7d33

Browse files
karusselleasbar
karussell
and
easbar
authored
Java/Janino scripting to replace our interpreter (graphhopper#2209)
* basic test with janino passing * move janino scripting to safe whitelisted approach * create a class to add encoded values on the fly but limit user input to expression * compilation unit via overwriting getValue method safely * minor fix regarding ClassName * ensure it is not called twice * separate parsing of condition and number * CustomWeighting: ensure Map is ordered (graphhopper#2162) * CustomWeighting: ensure Map is ordered * minor * cleaner junit5 assertThrows * one more assertThrow * change expression from factor multiplication to if-then-elseif-else and make speed factor working too * replace catch all with true and make max_speed working * no need to parse value which is always a number * use global max speed * minor cleanup and fix regarding sharedEV * found a way to avoid Unparser.unparse * use scripting in CustomWeighting. This revealed: we need if and if-else-if clauses and maybe a different variable initialization * reduce max speed and fix priority bug * cosmetics * minor truck.yml tweaks * adapt benchmarks * a bit more details in exception * see perf with ugly caching hack * revert and see bench * workaround bug with import conflict (two imports for enum 'NO') * inject class for constants, avoid static imports that can easily collide * special case of RouteNetwork for bike * introduce thread-safe caching; minor restructuring * reduce code duplication in ScriptHelper * move code out of ScriptHelper and rename it to ExpressionBuilder * less recursion for more clear protection * further cleanup and improved naming * adapt CustomModel.merge to scripting and throw exception if query-CustomModel contains factors >1 * improved docs about custom model * adapted docs to new format * removed CustomWeightingOld and all related classes * minor fixes * make debugging possible; every dynamic class has a different name now; fix two issues in source code creation (RoadClass->EncodedValue and import JsonFeature) * improve caching: remove oldest accessed entry, instead oldest inserted * introduce cache to keep certain classes in cache forever * try perf with disabled cache to see CH numbers again * Revert "try perf with disabled cache to see CH numbers again" This reverts commit 85c9743. * make it possible to disable dyn cache; clarify about the static cache which is populated via checkProfilesConsistency * allow multiple first_match and make names of encoded values even stricter * non-shared EncodedValue is too complex due to '.' we should use '$' or '__' * use non-shared encoded values * test again without cache * call in-area method only when necessary, improves speed where pre-conditional checks are done like road_class!=PRIMARY && in_area_custom1 * make area names stricter * already consider parts of the pull request reviews * Custom weighting helper cleanup (graphhopper#2211) * Chain CustomWeighting constructors, use static creator method * CustomWeighting no longer depends on CustomWeightingHelper * Extract parameter object for CustomWeighting * Use SpeedAndAccessProvider interface, add ExpressionBuilderTest * Change apply -> get * More fine-grained usage of try statement * revert DEFAULT to true * docs: remove confusing statement * better name for subclass * no need for the extra line in test * simple max_speed_fallback handling, still explicit * migrate to if-then notation * fix benchmarks for if-then notation * made Clause methods shorter * introduce different Op * merge speed_factor and max_speed into speed via introduced 'multiply with' and 'limit to' operators; cleanup * further simplify custom model: remove max_speed_fallback * bug fix * docs: adapt to know notation * multiply with -> by * minor cleanup * support for StringEncodedValue * UI fixes for new format; make isSharedEncodedValues method private again * bug fix for findMaxSpeed calculation * test perf: disable cache for comparison * test perf: again * enable cache to permanently store internal Weighting classes * uhm, test perf: without cache but push separately * Revert "uhm, test perf: without cache but push separately" This reverts commit 6bb9fd0. * minor improvements * removed SpeedAndAccessProvider * make sure that the internal cache will never be too big * fix package of JaninoCustomWeightingHelperSubclass * rename ExpressionBuilder to CustomModelParser * made CustomModelParser parser public and move create method to it from CustomWeighting * comment regarding internal cache method * create -> createWeighting * move CustomModel to different package to hide useInternalCache methods * limit size of the whole script, not single expressions * Rename CustomModelParser#create etc. * minor * rename internal and throw exception if found in query * fix contentString * Improve profiles.md * Fix road_surface->surface and else: "" -> else: null * Explain how to use else statement in JSON * Fix GH maps example * Warn instead of error in case the edge distance of very long edges differs from way geometry length (cherry picked from commit dfee21c) * rename * error message e.g. in case of encoded value wasn't added * distanceInfluence: use default value only if not initialized * less cryptic error message Co-authored-by: Andi <[email protected]>
1 parent 5c02f7b commit 13b7d33

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

44 files changed

+2323
-2052
lines changed

‎README.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -213,7 +213,7 @@ Here is a list of the more detailed features:
213213
* [Alternative routes](https://discuss.graphhopper.com/t/alternative-routes/424)
214214
* [Turn costs and restrictions](../stable/docs/core/turn-restrictions.md)
215215
* Country specific routing via SpatialRules
216-
* The core uses only a few dependencies (hppc, jts and slf4j)
216+
* The core uses only a few dependencies (hppc, jts, janino and slf4j)
217217
* Scales from small indoor-sized to world-wide-sized graphs
218218
* Finds nearest point on street e.g. to get elevation or 'snap to road' or being used as spatial index (see [#1485](https://github.com/graphhopper/graphhopper/pull/1485))
219219
* Calculates isochrones and [shortest path trees](https://github.com/graphhopper/graphhopper/pull/1577)

‎api/src/main/java/com/graphhopper/json/Statement.java

Lines changed: 104 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,104 @@
1+
/*
2+
* Licensed to GraphHopper GmbH under one or more contributor
3+
* license agreements. See the NOTICE file distributed with this work for
4+
* additional information regarding copyright ownership.
5+
*
6+
* GraphHopper GmbH licenses this file to you under the Apache License,
7+
* Version 2.0 (the "License"); you may not use this file except in
8+
* compliance with the License. You may obtain a copy of the License at
9+
*
10+
* http://www.apache.org/licenses/LICENSE-2.0
11+
*
12+
* Unless required by applicable law or agreed to in writing, software
13+
* distributed under the License is distributed on an "AS IS" BASIS,
14+
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
15+
* See the License for the specific language governing permissions and
16+
* limitations under the License.
17+
*/
18+
package com.graphhopper.json;
19+
20+
public class Statement {
21+
private final Keyword keyword;
22+
private final String expression;
23+
private final Op operation;
24+
private final double value;
25+
26+
private Statement(Keyword keyword, String expression, Op operation, double value) {
27+
this.keyword = keyword;
28+
this.expression = expression;
29+
this.value = value;
30+
this.operation = operation;
31+
}
32+
33+
public Keyword getKeyword() {
34+
return keyword;
35+
}
36+
37+
public String getExpression() {
38+
return expression;
39+
}
40+
41+
public Op getOperation() {
42+
return operation;
43+
}
44+
45+
public double getValue() {
46+
return value;
47+
}
48+
49+
public enum Keyword {
50+
IF("if"), ELSEIF("else if"), ELSE("else");
51+
52+
String name;
53+
54+
Keyword(String name) {
55+
this.name = name;
56+
}
57+
58+
public String getName() {
59+
return name;
60+
}
61+
}
62+
63+
public enum Op {
64+
MULTIPLY("multiply by"), LIMIT("limit to");
65+
66+
String name;
67+
68+
Op(String name) {
69+
this.name = name;
70+
}
71+
72+
public String getName() {
73+
return name;
74+
}
75+
76+
public String build(double value) {
77+
switch (this) {
78+
case MULTIPLY:
79+
return "value *= " + value;
80+
case LIMIT:
81+
return "value = Math.min(value," + value + ")";
82+
default:
83+
throw new IllegalArgumentException();
84+
}
85+
}
86+
}
87+
88+
@Override
89+
public String toString() {
90+
return keyword.getName() + ": " + expression + ", " + operation.getName() + ": " + value;
91+
}
92+
93+
public static Statement If(String expression, Op op, double value) {
94+
return new Statement(Keyword.IF, expression, op, value);
95+
}
96+
97+
public static Statement ElseIf(String expression, Op op, double value) {
98+
return new Statement(Keyword.ELSEIF, expression, op, value);
99+
}
100+
101+
public static Statement Else(Op op, double value) {
102+
return new Statement(Keyword.ELSE, null, op, value);
103+
}
104+
}

‎api/src/main/java/com/graphhopper/util/CustomModel.java

Lines changed: 65 additions & 137 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,8 @@
1717
*/
1818
package com.graphhopper.util;
1919

20+
import com.graphhopper.json.Statement;
21+
2022
import java.util.*;
2123

2224
/**
@@ -26,32 +28,43 @@ public class CustomModel {
2628

2729
public static final String KEY = "custom_model";
2830

29-
static double DEFAULT_D_I = 70;
30-
// optional:
31-
private Double maxSpeedFallback;
32-
private Double headingPenalty = Parameters.Routing.DEFAULT_HEADING_PENALTY;
33-
// default value derived from the cost for time e.g. 25€/hour and for distance 0.5€/km, for trucks this is usually larger
34-
private double distanceInfluence = DEFAULT_D_I;
35-
private Map<String, Object> speedFactorMap = new LinkedHashMap<>();
36-
private Map<String, Object> maxSpeedMap = new LinkedHashMap<>();
37-
private Map<String, Object> priorityMap = new LinkedHashMap<>();
31+
// e.g. 70 means that the time costs are 25€/hour and for the distance 0.5€/km (for trucks this is usually larger)
32+
static double DEFAULT_DISTANCE_INFLUENCE = 70;
33+
private Double distanceInfluence;
34+
private double headingPenalty = Parameters.Routing.DEFAULT_HEADING_PENALTY;
35+
private boolean internal;
36+
private List<Statement> speedStatements = new ArrayList<>();
37+
private List<Statement> priorityStatements = new ArrayList<>();
3838
private Map<String, JsonFeature> areas = new HashMap<>();
3939

4040
public CustomModel() {
4141
}
4242

4343
public CustomModel(CustomModel toCopy) {
44-
this.maxSpeedFallback = toCopy.maxSpeedFallback;
4544
this.headingPenalty = toCopy.headingPenalty;
4645
this.distanceInfluence = toCopy.distanceInfluence;
46+
// do not copy "internal"
4747

48-
speedFactorMap = deepCopy(toCopy.getSpeedFactor());
49-
maxSpeedMap = deepCopy(toCopy.getMaxSpeed());
50-
priorityMap = deepCopy(toCopy.getPriority());
48+
speedStatements = deepCopy(toCopy.getSpeed());
49+
priorityStatements = deepCopy(toCopy.getPriority());
5150

5251
areas.putAll(toCopy.getAreas());
5352
}
5453

54+
/**
55+
* This method is for internal usage only! Parsing a CustomModel is expensive and so we cache the result, which is
56+
* especially important for fast landmark queries (hybrid mode). Now this method ensures that all server-side custom
57+
* models are cached in a special internal cache which does not remove seldom accessed entries.
58+
*/
59+
public CustomModel internal() {
60+
this.internal = true;
61+
return this;
62+
}
63+
64+
public boolean isInternal() {
65+
return internal;
66+
}
67+
5568
private <T> T deepCopy(T originalObject) {
5669
if (originalObject instanceof List) {
5770
List<Object> newList = new ArrayList<>(((List) originalObject).size());
@@ -72,25 +85,22 @@ private <T> T deepCopy(T originalObject) {
7285
}
7386
}
7487

75-
public Map<String, Object> getSpeedFactor() {
76-
return speedFactorMap;
88+
public List<Statement> getSpeed() {
89+
return speedStatements;
7790
}
7891

79-
public Map<String, Object> getMaxSpeed() {
80-
return maxSpeedMap;
81-
}
82-
83-
public CustomModel setMaxSpeedFallback(Double maxSpeedFallback) {
84-
this.maxSpeedFallback = maxSpeedFallback;
92+
public CustomModel addToSpeed(Statement st) {
93+
getSpeed().add(st);
8594
return this;
8695
}
8796

88-
public Double getMaxSpeedFallback() {
89-
return maxSpeedFallback;
97+
public List<Statement> getPriority() {
98+
return priorityStatements;
9099
}
91100

92-
public Map<String, Object> getPriority() {
93-
return priorityMap;
101+
public CustomModel addToPriority(Statement st) {
102+
getPriority().add(st);
103+
return this;
94104
}
95105

96106
public CustomModel setAreas(Map<String, JsonFeature> areas) {
@@ -108,7 +118,7 @@ public CustomModel setDistanceInfluence(double distanceFactor) {
108118
}
109119

110120
public double getDistanceInfluence() {
111-
return distanceInfluence;
121+
return distanceInfluence == null ? DEFAULT_DISTANCE_INFLUENCE : distanceInfluence;
112122
}
113123

114124
public void setHeadingPenalty(double headingPenalty) {
@@ -126,43 +136,38 @@ public String toString() {
126136

127137
private String createContentString() {
128138
// used to check against stored custom models, see #2026
129-
return "distanceInfluence=" + distanceInfluence + "|speedFactor=" + speedFactorMap + "|maxSpeed=" + maxSpeedMap +
130-
"|maxSpeedFallback=" + maxSpeedFallback + "|priorityMap=" + priorityMap + "|areas=" + areas;
139+
return "distanceInfluence=" + distanceInfluence + "|headingPenalty=" + headingPenalty
140+
+ "|speedStatements=" + speedStatements + "|priorityStatements=" + priorityStatements + "|areas=" + areas;
131141
}
132142

133143
/**
134144
* A new CustomModel is created from the baseModel merged with the specified queryModel.
135145
*/
136146
public static CustomModel merge(CustomModel baseModel, CustomModel queryModel) {
137-
// avoid changing the specified CustomModel via deep copy otherwise the server-side CustomModel would be modified (same problem if queryModel would be used as target)
147+
if (queryModel.isInternal())
148+
throw new IllegalArgumentException("CustomModel in query cannot be internal");
149+
150+
// avoid changing the specified CustomModel via deep copy otherwise the server-side CustomModel would be
151+
// modified (same problem if queryModel would be used as target)
138152
CustomModel mergedCM = new CustomModel(baseModel);
139-
if (queryModel.maxSpeedFallback != null) {
140-
if (mergedCM.maxSpeedFallback != null && mergedCM.maxSpeedFallback > queryModel.maxSpeedFallback)
141-
throw new IllegalArgumentException("CustomModel in query can only use max_speed_fallback bigger or equal to " + mergedCM.maxSpeedFallback);
142-
mergedCM.maxSpeedFallback = queryModel.maxSpeedFallback;
143-
}
144-
if (Math.abs(queryModel.distanceInfluence - CustomModel.DEFAULT_D_I) > 0.01) {
145-
if (mergedCM.distanceInfluence > queryModel.distanceInfluence)
146-
throw new IllegalArgumentException("CustomModel in query can only use distance_influence bigger or equal to " + mergedCM.distanceInfluence);
153+
// we only overwrite the distance influence if a non-default value was used
154+
if (queryModel.distanceInfluence != null) {
155+
if (queryModel.distanceInfluence < mergedCM.getDistanceInfluence())
156+
throw new IllegalArgumentException("CustomModel in query can only use " +
157+
"distance_influence bigger or equal to " + mergedCM.getDistanceInfluence() +
158+
", given: " + queryModel.distanceInfluence);
147159
mergedCM.distanceInfluence = queryModel.distanceInfluence;
148160
}
149161

150-
// example
151-
// max_speed: { road_class: { secondary : 0.4 } }
152-
// or
153-
// priority: { max_weight: { "<3.501": 0.7 } }
154-
for (Map.Entry<String, Object> queryEntry : queryModel.getMaxSpeed().entrySet()) {
155-
Object value = mergedCM.maxSpeedMap.get(queryEntry.getKey());
156-
applyChange(mergedCM.maxSpeedMap, value, queryEntry);
157-
}
158-
for (Map.Entry<String, Object> queryEntry : queryModel.getSpeedFactor().entrySet()) {
159-
Object value = mergedCM.speedFactorMap.get(queryEntry.getKey());
160-
applyChange(mergedCM.speedFactorMap, value, queryEntry);
161-
}
162-
for (Map.Entry<String, Object> queryEntry : queryModel.getPriority().entrySet()) {
163-
Object value = mergedCM.priorityMap.get(queryEntry.getKey());
164-
applyChange(mergedCM.priorityMap, value, queryEntry);
165-
}
162+
checkFirst(queryModel.getSpeed());
163+
checkFirst(queryModel.getPriority());
164+
165+
check(queryModel.getPriority());
166+
check(queryModel.getSpeed());
167+
168+
mergedCM.speedStatements.addAll(queryModel.getSpeed());
169+
mergedCM.priorityStatements.addAll(queryModel.getPriority());
170+
166171
for (Map.Entry<String, JsonFeature> entry : queryModel.getAreas().entrySet()) {
167172
if (mergedCM.areas.containsKey(entry.getKey()))
168173
throw new IllegalArgumentException("area " + entry.getKey() + " already exists");
@@ -172,92 +177,15 @@ public static CustomModel merge(CustomModel baseModel, CustomModel queryModel) {
172177
return mergedCM;
173178
}
174179

175-
private static void applyChange(Map<String, Object> mergedSuperMap,
176-
Object mergedObj, Map.Entry<String, Object> querySuperEntry) {
177-
if (mergedObj == null) {
178-
// no need for a merge
179-
mergedSuperMap.put(querySuperEntry.getKey(), querySuperEntry.getValue());
180-
return;
181-
}
182-
if (!(mergedObj instanceof Map))
183-
throw new IllegalArgumentException(querySuperEntry.getKey() + ": entry is not a map: " + mergedObj);
184-
Object queryObj = querySuperEntry.getValue();
185-
if (!(queryObj instanceof Map))
186-
throw new IllegalArgumentException(querySuperEntry.getKey() + ": query entry is not a map: " + queryObj);
187-
188-
Map<Object, Object> mergedMap = (Map) mergedObj;
189-
Map<Object, Object> queryMap = (Map) queryObj;
190-
for (Map.Entry queryEntry : queryMap.entrySet()) {
191-
if (queryEntry.getKey() == null || queryEntry.getKey().toString().isEmpty())
192-
throw new IllegalArgumentException(querySuperEntry.getKey() + ": key cannot be null or empty");
193-
String key = queryEntry.getKey().toString();
194-
if (isComparison(key))
195-
continue;
196-
197-
Object mergedValue = mergedMap.get(key);
198-
if (mergedValue == null) {
199-
mergedMap.put(key, queryEntry.getValue());
200-
} else if (multiply(queryEntry.getValue(), mergedValue) != null) {
201-
// existing value needs to be multiplied
202-
mergedMap.put(key, multiply(queryEntry.getValue(), mergedValue));
203-
} else {
204-
throw new IllegalArgumentException(querySuperEntry.getKey() + ": cannot merge value " + queryEntry.getValue() + " for key " + key + ", merged value: " + mergedValue);
205-
}
206-
}
207-
208-
// now special handling for comparison keys start e.g. <2 or >3.0, see testMergeComparisonKeys
209-
// this could be simplified if CustomModel would be already an abstract syntax tree :)
210-
List<String> queryComparisonKeys = getComparisonKeys(queryMap);
211-
if (queryComparisonKeys.isEmpty())
212-
return;
213-
if (queryComparisonKeys.size() > 1)
214-
throw new IllegalArgumentException(querySuperEntry.getKey() + ": entry in " + querySuperEntry.getValue() + " must not contain more than one key comparison but contained " + queryComparisonKeys);
215-
char opChar = queryComparisonKeys.get(0).charAt(0);
216-
List<String> mergedComparisonKeys = getComparisonKeys(mergedMap);
217-
if (mergedComparisonKeys.isEmpty()) {
218-
mergedMap.put(queryComparisonKeys.get(0), queryMap.get(queryComparisonKeys.get(0)));
219-
} else if (mergedComparisonKeys.get(0).charAt(0) == opChar) {
220-
if (multiply(queryMap.get(queryComparisonKeys.get(0)), mergedMap.get(mergedComparisonKeys.get(0))) != 0)
221-
throw new IllegalArgumentException(querySuperEntry.getKey() + ": currently only blocking comparisons are allowed, but query was " + queryMap.get(queryComparisonKeys.get(0)) + " and server side: " + mergedMap.get(mergedComparisonKeys.get(0)));
222-
223-
try {
224-
double comparisonMergedValue = Double.parseDouble(mergedComparisonKeys.get(0).substring(1));
225-
double comparisonQueryValue = Double.parseDouble(queryComparisonKeys.get(0).substring(1));
226-
if (opChar == '<') {
227-
if (comparisonMergedValue > comparisonQueryValue)
228-
throw new IllegalArgumentException(querySuperEntry.getKey() + ": only use a comparison key with a bigger value than " + comparisonMergedValue + " but was " + comparisonQueryValue);
229-
} else if (opChar == '>') {
230-
if (comparisonMergedValue < comparisonQueryValue)
231-
throw new IllegalArgumentException(querySuperEntry.getKey() + ": only use a comparison key with a smaller value than " + comparisonMergedValue + " but was " + comparisonQueryValue);
232-
} else {
233-
throw new IllegalArgumentException(querySuperEntry.getKey() + ": only use a comparison key with < or > as operator but was " + opChar);
234-
}
235-
mergedMap.remove(mergedComparisonKeys.get(0));
236-
mergedMap.put(queryComparisonKeys.get(0), queryMap.get(queryComparisonKeys.get(0)));
237-
} catch (NumberFormatException ex) {
238-
throw new IllegalArgumentException(querySuperEntry.getKey() + ": number in one of the 'comparison' keys for " + querySuperEntry.getKey() + " wasn't parsable: " + queryComparisonKeys + " (" + mergedComparisonKeys + ")");
239-
}
240-
} else {
241-
throw new IllegalArgumentException(querySuperEntry.getKey() + ": comparison keys must match but did not: " + queryComparisonKeys.get(0) + " vs " + mergedComparisonKeys.get(0));
242-
}
243-
}
244-
245-
static Double multiply(Object queryValue, Object mergedValue) {
246-
if (queryValue instanceof Number && mergedValue instanceof Number)
247-
return ((Number) queryValue).doubleValue() * ((Number) mergedValue).doubleValue();
248-
return null;
249-
}
250-
251-
static boolean isComparison(String key) {
252-
return key.startsWith("<") || key.startsWith(">");
180+
private static void checkFirst(List<Statement> priority) {
181+
if (!priority.isEmpty() && priority.get(0).getKeyword() != Statement.Keyword.IF)
182+
throw new IllegalArgumentException("First statement needs to be an if statement but was " + priority.get(0).getKeyword().getName());
253183
}
254184

255-
private static List<String> getComparisonKeys(Map<Object, Object> map) {
256-
List<String> list = new ArrayList<>();
257-
for (Map.Entry queryEntry : map.entrySet()) {
258-
String key = queryEntry.getKey().toString();
259-
if (isComparison(key)) list.add(key);
185+
private static void check(List<Statement> list) {
186+
for (Statement statement : list) {
187+
if (statement.getOperation() == Statement.Op.MULTIPLY && statement.getValue() > 1)
188+
throw new IllegalArgumentException("factor cannot be larger than 1 but was " + statement.getValue());
260189
}
261-
return list;
262190
}
263191
}

0 commit comments

Comments
 (0)