Skip to content

Commit fb86eba

Browse files
authored
Merge pull request #626 from jkloetzke/conditional-tool-env
Conditional environment defiitions
2 parents cc3897d + 4650e5c commit fb86eba

File tree

6 files changed

+343
-56
lines changed

6 files changed

+343
-56
lines changed

doc/manual/configuration.rst

Lines changed: 79 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -1315,7 +1315,9 @@ The following settings are supported:
13151315
| | | environment: |
13161316
| | | FOO: value |
13171317
| | | BAR: baz |
1318-
| | | BAZ: "${VAR}" |
1318+
| | | BAZ: |
1319+
| | | value: "${VAR}" |
1320+
| | | if: "${CONDITION}" |
13191321
| | | |
13201322
| | | Value strings in this clause are subject to |
13211323
| | | :ref:`configuration-principle-subst`. |
@@ -1388,16 +1390,33 @@ name requirement. Both packages are based on the identical recipe
13881390
environment
13891391
~~~~~~~~~~~
13901392

1391-
Type: Dictionary (String -> String)
1393+
Type::
1394+
1395+
{
1396+
str : str | {
1397+
"value" : str,
1398+
Optional("if") : str | IfExpression
1399+
}
1400+
}
13921401

13931402
Defines environment variables in the scope of the current recipe. Any inherited
13941403
variables of the downstream recipe with the same name are overwritten. All
13951404
variables are passed to upstream recipes.
13961405

1397-
Example::
1406+
The definition of a variable can optionally be guarded by an ``if`` condition.
1407+
Only if the ``if`` property evaluates to true, the variable is actually
1408+
defined. Might be a string or an IfExpression. See
1409+
:ref:`configuration-principle-booleans` for details about the evaluation.
13981410

1399-
environment:
1400-
PKG_VERSION: "1.2.3"
1411+
Examples::
1412+
1413+
environment:
1414+
PKG_VERSION: "1.2.3"
1415+
1416+
environment:
1417+
PKG_VERSION:
1418+
value: "1.2.3"
1419+
if: "$(eq,$FOO,bar)"
14011420

14021421
All environment keys are eligible to variable substitution. The environment of
14031422
the recipe and inherited classes are merged together. Suppose the project has
@@ -1609,7 +1628,14 @@ Not available on Windows.
16091628
metaEnvironment
16101629
~~~~~~~~~~~~~~~
16111630

1612-
Type: Dictionary (String -> String)
1631+
Type::
1632+
1633+
{
1634+
str : str | {
1635+
"value" : str,
1636+
Optional("if") : str | IfExpression
1637+
}
1638+
}
16131639

16141640
metaEnvironment variables behave like :ref:`configuration-recipes-privateenv` variables.
16151641
They overrule other environment variables and can be used in all steps. In addition all
@@ -1622,6 +1648,11 @@ All metaEnvironment variables are subject to :ref:`string substitution
16221648
<configuration-principle-subst>`, unless the :ref:`policies-substituteMetaEnv`
16231649
policy is configured for the old behaviour.
16241650

1651+
The definition of a metaEnvironment variable can optionally be guarded by an
1652+
``if`` condition. Only if the ``if`` property evaluates to true, the variable
1653+
is actually defined. Might be a string or an IfExpression. See
1654+
:ref:`configuration-principle-booleans` for details about the evaluation.
1655+
16251656
.. _configuration-recipes-multipackage:
16261657

16271658
multiPackage
@@ -1664,7 +1695,14 @@ header files and other needed files to link with this library.
16641695
privateEnvironment
16651696
~~~~~~~~~~~~~~~~~~
16661697

1667-
Type: Dictionary (String -> String)
1698+
Type::
1699+
1700+
{
1701+
str : str | {
1702+
"value" : str,
1703+
Optional("if") : str | IfExpression
1704+
}
1705+
}
16681706

16691707
Defines environment variables just for the current recipe. Any inherited
16701708
variables with the same name of the downstream recipe or others that were
@@ -1680,6 +1718,11 @@ The ``privateEnvironment`` of the recipe and inherited classes are merged
16801718
together. See :ref:`configuration-recipes-env` for the merge and string
16811719
substitution behaviour.
16821720

1721+
The definition of a variable can optionally be guarded by an ``if`` condition.
1722+
Only if the ``if`` property evaluates to true, the variable is actually
1723+
defined. Might be a string or an IfExpression. See
1724+
:ref:`configuration-principle-booleans` for details about the evaluation.
1725+
16831726
.. _configuration-recipes-providedeps:
16841727

16851728
provideDeps
@@ -1735,6 +1778,9 @@ consuming recipes. Example::
17351778
netAccess: True
17361779
environment:
17371780
CC: gcc
1781+
CXX:
1782+
value: g++
1783+
if: "$ENABLE_CPP"
17381784
LD: ld
17391785
fingerprintIf: True
17401786
fingerprintScript: |
@@ -1761,6 +1807,11 @@ recipe they must define distinct variables because no particular order between
17611807
tools is defined. The values defined in this attribute are subject to variable
17621808
substitution.
17631809

1810+
The definition of an ``environment`` variable can optionally be guarded by an
1811+
``if`` condition. Only if the ``if`` property evaluates to true, the variable
1812+
is actually defined. Might be a string or an IfExpression. See
1813+
:ref:`configuration-principle-booleans` for details about the evaluation.
1814+
17641815
The ``fingerprintScript`` attribute defines a fingerprint script like in a
17651816
normal recipe by :ref:`configuration-recipes-fingerprintScript`. A fingerprint
17661817
script defined by a tool is implicitly added to the fingerprint scripts of all
@@ -1780,7 +1831,14 @@ by giving the relative path directly::
17801831
provideVars
17811832
~~~~~~~~~~~
17821833

1783-
Type: Dictionary (String -> String)
1834+
Type::
1835+
1836+
{
1837+
str : str | {
1838+
"value" : str,
1839+
Optional("if") : str | IfExpression
1840+
}
1841+
}
17841842

17851843
Declares arbitrary environment variables with values that should be passed to
17861844
the downstream recipe. The values of the declared variables are subject to
@@ -1792,6 +1850,11 @@ package environment. Example::
17921850
CROSS_COMPILE: "arm-linux-${ABI}-"
17931851

17941852

1853+
The definition of a variable can optionally be guarded by an ``if`` condition.
1854+
Only if the ``if`` property evaluates to true, the variable is actually
1855+
defined. Might be a string or an IfExpression. See
1856+
:ref:`configuration-principle-booleans` for details about the evaluation.
1857+
17951858
By default these provided variables are not picked up by downstream recipes. This
17961859
must be declared explicitly by a ``use: [environment]`` attribute in the
17971860
dependency section of the downstream recipe. Only then are the provided variables
@@ -1842,6 +1905,11 @@ actually used (i.e. the parent recipe defined ``sandbox`` in the ``use``
18421905
section and the user builds with ``--sandbox``). In this case the variables
18431906
defined here have a higher precedence that the ones defined in ``provideVars``.
18441907

1908+
The definition of an ``environment`` variable can optionally be guarded by an
1909+
``if`` condition. Only if the ``if`` property evaluates to true, the variable
1910+
is actually defined. Might be a string or an IfExpression. See
1911+
:ref:`configuration-principle-booleans` for details about the evaluation.
1912+
18451913
Variable substitution is possible for the mount paths and environment
18461914
variables. See :ref:`configuration-principle-subst` for the available
18471915
substations. The mount paths are also subject to an additional variable
@@ -1865,6 +1933,9 @@ Example::
18651933
- ["\\$SSH_AUTH_SOCK", "\\SSH_AUTH_SOCK", [nofail, nojenkins]]
18661934
environment:
18671935
AUTOCONF_BUILD: "x86_64-linux-gnu"
1936+
ORIGINAL_ARCH:
1937+
value: "$AUTOCONF_BUILD"
1938+
if: "$(ne,$AUTOCONF_BUILD,x86_64-linux-gnu)"
18681939
user: nobody
18691940

18701941
The example assumes that the variable ``MYREPO`` was set somewhere in the

pym/bob/input.py

Lines changed: 30 additions & 25 deletions
Original file line numberDiff line numberDiff line change
@@ -566,8 +566,8 @@ def prepare(self, coreStepRef, env):
566566
"""Create concrete tool for given step."""
567567
path = env.substitute(self.path, "provideTools::path")
568568
libs = [ env.substitute(l, "provideTools::libs") for l in self.libs ]
569-
environment = { k : env.substitute(v, "provideTools::environment::"+k)
570-
for k, v in self.environment.items() }
569+
environment = env.substituteCondDict(self.environment,
570+
"provideTools::environment")
571571
return CoreTool(coreStepRef, path, libs, self.netAccess, environment,
572572
self.fingerprintScript, self.fingerprintIf,
573573
self.fingerprintVars)
@@ -698,10 +698,8 @@ def __init__(self, coreStep, env, enabled, spec):
698698
if (m[0] != "") and (m[1] != ""):
699699
self.mounts.append(m)
700700
self.mounts.extend(recipeSet.getSandboxMounts())
701-
self.environment = {
702-
k : env.substitute(v, "providedSandbox::environment")
703-
for (k, v) in spec.get('environment', {}).items()
704-
}
701+
self.environment = env.substituteCondDict(spec.get('environment', {}),
702+
"providedSandbox::environment")
705703
self.user = recipeSet.getSandboxUser() or spec.get('user', "nobody")
706704

707705
# Calculate a "resultId" so that only identical sandboxes match
@@ -1793,14 +1791,22 @@ def validate(self, data):
17931791
return LayerSpec(name, RecipeSet.LAYERS_SCM_SCHEMA.validate(_data)[0])
17941792

17951793
class VarDefineValidator:
1796-
def __init__(self, keyword):
1797-
self.__varName = re.compile(r'^[A-Za-z_][A-Za-z0-9_]*$')
1794+
VAR_NAME = re.compile(r'^[A-Za-z_][A-Za-z0-9_]*$')
1795+
VAR_DEF = schema.Schema({
1796+
'value' : str,
1797+
schema.Optional("if"): schema.Or(str, IfExpression),
1798+
})
1799+
1800+
def __init__(self, keyword, conditional=True):
17981801
self.__keyword = keyword
1802+
self.__conditional = conditional
17991803

18001804
def validate(self, data):
18011805
if not isinstance(data, dict):
18021806
raise schema.SchemaUnexpectedTypeError(
18031807
"{}: must be a dictionary".format(self.__keyword), None)
1808+
1809+
data = data.copy()
18041810
for key,value in sorted(data.items()):
18051811
if not isinstance(key, str):
18061812
raise schema.SchemaUnexpectedTypeError(
@@ -1812,13 +1818,19 @@ def validate(self, data):
18121818
"{}: bad variable '{}'. Environment variables starting with 'BOB_' are reserved!"
18131819
.format(self.__keyword, key),
18141820
None)
1815-
if self.__varName.match(key) is None:
1821+
if self.VAR_NAME.match(key) is None:
18161822
raise schema.SchemaWrongKeyError(
18171823
"{}: bad variable name '{}'.".format(self.__keyword, key),
18181824
None)
1819-
if not isinstance(value, str):
1825+
if isinstance(value, dict) and self.__conditional:
1826+
self.VAR_DEF.validate(value, error=f"{self.__keyword}: {key}: invalid definition!")
1827+
data[key] = (value['value'], value.get('if'))
1828+
elif isinstance(value, str):
1829+
if self.__conditional:
1830+
data[key] = (value, None)
1831+
else:
18201832
raise schema.SchemaUnexpectedTypeError(
1821-
"{}: bad variable '{}'. Environment variable values must be strings!"
1833+
"{}: {}: bad variable definition type."
18221834
.format(self.__keyword, key),
18231835
None)
18241836
return data
@@ -2378,8 +2390,7 @@ def prepare(self, inputEnv, sandboxEnabled, inputStates, inputSandbox=None,
23782390
"__tools" : tools })
23792391
env = inputEnv.derive()
23802392
for i in self.__varSelf:
2381-
env.update(( (key, env.substitute(value, "environment::"+key))
2382-
for key, value in i.items() ))
2393+
env.update(env.substituteCondDict(i, "environment"))
23832394
states = { n : s.copy() for (n,s) in inputStates.items() }
23842395

23852396
# update plugin states
@@ -2455,8 +2466,7 @@ def prepare(self, inputEnv, sandboxEnabled, inputStates, inputSandbox=None,
24552466

24562467
if dep.envOverride:
24572468
thisDepEnv = thisDepEnv.derive(
2458-
{ key : env.substitute(value, "depends["+realName+"].environment["+key+"]")
2459-
for key, value in dep.envOverride.items() })
2469+
env.substituteCondDict(dep.envOverride, "depends["+realName+"].environment"))
24602470

24612471
r = self.__recipeSet.getRecipe(recipeName)
24622472
try:
@@ -2597,15 +2607,13 @@ def prepare(self, inputEnv, sandboxEnabled, inputStates, inputSandbox=None,
25972607

25982608
# apply private environment
25992609
for i in self.__varPrivate:
2600-
env = env.derive({ key : env.substitute(value, "privateEnvironment::"+key)
2601-
for key, value in i.items() })
2610+
env.update(env.substituteCondDict(i, "privateEnvironment"))
26022611

26032612
# meta variables override existing variables
26042613
if self.__recipeSet.getPolicy('substituteMetaEnv'):
2605-
metaEnv = { key : env.substitute(value, "metaEnvironment::"+key)
2606-
for key, value in self.__metaEnv.items() }
2614+
metaEnv = env.substituteCondDict(self.__metaEnv, "metaEnvironment")
26072615
else:
2608-
metaEnv = self.__metaEnv
2616+
metaEnv = { k : v[0] for k, v in self.__metaEnv.items() if env.evaluate(v[1], k) }
26092617
env.update(metaEnv)
26102618

26112619
# set fixed built-in variables
@@ -2711,10 +2719,7 @@ def prepare(self, inputEnv, sandboxEnabled, inputStates, inputSandbox=None,
27112719
[CoreRef(buildCoreStep)], doFingerprint, toolDepPackage, toolDepPackageWeak)
27122720

27132721
# provide environment
2714-
provideEnv = {}
2715-
for (key, value) in self.__provideVars.items():
2716-
provideEnv[key] = env.substitute(value, "provideVars::"+key)
2717-
packageCoreStep.providedEnv = provideEnv
2722+
packageCoreStep.providedEnv = env.substituteCondDict(self.__provideVars, "provideVars")
27182723

27192724
# provide tools
27202725
packageCoreStep.providedTools = { name : tool.prepare(packageCoreStep, env)
@@ -3305,7 +3310,7 @@ def removeWhiteList(x):
33053310
lambda x: updateDicRecursive(self.__commandConfig, x) if not self._ignoreCmdConfig else None
33063311
),
33073312
"environment" : BuiltinSetting(
3308-
VarDefineValidator("environment"),
3313+
VarDefineValidator("environment", conditional=False),
33093314
lambda x: self.__defaultEnv.update(x)
33103315
),
33113316
"fallbackMirror" : BuiltinSetting(self.MIRRORS_SCHEMA, updateFallbackMirror, True),

pym/bob/stringparser.py

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -556,6 +556,14 @@ def evaluate(self, condition, prop):
556556
s = self.substitute(condition, "condition on "+prop)
557557
return not isFalse(s)
558558

559+
def substituteCondDict(self, values, prop, nounset=True):
560+
try:
561+
return { key : self.substitute(value, key, nounset)
562+
for key, (value, condition) in values.items()
563+
if self.evaluate(condition, key) }
564+
except ParseError as e:
565+
raise ParseError(f"{prop}: {e.slogan}")
566+
559567
def touchReset(self):
560568
self.touched = self.touched + [ set() ]
561569

test/unit/test_input_env.py

Lines changed: 31 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,7 @@
77
import schema
88

99
from bob.input import Env, VarDefineValidator
10+
from bob.errors import ParseError
1011

1112
class TestEnv(TestCase):
1213

@@ -66,18 +67,47 @@ def testTouch(self):
6667
e1.touch(['foo'])
6768
self.assertEqual(e1.touchedKeys(), set(['foo']))
6869

70+
def testSubstituteCondDictErrors(self):
71+
e = Env({"A" : "a"})
72+
73+
self.assertEqual(e.substituteCondDict({ "X" : ("$A", None)}, "prop"),
74+
{"X" : "a"})
75+
76+
with self.assertRaises(ParseError) as exc:
77+
e.substituteCondDict({ "X" : ("$B", None)}, "|prop|")
78+
self.assertIn("|prop|", exc.exception.slogan)
79+
80+
self.assertEqual(e.substituteCondDict({ "X" : ("$B", None)}, "prop", nounset=False),
81+
{"X" : ""})
82+
6983

7084
class TestVarDefineValidator(TestCase):
7185
def setUp(self):
7286
self.v = VarDefineValidator("foo")
7387

7488
def testValid(self):
75-
self.assertEqual(self.v.validate({"FOO": "bar"}), {"FOO": "bar"})
89+
self.assertEqual(self.v.validate({"FOO": "bar"}), {"FOO": ("bar", None)})
90+
self.assertEqual(
91+
self.v.validate({"FOO": {"value" : "bar"}}),
92+
{"FOO": ("bar", None)})
93+
self.assertEqual(
94+
self.v.validate({"FOO": {"value" : "bar", "if" : "condition"}}),
95+
{"FOO": ("bar", "condition")})
7696

7797
def testWrongTypes(self):
7898
self.assertRaises(schema.SchemaError, self.v.validate, "boom")
7999
self.assertRaises(schema.SchemaError, self.v.validate, {1 : "bar"})
80100
self.assertRaises(schema.SchemaError, self.v.validate, {"foo" : True})
101+
self.assertRaises(schema.SchemaError, self.v.validate,
102+
{"foo" : {}})
103+
self.assertRaises(schema.SchemaError, self.v.validate,
104+
{"foo" : []})
105+
self.assertRaises(schema.SchemaError, self.v.validate,
106+
{"foo" : {"value" : 1}})
107+
self.assertRaises(schema.SchemaError, self.v.validate,
108+
{"foo" : {"value" : "bar", "if" : 1}})
109+
self.assertRaises(schema.SchemaError, self.v.validate,
110+
{"foo" : {"value" : "bar", "wrong-key" : "baz"}})
81111

82112
def testWrongNames(self):
83113
self.assertRaises(schema.SchemaError, self.v.validate, {"0abc" : "bar"})

0 commit comments

Comments
 (0)