Merge pull request hashicorp#208 from hashicorp/third-gen-sentinel-function-docs

Third gen sentinel function docs

Author: rberlind

governance/README.md

@@ -2,6 +2,6 @@
 
 Sentinel gives operations teams the governance capabilities they need to ensure that all infrastructure provisioned with Terraform Enterprise complies with their organization's provisioning rules. The files under this directory and its sub-directories provide some sample Sentinel policies for AWS, Microsoft Azure, Google Cloud Platform (GCP), and VMware as well as some cloud-agnostic policies.
 
-The policies are grouped into [first-generation](./first-generation) and [second-generation](./second-generation) directories. The first-generation policies were created in 2018 while the second-generation policies were created in 2019. We encourage users to use the second-generation policies and to model new policies on them.
+The policies are grouped into [first-generation](./first-generation), [second-generation](./second-generation), and [third-generation](./third-generation) directories. The first-generation policies were created in 2018. The second-generation policies were created in 2019 and made significant [improvements](./second-generation/README.md#improvements) including better information for resources that violated policies. Both the first-generation and the second-generation policies used the original v1 versions of the tfplan, tfstate, and tfconfig imports. The third-generation policies were created in 2020 and use the v2 versions of those imports. The third-generation policies also use [Sentinel Modules](https://docs.hashicorp.com/sentinel/extending/modules/) which allow the policies to be much shorter than the second-generation policies. You can read more about the important characterizations of the third-generation policies [here](./second-generation/README.md#important-characterizations-of-the-new-policies).
 
-If you would like to see an end-to-end process for managing Sentinel policies and policy sets with version control and Terraform Enterprise, see the [hashicorp/tfe-policies-example](https://github.com/hashicorp/tfe-policies-example) repository.
+Note, however, that the v2 imports can only be used with Terraform 0.12 and higher. We encourage users who are only enforcing policies against Terraform 0.12 and higher to use the third-generation policies and to model new policies on them. But if you still need to enforce policies against Terraform 0.11 configurations, then you should use the second-generation policies.

governance/third-generation/README.md

@@ -1,5 +1,4 @@
 # Third-Generation Sentinel Policies
-
 This directory and its sub-directories contain third-generation Sentinel policies and associated [Sentinel CLI](https://docs.hashicorp.com/sentinel/intro/getting-started/install) test cases and mocks which were created in 2020 for AWS, Microsoft Azure, Google Cloud Platform (GCP), and VMware. It also contains some some common, re-usable functions.
 
 Additionally, it contains [Policy Set](https://www.terraform.io/docs/cloud/sentinel/manage-policies.html#the-sentinel-hcl-configuration-file) configuration files so that the cloud-specific and cloud-agnostic policies can easily be added to Terraform Cloud organizations using [VCS Integrations](https://www.terraform.io/docs/cloud/vcs/index.html) after forking this repository.
@@ -33,25 +32,27 @@ These new third-generation policies have several important characteristics:
 1. The common function `evaluate_attribute`, which is in the tfplan-functions.sentinel and tfstate-functions.sentinel modules, can evaluate the values of any attribute of any resource even if it is deeply nested inside the resource. It does this by calling itself recursively.
 
 ## Common Functions
-You can find most of the functions used in the third-generation policies in the Sentinel modules in the [common functions](./common-functions) directory:
-  * [tfplan-functions.sentinel](./common-functions/tfplan-functions.sentinel)
-  * [tfstate-functions.sentinel](./common-functions/tfstate-functions.sentinel)
-  * [tfconfig-functions.sentinel](./common-functions/tfconfig-functions.sentinel)
-  * [tfrun-functions.sentinel](./common-functions/tfrun-functions.sentinel)
+You can find most of the common functions used in the third-generation policies in the Sentinel modules in the [common functions](./common-functions) directory:
+  * [tfplan-functions](./common-functions/tfplan-functions)
+  * [tfstate-functions](./common-functions/tfstate-functions)
+  * [tfconfig-functions](./common-functions/tfconfig-functions)
+  * [tfrun-functions](./common-functions/tfrun-functions)
 
-There are also some functions used to validate assumed roles for the AWS provider in [aws-functions.sentinel](./aws/aws-functions/aws-functions.sentinel).
+There are also some functions used to validate assumed roles for the AWS provider in [aws-functions](./aws/aws-functions).
 
-Unlike the second-generation common functions that were each defined in a separate file, all of the common functions that use any of the 4 Terraform Sentinel imports (tfplan/v2, tfstate/v2, tfconfig/v2, and tfrun) are defined in a single file. This makes it easier to import all of the functions that use one of those imports into the Sentinel CLI test cases, since those only need a single stanza such as this one for each module:
+Unlike the second-generation common functions that were each defined in a separate file, all of the common functions that use any of the 4 Terraform Sentinel imports (tfplan/v2, tfstate/v2, tfconfig/v2, and tfrun) are defined in a single file. This makes it easier to import all of the functions that use one of those imports into the Sentinel CLI test cases and Terraform Cloud policy sets, since those only need a single stanza such as this one for each module:
 ```
 "modules": {
   "tfplan-functions": {
-    "path": "../../../common-functions/tfplan-functions.sentinel"
+    "path": "../../../common-functions/tfplan-functions/tfplan-functions.sentinel"
   }
 }
 ```
-Test cases that use the other modules would either change both occurrences of "tfplan" in that stanza to "tfstate", "tfconfig", or "tfrun" or would add additional stanzas with those changes.
+Test cases that use the other modules would either change all three occurrences of "tfplan" in that stanza to "tfstate", "tfconfig", or "tfrun" or would add additional stanzas with those changes.
+
+We have put each Sentinel module in its own directory which also contains Markdown files for each of the module's functions under a docs directory. Each of these Markdown files describes the function, its declaration, its arguments, other common functions it uses, what it returns, and what it prints. It also gives examples of calling the function and sometimes lists some policies that call it.
 
-While having multiple functions in a single file and module does make examining the function code a bit harder, we think the reduced work associated with referencing the functions in the test cases justifies this.
+While having multiple Sentinel functions in a single file does make examining the function code a bit harder, we think the reduced work associated with referencing the functions in the test cases and policy sets justifies this.
 
 To use any of the functions in a new policy, be sure to include lines like these:
 ```
@@ -67,7 +68,7 @@ In this case, we are using `plan`, `state`, `config`, `run`, and `aws` as aliase
 We discuss these two modules together because they are essentially identical except for their use of the tfplan/v2 and tfstate/v2 imports.
 
 Each of these modules has several types of functions:
-  * `find_resources` and `find_datasources` functions that find resources or data sources of a specific type that are being created or updated.
+  * `find_resources` and `find_datasources` functions that find resources or data sources of a specific type that are being created or updated. Note that the tfplan versions of these functions only find resources that are being created or changed and data sources that are being created, changed, or read.
   * `find_resources_being_destroyed` and `find_datasources_being_destroyed` function that find resources or data sources that are being destroyed but not re-created.
   * The `find_blocks` function finds all blocks of a specific type in a single resource.
   * `filter_*` functions that filter a collection of resources, data sources, or blocks to a sub-collection that violates some condition. (When we say resources below, we are including data sources which are really just read-only resources.) The filter functions all accept a collection of resource changes (for tfplan/v2) or resources (for tfstate/v2), an attribute, a value or a list of values, and a boolean, `prtmsg`, which can be `true` or `false` and indicates whether the filter function should print violation messages. The filter functions return a map consisting of 2 items:
@@ -78,6 +79,10 @@ Each of these modules has several types of functions:
   * The `to_string` function which can convert any Sentinel object to a string. It is used to build the messages in the `messages` collection returned by the filter functions.
   * The `print_violations` function which can be called after calling one of the filter function to print the violation messages. This would only be called if the `prtmsg` argument had been set to `false` when calling the filter function. This is sometimes desirable especially if processing blocks of resources since your policy can then print some other message that gives the address of the resource with block-level violations before printing them.
 
+Documentation for each individual function can be found in these directories:
+  * [tfplan-functions](./common-functions/tfplan-functions/docs)
+  * [tfstate-functions](./common-functions/tfstate-functions/docs)
+
 ### The Functions of the tfconfig-functions Module
 The `tfconfig-functions` module has several types of functions:
   * `find_all_*` functions find all resources, data sources, provisioners, providers, variables, outputs, and module calls of all types.
@@ -89,19 +94,28 @@ The `tfconfig-functions` module has several types of functions:
   * Two filter functions, `filter_attribute_not_in_list` and `filter_attribute_in_list`, that are similar to the filter functions in the tfplan-functions module. However, these can only be used against top-level attributes of the items in the collection passed to them. Those collections can be any type of entity covered by the tfconfig/v2 import including resources, data sources, providers, provisioners, variables, outputs, and module calls.
   * The same `to_string` and `print_violations` functions that are in the tfplan-functions module.
 
+Documentation for each individual function can be found in this directory:
+  * [tfconfig-functions](./common-functions/tfconfig-functions/docs)
+
 ### The Functions of the tfrun-functions Module
 The `tfrun-functions` module has the following functions:
   * The `limit_proposed_monthly_cost` function validates that the proposed monthly cost estimate is less than the given limit.
   * The `limit_cost_and_percentage_increase` function validates that the proposed monthly cost estimate and percentage increase over the previous cost estimate ar both less than limits.
   * The `limit_cost_by_workspace_name` function validates that the monthly cost estimate is less than the limit in a map associated with a workspace name prefix or suffix that the current workspace has.
 
+Documentation for each individual function can be found in this directory:
+  * [tfrun-functions](./common-functions/tfrun-functions/docs)
+
 ### The Functions of the aws-functions Module
 The `aws-functions` module (which is located under in the aws/aws-functions directory) has the following functions:
   * The `determine_role_arn` function determines the ARN of a role set in the `role_arn` parameter of an AWS provider. It can only determine the role_arn if it is set to either a hard-coded value or to a reference to a single Terraform variable. It sets the role to "complex" if it finds a single non-variable reference or if it finds multiple references. It sets the role to "none" if no role arn is found.
   * The `get_assumed_roles` function gets all roles assumed by AWS providers in the current Terraform configuration. It calls the `determine_role_arn` function.
   * The `validate_assumed_roles_with_list` function validates assumed roles found by the `get_assumed_roles` function against a list of role ARNs.
   * The `validate_assumed_roles_with_map` function validates assumed roles found by the `get_assumed_roles` function against a map of role ARNs which are associated with regular expressions representing workspace names that are allowed to use them.
 
+Documentation for each individual function can be found in this directory:
+  * [aws-functions](./aws/aws-functions/docs)
+
 ## Mock Files and Test Cases
 Sentinel [mock files](https://www.terraform.io/docs/enterprise/sentinel/mock.html) and [test cases](https://docs.hashicorp.com/sentinel/commands/config#test-cases) have been provided under the test directory of each cloud so that all the policies can be tested with the [Sentinel CLI](https://docs.hashicorp.com/sentinel/commands). The mocks were generated from actual Terraform 0.12 plans run against Terraform code that provisioned resources in these clouds. The pass and fail mock files were edited to respectively pass and fail the associated Sentinel policies. Some policies, including those that have multiple rules, have multiple fail mock files with names that indicate which condition or conditions they fail.
 

governance/third-generation/aws/aws-functions/docs/determine_role_arn.md

@@ -0,0 +1,34 @@
+# determine_role_arn
+This function determines the ARN of an AWS IAM role assumed by the Terraform AWS provider using the [tfconfig/v2](https://www.terraform.io/docs/cloud/sentinel/import/tfconfig-v2.html) and [tfplan/v2](https://www.terraform.io/docs/cloud/sentinel/import/tfplan-v2.html) imports.
+
+It can only do this when the `role_arn` of the AWS provider is set to a hard-coded string or to a variable within the Terraform configuration. In the second case, the function cross-references the name of the variable in the tfconfig/v2 import with the actual value assigned to it in the tfplan/v2 import.
+
+## Sentinel Module
+This function is contained in the [aws-functions.sentinel](../aws-functions.sentinel) module.
+
+## Declaration
+`determine_role_arn = func(address, data)`
+
+## Arguments
+* **address**: the address of the provider which has the form `module_address:provider.alias`.
+* **data**: the data associated with the provider.
+
+## Common Functions Used
+None
+
+## What It Returns
+This function returns the ARN of the AWS IAM role of the provider. However, if no role was specified for an instance of the AWS provider, it returns "none", and if it finds finds a single non-variable reference or multiple references, it returns "complex".
+
+## What It Prints
+This function prints warning messages if the `role_arn` attribute was not a hard-coded string or a reference to a single variable.
+
+## Examples
+Here is an example of calling this function, assuming that the [tfconfig-functions.sentinel](../../../common-functions/tfconfig-functions/tfconfig-functions.sentinel) module has been imported with the alias `config`:
+```
+aws_providers = config.find_providers_by_type("aws")
+for aws_providers as address, data {
+  assumed_roles[address] = determine_role_arn(address, data)
+}
+```
+
+This function is called by the `get_assumed_roles` function contained in the same Sentinel module. In fact, the above code is extracted from that function. Since the two functions are in the same module, the call to the `get_assumed_roles` function does not include the `aws.` prefix.

governance/third-generation/aws/aws-functions/docs/get_assumed_roles.md

@@ -0,0 +1,30 @@
+# get_assumed_roles
+This function gets all roles assumed by any instances of the AWS provider in the current Terraform configuration using the [tfconfig/v2](https://www.terraform.io/docs/cloud/sentinel/import/tfconfig-v2.html) and [tfplan/v2](https://www.terraform.io/docs/cloud/sentinel/import/tfplan-v2.html) imports.
+
+The tfplan/v2 import is used by the `determine_role_arn` function that this function calls.
+
+## Sentinel Module
+This function is contained in the [aws-functions.sentinel](../aws-functions.sentinel) module.
+
+## Declaration
+`get_assumed_roles = func()`
+
+## Arguments
+None
+
+## Common Functions Used
+This function calls the `find_providers_by_type` function of the [tfconfig-functions.sentinel](../../../common-functions/tfconfig-functions/tfconfig-functions.sentinel) module and the `determine_role_arn` function of the [aws-functions.sentinel](../aws-functions.sentinel) module.
+
+## What It Returns
+This function returns a map indexed by the addresses of the provider instances contained in the workspace's Terraform configuration with values set to the actual ARNs of the AWS IAM roles assumed by those providers. However, if no role was specified for an instance of the AWS provider, it returns "none" for that instance, and if it finds finds a single non-variable reference or multiple references for an instance, it returns "complex" for that instance.
+
+## What It Prints
+This function does not print anything itself, but the `determine_role_arn` function prints warning messages if the `role_arn` attribute was not a hard-coded string or a reference to a single variable.
+
+## Examples
+Here is an example of calling this function:
+```
+assumed_roles = get_assumed_roles()
+```
+
+This function is called by the `validate_assumed_roles_with_list` and `validate_assumed_roles_with_map` functions contained in the same Sentinel module. Since the three functions are in the same module, the call to the `get_assumed_roles` function does not include the `aws.` prefix.

governance/third-generation/aws/aws-functions/docs/validate_assumed_roles_with_list.md

@@ -0,0 +1,31 @@
+# validate_assumed_roles_with_list
+This function checks whether all roles assumed by all instances of the AWS provider in the current Terraform configuration are in a specified list.
+
+## Sentinel Module
+This function is contained in the [aws-functions.sentinel](../aws-functions.sentinel) module.
+
+## Declaration
+`validate_assumed_roles_with_list = func(allowed_roles)`
+
+## Arguments
+* **allowed_roles**: a list of allowed AWS IAM role ARNs that can be assumed by AWS providers. If you want to a policy that calls this function to pass if a role assumed by an instance of the AWS provider contains a single non-variable reference or multiple references, include "complex" in the list.
+
+## Common Functions Used
+This function calls the the `get_assumed_roles` function of the [aws-functions.sentinel](../aws-functions.sentinel) module.
+
+## What It Returns
+This function returns `true` if all the AWS IAM roles assumed by AWS providers in the Terraform configuration of the current workspace were in the `allowed_roles` list. Otherwise, it returns `false`.
+
+## What It Prints
+This function prints a warning message for each role assumed by an AWS provider that is not in the `allowed_roles` list.
+
+## Examples
+Here is an example of calling this function, assuming that the aws-functions.sentinel file that contains it has been imported with the alias `aws`:
+```
+allowed_roles = [
+  "arn:aws:iam::123412341234:role/terraform-assumed-role",
+]
+roles_validated = aws.validate_assumed_roles_with_list(allowed_roles)
+```
+
+This function is used by the [restrict-assumed-roles.sentinel](../../restrict-assumed-roles.sentinel) policy.