use pagination in use-latest-module-versions policy

Author: rberlind

governance/third-generation/cloud-agnostic/http-examples/README.md

@@ -3,8 +3,6 @@ This directory contains examples of using the [HTTP import](https://docs.hashico
 
 Be sure to use Sentinel 0.15.2 or higher with these policies.
 
-These policies are essentially the same as the second-generation versions in this [directory](../../../second-generation/cloud-agnostic/http-examples), but the [use-latest-module-versions.sentinel](./use-latest-module-versions.sentinel) in this directory uses the [tfconfig/v2](https://www.terraform.io/docs/cloud/sentinel/import/tfconfig-v2.html) import instead of the older [tfconfig](https://www.terraform.io/docs/cloud/sentinel/import/tfconfig.html) import. It uses that import indirectly by calling some functions from the [tfconfig-functions](../../common-functions/tfconfig-functions) Sentinel module.
-
 ## Policies
 There are currently three example policies in this directory:
 * [check-external-http-api.sentinel](./check-external-http-api.sentinel)
@@ -20,12 +18,17 @@ sentinel test -run=check -verbose
 
 The second policy uses the HTTP import to call the Terraform Registry [List Modules API](https://www.terraform.io/docs/registry/api.html#list-modules) against a Terraform Cloud or Terraform Enterprise server in order to determine the most recent version of each module in the [Private Module Registry](https://www.terraform.io/docs/cloud/registry/index.html) (PMR) of an organization on that server or in the [public Terraform registry](https://registry.terraform.io). It then checks that the version constraints used in module calls allow the most recent version. This policy also uses parameters as described below.
 
+Since 9/6/2021, the second policy can retrieve versions of all private modules in a PMR since it uses pagination to keep calling the List Modules API. However, the policy limits the number of modules retrieved from the public registry to 100.
+
+This policy currently uses the `/api/registry/v1/modules` endpoint for private registries rather than the newer `/organizations/:organization_name/registry-modules` endpoint that can get both private and publicly curated modules. Note that publically curated modules are not available in TFE. The `/organizations/:organization_name/registry-modules` API endpoint is available in TFE since version v202106-1. We expect to create a version of this policy that will use the new API endpoint but we will keep this policy so that customers on older versions of TFE can still use it.
+
 The third policy uses the HTTP import to call a [NASA API](https://api.nasa.gov/) that retrieves a list of Near Earth Objects and warns if any of them are too close for comfort. This is based on an example from this HashiCorp [blog](https://www.hashicorp.com/blog/announcing-business-aware-policies-for-terraform-cloud-and-enterprise/) that announced the HTTP import and "Business-aware Policies". This policy also uses parameters as described below.
 
 ## Use of Parameters in use-latest-module-versions.sentinel
-The [use-latest-module-versions.sentinel](./use-latest-module-versions.sentinel) policy uses four parameters:
+The [use-latest-module-versions.sentinel](./use-latest-module-versions.sentinel) policy uses five parameters:
 * `public_registry` indicates whether the public Terraform registry is being used.  This is `false` by default, but could be set to `true`.
 * `address` gives the address of the Terraform Cloud or Terraform Enterprise server.  It defaults to `app.terraform.io` which is the address of the multi-tenant Terraform Cloud server that HashiCorp runs. You must specify a value for this if using a Terraform Enterprise server.
+* `limit` gives the maximum number of modules to retrieve in a single call to the List Modules API endpoint. It defaults to `100` which is the maximum value that can be set.
 * `organization` gives the name of an [organization](https://www.terraform.io/docs/cloud/users-teams-organizations/organizations.html) on the Terraform Cloud or Terraform Enterprise server specified by `address`. You must always specify a valid organization.
 * `token` gives a valid Terraform Cloud API token which can be a user, team, or organization token. See the [API tokens](https://www.terraform.io/docs/cloud/users-teams-organizations/api-tokens.html) document for more information.
 

governance/third-generation/cloud-agnostic/http-examples/use-latest-module-versions.hcl

@@ -6,6 +6,10 @@ param "address" {
   value = "registry.terraform.io"
 }
 
+param "limit" {
+  value = 100
+}
+
 param "organization" {
   value = "Azure"
 }

governance/third-generation/cloud-agnostic/http-examples/use-latest-module-versions.sentinel

@@ -11,6 +11,20 @@
 
 # Additionally, this policy uses Sentinel parameters
 
+# The policy supports pagination to get unlimited number of modules from private
+# registries. Since the public registry has a very large number of modules,
+# the policy does not use pagination for it and limits the total number of
+# modules to 100.
+
+# This policy currently uses the /api/registry/v1/modules endpoint for private
+# registries rather than the newer /organizations/:organization_name/registry-modules
+# endpoint that can get both private and publicly curated modules. Note that
+# publically curated modules are not available in TFE. The
+# /organizations/:organization_name/registry-modules endpoint is available
+# in TFE since version v202106-1. We expect to create a version of this policy
+# that will use the new API endpoint but we will keep this policy so that
+# customers on older versions of TFE can still use it.
+
 ##### Imports #####
 
 # Import common-functions/tfconfig-functions/tfconfig-functions.sentinel
@@ -28,6 +42,9 @@ import "version"
 param public_registry default false
 # The address of the Terraform Cloud or Terraform Enterprise server
 param address default "app.terraform.io"
+# The limit to use when retrieving modules from a registry
+# This cannot be greater than 100
+param limit default 100
 # The name of the Terraform Cloud or Terraform Enterprise organization
 param organization
 # A valid Terraform Cloud or Terraform Enterprise API token
@@ -36,21 +53,34 @@ param token
 ##### Functions #####
 
 # Retrieve modules from a module registry and give their paths and latest versions
-retrieve_latest_module_versions = func(public_registry, address, organization, token) {
+retrieve_latest_module_versions = func() {
+
+  modules = {}
 
   # Call the TFC Modules API and extract the response
+  # Treat public registry and private registry differently
   if public_registry {
-    # We are limiting the request to 20 verified modules for the
-    # namespace in the public registry matching the organzation parameter
-    req = http.request("https://" + address + "/v1/modules/"  +
-                       organization + "?limit=20&verified=true")
+    modules = get_public_registry_modules()
   } else {
-    req = http.request("https://" + address + "/api/registry/v1/modules/"  +
-                       organization)
-    req = req.with_header("Authorization", "Bearer " + token)
+    # private registry
+    # Pass the empty modules map and offset=0
+    modules = get_private_registry_modules(modules, 0)
   }
+
+  # modules is indexed by <name>/<provider> and contains most recent version
+  return modules
+}
+
+# Get up to `limit` verified modules from the public registry for a namespace
+# But pass the organization parameter
+get_public_registry_modules = func() {
+  req = http.request("https://" + address + "/v1/modules/"  +
+                     namespace + "?limit=" + limit + "&verified=true")
+
+  # Call TFE API to get modules and unmarshal results
   res = json.unmarshal(http.get(req).body)
 
+  # Initialize modules map
   modules = {}
 
   # Iterate over the modules and extract names, providers, and latest versions
@@ -59,18 +89,41 @@ retrieve_latest_module_versions = func(public_registry, address, organization, t
     modules[index] = m.version
   }
 
-  # modules is indexed by <name>/<provider> and contains most recent version
+  return modules
+}
+
+# Get all modules from the private registry by calling itself recursively
+# Start by calling with modules = {}, and offset = 0
+get_private_registry_modules = func(modules, offset) {
+  req = http.request("https://" + address + "/api/registry/v1/modules/"  +
+                    organization + "?limit=" + string(limit) +
+                    "&offset=" + string(offset))
+  req = req.with_header("Authorization", "Bearer " + token)
+
+  # Call TFE API to get modules and unmarshal results
+  res = json.unmarshal(http.get(req).body)
+
+  # Iterate over the modules and extract names, providers, and latest versions
+  for res.modules as m {
+    index = m.namespace + "/" + m.name + "/" + m.provider
+    modules[index] = m.version
+  }
+
+  # Make recursive function call if there are more modules
+  if res.meta.next_offset else 0 > 0 {
+    return get_private_registry_modules(modules, res.meta.next_offset)
+  }
+
   return modules
 }
 
 # Validate sources of modules in the registry
-validate_modules = func(public_registry, address, organization, token) {
+validate_modules = func() {
 
   validated = true
 
   # Get latest module versions from registry
-  discovered_modules = retrieve_latest_module_versions(public_registry, address,
-                       organization, token)
+  discovered_modules = retrieve_latest_module_versions()
 
   # Get all module addresses
   allModuleAddresses = config.find_descendant_modules("")
@@ -128,7 +181,7 @@ validate_modules = func(public_registry, address, organization, token) {
 ##### Rules #####
 
 # Call the validation function
-modules_validated = validate_modules(public_registry, address, organization, token)
+modules_validated = validate_modules()
 
 # Main rule
 main = rule {