Merge pull request hashicorp#81 from hashicorp/openshift-3.11

openshift 3.11

Author: vincentramirez

infrastructure-as-code/k8s-cluster-openshift-aws/README.md

@@ -1,12 +1,12 @@
 # Openshift Cluster in AWS
-This guide provisions an OpenShift Origin 3.7 cluster in AWS with 1 master node, 1 client node, and 1 bastion host. It uses ansible-playbook to deploy OpenShift to the master and client nodes from the bastion host after using Terraform to provision the AWS infrastructure. It is based on a [terraform-aws-openshift](https://github.com/dwmkerr/terraform-aws-openshift) repository created by Dave Kerr.
+This guide provisions an OpenShift Origin 3.11 cluster in AWS with 1 master node, 1 client node, and 1 bastion host. It uses ansible-playbook to deploy OpenShift to the master and client nodes from the bastion host after using Terraform to provision the AWS infrastructure. It is based on a [terraform-aws-openshift](https://github.com/dwmkerr/terraform-aws-openshift) repository created by Dave Kerr.
 
 While the original repository required the user to manually run ansible-playbook after provisioning the AWS infrastructure with Terraform, this guide uses a Terraform [remote-exec provisioner](https://www.terraform.io/docs/provisioners/remote-exec.html) to do that. It also uses several additional remote-exec and local-exec provisioners to automate the rest of the deployment, retrieve the OpenShift cluster keys, and write them to outputs. This is important since it allows workspaces that deploy pods and services to the cluster do that via workspace state sharing without any manual copying of the cluster keys.
 
 ## Reference Material
 * [OpenShift Origin](https://www.openshift.org/): the open source version of OpenShift, Red Hat's commercial implementation of Kubernetes.
 * [Kubernetes](https://kubernetes.io/): the open source system for automating deployment and management of containerized applications.
-* [openshift-ansible](https://github.com/openshift/openshift-ansible/tree/release-3.7): Ansible roles and playbooks for installing and managing OpenShift 3.7 clusters with Ansible.
+* [openshift-ansible](https://github.com/openshift/openshift-ansible/tree/release-3.11): Ansible roles and playbooks for installing and managing OpenShift 3.11 clusters with Ansible.
 * [ansible-playbook](https://docs.ansible.com/ansible/2.4/ansible-playbook.html): the actual ansible tool used to deploy the OpenShift cluster. This is used in the install-from-bastion.sh script.
 
 ## Estimated Time to Complete
@@ -16,7 +16,7 @@ While the original repository required the user to manually run ansible-playbook
 Our target persona is a developer or operations engineer who wants to provision an OpenShift cluster into AWS.
 
 ## Challenge
-The [advanced installation method](https://docs.openshift.com/container-platform/3.7/install_config/install/advanced_install.html) for OpenShift uses ansible-playbook to deploy OpenShift. Before doing that, the deployer must first provision some infrastructure and then configure an Ansible inventory file with suitable settings. Typically, ansible-playbook would be manually run on a bastion host even if a tool like Terraform had been used to provision the infrastructure.
+The [installation method](https://docs.openshift.com/container-platform/3.11/install/index.html) for OpenShift uses ansible-playbook to deploy OpenShift. Before doing that, the deployer must first provision some infrastructure and then configure an Ansible inventory file with suitable settings. Typically, ansible-playbook would be manually run on a bastion host even if a tool like Terraform had been used to provision the infrastructure.
 
 ## Solution
 This guide combines and completely automates the two steps mentioned above:
@@ -64,14 +64,14 @@ EOF
 
 1. If you do not already have a Terraform Enterprise (TFE) account, self-register for an evaluation at https://app.terraform.io/account/new.
 1. After getting access to your TFE account, create an organization for yourself. You might also want to review the [Getting Started](https://www.terraform.io/docs/enterprise/getting-started/index.html) documentation.
-1. Connect your TFE organization to GitHub. See the [Configuring Github Access](https://www.terraform.io/docs/enterprise/vcs/github.html)documentation.
+1. Connect your TFE organization to GitHub. See the [Configuring GitHub Access](https://www.terraform.io/docs/enterprise/vcs/github.html) documentation.
 
 If you want to use open source Terraform instead of TFE, you can create a copy of the included openshift.tfvars.example file, calling it openshift.auto.tfvars, set values for the variables in it, run `terraform init`, and then run `terraform apply`.
 
 ### Step 3: Configure a Terraform Enterprise Workspace
 1. Fork this repository by clicking the Fork button in the upper right corner of the screen and selecting your own personal GitHub account or organization.
 1. Create a workspace in your TFE organization called k8s-cluster-openshift.
-1. Configure the workspace to connect to the fork of this repository in your own Github account.
+1. Configure the workspace to connect to the fork of this repository in your own GitHub account.
 1. Set the Terraform Working Directory to "infrastructure-as-code/k8s-cluster-openshift-aws".
 1. On the Variables tab of your workspace, add the following variables to the Terraform variables: key_name, private_key_data, vault_addr, vault_user, and vault_k8s_auth_path. The first of these must be the name of the key pair you created above. The second must be the actual contents of the private key you downloaded as a pem file.  Be sure to mark this variable as sensitive so that it will not be visible after you save your variables. Set vault_addr to the address of your Vault server (e.g., "http://<your_vault_dns>:8200") and vault_user to your username on your Vault server. Finally, set vault_k8s_auth_path to something like "\<your username\>-openshift".
 1. HashiCorp SEs should also set the owner and ttl variables which are used by the AWS Lambda reaper function that terminates old EC2 instances.

infrastructure-as-code/k8s-cluster-openshift-aws/main.tf

@@ -131,6 +131,7 @@ resource "null_resource" "configure_k8s" {
 
   provisioner "remote-exec" {
     inline = [
+      "sleep 180",
       "kubectl create -f vault-reviewer.yaml",
       "kubectl create -f vault-reviewer-rbac.yaml",
       "kubectl get serviceaccount vault-reviewer -o yaml > vault-reviewer-service.yaml",

infrastructure-as-code/k8s-cluster-openshift-aws/modules/openshift/01-amis.tf

@@ -1,6 +1,6 @@
 # Define the RHEL 7.2 AMI by:
-# RedHat, Latest, x86_64, EBS, HVM, RHEL 7.2
-data "aws_ami" "rhel7_2" {
+# RedHat, Latest, x86_64, EBS, HVM, RHEL 7.5
+data "aws_ami" "rhel7_5" {
   most_recent = true
 
   owners = ["309956199498"] // Red Hat's account ID.
@@ -22,7 +22,7 @@ data "aws_ami" "rhel7_2" {
 
   filter {
     name   = "name"
-    values = ["RHEL-7.2*"]
+    values = ["RHEL-7.5*"]
   }
 }
 

infrastructure-as-code/k8s-cluster-openshift-aws/modules/openshift/05-nodes.tf

@@ -7,7 +7,7 @@ data "template_file" "setup-master" {
 
 //  Launch configuration for the master
 resource "aws_instance" "master" {
-  ami                  = "${data.aws_ami.rhel7_2.id}"
+  ami                  = "${data.aws_ami.rhel7_5.id}"
   # Master nodes require at least 16GB of memory.
   instance_type        = "m4.xlarge"
   subnet_id            = "${aws_subnet.public-subnet.id}"
@@ -51,7 +51,7 @@ data "template_file" "setup-node" {
 
 //  Create the two nodes.
 resource "aws_instance" "node1" {
-  ami                  = "${data.aws_ami.rhel7_2.id}"
+  ami                  = "${data.aws_ami.rhel7_5.id}"
   instance_type        = "${var.amisize}"
   subnet_id            = "${aws_subnet.public-subnet.id}"
   iam_instance_profile = "${aws_iam_instance_profile.openshift-instance-profile.id}"

infrastructure-as-code/k8s-cluster-openshift-aws/modules/openshift/07-bastion.tf

@@ -12,6 +12,7 @@ data "template_file" "inventory" {
     master_ip = "${aws_instance.master.public_ip}"
     private_key = "${var.private_key_data}"
     name_tag_prefix = "${var.name_tag_prefix}"
+    region = "${var.region}"
   }
 }
 

infrastructure-as-code/k8s-cluster-openshift-aws/modules/openshift/files/install-from-bastion.sh

@@ -5,19 +5,26 @@ exec > /home/ec2-user/install-openshift.log 2>&1
 
 # Install dev tools and Ansible 2.2
 sudo yum install -y "@Development Tools" python2-pip openssl-devel python-devel gcc libffi-devel
-sudo pip install -Iv ansible==2.4.3.0
+sudo pip install -Iv ansible==2.6.5
 
 # Clone the openshift-ansible repo, which contains the installer.
-git clone -b release-3.7 https://github.com/openshift/openshift-ansible
+#git clone -b release-3.11 https://github.com/openshift/openshift-ansible
+
+# Using specific commit since later one made it illegal to use the openshift_hostname variable
+git clone -n https://github.com/openshift/openshift-ansible
+cd openshift-ansible
+git checkout 7c8b4f07a46089640bc61b8a9b35fd8b5ed86245
+cd ..
+
 
 # Set up bastion to SSH to other servers
 echo "${private_key}" > /home/ec2-user/.ssh/private-key.pem
 chmod 400 /home/ec2-user/.ssh/private-key.pem
 #chown ec2-user:ec2-user /home/ec2-user/.ssh/private-key.pem
 eval $(ssh-agent)
 ssh-add /home/ec2-user/.ssh/private-key.pem
-ssh-keyscan -t rsa -H master.${name_tag_prefix}-openshift.local >> /home/ec2-user/.ssh/known_hosts
-ssh-keyscan -t rsa -H node1.${name_tag_prefix}-openshift.local >> /home/ec2-user/.ssh/known_hosts
+#ssh-keyscan -t rsa -H master.${name_tag_prefix}-openshift.local >> /home/ec2-user/.ssh/known_hosts
+#ssh-keyscan -t rsa -H node1.${name_tag_prefix}-openshift.local >> /home/ec2-user/.ssh/known_hosts
 
 # Create inventory.cfg file
 #cat > /home/inventory.cfg << EOF
@@ -45,7 +52,7 @@ ansible_become=true
 openshift_deployment_type=origin
 
 # OpenShift Release
-openshift_release=v3.7
+openshift_release=v3.11
 
 # We need a wildcard DNS setup for our public access to services, fortunately
 # we can use the superb xip.io to get one for free.
@@ -54,13 +61,22 @@ openshift_public_hostname=${master_ip}.xip.io
 openshift_master_default_subdomain=${master_ip}.xip.io
 
 # Use an htpasswd file as the indentity provider.
-openshift_master_identity_providers=[{'name': 'htpasswd_auth', 'login': 'true', 'challenge': 'true', 'kind': 'HTPasswdPasswordIdentityProvider', 'filename': '/etc/origin/master/htpasswd'}]
+openshift_master_identity_providers=[{'name': 'htpasswd_auth', 'login': 'true', 'challenge': 'true', 'kind': 'HTPasswdPasswordIdentityProvider'}]
+#, 'filename': '/etc/origin/master/htpasswd'
 
 # Uncomment the line below to enable metrics for the cluster.
 # openshift_hosted_metrics_deploy=true
 
-openshift_enable_service_catalog=false
-template_service_broker_install=false
+# Set the cluster_id.
+openshift_clusterid="openshift-cluster-${region}"
+
+# Define the standard set of node groups, as per:
+#   https://github.com/openshift/openshift-ansible#node-group-definition-and-mapping
+openshift_node_groups=[{'name': 'node-config-master', 'labels': ['node-role.kubernetes.io/master=true']}, {'name': 'node-config-infra', 'labels': ['node-role.kubernetes.io/infra=true']}, {'name': 'node-config-compute', 'labels': ['node-role.kubernetes.io/compute=true']}, {'name': 'node-config-master-infra', 'labels': ['node-role.kubernetes.io/infra=true,node-role.kubernetes.io/master=true']}, {'name': 'node-config-all-in-one', 'labels': ['node-role.kubernetes.io/infra=true,node-role.kubernetes.io/master=true,node-role.kubernetes.io/compute=true']}]
+
+
+#openshift_enable_service_catalog=false
+#template_service_broker_install=false
 
 #openshift_template_service_broker_namespaces=['openshift']
 
@@ -79,15 +95,18 @@ master.${name_tag_prefix}-openshift.local openshift_hostname=master.${name_tag_p
 
 # host group for nodes, includes region info
 [nodes]
-master.${name_tag_prefix}-openshift.local openshift_hostname=master.${name_tag_prefix}-openshift.local openshift_node_labels="{'region': 'infra', 'zone': 'default'}" openshift_schedulable=true
-node1.${name_tag_prefix}-openshift.local openshift_hostname=node1.${name_tag_prefix}-openshift.local openshift_node_labels="{'region': 'primary', 'zone': 'east'}"
+master.${name_tag_prefix}-openshift.local openshift_hostname=master.${name_tag_prefix}-openshift.local openshift_node_group_name='node-config-master-infra' openshift_schedulable=true
+node1.${name_tag_prefix}-openshift.local openshift_hostname=node1.${name_tag_prefix}-openshift.local openshift_node_group_name='node-config-compute'
 EOF
 
 # Change ownership of file to ec2-user
 #sudo chown ec2-user:ec2-user /home/ec2-user/inventory.cfg
 
 # Run the playbook.
-ANSIBLE_HOST_KEY_CHECKING=False /usr/local/bin/ansible-playbook -i ~/inventory.cfg ~/openshift-ansible/playbooks/byo/config.yml
+#ANSIBLE_HOST_KEY_CHECKING=False /usr/local/bin/ansible-playbook -i ~/inventory.cfg ~/openshift-ansible/playbooks/byo/config.yml
+
+ANSIBLE_HOST_KEY_CHECKING=False /usr/local/bin/ansible-playbook -i ~/inventory.cfg ./openshift-ansible/playbooks/prerequisites.yml
+ANSIBLE_HOST_KEY_CHECKING=False /usr/local/bin/ansible-playbook -i ~/inventory.cfg ./openshift-ansible/playbooks/deploy_cluster.yml
 
 # uncomment for verbose! -vvv
 

infrastructure-as-code/k8s-cluster-openshift-aws/scripts/postinstall-master.sh

@@ -6,6 +6,7 @@
 sleep 120
 
 # Create an htpasswd file, we'll use htpasswd auth for OpenShift.
+sudo mkdir -p /etc/origin/master
 sudo htpasswd -cb /etc/origin/master/htpasswd admin 123
 oc adm policy add-cluster-role-to-user cluster-admin admin