Done lab refactoring on OpenStack

SwitchEngines/README.md

@@ -41,31 +41,29 @@ Please, refer to your OS documentation for the proper way to do so
      v1.1.4. Skip the TF "Quick start tutorial" (Docker).
   1. [OpenStack
      CLI](https://docs.openstack.org/newton/user-guide/common/cli-install-openstack-command-line-clients.html)
-     version >= 5.7.0. It is recommended to install your original distribution
+     version >= 6.0.0. It is recommended to install your original distribution
      package named like `python-openstackclient`.
-  1. Grab or create a new project in your OpenStack account. The placeholder
-     used in this exercise is `<your-project-name>` with value `Cloud-MA-IaC`.
-  1. On the OpenStack Horizon dashboard, go to your project page, then
-     1. create new [Application
+  1. On the OpenStack dashboard, create a new project. The placeholder used in
+     this exercise is `<your-project-name>` with value `Cloud-MA-IaC`.
+  1. Go to your project page, then create new [Application
         Credentials](https://engines.switch.ch/horizon/identity/application_credentials/)
-        and save it as `~/.config/openstack/clouds.yaml`. With SwitchEngines, the
-        cloud name to use for this is `engines`. :warning: App credentials might
-        not work for some commands. A template is also available in repo file
-        [`conf/clouds.yaml.api_cred`](conf/clouds.yaml.api_cred). Alternatively,
+        and save it as `~/.config/openstack/clouds.yaml`. With SwitchEngines,
+        the cloud name to use for this is `engines`. :warning: App credentials
+        might not work for some commands. A template is also available in this
+        repo file
+        [`conf/clouds.yaml.app_cred`](conf/clouds.yaml.app_cred). Alternatively,
         you can use your [API
         credentials](https://engines.switch.ch/horizon/project/api_access/clouds.yaml)
-        with explicit project name/ID -- you'll have to add your API `password` from
-        your profile page's ["Credentials"
+        with explicit project name/ID -- you'll have to add your API
+        `password` from your profile page's ["Credentials"
         tab](https://engines.admin.switch.ch/users/profile). A template is
-        available in repo file
-        [`conf/clouds.yaml.app_cred`](conf/clouds.yaml.app_cred).
-    <!-- 1. download your [RC -->
-    <!--    file](https://engines.switch.ch/horizon/project/api_access/openrc/) and -->
-    <!--    install it (it asks for your API password): -->
-    <!--    ``` shell -->
-    <!--    $ source <your-project>-openrc.sh -->
-    <!--    ``` -->
-  1. Verify that your credentials are OK (:warning: it might reveal secrets!):
+        available in this repo file
+        [`conf/clouds.yaml.api_cred`](conf/clouds.yaml.app_cred). :warning:
+        Avoid mixing different authentication schemes in `clouds.yaml` or from
+        the environment (via sourcing so called OpenStack RC files).
+  1. Verify that your credentials are OK (:warning: it might reveal secrets!
+     If you have just one cloud configured, you can drop the switch
+     `--os-cloud=engines`, else adapt accordingly):
   ``` shell
   lcl$ $ openstack --os-cloud=engines [application] credential list
   ```
@@ -74,7 +72,7 @@ Please, refer to your OS documentation for the proper way to do so
 
 **Goal:** instruct TF to handle a single OpenStack instance.
 
-<a name="image-query"></a>Find out the smallest image to use for a Debian server:
+Find out the smallest image to use for a Debian server:
 
 ``` shell
 lcl$ openstack --os-cloud=engines image list --public --status=active --sort-column=Size -c ID -c Name -c Size --long
@@ -86,7 +84,7 @@ lcl$ openstack --os-cloud=engines image list --public --status=active --sort-col
 ```
 
 :bulb: We use the first ID found for the placeholder `<your-image-ID>`. In
-SwitchEngines this is 1.5GB.
+SwitchEngines, that has a size of ~1.5GB.
 
 Find out the smallest instance *flavor* that acommodates our Debian image.
 
@@ -106,6 +104,9 @@ Create a "sandbox" directory on your local machine
 in HCL language), the infrastructure *definition* file, with the following
 content (replace all `<...>` tokens with actual values):
 
+:bulb: Application credentials shall match those in file
+`~/.config/openstack/clouds.yaml`.
+
 ``` hcl
 terraform {
   required_version = ">= 0.14.9"
@@ -122,7 +123,7 @@ provider "openstack" {
   project_domain_id = "<your-project-ID>"
   application_credential_id     = "<your-app-cred-ID>"
   application_credential_secret = "<your-app-cred-secret>"
-  region = "ZH"
+  region = "<your-region>"
 }
 
 resource "openstack_compute_instance_v2" "app_server" {
@@ -229,8 +230,7 @@ Do you want to perform these actions?
 
   Enter a value: yes
 
-openstack_compute_instance_v2.app_server: Creating...
-openstack_compute_instance_v2.app_server: Still creating... [10s elapsed]
+...
 openstack_compute_instance_v2.app_server: Creation complete after 18s [id=f70aef53-51f9-4d12-a4a9-fe9a6cc5a58f]
 
 Apply complete! Resources: 1 added, 0 changed, 0 destroyed.
@@ -295,7 +295,7 @@ not, why? We'll deal with that in Task #4 below.
 Now, stop the running instance (its ID is shown above ;-):
 
 ``` shell
-lcl$ $ openstack server stop f70aef53-51f9-4d12-a4a9-fe9a6cc5a58f
+lcl$ openstack --os-cloud=engines server stop f70aef53-51f9-4d12-a4a9-fe9a6cc5a58f
 ```
 
 wait some seconds and test again:
@@ -310,114 +310,254 @@ resource. So, let's refresh it, and check again:
 
 ``` shell
 lcl$ terraform refresh
-aws_instance.app_server: Refreshing state... [id=i-0155ba9d77ee0a854]
-lcl$ terraform show | grep instance_state
-    instance_state                       = "stopped"
+...
+lcl$ terraform show | grep power_state
+    power_state         = "shutoff"
 ```
 
 Ah-ha!
 
+Hold on a second: our TF plan does not (explicitly) specify the desired status
+of a resource. What happens if we reapply the plan? Lets' try:
 
-### Task #4: change your infrastructure ###
+``` shell
+lcl$ terraform apply -auto-approve
+openstack_compute_instance_v2.app_server: Refreshing state... [id=...]
+...
+Terraform will perform the following actions:
+...
+      ~ power_state         = "shutoff" -> "active"
+...
+Apply complete! Resources: 0 added, 1 changed, 0 destroyed.
 
-Indeed, the instance doesn't have a (floating) *public* IP address, so we need
-to get one and associate it to our instance with the following snippet added
-to `main.tf`:
+lcl$ terraform show | grep power_state
+    power_state         = "active"
+```
 
-``` hcl
-resource "openstack_networking_floatingip_v2" "fip_1" {
-  pool = "public"
-}
+:warning: Apply was run in `-auto-approve` (non interactive) mode which
+assumes "yes" to all questions. Use with care!
+
+From the above commands' output we see that
+  * the local state is refreshed before doing anything,
+  * 1 change was applied: the instance has been switched on,
+  * OpenStack instances have an "active" default `power_state` :-)
 
-resource "openstack_compute_floatingip_associate_v2" "fip_1" {
-  # we use var/obj notation here ${x.y}
-  floating_ip = "${openstack_networking_floatingip_v2.fip_1.address}"
-  instance_id = "${openstack_compute_instance_v2.app_server.id}"
-  # to avoid getting "Resource not found"
-  wait_until_associated = true
-}
-```
 
-Let's see what happens:
+### Task #4: change your infrastructure ###
+
+**Goal:** modify the resource created before, and learn how to apply changes
+to a Terraform project.
+
+Replace the resource's `image_id` in `main.tf` with the second one found from
+the catalog query done above -- it should be a "Debian Bullseye 11
+(SWITCHengines)". Before applying our new plan, let's see what TF thinks of
+it:
+
 ``` shell
-lcl$ terraform apply -auto-approve
-openstack_compute_instance_v2.app_server: Refreshing state... [id=f70aef53-51f9-4d12-a4a9-fe9a6cc5a58f]
-...
+lcl$ terraform plan -out=change-image-ID.tfplan
+openstack_compute_instance_v2.app_server: Refreshing state... [id=1edad9e2-5459-4980-bbc5-a0c5b65bfb0d]
+
 Terraform will perform the following actions:
 
-  # openstack_compute_floatingip_associate_v2.fip_1 will be created
-  + resource "openstack_compute_floatingip_associate_v2" "fip_1" {
-      + floating_ip           = (known after apply)
-      ...
-    }
+  # openstack_compute_instance_v2.app_server will be updated in-place
+  ~ resource "openstack_compute_instance_v2" "app_server" {
+        id                  = "1edad9e2-5459-4980-bbc5-a0c5b65bfb0d"
+      ~ image_id            = "8674f1a5-f7d9-4975-af0b-d2e9e33c9152" -> "54ee4d6e-9155-4698-ab2b-45d9067e8e8e"
+        name                = "TF-managed"
+        tags                = []
+        # (13 unchanged attributes hidden)
 
-  # openstack_networking_floatingip_v2.fip_1 will be created
-  + resource "openstack_networking_floatingip_v2" "fip_1" {
-      + address    = (known after apply)
-      ...
+        # (1 unchanged block hidden)
     }
 
-Plan: 2 to add, 0 to change, 0 to destroy.
-openstack_networking_floatingip_v2.fip_1: Creating...
-openstack_networking_floatingip_v2.fip_1: Creation complete after 9s [id=81f7690f-e024-4eb8-bbbc-98a242c3b0c3]
-openstack_compute_floatingip_associate_v2.fip_1: Creating...
-openstack_compute_floatingip_associate_v2.fip_1: Creation complete after 6s [id=86.119.32.210/f70aef53-51f9-4d12-a4a9-fe9a6cc5a58f/]
+Plan: 0 to add, 1 to change, 0 to destroy.
+...
+Saved the plan to: change-image-ID.tfplan
+
+To perform exactly these actions, run the following command to apply:
+    terraform apply "change-image-ID.tfplan"
 ```
 
-Here we go. First, observe how TF had to refresh its local status before
-applying the new plan. Then, now the instance got the public IP address shown
-at the bottom of the command's output. However, a last bit is still missing,
-because the "default" *security group* only allows outbound traffic. Thus we
-need something like:
+:bulb: Remarks:
+  * The change we want to apply is *not* destructive.
+  * We saved our plan. :question: Why? It is not really necessary in a simple
+    scenario like ours, however a more complex IaC workflow might require plan
+    artifacts to be programmatically validated and versioned.
+
+Apply the saved plan:
+``` shell
+lcl$ terraform apply change-image-ID.tfplan
+$ terraform apply change-image-ID.tfplan
+...
+Apply complete! Resources: 0 added, 1 changed, 0 destroyed.
+```
+
+:bulb: What? Not asking for confirmation? Indeed, a *saved* plan is intended for
+automated workflows! Moreover, a saved plan will come handy for rolling back a
+broken infrastructure to the last working setup.
+
+:question: What if we did not save our plan, and called a plain apply command?
+Would the result be the same?
+
+
+### Task #5: input variables ###
+
+**Goal:** make a TF plan more flexible via input variables.
+
+Our original plan has all its content hard-coded. Let's make it more flexible
+with some input variables stored in a separate `variables.tf` file inside your
+TF sandbox:
 
 ``` hcl
-resource "openstack_compute_instance_v2" "app_server" {
-  ... # as above
- security_groups = ["default", "secgroup_tf"]
+variable "instance_name" {
+  description = "Value of the instance's name tag"
+  type        = string
+  default     = "AnotherAppServerInstance"
 }
+```
 
-resource "openstack_networking_secgroup_v2" "secgroup_tf" {
-  name        = "secgroup_tf"
-}
+Then modify the `main.tf` as follows:
 
-resource "openstack_networking_secgroup_rule_v2" "secgroup_rule_1" {
-  direction         = "ingress"
-  ethertype         = "IPv4"
-  protocol          = "tcp"
-  port_range_min    = 22
-  port_range_max    = 22
-  remote_ip_prefix  = "0.0.0.0/0"
-  security_group_id = "${openstack_networking_secgroup_v2.secgroup_tf.id}"
+``` hcl
+resource "openstack_compute_instance_v2" "app_server" {
+  ...
+  metadata = {
+    my_instance_name = var.instance_name
+  }
 }
+```
 
-resource "openstack_networking_floatingip_v2" "fip_1" ...
+Apply the changes:
+``` shell
+lcl$ terraform apply -auto-approve
+
+Plan: 0 to add, 1 to change, 0 to destroy.
 ...
+Apply complete! Resources: 0 added, 1 changed, 0 destroyed.
 ```
 
-Now, our instance is reachable via SSH (only!). But can we login? The reply is
-in Task #7.
+You should see the new tag added to the "Metadata" section of the Horizon
+dashboard:
+`https://engines.switch.ch/horizon/project/instances/<instance-ID>/`.
+
+:bulb: **Exercise:** input variables can also be passed on the `apply` command
+line. Find how to do that with another different value for the variable
+`instance_name`. :question: Would this last change be persistent if we rerun a
+plain `terraform apply`?
 
-### Task #5: input variables ###
 
 ### Task #6: queries with outputs ###
 
-### Task #7: SSH provisioning with Cloud-Init ###
+**Goal:** use output values to query a provisioned infrastructure.
+
+We have seen in the previous tasks that the infrastructure's status can be
+displayed via `terraform show`: a rather clumsy way, if you just want to
+extract some specific information. A better programmatic way of querying your
+infrastructure makes use of "outputs". In your sandbox, put the following in a
+file called `outputs.tf`:
 
-@@@@@ OLD part below@@@@
+``` hcl
+output "instance_id" {
+  description = "ID of the instance"
+  value       = openstack_compute_instance_v2.app_server.id
+}
+```
 
-Also, prepare a `~/terraform/OpenStack/outputs.tf` file with the appropriate
-contents to get the instance's public IP address (via `instance_public_ip`).
+We have declared two outputs. As usual with TF, before querying their
+associated values, we need to apply the changes:
 
+``` shell
+lcl$ terraform apply -auto-approve
+...
+Apply complete! Resources: 0 added, 0 changed, 0 destroyed.
+
+Outputs:
 
+instance_id = "93914f14-e521-4cc1-acfe-046bc3fa31be"
+```
 
-Then apply!
+So, we already got the needed information, but, within a workflow, it is more
+practical to do something like:
 
 ``` shell
-lcl$ terraform apply
+lcl$ terraform output instance_id
 ```
 
-Verify that you can SSH as user `debian` into your instance:
+:bulb: **Exercise:** Add an output item to display the instance metadata tag
+`my_instance_name`.
 
+:question: What if the `my_instance_name` tag is changed outside TF? Try f.i.:
 ``` shell
-lcl$ ssh debian@$(terraform output -raw instance_public_ip) -i ../tf-cloud-init
+lcl$ openstack --os-cloud=engines server set \
+       --property my_instance_name="Foo-Bar" 93914f14-e521-4cc1-acfe-046bc3fa31be
+```
+
+:question: What must be done to have TF respect that external change?
+
+:question: How to revert an external change via TF?
+
+
+### Task #7: networking and SSH provisioning with Cloud-Init ###
+
+**Goal:** use Cloud-Init to provision an SSH access to your TF-managed
+instance.
+
+Did you try to SSH into the instance you created via TF? It cannot work,
+because we did not instructed TF about networking, login, users, keys or
+anything else. **This is left entirely to you as an exercise.** With reference
+to the official TF documentation for the [OpenStack
+provider](https://registry.terraform.io/providers/terraform-provider-openstack/openstack/latest/docs/), you shall:
+
+  1. Destroy your infrastructure. Guess how ;-)
+  1. Create an SSH key pair `tf-cloud-init`.
+  1. Create a new cloud-init file
+     `~/terraform/OpenStack/scripts/add-ssh.yaml` with the following content:
+     ``` yaml
+     #cloud-config
+     #^^^^^^^^^^^^
+     # DO NOT TOUCH the first line!
+     ---
+     groups:
+       - debian: [root, sys]
+       - terraform
+
+     users:
+       - default
+       - name: terraform
+         gecos: terraform
+         primary_group: terraform
+         groups: users, admin
+         ssh_authorized_keys:
+           - <your-SSH-pub-key-on-one-line>
+     ```
+     :warning: **Mind that the first line of this file must spell exactly
+     `#cloud-config`**!
+  1. Modify the `main.tf` as follows:
+     1. add a resource block for a custom v2 network *security group*;
+     1. add resource blocks for v2 *security group rules* allowing ICMP ping
+        and TCP ingress ports 22 from anywhere with any egress port open. These
+        resources shall reference your security group;
+     1. add a resource block for a v2 *floating IP* from the public pool;
+     1. add a resource block for a v2 *floating IP association* referencing
+        your floating IP resource and your compute instance resource;
+     1. extend your `"app_server"` resource block to:
+        1. reference your custom security group as well as the default one,
+        1. associate a public IP address,
+        1. include the above cloud-init file,
+     1. add a TF output `"instance_public_ip"`.
+
+:bulb: In the above instruction, "reference" means that the keys shall take
+values dynamically from variable expansion, like:
+``` hcl
+  key = ${<object.attribute...>}
+```
+
+When done, *validate* your new plan and *apply* it. Verify that you can ping
+and connect via SSH as user `terraform` into your instance:
+
+``` shell
+lcl$ ping $(terraform output -raw instance_public_ip)
+...
+lcl$ ssh terraform@$(terraform output -raw instance_public_ip) -i /path/to/private/key/tf-cloud-init
+...
 ```

SwitchEngines/conf/clouds.yaml.api_cred

-# Project-agnostic -- API credentials
+# -*- mode: yaml -*-
+# OpenStack conf file with *API* credentials --
+# <https://engines.switch.ch/horizon/project/api_access>
 clouds:
   engines:
     auth:
-      auth_url: https://keystone.cloud.switch.ch:5000/v3
+      auth_url: "https://keystone.cloud.switch.ch:5000/v3"
       username: "<your-user-name>"
-      password: "<your-password"
+      password: "<your-password>"
+      # project_name: "<your-project>"
       user_domain_name: "Default"
     interface: "public"
     identity_api_version: 3

SwitchEngines/conf/clouds.yaml.app_cred

-# Project-specific -- application credentials.
+# -*- mode: yaml -*-
+# OpenStack conf file with *application* credentials.
 clouds:
-  Cloud-MA-IaC:
+  engines:
     auth:
-      auth_url: https://keystone.cloud.switch.ch:5000/v3
+      auth_url: "https://keystone.cloud.switch.ch:5000/v3"
       application_credential_id: "<your-app-cred-ID>"
       application_credential_secret: "<your-app-cred-secret>"
-      user_domain_name: Default
-      project_domain_name: Default
     interface: "public"
+    region: "<your-region>"
     identity_api_version: 3
     auth_type: "v3applicationcredential"

SwitchEngines/main.tf.basic

+# -*- mode: terraform -*-
 terraform {
-  required_version = ">= 0.14.9"
   required_providers {
     openstack = {
       source  = "terraform-provider-openstack/openstack"
-      # version = "~> 1.35.0"
       version = "~> 1.48.0"
     }
   }
+  required_version = ">= 0.15.0"
 }
 
 # Configure the OpenStack Provider
@@ -15,7 +15,7 @@ provider "openstack" {
   project_domain_id = "<your-project-ID>"
   application_credential_id     = "<your-app-cred-ID>"
   application_credential_secret = "<your-app-cred-secret>"
-  region = "LS"
+  region = "<your-region>"
 }
 
 resource "openstack_compute_instance_v2" "app_server" {
@@ -23,30 +23,4 @@ resource "openstack_compute_instance_v2" "app_server" {
   image_id        = "<your-image-ID>"
   flavor_name     = "<your-flavor>"
   key_pair        = "TF_lab"
-  security_groups = ["default", "secgroup_tf"]
-}
-
-resource "openstack_networking_secgroup_v2" "secgroup_tf" {
-  name        = "secgroup_tf"
-}
-
-resource "openstack_networking_secgroup_rule_v2" "secgroup_rule_1" {
-  direction         = "ingress"
-  ethertype         = "IPv4"
-  protocol          = "tcp"
-  port_range_min    = 22
-  port_range_max    = 22
-  remote_ip_prefix  = "0.0.0.0/0"
-  security_group_id = "${openstack_networking_secgroup_v2.secgroup_tf.id}"
-}
-
-resource "openstack_networking_floatingip_v2" "fip_1" {
-  pool = "public"
-}
-
-resource "openstack_compute_floatingip_associate_v2" "fip_1" {
-  floating_ip = "${openstack_networking_floatingip_v2.fip_1.address}"
-  instance_id = "${openstack_compute_instance_v2.app_server.id}"
-  # to avoid getting "Resource not found"
-  wait_until_associated = true
 }

SwitchEngines/scripts/add-ssh.yaml

+#cloud-config
+#^^^^^^^^^^^^
+# DO NOT TOUCH the first line!
+---
+groups:
+  - debian: [root, sys]
+  - terraform
+
+users:
+  - default
+  - name: terraform
+    gecos: terraform
+    primary_group: terraform
+    groups: users, admin
+    ssh_authorized_keys:
+      - <your-SSH-pub-key-on-one-line>

SwitchEngines/variables.tf

+# -*- mode: terraform -*-
+variable "instance_name" {
+  description = "Value of the instance's name tag"
+  type        = string
+  default     = "TF-Lab-AppServerInstance"
+}