More Cloud-init integration

Signed-off-by: Marco Emilio "sphakka" Poleggi <marcoep@ieee.org>

SwitchEngines/README.md

@@ -526,7 +526,7 @@ lcl$ openstack --os-cloud=engines server set \
 :question: How to revert an external change via TF?
 
 
-### Task #7: networking and SSH provisioning with Cloud-Init ###
+### Task #7: networking and SSH provisioning with Cloud-init ###
 
 **Goal:** complete the network configuration and use
 [Cloud-init](https://cloudinit.readthedocs.io/) to provision an SSH access to
@@ -534,9 +534,13 @@ your TF-managed instance.
 
 Did you try to SSH into the instance you created via TF? It cannot work,
 because we did not instructed TF about networking, login, users, keys or
-anything else. **This is left entirely to you as an exercise.** With reference
-to the official TF documentation for the [OpenStack
-provider](https://registry.terraform.io/providers/terraform-provider-openstack/openstack/latest/docs/), you shall:
+anything else.
+
+**This is left to you as an exercise.**
+
+With reference to the official TF documentation for the [OpenStack
+provider](https://registry.terraform.io/providers/terraform-provider-openstack/openstack/latest/docs/),
+you shall:
 
   1. Destroy your infrastructure. Guess how ;-)
   1. Create **locally** in your sandbox an SSH key pair named `tf-cloud-init`:
@@ -545,14 +549,13 @@ provider](https://registry.terraform.io/providers/terraform-provider-openstack/o
       lcl$ yes | ssh-keygen -t ed25519 -f keys/tf-cloud-init -C terraform@TF-lab
       ```
   1. Create a new cloud-init file
-     `~/terraform/OpenStack/conf/cloud-init.add-ssh.yaml` with the following content:
+     `conf/cloud-init.add-ssh.yaml` with the following content:
      ``` yaml
      #cloud-config
      #^^^^^^^^^^^^
      # DO NOT TOUCH the first line!
      ---
      groups:
-       - debian: [root, sys]
        - terraform
 
      users:
@@ -560,7 +563,7 @@ provider](https://registry.terraform.io/providers/terraform-provider-openstack/o
        - name: terraform
          gecos: terraform
          primary_group: terraform
-         groups: users, admin
+         groups: [users, admin]
          ssh_authorized_keys:
            - <your-SSH-pub-key-on-one-line>
      ```
@@ -577,8 +580,8 @@ provider](https://registry.terraform.io/providers/terraform-provider-openstack/o
         'data' port;
      1. extend your `"app_server"` resource block to:
         1. reference your custom security group as well as the default one,
-        1. include the above cloud-init file,
-     1. add two TF outputs: `"instance_public_ip"` and `"ssh_user"`.
+        1. include the above cloud-init file via a [`user_data` attribute](https://registry.terraform.io/providers/terraform-provider-openstack/openstack/latest/docs/resources/compute_instance_v2),
+     1. add two TF outputs: `"instance_public_ip"` (dynamic) and `"ssh_user"` (static).
 
 :bulb: In the above instruction, "reference" means that the configuration keys shall take
 values dynamically from variable expansion, like (`${}` is optional in simple cases):
@@ -595,3 +598,153 @@ lcl$ ping $(terraform output -raw instance_public_ip)
 lcl$ ssh -i keys/tf-cloud-init -o IdentitiesOnly=yes $(terraform output -raw ssh_user)@$(terraform output -raw instance_public_ip)
 ...
 ```
+
+### Task #8: further practice with Cloud-init  ###
+
+**Goal:** get more acquainted with Cloud-init: change the instance's default
+configuration, handle packages, integrate more with Terraform.
+
+You may already have realized that Cloud-init is a declarative configuration
+management tool for the Cloud ;-)
+
+As for the provisioning side, the mechanism to pass extra instance
+configuration, scripts, etc. is the ["user
+data"](https://cloudinit.readthedocs.io/en/latest/explanation/format.html). Of
+course, most if not all cloud images come with Cloud-init installed.
+
+The provisioning tool/platform (here Terraform) just does some proper
+formatting/marshaling of user data before sending them to the cloud.
+
+:bulb: Before trying more, here are some tips for [debugging your
+cloud-init](https://cloudinit.readthedocs.io/en/latest/howto/debugging.html#how-to-debug)
+user data (of course your instance must be accessible via SSH):
+
+**Q.** Where are the instance's Cloud-init logs? A. They're in files
+`/var/log/cloud-init*.log`.
+
+**Q.** Where are user-data files stored *in the instance*? A. Here they are:
+- Received file: `/var/lib/cloud/instance/user-data.txt `
+- Rendered file: `/var/lib/cloud/instance/cloud-config.txt`
+
+**Q.** [How to rerun Cloud-init](https://cloudinit.readthedocs.io/en/latest/howto/rerun_cloud_init.html) on the instance without reinstalling it?
+A. Run the command: `sudo cloud-init clean --logs --reboot`.
+
+
+**The following tasks are left to you as exercises.**
+
+
+#### Task #8.1: default user configuration, etc.  ####
+
+In Task #7, you added the extra user `terraform` while leaving untouched the
+default one. Actually, we might know nothing about this latter (it depends on
+the distribution), so why not impose *our* default user?
+
+Easy! You just need to modify the `conf/cloud-init.add-ssh.yaml` above and
+replace the `users` section with a suitable `default_user` dictionary under
+section
+[`system_info`](https://cloudinit.readthedocs.io/en/latest/reference/base_config_reference.html#system-info-keys):
+
+``` yaml
+system_info:
+  default_user:
+    name: terraform
+    ...
+```
+
+You may also have a look at the distro's original Cloud-init configuration
+`/etc/cloud/cloud.cfg` in your provisioned instance.
+
+Since our image is rather light, let's add some handy packages (names must be
+the official distro's) to our cloud-init YAML:
+``` yaml
+packages:
+  - curl
+  - nano
+  - ripgrep
+  - <add your favorite here>
+  - ...
+```
+
+See how that's agnostic of the package manager? Of course, since we're on
+Debian/Ubuntu, `apt` is actually called behind the scenes.
+
+Want to install a package out of the distro's tree? Quickly done with the
+generic shell command support:
+``` yaml
+runcmd:
+  - mkdir -p /run/tmp
+  - curl -Lo /run/tmp/kind https://kind.sigs.k8s.io/dl/v0.24.0/kind-linux-amd64
+  - install -o root -g root -m 0755 /run/tmp/kind /usr/local/bin/kind
+```
+
+Once done, rerun terraform apply as in Task #7. You should be able to connect
+as before with user `terraform`. Verify that the extra packages have been
+installed.
+
+
+#### Task #8.2: integrating Cloud-init with Terraform  ####
+
+So far Terraform didn't interfere at all with the user data: it just sent some
+kind of blob down the pipe. Now, suppose you have *several* user data files
+(configuration, scripts, etc.) to be provisioned: how to do that? Several
+`user_data` attributes? Nah, that wouldn't work.
+
+Enter the [`cloudinit-config` data source](https://registry.terraform.io/providers/hashicorp/cloudinit/latest/docs/data-sources/config)!
+
+[Data
+sources](https://developer.hashicorp.com/terraform/language/data-sources)
+provide special blocks to store information that can be used by resource
+blocks. You already used a data block for the FIP port association.
+
+The cloudinit-config data source supports the declaration of several parts
+which are *merged* together into a single Multi-part MIME piece, so that the
+result can be ingested by the instance resource's `user_data` attribute.
+
+You shall:
+
+1. Split your cloud-init YAML file above into 2 files:
+   - `conf/cloud-init.users.yaml` with basic user-related configuration, and
+   - `conf/cloud-init.packages.yaml` with package-related configuration.
+1. Modify your `main.tf`:
+   1. Add a cloudinit-config data block with two parts declaring each one of
+      the above YAML conf file.
+   1. Change, in the `app_server` resource block, the `user_data` assignment to
+      get the "rendered" output of the cloudinit-config data block.
+
+Then, rerun terraform apply, connect to your instance and verify that
+everything is there.
+
+:question: What if one needs to modify or override in the second cloud-init
+YAML file something already declared in the first one? E.g., adding a special
+package-provided group to the default user. There's risk of breaking things,
+unless the correct [user-data merge strategy is
+specified](https://cloudinit.readthedocs.io/en/latest/reference/merging.html).
+
+
+### Task #9: Provisioning KinD with with Terraform  ###
+
+**Please, complete the [K8s Lab
+tutorial](https://gitedu.hesge.ch/lsds/teaching/bachelor/cloud-and-deployment/lab-k8s)
+before starting this exercise.**
+
+Now you have all the skills to provision your micro cloud infrastructure based
+on KinD.
+
+You shall:
+
+1. Modify your `main.tf` to use a more capable flavor instance for your
+   `app_server`, like a `m1.medium` or `c1.large`.
+2. Extend your YAML files to
+   - install all the required packages. Tips: add the distro's `docker.io`
+     package, install `kubectl` and `kind` via explicit `runcmd` scripts;
+   - add any needed extra group for the default user.
+
+Run again terraform apply, wait a couple of minutes to let the installation of
+all extra packages to complete, then connect to your instance.
+
+Since KinD is not designed for remote use, you'll have to create your cluster
+on the instance as done during the K8s Lab:
+
+1. Write a minimal KinD configuration file for 2 worker nodes.
+1. Provision the cluster.
+1. Deploy the matallb load balancer service.