diff --git a/README.md b/README.md new file mode 100644 index 0000000..8dfc630 --- /dev/null +++ b/README.md @@ -0,0 +1,352 @@ +# TerraNova — Infrastructure as Code CI/CD (Mandatory Assignment 2) + +**Repository:** [einalien/assigment2-iac](https://git.ntnu.no/einalien/assigment2-iac) +**Platform:** NTNU SkyHiGh OpenStack · OpenTofu · GitHub Actions on `git.ntnu.no` +**Environments:** TEST → STAGE → PROD + +## Table of Contents + +1. [Purpose and Scope](#1-purpose-and-scope) + - [Implementation Status](#implementation-status) +2. [Architecture and Repository Structure](#2-architecture-and-repository-structure) + - [Environment Configuration](#environment-configuration) +3. [CI/CD Workflow](#3-cicd-workflow) + - [3.1 Developer Workflow and Pull Requests](#31-developer-workflow-and-pull-requests) + - [3.2 Continuous Integration](#32-continuous-integration--githubworkflowsciyml) + - [3.3 Deployment](#33-deployment--githubworkflowsdeployyml) +4. [Prerequisites and Credentials](#4-prerequisites-and-credentials) + - [GitHub Actions Environments: Required Configuration](#github-actions-environments-required-configuration) +5. [Running the Workflow End to End](#5-running-the-workflow-end-to-end) + - [5.1 Runner Verification](#51-runner-verification) + - [5.2 Automatic TEST and STAGE Deployment](#52-automatic-test-and-stage-deployment) + - [5.3 Manual Deployment through PROD](#53-manual-deployment-through-prod) + - [5.4 Local Development, Validation, and Manual Deployment](#54-local-development-validation-and-manual-deployment) +6. [Destroying Resources and Cleanup](#6-destroying-resources-and-cleanup) + - [6.1 Destroy an Individual Frontend Stack](#61-destroy-an-individual-frontend-stack) + - [6.2 Destroy All Stacks in One Environment](#62-destroy-all-stacks-in-one-environment) + - [6.3 Destroy the Configured Stacks in All Three Environments](#63-destroy-the-configured-stacks-in-all-three-environments) +7. [Security, State Management, and Design Decisions](#7-security-state-management-and-design-decisions) +8. [Verification and Evidence](#8-verification-and-evidence) +9. [Known Limitations and Improvements](#9-known-limitations-and-improvements) + +## 1. Purpose and scope + +This proof of concept addresses the problems described in the TerraNova scenario: unreviewed changes to `main`, inconsistent code quality, deployments from individual workstations, and configuration drift between Git and deployed infrastructure. Infrastructure is defined in version-controlled OpenTofu configurations and validated before changes are automatically deployed to TEST and STAGE. Production deployment requires a separate manual workflow run, with environment approval provided by GitHub environment protection rules **when those rules are configured**. + +The implementation provisions **two independently managed stacks per environment**: + +1. **Network:** a private OpenStack network, frontend/backend/database subnets, an external router, and router interfaces. +2. **Frontend:** a Debian 13 virtual machine running Nginx, with a private IP, a floating IP, and security group rules for HTTP and restricted SSH. + +The backend and database *subnets* are created as part of the network topology; there are no separate backend application or database VM stacks in this proof of concept. + +### Implementation status + +| Capability | Status | +| --- | --- | +| CI formatting, validation, linting, and security scanning | Implemented | +| Automatic TEST deployment after successful CI on `main` | Implemented; successful run observed | +| Automatic STAGE deployment after TEST | Implemented; successful run observed | +| Manual PROD deployment after TEST and STAGE | Implemented; successful run reported | +| Two OpenTofu stacks in each environment | Implemented | +| Environment- and stack-specific remote state | Implemented | +| Reusable network and VM modules | Implemented (local modules) | +| GitHub Actions destruction workflow | **Not implemented** | +| Manual destruction using OpenTofu CLI | Documented below; not automated | +| Independent, tagged releases of the reusable modules | **Not implemented**; modules are versioned with repository commits | + +The repository's workflow definitions alone do not prove that repository branch protection, required reviewers, instructor permissions, or runner availability have been enabled. These are administrative prerequisites described below. + +## 2. Architecture and repository structure + +```text +assigment2-iac/ +├── .github/workflows/ +│ ├── ci.yml # Formatting, validation, TFLint, Checkov +│ ├── deploy.yml # TEST → STAGE → conditional PROD +│ └── runner-test.yml # Manually triggered runner health check +├── bootstrap/ +│ ├── cloud_init/runner.yaml # Runner VM operating-system packages +│ └── *.tf # Runner network, router, VM, security groups +├── environments/ +│ ├── test/ +│ │ ├── network/ # TEST network state and configuration +│ │ └── frontend/ # TEST frontend state and configuration +│ ├── stage/ +│ │ ├── network/ +│ │ └── frontend/ +│ └── prod/ +│ ├── network/ +│ └── frontend/ +├── modules/ +│ ├── network/ # Reusable OpenStack network resources +│ └── vm/ # Reusable VM, SG, IP, optional volume attach +├── .tflint.hcl +├── .gitignore +└── README.md +``` + +Every environment has its own network and frontend root module, its own `deployment.tfvars`, and its own `.terraform.lock.hcl`. The frontend stack uses `terraform_remote_state` to read the network ID and frontend subnet ID from the corresponding environment's network state. Therefore **network must be created before frontend, and frontend must be destroyed before network**. + +### Environment configuration + +| Environment | Network CIDRs (frontend / backend / database) | Network state suffix | Frontend state suffix | +| --- | --- | --- | --- | +| TEST | `10.60.10.0/24` / `10.60.20.0/24` / `10.60.30.0/24` | `assignment2-test-network` | `assignment2-test-frontend` | +| STAGE | `10.70.10.0/24` / `10.70.20.0/24` / `10.70.30.0/24` | `assignment2-stage-network` | `assignment2-stage-frontend` | +| PROD | `10.80.10.0/24` / `10.80.20.0/24` / `10.80.30.0/24` | `assignment2-prod-network` | `assignment2-prod-frontend` | + +All six stacks use the **Kubernetes OpenTofu backend** in namespace `iac`, with separate `secret_suffix` values. This isolates their state even though they use the same existing Kubernetes cluster. The kOps configuration used to obtain access to that cluster is stored separately in Swift at `swift://demo-kops`. **The Swift kOps state store is not the OpenTofu state backend**. + +The `bootstrap/` root module creates the dedicated self-hosted runner VM and its supporting OpenStack network. Unlike the six workload stacks, its provided `providers.tf` does **not** declare a remote state backend; the bootstrap state must be safeguarded separately by the operator. + +## 3. CI/CD workflow + +### 3.1 Developer workflow and pull requests + +1. Pull the latest `main` and create a branch, for example `feat/add-security-rule` or `fix/frontend-config`. +2. Make changes to the appropriate environment, module, or workflow files. +3. Run local formatting, validation, and relevant lint/security checks (section 4). +4. Commit and push the feature branch. +5. Open a pull request targeting `main`. Review the OpenTofu changes and verify the CI job passes before merging. +6. Merge the reviewed change into `main`. The successful push CI run triggers the deployment workflow. + +Recommended repository settings: protect `main`, require pull requests, require successful CI checks, restrict direct pushes, and require review by another collaborator. **These policies must be verified in repository settings; they are not enforced by the YAML files themselves.** + +### 3.2 Continuous integration — `.github/workflows/ci.yml` + +CI is triggered on **pull requests targeting `main`**, **pushes to `main`**, or manual **Run workflow**. It runs on the self-hosted runner labeled `[self-hosted, linux, assignment2]` and performs: + +1. Checkout with `persist-credentials: false`. +2. Install OpenTofu **1.12.6**. +3. `tofu fmt -check -recursive` for consistent formatting. +4. `tofu init -backend=false -input=false -lockfile=readonly` and `tofu validate -no-color` for the bootstrap and all six environment stacks. Using `-backend=false` avoids accessing deployment state during static validation. +5. TFLint **v0.64.0** on the repository's OpenTofu configurations. +6. Checkov **3.4.0**, scanning both Terraform/OpenTofu and GitHub Actions definitions. + +A failed check causes CI to fail. Only a successful **push-originated** CI run on `main` meets the automatic deployment workflow's trigger condition; a pull-request CI run does not deploy infrastructure. + +### 3.3 Deployment — `.github/workflows/deploy.yml` + +```text +Pull request ──> CI validation ──> review/merge into main + │ + CI on push to main + │ success + ▼ + Deploy TEST + │ success + ▼ + Deploy STAGE + │ + PROD skipped + +Manual "Deploy Infrastructure" run from main + │ + └──> TEST ──> STAGE ──> [PROD environment approval*] ──> PROD + +* Approval is required only if configured in the GitHub PROD environment. +``` + +`deploy.yml` is triggered by a completed CI workflow on `main` or by `workflow_dispatch`. The TEST job checks that CI succeeded and was triggered by a push to `main`, or that the deployment was manually dispatched from `main`. STAGE has `needs: deploy-test`, so it runs only after TEST succeeds. PROD has `needs: deploy-stage` and an additional condition requiring a **manual `workflow_dispatch` from `main`**. There is no `deploy_prod` checkbox in this version: **every manual run of Deploy Infrastructure from `main` attempts PROD after TEST and STAGE**. + +For each environment, the deployment job checks OpenStack credentials, installs checksum-verified `kubectl` and `kOps`, obtains a short-lived **1-hour Kubernetes admin kubeconfig**, verifies access to the `iac` namespace, initializes the backends, then processes **network before frontend**. OpenTofu saves each plan and inspects the plan JSON with `jq`. The pipeline only allows `no-op`, `read`, `create`, and `update` resource actions; it rejects planned deletions and replacements. It then applies the exact saved plan, or skips application when there are no changes. Temporary plans, logs, and kubeconfig files are removed in cleanup steps. + +**Current limitation:** TEST has a `Verify TEST remote state` step which requires **non-empty existing managed state** in both TEST stacks. This protects the current TEST deployment but means the automated TEST job **cannot bootstrap TEST from completely empty state** or immediately recreate it after destroying both stacks. A local initial creation or a change to that preflight step is required. STAGE and PROD do not have this non-empty-state restriction. + +## 4. Prerequisites and credentials + +The workflow assumes the following pre-existing infrastructure and permissions: + +- An OpenStack project in NTNU **SkyHiGh**, with available quotas, suitable images/flavors, external network `ntnu-internal`, and an existing SSH keypair. +- OpenStack **application credentials** that can manage the intended network and compute resources. Separate deployment credentials should be configured per GitHub environment (`TEST`, `STAGE`, `PROD`). +- An accessible kOps Kubernetes cluster named `einalien.k8s.local`, whose kOps state is in `swift://demo-kops`. +- Kubernetes namespace **`iac`**, plus permission to read/update Secrets and use Lease objects needed for backend locking. +- A running, registered GitHub Actions self-hosted Linux runner labeled `self-hosted`, `linux`, `assignment2`, with outbound access to the OpenStack APIs, the cluster, GitHub Actions, and the relevant download endpoints. +- `jq`, Git, Python 3 with venv, curl, and the libraries required by the runner. `bootstrap/cloud_init/runner.yaml` installs these OS packages, **but does not register the GitHub runner agent**. Agent registration and long-running service setup must be completed separately. +- In GitHub's repository settings, environment names **TEST**, **STAGE**, and **PROD**, with appropriate environment-specific credentials. Configure **required reviewers for PROD** if a human approval gate is required; the YAML's manual trigger alone is not a reviewer approval. + +### GitHub Actions environments: required configuration + +Configure these values under each environment's secrets/variables as applicable: + +| Name | Type | Used for | +| --- | --- | --- | +| `OS_AUTH_URL` | Variable | OpenStack identity endpoint | +| `OS_REGION_NAME` | Variable | OpenStack region | +| `OS_APPLICATION_CREDENTIAL_ID` | Secret | Environment-specific OpenStack deployment credential ID | +| `OS_APPLICATION_CREDENTIAL_SECRET` | Secret | Environment-specific deployment credential secret | +| `KOPS_APPLICATION_CREDENTIAL_ID` | Secret | Credential used to read kOps state in Swift | +| `KOPS_APPLICATION_CREDENTIAL_SECRET` | Secret | Secret associated with the kOps credential | +| `SSH_KEY_NAME` | Variable | Name of an existing OpenStack SSH keypair | +| `ADMIN_CIDR` | Secret | Trusted IPv4 CIDR permitted to access SSH (port 22) | + +The workflow additionally sets `OS_AUTH_TYPE=v3applicationcredential`, `OS_IDENTITY_API_VERSION=3`, and `OS_INTERFACE=public`. No actual credentials should be committed to the repository. Store the application credential values only in GitHub Secrets or local, ignored credential files. + + + +## 5. Running the workflow end to end + +### 5.1 Runner verification + +1. Verify the runner VM is powered on in OpenStack. +2. Verify it appears **Online** in repository **Settings → Actions → Runners** with label `assignment2`. +3. Go to **Actions → Runner Test → Run workflow**, select `main`, and confirm that the runner and OpenTofu version checks succeed. + +The bootstrap configuration for the runner is in `bootstrap/`. An administrator with OpenStack credentials can initialize it locally using a private `terraform.tfvars` based on `bootstrap/terraform.tfvars.example`, then run `tofu init`, `tofu plan`, and `tofu apply`. Preserve the bootstrap state; do not destroy the runner VM while relying on it for CI/CD. + +### 5.2 Automatic TEST and STAGE deployment + +1. Create a branch, push it, and open a PR into `main`. +2. Wait for **CI** to pass and complete the review/merge process. +3. The push to `main` runs **CI** again. When it succeeds, **Deploy Infrastructure** automatically starts. +4. Verify the jobs **Deploy TEST → Deploy STAGE** succeeded. **Deploy PROD** should be **Skipped** on this automatic run. +5. Confirm the OpenStack network, router, subnets, and frontend VM exist in TEST and STAGE. + +**Note:** The automatic TEST path assumes the two TEST states were created before the run; see the limitation in section 3.3. + +### 5.3 Manual deployment through PROD + +1. Open the repository on `git.ntnu.no` and select **Actions → Deploy Infrastructure**. +2. Click **Run workflow** and select **`main`**. This manually invokes the pipeline even if no new commit has been pushed. +3. Wait for **Deploy TEST** and **Deploy STAGE** to succeed. +4. If the PROD environment has required reviewers, an authorized reviewer selects **Review deployments → PROD → Approve and deploy**. +5. Verify **Deploy PROD** succeeds, and inspect the created resources and floating IP in OpenStack. + +Do **not** select **Re-run all jobs** on an automatically triggered run to deploy PROD: it retains the original event and the PROD condition remains false. Start a **new manual workflow run** instead. + +### 5.4 Local development, validation, and manual deployment + +The commands below assume a Bash shell (e.g., WSL Ubuntu) from the repository root and OpenTofu 1.12.6 or a compatible version installed. Configure the OpenStack CLI/application credentials from a secure, untracked local file; never commit the actual values. + +```bash +# Clone and create a working branch +git clone https://git.ntnu.no/einalien/assigment2-iac.git +cd assigment2-iac +git switch -c feat/example-change + +# Format and validate all root configurations +tofu fmt -recursive +tofu fmt -check -recursive +for dir in bootstrap environments/{test,stage,prod}/{network,frontend}; do + tofu -chdir="$dir" init -backend=false -input=false -lockfile=readonly + tofu -chdir="$dir" validate -no-color +done + +# The CI workflow also runs TFLint and Checkov; see ci.yml for exact versions. +``` + +To use the remote Kubernetes state locally, first obtain a valid kubeconfig. **These commands require authorized access** and the operator's private kOps credentials: + +```bash +# Run in WSL/Linux, after privately loading kOps/Swift application credentials. +export KOPS_STATE_STORE="swift://demo-kops" +export KUBECONFIG="$HOME/.kube/assignment2-kubeconfig" +mkdir -p "$HOME/.kube" +umask 077 + +kops export kubeconfig einalien.k8s.local \ + --admin=1h \ + --state="$KOPS_STATE_STORE" \ + --kubeconfig="$KUBECONFIG" + +kubectl get namespace iac +``` + +Then load the appropriate **target environment's** OpenStack deployment credentials and set the two frontend variables privately. For example, after sourcing your own ignored credential file: + +```bash +# Placeholders: use your actual existing keypair and trusted CIDR. +export TF_VAR_ssh_key_name="YOUR_OPENSTACK_KEYPAIR_NAME" +export TF_VAR_admin_cidr="YOUR_TRUSTED_IP/32" +``` + +To **plan and create or update one environment** (substitute `test`, `stage`, or `prod`), run network first: + +```bash +ENVIRONMENT=stage + +for STACK in network frontend; do + DIR="environments/$ENVIRONMENT/$STACK" + tofu -chdir="$DIR" init -reconfigure -input=false -lockfile=readonly + tofu -chdir="$DIR" plan -var-file=deployment.tfvars + tofu -chdir="$DIR" apply -var-file=deployment.tfvars + done +``` + +OpenTofu prompts for confirmation of each apply. Choose the appropriate credentials **before** running this sequence, especially if the environments use different OpenStack application credentials. Local plans may include destructive changes; unlike `deploy.yml`, this command sequence does not implement the CI pipeline's JSON action filter. Review each plan carefully before approval. + +## 6. Destroying resources and cleanup + +**Current status:** `.github/workflows/deploy.yml` rejects deletions/replacements, and there is **no `.github/workflows/destroy.yml`** in the submitted implementation. Destruction therefore **cannot be triggered through GitHub Actions at present**. The following is a **manual operator procedure**, not an implemented destruction pipeline. It provides a reproducible CLI recipe for a selected stack or environment. A separate manually gated destroy workflow would be needed to fully satisfy the automated lifecycle requirement. + +**Warning:** Destruction deletes real OpenStack resources. Obtain authorization, capture the required deployment evidence, protect all remote state, and confirm the target environment before proceeding. Never destroy `bootstrap/` while it hosts the required runner. Never destroy the external kOps cluster because the six stack states depend on it. + +### 6.1 Destroy an individual frontend stack + +Ensure the local Kubernetes kubeconfig and correct OpenStack credentials are active as described in section 5.4. For example, for STAGE: + +```bash +ENVIRONMENT=stage +DIR="environments/$ENVIRONMENT/frontend" + +tofu -chdir="$DIR" init -reconfigure -input=false -lockfile=readonly +tofu -chdir="$DIR" plan -destroy -var-file=deployment.tfvars +tofu -chdir="$DIR" destroy -var-file=deployment.tfvars +``` + +This removes the frontend VM and its managed dependencies, while leaving the network stack intact. **Do not destroy the network first** while any frontend VM still uses its subnet and network remote-state outputs. + +### 6.2 Destroy all stacks in one environment + +Change the selected environment as needed (`test`, `stage`, `prod`). Use the **correct environment-specific** deployment credentials before running. Both stacks are processed in reverse dependency order: + +```bash +ENVIRONMENT=stage + +for STACK in frontend network; do + DIR="environments/$ENVIRONMENT/$STACK" + tofu -chdir="$DIR" init -reconfigure -input=false -lockfile=readonly + tofu -chdir="$DIR" plan -destroy -var-file=deployment.tfvars + tofu -chdir="$DIR" destroy -var-file=deployment.tfvars + done +``` + +### 6.3 Destroy the configured stacks in all three environments + +Run the **section 6.2 sequence separately for PROD, STAGE, and TEST**, in that order. **Switch to the correct OpenStack deployment credentials before each environment**. This deliberate, environment-by-environment process avoids accidentally using PROD permissions or configuration for another environment. Collect a successful OpenTofu destroy log and resource inventory screenshot for each environment if destruction evidence is required. + +After a complete TEST destroy, the current automated `deploy-test` job will fail its non-empty remote-state check. TEST must be re-created manually using section 5.4, or the preflight check must be adapted to accept empty state, before normal automated deployments can resume. + +## 7. Security, state management, and design decisions + +- **Git as the intended source of truth:** changes are reviewed and merged before automatic application, reducing configuration drift and unreviewed local changes. +- **Environment isolation:** TEST/STAGE/PROD have separate configuration folders, CIDRs, GitHub Environments, and Kubernetes state keys. Frontend reads only the corresponding network state. +- **State and locking:** the Kubernetes backend stores OpenTofu state in namespace `iac`. The pipeline checks Kubernetes API and Lease permissions before deploying. Protect access to the cluster and its Secrets, as state may include sensitive data. +- **Limited authentication exposure:** OpenStack uses application credentials and GitHub environment secrets; kOps generates a short-lived kubeconfig. The deployment workflow removes temporary kubeconfig, plans, and log files in cleanup steps. +- **Static quality gates:** formatting, validation, TFLint, and Checkov execute before automatic deployment. Provider selections are constrained by `.terraform.lock.hcl` and version requirements. +- **Conservative deployment:** each saved plan is checked for destructive actions before it can be applied, and concurrency prevents overlapping infrastructure deployments. Destructive changes require a distinct operator procedure. +- **Network exposure:** frontend permits HTTP on port 80 from the configured `http_cidr` (currently `0.0.0.0/0`), and SSH on port 22 only from `admin_cidr`. This is a demonstration Nginx service, not a hardened application delivery platform. +- **Reuse and versioning:** `modules/network` and `modules/vm` are reused across environments via relative source paths. Module code is therefore versioned at the **repository commit level**, with the same module revision used by each environment at a given commit. There is **no independent module release/tag or registry version pinning**. Provider versions are constrained in code and resolved in committed lock files. + +For reuse across multiple customers, a future iteration could move the modules into independently tagged releases, adopt reusable deployment workflows with customer-specific configuration, use separate customer state backends and accounts, and add policy-as-code checks and automated, approved destroy operations. + +## 8. Verification and evidence + +Verify a successful end-to-end deployment in **Actions → Deploy Infrastructure**: + +- Automatic CI-triggered run: **TEST succeeded**, **STAGE succeeded**, **PROD skipped** (expected). +- Manually dispatched run from `main`: **TEST succeeded**, **STAGE succeeded**, **PROD succeeded**. +- OpenStack dashboard: expected private networks, subnets, routers, and at least one frontend VM per environment. +- Optional workload test: identify a frontend floating IP from the OpenTofu outputs or OpenStack dashboard and verify the default Nginx HTTP page is reachable on port 80. + + +## 9. Known limitations and improvements + +1. **No GitHub Actions destroy workflow:** destruction is manual, and the assignment's automated destruction requirement is not fully met. +2. **TEST cannot automatically initialize empty state:** its remote-state preflight requires existing managed resources. STAGE and PROD permit initial creation. +3. **PROD approval depends on GitHub settings:** manual triggering is implemented in YAML, but required reviewers must be enabled administratively. +4. **Module versions are tied to the repository commit:** independently published and pinned module versions would improve customer reuse and rollback. +5. **Runner and kOps dependencies are external:** the self-hosted runner agent registration, continuous availability, kOps cluster, Kubernetes namespace, access control, and instructor permissions require separate setup and maintenance. +6. **Bootstrap state is not remotely configured in the provided code:** its state needs separate backup and lifecycle protection. +7. **CI/CD tests are infrastructure-focused:** end-to-end application or Nginx availability tests are not implemented; the assignment permits workload-specific tests to be omitted.