{"id":47,"date":"2024-10-06T00:36:21","date_gmt":"2024-10-06T00:36:21","guid":{"rendered":"https:\/\/blogs.oregonstate.edu\/ozzythecomputerguy\/?p=47"},"modified":"2024-10-06T00:50:53","modified_gmt":"2024-10-06T00:50:53","slug":"overcoming-azure-devops-pipeline-challenges-with-terraform-for-aks-deployments","status":"publish","type":"post","link":"https:\/\/blogs.oregonstate.edu\/ozzythecomputerguy\/2024\/10\/06\/overcoming-azure-devops-pipeline-challenges-with-terraform-for-aks-deployments\/","title":{"rendered":"Overcoming Azure DevOps Pipeline Challenges with Terraform for AKS Deployments"},"content":{"rendered":"\n

Day two of developing KubeConductor is officially done. <\/p>\n\n\n\n

Today, I focused on configuring an Azure DevOps CI\/CD pipeline to deploy an Azure Kubernetes Service (AKS) cluster using Terraform. I’ll touch on the hurdles encountered while provisioning the infrastructure and managing existing resources, as well as the steps taken to resolve them.<\/p>\n\n\n\n

The Code<\/h2>\n\n\n\n

My public repo can he found [here<\/a>]. <\/p>\n\n\n\n

aks-cluster.tf<\/strong>: This is the Terraform configuration file that provisions an Azure Kubernetes Service (AKS) cluster using the AzureRM provider.<\/p>\n\n\n\n

# aks-cluster.tf\nprovider \"azurerm\" {\n  features {}\n\n  client_id       = var.client_id\n  client_secret   = var.client_secret\n  tenant_id       = var.tenant_id\n  subscription_id = var.subscription_id\n}\n\n\nresource \"azurerm_resource_group\" \"default\" {\n  name     = \"terraform-aks-rg\"\n  location = \"West US\"\n}\n\nresource \"azurerm_kubernetes_cluster\" \"default\" {\n  name                = \"terraform-aks-cluster\"\n  location            = azurerm_resource_group.default.location\n  resource_group_name = azurerm_resource_group.default.name\n  dns_prefix          = \"terraform-aks\"\n\n  default_node_pool {\n    name            = \"default\"\n    node_count      = 2\n    vm_size         = \"Standard_DS2_v2\"\n    os_disk_size_gb = 30\n  }\n\n  identity {\n    type = \"SystemAssigned\"\n  }\n\n  role_based_access_control_enabled = true\n\n  tags = {\n    environment = \"Development\"\n  }\n}<\/code><\/pre>\n\n\n\n
variables.tf:<\/strong> This Terraform configuration file defines a set of variables used in the aks-cluster.tf<\/code> file to dynamically configure the Azure resources<\/p>\n\n\n\n
# variables.tf\nvariable \"resource_group_name\" {\n  description = \"Name of the Resource Group\"\n  default     = \"terraform-aks-rg\"\n}\n\nvariable \"client_id\" {\n  description = \"The Client ID of the Service Principal\"\n}\n\nvariable \"client_secret\" {\n  description = \"The Client Secret of the Service Principal\"\n}\n\nvariable \"tenant_id\" {\n  description = \"The Tenant ID of the Azure Active Directory\"\n}\n\nvariable \"subscription_id\" {\n  description = \"The Subscription ID where the resources will be created\"\n}<\/code><\/pre>\n\n\n\n
versions.tf:<\/strong> This Terraform configuration file specifies the required Terraform and provider versions needed. It helps ensure compatibility and stability by enforcing versioning.<\/p>\n\n\n\n
# versions.tf\nterraform {\n  required_version = \">= 0.14\"\n  required_providers {\n    azurerm = {\n      source  = \"hashicorp\/azurerm\"\n      version = \">= 2.56\"\n    }\n  }\n}<\/code><\/pre>\n\n\n\n
outputs.tf:<\/strong> This Terraform configuration file defines output variables for the infrastructure. In my case, these outputs are used to display values after the resources have been created. This functionality is not implemented yet, however. <\/p>\n\n\n\n
# outputs.tf\noutput \"kubernetes_cluster_name\" {\n  value = azurerm_kubernetes_cluster.default.name\n}\n\noutput \"resource_group_name\" {\n  value = azurerm_resource_group.default.name\n}<\/code><\/pre>\n\n\n\n
Initial Pipeline Configuration<\/strong><\/h2>\n\n\n\n
My goal for today was to automate the provisioning of an AKS cluster using Terraform in an Azure DevOps pipeline. The basic setup included a self-hosted Ubuntu agent and a multi-step pipeline with the following stages:<\/p>\n\n\n\n
Terraform Init:<\/strong> Initializes a new or existing Terraform configuration<\/p>\n\n\n\n
Terraform Plan: <\/strong>Generates an execution plan to show what changes Terraform will make to your infrastructure.<\/p>\n\n\n\n
Terraform Apply: <\/strong>Applies the changes required to reach the desired state of the configuration.<\/p>\n\n\n\n
Below is a screenshot of the steps passing the CI\/CD pipeline, which runs on a self-hosted Ubuntu machine. <\/p>\n\n\n\n
\"\"<\/figure>\n\n\n\n
<\/p>\n\n\n\n
Debugging Pipeline Environment Variables<\/h2>\n\n\n\n
From the beginning, I knew I wanted an application that mirrored real-world cloud infrastructure as closely as possible. This meant that I needed to be careful with how I handle secret values such as IDs, passwords, etc. I opted to use Azure Pipeline Group Variables to store the sensitive information relating to my cloud infrastructure. My CI\/CD pipeline fetches these variables on runtime, eliminating the need to hard-code these values. <\/p>\n\n\n\n
Terraform relies on a consistent mapping of environment variables for Azure authentication. I encountered errors with variables like client_id<\/code> and subscription_id<\/code> not being recognized correctly. To fix this issue, I standardized the variable mapping using the TF_VAR_<\/code> prefix to align Terraform\u2019s environment variables (TF_VAR_client_id<\/code>, TF_VAR_client_secret<\/code>, TF_VAR_tenant_id<\/code>, and TF_VAR_subscription_id<\/code>) with Azure\u2019s environment requirements (ARM_CLIENT_ID<\/code>, ARM_CLIENT_SECRET<\/code>, etc.).<\/p>\n\n\n\n
Here is the updated yml to match those requirements:<\/p>\n\n\n\n
# Updated azure-pipelines.yml with complete TF_VAR_ variable mapping\n\ntrigger:\n  branches:\n    include:\n      - main\n\npool:\n  name: \"SelfHostedUbuntu\"\n\nvariables:\n  - group: Terraform-SP-Credentials\n\njobs:\n  - job: \"Deploy_AKS\"\n    displayName: \"Provision AKS Cluster Using Terraform\"\n    steps:\n      # Step 1: Checkout Code\n      - checkout: self\n\n      # Step 2: Verify Terraform Version\n      - script: |\n          terraform --version\n        displayName: \"Verify Installed Terraform Version\"\n\n      # Step 3: Terraform Init (Set Working Directory and Pass All Variables with TF_VAR_ Prefix)\n      - script: |\n          terraform init\n        displayName: \"Terraform Init\"\n        workingDirectory: $(System.DefaultWorkingDirectory)\/terraform\n        env:\n          ARM_CLIENT_ID: $(appId)\n          ARM_CLIENT_SECRET: $(password)\n          ARM_TENANT_ID: $(tenant)\n          ARM_SUBSCRIPTION_ID: $(AZURE_SUBSCRIPTION_ID)\n          TF_VAR_client_id: $(appId) # Client ID\n          TF_VAR_client_secret: $(password) # Client Secret\n          TF_VAR_tenant_id: $(tenant) # Tenant ID\n          TF_VAR_subscription_id: $(AZURE_SUBSCRIPTION_ID) # Subscription ID\n\n      # Step 4: Terraform Plan (Pass All Variables with TF_VAR_ Prefix)\n      - script: |\n          terraform plan -out=tfplan\n        displayName: \"Terraform Plan\"\n        workingDirectory: $(System.DefaultWorkingDirectory)\/terraform\n        env:\n          ARM_CLIENT_ID: $(appId)\n          ARM_CLIENT_SECRET: $(password)\n          ARM_TENANT_ID: $(tenant)\n          ARM_SUBSCRIPTION_ID: $(AZURE_SUBSCRIPTION_ID)\n          TF_VAR_client_id: $(appId)\n          TF_VAR_client_secret: $(password)\n          TF_VAR_tenant_id: $(tenant)\n          TF_VAR_subscription_id: $(AZURE_SUBSCRIPTION_ID)\n      # Step 5: Terraform Apply (Set Working Directory and Pass All Variables)\n      - script: |\n          terraform apply -auto-approve tfplan\n        displayName: \"Terraform Apply\"\n        workingDirectory: $(System.DefaultWorkingDirectory)\/terraform\n        env:\n          ARM_CLIENT_ID: $(appId)\n          ARM_CLIENT_SECRET: $(password)\n          ARM_TENANT_ID: $(tenant)\n          ARM_SUBSCRIPTION_ID: $(AZURE_SUBSCRIPTION_ID)\n          TF_VAR_client_id: $(appId)\n          TF_VAR_client_secret: $(password)\n          TF_VAR_tenant_id: $(tenant)\n          TF_VAR_subscription_id: $(AZURE_SUBSCRIPTION_ID)<\/code><\/pre>\n\n\n\n
Azure CLI Summary:<\/strong><\/h2>\n\n\n\n
Here’s a list of Azure CLI commands I used today. Sometimes these were used for debugging, sanity checking, or they were just necessary for Terraform to work. <\/p>\n\n\n\n
1. Login to Azure Using Service Principal:<\/strong><\/h3>\n\n\n\n
Authenticates to Azure using Service Principal credentials (appId<\/code>, password<\/code>, and tenant<\/code>).<\/p>\n\n\n\n
az login --service-principal -u <appId> -p <password> --tenant <tenant><\/pre>\n\n\n\n
2. Retrieve AKS Cluster Credentials:<\/strong><\/h3>\n\n\n\n
Fetches the AKS cluster credentials and configures kubectl<\/code> to use this cluster.<\/p>\n\n\n\n
az aks get-credentials --resource-group <resource_group_name> --name <kubernetes_cluster_name><\/pre>\n\n\n\n
3. Browse Kubernetes Dashboard:<\/strong><\/h3>\n\n\n\n
Opens the Kubernetes dashboard for the specified AKS cluster in the Azure Portal.<\/p>\n\n\n\n
az aks browse --resource-group <resource_group_name> --name <kubernetes_cluster_name><\/pre>\n\n\n\n
4. Create Service Principal:<\/strong><\/h3>\n\n\n\n
Creates an Azure Active Directory Service Principal for authentication in Terraform.<\/p>\n\n\n\n
az ad sp create-for-rbac --skip-assignment<\/pre>\n\n\n\n
Takeaways<\/h2>\n\n\n\n
State Management Needs Improvement:<\/strong> The pipeline deploys the AKS cluster and pods successfully if the resource group does not exist. However, the pipeline fails since state management is not set up yet — this a task that requires configuring a remote state backend. <\/p>\n\n\n\n
Streamlined Pipeline Steps:<\/strong> The final pipeline configuration effectively provisions the AKS cluster but needs fine-tuning for state management and validation.<\/p>\n\n\n\n
Next Steps:<\/strong><\/h3>\n\n\n\n