Ansible playbook deployment in production using AWX

Q: How do you clone an Azure DevOps repository and create a child/nested branch (branch from a branch) in VS Code?

Answer Breakdown

1. Cloning the Repository from Azure DevOps in VS Code

Get the Clone URL:
• Navigate to your project in Azure DevOps.
• Go to Repos -> Files (or Branches).
• Click the Clone button in the top-right corner and copy the HTTPS or SSH URL (or click Clone in VS Code for a direct integrated launch). [1, 2, 3]
Clone in VS Code:
• Open Visual Studio Code.
• Open the Command Palette (Ctrl+Shift+P on Windows/Linux or Cmd+Shift+P on macOS) or click the Source Control icon on the activity bar.
• Type Git: Clone and press Enter.
• Paste the Azure DevOps repository URL and press Enter.
• Choose a local directory on your computer where the project files should reside, and click Select Repository Location.
• Click Open when prompted, and trust the authors if asked. [1, 2, 3, 4]

2. Creating a Branch from a Branch (Nested/Child Branch) in VS Code

In Git, all branches are created from a "base" or "source" branch. Creating a branch from an existing feature branch (a branch within a branch) simply means choosing your current feature branch (instead of main or master) as the reference point. [1, 2]

Method A: Using the Status Bar (Bottom Left)
• Switch to or make sure you are currently checked out on the parent branch you want to branch off from (for example, feature/login-ui).
• Click on the current branch name displayed in the bottom-left corner of the VS Code status bar.
• Select Create new branch from... at the top of the dropdown menu.
• Select your desired source/parent branch (e.g., origin/feature/login-ui or your local feature/login-ui).
• Type your new child branch name (e.g., feature/login-ui-validation) and press Enter. [1, 2, 3, 4, 5]
Method B: Using the Command Palette
• Press Ctrl+Shift+P (or Cmd+Shift+P).
• Type Git: Create Branch From... and select it.
• Choose the parent branch from the list.
• Enter the new branch name and press Enter. [1, 2]

3. Publishing (Pushing) the New Branch to Azure DevOps

By default, the newly created branch exists locally on your machine. [1, 2]
To push it up to Azure DevOps so it shows up in Azure Repos:
• Click the Publish Branch button in the Source Control panel or click the cloud/sync icon in the bottom status bar, or run git push -u origin <branch-name> in the VS Code terminal.
• Verify in your browser under Azure Repos > Branches that your new child branch is now tracked remotely. [1, 2, 3]

Q. How to create branch in branch in vscode and clone repository from azure devops

Step 1: Clone the Repository from Azure DevOps

You can clone your repo into VS Code using either the direct portal shortcut or a manual URL copy. [1]

Method A: Direct Portal Link (Fastest)

Open your web browser, navigate to your project in Azure DevOps, and go to Repos. [1]
Click the Clone button in the top right corner. [1]
Under the IDE section, click Clone in VS Code. [1]
Your browser will prompt you to open Visual Studio Code. Click Open link.
VS Code will open and ask you to select a local folder destination. Choose your folder, and the cloning process will begin. [1]

Method B: Manual Copy-Paste

Go to your Azure DevOps Repos, click Clone, ensure HTTPS is highlighted, and click the Copy icon next to the URL. [1]
Open VS Code, and click the Source Control icon (the branching tree logo) on the left sidebar. [1]
Click Clone Repository and paste the copied URL into the top prompt box. [1]
Press Enter, select a local folder destination, and click Open when the clone completes. [1, 2]
Note: If prompted to log in, authenticate using your Azure DevOps credentials or a Personal Access Token (PAT). [1, 2]

---

Step 2: Create a "Branch in a Branch"

To create a new sub-branch off of an existing non-main branch, you must first switch to that parent branch and then branch off it. [1]

Using the VS Code Interface (UI)

Switch to your parent branch: Click the branch name in the bottom-left corner of the VS Code status bar (it might say main or dev). Select the existing branch you want to use as your base.
Pull the latest changes: Click the Sync or Pull icon next to it to make sure your local copy of this parent branch is perfectly up-to-date.
Create the sub-branch: Click the branch name in the bottom-left corner again.
Select Create new branch... from the dropdown menu.
Type a name for your new sub-branch (e.g., feature/login-fix) and press Enter.
VS Code will automatically check out (switch you into) your new branch. [1, 2, 3, 4]

Using the VS Code Terminal (CLI)

If you prefer using the built-in terminal (Ctrl + \`` or Cmd + ``), run these quick commands:

bash

# 1. Switch to your parent branch (e.g., 'dev')
git checkout dev

# 2. Get the latest code updates
git pull

# 3. Create and switch to your new sub-branch
git checkout -b feature/new-sub-branch

Use code with caution.

---

Step 3: Publish Your New Branch to Azure DevOps

Your new branch currently exists only on your local machine. To send it back up to Azure DevOps: [1]

Look at the bottom-left corner or the Source Control sidebar for a cloud icon or a button labeled Publish Branch.
Click Publish Branch.
Go back to your Azure DevOps web browser, refresh the Branches page, and you will see your newly pushed branch tracking under your projec

Q1: How do you set up an end-to-end automation pipeline that continuously onboards credentials, inventories, templates, surveys, and schedules into AWX from Azure DevOps?

Answer: You treat AWX configuration as Infrastructure as Code (IaC) or Configuration as Code (CaC). Instead of manual UI clicks, you store all AWX definitions (YAML or JSON format) in an Azure Repos Git repository.
An Azure DevOps CI/CD pipeline is then configured to trigger whenever a change is made to this repo. The pipeline runs a step utilizing either the Ansible awx.awx collection (specifically modules like awx.awx.credential, awx.awx.inventory, awx.awx.job_template, etc.) or the AWX REST API / awx CLI tool. This pipeline authenticates to AWX and sequentially provisions or updates the resources. 

Q2: In what order must you provision these elements in AWX via the pipeline, and why?

Answer: Order matters due to object dependencies:
1. Organizations & Users: The base tenancy layer.
2. Credentials: Needed for authenticating Git projects, cloud syncs, and host logins.
3. Projects: Imports the actual Ansible playbooks from Git.
4. Inventories: Defines the target machines or hosts.
5. Job Templates: Ties the Project, Inventory, and Credentials together.
6. Surveys & Schedules: Attached directly as sub-properties or child configurations of the created Job Template. 

---

2. Deep Dive: Component Provisioning

Q3: How do you securely onboard AWX Credentials from an Azure DevOps Pipeline without hardcoding passwords in Git?

Answer: You should never store plain-text secrets in Git. 

Securely store secrets (SSH keys, service principal keys, database passwords) inside Azure Key Vault. 
Link the Key Vault to an Azure DevOps Variable Group using a Service Connection. 
In the pipeline step, inject these variables as secure environment variables. 
Execute the awx.awx.credential module, passing the pipeline secret tokens into the execution block. AWX will accept them and instantly encrypt them at rest within its PostgreSQL database. 

Q4: How do you automate the onboarding of an Inventory that maps to Azure Virtual Machines?

Answer: You onboard a Dynamic Inventory. 

Create a Microsoft Azure Resource Manager credential type in AWX using the pipeline (requiring Azure Client ID, Secret, and Tenant ID). 
Define the inventory via the pipeline using the awx.awx.inventory module. 
Create an Inventory Source linked to that Azure credential using an azure_rm plugin config file. Ensure the option update_on_launch is set to true via the configuration code so AWX dynamically pulls the latest live VMs from Azure every time a job executes. 

Q5: How is an AWX Job Template defined and onboarded programmatically from a pipeline?

Answer: It is defined using the awx.awx.job_template module. The pipeline must supply:
name: Unique identification string for the template.
project: The name of the SCM-linked project containing the code.
playbook: The exact relative path to the .yml file inside the repository.
inventory: The inventory object it will run against.
credentials: A list of associated machine, cloud, or network credentials. 

Q6: How do you implement a Survey on a Job Template using Configuration as Code?

Answer: In AWX, a survey is a JSON/YAML-structured structure of questions that maps user input to extra_vars.
When deploying the job template via the awx.awx.job_template module or API, you include a survey_spec block. 

yaml

survey_enabled: true
survey_spec:
  name: "Deployment Specifications"
  description: "User inputs"
  spec:
    - question_name: "Target Environment"
      variable: "target_env"
      type: "multiplechoice"
      choices: "Dev\nQA\nProd"
      required: true

The pipeline updates this block, instantly enabling or changing UI dropdowns for end-users. [

Q7: How do you programmatically configure a Schedule for an onboarded template?

Answer: You use the awx.awx.schedule module or post to the template's /schedules/ API endpoint. The key parameter is the rrule (Recurrence Rule string conforming to RFC 5545 standards). For example, DTSTART:20260301T000000Z RRULE:FREQ=DAILY;INTERVAL=1 schedules the template to run every day at midnight starting March 2026. 

---

3. Troubleshooting & Best Practices

Q8: If your Azure DevOps pipeline completes successfully, but the newly onboarded Job Template fails with an "Inventory not found" error, what is the likely cause?

Answer: This is a classic race condition or dependency mapping error. It happens when:

The Inventory name string mapped inside the Job Template definition task does not exactly match the actual name string created during the Inventory provisioning task.
The Inventory task failed or skipped, meaning the template is pointing to a non-existent object ID.

Remediation: Enforce explicit task execution blocks or dependencies (depends_on or sequential pipeline steps) to ensure the inventory object is fully verified and present before the template provisioning task runs.

Q9: How do you ensure your Azure DevOps pipeline updates existing assets (idempotency) instead of creating duplicate records in AWX?

Answer: You leverage the idempotent nature of the awx.awx Ansible modules. By setting state: present across your automation scripts, the modules automatically check AWX for an existing entity with the same unique identifiers (like name and organization). If it exists, AWX patches/updates the modified fields; if it does not exist, it creates a new entry, entirely avoiding duplication. [1]

Q: How would you architect a process to onboard data from Azure DevOps into AWX using a Job Template, Survey, and Schedule?

Answer:
To implement this onboarding flow, I would establish a decoupled, API-driven integration between Azure DevOps (ADO) and AWX. The architecture relies on three distinct layers:

The Blueprint (Job Template): In AWX, I define a Job Template linked to the Git repository containing our configuration playbooks. This template is bound to the target environment inventory, appropriate execution credentials, and variables.
The Dynamic Input (Survey): I configure an interactive Survey directly on the AWX Job Template to handle on-demand data onboarding. The survey defines schema-validated fields (e.g., text, dropdowns, integers). When Azure Pipelines triggers AWX via its API, it acts as a non-interactive user, injecting the onboarding payload directly into these survey fields as runtime extra variables (extra_vars).
The Trigger/Automated Cadence (Schedule): Depending on the lifecycle requirement, scheduling can occur in two ways:
• Event-Driven (Continuous Delivery): The onboarding is triggered dynamically from an Azure Pipeline whenever a developer pushes a data manifest. The pipeline executes a curl command or uses an Ansible AWX Extension targeting the AWX REST API endpoint POST /api/v2/job_templates/<id>/launch/ with the survey variables.
• Time-Driven (Batch Processing): If data onboarding happens periodically (e.g., nightly syncs), I create an AWX Schedule natively tied to that Job Template. Because the template prompts for survey fields, I define the default onboarding extra_vars right inside the AWX schedule configuration.

---

Q: please interpret ansible.builtin.shell: set -o pipefail grep -e hugepage_total /proc/meminfo |awk '{print 42}' in ansible playbook

This line runs a shell command on a remote server through Ansible, forces pipeline safety with pipefail, searches /proc/meminfo for hugepage info, and uses awk to print the fixed number 42. 

---

Direct Answer & Breakdown

yaml

- name: Check hugepage and output 42
  ansible.builtin.shell: set -o pipefail; grep -e hugepage_total /proc/meminfo | awk '{print 42}'

Let's break down each piece of the command:

ansible.builtin.shell: The Ansible Shell Module executes the command string via a shell (defaulting to /bin/sh) on the target managed node. 
set -o pipefail: A bash/sh command option. In a normal Linux pipeline (cmd1 | cmd2), the return code of the entire pipeline is normally just the return code of the last command (cmd2). With -o pipefail enabled, the pipeline retains a failure exit code if any command in the chain fails. 
grep -e hugepage_total /proc/meminfo: Searches the kernel stats file /proc/meminfo for lines matching the pattern hugepage_total.
| (The Pipe): Takes the standard output of the grep command and passes it directly as standard input to the awk command.
awk '{print 42}': Ignores whatever text grep piped into it and unconditionally prints the number 42 to the standard output.

---

Complete Playbook Example

Here is a practical example showing how to capture the output of this shell command into an Ansible variable and register it for later use:

yaml

---
- name: Example playbook for shell pipefail and awk
  hosts: all
  gather_facts: false

  tasks:
    - name: Run pipeline and register result
      ansible.builtin.shell: >
        set -o pipefail;
        grep -e hugepage_total /proc/meminfo | awk '{print 42}'
      register: hugepage_output
      changed_when: false

    - name: Display the captured result
      ansible.builtin.debug:
        msg: "The command output is {{ hugepage_output.stdout }}"

Key Tutor Notes to Keep in Mind

Why pipefail? Without set -o pipefail, if grep fails to find hugepage_total and returns a non-zero exit code (e.g., 1), awk might still execute and return 0, tricking Ansible into thinking the task succeeded cleanly. pipefail catches that middle-step failure.
Hardcoded Output: Notice that awk '{print 42}' does not actually process the grep results; it simply outputs 42 regardless of what grep found. If you wanted to print the actual value from /proc/meminfo, you would typically use awk '{print $2}' instead.
Best Practice Warning: In many cases, executing complex shell pipelines with ansible.builtin.shell can be replaced with native Ansible modules or simpler commands if you only need raw data. 

or

The expression ansible.builtin.shell: set "-o" pipefail && grep "-e" hugepage_total /proc/meminfo | awk '{print $2}' runs a shell command on a remote Linux server via Ansible, safely checking hugepage statistics with strict error handling. [1, 2]

---

Breaking Down the Command

ansible.builtin.shell: The Ansible module that executes your command through a shell (/bin/sh or /bin/bash if specified) on the target node rather than running the binary directly. [1, 2]
set -o pipefail: A Bash option that changes how piped commands report success or failure. Normally, a pipeline (cmd1 | cmd2) only returns the exit code of the last command (cmd2). If grep fails or returns nothing, awk might still exit with 0 (success). With pipefail, the entire pipeline returns a failure exit code if any command in the chain fails. [1]
grep -e hugepage_total /proc/meminfo: Searches the system file /proc/meminfo for lines matching the pattern hugepage_total. [1]
| (Pipe): Sends the text output of the grep command directly into the input of the awk command.
awk '{print $2}': Splits the incoming line into space-separated columns and prints out only the second column (which holds the numeric value of the hugepages). [1]

---

Complete Playbook Example

Here is a clean, production-ready playbook example that uses this pattern, registers the output into a variable, and displays it:

yaml

---
- name: Check total HugePages on remote host
  hosts: all
  become: false
  tasks:
    - name: Extract HugePage Total value
      ansible.builtin.shell:
        cmd: set -o pipefail && grep -e HugePages_Total /proc/meminfo | awk '{print $2}'
        executable: /bin/bash
      register: hugepage_total_result
      changed_when: false

    - name: Print the extracted value
      ansible.builtin.debug:
        msg: "The total hugepages count is: {{ hugepage_total_result.stdout }}"

Use code with caution.

---

Key Tutor Notes

Why use executable: /bin/bash? The default shell in some minimal containers or distributions is /bin/sh (often dash), which may not support advanced set -o options cleanly. Explicitly setting executable: /bin/bash ensures pipefail works as intended. [1]
Why changed_when: false? Reading data from /proc/meminfo is a read-only action. Marking changed_when: false stops Ansible from reporting this read task as a "changed" state in your playbook summary. [1]
Alternative approach: If you only need to read a file, Ansible has built-in file reading modules or native

Deep-Dive Interview Scenarios

Q1: How do you pass variable data from an Azure DevOps YAML pipeline into an AWX template survey?

Answer: AWX maps survey input fields directly to Ansible extra_vars. In the Azure Pipeline, I extract the data to be onboarded (e.g., from a JSON file or pipeline variable) and make a secure REST API call to AWX.

yaml

# Snippet from Azure DevOps Pipeline
- task: Bash@3
  inputs:
    targetType: 'inline'
    script: |
      curl -X POST https://yourdomain.com \
        -H "Authorization: Bearer $(AWX_TOKEN)" \
        -H "Content-Type: application/json" \
        -d '{
          "extra_vars": {
            "onboard_user_name": "$(NewUser)",
            "onboard_department": "$(DeptID)"
          }
        }'

Note: Ensure "Prompt on Launch" is checked for variables in the AWX Job Template so it accepts these overrides.

Q2: If an AWX template has a strict Survey enabled, what happens if Azure DevOps forgets to pass one of the parameters?

Answer: It depends on how the survey field is configured:

If the survey field is marked Required and has no default value, the API call from Azure DevOps will fail with a 400 Bad Request, and the job will not launch.
To prevent pipeline breaking, best practices dictate setting safe Default Answers within the AWX Survey configuration. If Azure DevOps omits a variable, AWX automatically falls back to that default value.

Q3: How do you handle automated Scheduling if the onboarding template relies heavily on custom data inputs for each run?

Answer: Running a survey-driven template on a fixed schedule can be tricky because schedules run without human input. To solve this, AWX allows you to inject unique extra_vars per schedule instance.

I would configure the specific schedule inside AWX and use its metadata block to pass the fixed batch data payload required for that specific run cadence.
Alternatively, if the data is highly dynamic, I shift the schedule entirely to Azure DevOps using cron triggers (schedules: in YAML). The pipeline fetches the fresh onboarding data first, then passes it down to the AWX API dynamically.

Q4: How do you ensure secure credential handoffs during this cross-platform onboarding process?

Answer: Under no circumstances should raw passwords or service tokens pass through the pipeline code.

For Azure DevOps: I store the AWX API token inside an Azure DevOps Variable Group marked as a secret, or pull it dynamically from Azure Key Vault during the pipeline run.
For AWX: Target system access credentials (like SSH keys or API tokens needed to process the onboarded data) are saved natively in AWX Credentials. The template utilizes them implicitly without exposing them back to Azure DevOps.

Question: How to  onboard data and configure AWX (Ansible Automation Platform) dynamically from an Azure DevOps (ADO) Git repository

To onboard data and configure AWX (Ansible Automation Platform) dynamically from an Azure DevOps (ADO) Git repository, the industry standard is to treat your AWX assets (Job Templates, Surveys, and Schedules) as Configuration as Code (CaC). 

Instead of configuring things manually in the AWX user interface, you store your infrastructure definitions in Azure Repos and run an onboarding pipeline that updates AWX via the official awx.awx Ansible collection. 

---

1. Structure Your Azure DevOps Repository

Create a directory structure in your Azure Repo to house your definitions:

text

awx-onboard/
├── azure-pipelines.yml          # Your ADO pipeline definition
├── inventory.yml                # Local runner inventory targeting your AWX server
├── onboard-awx.yml              # Main playbook executing the AWX collection
└── vars/
    └── awx_assets.yml           # Data file containing Templates, Surveys, and Schedules

2. Define the Onboarding Data (vars/awx_assets.yml)

Define the template configurations, user surveys, and schedules directly inside a YAML file.

yaml

---
# Define variables for your target Job Template
target_project: "My ADO Project Source"
target_inventory: "Production Infrastructure"

job_templates:
  - name: "App Deployment Template"
    description: "Deploys our core application via Azure DevOps triggers"
    playbook: "playbooks/deploy.yml"
    project: "{{ target_project }}"
    inventory: "{{ target_inventory }}"
    state: "present"
    
    # Survey definition mapped to the Job Template
    survey_enabled: true
    survey:
      name: "Deployment Prompts"
      description: "Inputs required for the application layout"
      spec:
        - question_name: "Target Environment"
          question_description: "Select target environment"
          variable: "target_env"
          type: "multiplechoice"
          choices: ["staging", "production"]
          default: "staging"
          required: true
        - question_name: "App Port Number"
          question_description: "Enter port number for the app"
          variable: "app_port"
          type: "integer"
          min: 1024
          max: 65535
          default: 8080
          required: true

# Schedule definition mapped to the above Job Template
schedules:
  - name: "Daily Nightly Staging Sync"
    unified_job_template: "App Deployment Template"
    rrule: "DTSTART:20260301T000000Z RRULE:FREQ=DAILY;INTERVAL=1" # Runs daily at midnight UTC
    extra_data:
      target_env: "staging"
      app_port: 9000
    state: "present"

3. Build the Configuration Playbook (onboard-awx.yml)

This playbook maps your YAML definitions to active API instructions using native awx.awx collection modules. 

yaml

---
- name: Onboard Templates, Surveys, and Schedules to AWX
  hosts: localhost
  gather_facts: false
  vars_files:
    - vars/awx_assets.yml

  tasks:
    - name: Ensure Job Templates and Surveys are created
      awx.awx.job_template:
        controller_host: "{{ lookup('env', 'AWX_HOST') }}"
        controller_username: "{{ lookup('env', 'AWX_USER') }}"
        controller_password: "{{ lookup('env', 'AWX_PASSWORD') }}"
        validate_certs: false
        name: "{{ item.name }}"
        description: "{{ item.description }}"
        playbook: "{{ item.playbook }}"
        project: "{{ item.project }}"
        inventory: "{{ item.inventory }}"
        survey_enabled: "{{ item.survey_enabled | default(false) }}"
        survey_spec: "{{ item.survey | default(omit) }}"
        state: "{{ item.state | default('present') }}"
      loop: "{{ job_templates }}"

    - name: Ensure Template Execution Schedules are defined
      awx.awx.schedule:
        controller_host: "{{ lookup('env', 'AWX_HOST') }}"
        controller_username: "{{ lookup('env', 'AWX_USER') }}"
        controller_password: "{{ lookup('env', 'AWX_PASSWORD') }}"
        validate_certs: false
        name: "{{ item.name }}"
        unified_job_template: "{{ item.unified_job_template }}"
        rrule: "{{ item.rrule }}"
        extra_data: "{{ item.extra_data | default(omit) }}"
        state: "{{ item.state | default('present') }}"
      loop: "{{ schedules }}"

4. Wire It into Azure Pipelines (azure-pipelines.yml)

Set up a pipeline that triggers automatically when changes are pushed to your awx-onboard/ directory.

yaml

trigger:
  branches:
    include:
      - main
  paths:
    include:
      - awx-onboard/*

pool:
  vmImage: 'ubuntu-latest'

variables:
  # Store these securely in ADO Service Connections or Variable Groups
  - group: AWX-Server-Credentials 

steps:
- task: UsePythonVersion@0
  inputs:
    versionSpec: '3.x'

- script: |
    pip install ansible
    ansible-galaxy collection install awx.awx
  displayName: 'Install Ansible and AWX Collection'

- script: |
    ansible-playbook onboard-awx.yml
  displayName: 'Sync Onboarding Data to AWX'
  env:
    AWX_HOST: $(AWX_URL)
    AWX_USER: $(AWX_USERNAME)
    AWX_PASSWORD: $(AWX_PAT_TOKEN)

Essential Rules to Remember:

Survey Mapping: Variables answered in an AWX Survey act dynamically as runtime extra_vars inside your core automation playbooks. 
Schedule Variables: When mapping schedules to templates utilizing surveys, verify that any variables defined inside extra_data accurately match the technical variable names inside your survey schema. 
Time Zones: The scheduling Recurrence Rule (rrule) syntax primarily relies on UTC timestamps. Be certain to account for Daylight Savings shifts when generating the string. 

Project Overview & Architecture

Project Title

Automated Self-Service AWX Onboarding Pipeline via Azure DevOps

The Problem Statement

An enterprise organization needs to onboard dozens of application teams to Ansible AWX. Manually creating AWX organizations, syncing multi-cloud/on-prem credentials, creating dynamic inventories, building job templates, configuring complex surveys, and setting up schedules takes days per team and introduces human error.

The Solution

A GitOps-driven, self-service onboarding framework.

Request: Application teams submit a simple JSON/YAML configuration file via a Pull Request (PR) in Azure DevOps.
Validation & Approval: ADO Pipelines validate the schema and trigger an approval gate.
Execution: Once approved, an Azure DevOps pipeline uses the awx.awx Ansible collection to completely configure AWX (Credentials, Inventories, Templates, Surveys, Schedules) in a fully idempotent manner.

---

Key Interview Questions & Answers

Q1: Can you walk me through the end-to-end flow of your AWX onboarding automation?

Answer: The workflow follows a strict GitOps model:

Step 1: The user creates a branch in the Azure Repos onboarding repository and adds or modifies a standardized template file (e.g., team-alpha.yaml).
Step 2: They open a Pull Request (PR). This triggers an Azure DevOps Validation Pipeline that runs linting tests and verifies that the YAML strictly matches our JSON schema.
Step 3: A team lead or platform engineer approves the PR, merging it into the main branch.
Step 4: The Azure DevOps Deployment Pipeline triggers. It extracts the configuration data from the YAML file and executes an Ansible playbook inside a secure Microsoft-hosted or self-hosted agent.
Step 5: This deployment playbook uses the awx.awx collection to talk directly to the AWX REST API, creating or updating the requested resources in order: Organization → Credentials → Inventory → Templates → Surveys → Schedules.

Q2: How do you handle sensitive credentials securely between Azure DevOps and AWX during onboarding?

Answer: We never store plaintext passwords or tokens in Git. We utilize two layers of security:

Azure DevOps Variable Groups / Azure Key Vault: Sensitive onboarding tokens (like the AWX admin token or seed passwords) are stored securely in Azure Key Vault and mapped to Azure DevOps Variable Groups.
AWX Managed Credentials: For target machines, the onboarding pipeline uses the awx.awx.credential module. It passes a reference to an external credential vault (like CyberArk or Azure Key Vault) or creates an AWX machine credential by pulling the initial secret securely from Key Vault during runtime, keeping it completely hidden from logs.

Q3: How do you attach a Survey to a Job Template programmatically using Ansible?

Answer: The awx.awx.job_template module allows us to pass a survey_spec parameter. The survey specification must be structured as a valid JSON object defining the questions, types (text, password, multiple-choice), and variable choices. We keep these survey definitions as block scalars inside our onboarding YAML files, convert them to JSON at runtime, and feed them into the template module.

Q4: How do you manage schedules idempotently without generating duplicate runs?

Answer: The awx.awx.schedule module requires a unified_job_template name and a specific rrule (Recurrence Rule string). To maintain idempotency, we enforce a strict naming convention for schedules (e.g., [Template Name] - Daily Sync). Because the module matches by name and associated template, running the pipeline multiple times updates the existing schedule rather than creating duplicates.

---

Technical Example: Implementation Code

1. The Self-Service Request (team-alpha.yaml)

Developers fill out this file to request their AWX environment:

yaml

team_name: "Alpha-Core"
credential:
  name: "Alpha-SSH-Key"
  kind: "ssh"
  username: "ubuntu"
inventory:
  name: "Alpha-Production-EC2"
  variables:
    ansible_connection: "ssh"
template:
  name: "Deploy-Alpha-App"
  project: "Core-Playbooks-Repo"
  playbook: "deploy.yml"
survey:
  enabled: true
  questions:
    - question_name: "Target Environment"
      variable: "target_env"
      type: "multiplechoice"
      choices: ["staging", "production"]
      required: true
schedule:
  enabled: true
  name: "Alpha-Nightly-Deploy"
  rrule: "DTSTART:20261001T000000Z RRULE:FREQ=DAILY;INTERVAL=1"

Use code with caution.

2. The Azure DevOps Pipeline (azure-pipelines.yml)

This pipeline triggers when the YAML file is merged into main.

yaml

trigger:
  branches:
    include:
      - main

pool:
  vmImage: 'ubuntu-latest'

variables:
- group: 'AWX-Platform-Secrets' # Contains AWX_HOST and AWX_TOKEN securely

steps:
- task: UsePythonVersion@0
  inputs:
    versionSpec: '3.x'

- script: |
    pip install ansible awxkit
    ansible-galaxy collection install awx.awx
  displayName: 'Install Prerequisites'

- script: |
    ansible-playbook -i localhost, playbook/onboard_team.yml \
      -e "awx_host=$(AWX_HOST)" \
      -e "awx_token=$(AWX_TOKEN)" \
      -e "@team-alpha.yaml"
  displayName: 'Execute AWX Onboarding Playbook'

Use code with caution.

3. The Core AWX Automation Playbook (onboard_team.yml)

This playbook reads the YAML input variables and provisions everything inside AWX.

yaml

---
- name: "Onboard Team to AWX"
  hosts: localhost
  gather_facts: false
  vars:
    # Fallback/Default values for safety
    controller_host: "{{ awx_host }}"
    controller_oauthtoken: "{{ awx_token }}"
    validate_certs: false

  tasks:
    - name: "Create AWX Organization"
      awx.awx.organization:
        name: "{{ team_name }}"
        state: present

    - name: "Create Machine Credential"
      awx.awx.credential:
        name: "{{ credential.name }}"
        organization: "{{ team_name }}"
        credential_type: "Machine"
        inputs:
          username: "{{ credential.username }}"
          # In prod, fetch real ssh_key data from Azure Key Vault or ADO secrets
          ssh_key_data: "{{ vault_ssh_private_key | default('dummy-key-data') }}" 
        state: present

    - name: "Create Inventory"
      awx.awx.inventory:
        name: "{{ inventory.name }}"
        organization: "{{ team_name }}"
        variables: "{{ inventory.variables }}"
        state: present

    - name: "Create Job Template with Survey"
      awx.awx.job_template:
        name: "{{ template.name }}"
        project: "{{ template.project }}"
        playbook: "{{ template.playbook }}"
        inventory: "{{ inventory.name }}"
        credential: "{{ credential.name }}"
        survey_enabled: "{{ survey.enabled }}"
        survey_spec:
          name: "Survey for {{ template.name }}"
          description: "Auto-generated survey"
          spec: "{{ survey.questions }}"
        state: present
      register: created_template

    - name: "Configure Automated Schedule"
      awx.awx.schedule:
        name: "{{ schedule.name }}"
        unified_job_template: "{{ template.name }}"
        rrule: "{{ schedule.rrule }}"
        state: present
      when: schedule.enabled | bool

Project Title: Multi-Tenant Enterprise Automation Hub Onboarding

Objective: Automate the configuration lifecycle of AWX (Ansible Tower) using Azure DevOps pipelines. Instead of manually clicking through the AWX UI, a centralized Git repository controls the configuration state. Changes made to configs are automatically applied to AWX.
Tech Stack: Azure DevOps Pipelines, Git (Azure Repos), Ansible/AWX API, awx.awx or infra.controller_configuration Ansible Collections.

---

 Key Interview Questions & Comprehensive Answers

Q1: Can you explain the end-to-end architecture of onboarding AWX data via Azure DevOps?

Answer: We implement Configuration as Code (CaC) using an Azure DevOps YAML Pipeline paired with Git repositories.

Azure Repos: Contains structured YAML variable files detailing AWX objects (Credentials, Inventories, Templates, Surveys, Schedules).
Azure Pipeline: Triggered upon pull request merges to the main branch. It securely retrieves an AWX access token using an Azure DevOps Service Connection or managed identity mapped to an Azure Key Vault.
Ansible Playbook Run: The pipeline invokes an agent running an Ansible playbook. This playbook consumes the structured YAML data utilizing the infra.controller_configuration collection to dynamically provision objects in AWX via its REST API.

Q2: How do you handle sensitive credentials securely when syncing them from Azure DevOps into AWX?

Answer: Under no circumstances are passwords or private keys stored in Git. We utilize a two-step protection mechanism:

Azure Key Vault: Sensitive materials (e.g., SSH keys, AWS/Azure service principal secrets) are stored securely in Azure Key Vault.
Variable Groups / Task Integration: The Azure Pipeline maps Key Vault secrets directly into env variables via AzureKeyVault@2. During execution, the Ansible playbook maps these short-lived environment secrets into the dynamic payload sent to the AWX credential API. The values are injected seamlessly during runtime and instantly masked in logs.

Q3: How do you structure templates, surveys, and schedules together inside code so AWX understands their relationships?

Answer: AWX enforces relational dependencies. To resolve this programmatically, our Ansible playbooks use ordered execution logic and distinct object grouping schemas:

Credentials & Inventories are provisioned first because they have zero internal dependencies.
Job Templates are created next, explicitly linking the previously generated credentials and inventory names.
Surveys are defined as structured JSON blocks embedded directly inside the schema definition of the Job Template itself.
Schedules are created last, targeting the exact name/ID of the completed Job Template.

---

 Structural Code Implementation Examples

The following files represent a minimal, fully declarative framework for provisioning an automated environment.

1. The Onboarding Data Definition (awx_assets.yml)

yaml

---
# Credential Data Definition
controller_credentials:
  - name: "ADO-Deployer-SSH-Key"
    credential_type: "Machine"
    organization: "Default"
    inputs:
      username: "deploy_user"
      ssh_key_data: "{{ env_ssh_private_key }}" # Injected dynamically via Azure Pipeline

# Inventory Data Definition
controller_inventories:
  - name: "Production-App-Inventory"
    organization: "Default"
    description: "Production Environment Inventory Managed via Azure DevOps"

# Template with Embedded Survey
controller_templates:
  - name: "Deploy-Microservice-Template"
    inventory: "Production-App-Inventory"
    project: "Central-Ansible-Playbooks"
    playbook: "deploy.yml"
    credentials:
      - "ADO-Deployer-SSH-Key"
    survey_enabled: true
    survey_spec:
      name: "Deployment parameters"
      description: "Provide deployment target tags"
      spec:
        - question_name: "Target Release Version"
          variable: "release_version"
          type: "text"
          required: true
          default: "latest"

# Schedule Mapping
controller_schedules:
  - name: "Nightly-Production-Sync"
    unified_job_template: "Deploy-Microservice-Template"
    rrule: "DTSTART;TZID=UTC:20261001T000000 RRULE:FREQ=DAILY;INTERVAL=1"
    extra_data:
      release_version: "stable"

2. The Execution Playbook (onboard_awx.yml)

yaml

---
- name: Onboard Infrastructure Data to AWX Controller
  hosts: localhost
  connection: local
  gather_facts: false
  tasks:
    - name: Include Declarative AWX Asset Configurations
      ansible.builtin.include_vars:
        file: awx_assets.yml

    - name: Build Credentials
      infra.controller_configuration.credential:
        controller_username: "{{ lookup('env', 'CONTROLLER_USERNAME') }}"
        controller_password: "{{ lookup('env', 'CONTROLLER_PASSWORD') }}"
        controller_host: "{{ lookup('env', 'CONTROLLER_HOST') }}"
        validate_certs: false
        name: "{{ item.name }}"
        credential_type: "{{ item.credential_type }}"
        organization: "{{ item.organization }}"
        inputs: "{{ item.inputs }}"
      loop: "{{ controller_credentials }}"

    - name: Build Inventories
      infra.controller_configuration.inventory:
        controller_host: "{{ lookup('env', 'CONTROLLER_HOST') }}"
        controller_username: "{{ lookup('env', 'CONTROLLER_USERNAME') }}"
        controller_password: "{{ lookup('env', 'CONTROLLER_PASSWORD') }}"
        validate_certs: false
        name: "{{ item.name }}"
        organization: "{{ item.organization }}"
      loop: "{{ controller_inventories }}"

    - name: Build Job Templates & Surveys
      infra.controller_configuration.job_template:
        controller_host: "{{ lookup('env', 'CONTROLLER_HOST') }}"
        controller_username: "{{ lookup('env', 'CONTROLLER_USERNAME') }}"
        controller_password: "{{ lookup('env', 'CONTROLLER_PASSWORD') }}"
        validate_certs: false
        name: "{{ item.name }}"
        inventory: "{{ item.inventory }}"
        project: "{{ item.project }}"
        playbook: "{{ item.playbook }}"
        credentials: "{{ item.credentials }}"
        survey_enabled: "{{ item.survey_enabled }}"
        survey_spec: "{{ item.survey_spec | default(omit) }}"
      loop: "{{ controller_templates }}"

    - name: Build Job Schedules
      infra.controller_configuration.schedule:
        controller_host: "{{ lookup('env', 'CONTROLLER_HOST') }}"
        controller_username: "{{ lookup('env', 'CONTROLLER_USERNAME') }}"
        controller_password: "{{ lookup('env', 'CONTROLLER_PASSWORD') }}"
        validate_certs: false
        name: "{{ item.name }}"
        unified_job_template: "{{ item.unified_job_template }}"
        rrule: "{{ item.rrule }}"
        extra_data: "{{ item.extra_data | default(omit) }}"
      loop: "{{ controller_schedules }}"

3. The Orchestrator (azure-pipelines.yml)

yaml

trigger:
  branches:
    include:
      - main

variables:
  - group: AWX-Engine-Properties # Stores non-sensitive keys like CONTROLLER_HOST

stages:
- stage: ProvisionAWX
  displayName: 'Sync Git Assets into AWX Engine'
  jobs:
  - job: ExecutionJob
    pool:
      vmImage: 'ubuntu-latest'
    steps:
    # Safely pull infrastructure secrets from enterprise Key Vault at runtime
    - task: AzureKeyVault@2
      inputs:
        azureSubscription: 'Azure-Enterprise-Service-Connection'
        KeyVaultName: 'KV-Prod-Secrets'
        SecretsFilter: 'AWXAdminPassword,TargetHostSSHPrivateKey'
        RunAsPreJob: false

    - script: |
        pip3 install ansible awxkit
        ansible-galaxy collection install infra.controller_configuration
      displayName: 'Install Execution Prerequisites'

    - script: |
        ansible-playbook onboard_awx.yml \
          -e "env_ssh_private_key='$(TargetHostSSHPrivateKey)'"
      displayName: 'Execute Declarative State Synchronization'
      env:
        CONTROLLER_HOST: $(CONTROLLER_HOST)
        CONTROLLER_USERNAME: 'admin'
        CONTROLLER_PASSWORD: $(AWXAdminPassword)

Q1: What is a Project in AWX/AAP, and why do we use SCM sync?

Answer: A project points AWX to a repository containing playbooks, roles, and variables. SCM sync ensures version control, eliminates manual file copying on control nodes, and supports multi-user collaboration.
Pro-tip: Always enable Update Revision on Launch for dev/staging or CI-driven workflows so jobs pull the latest commit automatically.

Q2: What are the prerequisites to create a Git/GitLab-backed project using SSH?

Answer:
1. Generate/configure an SSH key pair for AWX access.
2. Add the public key to your GitLab/GitHub account/deploy keys.
3. Create a Credential in AWX of type Source Control.
4. Reference the SCM SSH private key credential inside the project definition.

Q3: What happens behind the scenes when you save an AWX Project with SCM?

Answer: AWX clones/fetches the repository into a local internal project base path (often /var/lib/awx/projects/), validates directory structures, and parses available playbooks for use in Job Templates.

Q4: Troubleshooting — A project sync fails with an SSH permission denied (128/auth) error. How do you fix it?

Answer:
• Verify the SSH private key format/passphrase in AWX credentials.
• Ensure the git remote URL uses SSH format (git@gitlab.com:...) instead of HTTPS if using SSH keys.
• Check host key verification or known_hosts configuration if the git server fingerprint isn't trusted by the execution environment/container.

Question: What is meant by An AWX "Sourced from a Project

An AWX "Sourced from a Project" inventory imports static inventory files or custom inventory scripts directly from a Git/SCM project repository. 

Setup Steps

Sync Project: Push your inventory file (e.g., inventory/hosts.yml) into your Git repo and sync the project in AWX (Resources > Projects).
Create Inventory: Go to Resources > Inventories, click Add, and create a standard inventory. 
Add Source: Inside the inventory, go to the Sources tab and click Add:
• Source: Select Sourced from a project
• Project: Select your Git project
• Inventory file: Pick the relative path/file from the dropdown or type it manually (e.g., inventory/hosts.yml)
Sync Source: Save and trigger an inventory source sync.

Key Behavior & Limitations

Updates: Re-running source sync updates hosts/groups from Git.
Group Vars: Most users on Ansible Forum agree that group variables inside SCM-sourced inventory directories may not map cleanly as distinct group vars because ansible-inventory flattens behavior into host scope.
Empty Groups: Empty groups defined in SCM inventory files can be ignored/omitted during import. 

 Question : we have two branch in azure deveop one is dev and other is master and want to deploy playbook using awx and it is already tested in preprod through test awx and want to deploy on prod awx , please explain steps

To deploy your Ansible playbooks from Azure DevOps (dev and master) to Production AWX, you need to connect your Azure DevOps pipeline to the Prod AWX server using a webhook or API call.

Prerequisites

Prod AWX API Access: A service account or token with permissions to launch job templates in Prod AWX.
AWX Job Template: A job template configured in Prod AWX that points to your playbook and inventory.
Service Connection: Credentials stored in Azure DevOps to authenticate with Prod AWX securely.

---

Step 1: Configure Prod AWX Job Template

Create or verify a Job Template in Prod AWX that points to the correct production inventory and playbook.
Note the Job Template ID from the AWX URL (e.g., /#/templates/job_template/15 means ID is 15).
Enable Webhook on the job template if you want to trigger it via webhook, or use the AWX CLI (ansible-tower-cli) or curl via a REST API call in your pipeline.

Step 2: Store AWX Credentials in Azure DevOps

Go to your Azure DevOps project settings.
Select Service connections and click New service connection.
Choose Generic or Incoming Webhook (or use secret variables for AWX URL and API Token/User credentials).
Save your Prod AWX URL, username, and password (or personal access token) as Secret variables in your Azure DevOps pipeline or Library variable group.

Step 3: Create the Azure DevOps Pipeline

Create a YAML pipeline (e.g., azure-pipelines.yml) in your master branch (or configured to trigger on merges/pull requests to master).

Use an Azure DevOps script task to trigger the Prod AWX job using curl and the AWX REST API.

yaml

trigger:
  branches:
    include:
      - master

pool:
  vmImage: 'ubuntu-latest'

steps:
- task: Bash@TF
  displayName: 'Trigger Prod AWX Job'
  script: |
    # Define variables
    AWX_URL="https://yourcompany.com"
    JOB_TEMPLATE_ID="15"
    
    # Send API request to launch the job template
    response=$(curl -s -k -X POST \
      -H "Content-Type: application/json" \
      -u "$(AWX_USER):$(AWX_PASSWORD)" \
      --data '{"extra_vars": {"branch": "master"}}' \
      "$AWX_URL/api/v2/job_templates/$JOB_TEMPLATE_ID/launch/")
    
    echo "Response: $response"
    
    # Extract Job ID and check status (optional logic)
    job_id=$(echo $response | grep -o '"id":[0-9]*' | head -n1 | cut -d':' -f2)
    echo "Started AWX Job ID: $job_id"

Step 4: Add Approval Gates for Production

Go to your Azure DevOps Environments.
Create a production environment named prod-awx.
Add Approvals and checks to this environment, adding the specific team members or leads who must approve the deployment before curl executes against Prod AWX.
Reference the environment in your pipeline job:

yaml

jobs:
- deployment: DeployToProd
  environment: 'prod-awx'
  strategy:
    runOnce:
      deploy:
        steps:
        - script: echo "Running deployment via AWX..."

Step-by-Step Deployment Guide

[Azure DevOps (Dev/Master)] ➔ [Webhook / Sync] ➔ [Prod AWX] ➔ [Target Infrastructure]

Step 1: Freeze and Merge the Code

Create a Pull Request (PR) from dev to master in Azure DevOps.
Complete the peer review.
Merge the PR to make master your production-ready baseline.

Step 2: Configure Prod AWX Credentials

Log into Prod AWX.
Navigate to Credentials -> Add.
Create a Source Control credential.
Use a Personal Access Token (PAT) from Azure DevOps with Code (Read) scopes.

Step 3: Create the Project in Prod AWX

Navigate to Projects -> Add.
Name the project (e.g., Prod-Infrastructure-Playbooks).
Set Source Control Type to Git.
Paste your Azure DevOps master branch clone URL.
Select the Azure DevOps credential created in Step 2.
Set the SCM Branch explicitly to master.
Enable Update Revision on Launch to pull fresh code before every run.

Step 4: Set Up Inventory and Machine Credentials

Create your Production Inventory in AWX.
Add production target hosts or configure dynamic cloud plugins (Azure, AWS).
Add Machine Credentials (SSH keys or WinRM passwords) to access production servers.

Step 5: Create the Job Template

Navigate to Templates -> Add -> Job Template.
Link the Production Inventory, the Project (Step 3), and the Machine Credentials.
Select the specific playbook you want to run from the dropdown.
Save and click the Rocket Icon to launch.

---

Top 3 Interview Questions & Answers

Q1: How do you ensure that changes tested in Pre-Prod AWX don't break Prod AWX?

Answer: We enforce a strict Git branching strategy and environment separation. Pre-Prod AWX points exclusively to the dev branch, while Prod AWX points to the master branch. No code reaches production without passing code reviews and automated tests in Pre-Prod. We also use AWX Survey Prompts to double-check variables before execution.

Q2: How do you automate the sync between Azure DevOps and AWX so you don't have to click "Update Project" manually?

Answer: We configure an AWX Webhook. In Azure DevOps, we set up a Service Hook that triggers on code pushes to the master branch. This sends a payload to the Prod AWX webhook URL, automatically refreshing the project baseline.

Q3: How do you manage sensitive production data like passwords and API keys?

Answer: We never hardcode secrets in Git. We encrypt sensitive variables using Ansible Vault inside the repository or inject them securely at runtime using AWX Credential Types integrated with enterprise vaults (like CyberArk or Azure Key Vault).

---

Production Challenges & How to Troubleshoot Them

1. Authentication Failures with Azure DevOps

Challenge: Prod AWX fails to sync the project, throwing Authentication failed or Repository not found errors.
Troubleshooting: Check the Azure DevOps PAT expiration date. Ensure the PAT user has explicitly granted read permissions to the specific repository containing the playbooks.

2. Environment Drift (Pre-Prod vs. Prod)

Challenge: The playbook succeeds perfectly in Pre-Prod but fails in Prod due to subtle environment differences.
Troubleshooting: Compare host facts. Ensure python dependencies (pip packages) and Ansible collections match perfectly across both AWX cluster instances. Use execution environments (containers) to lock down dependencies.

3. Network and Firewall Blocks

Challenge: Prod AWX cannot reach the target production servers, resulting in UNREACHABLE! errors.
Troubleshooting: Validate network security groups (NSGs) and firewalls. Prod AWX often sits in a different subnet or VPC than Pre-Prod AWX. Ensure port 22 (SSH) or 5986 (WinRM) is open from the Prod AWX execution nodes to the target hosts.