AZURE LOCAL - Lessons Learned After Over One Year of Practical Experience

By Christian Medewitz, Pubished: May 25th, 2026

The 18-Hour Lockout That Changed Everything

Picture this: You're in the middle of a routine cluster update—v2509 to v2510—when suddenly the network driver fails. Network-Level Authentication kicks in, blocking domain login. You reach for the local Administrator account. The credentials were documented during deployment, but you discover something unexpected: Azure Local automatically renamed the account and changed the password during cluster creation as part of its hardening procedure. The new credentials aren't documented anywhere. You're completely locked out of a production system.

18 hours later, after a full rebuild, you create your first break-glass account.

I spent 12 months deploying Azure Local across multiple industrial sites. This is what I learned so you don't have to learn it the hard way.

Understanding the Three-Pillar Framework

Before diving into lessons learned, you need to understand how Azure Local deployment actually works. Most guides treat it as a monolithic process, but that's not how it plays out in the real world. After multiple deployments, I've found it breaks down into three distinct pillars, each owned by different teams with clear handoff criteria.

Pillar 1: Physical & On-Site (Field Team)

This is traditional datacenter work: unpacking hardware, racking equipment, installing the operating system, and getting nodes connected to Azure Arc. The field team's job ends when remote access to the server is possible and Arc registration shows "Connected" in the Azure portal, marking the handover to cloud management.

Key steps:

• Unpack, rack, and stack hardware

• BIOS configuration (TPM 2.0, Secure Boot, virtualization extensions)

• Firmware validation and patching

• Azure Local ISO installation

• Windows Server initial setup

• Create break-glass accounts (critical first step)

• Basic network configuration (static IP, VLANs, DNS)

• Install Azure Connected Machine Agent (Arc)

• Verify Arc registration

Handoff criteria: Remote access to the server is possible, Arc-registered nodes showing "Connected" status in Azure portal marks the handover to cloud management.

Pillar 2: Cluster Creation Phase (Cloud Team)

This is where Azure Local transforms individual servers into a converged infrastructure cluster. This work is done remotely by cloud/platform teams using Azure Portal or PowerShell. The cluster team's job ends when all nodes report healthy and storage is validated.

Key steps:

• Pre-cluster planning (naming, node count, network design)

• Define network intents (management, compute, storage)

• Provide Active Directory credentials and service accounts

• Execute cluster deployment via Azure Portal or automation

• Troubleshoot validation failures (expect this on first attempt)

• Verify cluster health and storage pools

Handoff criteria: Cluster operational with all nodes healthy, storage spaces functional.

Pillar 3: Post-Deployment, Operating and Maintaining the Cluster (Cloud/DevOps/App Teams)

This is where Azure Local becomes useful. Initial setup involves deploying AKS, virtual machines, networking, and workloads. Then comes the ongoing operational work: monitoring, patching, failover procedures, and version updates.

Part A - Initial Setup:

• Network controller deployment

• Storage validation, create and configure volumes

• AKS Arc deployment (with workload identity)

• Virtual machines and application workloads

• IoT Operations (if applicable)

• Backup configuration

Part B - Operational Excellence (ongoing):

• Monitoring and alerting setup

• Patch management (monthly/quarterly windows)

• Shutdown/startup sequence documentation

• Failover procedures

• Version update routines

• Operational runbooks

This framework matters because it clarifies who owns what and when the handoff happens. Let's talk about what goes wrong in each pillar.

Pillar 1 Lessons: Physical & On-Site

1. Break-Glass Accounts Are Not Optional

The failure scenario: During cluster upgrade, a network driver failed on non-certified hardware. Windows Defender's application signature checking blocked the NIC driver. Network-Level Authentication prevented domain login (no network = no domain controller). The original local Administrator credentials were documented during deployment, but Azure Local had automatically renamed the account and changed the password during cluster creation as part of its hardening procedure—without documenting the new credentials anywhere. Complete lockout. 18-hour recovery requiring full rebuild.

Root cause: Azure Local hardens the built-in Administrator account during cluster creation by:

• Renaming the local Administrator account

• Changing the Administrator password

• Not documenting the new credentials anywhere

The solution: Create 2+ local administrator accounts immediately after OS installation, before joining the domain or deploying the cluster. Test them. Document them. Store credentials in a password vault.

powershell

# Create break-glass account immediately after OS install
net user BreakGlass1 "YourSecurePassword14+" /add
net localgroup Administrators BreakGlass1 /add

# Create a second break-glass account
net user BreakGlass2 "AnotherSecurePassword14+" /add
net localgroup Administrators BreakGlass2 /add

# Verify accounts work
whoami

2. The Password Length Trap

Windows Server installation will happily accept a 9-character password like Admin123! during OS setup. You won't discover the problem until hours later when cluster deployment validation fails with cryptic errors that don't mention password requirements.

Requirements:

• 14-72 characters minimum

• Must contain 3 or more of: uppercase, lowercase, numbers, special characters

The silent failure happens because Windows Server doesn't enforce Azure Local's stricter requirements at OS install time. The cluster validation scripts do.

3. RAID Architecture: The Hardware Gotcha

Storage Spaces Direct (S2D) requires data disks in JBOD/HBA/Pass-Through mode—not hardware RAID. But your OS disk should have RAID 1 protection. This means you need separate controllers:

• Controller 1: OS disk (RAID mode via BOSS card or dedicated controller)

• Controller 2: Data disks (HBA/JBOD mode for Storage Spaces Direct)

Validation:

powershell

Get-PhysicalDisk | Select FriendlyName, BusType, MediaType, Size

# BusType must show "SAS", "SATA", or "NVMe"
# If BusType shows "RAID" → Storage Spaces Direct will fail

Common mistake: Using a single RAID controller with some disks in RAID mode and others in pass-through. This doesn't work reliably. You need dedicated controllers or a BOSS card for the OS disk.

Note: This is more relevant for non-certified Azure Local test rigs, as all certified hardware configurations come with BOSS cards included.

4. Single-Node Clusters: Not Recommended for Production

Azure Local treats on-premises clusters like Azure datacenters—it expects 24/7 uptime with redundancy. Single-node clusters break this model:

• No redundancy during updates or disruptions

• Portal experience degrades significantly during maintenance windows

• No failover, no high availability

Single-node clusters are only suitable for testing and lab environments. For production workloads, deploy a minimum of 2 nodes to maintain service during updates and failures.

5. SCONFIG & Network Configuration Gotchas

Azure Local cluster validation requires static IP addresses—DHCP will cause deployment failures. Configure static IPs using SCONFIG (Option 8) during initial setup.

Additional requirements:

• Disable all disconnected network adapters (enabled but disconnected NICs cause validation failures)

• Configure VLANs via PowerShell after SCONFIG:

powershell

# Set VLAN on physical adapter (before cluster deployment)
Set-NetAdapterAdvancedProperty -Name "Port1" -RegistryKeyword "VlanID" -RegistryValue 308

Post-deployment networking: After cluster deployment, physical adapters are bound to the virtual switch and show status "Not Present". To find the correct vSwitch name:

powershell

# Use this (correct)
Get-VMSwitchTeam

# Not this (shows "Not Present")
Get-NetAdapter

6. Arc Agent Registration & Troubleshooting

Arc registration is the handoff checkpoint from Pillar 1 to Pillar 2. Common registration issues include:

• Wrong resource group

• Server name mismatch

• Tenant/subscription mismatch

• Connectivity failures (proxy/firewall)

• Expired authentication tokens

Useful commands (run from C:\Program Files\AzureConnectedMachineAgent\):

powershell

# Check current status
.\azcmagent.exe show

# Test connectivity to Azure
.\azcmagent.exe check

# Disconnect (required before re-registering with different parameters)
.\azcmagent.exe disconnect

# Register/re-register
.\azcmagent.exe connect `
  --resource-group "rg-azurelocal-site01" `
  --location "norwayeast" `
  --subscription-id "your-sub-id" `
  --tenant-id "your-tenant-id"

Important: Always disconnect before re-registering with different parameters. You cannot change resource groups without disconnecting first.

Pillar 2 Lessons: Cluster Creation Phase

7. Network Intents & Architecture Patterns

Azure Local supports multiple deployment modes:

• AD-dependent (traditional domain-joined)

• AD-less (cloud-managed identity)

• Disconnected (air-gapped environments)

Each mode has different network topology requirements. Cluster size (3-node vs. 5-node) also affects whether you need Top-of-Rack (ToR) switches.

Key insight: Understand your deployment mode and cluster size before ordering network equipment. The required switch capabilities vary significantly between deployment patterns.

8. RDMA Protocols: iWARP vs RoCE

Azure Local supports two RDMA protocols for storage traffic:

• RoCE (RDMA over Converged Ethernet): UDP-based, requires extensive QoS configuration on switches, critical for failover/backup scenarios

• iWARP: TCP-based, different trade-offs

Critical consideration: Your Top-of-Rack switches must support your chosen RDMA protocol. RoCE requires more careful QoS planning, especially during high network activity (failovers, backup/restore operations).

9. Cluster Validation Scripts: Expect Initial Failure

Microsoft's cluster creation process downloads PowerShell validation scripts to your servers. These scripts are:

• Extremely slow (1+ hours per evaluation run)

• Cryptic error messages ("unknown network error")

• Extremely sensitive to configuration details

Real example: Validation script checked Active Directory service account's DisplayName property (not sAMAccountName). If DisplayName ≠ sAMAccountName → deployment fails.

powershell

# Fix for DisplayName mismatch
Set-ADUser -Identity svc-azurelocal -DisplayName "svc-azurelocal"

Set expectations correctly: Your first cluster deployment will likely fail. This is normal. It could be DNS, networking, Active Directory, or any of dozens of validation checks. Don't expect a smooth deployment on the first attempt, especially in production environments with existing technical debt.

Note: Specific validation errors evolve with each Azure Local version. The example above may be fixed in newer releases, but it illustrates the sensitivity level.

10. Switchless vs Top-of-Rack Design

2 nodes: Switchless design is feasible and cable-efficient (direct node-to-node connections)

3+ nodes: Top-of-Rack switch is strongly recommended

Reason: Adding nodes to a switchless cluster requires production rewiring and reconfiguration. With a ToR switch, adding nodes means connecting a single cable to the switch—no disruption to existing nodes.

Planning for growth: If there's any chance you'll expand beyond 2 nodes, introduce the ToR switch at the 3-node threshold.

11. AKS Arc: Uninstall Before Cluster Updates

Uninstalling AKS Arc before cluster updates is recommended but not strictly required. However, it prevents AKS from entering an orphaned state when AKS versions change during cluster updates.

Orphaned AKS operates on-premises but loses connection to Arc cloud components. Fixing this requires redeployment.

Solution: Use stateless configuration and automation for AKS deployment/redeployment. Treat AKS as ephemeral infrastructure that can be quickly rebuilt.

Pillar 3 Lessons: Post-Deployment, Operating and Maintaining

12. Network Settings Reset After Updates

RDP access, remote administration ports, and other node-level settings reset to defaults after cluster updates. You can automate re-application if needed, but from a security/hardening perspective, the defaults are often acceptable.

Recommendation: If you're only accessing nodes via iLO/out-of-band management, leaving RDP disabled is actually preferred.

13. SDN: Optional Day One, Permanent After Deployment

Software-Defined Networking (SDN) enables zero-trust micro-segmentation via Network Security Groups. But it adds significant operational complexity.

Critical constraint: You can opt into SDN later, but you cannot remove it once deployed. This is a one-way door.

Recommendation: Consider deferring SDN if your organization isn't ready for the additional complexity. You can always add it later when operational maturity improves.

14. Vendor-Assisted Installs: Helpful But Not a Substitute

Hardware vendors can help with cluster setup, storage configuration, and networking (RDMA protocols, Storage Spaces Direct). This assistance is valuable for initial deployment.

However: Vendor assistance does not replace the need for internal competency and operational procedures. You still need internal knowledge for long-term operations, or you'll be dependent on sustained external partnerships.

15. Infrastructure Planning: Build vs Buy Decision

Two paths forward:

Path A (Recommended): Build internal competency via small test rigs → iterative learning → mature automation. Avoids vendor dependency long-term.

Path B: Buy expert help—vendor or partner does the heavy lifting. Work measured in months, not hours.

Small test deployments reveal actual infrastructure needs vs. assumptions. This is critical for FinOps: cluster sizing affects licensing costs (especially SQL Server), and you need to determine if one large cluster or two smaller clusters better serves your needs.

Timeline: Expect months to 1+ year for most companies due to organizational adaptation, not just technical implementation.

16. Organizational Responsibility Shift

Deploying Azure Local creates a major change in organizational responsibilities:

Classic IT model:

• On-prem team → All on-premises infrastructure

• Network team → Physical firewalls, switches, routing

• Cloud team → Azure services

Azure Local model:

• Pillar 1: On-prem/field team (rack/stack, OS, Arc registration) → handoff

• Pillar 2-3: Cloud/platform teams (cluster deployment, operations)

• Network ownership shifts: From NetOps/NetSec (physical devices) to cloud teams (SDN, NSGs, virtual networking)

• IaC/automation: On-prem teams no longer manage clusters day-to-day; cloud teams use infrastructure-as-code

This is organizational transformation, not just technology deployment.

Strategic & Operational Lessons

17. Real-World Timeframe Expectations

Test Environment Deployment:

• Optimistic (everything pre-staged): Few days to 1 week

• Realistic: 1-3 weeks to multi-week

Blockers: Hardware availability, network setup, Active Directory preparation (service accounts, OUs, delegations, sites/services, DNS, reverse zones, DHCP), VPN to Azure test subscription, jump hosts.

Key success factor: Eliminate all dependencies and friction upfront. Don't wait for switches, subscriptions, credit cards, or approvals.

Production Deployment:

• Technical work: Similar to test environment

• Real blockers: NOT technical issues

  • Fitting Azure Local into existing environment

  • Technical debt resolution

  • Internal competency gaps

  • Resource availability and funding

  • Network/firewall/proxy configurations

  • Change management processes (no "YOLO" deployments)

  • CAB approvals, change windows, rollback procedures

  • Active Directory modifications in production (slow approval cycles)

Timeline: Multiple months to 1+ year depending on organizational maturity and existing technical debt.

18. Separation Strategy: Subscription & Resource Group Design

Microsoft constraint: One custom location per subscription, one custom location per Azure Local cluster.

Recommended architecture:

• One subscription per Azure Local cluster (represents a datacenter site)

• One resource group per cluster (contains all cluster resources)

• Cannot move resources between subscriptions or resource groups after creation (per Microsoft documentation)

Benefits:

• Easy decommissioning (delete entire subscription/resource group)

• Clear separation between sites/clusters

• Simplified billing and cost tracking per site

Caveat: Plan your resource group strategy upfront—you can't reorganize later.

19. Active Directory Cleanup & Decommissioning

When replacing or decommissioning servers (common due to resource availability issues, wrong sizing, or incompatible hardware for cluster expansion):

Cloud side: Easy cleanup via resource group deletion

On-premises side: Leaves orphaned objects:

• Active Directory computer accounts

• DNS records

• Other on-prem artifacts

Solution: Develop evaluation and cleanup scripts to validate and remove on-prem leftovers before redeployment or when repurposing hardware to different sites.

powershell

# Example cleanup check
Get-ADComputer -Filter "Name -like 'AZLOCAL-*'" | 
  Where-Object {$_.Enabled -eq $false -or $_.LastLogonDate -lt (Get-Date).AddDays(-30)}

Conclusion: The Path Forward

Azure Local isn't just another infrastructure platform—it's a fundamental shift in how we build and operate on-premises systems. The three-pillar framework (Physical, Cluster, Operations) provides clarity on responsibilities and handoffs, but the real challenge isn't technical.

The real challenge is organizational transformation: shifting network ownership from physical devices to software-defined networking, moving from change-averse operations to infrastructure-as-code, and building competency in new teams.

After 12 months across multiple deployments, here's my core advice:

1. Create break-glass accounts immediately (before anything else)

2. Plan for minimum 2 nodes for production clusters (single-node is testing only)

3. Expect first deployment to fail (validation scripts are sensitive)

4. Build internal competency (vendor help is useful but not sufficient)

5. Plan resource groups upfront (you can't reorganize later)

6. Set realistic timelines (months to 1+ year for production rollout)

The 18-hour lockout taught me that small details—like break-glass accounts—can mean the difference between a 5-minute fix and a multi-day rebuild. Every lesson in this article came from real failures and real recoveries.

Use this framework. Avoid these mistakes. Save yourself months of troubleshooting.