=== RUN   Test_DCGM_Exporter_Compatibility/AzureLinux3/scriptless_nbc
=== PAUSE Test_DCGM_Exporter_Compatibility/AzureLinux3/scriptless_nbc
=== CONT  Test_DCGM_Exporter_Compatibility/AzureLinux3/scriptless_nbc
    test_helpers.go:390: [3.876s] TAGS {Name:Test_DCGM_Exporter_Compatibility/AzureLinux3/scriptless_nbc ImageName:AzureLinuxV3gen2 OS:azurelinux Arch:amd64 NetworkIsolated:false NonAnonymousACR:false GPU:false WASM:false BootstrapTokenFallback:false KubeletCustomConfig:false Scriptless:false VHDCaching:false MockAzureChinaCloud:false VMSeriesCoverageTest:false}
    test_helpers.go:221: [9.957s] → running scenario...
    cluster.go:71: [9.957s] → preparing cluster...
    cluster.go:251: [9.957s] → get or create cluster abe2e-kubenet-v4-e1f58...
    cluster.go:259: [10.863s] ✓ get or create cluster abe2e-kubenet-v4-e1f58 done (0.9s)
    cluster.go:835: [10.863s] → setting up private DNS for API server...
    aks_model.go:307: [11.537s] → adding firewall rules...
    aks_model.go:336: [13.758s] Creating subnet AzureFirewallSubnet in VNet aks-vnet-13663576
    aks_model.go:355: [14.420s] Created firewall subnet with ID: /subscriptions/8ecadfc9-d1a3-4ea4-b844-0d9f87e4d7c8/resourceGroups/MC_abe2e-westus3_abe2e-kubenet-v4-e1f58_westus3/providers/Microsoft.Network/virtualNetworks/aks-vnet-13663576/subnets/AzureFirewallSubnet
    aks_model.go:369: [14.420s] Creating public IP abe2e-fw-pip
    aks_model.go:387: [15.235s] Created public IP with ID: /subscriptions/8ecadfc9-d1a3-4ea4-b844-0d9f87e4d7c8/resourceGroups/MC_abe2e-westus3_abe2e-kubenet-v4-e1f58_westus3/providers/Microsoft.Network/publicIPAddresses/abe2e-fw-pip
    aks_model.go:291: [15.235s] Firewall rules configured successfully
    cluster.go:874: [17.402s] private DNS zone "abe2e-kubenet-v4-ak2wn2my.hcp.westus3.azmk8s.io" → [4.249.99.50]
    cluster.go:875: [17.403s] ✓ setting up private DNS for API server done (6.5s)
    aks_model.go:405: [18.434s] Firewall private IP: 10.225.0.4
    aks_model.go:436: [18.434s] Adding route "vnet-local" to AKS route table "abe2e-fw-rt"
    aks_model.go:436: [19.615s] Adding route "default-route-to-firewall" to AKS route table "abe2e-fw-rt"
    aks_model.go:447: [20.133s] Successfully added firewall routes to AKS route table "abe2e-fw-rt"
    aks_model.go:448: [20.133s] ✓ adding firewall rules done (8.6s)
    cluster.go:727: [20.133s] → collecting garbage VMSS...
    kube.go:379: [20.133s] Creating daemonset debug-mariner-tolerated with image mcr.microsoft.com/cbl-mariner/base/core:2.0
    kube.go:379: [20.255s] Creating daemonset debugnonhost-mariner-tolerated with image mcr.microsoft.com/cbl-mariner/base/core:2.0
    kube.go:561: [20.325s] Creating proxy daemonset e2e-proxy with image mcr.microsoft.com/cbl-mariner/base/python:3
    cluster.go:777: [20.738s] → collecting garbage K8s nodes...
    cluster.go:811: [20.766s] ✓ collecting garbage K8s nodes done (0.0s)
    cluster.go:769: [20.767s] ✓ collecting garbage VMSS done (0.6s)
    cluster.go:133: [20.767s] ✓ preparing cluster done (10.8s)
    test_helpers.go:258: [20.767s] → preparing AKS node...
    vmss.go:476: [32.783s] → creating VMSS o2b8-2026-05-24-dcgmexportercompatibilityazurelinux3scrip...
    vmss.go:384: [33.054s] VMSS portal link: https://ms.portal.azure.com/#@microsoft.onmicrosoft.com/resource/subscriptions/8ecadfc9-d1a3-4ea4-b844-0d9f87e4d7c8/resourceGroups/MC_abe2e-westus3_abe2e-kubenet-v4-e1f58_westus3/providers/Microsoft.Compute/virtualMachineScaleSets/o2b8-2026-05-24-dcgmexportercompatibilityazurelinux3scrip/overview
    vmss.go:390: [33.054s] Managed cluster portal link: https://ms.portal.azure.com/#@microsoft.onmicrosoft.com/resource/subscriptions/8ecadfc9-d1a3-4ea4-b844-0d9f87e4d7c8/resourceGroups/MC_abe2e-westus3_abe2e-kubenet-v4-e1f58_westus3/providers/Microsoft.ContainerService/managedClust
... [The stack trace has been truncated as it exceeded the maximum allowed size. Please refer to the complete log available in the Test Run attachments for full details.]

=== RUN   Test_DCGM_Exporter_Compatibility/Ubuntu2404/default
=== PAUSE Test_DCGM_Exporter_Compatibility/Ubuntu2404/default
=== CONT  Test_DCGM_Exporter_Compatibility/Ubuntu2404/default
    azure.go:478: [0.000s] Looking up images in https://ms.portal.azure.com/#@microsoft.onmicrosoft.com/resource/subscriptions/c4c3550e-a965-4993-a50c-628fd38cd3e1/resourceGroups/aksvhdtestbuildrg/providers/Microsoft.Compute/galleries/PackerSigGalleryEastUS/images/aks-ubuntu-containerd-24.04-gen2/overview
    azure.go:567: [33.757s] Image version /subscriptions/c4c3550e-a965-4993-a50c-628fd38cd3e1/resourceGroups/aksvhdtestbuildrg/providers/Microsoft.Compute/galleries/PackerSigGalleryEastUS/images/2404gen2containerd/versions/1.1779601455.30864 is already in region westus3
    vhd.go:363: [33.757s] got version by tag buildId=165373865: https://ms.portal.azure.com/#@microsoft.onmicrosoft.com/resource/subscriptions/c4c3550e-a965-4993-a50c-628fd38cd3e1/resourceGroups/aksvhdtestbuildrg/providers/Microsoft.Compute/galleries/PackerSigGalleryEastUS/images/aks-ubuntu-containerd-24.04-gen2/versions/1.1779601455.30864/overview
    test_helpers.go:390: [33.757s] TAGS {Name:Test_DCGM_Exporter_Compatibility/Ubuntu2404/default ImageName:2404gen2containerd OS:ubuntu Arch:amd64 NetworkIsolated:false NonAnonymousACR:false GPU:false WASM:false BootstrapTokenFallback:false KubeletCustomConfig:false Scriptless:false VHDCaching:false MockAzureChinaCloud:false VMSeriesCoverageTest:false}
    test_helpers.go:221: [33.757s] → running scenario...
    test_helpers.go:258: [33.757s] → preparing AKS node...
    vmss.go:476: [33.757s] → creating VMSS 8pae-2026-05-24-dcgmexportercompatibilityubuntu2404defaul...
    vmss.go:384: [34.027s] VMSS portal link: https://ms.portal.azure.com/#@microsoft.onmicrosoft.com/resource/subscriptions/8ecadfc9-d1a3-4ea4-b844-0d9f87e4d7c8/resourceGroups/MC_abe2e-westus3_abe2e-kubenet-v4-e1f58_westus3/providers/Microsoft.Compute/virtualMachineScaleSets/8pae-2026-05-24-dcgmexportercompatibilityubuntu2404defaul/overview
    vmss.go:390: [34.027s] Managed cluster portal link: https://ms.portal.azure.com/#@microsoft.onmicrosoft.com/resource/subscriptions/8ecadfc9-d1a3-4ea4-b844-0d9f87e4d7c8/resourceGroups/MC_abe2e-westus3_abe2e-kubenet-v4-e1f58_westus3/providers/Microsoft.ContainerService/managedClusters/abe2e-kubenet-v4-e1f58/overview
    vmss.go:509: [37.881s] VM will be automatically deleted after the test finishes, to preserve it for debugging purposes set KEEP_VMSS=true or pause the test with a breakpoint before the test finishes or failed
    vmss.go:513: [37.881s] SSH Instructions: (may take a few minutes for the VM to be ready for SSH)
        ========================
        az network bastion ssh --target-resource-id "/subscriptions/8ecadfc9-d1a3-4ea4-b844-0d9f87e4d7c8/resourceGroups/MC_abe2e-westus3_abe2e-kubenet-v4-e1f58_westus3/providers/Microsoft.Compute/virtualMachineScaleSets/8pae-2026-05-24-dcgmexportercompatibilityubuntu2404defaul/virtualMachines/0" --name "abe2e-kubenet-v4-e1f58-bastion" --resource-group MC_abe2e-westus3_abe2e-kubenet-v4-e1f58_westus3 --auth-type ssh-key --username azureuser --ssh-key /tmp/private-key-1344314108
        
    bastionssh.go:304: [169.717s] Attempt 1/5 establishing SSH over bastion to 10.224.0.101
    vmss.go:563: [172.061s] VM reached running state
    vmss.go:533: [172.061s] ✓ creating VMSS 8pae-2026-05-24-dcgmexportercompatibilityubuntu2404defaul done (138.3s)
    test_helpers.go:346: [172.062s] ✓ preparing AKS node done (138.3s)
    test_helpers.go:251: [172.062s] Choosing the private ACR "abe2eprivatewestus3" for the vm validation
    test_helpers.go:410: [172.062s] → validating VM...
    test_helpers.go:825: [172.786s] SSH connectivity to 10.224.0.101 verified successfully
    scenario_gpu_managed_experience_test.go:203: [172.7
... [The stack trace has been truncated as it exceeded the maximum allowed size. Please refer to the complete log available in the Test Run attachments for full details.]

=== RUN   Test_DCGM_Exporter_Compatibility/AzureLinux3
=== PAUSE Test_DCGM_Exporter_Compatibility/AzureLinux3
=== CONT  Test_DCGM_Exporter_Compatibility/AzureLinux3
--- FAIL: Test_DCGM_Exporter_Compatibility/AzureLinux3 (0.00s)

=== RUN   Test_DCGM_Exporter_Compatibility/AzureLinux3/default
=== PAUSE Test_DCGM_Exporter_Compatibility/AzureLinux3/default
=== CONT  Test_DCGM_Exporter_Compatibility/AzureLinux3/default
    azure.go:478: [0.000s] Looking up images in https://ms.portal.azure.com/#@microsoft.onmicrosoft.com/resource/subscriptions/c4c3550e-a965-4993-a50c-628fd38cd3e1/resourceGroups/aksvhdtestbuildrg/providers/Microsoft.Compute/galleries/PackerSigGalleryEastUS/images/aks-azurelinux-v3-gen2/overview
    azure.go:567: [3.875s] Image version /subscriptions/c4c3550e-a965-4993-a50c-628fd38cd3e1/resourceGroups/aksvhdtestbuildrg/providers/Microsoft.Compute/galleries/PackerSigGalleryEastUS/images/AzureLinuxV3gen2/versions/1.1779601452.25781 is already in region westus3
    vhd.go:363: [3.875s] got version by tag buildId=165373865: https://ms.portal.azure.com/#@microsoft.onmicrosoft.com/resource/subscriptions/c4c3550e-a965-4993-a50c-628fd38cd3e1/resourceGroups/aksvhdtestbuildrg/providers/Microsoft.Compute/galleries/PackerSigGalleryEastUS/images/aks-azurelinux-v3-gen2/versions/1.1779601452.25781/overview
    test_helpers.go:390: [3.875s] TAGS {Name:Test_DCGM_Exporter_Compatibility/AzureLinux3/default ImageName:AzureLinuxV3gen2 OS:azurelinux Arch:amd64 NetworkIsolated:false NonAnonymousACR:false GPU:false WASM:false BootstrapTokenFallback:false KubeletCustomConfig:false Scriptless:false VHDCaching:false MockAzureChinaCloud:false VMSeriesCoverageTest:false}
    test_helpers.go:221: [9.958s] → running scenario...
    test_helpers.go:258: [20.767s] → preparing AKS node...
    vmss.go:476: [32.783s] → creating VMSS fj3e-2026-05-24-dcgmexportercompatibilityazurelinux3defau...
    vmss.go:384: [33.060s] VMSS portal link: https://ms.portal.azure.com/#@microsoft.onmicrosoft.com/resource/subscriptions/8ecadfc9-d1a3-4ea4-b844-0d9f87e4d7c8/resourceGroups/MC_abe2e-westus3_abe2e-kubenet-v4-e1f58_westus3/providers/Microsoft.Compute/virtualMachineScaleSets/fj3e-2026-05-24-dcgmexportercompatibilityazurelinux3defau/overview
    vmss.go:390: [33.060s] Managed cluster portal link: https://ms.portal.azure.com/#@microsoft.onmicrosoft.com/resource/subscriptions/8ecadfc9-d1a3-4ea4-b844-0d9f87e4d7c8/resourceGroups/MC_abe2e-westus3_abe2e-kubenet-v4-e1f58_westus3/providers/Microsoft.ContainerService/managedClusters/abe2e-kubenet-v4-e1f58/overview
    vmss.go:509: [38.309s] VM will be automatically deleted after the test finishes, to preserve it for debugging purposes set KEEP_VMSS=true or pause the test with a breakpoint before the test finishes or failed
    vmss.go:513: [38.309s] SSH Instructions: (may take a few minutes for the VM to be ready for SSH)
        ========================
        az network bastion ssh --target-resource-id "/subscriptions/8ecadfc9-d1a3-4ea4-b844-0d9f87e4d7c8/resourceGroups/MC_abe2e-westus3_abe2e-kubenet-v4-e1f58_westus3/providers/Microsoft.Compute/virtualMachineScaleSets/fj3e-2026-05-24-dcgmexportercompatibilityazurelinux3defau/virtualMachines/0" --name "abe2e-kubenet-v4-e1f58-bastion" --resource-group MC_abe2e-westus3_abe2e-kubenet-v4-e1f58_westus3 --auth-type ssh-key --username azureuser --ssh-key /tmp/private-key-1344314108
        
    bastionssh.go:304: [170.988s] Attempt 1/5 establishing SSH over bastion to 10.224.0.100
    vmss.go:563: [172.653s] VM reached running state
    vmss.go:533: [172.653s] ✓ creating VMSS fj3e-2026-05-24-dcgmexportercompatibilityazurelinux3defau done (139.9s)
    test_helpers.go:346: [172.653s] ✓ preparing AKS node done (151.9s)
    test_helpers.go:251: [172.653s] Choosing the private ACR "abe2eprivatewestus3" for the vm validation
    test_helpers.go:410: [172.653s] → validating VM...
    test_helpers.go:825: [173.263s] SSH connectivity to 10.224.0.100 verified successfully
    scenario_gpu_managed_experience_test.go:203: [173.264s] Expected versio
... [The stack trace has been truncated as it exceeded the maximum allowed size. Please refer to the complete log available in the Test Run attachments for full details.]