W1105 01:03:29.773] ************************************************************************** bootstrap.py is deprecated! test-infra oncall does not support any job still using bootstrap.py. Please migrate your job to podutils! https://github.com/kubernetes/test-infra/blob/master/prow/pod-utilities.md ************************************************************************** I1105 01:03:29.774] Args: --job=chewong-pull-azuredisk-csi-driver-e2e-multi-az --service-account=/etc/service-account/service-account.json --upload=gs://kubernetes-jenkins/logs --job=chewong-pull-azuredisk-csi-driver-e2e-multi-az --root=/go/src --service-account=/etc/service-account/service-account.json --repo=k8s.io/kubernetes=master --repo=sigs.k8s.io/azuredisk-csi-driver=master:b7c03af8eae7388e84115ace30cf2ff3285b0dc9,180:dbf2ba05476df3f4c50de82e8d2f2a5264329cb5 --upload=gs://kubernetes-upstream/pr-logs --timeout=460 --scenario=kubernetes_e2e -- --gce-ssh= --test=true --up=true --down=true --deployment=acsengine --build=bazel --provider=skeleton --ginkgo-parallel=1 --acsengine-agentpoolcount=2 --acsengine-admin-username=azureuser '--acsengine-creds=$AZURE_CREDENTIALS' --acsengine-orchestratorRelease=1.16 --acsengine-mastervmsize=Standard_DS2_v2 --acsengine-agentvmsize=Standard_D4s_v3 --acsengine-location=eastus2 '--acsengine-public-key=$AZURE_SSH_PUBLIC_KEY_FILE' --acsengine-template-url=https://raw.githubusercontent.com/kubernetes/cloud-provider-azure/master/tests/k8s-azure/manifest/linux-vmss-multi-zones.json --acsengine-download-url=https://github.com/Azure/aks-engine/releases/download/v0.39.1/aks-engine-v0.39.1-linux-amd64.tar.gz --test-azure-disk-csi-driver=True '--test_args=--ginkgo.focus=\[multi-az\]' --timeout=420m I1105 01:03:29.774] Bootstrap chewong-pull-azuredisk-csi-driver-e2e-multi-az... I1105 01:03:29.777] Builder: fabe53f1-ff67-11e9-918b-16773c0c7aed I1105 01:03:29.778] Image: gcr.io/k8s-testimages/kubekins-e2e:v20191103-6816af1-master I1105 01:03:29.778] Gubernator results at https://gubernator.k8s.io/build/kubernetes-upstream/pr-logs/pull/sigs.k8s.io_azuredisk-csi-driver/180/chewong-pull-azuredisk-csi-driver-e2e-multi-az/1191521311436836864 I1105 01:03:29.778] Call: gcloud auth activate-service-account --key-file=/etc/service-account/service-account.json W1105 01:03:30.289] Activated service account credentials for: [kubernetes-upstream@kubernetes-upstream.iam.gserviceaccount.com] I1105 01:03:30.319] process 195 exited with code 0 after 0.0m I1105 01:03:30.319] Call: gcloud config get-value account I1105 01:03:30.568] process 207 exited with code 0 after 0.0m I1105 01:03:30.568] Will upload results to gs://kubernetes-upstream/pr-logs using kubernetes-upstream@kubernetes-upstream.iam.gserviceaccount.com I1105 01:03:30.569] Root: /go/src I1105 01:03:30.569] cd to /go/src I1105 01:03:30.569] Checkout: /go/src/k8s.io/kubernetes master to /go/src/k8s.io/kubernetes I1105 01:03:30.569] Call: git init k8s.io/kubernetes I1105 01:03:30.573] Initialized empty Git repository in /go/src/k8s.io/kubernetes/.git/ I1105 01:03:30.573] process 219 exited with code 0 after 0.0m I1105 01:03:30.573] Call: git config --local user.name 'K8S Bootstrap' I1105 01:03:30.575] process 220 exited with code 0 after 0.0m I1105 01:03:30.576] Call: git config --local user.email k8s_bootstrap@localhost I1105 01:03:30.578] process 221 exited with code 0 after 0.0m I1105 01:03:30.578] Call: git fetch --quiet --tags https://github.com/kubernetes/kubernetes master I1105 01:04:29.893] process 222 exited with code 0 after 1.0m I1105 01:04:29.894] Call: git checkout -B test FETCH_HEAD W1105 01:04:32.095] Switched to a new branch 'test' I1105 01:04:32.104] process 233 exited with code 0 after 0.0m I1105 01:04:32.104] Call: git show -s --format=format:%ct HEAD I1105 01:04:32.121] process 234 exited with code 0 after 0.0m I1105 01:04:32.121] Checkout: /go/src/sigs.k8s.io/azuredisk-csi-driver master:b7c03af8eae7388e84115ace30cf2ff3285b0dc9,180:dbf2ba05476df3f4c50de82e8d2f2a5264329cb5 to /go/src/sigs.k8s.io/azuredisk-csi-driver I1105 01:04:32.121] Call: git init sigs.k8s.io/azuredisk-csi-driver I1105 01:04:32.125] Initialized empty Git repository in /go/src/sigs.k8s.io/azuredisk-csi-driver/.git/ I1105 01:04:32.125] process 235 exited with code 0 after 0.0m I1105 01:04:32.125] Call: git config --local user.name 'K8S Bootstrap' I1105 01:04:32.128] process 236 exited with code 0 after 0.0m I1105 01:04:32.128] Call: git config --local user.email k8s_bootstrap@localhost I1105 01:04:32.130] process 237 exited with code 0 after 0.0m I1105 01:04:32.131] Call: git fetch --quiet --tags https://github.com/kubernetes-sigs/azuredisk-csi-driver master +refs/pull/180/head:refs/pr/180 I1105 01:04:38.326] process 238 exited with code 0 after 0.1m I1105 01:04:38.327] Call: git checkout -B test b7c03af8eae7388e84115ace30cf2ff3285b0dc9 W1105 01:04:38.926] Switched to a new branch 'test' I1105 01:04:38.929] process 249 exited with code 0 after 0.0m I1105 01:04:38.930] Call: git show -s --format=format:%ct HEAD I1105 01:04:38.932] process 250 exited with code 0 after 0.0m I1105 01:04:38.933] Call: git merge --no-ff -m 'Merge +refs/pull/180/head:refs/pr/180' dbf2ba05476df3f4c50de82e8d2f2a5264329cb5 I1105 01:04:39.671] Merge made by the 'recursive' strategy. I1105 01:04:39.674] .../templates/csi-azuredisk-controller.yaml | 5 +- I1105 01:04:39.674] deploy/csi-azuredisk-controller.yaml | 3 +- I1105 01:04:39.674] .../storageclass-azuredisk-csi-topology.yaml | 17 +++ I1105 01:04:39.674] pkg/azuredisk/azuredisk.go | 1 + I1105 01:04:39.675] pkg/azuredisk/identityserver.go | 7 ++ I1105 01:04:39.675] pkg/azuredisk/nodeserver.go | 23 +++- I1105 01:04:39.675] test/e2e/driver/azuredisk_csi_driver.go | 19 ++- I1105 01:04:39.675] test/e2e/dynamic_provisioning.go | 134 +++++++++++++++------ I1105 01:04:39.675] test/e2e/testsuites/testsuites.go | 8 ++ I1105 01:04:39.675] 9 files changed, 170 insertions(+), 47 deletions(-) I1105 01:04:39.676] create mode 100644 deploy/example/storageclass-azuredisk-csi-topology.yaml I1105 01:04:39.676] process 251 exited with code 0 after 0.0m I1105 01:04:39.676] Configure environment... I1105 01:04:39.676] Call: git show -s --format=format:%ct HEAD I1105 01:04:39.693] process 267 exited with code 0 after 0.0m I1105 01:04:39.693] Call: gcloud auth activate-service-account --key-file=/etc/service-account/service-account.json W1105 01:04:40.317] Activated service account credentials for: [kubernetes-upstream@kubernetes-upstream.iam.gserviceaccount.com] I1105 01:04:40.426] process 268 exited with code 0 after 0.0m I1105 01:04:40.426] Call: gcloud config get-value account I1105 01:04:40.669] process 280 exited with code 0 after 0.0m I1105 01:04:40.670] Will upload results to gs://kubernetes-upstream/pr-logs using kubernetes-upstream@kubernetes-upstream.iam.gserviceaccount.com I1105 01:04:40.670] Call: bash -c ' set -o errexit set -o nounset export KUBE_ROOT=. source hack/lib/version.sh kube::version::get_version_vars echo $KUBE_GIT_VERSION ' I1105 01:04:40.918] process 292 exited with code 0 after 0.0m I1105 01:04:40.919] Start 1191521311436836864 at v1.18.0-alpha.0.230+b735a17163ac7c... I1105 01:04:40.921] Call: gsutil -q -h Content-Type:application/json cp /tmp/gsutil_UyAsba gs://kubernetes-upstream/pr-logs/pull/sigs.k8s.io_azuredisk-csi-driver/180/chewong-pull-azuredisk-csi-driver-e2e-multi-az/1191521311436836864/started.json I1105 01:04:42.455] process 325 exited with code 0 after 0.0m I1105 01:04:42.455] Call: gsutil -q -h Content-Type:text/plain -h 'x-goog-meta-link: gs://kubernetes-upstream/pr-logs/pull/sigs.k8s.io_azuredisk-csi-driver/180/chewong-pull-azuredisk-csi-driver-e2e-multi-az/1191521311436836864' cp /tmp/gsutil_xWCMzb gs://kubernetes-upstream/pr-logs/directory/chewong-pull-azuredisk-csi-driver-e2e-multi-az/1191521311436836864.txt I1105 01:04:43.606] process 505 exited with code 0 after 0.0m I1105 01:04:43.607] Call: /workspace/./test-infra/jenkins/../scenarios/kubernetes_e2e.py --gce-ssh= --test=true --up=true --down=true --deployment=acsengine --build=bazel --provider=skeleton --ginkgo-parallel=1 --acsengine-agentpoolcount=2 --acsengine-admin-username=azureuser --acsengine-creds=/etc/azure-cred/credentials --acsengine-orchestratorRelease=1.16 --acsengine-mastervmsize=Standard_DS2_v2 --acsengine-agentvmsize=Standard_D4s_v3 --acsengine-location=eastus2 --acsengine-public-key=/etc/azure-ssh/azure-ssh-pub --acsengine-template-url=https://raw.githubusercontent.com/kubernetes/cloud-provider-azure/master/tests/k8s-azure/manifest/linux-vmss-multi-zones.json --acsengine-download-url=https://github.com/Azure/aks-engine/releases/download/v0.39.1/aks-engine-v0.39.1-linux-amd64.tar.gz --test-azure-disk-csi-driver=True '--test_args=--ginkgo.focus=\[multi-az\]' --timeout=420m W1105 01:04:43.645] starts with local mode W1105 01:04:43.645] Environment: W1105 01:04:43.645] ARTIFACTS=/workspace/_artifacts W1105 01:04:43.646] AWS_SSH_PRIVATE_KEY_FILE=/root/.ssh/kube_aws_rsa W1105 01:04:43.646] AWS_SSH_PUBLIC_KEY_FILE=/root/.ssh/kube_aws_rsa.pub W1105 01:04:43.646] AZURE_CREDENTIALS=/etc/azure-cred/credentials W1105 01:04:43.646] AZURE_SSH_PUBLIC_KEY_FILE=/etc/azure-ssh/azure-ssh-pub W1105 01:04:43.646] AZ_STORAGE_CONTAINER_NAME=mystoragecontainer W1105 01:04:43.646] BAZEL_REMOTE_CACHE_ENABLED=false W1105 01:04:43.646] BAZEL_VERSION=0.23.2 W1105 01:04:43.647] BOOTSTRAP_MIGRATION=yes W1105 01:04:43.647] BUILD_ID=1191521311436836864 W1105 01:04:43.647] BUILD_NUMBER=1191521311436836864 W1105 01:04:43.647] CLOUDSDK_COMPONENT_MANAGER_DISABLE_UPDATE_CHECK=true W1105 01:04:43.647] CLOUDSDK_CONFIG=/go/src/k8s.io/kubernetes/.config/gcloud W1105 01:04:43.647] CLOUDSDK_CORE_DISABLE_PROMPTS=1 W1105 01:04:43.647] CLOUDSDK_EXPERIMENTAL_FAST_COMPONENT_UPDATE=false W1105 01:04:43.647] CLOUD_CONFIG=random W1105 01:04:43.648] DECK_PORT=tcp://10.0.160.83:80 W1105 01:04:43.648] DECK_PORT_80_TCP=tcp://10.0.160.83:80 W1105 01:04:43.648] DECK_PORT_80_TCP_ADDR=10.0.160.83 W1105 01:04:43.648] DECK_PORT_80_TCP_PORT=80 W1105 01:04:43.648] DECK_PORT_80_TCP_PROTO=tcp W1105 01:04:43.648] DECK_SERVICE_HOST=10.0.160.83 W1105 01:04:43.648] DECK_SERVICE_PORT=80 W1105 01:04:43.648] DOCKER_IN_DOCKER_ENABLED=true W1105 01:04:43.649] DOCKER_IN_DOCKER_IPV6_ENABLED=false W1105 01:04:43.649] GCS_ARTIFACTS_DIR=gs://kubernetes-upstream/pr-logs/pull/sigs.k8s.io_azuredisk-csi-driver/180/chewong-pull-azuredisk-csi-driver-e2e-multi-az/1191521311436836864/artifacts W1105 01:04:43.649] GOOGLE_APPLICATION_CREDENTIALS=/etc/service-account/service-account.json W1105 01:04:43.649] GOPATH=/go W1105 01:04:43.649] GO_TARBALL=go1.12.12.linux-amd64.tar.gz W1105 01:04:43.649] HOME=/workspace W1105 01:04:43.649] HOOK_PORT=tcp://10.0.144.133:8888 W1105 01:04:43.650] HOOK_PORT_8888_TCP=tcp://10.0.144.133:8888 W1105 01:04:43.650] HOOK_PORT_8888_TCP_ADDR=10.0.144.133 W1105 01:04:43.650] HOOK_PORT_8888_TCP_PORT=8888 W1105 01:04:43.650] HOOK_PORT_8888_TCP_PROTO=tcp W1105 01:04:43.650] HOOK_SERVICE_HOST=10.0.144.133 W1105 01:04:43.651] HOOK_SERVICE_PORT=8888 W1105 01:04:43.651] HOSTNAME=fabe53f1-ff67-11e9-918b-16773c0c7aed W1105 01:04:43.651] IMAGE=gcr.io/k8s-testimages/kubekins-e2e:v20191103-6816af1-master W1105 01:04:43.651] INSTANCE_PREFIX=bootstrap-e2e W1105 01:04:43.651] JENKINS_GCE_SSH_PRIVATE_KEY_FILE=/root/.ssh/google_compute_engine W1105 01:04:43.651] JENKINS_GCE_SSH_PUBLIC_KEY_FILE=/root/.ssh/google_compute_engine.pub W1105 01:04:43.651] JOB_NAME=chewong-pull-azuredisk-csi-driver-e2e-multi-az W1105 01:04:43.652] JOB_SPEC={"type":"presubmit","job":"chewong-pull-azuredisk-csi-driver-e2e-multi-az","buildid":"1191521311436836864","prowjobid":"fabe53f1-ff67-11e9-918b-16773c0c7aed","refs":{"org":"Azure","repo":"kubernetes","repo_link":"https://github.com/Azure/kubernetes","base_ref":"master","base_sha":"077f825c584903eeb5854040192c2977079343e9","base_link":"https://github.com/Azure/kubernetes/commit/077f825c584903eeb5854040192c2977079343e9","pulls":[{"number":113,"author":"chewong","sha":"879465972041635fd7bc8269da49153b67e63822","link":"https://github.com/Azure/kubernetes/pull/113","commit_link":"https://github.com/Azure/kubernetes/pull/113/commits/879465972041635fd7bc8269da49153b67e63822","author_link":"https://github.com/chewong"}]}} W1105 01:04:43.652] JOB_TYPE=presubmit W1105 01:04:43.652] KUBERNETES_PORT=tcp://10.0.0.1:443 W1105 01:04:43.652] KUBERNETES_PORT_443_TCP=tcp://10.0.0.1:443 W1105 01:04:43.653] KUBERNETES_PORT_443_TCP_ADDR=10.0.0.1 W1105 01:04:43.653] KUBERNETES_PORT_443_TCP_PORT=443 W1105 01:04:43.653] KUBERNETES_PORT_443_TCP_PROTO=tcp W1105 01:04:43.653] KUBERNETES_SERVICE_HOST=10.0.0.1 W1105 01:04:43.653] KUBERNETES_SERVICE_PORT=443 W1105 01:04:43.653] KUBERNETES_SERVICE_PORT_HTTPS=443 W1105 01:04:43.653] KUBETEST_IN_DOCKER=true W1105 01:04:43.654] KUBETEST_MANUAL_DUMP=y W1105 01:04:43.654] KUBE_AWS_INSTANCE_PREFIX=bootstrap-e2e W1105 01:04:43.654] KUBE_GCE_INSTANCE_PREFIX=bootstrap-e2e W1105 01:04:43.654] KUBE_SSH_KEY_PATH=/etc/ssh-key-secret/ssh-private W1105 01:04:43.654] KUBE_SSH_PUBLIC_KEY_PATH=/etc/ssh-key-secret/ssh-public W1105 01:04:43.654] KUBE_SSH_USER=azureuser W1105 01:04:43.654] KUBE_VERBOSE=0 W1105 01:04:43.655] NODE_NAME=fabe53f1-ff67-11e9-918b-16773c0c7aed W1105 01:04:43.655] PATH=/go/bin:/go/bin:/usr/local/go/bin:/google-cloud-sdk/bin:/workspace:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin W1105 01:04:43.655] PROW_JOB_ID=fabe53f1-ff67-11e9-918b-16773c0c7aed W1105 01:04:43.655] PULL_BASE_REF=master W1105 01:04:43.655] PULL_BASE_SHA=077f825c584903eeb5854040192c2977079343e9 W1105 01:04:43.655] PULL_NUMBER=113 W1105 01:04:43.655] PULL_PULL_SHA=879465972041635fd7bc8269da49153b67e63822 W1105 01:04:43.656] PULL_REFS=master:077f825c584903eeb5854040192c2977079343e9,113:879465972041635fd7bc8269da49153b67e63822 W1105 01:04:43.656] PWD=/workspace W1105 01:04:43.656] REGISTRY=k8sprowinternal.azurecr.io W1105 01:04:43.656] REPO_NAME=kubernetes W1105 01:04:43.656] REPO_OWNER=Azure W1105 01:04:43.656] SHLVL=2 W1105 01:04:43.656] SOURCE_DATE_EPOCH=1572910143 W1105 01:04:43.657] TERM=xterm W1105 01:04:43.657] TIDE_PORT=tcp://10.0.180.53:80 W1105 01:04:43.657] TIDE_PORT_80_TCP=tcp://10.0.180.53:80 W1105 01:04:43.657] TIDE_PORT_80_TCP_ADDR=10.0.180.53 W1105 01:04:43.657] TIDE_PORT_80_TCP_PORT=80 W1105 01:04:43.657] TIDE_PORT_80_TCP_PROTO=tcp W1105 01:04:43.657] TIDE_SERVICE_HOST=10.0.180.53 W1105 01:04:43.658] TIDE_SERVICE_PORT=80 W1105 01:04:43.658] WORKSPACE=/workspace W1105 01:04:43.658] _=./test-infra/jenkins/bootstrap.py W1105 01:04:43.658] Run: ('kubetest', '--dump=/workspace/_artifacts', '--gcp-service-account=/etc/service-account/service-account.json', '--build=bazel', '--up', '--down', '--test', '--deployment=acsengine', '--provider=skeleton', '--cluster=bootstrap-e2e', '--gcp-network=bootstrap-e2e', '--ginkgo-parallel=1', '--acsengine-agentpoolcount=2', '--acsengine-admin-username=azureuser', '--acsengine-creds=/etc/azure-cred/credentials', '--acsengine-orchestratorRelease=1.16', '--acsengine-mastervmsize=Standard_DS2_v2', '--acsengine-agentvmsize=Standard_D4s_v3', '--acsengine-location=eastus2', '--acsengine-public-key=/etc/azure-ssh/azure-ssh-pub', '--acsengine-template-url=https://raw.githubusercontent.com/kubernetes/cloud-provider-azure/master/tests/k8s-azure/manifest/linux-vmss-multi-zones.json', '--acsengine-download-url=https://github.com/Azure/aks-engine/releases/download/v0.39.1/aks-engine-v0.39.1-linux-amd64.tar.gz', '--test-azure-disk-csi-driver=True', '--test_args=--ginkgo.focus=\\[multi-az\\]', '--timeout=420m') W1105 01:04:43.661] 2019/11/05 01:04:43 main.go:332: Limiting testing to 7h0m0s W1105 01:04:43.662] 2019/11/05 01:04:43 azure.go:232: Reading credentials file /etc/azure-cred/credentials W1105 01:04:43.662] 2019/11/05 01:04:43 process.go:153: Running: make -C /go/src/k8s.io/kubernetes bazel-release W1105 01:04:43.741] Extracting Bazel installation... I1105 01:04:43.841] make: Entering directory '/go/src/k8s.io/kubernetes' W1105 01:04:53.148] Starting local Bazel server and connecting to it... W1105 01:04:54.902] Loading: W1105 01:04:54.907] Loading: 0 packages loaded W1105 01:04:55.909] Loading: 0 packages loaded W1105 01:04:57.493] Loading: 0 packages loaded W1105 01:04:58.910] Loading: 0 packages loaded W1105 01:05:00.760] Loading: 0 packages loaded W1105 01:05:01.910] Loading: 0 packages loaded W1105 01:05:01.911] currently loading: build/release-tars W1105 01:05:04.030] Loading: 0 packages loaded W1105 01:05:04.030] currently loading: build/release-tars W1105 01:05:04.413] Analyzing: target //build/release-tars:release-tars (1 packages loaded, 0 targets configured) W1105 01:05:05.867] Analyzing: target //build/release-tars:release-tars (149 packages loaded, 805 targets configured) W1105 01:05:07.540] Analyzing: target //build/release-tars:release-tars (1078 packages loaded, 3909 targets configured) W1105 01:05:09.490] Analyzing: target //build/release-tars:release-tars (1754 packages loaded, 12495 targets configured) W1105 01:05:11.706] Analyzing: target //build/release-tars:release-tars (2601 packages loaded, 22773 targets configured) W1105 01:05:14.255] Analyzing: target //build/release-tars:release-tars (2946 packages loaded, 26032 targets configured) W1105 01:05:17.229] Analyzing: target //build/release-tars:release-tars (3079 packages loaded, 28388 targets configured) W1105 01:05:21.576] Analyzing: target //build/release-tars:release-tars (3103 packages loaded, 34562 targets configured) W1105 01:05:25.603] Analyzing: target //build/release-tars:release-tars (3104 packages loaded, 36098 targets configured) W1105 01:05:31.637] INFO: Analysed target //build/release-tars:release-tars (3104 packages loaded, 36837 targets configured). W1105 01:05:31.640] Building: checking cached actions W1105 01:05:31.640] INFO: Found 1 target... W1105 01:05:31.807] [0 / 25] [-----] BazelWorkspaceStatusAction stable-status.txt ... (2 actions, 0 running) W1105 01:05:38.516] [97 / 4,426] GoCompilePkg vendor/golang.org/x/tools/go/internal/gcimporter/linux_amd64_stripped/go_default_library%/k8s.io/kubernetes/vendor/golang.org/x/tools/go/internal/gcimporter.a [for host]; 2s linux-sandbox ... (8 actions, 7 running) W1105 01:05:45.654] [121 / 4,426] GoStdlib external/io_bazel_rules_go/linux_amd64_pure_stripped/stdlib%/pkg; 9s linux-sandbox ... (8 actions, 7 running) W1105 01:05:53.838] [171 / 4,426] GoStdlib external/io_bazel_rules_go/linux_amd64_pure_stripped/stdlib%/pkg; 17s linux-sandbox ... (8 actions, 7 running) W1105 01:06:02.817] [249 / 4,426] GoStdlib external/io_bazel_rules_go/linux_amd64_pure_stripped/stdlib%/pkg; 26s linux-sandbox ... (8 actions, 7 running) W1105 01:06:13.719] [305 / 4,426] GoStdlib external/io_bazel_rules_go/linux_amd64_pure_stripped/stdlib%/pkg; 37s linux-sandbox ... (7 actions running) W1105 01:06:25.943] [358 / 4,426] GoCompilePkg vendor/github.com/syndtr/gocapability/capability/linux_amd64_stripped/go_default_library%/k8s.io/kubernetes/vendor/github.com/syndtr/gocapability/capability.a; 1s linux-sandbox ... (8 actions, 7 running) W1105 01:06:39.646] [526 / 4,426] GoCompilePkg vendor/github.com/quobyte/api/linux_amd64_stripped/go_default_library%/k8s.io/kubernetes/vendor/github.com/quobyte/api.a; 1s linux-sandbox ... (8 actions, 7 running) W1105 01:06:56.457] [625 / 4,426] GoCompilePkg vendor/go.etcd.io/etcd/clientv3/balancer/linux_amd64_stripped/go_default_library%/k8s.io/kubernetes/vendor/go.etcd.io/etcd/clientv3/balancer.a; 4s linux-sandbox ... (8 actions, 7 running) W1105 01:07:14.745] [787 / 4,426] GoCompilePkg vendor/github.com/imdario/mergo/linux_amd64_pure_stripped/go_default_library%/k8s.io/kubernetes/vendor/github.com/imdario/mergo.a; 0s linux-sandbox ... (8 actions, 7 running) W1105 01:07:35.766] [892 / 4,426] GoCompilePkg vendor/github.com/chai2010/gettext-go/gettext/linux_amd64_pure_stripped/go_default_library%/k8s.io/kubernetes/vendor/github.com/chai2010/gettext-go/gettext.a; 7s linux-sandbox ... (8 actions, 7 running) W1105 01:07:59.960] [1,035 / 4,426] GoCompilePkg vendor/github.com/globalsign/mgo/internal/json/linux_amd64_stripped/go_default_library%/k8s.io/kubernetes/vendor/github.com/globalsign/mgo/internal/json.a; 1s linux-sandbox ... (8 actions, 7 running) W1105 01:08:27.801] [1,268 / 4,426] GoCompilePkg vendor/github.com/miekg/dns/linux_amd64_stripped/go_default_library%/k8s.io/kubernetes/vendor/github.com/miekg/dns.a; 4s linux-sandbox ... (8 actions, 7 running) W1105 01:08:59.770] [1,483 / 4,426] GoCompilePkg vendor/github.com/vmware/govmomi/vim25/types/linux_amd64_pure_stripped/go_default_library%/k8s.io/kubernetes/vendor/github.com/vmware/govmomi/vim25/types.a; 5s linux-sandbox ... (8 actions, 7 running) W1105 01:09:37.549] [1,632 / 4,426] GoCompilePkg vendor/github.com/Azure/azure-sdk-for-go/services/compute/mgmt/2019-07-01/compute/linux_amd64_pure_stripped/go_default_library%/k8s.io/kubernetes/vendor/github.com/Azure/azure-sdk-for-go/services/compute/mgmt/2019-07-01/compute.a; 6s linux-sandbox ... (8 actions running) W1105 01:10:20.059] [1,803 / 4,426] ; 21s linux-sandbox ... (8 actions, 7 running) W1105 01:11:09.566] [2,100 / 4,426] GoCompilePkg staging/src/k8s.io/apiextensions-apiserver/pkg/apiserver/schema/objectmeta/linux_amd64_pure_stripped/go_default_library%/k8s.io/kubernetes/vendor/k8s.io/apiextensions-apiserver/pkg/apiserver/schema/objectmeta.a; 3s linux-sandbox ... (8 actions, 7 running) W1105 01:12:05.886] [2,465 / 4,426] GoCompilePkg pkg/generated/openapi/linux_amd64_pure_stripped/go_default_library%/k8s.io/kubernetes/pkg/generated/openapi.a; 5s linux-sandbox ... (8 actions, 7 running) W1105 01:13:10.635] [2,722 / 4,426] GoCompilePkg pkg/apis/rbac/v1beta1/linux_amd64_stripped/go_default_library%/k8s.io/kubernetes/pkg/apis/rbac/v1beta1.a; 6s linux-sandbox ... (8 actions, 7 running) W1105 01:14:29.249] [3,082 / 4,426] GoCompilePkg pkg/controller/replicaset/linux_amd64_pure_stripped/go_default_library%/k8s.io/kubernetes/pkg/controller/replicaset.a; 8s linux-sandbox ... (8 actions, 7 running) W1105 01:15:55.976] [3,494 / 4,426] GoLink cmd/kube-controller-manager/linux_amd64_pure_stripped/kube-controller-manager; 4s linux-sandbox ... (8 actions, 7 running) W1105 01:17:35.263] [3,981 / 4,426] GoCompilePkg staging/src/k8s.io/kubectl/pkg/drain/linux_amd64_stripped/go_default_library%/k8s.io/kubernetes/vendor/k8s.io/kubectl/pkg/drain.a; 0s linux-sandbox ... (8 actions, 7 running) W1105 01:19:29.475] [4,341 / 4,429] GoLink cmd/kube-apiserver/linux_amd64_pure_stripped/kube-apiserver; 29s linux-sandbox ... (8 actions, 7 running) W1105 01:21:30.187] Slow read: a 144455855-byte read from /workspace/.cache/bazel/_bazel_root/48d5366022b4e3197674c8d6e2bee219/execroot/io_k8s_kubernetes/bazel-out/k8-fastbuild/bin/cmd/kubelet/kubelet took 25634 ms. W1105 01:21:45.146] [4,417 / 4,429] GoLink cmd/genman/genman; 108s linux-sandbox ... (7 actions running) W1105 01:24:34.564] [4,426 / 4,429] GoLink cmd/genman/genman; 273s linux-sandbox W1105 01:24:36.422] Slow read: a 305846063-byte read from /workspace/.cache/bazel/_bazel_root/48d5366022b4e3197674c8d6e2bee219/execroot/io_k8s_kubernetes/bazel-out/k8-fastbuild/bin/cmd/genman/genman took 47441 ms. W1105 01:24:56.654] Target //build/release-tars:release-tars up-to-date: W1105 01:24:56.698] bazel-bin/build/release-tars/kubernetes-client-linux-amd64.tar.gz W1105 01:24:56.698] bazel-bin/build/release-tars/kubernetes-node-linux-amd64.tar.gz W1105 01:24:56.698] bazel-bin/build/release-tars/kubernetes-server-linux-amd64.tar.gz W1105 01:24:56.699] bazel-bin/build/release-tars/kubernetes-manifests.tar.gz W1105 01:24:56.699] bazel-bin/build/release-tars/kubernetes-test-portable.tar.gz W1105 01:24:56.699] bazel-bin/build/release-tars/kubernetes-test-linux-amd64.tar.gz W1105 01:24:56.699] bazel-bin/build/release-tars/kubernetes.tar.gz W1105 01:24:56.700] bazel-bin/build/release-tars/kubernetes-src.tar.gz W1105 01:24:56.951] INFO: Elapsed time: 1212.996s, Critical Path: 445.00s W1105 01:24:56.951] INFO: 4364 processes: 4364 linux-sandbox. W1105 01:24:57.099] INFO: Build completed successfully, 4429 total actions W1105 01:24:57.149] INFO: Build completed successfully, 4429 total actions W1105 01:24:57.201] 2019/11/05 01:24:57 process.go:155: Step 'make -C /go/src/k8s.io/kubernetes bazel-release' finished in 20m13.538981818s W1105 01:24:57.201] 2019/11/05 01:24:57 util.go:277: Flushing memory. I1105 01:24:57.302] make: Leaving directory '/go/src/k8s.io/kubernetes' W1105 01:25:18.082] 2019/11/05 01:25:18 azure.go:906: Deleting resource group: kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5. W1105 01:25:18.561] 2019/11/05 01:25:18 azure.go:787: downloading /workspace/aks978705698/kubernetes.json from https://raw.githubusercontent.com/kubernetes/cloud-provider-azure/master/tests/k8s-azure/manifest/linux-vmss-multi-zones.json. W1105 01:25:18.562] 2019/11/05 01:25:18 util.go:42: curl https://raw.githubusercontent.com/kubernetes/cloud-provider-azure/master/tests/k8s-azure/manifest/linux-vmss-multi-zones.json W1105 01:25:18.770] 2019/11/05 01:25:18 azure.go:529: downloading /workspace/aks-engine.tar.gz from https://github.com/Azure/aks-engine/releases/download/v0.39.1/aks-engine-v0.39.1-linux-amd64.tar.gz. W1105 01:25:18.771] 2019/11/05 01:25:18 util.go:42: curl https://github.com/Azure/aks-engine/releases/download/v0.39.1/aks-engine-v0.39.1-linux-amd64.tar.gz W1105 01:25:20.045] 2019/11/05 01:25:20 azure.go:556: Extracting tar file /workspace/aks-engine.tar.gz into directory /go/src/k8s.io/kubernetes . W1105 01:25:20.049] 2019/11/05 01:25:20 process.go:153: Running: tar -xzf /workspace/aks-engine.tar.gz --strip 1 W1105 01:25:20.204] 2019/11/05 01:25:20 process.go:155: Step 'tar -xzf /workspace/aks-engine.tar.gz --strip 1' finished in 154.886503ms W1105 01:25:20.204] 2019/11/05 01:25:20 process.go:153: Running: /go/src/k8s.io/kubernetes/aks-engine generate /workspace/aks978705698/kubernetes.json --output-directory /workspace/aks978705698 I1105 01:25:20.417] INFO[0000] Generating assets into /workspace/aks978705698... W1105 01:25:37.979] 2019/11/05 01:25:37 process.go:155: Step '/go/src/k8s.io/kubernetes/aks-engine generate /workspace/aks978705698/kubernetes.json --output-directory /workspace/aks978705698' finished in 17.774654767s W1105 01:25:37.983] 2019/11/05 01:25:37 azure.go:627: Setting kubeconfig env variable: kubeconfig path: /workspace/aks978705698/kubeconfig/kubeconfig.eastus2.json. W1105 01:25:37.983] 2019/11/05 01:25:37 azure.go:629: Creating resource group: kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5. W1105 01:25:37.983] 2019/11/05 01:25:37 azure.go:631: Creating Azure resource group: kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5 for cluster deployment. W1105 01:25:39.991] 2019/11/05 01:25:39 azure.go:636: Validating deployment ARM templates. W1105 01:25:42.375] 2019/11/05 01:25:42 azure.go:642: Deploying cluster kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5 in resource group kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5. W1105 01:35:53.094] 2019/11/05 01:35:53 process.go:153: Running: ./cluster/kubectl.sh --match-server-version=false version W1105 01:35:53.753] 2019/11/05 01:35:53 process.go:155: Step './cluster/kubectl.sh --match-server-version=false version' finished in 658.487962ms W1105 01:35:53.753] 2019/11/05 01:35:53 process.go:153: Running: ./cluster/kubectl.sh --match-server-version=false get nodes -oyaml W1105 01:35:55.070] 2019/11/05 01:35:55 process.go:155: Step './cluster/kubectl.sh --match-server-version=false get nodes -oyaml' finished in 1.317396126s W1105 01:35:55.071] 2019/11/05 01:35:55 azure.go:970: KUBE_TEST_REPO_LIST_DOWNLOAD_LOCATION not set. Using default test image repos. W1105 01:35:55.071] 2019/11/05 01:35:55 azure.go:970: KUBE_TEST_REPO_LIST_DOWNLOAD_LOCATION not set. Using default test image repos. W1105 01:35:55.071] 2019/11/05 01:35:55 process.go:153: Running: kubectl get nodes --no-headers W1105 01:35:55.573] 2019/11/05 01:35:55 process.go:155: Step 'kubectl get nodes --no-headers' finished in 502.265337ms W1105 01:35:55.573] 2019/11/05 01:35:55 e2e.go:474: Cluster nodes: W1105 01:35:55.573] k8s-agentpool-15826033-vmss000000 Ready agent 98s v1.15.0 W1105 01:35:55.573] k8s-agentpool-15826033-vmss000001 Ready agent 106s v1.15.0 W1105 01:35:55.574] k8s-master-15826033-0 Ready master 49s v1.15.0 W1105 01:35:55.574] k8s-master-15826033-1 Ready master 48s v1.15.0 W1105 01:35:55.574] k8s-master-15826033-2 Ready master 106s v1.15.0 W1105 01:35:55.574] k8s-master-15826033-3 Ready master 49s v1.15.0 W1105 01:35:55.574] k8s-master-15826033-4 Ready master 48s v1.15.0 W1105 01:35:55.574] 2019/11/05 01:35:55 process.go:153: Running: ./cluster/kubectl.sh --match-server-version=false version W1105 01:35:55.999] 2019/11/05 01:35:55 process.go:155: Step './cluster/kubectl.sh --match-server-version=false version' finished in 425.732537ms W1105 01:35:55.999] 2019/11/05 01:35:55 process.go:153: Running: make e2e-test I1105 01:35:56.099] go test -v -timeout=0 ./test/e2e "-ginkgo.noColor" I1105 01:36:49.374] Nov 5 01:36:49.373: INFO: The --provider flag is not set. Continuing as if --provider=skeleton had been used. I1105 01:36:49.374] === RUN TestE2E I1105 01:36:49.374] Running Suite: AzureDisk CSI Driver End-to-End Tests I1105 01:36:49.374] ==================================================== I1105 01:36:49.375] Random Seed: 1572917809 - Will randomize all specs I1105 01:36:49.375] Will run 16 of 16 specs I1105 01:36:49.375] I1105 01:36:52.236] 2019/11/05 01:36:52 Attempting docker login with Azure service principal I1105 01:36:53.857] 2019/11/05 01:36:53 docker login is successful I1105 01:36:53.857] 2019/11/05 01:36:53 Installing Azure Disk CSI Driver... I1105 01:36:53.858] make[1]: Entering directory '/go/src/sigs.k8s.io/azuredisk-csi-driver' I1105 01:36:53.859] # Use v2.11.0 helm to match tiller's version in clusters made by aks-engine I1105 01:36:53.860] curl https://raw.githubusercontent.com/helm/helm/master/scripts/get | DESIRED_VERSION=v2.11.0 bash I1105 01:36:54.463] % Total % Received % Xferd Average Speed Time Time Time Current I1105 01:36:54.463] Dload Upload Total Spent Left Speed I1105 01:36:54.503] 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 7034 100 7034 0 0 171k 0 --:--:-- --:--:-- --:--:-- 171k I1105 01:36:54.560] Downloading https://get.helm.sh/helm-v2.11.0-linux-amd64.tar.gz I1105 01:36:56.210] Preparing to install helm and tiller into /usr/local/bin I1105 01:36:56.238] helm installed into /usr/local/bin/helm I1105 01:36:56.265] tiller installed into /usr/local/bin/tiller I1105 01:36:56.266] Run 'helm init' to configure helm. I1105 01:36:56.282] # Make sure tiller is ready I1105 01:36:56.283] kubectl wait pod -l name=tiller --namespace kube-system --for condition=ready --timeout 5m I1105 01:36:56.798] pod/tiller-deploy-54c96cb5df-8d9fs condition met I1105 01:36:56.799] helm version I1105 01:36:56.856] Client: &version.Version{SemVer:"v2.11.0", GitCommit:"2e55dbe1fdb5fdb96b75ff144a339489417b146b", GitTreeState:"clean"} I1105 01:36:57.805] Server: &version.Version{SemVer:"v2.11.0", GitCommit:"2e55dbe1fdb5fdb96b75ff144a339489417b146b", GitTreeState:"clean"} I1105 01:36:57.807] # Only build and push the image if it does not exist in the registry I1105 01:36:57.807] docker pull k8sprowinternal.azurecr.io/azuredisk-csi:e2e-3e3ed045c6faa509dbd8d2890ee9f56aa2934887 || make azuredisk-container push I1105 01:36:58.033] Error response from daemon: manifest for k8sprowinternal.azurecr.io/azuredisk-csi:e2e-3e3ed045c6faa509dbd8d2890ee9f56aa2934887 not found: manifest unknown: manifest unknown I1105 01:36:58.037] make[2]: Entering directory '/go/src/sigs.k8s.io/azuredisk-csi-driver' I1105 01:36:58.042] if [ ! -d ./vendor ]; then dep ensure -vendor-only; fi I1105 01:36:58.043] CGO_ENABLED=0 GOOS=linux go build -a -ldflags "-X sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.driverVersion=e2e-3e3ed045c6faa509dbd8d2890ee9f56aa2934887 -X sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.gitCommit=3e3ed045c6faa509dbd8d2890ee9f56aa2934887 -X sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.buildDate=2019-11-05T01:36:58Z -X sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.DriverName=disk.csi.azure.com -X sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.topologyKey=topology.disk.csi.azure.com/zone -extldflags "-static"" -o _output/azurediskplugin ./pkg/azurediskplugin I1105 01:37:33.633] docker build --no-cache -t k8sprowinternal.azurecr.io/azuredisk-csi:e2e-3e3ed045c6faa509dbd8d2890ee9f56aa2934887 -f ./pkg/azurediskplugin/Dockerfile . I1105 01:37:35.788] Sending build context to Docker daemon 214.4MB I1105 01:37:35.830] Step 1/6 : FROM aksrepos.azurecr.io/fundamental/base-ubuntu:v0.0.5 I1105 01:37:36.352] v0.0.5: Pulling from fundamental/base-ubuntu I1105 01:37:36.353] 5667fdb72017: Pulling fs layer I1105 01:37:36.353] d83811f270d5: Pulling fs layer I1105 01:37:36.353] ee671aafb583: Pulling fs layer I1105 01:37:36.353] 7fc152dfb3a6: Pulling fs layer I1105 01:37:36.353] 7940edb1bae7: Pulling fs layer I1105 01:37:36.353] 2ac04aec52fc: Pulling fs layer I1105 01:37:36.354] 3d25cb8b687d: Pulling fs layer I1105 01:37:36.354] 2ac04aec52fc: Waiting I1105 01:37:36.354] 7fc152dfb3a6: Waiting I1105 01:37:36.354] 3d25cb8b687d: Waiting I1105 01:37:36.354] 7940edb1bae7: Waiting I1105 01:37:36.458] d83811f270d5: Download complete I1105 01:37:36.503] ee671aafb583: Download complete I1105 01:37:36.568] 7fc152dfb3a6: Verifying Checksum I1105 01:37:36.568] 7fc152dfb3a6: Download complete I1105 01:37:36.771] 5667fdb72017: Verifying Checksum I1105 01:37:36.772] 5667fdb72017: Download complete I1105 01:37:36.826] 7940edb1bae7: Verifying Checksum I1105 01:37:36.827] 7940edb1bae7: Download complete I1105 01:37:37.135] 3d25cb8b687d: Verifying Checksum I1105 01:37:37.135] 3d25cb8b687d: Download complete I1105 01:37:37.446] 2ac04aec52fc: Verifying Checksum I1105 01:37:37.446] 2ac04aec52fc: Download complete I1105 01:37:38.861] 5667fdb72017: Pull complete I1105 01:37:39.006] d83811f270d5: Pull complete I1105 01:37:39.154] ee671aafb583: Pull complete I1105 01:37:39.301] 7fc152dfb3a6: Pull complete I1105 01:37:39.476] 7940edb1bae7: Pull complete I1105 01:37:42.171] 2ac04aec52fc: Pull complete I1105 01:37:43.080] 3d25cb8b687d: Pull complete I1105 01:37:43.102] Digest: sha256:786f291c07d8b7a063d1cabd1a143a217767b34deeca1188565c46f496e50f03 I1105 01:37:43.290] Status: Downloaded newer image for aksrepos.azurecr.io/fundamental/base-ubuntu:v0.0.5 I1105 01:37:43.291] ---> 7730e8d0ed58 I1105 01:37:43.291] Step 2/6 : RUN apt-get update && apt-get install -y util-linux e2fsprogs mount ca-certificates udev xfsprogs I1105 01:37:49.880] ---> Running in 1e205b35cef1 I1105 01:37:50.869] Hit:1 http://azure.archive.ubuntu.com/ubuntu bionic InRelease I1105 01:37:50.869] Get:2 http://azure.archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB] I1105 01:37:50.870] Get:3 http://azure.archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB] I1105 01:37:51.041] Get:4 http://azure.archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages [995 kB] I1105 01:37:51.062] Get:5 http://azure.archive.ubuntu.com/ubuntu bionic-updates/multiverse amd64 Packages [9022 B] I1105 01:37:51.063] Get:6 http://azure.archive.ubuntu.com/ubuntu bionic-updates/universe amd64 Packages [1304 kB] I1105 01:37:51.080] Get:7 http://azure.archive.ubuntu.com/ubuntu bionic-updates/restricted amd64 Packages [23.2 kB] I1105 01:37:51.124] Get:8 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB] I1105 01:37:51.136] Get:9 http://azure.archive.ubuntu.com/ubuntu bionic-backports/universe amd64 Packages [4234 B] I1105 01:37:51.841] Get:10 http://security.ubuntu.com/ubuntu bionic-security/universe amd64 Packages [782 kB] I1105 01:37:52.947] Get:11 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages [700 kB] I1105 01:37:54.060] Get:12 http://security.ubuntu.com/ubuntu bionic-security/multiverse amd64 Packages [5944 B] I1105 01:37:54.349] Get:13 http://security.ubuntu.com/ubuntu bionic-security/restricted amd64 Packages [12.6 kB] I1105 01:37:54.363] Fetched 4089 kB in 4s (1160 kB/s) I1105 01:37:55.268] Reading package lists... I1105 01:37:56.231] Reading package lists... I1105 01:37:56.411] Building dependency tree... I1105 01:37:56.411] Reading state information... I1105 01:37:56.528] ca-certificates is already the newest version (20180409). I1105 01:37:56.528] e2fsprogs is already the newest version (1.44.1-1ubuntu1.2). I1105 01:37:56.528] mount is already the newest version (2.31.1-0.4ubuntu3.4). I1105 01:37:56.528] util-linux is already the newest version (2.31.1-0.4ubuntu3.4). I1105 01:37:56.528] The following additional packages will be installed: I1105 01:37:56.529] libreadline5 I1105 01:37:56.529] Suggested packages: I1105 01:37:56.529] xfsdump acl attr quota I1105 01:37:56.551] The following NEW packages will be installed: I1105 01:37:56.551] libreadline5 udev xfsprogs I1105 01:37:56.573] 0 upgraded, 3 newly installed, 0 to remove and 1 not upgraded. I1105 01:37:56.573] Need to get 1884 kB of archives. I1105 01:37:56.573] After this operation, 12.2 MB of additional disk space will be used. I1105 01:37:56.573] Get:1 http://azure.archive.ubuntu.com/ubuntu bionic-updates/main amd64 udev amd64 237-3ubuntu10.31 [1102 kB] I1105 01:37:56.753] Get:2 http://azure.archive.ubuntu.com/ubuntu bionic/main amd64 libreadline5 amd64 5.2+dfsg-3build1 [99.5 kB] I1105 01:37:56.914] Get:3 http://azure.archive.ubuntu.com/ubuntu bionic/main amd64 xfsprogs amd64 4.9.0+nmu1ubuntu2 [683 kB] I1105 01:37:57.176] debconf: delaying package configuration, since apt-utils is not installed I1105 01:37:57.209] Fetched 1884 kB in 0s (4940 kB/s) I1105 01:37:57.294] Selecting previously unselected package udev. I1105 01:37:57.300] (Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 6917 files and directories currently installed.) I1105 01:37:57.300] Preparing to unpack .../udev_237-3ubuntu10.31_amd64.deb ... I1105 01:37:57.343] Unpacking udev (237-3ubuntu10.31) ... I1105 01:37:57.648] Selecting previously unselected package libreadline5:amd64. I1105 01:37:57.649] Preparing to unpack .../libreadline5_5.2+dfsg-3build1_amd64.deb ... I1105 01:37:57.681] Unpacking libreadline5:amd64 (5.2+dfsg-3build1) ... I1105 01:37:57.808] Selecting previously unselected package xfsprogs. I1105 01:37:57.809] Preparing to unpack .../xfsprogs_4.9.0+nmu1ubuntu2_amd64.deb ... I1105 01:37:57.825] Unpacking xfsprogs (4.9.0+nmu1ubuntu2) ... I1105 01:37:58.062] Setting up udev (237-3ubuntu10.31) ... I1105 01:37:58.442] invoke-rc.d: could not determine current runlevel I1105 01:37:58.445] invoke-rc.d: policy-rc.d denied execution of start. I1105 01:37:58.501] Setting up libreadline5:amd64 (5.2+dfsg-3build1) ... I1105 01:37:58.543] Setting up xfsprogs (4.9.0+nmu1ubuntu2) ... I1105 01:37:58.591] Processing triggers for systemd (237-3ubuntu10.31) ... I1105 01:37:58.620] Processing triggers for libc-bin (2.27-3ubuntu1) ... I1105 01:38:00.398] Removing intermediate container 1e205b35cef1 I1105 01:38:00.398] ---> 40fbc7d7f773 I1105 01:38:00.398] Step 3/6 : LABEL maintainers="andyzhangx" I1105 01:38:00.673] ---> Running in 6c9b73012e7a I1105 01:38:01.146] Removing intermediate container 6c9b73012e7a I1105 01:38:01.146] ---> 956fe6db464f I1105 01:38:01.146] Step 4/6 : LABEL description="Azure Disk CSI Driver" I1105 01:38:01.311] ---> Running in 233da61ca944 I1105 01:38:02.102] Removing intermediate container 233da61ca944 I1105 01:38:02.102] ---> 9ddbc2cfe29f I1105 01:38:02.102] Step 5/6 : COPY ./_output/azurediskplugin /azurediskplugin I1105 01:38:04.439] ---> 573631df052d I1105 01:38:04.439] Step 6/6 : ENTRYPOINT ["/azurediskplugin"] I1105 01:38:04.583] ---> Running in 2660b9695037 I1105 01:38:05.113] Removing intermediate container 2660b9695037 I1105 01:38:05.113] ---> 585a77f52f87 I1105 01:38:05.256] Successfully built 585a77f52f87 I1105 01:38:05.501] Successfully tagged k8sprowinternal.azurecr.io/azuredisk-csi:e2e-3e3ed045c6faa509dbd8d2890ee9f56aa2934887 I1105 01:38:05.504] docker push k8sprowinternal.azurecr.io/azuredisk-csi:e2e-3e3ed045c6faa509dbd8d2890ee9f56aa2934887 I1105 01:38:05.570] The push refers to repository [k8sprowinternal.azurecr.io/azuredisk-csi] I1105 01:38:05.582] f821302637c0: Preparing I1105 01:38:05.582] d8d0bf7d8b3d: Preparing I1105 01:38:05.582] 77098f020d77: Preparing I1105 01:38:05.582] f780fac61456: Preparing I1105 01:38:05.582] f89c15b53b76: Preparing I1105 01:38:05.582] e80c789bc6ac: Preparing I1105 01:38:05.582] 6c3332381368: Preparing I1105 01:38:05.583] ef1a1ec5bba9: Preparing I1105 01:38:05.583] a1aa3da2a80a: Preparing I1105 01:38:05.583] ef1a1ec5bba9: Waiting I1105 01:38:05.583] 6c3332381368: Waiting I1105 01:38:05.583] e80c789bc6ac: Waiting I1105 01:38:05.583] a1aa3da2a80a: Waiting I1105 01:38:05.970] f89c15b53b76: Pushed I1105 01:38:06.368] e80c789bc6ac: Pushed I1105 01:38:06.731] 6c3332381368: Pushed I1105 01:38:07.122] ef1a1ec5bba9: Pushed I1105 01:38:07.686] d8d0bf7d8b3d: Pushed I1105 01:38:08.002] 77098f020d77: Pushed I1105 01:38:09.517] f821302637c0: Pushed I1105 01:38:12.104] a1aa3da2a80a: Pushed I1105 01:38:14.148] f780fac61456: Pushed I1105 01:38:15.011] e2e-3e3ed045c6faa509dbd8d2890ee9f56aa2934887: digest: sha256:75d4ff5d7cd845d6e810c67796ecb037e852fb849ec1eb5b3194564008e6d1e0 size: 2206 I1105 01:38:15.035] make[2]: Leaving directory '/go/src/sigs.k8s.io/azuredisk-csi-driver' I1105 01:38:15.035] helm install charts/latest/azuredisk-csi-driver -n azuredisk-csi-driver --namespace kube-system --wait \ I1105 01:38:15.035] --set image.pullPolicy=IfNotPresent \ I1105 01:38:15.036] --set image.repository=k8sprowinternal.azurecr.io/azuredisk-csi \ I1105 01:38:15.036] --set image.tag=e2e-3e3ed045c6faa509dbd8d2890ee9f56aa2934887 I1105 01:39:16.356] NAME: azuredisk-csi-driver I1105 01:39:17.955] LAST DEPLOYED: Tue Nov 5 01:38:16 2019 I1105 01:39:17.955] NAMESPACE: kube-system I1105 01:39:17.955] STATUS: DEPLOYED I1105 01:39:17.955] I1105 01:39:17.955] RESOURCES: I1105 01:39:17.956] ==> v1/Deployment I1105 01:39:17.956] NAME AGE I1105 01:39:17.956] csi-azuredisk-controller 53s I1105 01:39:17.956] I1105 01:39:17.956] ==> v1/Pod(related) I1105 01:39:17.956] I1105 01:39:17.956] NAME READY STATUS RESTARTS AGE I1105 01:39:17.956] csi-azuredisk-node-ghv5w 3/3 Running 0 53s I1105 01:39:17.957] csi-azuredisk-node-whq2g 3/3 Running 0 53s I1105 01:39:17.957] csi-azuredisk-controller-6f6587bfc6-8vkfl 6/6 Running 0 53s I1105 01:39:17.957] I1105 01:39:17.957] ==> v1/ServiceAccount I1105 01:39:17.957] I1105 01:39:17.957] NAME AGE I1105 01:39:17.958] csi-azuredisk-controller-sa 54s I1105 01:39:17.958] I1105 01:39:17.958] ==> v1beta1/CustomResourceDefinition I1105 01:39:17.958] csidrivers.csi.storage.k8s.io 54s I1105 01:39:17.959] csinodeinfos.csi.storage.k8s.io 54s I1105 01:39:17.959] I1105 01:39:17.959] ==> v1/ClusterRole I1105 01:39:17.959] azuredisk-external-provisioner-role 54s I1105 01:39:17.959] azuredisk-external-attacher-role 54s I1105 01:39:17.960] azuredisk-external-snapshotter-role 54s I1105 01:39:17.960] azuredisk-cluster-driver-registrar-role 54s I1105 01:39:17.960] I1105 01:39:17.960] ==> v1/ClusterRoleBinding I1105 01:39:17.960] azuredisk-csi-provisioner-binding 54s I1105 01:39:17.961] azuredisk-csi-attacher-binding 54s I1105 01:39:17.961] azuredisk-csi-snapshotter-binding 54s I1105 01:39:17.961] azuredisk-csi-driver-registrar-binding 54s I1105 01:39:17.961] I1105 01:39:17.961] ==> v1/Service I1105 01:39:17.962] csi-azuredisk-controller 54s I1105 01:39:17.962] I1105 01:39:17.962] ==> v1/DaemonSet I1105 01:39:17.962] csi-azuredisk-node 54s I1105 01:39:17.962] I1105 01:39:17.962] I1105 01:39:17.963] NOTES: I1105 01:39:17.963] The Azure Disk CSI Driver is getting deployed to your cluster. I1105 01:39:17.963] I1105 01:39:17.963] To check Azure Disk CSI Driver pods status, please run: I1105 01:39:17.963] I1105 01:39:17.963] kubectl --namespace=kube-system get pods --selector="release=azuredisk-csi-driver" --watch I1105 01:39:17.963] I1105 01:39:17.964] make[1]: Leaving directory '/go/src/sigs.k8s.io/azuredisk-csi-driver' I1105 01:39:17.964] 2019/11/05 01:39:17 Azure Disk CSI Driver installed I1105 01:39:17.964] [azuredisk-csi-e2e] Dynamic Provisioning [single-az] I1105 01:39:17.964] should create a volume on demand and mount it as readOnly in a pod I1105 01:39:17.964] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:126 I1105 01:39:17.965] [BeforeEach] [single-az] I1105 01:39:17.965] /go/src/sigs.k8s.io/azuredisk-csi-driver/vendor/k8s.io/kubernetes/test/e2e/framework/framework.go:150 I1105 01:39:17.965] STEP: Creating a kubernetes client I1105 01:39:17.966] Nov 5 01:39:17.959: INFO: >>> kubeConfig: /workspace/aks978705698/kubeconfig/kubeconfig.eastus2.json I1105 01:39:17.966] STEP: Building a namespace api object, basename azuredisk I1105 01:39:18.573] Nov 5 01:39:18.572: INFO: Found PodSecurityPolicies; assuming PodSecurityPolicy is enabled. I1105 01:39:18.937] Nov 5 01:39:18.937: INFO: Found ClusterRoles; assuming RBAC is enabled. I1105 01:39:19.018] STEP: Binding the e2e-test-privileged-psp PodSecurityPolicy to the default service account in azuredisk-3540 I1105 01:39:19.277] STEP: Waiting for a default service account to be provisioned in namespace I1105 01:39:19.352] [BeforeEach] [single-az] I1105 01:39:19.353] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:58 I1105 01:39:19.353] [It] should create a volume on demand and mount it as readOnly in a pod I1105 01:39:19.353] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:126 I1105 01:39:19.353] STEP: setting up the StorageClass I1105 01:39:19.353] STEP: creating a StorageClass I1105 01:39:19.438] STEP: setting up the PVC and PV I1105 01:39:19.439] STEP: creating a PVC I1105 01:39:19.523] STEP: waiting for PVC to be in phase "Bound" I1105 01:39:19.523] Nov 5 01:39:19.523: INFO: Waiting up to 5m0s for PersistentVolumeClaims [pvc-lvm42] to have phase Bound I1105 01:39:19.600] Nov 5 01:39:19.600: INFO: PersistentVolumeClaim pvc-lvm42 found but phase is Pending instead of Bound. I1105 01:39:21.678] Nov 5 01:39:21.678: INFO: PersistentVolumeClaim pvc-lvm42 found but phase is Pending instead of Bound. I1105 01:39:23.756] Nov 5 01:39:23.756: INFO: PersistentVolumeClaim pvc-lvm42 found but phase is Pending instead of Bound. I1105 01:39:25.850] Nov 5 01:39:25.850: INFO: PersistentVolumeClaim pvc-lvm42 found but phase is Pending instead of Bound. I1105 01:39:27.928] Nov 5 01:39:27.927: INFO: PersistentVolumeClaim pvc-lvm42 found but phase is Pending instead of Bound. I1105 01:39:30.005] Nov 5 01:39:30.005: INFO: PersistentVolumeClaim pvc-lvm42 found but phase is Pending instead of Bound. I1105 01:39:32.084] Nov 5 01:39:32.083: INFO: PersistentVolumeClaim pvc-lvm42 found but phase is Pending instead of Bound. I1105 01:39:34.161] Nov 5 01:39:34.161: INFO: PersistentVolumeClaim pvc-lvm42 found but phase is Pending instead of Bound. I1105 01:39:36.239] Nov 5 01:39:36.239: INFO: PersistentVolumeClaim pvc-lvm42 found but phase is Pending instead of Bound. I1105 01:39:38.317] Nov 5 01:39:38.316: INFO: PersistentVolumeClaim pvc-lvm42 found but phase is Pending instead of Bound. I1105 01:39:40.395] Nov 5 01:39:40.394: INFO: PersistentVolumeClaim pvc-lvm42 found but phase is Pending instead of Bound. I1105 01:39:42.472] Nov 5 01:39:42.472: INFO: PersistentVolumeClaim pvc-lvm42 found but phase is Pending instead of Bound. I1105 01:39:44.550] Nov 5 01:39:44.550: INFO: PersistentVolumeClaim pvc-lvm42 found but phase is Pending instead of Bound. I1105 01:39:46.633] Nov 5 01:39:46.632: INFO: PersistentVolumeClaim pvc-lvm42 found but phase is Pending instead of Bound. I1105 01:39:48.710] Nov 5 01:39:48.709: INFO: PersistentVolumeClaim pvc-lvm42 found but phase is Pending instead of Bound. I1105 01:39:50.787] Nov 5 01:39:50.787: INFO: PersistentVolumeClaim pvc-lvm42 found but phase is Pending instead of Bound. I1105 01:39:52.865] Nov 5 01:39:52.865: INFO: PersistentVolumeClaim pvc-lvm42 found but phase is Pending instead of Bound. I1105 01:39:54.944] Nov 5 01:39:54.944: INFO: PersistentVolumeClaim pvc-lvm42 found but phase is Pending instead of Bound. I1105 01:39:57.022] Nov 5 01:39:57.021: INFO: PersistentVolumeClaim pvc-lvm42 found but phase is Pending instead of Bound. I1105 01:39:59.106] Nov 5 01:39:59.105: INFO: PersistentVolumeClaim pvc-lvm42 found but phase is Pending instead of Bound. I1105 01:40:01.183] Nov 5 01:40:01.183: INFO: PersistentVolumeClaim pvc-lvm42 found but phase is Pending instead of Bound. I1105 01:40:03.261] Nov 5 01:40:03.260: INFO: PersistentVolumeClaim pvc-lvm42 found but phase is Pending instead of Bound. I1105 01:40:05.338] Nov 5 01:40:05.338: INFO: PersistentVolumeClaim pvc-lvm42 found but phase is Pending instead of Bound. I1105 01:40:07.417] Nov 5 01:40:07.416: INFO: PersistentVolumeClaim pvc-lvm42 found but phase is Pending instead of Bound. I1105 01:40:09.494] Nov 5 01:40:09.494: INFO: PersistentVolumeClaim pvc-lvm42 found but phase is Pending instead of Bound. I1105 01:40:11.572] Nov 5 01:40:11.572: INFO: PersistentVolumeClaim pvc-lvm42 found but phase is Pending instead of Bound. I1105 01:40:13.649] Nov 5 01:40:13.649: INFO: PersistentVolumeClaim pvc-lvm42 found but phase is Pending instead of Bound. I1105 01:40:15.728] Nov 5 01:40:15.727: INFO: PersistentVolumeClaim pvc-lvm42 found but phase is Pending instead of Bound. I1105 01:40:17.806] Nov 5 01:40:17.805: INFO: PersistentVolumeClaim pvc-lvm42 found but phase is Pending instead of Bound. I1105 01:40:19.884] Nov 5 01:40:19.883: INFO: PersistentVolumeClaim pvc-lvm42 found but phase is Pending instead of Bound. I1105 01:40:21.963] Nov 5 01:40:21.963: INFO: PersistentVolumeClaim pvc-lvm42 found but phase is Pending instead of Bound. I1105 01:40:24.041] Nov 5 01:40:24.041: INFO: PersistentVolumeClaim pvc-lvm42 found but phase is Pending instead of Bound. I1105 01:40:26.120] Nov 5 01:40:26.119: INFO: PersistentVolumeClaim pvc-lvm42 found but phase is Pending instead of Bound. I1105 01:40:28.197] Nov 5 01:40:28.197: INFO: PersistentVolumeClaim pvc-lvm42 found but phase is Pending instead of Bound. I1105 01:40:30.275] Nov 5 01:40:30.275: INFO: PersistentVolumeClaim pvc-lvm42 found but phase is Pending instead of Bound. I1105 01:40:32.353] Nov 5 01:40:32.353: INFO: PersistentVolumeClaim pvc-lvm42 found but phase is Pending instead of Bound. I1105 01:40:34.430] Nov 5 01:40:34.430: INFO: PersistentVolumeClaim pvc-lvm42 found but phase is Pending instead of Bound. I1105 01:40:36.508] Nov 5 01:40:36.508: INFO: PersistentVolumeClaim pvc-lvm42 found but phase is Pending instead of Bound. I1105 01:40:38.586] Nov 5 01:40:38.586: INFO: PersistentVolumeClaim pvc-lvm42 found but phase is Pending instead of Bound. I1105 01:40:40.664] Nov 5 01:40:40.663: INFO: PersistentVolumeClaim pvc-lvm42 found but phase is Pending instead of Bound. I1105 01:40:42.742] Nov 5 01:40:42.742: INFO: PersistentVolumeClaim pvc-lvm42 found but phase is Pending instead of Bound. I1105 01:40:44.820] Nov 5 01:40:44.819: INFO: PersistentVolumeClaim pvc-lvm42 found but phase is Pending instead of Bound. I1105 01:40:46.902] Nov 5 01:40:46.902: INFO: PersistentVolumeClaim pvc-lvm42 found but phase is Pending instead of Bound. I1105 01:40:48.980] Nov 5 01:40:48.979: INFO: PersistentVolumeClaim pvc-lvm42 found but phase is Pending instead of Bound. I1105 01:40:51.057] Nov 5 01:40:51.057: INFO: PersistentVolumeClaim pvc-lvm42 found but phase is Pending instead of Bound. I1105 01:40:53.169] Nov 5 01:40:53.168: INFO: PersistentVolumeClaim pvc-lvm42 found but phase is Pending instead of Bound. I1105 01:40:55.247] Nov 5 01:40:55.247: INFO: PersistentVolumeClaim pvc-lvm42 found but phase is Pending instead of Bound. I1105 01:40:57.336] Nov 5 01:40:57.336: INFO: PersistentVolumeClaim pvc-lvm42 found but phase is Pending instead of Bound. I1105 01:40:59.414] Nov 5 01:40:59.413: INFO: PersistentVolumeClaim pvc-lvm42 found but phase is Pending instead of Bound. I1105 01:41:01.492] Nov 5 01:41:01.491: INFO: PersistentVolumeClaim pvc-lvm42 found but phase is Pending instead of Bound. I1105 01:41:03.569] Nov 5 01:41:03.569: INFO: PersistentVolumeClaim pvc-lvm42 found but phase is Pending instead of Bound. I1105 01:41:05.646] Nov 5 01:41:05.646: INFO: PersistentVolumeClaim pvc-lvm42 found but phase is Pending instead of Bound. I1105 01:41:07.725] Nov 5 01:41:07.724: INFO: PersistentVolumeClaim pvc-lvm42 found but phase is Pending instead of Bound. I1105 01:41:09.803] Nov 5 01:41:09.803: INFO: PersistentVolumeClaim pvc-lvm42 found but phase is Pending instead of Bound. I1105 01:41:11.881] Nov 5 01:41:11.880: INFO: PersistentVolumeClaim pvc-lvm42 found but phase is Pending instead of Bound. I1105 01:41:13.958] Nov 5 01:41:13.958: INFO: PersistentVolumeClaim pvc-lvm42 found but phase is Pending instead of Bound. I1105 01:41:16.036] Nov 5 01:41:16.036: INFO: PersistentVolumeClaim pvc-lvm42 found but phase is Pending instead of Bound. I1105 01:41:18.118] Nov 5 01:41:18.118: INFO: PersistentVolumeClaim pvc-lvm42 found but phase is Pending instead of Bound. I1105 01:41:20.196] Nov 5 01:41:20.196: INFO: PersistentVolumeClaim pvc-lvm42 found but phase is Pending instead of Bound. I1105 01:41:22.274] Nov 5 01:41:22.274: INFO: PersistentVolumeClaim pvc-lvm42 found but phase is Pending instead of Bound. I1105 01:41:24.352] Nov 5 01:41:24.352: INFO: PersistentVolumeClaim pvc-lvm42 found but phase is Pending instead of Bound. I1105 01:41:26.430] Nov 5 01:41:26.430: INFO: PersistentVolumeClaim pvc-lvm42 found but phase is Pending instead of Bound. I1105 01:41:28.508] Nov 5 01:41:28.507: INFO: PersistentVolumeClaim pvc-lvm42 found but phase is Pending instead of Bound. I1105 01:41:30.586] Nov 5 01:41:30.585: INFO: PersistentVolumeClaim pvc-lvm42 found but phase is Pending instead of Bound. I1105 01:41:32.663] Nov 5 01:41:32.663: INFO: PersistentVolumeClaim pvc-lvm42 found but phase is Pending instead of Bound. I1105 01:41:34.741] Nov 5 01:41:34.741: INFO: PersistentVolumeClaim pvc-lvm42 found but phase is Pending instead of Bound. I1105 01:41:36.819] Nov 5 01:41:36.818: INFO: PersistentVolumeClaim pvc-lvm42 found but phase is Pending instead of Bound. I1105 01:41:38.897] Nov 5 01:41:38.897: INFO: PersistentVolumeClaim pvc-lvm42 found but phase is Pending instead of Bound. I1105 01:41:40.975] Nov 5 01:41:40.975: INFO: PersistentVolumeClaim pvc-lvm42 found but phase is Pending instead of Bound. I1105 01:41:43.053] Nov 5 01:41:43.053: INFO: PersistentVolumeClaim pvc-lvm42 found but phase is Pending instead of Bound. I1105 01:41:45.143] Nov 5 01:41:45.143: INFO: PersistentVolumeClaim pvc-lvm42 found but phase is Pending instead of Bound. I1105 01:41:47.221] Nov 5 01:41:47.221: INFO: PersistentVolumeClaim pvc-lvm42 found but phase is Pending instead of Bound. I1105 01:41:49.299] Nov 5 01:41:49.299: INFO: PersistentVolumeClaim pvc-lvm42 found but phase is Pending instead of Bound. I1105 01:41:51.377] Nov 5 01:41:51.377: INFO: PersistentVolumeClaim pvc-lvm42 found but phase is Pending instead of Bound. I1105 01:41:53.455] Nov 5 01:41:53.454: INFO: PersistentVolumeClaim pvc-lvm42 found but phase is Pending instead of Bound. I1105 01:41:55.533] Nov 5 01:41:55.532: INFO: PersistentVolumeClaim pvc-lvm42 found but phase is Pending instead of Bound. I1105 01:41:57.610] Nov 5 01:41:57.610: INFO: PersistentVolumeClaim pvc-lvm42 found but phase is Pending instead of Bound. I1105 01:41:59.688] Nov 5 01:41:59.688: INFO: PersistentVolumeClaim pvc-lvm42 found but phase is Pending instead of Bound. I1105 01:42:01.766] Nov 5 01:42:01.766: INFO: PersistentVolumeClaim pvc-lvm42 found but phase is Pending instead of Bound. I1105 01:42:03.844] Nov 5 01:42:03.844: INFO: PersistentVolumeClaim pvc-lvm42 found but phase is Pending instead of Bound. I1105 01:42:05.922] Nov 5 01:42:05.922: INFO: PersistentVolumeClaim pvc-lvm42 found but phase is Pending instead of Bound. I1105 01:42:08.000] Nov 5 01:42:08.000: INFO: PersistentVolumeClaim pvc-lvm42 found but phase is Pending instead of Bound. I1105 01:42:10.089] Nov 5 01:42:10.089: INFO: PersistentVolumeClaim pvc-lvm42 found but phase is Pending instead of Bound. I1105 01:42:12.167] Nov 5 01:42:12.167: INFO: PersistentVolumeClaim pvc-lvm42 found but phase is Pending instead of Bound. I1105 01:42:14.245] Nov 5 01:42:14.245: INFO: PersistentVolumeClaim pvc-lvm42 found but phase is Pending instead of Bound. I1105 01:42:16.323] Nov 5 01:42:16.323: INFO: PersistentVolumeClaim pvc-lvm42 found but phase is Pending instead of Bound. I1105 01:42:18.401] Nov 5 01:42:18.400: INFO: PersistentVolumeClaim pvc-lvm42 found but phase is Pending instead of Bound. I1105 01:42:20.478] Nov 5 01:42:20.478: INFO: PersistentVolumeClaim pvc-lvm42 found but phase is Pending instead of Bound. I1105 01:42:22.556] Nov 5 01:42:22.556: INFO: PersistentVolumeClaim pvc-lvm42 found but phase is Pending instead of Bound. I1105 01:42:24.634] Nov 5 01:42:24.634: INFO: PersistentVolumeClaim pvc-lvm42 found but phase is Pending instead of Bound. I1105 01:42:26.712] Nov 5 01:42:26.712: INFO: PersistentVolumeClaim pvc-lvm42 found but phase is Pending instead of Bound. I1105 01:42:28.790] Nov 5 01:42:28.790: INFO: PersistentVolumeClaim pvc-lvm42 found but phase is Pending instead of Bound. I1105 01:42:30.868] Nov 5 01:42:30.867: INFO: PersistentVolumeClaim pvc-lvm42 found but phase is Pending instead of Bound. I1105 01:42:32.949] Nov 5 01:42:32.948: INFO: PersistentVolumeClaim pvc-lvm42 found but phase is Pending instead of Bound. I1105 01:42:35.027] Nov 5 01:42:35.026: INFO: PersistentVolumeClaim pvc-lvm42 found but phase is Pending instead of Bound. I1105 01:42:37.133] Nov 5 01:42:37.133: INFO: PersistentVolumeClaim pvc-lvm42 found but phase is Pending instead of Bound. I1105 01:42:39.211] Nov 5 01:42:39.211: INFO: PersistentVolumeClaim pvc-lvm42 found but phase is Pending instead of Bound. I1105 01:42:41.289] Nov 5 01:42:41.288: INFO: PersistentVolumeClaim pvc-lvm42 found but phase is Pending instead of Bound. I1105 01:42:43.367] Nov 5 01:42:43.366: INFO: PersistentVolumeClaim pvc-lvm42 found but phase is Pending instead of Bound. I1105 01:42:45.444] Nov 5 01:42:45.444: INFO: PersistentVolumeClaim pvc-lvm42 found but phase is Pending instead of Bound. I1105 01:42:47.522] Nov 5 01:42:47.522: INFO: PersistentVolumeClaim pvc-lvm42 found but phase is Pending instead of Bound. I1105 01:42:49.600] Nov 5 01:42:49.600: INFO: PersistentVolumeClaim pvc-lvm42 found but phase is Pending instead of Bound. I1105 01:42:51.678] Nov 5 01:42:51.678: INFO: PersistentVolumeClaim pvc-lvm42 found but phase is Pending instead of Bound. I1105 01:42:53.755] Nov 5 01:42:53.755: INFO: PersistentVolumeClaim pvc-lvm42 found but phase is Pending instead of Bound. I1105 01:42:55.834] Nov 5 01:42:55.834: INFO: PersistentVolumeClaim pvc-lvm42 found but phase is Pending instead of Bound. I1105 01:42:57.912] Nov 5 01:42:57.912: INFO: PersistentVolumeClaim pvc-lvm42 found but phase is Pending instead of Bound. I1105 01:42:59.990] Nov 5 01:42:59.990: INFO: PersistentVolumeClaim pvc-lvm42 found but phase is Pending instead of Bound. I1105 01:43:02.069] Nov 5 01:43:02.068: INFO: PersistentVolumeClaim pvc-lvm42 found but phase is Pending instead of Bound. I1105 01:43:04.147] Nov 5 01:43:04.147: INFO: PersistentVolumeClaim pvc-lvm42 found but phase is Pending instead of Bound. I1105 01:43:06.225] Nov 5 01:43:06.225: INFO: PersistentVolumeClaim pvc-lvm42 found but phase is Pending instead of Bound. I1105 01:43:08.302] Nov 5 01:43:08.302: INFO: PersistentVolumeClaim pvc-lvm42 found but phase is Pending instead of Bound. I1105 01:43:10.380] Nov 5 01:43:10.380: INFO: PersistentVolumeClaim pvc-lvm42 found but phase is Pending instead of Bound. I1105 01:43:12.458] Nov 5 01:43:12.458: INFO: PersistentVolumeClaim pvc-lvm42 found but phase is Pending instead of Bound. I1105 01:43:14.536] Nov 5 01:43:14.536: INFO: PersistentVolumeClaim pvc-lvm42 found but phase is Pending instead of Bound. I1105 01:43:16.614] Nov 5 01:43:16.614: INFO: PersistentVolumeClaim pvc-lvm42 found but phase is Pending instead of Bound. I1105 01:43:18.691] Nov 5 01:43:18.691: INFO: PersistentVolumeClaim pvc-lvm42 found but phase is Pending instead of Bound. I1105 01:43:20.768] Nov 5 01:43:20.768: INFO: PersistentVolumeClaim pvc-lvm42 found but phase is Pending instead of Bound. I1105 01:43:22.847] Nov 5 01:43:22.846: INFO: PersistentVolumeClaim pvc-lvm42 found but phase is Pending instead of Bound. I1105 01:43:24.924] Nov 5 01:43:24.924: INFO: PersistentVolumeClaim pvc-lvm42 found but phase is Pending instead of Bound. I1105 01:43:27.002] Nov 5 01:43:27.002: INFO: PersistentVolumeClaim pvc-lvm42 found but phase is Pending instead of Bound. I1105 01:43:29.080] Nov 5 01:43:29.079: INFO: PersistentVolumeClaim pvc-lvm42 found but phase is Pending instead of Bound. I1105 01:43:31.158] Nov 5 01:43:31.158: INFO: PersistentVolumeClaim pvc-lvm42 found but phase is Pending instead of Bound. I1105 01:43:33.235] Nov 5 01:43:33.235: INFO: PersistentVolumeClaim pvc-lvm42 found but phase is Pending instead of Bound. I1105 01:43:35.314] Nov 5 01:43:35.313: INFO: PersistentVolumeClaim pvc-lvm42 found but phase is Pending instead of Bound. I1105 01:43:37.392] Nov 5 01:43:37.392: INFO: PersistentVolumeClaim pvc-lvm42 found but phase is Pending instead of Bound. I1105 01:43:39.470] Nov 5 01:43:39.470: INFO: PersistentVolumeClaim pvc-lvm42 found but phase is Pending instead of Bound. I1105 01:43:41.548] Nov 5 01:43:41.547: INFO: PersistentVolumeClaim pvc-lvm42 found but phase is Pending instead of Bound. I1105 01:43:43.625] Nov 5 01:43:43.625: INFO: PersistentVolumeClaim pvc-lvm42 found but phase is Pending instead of Bound. I1105 01:43:45.703] Nov 5 01:43:45.703: INFO: PersistentVolumeClaim pvc-lvm42 found but phase is Pending instead of Bound. I1105 01:43:47.781] Nov 5 01:43:47.781: INFO: PersistentVolumeClaim pvc-lvm42 found but phase is Pending instead of Bound. I1105 01:43:49.860] Nov 5 01:43:49.860: INFO: PersistentVolumeClaim pvc-lvm42 found but phase is Pending instead of Bound. I1105 01:43:51.941] Nov 5 01:43:51.940: INFO: PersistentVolumeClaim pvc-lvm42 found but phase is Pending instead of Bound. I1105 01:43:54.018] Nov 5 01:43:54.018: INFO: PersistentVolumeClaim pvc-lvm42 found but phase is Pending instead of Bound. I1105 01:43:56.096] Nov 5 01:43:56.096: INFO: PersistentVolumeClaim pvc-lvm42 found but phase is Pending instead of Bound. I1105 01:43:58.174] Nov 5 01:43:58.174: INFO: PersistentVolumeClaim pvc-lvm42 found but phase is Pending instead of Bound. I1105 01:44:00.252] Nov 5 01:44:00.252: INFO: PersistentVolumeClaim pvc-lvm42 found but phase is Pending instead of Bound. I1105 01:44:02.331] Nov 5 01:44:02.331: INFO: PersistentVolumeClaim pvc-lvm42 found but phase is Pending instead of Bound. I1105 01:44:04.409] Nov 5 01:44:04.409: INFO: PersistentVolumeClaim pvc-lvm42 found but phase is Pending instead of Bound. I1105 01:44:06.488] Nov 5 01:44:06.488: INFO: PersistentVolumeClaim pvc-lvm42 found but phase is Pending instead of Bound. I1105 01:44:08.565] Nov 5 01:44:08.565: INFO: PersistentVolumeClaim pvc-lvm42 found but phase is Pending instead of Bound. I1105 01:44:10.643] Nov 5 01:44:10.643: INFO: PersistentVolumeClaim pvc-lvm42 found but phase is Pending instead of Bound. I1105 01:44:12.721] Nov 5 01:44:12.721: INFO: PersistentVolumeClaim pvc-lvm42 found but phase is Pending instead of Bound. I1105 01:44:14.799] Nov 5 01:44:14.799: INFO: PersistentVolumeClaim pvc-lvm42 found but phase is Pending instead of Bound. I1105 01:44:16.877] Nov 5 01:44:16.877: INFO: PersistentVolumeClaim pvc-lvm42 found but phase is Pending instead of Bound. I1105 01:44:18.961] Nov 5 01:44:18.961: INFO: PersistentVolumeClaim pvc-lvm42 found but phase is Pending instead of Bound. I1105 01:44:20.962] Nov 5 01:44:20.961: INFO: Unexpected error occurred: PersistentVolumeClaims [pvc-lvm42] not all in phase Bound within 5m0s I1105 01:44:20.962] [AfterEach] [single-az] I1105 01:44:20.962] /go/src/sigs.k8s.io/azuredisk-csi-driver/vendor/k8s.io/kubernetes/test/e2e/framework/framework.go:151 I1105 01:44:20.963] STEP: Collecting events from namespace "azuredisk-3540". I1105 01:44:21.040] STEP: Found 3 events. I1105 01:44:21.041] Nov 5 01:44:21.040: INFO: At 2019-11-05 01:39:19 +0000 UTC - event for pvc-lvm42: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "disk.csi.azure.com" or manually created by system administrator I1105 01:44:21.041] Nov 5 01:44:21.040: INFO: At 2019-11-05 01:39:19 +0000 UTC - event for pvc-lvm42: {disk.csi.azure.com_csi-azuredisk-controller-6f6587bfc6-8vkfl_1376914b-ff6d-11e9-93ab-ba2bee589b8c } Provisioning: External provisioner is provisioning volume for claim "azuredisk-3540/pvc-lvm42" I1105 01:44:21.042] Nov 5 01:44:21.040: INFO: At 2019-11-05 01:39:19 +0000 UTC - event for pvc-lvm42: {disk.csi.azure.com_csi-azuredisk-controller-6f6587bfc6-8vkfl_1376914b-ff6d-11e9-93ab-ba2bee589b8c } ProvisioningFailed: failed to provision volume with StorageClass "azuredisk-3540-disk.csi.azure.com-dynamic-sc-xhvsx": rpc error: code = Unknown desc = compute.DisksClient#CreateOrUpdate: Failure sending request: StatusCode=400 -- Original Error: Code="InvalidAvailabilityZone" Message="The zone(s) '' for resource 'Microsoft.Compute/disks/pvc-5ad5d2fd-ab34-408e-8b14-487779c88eb4' is not supported. The supported zones for location 'eastus2' are '1,2,3'" I1105 01:44:21.122] Nov 5 01:44:21.122: INFO: POD NODE PHASE GRACE CONDITIONS I1105 01:44:21.122] Nov 5 01:44:21.122: INFO: I1105 01:44:21.353] Nov 5 01:44:21.352: INFO: I1105 01:44:21.353] Logging node info for node k8s-agentpool-15826033-vmss000000 I1105 01:44:21.437] Nov 5 01:44:21.433: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-agentpool-15826033-vmss000000,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-agentpool-15826033-vmss000000,UID:15fa874e-54ec-42cd-b243-d3baa512150f,ResourceVersion:2765,Generation:0,CreationTimestamp:2019-11-05 01:34:17 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{agentpool: agentpool,beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_D4s_v3,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 0,kubernetes.azure.com/cluster: kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5,kubernetes.azure.com/role: agent,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-agentpool-15826033-vmss000000,kubernetes.io/os: linux,kubernetes.io/role: agent,node-role.kubernetes.io/agent: ,storageprofile: managed,storagetier: Premium_LRS,topology.disk.csi.azure.com/zone: 0,},Annotations:map[string]string{csi.volume.kubernetes.io/nodeid: {"disk.csi.azure.com":"k8s-agentpool-15826033-vmss000000"},node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.2.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool-15826033-vmss/virtualMachines/0,Unschedulable:false,Taints:[],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-csi-disk.csi.azure.com: {{8 0} {} 8 DecimalSI},cpu: {{4 0} {} 4 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{16797044736 0} {} 16403364Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-csi-disk.csi.azure.com: {{8 0} {} 8 DecimalSI},cpu: {{4 0} {} 4 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{16010612736 0} {} 15635364Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 01:34:56 +0000 UTC 2019-11-05 01:34:56 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 01:43:49 +0000 UTC 2019-11-05 01:34:17 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 01:43:49 +0000 UTC 2019-11-05 01:34:17 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 01:43:49 +0000 UTC 2019-11-05 01:34:17 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 01:43:49 +0000 UTC 2019-11-05 01:34:29 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.0.4} {Hostname k8s-agentpool-15826033-vmss000000}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:afe00792177d49a98f6a44d03d08732d,SystemUUID:4F1777BF-A5C3-004E-81C4-90F0B54C02A0,BootID:13b6a628-ae89-44af-b891-89461030adcb,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[k8sprowinternal.azurecr.io/azuredisk-csi@sha256:75d4ff5d7cd845d6e810c67796ecb037e852fb849ec1eb5b3194564008e6d1e0 k8sprowinternal.azurecr.io/azuredisk-csi:e2e-3e3ed045c6faa509dbd8d2890ee9f56aa2934887] 286930414} {[k8s.gcr.io/kubernetes-dashboard-amd64@sha256:0ae6b69432e78069c5ce2bcde0fe409c5c4d6f0f4d9cd50a17974fea38898747 k8s.gcr.io/kubernetes-dashboard-amd64:v1.10.1] 121711221} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/metrics-server-amd64@sha256:49a9f12f7067d11f42c803dbe61ed2c1299959ad85cb315b25ff7eef8e6b8892 k8s.gcr.io/metrics-server-amd64:v0.2.1] 42541759} {[mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume@sha256:4fd30d43947d4a54fc89ead7985beecfd3c9b2a93a0655a373b1608ab90bd5af mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume:v0.0.7] 22909487} {[quay.io/k8scsi/livenessprobe@sha256:8c9c8a0f330acc2e558ae6624309447d02522e76aa35af23fa19b88abf4381cb quay.io/k8scsi/livenessprobe:v1.0.2] 17415381} {[quay.io/k8scsi/csi-node-driver-registrar@sha256:13daf82fb99e951a4bff8ae5fc7c17c3a8fe7130be6400990d8f6076c32d4599 quay.io/k8scsi/csi-node-driver-registrar:v1.1.0] 15815995} {[mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume@sha256:23d8c6033f02a1ecad05127ebdc931bb871264228661bc122704b0974e4d9fdd mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume:1.0.8] 1159025} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 01:44:21.437] Nov 5 01:44:21.434: INFO: I1105 01:44:21.437] Logging kubelet events for node k8s-agentpool-15826033-vmss000000 I1105 01:44:21.516] Nov 5 01:44:21.516: INFO: I1105 01:44:21.516] Logging pods the kubelet thinks is on node k8s-agentpool-15826033-vmss000000 I1105 01:44:21.628] Nov 5 01:44:21.628: INFO: azure-ip-masq-agent-vx86b started at 2019-11-05 01:34:34 +0000 UTC (0+1 container statuses recorded) I1105 01:44:21.629] Nov 5 01:44:21.628: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 01:44:21.629] Nov 5 01:44:21.628: INFO: keyvault-flexvolume-qhnfr started at 2019-11-05 01:34:34 +0000 UTC (0+1 container statuses recorded) I1105 01:44:21.629] Nov 5 01:44:21.628: INFO: Container keyvault-flexvolume ready: true, restart count 0 I1105 01:44:21.629] Nov 5 01:44:21.628: INFO: metrics-server-864ffbc5c-sdxps started at 2019-11-05 01:34:34 +0000 UTC (0+1 container statuses recorded) I1105 01:44:21.629] Nov 5 01:44:21.628: INFO: Container metrics-server ready: true, restart count 0 I1105 01:44:21.630] Nov 5 01:44:21.628: INFO: kube-proxy-wxblc started at 2019-11-05 01:34:34 +0000 UTC (0+1 container statuses recorded) I1105 01:44:21.630] Nov 5 01:44:21.628: INFO: Container kube-proxy ready: true, restart count 0 I1105 01:44:21.630] Nov 5 01:44:21.628: INFO: kubernetes-dashboard-66dd8b8df7-ks284 started at 2019-11-05 01:34:35 +0000 UTC (0+1 container statuses recorded) I1105 01:44:21.630] Nov 5 01:44:21.628: INFO: Container kubernetes-dashboard ready: true, restart count 0 I1105 01:44:21.630] Nov 5 01:44:21.628: INFO: csi-azuredisk-node-whq2g started at 2019-11-05 01:38:24 +0000 UTC (0+3 container statuses recorded) I1105 01:44:21.630] Nov 5 01:44:21.628: INFO: Container azuredisk ready: true, restart count 0 I1105 01:44:21.631] Nov 5 01:44:21.628: INFO: Container liveness-probe ready: true, restart count 0 I1105 01:44:21.631] Nov 5 01:44:21.628: INFO: Container node-driver-registrar ready: true, restart count 0 I1105 01:44:21.631] Nov 5 01:44:21.628: INFO: blobfuse-flexvol-installer-bsvqt started at 2019-11-05 01:34:34 +0000 UTC (0+1 container statuses recorded) I1105 01:44:21.631] Nov 5 01:44:21.628: INFO: Container blobfuse-flexvol-installer ready: true, restart count 0 I1105 01:44:21.881] Nov 5 01:44:21.880: INFO: I1105 01:44:21.881] Latency metrics for node k8s-agentpool-15826033-vmss000000 I1105 01:44:21.881] Nov 5 01:44:21.880: INFO: I1105 01:44:21.881] Logging node info for node k8s-agentpool-15826033-vmss000001 I1105 01:44:21.961] Nov 5 01:44:21.958: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-agentpool-15826033-vmss000001,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-agentpool-15826033-vmss000001,UID:4a8ea605-cc0b-4e3b-9060-d8867d269e6c,ResourceVersion:2740,Generation:0,CreationTimestamp:2019-11-05 01:34:09 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{agentpool: agentpool,beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_D4s_v3,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 1,kubernetes.azure.com/cluster: kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5,kubernetes.azure.com/role: agent,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-agentpool-15826033-vmss000001,kubernetes.io/os: linux,kubernetes.io/role: agent,node-role.kubernetes.io/agent: ,storageprofile: managed,storagetier: Premium_LRS,topology.disk.csi.azure.com/zone: 1,},Annotations:map[string]string{csi.volume.kubernetes.io/nodeid: {"disk.csi.azure.com":"k8s-agentpool-15826033-vmss000001"},node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.0.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool-15826033-vmss/virtualMachines/1,Unschedulable:false,Taints:[],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-csi-disk.csi.azure.com: {{8 0} {} 8 DecimalSI},cpu: {{4 0} {} 4 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{16797044736 0} {} 16403364Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-csi-disk.csi.azure.com: {{8 0} {} 8 DecimalSI},cpu: {{4 0} {} 4 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{16010612736 0} {} 15635364Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 01:34:57 +0000 UTC 2019-11-05 01:34:57 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 01:43:40 +0000 UTC 2019-11-05 01:34:08 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 01:43:40 +0000 UTC 2019-11-05 01:34:08 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 01:43:40 +0000 UTC 2019-11-05 01:34:08 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 01:43:40 +0000 UTC 2019-11-05 01:34:29 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.0.5} {Hostname k8s-agentpool-15826033-vmss000001}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:b6b9fbf4a2fd4fa5a7ae472415fe30b8,SystemUUID:0162B3A4-3955-9A46-ABD4-A37BD75848CC,BootID:ee93003f-d47c-4118-9975-57365b3443a5,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[k8sprowinternal.azurecr.io/azuredisk-csi@sha256:75d4ff5d7cd845d6e810c67796ecb037e852fb849ec1eb5b3194564008e6d1e0 k8sprowinternal.azurecr.io/azuredisk-csi:e2e-3e3ed045c6faa509dbd8d2890ee9f56aa2934887] 286930414} {[gcr.io/kubernetes-helm/tiller@sha256:f6d8f4ab9ba993b5f5b60a6edafe86352eabe474ffeb84cb6c79b8866dce45d1 gcr.io/kubernetes-helm/tiller:v2.11.0] 71821984} {[quay.io/k8scsi/csi-provisioner@sha256:d657c839dce87324fe2b677302913f9386f885f8746be7bea0ced5b0844e3433 quay.io/k8scsi/csi-provisioner:v1.3.1] 50596998} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[quay.io/k8scsi/csi-attacher@sha256:26fccd7a99d973845df1193b46ebdcc6ab8dc5f6e6be319750c471fce1742d13 quay.io/k8scsi/csi-attacher:v1.2.0] 46226754} {[quay.io/k8scsi/csi-cluster-driver-registrar@sha256:fafd75ae5442f192cfa8c2e792903aee30d5884b62e802e4464b0a895d21e3ef quay.io/k8scsi/csi-cluster-driver-registrar:v1.0.1] 45874691} {[quay.io/k8scsi/csi-snapshotter@sha256:a49e0da1af6f2bf717e41ba1eee8b5e6a1cbd66a709dd92cc43fe475fe2589eb quay.io/k8scsi/csi-snapshotter:v1.1.0] 44325352} {[k8s.gcr.io/coredns@sha256:e83beb5e43f8513fa735e77ffc5859640baea30a882a11cc75c4c3244a737d3c k8s.gcr.io/coredns:1.5.0] 42488424} {[mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume@sha256:4fd30d43947d4a54fc89ead7985beecfd3c9b2a93a0655a373b1608ab90bd5af mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume:v0.0.7] 22909487} {[quay.io/k8scsi/livenessprobe@sha256:8c9c8a0f330acc2e558ae6624309447d02522e76aa35af23fa19b88abf4381cb quay.io/k8scsi/livenessprobe:v1.0.2] 17415381} {[quay.io/k8scsi/csi-node-driver-registrar@sha256:13daf82fb99e951a4bff8ae5fc7c17c3a8fe7130be6400990d8f6076c32d4599 quay.io/k8scsi/csi-node-driver-registrar:v1.1.0] 15815995} {[quay.io/k8scsi/livenessprobe@sha256:dde617756e0f602adc566ab71fd885f1dad451ad3fb063ac991c95a2ff47aea5 quay.io/k8scsi/livenessprobe:v1.1.0] 14967303} {[mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume@sha256:23d8c6033f02a1ecad05127ebdc931bb871264228661bc122704b0974e4d9fdd mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume:1.0.8] 1159025} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 01:44:21.962] Nov 5 01:44:21.958: INFO: I1105 01:44:21.962] Logging kubelet events for node k8s-agentpool-15826033-vmss000001 I1105 01:44:22.040] Nov 5 01:44:22.040: INFO: I1105 01:44:22.040] Logging pods the kubelet thinks is on node k8s-agentpool-15826033-vmss000001 I1105 01:44:22.147] Nov 5 01:44:22.147: INFO: csi-azuredisk-node-ghv5w started at 2019-11-05 01:38:24 +0000 UTC (0+3 container statuses recorded) I1105 01:44:22.147] Nov 5 01:44:22.147: INFO: Container azuredisk ready: true, restart count 0 I1105 01:44:22.148] Nov 5 01:44:22.147: INFO: Container liveness-probe ready: true, restart count 0 I1105 01:44:22.148] Nov 5 01:44:22.147: INFO: Container node-driver-registrar ready: true, restart count 0 I1105 01:44:22.148] Nov 5 01:44:22.147: INFO: csi-azuredisk-controller-6f6587bfc6-8vkfl started at 2019-11-05 01:38:24 +0000 UTC (0+6 container statuses recorded) I1105 01:44:22.148] Nov 5 01:44:22.147: INFO: Container azuredisk ready: true, restart count 0 I1105 01:44:22.149] Nov 5 01:44:22.147: INFO: Container cluster-driver-registrar ready: true, restart count 0 I1105 01:44:22.149] Nov 5 01:44:22.147: INFO: Container csi-attacher ready: true, restart count 0 I1105 01:44:22.149] Nov 5 01:44:22.147: INFO: Container csi-provisioner ready: true, restart count 0 I1105 01:44:22.149] Nov 5 01:44:22.147: INFO: Container csi-snapshotter ready: true, restart count 0 I1105 01:44:22.149] Nov 5 01:44:22.147: INFO: Container liveness-probe ready: true, restart count 0 I1105 01:44:22.150] Nov 5 01:44:22.147: INFO: blobfuse-flexvol-installer-rw5dw started at 2019-11-05 01:34:35 +0000 UTC (0+1 container statuses recorded) I1105 01:44:22.150] Nov 5 01:44:22.147: INFO: Container blobfuse-flexvol-installer ready: true, restart count 0 I1105 01:44:22.150] Nov 5 01:44:22.147: INFO: coredns-7f68dcdbdb-jt522 started at 2019-11-05 01:34:35 +0000 UTC (0+1 container statuses recorded) I1105 01:44:22.150] Nov 5 01:44:22.147: INFO: Container coredns ready: true, restart count 0 I1105 01:44:22.151] Nov 5 01:44:22.147: INFO: azure-ip-masq-agent-6c6vn started at 2019-11-05 01:34:35 +0000 UTC (0+1 container statuses recorded) I1105 01:44:22.151] Nov 5 01:44:22.147: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 01:44:22.151] Nov 5 01:44:22.147: INFO: keyvault-flexvolume-rdlcj started at 2019-11-05 01:34:35 +0000 UTC (0+1 container statuses recorded) I1105 01:44:22.151] Nov 5 01:44:22.147: INFO: Container keyvault-flexvolume ready: true, restart count 0 I1105 01:44:22.151] Nov 5 01:44:22.147: INFO: kube-proxy-6kh57 started at 2019-11-05 01:34:35 +0000 UTC (0+1 container statuses recorded) I1105 01:44:22.152] Nov 5 01:44:22.147: INFO: Container kube-proxy ready: true, restart count 0 I1105 01:44:22.152] Nov 5 01:44:22.147: INFO: tiller-deploy-54c96cb5df-8d9fs started at 2019-11-05 01:34:35 +0000 UTC (0+1 container statuses recorded) I1105 01:44:22.152] Nov 5 01:44:22.147: INFO: Container tiller ready: true, restart count 0 I1105 01:44:22.405] Nov 5 01:44:22.404: INFO: I1105 01:44:22.405] Latency metrics for node k8s-agentpool-15826033-vmss000001 I1105 01:44:22.405] Nov 5 01:44:22.404: INFO: I1105 01:44:22.405] Logging node info for node k8s-master-15826033-0 I1105 01:44:22.484] Nov 5 01:44:22.482: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-15826033-0,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-15826033-0,UID:aceb0231-4645-4a9a-b296-90ada851986b,ResourceVersion:2697,Generation:0,CreationTimestamp:2019-11-05 01:35:06 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 1,kubernetes.azure.com/cluster: kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-15826033-0,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.4.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5/providers/Microsoft.Compute/virtualMachines/k8s-master-15826033-0,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 01:35:20 +0000 UTC 2019-11-05 01:35:20 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 01:43:28 +0000 UTC 2019-11-05 01:34:57 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 01:43:28 +0000 UTC 2019-11-05 01:34:57 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 01:43:28 +0000 UTC 2019-11-05 01:34:57 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 01:43:28 +0000 UTC 2019-11-05 01:35:07 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.255.5} {Hostname k8s-master-15826033-0}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:1915756c22a540dc8eb30493987e9a43,SystemUUID:742C969C-5A71-D744-8E61-F27E38B56EB2,BootID:03f0672e-8fa9-4309-956f-8e3c371902c3,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 01:44:22.485] Nov 5 01:44:22.482: INFO: I1105 01:44:22.485] Logging kubelet events for node k8s-master-15826033-0 I1105 01:44:22.565] Nov 5 01:44:22.565: INFO: I1105 01:44:22.565] Logging pods the kubelet thinks is on node k8s-master-15826033-0 I1105 01:44:22.672] Nov 5 01:44:22.671: INFO: cloud-controller-manager-k8s-master-15826033-0 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 01:44:22.672] Nov 5 01:44:22.672: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 01:44:22.672] Nov 5 01:44:22.672: INFO: kube-addon-manager-k8s-master-15826033-0 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 01:44:22.673] Nov 5 01:44:22.672: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 01:44:22.673] Nov 5 01:44:22.672: INFO: kube-apiserver-k8s-master-15826033-0 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 01:44:22.673] Nov 5 01:44:22.672: INFO: Container kube-apiserver ready: true, restart count 0 I1105 01:44:22.673] Nov 5 01:44:22.672: INFO: kube-controller-manager-k8s-master-15826033-0 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 01:44:22.674] Nov 5 01:44:22.672: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 01:44:22.674] Nov 5 01:44:22.672: INFO: kube-scheduler-k8s-master-15826033-0 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 01:44:22.674] Nov 5 01:44:22.672: INFO: Container kube-scheduler ready: true, restart count 0 I1105 01:44:22.674] Nov 5 01:44:22.672: INFO: azure-ip-masq-agent-wgdpx started at 2019-11-05 01:35:06 +0000 UTC (0+1 container statuses recorded) I1105 01:44:22.675] Nov 5 01:44:22.672: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 01:44:22.675] Nov 5 01:44:22.672: INFO: kube-proxy-7d2px started at 2019-11-05 01:35:06 +0000 UTC (0+1 container statuses recorded) I1105 01:44:22.675] Nov 5 01:44:22.672: INFO: Container kube-proxy ready: true, restart count 0 I1105 01:44:22.924] Nov 5 01:44:22.924: INFO: I1105 01:44:22.924] Latency metrics for node k8s-master-15826033-0 I1105 01:44:22.924] Nov 5 01:44:22.924: INFO: I1105 01:44:22.925] Logging node info for node k8s-master-15826033-1 I1105 01:44:23.003] Nov 5 01:44:23.001: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-15826033-1,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-15826033-1,UID:1a13ec61-c89a-456c-8c94-bfc8e717c2e7,ResourceVersion:2702,Generation:0,CreationTimestamp:2019-11-05 01:35:07 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 1,kubernetes.azure.com/cluster: kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-15826033-1,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.5.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5/providers/Microsoft.Compute/virtualMachines/k8s-master-15826033-1,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 01:35:31 +0000 UTC 2019-11-05 01:35:31 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 01:43:28 +0000 UTC 2019-11-05 01:35:00 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 01:43:28 +0000 UTC 2019-11-05 01:35:00 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 01:43:28 +0000 UTC 2019-11-05 01:35:00 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 01:43:28 +0000 UTC 2019-11-05 01:35:07 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.255.6} {Hostname k8s-master-15826033-1}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:e0f2bccddd5b4555b180d14081ab59a8,SystemUUID:AE126631-CAD2-B54F-B22E-1F499342A4D6,BootID:509b9a9d-b046-4861-aa35-8bcb77465de2,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 01:44:23.004] Nov 5 01:44:23.001: INFO: I1105 01:44:23.004] Logging kubelet events for node k8s-master-15826033-1 I1105 01:44:23.082] Nov 5 01:44:23.082: INFO: I1105 01:44:23.082] Logging pods the kubelet thinks is on node k8s-master-15826033-1 I1105 01:44:23.184] Nov 5 01:44:23.184: INFO: kube-scheduler-k8s-master-15826033-1 started at 2019-11-05 01:36:16 +0000 UTC (0+1 container statuses recorded) I1105 01:44:23.184] Nov 5 01:44:23.184: INFO: Container kube-scheduler ready: true, restart count 0 I1105 01:44:23.185] Nov 5 01:44:23.184: INFO: azure-ip-masq-agent-w6dpr started at 2019-11-05 01:35:10 +0000 UTC (0+1 container statuses recorded) I1105 01:44:23.185] Nov 5 01:44:23.184: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 01:44:23.185] Nov 5 01:44:23.184: INFO: kube-proxy-8kmr5 started at 2019-11-05 01:35:10 +0000 UTC (0+1 container statuses recorded) I1105 01:44:23.185] Nov 5 01:44:23.184: INFO: Container kube-proxy ready: true, restart count 0 I1105 01:44:23.185] Nov 5 01:44:23.184: INFO: cloud-controller-manager-k8s-master-15826033-1 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 01:44:23.186] Nov 5 01:44:23.184: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 01:44:23.186] Nov 5 01:44:23.184: INFO: kube-addon-manager-k8s-master-15826033-1 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 01:44:23.186] Nov 5 01:44:23.184: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 01:44:23.186] Nov 5 01:44:23.184: INFO: kube-apiserver-k8s-master-15826033-1 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 01:44:23.186] Nov 5 01:44:23.184: INFO: Container kube-apiserver ready: true, restart count 0 I1105 01:44:23.187] Nov 5 01:44:23.184: INFO: kube-controller-manager-k8s-master-15826033-1 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 01:44:23.187] Nov 5 01:44:23.184: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 01:44:23.435] Nov 5 01:44:23.435: INFO: I1105 01:44:23.435] Latency metrics for node k8s-master-15826033-1 I1105 01:44:23.435] Nov 5 01:44:23.435: INFO: I1105 01:44:23.436] Logging node info for node k8s-master-15826033-2 I1105 01:44:23.515] Nov 5 01:44:23.512: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-15826033-2,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-15826033-2,UID:acb3a851-d6df-4549-ba56-4b7e96553c17,ResourceVersion:2797,Generation:0,CreationTimestamp:2019-11-05 01:34:09 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 2,kubernetes.azure.com/cluster: kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-15826033-2,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.1.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5/providers/Microsoft.Compute/virtualMachines/k8s-master-15826033-2,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 01:34:44 +0000 UTC 2019-11-05 01:34:44 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 01:44:00 +0000 UTC 2019-11-05 01:34:07 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 01:44:00 +0000 UTC 2019-11-05 01:34:07 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 01:44:00 +0000 UTC 2019-11-05 01:34:07 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 01:44:00 +0000 UTC 2019-11-05 01:34:29 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{Hostname k8s-master-15826033-2} {InternalIP 10.240.255.7}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:0885b0c2f6174106bde4fa0eae2cbc6b,SystemUUID:C176121C-5D7A-6243-9B05-8B68F8022B64,BootID:30430638-7ab6-4761-b984-2387e1a3f5b2,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 01:44:23.515] Nov 5 01:44:23.512: INFO: I1105 01:44:23.515] Logging kubelet events for node k8s-master-15826033-2 I1105 01:44:23.593] Nov 5 01:44:23.593: INFO: I1105 01:44:23.593] Logging pods the kubelet thinks is on node k8s-master-15826033-2 I1105 01:44:23.698] Nov 5 01:44:23.697: INFO: kube-apiserver-k8s-master-15826033-2 started at 2019-11-05 01:33:54 +0000 UTC (0+1 container statuses recorded) I1105 01:44:23.698] Nov 5 01:44:23.697: INFO: Container kube-apiserver ready: true, restart count 0 I1105 01:44:23.698] Nov 5 01:44:23.697: INFO: kube-controller-manager-k8s-master-15826033-2 started at 2019-11-05 01:33:54 +0000 UTC (0+1 container statuses recorded) I1105 01:44:23.698] Nov 5 01:44:23.697: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 01:44:23.699] Nov 5 01:44:23.697: INFO: kube-scheduler-k8s-master-15826033-2 started at 2019-11-05 01:33:54 +0000 UTC (0+1 container statuses recorded) I1105 01:44:23.699] Nov 5 01:44:23.697: INFO: Container kube-scheduler ready: true, restart count 0 I1105 01:44:23.699] Nov 5 01:44:23.697: INFO: azure-ip-masq-agent-hw7s5 started at 2019-11-05 01:34:34 +0000 UTC (0+1 container statuses recorded) I1105 01:44:23.699] Nov 5 01:44:23.697: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 01:44:23.699] Nov 5 01:44:23.697: INFO: kube-proxy-5xpp6 started at 2019-11-05 01:34:34 +0000 UTC (0+1 container statuses recorded) I1105 01:44:23.699] Nov 5 01:44:23.697: INFO: Container kube-proxy ready: true, restart count 0 I1105 01:44:23.700] Nov 5 01:44:23.697: INFO: cloud-controller-manager-k8s-master-15826033-2 started at 2019-11-05 01:33:54 +0000 UTC (0+1 container statuses recorded) I1105 01:44:23.700] Nov 5 01:44:23.697: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 01:44:23.700] Nov 5 01:44:23.697: INFO: kube-addon-manager-k8s-master-15826033-2 started at 2019-11-05 01:33:54 +0000 UTC (0+1 container statuses recorded) I1105 01:44:23.700] Nov 5 01:44:23.697: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 01:44:23.946] Nov 5 01:44:23.945: INFO: I1105 01:44:23.946] Latency metrics for node k8s-master-15826033-2 I1105 01:44:23.946] Nov 5 01:44:23.945: INFO: I1105 01:44:23.946] Logging node info for node k8s-master-15826033-3 I1105 01:44:24.025] Nov 5 01:44:24.022: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-15826033-3,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-15826033-3,UID:b7702272-1876-434e-b6a0-17fa34c6757f,ResourceVersion:2695,Generation:0,CreationTimestamp:2019-11-05 01:35:06 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 0,kubernetes.azure.com/cluster: kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-15826033-3,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.3.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5/providers/Microsoft.Compute/virtualMachines/k8s-master-15826033-3,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 01:35:17 +0000 UTC 2019-11-05 01:35:17 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 01:43:27 +0000 UTC 2019-11-05 01:35:00 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 01:43:27 +0000 UTC 2019-11-05 01:35:00 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 01:43:27 +0000 UTC 2019-11-05 01:35:00 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 01:43:27 +0000 UTC 2019-11-05 01:35:06 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.255.8} {Hostname k8s-master-15826033-3}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:fbf366963961422c9d90bedf6639c8e1,SystemUUID:5388948D-1A8D-EA41-B8B8-7F4523EDF5AA,BootID:42f5884c-7418-486a-8cf3-75188a331f38,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 01:44:24.025] Nov 5 01:44:24.022: INFO: I1105 01:44:24.025] Logging kubelet events for node k8s-master-15826033-3 I1105 01:44:24.121] Nov 5 01:44:24.120: INFO: I1105 01:44:24.121] Logging pods the kubelet thinks is on node k8s-master-15826033-3 I1105 01:44:24.226] Nov 5 01:44:24.226: INFO: cloud-controller-manager-k8s-master-15826033-3 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 01:44:24.226] Nov 5 01:44:24.226: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 01:44:24.226] Nov 5 01:44:24.226: INFO: kube-addon-manager-k8s-master-15826033-3 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 01:44:24.226] Nov 5 01:44:24.226: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 01:44:24.227] Nov 5 01:44:24.226: INFO: kube-apiserver-k8s-master-15826033-3 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 01:44:24.227] Nov 5 01:44:24.226: INFO: Container kube-apiserver ready: true, restart count 0 I1105 01:44:24.227] Nov 5 01:44:24.226: INFO: kube-controller-manager-k8s-master-15826033-3 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 01:44:24.227] Nov 5 01:44:24.226: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 01:44:24.227] Nov 5 01:44:24.226: INFO: kube-scheduler-k8s-master-15826033-3 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 01:44:24.228] Nov 5 01:44:24.226: INFO: Container kube-scheduler ready: true, restart count 0 I1105 01:44:24.228] Nov 5 01:44:24.226: INFO: kube-proxy-xz24z started at 2019-11-05 01:35:06 +0000 UTC (0+1 container statuses recorded) I1105 01:44:24.228] Nov 5 01:44:24.226: INFO: Container kube-proxy ready: true, restart count 0 I1105 01:44:24.228] Nov 5 01:44:24.226: INFO: azure-ip-masq-agent-dqfpw started at 2019-11-05 01:35:06 +0000 UTC (0+1 container statuses recorded) I1105 01:44:24.228] Nov 5 01:44:24.226: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 01:44:24.482] Nov 5 01:44:24.482: INFO: I1105 01:44:24.482] Latency metrics for node k8s-master-15826033-3 I1105 01:44:24.482] Nov 5 01:44:24.482: INFO: I1105 01:44:24.482] Logging node info for node k8s-master-15826033-4 I1105 01:44:24.564] Nov 5 01:44:24.561: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-15826033-4,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-15826033-4,UID:276daef1-6c05-41e3-bacd-abc93d35cc6d,ResourceVersion:2706,Generation:0,CreationTimestamp:2019-11-05 01:35:07 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 0,kubernetes.azure.com/cluster: kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-15826033-4,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.6.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5/providers/Microsoft.Compute/virtualMachines/k8s-master-15826033-4,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 01:35:41 +0000 UTC 2019-11-05 01:35:41 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 01:43:29 +0000 UTC 2019-11-05 01:35:02 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 01:43:29 +0000 UTC 2019-11-05 01:35:02 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 01:43:29 +0000 UTC 2019-11-05 01:35:02 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 01:43:29 +0000 UTC 2019-11-05 01:35:08 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.255.9} {Hostname k8s-master-15826033-4}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:79051fc0bdeb4e32a56acbaf5972ce3a,SystemUUID:C71606B5-7F08-9341-A26D-8A8D9D84FCAE,BootID:4bdf7cf3-2b83-42c7-a7ba-926ebe9ca75b,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 01:44:24.564] Nov 5 01:44:24.561: INFO: I1105 01:44:24.565] Logging kubelet events for node k8s-master-15826033-4 I1105 01:44:24.642] Nov 5 01:44:24.642: INFO: I1105 01:44:24.642] Logging pods the kubelet thinks is on node k8s-master-15826033-4 I1105 01:44:24.749] Nov 5 01:44:24.749: INFO: azure-ip-masq-agent-9nrj8 started at 2019-11-05 01:35:11 +0000 UTC (0+1 container statuses recorded) I1105 01:44:24.750] Nov 5 01:44:24.749: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 01:44:24.750] Nov 5 01:44:24.749: INFO: kube-proxy-rmngk started at 2019-11-05 01:35:11 +0000 UTC (0+1 container statuses recorded) I1105 01:44:24.750] Nov 5 01:44:24.749: INFO: Container kube-proxy ready: true, restart count 0 I1105 01:44:24.750] Nov 5 01:44:24.749: INFO: kube-apiserver-k8s-master-15826033-4 started at 2019-11-05 01:34:55 +0000 UTC (0+1 container statuses recorded) I1105 01:44:24.751] Nov 5 01:44:24.749: INFO: Container kube-apiserver ready: true, restart count 0 I1105 01:44:24.751] Nov 5 01:44:24.749: INFO: kube-controller-manager-k8s-master-15826033-4 started at 2019-11-05 01:34:55 +0000 UTC (0+1 container statuses recorded) I1105 01:44:24.751] Nov 5 01:44:24.749: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 01:44:24.751] Nov 5 01:44:24.749: INFO: kube-scheduler-k8s-master-15826033-4 started at 2019-11-05 01:34:55 +0000 UTC (0+1 container statuses recorded) I1105 01:44:24.752] Nov 5 01:44:24.749: INFO: Container kube-scheduler ready: true, restart count 0 I1105 01:44:24.752] Nov 5 01:44:24.749: INFO: cloud-controller-manager-k8s-master-15826033-4 started at 2019-11-05 01:34:56 +0000 UTC (0+1 container statuses recorded) I1105 01:44:24.752] Nov 5 01:44:24.749: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 01:44:24.752] Nov 5 01:44:24.749: INFO: kube-addon-manager-k8s-master-15826033-4 started at 2019-11-05 01:34:56 +0000 UTC (0+1 container statuses recorded) I1105 01:44:24.753] Nov 5 01:44:24.749: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 01:44:24.998] Nov 5 01:44:24.998: INFO: I1105 01:44:24.998] Latency metrics for node k8s-master-15826033-4 I1105 01:44:24.999] Nov 5 01:44:24.998: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready I1105 01:44:25.077] STEP: Destroying namespace "azuredisk-3540" for this suite. I1105 01:44:39.324] Nov 5 01:44:39.324: INFO: Waiting up to 30s for server preferred namespaced resources to be successfully discovered I1105 01:44:42.314] Nov 5 01:44:42.314: INFO: namespace azuredisk-3540 deletion completed in 17.237396931s I1105 01:44:42.314] I1105 01:44:42.314] • Failure [324.355 seconds] I1105 01:44:42.314] [azuredisk-csi-e2e] Dynamic Provisioning I1105 01:44:42.315] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:33 I1105 01:44:42.315] [single-az] I1105 01:44:42.315] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:36 I1105 01:44:42.315] should create a volume on demand and mount it as readOnly in a pod [It] I1105 01:44:42.315] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:126 I1105 01:44:42.315] I1105 01:44:42.316] Unexpected error: I1105 01:44:42.316] <*errors.errorString | 0xc00075a320>: { I1105 01:44:42.316] s: "PersistentVolumeClaims [pvc-lvm42] not all in phase Bound within 5m0s", I1105 01:44:42.316] } I1105 01:44:42.316] PersistentVolumeClaims [pvc-lvm42] not all in phase Bound within 5m0s I1105 01:44:42.317] occurred I1105 01:44:42.317] I1105 01:44:42.317] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/testsuites/testsuites.go:278 I1105 01:44:42.317] ------------------------------ I1105 01:44:42.317] [azuredisk-csi-e2e] [single-az] Pre-Provisioned I1105 01:44:42.317] [env] should use a pre-provisioned volume and retain PV with reclaimPolicy "Retain" I1105 01:44:42.318] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/pre_provisioning.go:100 I1105 01:44:42.318] [BeforeEach] [azuredisk-csi-e2e] [single-az] Pre-Provisioned I1105 01:44:42.318] /go/src/sigs.k8s.io/azuredisk-csi-driver/vendor/k8s.io/kubernetes/test/e2e/framework/framework.go:150 I1105 01:44:42.318] STEP: Creating a kubernetes client I1105 01:44:42.318] Nov 5 01:44:42.314: INFO: >>> kubeConfig: /workspace/aks978705698/kubeconfig/kubeconfig.eastus2.json I1105 01:44:42.319] STEP: Building a namespace api object, basename azuredisk I1105 01:44:42.550] STEP: Binding the e2e-test-privileged-psp PodSecurityPolicy to the default service account in azuredisk-7528 I1105 01:44:42.809] STEP: Waiting for a default service account to be provisioned in namespace I1105 01:44:42.884] [BeforeEach] [azuredisk-csi-e2e] [single-az] Pre-Provisioned I1105 01:44:42.885] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/pre_provisioning.go:48 I1105 01:44:42.885] [It] [env] should use a pre-provisioned volume and retain PV with reclaimPolicy "Retain" I1105 01:44:42.886] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/pre_provisioning.go:100 I1105 01:44:45.294] STEP: Successfully provisioned AzureDisk volume: "/subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5/providers/Microsoft.Compute/disks/pre-provisioned-retain-reclaimPolicy" I1105 01:44:45.295] I1105 01:44:45.295] STEP: setting up the PV I1105 01:44:45.295] STEP: creating a PV I1105 01:44:45.377] STEP: setting up the PVC I1105 01:44:45.378] STEP: creating a PVC I1105 01:44:45.467] STEP: waiting for PVC to be in phase "Bound" I1105 01:44:45.467] Nov 5 01:44:45.466: INFO: Waiting up to 5m0s for PersistentVolumeClaims [pvc-x5b9j] to have phase Bound I1105 01:44:45.544] Nov 5 01:44:45.544: INFO: PersistentVolumeClaim pvc-x5b9j found and phase=Bound (77.264403ms) I1105 01:44:45.544] STEP: checking the PVC I1105 01:44:45.621] STEP: validating provisioned PV I1105 01:44:45.698] STEP: checking the PV I1105 01:44:45.699] [AfterEach] [azuredisk-csi-e2e] [single-az] Pre-Provisioned I1105 01:44:45.699] /go/src/sigs.k8s.io/azuredisk-csi-driver/vendor/k8s.io/kubernetes/test/e2e/framework/framework.go:151 I1105 01:44:45.700] STEP: Collecting events from namespace "azuredisk-7528". I1105 01:44:45.776] STEP: Found 0 events. I1105 01:44:45.854] Nov 5 01:44:45.853: INFO: POD NODE PHASE GRACE CONDITIONS I1105 01:44:45.854] Nov 5 01:44:45.853: INFO: I1105 01:44:46.085] Nov 5 01:44:46.085: INFO: I1105 01:44:46.085] Logging node info for node k8s-agentpool-15826033-vmss000000 I1105 01:44:46.166] Nov 5 01:44:46.162: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-agentpool-15826033-vmss000000,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-agentpool-15826033-vmss000000,UID:15fa874e-54ec-42cd-b243-d3baa512150f,ResourceVersion:2765,Generation:0,CreationTimestamp:2019-11-05 01:34:17 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{agentpool: agentpool,beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_D4s_v3,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 0,kubernetes.azure.com/cluster: kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5,kubernetes.azure.com/role: agent,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-agentpool-15826033-vmss000000,kubernetes.io/os: linux,kubernetes.io/role: agent,node-role.kubernetes.io/agent: ,storageprofile: managed,storagetier: Premium_LRS,topology.disk.csi.azure.com/zone: 0,},Annotations:map[string]string{csi.volume.kubernetes.io/nodeid: {"disk.csi.azure.com":"k8s-agentpool-15826033-vmss000000"},node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.2.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool-15826033-vmss/virtualMachines/0,Unschedulable:false,Taints:[],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-csi-disk.csi.azure.com: {{8 0} {} 8 DecimalSI},cpu: {{4 0} {} 4 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{16797044736 0} {} 16403364Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-csi-disk.csi.azure.com: {{8 0} {} 8 DecimalSI},cpu: {{4 0} {} 4 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{16010612736 0} {} 15635364Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 01:34:56 +0000 UTC 2019-11-05 01:34:56 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 01:43:49 +0000 UTC 2019-11-05 01:34:17 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 01:43:49 +0000 UTC 2019-11-05 01:34:17 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 01:43:49 +0000 UTC 2019-11-05 01:34:17 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 01:43:49 +0000 UTC 2019-11-05 01:34:29 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.0.4} {Hostname k8s-agentpool-15826033-vmss000000}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:afe00792177d49a98f6a44d03d08732d,SystemUUID:4F1777BF-A5C3-004E-81C4-90F0B54C02A0,BootID:13b6a628-ae89-44af-b891-89461030adcb,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[k8sprowinternal.azurecr.io/azuredisk-csi@sha256:75d4ff5d7cd845d6e810c67796ecb037e852fb849ec1eb5b3194564008e6d1e0 k8sprowinternal.azurecr.io/azuredisk-csi:e2e-3e3ed045c6faa509dbd8d2890ee9f56aa2934887] 286930414} {[k8s.gcr.io/kubernetes-dashboard-amd64@sha256:0ae6b69432e78069c5ce2bcde0fe409c5c4d6f0f4d9cd50a17974fea38898747 k8s.gcr.io/kubernetes-dashboard-amd64:v1.10.1] 121711221} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/metrics-server-amd64@sha256:49a9f12f7067d11f42c803dbe61ed2c1299959ad85cb315b25ff7eef8e6b8892 k8s.gcr.io/metrics-server-amd64:v0.2.1] 42541759} {[mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume@sha256:4fd30d43947d4a54fc89ead7985beecfd3c9b2a93a0655a373b1608ab90bd5af mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume:v0.0.7] 22909487} {[quay.io/k8scsi/livenessprobe@sha256:8c9c8a0f330acc2e558ae6624309447d02522e76aa35af23fa19b88abf4381cb quay.io/k8scsi/livenessprobe:v1.0.2] 17415381} {[quay.io/k8scsi/csi-node-driver-registrar@sha256:13daf82fb99e951a4bff8ae5fc7c17c3a8fe7130be6400990d8f6076c32d4599 quay.io/k8scsi/csi-node-driver-registrar:v1.1.0] 15815995} {[mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume@sha256:23d8c6033f02a1ecad05127ebdc931bb871264228661bc122704b0974e4d9fdd mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume:1.0.8] 1159025} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 01:44:46.166] Nov 5 01:44:46.163: INFO: I1105 01:44:46.166] Logging kubelet events for node k8s-agentpool-15826033-vmss000000 I1105 01:44:46.243] Nov 5 01:44:46.243: INFO: I1105 01:44:46.244] Logging pods the kubelet thinks is on node k8s-agentpool-15826033-vmss000000 I1105 01:44:46.324] Nov 5 01:44:46.324: INFO: kubernetes-dashboard-66dd8b8df7-ks284 started at 2019-11-05 01:34:35 +0000 UTC (0+1 container statuses recorded) I1105 01:44:46.325] Nov 5 01:44:46.324: INFO: Container kubernetes-dashboard ready: true, restart count 0 I1105 01:44:46.325] Nov 5 01:44:46.324: INFO: csi-azuredisk-node-whq2g started at 2019-11-05 01:38:24 +0000 UTC (0+3 container statuses recorded) I1105 01:44:46.325] Nov 5 01:44:46.324: INFO: Container azuredisk ready: true, restart count 0 I1105 01:44:46.325] Nov 5 01:44:46.324: INFO: Container liveness-probe ready: true, restart count 0 I1105 01:44:46.325] Nov 5 01:44:46.324: INFO: Container node-driver-registrar ready: true, restart count 0 I1105 01:44:46.326] Nov 5 01:44:46.324: INFO: blobfuse-flexvol-installer-bsvqt started at 2019-11-05 01:34:34 +0000 UTC (0+1 container statuses recorded) I1105 01:44:46.326] Nov 5 01:44:46.324: INFO: Container blobfuse-flexvol-installer ready: true, restart count 0 I1105 01:44:46.326] Nov 5 01:44:46.324: INFO: azure-ip-masq-agent-vx86b started at 2019-11-05 01:34:34 +0000 UTC (0+1 container statuses recorded) I1105 01:44:46.326] Nov 5 01:44:46.324: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 01:44:46.326] Nov 5 01:44:46.324: INFO: keyvault-flexvolume-qhnfr started at 2019-11-05 01:34:34 +0000 UTC (0+1 container statuses recorded) I1105 01:44:46.327] Nov 5 01:44:46.324: INFO: Container keyvault-flexvolume ready: true, restart count 0 I1105 01:44:46.327] Nov 5 01:44:46.324: INFO: metrics-server-864ffbc5c-sdxps started at 2019-11-05 01:34:34 +0000 UTC (0+1 container statuses recorded) I1105 01:44:46.327] Nov 5 01:44:46.324: INFO: Container metrics-server ready: true, restart count 0 I1105 01:44:46.327] Nov 5 01:44:46.324: INFO: kube-proxy-wxblc started at 2019-11-05 01:34:34 +0000 UTC (0+1 container statuses recorded) I1105 01:44:46.327] Nov 5 01:44:46.324: INFO: Container kube-proxy ready: true, restart count 0 I1105 01:44:46.574] Nov 5 01:44:46.574: INFO: I1105 01:44:46.574] Latency metrics for node k8s-agentpool-15826033-vmss000000 I1105 01:44:46.575] Nov 5 01:44:46.574: INFO: I1105 01:44:46.575] Logging node info for node k8s-agentpool-15826033-vmss000001 I1105 01:44:46.655] Nov 5 01:44:46.652: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-agentpool-15826033-vmss000001,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-agentpool-15826033-vmss000001,UID:4a8ea605-cc0b-4e3b-9060-d8867d269e6c,ResourceVersion:2926,Generation:0,CreationTimestamp:2019-11-05 01:34:09 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{agentpool: agentpool,beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_D4s_v3,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 1,kubernetes.azure.com/cluster: kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5,kubernetes.azure.com/role: agent,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-agentpool-15826033-vmss000001,kubernetes.io/os: linux,kubernetes.io/role: agent,node-role.kubernetes.io/agent: ,storageprofile: managed,storagetier: Premium_LRS,topology.disk.csi.azure.com/zone: 1,},Annotations:map[string]string{csi.volume.kubernetes.io/nodeid: {"disk.csi.azure.com":"k8s-agentpool-15826033-vmss000001"},node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.0.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool-15826033-vmss/virtualMachines/1,Unschedulable:false,Taints:[],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-csi-disk.csi.azure.com: {{8 0} {} 8 DecimalSI},cpu: {{4 0} {} 4 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{16797044736 0} {} 16403364Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-csi-disk.csi.azure.com: {{8 0} {} 8 DecimalSI},cpu: {{4 0} {} 4 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{16010612736 0} {} 15635364Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 01:34:57 +0000 UTC 2019-11-05 01:34:57 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 01:44:40 +0000 UTC 2019-11-05 01:34:08 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 01:44:40 +0000 UTC 2019-11-05 01:34:08 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 01:44:40 +0000 UTC 2019-11-05 01:34:08 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 01:44:40 +0000 UTC 2019-11-05 01:34:29 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.0.5} {Hostname k8s-agentpool-15826033-vmss000001}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:b6b9fbf4a2fd4fa5a7ae472415fe30b8,SystemUUID:0162B3A4-3955-9A46-ABD4-A37BD75848CC,BootID:ee93003f-d47c-4118-9975-57365b3443a5,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[k8sprowinternal.azurecr.io/azuredisk-csi@sha256:75d4ff5d7cd845d6e810c67796ecb037e852fb849ec1eb5b3194564008e6d1e0 k8sprowinternal.azurecr.io/azuredisk-csi:e2e-3e3ed045c6faa509dbd8d2890ee9f56aa2934887] 286930414} {[gcr.io/kubernetes-helm/tiller@sha256:f6d8f4ab9ba993b5f5b60a6edafe86352eabe474ffeb84cb6c79b8866dce45d1 gcr.io/kubernetes-helm/tiller:v2.11.0] 71821984} {[quay.io/k8scsi/csi-provisioner@sha256:d657c839dce87324fe2b677302913f9386f885f8746be7bea0ced5b0844e3433 quay.io/k8scsi/csi-provisioner:v1.3.1] 50596998} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[quay.io/k8scsi/csi-attacher@sha256:26fccd7a99d973845df1193b46ebdcc6ab8dc5f6e6be319750c471fce1742d13 quay.io/k8scsi/csi-attacher:v1.2.0] 46226754} {[quay.io/k8scsi/csi-cluster-driver-registrar@sha256:fafd75ae5442f192cfa8c2e792903aee30d5884b62e802e4464b0a895d21e3ef quay.io/k8scsi/csi-cluster-driver-registrar:v1.0.1] 45874691} {[quay.io/k8scsi/csi-snapshotter@sha256:a49e0da1af6f2bf717e41ba1eee8b5e6a1cbd66a709dd92cc43fe475fe2589eb quay.io/k8scsi/csi-snapshotter:v1.1.0] 44325352} {[k8s.gcr.io/coredns@sha256:e83beb5e43f8513fa735e77ffc5859640baea30a882a11cc75c4c3244a737d3c k8s.gcr.io/coredns:1.5.0] 42488424} {[mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume@sha256:4fd30d43947d4a54fc89ead7985beecfd3c9b2a93a0655a373b1608ab90bd5af mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume:v0.0.7] 22909487} {[quay.io/k8scsi/livenessprobe@sha256:8c9c8a0f330acc2e558ae6624309447d02522e76aa35af23fa19b88abf4381cb quay.io/k8scsi/livenessprobe:v1.0.2] 17415381} {[quay.io/k8scsi/csi-node-driver-registrar@sha256:13daf82fb99e951a4bff8ae5fc7c17c3a8fe7130be6400990d8f6076c32d4599 quay.io/k8scsi/csi-node-driver-registrar:v1.1.0] 15815995} {[quay.io/k8scsi/livenessprobe@sha256:dde617756e0f602adc566ab71fd885f1dad451ad3fb063ac991c95a2ff47aea5 quay.io/k8scsi/livenessprobe:v1.1.0] 14967303} {[mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume@sha256:23d8c6033f02a1ecad05127ebdc931bb871264228661bc122704b0974e4d9fdd mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume:1.0.8] 1159025} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 01:44:46.655] Nov 5 01:44:46.652: INFO: I1105 01:44:46.656] Logging kubelet events for node k8s-agentpool-15826033-vmss000001 I1105 01:44:46.734] Nov 5 01:44:46.734: INFO: I1105 01:44:46.734] Logging pods the kubelet thinks is on node k8s-agentpool-15826033-vmss000001 I1105 01:44:46.817] Nov 5 01:44:46.817: INFO: tiller-deploy-54c96cb5df-8d9fs started at 2019-11-05 01:34:35 +0000 UTC (0+1 container statuses recorded) I1105 01:44:46.818] Nov 5 01:44:46.817: INFO: Container tiller ready: true, restart count 0 I1105 01:44:46.818] Nov 5 01:44:46.817: INFO: csi-azuredisk-node-ghv5w started at 2019-11-05 01:38:24 +0000 UTC (0+3 container statuses recorded) I1105 01:44:46.818] Nov 5 01:44:46.817: INFO: Container azuredisk ready: true, restart count 0 I1105 01:44:46.819] Nov 5 01:44:46.817: INFO: Container liveness-probe ready: true, restart count 0 I1105 01:44:46.819] Nov 5 01:44:46.817: INFO: Container node-driver-registrar ready: true, restart count 0 I1105 01:44:46.820] Nov 5 01:44:46.817: INFO: csi-azuredisk-controller-6f6587bfc6-8vkfl started at 2019-11-05 01:38:24 +0000 UTC (0+6 container statuses recorded) I1105 01:44:46.820] Nov 5 01:44:46.817: INFO: Container azuredisk ready: true, restart count 0 I1105 01:44:46.820] Nov 5 01:44:46.817: INFO: Container cluster-driver-registrar ready: true, restart count 0 I1105 01:44:46.820] Nov 5 01:44:46.817: INFO: Container csi-attacher ready: true, restart count 0 I1105 01:44:46.821] Nov 5 01:44:46.817: INFO: Container csi-provisioner ready: true, restart count 0 I1105 01:44:46.821] Nov 5 01:44:46.817: INFO: Container csi-snapshotter ready: true, restart count 0 I1105 01:44:46.821] Nov 5 01:44:46.817: INFO: Container liveness-probe ready: true, restart count 0 I1105 01:44:46.821] Nov 5 01:44:46.817: INFO: blobfuse-flexvol-installer-rw5dw started at 2019-11-05 01:34:35 +0000 UTC (0+1 container statuses recorded) I1105 01:44:46.822] Nov 5 01:44:46.817: INFO: Container blobfuse-flexvol-installer ready: true, restart count 0 I1105 01:44:46.822] Nov 5 01:44:46.817: INFO: coredns-7f68dcdbdb-jt522 started at 2019-11-05 01:34:35 +0000 UTC (0+1 container statuses recorded) I1105 01:44:46.822] Nov 5 01:44:46.817: INFO: Container coredns ready: true, restart count 0 I1105 01:44:46.822] Nov 5 01:44:46.817: INFO: azure-ip-masq-agent-6c6vn started at 2019-11-05 01:34:35 +0000 UTC (0+1 container statuses recorded) I1105 01:44:46.822] Nov 5 01:44:46.817: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 01:44:46.823] Nov 5 01:44:46.817: INFO: keyvault-flexvolume-rdlcj started at 2019-11-05 01:34:35 +0000 UTC (0+1 container statuses recorded) I1105 01:44:46.823] Nov 5 01:44:46.817: INFO: Container keyvault-flexvolume ready: true, restart count 0 I1105 01:44:46.823] Nov 5 01:44:46.817: INFO: kube-proxy-6kh57 started at 2019-11-05 01:34:35 +0000 UTC (0+1 container statuses recorded) I1105 01:44:46.823] Nov 5 01:44:46.817: INFO: Container kube-proxy ready: true, restart count 0 I1105 01:44:47.073] Nov 5 01:44:47.073: INFO: I1105 01:44:47.073] Latency metrics for node k8s-agentpool-15826033-vmss000001 I1105 01:44:47.074] Nov 5 01:44:47.073: INFO: I1105 01:44:47.074] Logging node info for node k8s-master-15826033-0 I1105 01:44:47.153] Nov 5 01:44:47.150: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-15826033-0,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-15826033-0,UID:aceb0231-4645-4a9a-b296-90ada851986b,ResourceVersion:2874,Generation:0,CreationTimestamp:2019-11-05 01:35:06 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 1,kubernetes.azure.com/cluster: kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-15826033-0,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.4.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5/providers/Microsoft.Compute/virtualMachines/k8s-master-15826033-0,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 01:35:20 +0000 UTC 2019-11-05 01:35:20 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 01:44:28 +0000 UTC 2019-11-05 01:34:57 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 01:44:28 +0000 UTC 2019-11-05 01:34:57 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 01:44:28 +0000 UTC 2019-11-05 01:34:57 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 01:44:28 +0000 UTC 2019-11-05 01:35:07 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.255.5} {Hostname k8s-master-15826033-0}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:1915756c22a540dc8eb30493987e9a43,SystemUUID:742C969C-5A71-D744-8E61-F27E38B56EB2,BootID:03f0672e-8fa9-4309-956f-8e3c371902c3,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 01:44:47.153] Nov 5 01:44:47.151: INFO: I1105 01:44:47.153] Logging kubelet events for node k8s-master-15826033-0 I1105 01:44:47.232] Nov 5 01:44:47.232: INFO: I1105 01:44:47.233] Logging pods the kubelet thinks is on node k8s-master-15826033-0 I1105 01:44:47.316] Nov 5 01:44:47.315: INFO: cloud-controller-manager-k8s-master-15826033-0 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 01:44:47.316] Nov 5 01:44:47.315: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 01:44:47.316] Nov 5 01:44:47.315: INFO: kube-addon-manager-k8s-master-15826033-0 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 01:44:47.316] Nov 5 01:44:47.316: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 01:44:47.317] Nov 5 01:44:47.316: INFO: kube-apiserver-k8s-master-15826033-0 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 01:44:47.317] Nov 5 01:44:47.316: INFO: Container kube-apiserver ready: true, restart count 0 I1105 01:44:47.317] Nov 5 01:44:47.316: INFO: kube-controller-manager-k8s-master-15826033-0 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 01:44:47.317] Nov 5 01:44:47.316: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 01:44:47.318] Nov 5 01:44:47.316: INFO: kube-scheduler-k8s-master-15826033-0 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 01:44:47.318] Nov 5 01:44:47.316: INFO: Container kube-scheduler ready: true, restart count 0 I1105 01:44:47.318] Nov 5 01:44:47.316: INFO: azure-ip-masq-agent-wgdpx started at 2019-11-05 01:35:06 +0000 UTC (0+1 container statuses recorded) I1105 01:44:47.318] Nov 5 01:44:47.316: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 01:44:47.318] Nov 5 01:44:47.316: INFO: kube-proxy-7d2px started at 2019-11-05 01:35:06 +0000 UTC (0+1 container statuses recorded) I1105 01:44:47.319] Nov 5 01:44:47.316: INFO: Container kube-proxy ready: true, restart count 0 I1105 01:44:47.559] Nov 5 01:44:47.558: INFO: I1105 01:44:47.559] Latency metrics for node k8s-master-15826033-0 I1105 01:44:47.559] Nov 5 01:44:47.558: INFO: I1105 01:44:47.559] Logging node info for node k8s-master-15826033-1 I1105 01:44:47.638] Nov 5 01:44:47.636: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-15826033-1,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-15826033-1,UID:1a13ec61-c89a-456c-8c94-bfc8e717c2e7,ResourceVersion:2879,Generation:0,CreationTimestamp:2019-11-05 01:35:07 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 1,kubernetes.azure.com/cluster: kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-15826033-1,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.5.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5/providers/Microsoft.Compute/virtualMachines/k8s-master-15826033-1,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 01:35:31 +0000 UTC 2019-11-05 01:35:31 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 01:44:28 +0000 UTC 2019-11-05 01:35:00 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 01:44:28 +0000 UTC 2019-11-05 01:35:00 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 01:44:28 +0000 UTC 2019-11-05 01:35:00 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 01:44:28 +0000 UTC 2019-11-05 01:35:07 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.255.6} {Hostname k8s-master-15826033-1}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:e0f2bccddd5b4555b180d14081ab59a8,SystemUUID:AE126631-CAD2-B54F-B22E-1F499342A4D6,BootID:509b9a9d-b046-4861-aa35-8bcb77465de2,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 01:44:47.638] Nov 5 01:44:47.636: INFO: I1105 01:44:47.639] Logging kubelet events for node k8s-master-15826033-1 I1105 01:44:47.725] Nov 5 01:44:47.725: INFO: I1105 01:44:47.726] Logging pods the kubelet thinks is on node k8s-master-15826033-1 I1105 01:44:47.805] Nov 5 01:44:47.805: INFO: kube-addon-manager-k8s-master-15826033-1 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 01:44:47.806] Nov 5 01:44:47.805: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 01:44:47.806] Nov 5 01:44:47.805: INFO: kube-apiserver-k8s-master-15826033-1 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 01:44:47.806] Nov 5 01:44:47.805: INFO: Container kube-apiserver ready: true, restart count 0 I1105 01:44:47.806] Nov 5 01:44:47.805: INFO: kube-controller-manager-k8s-master-15826033-1 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 01:44:47.806] Nov 5 01:44:47.805: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 01:44:47.807] Nov 5 01:44:47.805: INFO: kube-scheduler-k8s-master-15826033-1 started at 2019-11-05 01:36:16 +0000 UTC (0+1 container statuses recorded) I1105 01:44:47.807] Nov 5 01:44:47.805: INFO: Container kube-scheduler ready: true, restart count 0 I1105 01:44:47.807] Nov 5 01:44:47.805: INFO: azure-ip-masq-agent-w6dpr started at 2019-11-05 01:35:10 +0000 UTC (0+1 container statuses recorded) I1105 01:44:47.807] Nov 5 01:44:47.805: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 01:44:47.807] Nov 5 01:44:47.805: INFO: kube-proxy-8kmr5 started at 2019-11-05 01:35:10 +0000 UTC (0+1 container statuses recorded) I1105 01:44:47.808] Nov 5 01:44:47.805: INFO: Container kube-proxy ready: true, restart count 0 I1105 01:44:47.808] Nov 5 01:44:47.805: INFO: cloud-controller-manager-k8s-master-15826033-1 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 01:44:47.808] Nov 5 01:44:47.805: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 01:44:48.052] Nov 5 01:44:48.052: INFO: I1105 01:44:48.053] Latency metrics for node k8s-master-15826033-1 I1105 01:44:48.053] Nov 5 01:44:48.052: INFO: I1105 01:44:48.053] Logging node info for node k8s-master-15826033-2 I1105 01:44:48.132] Nov 5 01:44:48.130: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-15826033-2,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-15826033-2,UID:acb3a851-d6df-4549-ba56-4b7e96553c17,ResourceVersion:2797,Generation:0,CreationTimestamp:2019-11-05 01:34:09 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 2,kubernetes.azure.com/cluster: kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-15826033-2,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.1.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5/providers/Microsoft.Compute/virtualMachines/k8s-master-15826033-2,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 01:34:44 +0000 UTC 2019-11-05 01:34:44 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 01:44:00 +0000 UTC 2019-11-05 01:34:07 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 01:44:00 +0000 UTC 2019-11-05 01:34:07 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 01:44:00 +0000 UTC 2019-11-05 01:34:07 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 01:44:00 +0000 UTC 2019-11-05 01:34:29 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{Hostname k8s-master-15826033-2} {InternalIP 10.240.255.7}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:0885b0c2f6174106bde4fa0eae2cbc6b,SystemUUID:C176121C-5D7A-6243-9B05-8B68F8022B64,BootID:30430638-7ab6-4761-b984-2387e1a3f5b2,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 01:44:48.132] Nov 5 01:44:48.130: INFO: I1105 01:44:48.133] Logging kubelet events for node k8s-master-15826033-2 I1105 01:44:48.211] Nov 5 01:44:48.211: INFO: I1105 01:44:48.212] Logging pods the kubelet thinks is on node k8s-master-15826033-2 I1105 01:44:48.292] Nov 5 01:44:48.291: INFO: kube-apiserver-k8s-master-15826033-2 started at 2019-11-05 01:33:54 +0000 UTC (0+1 container statuses recorded) I1105 01:44:48.292] Nov 5 01:44:48.291: INFO: Container kube-apiserver ready: true, restart count 0 I1105 01:44:48.292] Nov 5 01:44:48.291: INFO: kube-controller-manager-k8s-master-15826033-2 started at 2019-11-05 01:33:54 +0000 UTC (0+1 container statuses recorded) I1105 01:44:48.292] Nov 5 01:44:48.291: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 01:44:48.293] Nov 5 01:44:48.291: INFO: kube-scheduler-k8s-master-15826033-2 started at 2019-11-05 01:33:54 +0000 UTC (0+1 container statuses recorded) I1105 01:44:48.293] Nov 5 01:44:48.291: INFO: Container kube-scheduler ready: true, restart count 0 I1105 01:44:48.293] Nov 5 01:44:48.291: INFO: azure-ip-masq-agent-hw7s5 started at 2019-11-05 01:34:34 +0000 UTC (0+1 container statuses recorded) I1105 01:44:48.293] Nov 5 01:44:48.291: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 01:44:48.294] Nov 5 01:44:48.291: INFO: kube-proxy-5xpp6 started at 2019-11-05 01:34:34 +0000 UTC (0+1 container statuses recorded) I1105 01:44:48.294] Nov 5 01:44:48.291: INFO: Container kube-proxy ready: true, restart count 0 I1105 01:44:48.294] Nov 5 01:44:48.291: INFO: cloud-controller-manager-k8s-master-15826033-2 started at 2019-11-05 01:33:54 +0000 UTC (0+1 container statuses recorded) I1105 01:44:48.294] Nov 5 01:44:48.292: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 01:44:48.294] Nov 5 01:44:48.292: INFO: kube-addon-manager-k8s-master-15826033-2 started at 2019-11-05 01:33:54 +0000 UTC (0+1 container statuses recorded) I1105 01:44:48.295] Nov 5 01:44:48.292: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 01:44:48.538] Nov 5 01:44:48.538: INFO: I1105 01:44:48.539] Latency metrics for node k8s-master-15826033-2 I1105 01:44:48.539] Nov 5 01:44:48.538: INFO: I1105 01:44:48.539] Logging node info for node k8s-master-15826033-3 I1105 01:44:48.618] Nov 5 01:44:48.615: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-15826033-3,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-15826033-3,UID:b7702272-1876-434e-b6a0-17fa34c6757f,ResourceVersion:2872,Generation:0,CreationTimestamp:2019-11-05 01:35:06 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 0,kubernetes.azure.com/cluster: kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-15826033-3,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.3.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5/providers/Microsoft.Compute/virtualMachines/k8s-master-15826033-3,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 01:35:17 +0000 UTC 2019-11-05 01:35:17 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 01:44:27 +0000 UTC 2019-11-05 01:35:00 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 01:44:27 +0000 UTC 2019-11-05 01:35:00 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 01:44:27 +0000 UTC 2019-11-05 01:35:00 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 01:44:27 +0000 UTC 2019-11-05 01:35:06 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.255.8} {Hostname k8s-master-15826033-3}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:fbf366963961422c9d90bedf6639c8e1,SystemUUID:5388948D-1A8D-EA41-B8B8-7F4523EDF5AA,BootID:42f5884c-7418-486a-8cf3-75188a331f38,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 01:44:48.618] Nov 5 01:44:48.615: INFO: I1105 01:44:48.618] Logging kubelet events for node k8s-master-15826033-3 I1105 01:44:48.702] Nov 5 01:44:48.702: INFO: I1105 01:44:48.702] Logging pods the kubelet thinks is on node k8s-master-15826033-3 I1105 01:44:48.785] Nov 5 01:44:48.785: INFO: kube-controller-manager-k8s-master-15826033-3 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 01:44:48.785] Nov 5 01:44:48.785: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 01:44:48.786] Nov 5 01:44:48.785: INFO: kube-scheduler-k8s-master-15826033-3 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 01:44:48.786] Nov 5 01:44:48.785: INFO: Container kube-scheduler ready: true, restart count 0 I1105 01:44:48.786] Nov 5 01:44:48.785: INFO: kube-proxy-xz24z started at 2019-11-05 01:35:06 +0000 UTC (0+1 container statuses recorded) I1105 01:44:48.786] Nov 5 01:44:48.785: INFO: Container kube-proxy ready: true, restart count 0 I1105 01:44:48.786] Nov 5 01:44:48.785: INFO: azure-ip-masq-agent-dqfpw started at 2019-11-05 01:35:06 +0000 UTC (0+1 container statuses recorded) I1105 01:44:48.787] Nov 5 01:44:48.785: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 01:44:48.787] Nov 5 01:44:48.785: INFO: cloud-controller-manager-k8s-master-15826033-3 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 01:44:48.787] Nov 5 01:44:48.785: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 01:44:48.787] Nov 5 01:44:48.785: INFO: kube-addon-manager-k8s-master-15826033-3 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 01:44:48.787] Nov 5 01:44:48.785: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 01:44:48.788] Nov 5 01:44:48.785: INFO: kube-apiserver-k8s-master-15826033-3 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 01:44:48.788] Nov 5 01:44:48.785: INFO: Container kube-apiserver ready: true, restart count 0 I1105 01:44:49.032] Nov 5 01:44:49.032: INFO: I1105 01:44:49.033] Latency metrics for node k8s-master-15826033-3 I1105 01:44:49.033] Nov 5 01:44:49.032: INFO: I1105 01:44:49.033] Logging node info for node k8s-master-15826033-4 I1105 01:44:49.121] Nov 5 01:44:49.118: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-15826033-4,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-15826033-4,UID:276daef1-6c05-41e3-bacd-abc93d35cc6d,ResourceVersion:2882,Generation:0,CreationTimestamp:2019-11-05 01:35:07 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 0,kubernetes.azure.com/cluster: kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-15826033-4,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.6.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5/providers/Microsoft.Compute/virtualMachines/k8s-master-15826033-4,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 01:35:41 +0000 UTC 2019-11-05 01:35:41 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 01:44:29 +0000 UTC 2019-11-05 01:35:02 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 01:44:29 +0000 UTC 2019-11-05 01:35:02 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 01:44:29 +0000 UTC 2019-11-05 01:35:02 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 01:44:29 +0000 UTC 2019-11-05 01:35:08 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.255.9} {Hostname k8s-master-15826033-4}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:79051fc0bdeb4e32a56acbaf5972ce3a,SystemUUID:C71606B5-7F08-9341-A26D-8A8D9D84FCAE,BootID:4bdf7cf3-2b83-42c7-a7ba-926ebe9ca75b,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 01:44:49.121] Nov 5 01:44:49.118: INFO: I1105 01:44:49.121] Logging kubelet events for node k8s-master-15826033-4 I1105 01:44:49.200] Nov 5 01:44:49.200: INFO: I1105 01:44:49.200] Logging pods the kubelet thinks is on node k8s-master-15826033-4 I1105 01:44:49.282] Nov 5 01:44:49.282: INFO: kube-apiserver-k8s-master-15826033-4 started at 2019-11-05 01:34:55 +0000 UTC (0+1 container statuses recorded) I1105 01:44:49.282] Nov 5 01:44:49.282: INFO: Container kube-apiserver ready: true, restart count 0 I1105 01:44:49.283] Nov 5 01:44:49.282: INFO: kube-controller-manager-k8s-master-15826033-4 started at 2019-11-05 01:34:55 +0000 UTC (0+1 container statuses recorded) I1105 01:44:49.283] Nov 5 01:44:49.282: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 01:44:49.283] Nov 5 01:44:49.282: INFO: kube-scheduler-k8s-master-15826033-4 started at 2019-11-05 01:34:55 +0000 UTC (0+1 container statuses recorded) I1105 01:44:49.283] Nov 5 01:44:49.282: INFO: Container kube-scheduler ready: true, restart count 0 I1105 01:44:49.284] Nov 5 01:44:49.282: INFO: cloud-controller-manager-k8s-master-15826033-4 started at 2019-11-05 01:34:56 +0000 UTC (0+1 container statuses recorded) I1105 01:44:49.284] Nov 5 01:44:49.282: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 01:44:49.284] Nov 5 01:44:49.282: INFO: kube-addon-manager-k8s-master-15826033-4 started at 2019-11-05 01:34:56 +0000 UTC (0+1 container statuses recorded) I1105 01:44:49.284] Nov 5 01:44:49.282: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 01:44:49.285] Nov 5 01:44:49.282: INFO: azure-ip-masq-agent-9nrj8 started at 2019-11-05 01:35:11 +0000 UTC (0+1 container statuses recorded) I1105 01:44:49.285] Nov 5 01:44:49.282: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 01:44:49.285] Nov 5 01:44:49.282: INFO: kube-proxy-rmngk started at 2019-11-05 01:35:11 +0000 UTC (0+1 container statuses recorded) I1105 01:44:49.285] Nov 5 01:44:49.282: INFO: Container kube-proxy ready: true, restart count 0 I1105 01:44:49.530] Nov 5 01:44:49.530: INFO: I1105 01:44:49.531] Latency metrics for node k8s-master-15826033-4 I1105 01:44:49.531] Nov 5 01:44:49.530: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready I1105 01:44:49.608] STEP: Destroying namespace "azuredisk-7528" for this suite. I1105 01:45:03.850] Nov 5 01:45:03.850: INFO: Waiting up to 30s for server preferred namespaced resources to be successfully discovered I1105 01:45:06.785] Nov 5 01:45:06.785: INFO: namespace azuredisk-7528 deletion completed in 17.176618244s I1105 01:45:06.786] [AfterEach] [azuredisk-csi-e2e] [single-az] Pre-Provisioned I1105 01:45:06.786] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/pre_provisioning.go:54 I1105 01:45:12.285] I1105 01:45:12.285] •! Panic [29.971 seconds] I1105 01:45:12.286] [azuredisk-csi-e2e] [single-az] Pre-Provisioned I1105 01:45:12.286] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/pre_provisioning.go:36 I1105 01:45:12.286] [env] should use a pre-provisioned volume and retain PV with reclaimPolicy "Retain" [It] I1105 01:45:12.286] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/pre_provisioning.go:100 I1105 01:45:12.286] I1105 01:45:12.286] Test Panicked I1105 01:45:12.287] runtime error: invalid memory address or nil pointer dereference I1105 01:45:12.287] /usr/local/go/src/runtime/panic.go:82 I1105 01:45:12.287] I1105 01:45:12.287] Full Stack Trace I1105 01:45:12.287] /usr/local/go/src/runtime/panic.go:522 +0x1b5 I1105 01:45:12.288] sigs.k8s.io/azuredisk-csi-driver/test/e2e/testsuites.(*TestPersistentVolumeClaim).Cleanup(0xc000f20e40) I1105 01:45:12.288] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/testsuites/testsuites.go:313 +0x3b I1105 01:45:12.288] sigs.k8s.io/azuredisk-csi-driver/test/e2e/testsuites.(*PreProvisionedReclaimPolicyTest).Run(0xc00087d4e0, 0x20e1540, 0xc0001f1040, 0xc000f086e0) I1105 01:45:12.288] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/testsuites/pre_provisioned_reclaim_policy_tester.go:37 +0x10f I1105 01:45:12.289] sigs.k8s.io/azuredisk-csi-driver/test/e2e.glob..func2.4() I1105 01:45:12.289] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/pre_provisioning.go:123 +0x529 I1105 01:45:12.289] sigs.k8s.io/azuredisk-csi-driver/vendor/github.com/onsi/ginkgo/internal/leafnodes.(*runner).runSync(0xc000568120, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) I1105 01:45:12.289] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/suite_test.go:132 +0xa4 I1105 01:45:12.289] testing.tRunner(0xc0004fa300, 0x1e2da58) I1105 01:45:12.290] /usr/local/go/src/testing/testing.go:865 +0xc0 I1105 01:45:12.290] created by testing.(*T).Run I1105 01:45:12.290] /usr/local/go/src/testing/testing.go:916 +0x35a I1105 01:45:12.290] I1105 01:45:12.290] ------------------------------ I1105 01:45:12.290] [azuredisk-csi-e2e] Dynamic Provisioning [single-az] I1105 01:45:12.290] should create a volume on demand I1105 01:45:12.291] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:78 I1105 01:45:12.291] [BeforeEach] [single-az] I1105 01:45:12.291] /go/src/sigs.k8s.io/azuredisk-csi-driver/vendor/k8s.io/kubernetes/test/e2e/framework/framework.go:150 I1105 01:45:12.291] STEP: Creating a kubernetes client I1105 01:45:12.291] Nov 5 01:45:12.285: INFO: >>> kubeConfig: /workspace/aks978705698/kubeconfig/kubeconfig.eastus2.json I1105 01:45:12.291] STEP: Building a namespace api object, basename azuredisk I1105 01:45:12.540] STEP: Binding the e2e-test-privileged-psp PodSecurityPolicy to the default service account in azuredisk-5598 I1105 01:45:12.799] STEP: Waiting for a default service account to be provisioned in namespace I1105 01:45:12.874] [BeforeEach] [single-az] I1105 01:45:12.874] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:58 I1105 01:45:12.875] [It] should create a volume on demand I1105 01:45:12.875] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:78 I1105 01:45:12.875] STEP: setting up the StorageClass I1105 01:45:12.875] STEP: creating a StorageClass I1105 01:45:12.956] STEP: setting up the PVC and PV I1105 01:45:12.956] STEP: creating a PVC I1105 01:45:13.044] STEP: waiting for PVC to be in phase "Bound" I1105 01:45:13.045] Nov 5 01:45:13.044: INFO: Waiting up to 5m0s for PersistentVolumeClaims [pvc-6lwhr] to have phase Bound I1105 01:45:13.146] Nov 5 01:45:13.145: INFO: PersistentVolumeClaim pvc-6lwhr found but phase is Pending instead of Bound. I1105 01:45:15.224] Nov 5 01:45:15.223: INFO: PersistentVolumeClaim pvc-6lwhr found but phase is Pending instead of Bound. I1105 01:45:17.301] Nov 5 01:45:17.301: INFO: PersistentVolumeClaim pvc-6lwhr found but phase is Pending instead of Bound. I1105 01:45:19.379] Nov 5 01:45:19.379: INFO: PersistentVolumeClaim pvc-6lwhr found but phase is Pending instead of Bound. I1105 01:45:21.457] Nov 5 01:45:21.457: INFO: PersistentVolumeClaim pvc-6lwhr found but phase is Pending instead of Bound. I1105 01:45:23.535] Nov 5 01:45:23.535: INFO: PersistentVolumeClaim pvc-6lwhr found but phase is Pending instead of Bound. I1105 01:45:25.613] Nov 5 01:45:25.613: INFO: PersistentVolumeClaim pvc-6lwhr found but phase is Pending instead of Bound. I1105 01:45:27.691] Nov 5 01:45:27.690: INFO: PersistentVolumeClaim pvc-6lwhr found but phase is Pending instead of Bound. I1105 01:45:29.768] Nov 5 01:45:29.768: INFO: PersistentVolumeClaim pvc-6lwhr found but phase is Pending instead of Bound. I1105 01:45:31.846] Nov 5 01:45:31.846: INFO: PersistentVolumeClaim pvc-6lwhr found but phase is Pending instead of Bound. I1105 01:45:33.924] Nov 5 01:45:33.923: INFO: PersistentVolumeClaim pvc-6lwhr found but phase is Pending instead of Bound. I1105 01:45:36.012] Nov 5 01:45:36.012: INFO: PersistentVolumeClaim pvc-6lwhr found but phase is Pending instead of Bound. I1105 01:45:38.090] Nov 5 01:45:38.090: INFO: PersistentVolumeClaim pvc-6lwhr found but phase is Pending instead of Bound. I1105 01:45:40.168] Nov 5 01:45:40.168: INFO: PersistentVolumeClaim pvc-6lwhr found but phase is Pending instead of Bound. I1105 01:45:42.246] Nov 5 01:45:42.245: INFO: PersistentVolumeClaim pvc-6lwhr found but phase is Pending instead of Bound. I1105 01:45:44.325] Nov 5 01:45:44.325: INFO: PersistentVolumeClaim pvc-6lwhr found but phase is Pending instead of Bound. I1105 01:45:46.403] Nov 5 01:45:46.403: INFO: PersistentVolumeClaim pvc-6lwhr found but phase is Pending instead of Bound. I1105 01:45:48.481] Nov 5 01:45:48.481: INFO: PersistentVolumeClaim pvc-6lwhr found but phase is Pending instead of Bound. I1105 01:45:50.559] Nov 5 01:45:50.559: INFO: PersistentVolumeClaim pvc-6lwhr found but phase is Pending instead of Bound. I1105 01:45:52.637] Nov 5 01:45:52.637: INFO: PersistentVolumeClaim pvc-6lwhr found but phase is Pending instead of Bound. I1105 01:45:54.715] Nov 5 01:45:54.715: INFO: PersistentVolumeClaim pvc-6lwhr found but phase is Pending instead of Bound. I1105 01:45:56.794] Nov 5 01:45:56.793: INFO: PersistentVolumeClaim pvc-6lwhr found but phase is Pending instead of Bound. I1105 01:45:58.871] Nov 5 01:45:58.871: INFO: PersistentVolumeClaim pvc-6lwhr found but phase is Pending instead of Bound. I1105 01:46:00.949] Nov 5 01:46:00.949: INFO: PersistentVolumeClaim pvc-6lwhr found but phase is Pending instead of Bound. I1105 01:46:03.027] Nov 5 01:46:03.027: INFO: PersistentVolumeClaim pvc-6lwhr found but phase is Pending instead of Bound. I1105 01:46:05.113] Nov 5 01:46:05.113: INFO: PersistentVolumeClaim pvc-6lwhr found but phase is Pending instead of Bound. I1105 01:46:07.192] Nov 5 01:46:07.191: INFO: PersistentVolumeClaim pvc-6lwhr found but phase is Pending instead of Bound. I1105 01:46:09.273] Nov 5 01:46:09.272: INFO: PersistentVolumeClaim pvc-6lwhr found but phase is Pending instead of Bound. I1105 01:46:11.350] Nov 5 01:46:11.350: INFO: PersistentVolumeClaim pvc-6lwhr found but phase is Pending instead of Bound. I1105 01:46:13.428] Nov 5 01:46:13.428: INFO: PersistentVolumeClaim pvc-6lwhr found but phase is Pending instead of Bound. I1105 01:46:15.506] Nov 5 01:46:15.506: INFO: PersistentVolumeClaim pvc-6lwhr found but phase is Pending instead of Bound. I1105 01:46:17.584] Nov 5 01:46:17.584: INFO: PersistentVolumeClaim pvc-6lwhr found but phase is Pending instead of Bound. I1105 01:46:19.662] Nov 5 01:46:19.662: INFO: PersistentVolumeClaim pvc-6lwhr found but phase is Pending instead of Bound. I1105 01:46:21.740] Nov 5 01:46:21.740: INFO: PersistentVolumeClaim pvc-6lwhr found but phase is Pending instead of Bound. I1105 01:46:23.818] Nov 5 01:46:23.818: INFO: PersistentVolumeClaim pvc-6lwhr found but phase is Pending instead of Bound. I1105 01:46:25.896] Nov 5 01:46:25.896: INFO: PersistentVolumeClaim pvc-6lwhr found but phase is Pending instead of Bound. I1105 01:46:27.975] Nov 5 01:46:27.974: INFO: PersistentVolumeClaim pvc-6lwhr found but phase is Pending instead of Bound. I1105 01:46:30.056] Nov 5 01:46:30.055: INFO: PersistentVolumeClaim pvc-6lwhr found but phase is Pending instead of Bound. I1105 01:46:32.134] Nov 5 01:46:32.134: INFO: PersistentVolumeClaim pvc-6lwhr found but phase is Pending instead of Bound. I1105 01:46:34.212] Nov 5 01:46:34.212: INFO: PersistentVolumeClaim pvc-6lwhr found but phase is Pending instead of Bound. I1105 01:46:36.290] Nov 5 01:46:36.289: INFO: PersistentVolumeClaim pvc-6lwhr found but phase is Pending instead of Bound. I1105 01:46:38.368] Nov 5 01:46:38.367: INFO: PersistentVolumeClaim pvc-6lwhr found but phase is Pending instead of Bound. I1105 01:46:40.447] Nov 5 01:46:40.447: INFO: PersistentVolumeClaim pvc-6lwhr found but phase is Pending instead of Bound. I1105 01:46:42.525] Nov 5 01:46:42.525: INFO: PersistentVolumeClaim pvc-6lwhr found but phase is Pending instead of Bound. I1105 01:46:44.604] Nov 5 01:46:44.603: INFO: PersistentVolumeClaim pvc-6lwhr found but phase is Pending instead of Bound. I1105 01:46:46.681] Nov 5 01:46:46.681: INFO: PersistentVolumeClaim pvc-6lwhr found but phase is Pending instead of Bound. I1105 01:46:48.759] Nov 5 01:46:48.759: INFO: PersistentVolumeClaim pvc-6lwhr found but phase is Pending instead of Bound. I1105 01:46:50.837] Nov 5 01:46:50.837: INFO: PersistentVolumeClaim pvc-6lwhr found but phase is Pending instead of Bound. I1105 01:46:52.915] Nov 5 01:46:52.915: INFO: PersistentVolumeClaim pvc-6lwhr found but phase is Pending instead of Bound. I1105 01:46:54.993] Nov 5 01:46:54.992: INFO: PersistentVolumeClaim pvc-6lwhr found but phase is Pending instead of Bound. I1105 01:46:57.074] Nov 5 01:46:57.074: INFO: PersistentVolumeClaim pvc-6lwhr found but phase is Pending instead of Bound. I1105 01:46:59.152] Nov 5 01:46:59.152: INFO: PersistentVolumeClaim pvc-6lwhr found but phase is Pending instead of Bound. I1105 01:47:01.229] Nov 5 01:47:01.229: INFO: PersistentVolumeClaim pvc-6lwhr found but phase is Pending instead of Bound. I1105 01:47:03.306] Nov 5 01:47:03.306: INFO: PersistentVolumeClaim pvc-6lwhr found but phase is Pending instead of Bound. I1105 01:47:05.384] Nov 5 01:47:05.384: INFO: PersistentVolumeClaim pvc-6lwhr found but phase is Pending instead of Bound. I1105 01:47:07.462] Nov 5 01:47:07.462: INFO: PersistentVolumeClaim pvc-6lwhr found but phase is Pending instead of Bound. I1105 01:47:09.540] Nov 5 01:47:09.540: INFO: PersistentVolumeClaim pvc-6lwhr found but phase is Pending instead of Bound. I1105 01:47:11.618] Nov 5 01:47:11.618: INFO: PersistentVolumeClaim pvc-6lwhr found but phase is Pending instead of Bound. I1105 01:47:13.696] Nov 5 01:47:13.696: INFO: PersistentVolumeClaim pvc-6lwhr found but phase is Pending instead of Bound. I1105 01:47:15.774] Nov 5 01:47:15.774: INFO: PersistentVolumeClaim pvc-6lwhr found but phase is Pending instead of Bound. I1105 01:47:17.852] Nov 5 01:47:17.852: INFO: PersistentVolumeClaim pvc-6lwhr found but phase is Pending instead of Bound. I1105 01:47:19.930] Nov 5 01:47:19.930: INFO: PersistentVolumeClaim pvc-6lwhr found but phase is Pending instead of Bound. I1105 01:47:22.008] Nov 5 01:47:22.008: INFO: PersistentVolumeClaim pvc-6lwhr found but phase is Pending instead of Bound. I1105 01:47:24.086] Nov 5 01:47:24.086: INFO: PersistentVolumeClaim pvc-6lwhr found but phase is Pending instead of Bound. I1105 01:47:26.163] Nov 5 01:47:26.163: INFO: PersistentVolumeClaim pvc-6lwhr found but phase is Pending instead of Bound. I1105 01:47:28.246] Nov 5 01:47:28.246: INFO: PersistentVolumeClaim pvc-6lwhr found but phase is Pending instead of Bound. I1105 01:47:30.324] Nov 5 01:47:30.324: INFO: PersistentVolumeClaim pvc-6lwhr found but phase is Pending instead of Bound. I1105 01:47:32.403] Nov 5 01:47:32.403: INFO: PersistentVolumeClaim pvc-6lwhr found but phase is Pending instead of Bound. I1105 01:47:34.481] Nov 5 01:47:34.481: INFO: PersistentVolumeClaim pvc-6lwhr found but phase is Pending instead of Bound. I1105 01:47:36.563] Nov 5 01:47:36.563: INFO: PersistentVolumeClaim pvc-6lwhr found but phase is Pending instead of Bound. I1105 01:47:38.641] Nov 5 01:47:38.640: INFO: PersistentVolumeClaim pvc-6lwhr found but phase is Pending instead of Bound. I1105 01:47:40.718] Nov 5 01:47:40.718: INFO: PersistentVolumeClaim pvc-6lwhr found but phase is Pending instead of Bound. I1105 01:47:42.796] Nov 5 01:47:42.796: INFO: PersistentVolumeClaim pvc-6lwhr found but phase is Pending instead of Bound. I1105 01:47:44.874] Nov 5 01:47:44.874: INFO: PersistentVolumeClaim pvc-6lwhr found but phase is Pending instead of Bound. I1105 01:47:46.953] Nov 5 01:47:46.952: INFO: PersistentVolumeClaim pvc-6lwhr found but phase is Pending instead of Bound. I1105 01:47:49.031] Nov 5 01:47:49.030: INFO: PersistentVolumeClaim pvc-6lwhr found but phase is Pending instead of Bound. I1105 01:47:51.108] Nov 5 01:47:51.108: INFO: PersistentVolumeClaim pvc-6lwhr found but phase is Pending instead of Bound. I1105 01:47:53.188] Nov 5 01:47:53.188: INFO: PersistentVolumeClaim pvc-6lwhr found but phase is Pending instead of Bound. I1105 01:47:55.266] Nov 5 01:47:55.266: INFO: PersistentVolumeClaim pvc-6lwhr found but phase is Pending instead of Bound. I1105 01:47:57.344] Nov 5 01:47:57.343: INFO: PersistentVolumeClaim pvc-6lwhr found but phase is Pending instead of Bound. I1105 01:47:59.421] Nov 5 01:47:59.421: INFO: PersistentVolumeClaim pvc-6lwhr found but phase is Pending instead of Bound. I1105 01:48:01.499] Nov 5 01:48:01.499: INFO: PersistentVolumeClaim pvc-6lwhr found but phase is Pending instead of Bound. I1105 01:48:03.577] Nov 5 01:48:03.577: INFO: PersistentVolumeClaim pvc-6lwhr found but phase is Pending instead of Bound. I1105 01:48:05.655] Nov 5 01:48:05.654: INFO: PersistentVolumeClaim pvc-6lwhr found but phase is Pending instead of Bound. I1105 01:48:07.732] Nov 5 01:48:07.732: INFO: PersistentVolumeClaim pvc-6lwhr found but phase is Pending instead of Bound. I1105 01:48:09.820] Nov 5 01:48:09.820: INFO: PersistentVolumeClaim pvc-6lwhr found but phase is Pending instead of Bound. I1105 01:48:11.897] Nov 5 01:48:11.897: INFO: PersistentVolumeClaim pvc-6lwhr found but phase is Pending instead of Bound. I1105 01:48:13.975] Nov 5 01:48:13.975: INFO: PersistentVolumeClaim pvc-6lwhr found but phase is Pending instead of Bound. I1105 01:48:16.053] Nov 5 01:48:16.053: INFO: PersistentVolumeClaim pvc-6lwhr found but phase is Pending instead of Bound. I1105 01:48:18.132] Nov 5 01:48:18.132: INFO: PersistentVolumeClaim pvc-6lwhr found but phase is Pending instead of Bound. I1105 01:48:20.210] Nov 5 01:48:20.210: INFO: PersistentVolumeClaim pvc-6lwhr found but phase is Pending instead of Bound. I1105 01:48:22.288] Nov 5 01:48:22.288: INFO: PersistentVolumeClaim pvc-6lwhr found but phase is Pending instead of Bound. I1105 01:48:24.366] Nov 5 01:48:24.365: INFO: PersistentVolumeClaim pvc-6lwhr found but phase is Pending instead of Bound. I1105 01:48:26.443] Nov 5 01:48:26.443: INFO: PersistentVolumeClaim pvc-6lwhr found but phase is Pending instead of Bound. I1105 01:48:28.521] Nov 5 01:48:28.521: INFO: PersistentVolumeClaim pvc-6lwhr found but phase is Pending instead of Bound. I1105 01:48:30.599] Nov 5 01:48:30.598: INFO: PersistentVolumeClaim pvc-6lwhr found but phase is Pending instead of Bound. I1105 01:48:32.676] Nov 5 01:48:32.676: INFO: PersistentVolumeClaim pvc-6lwhr found but phase is Pending instead of Bound. I1105 01:48:34.754] Nov 5 01:48:34.754: INFO: PersistentVolumeClaim pvc-6lwhr found but phase is Pending instead of Bound. I1105 01:48:36.831] Nov 5 01:48:36.831: INFO: PersistentVolumeClaim pvc-6lwhr found but phase is Pending instead of Bound. I1105 01:48:38.909] Nov 5 01:48:38.909: INFO: PersistentVolumeClaim pvc-6lwhr found but phase is Pending instead of Bound. I1105 01:48:40.986] Nov 5 01:48:40.986: INFO: PersistentVolumeClaim pvc-6lwhr found but phase is Pending instead of Bound. I1105 01:48:43.063] Nov 5 01:48:43.063: INFO: PersistentVolumeClaim pvc-6lwhr found but phase is Pending instead of Bound. I1105 01:48:45.142] Nov 5 01:48:45.141: INFO: PersistentVolumeClaim pvc-6lwhr found but phase is Pending instead of Bound. I1105 01:48:47.220] Nov 5 01:48:47.220: INFO: PersistentVolumeClaim pvc-6lwhr found but phase is Pending instead of Bound. I1105 01:48:49.297] Nov 5 01:48:49.297: INFO: PersistentVolumeClaim pvc-6lwhr found but phase is Pending instead of Bound. I1105 01:48:51.375] Nov 5 01:48:51.374: INFO: PersistentVolumeClaim pvc-6lwhr found but phase is Pending instead of Bound. I1105 01:48:53.452] Nov 5 01:48:53.452: INFO: PersistentVolumeClaim pvc-6lwhr found but phase is Pending instead of Bound. I1105 01:48:55.530] Nov 5 01:48:55.529: INFO: PersistentVolumeClaim pvc-6lwhr found but phase is Pending instead of Bound. I1105 01:48:57.607] Nov 5 01:48:57.607: INFO: PersistentVolumeClaim pvc-6lwhr found but phase is Pending instead of Bound. I1105 01:48:59.685] Nov 5 01:48:59.685: INFO: PersistentVolumeClaim pvc-6lwhr found but phase is Pending instead of Bound. I1105 01:49:01.763] Nov 5 01:49:01.762: INFO: PersistentVolumeClaim pvc-6lwhr found but phase is Pending instead of Bound. I1105 01:49:03.840] Nov 5 01:49:03.840: INFO: PersistentVolumeClaim pvc-6lwhr found but phase is Pending instead of Bound. I1105 01:49:05.917] Nov 5 01:49:05.917: INFO: PersistentVolumeClaim pvc-6lwhr found but phase is Pending instead of Bound. I1105 01:49:07.995] Nov 5 01:49:07.995: INFO: PersistentVolumeClaim pvc-6lwhr found but phase is Pending instead of Bound. I1105 01:49:10.072] Nov 5 01:49:10.072: INFO: PersistentVolumeClaim pvc-6lwhr found but phase is Pending instead of Bound. I1105 01:49:12.150] Nov 5 01:49:12.150: INFO: PersistentVolumeClaim pvc-6lwhr found but phase is Pending instead of Bound. I1105 01:49:14.229] Nov 5 01:49:14.229: INFO: PersistentVolumeClaim pvc-6lwhr found but phase is Pending instead of Bound. I1105 01:49:16.308] Nov 5 01:49:16.307: INFO: PersistentVolumeClaim pvc-6lwhr found but phase is Pending instead of Bound. I1105 01:49:18.386] Nov 5 01:49:18.386: INFO: PersistentVolumeClaim pvc-6lwhr found but phase is Pending instead of Bound. I1105 01:49:20.465] Nov 5 01:49:20.464: INFO: PersistentVolumeClaim pvc-6lwhr found but phase is Pending instead of Bound. I1105 01:49:22.542] Nov 5 01:49:22.542: INFO: PersistentVolumeClaim pvc-6lwhr found but phase is Pending instead of Bound. I1105 01:49:24.619] Nov 5 01:49:24.619: INFO: PersistentVolumeClaim pvc-6lwhr found but phase is Pending instead of Bound. I1105 01:49:26.697] Nov 5 01:49:26.696: INFO: PersistentVolumeClaim pvc-6lwhr found but phase is Pending instead of Bound. I1105 01:49:28.774] Nov 5 01:49:28.774: INFO: PersistentVolumeClaim pvc-6lwhr found but phase is Pending instead of Bound. I1105 01:49:30.852] Nov 5 01:49:30.851: INFO: PersistentVolumeClaim pvc-6lwhr found but phase is Pending instead of Bound. I1105 01:49:32.930] Nov 5 01:49:32.930: INFO: PersistentVolumeClaim pvc-6lwhr found but phase is Pending instead of Bound. I1105 01:49:35.008] Nov 5 01:49:35.007: INFO: PersistentVolumeClaim pvc-6lwhr found but phase is Pending instead of Bound. I1105 01:49:37.105] Nov 5 01:49:37.105: INFO: PersistentVolumeClaim pvc-6lwhr found but phase is Pending instead of Bound. I1105 01:49:39.183] Nov 5 01:49:39.183: INFO: PersistentVolumeClaim pvc-6lwhr found but phase is Pending instead of Bound. I1105 01:49:41.261] Nov 5 01:49:41.260: INFO: PersistentVolumeClaim pvc-6lwhr found but phase is Pending instead of Bound. I1105 01:49:43.339] Nov 5 01:49:43.339: INFO: PersistentVolumeClaim pvc-6lwhr found but phase is Pending instead of Bound. I1105 01:49:45.426] Nov 5 01:49:45.425: INFO: PersistentVolumeClaim pvc-6lwhr found but phase is Pending instead of Bound. I1105 01:49:47.505] Nov 5 01:49:47.505: INFO: PersistentVolumeClaim pvc-6lwhr found but phase is Pending instead of Bound. I1105 01:49:49.584] Nov 5 01:49:49.584: INFO: PersistentVolumeClaim pvc-6lwhr found but phase is Pending instead of Bound. I1105 01:49:51.662] Nov 5 01:49:51.662: INFO: PersistentVolumeClaim pvc-6lwhr found but phase is Pending instead of Bound. I1105 01:49:53.740] Nov 5 01:49:53.739: INFO: PersistentVolumeClaim pvc-6lwhr found but phase is Pending instead of Bound. I1105 01:49:55.818] Nov 5 01:49:55.817: INFO: PersistentVolumeClaim pvc-6lwhr found but phase is Pending instead of Bound. I1105 01:49:57.895] Nov 5 01:49:57.895: INFO: PersistentVolumeClaim pvc-6lwhr found but phase is Pending instead of Bound. I1105 01:49:59.973] Nov 5 01:49:59.973: INFO: PersistentVolumeClaim pvc-6lwhr found but phase is Pending instead of Bound. I1105 01:50:02.051] Nov 5 01:50:02.051: INFO: PersistentVolumeClaim pvc-6lwhr found but phase is Pending instead of Bound. I1105 01:50:04.128] Nov 5 01:50:04.128: INFO: PersistentVolumeClaim pvc-6lwhr found but phase is Pending instead of Bound. I1105 01:50:06.206] Nov 5 01:50:06.206: INFO: PersistentVolumeClaim pvc-6lwhr found but phase is Pending instead of Bound. I1105 01:50:08.283] Nov 5 01:50:08.283: INFO: PersistentVolumeClaim pvc-6lwhr found but phase is Pending instead of Bound. I1105 01:50:10.361] Nov 5 01:50:10.361: INFO: PersistentVolumeClaim pvc-6lwhr found but phase is Pending instead of Bound. I1105 01:50:12.439] Nov 5 01:50:12.439: INFO: PersistentVolumeClaim pvc-6lwhr found but phase is Pending instead of Bound. I1105 01:50:14.439] Nov 5 01:50:14.439: INFO: Unexpected error occurred: PersistentVolumeClaims [pvc-6lwhr] not all in phase Bound within 5m0s I1105 01:50:14.439] [AfterEach] [single-az] I1105 01:50:14.440] /go/src/sigs.k8s.io/azuredisk-csi-driver/vendor/k8s.io/kubernetes/test/e2e/framework/framework.go:151 I1105 01:50:14.440] STEP: Collecting events from namespace "azuredisk-5598". I1105 01:50:14.517] STEP: Found 3 events. I1105 01:50:14.517] Nov 5 01:50:14.517: INFO: At 2019-11-05 01:45:13 +0000 UTC - event for pvc-6lwhr: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "disk.csi.azure.com" or manually created by system administrator I1105 01:50:14.518] Nov 5 01:50:14.517: INFO: At 2019-11-05 01:45:13 +0000 UTC - event for pvc-6lwhr: {disk.csi.azure.com_csi-azuredisk-controller-6f6587bfc6-8vkfl_1376914b-ff6d-11e9-93ab-ba2bee589b8c } Provisioning: External provisioner is provisioning volume for claim "azuredisk-5598/pvc-6lwhr" I1105 01:50:14.518] Nov 5 01:50:14.517: INFO: At 2019-11-05 01:45:13 +0000 UTC - event for pvc-6lwhr: {disk.csi.azure.com_csi-azuredisk-controller-6f6587bfc6-8vkfl_1376914b-ff6d-11e9-93ab-ba2bee589b8c } ProvisioningFailed: failed to provision volume with StorageClass "azuredisk-5598-disk.csi.azure.com-dynamic-sc-w6bd4": rpc error: code = Unknown desc = compute.DisksClient#CreateOrUpdate: Failure sending request: StatusCode=400 -- Original Error: Code="InvalidAvailabilityZone" Message="The zone(s) '' for resource 'Microsoft.Compute/disks/pvc-7849bd5a-21f5-41fe-8fd6-e9779892bcd8' is not supported. The supported zones for location 'eastus2' are '1,2,3'" I1105 01:50:14.594] Nov 5 01:50:14.594: INFO: POD NODE PHASE GRACE CONDITIONS I1105 01:50:14.594] Nov 5 01:50:14.594: INFO: I1105 01:50:14.824] Nov 5 01:50:14.823: INFO: I1105 01:50:14.824] Logging node info for node k8s-agentpool-15826033-vmss000000 I1105 01:50:14.904] Nov 5 01:50:14.900: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-agentpool-15826033-vmss000000,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-agentpool-15826033-vmss000000,UID:15fa874e-54ec-42cd-b243-d3baa512150f,ResourceVersion:3861,Generation:0,CreationTimestamp:2019-11-05 01:34:17 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{agentpool: agentpool,beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_D4s_v3,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 0,kubernetes.azure.com/cluster: kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5,kubernetes.azure.com/role: agent,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-agentpool-15826033-vmss000000,kubernetes.io/os: linux,kubernetes.io/role: agent,node-role.kubernetes.io/agent: ,storageprofile: managed,storagetier: Premium_LRS,topology.disk.csi.azure.com/zone: 0,},Annotations:map[string]string{csi.volume.kubernetes.io/nodeid: {"disk.csi.azure.com":"k8s-agentpool-15826033-vmss000000"},node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.2.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool-15826033-vmss/virtualMachines/0,Unschedulable:false,Taints:[],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-csi-disk.csi.azure.com: {{8 0} {} 8 DecimalSI},cpu: {{4 0} {} 4 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{16797044736 0} {} 16403364Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-csi-disk.csi.azure.com: {{8 0} {} 8 DecimalSI},cpu: {{4 0} {} 4 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{16010612736 0} {} 15635364Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 01:34:56 +0000 UTC 2019-11-05 01:34:56 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 01:49:50 +0000 UTC 2019-11-05 01:34:17 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 01:49:50 +0000 UTC 2019-11-05 01:34:17 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 01:49:50 +0000 UTC 2019-11-05 01:34:17 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 01:49:50 +0000 UTC 2019-11-05 01:34:29 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.0.4} {Hostname k8s-agentpool-15826033-vmss000000}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:afe00792177d49a98f6a44d03d08732d,SystemUUID:4F1777BF-A5C3-004E-81C4-90F0B54C02A0,BootID:13b6a628-ae89-44af-b891-89461030adcb,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[k8sprowinternal.azurecr.io/azuredisk-csi@sha256:75d4ff5d7cd845d6e810c67796ecb037e852fb849ec1eb5b3194564008e6d1e0 k8sprowinternal.azurecr.io/azuredisk-csi:e2e-3e3ed045c6faa509dbd8d2890ee9f56aa2934887] 286930414} {[k8s.gcr.io/kubernetes-dashboard-amd64@sha256:0ae6b69432e78069c5ce2bcde0fe409c5c4d6f0f4d9cd50a17974fea38898747 k8s.gcr.io/kubernetes-dashboard-amd64:v1.10.1] 121711221} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/metrics-server-amd64@sha256:49a9f12f7067d11f42c803dbe61ed2c1299959ad85cb315b25ff7eef8e6b8892 k8s.gcr.io/metrics-server-amd64:v0.2.1] 42541759} {[mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume@sha256:4fd30d43947d4a54fc89ead7985beecfd3c9b2a93a0655a373b1608ab90bd5af mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume:v0.0.7] 22909487} {[quay.io/k8scsi/livenessprobe@sha256:8c9c8a0f330acc2e558ae6624309447d02522e76aa35af23fa19b88abf4381cb quay.io/k8scsi/livenessprobe:v1.0.2] 17415381} {[quay.io/k8scsi/csi-node-driver-registrar@sha256:13daf82fb99e951a4bff8ae5fc7c17c3a8fe7130be6400990d8f6076c32d4599 quay.io/k8scsi/csi-node-driver-registrar:v1.1.0] 15815995} {[mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume@sha256:23d8c6033f02a1ecad05127ebdc931bb871264228661bc122704b0974e4d9fdd mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume:1.0.8] 1159025} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 01:50:14.904] Nov 5 01:50:14.901: INFO: I1105 01:50:14.904] Logging kubelet events for node k8s-agentpool-15826033-vmss000000 I1105 01:50:14.981] Nov 5 01:50:14.981: INFO: I1105 01:50:14.982] Logging pods the kubelet thinks is on node k8s-agentpool-15826033-vmss000000 I1105 01:50:15.064] Nov 5 01:50:15.064: INFO: keyvault-flexvolume-qhnfr started at 2019-11-05 01:34:34 +0000 UTC (0+1 container statuses recorded) I1105 01:50:15.064] Nov 5 01:50:15.064: INFO: Container keyvault-flexvolume ready: true, restart count 0 I1105 01:50:15.064] Nov 5 01:50:15.064: INFO: metrics-server-864ffbc5c-sdxps started at 2019-11-05 01:34:34 +0000 UTC (0+1 container statuses recorded) I1105 01:50:15.065] Nov 5 01:50:15.064: INFO: Container metrics-server ready: true, restart count 0 I1105 01:50:15.065] Nov 5 01:50:15.064: INFO: kube-proxy-wxblc started at 2019-11-05 01:34:34 +0000 UTC (0+1 container statuses recorded) I1105 01:50:15.065] Nov 5 01:50:15.064: INFO: Container kube-proxy ready: true, restart count 0 I1105 01:50:15.065] Nov 5 01:50:15.064: INFO: kubernetes-dashboard-66dd8b8df7-ks284 started at 2019-11-05 01:34:35 +0000 UTC (0+1 container statuses recorded) I1105 01:50:15.065] Nov 5 01:50:15.064: INFO: Container kubernetes-dashboard ready: true, restart count 0 I1105 01:50:15.065] Nov 5 01:50:15.064: INFO: csi-azuredisk-node-whq2g started at 2019-11-05 01:38:24 +0000 UTC (0+3 container statuses recorded) I1105 01:50:15.065] Nov 5 01:50:15.064: INFO: Container azuredisk ready: true, restart count 0 I1105 01:50:15.065] Nov 5 01:50:15.064: INFO: Container liveness-probe ready: true, restart count 0 I1105 01:50:15.066] Nov 5 01:50:15.064: INFO: Container node-driver-registrar ready: true, restart count 0 I1105 01:50:15.066] Nov 5 01:50:15.064: INFO: blobfuse-flexvol-installer-bsvqt started at 2019-11-05 01:34:34 +0000 UTC (0+1 container statuses recorded) I1105 01:50:15.066] Nov 5 01:50:15.064: INFO: Container blobfuse-flexvol-installer ready: true, restart count 0 I1105 01:50:15.066] Nov 5 01:50:15.064: INFO: azure-ip-masq-agent-vx86b started at 2019-11-05 01:34:34 +0000 UTC (0+1 container statuses recorded) I1105 01:50:15.066] Nov 5 01:50:15.064: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 01:50:15.313] Nov 5 01:50:15.313: INFO: I1105 01:50:15.314] Latency metrics for node k8s-agentpool-15826033-vmss000000 I1105 01:50:15.314] Nov 5 01:50:15.313: INFO: I1105 01:50:15.314] Logging node info for node k8s-agentpool-15826033-vmss000001 I1105 01:50:15.393] Nov 5 01:50:15.390: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-agentpool-15826033-vmss000001,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-agentpool-15826033-vmss000001,UID:4a8ea605-cc0b-4e3b-9060-d8867d269e6c,ResourceVersion:3835,Generation:0,CreationTimestamp:2019-11-05 01:34:09 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{agentpool: agentpool,beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_D4s_v3,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 1,kubernetes.azure.com/cluster: kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5,kubernetes.azure.com/role: agent,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-agentpool-15826033-vmss000001,kubernetes.io/os: linux,kubernetes.io/role: agent,node-role.kubernetes.io/agent: ,storageprofile: managed,storagetier: Premium_LRS,topology.disk.csi.azure.com/zone: 1,},Annotations:map[string]string{csi.volume.kubernetes.io/nodeid: {"disk.csi.azure.com":"k8s-agentpool-15826033-vmss000001"},node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.0.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool-15826033-vmss/virtualMachines/1,Unschedulable:false,Taints:[],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-csi-disk.csi.azure.com: {{8 0} {} 8 DecimalSI},cpu: {{4 0} {} 4 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{16797044736 0} {} 16403364Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-csi-disk.csi.azure.com: {{8 0} {} 8 DecimalSI},cpu: {{4 0} {} 4 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{16010612736 0} {} 15635364Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 01:34:57 +0000 UTC 2019-11-05 01:34:57 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 01:49:41 +0000 UTC 2019-11-05 01:34:08 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 01:49:41 +0000 UTC 2019-11-05 01:34:08 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 01:49:41 +0000 UTC 2019-11-05 01:34:08 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 01:49:41 +0000 UTC 2019-11-05 01:34:29 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.0.5} {Hostname k8s-agentpool-15826033-vmss000001}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:b6b9fbf4a2fd4fa5a7ae472415fe30b8,SystemUUID:0162B3A4-3955-9A46-ABD4-A37BD75848CC,BootID:ee93003f-d47c-4118-9975-57365b3443a5,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[k8sprowinternal.azurecr.io/azuredisk-csi@sha256:75d4ff5d7cd845d6e810c67796ecb037e852fb849ec1eb5b3194564008e6d1e0 k8sprowinternal.azurecr.io/azuredisk-csi:e2e-3e3ed045c6faa509dbd8d2890ee9f56aa2934887] 286930414} {[gcr.io/kubernetes-helm/tiller@sha256:f6d8f4ab9ba993b5f5b60a6edafe86352eabe474ffeb84cb6c79b8866dce45d1 gcr.io/kubernetes-helm/tiller:v2.11.0] 71821984} {[quay.io/k8scsi/csi-provisioner@sha256:d657c839dce87324fe2b677302913f9386f885f8746be7bea0ced5b0844e3433 quay.io/k8scsi/csi-provisioner:v1.3.1] 50596998} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[quay.io/k8scsi/csi-attacher@sha256:26fccd7a99d973845df1193b46ebdcc6ab8dc5f6e6be319750c471fce1742d13 quay.io/k8scsi/csi-attacher:v1.2.0] 46226754} {[quay.io/k8scsi/csi-cluster-driver-registrar@sha256:fafd75ae5442f192cfa8c2e792903aee30d5884b62e802e4464b0a895d21e3ef quay.io/k8scsi/csi-cluster-driver-registrar:v1.0.1] 45874691} {[quay.io/k8scsi/csi-snapshotter@sha256:a49e0da1af6f2bf717e41ba1eee8b5e6a1cbd66a709dd92cc43fe475fe2589eb quay.io/k8scsi/csi-snapshotter:v1.1.0] 44325352} {[k8s.gcr.io/coredns@sha256:e83beb5e43f8513fa735e77ffc5859640baea30a882a11cc75c4c3244a737d3c k8s.gcr.io/coredns:1.5.0] 42488424} {[mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume@sha256:4fd30d43947d4a54fc89ead7985beecfd3c9b2a93a0655a373b1608ab90bd5af mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume:v0.0.7] 22909487} {[quay.io/k8scsi/livenessprobe@sha256:8c9c8a0f330acc2e558ae6624309447d02522e76aa35af23fa19b88abf4381cb quay.io/k8scsi/livenessprobe:v1.0.2] 17415381} {[quay.io/k8scsi/csi-node-driver-registrar@sha256:13daf82fb99e951a4bff8ae5fc7c17c3a8fe7130be6400990d8f6076c32d4599 quay.io/k8scsi/csi-node-driver-registrar:v1.1.0] 15815995} {[quay.io/k8scsi/livenessprobe@sha256:dde617756e0f602adc566ab71fd885f1dad451ad3fb063ac991c95a2ff47aea5 quay.io/k8scsi/livenessprobe:v1.1.0] 14967303} {[mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume@sha256:23d8c6033f02a1ecad05127ebdc931bb871264228661bc122704b0974e4d9fdd mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume:1.0.8] 1159025} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 01:50:15.394] Nov 5 01:50:15.390: INFO: I1105 01:50:15.394] Logging kubelet events for node k8s-agentpool-15826033-vmss000001 I1105 01:50:15.472] Nov 5 01:50:15.472: INFO: I1105 01:50:15.472] Logging pods the kubelet thinks is on node k8s-agentpool-15826033-vmss000001 I1105 01:50:15.555] Nov 5 01:50:15.555: INFO: blobfuse-flexvol-installer-rw5dw started at 2019-11-05 01:34:35 +0000 UTC (0+1 container statuses recorded) I1105 01:50:15.555] Nov 5 01:50:15.555: INFO: Container blobfuse-flexvol-installer ready: true, restart count 0 I1105 01:50:15.555] Nov 5 01:50:15.555: INFO: coredns-7f68dcdbdb-jt522 started at 2019-11-05 01:34:35 +0000 UTC (0+1 container statuses recorded) I1105 01:50:15.555] Nov 5 01:50:15.555: INFO: Container coredns ready: true, restart count 0 I1105 01:50:15.556] Nov 5 01:50:15.555: INFO: azure-ip-masq-agent-6c6vn started at 2019-11-05 01:34:35 +0000 UTC (0+1 container statuses recorded) I1105 01:50:15.556] Nov 5 01:50:15.555: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 01:50:15.556] Nov 5 01:50:15.555: INFO: keyvault-flexvolume-rdlcj started at 2019-11-05 01:34:35 +0000 UTC (0+1 container statuses recorded) I1105 01:50:15.556] Nov 5 01:50:15.555: INFO: Container keyvault-flexvolume ready: true, restart count 0 I1105 01:50:15.556] Nov 5 01:50:15.555: INFO: kube-proxy-6kh57 started at 2019-11-05 01:34:35 +0000 UTC (0+1 container statuses recorded) I1105 01:50:15.556] Nov 5 01:50:15.555: INFO: Container kube-proxy ready: true, restart count 0 I1105 01:50:15.556] Nov 5 01:50:15.555: INFO: tiller-deploy-54c96cb5df-8d9fs started at 2019-11-05 01:34:35 +0000 UTC (0+1 container statuses recorded) I1105 01:50:15.557] Nov 5 01:50:15.555: INFO: Container tiller ready: true, restart count 0 I1105 01:50:15.557] Nov 5 01:50:15.555: INFO: csi-azuredisk-node-ghv5w started at 2019-11-05 01:38:24 +0000 UTC (0+3 container statuses recorded) I1105 01:50:15.557] Nov 5 01:50:15.555: INFO: Container azuredisk ready: true, restart count 0 I1105 01:50:15.557] Nov 5 01:50:15.555: INFO: Container liveness-probe ready: true, restart count 0 I1105 01:50:15.557] Nov 5 01:50:15.555: INFO: Container node-driver-registrar ready: true, restart count 0 I1105 01:50:15.557] Nov 5 01:50:15.555: INFO: csi-azuredisk-controller-6f6587bfc6-8vkfl started at 2019-11-05 01:38:24 +0000 UTC (0+6 container statuses recorded) I1105 01:50:15.557] Nov 5 01:50:15.555: INFO: Container azuredisk ready: true, restart count 0 I1105 01:50:15.558] Nov 5 01:50:15.555: INFO: Container cluster-driver-registrar ready: true, restart count 0 I1105 01:50:15.558] Nov 5 01:50:15.555: INFO: Container csi-attacher ready: true, restart count 0 I1105 01:50:15.558] Nov 5 01:50:15.555: INFO: Container csi-provisioner ready: true, restart count 0 I1105 01:50:15.558] Nov 5 01:50:15.555: INFO: Container csi-snapshotter ready: true, restart count 0 I1105 01:50:15.558] Nov 5 01:50:15.555: INFO: Container liveness-probe ready: true, restart count 0 I1105 01:50:15.813] Nov 5 01:50:15.813: INFO: I1105 01:50:15.814] Latency metrics for node k8s-agentpool-15826033-vmss000001 I1105 01:50:15.814] Nov 5 01:50:15.813: INFO: I1105 01:50:15.814] Logging node info for node k8s-master-15826033-0 I1105 01:50:15.893] Nov 5 01:50:15.891: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-15826033-0,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-15826033-0,UID:aceb0231-4645-4a9a-b296-90ada851986b,ResourceVersion:3796,Generation:0,CreationTimestamp:2019-11-05 01:35:06 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 1,kubernetes.azure.com/cluster: kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-15826033-0,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.4.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5/providers/Microsoft.Compute/virtualMachines/k8s-master-15826033-0,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 01:35:20 +0000 UTC 2019-11-05 01:35:20 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 01:49:28 +0000 UTC 2019-11-05 01:34:57 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 01:49:28 +0000 UTC 2019-11-05 01:34:57 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 01:49:28 +0000 UTC 2019-11-05 01:34:57 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 01:49:28 +0000 UTC 2019-11-05 01:35:07 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.255.5} {Hostname k8s-master-15826033-0}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:1915756c22a540dc8eb30493987e9a43,SystemUUID:742C969C-5A71-D744-8E61-F27E38B56EB2,BootID:03f0672e-8fa9-4309-956f-8e3c371902c3,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 01:50:15.893] Nov 5 01:50:15.891: INFO: I1105 01:50:15.893] Logging kubelet events for node k8s-master-15826033-0 I1105 01:50:15.972] Nov 5 01:50:15.972: INFO: I1105 01:50:15.972] Logging pods the kubelet thinks is on node k8s-master-15826033-0 I1105 01:50:16.056] Nov 5 01:50:16.055: INFO: azure-ip-masq-agent-wgdpx started at 2019-11-05 01:35:06 +0000 UTC (0+1 container statuses recorded) I1105 01:50:16.056] Nov 5 01:50:16.055: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 01:50:16.056] Nov 5 01:50:16.055: INFO: kube-proxy-7d2px started at 2019-11-05 01:35:06 +0000 UTC (0+1 container statuses recorded) I1105 01:50:16.056] Nov 5 01:50:16.055: INFO: Container kube-proxy ready: true, restart count 0 I1105 01:50:16.056] Nov 5 01:50:16.055: INFO: cloud-controller-manager-k8s-master-15826033-0 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 01:50:16.057] Nov 5 01:50:16.055: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 01:50:16.057] Nov 5 01:50:16.055: INFO: kube-addon-manager-k8s-master-15826033-0 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 01:50:16.057] Nov 5 01:50:16.055: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 01:50:16.057] Nov 5 01:50:16.055: INFO: kube-apiserver-k8s-master-15826033-0 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 01:50:16.057] Nov 5 01:50:16.055: INFO: Container kube-apiserver ready: true, restart count 0 I1105 01:50:16.057] Nov 5 01:50:16.055: INFO: kube-controller-manager-k8s-master-15826033-0 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 01:50:16.058] Nov 5 01:50:16.055: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 01:50:16.058] Nov 5 01:50:16.055: INFO: kube-scheduler-k8s-master-15826033-0 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 01:50:16.058] Nov 5 01:50:16.055: INFO: Container kube-scheduler ready: true, restart count 0 I1105 01:50:16.307] Nov 5 01:50:16.307: INFO: I1105 01:50:16.307] Latency metrics for node k8s-master-15826033-0 I1105 01:50:16.307] Nov 5 01:50:16.307: INFO: I1105 01:50:16.307] Logging node info for node k8s-master-15826033-1 I1105 01:50:16.386] Nov 5 01:50:16.384: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-15826033-1,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-15826033-1,UID:1a13ec61-c89a-456c-8c94-bfc8e717c2e7,ResourceVersion:3801,Generation:0,CreationTimestamp:2019-11-05 01:35:07 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 1,kubernetes.azure.com/cluster: kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-15826033-1,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.5.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5/providers/Microsoft.Compute/virtualMachines/k8s-master-15826033-1,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 01:35:31 +0000 UTC 2019-11-05 01:35:31 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 01:49:29 +0000 UTC 2019-11-05 01:35:00 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 01:49:29 +0000 UTC 2019-11-05 01:35:00 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 01:49:29 +0000 UTC 2019-11-05 01:35:00 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 01:49:29 +0000 UTC 2019-11-05 01:35:07 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.255.6} {Hostname k8s-master-15826033-1}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:e0f2bccddd5b4555b180d14081ab59a8,SystemUUID:AE126631-CAD2-B54F-B22E-1F499342A4D6,BootID:509b9a9d-b046-4861-aa35-8bcb77465de2,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 01:50:16.386] Nov 5 01:50:16.384: INFO: I1105 01:50:16.387] Logging kubelet events for node k8s-master-15826033-1 I1105 01:50:16.466] Nov 5 01:50:16.466: INFO: I1105 01:50:16.466] Logging pods the kubelet thinks is on node k8s-master-15826033-1 I1105 01:50:16.547] Nov 5 01:50:16.547: INFO: cloud-controller-manager-k8s-master-15826033-1 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 01:50:16.547] Nov 5 01:50:16.547: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 01:50:16.548] Nov 5 01:50:16.547: INFO: kube-addon-manager-k8s-master-15826033-1 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 01:50:16.548] Nov 5 01:50:16.547: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 01:50:16.548] Nov 5 01:50:16.547: INFO: kube-apiserver-k8s-master-15826033-1 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 01:50:16.548] Nov 5 01:50:16.547: INFO: Container kube-apiserver ready: true, restart count 0 I1105 01:50:16.548] Nov 5 01:50:16.547: INFO: kube-controller-manager-k8s-master-15826033-1 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 01:50:16.548] Nov 5 01:50:16.547: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 01:50:16.549] Nov 5 01:50:16.547: INFO: kube-scheduler-k8s-master-15826033-1 started at 2019-11-05 01:36:16 +0000 UTC (0+1 container statuses recorded) I1105 01:50:16.549] Nov 5 01:50:16.547: INFO: Container kube-scheduler ready: true, restart count 0 I1105 01:50:16.549] Nov 5 01:50:16.547: INFO: azure-ip-masq-agent-w6dpr started at 2019-11-05 01:35:10 +0000 UTC (0+1 container statuses recorded) I1105 01:50:16.549] Nov 5 01:50:16.547: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 01:50:16.549] Nov 5 01:50:16.547: INFO: kube-proxy-8kmr5 started at 2019-11-05 01:35:10 +0000 UTC (0+1 container statuses recorded) I1105 01:50:16.549] Nov 5 01:50:16.547: INFO: Container kube-proxy ready: true, restart count 0 I1105 01:50:16.793] Nov 5 01:50:16.793: INFO: I1105 01:50:16.793] Latency metrics for node k8s-master-15826033-1 I1105 01:50:16.793] Nov 5 01:50:16.793: INFO: I1105 01:50:16.793] Logging node info for node k8s-master-15826033-2 I1105 01:50:16.872] Nov 5 01:50:16.870: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-15826033-2,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-15826033-2,UID:acb3a851-d6df-4549-ba56-4b7e96553c17,ResourceVersion:3892,Generation:0,CreationTimestamp:2019-11-05 01:34:09 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 2,kubernetes.azure.com/cluster: kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-15826033-2,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.1.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5/providers/Microsoft.Compute/virtualMachines/k8s-master-15826033-2,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 01:34:44 +0000 UTC 2019-11-05 01:34:44 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 01:50:01 +0000 UTC 2019-11-05 01:34:07 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 01:50:01 +0000 UTC 2019-11-05 01:34:07 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 01:50:01 +0000 UTC 2019-11-05 01:34:07 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 01:50:01 +0000 UTC 2019-11-05 01:34:29 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{Hostname k8s-master-15826033-2} {InternalIP 10.240.255.7}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:0885b0c2f6174106bde4fa0eae2cbc6b,SystemUUID:C176121C-5D7A-6243-9B05-8B68F8022B64,BootID:30430638-7ab6-4761-b984-2387e1a3f5b2,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 01:50:16.873] Nov 5 01:50:16.870: INFO: I1105 01:50:16.873] Logging kubelet events for node k8s-master-15826033-2 I1105 01:50:16.952] Nov 5 01:50:16.952: INFO: I1105 01:50:16.952] Logging pods the kubelet thinks is on node k8s-master-15826033-2 I1105 01:50:17.034] Nov 5 01:50:17.034: INFO: kube-apiserver-k8s-master-15826033-2 started at 2019-11-05 01:33:54 +0000 UTC (0+1 container statuses recorded) I1105 01:50:17.035] Nov 5 01:50:17.034: INFO: Container kube-apiserver ready: true, restart count 0 I1105 01:50:17.035] Nov 5 01:50:17.034: INFO: kube-controller-manager-k8s-master-15826033-2 started at 2019-11-05 01:33:54 +0000 UTC (0+1 container statuses recorded) I1105 01:50:17.035] Nov 5 01:50:17.034: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 01:50:17.035] Nov 5 01:50:17.034: INFO: kube-scheduler-k8s-master-15826033-2 started at 2019-11-05 01:33:54 +0000 UTC (0+1 container statuses recorded) I1105 01:50:17.035] Nov 5 01:50:17.034: INFO: Container kube-scheduler ready: true, restart count 0 I1105 01:50:17.035] Nov 5 01:50:17.034: INFO: azure-ip-masq-agent-hw7s5 started at 2019-11-05 01:34:34 +0000 UTC (0+1 container statuses recorded) I1105 01:50:17.036] Nov 5 01:50:17.034: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 01:50:17.036] Nov 5 01:50:17.034: INFO: kube-proxy-5xpp6 started at 2019-11-05 01:34:34 +0000 UTC (0+1 container statuses recorded) I1105 01:50:17.036] Nov 5 01:50:17.034: INFO: Container kube-proxy ready: true, restart count 0 I1105 01:50:17.036] Nov 5 01:50:17.034: INFO: cloud-controller-manager-k8s-master-15826033-2 started at 2019-11-05 01:33:54 +0000 UTC (0+1 container statuses recorded) I1105 01:50:17.036] Nov 5 01:50:17.034: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 01:50:17.036] Nov 5 01:50:17.034: INFO: kube-addon-manager-k8s-master-15826033-2 started at 2019-11-05 01:33:54 +0000 UTC (0+1 container statuses recorded) I1105 01:50:17.036] Nov 5 01:50:17.034: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 01:50:17.312] Nov 5 01:50:17.312: INFO: I1105 01:50:17.312] Latency metrics for node k8s-master-15826033-2 I1105 01:50:17.313] Nov 5 01:50:17.312: INFO: I1105 01:50:17.313] Logging node info for node k8s-master-15826033-3 I1105 01:50:17.391] Nov 5 01:50:17.389: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-15826033-3,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-15826033-3,UID:b7702272-1876-434e-b6a0-17fa34c6757f,ResourceVersion:3790,Generation:0,CreationTimestamp:2019-11-05 01:35:06 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 0,kubernetes.azure.com/cluster: kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-15826033-3,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.3.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5/providers/Microsoft.Compute/virtualMachines/k8s-master-15826033-3,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 01:35:17 +0000 UTC 2019-11-05 01:35:17 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 01:49:28 +0000 UTC 2019-11-05 01:35:00 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 01:49:28 +0000 UTC 2019-11-05 01:35:00 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 01:49:28 +0000 UTC 2019-11-05 01:35:00 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 01:49:28 +0000 UTC 2019-11-05 01:35:06 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.255.8} {Hostname k8s-master-15826033-3}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:fbf366963961422c9d90bedf6639c8e1,SystemUUID:5388948D-1A8D-EA41-B8B8-7F4523EDF5AA,BootID:42f5884c-7418-486a-8cf3-75188a331f38,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 01:50:17.392] Nov 5 01:50:17.389: INFO: I1105 01:50:17.392] Logging kubelet events for node k8s-master-15826033-3 I1105 01:50:17.470] Nov 5 01:50:17.470: INFO: I1105 01:50:17.470] Logging pods the kubelet thinks is on node k8s-master-15826033-3 I1105 01:50:17.552] Nov 5 01:50:17.551: INFO: cloud-controller-manager-k8s-master-15826033-3 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 01:50:17.552] Nov 5 01:50:17.551: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 01:50:17.552] Nov 5 01:50:17.551: INFO: kube-addon-manager-k8s-master-15826033-3 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 01:50:17.552] Nov 5 01:50:17.551: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 01:50:17.552] Nov 5 01:50:17.551: INFO: kube-apiserver-k8s-master-15826033-3 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 01:50:17.552] Nov 5 01:50:17.551: INFO: Container kube-apiserver ready: true, restart count 0 I1105 01:50:17.552] Nov 5 01:50:17.551: INFO: kube-controller-manager-k8s-master-15826033-3 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 01:50:17.553] Nov 5 01:50:17.551: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 01:50:17.553] Nov 5 01:50:17.551: INFO: kube-scheduler-k8s-master-15826033-3 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 01:50:17.553] Nov 5 01:50:17.551: INFO: Container kube-scheduler ready: true, restart count 0 I1105 01:50:17.553] Nov 5 01:50:17.551: INFO: kube-proxy-xz24z started at 2019-11-05 01:35:06 +0000 UTC (0+1 container statuses recorded) I1105 01:50:17.553] Nov 5 01:50:17.551: INFO: Container kube-proxy ready: true, restart count 0 I1105 01:50:17.553] Nov 5 01:50:17.551: INFO: azure-ip-masq-agent-dqfpw started at 2019-11-05 01:35:06 +0000 UTC (0+1 container statuses recorded) I1105 01:50:17.554] Nov 5 01:50:17.551: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 01:50:17.801] Nov 5 01:50:17.801: INFO: I1105 01:50:17.802] Latency metrics for node k8s-master-15826033-3 I1105 01:50:17.802] Nov 5 01:50:17.801: INFO: I1105 01:50:17.802] Logging node info for node k8s-master-15826033-4 I1105 01:50:17.880] Nov 5 01:50:17.878: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-15826033-4,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-15826033-4,UID:276daef1-6c05-41e3-bacd-abc93d35cc6d,ResourceVersion:3804,Generation:0,CreationTimestamp:2019-11-05 01:35:07 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 0,kubernetes.azure.com/cluster: kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-15826033-4,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.6.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5/providers/Microsoft.Compute/virtualMachines/k8s-master-15826033-4,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 01:35:41 +0000 UTC 2019-11-05 01:35:41 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 01:49:30 +0000 UTC 2019-11-05 01:35:02 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 01:49:30 +0000 UTC 2019-11-05 01:35:02 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 01:49:30 +0000 UTC 2019-11-05 01:35:02 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 01:49:30 +0000 UTC 2019-11-05 01:35:08 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.255.9} {Hostname k8s-master-15826033-4}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:79051fc0bdeb4e32a56acbaf5972ce3a,SystemUUID:C71606B5-7F08-9341-A26D-8A8D9D84FCAE,BootID:4bdf7cf3-2b83-42c7-a7ba-926ebe9ca75b,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 01:50:17.880] Nov 5 01:50:17.878: INFO: I1105 01:50:17.881] Logging kubelet events for node k8s-master-15826033-4 I1105 01:50:17.971] Nov 5 01:50:17.971: INFO: I1105 01:50:17.971] Logging pods the kubelet thinks is on node k8s-master-15826033-4 I1105 01:50:18.052] Nov 5 01:50:18.052: INFO: cloud-controller-manager-k8s-master-15826033-4 started at 2019-11-05 01:34:56 +0000 UTC (0+1 container statuses recorded) I1105 01:50:18.052] Nov 5 01:50:18.052: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 01:50:18.053] Nov 5 01:50:18.052: INFO: kube-addon-manager-k8s-master-15826033-4 started at 2019-11-05 01:34:56 +0000 UTC (0+1 container statuses recorded) I1105 01:50:18.053] Nov 5 01:50:18.052: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 01:50:18.053] Nov 5 01:50:18.052: INFO: azure-ip-masq-agent-9nrj8 started at 2019-11-05 01:35:11 +0000 UTC (0+1 container statuses recorded) I1105 01:50:18.053] Nov 5 01:50:18.052: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 01:50:18.053] Nov 5 01:50:18.052: INFO: kube-proxy-rmngk started at 2019-11-05 01:35:11 +0000 UTC (0+1 container statuses recorded) I1105 01:50:18.053] Nov 5 01:50:18.052: INFO: Container kube-proxy ready: true, restart count 0 I1105 01:50:18.054] Nov 5 01:50:18.052: INFO: kube-apiserver-k8s-master-15826033-4 started at 2019-11-05 01:34:55 +0000 UTC (0+1 container statuses recorded) I1105 01:50:18.054] Nov 5 01:50:18.052: INFO: Container kube-apiserver ready: true, restart count 0 I1105 01:50:18.054] Nov 5 01:50:18.052: INFO: kube-controller-manager-k8s-master-15826033-4 started at 2019-11-05 01:34:55 +0000 UTC (0+1 container statuses recorded) I1105 01:50:18.054] Nov 5 01:50:18.052: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 01:50:18.054] Nov 5 01:50:18.052: INFO: kube-scheduler-k8s-master-15826033-4 started at 2019-11-05 01:34:55 +0000 UTC (0+1 container statuses recorded) I1105 01:50:18.054] Nov 5 01:50:18.052: INFO: Container kube-scheduler ready: true, restart count 0 I1105 01:50:18.301] Nov 5 01:50:18.301: INFO: I1105 01:50:18.301] Latency metrics for node k8s-master-15826033-4 I1105 01:50:18.301] Nov 5 01:50:18.301: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready I1105 01:50:18.381] STEP: Destroying namespace "azuredisk-5598" for this suite. I1105 01:50:32.622] Nov 5 01:50:32.622: INFO: Waiting up to 30s for server preferred namespaced resources to be successfully discovered I1105 01:50:35.564] Nov 5 01:50:35.563: INFO: namespace azuredisk-5598 deletion completed in 17.182801471s I1105 01:50:35.564] I1105 01:50:35.564] • Failure [323.278 seconds] I1105 01:50:35.564] [azuredisk-csi-e2e] Dynamic Provisioning I1105 01:50:35.564] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:33 I1105 01:50:35.564] [single-az] I1105 01:50:35.565] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:36 I1105 01:50:35.565] should create a volume on demand [It] I1105 01:50:35.565] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:78 I1105 01:50:35.565] I1105 01:50:35.565] Unexpected error: I1105 01:50:35.565] <*errors.errorString | 0xc0007206e0>: { I1105 01:50:35.565] s: "PersistentVolumeClaims [pvc-6lwhr] not all in phase Bound within 5m0s", I1105 01:50:35.565] } I1105 01:50:35.566] PersistentVolumeClaims [pvc-6lwhr] not all in phase Bound within 5m0s I1105 01:50:35.566] occurred I1105 01:50:35.566] I1105 01:50:35.566] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/testsuites/testsuites.go:278 I1105 01:50:35.566] ------------------------------ I1105 01:50:35.566] [azuredisk-csi-e2e] [single-az] Pre-Provisioned I1105 01:50:35.566] [env] should use a pre-provisioned volume and mount it as readOnly in a pod I1105 01:50:35.566] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/pre_provisioning.go:66 I1105 01:50:35.566] [BeforeEach] [azuredisk-csi-e2e] [single-az] Pre-Provisioned I1105 01:50:35.566] /go/src/sigs.k8s.io/azuredisk-csi-driver/vendor/k8s.io/kubernetes/test/e2e/framework/framework.go:150 I1105 01:50:35.567] STEP: Creating a kubernetes client I1105 01:50:35.567] Nov 5 01:50:35.564: INFO: >>> kubeConfig: /workspace/aks978705698/kubeconfig/kubeconfig.eastus2.json I1105 01:50:35.567] STEP: Building a namespace api object, basename azuredisk I1105 01:50:35.799] STEP: Binding the e2e-test-privileged-psp PodSecurityPolicy to the default service account in azuredisk-8276 I1105 01:50:36.057] STEP: Waiting for a default service account to be provisioned in namespace I1105 01:50:36.132] [BeforeEach] [azuredisk-csi-e2e] [single-az] Pre-Provisioned I1105 01:50:36.132] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/pre_provisioning.go:48 I1105 01:50:36.133] [It] [env] should use a pre-provisioned volume and mount it as readOnly in a pod I1105 01:50:36.133] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/pre_provisioning.go:66 I1105 01:50:38.020] STEP: Successfully provisioned AzureDisk volume: "/subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5/providers/Microsoft.Compute/disks/pre-provisioned-readOnly" I1105 01:50:38.020] I1105 01:50:38.020] STEP: setting up the PV I1105 01:50:38.020] STEP: creating a PV I1105 01:50:38.103] STEP: setting up the PVC I1105 01:50:38.103] STEP: creating a PVC I1105 01:50:38.186] STEP: waiting for PVC to be in phase "Bound" I1105 01:50:38.186] Nov 5 01:50:38.185: INFO: Waiting up to 5m0s for PersistentVolumeClaims [pvc-wsxcw] to have phase Bound I1105 01:50:38.263] Nov 5 01:50:38.263: INFO: PersistentVolumeClaim pvc-wsxcw found and phase=Bound (77.590805ms) I1105 01:50:38.263] STEP: checking the PVC I1105 01:50:38.340] STEP: validating provisioned PV I1105 01:50:38.417] STEP: checking the PV I1105 01:50:38.417] STEP: deploying the pod I1105 01:50:38.498] STEP: checking that the pods command exits with an error I1105 01:50:38.498] Nov 5 01:50:38.498: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-qd9pw" in namespace "azuredisk-8276" to be "Error status code" I1105 01:50:38.579] Nov 5 01:50:38.579: INFO: Pod "azuredisk-volume-tester-qd9pw": Phase="Pending", Reason="", readiness=false. Elapsed: 81.034932ms I1105 01:50:40.657] Nov 5 01:50:40.657: INFO: Pod "azuredisk-volume-tester-qd9pw": Phase="Pending", Reason="", readiness=false. Elapsed: 2.159270747s I1105 01:50:42.735] Nov 5 01:50:42.735: INFO: Pod "azuredisk-volume-tester-qd9pw": Phase="Pending", Reason="", readiness=false. Elapsed: 4.236977558s I1105 01:50:44.813] Nov 5 01:50:44.813: INFO: Pod "azuredisk-volume-tester-qd9pw": Phase="Pending", Reason="", readiness=false. Elapsed: 6.315079572s I1105 01:50:46.891] Nov 5 01:50:46.891: INFO: Pod "azuredisk-volume-tester-qd9pw": Phase="Pending", Reason="", readiness=false. Elapsed: 8.392888984s I1105 01:50:48.969] Nov 5 01:50:48.969: INFO: Pod "azuredisk-volume-tester-qd9pw": Phase="Pending", Reason="", readiness=false. Elapsed: 10.470616895s I1105 01:50:51.046] Nov 5 01:50:51.046: INFO: Pod "azuredisk-volume-tester-qd9pw": Phase="Pending", Reason="", readiness=false. Elapsed: 12.548086604s I1105 01:50:53.154] Nov 5 01:50:53.153: INFO: Pod "azuredisk-volume-tester-qd9pw": Phase="Pending", Reason="", readiness=false. Elapsed: 14.655396946s I1105 01:50:55.231] Nov 5 01:50:55.231: INFO: Pod "azuredisk-volume-tester-qd9pw": Phase="Pending", Reason="", readiness=false. Elapsed: 16.732653553s I1105 01:50:57.309] Nov 5 01:50:57.309: INFO: Pod "azuredisk-volume-tester-qd9pw": Phase="Pending", Reason="", readiness=false. Elapsed: 18.810863467s I1105 01:50:59.387] Nov 5 01:50:59.387: INFO: Pod "azuredisk-volume-tester-qd9pw": Phase="Pending", Reason="", readiness=false. Elapsed: 20.888948081s I1105 01:51:01.465] Nov 5 01:51:01.465: INFO: Pod "azuredisk-volume-tester-qd9pw": Phase="Pending", Reason="", readiness=false. Elapsed: 22.967116695s I1105 01:51:03.543] Nov 5 01:51:03.543: INFO: Pod "azuredisk-volume-tester-qd9pw": Phase="Pending", Reason="", readiness=false. Elapsed: 25.044802105s I1105 01:51:05.621] Nov 5 01:51:05.620: INFO: Pod "azuredisk-volume-tester-qd9pw": Phase="Pending", Reason="", readiness=false. Elapsed: 27.122406615s I1105 01:51:07.699] Nov 5 01:51:07.699: INFO: Pod "azuredisk-volume-tester-qd9pw": Phase="Pending", Reason="", readiness=false. Elapsed: 29.200900031s I1105 01:51:09.778] Nov 5 01:51:09.777: INFO: Pod "azuredisk-volume-tester-qd9pw": Phase="Pending", Reason="", readiness=false. Elapsed: 31.279336147s I1105 01:51:11.855] Nov 5 01:51:11.855: INFO: Pod "azuredisk-volume-tester-qd9pw": Phase="Pending", Reason="", readiness=false. Elapsed: 33.357088458s I1105 01:51:13.933] Nov 5 01:51:13.933: INFO: Pod "azuredisk-volume-tester-qd9pw": Phase="Pending", Reason="", readiness=false. Elapsed: 35.434702267s I1105 01:51:16.016] Nov 5 01:51:16.015: INFO: Pod "azuredisk-volume-tester-qd9pw": Phase="Pending", Reason="", readiness=false. Elapsed: 37.517350016s I1105 01:51:18.095] Nov 5 01:51:18.095: INFO: Pod "azuredisk-volume-tester-qd9pw": Phase="Pending", Reason="", readiness=false. Elapsed: 39.596707939s I1105 01:51:20.173] Nov 5 01:51:20.172: INFO: Pod "azuredisk-volume-tester-qd9pw": Phase="Pending", Reason="", readiness=false. Elapsed: 41.674368548s I1105 01:51:22.250] Nov 5 01:51:22.250: INFO: Pod "azuredisk-volume-tester-qd9pw": Phase="Pending", Reason="", readiness=false. Elapsed: 43.752028058s I1105 01:51:24.329] Nov 5 01:51:24.328: INFO: Pod "azuredisk-volume-tester-qd9pw": Phase="Pending", Reason="", readiness=false. Elapsed: 45.830365772s I1105 01:51:26.408] Nov 5 01:51:26.407: INFO: Pod "azuredisk-volume-tester-qd9pw": Phase="Pending", Reason="", readiness=false. Elapsed: 47.909207491s I1105 01:51:28.485] Nov 5 01:51:28.485: INFO: Pod "azuredisk-volume-tester-qd9pw": Phase="Pending", Reason="", readiness=false. Elapsed: 49.986992701s I1105 01:51:30.563] Nov 5 01:51:30.562: INFO: Pod "azuredisk-volume-tester-qd9pw": Phase="Pending", Reason="", readiness=false. Elapsed: 52.064486709s I1105 01:51:32.640] Nov 5 01:51:32.640: INFO: Pod "azuredisk-volume-tester-qd9pw": Phase="Pending", Reason="", readiness=false. Elapsed: 54.142219019s I1105 01:51:34.718] Nov 5 01:51:34.718: INFO: Pod "azuredisk-volume-tester-qd9pw": Phase="Pending", Reason="", readiness=false. Elapsed: 56.219752427s I1105 01:51:36.796] Nov 5 01:51:36.796: INFO: Pod "azuredisk-volume-tester-qd9pw": Phase="Pending", Reason="", readiness=false. Elapsed: 58.298187642s I1105 01:51:38.874] Nov 5 01:51:38.874: INFO: Pod "azuredisk-volume-tester-qd9pw": Phase="Pending", Reason="", readiness=false. Elapsed: 1m0.375807351s I1105 01:51:40.951] Nov 5 01:51:40.951: INFO: Pod "azuredisk-volume-tester-qd9pw": Phase="Pending", Reason="", readiness=false. Elapsed: 1m2.453323559s I1105 01:51:43.029] Nov 5 01:51:43.029: INFO: Pod "azuredisk-volume-tester-qd9pw": Phase="Pending", Reason="", readiness=false. Elapsed: 1m4.530798666s I1105 01:51:45.138] Nov 5 01:51:45.137: INFO: Pod "azuredisk-volume-tester-qd9pw": Phase="Pending", Reason="", readiness=false. Elapsed: 1m6.639499018s I1105 01:51:47.215] Nov 5 01:51:47.215: INFO: Pod "azuredisk-volume-tester-qd9pw": Phase="Pending", Reason="", readiness=false. Elapsed: 1m8.717160526s I1105 01:51:49.293] Nov 5 01:51:49.293: INFO: Pod "azuredisk-volume-tester-qd9pw": Phase="Pending", Reason="", readiness=false. Elapsed: 1m10.795096237s I1105 01:51:51.371] Nov 5 01:51:51.371: INFO: Pod "azuredisk-volume-tester-qd9pw": Phase="Pending", Reason="", readiness=false. Elapsed: 1m12.873193549s I1105 01:51:53.449] Nov 5 01:51:53.449: INFO: Pod "azuredisk-volume-tester-qd9pw": Phase="Failed", Reason="", readiness=false. Elapsed: 1m14.950682957s I1105 01:51:53.449] STEP: Saw pod failure I1105 01:51:53.449] Nov 5 01:51:53.449: INFO: Pod "azuredisk-volume-tester-qd9pw" satisfied condition "Error status code" I1105 01:51:53.449] STEP: checking that pod logs contain expected message I1105 01:51:53.532] Nov 5 01:51:53.531: INFO: deleting Pod "azuredisk-8276"/"azuredisk-volume-tester-qd9pw" I1105 01:51:53.614] Nov 5 01:51:53.614: INFO: Pod azuredisk-volume-tester-qd9pw has the following logs: /bin/sh: can't create /mnt/test-1/data: Read-only file system I1105 01:51:53.614] I1105 01:51:53.615] STEP: Deleting pod azuredisk-volume-tester-qd9pw in namespace azuredisk-8276 I1105 01:51:53.704] STEP: deleting PV "azuredisk-8276-disk.csi.azure.com-preprovsioned-pv-v648q" I1105 01:51:53.704] Nov 5 01:51:53.704: INFO: Deleting PersistentVolume "azuredisk-8276-disk.csi.azure.com-preprovsioned-pv-v648q" I1105 01:51:53.793] STEP: waiting for claim's PV "azuredisk-8276-disk.csi.azure.com-preprovsioned-pv-v648q" to be deleted I1105 01:51:53.794] Nov 5 01:51:53.793: INFO: Waiting up to 10m0s for PersistentVolume azuredisk-8276-disk.csi.azure.com-preprovsioned-pv-v648q to get deleted I1105 01:51:53.870] Nov 5 01:51:53.870: INFO: PersistentVolume azuredisk-8276-disk.csi.azure.com-preprovsioned-pv-v648q found and phase=Bound (76.859299ms) I1105 01:51:58.948] Nov 5 01:51:58.948: INFO: PersistentVolume azuredisk-8276-disk.csi.azure.com-preprovsioned-pv-v648q found and phase=Bound (5.154448511s) I1105 01:52:04.026] Nov 5 01:52:04.026: INFO: PersistentVolume azuredisk-8276-disk.csi.azure.com-preprovsioned-pv-v648q found and phase=Bound (10.232191624s) I1105 01:52:09.146] Nov 5 01:52:09.146: INFO: PersistentVolume azuredisk-8276-disk.csi.azure.com-preprovsioned-pv-v648q found and phase=Bound (15.352423168s) I1105 01:52:14.224] Nov 5 01:52:14.224: INFO: PersistentVolume azuredisk-8276-disk.csi.azure.com-preprovsioned-pv-v648q found and phase=Bound (20.430309282s) I1105 01:52:19.302] Nov 5 01:52:19.302: INFO: PersistentVolume azuredisk-8276-disk.csi.azure.com-preprovsioned-pv-v648q found and phase=Bound (25.508658798s) I1105 01:52:24.381] Nov 5 01:52:24.380: INFO: PersistentVolume azuredisk-8276-disk.csi.azure.com-preprovsioned-pv-v648q found and phase=Bound (30.587036115s) I1105 01:52:29.459] Nov 5 01:52:29.458: INFO: PersistentVolume azuredisk-8276-disk.csi.azure.com-preprovsioned-pv-v648q found and phase=Bound (35.665028928s) I1105 01:52:34.536] Nov 5 01:52:34.536: INFO: PersistentVolume azuredisk-8276-disk.csi.azure.com-preprovsioned-pv-v648q found and phase=Bound (40.74291834s) I1105 01:52:39.614] Nov 5 01:52:39.614: INFO: PersistentVolume azuredisk-8276-disk.csi.azure.com-preprovsioned-pv-v648q found and phase=Bound (45.820762651s) I1105 01:52:44.692] Nov 5 01:52:44.692: INFO: PersistentVolume azuredisk-8276-disk.csi.azure.com-preprovsioned-pv-v648q found and phase=Bound (50.898542462s) I1105 01:52:49.770] Nov 5 01:52:49.770: INFO: PersistentVolume azuredisk-8276-disk.csi.azure.com-preprovsioned-pv-v648q found and phase=Bound (55.976573074s) I1105 01:52:54.848] Nov 5 01:52:54.848: INFO: PersistentVolume azuredisk-8276-disk.csi.azure.com-preprovsioned-pv-v648q found and phase=Bound (1m1.054652187s) I1105 01:52:59.926] Nov 5 01:52:59.926: INFO: PersistentVolume azuredisk-8276-disk.csi.azure.com-preprovsioned-pv-v648q found and phase=Bound (1m6.132685798s) I1105 01:53:05.004] Nov 5 01:53:05.004: INFO: PersistentVolume azuredisk-8276-disk.csi.azure.com-preprovsioned-pv-v648q found and phase=Bound (1m11.21076031s) I1105 01:53:10.083] Nov 5 01:53:10.083: INFO: PersistentVolume azuredisk-8276-disk.csi.azure.com-preprovsioned-pv-v648q found and phase=Bound (1m16.289204924s) I1105 01:53:15.161] Nov 5 01:53:15.161: INFO: PersistentVolume azuredisk-8276-disk.csi.azure.com-preprovsioned-pv-v648q found and phase=Bound (1m21.367435737s) I1105 01:53:20.239] Nov 5 01:53:20.239: INFO: PersistentVolume azuredisk-8276-disk.csi.azure.com-preprovsioned-pv-v648q found and phase=Bound (1m26.445405547s) I1105 01:53:25.317] Nov 5 01:53:25.317: INFO: PersistentVolume azuredisk-8276-disk.csi.azure.com-preprovsioned-pv-v648q found and phase=Bound (1m31.52376066s) I1105 01:53:30.395] Nov 5 01:53:30.395: INFO: PersistentVolume azuredisk-8276-disk.csi.azure.com-preprovsioned-pv-v648q found and phase=Bound (1m36.601936571s) I1105 01:53:35.474] Nov 5 01:53:35.473: INFO: PersistentVolume azuredisk-8276-disk.csi.azure.com-preprovsioned-pv-v648q found and phase=Bound (1m41.680070881s) I1105 01:53:40.552] Nov 5 01:53:40.551: INFO: PersistentVolume azuredisk-8276-disk.csi.azure.com-preprovsioned-pv-v648q found and phase=Bound (1m46.758152391s) I1105 01:53:45.630] Nov 5 01:53:45.629: INFO: PersistentVolume azuredisk-8276-disk.csi.azure.com-preprovsioned-pv-v648q found and phase=Bound (1m51.836165s) I1105 01:53:50.710] Nov 5 01:53:50.710: INFO: PersistentVolume azuredisk-8276-disk.csi.azure.com-preprovsioned-pv-v648q found and phase=Bound (1m56.916192625s) I1105 01:53:55.788] Nov 5 01:53:55.788: INFO: PersistentVolume azuredisk-8276-disk.csi.azure.com-preprovsioned-pv-v648q found and phase=Bound (2m1.994633436s) I1105 01:54:00.868] Nov 5 01:54:00.868: INFO: PersistentVolume azuredisk-8276-disk.csi.azure.com-preprovsioned-pv-v648q found and phase=Bound (2m7.07462096s) I1105 01:54:05.946] Nov 5 01:54:05.946: INFO: PersistentVolume azuredisk-8276-disk.csi.azure.com-preprovsioned-pv-v648q found and phase=Bound (2m12.152387566s) I1105 01:54:11.024] Nov 5 01:54:11.024: INFO: PersistentVolume azuredisk-8276-disk.csi.azure.com-preprovsioned-pv-v648q found and phase=Bound (2m17.230489374s) I1105 01:54:16.104] Nov 5 01:54:16.103: INFO: PersistentVolume azuredisk-8276-disk.csi.azure.com-preprovsioned-pv-v648q found and phase=Bound (2m22.310085894s) I1105 01:54:21.182] Nov 5 01:54:21.182: INFO: PersistentVolume azuredisk-8276-disk.csi.azure.com-preprovsioned-pv-v648q found and phase=Bound (2m27.388353403s) I1105 01:54:26.260] Nov 5 01:54:26.260: INFO: PersistentVolume azuredisk-8276-disk.csi.azure.com-preprovsioned-pv-v648q found and phase=Bound (2m32.466959515s) I1105 01:54:31.339] Nov 5 01:54:31.339: INFO: PersistentVolume azuredisk-8276-disk.csi.azure.com-preprovsioned-pv-v648q found and phase=Bound (2m37.545294624s) I1105 01:54:36.423] Nov 5 01:54:36.423: INFO: PersistentVolume azuredisk-8276-disk.csi.azure.com-preprovsioned-pv-v648q found and phase=Bound (2m42.629399578s) I1105 01:54:41.501] Nov 5 01:54:41.501: INFO: PersistentVolume azuredisk-8276-disk.csi.azure.com-preprovsioned-pv-v648q found and phase=Bound (2m47.707596286s) I1105 01:54:46.583] Nov 5 01:54:46.583: INFO: PersistentVolume azuredisk-8276-disk.csi.azure.com-preprovsioned-pv-v648q found and phase=Bound (2m52.789473122s) I1105 01:54:51.661] Nov 5 01:54:51.661: INFO: PersistentVolume azuredisk-8276-disk.csi.azure.com-preprovsioned-pv-v648q found and phase=Bound (2m57.867513828s) I1105 01:54:56.739] Nov 5 01:54:56.739: INFO: PersistentVolume azuredisk-8276-disk.csi.azure.com-preprovsioned-pv-v648q found and phase=Bound (3m2.945199631s) I1105 01:55:01.817] Nov 5 01:55:01.817: INFO: PersistentVolume azuredisk-8276-disk.csi.azure.com-preprovsioned-pv-v648q found and phase=Bound (3m8.023283336s) I1105 01:55:06.895] Nov 5 01:55:06.895: INFO: PersistentVolume azuredisk-8276-disk.csi.azure.com-preprovsioned-pv-v648q found and phase=Bound (3m13.101473943s) I1105 01:55:11.973] Nov 5 01:55:11.973: INFO: PersistentVolume azuredisk-8276-disk.csi.azure.com-preprovsioned-pv-v648q found and phase=Bound (3m18.179575948s) I1105 01:55:17.051] Nov 5 01:55:17.051: INFO: PersistentVolume azuredisk-8276-disk.csi.azure.com-preprovsioned-pv-v648q found and phase=Bound (3m23.257653453s) I1105 01:55:22.129] Nov 5 01:55:22.128: INFO: PersistentVolume azuredisk-8276-disk.csi.azure.com-preprovsioned-pv-v648q found and phase=Bound (3m28.335126854s) I1105 01:55:27.206] Nov 5 01:55:27.206: INFO: PersistentVolume azuredisk-8276-disk.csi.azure.com-preprovsioned-pv-v648q found and phase=Bound (3m33.412634454s) I1105 01:55:32.284] Nov 5 01:55:32.284: INFO: PersistentVolume azuredisk-8276-disk.csi.azure.com-preprovsioned-pv-v648q found and phase=Bound (3m38.490197654s) I1105 01:55:37.362] Nov 5 01:55:37.361: INFO: PersistentVolume azuredisk-8276-disk.csi.azure.com-preprovsioned-pv-v648q found and phase=Bound (3m43.567940256s) I1105 01:55:42.439] Nov 5 01:55:42.439: INFO: PersistentVolume azuredisk-8276-disk.csi.azure.com-preprovsioned-pv-v648q found and phase=Bound (3m48.645779958s) I1105 01:55:47.517] Nov 5 01:55:47.517: INFO: PersistentVolume azuredisk-8276-disk.csi.azure.com-preprovsioned-pv-v648q found and phase=Bound (3m53.723573859s) I1105 01:55:52.602] Nov 5 01:55:52.602: INFO: PersistentVolume azuredisk-8276-disk.csi.azure.com-preprovsioned-pv-v648q found and phase=Bound (3m58.808435516s) I1105 01:55:57.680] Nov 5 01:55:57.679: INFO: PersistentVolume azuredisk-8276-disk.csi.azure.com-preprovsioned-pv-v648q found and phase=Bound (4m3.885977915s) I1105 01:56:02.757] Nov 5 01:56:02.757: INFO: PersistentVolume azuredisk-8276-disk.csi.azure.com-preprovsioned-pv-v648q found and phase=Bound (4m8.963415413s) I1105 01:56:07.835] Nov 5 01:56:07.835: INFO: PersistentVolume azuredisk-8276-disk.csi.azure.com-preprovsioned-pv-v648q found and phase=Bound (4m14.041177814s) I1105 01:56:12.912] Nov 5 01:56:12.912: INFO: PersistentVolume azuredisk-8276-disk.csi.azure.com-preprovsioned-pv-v648q found and phase=Bound (4m19.118792913s) I1105 01:56:17.991] Nov 5 01:56:17.991: INFO: PersistentVolume azuredisk-8276-disk.csi.azure.com-preprovsioned-pv-v648q found and phase=Bound (4m24.197817723s) I1105 01:56:23.069] Nov 5 01:56:23.069: INFO: PersistentVolume azuredisk-8276-disk.csi.azure.com-preprovsioned-pv-v648q found and phase=Bound (4m29.27522402s) I1105 01:56:28.146] Nov 5 01:56:28.146: INFO: PersistentVolume azuredisk-8276-disk.csi.azure.com-preprovsioned-pv-v648q found and phase=Bound (4m34.352776918s) I1105 01:56:33.224] Nov 5 01:56:33.224: INFO: PersistentVolume azuredisk-8276-disk.csi.azure.com-preprovsioned-pv-v648q found and phase=Bound (4m39.430288115s) I1105 01:56:38.302] Nov 5 01:56:38.302: INFO: PersistentVolume azuredisk-8276-disk.csi.azure.com-preprovsioned-pv-v648q found and phase=Bound (4m44.508311816s) I1105 01:56:43.380] Nov 5 01:56:43.380: INFO: PersistentVolume azuredisk-8276-disk.csi.azure.com-preprovsioned-pv-v648q found and phase=Bound (4m49.586439218s) I1105 01:56:48.458] Nov 5 01:56:48.458: INFO: PersistentVolume azuredisk-8276-disk.csi.azure.com-preprovsioned-pv-v648q found and phase=Bound (4m54.664171017s) I1105 01:56:53.536] Nov 5 01:56:53.535: INFO: PersistentVolume azuredisk-8276-disk.csi.azure.com-preprovsioned-pv-v648q found and phase=Bound (4m59.742144717s) I1105 01:56:58.614] Nov 5 01:56:58.614: INFO: PersistentVolume azuredisk-8276-disk.csi.azure.com-preprovsioned-pv-v648q found and phase=Bound (5m4.820713622s) I1105 01:57:03.693] Nov 5 01:57:03.693: INFO: PersistentVolume azuredisk-8276-disk.csi.azure.com-preprovsioned-pv-v648q found and phase=Bound (5m9.899248327s) I1105 01:57:08.770] Nov 5 01:57:08.770: INFO: PersistentVolume azuredisk-8276-disk.csi.azure.com-preprovsioned-pv-v648q found and phase=Bound (5m14.976913024s) I1105 01:57:13.849] Nov 5 01:57:13.848: INFO: PersistentVolume azuredisk-8276-disk.csi.azure.com-preprovsioned-pv-v648q found and phase=Bound (5m20.054915124s) I1105 01:57:18.927] Nov 5 01:57:18.927: INFO: PersistentVolume azuredisk-8276-disk.csi.azure.com-preprovsioned-pv-v648q found and phase=Bound (5m25.133254426s) I1105 01:57:24.009] Nov 5 01:57:24.008: INFO: PersistentVolume azuredisk-8276-disk.csi.azure.com-preprovsioned-pv-v648q found and phase=Bound (5m30.214854454s) I1105 01:57:29.117] Nov 5 01:57:29.117: INFO: PersistentVolume azuredisk-8276-disk.csi.azure.com-preprovsioned-pv-v648q found and phase=Bound (5m35.323377391s) I1105 01:57:34.195] Nov 5 01:57:34.195: INFO: PersistentVolume azuredisk-8276-disk.csi.azure.com-preprovsioned-pv-v648q found and phase=Bound (5m40.401647692s) I1105 01:57:39.273] Nov 5 01:57:39.273: INFO: PersistentVolume azuredisk-8276-disk.csi.azure.com-preprovsioned-pv-v648q found and phase=Bound (5m45.479354389s) I1105 01:57:44.351] Nov 5 01:57:44.350: INFO: PersistentVolume azuredisk-8276-disk.csi.azure.com-preprovsioned-pv-v648q found and phase=Bound (5m50.557137686s) I1105 01:57:49.429] Nov 5 01:57:49.429: INFO: PersistentVolume azuredisk-8276-disk.csi.azure.com-preprovsioned-pv-v648q found and phase=Bound (5m55.635340186s) I1105 01:57:54.507] Nov 5 01:57:54.506: INFO: PersistentVolume azuredisk-8276-disk.csi.azure.com-preprovsioned-pv-v648q found and phase=Bound (6m0.713003582s) I1105 01:57:59.585] Nov 5 01:57:59.585: INFO: PersistentVolume azuredisk-8276-disk.csi.azure.com-preprovsioned-pv-v648q found and phase=Bound (6m5.791177082s) I1105 01:58:04.663] Nov 5 01:58:04.662: INFO: PersistentVolume azuredisk-8276-disk.csi.azure.com-preprovsioned-pv-v648q found and phase=Bound (6m10.869100279s) I1105 01:58:09.741] Nov 5 01:58:09.740: INFO: PersistentVolume azuredisk-8276-disk.csi.azure.com-preprovsioned-pv-v648q found and phase=Bound (6m15.946935176s) I1105 01:58:14.818] Nov 5 01:58:14.818: INFO: PersistentVolume azuredisk-8276-disk.csi.azure.com-preprovsioned-pv-v648q found and phase=Bound (6m21.024856373s) I1105 01:58:19.896] Nov 5 01:58:19.896: INFO: PersistentVolume azuredisk-8276-disk.csi.azure.com-preprovsioned-pv-v648q found and phase=Bound (6m26.102458967s) I1105 01:58:24.974] Nov 5 01:58:24.974: INFO: PersistentVolume azuredisk-8276-disk.csi.azure.com-preprovsioned-pv-v648q found and phase=Bound (6m31.180454965s) I1105 01:58:30.056] Nov 5 01:58:30.055: INFO: PersistentVolume azuredisk-8276-disk.csi.azure.com-preprovsioned-pv-v648q found and phase=Bound (6m36.261854989s) I1105 01:58:35.133] Nov 5 01:58:35.133: INFO: PersistentVolume azuredisk-8276-disk.csi.azure.com-preprovsioned-pv-v648q found and phase=Bound (6m41.339392882s) I1105 01:58:40.210] Nov 5 01:58:40.210: INFO: PersistentVolume azuredisk-8276-disk.csi.azure.com-preprovsioned-pv-v648q found and phase=Bound (6m46.416832575s) I1105 01:58:45.289] Nov 5 01:58:45.288: INFO: PersistentVolume azuredisk-8276-disk.csi.azure.com-preprovsioned-pv-v648q found and phase=Bound (6m51.494870872s) I1105 01:58:50.366] Nov 5 01:58:50.366: INFO: PersistentVolume azuredisk-8276-disk.csi.azure.com-preprovsioned-pv-v648q found and phase=Bound (6m56.572705568s) I1105 01:58:55.452] Nov 5 01:58:55.452: INFO: PersistentVolume azuredisk-8276-disk.csi.azure.com-preprovsioned-pv-v648q found and phase=Bound (7m1.658874328s) I1105 01:59:00.530] Nov 5 01:59:00.530: INFO: PersistentVolume azuredisk-8276-disk.csi.azure.com-preprovsioned-pv-v648q found and phase=Bound (7m6.736762923s) I1105 01:59:05.608] Nov 5 01:59:05.608: INFO: PersistentVolume azuredisk-8276-disk.csi.azure.com-preprovsioned-pv-v648q found and phase=Bound (7m11.814653419s) I1105 01:59:10.686] Nov 5 01:59:10.686: INFO: PersistentVolume azuredisk-8276-disk.csi.azure.com-preprovsioned-pv-v648q found and phase=Bound (7m16.892604115s) I1105 01:59:15.764] Nov 5 01:59:15.764: INFO: PersistentVolume azuredisk-8276-disk.csi.azure.com-preprovsioned-pv-v648q found and phase=Bound (7m21.970389809s) I1105 01:59:20.842] Nov 5 01:59:20.842: INFO: PersistentVolume azuredisk-8276-disk.csi.azure.com-preprovsioned-pv-v648q found and phase=Bound (7m27.048578606s) I1105 01:59:25.920] Nov 5 01:59:25.920: INFO: PersistentVolume azuredisk-8276-disk.csi.azure.com-preprovsioned-pv-v648q found and phase=Bound (7m32.126206599s) I1105 01:59:30.998] Nov 5 01:59:30.997: INFO: PersistentVolume azuredisk-8276-disk.csi.azure.com-preprovsioned-pv-v648q found and phase=Bound (7m37.203959993s) I1105 01:59:36.083] Nov 5 01:59:36.082: INFO: PersistentVolume azuredisk-8276-disk.csi.azure.com-preprovsioned-pv-v648q found and phase=Bound (7m42.289036443s) I1105 01:59:41.160] Nov 5 01:59:41.160: INFO: PersistentVolume azuredisk-8276-disk.csi.azure.com-preprovsioned-pv-v648q found and phase=Bound (7m47.366393434s) I1105 01:59:46.238] Nov 5 01:59:46.237: INFO: PersistentVolume azuredisk-8276-disk.csi.azure.com-preprovsioned-pv-v648q found and phase=Bound (7m52.444113627s) I1105 01:59:51.315] Nov 5 01:59:51.315: INFO: PersistentVolume azuredisk-8276-disk.csi.azure.com-preprovsioned-pv-v648q found and phase=Bound (7m57.521442516s) I1105 01:59:56.393] Nov 5 01:59:56.393: INFO: PersistentVolume azuredisk-8276-disk.csi.azure.com-preprovsioned-pv-v648q found and phase=Bound (8m2.599201109s) I1105 02:00:01.470] Nov 5 02:00:01.470: INFO: PersistentVolume azuredisk-8276-disk.csi.azure.com-preprovsioned-pv-v648q found and phase=Bound (8m7.676612s) I1105 02:00:06.548] Nov 5 02:00:06.547: INFO: PersistentVolume azuredisk-8276-disk.csi.azure.com-preprovsioned-pv-v648q found and phase=Bound (8m12.75410309s) I1105 02:00:11.625] Nov 5 02:00:11.625: INFO: PersistentVolume azuredisk-8276-disk.csi.azure.com-preprovsioned-pv-v648q found and phase=Bound (8m17.831661881s) I1105 02:00:16.703] Nov 5 02:00:16.703: INFO: PersistentVolume azuredisk-8276-disk.csi.azure.com-preprovsioned-pv-v648q found and phase=Bound (8m22.909519675s) I1105 02:00:21.785] Nov 5 02:00:21.785: INFO: PersistentVolume azuredisk-8276-disk.csi.azure.com-preprovsioned-pv-v648q found and phase=Bound (8m27.991671801s) I1105 02:00:26.863] Nov 5 02:00:26.863: INFO: PersistentVolume azuredisk-8276-disk.csi.azure.com-preprovsioned-pv-v648q found and phase=Bound (8m33.069346693s) I1105 02:00:31.950] Nov 5 02:00:31.950: INFO: PersistentVolume azuredisk-8276-disk.csi.azure.com-preprovsioned-pv-v648q found and phase=Bound (8m38.156625059s) I1105 02:00:37.028] Nov 5 02:00:37.028: INFO: PersistentVolume azuredisk-8276-disk.csi.azure.com-preprovsioned-pv-v648q found and phase=Bound (8m43.234355851s) I1105 02:00:42.105] Nov 5 02:00:42.105: INFO: PersistentVolume azuredisk-8276-disk.csi.azure.com-preprovsioned-pv-v648q found and phase=Bound (8m48.31179374s) I1105 02:00:47.183] Nov 5 02:00:47.182: INFO: PersistentVolume azuredisk-8276-disk.csi.azure.com-preprovsioned-pv-v648q found and phase=Bound (8m53.389157229s) I1105 02:00:52.260] Nov 5 02:00:52.260: INFO: PersistentVolume azuredisk-8276-disk.csi.azure.com-preprovsioned-pv-v648q found and phase=Bound (8m58.466887221s) I1105 02:00:57.338] Nov 5 02:00:57.338: INFO: PersistentVolume azuredisk-8276-disk.csi.azure.com-preprovsioned-pv-v648q found and phase=Bound (9m3.544720113s) I1105 02:01:02.416] Nov 5 02:01:02.416: INFO: PersistentVolume azuredisk-8276-disk.csi.azure.com-preprovsioned-pv-v648q found and phase=Bound (9m8.622377203s) I1105 02:01:07.493] Nov 5 02:01:07.493: INFO: PersistentVolume azuredisk-8276-disk.csi.azure.com-preprovsioned-pv-v648q found and phase=Bound (9m13.69957529s) I1105 02:01:12.570] Nov 5 02:01:12.570: INFO: PersistentVolume azuredisk-8276-disk.csi.azure.com-preprovsioned-pv-v648q found and phase=Bound (9m18.776516775s) I1105 02:01:17.648] Nov 5 02:01:17.648: INFO: PersistentVolume azuredisk-8276-disk.csi.azure.com-preprovsioned-pv-v648q found and phase=Bound (9m23.854507168s) I1105 02:01:22.726] Nov 5 02:01:22.726: INFO: PersistentVolume azuredisk-8276-disk.csi.azure.com-preprovsioned-pv-v648q found and phase=Bound (9m28.93245616s) I1105 02:01:27.803] Nov 5 02:01:27.803: INFO: PersistentVolume azuredisk-8276-disk.csi.azure.com-preprovsioned-pv-v648q found and phase=Bound (9m34.009823848s) I1105 02:01:32.882] Nov 5 02:01:32.881: INFO: PersistentVolume azuredisk-8276-disk.csi.azure.com-preprovsioned-pv-v648q found and phase=Bound (9m39.088122643s) I1105 02:01:37.960] Nov 5 02:01:37.959: INFO: PersistentVolume azuredisk-8276-disk.csi.azure.com-preprovsioned-pv-v648q found and phase=Bound (9m44.166164436s) I1105 02:01:43.038] Nov 5 02:01:43.038: INFO: PersistentVolume azuredisk-8276-disk.csi.azure.com-preprovsioned-pv-v648q found and phase=Bound (9m49.244726333s) I1105 02:01:48.122] Nov 5 02:01:48.122: INFO: PersistentVolume azuredisk-8276-disk.csi.azure.com-preprovsioned-pv-v648q found and phase=Bound (9m54.328293269s) I1105 02:01:53.200] Nov 5 02:01:53.199: INFO: PersistentVolume azuredisk-8276-disk.csi.azure.com-preprovsioned-pv-v648q found and phase=Bound (9m59.40612686s) I1105 02:01:58.200] Nov 5 02:01:58.200: INFO: Unexpected error occurred: PersistentVolume azuredisk-8276-disk.csi.azure.com-preprovsioned-pv-v648q still exists within 10m0s I1105 02:01:58.200] [AfterEach] [azuredisk-csi-e2e] [single-az] Pre-Provisioned I1105 02:01:58.201] /go/src/sigs.k8s.io/azuredisk-csi-driver/vendor/k8s.io/kubernetes/test/e2e/framework/framework.go:151 I1105 02:01:58.201] STEP: Collecting events from namespace "azuredisk-8276". I1105 02:01:58.278] STEP: Found 7 events. I1105 02:01:58.279] Nov 5 02:01:58.278: INFO: At 2019-11-05 01:50:38 +0000 UTC - event for azuredisk-volume-tester-qd9pw: {default-scheduler } Scheduled: Successfully assigned azuredisk-8276/azuredisk-volume-tester-qd9pw to k8s-agentpool-15826033-vmss000000 I1105 02:01:58.279] Nov 5 02:01:58.278: INFO: At 2019-11-05 01:50:53 +0000 UTC - event for azuredisk-volume-tester-qd9pw: {attachdetach-controller } FailedAttachVolume: AttachVolume.Attach failed for volume "azuredisk-8276-disk.csi.azure.com-preprovsioned-pv-v648q" : attachment timeout for volume /subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5/providers/Microsoft.Compute/disks/pre-provisioned-readOnly I1105 02:01:58.279] Nov 5 02:01:58.278: INFO: At 2019-11-05 01:51:24 +0000 UTC - event for azuredisk-volume-tester-qd9pw: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "azuredisk-8276-disk.csi.azure.com-preprovsioned-pv-v648q" I1105 02:01:58.279] Nov 5 02:01:58.278: INFO: At 2019-11-05 01:51:50 +0000 UTC - event for azuredisk-volume-tester-qd9pw: {kubelet k8s-agentpool-15826033-vmss000000} Pulling: Pulling image "docker.io/library/busybox:1.29" I1105 02:01:58.280] Nov 5 02:01:58.278: INFO: At 2019-11-05 01:51:50 +0000 UTC - event for azuredisk-volume-tester-qd9pw: {kubelet k8s-agentpool-15826033-vmss000000} Pulled: Successfully pulled image "docker.io/library/busybox:1.29" I1105 02:01:58.280] Nov 5 02:01:58.278: INFO: At 2019-11-05 01:51:52 +0000 UTC - event for azuredisk-volume-tester-qd9pw: {kubelet k8s-agentpool-15826033-vmss000000} Created: Created container volume-tester I1105 02:01:58.280] Nov 5 02:01:58.278: INFO: At 2019-11-05 01:51:52 +0000 UTC - event for azuredisk-volume-tester-qd9pw: {kubelet k8s-agentpool-15826033-vmss000000} Started: Started container volume-tester I1105 02:01:58.355] Nov 5 02:01:58.355: INFO: POD NODE PHASE GRACE CONDITIONS I1105 02:01:58.355] Nov 5 02:01:58.355: INFO: I1105 02:01:58.584] Nov 5 02:01:58.584: INFO: I1105 02:01:58.584] Logging node info for node k8s-agentpool-15826033-vmss000000 I1105 02:01:58.666] Nov 5 02:01:58.663: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-agentpool-15826033-vmss000000,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-agentpool-15826033-vmss000000,UID:15fa874e-54ec-42cd-b243-d3baa512150f,ResourceVersion:5853,Generation:0,CreationTimestamp:2019-11-05 01:34:17 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{agentpool: agentpool,beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_D4s_v3,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 0,kubernetes.azure.com/cluster: kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5,kubernetes.azure.com/role: agent,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-agentpool-15826033-vmss000000,kubernetes.io/os: linux,kubernetes.io/role: agent,node-role.kubernetes.io/agent: ,storageprofile: managed,storagetier: Premium_LRS,topology.disk.csi.azure.com/zone: 0,},Annotations:map[string]string{csi.volume.kubernetes.io/nodeid: {"disk.csi.azure.com":"k8s-agentpool-15826033-vmss000000"},node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.2.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool-15826033-vmss/virtualMachines/0,Unschedulable:false,Taints:[],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-csi-disk.csi.azure.com: {{8 0} {} 8 DecimalSI},cpu: {{4 0} {} 4 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{16797044736 0} {} 16403364Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-csi-disk.csi.azure.com: {{8 0} {} 8 DecimalSI},cpu: {{4 0} {} 4 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{16010612736 0} {} 15635364Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 01:34:56 +0000 UTC 2019-11-05 01:34:56 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 02:01:21 +0000 UTC 2019-11-05 01:34:17 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 02:01:21 +0000 UTC 2019-11-05 01:34:17 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 02:01:21 +0000 UTC 2019-11-05 01:34:17 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 02:01:21 +0000 UTC 2019-11-05 01:34:29 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.0.4} {Hostname k8s-agentpool-15826033-vmss000000}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:afe00792177d49a98f6a44d03d08732d,SystemUUID:4F1777BF-A5C3-004E-81C4-90F0B54C02A0,BootID:13b6a628-ae89-44af-b891-89461030adcb,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[k8sprowinternal.azurecr.io/azuredisk-csi@sha256:75d4ff5d7cd845d6e810c67796ecb037e852fb849ec1eb5b3194564008e6d1e0 k8sprowinternal.azurecr.io/azuredisk-csi:e2e-3e3ed045c6faa509dbd8d2890ee9f56aa2934887] 286930414} {[k8s.gcr.io/kubernetes-dashboard-amd64@sha256:0ae6b69432e78069c5ce2bcde0fe409c5c4d6f0f4d9cd50a17974fea38898747 k8s.gcr.io/kubernetes-dashboard-amd64:v1.10.1] 121711221} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/metrics-server-amd64@sha256:49a9f12f7067d11f42c803dbe61ed2c1299959ad85cb315b25ff7eef8e6b8892 k8s.gcr.io/metrics-server-amd64:v0.2.1] 42541759} {[mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume@sha256:4fd30d43947d4a54fc89ead7985beecfd3c9b2a93a0655a373b1608ab90bd5af mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume:v0.0.7] 22909487} {[quay.io/k8scsi/livenessprobe@sha256:8c9c8a0f330acc2e558ae6624309447d02522e76aa35af23fa19b88abf4381cb quay.io/k8scsi/livenessprobe:v1.0.2] 17415381} {[quay.io/k8scsi/csi-node-driver-registrar@sha256:13daf82fb99e951a4bff8ae5fc7c17c3a8fe7130be6400990d8f6076c32d4599 quay.io/k8scsi/csi-node-driver-registrar:v1.1.0] 15815995} {[mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume@sha256:23d8c6033f02a1ecad05127ebdc931bb871264228661bc122704b0974e4d9fdd mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume:1.0.8] 1159025} {[busybox@sha256:8ccbac733d19c0dd4d70b4f0c1e12245b5fa3ad24758a11035ee505c629c0796 busybox:1.29] 1154361} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 02:01:58.666] Nov 5 02:01:58.663: INFO: I1105 02:01:58.667] Logging kubelet events for node k8s-agentpool-15826033-vmss000000 I1105 02:01:58.745] Nov 5 02:01:58.744: INFO: I1105 02:01:58.745] Logging pods the kubelet thinks is on node k8s-agentpool-15826033-vmss000000 I1105 02:01:58.831] Nov 5 02:01:58.830: INFO: kube-proxy-wxblc started at 2019-11-05 01:34:34 +0000 UTC (0+1 container statuses recorded) I1105 02:01:58.831] Nov 5 02:01:58.830: INFO: Container kube-proxy ready: true, restart count 0 I1105 02:01:58.831] Nov 5 02:01:58.830: INFO: kubernetes-dashboard-66dd8b8df7-ks284 started at 2019-11-05 01:34:35 +0000 UTC (0+1 container statuses recorded) I1105 02:01:58.831] Nov 5 02:01:58.830: INFO: Container kubernetes-dashboard ready: true, restart count 0 I1105 02:01:58.831] Nov 5 02:01:58.830: INFO: csi-azuredisk-node-whq2g started at 2019-11-05 01:38:24 +0000 UTC (0+3 container statuses recorded) I1105 02:01:58.831] Nov 5 02:01:58.830: INFO: Container azuredisk ready: true, restart count 0 I1105 02:01:58.832] Nov 5 02:01:58.830: INFO: Container liveness-probe ready: true, restart count 0 I1105 02:01:58.832] Nov 5 02:01:58.830: INFO: Container node-driver-registrar ready: true, restart count 0 I1105 02:01:58.832] Nov 5 02:01:58.830: INFO: blobfuse-flexvol-installer-bsvqt started at 2019-11-05 01:34:34 +0000 UTC (0+1 container statuses recorded) I1105 02:01:58.832] Nov 5 02:01:58.830: INFO: Container blobfuse-flexvol-installer ready: true, restart count 0 I1105 02:01:58.832] Nov 5 02:01:58.830: INFO: azure-ip-masq-agent-vx86b started at 2019-11-05 01:34:34 +0000 UTC (0+1 container statuses recorded) I1105 02:01:58.832] Nov 5 02:01:58.830: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 02:01:58.833] Nov 5 02:01:58.830: INFO: keyvault-flexvolume-qhnfr started at 2019-11-05 01:34:34 +0000 UTC (0+1 container statuses recorded) I1105 02:01:58.833] Nov 5 02:01:58.830: INFO: Container keyvault-flexvolume ready: true, restart count 0 I1105 02:01:58.833] Nov 5 02:01:58.830: INFO: metrics-server-864ffbc5c-sdxps started at 2019-11-05 01:34:34 +0000 UTC (0+1 container statuses recorded) I1105 02:01:58.833] Nov 5 02:01:58.830: INFO: Container metrics-server ready: true, restart count 0 I1105 02:01:59.084] Nov 5 02:01:59.084: INFO: I1105 02:01:59.085] Latency metrics for node k8s-agentpool-15826033-vmss000000 I1105 02:01:59.085] Nov 5 02:01:59.084: INFO: I1105 02:01:59.085] Logging node info for node k8s-agentpool-15826033-vmss000001 I1105 02:01:59.165] Nov 5 02:01:59.162: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-agentpool-15826033-vmss000001,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-agentpool-15826033-vmss000001,UID:4a8ea605-cc0b-4e3b-9060-d8867d269e6c,ResourceVersion:5916,Generation:0,CreationTimestamp:2019-11-05 01:34:09 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{agentpool: agentpool,beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_D4s_v3,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 1,kubernetes.azure.com/cluster: kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5,kubernetes.azure.com/role: agent,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-agentpool-15826033-vmss000001,kubernetes.io/os: linux,kubernetes.io/role: agent,node-role.kubernetes.io/agent: ,storageprofile: managed,storagetier: Premium_LRS,topology.disk.csi.azure.com/zone: 1,},Annotations:map[string]string{csi.volume.kubernetes.io/nodeid: {"disk.csi.azure.com":"k8s-agentpool-15826033-vmss000001"},node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.0.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool-15826033-vmss/virtualMachines/1,Unschedulable:false,Taints:[],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-csi-disk.csi.azure.com: {{8 0} {} 8 DecimalSI},cpu: {{4 0} {} 4 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{16797044736 0} {} 16403364Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-csi-disk.csi.azure.com: {{8 0} {} 8 DecimalSI},cpu: {{4 0} {} 4 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{16010612736 0} {} 15635364Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 01:34:57 +0000 UTC 2019-11-05 01:34:57 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 02:01:43 +0000 UTC 2019-11-05 01:34:08 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 02:01:43 +0000 UTC 2019-11-05 01:34:08 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 02:01:43 +0000 UTC 2019-11-05 01:34:08 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 02:01:43 +0000 UTC 2019-11-05 01:34:29 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.0.5} {Hostname k8s-agentpool-15826033-vmss000001}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:b6b9fbf4a2fd4fa5a7ae472415fe30b8,SystemUUID:0162B3A4-3955-9A46-ABD4-A37BD75848CC,BootID:ee93003f-d47c-4118-9975-57365b3443a5,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[k8sprowinternal.azurecr.io/azuredisk-csi@sha256:75d4ff5d7cd845d6e810c67796ecb037e852fb849ec1eb5b3194564008e6d1e0 k8sprowinternal.azurecr.io/azuredisk-csi:e2e-3e3ed045c6faa509dbd8d2890ee9f56aa2934887] 286930414} {[gcr.io/kubernetes-helm/tiller@sha256:f6d8f4ab9ba993b5f5b60a6edafe86352eabe474ffeb84cb6c79b8866dce45d1 gcr.io/kubernetes-helm/tiller:v2.11.0] 71821984} {[quay.io/k8scsi/csi-provisioner@sha256:d657c839dce87324fe2b677302913f9386f885f8746be7bea0ced5b0844e3433 quay.io/k8scsi/csi-provisioner:v1.3.1] 50596998} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[quay.io/k8scsi/csi-attacher@sha256:26fccd7a99d973845df1193b46ebdcc6ab8dc5f6e6be319750c471fce1742d13 quay.io/k8scsi/csi-attacher:v1.2.0] 46226754} {[quay.io/k8scsi/csi-cluster-driver-registrar@sha256:fafd75ae5442f192cfa8c2e792903aee30d5884b62e802e4464b0a895d21e3ef quay.io/k8scsi/csi-cluster-driver-registrar:v1.0.1] 45874691} {[quay.io/k8scsi/csi-snapshotter@sha256:a49e0da1af6f2bf717e41ba1eee8b5e6a1cbd66a709dd92cc43fe475fe2589eb quay.io/k8scsi/csi-snapshotter:v1.1.0] 44325352} {[k8s.gcr.io/coredns@sha256:e83beb5e43f8513fa735e77ffc5859640baea30a882a11cc75c4c3244a737d3c k8s.gcr.io/coredns:1.5.0] 42488424} {[mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume@sha256:4fd30d43947d4a54fc89ead7985beecfd3c9b2a93a0655a373b1608ab90bd5af mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume:v0.0.7] 22909487} {[quay.io/k8scsi/livenessprobe@sha256:8c9c8a0f330acc2e558ae6624309447d02522e76aa35af23fa19b88abf4381cb quay.io/k8scsi/livenessprobe:v1.0.2] 17415381} {[quay.io/k8scsi/csi-node-driver-registrar@sha256:13daf82fb99e951a4bff8ae5fc7c17c3a8fe7130be6400990d8f6076c32d4599 quay.io/k8scsi/csi-node-driver-registrar:v1.1.0] 15815995} {[quay.io/k8scsi/livenessprobe@sha256:dde617756e0f602adc566ab71fd885f1dad451ad3fb063ac991c95a2ff47aea5 quay.io/k8scsi/livenessprobe:v1.1.0] 14967303} {[mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume@sha256:23d8c6033f02a1ecad05127ebdc931bb871264228661bc122704b0974e4d9fdd mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume:1.0.8] 1159025} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 02:01:59.165] Nov 5 02:01:59.162: INFO: I1105 02:01:59.165] Logging kubelet events for node k8s-agentpool-15826033-vmss000001 I1105 02:01:59.242] Nov 5 02:01:59.242: INFO: I1105 02:01:59.243] Logging pods the kubelet thinks is on node k8s-agentpool-15826033-vmss000001 I1105 02:01:59.326] Nov 5 02:01:59.326: INFO: tiller-deploy-54c96cb5df-8d9fs started at 2019-11-05 01:34:35 +0000 UTC (0+1 container statuses recorded) I1105 02:01:59.326] Nov 5 02:01:59.326: INFO: Container tiller ready: true, restart count 0 I1105 02:01:59.327] Nov 5 02:01:59.326: INFO: csi-azuredisk-node-ghv5w started at 2019-11-05 01:38:24 +0000 UTC (0+3 container statuses recorded) I1105 02:01:59.327] Nov 5 02:01:59.326: INFO: Container azuredisk ready: true, restart count 0 I1105 02:01:59.327] Nov 5 02:01:59.326: INFO: Container liveness-probe ready: true, restart count 0 I1105 02:01:59.327] Nov 5 02:01:59.326: INFO: Container node-driver-registrar ready: true, restart count 0 I1105 02:01:59.327] Nov 5 02:01:59.326: INFO: csi-azuredisk-controller-6f6587bfc6-8vkfl started at 2019-11-05 01:38:24 +0000 UTC (0+6 container statuses recorded) I1105 02:01:59.327] Nov 5 02:01:59.326: INFO: Container azuredisk ready: true, restart count 0 I1105 02:01:59.328] Nov 5 02:01:59.326: INFO: Container cluster-driver-registrar ready: true, restart count 0 I1105 02:01:59.328] Nov 5 02:01:59.326: INFO: Container csi-attacher ready: true, restart count 0 I1105 02:01:59.328] Nov 5 02:01:59.326: INFO: Container csi-provisioner ready: true, restart count 0 I1105 02:01:59.328] Nov 5 02:01:59.326: INFO: Container csi-snapshotter ready: true, restart count 0 I1105 02:01:59.328] Nov 5 02:01:59.326: INFO: Container liveness-probe ready: true, restart count 0 I1105 02:01:59.328] Nov 5 02:01:59.326: INFO: blobfuse-flexvol-installer-rw5dw started at 2019-11-05 01:34:35 +0000 UTC (0+1 container statuses recorded) I1105 02:01:59.328] Nov 5 02:01:59.326: INFO: Container blobfuse-flexvol-installer ready: true, restart count 0 I1105 02:01:59.329] Nov 5 02:01:59.326: INFO: coredns-7f68dcdbdb-jt522 started at 2019-11-05 01:34:35 +0000 UTC (0+1 container statuses recorded) I1105 02:01:59.329] Nov 5 02:01:59.326: INFO: Container coredns ready: true, restart count 0 I1105 02:01:59.329] Nov 5 02:01:59.326: INFO: azure-ip-masq-agent-6c6vn started at 2019-11-05 01:34:35 +0000 UTC (0+1 container statuses recorded) I1105 02:01:59.329] Nov 5 02:01:59.326: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 02:01:59.329] Nov 5 02:01:59.326: INFO: keyvault-flexvolume-rdlcj started at 2019-11-05 01:34:35 +0000 UTC (0+1 container statuses recorded) I1105 02:01:59.329] Nov 5 02:01:59.326: INFO: Container keyvault-flexvolume ready: true, restart count 0 I1105 02:01:59.330] Nov 5 02:01:59.326: INFO: kube-proxy-6kh57 started at 2019-11-05 01:34:35 +0000 UTC (0+1 container statuses recorded) I1105 02:01:59.330] Nov 5 02:01:59.326: INFO: Container kube-proxy ready: true, restart count 0 I1105 02:01:59.574] Nov 5 02:01:59.574: INFO: I1105 02:01:59.574] Latency metrics for node k8s-agentpool-15826033-vmss000001 I1105 02:01:59.574] Nov 5 02:01:59.574: INFO: I1105 02:01:59.575] Logging node info for node k8s-master-15826033-0 I1105 02:01:59.654] Nov 5 02:01:59.652: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-15826033-0,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-15826033-0,UID:aceb0231-4645-4a9a-b296-90ada851986b,ResourceVersion:5879,Generation:0,CreationTimestamp:2019-11-05 01:35:06 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 1,kubernetes.azure.com/cluster: kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-15826033-0,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.4.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5/providers/Microsoft.Compute/virtualMachines/k8s-master-15826033-0,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 01:35:20 +0000 UTC 2019-11-05 01:35:20 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 02:01:30 +0000 UTC 2019-11-05 01:34:57 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 02:01:30 +0000 UTC 2019-11-05 01:34:57 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 02:01:30 +0000 UTC 2019-11-05 01:34:57 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 02:01:30 +0000 UTC 2019-11-05 01:35:07 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.255.5} {Hostname k8s-master-15826033-0}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:1915756c22a540dc8eb30493987e9a43,SystemUUID:742C969C-5A71-D744-8E61-F27E38B56EB2,BootID:03f0672e-8fa9-4309-956f-8e3c371902c3,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 02:01:59.654] Nov 5 02:01:59.652: INFO: I1105 02:01:59.654] Logging kubelet events for node k8s-master-15826033-0 I1105 02:01:59.732] Nov 5 02:01:59.732: INFO: I1105 02:01:59.732] Logging pods the kubelet thinks is on node k8s-master-15826033-0 I1105 02:01:59.814] Nov 5 02:01:59.814: INFO: cloud-controller-manager-k8s-master-15826033-0 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:01:59.814] Nov 5 02:01:59.814: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 02:01:59.814] Nov 5 02:01:59.814: INFO: kube-addon-manager-k8s-master-15826033-0 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:01:59.814] Nov 5 02:01:59.814: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 02:01:59.815] Nov 5 02:01:59.814: INFO: kube-apiserver-k8s-master-15826033-0 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:01:59.815] Nov 5 02:01:59.814: INFO: Container kube-apiserver ready: true, restart count 0 I1105 02:01:59.815] Nov 5 02:01:59.814: INFO: kube-controller-manager-k8s-master-15826033-0 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:01:59.815] Nov 5 02:01:59.814: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 02:01:59.815] Nov 5 02:01:59.814: INFO: kube-scheduler-k8s-master-15826033-0 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:01:59.815] Nov 5 02:01:59.814: INFO: Container kube-scheduler ready: true, restart count 0 I1105 02:01:59.815] Nov 5 02:01:59.814: INFO: azure-ip-masq-agent-wgdpx started at 2019-11-05 01:35:06 +0000 UTC (0+1 container statuses recorded) I1105 02:01:59.816] Nov 5 02:01:59.814: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 02:01:59.816] Nov 5 02:01:59.814: INFO: kube-proxy-7d2px started at 2019-11-05 01:35:06 +0000 UTC (0+1 container statuses recorded) I1105 02:01:59.816] Nov 5 02:01:59.814: INFO: Container kube-proxy ready: true, restart count 0 I1105 02:02:00.068] Nov 5 02:02:00.067: INFO: I1105 02:02:00.068] Latency metrics for node k8s-master-15826033-0 I1105 02:02:00.068] Nov 5 02:02:00.067: INFO: I1105 02:02:00.068] Logging node info for node k8s-master-15826033-1 I1105 02:02:00.147] Nov 5 02:02:00.144: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-15826033-1,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-15826033-1,UID:1a13ec61-c89a-456c-8c94-bfc8e717c2e7,ResourceVersion:5881,Generation:0,CreationTimestamp:2019-11-05 01:35:07 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 1,kubernetes.azure.com/cluster: kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-15826033-1,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.5.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5/providers/Microsoft.Compute/virtualMachines/k8s-master-15826033-1,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 01:35:31 +0000 UTC 2019-11-05 01:35:31 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 02:01:31 +0000 UTC 2019-11-05 01:35:00 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 02:01:31 +0000 UTC 2019-11-05 01:35:00 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 02:01:31 +0000 UTC 2019-11-05 01:35:00 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 02:01:31 +0000 UTC 2019-11-05 01:35:07 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.255.6} {Hostname k8s-master-15826033-1}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:e0f2bccddd5b4555b180d14081ab59a8,SystemUUID:AE126631-CAD2-B54F-B22E-1F499342A4D6,BootID:509b9a9d-b046-4861-aa35-8bcb77465de2,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 02:02:00.147] Nov 5 02:02:00.144: INFO: I1105 02:02:00.147] Logging kubelet events for node k8s-master-15826033-1 I1105 02:02:00.226] Nov 5 02:02:00.226: INFO: I1105 02:02:00.226] Logging pods the kubelet thinks is on node k8s-master-15826033-1 I1105 02:02:00.308] Nov 5 02:02:00.307: INFO: cloud-controller-manager-k8s-master-15826033-1 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:02:00.308] Nov 5 02:02:00.307: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 02:02:00.308] Nov 5 02:02:00.307: INFO: kube-addon-manager-k8s-master-15826033-1 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:02:00.308] Nov 5 02:02:00.308: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 02:02:00.308] Nov 5 02:02:00.308: INFO: kube-apiserver-k8s-master-15826033-1 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:02:00.309] Nov 5 02:02:00.308: INFO: Container kube-apiserver ready: true, restart count 0 I1105 02:02:00.309] Nov 5 02:02:00.308: INFO: kube-controller-manager-k8s-master-15826033-1 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:02:00.309] Nov 5 02:02:00.308: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 02:02:00.309] Nov 5 02:02:00.308: INFO: kube-scheduler-k8s-master-15826033-1 started at 2019-11-05 01:36:16 +0000 UTC (0+1 container statuses recorded) I1105 02:02:00.309] Nov 5 02:02:00.308: INFO: Container kube-scheduler ready: true, restart count 0 I1105 02:02:00.309] Nov 5 02:02:00.308: INFO: azure-ip-masq-agent-w6dpr started at 2019-11-05 01:35:10 +0000 UTC (0+1 container statuses recorded) I1105 02:02:00.309] Nov 5 02:02:00.308: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 02:02:00.310] Nov 5 02:02:00.308: INFO: kube-proxy-8kmr5 started at 2019-11-05 01:35:10 +0000 UTC (0+1 container statuses recorded) I1105 02:02:00.310] Nov 5 02:02:00.308: INFO: Container kube-proxy ready: true, restart count 0 I1105 02:02:00.553] Nov 5 02:02:00.553: INFO: I1105 02:02:00.553] Latency metrics for node k8s-master-15826033-1 I1105 02:02:00.553] Nov 5 02:02:00.553: INFO: I1105 02:02:00.553] Logging node info for node k8s-master-15826033-2 I1105 02:02:00.633] Nov 5 02:02:00.630: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-15826033-2,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-15826033-2,UID:acb3a851-d6df-4549-ba56-4b7e96553c17,ResourceVersion:5799,Generation:0,CreationTimestamp:2019-11-05 01:34:09 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 2,kubernetes.azure.com/cluster: kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-15826033-2,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.1.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5/providers/Microsoft.Compute/virtualMachines/k8s-master-15826033-2,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 01:34:44 +0000 UTC 2019-11-05 01:34:44 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 02:01:02 +0000 UTC 2019-11-05 01:34:07 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 02:01:02 +0000 UTC 2019-11-05 01:34:07 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 02:01:02 +0000 UTC 2019-11-05 01:34:07 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 02:01:02 +0000 UTC 2019-11-05 01:34:29 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{Hostname k8s-master-15826033-2} {InternalIP 10.240.255.7}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:0885b0c2f6174106bde4fa0eae2cbc6b,SystemUUID:C176121C-5D7A-6243-9B05-8B68F8022B64,BootID:30430638-7ab6-4761-b984-2387e1a3f5b2,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 02:02:00.633] Nov 5 02:02:00.630: INFO: I1105 02:02:00.633] Logging kubelet events for node k8s-master-15826033-2 I1105 02:02:00.711] Nov 5 02:02:00.711: INFO: I1105 02:02:00.711] Logging pods the kubelet thinks is on node k8s-master-15826033-2 I1105 02:02:00.793] Nov 5 02:02:00.793: INFO: cloud-controller-manager-k8s-master-15826033-2 started at 2019-11-05 01:33:54 +0000 UTC (0+1 container statuses recorded) I1105 02:02:00.793] Nov 5 02:02:00.793: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 02:02:00.794] Nov 5 02:02:00.793: INFO: kube-addon-manager-k8s-master-15826033-2 started at 2019-11-05 01:33:54 +0000 UTC (0+1 container statuses recorded) I1105 02:02:00.794] Nov 5 02:02:00.793: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 02:02:00.794] Nov 5 02:02:00.793: INFO: kube-apiserver-k8s-master-15826033-2 started at 2019-11-05 01:33:54 +0000 UTC (0+1 container statuses recorded) I1105 02:02:00.794] Nov 5 02:02:00.793: INFO: Container kube-apiserver ready: true, restart count 0 I1105 02:02:00.795] Nov 5 02:02:00.793: INFO: kube-controller-manager-k8s-master-15826033-2 started at 2019-11-05 01:33:54 +0000 UTC (0+1 container statuses recorded) I1105 02:02:00.795] Nov 5 02:02:00.793: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 02:02:00.795] Nov 5 02:02:00.793: INFO: kube-scheduler-k8s-master-15826033-2 started at 2019-11-05 01:33:54 +0000 UTC (0+1 container statuses recorded) I1105 02:02:00.796] Nov 5 02:02:00.793: INFO: Container kube-scheduler ready: true, restart count 0 I1105 02:02:00.796] Nov 5 02:02:00.793: INFO: azure-ip-masq-agent-hw7s5 started at 2019-11-05 01:34:34 +0000 UTC (0+1 container statuses recorded) I1105 02:02:00.796] Nov 5 02:02:00.793: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 02:02:00.796] Nov 5 02:02:00.793: INFO: kube-proxy-5xpp6 started at 2019-11-05 01:34:34 +0000 UTC (0+1 container statuses recorded) I1105 02:02:00.797] Nov 5 02:02:00.793: INFO: Container kube-proxy ready: true, restart count 0 I1105 02:02:01.038] Nov 5 02:02:01.038: INFO: I1105 02:02:01.038] Latency metrics for node k8s-master-15826033-2 I1105 02:02:01.038] Nov 5 02:02:01.038: INFO: I1105 02:02:01.038] Logging node info for node k8s-master-15826033-3 I1105 02:02:01.133] Nov 5 02:02:01.130: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-15826033-3,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-15826033-3,UID:b7702272-1876-434e-b6a0-17fa34c6757f,ResourceVersion:5874,Generation:0,CreationTimestamp:2019-11-05 01:35:06 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 0,kubernetes.azure.com/cluster: kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-15826033-3,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.3.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5/providers/Microsoft.Compute/virtualMachines/k8s-master-15826033-3,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 01:35:17 +0000 UTC 2019-11-05 01:35:17 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 02:01:29 +0000 UTC 2019-11-05 01:35:00 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 02:01:29 +0000 UTC 2019-11-05 01:35:00 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 02:01:29 +0000 UTC 2019-11-05 01:35:00 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 02:01:29 +0000 UTC 2019-11-05 01:35:06 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.255.8} {Hostname k8s-master-15826033-3}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:fbf366963961422c9d90bedf6639c8e1,SystemUUID:5388948D-1A8D-EA41-B8B8-7F4523EDF5AA,BootID:42f5884c-7418-486a-8cf3-75188a331f38,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 02:02:01.133] Nov 5 02:02:01.130: INFO: I1105 02:02:01.133] Logging kubelet events for node k8s-master-15826033-3 I1105 02:02:01.211] Nov 5 02:02:01.211: INFO: I1105 02:02:01.211] Logging pods the kubelet thinks is on node k8s-master-15826033-3 I1105 02:02:01.294] Nov 5 02:02:01.294: INFO: kube-scheduler-k8s-master-15826033-3 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:02:01.294] Nov 5 02:02:01.294: INFO: Container kube-scheduler ready: true, restart count 0 I1105 02:02:01.294] Nov 5 02:02:01.294: INFO: kube-proxy-xz24z started at 2019-11-05 01:35:06 +0000 UTC (0+1 container statuses recorded) I1105 02:02:01.295] Nov 5 02:02:01.294: INFO: Container kube-proxy ready: true, restart count 0 I1105 02:02:01.295] Nov 5 02:02:01.294: INFO: azure-ip-masq-agent-dqfpw started at 2019-11-05 01:35:06 +0000 UTC (0+1 container statuses recorded) I1105 02:02:01.295] Nov 5 02:02:01.294: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 02:02:01.295] Nov 5 02:02:01.294: INFO: cloud-controller-manager-k8s-master-15826033-3 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:02:01.295] Nov 5 02:02:01.294: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 02:02:01.295] Nov 5 02:02:01.294: INFO: kube-addon-manager-k8s-master-15826033-3 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:02:01.295] Nov 5 02:02:01.294: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 02:02:01.296] Nov 5 02:02:01.294: INFO: kube-apiserver-k8s-master-15826033-3 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:02:01.296] Nov 5 02:02:01.294: INFO: Container kube-apiserver ready: true, restart count 0 I1105 02:02:01.296] Nov 5 02:02:01.294: INFO: kube-controller-manager-k8s-master-15826033-3 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:02:01.296] Nov 5 02:02:01.294: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 02:02:01.546] Nov 5 02:02:01.546: INFO: I1105 02:02:01.546] Latency metrics for node k8s-master-15826033-3 I1105 02:02:01.546] Nov 5 02:02:01.546: INFO: I1105 02:02:01.547] Logging node info for node k8s-master-15826033-4 I1105 02:02:01.627] Nov 5 02:02:01.624: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-15826033-4,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-15826033-4,UID:276daef1-6c05-41e3-bacd-abc93d35cc6d,ResourceVersion:5883,Generation:0,CreationTimestamp:2019-11-05 01:35:07 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 0,kubernetes.azure.com/cluster: kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-15826033-4,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.6.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5/providers/Microsoft.Compute/virtualMachines/k8s-master-15826033-4,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 01:35:41 +0000 UTC 2019-11-05 01:35:41 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 02:01:31 +0000 UTC 2019-11-05 01:35:02 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 02:01:31 +0000 UTC 2019-11-05 01:35:02 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 02:01:31 +0000 UTC 2019-11-05 01:35:02 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 02:01:31 +0000 UTC 2019-11-05 01:35:08 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.255.9} {Hostname k8s-master-15826033-4}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:79051fc0bdeb4e32a56acbaf5972ce3a,SystemUUID:C71606B5-7F08-9341-A26D-8A8D9D84FCAE,BootID:4bdf7cf3-2b83-42c7-a7ba-926ebe9ca75b,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 02:02:01.627] Nov 5 02:02:01.625: INFO: I1105 02:02:01.627] Logging kubelet events for node k8s-master-15826033-4 I1105 02:02:01.706] Nov 5 02:02:01.706: INFO: I1105 02:02:01.706] Logging pods the kubelet thinks is on node k8s-master-15826033-4 I1105 02:02:01.788] Nov 5 02:02:01.788: INFO: azure-ip-masq-agent-9nrj8 started at 2019-11-05 01:35:11 +0000 UTC (0+1 container statuses recorded) I1105 02:02:01.788] Nov 5 02:02:01.788: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 02:02:01.788] Nov 5 02:02:01.788: INFO: kube-proxy-rmngk started at 2019-11-05 01:35:11 +0000 UTC (0+1 container statuses recorded) I1105 02:02:01.789] Nov 5 02:02:01.788: INFO: Container kube-proxy ready: true, restart count 0 I1105 02:02:01.789] Nov 5 02:02:01.788: INFO: kube-apiserver-k8s-master-15826033-4 started at 2019-11-05 01:34:55 +0000 UTC (0+1 container statuses recorded) I1105 02:02:01.789] Nov 5 02:02:01.788: INFO: Container kube-apiserver ready: true, restart count 0 I1105 02:02:01.789] Nov 5 02:02:01.788: INFO: kube-controller-manager-k8s-master-15826033-4 started at 2019-11-05 01:34:55 +0000 UTC (0+1 container statuses recorded) I1105 02:02:01.789] Nov 5 02:02:01.788: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 02:02:01.790] Nov 5 02:02:01.788: INFO: kube-scheduler-k8s-master-15826033-4 started at 2019-11-05 01:34:55 +0000 UTC (0+1 container statuses recorded) I1105 02:02:01.790] Nov 5 02:02:01.788: INFO: Container kube-scheduler ready: true, restart count 0 I1105 02:02:01.790] Nov 5 02:02:01.788: INFO: cloud-controller-manager-k8s-master-15826033-4 started at 2019-11-05 01:34:56 +0000 UTC (0+1 container statuses recorded) I1105 02:02:01.790] Nov 5 02:02:01.788: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 02:02:01.791] Nov 5 02:02:01.788: INFO: kube-addon-manager-k8s-master-15826033-4 started at 2019-11-05 01:34:56 +0000 UTC (0+1 container statuses recorded) I1105 02:02:01.791] Nov 5 02:02:01.788: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 02:02:02.037] Nov 5 02:02:02.037: INFO: I1105 02:02:02.038] Latency metrics for node k8s-master-15826033-4 I1105 02:02:02.038] Nov 5 02:02:02.037: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready I1105 02:02:02.116] STEP: Destroying namespace "azuredisk-8276" for this suite. I1105 02:02:16.356] Nov 5 02:02:16.355: INFO: Waiting up to 30s for server preferred namespaced resources to be successfully discovered I1105 02:02:19.325] Nov 5 02:02:19.324: INFO: namespace azuredisk-8276 deletion completed in 17.208105367s I1105 02:02:19.325] [AfterEach] [azuredisk-csi-e2e] [single-az] Pre-Provisioned I1105 02:02:19.325] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/pre_provisioning.go:54 I1105 02:02:19.957] I1105 02:02:19.957] • Failure [704.393 seconds] I1105 02:02:19.957] [azuredisk-csi-e2e] [single-az] Pre-Provisioned I1105 02:02:19.957] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/pre_provisioning.go:36 I1105 02:02:19.958] [env] should use a pre-provisioned volume and mount it as readOnly in a pod [It] I1105 02:02:19.958] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/pre_provisioning.go:66 I1105 02:02:19.958] I1105 02:02:19.958] Unexpected error: I1105 02:02:19.958] <*errors.errorString | 0xc00083e510>: { I1105 02:02:19.958] s: "PersistentVolume azuredisk-8276-disk.csi.azure.com-preprovsioned-pv-v648q still exists within 10m0s", I1105 02:02:19.959] } I1105 02:02:19.959] PersistentVolume azuredisk-8276-disk.csi.azure.com-preprovsioned-pv-v648q still exists within 10m0s I1105 02:02:19.959] occurred I1105 02:02:19.959] I1105 02:02:19.959] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/testsuites/testsuites.go:353 I1105 02:02:19.959] ------------------------------ I1105 02:02:19.959] [azuredisk-csi-e2e] Dynamic Provisioning [multi-az] I1105 02:02:19.960] should create a volume on demand and mount it as readOnly in a pod I1105 02:02:19.960] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:126 I1105 02:02:19.960] [BeforeEach] [multi-az] I1105 02:02:19.960] /go/src/sigs.k8s.io/azuredisk-csi-driver/vendor/k8s.io/kubernetes/test/e2e/framework/framework.go:150 I1105 02:02:19.960] STEP: Creating a kubernetes client I1105 02:02:19.960] Nov 5 02:02:19.957: INFO: >>> kubeConfig: /workspace/aks978705698/kubeconfig/kubeconfig.eastus2.json I1105 02:02:19.960] STEP: Building a namespace api object, basename azuredisk I1105 02:02:20.191] STEP: Binding the e2e-test-privileged-psp PodSecurityPolicy to the default service account in azuredisk-104 I1105 02:02:20.452] STEP: Waiting for a default service account to be provisioned in namespace I1105 02:02:20.527] [BeforeEach] [multi-az] I1105 02:02:20.528] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:58 I1105 02:02:20.679] [It] should create a volume on demand and mount it as readOnly in a pod I1105 02:02:20.679] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:126 I1105 02:02:20.679] STEP: setting up the StorageClass I1105 02:02:20.679] STEP: creating a StorageClass I1105 02:02:20.760] STEP: setting up the PVC and PV I1105 02:02:20.760] STEP: creating a PVC I1105 02:02:20.843] STEP: waiting for PVC to be in phase "Bound" I1105 02:02:20.843] Nov 5 02:02:20.843: INFO: Waiting up to 5m0s for PersistentVolumeClaims [pvc-q9qpz] to have phase Bound I1105 02:02:20.920] Nov 5 02:02:20.919: INFO: PersistentVolumeClaim pvc-q9qpz found but phase is Pending instead of Bound. I1105 02:02:22.997] Nov 5 02:02:22.997: INFO: PersistentVolumeClaim pvc-q9qpz found but phase is Pending instead of Bound. I1105 02:02:25.075] Nov 5 02:02:25.075: INFO: PersistentVolumeClaim pvc-q9qpz found but phase is Pending instead of Bound. I1105 02:02:27.153] Nov 5 02:02:27.152: INFO: PersistentVolumeClaim pvc-q9qpz found but phase is Pending instead of Bound. I1105 02:02:29.232] Nov 5 02:02:29.231: INFO: PersistentVolumeClaim pvc-q9qpz found but phase is Pending instead of Bound. I1105 02:02:31.309] Nov 5 02:02:31.308: INFO: PersistentVolumeClaim pvc-q9qpz found but phase is Pending instead of Bound. I1105 02:02:33.386] Nov 5 02:02:33.386: INFO: PersistentVolumeClaim pvc-q9qpz found but phase is Pending instead of Bound. I1105 02:02:35.469] Nov 5 02:02:35.469: INFO: PersistentVolumeClaim pvc-q9qpz found but phase is Pending instead of Bound. I1105 02:02:37.546] Nov 5 02:02:37.546: INFO: PersistentVolumeClaim pvc-q9qpz found but phase is Pending instead of Bound. I1105 02:02:39.624] Nov 5 02:02:39.624: INFO: PersistentVolumeClaim pvc-q9qpz found but phase is Pending instead of Bound. I1105 02:02:41.701] Nov 5 02:02:41.701: INFO: PersistentVolumeClaim pvc-q9qpz found but phase is Pending instead of Bound. I1105 02:02:43.779] Nov 5 02:02:43.779: INFO: PersistentVolumeClaim pvc-q9qpz found but phase is Pending instead of Bound. I1105 02:02:45.856] Nov 5 02:02:45.856: INFO: PersistentVolumeClaim pvc-q9qpz found but phase is Pending instead of Bound. I1105 02:02:47.934] Nov 5 02:02:47.934: INFO: PersistentVolumeClaim pvc-q9qpz found but phase is Pending instead of Bound. I1105 02:02:50.011] Nov 5 02:02:50.011: INFO: PersistentVolumeClaim pvc-q9qpz found but phase is Pending instead of Bound. I1105 02:02:52.089] Nov 5 02:02:52.089: INFO: PersistentVolumeClaim pvc-q9qpz found but phase is Pending instead of Bound. I1105 02:02:54.167] Nov 5 02:02:54.166: INFO: PersistentVolumeClaim pvc-q9qpz found but phase is Pending instead of Bound. I1105 02:02:56.244] Nov 5 02:02:56.243: INFO: PersistentVolumeClaim pvc-q9qpz found but phase is Pending instead of Bound. I1105 02:02:58.321] Nov 5 02:02:58.321: INFO: PersistentVolumeClaim pvc-q9qpz found but phase is Pending instead of Bound. I1105 02:03:00.398] Nov 5 02:03:00.398: INFO: PersistentVolumeClaim pvc-q9qpz found but phase is Pending instead of Bound. I1105 02:03:02.476] Nov 5 02:03:02.476: INFO: PersistentVolumeClaim pvc-q9qpz found but phase is Pending instead of Bound. I1105 02:03:04.554] Nov 5 02:03:04.553: INFO: PersistentVolumeClaim pvc-q9qpz found but phase is Pending instead of Bound. I1105 02:03:06.631] Nov 5 02:03:06.631: INFO: PersistentVolumeClaim pvc-q9qpz found but phase is Pending instead of Bound. I1105 02:03:08.714] Nov 5 02:03:08.713: INFO: PersistentVolumeClaim pvc-q9qpz found but phase is Pending instead of Bound. I1105 02:03:10.791] Nov 5 02:03:10.791: INFO: PersistentVolumeClaim pvc-q9qpz found but phase is Pending instead of Bound. I1105 02:03:12.872] Nov 5 02:03:12.872: INFO: PersistentVolumeClaim pvc-q9qpz found but phase is Pending instead of Bound. I1105 02:03:14.950] Nov 5 02:03:14.950: INFO: PersistentVolumeClaim pvc-q9qpz found but phase is Pending instead of Bound. I1105 02:03:17.027] Nov 5 02:03:17.027: INFO: PersistentVolumeClaim pvc-q9qpz found but phase is Pending instead of Bound. I1105 02:03:19.104] Nov 5 02:03:19.104: INFO: PersistentVolumeClaim pvc-q9qpz found but phase is Pending instead of Bound. I1105 02:03:21.182] Nov 5 02:03:21.182: INFO: PersistentVolumeClaim pvc-q9qpz found but phase is Pending instead of Bound. I1105 02:03:23.260] Nov 5 02:03:23.260: INFO: PersistentVolumeClaim pvc-q9qpz found but phase is Pending instead of Bound. I1105 02:03:25.338] Nov 5 02:03:25.337: INFO: PersistentVolumeClaim pvc-q9qpz found but phase is Pending instead of Bound. I1105 02:03:27.415] Nov 5 02:03:27.415: INFO: PersistentVolumeClaim pvc-q9qpz found but phase is Pending instead of Bound. I1105 02:03:29.492] Nov 5 02:03:29.492: INFO: PersistentVolumeClaim pvc-q9qpz found but phase is Pending instead of Bound. I1105 02:03:31.571] Nov 5 02:03:31.571: INFO: PersistentVolumeClaim pvc-q9qpz found but phase is Pending instead of Bound. I1105 02:03:33.649] Nov 5 02:03:33.649: INFO: PersistentVolumeClaim pvc-q9qpz found but phase is Pending instead of Bound. I1105 02:03:35.727] Nov 5 02:03:35.726: INFO: PersistentVolumeClaim pvc-q9qpz found but phase is Pending instead of Bound. I1105 02:03:37.804] Nov 5 02:03:37.804: INFO: PersistentVolumeClaim pvc-q9qpz found but phase is Pending instead of Bound. I1105 02:03:39.881] Nov 5 02:03:39.881: INFO: PersistentVolumeClaim pvc-q9qpz found but phase is Pending instead of Bound. I1105 02:03:41.959] Nov 5 02:03:41.959: INFO: PersistentVolumeClaim pvc-q9qpz found but phase is Pending instead of Bound. I1105 02:03:44.036] Nov 5 02:03:44.036: INFO: PersistentVolumeClaim pvc-q9qpz found but phase is Pending instead of Bound. I1105 02:03:46.115] Nov 5 02:03:46.114: INFO: PersistentVolumeClaim pvc-q9qpz found but phase is Pending instead of Bound. I1105 02:03:48.192] Nov 5 02:03:48.192: INFO: PersistentVolumeClaim pvc-q9qpz found but phase is Pending instead of Bound. I1105 02:03:50.270] Nov 5 02:03:50.270: INFO: PersistentVolumeClaim pvc-q9qpz found but phase is Pending instead of Bound. I1105 02:03:52.347] Nov 5 02:03:52.347: INFO: PersistentVolumeClaim pvc-q9qpz found but phase is Pending instead of Bound. I1105 02:03:54.425] Nov 5 02:03:54.424: INFO: PersistentVolumeClaim pvc-q9qpz found but phase is Pending instead of Bound. I1105 02:03:56.502] Nov 5 02:03:56.502: INFO: PersistentVolumeClaim pvc-q9qpz found but phase is Pending instead of Bound. I1105 02:03:58.580] Nov 5 02:03:58.579: INFO: PersistentVolumeClaim pvc-q9qpz found but phase is Pending instead of Bound. I1105 02:04:00.657] Nov 5 02:04:00.657: INFO: PersistentVolumeClaim pvc-q9qpz found but phase is Pending instead of Bound. I1105 02:04:02.735] Nov 5 02:04:02.735: INFO: PersistentVolumeClaim pvc-q9qpz found but phase is Pending instead of Bound. I1105 02:04:04.812] Nov 5 02:04:04.812: INFO: PersistentVolumeClaim pvc-q9qpz found but phase is Pending instead of Bound. I1105 02:04:06.891] Nov 5 02:04:06.890: INFO: PersistentVolumeClaim pvc-q9qpz found but phase is Pending instead of Bound. I1105 02:04:08.968] Nov 5 02:04:08.968: INFO: PersistentVolumeClaim pvc-q9qpz found but phase is Pending instead of Bound. I1105 02:04:11.045] Nov 5 02:04:11.045: INFO: PersistentVolumeClaim pvc-q9qpz found but phase is Pending instead of Bound. I1105 02:04:13.138] Nov 5 02:04:13.138: INFO: PersistentVolumeClaim pvc-q9qpz found but phase is Pending instead of Bound. I1105 02:04:15.215] Nov 5 02:04:15.215: INFO: PersistentVolumeClaim pvc-q9qpz found but phase is Pending instead of Bound. I1105 02:04:17.293] Nov 5 02:04:17.293: INFO: PersistentVolumeClaim pvc-q9qpz found but phase is Pending instead of Bound. I1105 02:04:19.371] Nov 5 02:04:19.370: INFO: PersistentVolumeClaim pvc-q9qpz found but phase is Pending instead of Bound. I1105 02:04:21.448] Nov 5 02:04:21.448: INFO: PersistentVolumeClaim pvc-q9qpz found but phase is Pending instead of Bound. I1105 02:04:23.526] Nov 5 02:04:23.525: INFO: PersistentVolumeClaim pvc-q9qpz found but phase is Pending instead of Bound. I1105 02:04:25.604] Nov 5 02:04:25.604: INFO: PersistentVolumeClaim pvc-q9qpz found but phase is Pending instead of Bound. I1105 02:04:27.682] Nov 5 02:04:27.681: INFO: PersistentVolumeClaim pvc-q9qpz found but phase is Pending instead of Bound. I1105 02:04:29.759] Nov 5 02:04:29.759: INFO: PersistentVolumeClaim pvc-q9qpz found but phase is Pending instead of Bound. I1105 02:04:31.837] Nov 5 02:04:31.837: INFO: PersistentVolumeClaim pvc-q9qpz found but phase is Pending instead of Bound. I1105 02:04:33.914] Nov 5 02:04:33.914: INFO: PersistentVolumeClaim pvc-q9qpz found but phase is Pending instead of Bound. I1105 02:04:35.992] Nov 5 02:04:35.991: INFO: PersistentVolumeClaim pvc-q9qpz found but phase is Pending instead of Bound. I1105 02:04:38.069] Nov 5 02:04:38.069: INFO: PersistentVolumeClaim pvc-q9qpz found but phase is Pending instead of Bound. I1105 02:04:40.147] Nov 5 02:04:40.147: INFO: PersistentVolumeClaim pvc-q9qpz found but phase is Pending instead of Bound. I1105 02:04:42.224] Nov 5 02:04:42.224: INFO: PersistentVolumeClaim pvc-q9qpz found but phase is Pending instead of Bound. I1105 02:04:44.303] Nov 5 02:04:44.302: INFO: PersistentVolumeClaim pvc-q9qpz found but phase is Pending instead of Bound. I1105 02:04:46.380] Nov 5 02:04:46.380: INFO: PersistentVolumeClaim pvc-q9qpz found but phase is Pending instead of Bound. I1105 02:04:48.458] Nov 5 02:04:48.457: INFO: PersistentVolumeClaim pvc-q9qpz found but phase is Pending instead of Bound. I1105 02:04:50.535] Nov 5 02:04:50.535: INFO: PersistentVolumeClaim pvc-q9qpz found but phase is Pending instead of Bound. I1105 02:04:52.613] Nov 5 02:04:52.612: INFO: PersistentVolumeClaim pvc-q9qpz found but phase is Pending instead of Bound. I1105 02:04:54.690] Nov 5 02:04:54.690: INFO: PersistentVolumeClaim pvc-q9qpz found but phase is Pending instead of Bound. I1105 02:04:56.768] Nov 5 02:04:56.768: INFO: PersistentVolumeClaim pvc-q9qpz found but phase is Pending instead of Bound. I1105 02:04:58.846] Nov 5 02:04:58.845: INFO: PersistentVolumeClaim pvc-q9qpz found but phase is Pending instead of Bound. I1105 02:05:00.923] Nov 5 02:05:00.923: INFO: PersistentVolumeClaim pvc-q9qpz found but phase is Pending instead of Bound. I1105 02:05:03.000] Nov 5 02:05:03.000: INFO: PersistentVolumeClaim pvc-q9qpz found but phase is Pending instead of Bound. I1105 02:05:05.079] Nov 5 02:05:05.078: INFO: PersistentVolumeClaim pvc-q9qpz found but phase is Pending instead of Bound. I1105 02:05:07.156] Nov 5 02:05:07.156: INFO: PersistentVolumeClaim pvc-q9qpz found but phase is Pending instead of Bound. I1105 02:05:09.233] Nov 5 02:05:09.233: INFO: PersistentVolumeClaim pvc-q9qpz found but phase is Pending instead of Bound. I1105 02:05:11.311] Nov 5 02:05:11.311: INFO: PersistentVolumeClaim pvc-q9qpz found but phase is Pending instead of Bound. I1105 02:05:13.389] Nov 5 02:05:13.389: INFO: PersistentVolumeClaim pvc-q9qpz found but phase is Pending instead of Bound. I1105 02:05:15.467] Nov 5 02:05:15.467: INFO: PersistentVolumeClaim pvc-q9qpz found but phase is Pending instead of Bound. I1105 02:05:17.545] Nov 5 02:05:17.545: INFO: PersistentVolumeClaim pvc-q9qpz found but phase is Pending instead of Bound. I1105 02:05:19.622] Nov 5 02:05:19.622: INFO: PersistentVolumeClaim pvc-q9qpz found but phase is Pending instead of Bound. I1105 02:05:21.699] Nov 5 02:05:21.699: INFO: PersistentVolumeClaim pvc-q9qpz found but phase is Pending instead of Bound. I1105 02:05:23.777] Nov 5 02:05:23.777: INFO: PersistentVolumeClaim pvc-q9qpz found but phase is Pending instead of Bound. I1105 02:05:25.854] Nov 5 02:05:25.854: INFO: PersistentVolumeClaim pvc-q9qpz found but phase is Pending instead of Bound. I1105 02:05:27.932] Nov 5 02:05:27.932: INFO: PersistentVolumeClaim pvc-q9qpz found but phase is Pending instead of Bound. I1105 02:05:30.009] Nov 5 02:05:30.009: INFO: PersistentVolumeClaim pvc-q9qpz found but phase is Pending instead of Bound. I1105 02:05:32.087] Nov 5 02:05:32.087: INFO: PersistentVolumeClaim pvc-q9qpz found but phase is Pending instead of Bound. I1105 02:05:34.165] Nov 5 02:05:34.165: INFO: PersistentVolumeClaim pvc-q9qpz found but phase is Pending instead of Bound. I1105 02:05:36.243] Nov 5 02:05:36.243: INFO: PersistentVolumeClaim pvc-q9qpz found but phase is Pending instead of Bound. I1105 02:05:38.320] Nov 5 02:05:38.320: INFO: PersistentVolumeClaim pvc-q9qpz found but phase is Pending instead of Bound. I1105 02:05:40.398] Nov 5 02:05:40.397: INFO: PersistentVolumeClaim pvc-q9qpz found but phase is Pending instead of Bound. I1105 02:05:42.476] Nov 5 02:05:42.475: INFO: PersistentVolumeClaim pvc-q9qpz found but phase is Pending instead of Bound. I1105 02:05:44.554] Nov 5 02:05:44.553: INFO: PersistentVolumeClaim pvc-q9qpz found but phase is Pending instead of Bound. I1105 02:05:46.632] Nov 5 02:05:46.632: INFO: PersistentVolumeClaim pvc-q9qpz found but phase is Pending instead of Bound. I1105 02:05:48.711] Nov 5 02:05:48.711: INFO: PersistentVolumeClaim pvc-q9qpz found but phase is Pending instead of Bound. I1105 02:05:50.789] Nov 5 02:05:50.789: INFO: PersistentVolumeClaim pvc-q9qpz found but phase is Pending instead of Bound. I1105 02:05:52.867] Nov 5 02:05:52.867: INFO: PersistentVolumeClaim pvc-q9qpz found but phase is Pending instead of Bound. I1105 02:05:54.945] Nov 5 02:05:54.945: INFO: PersistentVolumeClaim pvc-q9qpz found but phase is Pending instead of Bound. I1105 02:05:57.023] Nov 5 02:05:57.022: INFO: PersistentVolumeClaim pvc-q9qpz found but phase is Pending instead of Bound. I1105 02:05:59.101] Nov 5 02:05:59.101: INFO: PersistentVolumeClaim pvc-q9qpz found but phase is Pending instead of Bound. I1105 02:06:01.179] Nov 5 02:06:01.179: INFO: PersistentVolumeClaim pvc-q9qpz found but phase is Pending instead of Bound. I1105 02:06:03.258] Nov 5 02:06:03.257: INFO: PersistentVolumeClaim pvc-q9qpz found but phase is Pending instead of Bound. I1105 02:06:05.336] Nov 5 02:06:05.336: INFO: PersistentVolumeClaim pvc-q9qpz found but phase is Pending instead of Bound. I1105 02:06:07.414] Nov 5 02:06:07.414: INFO: PersistentVolumeClaim pvc-q9qpz found but phase is Pending instead of Bound. I1105 02:06:09.492] Nov 5 02:06:09.491: INFO: PersistentVolumeClaim pvc-q9qpz found but phase is Pending instead of Bound. I1105 02:06:11.570] Nov 5 02:06:11.570: INFO: PersistentVolumeClaim pvc-q9qpz found but phase is Pending instead of Bound. I1105 02:06:13.648] Nov 5 02:06:13.648: INFO: PersistentVolumeClaim pvc-q9qpz found but phase is Pending instead of Bound. I1105 02:06:15.728] Nov 5 02:06:15.728: INFO: PersistentVolumeClaim pvc-q9qpz found but phase is Pending instead of Bound. I1105 02:06:17.806] Nov 5 02:06:17.806: INFO: PersistentVolumeClaim pvc-q9qpz found but phase is Pending instead of Bound. I1105 02:06:19.884] Nov 5 02:06:19.884: INFO: PersistentVolumeClaim pvc-q9qpz found but phase is Pending instead of Bound. I1105 02:06:21.971] Nov 5 02:06:21.971: INFO: PersistentVolumeClaim pvc-q9qpz found but phase is Pending instead of Bound. I1105 02:06:24.050] Nov 5 02:06:24.050: INFO: PersistentVolumeClaim pvc-q9qpz found but phase is Pending instead of Bound. I1105 02:06:26.128] Nov 5 02:06:26.128: INFO: PersistentVolumeClaim pvc-q9qpz found but phase is Pending instead of Bound. I1105 02:06:28.207] Nov 5 02:06:28.206: INFO: PersistentVolumeClaim pvc-q9qpz found but phase is Pending instead of Bound. I1105 02:06:30.285] Nov 5 02:06:30.285: INFO: PersistentVolumeClaim pvc-q9qpz found but phase is Pending instead of Bound. I1105 02:06:32.368] Nov 5 02:06:32.367: INFO: PersistentVolumeClaim pvc-q9qpz found but phase is Pending instead of Bound. I1105 02:06:34.445] Nov 5 02:06:34.445: INFO: PersistentVolumeClaim pvc-q9qpz found but phase is Pending instead of Bound. I1105 02:06:36.524] Nov 5 02:06:36.523: INFO: PersistentVolumeClaim pvc-q9qpz found but phase is Pending instead of Bound. I1105 02:06:38.602] Nov 5 02:06:38.601: INFO: PersistentVolumeClaim pvc-q9qpz found but phase is Pending instead of Bound. I1105 02:06:40.680] Nov 5 02:06:40.680: INFO: PersistentVolumeClaim pvc-q9qpz found but phase is Pending instead of Bound. I1105 02:06:42.758] Nov 5 02:06:42.758: INFO: PersistentVolumeClaim pvc-q9qpz found but phase is Pending instead of Bound. I1105 02:06:44.845] Nov 5 02:06:44.845: INFO: PersistentVolumeClaim pvc-q9qpz found but phase is Pending instead of Bound. I1105 02:06:46.923] Nov 5 02:06:46.923: INFO: PersistentVolumeClaim pvc-q9qpz found but phase is Pending instead of Bound. I1105 02:06:49.001] Nov 5 02:06:49.001: INFO: PersistentVolumeClaim pvc-q9qpz found but phase is Pending instead of Bound. I1105 02:06:51.079] Nov 5 02:06:51.079: INFO: PersistentVolumeClaim pvc-q9qpz found but phase is Pending instead of Bound. I1105 02:06:53.159] Nov 5 02:06:53.159: INFO: PersistentVolumeClaim pvc-q9qpz found but phase is Pending instead of Bound. I1105 02:06:55.237] Nov 5 02:06:55.237: INFO: PersistentVolumeClaim pvc-q9qpz found but phase is Pending instead of Bound. I1105 02:06:57.316] Nov 5 02:06:57.316: INFO: PersistentVolumeClaim pvc-q9qpz found but phase is Pending instead of Bound. I1105 02:06:59.394] Nov 5 02:06:59.394: INFO: PersistentVolumeClaim pvc-q9qpz found but phase is Pending instead of Bound. I1105 02:07:01.473] Nov 5 02:07:01.472: INFO: PersistentVolumeClaim pvc-q9qpz found but phase is Pending instead of Bound. I1105 02:07:03.550] Nov 5 02:07:03.550: INFO: PersistentVolumeClaim pvc-q9qpz found but phase is Pending instead of Bound. I1105 02:07:05.629] Nov 5 02:07:05.628: INFO: PersistentVolumeClaim pvc-q9qpz found but phase is Pending instead of Bound. I1105 02:07:07.707] Nov 5 02:07:07.706: INFO: PersistentVolumeClaim pvc-q9qpz found but phase is Pending instead of Bound. I1105 02:07:09.785] Nov 5 02:07:09.785: INFO: PersistentVolumeClaim pvc-q9qpz found but phase is Pending instead of Bound. I1105 02:07:11.863] Nov 5 02:07:11.863: INFO: PersistentVolumeClaim pvc-q9qpz found but phase is Pending instead of Bound. I1105 02:07:13.943] Nov 5 02:07:13.943: INFO: PersistentVolumeClaim pvc-q9qpz found but phase is Pending instead of Bound. I1105 02:07:16.021] Nov 5 02:07:16.021: INFO: PersistentVolumeClaim pvc-q9qpz found but phase is Pending instead of Bound. I1105 02:07:18.100] Nov 5 02:07:18.100: INFO: PersistentVolumeClaim pvc-q9qpz found but phase is Pending instead of Bound. I1105 02:07:20.178] Nov 5 02:07:20.178: INFO: PersistentVolumeClaim pvc-q9qpz found but phase is Pending instead of Bound. I1105 02:07:22.178] Nov 5 02:07:22.178: INFO: Unexpected error occurred: PersistentVolumeClaims [pvc-q9qpz] not all in phase Bound within 5m0s I1105 02:07:22.179] [AfterEach] [multi-az] I1105 02:07:22.179] /go/src/sigs.k8s.io/azuredisk-csi-driver/vendor/k8s.io/kubernetes/test/e2e/framework/framework.go:151 I1105 02:07:22.179] STEP: Collecting events from namespace "azuredisk-104". I1105 02:07:22.257] STEP: Found 3 events. I1105 02:07:22.257] Nov 5 02:07:22.256: INFO: At 2019-11-05 02:02:20 +0000 UTC - event for pvc-q9qpz: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "disk.csi.azure.com" or manually created by system administrator I1105 02:07:22.257] Nov 5 02:07:22.256: INFO: At 2019-11-05 02:02:20 +0000 UTC - event for pvc-q9qpz: {disk.csi.azure.com_csi-azuredisk-controller-6f6587bfc6-8vkfl_1376914b-ff6d-11e9-93ab-ba2bee589b8c } Provisioning: External provisioner is provisioning volume for claim "azuredisk-104/pvc-q9qpz" I1105 02:07:22.258] Nov 5 02:07:22.256: INFO: At 2019-11-05 02:02:20 +0000 UTC - event for pvc-q9qpz: {disk.csi.azure.com_csi-azuredisk-controller-6f6587bfc6-8vkfl_1376914b-ff6d-11e9-93ab-ba2bee589b8c } ProvisioningFailed: failed to provision volume with StorageClass "azuredisk-104-disk.csi.azure.com-dynamic-sc-t6xqm": rpc error: code = Unknown desc = compute.DisksClient#CreateOrUpdate: Failure sending request: StatusCode=400 -- Original Error: Code="InvalidAvailabilityZone" Message="The zone(s) '' for resource 'Microsoft.Compute/disks/pvc-82b77b97-dc12-48e5-bf5d-0c0733f8a1f6' is not supported. The supported zones for location 'eastus2' are '1,2,3'" I1105 02:07:22.334] Nov 5 02:07:22.334: INFO: POD NODE PHASE GRACE CONDITIONS I1105 02:07:22.334] Nov 5 02:07:22.334: INFO: I1105 02:07:22.565] Nov 5 02:07:22.565: INFO: I1105 02:07:22.565] Logging node info for node k8s-agentpool-15826033-vmss000000 I1105 02:07:22.645] Nov 5 02:07:22.642: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-agentpool-15826033-vmss000000,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-agentpool-15826033-vmss000000,UID:15fa874e-54ec-42cd-b243-d3baa512150f,ResourceVersion:6756,Generation:0,CreationTimestamp:2019-11-05 01:34:17 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{agentpool: agentpool,beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_D4s_v3,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 0,kubernetes.azure.com/cluster: kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5,kubernetes.azure.com/role: agent,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-agentpool-15826033-vmss000000,kubernetes.io/os: linux,kubernetes.io/role: agent,node-role.kubernetes.io/agent: ,storageprofile: managed,storagetier: Premium_LRS,topology.disk.csi.azure.com/zone: 0,},Annotations:map[string]string{csi.volume.kubernetes.io/nodeid: {"disk.csi.azure.com":"k8s-agentpool-15826033-vmss000000"},node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.2.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool-15826033-vmss/virtualMachines/0,Unschedulable:false,Taints:[],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-csi-disk.csi.azure.com: {{8 0} {} 8 DecimalSI},cpu: {{4 0} {} 4 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{16797044736 0} {} 16403364Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-csi-disk.csi.azure.com: {{8 0} {} 8 DecimalSI},cpu: {{4 0} {} 4 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{16010612736 0} {} 15635364Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 01:34:56 +0000 UTC 2019-11-05 01:34:56 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 02:06:22 +0000 UTC 2019-11-05 01:34:17 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 02:06:22 +0000 UTC 2019-11-05 01:34:17 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 02:06:22 +0000 UTC 2019-11-05 01:34:17 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 02:06:22 +0000 UTC 2019-11-05 01:34:29 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.0.4} {Hostname k8s-agentpool-15826033-vmss000000}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:afe00792177d49a98f6a44d03d08732d,SystemUUID:4F1777BF-A5C3-004E-81C4-90F0B54C02A0,BootID:13b6a628-ae89-44af-b891-89461030adcb,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[k8sprowinternal.azurecr.io/azuredisk-csi@sha256:75d4ff5d7cd845d6e810c67796ecb037e852fb849ec1eb5b3194564008e6d1e0 k8sprowinternal.azurecr.io/azuredisk-csi:e2e-3e3ed045c6faa509dbd8d2890ee9f56aa2934887] 286930414} {[k8s.gcr.io/kubernetes-dashboard-amd64@sha256:0ae6b69432e78069c5ce2bcde0fe409c5c4d6f0f4d9cd50a17974fea38898747 k8s.gcr.io/kubernetes-dashboard-amd64:v1.10.1] 121711221} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/metrics-server-amd64@sha256:49a9f12f7067d11f42c803dbe61ed2c1299959ad85cb315b25ff7eef8e6b8892 k8s.gcr.io/metrics-server-amd64:v0.2.1] 42541759} {[mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume@sha256:4fd30d43947d4a54fc89ead7985beecfd3c9b2a93a0655a373b1608ab90bd5af mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume:v0.0.7] 22909487} {[quay.io/k8scsi/livenessprobe@sha256:8c9c8a0f330acc2e558ae6624309447d02522e76aa35af23fa19b88abf4381cb quay.io/k8scsi/livenessprobe:v1.0.2] 17415381} {[quay.io/k8scsi/csi-node-driver-registrar@sha256:13daf82fb99e951a4bff8ae5fc7c17c3a8fe7130be6400990d8f6076c32d4599 quay.io/k8scsi/csi-node-driver-registrar:v1.1.0] 15815995} {[mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume@sha256:23d8c6033f02a1ecad05127ebdc931bb871264228661bc122704b0974e4d9fdd mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume:1.0.8] 1159025} {[busybox@sha256:8ccbac733d19c0dd4d70b4f0c1e12245b5fa3ad24758a11035ee505c629c0796 busybox:1.29] 1154361} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 02:07:22.646] Nov 5 02:07:22.643: INFO: I1105 02:07:22.646] Logging kubelet events for node k8s-agentpool-15826033-vmss000000 I1105 02:07:22.724] Nov 5 02:07:22.724: INFO: I1105 02:07:22.725] Logging pods the kubelet thinks is on node k8s-agentpool-15826033-vmss000000 I1105 02:07:22.810] Nov 5 02:07:22.810: INFO: blobfuse-flexvol-installer-bsvqt started at 2019-11-05 01:34:34 +0000 UTC (0+1 container statuses recorded) I1105 02:07:22.811] Nov 5 02:07:22.810: INFO: Container blobfuse-flexvol-installer ready: true, restart count 0 I1105 02:07:22.811] Nov 5 02:07:22.810: INFO: azure-ip-masq-agent-vx86b started at 2019-11-05 01:34:34 +0000 UTC (0+1 container statuses recorded) I1105 02:07:22.811] Nov 5 02:07:22.810: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 02:07:22.811] Nov 5 02:07:22.810: INFO: keyvault-flexvolume-qhnfr started at 2019-11-05 01:34:34 +0000 UTC (0+1 container statuses recorded) I1105 02:07:22.812] Nov 5 02:07:22.810: INFO: Container keyvault-flexvolume ready: true, restart count 0 I1105 02:07:22.812] Nov 5 02:07:22.810: INFO: metrics-server-864ffbc5c-sdxps started at 2019-11-05 01:34:34 +0000 UTC (0+1 container statuses recorded) I1105 02:07:22.812] Nov 5 02:07:22.810: INFO: Container metrics-server ready: true, restart count 0 I1105 02:07:22.812] Nov 5 02:07:22.810: INFO: kube-proxy-wxblc started at 2019-11-05 01:34:34 +0000 UTC (0+1 container statuses recorded) I1105 02:07:22.812] Nov 5 02:07:22.810: INFO: Container kube-proxy ready: true, restart count 0 I1105 02:07:22.813] Nov 5 02:07:22.810: INFO: kubernetes-dashboard-66dd8b8df7-ks284 started at 2019-11-05 01:34:35 +0000 UTC (0+1 container statuses recorded) I1105 02:07:22.813] Nov 5 02:07:22.810: INFO: Container kubernetes-dashboard ready: true, restart count 0 I1105 02:07:22.813] Nov 5 02:07:22.810: INFO: csi-azuredisk-node-whq2g started at 2019-11-05 01:38:24 +0000 UTC (0+3 container statuses recorded) I1105 02:07:22.813] Nov 5 02:07:22.810: INFO: Container azuredisk ready: true, restart count 0 I1105 02:07:22.814] Nov 5 02:07:22.810: INFO: Container liveness-probe ready: true, restart count 0 I1105 02:07:22.814] Nov 5 02:07:22.810: INFO: Container node-driver-registrar ready: true, restart count 0 I1105 02:07:23.064] Nov 5 02:07:23.064: INFO: I1105 02:07:23.064] Latency metrics for node k8s-agentpool-15826033-vmss000000 I1105 02:07:23.064] Nov 5 02:07:23.064: INFO: I1105 02:07:23.064] Logging node info for node k8s-agentpool-15826033-vmss000001 I1105 02:07:23.145] Nov 5 02:07:23.141: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-agentpool-15826033-vmss000001,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-agentpool-15826033-vmss000001,UID:4a8ea605-cc0b-4e3b-9060-d8867d269e6c,ResourceVersion:6821,Generation:0,CreationTimestamp:2019-11-05 01:34:09 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{agentpool: agentpool,beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_D4s_v3,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 1,kubernetes.azure.com/cluster: kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5,kubernetes.azure.com/role: agent,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-agentpool-15826033-vmss000001,kubernetes.io/os: linux,kubernetes.io/role: agent,node-role.kubernetes.io/agent: ,storageprofile: managed,storagetier: Premium_LRS,topology.disk.csi.azure.com/zone: 1,},Annotations:map[string]string{csi.volume.kubernetes.io/nodeid: {"disk.csi.azure.com":"k8s-agentpool-15826033-vmss000001"},node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.0.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool-15826033-vmss/virtualMachines/1,Unschedulable:false,Taints:[],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-csi-disk.csi.azure.com: {{8 0} {} 8 DecimalSI},cpu: {{4 0} {} 4 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{16797044736 0} {} 16403364Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-csi-disk.csi.azure.com: {{8 0} {} 8 DecimalSI},cpu: {{4 0} {} 4 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{16010612736 0} {} 15635364Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 01:34:57 +0000 UTC 2019-11-05 01:34:57 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 02:06:44 +0000 UTC 2019-11-05 01:34:08 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 02:06:44 +0000 UTC 2019-11-05 01:34:08 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 02:06:44 +0000 UTC 2019-11-05 01:34:08 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 02:06:44 +0000 UTC 2019-11-05 01:34:29 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.0.5} {Hostname k8s-agentpool-15826033-vmss000001}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:b6b9fbf4a2fd4fa5a7ae472415fe30b8,SystemUUID:0162B3A4-3955-9A46-ABD4-A37BD75848CC,BootID:ee93003f-d47c-4118-9975-57365b3443a5,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[k8sprowinternal.azurecr.io/azuredisk-csi@sha256:75d4ff5d7cd845d6e810c67796ecb037e852fb849ec1eb5b3194564008e6d1e0 k8sprowinternal.azurecr.io/azuredisk-csi:e2e-3e3ed045c6faa509dbd8d2890ee9f56aa2934887] 286930414} {[gcr.io/kubernetes-helm/tiller@sha256:f6d8f4ab9ba993b5f5b60a6edafe86352eabe474ffeb84cb6c79b8866dce45d1 gcr.io/kubernetes-helm/tiller:v2.11.0] 71821984} {[quay.io/k8scsi/csi-provisioner@sha256:d657c839dce87324fe2b677302913f9386f885f8746be7bea0ced5b0844e3433 quay.io/k8scsi/csi-provisioner:v1.3.1] 50596998} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[quay.io/k8scsi/csi-attacher@sha256:26fccd7a99d973845df1193b46ebdcc6ab8dc5f6e6be319750c471fce1742d13 quay.io/k8scsi/csi-attacher:v1.2.0] 46226754} {[quay.io/k8scsi/csi-cluster-driver-registrar@sha256:fafd75ae5442f192cfa8c2e792903aee30d5884b62e802e4464b0a895d21e3ef quay.io/k8scsi/csi-cluster-driver-registrar:v1.0.1] 45874691} {[quay.io/k8scsi/csi-snapshotter@sha256:a49e0da1af6f2bf717e41ba1eee8b5e6a1cbd66a709dd92cc43fe475fe2589eb quay.io/k8scsi/csi-snapshotter:v1.1.0] 44325352} {[k8s.gcr.io/coredns@sha256:e83beb5e43f8513fa735e77ffc5859640baea30a882a11cc75c4c3244a737d3c k8s.gcr.io/coredns:1.5.0] 42488424} {[mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume@sha256:4fd30d43947d4a54fc89ead7985beecfd3c9b2a93a0655a373b1608ab90bd5af mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume:v0.0.7] 22909487} {[quay.io/k8scsi/livenessprobe@sha256:8c9c8a0f330acc2e558ae6624309447d02522e76aa35af23fa19b88abf4381cb quay.io/k8scsi/livenessprobe:v1.0.2] 17415381} {[quay.io/k8scsi/csi-node-driver-registrar@sha256:13daf82fb99e951a4bff8ae5fc7c17c3a8fe7130be6400990d8f6076c32d4599 quay.io/k8scsi/csi-node-driver-registrar:v1.1.0] 15815995} {[quay.io/k8scsi/livenessprobe@sha256:dde617756e0f602adc566ab71fd885f1dad451ad3fb063ac991c95a2ff47aea5 quay.io/k8scsi/livenessprobe:v1.1.0] 14967303} {[mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume@sha256:23d8c6033f02a1ecad05127ebdc931bb871264228661bc122704b0974e4d9fdd mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume:1.0.8] 1159025} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 02:07:23.145] Nov 5 02:07:23.141: INFO: I1105 02:07:23.145] Logging kubelet events for node k8s-agentpool-15826033-vmss000001 I1105 02:07:23.225] Nov 5 02:07:23.224: INFO: I1105 02:07:23.225] Logging pods the kubelet thinks is on node k8s-agentpool-15826033-vmss000001 I1105 02:07:23.313] Nov 5 02:07:23.313: INFO: csi-azuredisk-node-ghv5w started at 2019-11-05 01:38:24 +0000 UTC (0+3 container statuses recorded) I1105 02:07:23.313] Nov 5 02:07:23.313: INFO: Container azuredisk ready: true, restart count 0 I1105 02:07:23.313] Nov 5 02:07:23.313: INFO: Container liveness-probe ready: true, restart count 0 I1105 02:07:23.314] Nov 5 02:07:23.313: INFO: Container node-driver-registrar ready: true, restart count 0 I1105 02:07:23.314] Nov 5 02:07:23.313: INFO: csi-azuredisk-controller-6f6587bfc6-8vkfl started at 2019-11-05 01:38:24 +0000 UTC (0+6 container statuses recorded) I1105 02:07:23.314] Nov 5 02:07:23.313: INFO: Container azuredisk ready: true, restart count 0 I1105 02:07:23.314] Nov 5 02:07:23.313: INFO: Container cluster-driver-registrar ready: true, restart count 0 I1105 02:07:23.314] Nov 5 02:07:23.313: INFO: Container csi-attacher ready: true, restart count 0 I1105 02:07:23.314] Nov 5 02:07:23.313: INFO: Container csi-provisioner ready: true, restart count 0 I1105 02:07:23.315] Nov 5 02:07:23.313: INFO: Container csi-snapshotter ready: true, restart count 0 I1105 02:07:23.315] Nov 5 02:07:23.313: INFO: Container liveness-probe ready: true, restart count 0 I1105 02:07:23.315] Nov 5 02:07:23.313: INFO: blobfuse-flexvol-installer-rw5dw started at 2019-11-05 01:34:35 +0000 UTC (0+1 container statuses recorded) I1105 02:07:23.315] Nov 5 02:07:23.313: INFO: Container blobfuse-flexvol-installer ready: true, restart count 0 I1105 02:07:23.315] Nov 5 02:07:23.313: INFO: coredns-7f68dcdbdb-jt522 started at 2019-11-05 01:34:35 +0000 UTC (0+1 container statuses recorded) I1105 02:07:23.315] Nov 5 02:07:23.313: INFO: Container coredns ready: true, restart count 0 I1105 02:07:23.316] Nov 5 02:07:23.313: INFO: azure-ip-masq-agent-6c6vn started at 2019-11-05 01:34:35 +0000 UTC (0+1 container statuses recorded) I1105 02:07:23.316] Nov 5 02:07:23.313: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 02:07:23.316] Nov 5 02:07:23.313: INFO: keyvault-flexvolume-rdlcj started at 2019-11-05 01:34:35 +0000 UTC (0+1 container statuses recorded) I1105 02:07:23.316] Nov 5 02:07:23.313: INFO: Container keyvault-flexvolume ready: true, restart count 0 I1105 02:07:23.316] Nov 5 02:07:23.313: INFO: kube-proxy-6kh57 started at 2019-11-05 01:34:35 +0000 UTC (0+1 container statuses recorded) I1105 02:07:23.317] Nov 5 02:07:23.313: INFO: Container kube-proxy ready: true, restart count 0 I1105 02:07:23.317] Nov 5 02:07:23.313: INFO: tiller-deploy-54c96cb5df-8d9fs started at 2019-11-05 01:34:35 +0000 UTC (0+1 container statuses recorded) I1105 02:07:23.317] Nov 5 02:07:23.313: INFO: Container tiller ready: true, restart count 0 I1105 02:07:23.578] Nov 5 02:07:23.578: INFO: I1105 02:07:23.578] Latency metrics for node k8s-agentpool-15826033-vmss000001 I1105 02:07:23.578] Nov 5 02:07:23.578: INFO: I1105 02:07:23.579] Logging node info for node k8s-master-15826033-0 I1105 02:07:23.658] Nov 5 02:07:23.655: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-15826033-0,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-15826033-0,UID:aceb0231-4645-4a9a-b296-90ada851986b,ResourceVersion:6783,Generation:0,CreationTimestamp:2019-11-05 01:35:06 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 1,kubernetes.azure.com/cluster: kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-15826033-0,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.4.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5/providers/Microsoft.Compute/virtualMachines/k8s-master-15826033-0,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 01:35:20 +0000 UTC 2019-11-05 01:35:20 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 02:06:31 +0000 UTC 2019-11-05 01:34:57 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 02:06:31 +0000 UTC 2019-11-05 01:34:57 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 02:06:31 +0000 UTC 2019-11-05 01:34:57 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 02:06:31 +0000 UTC 2019-11-05 01:35:07 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.255.5} {Hostname k8s-master-15826033-0}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:1915756c22a540dc8eb30493987e9a43,SystemUUID:742C969C-5A71-D744-8E61-F27E38B56EB2,BootID:03f0672e-8fa9-4309-956f-8e3c371902c3,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 02:07:23.658] Nov 5 02:07:23.656: INFO: I1105 02:07:23.658] Logging kubelet events for node k8s-master-15826033-0 I1105 02:07:23.738] Nov 5 02:07:23.738: INFO: I1105 02:07:23.739] Logging pods the kubelet thinks is on node k8s-master-15826033-0 I1105 02:07:23.822] Nov 5 02:07:23.822: INFO: cloud-controller-manager-k8s-master-15826033-0 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:07:23.822] Nov 5 02:07:23.822: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 02:07:23.823] Nov 5 02:07:23.822: INFO: kube-addon-manager-k8s-master-15826033-0 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:07:23.823] Nov 5 02:07:23.822: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 02:07:23.823] Nov 5 02:07:23.822: INFO: kube-apiserver-k8s-master-15826033-0 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:07:23.823] Nov 5 02:07:23.822: INFO: Container kube-apiserver ready: true, restart count 0 I1105 02:07:23.824] Nov 5 02:07:23.822: INFO: kube-controller-manager-k8s-master-15826033-0 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:07:23.824] Nov 5 02:07:23.822: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 02:07:23.824] Nov 5 02:07:23.822: INFO: kube-scheduler-k8s-master-15826033-0 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:07:23.824] Nov 5 02:07:23.822: INFO: Container kube-scheduler ready: true, restart count 0 I1105 02:07:23.825] Nov 5 02:07:23.822: INFO: azure-ip-masq-agent-wgdpx started at 2019-11-05 01:35:06 +0000 UTC (0+1 container statuses recorded) I1105 02:07:23.825] Nov 5 02:07:23.822: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 02:07:23.825] Nov 5 02:07:23.822: INFO: kube-proxy-7d2px started at 2019-11-05 01:35:06 +0000 UTC (0+1 container statuses recorded) I1105 02:07:23.825] Nov 5 02:07:23.822: INFO: Container kube-proxy ready: true, restart count 0 I1105 02:07:24.071] Nov 5 02:07:24.071: INFO: I1105 02:07:24.072] Latency metrics for node k8s-master-15826033-0 I1105 02:07:24.072] Nov 5 02:07:24.071: INFO: I1105 02:07:24.072] Logging node info for node k8s-master-15826033-1 I1105 02:07:24.151] Nov 5 02:07:24.149: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-15826033-1,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-15826033-1,UID:1a13ec61-c89a-456c-8c94-bfc8e717c2e7,ResourceVersion:6784,Generation:0,CreationTimestamp:2019-11-05 01:35:07 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 1,kubernetes.azure.com/cluster: kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-15826033-1,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.5.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5/providers/Microsoft.Compute/virtualMachines/k8s-master-15826033-1,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 01:35:31 +0000 UTC 2019-11-05 01:35:31 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 02:06:31 +0000 UTC 2019-11-05 01:35:00 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 02:06:31 +0000 UTC 2019-11-05 01:35:00 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 02:06:31 +0000 UTC 2019-11-05 01:35:00 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 02:06:31 +0000 UTC 2019-11-05 01:35:07 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.255.6} {Hostname k8s-master-15826033-1}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:e0f2bccddd5b4555b180d14081ab59a8,SystemUUID:AE126631-CAD2-B54F-B22E-1F499342A4D6,BootID:509b9a9d-b046-4861-aa35-8bcb77465de2,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 02:07:24.151] Nov 5 02:07:24.149: INFO: I1105 02:07:24.152] Logging kubelet events for node k8s-master-15826033-1 I1105 02:07:24.230] Nov 5 02:07:24.230: INFO: I1105 02:07:24.230] Logging pods the kubelet thinks is on node k8s-master-15826033-1 I1105 02:07:24.314] Nov 5 02:07:24.313: INFO: cloud-controller-manager-k8s-master-15826033-1 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:07:24.314] Nov 5 02:07:24.313: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 02:07:24.314] Nov 5 02:07:24.313: INFO: kube-addon-manager-k8s-master-15826033-1 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:07:24.314] Nov 5 02:07:24.313: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 02:07:24.315] Nov 5 02:07:24.313: INFO: kube-apiserver-k8s-master-15826033-1 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:07:24.315] Nov 5 02:07:24.313: INFO: Container kube-apiserver ready: true, restart count 0 I1105 02:07:24.315] Nov 5 02:07:24.313: INFO: kube-controller-manager-k8s-master-15826033-1 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:07:24.316] Nov 5 02:07:24.313: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 02:07:24.316] Nov 5 02:07:24.313: INFO: kube-scheduler-k8s-master-15826033-1 started at 2019-11-05 01:36:16 +0000 UTC (0+1 container statuses recorded) I1105 02:07:24.316] Nov 5 02:07:24.313: INFO: Container kube-scheduler ready: true, restart count 0 I1105 02:07:24.316] Nov 5 02:07:24.313: INFO: azure-ip-masq-agent-w6dpr started at 2019-11-05 01:35:10 +0000 UTC (0+1 container statuses recorded) I1105 02:07:24.317] Nov 5 02:07:24.313: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 02:07:24.317] Nov 5 02:07:24.313: INFO: kube-proxy-8kmr5 started at 2019-11-05 01:35:10 +0000 UTC (0+1 container statuses recorded) I1105 02:07:24.317] Nov 5 02:07:24.313: INFO: Container kube-proxy ready: true, restart count 0 I1105 02:07:24.568] Nov 5 02:07:24.568: INFO: I1105 02:07:24.568] Latency metrics for node k8s-master-15826033-1 I1105 02:07:24.569] Nov 5 02:07:24.568: INFO: I1105 02:07:24.569] Logging node info for node k8s-master-15826033-2 I1105 02:07:24.649] Nov 5 02:07:24.646: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-15826033-2,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-15826033-2,UID:acb3a851-d6df-4549-ba56-4b7e96553c17,ResourceVersion:6876,Generation:0,CreationTimestamp:2019-11-05 01:34:09 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 2,kubernetes.azure.com/cluster: kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-15826033-2,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.1.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5/providers/Microsoft.Compute/virtualMachines/k8s-master-15826033-2,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 01:34:44 +0000 UTC 2019-11-05 01:34:44 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 02:07:03 +0000 UTC 2019-11-05 01:34:07 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 02:07:03 +0000 UTC 2019-11-05 01:34:07 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 02:07:03 +0000 UTC 2019-11-05 01:34:07 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 02:07:03 +0000 UTC 2019-11-05 01:34:29 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{Hostname k8s-master-15826033-2} {InternalIP 10.240.255.7}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:0885b0c2f6174106bde4fa0eae2cbc6b,SystemUUID:C176121C-5D7A-6243-9B05-8B68F8022B64,BootID:30430638-7ab6-4761-b984-2387e1a3f5b2,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 02:07:24.649] Nov 5 02:07:24.646: INFO: I1105 02:07:24.649] Logging kubelet events for node k8s-master-15826033-2 I1105 02:07:24.729] Nov 5 02:07:24.729: INFO: I1105 02:07:24.729] Logging pods the kubelet thinks is on node k8s-master-15826033-2 I1105 02:07:24.818] Nov 5 02:07:24.818: INFO: azure-ip-masq-agent-hw7s5 started at 2019-11-05 01:34:34 +0000 UTC (0+1 container statuses recorded) I1105 02:07:24.818] Nov 5 02:07:24.818: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 02:07:24.819] Nov 5 02:07:24.818: INFO: kube-proxy-5xpp6 started at 2019-11-05 01:34:34 +0000 UTC (0+1 container statuses recorded) I1105 02:07:24.819] Nov 5 02:07:24.818: INFO: Container kube-proxy ready: true, restart count 0 I1105 02:07:24.819] Nov 5 02:07:24.818: INFO: cloud-controller-manager-k8s-master-15826033-2 started at 2019-11-05 01:33:54 +0000 UTC (0+1 container statuses recorded) I1105 02:07:24.819] Nov 5 02:07:24.818: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 02:07:24.820] Nov 5 02:07:24.818: INFO: kube-addon-manager-k8s-master-15826033-2 started at 2019-11-05 01:33:54 +0000 UTC (0+1 container statuses recorded) I1105 02:07:24.820] Nov 5 02:07:24.818: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 02:07:24.820] Nov 5 02:07:24.818: INFO: kube-apiserver-k8s-master-15826033-2 started at 2019-11-05 01:33:54 +0000 UTC (0+1 container statuses recorded) I1105 02:07:24.820] Nov 5 02:07:24.818: INFO: Container kube-apiserver ready: true, restart count 0 I1105 02:07:24.820] Nov 5 02:07:24.818: INFO: kube-controller-manager-k8s-master-15826033-2 started at 2019-11-05 01:33:54 +0000 UTC (0+1 container statuses recorded) I1105 02:07:24.821] Nov 5 02:07:24.818: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 02:07:24.821] Nov 5 02:07:24.818: INFO: kube-scheduler-k8s-master-15826033-2 started at 2019-11-05 01:33:54 +0000 UTC (0+1 container statuses recorded) I1105 02:07:24.821] Nov 5 02:07:24.818: INFO: Container kube-scheduler ready: true, restart count 0 I1105 02:07:25.069] Nov 5 02:07:25.069: INFO: I1105 02:07:25.070] Latency metrics for node k8s-master-15826033-2 I1105 02:07:25.070] Nov 5 02:07:25.069: INFO: I1105 02:07:25.070] Logging node info for node k8s-master-15826033-3 I1105 02:07:25.149] Nov 5 02:07:25.147: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-15826033-3,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-15826033-3,UID:b7702272-1876-434e-b6a0-17fa34c6757f,ResourceVersion:6780,Generation:0,CreationTimestamp:2019-11-05 01:35:06 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 0,kubernetes.azure.com/cluster: kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-15826033-3,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.3.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5/providers/Microsoft.Compute/virtualMachines/k8s-master-15826033-3,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 01:35:17 +0000 UTC 2019-11-05 01:35:17 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 02:06:30 +0000 UTC 2019-11-05 01:35:00 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 02:06:30 +0000 UTC 2019-11-05 01:35:00 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 02:06:30 +0000 UTC 2019-11-05 01:35:00 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 02:06:30 +0000 UTC 2019-11-05 01:35:06 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.255.8} {Hostname k8s-master-15826033-3}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:fbf366963961422c9d90bedf6639c8e1,SystemUUID:5388948D-1A8D-EA41-B8B8-7F4523EDF5AA,BootID:42f5884c-7418-486a-8cf3-75188a331f38,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 02:07:25.150] Nov 5 02:07:25.147: INFO: I1105 02:07:25.150] Logging kubelet events for node k8s-master-15826033-3 I1105 02:07:25.229] Nov 5 02:07:25.229: INFO: I1105 02:07:25.229] Logging pods the kubelet thinks is on node k8s-master-15826033-3 I1105 02:07:25.313] Nov 5 02:07:25.312: INFO: kube-scheduler-k8s-master-15826033-3 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:07:25.313] Nov 5 02:07:25.312: INFO: Container kube-scheduler ready: true, restart count 0 I1105 02:07:25.313] Nov 5 02:07:25.312: INFO: kube-proxy-xz24z started at 2019-11-05 01:35:06 +0000 UTC (0+1 container statuses recorded) I1105 02:07:25.313] Nov 5 02:07:25.312: INFO: Container kube-proxy ready: true, restart count 0 I1105 02:07:25.313] Nov 5 02:07:25.312: INFO: azure-ip-masq-agent-dqfpw started at 2019-11-05 01:35:06 +0000 UTC (0+1 container statuses recorded) I1105 02:07:25.314] Nov 5 02:07:25.313: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 02:07:25.314] Nov 5 02:07:25.313: INFO: cloud-controller-manager-k8s-master-15826033-3 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:07:25.314] Nov 5 02:07:25.313: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 02:07:25.314] Nov 5 02:07:25.313: INFO: kube-addon-manager-k8s-master-15826033-3 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:07:25.314] Nov 5 02:07:25.313: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 02:07:25.314] Nov 5 02:07:25.313: INFO: kube-apiserver-k8s-master-15826033-3 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:07:25.315] Nov 5 02:07:25.313: INFO: Container kube-apiserver ready: true, restart count 0 I1105 02:07:25.315] Nov 5 02:07:25.313: INFO: kube-controller-manager-k8s-master-15826033-3 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:07:25.315] Nov 5 02:07:25.313: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 02:07:25.579] Nov 5 02:07:25.579: INFO: I1105 02:07:25.579] Latency metrics for node k8s-master-15826033-3 I1105 02:07:25.579] Nov 5 02:07:25.579: INFO: I1105 02:07:25.579] Logging node info for node k8s-master-15826033-4 I1105 02:07:25.659] Nov 5 02:07:25.657: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-15826033-4,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-15826033-4,UID:276daef1-6c05-41e3-bacd-abc93d35cc6d,ResourceVersion:6787,Generation:0,CreationTimestamp:2019-11-05 01:35:07 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 0,kubernetes.azure.com/cluster: kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-15826033-4,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.6.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5/providers/Microsoft.Compute/virtualMachines/k8s-master-15826033-4,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 01:35:41 +0000 UTC 2019-11-05 01:35:41 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 02:06:32 +0000 UTC 2019-11-05 01:35:02 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 02:06:32 +0000 UTC 2019-11-05 01:35:02 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 02:06:32 +0000 UTC 2019-11-05 01:35:02 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 02:06:32 +0000 UTC 2019-11-05 01:35:08 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.255.9} {Hostname k8s-master-15826033-4}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:79051fc0bdeb4e32a56acbaf5972ce3a,SystemUUID:C71606B5-7F08-9341-A26D-8A8D9D84FCAE,BootID:4bdf7cf3-2b83-42c7-a7ba-926ebe9ca75b,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 02:07:25.660] Nov 5 02:07:25.657: INFO: I1105 02:07:25.660] Logging kubelet events for node k8s-master-15826033-4 I1105 02:07:25.739] Nov 5 02:07:25.739: INFO: I1105 02:07:25.739] Logging pods the kubelet thinks is on node k8s-master-15826033-4 I1105 02:07:25.823] Nov 5 02:07:25.823: INFO: kube-apiserver-k8s-master-15826033-4 started at 2019-11-05 01:34:55 +0000 UTC (0+1 container statuses recorded) I1105 02:07:25.824] Nov 5 02:07:25.823: INFO: Container kube-apiserver ready: true, restart count 0 I1105 02:07:25.824] Nov 5 02:07:25.823: INFO: kube-controller-manager-k8s-master-15826033-4 started at 2019-11-05 01:34:55 +0000 UTC (0+1 container statuses recorded) I1105 02:07:25.824] Nov 5 02:07:25.823: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 02:07:25.824] Nov 5 02:07:25.823: INFO: kube-scheduler-k8s-master-15826033-4 started at 2019-11-05 01:34:55 +0000 UTC (0+1 container statuses recorded) I1105 02:07:25.825] Nov 5 02:07:25.823: INFO: Container kube-scheduler ready: true, restart count 0 I1105 02:07:25.825] Nov 5 02:07:25.823: INFO: cloud-controller-manager-k8s-master-15826033-4 started at 2019-11-05 01:34:56 +0000 UTC (0+1 container statuses recorded) I1105 02:07:25.825] Nov 5 02:07:25.823: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 02:07:25.825] Nov 5 02:07:25.823: INFO: kube-addon-manager-k8s-master-15826033-4 started at 2019-11-05 01:34:56 +0000 UTC (0+1 container statuses recorded) I1105 02:07:25.826] Nov 5 02:07:25.823: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 02:07:25.826] Nov 5 02:07:25.823: INFO: azure-ip-masq-agent-9nrj8 started at 2019-11-05 01:35:11 +0000 UTC (0+1 container statuses recorded) I1105 02:07:25.826] Nov 5 02:07:25.823: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 02:07:25.826] Nov 5 02:07:25.823: INFO: kube-proxy-rmngk started at 2019-11-05 01:35:11 +0000 UTC (0+1 container statuses recorded) I1105 02:07:25.827] Nov 5 02:07:25.823: INFO: Container kube-proxy ready: true, restart count 0 I1105 02:07:26.079] Nov 5 02:07:26.079: INFO: I1105 02:07:26.080] Latency metrics for node k8s-master-15826033-4 I1105 02:07:26.080] Nov 5 02:07:26.079: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready I1105 02:07:26.158] STEP: Destroying namespace "azuredisk-104" for this suite. I1105 02:07:40.399] Nov 5 02:07:40.399: INFO: Waiting up to 30s for server preferred namespaced resources to be successfully discovered I1105 02:07:43.386] Nov 5 02:07:43.385: INFO: namespace azuredisk-104 deletion completed in 17.227097882s I1105 02:07:43.386] I1105 02:07:43.386] • Failure [323.429 seconds] I1105 02:07:43.386] [azuredisk-csi-e2e] Dynamic Provisioning I1105 02:07:43.386] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:33 I1105 02:07:43.387] [multi-az] I1105 02:07:43.387] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:40 I1105 02:07:43.387] should create a volume on demand and mount it as readOnly in a pod [It] I1105 02:07:43.387] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:126 I1105 02:07:43.387] I1105 02:07:43.388] Unexpected error: I1105 02:07:43.388] <*errors.errorString | 0xc000e49b60>: { I1105 02:07:43.388] s: "PersistentVolumeClaims [pvc-q9qpz] not all in phase Bound within 5m0s", I1105 02:07:43.388] } I1105 02:07:43.388] PersistentVolumeClaims [pvc-q9qpz] not all in phase Bound within 5m0s I1105 02:07:43.388] occurred I1105 02:07:43.388] I1105 02:07:43.388] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/testsuites/testsuites.go:278 I1105 02:07:43.389] ------------------------------ I1105 02:07:43.389] [azuredisk-csi-e2e] Dynamic Provisioning [single-az] I1105 02:07:43.389] should create multiple PV objects, bind to PVCs and attach all to different pods on the same node I1105 02:07:43.389] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:151 I1105 02:07:43.389] [BeforeEach] [single-az] I1105 02:07:43.389] /go/src/sigs.k8s.io/azuredisk-csi-driver/vendor/k8s.io/kubernetes/test/e2e/framework/framework.go:150 I1105 02:07:43.389] STEP: Creating a kubernetes client I1105 02:07:43.390] Nov 5 02:07:43.386: INFO: >>> kubeConfig: /workspace/aks978705698/kubeconfig/kubeconfig.eastus2.json I1105 02:07:43.390] STEP: Building a namespace api object, basename azuredisk I1105 02:07:43.623] STEP: Binding the e2e-test-privileged-psp PodSecurityPolicy to the default service account in azuredisk-1141 I1105 02:07:43.887] STEP: Waiting for a default service account to be provisioned in namespace I1105 02:07:43.964] [BeforeEach] [single-az] I1105 02:07:43.964] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:58 I1105 02:07:43.964] [It] should create multiple PV objects, bind to PVCs and attach all to different pods on the same node I1105 02:07:43.964] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:151 I1105 02:07:43.964] STEP: setting up the StorageClass I1105 02:07:43.964] STEP: creating a StorageClass I1105 02:07:44.045] STEP: setting up the PVC and PV I1105 02:07:44.045] STEP: creating a PVC I1105 02:07:44.134] STEP: waiting for PVC to be in phase "Bound" I1105 02:07:44.134] Nov 5 02:07:44.134: INFO: Waiting up to 5m0s for PersistentVolumeClaims [pvc-rkt48] to have phase Bound I1105 02:07:44.211] Nov 5 02:07:44.211: INFO: PersistentVolumeClaim pvc-rkt48 found but phase is Pending instead of Bound. I1105 02:07:46.290] Nov 5 02:07:46.289: INFO: PersistentVolumeClaim pvc-rkt48 found but phase is Pending instead of Bound. I1105 02:07:48.368] Nov 5 02:07:48.368: INFO: PersistentVolumeClaim pvc-rkt48 found but phase is Pending instead of Bound. I1105 02:07:50.446] Nov 5 02:07:50.446: INFO: PersistentVolumeClaim pvc-rkt48 found but phase is Pending instead of Bound. I1105 02:07:52.529] Nov 5 02:07:52.528: INFO: PersistentVolumeClaim pvc-rkt48 found but phase is Pending instead of Bound. I1105 02:07:54.607] Nov 5 02:07:54.607: INFO: PersistentVolumeClaim pvc-rkt48 found but phase is Pending instead of Bound. I1105 02:07:56.685] Nov 5 02:07:56.685: INFO: PersistentVolumeClaim pvc-rkt48 found but phase is Pending instead of Bound. I1105 02:07:58.763] Nov 5 02:07:58.763: INFO: PersistentVolumeClaim pvc-rkt48 found but phase is Pending instead of Bound. I1105 02:08:00.842] Nov 5 02:08:00.842: INFO: PersistentVolumeClaim pvc-rkt48 found but phase is Pending instead of Bound. I1105 02:08:02.920] Nov 5 02:08:02.920: INFO: PersistentVolumeClaim pvc-rkt48 found but phase is Pending instead of Bound. I1105 02:08:04.998] Nov 5 02:08:04.998: INFO: PersistentVolumeClaim pvc-rkt48 found but phase is Pending instead of Bound. I1105 02:08:07.076] Nov 5 02:08:07.076: INFO: PersistentVolumeClaim pvc-rkt48 found but phase is Pending instead of Bound. I1105 02:08:09.162] Nov 5 02:08:09.162: INFO: PersistentVolumeClaim pvc-rkt48 found but phase is Pending instead of Bound. I1105 02:08:11.241] Nov 5 02:08:11.240: INFO: PersistentVolumeClaim pvc-rkt48 found but phase is Pending instead of Bound. I1105 02:08:13.319] Nov 5 02:08:13.318: INFO: PersistentVolumeClaim pvc-rkt48 found but phase is Pending instead of Bound. I1105 02:08:15.397] Nov 5 02:08:15.397: INFO: PersistentVolumeClaim pvc-rkt48 found but phase is Pending instead of Bound. I1105 02:08:17.475] Nov 5 02:08:17.475: INFO: PersistentVolumeClaim pvc-rkt48 found but phase is Pending instead of Bound. I1105 02:08:19.553] Nov 5 02:08:19.553: INFO: PersistentVolumeClaim pvc-rkt48 found but phase is Pending instead of Bound. I1105 02:08:21.632] Nov 5 02:08:21.632: INFO: PersistentVolumeClaim pvc-rkt48 found but phase is Pending instead of Bound. I1105 02:08:23.710] Nov 5 02:08:23.710: INFO: PersistentVolumeClaim pvc-rkt48 found but phase is Pending instead of Bound. I1105 02:08:25.789] Nov 5 02:08:25.788: INFO: PersistentVolumeClaim pvc-rkt48 found but phase is Pending instead of Bound. I1105 02:08:27.867] Nov 5 02:08:27.867: INFO: PersistentVolumeClaim pvc-rkt48 found but phase is Pending instead of Bound. I1105 02:08:29.945] Nov 5 02:08:29.945: INFO: PersistentVolumeClaim pvc-rkt48 found but phase is Pending instead of Bound. I1105 02:08:32.026] Nov 5 02:08:32.026: INFO: PersistentVolumeClaim pvc-rkt48 found but phase is Pending instead of Bound. I1105 02:08:34.104] Nov 5 02:08:34.104: INFO: PersistentVolumeClaim pvc-rkt48 found but phase is Pending instead of Bound. I1105 02:08:36.183] Nov 5 02:08:36.183: INFO: PersistentVolumeClaim pvc-rkt48 found but phase is Pending instead of Bound. I1105 02:08:38.261] Nov 5 02:08:38.261: INFO: PersistentVolumeClaim pvc-rkt48 found but phase is Pending instead of Bound. I1105 02:08:40.340] Nov 5 02:08:40.340: INFO: PersistentVolumeClaim pvc-rkt48 found but phase is Pending instead of Bound. I1105 02:08:42.418] Nov 5 02:08:42.418: INFO: PersistentVolumeClaim pvc-rkt48 found but phase is Pending instead of Bound. I1105 02:08:44.496] Nov 5 02:08:44.496: INFO: PersistentVolumeClaim pvc-rkt48 found but phase is Pending instead of Bound. I1105 02:08:46.575] Nov 5 02:08:46.575: INFO: PersistentVolumeClaim pvc-rkt48 found but phase is Pending instead of Bound. I1105 02:08:48.654] Nov 5 02:08:48.654: INFO: PersistentVolumeClaim pvc-rkt48 found but phase is Pending instead of Bound. I1105 02:08:50.732] Nov 5 02:08:50.731: INFO: PersistentVolumeClaim pvc-rkt48 found but phase is Pending instead of Bound. I1105 02:08:52.810] Nov 5 02:08:52.810: INFO: PersistentVolumeClaim pvc-rkt48 found but phase is Pending instead of Bound. I1105 02:08:54.888] Nov 5 02:08:54.887: INFO: PersistentVolumeClaim pvc-rkt48 found but phase is Pending instead of Bound. I1105 02:08:56.965] Nov 5 02:08:56.965: INFO: PersistentVolumeClaim pvc-rkt48 found but phase is Pending instead of Bound. I1105 02:08:59.044] Nov 5 02:08:59.044: INFO: PersistentVolumeClaim pvc-rkt48 found but phase is Pending instead of Bound. I1105 02:09:01.129] Nov 5 02:09:01.128: INFO: PersistentVolumeClaim pvc-rkt48 found but phase is Pending instead of Bound. I1105 02:09:03.206] Nov 5 02:09:03.206: INFO: PersistentVolumeClaim pvc-rkt48 found but phase is Pending instead of Bound. I1105 02:09:05.284] Nov 5 02:09:05.284: INFO: PersistentVolumeClaim pvc-rkt48 found but phase is Pending instead of Bound. I1105 02:09:07.362] Nov 5 02:09:07.362: INFO: PersistentVolumeClaim pvc-rkt48 found but phase is Pending instead of Bound. I1105 02:09:09.440] Nov 5 02:09:09.440: INFO: PersistentVolumeClaim pvc-rkt48 found but phase is Pending instead of Bound. I1105 02:09:11.518] Nov 5 02:09:11.517: INFO: PersistentVolumeClaim pvc-rkt48 found but phase is Pending instead of Bound. I1105 02:09:13.596] Nov 5 02:09:13.596: INFO: PersistentVolumeClaim pvc-rkt48 found but phase is Pending instead of Bound. I1105 02:09:15.674] Nov 5 02:09:15.674: INFO: PersistentVolumeClaim pvc-rkt48 found but phase is Pending instead of Bound. I1105 02:09:17.753] Nov 5 02:09:17.753: INFO: PersistentVolumeClaim pvc-rkt48 found but phase is Pending instead of Bound. I1105 02:09:19.831] Nov 5 02:09:19.830: INFO: PersistentVolumeClaim pvc-rkt48 found but phase is Pending instead of Bound. I1105 02:09:21.908] Nov 5 02:09:21.908: INFO: PersistentVolumeClaim pvc-rkt48 found but phase is Pending instead of Bound. I1105 02:09:23.991] Nov 5 02:09:23.991: INFO: PersistentVolumeClaim pvc-rkt48 found but phase is Pending instead of Bound. I1105 02:09:26.069] Nov 5 02:09:26.069: INFO: PersistentVolumeClaim pvc-rkt48 found but phase is Pending instead of Bound. I1105 02:09:28.148] Nov 5 02:09:28.148: INFO: PersistentVolumeClaim pvc-rkt48 found but phase is Pending instead of Bound. I1105 02:09:30.226] Nov 5 02:09:30.226: INFO: PersistentVolumeClaim pvc-rkt48 found but phase is Pending instead of Bound. I1105 02:09:32.304] Nov 5 02:09:32.303: INFO: PersistentVolumeClaim pvc-rkt48 found but phase is Pending instead of Bound. I1105 02:09:34.382] Nov 5 02:09:34.382: INFO: PersistentVolumeClaim pvc-rkt48 found but phase is Pending instead of Bound. I1105 02:09:36.466] Nov 5 02:09:36.466: INFO: PersistentVolumeClaim pvc-rkt48 found but phase is Pending instead of Bound. I1105 02:09:38.544] Nov 5 02:09:38.544: INFO: PersistentVolumeClaim pvc-rkt48 found but phase is Pending instead of Bound. I1105 02:09:40.632] Nov 5 02:09:40.631: INFO: PersistentVolumeClaim pvc-rkt48 found but phase is Pending instead of Bound. I1105 02:09:42.709] Nov 5 02:09:42.709: INFO: PersistentVolumeClaim pvc-rkt48 found but phase is Pending instead of Bound. I1105 02:09:44.787] Nov 5 02:09:44.787: INFO: PersistentVolumeClaim pvc-rkt48 found but phase is Pending instead of Bound. I1105 02:09:46.865] Nov 5 02:09:46.865: INFO: PersistentVolumeClaim pvc-rkt48 found but phase is Pending instead of Bound. I1105 02:09:48.942] Nov 5 02:09:48.942: INFO: PersistentVolumeClaim pvc-rkt48 found but phase is Pending instead of Bound. I1105 02:09:51.023] Nov 5 02:09:51.022: INFO: PersistentVolumeClaim pvc-rkt48 found but phase is Pending instead of Bound. I1105 02:09:53.127] Nov 5 02:09:53.127: INFO: PersistentVolumeClaim pvc-rkt48 found but phase is Pending instead of Bound. I1105 02:09:55.205] Nov 5 02:09:55.205: INFO: PersistentVolumeClaim pvc-rkt48 found but phase is Pending instead of Bound. I1105 02:09:57.284] Nov 5 02:09:57.284: INFO: PersistentVolumeClaim pvc-rkt48 found but phase is Pending instead of Bound. I1105 02:09:59.366] Nov 5 02:09:59.366: INFO: PersistentVolumeClaim pvc-rkt48 found but phase is Pending instead of Bound. I1105 02:10:01.444] Nov 5 02:10:01.444: INFO: PersistentVolumeClaim pvc-rkt48 found but phase is Pending instead of Bound. I1105 02:10:03.522] Nov 5 02:10:03.522: INFO: PersistentVolumeClaim pvc-rkt48 found but phase is Pending instead of Bound. I1105 02:10:05.599] Nov 5 02:10:05.599: INFO: PersistentVolumeClaim pvc-rkt48 found but phase is Pending instead of Bound. I1105 02:10:07.678] Nov 5 02:10:07.678: INFO: PersistentVolumeClaim pvc-rkt48 found but phase is Pending instead of Bound. I1105 02:10:09.756] Nov 5 02:10:09.755: INFO: PersistentVolumeClaim pvc-rkt48 found but phase is Pending instead of Bound. I1105 02:10:11.833] Nov 5 02:10:11.833: INFO: PersistentVolumeClaim pvc-rkt48 found but phase is Pending instead of Bound. I1105 02:10:13.911] Nov 5 02:10:13.911: INFO: PersistentVolumeClaim pvc-rkt48 found but phase is Pending instead of Bound. I1105 02:10:15.990] Nov 5 02:10:15.989: INFO: PersistentVolumeClaim pvc-rkt48 found but phase is Pending instead of Bound. I1105 02:10:18.068] Nov 5 02:10:18.068: INFO: PersistentVolumeClaim pvc-rkt48 found but phase is Pending instead of Bound. I1105 02:10:20.145] Nov 5 02:10:20.145: INFO: PersistentVolumeClaim pvc-rkt48 found but phase is Pending instead of Bound. I1105 02:10:22.223] Nov 5 02:10:22.223: INFO: PersistentVolumeClaim pvc-rkt48 found but phase is Pending instead of Bound. I1105 02:10:24.300] Nov 5 02:10:24.300: INFO: PersistentVolumeClaim pvc-rkt48 found but phase is Pending instead of Bound. I1105 02:10:26.378] Nov 5 02:10:26.378: INFO: PersistentVolumeClaim pvc-rkt48 found but phase is Pending instead of Bound. I1105 02:10:28.466] Nov 5 02:10:28.466: INFO: PersistentVolumeClaim pvc-rkt48 found but phase is Pending instead of Bound. I1105 02:10:30.544] Nov 5 02:10:30.544: INFO: PersistentVolumeClaim pvc-rkt48 found but phase is Pending instead of Bound. I1105 02:10:32.622] Nov 5 02:10:32.621: INFO: PersistentVolumeClaim pvc-rkt48 found but phase is Pending instead of Bound. I1105 02:10:34.700] Nov 5 02:10:34.700: INFO: PersistentVolumeClaim pvc-rkt48 found but phase is Pending instead of Bound. I1105 02:10:36.779] Nov 5 02:10:36.778: INFO: PersistentVolumeClaim pvc-rkt48 found but phase is Pending instead of Bound. I1105 02:10:38.862] Nov 5 02:10:38.862: INFO: PersistentVolumeClaim pvc-rkt48 found but phase is Pending instead of Bound. I1105 02:10:40.940] Nov 5 02:10:40.940: INFO: PersistentVolumeClaim pvc-rkt48 found but phase is Pending instead of Bound. I1105 02:10:43.018] Nov 5 02:10:43.018: INFO: PersistentVolumeClaim pvc-rkt48 found but phase is Pending instead of Bound. I1105 02:10:45.117] Nov 5 02:10:45.117: INFO: PersistentVolumeClaim pvc-rkt48 found but phase is Pending instead of Bound. I1105 02:10:47.195] Nov 5 02:10:47.195: INFO: PersistentVolumeClaim pvc-rkt48 found but phase is Pending instead of Bound. I1105 02:10:49.273] Nov 5 02:10:49.273: INFO: PersistentVolumeClaim pvc-rkt48 found but phase is Pending instead of Bound. I1105 02:10:51.351] Nov 5 02:10:51.351: INFO: PersistentVolumeClaim pvc-rkt48 found but phase is Pending instead of Bound. I1105 02:10:53.428] Nov 5 02:10:53.428: INFO: PersistentVolumeClaim pvc-rkt48 found but phase is Pending instead of Bound. I1105 02:10:55.506] Nov 5 02:10:55.506: INFO: PersistentVolumeClaim pvc-rkt48 found but phase is Pending instead of Bound. I1105 02:10:57.583] Nov 5 02:10:57.583: INFO: PersistentVolumeClaim pvc-rkt48 found but phase is Pending instead of Bound. I1105 02:10:59.661] Nov 5 02:10:59.661: INFO: PersistentVolumeClaim pvc-rkt48 found but phase is Pending instead of Bound. I1105 02:11:01.739] Nov 5 02:11:01.739: INFO: PersistentVolumeClaim pvc-rkt48 found but phase is Pending instead of Bound. I1105 02:11:03.817] Nov 5 02:11:03.816: INFO: PersistentVolumeClaim pvc-rkt48 found but phase is Pending instead of Bound. I1105 02:11:05.894] Nov 5 02:11:05.894: INFO: PersistentVolumeClaim pvc-rkt48 found but phase is Pending instead of Bound. I1105 02:11:07.972] Nov 5 02:11:07.971: INFO: PersistentVolumeClaim pvc-rkt48 found but phase is Pending instead of Bound. I1105 02:11:10.049] Nov 5 02:11:10.049: INFO: PersistentVolumeClaim pvc-rkt48 found but phase is Pending instead of Bound. I1105 02:11:12.126] Nov 5 02:11:12.126: INFO: PersistentVolumeClaim pvc-rkt48 found but phase is Pending instead of Bound. I1105 02:11:14.204] Nov 5 02:11:14.204: INFO: PersistentVolumeClaim pvc-rkt48 found but phase is Pending instead of Bound. I1105 02:11:16.282] Nov 5 02:11:16.282: INFO: PersistentVolumeClaim pvc-rkt48 found but phase is Pending instead of Bound. I1105 02:11:18.360] Nov 5 02:11:18.359: INFO: PersistentVolumeClaim pvc-rkt48 found but phase is Pending instead of Bound. I1105 02:11:20.438] Nov 5 02:11:20.438: INFO: PersistentVolumeClaim pvc-rkt48 found but phase is Pending instead of Bound. I1105 02:11:22.516] Nov 5 02:11:22.516: INFO: PersistentVolumeClaim pvc-rkt48 found but phase is Pending instead of Bound. I1105 02:11:24.594] Nov 5 02:11:24.594: INFO: PersistentVolumeClaim pvc-rkt48 found but phase is Pending instead of Bound. I1105 02:11:26.671] Nov 5 02:11:26.671: INFO: PersistentVolumeClaim pvc-rkt48 found but phase is Pending instead of Bound. I1105 02:11:28.750] Nov 5 02:11:28.749: INFO: PersistentVolumeClaim pvc-rkt48 found but phase is Pending instead of Bound. I1105 02:11:30.827] Nov 5 02:11:30.827: INFO: PersistentVolumeClaim pvc-rkt48 found but phase is Pending instead of Bound. I1105 02:11:32.905] Nov 5 02:11:32.904: INFO: PersistentVolumeClaim pvc-rkt48 found but phase is Pending instead of Bound. I1105 02:11:34.983] Nov 5 02:11:34.982: INFO: PersistentVolumeClaim pvc-rkt48 found but phase is Pending instead of Bound. I1105 02:11:37.060] Nov 5 02:11:37.060: INFO: PersistentVolumeClaim pvc-rkt48 found but phase is Pending instead of Bound. I1105 02:11:39.138] Nov 5 02:11:39.138: INFO: PersistentVolumeClaim pvc-rkt48 found but phase is Pending instead of Bound. I1105 02:11:41.215] Nov 5 02:11:41.215: INFO: PersistentVolumeClaim pvc-rkt48 found but phase is Pending instead of Bound. I1105 02:11:43.293] Nov 5 02:11:43.293: INFO: PersistentVolumeClaim pvc-rkt48 found but phase is Pending instead of Bound. I1105 02:11:45.370] Nov 5 02:11:45.370: INFO: PersistentVolumeClaim pvc-rkt48 found but phase is Pending instead of Bound. I1105 02:11:47.448] Nov 5 02:11:47.448: INFO: PersistentVolumeClaim pvc-rkt48 found but phase is Pending instead of Bound. I1105 02:11:49.526] Nov 5 02:11:49.525: INFO: PersistentVolumeClaim pvc-rkt48 found but phase is Pending instead of Bound. I1105 02:11:51.603] Nov 5 02:11:51.603: INFO: PersistentVolumeClaim pvc-rkt48 found but phase is Pending instead of Bound. I1105 02:11:53.681] Nov 5 02:11:53.681: INFO: PersistentVolumeClaim pvc-rkt48 found but phase is Pending instead of Bound. I1105 02:11:55.760] Nov 5 02:11:55.759: INFO: PersistentVolumeClaim pvc-rkt48 found but phase is Pending instead of Bound. I1105 02:11:57.838] Nov 5 02:11:57.837: INFO: PersistentVolumeClaim pvc-rkt48 found but phase is Pending instead of Bound. I1105 02:11:59.915] Nov 5 02:11:59.915: INFO: PersistentVolumeClaim pvc-rkt48 found but phase is Pending instead of Bound. I1105 02:12:01.993] Nov 5 02:12:01.993: INFO: PersistentVolumeClaim pvc-rkt48 found but phase is Pending instead of Bound. I1105 02:12:04.071] Nov 5 02:12:04.071: INFO: PersistentVolumeClaim pvc-rkt48 found but phase is Pending instead of Bound. I1105 02:12:06.149] Nov 5 02:12:06.148: INFO: PersistentVolumeClaim pvc-rkt48 found but phase is Pending instead of Bound. I1105 02:12:08.226] Nov 5 02:12:08.226: INFO: PersistentVolumeClaim pvc-rkt48 found but phase is Pending instead of Bound. I1105 02:12:10.303] Nov 5 02:12:10.303: INFO: PersistentVolumeClaim pvc-rkt48 found but phase is Pending instead of Bound. I1105 02:12:12.381] Nov 5 02:12:12.381: INFO: PersistentVolumeClaim pvc-rkt48 found but phase is Pending instead of Bound. I1105 02:12:14.459] Nov 5 02:12:14.459: INFO: PersistentVolumeClaim pvc-rkt48 found but phase is Pending instead of Bound. I1105 02:12:16.536] Nov 5 02:12:16.536: INFO: PersistentVolumeClaim pvc-rkt48 found but phase is Pending instead of Bound. I1105 02:12:18.614] Nov 5 02:12:18.614: INFO: PersistentVolumeClaim pvc-rkt48 found but phase is Pending instead of Bound. I1105 02:12:20.693] Nov 5 02:12:20.692: INFO: PersistentVolumeClaim pvc-rkt48 found but phase is Pending instead of Bound. I1105 02:12:22.770] Nov 5 02:12:22.770: INFO: PersistentVolumeClaim pvc-rkt48 found but phase is Pending instead of Bound. I1105 02:12:24.848] Nov 5 02:12:24.848: INFO: PersistentVolumeClaim pvc-rkt48 found but phase is Pending instead of Bound. I1105 02:12:26.926] Nov 5 02:12:26.925: INFO: PersistentVolumeClaim pvc-rkt48 found but phase is Pending instead of Bound. I1105 02:12:29.003] Nov 5 02:12:29.003: INFO: PersistentVolumeClaim pvc-rkt48 found but phase is Pending instead of Bound. I1105 02:12:31.081] Nov 5 02:12:31.081: INFO: PersistentVolumeClaim pvc-rkt48 found but phase is Pending instead of Bound. I1105 02:12:33.159] Nov 5 02:12:33.159: INFO: PersistentVolumeClaim pvc-rkt48 found but phase is Pending instead of Bound. I1105 02:12:35.237] Nov 5 02:12:35.236: INFO: PersistentVolumeClaim pvc-rkt48 found but phase is Pending instead of Bound. I1105 02:12:37.315] Nov 5 02:12:37.314: INFO: PersistentVolumeClaim pvc-rkt48 found but phase is Pending instead of Bound. I1105 02:12:39.392] Nov 5 02:12:39.392: INFO: PersistentVolumeClaim pvc-rkt48 found but phase is Pending instead of Bound. I1105 02:12:41.470] Nov 5 02:12:41.470: INFO: PersistentVolumeClaim pvc-rkt48 found but phase is Pending instead of Bound. I1105 02:12:43.548] Nov 5 02:12:43.548: INFO: PersistentVolumeClaim pvc-rkt48 found but phase is Pending instead of Bound. I1105 02:12:45.548] Nov 5 02:12:45.548: INFO: Unexpected error occurred: PersistentVolumeClaims [pvc-rkt48] not all in phase Bound within 5m0s I1105 02:12:45.549] [AfterEach] [single-az] I1105 02:12:45.549] /go/src/sigs.k8s.io/azuredisk-csi-driver/vendor/k8s.io/kubernetes/test/e2e/framework/framework.go:151 I1105 02:12:45.549] STEP: Collecting events from namespace "azuredisk-1141". I1105 02:12:45.627] STEP: Found 3 events. I1105 02:12:45.627] Nov 5 02:12:45.627: INFO: At 2019-11-05 02:07:44 +0000 UTC - event for pvc-rkt48: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "disk.csi.azure.com" or manually created by system administrator I1105 02:12:45.628] Nov 5 02:12:45.627: INFO: At 2019-11-05 02:07:44 +0000 UTC - event for pvc-rkt48: {disk.csi.azure.com_csi-azuredisk-controller-6f6587bfc6-8vkfl_1376914b-ff6d-11e9-93ab-ba2bee589b8c } Provisioning: External provisioner is provisioning volume for claim "azuredisk-1141/pvc-rkt48" I1105 02:12:45.628] Nov 5 02:12:45.627: INFO: At 2019-11-05 02:07:44 +0000 UTC - event for pvc-rkt48: {disk.csi.azure.com_csi-azuredisk-controller-6f6587bfc6-8vkfl_1376914b-ff6d-11e9-93ab-ba2bee589b8c } ProvisioningFailed: failed to provision volume with StorageClass "azuredisk-1141-disk.csi.azure.com-dynamic-sc-7cnrt": rpc error: code = Unknown desc = compute.DisksClient#CreateOrUpdate: Failure sending request: StatusCode=400 -- Original Error: Code="InvalidAvailabilityZone" Message="The zone(s) '' for resource 'Microsoft.Compute/disks/pvc-a4edd799-8b82-4ed1-849a-6368e074b2d7' is not supported. The supported zones for location 'eastus2' are '1,2,3'" I1105 02:12:45.705] Nov 5 02:12:45.704: INFO: POD NODE PHASE GRACE CONDITIONS I1105 02:12:45.705] Nov 5 02:12:45.704: INFO: I1105 02:12:45.936] Nov 5 02:12:45.936: INFO: I1105 02:12:45.936] Logging node info for node k8s-agentpool-15826033-vmss000000 I1105 02:12:46.017] Nov 5 02:12:46.013: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-agentpool-15826033-vmss000000,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-agentpool-15826033-vmss000000,UID:15fa874e-54ec-42cd-b243-d3baa512150f,ResourceVersion:7834,Generation:0,CreationTimestamp:2019-11-05 01:34:17 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{agentpool: agentpool,beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_D4s_v3,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 0,kubernetes.azure.com/cluster: kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5,kubernetes.azure.com/role: agent,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-agentpool-15826033-vmss000000,kubernetes.io/os: linux,kubernetes.io/role: agent,node-role.kubernetes.io/agent: ,storageprofile: managed,storagetier: Premium_LRS,topology.disk.csi.azure.com/zone: 0,},Annotations:map[string]string{csi.volume.kubernetes.io/nodeid: {"disk.csi.azure.com":"k8s-agentpool-15826033-vmss000000"},node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.2.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool-15826033-vmss/virtualMachines/0,Unschedulable:false,Taints:[],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-csi-disk.csi.azure.com: {{8 0} {} 8 DecimalSI},cpu: {{4 0} {} 4 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{16797044736 0} {} 16403364Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-csi-disk.csi.azure.com: {{8 0} {} 8 DecimalSI},cpu: {{4 0} {} 4 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{16010612736 0} {} 15635364Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 01:34:56 +0000 UTC 2019-11-05 01:34:56 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 02:12:23 +0000 UTC 2019-11-05 01:34:17 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 02:12:23 +0000 UTC 2019-11-05 01:34:17 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 02:12:23 +0000 UTC 2019-11-05 01:34:17 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 02:12:23 +0000 UTC 2019-11-05 01:34:29 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.0.4} {Hostname k8s-agentpool-15826033-vmss000000}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:afe00792177d49a98f6a44d03d08732d,SystemUUID:4F1777BF-A5C3-004E-81C4-90F0B54C02A0,BootID:13b6a628-ae89-44af-b891-89461030adcb,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[k8sprowinternal.azurecr.io/azuredisk-csi@sha256:75d4ff5d7cd845d6e810c67796ecb037e852fb849ec1eb5b3194564008e6d1e0 k8sprowinternal.azurecr.io/azuredisk-csi:e2e-3e3ed045c6faa509dbd8d2890ee9f56aa2934887] 286930414} {[k8s.gcr.io/kubernetes-dashboard-amd64@sha256:0ae6b69432e78069c5ce2bcde0fe409c5c4d6f0f4d9cd50a17974fea38898747 k8s.gcr.io/kubernetes-dashboard-amd64:v1.10.1] 121711221} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/metrics-server-amd64@sha256:49a9f12f7067d11f42c803dbe61ed2c1299959ad85cb315b25ff7eef8e6b8892 k8s.gcr.io/metrics-server-amd64:v0.2.1] 42541759} {[mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume@sha256:4fd30d43947d4a54fc89ead7985beecfd3c9b2a93a0655a373b1608ab90bd5af mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume:v0.0.7] 22909487} {[quay.io/k8scsi/livenessprobe@sha256:8c9c8a0f330acc2e558ae6624309447d02522e76aa35af23fa19b88abf4381cb quay.io/k8scsi/livenessprobe:v1.0.2] 17415381} {[quay.io/k8scsi/csi-node-driver-registrar@sha256:13daf82fb99e951a4bff8ae5fc7c17c3a8fe7130be6400990d8f6076c32d4599 quay.io/k8scsi/csi-node-driver-registrar:v1.1.0] 15815995} {[mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume@sha256:23d8c6033f02a1ecad05127ebdc931bb871264228661bc122704b0974e4d9fdd mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume:1.0.8] 1159025} {[busybox@sha256:8ccbac733d19c0dd4d70b4f0c1e12245b5fa3ad24758a11035ee505c629c0796 busybox:1.29] 1154361} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 02:12:46.017] Nov 5 02:12:46.014: INFO: I1105 02:12:46.017] Logging kubelet events for node k8s-agentpool-15826033-vmss000000 I1105 02:12:46.095] Nov 5 02:12:46.094: INFO: I1105 02:12:46.095] Logging pods the kubelet thinks is on node k8s-agentpool-15826033-vmss000000 I1105 02:12:46.177] Nov 5 02:12:46.177: INFO: blobfuse-flexvol-installer-bsvqt started at 2019-11-05 01:34:34 +0000 UTC (0+1 container statuses recorded) I1105 02:12:46.177] Nov 5 02:12:46.177: INFO: Container blobfuse-flexvol-installer ready: true, restart count 0 I1105 02:12:46.178] Nov 5 02:12:46.177: INFO: azure-ip-masq-agent-vx86b started at 2019-11-05 01:34:34 +0000 UTC (0+1 container statuses recorded) I1105 02:12:46.178] Nov 5 02:12:46.177: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 02:12:46.178] Nov 5 02:12:46.177: INFO: keyvault-flexvolume-qhnfr started at 2019-11-05 01:34:34 +0000 UTC (0+1 container statuses recorded) I1105 02:12:46.178] Nov 5 02:12:46.177: INFO: Container keyvault-flexvolume ready: true, restart count 0 I1105 02:12:46.179] Nov 5 02:12:46.177: INFO: metrics-server-864ffbc5c-sdxps started at 2019-11-05 01:34:34 +0000 UTC (0+1 container statuses recorded) I1105 02:12:46.179] Nov 5 02:12:46.177: INFO: Container metrics-server ready: true, restart count 0 I1105 02:12:46.179] Nov 5 02:12:46.177: INFO: kube-proxy-wxblc started at 2019-11-05 01:34:34 +0000 UTC (0+1 container statuses recorded) I1105 02:12:46.179] Nov 5 02:12:46.177: INFO: Container kube-proxy ready: true, restart count 0 I1105 02:12:46.180] Nov 5 02:12:46.177: INFO: kubernetes-dashboard-66dd8b8df7-ks284 started at 2019-11-05 01:34:35 +0000 UTC (0+1 container statuses recorded) I1105 02:12:46.180] Nov 5 02:12:46.177: INFO: Container kubernetes-dashboard ready: true, restart count 0 I1105 02:12:46.180] Nov 5 02:12:46.177: INFO: csi-azuredisk-node-whq2g started at 2019-11-05 01:38:24 +0000 UTC (0+3 container statuses recorded) I1105 02:12:46.181] Nov 5 02:12:46.177: INFO: Container azuredisk ready: true, restart count 0 I1105 02:12:46.181] Nov 5 02:12:46.177: INFO: Container liveness-probe ready: true, restart count 0 I1105 02:12:46.181] Nov 5 02:12:46.177: INFO: Container node-driver-registrar ready: true, restart count 0 I1105 02:12:46.430] Nov 5 02:12:46.430: INFO: I1105 02:12:46.430] Latency metrics for node k8s-agentpool-15826033-vmss000000 I1105 02:12:46.431] Nov 5 02:12:46.430: INFO: I1105 02:12:46.431] Logging node info for node k8s-agentpool-15826033-vmss000001 I1105 02:12:46.524] Nov 5 02:12:46.520: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-agentpool-15826033-vmss000001,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-agentpool-15826033-vmss000001,UID:4a8ea605-cc0b-4e3b-9060-d8867d269e6c,ResourceVersion:7899,Generation:0,CreationTimestamp:2019-11-05 01:34:09 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{agentpool: agentpool,beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_D4s_v3,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 1,kubernetes.azure.com/cluster: kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5,kubernetes.azure.com/role: agent,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-agentpool-15826033-vmss000001,kubernetes.io/os: linux,kubernetes.io/role: agent,node-role.kubernetes.io/agent: ,storageprofile: managed,storagetier: Premium_LRS,topology.disk.csi.azure.com/zone: 1,},Annotations:map[string]string{csi.volume.kubernetes.io/nodeid: {"disk.csi.azure.com":"k8s-agentpool-15826033-vmss000001"},node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.0.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool-15826033-vmss/virtualMachines/1,Unschedulable:false,Taints:[],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-csi-disk.csi.azure.com: {{8 0} {} 8 DecimalSI},cpu: {{4 0} {} 4 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{16797044736 0} {} 16403364Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-csi-disk.csi.azure.com: {{8 0} {} 8 DecimalSI},cpu: {{4 0} {} 4 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{16010612736 0} {} 15635364Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 01:34:57 +0000 UTC 2019-11-05 01:34:57 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 02:12:45 +0000 UTC 2019-11-05 01:34:08 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 02:12:45 +0000 UTC 2019-11-05 01:34:08 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 02:12:45 +0000 UTC 2019-11-05 01:34:08 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 02:12:45 +0000 UTC 2019-11-05 01:34:29 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.0.5} {Hostname k8s-agentpool-15826033-vmss000001}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:b6b9fbf4a2fd4fa5a7ae472415fe30b8,SystemUUID:0162B3A4-3955-9A46-ABD4-A37BD75848CC,BootID:ee93003f-d47c-4118-9975-57365b3443a5,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[k8sprowinternal.azurecr.io/azuredisk-csi@sha256:75d4ff5d7cd845d6e810c67796ecb037e852fb849ec1eb5b3194564008e6d1e0 k8sprowinternal.azurecr.io/azuredisk-csi:e2e-3e3ed045c6faa509dbd8d2890ee9f56aa2934887] 286930414} {[gcr.io/kubernetes-helm/tiller@sha256:f6d8f4ab9ba993b5f5b60a6edafe86352eabe474ffeb84cb6c79b8866dce45d1 gcr.io/kubernetes-helm/tiller:v2.11.0] 71821984} {[quay.io/k8scsi/csi-provisioner@sha256:d657c839dce87324fe2b677302913f9386f885f8746be7bea0ced5b0844e3433 quay.io/k8scsi/csi-provisioner:v1.3.1] 50596998} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[quay.io/k8scsi/csi-attacher@sha256:26fccd7a99d973845df1193b46ebdcc6ab8dc5f6e6be319750c471fce1742d13 quay.io/k8scsi/csi-attacher:v1.2.0] 46226754} {[quay.io/k8scsi/csi-cluster-driver-registrar@sha256:fafd75ae5442f192cfa8c2e792903aee30d5884b62e802e4464b0a895d21e3ef quay.io/k8scsi/csi-cluster-driver-registrar:v1.0.1] 45874691} {[quay.io/k8scsi/csi-snapshotter@sha256:a49e0da1af6f2bf717e41ba1eee8b5e6a1cbd66a709dd92cc43fe475fe2589eb quay.io/k8scsi/csi-snapshotter:v1.1.0] 44325352} {[k8s.gcr.io/coredns@sha256:e83beb5e43f8513fa735e77ffc5859640baea30a882a11cc75c4c3244a737d3c k8s.gcr.io/coredns:1.5.0] 42488424} {[mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume@sha256:4fd30d43947d4a54fc89ead7985beecfd3c9b2a93a0655a373b1608ab90bd5af mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume:v0.0.7] 22909487} {[quay.io/k8scsi/livenessprobe@sha256:8c9c8a0f330acc2e558ae6624309447d02522e76aa35af23fa19b88abf4381cb quay.io/k8scsi/livenessprobe:v1.0.2] 17415381} {[quay.io/k8scsi/csi-node-driver-registrar@sha256:13daf82fb99e951a4bff8ae5fc7c17c3a8fe7130be6400990d8f6076c32d4599 quay.io/k8scsi/csi-node-driver-registrar:v1.1.0] 15815995} {[quay.io/k8scsi/livenessprobe@sha256:dde617756e0f602adc566ab71fd885f1dad451ad3fb063ac991c95a2ff47aea5 quay.io/k8scsi/livenessprobe:v1.1.0] 14967303} {[mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume@sha256:23d8c6033f02a1ecad05127ebdc931bb871264228661bc122704b0974e4d9fdd mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume:1.0.8] 1159025} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 02:12:46.524] Nov 5 02:12:46.521: INFO: I1105 02:12:46.524] Logging kubelet events for node k8s-agentpool-15826033-vmss000001 I1105 02:12:46.602] Nov 5 02:12:46.602: INFO: I1105 02:12:46.603] Logging pods the kubelet thinks is on node k8s-agentpool-15826033-vmss000001 I1105 02:12:46.690] Nov 5 02:12:46.690: INFO: coredns-7f68dcdbdb-jt522 started at 2019-11-05 01:34:35 +0000 UTC (0+1 container statuses recorded) I1105 02:12:46.690] Nov 5 02:12:46.690: INFO: Container coredns ready: true, restart count 0 I1105 02:12:46.691] Nov 5 02:12:46.690: INFO: azure-ip-masq-agent-6c6vn started at 2019-11-05 01:34:35 +0000 UTC (0+1 container statuses recorded) I1105 02:12:46.691] Nov 5 02:12:46.690: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 02:12:46.691] Nov 5 02:12:46.690: INFO: keyvault-flexvolume-rdlcj started at 2019-11-05 01:34:35 +0000 UTC (0+1 container statuses recorded) I1105 02:12:46.691] Nov 5 02:12:46.690: INFO: Container keyvault-flexvolume ready: true, restart count 0 I1105 02:12:46.691] Nov 5 02:12:46.690: INFO: kube-proxy-6kh57 started at 2019-11-05 01:34:35 +0000 UTC (0+1 container statuses recorded) I1105 02:12:46.692] Nov 5 02:12:46.690: INFO: Container kube-proxy ready: true, restart count 0 I1105 02:12:46.692] Nov 5 02:12:46.690: INFO: tiller-deploy-54c96cb5df-8d9fs started at 2019-11-05 01:34:35 +0000 UTC (0+1 container statuses recorded) I1105 02:12:46.692] Nov 5 02:12:46.690: INFO: Container tiller ready: true, restart count 0 I1105 02:12:46.692] Nov 5 02:12:46.690: INFO: csi-azuredisk-node-ghv5w started at 2019-11-05 01:38:24 +0000 UTC (0+3 container statuses recorded) I1105 02:12:46.692] Nov 5 02:12:46.690: INFO: Container azuredisk ready: true, restart count 0 I1105 02:12:46.692] Nov 5 02:12:46.690: INFO: Container liveness-probe ready: true, restart count 0 I1105 02:12:46.693] Nov 5 02:12:46.690: INFO: Container node-driver-registrar ready: true, restart count 0 I1105 02:12:46.693] Nov 5 02:12:46.690: INFO: csi-azuredisk-controller-6f6587bfc6-8vkfl started at 2019-11-05 01:38:24 +0000 UTC (0+6 container statuses recorded) I1105 02:12:46.693] Nov 5 02:12:46.690: INFO: Container azuredisk ready: true, restart count 0 I1105 02:12:46.693] Nov 5 02:12:46.690: INFO: Container cluster-driver-registrar ready: true, restart count 0 I1105 02:12:46.693] Nov 5 02:12:46.690: INFO: Container csi-attacher ready: true, restart count 0 I1105 02:12:46.693] Nov 5 02:12:46.690: INFO: Container csi-provisioner ready: true, restart count 0 I1105 02:12:46.694] Nov 5 02:12:46.690: INFO: Container csi-snapshotter ready: true, restart count 0 I1105 02:12:46.694] Nov 5 02:12:46.690: INFO: Container liveness-probe ready: true, restart count 0 I1105 02:12:46.694] Nov 5 02:12:46.690: INFO: blobfuse-flexvol-installer-rw5dw started at 2019-11-05 01:34:35 +0000 UTC (0+1 container statuses recorded) I1105 02:12:46.694] Nov 5 02:12:46.690: INFO: Container blobfuse-flexvol-installer ready: true, restart count 0 I1105 02:12:46.946] Nov 5 02:12:46.946: INFO: I1105 02:12:46.947] Latency metrics for node k8s-agentpool-15826033-vmss000001 I1105 02:12:46.947] Nov 5 02:12:46.946: INFO: I1105 02:12:46.947] Logging node info for node k8s-master-15826033-0 I1105 02:12:47.026] Nov 5 02:12:47.024: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-15826033-0,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-15826033-0,UID:aceb0231-4645-4a9a-b296-90ada851986b,ResourceVersion:7861,Generation:0,CreationTimestamp:2019-11-05 01:35:06 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 1,kubernetes.azure.com/cluster: kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-15826033-0,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.4.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5/providers/Microsoft.Compute/virtualMachines/k8s-master-15826033-0,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 01:35:20 +0000 UTC 2019-11-05 01:35:20 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 02:12:31 +0000 UTC 2019-11-05 01:34:57 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 02:12:31 +0000 UTC 2019-11-05 01:34:57 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 02:12:31 +0000 UTC 2019-11-05 01:34:57 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 02:12:31 +0000 UTC 2019-11-05 01:35:07 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.255.5} {Hostname k8s-master-15826033-0}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:1915756c22a540dc8eb30493987e9a43,SystemUUID:742C969C-5A71-D744-8E61-F27E38B56EB2,BootID:03f0672e-8fa9-4309-956f-8e3c371902c3,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 02:12:47.026] Nov 5 02:12:47.024: INFO: I1105 02:12:47.026] Logging kubelet events for node k8s-master-15826033-0 I1105 02:12:47.105] Nov 5 02:12:47.104: INFO: I1105 02:12:47.105] Logging pods the kubelet thinks is on node k8s-master-15826033-0 I1105 02:12:47.186] Nov 5 02:12:47.186: INFO: kube-controller-manager-k8s-master-15826033-0 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:12:47.186] Nov 5 02:12:47.186: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 02:12:47.187] Nov 5 02:12:47.186: INFO: kube-scheduler-k8s-master-15826033-0 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:12:47.187] Nov 5 02:12:47.186: INFO: Container kube-scheduler ready: true, restart count 0 I1105 02:12:47.187] Nov 5 02:12:47.186: INFO: azure-ip-masq-agent-wgdpx started at 2019-11-05 01:35:06 +0000 UTC (0+1 container statuses recorded) I1105 02:12:47.187] Nov 5 02:12:47.186: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 02:12:47.188] Nov 5 02:12:47.186: INFO: kube-proxy-7d2px started at 2019-11-05 01:35:06 +0000 UTC (0+1 container statuses recorded) I1105 02:12:47.188] Nov 5 02:12:47.186: INFO: Container kube-proxy ready: true, restart count 0 I1105 02:12:47.188] Nov 5 02:12:47.186: INFO: cloud-controller-manager-k8s-master-15826033-0 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:12:47.188] Nov 5 02:12:47.186: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 02:12:47.188] Nov 5 02:12:47.186: INFO: kube-addon-manager-k8s-master-15826033-0 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:12:47.189] Nov 5 02:12:47.186: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 02:12:47.189] Nov 5 02:12:47.186: INFO: kube-apiserver-k8s-master-15826033-0 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:12:47.189] Nov 5 02:12:47.186: INFO: Container kube-apiserver ready: true, restart count 0 I1105 02:12:47.434] Nov 5 02:12:47.434: INFO: I1105 02:12:47.434] Latency metrics for node k8s-master-15826033-0 I1105 02:12:47.435] Nov 5 02:12:47.434: INFO: I1105 02:12:47.435] Logging node info for node k8s-master-15826033-1 I1105 02:12:47.516] Nov 5 02:12:47.513: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-15826033-1,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-15826033-1,UID:1a13ec61-c89a-456c-8c94-bfc8e717c2e7,ResourceVersion:7863,Generation:0,CreationTimestamp:2019-11-05 01:35:07 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 1,kubernetes.azure.com/cluster: kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-15826033-1,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.5.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5/providers/Microsoft.Compute/virtualMachines/k8s-master-15826033-1,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 01:35:31 +0000 UTC 2019-11-05 01:35:31 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 02:12:32 +0000 UTC 2019-11-05 01:35:00 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 02:12:32 +0000 UTC 2019-11-05 01:35:00 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 02:12:32 +0000 UTC 2019-11-05 01:35:00 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 02:12:32 +0000 UTC 2019-11-05 01:35:07 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.255.6} {Hostname k8s-master-15826033-1}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:e0f2bccddd5b4555b180d14081ab59a8,SystemUUID:AE126631-CAD2-B54F-B22E-1F499342A4D6,BootID:509b9a9d-b046-4861-aa35-8bcb77465de2,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 02:12:47.516] Nov 5 02:12:47.514: INFO: I1105 02:12:47.516] Logging kubelet events for node k8s-master-15826033-1 I1105 02:12:47.595] Nov 5 02:12:47.595: INFO: I1105 02:12:47.595] Logging pods the kubelet thinks is on node k8s-master-15826033-1 I1105 02:12:47.676] Nov 5 02:12:47.676: INFO: kube-addon-manager-k8s-master-15826033-1 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:12:47.676] Nov 5 02:12:47.676: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 02:12:47.677] Nov 5 02:12:47.676: INFO: kube-apiserver-k8s-master-15826033-1 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:12:47.677] Nov 5 02:12:47.676: INFO: Container kube-apiserver ready: true, restart count 0 I1105 02:12:47.677] Nov 5 02:12:47.676: INFO: kube-controller-manager-k8s-master-15826033-1 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:12:47.677] Nov 5 02:12:47.676: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 02:12:47.678] Nov 5 02:12:47.676: INFO: kube-scheduler-k8s-master-15826033-1 started at 2019-11-05 01:36:16 +0000 UTC (0+1 container statuses recorded) I1105 02:12:47.678] Nov 5 02:12:47.676: INFO: Container kube-scheduler ready: true, restart count 0 I1105 02:12:47.678] Nov 5 02:12:47.676: INFO: azure-ip-masq-agent-w6dpr started at 2019-11-05 01:35:10 +0000 UTC (0+1 container statuses recorded) I1105 02:12:47.678] Nov 5 02:12:47.676: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 02:12:47.679] Nov 5 02:12:47.676: INFO: kube-proxy-8kmr5 started at 2019-11-05 01:35:10 +0000 UTC (0+1 container statuses recorded) I1105 02:12:47.679] Nov 5 02:12:47.676: INFO: Container kube-proxy ready: true, restart count 0 I1105 02:12:47.679] Nov 5 02:12:47.676: INFO: cloud-controller-manager-k8s-master-15826033-1 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:12:47.679] Nov 5 02:12:47.676: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 02:12:47.937] Nov 5 02:12:47.936: INFO: I1105 02:12:47.937] Latency metrics for node k8s-master-15826033-1 I1105 02:12:47.937] Nov 5 02:12:47.937: INFO: I1105 02:12:47.938] Logging node info for node k8s-master-15826033-2 I1105 02:12:48.016] Nov 5 02:12:48.014: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-15826033-2,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-15826033-2,UID:acb3a851-d6df-4549-ba56-4b7e96553c17,ResourceVersion:7779,Generation:0,CreationTimestamp:2019-11-05 01:34:09 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 2,kubernetes.azure.com/cluster: kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-15826033-2,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.1.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5/providers/Microsoft.Compute/virtualMachines/k8s-master-15826033-2,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 01:34:44 +0000 UTC 2019-11-05 01:34:44 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 02:12:03 +0000 UTC 2019-11-05 01:34:07 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 02:12:03 +0000 UTC 2019-11-05 01:34:07 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 02:12:03 +0000 UTC 2019-11-05 01:34:07 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 02:12:03 +0000 UTC 2019-11-05 01:34:29 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{Hostname k8s-master-15826033-2} {InternalIP 10.240.255.7}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:0885b0c2f6174106bde4fa0eae2cbc6b,SystemUUID:C176121C-5D7A-6243-9B05-8B68F8022B64,BootID:30430638-7ab6-4761-b984-2387e1a3f5b2,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 02:12:48.017] Nov 5 02:12:48.014: INFO: I1105 02:12:48.017] Logging kubelet events for node k8s-master-15826033-2 I1105 02:12:48.098] Nov 5 02:12:48.098: INFO: I1105 02:12:48.098] Logging pods the kubelet thinks is on node k8s-master-15826033-2 I1105 02:12:48.180] Nov 5 02:12:48.180: INFO: kube-addon-manager-k8s-master-15826033-2 started at 2019-11-05 01:33:54 +0000 UTC (0+1 container statuses recorded) I1105 02:12:48.180] Nov 5 02:12:48.180: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 02:12:48.180] Nov 5 02:12:48.180: INFO: kube-apiserver-k8s-master-15826033-2 started at 2019-11-05 01:33:54 +0000 UTC (0+1 container statuses recorded) I1105 02:12:48.181] Nov 5 02:12:48.180: INFO: Container kube-apiserver ready: true, restart count 0 I1105 02:12:48.181] Nov 5 02:12:48.180: INFO: kube-controller-manager-k8s-master-15826033-2 started at 2019-11-05 01:33:54 +0000 UTC (0+1 container statuses recorded) I1105 02:12:48.181] Nov 5 02:12:48.180: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 02:12:48.181] Nov 5 02:12:48.180: INFO: kube-scheduler-k8s-master-15826033-2 started at 2019-11-05 01:33:54 +0000 UTC (0+1 container statuses recorded) I1105 02:12:48.181] Nov 5 02:12:48.180: INFO: Container kube-scheduler ready: true, restart count 0 I1105 02:12:48.182] Nov 5 02:12:48.180: INFO: azure-ip-masq-agent-hw7s5 started at 2019-11-05 01:34:34 +0000 UTC (0+1 container statuses recorded) I1105 02:12:48.182] Nov 5 02:12:48.180: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 02:12:48.182] Nov 5 02:12:48.180: INFO: kube-proxy-5xpp6 started at 2019-11-05 01:34:34 +0000 UTC (0+1 container statuses recorded) I1105 02:12:48.182] Nov 5 02:12:48.180: INFO: Container kube-proxy ready: true, restart count 0 I1105 02:12:48.182] Nov 5 02:12:48.180: INFO: cloud-controller-manager-k8s-master-15826033-2 started at 2019-11-05 01:33:54 +0000 UTC (0+1 container statuses recorded) I1105 02:12:48.183] Nov 5 02:12:48.180: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 02:12:48.433] Nov 5 02:12:48.432: INFO: I1105 02:12:48.433] Latency metrics for node k8s-master-15826033-2 I1105 02:12:48.433] Nov 5 02:12:48.432: INFO: I1105 02:12:48.433] Logging node info for node k8s-master-15826033-3 I1105 02:12:48.512] Nov 5 02:12:48.510: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-15826033-3,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-15826033-3,UID:b7702272-1876-434e-b6a0-17fa34c6757f,ResourceVersion:7859,Generation:0,CreationTimestamp:2019-11-05 01:35:06 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 0,kubernetes.azure.com/cluster: kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-15826033-3,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.3.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5/providers/Microsoft.Compute/virtualMachines/k8s-master-15826033-3,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 01:35:17 +0000 UTC 2019-11-05 01:35:17 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 02:12:31 +0000 UTC 2019-11-05 01:35:00 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 02:12:31 +0000 UTC 2019-11-05 01:35:00 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 02:12:31 +0000 UTC 2019-11-05 01:35:00 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 02:12:31 +0000 UTC 2019-11-05 01:35:06 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.255.8} {Hostname k8s-master-15826033-3}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:fbf366963961422c9d90bedf6639c8e1,SystemUUID:5388948D-1A8D-EA41-B8B8-7F4523EDF5AA,BootID:42f5884c-7418-486a-8cf3-75188a331f38,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 02:12:48.512] Nov 5 02:12:48.510: INFO: I1105 02:12:48.513] Logging kubelet events for node k8s-master-15826033-3 I1105 02:12:48.591] Nov 5 02:12:48.591: INFO: I1105 02:12:48.591] Logging pods the kubelet thinks is on node k8s-master-15826033-3 I1105 02:12:48.673] Nov 5 02:12:48.673: INFO: kube-scheduler-k8s-master-15826033-3 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:12:48.673] Nov 5 02:12:48.673: INFO: Container kube-scheduler ready: true, restart count 0 I1105 02:12:48.673] Nov 5 02:12:48.673: INFO: kube-proxy-xz24z started at 2019-11-05 01:35:06 +0000 UTC (0+1 container statuses recorded) I1105 02:12:48.674] Nov 5 02:12:48.673: INFO: Container kube-proxy ready: true, restart count 0 I1105 02:12:48.674] Nov 5 02:12:48.673: INFO: azure-ip-masq-agent-dqfpw started at 2019-11-05 01:35:06 +0000 UTC (0+1 container statuses recorded) I1105 02:12:48.674] Nov 5 02:12:48.673: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 02:12:48.674] Nov 5 02:12:48.673: INFO: cloud-controller-manager-k8s-master-15826033-3 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:12:48.675] Nov 5 02:12:48.673: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 02:12:48.675] Nov 5 02:12:48.673: INFO: kube-addon-manager-k8s-master-15826033-3 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:12:48.675] Nov 5 02:12:48.673: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 02:12:48.675] Nov 5 02:12:48.673: INFO: kube-apiserver-k8s-master-15826033-3 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:12:48.675] Nov 5 02:12:48.673: INFO: Container kube-apiserver ready: true, restart count 0 I1105 02:12:48.676] Nov 5 02:12:48.673: INFO: kube-controller-manager-k8s-master-15826033-3 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:12:48.676] Nov 5 02:12:48.673: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 02:12:48.919] Nov 5 02:12:48.919: INFO: I1105 02:12:48.919] Latency metrics for node k8s-master-15826033-3 I1105 02:12:48.920] Nov 5 02:12:48.919: INFO: I1105 02:12:48.920] Logging node info for node k8s-master-15826033-4 I1105 02:12:48.998] Nov 5 02:12:48.996: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-15826033-4,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-15826033-4,UID:276daef1-6c05-41e3-bacd-abc93d35cc6d,ResourceVersion:7865,Generation:0,CreationTimestamp:2019-11-05 01:35:07 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 0,kubernetes.azure.com/cluster: kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-15826033-4,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.6.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5/providers/Microsoft.Compute/virtualMachines/k8s-master-15826033-4,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 01:35:41 +0000 UTC 2019-11-05 01:35:41 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 02:12:32 +0000 UTC 2019-11-05 01:35:02 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 02:12:32 +0000 UTC 2019-11-05 01:35:02 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 02:12:32 +0000 UTC 2019-11-05 01:35:02 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 02:12:32 +0000 UTC 2019-11-05 01:35:08 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.255.9} {Hostname k8s-master-15826033-4}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:79051fc0bdeb4e32a56acbaf5972ce3a,SystemUUID:C71606B5-7F08-9341-A26D-8A8D9D84FCAE,BootID:4bdf7cf3-2b83-42c7-a7ba-926ebe9ca75b,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 02:12:48.999] Nov 5 02:12:48.996: INFO: I1105 02:12:48.999] Logging kubelet events for node k8s-master-15826033-4 I1105 02:12:49.077] Nov 5 02:12:49.077: INFO: I1105 02:12:49.078] Logging pods the kubelet thinks is on node k8s-master-15826033-4 I1105 02:12:49.163] Nov 5 02:12:49.163: INFO: azure-ip-masq-agent-9nrj8 started at 2019-11-05 01:35:11 +0000 UTC (0+1 container statuses recorded) I1105 02:12:49.163] Nov 5 02:12:49.163: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 02:12:49.164] Nov 5 02:12:49.163: INFO: kube-proxy-rmngk started at 2019-11-05 01:35:11 +0000 UTC (0+1 container statuses recorded) I1105 02:12:49.164] Nov 5 02:12:49.163: INFO: Container kube-proxy ready: true, restart count 0 I1105 02:12:49.164] Nov 5 02:12:49.163: INFO: kube-apiserver-k8s-master-15826033-4 started at 2019-11-05 01:34:55 +0000 UTC (0+1 container statuses recorded) I1105 02:12:49.164] Nov 5 02:12:49.163: INFO: Container kube-apiserver ready: true, restart count 0 I1105 02:12:49.164] Nov 5 02:12:49.163: INFO: kube-controller-manager-k8s-master-15826033-4 started at 2019-11-05 01:34:55 +0000 UTC (0+1 container statuses recorded) I1105 02:12:49.164] Nov 5 02:12:49.163: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 02:12:49.164] Nov 5 02:12:49.163: INFO: kube-scheduler-k8s-master-15826033-4 started at 2019-11-05 01:34:55 +0000 UTC (0+1 container statuses recorded) I1105 02:12:49.165] Nov 5 02:12:49.163: INFO: Container kube-scheduler ready: true, restart count 0 I1105 02:12:49.165] Nov 5 02:12:49.163: INFO: cloud-controller-manager-k8s-master-15826033-4 started at 2019-11-05 01:34:56 +0000 UTC (0+1 container statuses recorded) I1105 02:12:49.165] Nov 5 02:12:49.163: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 02:12:49.165] Nov 5 02:12:49.163: INFO: kube-addon-manager-k8s-master-15826033-4 started at 2019-11-05 01:34:56 +0000 UTC (0+1 container statuses recorded) I1105 02:12:49.165] Nov 5 02:12:49.163: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 02:12:49.421] Nov 5 02:12:49.421: INFO: I1105 02:12:49.421] Latency metrics for node k8s-master-15826033-4 I1105 02:12:49.421] Nov 5 02:12:49.421: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready I1105 02:12:49.499] STEP: Destroying namespace "azuredisk-1141" for this suite. I1105 02:13:03.739] Nov 5 02:13:03.738: INFO: Waiting up to 30s for server preferred namespaced resources to be successfully discovered I1105 02:13:06.693] Nov 5 02:13:06.692: INFO: namespace azuredisk-1141 deletion completed in 17.193807618s I1105 02:13:06.693] I1105 02:13:06.693] • Failure [323.307 seconds] I1105 02:13:06.693] [azuredisk-csi-e2e] Dynamic Provisioning I1105 02:13:06.694] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:33 I1105 02:13:06.694] [single-az] I1105 02:13:06.694] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:36 I1105 02:13:06.694] should create multiple PV objects, bind to PVCs and attach all to different pods on the same node [It] I1105 02:13:06.694] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:151 I1105 02:13:06.695] I1105 02:13:06.695] Unexpected error: I1105 02:13:06.695] <*errors.errorString | 0xc0001eb860>: { I1105 02:13:06.695] s: "PersistentVolumeClaims [pvc-rkt48] not all in phase Bound within 5m0s", I1105 02:13:06.695] } I1105 02:13:06.696] PersistentVolumeClaims [pvc-rkt48] not all in phase Bound within 5m0s I1105 02:13:06.696] occurred I1105 02:13:06.696] I1105 02:13:06.696] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/testsuites/testsuites.go:278 I1105 02:13:06.696] ------------------------------ I1105 02:13:06.696] [azuredisk-csi-e2e] Dynamic Provisioning [single-az] I1105 02:13:06.697] should create a raw block volume on demand I1105 02:13:06.697] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:101 I1105 02:13:06.697] [BeforeEach] [single-az] I1105 02:13:06.697] /go/src/sigs.k8s.io/azuredisk-csi-driver/vendor/k8s.io/kubernetes/test/e2e/framework/framework.go:150 I1105 02:13:06.697] STEP: Creating a kubernetes client I1105 02:13:06.698] Nov 5 02:13:06.693: INFO: >>> kubeConfig: /workspace/aks978705698/kubeconfig/kubeconfig.eastus2.json I1105 02:13:06.698] STEP: Building a namespace api object, basename azuredisk I1105 02:13:06.933] STEP: Binding the e2e-test-privileged-psp PodSecurityPolicy to the default service account in azuredisk-8234 I1105 02:13:07.192] STEP: Waiting for a default service account to be provisioned in namespace I1105 02:13:07.268] [BeforeEach] [single-az] I1105 02:13:07.268] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:58 I1105 02:13:07.268] [It] should create a raw block volume on demand I1105 02:13:07.269] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:101 I1105 02:13:07.269] STEP: setting up the StorageClass I1105 02:13:07.269] STEP: creating a StorageClass I1105 02:13:07.351] STEP: setting up the PVC and PV I1105 02:13:07.351] STEP: creating a PVC I1105 02:13:07.434] STEP: waiting for PVC to be in phase "Bound" I1105 02:13:07.434] Nov 5 02:13:07.434: INFO: Waiting up to 5m0s for PersistentVolumeClaims [pvc-zvbnt] to have phase Bound I1105 02:13:07.512] Nov 5 02:13:07.511: INFO: PersistentVolumeClaim pvc-zvbnt found but phase is Pending instead of Bound. I1105 02:13:09.589] Nov 5 02:13:09.589: INFO: PersistentVolumeClaim pvc-zvbnt found but phase is Pending instead of Bound. I1105 02:13:11.667] Nov 5 02:13:11.667: INFO: PersistentVolumeClaim pvc-zvbnt found but phase is Pending instead of Bound. I1105 02:13:13.745] Nov 5 02:13:13.745: INFO: PersistentVolumeClaim pvc-zvbnt found but phase is Pending instead of Bound. I1105 02:13:15.822] Nov 5 02:13:15.822: INFO: PersistentVolumeClaim pvc-zvbnt found but phase is Pending instead of Bound. I1105 02:13:17.900] Nov 5 02:13:17.900: INFO: PersistentVolumeClaim pvc-zvbnt found but phase is Pending instead of Bound. I1105 02:13:19.978] Nov 5 02:13:19.978: INFO: PersistentVolumeClaim pvc-zvbnt found but phase is Pending instead of Bound. I1105 02:13:22.056] Nov 5 02:13:22.055: INFO: PersistentVolumeClaim pvc-zvbnt found but phase is Pending instead of Bound. I1105 02:13:24.133] Nov 5 02:13:24.133: INFO: PersistentVolumeClaim pvc-zvbnt found but phase is Pending instead of Bound. I1105 02:13:26.211] Nov 5 02:13:26.211: INFO: PersistentVolumeClaim pvc-zvbnt found but phase is Pending instead of Bound. I1105 02:13:28.289] Nov 5 02:13:28.288: INFO: PersistentVolumeClaim pvc-zvbnt found but phase is Pending instead of Bound. I1105 02:13:30.366] Nov 5 02:13:30.366: INFO: PersistentVolumeClaim pvc-zvbnt found but phase is Pending instead of Bound. I1105 02:13:32.444] Nov 5 02:13:32.444: INFO: PersistentVolumeClaim pvc-zvbnt found but phase is Pending instead of Bound. I1105 02:13:34.523] Nov 5 02:13:34.523: INFO: PersistentVolumeClaim pvc-zvbnt found but phase is Pending instead of Bound. I1105 02:13:36.600] Nov 5 02:13:36.600: INFO: PersistentVolumeClaim pvc-zvbnt found but phase is Pending instead of Bound. I1105 02:13:38.678] Nov 5 02:13:38.678: INFO: PersistentVolumeClaim pvc-zvbnt found but phase is Pending instead of Bound. I1105 02:13:40.756] Nov 5 02:13:40.756: INFO: PersistentVolumeClaim pvc-zvbnt found but phase is Pending instead of Bound. I1105 02:13:42.841] Nov 5 02:13:42.841: INFO: PersistentVolumeClaim pvc-zvbnt found but phase is Pending instead of Bound. I1105 02:13:44.919] Nov 5 02:13:44.919: INFO: PersistentVolumeClaim pvc-zvbnt found but phase is Pending instead of Bound. I1105 02:13:46.996] Nov 5 02:13:46.996: INFO: PersistentVolumeClaim pvc-zvbnt found but phase is Pending instead of Bound. I1105 02:13:49.078] Nov 5 02:13:49.077: INFO: PersistentVolumeClaim pvc-zvbnt found but phase is Pending instead of Bound. I1105 02:13:51.155] Nov 5 02:13:51.155: INFO: PersistentVolumeClaim pvc-zvbnt found but phase is Pending instead of Bound. I1105 02:13:53.233] Nov 5 02:13:53.233: INFO: PersistentVolumeClaim pvc-zvbnt found but phase is Pending instead of Bound. I1105 02:13:55.311] Nov 5 02:13:55.311: INFO: PersistentVolumeClaim pvc-zvbnt found but phase is Pending instead of Bound. I1105 02:13:57.388] Nov 5 02:13:57.388: INFO: PersistentVolumeClaim pvc-zvbnt found but phase is Pending instead of Bound. I1105 02:13:59.466] Nov 5 02:13:59.466: INFO: PersistentVolumeClaim pvc-zvbnt found but phase is Pending instead of Bound. I1105 02:14:01.544] Nov 5 02:14:01.543: INFO: PersistentVolumeClaim pvc-zvbnt found but phase is Pending instead of Bound. I1105 02:14:03.621] Nov 5 02:14:03.621: INFO: PersistentVolumeClaim pvc-zvbnt found but phase is Pending instead of Bound. I1105 02:14:05.698] Nov 5 02:14:05.698: INFO: PersistentVolumeClaim pvc-zvbnt found but phase is Pending instead of Bound. I1105 02:14:07.776] Nov 5 02:14:07.776: INFO: PersistentVolumeClaim pvc-zvbnt found but phase is Pending instead of Bound. I1105 02:14:09.854] Nov 5 02:14:09.854: INFO: PersistentVolumeClaim pvc-zvbnt found but phase is Pending instead of Bound. I1105 02:14:11.931] Nov 5 02:14:11.931: INFO: PersistentVolumeClaim pvc-zvbnt found but phase is Pending instead of Bound. I1105 02:14:14.009] Nov 5 02:14:14.009: INFO: PersistentVolumeClaim pvc-zvbnt found but phase is Pending instead of Bound. I1105 02:14:16.088] Nov 5 02:14:16.088: INFO: PersistentVolumeClaim pvc-zvbnt found but phase is Pending instead of Bound. I1105 02:14:18.167] Nov 5 02:14:18.166: INFO: PersistentVolumeClaim pvc-zvbnt found but phase is Pending instead of Bound. I1105 02:14:20.244] Nov 5 02:14:20.244: INFO: PersistentVolumeClaim pvc-zvbnt found but phase is Pending instead of Bound. I1105 02:14:22.322] Nov 5 02:14:22.322: INFO: PersistentVolumeClaim pvc-zvbnt found but phase is Pending instead of Bound. I1105 02:14:24.399] Nov 5 02:14:24.399: INFO: PersistentVolumeClaim pvc-zvbnt found but phase is Pending instead of Bound. I1105 02:14:26.477] Nov 5 02:14:26.476: INFO: PersistentVolumeClaim pvc-zvbnt found but phase is Pending instead of Bound. I1105 02:14:28.554] Nov 5 02:14:28.554: INFO: PersistentVolumeClaim pvc-zvbnt found but phase is Pending instead of Bound. I1105 02:14:30.632] Nov 5 02:14:30.632: INFO: PersistentVolumeClaim pvc-zvbnt found but phase is Pending instead of Bound. I1105 02:14:32.710] Nov 5 02:14:32.710: INFO: PersistentVolumeClaim pvc-zvbnt found but phase is Pending instead of Bound. I1105 02:14:34.788] Nov 5 02:14:34.787: INFO: PersistentVolumeClaim pvc-zvbnt found but phase is Pending instead of Bound. I1105 02:14:36.865] Nov 5 02:14:36.865: INFO: PersistentVolumeClaim pvc-zvbnt found but phase is Pending instead of Bound. I1105 02:14:38.943] Nov 5 02:14:38.942: INFO: PersistentVolumeClaim pvc-zvbnt found but phase is Pending instead of Bound. I1105 02:14:41.020] Nov 5 02:14:41.020: INFO: PersistentVolumeClaim pvc-zvbnt found but phase is Pending instead of Bound. I1105 02:14:43.098] Nov 5 02:14:43.097: INFO: PersistentVolumeClaim pvc-zvbnt found but phase is Pending instead of Bound. I1105 02:14:45.175] Nov 5 02:14:45.175: INFO: PersistentVolumeClaim pvc-zvbnt found but phase is Pending instead of Bound. I1105 02:14:47.253] Nov 5 02:14:47.252: INFO: PersistentVolumeClaim pvc-zvbnt found but phase is Pending instead of Bound. I1105 02:14:49.331] Nov 5 02:14:49.331: INFO: PersistentVolumeClaim pvc-zvbnt found but phase is Pending instead of Bound. I1105 02:14:51.409] Nov 5 02:14:51.408: INFO: PersistentVolumeClaim pvc-zvbnt found but phase is Pending instead of Bound. I1105 02:14:53.486] Nov 5 02:14:53.486: INFO: PersistentVolumeClaim pvc-zvbnt found but phase is Pending instead of Bound. I1105 02:14:55.564] Nov 5 02:14:55.564: INFO: PersistentVolumeClaim pvc-zvbnt found but phase is Pending instead of Bound. I1105 02:14:57.642] Nov 5 02:14:57.642: INFO: PersistentVolumeClaim pvc-zvbnt found but phase is Pending instead of Bound. I1105 02:14:59.720] Nov 5 02:14:59.719: INFO: PersistentVolumeClaim pvc-zvbnt found but phase is Pending instead of Bound. I1105 02:15:01.797] Nov 5 02:15:01.797: INFO: PersistentVolumeClaim pvc-zvbnt found but phase is Pending instead of Bound. I1105 02:15:03.875] Nov 5 02:15:03.875: INFO: PersistentVolumeClaim pvc-zvbnt found but phase is Pending instead of Bound. I1105 02:15:05.954] Nov 5 02:15:05.954: INFO: PersistentVolumeClaim pvc-zvbnt found but phase is Pending instead of Bound. I1105 02:15:08.032] Nov 5 02:15:08.031: INFO: PersistentVolumeClaim pvc-zvbnt found but phase is Pending instead of Bound. I1105 02:15:10.109] Nov 5 02:15:10.109: INFO: PersistentVolumeClaim pvc-zvbnt found but phase is Pending instead of Bound. I1105 02:15:12.187] Nov 5 02:15:12.187: INFO: PersistentVolumeClaim pvc-zvbnt found but phase is Pending instead of Bound. I1105 02:15:14.264] Nov 5 02:15:14.264: INFO: PersistentVolumeClaim pvc-zvbnt found but phase is Pending instead of Bound. I1105 02:15:16.342] Nov 5 02:15:16.342: INFO: PersistentVolumeClaim pvc-zvbnt found but phase is Pending instead of Bound. I1105 02:15:18.420] Nov 5 02:15:18.419: INFO: PersistentVolumeClaim pvc-zvbnt found but phase is Pending instead of Bound. I1105 02:15:20.497] Nov 5 02:15:20.497: INFO: PersistentVolumeClaim pvc-zvbnt found but phase is Pending instead of Bound. I1105 02:15:22.574] Nov 5 02:15:22.574: INFO: PersistentVolumeClaim pvc-zvbnt found but phase is Pending instead of Bound. I1105 02:15:24.652] Nov 5 02:15:24.652: INFO: PersistentVolumeClaim pvc-zvbnt found but phase is Pending instead of Bound. I1105 02:15:26.730] Nov 5 02:15:26.730: INFO: PersistentVolumeClaim pvc-zvbnt found but phase is Pending instead of Bound. I1105 02:15:28.807] Nov 5 02:15:28.807: INFO: PersistentVolumeClaim pvc-zvbnt found but phase is Pending instead of Bound. I1105 02:15:30.885] Nov 5 02:15:30.885: INFO: PersistentVolumeClaim pvc-zvbnt found but phase is Pending instead of Bound. I1105 02:15:32.963] Nov 5 02:15:32.963: INFO: PersistentVolumeClaim pvc-zvbnt found but phase is Pending instead of Bound. I1105 02:15:35.040] Nov 5 02:15:35.040: INFO: PersistentVolumeClaim pvc-zvbnt found but phase is Pending instead of Bound. I1105 02:15:37.157] Nov 5 02:15:37.156: INFO: PersistentVolumeClaim pvc-zvbnt found but phase is Pending instead of Bound. I1105 02:15:39.235] Nov 5 02:15:39.234: INFO: PersistentVolumeClaim pvc-zvbnt found but phase is Pending instead of Bound. I1105 02:15:41.312] Nov 5 02:15:41.312: INFO: PersistentVolumeClaim pvc-zvbnt found but phase is Pending instead of Bound. I1105 02:15:43.390] Nov 5 02:15:43.389: INFO: PersistentVolumeClaim pvc-zvbnt found but phase is Pending instead of Bound. I1105 02:15:45.467] Nov 5 02:15:45.467: INFO: PersistentVolumeClaim pvc-zvbnt found but phase is Pending instead of Bound. I1105 02:15:47.545] Nov 5 02:15:47.545: INFO: PersistentVolumeClaim pvc-zvbnt found but phase is Pending instead of Bound. I1105 02:15:49.622] Nov 5 02:15:49.622: INFO: PersistentVolumeClaim pvc-zvbnt found but phase is Pending instead of Bound. I1105 02:15:51.699] Nov 5 02:15:51.699: INFO: PersistentVolumeClaim pvc-zvbnt found but phase is Pending instead of Bound. I1105 02:15:53.776] Nov 5 02:15:53.776: INFO: PersistentVolumeClaim pvc-zvbnt found but phase is Pending instead of Bound. I1105 02:15:55.854] Nov 5 02:15:55.854: INFO: PersistentVolumeClaim pvc-zvbnt found but phase is Pending instead of Bound. I1105 02:15:57.932] Nov 5 02:15:57.931: INFO: PersistentVolumeClaim pvc-zvbnt found but phase is Pending instead of Bound. I1105 02:16:00.015] Nov 5 02:16:00.015: INFO: PersistentVolumeClaim pvc-zvbnt found but phase is Pending instead of Bound. I1105 02:16:02.093] Nov 5 02:16:02.093: INFO: PersistentVolumeClaim pvc-zvbnt found but phase is Pending instead of Bound. I1105 02:16:04.170] Nov 5 02:16:04.170: INFO: PersistentVolumeClaim pvc-zvbnt found but phase is Pending instead of Bound. I1105 02:16:06.247] Nov 5 02:16:06.247: INFO: PersistentVolumeClaim pvc-zvbnt found but phase is Pending instead of Bound. I1105 02:16:08.325] Nov 5 02:16:08.325: INFO: PersistentVolumeClaim pvc-zvbnt found but phase is Pending instead of Bound. I1105 02:16:10.402] Nov 5 02:16:10.402: INFO: PersistentVolumeClaim pvc-zvbnt found but phase is Pending instead of Bound. I1105 02:16:12.480] Nov 5 02:16:12.479: INFO: PersistentVolumeClaim pvc-zvbnt found but phase is Pending instead of Bound. I1105 02:16:14.558] Nov 5 02:16:14.557: INFO: PersistentVolumeClaim pvc-zvbnt found but phase is Pending instead of Bound. I1105 02:16:16.635] Nov 5 02:16:16.635: INFO: PersistentVolumeClaim pvc-zvbnt found but phase is Pending instead of Bound. I1105 02:16:18.714] Nov 5 02:16:18.713: INFO: PersistentVolumeClaim pvc-zvbnt found but phase is Pending instead of Bound. I1105 02:16:20.792] Nov 5 02:16:20.791: INFO: PersistentVolumeClaim pvc-zvbnt found but phase is Pending instead of Bound. I1105 02:16:22.869] Nov 5 02:16:22.869: INFO: PersistentVolumeClaim pvc-zvbnt found but phase is Pending instead of Bound. I1105 02:16:24.947] Nov 5 02:16:24.946: INFO: PersistentVolumeClaim pvc-zvbnt found but phase is Pending instead of Bound. I1105 02:16:27.024] Nov 5 02:16:27.024: INFO: PersistentVolumeClaim pvc-zvbnt found but phase is Pending instead of Bound. I1105 02:16:29.118] Nov 5 02:16:29.118: INFO: PersistentVolumeClaim pvc-zvbnt found but phase is Pending instead of Bound. I1105 02:16:31.195] Nov 5 02:16:31.195: INFO: PersistentVolumeClaim pvc-zvbnt found but phase is Pending instead of Bound. I1105 02:16:33.272] Nov 5 02:16:33.272: INFO: PersistentVolumeClaim pvc-zvbnt found but phase is Pending instead of Bound. I1105 02:16:35.350] Nov 5 02:16:35.349: INFO: PersistentVolumeClaim pvc-zvbnt found but phase is Pending instead of Bound. I1105 02:16:37.427] Nov 5 02:16:37.427: INFO: PersistentVolumeClaim pvc-zvbnt found but phase is Pending instead of Bound. I1105 02:16:39.505] Nov 5 02:16:39.505: INFO: PersistentVolumeClaim pvc-zvbnt found but phase is Pending instead of Bound. I1105 02:16:41.582] Nov 5 02:16:41.582: INFO: PersistentVolumeClaim pvc-zvbnt found but phase is Pending instead of Bound. I1105 02:16:43.660] Nov 5 02:16:43.660: INFO: PersistentVolumeClaim pvc-zvbnt found but phase is Pending instead of Bound. I1105 02:16:45.737] Nov 5 02:16:45.737: INFO: PersistentVolumeClaim pvc-zvbnt found but phase is Pending instead of Bound. I1105 02:16:47.815] Nov 5 02:16:47.814: INFO: PersistentVolumeClaim pvc-zvbnt found but phase is Pending instead of Bound. I1105 02:16:49.892] Nov 5 02:16:49.892: INFO: PersistentVolumeClaim pvc-zvbnt found but phase is Pending instead of Bound. I1105 02:16:51.970] Nov 5 02:16:51.970: INFO: PersistentVolumeClaim pvc-zvbnt found but phase is Pending instead of Bound. I1105 02:16:54.047] Nov 5 02:16:54.047: INFO: PersistentVolumeClaim pvc-zvbnt found but phase is Pending instead of Bound. I1105 02:16:56.125] Nov 5 02:16:56.124: INFO: PersistentVolumeClaim pvc-zvbnt found but phase is Pending instead of Bound. I1105 02:16:58.203] Nov 5 02:16:58.203: INFO: PersistentVolumeClaim pvc-zvbnt found but phase is Pending instead of Bound. I1105 02:17:00.280] Nov 5 02:17:00.280: INFO: PersistentVolumeClaim pvc-zvbnt found but phase is Pending instead of Bound. I1105 02:17:02.358] Nov 5 02:17:02.357: INFO: PersistentVolumeClaim pvc-zvbnt found but phase is Pending instead of Bound. I1105 02:17:04.450] Nov 5 02:17:04.450: INFO: PersistentVolumeClaim pvc-zvbnt found but phase is Pending instead of Bound. I1105 02:17:06.527] Nov 5 02:17:06.527: INFO: PersistentVolumeClaim pvc-zvbnt found but phase is Pending instead of Bound. I1105 02:17:08.613] Nov 5 02:17:08.612: INFO: PersistentVolumeClaim pvc-zvbnt found but phase is Pending instead of Bound. I1105 02:17:10.690] Nov 5 02:17:10.690: INFO: PersistentVolumeClaim pvc-zvbnt found but phase is Pending instead of Bound. I1105 02:17:12.767] Nov 5 02:17:12.767: INFO: PersistentVolumeClaim pvc-zvbnt found but phase is Pending instead of Bound. I1105 02:17:14.851] Nov 5 02:17:14.850: INFO: PersistentVolumeClaim pvc-zvbnt found but phase is Pending instead of Bound. I1105 02:17:16.928] Nov 5 02:17:16.928: INFO: PersistentVolumeClaim pvc-zvbnt found but phase is Pending instead of Bound. I1105 02:17:19.006] Nov 5 02:17:19.005: INFO: PersistentVolumeClaim pvc-zvbnt found but phase is Pending instead of Bound. I1105 02:17:21.098] Nov 5 02:17:21.098: INFO: PersistentVolumeClaim pvc-zvbnt found but phase is Pending instead of Bound. I1105 02:17:23.175] Nov 5 02:17:23.175: INFO: PersistentVolumeClaim pvc-zvbnt found but phase is Pending instead of Bound. I1105 02:17:25.253] Nov 5 02:17:25.253: INFO: PersistentVolumeClaim pvc-zvbnt found but phase is Pending instead of Bound. I1105 02:17:27.330] Nov 5 02:17:27.330: INFO: PersistentVolumeClaim pvc-zvbnt found but phase is Pending instead of Bound. I1105 02:17:29.408] Nov 5 02:17:29.408: INFO: PersistentVolumeClaim pvc-zvbnt found but phase is Pending instead of Bound. I1105 02:17:31.486] Nov 5 02:17:31.485: INFO: PersistentVolumeClaim pvc-zvbnt found but phase is Pending instead of Bound. I1105 02:17:33.562] Nov 5 02:17:33.562: INFO: PersistentVolumeClaim pvc-zvbnt found but phase is Pending instead of Bound. I1105 02:17:35.640] Nov 5 02:17:35.640: INFO: PersistentVolumeClaim pvc-zvbnt found but phase is Pending instead of Bound. I1105 02:17:37.718] Nov 5 02:17:37.717: INFO: PersistentVolumeClaim pvc-zvbnt found but phase is Pending instead of Bound. I1105 02:17:39.796] Nov 5 02:17:39.796: INFO: PersistentVolumeClaim pvc-zvbnt found but phase is Pending instead of Bound. I1105 02:17:41.873] Nov 5 02:17:41.873: INFO: PersistentVolumeClaim pvc-zvbnt found but phase is Pending instead of Bound. I1105 02:17:43.951] Nov 5 02:17:43.951: INFO: PersistentVolumeClaim pvc-zvbnt found but phase is Pending instead of Bound. I1105 02:17:46.028] Nov 5 02:17:46.028: INFO: PersistentVolumeClaim pvc-zvbnt found but phase is Pending instead of Bound. I1105 02:17:48.113] Nov 5 02:17:48.113: INFO: PersistentVolumeClaim pvc-zvbnt found but phase is Pending instead of Bound. I1105 02:17:50.190] Nov 5 02:17:50.190: INFO: PersistentVolumeClaim pvc-zvbnt found but phase is Pending instead of Bound. I1105 02:17:52.268] Nov 5 02:17:52.268: INFO: PersistentVolumeClaim pvc-zvbnt found but phase is Pending instead of Bound. I1105 02:17:54.345] Nov 5 02:17:54.345: INFO: PersistentVolumeClaim pvc-zvbnt found but phase is Pending instead of Bound. I1105 02:17:56.428] Nov 5 02:17:56.428: INFO: PersistentVolumeClaim pvc-zvbnt found but phase is Pending instead of Bound. I1105 02:17:58.506] Nov 5 02:17:58.506: INFO: PersistentVolumeClaim pvc-zvbnt found but phase is Pending instead of Bound. I1105 02:18:00.584] Nov 5 02:18:00.583: INFO: PersistentVolumeClaim pvc-zvbnt found but phase is Pending instead of Bound. I1105 02:18:02.661] Nov 5 02:18:02.661: INFO: PersistentVolumeClaim pvc-zvbnt found but phase is Pending instead of Bound. I1105 02:18:04.738] Nov 5 02:18:04.738: INFO: PersistentVolumeClaim pvc-zvbnt found but phase is Pending instead of Bound. I1105 02:18:06.816] Nov 5 02:18:06.816: INFO: PersistentVolumeClaim pvc-zvbnt found but phase is Pending instead of Bound. I1105 02:18:08.816] Nov 5 02:18:08.816: INFO: Unexpected error occurred: PersistentVolumeClaims [pvc-zvbnt] not all in phase Bound within 5m0s I1105 02:18:08.817] [AfterEach] [single-az] I1105 02:18:08.817] /go/src/sigs.k8s.io/azuredisk-csi-driver/vendor/k8s.io/kubernetes/test/e2e/framework/framework.go:151 I1105 02:18:08.817] STEP: Collecting events from namespace "azuredisk-8234". I1105 02:18:08.894] STEP: Found 3 events. I1105 02:18:08.894] Nov 5 02:18:08.894: INFO: At 2019-11-05 02:13:07 +0000 UTC - event for pvc-zvbnt: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "disk.csi.azure.com" or manually created by system administrator I1105 02:18:08.894] Nov 5 02:18:08.894: INFO: At 2019-11-05 02:13:07 +0000 UTC - event for pvc-zvbnt: {disk.csi.azure.com_csi-azuredisk-controller-6f6587bfc6-8vkfl_1376914b-ff6d-11e9-93ab-ba2bee589b8c } Provisioning: External provisioner is provisioning volume for claim "azuredisk-8234/pvc-zvbnt" I1105 02:18:08.895] Nov 5 02:18:08.894: INFO: At 2019-11-05 02:13:07 +0000 UTC - event for pvc-zvbnt: {disk.csi.azure.com_csi-azuredisk-controller-6f6587bfc6-8vkfl_1376914b-ff6d-11e9-93ab-ba2bee589b8c } ProvisioningFailed: failed to provision volume with StorageClass "azuredisk-8234-disk.csi.azure.com-dynamic-sc-blrkl": rpc error: code = Unknown desc = compute.DisksClient#CreateOrUpdate: Failure sending request: StatusCode=400 -- Original Error: Code="InvalidAvailabilityZone" Message="The zone(s) '' for resource 'Microsoft.Compute/disks/pvc-19f49ea0-46d9-4631-9a8c-82314a429761' is not supported. The supported zones for location 'eastus2' are '1,2,3'" I1105 02:18:08.971] Nov 5 02:18:08.970: INFO: POD NODE PHASE GRACE CONDITIONS I1105 02:18:08.971] Nov 5 02:18:08.971: INFO: I1105 02:18:09.200] Nov 5 02:18:09.200: INFO: I1105 02:18:09.200] Logging node info for node k8s-agentpool-15826033-vmss000000 I1105 02:18:09.280] Nov 5 02:18:09.277: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-agentpool-15826033-vmss000000,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-agentpool-15826033-vmss000000,UID:15fa874e-54ec-42cd-b243-d3baa512150f,ResourceVersion:8737,Generation:0,CreationTimestamp:2019-11-05 01:34:17 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{agentpool: agentpool,beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_D4s_v3,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 0,kubernetes.azure.com/cluster: kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5,kubernetes.azure.com/role: agent,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-agentpool-15826033-vmss000000,kubernetes.io/os: linux,kubernetes.io/role: agent,node-role.kubernetes.io/agent: ,storageprofile: managed,storagetier: Premium_LRS,topology.disk.csi.azure.com/zone: 0,},Annotations:map[string]string{csi.volume.kubernetes.io/nodeid: {"disk.csi.azure.com":"k8s-agentpool-15826033-vmss000000"},node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.2.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool-15826033-vmss/virtualMachines/0,Unschedulable:false,Taints:[],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-csi-disk.csi.azure.com: {{8 0} {} 8 DecimalSI},cpu: {{4 0} {} 4 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{16797044736 0} {} 16403364Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-csi-disk.csi.azure.com: {{8 0} {} 8 DecimalSI},cpu: {{4 0} {} 4 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{16010612736 0} {} 15635364Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 01:34:56 +0000 UTC 2019-11-05 01:34:56 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 02:17:24 +0000 UTC 2019-11-05 01:34:17 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 02:17:24 +0000 UTC 2019-11-05 01:34:17 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 02:17:24 +0000 UTC 2019-11-05 01:34:17 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 02:17:24 +0000 UTC 2019-11-05 01:34:29 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.0.4} {Hostname k8s-agentpool-15826033-vmss000000}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:afe00792177d49a98f6a44d03d08732d,SystemUUID:4F1777BF-A5C3-004E-81C4-90F0B54C02A0,BootID:13b6a628-ae89-44af-b891-89461030adcb,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[k8sprowinternal.azurecr.io/azuredisk-csi@sha256:75d4ff5d7cd845d6e810c67796ecb037e852fb849ec1eb5b3194564008e6d1e0 k8sprowinternal.azurecr.io/azuredisk-csi:e2e-3e3ed045c6faa509dbd8d2890ee9f56aa2934887] 286930414} {[k8s.gcr.io/kubernetes-dashboard-amd64@sha256:0ae6b69432e78069c5ce2bcde0fe409c5c4d6f0f4d9cd50a17974fea38898747 k8s.gcr.io/kubernetes-dashboard-amd64:v1.10.1] 121711221} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/metrics-server-amd64@sha256:49a9f12f7067d11f42c803dbe61ed2c1299959ad85cb315b25ff7eef8e6b8892 k8s.gcr.io/metrics-server-amd64:v0.2.1] 42541759} {[mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume@sha256:4fd30d43947d4a54fc89ead7985beecfd3c9b2a93a0655a373b1608ab90bd5af mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume:v0.0.7] 22909487} {[quay.io/k8scsi/livenessprobe@sha256:8c9c8a0f330acc2e558ae6624309447d02522e76aa35af23fa19b88abf4381cb quay.io/k8scsi/livenessprobe:v1.0.2] 17415381} {[quay.io/k8scsi/csi-node-driver-registrar@sha256:13daf82fb99e951a4bff8ae5fc7c17c3a8fe7130be6400990d8f6076c32d4599 quay.io/k8scsi/csi-node-driver-registrar:v1.1.0] 15815995} {[mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume@sha256:23d8c6033f02a1ecad05127ebdc931bb871264228661bc122704b0974e4d9fdd mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume:1.0.8] 1159025} {[busybox@sha256:8ccbac733d19c0dd4d70b4f0c1e12245b5fa3ad24758a11035ee505c629c0796 busybox:1.29] 1154361} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 02:18:09.280] Nov 5 02:18:09.277: INFO: I1105 02:18:09.280] Logging kubelet events for node k8s-agentpool-15826033-vmss000000 I1105 02:18:09.360] Nov 5 02:18:09.360: INFO: I1105 02:18:09.360] Logging pods the kubelet thinks is on node k8s-agentpool-15826033-vmss000000 I1105 02:18:09.442] Nov 5 02:18:09.442: INFO: metrics-server-864ffbc5c-sdxps started at 2019-11-05 01:34:34 +0000 UTC (0+1 container statuses recorded) I1105 02:18:09.442] Nov 5 02:18:09.442: INFO: Container metrics-server ready: true, restart count 0 I1105 02:18:09.443] Nov 5 02:18:09.442: INFO: kube-proxy-wxblc started at 2019-11-05 01:34:34 +0000 UTC (0+1 container statuses recorded) I1105 02:18:09.443] Nov 5 02:18:09.442: INFO: Container kube-proxy ready: true, restart count 0 I1105 02:18:09.443] Nov 5 02:18:09.442: INFO: kubernetes-dashboard-66dd8b8df7-ks284 started at 2019-11-05 01:34:35 +0000 UTC (0+1 container statuses recorded) I1105 02:18:09.443] Nov 5 02:18:09.442: INFO: Container kubernetes-dashboard ready: true, restart count 0 I1105 02:18:09.443] Nov 5 02:18:09.442: INFO: csi-azuredisk-node-whq2g started at 2019-11-05 01:38:24 +0000 UTC (0+3 container statuses recorded) I1105 02:18:09.443] Nov 5 02:18:09.442: INFO: Container azuredisk ready: true, restart count 0 I1105 02:18:09.444] Nov 5 02:18:09.442: INFO: Container liveness-probe ready: true, restart count 0 I1105 02:18:09.444] Nov 5 02:18:09.442: INFO: Container node-driver-registrar ready: true, restart count 0 I1105 02:18:09.444] Nov 5 02:18:09.442: INFO: blobfuse-flexvol-installer-bsvqt started at 2019-11-05 01:34:34 +0000 UTC (0+1 container statuses recorded) I1105 02:18:09.444] Nov 5 02:18:09.442: INFO: Container blobfuse-flexvol-installer ready: true, restart count 0 I1105 02:18:09.444] Nov 5 02:18:09.442: INFO: azure-ip-masq-agent-vx86b started at 2019-11-05 01:34:34 +0000 UTC (0+1 container statuses recorded) I1105 02:18:09.444] Nov 5 02:18:09.442: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 02:18:09.445] Nov 5 02:18:09.442: INFO: keyvault-flexvolume-qhnfr started at 2019-11-05 01:34:34 +0000 UTC (0+1 container statuses recorded) I1105 02:18:09.445] Nov 5 02:18:09.442: INFO: Container keyvault-flexvolume ready: true, restart count 0 I1105 02:18:09.693] Nov 5 02:18:09.693: INFO: I1105 02:18:09.694] Latency metrics for node k8s-agentpool-15826033-vmss000000 I1105 02:18:09.694] Nov 5 02:18:09.693: INFO: I1105 02:18:09.694] Logging node info for node k8s-agentpool-15826033-vmss000001 I1105 02:18:09.773] Nov 5 02:18:09.770: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-agentpool-15826033-vmss000001,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-agentpool-15826033-vmss000001,UID:4a8ea605-cc0b-4e3b-9060-d8867d269e6c,ResourceVersion:8800,Generation:0,CreationTimestamp:2019-11-05 01:34:09 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{agentpool: agentpool,beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_D4s_v3,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 1,kubernetes.azure.com/cluster: kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5,kubernetes.azure.com/role: agent,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-agentpool-15826033-vmss000001,kubernetes.io/os: linux,kubernetes.io/role: agent,node-role.kubernetes.io/agent: ,storageprofile: managed,storagetier: Premium_LRS,topology.disk.csi.azure.com/zone: 1,},Annotations:map[string]string{csi.volume.kubernetes.io/nodeid: {"disk.csi.azure.com":"k8s-agentpool-15826033-vmss000001"},node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.0.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool-15826033-vmss/virtualMachines/1,Unschedulable:false,Taints:[],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-csi-disk.csi.azure.com: {{8 0} {} 8 DecimalSI},cpu: {{4 0} {} 4 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{16797044736 0} {} 16403364Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-csi-disk.csi.azure.com: {{8 0} {} 8 DecimalSI},cpu: {{4 0} {} 4 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{16010612736 0} {} 15635364Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 01:34:57 +0000 UTC 2019-11-05 01:34:57 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 02:17:46 +0000 UTC 2019-11-05 01:34:08 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 02:17:46 +0000 UTC 2019-11-05 01:34:08 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 02:17:46 +0000 UTC 2019-11-05 01:34:08 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 02:17:46 +0000 UTC 2019-11-05 01:34:29 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.0.5} {Hostname k8s-agentpool-15826033-vmss000001}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:b6b9fbf4a2fd4fa5a7ae472415fe30b8,SystemUUID:0162B3A4-3955-9A46-ABD4-A37BD75848CC,BootID:ee93003f-d47c-4118-9975-57365b3443a5,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[k8sprowinternal.azurecr.io/azuredisk-csi@sha256:75d4ff5d7cd845d6e810c67796ecb037e852fb849ec1eb5b3194564008e6d1e0 k8sprowinternal.azurecr.io/azuredisk-csi:e2e-3e3ed045c6faa509dbd8d2890ee9f56aa2934887] 286930414} {[gcr.io/kubernetes-helm/tiller@sha256:f6d8f4ab9ba993b5f5b60a6edafe86352eabe474ffeb84cb6c79b8866dce45d1 gcr.io/kubernetes-helm/tiller:v2.11.0] 71821984} {[quay.io/k8scsi/csi-provisioner@sha256:d657c839dce87324fe2b677302913f9386f885f8746be7bea0ced5b0844e3433 quay.io/k8scsi/csi-provisioner:v1.3.1] 50596998} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[quay.io/k8scsi/csi-attacher@sha256:26fccd7a99d973845df1193b46ebdcc6ab8dc5f6e6be319750c471fce1742d13 quay.io/k8scsi/csi-attacher:v1.2.0] 46226754} {[quay.io/k8scsi/csi-cluster-driver-registrar@sha256:fafd75ae5442f192cfa8c2e792903aee30d5884b62e802e4464b0a895d21e3ef quay.io/k8scsi/csi-cluster-driver-registrar:v1.0.1] 45874691} {[quay.io/k8scsi/csi-snapshotter@sha256:a49e0da1af6f2bf717e41ba1eee8b5e6a1cbd66a709dd92cc43fe475fe2589eb quay.io/k8scsi/csi-snapshotter:v1.1.0] 44325352} {[k8s.gcr.io/coredns@sha256:e83beb5e43f8513fa735e77ffc5859640baea30a882a11cc75c4c3244a737d3c k8s.gcr.io/coredns:1.5.0] 42488424} {[mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume@sha256:4fd30d43947d4a54fc89ead7985beecfd3c9b2a93a0655a373b1608ab90bd5af mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume:v0.0.7] 22909487} {[quay.io/k8scsi/livenessprobe@sha256:8c9c8a0f330acc2e558ae6624309447d02522e76aa35af23fa19b88abf4381cb quay.io/k8scsi/livenessprobe:v1.0.2] 17415381} {[quay.io/k8scsi/csi-node-driver-registrar@sha256:13daf82fb99e951a4bff8ae5fc7c17c3a8fe7130be6400990d8f6076c32d4599 quay.io/k8scsi/csi-node-driver-registrar:v1.1.0] 15815995} {[quay.io/k8scsi/livenessprobe@sha256:dde617756e0f602adc566ab71fd885f1dad451ad3fb063ac991c95a2ff47aea5 quay.io/k8scsi/livenessprobe:v1.1.0] 14967303} {[mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume@sha256:23d8c6033f02a1ecad05127ebdc931bb871264228661bc122704b0974e4d9fdd mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume:1.0.8] 1159025} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 02:18:09.774] Nov 5 02:18:09.770: INFO: I1105 02:18:09.774] Logging kubelet events for node k8s-agentpool-15826033-vmss000001 I1105 02:18:09.851] Nov 5 02:18:09.851: INFO: I1105 02:18:09.852] Logging pods the kubelet thinks is on node k8s-agentpool-15826033-vmss000001 I1105 02:18:09.935] Nov 5 02:18:09.935: INFO: csi-azuredisk-controller-6f6587bfc6-8vkfl started at 2019-11-05 01:38:24 +0000 UTC (0+6 container statuses recorded) I1105 02:18:09.936] Nov 5 02:18:09.935: INFO: Container azuredisk ready: true, restart count 0 I1105 02:18:09.936] Nov 5 02:18:09.935: INFO: Container cluster-driver-registrar ready: true, restart count 0 I1105 02:18:09.936] Nov 5 02:18:09.935: INFO: Container csi-attacher ready: true, restart count 0 I1105 02:18:09.936] Nov 5 02:18:09.935: INFO: Container csi-provisioner ready: true, restart count 0 I1105 02:18:09.936] Nov 5 02:18:09.935: INFO: Container csi-snapshotter ready: true, restart count 0 I1105 02:18:09.936] Nov 5 02:18:09.935: INFO: Container liveness-probe ready: true, restart count 0 I1105 02:18:09.937] Nov 5 02:18:09.935: INFO: blobfuse-flexvol-installer-rw5dw started at 2019-11-05 01:34:35 +0000 UTC (0+1 container statuses recorded) I1105 02:18:09.937] Nov 5 02:18:09.935: INFO: Container blobfuse-flexvol-installer ready: true, restart count 0 I1105 02:18:09.937] Nov 5 02:18:09.935: INFO: coredns-7f68dcdbdb-jt522 started at 2019-11-05 01:34:35 +0000 UTC (0+1 container statuses recorded) I1105 02:18:09.937] Nov 5 02:18:09.935: INFO: Container coredns ready: true, restart count 0 I1105 02:18:09.937] Nov 5 02:18:09.935: INFO: azure-ip-masq-agent-6c6vn started at 2019-11-05 01:34:35 +0000 UTC (0+1 container statuses recorded) I1105 02:18:09.938] Nov 5 02:18:09.935: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 02:18:09.938] Nov 5 02:18:09.935: INFO: keyvault-flexvolume-rdlcj started at 2019-11-05 01:34:35 +0000 UTC (0+1 container statuses recorded) I1105 02:18:09.938] Nov 5 02:18:09.935: INFO: Container keyvault-flexvolume ready: true, restart count 0 I1105 02:18:09.938] Nov 5 02:18:09.935: INFO: kube-proxy-6kh57 started at 2019-11-05 01:34:35 +0000 UTC (0+1 container statuses recorded) I1105 02:18:09.938] Nov 5 02:18:09.935: INFO: Container kube-proxy ready: true, restart count 0 I1105 02:18:09.938] Nov 5 02:18:09.935: INFO: tiller-deploy-54c96cb5df-8d9fs started at 2019-11-05 01:34:35 +0000 UTC (0+1 container statuses recorded) I1105 02:18:09.939] Nov 5 02:18:09.935: INFO: Container tiller ready: true, restart count 0 I1105 02:18:09.939] Nov 5 02:18:09.935: INFO: csi-azuredisk-node-ghv5w started at 2019-11-05 01:38:24 +0000 UTC (0+3 container statuses recorded) I1105 02:18:09.939] Nov 5 02:18:09.935: INFO: Container azuredisk ready: true, restart count 0 I1105 02:18:09.939] Nov 5 02:18:09.935: INFO: Container liveness-probe ready: true, restart count 0 I1105 02:18:09.939] Nov 5 02:18:09.935: INFO: Container node-driver-registrar ready: true, restart count 0 I1105 02:18:10.196] Nov 5 02:18:10.196: INFO: I1105 02:18:10.197] Latency metrics for node k8s-agentpool-15826033-vmss000001 I1105 02:18:10.197] Nov 5 02:18:10.196: INFO: I1105 02:18:10.197] Logging node info for node k8s-master-15826033-0 I1105 02:18:10.275] Nov 5 02:18:10.273: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-15826033-0,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-15826033-0,UID:aceb0231-4645-4a9a-b296-90ada851986b,ResourceVersion:8763,Generation:0,CreationTimestamp:2019-11-05 01:35:06 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 1,kubernetes.azure.com/cluster: kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-15826033-0,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.4.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5/providers/Microsoft.Compute/virtualMachines/k8s-master-15826033-0,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 01:35:20 +0000 UTC 2019-11-05 01:35:20 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 02:17:32 +0000 UTC 2019-11-05 01:34:57 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 02:17:32 +0000 UTC 2019-11-05 01:34:57 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 02:17:32 +0000 UTC 2019-11-05 01:34:57 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 02:17:32 +0000 UTC 2019-11-05 01:35:07 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.255.5} {Hostname k8s-master-15826033-0}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:1915756c22a540dc8eb30493987e9a43,SystemUUID:742C969C-5A71-D744-8E61-F27E38B56EB2,BootID:03f0672e-8fa9-4309-956f-8e3c371902c3,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 02:18:10.276] Nov 5 02:18:10.273: INFO: I1105 02:18:10.276] Logging kubelet events for node k8s-master-15826033-0 I1105 02:18:10.355] Nov 5 02:18:10.354: INFO: I1105 02:18:10.355] Logging pods the kubelet thinks is on node k8s-master-15826033-0 I1105 02:18:10.436] Nov 5 02:18:10.436: INFO: kube-apiserver-k8s-master-15826033-0 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:18:10.436] Nov 5 02:18:10.436: INFO: Container kube-apiserver ready: true, restart count 0 I1105 02:18:10.436] Nov 5 02:18:10.436: INFO: kube-controller-manager-k8s-master-15826033-0 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:18:10.437] Nov 5 02:18:10.436: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 02:18:10.437] Nov 5 02:18:10.436: INFO: kube-scheduler-k8s-master-15826033-0 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:18:10.437] Nov 5 02:18:10.436: INFO: Container kube-scheduler ready: true, restart count 0 I1105 02:18:10.437] Nov 5 02:18:10.436: INFO: azure-ip-masq-agent-wgdpx started at 2019-11-05 01:35:06 +0000 UTC (0+1 container statuses recorded) I1105 02:18:10.437] Nov 5 02:18:10.436: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 02:18:10.438] Nov 5 02:18:10.436: INFO: kube-proxy-7d2px started at 2019-11-05 01:35:06 +0000 UTC (0+1 container statuses recorded) I1105 02:18:10.438] Nov 5 02:18:10.436: INFO: Container kube-proxy ready: true, restart count 0 I1105 02:18:10.438] Nov 5 02:18:10.436: INFO: cloud-controller-manager-k8s-master-15826033-0 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:18:10.438] Nov 5 02:18:10.436: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 02:18:10.438] Nov 5 02:18:10.436: INFO: kube-addon-manager-k8s-master-15826033-0 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:18:10.439] Nov 5 02:18:10.436: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 02:18:10.690] Nov 5 02:18:10.690: INFO: I1105 02:18:10.691] Latency metrics for node k8s-master-15826033-0 I1105 02:18:10.691] Nov 5 02:18:10.690: INFO: I1105 02:18:10.691] Logging node info for node k8s-master-15826033-1 I1105 02:18:10.770] Nov 5 02:18:10.767: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-15826033-1,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-15826033-1,UID:1a13ec61-c89a-456c-8c94-bfc8e717c2e7,ResourceVersion:8765,Generation:0,CreationTimestamp:2019-11-05 01:35:07 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 1,kubernetes.azure.com/cluster: kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-15826033-1,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.5.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5/providers/Microsoft.Compute/virtualMachines/k8s-master-15826033-1,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 01:35:31 +0000 UTC 2019-11-05 01:35:31 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 02:17:33 +0000 UTC 2019-11-05 01:35:00 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 02:17:33 +0000 UTC 2019-11-05 01:35:00 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 02:17:33 +0000 UTC 2019-11-05 01:35:00 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 02:17:33 +0000 UTC 2019-11-05 01:35:07 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.255.6} {Hostname k8s-master-15826033-1}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:e0f2bccddd5b4555b180d14081ab59a8,SystemUUID:AE126631-CAD2-B54F-B22E-1F499342A4D6,BootID:509b9a9d-b046-4861-aa35-8bcb77465de2,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 02:18:10.770] Nov 5 02:18:10.768: INFO: I1105 02:18:10.770] Logging kubelet events for node k8s-master-15826033-1 I1105 02:18:10.849] Nov 5 02:18:10.848: INFO: I1105 02:18:10.849] Logging pods the kubelet thinks is on node k8s-master-15826033-1 I1105 02:18:10.929] Nov 5 02:18:10.929: INFO: kube-controller-manager-k8s-master-15826033-1 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:18:10.929] Nov 5 02:18:10.929: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 02:18:10.929] Nov 5 02:18:10.929: INFO: kube-scheduler-k8s-master-15826033-1 started at 2019-11-05 01:36:16 +0000 UTC (0+1 container statuses recorded) I1105 02:18:10.929] Nov 5 02:18:10.929: INFO: Container kube-scheduler ready: true, restart count 0 I1105 02:18:10.930] Nov 5 02:18:10.929: INFO: azure-ip-masq-agent-w6dpr started at 2019-11-05 01:35:10 +0000 UTC (0+1 container statuses recorded) I1105 02:18:10.930] Nov 5 02:18:10.929: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 02:18:10.930] Nov 5 02:18:10.929: INFO: kube-proxy-8kmr5 started at 2019-11-05 01:35:10 +0000 UTC (0+1 container statuses recorded) I1105 02:18:10.930] Nov 5 02:18:10.929: INFO: Container kube-proxy ready: true, restart count 0 I1105 02:18:10.930] Nov 5 02:18:10.929: INFO: cloud-controller-manager-k8s-master-15826033-1 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:18:10.931] Nov 5 02:18:10.929: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 02:18:10.931] Nov 5 02:18:10.929: INFO: kube-addon-manager-k8s-master-15826033-1 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:18:10.931] Nov 5 02:18:10.929: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 02:18:10.931] Nov 5 02:18:10.929: INFO: kube-apiserver-k8s-master-15826033-1 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:18:10.931] Nov 5 02:18:10.929: INFO: Container kube-apiserver ready: true, restart count 0 I1105 02:18:11.180] Nov 5 02:18:11.180: INFO: I1105 02:18:11.181] Latency metrics for node k8s-master-15826033-1 I1105 02:18:11.181] Nov 5 02:18:11.180: INFO: I1105 02:18:11.181] Logging node info for node k8s-master-15826033-2 I1105 02:18:11.260] Nov 5 02:18:11.257: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-15826033-2,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-15826033-2,UID:acb3a851-d6df-4549-ba56-4b7e96553c17,ResourceVersion:8854,Generation:0,CreationTimestamp:2019-11-05 01:34:09 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 2,kubernetes.azure.com/cluster: kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-15826033-2,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.1.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5/providers/Microsoft.Compute/virtualMachines/k8s-master-15826033-2,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 01:34:44 +0000 UTC 2019-11-05 01:34:44 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 02:18:04 +0000 UTC 2019-11-05 01:34:07 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 02:18:04 +0000 UTC 2019-11-05 01:34:07 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 02:18:04 +0000 UTC 2019-11-05 01:34:07 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 02:18:04 +0000 UTC 2019-11-05 01:34:29 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{Hostname k8s-master-15826033-2} {InternalIP 10.240.255.7}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:0885b0c2f6174106bde4fa0eae2cbc6b,SystemUUID:C176121C-5D7A-6243-9B05-8B68F8022B64,BootID:30430638-7ab6-4761-b984-2387e1a3f5b2,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 02:18:11.260] Nov 5 02:18:11.258: INFO: I1105 02:18:11.261] Logging kubelet events for node k8s-master-15826033-2 I1105 02:18:11.338] Nov 5 02:18:11.338: INFO: I1105 02:18:11.338] Logging pods the kubelet thinks is on node k8s-master-15826033-2 I1105 02:18:11.421] Nov 5 02:18:11.421: INFO: cloud-controller-manager-k8s-master-15826033-2 started at 2019-11-05 01:33:54 +0000 UTC (0+1 container statuses recorded) I1105 02:18:11.421] Nov 5 02:18:11.421: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 02:18:11.421] Nov 5 02:18:11.421: INFO: kube-addon-manager-k8s-master-15826033-2 started at 2019-11-05 01:33:54 +0000 UTC (0+1 container statuses recorded) I1105 02:18:11.421] Nov 5 02:18:11.421: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 02:18:11.421] Nov 5 02:18:11.421: INFO: kube-apiserver-k8s-master-15826033-2 started at 2019-11-05 01:33:54 +0000 UTC (0+1 container statuses recorded) I1105 02:18:11.422] Nov 5 02:18:11.421: INFO: Container kube-apiserver ready: true, restart count 0 I1105 02:18:11.422] Nov 5 02:18:11.421: INFO: kube-controller-manager-k8s-master-15826033-2 started at 2019-11-05 01:33:54 +0000 UTC (0+1 container statuses recorded) I1105 02:18:11.422] Nov 5 02:18:11.421: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 02:18:11.422] Nov 5 02:18:11.421: INFO: kube-scheduler-k8s-master-15826033-2 started at 2019-11-05 01:33:54 +0000 UTC (0+1 container statuses recorded) I1105 02:18:11.423] Nov 5 02:18:11.421: INFO: Container kube-scheduler ready: true, restart count 0 I1105 02:18:11.423] Nov 5 02:18:11.421: INFO: azure-ip-masq-agent-hw7s5 started at 2019-11-05 01:34:34 +0000 UTC (0+1 container statuses recorded) I1105 02:18:11.423] Nov 5 02:18:11.421: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 02:18:11.423] Nov 5 02:18:11.421: INFO: kube-proxy-5xpp6 started at 2019-11-05 01:34:34 +0000 UTC (0+1 container statuses recorded) I1105 02:18:11.423] Nov 5 02:18:11.421: INFO: Container kube-proxy ready: true, restart count 0 I1105 02:18:11.673] Nov 5 02:18:11.672: INFO: I1105 02:18:11.673] Latency metrics for node k8s-master-15826033-2 I1105 02:18:11.673] Nov 5 02:18:11.673: INFO: I1105 02:18:11.673] Logging node info for node k8s-master-15826033-3 I1105 02:18:11.752] Nov 5 02:18:11.749: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-15826033-3,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-15826033-3,UID:b7702272-1876-434e-b6a0-17fa34c6757f,ResourceVersion:8759,Generation:0,CreationTimestamp:2019-11-05 01:35:06 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 0,kubernetes.azure.com/cluster: kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-15826033-3,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.3.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5/providers/Microsoft.Compute/virtualMachines/k8s-master-15826033-3,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 01:35:17 +0000 UTC 2019-11-05 01:35:17 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 02:17:31 +0000 UTC 2019-11-05 01:35:00 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 02:17:31 +0000 UTC 2019-11-05 01:35:00 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 02:17:31 +0000 UTC 2019-11-05 01:35:00 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 02:17:31 +0000 UTC 2019-11-05 01:35:06 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.255.8} {Hostname k8s-master-15826033-3}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:fbf366963961422c9d90bedf6639c8e1,SystemUUID:5388948D-1A8D-EA41-B8B8-7F4523EDF5AA,BootID:42f5884c-7418-486a-8cf3-75188a331f38,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 02:18:11.752] Nov 5 02:18:11.750: INFO: I1105 02:18:11.752] Logging kubelet events for node k8s-master-15826033-3 I1105 02:18:11.830] Nov 5 02:18:11.830: INFO: I1105 02:18:11.831] Logging pods the kubelet thinks is on node k8s-master-15826033-3 I1105 02:18:11.912] Nov 5 02:18:11.912: INFO: cloud-controller-manager-k8s-master-15826033-3 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:18:11.912] Nov 5 02:18:11.912: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 02:18:11.913] Nov 5 02:18:11.912: INFO: kube-addon-manager-k8s-master-15826033-3 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:18:11.913] Nov 5 02:18:11.912: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 02:18:11.913] Nov 5 02:18:11.912: INFO: kube-apiserver-k8s-master-15826033-3 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:18:11.913] Nov 5 02:18:11.912: INFO: Container kube-apiserver ready: true, restart count 0 I1105 02:18:11.913] Nov 5 02:18:11.912: INFO: kube-controller-manager-k8s-master-15826033-3 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:18:11.913] Nov 5 02:18:11.912: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 02:18:11.913] Nov 5 02:18:11.912: INFO: kube-scheduler-k8s-master-15826033-3 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:18:11.914] Nov 5 02:18:11.912: INFO: Container kube-scheduler ready: true, restart count 0 I1105 02:18:11.914] Nov 5 02:18:11.912: INFO: kube-proxy-xz24z started at 2019-11-05 01:35:06 +0000 UTC (0+1 container statuses recorded) I1105 02:18:11.914] Nov 5 02:18:11.912: INFO: Container kube-proxy ready: true, restart count 0 I1105 02:18:11.914] Nov 5 02:18:11.912: INFO: azure-ip-masq-agent-dqfpw started at 2019-11-05 01:35:06 +0000 UTC (0+1 container statuses recorded) I1105 02:18:11.914] Nov 5 02:18:11.912: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 02:18:12.168] Nov 5 02:18:12.168: INFO: I1105 02:18:12.169] Latency metrics for node k8s-master-15826033-3 I1105 02:18:12.169] Nov 5 02:18:12.168: INFO: I1105 02:18:12.169] Logging node info for node k8s-master-15826033-4 I1105 02:18:12.247] Nov 5 02:18:12.245: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-15826033-4,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-15826033-4,UID:276daef1-6c05-41e3-bacd-abc93d35cc6d,ResourceVersion:8766,Generation:0,CreationTimestamp:2019-11-05 01:35:07 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 0,kubernetes.azure.com/cluster: kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-15826033-4,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.6.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5/providers/Microsoft.Compute/virtualMachines/k8s-master-15826033-4,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 01:35:41 +0000 UTC 2019-11-05 01:35:41 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 02:17:33 +0000 UTC 2019-11-05 01:35:02 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 02:17:33 +0000 UTC 2019-11-05 01:35:02 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 02:17:33 +0000 UTC 2019-11-05 01:35:02 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 02:17:33 +0000 UTC 2019-11-05 01:35:08 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.255.9} {Hostname k8s-master-15826033-4}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:79051fc0bdeb4e32a56acbaf5972ce3a,SystemUUID:C71606B5-7F08-9341-A26D-8A8D9D84FCAE,BootID:4bdf7cf3-2b83-42c7-a7ba-926ebe9ca75b,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 02:18:12.248] Nov 5 02:18:12.245: INFO: I1105 02:18:12.248] Logging kubelet events for node k8s-master-15826033-4 I1105 02:18:12.326] Nov 5 02:18:12.326: INFO: I1105 02:18:12.326] Logging pods the kubelet thinks is on node k8s-master-15826033-4 I1105 02:18:12.408] Nov 5 02:18:12.408: INFO: kube-scheduler-k8s-master-15826033-4 started at 2019-11-05 01:34:55 +0000 UTC (0+1 container statuses recorded) I1105 02:18:12.409] Nov 5 02:18:12.408: INFO: Container kube-scheduler ready: true, restart count 0 I1105 02:18:12.409] Nov 5 02:18:12.408: INFO: cloud-controller-manager-k8s-master-15826033-4 started at 2019-11-05 01:34:56 +0000 UTC (0+1 container statuses recorded) I1105 02:18:12.409] Nov 5 02:18:12.408: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 02:18:12.409] Nov 5 02:18:12.408: INFO: kube-addon-manager-k8s-master-15826033-4 started at 2019-11-05 01:34:56 +0000 UTC (0+1 container statuses recorded) I1105 02:18:12.409] Nov 5 02:18:12.408: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 02:18:12.410] Nov 5 02:18:12.408: INFO: azure-ip-masq-agent-9nrj8 started at 2019-11-05 01:35:11 +0000 UTC (0+1 container statuses recorded) I1105 02:18:12.410] Nov 5 02:18:12.408: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 02:18:12.410] Nov 5 02:18:12.408: INFO: kube-proxy-rmngk started at 2019-11-05 01:35:11 +0000 UTC (0+1 container statuses recorded) I1105 02:18:12.410] Nov 5 02:18:12.408: INFO: Container kube-proxy ready: true, restart count 0 I1105 02:18:12.410] Nov 5 02:18:12.408: INFO: kube-apiserver-k8s-master-15826033-4 started at 2019-11-05 01:34:55 +0000 UTC (0+1 container statuses recorded) I1105 02:18:12.410] Nov 5 02:18:12.408: INFO: Container kube-apiserver ready: true, restart count 0 I1105 02:18:12.411] Nov 5 02:18:12.408: INFO: kube-controller-manager-k8s-master-15826033-4 started at 2019-11-05 01:34:55 +0000 UTC (0+1 container statuses recorded) I1105 02:18:12.411] Nov 5 02:18:12.408: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 02:18:12.657] Nov 5 02:18:12.657: INFO: I1105 02:18:12.658] Latency metrics for node k8s-master-15826033-4 I1105 02:18:12.658] Nov 5 02:18:12.657: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready I1105 02:18:12.735] STEP: Destroying namespace "azuredisk-8234" for this suite. I1105 02:18:26.972] Nov 5 02:18:26.972: INFO: Waiting up to 30s for server preferred namespaced resources to be successfully discovered I1105 02:18:29.925] Nov 5 02:18:29.924: INFO: namespace azuredisk-8234 deletion completed in 17.189522955s I1105 02:18:29.925] I1105 02:18:29.925] • Failure [323.232 seconds] I1105 02:18:29.925] [azuredisk-csi-e2e] Dynamic Provisioning I1105 02:18:29.926] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:33 I1105 02:18:29.926] [single-az] I1105 02:18:29.926] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:36 I1105 02:18:29.926] should create a raw block volume on demand [It] I1105 02:18:29.927] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:101 I1105 02:18:29.927] I1105 02:18:29.927] Unexpected error: I1105 02:18:29.927] <*errors.errorString | 0xc000f8add0>: { I1105 02:18:29.927] s: "PersistentVolumeClaims [pvc-zvbnt] not all in phase Bound within 5m0s", I1105 02:18:29.927] } I1105 02:18:29.927] PersistentVolumeClaims [pvc-zvbnt] not all in phase Bound within 5m0s I1105 02:18:29.928] occurred I1105 02:18:29.928] I1105 02:18:29.928] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/testsuites/testsuites.go:278 I1105 02:18:29.928] ------------------------------ I1105 02:18:29.928] [azuredisk-csi-e2e] Dynamic Provisioning [single-az] I1105 02:18:29.928] should create a deployment object, write and read to it, delete the pod and write and read to it again I1105 02:18:29.928] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:193 I1105 02:18:29.929] [BeforeEach] [single-az] I1105 02:18:29.929] /go/src/sigs.k8s.io/azuredisk-csi-driver/vendor/k8s.io/kubernetes/test/e2e/framework/framework.go:150 I1105 02:18:29.929] STEP: Creating a kubernetes client I1105 02:18:29.929] Nov 5 02:18:29.925: INFO: >>> kubeConfig: /workspace/aks978705698/kubeconfig/kubeconfig.eastus2.json I1105 02:18:29.929] STEP: Building a namespace api object, basename azuredisk I1105 02:18:30.160] STEP: Binding the e2e-test-privileged-psp PodSecurityPolicy to the default service account in azuredisk-1124 I1105 02:18:30.419] STEP: Waiting for a default service account to be provisioned in namespace I1105 02:18:30.494] [BeforeEach] [single-az] I1105 02:18:30.495] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:58 I1105 02:18:30.495] [It] should create a deployment object, write and read to it, delete the pod and write and read to it again I1105 02:18:30.495] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:193 I1105 02:18:30.495] STEP: setting up the StorageClass I1105 02:18:30.495] STEP: creating a StorageClass I1105 02:18:30.576] STEP: setting up the PVC I1105 02:18:30.576] STEP: creating a PVC I1105 02:18:30.664] STEP: waiting for PVC to be in phase "Bound" I1105 02:18:30.664] Nov 5 02:18:30.664: INFO: Waiting up to 5m0s for PersistentVolumeClaims [pvc-fbpb7] to have phase Bound I1105 02:18:30.741] Nov 5 02:18:30.741: INFO: PersistentVolumeClaim pvc-fbpb7 found but phase is Pending instead of Bound. I1105 02:18:32.818] Nov 5 02:18:32.818: INFO: PersistentVolumeClaim pvc-fbpb7 found but phase is Pending instead of Bound. I1105 02:18:34.896] Nov 5 02:18:34.896: INFO: PersistentVolumeClaim pvc-fbpb7 found but phase is Pending instead of Bound. I1105 02:18:36.973] Nov 5 02:18:36.973: INFO: PersistentVolumeClaim pvc-fbpb7 found but phase is Pending instead of Bound. I1105 02:18:39.052] Nov 5 02:18:39.052: INFO: PersistentVolumeClaim pvc-fbpb7 found but phase is Pending instead of Bound. I1105 02:18:41.135] Nov 5 02:18:41.134: INFO: PersistentVolumeClaim pvc-fbpb7 found but phase is Pending instead of Bound. I1105 02:18:43.212] Nov 5 02:18:43.212: INFO: PersistentVolumeClaim pvc-fbpb7 found but phase is Pending instead of Bound. I1105 02:18:45.290] Nov 5 02:18:45.289: INFO: PersistentVolumeClaim pvc-fbpb7 found but phase is Pending instead of Bound. I1105 02:18:47.368] Nov 5 02:18:47.367: INFO: PersistentVolumeClaim pvc-fbpb7 found but phase is Pending instead of Bound. I1105 02:18:49.445] Nov 5 02:18:49.445: INFO: PersistentVolumeClaim pvc-fbpb7 found but phase is Pending instead of Bound. I1105 02:18:51.523] Nov 5 02:18:51.523: INFO: PersistentVolumeClaim pvc-fbpb7 found but phase is Pending instead of Bound. I1105 02:18:53.601] Nov 5 02:18:53.600: INFO: PersistentVolumeClaim pvc-fbpb7 found but phase is Pending instead of Bound. I1105 02:18:55.678] Nov 5 02:18:55.678: INFO: PersistentVolumeClaim pvc-fbpb7 found but phase is Pending instead of Bound. I1105 02:18:57.755] Nov 5 02:18:57.755: INFO: PersistentVolumeClaim pvc-fbpb7 found but phase is Pending instead of Bound. I1105 02:18:59.833] Nov 5 02:18:59.832: INFO: PersistentVolumeClaim pvc-fbpb7 found but phase is Pending instead of Bound. I1105 02:19:01.910] Nov 5 02:19:01.910: INFO: PersistentVolumeClaim pvc-fbpb7 found but phase is Pending instead of Bound. I1105 02:19:03.988] Nov 5 02:19:03.988: INFO: PersistentVolumeClaim pvc-fbpb7 found but phase is Pending instead of Bound. I1105 02:19:06.066] Nov 5 02:19:06.065: INFO: PersistentVolumeClaim pvc-fbpb7 found but phase is Pending instead of Bound. I1105 02:19:08.143] Nov 5 02:19:08.143: INFO: PersistentVolumeClaim pvc-fbpb7 found but phase is Pending instead of Bound. I1105 02:19:10.221] Nov 5 02:19:10.221: INFO: PersistentVolumeClaim pvc-fbpb7 found but phase is Pending instead of Bound. I1105 02:19:12.299] Nov 5 02:19:12.299: INFO: PersistentVolumeClaim pvc-fbpb7 found but phase is Pending instead of Bound. I1105 02:19:14.377] Nov 5 02:19:14.377: INFO: PersistentVolumeClaim pvc-fbpb7 found but phase is Pending instead of Bound. I1105 02:19:16.457] Nov 5 02:19:16.457: INFO: PersistentVolumeClaim pvc-fbpb7 found but phase is Pending instead of Bound. I1105 02:19:18.535] Nov 5 02:19:18.535: INFO: PersistentVolumeClaim pvc-fbpb7 found but phase is Pending instead of Bound. I1105 02:19:20.613] Nov 5 02:19:20.613: INFO: PersistentVolumeClaim pvc-fbpb7 found but phase is Pending instead of Bound. I1105 02:19:22.690] Nov 5 02:19:22.690: INFO: PersistentVolumeClaim pvc-fbpb7 found but phase is Pending instead of Bound. I1105 02:19:24.767] Nov 5 02:19:24.767: INFO: PersistentVolumeClaim pvc-fbpb7 found but phase is Pending instead of Bound. I1105 02:19:26.845] Nov 5 02:19:26.844: INFO: PersistentVolumeClaim pvc-fbpb7 found but phase is Pending instead of Bound. I1105 02:19:28.922] Nov 5 02:19:28.922: INFO: PersistentVolumeClaim pvc-fbpb7 found but phase is Pending instead of Bound. I1105 02:19:31.000] Nov 5 02:19:30.999: INFO: PersistentVolumeClaim pvc-fbpb7 found but phase is Pending instead of Bound. I1105 02:19:33.080] Nov 5 02:19:33.080: INFO: PersistentVolumeClaim pvc-fbpb7 found but phase is Pending instead of Bound. I1105 02:19:35.158] Nov 5 02:19:35.158: INFO: PersistentVolumeClaim pvc-fbpb7 found but phase is Pending instead of Bound. I1105 02:19:37.235] Nov 5 02:19:37.235: INFO: PersistentVolumeClaim pvc-fbpb7 found but phase is Pending instead of Bound. I1105 02:19:39.312] Nov 5 02:19:39.312: INFO: PersistentVolumeClaim pvc-fbpb7 found but phase is Pending instead of Bound. I1105 02:19:41.390] Nov 5 02:19:41.389: INFO: PersistentVolumeClaim pvc-fbpb7 found but phase is Pending instead of Bound. I1105 02:19:43.467] Nov 5 02:19:43.467: INFO: PersistentVolumeClaim pvc-fbpb7 found but phase is Pending instead of Bound. I1105 02:19:45.545] Nov 5 02:19:45.545: INFO: PersistentVolumeClaim pvc-fbpb7 found but phase is Pending instead of Bound. I1105 02:19:47.622] Nov 5 02:19:47.622: INFO: PersistentVolumeClaim pvc-fbpb7 found but phase is Pending instead of Bound. I1105 02:19:49.700] Nov 5 02:19:49.700: INFO: PersistentVolumeClaim pvc-fbpb7 found but phase is Pending instead of Bound. I1105 02:19:51.778] Nov 5 02:19:51.777: INFO: PersistentVolumeClaim pvc-fbpb7 found but phase is Pending instead of Bound. I1105 02:19:53.856] Nov 5 02:19:53.856: INFO: PersistentVolumeClaim pvc-fbpb7 found but phase is Pending instead of Bound. I1105 02:19:55.934] Nov 5 02:19:55.934: INFO: PersistentVolumeClaim pvc-fbpb7 found but phase is Pending instead of Bound. I1105 02:19:58.011] Nov 5 02:19:58.011: INFO: PersistentVolumeClaim pvc-fbpb7 found but phase is Pending instead of Bound. I1105 02:20:00.090] Nov 5 02:20:00.090: INFO: PersistentVolumeClaim pvc-fbpb7 found but phase is Pending instead of Bound. I1105 02:20:02.168] Nov 5 02:20:02.168: INFO: PersistentVolumeClaim pvc-fbpb7 found but phase is Pending instead of Bound. I1105 02:20:04.246] Nov 5 02:20:04.246: INFO: PersistentVolumeClaim pvc-fbpb7 found but phase is Pending instead of Bound. I1105 02:20:06.324] Nov 5 02:20:06.324: INFO: PersistentVolumeClaim pvc-fbpb7 found but phase is Pending instead of Bound. I1105 02:20:08.401] Nov 5 02:20:08.401: INFO: PersistentVolumeClaim pvc-fbpb7 found but phase is Pending instead of Bound. I1105 02:20:10.479] Nov 5 02:20:10.478: INFO: PersistentVolumeClaim pvc-fbpb7 found but phase is Pending instead of Bound. I1105 02:20:12.556] Nov 5 02:20:12.556: INFO: PersistentVolumeClaim pvc-fbpb7 found but phase is Pending instead of Bound. I1105 02:20:14.633] Nov 5 02:20:14.633: INFO: PersistentVolumeClaim pvc-fbpb7 found but phase is Pending instead of Bound. I1105 02:20:16.711] Nov 5 02:20:16.710: INFO: PersistentVolumeClaim pvc-fbpb7 found but phase is Pending instead of Bound. I1105 02:20:18.788] Nov 5 02:20:18.787: INFO: PersistentVolumeClaim pvc-fbpb7 found but phase is Pending instead of Bound. I1105 02:20:20.866] Nov 5 02:20:20.865: INFO: PersistentVolumeClaim pvc-fbpb7 found but phase is Pending instead of Bound. I1105 02:20:22.943] Nov 5 02:20:22.943: INFO: PersistentVolumeClaim pvc-fbpb7 found but phase is Pending instead of Bound. I1105 02:20:25.020] Nov 5 02:20:25.020: INFO: PersistentVolumeClaim pvc-fbpb7 found but phase is Pending instead of Bound. I1105 02:20:27.098] Nov 5 02:20:27.097: INFO: PersistentVolumeClaim pvc-fbpb7 found but phase is Pending instead of Bound. I1105 02:20:29.175] Nov 5 02:20:29.175: INFO: PersistentVolumeClaim pvc-fbpb7 found but phase is Pending instead of Bound. I1105 02:20:31.254] Nov 5 02:20:31.254: INFO: PersistentVolumeClaim pvc-fbpb7 found but phase is Pending instead of Bound. I1105 02:20:33.333] Nov 5 02:20:33.333: INFO: PersistentVolumeClaim pvc-fbpb7 found but phase is Pending instead of Bound. I1105 02:20:35.411] Nov 5 02:20:35.411: INFO: PersistentVolumeClaim pvc-fbpb7 found but phase is Pending instead of Bound. I1105 02:20:37.488] Nov 5 02:20:37.488: INFO: PersistentVolumeClaim pvc-fbpb7 found but phase is Pending instead of Bound. I1105 02:20:39.566] Nov 5 02:20:39.566: INFO: PersistentVolumeClaim pvc-fbpb7 found but phase is Pending instead of Bound. I1105 02:20:41.644] Nov 5 02:20:41.643: INFO: PersistentVolumeClaim pvc-fbpb7 found but phase is Pending instead of Bound. I1105 02:20:43.721] Nov 5 02:20:43.721: INFO: PersistentVolumeClaim pvc-fbpb7 found but phase is Pending instead of Bound. I1105 02:20:45.806] Nov 5 02:20:45.806: INFO: PersistentVolumeClaim pvc-fbpb7 found but phase is Pending instead of Bound. I1105 02:20:47.884] Nov 5 02:20:47.883: INFO: PersistentVolumeClaim pvc-fbpb7 found but phase is Pending instead of Bound. I1105 02:20:49.964] Nov 5 02:20:49.964: INFO: PersistentVolumeClaim pvc-fbpb7 found but phase is Pending instead of Bound. I1105 02:20:52.041] Nov 5 02:20:52.041: INFO: PersistentVolumeClaim pvc-fbpb7 found but phase is Pending instead of Bound. I1105 02:20:54.119] Nov 5 02:20:54.119: INFO: PersistentVolumeClaim pvc-fbpb7 found but phase is Pending instead of Bound. I1105 02:20:56.197] Nov 5 02:20:56.197: INFO: PersistentVolumeClaim pvc-fbpb7 found but phase is Pending instead of Bound. I1105 02:20:58.274] Nov 5 02:20:58.274: INFO: PersistentVolumeClaim pvc-fbpb7 found but phase is Pending instead of Bound. I1105 02:21:00.352] Nov 5 02:21:00.352: INFO: PersistentVolumeClaim pvc-fbpb7 found but phase is Pending instead of Bound. I1105 02:21:02.429] Nov 5 02:21:02.429: INFO: PersistentVolumeClaim pvc-fbpb7 found but phase is Pending instead of Bound. I1105 02:21:04.507] Nov 5 02:21:04.507: INFO: PersistentVolumeClaim pvc-fbpb7 found but phase is Pending instead of Bound. I1105 02:21:06.585] Nov 5 02:21:06.584: INFO: PersistentVolumeClaim pvc-fbpb7 found but phase is Pending instead of Bound. I1105 02:21:08.662] Nov 5 02:21:08.662: INFO: PersistentVolumeClaim pvc-fbpb7 found but phase is Pending instead of Bound. I1105 02:21:10.740] Nov 5 02:21:10.739: INFO: PersistentVolumeClaim pvc-fbpb7 found but phase is Pending instead of Bound. I1105 02:21:12.817] Nov 5 02:21:12.817: INFO: PersistentVolumeClaim pvc-fbpb7 found but phase is Pending instead of Bound. I1105 02:21:14.894] Nov 5 02:21:14.894: INFO: PersistentVolumeClaim pvc-fbpb7 found but phase is Pending instead of Bound. I1105 02:21:16.972] Nov 5 02:21:16.972: INFO: PersistentVolumeClaim pvc-fbpb7 found but phase is Pending instead of Bound. I1105 02:21:19.049] Nov 5 02:21:19.049: INFO: PersistentVolumeClaim pvc-fbpb7 found but phase is Pending instead of Bound. I1105 02:21:21.131] Nov 5 02:21:21.131: INFO: PersistentVolumeClaim pvc-fbpb7 found but phase is Pending instead of Bound. I1105 02:21:23.208] Nov 5 02:21:23.208: INFO: PersistentVolumeClaim pvc-fbpb7 found but phase is Pending instead of Bound. I1105 02:21:25.286] Nov 5 02:21:25.285: INFO: PersistentVolumeClaim pvc-fbpb7 found but phase is Pending instead of Bound. I1105 02:21:27.364] Nov 5 02:21:27.364: INFO: PersistentVolumeClaim pvc-fbpb7 found but phase is Pending instead of Bound. I1105 02:21:29.441] Nov 5 02:21:29.441: INFO: PersistentVolumeClaim pvc-fbpb7 found but phase is Pending instead of Bound. I1105 02:21:31.518] Nov 5 02:21:31.518: INFO: PersistentVolumeClaim pvc-fbpb7 found but phase is Pending instead of Bound. I1105 02:21:33.599] Nov 5 02:21:33.599: INFO: PersistentVolumeClaim pvc-fbpb7 found but phase is Pending instead of Bound. I1105 02:21:35.677] Nov 5 02:21:35.677: INFO: PersistentVolumeClaim pvc-fbpb7 found but phase is Pending instead of Bound. I1105 02:21:37.754] Nov 5 02:21:37.754: INFO: PersistentVolumeClaim pvc-fbpb7 found but phase is Pending instead of Bound. I1105 02:21:39.832] Nov 5 02:21:39.832: INFO: PersistentVolumeClaim pvc-fbpb7 found but phase is Pending instead of Bound. I1105 02:21:41.910] Nov 5 02:21:41.910: INFO: PersistentVolumeClaim pvc-fbpb7 found but phase is Pending instead of Bound. I1105 02:21:43.987] Nov 5 02:21:43.987: INFO: PersistentVolumeClaim pvc-fbpb7 found but phase is Pending instead of Bound. I1105 02:21:46.065] Nov 5 02:21:46.065: INFO: PersistentVolumeClaim pvc-fbpb7 found but phase is Pending instead of Bound. I1105 02:21:48.143] Nov 5 02:21:48.143: INFO: PersistentVolumeClaim pvc-fbpb7 found but phase is Pending instead of Bound. I1105 02:21:50.220] Nov 5 02:21:50.220: INFO: PersistentVolumeClaim pvc-fbpb7 found but phase is Pending instead of Bound. I1105 02:21:52.298] Nov 5 02:21:52.297: INFO: PersistentVolumeClaim pvc-fbpb7 found but phase is Pending instead of Bound. I1105 02:21:54.375] Nov 5 02:21:54.375: INFO: PersistentVolumeClaim pvc-fbpb7 found but phase is Pending instead of Bound. I1105 02:21:56.453] Nov 5 02:21:56.452: INFO: PersistentVolumeClaim pvc-fbpb7 found but phase is Pending instead of Bound. I1105 02:21:58.531] Nov 5 02:21:58.530: INFO: PersistentVolumeClaim pvc-fbpb7 found but phase is Pending instead of Bound. I1105 02:22:00.608] Nov 5 02:22:00.608: INFO: PersistentVolumeClaim pvc-fbpb7 found but phase is Pending instead of Bound. I1105 02:22:02.686] Nov 5 02:22:02.686: INFO: PersistentVolumeClaim pvc-fbpb7 found but phase is Pending instead of Bound. I1105 02:22:04.763] Nov 5 02:22:04.763: INFO: PersistentVolumeClaim pvc-fbpb7 found but phase is Pending instead of Bound. I1105 02:22:06.841] Nov 5 02:22:06.841: INFO: PersistentVolumeClaim pvc-fbpb7 found but phase is Pending instead of Bound. I1105 02:22:08.918] Nov 5 02:22:08.918: INFO: PersistentVolumeClaim pvc-fbpb7 found but phase is Pending instead of Bound. I1105 02:22:10.995] Nov 5 02:22:10.995: INFO: PersistentVolumeClaim pvc-fbpb7 found but phase is Pending instead of Bound. I1105 02:22:13.073] Nov 5 02:22:13.073: INFO: PersistentVolumeClaim pvc-fbpb7 found but phase is Pending instead of Bound. I1105 02:22:15.151] Nov 5 02:22:15.151: INFO: PersistentVolumeClaim pvc-fbpb7 found but phase is Pending instead of Bound. I1105 02:22:17.228] Nov 5 02:22:17.228: INFO: PersistentVolumeClaim pvc-fbpb7 found but phase is Pending instead of Bound. I1105 02:22:19.306] Nov 5 02:22:19.306: INFO: PersistentVolumeClaim pvc-fbpb7 found but phase is Pending instead of Bound. I1105 02:22:21.384] Nov 5 02:22:21.384: INFO: PersistentVolumeClaim pvc-fbpb7 found but phase is Pending instead of Bound. I1105 02:22:23.462] Nov 5 02:22:23.462: INFO: PersistentVolumeClaim pvc-fbpb7 found but phase is Pending instead of Bound. I1105 02:22:25.539] Nov 5 02:22:25.539: INFO: PersistentVolumeClaim pvc-fbpb7 found but phase is Pending instead of Bound. I1105 02:22:27.617] Nov 5 02:22:27.617: INFO: PersistentVolumeClaim pvc-fbpb7 found but phase is Pending instead of Bound. I1105 02:22:29.695] Nov 5 02:22:29.694: INFO: PersistentVolumeClaim pvc-fbpb7 found but phase is Pending instead of Bound. I1105 02:22:31.772] Nov 5 02:22:31.772: INFO: PersistentVolumeClaim pvc-fbpb7 found but phase is Pending instead of Bound. I1105 02:22:33.850] Nov 5 02:22:33.849: INFO: PersistentVolumeClaim pvc-fbpb7 found but phase is Pending instead of Bound. I1105 02:22:35.928] Nov 5 02:22:35.928: INFO: PersistentVolumeClaim pvc-fbpb7 found but phase is Pending instead of Bound. I1105 02:22:38.005] Nov 5 02:22:38.005: INFO: PersistentVolumeClaim pvc-fbpb7 found but phase is Pending instead of Bound. I1105 02:22:40.082] Nov 5 02:22:40.082: INFO: PersistentVolumeClaim pvc-fbpb7 found but phase is Pending instead of Bound. I1105 02:22:42.160] Nov 5 02:22:42.160: INFO: PersistentVolumeClaim pvc-fbpb7 found but phase is Pending instead of Bound. I1105 02:22:44.238] Nov 5 02:22:44.238: INFO: PersistentVolumeClaim pvc-fbpb7 found but phase is Pending instead of Bound. I1105 02:22:46.315] Nov 5 02:22:46.315: INFO: PersistentVolumeClaim pvc-fbpb7 found but phase is Pending instead of Bound. I1105 02:22:48.393] Nov 5 02:22:48.393: INFO: PersistentVolumeClaim pvc-fbpb7 found but phase is Pending instead of Bound. I1105 02:22:50.471] Nov 5 02:22:50.470: INFO: PersistentVolumeClaim pvc-fbpb7 found but phase is Pending instead of Bound. I1105 02:22:52.548] Nov 5 02:22:52.548: INFO: PersistentVolumeClaim pvc-fbpb7 found but phase is Pending instead of Bound. I1105 02:22:54.626] Nov 5 02:22:54.626: INFO: PersistentVolumeClaim pvc-fbpb7 found but phase is Pending instead of Bound. I1105 02:22:56.703] Nov 5 02:22:56.703: INFO: PersistentVolumeClaim pvc-fbpb7 found but phase is Pending instead of Bound. I1105 02:22:58.781] Nov 5 02:22:58.780: INFO: PersistentVolumeClaim pvc-fbpb7 found but phase is Pending instead of Bound. I1105 02:23:00.859] Nov 5 02:23:00.858: INFO: PersistentVolumeClaim pvc-fbpb7 found but phase is Pending instead of Bound. I1105 02:23:02.936] Nov 5 02:23:02.936: INFO: PersistentVolumeClaim pvc-fbpb7 found but phase is Pending instead of Bound. I1105 02:23:05.013] Nov 5 02:23:05.013: INFO: PersistentVolumeClaim pvc-fbpb7 found but phase is Pending instead of Bound. I1105 02:23:07.090] Nov 5 02:23:07.090: INFO: PersistentVolumeClaim pvc-fbpb7 found but phase is Pending instead of Bound. I1105 02:23:09.168] Nov 5 02:23:09.167: INFO: PersistentVolumeClaim pvc-fbpb7 found but phase is Pending instead of Bound. I1105 02:23:11.250] Nov 5 02:23:11.250: INFO: PersistentVolumeClaim pvc-fbpb7 found but phase is Pending instead of Bound. I1105 02:23:13.327] Nov 5 02:23:13.327: INFO: PersistentVolumeClaim pvc-fbpb7 found but phase is Pending instead of Bound. I1105 02:23:15.406] Nov 5 02:23:15.406: INFO: PersistentVolumeClaim pvc-fbpb7 found but phase is Pending instead of Bound. I1105 02:23:17.483] Nov 5 02:23:17.483: INFO: PersistentVolumeClaim pvc-fbpb7 found but phase is Pending instead of Bound. I1105 02:23:19.561] Nov 5 02:23:19.561: INFO: PersistentVolumeClaim pvc-fbpb7 found but phase is Pending instead of Bound. I1105 02:23:21.639] Nov 5 02:23:21.638: INFO: PersistentVolumeClaim pvc-fbpb7 found but phase is Pending instead of Bound. I1105 02:23:23.716] Nov 5 02:23:23.715: INFO: PersistentVolumeClaim pvc-fbpb7 found but phase is Pending instead of Bound. I1105 02:23:25.793] Nov 5 02:23:25.793: INFO: PersistentVolumeClaim pvc-fbpb7 found but phase is Pending instead of Bound. I1105 02:23:27.871] Nov 5 02:23:27.871: INFO: PersistentVolumeClaim pvc-fbpb7 found but phase is Pending instead of Bound. I1105 02:23:29.949] Nov 5 02:23:29.949: INFO: PersistentVolumeClaim pvc-fbpb7 found but phase is Pending instead of Bound. I1105 02:23:31.949] Nov 5 02:23:31.949: INFO: Unexpected error occurred: PersistentVolumeClaims [pvc-fbpb7] not all in phase Bound within 5m0s I1105 02:23:31.950] [AfterEach] [single-az] I1105 02:23:31.950] /go/src/sigs.k8s.io/azuredisk-csi-driver/vendor/k8s.io/kubernetes/test/e2e/framework/framework.go:151 I1105 02:23:31.950] STEP: Collecting events from namespace "azuredisk-1124". I1105 02:23:32.027] STEP: Found 3 events. I1105 02:23:32.027] Nov 5 02:23:32.027: INFO: At 2019-11-05 02:18:30 +0000 UTC - event for pvc-fbpb7: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "disk.csi.azure.com" or manually created by system administrator I1105 02:23:32.028] Nov 5 02:23:32.027: INFO: At 2019-11-05 02:18:30 +0000 UTC - event for pvc-fbpb7: {disk.csi.azure.com_csi-azuredisk-controller-6f6587bfc6-8vkfl_1376914b-ff6d-11e9-93ab-ba2bee589b8c } Provisioning: External provisioner is provisioning volume for claim "azuredisk-1124/pvc-fbpb7" I1105 02:23:32.028] Nov 5 02:23:32.027: INFO: At 2019-11-05 02:18:30 +0000 UTC - event for pvc-fbpb7: {disk.csi.azure.com_csi-azuredisk-controller-6f6587bfc6-8vkfl_1376914b-ff6d-11e9-93ab-ba2bee589b8c } ProvisioningFailed: failed to provision volume with StorageClass "azuredisk-1124-disk.csi.azure.com-dynamic-sc-f522k": rpc error: code = Unknown desc = compute.DisksClient#CreateOrUpdate: Failure sending request: StatusCode=400 -- Original Error: Code="InvalidAvailabilityZone" Message="The zone(s) '' for resource 'Microsoft.Compute/disks/pvc-b594f272-6962-4f9c-b843-41e5759c4632' is not supported. The supported zones for location 'eastus2' are '1,2,3'" I1105 02:23:32.104] Nov 5 02:23:32.104: INFO: POD NODE PHASE GRACE CONDITIONS I1105 02:23:32.104] Nov 5 02:23:32.104: INFO: I1105 02:23:32.334] Nov 5 02:23:32.334: INFO: I1105 02:23:32.334] Logging node info for node k8s-agentpool-15826033-vmss000000 I1105 02:23:32.413] Nov 5 02:23:32.410: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-agentpool-15826033-vmss000000,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-agentpool-15826033-vmss000000,UID:15fa874e-54ec-42cd-b243-d3baa512150f,ResourceVersion:9812,Generation:0,CreationTimestamp:2019-11-05 01:34:17 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{agentpool: agentpool,beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_D4s_v3,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 0,kubernetes.azure.com/cluster: kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5,kubernetes.azure.com/role: agent,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-agentpool-15826033-vmss000000,kubernetes.io/os: linux,kubernetes.io/role: agent,node-role.kubernetes.io/agent: ,storageprofile: managed,storagetier: Premium_LRS,topology.disk.csi.azure.com/zone: 0,},Annotations:map[string]string{csi.volume.kubernetes.io/nodeid: {"disk.csi.azure.com":"k8s-agentpool-15826033-vmss000000"},node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.2.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool-15826033-vmss/virtualMachines/0,Unschedulable:false,Taints:[],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-csi-disk.csi.azure.com: {{8 0} {} 8 DecimalSI},cpu: {{4 0} {} 4 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{16797044736 0} {} 16403364Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-csi-disk.csi.azure.com: {{8 0} {} 8 DecimalSI},cpu: {{4 0} {} 4 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{16010612736 0} {} 15635364Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 01:34:56 +0000 UTC 2019-11-05 01:34:56 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 02:23:25 +0000 UTC 2019-11-05 01:34:17 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 02:23:25 +0000 UTC 2019-11-05 01:34:17 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 02:23:25 +0000 UTC 2019-11-05 01:34:17 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 02:23:25 +0000 UTC 2019-11-05 01:34:29 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.0.4} {Hostname k8s-agentpool-15826033-vmss000000}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:afe00792177d49a98f6a44d03d08732d,SystemUUID:4F1777BF-A5C3-004E-81C4-90F0B54C02A0,BootID:13b6a628-ae89-44af-b891-89461030adcb,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[k8sprowinternal.azurecr.io/azuredisk-csi@sha256:75d4ff5d7cd845d6e810c67796ecb037e852fb849ec1eb5b3194564008e6d1e0 k8sprowinternal.azurecr.io/azuredisk-csi:e2e-3e3ed045c6faa509dbd8d2890ee9f56aa2934887] 286930414} {[k8s.gcr.io/kubernetes-dashboard-amd64@sha256:0ae6b69432e78069c5ce2bcde0fe409c5c4d6f0f4d9cd50a17974fea38898747 k8s.gcr.io/kubernetes-dashboard-amd64:v1.10.1] 121711221} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/metrics-server-amd64@sha256:49a9f12f7067d11f42c803dbe61ed2c1299959ad85cb315b25ff7eef8e6b8892 k8s.gcr.io/metrics-server-amd64:v0.2.1] 42541759} {[mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume@sha256:4fd30d43947d4a54fc89ead7985beecfd3c9b2a93a0655a373b1608ab90bd5af mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume:v0.0.7] 22909487} {[quay.io/k8scsi/livenessprobe@sha256:8c9c8a0f330acc2e558ae6624309447d02522e76aa35af23fa19b88abf4381cb quay.io/k8scsi/livenessprobe:v1.0.2] 17415381} {[quay.io/k8scsi/csi-node-driver-registrar@sha256:13daf82fb99e951a4bff8ae5fc7c17c3a8fe7130be6400990d8f6076c32d4599 quay.io/k8scsi/csi-node-driver-registrar:v1.1.0] 15815995} {[mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume@sha256:23d8c6033f02a1ecad05127ebdc931bb871264228661bc122704b0974e4d9fdd mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume:1.0.8] 1159025} {[busybox@sha256:8ccbac733d19c0dd4d70b4f0c1e12245b5fa3ad24758a11035ee505c629c0796 busybox:1.29] 1154361} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 02:23:32.414] Nov 5 02:23:32.411: INFO: I1105 02:23:32.414] Logging kubelet events for node k8s-agentpool-15826033-vmss000000 I1105 02:23:32.491] Nov 5 02:23:32.491: INFO: I1105 02:23:32.491] Logging pods the kubelet thinks is on node k8s-agentpool-15826033-vmss000000 I1105 02:23:32.574] Nov 5 02:23:32.573: INFO: blobfuse-flexvol-installer-bsvqt started at 2019-11-05 01:34:34 +0000 UTC (0+1 container statuses recorded) I1105 02:23:32.574] Nov 5 02:23:32.574: INFO: Container blobfuse-flexvol-installer ready: true, restart count 0 I1105 02:23:32.574] Nov 5 02:23:32.574: INFO: azure-ip-masq-agent-vx86b started at 2019-11-05 01:34:34 +0000 UTC (0+1 container statuses recorded) I1105 02:23:32.574] Nov 5 02:23:32.574: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 02:23:32.575] Nov 5 02:23:32.574: INFO: keyvault-flexvolume-qhnfr started at 2019-11-05 01:34:34 +0000 UTC (0+1 container statuses recorded) I1105 02:23:32.575] Nov 5 02:23:32.574: INFO: Container keyvault-flexvolume ready: true, restart count 0 I1105 02:23:32.575] Nov 5 02:23:32.574: INFO: metrics-server-864ffbc5c-sdxps started at 2019-11-05 01:34:34 +0000 UTC (0+1 container statuses recorded) I1105 02:23:32.575] Nov 5 02:23:32.574: INFO: Container metrics-server ready: true, restart count 0 I1105 02:23:32.575] Nov 5 02:23:32.574: INFO: kube-proxy-wxblc started at 2019-11-05 01:34:34 +0000 UTC (0+1 container statuses recorded) I1105 02:23:32.575] Nov 5 02:23:32.574: INFO: Container kube-proxy ready: true, restart count 0 I1105 02:23:32.576] Nov 5 02:23:32.574: INFO: kubernetes-dashboard-66dd8b8df7-ks284 started at 2019-11-05 01:34:35 +0000 UTC (0+1 container statuses recorded) I1105 02:23:32.576] Nov 5 02:23:32.574: INFO: Container kubernetes-dashboard ready: true, restart count 0 I1105 02:23:32.576] Nov 5 02:23:32.574: INFO: csi-azuredisk-node-whq2g started at 2019-11-05 01:38:24 +0000 UTC (0+3 container statuses recorded) I1105 02:23:32.576] Nov 5 02:23:32.574: INFO: Container azuredisk ready: true, restart count 0 I1105 02:23:32.576] Nov 5 02:23:32.574: INFO: Container liveness-probe ready: true, restart count 0 I1105 02:23:32.576] Nov 5 02:23:32.574: INFO: Container node-driver-registrar ready: true, restart count 0 I1105 02:23:32.831] Nov 5 02:23:32.831: INFO: I1105 02:23:32.832] Latency metrics for node k8s-agentpool-15826033-vmss000000 I1105 02:23:32.832] Nov 5 02:23:32.831: INFO: I1105 02:23:32.832] Logging node info for node k8s-agentpool-15826033-vmss000001 I1105 02:23:32.912] Nov 5 02:23:32.909: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-agentpool-15826033-vmss000001,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-agentpool-15826033-vmss000001,UID:4a8ea605-cc0b-4e3b-9060-d8867d269e6c,ResourceVersion:9703,Generation:0,CreationTimestamp:2019-11-05 01:34:09 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{agentpool: agentpool,beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_D4s_v3,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 1,kubernetes.azure.com/cluster: kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5,kubernetes.azure.com/role: agent,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-agentpool-15826033-vmss000001,kubernetes.io/os: linux,kubernetes.io/role: agent,node-role.kubernetes.io/agent: ,storageprofile: managed,storagetier: Premium_LRS,topology.disk.csi.azure.com/zone: 1,},Annotations:map[string]string{csi.volume.kubernetes.io/nodeid: {"disk.csi.azure.com":"k8s-agentpool-15826033-vmss000001"},node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.0.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool-15826033-vmss/virtualMachines/1,Unschedulable:false,Taints:[],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-csi-disk.csi.azure.com: {{8 0} {} 8 DecimalSI},cpu: {{4 0} {} 4 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{16797044736 0} {} 16403364Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-csi-disk.csi.azure.com: {{8 0} {} 8 DecimalSI},cpu: {{4 0} {} 4 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{16010612736 0} {} 15635364Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 01:34:57 +0000 UTC 2019-11-05 01:34:57 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 02:22:47 +0000 UTC 2019-11-05 01:34:08 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 02:22:47 +0000 UTC 2019-11-05 01:34:08 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 02:22:47 +0000 UTC 2019-11-05 01:34:08 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 02:22:47 +0000 UTC 2019-11-05 01:34:29 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.0.5} {Hostname k8s-agentpool-15826033-vmss000001}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:b6b9fbf4a2fd4fa5a7ae472415fe30b8,SystemUUID:0162B3A4-3955-9A46-ABD4-A37BD75848CC,BootID:ee93003f-d47c-4118-9975-57365b3443a5,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[k8sprowinternal.azurecr.io/azuredisk-csi@sha256:75d4ff5d7cd845d6e810c67796ecb037e852fb849ec1eb5b3194564008e6d1e0 k8sprowinternal.azurecr.io/azuredisk-csi:e2e-3e3ed045c6faa509dbd8d2890ee9f56aa2934887] 286930414} {[gcr.io/kubernetes-helm/tiller@sha256:f6d8f4ab9ba993b5f5b60a6edafe86352eabe474ffeb84cb6c79b8866dce45d1 gcr.io/kubernetes-helm/tiller:v2.11.0] 71821984} {[quay.io/k8scsi/csi-provisioner@sha256:d657c839dce87324fe2b677302913f9386f885f8746be7bea0ced5b0844e3433 quay.io/k8scsi/csi-provisioner:v1.3.1] 50596998} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[quay.io/k8scsi/csi-attacher@sha256:26fccd7a99d973845df1193b46ebdcc6ab8dc5f6e6be319750c471fce1742d13 quay.io/k8scsi/csi-attacher:v1.2.0] 46226754} {[quay.io/k8scsi/csi-cluster-driver-registrar@sha256:fafd75ae5442f192cfa8c2e792903aee30d5884b62e802e4464b0a895d21e3ef quay.io/k8scsi/csi-cluster-driver-registrar:v1.0.1] 45874691} {[quay.io/k8scsi/csi-snapshotter@sha256:a49e0da1af6f2bf717e41ba1eee8b5e6a1cbd66a709dd92cc43fe475fe2589eb quay.io/k8scsi/csi-snapshotter:v1.1.0] 44325352} {[k8s.gcr.io/coredns@sha256:e83beb5e43f8513fa735e77ffc5859640baea30a882a11cc75c4c3244a737d3c k8s.gcr.io/coredns:1.5.0] 42488424} {[mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume@sha256:4fd30d43947d4a54fc89ead7985beecfd3c9b2a93a0655a373b1608ab90bd5af mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume:v0.0.7] 22909487} {[quay.io/k8scsi/livenessprobe@sha256:8c9c8a0f330acc2e558ae6624309447d02522e76aa35af23fa19b88abf4381cb quay.io/k8scsi/livenessprobe:v1.0.2] 17415381} {[quay.io/k8scsi/csi-node-driver-registrar@sha256:13daf82fb99e951a4bff8ae5fc7c17c3a8fe7130be6400990d8f6076c32d4599 quay.io/k8scsi/csi-node-driver-registrar:v1.1.0] 15815995} {[quay.io/k8scsi/livenessprobe@sha256:dde617756e0f602adc566ab71fd885f1dad451ad3fb063ac991c95a2ff47aea5 quay.io/k8scsi/livenessprobe:v1.1.0] 14967303} {[mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume@sha256:23d8c6033f02a1ecad05127ebdc931bb871264228661bc122704b0974e4d9fdd mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume:1.0.8] 1159025} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 02:23:32.912] Nov 5 02:23:32.909: INFO: I1105 02:23:32.912] Logging kubelet events for node k8s-agentpool-15826033-vmss000001 I1105 02:23:32.990] Nov 5 02:23:32.990: INFO: I1105 02:23:32.990] Logging pods the kubelet thinks is on node k8s-agentpool-15826033-vmss000001 I1105 02:23:33.072] Nov 5 02:23:33.072: INFO: coredns-7f68dcdbdb-jt522 started at 2019-11-05 01:34:35 +0000 UTC (0+1 container statuses recorded) I1105 02:23:33.073] Nov 5 02:23:33.072: INFO: Container coredns ready: true, restart count 0 I1105 02:23:33.073] Nov 5 02:23:33.072: INFO: azure-ip-masq-agent-6c6vn started at 2019-11-05 01:34:35 +0000 UTC (0+1 container statuses recorded) I1105 02:23:33.073] Nov 5 02:23:33.072: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 02:23:33.073] Nov 5 02:23:33.072: INFO: keyvault-flexvolume-rdlcj started at 2019-11-05 01:34:35 +0000 UTC (0+1 container statuses recorded) I1105 02:23:33.073] Nov 5 02:23:33.072: INFO: Container keyvault-flexvolume ready: true, restart count 0 I1105 02:23:33.073] Nov 5 02:23:33.072: INFO: kube-proxy-6kh57 started at 2019-11-05 01:34:35 +0000 UTC (0+1 container statuses recorded) I1105 02:23:33.073] Nov 5 02:23:33.072: INFO: Container kube-proxy ready: true, restart count 0 I1105 02:23:33.074] Nov 5 02:23:33.072: INFO: tiller-deploy-54c96cb5df-8d9fs started at 2019-11-05 01:34:35 +0000 UTC (0+1 container statuses recorded) I1105 02:23:33.074] Nov 5 02:23:33.072: INFO: Container tiller ready: true, restart count 0 I1105 02:23:33.074] Nov 5 02:23:33.072: INFO: csi-azuredisk-node-ghv5w started at 2019-11-05 01:38:24 +0000 UTC (0+3 container statuses recorded) I1105 02:23:33.074] Nov 5 02:23:33.072: INFO: Container azuredisk ready: true, restart count 0 I1105 02:23:33.074] Nov 5 02:23:33.072: INFO: Container liveness-probe ready: true, restart count 0 I1105 02:23:33.074] Nov 5 02:23:33.072: INFO: Container node-driver-registrar ready: true, restart count 0 I1105 02:23:33.075] Nov 5 02:23:33.072: INFO: csi-azuredisk-controller-6f6587bfc6-8vkfl started at 2019-11-05 01:38:24 +0000 UTC (0+6 container statuses recorded) I1105 02:23:33.075] Nov 5 02:23:33.072: INFO: Container azuredisk ready: true, restart count 0 I1105 02:23:33.075] Nov 5 02:23:33.072: INFO: Container cluster-driver-registrar ready: true, restart count 0 I1105 02:23:33.075] Nov 5 02:23:33.072: INFO: Container csi-attacher ready: true, restart count 0 I1105 02:23:33.075] Nov 5 02:23:33.072: INFO: Container csi-provisioner ready: true, restart count 0 I1105 02:23:33.075] Nov 5 02:23:33.072: INFO: Container csi-snapshotter ready: true, restart count 0 I1105 02:23:33.075] Nov 5 02:23:33.072: INFO: Container liveness-probe ready: true, restart count 0 I1105 02:23:33.075] Nov 5 02:23:33.072: INFO: blobfuse-flexvol-installer-rw5dw started at 2019-11-05 01:34:35 +0000 UTC (0+1 container statuses recorded) I1105 02:23:33.076] Nov 5 02:23:33.072: INFO: Container blobfuse-flexvol-installer ready: true, restart count 0 I1105 02:23:33.323] Nov 5 02:23:33.322: INFO: I1105 02:23:33.323] Latency metrics for node k8s-agentpool-15826033-vmss000001 I1105 02:23:33.323] Nov 5 02:23:33.322: INFO: I1105 02:23:33.323] Logging node info for node k8s-master-15826033-0 I1105 02:23:33.402] Nov 5 02:23:33.400: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-15826033-0,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-15826033-0,UID:aceb0231-4645-4a9a-b296-90ada851986b,ResourceVersion:9663,Generation:0,CreationTimestamp:2019-11-05 01:35:06 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 1,kubernetes.azure.com/cluster: kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-15826033-0,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.4.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5/providers/Microsoft.Compute/virtualMachines/k8s-master-15826033-0,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 01:35:20 +0000 UTC 2019-11-05 01:35:20 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 02:22:33 +0000 UTC 2019-11-05 01:34:57 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 02:22:33 +0000 UTC 2019-11-05 01:34:57 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 02:22:33 +0000 UTC 2019-11-05 01:34:57 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 02:22:33 +0000 UTC 2019-11-05 01:35:07 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.255.5} {Hostname k8s-master-15826033-0}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:1915756c22a540dc8eb30493987e9a43,SystemUUID:742C969C-5A71-D744-8E61-F27E38B56EB2,BootID:03f0672e-8fa9-4309-956f-8e3c371902c3,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 02:23:33.402] Nov 5 02:23:33.400: INFO: I1105 02:23:33.403] Logging kubelet events for node k8s-master-15826033-0 I1105 02:23:33.482] Nov 5 02:23:33.482: INFO: I1105 02:23:33.482] Logging pods the kubelet thinks is on node k8s-master-15826033-0 I1105 02:23:33.564] Nov 5 02:23:33.564: INFO: kube-proxy-7d2px started at 2019-11-05 01:35:06 +0000 UTC (0+1 container statuses recorded) I1105 02:23:33.565] Nov 5 02:23:33.564: INFO: Container kube-proxy ready: true, restart count 0 I1105 02:23:33.565] Nov 5 02:23:33.564: INFO: cloud-controller-manager-k8s-master-15826033-0 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:23:33.565] Nov 5 02:23:33.564: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 02:23:33.565] Nov 5 02:23:33.564: INFO: kube-addon-manager-k8s-master-15826033-0 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:23:33.565] Nov 5 02:23:33.564: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 02:23:33.565] Nov 5 02:23:33.564: INFO: kube-apiserver-k8s-master-15826033-0 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:23:33.566] Nov 5 02:23:33.564: INFO: Container kube-apiserver ready: true, restart count 0 I1105 02:23:33.566] Nov 5 02:23:33.564: INFO: kube-controller-manager-k8s-master-15826033-0 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:23:33.566] Nov 5 02:23:33.564: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 02:23:33.566] Nov 5 02:23:33.564: INFO: kube-scheduler-k8s-master-15826033-0 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:23:33.566] Nov 5 02:23:33.564: INFO: Container kube-scheduler ready: true, restart count 0 I1105 02:23:33.566] Nov 5 02:23:33.564: INFO: azure-ip-masq-agent-wgdpx started at 2019-11-05 01:35:06 +0000 UTC (0+1 container statuses recorded) I1105 02:23:33.567] Nov 5 02:23:33.564: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 02:23:33.812] Nov 5 02:23:33.812: INFO: I1105 02:23:33.813] Latency metrics for node k8s-master-15826033-0 I1105 02:23:33.813] Nov 5 02:23:33.812: INFO: I1105 02:23:33.813] Logging node info for node k8s-master-15826033-1 I1105 02:23:33.908] Nov 5 02:23:33.906: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-15826033-1,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-15826033-1,UID:1a13ec61-c89a-456c-8c94-bfc8e717c2e7,ResourceVersion:9665,Generation:0,CreationTimestamp:2019-11-05 01:35:07 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 1,kubernetes.azure.com/cluster: kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-15826033-1,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.5.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5/providers/Microsoft.Compute/virtualMachines/k8s-master-15826033-1,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 01:35:31 +0000 UTC 2019-11-05 01:35:31 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 02:22:33 +0000 UTC 2019-11-05 01:35:00 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 02:22:33 +0000 UTC 2019-11-05 01:35:00 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 02:22:33 +0000 UTC 2019-11-05 01:35:00 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 02:22:33 +0000 UTC 2019-11-05 01:35:07 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.255.6} {Hostname k8s-master-15826033-1}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:e0f2bccddd5b4555b180d14081ab59a8,SystemUUID:AE126631-CAD2-B54F-B22E-1F499342A4D6,BootID:509b9a9d-b046-4861-aa35-8bcb77465de2,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 02:23:33.908] Nov 5 02:23:33.906: INFO: I1105 02:23:33.908] Logging kubelet events for node k8s-master-15826033-1 I1105 02:23:33.988] Nov 5 02:23:33.987: INFO: I1105 02:23:33.988] Logging pods the kubelet thinks is on node k8s-master-15826033-1 I1105 02:23:34.068] Nov 5 02:23:34.068: INFO: kube-controller-manager-k8s-master-15826033-1 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:23:34.069] Nov 5 02:23:34.068: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 02:23:34.069] Nov 5 02:23:34.068: INFO: kube-scheduler-k8s-master-15826033-1 started at 2019-11-05 01:36:16 +0000 UTC (0+1 container statuses recorded) I1105 02:23:34.069] Nov 5 02:23:34.068: INFO: Container kube-scheduler ready: true, restart count 0 I1105 02:23:34.069] Nov 5 02:23:34.068: INFO: azure-ip-masq-agent-w6dpr started at 2019-11-05 01:35:10 +0000 UTC (0+1 container statuses recorded) I1105 02:23:34.069] Nov 5 02:23:34.068: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 02:23:34.069] Nov 5 02:23:34.068: INFO: kube-proxy-8kmr5 started at 2019-11-05 01:35:10 +0000 UTC (0+1 container statuses recorded) I1105 02:23:34.069] Nov 5 02:23:34.068: INFO: Container kube-proxy ready: true, restart count 0 I1105 02:23:34.070] Nov 5 02:23:34.068: INFO: cloud-controller-manager-k8s-master-15826033-1 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:23:34.070] Nov 5 02:23:34.068: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 02:23:34.070] Nov 5 02:23:34.068: INFO: kube-addon-manager-k8s-master-15826033-1 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:23:34.070] Nov 5 02:23:34.068: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 02:23:34.070] Nov 5 02:23:34.068: INFO: kube-apiserver-k8s-master-15826033-1 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:23:34.070] Nov 5 02:23:34.068: INFO: Container kube-apiserver ready: true, restart count 0 I1105 02:23:34.314] Nov 5 02:23:34.314: INFO: I1105 02:23:34.314] Latency metrics for node k8s-master-15826033-1 I1105 02:23:34.314] Nov 5 02:23:34.314: INFO: I1105 02:23:34.314] Logging node info for node k8s-master-15826033-2 I1105 02:23:34.393] Nov 5 02:23:34.391: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-15826033-2,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-15826033-2,UID:acb3a851-d6df-4549-ba56-4b7e96553c17,ResourceVersion:9756,Generation:0,CreationTimestamp:2019-11-05 01:34:09 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 2,kubernetes.azure.com/cluster: kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-15826033-2,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.1.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5/providers/Microsoft.Compute/virtualMachines/k8s-master-15826033-2,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 01:34:44 +0000 UTC 2019-11-05 01:34:44 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 02:23:05 +0000 UTC 2019-11-05 01:34:07 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 02:23:05 +0000 UTC 2019-11-05 01:34:07 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 02:23:05 +0000 UTC 2019-11-05 01:34:07 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 02:23:05 +0000 UTC 2019-11-05 01:34:29 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{Hostname k8s-master-15826033-2} {InternalIP 10.240.255.7}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:0885b0c2f6174106bde4fa0eae2cbc6b,SystemUUID:C176121C-5D7A-6243-9B05-8B68F8022B64,BootID:30430638-7ab6-4761-b984-2387e1a3f5b2,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 02:23:34.393] Nov 5 02:23:34.391: INFO: I1105 02:23:34.393] Logging kubelet events for node k8s-master-15826033-2 I1105 02:23:34.472] Nov 5 02:23:34.472: INFO: I1105 02:23:34.472] Logging pods the kubelet thinks is on node k8s-master-15826033-2 I1105 02:23:34.567] Nov 5 02:23:34.567: INFO: cloud-controller-manager-k8s-master-15826033-2 started at 2019-11-05 01:33:54 +0000 UTC (0+1 container statuses recorded) I1105 02:23:34.567] Nov 5 02:23:34.567: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 02:23:34.568] Nov 5 02:23:34.567: INFO: kube-addon-manager-k8s-master-15826033-2 started at 2019-11-05 01:33:54 +0000 UTC (0+1 container statuses recorded) I1105 02:23:34.568] Nov 5 02:23:34.567: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 02:23:34.568] Nov 5 02:23:34.567: INFO: kube-apiserver-k8s-master-15826033-2 started at 2019-11-05 01:33:54 +0000 UTC (0+1 container statuses recorded) I1105 02:23:34.568] Nov 5 02:23:34.567: INFO: Container kube-apiserver ready: true, restart count 0 I1105 02:23:34.568] Nov 5 02:23:34.567: INFO: kube-controller-manager-k8s-master-15826033-2 started at 2019-11-05 01:33:54 +0000 UTC (0+1 container statuses recorded) I1105 02:23:34.568] Nov 5 02:23:34.567: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 02:23:34.569] Nov 5 02:23:34.567: INFO: kube-scheduler-k8s-master-15826033-2 started at 2019-11-05 01:33:54 +0000 UTC (0+1 container statuses recorded) I1105 02:23:34.569] Nov 5 02:23:34.567: INFO: Container kube-scheduler ready: true, restart count 0 I1105 02:23:34.569] Nov 5 02:23:34.567: INFO: azure-ip-masq-agent-hw7s5 started at 2019-11-05 01:34:34 +0000 UTC (0+1 container statuses recorded) I1105 02:23:34.569] Nov 5 02:23:34.567: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 02:23:34.569] Nov 5 02:23:34.567: INFO: kube-proxy-5xpp6 started at 2019-11-05 01:34:34 +0000 UTC (0+1 container statuses recorded) I1105 02:23:34.570] Nov 5 02:23:34.567: INFO: Container kube-proxy ready: true, restart count 0 I1105 02:23:34.824] Nov 5 02:23:34.824: INFO: I1105 02:23:34.825] Latency metrics for node k8s-master-15826033-2 I1105 02:23:34.825] Nov 5 02:23:34.824: INFO: I1105 02:23:34.825] Logging node info for node k8s-master-15826033-3 I1105 02:23:34.904] Nov 5 02:23:34.902: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-15826033-3,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-15826033-3,UID:b7702272-1876-434e-b6a0-17fa34c6757f,ResourceVersion:9836,Generation:0,CreationTimestamp:2019-11-05 01:35:06 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 0,kubernetes.azure.com/cluster: kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-15826033-3,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.3.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5/providers/Microsoft.Compute/virtualMachines/k8s-master-15826033-3,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 01:35:17 +0000 UTC 2019-11-05 01:35:17 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 02:23:32 +0000 UTC 2019-11-05 01:35:00 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 02:23:32 +0000 UTC 2019-11-05 01:35:00 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 02:23:32 +0000 UTC 2019-11-05 01:35:00 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 02:23:32 +0000 UTC 2019-11-05 01:35:06 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.255.8} {Hostname k8s-master-15826033-3}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:fbf366963961422c9d90bedf6639c8e1,SystemUUID:5388948D-1A8D-EA41-B8B8-7F4523EDF5AA,BootID:42f5884c-7418-486a-8cf3-75188a331f38,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 02:23:34.905] Nov 5 02:23:34.902: INFO: I1105 02:23:34.905] Logging kubelet events for node k8s-master-15826033-3 I1105 02:23:34.983] Nov 5 02:23:34.983: INFO: I1105 02:23:34.984] Logging pods the kubelet thinks is on node k8s-master-15826033-3 I1105 02:23:35.066] Nov 5 02:23:35.065: INFO: kube-controller-manager-k8s-master-15826033-3 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:23:35.066] Nov 5 02:23:35.065: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 02:23:35.066] Nov 5 02:23:35.065: INFO: kube-scheduler-k8s-master-15826033-3 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:23:35.066] Nov 5 02:23:35.065: INFO: Container kube-scheduler ready: true, restart count 0 I1105 02:23:35.066] Nov 5 02:23:35.065: INFO: kube-proxy-xz24z started at 2019-11-05 01:35:06 +0000 UTC (0+1 container statuses recorded) I1105 02:23:35.067] Nov 5 02:23:35.065: INFO: Container kube-proxy ready: true, restart count 0 I1105 02:23:35.067] Nov 5 02:23:35.065: INFO: azure-ip-masq-agent-dqfpw started at 2019-11-05 01:35:06 +0000 UTC (0+1 container statuses recorded) I1105 02:23:35.067] Nov 5 02:23:35.065: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 02:23:35.067] Nov 5 02:23:35.065: INFO: cloud-controller-manager-k8s-master-15826033-3 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:23:35.067] Nov 5 02:23:35.065: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 02:23:35.067] Nov 5 02:23:35.065: INFO: kube-addon-manager-k8s-master-15826033-3 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:23:35.068] Nov 5 02:23:35.065: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 02:23:35.068] Nov 5 02:23:35.065: INFO: kube-apiserver-k8s-master-15826033-3 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:23:35.068] Nov 5 02:23:35.065: INFO: Container kube-apiserver ready: true, restart count 0 I1105 02:23:35.318] Nov 5 02:23:35.317: INFO: I1105 02:23:35.318] Latency metrics for node k8s-master-15826033-3 I1105 02:23:35.318] Nov 5 02:23:35.317: INFO: I1105 02:23:35.318] Logging node info for node k8s-master-15826033-4 I1105 02:23:35.397] Nov 5 02:23:35.394: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-15826033-4,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-15826033-4,UID:276daef1-6c05-41e3-bacd-abc93d35cc6d,ResourceVersion:9842,Generation:0,CreationTimestamp:2019-11-05 01:35:07 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 0,kubernetes.azure.com/cluster: kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-15826033-4,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.6.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5/providers/Microsoft.Compute/virtualMachines/k8s-master-15826033-4,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 01:35:41 +0000 UTC 2019-11-05 01:35:41 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 02:23:34 +0000 UTC 2019-11-05 01:35:02 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 02:23:34 +0000 UTC 2019-11-05 01:35:02 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 02:23:34 +0000 UTC 2019-11-05 01:35:02 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 02:23:34 +0000 UTC 2019-11-05 01:35:08 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.255.9} {Hostname k8s-master-15826033-4}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:79051fc0bdeb4e32a56acbaf5972ce3a,SystemUUID:C71606B5-7F08-9341-A26D-8A8D9D84FCAE,BootID:4bdf7cf3-2b83-42c7-a7ba-926ebe9ca75b,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 02:23:35.397] Nov 5 02:23:35.394: INFO: I1105 02:23:35.397] Logging kubelet events for node k8s-master-15826033-4 I1105 02:23:35.476] Nov 5 02:23:35.475: INFO: I1105 02:23:35.476] Logging pods the kubelet thinks is on node k8s-master-15826033-4 I1105 02:23:35.557] Nov 5 02:23:35.557: INFO: cloud-controller-manager-k8s-master-15826033-4 started at 2019-11-05 01:34:56 +0000 UTC (0+1 container statuses recorded) I1105 02:23:35.558] Nov 5 02:23:35.557: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 02:23:35.558] Nov 5 02:23:35.557: INFO: kube-addon-manager-k8s-master-15826033-4 started at 2019-11-05 01:34:56 +0000 UTC (0+1 container statuses recorded) I1105 02:23:35.558] Nov 5 02:23:35.557: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 02:23:35.558] Nov 5 02:23:35.557: INFO: azure-ip-masq-agent-9nrj8 started at 2019-11-05 01:35:11 +0000 UTC (0+1 container statuses recorded) I1105 02:23:35.558] Nov 5 02:23:35.557: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 02:23:35.558] Nov 5 02:23:35.557: INFO: kube-proxy-rmngk started at 2019-11-05 01:35:11 +0000 UTC (0+1 container statuses recorded) I1105 02:23:35.559] Nov 5 02:23:35.557: INFO: Container kube-proxy ready: true, restart count 0 I1105 02:23:35.559] Nov 5 02:23:35.557: INFO: kube-apiserver-k8s-master-15826033-4 started at 2019-11-05 01:34:55 +0000 UTC (0+1 container statuses recorded) I1105 02:23:35.559] Nov 5 02:23:35.557: INFO: Container kube-apiserver ready: true, restart count 0 I1105 02:23:35.559] Nov 5 02:23:35.557: INFO: kube-controller-manager-k8s-master-15826033-4 started at 2019-11-05 01:34:55 +0000 UTC (0+1 container statuses recorded) I1105 02:23:35.559] Nov 5 02:23:35.557: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 02:23:35.559] Nov 5 02:23:35.557: INFO: kube-scheduler-k8s-master-15826033-4 started at 2019-11-05 01:34:55 +0000 UTC (0+1 container statuses recorded) I1105 02:23:35.560] Nov 5 02:23:35.557: INFO: Container kube-scheduler ready: true, restart count 0 I1105 02:23:35.807] Nov 5 02:23:35.807: INFO: I1105 02:23:35.807] Latency metrics for node k8s-master-15826033-4 I1105 02:23:35.807] Nov 5 02:23:35.807: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready I1105 02:23:35.885] STEP: Destroying namespace "azuredisk-1124" for this suite. I1105 02:23:50.128] Nov 5 02:23:50.128: INFO: Waiting up to 30s for server preferred namespaced resources to be successfully discovered I1105 02:23:53.112] Nov 5 02:23:53.112: INFO: namespace azuredisk-1124 deletion completed in 17.227183895s I1105 02:23:53.112] I1105 02:23:53.112] • Failure [323.187 seconds] I1105 02:23:53.113] [azuredisk-csi-e2e] Dynamic Provisioning I1105 02:23:53.113] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:33 I1105 02:23:53.113] [single-az] I1105 02:23:53.113] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:36 I1105 02:23:53.113] should create a deployment object, write and read to it, delete the pod and write and read to it again [It] I1105 02:23:53.113] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:193 I1105 02:23:53.113] I1105 02:23:53.114] Unexpected error: I1105 02:23:53.114] <*errors.errorString | 0xc000936d50>: { I1105 02:23:53.114] s: "PersistentVolumeClaims [pvc-fbpb7] not all in phase Bound within 5m0s", I1105 02:23:53.114] } I1105 02:23:53.114] PersistentVolumeClaims [pvc-fbpb7] not all in phase Bound within 5m0s I1105 02:23:53.114] occurred I1105 02:23:53.114] I1105 02:23:53.115] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/testsuites/testsuites.go:278 I1105 02:23:53.115] ------------------------------ I1105 02:23:53.115] [azuredisk-csi-e2e] Dynamic Provisioning [multi-az] I1105 02:23:53.115] [env] should retain PV with reclaimPolicy "Retain" I1105 02:23:53.115] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:240 I1105 02:23:53.115] [BeforeEach] [multi-az] I1105 02:23:53.115] /go/src/sigs.k8s.io/azuredisk-csi-driver/vendor/k8s.io/kubernetes/test/e2e/framework/framework.go:150 I1105 02:23:53.115] STEP: Creating a kubernetes client I1105 02:23:53.115] Nov 5 02:23:53.112: INFO: >>> kubeConfig: /workspace/aks978705698/kubeconfig/kubeconfig.eastus2.json I1105 02:23:53.116] STEP: Building a namespace api object, basename azuredisk I1105 02:23:53.351] STEP: Binding the e2e-test-privileged-psp PodSecurityPolicy to the default service account in azuredisk-7960 I1105 02:23:53.615] STEP: Waiting for a default service account to be provisioned in namespace I1105 02:23:53.690] [BeforeEach] [multi-az] I1105 02:23:53.691] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:58 I1105 02:23:53.691] [It] [env] should retain PV with reclaimPolicy "Retain" I1105 02:23:53.691] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:240 I1105 02:23:53.691] STEP: setting up the StorageClass I1105 02:23:53.691] STEP: creating a StorageClass I1105 02:23:53.773] STEP: setting up the PVC and PV I1105 02:23:53.773] STEP: creating a PVC I1105 02:23:53.857] STEP: waiting for PVC to be in phase "Bound" I1105 02:23:53.857] Nov 5 02:23:53.857: INFO: Waiting up to 5m0s for PersistentVolumeClaims [pvc-4vz8h] to have phase Bound I1105 02:23:53.936] Nov 5 02:23:53.935: INFO: PersistentVolumeClaim pvc-4vz8h found but phase is Pending instead of Bound. I1105 02:23:56.013] Nov 5 02:23:56.013: INFO: PersistentVolumeClaim pvc-4vz8h found but phase is Pending instead of Bound. I1105 02:23:58.091] Nov 5 02:23:58.090: INFO: PersistentVolumeClaim pvc-4vz8h found but phase is Pending instead of Bound. I1105 02:24:00.168] Nov 5 02:24:00.168: INFO: PersistentVolumeClaim pvc-4vz8h found but phase is Pending instead of Bound. I1105 02:24:02.246] Nov 5 02:24:02.246: INFO: PersistentVolumeClaim pvc-4vz8h found but phase is Pending instead of Bound. I1105 02:24:04.323] Nov 5 02:24:04.323: INFO: PersistentVolumeClaim pvc-4vz8h found but phase is Pending instead of Bound. I1105 02:24:06.401] Nov 5 02:24:06.401: INFO: PersistentVolumeClaim pvc-4vz8h found but phase is Pending instead of Bound. I1105 02:24:08.478] Nov 5 02:24:08.478: INFO: PersistentVolumeClaim pvc-4vz8h found but phase is Pending instead of Bound. I1105 02:24:10.556] Nov 5 02:24:10.556: INFO: PersistentVolumeClaim pvc-4vz8h found but phase is Pending instead of Bound. I1105 02:24:12.633] Nov 5 02:24:12.633: INFO: PersistentVolumeClaim pvc-4vz8h found but phase is Pending instead of Bound. I1105 02:24:14.711] Nov 5 02:24:14.711: INFO: PersistentVolumeClaim pvc-4vz8h found but phase is Pending instead of Bound. I1105 02:24:16.789] Nov 5 02:24:16.788: INFO: PersistentVolumeClaim pvc-4vz8h found but phase is Pending instead of Bound. I1105 02:24:18.867] Nov 5 02:24:18.866: INFO: PersistentVolumeClaim pvc-4vz8h found but phase is Pending instead of Bound. I1105 02:24:20.944] Nov 5 02:24:20.944: INFO: PersistentVolumeClaim pvc-4vz8h found but phase is Pending instead of Bound. I1105 02:24:23.022] Nov 5 02:24:23.022: INFO: PersistentVolumeClaim pvc-4vz8h found but phase is Pending instead of Bound. I1105 02:24:25.117] Nov 5 02:24:25.117: INFO: PersistentVolumeClaim pvc-4vz8h found but phase is Pending instead of Bound. I1105 02:24:27.195] Nov 5 02:24:27.195: INFO: PersistentVolumeClaim pvc-4vz8h found but phase is Pending instead of Bound. I1105 02:24:29.273] Nov 5 02:24:29.273: INFO: PersistentVolumeClaim pvc-4vz8h found but phase is Pending instead of Bound. I1105 02:24:31.351] Nov 5 02:24:31.351: INFO: PersistentVolumeClaim pvc-4vz8h found but phase is Pending instead of Bound. I1105 02:24:33.429] Nov 5 02:24:33.429: INFO: PersistentVolumeClaim pvc-4vz8h found but phase is Pending instead of Bound. I1105 02:24:35.507] Nov 5 02:24:35.507: INFO: PersistentVolumeClaim pvc-4vz8h found but phase is Pending instead of Bound. I1105 02:24:37.584] Nov 5 02:24:37.584: INFO: PersistentVolumeClaim pvc-4vz8h found but phase is Pending instead of Bound. I1105 02:24:39.662] Nov 5 02:24:39.662: INFO: PersistentVolumeClaim pvc-4vz8h found but phase is Pending instead of Bound. I1105 02:24:41.739] Nov 5 02:24:41.739: INFO: PersistentVolumeClaim pvc-4vz8h found but phase is Pending instead of Bound. I1105 02:24:43.822] Nov 5 02:24:43.821: INFO: PersistentVolumeClaim pvc-4vz8h found but phase is Pending instead of Bound. I1105 02:24:45.899] Nov 5 02:24:45.899: INFO: PersistentVolumeClaim pvc-4vz8h found but phase is Pending instead of Bound. I1105 02:24:47.977] Nov 5 02:24:47.977: INFO: PersistentVolumeClaim pvc-4vz8h found but phase is Pending instead of Bound. I1105 02:24:50.054] Nov 5 02:24:50.054: INFO: PersistentVolumeClaim pvc-4vz8h found but phase is Pending instead of Bound. I1105 02:24:52.133] Nov 5 02:24:52.133: INFO: PersistentVolumeClaim pvc-4vz8h found but phase is Pending instead of Bound. I1105 02:24:54.211] Nov 5 02:24:54.211: INFO: PersistentVolumeClaim pvc-4vz8h found but phase is Pending instead of Bound. I1105 02:24:56.288] Nov 5 02:24:56.288: INFO: PersistentVolumeClaim pvc-4vz8h found but phase is Pending instead of Bound. I1105 02:24:58.367] Nov 5 02:24:58.366: INFO: PersistentVolumeClaim pvc-4vz8h found but phase is Pending instead of Bound. I1105 02:25:00.447] Nov 5 02:25:00.446: INFO: PersistentVolumeClaim pvc-4vz8h found but phase is Pending instead of Bound. I1105 02:25:02.524] Nov 5 02:25:02.524: INFO: PersistentVolumeClaim pvc-4vz8h found but phase is Pending instead of Bound. I1105 02:25:04.602] Nov 5 02:25:04.602: INFO: PersistentVolumeClaim pvc-4vz8h found but phase is Pending instead of Bound. I1105 02:25:06.685] Nov 5 02:25:06.685: INFO: PersistentVolumeClaim pvc-4vz8h found but phase is Pending instead of Bound. I1105 02:25:08.762] Nov 5 02:25:08.762: INFO: PersistentVolumeClaim pvc-4vz8h found but phase is Pending instead of Bound. I1105 02:25:10.840] Nov 5 02:25:10.840: INFO: PersistentVolumeClaim pvc-4vz8h found but phase is Pending instead of Bound. I1105 02:25:12.917] Nov 5 02:25:12.917: INFO: PersistentVolumeClaim pvc-4vz8h found but phase is Pending instead of Bound. I1105 02:25:14.995] Nov 5 02:25:14.995: INFO: PersistentVolumeClaim pvc-4vz8h found but phase is Pending instead of Bound. I1105 02:25:17.073] Nov 5 02:25:17.073: INFO: PersistentVolumeClaim pvc-4vz8h found but phase is Pending instead of Bound. I1105 02:25:19.156] Nov 5 02:25:19.155: INFO: PersistentVolumeClaim pvc-4vz8h found but phase is Pending instead of Bound. I1105 02:25:21.234] Nov 5 02:25:21.233: INFO: PersistentVolumeClaim pvc-4vz8h found but phase is Pending instead of Bound. I1105 02:25:23.311] Nov 5 02:25:23.311: INFO: PersistentVolumeClaim pvc-4vz8h found but phase is Pending instead of Bound. I1105 02:25:25.388] Nov 5 02:25:25.388: INFO: PersistentVolumeClaim pvc-4vz8h found but phase is Pending instead of Bound. I1105 02:25:27.467] Nov 5 02:25:27.467: INFO: PersistentVolumeClaim pvc-4vz8h found but phase is Pending instead of Bound. I1105 02:25:29.545] Nov 5 02:25:29.544: INFO: PersistentVolumeClaim pvc-4vz8h found but phase is Pending instead of Bound. I1105 02:25:31.622] Nov 5 02:25:31.622: INFO: PersistentVolumeClaim pvc-4vz8h found but phase is Pending instead of Bound. I1105 02:25:33.700] Nov 5 02:25:33.700: INFO: PersistentVolumeClaim pvc-4vz8h found but phase is Pending instead of Bound. I1105 02:25:35.778] Nov 5 02:25:35.778: INFO: PersistentVolumeClaim pvc-4vz8h found but phase is Pending instead of Bound. I1105 02:25:37.856] Nov 5 02:25:37.856: INFO: PersistentVolumeClaim pvc-4vz8h found but phase is Pending instead of Bound. I1105 02:25:39.934] Nov 5 02:25:39.934: INFO: PersistentVolumeClaim pvc-4vz8h found but phase is Pending instead of Bound. I1105 02:25:42.012] Nov 5 02:25:42.011: INFO: PersistentVolumeClaim pvc-4vz8h found but phase is Pending instead of Bound. I1105 02:25:44.090] Nov 5 02:25:44.089: INFO: PersistentVolumeClaim pvc-4vz8h found but phase is Pending instead of Bound. I1105 02:25:46.167] Nov 5 02:25:46.167: INFO: PersistentVolumeClaim pvc-4vz8h found but phase is Pending instead of Bound. I1105 02:25:48.245] Nov 5 02:25:48.245: INFO: PersistentVolumeClaim pvc-4vz8h found but phase is Pending instead of Bound. I1105 02:25:50.323] Nov 5 02:25:50.323: INFO: PersistentVolumeClaim pvc-4vz8h found but phase is Pending instead of Bound. I1105 02:25:52.401] Nov 5 02:25:52.401: INFO: PersistentVolumeClaim pvc-4vz8h found but phase is Pending instead of Bound. I1105 02:25:54.479] Nov 5 02:25:54.478: INFO: PersistentVolumeClaim pvc-4vz8h found but phase is Pending instead of Bound. I1105 02:25:56.557] Nov 5 02:25:56.556: INFO: PersistentVolumeClaim pvc-4vz8h found but phase is Pending instead of Bound. I1105 02:25:58.634] Nov 5 02:25:58.634: INFO: PersistentVolumeClaim pvc-4vz8h found but phase is Pending instead of Bound. I1105 02:26:00.712] Nov 5 02:26:00.712: INFO: PersistentVolumeClaim pvc-4vz8h found but phase is Pending instead of Bound. I1105 02:26:02.790] Nov 5 02:26:02.790: INFO: PersistentVolumeClaim pvc-4vz8h found but phase is Pending instead of Bound. I1105 02:26:04.868] Nov 5 02:26:04.868: INFO: PersistentVolumeClaim pvc-4vz8h found but phase is Pending instead of Bound. I1105 02:26:06.946] Nov 5 02:26:06.946: INFO: PersistentVolumeClaim pvc-4vz8h found but phase is Pending instead of Bound. I1105 02:26:09.024] Nov 5 02:26:09.023: INFO: PersistentVolumeClaim pvc-4vz8h found but phase is Pending instead of Bound. I1105 02:26:11.102] Nov 5 02:26:11.101: INFO: PersistentVolumeClaim pvc-4vz8h found but phase is Pending instead of Bound. I1105 02:26:13.181] Nov 5 02:26:13.181: INFO: PersistentVolumeClaim pvc-4vz8h found but phase is Pending instead of Bound. I1105 02:26:15.259] Nov 5 02:26:15.259: INFO: PersistentVolumeClaim pvc-4vz8h found but phase is Pending instead of Bound. I1105 02:26:17.336] Nov 5 02:26:17.336: INFO: PersistentVolumeClaim pvc-4vz8h found but phase is Pending instead of Bound. I1105 02:26:19.415] Nov 5 02:26:19.414: INFO: PersistentVolumeClaim pvc-4vz8h found but phase is Pending instead of Bound. I1105 02:26:21.492] Nov 5 02:26:21.492: INFO: PersistentVolumeClaim pvc-4vz8h found but phase is Pending instead of Bound. I1105 02:26:23.571] Nov 5 02:26:23.571: INFO: PersistentVolumeClaim pvc-4vz8h found but phase is Pending instead of Bound. I1105 02:26:25.649] Nov 5 02:26:25.649: INFO: PersistentVolumeClaim pvc-4vz8h found but phase is Pending instead of Bound. I1105 02:26:27.726] Nov 5 02:26:27.726: INFO: PersistentVolumeClaim pvc-4vz8h found but phase is Pending instead of Bound. I1105 02:26:29.804] Nov 5 02:26:29.803: INFO: PersistentVolumeClaim pvc-4vz8h found but phase is Pending instead of Bound. I1105 02:26:31.882] Nov 5 02:26:31.882: INFO: PersistentVolumeClaim pvc-4vz8h found but phase is Pending instead of Bound. I1105 02:26:33.960] Nov 5 02:26:33.959: INFO: PersistentVolumeClaim pvc-4vz8h found but phase is Pending instead of Bound. I1105 02:26:36.038] Nov 5 02:26:36.038: INFO: PersistentVolumeClaim pvc-4vz8h found but phase is Pending instead of Bound. I1105 02:26:38.118] Nov 5 02:26:38.118: INFO: PersistentVolumeClaim pvc-4vz8h found but phase is Pending instead of Bound. I1105 02:26:40.196] Nov 5 02:26:40.196: INFO: PersistentVolumeClaim pvc-4vz8h found but phase is Pending instead of Bound. I1105 02:26:42.275] Nov 5 02:26:42.275: INFO: PersistentVolumeClaim pvc-4vz8h found but phase is Pending instead of Bound. I1105 02:26:44.357] Nov 5 02:26:44.356: INFO: PersistentVolumeClaim pvc-4vz8h found but phase is Pending instead of Bound. I1105 02:26:46.434] Nov 5 02:26:46.434: INFO: PersistentVolumeClaim pvc-4vz8h found but phase is Pending instead of Bound. I1105 02:26:48.512] Nov 5 02:26:48.512: INFO: PersistentVolumeClaim pvc-4vz8h found but phase is Pending instead of Bound. I1105 02:26:50.590] Nov 5 02:26:50.590: INFO: PersistentVolumeClaim pvc-4vz8h found but phase is Pending instead of Bound. I1105 02:26:52.668] Nov 5 02:26:52.668: INFO: PersistentVolumeClaim pvc-4vz8h found but phase is Pending instead of Bound. I1105 02:26:54.746] Nov 5 02:26:54.746: INFO: PersistentVolumeClaim pvc-4vz8h found but phase is Pending instead of Bound. I1105 02:26:56.824] Nov 5 02:26:56.824: INFO: PersistentVolumeClaim pvc-4vz8h found but phase is Pending instead of Bound. I1105 02:26:58.902] Nov 5 02:26:58.902: INFO: PersistentVolumeClaim pvc-4vz8h found but phase is Pending instead of Bound. I1105 02:27:00.980] Nov 5 02:27:00.980: INFO: PersistentVolumeClaim pvc-4vz8h found but phase is Pending instead of Bound. I1105 02:27:03.058] Nov 5 02:27:03.057: INFO: PersistentVolumeClaim pvc-4vz8h found but phase is Pending instead of Bound. I1105 02:27:05.145] Nov 5 02:27:05.145: INFO: PersistentVolumeClaim pvc-4vz8h found but phase is Pending instead of Bound. I1105 02:27:07.224] Nov 5 02:27:07.224: INFO: PersistentVolumeClaim pvc-4vz8h found but phase is Pending instead of Bound. I1105 02:27:09.302] Nov 5 02:27:09.302: INFO: PersistentVolumeClaim pvc-4vz8h found but phase is Pending instead of Bound. I1105 02:27:11.380] Nov 5 02:27:11.380: INFO: PersistentVolumeClaim pvc-4vz8h found but phase is Pending instead of Bound. I1105 02:27:13.458] Nov 5 02:27:13.458: INFO: PersistentVolumeClaim pvc-4vz8h found but phase is Pending instead of Bound. I1105 02:27:15.536] Nov 5 02:27:15.536: INFO: PersistentVolumeClaim pvc-4vz8h found but phase is Pending instead of Bound. I1105 02:27:17.614] Nov 5 02:27:17.614: INFO: PersistentVolumeClaim pvc-4vz8h found but phase is Pending instead of Bound. I1105 02:27:19.692] Nov 5 02:27:19.692: INFO: PersistentVolumeClaim pvc-4vz8h found but phase is Pending instead of Bound. I1105 02:27:21.770] Nov 5 02:27:21.770: INFO: PersistentVolumeClaim pvc-4vz8h found but phase is Pending instead of Bound. I1105 02:27:23.849] Nov 5 02:27:23.849: INFO: PersistentVolumeClaim pvc-4vz8h found but phase is Pending instead of Bound. I1105 02:27:25.927] Nov 5 02:27:25.927: INFO: PersistentVolumeClaim pvc-4vz8h found but phase is Pending instead of Bound. I1105 02:27:28.016] Nov 5 02:27:28.016: INFO: PersistentVolumeClaim pvc-4vz8h found but phase is Pending instead of Bound. I1105 02:27:30.094] Nov 5 02:27:30.094: INFO: PersistentVolumeClaim pvc-4vz8h found but phase is Pending instead of Bound. I1105 02:27:32.172] Nov 5 02:27:32.172: INFO: PersistentVolumeClaim pvc-4vz8h found but phase is Pending instead of Bound. I1105 02:27:34.250] Nov 5 02:27:34.250: INFO: PersistentVolumeClaim pvc-4vz8h found but phase is Pending instead of Bound. I1105 02:27:36.328] Nov 5 02:27:36.327: INFO: PersistentVolumeClaim pvc-4vz8h found but phase is Pending instead of Bound. I1105 02:27:38.406] Nov 5 02:27:38.406: INFO: PersistentVolumeClaim pvc-4vz8h found but phase is Pending instead of Bound. I1105 02:27:40.483] Nov 5 02:27:40.483: INFO: PersistentVolumeClaim pvc-4vz8h found but phase is Pending instead of Bound. I1105 02:27:42.562] Nov 5 02:27:42.561: INFO: PersistentVolumeClaim pvc-4vz8h found but phase is Pending instead of Bound. I1105 02:27:44.640] Nov 5 02:27:44.640: INFO: PersistentVolumeClaim pvc-4vz8h found but phase is Pending instead of Bound. I1105 02:27:46.718] Nov 5 02:27:46.718: INFO: PersistentVolumeClaim pvc-4vz8h found but phase is Pending instead of Bound. I1105 02:27:48.796] Nov 5 02:27:48.795: INFO: PersistentVolumeClaim pvc-4vz8h found but phase is Pending instead of Bound. I1105 02:27:50.873] Nov 5 02:27:50.873: INFO: PersistentVolumeClaim pvc-4vz8h found but phase is Pending instead of Bound. I1105 02:27:52.952] Nov 5 02:27:52.952: INFO: PersistentVolumeClaim pvc-4vz8h found but phase is Pending instead of Bound. I1105 02:27:55.030] Nov 5 02:27:55.030: INFO: PersistentVolumeClaim pvc-4vz8h found but phase is Pending instead of Bound. I1105 02:27:57.149] Nov 5 02:27:57.149: INFO: PersistentVolumeClaim pvc-4vz8h found but phase is Pending instead of Bound. I1105 02:27:59.227] Nov 5 02:27:59.227: INFO: PersistentVolumeClaim pvc-4vz8h found but phase is Pending instead of Bound. I1105 02:28:01.305] Nov 5 02:28:01.304: INFO: PersistentVolumeClaim pvc-4vz8h found but phase is Pending instead of Bound. I1105 02:28:03.382] Nov 5 02:28:03.382: INFO: PersistentVolumeClaim pvc-4vz8h found but phase is Pending instead of Bound. I1105 02:28:05.460] Nov 5 02:28:05.459: INFO: PersistentVolumeClaim pvc-4vz8h found but phase is Pending instead of Bound. I1105 02:28:07.537] Nov 5 02:28:07.537: INFO: PersistentVolumeClaim pvc-4vz8h found but phase is Pending instead of Bound. I1105 02:28:09.627] Nov 5 02:28:09.626: INFO: PersistentVolumeClaim pvc-4vz8h found but phase is Pending instead of Bound. I1105 02:28:11.705] Nov 5 02:28:11.705: INFO: PersistentVolumeClaim pvc-4vz8h found but phase is Pending instead of Bound. I1105 02:28:13.783] Nov 5 02:28:13.783: INFO: PersistentVolumeClaim pvc-4vz8h found but phase is Pending instead of Bound. I1105 02:28:15.860] Nov 5 02:28:15.860: INFO: PersistentVolumeClaim pvc-4vz8h found but phase is Pending instead of Bound. I1105 02:28:17.945] Nov 5 02:28:17.944: INFO: PersistentVolumeClaim pvc-4vz8h found but phase is Pending instead of Bound. I1105 02:28:20.022] Nov 5 02:28:20.022: INFO: PersistentVolumeClaim pvc-4vz8h found but phase is Pending instead of Bound. I1105 02:28:22.100] Nov 5 02:28:22.100: INFO: PersistentVolumeClaim pvc-4vz8h found but phase is Pending instead of Bound. I1105 02:28:24.178] Nov 5 02:28:24.177: INFO: PersistentVolumeClaim pvc-4vz8h found but phase is Pending instead of Bound. I1105 02:28:26.256] Nov 5 02:28:26.256: INFO: PersistentVolumeClaim pvc-4vz8h found but phase is Pending instead of Bound. I1105 02:28:28.334] Nov 5 02:28:28.333: INFO: PersistentVolumeClaim pvc-4vz8h found but phase is Pending instead of Bound. I1105 02:28:30.411] Nov 5 02:28:30.411: INFO: PersistentVolumeClaim pvc-4vz8h found but phase is Pending instead of Bound. I1105 02:28:32.489] Nov 5 02:28:32.489: INFO: PersistentVolumeClaim pvc-4vz8h found but phase is Pending instead of Bound. I1105 02:28:34.567] Nov 5 02:28:34.567: INFO: PersistentVolumeClaim pvc-4vz8h found but phase is Pending instead of Bound. I1105 02:28:36.645] Nov 5 02:28:36.645: INFO: PersistentVolumeClaim pvc-4vz8h found but phase is Pending instead of Bound. I1105 02:28:38.732] Nov 5 02:28:38.732: INFO: PersistentVolumeClaim pvc-4vz8h found but phase is Pending instead of Bound. I1105 02:28:40.810] Nov 5 02:28:40.810: INFO: PersistentVolumeClaim pvc-4vz8h found but phase is Pending instead of Bound. I1105 02:28:42.888] Nov 5 02:28:42.887: INFO: PersistentVolumeClaim pvc-4vz8h found but phase is Pending instead of Bound. I1105 02:28:44.967] Nov 5 02:28:44.966: INFO: PersistentVolumeClaim pvc-4vz8h found but phase is Pending instead of Bound. I1105 02:28:47.044] Nov 5 02:28:47.044: INFO: PersistentVolumeClaim pvc-4vz8h found but phase is Pending instead of Bound. I1105 02:28:49.128] Nov 5 02:28:49.127: INFO: PersistentVolumeClaim pvc-4vz8h found but phase is Pending instead of Bound. I1105 02:28:51.206] Nov 5 02:28:51.205: INFO: PersistentVolumeClaim pvc-4vz8h found but phase is Pending instead of Bound. I1105 02:28:53.283] Nov 5 02:28:53.283: INFO: PersistentVolumeClaim pvc-4vz8h found but phase is Pending instead of Bound. I1105 02:28:55.284] Nov 5 02:28:55.283: INFO: Unexpected error occurred: PersistentVolumeClaims [pvc-4vz8h] not all in phase Bound within 5m0s I1105 02:28:55.284] [AfterEach] [multi-az] I1105 02:28:55.285] /go/src/sigs.k8s.io/azuredisk-csi-driver/vendor/k8s.io/kubernetes/test/e2e/framework/framework.go:151 I1105 02:28:55.285] STEP: Collecting events from namespace "azuredisk-7960". I1105 02:28:55.362] STEP: Found 3 events. I1105 02:28:55.362] Nov 5 02:28:55.362: INFO: At 2019-11-05 02:23:53 +0000 UTC - event for pvc-4vz8h: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "disk.csi.azure.com" or manually created by system administrator I1105 02:28:55.363] Nov 5 02:28:55.362: INFO: At 2019-11-05 02:23:53 +0000 UTC - event for pvc-4vz8h: {disk.csi.azure.com_csi-azuredisk-controller-6f6587bfc6-8vkfl_1376914b-ff6d-11e9-93ab-ba2bee589b8c } Provisioning: External provisioner is provisioning volume for claim "azuredisk-7960/pvc-4vz8h" I1105 02:28:55.363] Nov 5 02:28:55.362: INFO: At 2019-11-05 02:23:53 +0000 UTC - event for pvc-4vz8h: {disk.csi.azure.com_csi-azuredisk-controller-6f6587bfc6-8vkfl_1376914b-ff6d-11e9-93ab-ba2bee589b8c } ProvisioningFailed: failed to provision volume with StorageClass "azuredisk-7960-disk.csi.azure.com-dynamic-sc-jcdtm": rpc error: code = Unknown desc = compute.DisksClient#CreateOrUpdate: Failure sending request: StatusCode=400 -- Original Error: Code="InvalidAvailabilityZone" Message="The zone(s) '' for resource 'Microsoft.Compute/disks/pvc-bab2ead6-6996-424b-8320-48fd7d56fff4' is not supported. The supported zones for location 'eastus2' are '1,2,3'" I1105 02:28:55.439] Nov 5 02:28:55.439: INFO: POD NODE PHASE GRACE CONDITIONS I1105 02:28:55.440] Nov 5 02:28:55.439: INFO: I1105 02:28:55.670] Nov 5 02:28:55.670: INFO: I1105 02:28:55.670] Logging node info for node k8s-agentpool-15826033-vmss000000 I1105 02:28:55.750] Nov 5 02:28:55.747: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-agentpool-15826033-vmss000000,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-agentpool-15826033-vmss000000,UID:15fa874e-54ec-42cd-b243-d3baa512150f,ResourceVersion:10714,Generation:0,CreationTimestamp:2019-11-05 01:34:17 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{agentpool: agentpool,beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_D4s_v3,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 0,kubernetes.azure.com/cluster: kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5,kubernetes.azure.com/role: agent,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-agentpool-15826033-vmss000000,kubernetes.io/os: linux,kubernetes.io/role: agent,node-role.kubernetes.io/agent: ,storageprofile: managed,storagetier: Premium_LRS,topology.disk.csi.azure.com/zone: 0,},Annotations:map[string]string{csi.volume.kubernetes.io/nodeid: {"disk.csi.azure.com":"k8s-agentpool-15826033-vmss000000"},node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.2.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool-15826033-vmss/virtualMachines/0,Unschedulable:false,Taints:[],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-csi-disk.csi.azure.com: {{8 0} {} 8 DecimalSI},cpu: {{4 0} {} 4 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{16797044736 0} {} 16403364Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-csi-disk.csi.azure.com: {{8 0} {} 8 DecimalSI},cpu: {{4 0} {} 4 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{16010612736 0} {} 15635364Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 01:34:56 +0000 UTC 2019-11-05 01:34:56 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 02:28:26 +0000 UTC 2019-11-05 01:34:17 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 02:28:26 +0000 UTC 2019-11-05 01:34:17 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 02:28:26 +0000 UTC 2019-11-05 01:34:17 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 02:28:26 +0000 UTC 2019-11-05 01:34:29 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.0.4} {Hostname k8s-agentpool-15826033-vmss000000}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:afe00792177d49a98f6a44d03d08732d,SystemUUID:4F1777BF-A5C3-004E-81C4-90F0B54C02A0,BootID:13b6a628-ae89-44af-b891-89461030adcb,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[k8sprowinternal.azurecr.io/azuredisk-csi@sha256:75d4ff5d7cd845d6e810c67796ecb037e852fb849ec1eb5b3194564008e6d1e0 k8sprowinternal.azurecr.io/azuredisk-csi:e2e-3e3ed045c6faa509dbd8d2890ee9f56aa2934887] 286930414} {[k8s.gcr.io/kubernetes-dashboard-amd64@sha256:0ae6b69432e78069c5ce2bcde0fe409c5c4d6f0f4d9cd50a17974fea38898747 k8s.gcr.io/kubernetes-dashboard-amd64:v1.10.1] 121711221} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/metrics-server-amd64@sha256:49a9f12f7067d11f42c803dbe61ed2c1299959ad85cb315b25ff7eef8e6b8892 k8s.gcr.io/metrics-server-amd64:v0.2.1] 42541759} {[mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume@sha256:4fd30d43947d4a54fc89ead7985beecfd3c9b2a93a0655a373b1608ab90bd5af mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume:v0.0.7] 22909487} {[quay.io/k8scsi/livenessprobe@sha256:8c9c8a0f330acc2e558ae6624309447d02522e76aa35af23fa19b88abf4381cb quay.io/k8scsi/livenessprobe:v1.0.2] 17415381} {[quay.io/k8scsi/csi-node-driver-registrar@sha256:13daf82fb99e951a4bff8ae5fc7c17c3a8fe7130be6400990d8f6076c32d4599 quay.io/k8scsi/csi-node-driver-registrar:v1.1.0] 15815995} {[mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume@sha256:23d8c6033f02a1ecad05127ebdc931bb871264228661bc122704b0974e4d9fdd mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume:1.0.8] 1159025} {[busybox@sha256:8ccbac733d19c0dd4d70b4f0c1e12245b5fa3ad24758a11035ee505c629c0796 busybox:1.29] 1154361} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 02:28:55.750] Nov 5 02:28:55.747: INFO: I1105 02:28:55.751] Logging kubelet events for node k8s-agentpool-15826033-vmss000000 I1105 02:28:55.829] Nov 5 02:28:55.828: INFO: I1105 02:28:55.829] Logging pods the kubelet thinks is on node k8s-agentpool-15826033-vmss000000 I1105 02:28:55.916] Nov 5 02:28:55.916: INFO: metrics-server-864ffbc5c-sdxps started at 2019-11-05 01:34:34 +0000 UTC (0+1 container statuses recorded) I1105 02:28:55.916] Nov 5 02:28:55.916: INFO: Container metrics-server ready: true, restart count 0 I1105 02:28:55.917] Nov 5 02:28:55.916: INFO: kube-proxy-wxblc started at 2019-11-05 01:34:34 +0000 UTC (0+1 container statuses recorded) I1105 02:28:55.917] Nov 5 02:28:55.916: INFO: Container kube-proxy ready: true, restart count 0 I1105 02:28:55.917] Nov 5 02:28:55.916: INFO: kubernetes-dashboard-66dd8b8df7-ks284 started at 2019-11-05 01:34:35 +0000 UTC (0+1 container statuses recorded) I1105 02:28:55.917] Nov 5 02:28:55.916: INFO: Container kubernetes-dashboard ready: true, restart count 0 I1105 02:28:55.917] Nov 5 02:28:55.916: INFO: csi-azuredisk-node-whq2g started at 2019-11-05 01:38:24 +0000 UTC (0+3 container statuses recorded) I1105 02:28:55.918] Nov 5 02:28:55.916: INFO: Container azuredisk ready: true, restart count 0 I1105 02:28:55.918] Nov 5 02:28:55.916: INFO: Container liveness-probe ready: true, restart count 0 I1105 02:28:55.918] Nov 5 02:28:55.916: INFO: Container node-driver-registrar ready: true, restart count 0 I1105 02:28:55.918] Nov 5 02:28:55.916: INFO: blobfuse-flexvol-installer-bsvqt started at 2019-11-05 01:34:34 +0000 UTC (0+1 container statuses recorded) I1105 02:28:55.918] Nov 5 02:28:55.916: INFO: Container blobfuse-flexvol-installer ready: true, restart count 0 I1105 02:28:55.919] Nov 5 02:28:55.916: INFO: azure-ip-masq-agent-vx86b started at 2019-11-05 01:34:34 +0000 UTC (0+1 container statuses recorded) I1105 02:28:55.919] Nov 5 02:28:55.916: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 02:28:55.919] Nov 5 02:28:55.916: INFO: keyvault-flexvolume-qhnfr started at 2019-11-05 01:34:34 +0000 UTC (0+1 container statuses recorded) I1105 02:28:55.919] Nov 5 02:28:55.916: INFO: Container keyvault-flexvolume ready: true, restart count 0 I1105 02:28:56.165] Nov 5 02:28:56.165: INFO: I1105 02:28:56.165] Latency metrics for node k8s-agentpool-15826033-vmss000000 I1105 02:28:56.166] Nov 5 02:28:56.165: INFO: I1105 02:28:56.166] Logging node info for node k8s-agentpool-15826033-vmss000001 I1105 02:28:56.246] Nov 5 02:28:56.242: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-agentpool-15826033-vmss000001,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-agentpool-15826033-vmss000001,UID:4a8ea605-cc0b-4e3b-9060-d8867d269e6c,ResourceVersion:10778,Generation:0,CreationTimestamp:2019-11-05 01:34:09 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{agentpool: agentpool,beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_D4s_v3,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 1,kubernetes.azure.com/cluster: kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5,kubernetes.azure.com/role: agent,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-agentpool-15826033-vmss000001,kubernetes.io/os: linux,kubernetes.io/role: agent,node-role.kubernetes.io/agent: ,storageprofile: managed,storagetier: Premium_LRS,topology.disk.csi.azure.com/zone: 1,},Annotations:map[string]string{csi.volume.kubernetes.io/nodeid: {"disk.csi.azure.com":"k8s-agentpool-15826033-vmss000001"},node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.0.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool-15826033-vmss/virtualMachines/1,Unschedulable:false,Taints:[],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-csi-disk.csi.azure.com: {{8 0} {} 8 DecimalSI},cpu: {{4 0} {} 4 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{16797044736 0} {} 16403364Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-csi-disk.csi.azure.com: {{8 0} {} 8 DecimalSI},cpu: {{4 0} {} 4 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{16010612736 0} {} 15635364Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 01:34:57 +0000 UTC 2019-11-05 01:34:57 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 02:28:48 +0000 UTC 2019-11-05 01:34:08 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 02:28:48 +0000 UTC 2019-11-05 01:34:08 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 02:28:48 +0000 UTC 2019-11-05 01:34:08 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 02:28:48 +0000 UTC 2019-11-05 01:34:29 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.0.5} {Hostname k8s-agentpool-15826033-vmss000001}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:b6b9fbf4a2fd4fa5a7ae472415fe30b8,SystemUUID:0162B3A4-3955-9A46-ABD4-A37BD75848CC,BootID:ee93003f-d47c-4118-9975-57365b3443a5,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[k8sprowinternal.azurecr.io/azuredisk-csi@sha256:75d4ff5d7cd845d6e810c67796ecb037e852fb849ec1eb5b3194564008e6d1e0 k8sprowinternal.azurecr.io/azuredisk-csi:e2e-3e3ed045c6faa509dbd8d2890ee9f56aa2934887] 286930414} {[gcr.io/kubernetes-helm/tiller@sha256:f6d8f4ab9ba993b5f5b60a6edafe86352eabe474ffeb84cb6c79b8866dce45d1 gcr.io/kubernetes-helm/tiller:v2.11.0] 71821984} {[quay.io/k8scsi/csi-provisioner@sha256:d657c839dce87324fe2b677302913f9386f885f8746be7bea0ced5b0844e3433 quay.io/k8scsi/csi-provisioner:v1.3.1] 50596998} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[quay.io/k8scsi/csi-attacher@sha256:26fccd7a99d973845df1193b46ebdcc6ab8dc5f6e6be319750c471fce1742d13 quay.io/k8scsi/csi-attacher:v1.2.0] 46226754} {[quay.io/k8scsi/csi-cluster-driver-registrar@sha256:fafd75ae5442f192cfa8c2e792903aee30d5884b62e802e4464b0a895d21e3ef quay.io/k8scsi/csi-cluster-driver-registrar:v1.0.1] 45874691} {[quay.io/k8scsi/csi-snapshotter@sha256:a49e0da1af6f2bf717e41ba1eee8b5e6a1cbd66a709dd92cc43fe475fe2589eb quay.io/k8scsi/csi-snapshotter:v1.1.0] 44325352} {[k8s.gcr.io/coredns@sha256:e83beb5e43f8513fa735e77ffc5859640baea30a882a11cc75c4c3244a737d3c k8s.gcr.io/coredns:1.5.0] 42488424} {[mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume@sha256:4fd30d43947d4a54fc89ead7985beecfd3c9b2a93a0655a373b1608ab90bd5af mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume:v0.0.7] 22909487} {[quay.io/k8scsi/livenessprobe@sha256:8c9c8a0f330acc2e558ae6624309447d02522e76aa35af23fa19b88abf4381cb quay.io/k8scsi/livenessprobe:v1.0.2] 17415381} {[quay.io/k8scsi/csi-node-driver-registrar@sha256:13daf82fb99e951a4bff8ae5fc7c17c3a8fe7130be6400990d8f6076c32d4599 quay.io/k8scsi/csi-node-driver-registrar:v1.1.0] 15815995} {[quay.io/k8scsi/livenessprobe@sha256:dde617756e0f602adc566ab71fd885f1dad451ad3fb063ac991c95a2ff47aea5 quay.io/k8scsi/livenessprobe:v1.1.0] 14967303} {[mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume@sha256:23d8c6033f02a1ecad05127ebdc931bb871264228661bc122704b0974e4d9fdd mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume:1.0.8] 1159025} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 02:28:56.246] Nov 5 02:28:56.243: INFO: I1105 02:28:56.247] Logging kubelet events for node k8s-agentpool-15826033-vmss000001 I1105 02:28:56.324] Nov 5 02:28:56.324: INFO: I1105 02:28:56.324] Logging pods the kubelet thinks is on node k8s-agentpool-15826033-vmss000001 I1105 02:28:56.407] Nov 5 02:28:56.407: INFO: csi-azuredisk-node-ghv5w started at 2019-11-05 01:38:24 +0000 UTC (0+3 container statuses recorded) I1105 02:28:56.407] Nov 5 02:28:56.407: INFO: Container azuredisk ready: true, restart count 0 I1105 02:28:56.407] Nov 5 02:28:56.407: INFO: Container liveness-probe ready: true, restart count 0 I1105 02:28:56.407] Nov 5 02:28:56.407: INFO: Container node-driver-registrar ready: true, restart count 0 I1105 02:28:56.408] Nov 5 02:28:56.407: INFO: csi-azuredisk-controller-6f6587bfc6-8vkfl started at 2019-11-05 01:38:24 +0000 UTC (0+6 container statuses recorded) I1105 02:28:56.408] Nov 5 02:28:56.407: INFO: Container azuredisk ready: true, restart count 0 I1105 02:28:56.408] Nov 5 02:28:56.407: INFO: Container cluster-driver-registrar ready: true, restart count 0 I1105 02:28:56.408] Nov 5 02:28:56.407: INFO: Container csi-attacher ready: true, restart count 0 I1105 02:28:56.408] Nov 5 02:28:56.407: INFO: Container csi-provisioner ready: true, restart count 0 I1105 02:28:56.409] Nov 5 02:28:56.407: INFO: Container csi-snapshotter ready: true, restart count 0 I1105 02:28:56.409] Nov 5 02:28:56.407: INFO: Container liveness-probe ready: true, restart count 0 I1105 02:28:56.409] Nov 5 02:28:56.407: INFO: blobfuse-flexvol-installer-rw5dw started at 2019-11-05 01:34:35 +0000 UTC (0+1 container statuses recorded) I1105 02:28:56.409] Nov 5 02:28:56.407: INFO: Container blobfuse-flexvol-installer ready: true, restart count 0 I1105 02:28:56.409] Nov 5 02:28:56.407: INFO: coredns-7f68dcdbdb-jt522 started at 2019-11-05 01:34:35 +0000 UTC (0+1 container statuses recorded) I1105 02:28:56.410] Nov 5 02:28:56.407: INFO: Container coredns ready: true, restart count 0 I1105 02:28:56.410] Nov 5 02:28:56.407: INFO: azure-ip-masq-agent-6c6vn started at 2019-11-05 01:34:35 +0000 UTC (0+1 container statuses recorded) I1105 02:28:56.410] Nov 5 02:28:56.407: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 02:28:56.410] Nov 5 02:28:56.407: INFO: keyvault-flexvolume-rdlcj started at 2019-11-05 01:34:35 +0000 UTC (0+1 container statuses recorded) I1105 02:28:56.410] Nov 5 02:28:56.407: INFO: Container keyvault-flexvolume ready: true, restart count 0 I1105 02:28:56.411] Nov 5 02:28:56.407: INFO: kube-proxy-6kh57 started at 2019-11-05 01:34:35 +0000 UTC (0+1 container statuses recorded) I1105 02:28:56.411] Nov 5 02:28:56.407: INFO: Container kube-proxy ready: true, restart count 0 I1105 02:28:56.411] Nov 5 02:28:56.407: INFO: tiller-deploy-54c96cb5df-8d9fs started at 2019-11-05 01:34:35 +0000 UTC (0+1 container statuses recorded) I1105 02:28:56.411] Nov 5 02:28:56.407: INFO: Container tiller ready: true, restart count 0 I1105 02:28:56.661] Nov 5 02:28:56.661: INFO: I1105 02:28:56.661] Latency metrics for node k8s-agentpool-15826033-vmss000001 I1105 02:28:56.661] Nov 5 02:28:56.661: INFO: I1105 02:28:56.662] Logging node info for node k8s-master-15826033-0 I1105 02:28:56.740] Nov 5 02:28:56.737: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-15826033-0,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-15826033-0,UID:aceb0231-4645-4a9a-b296-90ada851986b,ResourceVersion:10740,Generation:0,CreationTimestamp:2019-11-05 01:35:06 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 1,kubernetes.azure.com/cluster: kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-15826033-0,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.4.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5/providers/Microsoft.Compute/virtualMachines/k8s-master-15826033-0,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 01:35:20 +0000 UTC 2019-11-05 01:35:20 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 02:28:34 +0000 UTC 2019-11-05 01:34:57 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 02:28:34 +0000 UTC 2019-11-05 01:34:57 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 02:28:34 +0000 UTC 2019-11-05 01:34:57 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 02:28:34 +0000 UTC 2019-11-05 01:35:07 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.255.5} {Hostname k8s-master-15826033-0}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:1915756c22a540dc8eb30493987e9a43,SystemUUID:742C969C-5A71-D744-8E61-F27E38B56EB2,BootID:03f0672e-8fa9-4309-956f-8e3c371902c3,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 02:28:56.740] Nov 5 02:28:56.738: INFO: I1105 02:28:56.740] Logging kubelet events for node k8s-master-15826033-0 I1105 02:28:56.820] Nov 5 02:28:56.819: INFO: I1105 02:28:56.820] Logging pods the kubelet thinks is on node k8s-master-15826033-0 I1105 02:28:56.901] Nov 5 02:28:56.901: INFO: azure-ip-masq-agent-wgdpx started at 2019-11-05 01:35:06 +0000 UTC (0+1 container statuses recorded) I1105 02:28:56.902] Nov 5 02:28:56.901: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 02:28:56.902] Nov 5 02:28:56.901: INFO: kube-proxy-7d2px started at 2019-11-05 01:35:06 +0000 UTC (0+1 container statuses recorded) I1105 02:28:56.902] Nov 5 02:28:56.901: INFO: Container kube-proxy ready: true, restart count 0 I1105 02:28:56.902] Nov 5 02:28:56.901: INFO: cloud-controller-manager-k8s-master-15826033-0 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:28:56.902] Nov 5 02:28:56.901: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 02:28:56.903] Nov 5 02:28:56.901: INFO: kube-addon-manager-k8s-master-15826033-0 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:28:56.903] Nov 5 02:28:56.901: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 02:28:56.903] Nov 5 02:28:56.901: INFO: kube-apiserver-k8s-master-15826033-0 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:28:56.903] Nov 5 02:28:56.901: INFO: Container kube-apiserver ready: true, restart count 0 I1105 02:28:56.903] Nov 5 02:28:56.901: INFO: kube-controller-manager-k8s-master-15826033-0 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:28:56.904] Nov 5 02:28:56.901: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 02:28:56.904] Nov 5 02:28:56.901: INFO: kube-scheduler-k8s-master-15826033-0 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:28:56.904] Nov 5 02:28:56.901: INFO: Container kube-scheduler ready: true, restart count 0 I1105 02:28:57.156] Nov 5 02:28:57.156: INFO: I1105 02:28:57.157] Latency metrics for node k8s-master-15826033-0 I1105 02:28:57.157] Nov 5 02:28:57.156: INFO: I1105 02:28:57.157] Logging node info for node k8s-master-15826033-1 I1105 02:28:57.236] Nov 5 02:28:57.234: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-15826033-1,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-15826033-1,UID:1a13ec61-c89a-456c-8c94-bfc8e717c2e7,ResourceVersion:10741,Generation:0,CreationTimestamp:2019-11-05 01:35:07 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 1,kubernetes.azure.com/cluster: kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-15826033-1,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.5.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5/providers/Microsoft.Compute/virtualMachines/k8s-master-15826033-1,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 01:35:31 +0000 UTC 2019-11-05 01:35:31 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 02:28:34 +0000 UTC 2019-11-05 01:35:00 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 02:28:34 +0000 UTC 2019-11-05 01:35:00 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 02:28:34 +0000 UTC 2019-11-05 01:35:00 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 02:28:34 +0000 UTC 2019-11-05 01:35:07 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.255.6} {Hostname k8s-master-15826033-1}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:e0f2bccddd5b4555b180d14081ab59a8,SystemUUID:AE126631-CAD2-B54F-B22E-1F499342A4D6,BootID:509b9a9d-b046-4861-aa35-8bcb77465de2,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 02:28:57.237] Nov 5 02:28:57.234: INFO: I1105 02:28:57.237] Logging kubelet events for node k8s-master-15826033-1 I1105 02:28:57.315] Nov 5 02:28:57.315: INFO: I1105 02:28:57.316] Logging pods the kubelet thinks is on node k8s-master-15826033-1 I1105 02:28:57.396] Nov 5 02:28:57.395: INFO: azure-ip-masq-agent-w6dpr started at 2019-11-05 01:35:10 +0000 UTC (0+1 container statuses recorded) I1105 02:28:57.396] Nov 5 02:28:57.395: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 02:28:57.396] Nov 5 02:28:57.395: INFO: kube-proxy-8kmr5 started at 2019-11-05 01:35:10 +0000 UTC (0+1 container statuses recorded) I1105 02:28:57.396] Nov 5 02:28:57.395: INFO: Container kube-proxy ready: true, restart count 0 I1105 02:28:57.396] Nov 5 02:28:57.395: INFO: cloud-controller-manager-k8s-master-15826033-1 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:28:57.397] Nov 5 02:28:57.395: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 02:28:57.397] Nov 5 02:28:57.395: INFO: kube-addon-manager-k8s-master-15826033-1 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:28:57.397] Nov 5 02:28:57.395: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 02:28:57.397] Nov 5 02:28:57.395: INFO: kube-apiserver-k8s-master-15826033-1 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:28:57.397] Nov 5 02:28:57.395: INFO: Container kube-apiserver ready: true, restart count 0 I1105 02:28:57.397] Nov 5 02:28:57.395: INFO: kube-controller-manager-k8s-master-15826033-1 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:28:57.398] Nov 5 02:28:57.395: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 02:28:57.398] Nov 5 02:28:57.395: INFO: kube-scheduler-k8s-master-15826033-1 started at 2019-11-05 01:36:16 +0000 UTC (0+1 container statuses recorded) I1105 02:28:57.398] Nov 5 02:28:57.396: INFO: Container kube-scheduler ready: true, restart count 0 I1105 02:28:57.646] Nov 5 02:28:57.646: INFO: I1105 02:28:57.647] Latency metrics for node k8s-master-15826033-1 I1105 02:28:57.647] Nov 5 02:28:57.646: INFO: I1105 02:28:57.647] Logging node info for node k8s-master-15826033-2 I1105 02:28:57.726] Nov 5 02:28:57.724: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-15826033-2,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-15826033-2,UID:acb3a851-d6df-4549-ba56-4b7e96553c17,ResourceVersion:10656,Generation:0,CreationTimestamp:2019-11-05 01:34:09 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 2,kubernetes.azure.com/cluster: kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-15826033-2,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.1.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5/providers/Microsoft.Compute/virtualMachines/k8s-master-15826033-2,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 01:34:44 +0000 UTC 2019-11-05 01:34:44 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 02:28:05 +0000 UTC 2019-11-05 01:34:07 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 02:28:05 +0000 UTC 2019-11-05 01:34:07 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 02:28:05 +0000 UTC 2019-11-05 01:34:07 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 02:28:05 +0000 UTC 2019-11-05 01:34:29 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{Hostname k8s-master-15826033-2} {InternalIP 10.240.255.7}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:0885b0c2f6174106bde4fa0eae2cbc6b,SystemUUID:C176121C-5D7A-6243-9B05-8B68F8022B64,BootID:30430638-7ab6-4761-b984-2387e1a3f5b2,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 02:28:57.727] Nov 5 02:28:57.724: INFO: I1105 02:28:57.727] Logging kubelet events for node k8s-master-15826033-2 I1105 02:28:57.806] Nov 5 02:28:57.806: INFO: I1105 02:28:57.806] Logging pods the kubelet thinks is on node k8s-master-15826033-2 I1105 02:28:57.888] Nov 5 02:28:57.888: INFO: cloud-controller-manager-k8s-master-15826033-2 started at 2019-11-05 01:33:54 +0000 UTC (0+1 container statuses recorded) I1105 02:28:57.888] Nov 5 02:28:57.888: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 02:28:57.888] Nov 5 02:28:57.888: INFO: kube-addon-manager-k8s-master-15826033-2 started at 2019-11-05 01:33:54 +0000 UTC (0+1 container statuses recorded) I1105 02:28:57.889] Nov 5 02:28:57.888: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 02:28:57.889] Nov 5 02:28:57.888: INFO: kube-apiserver-k8s-master-15826033-2 started at 2019-11-05 01:33:54 +0000 UTC (0+1 container statuses recorded) I1105 02:28:57.889] Nov 5 02:28:57.888: INFO: Container kube-apiserver ready: true, restart count 0 I1105 02:28:57.889] Nov 5 02:28:57.888: INFO: kube-controller-manager-k8s-master-15826033-2 started at 2019-11-05 01:33:54 +0000 UTC (0+1 container statuses recorded) I1105 02:28:57.889] Nov 5 02:28:57.888: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 02:28:57.889] Nov 5 02:28:57.888: INFO: kube-scheduler-k8s-master-15826033-2 started at 2019-11-05 01:33:54 +0000 UTC (0+1 container statuses recorded) I1105 02:28:57.890] Nov 5 02:28:57.888: INFO: Container kube-scheduler ready: true, restart count 0 I1105 02:28:57.890] Nov 5 02:28:57.888: INFO: azure-ip-masq-agent-hw7s5 started at 2019-11-05 01:34:34 +0000 UTC (0+1 container statuses recorded) I1105 02:28:57.890] Nov 5 02:28:57.888: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 02:28:57.890] Nov 5 02:28:57.888: INFO: kube-proxy-5xpp6 started at 2019-11-05 01:34:34 +0000 UTC (0+1 container statuses recorded) I1105 02:28:57.890] Nov 5 02:28:57.888: INFO: Container kube-proxy ready: true, restart count 0 I1105 02:28:58.139] Nov 5 02:28:58.138: INFO: I1105 02:28:58.139] Latency metrics for node k8s-master-15826033-2 I1105 02:28:58.139] Nov 5 02:28:58.138: INFO: I1105 02:28:58.139] Logging node info for node k8s-master-15826033-3 I1105 02:28:58.218] Nov 5 02:28:58.215: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-15826033-3,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-15826033-3,UID:b7702272-1876-434e-b6a0-17fa34c6757f,ResourceVersion:10737,Generation:0,CreationTimestamp:2019-11-05 01:35:06 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 0,kubernetes.azure.com/cluster: kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-15826033-3,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.3.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5/providers/Microsoft.Compute/virtualMachines/k8s-master-15826033-3,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 01:35:17 +0000 UTC 2019-11-05 01:35:17 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 02:28:33 +0000 UTC 2019-11-05 01:35:00 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 02:28:33 +0000 UTC 2019-11-05 01:35:00 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 02:28:33 +0000 UTC 2019-11-05 01:35:00 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 02:28:33 +0000 UTC 2019-11-05 01:35:06 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.255.8} {Hostname k8s-master-15826033-3}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:fbf366963961422c9d90bedf6639c8e1,SystemUUID:5388948D-1A8D-EA41-B8B8-7F4523EDF5AA,BootID:42f5884c-7418-486a-8cf3-75188a331f38,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 02:28:58.218] Nov 5 02:28:58.216: INFO: I1105 02:28:58.218] Logging kubelet events for node k8s-master-15826033-3 I1105 02:28:58.297] Nov 5 02:28:58.297: INFO: I1105 02:28:58.297] Logging pods the kubelet thinks is on node k8s-master-15826033-3 I1105 02:28:58.379] Nov 5 02:28:58.378: INFO: kube-addon-manager-k8s-master-15826033-3 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:28:58.379] Nov 5 02:28:58.379: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 02:28:58.379] Nov 5 02:28:58.379: INFO: kube-apiserver-k8s-master-15826033-3 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:28:58.379] Nov 5 02:28:58.379: INFO: Container kube-apiserver ready: true, restart count 0 I1105 02:28:58.380] Nov 5 02:28:58.379: INFO: kube-controller-manager-k8s-master-15826033-3 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:28:58.380] Nov 5 02:28:58.379: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 02:28:58.380] Nov 5 02:28:58.379: INFO: kube-scheduler-k8s-master-15826033-3 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:28:58.380] Nov 5 02:28:58.379: INFO: Container kube-scheduler ready: true, restart count 0 I1105 02:28:58.380] Nov 5 02:28:58.379: INFO: kube-proxy-xz24z started at 2019-11-05 01:35:06 +0000 UTC (0+1 container statuses recorded) I1105 02:28:58.380] Nov 5 02:28:58.379: INFO: Container kube-proxy ready: true, restart count 0 I1105 02:28:58.381] Nov 5 02:28:58.379: INFO: azure-ip-masq-agent-dqfpw started at 2019-11-05 01:35:06 +0000 UTC (0+1 container statuses recorded) I1105 02:28:58.381] Nov 5 02:28:58.379: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 02:28:58.381] Nov 5 02:28:58.379: INFO: cloud-controller-manager-k8s-master-15826033-3 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:28:58.381] Nov 5 02:28:58.379: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 02:28:58.627] Nov 5 02:28:58.627: INFO: I1105 02:28:58.627] Latency metrics for node k8s-master-15826033-3 I1105 02:28:58.628] Nov 5 02:28:58.627: INFO: I1105 02:28:58.628] Logging node info for node k8s-master-15826033-4 I1105 02:28:58.707] Nov 5 02:28:58.704: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-15826033-4,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-15826033-4,UID:276daef1-6c05-41e3-bacd-abc93d35cc6d,ResourceVersion:10743,Generation:0,CreationTimestamp:2019-11-05 01:35:07 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 0,kubernetes.azure.com/cluster: kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-15826033-4,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.6.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5/providers/Microsoft.Compute/virtualMachines/k8s-master-15826033-4,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 01:35:41 +0000 UTC 2019-11-05 01:35:41 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 02:28:34 +0000 UTC 2019-11-05 01:35:02 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 02:28:34 +0000 UTC 2019-11-05 01:35:02 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 02:28:34 +0000 UTC 2019-11-05 01:35:02 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 02:28:34 +0000 UTC 2019-11-05 01:35:08 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.255.9} {Hostname k8s-master-15826033-4}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:79051fc0bdeb4e32a56acbaf5972ce3a,SystemUUID:C71606B5-7F08-9341-A26D-8A8D9D84FCAE,BootID:4bdf7cf3-2b83-42c7-a7ba-926ebe9ca75b,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 02:28:58.707] Nov 5 02:28:58.704: INFO: I1105 02:28:58.707] Logging kubelet events for node k8s-master-15826033-4 I1105 02:28:58.786] Nov 5 02:28:58.786: INFO: I1105 02:28:58.786] Logging pods the kubelet thinks is on node k8s-master-15826033-4 I1105 02:28:58.868] Nov 5 02:28:58.868: INFO: kube-addon-manager-k8s-master-15826033-4 started at 2019-11-05 01:34:56 +0000 UTC (0+1 container statuses recorded) I1105 02:28:58.868] Nov 5 02:28:58.868: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 02:28:58.868] Nov 5 02:28:58.868: INFO: azure-ip-masq-agent-9nrj8 started at 2019-11-05 01:35:11 +0000 UTC (0+1 container statuses recorded) I1105 02:28:58.869] Nov 5 02:28:58.868: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 02:28:58.869] Nov 5 02:28:58.868: INFO: kube-proxy-rmngk started at 2019-11-05 01:35:11 +0000 UTC (0+1 container statuses recorded) I1105 02:28:58.869] Nov 5 02:28:58.868: INFO: Container kube-proxy ready: true, restart count 0 I1105 02:28:58.869] Nov 5 02:28:58.868: INFO: kube-apiserver-k8s-master-15826033-4 started at 2019-11-05 01:34:55 +0000 UTC (0+1 container statuses recorded) I1105 02:28:58.869] Nov 5 02:28:58.868: INFO: Container kube-apiserver ready: true, restart count 0 I1105 02:28:58.869] Nov 5 02:28:58.868: INFO: kube-controller-manager-k8s-master-15826033-4 started at 2019-11-05 01:34:55 +0000 UTC (0+1 container statuses recorded) I1105 02:28:58.870] Nov 5 02:28:58.868: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 02:28:58.870] Nov 5 02:28:58.868: INFO: kube-scheduler-k8s-master-15826033-4 started at 2019-11-05 01:34:55 +0000 UTC (0+1 container statuses recorded) I1105 02:28:58.870] Nov 5 02:28:58.868: INFO: Container kube-scheduler ready: true, restart count 0 I1105 02:28:58.870] Nov 5 02:28:58.868: INFO: cloud-controller-manager-k8s-master-15826033-4 started at 2019-11-05 01:34:56 +0000 UTC (0+1 container statuses recorded) I1105 02:28:58.870] Nov 5 02:28:58.868: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 02:28:59.120] Nov 5 02:28:59.120: INFO: I1105 02:28:59.120] Latency metrics for node k8s-master-15826033-4 I1105 02:28:59.120] Nov 5 02:28:59.120: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready I1105 02:28:59.199] STEP: Destroying namespace "azuredisk-7960" for this suite. I1105 02:29:13.437] Nov 5 02:29:13.436: INFO: Waiting up to 30s for server preferred namespaced resources to be successfully discovered I1105 02:29:16.363] Nov 5 02:29:16.362: INFO: namespace azuredisk-7960 deletion completed in 17.163684921s I1105 02:29:16.363] I1105 02:29:16.363] • Failure [323.250 seconds] I1105 02:29:16.363] [azuredisk-csi-e2e] Dynamic Provisioning I1105 02:29:16.363] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:33 I1105 02:29:16.364] [multi-az] I1105 02:29:16.364] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:40 I1105 02:29:16.364] [env] should retain PV with reclaimPolicy "Retain" [It] I1105 02:29:16.364] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:240 I1105 02:29:16.364] I1105 02:29:16.365] Unexpected error: I1105 02:29:16.365] <*errors.errorString | 0xc000ca4c30>: { I1105 02:29:16.365] s: "PersistentVolumeClaims [pvc-4vz8h] not all in phase Bound within 5m0s", I1105 02:29:16.365] } I1105 02:29:16.365] PersistentVolumeClaims [pvc-4vz8h] not all in phase Bound within 5m0s I1105 02:29:16.365] occurred I1105 02:29:16.365] I1105 02:29:16.366] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/testsuites/testsuites.go:278 I1105 02:29:16.366] ------------------------------ I1105 02:29:16.366] [azuredisk-csi-e2e] Dynamic Provisioning [multi-az] I1105 02:29:16.366] should create a deployment object, write and read to it, delete the pod and write and read to it again I1105 02:29:16.366] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:193 I1105 02:29:16.366] [BeforeEach] [multi-az] I1105 02:29:16.366] /go/src/sigs.k8s.io/azuredisk-csi-driver/vendor/k8s.io/kubernetes/test/e2e/framework/framework.go:150 I1105 02:29:16.367] STEP: Creating a kubernetes client I1105 02:29:16.367] Nov 5 02:29:16.362: INFO: >>> kubeConfig: /workspace/aks978705698/kubeconfig/kubeconfig.eastus2.json I1105 02:29:16.367] STEP: Building a namespace api object, basename azuredisk I1105 02:29:16.613] STEP: Binding the e2e-test-privileged-psp PodSecurityPolicy to the default service account in azuredisk-7865 I1105 02:29:16.873] STEP: Waiting for a default service account to be provisioned in namespace I1105 02:29:16.949] [BeforeEach] [multi-az] I1105 02:29:16.949] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:58 I1105 02:29:16.949] [It] should create a deployment object, write and read to it, delete the pod and write and read to it again I1105 02:29:16.949] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:193 I1105 02:29:16.949] STEP: setting up the StorageClass I1105 02:29:16.949] STEP: creating a StorageClass I1105 02:29:17.029] STEP: setting up the PVC I1105 02:29:17.030] STEP: creating a PVC I1105 02:29:17.162] STEP: waiting for PVC to be in phase "Bound" I1105 02:29:17.162] Nov 5 02:29:17.161: INFO: Waiting up to 5m0s for PersistentVolumeClaims [pvc-sjl96] to have phase Bound I1105 02:29:17.239] Nov 5 02:29:17.239: INFO: PersistentVolumeClaim pvc-sjl96 found but phase is Pending instead of Bound. I1105 02:29:19.317] Nov 5 02:29:19.317: INFO: PersistentVolumeClaim pvc-sjl96 found but phase is Pending instead of Bound. I1105 02:29:21.395] Nov 5 02:29:21.395: INFO: PersistentVolumeClaim pvc-sjl96 found but phase is Pending instead of Bound. I1105 02:29:23.473] Nov 5 02:29:23.473: INFO: PersistentVolumeClaim pvc-sjl96 found but phase is Pending instead of Bound. I1105 02:29:25.552] Nov 5 02:29:25.552: INFO: PersistentVolumeClaim pvc-sjl96 found but phase is Pending instead of Bound. I1105 02:29:27.630] Nov 5 02:29:27.630: INFO: PersistentVolumeClaim pvc-sjl96 found but phase is Pending instead of Bound. I1105 02:29:29.708] Nov 5 02:29:29.708: INFO: PersistentVolumeClaim pvc-sjl96 found but phase is Pending instead of Bound. I1105 02:29:31.786] Nov 5 02:29:31.786: INFO: PersistentVolumeClaim pvc-sjl96 found but phase is Pending instead of Bound. I1105 02:29:33.864] Nov 5 02:29:33.864: INFO: PersistentVolumeClaim pvc-sjl96 found but phase is Pending instead of Bound. I1105 02:29:35.942] Nov 5 02:29:35.942: INFO: PersistentVolumeClaim pvc-sjl96 found but phase is Pending instead of Bound. I1105 02:29:38.020] Nov 5 02:29:38.020: INFO: PersistentVolumeClaim pvc-sjl96 found but phase is Pending instead of Bound. I1105 02:29:40.098] Nov 5 02:29:40.098: INFO: PersistentVolumeClaim pvc-sjl96 found but phase is Pending instead of Bound. I1105 02:29:42.180] Nov 5 02:29:42.179: INFO: PersistentVolumeClaim pvc-sjl96 found but phase is Pending instead of Bound. I1105 02:29:44.257] Nov 5 02:29:44.257: INFO: PersistentVolumeClaim pvc-sjl96 found but phase is Pending instead of Bound. I1105 02:29:46.335] Nov 5 02:29:46.335: INFO: PersistentVolumeClaim pvc-sjl96 found but phase is Pending instead of Bound. I1105 02:29:48.418] Nov 5 02:29:48.417: INFO: PersistentVolumeClaim pvc-sjl96 found but phase is Pending instead of Bound. I1105 02:29:50.495] Nov 5 02:29:50.495: INFO: PersistentVolumeClaim pvc-sjl96 found but phase is Pending instead of Bound. I1105 02:29:52.573] Nov 5 02:29:52.573: INFO: PersistentVolumeClaim pvc-sjl96 found but phase is Pending instead of Bound. I1105 02:29:54.651] Nov 5 02:29:54.650: INFO: PersistentVolumeClaim pvc-sjl96 found but phase is Pending instead of Bound. I1105 02:29:56.728] Nov 5 02:29:56.728: INFO: PersistentVolumeClaim pvc-sjl96 found but phase is Pending instead of Bound. I1105 02:29:58.806] Nov 5 02:29:58.806: INFO: PersistentVolumeClaim pvc-sjl96 found but phase is Pending instead of Bound. I1105 02:30:00.883] Nov 5 02:30:00.883: INFO: PersistentVolumeClaim pvc-sjl96 found but phase is Pending instead of Bound. I1105 02:30:02.961] Nov 5 02:30:02.961: INFO: PersistentVolumeClaim pvc-sjl96 found but phase is Pending instead of Bound. I1105 02:30:05.039] Nov 5 02:30:05.039: INFO: PersistentVolumeClaim pvc-sjl96 found but phase is Pending instead of Bound. I1105 02:30:07.117] Nov 5 02:30:07.116: INFO: PersistentVolumeClaim pvc-sjl96 found but phase is Pending instead of Bound. I1105 02:30:09.194] Nov 5 02:30:09.194: INFO: PersistentVolumeClaim pvc-sjl96 found but phase is Pending instead of Bound. I1105 02:30:11.272] Nov 5 02:30:11.272: INFO: PersistentVolumeClaim pvc-sjl96 found but phase is Pending instead of Bound. I1105 02:30:13.350] Nov 5 02:30:13.350: INFO: PersistentVolumeClaim pvc-sjl96 found but phase is Pending instead of Bound. I1105 02:30:15.427] Nov 5 02:30:15.427: INFO: PersistentVolumeClaim pvc-sjl96 found but phase is Pending instead of Bound. I1105 02:30:17.505] Nov 5 02:30:17.505: INFO: PersistentVolumeClaim pvc-sjl96 found but phase is Pending instead of Bound. I1105 02:30:19.583] Nov 5 02:30:19.583: INFO: PersistentVolumeClaim pvc-sjl96 found but phase is Pending instead of Bound. I1105 02:30:21.660] Nov 5 02:30:21.660: INFO: PersistentVolumeClaim pvc-sjl96 found but phase is Pending instead of Bound. I1105 02:30:23.738] Nov 5 02:30:23.738: INFO: PersistentVolumeClaim pvc-sjl96 found but phase is Pending instead of Bound. I1105 02:30:25.816] Nov 5 02:30:25.816: INFO: PersistentVolumeClaim pvc-sjl96 found but phase is Pending instead of Bound. I1105 02:30:27.894] Nov 5 02:30:27.894: INFO: PersistentVolumeClaim pvc-sjl96 found but phase is Pending instead of Bound. I1105 02:30:29.972] Nov 5 02:30:29.971: INFO: PersistentVolumeClaim pvc-sjl96 found but phase is Pending instead of Bound. I1105 02:30:32.049] Nov 5 02:30:32.049: INFO: PersistentVolumeClaim pvc-sjl96 found but phase is Pending instead of Bound. I1105 02:30:34.126] Nov 5 02:30:34.126: INFO: PersistentVolumeClaim pvc-sjl96 found but phase is Pending instead of Bound. I1105 02:30:36.203] Nov 5 02:30:36.203: INFO: PersistentVolumeClaim pvc-sjl96 found but phase is Pending instead of Bound. I1105 02:30:38.281] Nov 5 02:30:38.281: INFO: PersistentVolumeClaim pvc-sjl96 found but phase is Pending instead of Bound. I1105 02:30:40.359] Nov 5 02:30:40.359: INFO: PersistentVolumeClaim pvc-sjl96 found but phase is Pending instead of Bound. I1105 02:30:42.436] Nov 5 02:30:42.436: INFO: PersistentVolumeClaim pvc-sjl96 found but phase is Pending instead of Bound. I1105 02:30:44.515] Nov 5 02:30:44.515: INFO: PersistentVolumeClaim pvc-sjl96 found but phase is Pending instead of Bound. I1105 02:30:46.593] Nov 5 02:30:46.592: INFO: PersistentVolumeClaim pvc-sjl96 found but phase is Pending instead of Bound. I1105 02:30:48.670] Nov 5 02:30:48.670: INFO: PersistentVolumeClaim pvc-sjl96 found but phase is Pending instead of Bound. I1105 02:30:50.748] Nov 5 02:30:50.748: INFO: PersistentVolumeClaim pvc-sjl96 found but phase is Pending instead of Bound. I1105 02:30:52.828] Nov 5 02:30:52.828: INFO: PersistentVolumeClaim pvc-sjl96 found but phase is Pending instead of Bound. I1105 02:30:54.906] Nov 5 02:30:54.905: INFO: PersistentVolumeClaim pvc-sjl96 found but phase is Pending instead of Bound. I1105 02:30:56.983] Nov 5 02:30:56.983: INFO: PersistentVolumeClaim pvc-sjl96 found but phase is Pending instead of Bound. I1105 02:30:59.060] Nov 5 02:30:59.060: INFO: PersistentVolumeClaim pvc-sjl96 found but phase is Pending instead of Bound. I1105 02:31:01.138] Nov 5 02:31:01.138: INFO: PersistentVolumeClaim pvc-sjl96 found but phase is Pending instead of Bound. I1105 02:31:03.216] Nov 5 02:31:03.216: INFO: PersistentVolumeClaim pvc-sjl96 found but phase is Pending instead of Bound. I1105 02:31:05.297] Nov 5 02:31:05.296: INFO: PersistentVolumeClaim pvc-sjl96 found but phase is Pending instead of Bound. I1105 02:31:07.374] Nov 5 02:31:07.374: INFO: PersistentVolumeClaim pvc-sjl96 found but phase is Pending instead of Bound. I1105 02:31:09.453] Nov 5 02:31:09.453: INFO: PersistentVolumeClaim pvc-sjl96 found but phase is Pending instead of Bound. I1105 02:31:11.531] Nov 5 02:31:11.530: INFO: PersistentVolumeClaim pvc-sjl96 found but phase is Pending instead of Bound. I1105 02:31:13.608] Nov 5 02:31:13.608: INFO: PersistentVolumeClaim pvc-sjl96 found but phase is Pending instead of Bound. I1105 02:31:15.685] Nov 5 02:31:15.685: INFO: PersistentVolumeClaim pvc-sjl96 found but phase is Pending instead of Bound. I1105 02:31:17.764] Nov 5 02:31:17.763: INFO: PersistentVolumeClaim pvc-sjl96 found but phase is Pending instead of Bound. I1105 02:31:19.841] Nov 5 02:31:19.841: INFO: PersistentVolumeClaim pvc-sjl96 found but phase is Pending instead of Bound. I1105 02:31:21.918] Nov 5 02:31:21.918: INFO: PersistentVolumeClaim pvc-sjl96 found but phase is Pending instead of Bound. I1105 02:31:23.998] Nov 5 02:31:23.997: INFO: PersistentVolumeClaim pvc-sjl96 found but phase is Pending instead of Bound. I1105 02:31:26.075] Nov 5 02:31:26.075: INFO: PersistentVolumeClaim pvc-sjl96 found but phase is Pending instead of Bound. I1105 02:31:28.153] Nov 5 02:31:28.153: INFO: PersistentVolumeClaim pvc-sjl96 found but phase is Pending instead of Bound. I1105 02:31:30.231] Nov 5 02:31:30.230: INFO: PersistentVolumeClaim pvc-sjl96 found but phase is Pending instead of Bound. I1105 02:31:32.308] Nov 5 02:31:32.308: INFO: PersistentVolumeClaim pvc-sjl96 found but phase is Pending instead of Bound. I1105 02:31:34.389] Nov 5 02:31:34.389: INFO: PersistentVolumeClaim pvc-sjl96 found but phase is Pending instead of Bound. I1105 02:31:36.467] Nov 5 02:31:36.467: INFO: PersistentVolumeClaim pvc-sjl96 found but phase is Pending instead of Bound. I1105 02:31:38.545] Nov 5 02:31:38.545: INFO: PersistentVolumeClaim pvc-sjl96 found but phase is Pending instead of Bound. I1105 02:31:40.622] Nov 5 02:31:40.622: INFO: PersistentVolumeClaim pvc-sjl96 found but phase is Pending instead of Bound. I1105 02:31:42.703] Nov 5 02:31:42.703: INFO: PersistentVolumeClaim pvc-sjl96 found but phase is Pending instead of Bound. I1105 02:31:44.781] Nov 5 02:31:44.781: INFO: PersistentVolumeClaim pvc-sjl96 found but phase is Pending instead of Bound. I1105 02:31:46.858] Nov 5 02:31:46.858: INFO: PersistentVolumeClaim pvc-sjl96 found but phase is Pending instead of Bound. I1105 02:31:48.936] Nov 5 02:31:48.935: INFO: PersistentVolumeClaim pvc-sjl96 found but phase is Pending instead of Bound. I1105 02:31:51.013] Nov 5 02:31:51.012: INFO: PersistentVolumeClaim pvc-sjl96 found but phase is Pending instead of Bound. I1105 02:31:53.128] Nov 5 02:31:53.128: INFO: PersistentVolumeClaim pvc-sjl96 found but phase is Pending instead of Bound. I1105 02:31:55.206] Nov 5 02:31:55.206: INFO: PersistentVolumeClaim pvc-sjl96 found but phase is Pending instead of Bound. I1105 02:31:57.284] Nov 5 02:31:57.284: INFO: PersistentVolumeClaim pvc-sjl96 found but phase is Pending instead of Bound. I1105 02:31:59.361] Nov 5 02:31:59.361: INFO: PersistentVolumeClaim pvc-sjl96 found but phase is Pending instead of Bound. I1105 02:32:01.439] Nov 5 02:32:01.439: INFO: PersistentVolumeClaim pvc-sjl96 found but phase is Pending instead of Bound. I1105 02:32:03.517] Nov 5 02:32:03.517: INFO: PersistentVolumeClaim pvc-sjl96 found but phase is Pending instead of Bound. I1105 02:32:05.595] Nov 5 02:32:05.595: INFO: PersistentVolumeClaim pvc-sjl96 found but phase is Pending instead of Bound. I1105 02:32:07.673] Nov 5 02:32:07.672: INFO: PersistentVolumeClaim pvc-sjl96 found but phase is Pending instead of Bound. I1105 02:32:09.757] Nov 5 02:32:09.757: INFO: PersistentVolumeClaim pvc-sjl96 found but phase is Pending instead of Bound. I1105 02:32:11.836] Nov 5 02:32:11.835: INFO: PersistentVolumeClaim pvc-sjl96 found but phase is Pending instead of Bound. I1105 02:32:13.914] Nov 5 02:32:13.913: INFO: PersistentVolumeClaim pvc-sjl96 found but phase is Pending instead of Bound. I1105 02:32:15.997] Nov 5 02:32:15.997: INFO: PersistentVolumeClaim pvc-sjl96 found but phase is Pending instead of Bound. I1105 02:32:18.074] Nov 5 02:32:18.074: INFO: PersistentVolumeClaim pvc-sjl96 found but phase is Pending instead of Bound. I1105 02:32:20.156] Nov 5 02:32:20.156: INFO: PersistentVolumeClaim pvc-sjl96 found but phase is Pending instead of Bound. I1105 02:32:22.233] Nov 5 02:32:22.233: INFO: PersistentVolumeClaim pvc-sjl96 found but phase is Pending instead of Bound. I1105 02:32:24.311] Nov 5 02:32:24.311: INFO: PersistentVolumeClaim pvc-sjl96 found but phase is Pending instead of Bound. I1105 02:32:26.388] Nov 5 02:32:26.388: INFO: PersistentVolumeClaim pvc-sjl96 found but phase is Pending instead of Bound. I1105 02:32:28.466] Nov 5 02:32:28.465: INFO: PersistentVolumeClaim pvc-sjl96 found but phase is Pending instead of Bound. I1105 02:32:30.543] Nov 5 02:32:30.543: INFO: PersistentVolumeClaim pvc-sjl96 found but phase is Pending instead of Bound. I1105 02:32:32.621] Nov 5 02:32:32.621: INFO: PersistentVolumeClaim pvc-sjl96 found but phase is Pending instead of Bound. I1105 02:32:34.699] Nov 5 02:32:34.699: INFO: PersistentVolumeClaim pvc-sjl96 found but phase is Pending instead of Bound. I1105 02:32:36.776] Nov 5 02:32:36.776: INFO: PersistentVolumeClaim pvc-sjl96 found but phase is Pending instead of Bound. I1105 02:32:38.854] Nov 5 02:32:38.854: INFO: PersistentVolumeClaim pvc-sjl96 found but phase is Pending instead of Bound. I1105 02:32:40.932] Nov 5 02:32:40.931: INFO: PersistentVolumeClaim pvc-sjl96 found but phase is Pending instead of Bound. I1105 02:32:43.009] Nov 5 02:32:43.009: INFO: PersistentVolumeClaim pvc-sjl96 found but phase is Pending instead of Bound. I1105 02:32:45.131] Nov 5 02:32:45.131: INFO: PersistentVolumeClaim pvc-sjl96 found but phase is Pending instead of Bound. I1105 02:32:47.208] Nov 5 02:32:47.208: INFO: PersistentVolumeClaim pvc-sjl96 found but phase is Pending instead of Bound. I1105 02:32:49.286] Nov 5 02:32:49.286: INFO: PersistentVolumeClaim pvc-sjl96 found but phase is Pending instead of Bound. I1105 02:32:51.364] Nov 5 02:32:51.363: INFO: PersistentVolumeClaim pvc-sjl96 found but phase is Pending instead of Bound. I1105 02:32:53.450] Nov 5 02:32:53.449: INFO: PersistentVolumeClaim pvc-sjl96 found but phase is Pending instead of Bound. I1105 02:32:55.527] Nov 5 02:32:55.527: INFO: PersistentVolumeClaim pvc-sjl96 found but phase is Pending instead of Bound. I1105 02:32:57.605] Nov 5 02:32:57.605: INFO: PersistentVolumeClaim pvc-sjl96 found but phase is Pending instead of Bound. I1105 02:32:59.683] Nov 5 02:32:59.683: INFO: PersistentVolumeClaim pvc-sjl96 found but phase is Pending instead of Bound. I1105 02:33:01.760] Nov 5 02:33:01.760: INFO: PersistentVolumeClaim pvc-sjl96 found but phase is Pending instead of Bound. I1105 02:33:03.838] Nov 5 02:33:03.838: INFO: PersistentVolumeClaim pvc-sjl96 found but phase is Pending instead of Bound. I1105 02:33:05.915] Nov 5 02:33:05.915: INFO: PersistentVolumeClaim pvc-sjl96 found but phase is Pending instead of Bound. I1105 02:33:07.993] Nov 5 02:33:07.993: INFO: PersistentVolumeClaim pvc-sjl96 found but phase is Pending instead of Bound. I1105 02:33:10.071] Nov 5 02:33:10.071: INFO: PersistentVolumeClaim pvc-sjl96 found but phase is Pending instead of Bound. I1105 02:33:12.149] Nov 5 02:33:12.149: INFO: PersistentVolumeClaim pvc-sjl96 found but phase is Pending instead of Bound. I1105 02:33:14.226] Nov 5 02:33:14.226: INFO: PersistentVolumeClaim pvc-sjl96 found but phase is Pending instead of Bound. I1105 02:33:16.304] Nov 5 02:33:16.304: INFO: PersistentVolumeClaim pvc-sjl96 found but phase is Pending instead of Bound. I1105 02:33:18.385] Nov 5 02:33:18.385: INFO: PersistentVolumeClaim pvc-sjl96 found but phase is Pending instead of Bound. I1105 02:33:20.463] Nov 5 02:33:20.462: INFO: PersistentVolumeClaim pvc-sjl96 found but phase is Pending instead of Bound. I1105 02:33:22.542] Nov 5 02:33:22.542: INFO: PersistentVolumeClaim pvc-sjl96 found but phase is Pending instead of Bound. I1105 02:33:24.619] Nov 5 02:33:24.619: INFO: PersistentVolumeClaim pvc-sjl96 found but phase is Pending instead of Bound. I1105 02:33:26.697] Nov 5 02:33:26.696: INFO: PersistentVolumeClaim pvc-sjl96 found but phase is Pending instead of Bound. I1105 02:33:28.774] Nov 5 02:33:28.774: INFO: PersistentVolumeClaim pvc-sjl96 found but phase is Pending instead of Bound. I1105 02:33:30.851] Nov 5 02:33:30.851: INFO: PersistentVolumeClaim pvc-sjl96 found but phase is Pending instead of Bound. I1105 02:33:32.928] Nov 5 02:33:32.928: INFO: PersistentVolumeClaim pvc-sjl96 found but phase is Pending instead of Bound. I1105 02:33:35.006] Nov 5 02:33:35.006: INFO: PersistentVolumeClaim pvc-sjl96 found but phase is Pending instead of Bound. I1105 02:33:37.095] Nov 5 02:33:37.095: INFO: PersistentVolumeClaim pvc-sjl96 found but phase is Pending instead of Bound. I1105 02:33:39.173] Nov 5 02:33:39.173: INFO: PersistentVolumeClaim pvc-sjl96 found but phase is Pending instead of Bound. I1105 02:33:41.250] Nov 5 02:33:41.250: INFO: PersistentVolumeClaim pvc-sjl96 found but phase is Pending instead of Bound. I1105 02:33:43.327] Nov 5 02:33:43.327: INFO: PersistentVolumeClaim pvc-sjl96 found but phase is Pending instead of Bound. I1105 02:33:45.405] Nov 5 02:33:45.404: INFO: PersistentVolumeClaim pvc-sjl96 found but phase is Pending instead of Bound. I1105 02:33:47.482] Nov 5 02:33:47.482: INFO: PersistentVolumeClaim pvc-sjl96 found but phase is Pending instead of Bound. I1105 02:33:49.560] Nov 5 02:33:49.560: INFO: PersistentVolumeClaim pvc-sjl96 found but phase is Pending instead of Bound. I1105 02:33:51.637] Nov 5 02:33:51.637: INFO: PersistentVolumeClaim pvc-sjl96 found but phase is Pending instead of Bound. I1105 02:33:53.715] Nov 5 02:33:53.715: INFO: PersistentVolumeClaim pvc-sjl96 found but phase is Pending instead of Bound. I1105 02:33:55.792] Nov 5 02:33:55.792: INFO: PersistentVolumeClaim pvc-sjl96 found but phase is Pending instead of Bound. I1105 02:33:57.870] Nov 5 02:33:57.870: INFO: PersistentVolumeClaim pvc-sjl96 found but phase is Pending instead of Bound. I1105 02:33:59.948] Nov 5 02:33:59.948: INFO: PersistentVolumeClaim pvc-sjl96 found but phase is Pending instead of Bound. I1105 02:34:02.026] Nov 5 02:34:02.026: INFO: PersistentVolumeClaim pvc-sjl96 found but phase is Pending instead of Bound. I1105 02:34:04.104] Nov 5 02:34:04.104: INFO: PersistentVolumeClaim pvc-sjl96 found but phase is Pending instead of Bound. I1105 02:34:06.182] Nov 5 02:34:06.181: INFO: PersistentVolumeClaim pvc-sjl96 found but phase is Pending instead of Bound. I1105 02:34:08.259] Nov 5 02:34:08.259: INFO: PersistentVolumeClaim pvc-sjl96 found but phase is Pending instead of Bound. I1105 02:34:10.337] Nov 5 02:34:10.336: INFO: PersistentVolumeClaim pvc-sjl96 found but phase is Pending instead of Bound. I1105 02:34:12.414] Nov 5 02:34:12.414: INFO: PersistentVolumeClaim pvc-sjl96 found but phase is Pending instead of Bound. I1105 02:34:14.492] Nov 5 02:34:14.491: INFO: PersistentVolumeClaim pvc-sjl96 found but phase is Pending instead of Bound. I1105 02:34:16.578] Nov 5 02:34:16.577: INFO: PersistentVolumeClaim pvc-sjl96 found but phase is Pending instead of Bound. I1105 02:34:18.578] Nov 5 02:34:18.578: INFO: Unexpected error occurred: PersistentVolumeClaims [pvc-sjl96] not all in phase Bound within 5m0s I1105 02:34:18.578] [AfterEach] [multi-az] I1105 02:34:18.578] /go/src/sigs.k8s.io/azuredisk-csi-driver/vendor/k8s.io/kubernetes/test/e2e/framework/framework.go:151 I1105 02:34:18.578] STEP: Collecting events from namespace "azuredisk-7865". I1105 02:34:18.656] STEP: Found 3 events. I1105 02:34:18.656] Nov 5 02:34:18.656: INFO: At 2019-11-05 02:29:17 +0000 UTC - event for pvc-sjl96: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "disk.csi.azure.com" or manually created by system administrator I1105 02:34:18.656] Nov 5 02:34:18.656: INFO: At 2019-11-05 02:29:17 +0000 UTC - event for pvc-sjl96: {disk.csi.azure.com_csi-azuredisk-controller-6f6587bfc6-8vkfl_1376914b-ff6d-11e9-93ab-ba2bee589b8c } Provisioning: External provisioner is provisioning volume for claim "azuredisk-7865/pvc-sjl96" I1105 02:34:18.657] Nov 5 02:34:18.656: INFO: At 2019-11-05 02:29:17 +0000 UTC - event for pvc-sjl96: {disk.csi.azure.com_csi-azuredisk-controller-6f6587bfc6-8vkfl_1376914b-ff6d-11e9-93ab-ba2bee589b8c } ProvisioningFailed: failed to provision volume with StorageClass "azuredisk-7865-disk.csi.azure.com-dynamic-sc-x26jr": rpc error: code = Unknown desc = compute.DisksClient#CreateOrUpdate: Failure sending request: StatusCode=400 -- Original Error: Code="InvalidAvailabilityZone" Message="The zone(s) '' for resource 'Microsoft.Compute/disks/pvc-07395b67-8969-4afb-b7ec-4fe833ff0e95' is not supported. The supported zones for location 'eastus2' are '1,2,3'" I1105 02:34:18.740] Nov 5 02:34:18.740: INFO: POD NODE PHASE GRACE CONDITIONS I1105 02:34:18.741] Nov 5 02:34:18.740: INFO: I1105 02:34:18.971] Nov 5 02:34:18.971: INFO: I1105 02:34:18.972] Logging node info for node k8s-agentpool-15826033-vmss000000 I1105 02:34:19.051] Nov 5 02:34:19.048: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-agentpool-15826033-vmss000000,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-agentpool-15826033-vmss000000,UID:15fa874e-54ec-42cd-b243-d3baa512150f,ResourceVersion:11615,Generation:0,CreationTimestamp:2019-11-05 01:34:17 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{agentpool: agentpool,beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_D4s_v3,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 0,kubernetes.azure.com/cluster: kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5,kubernetes.azure.com/role: agent,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-agentpool-15826033-vmss000000,kubernetes.io/os: linux,kubernetes.io/role: agent,node-role.kubernetes.io/agent: ,storageprofile: managed,storagetier: Premium_LRS,topology.disk.csi.azure.com/zone: 0,},Annotations:map[string]string{csi.volume.kubernetes.io/nodeid: {"disk.csi.azure.com":"k8s-agentpool-15826033-vmss000000"},node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.2.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool-15826033-vmss/virtualMachines/0,Unschedulable:false,Taints:[],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-csi-disk.csi.azure.com: {{8 0} {} 8 DecimalSI},cpu: {{4 0} {} 4 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{16797044736 0} {} 16403364Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-csi-disk.csi.azure.com: {{8 0} {} 8 DecimalSI},cpu: {{4 0} {} 4 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{16010612736 0} {} 15635364Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 01:34:56 +0000 UTC 2019-11-05 01:34:56 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 02:33:26 +0000 UTC 2019-11-05 01:34:17 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 02:33:26 +0000 UTC 2019-11-05 01:34:17 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 02:33:26 +0000 UTC 2019-11-05 01:34:17 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 02:33:26 +0000 UTC 2019-11-05 01:34:29 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.0.4} {Hostname k8s-agentpool-15826033-vmss000000}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:afe00792177d49a98f6a44d03d08732d,SystemUUID:4F1777BF-A5C3-004E-81C4-90F0B54C02A0,BootID:13b6a628-ae89-44af-b891-89461030adcb,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[k8sprowinternal.azurecr.io/azuredisk-csi@sha256:75d4ff5d7cd845d6e810c67796ecb037e852fb849ec1eb5b3194564008e6d1e0 k8sprowinternal.azurecr.io/azuredisk-csi:e2e-3e3ed045c6faa509dbd8d2890ee9f56aa2934887] 286930414} {[k8s.gcr.io/kubernetes-dashboard-amd64@sha256:0ae6b69432e78069c5ce2bcde0fe409c5c4d6f0f4d9cd50a17974fea38898747 k8s.gcr.io/kubernetes-dashboard-amd64:v1.10.1] 121711221} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/metrics-server-amd64@sha256:49a9f12f7067d11f42c803dbe61ed2c1299959ad85cb315b25ff7eef8e6b8892 k8s.gcr.io/metrics-server-amd64:v0.2.1] 42541759} {[mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume@sha256:4fd30d43947d4a54fc89ead7985beecfd3c9b2a93a0655a373b1608ab90bd5af mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume:v0.0.7] 22909487} {[quay.io/k8scsi/livenessprobe@sha256:8c9c8a0f330acc2e558ae6624309447d02522e76aa35af23fa19b88abf4381cb quay.io/k8scsi/livenessprobe:v1.0.2] 17415381} {[quay.io/k8scsi/csi-node-driver-registrar@sha256:13daf82fb99e951a4bff8ae5fc7c17c3a8fe7130be6400990d8f6076c32d4599 quay.io/k8scsi/csi-node-driver-registrar:v1.1.0] 15815995} {[mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume@sha256:23d8c6033f02a1ecad05127ebdc931bb871264228661bc122704b0974e4d9fdd mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume:1.0.8] 1159025} {[busybox@sha256:8ccbac733d19c0dd4d70b4f0c1e12245b5fa3ad24758a11035ee505c629c0796 busybox:1.29] 1154361} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 02:34:19.052] Nov 5 02:34:19.049: INFO: I1105 02:34:19.052] Logging kubelet events for node k8s-agentpool-15826033-vmss000000 I1105 02:34:19.132] Nov 5 02:34:19.132: INFO: I1105 02:34:19.133] Logging pods the kubelet thinks is on node k8s-agentpool-15826033-vmss000000 I1105 02:34:19.215] Nov 5 02:34:19.215: INFO: csi-azuredisk-node-whq2g started at 2019-11-05 01:38:24 +0000 UTC (0+3 container statuses recorded) I1105 02:34:19.215] Nov 5 02:34:19.215: INFO: Container azuredisk ready: true, restart count 0 I1105 02:34:19.216] Nov 5 02:34:19.215: INFO: Container liveness-probe ready: true, restart count 0 I1105 02:34:19.216] Nov 5 02:34:19.215: INFO: Container node-driver-registrar ready: true, restart count 0 I1105 02:34:19.216] Nov 5 02:34:19.215: INFO: blobfuse-flexvol-installer-bsvqt started at 2019-11-05 01:34:34 +0000 UTC (0+1 container statuses recorded) I1105 02:34:19.216] Nov 5 02:34:19.215: INFO: Container blobfuse-flexvol-installer ready: true, restart count 0 I1105 02:34:19.216] Nov 5 02:34:19.215: INFO: azure-ip-masq-agent-vx86b started at 2019-11-05 01:34:34 +0000 UTC (0+1 container statuses recorded) I1105 02:34:19.217] Nov 5 02:34:19.215: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 02:34:19.217] Nov 5 02:34:19.215: INFO: keyvault-flexvolume-qhnfr started at 2019-11-05 01:34:34 +0000 UTC (0+1 container statuses recorded) I1105 02:34:19.217] Nov 5 02:34:19.215: INFO: Container keyvault-flexvolume ready: true, restart count 0 I1105 02:34:19.218] Nov 5 02:34:19.215: INFO: metrics-server-864ffbc5c-sdxps started at 2019-11-05 01:34:34 +0000 UTC (0+1 container statuses recorded) I1105 02:34:19.218] Nov 5 02:34:19.215: INFO: Container metrics-server ready: true, restart count 0 I1105 02:34:19.218] Nov 5 02:34:19.215: INFO: kube-proxy-wxblc started at 2019-11-05 01:34:34 +0000 UTC (0+1 container statuses recorded) I1105 02:34:19.218] Nov 5 02:34:19.215: INFO: Container kube-proxy ready: true, restart count 0 I1105 02:34:19.219] Nov 5 02:34:19.215: INFO: kubernetes-dashboard-66dd8b8df7-ks284 started at 2019-11-05 01:34:35 +0000 UTC (0+1 container statuses recorded) I1105 02:34:19.219] Nov 5 02:34:19.215: INFO: Container kubernetes-dashboard ready: true, restart count 0 I1105 02:34:19.466] Nov 5 02:34:19.466: INFO: I1105 02:34:19.467] Latency metrics for node k8s-agentpool-15826033-vmss000000 I1105 02:34:19.467] Nov 5 02:34:19.466: INFO: I1105 02:34:19.467] Logging node info for node k8s-agentpool-15826033-vmss000001 I1105 02:34:19.547] Nov 5 02:34:19.543: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-agentpool-15826033-vmss000001,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-agentpool-15826033-vmss000001,UID:4a8ea605-cc0b-4e3b-9060-d8867d269e6c,ResourceVersion:11683,Generation:0,CreationTimestamp:2019-11-05 01:34:09 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{agentpool: agentpool,beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_D4s_v3,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 1,kubernetes.azure.com/cluster: kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5,kubernetes.azure.com/role: agent,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-agentpool-15826033-vmss000001,kubernetes.io/os: linux,kubernetes.io/role: agent,node-role.kubernetes.io/agent: ,storageprofile: managed,storagetier: Premium_LRS,topology.disk.csi.azure.com/zone: 1,},Annotations:map[string]string{csi.volume.kubernetes.io/nodeid: {"disk.csi.azure.com":"k8s-agentpool-15826033-vmss000001"},node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.0.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool-15826033-vmss/virtualMachines/1,Unschedulable:false,Taints:[],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-csi-disk.csi.azure.com: {{8 0} {} 8 DecimalSI},cpu: {{4 0} {} 4 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{16797044736 0} {} 16403364Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-csi-disk.csi.azure.com: {{8 0} {} 8 DecimalSI},cpu: {{4 0} {} 4 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{16010612736 0} {} 15635364Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 01:34:57 +0000 UTC 2019-11-05 01:34:57 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 02:33:48 +0000 UTC 2019-11-05 01:34:08 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 02:33:48 +0000 UTC 2019-11-05 01:34:08 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 02:33:48 +0000 UTC 2019-11-05 01:34:08 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 02:33:48 +0000 UTC 2019-11-05 01:34:29 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.0.5} {Hostname k8s-agentpool-15826033-vmss000001}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:b6b9fbf4a2fd4fa5a7ae472415fe30b8,SystemUUID:0162B3A4-3955-9A46-ABD4-A37BD75848CC,BootID:ee93003f-d47c-4118-9975-57365b3443a5,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[k8sprowinternal.azurecr.io/azuredisk-csi@sha256:75d4ff5d7cd845d6e810c67796ecb037e852fb849ec1eb5b3194564008e6d1e0 k8sprowinternal.azurecr.io/azuredisk-csi:e2e-3e3ed045c6faa509dbd8d2890ee9f56aa2934887] 286930414} {[gcr.io/kubernetes-helm/tiller@sha256:f6d8f4ab9ba993b5f5b60a6edafe86352eabe474ffeb84cb6c79b8866dce45d1 gcr.io/kubernetes-helm/tiller:v2.11.0] 71821984} {[quay.io/k8scsi/csi-provisioner@sha256:d657c839dce87324fe2b677302913f9386f885f8746be7bea0ced5b0844e3433 quay.io/k8scsi/csi-provisioner:v1.3.1] 50596998} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[quay.io/k8scsi/csi-attacher@sha256:26fccd7a99d973845df1193b46ebdcc6ab8dc5f6e6be319750c471fce1742d13 quay.io/k8scsi/csi-attacher:v1.2.0] 46226754} {[quay.io/k8scsi/csi-cluster-driver-registrar@sha256:fafd75ae5442f192cfa8c2e792903aee30d5884b62e802e4464b0a895d21e3ef quay.io/k8scsi/csi-cluster-driver-registrar:v1.0.1] 45874691} {[quay.io/k8scsi/csi-snapshotter@sha256:a49e0da1af6f2bf717e41ba1eee8b5e6a1cbd66a709dd92cc43fe475fe2589eb quay.io/k8scsi/csi-snapshotter:v1.1.0] 44325352} {[k8s.gcr.io/coredns@sha256:e83beb5e43f8513fa735e77ffc5859640baea30a882a11cc75c4c3244a737d3c k8s.gcr.io/coredns:1.5.0] 42488424} {[mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume@sha256:4fd30d43947d4a54fc89ead7985beecfd3c9b2a93a0655a373b1608ab90bd5af mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume:v0.0.7] 22909487} {[quay.io/k8scsi/livenessprobe@sha256:8c9c8a0f330acc2e558ae6624309447d02522e76aa35af23fa19b88abf4381cb quay.io/k8scsi/livenessprobe:v1.0.2] 17415381} {[quay.io/k8scsi/csi-node-driver-registrar@sha256:13daf82fb99e951a4bff8ae5fc7c17c3a8fe7130be6400990d8f6076c32d4599 quay.io/k8scsi/csi-node-driver-registrar:v1.1.0] 15815995} {[quay.io/k8scsi/livenessprobe@sha256:dde617756e0f602adc566ab71fd885f1dad451ad3fb063ac991c95a2ff47aea5 quay.io/k8scsi/livenessprobe:v1.1.0] 14967303} {[mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume@sha256:23d8c6033f02a1ecad05127ebdc931bb871264228661bc122704b0974e4d9fdd mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume:1.0.8] 1159025} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 02:34:19.547] Nov 5 02:34:19.544: INFO: I1105 02:34:19.547] Logging kubelet events for node k8s-agentpool-15826033-vmss000001 I1105 02:34:19.626] Nov 5 02:34:19.626: INFO: I1105 02:34:19.626] Logging pods the kubelet thinks is on node k8s-agentpool-15826033-vmss000001 I1105 02:34:19.709] Nov 5 02:34:19.709: INFO: azure-ip-masq-agent-6c6vn started at 2019-11-05 01:34:35 +0000 UTC (0+1 container statuses recorded) I1105 02:34:19.709] Nov 5 02:34:19.709: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 02:34:19.710] Nov 5 02:34:19.709: INFO: keyvault-flexvolume-rdlcj started at 2019-11-05 01:34:35 +0000 UTC (0+1 container statuses recorded) I1105 02:34:19.710] Nov 5 02:34:19.709: INFO: Container keyvault-flexvolume ready: true, restart count 0 I1105 02:34:19.710] Nov 5 02:34:19.709: INFO: kube-proxy-6kh57 started at 2019-11-05 01:34:35 +0000 UTC (0+1 container statuses recorded) I1105 02:34:19.710] Nov 5 02:34:19.709: INFO: Container kube-proxy ready: true, restart count 0 I1105 02:34:19.710] Nov 5 02:34:19.709: INFO: tiller-deploy-54c96cb5df-8d9fs started at 2019-11-05 01:34:35 +0000 UTC (0+1 container statuses recorded) I1105 02:34:19.711] Nov 5 02:34:19.709: INFO: Container tiller ready: true, restart count 0 I1105 02:34:19.711] Nov 5 02:34:19.709: INFO: csi-azuredisk-node-ghv5w started at 2019-11-05 01:38:24 +0000 UTC (0+3 container statuses recorded) I1105 02:34:19.711] Nov 5 02:34:19.709: INFO: Container azuredisk ready: true, restart count 0 I1105 02:34:19.711] Nov 5 02:34:19.709: INFO: Container liveness-probe ready: true, restart count 0 I1105 02:34:19.711] Nov 5 02:34:19.709: INFO: Container node-driver-registrar ready: true, restart count 0 I1105 02:34:19.711] Nov 5 02:34:19.709: INFO: csi-azuredisk-controller-6f6587bfc6-8vkfl started at 2019-11-05 01:38:24 +0000 UTC (0+6 container statuses recorded) I1105 02:34:19.712] Nov 5 02:34:19.709: INFO: Container azuredisk ready: true, restart count 0 I1105 02:34:19.712] Nov 5 02:34:19.709: INFO: Container cluster-driver-registrar ready: true, restart count 0 I1105 02:34:19.712] Nov 5 02:34:19.709: INFO: Container csi-attacher ready: true, restart count 0 I1105 02:34:19.712] Nov 5 02:34:19.709: INFO: Container csi-provisioner ready: true, restart count 0 I1105 02:34:19.712] Nov 5 02:34:19.709: INFO: Container csi-snapshotter ready: true, restart count 0 I1105 02:34:19.712] Nov 5 02:34:19.709: INFO: Container liveness-probe ready: true, restart count 0 I1105 02:34:19.713] Nov 5 02:34:19.709: INFO: blobfuse-flexvol-installer-rw5dw started at 2019-11-05 01:34:35 +0000 UTC (0+1 container statuses recorded) I1105 02:34:19.713] Nov 5 02:34:19.709: INFO: Container blobfuse-flexvol-installer ready: true, restart count 0 I1105 02:34:19.713] Nov 5 02:34:19.709: INFO: coredns-7f68dcdbdb-jt522 started at 2019-11-05 01:34:35 +0000 UTC (0+1 container statuses recorded) I1105 02:34:19.713] Nov 5 02:34:19.709: INFO: Container coredns ready: true, restart count 0 I1105 02:34:19.973] Nov 5 02:34:19.973: INFO: I1105 02:34:19.974] Latency metrics for node k8s-agentpool-15826033-vmss000001 I1105 02:34:19.974] Nov 5 02:34:19.973: INFO: I1105 02:34:19.974] Logging node info for node k8s-master-15826033-0 I1105 02:34:20.055] Nov 5 02:34:20.052: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-15826033-0,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-15826033-0,UID:aceb0231-4645-4a9a-b296-90ada851986b,ResourceVersion:11643,Generation:0,CreationTimestamp:2019-11-05 01:35:06 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 1,kubernetes.azure.com/cluster: kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-15826033-0,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.4.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5/providers/Microsoft.Compute/virtualMachines/k8s-master-15826033-0,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 01:35:20 +0000 UTC 2019-11-05 01:35:20 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 02:33:34 +0000 UTC 2019-11-05 01:34:57 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 02:33:34 +0000 UTC 2019-11-05 01:34:57 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 02:33:34 +0000 UTC 2019-11-05 01:34:57 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 02:33:34 +0000 UTC 2019-11-05 01:35:07 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.255.5} {Hostname k8s-master-15826033-0}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:1915756c22a540dc8eb30493987e9a43,SystemUUID:742C969C-5A71-D744-8E61-F27E38B56EB2,BootID:03f0672e-8fa9-4309-956f-8e3c371902c3,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 02:34:20.055] Nov 5 02:34:20.053: INFO: I1105 02:34:20.055] Logging kubelet events for node k8s-master-15826033-0 I1105 02:34:20.135] Nov 5 02:34:20.135: INFO: I1105 02:34:20.135] Logging pods the kubelet thinks is on node k8s-master-15826033-0 I1105 02:34:20.216] Nov 5 02:34:20.216: INFO: azure-ip-masq-agent-wgdpx started at 2019-11-05 01:35:06 +0000 UTC (0+1 container statuses recorded) I1105 02:34:20.217] Nov 5 02:34:20.216: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 02:34:20.217] Nov 5 02:34:20.216: INFO: kube-proxy-7d2px started at 2019-11-05 01:35:06 +0000 UTC (0+1 container statuses recorded) I1105 02:34:20.217] Nov 5 02:34:20.216: INFO: Container kube-proxy ready: true, restart count 0 I1105 02:34:20.217] Nov 5 02:34:20.216: INFO: cloud-controller-manager-k8s-master-15826033-0 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:34:20.217] Nov 5 02:34:20.216: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 02:34:20.218] Nov 5 02:34:20.216: INFO: kube-addon-manager-k8s-master-15826033-0 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:34:20.218] Nov 5 02:34:20.216: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 02:34:20.218] Nov 5 02:34:20.216: INFO: kube-apiserver-k8s-master-15826033-0 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:34:20.218] Nov 5 02:34:20.216: INFO: Container kube-apiserver ready: true, restart count 0 I1105 02:34:20.218] Nov 5 02:34:20.216: INFO: kube-controller-manager-k8s-master-15826033-0 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:34:20.218] Nov 5 02:34:20.216: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 02:34:20.218] Nov 5 02:34:20.216: INFO: kube-scheduler-k8s-master-15826033-0 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:34:20.219] Nov 5 02:34:20.216: INFO: Container kube-scheduler ready: true, restart count 0 I1105 02:34:20.467] Nov 5 02:34:20.467: INFO: I1105 02:34:20.467] Latency metrics for node k8s-master-15826033-0 I1105 02:34:20.467] Nov 5 02:34:20.467: INFO: I1105 02:34:20.468] Logging node info for node k8s-master-15826033-1 I1105 02:34:20.547] Nov 5 02:34:20.544: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-15826033-1,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-15826033-1,UID:1a13ec61-c89a-456c-8c94-bfc8e717c2e7,ResourceVersion:11644,Generation:0,CreationTimestamp:2019-11-05 01:35:07 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 1,kubernetes.azure.com/cluster: kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-15826033-1,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.5.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5/providers/Microsoft.Compute/virtualMachines/k8s-master-15826033-1,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 01:35:31 +0000 UTC 2019-11-05 01:35:31 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 02:33:35 +0000 UTC 2019-11-05 01:35:00 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 02:33:35 +0000 UTC 2019-11-05 01:35:00 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 02:33:35 +0000 UTC 2019-11-05 01:35:00 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 02:33:35 +0000 UTC 2019-11-05 01:35:07 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.255.6} {Hostname k8s-master-15826033-1}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:e0f2bccddd5b4555b180d14081ab59a8,SystemUUID:AE126631-CAD2-B54F-B22E-1F499342A4D6,BootID:509b9a9d-b046-4861-aa35-8bcb77465de2,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 02:34:20.547] Nov 5 02:34:20.544: INFO: I1105 02:34:20.547] Logging kubelet events for node k8s-master-15826033-1 I1105 02:34:20.627] Nov 5 02:34:20.626: INFO: I1105 02:34:20.627] Logging pods the kubelet thinks is on node k8s-master-15826033-1 I1105 02:34:20.707] Nov 5 02:34:20.707: INFO: kube-proxy-8kmr5 started at 2019-11-05 01:35:10 +0000 UTC (0+1 container statuses recorded) I1105 02:34:20.707] Nov 5 02:34:20.707: INFO: Container kube-proxy ready: true, restart count 0 I1105 02:34:20.708] Nov 5 02:34:20.707: INFO: cloud-controller-manager-k8s-master-15826033-1 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:34:20.708] Nov 5 02:34:20.707: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 02:34:20.708] Nov 5 02:34:20.707: INFO: kube-addon-manager-k8s-master-15826033-1 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:34:20.708] Nov 5 02:34:20.707: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 02:34:20.708] Nov 5 02:34:20.707: INFO: kube-apiserver-k8s-master-15826033-1 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:34:20.708] Nov 5 02:34:20.707: INFO: Container kube-apiserver ready: true, restart count 0 I1105 02:34:20.709] Nov 5 02:34:20.707: INFO: kube-controller-manager-k8s-master-15826033-1 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:34:20.709] Nov 5 02:34:20.707: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 02:34:20.709] Nov 5 02:34:20.707: INFO: kube-scheduler-k8s-master-15826033-1 started at 2019-11-05 01:36:16 +0000 UTC (0+1 container statuses recorded) I1105 02:34:20.709] Nov 5 02:34:20.707: INFO: Container kube-scheduler ready: true, restart count 0 I1105 02:34:20.709] Nov 5 02:34:20.707: INFO: azure-ip-masq-agent-w6dpr started at 2019-11-05 01:35:10 +0000 UTC (0+1 container statuses recorded) I1105 02:34:20.709] Nov 5 02:34:20.707: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 02:34:20.955] Nov 5 02:34:20.955: INFO: I1105 02:34:20.955] Latency metrics for node k8s-master-15826033-1 I1105 02:34:20.955] Nov 5 02:34:20.955: INFO: I1105 02:34:20.955] Logging node info for node k8s-master-15826033-2 I1105 02:34:21.034] Nov 5 02:34:21.032: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-15826033-2,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-15826033-2,UID:acb3a851-d6df-4549-ba56-4b7e96553c17,ResourceVersion:11735,Generation:0,CreationTimestamp:2019-11-05 01:34:09 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 2,kubernetes.azure.com/cluster: kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-15826033-2,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.1.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5/providers/Microsoft.Compute/virtualMachines/k8s-master-15826033-2,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 01:34:44 +0000 UTC 2019-11-05 01:34:44 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 02:34:06 +0000 UTC 2019-11-05 01:34:07 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 02:34:06 +0000 UTC 2019-11-05 01:34:07 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 02:34:06 +0000 UTC 2019-11-05 01:34:07 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 02:34:06 +0000 UTC 2019-11-05 01:34:29 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{Hostname k8s-master-15826033-2} {InternalIP 10.240.255.7}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:0885b0c2f6174106bde4fa0eae2cbc6b,SystemUUID:C176121C-5D7A-6243-9B05-8B68F8022B64,BootID:30430638-7ab6-4761-b984-2387e1a3f5b2,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 02:34:21.034] Nov 5 02:34:21.032: INFO: I1105 02:34:21.034] Logging kubelet events for node k8s-master-15826033-2 I1105 02:34:21.130] Nov 5 02:34:21.130: INFO: I1105 02:34:21.130] Logging pods the kubelet thinks is on node k8s-master-15826033-2 I1105 02:34:21.213] Nov 5 02:34:21.212: INFO: azure-ip-masq-agent-hw7s5 started at 2019-11-05 01:34:34 +0000 UTC (0+1 container statuses recorded) I1105 02:34:21.213] Nov 5 02:34:21.212: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 02:34:21.213] Nov 5 02:34:21.212: INFO: kube-proxy-5xpp6 started at 2019-11-05 01:34:34 +0000 UTC (0+1 container statuses recorded) I1105 02:34:21.213] Nov 5 02:34:21.212: INFO: Container kube-proxy ready: true, restart count 0 I1105 02:34:21.213] Nov 5 02:34:21.212: INFO: cloud-controller-manager-k8s-master-15826033-2 started at 2019-11-05 01:33:54 +0000 UTC (0+1 container statuses recorded) I1105 02:34:21.213] Nov 5 02:34:21.212: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 02:34:21.214] Nov 5 02:34:21.212: INFO: kube-addon-manager-k8s-master-15826033-2 started at 2019-11-05 01:33:54 +0000 UTC (0+1 container statuses recorded) I1105 02:34:21.214] Nov 5 02:34:21.212: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 02:34:21.214] Nov 5 02:34:21.212: INFO: kube-apiserver-k8s-master-15826033-2 started at 2019-11-05 01:33:54 +0000 UTC (0+1 container statuses recorded) I1105 02:34:21.214] Nov 5 02:34:21.212: INFO: Container kube-apiserver ready: true, restart count 0 I1105 02:34:21.214] Nov 5 02:34:21.212: INFO: kube-controller-manager-k8s-master-15826033-2 started at 2019-11-05 01:33:54 +0000 UTC (0+1 container statuses recorded) I1105 02:34:21.214] Nov 5 02:34:21.212: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 02:34:21.214] Nov 5 02:34:21.212: INFO: kube-scheduler-k8s-master-15826033-2 started at 2019-11-05 01:33:54 +0000 UTC (0+1 container statuses recorded) I1105 02:34:21.215] Nov 5 02:34:21.212: INFO: Container kube-scheduler ready: true, restart count 0 I1105 02:34:21.461] Nov 5 02:34:21.461: INFO: I1105 02:34:21.461] Latency metrics for node k8s-master-15826033-2 I1105 02:34:21.461] Nov 5 02:34:21.461: INFO: I1105 02:34:21.461] Logging node info for node k8s-master-15826033-3 I1105 02:34:21.541] Nov 5 02:34:21.538: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-15826033-3,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-15826033-3,UID:b7702272-1876-434e-b6a0-17fa34c6757f,ResourceVersion:11641,Generation:0,CreationTimestamp:2019-11-05 01:35:06 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 0,kubernetes.azure.com/cluster: kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-15826033-3,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.3.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5/providers/Microsoft.Compute/virtualMachines/k8s-master-15826033-3,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 01:35:17 +0000 UTC 2019-11-05 01:35:17 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 02:33:33 +0000 UTC 2019-11-05 01:35:00 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 02:33:33 +0000 UTC 2019-11-05 01:35:00 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 02:33:33 +0000 UTC 2019-11-05 01:35:00 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 02:33:33 +0000 UTC 2019-11-05 01:35:06 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.255.8} {Hostname k8s-master-15826033-3}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:fbf366963961422c9d90bedf6639c8e1,SystemUUID:5388948D-1A8D-EA41-B8B8-7F4523EDF5AA,BootID:42f5884c-7418-486a-8cf3-75188a331f38,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 02:34:21.541] Nov 5 02:34:21.538: INFO: I1105 02:34:21.541] Logging kubelet events for node k8s-master-15826033-3 I1105 02:34:21.619] Nov 5 02:34:21.619: INFO: I1105 02:34:21.619] Logging pods the kubelet thinks is on node k8s-master-15826033-3 I1105 02:34:21.701] Nov 5 02:34:21.701: INFO: kube-proxy-xz24z started at 2019-11-05 01:35:06 +0000 UTC (0+1 container statuses recorded) I1105 02:34:21.701] Nov 5 02:34:21.701: INFO: Container kube-proxy ready: true, restart count 0 I1105 02:34:21.701] Nov 5 02:34:21.701: INFO: azure-ip-masq-agent-dqfpw started at 2019-11-05 01:35:06 +0000 UTC (0+1 container statuses recorded) I1105 02:34:21.701] Nov 5 02:34:21.701: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 02:34:21.702] Nov 5 02:34:21.701: INFO: cloud-controller-manager-k8s-master-15826033-3 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:34:21.702] Nov 5 02:34:21.701: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 02:34:21.702] Nov 5 02:34:21.701: INFO: kube-addon-manager-k8s-master-15826033-3 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:34:21.702] Nov 5 02:34:21.701: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 02:34:21.702] Nov 5 02:34:21.701: INFO: kube-apiserver-k8s-master-15826033-3 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:34:21.703] Nov 5 02:34:21.701: INFO: Container kube-apiserver ready: true, restart count 0 I1105 02:34:21.703] Nov 5 02:34:21.701: INFO: kube-controller-manager-k8s-master-15826033-3 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:34:21.703] Nov 5 02:34:21.701: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 02:34:21.703] Nov 5 02:34:21.701: INFO: kube-scheduler-k8s-master-15826033-3 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:34:21.703] Nov 5 02:34:21.701: INFO: Container kube-scheduler ready: true, restart count 0 I1105 02:34:21.958] Nov 5 02:34:21.958: INFO: I1105 02:34:21.958] Latency metrics for node k8s-master-15826033-3 I1105 02:34:21.958] Nov 5 02:34:21.958: INFO: I1105 02:34:21.958] Logging node info for node k8s-master-15826033-4 I1105 02:34:22.037] Nov 5 02:34:22.035: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-15826033-4,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-15826033-4,UID:276daef1-6c05-41e3-bacd-abc93d35cc6d,ResourceVersion:11646,Generation:0,CreationTimestamp:2019-11-05 01:35:07 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 0,kubernetes.azure.com/cluster: kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-15826033-4,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.6.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5/providers/Microsoft.Compute/virtualMachines/k8s-master-15826033-4,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 01:35:41 +0000 UTC 2019-11-05 01:35:41 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 02:33:35 +0000 UTC 2019-11-05 01:35:02 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 02:33:35 +0000 UTC 2019-11-05 01:35:02 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 02:33:35 +0000 UTC 2019-11-05 01:35:02 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 02:33:35 +0000 UTC 2019-11-05 01:35:08 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.255.9} {Hostname k8s-master-15826033-4}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:79051fc0bdeb4e32a56acbaf5972ce3a,SystemUUID:C71606B5-7F08-9341-A26D-8A8D9D84FCAE,BootID:4bdf7cf3-2b83-42c7-a7ba-926ebe9ca75b,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 02:34:22.037] Nov 5 02:34:22.035: INFO: I1105 02:34:22.037] Logging kubelet events for node k8s-master-15826033-4 I1105 02:34:22.118] Nov 5 02:34:22.118: INFO: I1105 02:34:22.118] Logging pods the kubelet thinks is on node k8s-master-15826033-4 I1105 02:34:22.200] Nov 5 02:34:22.199: INFO: cloud-controller-manager-k8s-master-15826033-4 started at 2019-11-05 01:34:56 +0000 UTC (0+1 container statuses recorded) I1105 02:34:22.200] Nov 5 02:34:22.199: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 02:34:22.200] Nov 5 02:34:22.199: INFO: kube-addon-manager-k8s-master-15826033-4 started at 2019-11-05 01:34:56 +0000 UTC (0+1 container statuses recorded) I1105 02:34:22.200] Nov 5 02:34:22.199: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 02:34:22.200] Nov 5 02:34:22.199: INFO: azure-ip-masq-agent-9nrj8 started at 2019-11-05 01:35:11 +0000 UTC (0+1 container statuses recorded) I1105 02:34:22.200] Nov 5 02:34:22.199: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 02:34:22.201] Nov 5 02:34:22.199: INFO: kube-proxy-rmngk started at 2019-11-05 01:35:11 +0000 UTC (0+1 container statuses recorded) I1105 02:34:22.201] Nov 5 02:34:22.199: INFO: Container kube-proxy ready: true, restart count 0 I1105 02:34:22.201] Nov 5 02:34:22.199: INFO: kube-apiserver-k8s-master-15826033-4 started at 2019-11-05 01:34:55 +0000 UTC (0+1 container statuses recorded) I1105 02:34:22.201] Nov 5 02:34:22.199: INFO: Container kube-apiserver ready: true, restart count 0 I1105 02:34:22.201] Nov 5 02:34:22.199: INFO: kube-controller-manager-k8s-master-15826033-4 started at 2019-11-05 01:34:55 +0000 UTC (0+1 container statuses recorded) I1105 02:34:22.201] Nov 5 02:34:22.199: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 02:34:22.202] Nov 5 02:34:22.199: INFO: kube-scheduler-k8s-master-15826033-4 started at 2019-11-05 01:34:55 +0000 UTC (0+1 container statuses recorded) I1105 02:34:22.202] Nov 5 02:34:22.199: INFO: Container kube-scheduler ready: true, restart count 0 I1105 02:34:22.444] Nov 5 02:34:22.444: INFO: I1105 02:34:22.445] Latency metrics for node k8s-master-15826033-4 I1105 02:34:22.445] Nov 5 02:34:22.444: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready I1105 02:34:22.524] STEP: Destroying namespace "azuredisk-7865" for this suite. I1105 02:34:34.760] Nov 5 02:34:34.760: INFO: Waiting up to 30s for server preferred namespaced resources to be successfully discovered I1105 02:34:37.724] Nov 5 02:34:37.724: INFO: namespace azuredisk-7865 deletion completed in 15.20075961s I1105 02:34:37.724] I1105 02:34:37.725] • Failure [321.362 seconds] I1105 02:34:37.725] [azuredisk-csi-e2e] Dynamic Provisioning I1105 02:34:37.725] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:33 I1105 02:34:37.725] [multi-az] I1105 02:34:37.725] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:40 I1105 02:34:37.725] should create a deployment object, write and read to it, delete the pod and write and read to it again [It] I1105 02:34:37.725] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:193 I1105 02:34:37.726] I1105 02:34:37.726] Unexpected error: I1105 02:34:37.726] <*errors.errorString | 0xc0001eaab0>: { I1105 02:34:37.726] s: "PersistentVolumeClaims [pvc-sjl96] not all in phase Bound within 5m0s", I1105 02:34:37.726] } I1105 02:34:37.726] PersistentVolumeClaims [pvc-sjl96] not all in phase Bound within 5m0s I1105 02:34:37.726] occurred I1105 02:34:37.727] I1105 02:34:37.727] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/testsuites/testsuites.go:278 I1105 02:34:37.727] ------------------------------ I1105 02:34:37.727] [azuredisk-csi-e2e] Dynamic Provisioning [multi-az] I1105 02:34:37.727] should create multiple PV objects, bind to PVCs and attach all to different pods on the same node I1105 02:34:37.727] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:151 I1105 02:34:37.727] [BeforeEach] [multi-az] I1105 02:34:37.727] /go/src/sigs.k8s.io/azuredisk-csi-driver/vendor/k8s.io/kubernetes/test/e2e/framework/framework.go:150 I1105 02:34:37.728] STEP: Creating a kubernetes client I1105 02:34:37.728] Nov 5 02:34:37.724: INFO: >>> kubeConfig: /workspace/aks978705698/kubeconfig/kubeconfig.eastus2.json I1105 02:34:37.728] STEP: Building a namespace api object, basename azuredisk I1105 02:34:37.989] STEP: Binding the e2e-test-privileged-psp PodSecurityPolicy to the default service account in azuredisk-1320 I1105 02:34:38.256] STEP: Waiting for a default service account to be provisioned in namespace I1105 02:34:38.332] [BeforeEach] [multi-az] I1105 02:34:38.332] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:58 I1105 02:34:38.332] [It] should create multiple PV objects, bind to PVCs and attach all to different pods on the same node I1105 02:34:38.333] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:151 I1105 02:34:38.333] STEP: setting up the StorageClass I1105 02:34:38.333] STEP: creating a StorageClass I1105 02:34:38.414] STEP: setting up the PVC and PV I1105 02:34:38.414] STEP: creating a PVC I1105 02:34:38.499] STEP: waiting for PVC to be in phase "Bound" I1105 02:34:38.499] Nov 5 02:34:38.499: INFO: Waiting up to 5m0s for PersistentVolumeClaims [pvc-pvkjx] to have phase Bound I1105 02:34:38.576] Nov 5 02:34:38.576: INFO: PersistentVolumeClaim pvc-pvkjx found but phase is Pending instead of Bound. I1105 02:34:40.654] Nov 5 02:34:40.654: INFO: PersistentVolumeClaim pvc-pvkjx found but phase is Pending instead of Bound. I1105 02:34:42.732] Nov 5 02:34:42.731: INFO: PersistentVolumeClaim pvc-pvkjx found but phase is Pending instead of Bound. I1105 02:34:44.809] Nov 5 02:34:44.809: INFO: PersistentVolumeClaim pvc-pvkjx found but phase is Pending instead of Bound. I1105 02:34:46.886] Nov 5 02:34:46.886: INFO: PersistentVolumeClaim pvc-pvkjx found but phase is Pending instead of Bound. I1105 02:34:48.964] Nov 5 02:34:48.964: INFO: PersistentVolumeClaim pvc-pvkjx found but phase is Pending instead of Bound. I1105 02:34:51.042] Nov 5 02:34:51.042: INFO: PersistentVolumeClaim pvc-pvkjx found but phase is Pending instead of Bound. I1105 02:34:53.132] Nov 5 02:34:53.132: INFO: PersistentVolumeClaim pvc-pvkjx found but phase is Pending instead of Bound. I1105 02:34:55.209] Nov 5 02:34:55.209: INFO: PersistentVolumeClaim pvc-pvkjx found but phase is Pending instead of Bound. I1105 02:34:57.286] Nov 5 02:34:57.286: INFO: PersistentVolumeClaim pvc-pvkjx found but phase is Pending instead of Bound. I1105 02:34:59.364] Nov 5 02:34:59.364: INFO: PersistentVolumeClaim pvc-pvkjx found but phase is Pending instead of Bound. I1105 02:35:01.441] Nov 5 02:35:01.441: INFO: PersistentVolumeClaim pvc-pvkjx found but phase is Pending instead of Bound. I1105 02:35:03.521] Nov 5 02:35:03.520: INFO: PersistentVolumeClaim pvc-pvkjx found but phase is Pending instead of Bound. I1105 02:35:05.598] Nov 5 02:35:05.598: INFO: PersistentVolumeClaim pvc-pvkjx found but phase is Pending instead of Bound. I1105 02:35:07.677] Nov 5 02:35:07.676: INFO: PersistentVolumeClaim pvc-pvkjx found but phase is Pending instead of Bound. I1105 02:35:09.754] Nov 5 02:35:09.753: INFO: PersistentVolumeClaim pvc-pvkjx found but phase is Pending instead of Bound. I1105 02:35:11.831] Nov 5 02:35:11.831: INFO: PersistentVolumeClaim pvc-pvkjx found but phase is Pending instead of Bound. I1105 02:35:13.908] Nov 5 02:35:13.908: INFO: PersistentVolumeClaim pvc-pvkjx found but phase is Pending instead of Bound. I1105 02:35:15.989] Nov 5 02:35:15.989: INFO: PersistentVolumeClaim pvc-pvkjx found but phase is Pending instead of Bound. I1105 02:35:18.067] Nov 5 02:35:18.067: INFO: PersistentVolumeClaim pvc-pvkjx found but phase is Pending instead of Bound. I1105 02:35:20.144] Nov 5 02:35:20.144: INFO: PersistentVolumeClaim pvc-pvkjx found but phase is Pending instead of Bound. I1105 02:35:22.222] Nov 5 02:35:22.222: INFO: PersistentVolumeClaim pvc-pvkjx found but phase is Pending instead of Bound. I1105 02:35:24.309] Nov 5 02:35:24.309: INFO: PersistentVolumeClaim pvc-pvkjx found but phase is Pending instead of Bound. I1105 02:35:26.387] Nov 5 02:35:26.387: INFO: PersistentVolumeClaim pvc-pvkjx found but phase is Pending instead of Bound. I1105 02:35:28.465] Nov 5 02:35:28.465: INFO: PersistentVolumeClaim pvc-pvkjx found but phase is Pending instead of Bound. I1105 02:35:30.543] Nov 5 02:35:30.543: INFO: PersistentVolumeClaim pvc-pvkjx found but phase is Pending instead of Bound. I1105 02:35:32.620] Nov 5 02:35:32.620: INFO: PersistentVolumeClaim pvc-pvkjx found but phase is Pending instead of Bound. I1105 02:35:34.698] Nov 5 02:35:34.698: INFO: PersistentVolumeClaim pvc-pvkjx found but phase is Pending instead of Bound. I1105 02:35:36.776] Nov 5 02:35:36.775: INFO: PersistentVolumeClaim pvc-pvkjx found but phase is Pending instead of Bound. I1105 02:35:38.861] Nov 5 02:35:38.861: INFO: PersistentVolumeClaim pvc-pvkjx found but phase is Pending instead of Bound. I1105 02:35:40.939] Nov 5 02:35:40.939: INFO: PersistentVolumeClaim pvc-pvkjx found but phase is Pending instead of Bound. I1105 02:35:43.017] Nov 5 02:35:43.017: INFO: PersistentVolumeClaim pvc-pvkjx found but phase is Pending instead of Bound. I1105 02:35:45.120] Nov 5 02:35:45.119: INFO: PersistentVolumeClaim pvc-pvkjx found but phase is Pending instead of Bound. I1105 02:35:47.197] Nov 5 02:35:47.197: INFO: PersistentVolumeClaim pvc-pvkjx found but phase is Pending instead of Bound. I1105 02:35:49.275] Nov 5 02:35:49.275: INFO: PersistentVolumeClaim pvc-pvkjx found but phase is Pending instead of Bound. I1105 02:35:51.352] Nov 5 02:35:51.352: INFO: PersistentVolumeClaim pvc-pvkjx found but phase is Pending instead of Bound. I1105 02:35:53.430] Nov 5 02:35:53.430: INFO: PersistentVolumeClaim pvc-pvkjx found but phase is Pending instead of Bound. I1105 02:35:55.507] Nov 5 02:35:55.507: INFO: PersistentVolumeClaim pvc-pvkjx found but phase is Pending instead of Bound. I1105 02:35:57.585] Nov 5 02:35:57.585: INFO: PersistentVolumeClaim pvc-pvkjx found but phase is Pending instead of Bound. I1105 02:35:59.663] Nov 5 02:35:59.663: INFO: PersistentVolumeClaim pvc-pvkjx found but phase is Pending instead of Bound. I1105 02:36:01.740] Nov 5 02:36:01.740: INFO: PersistentVolumeClaim pvc-pvkjx found but phase is Pending instead of Bound. I1105 02:36:03.818] Nov 5 02:36:03.817: INFO: PersistentVolumeClaim pvc-pvkjx found but phase is Pending instead of Bound. I1105 02:36:05.895] Nov 5 02:36:05.895: INFO: PersistentVolumeClaim pvc-pvkjx found but phase is Pending instead of Bound. I1105 02:36:07.973] Nov 5 02:36:07.973: INFO: PersistentVolumeClaim pvc-pvkjx found but phase is Pending instead of Bound. I1105 02:36:10.050] Nov 5 02:36:10.050: INFO: PersistentVolumeClaim pvc-pvkjx found but phase is Pending instead of Bound. I1105 02:36:12.129] Nov 5 02:36:12.129: INFO: PersistentVolumeClaim pvc-pvkjx found but phase is Pending instead of Bound. I1105 02:36:14.206] Nov 5 02:36:14.206: INFO: PersistentVolumeClaim pvc-pvkjx found but phase is Pending instead of Bound. I1105 02:36:16.284] Nov 5 02:36:16.284: INFO: PersistentVolumeClaim pvc-pvkjx found but phase is Pending instead of Bound. I1105 02:36:18.362] Nov 5 02:36:18.362: INFO: PersistentVolumeClaim pvc-pvkjx found but phase is Pending instead of Bound. I1105 02:36:20.439] Nov 5 02:36:20.439: INFO: PersistentVolumeClaim pvc-pvkjx found but phase is Pending instead of Bound. I1105 02:36:22.516] Nov 5 02:36:22.516: INFO: PersistentVolumeClaim pvc-pvkjx found but phase is Pending instead of Bound. I1105 02:36:24.594] Nov 5 02:36:24.594: INFO: PersistentVolumeClaim pvc-pvkjx found but phase is Pending instead of Bound. I1105 02:36:26.679] Nov 5 02:36:26.679: INFO: PersistentVolumeClaim pvc-pvkjx found but phase is Pending instead of Bound. I1105 02:36:28.759] Nov 5 02:36:28.759: INFO: PersistentVolumeClaim pvc-pvkjx found but phase is Pending instead of Bound. I1105 02:36:30.837] Nov 5 02:36:30.837: INFO: PersistentVolumeClaim pvc-pvkjx found but phase is Pending instead of Bound. I1105 02:36:32.914] Nov 5 02:36:32.914: INFO: PersistentVolumeClaim pvc-pvkjx found but phase is Pending instead of Bound. I1105 02:36:34.991] Nov 5 02:36:34.991: INFO: PersistentVolumeClaim pvc-pvkjx found but phase is Pending instead of Bound. I1105 02:36:37.069] Nov 5 02:36:37.068: INFO: PersistentVolumeClaim pvc-pvkjx found but phase is Pending instead of Bound. I1105 02:36:39.147] Nov 5 02:36:39.147: INFO: PersistentVolumeClaim pvc-pvkjx found but phase is Pending instead of Bound. I1105 02:36:41.224] Nov 5 02:36:41.224: INFO: PersistentVolumeClaim pvc-pvkjx found but phase is Pending instead of Bound. I1105 02:36:43.301] Nov 5 02:36:43.301: INFO: PersistentVolumeClaim pvc-pvkjx found but phase is Pending instead of Bound. I1105 02:36:45.379] Nov 5 02:36:45.379: INFO: PersistentVolumeClaim pvc-pvkjx found but phase is Pending instead of Bound. I1105 02:36:47.457] Nov 5 02:36:47.456: INFO: PersistentVolumeClaim pvc-pvkjx found but phase is Pending instead of Bound. I1105 02:36:49.534] Nov 5 02:36:49.534: INFO: PersistentVolumeClaim pvc-pvkjx found but phase is Pending instead of Bound. I1105 02:36:51.612] Nov 5 02:36:51.612: INFO: PersistentVolumeClaim pvc-pvkjx found but phase is Pending instead of Bound. I1105 02:36:53.690] Nov 5 02:36:53.690: INFO: PersistentVolumeClaim pvc-pvkjx found but phase is Pending instead of Bound. I1105 02:36:55.767] Nov 5 02:36:55.767: INFO: PersistentVolumeClaim pvc-pvkjx found but phase is Pending instead of Bound. I1105 02:36:57.845] Nov 5 02:36:57.845: INFO: PersistentVolumeClaim pvc-pvkjx found but phase is Pending instead of Bound. I1105 02:36:59.922] Nov 5 02:36:59.922: INFO: PersistentVolumeClaim pvc-pvkjx found but phase is Pending instead of Bound. I1105 02:37:02.000] Nov 5 02:37:02.000: INFO: PersistentVolumeClaim pvc-pvkjx found but phase is Pending instead of Bound. I1105 02:37:04.078] Nov 5 02:37:04.077: INFO: PersistentVolumeClaim pvc-pvkjx found but phase is Pending instead of Bound. I1105 02:37:06.156] Nov 5 02:37:06.155: INFO: PersistentVolumeClaim pvc-pvkjx found but phase is Pending instead of Bound. I1105 02:37:08.234] Nov 5 02:37:08.233: INFO: PersistentVolumeClaim pvc-pvkjx found but phase is Pending instead of Bound. I1105 02:37:10.312] Nov 5 02:37:10.311: INFO: PersistentVolumeClaim pvc-pvkjx found but phase is Pending instead of Bound. I1105 02:37:12.389] Nov 5 02:37:12.389: INFO: PersistentVolumeClaim pvc-pvkjx found but phase is Pending instead of Bound. I1105 02:37:14.470] Nov 5 02:37:14.469: INFO: PersistentVolumeClaim pvc-pvkjx found but phase is Pending instead of Bound. I1105 02:37:16.547] Nov 5 02:37:16.547: INFO: PersistentVolumeClaim pvc-pvkjx found but phase is Pending instead of Bound. I1105 02:37:18.626] Nov 5 02:37:18.626: INFO: PersistentVolumeClaim pvc-pvkjx found but phase is Pending instead of Bound. I1105 02:37:20.704] Nov 5 02:37:20.704: INFO: PersistentVolumeClaim pvc-pvkjx found but phase is Pending instead of Bound. I1105 02:37:22.782] Nov 5 02:37:22.782: INFO: PersistentVolumeClaim pvc-pvkjx found but phase is Pending instead of Bound. I1105 02:37:24.860] Nov 5 02:37:24.859: INFO: PersistentVolumeClaim pvc-pvkjx found but phase is Pending instead of Bound. I1105 02:37:26.937] Nov 5 02:37:26.937: INFO: PersistentVolumeClaim pvc-pvkjx found but phase is Pending instead of Bound. I1105 02:37:29.015] Nov 5 02:37:29.014: INFO: PersistentVolumeClaim pvc-pvkjx found but phase is Pending instead of Bound. I1105 02:37:31.092] Nov 5 02:37:31.092: INFO: PersistentVolumeClaim pvc-pvkjx found but phase is Pending instead of Bound. I1105 02:37:33.170] Nov 5 02:37:33.169: INFO: PersistentVolumeClaim pvc-pvkjx found but phase is Pending instead of Bound. I1105 02:37:35.247] Nov 5 02:37:35.247: INFO: PersistentVolumeClaim pvc-pvkjx found but phase is Pending instead of Bound. I1105 02:37:37.325] Nov 5 02:37:37.324: INFO: PersistentVolumeClaim pvc-pvkjx found but phase is Pending instead of Bound. I1105 02:37:39.404] Nov 5 02:37:39.403: INFO: PersistentVolumeClaim pvc-pvkjx found but phase is Pending instead of Bound. I1105 02:37:41.481] Nov 5 02:37:41.481: INFO: PersistentVolumeClaim pvc-pvkjx found but phase is Pending instead of Bound. I1105 02:37:43.559] Nov 5 02:37:43.558: INFO: PersistentVolumeClaim pvc-pvkjx found but phase is Pending instead of Bound. I1105 02:37:45.636] Nov 5 02:37:45.636: INFO: PersistentVolumeClaim pvc-pvkjx found but phase is Pending instead of Bound. I1105 02:37:47.714] Nov 5 02:37:47.714: INFO: PersistentVolumeClaim pvc-pvkjx found but phase is Pending instead of Bound. I1105 02:37:49.791] Nov 5 02:37:49.791: INFO: PersistentVolumeClaim pvc-pvkjx found but phase is Pending instead of Bound. I1105 02:37:51.869] Nov 5 02:37:51.869: INFO: PersistentVolumeClaim pvc-pvkjx found but phase is Pending instead of Bound. I1105 02:37:53.947] Nov 5 02:37:53.947: INFO: PersistentVolumeClaim pvc-pvkjx found but phase is Pending instead of Bound. I1105 02:37:56.024] Nov 5 02:37:56.024: INFO: PersistentVolumeClaim pvc-pvkjx found but phase is Pending instead of Bound. I1105 02:37:58.102] Nov 5 02:37:58.101: INFO: PersistentVolumeClaim pvc-pvkjx found but phase is Pending instead of Bound. I1105 02:38:00.179] Nov 5 02:38:00.179: INFO: PersistentVolumeClaim pvc-pvkjx found but phase is Pending instead of Bound. I1105 02:38:02.257] Nov 5 02:38:02.256: INFO: PersistentVolumeClaim pvc-pvkjx found but phase is Pending instead of Bound. I1105 02:38:04.334] Nov 5 02:38:04.333: INFO: PersistentVolumeClaim pvc-pvkjx found but phase is Pending instead of Bound. I1105 02:38:06.411] Nov 5 02:38:06.411: INFO: PersistentVolumeClaim pvc-pvkjx found but phase is Pending instead of Bound. I1105 02:38:08.488] Nov 5 02:38:08.488: INFO: PersistentVolumeClaim pvc-pvkjx found but phase is Pending instead of Bound. I1105 02:38:10.566] Nov 5 02:38:10.566: INFO: PersistentVolumeClaim pvc-pvkjx found but phase is Pending instead of Bound. I1105 02:38:12.643] Nov 5 02:38:12.643: INFO: PersistentVolumeClaim pvc-pvkjx found but phase is Pending instead of Bound. I1105 02:38:14.721] Nov 5 02:38:14.720: INFO: PersistentVolumeClaim pvc-pvkjx found but phase is Pending instead of Bound. I1105 02:38:16.798] Nov 5 02:38:16.798: INFO: PersistentVolumeClaim pvc-pvkjx found but phase is Pending instead of Bound. I1105 02:38:18.876] Nov 5 02:38:18.876: INFO: PersistentVolumeClaim pvc-pvkjx found but phase is Pending instead of Bound. I1105 02:38:20.954] Nov 5 02:38:20.953: INFO: PersistentVolumeClaim pvc-pvkjx found but phase is Pending instead of Bound. I1105 02:38:23.031] Nov 5 02:38:23.031: INFO: PersistentVolumeClaim pvc-pvkjx found but phase is Pending instead of Bound. I1105 02:38:25.124] Nov 5 02:38:25.124: INFO: PersistentVolumeClaim pvc-pvkjx found but phase is Pending instead of Bound. I1105 02:38:27.202] Nov 5 02:38:27.202: INFO: PersistentVolumeClaim pvc-pvkjx found but phase is Pending instead of Bound. I1105 02:38:29.280] Nov 5 02:38:29.280: INFO: PersistentVolumeClaim pvc-pvkjx found but phase is Pending instead of Bound. I1105 02:38:31.358] Nov 5 02:38:31.358: INFO: PersistentVolumeClaim pvc-pvkjx found but phase is Pending instead of Bound. I1105 02:38:33.435] Nov 5 02:38:33.435: INFO: PersistentVolumeClaim pvc-pvkjx found but phase is Pending instead of Bound. I1105 02:38:35.513] Nov 5 02:38:35.513: INFO: PersistentVolumeClaim pvc-pvkjx found but phase is Pending instead of Bound. I1105 02:38:37.590] Nov 5 02:38:37.590: INFO: PersistentVolumeClaim pvc-pvkjx found but phase is Pending instead of Bound. I1105 02:38:39.668] Nov 5 02:38:39.668: INFO: PersistentVolumeClaim pvc-pvkjx found but phase is Pending instead of Bound. I1105 02:38:41.745] Nov 5 02:38:41.745: INFO: PersistentVolumeClaim pvc-pvkjx found but phase is Pending instead of Bound. I1105 02:38:43.823] Nov 5 02:38:43.823: INFO: PersistentVolumeClaim pvc-pvkjx found but phase is Pending instead of Bound. I1105 02:38:45.901] Nov 5 02:38:45.901: INFO: PersistentVolumeClaim pvc-pvkjx found but phase is Pending instead of Bound. I1105 02:38:47.978] Nov 5 02:38:47.978: INFO: PersistentVolumeClaim pvc-pvkjx found but phase is Pending instead of Bound. I1105 02:38:50.056] Nov 5 02:38:50.055: INFO: PersistentVolumeClaim pvc-pvkjx found but phase is Pending instead of Bound. I1105 02:38:52.133] Nov 5 02:38:52.133: INFO: PersistentVolumeClaim pvc-pvkjx found but phase is Pending instead of Bound. I1105 02:38:54.210] Nov 5 02:38:54.210: INFO: PersistentVolumeClaim pvc-pvkjx found but phase is Pending instead of Bound. I1105 02:38:56.289] Nov 5 02:38:56.289: INFO: PersistentVolumeClaim pvc-pvkjx found but phase is Pending instead of Bound. I1105 02:38:58.366] Nov 5 02:38:58.366: INFO: PersistentVolumeClaim pvc-pvkjx found but phase is Pending instead of Bound. I1105 02:39:00.452] Nov 5 02:39:00.452: INFO: PersistentVolumeClaim pvc-pvkjx found but phase is Pending instead of Bound. I1105 02:39:02.531] Nov 5 02:39:02.530: INFO: PersistentVolumeClaim pvc-pvkjx found but phase is Pending instead of Bound. I1105 02:39:04.611] Nov 5 02:39:04.611: INFO: PersistentVolumeClaim pvc-pvkjx found but phase is Pending instead of Bound. I1105 02:39:06.689] Nov 5 02:39:06.689: INFO: PersistentVolumeClaim pvc-pvkjx found but phase is Pending instead of Bound. I1105 02:39:08.771] Nov 5 02:39:08.771: INFO: PersistentVolumeClaim pvc-pvkjx found but phase is Pending instead of Bound. I1105 02:39:10.849] Nov 5 02:39:10.849: INFO: PersistentVolumeClaim pvc-pvkjx found but phase is Pending instead of Bound. I1105 02:39:12.926] Nov 5 02:39:12.926: INFO: PersistentVolumeClaim pvc-pvkjx found but phase is Pending instead of Bound. I1105 02:39:15.004] Nov 5 02:39:15.004: INFO: PersistentVolumeClaim pvc-pvkjx found but phase is Pending instead of Bound. I1105 02:39:17.082] Nov 5 02:39:17.082: INFO: PersistentVolumeClaim pvc-pvkjx found but phase is Pending instead of Bound. I1105 02:39:19.160] Nov 5 02:39:19.160: INFO: PersistentVolumeClaim pvc-pvkjx found but phase is Pending instead of Bound. I1105 02:39:21.238] Nov 5 02:39:21.237: INFO: PersistentVolumeClaim pvc-pvkjx found but phase is Pending instead of Bound. I1105 02:39:23.315] Nov 5 02:39:23.315: INFO: PersistentVolumeClaim pvc-pvkjx found but phase is Pending instead of Bound. I1105 02:39:25.394] Nov 5 02:39:25.394: INFO: PersistentVolumeClaim pvc-pvkjx found but phase is Pending instead of Bound. I1105 02:39:27.472] Nov 5 02:39:27.472: INFO: PersistentVolumeClaim pvc-pvkjx found but phase is Pending instead of Bound. I1105 02:39:29.550] Nov 5 02:39:29.550: INFO: PersistentVolumeClaim pvc-pvkjx found but phase is Pending instead of Bound. I1105 02:39:31.628] Nov 5 02:39:31.627: INFO: PersistentVolumeClaim pvc-pvkjx found but phase is Pending instead of Bound. I1105 02:39:33.706] Nov 5 02:39:33.705: INFO: PersistentVolumeClaim pvc-pvkjx found but phase is Pending instead of Bound. I1105 02:39:35.784] Nov 5 02:39:35.784: INFO: PersistentVolumeClaim pvc-pvkjx found but phase is Pending instead of Bound. I1105 02:39:37.862] Nov 5 02:39:37.862: INFO: PersistentVolumeClaim pvc-pvkjx found but phase is Pending instead of Bound. I1105 02:39:39.862] Nov 5 02:39:39.862: INFO: Unexpected error occurred: PersistentVolumeClaims [pvc-pvkjx] not all in phase Bound within 5m0s I1105 02:39:39.863] [AfterEach] [multi-az] I1105 02:39:39.863] /go/src/sigs.k8s.io/azuredisk-csi-driver/vendor/k8s.io/kubernetes/test/e2e/framework/framework.go:151 I1105 02:39:39.863] STEP: Collecting events from namespace "azuredisk-1320". I1105 02:39:39.946] STEP: Found 3 events. I1105 02:39:39.947] Nov 5 02:39:39.946: INFO: At 2019-11-05 02:34:38 +0000 UTC - event for pvc-pvkjx: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "disk.csi.azure.com" or manually created by system administrator I1105 02:39:39.947] Nov 5 02:39:39.946: INFO: At 2019-11-05 02:34:38 +0000 UTC - event for pvc-pvkjx: {disk.csi.azure.com_csi-azuredisk-controller-6f6587bfc6-8vkfl_1376914b-ff6d-11e9-93ab-ba2bee589b8c } Provisioning: External provisioner is provisioning volume for claim "azuredisk-1320/pvc-pvkjx" I1105 02:39:39.947] Nov 5 02:39:39.946: INFO: At 2019-11-05 02:34:38 +0000 UTC - event for pvc-pvkjx: {disk.csi.azure.com_csi-azuredisk-controller-6f6587bfc6-8vkfl_1376914b-ff6d-11e9-93ab-ba2bee589b8c } ProvisioningFailed: failed to provision volume with StorageClass "azuredisk-1320-disk.csi.azure.com-dynamic-sc-hzd8j": rpc error: code = Unknown desc = compute.DisksClient#CreateOrUpdate: Failure sending request: StatusCode=400 -- Original Error: Code="InvalidAvailabilityZone" Message="The zone(s) '' for resource 'Microsoft.Compute/disks/pvc-a0ed7457-27bb-4e63-90ae-b23b04e1b9de' is not supported. The supported zones for location 'eastus2' are '1,2,3'" I1105 02:39:40.023] Nov 5 02:39:40.023: INFO: POD NODE PHASE GRACE CONDITIONS I1105 02:39:40.024] Nov 5 02:39:40.023: INFO: I1105 02:39:40.254] Nov 5 02:39:40.254: INFO: I1105 02:39:40.255] Logging node info for node k8s-agentpool-15826033-vmss000000 I1105 02:39:40.335] Nov 5 02:39:40.332: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-agentpool-15826033-vmss000000,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-agentpool-15826033-vmss000000,UID:15fa874e-54ec-42cd-b243-d3baa512150f,ResourceVersion:12700,Generation:0,CreationTimestamp:2019-11-05 01:34:17 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{agentpool: agentpool,beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_D4s_v3,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 0,kubernetes.azure.com/cluster: kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5,kubernetes.azure.com/role: agent,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-agentpool-15826033-vmss000000,kubernetes.io/os: linux,kubernetes.io/role: agent,node-role.kubernetes.io/agent: ,storageprofile: managed,storagetier: Premium_LRS,topology.disk.csi.azure.com/zone: 0,},Annotations:map[string]string{csi.volume.kubernetes.io/nodeid: {"disk.csi.azure.com":"k8s-agentpool-15826033-vmss000000"},node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.2.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool-15826033-vmss/virtualMachines/0,Unschedulable:false,Taints:[],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-csi-disk.csi.azure.com: {{8 0} {} 8 DecimalSI},cpu: {{4 0} {} 4 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{16797044736 0} {} 16403364Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-csi-disk.csi.azure.com: {{8 0} {} 8 DecimalSI},cpu: {{4 0} {} 4 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{16010612736 0} {} 15635364Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 01:34:56 +0000 UTC 2019-11-05 01:34:56 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 02:39:27 +0000 UTC 2019-11-05 01:34:17 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 02:39:27 +0000 UTC 2019-11-05 01:34:17 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 02:39:27 +0000 UTC 2019-11-05 01:34:17 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 02:39:27 +0000 UTC 2019-11-05 01:34:29 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.0.4} {Hostname k8s-agentpool-15826033-vmss000000}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:afe00792177d49a98f6a44d03d08732d,SystemUUID:4F1777BF-A5C3-004E-81C4-90F0B54C02A0,BootID:13b6a628-ae89-44af-b891-89461030adcb,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[k8sprowinternal.azurecr.io/azuredisk-csi@sha256:75d4ff5d7cd845d6e810c67796ecb037e852fb849ec1eb5b3194564008e6d1e0 k8sprowinternal.azurecr.io/azuredisk-csi:e2e-3e3ed045c6faa509dbd8d2890ee9f56aa2934887] 286930414} {[k8s.gcr.io/kubernetes-dashboard-amd64@sha256:0ae6b69432e78069c5ce2bcde0fe409c5c4d6f0f4d9cd50a17974fea38898747 k8s.gcr.io/kubernetes-dashboard-amd64:v1.10.1] 121711221} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/metrics-server-amd64@sha256:49a9f12f7067d11f42c803dbe61ed2c1299959ad85cb315b25ff7eef8e6b8892 k8s.gcr.io/metrics-server-amd64:v0.2.1] 42541759} {[mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume@sha256:4fd30d43947d4a54fc89ead7985beecfd3c9b2a93a0655a373b1608ab90bd5af mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume:v0.0.7] 22909487} {[quay.io/k8scsi/livenessprobe@sha256:8c9c8a0f330acc2e558ae6624309447d02522e76aa35af23fa19b88abf4381cb quay.io/k8scsi/livenessprobe:v1.0.2] 17415381} {[quay.io/k8scsi/csi-node-driver-registrar@sha256:13daf82fb99e951a4bff8ae5fc7c17c3a8fe7130be6400990d8f6076c32d4599 quay.io/k8scsi/csi-node-driver-registrar:v1.1.0] 15815995} {[mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume@sha256:23d8c6033f02a1ecad05127ebdc931bb871264228661bc122704b0974e4d9fdd mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume:1.0.8] 1159025} {[busybox@sha256:8ccbac733d19c0dd4d70b4f0c1e12245b5fa3ad24758a11035ee505c629c0796 busybox:1.29] 1154361} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 02:39:40.335] Nov 5 02:39:40.332: INFO: I1105 02:39:40.335] Logging kubelet events for node k8s-agentpool-15826033-vmss000000 I1105 02:39:40.409] Nov 5 02:39:40.409: INFO: I1105 02:39:40.409] Logging pods the kubelet thinks is on node k8s-agentpool-15826033-vmss000000 I1105 02:39:40.492] Nov 5 02:39:40.492: INFO: kube-proxy-wxblc started at 2019-11-05 01:34:34 +0000 UTC (0+1 container statuses recorded) I1105 02:39:40.492] Nov 5 02:39:40.492: INFO: Container kube-proxy ready: true, restart count 0 I1105 02:39:40.492] Nov 5 02:39:40.492: INFO: kubernetes-dashboard-66dd8b8df7-ks284 started at 2019-11-05 01:34:35 +0000 UTC (0+1 container statuses recorded) I1105 02:39:40.493] Nov 5 02:39:40.492: INFO: Container kubernetes-dashboard ready: true, restart count 0 I1105 02:39:40.493] Nov 5 02:39:40.492: INFO: csi-azuredisk-node-whq2g started at 2019-11-05 01:38:24 +0000 UTC (0+3 container statuses recorded) I1105 02:39:40.493] Nov 5 02:39:40.492: INFO: Container azuredisk ready: true, restart count 0 I1105 02:39:40.493] Nov 5 02:39:40.492: INFO: Container liveness-probe ready: true, restart count 0 I1105 02:39:40.494] Nov 5 02:39:40.492: INFO: Container node-driver-registrar ready: true, restart count 0 I1105 02:39:40.494] Nov 5 02:39:40.492: INFO: blobfuse-flexvol-installer-bsvqt started at 2019-11-05 01:34:34 +0000 UTC (0+1 container statuses recorded) I1105 02:39:40.494] Nov 5 02:39:40.492: INFO: Container blobfuse-flexvol-installer ready: true, restart count 0 I1105 02:39:40.494] Nov 5 02:39:40.492: INFO: azure-ip-masq-agent-vx86b started at 2019-11-05 01:34:34 +0000 UTC (0+1 container statuses recorded) I1105 02:39:40.494] Nov 5 02:39:40.492: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 02:39:40.495] Nov 5 02:39:40.492: INFO: keyvault-flexvolume-qhnfr started at 2019-11-05 01:34:34 +0000 UTC (0+1 container statuses recorded) I1105 02:39:40.495] Nov 5 02:39:40.492: INFO: Container keyvault-flexvolume ready: true, restart count 0 I1105 02:39:40.495] Nov 5 02:39:40.492: INFO: metrics-server-864ffbc5c-sdxps started at 2019-11-05 01:34:34 +0000 UTC (0+1 container statuses recorded) I1105 02:39:40.495] Nov 5 02:39:40.492: INFO: Container metrics-server ready: true, restart count 0 I1105 02:39:40.741] Nov 5 02:39:40.741: INFO: I1105 02:39:40.742] Latency metrics for node k8s-agentpool-15826033-vmss000000 I1105 02:39:40.742] Nov 5 02:39:40.741: INFO: I1105 02:39:40.742] Logging node info for node k8s-agentpool-15826033-vmss000001 I1105 02:39:40.822] Nov 5 02:39:40.818: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-agentpool-15826033-vmss000001,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-agentpool-15826033-vmss000001,UID:4a8ea605-cc0b-4e3b-9060-d8867d269e6c,ResourceVersion:12590,Generation:0,CreationTimestamp:2019-11-05 01:34:09 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{agentpool: agentpool,beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_D4s_v3,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 1,kubernetes.azure.com/cluster: kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5,kubernetes.azure.com/role: agent,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-agentpool-15826033-vmss000001,kubernetes.io/os: linux,kubernetes.io/role: agent,node-role.kubernetes.io/agent: ,storageprofile: managed,storagetier: Premium_LRS,topology.disk.csi.azure.com/zone: 1,},Annotations:map[string]string{csi.volume.kubernetes.io/nodeid: {"disk.csi.azure.com":"k8s-agentpool-15826033-vmss000001"},node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.0.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool-15826033-vmss/virtualMachines/1,Unschedulable:false,Taints:[],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-csi-disk.csi.azure.com: {{8 0} {} 8 DecimalSI},cpu: {{4 0} {} 4 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{16797044736 0} {} 16403364Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-csi-disk.csi.azure.com: {{8 0} {} 8 DecimalSI},cpu: {{4 0} {} 4 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{16010612736 0} {} 15635364Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 01:34:57 +0000 UTC 2019-11-05 01:34:57 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 02:38:49 +0000 UTC 2019-11-05 01:34:08 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 02:38:49 +0000 UTC 2019-11-05 01:34:08 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 02:38:49 +0000 UTC 2019-11-05 01:34:08 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 02:38:49 +0000 UTC 2019-11-05 01:34:29 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.0.5} {Hostname k8s-agentpool-15826033-vmss000001}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:b6b9fbf4a2fd4fa5a7ae472415fe30b8,SystemUUID:0162B3A4-3955-9A46-ABD4-A37BD75848CC,BootID:ee93003f-d47c-4118-9975-57365b3443a5,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[k8sprowinternal.azurecr.io/azuredisk-csi@sha256:75d4ff5d7cd845d6e810c67796ecb037e852fb849ec1eb5b3194564008e6d1e0 k8sprowinternal.azurecr.io/azuredisk-csi:e2e-3e3ed045c6faa509dbd8d2890ee9f56aa2934887] 286930414} {[gcr.io/kubernetes-helm/tiller@sha256:f6d8f4ab9ba993b5f5b60a6edafe86352eabe474ffeb84cb6c79b8866dce45d1 gcr.io/kubernetes-helm/tiller:v2.11.0] 71821984} {[quay.io/k8scsi/csi-provisioner@sha256:d657c839dce87324fe2b677302913f9386f885f8746be7bea0ced5b0844e3433 quay.io/k8scsi/csi-provisioner:v1.3.1] 50596998} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[quay.io/k8scsi/csi-attacher@sha256:26fccd7a99d973845df1193b46ebdcc6ab8dc5f6e6be319750c471fce1742d13 quay.io/k8scsi/csi-attacher:v1.2.0] 46226754} {[quay.io/k8scsi/csi-cluster-driver-registrar@sha256:fafd75ae5442f192cfa8c2e792903aee30d5884b62e802e4464b0a895d21e3ef quay.io/k8scsi/csi-cluster-driver-registrar:v1.0.1] 45874691} {[quay.io/k8scsi/csi-snapshotter@sha256:a49e0da1af6f2bf717e41ba1eee8b5e6a1cbd66a709dd92cc43fe475fe2589eb quay.io/k8scsi/csi-snapshotter:v1.1.0] 44325352} {[k8s.gcr.io/coredns@sha256:e83beb5e43f8513fa735e77ffc5859640baea30a882a11cc75c4c3244a737d3c k8s.gcr.io/coredns:1.5.0] 42488424} {[mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume@sha256:4fd30d43947d4a54fc89ead7985beecfd3c9b2a93a0655a373b1608ab90bd5af mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume:v0.0.7] 22909487} {[quay.io/k8scsi/livenessprobe@sha256:8c9c8a0f330acc2e558ae6624309447d02522e76aa35af23fa19b88abf4381cb quay.io/k8scsi/livenessprobe:v1.0.2] 17415381} {[quay.io/k8scsi/csi-node-driver-registrar@sha256:13daf82fb99e951a4bff8ae5fc7c17c3a8fe7130be6400990d8f6076c32d4599 quay.io/k8scsi/csi-node-driver-registrar:v1.1.0] 15815995} {[quay.io/k8scsi/livenessprobe@sha256:dde617756e0f602adc566ab71fd885f1dad451ad3fb063ac991c95a2ff47aea5 quay.io/k8scsi/livenessprobe:v1.1.0] 14967303} {[mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume@sha256:23d8c6033f02a1ecad05127ebdc931bb871264228661bc122704b0974e4d9fdd mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume:1.0.8] 1159025} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 02:39:40.822] Nov 5 02:39:40.818: INFO: I1105 02:39:40.822] Logging kubelet events for node k8s-agentpool-15826033-vmss000001 I1105 02:39:40.896] Nov 5 02:39:40.896: INFO: I1105 02:39:40.896] Logging pods the kubelet thinks is on node k8s-agentpool-15826033-vmss000001 I1105 02:39:40.978] Nov 5 02:39:40.978: INFO: kube-proxy-6kh57 started at 2019-11-05 01:34:35 +0000 UTC (0+1 container statuses recorded) I1105 02:39:40.979] Nov 5 02:39:40.978: INFO: Container kube-proxy ready: true, restart count 0 I1105 02:39:40.979] Nov 5 02:39:40.978: INFO: tiller-deploy-54c96cb5df-8d9fs started at 2019-11-05 01:34:35 +0000 UTC (0+1 container statuses recorded) I1105 02:39:40.979] Nov 5 02:39:40.978: INFO: Container tiller ready: true, restart count 0 I1105 02:39:40.979] Nov 5 02:39:40.978: INFO: csi-azuredisk-node-ghv5w started at 2019-11-05 01:38:24 +0000 UTC (0+3 container statuses recorded) I1105 02:39:40.980] Nov 5 02:39:40.978: INFO: Container azuredisk ready: true, restart count 0 I1105 02:39:40.980] Nov 5 02:39:40.978: INFO: Container liveness-probe ready: true, restart count 0 I1105 02:39:40.980] Nov 5 02:39:40.978: INFO: Container node-driver-registrar ready: true, restart count 0 I1105 02:39:40.980] Nov 5 02:39:40.978: INFO: csi-azuredisk-controller-6f6587bfc6-8vkfl started at 2019-11-05 01:38:24 +0000 UTC (0+6 container statuses recorded) I1105 02:39:40.981] Nov 5 02:39:40.978: INFO: Container azuredisk ready: true, restart count 0 I1105 02:39:40.981] Nov 5 02:39:40.978: INFO: Container cluster-driver-registrar ready: true, restart count 0 I1105 02:39:40.981] Nov 5 02:39:40.978: INFO: Container csi-attacher ready: true, restart count 0 I1105 02:39:40.981] Nov 5 02:39:40.978: INFO: Container csi-provisioner ready: true, restart count 0 I1105 02:39:40.981] Nov 5 02:39:40.978: INFO: Container csi-snapshotter ready: true, restart count 0 I1105 02:39:40.982] Nov 5 02:39:40.978: INFO: Container liveness-probe ready: true, restart count 0 I1105 02:39:40.982] Nov 5 02:39:40.978: INFO: blobfuse-flexvol-installer-rw5dw started at 2019-11-05 01:34:35 +0000 UTC (0+1 container statuses recorded) I1105 02:39:40.982] Nov 5 02:39:40.978: INFO: Container blobfuse-flexvol-installer ready: true, restart count 0 I1105 02:39:40.982] Nov 5 02:39:40.978: INFO: coredns-7f68dcdbdb-jt522 started at 2019-11-05 01:34:35 +0000 UTC (0+1 container statuses recorded) I1105 02:39:40.982] Nov 5 02:39:40.978: INFO: Container coredns ready: true, restart count 0 I1105 02:39:40.983] Nov 5 02:39:40.978: INFO: azure-ip-masq-agent-6c6vn started at 2019-11-05 01:34:35 +0000 UTC (0+1 container statuses recorded) I1105 02:39:40.983] Nov 5 02:39:40.978: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 02:39:40.983] Nov 5 02:39:40.978: INFO: keyvault-flexvolume-rdlcj started at 2019-11-05 01:34:35 +0000 UTC (0+1 container statuses recorded) I1105 02:39:40.983] Nov 5 02:39:40.978: INFO: Container keyvault-flexvolume ready: true, restart count 0 I1105 02:39:41.246] Nov 5 02:39:41.245: INFO: I1105 02:39:41.246] Latency metrics for node k8s-agentpool-15826033-vmss000001 I1105 02:39:41.246] Nov 5 02:39:41.245: INFO: I1105 02:39:41.246] Logging node info for node k8s-master-15826033-0 I1105 02:39:41.325] Nov 5 02:39:41.323: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-15826033-0,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-15826033-0,UID:aceb0231-4645-4a9a-b296-90ada851986b,ResourceVersion:12724,Generation:0,CreationTimestamp:2019-11-05 01:35:06 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 1,kubernetes.azure.com/cluster: kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-15826033-0,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.4.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5/providers/Microsoft.Compute/virtualMachines/k8s-master-15826033-0,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 01:35:20 +0000 UTC 2019-11-05 01:35:20 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 02:39:35 +0000 UTC 2019-11-05 01:34:57 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 02:39:35 +0000 UTC 2019-11-05 01:34:57 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 02:39:35 +0000 UTC 2019-11-05 01:34:57 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 02:39:35 +0000 UTC 2019-11-05 01:35:07 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.255.5} {Hostname k8s-master-15826033-0}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:1915756c22a540dc8eb30493987e9a43,SystemUUID:742C969C-5A71-D744-8E61-F27E38B56EB2,BootID:03f0672e-8fa9-4309-956f-8e3c371902c3,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 02:39:41.326] Nov 5 02:39:41.323: INFO: I1105 02:39:41.326] Logging kubelet events for node k8s-master-15826033-0 I1105 02:39:41.401] Nov 5 02:39:41.401: INFO: I1105 02:39:41.401] Logging pods the kubelet thinks is on node k8s-master-15826033-0 I1105 02:39:41.482] Nov 5 02:39:41.482: INFO: cloud-controller-manager-k8s-master-15826033-0 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:39:41.482] Nov 5 02:39:41.482: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 02:39:41.483] Nov 5 02:39:41.482: INFO: kube-addon-manager-k8s-master-15826033-0 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:39:41.483] Nov 5 02:39:41.482: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 02:39:41.483] Nov 5 02:39:41.482: INFO: kube-apiserver-k8s-master-15826033-0 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:39:41.483] Nov 5 02:39:41.482: INFO: Container kube-apiserver ready: true, restart count 0 I1105 02:39:41.483] Nov 5 02:39:41.482: INFO: kube-controller-manager-k8s-master-15826033-0 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:39:41.483] Nov 5 02:39:41.482: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 02:39:41.484] Nov 5 02:39:41.482: INFO: kube-scheduler-k8s-master-15826033-0 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:39:41.484] Nov 5 02:39:41.482: INFO: Container kube-scheduler ready: true, restart count 0 I1105 02:39:41.484] Nov 5 02:39:41.482: INFO: azure-ip-masq-agent-wgdpx started at 2019-11-05 01:35:06 +0000 UTC (0+1 container statuses recorded) I1105 02:39:41.484] Nov 5 02:39:41.482: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 02:39:41.484] Nov 5 02:39:41.482: INFO: kube-proxy-7d2px started at 2019-11-05 01:35:06 +0000 UTC (0+1 container statuses recorded) I1105 02:39:41.484] Nov 5 02:39:41.482: INFO: Container kube-proxy ready: true, restart count 0 I1105 02:39:41.730] Nov 5 02:39:41.730: INFO: I1105 02:39:41.731] Latency metrics for node k8s-master-15826033-0 I1105 02:39:41.731] Nov 5 02:39:41.730: INFO: I1105 02:39:41.731] Logging node info for node k8s-master-15826033-1 I1105 02:39:41.810] Nov 5 02:39:41.808: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-15826033-1,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-15826033-1,UID:1a13ec61-c89a-456c-8c94-bfc8e717c2e7,ResourceVersion:12727,Generation:0,CreationTimestamp:2019-11-05 01:35:07 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 1,kubernetes.azure.com/cluster: kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-15826033-1,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.5.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5/providers/Microsoft.Compute/virtualMachines/k8s-master-15826033-1,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 01:35:31 +0000 UTC 2019-11-05 01:35:31 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 02:39:35 +0000 UTC 2019-11-05 01:35:00 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 02:39:35 +0000 UTC 2019-11-05 01:35:00 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 02:39:35 +0000 UTC 2019-11-05 01:35:00 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 02:39:35 +0000 UTC 2019-11-05 01:35:07 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.255.6} {Hostname k8s-master-15826033-1}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:e0f2bccddd5b4555b180d14081ab59a8,SystemUUID:AE126631-CAD2-B54F-B22E-1F499342A4D6,BootID:509b9a9d-b046-4861-aa35-8bcb77465de2,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 02:39:41.811] Nov 5 02:39:41.808: INFO: I1105 02:39:41.811] Logging kubelet events for node k8s-master-15826033-1 I1105 02:39:41.886] Nov 5 02:39:41.886: INFO: I1105 02:39:41.886] Logging pods the kubelet thinks is on node k8s-master-15826033-1 I1105 02:39:41.967] Nov 5 02:39:41.967: INFO: kube-proxy-8kmr5 started at 2019-11-05 01:35:10 +0000 UTC (0+1 container statuses recorded) I1105 02:39:41.967] Nov 5 02:39:41.967: INFO: Container kube-proxy ready: true, restart count 0 I1105 02:39:41.968] Nov 5 02:39:41.967: INFO: cloud-controller-manager-k8s-master-15826033-1 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:39:41.968] Nov 5 02:39:41.967: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 02:39:41.968] Nov 5 02:39:41.967: INFO: kube-addon-manager-k8s-master-15826033-1 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:39:41.968] Nov 5 02:39:41.967: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 02:39:41.968] Nov 5 02:39:41.967: INFO: kube-apiserver-k8s-master-15826033-1 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:39:41.969] Nov 5 02:39:41.967: INFO: Container kube-apiserver ready: true, restart count 0 I1105 02:39:41.969] Nov 5 02:39:41.967: INFO: kube-controller-manager-k8s-master-15826033-1 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:39:41.969] Nov 5 02:39:41.967: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 02:39:41.970] Nov 5 02:39:41.967: INFO: kube-scheduler-k8s-master-15826033-1 started at 2019-11-05 01:36:16 +0000 UTC (0+1 container statuses recorded) I1105 02:39:41.970] Nov 5 02:39:41.967: INFO: Container kube-scheduler ready: true, restart count 0 I1105 02:39:41.970] Nov 5 02:39:41.967: INFO: azure-ip-masq-agent-w6dpr started at 2019-11-05 01:35:10 +0000 UTC (0+1 container statuses recorded) I1105 02:39:41.970] Nov 5 02:39:41.967: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 02:39:42.212] Nov 5 02:39:42.212: INFO: I1105 02:39:42.212] Latency metrics for node k8s-master-15826033-1 I1105 02:39:42.213] Nov 5 02:39:42.212: INFO: I1105 02:39:42.213] Logging node info for node k8s-master-15826033-2 I1105 02:39:42.303] Nov 5 02:39:42.301: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-15826033-2,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-15826033-2,UID:acb3a851-d6df-4549-ba56-4b7e96553c17,ResourceVersion:12641,Generation:0,CreationTimestamp:2019-11-05 01:34:09 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 2,kubernetes.azure.com/cluster: kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-15826033-2,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.1.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5/providers/Microsoft.Compute/virtualMachines/k8s-master-15826033-2,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 01:34:44 +0000 UTC 2019-11-05 01:34:44 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 02:39:07 +0000 UTC 2019-11-05 01:34:07 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 02:39:07 +0000 UTC 2019-11-05 01:34:07 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 02:39:07 +0000 UTC 2019-11-05 01:34:07 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 02:39:07 +0000 UTC 2019-11-05 01:34:29 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{Hostname k8s-master-15826033-2} {InternalIP 10.240.255.7}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:0885b0c2f6174106bde4fa0eae2cbc6b,SystemUUID:C176121C-5D7A-6243-9B05-8B68F8022B64,BootID:30430638-7ab6-4761-b984-2387e1a3f5b2,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 02:39:42.304] Nov 5 02:39:42.301: INFO: I1105 02:39:42.304] Logging kubelet events for node k8s-master-15826033-2 I1105 02:39:42.379] Nov 5 02:39:42.379: INFO: I1105 02:39:42.379] Logging pods the kubelet thinks is on node k8s-master-15826033-2 I1105 02:39:42.461] Nov 5 02:39:42.461: INFO: kube-apiserver-k8s-master-15826033-2 started at 2019-11-05 01:33:54 +0000 UTC (0+1 container statuses recorded) I1105 02:39:42.462] Nov 5 02:39:42.461: INFO: Container kube-apiserver ready: true, restart count 0 I1105 02:39:42.462] Nov 5 02:39:42.461: INFO: kube-controller-manager-k8s-master-15826033-2 started at 2019-11-05 01:33:54 +0000 UTC (0+1 container statuses recorded) I1105 02:39:42.462] Nov 5 02:39:42.461: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 02:39:42.462] Nov 5 02:39:42.461: INFO: kube-scheduler-k8s-master-15826033-2 started at 2019-11-05 01:33:54 +0000 UTC (0+1 container statuses recorded) I1105 02:39:42.463] Nov 5 02:39:42.461: INFO: Container kube-scheduler ready: true, restart count 0 I1105 02:39:42.463] Nov 5 02:39:42.461: INFO: azure-ip-masq-agent-hw7s5 started at 2019-11-05 01:34:34 +0000 UTC (0+1 container statuses recorded) I1105 02:39:42.463] Nov 5 02:39:42.461: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 02:39:42.463] Nov 5 02:39:42.461: INFO: kube-proxy-5xpp6 started at 2019-11-05 01:34:34 +0000 UTC (0+1 container statuses recorded) I1105 02:39:42.463] Nov 5 02:39:42.461: INFO: Container kube-proxy ready: true, restart count 0 I1105 02:39:42.463] Nov 5 02:39:42.461: INFO: cloud-controller-manager-k8s-master-15826033-2 started at 2019-11-05 01:33:54 +0000 UTC (0+1 container statuses recorded) I1105 02:39:42.464] Nov 5 02:39:42.461: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 02:39:42.464] Nov 5 02:39:42.461: INFO: kube-addon-manager-k8s-master-15826033-2 started at 2019-11-05 01:33:54 +0000 UTC (0+1 container statuses recorded) I1105 02:39:42.464] Nov 5 02:39:42.461: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 02:39:42.712] Nov 5 02:39:42.712: INFO: I1105 02:39:42.712] Latency metrics for node k8s-master-15826033-2 I1105 02:39:42.712] Nov 5 02:39:42.712: INFO: I1105 02:39:42.713] Logging node info for node k8s-master-15826033-3 I1105 02:39:42.791] Nov 5 02:39:42.789: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-15826033-3,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-15826033-3,UID:b7702272-1876-434e-b6a0-17fa34c6757f,ResourceVersion:12723,Generation:0,CreationTimestamp:2019-11-05 01:35:06 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 0,kubernetes.azure.com/cluster: kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-15826033-3,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.3.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5/providers/Microsoft.Compute/virtualMachines/k8s-master-15826033-3,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 01:35:17 +0000 UTC 2019-11-05 01:35:17 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 02:39:34 +0000 UTC 2019-11-05 01:35:00 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 02:39:34 +0000 UTC 2019-11-05 01:35:00 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 02:39:34 +0000 UTC 2019-11-05 01:35:00 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 02:39:34 +0000 UTC 2019-11-05 01:35:06 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.255.8} {Hostname k8s-master-15826033-3}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:fbf366963961422c9d90bedf6639c8e1,SystemUUID:5388948D-1A8D-EA41-B8B8-7F4523EDF5AA,BootID:42f5884c-7418-486a-8cf3-75188a331f38,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 02:39:42.792] Nov 5 02:39:42.789: INFO: I1105 02:39:42.792] Logging kubelet events for node k8s-master-15826033-3 I1105 02:39:42.867] Nov 5 02:39:42.867: INFO: I1105 02:39:42.867] Logging pods the kubelet thinks is on node k8s-master-15826033-3 I1105 02:39:42.949] Nov 5 02:39:42.949: INFO: azure-ip-masq-agent-dqfpw started at 2019-11-05 01:35:06 +0000 UTC (0+1 container statuses recorded) I1105 02:39:42.949] Nov 5 02:39:42.949: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 02:39:42.949] Nov 5 02:39:42.949: INFO: cloud-controller-manager-k8s-master-15826033-3 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:39:42.950] Nov 5 02:39:42.949: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 02:39:42.950] Nov 5 02:39:42.949: INFO: kube-addon-manager-k8s-master-15826033-3 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:39:42.950] Nov 5 02:39:42.949: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 02:39:42.951] Nov 5 02:39:42.949: INFO: kube-apiserver-k8s-master-15826033-3 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:39:42.951] Nov 5 02:39:42.949: INFO: Container kube-apiserver ready: true, restart count 0 I1105 02:39:42.951] Nov 5 02:39:42.949: INFO: kube-controller-manager-k8s-master-15826033-3 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:39:42.951] Nov 5 02:39:42.949: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 02:39:42.951] Nov 5 02:39:42.949: INFO: kube-scheduler-k8s-master-15826033-3 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:39:42.952] Nov 5 02:39:42.949: INFO: Container kube-scheduler ready: true, restart count 0 I1105 02:39:42.952] Nov 5 02:39:42.949: INFO: kube-proxy-xz24z started at 2019-11-05 01:35:06 +0000 UTC (0+1 container statuses recorded) I1105 02:39:42.952] Nov 5 02:39:42.949: INFO: Container kube-proxy ready: true, restart count 0 I1105 02:39:43.204] Nov 5 02:39:43.204: INFO: I1105 02:39:43.205] Latency metrics for node k8s-master-15826033-3 I1105 02:39:43.205] Nov 5 02:39:43.204: INFO: I1105 02:39:43.205] Logging node info for node k8s-master-15826033-4 I1105 02:39:43.285] Nov 5 02:39:43.282: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-15826033-4,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-15826033-4,UID:276daef1-6c05-41e3-bacd-abc93d35cc6d,ResourceVersion:12730,Generation:0,CreationTimestamp:2019-11-05 01:35:07 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 0,kubernetes.azure.com/cluster: kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-15826033-4,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.6.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5/providers/Microsoft.Compute/virtualMachines/k8s-master-15826033-4,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 01:35:41 +0000 UTC 2019-11-05 01:35:41 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 02:39:36 +0000 UTC 2019-11-05 01:35:02 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 02:39:36 +0000 UTC 2019-11-05 01:35:02 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 02:39:36 +0000 UTC 2019-11-05 01:35:02 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 02:39:36 +0000 UTC 2019-11-05 01:35:08 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.255.9} {Hostname k8s-master-15826033-4}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:79051fc0bdeb4e32a56acbaf5972ce3a,SystemUUID:C71606B5-7F08-9341-A26D-8A8D9D84FCAE,BootID:4bdf7cf3-2b83-42c7-a7ba-926ebe9ca75b,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 02:39:43.285] Nov 5 02:39:43.282: INFO: I1105 02:39:43.285] Logging kubelet events for node k8s-master-15826033-4 I1105 02:39:43.361] Nov 5 02:39:43.361: INFO: I1105 02:39:43.362] Logging pods the kubelet thinks is on node k8s-master-15826033-4 I1105 02:39:43.446] Nov 5 02:39:43.446: INFO: kube-addon-manager-k8s-master-15826033-4 started at 2019-11-05 01:34:56 +0000 UTC (0+1 container statuses recorded) I1105 02:39:43.446] Nov 5 02:39:43.446: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 02:39:43.446] Nov 5 02:39:43.446: INFO: azure-ip-masq-agent-9nrj8 started at 2019-11-05 01:35:11 +0000 UTC (0+1 container statuses recorded) I1105 02:39:43.447] Nov 5 02:39:43.446: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 02:39:43.447] Nov 5 02:39:43.446: INFO: kube-proxy-rmngk started at 2019-11-05 01:35:11 +0000 UTC (0+1 container statuses recorded) I1105 02:39:43.447] Nov 5 02:39:43.446: INFO: Container kube-proxy ready: true, restart count 0 I1105 02:39:43.447] Nov 5 02:39:43.446: INFO: kube-apiserver-k8s-master-15826033-4 started at 2019-11-05 01:34:55 +0000 UTC (0+1 container statuses recorded) I1105 02:39:43.447] Nov 5 02:39:43.446: INFO: Container kube-apiserver ready: true, restart count 0 I1105 02:39:43.448] Nov 5 02:39:43.446: INFO: kube-controller-manager-k8s-master-15826033-4 started at 2019-11-05 01:34:55 +0000 UTC (0+1 container statuses recorded) I1105 02:39:43.448] Nov 5 02:39:43.446: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 02:39:43.448] Nov 5 02:39:43.446: INFO: kube-scheduler-k8s-master-15826033-4 started at 2019-11-05 01:34:55 +0000 UTC (0+1 container statuses recorded) I1105 02:39:43.448] Nov 5 02:39:43.446: INFO: Container kube-scheduler ready: true, restart count 0 I1105 02:39:43.449] Nov 5 02:39:43.446: INFO: cloud-controller-manager-k8s-master-15826033-4 started at 2019-11-05 01:34:56 +0000 UTC (0+1 container statuses recorded) I1105 02:39:43.449] Nov 5 02:39:43.446: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 02:39:43.693] Nov 5 02:39:43.693: INFO: I1105 02:39:43.693] Latency metrics for node k8s-master-15826033-4 I1105 02:39:43.694] Nov 5 02:39:43.693: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready I1105 02:39:43.772] STEP: Destroying namespace "azuredisk-1320" for this suite. I1105 02:39:58.010] Nov 5 02:39:58.010: INFO: Waiting up to 30s for server preferred namespaced resources to be successfully discovered I1105 02:40:00.939] Nov 5 02:40:00.939: INFO: namespace azuredisk-1320 deletion completed in 17.166832043s I1105 02:40:00.940] I1105 02:40:00.940] • Failure [323.215 seconds] I1105 02:40:00.940] [azuredisk-csi-e2e] Dynamic Provisioning I1105 02:40:00.940] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:33 I1105 02:40:00.940] [multi-az] I1105 02:40:00.940] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:40 I1105 02:40:00.941] should create multiple PV objects, bind to PVCs and attach all to different pods on the same node [It] I1105 02:40:00.941] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:151 I1105 02:40:00.941] I1105 02:40:00.941] Unexpected error: I1105 02:40:00.941] <*errors.errorString | 0xc000937380>: { I1105 02:40:00.942] s: "PersistentVolumeClaims [pvc-pvkjx] not all in phase Bound within 5m0s", I1105 02:40:00.942] } I1105 02:40:00.942] PersistentVolumeClaims [pvc-pvkjx] not all in phase Bound within 5m0s I1105 02:40:00.942] occurred I1105 02:40:00.942] I1105 02:40:00.942] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/testsuites/testsuites.go:278 I1105 02:40:00.943] ------------------------------ I1105 02:40:00.943] [azuredisk-csi-e2e] Dynamic Provisioning [single-az] I1105 02:40:00.943] [env] should retain PV with reclaimPolicy "Retain" I1105 02:40:00.943] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:240 I1105 02:40:00.943] [BeforeEach] [single-az] I1105 02:40:00.943] /go/src/sigs.k8s.io/azuredisk-csi-driver/vendor/k8s.io/kubernetes/test/e2e/framework/framework.go:150 I1105 02:40:00.944] STEP: Creating a kubernetes client I1105 02:40:00.944] Nov 5 02:40:00.939: INFO: >>> kubeConfig: /workspace/aks978705698/kubeconfig/kubeconfig.eastus2.json I1105 02:40:00.944] STEP: Building a namespace api object, basename azuredisk I1105 02:40:01.211] STEP: Binding the e2e-test-privileged-psp PodSecurityPolicy to the default service account in azuredisk-9787 I1105 02:40:01.471] STEP: Waiting for a default service account to be provisioned in namespace I1105 02:40:01.547] [BeforeEach] [single-az] I1105 02:40:01.547] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:58 I1105 02:40:01.547] [It] [env] should retain PV with reclaimPolicy "Retain" I1105 02:40:01.547] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:240 I1105 02:40:01.548] STEP: setting up the StorageClass I1105 02:40:01.548] STEP: creating a StorageClass I1105 02:40:01.632] STEP: setting up the PVC and PV I1105 02:40:01.632] STEP: creating a PVC I1105 02:40:01.717] STEP: waiting for PVC to be in phase "Bound" I1105 02:40:01.717] Nov 5 02:40:01.717: INFO: Waiting up to 5m0s for PersistentVolumeClaims [pvc-nngqh] to have phase Bound I1105 02:40:01.794] Nov 5 02:40:01.794: INFO: PersistentVolumeClaim pvc-nngqh found but phase is Pending instead of Bound. I1105 02:40:03.872] Nov 5 02:40:03.872: INFO: PersistentVolumeClaim pvc-nngqh found but phase is Pending instead of Bound. I1105 02:40:05.949] Nov 5 02:40:05.949: INFO: PersistentVolumeClaim pvc-nngqh found but phase is Pending instead of Bound. I1105 02:40:08.027] Nov 5 02:40:08.027: INFO: PersistentVolumeClaim pvc-nngqh found but phase is Pending instead of Bound. I1105 02:40:10.105] Nov 5 02:40:10.105: INFO: PersistentVolumeClaim pvc-nngqh found but phase is Pending instead of Bound. I1105 02:40:12.182] Nov 5 02:40:12.182: INFO: PersistentVolumeClaim pvc-nngqh found but phase is Pending instead of Bound. I1105 02:40:14.260] Nov 5 02:40:14.259: INFO: PersistentVolumeClaim pvc-nngqh found but phase is Pending instead of Bound. I1105 02:40:16.338] Nov 5 02:40:16.337: INFO: PersistentVolumeClaim pvc-nngqh found but phase is Pending instead of Bound. I1105 02:40:18.415] Nov 5 02:40:18.415: INFO: PersistentVolumeClaim pvc-nngqh found but phase is Pending instead of Bound. I1105 02:40:20.493] Nov 5 02:40:20.493: INFO: PersistentVolumeClaim pvc-nngqh found but phase is Pending instead of Bound. I1105 02:40:22.571] Nov 5 02:40:22.571: INFO: PersistentVolumeClaim pvc-nngqh found but phase is Pending instead of Bound. I1105 02:40:24.649] Nov 5 02:40:24.649: INFO: PersistentVolumeClaim pvc-nngqh found but phase is Pending instead of Bound. I1105 02:40:26.727] Nov 5 02:40:26.727: INFO: PersistentVolumeClaim pvc-nngqh found but phase is Pending instead of Bound. I1105 02:40:28.804] Nov 5 02:40:28.804: INFO: PersistentVolumeClaim pvc-nngqh found but phase is Pending instead of Bound. I1105 02:40:30.882] Nov 5 02:40:30.882: INFO: PersistentVolumeClaim pvc-nngqh found but phase is Pending instead of Bound. I1105 02:40:32.960] Nov 5 02:40:32.960: INFO: PersistentVolumeClaim pvc-nngqh found but phase is Pending instead of Bound. I1105 02:40:35.038] Nov 5 02:40:35.037: INFO: PersistentVolumeClaim pvc-nngqh found but phase is Pending instead of Bound. I1105 02:40:37.138] Nov 5 02:40:37.138: INFO: PersistentVolumeClaim pvc-nngqh found but phase is Pending instead of Bound. I1105 02:40:39.217] Nov 5 02:40:39.217: INFO: PersistentVolumeClaim pvc-nngqh found but phase is Pending instead of Bound. I1105 02:40:41.295] Nov 5 02:40:41.295: INFO: PersistentVolumeClaim pvc-nngqh found but phase is Pending instead of Bound. I1105 02:40:43.372] Nov 5 02:40:43.372: INFO: PersistentVolumeClaim pvc-nngqh found but phase is Pending instead of Bound. I1105 02:40:45.450] Nov 5 02:40:45.450: INFO: PersistentVolumeClaim pvc-nngqh found but phase is Pending instead of Bound. I1105 02:40:47.527] Nov 5 02:40:47.527: INFO: PersistentVolumeClaim pvc-nngqh found but phase is Pending instead of Bound. I1105 02:40:49.606] Nov 5 02:40:49.605: INFO: PersistentVolumeClaim pvc-nngqh found but phase is Pending instead of Bound. I1105 02:40:51.689] Nov 5 02:40:51.689: INFO: PersistentVolumeClaim pvc-nngqh found but phase is Pending instead of Bound. I1105 02:40:53.767] Nov 5 02:40:53.767: INFO: PersistentVolumeClaim pvc-nngqh found but phase is Pending instead of Bound. I1105 02:40:55.845] Nov 5 02:40:55.844: INFO: PersistentVolumeClaim pvc-nngqh found but phase is Pending instead of Bound. I1105 02:40:57.923] Nov 5 02:40:57.923: INFO: PersistentVolumeClaim pvc-nngqh found but phase is Pending instead of Bound. I1105 02:41:00.001] Nov 5 02:41:00.001: INFO: PersistentVolumeClaim pvc-nngqh found but phase is Pending instead of Bound. I1105 02:41:02.079] Nov 5 02:41:02.079: INFO: PersistentVolumeClaim pvc-nngqh found but phase is Pending instead of Bound. I1105 02:41:04.157] Nov 5 02:41:04.157: INFO: PersistentVolumeClaim pvc-nngqh found but phase is Pending instead of Bound. I1105 02:41:06.236] Nov 5 02:41:06.236: INFO: PersistentVolumeClaim pvc-nngqh found but phase is Pending instead of Bound. I1105 02:41:08.314] Nov 5 02:41:08.314: INFO: PersistentVolumeClaim pvc-nngqh found but phase is Pending instead of Bound. I1105 02:41:10.392] Nov 5 02:41:10.391: INFO: PersistentVolumeClaim pvc-nngqh found but phase is Pending instead of Bound. I1105 02:41:12.469] Nov 5 02:41:12.469: INFO: PersistentVolumeClaim pvc-nngqh found but phase is Pending instead of Bound. I1105 02:41:14.546] Nov 5 02:41:14.546: INFO: PersistentVolumeClaim pvc-nngqh found but phase is Pending instead of Bound. I1105 02:41:16.624] Nov 5 02:41:16.624: INFO: PersistentVolumeClaim pvc-nngqh found but phase is Pending instead of Bound. I1105 02:41:18.702] Nov 5 02:41:18.702: INFO: PersistentVolumeClaim pvc-nngqh found but phase is Pending instead of Bound. I1105 02:41:20.779] Nov 5 02:41:20.779: INFO: PersistentVolumeClaim pvc-nngqh found but phase is Pending instead of Bound. I1105 02:41:22.857] Nov 5 02:41:22.856: INFO: PersistentVolumeClaim pvc-nngqh found but phase is Pending instead of Bound. I1105 02:41:24.937] Nov 5 02:41:24.937: INFO: PersistentVolumeClaim pvc-nngqh found but phase is Pending instead of Bound. I1105 02:41:27.015] Nov 5 02:41:27.015: INFO: PersistentVolumeClaim pvc-nngqh found but phase is Pending instead of Bound. I1105 02:41:29.148] Nov 5 02:41:29.147: INFO: PersistentVolumeClaim pvc-nngqh found but phase is Pending instead of Bound. I1105 02:41:31.229] Nov 5 02:41:31.229: INFO: PersistentVolumeClaim pvc-nngqh found but phase is Pending instead of Bound. I1105 02:41:33.307] Nov 5 02:41:33.306: INFO: PersistentVolumeClaim pvc-nngqh found but phase is Pending instead of Bound. I1105 02:41:35.385] Nov 5 02:41:35.384: INFO: PersistentVolumeClaim pvc-nngqh found but phase is Pending instead of Bound. I1105 02:41:37.465] Nov 5 02:41:37.465: INFO: PersistentVolumeClaim pvc-nngqh found but phase is Pending instead of Bound. I1105 02:41:39.543] Nov 5 02:41:39.542: INFO: PersistentVolumeClaim pvc-nngqh found but phase is Pending instead of Bound. I1105 02:41:41.620] Nov 5 02:41:41.620: INFO: PersistentVolumeClaim pvc-nngqh found but phase is Pending instead of Bound. I1105 02:41:43.697] Nov 5 02:41:43.697: INFO: PersistentVolumeClaim pvc-nngqh found but phase is Pending instead of Bound. I1105 02:41:45.775] Nov 5 02:41:45.775: INFO: PersistentVolumeClaim pvc-nngqh found but phase is Pending instead of Bound. I1105 02:41:47.853] Nov 5 02:41:47.853: INFO: PersistentVolumeClaim pvc-nngqh found but phase is Pending instead of Bound. I1105 02:41:49.931] Nov 5 02:41:49.931: INFO: PersistentVolumeClaim pvc-nngqh found but phase is Pending instead of Bound. I1105 02:41:52.008] Nov 5 02:41:52.008: INFO: PersistentVolumeClaim pvc-nngqh found but phase is Pending instead of Bound. I1105 02:41:54.086] Nov 5 02:41:54.085: INFO: PersistentVolumeClaim pvc-nngqh found but phase is Pending instead of Bound. I1105 02:41:56.163] Nov 5 02:41:56.163: INFO: PersistentVolumeClaim pvc-nngqh found but phase is Pending instead of Bound. I1105 02:41:58.242] Nov 5 02:41:58.241: INFO: PersistentVolumeClaim pvc-nngqh found but phase is Pending instead of Bound. I1105 02:42:00.320] Nov 5 02:42:00.320: INFO: PersistentVolumeClaim pvc-nngqh found but phase is Pending instead of Bound. I1105 02:42:02.398] Nov 5 02:42:02.398: INFO: PersistentVolumeClaim pvc-nngqh found but phase is Pending instead of Bound. I1105 02:42:04.476] Nov 5 02:42:04.476: INFO: PersistentVolumeClaim pvc-nngqh found but phase is Pending instead of Bound. I1105 02:42:06.554] Nov 5 02:42:06.554: INFO: PersistentVolumeClaim pvc-nngqh found but phase is Pending instead of Bound. I1105 02:42:08.631] Nov 5 02:42:08.631: INFO: PersistentVolumeClaim pvc-nngqh found but phase is Pending instead of Bound. I1105 02:42:10.709] Nov 5 02:42:10.709: INFO: PersistentVolumeClaim pvc-nngqh found but phase is Pending instead of Bound. I1105 02:42:12.787] Nov 5 02:42:12.786: INFO: PersistentVolumeClaim pvc-nngqh found but phase is Pending instead of Bound. I1105 02:42:14.864] Nov 5 02:42:14.864: INFO: PersistentVolumeClaim pvc-nngqh found but phase is Pending instead of Bound. I1105 02:42:16.942] Nov 5 02:42:16.942: INFO: PersistentVolumeClaim pvc-nngqh found but phase is Pending instead of Bound. I1105 02:42:19.020] Nov 5 02:42:19.020: INFO: PersistentVolumeClaim pvc-nngqh found but phase is Pending instead of Bound. I1105 02:42:21.128] Nov 5 02:42:21.128: INFO: PersistentVolumeClaim pvc-nngqh found but phase is Pending instead of Bound. I1105 02:42:23.207] Nov 5 02:42:23.207: INFO: PersistentVolumeClaim pvc-nngqh found but phase is Pending instead of Bound. I1105 02:42:25.285] Nov 5 02:42:25.285: INFO: PersistentVolumeClaim pvc-nngqh found but phase is Pending instead of Bound. I1105 02:42:27.363] Nov 5 02:42:27.363: INFO: PersistentVolumeClaim pvc-nngqh found but phase is Pending instead of Bound. I1105 02:42:29.441] Nov 5 02:42:29.440: INFO: PersistentVolumeClaim pvc-nngqh found but phase is Pending instead of Bound. I1105 02:42:31.519] Nov 5 02:42:31.518: INFO: PersistentVolumeClaim pvc-nngqh found but phase is Pending instead of Bound. I1105 02:42:33.597] Nov 5 02:42:33.596: INFO: PersistentVolumeClaim pvc-nngqh found but phase is Pending instead of Bound. I1105 02:42:35.674] Nov 5 02:42:35.674: INFO: PersistentVolumeClaim pvc-nngqh found but phase is Pending instead of Bound. I1105 02:42:37.752] Nov 5 02:42:37.752: INFO: PersistentVolumeClaim pvc-nngqh found but phase is Pending instead of Bound. I1105 02:42:39.831] Nov 5 02:42:39.831: INFO: PersistentVolumeClaim pvc-nngqh found but phase is Pending instead of Bound. I1105 02:42:41.909] Nov 5 02:42:41.909: INFO: PersistentVolumeClaim pvc-nngqh found but phase is Pending instead of Bound. I1105 02:42:43.987] Nov 5 02:42:43.986: INFO: PersistentVolumeClaim pvc-nngqh found but phase is Pending instead of Bound. I1105 02:42:46.064] Nov 5 02:42:46.064: INFO: PersistentVolumeClaim pvc-nngqh found but phase is Pending instead of Bound. I1105 02:42:48.142] Nov 5 02:42:48.142: INFO: PersistentVolumeClaim pvc-nngqh found but phase is Pending instead of Bound. I1105 02:42:50.220] Nov 5 02:42:50.220: INFO: PersistentVolumeClaim pvc-nngqh found but phase is Pending instead of Bound. I1105 02:42:52.299] Nov 5 02:42:52.298: INFO: PersistentVolumeClaim pvc-nngqh found but phase is Pending instead of Bound. I1105 02:42:54.377] Nov 5 02:42:54.377: INFO: PersistentVolumeClaim pvc-nngqh found but phase is Pending instead of Bound. I1105 02:42:56.455] Nov 5 02:42:56.455: INFO: PersistentVolumeClaim pvc-nngqh found but phase is Pending instead of Bound. I1105 02:42:58.532] Nov 5 02:42:58.532: INFO: PersistentVolumeClaim pvc-nngqh found but phase is Pending instead of Bound. I1105 02:43:00.611] Nov 5 02:43:00.611: INFO: PersistentVolumeClaim pvc-nngqh found but phase is Pending instead of Bound. I1105 02:43:02.689] Nov 5 02:43:02.688: INFO: PersistentVolumeClaim pvc-nngqh found but phase is Pending instead of Bound. I1105 02:43:04.767] Nov 5 02:43:04.766: INFO: PersistentVolumeClaim pvc-nngqh found but phase is Pending instead of Bound. I1105 02:43:06.845] Nov 5 02:43:06.845: INFO: PersistentVolumeClaim pvc-nngqh found but phase is Pending instead of Bound. I1105 02:43:08.924] Nov 5 02:43:08.923: INFO: PersistentVolumeClaim pvc-nngqh found but phase is Pending instead of Bound. I1105 02:43:11.002] Nov 5 02:43:11.001: INFO: PersistentVolumeClaim pvc-nngqh found but phase is Pending instead of Bound. I1105 02:43:13.082] Nov 5 02:43:13.082: INFO: PersistentVolumeClaim pvc-nngqh found but phase is Pending instead of Bound. I1105 02:43:15.169] Nov 5 02:43:15.169: INFO: PersistentVolumeClaim pvc-nngqh found but phase is Pending instead of Bound. I1105 02:43:17.247] Nov 5 02:43:17.247: INFO: PersistentVolumeClaim pvc-nngqh found but phase is Pending instead of Bound. I1105 02:43:19.325] Nov 5 02:43:19.325: INFO: PersistentVolumeClaim pvc-nngqh found but phase is Pending instead of Bound. I1105 02:43:21.403] Nov 5 02:43:21.403: INFO: PersistentVolumeClaim pvc-nngqh found but phase is Pending instead of Bound. I1105 02:43:23.480] Nov 5 02:43:23.480: INFO: PersistentVolumeClaim pvc-nngqh found but phase is Pending instead of Bound. I1105 02:43:25.558] Nov 5 02:43:25.557: INFO: PersistentVolumeClaim pvc-nngqh found but phase is Pending instead of Bound. I1105 02:43:27.636] Nov 5 02:43:27.636: INFO: PersistentVolumeClaim pvc-nngqh found but phase is Pending instead of Bound. I1105 02:43:29.714] Nov 5 02:43:29.713: INFO: PersistentVolumeClaim pvc-nngqh found but phase is Pending instead of Bound. I1105 02:43:31.792] Nov 5 02:43:31.791: INFO: PersistentVolumeClaim pvc-nngqh found but phase is Pending instead of Bound. I1105 02:43:33.870] Nov 5 02:43:33.870: INFO: PersistentVolumeClaim pvc-nngqh found but phase is Pending instead of Bound. I1105 02:43:35.948] Nov 5 02:43:35.948: INFO: PersistentVolumeClaim pvc-nngqh found but phase is Pending instead of Bound. I1105 02:43:38.026] Nov 5 02:43:38.026: INFO: PersistentVolumeClaim pvc-nngqh found but phase is Pending instead of Bound. I1105 02:43:40.104] Nov 5 02:43:40.104: INFO: PersistentVolumeClaim pvc-nngqh found but phase is Pending instead of Bound. I1105 02:43:42.182] Nov 5 02:43:42.182: INFO: PersistentVolumeClaim pvc-nngqh found but phase is Pending instead of Bound. I1105 02:43:44.261] Nov 5 02:43:44.260: INFO: PersistentVolumeClaim pvc-nngqh found but phase is Pending instead of Bound. I1105 02:43:46.339] Nov 5 02:43:46.338: INFO: PersistentVolumeClaim pvc-nngqh found but phase is Pending instead of Bound. I1105 02:43:48.421] Nov 5 02:43:48.421: INFO: PersistentVolumeClaim pvc-nngqh found but phase is Pending instead of Bound. I1105 02:43:50.502] Nov 5 02:43:50.502: INFO: PersistentVolumeClaim pvc-nngqh found but phase is Pending instead of Bound. I1105 02:43:52.580] Nov 5 02:43:52.580: INFO: PersistentVolumeClaim pvc-nngqh found but phase is Pending instead of Bound. I1105 02:43:54.658] Nov 5 02:43:54.658: INFO: PersistentVolumeClaim pvc-nngqh found but phase is Pending instead of Bound. I1105 02:43:56.735] Nov 5 02:43:56.735: INFO: PersistentVolumeClaim pvc-nngqh found but phase is Pending instead of Bound. I1105 02:43:58.814] Nov 5 02:43:58.813: INFO: PersistentVolumeClaim pvc-nngqh found but phase is Pending instead of Bound. I1105 02:44:00.891] Nov 5 02:44:00.891: INFO: PersistentVolumeClaim pvc-nngqh found but phase is Pending instead of Bound. I1105 02:44:02.969] Nov 5 02:44:02.969: INFO: PersistentVolumeClaim pvc-nngqh found but phase is Pending instead of Bound. I1105 02:44:05.047] Nov 5 02:44:05.047: INFO: PersistentVolumeClaim pvc-nngqh found but phase is Pending instead of Bound. I1105 02:44:07.125] Nov 5 02:44:07.125: INFO: PersistentVolumeClaim pvc-nngqh found but phase is Pending instead of Bound. I1105 02:44:09.203] Nov 5 02:44:09.203: INFO: PersistentVolumeClaim pvc-nngqh found but phase is Pending instead of Bound. I1105 02:44:11.281] Nov 5 02:44:11.281: INFO: PersistentVolumeClaim pvc-nngqh found but phase is Pending instead of Bound. I1105 02:44:13.360] Nov 5 02:44:13.359: INFO: PersistentVolumeClaim pvc-nngqh found but phase is Pending instead of Bound. I1105 02:44:15.442] Nov 5 02:44:15.442: INFO: PersistentVolumeClaim pvc-nngqh found but phase is Pending instead of Bound. I1105 02:44:17.520] Nov 5 02:44:17.519: INFO: PersistentVolumeClaim pvc-nngqh found but phase is Pending instead of Bound. I1105 02:44:19.597] Nov 5 02:44:19.597: INFO: PersistentVolumeClaim pvc-nngqh found but phase is Pending instead of Bound. I1105 02:44:21.676] Nov 5 02:44:21.675: INFO: PersistentVolumeClaim pvc-nngqh found but phase is Pending instead of Bound. I1105 02:44:23.758] Nov 5 02:44:23.758: INFO: PersistentVolumeClaim pvc-nngqh found but phase is Pending instead of Bound. I1105 02:44:25.843] Nov 5 02:44:25.842: INFO: PersistentVolumeClaim pvc-nngqh found but phase is Pending instead of Bound. I1105 02:44:27.924] Nov 5 02:44:27.924: INFO: PersistentVolumeClaim pvc-nngqh found but phase is Pending instead of Bound. I1105 02:44:30.002] Nov 5 02:44:30.002: INFO: PersistentVolumeClaim pvc-nngqh found but phase is Pending instead of Bound. I1105 02:44:32.080] Nov 5 02:44:32.080: INFO: PersistentVolumeClaim pvc-nngqh found but phase is Pending instead of Bound. I1105 02:44:34.159] Nov 5 02:44:34.159: INFO: PersistentVolumeClaim pvc-nngqh found but phase is Pending instead of Bound. I1105 02:44:36.237] Nov 5 02:44:36.236: INFO: PersistentVolumeClaim pvc-nngqh found but phase is Pending instead of Bound. I1105 02:44:38.314] Nov 5 02:44:38.314: INFO: PersistentVolumeClaim pvc-nngqh found but phase is Pending instead of Bound. I1105 02:44:40.392] Nov 5 02:44:40.392: INFO: PersistentVolumeClaim pvc-nngqh found but phase is Pending instead of Bound. I1105 02:44:42.470] Nov 5 02:44:42.470: INFO: PersistentVolumeClaim pvc-nngqh found but phase is Pending instead of Bound. I1105 02:44:44.549] Nov 5 02:44:44.549: INFO: PersistentVolumeClaim pvc-nngqh found but phase is Pending instead of Bound. I1105 02:44:46.627] Nov 5 02:44:46.627: INFO: PersistentVolumeClaim pvc-nngqh found but phase is Pending instead of Bound. I1105 02:44:48.705] Nov 5 02:44:48.705: INFO: PersistentVolumeClaim pvc-nngqh found but phase is Pending instead of Bound. I1105 02:44:50.783] Nov 5 02:44:50.782: INFO: PersistentVolumeClaim pvc-nngqh found but phase is Pending instead of Bound. I1105 02:44:52.861] Nov 5 02:44:52.860: INFO: PersistentVolumeClaim pvc-nngqh found but phase is Pending instead of Bound. I1105 02:44:54.941] Nov 5 02:44:54.941: INFO: PersistentVolumeClaim pvc-nngqh found but phase is Pending instead of Bound. I1105 02:44:57.019] Nov 5 02:44:57.019: INFO: PersistentVolumeClaim pvc-nngqh found but phase is Pending instead of Bound. I1105 02:44:59.096] Nov 5 02:44:59.096: INFO: PersistentVolumeClaim pvc-nngqh found but phase is Pending instead of Bound. I1105 02:45:01.174] Nov 5 02:45:01.174: INFO: PersistentVolumeClaim pvc-nngqh found but phase is Pending instead of Bound. I1105 02:45:03.175] Nov 5 02:45:03.174: INFO: Unexpected error occurred: PersistentVolumeClaims [pvc-nngqh] not all in phase Bound within 5m0s I1105 02:45:03.175] [AfterEach] [single-az] I1105 02:45:03.175] /go/src/sigs.k8s.io/azuredisk-csi-driver/vendor/k8s.io/kubernetes/test/e2e/framework/framework.go:151 I1105 02:45:03.176] STEP: Collecting events from namespace "azuredisk-9787". I1105 02:45:03.253] STEP: Found 3 events. I1105 02:45:03.253] Nov 5 02:45:03.252: INFO: At 2019-11-05 02:40:01 +0000 UTC - event for pvc-nngqh: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "disk.csi.azure.com" or manually created by system administrator I1105 02:45:03.253] Nov 5 02:45:03.252: INFO: At 2019-11-05 02:40:01 +0000 UTC - event for pvc-nngqh: {disk.csi.azure.com_csi-azuredisk-controller-6f6587bfc6-8vkfl_1376914b-ff6d-11e9-93ab-ba2bee589b8c } Provisioning: External provisioner is provisioning volume for claim "azuredisk-9787/pvc-nngqh" I1105 02:45:03.254] Nov 5 02:45:03.252: INFO: At 2019-11-05 02:40:01 +0000 UTC - event for pvc-nngqh: {disk.csi.azure.com_csi-azuredisk-controller-6f6587bfc6-8vkfl_1376914b-ff6d-11e9-93ab-ba2bee589b8c } ProvisioningFailed: failed to provision volume with StorageClass "azuredisk-9787-disk.csi.azure.com-dynamic-sc-tp74c": rpc error: code = Unknown desc = compute.DisksClient#CreateOrUpdate: Failure sending request: StatusCode=400 -- Original Error: Code="InvalidAvailabilityZone" Message="The zone(s) '' for resource 'Microsoft.Compute/disks/pvc-0ef79f1a-4817-47f1-986d-95e6ab0bce77' is not supported. The supported zones for location 'eastus2' are '1,2,3'" I1105 02:45:03.330] Nov 5 02:45:03.330: INFO: POD NODE PHASE GRACE CONDITIONS I1105 02:45:03.330] Nov 5 02:45:03.330: INFO: I1105 02:45:03.561] Nov 5 02:45:03.560: INFO: I1105 02:45:03.561] Logging node info for node k8s-agentpool-15826033-vmss000000 I1105 02:45:03.641] Nov 5 02:45:03.637: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-agentpool-15826033-vmss000000,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-agentpool-15826033-vmss000000,UID:15fa874e-54ec-42cd-b243-d3baa512150f,ResourceVersion:13603,Generation:0,CreationTimestamp:2019-11-05 01:34:17 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{agentpool: agentpool,beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_D4s_v3,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 0,kubernetes.azure.com/cluster: kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5,kubernetes.azure.com/role: agent,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-agentpool-15826033-vmss000000,kubernetes.io/os: linux,kubernetes.io/role: agent,node-role.kubernetes.io/agent: ,storageprofile: managed,storagetier: Premium_LRS,topology.disk.csi.azure.com/zone: 0,},Annotations:map[string]string{csi.volume.kubernetes.io/nodeid: {"disk.csi.azure.com":"k8s-agentpool-15826033-vmss000000"},node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.2.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool-15826033-vmss/virtualMachines/0,Unschedulable:false,Taints:[],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-csi-disk.csi.azure.com: {{8 0} {} 8 DecimalSI},cpu: {{4 0} {} 4 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{16797044736 0} {} 16403364Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-csi-disk.csi.azure.com: {{8 0} {} 8 DecimalSI},cpu: {{4 0} {} 4 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{16010612736 0} {} 15635364Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 01:34:56 +0000 UTC 2019-11-05 01:34:56 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 02:44:28 +0000 UTC 2019-11-05 01:34:17 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 02:44:28 +0000 UTC 2019-11-05 01:34:17 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 02:44:28 +0000 UTC 2019-11-05 01:34:17 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 02:44:28 +0000 UTC 2019-11-05 01:34:29 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.0.4} {Hostname k8s-agentpool-15826033-vmss000000}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:afe00792177d49a98f6a44d03d08732d,SystemUUID:4F1777BF-A5C3-004E-81C4-90F0B54C02A0,BootID:13b6a628-ae89-44af-b891-89461030adcb,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[k8sprowinternal.azurecr.io/azuredisk-csi@sha256:75d4ff5d7cd845d6e810c67796ecb037e852fb849ec1eb5b3194564008e6d1e0 k8sprowinternal.azurecr.io/azuredisk-csi:e2e-3e3ed045c6faa509dbd8d2890ee9f56aa2934887] 286930414} {[k8s.gcr.io/kubernetes-dashboard-amd64@sha256:0ae6b69432e78069c5ce2bcde0fe409c5c4d6f0f4d9cd50a17974fea38898747 k8s.gcr.io/kubernetes-dashboard-amd64:v1.10.1] 121711221} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/metrics-server-amd64@sha256:49a9f12f7067d11f42c803dbe61ed2c1299959ad85cb315b25ff7eef8e6b8892 k8s.gcr.io/metrics-server-amd64:v0.2.1] 42541759} {[mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume@sha256:4fd30d43947d4a54fc89ead7985beecfd3c9b2a93a0655a373b1608ab90bd5af mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume:v0.0.7] 22909487} {[quay.io/k8scsi/livenessprobe@sha256:8c9c8a0f330acc2e558ae6624309447d02522e76aa35af23fa19b88abf4381cb quay.io/k8scsi/livenessprobe:v1.0.2] 17415381} {[quay.io/k8scsi/csi-node-driver-registrar@sha256:13daf82fb99e951a4bff8ae5fc7c17c3a8fe7130be6400990d8f6076c32d4599 quay.io/k8scsi/csi-node-driver-registrar:v1.1.0] 15815995} {[mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume@sha256:23d8c6033f02a1ecad05127ebdc931bb871264228661bc122704b0974e4d9fdd mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume:1.0.8] 1159025} {[busybox@sha256:8ccbac733d19c0dd4d70b4f0c1e12245b5fa3ad24758a11035ee505c629c0796 busybox:1.29] 1154361} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 02:45:03.641] Nov 5 02:45:03.638: INFO: I1105 02:45:03.641] Logging kubelet events for node k8s-agentpool-15826033-vmss000000 I1105 02:45:03.715] Nov 5 02:45:03.715: INFO: I1105 02:45:03.715] Logging pods the kubelet thinks is on node k8s-agentpool-15826033-vmss000000 I1105 02:45:03.797] Nov 5 02:45:03.797: INFO: blobfuse-flexvol-installer-bsvqt started at 2019-11-05 01:34:34 +0000 UTC (0+1 container statuses recorded) I1105 02:45:03.798] Nov 5 02:45:03.797: INFO: Container blobfuse-flexvol-installer ready: true, restart count 0 I1105 02:45:03.798] Nov 5 02:45:03.797: INFO: azure-ip-masq-agent-vx86b started at 2019-11-05 01:34:34 +0000 UTC (0+1 container statuses recorded) I1105 02:45:03.798] Nov 5 02:45:03.797: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 02:45:03.798] Nov 5 02:45:03.797: INFO: keyvault-flexvolume-qhnfr started at 2019-11-05 01:34:34 +0000 UTC (0+1 container statuses recorded) I1105 02:45:03.799] Nov 5 02:45:03.797: INFO: Container keyvault-flexvolume ready: true, restart count 0 I1105 02:45:03.799] Nov 5 02:45:03.797: INFO: metrics-server-864ffbc5c-sdxps started at 2019-11-05 01:34:34 +0000 UTC (0+1 container statuses recorded) I1105 02:45:03.799] Nov 5 02:45:03.797: INFO: Container metrics-server ready: true, restart count 0 I1105 02:45:03.799] Nov 5 02:45:03.797: INFO: kube-proxy-wxblc started at 2019-11-05 01:34:34 +0000 UTC (0+1 container statuses recorded) I1105 02:45:03.800] Nov 5 02:45:03.797: INFO: Container kube-proxy ready: true, restart count 0 I1105 02:45:03.800] Nov 5 02:45:03.797: INFO: kubernetes-dashboard-66dd8b8df7-ks284 started at 2019-11-05 01:34:35 +0000 UTC (0+1 container statuses recorded) I1105 02:45:03.800] Nov 5 02:45:03.797: INFO: Container kubernetes-dashboard ready: true, restart count 0 I1105 02:45:03.800] Nov 5 02:45:03.797: INFO: csi-azuredisk-node-whq2g started at 2019-11-05 01:38:24 +0000 UTC (0+3 container statuses recorded) I1105 02:45:03.801] Nov 5 02:45:03.797: INFO: Container azuredisk ready: true, restart count 0 I1105 02:45:03.801] Nov 5 02:45:03.797: INFO: Container liveness-probe ready: true, restart count 0 I1105 02:45:03.801] Nov 5 02:45:03.797: INFO: Container node-driver-registrar ready: true, restart count 0 I1105 02:45:04.054] Nov 5 02:45:04.054: INFO: I1105 02:45:04.054] Latency metrics for node k8s-agentpool-15826033-vmss000000 I1105 02:45:04.054] Nov 5 02:45:04.054: INFO: I1105 02:45:04.055] Logging node info for node k8s-agentpool-15826033-vmss000001 I1105 02:45:04.135] Nov 5 02:45:04.131: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-agentpool-15826033-vmss000001,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-agentpool-15826033-vmss000001,UID:4a8ea605-cc0b-4e3b-9060-d8867d269e6c,ResourceVersion:13671,Generation:0,CreationTimestamp:2019-11-05 01:34:09 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{agentpool: agentpool,beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_D4s_v3,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 1,kubernetes.azure.com/cluster: kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5,kubernetes.azure.com/role: agent,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-agentpool-15826033-vmss000001,kubernetes.io/os: linux,kubernetes.io/role: agent,node-role.kubernetes.io/agent: ,storageprofile: managed,storagetier: Premium_LRS,topology.disk.csi.azure.com/zone: 1,},Annotations:map[string]string{csi.volume.kubernetes.io/nodeid: {"disk.csi.azure.com":"k8s-agentpool-15826033-vmss000001"},node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.0.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool-15826033-vmss/virtualMachines/1,Unschedulable:false,Taints:[],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-csi-disk.csi.azure.com: {{8 0} {} 8 DecimalSI},cpu: {{4 0} {} 4 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{16797044736 0} {} 16403364Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-csi-disk.csi.azure.com: {{8 0} {} 8 DecimalSI},cpu: {{4 0} {} 4 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{16010612736 0} {} 15635364Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 01:34:57 +0000 UTC 2019-11-05 01:34:57 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 02:44:50 +0000 UTC 2019-11-05 01:34:08 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 02:44:50 +0000 UTC 2019-11-05 01:34:08 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 02:44:50 +0000 UTC 2019-11-05 01:34:08 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 02:44:50 +0000 UTC 2019-11-05 01:34:29 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.0.5} {Hostname k8s-agentpool-15826033-vmss000001}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:b6b9fbf4a2fd4fa5a7ae472415fe30b8,SystemUUID:0162B3A4-3955-9A46-ABD4-A37BD75848CC,BootID:ee93003f-d47c-4118-9975-57365b3443a5,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[k8sprowinternal.azurecr.io/azuredisk-csi@sha256:75d4ff5d7cd845d6e810c67796ecb037e852fb849ec1eb5b3194564008e6d1e0 k8sprowinternal.azurecr.io/azuredisk-csi:e2e-3e3ed045c6faa509dbd8d2890ee9f56aa2934887] 286930414} {[gcr.io/kubernetes-helm/tiller@sha256:f6d8f4ab9ba993b5f5b60a6edafe86352eabe474ffeb84cb6c79b8866dce45d1 gcr.io/kubernetes-helm/tiller:v2.11.0] 71821984} {[quay.io/k8scsi/csi-provisioner@sha256:d657c839dce87324fe2b677302913f9386f885f8746be7bea0ced5b0844e3433 quay.io/k8scsi/csi-provisioner:v1.3.1] 50596998} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[quay.io/k8scsi/csi-attacher@sha256:26fccd7a99d973845df1193b46ebdcc6ab8dc5f6e6be319750c471fce1742d13 quay.io/k8scsi/csi-attacher:v1.2.0] 46226754} {[quay.io/k8scsi/csi-cluster-driver-registrar@sha256:fafd75ae5442f192cfa8c2e792903aee30d5884b62e802e4464b0a895d21e3ef quay.io/k8scsi/csi-cluster-driver-registrar:v1.0.1] 45874691} {[quay.io/k8scsi/csi-snapshotter@sha256:a49e0da1af6f2bf717e41ba1eee8b5e6a1cbd66a709dd92cc43fe475fe2589eb quay.io/k8scsi/csi-snapshotter:v1.1.0] 44325352} {[k8s.gcr.io/coredns@sha256:e83beb5e43f8513fa735e77ffc5859640baea30a882a11cc75c4c3244a737d3c k8s.gcr.io/coredns:1.5.0] 42488424} {[mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume@sha256:4fd30d43947d4a54fc89ead7985beecfd3c9b2a93a0655a373b1608ab90bd5af mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume:v0.0.7] 22909487} {[quay.io/k8scsi/livenessprobe@sha256:8c9c8a0f330acc2e558ae6624309447d02522e76aa35af23fa19b88abf4381cb quay.io/k8scsi/livenessprobe:v1.0.2] 17415381} {[quay.io/k8scsi/csi-node-driver-registrar@sha256:13daf82fb99e951a4bff8ae5fc7c17c3a8fe7130be6400990d8f6076c32d4599 quay.io/k8scsi/csi-node-driver-registrar:v1.1.0] 15815995} {[quay.io/k8scsi/livenessprobe@sha256:dde617756e0f602adc566ab71fd885f1dad451ad3fb063ac991c95a2ff47aea5 quay.io/k8scsi/livenessprobe:v1.1.0] 14967303} {[mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume@sha256:23d8c6033f02a1ecad05127ebdc931bb871264228661bc122704b0974e4d9fdd mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume:1.0.8] 1159025} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 02:45:04.135] Nov 5 02:45:04.132: INFO: I1105 02:45:04.135] Logging kubelet events for node k8s-agentpool-15826033-vmss000001 I1105 02:45:04.209] Nov 5 02:45:04.208: INFO: I1105 02:45:04.209] Logging pods the kubelet thinks is on node k8s-agentpool-15826033-vmss000001 I1105 02:45:04.292] Nov 5 02:45:04.291: INFO: keyvault-flexvolume-rdlcj started at 2019-11-05 01:34:35 +0000 UTC (0+1 container statuses recorded) I1105 02:45:04.292] Nov 5 02:45:04.292: INFO: Container keyvault-flexvolume ready: true, restart count 0 I1105 02:45:04.292] Nov 5 02:45:04.292: INFO: kube-proxy-6kh57 started at 2019-11-05 01:34:35 +0000 UTC (0+1 container statuses recorded) I1105 02:45:04.293] Nov 5 02:45:04.292: INFO: Container kube-proxy ready: true, restart count 0 I1105 02:45:04.293] Nov 5 02:45:04.292: INFO: tiller-deploy-54c96cb5df-8d9fs started at 2019-11-05 01:34:35 +0000 UTC (0+1 container statuses recorded) I1105 02:45:04.293] Nov 5 02:45:04.292: INFO: Container tiller ready: true, restart count 0 I1105 02:45:04.293] Nov 5 02:45:04.292: INFO: csi-azuredisk-node-ghv5w started at 2019-11-05 01:38:24 +0000 UTC (0+3 container statuses recorded) I1105 02:45:04.293] Nov 5 02:45:04.292: INFO: Container azuredisk ready: true, restart count 0 I1105 02:45:04.294] Nov 5 02:45:04.292: INFO: Container liveness-probe ready: true, restart count 0 I1105 02:45:04.294] Nov 5 02:45:04.292: INFO: Container node-driver-registrar ready: true, restart count 0 I1105 02:45:04.294] Nov 5 02:45:04.292: INFO: csi-azuredisk-controller-6f6587bfc6-8vkfl started at 2019-11-05 01:38:24 +0000 UTC (0+6 container statuses recorded) I1105 02:45:04.294] Nov 5 02:45:04.292: INFO: Container azuredisk ready: true, restart count 0 I1105 02:45:04.295] Nov 5 02:45:04.292: INFO: Container cluster-driver-registrar ready: true, restart count 0 I1105 02:45:04.295] Nov 5 02:45:04.292: INFO: Container csi-attacher ready: true, restart count 0 I1105 02:45:04.295] Nov 5 02:45:04.292: INFO: Container csi-provisioner ready: true, restart count 0 I1105 02:45:04.295] Nov 5 02:45:04.292: INFO: Container csi-snapshotter ready: true, restart count 0 I1105 02:45:04.295] Nov 5 02:45:04.292: INFO: Container liveness-probe ready: true, restart count 0 I1105 02:45:04.296] Nov 5 02:45:04.292: INFO: blobfuse-flexvol-installer-rw5dw started at 2019-11-05 01:34:35 +0000 UTC (0+1 container statuses recorded) I1105 02:45:04.296] Nov 5 02:45:04.292: INFO: Container blobfuse-flexvol-installer ready: true, restart count 0 I1105 02:45:04.296] Nov 5 02:45:04.292: INFO: coredns-7f68dcdbdb-jt522 started at 2019-11-05 01:34:35 +0000 UTC (0+1 container statuses recorded) I1105 02:45:04.296] Nov 5 02:45:04.292: INFO: Container coredns ready: true, restart count 0 I1105 02:45:04.297] Nov 5 02:45:04.292: INFO: azure-ip-masq-agent-6c6vn started at 2019-11-05 01:34:35 +0000 UTC (0+1 container statuses recorded) I1105 02:45:04.297] Nov 5 02:45:04.292: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 02:45:04.542] Nov 5 02:45:04.541: INFO: I1105 02:45:04.542] Latency metrics for node k8s-agentpool-15826033-vmss000001 I1105 02:45:04.542] Nov 5 02:45:04.541: INFO: I1105 02:45:04.542] Logging node info for node k8s-master-15826033-0 I1105 02:45:04.621] Nov 5 02:45:04.618: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-15826033-0,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-15826033-0,UID:aceb0231-4645-4a9a-b296-90ada851986b,ResourceVersion:13628,Generation:0,CreationTimestamp:2019-11-05 01:35:06 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 1,kubernetes.azure.com/cluster: kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-15826033-0,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.4.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5/providers/Microsoft.Compute/virtualMachines/k8s-master-15826033-0,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 01:35:20 +0000 UTC 2019-11-05 01:35:20 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 02:44:36 +0000 UTC 2019-11-05 01:34:57 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 02:44:36 +0000 UTC 2019-11-05 01:34:57 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 02:44:36 +0000 UTC 2019-11-05 01:34:57 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 02:44:36 +0000 UTC 2019-11-05 01:35:07 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.255.5} {Hostname k8s-master-15826033-0}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:1915756c22a540dc8eb30493987e9a43,SystemUUID:742C969C-5A71-D744-8E61-F27E38B56EB2,BootID:03f0672e-8fa9-4309-956f-8e3c371902c3,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 02:45:04.621] Nov 5 02:45:04.619: INFO: I1105 02:45:04.622] Logging kubelet events for node k8s-master-15826033-0 I1105 02:45:04.696] Nov 5 02:45:04.696: INFO: I1105 02:45:04.696] Logging pods the kubelet thinks is on node k8s-master-15826033-0 I1105 02:45:04.778] Nov 5 02:45:04.778: INFO: kube-proxy-7d2px started at 2019-11-05 01:35:06 +0000 UTC (0+1 container statuses recorded) I1105 02:45:04.779] Nov 5 02:45:04.778: INFO: Container kube-proxy ready: true, restart count 0 I1105 02:45:04.779] Nov 5 02:45:04.778: INFO: cloud-controller-manager-k8s-master-15826033-0 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:45:04.779] Nov 5 02:45:04.778: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 02:45:04.779] Nov 5 02:45:04.778: INFO: kube-addon-manager-k8s-master-15826033-0 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:45:04.780] Nov 5 02:45:04.778: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 02:45:04.780] Nov 5 02:45:04.778: INFO: kube-apiserver-k8s-master-15826033-0 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:45:04.780] Nov 5 02:45:04.778: INFO: Container kube-apiserver ready: true, restart count 0 I1105 02:45:04.780] Nov 5 02:45:04.778: INFO: kube-controller-manager-k8s-master-15826033-0 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:45:04.781] Nov 5 02:45:04.778: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 02:45:04.781] Nov 5 02:45:04.778: INFO: kube-scheduler-k8s-master-15826033-0 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:45:04.781] Nov 5 02:45:04.778: INFO: Container kube-scheduler ready: true, restart count 0 I1105 02:45:04.781] Nov 5 02:45:04.778: INFO: azure-ip-masq-agent-wgdpx started at 2019-11-05 01:35:06 +0000 UTC (0+1 container statuses recorded) I1105 02:45:04.782] Nov 5 02:45:04.778: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 02:45:05.033] Nov 5 02:45:05.032: INFO: I1105 02:45:05.033] Latency metrics for node k8s-master-15826033-0 I1105 02:45:05.033] Nov 5 02:45:05.032: INFO: I1105 02:45:05.034] Logging node info for node k8s-master-15826033-1 I1105 02:45:05.151] Nov 5 02:45:05.148: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-15826033-1,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-15826033-1,UID:1a13ec61-c89a-456c-8c94-bfc8e717c2e7,ResourceVersion:13630,Generation:0,CreationTimestamp:2019-11-05 01:35:07 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 1,kubernetes.azure.com/cluster: kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-15826033-1,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.5.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5/providers/Microsoft.Compute/virtualMachines/k8s-master-15826033-1,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 01:35:31 +0000 UTC 2019-11-05 01:35:31 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 02:44:36 +0000 UTC 2019-11-05 01:35:00 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 02:44:36 +0000 UTC 2019-11-05 01:35:00 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 02:44:36 +0000 UTC 2019-11-05 01:35:00 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 02:44:36 +0000 UTC 2019-11-05 01:35:07 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.255.6} {Hostname k8s-master-15826033-1}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:e0f2bccddd5b4555b180d14081ab59a8,SystemUUID:AE126631-CAD2-B54F-B22E-1F499342A4D6,BootID:509b9a9d-b046-4861-aa35-8bcb77465de2,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 02:45:05.152] Nov 5 02:45:05.149: INFO: I1105 02:45:05.152] Logging kubelet events for node k8s-master-15826033-1 I1105 02:45:05.226] Nov 5 02:45:05.226: INFO: I1105 02:45:05.226] Logging pods the kubelet thinks is on node k8s-master-15826033-1 I1105 02:45:05.307] Nov 5 02:45:05.306: INFO: kube-controller-manager-k8s-master-15826033-1 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:45:05.307] Nov 5 02:45:05.306: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 02:45:05.307] Nov 5 02:45:05.306: INFO: kube-scheduler-k8s-master-15826033-1 started at 2019-11-05 01:36:16 +0000 UTC (0+1 container statuses recorded) I1105 02:45:05.307] Nov 5 02:45:05.306: INFO: Container kube-scheduler ready: true, restart count 0 I1105 02:45:05.307] Nov 5 02:45:05.306: INFO: azure-ip-masq-agent-w6dpr started at 2019-11-05 01:35:10 +0000 UTC (0+1 container statuses recorded) I1105 02:45:05.308] Nov 5 02:45:05.306: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 02:45:05.308] Nov 5 02:45:05.306: INFO: kube-proxy-8kmr5 started at 2019-11-05 01:35:10 +0000 UTC (0+1 container statuses recorded) I1105 02:45:05.308] Nov 5 02:45:05.306: INFO: Container kube-proxy ready: true, restart count 0 I1105 02:45:05.308] Nov 5 02:45:05.306: INFO: cloud-controller-manager-k8s-master-15826033-1 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:45:05.308] Nov 5 02:45:05.306: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 02:45:05.309] Nov 5 02:45:05.306: INFO: kube-addon-manager-k8s-master-15826033-1 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:45:05.309] Nov 5 02:45:05.306: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 02:45:05.309] Nov 5 02:45:05.306: INFO: kube-apiserver-k8s-master-15826033-1 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:45:05.309] Nov 5 02:45:05.306: INFO: Container kube-apiserver ready: true, restart count 0 I1105 02:45:05.550] Nov 5 02:45:05.550: INFO: I1105 02:45:05.551] Latency metrics for node k8s-master-15826033-1 I1105 02:45:05.551] Nov 5 02:45:05.550: INFO: I1105 02:45:05.551] Logging node info for node k8s-master-15826033-2 I1105 02:45:05.629] Nov 5 02:45:05.627: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-15826033-2,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-15826033-2,UID:acb3a851-d6df-4549-ba56-4b7e96553c17,ResourceVersion:13544,Generation:0,CreationTimestamp:2019-11-05 01:34:09 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 2,kubernetes.azure.com/cluster: kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-15826033-2,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.1.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5/providers/Microsoft.Compute/virtualMachines/k8s-master-15826033-2,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 01:34:44 +0000 UTC 2019-11-05 01:34:44 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 02:44:07 +0000 UTC 2019-11-05 01:34:07 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 02:44:07 +0000 UTC 2019-11-05 01:34:07 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 02:44:07 +0000 UTC 2019-11-05 01:34:07 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 02:44:07 +0000 UTC 2019-11-05 01:34:29 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{Hostname k8s-master-15826033-2} {InternalIP 10.240.255.7}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:0885b0c2f6174106bde4fa0eae2cbc6b,SystemUUID:C176121C-5D7A-6243-9B05-8B68F8022B64,BootID:30430638-7ab6-4761-b984-2387e1a3f5b2,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 02:45:05.630] Nov 5 02:45:05.627: INFO: I1105 02:45:05.630] Logging kubelet events for node k8s-master-15826033-2 I1105 02:45:05.704] Nov 5 02:45:05.704: INFO: I1105 02:45:05.705] Logging pods the kubelet thinks is on node k8s-master-15826033-2 I1105 02:45:05.786] Nov 5 02:45:05.786: INFO: kube-addon-manager-k8s-master-15826033-2 started at 2019-11-05 01:33:54 +0000 UTC (0+1 container statuses recorded) I1105 02:45:05.786] Nov 5 02:45:05.786: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 02:45:05.786] Nov 5 02:45:05.786: INFO: kube-apiserver-k8s-master-15826033-2 started at 2019-11-05 01:33:54 +0000 UTC (0+1 container statuses recorded) I1105 02:45:05.786] Nov 5 02:45:05.786: INFO: Container kube-apiserver ready: true, restart count 0 I1105 02:45:05.787] Nov 5 02:45:05.786: INFO: kube-controller-manager-k8s-master-15826033-2 started at 2019-11-05 01:33:54 +0000 UTC (0+1 container statuses recorded) I1105 02:45:05.787] Nov 5 02:45:05.786: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 02:45:05.787] Nov 5 02:45:05.786: INFO: kube-scheduler-k8s-master-15826033-2 started at 2019-11-05 01:33:54 +0000 UTC (0+1 container statuses recorded) I1105 02:45:05.787] Nov 5 02:45:05.786: INFO: Container kube-scheduler ready: true, restart count 0 I1105 02:45:05.787] Nov 5 02:45:05.786: INFO: azure-ip-masq-agent-hw7s5 started at 2019-11-05 01:34:34 +0000 UTC (0+1 container statuses recorded) I1105 02:45:05.788] Nov 5 02:45:05.786: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 02:45:05.788] Nov 5 02:45:05.786: INFO: kube-proxy-5xpp6 started at 2019-11-05 01:34:34 +0000 UTC (0+1 container statuses recorded) I1105 02:45:05.788] Nov 5 02:45:05.786: INFO: Container kube-proxy ready: true, restart count 0 I1105 02:45:05.788] Nov 5 02:45:05.786: INFO: cloud-controller-manager-k8s-master-15826033-2 started at 2019-11-05 01:33:54 +0000 UTC (0+1 container statuses recorded) I1105 02:45:05.789] Nov 5 02:45:05.786: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 02:45:06.044] Nov 5 02:45:06.044: INFO: I1105 02:45:06.044] Latency metrics for node k8s-master-15826033-2 I1105 02:45:06.044] Nov 5 02:45:06.044: INFO: I1105 02:45:06.045] Logging node info for node k8s-master-15826033-3 I1105 02:45:06.129] Nov 5 02:45:06.127: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-15826033-3,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-15826033-3,UID:b7702272-1876-434e-b6a0-17fa34c6757f,ResourceVersion:13624,Generation:0,CreationTimestamp:2019-11-05 01:35:06 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 0,kubernetes.azure.com/cluster: kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-15826033-3,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.3.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5/providers/Microsoft.Compute/virtualMachines/k8s-master-15826033-3,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 01:35:17 +0000 UTC 2019-11-05 01:35:17 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 02:44:35 +0000 UTC 2019-11-05 01:35:00 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 02:44:35 +0000 UTC 2019-11-05 01:35:00 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 02:44:35 +0000 UTC 2019-11-05 01:35:00 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 02:44:35 +0000 UTC 2019-11-05 01:35:06 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.255.8} {Hostname k8s-master-15826033-3}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:fbf366963961422c9d90bedf6639c8e1,SystemUUID:5388948D-1A8D-EA41-B8B8-7F4523EDF5AA,BootID:42f5884c-7418-486a-8cf3-75188a331f38,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 02:45:06.130] Nov 5 02:45:06.127: INFO: I1105 02:45:06.130] Logging kubelet events for node k8s-master-15826033-3 I1105 02:45:06.205] Nov 5 02:45:06.204: INFO: I1105 02:45:06.205] Logging pods the kubelet thinks is on node k8s-master-15826033-3 I1105 02:45:06.286] Nov 5 02:45:06.286: INFO: kube-scheduler-k8s-master-15826033-3 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:45:06.287] Nov 5 02:45:06.286: INFO: Container kube-scheduler ready: true, restart count 0 I1105 02:45:06.287] Nov 5 02:45:06.286: INFO: kube-proxy-xz24z started at 2019-11-05 01:35:06 +0000 UTC (0+1 container statuses recorded) I1105 02:45:06.287] Nov 5 02:45:06.286: INFO: Container kube-proxy ready: true, restart count 0 I1105 02:45:06.287] Nov 5 02:45:06.286: INFO: azure-ip-masq-agent-dqfpw started at 2019-11-05 01:35:06 +0000 UTC (0+1 container statuses recorded) I1105 02:45:06.288] Nov 5 02:45:06.286: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 02:45:06.288] Nov 5 02:45:06.286: INFO: cloud-controller-manager-k8s-master-15826033-3 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:45:06.288] Nov 5 02:45:06.286: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 02:45:06.288] Nov 5 02:45:06.286: INFO: kube-addon-manager-k8s-master-15826033-3 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:45:06.289] Nov 5 02:45:06.286: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 02:45:06.289] Nov 5 02:45:06.286: INFO: kube-apiserver-k8s-master-15826033-3 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:45:06.289] Nov 5 02:45:06.286: INFO: Container kube-apiserver ready: true, restart count 0 I1105 02:45:06.289] Nov 5 02:45:06.286: INFO: kube-controller-manager-k8s-master-15826033-3 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:45:06.290] Nov 5 02:45:06.286: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 02:45:06.534] Nov 5 02:45:06.534: INFO: I1105 02:45:06.534] Latency metrics for node k8s-master-15826033-3 I1105 02:45:06.534] Nov 5 02:45:06.534: INFO: I1105 02:45:06.535] Logging node info for node k8s-master-15826033-4 I1105 02:45:06.614] Nov 5 02:45:06.611: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-15826033-4,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-15826033-4,UID:276daef1-6c05-41e3-bacd-abc93d35cc6d,ResourceVersion:13631,Generation:0,CreationTimestamp:2019-11-05 01:35:07 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 0,kubernetes.azure.com/cluster: kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-15826033-4,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.6.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5/providers/Microsoft.Compute/virtualMachines/k8s-master-15826033-4,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 01:35:41 +0000 UTC 2019-11-05 01:35:41 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 02:44:36 +0000 UTC 2019-11-05 01:35:02 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 02:44:36 +0000 UTC 2019-11-05 01:35:02 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 02:44:36 +0000 UTC 2019-11-05 01:35:02 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 02:44:36 +0000 UTC 2019-11-05 01:35:08 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.255.9} {Hostname k8s-master-15826033-4}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:79051fc0bdeb4e32a56acbaf5972ce3a,SystemUUID:C71606B5-7F08-9341-A26D-8A8D9D84FCAE,BootID:4bdf7cf3-2b83-42c7-a7ba-926ebe9ca75b,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 02:45:06.614] Nov 5 02:45:06.611: INFO: I1105 02:45:06.614] Logging kubelet events for node k8s-master-15826033-4 I1105 02:45:06.689] Nov 5 02:45:06.689: INFO: I1105 02:45:06.689] Logging pods the kubelet thinks is on node k8s-master-15826033-4 I1105 02:45:06.771] Nov 5 02:45:06.771: INFO: azure-ip-masq-agent-9nrj8 started at 2019-11-05 01:35:11 +0000 UTC (0+1 container statuses recorded) I1105 02:45:06.771] Nov 5 02:45:06.771: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 02:45:06.772] Nov 5 02:45:06.771: INFO: kube-proxy-rmngk started at 2019-11-05 01:35:11 +0000 UTC (0+1 container statuses recorded) I1105 02:45:06.772] Nov 5 02:45:06.771: INFO: Container kube-proxy ready: true, restart count 0 I1105 02:45:06.772] Nov 5 02:45:06.771: INFO: kube-apiserver-k8s-master-15826033-4 started at 2019-11-05 01:34:55 +0000 UTC (0+1 container statuses recorded) I1105 02:45:06.772] Nov 5 02:45:06.771: INFO: Container kube-apiserver ready: true, restart count 0 I1105 02:45:06.772] Nov 5 02:45:06.771: INFO: kube-controller-manager-k8s-master-15826033-4 started at 2019-11-05 01:34:55 +0000 UTC (0+1 container statuses recorded) I1105 02:45:06.773] Nov 5 02:45:06.771: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 02:45:06.773] Nov 5 02:45:06.771: INFO: kube-scheduler-k8s-master-15826033-4 started at 2019-11-05 01:34:55 +0000 UTC (0+1 container statuses recorded) I1105 02:45:06.773] Nov 5 02:45:06.771: INFO: Container kube-scheduler ready: true, restart count 0 I1105 02:45:06.773] Nov 5 02:45:06.771: INFO: cloud-controller-manager-k8s-master-15826033-4 started at 2019-11-05 01:34:56 +0000 UTC (0+1 container statuses recorded) I1105 02:45:06.773] Nov 5 02:45:06.771: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 02:45:06.774] Nov 5 02:45:06.771: INFO: kube-addon-manager-k8s-master-15826033-4 started at 2019-11-05 01:34:56 +0000 UTC (0+1 container statuses recorded) I1105 02:45:06.774] Nov 5 02:45:06.771: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 02:45:07.016] Nov 5 02:45:07.016: INFO: I1105 02:45:07.016] Latency metrics for node k8s-master-15826033-4 I1105 02:45:07.016] Nov 5 02:45:07.016: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready I1105 02:45:07.094] STEP: Destroying namespace "azuredisk-9787" for this suite. I1105 02:45:21.332] Nov 5 02:45:21.332: INFO: Waiting up to 30s for server preferred namespaced resources to be successfully discovered I1105 02:45:24.199] Nov 5 02:45:24.199: INFO: namespace azuredisk-9787 deletion completed in 17.105065146s I1105 02:45:24.200] I1105 02:45:24.200] • Failure [323.260 seconds] I1105 02:45:24.200] [azuredisk-csi-e2e] Dynamic Provisioning I1105 02:45:24.200] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:33 I1105 02:45:24.200] [single-az] I1105 02:45:24.201] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:36 I1105 02:45:24.201] [env] should retain PV with reclaimPolicy "Retain" [It] I1105 02:45:24.201] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:240 I1105 02:45:24.201] I1105 02:45:24.201] Unexpected error: I1105 02:45:24.202] <*errors.errorString | 0xc000e48000>: { I1105 02:45:24.202] s: "PersistentVolumeClaims [pvc-nngqh] not all in phase Bound within 5m0s", I1105 02:45:24.202] } I1105 02:45:24.202] PersistentVolumeClaims [pvc-nngqh] not all in phase Bound within 5m0s I1105 02:45:24.202] occurred I1105 02:45:24.202] I1105 02:45:24.203] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/testsuites/testsuites.go:278 I1105 02:45:24.203] ------------------------------ I1105 02:45:24.203] [azuredisk-csi-e2e] Dynamic Provisioning [single-az] I1105 02:45:24.203] should delete PV with reclaimPolicy "Delete" I1105 02:45:24.203] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:221 I1105 02:45:24.203] [BeforeEach] [single-az] I1105 02:45:24.203] /go/src/sigs.k8s.io/azuredisk-csi-driver/vendor/k8s.io/kubernetes/test/e2e/framework/framework.go:150 I1105 02:45:24.204] STEP: Creating a kubernetes client I1105 02:45:24.204] Nov 5 02:45:24.199: INFO: >>> kubeConfig: /workspace/aks978705698/kubeconfig/kubeconfig.eastus2.json I1105 02:45:24.204] STEP: Building a namespace api object, basename azuredisk I1105 02:45:24.436] STEP: Binding the e2e-test-privileged-psp PodSecurityPolicy to the default service account in azuredisk-8754 I1105 02:45:24.700] STEP: Waiting for a default service account to be provisioned in namespace I1105 02:45:24.775] [BeforeEach] [single-az] I1105 02:45:24.776] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:58 I1105 02:45:24.776] [It] should delete PV with reclaimPolicy "Delete" I1105 02:45:24.776] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:221 I1105 02:45:24.776] STEP: setting up the StorageClass I1105 02:45:24.777] STEP: creating a StorageClass I1105 02:45:24.858] STEP: setting up the PVC and PV I1105 02:45:24.858] STEP: creating a PVC I1105 02:45:24.946] STEP: waiting for PVC to be in phase "Bound" I1105 02:45:24.947] Nov 5 02:45:24.946: INFO: Waiting up to 5m0s for PersistentVolumeClaims [pvc-6ch89] to have phase Bound I1105 02:45:25.023] Nov 5 02:45:25.023: INFO: PersistentVolumeClaim pvc-6ch89 found but phase is Pending instead of Bound. I1105 02:45:27.101] Nov 5 02:45:27.101: INFO: PersistentVolumeClaim pvc-6ch89 found but phase is Pending instead of Bound. I1105 02:45:29.179] Nov 5 02:45:29.179: INFO: PersistentVolumeClaim pvc-6ch89 found but phase is Pending instead of Bound. I1105 02:45:31.256] Nov 5 02:45:31.256: INFO: PersistentVolumeClaim pvc-6ch89 found but phase is Pending instead of Bound. I1105 02:45:33.335] Nov 5 02:45:33.335: INFO: PersistentVolumeClaim pvc-6ch89 found but phase is Pending instead of Bound. I1105 02:45:35.412] Nov 5 02:45:35.412: INFO: PersistentVolumeClaim pvc-6ch89 found but phase is Pending instead of Bound. I1105 02:45:37.490] Nov 5 02:45:37.490: INFO: PersistentVolumeClaim pvc-6ch89 found but phase is Pending instead of Bound. I1105 02:45:39.568] Nov 5 02:45:39.568: INFO: PersistentVolumeClaim pvc-6ch89 found but phase is Pending instead of Bound. I1105 02:45:41.646] Nov 5 02:45:41.645: INFO: PersistentVolumeClaim pvc-6ch89 found but phase is Pending instead of Bound. I1105 02:45:43.723] Nov 5 02:45:43.723: INFO: PersistentVolumeClaim pvc-6ch89 found but phase is Pending instead of Bound. I1105 02:45:45.800] Nov 5 02:45:45.800: INFO: PersistentVolumeClaim pvc-6ch89 found but phase is Pending instead of Bound. I1105 02:45:47.878] Nov 5 02:45:47.878: INFO: PersistentVolumeClaim pvc-6ch89 found but phase is Pending instead of Bound. I1105 02:45:49.959] Nov 5 02:45:49.959: INFO: PersistentVolumeClaim pvc-6ch89 found but phase is Pending instead of Bound. I1105 02:45:52.039] Nov 5 02:45:52.039: INFO: PersistentVolumeClaim pvc-6ch89 found but phase is Pending instead of Bound. I1105 02:45:54.119] Nov 5 02:45:54.119: INFO: PersistentVolumeClaim pvc-6ch89 found but phase is Pending instead of Bound. I1105 02:45:56.197] Nov 5 02:45:56.197: INFO: PersistentVolumeClaim pvc-6ch89 found but phase is Pending instead of Bound. I1105 02:45:58.274] Nov 5 02:45:58.274: INFO: PersistentVolumeClaim pvc-6ch89 found but phase is Pending instead of Bound. I1105 02:46:00.352] Nov 5 02:46:00.351: INFO: PersistentVolumeClaim pvc-6ch89 found but phase is Pending instead of Bound. I1105 02:46:02.429] Nov 5 02:46:02.429: INFO: PersistentVolumeClaim pvc-6ch89 found but phase is Pending instead of Bound. I1105 02:46:04.506] Nov 5 02:46:04.506: INFO: PersistentVolumeClaim pvc-6ch89 found but phase is Pending instead of Bound. I1105 02:46:06.584] Nov 5 02:46:06.584: INFO: PersistentVolumeClaim pvc-6ch89 found but phase is Pending instead of Bound. I1105 02:46:08.662] Nov 5 02:46:08.661: INFO: PersistentVolumeClaim pvc-6ch89 found but phase is Pending instead of Bound. I1105 02:46:10.739] Nov 5 02:46:10.738: INFO: PersistentVolumeClaim pvc-6ch89 found but phase is Pending instead of Bound. I1105 02:46:12.816] Nov 5 02:46:12.816: INFO: PersistentVolumeClaim pvc-6ch89 found but phase is Pending instead of Bound. I1105 02:46:14.894] Nov 5 02:46:14.894: INFO: PersistentVolumeClaim pvc-6ch89 found but phase is Pending instead of Bound. I1105 02:46:16.971] Nov 5 02:46:16.971: INFO: PersistentVolumeClaim pvc-6ch89 found but phase is Pending instead of Bound. I1105 02:46:19.049] Nov 5 02:46:19.049: INFO: PersistentVolumeClaim pvc-6ch89 found but phase is Pending instead of Bound. I1105 02:46:21.142] Nov 5 02:46:21.142: INFO: PersistentVolumeClaim pvc-6ch89 found but phase is Pending instead of Bound. I1105 02:46:23.219] Nov 5 02:46:23.219: INFO: PersistentVolumeClaim pvc-6ch89 found but phase is Pending instead of Bound. I1105 02:46:25.296] Nov 5 02:46:25.296: INFO: PersistentVolumeClaim pvc-6ch89 found but phase is Pending instead of Bound. I1105 02:46:27.374] Nov 5 02:46:27.374: INFO: PersistentVolumeClaim pvc-6ch89 found but phase is Pending instead of Bound. I1105 02:46:29.452] Nov 5 02:46:29.452: INFO: PersistentVolumeClaim pvc-6ch89 found but phase is Pending instead of Bound. I1105 02:46:31.530] Nov 5 02:46:31.530: INFO: PersistentVolumeClaim pvc-6ch89 found but phase is Pending instead of Bound. I1105 02:46:33.607] Nov 5 02:46:33.607: INFO: PersistentVolumeClaim pvc-6ch89 found but phase is Pending instead of Bound. I1105 02:46:35.685] Nov 5 02:46:35.685: INFO: PersistentVolumeClaim pvc-6ch89 found but phase is Pending instead of Bound. I1105 02:46:37.763] Nov 5 02:46:37.763: INFO: PersistentVolumeClaim pvc-6ch89 found but phase is Pending instead of Bound. I1105 02:46:39.841] Nov 5 02:46:39.840: INFO: PersistentVolumeClaim pvc-6ch89 found but phase is Pending instead of Bound. I1105 02:46:41.918] Nov 5 02:46:41.918: INFO: PersistentVolumeClaim pvc-6ch89 found but phase is Pending instead of Bound. I1105 02:46:43.996] Nov 5 02:46:43.996: INFO: PersistentVolumeClaim pvc-6ch89 found but phase is Pending instead of Bound. I1105 02:46:46.074] Nov 5 02:46:46.074: INFO: PersistentVolumeClaim pvc-6ch89 found but phase is Pending instead of Bound. I1105 02:46:48.151] Nov 5 02:46:48.151: INFO: PersistentVolumeClaim pvc-6ch89 found but phase is Pending instead of Bound. I1105 02:46:50.229] Nov 5 02:46:50.228: INFO: PersistentVolumeClaim pvc-6ch89 found but phase is Pending instead of Bound. I1105 02:46:52.328] Nov 5 02:46:52.328: INFO: PersistentVolumeClaim pvc-6ch89 found but phase is Pending instead of Bound. I1105 02:46:54.406] Nov 5 02:46:54.406: INFO: PersistentVolumeClaim pvc-6ch89 found but phase is Pending instead of Bound. I1105 02:46:56.483] Nov 5 02:46:56.483: INFO: PersistentVolumeClaim pvc-6ch89 found but phase is Pending instead of Bound. I1105 02:46:58.561] Nov 5 02:46:58.561: INFO: PersistentVolumeClaim pvc-6ch89 found but phase is Pending instead of Bound. I1105 02:47:00.641] Nov 5 02:47:00.640: INFO: PersistentVolumeClaim pvc-6ch89 found but phase is Pending instead of Bound. I1105 02:47:02.718] Nov 5 02:47:02.717: INFO: PersistentVolumeClaim pvc-6ch89 found but phase is Pending instead of Bound. I1105 02:47:04.795] Nov 5 02:47:04.795: INFO: PersistentVolumeClaim pvc-6ch89 found but phase is Pending instead of Bound. I1105 02:47:06.873] Nov 5 02:47:06.873: INFO: PersistentVolumeClaim pvc-6ch89 found but phase is Pending instead of Bound. I1105 02:47:08.950] Nov 5 02:47:08.950: INFO: PersistentVolumeClaim pvc-6ch89 found but phase is Pending instead of Bound. I1105 02:47:11.028] Nov 5 02:47:11.027: INFO: PersistentVolumeClaim pvc-6ch89 found but phase is Pending instead of Bound. I1105 02:47:13.114] Nov 5 02:47:13.114: INFO: PersistentVolumeClaim pvc-6ch89 found but phase is Pending instead of Bound. I1105 02:47:15.191] Nov 5 02:47:15.191: INFO: PersistentVolumeClaim pvc-6ch89 found but phase is Pending instead of Bound. I1105 02:47:17.269] Nov 5 02:47:17.269: INFO: PersistentVolumeClaim pvc-6ch89 found but phase is Pending instead of Bound. I1105 02:47:19.347] Nov 5 02:47:19.346: INFO: PersistentVolumeClaim pvc-6ch89 found but phase is Pending instead of Bound. I1105 02:47:21.428] Nov 5 02:47:21.428: INFO: PersistentVolumeClaim pvc-6ch89 found but phase is Pending instead of Bound. I1105 02:47:23.506] Nov 5 02:47:23.506: INFO: PersistentVolumeClaim pvc-6ch89 found but phase is Pending instead of Bound. I1105 02:47:25.584] Nov 5 02:47:25.584: INFO: PersistentVolumeClaim pvc-6ch89 found but phase is Pending instead of Bound. I1105 02:47:27.661] Nov 5 02:47:27.661: INFO: PersistentVolumeClaim pvc-6ch89 found but phase is Pending instead of Bound. I1105 02:47:29.739] Nov 5 02:47:29.739: INFO: PersistentVolumeClaim pvc-6ch89 found but phase is Pending instead of Bound. I1105 02:47:31.817] Nov 5 02:47:31.817: INFO: PersistentVolumeClaim pvc-6ch89 found but phase is Pending instead of Bound. I1105 02:47:33.894] Nov 5 02:47:33.894: INFO: PersistentVolumeClaim pvc-6ch89 found but phase is Pending instead of Bound. I1105 02:47:35.972] Nov 5 02:47:35.972: INFO: PersistentVolumeClaim pvc-6ch89 found but phase is Pending instead of Bound. I1105 02:47:38.050] Nov 5 02:47:38.050: INFO: PersistentVolumeClaim pvc-6ch89 found but phase is Pending instead of Bound. I1105 02:47:40.128] Nov 5 02:47:40.128: INFO: PersistentVolumeClaim pvc-6ch89 found but phase is Pending instead of Bound. I1105 02:47:42.206] Nov 5 02:47:42.206: INFO: PersistentVolumeClaim pvc-6ch89 found but phase is Pending instead of Bound. I1105 02:47:44.285] Nov 5 02:47:44.285: INFO: PersistentVolumeClaim pvc-6ch89 found but phase is Pending instead of Bound. I1105 02:47:46.363] Nov 5 02:47:46.363: INFO: PersistentVolumeClaim pvc-6ch89 found but phase is Pending instead of Bound. I1105 02:47:48.444] Nov 5 02:47:48.444: INFO: PersistentVolumeClaim pvc-6ch89 found but phase is Pending instead of Bound. I1105 02:47:50.522] Nov 5 02:47:50.522: INFO: PersistentVolumeClaim pvc-6ch89 found but phase is Pending instead of Bound. I1105 02:47:52.600] Nov 5 02:47:52.599: INFO: PersistentVolumeClaim pvc-6ch89 found but phase is Pending instead of Bound. I1105 02:47:54.677] Nov 5 02:47:54.677: INFO: PersistentVolumeClaim pvc-6ch89 found but phase is Pending instead of Bound. I1105 02:47:56.755] Nov 5 02:47:56.754: INFO: PersistentVolumeClaim pvc-6ch89 found but phase is Pending instead of Bound. I1105 02:47:58.832] Nov 5 02:47:58.832: INFO: PersistentVolumeClaim pvc-6ch89 found but phase is Pending instead of Bound. I1105 02:48:00.909] Nov 5 02:48:00.909: INFO: PersistentVolumeClaim pvc-6ch89 found but phase is Pending instead of Bound. I1105 02:48:02.987] Nov 5 02:48:02.987: INFO: PersistentVolumeClaim pvc-6ch89 found but phase is Pending instead of Bound. I1105 02:48:05.065] Nov 5 02:48:05.064: INFO: PersistentVolumeClaim pvc-6ch89 found but phase is Pending instead of Bound. I1105 02:48:07.142] Nov 5 02:48:07.142: INFO: PersistentVolumeClaim pvc-6ch89 found but phase is Pending instead of Bound. I1105 02:48:09.220] Nov 5 02:48:09.220: INFO: PersistentVolumeClaim pvc-6ch89 found but phase is Pending instead of Bound. I1105 02:48:11.298] Nov 5 02:48:11.298: INFO: PersistentVolumeClaim pvc-6ch89 found but phase is Pending instead of Bound. I1105 02:48:13.375] Nov 5 02:48:13.375: INFO: PersistentVolumeClaim pvc-6ch89 found but phase is Pending instead of Bound. I1105 02:48:15.453] Nov 5 02:48:15.453: INFO: PersistentVolumeClaim pvc-6ch89 found but phase is Pending instead of Bound. I1105 02:48:17.530] Nov 5 02:48:17.530: INFO: PersistentVolumeClaim pvc-6ch89 found but phase is Pending instead of Bound. I1105 02:48:19.608] Nov 5 02:48:19.607: INFO: PersistentVolumeClaim pvc-6ch89 found but phase is Pending instead of Bound. I1105 02:48:21.685] Nov 5 02:48:21.685: INFO: PersistentVolumeClaim pvc-6ch89 found but phase is Pending instead of Bound. I1105 02:48:23.763] Nov 5 02:48:23.763: INFO: PersistentVolumeClaim pvc-6ch89 found but phase is Pending instead of Bound. I1105 02:48:25.840] Nov 5 02:48:25.840: INFO: PersistentVolumeClaim pvc-6ch89 found but phase is Pending instead of Bound. I1105 02:48:27.918] Nov 5 02:48:27.918: INFO: PersistentVolumeClaim pvc-6ch89 found but phase is Pending instead of Bound. I1105 02:48:29.996] Nov 5 02:48:29.996: INFO: PersistentVolumeClaim pvc-6ch89 found but phase is Pending instead of Bound. I1105 02:48:32.074] Nov 5 02:48:32.073: INFO: PersistentVolumeClaim pvc-6ch89 found but phase is Pending instead of Bound. I1105 02:48:34.152] Nov 5 02:48:34.151: INFO: PersistentVolumeClaim pvc-6ch89 found but phase is Pending instead of Bound. I1105 02:48:36.229] Nov 5 02:48:36.229: INFO: PersistentVolumeClaim pvc-6ch89 found but phase is Pending instead of Bound. I1105 02:48:38.306] Nov 5 02:48:38.306: INFO: PersistentVolumeClaim pvc-6ch89 found but phase is Pending instead of Bound. I1105 02:48:40.384] Nov 5 02:48:40.384: INFO: PersistentVolumeClaim pvc-6ch89 found but phase is Pending instead of Bound. I1105 02:48:42.462] Nov 5 02:48:42.462: INFO: PersistentVolumeClaim pvc-6ch89 found but phase is Pending instead of Bound. I1105 02:48:44.539] Nov 5 02:48:44.539: INFO: PersistentVolumeClaim pvc-6ch89 found but phase is Pending instead of Bound. I1105 02:48:46.618] Nov 5 02:48:46.618: INFO: PersistentVolumeClaim pvc-6ch89 found but phase is Pending instead of Bound. I1105 02:48:48.695] Nov 5 02:48:48.695: INFO: PersistentVolumeClaim pvc-6ch89 found but phase is Pending instead of Bound. I1105 02:48:50.773] Nov 5 02:48:50.773: INFO: PersistentVolumeClaim pvc-6ch89 found but phase is Pending instead of Bound. I1105 02:48:52.850] Nov 5 02:48:52.850: INFO: PersistentVolumeClaim pvc-6ch89 found but phase is Pending instead of Bound. I1105 02:48:54.928] Nov 5 02:48:54.928: INFO: PersistentVolumeClaim pvc-6ch89 found but phase is Pending instead of Bound. I1105 02:48:57.006] Nov 5 02:48:57.006: INFO: PersistentVolumeClaim pvc-6ch89 found but phase is Pending instead of Bound. I1105 02:48:59.083] Nov 5 02:48:59.083: INFO: PersistentVolumeClaim pvc-6ch89 found but phase is Pending instead of Bound. I1105 02:49:01.161] Nov 5 02:49:01.160: INFO: PersistentVolumeClaim pvc-6ch89 found but phase is Pending instead of Bound. I1105 02:49:03.239] Nov 5 02:49:03.238: INFO: PersistentVolumeClaim pvc-6ch89 found but phase is Pending instead of Bound. I1105 02:49:05.319] Nov 5 02:49:05.319: INFO: PersistentVolumeClaim pvc-6ch89 found but phase is Pending instead of Bound. I1105 02:49:07.397] Nov 5 02:49:07.397: INFO: PersistentVolumeClaim pvc-6ch89 found but phase is Pending instead of Bound. I1105 02:49:09.474] Nov 5 02:49:09.474: INFO: PersistentVolumeClaim pvc-6ch89 found but phase is Pending instead of Bound. I1105 02:49:11.552] Nov 5 02:49:11.552: INFO: PersistentVolumeClaim pvc-6ch89 found but phase is Pending instead of Bound. I1105 02:49:13.629] Nov 5 02:49:13.629: INFO: PersistentVolumeClaim pvc-6ch89 found but phase is Pending instead of Bound. I1105 02:49:15.707] Nov 5 02:49:15.707: INFO: PersistentVolumeClaim pvc-6ch89 found but phase is Pending instead of Bound. I1105 02:49:17.785] Nov 5 02:49:17.785: INFO: PersistentVolumeClaim pvc-6ch89 found but phase is Pending instead of Bound. I1105 02:49:19.863] Nov 5 02:49:19.863: INFO: PersistentVolumeClaim pvc-6ch89 found but phase is Pending instead of Bound. I1105 02:49:21.941] Nov 5 02:49:21.941: INFO: PersistentVolumeClaim pvc-6ch89 found but phase is Pending instead of Bound. I1105 02:49:24.018] Nov 5 02:49:24.018: INFO: PersistentVolumeClaim pvc-6ch89 found but phase is Pending instead of Bound. I1105 02:49:26.096] Nov 5 02:49:26.096: INFO: PersistentVolumeClaim pvc-6ch89 found but phase is Pending instead of Bound. I1105 02:49:28.173] Nov 5 02:49:28.173: INFO: PersistentVolumeClaim pvc-6ch89 found but phase is Pending instead of Bound. I1105 02:49:30.250] Nov 5 02:49:30.250: INFO: PersistentVolumeClaim pvc-6ch89 found but phase is Pending instead of Bound. I1105 02:49:32.328] Nov 5 02:49:32.328: INFO: PersistentVolumeClaim pvc-6ch89 found but phase is Pending instead of Bound. I1105 02:49:34.405] Nov 5 02:49:34.405: INFO: PersistentVolumeClaim pvc-6ch89 found but phase is Pending instead of Bound. I1105 02:49:36.483] Nov 5 02:49:36.483: INFO: PersistentVolumeClaim pvc-6ch89 found but phase is Pending instead of Bound. I1105 02:49:38.561] Nov 5 02:49:38.561: INFO: PersistentVolumeClaim pvc-6ch89 found but phase is Pending instead of Bound. I1105 02:49:40.638] Nov 5 02:49:40.638: INFO: PersistentVolumeClaim pvc-6ch89 found but phase is Pending instead of Bound. I1105 02:49:42.716] Nov 5 02:49:42.716: INFO: PersistentVolumeClaim pvc-6ch89 found but phase is Pending instead of Bound. I1105 02:49:44.793] Nov 5 02:49:44.793: INFO: PersistentVolumeClaim pvc-6ch89 found but phase is Pending instead of Bound. I1105 02:49:46.871] Nov 5 02:49:46.871: INFO: PersistentVolumeClaim pvc-6ch89 found but phase is Pending instead of Bound. I1105 02:49:48.949] Nov 5 02:49:48.948: INFO: PersistentVolumeClaim pvc-6ch89 found but phase is Pending instead of Bound. I1105 02:49:51.026] Nov 5 02:49:51.026: INFO: PersistentVolumeClaim pvc-6ch89 found but phase is Pending instead of Bound. I1105 02:49:53.144] Nov 5 02:49:53.144: INFO: PersistentVolumeClaim pvc-6ch89 found but phase is Pending instead of Bound. I1105 02:49:55.222] Nov 5 02:49:55.222: INFO: PersistentVolumeClaim pvc-6ch89 found but phase is Pending instead of Bound. I1105 02:49:57.300] Nov 5 02:49:57.300: INFO: PersistentVolumeClaim pvc-6ch89 found but phase is Pending instead of Bound. I1105 02:49:59.377] Nov 5 02:49:59.377: INFO: PersistentVolumeClaim pvc-6ch89 found but phase is Pending instead of Bound. I1105 02:50:01.455] Nov 5 02:50:01.455: INFO: PersistentVolumeClaim pvc-6ch89 found but phase is Pending instead of Bound. I1105 02:50:03.532] Nov 5 02:50:03.532: INFO: PersistentVolumeClaim pvc-6ch89 found but phase is Pending instead of Bound. I1105 02:50:05.610] Nov 5 02:50:05.609: INFO: PersistentVolumeClaim pvc-6ch89 found but phase is Pending instead of Bound. I1105 02:50:07.687] Nov 5 02:50:07.687: INFO: PersistentVolumeClaim pvc-6ch89 found but phase is Pending instead of Bound. I1105 02:50:09.765] Nov 5 02:50:09.765: INFO: PersistentVolumeClaim pvc-6ch89 found but phase is Pending instead of Bound. I1105 02:50:11.843] Nov 5 02:50:11.842: INFO: PersistentVolumeClaim pvc-6ch89 found but phase is Pending instead of Bound. I1105 02:50:13.920] Nov 5 02:50:13.920: INFO: PersistentVolumeClaim pvc-6ch89 found but phase is Pending instead of Bound. I1105 02:50:15.997] Nov 5 02:50:15.997: INFO: PersistentVolumeClaim pvc-6ch89 found but phase is Pending instead of Bound. I1105 02:50:18.075] Nov 5 02:50:18.075: INFO: PersistentVolumeClaim pvc-6ch89 found but phase is Pending instead of Bound. I1105 02:50:20.152] Nov 5 02:50:20.152: INFO: PersistentVolumeClaim pvc-6ch89 found but phase is Pending instead of Bound. I1105 02:50:22.230] Nov 5 02:50:22.230: INFO: PersistentVolumeClaim pvc-6ch89 found but phase is Pending instead of Bound. I1105 02:50:24.308] Nov 5 02:50:24.307: INFO: PersistentVolumeClaim pvc-6ch89 found but phase is Pending instead of Bound. I1105 02:50:26.308] Nov 5 02:50:26.308: INFO: Unexpected error occurred: PersistentVolumeClaims [pvc-6ch89] not all in phase Bound within 5m0s I1105 02:50:26.308] [AfterEach] [single-az] I1105 02:50:26.308] /go/src/sigs.k8s.io/azuredisk-csi-driver/vendor/k8s.io/kubernetes/test/e2e/framework/framework.go:151 I1105 02:50:26.309] STEP: Collecting events from namespace "azuredisk-8754". I1105 02:50:26.386] STEP: Found 3 events. I1105 02:50:26.386] Nov 5 02:50:26.386: INFO: At 2019-11-05 02:45:24 +0000 UTC - event for pvc-6ch89: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "disk.csi.azure.com" or manually created by system administrator I1105 02:50:26.387] Nov 5 02:50:26.386: INFO: At 2019-11-05 02:45:24 +0000 UTC - event for pvc-6ch89: {disk.csi.azure.com_csi-azuredisk-controller-6f6587bfc6-8vkfl_1376914b-ff6d-11e9-93ab-ba2bee589b8c } Provisioning: External provisioner is provisioning volume for claim "azuredisk-8754/pvc-6ch89" I1105 02:50:26.387] Nov 5 02:50:26.386: INFO: At 2019-11-05 02:45:25 +0000 UTC - event for pvc-6ch89: {disk.csi.azure.com_csi-azuredisk-controller-6f6587bfc6-8vkfl_1376914b-ff6d-11e9-93ab-ba2bee589b8c } ProvisioningFailed: failed to provision volume with StorageClass "azuredisk-8754-disk.csi.azure.com-dynamic-sc-rvqkb": rpc error: code = Unknown desc = compute.DisksClient#CreateOrUpdate: Failure sending request: StatusCode=400 -- Original Error: Code="InvalidAvailabilityZone" Message="The zone(s) '' for resource 'Microsoft.Compute/disks/pvc-5aa7ecda-04df-4896-b774-e1f87555735f' is not supported. The supported zones for location 'eastus2' are '1,2,3'" I1105 02:50:26.463] Nov 5 02:50:26.463: INFO: POD NODE PHASE GRACE CONDITIONS I1105 02:50:26.463] Nov 5 02:50:26.463: INFO: I1105 02:50:26.694] Nov 5 02:50:26.694: INFO: I1105 02:50:26.694] Logging node info for node k8s-agentpool-15826033-vmss000000 I1105 02:50:26.774] Nov 5 02:50:26.771: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-agentpool-15826033-vmss000000,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-agentpool-15826033-vmss000000,UID:15fa874e-54ec-42cd-b243-d3baa512150f,ResourceVersion:14505,Generation:0,CreationTimestamp:2019-11-05 01:34:17 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{agentpool: agentpool,beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_D4s_v3,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 0,kubernetes.azure.com/cluster: kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5,kubernetes.azure.com/role: agent,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-agentpool-15826033-vmss000000,kubernetes.io/os: linux,kubernetes.io/role: agent,node-role.kubernetes.io/agent: ,storageprofile: managed,storagetier: Premium_LRS,topology.disk.csi.azure.com/zone: 0,},Annotations:map[string]string{csi.volume.kubernetes.io/nodeid: {"disk.csi.azure.com":"k8s-agentpool-15826033-vmss000000"},node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.2.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool-15826033-vmss/virtualMachines/0,Unschedulable:false,Taints:[],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-csi-disk.csi.azure.com: {{8 0} {} 8 DecimalSI},cpu: {{4 0} {} 4 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{16797044736 0} {} 16403364Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-csi-disk.csi.azure.com: {{8 0} {} 8 DecimalSI},cpu: {{4 0} {} 4 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{16010612736 0} {} 15635364Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 01:34:56 +0000 UTC 2019-11-05 01:34:56 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 02:49:29 +0000 UTC 2019-11-05 01:34:17 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 02:49:29 +0000 UTC 2019-11-05 01:34:17 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 02:49:29 +0000 UTC 2019-11-05 01:34:17 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 02:49:29 +0000 UTC 2019-11-05 01:34:29 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.0.4} {Hostname k8s-agentpool-15826033-vmss000000}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:afe00792177d49a98f6a44d03d08732d,SystemUUID:4F1777BF-A5C3-004E-81C4-90F0B54C02A0,BootID:13b6a628-ae89-44af-b891-89461030adcb,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[k8sprowinternal.azurecr.io/azuredisk-csi@sha256:75d4ff5d7cd845d6e810c67796ecb037e852fb849ec1eb5b3194564008e6d1e0 k8sprowinternal.azurecr.io/azuredisk-csi:e2e-3e3ed045c6faa509dbd8d2890ee9f56aa2934887] 286930414} {[k8s.gcr.io/kubernetes-dashboard-amd64@sha256:0ae6b69432e78069c5ce2bcde0fe409c5c4d6f0f4d9cd50a17974fea38898747 k8s.gcr.io/kubernetes-dashboard-amd64:v1.10.1] 121711221} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/metrics-server-amd64@sha256:49a9f12f7067d11f42c803dbe61ed2c1299959ad85cb315b25ff7eef8e6b8892 k8s.gcr.io/metrics-server-amd64:v0.2.1] 42541759} {[mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume@sha256:4fd30d43947d4a54fc89ead7985beecfd3c9b2a93a0655a373b1608ab90bd5af mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume:v0.0.7] 22909487} {[quay.io/k8scsi/livenessprobe@sha256:8c9c8a0f330acc2e558ae6624309447d02522e76aa35af23fa19b88abf4381cb quay.io/k8scsi/livenessprobe:v1.0.2] 17415381} {[quay.io/k8scsi/csi-node-driver-registrar@sha256:13daf82fb99e951a4bff8ae5fc7c17c3a8fe7130be6400990d8f6076c32d4599 quay.io/k8scsi/csi-node-driver-registrar:v1.1.0] 15815995} {[mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume@sha256:23d8c6033f02a1ecad05127ebdc931bb871264228661bc122704b0974e4d9fdd mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume:1.0.8] 1159025} {[busybox@sha256:8ccbac733d19c0dd4d70b4f0c1e12245b5fa3ad24758a11035ee505c629c0796 busybox:1.29] 1154361} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 02:50:26.774] Nov 5 02:50:26.771: INFO: I1105 02:50:26.774] Logging kubelet events for node k8s-agentpool-15826033-vmss000000 I1105 02:50:26.849] Nov 5 02:50:26.849: INFO: I1105 02:50:26.849] Logging pods the kubelet thinks is on node k8s-agentpool-15826033-vmss000000 I1105 02:50:26.933] Nov 5 02:50:26.933: INFO: csi-azuredisk-node-whq2g started at 2019-11-05 01:38:24 +0000 UTC (0+3 container statuses recorded) I1105 02:50:26.933] Nov 5 02:50:26.933: INFO: Container azuredisk ready: true, restart count 0 I1105 02:50:26.933] Nov 5 02:50:26.933: INFO: Container liveness-probe ready: true, restart count 0 I1105 02:50:26.934] Nov 5 02:50:26.933: INFO: Container node-driver-registrar ready: true, restart count 0 I1105 02:50:26.934] Nov 5 02:50:26.933: INFO: blobfuse-flexvol-installer-bsvqt started at 2019-11-05 01:34:34 +0000 UTC (0+1 container statuses recorded) I1105 02:50:26.934] Nov 5 02:50:26.933: INFO: Container blobfuse-flexvol-installer ready: true, restart count 0 I1105 02:50:26.934] Nov 5 02:50:26.933: INFO: azure-ip-masq-agent-vx86b started at 2019-11-05 01:34:34 +0000 UTC (0+1 container statuses recorded) I1105 02:50:26.935] Nov 5 02:50:26.933: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 02:50:26.935] Nov 5 02:50:26.933: INFO: keyvault-flexvolume-qhnfr started at 2019-11-05 01:34:34 +0000 UTC (0+1 container statuses recorded) I1105 02:50:26.935] Nov 5 02:50:26.933: INFO: Container keyvault-flexvolume ready: true, restart count 0 I1105 02:50:26.935] Nov 5 02:50:26.933: INFO: metrics-server-864ffbc5c-sdxps started at 2019-11-05 01:34:34 +0000 UTC (0+1 container statuses recorded) I1105 02:50:26.935] Nov 5 02:50:26.933: INFO: Container metrics-server ready: true, restart count 0 I1105 02:50:26.935] Nov 5 02:50:26.933: INFO: kube-proxy-wxblc started at 2019-11-05 01:34:34 +0000 UTC (0+1 container statuses recorded) I1105 02:50:26.936] Nov 5 02:50:26.933: INFO: Container kube-proxy ready: true, restart count 0 I1105 02:50:26.936] Nov 5 02:50:26.933: INFO: kubernetes-dashboard-66dd8b8df7-ks284 started at 2019-11-05 01:34:35 +0000 UTC (0+1 container statuses recorded) I1105 02:50:26.936] Nov 5 02:50:26.933: INFO: Container kubernetes-dashboard ready: true, restart count 0 I1105 02:50:27.190] Nov 5 02:50:27.190: INFO: I1105 02:50:27.190] Latency metrics for node k8s-agentpool-15826033-vmss000000 I1105 02:50:27.191] Nov 5 02:50:27.190: INFO: I1105 02:50:27.191] Logging node info for node k8s-agentpool-15826033-vmss000001 I1105 02:50:27.271] Nov 5 02:50:27.267: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-agentpool-15826033-vmss000001,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-agentpool-15826033-vmss000001,UID:4a8ea605-cc0b-4e3b-9060-d8867d269e6c,ResourceVersion:14574,Generation:0,CreationTimestamp:2019-11-05 01:34:09 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{agentpool: agentpool,beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_D4s_v3,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 1,kubernetes.azure.com/cluster: kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5,kubernetes.azure.com/role: agent,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-agentpool-15826033-vmss000001,kubernetes.io/os: linux,kubernetes.io/role: agent,node-role.kubernetes.io/agent: ,storageprofile: managed,storagetier: Premium_LRS,topology.disk.csi.azure.com/zone: 1,},Annotations:map[string]string{csi.volume.kubernetes.io/nodeid: {"disk.csi.azure.com":"k8s-agentpool-15826033-vmss000001"},node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.0.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool-15826033-vmss/virtualMachines/1,Unschedulable:false,Taints:[],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-csi-disk.csi.azure.com: {{8 0} {} 8 DecimalSI},cpu: {{4 0} {} 4 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{16797044736 0} {} 16403364Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-csi-disk.csi.azure.com: {{8 0} {} 8 DecimalSI},cpu: {{4 0} {} 4 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{16010612736 0} {} 15635364Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 01:34:57 +0000 UTC 2019-11-05 01:34:57 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 02:49:51 +0000 UTC 2019-11-05 01:34:08 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 02:49:51 +0000 UTC 2019-11-05 01:34:08 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 02:49:51 +0000 UTC 2019-11-05 01:34:08 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 02:49:51 +0000 UTC 2019-11-05 01:34:29 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.0.5} {Hostname k8s-agentpool-15826033-vmss000001}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:b6b9fbf4a2fd4fa5a7ae472415fe30b8,SystemUUID:0162B3A4-3955-9A46-ABD4-A37BD75848CC,BootID:ee93003f-d47c-4118-9975-57365b3443a5,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[k8sprowinternal.azurecr.io/azuredisk-csi@sha256:75d4ff5d7cd845d6e810c67796ecb037e852fb849ec1eb5b3194564008e6d1e0 k8sprowinternal.azurecr.io/azuredisk-csi:e2e-3e3ed045c6faa509dbd8d2890ee9f56aa2934887] 286930414} {[gcr.io/kubernetes-helm/tiller@sha256:f6d8f4ab9ba993b5f5b60a6edafe86352eabe474ffeb84cb6c79b8866dce45d1 gcr.io/kubernetes-helm/tiller:v2.11.0] 71821984} {[quay.io/k8scsi/csi-provisioner@sha256:d657c839dce87324fe2b677302913f9386f885f8746be7bea0ced5b0844e3433 quay.io/k8scsi/csi-provisioner:v1.3.1] 50596998} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[quay.io/k8scsi/csi-attacher@sha256:26fccd7a99d973845df1193b46ebdcc6ab8dc5f6e6be319750c471fce1742d13 quay.io/k8scsi/csi-attacher:v1.2.0] 46226754} {[quay.io/k8scsi/csi-cluster-driver-registrar@sha256:fafd75ae5442f192cfa8c2e792903aee30d5884b62e802e4464b0a895d21e3ef quay.io/k8scsi/csi-cluster-driver-registrar:v1.0.1] 45874691} {[quay.io/k8scsi/csi-snapshotter@sha256:a49e0da1af6f2bf717e41ba1eee8b5e6a1cbd66a709dd92cc43fe475fe2589eb quay.io/k8scsi/csi-snapshotter:v1.1.0] 44325352} {[k8s.gcr.io/coredns@sha256:e83beb5e43f8513fa735e77ffc5859640baea30a882a11cc75c4c3244a737d3c k8s.gcr.io/coredns:1.5.0] 42488424} {[mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume@sha256:4fd30d43947d4a54fc89ead7985beecfd3c9b2a93a0655a373b1608ab90bd5af mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume:v0.0.7] 22909487} {[quay.io/k8scsi/livenessprobe@sha256:8c9c8a0f330acc2e558ae6624309447d02522e76aa35af23fa19b88abf4381cb quay.io/k8scsi/livenessprobe:v1.0.2] 17415381} {[quay.io/k8scsi/csi-node-driver-registrar@sha256:13daf82fb99e951a4bff8ae5fc7c17c3a8fe7130be6400990d8f6076c32d4599 quay.io/k8scsi/csi-node-driver-registrar:v1.1.0] 15815995} {[quay.io/k8scsi/livenessprobe@sha256:dde617756e0f602adc566ab71fd885f1dad451ad3fb063ac991c95a2ff47aea5 quay.io/k8scsi/livenessprobe:v1.1.0] 14967303} {[mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume@sha256:23d8c6033f02a1ecad05127ebdc931bb871264228661bc122704b0974e4d9fdd mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume:1.0.8] 1159025} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 02:50:27.271] Nov 5 02:50:27.267: INFO: I1105 02:50:27.271] Logging kubelet events for node k8s-agentpool-15826033-vmss000001 I1105 02:50:27.345] Nov 5 02:50:27.345: INFO: I1105 02:50:27.345] Logging pods the kubelet thinks is on node k8s-agentpool-15826033-vmss000001 I1105 02:50:27.427] Nov 5 02:50:27.427: INFO: coredns-7f68dcdbdb-jt522 started at 2019-11-05 01:34:35 +0000 UTC (0+1 container statuses recorded) I1105 02:50:27.427] Nov 5 02:50:27.427: INFO: Container coredns ready: true, restart count 0 I1105 02:50:27.428] Nov 5 02:50:27.427: INFO: azure-ip-masq-agent-6c6vn started at 2019-11-05 01:34:35 +0000 UTC (0+1 container statuses recorded) I1105 02:50:27.428] Nov 5 02:50:27.427: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 02:50:27.428] Nov 5 02:50:27.427: INFO: keyvault-flexvolume-rdlcj started at 2019-11-05 01:34:35 +0000 UTC (0+1 container statuses recorded) I1105 02:50:27.428] Nov 5 02:50:27.427: INFO: Container keyvault-flexvolume ready: true, restart count 0 I1105 02:50:27.428] Nov 5 02:50:27.427: INFO: kube-proxy-6kh57 started at 2019-11-05 01:34:35 +0000 UTC (0+1 container statuses recorded) I1105 02:50:27.428] Nov 5 02:50:27.427: INFO: Container kube-proxy ready: true, restart count 0 I1105 02:50:27.429] Nov 5 02:50:27.427: INFO: tiller-deploy-54c96cb5df-8d9fs started at 2019-11-05 01:34:35 +0000 UTC (0+1 container statuses recorded) I1105 02:50:27.429] Nov 5 02:50:27.427: INFO: Container tiller ready: true, restart count 0 I1105 02:50:27.429] Nov 5 02:50:27.427: INFO: csi-azuredisk-node-ghv5w started at 2019-11-05 01:38:24 +0000 UTC (0+3 container statuses recorded) I1105 02:50:27.429] Nov 5 02:50:27.427: INFO: Container azuredisk ready: true, restart count 0 I1105 02:50:27.429] Nov 5 02:50:27.427: INFO: Container liveness-probe ready: true, restart count 0 I1105 02:50:27.430] Nov 5 02:50:27.427: INFO: Container node-driver-registrar ready: true, restart count 0 I1105 02:50:27.430] Nov 5 02:50:27.427: INFO: csi-azuredisk-controller-6f6587bfc6-8vkfl started at 2019-11-05 01:38:24 +0000 UTC (0+6 container statuses recorded) I1105 02:50:27.430] Nov 5 02:50:27.427: INFO: Container azuredisk ready: true, restart count 0 I1105 02:50:27.430] Nov 5 02:50:27.427: INFO: Container cluster-driver-registrar ready: true, restart count 0 I1105 02:50:27.430] Nov 5 02:50:27.427: INFO: Container csi-attacher ready: true, restart count 0 I1105 02:50:27.430] Nov 5 02:50:27.427: INFO: Container csi-provisioner ready: true, restart count 0 I1105 02:50:27.430] Nov 5 02:50:27.427: INFO: Container csi-snapshotter ready: true, restart count 0 I1105 02:50:27.431] Nov 5 02:50:27.427: INFO: Container liveness-probe ready: true, restart count 0 I1105 02:50:27.431] Nov 5 02:50:27.427: INFO: blobfuse-flexvol-installer-rw5dw started at 2019-11-05 01:34:35 +0000 UTC (0+1 container statuses recorded) I1105 02:50:27.431] Nov 5 02:50:27.427: INFO: Container blobfuse-flexvol-installer ready: true, restart count 0 I1105 02:50:27.696] Nov 5 02:50:27.696: INFO: I1105 02:50:27.697] Latency metrics for node k8s-agentpool-15826033-vmss000001 I1105 02:50:27.697] Nov 5 02:50:27.696: INFO: I1105 02:50:27.697] Logging node info for node k8s-master-15826033-0 I1105 02:50:27.776] Nov 5 02:50:27.774: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-15826033-0,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-15826033-0,UID:aceb0231-4645-4a9a-b296-90ada851986b,ResourceVersion:14527,Generation:0,CreationTimestamp:2019-11-05 01:35:06 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 1,kubernetes.azure.com/cluster: kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-15826033-0,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.4.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5/providers/Microsoft.Compute/virtualMachines/k8s-master-15826033-0,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 01:35:20 +0000 UTC 2019-11-05 01:35:20 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 02:49:36 +0000 UTC 2019-11-05 01:34:57 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 02:49:36 +0000 UTC 2019-11-05 01:34:57 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 02:49:36 +0000 UTC 2019-11-05 01:34:57 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 02:49:36 +0000 UTC 2019-11-05 01:35:07 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.255.5} {Hostname k8s-master-15826033-0}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:1915756c22a540dc8eb30493987e9a43,SystemUUID:742C969C-5A71-D744-8E61-F27E38B56EB2,BootID:03f0672e-8fa9-4309-956f-8e3c371902c3,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 02:50:27.776] Nov 5 02:50:27.774: INFO: I1105 02:50:27.777] Logging kubelet events for node k8s-master-15826033-0 I1105 02:50:27.851] Nov 5 02:50:27.851: INFO: I1105 02:50:27.852] Logging pods the kubelet thinks is on node k8s-master-15826033-0 I1105 02:50:27.934] Nov 5 02:50:27.934: INFO: cloud-controller-manager-k8s-master-15826033-0 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:50:27.934] Nov 5 02:50:27.934: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 02:50:27.934] Nov 5 02:50:27.934: INFO: kube-addon-manager-k8s-master-15826033-0 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:50:27.935] Nov 5 02:50:27.934: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 02:50:27.935] Nov 5 02:50:27.934: INFO: kube-apiserver-k8s-master-15826033-0 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:50:27.935] Nov 5 02:50:27.934: INFO: Container kube-apiserver ready: true, restart count 0 I1105 02:50:27.935] Nov 5 02:50:27.934: INFO: kube-controller-manager-k8s-master-15826033-0 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:50:27.935] Nov 5 02:50:27.934: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 02:50:27.936] Nov 5 02:50:27.934: INFO: kube-scheduler-k8s-master-15826033-0 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:50:27.936] Nov 5 02:50:27.934: INFO: Container kube-scheduler ready: true, restart count 0 I1105 02:50:27.936] Nov 5 02:50:27.934: INFO: azure-ip-masq-agent-wgdpx started at 2019-11-05 01:35:06 +0000 UTC (0+1 container statuses recorded) I1105 02:50:27.936] Nov 5 02:50:27.934: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 02:50:27.936] Nov 5 02:50:27.934: INFO: kube-proxy-7d2px started at 2019-11-05 01:35:06 +0000 UTC (0+1 container statuses recorded) I1105 02:50:27.936] Nov 5 02:50:27.934: INFO: Container kube-proxy ready: true, restart count 0 I1105 02:50:28.186] Nov 5 02:50:28.185: INFO: I1105 02:50:28.186] Latency metrics for node k8s-master-15826033-0 I1105 02:50:28.186] Nov 5 02:50:28.185: INFO: I1105 02:50:28.186] Logging node info for node k8s-master-15826033-1 I1105 02:50:28.265] Nov 5 02:50:28.263: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-15826033-1,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-15826033-1,UID:1a13ec61-c89a-456c-8c94-bfc8e717c2e7,ResourceVersion:14528,Generation:0,CreationTimestamp:2019-11-05 01:35:07 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 1,kubernetes.azure.com/cluster: kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-15826033-1,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.5.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5/providers/Microsoft.Compute/virtualMachines/k8s-master-15826033-1,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 01:35:31 +0000 UTC 2019-11-05 01:35:31 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 02:49:37 +0000 UTC 2019-11-05 01:35:00 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 02:49:37 +0000 UTC 2019-11-05 01:35:00 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 02:49:37 +0000 UTC 2019-11-05 01:35:00 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 02:49:37 +0000 UTC 2019-11-05 01:35:07 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.255.6} {Hostname k8s-master-15826033-1}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:e0f2bccddd5b4555b180d14081ab59a8,SystemUUID:AE126631-CAD2-B54F-B22E-1F499342A4D6,BootID:509b9a9d-b046-4861-aa35-8bcb77465de2,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 02:50:28.265] Nov 5 02:50:28.263: INFO: I1105 02:50:28.266] Logging kubelet events for node k8s-master-15826033-1 I1105 02:50:28.341] Nov 5 02:50:28.340: INFO: I1105 02:50:28.341] Logging pods the kubelet thinks is on node k8s-master-15826033-1 I1105 02:50:28.421] Nov 5 02:50:28.421: INFO: cloud-controller-manager-k8s-master-15826033-1 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:50:28.421] Nov 5 02:50:28.421: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 02:50:28.422] Nov 5 02:50:28.421: INFO: kube-addon-manager-k8s-master-15826033-1 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:50:28.422] Nov 5 02:50:28.421: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 02:50:28.422] Nov 5 02:50:28.421: INFO: kube-apiserver-k8s-master-15826033-1 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:50:28.422] Nov 5 02:50:28.421: INFO: Container kube-apiserver ready: true, restart count 0 I1105 02:50:28.423] Nov 5 02:50:28.421: INFO: kube-controller-manager-k8s-master-15826033-1 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:50:28.423] Nov 5 02:50:28.421: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 02:50:28.423] Nov 5 02:50:28.421: INFO: kube-scheduler-k8s-master-15826033-1 started at 2019-11-05 01:36:16 +0000 UTC (0+1 container statuses recorded) I1105 02:50:28.423] Nov 5 02:50:28.421: INFO: Container kube-scheduler ready: true, restart count 0 I1105 02:50:28.424] Nov 5 02:50:28.421: INFO: azure-ip-masq-agent-w6dpr started at 2019-11-05 01:35:10 +0000 UTC (0+1 container statuses recorded) I1105 02:50:28.424] Nov 5 02:50:28.421: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 02:50:28.424] Nov 5 02:50:28.421: INFO: kube-proxy-8kmr5 started at 2019-11-05 01:35:10 +0000 UTC (0+1 container statuses recorded) I1105 02:50:28.424] Nov 5 02:50:28.421: INFO: Container kube-proxy ready: true, restart count 0 I1105 02:50:28.677] Nov 5 02:50:28.677: INFO: I1105 02:50:28.677] Latency metrics for node k8s-master-15826033-1 I1105 02:50:28.677] Nov 5 02:50:28.677: INFO: I1105 02:50:28.677] Logging node info for node k8s-master-15826033-2 I1105 02:50:28.758] Nov 5 02:50:28.756: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-15826033-2,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-15826033-2,UID:acb3a851-d6df-4549-ba56-4b7e96553c17,ResourceVersion:14620,Generation:0,CreationTimestamp:2019-11-05 01:34:09 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 2,kubernetes.azure.com/cluster: kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-15826033-2,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.1.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5/providers/Microsoft.Compute/virtualMachines/k8s-master-15826033-2,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 01:34:44 +0000 UTC 2019-11-05 01:34:44 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 02:50:08 +0000 UTC 2019-11-05 01:34:07 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 02:50:08 +0000 UTC 2019-11-05 01:34:07 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 02:50:08 +0000 UTC 2019-11-05 01:34:07 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 02:50:08 +0000 UTC 2019-11-05 01:34:29 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{Hostname k8s-master-15826033-2} {InternalIP 10.240.255.7}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:0885b0c2f6174106bde4fa0eae2cbc6b,SystemUUID:C176121C-5D7A-6243-9B05-8B68F8022B64,BootID:30430638-7ab6-4761-b984-2387e1a3f5b2,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 02:50:28.758] Nov 5 02:50:28.756: INFO: I1105 02:50:28.759] Logging kubelet events for node k8s-master-15826033-2 I1105 02:50:28.833] Nov 5 02:50:28.833: INFO: I1105 02:50:28.834] Logging pods the kubelet thinks is on node k8s-master-15826033-2 I1105 02:50:28.915] Nov 5 02:50:28.915: INFO: azure-ip-masq-agent-hw7s5 started at 2019-11-05 01:34:34 +0000 UTC (0+1 container statuses recorded) I1105 02:50:28.915] Nov 5 02:50:28.915: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 02:50:28.915] Nov 5 02:50:28.915: INFO: kube-proxy-5xpp6 started at 2019-11-05 01:34:34 +0000 UTC (0+1 container statuses recorded) I1105 02:50:28.916] Nov 5 02:50:28.915: INFO: Container kube-proxy ready: true, restart count 0 I1105 02:50:28.916] Nov 5 02:50:28.915: INFO: cloud-controller-manager-k8s-master-15826033-2 started at 2019-11-05 01:33:54 +0000 UTC (0+1 container statuses recorded) I1105 02:50:28.916] Nov 5 02:50:28.915: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 02:50:28.916] Nov 5 02:50:28.915: INFO: kube-addon-manager-k8s-master-15826033-2 started at 2019-11-05 01:33:54 +0000 UTC (0+1 container statuses recorded) I1105 02:50:28.916] Nov 5 02:50:28.915: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 02:50:28.916] Nov 5 02:50:28.915: INFO: kube-apiserver-k8s-master-15826033-2 started at 2019-11-05 01:33:54 +0000 UTC (0+1 container statuses recorded) I1105 02:50:28.916] Nov 5 02:50:28.915: INFO: Container kube-apiserver ready: true, restart count 0 I1105 02:50:28.917] Nov 5 02:50:28.915: INFO: kube-controller-manager-k8s-master-15826033-2 started at 2019-11-05 01:33:54 +0000 UTC (0+1 container statuses recorded) I1105 02:50:28.917] Nov 5 02:50:28.915: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 02:50:28.917] Nov 5 02:50:28.915: INFO: kube-scheduler-k8s-master-15826033-2 started at 2019-11-05 01:33:54 +0000 UTC (0+1 container statuses recorded) I1105 02:50:28.917] Nov 5 02:50:28.915: INFO: Container kube-scheduler ready: true, restart count 0 I1105 02:50:29.166] Nov 5 02:50:29.166: INFO: I1105 02:50:29.167] Latency metrics for node k8s-master-15826033-2 I1105 02:50:29.167] Nov 5 02:50:29.166: INFO: I1105 02:50:29.167] Logging node info for node k8s-master-15826033-3 I1105 02:50:29.246] Nov 5 02:50:29.244: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-15826033-3,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-15826033-3,UID:b7702272-1876-434e-b6a0-17fa34c6757f,ResourceVersion:14525,Generation:0,CreationTimestamp:2019-11-05 01:35:06 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 0,kubernetes.azure.com/cluster: kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-15826033-3,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.3.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5/providers/Microsoft.Compute/virtualMachines/k8s-master-15826033-3,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 01:35:17 +0000 UTC 2019-11-05 01:35:17 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 02:49:35 +0000 UTC 2019-11-05 01:35:00 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 02:49:35 +0000 UTC 2019-11-05 01:35:00 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 02:49:35 +0000 UTC 2019-11-05 01:35:00 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 02:49:35 +0000 UTC 2019-11-05 01:35:06 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.255.8} {Hostname k8s-master-15826033-3}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:fbf366963961422c9d90bedf6639c8e1,SystemUUID:5388948D-1A8D-EA41-B8B8-7F4523EDF5AA,BootID:42f5884c-7418-486a-8cf3-75188a331f38,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 02:50:29.247] Nov 5 02:50:29.244: INFO: I1105 02:50:29.247] Logging kubelet events for node k8s-master-15826033-3 I1105 02:50:29.321] Nov 5 02:50:29.321: INFO: I1105 02:50:29.322] Logging pods the kubelet thinks is on node k8s-master-15826033-3 I1105 02:50:29.403] Nov 5 02:50:29.403: INFO: azure-ip-masq-agent-dqfpw started at 2019-11-05 01:35:06 +0000 UTC (0+1 container statuses recorded) I1105 02:50:29.404] Nov 5 02:50:29.403: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 02:50:29.404] Nov 5 02:50:29.403: INFO: cloud-controller-manager-k8s-master-15826033-3 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:50:29.404] Nov 5 02:50:29.403: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 02:50:29.404] Nov 5 02:50:29.403: INFO: kube-addon-manager-k8s-master-15826033-3 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:50:29.404] Nov 5 02:50:29.403: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 02:50:29.405] Nov 5 02:50:29.403: INFO: kube-apiserver-k8s-master-15826033-3 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:50:29.405] Nov 5 02:50:29.403: INFO: Container kube-apiserver ready: true, restart count 0 I1105 02:50:29.405] Nov 5 02:50:29.403: INFO: kube-controller-manager-k8s-master-15826033-3 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:50:29.405] Nov 5 02:50:29.403: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 02:50:29.406] Nov 5 02:50:29.403: INFO: kube-scheduler-k8s-master-15826033-3 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:50:29.406] Nov 5 02:50:29.403: INFO: Container kube-scheduler ready: true, restart count 0 I1105 02:50:29.406] Nov 5 02:50:29.403: INFO: kube-proxy-xz24z started at 2019-11-05 01:35:06 +0000 UTC (0+1 container statuses recorded) I1105 02:50:29.406] Nov 5 02:50:29.403: INFO: Container kube-proxy ready: true, restart count 0 I1105 02:50:29.658] Nov 5 02:50:29.658: INFO: I1105 02:50:29.659] Latency metrics for node k8s-master-15826033-3 I1105 02:50:29.659] Nov 5 02:50:29.658: INFO: I1105 02:50:29.659] Logging node info for node k8s-master-15826033-4 I1105 02:50:29.738] Nov 5 02:50:29.736: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-15826033-4,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-15826033-4,UID:276daef1-6c05-41e3-bacd-abc93d35cc6d,ResourceVersion:14532,Generation:0,CreationTimestamp:2019-11-05 01:35:07 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 0,kubernetes.azure.com/cluster: kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-15826033-4,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.6.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5/providers/Microsoft.Compute/virtualMachines/k8s-master-15826033-4,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 01:35:41 +0000 UTC 2019-11-05 01:35:41 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 02:49:37 +0000 UTC 2019-11-05 01:35:02 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 02:49:37 +0000 UTC 2019-11-05 01:35:02 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 02:49:37 +0000 UTC 2019-11-05 01:35:02 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 02:49:37 +0000 UTC 2019-11-05 01:35:08 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.255.9} {Hostname k8s-master-15826033-4}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:79051fc0bdeb4e32a56acbaf5972ce3a,SystemUUID:C71606B5-7F08-9341-A26D-8A8D9D84FCAE,BootID:4bdf7cf3-2b83-42c7-a7ba-926ebe9ca75b,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 02:50:29.738] Nov 5 02:50:29.736: INFO: I1105 02:50:29.739] Logging kubelet events for node k8s-master-15826033-4 I1105 02:50:29.832] Nov 5 02:50:29.832: INFO: I1105 02:50:29.832] Logging pods the kubelet thinks is on node k8s-master-15826033-4 I1105 02:50:29.915] Nov 5 02:50:29.915: INFO: cloud-controller-manager-k8s-master-15826033-4 started at 2019-11-05 01:34:56 +0000 UTC (0+1 container statuses recorded) I1105 02:50:29.915] Nov 5 02:50:29.915: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 02:50:29.915] Nov 5 02:50:29.915: INFO: kube-addon-manager-k8s-master-15826033-4 started at 2019-11-05 01:34:56 +0000 UTC (0+1 container statuses recorded) I1105 02:50:29.916] Nov 5 02:50:29.915: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 02:50:29.916] Nov 5 02:50:29.915: INFO: azure-ip-masq-agent-9nrj8 started at 2019-11-05 01:35:11 +0000 UTC (0+1 container statuses recorded) I1105 02:50:29.916] Nov 5 02:50:29.915: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 02:50:29.916] Nov 5 02:50:29.915: INFO: kube-proxy-rmngk started at 2019-11-05 01:35:11 +0000 UTC (0+1 container statuses recorded) I1105 02:50:29.916] Nov 5 02:50:29.915: INFO: Container kube-proxy ready: true, restart count 0 I1105 02:50:29.916] Nov 5 02:50:29.915: INFO: kube-apiserver-k8s-master-15826033-4 started at 2019-11-05 01:34:55 +0000 UTC (0+1 container statuses recorded) I1105 02:50:29.916] Nov 5 02:50:29.915: INFO: Container kube-apiserver ready: true, restart count 0 I1105 02:50:29.917] Nov 5 02:50:29.915: INFO: kube-controller-manager-k8s-master-15826033-4 started at 2019-11-05 01:34:55 +0000 UTC (0+1 container statuses recorded) I1105 02:50:29.917] Nov 5 02:50:29.915: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 02:50:29.917] Nov 5 02:50:29.915: INFO: kube-scheduler-k8s-master-15826033-4 started at 2019-11-05 01:34:55 +0000 UTC (0+1 container statuses recorded) I1105 02:50:29.917] Nov 5 02:50:29.915: INFO: Container kube-scheduler ready: true, restart count 0 I1105 02:50:30.167] Nov 5 02:50:30.167: INFO: I1105 02:50:30.167] Latency metrics for node k8s-master-15826033-4 I1105 02:50:30.168] Nov 5 02:50:30.167: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready I1105 02:50:30.246] STEP: Destroying namespace "azuredisk-8754" for this suite. I1105 02:50:44.484] Nov 5 02:50:44.484: INFO: Waiting up to 30s for server preferred namespaced resources to be successfully discovered I1105 02:50:47.468] Nov 5 02:50:47.467: INFO: namespace azuredisk-8754 deletion completed in 17.22158638s I1105 02:50:47.468] I1105 02:50:47.468] • Failure [323.268 seconds] I1105 02:50:47.468] [azuredisk-csi-e2e] Dynamic Provisioning I1105 02:50:47.468] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:33 I1105 02:50:47.468] [single-az] I1105 02:50:47.469] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:36 I1105 02:50:47.469] should delete PV with reclaimPolicy "Delete" [It] I1105 02:50:47.469] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:221 I1105 02:50:47.469] I1105 02:50:47.469] Unexpected error: I1105 02:50:47.469] <*errors.errorString | 0xc0001eaac0>: { I1105 02:50:47.469] s: "PersistentVolumeClaims [pvc-6ch89] not all in phase Bound within 5m0s", I1105 02:50:47.470] } I1105 02:50:47.470] PersistentVolumeClaims [pvc-6ch89] not all in phase Bound within 5m0s I1105 02:50:47.470] occurred I1105 02:50:47.470] I1105 02:50:47.470] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/testsuites/testsuites.go:278 I1105 02:50:47.470] ------------------------------ I1105 02:50:47.470] [azuredisk-csi-e2e] Dynamic Provisioning [multi-az] I1105 02:50:47.470] should create a volume on demand I1105 02:50:47.471] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:78 I1105 02:50:47.471] [BeforeEach] [multi-az] I1105 02:50:47.471] /go/src/sigs.k8s.io/azuredisk-csi-driver/vendor/k8s.io/kubernetes/test/e2e/framework/framework.go:150 I1105 02:50:47.471] STEP: Creating a kubernetes client I1105 02:50:47.471] Nov 5 02:50:47.468: INFO: >>> kubeConfig: /workspace/aks978705698/kubeconfig/kubeconfig.eastus2.json I1105 02:50:47.471] STEP: Building a namespace api object, basename azuredisk I1105 02:50:47.706] STEP: Binding the e2e-test-privileged-psp PodSecurityPolicy to the default service account in azuredisk-8159 I1105 02:50:47.965] STEP: Waiting for a default service account to be provisioned in namespace I1105 02:50:48.040] [BeforeEach] [multi-az] I1105 02:50:48.040] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:58 I1105 02:50:48.040] [It] should create a volume on demand I1105 02:50:48.040] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:78 I1105 02:50:48.041] STEP: setting up the StorageClass I1105 02:50:48.041] STEP: creating a StorageClass I1105 02:50:48.131] STEP: setting up the PVC and PV I1105 02:50:48.131] STEP: creating a PVC I1105 02:50:48.214] STEP: waiting for PVC to be in phase "Bound" I1105 02:50:48.214] Nov 5 02:50:48.214: INFO: Waiting up to 5m0s for PersistentVolumeClaims [pvc-5mr2q] to have phase Bound I1105 02:50:48.291] Nov 5 02:50:48.291: INFO: PersistentVolumeClaim pvc-5mr2q found but phase is Pending instead of Bound. I1105 02:50:50.368] Nov 5 02:50:50.368: INFO: PersistentVolumeClaim pvc-5mr2q found but phase is Pending instead of Bound. I1105 02:50:52.446] Nov 5 02:50:52.446: INFO: PersistentVolumeClaim pvc-5mr2q found but phase is Pending instead of Bound. I1105 02:50:54.524] Nov 5 02:50:54.524: INFO: PersistentVolumeClaim pvc-5mr2q found but phase is Pending instead of Bound. I1105 02:50:56.602] Nov 5 02:50:56.602: INFO: PersistentVolumeClaim pvc-5mr2q found but phase is Pending instead of Bound. I1105 02:50:58.680] Nov 5 02:50:58.680: INFO: PersistentVolumeClaim pvc-5mr2q found but phase is Pending instead of Bound. I1105 02:51:00.758] Nov 5 02:51:00.758: INFO: PersistentVolumeClaim pvc-5mr2q found but phase is Pending instead of Bound. I1105 02:51:02.836] Nov 5 02:51:02.835: INFO: PersistentVolumeClaim pvc-5mr2q found but phase is Pending instead of Bound. I1105 02:51:04.913] Nov 5 02:51:04.913: INFO: PersistentVolumeClaim pvc-5mr2q found but phase is Pending instead of Bound. I1105 02:51:06.991] Nov 5 02:51:06.990: INFO: PersistentVolumeClaim pvc-5mr2q found but phase is Pending instead of Bound. I1105 02:51:09.069] Nov 5 02:51:09.069: INFO: PersistentVolumeClaim pvc-5mr2q found but phase is Pending instead of Bound. I1105 02:51:11.147] Nov 5 02:51:11.146: INFO: PersistentVolumeClaim pvc-5mr2q found but phase is Pending instead of Bound. I1105 02:51:13.224] Nov 5 02:51:13.224: INFO: PersistentVolumeClaim pvc-5mr2q found but phase is Pending instead of Bound. I1105 02:51:15.302] Nov 5 02:51:15.302: INFO: PersistentVolumeClaim pvc-5mr2q found but phase is Pending instead of Bound. I1105 02:51:17.380] Nov 5 02:51:17.380: INFO: PersistentVolumeClaim pvc-5mr2q found but phase is Pending instead of Bound. I1105 02:51:19.457] Nov 5 02:51:19.457: INFO: PersistentVolumeClaim pvc-5mr2q found but phase is Pending instead of Bound. I1105 02:51:21.534] Nov 5 02:51:21.534: INFO: PersistentVolumeClaim pvc-5mr2q found but phase is Pending instead of Bound. I1105 02:51:23.613] Nov 5 02:51:23.612: INFO: PersistentVolumeClaim pvc-5mr2q found but phase is Pending instead of Bound. I1105 02:51:25.690] Nov 5 02:51:25.690: INFO: PersistentVolumeClaim pvc-5mr2q found but phase is Pending instead of Bound. I1105 02:51:27.770] Nov 5 02:51:27.770: INFO: PersistentVolumeClaim pvc-5mr2q found but phase is Pending instead of Bound. I1105 02:51:29.848] Nov 5 02:51:29.848: INFO: PersistentVolumeClaim pvc-5mr2q found but phase is Pending instead of Bound. I1105 02:51:31.927] Nov 5 02:51:31.927: INFO: PersistentVolumeClaim pvc-5mr2q found but phase is Pending instead of Bound. I1105 02:51:34.008] Nov 5 02:51:34.008: INFO: PersistentVolumeClaim pvc-5mr2q found but phase is Pending instead of Bound. I1105 02:51:36.086] Nov 5 02:51:36.086: INFO: PersistentVolumeClaim pvc-5mr2q found but phase is Pending instead of Bound. I1105 02:51:38.164] Nov 5 02:51:38.164: INFO: PersistentVolumeClaim pvc-5mr2q found but phase is Pending instead of Bound. I1105 02:51:40.242] Nov 5 02:51:40.242: INFO: PersistentVolumeClaim pvc-5mr2q found but phase is Pending instead of Bound. I1105 02:51:42.320] Nov 5 02:51:42.319: INFO: PersistentVolumeClaim pvc-5mr2q found but phase is Pending instead of Bound. I1105 02:51:44.397] Nov 5 02:51:44.396: INFO: PersistentVolumeClaim pvc-5mr2q found but phase is Pending instead of Bound. I1105 02:51:46.474] Nov 5 02:51:46.474: INFO: PersistentVolumeClaim pvc-5mr2q found but phase is Pending instead of Bound. I1105 02:51:48.552] Nov 5 02:51:48.552: INFO: PersistentVolumeClaim pvc-5mr2q found but phase is Pending instead of Bound. I1105 02:51:50.630] Nov 5 02:51:50.630: INFO: PersistentVolumeClaim pvc-5mr2q found but phase is Pending instead of Bound. I1105 02:51:52.707] Nov 5 02:51:52.707: INFO: PersistentVolumeClaim pvc-5mr2q found but phase is Pending instead of Bound. I1105 02:51:54.792] Nov 5 02:51:54.790: INFO: PersistentVolumeClaim pvc-5mr2q found but phase is Pending instead of Bound. I1105 02:51:56.868] Nov 5 02:51:56.867: INFO: PersistentVolumeClaim pvc-5mr2q found but phase is Pending instead of Bound. I1105 02:51:58.945] Nov 5 02:51:58.945: INFO: PersistentVolumeClaim pvc-5mr2q found but phase is Pending instead of Bound. I1105 02:52:01.023] Nov 5 02:52:01.023: INFO: PersistentVolumeClaim pvc-5mr2q found but phase is Pending instead of Bound. I1105 02:52:03.101] Nov 5 02:52:03.101: INFO: PersistentVolumeClaim pvc-5mr2q found but phase is Pending instead of Bound. I1105 02:52:05.179] Nov 5 02:52:05.179: INFO: PersistentVolumeClaim pvc-5mr2q found but phase is Pending instead of Bound. I1105 02:52:07.257] Nov 5 02:52:07.257: INFO: PersistentVolumeClaim pvc-5mr2q found but phase is Pending instead of Bound. I1105 02:52:09.334] Nov 5 02:52:09.334: INFO: PersistentVolumeClaim pvc-5mr2q found but phase is Pending instead of Bound. I1105 02:52:11.412] Nov 5 02:52:11.411: INFO: PersistentVolumeClaim pvc-5mr2q found but phase is Pending instead of Bound. I1105 02:52:13.489] Nov 5 02:52:13.489: INFO: PersistentVolumeClaim pvc-5mr2q found but phase is Pending instead of Bound. I1105 02:52:15.567] Nov 5 02:52:15.567: INFO: PersistentVolumeClaim pvc-5mr2q found but phase is Pending instead of Bound. I1105 02:52:17.645] Nov 5 02:52:17.645: INFO: PersistentVolumeClaim pvc-5mr2q found but phase is Pending instead of Bound. I1105 02:52:19.724] Nov 5 02:52:19.724: INFO: PersistentVolumeClaim pvc-5mr2q found but phase is Pending instead of Bound. I1105 02:52:21.802] Nov 5 02:52:21.802: INFO: PersistentVolumeClaim pvc-5mr2q found but phase is Pending instead of Bound. I1105 02:52:23.883] Nov 5 02:52:23.883: INFO: PersistentVolumeClaim pvc-5mr2q found but phase is Pending instead of Bound. I1105 02:52:25.961] Nov 5 02:52:25.961: INFO: PersistentVolumeClaim pvc-5mr2q found but phase is Pending instead of Bound. I1105 02:52:28.396] Nov 5 02:52:28.039: INFO: PersistentVolumeClaim pvc-5mr2q found but phase is Pending instead of Bound. I1105 02:52:30.123] Nov 5 02:52:30.123: INFO: PersistentVolumeClaim pvc-5mr2q found but phase is Pending instead of Bound. I1105 02:52:32.201] Nov 5 02:52:32.200: INFO: PersistentVolumeClaim pvc-5mr2q found but phase is Pending instead of Bound. I1105 02:52:34.786] Nov 5 02:52:34.279: INFO: PersistentVolumeClaim pvc-5mr2q found but phase is Pending instead of Bound. I1105 02:52:36.356] Nov 5 02:52:36.356: INFO: PersistentVolumeClaim pvc-5mr2q found but phase is Pending instead of Bound. I1105 02:52:38.434] Nov 5 02:52:38.434: INFO: PersistentVolumeClaim pvc-5mr2q found but phase is Pending instead of Bound. I1105 02:52:41.007] Nov 5 02:52:40.511: INFO: PersistentVolumeClaim pvc-5mr2q found but phase is Pending instead of Bound. I1105 02:52:42.590] Nov 5 02:52:42.589: INFO: PersistentVolumeClaim pvc-5mr2q found but phase is Pending instead of Bound. I1105 02:52:44.667] Nov 5 02:52:44.667: INFO: PersistentVolumeClaim pvc-5mr2q found but phase is Pending instead of Bound. I1105 02:52:46.910] Nov 5 02:52:46.745: INFO: PersistentVolumeClaim pvc-5mr2q found but phase is Pending instead of Bound. I1105 02:52:48.824] Nov 5 02:52:48.823: INFO: PersistentVolumeClaim pvc-5mr2q found but phase is Pending instead of Bound. I1105 02:52:50.910] Nov 5 02:52:50.906: INFO: PersistentVolumeClaim pvc-5mr2q found but phase is Pending instead of Bound. I1105 02:52:52.990] Nov 5 02:52:52.986: INFO: PersistentVolumeClaim pvc-5mr2q found but phase is Pending instead of Bound. I1105 02:52:55.064] Nov 5 02:52:55.063: INFO: PersistentVolumeClaim pvc-5mr2q found but phase is Pending instead of Bound. I1105 02:52:57.142] Nov 5 02:52:57.141: INFO: PersistentVolumeClaim pvc-5mr2q found but phase is Pending instead of Bound. I1105 02:52:59.219] Nov 5 02:52:59.219: INFO: PersistentVolumeClaim pvc-5mr2q found but phase is Pending instead of Bound. I1105 02:53:01.297] Nov 5 02:53:01.296: INFO: PersistentVolumeClaim pvc-5mr2q found but phase is Pending instead of Bound. I1105 02:53:05.291] Nov 5 02:53:03.374: INFO: PersistentVolumeClaim pvc-5mr2q found but phase is Pending instead of Bound. I1105 02:53:05.453] Nov 5 02:53:05.453: INFO: PersistentVolumeClaim pvc-5mr2q found but phase is Pending instead of Bound. I1105 02:53:07.530] Nov 5 02:53:07.530: INFO: PersistentVolumeClaim pvc-5mr2q found but phase is Pending instead of Bound. I1105 02:53:09.609] Nov 5 02:53:09.608: INFO: PersistentVolumeClaim pvc-5mr2q found but phase is Pending instead of Bound. I1105 02:53:11.686] Nov 5 02:53:11.686: INFO: PersistentVolumeClaim pvc-5mr2q found but phase is Pending instead of Bound. I1105 02:53:13.764] Nov 5 02:53:13.764: INFO: PersistentVolumeClaim pvc-5mr2q found but phase is Pending instead of Bound. I1105 02:53:15.842] Nov 5 02:53:15.842: INFO: PersistentVolumeClaim pvc-5mr2q found but phase is Pending instead of Bound. I1105 02:53:17.920] Nov 5 02:53:17.920: INFO: PersistentVolumeClaim pvc-5mr2q found but phase is Pending instead of Bound. I1105 02:53:20.017] Nov 5 02:53:20.014: INFO: PersistentVolumeClaim pvc-5mr2q found but phase is Pending instead of Bound. I1105 02:53:22.092] Nov 5 02:53:22.092: INFO: PersistentVolumeClaim pvc-5mr2q found but phase is Pending instead of Bound. I1105 02:53:24.170] Nov 5 02:53:24.169: INFO: PersistentVolumeClaim pvc-5mr2q found but phase is Pending instead of Bound. I1105 02:53:26.247] Nov 5 02:53:26.247: INFO: PersistentVolumeClaim pvc-5mr2q found but phase is Pending instead of Bound. I1105 02:53:28.999] Nov 5 02:53:28.325: INFO: PersistentVolumeClaim pvc-5mr2q found but phase is Pending instead of Bound. I1105 02:53:30.403] Nov 5 02:53:30.403: INFO: PersistentVolumeClaim pvc-5mr2q found but phase is Pending instead of Bound. I1105 02:53:32.490] Nov 5 02:53:32.486: INFO: PersistentVolumeClaim pvc-5mr2q found but phase is Pending instead of Bound. I1105 02:53:34.563] Nov 5 02:53:34.563: INFO: PersistentVolumeClaim pvc-5mr2q found but phase is Pending instead of Bound. I1105 02:53:36.641] Nov 5 02:53:36.641: INFO: PersistentVolumeClaim pvc-5mr2q found but phase is Pending instead of Bound. I1105 02:53:38.722] Nov 5 02:53:38.721: INFO: PersistentVolumeClaim pvc-5mr2q found but phase is Pending instead of Bound. I1105 02:53:40.799] Nov 5 02:53:40.799: INFO: PersistentVolumeClaim pvc-5mr2q found but phase is Pending instead of Bound. I1105 02:53:42.879] Nov 5 02:53:42.879: INFO: PersistentVolumeClaim pvc-5mr2q found but phase is Pending instead of Bound. I1105 02:53:44.960] Nov 5 02:53:44.960: INFO: PersistentVolumeClaim pvc-5mr2q found but phase is Pending instead of Bound. I1105 02:53:47.042] Nov 5 02:53:47.038: INFO: PersistentVolumeClaim pvc-5mr2q found but phase is Pending instead of Bound. I1105 02:53:49.136] Nov 5 02:53:49.135: INFO: PersistentVolumeClaim pvc-5mr2q found but phase is Pending instead of Bound. I1105 02:53:51.218] Nov 5 02:53:51.218: INFO: PersistentVolumeClaim pvc-5mr2q found but phase is Pending instead of Bound. I1105 02:53:53.296] Nov 5 02:53:53.296: INFO: PersistentVolumeClaim pvc-5mr2q found but phase is Pending instead of Bound. I1105 02:53:55.374] Nov 5 02:53:55.374: INFO: PersistentVolumeClaim pvc-5mr2q found but phase is Pending instead of Bound. I1105 02:53:57.451] Nov 5 02:53:57.451: INFO: PersistentVolumeClaim pvc-5mr2q found but phase is Pending instead of Bound. I1105 02:53:59.530] Nov 5 02:53:59.529: INFO: PersistentVolumeClaim pvc-5mr2q found but phase is Pending instead of Bound. I1105 02:54:01.607] Nov 5 02:54:01.607: INFO: PersistentVolumeClaim pvc-5mr2q found but phase is Pending instead of Bound. I1105 02:54:03.685] Nov 5 02:54:03.685: INFO: PersistentVolumeClaim pvc-5mr2q found but phase is Pending instead of Bound. I1105 02:54:05.763] Nov 5 02:54:05.763: INFO: PersistentVolumeClaim pvc-5mr2q found but phase is Pending instead of Bound. I1105 02:54:07.842] Nov 5 02:54:07.841: INFO: PersistentVolumeClaim pvc-5mr2q found but phase is Pending instead of Bound. I1105 02:54:09.919] Nov 5 02:54:09.919: INFO: PersistentVolumeClaim pvc-5mr2q found but phase is Pending instead of Bound. I1105 02:54:11.997] Nov 5 02:54:11.996: INFO: PersistentVolumeClaim pvc-5mr2q found but phase is Pending instead of Bound. I1105 02:54:14.074] Nov 5 02:54:14.074: INFO: PersistentVolumeClaim pvc-5mr2q found but phase is Pending instead of Bound. I1105 02:54:16.152] Nov 5 02:54:16.152: INFO: PersistentVolumeClaim pvc-5mr2q found but phase is Pending instead of Bound. I1105 02:54:18.230] Nov 5 02:54:18.229: INFO: PersistentVolumeClaim pvc-5mr2q found but phase is Pending instead of Bound. I1105 02:54:20.307] Nov 5 02:54:20.307: INFO: PersistentVolumeClaim pvc-5mr2q found but phase is Pending instead of Bound. I1105 02:54:22.385] Nov 5 02:54:22.385: INFO: PersistentVolumeClaim pvc-5mr2q found but phase is Pending instead of Bound. I1105 02:54:24.471] Nov 5 02:54:24.470: INFO: PersistentVolumeClaim pvc-5mr2q found but phase is Pending instead of Bound. I1105 02:54:26.549] Nov 5 02:54:26.548: INFO: PersistentVolumeClaim pvc-5mr2q found but phase is Pending instead of Bound. I1105 02:54:28.630] Nov 5 02:54:28.630: INFO: PersistentVolumeClaim pvc-5mr2q found but phase is Pending instead of Bound. I1105 02:54:30.708] Nov 5 02:54:30.708: INFO: PersistentVolumeClaim pvc-5mr2q found but phase is Pending instead of Bound. I1105 02:54:32.786] Nov 5 02:54:32.786: INFO: PersistentVolumeClaim pvc-5mr2q found but phase is Pending instead of Bound. I1105 02:54:34.870] Nov 5 02:54:34.870: INFO: PersistentVolumeClaim pvc-5mr2q found but phase is Pending instead of Bound. I1105 02:54:36.952] Nov 5 02:54:36.952: INFO: PersistentVolumeClaim pvc-5mr2q found but phase is Pending instead of Bound. I1105 02:54:39.029] Nov 5 02:54:39.029: INFO: PersistentVolumeClaim pvc-5mr2q found but phase is Pending instead of Bound. I1105 02:54:41.130] Nov 5 02:54:41.130: INFO: PersistentVolumeClaim pvc-5mr2q found but phase is Pending instead of Bound. I1105 02:54:43.207] Nov 5 02:54:43.207: INFO: PersistentVolumeClaim pvc-5mr2q found but phase is Pending instead of Bound. I1105 02:54:45.285] Nov 5 02:54:45.285: INFO: PersistentVolumeClaim pvc-5mr2q found but phase is Pending instead of Bound. I1105 02:54:47.363] Nov 5 02:54:47.363: INFO: PersistentVolumeClaim pvc-5mr2q found but phase is Pending instead of Bound. I1105 02:54:49.441] Nov 5 02:54:49.441: INFO: PersistentVolumeClaim pvc-5mr2q found but phase is Pending instead of Bound. I1105 02:54:51.518] Nov 5 02:54:51.518: INFO: PersistentVolumeClaim pvc-5mr2q found but phase is Pending instead of Bound. I1105 02:54:53.596] Nov 5 02:54:53.596: INFO: PersistentVolumeClaim pvc-5mr2q found but phase is Pending instead of Bound. I1105 02:54:55.675] Nov 5 02:54:55.675: INFO: PersistentVolumeClaim pvc-5mr2q found but phase is Pending instead of Bound. I1105 02:54:57.753] Nov 5 02:54:57.753: INFO: PersistentVolumeClaim pvc-5mr2q found but phase is Pending instead of Bound. I1105 02:54:59.831] Nov 5 02:54:59.831: INFO: PersistentVolumeClaim pvc-5mr2q found but phase is Pending instead of Bound. I1105 02:55:01.909] Nov 5 02:55:01.909: INFO: PersistentVolumeClaim pvc-5mr2q found but phase is Pending instead of Bound. I1105 02:55:03.992] Nov 5 02:55:03.992: INFO: PersistentVolumeClaim pvc-5mr2q found but phase is Pending instead of Bound. I1105 02:55:06.074] Nov 5 02:55:06.070: INFO: PersistentVolumeClaim pvc-5mr2q found but phase is Pending instead of Bound. I1105 02:55:08.148] Nov 5 02:55:08.147: INFO: PersistentVolumeClaim pvc-5mr2q found but phase is Pending instead of Bound. I1105 02:55:10.225] Nov 5 02:55:10.225: INFO: PersistentVolumeClaim pvc-5mr2q found but phase is Pending instead of Bound. I1105 02:55:12.303] Nov 5 02:55:12.303: INFO: PersistentVolumeClaim pvc-5mr2q found but phase is Pending instead of Bound. I1105 02:55:14.381] Nov 5 02:55:14.381: INFO: PersistentVolumeClaim pvc-5mr2q found but phase is Pending instead of Bound. I1105 02:55:16.458] Nov 5 02:55:16.458: INFO: PersistentVolumeClaim pvc-5mr2q found but phase is Pending instead of Bound. I1105 02:55:18.536] Nov 5 02:55:18.536: INFO: PersistentVolumeClaim pvc-5mr2q found but phase is Pending instead of Bound. I1105 02:55:20.614] Nov 5 02:55:20.614: INFO: PersistentVolumeClaim pvc-5mr2q found but phase is Pending instead of Bound. I1105 02:55:22.692] Nov 5 02:55:22.692: INFO: PersistentVolumeClaim pvc-5mr2q found but phase is Pending instead of Bound. I1105 02:55:24.770] Nov 5 02:55:24.770: INFO: PersistentVolumeClaim pvc-5mr2q found but phase is Pending instead of Bound. I1105 02:55:26.848] Nov 5 02:55:26.847: INFO: PersistentVolumeClaim pvc-5mr2q found but phase is Pending instead of Bound. I1105 02:55:28.927] Nov 5 02:55:28.927: INFO: PersistentVolumeClaim pvc-5mr2q found but phase is Pending instead of Bound. I1105 02:55:31.005] Nov 5 02:55:31.005: INFO: PersistentVolumeClaim pvc-5mr2q found but phase is Pending instead of Bound. I1105 02:55:33.083] Nov 5 02:55:33.083: INFO: PersistentVolumeClaim pvc-5mr2q found but phase is Pending instead of Bound. I1105 02:55:35.161] Nov 5 02:55:35.161: INFO: PersistentVolumeClaim pvc-5mr2q found but phase is Pending instead of Bound. I1105 02:55:37.239] Nov 5 02:55:37.239: INFO: PersistentVolumeClaim pvc-5mr2q found but phase is Pending instead of Bound. I1105 02:55:39.317] Nov 5 02:55:39.316: INFO: PersistentVolumeClaim pvc-5mr2q found but phase is Pending instead of Bound. I1105 02:55:41.395] Nov 5 02:55:41.395: INFO: PersistentVolumeClaim pvc-5mr2q found but phase is Pending instead of Bound. I1105 02:55:43.490] Nov 5 02:55:43.472: INFO: PersistentVolumeClaim pvc-5mr2q found but phase is Pending instead of Bound. I1105 02:55:45.550] Nov 5 02:55:45.550: INFO: PersistentVolumeClaim pvc-5mr2q found but phase is Pending instead of Bound. I1105 02:55:47.637] Nov 5 02:55:47.630: INFO: PersistentVolumeClaim pvc-5mr2q found but phase is Pending instead of Bound. I1105 02:55:49.630] Nov 5 02:55:49.630: INFO: Unexpected error occurred: PersistentVolumeClaims [pvc-5mr2q] not all in phase Bound within 5m0s I1105 02:55:49.631] [AfterEach] [multi-az] I1105 02:55:49.631] /go/src/sigs.k8s.io/azuredisk-csi-driver/vendor/k8s.io/kubernetes/test/e2e/framework/framework.go:151 I1105 02:55:49.631] STEP: Collecting events from namespace "azuredisk-8159". I1105 02:55:49.708] STEP: Found 3 events. I1105 02:55:49.709] Nov 5 02:55:49.708: INFO: At 2019-11-05 02:50:48 +0000 UTC - event for pvc-5mr2q: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "disk.csi.azure.com" or manually created by system administrator I1105 02:55:49.709] Nov 5 02:55:49.708: INFO: At 2019-11-05 02:50:48 +0000 UTC - event for pvc-5mr2q: {disk.csi.azure.com_csi-azuredisk-controller-6f6587bfc6-8vkfl_1376914b-ff6d-11e9-93ab-ba2bee589b8c } Provisioning: External provisioner is provisioning volume for claim "azuredisk-8159/pvc-5mr2q" I1105 02:55:49.710] Nov 5 02:55:49.708: INFO: At 2019-11-05 02:50:48 +0000 UTC - event for pvc-5mr2q: {disk.csi.azure.com_csi-azuredisk-controller-6f6587bfc6-8vkfl_1376914b-ff6d-11e9-93ab-ba2bee589b8c } ProvisioningFailed: failed to provision volume with StorageClass "azuredisk-8159-disk.csi.azure.com-dynamic-sc-98mjd": rpc error: code = Unknown desc = compute.DisksClient#CreateOrUpdate: Failure sending request: StatusCode=400 -- Original Error: Code="InvalidAvailabilityZone" Message="The zone(s) '' for resource 'Microsoft.Compute/disks/pvc-50873950-4b4d-410a-9ab3-101eb7a2c561' is not supported. The supported zones for location 'eastus2' are '1,2,3'" I1105 02:55:49.786] Nov 5 02:55:49.786: INFO: POD NODE PHASE GRACE CONDITIONS I1105 02:55:49.786] Nov 5 02:55:49.786: INFO: I1105 02:55:50.022] Nov 5 02:55:50.022: INFO: I1105 02:55:50.022] Logging node info for node k8s-agentpool-15826033-vmss000000 I1105 02:55:50.102] Nov 5 02:55:50.099: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-agentpool-15826033-vmss000000,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-agentpool-15826033-vmss000000,UID:15fa874e-54ec-42cd-b243-d3baa512150f,ResourceVersion:15583,Generation:0,CreationTimestamp:2019-11-05 01:34:17 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{agentpool: agentpool,beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_D4s_v3,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 0,kubernetes.azure.com/cluster: kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5,kubernetes.azure.com/role: agent,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-agentpool-15826033-vmss000000,kubernetes.io/os: linux,kubernetes.io/role: agent,node-role.kubernetes.io/agent: ,storageprofile: managed,storagetier: Premium_LRS,topology.disk.csi.azure.com/zone: 0,},Annotations:map[string]string{csi.volume.kubernetes.io/nodeid: {"disk.csi.azure.com":"k8s-agentpool-15826033-vmss000000"},node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.2.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool-15826033-vmss/virtualMachines/0,Unschedulable:false,Taints:[],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-csi-disk.csi.azure.com: {{8 0} {} 8 DecimalSI},cpu: {{4 0} {} 4 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{16797044736 0} {} 16403364Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-csi-disk.csi.azure.com: {{8 0} {} 8 DecimalSI},cpu: {{4 0} {} 4 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{16010612736 0} {} 15635364Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 01:34:56 +0000 UTC 2019-11-05 01:34:56 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 02:55:30 +0000 UTC 2019-11-05 01:34:17 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 02:55:30 +0000 UTC 2019-11-05 01:34:17 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 02:55:30 +0000 UTC 2019-11-05 01:34:17 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 02:55:30 +0000 UTC 2019-11-05 01:34:29 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.0.4} {Hostname k8s-agentpool-15826033-vmss000000}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:afe00792177d49a98f6a44d03d08732d,SystemUUID:4F1777BF-A5C3-004E-81C4-90F0B54C02A0,BootID:13b6a628-ae89-44af-b891-89461030adcb,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[k8sprowinternal.azurecr.io/azuredisk-csi@sha256:75d4ff5d7cd845d6e810c67796ecb037e852fb849ec1eb5b3194564008e6d1e0 k8sprowinternal.azurecr.io/azuredisk-csi:e2e-3e3ed045c6faa509dbd8d2890ee9f56aa2934887] 286930414} {[k8s.gcr.io/kubernetes-dashboard-amd64@sha256:0ae6b69432e78069c5ce2bcde0fe409c5c4d6f0f4d9cd50a17974fea38898747 k8s.gcr.io/kubernetes-dashboard-amd64:v1.10.1] 121711221} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/metrics-server-amd64@sha256:49a9f12f7067d11f42c803dbe61ed2c1299959ad85cb315b25ff7eef8e6b8892 k8s.gcr.io/metrics-server-amd64:v0.2.1] 42541759} {[mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume@sha256:4fd30d43947d4a54fc89ead7985beecfd3c9b2a93a0655a373b1608ab90bd5af mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume:v0.0.7] 22909487} {[quay.io/k8scsi/livenessprobe@sha256:8c9c8a0f330acc2e558ae6624309447d02522e76aa35af23fa19b88abf4381cb quay.io/k8scsi/livenessprobe:v1.0.2] 17415381} {[quay.io/k8scsi/csi-node-driver-registrar@sha256:13daf82fb99e951a4bff8ae5fc7c17c3a8fe7130be6400990d8f6076c32d4599 quay.io/k8scsi/csi-node-driver-registrar:v1.1.0] 15815995} {[mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume@sha256:23d8c6033f02a1ecad05127ebdc931bb871264228661bc122704b0974e4d9fdd mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume:1.0.8] 1159025} {[busybox@sha256:8ccbac733d19c0dd4d70b4f0c1e12245b5fa3ad24758a11035ee505c629c0796 busybox:1.29] 1154361} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 02:55:50.102] Nov 5 02:55:50.099: INFO: I1105 02:55:50.102] Logging kubelet events for node k8s-agentpool-15826033-vmss000000 I1105 02:55:50.179] Nov 5 02:55:50.179: INFO: I1105 02:55:50.179] Logging pods the kubelet thinks is on node k8s-agentpool-15826033-vmss000000 I1105 02:55:50.262] Nov 5 02:55:50.262: INFO: csi-azuredisk-node-whq2g started at 2019-11-05 01:38:24 +0000 UTC (0+3 container statuses recorded) I1105 02:55:50.262] Nov 5 02:55:50.262: INFO: Container azuredisk ready: true, restart count 0 I1105 02:55:50.262] Nov 5 02:55:50.262: INFO: Container liveness-probe ready: true, restart count 0 I1105 02:55:50.263] Nov 5 02:55:50.262: INFO: Container node-driver-registrar ready: true, restart count 0 I1105 02:55:50.263] Nov 5 02:55:50.262: INFO: blobfuse-flexvol-installer-bsvqt started at 2019-11-05 01:34:34 +0000 UTC (0+1 container statuses recorded) I1105 02:55:50.263] Nov 5 02:55:50.262: INFO: Container blobfuse-flexvol-installer ready: true, restart count 0 I1105 02:55:50.263] Nov 5 02:55:50.262: INFO: azure-ip-masq-agent-vx86b started at 2019-11-05 01:34:34 +0000 UTC (0+1 container statuses recorded) I1105 02:55:50.263] Nov 5 02:55:50.262: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 02:55:50.264] Nov 5 02:55:50.262: INFO: keyvault-flexvolume-qhnfr started at 2019-11-05 01:34:34 +0000 UTC (0+1 container statuses recorded) I1105 02:55:50.264] Nov 5 02:55:50.262: INFO: Container keyvault-flexvolume ready: true, restart count 0 I1105 02:55:50.264] Nov 5 02:55:50.262: INFO: metrics-server-864ffbc5c-sdxps started at 2019-11-05 01:34:34 +0000 UTC (0+1 container statuses recorded) I1105 02:55:50.264] Nov 5 02:55:50.262: INFO: Container metrics-server ready: true, restart count 0 I1105 02:55:50.264] Nov 5 02:55:50.262: INFO: kube-proxy-wxblc started at 2019-11-05 01:34:34 +0000 UTC (0+1 container statuses recorded) I1105 02:55:50.265] Nov 5 02:55:50.262: INFO: Container kube-proxy ready: true, restart count 0 I1105 02:55:50.265] Nov 5 02:55:50.262: INFO: kubernetes-dashboard-66dd8b8df7-ks284 started at 2019-11-05 01:34:35 +0000 UTC (0+1 container statuses recorded) I1105 02:55:50.265] Nov 5 02:55:50.262: INFO: Container kubernetes-dashboard ready: true, restart count 0 I1105 02:55:50.511] Nov 5 02:55:50.511: INFO: I1105 02:55:50.512] Latency metrics for node k8s-agentpool-15826033-vmss000000 I1105 02:55:50.512] Nov 5 02:55:50.511: INFO: I1105 02:55:50.512] Logging node info for node k8s-agentpool-15826033-vmss000001 I1105 02:55:50.594] Nov 5 02:55:50.588: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-agentpool-15826033-vmss000001,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-agentpool-15826033-vmss000001,UID:4a8ea605-cc0b-4e3b-9060-d8867d269e6c,ResourceVersion:15474,Generation:0,CreationTimestamp:2019-11-05 01:34:09 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{agentpool: agentpool,beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_D4s_v3,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 1,kubernetes.azure.com/cluster: kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5,kubernetes.azure.com/role: agent,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-agentpool-15826033-vmss000001,kubernetes.io/os: linux,kubernetes.io/role: agent,node-role.kubernetes.io/agent: ,storageprofile: managed,storagetier: Premium_LRS,topology.disk.csi.azure.com/zone: 1,},Annotations:map[string]string{csi.volume.kubernetes.io/nodeid: {"disk.csi.azure.com":"k8s-agentpool-15826033-vmss000001"},node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.0.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool-15826033-vmss/virtualMachines/1,Unschedulable:false,Taints:[],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-csi-disk.csi.azure.com: {{8 0} {} 8 DecimalSI},cpu: {{4 0} {} 4 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{16797044736 0} {} 16403364Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-csi-disk.csi.azure.com: {{8 0} {} 8 DecimalSI},cpu: {{4 0} {} 4 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{16010612736 0} {} 15635364Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 01:34:57 +0000 UTC 2019-11-05 01:34:57 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 02:54:52 +0000 UTC 2019-11-05 01:34:08 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 02:54:52 +0000 UTC 2019-11-05 01:34:08 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 02:54:52 +0000 UTC 2019-11-05 01:34:08 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 02:54:52 +0000 UTC 2019-11-05 01:34:29 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.0.5} {Hostname k8s-agentpool-15826033-vmss000001}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:b6b9fbf4a2fd4fa5a7ae472415fe30b8,SystemUUID:0162B3A4-3955-9A46-ABD4-A37BD75848CC,BootID:ee93003f-d47c-4118-9975-57365b3443a5,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[k8sprowinternal.azurecr.io/azuredisk-csi@sha256:75d4ff5d7cd845d6e810c67796ecb037e852fb849ec1eb5b3194564008e6d1e0 k8sprowinternal.azurecr.io/azuredisk-csi:e2e-3e3ed045c6faa509dbd8d2890ee9f56aa2934887] 286930414} {[gcr.io/kubernetes-helm/tiller@sha256:f6d8f4ab9ba993b5f5b60a6edafe86352eabe474ffeb84cb6c79b8866dce45d1 gcr.io/kubernetes-helm/tiller:v2.11.0] 71821984} {[quay.io/k8scsi/csi-provisioner@sha256:d657c839dce87324fe2b677302913f9386f885f8746be7bea0ced5b0844e3433 quay.io/k8scsi/csi-provisioner:v1.3.1] 50596998} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[quay.io/k8scsi/csi-attacher@sha256:26fccd7a99d973845df1193b46ebdcc6ab8dc5f6e6be319750c471fce1742d13 quay.io/k8scsi/csi-attacher:v1.2.0] 46226754} {[quay.io/k8scsi/csi-cluster-driver-registrar@sha256:fafd75ae5442f192cfa8c2e792903aee30d5884b62e802e4464b0a895d21e3ef quay.io/k8scsi/csi-cluster-driver-registrar:v1.0.1] 45874691} {[quay.io/k8scsi/csi-snapshotter@sha256:a49e0da1af6f2bf717e41ba1eee8b5e6a1cbd66a709dd92cc43fe475fe2589eb quay.io/k8scsi/csi-snapshotter:v1.1.0] 44325352} {[k8s.gcr.io/coredns@sha256:e83beb5e43f8513fa735e77ffc5859640baea30a882a11cc75c4c3244a737d3c k8s.gcr.io/coredns:1.5.0] 42488424} {[mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume@sha256:4fd30d43947d4a54fc89ead7985beecfd3c9b2a93a0655a373b1608ab90bd5af mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume:v0.0.7] 22909487} {[quay.io/k8scsi/livenessprobe@sha256:8c9c8a0f330acc2e558ae6624309447d02522e76aa35af23fa19b88abf4381cb quay.io/k8scsi/livenessprobe:v1.0.2] 17415381} {[quay.io/k8scsi/csi-node-driver-registrar@sha256:13daf82fb99e951a4bff8ae5fc7c17c3a8fe7130be6400990d8f6076c32d4599 quay.io/k8scsi/csi-node-driver-registrar:v1.1.0] 15815995} {[quay.io/k8scsi/livenessprobe@sha256:dde617756e0f602adc566ab71fd885f1dad451ad3fb063ac991c95a2ff47aea5 quay.io/k8scsi/livenessprobe:v1.1.0] 14967303} {[mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume@sha256:23d8c6033f02a1ecad05127ebdc931bb871264228661bc122704b0974e4d9fdd mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume:1.0.8] 1159025} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 02:55:50.595] Nov 5 02:55:50.588: INFO: I1105 02:55:50.595] Logging kubelet events for node k8s-agentpool-15826033-vmss000001 I1105 02:55:50.669] Nov 5 02:55:50.665: INFO: I1105 02:55:50.670] Logging pods the kubelet thinks is on node k8s-agentpool-15826033-vmss000001 I1105 02:55:50.748] Nov 5 02:55:50.748: INFO: blobfuse-flexvol-installer-rw5dw started at 2019-11-05 01:34:35 +0000 UTC (0+1 container statuses recorded) I1105 02:55:50.748] Nov 5 02:55:50.748: INFO: Container blobfuse-flexvol-installer ready: true, restart count 0 I1105 02:55:50.749] Nov 5 02:55:50.748: INFO: coredns-7f68dcdbdb-jt522 started at 2019-11-05 01:34:35 +0000 UTC (0+1 container statuses recorded) I1105 02:55:50.749] Nov 5 02:55:50.748: INFO: Container coredns ready: true, restart count 0 I1105 02:55:50.749] Nov 5 02:55:50.748: INFO: azure-ip-masq-agent-6c6vn started at 2019-11-05 01:34:35 +0000 UTC (0+1 container statuses recorded) I1105 02:55:50.749] Nov 5 02:55:50.748: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 02:55:50.750] Nov 5 02:55:50.748: INFO: keyvault-flexvolume-rdlcj started at 2019-11-05 01:34:35 +0000 UTC (0+1 container statuses recorded) I1105 02:55:50.750] Nov 5 02:55:50.748: INFO: Container keyvault-flexvolume ready: true, restart count 0 I1105 02:55:50.750] Nov 5 02:55:50.748: INFO: kube-proxy-6kh57 started at 2019-11-05 01:34:35 +0000 UTC (0+1 container statuses recorded) I1105 02:55:50.750] Nov 5 02:55:50.748: INFO: Container kube-proxy ready: true, restart count 0 I1105 02:55:50.751] Nov 5 02:55:50.748: INFO: tiller-deploy-54c96cb5df-8d9fs started at 2019-11-05 01:34:35 +0000 UTC (0+1 container statuses recorded) I1105 02:55:50.751] Nov 5 02:55:50.748: INFO: Container tiller ready: true, restart count 0 I1105 02:55:50.751] Nov 5 02:55:50.748: INFO: csi-azuredisk-node-ghv5w started at 2019-11-05 01:38:24 +0000 UTC (0+3 container statuses recorded) I1105 02:55:50.751] Nov 5 02:55:50.748: INFO: Container azuredisk ready: true, restart count 0 I1105 02:55:50.751] Nov 5 02:55:50.748: INFO: Container liveness-probe ready: true, restart count 0 I1105 02:55:50.752] Nov 5 02:55:50.748: INFO: Container node-driver-registrar ready: true, restart count 0 I1105 02:55:50.752] Nov 5 02:55:50.748: INFO: csi-azuredisk-controller-6f6587bfc6-8vkfl started at 2019-11-05 01:38:24 +0000 UTC (0+6 container statuses recorded) I1105 02:55:50.752] Nov 5 02:55:50.748: INFO: Container azuredisk ready: true, restart count 0 I1105 02:55:50.752] Nov 5 02:55:50.748: INFO: Container cluster-driver-registrar ready: true, restart count 0 I1105 02:55:50.753] Nov 5 02:55:50.748: INFO: Container csi-attacher ready: true, restart count 0 I1105 02:55:50.753] Nov 5 02:55:50.748: INFO: Container csi-provisioner ready: true, restart count 0 I1105 02:55:50.753] Nov 5 02:55:50.748: INFO: Container csi-snapshotter ready: true, restart count 0 I1105 02:55:50.753] Nov 5 02:55:50.748: INFO: Container liveness-probe ready: true, restart count 0 I1105 02:55:51.008] Nov 5 02:55:51.008: INFO: I1105 02:55:51.008] Latency metrics for node k8s-agentpool-15826033-vmss000001 I1105 02:55:51.008] Nov 5 02:55:51.008: INFO: I1105 02:55:51.008] Logging node info for node k8s-master-15826033-0 I1105 02:55:51.087] Nov 5 02:55:51.085: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-15826033-0,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-15826033-0,UID:aceb0231-4645-4a9a-b296-90ada851986b,ResourceVersion:15602,Generation:0,CreationTimestamp:2019-11-05 01:35:06 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 1,kubernetes.azure.com/cluster: kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-15826033-0,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.4.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5/providers/Microsoft.Compute/virtualMachines/k8s-master-15826033-0,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 01:35:20 +0000 UTC 2019-11-05 01:35:20 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 02:55:37 +0000 UTC 2019-11-05 01:34:57 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 02:55:37 +0000 UTC 2019-11-05 01:34:57 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 02:55:37 +0000 UTC 2019-11-05 01:34:57 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 02:55:37 +0000 UTC 2019-11-05 01:35:07 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.255.5} {Hostname k8s-master-15826033-0}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:1915756c22a540dc8eb30493987e9a43,SystemUUID:742C969C-5A71-D744-8E61-F27E38B56EB2,BootID:03f0672e-8fa9-4309-956f-8e3c371902c3,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 02:55:51.088] Nov 5 02:55:51.085: INFO: I1105 02:55:51.088] Logging kubelet events for node k8s-master-15826033-0 I1105 02:55:51.162] Nov 5 02:55:51.162: INFO: I1105 02:55:51.163] Logging pods the kubelet thinks is on node k8s-master-15826033-0 I1105 02:55:51.245] Nov 5 02:55:51.245: INFO: cloud-controller-manager-k8s-master-15826033-0 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:55:51.245] Nov 5 02:55:51.245: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 02:55:51.245] Nov 5 02:55:51.245: INFO: kube-addon-manager-k8s-master-15826033-0 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:55:51.246] Nov 5 02:55:51.245: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 02:55:51.246] Nov 5 02:55:51.245: INFO: kube-apiserver-k8s-master-15826033-0 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:55:51.246] Nov 5 02:55:51.245: INFO: Container kube-apiserver ready: true, restart count 0 I1105 02:55:51.246] Nov 5 02:55:51.245: INFO: kube-controller-manager-k8s-master-15826033-0 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:55:51.246] Nov 5 02:55:51.245: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 02:55:51.247] Nov 5 02:55:51.245: INFO: kube-scheduler-k8s-master-15826033-0 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:55:51.247] Nov 5 02:55:51.245: INFO: Container kube-scheduler ready: true, restart count 0 I1105 02:55:51.247] Nov 5 02:55:51.245: INFO: azure-ip-masq-agent-wgdpx started at 2019-11-05 01:35:06 +0000 UTC (0+1 container statuses recorded) I1105 02:55:51.247] Nov 5 02:55:51.245: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 02:55:51.247] Nov 5 02:55:51.245: INFO: kube-proxy-7d2px started at 2019-11-05 01:35:06 +0000 UTC (0+1 container statuses recorded) I1105 02:55:51.247] Nov 5 02:55:51.245: INFO: Container kube-proxy ready: true, restart count 0 I1105 02:55:51.496] Nov 5 02:55:51.496: INFO: I1105 02:55:51.497] Latency metrics for node k8s-master-15826033-0 I1105 02:55:51.497] Nov 5 02:55:51.496: INFO: I1105 02:55:51.497] Logging node info for node k8s-master-15826033-1 I1105 02:55:51.577] Nov 5 02:55:51.573: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-15826033-1,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-15826033-1,UID:1a13ec61-c89a-456c-8c94-bfc8e717c2e7,ResourceVersion:15606,Generation:0,CreationTimestamp:2019-11-05 01:35:07 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 1,kubernetes.azure.com/cluster: kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-15826033-1,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.5.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5/providers/Microsoft.Compute/virtualMachines/k8s-master-15826033-1,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 01:35:31 +0000 UTC 2019-11-05 01:35:31 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 02:55:37 +0000 UTC 2019-11-05 01:35:00 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 02:55:37 +0000 UTC 2019-11-05 01:35:00 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 02:55:37 +0000 UTC 2019-11-05 01:35:00 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 02:55:37 +0000 UTC 2019-11-05 01:35:07 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.255.6} {Hostname k8s-master-15826033-1}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:e0f2bccddd5b4555b180d14081ab59a8,SystemUUID:AE126631-CAD2-B54F-B22E-1F499342A4D6,BootID:509b9a9d-b046-4861-aa35-8bcb77465de2,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 02:55:51.577] Nov 5 02:55:51.573: INFO: I1105 02:55:51.577] Logging kubelet events for node k8s-master-15826033-1 I1105 02:55:51.651] Nov 5 02:55:51.651: INFO: I1105 02:55:51.651] Logging pods the kubelet thinks is on node k8s-master-15826033-1 I1105 02:55:51.746] Nov 5 02:55:51.739: INFO: cloud-controller-manager-k8s-master-15826033-1 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:55:51.746] Nov 5 02:55:51.739: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 02:55:51.746] Nov 5 02:55:51.739: INFO: kube-addon-manager-k8s-master-15826033-1 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:55:51.746] Nov 5 02:55:51.739: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 02:55:51.747] Nov 5 02:55:51.739: INFO: kube-apiserver-k8s-master-15826033-1 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:55:51.747] Nov 5 02:55:51.739: INFO: Container kube-apiserver ready: true, restart count 0 I1105 02:55:51.747] Nov 5 02:55:51.739: INFO: kube-controller-manager-k8s-master-15826033-1 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:55:51.748] Nov 5 02:55:51.739: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 02:55:51.748] Nov 5 02:55:51.739: INFO: kube-scheduler-k8s-master-15826033-1 started at 2019-11-05 01:36:16 +0000 UTC (0+1 container statuses recorded) I1105 02:55:51.748] Nov 5 02:55:51.739: INFO: Container kube-scheduler ready: true, restart count 0 I1105 02:55:51.748] Nov 5 02:55:51.739: INFO: azure-ip-masq-agent-w6dpr started at 2019-11-05 01:35:10 +0000 UTC (0+1 container statuses recorded) I1105 02:55:51.748] Nov 5 02:55:51.739: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 02:55:51.749] Nov 5 02:55:51.739: INFO: kube-proxy-8kmr5 started at 2019-11-05 01:35:10 +0000 UTC (0+1 container statuses recorded) I1105 02:55:51.749] Nov 5 02:55:51.739: INFO: Container kube-proxy ready: true, restart count 0 I1105 02:55:51.998] Nov 5 02:55:51.997: INFO: I1105 02:55:51.998] Latency metrics for node k8s-master-15826033-1 I1105 02:55:51.998] Nov 5 02:55:51.997: INFO: I1105 02:55:51.998] Logging node info for node k8s-master-15826033-2 I1105 02:55:52.085] Nov 5 02:55:52.078: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-15826033-2,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-15826033-2,UID:acb3a851-d6df-4549-ba56-4b7e96553c17,ResourceVersion:15522,Generation:0,CreationTimestamp:2019-11-05 01:34:09 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 2,kubernetes.azure.com/cluster: kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-15826033-2,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.1.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5/providers/Microsoft.Compute/virtualMachines/k8s-master-15826033-2,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 01:34:44 +0000 UTC 2019-11-05 01:34:44 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 02:55:09 +0000 UTC 2019-11-05 01:34:07 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 02:55:09 +0000 UTC 2019-11-05 01:34:07 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 02:55:09 +0000 UTC 2019-11-05 01:34:07 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 02:55:09 +0000 UTC 2019-11-05 01:34:29 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{Hostname k8s-master-15826033-2} {InternalIP 10.240.255.7}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:0885b0c2f6174106bde4fa0eae2cbc6b,SystemUUID:C176121C-5D7A-6243-9B05-8B68F8022B64,BootID:30430638-7ab6-4761-b984-2387e1a3f5b2,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 02:55:52.085] Nov 5 02:55:52.078: INFO: I1105 02:55:52.085] Logging kubelet events for node k8s-master-15826033-2 I1105 02:55:52.162] Nov 5 02:55:52.162: INFO: I1105 02:55:52.162] Logging pods the kubelet thinks is on node k8s-master-15826033-2 I1105 02:55:52.245] Nov 5 02:55:52.244: INFO: cloud-controller-manager-k8s-master-15826033-2 started at 2019-11-05 01:33:54 +0000 UTC (0+1 container statuses recorded) I1105 02:55:52.245] Nov 5 02:55:52.244: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 02:55:52.245] Nov 5 02:55:52.244: INFO: kube-addon-manager-k8s-master-15826033-2 started at 2019-11-05 01:33:54 +0000 UTC (0+1 container statuses recorded) I1105 02:55:52.245] Nov 5 02:55:52.244: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 02:55:52.246] Nov 5 02:55:52.244: INFO: kube-apiserver-k8s-master-15826033-2 started at 2019-11-05 01:33:54 +0000 UTC (0+1 container statuses recorded) I1105 02:55:52.246] Nov 5 02:55:52.244: INFO: Container kube-apiserver ready: true, restart count 0 I1105 02:55:52.246] Nov 5 02:55:52.244: INFO: kube-controller-manager-k8s-master-15826033-2 started at 2019-11-05 01:33:54 +0000 UTC (0+1 container statuses recorded) I1105 02:55:52.246] Nov 5 02:55:52.244: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 02:55:52.247] Nov 5 02:55:52.244: INFO: kube-scheduler-k8s-master-15826033-2 started at 2019-11-05 01:33:54 +0000 UTC (0+1 container statuses recorded) I1105 02:55:52.247] Nov 5 02:55:52.244: INFO: Container kube-scheduler ready: true, restart count 0 I1105 02:55:52.247] Nov 5 02:55:52.244: INFO: azure-ip-masq-agent-hw7s5 started at 2019-11-05 01:34:34 +0000 UTC (0+1 container statuses recorded) I1105 02:55:52.247] Nov 5 02:55:52.244: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 02:55:52.247] Nov 5 02:55:52.244: INFO: kube-proxy-5xpp6 started at 2019-11-05 01:34:34 +0000 UTC (0+1 container statuses recorded) I1105 02:55:52.248] Nov 5 02:55:52.244: INFO: Container kube-proxy ready: true, restart count 0 I1105 02:55:52.503] Nov 5 02:55:52.503: INFO: I1105 02:55:52.504] Latency metrics for node k8s-master-15826033-2 I1105 02:55:52.504] Nov 5 02:55:52.503: INFO: I1105 02:55:52.504] Logging node info for node k8s-master-15826033-3 I1105 02:55:52.584] Nov 5 02:55:52.581: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-15826033-3,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-15826033-3,UID:b7702272-1876-434e-b6a0-17fa34c6757f,ResourceVersion:15600,Generation:0,CreationTimestamp:2019-11-05 01:35:06 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 0,kubernetes.azure.com/cluster: kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-15826033-3,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.3.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5/providers/Microsoft.Compute/virtualMachines/k8s-master-15826033-3,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 01:35:17 +0000 UTC 2019-11-05 01:35:17 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 02:55:36 +0000 UTC 2019-11-05 01:35:00 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 02:55:36 +0000 UTC 2019-11-05 01:35:00 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 02:55:36 +0000 UTC 2019-11-05 01:35:00 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 02:55:36 +0000 UTC 2019-11-05 01:35:06 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.255.8} {Hostname k8s-master-15826033-3}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:fbf366963961422c9d90bedf6639c8e1,SystemUUID:5388948D-1A8D-EA41-B8B8-7F4523EDF5AA,BootID:42f5884c-7418-486a-8cf3-75188a331f38,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 02:55:52.584] Nov 5 02:55:52.581: INFO: I1105 02:55:52.585] Logging kubelet events for node k8s-master-15826033-3 I1105 02:55:52.658] Nov 5 02:55:52.658: INFO: I1105 02:55:52.659] Logging pods the kubelet thinks is on node k8s-master-15826033-3 I1105 02:55:52.739] Nov 5 02:55:52.739: INFO: kube-proxy-xz24z started at 2019-11-05 01:35:06 +0000 UTC (0+1 container statuses recorded) I1105 02:55:52.740] Nov 5 02:55:52.739: INFO: Container kube-proxy ready: true, restart count 0 I1105 02:55:52.740] Nov 5 02:55:52.739: INFO: azure-ip-masq-agent-dqfpw started at 2019-11-05 01:35:06 +0000 UTC (0+1 container statuses recorded) I1105 02:55:52.740] Nov 5 02:55:52.739: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 02:55:52.740] Nov 5 02:55:52.739: INFO: cloud-controller-manager-k8s-master-15826033-3 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:55:52.740] Nov 5 02:55:52.739: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 02:55:52.740] Nov 5 02:55:52.739: INFO: kube-addon-manager-k8s-master-15826033-3 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:55:52.740] Nov 5 02:55:52.739: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 02:55:52.741] Nov 5 02:55:52.739: INFO: kube-apiserver-k8s-master-15826033-3 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:55:52.741] Nov 5 02:55:52.739: INFO: Container kube-apiserver ready: true, restart count 0 I1105 02:55:52.741] Nov 5 02:55:52.739: INFO: kube-controller-manager-k8s-master-15826033-3 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:55:52.741] Nov 5 02:55:52.739: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 02:55:52.741] Nov 5 02:55:52.739: INFO: kube-scheduler-k8s-master-15826033-3 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 02:55:52.741] Nov 5 02:55:52.739: INFO: Container kube-scheduler ready: true, restart count 0 I1105 02:55:52.988] Nov 5 02:55:52.988: INFO: I1105 02:55:53.312] Latency metrics for node k8s-master-15826033-3 I1105 02:55:53.312] Nov 5 02:55:52.988: INFO: I1105 02:55:53.312] Logging node info for node k8s-master-15826033-4 I1105 02:55:53.316] Nov 5 02:55:53.066: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-15826033-4,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-15826033-4,UID:276daef1-6c05-41e3-bacd-abc93d35cc6d,ResourceVersion:15607,Generation:0,CreationTimestamp:2019-11-05 01:35:07 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 0,kubernetes.azure.com/cluster: kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-15826033-4,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.6.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5/providers/Microsoft.Compute/virtualMachines/k8s-master-15826033-4,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 01:35:41 +0000 UTC 2019-11-05 01:35:41 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 02:55:38 +0000 UTC 2019-11-05 01:35:02 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 02:55:38 +0000 UTC 2019-11-05 01:35:02 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 02:55:38 +0000 UTC 2019-11-05 01:35:02 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 02:55:38 +0000 UTC 2019-11-05 01:35:08 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.255.9} {Hostname k8s-master-15826033-4}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:79051fc0bdeb4e32a56acbaf5972ce3a,SystemUUID:C71606B5-7F08-9341-A26D-8A8D9D84FCAE,BootID:4bdf7cf3-2b83-42c7-a7ba-926ebe9ca75b,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 02:55:53.316] Nov 5 02:55:53.066: INFO: I1105 02:55:53.316] Logging kubelet events for node k8s-master-15826033-4 I1105 02:55:53.316] Nov 5 02:55:53.143: INFO: I1105 02:55:53.316] Logging pods the kubelet thinks is on node k8s-master-15826033-4 I1105 02:55:53.317] Nov 5 02:55:53.225: INFO: cloud-controller-manager-k8s-master-15826033-4 started at 2019-11-05 01:34:56 +0000 UTC (0+1 container statuses recorded) I1105 02:55:53.317] Nov 5 02:55:53.225: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 02:55:53.317] Nov 5 02:55:53.225: INFO: kube-addon-manager-k8s-master-15826033-4 started at 2019-11-05 01:34:56 +0000 UTC (0+1 container statuses recorded) I1105 02:55:53.318] Nov 5 02:55:53.225: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 02:55:53.318] Nov 5 02:55:53.225: INFO: azure-ip-masq-agent-9nrj8 started at 2019-11-05 01:35:11 +0000 UTC (0+1 container statuses recorded) I1105 02:55:53.318] Nov 5 02:55:53.225: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 02:55:53.318] Nov 5 02:55:53.225: INFO: kube-proxy-rmngk started at 2019-11-05 01:35:11 +0000 UTC (0+1 container statuses recorded) I1105 02:55:53.319] Nov 5 02:55:53.225: INFO: Container kube-proxy ready: true, restart count 0 I1105 02:55:53.319] Nov 5 02:55:53.225: INFO: kube-apiserver-k8s-master-15826033-4 started at 2019-11-05 01:34:55 +0000 UTC (0+1 container statuses recorded) I1105 02:55:53.319] Nov 5 02:55:53.225: INFO: Container kube-apiserver ready: true, restart count 0 I1105 02:55:53.319] Nov 5 02:55:53.225: INFO: kube-controller-manager-k8s-master-15826033-4 started at 2019-11-05 01:34:55 +0000 UTC (0+1 container statuses recorded) I1105 02:55:53.319] Nov 5 02:55:53.225: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 02:55:53.320] Nov 5 02:55:53.225: INFO: kube-scheduler-k8s-master-15826033-4 started at 2019-11-05 01:34:55 +0000 UTC (0+1 container statuses recorded) I1105 02:55:53.320] Nov 5 02:55:53.225: INFO: Container kube-scheduler ready: true, restart count 0 I1105 02:55:53.478] Nov 5 02:55:53.477: INFO: I1105 02:55:53.478] Latency metrics for node k8s-master-15826033-4 I1105 02:55:53.478] Nov 5 02:55:53.477: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready I1105 02:55:53.555] STEP: Destroying namespace "azuredisk-8159" for this suite. I1105 02:56:07.798] Nov 5 02:56:07.797: INFO: Waiting up to 30s for server preferred namespaced resources to be successfully discovered I1105 02:56:10.760] Nov 5 02:56:10.760: INFO: namespace azuredisk-8159 deletion completed in 17.205049907s I1105 02:56:10.760] I1105 02:56:10.761] • Failure [323.292 seconds] I1105 02:56:10.761] [azuredisk-csi-e2e] Dynamic Provisioning I1105 02:56:10.761] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:33 I1105 02:56:10.761] [multi-az] I1105 02:56:10.761] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:40 I1105 02:56:10.761] should create a volume on demand [It] I1105 02:56:10.762] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:78 I1105 02:56:10.762] I1105 02:56:10.762] Unexpected error: I1105 02:56:10.762] <*errors.errorString | 0xc00083ebf0>: { I1105 02:56:10.762] s: "PersistentVolumeClaims [pvc-5mr2q] not all in phase Bound within 5m0s", I1105 02:56:10.762] } I1105 02:56:10.762] PersistentVolumeClaims [pvc-5mr2q] not all in phase Bound within 5m0s I1105 02:56:10.763] occurred I1105 02:56:10.763] I1105 02:56:10.763] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/testsuites/testsuites.go:278 I1105 02:56:10.763] ------------------------------ I1105 02:56:10.763] [azuredisk-csi-e2e] Dynamic Provisioning [multi-az] I1105 02:56:10.763] should create a raw block volume on demand I1105 02:56:10.763] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:101 I1105 02:56:10.763] [BeforeEach] [multi-az] I1105 02:56:10.764] /go/src/sigs.k8s.io/azuredisk-csi-driver/vendor/k8s.io/kubernetes/test/e2e/framework/framework.go:150 I1105 02:56:10.764] STEP: Creating a kubernetes client I1105 02:56:10.764] Nov 5 02:56:10.760: INFO: >>> kubeConfig: /workspace/aks978705698/kubeconfig/kubeconfig.eastus2.json I1105 02:56:10.764] STEP: Building a namespace api object, basename azuredisk I1105 02:56:10.998] STEP: Binding the e2e-test-privileged-psp PodSecurityPolicy to the default service account in azuredisk-7632 I1105 02:56:11.338] STEP: Waiting for a default service account to be provisioned in namespace I1105 02:56:11.345] [BeforeEach] [multi-az] I1105 02:56:11.346] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:58 I1105 02:56:11.346] [It] should create a raw block volume on demand I1105 02:56:11.346] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:101 I1105 02:56:11.346] STEP: setting up the StorageClass I1105 02:56:11.347] STEP: creating a StorageClass I1105 02:56:11.424] STEP: setting up the PVC and PV I1105 02:56:11.424] STEP: creating a PVC I1105 02:56:11.513] STEP: waiting for PVC to be in phase "Bound" I1105 02:56:11.514] Nov 5 02:56:11.510: INFO: Waiting up to 5m0s for PersistentVolumeClaims [pvc-vshfr] to have phase Bound I1105 02:56:11.588] Nov 5 02:56:11.587: INFO: PersistentVolumeClaim pvc-vshfr found but phase is Pending instead of Bound. I1105 02:56:13.665] Nov 5 02:56:13.665: INFO: PersistentVolumeClaim pvc-vshfr found but phase is Pending instead of Bound. I1105 02:56:15.744] Nov 5 02:56:15.743: INFO: PersistentVolumeClaim pvc-vshfr found but phase is Pending instead of Bound. I1105 02:56:17.827] Nov 5 02:56:17.826: INFO: PersistentVolumeClaim pvc-vshfr found but phase is Pending instead of Bound. I1105 02:56:19.904] Nov 5 02:56:19.904: INFO: PersistentVolumeClaim pvc-vshfr found but phase is Pending instead of Bound. I1105 02:56:21.985] Nov 5 02:56:21.984: INFO: PersistentVolumeClaim pvc-vshfr found but phase is Pending instead of Bound. I1105 02:56:24.063] Nov 5 02:56:24.063: INFO: PersistentVolumeClaim pvc-vshfr found but phase is Pending instead of Bound. I1105 02:56:26.142] Nov 5 02:56:26.141: INFO: PersistentVolumeClaim pvc-vshfr found but phase is Pending instead of Bound. I1105 02:56:28.224] Nov 5 02:56:28.219: INFO: PersistentVolumeClaim pvc-vshfr found but phase is Pending instead of Bound. I1105 02:56:30.302] Nov 5 02:56:30.298: INFO: PersistentVolumeClaim pvc-vshfr found but phase is Pending instead of Bound. I1105 02:56:32.386] Nov 5 02:56:32.382: INFO: PersistentVolumeClaim pvc-vshfr found but phase is Pending instead of Bound. I1105 02:56:34.460] Nov 5 02:56:34.459: INFO: PersistentVolumeClaim pvc-vshfr found but phase is Pending instead of Bound. I1105 02:56:36.542] Nov 5 02:56:36.542: INFO: PersistentVolumeClaim pvc-vshfr found but phase is Pending instead of Bound. I1105 02:56:38.638] Nov 5 02:56:38.634: INFO: PersistentVolumeClaim pvc-vshfr found but phase is Pending instead of Bound. I1105 02:56:40.712] Nov 5 02:56:40.711: INFO: PersistentVolumeClaim pvc-vshfr found but phase is Pending instead of Bound. I1105 02:56:42.789] Nov 5 02:56:42.789: INFO: PersistentVolumeClaim pvc-vshfr found but phase is Pending instead of Bound. I1105 02:56:44.867] Nov 5 02:56:44.867: INFO: PersistentVolumeClaim pvc-vshfr found but phase is Pending instead of Bound. I1105 02:56:46.944] Nov 5 02:56:46.944: INFO: PersistentVolumeClaim pvc-vshfr found but phase is Pending instead of Bound. I1105 02:56:49.022] Nov 5 02:56:49.022: INFO: PersistentVolumeClaim pvc-vshfr found but phase is Pending instead of Bound. I1105 02:56:51.290] Nov 5 02:56:51.099: INFO: PersistentVolumeClaim pvc-vshfr found but phase is Pending instead of Bound. I1105 02:56:53.177] Nov 5 02:56:53.177: INFO: PersistentVolumeClaim pvc-vshfr found but phase is Pending instead of Bound. I1105 02:56:55.258] Nov 5 02:56:55.258: INFO: PersistentVolumeClaim pvc-vshfr found but phase is Pending instead of Bound. I1105 02:56:57.336] Nov 5 02:56:57.336: INFO: PersistentVolumeClaim pvc-vshfr found but phase is Pending instead of Bound. I1105 02:57:00.401] Nov 5 02:56:59.413: INFO: PersistentVolumeClaim pvc-vshfr found but phase is Pending instead of Bound. I1105 02:57:01.492] Nov 5 02:57:01.491: INFO: PersistentVolumeClaim pvc-vshfr found but phase is Pending instead of Bound. I1105 02:57:03.570] Nov 5 02:57:03.569: INFO: PersistentVolumeClaim pvc-vshfr found but phase is Pending instead of Bound. I1105 02:57:05.647] Nov 5 02:57:05.647: INFO: PersistentVolumeClaim pvc-vshfr found but phase is Pending instead of Bound. I1105 02:57:07.725] Nov 5 02:57:07.725: INFO: PersistentVolumeClaim pvc-vshfr found but phase is Pending instead of Bound. I1105 02:57:09.803] Nov 5 02:57:09.802: INFO: PersistentVolumeClaim pvc-vshfr found but phase is Pending instead of Bound. I1105 02:57:11.881] Nov 5 02:57:11.880: INFO: PersistentVolumeClaim pvc-vshfr found but phase is Pending instead of Bound. I1105 02:57:13.962] Nov 5 02:57:13.961: INFO: PersistentVolumeClaim pvc-vshfr found but phase is Pending instead of Bound. I1105 02:57:16.039] Nov 5 02:57:16.039: INFO: PersistentVolumeClaim pvc-vshfr found but phase is Pending instead of Bound. I1105 02:57:18.452] Nov 5 02:57:18.117: INFO: PersistentVolumeClaim pvc-vshfr found but phase is Pending instead of Bound. I1105 02:57:20.195] Nov 5 02:57:20.195: INFO: PersistentVolumeClaim pvc-vshfr found but phase is Pending instead of Bound. I1105 02:57:23.114] Nov 5 02:57:22.272: INFO: PersistentVolumeClaim pvc-vshfr found but phase is Pending instead of Bound. I1105 02:57:24.351] Nov 5 02:57:24.350: INFO: PersistentVolumeClaim pvc-vshfr found but phase is Pending instead of Bound. I1105 02:57:27.518] Nov 5 02:57:26.428: INFO: PersistentVolumeClaim pvc-vshfr found but phase is Pending instead of Bound. I1105 02:57:28.506] Nov 5 02:57:28.506: INFO: PersistentVolumeClaim pvc-vshfr found but phase is Pending instead of Bound. I1105 02:57:30.584] Nov 5 02:57:30.584: INFO: PersistentVolumeClaim pvc-vshfr found but phase is Pending instead of Bound. I1105 02:57:32.662] Nov 5 02:57:32.662: INFO: PersistentVolumeClaim pvc-vshfr found but phase is Pending instead of Bound. I1105 02:57:34.740] Nov 5 02:57:34.740: INFO: PersistentVolumeClaim pvc-vshfr found but phase is Pending instead of Bound. I1105 02:57:52.579] Nov 5 02:57:36.822: INFO: PersistentVolumeClaim pvc-vshfr found but phase is Pending instead of Bound. I1105 02:57:52.579] Nov 5 02:57:38.899: INFO: PersistentVolumeClaim pvc-vshfr found but phase is Pending instead of Bound. I1105 02:57:52.579] Nov 5 02:57:40.979: INFO: PersistentVolumeClaim pvc-vshfr found but phase is Pending instead of Bound. I1105 02:57:52.579] Nov 5 02:57:43.057: INFO: PersistentVolumeClaim pvc-vshfr found but phase is Pending instead of Bound. I1105 02:57:52.580] Nov 5 02:57:45.146: INFO: PersistentVolumeClaim pvc-vshfr found but phase is Pending instead of Bound. I1105 02:57:52.580] Nov 5 02:57:47.224: INFO: PersistentVolumeClaim pvc-vshfr found but phase is Pending instead of Bound. I1105 02:57:52.580] Nov 5 02:57:49.303: INFO: PersistentVolumeClaim pvc-vshfr found but phase is Pending instead of Bound. I1105 02:57:52.580] Nov 5 02:57:51.381: INFO: PersistentVolumeClaim pvc-vshfr found but phase is Pending instead of Bound. I1105 02:57:53.459] Nov 5 02:57:53.459: INFO: PersistentVolumeClaim pvc-vshfr found but phase is Pending instead of Bound. I1105 02:58:10.776] Nov 5 02:57:55.537: INFO: PersistentVolumeClaim pvc-vshfr found but phase is Pending instead of Bound. I1105 02:58:10.777] Nov 5 02:57:57.615: INFO: PersistentVolumeClaim pvc-vshfr found but phase is Pending instead of Bound. I1105 02:58:10.777] Nov 5 02:57:59.693: INFO: PersistentVolumeClaim pvc-vshfr found but phase is Pending instead of Bound. I1105 02:58:10.777] Nov 5 02:58:01.771: INFO: PersistentVolumeClaim pvc-vshfr found but phase is Pending instead of Bound. I1105 02:58:10.777] Nov 5 02:58:03.848: INFO: PersistentVolumeClaim pvc-vshfr found but phase is Pending instead of Bound. I1105 02:58:10.778] Nov 5 02:58:05.927: INFO: PersistentVolumeClaim pvc-vshfr found but phase is Pending instead of Bound. I1105 02:58:10.778] Nov 5 02:58:08.014: INFO: PersistentVolumeClaim pvc-vshfr found but phase is Pending instead of Bound. I1105 02:58:10.778] Nov 5 02:58:10.092: INFO: PersistentVolumeClaim pvc-vshfr found but phase is Pending instead of Bound. I1105 02:58:12.170] Nov 5 02:58:12.170: INFO: PersistentVolumeClaim pvc-vshfr found but phase is Pending instead of Bound. I1105 02:58:19.239] Nov 5 02:58:14.248: INFO: PersistentVolumeClaim pvc-vshfr found but phase is Pending instead of Bound. I1105 02:58:19.240] Nov 5 02:58:16.326: INFO: PersistentVolumeClaim pvc-vshfr found but phase is Pending instead of Bound. I1105 02:58:19.240] Nov 5 02:58:18.404: INFO: PersistentVolumeClaim pvc-vshfr found but phase is Pending instead of Bound. I1105 02:58:20.482] Nov 5 02:58:20.482: INFO: PersistentVolumeClaim pvc-vshfr found but phase is Pending instead of Bound. I1105 02:58:23.445] Nov 5 02:58:22.559: INFO: PersistentVolumeClaim pvc-vshfr found but phase is Pending instead of Bound. I1105 02:58:24.637] Nov 5 02:58:24.637: INFO: PersistentVolumeClaim pvc-vshfr found but phase is Pending instead of Bound. I1105 02:58:26.715] Nov 5 02:58:26.715: INFO: PersistentVolumeClaim pvc-vshfr found but phase is Pending instead of Bound. I1105 02:58:28.792] Nov 5 02:58:28.792: INFO: PersistentVolumeClaim pvc-vshfr found but phase is Pending instead of Bound. I1105 02:58:30.870] Nov 5 02:58:30.870: INFO: PersistentVolumeClaim pvc-vshfr found but phase is Pending instead of Bound. I1105 02:58:32.948] Nov 5 02:58:32.947: INFO: PersistentVolumeClaim pvc-vshfr found but phase is Pending instead of Bound. I1105 02:58:35.025] Nov 5 02:58:35.025: INFO: PersistentVolumeClaim pvc-vshfr found but phase is Pending instead of Bound. I1105 02:58:37.120] Nov 5 02:58:37.120: INFO: PersistentVolumeClaim pvc-vshfr found but phase is Pending instead of Bound. I1105 02:58:39.198] Nov 5 02:58:39.198: INFO: PersistentVolumeClaim pvc-vshfr found but phase is Pending instead of Bound. I1105 02:58:41.276] Nov 5 02:58:41.276: INFO: PersistentVolumeClaim pvc-vshfr found but phase is Pending instead of Bound. I1105 02:58:43.354] Nov 5 02:58:43.354: INFO: PersistentVolumeClaim pvc-vshfr found but phase is Pending instead of Bound. I1105 02:58:45.438] Nov 5 02:58:45.434: INFO: PersistentVolumeClaim pvc-vshfr found but phase is Pending instead of Bound. I1105 02:58:47.512] Nov 5 02:58:47.512: INFO: PersistentVolumeClaim pvc-vshfr found but phase is Pending instead of Bound. I1105 02:58:49.590] Nov 5 02:58:49.590: INFO: PersistentVolumeClaim pvc-vshfr found but phase is Pending instead of Bound. I1105 02:58:51.669] Nov 5 02:58:51.669: INFO: PersistentVolumeClaim pvc-vshfr found but phase is Pending instead of Bound. I1105 02:58:53.747] Nov 5 02:58:53.746: INFO: PersistentVolumeClaim pvc-vshfr found but phase is Pending instead of Bound. I1105 02:58:55.827] Nov 5 02:58:55.826: INFO: PersistentVolumeClaim pvc-vshfr found but phase is Pending instead of Bound. I1105 02:58:57.906] Nov 5 02:58:57.905: INFO: PersistentVolumeClaim pvc-vshfr found but phase is Pending instead of Bound. I1105 02:58:59.983] Nov 5 02:58:59.982: INFO: PersistentVolumeClaim pvc-vshfr found but phase is Pending instead of Bound. I1105 02:59:02.060] Nov 5 02:59:02.060: INFO: PersistentVolumeClaim pvc-vshfr found but phase is Pending instead of Bound. I1105 02:59:04.143] Nov 5 02:59:04.138: INFO: PersistentVolumeClaim pvc-vshfr found but phase is Pending instead of Bound. I1105 02:59:06.215] Nov 5 02:59:06.215: INFO: PersistentVolumeClaim pvc-vshfr found but phase is Pending instead of Bound. I1105 02:59:08.294] Nov 5 02:59:08.293: INFO: PersistentVolumeClaim pvc-vshfr found but phase is Pending instead of Bound. I1105 02:59:10.372] Nov 5 02:59:10.372: INFO: PersistentVolumeClaim pvc-vshfr found but phase is Pending instead of Bound. I1105 02:59:12.457] Nov 5 02:59:12.457: INFO: PersistentVolumeClaim pvc-vshfr found but phase is Pending instead of Bound. I1105 02:59:14.535] Nov 5 02:59:14.534: INFO: PersistentVolumeClaim pvc-vshfr found but phase is Pending instead of Bound. I1105 02:59:16.612] Nov 5 02:59:16.612: INFO: PersistentVolumeClaim pvc-vshfr found but phase is Pending instead of Bound. I1105 02:59:18.690] Nov 5 02:59:18.689: INFO: PersistentVolumeClaim pvc-vshfr found but phase is Pending instead of Bound. I1105 02:59:20.778] Nov 5 02:59:20.776: INFO: PersistentVolumeClaim pvc-vshfr found but phase is Pending instead of Bound. I1105 02:59:22.854] Nov 5 02:59:22.854: INFO: PersistentVolumeClaim pvc-vshfr found but phase is Pending instead of Bound. I1105 02:59:24.932] Nov 5 02:59:24.932: INFO: PersistentVolumeClaim pvc-vshfr found but phase is Pending instead of Bound. I1105 02:59:27.010] Nov 5 02:59:27.009: INFO: PersistentVolumeClaim pvc-vshfr found but phase is Pending instead of Bound. I1105 02:59:29.410] Nov 5 02:59:29.131: INFO: PersistentVolumeClaim pvc-vshfr found but phase is Pending instead of Bound. I1105 02:59:31.209] Nov 5 02:59:31.209: INFO: PersistentVolumeClaim pvc-vshfr found but phase is Pending instead of Bound. I1105 02:59:33.287] Nov 5 02:59:33.287: INFO: PersistentVolumeClaim pvc-vshfr found but phase is Pending instead of Bound. I1105 02:59:35.364] Nov 5 02:59:35.364: INFO: PersistentVolumeClaim pvc-vshfr found but phase is Pending instead of Bound. I1105 02:59:37.442] Nov 5 02:59:37.441: INFO: PersistentVolumeClaim pvc-vshfr found but phase is Pending instead of Bound. I1105 02:59:39.520] Nov 5 02:59:39.520: INFO: PersistentVolumeClaim pvc-vshfr found but phase is Pending instead of Bound. I1105 02:59:41.598] Nov 5 02:59:41.598: INFO: PersistentVolumeClaim pvc-vshfr found but phase is Pending instead of Bound. I1105 02:59:43.676] Nov 5 02:59:43.676: INFO: PersistentVolumeClaim pvc-vshfr found but phase is Pending instead of Bound. I1105 02:59:45.754] Nov 5 02:59:45.753: INFO: PersistentVolumeClaim pvc-vshfr found but phase is Pending instead of Bound. I1105 02:59:47.831] Nov 5 02:59:47.831: INFO: PersistentVolumeClaim pvc-vshfr found but phase is Pending instead of Bound. I1105 02:59:49.909] Nov 5 02:59:49.909: INFO: PersistentVolumeClaim pvc-vshfr found but phase is Pending instead of Bound. I1105 02:59:51.987] Nov 5 02:59:51.986: INFO: PersistentVolumeClaim pvc-vshfr found but phase is Pending instead of Bound. I1105 02:59:54.066] Nov 5 02:59:54.064: INFO: PersistentVolumeClaim pvc-vshfr found but phase is Pending instead of Bound. I1105 02:59:56.143] Nov 5 02:59:56.143: INFO: PersistentVolumeClaim pvc-vshfr found but phase is Pending instead of Bound. I1105 02:59:58.220] Nov 5 02:59:58.220: INFO: PersistentVolumeClaim pvc-vshfr found but phase is Pending instead of Bound. I1105 03:00:00.298] Nov 5 03:00:00.298: INFO: PersistentVolumeClaim pvc-vshfr found but phase is Pending instead of Bound. I1105 03:00:02.376] Nov 5 03:00:02.376: INFO: PersistentVolumeClaim pvc-vshfr found but phase is Pending instead of Bound. I1105 03:00:04.458] Nov 5 03:00:04.458: INFO: PersistentVolumeClaim pvc-vshfr found but phase is Pending instead of Bound. I1105 03:00:06.543] Nov 5 03:00:06.542: INFO: PersistentVolumeClaim pvc-vshfr found but phase is Pending instead of Bound. I1105 03:00:08.620] Nov 5 03:00:08.619: INFO: PersistentVolumeClaim pvc-vshfr found but phase is Pending instead of Bound. I1105 03:00:10.697] Nov 5 03:00:10.697: INFO: PersistentVolumeClaim pvc-vshfr found but phase is Pending instead of Bound. I1105 03:00:12.775] Nov 5 03:00:12.775: INFO: PersistentVolumeClaim pvc-vshfr found but phase is Pending instead of Bound. I1105 03:00:14.853] Nov 5 03:00:14.853: INFO: PersistentVolumeClaim pvc-vshfr found but phase is Pending instead of Bound. I1105 03:00:16.931] Nov 5 03:00:16.931: INFO: PersistentVolumeClaim pvc-vshfr found but phase is Pending instead of Bound. I1105 03:00:19.009] Nov 5 03:00:19.009: INFO: PersistentVolumeClaim pvc-vshfr found but phase is Pending instead of Bound. I1105 03:00:21.156] Nov 5 03:00:21.156: INFO: PersistentVolumeClaim pvc-vshfr found but phase is Pending instead of Bound. I1105 03:00:23.233] Nov 5 03:00:23.233: INFO: PersistentVolumeClaim pvc-vshfr found but phase is Pending instead of Bound. I1105 03:00:25.311] Nov 5 03:00:25.311: INFO: PersistentVolumeClaim pvc-vshfr found but phase is Pending instead of Bound. I1105 03:00:27.392] Nov 5 03:00:27.391: INFO: PersistentVolumeClaim pvc-vshfr found but phase is Pending instead of Bound. I1105 03:00:29.470] Nov 5 03:00:29.470: INFO: PersistentVolumeClaim pvc-vshfr found but phase is Pending instead of Bound. I1105 03:00:31.548] Nov 5 03:00:31.548: INFO: PersistentVolumeClaim pvc-vshfr found but phase is Pending instead of Bound. I1105 03:00:33.632] Nov 5 03:00:33.631: INFO: PersistentVolumeClaim pvc-vshfr found but phase is Pending instead of Bound. I1105 03:00:35.709] Nov 5 03:00:35.709: INFO: PersistentVolumeClaim pvc-vshfr found but phase is Pending instead of Bound. I1105 03:00:37.787] Nov 5 03:00:37.787: INFO: PersistentVolumeClaim pvc-vshfr found but phase is Pending instead of Bound. I1105 03:00:39.867] Nov 5 03:00:39.867: INFO: PersistentVolumeClaim pvc-vshfr found but phase is Pending instead of Bound. I1105 03:00:41.945] Nov 5 03:00:41.944: INFO: PersistentVolumeClaim pvc-vshfr found but phase is Pending instead of Bound. I1105 03:00:44.022] Nov 5 03:00:44.022: INFO: PersistentVolumeClaim pvc-vshfr found but phase is Pending instead of Bound. I1105 03:00:46.101] Nov 5 03:00:46.101: INFO: PersistentVolumeClaim pvc-vshfr found but phase is Pending instead of Bound. I1105 03:00:48.178] Nov 5 03:00:48.178: INFO: PersistentVolumeClaim pvc-vshfr found but phase is Pending instead of Bound. I1105 03:00:50.262] Nov 5 03:00:50.261: INFO: PersistentVolumeClaim pvc-vshfr found but phase is Pending instead of Bound. I1105 03:00:52.340] Nov 5 03:00:52.339: INFO: PersistentVolumeClaim pvc-vshfr found but phase is Pending instead of Bound. I1105 03:00:54.418] Nov 5 03:00:54.417: INFO: PersistentVolumeClaim pvc-vshfr found but phase is Pending instead of Bound. I1105 03:00:56.502] Nov 5 03:00:56.502: INFO: PersistentVolumeClaim pvc-vshfr found but phase is Pending instead of Bound. I1105 03:00:58.579] Nov 5 03:00:58.579: INFO: PersistentVolumeClaim pvc-vshfr found but phase is Pending instead of Bound. I1105 03:01:00.657] Nov 5 03:01:00.656: INFO: PersistentVolumeClaim pvc-vshfr found but phase is Pending instead of Bound. I1105 03:01:02.736] Nov 5 03:01:02.735: INFO: PersistentVolumeClaim pvc-vshfr found but phase is Pending instead of Bound. I1105 03:01:04.813] Nov 5 03:01:04.813: INFO: PersistentVolumeClaim pvc-vshfr found but phase is Pending instead of Bound. I1105 03:01:06.898] Nov 5 03:01:06.894: INFO: PersistentVolumeClaim pvc-vshfr found but phase is Pending instead of Bound. I1105 03:01:08.974] Nov 5 03:01:08.974: INFO: PersistentVolumeClaim pvc-vshfr found but phase is Pending instead of Bound. I1105 03:01:11.052] Nov 5 03:01:11.051: INFO: PersistentVolumeClaim pvc-vshfr found but phase is Pending instead of Bound. I1105 03:01:13.055] Nov 5 03:01:13.053: INFO: Unexpected error occurred: PersistentVolumeClaims [pvc-vshfr] not all in phase Bound within 5m0s I1105 03:01:13.056] [AfterEach] [multi-az] I1105 03:01:13.056] /go/src/sigs.k8s.io/azuredisk-csi-driver/vendor/k8s.io/kubernetes/test/e2e/framework/framework.go:151 I1105 03:01:13.056] STEP: Collecting events from namespace "azuredisk-7632". I1105 03:01:13.141] STEP: Found 3 events. I1105 03:01:13.142] Nov 5 03:01:13.141: INFO: At 2019-11-05 02:56:11 +0000 UTC - event for pvc-vshfr: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "disk.csi.azure.com" or manually created by system administrator I1105 03:01:13.142] Nov 5 03:01:13.141: INFO: At 2019-11-05 02:56:11 +0000 UTC - event for pvc-vshfr: {disk.csi.azure.com_csi-azuredisk-controller-6f6587bfc6-8vkfl_1376914b-ff6d-11e9-93ab-ba2bee589b8c } Provisioning: External provisioner is provisioning volume for claim "azuredisk-7632/pvc-vshfr" I1105 03:01:13.142] Nov 5 03:01:13.141: INFO: At 2019-11-05 02:56:11 +0000 UTC - event for pvc-vshfr: {disk.csi.azure.com_csi-azuredisk-controller-6f6587bfc6-8vkfl_1376914b-ff6d-11e9-93ab-ba2bee589b8c } ProvisioningFailed: failed to provision volume with StorageClass "azuredisk-7632-disk.csi.azure.com-dynamic-sc-4nmdf": rpc error: code = Unknown desc = compute.DisksClient#CreateOrUpdate: Failure sending request: StatusCode=400 -- Original Error: Code="InvalidAvailabilityZone" Message="The zone(s) '' for resource 'Microsoft.Compute/disks/pvc-b8b7ef88-1067-43be-b11f-20e225275c8f' is not supported. The supported zones for location 'eastus2' are '1,2,3'" I1105 03:01:13.218] Nov 5 03:01:13.218: INFO: POD NODE PHASE GRACE CONDITIONS I1105 03:01:13.218] Nov 5 03:01:13.218: INFO: I1105 03:01:13.448] Nov 5 03:01:13.448: INFO: I1105 03:01:13.448] Logging node info for node k8s-agentpool-15826033-vmss000000 I1105 03:01:13.528] Nov 5 03:01:13.525: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-agentpool-15826033-vmss000000,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-agentpool-15826033-vmss000000,UID:15fa874e-54ec-42cd-b243-d3baa512150f,ResourceVersion:16484,Generation:0,CreationTimestamp:2019-11-05 01:34:17 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{agentpool: agentpool,beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_D4s_v3,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 0,kubernetes.azure.com/cluster: kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5,kubernetes.azure.com/role: agent,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-agentpool-15826033-vmss000000,kubernetes.io/os: linux,kubernetes.io/role: agent,node-role.kubernetes.io/agent: ,storageprofile: managed,storagetier: Premium_LRS,topology.disk.csi.azure.com/zone: 0,},Annotations:map[string]string{csi.volume.kubernetes.io/nodeid: {"disk.csi.azure.com":"k8s-agentpool-15826033-vmss000000"},node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.2.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool-15826033-vmss/virtualMachines/0,Unschedulable:false,Taints:[],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-csi-disk.csi.azure.com: {{8 0} {} 8 DecimalSI},cpu: {{4 0} {} 4 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{16797044736 0} {} 16403364Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-csi-disk.csi.azure.com: {{8 0} {} 8 DecimalSI},cpu: {{4 0} {} 4 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{16010612736 0} {} 15635364Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 01:34:56 +0000 UTC 2019-11-05 01:34:56 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 03:00:31 +0000 UTC 2019-11-05 01:34:17 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 03:00:31 +0000 UTC 2019-11-05 01:34:17 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 03:00:31 +0000 UTC 2019-11-05 01:34:17 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 03:00:31 +0000 UTC 2019-11-05 01:34:29 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.0.4} {Hostname k8s-agentpool-15826033-vmss000000}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:afe00792177d49a98f6a44d03d08732d,SystemUUID:4F1777BF-A5C3-004E-81C4-90F0B54C02A0,BootID:13b6a628-ae89-44af-b891-89461030adcb,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[k8sprowinternal.azurecr.io/azuredisk-csi@sha256:75d4ff5d7cd845d6e810c67796ecb037e852fb849ec1eb5b3194564008e6d1e0 k8sprowinternal.azurecr.io/azuredisk-csi:e2e-3e3ed045c6faa509dbd8d2890ee9f56aa2934887] 286930414} {[k8s.gcr.io/kubernetes-dashboard-amd64@sha256:0ae6b69432e78069c5ce2bcde0fe409c5c4d6f0f4d9cd50a17974fea38898747 k8s.gcr.io/kubernetes-dashboard-amd64:v1.10.1] 121711221} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/metrics-server-amd64@sha256:49a9f12f7067d11f42c803dbe61ed2c1299959ad85cb315b25ff7eef8e6b8892 k8s.gcr.io/metrics-server-amd64:v0.2.1] 42541759} {[mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume@sha256:4fd30d43947d4a54fc89ead7985beecfd3c9b2a93a0655a373b1608ab90bd5af mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume:v0.0.7] 22909487} {[quay.io/k8scsi/livenessprobe@sha256:8c9c8a0f330acc2e558ae6624309447d02522e76aa35af23fa19b88abf4381cb quay.io/k8scsi/livenessprobe:v1.0.2] 17415381} {[quay.io/k8scsi/csi-node-driver-registrar@sha256:13daf82fb99e951a4bff8ae5fc7c17c3a8fe7130be6400990d8f6076c32d4599 quay.io/k8scsi/csi-node-driver-registrar:v1.1.0] 15815995} {[mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume@sha256:23d8c6033f02a1ecad05127ebdc931bb871264228661bc122704b0974e4d9fdd mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume:1.0.8] 1159025} {[busybox@sha256:8ccbac733d19c0dd4d70b4f0c1e12245b5fa3ad24758a11035ee505c629c0796 busybox:1.29] 1154361} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 03:01:13.528] Nov 5 03:01:13.525: INFO: I1105 03:01:13.529] Logging kubelet events for node k8s-agentpool-15826033-vmss000000 I1105 03:01:13.603] Nov 5 03:01:13.603: INFO: I1105 03:01:13.603] Logging pods the kubelet thinks is on node k8s-agentpool-15826033-vmss000000 I1105 03:01:13.685] Nov 5 03:01:13.685: INFO: azure-ip-masq-agent-vx86b started at 2019-11-05 01:34:34 +0000 UTC (0+1 container statuses recorded) I1105 03:01:13.685] Nov 5 03:01:13.685: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 03:01:13.686] Nov 5 03:01:13.685: INFO: keyvault-flexvolume-qhnfr started at 2019-11-05 01:34:34 +0000 UTC (0+1 container statuses recorded) I1105 03:01:13.686] Nov 5 03:01:13.685: INFO: Container keyvault-flexvolume ready: true, restart count 0 I1105 03:01:13.686] Nov 5 03:01:13.685: INFO: metrics-server-864ffbc5c-sdxps started at 2019-11-05 01:34:34 +0000 UTC (0+1 container statuses recorded) I1105 03:01:13.686] Nov 5 03:01:13.685: INFO: Container metrics-server ready: true, restart count 0 I1105 03:01:13.686] Nov 5 03:01:13.685: INFO: kube-proxy-wxblc started at 2019-11-05 01:34:34 +0000 UTC (0+1 container statuses recorded) I1105 03:01:13.687] Nov 5 03:01:13.685: INFO: Container kube-proxy ready: true, restart count 0 I1105 03:01:13.687] Nov 5 03:01:13.685: INFO: kubernetes-dashboard-66dd8b8df7-ks284 started at 2019-11-05 01:34:35 +0000 UTC (0+1 container statuses recorded) I1105 03:01:13.687] Nov 5 03:01:13.685: INFO: Container kubernetes-dashboard ready: true, restart count 0 I1105 03:01:13.687] Nov 5 03:01:13.685: INFO: csi-azuredisk-node-whq2g started at 2019-11-05 01:38:24 +0000 UTC (0+3 container statuses recorded) I1105 03:01:13.687] Nov 5 03:01:13.685: INFO: Container azuredisk ready: true, restart count 0 I1105 03:01:13.688] Nov 5 03:01:13.685: INFO: Container liveness-probe ready: true, restart count 0 I1105 03:01:13.688] Nov 5 03:01:13.685: INFO: Container node-driver-registrar ready: true, restart count 0 I1105 03:01:13.688] Nov 5 03:01:13.685: INFO: blobfuse-flexvol-installer-bsvqt started at 2019-11-05 01:34:34 +0000 UTC (0+1 container statuses recorded) I1105 03:01:13.688] Nov 5 03:01:13.685: INFO: Container blobfuse-flexvol-installer ready: true, restart count 0 I1105 03:01:13.941] Nov 5 03:01:13.940: INFO: I1105 03:01:13.941] Latency metrics for node k8s-agentpool-15826033-vmss000000 I1105 03:01:13.941] Nov 5 03:01:13.941: INFO: I1105 03:01:13.941] Logging node info for node k8s-agentpool-15826033-vmss000001 I1105 03:01:14.021] Nov 5 03:01:14.018: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-agentpool-15826033-vmss000001,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-agentpool-15826033-vmss000001,UID:4a8ea605-cc0b-4e3b-9060-d8867d269e6c,ResourceVersion:16554,Generation:0,CreationTimestamp:2019-11-05 01:34:09 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{agentpool: agentpool,beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_D4s_v3,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 1,kubernetes.azure.com/cluster: kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5,kubernetes.azure.com/role: agent,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-agentpool-15826033-vmss000001,kubernetes.io/os: linux,kubernetes.io/role: agent,node-role.kubernetes.io/agent: ,storageprofile: managed,storagetier: Premium_LRS,topology.disk.csi.azure.com/zone: 1,},Annotations:map[string]string{csi.volume.kubernetes.io/nodeid: {"disk.csi.azure.com":"k8s-agentpool-15826033-vmss000001"},node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.0.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool-15826033-vmss/virtualMachines/1,Unschedulable:false,Taints:[],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-csi-disk.csi.azure.com: {{8 0} {} 8 DecimalSI},cpu: {{4 0} {} 4 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{16797044736 0} {} 16403364Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-csi-disk.csi.azure.com: {{8 0} {} 8 DecimalSI},cpu: {{4 0} {} 4 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{16010612736 0} {} 15635364Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 01:34:57 +0000 UTC 2019-11-05 01:34:57 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 03:00:53 +0000 UTC 2019-11-05 01:34:08 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 03:00:53 +0000 UTC 2019-11-05 01:34:08 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 03:00:53 +0000 UTC 2019-11-05 01:34:08 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 03:00:53 +0000 UTC 2019-11-05 01:34:29 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.0.5} {Hostname k8s-agentpool-15826033-vmss000001}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:b6b9fbf4a2fd4fa5a7ae472415fe30b8,SystemUUID:0162B3A4-3955-9A46-ABD4-A37BD75848CC,BootID:ee93003f-d47c-4118-9975-57365b3443a5,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[k8sprowinternal.azurecr.io/azuredisk-csi@sha256:75d4ff5d7cd845d6e810c67796ecb037e852fb849ec1eb5b3194564008e6d1e0 k8sprowinternal.azurecr.io/azuredisk-csi:e2e-3e3ed045c6faa509dbd8d2890ee9f56aa2934887] 286930414} {[gcr.io/kubernetes-helm/tiller@sha256:f6d8f4ab9ba993b5f5b60a6edafe86352eabe474ffeb84cb6c79b8866dce45d1 gcr.io/kubernetes-helm/tiller:v2.11.0] 71821984} {[quay.io/k8scsi/csi-provisioner@sha256:d657c839dce87324fe2b677302913f9386f885f8746be7bea0ced5b0844e3433 quay.io/k8scsi/csi-provisioner:v1.3.1] 50596998} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[quay.io/k8scsi/csi-attacher@sha256:26fccd7a99d973845df1193b46ebdcc6ab8dc5f6e6be319750c471fce1742d13 quay.io/k8scsi/csi-attacher:v1.2.0] 46226754} {[quay.io/k8scsi/csi-cluster-driver-registrar@sha256:fafd75ae5442f192cfa8c2e792903aee30d5884b62e802e4464b0a895d21e3ef quay.io/k8scsi/csi-cluster-driver-registrar:v1.0.1] 45874691} {[quay.io/k8scsi/csi-snapshotter@sha256:a49e0da1af6f2bf717e41ba1eee8b5e6a1cbd66a709dd92cc43fe475fe2589eb quay.io/k8scsi/csi-snapshotter:v1.1.0] 44325352} {[k8s.gcr.io/coredns@sha256:e83beb5e43f8513fa735e77ffc5859640baea30a882a11cc75c4c3244a737d3c k8s.gcr.io/coredns:1.5.0] 42488424} {[mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume@sha256:4fd30d43947d4a54fc89ead7985beecfd3c9b2a93a0655a373b1608ab90bd5af mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume:v0.0.7] 22909487} {[quay.io/k8scsi/livenessprobe@sha256:8c9c8a0f330acc2e558ae6624309447d02522e76aa35af23fa19b88abf4381cb quay.io/k8scsi/livenessprobe:v1.0.2] 17415381} {[quay.io/k8scsi/csi-node-driver-registrar@sha256:13daf82fb99e951a4bff8ae5fc7c17c3a8fe7130be6400990d8f6076c32d4599 quay.io/k8scsi/csi-node-driver-registrar:v1.1.0] 15815995} {[quay.io/k8scsi/livenessprobe@sha256:dde617756e0f602adc566ab71fd885f1dad451ad3fb063ac991c95a2ff47aea5 quay.io/k8scsi/livenessprobe:v1.1.0] 14967303} {[mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume@sha256:23d8c6033f02a1ecad05127ebdc931bb871264228661bc122704b0974e4d9fdd mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume:1.0.8] 1159025} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 03:01:14.022] Nov 5 03:01:14.018: INFO: I1105 03:01:14.022] Logging kubelet events for node k8s-agentpool-15826033-vmss000001 I1105 03:01:14.100] Nov 5 03:01:14.100: INFO: I1105 03:01:14.101] Logging pods the kubelet thinks is on node k8s-agentpool-15826033-vmss000001 I1105 03:01:14.184] Nov 5 03:01:14.183: INFO: kube-proxy-6kh57 started at 2019-11-05 01:34:35 +0000 UTC (0+1 container statuses recorded) I1105 03:01:14.184] Nov 5 03:01:14.183: INFO: Container kube-proxy ready: true, restart count 0 I1105 03:01:14.184] Nov 5 03:01:14.183: INFO: tiller-deploy-54c96cb5df-8d9fs started at 2019-11-05 01:34:35 +0000 UTC (0+1 container statuses recorded) I1105 03:01:14.184] Nov 5 03:01:14.183: INFO: Container tiller ready: true, restart count 0 I1105 03:01:14.185] Nov 5 03:01:14.183: INFO: csi-azuredisk-node-ghv5w started at 2019-11-05 01:38:24 +0000 UTC (0+3 container statuses recorded) I1105 03:01:14.185] Nov 5 03:01:14.183: INFO: Container azuredisk ready: true, restart count 0 I1105 03:01:14.185] Nov 5 03:01:14.183: INFO: Container liveness-probe ready: true, restart count 0 I1105 03:01:14.185] Nov 5 03:01:14.183: INFO: Container node-driver-registrar ready: true, restart count 0 I1105 03:01:14.186] Nov 5 03:01:14.183: INFO: csi-azuredisk-controller-6f6587bfc6-8vkfl started at 2019-11-05 01:38:24 +0000 UTC (0+6 container statuses recorded) I1105 03:01:14.186] Nov 5 03:01:14.183: INFO: Container azuredisk ready: true, restart count 0 I1105 03:01:14.186] Nov 5 03:01:14.183: INFO: Container cluster-driver-registrar ready: true, restart count 0 I1105 03:01:14.186] Nov 5 03:01:14.183: INFO: Container csi-attacher ready: true, restart count 0 I1105 03:01:14.186] Nov 5 03:01:14.184: INFO: Container csi-provisioner ready: true, restart count 0 I1105 03:01:14.186] Nov 5 03:01:14.184: INFO: Container csi-snapshotter ready: true, restart count 0 I1105 03:01:14.187] Nov 5 03:01:14.184: INFO: Container liveness-probe ready: true, restart count 0 I1105 03:01:14.187] Nov 5 03:01:14.184: INFO: blobfuse-flexvol-installer-rw5dw started at 2019-11-05 01:34:35 +0000 UTC (0+1 container statuses recorded) I1105 03:01:14.187] Nov 5 03:01:14.184: INFO: Container blobfuse-flexvol-installer ready: true, restart count 0 I1105 03:01:14.187] Nov 5 03:01:14.184: INFO: coredns-7f68dcdbdb-jt522 started at 2019-11-05 01:34:35 +0000 UTC (0+1 container statuses recorded) I1105 03:01:14.188] Nov 5 03:01:14.184: INFO: Container coredns ready: true, restart count 0 I1105 03:01:14.188] Nov 5 03:01:14.184: INFO: azure-ip-masq-agent-6c6vn started at 2019-11-05 01:34:35 +0000 UTC (0+1 container statuses recorded) I1105 03:01:14.188] Nov 5 03:01:14.184: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 03:01:14.188] Nov 5 03:01:14.184: INFO: keyvault-flexvolume-rdlcj started at 2019-11-05 01:34:35 +0000 UTC (0+1 container statuses recorded) I1105 03:01:14.188] Nov 5 03:01:14.184: INFO: Container keyvault-flexvolume ready: true, restart count 0 I1105 03:01:14.447] Nov 5 03:01:14.446: INFO: I1105 03:01:14.447] Latency metrics for node k8s-agentpool-15826033-vmss000001 I1105 03:01:14.447] Nov 5 03:01:14.446: INFO: I1105 03:01:14.447] Logging node info for node k8s-master-15826033-0 I1105 03:01:14.526] Nov 5 03:01:14.524: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-15826033-0,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-15826033-0,UID:aceb0231-4645-4a9a-b296-90ada851986b,ResourceVersion:16506,Generation:0,CreationTimestamp:2019-11-05 01:35:06 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 1,kubernetes.azure.com/cluster: kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-15826033-0,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.4.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5/providers/Microsoft.Compute/virtualMachines/k8s-master-15826033-0,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 01:35:20 +0000 UTC 2019-11-05 01:35:20 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 03:00:38 +0000 UTC 2019-11-05 01:34:57 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 03:00:38 +0000 UTC 2019-11-05 01:34:57 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 03:00:38 +0000 UTC 2019-11-05 01:34:57 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 03:00:38 +0000 UTC 2019-11-05 01:35:07 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.255.5} {Hostname k8s-master-15826033-0}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:1915756c22a540dc8eb30493987e9a43,SystemUUID:742C969C-5A71-D744-8E61-F27E38B56EB2,BootID:03f0672e-8fa9-4309-956f-8e3c371902c3,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 03:01:14.526] Nov 5 03:01:14.524: INFO: I1105 03:01:14.527] Logging kubelet events for node k8s-master-15826033-0 I1105 03:01:14.608] Nov 5 03:01:14.608: INFO: I1105 03:01:14.608] Logging pods the kubelet thinks is on node k8s-master-15826033-0 I1105 03:01:14.689] Nov 5 03:01:14.689: INFO: cloud-controller-manager-k8s-master-15826033-0 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 03:01:14.690] Nov 5 03:01:14.689: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 03:01:14.690] Nov 5 03:01:14.689: INFO: kube-addon-manager-k8s-master-15826033-0 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 03:01:14.690] Nov 5 03:01:14.689: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 03:01:14.690] Nov 5 03:01:14.689: INFO: kube-apiserver-k8s-master-15826033-0 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 03:01:14.690] Nov 5 03:01:14.689: INFO: Container kube-apiserver ready: true, restart count 0 I1105 03:01:14.690] Nov 5 03:01:14.689: INFO: kube-controller-manager-k8s-master-15826033-0 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 03:01:14.690] Nov 5 03:01:14.689: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 03:01:14.690] Nov 5 03:01:14.689: INFO: kube-scheduler-k8s-master-15826033-0 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 03:01:14.691] Nov 5 03:01:14.689: INFO: Container kube-scheduler ready: true, restart count 0 I1105 03:01:14.691] Nov 5 03:01:14.689: INFO: azure-ip-masq-agent-wgdpx started at 2019-11-05 01:35:06 +0000 UTC (0+1 container statuses recorded) I1105 03:01:14.691] Nov 5 03:01:14.689: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 03:01:14.691] Nov 5 03:01:14.689: INFO: kube-proxy-7d2px started at 2019-11-05 01:35:06 +0000 UTC (0+1 container statuses recorded) I1105 03:01:14.691] Nov 5 03:01:14.689: INFO: Container kube-proxy ready: true, restart count 0 I1105 03:01:17.086] Nov 5 03:01:17.086: INFO: I1105 03:01:17.086] Latency metrics for node k8s-master-15826033-0 I1105 03:01:17.087] Nov 5 03:01:17.086: INFO: I1105 03:01:17.087] Logging node info for node k8s-master-15826033-1 I1105 03:01:17.173] Nov 5 03:01:17.166: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-15826033-1,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-15826033-1,UID:1a13ec61-c89a-456c-8c94-bfc8e717c2e7,ResourceVersion:16507,Generation:0,CreationTimestamp:2019-11-05 01:35:07 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 1,kubernetes.azure.com/cluster: kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-15826033-1,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.5.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5/providers/Microsoft.Compute/virtualMachines/k8s-master-15826033-1,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 01:35:31 +0000 UTC 2019-11-05 01:35:31 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 03:00:38 +0000 UTC 2019-11-05 01:35:00 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 03:00:38 +0000 UTC 2019-11-05 01:35:00 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 03:00:38 +0000 UTC 2019-11-05 01:35:00 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 03:00:38 +0000 UTC 2019-11-05 01:35:07 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.255.6} {Hostname k8s-master-15826033-1}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:e0f2bccddd5b4555b180d14081ab59a8,SystemUUID:AE126631-CAD2-B54F-B22E-1F499342A4D6,BootID:509b9a9d-b046-4861-aa35-8bcb77465de2,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 03:01:17.173] Nov 5 03:01:17.166: INFO: I1105 03:01:17.173] Logging kubelet events for node k8s-master-15826033-1 I1105 03:01:17.246] Nov 5 03:01:17.246: INFO: I1105 03:01:17.246] Logging pods the kubelet thinks is on node k8s-master-15826033-1 I1105 03:01:17.328] Nov 5 03:01:17.327: INFO: cloud-controller-manager-k8s-master-15826033-1 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 03:01:17.328] Nov 5 03:01:17.327: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 03:01:17.328] Nov 5 03:01:17.327: INFO: kube-addon-manager-k8s-master-15826033-1 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 03:01:17.329] Nov 5 03:01:17.327: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 03:01:17.329] Nov 5 03:01:17.327: INFO: kube-apiserver-k8s-master-15826033-1 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 03:01:17.329] Nov 5 03:01:17.327: INFO: Container kube-apiserver ready: true, restart count 0 I1105 03:01:17.329] Nov 5 03:01:17.327: INFO: kube-controller-manager-k8s-master-15826033-1 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 03:01:17.330] Nov 5 03:01:17.327: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 03:01:17.330] Nov 5 03:01:17.327: INFO: kube-scheduler-k8s-master-15826033-1 started at 2019-11-05 01:36:16 +0000 UTC (0+1 container statuses recorded) I1105 03:01:17.330] Nov 5 03:01:17.327: INFO: Container kube-scheduler ready: true, restart count 0 I1105 03:01:17.330] Nov 5 03:01:17.327: INFO: azure-ip-masq-agent-w6dpr started at 2019-11-05 01:35:10 +0000 UTC (0+1 container statuses recorded) I1105 03:01:17.331] Nov 5 03:01:17.327: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 03:01:17.331] Nov 5 03:01:17.327: INFO: kube-proxy-8kmr5 started at 2019-11-05 01:35:10 +0000 UTC (0+1 container statuses recorded) I1105 03:01:17.331] Nov 5 03:01:17.327: INFO: Container kube-proxy ready: true, restart count 0 I1105 03:01:17.581] Nov 5 03:01:17.581: INFO: I1105 03:01:18.629] Latency metrics for node k8s-master-15826033-1 I1105 03:01:18.630] Nov 5 03:01:17.581: INFO: I1105 03:01:18.630] Logging node info for node k8s-master-15826033-2 I1105 03:01:18.634] Nov 5 03:01:17.659: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-15826033-2,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-15826033-2,UID:acb3a851-d6df-4549-ba56-4b7e96553c17,ResourceVersion:16599,Generation:0,CreationTimestamp:2019-11-05 01:34:09 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 2,kubernetes.azure.com/cluster: kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-15826033-2,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.1.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5/providers/Microsoft.Compute/virtualMachines/k8s-master-15826033-2,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 01:34:44 +0000 UTC 2019-11-05 01:34:44 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 03:01:09 +0000 UTC 2019-11-05 01:34:07 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 03:01:09 +0000 UTC 2019-11-05 01:34:07 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 03:01:09 +0000 UTC 2019-11-05 01:34:07 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 03:01:09 +0000 UTC 2019-11-05 01:34:29 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{Hostname k8s-master-15826033-2} {InternalIP 10.240.255.7}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:0885b0c2f6174106bde4fa0eae2cbc6b,SystemUUID:C176121C-5D7A-6243-9B05-8B68F8022B64,BootID:30430638-7ab6-4761-b984-2387e1a3f5b2,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 03:01:18.634] Nov 5 03:01:17.659: INFO: I1105 03:01:18.634] Logging kubelet events for node k8s-master-15826033-2 I1105 03:01:18.634] Nov 5 03:01:17.736: INFO: I1105 03:01:18.634] Logging pods the kubelet thinks is on node k8s-master-15826033-2 I1105 03:01:18.635] Nov 5 03:01:17.818: INFO: cloud-controller-manager-k8s-master-15826033-2 started at 2019-11-05 01:33:54 +0000 UTC (0+1 container statuses recorded) I1105 03:01:18.635] Nov 5 03:01:17.818: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 03:01:18.635] Nov 5 03:01:17.818: INFO: kube-addon-manager-k8s-master-15826033-2 started at 2019-11-05 01:33:54 +0000 UTC (0+1 container statuses recorded) I1105 03:01:18.635] Nov 5 03:01:17.818: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 03:01:18.636] Nov 5 03:01:17.818: INFO: kube-apiserver-k8s-master-15826033-2 started at 2019-11-05 01:33:54 +0000 UTC (0+1 container statuses recorded) I1105 03:01:18.636] Nov 5 03:01:17.818: INFO: Container kube-apiserver ready: true, restart count 0 I1105 03:01:18.636] Nov 5 03:01:17.818: INFO: kube-controller-manager-k8s-master-15826033-2 started at 2019-11-05 01:33:54 +0000 UTC (0+1 container statuses recorded) I1105 03:01:18.636] Nov 5 03:01:17.818: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 03:01:18.637] Nov 5 03:01:17.818: INFO: kube-scheduler-k8s-master-15826033-2 started at 2019-11-05 01:33:54 +0000 UTC (0+1 container statuses recorded) I1105 03:01:18.637] Nov 5 03:01:17.818: INFO: Container kube-scheduler ready: true, restart count 0 I1105 03:01:18.637] Nov 5 03:01:17.818: INFO: azure-ip-masq-agent-hw7s5 started at 2019-11-05 01:34:34 +0000 UTC (0+1 container statuses recorded) I1105 03:01:18.637] Nov 5 03:01:17.818: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 03:01:18.638] Nov 5 03:01:17.818: INFO: kube-proxy-5xpp6 started at 2019-11-05 01:34:34 +0000 UTC (0+1 container statuses recorded) I1105 03:01:18.638] Nov 5 03:01:17.818: INFO: Container kube-proxy ready: true, restart count 0 I1105 03:01:18.638] Nov 5 03:01:18.069: INFO: I1105 03:01:18.638] Latency metrics for node k8s-master-15826033-2 I1105 03:01:18.638] Nov 5 03:01:18.069: INFO: I1105 03:01:18.639] Logging node info for node k8s-master-15826033-3 I1105 03:01:18.642] Nov 5 03:01:18.146: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-15826033-3,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-15826033-3,UID:b7702272-1876-434e-b6a0-17fa34c6757f,ResourceVersion:16501,Generation:0,CreationTimestamp:2019-11-05 01:35:06 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 0,kubernetes.azure.com/cluster: kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-15826033-3,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.3.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5/providers/Microsoft.Compute/virtualMachines/k8s-master-15826033-3,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 01:35:17 +0000 UTC 2019-11-05 01:35:17 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 03:00:37 +0000 UTC 2019-11-05 01:35:00 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 03:00:37 +0000 UTC 2019-11-05 01:35:00 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 03:00:37 +0000 UTC 2019-11-05 01:35:00 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 03:00:37 +0000 UTC 2019-11-05 01:35:06 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.255.8} {Hostname k8s-master-15826033-3}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:fbf366963961422c9d90bedf6639c8e1,SystemUUID:5388948D-1A8D-EA41-B8B8-7F4523EDF5AA,BootID:42f5884c-7418-486a-8cf3-75188a331f38,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 03:01:18.642] Nov 5 03:01:18.146: INFO: I1105 03:01:18.642] Logging kubelet events for node k8s-master-15826033-3 I1105 03:01:18.642] Nov 5 03:01:18.223: INFO: I1105 03:01:18.643] Logging pods the kubelet thinks is on node k8s-master-15826033-3 I1105 03:01:18.643] Nov 5 03:01:18.306: INFO: azure-ip-masq-agent-dqfpw started at 2019-11-05 01:35:06 +0000 UTC (0+1 container statuses recorded) I1105 03:01:18.643] Nov 5 03:01:18.306: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 03:01:18.643] Nov 5 03:01:18.306: INFO: cloud-controller-manager-k8s-master-15826033-3 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 03:01:18.644] Nov 5 03:01:18.306: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 03:01:18.644] Nov 5 03:01:18.306: INFO: kube-addon-manager-k8s-master-15826033-3 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 03:01:18.644] Nov 5 03:01:18.306: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 03:01:18.644] Nov 5 03:01:18.306: INFO: kube-apiserver-k8s-master-15826033-3 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 03:01:18.645] Nov 5 03:01:18.306: INFO: Container kube-apiserver ready: true, restart count 0 I1105 03:01:18.645] Nov 5 03:01:18.306: INFO: kube-controller-manager-k8s-master-15826033-3 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 03:01:18.645] Nov 5 03:01:18.306: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 03:01:18.645] Nov 5 03:01:18.306: INFO: kube-scheduler-k8s-master-15826033-3 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 03:01:18.646] Nov 5 03:01:18.306: INFO: Container kube-scheduler ready: true, restart count 0 I1105 03:01:18.646] Nov 5 03:01:18.306: INFO: kube-proxy-xz24z started at 2019-11-05 01:35:06 +0000 UTC (0+1 container statuses recorded) I1105 03:01:18.646] Nov 5 03:01:18.306: INFO: Container kube-proxy ready: true, restart count 0 I1105 03:01:18.646] Nov 5 03:01:18.556: INFO: I1105 03:01:18.647] Latency metrics for node k8s-master-15826033-3 I1105 03:01:18.647] Nov 5 03:01:18.556: INFO: I1105 03:01:18.647] Logging node info for node k8s-master-15826033-4 I1105 03:01:18.650] Nov 5 03:01:18.634: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-15826033-4,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-15826033-4,UID:276daef1-6c05-41e3-bacd-abc93d35cc6d,ResourceVersion:16510,Generation:0,CreationTimestamp:2019-11-05 01:35:07 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 0,kubernetes.azure.com/cluster: kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-15826033-4,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.6.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5/providers/Microsoft.Compute/virtualMachines/k8s-master-15826033-4,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 01:35:41 +0000 UTC 2019-11-05 01:35:41 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 03:00:38 +0000 UTC 2019-11-05 01:35:02 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 03:00:38 +0000 UTC 2019-11-05 01:35:02 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 03:00:38 +0000 UTC 2019-11-05 01:35:02 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 03:00:38 +0000 UTC 2019-11-05 01:35:08 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.255.9} {Hostname k8s-master-15826033-4}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:79051fc0bdeb4e32a56acbaf5972ce3a,SystemUUID:C71606B5-7F08-9341-A26D-8A8D9D84FCAE,BootID:4bdf7cf3-2b83-42c7-a7ba-926ebe9ca75b,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 03:01:18.650] Nov 5 03:01:18.634: INFO: I1105 03:01:18.651] Logging kubelet events for node k8s-master-15826033-4 I1105 03:01:18.711] Nov 5 03:01:18.711: INFO: I1105 03:01:18.712] Logging pods the kubelet thinks is on node k8s-master-15826033-4 I1105 03:01:18.794] Nov 5 03:01:18.793: INFO: kube-apiserver-k8s-master-15826033-4 started at 2019-11-05 01:34:55 +0000 UTC (0+1 container statuses recorded) I1105 03:01:18.794] Nov 5 03:01:18.793: INFO: Container kube-apiserver ready: true, restart count 0 I1105 03:01:18.794] Nov 5 03:01:18.793: INFO: kube-controller-manager-k8s-master-15826033-4 started at 2019-11-05 01:34:55 +0000 UTC (0+1 container statuses recorded) I1105 03:01:18.795] Nov 5 03:01:18.793: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 03:01:18.795] Nov 5 03:01:18.793: INFO: kube-scheduler-k8s-master-15826033-4 started at 2019-11-05 01:34:55 +0000 UTC (0+1 container statuses recorded) I1105 03:01:18.795] Nov 5 03:01:18.793: INFO: Container kube-scheduler ready: true, restart count 0 I1105 03:01:18.795] Nov 5 03:01:18.793: INFO: cloud-controller-manager-k8s-master-15826033-4 started at 2019-11-05 01:34:56 +0000 UTC (0+1 container statuses recorded) I1105 03:01:18.795] Nov 5 03:01:18.793: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 03:01:18.796] Nov 5 03:01:18.793: INFO: kube-addon-manager-k8s-master-15826033-4 started at 2019-11-05 01:34:56 +0000 UTC (0+1 container statuses recorded) I1105 03:01:18.796] Nov 5 03:01:18.793: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 03:01:18.796] Nov 5 03:01:18.793: INFO: azure-ip-masq-agent-9nrj8 started at 2019-11-05 01:35:11 +0000 UTC (0+1 container statuses recorded) I1105 03:01:18.796] Nov 5 03:01:18.793: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 03:01:18.797] Nov 5 03:01:18.793: INFO: kube-proxy-rmngk started at 2019-11-05 01:35:11 +0000 UTC (0+1 container statuses recorded) I1105 03:01:18.797] Nov 5 03:01:18.793: INFO: Container kube-proxy ready: true, restart count 0 I1105 03:01:19.046] Nov 5 03:01:19.043: INFO: I1105 03:01:19.046] Latency metrics for node k8s-master-15826033-4 I1105 03:01:19.046] Nov 5 03:01:19.043: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready I1105 03:01:19.125] STEP: Destroying namespace "azuredisk-7632" for this suite. I1105 03:01:33.375] Nov 5 03:01:33.375: INFO: Waiting up to 30s for server preferred namespaced resources to be successfully discovered I1105 03:01:36.299] Nov 5 03:01:36.299: INFO: namespace azuredisk-7632 deletion completed in 17.174158191s I1105 03:01:36.790] I1105 03:01:36.792] • Failure [325.538 seconds] I1105 03:01:36.792] [azuredisk-csi-e2e] Dynamic Provisioning I1105 03:01:36.792] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:33 I1105 03:01:36.792] [multi-az] I1105 03:01:36.792] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:40 I1105 03:01:36.792] should create a raw block volume on demand [It] I1105 03:01:36.793] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:101 I1105 03:01:36.793] I1105 03:01:36.793] Unexpected error: I1105 03:01:36.793] <*errors.errorString | 0xc0008ffbe0>: { I1105 03:01:36.793] s: "PersistentVolumeClaims [pvc-vshfr] not all in phase Bound within 5m0s", I1105 03:01:36.794] } I1105 03:01:36.794] PersistentVolumeClaims [pvc-vshfr] not all in phase Bound within 5m0s I1105 03:01:36.794] occurred I1105 03:01:36.794] I1105 03:01:36.794] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/testsuites/testsuites.go:278 I1105 03:01:36.794] ------------------------------ I1105 03:01:36.794] [azuredisk-csi-e2e] Dynamic Provisioning [multi-az] I1105 03:01:36.795] should delete PV with reclaimPolicy "Delete" I1105 03:01:36.795] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:221 I1105 03:01:36.795] [BeforeEach] [multi-az] I1105 03:01:36.795] /go/src/sigs.k8s.io/azuredisk-csi-driver/vendor/k8s.io/kubernetes/test/e2e/framework/framework.go:150 I1105 03:01:36.795] STEP: Creating a kubernetes client I1105 03:01:36.795] Nov 5 03:01:36.299: INFO: >>> kubeConfig: /workspace/aks978705698/kubeconfig/kubeconfig.eastus2.json I1105 03:01:36.796] STEP: Building a namespace api object, basename azuredisk I1105 03:01:36.796] STEP: Binding the e2e-test-privileged-psp PodSecurityPolicy to the default service account in azuredisk-6855 I1105 03:01:36.809] STEP: Waiting for a default service account to be provisioned in namespace I1105 03:01:36.882] [BeforeEach] [multi-az] I1105 03:01:36.882] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:58 I1105 03:01:36.882] [It] should delete PV with reclaimPolicy "Delete" I1105 03:01:36.882] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:221 I1105 03:01:36.883] STEP: setting up the StorageClass I1105 03:01:36.883] STEP: creating a StorageClass I1105 03:01:36.963] STEP: setting up the PVC and PV I1105 03:01:36.964] STEP: creating a PVC I1105 03:01:37.052] STEP: waiting for PVC to be in phase "Bound" I1105 03:01:37.052] Nov 5 03:01:37.050: INFO: Waiting up to 5m0s for PersistentVolumeClaims [pvc-gtw85] to have phase Bound I1105 03:01:37.158] Nov 5 03:01:37.150: INFO: PersistentVolumeClaim pvc-gtw85 found but phase is Pending instead of Bound. I1105 03:01:39.234] Nov 5 03:01:39.233: INFO: PersistentVolumeClaim pvc-gtw85 found but phase is Pending instead of Bound. I1105 03:01:41.312] Nov 5 03:01:41.312: INFO: PersistentVolumeClaim pvc-gtw85 found but phase is Pending instead of Bound. I1105 03:01:43.390] Nov 5 03:01:43.390: INFO: PersistentVolumeClaim pvc-gtw85 found but phase is Pending instead of Bound. I1105 03:01:45.467] Nov 5 03:01:45.467: INFO: PersistentVolumeClaim pvc-gtw85 found but phase is Pending instead of Bound. I1105 03:01:47.545] Nov 5 03:01:47.544: INFO: PersistentVolumeClaim pvc-gtw85 found but phase is Pending instead of Bound. I1105 03:01:49.627] Nov 5 03:01:49.627: INFO: PersistentVolumeClaim pvc-gtw85 found but phase is Pending instead of Bound. I1105 03:01:51.713] Nov 5 03:01:51.710: INFO: PersistentVolumeClaim pvc-gtw85 found but phase is Pending instead of Bound. I1105 03:01:53.787] Nov 5 03:01:53.787: INFO: PersistentVolumeClaim pvc-gtw85 found but phase is Pending instead of Bound. I1105 03:01:55.864] Nov 5 03:01:55.864: INFO: PersistentVolumeClaim pvc-gtw85 found but phase is Pending instead of Bound. I1105 03:01:57.949] Nov 5 03:01:57.948: INFO: PersistentVolumeClaim pvc-gtw85 found but phase is Pending instead of Bound. I1105 03:02:00.027] Nov 5 03:02:00.026: INFO: PersistentVolumeClaim pvc-gtw85 found but phase is Pending instead of Bound. I1105 03:02:02.104] Nov 5 03:02:02.104: INFO: PersistentVolumeClaim pvc-gtw85 found but phase is Pending instead of Bound. I1105 03:02:04.182] Nov 5 03:02:04.181: INFO: PersistentVolumeClaim pvc-gtw85 found but phase is Pending instead of Bound. I1105 03:02:06.259] Nov 5 03:02:06.259: INFO: PersistentVolumeClaim pvc-gtw85 found but phase is Pending instead of Bound. I1105 03:02:08.339] Nov 5 03:02:08.339: INFO: PersistentVolumeClaim pvc-gtw85 found but phase is Pending instead of Bound. I1105 03:02:10.417] Nov 5 03:02:10.416: INFO: PersistentVolumeClaim pvc-gtw85 found but phase is Pending instead of Bound. I1105 03:02:12.502] Nov 5 03:02:12.498: INFO: PersistentVolumeClaim pvc-gtw85 found but phase is Pending instead of Bound. I1105 03:02:14.576] Nov 5 03:02:14.575: INFO: PersistentVolumeClaim pvc-gtw85 found but phase is Pending instead of Bound. I1105 03:02:16.654] Nov 5 03:02:16.654: INFO: PersistentVolumeClaim pvc-gtw85 found but phase is Pending instead of Bound. I1105 03:02:18.732] Nov 5 03:02:18.731: INFO: PersistentVolumeClaim pvc-gtw85 found but phase is Pending instead of Bound. I1105 03:02:20.904] Nov 5 03:02:20.809: INFO: PersistentVolumeClaim pvc-gtw85 found but phase is Pending instead of Bound. I1105 03:02:22.887] Nov 5 03:02:22.887: INFO: PersistentVolumeClaim pvc-gtw85 found but phase is Pending instead of Bound. I1105 03:02:24.965] Nov 5 03:02:24.965: INFO: PersistentVolumeClaim pvc-gtw85 found but phase is Pending instead of Bound. I1105 03:02:27.042] Nov 5 03:02:27.042: INFO: PersistentVolumeClaim pvc-gtw85 found but phase is Pending instead of Bound. I1105 03:02:29.134] Nov 5 03:02:29.134: INFO: PersistentVolumeClaim pvc-gtw85 found but phase is Pending instead of Bound. I1105 03:02:31.212] Nov 5 03:02:31.212: INFO: PersistentVolumeClaim pvc-gtw85 found but phase is Pending instead of Bound. I1105 03:02:33.289] Nov 5 03:02:33.289: INFO: PersistentVolumeClaim pvc-gtw85 found but phase is Pending instead of Bound. I1105 03:02:35.367] Nov 5 03:02:35.367: INFO: PersistentVolumeClaim pvc-gtw85 found but phase is Pending instead of Bound. I1105 03:02:37.445] Nov 5 03:02:37.445: INFO: PersistentVolumeClaim pvc-gtw85 found but phase is Pending instead of Bound. I1105 03:02:39.522] Nov 5 03:02:39.522: INFO: PersistentVolumeClaim pvc-gtw85 found but phase is Pending instead of Bound. I1105 03:02:41.600] Nov 5 03:02:41.600: INFO: PersistentVolumeClaim pvc-gtw85 found but phase is Pending instead of Bound. I1105 03:02:43.692] Nov 5 03:02:43.691: INFO: PersistentVolumeClaim pvc-gtw85 found but phase is Pending instead of Bound. I1105 03:02:45.769] Nov 5 03:02:45.769: INFO: PersistentVolumeClaim pvc-gtw85 found but phase is Pending instead of Bound. I1105 03:02:47.847] Nov 5 03:02:47.846: INFO: PersistentVolumeClaim pvc-gtw85 found but phase is Pending instead of Bound. I1105 03:02:49.924] Nov 5 03:02:49.924: INFO: PersistentVolumeClaim pvc-gtw85 found but phase is Pending instead of Bound. I1105 03:02:52.009] Nov 5 03:02:52.009: INFO: PersistentVolumeClaim pvc-gtw85 found but phase is Pending instead of Bound. I1105 03:02:54.086] Nov 5 03:02:54.086: INFO: PersistentVolumeClaim pvc-gtw85 found but phase is Pending instead of Bound. I1105 03:02:56.769] Nov 5 03:02:56.164: INFO: PersistentVolumeClaim pvc-gtw85 found but phase is Pending instead of Bound. I1105 03:02:58.241] Nov 5 03:02:58.241: INFO: PersistentVolumeClaim pvc-gtw85 found but phase is Pending instead of Bound. I1105 03:03:00.319] Nov 5 03:03:00.319: INFO: PersistentVolumeClaim pvc-gtw85 found but phase is Pending instead of Bound. I1105 03:03:02.397] Nov 5 03:03:02.397: INFO: PersistentVolumeClaim pvc-gtw85 found but phase is Pending instead of Bound. I1105 03:03:04.474] Nov 5 03:03:04.474: INFO: PersistentVolumeClaim pvc-gtw85 found but phase is Pending instead of Bound. I1105 03:03:06.552] Nov 5 03:03:06.551: INFO: PersistentVolumeClaim pvc-gtw85 found but phase is Pending instead of Bound. I1105 03:03:08.637] Nov 5 03:03:08.636: INFO: PersistentVolumeClaim pvc-gtw85 found but phase is Pending instead of Bound. I1105 03:03:10.714] Nov 5 03:03:10.713: INFO: PersistentVolumeClaim pvc-gtw85 found but phase is Pending instead of Bound. I1105 03:03:12.791] Nov 5 03:03:12.791: INFO: PersistentVolumeClaim pvc-gtw85 found but phase is Pending instead of Bound. I1105 03:03:14.872] Nov 5 03:03:14.871: INFO: PersistentVolumeClaim pvc-gtw85 found but phase is Pending instead of Bound. I1105 03:03:16.949] Nov 5 03:03:16.949: INFO: PersistentVolumeClaim pvc-gtw85 found but phase is Pending instead of Bound. I1105 03:03:19.029] Nov 5 03:03:19.026: INFO: PersistentVolumeClaim pvc-gtw85 found but phase is Pending instead of Bound. I1105 03:03:21.120] Nov 5 03:03:21.120: INFO: PersistentVolumeClaim pvc-gtw85 found but phase is Pending instead of Bound. I1105 03:03:23.198] Nov 5 03:03:23.197: INFO: PersistentVolumeClaim pvc-gtw85 found but phase is Pending instead of Bound. I1105 03:03:25.275] Nov 5 03:03:25.275: INFO: PersistentVolumeClaim pvc-gtw85 found but phase is Pending instead of Bound. I1105 03:03:27.353] Nov 5 03:03:27.353: INFO: PersistentVolumeClaim pvc-gtw85 found but phase is Pending instead of Bound. I1105 03:03:29.430] Nov 5 03:03:29.430: INFO: PersistentVolumeClaim pvc-gtw85 found but phase is Pending instead of Bound. I1105 03:03:31.508] Nov 5 03:03:31.508: INFO: PersistentVolumeClaim pvc-gtw85 found but phase is Pending instead of Bound. I1105 03:03:33.591] Nov 5 03:03:33.590: INFO: PersistentVolumeClaim pvc-gtw85 found but phase is Pending instead of Bound. I1105 03:03:35.668] Nov 5 03:03:35.668: INFO: PersistentVolumeClaim pvc-gtw85 found but phase is Pending instead of Bound. I1105 03:03:37.746] Nov 5 03:03:37.745: INFO: PersistentVolumeClaim pvc-gtw85 found but phase is Pending instead of Bound. I1105 03:03:39.824] Nov 5 03:03:39.823: INFO: PersistentVolumeClaim pvc-gtw85 found but phase is Pending instead of Bound. I1105 03:03:41.901] Nov 5 03:03:41.901: INFO: PersistentVolumeClaim pvc-gtw85 found but phase is Pending instead of Bound. I1105 03:03:43.979] Nov 5 03:03:43.978: INFO: PersistentVolumeClaim pvc-gtw85 found but phase is Pending instead of Bound. I1105 03:03:46.062] Nov 5 03:03:46.061: INFO: PersistentVolumeClaim pvc-gtw85 found but phase is Pending instead of Bound. I1105 03:03:48.139] Nov 5 03:03:48.139: INFO: PersistentVolumeClaim pvc-gtw85 found but phase is Pending instead of Bound. I1105 03:03:50.217] Nov 5 03:03:50.217: INFO: PersistentVolumeClaim pvc-gtw85 found but phase is Pending instead of Bound. I1105 03:03:52.327] Nov 5 03:03:52.326: INFO: PersistentVolumeClaim pvc-gtw85 found but phase is Pending instead of Bound. I1105 03:03:54.658] Nov 5 03:03:54.404: INFO: PersistentVolumeClaim pvc-gtw85 found but phase is Pending instead of Bound. I1105 03:03:56.489] Nov 5 03:03:56.489: INFO: PersistentVolumeClaim pvc-gtw85 found but phase is Pending instead of Bound. I1105 03:03:58.567] Nov 5 03:03:58.567: INFO: PersistentVolumeClaim pvc-gtw85 found but phase is Pending instead of Bound. I1105 03:04:00.833] Nov 5 03:04:00.644: INFO: PersistentVolumeClaim pvc-gtw85 found but phase is Pending instead of Bound. I1105 03:04:02.721] Nov 5 03:04:02.721: INFO: PersistentVolumeClaim pvc-gtw85 found but phase is Pending instead of Bound. I1105 03:04:04.799] Nov 5 03:04:04.799: INFO: PersistentVolumeClaim pvc-gtw85 found but phase is Pending instead of Bound. I1105 03:04:06.877] Nov 5 03:04:06.877: INFO: PersistentVolumeClaim pvc-gtw85 found but phase is Pending instead of Bound. I1105 03:04:08.954] Nov 5 03:04:08.954: INFO: PersistentVolumeClaim pvc-gtw85 found but phase is Pending instead of Bound. I1105 03:04:11.039] Nov 5 03:04:11.034: INFO: PersistentVolumeClaim pvc-gtw85 found but phase is Pending instead of Bound. I1105 03:04:13.118] Nov 5 03:04:13.118: INFO: PersistentVolumeClaim pvc-gtw85 found but phase is Pending instead of Bound. I1105 03:04:15.200] Nov 5 03:04:15.200: INFO: PersistentVolumeClaim pvc-gtw85 found but phase is Pending instead of Bound. I1105 03:04:17.278] Nov 5 03:04:17.278: INFO: PersistentVolumeClaim pvc-gtw85 found but phase is Pending instead of Bound. I1105 03:04:19.355] Nov 5 03:04:19.355: INFO: PersistentVolumeClaim pvc-gtw85 found but phase is Pending instead of Bound. I1105 03:04:21.434] Nov 5 03:04:21.434: INFO: PersistentVolumeClaim pvc-gtw85 found but phase is Pending instead of Bound. I1105 03:04:23.511] Nov 5 03:04:23.511: INFO: PersistentVolumeClaim pvc-gtw85 found but phase is Pending instead of Bound. I1105 03:04:25.602] Nov 5 03:04:25.597: INFO: PersistentVolumeClaim pvc-gtw85 found but phase is Pending instead of Bound. I1105 03:04:27.682] Nov 5 03:04:27.678: INFO: PersistentVolumeClaim pvc-gtw85 found but phase is Pending instead of Bound. I1105 03:04:29.766] Nov 5 03:04:29.762: INFO: PersistentVolumeClaim pvc-gtw85 found but phase is Pending instead of Bound. I1105 03:04:31.839] Nov 5 03:04:31.839: INFO: PersistentVolumeClaim pvc-gtw85 found but phase is Pending instead of Bound. I1105 03:04:33.917] Nov 5 03:04:33.917: INFO: PersistentVolumeClaim pvc-gtw85 found but phase is Pending instead of Bound. I1105 03:04:36.009] Nov 5 03:04:36.009: INFO: PersistentVolumeClaim pvc-gtw85 found but phase is Pending instead of Bound. I1105 03:04:38.086] Nov 5 03:04:38.086: INFO: PersistentVolumeClaim pvc-gtw85 found but phase is Pending instead of Bound. I1105 03:04:40.164] Nov 5 03:04:40.164: INFO: PersistentVolumeClaim pvc-gtw85 found but phase is Pending instead of Bound. I1105 03:04:42.242] Nov 5 03:04:42.242: INFO: PersistentVolumeClaim pvc-gtw85 found but phase is Pending instead of Bound. I1105 03:04:44.319] Nov 5 03:04:44.319: INFO: PersistentVolumeClaim pvc-gtw85 found but phase is Pending instead of Bound. I1105 03:04:46.397] Nov 5 03:04:46.396: INFO: PersistentVolumeClaim pvc-gtw85 found but phase is Pending instead of Bound. I1105 03:04:48.478] Nov 5 03:04:48.477: INFO: PersistentVolumeClaim pvc-gtw85 found but phase is Pending instead of Bound. I1105 03:04:50.555] Nov 5 03:04:50.554: INFO: PersistentVolumeClaim pvc-gtw85 found but phase is Pending instead of Bound. I1105 03:04:52.632] Nov 5 03:04:52.632: INFO: PersistentVolumeClaim pvc-gtw85 found but phase is Pending instead of Bound. I1105 03:04:54.710] Nov 5 03:04:54.710: INFO: PersistentVolumeClaim pvc-gtw85 found but phase is Pending instead of Bound. I1105 03:04:56.792] Nov 5 03:04:56.792: INFO: PersistentVolumeClaim pvc-gtw85 found but phase is Pending instead of Bound. I1105 03:04:58.873] Nov 5 03:04:58.871: INFO: PersistentVolumeClaim pvc-gtw85 found but phase is Pending instead of Bound. I1105 03:05:00.949] Nov 5 03:05:00.948: INFO: PersistentVolumeClaim pvc-gtw85 found but phase is Pending instead of Bound. I1105 03:05:03.026] Nov 5 03:05:03.026: INFO: PersistentVolumeClaim pvc-gtw85 found but phase is Pending instead of Bound. I1105 03:05:05.129] Nov 5 03:05:05.128: INFO: PersistentVolumeClaim pvc-gtw85 found but phase is Pending instead of Bound. I1105 03:05:07.207] Nov 5 03:05:07.206: INFO: PersistentVolumeClaim pvc-gtw85 found but phase is Pending instead of Bound. I1105 03:05:09.286] Nov 5 03:05:09.285: INFO: PersistentVolumeClaim pvc-gtw85 found but phase is Pending instead of Bound. I1105 03:05:11.363] Nov 5 03:05:11.362: INFO: PersistentVolumeClaim pvc-gtw85 found but phase is Pending instead of Bound. I1105 03:05:13.442] Nov 5 03:05:13.440: INFO: PersistentVolumeClaim pvc-gtw85 found but phase is Pending instead of Bound. I1105 03:05:15.518] Nov 5 03:05:15.518: INFO: PersistentVolumeClaim pvc-gtw85 found but phase is Pending instead of Bound. I1105 03:05:18.111] Nov 5 03:05:17.596: INFO: PersistentVolumeClaim pvc-gtw85 found but phase is Pending instead of Bound. I1105 03:05:19.674] Nov 5 03:05:19.674: INFO: PersistentVolumeClaim pvc-gtw85 found but phase is Pending instead of Bound. I1105 03:05:21.753] Nov 5 03:05:21.753: INFO: PersistentVolumeClaim pvc-gtw85 found but phase is Pending instead of Bound. I1105 03:05:23.833] Nov 5 03:05:23.833: INFO: PersistentVolumeClaim pvc-gtw85 found but phase is Pending instead of Bound. I1105 03:05:25.956] Nov 5 03:05:25.910: INFO: PersistentVolumeClaim pvc-gtw85 found but phase is Pending instead of Bound. I1105 03:05:27.998] Nov 5 03:05:27.994: INFO: PersistentVolumeClaim pvc-gtw85 found but phase is Pending instead of Bound. I1105 03:05:30.072] Nov 5 03:05:30.072: INFO: PersistentVolumeClaim pvc-gtw85 found but phase is Pending instead of Bound. I1105 03:05:32.150] Nov 5 03:05:32.149: INFO: PersistentVolumeClaim pvc-gtw85 found but phase is Pending instead of Bound. I1105 03:05:34.227] Nov 5 03:05:34.227: INFO: PersistentVolumeClaim pvc-gtw85 found but phase is Pending instead of Bound. I1105 03:05:39.008] Nov 5 03:05:36.305: INFO: PersistentVolumeClaim pvc-gtw85 found but phase is Pending instead of Bound. I1105 03:05:39.009] Nov 5 03:05:38.382: INFO: PersistentVolumeClaim pvc-gtw85 found but phase is Pending instead of Bound. I1105 03:05:40.460] Nov 5 03:05:40.460: INFO: PersistentVolumeClaim pvc-gtw85 found but phase is Pending instead of Bound. I1105 03:05:42.537] Nov 5 03:05:42.537: INFO: PersistentVolumeClaim pvc-gtw85 found but phase is Pending instead of Bound. I1105 03:05:44.615] Nov 5 03:05:44.615: INFO: PersistentVolumeClaim pvc-gtw85 found but phase is Pending instead of Bound. I1105 03:05:46.693] Nov 5 03:05:46.693: INFO: PersistentVolumeClaim pvc-gtw85 found but phase is Pending instead of Bound. I1105 03:05:48.771] Nov 5 03:05:48.770: INFO: PersistentVolumeClaim pvc-gtw85 found but phase is Pending instead of Bound. I1105 03:05:50.853] Nov 5 03:05:50.852: INFO: PersistentVolumeClaim pvc-gtw85 found but phase is Pending instead of Bound. I1105 03:05:52.930] Nov 5 03:05:52.930: INFO: PersistentVolumeClaim pvc-gtw85 found but phase is Pending instead of Bound. I1105 03:05:56.118] Nov 5 03:05:55.008: INFO: PersistentVolumeClaim pvc-gtw85 found but phase is Pending instead of Bound. I1105 03:05:57.086] Nov 5 03:05:57.085: INFO: PersistentVolumeClaim pvc-gtw85 found but phase is Pending instead of Bound. I1105 03:05:59.163] Nov 5 03:05:59.163: INFO: PersistentVolumeClaim pvc-gtw85 found but phase is Pending instead of Bound. I1105 03:06:01.251] Nov 5 03:06:01.251: INFO: PersistentVolumeClaim pvc-gtw85 found but phase is Pending instead of Bound. I1105 03:06:03.328] Nov 5 03:06:03.328: INFO: PersistentVolumeClaim pvc-gtw85 found but phase is Pending instead of Bound. I1105 03:06:05.406] Nov 5 03:06:05.405: INFO: PersistentVolumeClaim pvc-gtw85 found but phase is Pending instead of Bound. I1105 03:06:08.559] Nov 5 03:06:07.483: INFO: PersistentVolumeClaim pvc-gtw85 found but phase is Pending instead of Bound. I1105 03:06:09.561] Nov 5 03:06:09.561: INFO: PersistentVolumeClaim pvc-gtw85 found but phase is Pending instead of Bound. I1105 03:06:12.413] Nov 5 03:06:11.639: INFO: PersistentVolumeClaim pvc-gtw85 found but phase is Pending instead of Bound. I1105 03:06:13.722] Nov 5 03:06:13.716: INFO: PersistentVolumeClaim pvc-gtw85 found but phase is Pending instead of Bound. I1105 03:06:15.798] Nov 5 03:06:15.794: INFO: PersistentVolumeClaim pvc-gtw85 found but phase is Pending instead of Bound. I1105 03:06:17.886] Nov 5 03:06:17.878: INFO: PersistentVolumeClaim pvc-gtw85 found but phase is Pending instead of Bound. I1105 03:06:19.966] Nov 5 03:06:19.962: INFO: PersistentVolumeClaim pvc-gtw85 found but phase is Pending instead of Bound. I1105 03:06:22.040] Nov 5 03:06:22.040: INFO: PersistentVolumeClaim pvc-gtw85 found but phase is Pending instead of Bound. I1105 03:06:24.118] Nov 5 03:06:24.118: INFO: PersistentVolumeClaim pvc-gtw85 found but phase is Pending instead of Bound. I1105 03:06:26.196] Nov 5 03:06:26.196: INFO: PersistentVolumeClaim pvc-gtw85 found but phase is Pending instead of Bound. I1105 03:06:28.274] Nov 5 03:06:28.274: INFO: PersistentVolumeClaim pvc-gtw85 found but phase is Pending instead of Bound. I1105 03:06:34.083] Nov 5 03:06:30.351: INFO: PersistentVolumeClaim pvc-gtw85 found but phase is Pending instead of Bound. I1105 03:06:34.083] Nov 5 03:06:32.433: INFO: PersistentVolumeClaim pvc-gtw85 found but phase is Pending instead of Bound. I1105 03:06:34.511] Nov 5 03:06:34.511: INFO: PersistentVolumeClaim pvc-gtw85 found but phase is Pending instead of Bound. I1105 03:06:36.589] Nov 5 03:06:36.588: INFO: PersistentVolumeClaim pvc-gtw85 found but phase is Pending instead of Bound. I1105 03:06:38.589] Nov 5 03:06:38.589: INFO: Unexpected error occurred: PersistentVolumeClaims [pvc-gtw85] not all in phase Bound within 5m0s I1105 03:06:38.590] [AfterEach] [multi-az] I1105 03:06:38.590] /go/src/sigs.k8s.io/azuredisk-csi-driver/vendor/k8s.io/kubernetes/test/e2e/framework/framework.go:151 I1105 03:06:38.590] STEP: Collecting events from namespace "azuredisk-6855". I1105 03:06:38.667] STEP: Found 3 events. I1105 03:06:38.667] Nov 5 03:06:38.667: INFO: At 2019-11-05 03:01:37 +0000 UTC - event for pvc-gtw85: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "disk.csi.azure.com" or manually created by system administrator I1105 03:06:38.668] Nov 5 03:06:38.667: INFO: At 2019-11-05 03:01:37 +0000 UTC - event for pvc-gtw85: {disk.csi.azure.com_csi-azuredisk-controller-6f6587bfc6-8vkfl_1376914b-ff6d-11e9-93ab-ba2bee589b8c } Provisioning: External provisioner is provisioning volume for claim "azuredisk-6855/pvc-gtw85" I1105 03:06:38.669] Nov 5 03:06:38.667: INFO: At 2019-11-05 03:01:37 +0000 UTC - event for pvc-gtw85: {disk.csi.azure.com_csi-azuredisk-controller-6f6587bfc6-8vkfl_1376914b-ff6d-11e9-93ab-ba2bee589b8c } ProvisioningFailed: failed to provision volume with StorageClass "azuredisk-6855-disk.csi.azure.com-dynamic-sc-g682z": rpc error: code = Unknown desc = compute.DisksClient#CreateOrUpdate: Failure sending request: StatusCode=400 -- Original Error: Code="InvalidAvailabilityZone" Message="The zone(s) '' for resource 'Microsoft.Compute/disks/pvc-818f53fd-e3ad-4de7-9f8b-8e5b9f98b04d' is not supported. The supported zones for location 'eastus2' are '1,2,3'" I1105 03:06:38.744] Nov 5 03:06:38.744: INFO: POD NODE PHASE GRACE CONDITIONS I1105 03:06:38.745] Nov 5 03:06:38.744: INFO: I1105 03:06:38.974] Nov 5 03:06:38.974: INFO: I1105 03:06:38.975] Logging node info for node k8s-agentpool-15826033-vmss000000 I1105 03:06:39.055] Nov 5 03:06:39.052: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-agentpool-15826033-vmss000000,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-agentpool-15826033-vmss000000,UID:15fa874e-54ec-42cd-b243-d3baa512150f,ResourceVersion:17562,Generation:0,CreationTimestamp:2019-11-05 01:34:17 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{agentpool: agentpool,beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_D4s_v3,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 0,kubernetes.azure.com/cluster: kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5,kubernetes.azure.com/role: agent,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-agentpool-15826033-vmss000000,kubernetes.io/os: linux,kubernetes.io/role: agent,node-role.kubernetes.io/agent: ,storageprofile: managed,storagetier: Premium_LRS,topology.disk.csi.azure.com/zone: 0,},Annotations:map[string]string{csi.volume.kubernetes.io/nodeid: {"disk.csi.azure.com":"k8s-agentpool-15826033-vmss000000"},node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.2.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool-15826033-vmss/virtualMachines/0,Unschedulable:false,Taints:[],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-csi-disk.csi.azure.com: {{8 0} {} 8 DecimalSI},cpu: {{4 0} {} 4 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{16797044736 0} {} 16403364Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-csi-disk.csi.azure.com: {{8 0} {} 8 DecimalSI},cpu: {{4 0} {} 4 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{16010612736 0} {} 15635364Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 01:34:56 +0000 UTC 2019-11-05 01:34:56 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 03:06:32 +0000 UTC 2019-11-05 01:34:17 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 03:06:32 +0000 UTC 2019-11-05 01:34:17 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 03:06:32 +0000 UTC 2019-11-05 01:34:17 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 03:06:32 +0000 UTC 2019-11-05 01:34:29 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.0.4} {Hostname k8s-agentpool-15826033-vmss000000}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:afe00792177d49a98f6a44d03d08732d,SystemUUID:4F1777BF-A5C3-004E-81C4-90F0B54C02A0,BootID:13b6a628-ae89-44af-b891-89461030adcb,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[k8sprowinternal.azurecr.io/azuredisk-csi@sha256:75d4ff5d7cd845d6e810c67796ecb037e852fb849ec1eb5b3194564008e6d1e0 k8sprowinternal.azurecr.io/azuredisk-csi:e2e-3e3ed045c6faa509dbd8d2890ee9f56aa2934887] 286930414} {[k8s.gcr.io/kubernetes-dashboard-amd64@sha256:0ae6b69432e78069c5ce2bcde0fe409c5c4d6f0f4d9cd50a17974fea38898747 k8s.gcr.io/kubernetes-dashboard-amd64:v1.10.1] 121711221} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/metrics-server-amd64@sha256:49a9f12f7067d11f42c803dbe61ed2c1299959ad85cb315b25ff7eef8e6b8892 k8s.gcr.io/metrics-server-amd64:v0.2.1] 42541759} {[mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume@sha256:4fd30d43947d4a54fc89ead7985beecfd3c9b2a93a0655a373b1608ab90bd5af mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume:v0.0.7] 22909487} {[quay.io/k8scsi/livenessprobe@sha256:8c9c8a0f330acc2e558ae6624309447d02522e76aa35af23fa19b88abf4381cb quay.io/k8scsi/livenessprobe:v1.0.2] 17415381} {[quay.io/k8scsi/csi-node-driver-registrar@sha256:13daf82fb99e951a4bff8ae5fc7c17c3a8fe7130be6400990d8f6076c32d4599 quay.io/k8scsi/csi-node-driver-registrar:v1.1.0] 15815995} {[mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume@sha256:23d8c6033f02a1ecad05127ebdc931bb871264228661bc122704b0974e4d9fdd mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume:1.0.8] 1159025} {[busybox@sha256:8ccbac733d19c0dd4d70b4f0c1e12245b5fa3ad24758a11035ee505c629c0796 busybox:1.29] 1154361} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 03:06:39.055] Nov 5 03:06:39.052: INFO: I1105 03:06:39.055] Logging kubelet events for node k8s-agentpool-15826033-vmss000000 I1105 03:06:39.130] Nov 5 03:06:39.129: INFO: I1105 03:06:39.130] Logging pods the kubelet thinks is on node k8s-agentpool-15826033-vmss000000 I1105 03:06:39.212] Nov 5 03:06:39.212: INFO: keyvault-flexvolume-qhnfr started at 2019-11-05 01:34:34 +0000 UTC (0+1 container statuses recorded) I1105 03:06:39.212] Nov 5 03:06:39.212: INFO: Container keyvault-flexvolume ready: true, restart count 0 I1105 03:06:39.213] Nov 5 03:06:39.212: INFO: metrics-server-864ffbc5c-sdxps started at 2019-11-05 01:34:34 +0000 UTC (0+1 container statuses recorded) I1105 03:06:39.213] Nov 5 03:06:39.212: INFO: Container metrics-server ready: true, restart count 0 I1105 03:06:39.213] Nov 5 03:06:39.212: INFO: kube-proxy-wxblc started at 2019-11-05 01:34:34 +0000 UTC (0+1 container statuses recorded) I1105 03:06:39.213] Nov 5 03:06:39.212: INFO: Container kube-proxy ready: true, restart count 0 I1105 03:06:39.213] Nov 5 03:06:39.212: INFO: kubernetes-dashboard-66dd8b8df7-ks284 started at 2019-11-05 01:34:35 +0000 UTC (0+1 container statuses recorded) I1105 03:06:39.213] Nov 5 03:06:39.212: INFO: Container kubernetes-dashboard ready: true, restart count 0 I1105 03:06:39.214] Nov 5 03:06:39.212: INFO: csi-azuredisk-node-whq2g started at 2019-11-05 01:38:24 +0000 UTC (0+3 container statuses recorded) I1105 03:06:39.214] Nov 5 03:06:39.212: INFO: Container azuredisk ready: true, restart count 0 I1105 03:06:39.214] Nov 5 03:06:39.212: INFO: Container liveness-probe ready: true, restart count 0 I1105 03:06:39.214] Nov 5 03:06:39.212: INFO: Container node-driver-registrar ready: true, restart count 0 I1105 03:06:39.214] Nov 5 03:06:39.212: INFO: blobfuse-flexvol-installer-bsvqt started at 2019-11-05 01:34:34 +0000 UTC (0+1 container statuses recorded) I1105 03:06:39.214] Nov 5 03:06:39.212: INFO: Container blobfuse-flexvol-installer ready: true, restart count 0 I1105 03:06:39.215] Nov 5 03:06:39.212: INFO: azure-ip-masq-agent-vx86b started at 2019-11-05 01:34:34 +0000 UTC (0+1 container statuses recorded) I1105 03:06:39.215] Nov 5 03:06:39.212: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 03:06:39.480] Nov 5 03:06:39.480: INFO: I1105 03:06:39.480] Latency metrics for node k8s-agentpool-15826033-vmss000000 I1105 03:06:39.481] Nov 5 03:06:39.480: INFO: I1105 03:06:39.481] Logging node info for node k8s-agentpool-15826033-vmss000001 I1105 03:06:39.561] Nov 5 03:06:39.557: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-agentpool-15826033-vmss000001,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-agentpool-15826033-vmss000001,UID:4a8ea605-cc0b-4e3b-9060-d8867d269e6c,ResourceVersion:17456,Generation:0,CreationTimestamp:2019-11-05 01:34:09 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{agentpool: agentpool,beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_D4s_v3,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 1,kubernetes.azure.com/cluster: kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5,kubernetes.azure.com/role: agent,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-agentpool-15826033-vmss000001,kubernetes.io/os: linux,kubernetes.io/role: agent,node-role.kubernetes.io/agent: ,storageprofile: managed,storagetier: Premium_LRS,topology.disk.csi.azure.com/zone: 1,},Annotations:map[string]string{csi.volume.kubernetes.io/nodeid: {"disk.csi.azure.com":"k8s-agentpool-15826033-vmss000001"},node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.0.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool-15826033-vmss/virtualMachines/1,Unschedulable:false,Taints:[],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-csi-disk.csi.azure.com: {{8 0} {} 8 DecimalSI},cpu: {{4 0} {} 4 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{16797044736 0} {} 16403364Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-csi-disk.csi.azure.com: {{8 0} {} 8 DecimalSI},cpu: {{4 0} {} 4 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{16010612736 0} {} 15635364Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 01:34:57 +0000 UTC 2019-11-05 01:34:57 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 03:05:54 +0000 UTC 2019-11-05 01:34:08 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 03:05:54 +0000 UTC 2019-11-05 01:34:08 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 03:05:54 +0000 UTC 2019-11-05 01:34:08 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 03:05:54 +0000 UTC 2019-11-05 01:34:29 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.0.5} {Hostname k8s-agentpool-15826033-vmss000001}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:b6b9fbf4a2fd4fa5a7ae472415fe30b8,SystemUUID:0162B3A4-3955-9A46-ABD4-A37BD75848CC,BootID:ee93003f-d47c-4118-9975-57365b3443a5,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[k8sprowinternal.azurecr.io/azuredisk-csi@sha256:75d4ff5d7cd845d6e810c67796ecb037e852fb849ec1eb5b3194564008e6d1e0 k8sprowinternal.azurecr.io/azuredisk-csi:e2e-3e3ed045c6faa509dbd8d2890ee9f56aa2934887] 286930414} {[gcr.io/kubernetes-helm/tiller@sha256:f6d8f4ab9ba993b5f5b60a6edafe86352eabe474ffeb84cb6c79b8866dce45d1 gcr.io/kubernetes-helm/tiller:v2.11.0] 71821984} {[quay.io/k8scsi/csi-provisioner@sha256:d657c839dce87324fe2b677302913f9386f885f8746be7bea0ced5b0844e3433 quay.io/k8scsi/csi-provisioner:v1.3.1] 50596998} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[quay.io/k8scsi/csi-attacher@sha256:26fccd7a99d973845df1193b46ebdcc6ab8dc5f6e6be319750c471fce1742d13 quay.io/k8scsi/csi-attacher:v1.2.0] 46226754} {[quay.io/k8scsi/csi-cluster-driver-registrar@sha256:fafd75ae5442f192cfa8c2e792903aee30d5884b62e802e4464b0a895d21e3ef quay.io/k8scsi/csi-cluster-driver-registrar:v1.0.1] 45874691} {[quay.io/k8scsi/csi-snapshotter@sha256:a49e0da1af6f2bf717e41ba1eee8b5e6a1cbd66a709dd92cc43fe475fe2589eb quay.io/k8scsi/csi-snapshotter:v1.1.0] 44325352} {[k8s.gcr.io/coredns@sha256:e83beb5e43f8513fa735e77ffc5859640baea30a882a11cc75c4c3244a737d3c k8s.gcr.io/coredns:1.5.0] 42488424} {[mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume@sha256:4fd30d43947d4a54fc89ead7985beecfd3c9b2a93a0655a373b1608ab90bd5af mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume:v0.0.7] 22909487} {[quay.io/k8scsi/livenessprobe@sha256:8c9c8a0f330acc2e558ae6624309447d02522e76aa35af23fa19b88abf4381cb quay.io/k8scsi/livenessprobe:v1.0.2] 17415381} {[quay.io/k8scsi/csi-node-driver-registrar@sha256:13daf82fb99e951a4bff8ae5fc7c17c3a8fe7130be6400990d8f6076c32d4599 quay.io/k8scsi/csi-node-driver-registrar:v1.1.0] 15815995} {[quay.io/k8scsi/livenessprobe@sha256:dde617756e0f602adc566ab71fd885f1dad451ad3fb063ac991c95a2ff47aea5 quay.io/k8scsi/livenessprobe:v1.1.0] 14967303} {[mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume@sha256:23d8c6033f02a1ecad05127ebdc931bb871264228661bc122704b0974e4d9fdd mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume:1.0.8] 1159025} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 03:06:41.863] Nov 5 03:06:39.558: INFO: I1105 03:06:41.864] Logging kubelet events for node k8s-agentpool-15826033-vmss000001 I1105 03:06:41.864] Nov 5 03:06:39.635: INFO: I1105 03:06:41.864] Logging pods the kubelet thinks is on node k8s-agentpool-15826033-vmss000001 I1105 03:06:41.864] Nov 5 03:06:39.718: INFO: csi-azuredisk-controller-6f6587bfc6-8vkfl started at 2019-11-05 01:38:24 +0000 UTC (0+6 container statuses recorded) I1105 03:06:41.865] Nov 5 03:06:39.718: INFO: Container azuredisk ready: true, restart count 0 I1105 03:06:41.865] Nov 5 03:06:39.718: INFO: Container cluster-driver-registrar ready: true, restart count 0 I1105 03:06:41.865] Nov 5 03:06:39.718: INFO: Container csi-attacher ready: true, restart count 0 I1105 03:06:41.866] Nov 5 03:06:39.718: INFO: Container csi-provisioner ready: true, restart count 0 I1105 03:06:41.866] Nov 5 03:06:39.718: INFO: Container csi-snapshotter ready: true, restart count 0 I1105 03:06:41.867] Nov 5 03:06:39.718: INFO: Container liveness-probe ready: true, restart count 0 I1105 03:06:41.867] Nov 5 03:06:39.718: INFO: blobfuse-flexvol-installer-rw5dw started at 2019-11-05 01:34:35 +0000 UTC (0+1 container statuses recorded) I1105 03:06:41.867] Nov 5 03:06:39.718: INFO: Container blobfuse-flexvol-installer ready: true, restart count 0 I1105 03:06:41.868] Nov 5 03:06:39.718: INFO: coredns-7f68dcdbdb-jt522 started at 2019-11-05 01:34:35 +0000 UTC (0+1 container statuses recorded) I1105 03:06:41.868] Nov 5 03:06:39.718: INFO: Container coredns ready: true, restart count 0 I1105 03:06:41.869] Nov 5 03:06:39.718: INFO: azure-ip-masq-agent-6c6vn started at 2019-11-05 01:34:35 +0000 UTC (0+1 container statuses recorded) I1105 03:06:41.869] Nov 5 03:06:39.718: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 03:06:41.869] Nov 5 03:06:39.718: INFO: keyvault-flexvolume-rdlcj started at 2019-11-05 01:34:35 +0000 UTC (0+1 container statuses recorded) I1105 03:06:41.869] Nov 5 03:06:39.718: INFO: Container keyvault-flexvolume ready: true, restart count 0 I1105 03:06:41.870] Nov 5 03:06:39.718: INFO: kube-proxy-6kh57 started at 2019-11-05 01:34:35 +0000 UTC (0+1 container statuses recorded) I1105 03:06:41.870] Nov 5 03:06:39.718: INFO: Container kube-proxy ready: true, restart count 0 I1105 03:06:41.870] Nov 5 03:06:39.718: INFO: tiller-deploy-54c96cb5df-8d9fs started at 2019-11-05 01:34:35 +0000 UTC (0+1 container statuses recorded) I1105 03:06:41.871] Nov 5 03:06:39.718: INFO: Container tiller ready: true, restart count 0 I1105 03:06:41.871] Nov 5 03:06:39.718: INFO: csi-azuredisk-node-ghv5w started at 2019-11-05 01:38:24 +0000 UTC (0+3 container statuses recorded) I1105 03:06:41.871] Nov 5 03:06:39.718: INFO: Container azuredisk ready: true, restart count 0 I1105 03:06:41.871] Nov 5 03:06:39.718: INFO: Container liveness-probe ready: true, restart count 0 I1105 03:06:41.872] Nov 5 03:06:39.718: INFO: Container node-driver-registrar ready: true, restart count 0 I1105 03:06:42.228] Nov 5 03:06:42.228: INFO: I1105 03:06:42.560] Latency metrics for node k8s-agentpool-15826033-vmss000001 I1105 03:06:42.560] Nov 5 03:06:42.228: INFO: I1105 03:06:42.560] Logging node info for node k8s-master-15826033-0 I1105 03:06:42.563] Nov 5 03:06:42.306: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-15826033-0,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-15826033-0,UID:aceb0231-4645-4a9a-b296-90ada851986b,ResourceVersion:17582,Generation:0,CreationTimestamp:2019-11-05 01:35:06 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 1,kubernetes.azure.com/cluster: kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-15826033-0,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.4.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5/providers/Microsoft.Compute/virtualMachines/k8s-master-15826033-0,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 01:35:20 +0000 UTC 2019-11-05 01:35:20 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 03:06:38 +0000 UTC 2019-11-05 01:34:57 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 03:06:38 +0000 UTC 2019-11-05 01:34:57 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 03:06:38 +0000 UTC 2019-11-05 01:34:57 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 03:06:38 +0000 UTC 2019-11-05 01:35:07 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.255.5} {Hostname k8s-master-15826033-0}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:1915756c22a540dc8eb30493987e9a43,SystemUUID:742C969C-5A71-D744-8E61-F27E38B56EB2,BootID:03f0672e-8fa9-4309-956f-8e3c371902c3,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 03:06:42.564] Nov 5 03:06:42.306: INFO: I1105 03:06:42.564] Logging kubelet events for node k8s-master-15826033-0 I1105 03:06:42.564] Nov 5 03:06:42.383: INFO: I1105 03:06:42.564] Logging pods the kubelet thinks is on node k8s-master-15826033-0 I1105 03:06:42.565] Nov 5 03:06:42.464: INFO: azure-ip-masq-agent-wgdpx started at 2019-11-05 01:35:06 +0000 UTC (0+1 container statuses recorded) I1105 03:06:42.565] Nov 5 03:06:42.464: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 03:06:42.565] Nov 5 03:06:42.464: INFO: kube-proxy-7d2px started at 2019-11-05 01:35:06 +0000 UTC (0+1 container statuses recorded) I1105 03:06:42.565] Nov 5 03:06:42.464: INFO: Container kube-proxy ready: true, restart count 0 I1105 03:06:42.566] Nov 5 03:06:42.464: INFO: cloud-controller-manager-k8s-master-15826033-0 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 03:06:42.566] Nov 5 03:06:42.464: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 03:06:42.566] Nov 5 03:06:42.464: INFO: kube-addon-manager-k8s-master-15826033-0 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 03:06:42.566] Nov 5 03:06:42.464: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 03:06:42.567] Nov 5 03:06:42.464: INFO: kube-apiserver-k8s-master-15826033-0 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 03:06:42.567] Nov 5 03:06:42.464: INFO: Container kube-apiserver ready: true, restart count 0 I1105 03:06:42.567] Nov 5 03:06:42.464: INFO: kube-controller-manager-k8s-master-15826033-0 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 03:06:42.567] Nov 5 03:06:42.464: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 03:06:42.568] Nov 5 03:06:42.464: INFO: kube-scheduler-k8s-master-15826033-0 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 03:06:42.568] Nov 5 03:06:42.464: INFO: Container kube-scheduler ready: true, restart count 0 I1105 03:06:42.718] Nov 5 03:06:42.717: INFO: I1105 03:06:42.718] Latency metrics for node k8s-master-15826033-0 I1105 03:06:42.718] Nov 5 03:06:42.717: INFO: I1105 03:06:42.718] Logging node info for node k8s-master-15826033-1 I1105 03:06:42.801] Nov 5 03:06:42.798: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-15826033-1,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-15826033-1,UID:1a13ec61-c89a-456c-8c94-bfc8e717c2e7,ResourceVersion:17585,Generation:0,CreationTimestamp:2019-11-05 01:35:07 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 1,kubernetes.azure.com/cluster: kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-15826033-1,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.5.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5/providers/Microsoft.Compute/virtualMachines/k8s-master-15826033-1,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 01:35:31 +0000 UTC 2019-11-05 01:35:31 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 03:06:39 +0000 UTC 2019-11-05 01:35:00 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 03:06:39 +0000 UTC 2019-11-05 01:35:00 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 03:06:39 +0000 UTC 2019-11-05 01:35:00 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 03:06:39 +0000 UTC 2019-11-05 01:35:07 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.255.6} {Hostname k8s-master-15826033-1}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:e0f2bccddd5b4555b180d14081ab59a8,SystemUUID:AE126631-CAD2-B54F-B22E-1F499342A4D6,BootID:509b9a9d-b046-4861-aa35-8bcb77465de2,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 03:06:42.802] Nov 5 03:06:42.798: INFO: I1105 03:06:42.802] Logging kubelet events for node k8s-master-15826033-1 I1105 03:06:42.878] Nov 5 03:06:42.877: INFO: I1105 03:06:42.878] Logging pods the kubelet thinks is on node k8s-master-15826033-1 I1105 03:06:42.965] Nov 5 03:06:42.959: INFO: cloud-controller-manager-k8s-master-15826033-1 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 03:06:42.965] Nov 5 03:06:42.959: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 03:06:42.966] Nov 5 03:06:42.959: INFO: kube-addon-manager-k8s-master-15826033-1 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 03:06:42.966] Nov 5 03:06:42.959: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 03:06:42.966] Nov 5 03:06:42.959: INFO: kube-apiserver-k8s-master-15826033-1 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 03:06:42.966] Nov 5 03:06:42.959: INFO: Container kube-apiserver ready: true, restart count 0 I1105 03:06:42.967] Nov 5 03:06:42.959: INFO: kube-controller-manager-k8s-master-15826033-1 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 03:06:42.967] Nov 5 03:06:42.959: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 03:06:42.967] Nov 5 03:06:42.959: INFO: kube-scheduler-k8s-master-15826033-1 started at 2019-11-05 01:36:16 +0000 UTC (0+1 container statuses recorded) I1105 03:06:42.967] Nov 5 03:06:42.959: INFO: Container kube-scheduler ready: true, restart count 0 I1105 03:06:42.968] Nov 5 03:06:42.959: INFO: azure-ip-masq-agent-w6dpr started at 2019-11-05 01:35:10 +0000 UTC (0+1 container statuses recorded) I1105 03:06:42.968] Nov 5 03:06:42.959: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 03:06:42.968] Nov 5 03:06:42.959: INFO: kube-proxy-8kmr5 started at 2019-11-05 01:35:10 +0000 UTC (0+1 container statuses recorded) I1105 03:06:42.968] Nov 5 03:06:42.959: INFO: Container kube-proxy ready: true, restart count 0 I1105 03:06:43.224] Nov 5 03:06:43.224: INFO: I1105 03:06:43.225] Latency metrics for node k8s-master-15826033-1 I1105 03:06:43.225] Nov 5 03:06:43.224: INFO: I1105 03:06:43.225] Logging node info for node k8s-master-15826033-2 I1105 03:06:43.313] Nov 5 03:06:43.302: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-15826033-2,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-15826033-2,UID:acb3a851-d6df-4549-ba56-4b7e96553c17,ResourceVersion:17501,Generation:0,CreationTimestamp:2019-11-05 01:34:09 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 2,kubernetes.azure.com/cluster: kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-15826033-2,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.1.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5/providers/Microsoft.Compute/virtualMachines/k8s-master-15826033-2,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 01:34:44 +0000 UTC 2019-11-05 01:34:44 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 03:06:10 +0000 UTC 2019-11-05 01:34:07 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 03:06:10 +0000 UTC 2019-11-05 01:34:07 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 03:06:10 +0000 UTC 2019-11-05 01:34:07 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 03:06:10 +0000 UTC 2019-11-05 01:34:29 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{Hostname k8s-master-15826033-2} {InternalIP 10.240.255.7}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:0885b0c2f6174106bde4fa0eae2cbc6b,SystemUUID:C176121C-5D7A-6243-9B05-8B68F8022B64,BootID:30430638-7ab6-4761-b984-2387e1a3f5b2,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 03:06:43.313] Nov 5 03:06:43.302: INFO: I1105 03:06:43.313] Logging kubelet events for node k8s-master-15826033-2 I1105 03:06:43.384] Nov 5 03:06:43.382: INFO: I1105 03:06:43.385] Logging pods the kubelet thinks is on node k8s-master-15826033-2 I1105 03:06:43.465] Nov 5 03:06:43.465: INFO: kube-apiserver-k8s-master-15826033-2 started at 2019-11-05 01:33:54 +0000 UTC (0+1 container statuses recorded) I1105 03:06:43.466] Nov 5 03:06:43.465: INFO: Container kube-apiserver ready: true, restart count 0 I1105 03:06:43.466] Nov 5 03:06:43.465: INFO: kube-controller-manager-k8s-master-15826033-2 started at 2019-11-05 01:33:54 +0000 UTC (0+1 container statuses recorded) I1105 03:06:43.466] Nov 5 03:06:43.465: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 03:06:43.466] Nov 5 03:06:43.465: INFO: kube-scheduler-k8s-master-15826033-2 started at 2019-11-05 01:33:54 +0000 UTC (0+1 container statuses recorded) I1105 03:06:43.466] Nov 5 03:06:43.465: INFO: Container kube-scheduler ready: true, restart count 0 I1105 03:06:43.466] Nov 5 03:06:43.465: INFO: azure-ip-masq-agent-hw7s5 started at 2019-11-05 01:34:34 +0000 UTC (0+1 container statuses recorded) I1105 03:06:43.467] Nov 5 03:06:43.465: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 03:06:43.467] Nov 5 03:06:43.465: INFO: kube-proxy-5xpp6 started at 2019-11-05 01:34:34 +0000 UTC (0+1 container statuses recorded) I1105 03:06:43.467] Nov 5 03:06:43.465: INFO: Container kube-proxy ready: true, restart count 0 I1105 03:06:43.467] Nov 5 03:06:43.465: INFO: cloud-controller-manager-k8s-master-15826033-2 started at 2019-11-05 01:33:54 +0000 UTC (0+1 container statuses recorded) I1105 03:06:43.467] Nov 5 03:06:43.465: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 03:06:43.468] Nov 5 03:06:43.465: INFO: kube-addon-manager-k8s-master-15826033-2 started at 2019-11-05 01:33:54 +0000 UTC (0+1 container statuses recorded) I1105 03:06:43.468] Nov 5 03:06:43.465: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 03:06:43.727] Nov 5 03:06:43.726: INFO: I1105 03:06:46.415] Latency metrics for node k8s-master-15826033-2 I1105 03:06:46.415] Nov 5 03:06:43.726: INFO: I1105 03:06:46.417] Logging node info for node k8s-master-15826033-3 I1105 03:06:46.420] Nov 5 03:06:43.804: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-15826033-3,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-15826033-3,UID:b7702272-1876-434e-b6a0-17fa34c6757f,ResourceVersion:17579,Generation:0,CreationTimestamp:2019-11-05 01:35:06 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 0,kubernetes.azure.com/cluster: kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-15826033-3,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.3.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5/providers/Microsoft.Compute/virtualMachines/k8s-master-15826033-3,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 01:35:17 +0000 UTC 2019-11-05 01:35:17 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 03:06:37 +0000 UTC 2019-11-05 01:35:00 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 03:06:37 +0000 UTC 2019-11-05 01:35:00 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 03:06:37 +0000 UTC 2019-11-05 01:35:00 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 03:06:37 +0000 UTC 2019-11-05 01:35:06 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.255.8} {Hostname k8s-master-15826033-3}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:fbf366963961422c9d90bedf6639c8e1,SystemUUID:5388948D-1A8D-EA41-B8B8-7F4523EDF5AA,BootID:42f5884c-7418-486a-8cf3-75188a331f38,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 03:06:46.421] Nov 5 03:06:43.804: INFO: I1105 03:06:46.421] Logging kubelet events for node k8s-master-15826033-3 I1105 03:06:46.421] Nov 5 03:06:43.884: INFO: I1105 03:06:46.422] Logging pods the kubelet thinks is on node k8s-master-15826033-3 I1105 03:06:46.422] Nov 5 03:06:43.966: INFO: cloud-controller-manager-k8s-master-15826033-3 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 03:06:46.422] Nov 5 03:06:43.966: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 03:06:46.423] Nov 5 03:06:43.966: INFO: kube-addon-manager-k8s-master-15826033-3 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 03:06:46.423] Nov 5 03:06:43.966: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 03:06:46.423] Nov 5 03:06:43.966: INFO: kube-apiserver-k8s-master-15826033-3 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 03:06:46.423] Nov 5 03:06:43.966: INFO: Container kube-apiserver ready: true, restart count 0 I1105 03:06:46.424] Nov 5 03:06:43.966: INFO: kube-controller-manager-k8s-master-15826033-3 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 03:06:46.424] Nov 5 03:06:43.966: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 03:06:46.424] Nov 5 03:06:43.966: INFO: kube-scheduler-k8s-master-15826033-3 started at 2019-11-05 01:34:54 +0000 UTC (0+1 container statuses recorded) I1105 03:06:46.425] Nov 5 03:06:43.966: INFO: Container kube-scheduler ready: true, restart count 0 I1105 03:06:46.425] Nov 5 03:06:43.966: INFO: kube-proxy-xz24z started at 2019-11-05 01:35:06 +0000 UTC (0+1 container statuses recorded) I1105 03:06:46.425] Nov 5 03:06:43.966: INFO: Container kube-proxy ready: true, restart count 0 I1105 03:06:46.425] Nov 5 03:06:43.966: INFO: azure-ip-masq-agent-dqfpw started at 2019-11-05 01:35:06 +0000 UTC (0+1 container statuses recorded) I1105 03:06:46.426] Nov 5 03:06:43.966: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 03:06:46.426] Nov 5 03:06:44.221: INFO: I1105 03:06:46.426] Latency metrics for node k8s-master-15826033-3 I1105 03:06:46.426] Nov 5 03:06:44.221: INFO: I1105 03:06:46.427] Logging node info for node k8s-master-15826033-4 I1105 03:06:46.430] Nov 5 03:06:44.298: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-15826033-4,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-15826033-4,UID:276daef1-6c05-41e3-bacd-abc93d35cc6d,ResourceVersion:17588,Generation:0,CreationTimestamp:2019-11-05 01:35:07 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 0,kubernetes.azure.com/cluster: kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-15826033-4,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.6.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5/providers/Microsoft.Compute/virtualMachines/k8s-master-15826033-4,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 01:35:41 +0000 UTC 2019-11-05 01:35:41 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 03:06:39 +0000 UTC 2019-11-05 01:35:02 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 03:06:39 +0000 UTC 2019-11-05 01:35:02 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 03:06:39 +0000 UTC 2019-11-05 01:35:02 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 03:06:39 +0000 UTC 2019-11-05 01:35:08 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.255.9} {Hostname k8s-master-15826033-4}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:79051fc0bdeb4e32a56acbaf5972ce3a,SystemUUID:C71606B5-7F08-9341-A26D-8A8D9D84FCAE,BootID:4bdf7cf3-2b83-42c7-a7ba-926ebe9ca75b,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 03:06:46.430] Nov 5 03:06:44.298: INFO: I1105 03:06:46.430] Logging kubelet events for node k8s-master-15826033-4 I1105 03:06:46.430] Nov 5 03:06:44.376: INFO: I1105 03:06:46.430] Logging pods the kubelet thinks is on node k8s-master-15826033-4 I1105 03:06:46.431] Nov 5 03:06:44.458: INFO: kube-controller-manager-k8s-master-15826033-4 started at 2019-11-05 01:34:55 +0000 UTC (0+1 container statuses recorded) I1105 03:06:46.431] Nov 5 03:06:44.458: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 03:06:46.431] Nov 5 03:06:44.458: INFO: kube-scheduler-k8s-master-15826033-4 started at 2019-11-05 01:34:55 +0000 UTC (0+1 container statuses recorded) I1105 03:06:46.431] Nov 5 03:06:44.458: INFO: Container kube-scheduler ready: true, restart count 0 I1105 03:06:46.432] Nov 5 03:06:44.458: INFO: cloud-controller-manager-k8s-master-15826033-4 started at 2019-11-05 01:34:56 +0000 UTC (0+1 container statuses recorded) I1105 03:06:46.432] Nov 5 03:06:44.458: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 03:06:46.432] Nov 5 03:06:44.458: INFO: kube-addon-manager-k8s-master-15826033-4 started at 2019-11-05 01:34:56 +0000 UTC (0+1 container statuses recorded) I1105 03:06:46.433] Nov 5 03:06:44.458: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 03:06:46.433] Nov 5 03:06:44.458: INFO: azure-ip-masq-agent-9nrj8 started at 2019-11-05 01:35:11 +0000 UTC (0+1 container statuses recorded) I1105 03:06:46.433] Nov 5 03:06:44.458: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 03:06:46.433] Nov 5 03:06:44.458: INFO: kube-proxy-rmngk started at 2019-11-05 01:35:11 +0000 UTC (0+1 container statuses recorded) I1105 03:06:46.434] Nov 5 03:06:44.458: INFO: Container kube-proxy ready: true, restart count 0 I1105 03:06:46.434] Nov 5 03:06:44.458: INFO: kube-apiserver-k8s-master-15826033-4 started at 2019-11-05 01:34:55 +0000 UTC (0+1 container statuses recorded) I1105 03:06:46.434] Nov 5 03:06:44.458: INFO: Container kube-apiserver ready: true, restart count 0 I1105 03:06:46.628] Nov 5 03:06:46.627: INFO: I1105 03:06:46.766] Latency metrics for node k8s-master-15826033-4 I1105 03:06:46.766] Nov 5 03:06:46.628: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready I1105 03:06:46.766] STEP: Destroying namespace "azuredisk-6855" for this suite. I1105 03:07:00.945] Nov 5 03:07:00.945: INFO: Waiting up to 30s for server preferred namespaced resources to be successfully discovered I1105 03:07:03.941] Nov 5 03:07:03.941: INFO: namespace azuredisk-6855 deletion completed in 17.234891021s I1105 03:07:03.941] I1105 03:07:03.942] • Failure [327.642 seconds] I1105 03:07:03.942] [azuredisk-csi-e2e] Dynamic Provisioning I1105 03:07:03.942] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:33 I1105 03:07:03.942] [multi-az] I1105 03:07:03.942] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:40 I1105 03:07:03.943] should delete PV with reclaimPolicy "Delete" [It] I1105 03:07:03.943] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:221 I1105 03:07:03.943] I1105 03:07:03.943] Unexpected error: I1105 03:07:03.944] <*errors.errorString | 0xc00073c9a0>: { I1105 03:07:03.944] s: "PersistentVolumeClaims [pvc-gtw85] not all in phase Bound within 5m0s", I1105 03:07:03.944] } I1105 03:07:03.944] PersistentVolumeClaims [pvc-gtw85] not all in phase Bound within 5m0s I1105 03:07:03.945] occurred I1105 03:07:03.945] I1105 03:07:03.945] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/testsuites/testsuites.go:278 I1105 03:07:03.945] ------------------------------ I1105 03:07:03.945] 2019/11/05 03:07:03 Uninstalling Azure Disk CSI Driver... I1105 03:07:03.946] make[1]: Entering directory '/go/src/sigs.k8s.io/azuredisk-csi-driver' I1105 03:07:03.946] helm delete --purge azuredisk-csi-driver I1105 03:07:12.258] release "azuredisk-csi-driver" deleted I1105 03:07:13.859] make[1]: Leaving directory '/go/src/sigs.k8s.io/azuredisk-csi-driver' I1105 03:07:13.859] 2019/11/05 03:07:12 Azure Disk CSI Driver uninstalled I1105 03:07:13.861] I1105 03:07:13.861] I1105 03:07:13.862] Summarizing 16 Failures: I1105 03:07:13.862] I1105 03:07:13.862] [Fail] [azuredisk-csi-e2e] Dynamic Provisioning [single-az] [It] should create a volume on demand and mount it as readOnly in a pod I1105 03:07:13.863] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/testsuites/testsuites.go:278 I1105 03:07:13.863] I1105 03:07:13.863] [Panic!] [azuredisk-csi-e2e] [single-az] Pre-Provisioned [It] [env] should use a pre-provisioned volume and retain PV with reclaimPolicy "Retain" I1105 03:07:13.863] /usr/local/go/src/runtime/panic.go:82 I1105 03:07:13.864] I1105 03:07:13.864] [Fail] [azuredisk-csi-e2e] Dynamic Provisioning [single-az] [It] should create a volume on demand I1105 03:07:13.864] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/testsuites/testsuites.go:278 I1105 03:07:13.865] I1105 03:07:13.865] [Fail] [azuredisk-csi-e2e] [single-az] Pre-Provisioned [It] [env] should use a pre-provisioned volume and mount it as readOnly in a pod I1105 03:07:13.865] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/testsuites/testsuites.go:353 I1105 03:07:13.866] I1105 03:07:13.866] [Fail] [azuredisk-csi-e2e] Dynamic Provisioning [multi-az] [It] should create a volume on demand and mount it as readOnly in a pod I1105 03:07:13.867] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/testsuites/testsuites.go:278 I1105 03:07:13.867] I1105 03:07:13.867] [Fail] [azuredisk-csi-e2e] Dynamic Provisioning [single-az] [It] should create multiple PV objects, bind to PVCs and attach all to different pods on the same node I1105 03:07:13.868] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/testsuites/testsuites.go:278 I1105 03:07:13.868] I1105 03:07:13.868] [Fail] [azuredisk-csi-e2e] Dynamic Provisioning [single-az] [It] should create a raw block volume on demand I1105 03:07:13.869] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/testsuites/testsuites.go:278 I1105 03:07:13.869] I1105 03:07:13.869] [Fail] [azuredisk-csi-e2e] Dynamic Provisioning [single-az] [It] should create a deployment object, write and read to it, delete the pod and write and read to it again I1105 03:07:13.870] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/testsuites/testsuites.go:278 I1105 03:07:13.870] I1105 03:07:13.870] [Fail] [azuredisk-csi-e2e] Dynamic Provisioning [multi-az] [It] [env] should retain PV with reclaimPolicy "Retain" I1105 03:07:13.871] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/testsuites/testsuites.go:278 I1105 03:07:13.871] I1105 03:07:13.872] [Fail] [azuredisk-csi-e2e] Dynamic Provisioning [multi-az] [It] should create a deployment object, write and read to it, delete the pod and write and read to it again I1105 03:07:13.872] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/testsuites/testsuites.go:278 I1105 03:07:13.872] I1105 03:07:13.873] [Fail] [azuredisk-csi-e2e] Dynamic Provisioning [multi-az] [It] should create multiple PV objects, bind to PVCs and attach all to different pods on the same node I1105 03:07:13.873] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/testsuites/testsuites.go:278 I1105 03:07:13.874] I1105 03:07:13.874] [Fail] [azuredisk-csi-e2e] Dynamic Provisioning [single-az] [It] [env] should retain PV with reclaimPolicy "Retain" I1105 03:07:13.874] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/testsuites/testsuites.go:278 I1105 03:07:13.875] I1105 03:07:13.875] [Fail] [azuredisk-csi-e2e] Dynamic Provisioning [single-az] [It] should delete PV with reclaimPolicy "Delete" I1105 03:07:13.875] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/testsuites/testsuites.go:278 I1105 03:07:13.875] I1105 03:07:13.876] [Fail] [azuredisk-csi-e2e] Dynamic Provisioning [multi-az] [It] should create a volume on demand I1105 03:07:13.876] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/testsuites/testsuites.go:278 I1105 03:07:13.876] I1105 03:07:13.876] [Fail] [azuredisk-csi-e2e] Dynamic Provisioning [multi-az] [It] should create a raw block volume on demand I1105 03:07:13.877] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/testsuites/testsuites.go:278 I1105 03:07:13.877] I1105 03:07:13.877] [Fail] [azuredisk-csi-e2e] Dynamic Provisioning [multi-az] [It] should delete PV with reclaimPolicy "Delete" I1105 03:07:13.877] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/testsuites/testsuites.go:278 I1105 03:07:13.877] I1105 03:07:13.878] Ran 16 of 16 Specs in 5424.486 seconds I1105 03:07:13.878] FAIL! -- 0 Passed | 16 Failed | 0 Pending | 0 Skipped I1105 03:07:13.878] --- FAIL: TestE2E (5424.49s) I1105 03:07:13.878] FAIL I1105 03:07:13.878] FAIL sigs.k8s.io/azuredisk-csi-driver/test/e2e 5424.546s W1105 03:07:19.632] make: *** [Makefile:59: e2e-test] Error 1 W1105 03:07:19.632] 2019/11/05 03:07:19 process.go:155: Step 'make e2e-test' finished in 1h31m23.633275644s W1105 03:07:19.632] 2019/11/05 03:07:19 azure.go:912: Skippng DumpClusterLogs due to CCM not being enabled. W1105 03:07:19.633] 2019/11/05 03:07:19 azure.go:906: Deleting resource group: kubetest-4093c0d1-ff68-11e9-b04c-02425d6816e5. W1105 03:15:13.280] 2019/11/05 03:15:13 process.go:96: Saved XML output to /workspace/_artifacts/junit_runner.xml. W1105 03:15:13.282] 2019/11/05 03:15:13 process.go:153: Running: bash -c . hack/lib/version.sh && KUBE_ROOT=. kube::version::get_version_vars && echo "${KUBE_GIT_VERSION-}" W1105 03:15:14.011] 2019/11/05 03:15:14 process.go:155: Step 'bash -c . hack/lib/version.sh && KUBE_ROOT=. kube::version::get_version_vars && echo "${KUBE_GIT_VERSION-}"' finished in 729.894649ms W1105 03:15:14.011] 2019/11/05 03:15:14 main.go:319: Something went wrong: encountered 1 errors: [error during make e2e-test: exit status 2] W1105 03:15:14.013] Traceback (most recent call last): W1105 03:15:14.013] File "/workspace/./test-infra/jenkins/../scenarios/kubernetes_e2e.py", line 778, in W1105 03:15:14.014] main(parse_args()) W1105 03:15:14.014] File "/workspace/./test-infra/jenkins/../scenarios/kubernetes_e2e.py", line 626, in main W1105 03:15:14.014] mode.start(runner_args) W1105 03:15:14.015] File "/workspace/./test-infra/jenkins/../scenarios/kubernetes_e2e.py", line 262, in start W1105 03:15:14.015] check_env(env, self.command, *args) W1105 03:15:14.015] File "/workspace/./test-infra/jenkins/../scenarios/kubernetes_e2e.py", line 111, in check_env W1105 03:15:14.015] subprocess.check_call(cmd, env=env) W1105 03:15:14.015] File "/usr/lib/python2.7/subprocess.py", line 190, in check_call W1105 03:15:14.015] raise CalledProcessError(retcode, cmd) W1105 03:15:14.016] subprocess.CalledProcessError: Command '('kubetest', '--dump=/workspace/_artifacts', '--gcp-service-account=/etc/service-account/service-account.json', '--build=bazel', '--up', '--down', '--test', '--deployment=acsengine', '--provider=skeleton', '--cluster=bootstrap-e2e', '--gcp-network=bootstrap-e2e', '--ginkgo-parallel=1', '--acsengine-agentpoolcount=2', '--acsengine-admin-username=azureuser', '--acsengine-creds=/etc/azure-cred/credentials', '--acsengine-orchestratorRelease=1.16', '--acsengine-mastervmsize=Standard_DS2_v2', '--acsengine-agentvmsize=Standard_D4s_v3', '--acsengine-location=eastus2', '--acsengine-public-key=/etc/azure-ssh/azure-ssh-pub', '--acsengine-template-url=https://raw.githubusercontent.com/kubernetes/cloud-provider-azure/master/tests/k8s-azure/manifest/linux-vmss-multi-zones.json', '--acsengine-download-url=https://github.com/Azure/aks-engine/releases/download/v0.39.1/aks-engine-v0.39.1-linux-amd64.tar.gz', '--test-azure-disk-csi-driver=True', '--test_args=--ginkgo.focus=\\[multi-az\\]', '--timeout=420m')' returned non-zero exit status 1 E1105 03:15:14.022] Command failed I1105 03:15:14.022] process 685 exited with code 1 after 130.5m E1105 03:15:14.023] FAIL: chewong-pull-azuredisk-csi-driver-e2e-multi-az I1105 03:15:14.023] Call: gcloud auth activate-service-account --key-file=/etc/service-account/service-account.json W1105 03:15:15.011] Activated service account credentials for: [kubernetes-upstream@kubernetes-upstream.iam.gserviceaccount.com] I1105 03:15:15.044] process 19545 exited with code 0 after 0.0m I1105 03:15:15.044] Call: gcloud config get-value account I1105 03:15:15.298] process 19557 exited with code 0 after 0.0m I1105 03:15:15.298] Will upload results to gs://kubernetes-upstream/pr-logs using kubernetes-upstream@kubernetes-upstream.iam.gserviceaccount.com I1105 03:15:15.298] Upload result and artifacts... I1105 03:15:15.299] Gubernator results at https://gubernator.k8s.io/build/kubernetes-upstream/pr-logs/pull/sigs.k8s.io_azuredisk-csi-driver/180/chewong-pull-azuredisk-csi-driver-e2e-multi-az/1191521311436836864 I1105 03:15:15.299] Call: gsutil ls gs://kubernetes-upstream/pr-logs/pull/sigs.k8s.io_azuredisk-csi-driver/180/chewong-pull-azuredisk-csi-driver-e2e-multi-az/1191521311436836864/artifacts W1105 03:15:16.484] CommandException: One or more URLs matched no objects. E1105 03:15:16.559] Command failed I1105 03:15:16.559] process 19569 exited with code 1 after 0.0m W1105 03:15:16.560] Remote dir gs://kubernetes-upstream/pr-logs/pull/sigs.k8s.io_azuredisk-csi-driver/180/chewong-pull-azuredisk-csi-driver-e2e-multi-az/1191521311436836864/artifacts not exist yet I1105 03:15:16.560] Call: gsutil -m -q -o GSUtil:use_magicfile=True cp -r -c -z log,txt,xml /workspace/_artifacts gs://kubernetes-upstream/pr-logs/pull/sigs.k8s.io_azuredisk-csi-driver/180/chewong-pull-azuredisk-csi-driver-e2e-multi-az/1191521311436836864/artifacts I1105 03:15:18.072] process 19713 exited with code 0 after 0.0m I1105 03:15:18.072] Call: git rev-parse HEAD I1105 03:15:18.076] process 20242 exited with code 0 after 0.0m I1105 03:15:18.076] Call: git rev-parse HEAD I1105 03:15:18.080] process 20243 exited with code 0 after 0.0m I1105 03:15:18.080] Call: gsutil stat gs://kubernetes-upstream/pr-logs/directory/chewong-pull-azuredisk-csi-driver-e2e-multi-az/jobResultsCache.json I1105 03:15:18.904] process 20244 exited with code 0 after 0.0m I1105 03:15:18.905] Call: gsutil -q cat 'gs://kubernetes-upstream/pr-logs/directory/chewong-pull-azuredisk-csi-driver-e2e-multi-az/jobResultsCache.json#1572914852359342' I1105 03:15:20.010] process 20388 exited with code 0 after 0.0m I1105 03:15:20.010] Call: gsutil -q -h Content-Type:application/json -h x-goog-if-generation-match:1572914852359342 cp /tmp/gsutil_Iw8WSy gs://kubernetes-upstream/pr-logs/directory/chewong-pull-azuredisk-csi-driver-e2e-multi-az/jobResultsCache.json I1105 03:15:21.236] process 20534 exited with code 0 after 0.0m I1105 03:15:21.236] Call: gsutil stat gs://kubernetes-upstream/pr-logs/pull/sigs.k8s.io_azuredisk-csi-driver/180/chewong-pull-azuredisk-csi-driver-e2e-multi-az/jobResultsCache.json I1105 03:15:22.049] process 20714 exited with code 0 after 0.0m I1105 03:15:22.050] Call: gsutil -q cat 'gs://kubernetes-upstream/pr-logs/pull/sigs.k8s.io_azuredisk-csi-driver/180/chewong-pull-azuredisk-csi-driver-e2e-multi-az/jobResultsCache.json#1572914855620853' I1105 03:15:23.084] process 20858 exited with code 0 after 0.0m I1105 03:15:23.085] Call: gsutil -q -h Content-Type:application/json -h x-goog-if-generation-match:1572914855620853 cp /tmp/gsutil_C3yg5x gs://kubernetes-upstream/pr-logs/pull/sigs.k8s.io_azuredisk-csi-driver/180/chewong-pull-azuredisk-csi-driver-e2e-multi-az/jobResultsCache.json I1105 03:15:24.591] process 21004 exited with code 0 after 0.0m I1105 03:15:24.591] Call: gsutil -q -h Content-Type:application/json cp /tmp/gsutil_rGPA7O gs://kubernetes-upstream/pr-logs/pull/sigs.k8s.io_azuredisk-csi-driver/180/chewong-pull-azuredisk-csi-driver-e2e-multi-az/1191521311436836864/finished.json I1105 03:15:25.911] process 21184 exited with code 0 after 0.0m I1105 03:15:25.912] Call: gsutil -q -h Content-Type:text/plain -h 'Cache-Control:private, max-age=0, no-transform' cp /tmp/gsutil_YEYtvV gs://kubernetes-upstream/pr-logs/pull/sigs.k8s.io_azuredisk-csi-driver/180/chewong-pull-azuredisk-csi-driver-e2e-multi-az/latest-build.txt I1105 03:15:27.187] process 21364 exited with code 0 after 0.0m I1105 03:15:27.188] Call: gsutil -q -h Content-Type:text/plain -h 'Cache-Control:private, max-age=0, no-transform' cp /tmp/gsutil_KDT0un gs://kubernetes-upstream/pr-logs/directory/chewong-pull-azuredisk-csi-driver-e2e-multi-az/latest-build.txt I1105 03:15:28.398] process 21544 exited with code 0 after 0.0m