W1105 19:36:28.117] ************************************************************************** bootstrap.py is deprecated! test-infra oncall does not support any job still using bootstrap.py. Please migrate your job to podutils! https://github.com/kubernetes/test-infra/blob/master/prow/pod-utilities.md ************************************************************************** I1105 19:36:28.117] Args: --job=chewong-pull-azuredisk-csi-driver-e2e-multi-az --service-account=/etc/service-account/service-account.json --upload=gs://kubernetes-jenkins/logs --job=chewong-pull-azuredisk-csi-driver-e2e-multi-az --root=/go/src --service-account=/etc/service-account/service-account.json --repo=k8s.io/kubernetes=master --repo=sigs.k8s.io/azuredisk-csi-driver=master:b7c03af8eae7388e84115ace30cf2ff3285b0dc9,180:dbf2ba05476df3f4c50de82e8d2f2a5264329cb5 --upload=gs://kubernetes-upstream/pr-logs --timeout=460 --scenario=kubernetes_e2e -- --gce-ssh= --test=true --up=true --down=true --deployment=acsengine --build=bazel --provider=skeleton --ginkgo-parallel=1 --acsengine-agentpoolcount=2 --acsengine-admin-username=azureuser '--acsengine-creds=$AZURE_CREDENTIALS' --acsengine-orchestratorRelease=1.16 --acsengine-mastervmsize=Standard_DS2_v2 --acsengine-agentvmsize=Standard_D4s_v3 --acsengine-location=eastus2 '--acsengine-public-key=$AZURE_SSH_PUBLIC_KEY_FILE' --acsengine-template-url=https://raw.githubusercontent.com/kubernetes-sigs/cloud-provider-azure/master/tests/k8s-azure/manifest/linux-vmss-multi-zones.json --acsengine-download-url=https://github.com/Azure/aks-engine/releases/download/v0.39.1/aks-engine-v0.39.1-linux-amd64.tar.gz --test-azure-disk-csi-driver=True '--test_args=--ginkgo.focus=\[multi-az\]' --timeout=420m I1105 19:36:28.117] Bootstrap chewong-pull-azuredisk-csi-driver-e2e-multi-az... I1105 19:36:28.120] Builder: 7c54201a-0003-11ea-918b-16773c0c7aed I1105 19:36:28.120] Image: gcr.io/k8s-testimages/kubekins-e2e:v20191103-6816af1-master I1105 19:36:28.121] Gubernator results at https://gubernator.k8s.io/build/kubernetes-upstream/pr-logs/pull/sigs.k8s.io_azuredisk-csi-driver/180/chewong-pull-azuredisk-csi-driver-e2e-multi-az/1191801407074734080 I1105 19:36:28.121] Call: gcloud auth activate-service-account --key-file=/etc/service-account/service-account.json W1105 19:36:28.624] Activated service account credentials for: [kubernetes-upstream@kubernetes-upstream.iam.gserviceaccount.com] I1105 19:36:28.653] process 193 exited with code 0 after 0.0m I1105 19:36:28.654] Call: gcloud config get-value account I1105 19:36:28.891] process 205 exited with code 0 after 0.0m I1105 19:36:28.892] Will upload results to gs://kubernetes-upstream/pr-logs using kubernetes-upstream@kubernetes-upstream.iam.gserviceaccount.com I1105 19:36:28.892] Root: /go/src I1105 19:36:28.892] cd to /go/src I1105 19:36:28.892] Checkout: /go/src/k8s.io/kubernetes master to /go/src/k8s.io/kubernetes I1105 19:36:28.892] Call: git init k8s.io/kubernetes I1105 19:36:28.896] Initialized empty Git repository in /go/src/k8s.io/kubernetes/.git/ I1105 19:36:28.896] process 217 exited with code 0 after 0.0m I1105 19:36:28.896] Call: git config --local user.name 'K8S Bootstrap' I1105 19:36:28.899] process 218 exited with code 0 after 0.0m I1105 19:36:28.899] Call: git config --local user.email k8s_bootstrap@localhost I1105 19:36:28.901] process 219 exited with code 0 after 0.0m I1105 19:36:28.902] Call: git fetch --quiet --tags https://github.com/kubernetes/kubernetes master I1105 19:37:28.485] process 220 exited with code 0 after 1.0m I1105 19:37:28.486] Call: git checkout -B test FETCH_HEAD W1105 19:37:30.577] Switched to a new branch 'test' I1105 19:37:30.585] process 231 exited with code 0 after 0.0m I1105 19:37:30.586] Call: git show -s --format=format:%ct HEAD I1105 19:37:30.589] process 232 exited with code 0 after 0.0m I1105 19:37:30.589] Checkout: /go/src/sigs.k8s.io/azuredisk-csi-driver master:b7c03af8eae7388e84115ace30cf2ff3285b0dc9,180:dbf2ba05476df3f4c50de82e8d2f2a5264329cb5 to /go/src/sigs.k8s.io/azuredisk-csi-driver I1105 19:37:30.589] Call: git init sigs.k8s.io/azuredisk-csi-driver I1105 19:37:30.593] Initialized empty Git repository in /go/src/sigs.k8s.io/azuredisk-csi-driver/.git/ I1105 19:37:30.593] process 233 exited with code 0 after 0.0m I1105 19:37:30.593] Call: git config --local user.name 'K8S Bootstrap' I1105 19:37:30.596] process 234 exited with code 0 after 0.0m I1105 19:37:30.597] Call: git config --local user.email k8s_bootstrap@localhost I1105 19:37:30.599] process 235 exited with code 0 after 0.0m I1105 19:37:30.600] Call: git fetch --quiet --tags https://github.com/kubernetes-sigs/azuredisk-csi-driver master +refs/pull/180/head:refs/pr/180 I1105 19:37:37.534] process 236 exited with code 0 after 0.1m I1105 19:37:37.535] Call: git checkout -B test b7c03af8eae7388e84115ace30cf2ff3285b0dc9 W1105 19:37:38.110] Switched to a new branch 'test' I1105 19:37:38.113] process 247 exited with code 0 after 0.0m I1105 19:37:38.113] Call: git show -s --format=format:%ct HEAD I1105 19:37:38.116] process 248 exited with code 0 after 0.0m I1105 19:37:38.117] Call: git merge --no-ff -m 'Merge +refs/pull/180/head:refs/pr/180' dbf2ba05476df3f4c50de82e8d2f2a5264329cb5 I1105 19:37:38.260] Merge made by the 'recursive' strategy. I1105 19:37:38.263] .../templates/csi-azuredisk-controller.yaml | 5 +- I1105 19:37:38.264] deploy/csi-azuredisk-controller.yaml | 3 +- I1105 19:37:38.264] .../storageclass-azuredisk-csi-topology.yaml | 17 +++ I1105 19:37:38.264] pkg/azuredisk/azuredisk.go | 1 + I1105 19:37:38.264] pkg/azuredisk/identityserver.go | 7 ++ I1105 19:37:38.264] pkg/azuredisk/nodeserver.go | 23 +++- I1105 19:37:38.264] test/e2e/driver/azuredisk_csi_driver.go | 19 ++- I1105 19:37:38.264] test/e2e/dynamic_provisioning.go | 134 +++++++++++++++------ I1105 19:37:38.264] test/e2e/testsuites/testsuites.go | 8 ++ I1105 19:37:38.265] 9 files changed, 170 insertions(+), 47 deletions(-) I1105 19:37:38.265] create mode 100644 deploy/example/storageclass-azuredisk-csi-topology.yaml I1105 19:37:38.265] process 249 exited with code 0 after 0.0m I1105 19:37:38.265] Configure environment... I1105 19:37:38.265] Call: git show -s --format=format:%ct HEAD I1105 19:37:38.268] process 265 exited with code 0 after 0.0m I1105 19:37:38.269] Call: gcloud auth activate-service-account --key-file=/etc/service-account/service-account.json W1105 19:37:38.868] Activated service account credentials for: [kubernetes-upstream@kubernetes-upstream.iam.gserviceaccount.com] I1105 19:37:38.974] process 266 exited with code 0 after 0.0m I1105 19:37:38.974] Call: gcloud config get-value account I1105 19:37:39.209] process 278 exited with code 0 after 0.0m I1105 19:37:39.209] Will upload results to gs://kubernetes-upstream/pr-logs using kubernetes-upstream@kubernetes-upstream.iam.gserviceaccount.com I1105 19:37:39.209] Call: bash -c ' set -o errexit set -o nounset export KUBE_ROOT=. source hack/lib/version.sh kube::version::get_version_vars echo $KUBE_GIT_VERSION ' I1105 19:37:39.794] process 290 exited with code 0 after 0.0m I1105 19:37:39.794] Start 1191801407074734080 at v1.18.0-alpha.0.267+cbe7c6e3bea120... I1105 19:37:39.796] Call: gsutil -q -h Content-Type:application/json cp /tmp/gsutil_w6mFWO gs://kubernetes-upstream/pr-logs/pull/sigs.k8s.io_azuredisk-csi-driver/180/chewong-pull-azuredisk-csi-driver-e2e-multi-az/1191801407074734080/started.json I1105 19:37:41.420] process 323 exited with code 0 after 0.0m I1105 19:37:41.421] Call: gsutil -q -h Content-Type:text/plain -h 'x-goog-meta-link: gs://kubernetes-upstream/pr-logs/pull/sigs.k8s.io_azuredisk-csi-driver/180/chewong-pull-azuredisk-csi-driver-e2e-multi-az/1191801407074734080' cp /tmp/gsutil_T2LO4x gs://kubernetes-upstream/pr-logs/directory/chewong-pull-azuredisk-csi-driver-e2e-multi-az/1191801407074734080.txt I1105 19:37:42.564] process 503 exited with code 0 after 0.0m I1105 19:37:42.565] Call: /workspace/./test-infra/jenkins/../scenarios/kubernetes_e2e.py --gce-ssh= --test=true --up=true --down=true --deployment=acsengine --build=bazel --provider=skeleton --ginkgo-parallel=1 --acsengine-agentpoolcount=2 --acsengine-admin-username=azureuser --acsengine-creds=/etc/azure-cred/credentials --acsengine-orchestratorRelease=1.16 --acsengine-mastervmsize=Standard_DS2_v2 --acsengine-agentvmsize=Standard_D4s_v3 --acsengine-location=eastus2 --acsengine-public-key=/etc/azure-ssh/azure-ssh-pub --acsengine-template-url=https://raw.githubusercontent.com/kubernetes-sigs/cloud-provider-azure/master/tests/k8s-azure/manifest/linux-vmss-multi-zones.json --acsengine-download-url=https://github.com/Azure/aks-engine/releases/download/v0.39.1/aks-engine-v0.39.1-linux-amd64.tar.gz --test-azure-disk-csi-driver=True '--test_args=--ginkgo.focus=\[multi-az\]' --timeout=420m W1105 19:37:42.601] starts with local mode W1105 19:37:42.601] Environment: W1105 19:37:42.602] ARTIFACTS=/workspace/_artifacts W1105 19:37:42.602] AWS_SSH_PRIVATE_KEY_FILE=/root/.ssh/kube_aws_rsa W1105 19:37:42.602] AWS_SSH_PUBLIC_KEY_FILE=/root/.ssh/kube_aws_rsa.pub W1105 19:37:42.602] AZURE_CREDENTIALS=/etc/azure-cred/credentials W1105 19:37:42.602] AZURE_SSH_PUBLIC_KEY_FILE=/etc/azure-ssh/azure-ssh-pub W1105 19:37:42.602] AZ_STORAGE_CONTAINER_NAME=mystoragecontainer W1105 19:37:42.603] BAZEL_REMOTE_CACHE_ENABLED=false W1105 19:37:42.603] BAZEL_VERSION=0.23.2 W1105 19:37:42.603] BOOTSTRAP_MIGRATION=yes W1105 19:37:42.603] BUILD_ID=1191801407074734080 W1105 19:37:42.603] BUILD_NUMBER=1191801407074734080 W1105 19:37:42.603] CLOUDSDK_COMPONENT_MANAGER_DISABLE_UPDATE_CHECK=true W1105 19:37:42.603] CLOUDSDK_CONFIG=/go/src/k8s.io/kubernetes/.config/gcloud W1105 19:37:42.604] CLOUDSDK_CORE_DISABLE_PROMPTS=1 W1105 19:37:42.604] CLOUDSDK_EXPERIMENTAL_FAST_COMPONENT_UPDATE=false W1105 19:37:42.604] CLOUD_CONFIG=random W1105 19:37:42.604] DECK_PORT=tcp://10.0.160.83:80 W1105 19:37:42.604] DECK_PORT_80_TCP=tcp://10.0.160.83:80 W1105 19:37:42.604] DECK_PORT_80_TCP_ADDR=10.0.160.83 W1105 19:37:42.604] DECK_PORT_80_TCP_PORT=80 W1105 19:37:42.605] DECK_PORT_80_TCP_PROTO=tcp W1105 19:37:42.605] DECK_SERVICE_HOST=10.0.160.83 W1105 19:37:42.605] DECK_SERVICE_PORT=80 W1105 19:37:42.605] DOCKER_IN_DOCKER_ENABLED=true W1105 19:37:42.605] DOCKER_IN_DOCKER_IPV6_ENABLED=false W1105 19:37:42.605] GCS_ARTIFACTS_DIR=gs://kubernetes-upstream/pr-logs/pull/sigs.k8s.io_azuredisk-csi-driver/180/chewong-pull-azuredisk-csi-driver-e2e-multi-az/1191801407074734080/artifacts W1105 19:37:42.606] GOOGLE_APPLICATION_CREDENTIALS=/etc/service-account/service-account.json W1105 19:37:42.606] GOPATH=/go W1105 19:37:42.606] GO_TARBALL=go1.12.12.linux-amd64.tar.gz W1105 19:37:42.606] HOME=/workspace W1105 19:37:42.606] HOOK_PORT=tcp://10.0.144.133:8888 W1105 19:37:42.606] HOOK_PORT_8888_TCP=tcp://10.0.144.133:8888 W1105 19:37:42.606] HOOK_PORT_8888_TCP_ADDR=10.0.144.133 W1105 19:37:42.607] HOOK_PORT_8888_TCP_PORT=8888 W1105 19:37:42.607] HOOK_PORT_8888_TCP_PROTO=tcp W1105 19:37:42.607] HOOK_SERVICE_HOST=10.0.144.133 W1105 19:37:42.607] HOOK_SERVICE_PORT=8888 W1105 19:37:42.607] HOSTNAME=7c54201a-0003-11ea-918b-16773c0c7aed W1105 19:37:42.607] IMAGE=gcr.io/k8s-testimages/kubekins-e2e:v20191103-6816af1-master W1105 19:37:42.607] INSTANCE_PREFIX=bootstrap-e2e W1105 19:37:42.608] JENKINS_GCE_SSH_PRIVATE_KEY_FILE=/root/.ssh/google_compute_engine W1105 19:37:42.608] JENKINS_GCE_SSH_PUBLIC_KEY_FILE=/root/.ssh/google_compute_engine.pub W1105 19:37:42.608] JOB_NAME=chewong-pull-azuredisk-csi-driver-e2e-multi-az W1105 19:37:42.608] JOB_SPEC={"type":"presubmit","job":"chewong-pull-azuredisk-csi-driver-e2e-multi-az","buildid":"1191801407074734080","prowjobid":"7c54201a-0003-11ea-918b-16773c0c7aed","refs":{"org":"Azure","repo":"kubernetes","repo_link":"https://github.com/Azure/kubernetes","base_ref":"master","base_sha":"cbe7c6e3bea120a6828aae4215696f3b078e8dcb","base_link":"https://github.com/Azure/kubernetes/commit/cbe7c6e3bea120a6828aae4215696f3b078e8dcb","pulls":[{"number":113,"author":"chewong","sha":"879465972041635fd7bc8269da49153b67e63822","link":"https://github.com/Azure/kubernetes/pull/113","commit_link":"https://github.com/Azure/kubernetes/pull/113/commits/879465972041635fd7bc8269da49153b67e63822","author_link":"https://github.com/chewong"}]}} W1105 19:37:42.609] JOB_TYPE=presubmit W1105 19:37:42.609] KUBERNETES_PORT=tcp://10.0.0.1:443 W1105 19:37:42.609] KUBERNETES_PORT_443_TCP=tcp://10.0.0.1:443 W1105 19:37:42.609] KUBERNETES_PORT_443_TCP_ADDR=10.0.0.1 W1105 19:37:42.609] KUBERNETES_PORT_443_TCP_PORT=443 W1105 19:37:42.609] KUBERNETES_PORT_443_TCP_PROTO=tcp W1105 19:37:42.610] KUBERNETES_SERVICE_HOST=10.0.0.1 W1105 19:37:42.610] KUBERNETES_SERVICE_PORT=443 W1105 19:37:42.610] KUBERNETES_SERVICE_PORT_HTTPS=443 W1105 19:37:42.610] KUBETEST_IN_DOCKER=true W1105 19:37:42.610] KUBETEST_MANUAL_DUMP=y W1105 19:37:42.610] KUBE_AWS_INSTANCE_PREFIX=bootstrap-e2e W1105 19:37:42.610] KUBE_GCE_INSTANCE_PREFIX=bootstrap-e2e W1105 19:37:42.611] KUBE_SSH_KEY_PATH=/etc/ssh-key-secret/ssh-private W1105 19:37:42.611] KUBE_SSH_PUBLIC_KEY_PATH=/etc/ssh-key-secret/ssh-public W1105 19:37:42.611] KUBE_SSH_USER=azureuser W1105 19:37:42.611] KUBE_VERBOSE=0 W1105 19:37:42.611] NODE_NAME=7c54201a-0003-11ea-918b-16773c0c7aed W1105 19:37:42.611] PATH=/go/bin:/go/bin:/usr/local/go/bin:/google-cloud-sdk/bin:/workspace:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin W1105 19:37:42.612] PROW_JOB_ID=7c54201a-0003-11ea-918b-16773c0c7aed W1105 19:37:42.612] PULL_BASE_REF=master W1105 19:37:42.612] PULL_BASE_SHA=cbe7c6e3bea120a6828aae4215696f3b078e8dcb W1105 19:37:42.612] PULL_NUMBER=113 W1105 19:37:42.612] PULL_PULL_SHA=879465972041635fd7bc8269da49153b67e63822 W1105 19:37:42.612] PULL_REFS=master:cbe7c6e3bea120a6828aae4215696f3b078e8dcb,113:879465972041635fd7bc8269da49153b67e63822 W1105 19:37:42.612] PWD=/workspace W1105 19:37:42.612] REGISTRY=k8sprowinternal.azurecr.io W1105 19:37:42.613] REPO_NAME=kubernetes W1105 19:37:42.613] REPO_OWNER=Azure W1105 19:37:42.613] SHLVL=2 W1105 19:37:42.613] SOURCE_DATE_EPOCH=1572972546 W1105 19:37:42.613] TERM=xterm W1105 19:37:42.613] TIDE_PORT=tcp://10.0.180.53:80 W1105 19:37:42.613] TIDE_PORT_80_TCP=tcp://10.0.180.53:80 W1105 19:37:42.613] TIDE_PORT_80_TCP_ADDR=10.0.180.53 W1105 19:37:42.613] TIDE_PORT_80_TCP_PORT=80 W1105 19:37:42.613] TIDE_PORT_80_TCP_PROTO=tcp W1105 19:37:42.614] TIDE_SERVICE_HOST=10.0.180.53 W1105 19:37:42.614] TIDE_SERVICE_PORT=80 W1105 19:37:42.614] WORKSPACE=/workspace W1105 19:37:42.614] _=./test-infra/jenkins/bootstrap.py W1105 19:37:42.614] Run: ('kubetest', '--dump=/workspace/_artifacts', '--gcp-service-account=/etc/service-account/service-account.json', '--build=bazel', '--up', '--down', '--test', '--deployment=acsengine', '--provider=skeleton', '--cluster=bootstrap-e2e', '--gcp-network=bootstrap-e2e', '--ginkgo-parallel=1', '--acsengine-agentpoolcount=2', '--acsengine-admin-username=azureuser', '--acsengine-creds=/etc/azure-cred/credentials', '--acsengine-orchestratorRelease=1.16', '--acsengine-mastervmsize=Standard_DS2_v2', '--acsengine-agentvmsize=Standard_D4s_v3', '--acsengine-location=eastus2', '--acsengine-public-key=/etc/azure-ssh/azure-ssh-pub', '--acsengine-template-url=https://raw.githubusercontent.com/kubernetes-sigs/cloud-provider-azure/master/tests/k8s-azure/manifest/linux-vmss-multi-zones.json', '--acsengine-download-url=https://github.com/Azure/aks-engine/releases/download/v0.39.1/aks-engine-v0.39.1-linux-amd64.tar.gz', '--test-azure-disk-csi-driver=True', '--test_args=--ginkgo.focus=\\[multi-az\\]', '--timeout=420m') W1105 19:37:42.617] 2019/11/05 19:37:42 main.go:332: Limiting testing to 7h0m0s W1105 19:37:42.618] 2019/11/05 19:37:42 azure.go:232: Reading credentials file /etc/azure-cred/credentials W1105 19:37:42.618] 2019/11/05 19:37:42 process.go:153: Running: make -C /go/src/k8s.io/kubernetes bazel-release W1105 19:37:42.693] Extracting Bazel installation... I1105 19:37:42.793] make: Entering directory '/go/src/k8s.io/kubernetes' W1105 19:37:51.288] Starting local Bazel server and connecting to it... W1105 19:37:52.953] Loading: W1105 19:37:52.956] Loading: 0 packages loaded W1105 19:37:53.958] Loading: 0 packages loaded W1105 19:37:55.534] Loading: 0 packages loaded W1105 19:37:56.959] Loading: 0 packages loaded W1105 19:37:58.893] Loading: 0 packages loaded W1105 19:37:59.960] Loading: 0 packages loaded W1105 19:37:59.960] currently loading: build/release-tars W1105 19:38:02.174] Loading: 0 packages loaded W1105 19:38:02.174] currently loading: build/release-tars W1105 19:38:02.509] Analyzing: target //build/release-tars:release-tars (1 packages loaded, 0 targets configured) W1105 19:38:03.965] Analyzing: target //build/release-tars:release-tars (247 packages loaded, 1200 targets configured) W1105 19:38:05.635] Analyzing: target //build/release-tars:release-tars (1078 packages loaded, 4251 targets configured) W1105 19:38:07.558] Analyzing: target //build/release-tars:release-tars (1821 packages loaded, 12492 targets configured) W1105 19:38:09.772] Analyzing: target //build/release-tars:release-tars (2754 packages loaded, 22570 targets configured) W1105 19:38:12.972] Analyzing: target //build/release-tars:release-tars (3059 packages loaded, 27338 targets configured) W1105 19:38:16.052] Analyzing: target //build/release-tars:release-tars (3097 packages loaded, 32485 targets configured) W1105 19:38:22.501] Analyzing: target //build/release-tars:release-tars (3104 packages loaded, 35458 targets configured) W1105 19:38:29.253] INFO: Analysed target //build/release-tars:release-tars (3104 packages loaded, 36821 targets configured). W1105 19:38:29.256] Building: checking cached actions W1105 19:38:29.257] INFO: Found 1 target... W1105 19:38:29.441] [0 / 2] [-----] BazelWorkspaceStatusAction stable-status.txt W1105 19:38:36.162] [86 / 4,426] GoStdlib external/io_bazel_rules_go/linux_amd64_pure_stripped/stdlib%/pkg; 3s linux-sandbox ... (8 actions, 7 running) W1105 19:38:42.780] [109 / 4,426] GoStdlib external/io_bazel_rules_go/linux_amd64_pure_stripped/stdlib%/pkg; 9s linux-sandbox ... (8 actions, 7 running) W1105 19:38:50.894] [129 / 4,426] GoStdlib external/io_bazel_rules_go/linux_amd64_pure_stripped/stdlib%/pkg; 18s linux-sandbox ... (8 actions running) W1105 19:38:59.606] [179 / 4,426] GoStdlib external/io_bazel_rules_go/linux_amd64_pure_stripped/stdlib%/pkg; 26s linux-sandbox ... (8 actions running) W1105 19:39:09.654] [235 / 4,426] GoStdlib external/io_bazel_rules_go/linux_amd64_pure_stripped/stdlib%/pkg; 36s linux-sandbox ... (8 actions running) W1105 19:39:21.179] [303 / 4,426] GoCompilePkg vendor/github.com/modern-go/reflect2/linux_amd64_pure_stripped/go_default_library%/k8s.io/kubernetes/vendor/github.com/modern-go/reflect2.a; 1s linux-sandbox ... (8 actions running) W1105 19:39:34.436] [401 / 4,426] GoCompilePkg vendor/github.com/globalsign/mgo/internal/json/linux_amd64_pure_stripped/go_default_library%/k8s.io/kubernetes/vendor/github.com/globalsign/mgo/internal/json.a; 2s linux-sandbox ... (8 actions running) W1105 19:39:49.679] [503 / 4,426] GoCompilePkg vendor/github.com/thecodeteam/goscaleio/linux_amd64_stripped/go_default_library%/k8s.io/kubernetes/vendor/github.com/thecodeteam/goscaleio.a; 2s linux-sandbox ... (8 actions running) W1105 19:40:08.464] [606 / 4,426] GoCompilePkg vendor/github.com/prometheus/procfs/linux_amd64_pure_stripped/go_default_library%/k8s.io/kubernetes/vendor/github.com/prometheus/procfs.a; 6s linux-sandbox ... (8 actions, 7 running) W1105 19:40:29.950] [754 / 4,426] GoCompilePkg vendor/github.com/vmware/govmomi/vim25/types/linux_amd64_pure_stripped/go_default_library%/k8s.io/kubernetes/vendor/github.com/vmware/govmomi/vim25/types.a; 8s linux-sandbox ... (8 actions, 6 running) W1105 19:40:29.950] [754 / 4,426] GoCompilePkg vendor/github.com/vmware/govmomi/vim25/types/linux_amd64_pure_stripped/go_default_library%/k8s.io/kubernetes/vendor/github.com/vmware/govmomi/vim25/types.a; 8s linux-sandbox ... (8 actions, 7 running) W1105 19:40:53.574] [916 / 4,426] GoCompilePkg vendor/github.com/aws/aws-sdk-go/service/ec2/linux_amd64_pure_stripped/go_default_library%/k8s.io/kubernetes/vendor/github.com/aws/aws-sdk-go/service/ec2.a; 5s linux-sandbox ... (8 actions, 7 running) W1105 19:41:20.860] [1,174 / 4,426] GoCompilePkg vendor/github.com/Azure/azure-sdk-for-go/services/network/mgmt/2019-06-01/network/linux_amd64_stripped/go_default_library%/k8s.io/kubernetes/vendor/github.com/Azure/azure-sdk-for-go/services/network/mgmt/2019-06-01/network.a; 14s linux-sandbox ... (8 actions, 7 running) W1105 19:41:52.178] [1,417 / 4,426] GoCompilePkg vendor/sigs.k8s.io/kustomize/pkg/patch/transformer/linux_amd64_stripped/go_default_library%/k8s.io/kubernetes/vendor/sigs.k8s.io/kustomize/pkg/patch/transformer.a; 3s linux-sandbox ... (8 actions, 7 running) W1105 19:42:28.967] [1,562 / 4,426] ; 7s linux-sandbox ... (8 actions, 7 running) W1105 19:43:12.468] [1,687 / 4,426] ; 32s linux-sandbox ... (8 actions, 7 running) W1105 19:44:00.515] [1,927 / 4,426] GoCompilePkg staging/src/k8s.io/client-go/listers/core/v1/linux_amd64_stripped/go_default_library%/k8s.io/kubernetes/vendor/k8s.io/client-go/listers/core/v1.a; 2s linux-sandbox ... (8 actions, 7 running) W1105 19:44:55.794] [2,303 / 4,426] GoCompilePkg pkg/generated/openapi/linux_amd64_stripped/go_default_library%/k8s.io/kubernetes/pkg/generated/openapi.a; 8s linux-sandbox ... (8 actions, 7 running) W1105 19:45:59.534] [2,612 / 4,426] GoCompilePkg staging/src/k8s.io/kubectl/pkg/cmd/util/linux_amd64_stripped/go_default_library%/k8s.io/kubernetes/vendor/k8s.io/kubectl/pkg/cmd/util.a; 3s linux-sandbox ... (8 actions, 7 running) W1105 19:47:12.945] [2,853 / 4,426] GoCompilePkg staging/src/k8s.io/apiserver/pkg/server/linux_amd64_stripped/go_default_library%/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server.a; 7s linux-sandbox ... (8 actions, 7 running) W1105 19:48:39.118] [3,182 / 4,426] GoLink cmd/kubelet/kubelet; 15s linux-sandbox ... (8 actions, 7 running) W1105 19:50:24.330] [3,316 / 4,426] GoLink cmd/genkubedocs/genkubedocs; 119s linux-sandbox ... (8 actions, 6 running) W1105 19:50:24.332] [3,316 / 4,426] GoLink cmd/genkubedocs/genkubedocs; 119s linux-sandbox ... (8 actions, 7 running) W1105 19:52:17.227] [3,564 / 4,426] GoLink test/e2e/linux_amd64_stripped/_go_default_test-cgo; 31s linux-sandbox ... (8 actions, 7 running) W1105 19:54:27.199] [4,109 / 4,429] GoCompilePkg staging/src/k8s.io/client-go/informers/core/linux_amd64_pure_stripped/go_default_library%/k8s.io/kubernetes/vendor/k8s.io/client-go/informers/core.a; 3s linux-sandbox ... (8 actions running) W1105 19:56:56.652] [4,419 / 4,429] GZIP build/kube-controller-manager-internal-layer.tar.gz; 5s linux-sandbox ... (2 actions running) W1105 19:58:30.626] Target //build/release-tars:release-tars up-to-date: W1105 19:58:30.627] bazel-bin/build/release-tars/kubernetes-client-linux-amd64.tar.gz W1105 19:58:30.627] bazel-bin/build/release-tars/kubernetes-node-linux-amd64.tar.gz W1105 19:58:30.630] bazel-bin/build/release-tars/kubernetes-server-linux-amd64.tar.gz W1105 19:58:30.631] bazel-bin/build/release-tars/kubernetes-manifests.tar.gz W1105 19:58:30.631] bazel-bin/build/release-tars/kubernetes-test-portable.tar.gz W1105 19:58:30.632] bazel-bin/build/release-tars/kubernetes-test-linux-amd64.tar.gz W1105 19:58:30.633] bazel-bin/build/release-tars/kubernetes.tar.gz W1105 19:58:30.633] bazel-bin/build/release-tars/kubernetes-src.tar.gz W1105 19:58:30.671] INFO: Elapsed time: 1247.982s, Critical Path: 360.39s W1105 19:58:30.672] INFO: 4364 processes: 4364 linux-sandbox. W1105 19:58:30.677] INFO: Build completed successfully, 4429 total actions W1105 19:58:30.680] INFO: Build completed successfully, 4429 total actions W1105 19:58:30.697] 2019/11/05 19:58:30 process.go:155: Step 'make -C /go/src/k8s.io/kubernetes bazel-release' finished in 20m48.076854646s W1105 19:58:30.703] 2019/11/05 19:58:30 util.go:277: Flushing memory. I1105 19:58:30.804] make: Leaving directory '/go/src/k8s.io/kubernetes' W1105 19:59:15.613] 2019/11/05 19:59:15 azure.go:906: Deleting resource group: kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b. W1105 19:59:16.300] 2019/11/05 19:59:16 azure.go:787: downloading /workspace/aks971561099/kubernetes.json from https://raw.githubusercontent.com/kubernetes-sigs/cloud-provider-azure/master/tests/k8s-azure/manifest/linux-vmss-multi-zones.json. W1105 19:59:16.300] 2019/11/05 19:59:16 util.go:42: curl https://raw.githubusercontent.com/kubernetes-sigs/cloud-provider-azure/master/tests/k8s-azure/manifest/linux-vmss-multi-zones.json W1105 19:59:16.698] 2019/11/05 19:59:16 azure.go:529: downloading /workspace/aks-engine.tar.gz from https://github.com/Azure/aks-engine/releases/download/v0.39.1/aks-engine-v0.39.1-linux-amd64.tar.gz. W1105 19:59:16.698] 2019/11/05 19:59:16 util.go:42: curl https://github.com/Azure/aks-engine/releases/download/v0.39.1/aks-engine-v0.39.1-linux-amd64.tar.gz W1105 19:59:18.004] 2019/11/05 19:59:18 azure.go:556: Extracting tar file /workspace/aks-engine.tar.gz into directory /go/src/k8s.io/kubernetes . W1105 19:59:18.004] 2019/11/05 19:59:18 process.go:153: Running: tar -xzf /workspace/aks-engine.tar.gz --strip 1 W1105 19:59:20.605] 2019/11/05 19:59:20 process.go:155: Step 'tar -xzf /workspace/aks-engine.tar.gz --strip 1' finished in 2.600806913s W1105 19:59:20.605] 2019/11/05 19:59:20 process.go:153: Running: /go/src/k8s.io/kubernetes/aks-engine generate /workspace/aks971561099/kubernetes.json --output-directory /workspace/aks971561099 I1105 19:59:21.578] INFO[0000] Generating assets into /workspace/aks971561099... W1105 19:59:39.648] 2019/11/05 19:59:39 process.go:155: Step '/go/src/k8s.io/kubernetes/aks-engine generate /workspace/aks971561099/kubernetes.json --output-directory /workspace/aks971561099' finished in 19.04311955s W1105 19:59:39.654] 2019/11/05 19:59:39 azure.go:627: Setting kubeconfig env variable: kubeconfig path: /workspace/aks971561099/kubeconfig/kubeconfig.eastus2.json. W1105 19:59:39.655] 2019/11/05 19:59:39 azure.go:629: Creating resource group: kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b. W1105 19:59:39.655] 2019/11/05 19:59:39 azure.go:631: Creating Azure resource group: kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b for cluster deployment. W1105 19:59:41.157] 2019/11/05 19:59:41 azure.go:636: Validating deployment ARM templates. W1105 19:59:43.556] 2019/11/05 19:59:43 azure.go:642: Deploying cluster kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b in resource group kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b. W1105 20:10:55.058] 2019/11/05 20:10:55 process.go:153: Running: ./cluster/kubectl.sh --match-server-version=false version W1105 20:10:56.466] 2019/11/05 20:10:56 process.go:155: Step './cluster/kubectl.sh --match-server-version=false version' finished in 1.405194078s W1105 20:10:56.466] 2019/11/05 20:10:56 process.go:153: Running: ./cluster/kubectl.sh --match-server-version=false get nodes -oyaml W1105 20:11:29.192] 2019/11/05 20:11:29 process.go:155: Step './cluster/kubectl.sh --match-server-version=false get nodes -oyaml' finished in 32.729228506s W1105 20:11:29.193] 2019/11/05 20:11:29 azure.go:970: KUBE_TEST_REPO_LIST_DOWNLOAD_LOCATION not set. Using default test image repos. W1105 20:11:29.193] 2019/11/05 20:11:29 azure.go:970: KUBE_TEST_REPO_LIST_DOWNLOAD_LOCATION not set. Using default test image repos. W1105 20:11:29.193] 2019/11/05 20:11:29 process.go:153: Running: kubectl get nodes --no-headers W1105 20:11:30.202] 2019/11/05 20:11:30 process.go:155: Step 'kubectl get nodes --no-headers' finished in 1.006301616s W1105 20:11:30.202] 2019/11/05 20:11:30 e2e.go:474: Cluster nodes: W1105 20:11:30.202] k8s-agentpool-97435445-vmss000000 Ready agent 3m56s v1.15.0 W1105 20:11:30.202] k8s-agentpool-97435445-vmss000001 Ready agent 3m56s v1.15.0 W1105 20:11:30.202] k8s-master-97435445-0 Ready master 2m54s v1.15.0 W1105 20:11:30.203] k8s-master-97435445-1 Ready master 3m56s v1.15.0 W1105 20:11:30.203] k8s-master-97435445-2 Ready master 2m11s v1.15.0 W1105 20:11:30.203] k8s-master-97435445-3 Ready master 2m9s v1.15.0 W1105 20:11:30.203] k8s-master-97435445-4 Ready master 2m54s v1.15.0 W1105 20:11:30.203] 2019/11/05 20:11:30 process.go:153: Running: ./cluster/kubectl.sh --match-server-version=false version W1105 20:11:37.211] 2019/11/05 20:11:37 process.go:155: Step './cluster/kubectl.sh --match-server-version=false version' finished in 7.01116976s W1105 20:11:37.212] 2019/11/05 20:11:37 process.go:153: Running: make e2e-test I1105 20:11:37.314] go test -v -timeout=0 ./test/e2e "-ginkgo.noColor" I1105 20:14:28.909] Nov 5 20:14:28.909: INFO: The --provider flag is not set. Continuing as if --provider=skeleton had been used. I1105 20:14:28.910] === RUN TestE2E I1105 20:14:28.910] Running Suite: AzureDisk CSI Driver End-to-End Tests I1105 20:14:28.911] ==================================================== I1105 20:14:28.911] Random Seed: 1572984868 - Will randomize all specs I1105 20:14:28.912] Will run 16 of 16 specs I1105 20:14:28.912] I1105 20:14:31.190] 2019/11/05 20:14:31 Attempting docker login with Azure service principal I1105 20:14:35.575] 2019/11/05 20:14:35 docker login is successful I1105 20:14:35.578] 2019/11/05 20:14:35 Installing Azure Disk CSI Driver... I1105 20:14:35.590] make[1]: Entering directory '/go/src/sigs.k8s.io/azuredisk-csi-driver' I1105 20:14:35.594] # Use v2.11.0 helm to match tiller's version in clusters made by aks-engine I1105 20:14:35.603] curl https://raw.githubusercontent.com/helm/helm/master/scripts/get | DESIRED_VERSION=v2.11.0 bash I1105 20:14:35.678] % Total % Received % Xferd Average Speed Time Time Time Current I1105 20:14:35.682] Dload Upload Total Spent Left Speed I1105 20:14:35.742] 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 7034 100 7034 0 0 112k 0 --:--:-- --:--:-- --:--:-- 112k I1105 20:14:35.766] Downloading https://get.helm.sh/helm-v2.11.0-linux-amd64.tar.gz I1105 20:14:37.541] Preparing to install helm and tiller into /usr/local/bin I1105 20:14:37.626] helm installed into /usr/local/bin/helm I1105 20:14:37.670] tiller installed into /usr/local/bin/tiller I1105 20:14:37.710] Run 'helm init' to configure helm. I1105 20:14:37.738] # Make sure tiller is ready I1105 20:14:37.738] kubectl wait pod -l name=tiller --namespace kube-system --for condition=ready --timeout 5m I1105 20:14:38.428] pod/tiller-deploy-54c96cb5df-pt7x6 condition met I1105 20:14:38.454] helm version I1105 20:14:38.584] Client: &version.Version{SemVer:"v2.11.0", GitCommit:"2e55dbe1fdb5fdb96b75ff144a339489417b146b", GitTreeState:"clean"} I1105 20:14:39.559] Server: &version.Version{SemVer:"v2.11.0", GitCommit:"2e55dbe1fdb5fdb96b75ff144a339489417b146b", GitTreeState:"clean"} I1105 20:14:39.565] # Only build and push the image if it does not exist in the registry I1105 20:14:39.565] docker pull k8sprowinternal.azurecr.io/azuredisk-csi:e2e-3e3ed045c6faa509dbd8d2890ee9f56aa2934887 || make azuredisk-container push I1105 20:14:40.036] e2e-3e3ed045c6faa509dbd8d2890ee9f56aa2934887: Pulling from azuredisk-csi I1105 20:14:40.036] 5667fdb72017: Pulling fs layer I1105 20:14:40.036] d83811f270d5: Pulling fs layer I1105 20:14:40.036] ee671aafb583: Pulling fs layer I1105 20:14:40.037] 7fc152dfb3a6: Pulling fs layer I1105 20:14:40.037] 7940edb1bae7: Pulling fs layer I1105 20:14:40.037] 2ac04aec52fc: Pulling fs layer I1105 20:14:40.037] 3d25cb8b687d: Pulling fs layer I1105 20:14:40.037] c307cc385e02: Pulling fs layer I1105 20:14:40.037] 598f9736ca9b: Pulling fs layer I1105 20:14:40.037] 7fc152dfb3a6: Waiting I1105 20:14:40.038] 2ac04aec52fc: Waiting I1105 20:14:40.038] 3d25cb8b687d: Waiting I1105 20:14:40.038] c307cc385e02: Waiting I1105 20:14:40.038] 7940edb1bae7: Waiting I1105 20:14:40.038] 598f9736ca9b: Waiting I1105 20:14:40.346] d83811f270d5: Download complete I1105 20:14:40.482] ee671aafb583: Verifying Checksum I1105 20:14:40.482] ee671aafb583: Download complete I1105 20:14:40.601] 7fc152dfb3a6: Verifying Checksum I1105 20:14:40.601] 7fc152dfb3a6: Download complete I1105 20:14:40.670] 5667fdb72017: Verifying Checksum I1105 20:14:40.670] 5667fdb72017: Download complete I1105 20:14:40.975] 7940edb1bae7: Verifying Checksum I1105 20:14:40.975] 7940edb1bae7: Download complete I1105 20:14:41.270] 3d25cb8b687d: Verifying Checksum I1105 20:14:41.270] 3d25cb8b687d: Download complete I1105 20:14:41.290] 2ac04aec52fc: Verifying Checksum I1105 20:14:41.291] 2ac04aec52fc: Download complete I1105 20:14:41.317] c307cc385e02: Verifying Checksum I1105 20:14:41.317] c307cc385e02: Download complete I1105 20:14:41.689] 598f9736ca9b: Verifying Checksum I1105 20:14:41.690] 598f9736ca9b: Download complete I1105 20:14:59.871] 5667fdb72017: Pull complete I1105 20:15:08.066] d83811f270d5: Pull complete I1105 20:15:27.870] ee671aafb583: Pull complete I1105 20:15:39.010] 7fc152dfb3a6: Pull complete I1105 20:15:56.066] 7940edb1bae7: Pull complete I1105 20:15:59.161] 2ac04aec52fc: Pull complete I1105 20:16:00.269] 3d25cb8b687d: Pull complete I1105 20:16:00.990] c307cc385e02: Pull complete I1105 20:16:02.025] 598f9736ca9b: Pull complete I1105 20:16:02.048] Digest: sha256:75d4ff5d7cd845d6e810c67796ecb037e852fb849ec1eb5b3194564008e6d1e0 I1105 20:16:02.065] Status: Downloaded newer image for k8sprowinternal.azurecr.io/azuredisk-csi:e2e-3e3ed045c6faa509dbd8d2890ee9f56aa2934887 I1105 20:16:02.066] k8sprowinternal.azurecr.io/azuredisk-csi:e2e-3e3ed045c6faa509dbd8d2890ee9f56aa2934887 I1105 20:16:02.069] helm install charts/latest/azuredisk-csi-driver -n azuredisk-csi-driver --namespace kube-system --wait \ I1105 20:16:02.070] --set image.pullPolicy=IfNotPresent \ I1105 20:16:02.070] --set image.repository=k8sprowinternal.azurecr.io/azuredisk-csi \ I1105 20:16:02.070] --set image.tag=e2e-3e3ed045c6faa509dbd8d2890ee9f56aa2934887 I1105 20:17:01.420] NAME: azuredisk-csi-driver I1105 20:17:02.836] LAST DEPLOYED: Tue Nov 5 20:16:03 2019 I1105 20:17:02.836] NAMESPACE: kube-system I1105 20:17:02.836] STATUS: DEPLOYED I1105 20:17:02.836] I1105 20:17:02.836] RESOURCES: I1105 20:17:02.836] ==> v1/DaemonSet I1105 20:17:02.836] NAME AGE I1105 20:17:02.836] csi-azuredisk-node 53s I1105 20:17:02.837] I1105 20:17:02.837] ==> v1/Deployment I1105 20:17:02.837] csi-azuredisk-controller 53s I1105 20:17:02.837] I1105 20:17:02.837] ==> v1/Pod(related) I1105 20:17:02.837] I1105 20:17:02.837] NAME READY STATUS RESTARTS AGE I1105 20:17:02.837] csi-azuredisk-node-5wwhz 3/3 Running 0 53s I1105 20:17:02.838] csi-azuredisk-node-q2dmt 3/3 Running 0 53s I1105 20:17:02.838] csi-azuredisk-controller-6f6587bfc6-62nrt 6/6 Running 0 53s I1105 20:17:02.838] I1105 20:17:02.838] ==> v1/ServiceAccount I1105 20:17:02.838] I1105 20:17:02.838] NAME AGE I1105 20:17:02.839] csi-azuredisk-controller-sa 54s I1105 20:17:02.839] I1105 20:17:02.839] ==> v1beta1/CustomResourceDefinition I1105 20:17:02.839] csidrivers.csi.storage.k8s.io 54s I1105 20:17:02.839] csinodeinfos.csi.storage.k8s.io 54s I1105 20:17:02.839] I1105 20:17:02.839] ==> v1/ClusterRole I1105 20:17:02.839] azuredisk-external-snapshotter-role 54s I1105 20:17:02.839] azuredisk-external-provisioner-role 54s I1105 20:17:02.840] azuredisk-cluster-driver-registrar-role 54s I1105 20:17:02.840] azuredisk-external-attacher-role 54s I1105 20:17:02.840] I1105 20:17:02.840] ==> v1/ClusterRoleBinding I1105 20:17:02.840] azuredisk-csi-driver-registrar-binding 54s I1105 20:17:02.840] azuredisk-csi-snapshotter-binding 54s I1105 20:17:02.840] azuredisk-csi-attacher-binding 54s I1105 20:17:02.840] azuredisk-csi-provisioner-binding 53s I1105 20:17:02.841] I1105 20:17:02.841] ==> v1/Service I1105 20:17:02.841] csi-azuredisk-controller 53s I1105 20:17:02.841] I1105 20:17:02.841] I1105 20:17:02.841] NOTES: I1105 20:17:02.841] The Azure Disk CSI Driver is getting deployed to your cluster. I1105 20:17:02.841] I1105 20:17:02.841] To check Azure Disk CSI Driver pods status, please run: I1105 20:17:02.842] I1105 20:17:02.842] kubectl --namespace=kube-system get pods --selector="release=azuredisk-csi-driver" --watch I1105 20:17:02.842] I1105 20:17:02.842] make[1]: Leaving directory '/go/src/sigs.k8s.io/azuredisk-csi-driver' I1105 20:17:02.842] 2019/11/05 20:17:02 Azure Disk CSI Driver installed I1105 20:17:02.842] [azuredisk-csi-e2e] Dynamic Provisioning [multi-az] I1105 20:17:02.842] should create a volume on demand I1105 20:17:02.842] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:78 I1105 20:17:02.842] [BeforeEach] [multi-az] I1105 20:17:02.843] /go/src/sigs.k8s.io/azuredisk-csi-driver/vendor/k8s.io/kubernetes/test/e2e/framework/framework.go:150 I1105 20:17:02.843] STEP: Creating a kubernetes client I1105 20:17:02.843] Nov 5 20:17:02.838: INFO: >>> kubeConfig: /workspace/aks971561099/kubeconfig/kubeconfig.eastus2.json I1105 20:17:02.845] STEP: Building a namespace api object, basename azuredisk I1105 20:17:03.470] Nov 5 20:17:03.470: INFO: Found PodSecurityPolicies; assuming PodSecurityPolicy is enabled. I1105 20:17:03.837] Nov 5 20:17:03.837: INFO: Found ClusterRoles; assuming RBAC is enabled. I1105 20:17:03.920] STEP: Binding the e2e-test-privileged-psp PodSecurityPolicy to the default service account in azuredisk-8307 I1105 20:17:04.189] STEP: Waiting for a default service account to be provisioned in namespace I1105 20:17:04.265] [BeforeEach] [multi-az] I1105 20:17:04.266] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:58 I1105 20:17:04.345] [It] should create a volume on demand I1105 20:17:04.346] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:78 I1105 20:17:04.346] STEP: setting up the StorageClass I1105 20:17:04.346] STEP: creating a StorageClass I1105 20:17:04.428] STEP: setting up the PVC and PV I1105 20:17:04.428] STEP: creating a PVC I1105 20:17:04.521] STEP: waiting for PVC to be in phase "Bound" I1105 20:17:04.522] Nov 5 20:17:04.521: INFO: Waiting up to 5m0s for PersistentVolumeClaims [pvc-9g8k7] to have phase Bound I1105 20:17:04.600] Nov 5 20:17:04.600: INFO: PersistentVolumeClaim pvc-9g8k7 found but phase is Pending instead of Bound. I1105 20:17:06.680] Nov 5 20:17:06.680: INFO: PersistentVolumeClaim pvc-9g8k7 found but phase is Pending instead of Bound. I1105 20:17:08.759] Nov 5 20:17:08.759: INFO: PersistentVolumeClaim pvc-9g8k7 found but phase is Pending instead of Bound. I1105 20:17:10.838] Nov 5 20:17:10.838: INFO: PersistentVolumeClaim pvc-9g8k7 found but phase is Pending instead of Bound. I1105 20:17:12.917] Nov 5 20:17:12.917: INFO: PersistentVolumeClaim pvc-9g8k7 found but phase is Pending instead of Bound. I1105 20:17:14.996] Nov 5 20:17:14.996: INFO: PersistentVolumeClaim pvc-9g8k7 found but phase is Pending instead of Bound. I1105 20:17:17.074] Nov 5 20:17:17.074: INFO: PersistentVolumeClaim pvc-9g8k7 found but phase is Pending instead of Bound. I1105 20:17:19.153] Nov 5 20:17:19.153: INFO: PersistentVolumeClaim pvc-9g8k7 found but phase is Pending instead of Bound. I1105 20:17:21.232] Nov 5 20:17:21.232: INFO: PersistentVolumeClaim pvc-9g8k7 found but phase is Pending instead of Bound. I1105 20:17:23.311] Nov 5 20:17:23.311: INFO: PersistentVolumeClaim pvc-9g8k7 found but phase is Pending instead of Bound. I1105 20:17:25.390] Nov 5 20:17:25.390: INFO: PersistentVolumeClaim pvc-9g8k7 found but phase is Pending instead of Bound. I1105 20:17:27.470] Nov 5 20:17:27.469: INFO: PersistentVolumeClaim pvc-9g8k7 found but phase is Pending instead of Bound. I1105 20:17:29.548] Nov 5 20:17:29.548: INFO: PersistentVolumeClaim pvc-9g8k7 found but phase is Pending instead of Bound. I1105 20:17:31.627] Nov 5 20:17:31.627: INFO: PersistentVolumeClaim pvc-9g8k7 found but phase is Pending instead of Bound. I1105 20:17:33.708] Nov 5 20:17:33.707: INFO: PersistentVolumeClaim pvc-9g8k7 found but phase is Pending instead of Bound. I1105 20:17:35.787] Nov 5 20:17:35.787: INFO: PersistentVolumeClaim pvc-9g8k7 found but phase is Pending instead of Bound. I1105 20:17:37.866] Nov 5 20:17:37.865: INFO: PersistentVolumeClaim pvc-9g8k7 found but phase is Pending instead of Bound. I1105 20:17:39.944] Nov 5 20:17:39.944: INFO: PersistentVolumeClaim pvc-9g8k7 found but phase is Pending instead of Bound. I1105 20:17:42.024] Nov 5 20:17:42.024: INFO: PersistentVolumeClaim pvc-9g8k7 found but phase is Pending instead of Bound. I1105 20:17:44.104] Nov 5 20:17:44.104: INFO: PersistentVolumeClaim pvc-9g8k7 found but phase is Pending instead of Bound. I1105 20:17:46.218] Nov 5 20:17:46.218: INFO: PersistentVolumeClaim pvc-9g8k7 found but phase is Pending instead of Bound. I1105 20:17:48.296] Nov 5 20:17:48.296: INFO: PersistentVolumeClaim pvc-9g8k7 found but phase is Pending instead of Bound. I1105 20:17:50.376] Nov 5 20:17:50.375: INFO: PersistentVolumeClaim pvc-9g8k7 found but phase is Pending instead of Bound. I1105 20:17:52.454] Nov 5 20:17:52.454: INFO: PersistentVolumeClaim pvc-9g8k7 found but phase is Pending instead of Bound. I1105 20:17:54.534] Nov 5 20:17:54.533: INFO: PersistentVolumeClaim pvc-9g8k7 found but phase is Pending instead of Bound. I1105 20:17:56.613] Nov 5 20:17:56.612: INFO: PersistentVolumeClaim pvc-9g8k7 found but phase is Pending instead of Bound. I1105 20:17:58.698] Nov 5 20:17:58.698: INFO: PersistentVolumeClaim pvc-9g8k7 found but phase is Pending instead of Bound. I1105 20:18:00.777] Nov 5 20:18:00.777: INFO: PersistentVolumeClaim pvc-9g8k7 found but phase is Pending instead of Bound. I1105 20:18:02.857] Nov 5 20:18:02.857: INFO: PersistentVolumeClaim pvc-9g8k7 found but phase is Pending instead of Bound. I1105 20:18:04.937] Nov 5 20:18:04.936: INFO: PersistentVolumeClaim pvc-9g8k7 found but phase is Pending instead of Bound. I1105 20:18:07.016] Nov 5 20:18:07.016: INFO: PersistentVolumeClaim pvc-9g8k7 found but phase is Pending instead of Bound. I1105 20:18:09.095] Nov 5 20:18:09.095: INFO: PersistentVolumeClaim pvc-9g8k7 found but phase is Pending instead of Bound. I1105 20:18:11.174] Nov 5 20:18:11.174: INFO: PersistentVolumeClaim pvc-9g8k7 found but phase is Pending instead of Bound. I1105 20:18:13.253] Nov 5 20:18:13.253: INFO: PersistentVolumeClaim pvc-9g8k7 found but phase is Pending instead of Bound. I1105 20:18:15.332] Nov 5 20:18:15.332: INFO: PersistentVolumeClaim pvc-9g8k7 found but phase is Pending instead of Bound. I1105 20:18:17.422] Nov 5 20:18:17.422: INFO: PersistentVolumeClaim pvc-9g8k7 found but phase is Pending instead of Bound. I1105 20:18:19.501] Nov 5 20:18:19.501: INFO: PersistentVolumeClaim pvc-9g8k7 found but phase is Pending instead of Bound. I1105 20:18:21.579] Nov 5 20:18:21.579: INFO: PersistentVolumeClaim pvc-9g8k7 found but phase is Pending instead of Bound. I1105 20:18:23.658] Nov 5 20:18:23.658: INFO: PersistentVolumeClaim pvc-9g8k7 found but phase is Pending instead of Bound. I1105 20:18:25.740] Nov 5 20:18:25.740: INFO: PersistentVolumeClaim pvc-9g8k7 found but phase is Pending instead of Bound. I1105 20:18:27.819] Nov 5 20:18:27.819: INFO: PersistentVolumeClaim pvc-9g8k7 found but phase is Pending instead of Bound. I1105 20:18:29.899] Nov 5 20:18:29.898: INFO: PersistentVolumeClaim pvc-9g8k7 found but phase is Pending instead of Bound. I1105 20:18:31.978] Nov 5 20:18:31.978: INFO: PersistentVolumeClaim pvc-9g8k7 found but phase is Pending instead of Bound. I1105 20:18:34.057] Nov 5 20:18:34.057: INFO: PersistentVolumeClaim pvc-9g8k7 found but phase is Pending instead of Bound. I1105 20:18:36.137] Nov 5 20:18:36.136: INFO: PersistentVolumeClaim pvc-9g8k7 found but phase is Pending instead of Bound. I1105 20:18:38.226] Nov 5 20:18:38.226: INFO: PersistentVolumeClaim pvc-9g8k7 found but phase is Pending instead of Bound. I1105 20:18:40.306] Nov 5 20:18:40.306: INFO: PersistentVolumeClaim pvc-9g8k7 found but phase is Pending instead of Bound. I1105 20:18:42.385] Nov 5 20:18:42.385: INFO: PersistentVolumeClaim pvc-9g8k7 found but phase is Pending instead of Bound. I1105 20:18:44.464] Nov 5 20:18:44.464: INFO: PersistentVolumeClaim pvc-9g8k7 found but phase is Pending instead of Bound. I1105 20:18:46.543] Nov 5 20:18:46.543: INFO: PersistentVolumeClaim pvc-9g8k7 found but phase is Pending instead of Bound. I1105 20:18:48.623] Nov 5 20:18:48.622: INFO: PersistentVolumeClaim pvc-9g8k7 found but phase is Pending instead of Bound. I1105 20:18:50.702] Nov 5 20:18:50.701: INFO: PersistentVolumeClaim pvc-9g8k7 found but phase is Pending instead of Bound. I1105 20:18:52.780] Nov 5 20:18:52.780: INFO: PersistentVolumeClaim pvc-9g8k7 found but phase is Pending instead of Bound. I1105 20:18:54.863] Nov 5 20:18:54.862: INFO: PersistentVolumeClaim pvc-9g8k7 found but phase is Pending instead of Bound. I1105 20:18:56.945] Nov 5 20:18:56.945: INFO: PersistentVolumeClaim pvc-9g8k7 found but phase is Pending instead of Bound. I1105 20:18:59.026] Nov 5 20:18:59.026: INFO: PersistentVolumeClaim pvc-9g8k7 found but phase is Pending instead of Bound. I1105 20:19:01.105] Nov 5 20:19:01.105: INFO: PersistentVolumeClaim pvc-9g8k7 found but phase is Pending instead of Bound. I1105 20:19:03.184] Nov 5 20:19:03.184: INFO: PersistentVolumeClaim pvc-9g8k7 found but phase is Pending instead of Bound. I1105 20:19:05.263] Nov 5 20:19:05.263: INFO: PersistentVolumeClaim pvc-9g8k7 found but phase is Pending instead of Bound. I1105 20:19:07.341] Nov 5 20:19:07.341: INFO: PersistentVolumeClaim pvc-9g8k7 found but phase is Pending instead of Bound. I1105 20:19:09.420] Nov 5 20:19:09.420: INFO: PersistentVolumeClaim pvc-9g8k7 found but phase is Pending instead of Bound. I1105 20:19:11.499] Nov 5 20:19:11.499: INFO: PersistentVolumeClaim pvc-9g8k7 found but phase is Pending instead of Bound. I1105 20:19:13.578] Nov 5 20:19:13.578: INFO: PersistentVolumeClaim pvc-9g8k7 found but phase is Pending instead of Bound. I1105 20:19:15.658] Nov 5 20:19:15.658: INFO: PersistentVolumeClaim pvc-9g8k7 found but phase is Pending instead of Bound. I1105 20:19:17.737] Nov 5 20:19:17.737: INFO: PersistentVolumeClaim pvc-9g8k7 found but phase is Pending instead of Bound. I1105 20:19:19.816] Nov 5 20:19:19.816: INFO: PersistentVolumeClaim pvc-9g8k7 found but phase is Pending instead of Bound. I1105 20:19:21.895] Nov 5 20:19:21.895: INFO: PersistentVolumeClaim pvc-9g8k7 found but phase is Pending instead of Bound. I1105 20:19:23.974] Nov 5 20:19:23.974: INFO: PersistentVolumeClaim pvc-9g8k7 found but phase is Pending instead of Bound. I1105 20:19:26.053] Nov 5 20:19:26.053: INFO: PersistentVolumeClaim pvc-9g8k7 found but phase is Pending instead of Bound. I1105 20:19:28.132] Nov 5 20:19:28.132: INFO: PersistentVolumeClaim pvc-9g8k7 found but phase is Pending instead of Bound. I1105 20:19:30.211] Nov 5 20:19:30.211: INFO: PersistentVolumeClaim pvc-9g8k7 found but phase is Pending instead of Bound. I1105 20:19:32.290] Nov 5 20:19:32.290: INFO: PersistentVolumeClaim pvc-9g8k7 found but phase is Pending instead of Bound. I1105 20:19:34.370] Nov 5 20:19:34.369: INFO: PersistentVolumeClaim pvc-9g8k7 found but phase is Pending instead of Bound. I1105 20:19:36.449] Nov 5 20:19:36.448: INFO: PersistentVolumeClaim pvc-9g8k7 found but phase is Pending instead of Bound. I1105 20:19:38.527] Nov 5 20:19:38.527: INFO: PersistentVolumeClaim pvc-9g8k7 found but phase is Pending instead of Bound. I1105 20:19:40.606] Nov 5 20:19:40.606: INFO: PersistentVolumeClaim pvc-9g8k7 found but phase is Pending instead of Bound. I1105 20:19:42.684] Nov 5 20:19:42.684: INFO: PersistentVolumeClaim pvc-9g8k7 found but phase is Pending instead of Bound. I1105 20:19:44.763] Nov 5 20:19:44.763: INFO: PersistentVolumeClaim pvc-9g8k7 found but phase is Pending instead of Bound. I1105 20:19:46.842] Nov 5 20:19:46.842: INFO: PersistentVolumeClaim pvc-9g8k7 found but phase is Pending instead of Bound. I1105 20:19:48.921] Nov 5 20:19:48.921: INFO: PersistentVolumeClaim pvc-9g8k7 found but phase is Pending instead of Bound. I1105 20:19:51.001] Nov 5 20:19:51.000: INFO: PersistentVolumeClaim pvc-9g8k7 found but phase is Pending instead of Bound. I1105 20:19:53.080] Nov 5 20:19:53.079: INFO: PersistentVolumeClaim pvc-9g8k7 found but phase is Pending instead of Bound. I1105 20:19:55.158] Nov 5 20:19:55.158: INFO: PersistentVolumeClaim pvc-9g8k7 found but phase is Pending instead of Bound. I1105 20:19:57.237] Nov 5 20:19:57.237: INFO: PersistentVolumeClaim pvc-9g8k7 found but phase is Pending instead of Bound. I1105 20:19:59.316] Nov 5 20:19:59.316: INFO: PersistentVolumeClaim pvc-9g8k7 found but phase is Pending instead of Bound. I1105 20:20:01.395] Nov 5 20:20:01.395: INFO: PersistentVolumeClaim pvc-9g8k7 found but phase is Pending instead of Bound. I1105 20:20:03.474] Nov 5 20:20:03.474: INFO: PersistentVolumeClaim pvc-9g8k7 found but phase is Pending instead of Bound. I1105 20:20:05.553] Nov 5 20:20:05.553: INFO: PersistentVolumeClaim pvc-9g8k7 found but phase is Pending instead of Bound. I1105 20:20:07.633] Nov 5 20:20:07.632: INFO: PersistentVolumeClaim pvc-9g8k7 found but phase is Pending instead of Bound. I1105 20:20:09.711] Nov 5 20:20:09.711: INFO: PersistentVolumeClaim pvc-9g8k7 found but phase is Pending instead of Bound. I1105 20:20:11.790] Nov 5 20:20:11.790: INFO: PersistentVolumeClaim pvc-9g8k7 found but phase is Pending instead of Bound. I1105 20:20:13.870] Nov 5 20:20:13.869: INFO: PersistentVolumeClaim pvc-9g8k7 found but phase is Pending instead of Bound. I1105 20:20:15.949] Nov 5 20:20:15.948: INFO: PersistentVolumeClaim pvc-9g8k7 found but phase is Pending instead of Bound. I1105 20:20:18.028] Nov 5 20:20:18.028: INFO: PersistentVolumeClaim pvc-9g8k7 found but phase is Pending instead of Bound. I1105 20:20:20.106] Nov 5 20:20:20.106: INFO: PersistentVolumeClaim pvc-9g8k7 found but phase is Pending instead of Bound. I1105 20:20:22.220] Nov 5 20:20:22.219: INFO: PersistentVolumeClaim pvc-9g8k7 found but phase is Pending instead of Bound. I1105 20:20:24.299] Nov 5 20:20:24.298: INFO: PersistentVolumeClaim pvc-9g8k7 found but phase is Pending instead of Bound. I1105 20:20:26.377] Nov 5 20:20:26.377: INFO: PersistentVolumeClaim pvc-9g8k7 found but phase is Pending instead of Bound. I1105 20:20:28.456] Nov 5 20:20:28.456: INFO: PersistentVolumeClaim pvc-9g8k7 found but phase is Pending instead of Bound. I1105 20:20:30.535] Nov 5 20:20:30.535: INFO: PersistentVolumeClaim pvc-9g8k7 found but phase is Pending instead of Bound. I1105 20:20:32.613] Nov 5 20:20:32.613: INFO: PersistentVolumeClaim pvc-9g8k7 found but phase is Pending instead of Bound. I1105 20:20:34.693] Nov 5 20:20:34.692: INFO: PersistentVolumeClaim pvc-9g8k7 found but phase is Pending instead of Bound. I1105 20:20:36.772] Nov 5 20:20:36.771: INFO: PersistentVolumeClaim pvc-9g8k7 found but phase is Pending instead of Bound. I1105 20:20:38.852] Nov 5 20:20:38.852: INFO: PersistentVolumeClaim pvc-9g8k7 found but phase is Pending instead of Bound. I1105 20:20:40.931] Nov 5 20:20:40.931: INFO: PersistentVolumeClaim pvc-9g8k7 found but phase is Pending instead of Bound. I1105 20:20:43.010] Nov 5 20:20:43.010: INFO: PersistentVolumeClaim pvc-9g8k7 found but phase is Pending instead of Bound. I1105 20:20:45.089] Nov 5 20:20:45.088: INFO: PersistentVolumeClaim pvc-9g8k7 found but phase is Pending instead of Bound. I1105 20:20:47.168] Nov 5 20:20:47.168: INFO: PersistentVolumeClaim pvc-9g8k7 found but phase is Pending instead of Bound. I1105 20:20:49.247] Nov 5 20:20:49.247: INFO: PersistentVolumeClaim pvc-9g8k7 found but phase is Pending instead of Bound. I1105 20:20:51.326] Nov 5 20:20:51.326: INFO: PersistentVolumeClaim pvc-9g8k7 found but phase is Pending instead of Bound. I1105 20:20:53.406] Nov 5 20:20:53.406: INFO: PersistentVolumeClaim pvc-9g8k7 found but phase is Pending instead of Bound. I1105 20:20:55.485] Nov 5 20:20:55.485: INFO: PersistentVolumeClaim pvc-9g8k7 found but phase is Pending instead of Bound. I1105 20:20:57.568] Nov 5 20:20:57.568: INFO: PersistentVolumeClaim pvc-9g8k7 found but phase is Pending instead of Bound. I1105 20:20:59.648] Nov 5 20:20:59.648: INFO: PersistentVolumeClaim pvc-9g8k7 found but phase is Pending instead of Bound. I1105 20:21:01.728] Nov 5 20:21:01.728: INFO: PersistentVolumeClaim pvc-9g8k7 found but phase is Pending instead of Bound. I1105 20:21:03.807] Nov 5 20:21:03.807: INFO: PersistentVolumeClaim pvc-9g8k7 found but phase is Pending instead of Bound. I1105 20:21:05.888] Nov 5 20:21:05.888: INFO: PersistentVolumeClaim pvc-9g8k7 found but phase is Pending instead of Bound. I1105 20:21:07.967] Nov 5 20:21:07.967: INFO: PersistentVolumeClaim pvc-9g8k7 found but phase is Pending instead of Bound. I1105 20:21:10.046] Nov 5 20:21:10.046: INFO: PersistentVolumeClaim pvc-9g8k7 found but phase is Pending instead of Bound. I1105 20:21:12.125] Nov 5 20:21:12.125: INFO: PersistentVolumeClaim pvc-9g8k7 found but phase is Pending instead of Bound. I1105 20:21:14.213] Nov 5 20:21:14.213: INFO: PersistentVolumeClaim pvc-9g8k7 found but phase is Pending instead of Bound. I1105 20:21:16.292] Nov 5 20:21:16.292: INFO: PersistentVolumeClaim pvc-9g8k7 found but phase is Pending instead of Bound. I1105 20:21:18.371] Nov 5 20:21:18.371: INFO: PersistentVolumeClaim pvc-9g8k7 found but phase is Pending instead of Bound. I1105 20:21:20.450] Nov 5 20:21:20.450: INFO: PersistentVolumeClaim pvc-9g8k7 found but phase is Pending instead of Bound. I1105 20:21:22.530] Nov 5 20:21:22.530: INFO: PersistentVolumeClaim pvc-9g8k7 found but phase is Pending instead of Bound. I1105 20:21:24.609] Nov 5 20:21:24.609: INFO: PersistentVolumeClaim pvc-9g8k7 found but phase is Pending instead of Bound. I1105 20:21:26.689] Nov 5 20:21:26.688: INFO: PersistentVolumeClaim pvc-9g8k7 found but phase is Pending instead of Bound. I1105 20:21:28.768] Nov 5 20:21:28.767: INFO: PersistentVolumeClaim pvc-9g8k7 found but phase is Pending instead of Bound. I1105 20:21:30.847] Nov 5 20:21:30.847: INFO: PersistentVolumeClaim pvc-9g8k7 found but phase is Pending instead of Bound. I1105 20:21:32.926] Nov 5 20:21:32.926: INFO: PersistentVolumeClaim pvc-9g8k7 found but phase is Pending instead of Bound. I1105 20:21:35.006] Nov 5 20:21:35.006: INFO: PersistentVolumeClaim pvc-9g8k7 found but phase is Pending instead of Bound. I1105 20:21:37.085] Nov 5 20:21:37.084: INFO: PersistentVolumeClaim pvc-9g8k7 found but phase is Pending instead of Bound. I1105 20:21:39.164] Nov 5 20:21:39.164: INFO: PersistentVolumeClaim pvc-9g8k7 found but phase is Pending instead of Bound. I1105 20:21:41.243] Nov 5 20:21:41.243: INFO: PersistentVolumeClaim pvc-9g8k7 found but phase is Pending instead of Bound. I1105 20:21:43.322] Nov 5 20:21:43.321: INFO: PersistentVolumeClaim pvc-9g8k7 found but phase is Pending instead of Bound. I1105 20:21:45.400] Nov 5 20:21:45.400: INFO: PersistentVolumeClaim pvc-9g8k7 found but phase is Pending instead of Bound. I1105 20:21:47.479] Nov 5 20:21:47.479: INFO: PersistentVolumeClaim pvc-9g8k7 found but phase is Pending instead of Bound. I1105 20:21:49.558] Nov 5 20:21:49.558: INFO: PersistentVolumeClaim pvc-9g8k7 found but phase is Pending instead of Bound. I1105 20:21:51.637] Nov 5 20:21:51.637: INFO: PersistentVolumeClaim pvc-9g8k7 found but phase is Pending instead of Bound. I1105 20:21:53.717] Nov 5 20:21:53.717: INFO: PersistentVolumeClaim pvc-9g8k7 found but phase is Pending instead of Bound. I1105 20:21:55.797] Nov 5 20:21:55.797: INFO: PersistentVolumeClaim pvc-9g8k7 found but phase is Pending instead of Bound. I1105 20:21:57.876] Nov 5 20:21:57.876: INFO: PersistentVolumeClaim pvc-9g8k7 found but phase is Pending instead of Bound. I1105 20:21:59.955] Nov 5 20:21:59.955: INFO: PersistentVolumeClaim pvc-9g8k7 found but phase is Pending instead of Bound. I1105 20:22:02.035] Nov 5 20:22:02.035: INFO: PersistentVolumeClaim pvc-9g8k7 found but phase is Pending instead of Bound. I1105 20:22:04.115] Nov 5 20:22:04.114: INFO: PersistentVolumeClaim pvc-9g8k7 found but phase is Pending instead of Bound. I1105 20:22:06.115] Nov 5 20:22:06.115: INFO: Unexpected error occurred: PersistentVolumeClaims [pvc-9g8k7] not all in phase Bound within 5m0s I1105 20:22:06.115] [AfterEach] [multi-az] I1105 20:22:06.116] /go/src/sigs.k8s.io/azuredisk-csi-driver/vendor/k8s.io/kubernetes/test/e2e/framework/framework.go:151 I1105 20:22:06.116] STEP: Collecting events from namespace "azuredisk-8307". I1105 20:22:06.213] STEP: Found 3 events. I1105 20:22:06.213] Nov 5 20:22:06.213: INFO: At 2019-11-05 20:17:04 +0000 UTC - event for pvc-9g8k7: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "disk.csi.azure.com" or manually created by system administrator I1105 20:22:06.213] Nov 5 20:22:06.213: INFO: At 2019-11-05 20:17:04 +0000 UTC - event for pvc-9g8k7: {disk.csi.azure.com_csi-azuredisk-controller-6f6587bfc6-62nrt_3930bc69-0009-11ea-8cf5-829a27ff7d5a } Provisioning: External provisioner is provisioning volume for claim "azuredisk-8307/pvc-9g8k7" I1105 20:22:06.214] Nov 5 20:22:06.213: INFO: At 2019-11-05 20:17:05 +0000 UTC - event for pvc-9g8k7: {disk.csi.azure.com_csi-azuredisk-controller-6f6587bfc6-62nrt_3930bc69-0009-11ea-8cf5-829a27ff7d5a } ProvisioningFailed: failed to provision volume with StorageClass "azuredisk-8307-disk.csi.azure.com-dynamic-sc-5dsj4": rpc error: code = Unknown desc = compute.DisksClient#CreateOrUpdate: Failure sending request: StatusCode=400 -- Original Error: Code="InvalidAvailabilityZone" Message="The zone(s) '' for resource 'Microsoft.Compute/disks/pvc-bf33a003-e1f4-4125-b63b-8b0663bda0f8' is not supported. The supported zones for location 'eastus2' are '1,2,3'" I1105 20:22:06.293] Nov 5 20:22:06.293: INFO: POD NODE PHASE GRACE CONDITIONS I1105 20:22:06.293] Nov 5 20:22:06.293: INFO: I1105 20:22:06.533] Nov 5 20:22:06.532: INFO: I1105 20:22:06.533] Logging node info for node k8s-agentpool-97435445-vmss000000 I1105 20:22:06.624] Nov 5 20:22:06.620: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-agentpool-97435445-vmss000000,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-agentpool-97435445-vmss000000,UID:e955dc47-8c8f-4142-9241-3d409f32e42d,ResourceVersion:3450,Generation:0,CreationTimestamp:2019-11-05 20:07:34 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{agentpool: agentpool,beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_D4s_v3,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 0,kubernetes.azure.com/cluster: kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b,kubernetes.azure.com/role: agent,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-agentpool-97435445-vmss000000,kubernetes.io/os: linux,kubernetes.io/role: agent,node-role.kubernetes.io/agent: ,storageprofile: managed,storagetier: Premium_LRS,topology.disk.csi.azure.com/zone: 0,},Annotations:map[string]string{csi.volume.kubernetes.io/nodeid: {"disk.csi.azure.com":"k8s-agentpool-97435445-vmss000000"},node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.2.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool-97435445-vmss/virtualMachines/0,Unschedulable:false,Taints:[],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-csi-disk.csi.azure.com: {{8 0} {} 8 DecimalSI},cpu: {{4 0} {} 4 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{16797040640 0} {} BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-csi-disk.csi.azure.com: {{8 0} {} 8 DecimalSI},cpu: {{4 0} {} 4 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{16010608640 0} {} BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 20:08:01 +0000 UTC 2019-11-05 20:08:01 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 20:21:37 +0000 UTC 2019-11-05 20:07:27 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 20:21:37 +0000 UTC 2019-11-05 20:07:27 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 20:21:37 +0000 UTC 2019-11-05 20:07:27 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 20:21:37 +0000 UTC 2019-11-05 20:07:55 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.0.4} {Hostname k8s-agentpool-97435445-vmss000000}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:bfb55325809340228149d2d93f50df58,SystemUUID:CB2ED036-0439-F641-BC7F-640C3638C027,BootID:a917f4d1-549d-471d-b7dc-47ae1715d97d,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[k8sprowinternal.azurecr.io/azuredisk-csi@sha256:75d4ff5d7cd845d6e810c67796ecb037e852fb849ec1eb5b3194564008e6d1e0 k8sprowinternal.azurecr.io/azuredisk-csi:e2e-3e3ed045c6faa509dbd8d2890ee9f56aa2934887] 286930414} {[k8s.gcr.io/kubernetes-dashboard-amd64@sha256:0ae6b69432e78069c5ce2bcde0fe409c5c4d6f0f4d9cd50a17974fea38898747 k8s.gcr.io/kubernetes-dashboard-amd64:v1.10.1] 121711221} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/metrics-server-amd64@sha256:49a9f12f7067d11f42c803dbe61ed2c1299959ad85cb315b25ff7eef8e6b8892 k8s.gcr.io/metrics-server-amd64:v0.2.1] 42541759} {[k8s.gcr.io/coredns@sha256:e83beb5e43f8513fa735e77ffc5859640baea30a882a11cc75c4c3244a737d3c k8s.gcr.io/coredns:1.5.0] 42488424} {[mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume@sha256:4fd30d43947d4a54fc89ead7985beecfd3c9b2a93a0655a373b1608ab90bd5af mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume:v0.0.7] 22909487} {[quay.io/k8scsi/livenessprobe@sha256:8c9c8a0f330acc2e558ae6624309447d02522e76aa35af23fa19b88abf4381cb quay.io/k8scsi/livenessprobe:v1.0.2] 17415381} {[quay.io/k8scsi/csi-node-driver-registrar@sha256:13daf82fb99e951a4bff8ae5fc7c17c3a8fe7130be6400990d8f6076c32d4599 quay.io/k8scsi/csi-node-driver-registrar:v1.1.0] 15815995} {[mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume@sha256:23d8c6033f02a1ecad05127ebdc931bb871264228661bc122704b0974e4d9fdd mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume:1.0.8] 1159025} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 20:22:06.624] Nov 5 20:22:06.621: INFO: I1105 20:22:06.625] Logging kubelet events for node k8s-agentpool-97435445-vmss000000 I1105 20:22:06.705] Nov 5 20:22:06.705: INFO: I1105 20:22:06.705] Logging pods the kubelet thinks is on node k8s-agentpool-97435445-vmss000000 I1105 20:22:06.821] Nov 5 20:22:06.821: INFO: kube-proxy-8mt7f started at 2019-11-05 20:08:01 +0000 UTC (0+1 container statuses recorded) I1105 20:22:06.822] Nov 5 20:22:06.821: INFO: Container kube-proxy ready: true, restart count 0 I1105 20:22:06.822] Nov 5 20:22:06.821: INFO: kubernetes-dashboard-66dd8b8df7-t6j4l started at 2019-11-05 20:08:01 +0000 UTC (0+1 container statuses recorded) I1105 20:22:06.822] Nov 5 20:22:06.821: INFO: Container kubernetes-dashboard ready: true, restart count 0 I1105 20:22:06.822] Nov 5 20:22:06.821: INFO: csi-azuredisk-node-5wwhz started at 2019-11-05 20:16:09 +0000 UTC (0+3 container statuses recorded) I1105 20:22:06.822] Nov 5 20:22:06.821: INFO: Container azuredisk ready: true, restart count 0 I1105 20:22:06.823] Nov 5 20:22:06.821: INFO: Container liveness-probe ready: true, restart count 0 I1105 20:22:06.823] Nov 5 20:22:06.821: INFO: Container node-driver-registrar ready: true, restart count 0 I1105 20:22:06.823] Nov 5 20:22:06.821: INFO: blobfuse-flexvol-installer-p7hdb started at 2019-11-05 20:08:01 +0000 UTC (0+1 container statuses recorded) I1105 20:22:06.823] Nov 5 20:22:06.821: INFO: Container blobfuse-flexvol-installer ready: true, restart count 0 I1105 20:22:06.823] Nov 5 20:22:06.821: INFO: coredns-7f68dcdbdb-zh8g5 started at 2019-11-05 20:08:01 +0000 UTC (0+1 container statuses recorded) I1105 20:22:06.823] Nov 5 20:22:06.821: INFO: Container coredns ready: true, restart count 0 I1105 20:22:06.824] Nov 5 20:22:06.821: INFO: azure-ip-masq-agent-m7tv9 started at 2019-11-05 20:08:01 +0000 UTC (0+1 container statuses recorded) I1105 20:22:06.824] Nov 5 20:22:06.821: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 20:22:06.824] Nov 5 20:22:06.821: INFO: keyvault-flexvolume-smspt started at 2019-11-05 20:08:01 +0000 UTC (0+1 container statuses recorded) I1105 20:22:06.825] Nov 5 20:22:06.821: INFO: Container keyvault-flexvolume ready: true, restart count 0 I1105 20:22:06.825] Nov 5 20:22:06.821: INFO: metrics-server-864ffbc5c-wl84z started at 2019-11-05 20:08:01 +0000 UTC (0+1 container statuses recorded) I1105 20:22:06.825] Nov 5 20:22:06.821: INFO: Container metrics-server ready: true, restart count 0 I1105 20:22:07.089] Nov 5 20:22:07.089: INFO: I1105 20:22:07.089] Latency metrics for node k8s-agentpool-97435445-vmss000000 I1105 20:22:07.090] Nov 5 20:22:07.089: INFO: I1105 20:22:07.090] Logging node info for node k8s-agentpool-97435445-vmss000001 I1105 20:22:07.172] Nov 5 20:22:07.168: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-agentpool-97435445-vmss000001,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-agentpool-97435445-vmss000001,UID:75d3cc50-ca6d-45e0-a42a-6090e8d537dd,ResourceVersion:3448,Generation:0,CreationTimestamp:2019-11-05 20:07:34 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{agentpool: agentpool,beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_D4s_v3,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 1,kubernetes.azure.com/cluster: kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b,kubernetes.azure.com/role: agent,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-agentpool-97435445-vmss000001,kubernetes.io/os: linux,kubernetes.io/role: agent,node-role.kubernetes.io/agent: ,storageprofile: managed,storagetier: Premium_LRS,topology.disk.csi.azure.com/zone: 1,},Annotations:map[string]string{csi.volume.kubernetes.io/nodeid: {"disk.csi.azure.com":"k8s-agentpool-97435445-vmss000001"},node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.0.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool-97435445-vmss/virtualMachines/1,Unschedulable:false,Taints:[],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-csi-disk.csi.azure.com: {{8 0} {} 8 DecimalSI},cpu: {{4 0} {} 4 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{16797044736 0} {} 16403364Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-csi-disk.csi.azure.com: {{8 0} {} 8 DecimalSI},cpu: {{4 0} {} 4 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{16010612736 0} {} 15635364Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 20:08:25 +0000 UTC 2019-11-05 20:08:25 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 20:21:36 +0000 UTC 2019-11-05 20:07:26 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 20:21:36 +0000 UTC 2019-11-05 20:07:26 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 20:21:36 +0000 UTC 2019-11-05 20:07:26 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 20:21:36 +0000 UTC 2019-11-05 20:07:55 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.0.5} {Hostname k8s-agentpool-97435445-vmss000001}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:e78def54b2f545c6b94d88bbba523c60,SystemUUID:40AE8188-86AB-D848-84A4-D9254E1258BC,BootID:8587ec06-f4f4-43c1-86cd-06c295b05b63,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[k8sprowinternal.azurecr.io/azuredisk-csi@sha256:75d4ff5d7cd845d6e810c67796ecb037e852fb849ec1eb5b3194564008e6d1e0 k8sprowinternal.azurecr.io/azuredisk-csi:e2e-3e3ed045c6faa509dbd8d2890ee9f56aa2934887] 286930414} {[gcr.io/kubernetes-helm/tiller@sha256:f6d8f4ab9ba993b5f5b60a6edafe86352eabe474ffeb84cb6c79b8866dce45d1 gcr.io/kubernetes-helm/tiller:v2.11.0] 71821984} {[quay.io/k8scsi/csi-provisioner@sha256:d657c839dce87324fe2b677302913f9386f885f8746be7bea0ced5b0844e3433 quay.io/k8scsi/csi-provisioner:v1.3.1] 50596998} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[quay.io/k8scsi/csi-attacher@sha256:26fccd7a99d973845df1193b46ebdcc6ab8dc5f6e6be319750c471fce1742d13 quay.io/k8scsi/csi-attacher:v1.2.0] 46226754} {[quay.io/k8scsi/csi-cluster-driver-registrar@sha256:fafd75ae5442f192cfa8c2e792903aee30d5884b62e802e4464b0a895d21e3ef quay.io/k8scsi/csi-cluster-driver-registrar:v1.0.1] 45874691} {[quay.io/k8scsi/csi-snapshotter@sha256:a49e0da1af6f2bf717e41ba1eee8b5e6a1cbd66a709dd92cc43fe475fe2589eb quay.io/k8scsi/csi-snapshotter:v1.1.0] 44325352} {[mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume@sha256:4fd30d43947d4a54fc89ead7985beecfd3c9b2a93a0655a373b1608ab90bd5af mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume:v0.0.7] 22909487} {[quay.io/k8scsi/livenessprobe@sha256:8c9c8a0f330acc2e558ae6624309447d02522e76aa35af23fa19b88abf4381cb quay.io/k8scsi/livenessprobe:v1.0.2] 17415381} {[quay.io/k8scsi/csi-node-driver-registrar@sha256:13daf82fb99e951a4bff8ae5fc7c17c3a8fe7130be6400990d8f6076c32d4599 quay.io/k8scsi/csi-node-driver-registrar:v1.1.0] 15815995} {[quay.io/k8scsi/livenessprobe@sha256:dde617756e0f602adc566ab71fd885f1dad451ad3fb063ac991c95a2ff47aea5 quay.io/k8scsi/livenessprobe:v1.1.0] 14967303} {[mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume@sha256:23d8c6033f02a1ecad05127ebdc931bb871264228661bc122704b0974e4d9fdd mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume:1.0.8] 1159025} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 20:22:07.172] Nov 5 20:22:07.169: INFO: I1105 20:22:07.173] Logging kubelet events for node k8s-agentpool-97435445-vmss000001 I1105 20:22:07.252] Nov 5 20:22:07.252: INFO: I1105 20:22:07.253] Logging pods the kubelet thinks is on node k8s-agentpool-97435445-vmss000001 I1105 20:22:07.368] Nov 5 20:22:07.367: INFO: azure-ip-masq-agent-gdhcq started at 2019-11-05 20:07:59 +0000 UTC (0+1 container statuses recorded) I1105 20:22:07.368] Nov 5 20:22:07.367: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 20:22:07.368] Nov 5 20:22:07.367: INFO: keyvault-flexvolume-6bffx started at 2019-11-05 20:07:59 +0000 UTC (0+1 container statuses recorded) I1105 20:22:07.368] Nov 5 20:22:07.367: INFO: Container keyvault-flexvolume ready: true, restart count 0 I1105 20:22:07.368] Nov 5 20:22:07.367: INFO: kube-proxy-kjrpb started at 2019-11-05 20:08:00 +0000 UTC (0+1 container statuses recorded) I1105 20:22:07.368] Nov 5 20:22:07.367: INFO: Container kube-proxy ready: true, restart count 0 I1105 20:22:07.369] Nov 5 20:22:07.367: INFO: tiller-deploy-54c96cb5df-pt7x6 started at 2019-11-05 20:08:00 +0000 UTC (0+1 container statuses recorded) I1105 20:22:07.369] Nov 5 20:22:07.367: INFO: Container tiller ready: true, restart count 0 I1105 20:22:07.369] Nov 5 20:22:07.367: INFO: csi-azuredisk-node-q2dmt started at 2019-11-05 20:16:09 +0000 UTC (0+3 container statuses recorded) I1105 20:22:07.369] Nov 5 20:22:07.367: INFO: Container azuredisk ready: true, restart count 0 I1105 20:22:07.369] Nov 5 20:22:07.367: INFO: Container liveness-probe ready: true, restart count 0 I1105 20:22:07.369] Nov 5 20:22:07.367: INFO: Container node-driver-registrar ready: true, restart count 0 I1105 20:22:07.370] Nov 5 20:22:07.367: INFO: csi-azuredisk-controller-6f6587bfc6-62nrt started at 2019-11-05 20:16:09 +0000 UTC (0+6 container statuses recorded) I1105 20:22:07.370] Nov 5 20:22:07.367: INFO: Container azuredisk ready: true, restart count 0 I1105 20:22:07.370] Nov 5 20:22:07.367: INFO: Container cluster-driver-registrar ready: true, restart count 0 I1105 20:22:07.370] Nov 5 20:22:07.367: INFO: Container csi-attacher ready: true, restart count 0 I1105 20:22:07.370] Nov 5 20:22:07.367: INFO: Container csi-provisioner ready: true, restart count 0 I1105 20:22:07.370] Nov 5 20:22:07.367: INFO: Container csi-snapshotter ready: true, restart count 0 I1105 20:22:07.370] Nov 5 20:22:07.367: INFO: Container liveness-probe ready: true, restart count 0 I1105 20:22:07.371] Nov 5 20:22:07.367: INFO: blobfuse-flexvol-installer-j4s7t started at 2019-11-05 20:07:59 +0000 UTC (0+1 container statuses recorded) I1105 20:22:07.371] Nov 5 20:22:07.367: INFO: Container blobfuse-flexvol-installer ready: true, restart count 0 I1105 20:22:07.623] Nov 5 20:22:07.623: INFO: I1105 20:22:07.624] Latency metrics for node k8s-agentpool-97435445-vmss000001 I1105 20:22:07.624] Nov 5 20:22:07.623: INFO: I1105 20:22:07.624] Logging node info for node k8s-master-97435445-0 I1105 20:22:07.706] Nov 5 20:22:07.703: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-97435445-0,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-97435445-0,UID:df3ae36f-961c-4729-a8f5-34cda54f4ff9,ResourceVersion:3518,Generation:0,CreationTimestamp:2019-11-05 20:08:36 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 2,kubernetes.azure.com/cluster: kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-97435445-0,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.4.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b/providers/Microsoft.Compute/virtualMachines/k8s-master-97435445-0,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 20:08:46 +0000 UTC 2019-11-05 20:08:46 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 20:21:58 +0000 UTC 2019-11-05 20:08:28 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 20:21:58 +0000 UTC 2019-11-05 20:08:28 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 20:21:58 +0000 UTC 2019-11-05 20:08:28 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 20:21:58 +0000 UTC 2019-11-05 20:08:36 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.255.5} {Hostname k8s-master-97435445-0}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:3c381865192341c2868672e8626cef7d,SystemUUID:F3D259D3-7EB8-B844-A1F0-F954DCD0D76B,BootID:4948329e-cf57-48c3-aa83-5297bef70370,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 20:22:07.706] Nov 5 20:22:07.703: INFO: I1105 20:22:07.706] Logging kubelet events for node k8s-master-97435445-0 I1105 20:22:07.792] Nov 5 20:22:07.792: INFO: I1105 20:22:07.793] Logging pods the kubelet thinks is on node k8s-master-97435445-0 I1105 20:22:07.902] Nov 5 20:22:07.902: INFO: azure-ip-masq-agent-q6dcl started at 2019-11-05 20:08:38 +0000 UTC (0+1 container statuses recorded) I1105 20:22:07.902] Nov 5 20:22:07.902: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 20:22:07.902] Nov 5 20:22:07.902: INFO: kube-proxy-5wnd2 started at 2019-11-05 20:08:38 +0000 UTC (0+1 container statuses recorded) I1105 20:22:07.902] Nov 5 20:22:07.902: INFO: Container kube-proxy ready: true, restart count 0 I1105 20:22:07.903] Nov 5 20:22:07.902: INFO: cloud-controller-manager-k8s-master-97435445-0 started at 2019-11-05 20:08:22 +0000 UTC (0+1 container statuses recorded) I1105 20:22:07.903] Nov 5 20:22:07.902: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 20:22:07.903] Nov 5 20:22:07.902: INFO: kube-addon-manager-k8s-master-97435445-0 started at 2019-11-05 20:08:48 +0000 UTC (0+1 container statuses recorded) I1105 20:22:07.903] Nov 5 20:22:07.902: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 20:22:07.903] Nov 5 20:22:07.902: INFO: kube-apiserver-k8s-master-97435445-0 started at 2019-11-05 20:08:22 +0000 UTC (0+1 container statuses recorded) I1105 20:22:07.903] Nov 5 20:22:07.902: INFO: Container kube-apiserver ready: true, restart count 0 I1105 20:22:07.904] Nov 5 20:22:07.902: INFO: kube-controller-manager-k8s-master-97435445-0 started at 2019-11-05 20:08:22 +0000 UTC (0+1 container statuses recorded) I1105 20:22:07.904] Nov 5 20:22:07.902: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 20:22:07.904] Nov 5 20:22:07.902: INFO: kube-scheduler-k8s-master-97435445-0 started at 2019-11-05 20:08:22 +0000 UTC (0+1 container statuses recorded) I1105 20:22:07.904] Nov 5 20:22:07.902: INFO: Container kube-scheduler ready: true, restart count 0 I1105 20:22:08.155] Nov 5 20:22:08.154: INFO: I1105 20:22:08.155] Latency metrics for node k8s-master-97435445-0 I1105 20:22:08.155] Nov 5 20:22:08.154: INFO: I1105 20:22:08.155] Logging node info for node k8s-master-97435445-1 I1105 20:22:08.236] Nov 5 20:22:08.233: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-97435445-1,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-97435445-1,UID:05534a67-d493-4ab5-9b0f-5ff5490e234f,ResourceVersion:3420,Generation:0,CreationTimestamp:2019-11-05 20:07:34 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 0,kubernetes.azure.com/cluster: kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-97435445-1,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.1.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b/providers/Microsoft.Compute/virtualMachines/k8s-master-97435445-1,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 20:08:26 +0000 UTC 2019-11-05 20:08:26 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 20:21:26 +0000 UTC 2019-11-05 20:07:28 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 20:21:26 +0000 UTC 2019-11-05 20:07:28 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 20:21:26 +0000 UTC 2019-11-05 20:07:28 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 20:21:26 +0000 UTC 2019-11-05 20:07:55 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{Hostname k8s-master-97435445-1} {InternalIP 10.240.255.6}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:079b605103b74acabac18fd2ea63368a,SystemUUID:38220EEC-BEFF-4C46-AA05-5A214AAFDC28,BootID:b1b45df0-d925-4bcc-9830-0202df36ebb4,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 20:22:08.236] Nov 5 20:22:08.233: INFO: I1105 20:22:08.236] Logging kubelet events for node k8s-master-97435445-1 I1105 20:22:08.317] Nov 5 20:22:08.317: INFO: I1105 20:22:08.317] Logging pods the kubelet thinks is on node k8s-master-97435445-1 I1105 20:22:08.426] Nov 5 20:22:08.425: INFO: cloud-controller-manager-k8s-master-97435445-1 started at 2019-11-05 20:07:22 +0000 UTC (0+1 container statuses recorded) I1105 20:22:08.426] Nov 5 20:22:08.425: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 20:22:08.426] Nov 5 20:22:08.425: INFO: kube-addon-manager-k8s-master-97435445-1 started at 2019-11-05 20:07:22 +0000 UTC (0+1 container statuses recorded) I1105 20:22:08.426] Nov 5 20:22:08.425: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 20:22:08.426] Nov 5 20:22:08.425: INFO: kube-apiserver-k8s-master-97435445-1 started at 2019-11-05 20:07:22 +0000 UTC (0+1 container statuses recorded) I1105 20:22:08.426] Nov 5 20:22:08.425: INFO: Container kube-apiserver ready: true, restart count 0 I1105 20:22:08.427] Nov 5 20:22:08.425: INFO: kube-controller-manager-k8s-master-97435445-1 started at 2019-11-05 20:07:22 +0000 UTC (0+1 container statuses recorded) I1105 20:22:08.427] Nov 5 20:22:08.425: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 20:22:08.427] Nov 5 20:22:08.425: INFO: kube-scheduler-k8s-master-97435445-1 started at 2019-11-05 20:07:22 +0000 UTC (0+1 container statuses recorded) I1105 20:22:08.427] Nov 5 20:22:08.425: INFO: Container kube-scheduler ready: true, restart count 0 I1105 20:22:08.427] Nov 5 20:22:08.425: INFO: azure-ip-masq-agent-n29bm started at 2019-11-05 20:07:59 +0000 UTC (0+1 container statuses recorded) I1105 20:22:08.427] Nov 5 20:22:08.425: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 20:22:08.427] Nov 5 20:22:08.425: INFO: kube-proxy-h4s2c started at 2019-11-05 20:08:00 +0000 UTC (0+1 container statuses recorded) I1105 20:22:08.428] Nov 5 20:22:08.425: INFO: Container kube-proxy ready: true, restart count 0 I1105 20:22:08.686] Nov 5 20:22:08.686: INFO: I1105 20:22:08.686] Latency metrics for node k8s-master-97435445-1 I1105 20:22:08.686] Nov 5 20:22:08.686: INFO: I1105 20:22:08.686] Logging node info for node k8s-master-97435445-2 I1105 20:22:08.768] Nov 5 20:22:08.765: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-97435445-2,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-97435445-2,UID:fa55c895-1a93-4743-885e-03fce84b5332,ResourceVersion:3466,Generation:0,CreationTimestamp:2019-11-05 20:09:19 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 0,kubernetes.azure.com/cluster: kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-97435445-2,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.5.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b/providers/Microsoft.Compute/virtualMachines/k8s-master-97435445-2,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 20:09:19 +0000 UTC 2019-11-05 20:09:19 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 20:21:41 +0000 UTC 2019-11-05 20:09:12 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 20:21:41 +0000 UTC 2019-11-05 20:09:12 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 20:21:41 +0000 UTC 2019-11-05 20:09:12 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 20:21:41 +0000 UTC 2019-11-05 20:09:19 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.255.7} {Hostname k8s-master-97435445-2}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:d19f6709138f496da43c095cec7b2360,SystemUUID:10116777-A8D7-CB4A-A8B7-7EBCF2F23B62,BootID:50cd14a9-5497-4972-bf60-cf59a0cbcb5f,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 20:22:08.768] Nov 5 20:22:08.765: INFO: I1105 20:22:08.768] Logging kubelet events for node k8s-master-97435445-2 I1105 20:22:08.849] Nov 5 20:22:08.849: INFO: I1105 20:22:08.849] Logging pods the kubelet thinks is on node k8s-master-97435445-2 I1105 20:22:08.971] Nov 5 20:22:08.970: INFO: kube-addon-manager-k8s-master-97435445-2 started at 2019-11-05 20:09:05 +0000 UTC (0+1 container statuses recorded) I1105 20:22:08.971] Nov 5 20:22:08.970: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 20:22:08.971] Nov 5 20:22:08.970: INFO: azure-ip-masq-agent-sq7p5 started at 2019-11-05 20:09:19 +0000 UTC (0+1 container statuses recorded) I1105 20:22:08.971] Nov 5 20:22:08.970: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 20:22:08.971] Nov 5 20:22:08.970: INFO: kube-proxy-pz426 started at 2019-11-05 20:09:19 +0000 UTC (0+1 container statuses recorded) I1105 20:22:08.972] Nov 5 20:22:08.970: INFO: Container kube-proxy ready: true, restart count 0 I1105 20:22:08.972] Nov 5 20:22:08.970: INFO: kube-apiserver-k8s-master-97435445-2 started at 2019-11-05 20:09:05 +0000 UTC (0+1 container statuses recorded) I1105 20:22:08.972] Nov 5 20:22:08.970: INFO: Container kube-apiserver ready: true, restart count 0 I1105 20:22:08.972] Nov 5 20:22:08.970: INFO: kube-controller-manager-k8s-master-97435445-2 started at 2019-11-05 20:09:05 +0000 UTC (0+1 container statuses recorded) I1105 20:22:08.972] Nov 5 20:22:08.970: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 20:22:08.972] Nov 5 20:22:08.970: INFO: kube-scheduler-k8s-master-97435445-2 started at 2019-11-05 20:09:05 +0000 UTC (0+1 container statuses recorded) I1105 20:22:08.972] Nov 5 20:22:08.970: INFO: Container kube-scheduler ready: true, restart count 0 I1105 20:22:08.973] Nov 5 20:22:08.970: INFO: cloud-controller-manager-k8s-master-97435445-2 started at 2019-11-05 20:09:05 +0000 UTC (0+1 container statuses recorded) I1105 20:22:08.973] Nov 5 20:22:08.970: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 20:22:09.224] Nov 5 20:22:09.224: INFO: I1105 20:22:09.224] Latency metrics for node k8s-master-97435445-2 I1105 20:22:09.224] Nov 5 20:22:09.224: INFO: I1105 20:22:09.224] Logging node info for node k8s-master-97435445-3 I1105 20:22:09.305] Nov 5 20:22:09.303: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-97435445-3,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-97435445-3,UID:8ffd1d0d-4487-4680-a2f6-e724387be637,ResourceVersion:3472,Generation:0,CreationTimestamp:2019-11-05 20:09:21 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 1,kubernetes.azure.com/cluster: kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-97435445-3,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.6.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b/providers/Microsoft.Compute/virtualMachines/k8s-master-97435445-3,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 20:09:39 +0000 UTC 2019-11-05 20:09:39 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 20:21:42 +0000 UTC 2019-11-05 20:09:18 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 20:21:42 +0000 UTC 2019-11-05 20:09:18 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 20:21:42 +0000 UTC 2019-11-05 20:09:18 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 20:21:42 +0000 UTC 2019-11-05 20:09:21 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.255.8} {Hostname k8s-master-97435445-3}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:266064173bfb4297898c6c25769e8b20,SystemUUID:E6E44241-5015-7740-9A81-A668A8427206,BootID:579d658c-b0ad-47f1-a993-1ccea2b55789,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 20:22:09.306] Nov 5 20:22:09.303: INFO: I1105 20:22:09.306] Logging kubelet events for node k8s-master-97435445-3 I1105 20:22:09.386] Nov 5 20:22:09.386: INFO: I1105 20:22:09.386] Logging pods the kubelet thinks is on node k8s-master-97435445-3 I1105 20:22:09.497] Nov 5 20:22:09.496: INFO: kube-addon-manager-k8s-master-97435445-3 started at 2019-11-05 20:09:05 +0000 UTC (0+1 container statuses recorded) I1105 20:22:09.497] Nov 5 20:22:09.496: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 20:22:09.497] Nov 5 20:22:09.496: INFO: kube-apiserver-k8s-master-97435445-3 started at 2019-11-05 20:09:05 +0000 UTC (0+1 container statuses recorded) I1105 20:22:09.497] Nov 5 20:22:09.496: INFO: Container kube-apiserver ready: true, restart count 0 I1105 20:22:09.498] Nov 5 20:22:09.497: INFO: kube-controller-manager-k8s-master-97435445-3 started at 2019-11-05 20:09:05 +0000 UTC (0+1 container statuses recorded) I1105 20:22:09.498] Nov 5 20:22:09.497: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 20:22:09.498] Nov 5 20:22:09.497: INFO: kube-scheduler-k8s-master-97435445-3 started at 2019-11-05 20:09:05 +0000 UTC (0+1 container statuses recorded) I1105 20:22:09.498] Nov 5 20:22:09.497: INFO: Container kube-scheduler ready: true, restart count 0 I1105 20:22:09.498] Nov 5 20:22:09.497: INFO: azure-ip-masq-agent-747cl started at 2019-11-05 20:09:21 +0000 UTC (0+1 container statuses recorded) I1105 20:22:09.499] Nov 5 20:22:09.497: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 20:22:09.499] Nov 5 20:22:09.497: INFO: kube-proxy-k45g8 started at 2019-11-05 20:09:21 +0000 UTC (0+1 container statuses recorded) I1105 20:22:09.499] Nov 5 20:22:09.497: INFO: Container kube-proxy ready: true, restart count 0 I1105 20:22:09.499] Nov 5 20:22:09.497: INFO: cloud-controller-manager-k8s-master-97435445-3 started at 2019-11-05 20:09:04 +0000 UTC (0+1 container statuses recorded) I1105 20:22:09.499] Nov 5 20:22:09.497: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 20:22:09.755] Nov 5 20:22:09.755: INFO: I1105 20:22:09.755] Latency metrics for node k8s-master-97435445-3 I1105 20:22:09.755] Nov 5 20:22:09.755: INFO: I1105 20:22:09.755] Logging node info for node k8s-master-97435445-4 I1105 20:22:09.836] Nov 5 20:22:09.834: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-97435445-4,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-97435445-4,UID:6d547b41-3f92-4cc4-a2d7-dda2071776a2,ResourceVersion:3517,Generation:0,CreationTimestamp:2019-11-05 20:08:36 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 1,kubernetes.azure.com/cluster: kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-97435445-4,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.3.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b/providers/Microsoft.Compute/virtualMachines/k8s-master-97435445-4,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 20:08:59 +0000 UTC 2019-11-05 20:08:59 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 20:21:58 +0000 UTC 2019-11-05 20:08:29 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 20:21:58 +0000 UTC 2019-11-05 20:08:29 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 20:21:58 +0000 UTC 2019-11-05 20:08:29 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 20:21:58 +0000 UTC 2019-11-05 20:08:36 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.255.9} {Hostname k8s-master-97435445-4}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:56b393c61eb34ba49ebda5279eeedb30,SystemUUID:FD239C23-618C-0049-8734-9AF3FDC9E3C1,BootID:f342ea44-e719-4513-807d-5def7ac606e2,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 20:22:09.837] Nov 5 20:22:09.834: INFO: I1105 20:22:09.837] Logging kubelet events for node k8s-master-97435445-4 I1105 20:22:09.918] Nov 5 20:22:09.918: INFO: I1105 20:22:09.918] Logging pods the kubelet thinks is on node k8s-master-97435445-4 I1105 20:22:10.026] Nov 5 20:22:10.026: INFO: cloud-controller-manager-k8s-master-97435445-4 started at 2019-11-05 20:08:46 +0000 UTC (0+1 container statuses recorded) I1105 20:22:10.026] Nov 5 20:22:10.026: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 20:22:10.027] Nov 5 20:22:10.026: INFO: kube-addon-manager-k8s-master-97435445-4 started at 2019-11-05 20:08:23 +0000 UTC (0+1 container statuses recorded) I1105 20:22:10.027] Nov 5 20:22:10.026: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 20:22:10.027] Nov 5 20:22:10.026: INFO: kube-apiserver-k8s-master-97435445-4 started at 2019-11-05 20:08:23 +0000 UTC (0+1 container statuses recorded) I1105 20:22:10.027] Nov 5 20:22:10.026: INFO: Container kube-apiserver ready: true, restart count 0 I1105 20:22:10.027] Nov 5 20:22:10.026: INFO: kube-controller-manager-k8s-master-97435445-4 started at 2019-11-05 20:08:23 +0000 UTC (0+1 container statuses recorded) I1105 20:22:10.027] Nov 5 20:22:10.026: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 20:22:10.027] Nov 5 20:22:10.026: INFO: kube-scheduler-k8s-master-97435445-4 started at 2019-11-05 20:08:23 +0000 UTC (0+1 container statuses recorded) I1105 20:22:10.027] Nov 5 20:22:10.026: INFO: Container kube-scheduler ready: true, restart count 0 I1105 20:22:10.028] Nov 5 20:22:10.026: INFO: kube-proxy-vfbfr started at 2019-11-05 20:08:36 +0000 UTC (0+1 container statuses recorded) I1105 20:22:10.028] Nov 5 20:22:10.026: INFO: Container kube-proxy ready: true, restart count 0 I1105 20:22:10.028] Nov 5 20:22:10.026: INFO: azure-ip-masq-agent-kmkll started at 2019-11-05 20:08:36 +0000 UTC (0+1 container statuses recorded) I1105 20:22:10.028] Nov 5 20:22:10.026: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 20:22:10.321] Nov 5 20:22:10.321: INFO: I1105 20:22:10.321] Latency metrics for node k8s-master-97435445-4 I1105 20:22:10.322] Nov 5 20:22:10.321: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready I1105 20:22:10.402] STEP: Destroying namespace "azuredisk-8307" for this suite. I1105 20:22:24.655] Nov 5 20:22:24.654: INFO: Waiting up to 30s for server preferred namespaced resources to be successfully discovered I1105 20:22:27.757] Nov 5 20:22:27.757: INFO: namespace azuredisk-8307 deletion completed in 17.354518415s I1105 20:22:27.757] I1105 20:22:27.758] • Failure [324.918 seconds] I1105 20:22:27.758] [azuredisk-csi-e2e] Dynamic Provisioning I1105 20:22:27.758] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:33 I1105 20:22:27.758] [multi-az] I1105 20:22:27.758] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:40 I1105 20:22:27.758] should create a volume on demand [It] I1105 20:22:27.758] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:78 I1105 20:22:27.758] I1105 20:22:27.759] Unexpected error: I1105 20:22:27.759] <*errors.errorString | 0xc000916a40>: { I1105 20:22:27.759] s: "PersistentVolumeClaims [pvc-9g8k7] not all in phase Bound within 5m0s", I1105 20:22:27.759] } I1105 20:22:27.759] PersistentVolumeClaims [pvc-9g8k7] not all in phase Bound within 5m0s I1105 20:22:27.759] occurred I1105 20:22:27.759] I1105 20:22:27.759] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/testsuites/testsuites.go:278 I1105 20:22:27.759] ------------------------------ I1105 20:22:27.760] [azuredisk-csi-e2e] Dynamic Provisioning [single-az] I1105 20:22:27.760] should create a raw block volume on demand I1105 20:22:27.760] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:101 I1105 20:22:27.760] [BeforeEach] [single-az] I1105 20:22:27.760] /go/src/sigs.k8s.io/azuredisk-csi-driver/vendor/k8s.io/kubernetes/test/e2e/framework/framework.go:150 I1105 20:22:27.760] STEP: Creating a kubernetes client I1105 20:22:27.760] Nov 5 20:22:27.757: INFO: >>> kubeConfig: /workspace/aks971561099/kubeconfig/kubeconfig.eastus2.json I1105 20:22:27.760] STEP: Building a namespace api object, basename azuredisk I1105 20:22:28.003] STEP: Binding the e2e-test-privileged-psp PodSecurityPolicy to the default service account in azuredisk-1082 I1105 20:22:28.281] STEP: Waiting for a default service account to be provisioned in namespace I1105 20:22:28.357] [BeforeEach] [single-az] I1105 20:22:28.357] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:58 I1105 20:22:28.357] [It] should create a raw block volume on demand I1105 20:22:28.357] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:101 I1105 20:22:28.357] STEP: setting up the StorageClass I1105 20:22:28.357] STEP: creating a StorageClass I1105 20:22:28.440] STEP: setting up the PVC and PV I1105 20:22:28.440] STEP: creating a PVC I1105 20:22:28.530] STEP: waiting for PVC to be in phase "Bound" I1105 20:22:28.531] Nov 5 20:22:28.530: INFO: Waiting up to 5m0s for PersistentVolumeClaims [pvc-tvv9z] to have phase Bound I1105 20:22:28.609] Nov 5 20:22:28.609: INFO: PersistentVolumeClaim pvc-tvv9z found but phase is Pending instead of Bound. I1105 20:22:30.690] Nov 5 20:22:30.689: INFO: PersistentVolumeClaim pvc-tvv9z found but phase is Pending instead of Bound. I1105 20:22:32.769] Nov 5 20:22:32.769: INFO: PersistentVolumeClaim pvc-tvv9z found but phase is Pending instead of Bound. I1105 20:22:34.849] Nov 5 20:22:34.849: INFO: PersistentVolumeClaim pvc-tvv9z found but phase is Pending instead of Bound. I1105 20:22:36.931] Nov 5 20:22:36.931: INFO: PersistentVolumeClaim pvc-tvv9z found but phase is Pending instead of Bound. I1105 20:22:39.010] Nov 5 20:22:39.010: INFO: PersistentVolumeClaim pvc-tvv9z found but phase is Pending instead of Bound. I1105 20:22:41.089] Nov 5 20:22:41.089: INFO: PersistentVolumeClaim pvc-tvv9z found but phase is Pending instead of Bound. I1105 20:22:43.169] Nov 5 20:22:43.169: INFO: PersistentVolumeClaim pvc-tvv9z found but phase is Pending instead of Bound. I1105 20:22:45.248] Nov 5 20:22:45.248: INFO: PersistentVolumeClaim pvc-tvv9z found but phase is Pending instead of Bound. I1105 20:22:47.327] Nov 5 20:22:47.327: INFO: PersistentVolumeClaim pvc-tvv9z found but phase is Pending instead of Bound. I1105 20:22:49.406] Nov 5 20:22:49.406: INFO: PersistentVolumeClaim pvc-tvv9z found but phase is Pending instead of Bound. I1105 20:22:51.486] Nov 5 20:22:51.485: INFO: PersistentVolumeClaim pvc-tvv9z found but phase is Pending instead of Bound. I1105 20:22:53.565] Nov 5 20:22:53.565: INFO: PersistentVolumeClaim pvc-tvv9z found but phase is Pending instead of Bound. I1105 20:22:55.645] Nov 5 20:22:55.644: INFO: PersistentVolumeClaim pvc-tvv9z found but phase is Pending instead of Bound. I1105 20:22:57.724] Nov 5 20:22:57.724: INFO: PersistentVolumeClaim pvc-tvv9z found but phase is Pending instead of Bound. I1105 20:22:59.804] Nov 5 20:22:59.803: INFO: PersistentVolumeClaim pvc-tvv9z found but phase is Pending instead of Bound. I1105 20:23:01.882] Nov 5 20:23:01.882: INFO: PersistentVolumeClaim pvc-tvv9z found but phase is Pending instead of Bound. I1105 20:23:03.961] Nov 5 20:23:03.961: INFO: PersistentVolumeClaim pvc-tvv9z found but phase is Pending instead of Bound. I1105 20:23:06.041] Nov 5 20:23:06.041: INFO: PersistentVolumeClaim pvc-tvv9z found but phase is Pending instead of Bound. I1105 20:23:08.120] Nov 5 20:23:08.120: INFO: PersistentVolumeClaim pvc-tvv9z found but phase is Pending instead of Bound. I1105 20:23:10.213] Nov 5 20:23:10.213: INFO: PersistentVolumeClaim pvc-tvv9z found but phase is Pending instead of Bound. I1105 20:23:12.292] Nov 5 20:23:12.292: INFO: PersistentVolumeClaim pvc-tvv9z found but phase is Pending instead of Bound. I1105 20:23:14.371] Nov 5 20:23:14.371: INFO: PersistentVolumeClaim pvc-tvv9z found but phase is Pending instead of Bound. I1105 20:23:16.450] Nov 5 20:23:16.450: INFO: PersistentVolumeClaim pvc-tvv9z found but phase is Pending instead of Bound. I1105 20:23:18.530] Nov 5 20:23:18.530: INFO: PersistentVolumeClaim pvc-tvv9z found but phase is Pending instead of Bound. I1105 20:23:20.609] Nov 5 20:23:20.609: INFO: PersistentVolumeClaim pvc-tvv9z found but phase is Pending instead of Bound. I1105 20:23:22.690] Nov 5 20:23:22.689: INFO: PersistentVolumeClaim pvc-tvv9z found but phase is Pending instead of Bound. I1105 20:23:24.769] Nov 5 20:23:24.769: INFO: PersistentVolumeClaim pvc-tvv9z found but phase is Pending instead of Bound. I1105 20:23:26.848] Nov 5 20:23:26.848: INFO: PersistentVolumeClaim pvc-tvv9z found but phase is Pending instead of Bound. I1105 20:23:28.927] Nov 5 20:23:28.927: INFO: PersistentVolumeClaim pvc-tvv9z found but phase is Pending instead of Bound. I1105 20:23:31.006] Nov 5 20:23:31.006: INFO: PersistentVolumeClaim pvc-tvv9z found but phase is Pending instead of Bound. I1105 20:23:33.085] Nov 5 20:23:33.085: INFO: PersistentVolumeClaim pvc-tvv9z found but phase is Pending instead of Bound. I1105 20:23:35.164] Nov 5 20:23:35.164: INFO: PersistentVolumeClaim pvc-tvv9z found but phase is Pending instead of Bound. I1105 20:23:37.243] Nov 5 20:23:37.243: INFO: PersistentVolumeClaim pvc-tvv9z found but phase is Pending instead of Bound. I1105 20:23:39.323] Nov 5 20:23:39.323: INFO: PersistentVolumeClaim pvc-tvv9z found but phase is Pending instead of Bound. I1105 20:23:41.402] Nov 5 20:23:41.402: INFO: PersistentVolumeClaim pvc-tvv9z found but phase is Pending instead of Bound. I1105 20:23:43.481] Nov 5 20:23:43.481: INFO: PersistentVolumeClaim pvc-tvv9z found but phase is Pending instead of Bound. I1105 20:23:45.560] Nov 5 20:23:45.560: INFO: PersistentVolumeClaim pvc-tvv9z found but phase is Pending instead of Bound. I1105 20:23:47.640] Nov 5 20:23:47.640: INFO: PersistentVolumeClaim pvc-tvv9z found but phase is Pending instead of Bound. I1105 20:23:49.719] Nov 5 20:23:49.719: INFO: PersistentVolumeClaim pvc-tvv9z found but phase is Pending instead of Bound. I1105 20:23:51.798] Nov 5 20:23:51.798: INFO: PersistentVolumeClaim pvc-tvv9z found but phase is Pending instead of Bound. I1105 20:23:53.877] Nov 5 20:23:53.877: INFO: PersistentVolumeClaim pvc-tvv9z found but phase is Pending instead of Bound. I1105 20:23:55.956] Nov 5 20:23:55.956: INFO: PersistentVolumeClaim pvc-tvv9z found but phase is Pending instead of Bound. I1105 20:23:58.040] Nov 5 20:23:58.040: INFO: PersistentVolumeClaim pvc-tvv9z found but phase is Pending instead of Bound. I1105 20:24:00.122] Nov 5 20:24:00.122: INFO: PersistentVolumeClaim pvc-tvv9z found but phase is Pending instead of Bound. I1105 20:24:02.218] Nov 5 20:24:02.218: INFO: PersistentVolumeClaim pvc-tvv9z found but phase is Pending instead of Bound. I1105 20:24:04.297] Nov 5 20:24:04.297: INFO: PersistentVolumeClaim pvc-tvv9z found but phase is Pending instead of Bound. I1105 20:24:06.377] Nov 5 20:24:06.376: INFO: PersistentVolumeClaim pvc-tvv9z found but phase is Pending instead of Bound. I1105 20:24:08.456] Nov 5 20:24:08.456: INFO: PersistentVolumeClaim pvc-tvv9z found but phase is Pending instead of Bound. I1105 20:24:10.536] Nov 5 20:24:10.536: INFO: PersistentVolumeClaim pvc-tvv9z found but phase is Pending instead of Bound. I1105 20:24:12.623] Nov 5 20:24:12.622: INFO: PersistentVolumeClaim pvc-tvv9z found but phase is Pending instead of Bound. I1105 20:24:14.702] Nov 5 20:24:14.701: INFO: PersistentVolumeClaim pvc-tvv9z found but phase is Pending instead of Bound. I1105 20:24:16.781] Nov 5 20:24:16.781: INFO: PersistentVolumeClaim pvc-tvv9z found but phase is Pending instead of Bound. I1105 20:24:18.868] Nov 5 20:24:18.867: INFO: PersistentVolumeClaim pvc-tvv9z found but phase is Pending instead of Bound. I1105 20:24:20.947] Nov 5 20:24:20.947: INFO: PersistentVolumeClaim pvc-tvv9z found but phase is Pending instead of Bound. I1105 20:24:23.028] Nov 5 20:24:23.028: INFO: PersistentVolumeClaim pvc-tvv9z found but phase is Pending instead of Bound. I1105 20:24:25.108] Nov 5 20:24:25.107: INFO: PersistentVolumeClaim pvc-tvv9z found but phase is Pending instead of Bound. I1105 20:24:27.186] Nov 5 20:24:27.186: INFO: PersistentVolumeClaim pvc-tvv9z found but phase is Pending instead of Bound. I1105 20:24:29.265] Nov 5 20:24:29.265: INFO: PersistentVolumeClaim pvc-tvv9z found but phase is Pending instead of Bound. I1105 20:24:31.344] Nov 5 20:24:31.344: INFO: PersistentVolumeClaim pvc-tvv9z found but phase is Pending instead of Bound. I1105 20:24:33.424] Nov 5 20:24:33.424: INFO: PersistentVolumeClaim pvc-tvv9z found but phase is Pending instead of Bound. I1105 20:24:35.502] Nov 5 20:24:35.502: INFO: PersistentVolumeClaim pvc-tvv9z found but phase is Pending instead of Bound. I1105 20:24:37.582] Nov 5 20:24:37.581: INFO: PersistentVolumeClaim pvc-tvv9z found but phase is Pending instead of Bound. I1105 20:24:39.661] Nov 5 20:24:39.661: INFO: PersistentVolumeClaim pvc-tvv9z found but phase is Pending instead of Bound. I1105 20:24:41.740] Nov 5 20:24:41.740: INFO: PersistentVolumeClaim pvc-tvv9z found but phase is Pending instead of Bound. I1105 20:24:43.819] Nov 5 20:24:43.819: INFO: PersistentVolumeClaim pvc-tvv9z found but phase is Pending instead of Bound. I1105 20:24:45.898] Nov 5 20:24:45.898: INFO: PersistentVolumeClaim pvc-tvv9z found but phase is Pending instead of Bound. I1105 20:24:47.978] Nov 5 20:24:47.977: INFO: PersistentVolumeClaim pvc-tvv9z found but phase is Pending instead of Bound. I1105 20:24:50.057] Nov 5 20:24:50.057: INFO: PersistentVolumeClaim pvc-tvv9z found but phase is Pending instead of Bound. I1105 20:24:52.138] Nov 5 20:24:52.138: INFO: PersistentVolumeClaim pvc-tvv9z found but phase is Pending instead of Bound. I1105 20:24:54.218] Nov 5 20:24:54.217: INFO: PersistentVolumeClaim pvc-tvv9z found but phase is Pending instead of Bound. I1105 20:24:56.298] Nov 5 20:24:56.298: INFO: PersistentVolumeClaim pvc-tvv9z found but phase is Pending instead of Bound. I1105 20:24:58.378] Nov 5 20:24:58.377: INFO: PersistentVolumeClaim pvc-tvv9z found but phase is Pending instead of Bound. I1105 20:25:00.457] Nov 5 20:25:00.457: INFO: PersistentVolumeClaim pvc-tvv9z found but phase is Pending instead of Bound. I1105 20:25:02.536] Nov 5 20:25:02.536: INFO: PersistentVolumeClaim pvc-tvv9z found but phase is Pending instead of Bound. I1105 20:25:04.615] Nov 5 20:25:04.615: INFO: PersistentVolumeClaim pvc-tvv9z found but phase is Pending instead of Bound. I1105 20:25:06.695] Nov 5 20:25:06.694: INFO: PersistentVolumeClaim pvc-tvv9z found but phase is Pending instead of Bound. I1105 20:25:08.774] Nov 5 20:25:08.774: INFO: PersistentVolumeClaim pvc-tvv9z found but phase is Pending instead of Bound. I1105 20:25:10.856] Nov 5 20:25:10.856: INFO: PersistentVolumeClaim pvc-tvv9z found but phase is Pending instead of Bound. I1105 20:25:12.935] Nov 5 20:25:12.935: INFO: PersistentVolumeClaim pvc-tvv9z found but phase is Pending instead of Bound. I1105 20:25:15.015] Nov 5 20:25:15.015: INFO: PersistentVolumeClaim pvc-tvv9z found but phase is Pending instead of Bound. I1105 20:25:17.095] Nov 5 20:25:17.095: INFO: PersistentVolumeClaim pvc-tvv9z found but phase is Pending instead of Bound. I1105 20:25:19.175] Nov 5 20:25:19.175: INFO: PersistentVolumeClaim pvc-tvv9z found but phase is Pending instead of Bound. I1105 20:25:21.254] Nov 5 20:25:21.254: INFO: PersistentVolumeClaim pvc-tvv9z found but phase is Pending instead of Bound. I1105 20:25:23.333] Nov 5 20:25:23.333: INFO: PersistentVolumeClaim pvc-tvv9z found but phase is Pending instead of Bound. I1105 20:25:25.413] Nov 5 20:25:25.412: INFO: PersistentVolumeClaim pvc-tvv9z found but phase is Pending instead of Bound. I1105 20:25:27.492] Nov 5 20:25:27.492: INFO: PersistentVolumeClaim pvc-tvv9z found but phase is Pending instead of Bound. I1105 20:25:29.571] Nov 5 20:25:29.571: INFO: PersistentVolumeClaim pvc-tvv9z found but phase is Pending instead of Bound. I1105 20:25:31.650] Nov 5 20:25:31.650: INFO: PersistentVolumeClaim pvc-tvv9z found but phase is Pending instead of Bound. I1105 20:25:33.735] Nov 5 20:25:33.735: INFO: PersistentVolumeClaim pvc-tvv9z found but phase is Pending instead of Bound. I1105 20:25:35.814] Nov 5 20:25:35.814: INFO: PersistentVolumeClaim pvc-tvv9z found but phase is Pending instead of Bound. I1105 20:25:37.893] Nov 5 20:25:37.893: INFO: PersistentVolumeClaim pvc-tvv9z found but phase is Pending instead of Bound. I1105 20:25:39.973] Nov 5 20:25:39.972: INFO: PersistentVolumeClaim pvc-tvv9z found but phase is Pending instead of Bound. I1105 20:25:42.052] Nov 5 20:25:42.052: INFO: PersistentVolumeClaim pvc-tvv9z found but phase is Pending instead of Bound. I1105 20:25:44.131] Nov 5 20:25:44.131: INFO: PersistentVolumeClaim pvc-tvv9z found but phase is Pending instead of Bound. I1105 20:25:46.214] Nov 5 20:25:46.214: INFO: PersistentVolumeClaim pvc-tvv9z found but phase is Pending instead of Bound. I1105 20:25:48.293] Nov 5 20:25:48.293: INFO: PersistentVolumeClaim pvc-tvv9z found but phase is Pending instead of Bound. I1105 20:25:50.373] Nov 5 20:25:50.373: INFO: PersistentVolumeClaim pvc-tvv9z found but phase is Pending instead of Bound. I1105 20:25:52.452] Nov 5 20:25:52.452: INFO: PersistentVolumeClaim pvc-tvv9z found but phase is Pending instead of Bound. I1105 20:25:54.531] Nov 5 20:25:54.531: INFO: PersistentVolumeClaim pvc-tvv9z found but phase is Pending instead of Bound. I1105 20:25:56.610] Nov 5 20:25:56.610: INFO: PersistentVolumeClaim pvc-tvv9z found but phase is Pending instead of Bound. I1105 20:25:58.689] Nov 5 20:25:58.689: INFO: PersistentVolumeClaim pvc-tvv9z found but phase is Pending instead of Bound. I1105 20:26:00.768] Nov 5 20:26:00.768: INFO: PersistentVolumeClaim pvc-tvv9z found but phase is Pending instead of Bound. I1105 20:26:02.847] Nov 5 20:26:02.847: INFO: PersistentVolumeClaim pvc-tvv9z found but phase is Pending instead of Bound. I1105 20:26:04.926] Nov 5 20:26:04.926: INFO: PersistentVolumeClaim pvc-tvv9z found but phase is Pending instead of Bound. I1105 20:26:07.005] Nov 5 20:26:07.005: INFO: PersistentVolumeClaim pvc-tvv9z found but phase is Pending instead of Bound. I1105 20:26:09.085] Nov 5 20:26:09.085: INFO: PersistentVolumeClaim pvc-tvv9z found but phase is Pending instead of Bound. I1105 20:26:11.165] Nov 5 20:26:11.165: INFO: PersistentVolumeClaim pvc-tvv9z found but phase is Pending instead of Bound. I1105 20:26:13.245] Nov 5 20:26:13.245: INFO: PersistentVolumeClaim pvc-tvv9z found but phase is Pending instead of Bound. I1105 20:26:15.324] Nov 5 20:26:15.324: INFO: PersistentVolumeClaim pvc-tvv9z found but phase is Pending instead of Bound. I1105 20:26:17.403] Nov 5 20:26:17.402: INFO: PersistentVolumeClaim pvc-tvv9z found but phase is Pending instead of Bound. I1105 20:26:19.482] Nov 5 20:26:19.482: INFO: PersistentVolumeClaim pvc-tvv9z found but phase is Pending instead of Bound. I1105 20:26:21.562] Nov 5 20:26:21.562: INFO: PersistentVolumeClaim pvc-tvv9z found but phase is Pending instead of Bound. I1105 20:26:23.641] Nov 5 20:26:23.641: INFO: PersistentVolumeClaim pvc-tvv9z found but phase is Pending instead of Bound. I1105 20:26:25.721] Nov 5 20:26:25.721: INFO: PersistentVolumeClaim pvc-tvv9z found but phase is Pending instead of Bound. I1105 20:26:27.802] Nov 5 20:26:27.802: INFO: PersistentVolumeClaim pvc-tvv9z found but phase is Pending instead of Bound. I1105 20:26:29.882] Nov 5 20:26:29.882: INFO: PersistentVolumeClaim pvc-tvv9z found but phase is Pending instead of Bound. I1105 20:26:31.962] Nov 5 20:26:31.961: INFO: PersistentVolumeClaim pvc-tvv9z found but phase is Pending instead of Bound. I1105 20:26:34.041] Nov 5 20:26:34.041: INFO: PersistentVolumeClaim pvc-tvv9z found but phase is Pending instead of Bound. I1105 20:26:36.123] Nov 5 20:26:36.123: INFO: PersistentVolumeClaim pvc-tvv9z found but phase is Pending instead of Bound. I1105 20:26:38.211] Nov 5 20:26:38.210: INFO: PersistentVolumeClaim pvc-tvv9z found but phase is Pending instead of Bound. I1105 20:26:40.290] Nov 5 20:26:40.290: INFO: PersistentVolumeClaim pvc-tvv9z found but phase is Pending instead of Bound. I1105 20:26:42.370] Nov 5 20:26:42.369: INFO: PersistentVolumeClaim pvc-tvv9z found but phase is Pending instead of Bound. I1105 20:26:44.449] Nov 5 20:26:44.449: INFO: PersistentVolumeClaim pvc-tvv9z found but phase is Pending instead of Bound. I1105 20:26:46.529] Nov 5 20:26:46.529: INFO: PersistentVolumeClaim pvc-tvv9z found but phase is Pending instead of Bound. I1105 20:26:48.609] Nov 5 20:26:48.608: INFO: PersistentVolumeClaim pvc-tvv9z found but phase is Pending instead of Bound. I1105 20:26:50.687] Nov 5 20:26:50.687: INFO: PersistentVolumeClaim pvc-tvv9z found but phase is Pending instead of Bound. I1105 20:26:52.766] Nov 5 20:26:52.766: INFO: PersistentVolumeClaim pvc-tvv9z found but phase is Pending instead of Bound. I1105 20:26:54.846] Nov 5 20:26:54.846: INFO: PersistentVolumeClaim pvc-tvv9z found but phase is Pending instead of Bound. I1105 20:26:56.926] Nov 5 20:26:56.925: INFO: PersistentVolumeClaim pvc-tvv9z found but phase is Pending instead of Bound. I1105 20:26:59.005] Nov 5 20:26:59.005: INFO: PersistentVolumeClaim pvc-tvv9z found but phase is Pending instead of Bound. I1105 20:27:01.084] Nov 5 20:27:01.084: INFO: PersistentVolumeClaim pvc-tvv9z found but phase is Pending instead of Bound. I1105 20:27:03.163] Nov 5 20:27:03.163: INFO: PersistentVolumeClaim pvc-tvv9z found but phase is Pending instead of Bound. I1105 20:27:05.243] Nov 5 20:27:05.242: INFO: PersistentVolumeClaim pvc-tvv9z found but phase is Pending instead of Bound. I1105 20:27:07.322] Nov 5 20:27:07.321: INFO: PersistentVolumeClaim pvc-tvv9z found but phase is Pending instead of Bound. I1105 20:27:09.401] Nov 5 20:27:09.401: INFO: PersistentVolumeClaim pvc-tvv9z found but phase is Pending instead of Bound. I1105 20:27:11.480] Nov 5 20:27:11.480: INFO: PersistentVolumeClaim pvc-tvv9z found but phase is Pending instead of Bound. I1105 20:27:13.560] Nov 5 20:27:13.560: INFO: PersistentVolumeClaim pvc-tvv9z found but phase is Pending instead of Bound. I1105 20:27:15.639] Nov 5 20:27:15.638: INFO: PersistentVolumeClaim pvc-tvv9z found but phase is Pending instead of Bound. I1105 20:27:17.718] Nov 5 20:27:17.718: INFO: PersistentVolumeClaim pvc-tvv9z found but phase is Pending instead of Bound. I1105 20:27:19.798] Nov 5 20:27:19.797: INFO: PersistentVolumeClaim pvc-tvv9z found but phase is Pending instead of Bound. I1105 20:27:21.877] Nov 5 20:27:21.877: INFO: PersistentVolumeClaim pvc-tvv9z found but phase is Pending instead of Bound. I1105 20:27:23.957] Nov 5 20:27:23.957: INFO: PersistentVolumeClaim pvc-tvv9z found but phase is Pending instead of Bound. I1105 20:27:26.036] Nov 5 20:27:26.036: INFO: PersistentVolumeClaim pvc-tvv9z found but phase is Pending instead of Bound. I1105 20:27:28.116] Nov 5 20:27:28.115: INFO: PersistentVolumeClaim pvc-tvv9z found but phase is Pending instead of Bound. I1105 20:27:30.116] Nov 5 20:27:30.116: INFO: Unexpected error occurred: PersistentVolumeClaims [pvc-tvv9z] not all in phase Bound within 5m0s I1105 20:27:30.116] [AfterEach] [single-az] I1105 20:27:30.117] /go/src/sigs.k8s.io/azuredisk-csi-driver/vendor/k8s.io/kubernetes/test/e2e/framework/framework.go:151 I1105 20:27:30.117] STEP: Collecting events from namespace "azuredisk-1082". I1105 20:27:30.221] STEP: Found 3 events. I1105 20:27:30.221] Nov 5 20:27:30.221: INFO: At 2019-11-05 20:22:28 +0000 UTC - event for pvc-tvv9z: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "disk.csi.azure.com" or manually created by system administrator I1105 20:27:30.222] Nov 5 20:27:30.221: INFO: At 2019-11-05 20:22:28 +0000 UTC - event for pvc-tvv9z: {disk.csi.azure.com_csi-azuredisk-controller-6f6587bfc6-62nrt_3930bc69-0009-11ea-8cf5-829a27ff7d5a } Provisioning: External provisioner is provisioning volume for claim "azuredisk-1082/pvc-tvv9z" I1105 20:27:30.222] Nov 5 20:27:30.221: INFO: At 2019-11-05 20:22:28 +0000 UTC - event for pvc-tvv9z: {disk.csi.azure.com_csi-azuredisk-controller-6f6587bfc6-62nrt_3930bc69-0009-11ea-8cf5-829a27ff7d5a } ProvisioningFailed: failed to provision volume with StorageClass "azuredisk-1082-disk.csi.azure.com-dynamic-sc-9lfxl": rpc error: code = Unknown desc = compute.DisksClient#CreateOrUpdate: Failure sending request: StatusCode=400 -- Original Error: Code="InvalidAvailabilityZone" Message="The zone(s) '' for resource 'Microsoft.Compute/disks/pvc-1992216b-619e-4853-9cbf-6eed3e7ef781' is not supported. The supported zones for location 'eastus2' are '1,2,3'" I1105 20:27:30.299] Nov 5 20:27:30.299: INFO: POD NODE PHASE GRACE CONDITIONS I1105 20:27:30.299] Nov 5 20:27:30.299: INFO: I1105 20:27:30.534] Nov 5 20:27:30.534: INFO: I1105 20:27:30.534] Logging node info for node k8s-agentpool-97435445-vmss000000 I1105 20:27:30.708] Nov 5 20:27:30.705: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-agentpool-97435445-vmss000000,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-agentpool-97435445-vmss000000,UID:e955dc47-8c8f-4142-9241-3d409f32e42d,ResourceVersion:4354,Generation:0,CreationTimestamp:2019-11-05 20:07:34 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{agentpool: agentpool,beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_D4s_v3,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 0,kubernetes.azure.com/cluster: kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b,kubernetes.azure.com/role: agent,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-agentpool-97435445-vmss000000,kubernetes.io/os: linux,kubernetes.io/role: agent,node-role.kubernetes.io/agent: ,storageprofile: managed,storagetier: Premium_LRS,topology.disk.csi.azure.com/zone: 0,},Annotations:map[string]string{csi.volume.kubernetes.io/nodeid: {"disk.csi.azure.com":"k8s-agentpool-97435445-vmss000000"},node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.2.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool-97435445-vmss/virtualMachines/0,Unschedulable:false,Taints:[],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-csi-disk.csi.azure.com: {{8 0} {} 8 DecimalSI},cpu: {{4 0} {} 4 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{16797040640 0} {} BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-csi-disk.csi.azure.com: {{8 0} {} 8 DecimalSI},cpu: {{4 0} {} 4 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{16010608640 0} {} BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 20:08:01 +0000 UTC 2019-11-05 20:08:01 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 20:26:38 +0000 UTC 2019-11-05 20:07:27 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 20:26:38 +0000 UTC 2019-11-05 20:07:27 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 20:26:38 +0000 UTC 2019-11-05 20:07:27 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 20:26:38 +0000 UTC 2019-11-05 20:07:55 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.0.4} {Hostname k8s-agentpool-97435445-vmss000000}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:bfb55325809340228149d2d93f50df58,SystemUUID:CB2ED036-0439-F641-BC7F-640C3638C027,BootID:a917f4d1-549d-471d-b7dc-47ae1715d97d,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[k8sprowinternal.azurecr.io/azuredisk-csi@sha256:75d4ff5d7cd845d6e810c67796ecb037e852fb849ec1eb5b3194564008e6d1e0 k8sprowinternal.azurecr.io/azuredisk-csi:e2e-3e3ed045c6faa509dbd8d2890ee9f56aa2934887] 286930414} {[k8s.gcr.io/kubernetes-dashboard-amd64@sha256:0ae6b69432e78069c5ce2bcde0fe409c5c4d6f0f4d9cd50a17974fea38898747 k8s.gcr.io/kubernetes-dashboard-amd64:v1.10.1] 121711221} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/metrics-server-amd64@sha256:49a9f12f7067d11f42c803dbe61ed2c1299959ad85cb315b25ff7eef8e6b8892 k8s.gcr.io/metrics-server-amd64:v0.2.1] 42541759} {[k8s.gcr.io/coredns@sha256:e83beb5e43f8513fa735e77ffc5859640baea30a882a11cc75c4c3244a737d3c k8s.gcr.io/coredns:1.5.0] 42488424} {[mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume@sha256:4fd30d43947d4a54fc89ead7985beecfd3c9b2a93a0655a373b1608ab90bd5af mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume:v0.0.7] 22909487} {[quay.io/k8scsi/livenessprobe@sha256:8c9c8a0f330acc2e558ae6624309447d02522e76aa35af23fa19b88abf4381cb quay.io/k8scsi/livenessprobe:v1.0.2] 17415381} {[quay.io/k8scsi/csi-node-driver-registrar@sha256:13daf82fb99e951a4bff8ae5fc7c17c3a8fe7130be6400990d8f6076c32d4599 quay.io/k8scsi/csi-node-driver-registrar:v1.1.0] 15815995} {[mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume@sha256:23d8c6033f02a1ecad05127ebdc931bb871264228661bc122704b0974e4d9fdd mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume:1.0.8] 1159025} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 20:27:30.708] Nov 5 20:27:30.705: INFO: I1105 20:27:30.708] Logging kubelet events for node k8s-agentpool-97435445-vmss000000 I1105 20:27:30.790] Nov 5 20:27:30.790: INFO: I1105 20:27:30.790] Logging pods the kubelet thinks is on node k8s-agentpool-97435445-vmss000000 I1105 20:27:30.876] Nov 5 20:27:30.876: INFO: blobfuse-flexvol-installer-p7hdb started at 2019-11-05 20:08:01 +0000 UTC (0+1 container statuses recorded) I1105 20:27:30.876] Nov 5 20:27:30.876: INFO: Container blobfuse-flexvol-installer ready: true, restart count 0 I1105 20:27:30.876] Nov 5 20:27:30.876: INFO: coredns-7f68dcdbdb-zh8g5 started at 2019-11-05 20:08:01 +0000 UTC (0+1 container statuses recorded) I1105 20:27:30.876] Nov 5 20:27:30.876: INFO: Container coredns ready: true, restart count 0 I1105 20:27:30.877] Nov 5 20:27:30.876: INFO: azure-ip-masq-agent-m7tv9 started at 2019-11-05 20:08:01 +0000 UTC (0+1 container statuses recorded) I1105 20:27:30.877] Nov 5 20:27:30.876: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 20:27:30.877] Nov 5 20:27:30.876: INFO: keyvault-flexvolume-smspt started at 2019-11-05 20:08:01 +0000 UTC (0+1 container statuses recorded) I1105 20:27:30.877] Nov 5 20:27:30.876: INFO: Container keyvault-flexvolume ready: true, restart count 0 I1105 20:27:30.877] Nov 5 20:27:30.876: INFO: metrics-server-864ffbc5c-wl84z started at 2019-11-05 20:08:01 +0000 UTC (0+1 container statuses recorded) I1105 20:27:30.878] Nov 5 20:27:30.876: INFO: Container metrics-server ready: true, restart count 0 I1105 20:27:30.878] Nov 5 20:27:30.876: INFO: kube-proxy-8mt7f started at 2019-11-05 20:08:01 +0000 UTC (0+1 container statuses recorded) I1105 20:27:30.878] Nov 5 20:27:30.876: INFO: Container kube-proxy ready: true, restart count 0 I1105 20:27:30.878] Nov 5 20:27:30.876: INFO: kubernetes-dashboard-66dd8b8df7-t6j4l started at 2019-11-05 20:08:01 +0000 UTC (0+1 container statuses recorded) I1105 20:27:30.878] Nov 5 20:27:30.876: INFO: Container kubernetes-dashboard ready: true, restart count 0 I1105 20:27:30.879] Nov 5 20:27:30.876: INFO: csi-azuredisk-node-5wwhz started at 2019-11-05 20:16:09 +0000 UTC (0+3 container statuses recorded) I1105 20:27:30.879] Nov 5 20:27:30.876: INFO: Container azuredisk ready: true, restart count 0 I1105 20:27:30.879] Nov 5 20:27:30.876: INFO: Container liveness-probe ready: true, restart count 0 I1105 20:27:30.879] Nov 5 20:27:30.876: INFO: Container node-driver-registrar ready: true, restart count 0 I1105 20:27:31.136] Nov 5 20:27:31.136: INFO: I1105 20:27:31.136] Latency metrics for node k8s-agentpool-97435445-vmss000000 I1105 20:27:31.136] Nov 5 20:27:31.136: INFO: I1105 20:27:31.137] Logging node info for node k8s-agentpool-97435445-vmss000001 I1105 20:27:31.218] Nov 5 20:27:31.215: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-agentpool-97435445-vmss000001,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-agentpool-97435445-vmss000001,UID:75d3cc50-ca6d-45e0-a42a-6090e8d537dd,ResourceVersion:4347,Generation:0,CreationTimestamp:2019-11-05 20:07:34 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{agentpool: agentpool,beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_D4s_v3,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 1,kubernetes.azure.com/cluster: kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b,kubernetes.azure.com/role: agent,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-agentpool-97435445-vmss000001,kubernetes.io/os: linux,kubernetes.io/role: agent,node-role.kubernetes.io/agent: ,storageprofile: managed,storagetier: Premium_LRS,topology.disk.csi.azure.com/zone: 1,},Annotations:map[string]string{csi.volume.kubernetes.io/nodeid: {"disk.csi.azure.com":"k8s-agentpool-97435445-vmss000001"},node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.0.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool-97435445-vmss/virtualMachines/1,Unschedulable:false,Taints:[],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-csi-disk.csi.azure.com: {{8 0} {} 8 DecimalSI},cpu: {{4 0} {} 4 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{16797044736 0} {} 16403364Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-csi-disk.csi.azure.com: {{8 0} {} 8 DecimalSI},cpu: {{4 0} {} 4 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{16010612736 0} {} 15635364Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 20:08:25 +0000 UTC 2019-11-05 20:08:25 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 20:26:37 +0000 UTC 2019-11-05 20:07:26 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 20:26:37 +0000 UTC 2019-11-05 20:07:26 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 20:26:37 +0000 UTC 2019-11-05 20:07:26 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 20:26:37 +0000 UTC 2019-11-05 20:07:55 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.0.5} {Hostname k8s-agentpool-97435445-vmss000001}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:e78def54b2f545c6b94d88bbba523c60,SystemUUID:40AE8188-86AB-D848-84A4-D9254E1258BC,BootID:8587ec06-f4f4-43c1-86cd-06c295b05b63,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[k8sprowinternal.azurecr.io/azuredisk-csi@sha256:75d4ff5d7cd845d6e810c67796ecb037e852fb849ec1eb5b3194564008e6d1e0 k8sprowinternal.azurecr.io/azuredisk-csi:e2e-3e3ed045c6faa509dbd8d2890ee9f56aa2934887] 286930414} {[gcr.io/kubernetes-helm/tiller@sha256:f6d8f4ab9ba993b5f5b60a6edafe86352eabe474ffeb84cb6c79b8866dce45d1 gcr.io/kubernetes-helm/tiller:v2.11.0] 71821984} {[quay.io/k8scsi/csi-provisioner@sha256:d657c839dce87324fe2b677302913f9386f885f8746be7bea0ced5b0844e3433 quay.io/k8scsi/csi-provisioner:v1.3.1] 50596998} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[quay.io/k8scsi/csi-attacher@sha256:26fccd7a99d973845df1193b46ebdcc6ab8dc5f6e6be319750c471fce1742d13 quay.io/k8scsi/csi-attacher:v1.2.0] 46226754} {[quay.io/k8scsi/csi-cluster-driver-registrar@sha256:fafd75ae5442f192cfa8c2e792903aee30d5884b62e802e4464b0a895d21e3ef quay.io/k8scsi/csi-cluster-driver-registrar:v1.0.1] 45874691} {[quay.io/k8scsi/csi-snapshotter@sha256:a49e0da1af6f2bf717e41ba1eee8b5e6a1cbd66a709dd92cc43fe475fe2589eb quay.io/k8scsi/csi-snapshotter:v1.1.0] 44325352} {[mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume@sha256:4fd30d43947d4a54fc89ead7985beecfd3c9b2a93a0655a373b1608ab90bd5af mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume:v0.0.7] 22909487} {[quay.io/k8scsi/livenessprobe@sha256:8c9c8a0f330acc2e558ae6624309447d02522e76aa35af23fa19b88abf4381cb quay.io/k8scsi/livenessprobe:v1.0.2] 17415381} {[quay.io/k8scsi/csi-node-driver-registrar@sha256:13daf82fb99e951a4bff8ae5fc7c17c3a8fe7130be6400990d8f6076c32d4599 quay.io/k8scsi/csi-node-driver-registrar:v1.1.0] 15815995} {[quay.io/k8scsi/livenessprobe@sha256:dde617756e0f602adc566ab71fd885f1dad451ad3fb063ac991c95a2ff47aea5 quay.io/k8scsi/livenessprobe:v1.1.0] 14967303} {[mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume@sha256:23d8c6033f02a1ecad05127ebdc931bb871264228661bc122704b0974e4d9fdd mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume:1.0.8] 1159025} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 20:27:31.219] Nov 5 20:27:31.215: INFO: I1105 20:27:31.219] Logging kubelet events for node k8s-agentpool-97435445-vmss000001 I1105 20:27:31.299] Nov 5 20:27:31.299: INFO: I1105 20:27:31.299] Logging pods the kubelet thinks is on node k8s-agentpool-97435445-vmss000001 I1105 20:27:31.384] Nov 5 20:27:31.383: INFO: azure-ip-masq-agent-gdhcq started at 2019-11-05 20:07:59 +0000 UTC (0+1 container statuses recorded) I1105 20:27:31.384] Nov 5 20:27:31.384: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 20:27:31.384] Nov 5 20:27:31.384: INFO: keyvault-flexvolume-6bffx started at 2019-11-05 20:07:59 +0000 UTC (0+1 container statuses recorded) I1105 20:27:31.384] Nov 5 20:27:31.384: INFO: Container keyvault-flexvolume ready: true, restart count 0 I1105 20:27:31.384] Nov 5 20:27:31.384: INFO: kube-proxy-kjrpb started at 2019-11-05 20:08:00 +0000 UTC (0+1 container statuses recorded) I1105 20:27:31.385] Nov 5 20:27:31.384: INFO: Container kube-proxy ready: true, restart count 0 I1105 20:27:31.385] Nov 5 20:27:31.384: INFO: tiller-deploy-54c96cb5df-pt7x6 started at 2019-11-05 20:08:00 +0000 UTC (0+1 container statuses recorded) I1105 20:27:31.385] Nov 5 20:27:31.384: INFO: Container tiller ready: true, restart count 0 I1105 20:27:31.385] Nov 5 20:27:31.384: INFO: csi-azuredisk-node-q2dmt started at 2019-11-05 20:16:09 +0000 UTC (0+3 container statuses recorded) I1105 20:27:31.385] Nov 5 20:27:31.384: INFO: Container azuredisk ready: true, restart count 0 I1105 20:27:31.385] Nov 5 20:27:31.384: INFO: Container liveness-probe ready: true, restart count 0 I1105 20:27:31.386] Nov 5 20:27:31.384: INFO: Container node-driver-registrar ready: true, restart count 0 I1105 20:27:31.386] Nov 5 20:27:31.384: INFO: csi-azuredisk-controller-6f6587bfc6-62nrt started at 2019-11-05 20:16:09 +0000 UTC (0+6 container statuses recorded) I1105 20:27:31.386] Nov 5 20:27:31.384: INFO: Container azuredisk ready: true, restart count 0 I1105 20:27:31.386] Nov 5 20:27:31.384: INFO: Container cluster-driver-registrar ready: true, restart count 0 I1105 20:27:31.386] Nov 5 20:27:31.384: INFO: Container csi-attacher ready: true, restart count 0 I1105 20:27:31.386] Nov 5 20:27:31.384: INFO: Container csi-provisioner ready: true, restart count 0 I1105 20:27:31.387] Nov 5 20:27:31.384: INFO: Container csi-snapshotter ready: true, restart count 0 I1105 20:27:31.387] Nov 5 20:27:31.384: INFO: Container liveness-probe ready: true, restart count 0 I1105 20:27:31.387] Nov 5 20:27:31.384: INFO: blobfuse-flexvol-installer-j4s7t started at 2019-11-05 20:07:59 +0000 UTC (0+1 container statuses recorded) I1105 20:27:31.387] Nov 5 20:27:31.384: INFO: Container blobfuse-flexvol-installer ready: true, restart count 0 I1105 20:27:31.639] Nov 5 20:27:31.639: INFO: I1105 20:27:31.640] Latency metrics for node k8s-agentpool-97435445-vmss000001 I1105 20:27:31.640] Nov 5 20:27:31.639: INFO: I1105 20:27:31.640] Logging node info for node k8s-master-97435445-0 I1105 20:27:31.721] Nov 5 20:27:31.718: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-97435445-0,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-97435445-0,UID:df3ae36f-961c-4729-a8f5-34cda54f4ff9,ResourceVersion:4417,Generation:0,CreationTimestamp:2019-11-05 20:08:36 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 2,kubernetes.azure.com/cluster: kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-97435445-0,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.4.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b/providers/Microsoft.Compute/virtualMachines/k8s-master-97435445-0,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 20:08:46 +0000 UTC 2019-11-05 20:08:46 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 20:26:59 +0000 UTC 2019-11-05 20:08:28 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 20:26:59 +0000 UTC 2019-11-05 20:08:28 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 20:26:59 +0000 UTC 2019-11-05 20:08:28 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 20:26:59 +0000 UTC 2019-11-05 20:08:36 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.255.5} {Hostname k8s-master-97435445-0}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:3c381865192341c2868672e8626cef7d,SystemUUID:F3D259D3-7EB8-B844-A1F0-F954DCD0D76B,BootID:4948329e-cf57-48c3-aa83-5297bef70370,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 20:27:31.721] Nov 5 20:27:31.719: INFO: I1105 20:27:31.721] Logging kubelet events for node k8s-master-97435445-0 I1105 20:27:31.804] Nov 5 20:27:31.804: INFO: I1105 20:27:31.805] Logging pods the kubelet thinks is on node k8s-master-97435445-0 I1105 20:27:31.888] Nov 5 20:27:31.888: INFO: kube-proxy-5wnd2 started at 2019-11-05 20:08:38 +0000 UTC (0+1 container statuses recorded) I1105 20:27:31.889] Nov 5 20:27:31.888: INFO: Container kube-proxy ready: true, restart count 0 I1105 20:27:31.889] Nov 5 20:27:31.888: INFO: cloud-controller-manager-k8s-master-97435445-0 started at 2019-11-05 20:08:22 +0000 UTC (0+1 container statuses recorded) I1105 20:27:31.889] Nov 5 20:27:31.888: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 20:27:31.889] Nov 5 20:27:31.888: INFO: kube-addon-manager-k8s-master-97435445-0 started at 2019-11-05 20:08:48 +0000 UTC (0+1 container statuses recorded) I1105 20:27:31.889] Nov 5 20:27:31.888: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 20:27:31.889] Nov 5 20:27:31.888: INFO: kube-apiserver-k8s-master-97435445-0 started at 2019-11-05 20:08:22 +0000 UTC (0+1 container statuses recorded) I1105 20:27:31.889] Nov 5 20:27:31.888: INFO: Container kube-apiserver ready: true, restart count 0 I1105 20:27:31.890] Nov 5 20:27:31.888: INFO: kube-controller-manager-k8s-master-97435445-0 started at 2019-11-05 20:08:22 +0000 UTC (0+1 container statuses recorded) I1105 20:27:31.890] Nov 5 20:27:31.888: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 20:27:31.890] Nov 5 20:27:31.888: INFO: kube-scheduler-k8s-master-97435445-0 started at 2019-11-05 20:08:22 +0000 UTC (0+1 container statuses recorded) I1105 20:27:31.890] Nov 5 20:27:31.888: INFO: Container kube-scheduler ready: true, restart count 0 I1105 20:27:31.890] Nov 5 20:27:31.888: INFO: azure-ip-masq-agent-q6dcl started at 2019-11-05 20:08:38 +0000 UTC (0+1 container statuses recorded) I1105 20:27:31.890] Nov 5 20:27:31.888: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 20:27:32.148] Nov 5 20:27:32.148: INFO: I1105 20:27:32.148] Latency metrics for node k8s-master-97435445-0 I1105 20:27:32.148] Nov 5 20:27:32.148: INFO: I1105 20:27:32.148] Logging node info for node k8s-master-97435445-1 I1105 20:27:32.231] Nov 5 20:27:32.229: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-97435445-1,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-97435445-1,UID:05534a67-d493-4ab5-9b0f-5ff5490e234f,ResourceVersion:4492,Generation:0,CreationTimestamp:2019-11-05 20:07:34 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 0,kubernetes.azure.com/cluster: kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-97435445-1,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.1.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b/providers/Microsoft.Compute/virtualMachines/k8s-master-97435445-1,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 20:08:26 +0000 UTC 2019-11-05 20:08:26 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 20:27:27 +0000 UTC 2019-11-05 20:07:28 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 20:27:27 +0000 UTC 2019-11-05 20:07:28 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 20:27:27 +0000 UTC 2019-11-05 20:07:28 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 20:27:27 +0000 UTC 2019-11-05 20:07:55 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{Hostname k8s-master-97435445-1} {InternalIP 10.240.255.6}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:079b605103b74acabac18fd2ea63368a,SystemUUID:38220EEC-BEFF-4C46-AA05-5A214AAFDC28,BootID:b1b45df0-d925-4bcc-9830-0202df36ebb4,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 20:27:32.231] Nov 5 20:27:32.229: INFO: I1105 20:27:32.232] Logging kubelet events for node k8s-master-97435445-1 I1105 20:27:32.314] Nov 5 20:27:32.314: INFO: I1105 20:27:32.315] Logging pods the kubelet thinks is on node k8s-master-97435445-1 I1105 20:27:32.398] Nov 5 20:27:32.398: INFO: kube-addon-manager-k8s-master-97435445-1 started at 2019-11-05 20:07:22 +0000 UTC (0+1 container statuses recorded) I1105 20:27:32.398] Nov 5 20:27:32.398: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 20:27:32.398] Nov 5 20:27:32.398: INFO: kube-apiserver-k8s-master-97435445-1 started at 2019-11-05 20:07:22 +0000 UTC (0+1 container statuses recorded) I1105 20:27:32.398] Nov 5 20:27:32.398: INFO: Container kube-apiserver ready: true, restart count 0 I1105 20:27:32.399] Nov 5 20:27:32.398: INFO: kube-controller-manager-k8s-master-97435445-1 started at 2019-11-05 20:07:22 +0000 UTC (0+1 container statuses recorded) I1105 20:27:32.399] Nov 5 20:27:32.398: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 20:27:32.399] Nov 5 20:27:32.398: INFO: kube-scheduler-k8s-master-97435445-1 started at 2019-11-05 20:07:22 +0000 UTC (0+1 container statuses recorded) I1105 20:27:32.399] Nov 5 20:27:32.398: INFO: Container kube-scheduler ready: true, restart count 0 I1105 20:27:32.399] Nov 5 20:27:32.398: INFO: azure-ip-masq-agent-n29bm started at 2019-11-05 20:07:59 +0000 UTC (0+1 container statuses recorded) I1105 20:27:32.399] Nov 5 20:27:32.398: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 20:27:32.400] Nov 5 20:27:32.398: INFO: kube-proxy-h4s2c started at 2019-11-05 20:08:00 +0000 UTC (0+1 container statuses recorded) I1105 20:27:32.400] Nov 5 20:27:32.398: INFO: Container kube-proxy ready: true, restart count 0 I1105 20:27:32.400] Nov 5 20:27:32.398: INFO: cloud-controller-manager-k8s-master-97435445-1 started at 2019-11-05 20:07:22 +0000 UTC (0+1 container statuses recorded) I1105 20:27:32.400] Nov 5 20:27:32.398: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 20:27:32.654] Nov 5 20:27:32.654: INFO: I1105 20:27:32.654] Latency metrics for node k8s-master-97435445-1 I1105 20:27:32.654] Nov 5 20:27:32.654: INFO: I1105 20:27:32.654] Logging node info for node k8s-master-97435445-2 I1105 20:27:32.743] Nov 5 20:27:32.739: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-97435445-2,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-97435445-2,UID:fa55c895-1a93-4743-885e-03fce84b5332,ResourceVersion:4366,Generation:0,CreationTimestamp:2019-11-05 20:09:19 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 0,kubernetes.azure.com/cluster: kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-97435445-2,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.5.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b/providers/Microsoft.Compute/virtualMachines/k8s-master-97435445-2,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 20:09:19 +0000 UTC 2019-11-05 20:09:19 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 20:26:42 +0000 UTC 2019-11-05 20:09:12 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 20:26:42 +0000 UTC 2019-11-05 20:09:12 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 20:26:42 +0000 UTC 2019-11-05 20:09:12 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 20:26:42 +0000 UTC 2019-11-05 20:09:19 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.255.7} {Hostname k8s-master-97435445-2}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:d19f6709138f496da43c095cec7b2360,SystemUUID:10116777-A8D7-CB4A-A8B7-7EBCF2F23B62,BootID:50cd14a9-5497-4972-bf60-cf59a0cbcb5f,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 20:27:32.743] Nov 5 20:27:32.739: INFO: I1105 20:27:32.743] Logging kubelet events for node k8s-master-97435445-2 I1105 20:27:32.823] Nov 5 20:27:32.823: INFO: I1105 20:27:32.823] Logging pods the kubelet thinks is on node k8s-master-97435445-2 I1105 20:27:32.907] Nov 5 20:27:32.907: INFO: kube-proxy-pz426 started at 2019-11-05 20:09:19 +0000 UTC (0+1 container statuses recorded) I1105 20:27:32.907] Nov 5 20:27:32.907: INFO: Container kube-proxy ready: true, restart count 0 I1105 20:27:32.907] Nov 5 20:27:32.907: INFO: kube-apiserver-k8s-master-97435445-2 started at 2019-11-05 20:09:05 +0000 UTC (0+1 container statuses recorded) I1105 20:27:32.907] Nov 5 20:27:32.907: INFO: Container kube-apiserver ready: true, restart count 0 I1105 20:27:32.908] Nov 5 20:27:32.907: INFO: kube-controller-manager-k8s-master-97435445-2 started at 2019-11-05 20:09:05 +0000 UTC (0+1 container statuses recorded) I1105 20:27:32.908] Nov 5 20:27:32.907: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 20:27:32.908] Nov 5 20:27:32.907: INFO: kube-scheduler-k8s-master-97435445-2 started at 2019-11-05 20:09:05 +0000 UTC (0+1 container statuses recorded) I1105 20:27:32.908] Nov 5 20:27:32.907: INFO: Container kube-scheduler ready: true, restart count 0 I1105 20:27:32.908] Nov 5 20:27:32.907: INFO: cloud-controller-manager-k8s-master-97435445-2 started at 2019-11-05 20:09:05 +0000 UTC (0+1 container statuses recorded) I1105 20:27:32.908] Nov 5 20:27:32.907: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 20:27:32.909] Nov 5 20:27:32.907: INFO: kube-addon-manager-k8s-master-97435445-2 started at 2019-11-05 20:09:05 +0000 UTC (0+1 container statuses recorded) I1105 20:27:32.909] Nov 5 20:27:32.907: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 20:27:32.909] Nov 5 20:27:32.907: INFO: azure-ip-masq-agent-sq7p5 started at 2019-11-05 20:09:19 +0000 UTC (0+1 container statuses recorded) I1105 20:27:32.909] Nov 5 20:27:32.907: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 20:27:33.160] Nov 5 20:27:33.159: INFO: I1105 20:27:33.160] Latency metrics for node k8s-master-97435445-2 I1105 20:27:33.160] Nov 5 20:27:33.159: INFO: I1105 20:27:33.160] Logging node info for node k8s-master-97435445-3 I1105 20:27:33.241] Nov 5 20:27:33.238: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-97435445-3,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-97435445-3,UID:8ffd1d0d-4487-4680-a2f6-e724387be637,ResourceVersion:4369,Generation:0,CreationTimestamp:2019-11-05 20:09:21 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 1,kubernetes.azure.com/cluster: kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-97435445-3,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.6.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b/providers/Microsoft.Compute/virtualMachines/k8s-master-97435445-3,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 20:09:39 +0000 UTC 2019-11-05 20:09:39 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 20:26:43 +0000 UTC 2019-11-05 20:09:18 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 20:26:43 +0000 UTC 2019-11-05 20:09:18 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 20:26:43 +0000 UTC 2019-11-05 20:09:18 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 20:26:43 +0000 UTC 2019-11-05 20:09:21 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.255.8} {Hostname k8s-master-97435445-3}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:266064173bfb4297898c6c25769e8b20,SystemUUID:E6E44241-5015-7740-9A81-A668A8427206,BootID:579d658c-b0ad-47f1-a993-1ccea2b55789,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 20:27:33.241] Nov 5 20:27:33.239: INFO: I1105 20:27:33.241] Logging kubelet events for node k8s-master-97435445-3 I1105 20:27:33.322] Nov 5 20:27:33.322: INFO: I1105 20:27:33.323] Logging pods the kubelet thinks is on node k8s-master-97435445-3 I1105 20:27:33.407] Nov 5 20:27:33.407: INFO: kube-proxy-k45g8 started at 2019-11-05 20:09:21 +0000 UTC (0+1 container statuses recorded) I1105 20:27:33.407] Nov 5 20:27:33.407: INFO: Container kube-proxy ready: true, restart count 0 I1105 20:27:33.408] Nov 5 20:27:33.407: INFO: cloud-controller-manager-k8s-master-97435445-3 started at 2019-11-05 20:09:04 +0000 UTC (0+1 container statuses recorded) I1105 20:27:33.408] Nov 5 20:27:33.407: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 20:27:33.408] Nov 5 20:27:33.407: INFO: kube-addon-manager-k8s-master-97435445-3 started at 2019-11-05 20:09:05 +0000 UTC (0+1 container statuses recorded) I1105 20:27:33.408] Nov 5 20:27:33.407: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 20:27:33.408] Nov 5 20:27:33.407: INFO: kube-apiserver-k8s-master-97435445-3 started at 2019-11-05 20:09:05 +0000 UTC (0+1 container statuses recorded) I1105 20:27:33.408] Nov 5 20:27:33.407: INFO: Container kube-apiserver ready: true, restart count 0 I1105 20:27:33.409] Nov 5 20:27:33.407: INFO: kube-controller-manager-k8s-master-97435445-3 started at 2019-11-05 20:09:05 +0000 UTC (0+1 container statuses recorded) I1105 20:27:33.409] Nov 5 20:27:33.407: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 20:27:33.409] Nov 5 20:27:33.407: INFO: kube-scheduler-k8s-master-97435445-3 started at 2019-11-05 20:09:05 +0000 UTC (0+1 container statuses recorded) I1105 20:27:33.409] Nov 5 20:27:33.407: INFO: Container kube-scheduler ready: true, restart count 0 I1105 20:27:33.409] Nov 5 20:27:33.407: INFO: azure-ip-masq-agent-747cl started at 2019-11-05 20:09:21 +0000 UTC (0+1 container statuses recorded) I1105 20:27:33.409] Nov 5 20:27:33.407: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 20:27:33.665] Nov 5 20:27:33.665: INFO: I1105 20:27:33.665] Latency metrics for node k8s-master-97435445-3 I1105 20:27:33.666] Nov 5 20:27:33.665: INFO: I1105 20:27:33.666] Logging node info for node k8s-master-97435445-4 I1105 20:27:33.749] Nov 5 20:27:33.746: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-97435445-4,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-97435445-4,UID:6d547b41-3f92-4cc4-a2d7-dda2071776a2,ResourceVersion:4416,Generation:0,CreationTimestamp:2019-11-05 20:08:36 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 1,kubernetes.azure.com/cluster: kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-97435445-4,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.3.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b/providers/Microsoft.Compute/virtualMachines/k8s-master-97435445-4,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 20:08:59 +0000 UTC 2019-11-05 20:08:59 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 20:26:58 +0000 UTC 2019-11-05 20:08:29 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 20:26:58 +0000 UTC 2019-11-05 20:08:29 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 20:26:58 +0000 UTC 2019-11-05 20:08:29 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 20:26:58 +0000 UTC 2019-11-05 20:08:36 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.255.9} {Hostname k8s-master-97435445-4}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:56b393c61eb34ba49ebda5279eeedb30,SystemUUID:FD239C23-618C-0049-8734-9AF3FDC9E3C1,BootID:f342ea44-e719-4513-807d-5def7ac606e2,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 20:27:33.749] Nov 5 20:27:33.747: INFO: I1105 20:27:33.749] Logging kubelet events for node k8s-master-97435445-4 I1105 20:27:33.839] Nov 5 20:27:33.839: INFO: I1105 20:27:33.839] Logging pods the kubelet thinks is on node k8s-master-97435445-4 I1105 20:27:33.923] Nov 5 20:27:33.923: INFO: kube-apiserver-k8s-master-97435445-4 started at 2019-11-05 20:08:23 +0000 UTC (0+1 container statuses recorded) I1105 20:27:33.923] Nov 5 20:27:33.923: INFO: Container kube-apiserver ready: true, restart count 0 I1105 20:27:33.923] Nov 5 20:27:33.923: INFO: kube-controller-manager-k8s-master-97435445-4 started at 2019-11-05 20:08:23 +0000 UTC (0+1 container statuses recorded) I1105 20:27:33.923] Nov 5 20:27:33.923: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 20:27:33.924] Nov 5 20:27:33.923: INFO: kube-scheduler-k8s-master-97435445-4 started at 2019-11-05 20:08:23 +0000 UTC (0+1 container statuses recorded) I1105 20:27:33.924] Nov 5 20:27:33.923: INFO: Container kube-scheduler ready: true, restart count 0 I1105 20:27:33.924] Nov 5 20:27:33.923: INFO: kube-proxy-vfbfr started at 2019-11-05 20:08:36 +0000 UTC (0+1 container statuses recorded) I1105 20:27:33.924] Nov 5 20:27:33.923: INFO: Container kube-proxy ready: true, restart count 0 I1105 20:27:33.924] Nov 5 20:27:33.923: INFO: azure-ip-masq-agent-kmkll started at 2019-11-05 20:08:36 +0000 UTC (0+1 container statuses recorded) I1105 20:27:33.924] Nov 5 20:27:33.923: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 20:27:33.925] Nov 5 20:27:33.923: INFO: cloud-controller-manager-k8s-master-97435445-4 started at 2019-11-05 20:08:46 +0000 UTC (0+1 container statuses recorded) I1105 20:27:33.925] Nov 5 20:27:33.923: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 20:27:33.925] Nov 5 20:27:33.923: INFO: kube-addon-manager-k8s-master-97435445-4 started at 2019-11-05 20:08:23 +0000 UTC (0+1 container statuses recorded) I1105 20:27:33.925] Nov 5 20:27:33.923: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 20:27:34.180] Nov 5 20:27:34.180: INFO: I1105 20:27:34.180] Latency metrics for node k8s-master-97435445-4 I1105 20:27:34.180] Nov 5 20:27:34.180: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready I1105 20:27:34.287] STEP: Destroying namespace "azuredisk-1082" for this suite. I1105 20:27:48.538] Nov 5 20:27:48.538: INFO: Waiting up to 30s for server preferred namespaced resources to be successfully discovered I1105 20:27:51.599] Nov 5 20:27:51.598: INFO: namespace azuredisk-1082 deletion completed in 17.311858989s I1105 20:27:51.599] I1105 20:27:51.599] • Failure [323.841 seconds] I1105 20:27:51.599] [azuredisk-csi-e2e] Dynamic Provisioning I1105 20:27:51.599] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:33 I1105 20:27:51.599] [single-az] I1105 20:27:51.600] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:36 I1105 20:27:51.600] should create a raw block volume on demand [It] I1105 20:27:51.600] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:101 I1105 20:27:51.600] I1105 20:27:51.600] Unexpected error: I1105 20:27:51.600] <*errors.errorString | 0xc000a10b70>: { I1105 20:27:51.600] s: "PersistentVolumeClaims [pvc-tvv9z] not all in phase Bound within 5m0s", I1105 20:27:51.600] } I1105 20:27:51.601] PersistentVolumeClaims [pvc-tvv9z] not all in phase Bound within 5m0s I1105 20:27:51.601] occurred I1105 20:27:51.601] I1105 20:27:51.601] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/testsuites/testsuites.go:278 I1105 20:27:51.601] ------------------------------ I1105 20:27:51.601] [azuredisk-csi-e2e] Dynamic Provisioning [multi-az] I1105 20:27:51.601] [env] should retain PV with reclaimPolicy "Retain" I1105 20:27:51.601] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:240 I1105 20:27:51.602] [BeforeEach] [multi-az] I1105 20:27:51.602] /go/src/sigs.k8s.io/azuredisk-csi-driver/vendor/k8s.io/kubernetes/test/e2e/framework/framework.go:150 I1105 20:27:51.602] STEP: Creating a kubernetes client I1105 20:27:51.602] Nov 5 20:27:51.599: INFO: >>> kubeConfig: /workspace/aks971561099/kubeconfig/kubeconfig.eastus2.json I1105 20:27:51.602] STEP: Building a namespace api object, basename azuredisk I1105 20:27:51.838] STEP: Binding the e2e-test-privileged-psp PodSecurityPolicy to the default service account in azuredisk-8704 I1105 20:27:52.105] STEP: Waiting for a default service account to be provisioned in namespace I1105 20:27:52.182] [BeforeEach] [multi-az] I1105 20:27:52.182] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:58 I1105 20:27:52.182] [It] [env] should retain PV with reclaimPolicy "Retain" I1105 20:27:52.183] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:240 I1105 20:27:52.183] STEP: setting up the StorageClass I1105 20:27:52.183] STEP: creating a StorageClass I1105 20:27:52.270] STEP: setting up the PVC and PV I1105 20:27:52.270] STEP: creating a PVC I1105 20:27:52.360] STEP: waiting for PVC to be in phase "Bound" I1105 20:27:52.360] Nov 5 20:27:52.359: INFO: Waiting up to 5m0s for PersistentVolumeClaims [pvc-vrjw8] to have phase Bound I1105 20:27:52.438] Nov 5 20:27:52.438: INFO: PersistentVolumeClaim pvc-vrjw8 found but phase is Pending instead of Bound. I1105 20:27:54.518] Nov 5 20:27:54.517: INFO: PersistentVolumeClaim pvc-vrjw8 found but phase is Pending instead of Bound. I1105 20:27:56.597] Nov 5 20:27:56.596: INFO: PersistentVolumeClaim pvc-vrjw8 found but phase is Pending instead of Bound. I1105 20:27:58.676] Nov 5 20:27:58.675: INFO: PersistentVolumeClaim pvc-vrjw8 found but phase is Pending instead of Bound. I1105 20:28:00.755] Nov 5 20:28:00.755: INFO: PersistentVolumeClaim pvc-vrjw8 found but phase is Pending instead of Bound. I1105 20:28:02.835] Nov 5 20:28:02.834: INFO: PersistentVolumeClaim pvc-vrjw8 found but phase is Pending instead of Bound. I1105 20:28:04.914] Nov 5 20:28:04.914: INFO: PersistentVolumeClaim pvc-vrjw8 found but phase is Pending instead of Bound. I1105 20:28:06.993] Nov 5 20:28:06.993: INFO: PersistentVolumeClaim pvc-vrjw8 found but phase is Pending instead of Bound. I1105 20:28:09.072] Nov 5 20:28:09.072: INFO: PersistentVolumeClaim pvc-vrjw8 found but phase is Pending instead of Bound. I1105 20:28:11.152] Nov 5 20:28:11.152: INFO: PersistentVolumeClaim pvc-vrjw8 found but phase is Pending instead of Bound. I1105 20:28:13.232] Nov 5 20:28:13.232: INFO: PersistentVolumeClaim pvc-vrjw8 found but phase is Pending instead of Bound. I1105 20:28:15.312] Nov 5 20:28:15.312: INFO: PersistentVolumeClaim pvc-vrjw8 found but phase is Pending instead of Bound. I1105 20:28:17.391] Nov 5 20:28:17.391: INFO: PersistentVolumeClaim pvc-vrjw8 found but phase is Pending instead of Bound. I1105 20:28:19.471] Nov 5 20:28:19.470: INFO: PersistentVolumeClaim pvc-vrjw8 found but phase is Pending instead of Bound. I1105 20:28:21.550] Nov 5 20:28:21.550: INFO: PersistentVolumeClaim pvc-vrjw8 found but phase is Pending instead of Bound. I1105 20:28:23.629] Nov 5 20:28:23.629: INFO: PersistentVolumeClaim pvc-vrjw8 found but phase is Pending instead of Bound. I1105 20:28:25.708] Nov 5 20:28:25.708: INFO: PersistentVolumeClaim pvc-vrjw8 found but phase is Pending instead of Bound. I1105 20:28:27.787] Nov 5 20:28:27.787: INFO: PersistentVolumeClaim pvc-vrjw8 found but phase is Pending instead of Bound. I1105 20:28:29.867] Nov 5 20:28:29.867: INFO: PersistentVolumeClaim pvc-vrjw8 found but phase is Pending instead of Bound. I1105 20:28:31.946] Nov 5 20:28:31.945: INFO: PersistentVolumeClaim pvc-vrjw8 found but phase is Pending instead of Bound. I1105 20:28:34.025] Nov 5 20:28:34.025: INFO: PersistentVolumeClaim pvc-vrjw8 found but phase is Pending instead of Bound. I1105 20:28:36.104] Nov 5 20:28:36.104: INFO: PersistentVolumeClaim pvc-vrjw8 found but phase is Pending instead of Bound. I1105 20:28:38.210] Nov 5 20:28:38.210: INFO: PersistentVolumeClaim pvc-vrjw8 found but phase is Pending instead of Bound. I1105 20:28:40.290] Nov 5 20:28:40.290: INFO: PersistentVolumeClaim pvc-vrjw8 found but phase is Pending instead of Bound. I1105 20:28:42.370] Nov 5 20:28:42.369: INFO: PersistentVolumeClaim pvc-vrjw8 found but phase is Pending instead of Bound. I1105 20:28:44.449] Nov 5 20:28:44.448: INFO: PersistentVolumeClaim pvc-vrjw8 found but phase is Pending instead of Bound. I1105 20:28:46.529] Nov 5 20:28:46.528: INFO: PersistentVolumeClaim pvc-vrjw8 found but phase is Pending instead of Bound. I1105 20:28:48.608] Nov 5 20:28:48.607: INFO: PersistentVolumeClaim pvc-vrjw8 found but phase is Pending instead of Bound. I1105 20:28:50.686] Nov 5 20:28:50.686: INFO: PersistentVolumeClaim pvc-vrjw8 found but phase is Pending instead of Bound. I1105 20:28:52.766] Nov 5 20:28:52.766: INFO: PersistentVolumeClaim pvc-vrjw8 found but phase is Pending instead of Bound. I1105 20:28:54.846] Nov 5 20:28:54.846: INFO: PersistentVolumeClaim pvc-vrjw8 found but phase is Pending instead of Bound. I1105 20:28:56.926] Nov 5 20:28:56.925: INFO: PersistentVolumeClaim pvc-vrjw8 found but phase is Pending instead of Bound. I1105 20:28:59.005] Nov 5 20:28:59.004: INFO: PersistentVolumeClaim pvc-vrjw8 found but phase is Pending instead of Bound. I1105 20:29:01.084] Nov 5 20:29:01.084: INFO: PersistentVolumeClaim pvc-vrjw8 found but phase is Pending instead of Bound. I1105 20:29:03.163] Nov 5 20:29:03.162: INFO: PersistentVolumeClaim pvc-vrjw8 found but phase is Pending instead of Bound. I1105 20:29:05.242] Nov 5 20:29:05.242: INFO: PersistentVolumeClaim pvc-vrjw8 found but phase is Pending instead of Bound. I1105 20:29:07.321] Nov 5 20:29:07.321: INFO: PersistentVolumeClaim pvc-vrjw8 found but phase is Pending instead of Bound. I1105 20:29:09.401] Nov 5 20:29:09.401: INFO: PersistentVolumeClaim pvc-vrjw8 found but phase is Pending instead of Bound. I1105 20:29:11.482] Nov 5 20:29:11.482: INFO: PersistentVolumeClaim pvc-vrjw8 found but phase is Pending instead of Bound. I1105 20:29:13.568] Nov 5 20:29:13.568: INFO: PersistentVolumeClaim pvc-vrjw8 found but phase is Pending instead of Bound. I1105 20:29:15.648] Nov 5 20:29:15.648: INFO: PersistentVolumeClaim pvc-vrjw8 found but phase is Pending instead of Bound. I1105 20:29:17.727] Nov 5 20:29:17.727: INFO: PersistentVolumeClaim pvc-vrjw8 found but phase is Pending instead of Bound. I1105 20:29:19.807] Nov 5 20:29:19.807: INFO: PersistentVolumeClaim pvc-vrjw8 found but phase is Pending instead of Bound. I1105 20:29:21.887] Nov 5 20:29:21.886: INFO: PersistentVolumeClaim pvc-vrjw8 found but phase is Pending instead of Bound. I1105 20:29:23.966] Nov 5 20:29:23.966: INFO: PersistentVolumeClaim pvc-vrjw8 found but phase is Pending instead of Bound. I1105 20:29:26.045] Nov 5 20:29:26.045: INFO: PersistentVolumeClaim pvc-vrjw8 found but phase is Pending instead of Bound. I1105 20:29:28.124] Nov 5 20:29:28.124: INFO: PersistentVolumeClaim pvc-vrjw8 found but phase is Pending instead of Bound. I1105 20:29:30.211] Nov 5 20:29:30.211: INFO: PersistentVolumeClaim pvc-vrjw8 found but phase is Pending instead of Bound. I1105 20:29:32.291] Nov 5 20:29:32.290: INFO: PersistentVolumeClaim pvc-vrjw8 found but phase is Pending instead of Bound. I1105 20:29:34.371] Nov 5 20:29:34.370: INFO: PersistentVolumeClaim pvc-vrjw8 found but phase is Pending instead of Bound. I1105 20:29:36.450] Nov 5 20:29:36.450: INFO: PersistentVolumeClaim pvc-vrjw8 found but phase is Pending instead of Bound. I1105 20:29:38.530] Nov 5 20:29:38.529: INFO: PersistentVolumeClaim pvc-vrjw8 found but phase is Pending instead of Bound. I1105 20:29:40.609] Nov 5 20:29:40.609: INFO: PersistentVolumeClaim pvc-vrjw8 found but phase is Pending instead of Bound. I1105 20:29:42.687] Nov 5 20:29:42.687: INFO: PersistentVolumeClaim pvc-vrjw8 found but phase is Pending instead of Bound. I1105 20:29:44.767] Nov 5 20:29:44.767: INFO: PersistentVolumeClaim pvc-vrjw8 found but phase is Pending instead of Bound. I1105 20:29:46.846] Nov 5 20:29:46.846: INFO: PersistentVolumeClaim pvc-vrjw8 found but phase is Pending instead of Bound. I1105 20:29:48.926] Nov 5 20:29:48.926: INFO: PersistentVolumeClaim pvc-vrjw8 found but phase is Pending instead of Bound. I1105 20:29:51.005] Nov 5 20:29:51.005: INFO: PersistentVolumeClaim pvc-vrjw8 found but phase is Pending instead of Bound. I1105 20:29:53.086] Nov 5 20:29:53.086: INFO: PersistentVolumeClaim pvc-vrjw8 found but phase is Pending instead of Bound. I1105 20:29:55.165] Nov 5 20:29:55.164: INFO: PersistentVolumeClaim pvc-vrjw8 found but phase is Pending instead of Bound. I1105 20:29:57.243] Nov 5 20:29:57.243: INFO: PersistentVolumeClaim pvc-vrjw8 found but phase is Pending instead of Bound. I1105 20:29:59.323] Nov 5 20:29:59.322: INFO: PersistentVolumeClaim pvc-vrjw8 found but phase is Pending instead of Bound. I1105 20:30:01.402] Nov 5 20:30:01.402: INFO: PersistentVolumeClaim pvc-vrjw8 found but phase is Pending instead of Bound. I1105 20:30:03.481] Nov 5 20:30:03.481: INFO: PersistentVolumeClaim pvc-vrjw8 found but phase is Pending instead of Bound. I1105 20:30:05.561] Nov 5 20:30:05.561: INFO: PersistentVolumeClaim pvc-vrjw8 found but phase is Pending instead of Bound. I1105 20:30:07.641] Nov 5 20:30:07.641: INFO: PersistentVolumeClaim pvc-vrjw8 found but phase is Pending instead of Bound. I1105 20:30:09.720] Nov 5 20:30:09.720: INFO: PersistentVolumeClaim pvc-vrjw8 found but phase is Pending instead of Bound. I1105 20:30:11.800] Nov 5 20:30:11.799: INFO: PersistentVolumeClaim pvc-vrjw8 found but phase is Pending instead of Bound. I1105 20:30:13.879] Nov 5 20:30:13.879: INFO: PersistentVolumeClaim pvc-vrjw8 found but phase is Pending instead of Bound. I1105 20:30:15.958] Nov 5 20:30:15.958: INFO: PersistentVolumeClaim pvc-vrjw8 found but phase is Pending instead of Bound. I1105 20:30:18.038] Nov 5 20:30:18.037: INFO: PersistentVolumeClaim pvc-vrjw8 found but phase is Pending instead of Bound. I1105 20:30:20.117] Nov 5 20:30:20.117: INFO: PersistentVolumeClaim pvc-vrjw8 found but phase is Pending instead of Bound. I1105 20:30:22.211] Nov 5 20:30:22.210: INFO: PersistentVolumeClaim pvc-vrjw8 found but phase is Pending instead of Bound. I1105 20:30:24.290] Nov 5 20:30:24.290: INFO: PersistentVolumeClaim pvc-vrjw8 found but phase is Pending instead of Bound. I1105 20:30:26.370] Nov 5 20:30:26.370: INFO: PersistentVolumeClaim pvc-vrjw8 found but phase is Pending instead of Bound. I1105 20:30:28.450] Nov 5 20:30:28.449: INFO: PersistentVolumeClaim pvc-vrjw8 found but phase is Pending instead of Bound. I1105 20:30:30.530] Nov 5 20:30:30.530: INFO: PersistentVolumeClaim pvc-vrjw8 found but phase is Pending instead of Bound. I1105 20:30:32.617] Nov 5 20:30:32.617: INFO: PersistentVolumeClaim pvc-vrjw8 found but phase is Pending instead of Bound. I1105 20:30:34.696] Nov 5 20:30:34.696: INFO: PersistentVolumeClaim pvc-vrjw8 found but phase is Pending instead of Bound. I1105 20:30:36.774] Nov 5 20:30:36.774: INFO: PersistentVolumeClaim pvc-vrjw8 found but phase is Pending instead of Bound. I1105 20:30:38.858] Nov 5 20:30:38.857: INFO: PersistentVolumeClaim pvc-vrjw8 found but phase is Pending instead of Bound. I1105 20:30:40.937] Nov 5 20:30:40.936: INFO: PersistentVolumeClaim pvc-vrjw8 found but phase is Pending instead of Bound. I1105 20:30:43.016] Nov 5 20:30:43.015: INFO: PersistentVolumeClaim pvc-vrjw8 found but phase is Pending instead of Bound. I1105 20:30:45.095] Nov 5 20:30:45.095: INFO: PersistentVolumeClaim pvc-vrjw8 found but phase is Pending instead of Bound. I1105 20:30:47.174] Nov 5 20:30:47.173: INFO: PersistentVolumeClaim pvc-vrjw8 found but phase is Pending instead of Bound. I1105 20:30:49.252] Nov 5 20:30:49.252: INFO: PersistentVolumeClaim pvc-vrjw8 found but phase is Pending instead of Bound. I1105 20:30:51.331] Nov 5 20:30:51.331: INFO: PersistentVolumeClaim pvc-vrjw8 found but phase is Pending instead of Bound. I1105 20:30:53.413] Nov 5 20:30:53.413: INFO: PersistentVolumeClaim pvc-vrjw8 found but phase is Pending instead of Bound. I1105 20:30:55.494] Nov 5 20:30:55.494: INFO: PersistentVolumeClaim pvc-vrjw8 found but phase is Pending instead of Bound. I1105 20:30:57.574] Nov 5 20:30:57.573: INFO: PersistentVolumeClaim pvc-vrjw8 found but phase is Pending instead of Bound. I1105 20:30:59.653] Nov 5 20:30:59.653: INFO: PersistentVolumeClaim pvc-vrjw8 found but phase is Pending instead of Bound. I1105 20:31:01.732] Nov 5 20:31:01.732: INFO: PersistentVolumeClaim pvc-vrjw8 found but phase is Pending instead of Bound. I1105 20:31:03.811] Nov 5 20:31:03.811: INFO: PersistentVolumeClaim pvc-vrjw8 found but phase is Pending instead of Bound. I1105 20:31:05.890] Nov 5 20:31:05.890: INFO: PersistentVolumeClaim pvc-vrjw8 found but phase is Pending instead of Bound. I1105 20:31:07.971] Nov 5 20:31:07.970: INFO: PersistentVolumeClaim pvc-vrjw8 found but phase is Pending instead of Bound. I1105 20:31:10.050] Nov 5 20:31:10.050: INFO: PersistentVolumeClaim pvc-vrjw8 found but phase is Pending instead of Bound. I1105 20:31:12.129] Nov 5 20:31:12.129: INFO: PersistentVolumeClaim pvc-vrjw8 found but phase is Pending instead of Bound. I1105 20:31:14.217] Nov 5 20:31:14.216: INFO: PersistentVolumeClaim pvc-vrjw8 found but phase is Pending instead of Bound. I1105 20:31:16.296] Nov 5 20:31:16.296: INFO: PersistentVolumeClaim pvc-vrjw8 found but phase is Pending instead of Bound. I1105 20:31:18.376] Nov 5 20:31:18.375: INFO: PersistentVolumeClaim pvc-vrjw8 found but phase is Pending instead of Bound. I1105 20:31:20.461] Nov 5 20:31:20.460: INFO: PersistentVolumeClaim pvc-vrjw8 found but phase is Pending instead of Bound. I1105 20:31:22.540] Nov 5 20:31:22.540: INFO: PersistentVolumeClaim pvc-vrjw8 found but phase is Pending instead of Bound. I1105 20:31:24.619] Nov 5 20:31:24.619: INFO: PersistentVolumeClaim pvc-vrjw8 found but phase is Pending instead of Bound. I1105 20:31:26.698] Nov 5 20:31:26.698: INFO: PersistentVolumeClaim pvc-vrjw8 found but phase is Pending instead of Bound. I1105 20:31:28.778] Nov 5 20:31:28.777: INFO: PersistentVolumeClaim pvc-vrjw8 found but phase is Pending instead of Bound. I1105 20:31:30.865] Nov 5 20:31:30.865: INFO: PersistentVolumeClaim pvc-vrjw8 found but phase is Pending instead of Bound. I1105 20:31:32.946] Nov 5 20:31:32.945: INFO: PersistentVolumeClaim pvc-vrjw8 found but phase is Pending instead of Bound. I1105 20:31:35.025] Nov 5 20:31:35.025: INFO: PersistentVolumeClaim pvc-vrjw8 found but phase is Pending instead of Bound. I1105 20:31:37.105] Nov 5 20:31:37.105: INFO: PersistentVolumeClaim pvc-vrjw8 found but phase is Pending instead of Bound. I1105 20:31:39.185] Nov 5 20:31:39.184: INFO: PersistentVolumeClaim pvc-vrjw8 found but phase is Pending instead of Bound. I1105 20:31:41.264] Nov 5 20:31:41.264: INFO: PersistentVolumeClaim pvc-vrjw8 found but phase is Pending instead of Bound. I1105 20:31:43.349] Nov 5 20:31:43.349: INFO: PersistentVolumeClaim pvc-vrjw8 found but phase is Pending instead of Bound. I1105 20:31:45.428] Nov 5 20:31:45.428: INFO: PersistentVolumeClaim pvc-vrjw8 found but phase is Pending instead of Bound. I1105 20:31:47.507] Nov 5 20:31:47.507: INFO: PersistentVolumeClaim pvc-vrjw8 found but phase is Pending instead of Bound. I1105 20:31:49.586] Nov 5 20:31:49.586: INFO: PersistentVolumeClaim pvc-vrjw8 found but phase is Pending instead of Bound. I1105 20:31:51.668] Nov 5 20:31:51.668: INFO: PersistentVolumeClaim pvc-vrjw8 found but phase is Pending instead of Bound. I1105 20:31:53.747] Nov 5 20:31:53.747: INFO: PersistentVolumeClaim pvc-vrjw8 found but phase is Pending instead of Bound. I1105 20:31:55.827] Nov 5 20:31:55.827: INFO: PersistentVolumeClaim pvc-vrjw8 found but phase is Pending instead of Bound. I1105 20:31:57.910] Nov 5 20:31:57.910: INFO: PersistentVolumeClaim pvc-vrjw8 found but phase is Pending instead of Bound. I1105 20:31:59.989] Nov 5 20:31:59.989: INFO: PersistentVolumeClaim pvc-vrjw8 found but phase is Pending instead of Bound. I1105 20:32:02.068] Nov 5 20:32:02.068: INFO: PersistentVolumeClaim pvc-vrjw8 found but phase is Pending instead of Bound. I1105 20:32:04.147] Nov 5 20:32:04.147: INFO: PersistentVolumeClaim pvc-vrjw8 found but phase is Pending instead of Bound. I1105 20:32:06.227] Nov 5 20:32:06.227: INFO: PersistentVolumeClaim pvc-vrjw8 found but phase is Pending instead of Bound. I1105 20:32:08.307] Nov 5 20:32:08.306: INFO: PersistentVolumeClaim pvc-vrjw8 found but phase is Pending instead of Bound. I1105 20:32:10.385] Nov 5 20:32:10.385: INFO: PersistentVolumeClaim pvc-vrjw8 found but phase is Pending instead of Bound. I1105 20:32:12.466] Nov 5 20:32:12.466: INFO: PersistentVolumeClaim pvc-vrjw8 found but phase is Pending instead of Bound. I1105 20:32:14.545] Nov 5 20:32:14.545: INFO: PersistentVolumeClaim pvc-vrjw8 found but phase is Pending instead of Bound. I1105 20:32:16.625] Nov 5 20:32:16.625: INFO: PersistentVolumeClaim pvc-vrjw8 found but phase is Pending instead of Bound. I1105 20:32:18.704] Nov 5 20:32:18.704: INFO: PersistentVolumeClaim pvc-vrjw8 found but phase is Pending instead of Bound. I1105 20:32:20.786] Nov 5 20:32:20.786: INFO: PersistentVolumeClaim pvc-vrjw8 found but phase is Pending instead of Bound. I1105 20:32:22.865] Nov 5 20:32:22.865: INFO: PersistentVolumeClaim pvc-vrjw8 found but phase is Pending instead of Bound. I1105 20:32:24.946] Nov 5 20:32:24.946: INFO: PersistentVolumeClaim pvc-vrjw8 found but phase is Pending instead of Bound. I1105 20:32:27.025] Nov 5 20:32:27.025: INFO: PersistentVolumeClaim pvc-vrjw8 found but phase is Pending instead of Bound. I1105 20:32:29.105] Nov 5 20:32:29.104: INFO: PersistentVolumeClaim pvc-vrjw8 found but phase is Pending instead of Bound. I1105 20:32:31.184] Nov 5 20:32:31.184: INFO: PersistentVolumeClaim pvc-vrjw8 found but phase is Pending instead of Bound. I1105 20:32:33.263] Nov 5 20:32:33.263: INFO: PersistentVolumeClaim pvc-vrjw8 found but phase is Pending instead of Bound. I1105 20:32:35.349] Nov 5 20:32:35.348: INFO: PersistentVolumeClaim pvc-vrjw8 found but phase is Pending instead of Bound. I1105 20:32:37.428] Nov 5 20:32:37.428: INFO: PersistentVolumeClaim pvc-vrjw8 found but phase is Pending instead of Bound. I1105 20:32:39.508] Nov 5 20:32:39.508: INFO: PersistentVolumeClaim pvc-vrjw8 found but phase is Pending instead of Bound. I1105 20:32:41.587] Nov 5 20:32:41.587: INFO: PersistentVolumeClaim pvc-vrjw8 found but phase is Pending instead of Bound. I1105 20:32:43.667] Nov 5 20:32:43.666: INFO: PersistentVolumeClaim pvc-vrjw8 found but phase is Pending instead of Bound. I1105 20:32:45.747] Nov 5 20:32:45.746: INFO: PersistentVolumeClaim pvc-vrjw8 found but phase is Pending instead of Bound. I1105 20:32:47.826] Nov 5 20:32:47.826: INFO: PersistentVolumeClaim pvc-vrjw8 found but phase is Pending instead of Bound. I1105 20:32:49.905] Nov 5 20:32:49.905: INFO: PersistentVolumeClaim pvc-vrjw8 found but phase is Pending instead of Bound. I1105 20:32:51.985] Nov 5 20:32:51.984: INFO: PersistentVolumeClaim pvc-vrjw8 found but phase is Pending instead of Bound. I1105 20:32:53.985] Nov 5 20:32:53.985: INFO: Unexpected error occurred: PersistentVolumeClaims [pvc-vrjw8] not all in phase Bound within 5m0s I1105 20:32:53.985] [AfterEach] [multi-az] I1105 20:32:53.986] /go/src/sigs.k8s.io/azuredisk-csi-driver/vendor/k8s.io/kubernetes/test/e2e/framework/framework.go:151 I1105 20:32:53.986] STEP: Collecting events from namespace "azuredisk-8704". I1105 20:32:54.065] STEP: Found 3 events. I1105 20:32:54.066] Nov 5 20:32:54.065: INFO: At 2019-11-05 20:27:52 +0000 UTC - event for pvc-vrjw8: {disk.csi.azure.com_csi-azuredisk-controller-6f6587bfc6-62nrt_3930bc69-0009-11ea-8cf5-829a27ff7d5a } Provisioning: External provisioner is provisioning volume for claim "azuredisk-8704/pvc-vrjw8" I1105 20:32:54.066] Nov 5 20:32:54.065: INFO: At 2019-11-05 20:27:52 +0000 UTC - event for pvc-vrjw8: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "disk.csi.azure.com" or manually created by system administrator I1105 20:32:54.067] Nov 5 20:32:54.065: INFO: At 2019-11-05 20:27:52 +0000 UTC - event for pvc-vrjw8: {disk.csi.azure.com_csi-azuredisk-controller-6f6587bfc6-62nrt_3930bc69-0009-11ea-8cf5-829a27ff7d5a } ProvisioningFailed: failed to provision volume with StorageClass "azuredisk-8704-disk.csi.azure.com-dynamic-sc-njm8g": rpc error: code = Unknown desc = compute.DisksClient#CreateOrUpdate: Failure sending request: StatusCode=400 -- Original Error: Code="InvalidAvailabilityZone" Message="The zone(s) '' for resource 'Microsoft.Compute/disks/pvc-407455f4-7515-4f11-accd-f7a60ea556ce' is not supported. The supported zones for location 'eastus2' are '1,2,3'" I1105 20:32:54.145] Nov 5 20:32:54.145: INFO: POD NODE PHASE GRACE CONDITIONS I1105 20:32:54.145] Nov 5 20:32:54.145: INFO: I1105 20:32:54.379] Nov 5 20:32:54.379: INFO: I1105 20:32:54.379] Logging node info for node k8s-agentpool-97435445-vmss000000 I1105 20:32:54.462] Nov 5 20:32:54.459: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-agentpool-97435445-vmss000000,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-agentpool-97435445-vmss000000,UID:e955dc47-8c8f-4142-9241-3d409f32e42d,ResourceVersion:5431,Generation:0,CreationTimestamp:2019-11-05 20:07:34 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{agentpool: agentpool,beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_D4s_v3,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 0,kubernetes.azure.com/cluster: kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b,kubernetes.azure.com/role: agent,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-agentpool-97435445-vmss000000,kubernetes.io/os: linux,kubernetes.io/role: agent,node-role.kubernetes.io/agent: ,storageprofile: managed,storagetier: Premium_LRS,topology.disk.csi.azure.com/zone: 0,},Annotations:map[string]string{csi.volume.kubernetes.io/nodeid: {"disk.csi.azure.com":"k8s-agentpool-97435445-vmss000000"},node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.2.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool-97435445-vmss/virtualMachines/0,Unschedulable:false,Taints:[],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-csi-disk.csi.azure.com: {{8 0} {} 8 DecimalSI},cpu: {{4 0} {} 4 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{16797040640 0} {} BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-csi-disk.csi.azure.com: {{8 0} {} 8 DecimalSI},cpu: {{4 0} {} 4 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{16010608640 0} {} BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 20:08:01 +0000 UTC 2019-11-05 20:08:01 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 20:32:39 +0000 UTC 2019-11-05 20:07:27 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 20:32:39 +0000 UTC 2019-11-05 20:07:27 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 20:32:39 +0000 UTC 2019-11-05 20:07:27 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 20:32:39 +0000 UTC 2019-11-05 20:07:55 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.0.4} {Hostname k8s-agentpool-97435445-vmss000000}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:bfb55325809340228149d2d93f50df58,SystemUUID:CB2ED036-0439-F641-BC7F-640C3638C027,BootID:a917f4d1-549d-471d-b7dc-47ae1715d97d,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[k8sprowinternal.azurecr.io/azuredisk-csi@sha256:75d4ff5d7cd845d6e810c67796ecb037e852fb849ec1eb5b3194564008e6d1e0 k8sprowinternal.azurecr.io/azuredisk-csi:e2e-3e3ed045c6faa509dbd8d2890ee9f56aa2934887] 286930414} {[k8s.gcr.io/kubernetes-dashboard-amd64@sha256:0ae6b69432e78069c5ce2bcde0fe409c5c4d6f0f4d9cd50a17974fea38898747 k8s.gcr.io/kubernetes-dashboard-amd64:v1.10.1] 121711221} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/metrics-server-amd64@sha256:49a9f12f7067d11f42c803dbe61ed2c1299959ad85cb315b25ff7eef8e6b8892 k8s.gcr.io/metrics-server-amd64:v0.2.1] 42541759} {[k8s.gcr.io/coredns@sha256:e83beb5e43f8513fa735e77ffc5859640baea30a882a11cc75c4c3244a737d3c k8s.gcr.io/coredns:1.5.0] 42488424} {[mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume@sha256:4fd30d43947d4a54fc89ead7985beecfd3c9b2a93a0655a373b1608ab90bd5af mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume:v0.0.7] 22909487} {[quay.io/k8scsi/livenessprobe@sha256:8c9c8a0f330acc2e558ae6624309447d02522e76aa35af23fa19b88abf4381cb quay.io/k8scsi/livenessprobe:v1.0.2] 17415381} {[quay.io/k8scsi/csi-node-driver-registrar@sha256:13daf82fb99e951a4bff8ae5fc7c17c3a8fe7130be6400990d8f6076c32d4599 quay.io/k8scsi/csi-node-driver-registrar:v1.1.0] 15815995} {[mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume@sha256:23d8c6033f02a1ecad05127ebdc931bb871264228661bc122704b0974e4d9fdd mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume:1.0.8] 1159025} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 20:32:54.462] Nov 5 20:32:54.459: INFO: I1105 20:32:54.462] Logging kubelet events for node k8s-agentpool-97435445-vmss000000 I1105 20:32:54.542] Nov 5 20:32:54.542: INFO: I1105 20:32:54.543] Logging pods the kubelet thinks is on node k8s-agentpool-97435445-vmss000000 I1105 20:32:54.630] Nov 5 20:32:54.630: INFO: kube-proxy-8mt7f started at 2019-11-05 20:08:01 +0000 UTC (0+1 container statuses recorded) I1105 20:32:54.630] Nov 5 20:32:54.630: INFO: Container kube-proxy ready: true, restart count 0 I1105 20:32:54.630] Nov 5 20:32:54.630: INFO: kubernetes-dashboard-66dd8b8df7-t6j4l started at 2019-11-05 20:08:01 +0000 UTC (0+1 container statuses recorded) I1105 20:32:54.631] Nov 5 20:32:54.630: INFO: Container kubernetes-dashboard ready: true, restart count 0 I1105 20:32:54.631] Nov 5 20:32:54.630: INFO: csi-azuredisk-node-5wwhz started at 2019-11-05 20:16:09 +0000 UTC (0+3 container statuses recorded) I1105 20:32:54.631] Nov 5 20:32:54.630: INFO: Container azuredisk ready: true, restart count 0 I1105 20:32:54.631] Nov 5 20:32:54.630: INFO: Container liveness-probe ready: true, restart count 0 I1105 20:32:54.631] Nov 5 20:32:54.630: INFO: Container node-driver-registrar ready: true, restart count 0 I1105 20:32:54.631] Nov 5 20:32:54.630: INFO: blobfuse-flexvol-installer-p7hdb started at 2019-11-05 20:08:01 +0000 UTC (0+1 container statuses recorded) I1105 20:32:54.631] Nov 5 20:32:54.630: INFO: Container blobfuse-flexvol-installer ready: true, restart count 0 I1105 20:32:54.632] Nov 5 20:32:54.630: INFO: coredns-7f68dcdbdb-zh8g5 started at 2019-11-05 20:08:01 +0000 UTC (0+1 container statuses recorded) I1105 20:32:54.632] Nov 5 20:32:54.630: INFO: Container coredns ready: true, restart count 0 I1105 20:32:54.632] Nov 5 20:32:54.630: INFO: azure-ip-masq-agent-m7tv9 started at 2019-11-05 20:08:01 +0000 UTC (0+1 container statuses recorded) I1105 20:32:54.632] Nov 5 20:32:54.630: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 20:32:54.632] Nov 5 20:32:54.630: INFO: keyvault-flexvolume-smspt started at 2019-11-05 20:08:01 +0000 UTC (0+1 container statuses recorded) I1105 20:32:54.632] Nov 5 20:32:54.630: INFO: Container keyvault-flexvolume ready: true, restart count 0 I1105 20:32:54.632] Nov 5 20:32:54.630: INFO: metrics-server-864ffbc5c-wl84z started at 2019-11-05 20:08:01 +0000 UTC (0+1 container statuses recorded) I1105 20:32:54.633] Nov 5 20:32:54.630: INFO: Container metrics-server ready: true, restart count 0 I1105 20:32:54.885] Nov 5 20:32:54.885: INFO: I1105 20:32:54.886] Latency metrics for node k8s-agentpool-97435445-vmss000000 I1105 20:32:54.886] Nov 5 20:32:54.885: INFO: I1105 20:32:54.886] Logging node info for node k8s-agentpool-97435445-vmss000001 I1105 20:32:54.969] Nov 5 20:32:54.965: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-agentpool-97435445-vmss000001,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-agentpool-97435445-vmss000001,UID:75d3cc50-ca6d-45e0-a42a-6090e8d537dd,ResourceVersion:5428,Generation:0,CreationTimestamp:2019-11-05 20:07:34 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{agentpool: agentpool,beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_D4s_v3,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 1,kubernetes.azure.com/cluster: kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b,kubernetes.azure.com/role: agent,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-agentpool-97435445-vmss000001,kubernetes.io/os: linux,kubernetes.io/role: agent,node-role.kubernetes.io/agent: ,storageprofile: managed,storagetier: Premium_LRS,topology.disk.csi.azure.com/zone: 1,},Annotations:map[string]string{csi.volume.kubernetes.io/nodeid: {"disk.csi.azure.com":"k8s-agentpool-97435445-vmss000001"},node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.0.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool-97435445-vmss/virtualMachines/1,Unschedulable:false,Taints:[],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-csi-disk.csi.azure.com: {{8 0} {} 8 DecimalSI},cpu: {{4 0} {} 4 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{16797044736 0} {} 16403364Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-csi-disk.csi.azure.com: {{8 0} {} 8 DecimalSI},cpu: {{4 0} {} 4 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{16010612736 0} {} 15635364Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 20:08:25 +0000 UTC 2019-11-05 20:08:25 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 20:32:38 +0000 UTC 2019-11-05 20:07:26 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 20:32:38 +0000 UTC 2019-11-05 20:07:26 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 20:32:38 +0000 UTC 2019-11-05 20:07:26 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 20:32:38 +0000 UTC 2019-11-05 20:07:55 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.0.5} {Hostname k8s-agentpool-97435445-vmss000001}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:e78def54b2f545c6b94d88bbba523c60,SystemUUID:40AE8188-86AB-D848-84A4-D9254E1258BC,BootID:8587ec06-f4f4-43c1-86cd-06c295b05b63,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[k8sprowinternal.azurecr.io/azuredisk-csi@sha256:75d4ff5d7cd845d6e810c67796ecb037e852fb849ec1eb5b3194564008e6d1e0 k8sprowinternal.azurecr.io/azuredisk-csi:e2e-3e3ed045c6faa509dbd8d2890ee9f56aa2934887] 286930414} {[gcr.io/kubernetes-helm/tiller@sha256:f6d8f4ab9ba993b5f5b60a6edafe86352eabe474ffeb84cb6c79b8866dce45d1 gcr.io/kubernetes-helm/tiller:v2.11.0] 71821984} {[quay.io/k8scsi/csi-provisioner@sha256:d657c839dce87324fe2b677302913f9386f885f8746be7bea0ced5b0844e3433 quay.io/k8scsi/csi-provisioner:v1.3.1] 50596998} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[quay.io/k8scsi/csi-attacher@sha256:26fccd7a99d973845df1193b46ebdcc6ab8dc5f6e6be319750c471fce1742d13 quay.io/k8scsi/csi-attacher:v1.2.0] 46226754} {[quay.io/k8scsi/csi-cluster-driver-registrar@sha256:fafd75ae5442f192cfa8c2e792903aee30d5884b62e802e4464b0a895d21e3ef quay.io/k8scsi/csi-cluster-driver-registrar:v1.0.1] 45874691} {[quay.io/k8scsi/csi-snapshotter@sha256:a49e0da1af6f2bf717e41ba1eee8b5e6a1cbd66a709dd92cc43fe475fe2589eb quay.io/k8scsi/csi-snapshotter:v1.1.0] 44325352} {[mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume@sha256:4fd30d43947d4a54fc89ead7985beecfd3c9b2a93a0655a373b1608ab90bd5af mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume:v0.0.7] 22909487} {[quay.io/k8scsi/livenessprobe@sha256:8c9c8a0f330acc2e558ae6624309447d02522e76aa35af23fa19b88abf4381cb quay.io/k8scsi/livenessprobe:v1.0.2] 17415381} {[quay.io/k8scsi/csi-node-driver-registrar@sha256:13daf82fb99e951a4bff8ae5fc7c17c3a8fe7130be6400990d8f6076c32d4599 quay.io/k8scsi/csi-node-driver-registrar:v1.1.0] 15815995} {[quay.io/k8scsi/livenessprobe@sha256:dde617756e0f602adc566ab71fd885f1dad451ad3fb063ac991c95a2ff47aea5 quay.io/k8scsi/livenessprobe:v1.1.0] 14967303} {[mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume@sha256:23d8c6033f02a1ecad05127ebdc931bb871264228661bc122704b0974e4d9fdd mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume:1.0.8] 1159025} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 20:32:54.969] Nov 5 20:32:54.965: INFO: I1105 20:32:54.969] Logging kubelet events for node k8s-agentpool-97435445-vmss000001 I1105 20:32:55.051] Nov 5 20:32:55.051: INFO: I1105 20:32:55.052] Logging pods the kubelet thinks is on node k8s-agentpool-97435445-vmss000001 I1105 20:32:55.140] Nov 5 20:32:55.139: INFO: csi-azuredisk-controller-6f6587bfc6-62nrt started at 2019-11-05 20:16:09 +0000 UTC (0+6 container statuses recorded) I1105 20:32:55.140] Nov 5 20:32:55.139: INFO: Container azuredisk ready: true, restart count 0 I1105 20:32:55.140] Nov 5 20:32:55.139: INFO: Container cluster-driver-registrar ready: true, restart count 0 I1105 20:32:55.140] Nov 5 20:32:55.139: INFO: Container csi-attacher ready: true, restart count 0 I1105 20:32:55.140] Nov 5 20:32:55.139: INFO: Container csi-provisioner ready: true, restart count 0 I1105 20:32:55.141] Nov 5 20:32:55.139: INFO: Container csi-snapshotter ready: true, restart count 0 I1105 20:32:55.141] Nov 5 20:32:55.139: INFO: Container liveness-probe ready: true, restart count 0 I1105 20:32:55.141] Nov 5 20:32:55.139: INFO: blobfuse-flexvol-installer-j4s7t started at 2019-11-05 20:07:59 +0000 UTC (0+1 container statuses recorded) I1105 20:32:55.141] Nov 5 20:32:55.139: INFO: Container blobfuse-flexvol-installer ready: true, restart count 0 I1105 20:32:55.141] Nov 5 20:32:55.139: INFO: azure-ip-masq-agent-gdhcq started at 2019-11-05 20:07:59 +0000 UTC (0+1 container statuses recorded) I1105 20:32:55.141] Nov 5 20:32:55.139: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 20:32:55.142] Nov 5 20:32:55.139: INFO: keyvault-flexvolume-6bffx started at 2019-11-05 20:07:59 +0000 UTC (0+1 container statuses recorded) I1105 20:32:55.142] Nov 5 20:32:55.139: INFO: Container keyvault-flexvolume ready: true, restart count 0 I1105 20:32:55.142] Nov 5 20:32:55.139: INFO: kube-proxy-kjrpb started at 2019-11-05 20:08:00 +0000 UTC (0+1 container statuses recorded) I1105 20:32:55.142] Nov 5 20:32:55.139: INFO: Container kube-proxy ready: true, restart count 0 I1105 20:32:55.142] Nov 5 20:32:55.139: INFO: tiller-deploy-54c96cb5df-pt7x6 started at 2019-11-05 20:08:00 +0000 UTC (0+1 container statuses recorded) I1105 20:32:55.143] Nov 5 20:32:55.139: INFO: Container tiller ready: true, restart count 0 I1105 20:32:55.143] Nov 5 20:32:55.139: INFO: csi-azuredisk-node-q2dmt started at 2019-11-05 20:16:09 +0000 UTC (0+3 container statuses recorded) I1105 20:32:55.143] Nov 5 20:32:55.139: INFO: Container azuredisk ready: true, restart count 0 I1105 20:32:55.143] Nov 5 20:32:55.139: INFO: Container liveness-probe ready: true, restart count 0 I1105 20:32:55.143] Nov 5 20:32:55.140: INFO: Container node-driver-registrar ready: true, restart count 0 I1105 20:32:55.396] Nov 5 20:32:55.396: INFO: I1105 20:32:55.396] Latency metrics for node k8s-agentpool-97435445-vmss000001 I1105 20:32:55.396] Nov 5 20:32:55.396: INFO: I1105 20:32:55.397] Logging node info for node k8s-master-97435445-0 I1105 20:32:55.481] Nov 5 20:32:55.479: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-97435445-0,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-97435445-0,UID:df3ae36f-961c-4729-a8f5-34cda54f4ff9,ResourceVersion:5315,Generation:0,CreationTimestamp:2019-11-05 20:08:36 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 2,kubernetes.azure.com/cluster: kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-97435445-0,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.4.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b/providers/Microsoft.Compute/virtualMachines/k8s-master-97435445-0,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 20:08:46 +0000 UTC 2019-11-05 20:08:46 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 20:31:59 +0000 UTC 2019-11-05 20:08:28 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 20:31:59 +0000 UTC 2019-11-05 20:08:28 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 20:31:59 +0000 UTC 2019-11-05 20:08:28 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 20:31:59 +0000 UTC 2019-11-05 20:08:36 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.255.5} {Hostname k8s-master-97435445-0}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:3c381865192341c2868672e8626cef7d,SystemUUID:F3D259D3-7EB8-B844-A1F0-F954DCD0D76B,BootID:4948329e-cf57-48c3-aa83-5297bef70370,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 20:32:55.482] Nov 5 20:32:55.479: INFO: I1105 20:32:55.482] Logging kubelet events for node k8s-master-97435445-0 I1105 20:32:55.563] Nov 5 20:32:55.563: INFO: I1105 20:32:55.564] Logging pods the kubelet thinks is on node k8s-master-97435445-0 I1105 20:32:55.648] Nov 5 20:32:55.648: INFO: cloud-controller-manager-k8s-master-97435445-0 started at 2019-11-05 20:08:22 +0000 UTC (0+1 container statuses recorded) I1105 20:32:55.648] Nov 5 20:32:55.648: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 20:32:55.648] Nov 5 20:32:55.648: INFO: kube-addon-manager-k8s-master-97435445-0 started at 2019-11-05 20:08:48 +0000 UTC (0+1 container statuses recorded) I1105 20:32:55.648] Nov 5 20:32:55.648: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 20:32:55.648] Nov 5 20:32:55.648: INFO: kube-apiserver-k8s-master-97435445-0 started at 2019-11-05 20:08:22 +0000 UTC (0+1 container statuses recorded) I1105 20:32:55.649] Nov 5 20:32:55.648: INFO: Container kube-apiserver ready: true, restart count 0 I1105 20:32:55.649] Nov 5 20:32:55.648: INFO: kube-controller-manager-k8s-master-97435445-0 started at 2019-11-05 20:08:22 +0000 UTC (0+1 container statuses recorded) I1105 20:32:55.649] Nov 5 20:32:55.648: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 20:32:55.649] Nov 5 20:32:55.648: INFO: kube-scheduler-k8s-master-97435445-0 started at 2019-11-05 20:08:22 +0000 UTC (0+1 container statuses recorded) I1105 20:32:55.649] Nov 5 20:32:55.648: INFO: Container kube-scheduler ready: true, restart count 0 I1105 20:32:55.650] Nov 5 20:32:55.648: INFO: azure-ip-masq-agent-q6dcl started at 2019-11-05 20:08:38 +0000 UTC (0+1 container statuses recorded) I1105 20:32:55.650] Nov 5 20:32:55.648: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 20:32:55.650] Nov 5 20:32:55.648: INFO: kube-proxy-5wnd2 started at 2019-11-05 20:08:38 +0000 UTC (0+1 container statuses recorded) I1105 20:32:55.650] Nov 5 20:32:55.648: INFO: Container kube-proxy ready: true, restart count 0 I1105 20:32:55.901] Nov 5 20:32:55.901: INFO: I1105 20:32:55.901] Latency metrics for node k8s-master-97435445-0 I1105 20:32:55.901] Nov 5 20:32:55.901: INFO: I1105 20:32:55.901] Logging node info for node k8s-master-97435445-1 I1105 20:32:55.983] Nov 5 20:32:55.980: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-97435445-1,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-97435445-1,UID:05534a67-d493-4ab5-9b0f-5ff5490e234f,ResourceVersion:5395,Generation:0,CreationTimestamp:2019-11-05 20:07:34 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 0,kubernetes.azure.com/cluster: kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-97435445-1,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.1.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b/providers/Microsoft.Compute/virtualMachines/k8s-master-97435445-1,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 20:08:26 +0000 UTC 2019-11-05 20:08:26 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 20:32:27 +0000 UTC 2019-11-05 20:07:28 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 20:32:27 +0000 UTC 2019-11-05 20:07:28 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 20:32:27 +0000 UTC 2019-11-05 20:07:28 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 20:32:27 +0000 UTC 2019-11-05 20:07:55 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{Hostname k8s-master-97435445-1} {InternalIP 10.240.255.6}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:079b605103b74acabac18fd2ea63368a,SystemUUID:38220EEC-BEFF-4C46-AA05-5A214AAFDC28,BootID:b1b45df0-d925-4bcc-9830-0202df36ebb4,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 20:32:55.983] Nov 5 20:32:55.980: INFO: I1105 20:32:55.983] Logging kubelet events for node k8s-master-97435445-1 I1105 20:32:56.064] Nov 5 20:32:56.064: INFO: I1105 20:32:56.064] Logging pods the kubelet thinks is on node k8s-master-97435445-1 I1105 20:32:56.148] Nov 5 20:32:56.148: INFO: kube-addon-manager-k8s-master-97435445-1 started at 2019-11-05 20:07:22 +0000 UTC (0+1 container statuses recorded) I1105 20:32:56.148] Nov 5 20:32:56.148: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 20:32:56.149] Nov 5 20:32:56.148: INFO: kube-apiserver-k8s-master-97435445-1 started at 2019-11-05 20:07:22 +0000 UTC (0+1 container statuses recorded) I1105 20:32:56.149] Nov 5 20:32:56.148: INFO: Container kube-apiserver ready: true, restart count 0 I1105 20:32:56.149] Nov 5 20:32:56.148: INFO: kube-controller-manager-k8s-master-97435445-1 started at 2019-11-05 20:07:22 +0000 UTC (0+1 container statuses recorded) I1105 20:32:56.149] Nov 5 20:32:56.148: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 20:32:56.149] Nov 5 20:32:56.148: INFO: kube-scheduler-k8s-master-97435445-1 started at 2019-11-05 20:07:22 +0000 UTC (0+1 container statuses recorded) I1105 20:32:56.150] Nov 5 20:32:56.148: INFO: Container kube-scheduler ready: true, restart count 0 I1105 20:32:56.150] Nov 5 20:32:56.148: INFO: azure-ip-masq-agent-n29bm started at 2019-11-05 20:07:59 +0000 UTC (0+1 container statuses recorded) I1105 20:32:56.150] Nov 5 20:32:56.148: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 20:32:56.150] Nov 5 20:32:56.148: INFO: kube-proxy-h4s2c started at 2019-11-05 20:08:00 +0000 UTC (0+1 container statuses recorded) I1105 20:32:56.150] Nov 5 20:32:56.148: INFO: Container kube-proxy ready: true, restart count 0 I1105 20:32:56.150] Nov 5 20:32:56.148: INFO: cloud-controller-manager-k8s-master-97435445-1 started at 2019-11-05 20:07:22 +0000 UTC (0+1 container statuses recorded) I1105 20:32:56.151] Nov 5 20:32:56.148: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 20:32:56.402] Nov 5 20:32:56.402: INFO: I1105 20:32:56.403] Latency metrics for node k8s-master-97435445-1 I1105 20:32:56.403] Nov 5 20:32:56.402: INFO: I1105 20:32:56.403] Logging node info for node k8s-master-97435445-2 I1105 20:32:56.485] Nov 5 20:32:56.482: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-97435445-2,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-97435445-2,UID:fa55c895-1a93-4743-885e-03fce84b5332,ResourceVersion:5444,Generation:0,CreationTimestamp:2019-11-05 20:09:19 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 0,kubernetes.azure.com/cluster: kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-97435445-2,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.5.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b/providers/Microsoft.Compute/virtualMachines/k8s-master-97435445-2,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 20:09:19 +0000 UTC 2019-11-05 20:09:19 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 20:32:43 +0000 UTC 2019-11-05 20:09:12 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 20:32:43 +0000 UTC 2019-11-05 20:09:12 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 20:32:43 +0000 UTC 2019-11-05 20:09:12 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 20:32:43 +0000 UTC 2019-11-05 20:09:19 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.255.7} {Hostname k8s-master-97435445-2}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:d19f6709138f496da43c095cec7b2360,SystemUUID:10116777-A8D7-CB4A-A8B7-7EBCF2F23B62,BootID:50cd14a9-5497-4972-bf60-cf59a0cbcb5f,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 20:32:56.485] Nov 5 20:32:56.482: INFO: I1105 20:32:56.485] Logging kubelet events for node k8s-master-97435445-2 I1105 20:32:56.567] Nov 5 20:32:56.567: INFO: I1105 20:32:56.567] Logging pods the kubelet thinks is on node k8s-master-97435445-2 I1105 20:32:56.652] Nov 5 20:32:56.651: INFO: kube-apiserver-k8s-master-97435445-2 started at 2019-11-05 20:09:05 +0000 UTC (0+1 container statuses recorded) I1105 20:32:56.652] Nov 5 20:32:56.651: INFO: Container kube-apiserver ready: true, restart count 0 I1105 20:32:56.652] Nov 5 20:32:56.651: INFO: kube-controller-manager-k8s-master-97435445-2 started at 2019-11-05 20:09:05 +0000 UTC (0+1 container statuses recorded) I1105 20:32:56.652] Nov 5 20:32:56.651: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 20:32:56.652] Nov 5 20:32:56.651: INFO: kube-scheduler-k8s-master-97435445-2 started at 2019-11-05 20:09:05 +0000 UTC (0+1 container statuses recorded) I1105 20:32:56.652] Nov 5 20:32:56.651: INFO: Container kube-scheduler ready: true, restart count 0 I1105 20:32:56.653] Nov 5 20:32:56.651: INFO: cloud-controller-manager-k8s-master-97435445-2 started at 2019-11-05 20:09:05 +0000 UTC (0+1 container statuses recorded) I1105 20:32:56.653] Nov 5 20:32:56.651: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 20:32:56.653] Nov 5 20:32:56.651: INFO: kube-addon-manager-k8s-master-97435445-2 started at 2019-11-05 20:09:05 +0000 UTC (0+1 container statuses recorded) I1105 20:32:56.653] Nov 5 20:32:56.651: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 20:32:56.653] Nov 5 20:32:56.651: INFO: azure-ip-masq-agent-sq7p5 started at 2019-11-05 20:09:19 +0000 UTC (0+1 container statuses recorded) I1105 20:32:56.653] Nov 5 20:32:56.651: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 20:32:56.653] Nov 5 20:32:56.651: INFO: kube-proxy-pz426 started at 2019-11-05 20:09:19 +0000 UTC (0+1 container statuses recorded) I1105 20:32:56.653] Nov 5 20:32:56.651: INFO: Container kube-proxy ready: true, restart count 0 I1105 20:32:56.912] Nov 5 20:32:56.911: INFO: I1105 20:32:56.912] Latency metrics for node k8s-master-97435445-2 I1105 20:32:56.912] Nov 5 20:32:56.911: INFO: I1105 20:32:56.912] Logging node info for node k8s-master-97435445-3 I1105 20:32:56.993] Nov 5 20:32:56.991: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-97435445-3,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-97435445-3,UID:8ffd1d0d-4487-4680-a2f6-e724387be637,ResourceVersion:5447,Generation:0,CreationTimestamp:2019-11-05 20:09:21 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 1,kubernetes.azure.com/cluster: kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-97435445-3,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.6.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b/providers/Microsoft.Compute/virtualMachines/k8s-master-97435445-3,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 20:09:39 +0000 UTC 2019-11-05 20:09:39 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 20:32:44 +0000 UTC 2019-11-05 20:09:18 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 20:32:44 +0000 UTC 2019-11-05 20:09:18 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 20:32:44 +0000 UTC 2019-11-05 20:09:18 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 20:32:44 +0000 UTC 2019-11-05 20:09:21 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.255.8} {Hostname k8s-master-97435445-3}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:266064173bfb4297898c6c25769e8b20,SystemUUID:E6E44241-5015-7740-9A81-A668A8427206,BootID:579d658c-b0ad-47f1-a993-1ccea2b55789,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 20:32:56.993] Nov 5 20:32:56.991: INFO: I1105 20:32:56.993] Logging kubelet events for node k8s-master-97435445-3 I1105 20:32:57.075] Nov 5 20:32:57.075: INFO: I1105 20:32:57.075] Logging pods the kubelet thinks is on node k8s-master-97435445-3 I1105 20:32:57.160] Nov 5 20:32:57.159: INFO: cloud-controller-manager-k8s-master-97435445-3 started at 2019-11-05 20:09:04 +0000 UTC (0+1 container statuses recorded) I1105 20:32:57.160] Nov 5 20:32:57.159: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 20:32:57.160] Nov 5 20:32:57.159: INFO: kube-addon-manager-k8s-master-97435445-3 started at 2019-11-05 20:09:05 +0000 UTC (0+1 container statuses recorded) I1105 20:32:57.160] Nov 5 20:32:57.159: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 20:32:57.160] Nov 5 20:32:57.159: INFO: kube-apiserver-k8s-master-97435445-3 started at 2019-11-05 20:09:05 +0000 UTC (0+1 container statuses recorded) I1105 20:32:57.161] Nov 5 20:32:57.160: INFO: Container kube-apiserver ready: true, restart count 0 I1105 20:32:57.161] Nov 5 20:32:57.160: INFO: kube-controller-manager-k8s-master-97435445-3 started at 2019-11-05 20:09:05 +0000 UTC (0+1 container statuses recorded) I1105 20:32:57.161] Nov 5 20:32:57.160: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 20:32:57.161] Nov 5 20:32:57.160: INFO: kube-scheduler-k8s-master-97435445-3 started at 2019-11-05 20:09:05 +0000 UTC (0+1 container statuses recorded) I1105 20:32:57.161] Nov 5 20:32:57.160: INFO: Container kube-scheduler ready: true, restart count 0 I1105 20:32:57.161] Nov 5 20:32:57.160: INFO: azure-ip-masq-agent-747cl started at 2019-11-05 20:09:21 +0000 UTC (0+1 container statuses recorded) I1105 20:32:57.162] Nov 5 20:32:57.160: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 20:32:57.162] Nov 5 20:32:57.160: INFO: kube-proxy-k45g8 started at 2019-11-05 20:09:21 +0000 UTC (0+1 container statuses recorded) I1105 20:32:57.162] Nov 5 20:32:57.160: INFO: Container kube-proxy ready: true, restart count 0 I1105 20:32:57.423] Nov 5 20:32:57.423: INFO: I1105 20:32:57.423] Latency metrics for node k8s-master-97435445-3 I1105 20:32:57.423] Nov 5 20:32:57.423: INFO: I1105 20:32:57.423] Logging node info for node k8s-master-97435445-4 I1105 20:32:57.505] Nov 5 20:32:57.502: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-97435445-4,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-97435445-4,UID:6d547b41-3f92-4cc4-a2d7-dda2071776a2,ResourceVersion:5314,Generation:0,CreationTimestamp:2019-11-05 20:08:36 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 1,kubernetes.azure.com/cluster: kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-97435445-4,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.3.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b/providers/Microsoft.Compute/virtualMachines/k8s-master-97435445-4,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 20:08:59 +0000 UTC 2019-11-05 20:08:59 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 20:31:59 +0000 UTC 2019-11-05 20:08:29 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 20:31:59 +0000 UTC 2019-11-05 20:08:29 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 20:31:59 +0000 UTC 2019-11-05 20:08:29 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 20:31:59 +0000 UTC 2019-11-05 20:08:36 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.255.9} {Hostname k8s-master-97435445-4}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:56b393c61eb34ba49ebda5279eeedb30,SystemUUID:FD239C23-618C-0049-8734-9AF3FDC9E3C1,BootID:f342ea44-e719-4513-807d-5def7ac606e2,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 20:32:57.505] Nov 5 20:32:57.502: INFO: I1105 20:32:57.505] Logging kubelet events for node k8s-master-97435445-4 I1105 20:32:57.587] Nov 5 20:32:57.586: INFO: I1105 20:32:57.587] Logging pods the kubelet thinks is on node k8s-master-97435445-4 I1105 20:32:57.671] Nov 5 20:32:57.671: INFO: kube-apiserver-k8s-master-97435445-4 started at 2019-11-05 20:08:23 +0000 UTC (0+1 container statuses recorded) I1105 20:32:57.671] Nov 5 20:32:57.671: INFO: Container kube-apiserver ready: true, restart count 0 I1105 20:32:57.671] Nov 5 20:32:57.671: INFO: kube-controller-manager-k8s-master-97435445-4 started at 2019-11-05 20:08:23 +0000 UTC (0+1 container statuses recorded) I1105 20:32:57.672] Nov 5 20:32:57.671: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 20:32:57.672] Nov 5 20:32:57.671: INFO: kube-scheduler-k8s-master-97435445-4 started at 2019-11-05 20:08:23 +0000 UTC (0+1 container statuses recorded) I1105 20:32:57.672] Nov 5 20:32:57.671: INFO: Container kube-scheduler ready: true, restart count 0 I1105 20:32:57.672] Nov 5 20:32:57.671: INFO: kube-proxy-vfbfr started at 2019-11-05 20:08:36 +0000 UTC (0+1 container statuses recorded) I1105 20:32:57.672] Nov 5 20:32:57.671: INFO: Container kube-proxy ready: true, restart count 0 I1105 20:32:57.672] Nov 5 20:32:57.671: INFO: azure-ip-masq-agent-kmkll started at 2019-11-05 20:08:36 +0000 UTC (0+1 container statuses recorded) I1105 20:32:57.673] Nov 5 20:32:57.671: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 20:32:57.673] Nov 5 20:32:57.671: INFO: cloud-controller-manager-k8s-master-97435445-4 started at 2019-11-05 20:08:46 +0000 UTC (0+1 container statuses recorded) I1105 20:32:57.673] Nov 5 20:32:57.671: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 20:32:57.673] Nov 5 20:32:57.671: INFO: kube-addon-manager-k8s-master-97435445-4 started at 2019-11-05 20:08:23 +0000 UTC (0+1 container statuses recorded) I1105 20:32:57.673] Nov 5 20:32:57.671: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 20:32:57.949] Nov 5 20:32:57.949: INFO: I1105 20:32:57.949] Latency metrics for node k8s-master-97435445-4 I1105 20:32:57.950] Nov 5 20:32:57.949: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready I1105 20:32:58.030] STEP: Destroying namespace "azuredisk-8704" for this suite. I1105 20:33:12.288] Nov 5 20:33:12.288: INFO: Waiting up to 30s for server preferred namespaced resources to be successfully discovered I1105 20:33:15.275] Nov 5 20:33:15.274: INFO: namespace azuredisk-8704 deletion completed in 17.245135385s I1105 20:33:15.275] I1105 20:33:15.275] • Failure [323.676 seconds] I1105 20:33:15.275] [azuredisk-csi-e2e] Dynamic Provisioning I1105 20:33:15.276] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:33 I1105 20:33:15.276] [multi-az] I1105 20:33:15.276] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:40 I1105 20:33:15.276] [env] should retain PV with reclaimPolicy "Retain" [It] I1105 20:33:15.276] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:240 I1105 20:33:15.276] I1105 20:33:15.277] Unexpected error: I1105 20:33:15.277] <*errors.errorString | 0xc000a3c170>: { I1105 20:33:15.277] s: "PersistentVolumeClaims [pvc-vrjw8] not all in phase Bound within 5m0s", I1105 20:33:15.277] } I1105 20:33:15.277] PersistentVolumeClaims [pvc-vrjw8] not all in phase Bound within 5m0s I1105 20:33:15.277] occurred I1105 20:33:15.277] I1105 20:33:15.278] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/testsuites/testsuites.go:278 I1105 20:33:15.278] ------------------------------ I1105 20:33:15.278] [azuredisk-csi-e2e] Dynamic Provisioning [single-az] I1105 20:33:15.278] should create multiple PV objects, bind to PVCs and attach all to different pods on the same node I1105 20:33:15.278] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:151 I1105 20:33:15.278] [BeforeEach] [single-az] I1105 20:33:15.278] /go/src/sigs.k8s.io/azuredisk-csi-driver/vendor/k8s.io/kubernetes/test/e2e/framework/framework.go:150 I1105 20:33:15.278] STEP: Creating a kubernetes client I1105 20:33:15.279] Nov 5 20:33:15.275: INFO: >>> kubeConfig: /workspace/aks971561099/kubeconfig/kubeconfig.eastus2.json I1105 20:33:15.279] STEP: Building a namespace api object, basename azuredisk I1105 20:33:15.524] STEP: Binding the e2e-test-privileged-psp PodSecurityPolicy to the default service account in azuredisk-5245 I1105 20:33:15.787] STEP: Waiting for a default service account to be provisioned in namespace I1105 20:33:15.864] [BeforeEach] [single-az] I1105 20:33:15.864] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:58 I1105 20:33:15.864] [It] should create multiple PV objects, bind to PVCs and attach all to different pods on the same node I1105 20:33:15.864] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:151 I1105 20:33:15.865] STEP: setting up the StorageClass I1105 20:33:15.865] STEP: creating a StorageClass I1105 20:33:15.947] STEP: setting up the PVC and PV I1105 20:33:15.948] STEP: creating a PVC I1105 20:33:16.035] STEP: waiting for PVC to be in phase "Bound" I1105 20:33:16.036] Nov 5 20:33:16.035: INFO: Waiting up to 5m0s for PersistentVolumeClaims [pvc-fzxvb] to have phase Bound I1105 20:33:16.114] Nov 5 20:33:16.114: INFO: PersistentVolumeClaim pvc-fzxvb found but phase is Pending instead of Bound. I1105 20:33:18.211] Nov 5 20:33:18.211: INFO: PersistentVolumeClaim pvc-fzxvb found but phase is Pending instead of Bound. I1105 20:33:20.290] Nov 5 20:33:20.290: INFO: PersistentVolumeClaim pvc-fzxvb found but phase is Pending instead of Bound. I1105 20:33:22.370] Nov 5 20:33:22.370: INFO: PersistentVolumeClaim pvc-fzxvb found but phase is Pending instead of Bound. I1105 20:33:24.449] Nov 5 20:33:24.449: INFO: PersistentVolumeClaim pvc-fzxvb found but phase is Pending instead of Bound. I1105 20:33:26.529] Nov 5 20:33:26.528: INFO: PersistentVolumeClaim pvc-fzxvb found but phase is Pending instead of Bound. I1105 20:33:28.609] Nov 5 20:33:28.609: INFO: PersistentVolumeClaim pvc-fzxvb found but phase is Pending instead of Bound. I1105 20:33:30.688] Nov 5 20:33:30.688: INFO: PersistentVolumeClaim pvc-fzxvb found but phase is Pending instead of Bound. I1105 20:33:32.767] Nov 5 20:33:32.767: INFO: PersistentVolumeClaim pvc-fzxvb found but phase is Pending instead of Bound. I1105 20:33:34.847] Nov 5 20:33:34.847: INFO: PersistentVolumeClaim pvc-fzxvb found but phase is Pending instead of Bound. I1105 20:33:36.926] Nov 5 20:33:36.926: INFO: PersistentVolumeClaim pvc-fzxvb found but phase is Pending instead of Bound. I1105 20:33:39.012] Nov 5 20:33:39.012: INFO: PersistentVolumeClaim pvc-fzxvb found but phase is Pending instead of Bound. I1105 20:33:41.092] Nov 5 20:33:41.092: INFO: PersistentVolumeClaim pvc-fzxvb found but phase is Pending instead of Bound. I1105 20:33:43.203] Nov 5 20:33:43.202: INFO: PersistentVolumeClaim pvc-fzxvb found but phase is Pending instead of Bound. I1105 20:33:45.282] Nov 5 20:33:45.282: INFO: PersistentVolumeClaim pvc-fzxvb found but phase is Pending instead of Bound. I1105 20:33:47.360] Nov 5 20:33:47.360: INFO: PersistentVolumeClaim pvc-fzxvb found but phase is Pending instead of Bound. I1105 20:33:49.440] Nov 5 20:33:49.440: INFO: PersistentVolumeClaim pvc-fzxvb found but phase is Pending instead of Bound. I1105 20:33:51.520] Nov 5 20:33:51.520: INFO: PersistentVolumeClaim pvc-fzxvb found but phase is Pending instead of Bound. I1105 20:33:53.599] Nov 5 20:33:53.599: INFO: PersistentVolumeClaim pvc-fzxvb found but phase is Pending instead of Bound. I1105 20:33:55.679] Nov 5 20:33:55.679: INFO: PersistentVolumeClaim pvc-fzxvb found but phase is Pending instead of Bound. I1105 20:33:57.758] Nov 5 20:33:57.758: INFO: PersistentVolumeClaim pvc-fzxvb found but phase is Pending instead of Bound. I1105 20:33:59.838] Nov 5 20:33:59.838: INFO: PersistentVolumeClaim pvc-fzxvb found but phase is Pending instead of Bound. I1105 20:34:01.918] Nov 5 20:34:01.918: INFO: PersistentVolumeClaim pvc-fzxvb found but phase is Pending instead of Bound. I1105 20:34:03.998] Nov 5 20:34:03.997: INFO: PersistentVolumeClaim pvc-fzxvb found but phase is Pending instead of Bound. I1105 20:34:06.077] Nov 5 20:34:06.077: INFO: PersistentVolumeClaim pvc-fzxvb found but phase is Pending instead of Bound. I1105 20:34:08.157] Nov 5 20:34:08.156: INFO: PersistentVolumeClaim pvc-fzxvb found but phase is Pending instead of Bound. I1105 20:34:10.236] Nov 5 20:34:10.236: INFO: PersistentVolumeClaim pvc-fzxvb found but phase is Pending instead of Bound. I1105 20:34:12.315] Nov 5 20:34:12.315: INFO: PersistentVolumeClaim pvc-fzxvb found but phase is Pending instead of Bound. I1105 20:34:14.396] Nov 5 20:34:14.396: INFO: PersistentVolumeClaim pvc-fzxvb found but phase is Pending instead of Bound. I1105 20:34:16.476] Nov 5 20:34:16.475: INFO: PersistentVolumeClaim pvc-fzxvb found but phase is Pending instead of Bound. I1105 20:34:18.556] Nov 5 20:34:18.556: INFO: PersistentVolumeClaim pvc-fzxvb found but phase is Pending instead of Bound. I1105 20:34:20.635] Nov 5 20:34:20.635: INFO: PersistentVolumeClaim pvc-fzxvb found but phase is Pending instead of Bound. I1105 20:34:22.715] Nov 5 20:34:22.715: INFO: PersistentVolumeClaim pvc-fzxvb found but phase is Pending instead of Bound. I1105 20:34:24.795] Nov 5 20:34:24.795: INFO: PersistentVolumeClaim pvc-fzxvb found but phase is Pending instead of Bound. I1105 20:34:26.874] Nov 5 20:34:26.874: INFO: PersistentVolumeClaim pvc-fzxvb found but phase is Pending instead of Bound. I1105 20:34:28.954] Nov 5 20:34:28.954: INFO: PersistentVolumeClaim pvc-fzxvb found but phase is Pending instead of Bound. I1105 20:34:31.033] Nov 5 20:34:31.033: INFO: PersistentVolumeClaim pvc-fzxvb found but phase is Pending instead of Bound. I1105 20:34:33.113] Nov 5 20:34:33.113: INFO: PersistentVolumeClaim pvc-fzxvb found but phase is Pending instead of Bound. I1105 20:34:35.193] Nov 5 20:34:35.192: INFO: PersistentVolumeClaim pvc-fzxvb found but phase is Pending instead of Bound. I1105 20:34:37.273] Nov 5 20:34:37.272: INFO: PersistentVolumeClaim pvc-fzxvb found but phase is Pending instead of Bound. I1105 20:34:39.352] Nov 5 20:34:39.352: INFO: PersistentVolumeClaim pvc-fzxvb found but phase is Pending instead of Bound. I1105 20:34:41.431] Nov 5 20:34:41.431: INFO: PersistentVolumeClaim pvc-fzxvb found but phase is Pending instead of Bound. I1105 20:34:43.510] Nov 5 20:34:43.510: INFO: PersistentVolumeClaim pvc-fzxvb found but phase is Pending instead of Bound. I1105 20:34:45.589] Nov 5 20:34:45.589: INFO: PersistentVolumeClaim pvc-fzxvb found but phase is Pending instead of Bound. I1105 20:34:47.670] Nov 5 20:34:47.670: INFO: PersistentVolumeClaim pvc-fzxvb found but phase is Pending instead of Bound. I1105 20:34:49.749] Nov 5 20:34:49.749: INFO: PersistentVolumeClaim pvc-fzxvb found but phase is Pending instead of Bound. I1105 20:34:51.829] Nov 5 20:34:51.829: INFO: PersistentVolumeClaim pvc-fzxvb found but phase is Pending instead of Bound. I1105 20:34:53.909] Nov 5 20:34:53.909: INFO: PersistentVolumeClaim pvc-fzxvb found but phase is Pending instead of Bound. I1105 20:34:55.989] Nov 5 20:34:55.988: INFO: PersistentVolumeClaim pvc-fzxvb found but phase is Pending instead of Bound. I1105 20:34:58.069] Nov 5 20:34:58.069: INFO: PersistentVolumeClaim pvc-fzxvb found but phase is Pending instead of Bound. I1105 20:35:00.148] Nov 5 20:35:00.148: INFO: PersistentVolumeClaim pvc-fzxvb found but phase is Pending instead of Bound. I1105 20:35:02.234] Nov 5 20:35:02.234: INFO: PersistentVolumeClaim pvc-fzxvb found but phase is Pending instead of Bound. I1105 20:35:04.315] Nov 5 20:35:04.314: INFO: PersistentVolumeClaim pvc-fzxvb found but phase is Pending instead of Bound. I1105 20:35:06.394] Nov 5 20:35:06.394: INFO: PersistentVolumeClaim pvc-fzxvb found but phase is Pending instead of Bound. I1105 20:35:08.473] Nov 5 20:35:08.473: INFO: PersistentVolumeClaim pvc-fzxvb found but phase is Pending instead of Bound. I1105 20:35:10.553] Nov 5 20:35:10.553: INFO: PersistentVolumeClaim pvc-fzxvb found but phase is Pending instead of Bound. I1105 20:35:12.633] Nov 5 20:35:12.633: INFO: PersistentVolumeClaim pvc-fzxvb found but phase is Pending instead of Bound. I1105 20:35:14.712] Nov 5 20:35:14.711: INFO: PersistentVolumeClaim pvc-fzxvb found but phase is Pending instead of Bound. I1105 20:35:16.791] Nov 5 20:35:16.791: INFO: PersistentVolumeClaim pvc-fzxvb found but phase is Pending instead of Bound. I1105 20:35:18.871] Nov 5 20:35:18.871: INFO: PersistentVolumeClaim pvc-fzxvb found but phase is Pending instead of Bound. I1105 20:35:20.950] Nov 5 20:35:20.950: INFO: PersistentVolumeClaim pvc-fzxvb found but phase is Pending instead of Bound. I1105 20:35:23.030] Nov 5 20:35:23.029: INFO: PersistentVolumeClaim pvc-fzxvb found but phase is Pending instead of Bound. I1105 20:35:25.110] Nov 5 20:35:25.110: INFO: PersistentVolumeClaim pvc-fzxvb found but phase is Pending instead of Bound. I1105 20:35:27.190] Nov 5 20:35:27.188: INFO: PersistentVolumeClaim pvc-fzxvb found but phase is Pending instead of Bound. I1105 20:35:29.270] Nov 5 20:35:29.270: INFO: PersistentVolumeClaim pvc-fzxvb found but phase is Pending instead of Bound. I1105 20:35:31.350] Nov 5 20:35:31.350: INFO: PersistentVolumeClaim pvc-fzxvb found but phase is Pending instead of Bound. I1105 20:35:33.429] Nov 5 20:35:33.429: INFO: PersistentVolumeClaim pvc-fzxvb found but phase is Pending instead of Bound. I1105 20:35:35.509] Nov 5 20:35:35.508: INFO: PersistentVolumeClaim pvc-fzxvb found but phase is Pending instead of Bound. I1105 20:35:37.588] Nov 5 20:35:37.587: INFO: PersistentVolumeClaim pvc-fzxvb found but phase is Pending instead of Bound. I1105 20:35:39.667] Nov 5 20:35:39.667: INFO: PersistentVolumeClaim pvc-fzxvb found but phase is Pending instead of Bound. I1105 20:35:41.747] Nov 5 20:35:41.747: INFO: PersistentVolumeClaim pvc-fzxvb found but phase is Pending instead of Bound. I1105 20:35:43.826] Nov 5 20:35:43.826: INFO: PersistentVolumeClaim pvc-fzxvb found but phase is Pending instead of Bound. I1105 20:35:45.905] Nov 5 20:35:45.905: INFO: PersistentVolumeClaim pvc-fzxvb found but phase is Pending instead of Bound. I1105 20:35:47.984] Nov 5 20:35:47.984: INFO: PersistentVolumeClaim pvc-fzxvb found but phase is Pending instead of Bound. I1105 20:35:50.064] Nov 5 20:35:50.063: INFO: PersistentVolumeClaim pvc-fzxvb found but phase is Pending instead of Bound. I1105 20:35:52.143] Nov 5 20:35:52.142: INFO: PersistentVolumeClaim pvc-fzxvb found but phase is Pending instead of Bound. I1105 20:35:54.241] Nov 5 20:35:54.241: INFO: PersistentVolumeClaim pvc-fzxvb found but phase is Pending instead of Bound. I1105 20:35:56.321] Nov 5 20:35:56.321: INFO: PersistentVolumeClaim pvc-fzxvb found but phase is Pending instead of Bound. I1105 20:35:58.401] Nov 5 20:35:58.401: INFO: PersistentVolumeClaim pvc-fzxvb found but phase is Pending instead of Bound. I1105 20:36:00.482] Nov 5 20:36:00.482: INFO: PersistentVolumeClaim pvc-fzxvb found but phase is Pending instead of Bound. I1105 20:36:02.561] Nov 5 20:36:02.561: INFO: PersistentVolumeClaim pvc-fzxvb found but phase is Pending instead of Bound. I1105 20:36:04.641] Nov 5 20:36:04.641: INFO: PersistentVolumeClaim pvc-fzxvb found but phase is Pending instead of Bound. I1105 20:36:06.720] Nov 5 20:36:06.719: INFO: PersistentVolumeClaim pvc-fzxvb found but phase is Pending instead of Bound. I1105 20:36:08.799] Nov 5 20:36:08.799: INFO: PersistentVolumeClaim pvc-fzxvb found but phase is Pending instead of Bound. I1105 20:36:10.878] Nov 5 20:36:10.878: INFO: PersistentVolumeClaim pvc-fzxvb found but phase is Pending instead of Bound. I1105 20:36:12.957] Nov 5 20:36:12.957: INFO: PersistentVolumeClaim pvc-fzxvb found but phase is Pending instead of Bound. I1105 20:36:15.037] Nov 5 20:36:15.037: INFO: PersistentVolumeClaim pvc-fzxvb found but phase is Pending instead of Bound. I1105 20:36:17.116] Nov 5 20:36:17.116: INFO: PersistentVolumeClaim pvc-fzxvb found but phase is Pending instead of Bound. I1105 20:36:19.196] Nov 5 20:36:19.196: INFO: PersistentVolumeClaim pvc-fzxvb found but phase is Pending instead of Bound. I1105 20:36:21.275] Nov 5 20:36:21.275: INFO: PersistentVolumeClaim pvc-fzxvb found but phase is Pending instead of Bound. I1105 20:36:23.355] Nov 5 20:36:23.355: INFO: PersistentVolumeClaim pvc-fzxvb found but phase is Pending instead of Bound. I1105 20:36:25.434] Nov 5 20:36:25.434: INFO: PersistentVolumeClaim pvc-fzxvb found but phase is Pending instead of Bound. I1105 20:36:27.513] Nov 5 20:36:27.513: INFO: PersistentVolumeClaim pvc-fzxvb found but phase is Pending instead of Bound. I1105 20:36:29.593] Nov 5 20:36:29.592: INFO: PersistentVolumeClaim pvc-fzxvb found but phase is Pending instead of Bound. I1105 20:36:31.675] Nov 5 20:36:31.674: INFO: PersistentVolumeClaim pvc-fzxvb found but phase is Pending instead of Bound. I1105 20:36:33.758] Nov 5 20:36:33.758: INFO: PersistentVolumeClaim pvc-fzxvb found but phase is Pending instead of Bound. I1105 20:36:35.838] Nov 5 20:36:35.837: INFO: PersistentVolumeClaim pvc-fzxvb found but phase is Pending instead of Bound. I1105 20:36:37.917] Nov 5 20:36:37.916: INFO: PersistentVolumeClaim pvc-fzxvb found but phase is Pending instead of Bound. I1105 20:36:39.997] Nov 5 20:36:39.996: INFO: PersistentVolumeClaim pvc-fzxvb found but phase is Pending instead of Bound. I1105 20:36:42.076] Nov 5 20:36:42.076: INFO: PersistentVolumeClaim pvc-fzxvb found but phase is Pending instead of Bound. I1105 20:36:44.155] Nov 5 20:36:44.155: INFO: PersistentVolumeClaim pvc-fzxvb found but phase is Pending instead of Bound. I1105 20:36:46.235] Nov 5 20:36:46.234: INFO: PersistentVolumeClaim pvc-fzxvb found but phase is Pending instead of Bound. I1105 20:36:48.316] Nov 5 20:36:48.316: INFO: PersistentVolumeClaim pvc-fzxvb found but phase is Pending instead of Bound. I1105 20:36:50.396] Nov 5 20:36:50.396: INFO: PersistentVolumeClaim pvc-fzxvb found but phase is Pending instead of Bound. I1105 20:36:52.476] Nov 5 20:36:52.475: INFO: PersistentVolumeClaim pvc-fzxvb found but phase is Pending instead of Bound. I1105 20:36:54.555] Nov 5 20:36:54.555: INFO: PersistentVolumeClaim pvc-fzxvb found but phase is Pending instead of Bound. I1105 20:36:56.634] Nov 5 20:36:56.634: INFO: PersistentVolumeClaim pvc-fzxvb found but phase is Pending instead of Bound. I1105 20:36:58.713] Nov 5 20:36:58.713: INFO: PersistentVolumeClaim pvc-fzxvb found but phase is Pending instead of Bound. I1105 20:37:00.792] Nov 5 20:37:00.792: INFO: PersistentVolumeClaim pvc-fzxvb found but phase is Pending instead of Bound. I1105 20:37:02.878] Nov 5 20:37:02.878: INFO: PersistentVolumeClaim pvc-fzxvb found but phase is Pending instead of Bound. I1105 20:37:04.957] Nov 5 20:37:04.957: INFO: PersistentVolumeClaim pvc-fzxvb found but phase is Pending instead of Bound. I1105 20:37:07.036] Nov 5 20:37:07.036: INFO: PersistentVolumeClaim pvc-fzxvb found but phase is Pending instead of Bound. I1105 20:37:09.121] Nov 5 20:37:09.121: INFO: PersistentVolumeClaim pvc-fzxvb found but phase is Pending instead of Bound. I1105 20:37:11.200] Nov 5 20:37:11.200: INFO: PersistentVolumeClaim pvc-fzxvb found but phase is Pending instead of Bound. I1105 20:37:13.278] Nov 5 20:37:13.278: INFO: PersistentVolumeClaim pvc-fzxvb found but phase is Pending instead of Bound. I1105 20:37:15.358] Nov 5 20:37:15.358: INFO: PersistentVolumeClaim pvc-fzxvb found but phase is Pending instead of Bound. I1105 20:37:17.438] Nov 5 20:37:17.438: INFO: PersistentVolumeClaim pvc-fzxvb found but phase is Pending instead of Bound. I1105 20:37:19.518] Nov 5 20:37:19.517: INFO: PersistentVolumeClaim pvc-fzxvb found but phase is Pending instead of Bound. I1105 20:37:21.597] Nov 5 20:37:21.596: INFO: PersistentVolumeClaim pvc-fzxvb found but phase is Pending instead of Bound. I1105 20:37:23.676] Nov 5 20:37:23.676: INFO: PersistentVolumeClaim pvc-fzxvb found but phase is Pending instead of Bound. I1105 20:37:25.763] Nov 5 20:37:25.763: INFO: PersistentVolumeClaim pvc-fzxvb found but phase is Pending instead of Bound. I1105 20:37:27.842] Nov 5 20:37:27.842: INFO: PersistentVolumeClaim pvc-fzxvb found but phase is Pending instead of Bound. I1105 20:37:29.922] Nov 5 20:37:29.922: INFO: PersistentVolumeClaim pvc-fzxvb found but phase is Pending instead of Bound. I1105 20:37:32.001] Nov 5 20:37:32.001: INFO: PersistentVolumeClaim pvc-fzxvb found but phase is Pending instead of Bound. I1105 20:37:34.081] Nov 5 20:37:34.080: INFO: PersistentVolumeClaim pvc-fzxvb found but phase is Pending instead of Bound. I1105 20:37:36.160] Nov 5 20:37:36.160: INFO: PersistentVolumeClaim pvc-fzxvb found but phase is Pending instead of Bound. I1105 20:37:38.240] Nov 5 20:37:38.239: INFO: PersistentVolumeClaim pvc-fzxvb found but phase is Pending instead of Bound. I1105 20:37:40.319] Nov 5 20:37:40.318: INFO: PersistentVolumeClaim pvc-fzxvb found but phase is Pending instead of Bound. I1105 20:37:42.398] Nov 5 20:37:42.398: INFO: PersistentVolumeClaim pvc-fzxvb found but phase is Pending instead of Bound. I1105 20:37:44.482] Nov 5 20:37:44.482: INFO: PersistentVolumeClaim pvc-fzxvb found but phase is Pending instead of Bound. I1105 20:37:46.562] Nov 5 20:37:46.562: INFO: PersistentVolumeClaim pvc-fzxvb found but phase is Pending instead of Bound. I1105 20:37:48.641] Nov 5 20:37:48.641: INFO: PersistentVolumeClaim pvc-fzxvb found but phase is Pending instead of Bound. I1105 20:37:50.720] Nov 5 20:37:50.720: INFO: PersistentVolumeClaim pvc-fzxvb found but phase is Pending instead of Bound. I1105 20:37:52.804] Nov 5 20:37:52.804: INFO: PersistentVolumeClaim pvc-fzxvb found but phase is Pending instead of Bound. I1105 20:37:54.883] Nov 5 20:37:54.883: INFO: PersistentVolumeClaim pvc-fzxvb found but phase is Pending instead of Bound. I1105 20:37:56.971] Nov 5 20:37:56.970: INFO: PersistentVolumeClaim pvc-fzxvb found but phase is Pending instead of Bound. I1105 20:37:59.050] Nov 5 20:37:59.050: INFO: PersistentVolumeClaim pvc-fzxvb found but phase is Pending instead of Bound. I1105 20:38:01.130] Nov 5 20:38:01.129: INFO: PersistentVolumeClaim pvc-fzxvb found but phase is Pending instead of Bound. I1105 20:38:03.209] Nov 5 20:38:03.209: INFO: PersistentVolumeClaim pvc-fzxvb found but phase is Pending instead of Bound. I1105 20:38:05.288] Nov 5 20:38:05.288: INFO: PersistentVolumeClaim pvc-fzxvb found but phase is Pending instead of Bound. I1105 20:38:07.370] Nov 5 20:38:07.369: INFO: PersistentVolumeClaim pvc-fzxvb found but phase is Pending instead of Bound. I1105 20:38:09.449] Nov 5 20:38:09.449: INFO: PersistentVolumeClaim pvc-fzxvb found but phase is Pending instead of Bound. I1105 20:38:11.528] Nov 5 20:38:11.528: INFO: PersistentVolumeClaim pvc-fzxvb found but phase is Pending instead of Bound. I1105 20:38:13.607] Nov 5 20:38:13.607: INFO: PersistentVolumeClaim pvc-fzxvb found but phase is Pending instead of Bound. I1105 20:38:15.686] Nov 5 20:38:15.686: INFO: PersistentVolumeClaim pvc-fzxvb found but phase is Pending instead of Bound. I1105 20:38:17.687] Nov 5 20:38:17.686: INFO: Unexpected error occurred: PersistentVolumeClaims [pvc-fzxvb] not all in phase Bound within 5m0s I1105 20:38:17.687] [AfterEach] [single-az] I1105 20:38:17.687] /go/src/sigs.k8s.io/azuredisk-csi-driver/vendor/k8s.io/kubernetes/test/e2e/framework/framework.go:151 I1105 20:38:17.687] STEP: Collecting events from namespace "azuredisk-5245". I1105 20:38:17.766] STEP: Found 3 events. I1105 20:38:17.767] Nov 5 20:38:17.766: INFO: At 2019-11-05 20:33:15 +0000 UTC - event for pvc-fzxvb: {disk.csi.azure.com_csi-azuredisk-controller-6f6587bfc6-62nrt_3930bc69-0009-11ea-8cf5-829a27ff7d5a } Provisioning: External provisioner is provisioning volume for claim "azuredisk-5245/pvc-fzxvb" I1105 20:38:17.767] Nov 5 20:38:17.766: INFO: At 2019-11-05 20:33:16 +0000 UTC - event for pvc-fzxvb: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "disk.csi.azure.com" or manually created by system administrator I1105 20:38:17.767] Nov 5 20:38:17.766: INFO: At 2019-11-05 20:33:16 +0000 UTC - event for pvc-fzxvb: {disk.csi.azure.com_csi-azuredisk-controller-6f6587bfc6-62nrt_3930bc69-0009-11ea-8cf5-829a27ff7d5a } ProvisioningFailed: failed to provision volume with StorageClass "azuredisk-5245-disk.csi.azure.com-dynamic-sc-ps5s2": rpc error: code = Unknown desc = compute.DisksClient#CreateOrUpdate: Failure sending request: StatusCode=400 -- Original Error: Code="InvalidAvailabilityZone" Message="The zone(s) '' for resource 'Microsoft.Compute/disks/pvc-becb7110-0575-4408-878d-f00430c8097b' is not supported. The supported zones for location 'eastus2' are '1,2,3'" I1105 20:38:17.845] Nov 5 20:38:17.845: INFO: POD NODE PHASE GRACE CONDITIONS I1105 20:38:17.845] Nov 5 20:38:17.845: INFO: I1105 20:38:18.081] Nov 5 20:38:18.081: INFO: I1105 20:38:18.081] Logging node info for node k8s-agentpool-97435445-vmss000000 I1105 20:38:18.163] Nov 5 20:38:18.160: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-agentpool-97435445-vmss000000,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-agentpool-97435445-vmss000000,UID:e955dc47-8c8f-4142-9241-3d409f32e42d,ResourceVersion:6331,Generation:0,CreationTimestamp:2019-11-05 20:07:34 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{agentpool: agentpool,beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_D4s_v3,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 0,kubernetes.azure.com/cluster: kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b,kubernetes.azure.com/role: agent,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-agentpool-97435445-vmss000000,kubernetes.io/os: linux,kubernetes.io/role: agent,node-role.kubernetes.io/agent: ,storageprofile: managed,storagetier: Premium_LRS,topology.disk.csi.azure.com/zone: 0,},Annotations:map[string]string{csi.volume.kubernetes.io/nodeid: {"disk.csi.azure.com":"k8s-agentpool-97435445-vmss000000"},node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.2.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool-97435445-vmss/virtualMachines/0,Unschedulable:false,Taints:[],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-csi-disk.csi.azure.com: {{8 0} {} 8 DecimalSI},cpu: {{4 0} {} 4 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{16797040640 0} {} BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-csi-disk.csi.azure.com: {{8 0} {} 8 DecimalSI},cpu: {{4 0} {} 4 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{16010608640 0} {} BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 20:08:01 +0000 UTC 2019-11-05 20:08:01 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 20:37:40 +0000 UTC 2019-11-05 20:07:27 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 20:37:40 +0000 UTC 2019-11-05 20:07:27 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 20:37:40 +0000 UTC 2019-11-05 20:07:27 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 20:37:40 +0000 UTC 2019-11-05 20:07:55 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.0.4} {Hostname k8s-agentpool-97435445-vmss000000}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:bfb55325809340228149d2d93f50df58,SystemUUID:CB2ED036-0439-F641-BC7F-640C3638C027,BootID:a917f4d1-549d-471d-b7dc-47ae1715d97d,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[k8sprowinternal.azurecr.io/azuredisk-csi@sha256:75d4ff5d7cd845d6e810c67796ecb037e852fb849ec1eb5b3194564008e6d1e0 k8sprowinternal.azurecr.io/azuredisk-csi:e2e-3e3ed045c6faa509dbd8d2890ee9f56aa2934887] 286930414} {[k8s.gcr.io/kubernetes-dashboard-amd64@sha256:0ae6b69432e78069c5ce2bcde0fe409c5c4d6f0f4d9cd50a17974fea38898747 k8s.gcr.io/kubernetes-dashboard-amd64:v1.10.1] 121711221} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/metrics-server-amd64@sha256:49a9f12f7067d11f42c803dbe61ed2c1299959ad85cb315b25ff7eef8e6b8892 k8s.gcr.io/metrics-server-amd64:v0.2.1] 42541759} {[k8s.gcr.io/coredns@sha256:e83beb5e43f8513fa735e77ffc5859640baea30a882a11cc75c4c3244a737d3c k8s.gcr.io/coredns:1.5.0] 42488424} {[mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume@sha256:4fd30d43947d4a54fc89ead7985beecfd3c9b2a93a0655a373b1608ab90bd5af mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume:v0.0.7] 22909487} {[quay.io/k8scsi/livenessprobe@sha256:8c9c8a0f330acc2e558ae6624309447d02522e76aa35af23fa19b88abf4381cb quay.io/k8scsi/livenessprobe:v1.0.2] 17415381} {[quay.io/k8scsi/csi-node-driver-registrar@sha256:13daf82fb99e951a4bff8ae5fc7c17c3a8fe7130be6400990d8f6076c32d4599 quay.io/k8scsi/csi-node-driver-registrar:v1.1.0] 15815995} {[mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume@sha256:23d8c6033f02a1ecad05127ebdc931bb871264228661bc122704b0974e4d9fdd mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume:1.0.8] 1159025} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 20:38:18.163] Nov 5 20:38:18.160: INFO: I1105 20:38:18.164] Logging kubelet events for node k8s-agentpool-97435445-vmss000000 I1105 20:38:18.243] Nov 5 20:38:18.243: INFO: I1105 20:38:18.244] Logging pods the kubelet thinks is on node k8s-agentpool-97435445-vmss000000 I1105 20:38:18.328] Nov 5 20:38:18.328: INFO: coredns-7f68dcdbdb-zh8g5 started at 2019-11-05 20:08:01 +0000 UTC (0+1 container statuses recorded) I1105 20:38:18.328] Nov 5 20:38:18.328: INFO: Container coredns ready: true, restart count 0 I1105 20:38:18.329] Nov 5 20:38:18.328: INFO: azure-ip-masq-agent-m7tv9 started at 2019-11-05 20:08:01 +0000 UTC (0+1 container statuses recorded) I1105 20:38:18.329] Nov 5 20:38:18.328: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 20:38:18.329] Nov 5 20:38:18.328: INFO: keyvault-flexvolume-smspt started at 2019-11-05 20:08:01 +0000 UTC (0+1 container statuses recorded) I1105 20:38:18.329] Nov 5 20:38:18.328: INFO: Container keyvault-flexvolume ready: true, restart count 0 I1105 20:38:18.329] Nov 5 20:38:18.328: INFO: metrics-server-864ffbc5c-wl84z started at 2019-11-05 20:08:01 +0000 UTC (0+1 container statuses recorded) I1105 20:38:18.330] Nov 5 20:38:18.328: INFO: Container metrics-server ready: true, restart count 0 I1105 20:38:18.330] Nov 5 20:38:18.328: INFO: kube-proxy-8mt7f started at 2019-11-05 20:08:01 +0000 UTC (0+1 container statuses recorded) I1105 20:38:18.330] Nov 5 20:38:18.328: INFO: Container kube-proxy ready: true, restart count 0 I1105 20:38:18.330] Nov 5 20:38:18.328: INFO: kubernetes-dashboard-66dd8b8df7-t6j4l started at 2019-11-05 20:08:01 +0000 UTC (0+1 container statuses recorded) I1105 20:38:18.330] Nov 5 20:38:18.328: INFO: Container kubernetes-dashboard ready: true, restart count 0 I1105 20:38:18.331] Nov 5 20:38:18.328: INFO: csi-azuredisk-node-5wwhz started at 2019-11-05 20:16:09 +0000 UTC (0+3 container statuses recorded) I1105 20:38:18.331] Nov 5 20:38:18.328: INFO: Container azuredisk ready: true, restart count 0 I1105 20:38:18.331] Nov 5 20:38:18.328: INFO: Container liveness-probe ready: true, restart count 0 I1105 20:38:18.331] Nov 5 20:38:18.328: INFO: Container node-driver-registrar ready: true, restart count 0 I1105 20:38:18.331] Nov 5 20:38:18.328: INFO: blobfuse-flexvol-installer-p7hdb started at 2019-11-05 20:08:01 +0000 UTC (0+1 container statuses recorded) I1105 20:38:18.331] Nov 5 20:38:18.328: INFO: Container blobfuse-flexvol-installer ready: true, restart count 0 I1105 20:38:18.590] Nov 5 20:38:18.590: INFO: I1105 20:38:18.590] Latency metrics for node k8s-agentpool-97435445-vmss000000 I1105 20:38:18.590] Nov 5 20:38:18.590: INFO: I1105 20:38:18.590] Logging node info for node k8s-agentpool-97435445-vmss000001 I1105 20:38:18.673] Nov 5 20:38:18.669: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-agentpool-97435445-vmss000001,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-agentpool-97435445-vmss000001,UID:75d3cc50-ca6d-45e0-a42a-6090e8d537dd,ResourceVersion:6330,Generation:0,CreationTimestamp:2019-11-05 20:07:34 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{agentpool: agentpool,beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_D4s_v3,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 1,kubernetes.azure.com/cluster: kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b,kubernetes.azure.com/role: agent,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-agentpool-97435445-vmss000001,kubernetes.io/os: linux,kubernetes.io/role: agent,node-role.kubernetes.io/agent: ,storageprofile: managed,storagetier: Premium_LRS,topology.disk.csi.azure.com/zone: 1,},Annotations:map[string]string{csi.volume.kubernetes.io/nodeid: {"disk.csi.azure.com":"k8s-agentpool-97435445-vmss000001"},node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.0.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool-97435445-vmss/virtualMachines/1,Unschedulable:false,Taints:[],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-csi-disk.csi.azure.com: {{8 0} {} 8 DecimalSI},cpu: {{4 0} {} 4 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{16797044736 0} {} 16403364Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-csi-disk.csi.azure.com: {{8 0} {} 8 DecimalSI},cpu: {{4 0} {} 4 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{16010612736 0} {} 15635364Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 20:08:25 +0000 UTC 2019-11-05 20:08:25 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 20:37:39 +0000 UTC 2019-11-05 20:07:26 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 20:37:39 +0000 UTC 2019-11-05 20:07:26 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 20:37:39 +0000 UTC 2019-11-05 20:07:26 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 20:37:39 +0000 UTC 2019-11-05 20:07:55 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.0.5} {Hostname k8s-agentpool-97435445-vmss000001}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:e78def54b2f545c6b94d88bbba523c60,SystemUUID:40AE8188-86AB-D848-84A4-D9254E1258BC,BootID:8587ec06-f4f4-43c1-86cd-06c295b05b63,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[k8sprowinternal.azurecr.io/azuredisk-csi@sha256:75d4ff5d7cd845d6e810c67796ecb037e852fb849ec1eb5b3194564008e6d1e0 k8sprowinternal.azurecr.io/azuredisk-csi:e2e-3e3ed045c6faa509dbd8d2890ee9f56aa2934887] 286930414} {[gcr.io/kubernetes-helm/tiller@sha256:f6d8f4ab9ba993b5f5b60a6edafe86352eabe474ffeb84cb6c79b8866dce45d1 gcr.io/kubernetes-helm/tiller:v2.11.0] 71821984} {[quay.io/k8scsi/csi-provisioner@sha256:d657c839dce87324fe2b677302913f9386f885f8746be7bea0ced5b0844e3433 quay.io/k8scsi/csi-provisioner:v1.3.1] 50596998} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[quay.io/k8scsi/csi-attacher@sha256:26fccd7a99d973845df1193b46ebdcc6ab8dc5f6e6be319750c471fce1742d13 quay.io/k8scsi/csi-attacher:v1.2.0] 46226754} {[quay.io/k8scsi/csi-cluster-driver-registrar@sha256:fafd75ae5442f192cfa8c2e792903aee30d5884b62e802e4464b0a895d21e3ef quay.io/k8scsi/csi-cluster-driver-registrar:v1.0.1] 45874691} {[quay.io/k8scsi/csi-snapshotter@sha256:a49e0da1af6f2bf717e41ba1eee8b5e6a1cbd66a709dd92cc43fe475fe2589eb quay.io/k8scsi/csi-snapshotter:v1.1.0] 44325352} {[mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume@sha256:4fd30d43947d4a54fc89ead7985beecfd3c9b2a93a0655a373b1608ab90bd5af mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume:v0.0.7] 22909487} {[quay.io/k8scsi/livenessprobe@sha256:8c9c8a0f330acc2e558ae6624309447d02522e76aa35af23fa19b88abf4381cb quay.io/k8scsi/livenessprobe:v1.0.2] 17415381} {[quay.io/k8scsi/csi-node-driver-registrar@sha256:13daf82fb99e951a4bff8ae5fc7c17c3a8fe7130be6400990d8f6076c32d4599 quay.io/k8scsi/csi-node-driver-registrar:v1.1.0] 15815995} {[quay.io/k8scsi/livenessprobe@sha256:dde617756e0f602adc566ab71fd885f1dad451ad3fb063ac991c95a2ff47aea5 quay.io/k8scsi/livenessprobe:v1.1.0] 14967303} {[mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume@sha256:23d8c6033f02a1ecad05127ebdc931bb871264228661bc122704b0974e4d9fdd mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume:1.0.8] 1159025} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 20:38:18.673] Nov 5 20:38:18.669: INFO: I1105 20:38:18.673] Logging kubelet events for node k8s-agentpool-97435445-vmss000001 I1105 20:38:18.754] Nov 5 20:38:18.754: INFO: I1105 20:38:18.754] Logging pods the kubelet thinks is on node k8s-agentpool-97435445-vmss000001 I1105 20:38:18.838] Nov 5 20:38:18.838: INFO: keyvault-flexvolume-6bffx started at 2019-11-05 20:07:59 +0000 UTC (0+1 container statuses recorded) I1105 20:38:18.839] Nov 5 20:38:18.838: INFO: Container keyvault-flexvolume ready: true, restart count 0 I1105 20:38:18.839] Nov 5 20:38:18.838: INFO: kube-proxy-kjrpb started at 2019-11-05 20:08:00 +0000 UTC (0+1 container statuses recorded) I1105 20:38:18.839] Nov 5 20:38:18.838: INFO: Container kube-proxy ready: true, restart count 0 I1105 20:38:18.839] Nov 5 20:38:18.838: INFO: tiller-deploy-54c96cb5df-pt7x6 started at 2019-11-05 20:08:00 +0000 UTC (0+1 container statuses recorded) I1105 20:38:18.839] Nov 5 20:38:18.838: INFO: Container tiller ready: true, restart count 0 I1105 20:38:18.839] Nov 5 20:38:18.838: INFO: csi-azuredisk-node-q2dmt started at 2019-11-05 20:16:09 +0000 UTC (0+3 container statuses recorded) I1105 20:38:18.840] Nov 5 20:38:18.838: INFO: Container azuredisk ready: true, restart count 0 I1105 20:38:18.840] Nov 5 20:38:18.838: INFO: Container liveness-probe ready: true, restart count 0 I1105 20:38:18.840] Nov 5 20:38:18.838: INFO: Container node-driver-registrar ready: true, restart count 0 I1105 20:38:18.840] Nov 5 20:38:18.838: INFO: csi-azuredisk-controller-6f6587bfc6-62nrt started at 2019-11-05 20:16:09 +0000 UTC (0+6 container statuses recorded) I1105 20:38:18.840] Nov 5 20:38:18.838: INFO: Container azuredisk ready: true, restart count 0 I1105 20:38:18.840] Nov 5 20:38:18.838: INFO: Container cluster-driver-registrar ready: true, restart count 0 I1105 20:38:18.840] Nov 5 20:38:18.838: INFO: Container csi-attacher ready: true, restart count 0 I1105 20:38:18.841] Nov 5 20:38:18.838: INFO: Container csi-provisioner ready: true, restart count 0 I1105 20:38:18.841] Nov 5 20:38:18.838: INFO: Container csi-snapshotter ready: true, restart count 0 I1105 20:38:18.841] Nov 5 20:38:18.838: INFO: Container liveness-probe ready: true, restart count 0 I1105 20:38:18.841] Nov 5 20:38:18.838: INFO: blobfuse-flexvol-installer-j4s7t started at 2019-11-05 20:07:59 +0000 UTC (0+1 container statuses recorded) I1105 20:38:18.841] Nov 5 20:38:18.838: INFO: Container blobfuse-flexvol-installer ready: true, restart count 0 I1105 20:38:18.841] Nov 5 20:38:18.838: INFO: azure-ip-masq-agent-gdhcq started at 2019-11-05 20:07:59 +0000 UTC (0+1 container statuses recorded) I1105 20:38:18.842] Nov 5 20:38:18.838: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 20:38:19.098] Nov 5 20:38:19.097: INFO: I1105 20:38:19.098] Latency metrics for node k8s-agentpool-97435445-vmss000001 I1105 20:38:19.098] Nov 5 20:38:19.097: INFO: I1105 20:38:19.098] Logging node info for node k8s-master-97435445-0 I1105 20:38:19.179] Nov 5 20:38:19.177: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-97435445-0,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-97435445-0,UID:df3ae36f-961c-4729-a8f5-34cda54f4ff9,ResourceVersion:6391,Generation:0,CreationTimestamp:2019-11-05 20:08:36 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 2,kubernetes.azure.com/cluster: kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-97435445-0,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.4.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b/providers/Microsoft.Compute/virtualMachines/k8s-master-97435445-0,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 20:08:46 +0000 UTC 2019-11-05 20:08:46 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 20:38:00 +0000 UTC 2019-11-05 20:08:28 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 20:38:00 +0000 UTC 2019-11-05 20:08:28 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 20:38:00 +0000 UTC 2019-11-05 20:08:28 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 20:38:00 +0000 UTC 2019-11-05 20:08:36 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.255.5} {Hostname k8s-master-97435445-0}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:3c381865192341c2868672e8626cef7d,SystemUUID:F3D259D3-7EB8-B844-A1F0-F954DCD0D76B,BootID:4948329e-cf57-48c3-aa83-5297bef70370,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 20:38:19.179] Nov 5 20:38:19.177: INFO: I1105 20:38:19.179] Logging kubelet events for node k8s-master-97435445-0 I1105 20:38:19.261] Nov 5 20:38:19.261: INFO: I1105 20:38:19.261] Logging pods the kubelet thinks is on node k8s-master-97435445-0 I1105 20:38:19.346] Nov 5 20:38:19.346: INFO: cloud-controller-manager-k8s-master-97435445-0 started at 2019-11-05 20:08:22 +0000 UTC (0+1 container statuses recorded) I1105 20:38:19.346] Nov 5 20:38:19.346: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 20:38:19.346] Nov 5 20:38:19.346: INFO: kube-addon-manager-k8s-master-97435445-0 started at 2019-11-05 20:08:48 +0000 UTC (0+1 container statuses recorded) I1105 20:38:19.346] Nov 5 20:38:19.346: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 20:38:19.346] Nov 5 20:38:19.346: INFO: kube-apiserver-k8s-master-97435445-0 started at 2019-11-05 20:08:22 +0000 UTC (0+1 container statuses recorded) I1105 20:38:19.347] Nov 5 20:38:19.346: INFO: Container kube-apiserver ready: true, restart count 0 I1105 20:38:19.347] Nov 5 20:38:19.346: INFO: kube-controller-manager-k8s-master-97435445-0 started at 2019-11-05 20:08:22 +0000 UTC (0+1 container statuses recorded) I1105 20:38:19.347] Nov 5 20:38:19.346: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 20:38:19.347] Nov 5 20:38:19.346: INFO: kube-scheduler-k8s-master-97435445-0 started at 2019-11-05 20:08:22 +0000 UTC (0+1 container statuses recorded) I1105 20:38:19.347] Nov 5 20:38:19.346: INFO: Container kube-scheduler ready: true, restart count 0 I1105 20:38:19.347] Nov 5 20:38:19.346: INFO: azure-ip-masq-agent-q6dcl started at 2019-11-05 20:08:38 +0000 UTC (0+1 container statuses recorded) I1105 20:38:19.347] Nov 5 20:38:19.346: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 20:38:19.348] Nov 5 20:38:19.346: INFO: kube-proxy-5wnd2 started at 2019-11-05 20:08:38 +0000 UTC (0+1 container statuses recorded) I1105 20:38:19.348] Nov 5 20:38:19.346: INFO: Container kube-proxy ready: true, restart count 0 I1105 20:38:19.597] Nov 5 20:38:19.597: INFO: I1105 20:38:19.597] Latency metrics for node k8s-master-97435445-0 I1105 20:38:19.597] Nov 5 20:38:19.597: INFO: I1105 20:38:19.598] Logging node info for node k8s-master-97435445-1 I1105 20:38:19.678] Nov 5 20:38:19.676: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-97435445-1,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-97435445-1,UID:05534a67-d493-4ab5-9b0f-5ff5490e234f,ResourceVersion:6294,Generation:0,CreationTimestamp:2019-11-05 20:07:34 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 0,kubernetes.azure.com/cluster: kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-97435445-1,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.1.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b/providers/Microsoft.Compute/virtualMachines/k8s-master-97435445-1,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 20:08:26 +0000 UTC 2019-11-05 20:08:26 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 20:37:28 +0000 UTC 2019-11-05 20:07:28 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 20:37:28 +0000 UTC 2019-11-05 20:07:28 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 20:37:28 +0000 UTC 2019-11-05 20:07:28 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 20:37:28 +0000 UTC 2019-11-05 20:07:55 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{Hostname k8s-master-97435445-1} {InternalIP 10.240.255.6}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:079b605103b74acabac18fd2ea63368a,SystemUUID:38220EEC-BEFF-4C46-AA05-5A214AAFDC28,BootID:b1b45df0-d925-4bcc-9830-0202df36ebb4,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 20:38:19.678] Nov 5 20:38:19.676: INFO: I1105 20:38:19.678] Logging kubelet events for node k8s-master-97435445-1 I1105 20:38:19.759] Nov 5 20:38:19.759: INFO: I1105 20:38:19.759] Logging pods the kubelet thinks is on node k8s-master-97435445-1 I1105 20:38:19.843] Nov 5 20:38:19.843: INFO: kube-addon-manager-k8s-master-97435445-1 started at 2019-11-05 20:07:22 +0000 UTC (0+1 container statuses recorded) I1105 20:38:19.844] Nov 5 20:38:19.843: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 20:38:19.844] Nov 5 20:38:19.843: INFO: kube-apiserver-k8s-master-97435445-1 started at 2019-11-05 20:07:22 +0000 UTC (0+1 container statuses recorded) I1105 20:38:19.844] Nov 5 20:38:19.843: INFO: Container kube-apiserver ready: true, restart count 0 I1105 20:38:19.844] Nov 5 20:38:19.843: INFO: kube-controller-manager-k8s-master-97435445-1 started at 2019-11-05 20:07:22 +0000 UTC (0+1 container statuses recorded) I1105 20:38:19.844] Nov 5 20:38:19.843: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 20:38:19.844] Nov 5 20:38:19.843: INFO: kube-scheduler-k8s-master-97435445-1 started at 2019-11-05 20:07:22 +0000 UTC (0+1 container statuses recorded) I1105 20:38:19.844] Nov 5 20:38:19.843: INFO: Container kube-scheduler ready: true, restart count 0 I1105 20:38:19.845] Nov 5 20:38:19.843: INFO: azure-ip-masq-agent-n29bm started at 2019-11-05 20:07:59 +0000 UTC (0+1 container statuses recorded) I1105 20:38:19.845] Nov 5 20:38:19.843: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 20:38:19.845] Nov 5 20:38:19.843: INFO: kube-proxy-h4s2c started at 2019-11-05 20:08:00 +0000 UTC (0+1 container statuses recorded) I1105 20:38:19.845] Nov 5 20:38:19.843: INFO: Container kube-proxy ready: true, restart count 0 I1105 20:38:19.845] Nov 5 20:38:19.843: INFO: cloud-controller-manager-k8s-master-97435445-1 started at 2019-11-05 20:07:22 +0000 UTC (0+1 container statuses recorded) I1105 20:38:19.845] Nov 5 20:38:19.843: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 20:38:20.098] Nov 5 20:38:20.098: INFO: I1105 20:38:20.098] Latency metrics for node k8s-master-97435445-1 I1105 20:38:20.098] Nov 5 20:38:20.098: INFO: I1105 20:38:20.098] Logging node info for node k8s-master-97435445-2 I1105 20:38:20.183] Nov 5 20:38:20.180: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-97435445-2,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-97435445-2,UID:fa55c895-1a93-4743-885e-03fce84b5332,ResourceVersion:6344,Generation:0,CreationTimestamp:2019-11-05 20:09:19 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 0,kubernetes.azure.com/cluster: kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-97435445-2,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.5.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b/providers/Microsoft.Compute/virtualMachines/k8s-master-97435445-2,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 20:09:19 +0000 UTC 2019-11-05 20:09:19 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 20:37:44 +0000 UTC 2019-11-05 20:09:12 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 20:37:44 +0000 UTC 2019-11-05 20:09:12 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 20:37:44 +0000 UTC 2019-11-05 20:09:12 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 20:37:44 +0000 UTC 2019-11-05 20:09:19 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.255.7} {Hostname k8s-master-97435445-2}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:d19f6709138f496da43c095cec7b2360,SystemUUID:10116777-A8D7-CB4A-A8B7-7EBCF2F23B62,BootID:50cd14a9-5497-4972-bf60-cf59a0cbcb5f,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 20:38:20.183] Nov 5 20:38:20.180: INFO: I1105 20:38:20.183] Logging kubelet events for node k8s-master-97435445-2 I1105 20:38:20.263] Nov 5 20:38:20.263: INFO: I1105 20:38:20.264] Logging pods the kubelet thinks is on node k8s-master-97435445-2 I1105 20:38:20.346] Nov 5 20:38:20.346: INFO: cloud-controller-manager-k8s-master-97435445-2 started at 2019-11-05 20:09:05 +0000 UTC (0+1 container statuses recorded) I1105 20:38:20.346] Nov 5 20:38:20.346: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 20:38:20.346] Nov 5 20:38:20.346: INFO: kube-addon-manager-k8s-master-97435445-2 started at 2019-11-05 20:09:05 +0000 UTC (0+1 container statuses recorded) I1105 20:38:20.347] Nov 5 20:38:20.346: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 20:38:20.347] Nov 5 20:38:20.346: INFO: azure-ip-masq-agent-sq7p5 started at 2019-11-05 20:09:19 +0000 UTC (0+1 container statuses recorded) I1105 20:38:20.347] Nov 5 20:38:20.346: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 20:38:20.347] Nov 5 20:38:20.346: INFO: kube-proxy-pz426 started at 2019-11-05 20:09:19 +0000 UTC (0+1 container statuses recorded) I1105 20:38:20.347] Nov 5 20:38:20.346: INFO: Container kube-proxy ready: true, restart count 0 I1105 20:38:20.347] Nov 5 20:38:20.346: INFO: kube-apiserver-k8s-master-97435445-2 started at 2019-11-05 20:09:05 +0000 UTC (0+1 container statuses recorded) I1105 20:38:20.348] Nov 5 20:38:20.346: INFO: Container kube-apiserver ready: true, restart count 0 I1105 20:38:20.348] Nov 5 20:38:20.346: INFO: kube-controller-manager-k8s-master-97435445-2 started at 2019-11-05 20:09:05 +0000 UTC (0+1 container statuses recorded) I1105 20:38:20.348] Nov 5 20:38:20.346: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 20:38:20.348] Nov 5 20:38:20.346: INFO: kube-scheduler-k8s-master-97435445-2 started at 2019-11-05 20:09:05 +0000 UTC (0+1 container statuses recorded) I1105 20:38:20.348] Nov 5 20:38:20.346: INFO: Container kube-scheduler ready: true, restart count 0 I1105 20:38:20.598] Nov 5 20:38:20.597: INFO: I1105 20:38:20.598] Latency metrics for node k8s-master-97435445-2 I1105 20:38:20.598] Nov 5 20:38:20.597: INFO: I1105 20:38:20.598] Logging node info for node k8s-master-97435445-3 I1105 20:38:20.678] Nov 5 20:38:20.676: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-97435445-3,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-97435445-3,UID:8ffd1d0d-4487-4680-a2f6-e724387be637,ResourceVersion:6348,Generation:0,CreationTimestamp:2019-11-05 20:09:21 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 1,kubernetes.azure.com/cluster: kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-97435445-3,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.6.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b/providers/Microsoft.Compute/virtualMachines/k8s-master-97435445-3,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 20:09:39 +0000 UTC 2019-11-05 20:09:39 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 20:37:45 +0000 UTC 2019-11-05 20:09:18 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 20:37:45 +0000 UTC 2019-11-05 20:09:18 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 20:37:45 +0000 UTC 2019-11-05 20:09:18 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 20:37:45 +0000 UTC 2019-11-05 20:09:21 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.255.8} {Hostname k8s-master-97435445-3}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:266064173bfb4297898c6c25769e8b20,SystemUUID:E6E44241-5015-7740-9A81-A668A8427206,BootID:579d658c-b0ad-47f1-a993-1ccea2b55789,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 20:38:20.679] Nov 5 20:38:20.676: INFO: I1105 20:38:20.679] Logging kubelet events for node k8s-master-97435445-3 I1105 20:38:20.761] Nov 5 20:38:20.760: INFO: I1105 20:38:20.761] Logging pods the kubelet thinks is on node k8s-master-97435445-3 I1105 20:38:20.845] Nov 5 20:38:20.845: INFO: cloud-controller-manager-k8s-master-97435445-3 started at 2019-11-05 20:09:04 +0000 UTC (0+1 container statuses recorded) I1105 20:38:20.845] Nov 5 20:38:20.845: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 20:38:20.846] Nov 5 20:38:20.845: INFO: kube-addon-manager-k8s-master-97435445-3 started at 2019-11-05 20:09:05 +0000 UTC (0+1 container statuses recorded) I1105 20:38:20.846] Nov 5 20:38:20.845: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 20:38:20.846] Nov 5 20:38:20.845: INFO: kube-apiserver-k8s-master-97435445-3 started at 2019-11-05 20:09:05 +0000 UTC (0+1 container statuses recorded) I1105 20:38:20.846] Nov 5 20:38:20.845: INFO: Container kube-apiserver ready: true, restart count 0 I1105 20:38:20.846] Nov 5 20:38:20.845: INFO: kube-controller-manager-k8s-master-97435445-3 started at 2019-11-05 20:09:05 +0000 UTC (0+1 container statuses recorded) I1105 20:38:20.846] Nov 5 20:38:20.845: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 20:38:20.847] Nov 5 20:38:20.845: INFO: kube-scheduler-k8s-master-97435445-3 started at 2019-11-05 20:09:05 +0000 UTC (0+1 container statuses recorded) I1105 20:38:20.847] Nov 5 20:38:20.845: INFO: Container kube-scheduler ready: true, restart count 0 I1105 20:38:20.847] Nov 5 20:38:20.845: INFO: azure-ip-masq-agent-747cl started at 2019-11-05 20:09:21 +0000 UTC (0+1 container statuses recorded) I1105 20:38:20.847] Nov 5 20:38:20.845: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 20:38:20.847] Nov 5 20:38:20.845: INFO: kube-proxy-k45g8 started at 2019-11-05 20:09:21 +0000 UTC (0+1 container statuses recorded) I1105 20:38:20.847] Nov 5 20:38:20.845: INFO: Container kube-proxy ready: true, restart count 0 I1105 20:38:21.099] Nov 5 20:38:21.099: INFO: I1105 20:38:21.100] Latency metrics for node k8s-master-97435445-3 I1105 20:38:21.100] Nov 5 20:38:21.099: INFO: I1105 20:38:21.100] Logging node info for node k8s-master-97435445-4 I1105 20:38:21.180] Nov 5 20:38:21.178: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-97435445-4,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-97435445-4,UID:6d547b41-3f92-4cc4-a2d7-dda2071776a2,ResourceVersion:6390,Generation:0,CreationTimestamp:2019-11-05 20:08:36 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 1,kubernetes.azure.com/cluster: kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-97435445-4,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.3.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b/providers/Microsoft.Compute/virtualMachines/k8s-master-97435445-4,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 20:08:59 +0000 UTC 2019-11-05 20:08:59 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 20:38:00 +0000 UTC 2019-11-05 20:08:29 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 20:38:00 +0000 UTC 2019-11-05 20:08:29 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 20:38:00 +0000 UTC 2019-11-05 20:08:29 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 20:38:00 +0000 UTC 2019-11-05 20:08:36 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.255.9} {Hostname k8s-master-97435445-4}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:56b393c61eb34ba49ebda5279eeedb30,SystemUUID:FD239C23-618C-0049-8734-9AF3FDC9E3C1,BootID:f342ea44-e719-4513-807d-5def7ac606e2,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 20:38:21.180] Nov 5 20:38:21.178: INFO: I1105 20:38:21.180] Logging kubelet events for node k8s-master-97435445-4 I1105 20:38:21.264] Nov 5 20:38:21.264: INFO: I1105 20:38:21.264] Logging pods the kubelet thinks is on node k8s-master-97435445-4 I1105 20:38:21.348] Nov 5 20:38:21.348: INFO: azure-ip-masq-agent-kmkll started at 2019-11-05 20:08:36 +0000 UTC (0+1 container statuses recorded) I1105 20:38:21.348] Nov 5 20:38:21.348: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 20:38:21.348] Nov 5 20:38:21.348: INFO: cloud-controller-manager-k8s-master-97435445-4 started at 2019-11-05 20:08:46 +0000 UTC (0+1 container statuses recorded) I1105 20:38:21.348] Nov 5 20:38:21.348: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 20:38:21.349] Nov 5 20:38:21.348: INFO: kube-addon-manager-k8s-master-97435445-4 started at 2019-11-05 20:08:23 +0000 UTC (0+1 container statuses recorded) I1105 20:38:21.349] Nov 5 20:38:21.348: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 20:38:21.349] Nov 5 20:38:21.348: INFO: kube-apiserver-k8s-master-97435445-4 started at 2019-11-05 20:08:23 +0000 UTC (0+1 container statuses recorded) I1105 20:38:21.349] Nov 5 20:38:21.348: INFO: Container kube-apiserver ready: true, restart count 0 I1105 20:38:21.349] Nov 5 20:38:21.348: INFO: kube-controller-manager-k8s-master-97435445-4 started at 2019-11-05 20:08:23 +0000 UTC (0+1 container statuses recorded) I1105 20:38:21.350] Nov 5 20:38:21.348: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 20:38:21.350] Nov 5 20:38:21.348: INFO: kube-scheduler-k8s-master-97435445-4 started at 2019-11-05 20:08:23 +0000 UTC (0+1 container statuses recorded) I1105 20:38:21.350] Nov 5 20:38:21.348: INFO: Container kube-scheduler ready: true, restart count 0 I1105 20:38:21.350] Nov 5 20:38:21.348: INFO: kube-proxy-vfbfr started at 2019-11-05 20:08:36 +0000 UTC (0+1 container statuses recorded) I1105 20:38:21.350] Nov 5 20:38:21.348: INFO: Container kube-proxy ready: true, restart count 0 I1105 20:38:21.603] Nov 5 20:38:21.603: INFO: I1105 20:38:21.603] Latency metrics for node k8s-master-97435445-4 I1105 20:38:21.603] Nov 5 20:38:21.603: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready I1105 20:38:21.687] STEP: Destroying namespace "azuredisk-5245" for this suite. I1105 20:38:35.934] Nov 5 20:38:35.933: INFO: Waiting up to 30s for server preferred namespaced resources to be successfully discovered I1105 20:38:38.940] Nov 5 20:38:38.940: INFO: namespace azuredisk-5245 deletion completed in 17.253648998s I1105 20:38:38.941] I1105 20:38:38.941] • Failure [323.666 seconds] I1105 20:38:38.941] [azuredisk-csi-e2e] Dynamic Provisioning I1105 20:38:38.941] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:33 I1105 20:38:38.941] [single-az] I1105 20:38:38.942] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:36 I1105 20:38:38.942] should create multiple PV objects, bind to PVCs and attach all to different pods on the same node [It] I1105 20:38:38.942] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:151 I1105 20:38:38.942] I1105 20:38:38.942] Unexpected error: I1105 20:38:38.942] <*errors.errorString | 0xc000cb51d0>: { I1105 20:38:38.942] s: "PersistentVolumeClaims [pvc-fzxvb] not all in phase Bound within 5m0s", I1105 20:38:38.942] } I1105 20:38:38.943] PersistentVolumeClaims [pvc-fzxvb] not all in phase Bound within 5m0s I1105 20:38:38.943] occurred I1105 20:38:38.943] I1105 20:38:38.943] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/testsuites/testsuites.go:278 I1105 20:38:38.943] ------------------------------ I1105 20:38:38.943] [azuredisk-csi-e2e] Dynamic Provisioning [multi-az] I1105 20:38:38.943] should create a deployment object, write and read to it, delete the pod and write and read to it again I1105 20:38:38.943] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:193 I1105 20:38:38.943] [BeforeEach] [multi-az] I1105 20:38:38.944] /go/src/sigs.k8s.io/azuredisk-csi-driver/vendor/k8s.io/kubernetes/test/e2e/framework/framework.go:150 I1105 20:38:38.944] STEP: Creating a kubernetes client I1105 20:38:38.944] Nov 5 20:38:38.941: INFO: >>> kubeConfig: /workspace/aks971561099/kubeconfig/kubeconfig.eastus2.json I1105 20:38:38.944] STEP: Building a namespace api object, basename azuredisk I1105 20:38:39.181] STEP: Binding the e2e-test-privileged-psp PodSecurityPolicy to the default service account in azuredisk-507 I1105 20:38:39.447] STEP: Waiting for a default service account to be provisioned in namespace I1105 20:38:39.524] [BeforeEach] [multi-az] I1105 20:38:39.524] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:58 I1105 20:38:39.524] [It] should create a deployment object, write and read to it, delete the pod and write and read to it again I1105 20:38:39.524] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:193 I1105 20:38:39.525] STEP: setting up the StorageClass I1105 20:38:39.525] STEP: creating a StorageClass I1105 20:38:39.607] STEP: setting up the PVC I1105 20:38:39.607] STEP: creating a PVC I1105 20:38:39.694] STEP: waiting for PVC to be in phase "Bound" I1105 20:38:39.695] Nov 5 20:38:39.694: INFO: Waiting up to 5m0s for PersistentVolumeClaims [pvc-mkkrb] to have phase Bound I1105 20:38:39.778] Nov 5 20:38:39.778: INFO: PersistentVolumeClaim pvc-mkkrb found but phase is Pending instead of Bound. I1105 20:38:41.858] Nov 5 20:38:41.858: INFO: PersistentVolumeClaim pvc-mkkrb found but phase is Pending instead of Bound. I1105 20:38:43.937] Nov 5 20:38:43.937: INFO: PersistentVolumeClaim pvc-mkkrb found but phase is Pending instead of Bound. I1105 20:38:46.016] Nov 5 20:38:46.016: INFO: PersistentVolumeClaim pvc-mkkrb found but phase is Pending instead of Bound. I1105 20:38:48.096] Nov 5 20:38:48.096: INFO: PersistentVolumeClaim pvc-mkkrb found but phase is Pending instead of Bound. I1105 20:38:50.212] Nov 5 20:38:50.212: INFO: PersistentVolumeClaim pvc-mkkrb found but phase is Pending instead of Bound. I1105 20:38:52.291] Nov 5 20:38:52.291: INFO: PersistentVolumeClaim pvc-mkkrb found but phase is Pending instead of Bound. I1105 20:38:54.371] Nov 5 20:38:54.370: INFO: PersistentVolumeClaim pvc-mkkrb found but phase is Pending instead of Bound. I1105 20:38:56.456] Nov 5 20:38:56.456: INFO: PersistentVolumeClaim pvc-mkkrb found but phase is Pending instead of Bound. I1105 20:38:58.535] Nov 5 20:38:58.535: INFO: PersistentVolumeClaim pvc-mkkrb found but phase is Pending instead of Bound. I1105 20:39:00.614] Nov 5 20:39:00.614: INFO: PersistentVolumeClaim pvc-mkkrb found but phase is Pending instead of Bound. I1105 20:39:02.697] Nov 5 20:39:02.697: INFO: PersistentVolumeClaim pvc-mkkrb found but phase is Pending instead of Bound. I1105 20:39:04.777] Nov 5 20:39:04.777: INFO: PersistentVolumeClaim pvc-mkkrb found but phase is Pending instead of Bound. I1105 20:39:06.862] Nov 5 20:39:06.862: INFO: PersistentVolumeClaim pvc-mkkrb found but phase is Pending instead of Bound. I1105 20:39:08.941] Nov 5 20:39:08.941: INFO: PersistentVolumeClaim pvc-mkkrb found but phase is Pending instead of Bound. I1105 20:39:11.021] Nov 5 20:39:11.021: INFO: PersistentVolumeClaim pvc-mkkrb found but phase is Pending instead of Bound. I1105 20:39:13.100] Nov 5 20:39:13.100: INFO: PersistentVolumeClaim pvc-mkkrb found but phase is Pending instead of Bound. I1105 20:39:15.179] Nov 5 20:39:15.179: INFO: PersistentVolumeClaim pvc-mkkrb found but phase is Pending instead of Bound. I1105 20:39:17.258] Nov 5 20:39:17.258: INFO: PersistentVolumeClaim pvc-mkkrb found but phase is Pending instead of Bound. I1105 20:39:19.337] Nov 5 20:39:19.337: INFO: PersistentVolumeClaim pvc-mkkrb found but phase is Pending instead of Bound. I1105 20:39:21.417] Nov 5 20:39:21.417: INFO: PersistentVolumeClaim pvc-mkkrb found but phase is Pending instead of Bound. I1105 20:39:23.496] Nov 5 20:39:23.496: INFO: PersistentVolumeClaim pvc-mkkrb found but phase is Pending instead of Bound. I1105 20:39:25.576] Nov 5 20:39:25.576: INFO: PersistentVolumeClaim pvc-mkkrb found but phase is Pending instead of Bound. I1105 20:39:27.655] Nov 5 20:39:27.655: INFO: PersistentVolumeClaim pvc-mkkrb found but phase is Pending instead of Bound. I1105 20:39:29.735] Nov 5 20:39:29.734: INFO: PersistentVolumeClaim pvc-mkkrb found but phase is Pending instead of Bound. I1105 20:39:31.813] Nov 5 20:39:31.813: INFO: PersistentVolumeClaim pvc-mkkrb found but phase is Pending instead of Bound. I1105 20:39:33.893] Nov 5 20:39:33.892: INFO: PersistentVolumeClaim pvc-mkkrb found but phase is Pending instead of Bound. I1105 20:39:35.972] Nov 5 20:39:35.972: INFO: PersistentVolumeClaim pvc-mkkrb found but phase is Pending instead of Bound. I1105 20:39:38.052] Nov 5 20:39:38.051: INFO: PersistentVolumeClaim pvc-mkkrb found but phase is Pending instead of Bound. I1105 20:39:40.131] Nov 5 20:39:40.131: INFO: PersistentVolumeClaim pvc-mkkrb found but phase is Pending instead of Bound. I1105 20:39:42.217] Nov 5 20:39:42.217: INFO: PersistentVolumeClaim pvc-mkkrb found but phase is Pending instead of Bound. I1105 20:39:44.296] Nov 5 20:39:44.295: INFO: PersistentVolumeClaim pvc-mkkrb found but phase is Pending instead of Bound. I1105 20:39:46.375] Nov 5 20:39:46.375: INFO: PersistentVolumeClaim pvc-mkkrb found but phase is Pending instead of Bound. I1105 20:39:48.454] Nov 5 20:39:48.454: INFO: PersistentVolumeClaim pvc-mkkrb found but phase is Pending instead of Bound. I1105 20:39:50.534] Nov 5 20:39:50.533: INFO: PersistentVolumeClaim pvc-mkkrb found but phase is Pending instead of Bound. I1105 20:39:52.613] Nov 5 20:39:52.612: INFO: PersistentVolumeClaim pvc-mkkrb found but phase is Pending instead of Bound. I1105 20:39:54.691] Nov 5 20:39:54.691: INFO: PersistentVolumeClaim pvc-mkkrb found but phase is Pending instead of Bound. I1105 20:39:56.770] Nov 5 20:39:56.770: INFO: PersistentVolumeClaim pvc-mkkrb found but phase is Pending instead of Bound. I1105 20:39:58.849] Nov 5 20:39:58.849: INFO: PersistentVolumeClaim pvc-mkkrb found but phase is Pending instead of Bound. I1105 20:40:00.928] Nov 5 20:40:00.928: INFO: PersistentVolumeClaim pvc-mkkrb found but phase is Pending instead of Bound. I1105 20:40:03.007] Nov 5 20:40:03.007: INFO: PersistentVolumeClaim pvc-mkkrb found but phase is Pending instead of Bound. I1105 20:40:05.087] Nov 5 20:40:05.087: INFO: PersistentVolumeClaim pvc-mkkrb found but phase is Pending instead of Bound. I1105 20:40:07.166] Nov 5 20:40:07.166: INFO: PersistentVolumeClaim pvc-mkkrb found but phase is Pending instead of Bound. I1105 20:40:09.245] Nov 5 20:40:09.245: INFO: PersistentVolumeClaim pvc-mkkrb found but phase is Pending instead of Bound. I1105 20:40:11.324] Nov 5 20:40:11.324: INFO: PersistentVolumeClaim pvc-mkkrb found but phase is Pending instead of Bound. I1105 20:40:13.404] Nov 5 20:40:13.404: INFO: PersistentVolumeClaim pvc-mkkrb found but phase is Pending instead of Bound. I1105 20:40:15.484] Nov 5 20:40:15.484: INFO: PersistentVolumeClaim pvc-mkkrb found but phase is Pending instead of Bound. I1105 20:40:17.564] Nov 5 20:40:17.564: INFO: PersistentVolumeClaim pvc-mkkrb found but phase is Pending instead of Bound. I1105 20:40:19.643] Nov 5 20:40:19.643: INFO: PersistentVolumeClaim pvc-mkkrb found but phase is Pending instead of Bound. I1105 20:40:21.723] Nov 5 20:40:21.723: INFO: PersistentVolumeClaim pvc-mkkrb found but phase is Pending instead of Bound. I1105 20:40:23.802] Nov 5 20:40:23.802: INFO: PersistentVolumeClaim pvc-mkkrb found but phase is Pending instead of Bound. I1105 20:40:25.882] Nov 5 20:40:25.881: INFO: PersistentVolumeClaim pvc-mkkrb found but phase is Pending instead of Bound. I1105 20:40:27.962] Nov 5 20:40:27.962: INFO: PersistentVolumeClaim pvc-mkkrb found but phase is Pending instead of Bound. I1105 20:40:30.041] Nov 5 20:40:30.041: INFO: PersistentVolumeClaim pvc-mkkrb found but phase is Pending instead of Bound. I1105 20:40:32.120] Nov 5 20:40:32.120: INFO: PersistentVolumeClaim pvc-mkkrb found but phase is Pending instead of Bound. I1105 20:40:34.221] Nov 5 20:40:34.221: INFO: PersistentVolumeClaim pvc-mkkrb found but phase is Pending instead of Bound. I1105 20:40:36.301] Nov 5 20:40:36.300: INFO: PersistentVolumeClaim pvc-mkkrb found but phase is Pending instead of Bound. I1105 20:40:38.380] Nov 5 20:40:38.380: INFO: PersistentVolumeClaim pvc-mkkrb found but phase is Pending instead of Bound. I1105 20:40:40.460] Nov 5 20:40:40.460: INFO: PersistentVolumeClaim pvc-mkkrb found but phase is Pending instead of Bound. I1105 20:40:42.540] Nov 5 20:40:42.539: INFO: PersistentVolumeClaim pvc-mkkrb found but phase is Pending instead of Bound. I1105 20:40:44.619] Nov 5 20:40:44.619: INFO: PersistentVolumeClaim pvc-mkkrb found but phase is Pending instead of Bound. I1105 20:40:46.699] Nov 5 20:40:46.698: INFO: PersistentVolumeClaim pvc-mkkrb found but phase is Pending instead of Bound. I1105 20:40:48.778] Nov 5 20:40:48.778: INFO: PersistentVolumeClaim pvc-mkkrb found but phase is Pending instead of Bound. I1105 20:40:50.858] Nov 5 20:40:50.858: INFO: PersistentVolumeClaim pvc-mkkrb found but phase is Pending instead of Bound. I1105 20:40:52.937] Nov 5 20:40:52.937: INFO: PersistentVolumeClaim pvc-mkkrb found but phase is Pending instead of Bound. I1105 20:40:55.024] Nov 5 20:40:55.024: INFO: PersistentVolumeClaim pvc-mkkrb found but phase is Pending instead of Bound. I1105 20:40:57.104] Nov 5 20:40:57.104: INFO: PersistentVolumeClaim pvc-mkkrb found but phase is Pending instead of Bound. I1105 20:40:59.183] Nov 5 20:40:59.183: INFO: PersistentVolumeClaim pvc-mkkrb found but phase is Pending instead of Bound. I1105 20:41:01.262] Nov 5 20:41:01.262: INFO: PersistentVolumeClaim pvc-mkkrb found but phase is Pending instead of Bound. I1105 20:41:03.342] Nov 5 20:41:03.341: INFO: PersistentVolumeClaim pvc-mkkrb found but phase is Pending instead of Bound. I1105 20:41:05.421] Nov 5 20:41:05.421: INFO: PersistentVolumeClaim pvc-mkkrb found but phase is Pending instead of Bound. I1105 20:41:07.500] Nov 5 20:41:07.500: INFO: PersistentVolumeClaim pvc-mkkrb found but phase is Pending instead of Bound. I1105 20:41:09.579] Nov 5 20:41:09.579: INFO: PersistentVolumeClaim pvc-mkkrb found but phase is Pending instead of Bound. I1105 20:41:11.659] Nov 5 20:41:11.658: INFO: PersistentVolumeClaim pvc-mkkrb found but phase is Pending instead of Bound. I1105 20:41:13.738] Nov 5 20:41:13.738: INFO: PersistentVolumeClaim pvc-mkkrb found but phase is Pending instead of Bound. I1105 20:41:15.817] Nov 5 20:41:15.817: INFO: PersistentVolumeClaim pvc-mkkrb found but phase is Pending instead of Bound. I1105 20:41:17.896] Nov 5 20:41:17.895: INFO: PersistentVolumeClaim pvc-mkkrb found but phase is Pending instead of Bound. I1105 20:41:19.975] Nov 5 20:41:19.975: INFO: PersistentVolumeClaim pvc-mkkrb found but phase is Pending instead of Bound. I1105 20:41:22.054] Nov 5 20:41:22.054: INFO: PersistentVolumeClaim pvc-mkkrb found but phase is Pending instead of Bound. I1105 20:41:24.134] Nov 5 20:41:24.134: INFO: PersistentVolumeClaim pvc-mkkrb found but phase is Pending instead of Bound. I1105 20:41:26.218] Nov 5 20:41:26.218: INFO: PersistentVolumeClaim pvc-mkkrb found but phase is Pending instead of Bound. I1105 20:41:28.297] Nov 5 20:41:28.297: INFO: PersistentVolumeClaim pvc-mkkrb found but phase is Pending instead of Bound. I1105 20:41:30.376] Nov 5 20:41:30.376: INFO: PersistentVolumeClaim pvc-mkkrb found but phase is Pending instead of Bound. I1105 20:41:32.460] Nov 5 20:41:32.460: INFO: PersistentVolumeClaim pvc-mkkrb found but phase is Pending instead of Bound. I1105 20:41:34.542] Nov 5 20:41:34.541: INFO: PersistentVolumeClaim pvc-mkkrb found but phase is Pending instead of Bound. I1105 20:41:36.621] Nov 5 20:41:36.620: INFO: PersistentVolumeClaim pvc-mkkrb found but phase is Pending instead of Bound. I1105 20:41:38.700] Nov 5 20:41:38.699: INFO: PersistentVolumeClaim pvc-mkkrb found but phase is Pending instead of Bound. I1105 20:41:40.778] Nov 5 20:41:40.778: INFO: PersistentVolumeClaim pvc-mkkrb found but phase is Pending instead of Bound. I1105 20:41:42.860] Nov 5 20:41:42.860: INFO: PersistentVolumeClaim pvc-mkkrb found but phase is Pending instead of Bound. I1105 20:41:44.939] Nov 5 20:41:44.939: INFO: PersistentVolumeClaim pvc-mkkrb found but phase is Pending instead of Bound. I1105 20:41:47.018] Nov 5 20:41:47.018: INFO: PersistentVolumeClaim pvc-mkkrb found but phase is Pending instead of Bound. I1105 20:41:49.100] Nov 5 20:41:49.100: INFO: PersistentVolumeClaim pvc-mkkrb found but phase is Pending instead of Bound. I1105 20:41:51.179] Nov 5 20:41:51.179: INFO: PersistentVolumeClaim pvc-mkkrb found but phase is Pending instead of Bound. I1105 20:41:53.259] Nov 5 20:41:53.258: INFO: PersistentVolumeClaim pvc-mkkrb found but phase is Pending instead of Bound. I1105 20:41:55.341] Nov 5 20:41:55.340: INFO: PersistentVolumeClaim pvc-mkkrb found but phase is Pending instead of Bound. I1105 20:41:57.420] Nov 5 20:41:57.420: INFO: PersistentVolumeClaim pvc-mkkrb found but phase is Pending instead of Bound. I1105 20:41:59.499] Nov 5 20:41:59.499: INFO: PersistentVolumeClaim pvc-mkkrb found but phase is Pending instead of Bound. I1105 20:42:01.579] Nov 5 20:42:01.579: INFO: PersistentVolumeClaim pvc-mkkrb found but phase is Pending instead of Bound. I1105 20:42:03.658] Nov 5 20:42:03.658: INFO: PersistentVolumeClaim pvc-mkkrb found but phase is Pending instead of Bound. I1105 20:42:05.738] Nov 5 20:42:05.737: INFO: PersistentVolumeClaim pvc-mkkrb found but phase is Pending instead of Bound. I1105 20:42:07.817] Nov 5 20:42:07.816: INFO: PersistentVolumeClaim pvc-mkkrb found but phase is Pending instead of Bound. I1105 20:42:28.485] Nov 5 20:42:28.485: INFO: PersistentVolumeClaim pvc-mkkrb found but phase is Pending instead of Bound. I1105 20:42:30.559] Nov 5 20:42:30.558: INFO: PersistentVolumeClaim pvc-mkkrb found but phase is Pending instead of Bound. I1105 20:42:32.644] Nov 5 20:42:32.644: INFO: PersistentVolumeClaim pvc-mkkrb found but phase is Pending instead of Bound. I1105 20:42:34.729] Nov 5 20:42:34.728: INFO: PersistentVolumeClaim pvc-mkkrb found but phase is Pending instead of Bound. I1105 20:42:36.814] Nov 5 20:42:36.814: INFO: PersistentVolumeClaim pvc-mkkrb found but phase is Pending instead of Bound. I1105 20:42:38.899] Nov 5 20:42:38.899: INFO: PersistentVolumeClaim pvc-mkkrb found but phase is Pending instead of Bound. I1105 20:42:40.984] Nov 5 20:42:40.984: INFO: PersistentVolumeClaim pvc-mkkrb found but phase is Pending instead of Bound. I1105 20:42:43.072] Nov 5 20:42:43.072: INFO: PersistentVolumeClaim pvc-mkkrb found but phase is Pending instead of Bound. I1105 20:42:45.157] Nov 5 20:42:45.157: INFO: PersistentVolumeClaim pvc-mkkrb found but phase is Pending instead of Bound. I1105 20:42:47.242] Nov 5 20:42:47.242: INFO: PersistentVolumeClaim pvc-mkkrb found but phase is Pending instead of Bound. I1105 20:42:49.327] Nov 5 20:42:49.326: INFO: PersistentVolumeClaim pvc-mkkrb found but phase is Pending instead of Bound. I1105 20:42:51.411] Nov 5 20:42:51.411: INFO: PersistentVolumeClaim pvc-mkkrb found but phase is Pending instead of Bound. I1105 20:42:53.496] Nov 5 20:42:53.495: INFO: PersistentVolumeClaim pvc-mkkrb found but phase is Pending instead of Bound. I1105 20:42:55.581] Nov 5 20:42:55.580: INFO: PersistentVolumeClaim pvc-mkkrb found but phase is Pending instead of Bound. I1105 20:42:57.666] Nov 5 20:42:57.665: INFO: PersistentVolumeClaim pvc-mkkrb found but phase is Pending instead of Bound. I1105 20:42:59.750] Nov 5 20:42:59.750: INFO: PersistentVolumeClaim pvc-mkkrb found but phase is Pending instead of Bound. I1105 20:43:01.834] Nov 5 20:43:01.834: INFO: PersistentVolumeClaim pvc-mkkrb found but phase is Pending instead of Bound. I1105 20:43:03.919] Nov 5 20:43:03.919: INFO: PersistentVolumeClaim pvc-mkkrb found but phase is Pending instead of Bound. I1105 20:43:06.004] Nov 5 20:43:06.004: INFO: PersistentVolumeClaim pvc-mkkrb found but phase is Pending instead of Bound. I1105 20:43:08.088] Nov 5 20:43:08.088: INFO: PersistentVolumeClaim pvc-mkkrb found but phase is Pending instead of Bound. I1105 20:43:10.221] Nov 5 20:43:10.220: INFO: PersistentVolumeClaim pvc-mkkrb found but phase is Pending instead of Bound. I1105 20:43:12.306] Nov 5 20:43:12.305: INFO: PersistentVolumeClaim pvc-mkkrb found but phase is Pending instead of Bound. I1105 20:43:14.390] Nov 5 20:43:14.390: INFO: PersistentVolumeClaim pvc-mkkrb found but phase is Pending instead of Bound. I1105 20:43:16.475] Nov 5 20:43:16.475: INFO: PersistentVolumeClaim pvc-mkkrb found but phase is Pending instead of Bound. I1105 20:43:18.560] Nov 5 20:43:18.560: INFO: PersistentVolumeClaim pvc-mkkrb found but phase is Pending instead of Bound. I1105 20:43:20.646] Nov 5 20:43:20.646: INFO: PersistentVolumeClaim pvc-mkkrb found but phase is Pending instead of Bound. I1105 20:43:22.731] Nov 5 20:43:22.731: INFO: PersistentVolumeClaim pvc-mkkrb found but phase is Pending instead of Bound. I1105 20:43:24.816] Nov 5 20:43:24.816: INFO: PersistentVolumeClaim pvc-mkkrb found but phase is Pending instead of Bound. I1105 20:43:26.902] Nov 5 20:43:26.902: INFO: PersistentVolumeClaim pvc-mkkrb found but phase is Pending instead of Bound. I1105 20:43:28.987] Nov 5 20:43:28.986: INFO: PersistentVolumeClaim pvc-mkkrb found but phase is Pending instead of Bound. I1105 20:43:31.071] Nov 5 20:43:31.071: INFO: PersistentVolumeClaim pvc-mkkrb found but phase is Pending instead of Bound. I1105 20:43:33.155] Nov 5 20:43:33.155: INFO: PersistentVolumeClaim pvc-mkkrb found but phase is Pending instead of Bound. I1105 20:43:35.240] Nov 5 20:43:35.240: INFO: PersistentVolumeClaim pvc-mkkrb found but phase is Pending instead of Bound. I1105 20:43:37.324] Nov 5 20:43:37.324: INFO: PersistentVolumeClaim pvc-mkkrb found but phase is Pending instead of Bound. I1105 20:43:39.409] Nov 5 20:43:39.409: INFO: PersistentVolumeClaim pvc-mkkrb found but phase is Pending instead of Bound. I1105 20:43:41.409] Nov 5 20:43:41.409: INFO: Unexpected error occurred: PersistentVolumeClaims [pvc-mkkrb] not all in phase Bound within 5m0s I1105 20:43:41.410] [AfterEach] [multi-az] I1105 20:43:41.410] /go/src/sigs.k8s.io/azuredisk-csi-driver/vendor/k8s.io/kubernetes/test/e2e/framework/framework.go:151 I1105 20:43:41.410] STEP: Collecting events from namespace "azuredisk-507". I1105 20:43:41.500] STEP: Found 3 events. I1105 20:43:41.500] Nov 5 20:43:41.500: INFO: At 2019-11-05 20:38:39 +0000 UTC - event for pvc-mkkrb: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "disk.csi.azure.com" or manually created by system administrator I1105 20:43:41.500] Nov 5 20:43:41.500: INFO: At 2019-11-05 20:38:39 +0000 UTC - event for pvc-mkkrb: {disk.csi.azure.com_csi-azuredisk-controller-6f6587bfc6-62nrt_3930bc69-0009-11ea-8cf5-829a27ff7d5a } Provisioning: External provisioner is provisioning volume for claim "azuredisk-507/pvc-mkkrb" I1105 20:43:41.501] Nov 5 20:43:41.500: INFO: At 2019-11-05 20:38:39 +0000 UTC - event for pvc-mkkrb: {disk.csi.azure.com_csi-azuredisk-controller-6f6587bfc6-62nrt_3930bc69-0009-11ea-8cf5-829a27ff7d5a } ProvisioningFailed: failed to provision volume with StorageClass "azuredisk-507-disk.csi.azure.com-dynamic-sc-qkrmz": rpc error: code = Unknown desc = compute.DisksClient#CreateOrUpdate: Failure sending request: StatusCode=400 -- Original Error: Code="InvalidAvailabilityZone" Message="The zone(s) '' for resource 'Microsoft.Compute/disks/pvc-5b8b4b6c-a77e-4ceb-9226-ec3f3ed561ec' is not supported. The supported zones for location 'eastus2' are '1,2,3'" I1105 20:43:41.584] Nov 5 20:43:41.583: INFO: POD NODE PHASE GRACE CONDITIONS I1105 20:43:41.584] Nov 5 20:43:41.583: INFO: I1105 20:43:41.915] Nov 5 20:43:41.915: INFO: I1105 20:43:41.916] Logging node info for node k8s-agentpool-97435445-vmss000000 I1105 20:43:42.003] Nov 5 20:43:41.999: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-agentpool-97435445-vmss000000,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-agentpool-97435445-vmss000000,UID:e955dc47-8c8f-4142-9241-3d409f32e42d,ResourceVersion:7407,Generation:0,CreationTimestamp:2019-11-05 20:07:34 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{agentpool: agentpool,beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_D4s_v3,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 0,kubernetes.azure.com/cluster: kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b,kubernetes.azure.com/role: agent,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-agentpool-97435445-vmss000000,kubernetes.io/os: linux,kubernetes.io/role: agent,node-role.kubernetes.io/agent: ,storageprofile: managed,storagetier: Premium_LRS,topology.disk.csi.azure.com/zone: 0,},Annotations:map[string]string{csi.volume.kubernetes.io/nodeid: {"disk.csi.azure.com":"k8s-agentpool-97435445-vmss000000"},node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.2.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool-97435445-vmss/virtualMachines/0,Unschedulable:false,Taints:[],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-csi-disk.csi.azure.com: {{8 0} {} 8 DecimalSI},cpu: {{4 0} {} 4 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{16797040640 0} {} BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-csi-disk.csi.azure.com: {{8 0} {} 8 DecimalSI},cpu: {{4 0} {} 4 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{16010608640 0} {} BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 20:08:01 +0000 UTC 2019-11-05 20:08:01 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 20:43:41 +0000 UTC 2019-11-05 20:07:27 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 20:43:41 +0000 UTC 2019-11-05 20:07:27 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 20:43:41 +0000 UTC 2019-11-05 20:07:27 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 20:43:41 +0000 UTC 2019-11-05 20:07:55 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.0.4} {Hostname k8s-agentpool-97435445-vmss000000}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:bfb55325809340228149d2d93f50df58,SystemUUID:CB2ED036-0439-F641-BC7F-640C3638C027,BootID:a917f4d1-549d-471d-b7dc-47ae1715d97d,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[k8sprowinternal.azurecr.io/azuredisk-csi@sha256:75d4ff5d7cd845d6e810c67796ecb037e852fb849ec1eb5b3194564008e6d1e0 k8sprowinternal.azurecr.io/azuredisk-csi:e2e-3e3ed045c6faa509dbd8d2890ee9f56aa2934887] 286930414} {[k8s.gcr.io/kubernetes-dashboard-amd64@sha256:0ae6b69432e78069c5ce2bcde0fe409c5c4d6f0f4d9cd50a17974fea38898747 k8s.gcr.io/kubernetes-dashboard-amd64:v1.10.1] 121711221} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/metrics-server-amd64@sha256:49a9f12f7067d11f42c803dbe61ed2c1299959ad85cb315b25ff7eef8e6b8892 k8s.gcr.io/metrics-server-amd64:v0.2.1] 42541759} {[k8s.gcr.io/coredns@sha256:e83beb5e43f8513fa735e77ffc5859640baea30a882a11cc75c4c3244a737d3c k8s.gcr.io/coredns:1.5.0] 42488424} {[mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume@sha256:4fd30d43947d4a54fc89ead7985beecfd3c9b2a93a0655a373b1608ab90bd5af mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume:v0.0.7] 22909487} {[quay.io/k8scsi/livenessprobe@sha256:8c9c8a0f330acc2e558ae6624309447d02522e76aa35af23fa19b88abf4381cb quay.io/k8scsi/livenessprobe:v1.0.2] 17415381} {[quay.io/k8scsi/csi-node-driver-registrar@sha256:13daf82fb99e951a4bff8ae5fc7c17c3a8fe7130be6400990d8f6076c32d4599 quay.io/k8scsi/csi-node-driver-registrar:v1.1.0] 15815995} {[mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume@sha256:23d8c6033f02a1ecad05127ebdc931bb871264228661bc122704b0974e4d9fdd mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume:1.0.8] 1159025} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 20:43:42.003] Nov 5 20:43:42.000: INFO: I1105 20:43:42.003] Logging kubelet events for node k8s-agentpool-97435445-vmss000000 I1105 20:43:42.089] Nov 5 20:43:42.089: INFO: I1105 20:43:42.090] Logging pods the kubelet thinks is on node k8s-agentpool-97435445-vmss000000 I1105 20:43:42.235] Nov 5 20:43:42.235: INFO: coredns-7f68dcdbdb-zh8g5 started at 2019-11-05 20:08:01 +0000 UTC (0+1 container statuses recorded) I1105 20:43:42.235] Nov 5 20:43:42.235: INFO: Container coredns ready: true, restart count 0 I1105 20:43:42.236] Nov 5 20:43:42.235: INFO: azure-ip-masq-agent-m7tv9 started at 2019-11-05 20:08:01 +0000 UTC (0+1 container statuses recorded) I1105 20:43:42.236] Nov 5 20:43:42.235: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 20:43:42.236] Nov 5 20:43:42.235: INFO: keyvault-flexvolume-smspt started at 2019-11-05 20:08:01 +0000 UTC (0+1 container statuses recorded) I1105 20:43:42.236] Nov 5 20:43:42.235: INFO: Container keyvault-flexvolume ready: true, restart count 0 I1105 20:43:42.236] Nov 5 20:43:42.235: INFO: metrics-server-864ffbc5c-wl84z started at 2019-11-05 20:08:01 +0000 UTC (0+1 container statuses recorded) I1105 20:43:42.236] Nov 5 20:43:42.235: INFO: Container metrics-server ready: true, restart count 0 I1105 20:43:42.237] Nov 5 20:43:42.235: INFO: kube-proxy-8mt7f started at 2019-11-05 20:08:01 +0000 UTC (0+1 container statuses recorded) I1105 20:43:42.237] Nov 5 20:43:42.235: INFO: Container kube-proxy ready: true, restart count 0 I1105 20:43:42.237] Nov 5 20:43:42.235: INFO: kubernetes-dashboard-66dd8b8df7-t6j4l started at 2019-11-05 20:08:01 +0000 UTC (0+1 container statuses recorded) I1105 20:43:42.237] Nov 5 20:43:42.235: INFO: Container kubernetes-dashboard ready: true, restart count 0 I1105 20:43:42.237] Nov 5 20:43:42.235: INFO: csi-azuredisk-node-5wwhz started at 2019-11-05 20:16:09 +0000 UTC (0+3 container statuses recorded) I1105 20:43:42.237] Nov 5 20:43:42.235: INFO: Container azuredisk ready: true, restart count 0 I1105 20:43:42.237] Nov 5 20:43:42.235: INFO: Container liveness-probe ready: true, restart count 0 I1105 20:43:42.237] Nov 5 20:43:42.235: INFO: Container node-driver-registrar ready: true, restart count 0 I1105 20:43:42.238] Nov 5 20:43:42.235: INFO: blobfuse-flexvol-installer-p7hdb started at 2019-11-05 20:08:01 +0000 UTC (0+1 container statuses recorded) I1105 20:43:42.238] Nov 5 20:43:42.235: INFO: Container blobfuse-flexvol-installer ready: true, restart count 0 I1105 20:43:42.509] Nov 5 20:43:42.509: INFO: I1105 20:43:42.509] Latency metrics for node k8s-agentpool-97435445-vmss000000 I1105 20:43:42.510] Nov 5 20:43:42.509: INFO: I1105 20:43:42.510] Logging node info for node k8s-agentpool-97435445-vmss000001 I1105 20:43:42.626] Nov 5 20:43:42.623: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-agentpool-97435445-vmss000001,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-agentpool-97435445-vmss000001,UID:75d3cc50-ca6d-45e0-a42a-6090e8d537dd,ResourceVersion:7405,Generation:0,CreationTimestamp:2019-11-05 20:07:34 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{agentpool: agentpool,beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_D4s_v3,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 1,kubernetes.azure.com/cluster: kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b,kubernetes.azure.com/role: agent,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-agentpool-97435445-vmss000001,kubernetes.io/os: linux,kubernetes.io/role: agent,node-role.kubernetes.io/agent: ,storageprofile: managed,storagetier: Premium_LRS,topology.disk.csi.azure.com/zone: 1,},Annotations:map[string]string{csi.volume.kubernetes.io/nodeid: {"disk.csi.azure.com":"k8s-agentpool-97435445-vmss000001"},node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.0.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool-97435445-vmss/virtualMachines/1,Unschedulable:false,Taints:[],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-csi-disk.csi.azure.com: {{8 0} {} 8 DecimalSI},cpu: {{4 0} {} 4 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{16797044736 0} {} 16403364Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-csi-disk.csi.azure.com: {{8 0} {} 8 DecimalSI},cpu: {{4 0} {} 4 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{16010612736 0} {} 15635364Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 20:08:25 +0000 UTC 2019-11-05 20:08:25 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 20:43:40 +0000 UTC 2019-11-05 20:07:26 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 20:43:40 +0000 UTC 2019-11-05 20:07:26 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 20:43:40 +0000 UTC 2019-11-05 20:07:26 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 20:43:40 +0000 UTC 2019-11-05 20:07:55 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.0.5} {Hostname k8s-agentpool-97435445-vmss000001}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:e78def54b2f545c6b94d88bbba523c60,SystemUUID:40AE8188-86AB-D848-84A4-D9254E1258BC,BootID:8587ec06-f4f4-43c1-86cd-06c295b05b63,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[k8sprowinternal.azurecr.io/azuredisk-csi@sha256:75d4ff5d7cd845d6e810c67796ecb037e852fb849ec1eb5b3194564008e6d1e0 k8sprowinternal.azurecr.io/azuredisk-csi:e2e-3e3ed045c6faa509dbd8d2890ee9f56aa2934887] 286930414} {[gcr.io/kubernetes-helm/tiller@sha256:f6d8f4ab9ba993b5f5b60a6edafe86352eabe474ffeb84cb6c79b8866dce45d1 gcr.io/kubernetes-helm/tiller:v2.11.0] 71821984} {[quay.io/k8scsi/csi-provisioner@sha256:d657c839dce87324fe2b677302913f9386f885f8746be7bea0ced5b0844e3433 quay.io/k8scsi/csi-provisioner:v1.3.1] 50596998} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[quay.io/k8scsi/csi-attacher@sha256:26fccd7a99d973845df1193b46ebdcc6ab8dc5f6e6be319750c471fce1742d13 quay.io/k8scsi/csi-attacher:v1.2.0] 46226754} {[quay.io/k8scsi/csi-cluster-driver-registrar@sha256:fafd75ae5442f192cfa8c2e792903aee30d5884b62e802e4464b0a895d21e3ef quay.io/k8scsi/csi-cluster-driver-registrar:v1.0.1] 45874691} {[quay.io/k8scsi/csi-snapshotter@sha256:a49e0da1af6f2bf717e41ba1eee8b5e6a1cbd66a709dd92cc43fe475fe2589eb quay.io/k8scsi/csi-snapshotter:v1.1.0] 44325352} {[mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume@sha256:4fd30d43947d4a54fc89ead7985beecfd3c9b2a93a0655a373b1608ab90bd5af mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume:v0.0.7] 22909487} {[quay.io/k8scsi/livenessprobe@sha256:8c9c8a0f330acc2e558ae6624309447d02522e76aa35af23fa19b88abf4381cb quay.io/k8scsi/livenessprobe:v1.0.2] 17415381} {[quay.io/k8scsi/csi-node-driver-registrar@sha256:13daf82fb99e951a4bff8ae5fc7c17c3a8fe7130be6400990d8f6076c32d4599 quay.io/k8scsi/csi-node-driver-registrar:v1.1.0] 15815995} {[quay.io/k8scsi/livenessprobe@sha256:dde617756e0f602adc566ab71fd885f1dad451ad3fb063ac991c95a2ff47aea5 quay.io/k8scsi/livenessprobe:v1.1.0] 14967303} {[mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume@sha256:23d8c6033f02a1ecad05127ebdc931bb871264228661bc122704b0974e4d9fdd mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume:1.0.8] 1159025} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 20:43:42.627] Nov 5 20:43:42.623: INFO: I1105 20:43:42.627] Logging kubelet events for node k8s-agentpool-97435445-vmss000001 I1105 20:43:42.712] Nov 5 20:43:42.712: INFO: I1105 20:43:42.712] Logging pods the kubelet thinks is on node k8s-agentpool-97435445-vmss000001 I1105 20:43:42.802] Nov 5 20:43:42.801: INFO: tiller-deploy-54c96cb5df-pt7x6 started at 2019-11-05 20:08:00 +0000 UTC (0+1 container statuses recorded) I1105 20:43:42.802] Nov 5 20:43:42.801: INFO: Container tiller ready: true, restart count 0 I1105 20:43:42.802] Nov 5 20:43:42.802: INFO: csi-azuredisk-node-q2dmt started at 2019-11-05 20:16:09 +0000 UTC (0+3 container statuses recorded) I1105 20:43:42.802] Nov 5 20:43:42.802: INFO: Container azuredisk ready: true, restart count 0 I1105 20:43:42.802] Nov 5 20:43:42.802: INFO: Container liveness-probe ready: true, restart count 0 I1105 20:43:42.802] Nov 5 20:43:42.802: INFO: Container node-driver-registrar ready: true, restart count 0 I1105 20:43:42.803] Nov 5 20:43:42.802: INFO: csi-azuredisk-controller-6f6587bfc6-62nrt started at 2019-11-05 20:16:09 +0000 UTC (0+6 container statuses recorded) I1105 20:43:42.803] Nov 5 20:43:42.802: INFO: Container azuredisk ready: true, restart count 0 I1105 20:43:42.803] Nov 5 20:43:42.802: INFO: Container cluster-driver-registrar ready: true, restart count 0 I1105 20:43:42.803] Nov 5 20:43:42.802: INFO: Container csi-attacher ready: true, restart count 0 I1105 20:43:42.803] Nov 5 20:43:42.802: INFO: Container csi-provisioner ready: true, restart count 0 I1105 20:43:42.803] Nov 5 20:43:42.802: INFO: Container csi-snapshotter ready: true, restart count 0 I1105 20:43:42.803] Nov 5 20:43:42.802: INFO: Container liveness-probe ready: true, restart count 0 I1105 20:43:42.803] Nov 5 20:43:42.802: INFO: blobfuse-flexvol-installer-j4s7t started at 2019-11-05 20:07:59 +0000 UTC (0+1 container statuses recorded) I1105 20:43:42.804] Nov 5 20:43:42.802: INFO: Container blobfuse-flexvol-installer ready: true, restart count 0 I1105 20:43:42.804] Nov 5 20:43:42.802: INFO: azure-ip-masq-agent-gdhcq started at 2019-11-05 20:07:59 +0000 UTC (0+1 container statuses recorded) I1105 20:43:42.804] Nov 5 20:43:42.802: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 20:43:42.804] Nov 5 20:43:42.802: INFO: keyvault-flexvolume-6bffx started at 2019-11-05 20:07:59 +0000 UTC (0+1 container statuses recorded) I1105 20:43:42.804] Nov 5 20:43:42.802: INFO: Container keyvault-flexvolume ready: true, restart count 0 I1105 20:43:42.804] Nov 5 20:43:42.802: INFO: kube-proxy-kjrpb started at 2019-11-05 20:08:00 +0000 UTC (0+1 container statuses recorded) I1105 20:43:42.804] Nov 5 20:43:42.802: INFO: Container kube-proxy ready: true, restart count 0 I1105 20:43:43.081] Nov 5 20:43:43.081: INFO: I1105 20:43:43.081] Latency metrics for node k8s-agentpool-97435445-vmss000001 I1105 20:43:43.082] Nov 5 20:43:43.081: INFO: I1105 20:43:43.082] Logging node info for node k8s-master-97435445-0 I1105 20:43:43.170] Nov 5 20:43:43.168: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-97435445-0,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-97435445-0,UID:df3ae36f-961c-4729-a8f5-34cda54f4ff9,ResourceVersion:7295,Generation:0,CreationTimestamp:2019-11-05 20:08:36 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 2,kubernetes.azure.com/cluster: kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-97435445-0,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.4.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b/providers/Microsoft.Compute/virtualMachines/k8s-master-97435445-0,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 20:08:46 +0000 UTC 2019-11-05 20:08:46 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 20:43:01 +0000 UTC 2019-11-05 20:08:28 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 20:43:01 +0000 UTC 2019-11-05 20:08:28 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 20:43:01 +0000 UTC 2019-11-05 20:08:28 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 20:43:01 +0000 UTC 2019-11-05 20:08:36 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.255.5} {Hostname k8s-master-97435445-0}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:3c381865192341c2868672e8626cef7d,SystemUUID:F3D259D3-7EB8-B844-A1F0-F954DCD0D76B,BootID:4948329e-cf57-48c3-aa83-5297bef70370,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 20:43:43.170] Nov 5 20:43:43.168: INFO: I1105 20:43:43.170] Logging kubelet events for node k8s-master-97435445-0 I1105 20:43:43.257] Nov 5 20:43:43.257: INFO: I1105 20:43:43.257] Logging pods the kubelet thinks is on node k8s-master-97435445-0 I1105 20:43:43.347] Nov 5 20:43:43.347: INFO: kube-scheduler-k8s-master-97435445-0 started at 2019-11-05 20:08:22 +0000 UTC (0+1 container statuses recorded) I1105 20:43:43.347] Nov 5 20:43:43.347: INFO: Container kube-scheduler ready: true, restart count 0 I1105 20:43:43.347] Nov 5 20:43:43.347: INFO: azure-ip-masq-agent-q6dcl started at 2019-11-05 20:08:38 +0000 UTC (0+1 container statuses recorded) I1105 20:43:43.348] Nov 5 20:43:43.347: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 20:43:43.348] Nov 5 20:43:43.347: INFO: kube-proxy-5wnd2 started at 2019-11-05 20:08:38 +0000 UTC (0+1 container statuses recorded) I1105 20:43:43.348] Nov 5 20:43:43.347: INFO: Container kube-proxy ready: true, restart count 0 I1105 20:43:43.348] Nov 5 20:43:43.347: INFO: cloud-controller-manager-k8s-master-97435445-0 started at 2019-11-05 20:08:22 +0000 UTC (0+1 container statuses recorded) I1105 20:43:43.348] Nov 5 20:43:43.347: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 20:43:43.348] Nov 5 20:43:43.347: INFO: kube-addon-manager-k8s-master-97435445-0 started at 2019-11-05 20:08:48 +0000 UTC (0+1 container statuses recorded) I1105 20:43:43.348] Nov 5 20:43:43.347: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 20:43:43.349] Nov 5 20:43:43.347: INFO: kube-apiserver-k8s-master-97435445-0 started at 2019-11-05 20:08:22 +0000 UTC (0+1 container statuses recorded) I1105 20:43:43.349] Nov 5 20:43:43.347: INFO: Container kube-apiserver ready: true, restart count 0 I1105 20:43:43.349] Nov 5 20:43:43.347: INFO: kube-controller-manager-k8s-master-97435445-0 started at 2019-11-05 20:08:22 +0000 UTC (0+1 container statuses recorded) I1105 20:43:43.349] Nov 5 20:43:43.347: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 20:43:43.631] Nov 5 20:43:43.631: INFO: I1105 20:43:43.631] Latency metrics for node k8s-master-97435445-0 I1105 20:43:43.631] Nov 5 20:43:43.631: INFO: I1105 20:43:43.631] Logging node info for node k8s-master-97435445-1 I1105 20:43:43.718] Nov 5 20:43:43.715: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-97435445-1,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-97435445-1,UID:05534a67-d493-4ab5-9b0f-5ff5490e234f,ResourceVersion:7372,Generation:0,CreationTimestamp:2019-11-05 20:07:34 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 0,kubernetes.azure.com/cluster: kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-97435445-1,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.1.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b/providers/Microsoft.Compute/virtualMachines/k8s-master-97435445-1,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 20:08:26 +0000 UTC 2019-11-05 20:08:26 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 20:43:29 +0000 UTC 2019-11-05 20:07:28 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 20:43:29 +0000 UTC 2019-11-05 20:07:28 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 20:43:29 +0000 UTC 2019-11-05 20:07:28 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 20:43:29 +0000 UTC 2019-11-05 20:07:55 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{Hostname k8s-master-97435445-1} {InternalIP 10.240.255.6}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:079b605103b74acabac18fd2ea63368a,SystemUUID:38220EEC-BEFF-4C46-AA05-5A214AAFDC28,BootID:b1b45df0-d925-4bcc-9830-0202df36ebb4,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 20:43:43.718] Nov 5 20:43:43.716: INFO: I1105 20:43:43.718] Logging kubelet events for node k8s-master-97435445-1 I1105 20:43:43.804] Nov 5 20:43:43.804: INFO: I1105 20:43:43.804] Logging pods the kubelet thinks is on node k8s-master-97435445-1 I1105 20:43:43.894] Nov 5 20:43:43.894: INFO: cloud-controller-manager-k8s-master-97435445-1 started at 2019-11-05 20:07:22 +0000 UTC (0+1 container statuses recorded) I1105 20:43:43.894] Nov 5 20:43:43.894: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 20:43:43.895] Nov 5 20:43:43.894: INFO: kube-addon-manager-k8s-master-97435445-1 started at 2019-11-05 20:07:22 +0000 UTC (0+1 container statuses recorded) I1105 20:43:43.895] Nov 5 20:43:43.894: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 20:43:43.895] Nov 5 20:43:43.894: INFO: kube-apiserver-k8s-master-97435445-1 started at 2019-11-05 20:07:22 +0000 UTC (0+1 container statuses recorded) I1105 20:43:43.895] Nov 5 20:43:43.894: INFO: Container kube-apiserver ready: true, restart count 0 I1105 20:43:43.895] Nov 5 20:43:43.894: INFO: kube-controller-manager-k8s-master-97435445-1 started at 2019-11-05 20:07:22 +0000 UTC (0+1 container statuses recorded) I1105 20:43:43.896] Nov 5 20:43:43.894: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 20:43:43.896] Nov 5 20:43:43.894: INFO: kube-scheduler-k8s-master-97435445-1 started at 2019-11-05 20:07:22 +0000 UTC (0+1 container statuses recorded) I1105 20:43:43.896] Nov 5 20:43:43.894: INFO: Container kube-scheduler ready: true, restart count 0 I1105 20:43:43.896] Nov 5 20:43:43.894: INFO: azure-ip-masq-agent-n29bm started at 2019-11-05 20:07:59 +0000 UTC (0+1 container statuses recorded) I1105 20:43:43.896] Nov 5 20:43:43.894: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 20:43:43.896] Nov 5 20:43:43.894: INFO: kube-proxy-h4s2c started at 2019-11-05 20:08:00 +0000 UTC (0+1 container statuses recorded) I1105 20:43:43.897] Nov 5 20:43:43.894: INFO: Container kube-proxy ready: true, restart count 0 I1105 20:43:44.168] Nov 5 20:43:44.168: INFO: I1105 20:43:44.168] Latency metrics for node k8s-master-97435445-1 I1105 20:43:44.169] Nov 5 20:43:44.168: INFO: I1105 20:43:44.169] Logging node info for node k8s-master-97435445-2 I1105 20:43:44.255] Nov 5 20:43:44.252: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-97435445-2,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-97435445-2,UID:fa55c895-1a93-4743-885e-03fce84b5332,ResourceVersion:7246,Generation:0,CreationTimestamp:2019-11-05 20:09:19 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 0,kubernetes.azure.com/cluster: kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-97435445-2,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.5.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b/providers/Microsoft.Compute/virtualMachines/k8s-master-97435445-2,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 20:09:19 +0000 UTC 2019-11-05 20:09:19 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 20:42:45 +0000 UTC 2019-11-05 20:09:12 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 20:42:45 +0000 UTC 2019-11-05 20:09:12 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 20:42:45 +0000 UTC 2019-11-05 20:09:12 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 20:42:45 +0000 UTC 2019-11-05 20:09:19 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.255.7} {Hostname k8s-master-97435445-2}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:d19f6709138f496da43c095cec7b2360,SystemUUID:10116777-A8D7-CB4A-A8B7-7EBCF2F23B62,BootID:50cd14a9-5497-4972-bf60-cf59a0cbcb5f,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 20:43:44.255] Nov 5 20:43:44.253: INFO: I1105 20:43:44.256] Logging kubelet events for node k8s-master-97435445-2 I1105 20:43:44.342] Nov 5 20:43:44.342: INFO: I1105 20:43:44.342] Logging pods the kubelet thinks is on node k8s-master-97435445-2 I1105 20:43:44.431] Nov 5 20:43:44.430: INFO: azure-ip-masq-agent-sq7p5 started at 2019-11-05 20:09:19 +0000 UTC (0+1 container statuses recorded) I1105 20:43:44.431] Nov 5 20:43:44.430: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 20:43:44.431] Nov 5 20:43:44.430: INFO: kube-proxy-pz426 started at 2019-11-05 20:09:19 +0000 UTC (0+1 container statuses recorded) I1105 20:43:44.431] Nov 5 20:43:44.430: INFO: Container kube-proxy ready: true, restart count 0 I1105 20:43:44.431] Nov 5 20:43:44.430: INFO: kube-apiserver-k8s-master-97435445-2 started at 2019-11-05 20:09:05 +0000 UTC (0+1 container statuses recorded) I1105 20:43:44.431] Nov 5 20:43:44.430: INFO: Container kube-apiserver ready: true, restart count 0 I1105 20:43:44.432] Nov 5 20:43:44.430: INFO: kube-controller-manager-k8s-master-97435445-2 started at 2019-11-05 20:09:05 +0000 UTC (0+1 container statuses recorded) I1105 20:43:44.432] Nov 5 20:43:44.430: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 20:43:44.432] Nov 5 20:43:44.430: INFO: kube-scheduler-k8s-master-97435445-2 started at 2019-11-05 20:09:05 +0000 UTC (0+1 container statuses recorded) I1105 20:43:44.432] Nov 5 20:43:44.430: INFO: Container kube-scheduler ready: true, restart count 0 I1105 20:43:44.432] Nov 5 20:43:44.430: INFO: cloud-controller-manager-k8s-master-97435445-2 started at 2019-11-05 20:09:05 +0000 UTC (0+1 container statuses recorded) I1105 20:43:44.432] Nov 5 20:43:44.430: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 20:43:44.432] Nov 5 20:43:44.430: INFO: kube-addon-manager-k8s-master-97435445-2 started at 2019-11-05 20:09:05 +0000 UTC (0+1 container statuses recorded) I1105 20:43:44.433] Nov 5 20:43:44.430: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 20:43:44.701] Nov 5 20:43:44.701: INFO: I1105 20:43:44.701] Latency metrics for node k8s-master-97435445-2 I1105 20:43:44.701] Nov 5 20:43:44.701: INFO: I1105 20:43:44.702] Logging node info for node k8s-master-97435445-3 I1105 20:43:44.788] Nov 5 20:43:44.785: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-97435445-3,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-97435445-3,UID:8ffd1d0d-4487-4680-a2f6-e724387be637,ResourceVersion:7248,Generation:0,CreationTimestamp:2019-11-05 20:09:21 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 1,kubernetes.azure.com/cluster: kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-97435445-3,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.6.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b/providers/Microsoft.Compute/virtualMachines/k8s-master-97435445-3,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 20:09:39 +0000 UTC 2019-11-05 20:09:39 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 20:42:46 +0000 UTC 2019-11-05 20:09:18 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 20:42:46 +0000 UTC 2019-11-05 20:09:18 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 20:42:46 +0000 UTC 2019-11-05 20:09:18 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 20:42:46 +0000 UTC 2019-11-05 20:09:21 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.255.8} {Hostname k8s-master-97435445-3}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:266064173bfb4297898c6c25769e8b20,SystemUUID:E6E44241-5015-7740-9A81-A668A8427206,BootID:579d658c-b0ad-47f1-a993-1ccea2b55789,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 20:43:44.788] Nov 5 20:43:44.785: INFO: I1105 20:43:44.788] Logging kubelet events for node k8s-master-97435445-3 I1105 20:43:44.875] Nov 5 20:43:44.875: INFO: I1105 20:43:44.875] Logging pods the kubelet thinks is on node k8s-master-97435445-3 I1105 20:43:44.965] Nov 5 20:43:44.965: INFO: kube-proxy-k45g8 started at 2019-11-05 20:09:21 +0000 UTC (0+1 container statuses recorded) I1105 20:43:44.965] Nov 5 20:43:44.965: INFO: Container kube-proxy ready: true, restart count 0 I1105 20:43:44.965] Nov 5 20:43:44.965: INFO: cloud-controller-manager-k8s-master-97435445-3 started at 2019-11-05 20:09:04 +0000 UTC (0+1 container statuses recorded) I1105 20:43:44.966] Nov 5 20:43:44.965: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 20:43:44.966] Nov 5 20:43:44.965: INFO: kube-addon-manager-k8s-master-97435445-3 started at 2019-11-05 20:09:05 +0000 UTC (0+1 container statuses recorded) I1105 20:43:44.966] Nov 5 20:43:44.965: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 20:43:44.966] Nov 5 20:43:44.965: INFO: kube-apiserver-k8s-master-97435445-3 started at 2019-11-05 20:09:05 +0000 UTC (0+1 container statuses recorded) I1105 20:43:44.966] Nov 5 20:43:44.965: INFO: Container kube-apiserver ready: true, restart count 0 I1105 20:43:44.966] Nov 5 20:43:44.965: INFO: kube-controller-manager-k8s-master-97435445-3 started at 2019-11-05 20:09:05 +0000 UTC (0+1 container statuses recorded) I1105 20:43:44.967] Nov 5 20:43:44.965: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 20:43:44.967] Nov 5 20:43:44.965: INFO: kube-scheduler-k8s-master-97435445-3 started at 2019-11-05 20:09:05 +0000 UTC (0+1 container statuses recorded) I1105 20:43:44.967] Nov 5 20:43:44.965: INFO: Container kube-scheduler ready: true, restart count 0 I1105 20:43:44.967] Nov 5 20:43:44.965: INFO: azure-ip-masq-agent-747cl started at 2019-11-05 20:09:21 +0000 UTC (0+1 container statuses recorded) I1105 20:43:44.967] Nov 5 20:43:44.965: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 20:43:45.238] Nov 5 20:43:45.237: INFO: I1105 20:43:45.238] Latency metrics for node k8s-master-97435445-3 I1105 20:43:45.238] Nov 5 20:43:45.237: INFO: I1105 20:43:45.238] Logging node info for node k8s-master-97435445-4 I1105 20:43:45.325] Nov 5 20:43:45.322: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-97435445-4,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-97435445-4,UID:6d547b41-3f92-4cc4-a2d7-dda2071776a2,ResourceVersion:7292,Generation:0,CreationTimestamp:2019-11-05 20:08:36 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 1,kubernetes.azure.com/cluster: kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-97435445-4,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.3.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b/providers/Microsoft.Compute/virtualMachines/k8s-master-97435445-4,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 20:08:59 +0000 UTC 2019-11-05 20:08:59 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 20:43:00 +0000 UTC 2019-11-05 20:08:29 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 20:43:00 +0000 UTC 2019-11-05 20:08:29 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 20:43:00 +0000 UTC 2019-11-05 20:08:29 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 20:43:00 +0000 UTC 2019-11-05 20:08:36 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.255.9} {Hostname k8s-master-97435445-4}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:56b393c61eb34ba49ebda5279eeedb30,SystemUUID:FD239C23-618C-0049-8734-9AF3FDC9E3C1,BootID:f342ea44-e719-4513-807d-5def7ac606e2,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 20:43:45.325] Nov 5 20:43:45.323: INFO: I1105 20:43:45.325] Logging kubelet events for node k8s-master-97435445-4 I1105 20:43:45.412] Nov 5 20:43:45.412: INFO: I1105 20:43:45.412] Logging pods the kubelet thinks is on node k8s-master-97435445-4 I1105 20:43:45.503] Nov 5 20:43:45.503: INFO: kube-apiserver-k8s-master-97435445-4 started at 2019-11-05 20:08:23 +0000 UTC (0+1 container statuses recorded) I1105 20:43:45.503] Nov 5 20:43:45.503: INFO: Container kube-apiserver ready: true, restart count 0 I1105 20:43:45.504] Nov 5 20:43:45.503: INFO: kube-controller-manager-k8s-master-97435445-4 started at 2019-11-05 20:08:23 +0000 UTC (0+1 container statuses recorded) I1105 20:43:45.504] Nov 5 20:43:45.503: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 20:43:45.504] Nov 5 20:43:45.503: INFO: kube-scheduler-k8s-master-97435445-4 started at 2019-11-05 20:08:23 +0000 UTC (0+1 container statuses recorded) I1105 20:43:45.504] Nov 5 20:43:45.503: INFO: Container kube-scheduler ready: true, restart count 0 I1105 20:43:45.504] Nov 5 20:43:45.503: INFO: kube-proxy-vfbfr started at 2019-11-05 20:08:36 +0000 UTC (0+1 container statuses recorded) I1105 20:43:45.504] Nov 5 20:43:45.503: INFO: Container kube-proxy ready: true, restart count 0 I1105 20:43:45.505] Nov 5 20:43:45.503: INFO: azure-ip-masq-agent-kmkll started at 2019-11-05 20:08:36 +0000 UTC (0+1 container statuses recorded) I1105 20:43:45.505] Nov 5 20:43:45.503: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 20:43:45.505] Nov 5 20:43:45.503: INFO: cloud-controller-manager-k8s-master-97435445-4 started at 2019-11-05 20:08:46 +0000 UTC (0+1 container statuses recorded) I1105 20:43:45.505] Nov 5 20:43:45.503: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 20:43:45.505] Nov 5 20:43:45.503: INFO: kube-addon-manager-k8s-master-97435445-4 started at 2019-11-05 20:08:23 +0000 UTC (0+1 container statuses recorded) I1105 20:43:45.506] Nov 5 20:43:45.503: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 20:43:45.791] Nov 5 20:43:45.791: INFO: I1105 20:43:45.791] Latency metrics for node k8s-master-97435445-4 I1105 20:43:45.791] Nov 5 20:43:45.791: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready I1105 20:43:45.877] STEP: Destroying namespace "azuredisk-507" for this suite. I1105 20:44:00.144] Nov 5 20:44:00.144: INFO: Waiting up to 30s for server preferred namespaced resources to be successfully discovered I1105 20:44:03.335] Nov 5 20:44:03.335: INFO: namespace azuredisk-507 deletion completed in 17.457902742s I1105 20:44:03.335] I1105 20:44:03.335] • Failure [324.394 seconds] I1105 20:44:03.335] [azuredisk-csi-e2e] Dynamic Provisioning I1105 20:44:03.335] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:33 I1105 20:44:03.335] [multi-az] I1105 20:44:03.336] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:40 I1105 20:44:03.336] should create a deployment object, write and read to it, delete the pod and write and read to it again [It] I1105 20:44:03.336] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:193 I1105 20:44:03.336] I1105 20:44:03.336] Unexpected error: I1105 20:44:03.336] <*errors.errorString | 0xc0008c9170>: { I1105 20:44:03.337] s: "PersistentVolumeClaims [pvc-mkkrb] not all in phase Bound within 5m0s", I1105 20:44:03.337] } I1105 20:44:03.337] PersistentVolumeClaims [pvc-mkkrb] not all in phase Bound within 5m0s I1105 20:44:03.337] occurred I1105 20:44:03.337] I1105 20:44:03.337] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/testsuites/testsuites.go:278 I1105 20:44:03.337] ------------------------------ I1105 20:44:03.337] [azuredisk-csi-e2e] [single-az] Pre-Provisioned I1105 20:44:03.338] [env] should use a pre-provisioned volume and retain PV with reclaimPolicy "Retain" I1105 20:44:03.338] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/pre_provisioning.go:100 I1105 20:44:03.338] [BeforeEach] [azuredisk-csi-e2e] [single-az] Pre-Provisioned I1105 20:44:03.338] /go/src/sigs.k8s.io/azuredisk-csi-driver/vendor/k8s.io/kubernetes/test/e2e/framework/framework.go:150 I1105 20:44:03.338] STEP: Creating a kubernetes client I1105 20:44:03.338] Nov 5 20:44:03.335: INFO: >>> kubeConfig: /workspace/aks971561099/kubeconfig/kubeconfig.eastus2.json I1105 20:44:03.338] STEP: Building a namespace api object, basename azuredisk I1105 20:44:03.595] STEP: Binding the e2e-test-privileged-psp PodSecurityPolicy to the default service account in azuredisk-9275 I1105 20:44:03.869] STEP: Waiting for a default service account to be provisioned in namespace I1105 20:44:03.950] [BeforeEach] [azuredisk-csi-e2e] [single-az] Pre-Provisioned I1105 20:44:03.951] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/pre_provisioning.go:48 I1105 20:44:03.951] [It] [env] should use a pre-provisioned volume and retain PV with reclaimPolicy "Retain" I1105 20:44:03.951] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/pre_provisioning.go:100 I1105 20:44:06.268] STEP: Successfully provisioned AzureDisk volume: "/subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b/providers/Microsoft.Compute/disks/pre-provisioned-retain-reclaimPolicy" I1105 20:44:06.268] I1105 20:44:06.268] STEP: setting up the PV I1105 20:44:06.269] STEP: creating a PV I1105 20:44:06.363] STEP: setting up the PVC I1105 20:44:06.364] STEP: creating a PVC I1105 20:44:06.455] STEP: waiting for PVC to be in phase "Bound" I1105 20:44:06.455] Nov 5 20:44:06.455: INFO: Waiting up to 5m0s for PersistentVolumeClaims [pvc-znn4p] to have phase Bound I1105 20:44:06.540] Nov 5 20:44:06.540: INFO: PersistentVolumeClaim pvc-znn4p found and phase=Bound (84.96004ms) I1105 20:44:06.540] STEP: checking the PVC I1105 20:44:06.628] STEP: validating provisioned PV I1105 20:44:06.713] STEP: checking the PV I1105 20:44:06.713] [AfterEach] [azuredisk-csi-e2e] [single-az] Pre-Provisioned I1105 20:44:06.713] /go/src/sigs.k8s.io/azuredisk-csi-driver/vendor/k8s.io/kubernetes/test/e2e/framework/framework.go:151 I1105 20:44:06.713] STEP: Collecting events from namespace "azuredisk-9275". I1105 20:44:06.797] STEP: Found 0 events. I1105 20:44:06.881] Nov 5 20:44:06.881: INFO: POD NODE PHASE GRACE CONDITIONS I1105 20:44:06.882] Nov 5 20:44:06.881: INFO: I1105 20:44:07.131] Nov 5 20:44:07.131: INFO: I1105 20:44:07.131] Logging node info for node k8s-agentpool-97435445-vmss000000 I1105 20:44:07.218] Nov 5 20:44:07.215: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-agentpool-97435445-vmss000000,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-agentpool-97435445-vmss000000,UID:e955dc47-8c8f-4142-9241-3d409f32e42d,ResourceVersion:7407,Generation:0,CreationTimestamp:2019-11-05 20:07:34 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{agentpool: agentpool,beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_D4s_v3,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 0,kubernetes.azure.com/cluster: kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b,kubernetes.azure.com/role: agent,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-agentpool-97435445-vmss000000,kubernetes.io/os: linux,kubernetes.io/role: agent,node-role.kubernetes.io/agent: ,storageprofile: managed,storagetier: Premium_LRS,topology.disk.csi.azure.com/zone: 0,},Annotations:map[string]string{csi.volume.kubernetes.io/nodeid: {"disk.csi.azure.com":"k8s-agentpool-97435445-vmss000000"},node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.2.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool-97435445-vmss/virtualMachines/0,Unschedulable:false,Taints:[],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-csi-disk.csi.azure.com: {{8 0} {} 8 DecimalSI},cpu: {{4 0} {} 4 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{16797040640 0} {} BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-csi-disk.csi.azure.com: {{8 0} {} 8 DecimalSI},cpu: {{4 0} {} 4 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{16010608640 0} {} BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 20:08:01 +0000 UTC 2019-11-05 20:08:01 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 20:43:41 +0000 UTC 2019-11-05 20:07:27 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 20:43:41 +0000 UTC 2019-11-05 20:07:27 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 20:43:41 +0000 UTC 2019-11-05 20:07:27 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 20:43:41 +0000 UTC 2019-11-05 20:07:55 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.0.4} {Hostname k8s-agentpool-97435445-vmss000000}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:bfb55325809340228149d2d93f50df58,SystemUUID:CB2ED036-0439-F641-BC7F-640C3638C027,BootID:a917f4d1-549d-471d-b7dc-47ae1715d97d,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[k8sprowinternal.azurecr.io/azuredisk-csi@sha256:75d4ff5d7cd845d6e810c67796ecb037e852fb849ec1eb5b3194564008e6d1e0 k8sprowinternal.azurecr.io/azuredisk-csi:e2e-3e3ed045c6faa509dbd8d2890ee9f56aa2934887] 286930414} {[k8s.gcr.io/kubernetes-dashboard-amd64@sha256:0ae6b69432e78069c5ce2bcde0fe409c5c4d6f0f4d9cd50a17974fea38898747 k8s.gcr.io/kubernetes-dashboard-amd64:v1.10.1] 121711221} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/metrics-server-amd64@sha256:49a9f12f7067d11f42c803dbe61ed2c1299959ad85cb315b25ff7eef8e6b8892 k8s.gcr.io/metrics-server-amd64:v0.2.1] 42541759} {[k8s.gcr.io/coredns@sha256:e83beb5e43f8513fa735e77ffc5859640baea30a882a11cc75c4c3244a737d3c k8s.gcr.io/coredns:1.5.0] 42488424} {[mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume@sha256:4fd30d43947d4a54fc89ead7985beecfd3c9b2a93a0655a373b1608ab90bd5af mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume:v0.0.7] 22909487} {[quay.io/k8scsi/livenessprobe@sha256:8c9c8a0f330acc2e558ae6624309447d02522e76aa35af23fa19b88abf4381cb quay.io/k8scsi/livenessprobe:v1.0.2] 17415381} {[quay.io/k8scsi/csi-node-driver-registrar@sha256:13daf82fb99e951a4bff8ae5fc7c17c3a8fe7130be6400990d8f6076c32d4599 quay.io/k8scsi/csi-node-driver-registrar:v1.1.0] 15815995} {[mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume@sha256:23d8c6033f02a1ecad05127ebdc931bb871264228661bc122704b0974e4d9fdd mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume:1.0.8] 1159025} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 20:44:07.219] Nov 5 20:44:07.216: INFO: I1105 20:44:07.219] Logging kubelet events for node k8s-agentpool-97435445-vmss000000 I1105 20:44:07.304] Nov 5 20:44:07.303: INFO: I1105 20:44:07.304] Logging pods the kubelet thinks is on node k8s-agentpool-97435445-vmss000000 I1105 20:44:07.393] Nov 5 20:44:07.393: INFO: kubernetes-dashboard-66dd8b8df7-t6j4l started at 2019-11-05 20:08:01 +0000 UTC (0+1 container statuses recorded) I1105 20:44:07.393] Nov 5 20:44:07.393: INFO: Container kubernetes-dashboard ready: true, restart count 0 I1105 20:44:07.394] Nov 5 20:44:07.393: INFO: csi-azuredisk-node-5wwhz started at 2019-11-05 20:16:09 +0000 UTC (0+3 container statuses recorded) I1105 20:44:07.394] Nov 5 20:44:07.393: INFO: Container azuredisk ready: true, restart count 0 I1105 20:44:07.394] Nov 5 20:44:07.393: INFO: Container liveness-probe ready: true, restart count 0 I1105 20:44:07.394] Nov 5 20:44:07.393: INFO: Container node-driver-registrar ready: true, restart count 0 I1105 20:44:07.394] Nov 5 20:44:07.393: INFO: blobfuse-flexvol-installer-p7hdb started at 2019-11-05 20:08:01 +0000 UTC (0+1 container statuses recorded) I1105 20:44:07.394] Nov 5 20:44:07.393: INFO: Container blobfuse-flexvol-installer ready: true, restart count 0 I1105 20:44:07.394] Nov 5 20:44:07.393: INFO: coredns-7f68dcdbdb-zh8g5 started at 2019-11-05 20:08:01 +0000 UTC (0+1 container statuses recorded) I1105 20:44:07.395] Nov 5 20:44:07.393: INFO: Container coredns ready: true, restart count 0 I1105 20:44:07.395] Nov 5 20:44:07.393: INFO: azure-ip-masq-agent-m7tv9 started at 2019-11-05 20:08:01 +0000 UTC (0+1 container statuses recorded) I1105 20:44:07.395] Nov 5 20:44:07.393: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 20:44:07.395] Nov 5 20:44:07.393: INFO: keyvault-flexvolume-smspt started at 2019-11-05 20:08:01 +0000 UTC (0+1 container statuses recorded) I1105 20:44:07.395] Nov 5 20:44:07.393: INFO: Container keyvault-flexvolume ready: true, restart count 0 I1105 20:44:07.395] Nov 5 20:44:07.393: INFO: metrics-server-864ffbc5c-wl84z started at 2019-11-05 20:08:01 +0000 UTC (0+1 container statuses recorded) I1105 20:44:07.395] Nov 5 20:44:07.393: INFO: Container metrics-server ready: true, restart count 0 I1105 20:44:07.396] Nov 5 20:44:07.393: INFO: kube-proxy-8mt7f started at 2019-11-05 20:08:01 +0000 UTC (0+1 container statuses recorded) I1105 20:44:07.396] Nov 5 20:44:07.393: INFO: Container kube-proxy ready: true, restart count 0 I1105 20:44:07.659] Nov 5 20:44:07.659: INFO: I1105 20:44:07.659] Latency metrics for node k8s-agentpool-97435445-vmss000000 I1105 20:44:07.659] Nov 5 20:44:07.659: INFO: I1105 20:44:07.660] Logging node info for node k8s-agentpool-97435445-vmss000001 I1105 20:44:07.747] Nov 5 20:44:07.743: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-agentpool-97435445-vmss000001,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-agentpool-97435445-vmss000001,UID:75d3cc50-ca6d-45e0-a42a-6090e8d537dd,ResourceVersion:7405,Generation:0,CreationTimestamp:2019-11-05 20:07:34 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{agentpool: agentpool,beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_D4s_v3,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 1,kubernetes.azure.com/cluster: kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b,kubernetes.azure.com/role: agent,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-agentpool-97435445-vmss000001,kubernetes.io/os: linux,kubernetes.io/role: agent,node-role.kubernetes.io/agent: ,storageprofile: managed,storagetier: Premium_LRS,topology.disk.csi.azure.com/zone: 1,},Annotations:map[string]string{csi.volume.kubernetes.io/nodeid: {"disk.csi.azure.com":"k8s-agentpool-97435445-vmss000001"},node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.0.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool-97435445-vmss/virtualMachines/1,Unschedulable:false,Taints:[],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-csi-disk.csi.azure.com: {{8 0} {} 8 DecimalSI},cpu: {{4 0} {} 4 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{16797044736 0} {} 16403364Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-csi-disk.csi.azure.com: {{8 0} {} 8 DecimalSI},cpu: {{4 0} {} 4 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{16010612736 0} {} 15635364Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 20:08:25 +0000 UTC 2019-11-05 20:08:25 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 20:43:40 +0000 UTC 2019-11-05 20:07:26 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 20:43:40 +0000 UTC 2019-11-05 20:07:26 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 20:43:40 +0000 UTC 2019-11-05 20:07:26 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 20:43:40 +0000 UTC 2019-11-05 20:07:55 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.0.5} {Hostname k8s-agentpool-97435445-vmss000001}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:e78def54b2f545c6b94d88bbba523c60,SystemUUID:40AE8188-86AB-D848-84A4-D9254E1258BC,BootID:8587ec06-f4f4-43c1-86cd-06c295b05b63,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[k8sprowinternal.azurecr.io/azuredisk-csi@sha256:75d4ff5d7cd845d6e810c67796ecb037e852fb849ec1eb5b3194564008e6d1e0 k8sprowinternal.azurecr.io/azuredisk-csi:e2e-3e3ed045c6faa509dbd8d2890ee9f56aa2934887] 286930414} {[gcr.io/kubernetes-helm/tiller@sha256:f6d8f4ab9ba993b5f5b60a6edafe86352eabe474ffeb84cb6c79b8866dce45d1 gcr.io/kubernetes-helm/tiller:v2.11.0] 71821984} {[quay.io/k8scsi/csi-provisioner@sha256:d657c839dce87324fe2b677302913f9386f885f8746be7bea0ced5b0844e3433 quay.io/k8scsi/csi-provisioner:v1.3.1] 50596998} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[quay.io/k8scsi/csi-attacher@sha256:26fccd7a99d973845df1193b46ebdcc6ab8dc5f6e6be319750c471fce1742d13 quay.io/k8scsi/csi-attacher:v1.2.0] 46226754} {[quay.io/k8scsi/csi-cluster-driver-registrar@sha256:fafd75ae5442f192cfa8c2e792903aee30d5884b62e802e4464b0a895d21e3ef quay.io/k8scsi/csi-cluster-driver-registrar:v1.0.1] 45874691} {[quay.io/k8scsi/csi-snapshotter@sha256:a49e0da1af6f2bf717e41ba1eee8b5e6a1cbd66a709dd92cc43fe475fe2589eb quay.io/k8scsi/csi-snapshotter:v1.1.0] 44325352} {[mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume@sha256:4fd30d43947d4a54fc89ead7985beecfd3c9b2a93a0655a373b1608ab90bd5af mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume:v0.0.7] 22909487} {[quay.io/k8scsi/livenessprobe@sha256:8c9c8a0f330acc2e558ae6624309447d02522e76aa35af23fa19b88abf4381cb quay.io/k8scsi/livenessprobe:v1.0.2] 17415381} {[quay.io/k8scsi/csi-node-driver-registrar@sha256:13daf82fb99e951a4bff8ae5fc7c17c3a8fe7130be6400990d8f6076c32d4599 quay.io/k8scsi/csi-node-driver-registrar:v1.1.0] 15815995} {[quay.io/k8scsi/livenessprobe@sha256:dde617756e0f602adc566ab71fd885f1dad451ad3fb063ac991c95a2ff47aea5 quay.io/k8scsi/livenessprobe:v1.1.0] 14967303} {[mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume@sha256:23d8c6033f02a1ecad05127ebdc931bb871264228661bc122704b0974e4d9fdd mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume:1.0.8] 1159025} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 20:44:07.747] Nov 5 20:44:07.743: INFO: I1105 20:44:07.747] Logging kubelet events for node k8s-agentpool-97435445-vmss000001 I1105 20:44:07.832] Nov 5 20:44:07.832: INFO: I1105 20:44:07.832] Logging pods the kubelet thinks is on node k8s-agentpool-97435445-vmss000001 I1105 20:44:07.920] Nov 5 20:44:07.920: INFO: csi-azuredisk-controller-6f6587bfc6-62nrt started at 2019-11-05 20:16:09 +0000 UTC (0+6 container statuses recorded) I1105 20:44:07.920] Nov 5 20:44:07.920: INFO: Container azuredisk ready: true, restart count 0 I1105 20:44:07.921] Nov 5 20:44:07.920: INFO: Container cluster-driver-registrar ready: true, restart count 0 I1105 20:44:07.921] Nov 5 20:44:07.920: INFO: Container csi-attacher ready: true, restart count 0 I1105 20:44:07.921] Nov 5 20:44:07.920: INFO: Container csi-provisioner ready: true, restart count 0 I1105 20:44:07.921] Nov 5 20:44:07.920: INFO: Container csi-snapshotter ready: true, restart count 0 I1105 20:44:07.921] Nov 5 20:44:07.920: INFO: Container liveness-probe ready: true, restart count 0 I1105 20:44:07.921] Nov 5 20:44:07.920: INFO: blobfuse-flexvol-installer-j4s7t started at 2019-11-05 20:07:59 +0000 UTC (0+1 container statuses recorded) I1105 20:44:07.921] Nov 5 20:44:07.920: INFO: Container blobfuse-flexvol-installer ready: true, restart count 0 I1105 20:44:07.922] Nov 5 20:44:07.920: INFO: azure-ip-masq-agent-gdhcq started at 2019-11-05 20:07:59 +0000 UTC (0+1 container statuses recorded) I1105 20:44:07.922] Nov 5 20:44:07.920: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 20:44:07.922] Nov 5 20:44:07.920: INFO: keyvault-flexvolume-6bffx started at 2019-11-05 20:07:59 +0000 UTC (0+1 container statuses recorded) I1105 20:44:07.922] Nov 5 20:44:07.920: INFO: Container keyvault-flexvolume ready: true, restart count 0 I1105 20:44:07.922] Nov 5 20:44:07.920: INFO: kube-proxy-kjrpb started at 2019-11-05 20:08:00 +0000 UTC (0+1 container statuses recorded) I1105 20:44:07.922] Nov 5 20:44:07.920: INFO: Container kube-proxy ready: true, restart count 0 I1105 20:44:07.922] Nov 5 20:44:07.920: INFO: tiller-deploy-54c96cb5df-pt7x6 started at 2019-11-05 20:08:00 +0000 UTC (0+1 container statuses recorded) I1105 20:44:07.923] Nov 5 20:44:07.920: INFO: Container tiller ready: true, restart count 0 I1105 20:44:07.923] Nov 5 20:44:07.920: INFO: csi-azuredisk-node-q2dmt started at 2019-11-05 20:16:09 +0000 UTC (0+3 container statuses recorded) I1105 20:44:07.923] Nov 5 20:44:07.920: INFO: Container azuredisk ready: true, restart count 0 I1105 20:44:07.923] Nov 5 20:44:07.920: INFO: Container liveness-probe ready: true, restart count 0 I1105 20:44:07.923] Nov 5 20:44:07.920: INFO: Container node-driver-registrar ready: true, restart count 0 I1105 20:44:08.188] Nov 5 20:44:08.188: INFO: I1105 20:44:08.188] Latency metrics for node k8s-agentpool-97435445-vmss000001 I1105 20:44:08.188] Nov 5 20:44:08.188: INFO: I1105 20:44:08.188] Logging node info for node k8s-master-97435445-0 I1105 20:44:08.275] Nov 5 20:44:08.272: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-97435445-0,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-97435445-0,UID:df3ae36f-961c-4729-a8f5-34cda54f4ff9,ResourceVersion:7476,Generation:0,CreationTimestamp:2019-11-05 20:08:36 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 2,kubernetes.azure.com/cluster: kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-97435445-0,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.4.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b/providers/Microsoft.Compute/virtualMachines/k8s-master-97435445-0,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 20:08:46 +0000 UTC 2019-11-05 20:08:46 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 20:44:01 +0000 UTC 2019-11-05 20:08:28 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 20:44:01 +0000 UTC 2019-11-05 20:08:28 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 20:44:01 +0000 UTC 2019-11-05 20:08:28 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 20:44:01 +0000 UTC 2019-11-05 20:08:36 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.255.5} {Hostname k8s-master-97435445-0}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:3c381865192341c2868672e8626cef7d,SystemUUID:F3D259D3-7EB8-B844-A1F0-F954DCD0D76B,BootID:4948329e-cf57-48c3-aa83-5297bef70370,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 20:44:08.275] Nov 5 20:44:08.272: INFO: I1105 20:44:08.275] Logging kubelet events for node k8s-master-97435445-0 I1105 20:44:08.361] Nov 5 20:44:08.361: INFO: I1105 20:44:08.362] Logging pods the kubelet thinks is on node k8s-master-97435445-0 I1105 20:44:08.449] Nov 5 20:44:08.449: INFO: kube-controller-manager-k8s-master-97435445-0 started at 2019-11-05 20:08:22 +0000 UTC (0+1 container statuses recorded) I1105 20:44:08.449] Nov 5 20:44:08.449: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 20:44:08.450] Nov 5 20:44:08.449: INFO: kube-scheduler-k8s-master-97435445-0 started at 2019-11-05 20:08:22 +0000 UTC (0+1 container statuses recorded) I1105 20:44:08.450] Nov 5 20:44:08.449: INFO: Container kube-scheduler ready: true, restart count 0 I1105 20:44:08.450] Nov 5 20:44:08.449: INFO: azure-ip-masq-agent-q6dcl started at 2019-11-05 20:08:38 +0000 UTC (0+1 container statuses recorded) I1105 20:44:08.450] Nov 5 20:44:08.449: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 20:44:08.450] Nov 5 20:44:08.449: INFO: kube-proxy-5wnd2 started at 2019-11-05 20:08:38 +0000 UTC (0+1 container statuses recorded) I1105 20:44:08.450] Nov 5 20:44:08.449: INFO: Container kube-proxy ready: true, restart count 0 I1105 20:44:08.450] Nov 5 20:44:08.449: INFO: cloud-controller-manager-k8s-master-97435445-0 started at 2019-11-05 20:08:22 +0000 UTC (0+1 container statuses recorded) I1105 20:44:08.451] Nov 5 20:44:08.449: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 20:44:08.451] Nov 5 20:44:08.449: INFO: kube-addon-manager-k8s-master-97435445-0 started at 2019-11-05 20:08:48 +0000 UTC (0+1 container statuses recorded) I1105 20:44:08.451] Nov 5 20:44:08.449: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 20:44:08.451] Nov 5 20:44:08.449: INFO: kube-apiserver-k8s-master-97435445-0 started at 2019-11-05 20:08:22 +0000 UTC (0+1 container statuses recorded) I1105 20:44:08.451] Nov 5 20:44:08.449: INFO: Container kube-apiserver ready: true, restart count 0 I1105 20:44:08.731] Nov 5 20:44:08.731: INFO: I1105 20:44:08.732] Latency metrics for node k8s-master-97435445-0 I1105 20:44:08.732] Nov 5 20:44:08.731: INFO: I1105 20:44:08.732] Logging node info for node k8s-master-97435445-1 I1105 20:44:08.818] Nov 5 20:44:08.816: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-97435445-1,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-97435445-1,UID:05534a67-d493-4ab5-9b0f-5ff5490e234f,ResourceVersion:7372,Generation:0,CreationTimestamp:2019-11-05 20:07:34 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 0,kubernetes.azure.com/cluster: kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-97435445-1,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.1.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b/providers/Microsoft.Compute/virtualMachines/k8s-master-97435445-1,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 20:08:26 +0000 UTC 2019-11-05 20:08:26 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 20:43:29 +0000 UTC 2019-11-05 20:07:28 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 20:43:29 +0000 UTC 2019-11-05 20:07:28 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 20:43:29 +0000 UTC 2019-11-05 20:07:28 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 20:43:29 +0000 UTC 2019-11-05 20:07:55 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{Hostname k8s-master-97435445-1} {InternalIP 10.240.255.6}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:079b605103b74acabac18fd2ea63368a,SystemUUID:38220EEC-BEFF-4C46-AA05-5A214AAFDC28,BootID:b1b45df0-d925-4bcc-9830-0202df36ebb4,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 20:44:08.818] Nov 5 20:44:08.816: INFO: I1105 20:44:08.819] Logging kubelet events for node k8s-master-97435445-1 I1105 20:44:08.904] Nov 5 20:44:08.904: INFO: I1105 20:44:08.905] Logging pods the kubelet thinks is on node k8s-master-97435445-1 I1105 20:44:08.993] Nov 5 20:44:08.993: INFO: azure-ip-masq-agent-n29bm started at 2019-11-05 20:07:59 +0000 UTC (0+1 container statuses recorded) I1105 20:44:08.993] Nov 5 20:44:08.993: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 20:44:08.993] Nov 5 20:44:08.993: INFO: kube-proxy-h4s2c started at 2019-11-05 20:08:00 +0000 UTC (0+1 container statuses recorded) I1105 20:44:08.993] Nov 5 20:44:08.993: INFO: Container kube-proxy ready: true, restart count 0 I1105 20:44:08.994] Nov 5 20:44:08.993: INFO: cloud-controller-manager-k8s-master-97435445-1 started at 2019-11-05 20:07:22 +0000 UTC (0+1 container statuses recorded) I1105 20:44:08.994] Nov 5 20:44:08.993: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 20:44:08.994] Nov 5 20:44:08.993: INFO: kube-addon-manager-k8s-master-97435445-1 started at 2019-11-05 20:07:22 +0000 UTC (0+1 container statuses recorded) I1105 20:44:08.994] Nov 5 20:44:08.993: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 20:44:08.994] Nov 5 20:44:08.993: INFO: kube-apiserver-k8s-master-97435445-1 started at 2019-11-05 20:07:22 +0000 UTC (0+1 container statuses recorded) I1105 20:44:08.994] Nov 5 20:44:08.993: INFO: Container kube-apiserver ready: true, restart count 0 I1105 20:44:08.995] Nov 5 20:44:08.993: INFO: kube-controller-manager-k8s-master-97435445-1 started at 2019-11-05 20:07:22 +0000 UTC (0+1 container statuses recorded) I1105 20:44:08.995] Nov 5 20:44:08.993: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 20:44:08.995] Nov 5 20:44:08.993: INFO: kube-scheduler-k8s-master-97435445-1 started at 2019-11-05 20:07:22 +0000 UTC (0+1 container statuses recorded) I1105 20:44:08.995] Nov 5 20:44:08.993: INFO: Container kube-scheduler ready: true, restart count 0 I1105 20:44:09.267] Nov 5 20:44:09.267: INFO: I1105 20:44:09.267] Latency metrics for node k8s-master-97435445-1 I1105 20:44:09.267] Nov 5 20:44:09.267: INFO: I1105 20:44:09.267] Logging node info for node k8s-master-97435445-2 I1105 20:44:09.354] Nov 5 20:44:09.351: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-97435445-2,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-97435445-2,UID:fa55c895-1a93-4743-885e-03fce84b5332,ResourceVersion:7419,Generation:0,CreationTimestamp:2019-11-05 20:09:19 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 0,kubernetes.azure.com/cluster: kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-97435445-2,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.5.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b/providers/Microsoft.Compute/virtualMachines/k8s-master-97435445-2,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 20:09:19 +0000 UTC 2019-11-05 20:09:19 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 20:43:45 +0000 UTC 2019-11-05 20:09:12 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 20:43:45 +0000 UTC 2019-11-05 20:09:12 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 20:43:45 +0000 UTC 2019-11-05 20:09:12 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 20:43:45 +0000 UTC 2019-11-05 20:09:19 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.255.7} {Hostname k8s-master-97435445-2}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:d19f6709138f496da43c095cec7b2360,SystemUUID:10116777-A8D7-CB4A-A8B7-7EBCF2F23B62,BootID:50cd14a9-5497-4972-bf60-cf59a0cbcb5f,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 20:44:09.354] Nov 5 20:44:09.351: INFO: I1105 20:44:09.354] Logging kubelet events for node k8s-master-97435445-2 I1105 20:44:09.440] Nov 5 20:44:09.440: INFO: I1105 20:44:09.441] Logging pods the kubelet thinks is on node k8s-master-97435445-2 I1105 20:44:09.527] Nov 5 20:44:09.527: INFO: kube-apiserver-k8s-master-97435445-2 started at 2019-11-05 20:09:05 +0000 UTC (0+1 container statuses recorded) I1105 20:44:09.527] Nov 5 20:44:09.527: INFO: Container kube-apiserver ready: true, restart count 0 I1105 20:44:09.528] Nov 5 20:44:09.527: INFO: kube-controller-manager-k8s-master-97435445-2 started at 2019-11-05 20:09:05 +0000 UTC (0+1 container statuses recorded) I1105 20:44:09.528] Nov 5 20:44:09.527: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 20:44:09.528] Nov 5 20:44:09.527: INFO: kube-scheduler-k8s-master-97435445-2 started at 2019-11-05 20:09:05 +0000 UTC (0+1 container statuses recorded) I1105 20:44:09.528] Nov 5 20:44:09.527: INFO: Container kube-scheduler ready: true, restart count 0 I1105 20:44:09.528] Nov 5 20:44:09.527: INFO: cloud-controller-manager-k8s-master-97435445-2 started at 2019-11-05 20:09:05 +0000 UTC (0+1 container statuses recorded) I1105 20:44:09.528] Nov 5 20:44:09.527: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 20:44:09.528] Nov 5 20:44:09.527: INFO: kube-addon-manager-k8s-master-97435445-2 started at 2019-11-05 20:09:05 +0000 UTC (0+1 container statuses recorded) I1105 20:44:09.529] Nov 5 20:44:09.527: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 20:44:09.529] Nov 5 20:44:09.527: INFO: azure-ip-masq-agent-sq7p5 started at 2019-11-05 20:09:19 +0000 UTC (0+1 container statuses recorded) I1105 20:44:09.529] Nov 5 20:44:09.527: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 20:44:09.529] Nov 5 20:44:09.527: INFO: kube-proxy-pz426 started at 2019-11-05 20:09:19 +0000 UTC (0+1 container statuses recorded) I1105 20:44:09.529] Nov 5 20:44:09.527: INFO: Container kube-proxy ready: true, restart count 0 I1105 20:44:09.799] Nov 5 20:44:09.799: INFO: I1105 20:44:09.800] Latency metrics for node k8s-master-97435445-2 I1105 20:44:09.800] Nov 5 20:44:09.799: INFO: I1105 20:44:09.800] Logging node info for node k8s-master-97435445-3 I1105 20:44:09.887] Nov 5 20:44:09.884: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-97435445-3,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-97435445-3,UID:8ffd1d0d-4487-4680-a2f6-e724387be637,ResourceVersion:7424,Generation:0,CreationTimestamp:2019-11-05 20:09:21 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 1,kubernetes.azure.com/cluster: kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-97435445-3,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.6.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b/providers/Microsoft.Compute/virtualMachines/k8s-master-97435445-3,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 20:09:39 +0000 UTC 2019-11-05 20:09:39 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 20:43:46 +0000 UTC 2019-11-05 20:09:18 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 20:43:46 +0000 UTC 2019-11-05 20:09:18 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 20:43:46 +0000 UTC 2019-11-05 20:09:18 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 20:43:46 +0000 UTC 2019-11-05 20:09:21 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.255.8} {Hostname k8s-master-97435445-3}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:266064173bfb4297898c6c25769e8b20,SystemUUID:E6E44241-5015-7740-9A81-A668A8427206,BootID:579d658c-b0ad-47f1-a993-1ccea2b55789,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 20:44:09.887] Nov 5 20:44:09.884: INFO: I1105 20:44:09.887] Logging kubelet events for node k8s-master-97435445-3 I1105 20:44:09.973] Nov 5 20:44:09.973: INFO: I1105 20:44:09.974] Logging pods the kubelet thinks is on node k8s-master-97435445-3 I1105 20:44:10.063] Nov 5 20:44:10.062: INFO: kube-proxy-k45g8 started at 2019-11-05 20:09:21 +0000 UTC (0+1 container statuses recorded) I1105 20:44:10.063] Nov 5 20:44:10.062: INFO: Container kube-proxy ready: true, restart count 0 I1105 20:44:10.063] Nov 5 20:44:10.062: INFO: cloud-controller-manager-k8s-master-97435445-3 started at 2019-11-05 20:09:04 +0000 UTC (0+1 container statuses recorded) I1105 20:44:10.064] Nov 5 20:44:10.062: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 20:44:10.064] Nov 5 20:44:10.062: INFO: kube-addon-manager-k8s-master-97435445-3 started at 2019-11-05 20:09:05 +0000 UTC (0+1 container statuses recorded) I1105 20:44:10.064] Nov 5 20:44:10.062: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 20:44:10.064] Nov 5 20:44:10.062: INFO: kube-apiserver-k8s-master-97435445-3 started at 2019-11-05 20:09:05 +0000 UTC (0+1 container statuses recorded) I1105 20:44:10.064] Nov 5 20:44:10.062: INFO: Container kube-apiserver ready: true, restart count 0 I1105 20:44:10.065] Nov 5 20:44:10.062: INFO: kube-controller-manager-k8s-master-97435445-3 started at 2019-11-05 20:09:05 +0000 UTC (0+1 container statuses recorded) I1105 20:44:10.065] Nov 5 20:44:10.062: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 20:44:10.065] Nov 5 20:44:10.062: INFO: kube-scheduler-k8s-master-97435445-3 started at 2019-11-05 20:09:05 +0000 UTC (0+1 container statuses recorded) I1105 20:44:10.065] Nov 5 20:44:10.062: INFO: Container kube-scheduler ready: true, restart count 0 I1105 20:44:10.065] Nov 5 20:44:10.062: INFO: azure-ip-masq-agent-747cl started at 2019-11-05 20:09:21 +0000 UTC (0+1 container statuses recorded) I1105 20:44:10.065] Nov 5 20:44:10.062: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 20:44:10.346] Nov 5 20:44:10.346: INFO: I1105 20:44:10.346] Latency metrics for node k8s-master-97435445-3 I1105 20:44:10.346] Nov 5 20:44:10.346: INFO: I1105 20:44:10.347] Logging node info for node k8s-master-97435445-4 I1105 20:44:10.432] Nov 5 20:44:10.430: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-97435445-4,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-97435445-4,UID:6d547b41-3f92-4cc4-a2d7-dda2071776a2,ResourceVersion:7475,Generation:0,CreationTimestamp:2019-11-05 20:08:36 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 1,kubernetes.azure.com/cluster: kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-97435445-4,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.3.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b/providers/Microsoft.Compute/virtualMachines/k8s-master-97435445-4,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 20:08:59 +0000 UTC 2019-11-05 20:08:59 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 20:44:00 +0000 UTC 2019-11-05 20:08:29 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 20:44:00 +0000 UTC 2019-11-05 20:08:29 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 20:44:00 +0000 UTC 2019-11-05 20:08:29 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 20:44:00 +0000 UTC 2019-11-05 20:08:36 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.255.9} {Hostname k8s-master-97435445-4}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:56b393c61eb34ba49ebda5279eeedb30,SystemUUID:FD239C23-618C-0049-8734-9AF3FDC9E3C1,BootID:f342ea44-e719-4513-807d-5def7ac606e2,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 20:44:10.433] Nov 5 20:44:10.430: INFO: I1105 20:44:10.433] Logging kubelet events for node k8s-master-97435445-4 I1105 20:44:10.519] Nov 5 20:44:10.519: INFO: I1105 20:44:10.519] Logging pods the kubelet thinks is on node k8s-master-97435445-4 I1105 20:44:10.628] Nov 5 20:44:10.628: INFO: kube-apiserver-k8s-master-97435445-4 started at 2019-11-05 20:08:23 +0000 UTC (0+1 container statuses recorded) I1105 20:44:10.629] Nov 5 20:44:10.628: INFO: Container kube-apiserver ready: true, restart count 0 I1105 20:44:10.629] Nov 5 20:44:10.628: INFO: kube-controller-manager-k8s-master-97435445-4 started at 2019-11-05 20:08:23 +0000 UTC (0+1 container statuses recorded) I1105 20:44:10.629] Nov 5 20:44:10.628: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 20:44:10.629] Nov 5 20:44:10.628: INFO: kube-scheduler-k8s-master-97435445-4 started at 2019-11-05 20:08:23 +0000 UTC (0+1 container statuses recorded) I1105 20:44:10.629] Nov 5 20:44:10.628: INFO: Container kube-scheduler ready: true, restart count 0 I1105 20:44:10.629] Nov 5 20:44:10.628: INFO: kube-proxy-vfbfr started at 2019-11-05 20:08:36 +0000 UTC (0+1 container statuses recorded) I1105 20:44:10.630] Nov 5 20:44:10.628: INFO: Container kube-proxy ready: true, restart count 0 I1105 20:44:10.630] Nov 5 20:44:10.628: INFO: azure-ip-masq-agent-kmkll started at 2019-11-05 20:08:36 +0000 UTC (0+1 container statuses recorded) I1105 20:44:10.630] Nov 5 20:44:10.628: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 20:44:10.630] Nov 5 20:44:10.628: INFO: cloud-controller-manager-k8s-master-97435445-4 started at 2019-11-05 20:08:46 +0000 UTC (0+1 container statuses recorded) I1105 20:44:10.630] Nov 5 20:44:10.628: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 20:44:10.630] Nov 5 20:44:10.628: INFO: kube-addon-manager-k8s-master-97435445-4 started at 2019-11-05 20:08:23 +0000 UTC (0+1 container statuses recorded) I1105 20:44:10.630] Nov 5 20:44:10.628: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 20:44:10.901] Nov 5 20:44:10.901: INFO: I1105 20:44:10.901] Latency metrics for node k8s-master-97435445-4 I1105 20:44:10.901] Nov 5 20:44:10.901: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready I1105 20:44:10.988] STEP: Destroying namespace "azuredisk-9275" for this suite. I1105 20:44:25.249] Nov 5 20:44:25.249: INFO: Waiting up to 30s for server preferred namespaced resources to be successfully discovered I1105 20:44:28.411] Nov 5 20:44:28.411: INFO: namespace azuredisk-9275 deletion completed in 17.422891805s I1105 20:44:28.411] [AfterEach] [azuredisk-csi-e2e] [single-az] Pre-Provisioned I1105 20:44:28.411] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/pre_provisioning.go:54 I1105 20:44:33.878] I1105 20:44:33.878] •! Panic [30.543 seconds] I1105 20:44:33.879] [azuredisk-csi-e2e] [single-az] Pre-Provisioned I1105 20:44:33.879] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/pre_provisioning.go:36 I1105 20:44:33.879] [env] should use a pre-provisioned volume and retain PV with reclaimPolicy "Retain" [It] I1105 20:44:33.879] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/pre_provisioning.go:100 I1105 20:44:33.879] I1105 20:44:33.879] Test Panicked I1105 20:44:33.879] runtime error: invalid memory address or nil pointer dereference I1105 20:44:33.879] /usr/local/go/src/runtime/panic.go:82 I1105 20:44:33.880] I1105 20:44:33.880] Full Stack Trace I1105 20:44:33.880] /usr/local/go/src/runtime/panic.go:522 +0x1b5 I1105 20:44:33.880] sigs.k8s.io/azuredisk-csi-driver/test/e2e/testsuites.(*TestPersistentVolumeClaim).Cleanup(0xc0010761e0) I1105 20:44:33.880] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/testsuites/testsuites.go:313 +0x3b I1105 20:44:33.880] sigs.k8s.io/azuredisk-csi-driver/test/e2e/testsuites.(*PreProvisionedReclaimPolicyTest).Run(0xc0009ad4e0, 0x20e1540, 0xc000d66140, 0xc000edbce0) I1105 20:44:33.881] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/testsuites/pre_provisioned_reclaim_policy_tester.go:37 +0x10f I1105 20:44:33.881] sigs.k8s.io/azuredisk-csi-driver/test/e2e.glob..func2.4() I1105 20:44:33.881] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/pre_provisioning.go:123 +0x529 I1105 20:44:33.881] sigs.k8s.io/azuredisk-csi-driver/vendor/github.com/onsi/ginkgo/internal/leafnodes.(*runner).runSync(0xc0002a6ae0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) I1105 20:44:33.881] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/suite_test.go:132 +0xa4 I1105 20:44:33.882] testing.tRunner(0xc0002cb100, 0x1e2da58) I1105 20:44:33.882] /usr/local/go/src/testing/testing.go:865 +0xc0 I1105 20:44:33.882] created by testing.(*T).Run I1105 20:44:33.882] /usr/local/go/src/testing/testing.go:916 +0x35a I1105 20:44:33.882] I1105 20:44:33.882] ------------------------------ I1105 20:44:33.883] [azuredisk-csi-e2e] Dynamic Provisioning [multi-az] I1105 20:44:33.883] should create multiple PV objects, bind to PVCs and attach all to different pods on the same node I1105 20:44:33.883] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:151 I1105 20:44:33.883] [BeforeEach] [multi-az] I1105 20:44:33.883] /go/src/sigs.k8s.io/azuredisk-csi-driver/vendor/k8s.io/kubernetes/test/e2e/framework/framework.go:150 I1105 20:44:33.883] STEP: Creating a kubernetes client I1105 20:44:33.883] Nov 5 20:44:33.878: INFO: >>> kubeConfig: /workspace/aks971561099/kubeconfig/kubeconfig.eastus2.json I1105 20:44:33.884] STEP: Building a namespace api object, basename azuredisk I1105 20:44:34.136] STEP: Binding the e2e-test-privileged-psp PodSecurityPolicy to the default service account in azuredisk-6880 I1105 20:44:34.411] STEP: Waiting for a default service account to be provisioned in namespace I1105 20:44:34.499] [BeforeEach] [multi-az] I1105 20:44:34.500] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:58 I1105 20:44:34.500] [It] should create multiple PV objects, bind to PVCs and attach all to different pods on the same node I1105 20:44:34.500] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:151 I1105 20:44:34.500] STEP: setting up the StorageClass I1105 20:44:34.500] STEP: creating a StorageClass I1105 20:44:34.609] STEP: setting up the PVC and PV I1105 20:44:34.609] STEP: creating a PVC I1105 20:44:34.711] STEP: waiting for PVC to be in phase "Bound" I1105 20:44:34.711] Nov 5 20:44:34.711: INFO: Waiting up to 5m0s for PersistentVolumeClaims [pvc-m2g49] to have phase Bound I1105 20:44:34.795] Nov 5 20:44:34.795: INFO: PersistentVolumeClaim pvc-m2g49 found but phase is Pending instead of Bound. I1105 20:44:36.880] Nov 5 20:44:36.880: INFO: PersistentVolumeClaim pvc-m2g49 found but phase is Pending instead of Bound. I1105 20:44:38.965] Nov 5 20:44:38.964: INFO: PersistentVolumeClaim pvc-m2g49 found but phase is Pending instead of Bound. I1105 20:44:41.049] Nov 5 20:44:41.049: INFO: PersistentVolumeClaim pvc-m2g49 found but phase is Pending instead of Bound. I1105 20:44:43.134] Nov 5 20:44:43.134: INFO: PersistentVolumeClaim pvc-m2g49 found but phase is Pending instead of Bound. I1105 20:44:45.218] Nov 5 20:44:45.218: INFO: PersistentVolumeClaim pvc-m2g49 found but phase is Pending instead of Bound. I1105 20:44:47.303] Nov 5 20:44:47.303: INFO: PersistentVolumeClaim pvc-m2g49 found but phase is Pending instead of Bound. I1105 20:44:49.387] Nov 5 20:44:49.387: INFO: PersistentVolumeClaim pvc-m2g49 found but phase is Pending instead of Bound. I1105 20:44:51.472] Nov 5 20:44:51.472: INFO: PersistentVolumeClaim pvc-m2g49 found but phase is Pending instead of Bound. I1105 20:44:53.556] Nov 5 20:44:53.556: INFO: PersistentVolumeClaim pvc-m2g49 found but phase is Pending instead of Bound. I1105 20:44:55.641] Nov 5 20:44:55.641: INFO: PersistentVolumeClaim pvc-m2g49 found but phase is Pending instead of Bound. I1105 20:44:57.725] Nov 5 20:44:57.725: INFO: PersistentVolumeClaim pvc-m2g49 found but phase is Pending instead of Bound. I1105 20:44:59.810] Nov 5 20:44:59.810: INFO: PersistentVolumeClaim pvc-m2g49 found but phase is Pending instead of Bound. I1105 20:45:01.894] Nov 5 20:45:01.894: INFO: PersistentVolumeClaim pvc-m2g49 found but phase is Pending instead of Bound. I1105 20:45:03.980] Nov 5 20:45:03.980: INFO: PersistentVolumeClaim pvc-m2g49 found but phase is Pending instead of Bound. I1105 20:45:06.064] Nov 5 20:45:06.064: INFO: PersistentVolumeClaim pvc-m2g49 found but phase is Pending instead of Bound. I1105 20:45:08.151] Nov 5 20:45:08.151: INFO: PersistentVolumeClaim pvc-m2g49 found but phase is Pending instead of Bound. I1105 20:45:10.235] Nov 5 20:45:10.235: INFO: PersistentVolumeClaim pvc-m2g49 found but phase is Pending instead of Bound. I1105 20:45:12.320] Nov 5 20:45:12.320: INFO: PersistentVolumeClaim pvc-m2g49 found but phase is Pending instead of Bound. I1105 20:45:14.405] Nov 5 20:45:14.404: INFO: PersistentVolumeClaim pvc-m2g49 found but phase is Pending instead of Bound. I1105 20:45:16.490] Nov 5 20:45:16.490: INFO: PersistentVolumeClaim pvc-m2g49 found but phase is Pending instead of Bound. I1105 20:45:18.574] Nov 5 20:45:18.574: INFO: PersistentVolumeClaim pvc-m2g49 found but phase is Pending instead of Bound. I1105 20:45:20.659] Nov 5 20:45:20.658: INFO: PersistentVolumeClaim pvc-m2g49 found but phase is Pending instead of Bound. I1105 20:45:22.743] Nov 5 20:45:22.742: INFO: PersistentVolumeClaim pvc-m2g49 found but phase is Pending instead of Bound. I1105 20:45:24.829] Nov 5 20:45:24.828: INFO: PersistentVolumeClaim pvc-m2g49 found but phase is Pending instead of Bound. I1105 20:45:26.913] Nov 5 20:45:26.913: INFO: PersistentVolumeClaim pvc-m2g49 found but phase is Pending instead of Bound. I1105 20:45:28.997] Nov 5 20:45:28.997: INFO: PersistentVolumeClaim pvc-m2g49 found but phase is Pending instead of Bound. I1105 20:45:31.082] Nov 5 20:45:31.082: INFO: PersistentVolumeClaim pvc-m2g49 found but phase is Pending instead of Bound. I1105 20:45:33.166] Nov 5 20:45:33.166: INFO: PersistentVolumeClaim pvc-m2g49 found but phase is Pending instead of Bound. I1105 20:45:35.253] Nov 5 20:45:35.253: INFO: PersistentVolumeClaim pvc-m2g49 found but phase is Pending instead of Bound. I1105 20:45:37.337] Nov 5 20:45:37.337: INFO: PersistentVolumeClaim pvc-m2g49 found but phase is Pending instead of Bound. I1105 20:45:39.422] Nov 5 20:45:39.422: INFO: PersistentVolumeClaim pvc-m2g49 found but phase is Pending instead of Bound. I1105 20:45:41.507] Nov 5 20:45:41.506: INFO: PersistentVolumeClaim pvc-m2g49 found but phase is Pending instead of Bound. I1105 20:45:43.591] Nov 5 20:45:43.591: INFO: PersistentVolumeClaim pvc-m2g49 found but phase is Pending instead of Bound. I1105 20:45:45.676] Nov 5 20:45:45.676: INFO: PersistentVolumeClaim pvc-m2g49 found but phase is Pending instead of Bound. I1105 20:45:47.760] Nov 5 20:45:47.760: INFO: PersistentVolumeClaim pvc-m2g49 found but phase is Pending instead of Bound. I1105 20:45:49.844] Nov 5 20:45:49.844: INFO: PersistentVolumeClaim pvc-m2g49 found but phase is Pending instead of Bound. I1105 20:45:51.929] Nov 5 20:45:51.928: INFO: PersistentVolumeClaim pvc-m2g49 found but phase is Pending instead of Bound. I1105 20:45:54.013] Nov 5 20:45:54.013: INFO: PersistentVolumeClaim pvc-m2g49 found but phase is Pending instead of Bound. I1105 20:45:56.097] Nov 5 20:45:56.097: INFO: PersistentVolumeClaim pvc-m2g49 found but phase is Pending instead of Bound. I1105 20:45:58.216] Nov 5 20:45:58.216: INFO: PersistentVolumeClaim pvc-m2g49 found but phase is Pending instead of Bound. I1105 20:46:00.301] Nov 5 20:46:00.301: INFO: PersistentVolumeClaim pvc-m2g49 found but phase is Pending instead of Bound. I1105 20:46:02.386] Nov 5 20:46:02.385: INFO: PersistentVolumeClaim pvc-m2g49 found but phase is Pending instead of Bound. I1105 20:46:04.470] Nov 5 20:46:04.470: INFO: PersistentVolumeClaim pvc-m2g49 found but phase is Pending instead of Bound. I1105 20:46:06.555] Nov 5 20:46:06.555: INFO: PersistentVolumeClaim pvc-m2g49 found but phase is Pending instead of Bound. I1105 20:46:08.644] Nov 5 20:46:08.643: INFO: PersistentVolumeClaim pvc-m2g49 found but phase is Pending instead of Bound. I1105 20:46:10.728] Nov 5 20:46:10.728: INFO: PersistentVolumeClaim pvc-m2g49 found but phase is Pending instead of Bound. I1105 20:46:12.813] Nov 5 20:46:12.813: INFO: PersistentVolumeClaim pvc-m2g49 found but phase is Pending instead of Bound. I1105 20:46:14.893] Nov 5 20:46:14.892: INFO: PersistentVolumeClaim pvc-m2g49 found but phase is Pending instead of Bound. I1105 20:46:16.972] Nov 5 20:46:16.971: INFO: PersistentVolumeClaim pvc-m2g49 found but phase is Pending instead of Bound. I1105 20:46:19.051] Nov 5 20:46:19.050: INFO: PersistentVolumeClaim pvc-m2g49 found but phase is Pending instead of Bound. I1105 20:46:21.131] Nov 5 20:46:21.131: INFO: PersistentVolumeClaim pvc-m2g49 found but phase is Pending instead of Bound. I1105 20:46:23.211] Nov 5 20:46:23.210: INFO: PersistentVolumeClaim pvc-m2g49 found but phase is Pending instead of Bound. I1105 20:46:25.290] Nov 5 20:46:25.290: INFO: PersistentVolumeClaim pvc-m2g49 found but phase is Pending instead of Bound. I1105 20:46:27.369] Nov 5 20:46:27.369: INFO: PersistentVolumeClaim pvc-m2g49 found but phase is Pending instead of Bound. I1105 20:46:29.449] Nov 5 20:46:29.449: INFO: PersistentVolumeClaim pvc-m2g49 found but phase is Pending instead of Bound. I1105 20:46:31.529] Nov 5 20:46:31.529: INFO: PersistentVolumeClaim pvc-m2g49 found but phase is Pending instead of Bound. I1105 20:46:33.609] Nov 5 20:46:33.609: INFO: PersistentVolumeClaim pvc-m2g49 found but phase is Pending instead of Bound. I1105 20:46:35.688] Nov 5 20:46:35.688: INFO: PersistentVolumeClaim pvc-m2g49 found but phase is Pending instead of Bound. I1105 20:46:37.767] Nov 5 20:46:37.767: INFO: PersistentVolumeClaim pvc-m2g49 found but phase is Pending instead of Bound. I1105 20:46:39.847] Nov 5 20:46:39.847: INFO: PersistentVolumeClaim pvc-m2g49 found but phase is Pending instead of Bound. I1105 20:46:41.927] Nov 5 20:46:41.927: INFO: PersistentVolumeClaim pvc-m2g49 found but phase is Pending instead of Bound. I1105 20:46:44.006] Nov 5 20:46:44.006: INFO: PersistentVolumeClaim pvc-m2g49 found but phase is Pending instead of Bound. I1105 20:46:46.085] Nov 5 20:46:46.084: INFO: PersistentVolumeClaim pvc-m2g49 found but phase is Pending instead of Bound. I1105 20:46:48.164] Nov 5 20:46:48.164: INFO: PersistentVolumeClaim pvc-m2g49 found but phase is Pending instead of Bound. I1105 20:46:50.244] Nov 5 20:46:50.243: INFO: PersistentVolumeClaim pvc-m2g49 found but phase is Pending instead of Bound. I1105 20:46:52.323] Nov 5 20:46:52.322: INFO: PersistentVolumeClaim pvc-m2g49 found but phase is Pending instead of Bound. I1105 20:46:54.402] Nov 5 20:46:54.402: INFO: PersistentVolumeClaim pvc-m2g49 found but phase is Pending instead of Bound. I1105 20:46:56.482] Nov 5 20:46:56.482: INFO: PersistentVolumeClaim pvc-m2g49 found but phase is Pending instead of Bound. I1105 20:46:58.561] Nov 5 20:46:58.561: INFO: PersistentVolumeClaim pvc-m2g49 found but phase is Pending instead of Bound. I1105 20:47:00.641] Nov 5 20:47:00.641: INFO: PersistentVolumeClaim pvc-m2g49 found but phase is Pending instead of Bound. I1105 20:47:02.720] Nov 5 20:47:02.720: INFO: PersistentVolumeClaim pvc-m2g49 found but phase is Pending instead of Bound. I1105 20:47:04.799] Nov 5 20:47:04.799: INFO: PersistentVolumeClaim pvc-m2g49 found but phase is Pending instead of Bound. I1105 20:47:06.879] Nov 5 20:47:06.879: INFO: PersistentVolumeClaim pvc-m2g49 found but phase is Pending instead of Bound. I1105 20:47:08.958] Nov 5 20:47:08.958: INFO: PersistentVolumeClaim pvc-m2g49 found but phase is Pending instead of Bound. I1105 20:47:11.038] Nov 5 20:47:11.037: INFO: PersistentVolumeClaim pvc-m2g49 found but phase is Pending instead of Bound. I1105 20:47:13.117] Nov 5 20:47:13.116: INFO: PersistentVolumeClaim pvc-m2g49 found but phase is Pending instead of Bound. I1105 20:47:15.196] Nov 5 20:47:15.195: INFO: PersistentVolumeClaim pvc-m2g49 found but phase is Pending instead of Bound. I1105 20:47:17.279] Nov 5 20:47:17.279: INFO: PersistentVolumeClaim pvc-m2g49 found but phase is Pending instead of Bound. I1105 20:47:19.359] Nov 5 20:47:19.358: INFO: PersistentVolumeClaim pvc-m2g49 found but phase is Pending instead of Bound. I1105 20:47:21.438] Nov 5 20:47:21.438: INFO: PersistentVolumeClaim pvc-m2g49 found but phase is Pending instead of Bound. I1105 20:47:23.517] Nov 5 20:47:23.516: INFO: PersistentVolumeClaim pvc-m2g49 found but phase is Pending instead of Bound. I1105 20:47:25.596] Nov 5 20:47:25.595: INFO: PersistentVolumeClaim pvc-m2g49 found but phase is Pending instead of Bound. I1105 20:47:27.675] Nov 5 20:47:27.675: INFO: PersistentVolumeClaim pvc-m2g49 found but phase is Pending instead of Bound. I1105 20:47:29.757] Nov 5 20:47:29.757: INFO: PersistentVolumeClaim pvc-m2g49 found but phase is Pending instead of Bound. I1105 20:47:31.843] Nov 5 20:47:31.843: INFO: PersistentVolumeClaim pvc-m2g49 found but phase is Pending instead of Bound. I1105 20:47:33.923] Nov 5 20:47:33.923: INFO: PersistentVolumeClaim pvc-m2g49 found but phase is Pending instead of Bound. I1105 20:47:36.003] Nov 5 20:47:36.003: INFO: PersistentVolumeClaim pvc-m2g49 found but phase is Pending instead of Bound. I1105 20:47:38.082] Nov 5 20:47:38.082: INFO: PersistentVolumeClaim pvc-m2g49 found but phase is Pending instead of Bound. I1105 20:47:40.162] Nov 5 20:47:40.161: INFO: PersistentVolumeClaim pvc-m2g49 found but phase is Pending instead of Bound. I1105 20:47:42.241] Nov 5 20:47:42.241: INFO: PersistentVolumeClaim pvc-m2g49 found but phase is Pending instead of Bound. I1105 20:47:44.321] Nov 5 20:47:44.320: INFO: PersistentVolumeClaim pvc-m2g49 found but phase is Pending instead of Bound. I1105 20:47:46.401] Nov 5 20:47:46.401: INFO: PersistentVolumeClaim pvc-m2g49 found but phase is Pending instead of Bound. I1105 20:47:48.480] Nov 5 20:47:48.480: INFO: PersistentVolumeClaim pvc-m2g49 found but phase is Pending instead of Bound. I1105 20:47:50.560] Nov 5 20:47:50.560: INFO: PersistentVolumeClaim pvc-m2g49 found but phase is Pending instead of Bound. I1105 20:47:52.639] Nov 5 20:47:52.639: INFO: PersistentVolumeClaim pvc-m2g49 found but phase is Pending instead of Bound. I1105 20:47:54.718] Nov 5 20:47:54.718: INFO: PersistentVolumeClaim pvc-m2g49 found but phase is Pending instead of Bound. I1105 20:47:56.797] Nov 5 20:47:56.797: INFO: PersistentVolumeClaim pvc-m2g49 found but phase is Pending instead of Bound. I1105 20:47:58.877] Nov 5 20:47:58.877: INFO: PersistentVolumeClaim pvc-m2g49 found but phase is Pending instead of Bound. I1105 20:48:00.956] Nov 5 20:48:00.956: INFO: PersistentVolumeClaim pvc-m2g49 found but phase is Pending instead of Bound. I1105 20:48:03.035] Nov 5 20:48:03.035: INFO: PersistentVolumeClaim pvc-m2g49 found but phase is Pending instead of Bound. I1105 20:48:05.114] Nov 5 20:48:05.114: INFO: PersistentVolumeClaim pvc-m2g49 found but phase is Pending instead of Bound. I1105 20:48:07.201] Nov 5 20:48:07.201: INFO: PersistentVolumeClaim pvc-m2g49 found but phase is Pending instead of Bound. I1105 20:48:09.281] Nov 5 20:48:09.280: INFO: PersistentVolumeClaim pvc-m2g49 found but phase is Pending instead of Bound. I1105 20:48:11.360] Nov 5 20:48:11.360: INFO: PersistentVolumeClaim pvc-m2g49 found but phase is Pending instead of Bound. I1105 20:48:13.439] Nov 5 20:48:13.439: INFO: PersistentVolumeClaim pvc-m2g49 found but phase is Pending instead of Bound. I1105 20:48:15.518] Nov 5 20:48:15.518: INFO: PersistentVolumeClaim pvc-m2g49 found but phase is Pending instead of Bound. I1105 20:48:17.598] Nov 5 20:48:17.598: INFO: PersistentVolumeClaim pvc-m2g49 found but phase is Pending instead of Bound. I1105 20:48:19.677] Nov 5 20:48:19.677: INFO: PersistentVolumeClaim pvc-m2g49 found but phase is Pending instead of Bound. I1105 20:48:21.757] Nov 5 20:48:21.756: INFO: PersistentVolumeClaim pvc-m2g49 found but phase is Pending instead of Bound. I1105 20:48:23.835] Nov 5 20:48:23.835: INFO: PersistentVolumeClaim pvc-m2g49 found but phase is Pending instead of Bound. I1105 20:48:25.920] Nov 5 20:48:25.920: INFO: PersistentVolumeClaim pvc-m2g49 found but phase is Pending instead of Bound. I1105 20:48:28.002] Nov 5 20:48:28.002: INFO: PersistentVolumeClaim pvc-m2g49 found but phase is Pending instead of Bound. I1105 20:48:30.081] Nov 5 20:48:30.081: INFO: PersistentVolumeClaim pvc-m2g49 found but phase is Pending instead of Bound. I1105 20:48:32.161] Nov 5 20:48:32.160: INFO: PersistentVolumeClaim pvc-m2g49 found but phase is Pending instead of Bound. I1105 20:48:34.239] Nov 5 20:48:34.239: INFO: PersistentVolumeClaim pvc-m2g49 found but phase is Pending instead of Bound. I1105 20:48:36.319] Nov 5 20:48:36.318: INFO: PersistentVolumeClaim pvc-m2g49 found but phase is Pending instead of Bound. I1105 20:48:38.398] Nov 5 20:48:38.398: INFO: PersistentVolumeClaim pvc-m2g49 found but phase is Pending instead of Bound. I1105 20:48:40.477] Nov 5 20:48:40.476: INFO: PersistentVolumeClaim pvc-m2g49 found but phase is Pending instead of Bound. I1105 20:48:42.556] Nov 5 20:48:42.556: INFO: PersistentVolumeClaim pvc-m2g49 found but phase is Pending instead of Bound. I1105 20:48:44.636] Nov 5 20:48:44.636: INFO: PersistentVolumeClaim pvc-m2g49 found but phase is Pending instead of Bound. I1105 20:48:46.716] Nov 5 20:48:46.715: INFO: PersistentVolumeClaim pvc-m2g49 found but phase is Pending instead of Bound. I1105 20:48:48.795] Nov 5 20:48:48.795: INFO: PersistentVolumeClaim pvc-m2g49 found but phase is Pending instead of Bound. I1105 20:48:50.874] Nov 5 20:48:50.874: INFO: PersistentVolumeClaim pvc-m2g49 found but phase is Pending instead of Bound. I1105 20:48:52.953] Nov 5 20:48:52.953: INFO: PersistentVolumeClaim pvc-m2g49 found but phase is Pending instead of Bound. I1105 20:48:55.034] Nov 5 20:48:55.033: INFO: PersistentVolumeClaim pvc-m2g49 found but phase is Pending instead of Bound. I1105 20:48:57.113] Nov 5 20:48:57.112: INFO: PersistentVolumeClaim pvc-m2g49 found but phase is Pending instead of Bound. I1105 20:48:59.192] Nov 5 20:48:59.192: INFO: PersistentVolumeClaim pvc-m2g49 found but phase is Pending instead of Bound. I1105 20:49:01.271] Nov 5 20:49:01.271: INFO: PersistentVolumeClaim pvc-m2g49 found but phase is Pending instead of Bound. I1105 20:49:03.350] Nov 5 20:49:03.350: INFO: PersistentVolumeClaim pvc-m2g49 found but phase is Pending instead of Bound. I1105 20:49:05.430] Nov 5 20:49:05.429: INFO: PersistentVolumeClaim pvc-m2g49 found but phase is Pending instead of Bound. I1105 20:49:07.509] Nov 5 20:49:07.509: INFO: PersistentVolumeClaim pvc-m2g49 found but phase is Pending instead of Bound. I1105 20:49:09.589] Nov 5 20:49:09.589: INFO: PersistentVolumeClaim pvc-m2g49 found but phase is Pending instead of Bound. I1105 20:49:11.669] Nov 5 20:49:11.669: INFO: PersistentVolumeClaim pvc-m2g49 found but phase is Pending instead of Bound. I1105 20:49:13.749] Nov 5 20:49:13.749: INFO: PersistentVolumeClaim pvc-m2g49 found but phase is Pending instead of Bound. I1105 20:49:15.828] Nov 5 20:49:15.828: INFO: PersistentVolumeClaim pvc-m2g49 found but phase is Pending instead of Bound. I1105 20:49:17.908] Nov 5 20:49:17.908: INFO: PersistentVolumeClaim pvc-m2g49 found but phase is Pending instead of Bound. I1105 20:49:19.987] Nov 5 20:49:19.987: INFO: PersistentVolumeClaim pvc-m2g49 found but phase is Pending instead of Bound. I1105 20:49:22.067] Nov 5 20:49:22.066: INFO: PersistentVolumeClaim pvc-m2g49 found but phase is Pending instead of Bound. I1105 20:49:24.160] Nov 5 20:49:24.160: INFO: PersistentVolumeClaim pvc-m2g49 found but phase is Pending instead of Bound. I1105 20:49:26.252] Nov 5 20:49:26.252: INFO: PersistentVolumeClaim pvc-m2g49 found but phase is Pending instead of Bound. I1105 20:49:28.331] Nov 5 20:49:28.331: INFO: PersistentVolumeClaim pvc-m2g49 found but phase is Pending instead of Bound. I1105 20:49:30.411] Nov 5 20:49:30.411: INFO: PersistentVolumeClaim pvc-m2g49 found but phase is Pending instead of Bound. I1105 20:49:32.490] Nov 5 20:49:32.490: INFO: PersistentVolumeClaim pvc-m2g49 found but phase is Pending instead of Bound. I1105 20:49:34.570] Nov 5 20:49:34.569: INFO: PersistentVolumeClaim pvc-m2g49 found but phase is Pending instead of Bound. I1105 20:49:36.570] Nov 5 20:49:36.570: INFO: Unexpected error occurred: PersistentVolumeClaims [pvc-m2g49] not all in phase Bound within 5m0s I1105 20:49:36.570] [AfterEach] [multi-az] I1105 20:49:36.570] /go/src/sigs.k8s.io/azuredisk-csi-driver/vendor/k8s.io/kubernetes/test/e2e/framework/framework.go:151 I1105 20:49:36.571] STEP: Collecting events from namespace "azuredisk-6880". I1105 20:49:36.652] STEP: Found 3 events. I1105 20:49:36.653] Nov 5 20:49:36.652: INFO: At 2019-11-05 20:44:34 +0000 UTC - event for pvc-m2g49: {disk.csi.azure.com_csi-azuredisk-controller-6f6587bfc6-62nrt_3930bc69-0009-11ea-8cf5-829a27ff7d5a } Provisioning: External provisioner is provisioning volume for claim "azuredisk-6880/pvc-m2g49" I1105 20:49:36.653] Nov 5 20:49:36.652: INFO: At 2019-11-05 20:44:34 +0000 UTC - event for pvc-m2g49: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "disk.csi.azure.com" or manually created by system administrator I1105 20:49:36.654] Nov 5 20:49:36.652: INFO: At 2019-11-05 20:44:34 +0000 UTC - event for pvc-m2g49: {disk.csi.azure.com_csi-azuredisk-controller-6f6587bfc6-62nrt_3930bc69-0009-11ea-8cf5-829a27ff7d5a } ProvisioningFailed: failed to provision volume with StorageClass "azuredisk-6880-disk.csi.azure.com-dynamic-sc-b2lt8": rpc error: code = Unknown desc = compute.DisksClient#CreateOrUpdate: Failure sending request: StatusCode=400 -- Original Error: Code="InvalidAvailabilityZone" Message="The zone(s) '' for resource 'Microsoft.Compute/disks/pvc-679b1f36-48d8-49c5-a05f-83d3128d4fe8' is not supported. The supported zones for location 'eastus2' are '1,2,3'" I1105 20:49:36.731] Nov 5 20:49:36.731: INFO: POD NODE PHASE GRACE CONDITIONS I1105 20:49:36.731] Nov 5 20:49:36.731: INFO: I1105 20:49:36.965] Nov 5 20:49:36.965: INFO: I1105 20:49:36.966] Logging node info for node k8s-agentpool-97435445-vmss000000 I1105 20:49:37.047] Nov 5 20:49:37.044: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-agentpool-97435445-vmss000000,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-agentpool-97435445-vmss000000,UID:e955dc47-8c8f-4142-9241-3d409f32e42d,ResourceVersion:8326,Generation:0,CreationTimestamp:2019-11-05 20:07:34 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{agentpool: agentpool,beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_D4s_v3,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 0,kubernetes.azure.com/cluster: kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b,kubernetes.azure.com/role: agent,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-agentpool-97435445-vmss000000,kubernetes.io/os: linux,kubernetes.io/role: agent,node-role.kubernetes.io/agent: ,storageprofile: managed,storagetier: Premium_LRS,topology.disk.csi.azure.com/zone: 0,},Annotations:map[string]string{csi.volume.kubernetes.io/nodeid: {"disk.csi.azure.com":"k8s-agentpool-97435445-vmss000000"},node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.2.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool-97435445-vmss/virtualMachines/0,Unschedulable:false,Taints:[],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-csi-disk.csi.azure.com: {{8 0} {} 8 DecimalSI},cpu: {{4 0} {} 4 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{16797040640 0} {} BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-csi-disk.csi.azure.com: {{8 0} {} 8 DecimalSI},cpu: {{4 0} {} 4 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{16010608640 0} {} BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 20:08:01 +0000 UTC 2019-11-05 20:08:01 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 20:48:42 +0000 UTC 2019-11-05 20:07:27 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 20:48:42 +0000 UTC 2019-11-05 20:07:27 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 20:48:42 +0000 UTC 2019-11-05 20:07:27 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 20:48:42 +0000 UTC 2019-11-05 20:07:55 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.0.4} {Hostname k8s-agentpool-97435445-vmss000000}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:bfb55325809340228149d2d93f50df58,SystemUUID:CB2ED036-0439-F641-BC7F-640C3638C027,BootID:a917f4d1-549d-471d-b7dc-47ae1715d97d,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[k8sprowinternal.azurecr.io/azuredisk-csi@sha256:75d4ff5d7cd845d6e810c67796ecb037e852fb849ec1eb5b3194564008e6d1e0 k8sprowinternal.azurecr.io/azuredisk-csi:e2e-3e3ed045c6faa509dbd8d2890ee9f56aa2934887] 286930414} {[k8s.gcr.io/kubernetes-dashboard-amd64@sha256:0ae6b69432e78069c5ce2bcde0fe409c5c4d6f0f4d9cd50a17974fea38898747 k8s.gcr.io/kubernetes-dashboard-amd64:v1.10.1] 121711221} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/metrics-server-amd64@sha256:49a9f12f7067d11f42c803dbe61ed2c1299959ad85cb315b25ff7eef8e6b8892 k8s.gcr.io/metrics-server-amd64:v0.2.1] 42541759} {[k8s.gcr.io/coredns@sha256:e83beb5e43f8513fa735e77ffc5859640baea30a882a11cc75c4c3244a737d3c k8s.gcr.io/coredns:1.5.0] 42488424} {[mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume@sha256:4fd30d43947d4a54fc89ead7985beecfd3c9b2a93a0655a373b1608ab90bd5af mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume:v0.0.7] 22909487} {[quay.io/k8scsi/livenessprobe@sha256:8c9c8a0f330acc2e558ae6624309447d02522e76aa35af23fa19b88abf4381cb quay.io/k8scsi/livenessprobe:v1.0.2] 17415381} {[quay.io/k8scsi/csi-node-driver-registrar@sha256:13daf82fb99e951a4bff8ae5fc7c17c3a8fe7130be6400990d8f6076c32d4599 quay.io/k8scsi/csi-node-driver-registrar:v1.1.0] 15815995} {[mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume@sha256:23d8c6033f02a1ecad05127ebdc931bb871264228661bc122704b0974e4d9fdd mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume:1.0.8] 1159025} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 20:49:37.048] Nov 5 20:49:37.045: INFO: I1105 20:49:37.048] Logging kubelet events for node k8s-agentpool-97435445-vmss000000 I1105 20:49:37.128] Nov 5 20:49:37.128: INFO: I1105 20:49:37.128] Logging pods the kubelet thinks is on node k8s-agentpool-97435445-vmss000000 I1105 20:49:37.212] Nov 5 20:49:37.212: INFO: blobfuse-flexvol-installer-p7hdb started at 2019-11-05 20:08:01 +0000 UTC (0+1 container statuses recorded) I1105 20:49:37.213] Nov 5 20:49:37.212: INFO: Container blobfuse-flexvol-installer ready: true, restart count 0 I1105 20:49:37.213] Nov 5 20:49:37.212: INFO: coredns-7f68dcdbdb-zh8g5 started at 2019-11-05 20:08:01 +0000 UTC (0+1 container statuses recorded) I1105 20:49:37.213] Nov 5 20:49:37.212: INFO: Container coredns ready: true, restart count 0 I1105 20:49:37.213] Nov 5 20:49:37.212: INFO: azure-ip-masq-agent-m7tv9 started at 2019-11-05 20:08:01 +0000 UTC (0+1 container statuses recorded) I1105 20:49:37.213] Nov 5 20:49:37.212: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 20:49:37.214] Nov 5 20:49:37.212: INFO: keyvault-flexvolume-smspt started at 2019-11-05 20:08:01 +0000 UTC (0+1 container statuses recorded) I1105 20:49:37.214] Nov 5 20:49:37.212: INFO: Container keyvault-flexvolume ready: true, restart count 0 I1105 20:49:37.214] Nov 5 20:49:37.212: INFO: metrics-server-864ffbc5c-wl84z started at 2019-11-05 20:08:01 +0000 UTC (0+1 container statuses recorded) I1105 20:49:37.214] Nov 5 20:49:37.212: INFO: Container metrics-server ready: true, restart count 0 I1105 20:49:37.214] Nov 5 20:49:37.212: INFO: kube-proxy-8mt7f started at 2019-11-05 20:08:01 +0000 UTC (0+1 container statuses recorded) I1105 20:49:37.214] Nov 5 20:49:37.212: INFO: Container kube-proxy ready: true, restart count 0 I1105 20:49:37.214] Nov 5 20:49:37.212: INFO: kubernetes-dashboard-66dd8b8df7-t6j4l started at 2019-11-05 20:08:01 +0000 UTC (0+1 container statuses recorded) I1105 20:49:37.215] Nov 5 20:49:37.212: INFO: Container kubernetes-dashboard ready: true, restart count 0 I1105 20:49:37.215] Nov 5 20:49:37.212: INFO: csi-azuredisk-node-5wwhz started at 2019-11-05 20:16:09 +0000 UTC (0+3 container statuses recorded) I1105 20:49:37.215] Nov 5 20:49:37.212: INFO: Container azuredisk ready: true, restart count 0 I1105 20:49:37.215] Nov 5 20:49:37.212: INFO: Container liveness-probe ready: true, restart count 0 I1105 20:49:37.215] Nov 5 20:49:37.212: INFO: Container node-driver-registrar ready: true, restart count 0 I1105 20:49:37.475] Nov 5 20:49:37.475: INFO: I1105 20:49:37.475] Latency metrics for node k8s-agentpool-97435445-vmss000000 I1105 20:49:37.475] Nov 5 20:49:37.475: INFO: I1105 20:49:37.475] Logging node info for node k8s-agentpool-97435445-vmss000001 I1105 20:49:37.559] Nov 5 20:49:37.555: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-agentpool-97435445-vmss000001,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-agentpool-97435445-vmss000001,UID:75d3cc50-ca6d-45e0-a42a-6090e8d537dd,ResourceVersion:8322,Generation:0,CreationTimestamp:2019-11-05 20:07:34 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{agentpool: agentpool,beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_D4s_v3,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 1,kubernetes.azure.com/cluster: kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b,kubernetes.azure.com/role: agent,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-agentpool-97435445-vmss000001,kubernetes.io/os: linux,kubernetes.io/role: agent,node-role.kubernetes.io/agent: ,storageprofile: managed,storagetier: Premium_LRS,topology.disk.csi.azure.com/zone: 1,},Annotations:map[string]string{csi.volume.kubernetes.io/nodeid: {"disk.csi.azure.com":"k8s-agentpool-97435445-vmss000001"},node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.0.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool-97435445-vmss/virtualMachines/1,Unschedulable:false,Taints:[],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-csi-disk.csi.azure.com: {{8 0} {} 8 DecimalSI},cpu: {{4 0} {} 4 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{16797044736 0} {} 16403364Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-csi-disk.csi.azure.com: {{8 0} {} 8 DecimalSI},cpu: {{4 0} {} 4 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{16010612736 0} {} 15635364Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 20:08:25 +0000 UTC 2019-11-05 20:08:25 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 20:48:41 +0000 UTC 2019-11-05 20:07:26 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 20:48:41 +0000 UTC 2019-11-05 20:07:26 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 20:48:41 +0000 UTC 2019-11-05 20:07:26 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 20:48:41 +0000 UTC 2019-11-05 20:07:55 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.0.5} {Hostname k8s-agentpool-97435445-vmss000001}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:e78def54b2f545c6b94d88bbba523c60,SystemUUID:40AE8188-86AB-D848-84A4-D9254E1258BC,BootID:8587ec06-f4f4-43c1-86cd-06c295b05b63,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[k8sprowinternal.azurecr.io/azuredisk-csi@sha256:75d4ff5d7cd845d6e810c67796ecb037e852fb849ec1eb5b3194564008e6d1e0 k8sprowinternal.azurecr.io/azuredisk-csi:e2e-3e3ed045c6faa509dbd8d2890ee9f56aa2934887] 286930414} {[gcr.io/kubernetes-helm/tiller@sha256:f6d8f4ab9ba993b5f5b60a6edafe86352eabe474ffeb84cb6c79b8866dce45d1 gcr.io/kubernetes-helm/tiller:v2.11.0] 71821984} {[quay.io/k8scsi/csi-provisioner@sha256:d657c839dce87324fe2b677302913f9386f885f8746be7bea0ced5b0844e3433 quay.io/k8scsi/csi-provisioner:v1.3.1] 50596998} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[quay.io/k8scsi/csi-attacher@sha256:26fccd7a99d973845df1193b46ebdcc6ab8dc5f6e6be319750c471fce1742d13 quay.io/k8scsi/csi-attacher:v1.2.0] 46226754} {[quay.io/k8scsi/csi-cluster-driver-registrar@sha256:fafd75ae5442f192cfa8c2e792903aee30d5884b62e802e4464b0a895d21e3ef quay.io/k8scsi/csi-cluster-driver-registrar:v1.0.1] 45874691} {[quay.io/k8scsi/csi-snapshotter@sha256:a49e0da1af6f2bf717e41ba1eee8b5e6a1cbd66a709dd92cc43fe475fe2589eb quay.io/k8scsi/csi-snapshotter:v1.1.0] 44325352} {[mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume@sha256:4fd30d43947d4a54fc89ead7985beecfd3c9b2a93a0655a373b1608ab90bd5af mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume:v0.0.7] 22909487} {[quay.io/k8scsi/livenessprobe@sha256:8c9c8a0f330acc2e558ae6624309447d02522e76aa35af23fa19b88abf4381cb quay.io/k8scsi/livenessprobe:v1.0.2] 17415381} {[quay.io/k8scsi/csi-node-driver-registrar@sha256:13daf82fb99e951a4bff8ae5fc7c17c3a8fe7130be6400990d8f6076c32d4599 quay.io/k8scsi/csi-node-driver-registrar:v1.1.0] 15815995} {[quay.io/k8scsi/livenessprobe@sha256:dde617756e0f602adc566ab71fd885f1dad451ad3fb063ac991c95a2ff47aea5 quay.io/k8scsi/livenessprobe:v1.1.0] 14967303} {[mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume@sha256:23d8c6033f02a1ecad05127ebdc931bb871264228661bc122704b0974e4d9fdd mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume:1.0.8] 1159025} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 20:49:37.559] Nov 5 20:49:37.555: INFO: I1105 20:49:37.559] Logging kubelet events for node k8s-agentpool-97435445-vmss000001 I1105 20:49:37.640] Nov 5 20:49:37.640: INFO: I1105 20:49:37.640] Logging pods the kubelet thinks is on node k8s-agentpool-97435445-vmss000001 I1105 20:49:37.725] Nov 5 20:49:37.725: INFO: blobfuse-flexvol-installer-j4s7t started at 2019-11-05 20:07:59 +0000 UTC (0+1 container statuses recorded) I1105 20:49:37.726] Nov 5 20:49:37.725: INFO: Container blobfuse-flexvol-installer ready: true, restart count 0 I1105 20:49:37.726] Nov 5 20:49:37.725: INFO: azure-ip-masq-agent-gdhcq started at 2019-11-05 20:07:59 +0000 UTC (0+1 container statuses recorded) I1105 20:49:37.726] Nov 5 20:49:37.725: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 20:49:37.726] Nov 5 20:49:37.725: INFO: keyvault-flexvolume-6bffx started at 2019-11-05 20:07:59 +0000 UTC (0+1 container statuses recorded) I1105 20:49:37.726] Nov 5 20:49:37.725: INFO: Container keyvault-flexvolume ready: true, restart count 0 I1105 20:49:37.726] Nov 5 20:49:37.725: INFO: kube-proxy-kjrpb started at 2019-11-05 20:08:00 +0000 UTC (0+1 container statuses recorded) I1105 20:49:37.726] Nov 5 20:49:37.725: INFO: Container kube-proxy ready: true, restart count 0 I1105 20:49:37.726] Nov 5 20:49:37.725: INFO: tiller-deploy-54c96cb5df-pt7x6 started at 2019-11-05 20:08:00 +0000 UTC (0+1 container statuses recorded) I1105 20:49:37.727] Nov 5 20:49:37.725: INFO: Container tiller ready: true, restart count 0 I1105 20:49:37.727] Nov 5 20:49:37.725: INFO: csi-azuredisk-node-q2dmt started at 2019-11-05 20:16:09 +0000 UTC (0+3 container statuses recorded) I1105 20:49:37.727] Nov 5 20:49:37.725: INFO: Container azuredisk ready: true, restart count 0 I1105 20:49:37.727] Nov 5 20:49:37.725: INFO: Container liveness-probe ready: true, restart count 0 I1105 20:49:37.727] Nov 5 20:49:37.725: INFO: Container node-driver-registrar ready: true, restart count 0 I1105 20:49:37.727] Nov 5 20:49:37.725: INFO: csi-azuredisk-controller-6f6587bfc6-62nrt started at 2019-11-05 20:16:09 +0000 UTC (0+6 container statuses recorded) I1105 20:49:37.727] Nov 5 20:49:37.725: INFO: Container azuredisk ready: true, restart count 0 I1105 20:49:37.727] Nov 5 20:49:37.725: INFO: Container cluster-driver-registrar ready: true, restart count 0 I1105 20:49:37.728] Nov 5 20:49:37.725: INFO: Container csi-attacher ready: true, restart count 0 I1105 20:49:37.728] Nov 5 20:49:37.725: INFO: Container csi-provisioner ready: true, restart count 0 I1105 20:49:37.728] Nov 5 20:49:37.725: INFO: Container csi-snapshotter ready: true, restart count 0 I1105 20:49:37.728] Nov 5 20:49:37.725: INFO: Container liveness-probe ready: true, restart count 0 I1105 20:49:37.985] Nov 5 20:49:37.985: INFO: I1105 20:49:37.985] Latency metrics for node k8s-agentpool-97435445-vmss000001 I1105 20:49:37.986] Nov 5 20:49:37.985: INFO: I1105 20:49:37.986] Logging node info for node k8s-master-97435445-0 I1105 20:49:38.067] Nov 5 20:49:38.065: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-97435445-0,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-97435445-0,UID:df3ae36f-961c-4729-a8f5-34cda54f4ff9,ResourceVersion:8388,Generation:0,CreationTimestamp:2019-11-05 20:08:36 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 2,kubernetes.azure.com/cluster: kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-97435445-0,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.4.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b/providers/Microsoft.Compute/virtualMachines/k8s-master-97435445-0,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 20:08:46 +0000 UTC 2019-11-05 20:08:46 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 20:49:02 +0000 UTC 2019-11-05 20:08:28 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 20:49:02 +0000 UTC 2019-11-05 20:08:28 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 20:49:02 +0000 UTC 2019-11-05 20:08:28 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 20:49:02 +0000 UTC 2019-11-05 20:08:36 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.255.5} {Hostname k8s-master-97435445-0}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:3c381865192341c2868672e8626cef7d,SystemUUID:F3D259D3-7EB8-B844-A1F0-F954DCD0D76B,BootID:4948329e-cf57-48c3-aa83-5297bef70370,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 20:49:38.067] Nov 5 20:49:38.065: INFO: I1105 20:49:38.068] Logging kubelet events for node k8s-master-97435445-0 I1105 20:49:38.155] Nov 5 20:49:38.155: INFO: I1105 20:49:38.155] Logging pods the kubelet thinks is on node k8s-master-97435445-0 I1105 20:49:38.239] Nov 5 20:49:38.239: INFO: kube-addon-manager-k8s-master-97435445-0 started at 2019-11-05 20:08:48 +0000 UTC (0+1 container statuses recorded) I1105 20:49:38.240] Nov 5 20:49:38.239: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 20:49:38.240] Nov 5 20:49:38.239: INFO: kube-apiserver-k8s-master-97435445-0 started at 2019-11-05 20:08:22 +0000 UTC (0+1 container statuses recorded) I1105 20:49:38.240] Nov 5 20:49:38.239: INFO: Container kube-apiserver ready: true, restart count 0 I1105 20:49:38.240] Nov 5 20:49:38.239: INFO: kube-controller-manager-k8s-master-97435445-0 started at 2019-11-05 20:08:22 +0000 UTC (0+1 container statuses recorded) I1105 20:49:38.240] Nov 5 20:49:38.239: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 20:49:38.240] Nov 5 20:49:38.239: INFO: kube-scheduler-k8s-master-97435445-0 started at 2019-11-05 20:08:22 +0000 UTC (0+1 container statuses recorded) I1105 20:49:38.240] Nov 5 20:49:38.239: INFO: Container kube-scheduler ready: true, restart count 0 I1105 20:49:38.241] Nov 5 20:49:38.239: INFO: azure-ip-masq-agent-q6dcl started at 2019-11-05 20:08:38 +0000 UTC (0+1 container statuses recorded) I1105 20:49:38.241] Nov 5 20:49:38.239: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 20:49:38.241] Nov 5 20:49:38.239: INFO: kube-proxy-5wnd2 started at 2019-11-05 20:08:38 +0000 UTC (0+1 container statuses recorded) I1105 20:49:38.241] Nov 5 20:49:38.239: INFO: Container kube-proxy ready: true, restart count 0 I1105 20:49:38.241] Nov 5 20:49:38.239: INFO: cloud-controller-manager-k8s-master-97435445-0 started at 2019-11-05 20:08:22 +0000 UTC (0+1 container statuses recorded) I1105 20:49:38.241] Nov 5 20:49:38.239: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 20:49:38.494] Nov 5 20:49:38.493: INFO: I1105 20:49:38.494] Latency metrics for node k8s-master-97435445-0 I1105 20:49:38.494] Nov 5 20:49:38.493: INFO: I1105 20:49:38.494] Logging node info for node k8s-master-97435445-1 I1105 20:49:38.575] Nov 5 20:49:38.573: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-97435445-1,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-97435445-1,UID:05534a67-d493-4ab5-9b0f-5ff5490e234f,ResourceVersion:8464,Generation:0,CreationTimestamp:2019-11-05 20:07:34 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 0,kubernetes.azure.com/cluster: kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-97435445-1,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.1.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b/providers/Microsoft.Compute/virtualMachines/k8s-master-97435445-1,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 20:08:26 +0000 UTC 2019-11-05 20:08:26 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 20:49:29 +0000 UTC 2019-11-05 20:07:28 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 20:49:29 +0000 UTC 2019-11-05 20:07:28 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 20:49:29 +0000 UTC 2019-11-05 20:07:28 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 20:49:29 +0000 UTC 2019-11-05 20:07:55 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{Hostname k8s-master-97435445-1} {InternalIP 10.240.255.6}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:079b605103b74acabac18fd2ea63368a,SystemUUID:38220EEC-BEFF-4C46-AA05-5A214AAFDC28,BootID:b1b45df0-d925-4bcc-9830-0202df36ebb4,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 20:49:38.575] Nov 5 20:49:38.573: INFO: I1105 20:49:38.576] Logging kubelet events for node k8s-master-97435445-1 I1105 20:49:38.661] Nov 5 20:49:38.661: INFO: I1105 20:49:38.661] Logging pods the kubelet thinks is on node k8s-master-97435445-1 I1105 20:49:38.745] Nov 5 20:49:38.745: INFO: kube-proxy-h4s2c started at 2019-11-05 20:08:00 +0000 UTC (0+1 container statuses recorded) I1105 20:49:38.745] Nov 5 20:49:38.745: INFO: Container kube-proxy ready: true, restart count 0 I1105 20:49:38.745] Nov 5 20:49:38.745: INFO: cloud-controller-manager-k8s-master-97435445-1 started at 2019-11-05 20:07:22 +0000 UTC (0+1 container statuses recorded) I1105 20:49:38.745] Nov 5 20:49:38.745: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 20:49:38.746] Nov 5 20:49:38.745: INFO: kube-addon-manager-k8s-master-97435445-1 started at 2019-11-05 20:07:22 +0000 UTC (0+1 container statuses recorded) I1105 20:49:38.746] Nov 5 20:49:38.745: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 20:49:38.746] Nov 5 20:49:38.745: INFO: kube-apiserver-k8s-master-97435445-1 started at 2019-11-05 20:07:22 +0000 UTC (0+1 container statuses recorded) I1105 20:49:38.746] Nov 5 20:49:38.745: INFO: Container kube-apiserver ready: true, restart count 0 I1105 20:49:38.746] Nov 5 20:49:38.745: INFO: kube-controller-manager-k8s-master-97435445-1 started at 2019-11-05 20:07:22 +0000 UTC (0+1 container statuses recorded) I1105 20:49:38.746] Nov 5 20:49:38.745: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 20:49:38.746] Nov 5 20:49:38.745: INFO: kube-scheduler-k8s-master-97435445-1 started at 2019-11-05 20:07:22 +0000 UTC (0+1 container statuses recorded) I1105 20:49:38.747] Nov 5 20:49:38.745: INFO: Container kube-scheduler ready: true, restart count 0 I1105 20:49:38.747] Nov 5 20:49:38.745: INFO: azure-ip-masq-agent-n29bm started at 2019-11-05 20:07:59 +0000 UTC (0+1 container statuses recorded) I1105 20:49:38.747] Nov 5 20:49:38.745: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 20:49:39.024] Nov 5 20:49:39.024: INFO: I1105 20:49:39.024] Latency metrics for node k8s-master-97435445-1 I1105 20:49:39.024] Nov 5 20:49:39.024: INFO: I1105 20:49:39.025] Logging node info for node k8s-master-97435445-2 I1105 20:49:39.105] Nov 5 20:49:39.103: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-97435445-2,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-97435445-2,UID:fa55c895-1a93-4743-885e-03fce84b5332,ResourceVersion:8338,Generation:0,CreationTimestamp:2019-11-05 20:09:19 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 0,kubernetes.azure.com/cluster: kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-97435445-2,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.5.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b/providers/Microsoft.Compute/virtualMachines/k8s-master-97435445-2,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 20:09:19 +0000 UTC 2019-11-05 20:09:19 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 20:48:46 +0000 UTC 2019-11-05 20:09:12 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 20:48:46 +0000 UTC 2019-11-05 20:09:12 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 20:48:46 +0000 UTC 2019-11-05 20:09:12 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 20:48:46 +0000 UTC 2019-11-05 20:09:19 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.255.7} {Hostname k8s-master-97435445-2}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:d19f6709138f496da43c095cec7b2360,SystemUUID:10116777-A8D7-CB4A-A8B7-7EBCF2F23B62,BootID:50cd14a9-5497-4972-bf60-cf59a0cbcb5f,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 20:49:39.105] Nov 5 20:49:39.103: INFO: I1105 20:49:39.106] Logging kubelet events for node k8s-master-97435445-2 I1105 20:49:39.186] Nov 5 20:49:39.186: INFO: I1105 20:49:39.187] Logging pods the kubelet thinks is on node k8s-master-97435445-2 I1105 20:49:39.270] Nov 5 20:49:39.270: INFO: kube-proxy-pz426 started at 2019-11-05 20:09:19 +0000 UTC (0+1 container statuses recorded) I1105 20:49:39.270] Nov 5 20:49:39.270: INFO: Container kube-proxy ready: true, restart count 0 I1105 20:49:39.270] Nov 5 20:49:39.270: INFO: kube-apiserver-k8s-master-97435445-2 started at 2019-11-05 20:09:05 +0000 UTC (0+1 container statuses recorded) I1105 20:49:39.270] Nov 5 20:49:39.270: INFO: Container kube-apiserver ready: true, restart count 0 I1105 20:49:39.271] Nov 5 20:49:39.270: INFO: kube-controller-manager-k8s-master-97435445-2 started at 2019-11-05 20:09:05 +0000 UTC (0+1 container statuses recorded) I1105 20:49:39.271] Nov 5 20:49:39.270: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 20:49:39.271] Nov 5 20:49:39.270: INFO: kube-scheduler-k8s-master-97435445-2 started at 2019-11-05 20:09:05 +0000 UTC (0+1 container statuses recorded) I1105 20:49:39.271] Nov 5 20:49:39.270: INFO: Container kube-scheduler ready: true, restart count 0 I1105 20:49:39.271] Nov 5 20:49:39.270: INFO: cloud-controller-manager-k8s-master-97435445-2 started at 2019-11-05 20:09:05 +0000 UTC (0+1 container statuses recorded) I1105 20:49:39.271] Nov 5 20:49:39.270: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 20:49:39.272] Nov 5 20:49:39.270: INFO: kube-addon-manager-k8s-master-97435445-2 started at 2019-11-05 20:09:05 +0000 UTC (0+1 container statuses recorded) I1105 20:49:39.272] Nov 5 20:49:39.270: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 20:49:39.272] Nov 5 20:49:39.270: INFO: azure-ip-masq-agent-sq7p5 started at 2019-11-05 20:09:19 +0000 UTC (0+1 container statuses recorded) I1105 20:49:39.272] Nov 5 20:49:39.270: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 20:49:39.528] Nov 5 20:49:39.528: INFO: I1105 20:49:39.528] Latency metrics for node k8s-master-97435445-2 I1105 20:49:39.528] Nov 5 20:49:39.528: INFO: I1105 20:49:39.529] Logging node info for node k8s-master-97435445-3 I1105 20:49:39.609] Nov 5 20:49:39.607: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-97435445-3,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-97435445-3,UID:8ffd1d0d-4487-4680-a2f6-e724387be637,ResourceVersion:8339,Generation:0,CreationTimestamp:2019-11-05 20:09:21 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 1,kubernetes.azure.com/cluster: kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-97435445-3,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.6.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b/providers/Microsoft.Compute/virtualMachines/k8s-master-97435445-3,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 20:09:39 +0000 UTC 2019-11-05 20:09:39 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 20:48:47 +0000 UTC 2019-11-05 20:09:18 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 20:48:47 +0000 UTC 2019-11-05 20:09:18 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 20:48:47 +0000 UTC 2019-11-05 20:09:18 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 20:48:47 +0000 UTC 2019-11-05 20:09:21 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.255.8} {Hostname k8s-master-97435445-3}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:266064173bfb4297898c6c25769e8b20,SystemUUID:E6E44241-5015-7740-9A81-A668A8427206,BootID:579d658c-b0ad-47f1-a993-1ccea2b55789,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 20:49:39.610] Nov 5 20:49:39.607: INFO: I1105 20:49:39.610] Logging kubelet events for node k8s-master-97435445-3 I1105 20:49:39.699] Nov 5 20:49:39.699: INFO: I1105 20:49:39.699] Logging pods the kubelet thinks is on node k8s-master-97435445-3 I1105 20:49:39.784] Nov 5 20:49:39.784: INFO: azure-ip-masq-agent-747cl started at 2019-11-05 20:09:21 +0000 UTC (0+1 container statuses recorded) I1105 20:49:39.784] Nov 5 20:49:39.784: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 20:49:39.784] Nov 5 20:49:39.784: INFO: kube-proxy-k45g8 started at 2019-11-05 20:09:21 +0000 UTC (0+1 container statuses recorded) I1105 20:49:39.784] Nov 5 20:49:39.784: INFO: Container kube-proxy ready: true, restart count 0 I1105 20:49:39.785] Nov 5 20:49:39.784: INFO: cloud-controller-manager-k8s-master-97435445-3 started at 2019-11-05 20:09:04 +0000 UTC (0+1 container statuses recorded) I1105 20:49:39.785] Nov 5 20:49:39.784: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 20:49:39.785] Nov 5 20:49:39.784: INFO: kube-addon-manager-k8s-master-97435445-3 started at 2019-11-05 20:09:05 +0000 UTC (0+1 container statuses recorded) I1105 20:49:39.785] Nov 5 20:49:39.784: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 20:49:39.785] Nov 5 20:49:39.784: INFO: kube-apiserver-k8s-master-97435445-3 started at 2019-11-05 20:09:05 +0000 UTC (0+1 container statuses recorded) I1105 20:49:39.785] Nov 5 20:49:39.784: INFO: Container kube-apiserver ready: true, restart count 0 I1105 20:49:39.786] Nov 5 20:49:39.784: INFO: kube-controller-manager-k8s-master-97435445-3 started at 2019-11-05 20:09:05 +0000 UTC (0+1 container statuses recorded) I1105 20:49:39.786] Nov 5 20:49:39.784: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 20:49:39.786] Nov 5 20:49:39.784: INFO: kube-scheduler-k8s-master-97435445-3 started at 2019-11-05 20:09:05 +0000 UTC (0+1 container statuses recorded) I1105 20:49:39.786] Nov 5 20:49:39.784: INFO: Container kube-scheduler ready: true, restart count 0 I1105 20:49:40.035] Nov 5 20:49:40.035: INFO: I1105 20:49:40.036] Latency metrics for node k8s-master-97435445-3 I1105 20:49:40.036] Nov 5 20:49:40.035: INFO: I1105 20:49:40.036] Logging node info for node k8s-master-97435445-4 I1105 20:49:40.117] Nov 5 20:49:40.115: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-97435445-4,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-97435445-4,UID:6d547b41-3f92-4cc4-a2d7-dda2071776a2,ResourceVersion:8384,Generation:0,CreationTimestamp:2019-11-05 20:08:36 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 1,kubernetes.azure.com/cluster: kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-97435445-4,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.3.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b/providers/Microsoft.Compute/virtualMachines/k8s-master-97435445-4,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 20:08:59 +0000 UTC 2019-11-05 20:08:59 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 20:49:01 +0000 UTC 2019-11-05 20:08:29 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 20:49:01 +0000 UTC 2019-11-05 20:08:29 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 20:49:01 +0000 UTC 2019-11-05 20:08:29 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 20:49:01 +0000 UTC 2019-11-05 20:08:36 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.255.9} {Hostname k8s-master-97435445-4}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:56b393c61eb34ba49ebda5279eeedb30,SystemUUID:FD239C23-618C-0049-8734-9AF3FDC9E3C1,BootID:f342ea44-e719-4513-807d-5def7ac606e2,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 20:49:40.118] Nov 5 20:49:40.115: INFO: I1105 20:49:40.118] Logging kubelet events for node k8s-master-97435445-4 I1105 20:49:40.199] Nov 5 20:49:40.199: INFO: I1105 20:49:40.199] Logging pods the kubelet thinks is on node k8s-master-97435445-4 I1105 20:49:40.283] Nov 5 20:49:40.282: INFO: cloud-controller-manager-k8s-master-97435445-4 started at 2019-11-05 20:08:46 +0000 UTC (0+1 container statuses recorded) I1105 20:49:40.283] Nov 5 20:49:40.282: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 20:49:40.283] Nov 5 20:49:40.282: INFO: kube-addon-manager-k8s-master-97435445-4 started at 2019-11-05 20:08:23 +0000 UTC (0+1 container statuses recorded) I1105 20:49:40.283] Nov 5 20:49:40.282: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 20:49:40.283] Nov 5 20:49:40.282: INFO: kube-apiserver-k8s-master-97435445-4 started at 2019-11-05 20:08:23 +0000 UTC (0+1 container statuses recorded) I1105 20:49:40.284] Nov 5 20:49:40.282: INFO: Container kube-apiserver ready: true, restart count 0 I1105 20:49:40.284] Nov 5 20:49:40.282: INFO: kube-controller-manager-k8s-master-97435445-4 started at 2019-11-05 20:08:23 +0000 UTC (0+1 container statuses recorded) I1105 20:49:40.284] Nov 5 20:49:40.282: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 20:49:40.284] Nov 5 20:49:40.282: INFO: kube-scheduler-k8s-master-97435445-4 started at 2019-11-05 20:08:23 +0000 UTC (0+1 container statuses recorded) I1105 20:49:40.284] Nov 5 20:49:40.282: INFO: Container kube-scheduler ready: true, restart count 0 I1105 20:49:40.284] Nov 5 20:49:40.282: INFO: kube-proxy-vfbfr started at 2019-11-05 20:08:36 +0000 UTC (0+1 container statuses recorded) I1105 20:49:40.285] Nov 5 20:49:40.282: INFO: Container kube-proxy ready: true, restart count 0 I1105 20:49:40.285] Nov 5 20:49:40.282: INFO: azure-ip-masq-agent-kmkll started at 2019-11-05 20:08:36 +0000 UTC (0+1 container statuses recorded) I1105 20:49:40.285] Nov 5 20:49:40.282: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 20:49:40.545] Nov 5 20:49:40.545: INFO: I1105 20:49:40.545] Latency metrics for node k8s-master-97435445-4 I1105 20:49:40.545] Nov 5 20:49:40.545: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready I1105 20:49:40.625] STEP: Destroying namespace "azuredisk-6880" for this suite. I1105 20:49:54.885] Nov 5 20:49:54.885: INFO: Waiting up to 30s for server preferred namespaced resources to be successfully discovered I1105 20:49:57.899] Nov 5 20:49:57.898: INFO: namespace azuredisk-6880 deletion completed in 17.27376004s I1105 20:49:57.899] I1105 20:49:57.899] • Failure [324.020 seconds] I1105 20:49:57.899] [azuredisk-csi-e2e] Dynamic Provisioning I1105 20:49:57.899] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:33 I1105 20:49:57.899] [multi-az] I1105 20:49:57.899] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:40 I1105 20:49:57.900] should create multiple PV objects, bind to PVCs and attach all to different pods on the same node [It] I1105 20:49:57.900] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:151 I1105 20:49:57.900] I1105 20:49:57.900] Unexpected error: I1105 20:49:57.900] <*errors.errorString | 0xc000ce7640>: { I1105 20:49:57.900] s: "PersistentVolumeClaims [pvc-m2g49] not all in phase Bound within 5m0s", I1105 20:49:57.901] } I1105 20:49:57.901] PersistentVolumeClaims [pvc-m2g49] not all in phase Bound within 5m0s I1105 20:49:57.901] occurred I1105 20:49:57.901] I1105 20:49:57.901] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/testsuites/testsuites.go:278 I1105 20:49:57.901] ------------------------------ I1105 20:49:57.901] [azuredisk-csi-e2e] Dynamic Provisioning [single-az] I1105 20:49:57.902] should create a deployment object, write and read to it, delete the pod and write and read to it again I1105 20:49:57.902] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:193 I1105 20:49:57.902] [BeforeEach] [single-az] I1105 20:49:57.902] /go/src/sigs.k8s.io/azuredisk-csi-driver/vendor/k8s.io/kubernetes/test/e2e/framework/framework.go:150 I1105 20:49:57.902] STEP: Creating a kubernetes client I1105 20:49:57.902] Nov 5 20:49:57.899: INFO: >>> kubeConfig: /workspace/aks971561099/kubeconfig/kubeconfig.eastus2.json I1105 20:49:57.903] STEP: Building a namespace api object, basename azuredisk I1105 20:49:58.139] STEP: Binding the e2e-test-privileged-psp PodSecurityPolicy to the default service account in azuredisk-8958 I1105 20:49:58.408] STEP: Waiting for a default service account to be provisioned in namespace I1105 20:49:58.485] [BeforeEach] [single-az] I1105 20:49:58.485] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:58 I1105 20:49:58.485] [It] should create a deployment object, write and read to it, delete the pod and write and read to it again I1105 20:49:58.485] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:193 I1105 20:49:58.485] STEP: setting up the StorageClass I1105 20:49:58.485] STEP: creating a StorageClass I1105 20:49:58.567] STEP: setting up the PVC I1105 20:49:58.568] STEP: creating a PVC I1105 20:49:58.655] STEP: waiting for PVC to be in phase "Bound" I1105 20:49:58.655] Nov 5 20:49:58.655: INFO: Waiting up to 5m0s for PersistentVolumeClaims [pvc-6nqs8] to have phase Bound I1105 20:49:58.734] Nov 5 20:49:58.734: INFO: PersistentVolumeClaim pvc-6nqs8 found but phase is Pending instead of Bound. I1105 20:50:00.813] Nov 5 20:50:00.813: INFO: PersistentVolumeClaim pvc-6nqs8 found but phase is Pending instead of Bound. I1105 20:50:02.892] Nov 5 20:50:02.892: INFO: PersistentVolumeClaim pvc-6nqs8 found but phase is Pending instead of Bound. I1105 20:50:04.971] Nov 5 20:50:04.971: INFO: PersistentVolumeClaim pvc-6nqs8 found but phase is Pending instead of Bound. I1105 20:50:07.057] Nov 5 20:50:07.057: INFO: PersistentVolumeClaim pvc-6nqs8 found but phase is Pending instead of Bound. I1105 20:50:09.136] Nov 5 20:50:09.136: INFO: PersistentVolumeClaim pvc-6nqs8 found but phase is Pending instead of Bound. I1105 20:50:11.210] Nov 5 20:50:11.210: INFO: PersistentVolumeClaim pvc-6nqs8 found but phase is Pending instead of Bound. I1105 20:50:13.285] Nov 5 20:50:13.285: INFO: PersistentVolumeClaim pvc-6nqs8 found but phase is Pending instead of Bound. I1105 20:50:15.359] Nov 5 20:50:15.359: INFO: PersistentVolumeClaim pvc-6nqs8 found but phase is Pending instead of Bound. I1105 20:50:17.434] Nov 5 20:50:17.433: INFO: PersistentVolumeClaim pvc-6nqs8 found but phase is Pending instead of Bound. I1105 20:50:19.508] Nov 5 20:50:19.508: INFO: PersistentVolumeClaim pvc-6nqs8 found but phase is Pending instead of Bound. I1105 20:50:21.582] Nov 5 20:50:21.582: INFO: PersistentVolumeClaim pvc-6nqs8 found but phase is Pending instead of Bound. I1105 20:50:23.656] Nov 5 20:50:23.656: INFO: PersistentVolumeClaim pvc-6nqs8 found but phase is Pending instead of Bound. I1105 20:50:25.730] Nov 5 20:50:25.730: INFO: PersistentVolumeClaim pvc-6nqs8 found but phase is Pending instead of Bound. I1105 20:50:27.805] Nov 5 20:50:27.804: INFO: PersistentVolumeClaim pvc-6nqs8 found but phase is Pending instead of Bound. I1105 20:50:29.879] Nov 5 20:50:29.879: INFO: PersistentVolumeClaim pvc-6nqs8 found but phase is Pending instead of Bound. I1105 20:50:31.953] Nov 5 20:50:31.953: INFO: PersistentVolumeClaim pvc-6nqs8 found but phase is Pending instead of Bound. I1105 20:50:34.027] Nov 5 20:50:34.027: INFO: PersistentVolumeClaim pvc-6nqs8 found but phase is Pending instead of Bound. I1105 20:50:36.101] Nov 5 20:50:36.101: INFO: PersistentVolumeClaim pvc-6nqs8 found but phase is Pending instead of Bound. I1105 20:50:38.210] Nov 5 20:50:38.209: INFO: PersistentVolumeClaim pvc-6nqs8 found but phase is Pending instead of Bound. I1105 20:50:40.283] Nov 5 20:50:40.283: INFO: PersistentVolumeClaim pvc-6nqs8 found but phase is Pending instead of Bound. I1105 20:50:42.358] Nov 5 20:50:42.357: INFO: PersistentVolumeClaim pvc-6nqs8 found but phase is Pending instead of Bound. I1105 20:50:44.432] Nov 5 20:50:44.432: INFO: PersistentVolumeClaim pvc-6nqs8 found but phase is Pending instead of Bound. I1105 20:50:46.511] Nov 5 20:50:46.510: INFO: PersistentVolumeClaim pvc-6nqs8 found but phase is Pending instead of Bound. I1105 20:50:48.584] Nov 5 20:50:48.584: INFO: PersistentVolumeClaim pvc-6nqs8 found but phase is Pending instead of Bound. I1105 20:50:50.658] Nov 5 20:50:50.658: INFO: PersistentVolumeClaim pvc-6nqs8 found but phase is Pending instead of Bound. I1105 20:50:52.734] Nov 5 20:50:52.734: INFO: PersistentVolumeClaim pvc-6nqs8 found but phase is Pending instead of Bound. I1105 20:50:54.808] Nov 5 20:50:54.808: INFO: PersistentVolumeClaim pvc-6nqs8 found but phase is Pending instead of Bound. I1105 20:50:56.882] Nov 5 20:50:56.882: INFO: PersistentVolumeClaim pvc-6nqs8 found but phase is Pending instead of Bound. I1105 20:50:58.957] Nov 5 20:50:58.957: INFO: PersistentVolumeClaim pvc-6nqs8 found but phase is Pending instead of Bound. I1105 20:51:01.032] Nov 5 20:51:01.031: INFO: PersistentVolumeClaim pvc-6nqs8 found but phase is Pending instead of Bound. I1105 20:51:03.106] Nov 5 20:51:03.106: INFO: PersistentVolumeClaim pvc-6nqs8 found but phase is Pending instead of Bound. I1105 20:51:05.180] Nov 5 20:51:05.180: INFO: PersistentVolumeClaim pvc-6nqs8 found but phase is Pending instead of Bound. I1105 20:51:07.254] Nov 5 20:51:07.254: INFO: PersistentVolumeClaim pvc-6nqs8 found but phase is Pending instead of Bound. I1105 20:51:09.329] Nov 5 20:51:09.329: INFO: PersistentVolumeClaim pvc-6nqs8 found but phase is Pending instead of Bound. I1105 20:51:11.403] Nov 5 20:51:11.403: INFO: PersistentVolumeClaim pvc-6nqs8 found but phase is Pending instead of Bound. I1105 20:51:13.476] Nov 5 20:51:13.476: INFO: PersistentVolumeClaim pvc-6nqs8 found but phase is Pending instead of Bound. I1105 20:51:15.551] Nov 5 20:51:15.551: INFO: PersistentVolumeClaim pvc-6nqs8 found but phase is Pending instead of Bound. I1105 20:51:17.625] Nov 5 20:51:17.625: INFO: PersistentVolumeClaim pvc-6nqs8 found but phase is Pending instead of Bound. I1105 20:51:19.700] Nov 5 20:51:19.699: INFO: PersistentVolumeClaim pvc-6nqs8 found but phase is Pending instead of Bound. I1105 20:51:21.774] Nov 5 20:51:21.774: INFO: PersistentVolumeClaim pvc-6nqs8 found but phase is Pending instead of Bound. I1105 20:51:23.848] Nov 5 20:51:23.848: INFO: PersistentVolumeClaim pvc-6nqs8 found but phase is Pending instead of Bound. I1105 20:51:25.922] Nov 5 20:51:25.922: INFO: PersistentVolumeClaim pvc-6nqs8 found but phase is Pending instead of Bound. I1105 20:51:27.996] Nov 5 20:51:27.996: INFO: PersistentVolumeClaim pvc-6nqs8 found but phase is Pending instead of Bound. I1105 20:51:30.071] Nov 5 20:51:30.070: INFO: PersistentVolumeClaim pvc-6nqs8 found but phase is Pending instead of Bound. I1105 20:51:32.145] Nov 5 20:51:32.145: INFO: PersistentVolumeClaim pvc-6nqs8 found but phase is Pending instead of Bound. I1105 20:51:34.220] Nov 5 20:51:34.220: INFO: PersistentVolumeClaim pvc-6nqs8 found but phase is Pending instead of Bound. I1105 20:51:36.294] Nov 5 20:51:36.294: INFO: PersistentVolumeClaim pvc-6nqs8 found but phase is Pending instead of Bound. I1105 20:51:38.369] Nov 5 20:51:38.369: INFO: PersistentVolumeClaim pvc-6nqs8 found but phase is Pending instead of Bound. I1105 20:51:40.443] Nov 5 20:51:40.443: INFO: PersistentVolumeClaim pvc-6nqs8 found but phase is Pending instead of Bound. I1105 20:51:42.518] Nov 5 20:51:42.518: INFO: PersistentVolumeClaim pvc-6nqs8 found but phase is Pending instead of Bound. I1105 20:51:44.594] Nov 5 20:51:44.593: INFO: PersistentVolumeClaim pvc-6nqs8 found but phase is Pending instead of Bound. I1105 20:51:46.667] Nov 5 20:51:46.667: INFO: PersistentVolumeClaim pvc-6nqs8 found but phase is Pending instead of Bound. I1105 20:51:48.741] Nov 5 20:51:48.741: INFO: PersistentVolumeClaim pvc-6nqs8 found but phase is Pending instead of Bound. I1105 20:51:50.816] Nov 5 20:51:50.816: INFO: PersistentVolumeClaim pvc-6nqs8 found but phase is Pending instead of Bound. I1105 20:51:52.890] Nov 5 20:51:52.890: INFO: PersistentVolumeClaim pvc-6nqs8 found but phase is Pending instead of Bound. I1105 20:51:54.964] Nov 5 20:51:54.964: INFO: PersistentVolumeClaim pvc-6nqs8 found but phase is Pending instead of Bound. I1105 20:51:57.038] Nov 5 20:51:57.038: INFO: PersistentVolumeClaim pvc-6nqs8 found but phase is Pending instead of Bound. I1105 20:51:59.113] Nov 5 20:51:59.112: INFO: PersistentVolumeClaim pvc-6nqs8 found but phase is Pending instead of Bound. I1105 20:52:01.186] Nov 5 20:52:01.186: INFO: PersistentVolumeClaim pvc-6nqs8 found but phase is Pending instead of Bound. I1105 20:52:03.261] Nov 5 20:52:03.260: INFO: PersistentVolumeClaim pvc-6nqs8 found but phase is Pending instead of Bound. I1105 20:52:05.335] Nov 5 20:52:05.334: INFO: PersistentVolumeClaim pvc-6nqs8 found but phase is Pending instead of Bound. I1105 20:52:07.409] Nov 5 20:52:07.409: INFO: PersistentVolumeClaim pvc-6nqs8 found but phase is Pending instead of Bound. I1105 20:52:09.484] Nov 5 20:52:09.483: INFO: PersistentVolumeClaim pvc-6nqs8 found but phase is Pending instead of Bound. I1105 20:52:11.558] Nov 5 20:52:11.557: INFO: PersistentVolumeClaim pvc-6nqs8 found but phase is Pending instead of Bound. I1105 20:52:13.632] Nov 5 20:52:13.632: INFO: PersistentVolumeClaim pvc-6nqs8 found but phase is Pending instead of Bound. I1105 20:52:15.706] Nov 5 20:52:15.706: INFO: PersistentVolumeClaim pvc-6nqs8 found but phase is Pending instead of Bound. I1105 20:52:17.780] Nov 5 20:52:17.780: INFO: PersistentVolumeClaim pvc-6nqs8 found but phase is Pending instead of Bound. I1105 20:52:19.854] Nov 5 20:52:19.854: INFO: PersistentVolumeClaim pvc-6nqs8 found but phase is Pending instead of Bound. I1105 20:52:21.928] Nov 5 20:52:21.928: INFO: PersistentVolumeClaim pvc-6nqs8 found but phase is Pending instead of Bound. I1105 20:52:24.002] Nov 5 20:52:24.002: INFO: PersistentVolumeClaim pvc-6nqs8 found but phase is Pending instead of Bound. I1105 20:52:26.077] Nov 5 20:52:26.076: INFO: PersistentVolumeClaim pvc-6nqs8 found but phase is Pending instead of Bound. I1105 20:52:28.151] Nov 5 20:52:28.151: INFO: PersistentVolumeClaim pvc-6nqs8 found but phase is Pending instead of Bound. I1105 20:52:30.225] Nov 5 20:52:30.224: INFO: PersistentVolumeClaim pvc-6nqs8 found but phase is Pending instead of Bound. I1105 20:52:32.300] Nov 5 20:52:32.299: INFO: PersistentVolumeClaim pvc-6nqs8 found but phase is Pending instead of Bound. I1105 20:52:34.374] Nov 5 20:52:34.374: INFO: PersistentVolumeClaim pvc-6nqs8 found but phase is Pending instead of Bound. I1105 20:52:36.448] Nov 5 20:52:36.448: INFO: PersistentVolumeClaim pvc-6nqs8 found but phase is Pending instead of Bound. I1105 20:52:38.522] Nov 5 20:52:38.522: INFO: PersistentVolumeClaim pvc-6nqs8 found but phase is Pending instead of Bound. I1105 20:52:40.596] Nov 5 20:52:40.596: INFO: PersistentVolumeClaim pvc-6nqs8 found but phase is Pending instead of Bound. I1105 20:52:42.670] Nov 5 20:52:42.670: INFO: PersistentVolumeClaim pvc-6nqs8 found but phase is Pending instead of Bound. I1105 20:52:44.744] Nov 5 20:52:44.744: INFO: PersistentVolumeClaim pvc-6nqs8 found but phase is Pending instead of Bound. I1105 20:52:46.819] Nov 5 20:52:46.819: INFO: PersistentVolumeClaim pvc-6nqs8 found but phase is Pending instead of Bound. I1105 20:52:48.893] Nov 5 20:52:48.893: INFO: PersistentVolumeClaim pvc-6nqs8 found but phase is Pending instead of Bound. I1105 20:52:50.967] Nov 5 20:52:50.967: INFO: PersistentVolumeClaim pvc-6nqs8 found but phase is Pending instead of Bound. I1105 20:52:53.041] Nov 5 20:52:53.041: INFO: PersistentVolumeClaim pvc-6nqs8 found but phase is Pending instead of Bound. I1105 20:52:55.115] Nov 5 20:52:55.115: INFO: PersistentVolumeClaim pvc-6nqs8 found but phase is Pending instead of Bound. I1105 20:52:57.190] Nov 5 20:52:57.189: INFO: PersistentVolumeClaim pvc-6nqs8 found but phase is Pending instead of Bound. I1105 20:52:59.264] Nov 5 20:52:59.264: INFO: PersistentVolumeClaim pvc-6nqs8 found but phase is Pending instead of Bound. I1105 20:53:01.338] Nov 5 20:53:01.337: INFO: PersistentVolumeClaim pvc-6nqs8 found but phase is Pending instead of Bound. I1105 20:53:03.412] Nov 5 20:53:03.412: INFO: PersistentVolumeClaim pvc-6nqs8 found but phase is Pending instead of Bound. I1105 20:53:05.491] Nov 5 20:53:05.490: INFO: PersistentVolumeClaim pvc-6nqs8 found but phase is Pending instead of Bound. I1105 20:53:07.569] Nov 5 20:53:07.569: INFO: PersistentVolumeClaim pvc-6nqs8 found but phase is Pending instead of Bound. I1105 20:53:09.648] Nov 5 20:53:09.648: INFO: PersistentVolumeClaim pvc-6nqs8 found but phase is Pending instead of Bound. I1105 20:53:11.728] Nov 5 20:53:11.728: INFO: PersistentVolumeClaim pvc-6nqs8 found but phase is Pending instead of Bound. I1105 20:53:13.807] Nov 5 20:53:13.806: INFO: PersistentVolumeClaim pvc-6nqs8 found but phase is Pending instead of Bound. I1105 20:53:15.885] Nov 5 20:53:15.885: INFO: PersistentVolumeClaim pvc-6nqs8 found but phase is Pending instead of Bound. I1105 20:53:17.965] Nov 5 20:53:17.965: INFO: PersistentVolumeClaim pvc-6nqs8 found but phase is Pending instead of Bound. I1105 20:53:20.044] Nov 5 20:53:20.044: INFO: PersistentVolumeClaim pvc-6nqs8 found but phase is Pending instead of Bound. I1105 20:53:22.123] Nov 5 20:53:22.123: INFO: PersistentVolumeClaim pvc-6nqs8 found but phase is Pending instead of Bound. I1105 20:53:24.202] Nov 5 20:53:24.202: INFO: PersistentVolumeClaim pvc-6nqs8 found but phase is Pending instead of Bound. I1105 20:53:26.282] Nov 5 20:53:26.282: INFO: PersistentVolumeClaim pvc-6nqs8 found but phase is Pending instead of Bound. I1105 20:53:28.361] Nov 5 20:53:28.361: INFO: PersistentVolumeClaim pvc-6nqs8 found but phase is Pending instead of Bound. I1105 20:53:30.442] Nov 5 20:53:30.442: INFO: PersistentVolumeClaim pvc-6nqs8 found but phase is Pending instead of Bound. I1105 20:53:32.521] Nov 5 20:53:32.521: INFO: PersistentVolumeClaim pvc-6nqs8 found but phase is Pending instead of Bound. I1105 20:53:34.628] Nov 5 20:53:34.627: INFO: PersistentVolumeClaim pvc-6nqs8 found but phase is Pending instead of Bound. I1105 20:53:36.713] Nov 5 20:53:36.713: INFO: PersistentVolumeClaim pvc-6nqs8 found but phase is Pending instead of Bound. I1105 20:53:38.792] Nov 5 20:53:38.792: INFO: PersistentVolumeClaim pvc-6nqs8 found but phase is Pending instead of Bound. I1105 20:53:40.871] Nov 5 20:53:40.871: INFO: PersistentVolumeClaim pvc-6nqs8 found but phase is Pending instead of Bound. I1105 20:53:42.950] Nov 5 20:53:42.950: INFO: PersistentVolumeClaim pvc-6nqs8 found but phase is Pending instead of Bound. I1105 20:53:45.029] Nov 5 20:53:45.029: INFO: PersistentVolumeClaim pvc-6nqs8 found but phase is Pending instead of Bound. I1105 20:53:47.108] Nov 5 20:53:47.108: INFO: PersistentVolumeClaim pvc-6nqs8 found but phase is Pending instead of Bound. I1105 20:53:49.188] Nov 5 20:53:49.187: INFO: PersistentVolumeClaim pvc-6nqs8 found but phase is Pending instead of Bound. I1105 20:53:51.267] Nov 5 20:53:51.267: INFO: PersistentVolumeClaim pvc-6nqs8 found but phase is Pending instead of Bound. I1105 20:53:53.346] Nov 5 20:53:53.346: INFO: PersistentVolumeClaim pvc-6nqs8 found but phase is Pending instead of Bound. I1105 20:53:55.426] Nov 5 20:53:55.426: INFO: PersistentVolumeClaim pvc-6nqs8 found but phase is Pending instead of Bound. I1105 20:53:57.506] Nov 5 20:53:57.506: INFO: PersistentVolumeClaim pvc-6nqs8 found but phase is Pending instead of Bound. I1105 20:53:59.585] Nov 5 20:53:59.585: INFO: PersistentVolumeClaim pvc-6nqs8 found but phase is Pending instead of Bound. I1105 20:54:01.664] Nov 5 20:54:01.664: INFO: PersistentVolumeClaim pvc-6nqs8 found but phase is Pending instead of Bound. I1105 20:54:03.743] Nov 5 20:54:03.743: INFO: PersistentVolumeClaim pvc-6nqs8 found but phase is Pending instead of Bound. I1105 20:54:05.823] Nov 5 20:54:05.823: INFO: PersistentVolumeClaim pvc-6nqs8 found but phase is Pending instead of Bound. I1105 20:54:07.903] Nov 5 20:54:07.902: INFO: PersistentVolumeClaim pvc-6nqs8 found but phase is Pending instead of Bound. I1105 20:54:09.982] Nov 5 20:54:09.981: INFO: PersistentVolumeClaim pvc-6nqs8 found but phase is Pending instead of Bound. I1105 20:54:12.061] Nov 5 20:54:12.060: INFO: PersistentVolumeClaim pvc-6nqs8 found but phase is Pending instead of Bound. I1105 20:54:14.140] Nov 5 20:54:14.140: INFO: PersistentVolumeClaim pvc-6nqs8 found but phase is Pending instead of Bound. I1105 20:54:16.223] Nov 5 20:54:16.223: INFO: PersistentVolumeClaim pvc-6nqs8 found but phase is Pending instead of Bound. I1105 20:54:18.302] Nov 5 20:54:18.302: INFO: PersistentVolumeClaim pvc-6nqs8 found but phase is Pending instead of Bound. I1105 20:54:20.381] Nov 5 20:54:20.381: INFO: PersistentVolumeClaim pvc-6nqs8 found but phase is Pending instead of Bound. I1105 20:54:22.460] Nov 5 20:54:22.460: INFO: PersistentVolumeClaim pvc-6nqs8 found but phase is Pending instead of Bound. I1105 20:54:24.539] Nov 5 20:54:24.539: INFO: PersistentVolumeClaim pvc-6nqs8 found but phase is Pending instead of Bound. I1105 20:54:26.619] Nov 5 20:54:26.619: INFO: PersistentVolumeClaim pvc-6nqs8 found but phase is Pending instead of Bound. I1105 20:54:28.705] Nov 5 20:54:28.705: INFO: PersistentVolumeClaim pvc-6nqs8 found but phase is Pending instead of Bound. I1105 20:54:30.784] Nov 5 20:54:30.784: INFO: PersistentVolumeClaim pvc-6nqs8 found but phase is Pending instead of Bound. I1105 20:54:32.863] Nov 5 20:54:32.863: INFO: PersistentVolumeClaim pvc-6nqs8 found but phase is Pending instead of Bound. I1105 20:54:34.946] Nov 5 20:54:34.946: INFO: PersistentVolumeClaim pvc-6nqs8 found but phase is Pending instead of Bound. I1105 20:54:37.026] Nov 5 20:54:37.026: INFO: PersistentVolumeClaim pvc-6nqs8 found but phase is Pending instead of Bound. I1105 20:54:39.106] Nov 5 20:54:39.106: INFO: PersistentVolumeClaim pvc-6nqs8 found but phase is Pending instead of Bound. I1105 20:54:41.185] Nov 5 20:54:41.184: INFO: PersistentVolumeClaim pvc-6nqs8 found but phase is Pending instead of Bound. I1105 20:54:43.268] Nov 5 20:54:43.268: INFO: PersistentVolumeClaim pvc-6nqs8 found but phase is Pending instead of Bound. I1105 20:54:45.348] Nov 5 20:54:45.348: INFO: PersistentVolumeClaim pvc-6nqs8 found but phase is Pending instead of Bound. I1105 20:54:47.427] Nov 5 20:54:47.427: INFO: PersistentVolumeClaim pvc-6nqs8 found but phase is Pending instead of Bound. I1105 20:54:49.507] Nov 5 20:54:49.506: INFO: PersistentVolumeClaim pvc-6nqs8 found but phase is Pending instead of Bound. I1105 20:54:51.586] Nov 5 20:54:51.586: INFO: PersistentVolumeClaim pvc-6nqs8 found but phase is Pending instead of Bound. I1105 20:54:53.665] Nov 5 20:54:53.665: INFO: PersistentVolumeClaim pvc-6nqs8 found but phase is Pending instead of Bound. I1105 20:54:55.745] Nov 5 20:54:55.744: INFO: PersistentVolumeClaim pvc-6nqs8 found but phase is Pending instead of Bound. I1105 20:54:57.823] Nov 5 20:54:57.823: INFO: PersistentVolumeClaim pvc-6nqs8 found but phase is Pending instead of Bound. I1105 20:54:59.824] Nov 5 20:54:59.823: INFO: Unexpected error occurred: PersistentVolumeClaims [pvc-6nqs8] not all in phase Bound within 5m0s I1105 20:54:59.824] [AfterEach] [single-az] I1105 20:54:59.824] /go/src/sigs.k8s.io/azuredisk-csi-driver/vendor/k8s.io/kubernetes/test/e2e/framework/framework.go:151 I1105 20:54:59.825] STEP: Collecting events from namespace "azuredisk-8958". I1105 20:54:59.904] STEP: Found 3 events. I1105 20:54:59.904] Nov 5 20:54:59.904: INFO: At 2019-11-05 20:49:58 +0000 UTC - event for pvc-6nqs8: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "disk.csi.azure.com" or manually created by system administrator I1105 20:54:59.904] Nov 5 20:54:59.904: INFO: At 2019-11-05 20:49:58 +0000 UTC - event for pvc-6nqs8: {disk.csi.azure.com_csi-azuredisk-controller-6f6587bfc6-62nrt_3930bc69-0009-11ea-8cf5-829a27ff7d5a } Provisioning: External provisioner is provisioning volume for claim "azuredisk-8958/pvc-6nqs8" I1105 20:54:59.905] Nov 5 20:54:59.904: INFO: At 2019-11-05 20:49:58 +0000 UTC - event for pvc-6nqs8: {disk.csi.azure.com_csi-azuredisk-controller-6f6587bfc6-62nrt_3930bc69-0009-11ea-8cf5-829a27ff7d5a } ProvisioningFailed: failed to provision volume with StorageClass "azuredisk-8958-disk.csi.azure.com-dynamic-sc-jphrz": rpc error: code = Unknown desc = compute.DisksClient#CreateOrUpdate: Failure sending request: StatusCode=400 -- Original Error: Code="InvalidAvailabilityZone" Message="The zone(s) '' for resource 'Microsoft.Compute/disks/pvc-8a9d1e51-fbae-4b8e-86b0-e707a779262a' is not supported. The supported zones for location 'eastus2' are '1,2,3'" I1105 20:54:59.982] Nov 5 20:54:59.982: INFO: POD NODE PHASE GRACE CONDITIONS I1105 20:54:59.982] Nov 5 20:54:59.982: INFO: I1105 20:55:00.218] Nov 5 20:55:00.217: INFO: I1105 20:55:00.218] Logging node info for node k8s-agentpool-97435445-vmss000000 I1105 20:55:00.299] Nov 5 20:55:00.296: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-agentpool-97435445-vmss000000,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-agentpool-97435445-vmss000000,UID:e955dc47-8c8f-4142-9241-3d409f32e42d,ResourceVersion:9403,Generation:0,CreationTimestamp:2019-11-05 20:07:34 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{agentpool: agentpool,beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_D4s_v3,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 0,kubernetes.azure.com/cluster: kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b,kubernetes.azure.com/role: agent,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-agentpool-97435445-vmss000000,kubernetes.io/os: linux,kubernetes.io/role: agent,node-role.kubernetes.io/agent: ,storageprofile: managed,storagetier: Premium_LRS,topology.disk.csi.azure.com/zone: 0,},Annotations:map[string]string{csi.volume.kubernetes.io/nodeid: {"disk.csi.azure.com":"k8s-agentpool-97435445-vmss000000"},node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.2.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool-97435445-vmss/virtualMachines/0,Unschedulable:false,Taints:[],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-csi-disk.csi.azure.com: {{8 0} {} 8 DecimalSI},cpu: {{4 0} {} 4 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{16797040640 0} {} BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-csi-disk.csi.azure.com: {{8 0} {} 8 DecimalSI},cpu: {{4 0} {} 4 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{16010608640 0} {} BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 20:08:01 +0000 UTC 2019-11-05 20:08:01 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 20:54:43 +0000 UTC 2019-11-05 20:07:27 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 20:54:43 +0000 UTC 2019-11-05 20:07:27 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 20:54:43 +0000 UTC 2019-11-05 20:07:27 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 20:54:43 +0000 UTC 2019-11-05 20:07:55 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.0.4} {Hostname k8s-agentpool-97435445-vmss000000}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:bfb55325809340228149d2d93f50df58,SystemUUID:CB2ED036-0439-F641-BC7F-640C3638C027,BootID:a917f4d1-549d-471d-b7dc-47ae1715d97d,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[k8sprowinternal.azurecr.io/azuredisk-csi@sha256:75d4ff5d7cd845d6e810c67796ecb037e852fb849ec1eb5b3194564008e6d1e0 k8sprowinternal.azurecr.io/azuredisk-csi:e2e-3e3ed045c6faa509dbd8d2890ee9f56aa2934887] 286930414} {[k8s.gcr.io/kubernetes-dashboard-amd64@sha256:0ae6b69432e78069c5ce2bcde0fe409c5c4d6f0f4d9cd50a17974fea38898747 k8s.gcr.io/kubernetes-dashboard-amd64:v1.10.1] 121711221} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/metrics-server-amd64@sha256:49a9f12f7067d11f42c803dbe61ed2c1299959ad85cb315b25ff7eef8e6b8892 k8s.gcr.io/metrics-server-amd64:v0.2.1] 42541759} {[k8s.gcr.io/coredns@sha256:e83beb5e43f8513fa735e77ffc5859640baea30a882a11cc75c4c3244a737d3c k8s.gcr.io/coredns:1.5.0] 42488424} {[mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume@sha256:4fd30d43947d4a54fc89ead7985beecfd3c9b2a93a0655a373b1608ab90bd5af mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume:v0.0.7] 22909487} {[quay.io/k8scsi/livenessprobe@sha256:8c9c8a0f330acc2e558ae6624309447d02522e76aa35af23fa19b88abf4381cb quay.io/k8scsi/livenessprobe:v1.0.2] 17415381} {[quay.io/k8scsi/csi-node-driver-registrar@sha256:13daf82fb99e951a4bff8ae5fc7c17c3a8fe7130be6400990d8f6076c32d4599 quay.io/k8scsi/csi-node-driver-registrar:v1.1.0] 15815995} {[mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume@sha256:23d8c6033f02a1ecad05127ebdc931bb871264228661bc122704b0974e4d9fdd mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume:1.0.8] 1159025} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 20:55:00.300] Nov 5 20:55:00.296: INFO: I1105 20:55:00.300] Logging kubelet events for node k8s-agentpool-97435445-vmss000000 I1105 20:55:00.380] Nov 5 20:55:00.380: INFO: I1105 20:55:00.380] Logging pods the kubelet thinks is on node k8s-agentpool-97435445-vmss000000 I1105 20:55:00.468] Nov 5 20:55:00.468: INFO: coredns-7f68dcdbdb-zh8g5 started at 2019-11-05 20:08:01 +0000 UTC (0+1 container statuses recorded) I1105 20:55:00.469] Nov 5 20:55:00.468: INFO: Container coredns ready: true, restart count 0 I1105 20:55:00.469] Nov 5 20:55:00.468: INFO: azure-ip-masq-agent-m7tv9 started at 2019-11-05 20:08:01 +0000 UTC (0+1 container statuses recorded) I1105 20:55:00.469] Nov 5 20:55:00.468: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 20:55:00.469] Nov 5 20:55:00.468: INFO: keyvault-flexvolume-smspt started at 2019-11-05 20:08:01 +0000 UTC (0+1 container statuses recorded) I1105 20:55:00.469] Nov 5 20:55:00.468: INFO: Container keyvault-flexvolume ready: true, restart count 0 I1105 20:55:00.469] Nov 5 20:55:00.468: INFO: metrics-server-864ffbc5c-wl84z started at 2019-11-05 20:08:01 +0000 UTC (0+1 container statuses recorded) I1105 20:55:00.469] Nov 5 20:55:00.468: INFO: Container metrics-server ready: true, restart count 0 I1105 20:55:00.470] Nov 5 20:55:00.468: INFO: kube-proxy-8mt7f started at 2019-11-05 20:08:01 +0000 UTC (0+1 container statuses recorded) I1105 20:55:00.470] Nov 5 20:55:00.468: INFO: Container kube-proxy ready: true, restart count 0 I1105 20:55:00.470] Nov 5 20:55:00.468: INFO: kubernetes-dashboard-66dd8b8df7-t6j4l started at 2019-11-05 20:08:01 +0000 UTC (0+1 container statuses recorded) I1105 20:55:00.470] Nov 5 20:55:00.468: INFO: Container kubernetes-dashboard ready: true, restart count 0 I1105 20:55:00.470] Nov 5 20:55:00.468: INFO: csi-azuredisk-node-5wwhz started at 2019-11-05 20:16:09 +0000 UTC (0+3 container statuses recorded) I1105 20:55:00.470] Nov 5 20:55:00.468: INFO: Container azuredisk ready: true, restart count 0 I1105 20:55:00.470] Nov 5 20:55:00.468: INFO: Container liveness-probe ready: true, restart count 0 I1105 20:55:00.471] Nov 5 20:55:00.468: INFO: Container node-driver-registrar ready: true, restart count 0 I1105 20:55:00.471] Nov 5 20:55:00.468: INFO: blobfuse-flexvol-installer-p7hdb started at 2019-11-05 20:08:01 +0000 UTC (0+1 container statuses recorded) I1105 20:55:00.471] Nov 5 20:55:00.468: INFO: Container blobfuse-flexvol-installer ready: true, restart count 0 I1105 20:55:00.741] Nov 5 20:55:00.740: INFO: I1105 20:55:00.741] Latency metrics for node k8s-agentpool-97435445-vmss000000 I1105 20:55:00.741] Nov 5 20:55:00.741: INFO: I1105 20:55:00.741] Logging node info for node k8s-agentpool-97435445-vmss000001 I1105 20:55:00.823] Nov 5 20:55:00.819: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-agentpool-97435445-vmss000001,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-agentpool-97435445-vmss000001,UID:75d3cc50-ca6d-45e0-a42a-6090e8d537dd,ResourceVersion:9398,Generation:0,CreationTimestamp:2019-11-05 20:07:34 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{agentpool: agentpool,beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_D4s_v3,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 1,kubernetes.azure.com/cluster: kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b,kubernetes.azure.com/role: agent,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-agentpool-97435445-vmss000001,kubernetes.io/os: linux,kubernetes.io/role: agent,node-role.kubernetes.io/agent: ,storageprofile: managed,storagetier: Premium_LRS,topology.disk.csi.azure.com/zone: 1,},Annotations:map[string]string{csi.volume.kubernetes.io/nodeid: {"disk.csi.azure.com":"k8s-agentpool-97435445-vmss000001"},node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.0.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool-97435445-vmss/virtualMachines/1,Unschedulable:false,Taints:[],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-csi-disk.csi.azure.com: {{8 0} {} 8 DecimalSI},cpu: {{4 0} {} 4 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{16797044736 0} {} 16403364Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-csi-disk.csi.azure.com: {{8 0} {} 8 DecimalSI},cpu: {{4 0} {} 4 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{16010612736 0} {} 15635364Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 20:08:25 +0000 UTC 2019-11-05 20:08:25 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 20:54:42 +0000 UTC 2019-11-05 20:07:26 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 20:54:42 +0000 UTC 2019-11-05 20:07:26 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 20:54:42 +0000 UTC 2019-11-05 20:07:26 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 20:54:42 +0000 UTC 2019-11-05 20:07:55 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.0.5} {Hostname k8s-agentpool-97435445-vmss000001}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:e78def54b2f545c6b94d88bbba523c60,SystemUUID:40AE8188-86AB-D848-84A4-D9254E1258BC,BootID:8587ec06-f4f4-43c1-86cd-06c295b05b63,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[k8sprowinternal.azurecr.io/azuredisk-csi@sha256:75d4ff5d7cd845d6e810c67796ecb037e852fb849ec1eb5b3194564008e6d1e0 k8sprowinternal.azurecr.io/azuredisk-csi:e2e-3e3ed045c6faa509dbd8d2890ee9f56aa2934887] 286930414} {[gcr.io/kubernetes-helm/tiller@sha256:f6d8f4ab9ba993b5f5b60a6edafe86352eabe474ffeb84cb6c79b8866dce45d1 gcr.io/kubernetes-helm/tiller:v2.11.0] 71821984} {[quay.io/k8scsi/csi-provisioner@sha256:d657c839dce87324fe2b677302913f9386f885f8746be7bea0ced5b0844e3433 quay.io/k8scsi/csi-provisioner:v1.3.1] 50596998} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[quay.io/k8scsi/csi-attacher@sha256:26fccd7a99d973845df1193b46ebdcc6ab8dc5f6e6be319750c471fce1742d13 quay.io/k8scsi/csi-attacher:v1.2.0] 46226754} {[quay.io/k8scsi/csi-cluster-driver-registrar@sha256:fafd75ae5442f192cfa8c2e792903aee30d5884b62e802e4464b0a895d21e3ef quay.io/k8scsi/csi-cluster-driver-registrar:v1.0.1] 45874691} {[quay.io/k8scsi/csi-snapshotter@sha256:a49e0da1af6f2bf717e41ba1eee8b5e6a1cbd66a709dd92cc43fe475fe2589eb quay.io/k8scsi/csi-snapshotter:v1.1.0] 44325352} {[mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume@sha256:4fd30d43947d4a54fc89ead7985beecfd3c9b2a93a0655a373b1608ab90bd5af mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume:v0.0.7] 22909487} {[quay.io/k8scsi/livenessprobe@sha256:8c9c8a0f330acc2e558ae6624309447d02522e76aa35af23fa19b88abf4381cb quay.io/k8scsi/livenessprobe:v1.0.2] 17415381} {[quay.io/k8scsi/csi-node-driver-registrar@sha256:13daf82fb99e951a4bff8ae5fc7c17c3a8fe7130be6400990d8f6076c32d4599 quay.io/k8scsi/csi-node-driver-registrar:v1.1.0] 15815995} {[quay.io/k8scsi/livenessprobe@sha256:dde617756e0f602adc566ab71fd885f1dad451ad3fb063ac991c95a2ff47aea5 quay.io/k8scsi/livenessprobe:v1.1.0] 14967303} {[mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume@sha256:23d8c6033f02a1ecad05127ebdc931bb871264228661bc122704b0974e4d9fdd mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume:1.0.8] 1159025} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 20:55:00.823] Nov 5 20:55:00.819: INFO: I1105 20:55:00.823] Logging kubelet events for node k8s-agentpool-97435445-vmss000001 I1105 20:55:00.902] Nov 5 20:55:00.902: INFO: I1105 20:55:00.903] Logging pods the kubelet thinks is on node k8s-agentpool-97435445-vmss000001 I1105 20:55:00.987] Nov 5 20:55:00.987: INFO: tiller-deploy-54c96cb5df-pt7x6 started at 2019-11-05 20:08:00 +0000 UTC (0+1 container statuses recorded) I1105 20:55:00.988] Nov 5 20:55:00.987: INFO: Container tiller ready: true, restart count 0 I1105 20:55:00.988] Nov 5 20:55:00.987: INFO: csi-azuredisk-node-q2dmt started at 2019-11-05 20:16:09 +0000 UTC (0+3 container statuses recorded) I1105 20:55:00.988] Nov 5 20:55:00.987: INFO: Container azuredisk ready: true, restart count 0 I1105 20:55:00.988] Nov 5 20:55:00.987: INFO: Container liveness-probe ready: true, restart count 0 I1105 20:55:00.988] Nov 5 20:55:00.987: INFO: Container node-driver-registrar ready: true, restart count 0 I1105 20:55:00.988] Nov 5 20:55:00.987: INFO: csi-azuredisk-controller-6f6587bfc6-62nrt started at 2019-11-05 20:16:09 +0000 UTC (0+6 container statuses recorded) I1105 20:55:00.988] Nov 5 20:55:00.987: INFO: Container azuredisk ready: true, restart count 0 I1105 20:55:00.989] Nov 5 20:55:00.987: INFO: Container cluster-driver-registrar ready: true, restart count 0 I1105 20:55:00.989] Nov 5 20:55:00.987: INFO: Container csi-attacher ready: true, restart count 0 I1105 20:55:00.989] Nov 5 20:55:00.987: INFO: Container csi-provisioner ready: true, restart count 0 I1105 20:55:00.989] Nov 5 20:55:00.987: INFO: Container csi-snapshotter ready: true, restart count 0 I1105 20:55:00.989] Nov 5 20:55:00.987: INFO: Container liveness-probe ready: true, restart count 0 I1105 20:55:00.989] Nov 5 20:55:00.987: INFO: blobfuse-flexvol-installer-j4s7t started at 2019-11-05 20:07:59 +0000 UTC (0+1 container statuses recorded) I1105 20:55:00.989] Nov 5 20:55:00.987: INFO: Container blobfuse-flexvol-installer ready: true, restart count 0 I1105 20:55:00.989] Nov 5 20:55:00.987: INFO: azure-ip-masq-agent-gdhcq started at 2019-11-05 20:07:59 +0000 UTC (0+1 container statuses recorded) I1105 20:55:00.990] Nov 5 20:55:00.987: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 20:55:00.990] Nov 5 20:55:00.987: INFO: keyvault-flexvolume-6bffx started at 2019-11-05 20:07:59 +0000 UTC (0+1 container statuses recorded) I1105 20:55:00.990] Nov 5 20:55:00.987: INFO: Container keyvault-flexvolume ready: true, restart count 0 I1105 20:55:00.990] Nov 5 20:55:00.987: INFO: kube-proxy-kjrpb started at 2019-11-05 20:08:00 +0000 UTC (0+1 container statuses recorded) I1105 20:55:00.990] Nov 5 20:55:00.987: INFO: Container kube-proxy ready: true, restart count 0 I1105 20:55:01.247] Nov 5 20:55:01.246: INFO: I1105 20:55:01.247] Latency metrics for node k8s-agentpool-97435445-vmss000001 I1105 20:55:01.247] Nov 5 20:55:01.246: INFO: I1105 20:55:01.247] Logging node info for node k8s-master-97435445-0 I1105 20:55:01.328] Nov 5 20:55:01.325: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-97435445-0,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-97435445-0,UID:df3ae36f-961c-4729-a8f5-34cda54f4ff9,ResourceVersion:9287,Generation:0,CreationTimestamp:2019-11-05 20:08:36 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 2,kubernetes.azure.com/cluster: kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-97435445-0,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.4.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b/providers/Microsoft.Compute/virtualMachines/k8s-master-97435445-0,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 20:08:46 +0000 UTC 2019-11-05 20:08:46 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 20:54:03 +0000 UTC 2019-11-05 20:08:28 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 20:54:03 +0000 UTC 2019-11-05 20:08:28 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 20:54:03 +0000 UTC 2019-11-05 20:08:28 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 20:54:03 +0000 UTC 2019-11-05 20:08:36 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.255.5} {Hostname k8s-master-97435445-0}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:3c381865192341c2868672e8626cef7d,SystemUUID:F3D259D3-7EB8-B844-A1F0-F954DCD0D76B,BootID:4948329e-cf57-48c3-aa83-5297bef70370,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 20:55:01.328] Nov 5 20:55:01.325: INFO: I1105 20:55:01.328] Logging kubelet events for node k8s-master-97435445-0 I1105 20:55:01.409] Nov 5 20:55:01.409: INFO: I1105 20:55:01.409] Logging pods the kubelet thinks is on node k8s-master-97435445-0 I1105 20:55:01.493] Nov 5 20:55:01.493: INFO: kube-proxy-5wnd2 started at 2019-11-05 20:08:38 +0000 UTC (0+1 container statuses recorded) I1105 20:55:01.494] Nov 5 20:55:01.493: INFO: Container kube-proxy ready: true, restart count 0 I1105 20:55:01.494] Nov 5 20:55:01.493: INFO: cloud-controller-manager-k8s-master-97435445-0 started at 2019-11-05 20:08:22 +0000 UTC (0+1 container statuses recorded) I1105 20:55:01.494] Nov 5 20:55:01.493: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 20:55:01.494] Nov 5 20:55:01.493: INFO: kube-addon-manager-k8s-master-97435445-0 started at 2019-11-05 20:08:48 +0000 UTC (0+1 container statuses recorded) I1105 20:55:01.494] Nov 5 20:55:01.493: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 20:55:01.494] Nov 5 20:55:01.493: INFO: kube-apiserver-k8s-master-97435445-0 started at 2019-11-05 20:08:22 +0000 UTC (0+1 container statuses recorded) I1105 20:55:01.494] Nov 5 20:55:01.493: INFO: Container kube-apiserver ready: true, restart count 0 I1105 20:55:01.495] Nov 5 20:55:01.493: INFO: kube-controller-manager-k8s-master-97435445-0 started at 2019-11-05 20:08:22 +0000 UTC (0+1 container statuses recorded) I1105 20:55:01.495] Nov 5 20:55:01.493: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 20:55:01.495] Nov 5 20:55:01.493: INFO: kube-scheduler-k8s-master-97435445-0 started at 2019-11-05 20:08:22 +0000 UTC (0+1 container statuses recorded) I1105 20:55:01.495] Nov 5 20:55:01.493: INFO: Container kube-scheduler ready: true, restart count 0 I1105 20:55:01.495] Nov 5 20:55:01.493: INFO: azure-ip-masq-agent-q6dcl started at 2019-11-05 20:08:38 +0000 UTC (0+1 container statuses recorded) I1105 20:55:01.495] Nov 5 20:55:01.493: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 20:55:01.767] Nov 5 20:55:01.767: INFO: I1105 20:55:01.767] Latency metrics for node k8s-master-97435445-0 I1105 20:55:01.767] Nov 5 20:55:01.767: INFO: I1105 20:55:01.767] Logging node info for node k8s-master-97435445-1 I1105 20:55:01.848] Nov 5 20:55:01.845: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-97435445-1,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-97435445-1,UID:05534a67-d493-4ab5-9b0f-5ff5490e234f,ResourceVersion:9366,Generation:0,CreationTimestamp:2019-11-05 20:07:34 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 0,kubernetes.azure.com/cluster: kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-97435445-1,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.1.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b/providers/Microsoft.Compute/virtualMachines/k8s-master-97435445-1,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 20:08:26 +0000 UTC 2019-11-05 20:08:26 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 20:54:30 +0000 UTC 2019-11-05 20:07:28 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 20:54:30 +0000 UTC 2019-11-05 20:07:28 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 20:54:30 +0000 UTC 2019-11-05 20:07:28 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 20:54:30 +0000 UTC 2019-11-05 20:07:55 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{Hostname k8s-master-97435445-1} {InternalIP 10.240.255.6}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:079b605103b74acabac18fd2ea63368a,SystemUUID:38220EEC-BEFF-4C46-AA05-5A214AAFDC28,BootID:b1b45df0-d925-4bcc-9830-0202df36ebb4,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 20:55:01.849] Nov 5 20:55:01.846: INFO: I1105 20:55:01.849] Logging kubelet events for node k8s-master-97435445-1 I1105 20:55:01.929] Nov 5 20:55:01.929: INFO: I1105 20:55:01.929] Logging pods the kubelet thinks is on node k8s-master-97435445-1 I1105 20:55:02.013] Nov 5 20:55:02.013: INFO: cloud-controller-manager-k8s-master-97435445-1 started at 2019-11-05 20:07:22 +0000 UTC (0+1 container statuses recorded) I1105 20:55:02.014] Nov 5 20:55:02.013: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 20:55:02.014] Nov 5 20:55:02.013: INFO: kube-addon-manager-k8s-master-97435445-1 started at 2019-11-05 20:07:22 +0000 UTC (0+1 container statuses recorded) I1105 20:55:02.014] Nov 5 20:55:02.013: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 20:55:02.014] Nov 5 20:55:02.013: INFO: kube-apiserver-k8s-master-97435445-1 started at 2019-11-05 20:07:22 +0000 UTC (0+1 container statuses recorded) I1105 20:55:02.014] Nov 5 20:55:02.013: INFO: Container kube-apiserver ready: true, restart count 0 I1105 20:55:02.014] Nov 5 20:55:02.013: INFO: kube-controller-manager-k8s-master-97435445-1 started at 2019-11-05 20:07:22 +0000 UTC (0+1 container statuses recorded) I1105 20:55:02.014] Nov 5 20:55:02.013: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 20:55:02.015] Nov 5 20:55:02.013: INFO: kube-scheduler-k8s-master-97435445-1 started at 2019-11-05 20:07:22 +0000 UTC (0+1 container statuses recorded) I1105 20:55:02.015] Nov 5 20:55:02.013: INFO: Container kube-scheduler ready: true, restart count 0 I1105 20:55:02.015] Nov 5 20:55:02.013: INFO: azure-ip-masq-agent-n29bm started at 2019-11-05 20:07:59 +0000 UTC (0+1 container statuses recorded) I1105 20:55:02.015] Nov 5 20:55:02.013: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 20:55:02.015] Nov 5 20:55:02.013: INFO: kube-proxy-h4s2c started at 2019-11-05 20:08:00 +0000 UTC (0+1 container statuses recorded) I1105 20:55:02.015] Nov 5 20:55:02.013: INFO: Container kube-proxy ready: true, restart count 0 I1105 20:55:02.310] Nov 5 20:55:02.310: INFO: I1105 20:55:02.310] Latency metrics for node k8s-master-97435445-1 I1105 20:55:02.311] Nov 5 20:55:02.310: INFO: I1105 20:55:02.311] Logging node info for node k8s-master-97435445-2 I1105 20:55:02.391] Nov 5 20:55:02.389: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-97435445-2,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-97435445-2,UID:fa55c895-1a93-4743-885e-03fce84b5332,ResourceVersion:9414,Generation:0,CreationTimestamp:2019-11-05 20:09:19 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 0,kubernetes.azure.com/cluster: kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-97435445-2,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.5.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b/providers/Microsoft.Compute/virtualMachines/k8s-master-97435445-2,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 20:09:19 +0000 UTC 2019-11-05 20:09:19 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 20:54:47 +0000 UTC 2019-11-05 20:09:12 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 20:54:47 +0000 UTC 2019-11-05 20:09:12 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 20:54:47 +0000 UTC 2019-11-05 20:09:12 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 20:54:47 +0000 UTC 2019-11-05 20:09:19 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.255.7} {Hostname k8s-master-97435445-2}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:d19f6709138f496da43c095cec7b2360,SystemUUID:10116777-A8D7-CB4A-A8B7-7EBCF2F23B62,BootID:50cd14a9-5497-4972-bf60-cf59a0cbcb5f,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 20:55:02.391] Nov 5 20:55:02.389: INFO: I1105 20:55:02.392] Logging kubelet events for node k8s-master-97435445-2 I1105 20:55:02.473] Nov 5 20:55:02.473: INFO: I1105 20:55:02.473] Logging pods the kubelet thinks is on node k8s-master-97435445-2 I1105 20:55:02.564] Nov 5 20:55:02.564: INFO: cloud-controller-manager-k8s-master-97435445-2 started at 2019-11-05 20:09:05 +0000 UTC (0+1 container statuses recorded) I1105 20:55:02.565] Nov 5 20:55:02.564: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 20:55:02.565] Nov 5 20:55:02.564: INFO: kube-addon-manager-k8s-master-97435445-2 started at 2019-11-05 20:09:05 +0000 UTC (0+1 container statuses recorded) I1105 20:55:02.565] Nov 5 20:55:02.564: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 20:55:02.565] Nov 5 20:55:02.564: INFO: azure-ip-masq-agent-sq7p5 started at 2019-11-05 20:09:19 +0000 UTC (0+1 container statuses recorded) I1105 20:55:02.565] Nov 5 20:55:02.564: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 20:55:02.565] Nov 5 20:55:02.564: INFO: kube-proxy-pz426 started at 2019-11-05 20:09:19 +0000 UTC (0+1 container statuses recorded) I1105 20:55:02.566] Nov 5 20:55:02.564: INFO: Container kube-proxy ready: true, restart count 0 I1105 20:55:02.566] Nov 5 20:55:02.564: INFO: kube-apiserver-k8s-master-97435445-2 started at 2019-11-05 20:09:05 +0000 UTC (0+1 container statuses recorded) I1105 20:55:02.566] Nov 5 20:55:02.564: INFO: Container kube-apiserver ready: true, restart count 0 I1105 20:55:02.566] Nov 5 20:55:02.564: INFO: kube-controller-manager-k8s-master-97435445-2 started at 2019-11-05 20:09:05 +0000 UTC (0+1 container statuses recorded) I1105 20:55:02.566] Nov 5 20:55:02.564: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 20:55:02.566] Nov 5 20:55:02.564: INFO: kube-scheduler-k8s-master-97435445-2 started at 2019-11-05 20:09:05 +0000 UTC (0+1 container statuses recorded) I1105 20:55:02.566] Nov 5 20:55:02.564: INFO: Container kube-scheduler ready: true, restart count 0 I1105 20:55:02.818] Nov 5 20:55:02.818: INFO: I1105 20:55:02.818] Latency metrics for node k8s-master-97435445-2 I1105 20:55:02.819] Nov 5 20:55:02.818: INFO: I1105 20:55:02.819] Logging node info for node k8s-master-97435445-3 I1105 20:55:02.900] Nov 5 20:55:02.898: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-97435445-3,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-97435445-3,UID:8ffd1d0d-4487-4680-a2f6-e724387be637,ResourceVersion:9417,Generation:0,CreationTimestamp:2019-11-05 20:09:21 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 1,kubernetes.azure.com/cluster: kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-97435445-3,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.6.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b/providers/Microsoft.Compute/virtualMachines/k8s-master-97435445-3,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 20:09:39 +0000 UTC 2019-11-05 20:09:39 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 20:54:47 +0000 UTC 2019-11-05 20:09:18 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 20:54:47 +0000 UTC 2019-11-05 20:09:18 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 20:54:47 +0000 UTC 2019-11-05 20:09:18 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 20:54:47 +0000 UTC 2019-11-05 20:09:21 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.255.8} {Hostname k8s-master-97435445-3}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:266064173bfb4297898c6c25769e8b20,SystemUUID:E6E44241-5015-7740-9A81-A668A8427206,BootID:579d658c-b0ad-47f1-a993-1ccea2b55789,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 20:55:02.900] Nov 5 20:55:02.898: INFO: I1105 20:55:02.901] Logging kubelet events for node k8s-master-97435445-3 I1105 20:55:02.982] Nov 5 20:55:02.982: INFO: I1105 20:55:02.982] Logging pods the kubelet thinks is on node k8s-master-97435445-3 I1105 20:55:03.066] Nov 5 20:55:03.066: INFO: kube-scheduler-k8s-master-97435445-3 started at 2019-11-05 20:09:05 +0000 UTC (0+1 container statuses recorded) I1105 20:55:03.066] Nov 5 20:55:03.066: INFO: Container kube-scheduler ready: true, restart count 0 I1105 20:55:03.066] Nov 5 20:55:03.066: INFO: azure-ip-masq-agent-747cl started at 2019-11-05 20:09:21 +0000 UTC (0+1 container statuses recorded) I1105 20:55:03.066] Nov 5 20:55:03.066: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 20:55:03.067] Nov 5 20:55:03.066: INFO: kube-proxy-k45g8 started at 2019-11-05 20:09:21 +0000 UTC (0+1 container statuses recorded) I1105 20:55:03.067] Nov 5 20:55:03.066: INFO: Container kube-proxy ready: true, restart count 0 I1105 20:55:03.067] Nov 5 20:55:03.066: INFO: cloud-controller-manager-k8s-master-97435445-3 started at 2019-11-05 20:09:04 +0000 UTC (0+1 container statuses recorded) I1105 20:55:03.067] Nov 5 20:55:03.066: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 20:55:03.067] Nov 5 20:55:03.066: INFO: kube-addon-manager-k8s-master-97435445-3 started at 2019-11-05 20:09:05 +0000 UTC (0+1 container statuses recorded) I1105 20:55:03.067] Nov 5 20:55:03.066: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 20:55:03.067] Nov 5 20:55:03.066: INFO: kube-apiserver-k8s-master-97435445-3 started at 2019-11-05 20:09:05 +0000 UTC (0+1 container statuses recorded) I1105 20:55:03.068] Nov 5 20:55:03.066: INFO: Container kube-apiserver ready: true, restart count 0 I1105 20:55:03.068] Nov 5 20:55:03.066: INFO: kube-controller-manager-k8s-master-97435445-3 started at 2019-11-05 20:09:05 +0000 UTC (0+1 container statuses recorded) I1105 20:55:03.068] Nov 5 20:55:03.066: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 20:55:03.317] Nov 5 20:55:03.317: INFO: I1105 20:55:03.318] Latency metrics for node k8s-master-97435445-3 I1105 20:55:03.318] Nov 5 20:55:03.317: INFO: I1105 20:55:03.318] Logging node info for node k8s-master-97435445-4 I1105 20:55:03.398] Nov 5 20:55:03.396: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-97435445-4,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-97435445-4,UID:6d547b41-3f92-4cc4-a2d7-dda2071776a2,ResourceVersion:9459,Generation:0,CreationTimestamp:2019-11-05 20:08:36 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 1,kubernetes.azure.com/cluster: kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-97435445-4,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.3.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b/providers/Microsoft.Compute/virtualMachines/k8s-master-97435445-4,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 20:08:59 +0000 UTC 2019-11-05 20:08:59 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 20:55:02 +0000 UTC 2019-11-05 20:08:29 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 20:55:02 +0000 UTC 2019-11-05 20:08:29 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 20:55:02 +0000 UTC 2019-11-05 20:08:29 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 20:55:02 +0000 UTC 2019-11-05 20:08:36 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.255.9} {Hostname k8s-master-97435445-4}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:56b393c61eb34ba49ebda5279eeedb30,SystemUUID:FD239C23-618C-0049-8734-9AF3FDC9E3C1,BootID:f342ea44-e719-4513-807d-5def7ac606e2,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 20:55:03.399] Nov 5 20:55:03.396: INFO: I1105 20:55:03.399] Logging kubelet events for node k8s-master-97435445-4 I1105 20:55:03.479] Nov 5 20:55:03.479: INFO: I1105 20:55:03.479] Logging pods the kubelet thinks is on node k8s-master-97435445-4 I1105 20:55:03.563] Nov 5 20:55:03.563: INFO: kube-controller-manager-k8s-master-97435445-4 started at 2019-11-05 20:08:23 +0000 UTC (0+1 container statuses recorded) I1105 20:55:03.563] Nov 5 20:55:03.563: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 20:55:03.564] Nov 5 20:55:03.563: INFO: kube-scheduler-k8s-master-97435445-4 started at 2019-11-05 20:08:23 +0000 UTC (0+1 container statuses recorded) I1105 20:55:03.564] Nov 5 20:55:03.563: INFO: Container kube-scheduler ready: true, restart count 0 I1105 20:55:03.564] Nov 5 20:55:03.563: INFO: kube-proxy-vfbfr started at 2019-11-05 20:08:36 +0000 UTC (0+1 container statuses recorded) I1105 20:55:03.564] Nov 5 20:55:03.563: INFO: Container kube-proxy ready: true, restart count 0 I1105 20:55:03.564] Nov 5 20:55:03.563: INFO: azure-ip-masq-agent-kmkll started at 2019-11-05 20:08:36 +0000 UTC (0+1 container statuses recorded) I1105 20:55:03.564] Nov 5 20:55:03.563: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 20:55:03.565] Nov 5 20:55:03.563: INFO: cloud-controller-manager-k8s-master-97435445-4 started at 2019-11-05 20:08:46 +0000 UTC (0+1 container statuses recorded) I1105 20:55:03.565] Nov 5 20:55:03.563: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 20:55:03.565] Nov 5 20:55:03.563: INFO: kube-addon-manager-k8s-master-97435445-4 started at 2019-11-05 20:08:23 +0000 UTC (0+1 container statuses recorded) I1105 20:55:03.566] Nov 5 20:55:03.563: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 20:55:03.566] Nov 5 20:55:03.563: INFO: kube-apiserver-k8s-master-97435445-4 started at 2019-11-05 20:08:23 +0000 UTC (0+1 container statuses recorded) I1105 20:55:03.566] Nov 5 20:55:03.563: INFO: Container kube-apiserver ready: true, restart count 0 I1105 20:55:03.833] Nov 5 20:55:03.833: INFO: I1105 20:55:03.833] Latency metrics for node k8s-master-97435445-4 I1105 20:55:03.833] Nov 5 20:55:03.833: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready I1105 20:55:03.913] STEP: Destroying namespace "azuredisk-8958" for this suite. I1105 20:55:18.155] Nov 5 20:55:18.155: INFO: Waiting up to 30s for server preferred namespaced resources to be successfully discovered I1105 20:55:21.194] Nov 5 20:55:21.193: INFO: namespace azuredisk-8958 deletion completed in 17.280808884s I1105 20:55:21.194] I1105 20:55:21.194] • Failure [323.295 seconds] I1105 20:55:21.194] [azuredisk-csi-e2e] Dynamic Provisioning I1105 20:55:21.194] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:33 I1105 20:55:21.194] [single-az] I1105 20:55:21.195] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:36 I1105 20:55:21.195] should create a deployment object, write and read to it, delete the pod and write and read to it again [It] I1105 20:55:21.195] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:193 I1105 20:55:21.195] I1105 20:55:21.195] Unexpected error: I1105 20:55:21.196] <*errors.errorString | 0xc0007b1120>: { I1105 20:55:21.196] s: "PersistentVolumeClaims [pvc-6nqs8] not all in phase Bound within 5m0s", I1105 20:55:21.196] } I1105 20:55:21.196] PersistentVolumeClaims [pvc-6nqs8] not all in phase Bound within 5m0s I1105 20:55:21.196] occurred I1105 20:55:21.196] I1105 20:55:21.196] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/testsuites/testsuites.go:278 I1105 20:55:21.201] ------------------------------ I1105 20:55:21.201] [azuredisk-csi-e2e] Dynamic Provisioning [multi-az] I1105 20:55:21.202] should create a raw block volume on demand I1105 20:55:21.202] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:101 I1105 20:55:21.202] [BeforeEach] [multi-az] I1105 20:55:21.202] /go/src/sigs.k8s.io/azuredisk-csi-driver/vendor/k8s.io/kubernetes/test/e2e/framework/framework.go:150 I1105 20:55:21.202] STEP: Creating a kubernetes client I1105 20:55:21.202] Nov 5 20:55:21.193: INFO: >>> kubeConfig: /workspace/aks971561099/kubeconfig/kubeconfig.eastus2.json I1105 20:55:21.203] STEP: Building a namespace api object, basename azuredisk I1105 20:55:21.439] STEP: Binding the e2e-test-privileged-psp PodSecurityPolicy to the default service account in azuredisk-9464 I1105 20:55:21.707] STEP: Waiting for a default service account to be provisioned in namespace I1105 20:55:21.784] [BeforeEach] [multi-az] I1105 20:55:21.784] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:58 I1105 20:55:21.784] [It] should create a raw block volume on demand I1105 20:55:21.784] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:101 I1105 20:55:21.785] STEP: setting up the StorageClass I1105 20:55:21.785] STEP: creating a StorageClass I1105 20:55:21.878] STEP: setting up the PVC and PV I1105 20:55:21.878] STEP: creating a PVC I1105 20:55:21.967] STEP: waiting for PVC to be in phase "Bound" I1105 20:55:21.967] Nov 5 20:55:21.967: INFO: Waiting up to 5m0s for PersistentVolumeClaims [pvc-hqwk4] to have phase Bound I1105 20:55:22.045] Nov 5 20:55:22.045: INFO: PersistentVolumeClaim pvc-hqwk4 found but phase is Pending instead of Bound. I1105 20:55:24.125] Nov 5 20:55:24.125: INFO: PersistentVolumeClaim pvc-hqwk4 found but phase is Pending instead of Bound. I1105 20:55:26.212] Nov 5 20:55:26.212: INFO: PersistentVolumeClaim pvc-hqwk4 found but phase is Pending instead of Bound. I1105 20:55:28.291] Nov 5 20:55:28.291: INFO: PersistentVolumeClaim pvc-hqwk4 found but phase is Pending instead of Bound. I1105 20:55:30.370] Nov 5 20:55:30.370: INFO: PersistentVolumeClaim pvc-hqwk4 found but phase is Pending instead of Bound. I1105 20:55:32.450] Nov 5 20:55:32.450: INFO: PersistentVolumeClaim pvc-hqwk4 found but phase is Pending instead of Bound. I1105 20:55:34.529] Nov 5 20:55:34.529: INFO: PersistentVolumeClaim pvc-hqwk4 found but phase is Pending instead of Bound. I1105 20:55:36.608] Nov 5 20:55:36.608: INFO: PersistentVolumeClaim pvc-hqwk4 found but phase is Pending instead of Bound. I1105 20:55:38.687] Nov 5 20:55:38.687: INFO: PersistentVolumeClaim pvc-hqwk4 found but phase is Pending instead of Bound. I1105 20:55:40.766] Nov 5 20:55:40.766: INFO: PersistentVolumeClaim pvc-hqwk4 found but phase is Pending instead of Bound. I1105 20:55:42.845] Nov 5 20:55:42.845: INFO: PersistentVolumeClaim pvc-hqwk4 found but phase is Pending instead of Bound. I1105 20:55:44.924] Nov 5 20:55:44.923: INFO: PersistentVolumeClaim pvc-hqwk4 found but phase is Pending instead of Bound. I1105 20:55:47.003] Nov 5 20:55:47.002: INFO: PersistentVolumeClaim pvc-hqwk4 found but phase is Pending instead of Bound. I1105 20:55:49.081] Nov 5 20:55:49.081: INFO: PersistentVolumeClaim pvc-hqwk4 found but phase is Pending instead of Bound. I1105 20:55:51.161] Nov 5 20:55:51.161: INFO: PersistentVolumeClaim pvc-hqwk4 found but phase is Pending instead of Bound. I1105 20:55:53.241] Nov 5 20:55:53.241: INFO: PersistentVolumeClaim pvc-hqwk4 found but phase is Pending instead of Bound. I1105 20:55:55.320] Nov 5 20:55:55.320: INFO: PersistentVolumeClaim pvc-hqwk4 found but phase is Pending instead of Bound. I1105 20:55:57.399] Nov 5 20:55:57.398: INFO: PersistentVolumeClaim pvc-hqwk4 found but phase is Pending instead of Bound. I1105 20:55:59.478] Nov 5 20:55:59.478: INFO: PersistentVolumeClaim pvc-hqwk4 found but phase is Pending instead of Bound. I1105 20:56:01.558] Nov 5 20:56:01.557: INFO: PersistentVolumeClaim pvc-hqwk4 found but phase is Pending instead of Bound. I1105 20:56:03.637] Nov 5 20:56:03.637: INFO: PersistentVolumeClaim pvc-hqwk4 found but phase is Pending instead of Bound. I1105 20:56:05.720] Nov 5 20:56:05.720: INFO: PersistentVolumeClaim pvc-hqwk4 found but phase is Pending instead of Bound. I1105 20:56:07.800] Nov 5 20:56:07.799: INFO: PersistentVolumeClaim pvc-hqwk4 found but phase is Pending instead of Bound. I1105 20:56:09.879] Nov 5 20:56:09.879: INFO: PersistentVolumeClaim pvc-hqwk4 found but phase is Pending instead of Bound. I1105 20:56:11.959] Nov 5 20:56:11.959: INFO: PersistentVolumeClaim pvc-hqwk4 found but phase is Pending instead of Bound. I1105 20:56:14.038] Nov 5 20:56:14.038: INFO: PersistentVolumeClaim pvc-hqwk4 found but phase is Pending instead of Bound. I1105 20:56:16.117] Nov 5 20:56:16.117: INFO: PersistentVolumeClaim pvc-hqwk4 found but phase is Pending instead of Bound. I1105 20:56:18.214] Nov 5 20:56:18.214: INFO: PersistentVolumeClaim pvc-hqwk4 found but phase is Pending instead of Bound. I1105 20:56:20.293] Nov 5 20:56:20.293: INFO: PersistentVolumeClaim pvc-hqwk4 found but phase is Pending instead of Bound. I1105 20:56:22.372] Nov 5 20:56:22.372: INFO: PersistentVolumeClaim pvc-hqwk4 found but phase is Pending instead of Bound. I1105 20:56:24.451] Nov 5 20:56:24.451: INFO: PersistentVolumeClaim pvc-hqwk4 found but phase is Pending instead of Bound. I1105 20:56:26.531] Nov 5 20:56:26.531: INFO: PersistentVolumeClaim pvc-hqwk4 found but phase is Pending instead of Bound. I1105 20:56:28.610] Nov 5 20:56:28.610: INFO: PersistentVolumeClaim pvc-hqwk4 found but phase is Pending instead of Bound. I1105 20:56:30.689] Nov 5 20:56:30.689: INFO: PersistentVolumeClaim pvc-hqwk4 found but phase is Pending instead of Bound. I1105 20:56:32.768] Nov 5 20:56:32.767: INFO: PersistentVolumeClaim pvc-hqwk4 found but phase is Pending instead of Bound. I1105 20:56:34.847] Nov 5 20:56:34.847: INFO: PersistentVolumeClaim pvc-hqwk4 found but phase is Pending instead of Bound. I1105 20:56:36.927] Nov 5 20:56:36.927: INFO: PersistentVolumeClaim pvc-hqwk4 found but phase is Pending instead of Bound. I1105 20:56:39.006] Nov 5 20:56:39.006: INFO: PersistentVolumeClaim pvc-hqwk4 found but phase is Pending instead of Bound. I1105 20:56:41.086] Nov 5 20:56:41.085: INFO: PersistentVolumeClaim pvc-hqwk4 found but phase is Pending instead of Bound. I1105 20:56:43.165] Nov 5 20:56:43.164: INFO: PersistentVolumeClaim pvc-hqwk4 found but phase is Pending instead of Bound. I1105 20:56:45.244] Nov 5 20:56:45.244: INFO: PersistentVolumeClaim pvc-hqwk4 found but phase is Pending instead of Bound. I1105 20:56:47.323] Nov 5 20:56:47.322: INFO: PersistentVolumeClaim pvc-hqwk4 found but phase is Pending instead of Bound. I1105 20:56:49.402] Nov 5 20:56:49.402: INFO: PersistentVolumeClaim pvc-hqwk4 found but phase is Pending instead of Bound. I1105 20:56:51.482] Nov 5 20:56:51.482: INFO: PersistentVolumeClaim pvc-hqwk4 found but phase is Pending instead of Bound. I1105 20:56:53.561] Nov 5 20:56:53.561: INFO: PersistentVolumeClaim pvc-hqwk4 found but phase is Pending instead of Bound. I1105 20:56:55.641] Nov 5 20:56:55.640: INFO: PersistentVolumeClaim pvc-hqwk4 found but phase is Pending instead of Bound. I1105 20:56:57.719] Nov 5 20:56:57.719: INFO: PersistentVolumeClaim pvc-hqwk4 found but phase is Pending instead of Bound. I1105 20:56:59.799] Nov 5 20:56:59.798: INFO: PersistentVolumeClaim pvc-hqwk4 found but phase is Pending instead of Bound. I1105 20:57:01.878] Nov 5 20:57:01.878: INFO: PersistentVolumeClaim pvc-hqwk4 found but phase is Pending instead of Bound. I1105 20:57:03.957] Nov 5 20:57:03.957: INFO: PersistentVolumeClaim pvc-hqwk4 found but phase is Pending instead of Bound. I1105 20:57:06.037] Nov 5 20:57:06.036: INFO: PersistentVolumeClaim pvc-hqwk4 found but phase is Pending instead of Bound. I1105 20:57:08.115] Nov 5 20:57:08.115: INFO: PersistentVolumeClaim pvc-hqwk4 found but phase is Pending instead of Bound. I1105 20:57:10.230] Nov 5 20:57:10.230: INFO: PersistentVolumeClaim pvc-hqwk4 found but phase is Pending instead of Bound. I1105 20:57:12.310] Nov 5 20:57:12.310: INFO: PersistentVolumeClaim pvc-hqwk4 found but phase is Pending instead of Bound. I1105 20:57:14.389] Nov 5 20:57:14.389: INFO: PersistentVolumeClaim pvc-hqwk4 found but phase is Pending instead of Bound. I1105 20:57:16.468] Nov 5 20:57:16.468: INFO: PersistentVolumeClaim pvc-hqwk4 found but phase is Pending instead of Bound. I1105 20:57:18.547] Nov 5 20:57:18.547: INFO: PersistentVolumeClaim pvc-hqwk4 found but phase is Pending instead of Bound. I1105 20:57:20.627] Nov 5 20:57:20.627: INFO: PersistentVolumeClaim pvc-hqwk4 found but phase is Pending instead of Bound. I1105 20:57:22.706] Nov 5 20:57:22.706: INFO: PersistentVolumeClaim pvc-hqwk4 found but phase is Pending instead of Bound. I1105 20:57:24.785] Nov 5 20:57:24.785: INFO: PersistentVolumeClaim pvc-hqwk4 found but phase is Pending instead of Bound. I1105 20:57:26.865] Nov 5 20:57:26.864: INFO: PersistentVolumeClaim pvc-hqwk4 found but phase is Pending instead of Bound. I1105 20:57:28.944] Nov 5 20:57:28.944: INFO: PersistentVolumeClaim pvc-hqwk4 found but phase is Pending instead of Bound. I1105 20:57:31.023] Nov 5 20:57:31.023: INFO: PersistentVolumeClaim pvc-hqwk4 found but phase is Pending instead of Bound. I1105 20:57:33.102] Nov 5 20:57:33.102: INFO: PersistentVolumeClaim pvc-hqwk4 found but phase is Pending instead of Bound. I1105 20:57:35.183] Nov 5 20:57:35.183: INFO: PersistentVolumeClaim pvc-hqwk4 found but phase is Pending instead of Bound. I1105 20:57:37.262] Nov 5 20:57:37.262: INFO: PersistentVolumeClaim pvc-hqwk4 found but phase is Pending instead of Bound. I1105 20:57:39.342] Nov 5 20:57:39.342: INFO: PersistentVolumeClaim pvc-hqwk4 found but phase is Pending instead of Bound. I1105 20:57:41.532] Nov 5 20:57:41.532: INFO: PersistentVolumeClaim pvc-hqwk4 found but phase is Pending instead of Bound. I1105 20:57:43.611] Nov 5 20:57:43.611: INFO: PersistentVolumeClaim pvc-hqwk4 found but phase is Pending instead of Bound. I1105 20:57:45.690] Nov 5 20:57:45.690: INFO: PersistentVolumeClaim pvc-hqwk4 found but phase is Pending instead of Bound. I1105 20:57:47.769] Nov 5 20:57:47.769: INFO: PersistentVolumeClaim pvc-hqwk4 found but phase is Pending instead of Bound. I1105 20:57:49.849] Nov 5 20:57:49.849: INFO: PersistentVolumeClaim pvc-hqwk4 found but phase is Pending instead of Bound. I1105 20:57:52.549] Nov 5 20:57:52.549: INFO: PersistentVolumeClaim pvc-hqwk4 found but phase is Pending instead of Bound. I1105 20:57:54.628] Nov 5 20:57:54.628: INFO: PersistentVolumeClaim pvc-hqwk4 found but phase is Pending instead of Bound. I1105 20:57:56.720] Nov 5 20:57:56.720: INFO: PersistentVolumeClaim pvc-hqwk4 found but phase is Pending instead of Bound. I1105 20:57:58.800] Nov 5 20:57:58.800: INFO: PersistentVolumeClaim pvc-hqwk4 found but phase is Pending instead of Bound. I1105 20:58:00.881] Nov 5 20:58:00.880: INFO: PersistentVolumeClaim pvc-hqwk4 found but phase is Pending instead of Bound. I1105 20:58:02.960] Nov 5 20:58:02.959: INFO: PersistentVolumeClaim pvc-hqwk4 found but phase is Pending instead of Bound. I1105 20:58:05.039] Nov 5 20:58:05.039: INFO: PersistentVolumeClaim pvc-hqwk4 found but phase is Pending instead of Bound. I1105 20:58:07.118] Nov 5 20:58:07.118: INFO: PersistentVolumeClaim pvc-hqwk4 found but phase is Pending instead of Bound. I1105 20:58:09.197] Nov 5 20:58:09.197: INFO: PersistentVolumeClaim pvc-hqwk4 found but phase is Pending instead of Bound. I1105 20:58:11.277] Nov 5 20:58:11.277: INFO: PersistentVolumeClaim pvc-hqwk4 found but phase is Pending instead of Bound. I1105 20:58:13.356] Nov 5 20:58:13.356: INFO: PersistentVolumeClaim pvc-hqwk4 found but phase is Pending instead of Bound. I1105 20:58:15.439] Nov 5 20:58:15.439: INFO: PersistentVolumeClaim pvc-hqwk4 found but phase is Pending instead of Bound. I1105 20:58:17.519] Nov 5 20:58:17.519: INFO: PersistentVolumeClaim pvc-hqwk4 found but phase is Pending instead of Bound. I1105 20:58:19.597] Nov 5 20:58:19.597: INFO: PersistentVolumeClaim pvc-hqwk4 found but phase is Pending instead of Bound. I1105 20:58:21.677] Nov 5 20:58:21.677: INFO: PersistentVolumeClaim pvc-hqwk4 found but phase is Pending instead of Bound. I1105 20:58:23.757] Nov 5 20:58:23.756: INFO: PersistentVolumeClaim pvc-hqwk4 found but phase is Pending instead of Bound. I1105 20:58:25.836] Nov 5 20:58:25.835: INFO: PersistentVolumeClaim pvc-hqwk4 found but phase is Pending instead of Bound. I1105 20:58:27.915] Nov 5 20:58:27.914: INFO: PersistentVolumeClaim pvc-hqwk4 found but phase is Pending instead of Bound. I1105 20:58:30.006] Nov 5 20:58:30.005: INFO: PersistentVolumeClaim pvc-hqwk4 found but phase is Pending instead of Bound. I1105 20:58:32.084] Nov 5 20:58:32.084: INFO: PersistentVolumeClaim pvc-hqwk4 found but phase is Pending instead of Bound. I1105 20:58:34.164] Nov 5 20:58:34.164: INFO: PersistentVolumeClaim pvc-hqwk4 found but phase is Pending instead of Bound. I1105 20:58:36.244] Nov 5 20:58:36.244: INFO: PersistentVolumeClaim pvc-hqwk4 found but phase is Pending instead of Bound. I1105 20:58:38.323] Nov 5 20:58:38.323: INFO: PersistentVolumeClaim pvc-hqwk4 found but phase is Pending instead of Bound. I1105 20:58:40.402] Nov 5 20:58:40.402: INFO: PersistentVolumeClaim pvc-hqwk4 found but phase is Pending instead of Bound. I1105 20:58:42.481] Nov 5 20:58:42.481: INFO: PersistentVolumeClaim pvc-hqwk4 found but phase is Pending instead of Bound. I1105 20:58:44.560] Nov 5 20:58:44.560: INFO: PersistentVolumeClaim pvc-hqwk4 found but phase is Pending instead of Bound. I1105 20:58:46.639] Nov 5 20:58:46.639: INFO: PersistentVolumeClaim pvc-hqwk4 found but phase is Pending instead of Bound. I1105 20:58:48.722] Nov 5 20:58:48.721: INFO: PersistentVolumeClaim pvc-hqwk4 found but phase is Pending instead of Bound. I1105 20:58:50.801] Nov 5 20:58:50.801: INFO: PersistentVolumeClaim pvc-hqwk4 found but phase is Pending instead of Bound. I1105 20:58:52.880] Nov 5 20:58:52.880: INFO: PersistentVolumeClaim pvc-hqwk4 found but phase is Pending instead of Bound. I1105 20:58:54.960] Nov 5 20:58:54.960: INFO: PersistentVolumeClaim pvc-hqwk4 found but phase is Pending instead of Bound. I1105 20:58:57.039] Nov 5 20:58:57.039: INFO: PersistentVolumeClaim pvc-hqwk4 found but phase is Pending instead of Bound. I1105 20:58:59.119] Nov 5 20:58:59.119: INFO: PersistentVolumeClaim pvc-hqwk4 found but phase is Pending instead of Bound. I1105 20:59:01.199] Nov 5 20:59:01.198: INFO: PersistentVolumeClaim pvc-hqwk4 found but phase is Pending instead of Bound. I1105 20:59:03.278] Nov 5 20:59:03.278: INFO: PersistentVolumeClaim pvc-hqwk4 found but phase is Pending instead of Bound. I1105 20:59:05.357] Nov 5 20:59:05.357: INFO: PersistentVolumeClaim pvc-hqwk4 found but phase is Pending instead of Bound. I1105 20:59:07.437] Nov 5 20:59:07.436: INFO: PersistentVolumeClaim pvc-hqwk4 found but phase is Pending instead of Bound. I1105 20:59:09.516] Nov 5 20:59:09.515: INFO: PersistentVolumeClaim pvc-hqwk4 found but phase is Pending instead of Bound. I1105 20:59:11.594] Nov 5 20:59:11.594: INFO: PersistentVolumeClaim pvc-hqwk4 found but phase is Pending instead of Bound. I1105 20:59:13.673] Nov 5 20:59:13.673: INFO: PersistentVolumeClaim pvc-hqwk4 found but phase is Pending instead of Bound. I1105 20:59:15.753] Nov 5 20:59:15.753: INFO: PersistentVolumeClaim pvc-hqwk4 found but phase is Pending instead of Bound. I1105 20:59:17.832] Nov 5 20:59:17.832: INFO: PersistentVolumeClaim pvc-hqwk4 found but phase is Pending instead of Bound. I1105 20:59:19.912] Nov 5 20:59:19.912: INFO: PersistentVolumeClaim pvc-hqwk4 found but phase is Pending instead of Bound. I1105 20:59:21.991] Nov 5 20:59:21.991: INFO: PersistentVolumeClaim pvc-hqwk4 found but phase is Pending instead of Bound. I1105 20:59:24.070] Nov 5 20:59:24.069: INFO: PersistentVolumeClaim pvc-hqwk4 found but phase is Pending instead of Bound. I1105 20:59:26.149] Nov 5 20:59:26.149: INFO: PersistentVolumeClaim pvc-hqwk4 found but phase is Pending instead of Bound. I1105 20:59:28.228] Nov 5 20:59:28.228: INFO: PersistentVolumeClaim pvc-hqwk4 found but phase is Pending instead of Bound. I1105 20:59:30.307] Nov 5 20:59:30.307: INFO: PersistentVolumeClaim pvc-hqwk4 found but phase is Pending instead of Bound. I1105 20:59:32.386] Nov 5 20:59:32.386: INFO: PersistentVolumeClaim pvc-hqwk4 found but phase is Pending instead of Bound. I1105 20:59:34.465] Nov 5 20:59:34.465: INFO: PersistentVolumeClaim pvc-hqwk4 found but phase is Pending instead of Bound. I1105 20:59:36.543] Nov 5 20:59:36.543: INFO: PersistentVolumeClaim pvc-hqwk4 found but phase is Pending instead of Bound. I1105 20:59:38.655] Nov 5 20:59:38.655: INFO: PersistentVolumeClaim pvc-hqwk4 found but phase is Pending instead of Bound. I1105 20:59:40.736] Nov 5 20:59:40.736: INFO: PersistentVolumeClaim pvc-hqwk4 found but phase is Pending instead of Bound. I1105 20:59:42.815] Nov 5 20:59:42.815: INFO: PersistentVolumeClaim pvc-hqwk4 found but phase is Pending instead of Bound. I1105 20:59:44.895] Nov 5 20:59:44.894: INFO: PersistentVolumeClaim pvc-hqwk4 found but phase is Pending instead of Bound. I1105 20:59:46.975] Nov 5 20:59:46.974: INFO: PersistentVolumeClaim pvc-hqwk4 found but phase is Pending instead of Bound. I1105 20:59:49.054] Nov 5 20:59:49.053: INFO: PersistentVolumeClaim pvc-hqwk4 found but phase is Pending instead of Bound. I1105 20:59:51.133] Nov 5 20:59:51.133: INFO: PersistentVolumeClaim pvc-hqwk4 found but phase is Pending instead of Bound. I1105 20:59:53.212] Nov 5 20:59:53.212: INFO: PersistentVolumeClaim pvc-hqwk4 found but phase is Pending instead of Bound. I1105 20:59:55.291] Nov 5 20:59:55.291: INFO: PersistentVolumeClaim pvc-hqwk4 found but phase is Pending instead of Bound. I1105 20:59:57.370] Nov 5 20:59:57.370: INFO: PersistentVolumeClaim pvc-hqwk4 found but phase is Pending instead of Bound. I1105 20:59:59.449] Nov 5 20:59:59.449: INFO: PersistentVolumeClaim pvc-hqwk4 found but phase is Pending instead of Bound. I1105 21:00:01.528] Nov 5 21:00:01.528: INFO: PersistentVolumeClaim pvc-hqwk4 found but phase is Pending instead of Bound. I1105 21:00:03.607] Nov 5 21:00:03.607: INFO: PersistentVolumeClaim pvc-hqwk4 found but phase is Pending instead of Bound. I1105 21:00:05.686] Nov 5 21:00:05.686: INFO: PersistentVolumeClaim pvc-hqwk4 found but phase is Pending instead of Bound. I1105 21:00:07.766] Nov 5 21:00:07.765: INFO: PersistentVolumeClaim pvc-hqwk4 found but phase is Pending instead of Bound. I1105 21:00:09.845] Nov 5 21:00:09.845: INFO: PersistentVolumeClaim pvc-hqwk4 found but phase is Pending instead of Bound. I1105 21:00:11.924] Nov 5 21:00:11.924: INFO: PersistentVolumeClaim pvc-hqwk4 found but phase is Pending instead of Bound. I1105 21:00:14.003] Nov 5 21:00:14.003: INFO: PersistentVolumeClaim pvc-hqwk4 found but phase is Pending instead of Bound. I1105 21:00:16.082] Nov 5 21:00:16.081: INFO: PersistentVolumeClaim pvc-hqwk4 found but phase is Pending instead of Bound. I1105 21:00:18.160] Nov 5 21:00:18.160: INFO: PersistentVolumeClaim pvc-hqwk4 found but phase is Pending instead of Bound. I1105 21:00:20.240] Nov 5 21:00:20.240: INFO: PersistentVolumeClaim pvc-hqwk4 found but phase is Pending instead of Bound. I1105 21:00:22.240] Nov 5 21:00:22.240: INFO: Unexpected error occurred: PersistentVolumeClaims [pvc-hqwk4] not all in phase Bound within 5m0s I1105 21:00:22.241] [AfterEach] [multi-az] I1105 21:00:22.241] /go/src/sigs.k8s.io/azuredisk-csi-driver/vendor/k8s.io/kubernetes/test/e2e/framework/framework.go:151 I1105 21:00:22.241] STEP: Collecting events from namespace "azuredisk-9464". I1105 21:00:22.321] STEP: Found 3 events. I1105 21:00:22.322] Nov 5 21:00:22.321: INFO: At 2019-11-05 20:55:21 +0000 UTC - event for pvc-hqwk4: {disk.csi.azure.com_csi-azuredisk-controller-6f6587bfc6-62nrt_3930bc69-0009-11ea-8cf5-829a27ff7d5a } Provisioning: External provisioner is provisioning volume for claim "azuredisk-9464/pvc-hqwk4" I1105 21:00:22.322] Nov 5 21:00:22.321: INFO: At 2019-11-05 20:55:21 +0000 UTC - event for pvc-hqwk4: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "disk.csi.azure.com" or manually created by system administrator I1105 21:00:22.322] Nov 5 21:00:22.321: INFO: At 2019-11-05 20:55:22 +0000 UTC - event for pvc-hqwk4: {disk.csi.azure.com_csi-azuredisk-controller-6f6587bfc6-62nrt_3930bc69-0009-11ea-8cf5-829a27ff7d5a } ProvisioningFailed: failed to provision volume with StorageClass "azuredisk-9464-disk.csi.azure.com-dynamic-sc-5ks6s": rpc error: code = Unknown desc = compute.DisksClient#CreateOrUpdate: Failure sending request: StatusCode=400 -- Original Error: Code="InvalidAvailabilityZone" Message="The zone(s) '' for resource 'Microsoft.Compute/disks/pvc-def232e0-f8dd-4d3c-8404-450062786564' is not supported. The supported zones for location 'eastus2' are '1,2,3'" I1105 21:00:22.399] Nov 5 21:00:22.399: INFO: POD NODE PHASE GRACE CONDITIONS I1105 21:00:22.400] Nov 5 21:00:22.399: INFO: I1105 21:00:22.641] Nov 5 21:00:22.641: INFO: I1105 21:00:22.642] Logging node info for node k8s-agentpool-97435445-vmss000000 I1105 21:00:22.723] Nov 5 21:00:22.720: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-agentpool-97435445-vmss000000,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-agentpool-97435445-vmss000000,UID:e955dc47-8c8f-4142-9241-3d409f32e42d,ResourceVersion:10303,Generation:0,CreationTimestamp:2019-11-05 20:07:34 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{agentpool: agentpool,beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_D4s_v3,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 0,kubernetes.azure.com/cluster: kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b,kubernetes.azure.com/role: agent,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-agentpool-97435445-vmss000000,kubernetes.io/os: linux,kubernetes.io/role: agent,node-role.kubernetes.io/agent: ,storageprofile: managed,storagetier: Premium_LRS,topology.disk.csi.azure.com/zone: 0,},Annotations:map[string]string{csi.volume.kubernetes.io/nodeid: {"disk.csi.azure.com":"k8s-agentpool-97435445-vmss000000"},node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.2.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool-97435445-vmss/virtualMachines/0,Unschedulable:false,Taints:[],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-csi-disk.csi.azure.com: {{8 0} {} 8 DecimalSI},cpu: {{4 0} {} 4 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{16797040640 0} {} BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-csi-disk.csi.azure.com: {{8 0} {} 8 DecimalSI},cpu: {{4 0} {} 4 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{16010608640 0} {} BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 20:08:01 +0000 UTC 2019-11-05 20:08:01 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 20:59:43 +0000 UTC 2019-11-05 20:07:27 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 20:59:43 +0000 UTC 2019-11-05 20:07:27 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 20:59:43 +0000 UTC 2019-11-05 20:07:27 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 20:59:43 +0000 UTC 2019-11-05 20:07:55 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.0.4} {Hostname k8s-agentpool-97435445-vmss000000}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:bfb55325809340228149d2d93f50df58,SystemUUID:CB2ED036-0439-F641-BC7F-640C3638C027,BootID:a917f4d1-549d-471d-b7dc-47ae1715d97d,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[k8sprowinternal.azurecr.io/azuredisk-csi@sha256:75d4ff5d7cd845d6e810c67796ecb037e852fb849ec1eb5b3194564008e6d1e0 k8sprowinternal.azurecr.io/azuredisk-csi:e2e-3e3ed045c6faa509dbd8d2890ee9f56aa2934887] 286930414} {[k8s.gcr.io/kubernetes-dashboard-amd64@sha256:0ae6b69432e78069c5ce2bcde0fe409c5c4d6f0f4d9cd50a17974fea38898747 k8s.gcr.io/kubernetes-dashboard-amd64:v1.10.1] 121711221} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/metrics-server-amd64@sha256:49a9f12f7067d11f42c803dbe61ed2c1299959ad85cb315b25ff7eef8e6b8892 k8s.gcr.io/metrics-server-amd64:v0.2.1] 42541759} {[k8s.gcr.io/coredns@sha256:e83beb5e43f8513fa735e77ffc5859640baea30a882a11cc75c4c3244a737d3c k8s.gcr.io/coredns:1.5.0] 42488424} {[mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume@sha256:4fd30d43947d4a54fc89ead7985beecfd3c9b2a93a0655a373b1608ab90bd5af mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume:v0.0.7] 22909487} {[quay.io/k8scsi/livenessprobe@sha256:8c9c8a0f330acc2e558ae6624309447d02522e76aa35af23fa19b88abf4381cb quay.io/k8scsi/livenessprobe:v1.0.2] 17415381} {[quay.io/k8scsi/csi-node-driver-registrar@sha256:13daf82fb99e951a4bff8ae5fc7c17c3a8fe7130be6400990d8f6076c32d4599 quay.io/k8scsi/csi-node-driver-registrar:v1.1.0] 15815995} {[mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume@sha256:23d8c6033f02a1ecad05127ebdc931bb871264228661bc122704b0974e4d9fdd mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume:1.0.8] 1159025} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 21:00:22.723] Nov 5 21:00:22.720: INFO: I1105 21:00:22.724] Logging kubelet events for node k8s-agentpool-97435445-vmss000000 I1105 21:00:22.803] Nov 5 21:00:22.803: INFO: I1105 21:00:22.803] Logging pods the kubelet thinks is on node k8s-agentpool-97435445-vmss000000 I1105 21:00:22.888] Nov 5 21:00:22.888: INFO: kube-proxy-8mt7f started at 2019-11-05 20:08:01 +0000 UTC (0+1 container statuses recorded) I1105 21:00:22.888] Nov 5 21:00:22.888: INFO: Container kube-proxy ready: true, restart count 0 I1105 21:00:22.889] Nov 5 21:00:22.888: INFO: kubernetes-dashboard-66dd8b8df7-t6j4l started at 2019-11-05 20:08:01 +0000 UTC (0+1 container statuses recorded) I1105 21:00:22.889] Nov 5 21:00:22.888: INFO: Container kubernetes-dashboard ready: true, restart count 0 I1105 21:00:22.889] Nov 5 21:00:22.888: INFO: csi-azuredisk-node-5wwhz started at 2019-11-05 20:16:09 +0000 UTC (0+3 container statuses recorded) I1105 21:00:22.889] Nov 5 21:00:22.888: INFO: Container azuredisk ready: true, restart count 0 I1105 21:00:22.889] Nov 5 21:00:22.888: INFO: Container liveness-probe ready: true, restart count 0 I1105 21:00:22.889] Nov 5 21:00:22.888: INFO: Container node-driver-registrar ready: true, restart count 0 I1105 21:00:22.890] Nov 5 21:00:22.888: INFO: blobfuse-flexvol-installer-p7hdb started at 2019-11-05 20:08:01 +0000 UTC (0+1 container statuses recorded) I1105 21:00:22.890] Nov 5 21:00:22.888: INFO: Container blobfuse-flexvol-installer ready: true, restart count 0 I1105 21:00:22.890] Nov 5 21:00:22.888: INFO: coredns-7f68dcdbdb-zh8g5 started at 2019-11-05 20:08:01 +0000 UTC (0+1 container statuses recorded) I1105 21:00:22.890] Nov 5 21:00:22.888: INFO: Container coredns ready: true, restart count 0 I1105 21:00:22.890] Nov 5 21:00:22.888: INFO: azure-ip-masq-agent-m7tv9 started at 2019-11-05 20:08:01 +0000 UTC (0+1 container statuses recorded) I1105 21:00:22.890] Nov 5 21:00:22.888: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 21:00:22.891] Nov 5 21:00:22.888: INFO: keyvault-flexvolume-smspt started at 2019-11-05 20:08:01 +0000 UTC (0+1 container statuses recorded) I1105 21:00:22.891] Nov 5 21:00:22.888: INFO: Container keyvault-flexvolume ready: true, restart count 0 I1105 21:00:22.891] Nov 5 21:00:22.888: INFO: metrics-server-864ffbc5c-wl84z started at 2019-11-05 20:08:01 +0000 UTC (0+1 container statuses recorded) I1105 21:00:22.891] Nov 5 21:00:22.888: INFO: Container metrics-server ready: true, restart count 0 I1105 21:00:23.143] Nov 5 21:00:23.143: INFO: I1105 21:00:23.143] Latency metrics for node k8s-agentpool-97435445-vmss000000 I1105 21:00:23.144] Nov 5 21:00:23.143: INFO: I1105 21:00:23.144] Logging node info for node k8s-agentpool-97435445-vmss000001 I1105 21:00:23.225] Nov 5 21:00:23.221: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-agentpool-97435445-vmss000001,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-agentpool-97435445-vmss000001,UID:75d3cc50-ca6d-45e0-a42a-6090e8d537dd,ResourceVersion:10301,Generation:0,CreationTimestamp:2019-11-05 20:07:34 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{agentpool: agentpool,beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_D4s_v3,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 1,kubernetes.azure.com/cluster: kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b,kubernetes.azure.com/role: agent,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-agentpool-97435445-vmss000001,kubernetes.io/os: linux,kubernetes.io/role: agent,node-role.kubernetes.io/agent: ,storageprofile: managed,storagetier: Premium_LRS,topology.disk.csi.azure.com/zone: 1,},Annotations:map[string]string{csi.volume.kubernetes.io/nodeid: {"disk.csi.azure.com":"k8s-agentpool-97435445-vmss000001"},node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.0.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool-97435445-vmss/virtualMachines/1,Unschedulable:false,Taints:[],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-csi-disk.csi.azure.com: {{8 0} {} 8 DecimalSI},cpu: {{4 0} {} 4 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{16797044736 0} {} 16403364Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-csi-disk.csi.azure.com: {{8 0} {} 8 DecimalSI},cpu: {{4 0} {} 4 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{16010612736 0} {} 15635364Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 20:08:25 +0000 UTC 2019-11-05 20:08:25 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 20:59:42 +0000 UTC 2019-11-05 20:07:26 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 20:59:42 +0000 UTC 2019-11-05 20:07:26 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 20:59:42 +0000 UTC 2019-11-05 20:07:26 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 20:59:42 +0000 UTC 2019-11-05 20:07:55 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.0.5} {Hostname k8s-agentpool-97435445-vmss000001}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:e78def54b2f545c6b94d88bbba523c60,SystemUUID:40AE8188-86AB-D848-84A4-D9254E1258BC,BootID:8587ec06-f4f4-43c1-86cd-06c295b05b63,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[k8sprowinternal.azurecr.io/azuredisk-csi@sha256:75d4ff5d7cd845d6e810c67796ecb037e852fb849ec1eb5b3194564008e6d1e0 k8sprowinternal.azurecr.io/azuredisk-csi:e2e-3e3ed045c6faa509dbd8d2890ee9f56aa2934887] 286930414} {[gcr.io/kubernetes-helm/tiller@sha256:f6d8f4ab9ba993b5f5b60a6edafe86352eabe474ffeb84cb6c79b8866dce45d1 gcr.io/kubernetes-helm/tiller:v2.11.0] 71821984} {[quay.io/k8scsi/csi-provisioner@sha256:d657c839dce87324fe2b677302913f9386f885f8746be7bea0ced5b0844e3433 quay.io/k8scsi/csi-provisioner:v1.3.1] 50596998} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[quay.io/k8scsi/csi-attacher@sha256:26fccd7a99d973845df1193b46ebdcc6ab8dc5f6e6be319750c471fce1742d13 quay.io/k8scsi/csi-attacher:v1.2.0] 46226754} {[quay.io/k8scsi/csi-cluster-driver-registrar@sha256:fafd75ae5442f192cfa8c2e792903aee30d5884b62e802e4464b0a895d21e3ef quay.io/k8scsi/csi-cluster-driver-registrar:v1.0.1] 45874691} {[quay.io/k8scsi/csi-snapshotter@sha256:a49e0da1af6f2bf717e41ba1eee8b5e6a1cbd66a709dd92cc43fe475fe2589eb quay.io/k8scsi/csi-snapshotter:v1.1.0] 44325352} {[mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume@sha256:4fd30d43947d4a54fc89ead7985beecfd3c9b2a93a0655a373b1608ab90bd5af mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume:v0.0.7] 22909487} {[quay.io/k8scsi/livenessprobe@sha256:8c9c8a0f330acc2e558ae6624309447d02522e76aa35af23fa19b88abf4381cb quay.io/k8scsi/livenessprobe:v1.0.2] 17415381} {[quay.io/k8scsi/csi-node-driver-registrar@sha256:13daf82fb99e951a4bff8ae5fc7c17c3a8fe7130be6400990d8f6076c32d4599 quay.io/k8scsi/csi-node-driver-registrar:v1.1.0] 15815995} {[quay.io/k8scsi/livenessprobe@sha256:dde617756e0f602adc566ab71fd885f1dad451ad3fb063ac991c95a2ff47aea5 quay.io/k8scsi/livenessprobe:v1.1.0] 14967303} {[mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume@sha256:23d8c6033f02a1ecad05127ebdc931bb871264228661bc122704b0974e4d9fdd mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume:1.0.8] 1159025} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 21:00:23.225] Nov 5 21:00:23.222: INFO: I1105 21:00:23.225] Logging kubelet events for node k8s-agentpool-97435445-vmss000001 I1105 21:00:23.312] Nov 5 21:00:23.312: INFO: I1105 21:00:23.312] Logging pods the kubelet thinks is on node k8s-agentpool-97435445-vmss000001 I1105 21:00:23.396] Nov 5 21:00:23.396: INFO: tiller-deploy-54c96cb5df-pt7x6 started at 2019-11-05 20:08:00 +0000 UTC (0+1 container statuses recorded) I1105 21:00:23.396] Nov 5 21:00:23.396: INFO: Container tiller ready: true, restart count 0 I1105 21:00:23.396] Nov 5 21:00:23.396: INFO: csi-azuredisk-node-q2dmt started at 2019-11-05 20:16:09 +0000 UTC (0+3 container statuses recorded) I1105 21:00:23.397] Nov 5 21:00:23.396: INFO: Container azuredisk ready: true, restart count 0 I1105 21:00:23.397] Nov 5 21:00:23.396: INFO: Container liveness-probe ready: true, restart count 0 I1105 21:00:23.397] Nov 5 21:00:23.396: INFO: Container node-driver-registrar ready: true, restart count 0 I1105 21:00:23.397] Nov 5 21:00:23.396: INFO: csi-azuredisk-controller-6f6587bfc6-62nrt started at 2019-11-05 20:16:09 +0000 UTC (0+6 container statuses recorded) I1105 21:00:23.397] Nov 5 21:00:23.396: INFO: Container azuredisk ready: true, restart count 0 I1105 21:00:23.397] Nov 5 21:00:23.396: INFO: Container cluster-driver-registrar ready: true, restart count 0 I1105 21:00:23.397] Nov 5 21:00:23.396: INFO: Container csi-attacher ready: true, restart count 0 I1105 21:00:23.398] Nov 5 21:00:23.396: INFO: Container csi-provisioner ready: true, restart count 0 I1105 21:00:23.398] Nov 5 21:00:23.396: INFO: Container csi-snapshotter ready: true, restart count 0 I1105 21:00:23.398] Nov 5 21:00:23.396: INFO: Container liveness-probe ready: true, restart count 0 I1105 21:00:23.398] Nov 5 21:00:23.396: INFO: blobfuse-flexvol-installer-j4s7t started at 2019-11-05 20:07:59 +0000 UTC (0+1 container statuses recorded) I1105 21:00:23.398] Nov 5 21:00:23.396: INFO: Container blobfuse-flexvol-installer ready: true, restart count 0 I1105 21:00:23.399] Nov 5 21:00:23.396: INFO: azure-ip-masq-agent-gdhcq started at 2019-11-05 20:07:59 +0000 UTC (0+1 container statuses recorded) I1105 21:00:23.399] Nov 5 21:00:23.396: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 21:00:23.399] Nov 5 21:00:23.396: INFO: keyvault-flexvolume-6bffx started at 2019-11-05 20:07:59 +0000 UTC (0+1 container statuses recorded) I1105 21:00:23.399] Nov 5 21:00:23.396: INFO: Container keyvault-flexvolume ready: true, restart count 0 I1105 21:00:23.399] Nov 5 21:00:23.396: INFO: kube-proxy-kjrpb started at 2019-11-05 20:08:00 +0000 UTC (0+1 container statuses recorded) I1105 21:00:23.399] Nov 5 21:00:23.396: INFO: Container kube-proxy ready: true, restart count 0 I1105 21:00:23.667] Nov 5 21:00:23.667: INFO: I1105 21:00:23.667] Latency metrics for node k8s-agentpool-97435445-vmss000001 I1105 21:00:23.667] Nov 5 21:00:23.667: INFO: I1105 21:00:23.668] Logging node info for node k8s-master-97435445-0 I1105 21:00:23.748] Nov 5 21:00:23.746: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-97435445-0,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-97435445-0,UID:df3ae36f-961c-4729-a8f5-34cda54f4ff9,ResourceVersion:10362,Generation:0,CreationTimestamp:2019-11-05 20:08:36 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 2,kubernetes.azure.com/cluster: kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-97435445-0,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.4.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b/providers/Microsoft.Compute/virtualMachines/k8s-master-97435445-0,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 20:08:46 +0000 UTC 2019-11-05 20:08:46 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 21:00:03 +0000 UTC 2019-11-05 20:08:28 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 21:00:03 +0000 UTC 2019-11-05 20:08:28 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 21:00:03 +0000 UTC 2019-11-05 20:08:28 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 21:00:03 +0000 UTC 2019-11-05 20:08:36 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.255.5} {Hostname k8s-master-97435445-0}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:3c381865192341c2868672e8626cef7d,SystemUUID:F3D259D3-7EB8-B844-A1F0-F954DCD0D76B,BootID:4948329e-cf57-48c3-aa83-5297bef70370,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 21:00:23.749] Nov 5 21:00:23.746: INFO: I1105 21:00:23.749] Logging kubelet events for node k8s-master-97435445-0 I1105 21:00:23.832] Nov 5 21:00:23.832: INFO: I1105 21:00:23.832] Logging pods the kubelet thinks is on node k8s-master-97435445-0 I1105 21:00:23.917] Nov 5 21:00:23.917: INFO: azure-ip-masq-agent-q6dcl started at 2019-11-05 20:08:38 +0000 UTC (0+1 container statuses recorded) I1105 21:00:23.918] Nov 5 21:00:23.917: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 21:00:23.918] Nov 5 21:00:23.917: INFO: kube-proxy-5wnd2 started at 2019-11-05 20:08:38 +0000 UTC (0+1 container statuses recorded) I1105 21:00:23.918] Nov 5 21:00:23.917: INFO: Container kube-proxy ready: true, restart count 0 I1105 21:00:23.918] Nov 5 21:00:23.917: INFO: cloud-controller-manager-k8s-master-97435445-0 started at 2019-11-05 20:08:22 +0000 UTC (0+1 container statuses recorded) I1105 21:00:23.918] Nov 5 21:00:23.917: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 21:00:23.919] Nov 5 21:00:23.917: INFO: kube-addon-manager-k8s-master-97435445-0 started at 2019-11-05 20:08:48 +0000 UTC (0+1 container statuses recorded) I1105 21:00:23.919] Nov 5 21:00:23.917: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 21:00:23.919] Nov 5 21:00:23.917: INFO: kube-apiserver-k8s-master-97435445-0 started at 2019-11-05 20:08:22 +0000 UTC (0+1 container statuses recorded) I1105 21:00:23.919] Nov 5 21:00:23.917: INFO: Container kube-apiserver ready: true, restart count 0 I1105 21:00:23.919] Nov 5 21:00:23.917: INFO: kube-controller-manager-k8s-master-97435445-0 started at 2019-11-05 20:08:22 +0000 UTC (0+1 container statuses recorded) I1105 21:00:23.919] Nov 5 21:00:23.917: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 21:00:23.920] Nov 5 21:00:23.917: INFO: kube-scheduler-k8s-master-97435445-0 started at 2019-11-05 20:08:22 +0000 UTC (0+1 container statuses recorded) I1105 21:00:23.920] Nov 5 21:00:23.917: INFO: Container kube-scheduler ready: true, restart count 0 I1105 21:00:24.171] Nov 5 21:00:24.170: INFO: I1105 21:00:24.171] Latency metrics for node k8s-master-97435445-0 I1105 21:00:24.171] Nov 5 21:00:24.170: INFO: I1105 21:00:24.171] Logging node info for node k8s-master-97435445-1 I1105 21:00:24.252] Nov 5 21:00:24.250: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-97435445-1,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-97435445-1,UID:05534a67-d493-4ab5-9b0f-5ff5490e234f,ResourceVersion:10265,Generation:0,CreationTimestamp:2019-11-05 20:07:34 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 0,kubernetes.azure.com/cluster: kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-97435445-1,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.1.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b/providers/Microsoft.Compute/virtualMachines/k8s-master-97435445-1,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 20:08:26 +0000 UTC 2019-11-05 20:08:26 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 20:59:31 +0000 UTC 2019-11-05 20:07:28 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 20:59:31 +0000 UTC 2019-11-05 20:07:28 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 20:59:31 +0000 UTC 2019-11-05 20:07:28 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 20:59:31 +0000 UTC 2019-11-05 20:07:55 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{Hostname k8s-master-97435445-1} {InternalIP 10.240.255.6}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:079b605103b74acabac18fd2ea63368a,SystemUUID:38220EEC-BEFF-4C46-AA05-5A214AAFDC28,BootID:b1b45df0-d925-4bcc-9830-0202df36ebb4,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 21:00:24.252] Nov 5 21:00:24.250: INFO: I1105 21:00:24.252] Logging kubelet events for node k8s-master-97435445-1 I1105 21:00:24.334] Nov 5 21:00:24.334: INFO: I1105 21:00:24.334] Logging pods the kubelet thinks is on node k8s-master-97435445-1 I1105 21:00:24.418] Nov 5 21:00:24.418: INFO: kube-scheduler-k8s-master-97435445-1 started at 2019-11-05 20:07:22 +0000 UTC (0+1 container statuses recorded) I1105 21:00:24.418] Nov 5 21:00:24.418: INFO: Container kube-scheduler ready: true, restart count 0 I1105 21:00:24.419] Nov 5 21:00:24.418: INFO: azure-ip-masq-agent-n29bm started at 2019-11-05 20:07:59 +0000 UTC (0+1 container statuses recorded) I1105 21:00:24.419] Nov 5 21:00:24.418: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 21:00:24.419] Nov 5 21:00:24.418: INFO: kube-proxy-h4s2c started at 2019-11-05 20:08:00 +0000 UTC (0+1 container statuses recorded) I1105 21:00:24.419] Nov 5 21:00:24.418: INFO: Container kube-proxy ready: true, restart count 0 I1105 21:00:24.419] Nov 5 21:00:24.418: INFO: cloud-controller-manager-k8s-master-97435445-1 started at 2019-11-05 20:07:22 +0000 UTC (0+1 container statuses recorded) I1105 21:00:24.420] Nov 5 21:00:24.418: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 21:00:24.420] Nov 5 21:00:24.418: INFO: kube-addon-manager-k8s-master-97435445-1 started at 2019-11-05 20:07:22 +0000 UTC (0+1 container statuses recorded) I1105 21:00:24.420] Nov 5 21:00:24.418: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 21:00:24.420] Nov 5 21:00:24.418: INFO: kube-apiserver-k8s-master-97435445-1 started at 2019-11-05 20:07:22 +0000 UTC (0+1 container statuses recorded) I1105 21:00:24.420] Nov 5 21:00:24.418: INFO: Container kube-apiserver ready: true, restart count 0 I1105 21:00:24.421] Nov 5 21:00:24.418: INFO: kube-controller-manager-k8s-master-97435445-1 started at 2019-11-05 20:07:22 +0000 UTC (0+1 container statuses recorded) I1105 21:00:24.421] Nov 5 21:00:24.418: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 21:00:24.682] Nov 5 21:00:24.682: INFO: I1105 21:00:24.683] Latency metrics for node k8s-master-97435445-1 I1105 21:00:24.683] Nov 5 21:00:24.682: INFO: I1105 21:00:24.683] Logging node info for node k8s-master-97435445-2 I1105 21:00:24.764] Nov 5 21:00:24.761: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-97435445-2,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-97435445-2,UID:fa55c895-1a93-4743-885e-03fce84b5332,ResourceVersion:10316,Generation:0,CreationTimestamp:2019-11-05 20:09:19 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 0,kubernetes.azure.com/cluster: kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-97435445-2,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.5.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b/providers/Microsoft.Compute/virtualMachines/k8s-master-97435445-2,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 20:09:19 +0000 UTC 2019-11-05 20:09:19 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 20:59:48 +0000 UTC 2019-11-05 20:09:12 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 20:59:48 +0000 UTC 2019-11-05 20:09:12 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 20:59:48 +0000 UTC 2019-11-05 20:09:12 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 20:59:48 +0000 UTC 2019-11-05 20:09:19 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.255.7} {Hostname k8s-master-97435445-2}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:d19f6709138f496da43c095cec7b2360,SystemUUID:10116777-A8D7-CB4A-A8B7-7EBCF2F23B62,BootID:50cd14a9-5497-4972-bf60-cf59a0cbcb5f,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 21:00:24.764] Nov 5 21:00:24.761: INFO: I1105 21:00:24.764] Logging kubelet events for node k8s-master-97435445-2 I1105 21:00:24.849] Nov 5 21:00:24.849: INFO: I1105 21:00:24.849] Logging pods the kubelet thinks is on node k8s-master-97435445-2 I1105 21:00:24.932] Nov 5 21:00:24.931: INFO: kube-scheduler-k8s-master-97435445-2 started at 2019-11-05 20:09:05 +0000 UTC (0+1 container statuses recorded) I1105 21:00:24.932] Nov 5 21:00:24.931: INFO: Container kube-scheduler ready: true, restart count 0 I1105 21:00:24.932] Nov 5 21:00:24.931: INFO: cloud-controller-manager-k8s-master-97435445-2 started at 2019-11-05 20:09:05 +0000 UTC (0+1 container statuses recorded) I1105 21:00:24.932] Nov 5 21:00:24.931: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 21:00:24.933] Nov 5 21:00:24.931: INFO: kube-addon-manager-k8s-master-97435445-2 started at 2019-11-05 20:09:05 +0000 UTC (0+1 container statuses recorded) I1105 21:00:24.933] Nov 5 21:00:24.931: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 21:00:24.933] Nov 5 21:00:24.931: INFO: azure-ip-masq-agent-sq7p5 started at 2019-11-05 20:09:19 +0000 UTC (0+1 container statuses recorded) I1105 21:00:24.933] Nov 5 21:00:24.931: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 21:00:24.933] Nov 5 21:00:24.931: INFO: kube-proxy-pz426 started at 2019-11-05 20:09:19 +0000 UTC (0+1 container statuses recorded) I1105 21:00:24.934] Nov 5 21:00:24.931: INFO: Container kube-proxy ready: true, restart count 0 I1105 21:00:24.934] Nov 5 21:00:24.931: INFO: kube-apiserver-k8s-master-97435445-2 started at 2019-11-05 20:09:05 +0000 UTC (0+1 container statuses recorded) I1105 21:00:24.934] Nov 5 21:00:24.931: INFO: Container kube-apiserver ready: true, restart count 0 I1105 21:00:24.934] Nov 5 21:00:24.931: INFO: kube-controller-manager-k8s-master-97435445-2 started at 2019-11-05 20:09:05 +0000 UTC (0+1 container statuses recorded) I1105 21:00:24.935] Nov 5 21:00:24.931: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 21:00:25.187] Nov 5 21:00:25.187: INFO: I1105 21:00:25.187] Latency metrics for node k8s-master-97435445-2 I1105 21:00:25.187] Nov 5 21:00:25.187: INFO: I1105 21:00:25.187] Logging node info for node k8s-master-97435445-3 I1105 21:00:25.268] Nov 5 21:00:25.265: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-97435445-3,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-97435445-3,UID:8ffd1d0d-4487-4680-a2f6-e724387be637,ResourceVersion:10318,Generation:0,CreationTimestamp:2019-11-05 20:09:21 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 1,kubernetes.azure.com/cluster: kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-97435445-3,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.6.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b/providers/Microsoft.Compute/virtualMachines/k8s-master-97435445-3,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 20:09:39 +0000 UTC 2019-11-05 20:09:39 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 20:59:48 +0000 UTC 2019-11-05 20:09:18 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 20:59:48 +0000 UTC 2019-11-05 20:09:18 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 20:59:48 +0000 UTC 2019-11-05 20:09:18 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 20:59:48 +0000 UTC 2019-11-05 20:09:21 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.255.8} {Hostname k8s-master-97435445-3}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:266064173bfb4297898c6c25769e8b20,SystemUUID:E6E44241-5015-7740-9A81-A668A8427206,BootID:579d658c-b0ad-47f1-a993-1ccea2b55789,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 21:00:25.268] Nov 5 21:00:25.265: INFO: I1105 21:00:25.268] Logging kubelet events for node k8s-master-97435445-3 I1105 21:00:25.350] Nov 5 21:00:25.350: INFO: I1105 21:00:25.350] Logging pods the kubelet thinks is on node k8s-master-97435445-3 I1105 21:00:25.433] Nov 5 21:00:25.433: INFO: kube-proxy-k45g8 started at 2019-11-05 20:09:21 +0000 UTC (0+1 container statuses recorded) I1105 21:00:25.433] Nov 5 21:00:25.433: INFO: Container kube-proxy ready: true, restart count 0 I1105 21:00:25.433] Nov 5 21:00:25.433: INFO: cloud-controller-manager-k8s-master-97435445-3 started at 2019-11-05 20:09:04 +0000 UTC (0+1 container statuses recorded) I1105 21:00:25.434] Nov 5 21:00:25.433: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 21:00:25.434] Nov 5 21:00:25.433: INFO: kube-addon-manager-k8s-master-97435445-3 started at 2019-11-05 20:09:05 +0000 UTC (0+1 container statuses recorded) I1105 21:00:25.434] Nov 5 21:00:25.433: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 21:00:25.434] Nov 5 21:00:25.433: INFO: kube-apiserver-k8s-master-97435445-3 started at 2019-11-05 20:09:05 +0000 UTC (0+1 container statuses recorded) I1105 21:00:25.434] Nov 5 21:00:25.433: INFO: Container kube-apiserver ready: true, restart count 0 I1105 21:00:25.435] Nov 5 21:00:25.433: INFO: kube-controller-manager-k8s-master-97435445-3 started at 2019-11-05 20:09:05 +0000 UTC (0+1 container statuses recorded) I1105 21:00:25.435] Nov 5 21:00:25.433: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 21:00:25.435] Nov 5 21:00:25.433: INFO: kube-scheduler-k8s-master-97435445-3 started at 2019-11-05 20:09:05 +0000 UTC (0+1 container statuses recorded) I1105 21:00:25.435] Nov 5 21:00:25.433: INFO: Container kube-scheduler ready: true, restart count 0 I1105 21:00:25.435] Nov 5 21:00:25.433: INFO: azure-ip-masq-agent-747cl started at 2019-11-05 20:09:21 +0000 UTC (0+1 container statuses recorded) I1105 21:00:25.435] Nov 5 21:00:25.433: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 21:00:25.693] Nov 5 21:00:25.693: INFO: I1105 21:00:25.693] Latency metrics for node k8s-master-97435445-3 I1105 21:00:25.693] Nov 5 21:00:25.693: INFO: I1105 21:00:25.693] Logging node info for node k8s-master-97435445-4 I1105 21:00:25.785] Nov 5 21:00:25.782: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-97435445-4,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-97435445-4,UID:6d547b41-3f92-4cc4-a2d7-dda2071776a2,ResourceVersion:10360,Generation:0,CreationTimestamp:2019-11-05 20:08:36 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 1,kubernetes.azure.com/cluster: kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-97435445-4,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.3.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b/providers/Microsoft.Compute/virtualMachines/k8s-master-97435445-4,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 20:08:59 +0000 UTC 2019-11-05 20:08:59 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 21:00:03 +0000 UTC 2019-11-05 20:08:29 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 21:00:03 +0000 UTC 2019-11-05 20:08:29 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 21:00:03 +0000 UTC 2019-11-05 20:08:29 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 21:00:03 +0000 UTC 2019-11-05 20:08:36 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.255.9} {Hostname k8s-master-97435445-4}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:56b393c61eb34ba49ebda5279eeedb30,SystemUUID:FD239C23-618C-0049-8734-9AF3FDC9E3C1,BootID:f342ea44-e719-4513-807d-5def7ac606e2,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 21:00:25.785] Nov 5 21:00:25.783: INFO: I1105 21:00:25.785] Logging kubelet events for node k8s-master-97435445-4 I1105 21:00:25.867] Nov 5 21:00:25.866: INFO: I1105 21:00:25.867] Logging pods the kubelet thinks is on node k8s-master-97435445-4 I1105 21:00:25.950] Nov 5 21:00:25.950: INFO: cloud-controller-manager-k8s-master-97435445-4 started at 2019-11-05 20:08:46 +0000 UTC (0+1 container statuses recorded) I1105 21:00:25.950] Nov 5 21:00:25.950: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 21:00:25.951] Nov 5 21:00:25.950: INFO: kube-addon-manager-k8s-master-97435445-4 started at 2019-11-05 20:08:23 +0000 UTC (0+1 container statuses recorded) I1105 21:00:25.951] Nov 5 21:00:25.950: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 21:00:25.951] Nov 5 21:00:25.950: INFO: kube-apiserver-k8s-master-97435445-4 started at 2019-11-05 20:08:23 +0000 UTC (0+1 container statuses recorded) I1105 21:00:25.951] Nov 5 21:00:25.950: INFO: Container kube-apiserver ready: true, restart count 0 I1105 21:00:25.951] Nov 5 21:00:25.950: INFO: kube-controller-manager-k8s-master-97435445-4 started at 2019-11-05 20:08:23 +0000 UTC (0+1 container statuses recorded) I1105 21:00:25.951] Nov 5 21:00:25.950: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 21:00:25.952] Nov 5 21:00:25.950: INFO: kube-scheduler-k8s-master-97435445-4 started at 2019-11-05 20:08:23 +0000 UTC (0+1 container statuses recorded) I1105 21:00:25.952] Nov 5 21:00:25.950: INFO: Container kube-scheduler ready: true, restart count 0 I1105 21:00:25.952] Nov 5 21:00:25.950: INFO: kube-proxy-vfbfr started at 2019-11-05 20:08:36 +0000 UTC (0+1 container statuses recorded) I1105 21:00:25.952] Nov 5 21:00:25.950: INFO: Container kube-proxy ready: true, restart count 0 I1105 21:00:25.952] Nov 5 21:00:25.950: INFO: azure-ip-masq-agent-kmkll started at 2019-11-05 20:08:36 +0000 UTC (0+1 container statuses recorded) I1105 21:00:25.952] Nov 5 21:00:25.950: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 21:00:26.233] Nov 5 21:00:26.232: INFO: I1105 21:00:26.233] Latency metrics for node k8s-master-97435445-4 I1105 21:00:26.233] Nov 5 21:00:26.232: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready I1105 21:00:26.313] STEP: Destroying namespace "azuredisk-9464" for this suite. I1105 21:00:40.558] Nov 5 21:00:40.558: INFO: Waiting up to 30s for server preferred namespaced resources to be successfully discovered I1105 21:00:43.572] Nov 5 21:00:43.572: INFO: namespace azuredisk-9464 deletion completed in 17.25911846s I1105 21:00:43.572] I1105 21:00:43.572] • Failure [322.378 seconds] I1105 21:00:43.573] [azuredisk-csi-e2e] Dynamic Provisioning I1105 21:00:43.573] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:33 I1105 21:00:43.573] [multi-az] I1105 21:00:43.573] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:40 I1105 21:00:43.573] should create a raw block volume on demand [It] I1105 21:00:43.573] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:101 I1105 21:00:43.573] I1105 21:00:43.573] Unexpected error: I1105 21:00:43.573] <*errors.errorString | 0xc0006b5660>: { I1105 21:00:43.574] s: "PersistentVolumeClaims [pvc-hqwk4] not all in phase Bound within 5m0s", I1105 21:00:43.574] } I1105 21:00:43.574] PersistentVolumeClaims [pvc-hqwk4] not all in phase Bound within 5m0s I1105 21:00:43.574] occurred I1105 21:00:43.574] I1105 21:00:43.574] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/testsuites/testsuites.go:278 I1105 21:00:43.574] ------------------------------ I1105 21:00:43.575] [azuredisk-csi-e2e] Dynamic Provisioning [single-az] I1105 21:00:43.575] should create a volume on demand and mount it as readOnly in a pod I1105 21:00:43.575] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:126 I1105 21:00:43.575] [BeforeEach] [single-az] I1105 21:00:43.575] /go/src/sigs.k8s.io/azuredisk-csi-driver/vendor/k8s.io/kubernetes/test/e2e/framework/framework.go:150 I1105 21:00:43.575] STEP: Creating a kubernetes client I1105 21:00:43.575] Nov 5 21:00:43.572: INFO: >>> kubeConfig: /workspace/aks971561099/kubeconfig/kubeconfig.eastus2.json I1105 21:00:43.576] STEP: Building a namespace api object, basename azuredisk I1105 21:00:43.811] STEP: Binding the e2e-test-privileged-psp PodSecurityPolicy to the default service account in azuredisk-6633 I1105 21:00:44.074] STEP: Waiting for a default service account to be provisioned in namespace I1105 21:00:44.150] [BeforeEach] [single-az] I1105 21:00:44.150] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:58 I1105 21:00:44.151] [It] should create a volume on demand and mount it as readOnly in a pod I1105 21:00:44.151] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:126 I1105 21:00:44.151] STEP: setting up the StorageClass I1105 21:00:44.151] STEP: creating a StorageClass I1105 21:00:44.232] STEP: setting up the PVC and PV I1105 21:00:44.233] STEP: creating a PVC I1105 21:00:44.319] STEP: waiting for PVC to be in phase "Bound" I1105 21:00:44.319] Nov 5 21:00:44.319: INFO: Waiting up to 5m0s for PersistentVolumeClaims [pvc-zkl6h] to have phase Bound I1105 21:00:44.398] Nov 5 21:00:44.397: INFO: PersistentVolumeClaim pvc-zkl6h found but phase is Pending instead of Bound. I1105 21:00:46.477] Nov 5 21:00:46.476: INFO: PersistentVolumeClaim pvc-zkl6h found but phase is Pending instead of Bound. I1105 21:00:48.556] Nov 5 21:00:48.555: INFO: PersistentVolumeClaim pvc-zkl6h found but phase is Pending instead of Bound. I1105 21:00:50.645] Nov 5 21:00:50.645: INFO: PersistentVolumeClaim pvc-zkl6h found but phase is Pending instead of Bound. I1105 21:00:52.724] Nov 5 21:00:52.724: INFO: PersistentVolumeClaim pvc-zkl6h found but phase is Pending instead of Bound. I1105 21:00:54.804] Nov 5 21:00:54.803: INFO: PersistentVolumeClaim pvc-zkl6h found but phase is Pending instead of Bound. I1105 21:00:56.883] Nov 5 21:00:56.883: INFO: PersistentVolumeClaim pvc-zkl6h found but phase is Pending instead of Bound. I1105 21:00:58.962] Nov 5 21:00:58.962: INFO: PersistentVolumeClaim pvc-zkl6h found but phase is Pending instead of Bound. I1105 21:01:01.041] Nov 5 21:01:01.040: INFO: PersistentVolumeClaim pvc-zkl6h found but phase is Pending instead of Bound. I1105 21:01:03.119] Nov 5 21:01:03.119: INFO: PersistentVolumeClaim pvc-zkl6h found but phase is Pending instead of Bound. I1105 21:01:05.201] Nov 5 21:01:05.201: INFO: PersistentVolumeClaim pvc-zkl6h found but phase is Pending instead of Bound. I1105 21:01:07.280] Nov 5 21:01:07.280: INFO: PersistentVolumeClaim pvc-zkl6h found but phase is Pending instead of Bound. I1105 21:01:09.360] Nov 5 21:01:09.360: INFO: PersistentVolumeClaim pvc-zkl6h found but phase is Pending instead of Bound. I1105 21:01:11.439] Nov 5 21:01:11.439: INFO: PersistentVolumeClaim pvc-zkl6h found but phase is Pending instead of Bound. I1105 21:01:13.519] Nov 5 21:01:13.519: INFO: PersistentVolumeClaim pvc-zkl6h found but phase is Pending instead of Bound. I1105 21:01:15.598] Nov 5 21:01:15.598: INFO: PersistentVolumeClaim pvc-zkl6h found but phase is Pending instead of Bound. I1105 21:01:17.677] Nov 5 21:01:17.677: INFO: PersistentVolumeClaim pvc-zkl6h found but phase is Pending instead of Bound. I1105 21:01:19.758] Nov 5 21:01:19.758: INFO: PersistentVolumeClaim pvc-zkl6h found but phase is Pending instead of Bound. I1105 21:01:21.838] Nov 5 21:01:21.838: INFO: PersistentVolumeClaim pvc-zkl6h found but phase is Pending instead of Bound. I1105 21:01:23.917] Nov 5 21:01:23.917: INFO: PersistentVolumeClaim pvc-zkl6h found but phase is Pending instead of Bound. I1105 21:01:25.998] Nov 5 21:01:25.997: INFO: PersistentVolumeClaim pvc-zkl6h found but phase is Pending instead of Bound. I1105 21:01:28.081] Nov 5 21:01:28.081: INFO: PersistentVolumeClaim pvc-zkl6h found but phase is Pending instead of Bound. I1105 21:01:30.161] Nov 5 21:01:30.160: INFO: PersistentVolumeClaim pvc-zkl6h found but phase is Pending instead of Bound. I1105 21:01:32.240] Nov 5 21:01:32.240: INFO: PersistentVolumeClaim pvc-zkl6h found but phase is Pending instead of Bound. I1105 21:01:34.320] Nov 5 21:01:34.320: INFO: PersistentVolumeClaim pvc-zkl6h found but phase is Pending instead of Bound. I1105 21:01:36.400] Nov 5 21:01:36.399: INFO: PersistentVolumeClaim pvc-zkl6h found but phase is Pending instead of Bound. I1105 21:01:38.479] Nov 5 21:01:38.479: INFO: PersistentVolumeClaim pvc-zkl6h found but phase is Pending instead of Bound. I1105 21:01:40.559] Nov 5 21:01:40.559: INFO: PersistentVolumeClaim pvc-zkl6h found but phase is Pending instead of Bound. I1105 21:01:42.638] Nov 5 21:01:42.638: INFO: PersistentVolumeClaim pvc-zkl6h found but phase is Pending instead of Bound. I1105 21:01:44.719] Nov 5 21:01:44.719: INFO: PersistentVolumeClaim pvc-zkl6h found but phase is Pending instead of Bound. I1105 21:01:46.798] Nov 5 21:01:46.797: INFO: PersistentVolumeClaim pvc-zkl6h found but phase is Pending instead of Bound. I1105 21:01:48.877] Nov 5 21:01:48.877: INFO: PersistentVolumeClaim pvc-zkl6h found but phase is Pending instead of Bound. I1105 21:01:50.956] Nov 5 21:01:50.955: INFO: PersistentVolumeClaim pvc-zkl6h found but phase is Pending instead of Bound. I1105 21:01:53.035] Nov 5 21:01:53.034: INFO: PersistentVolumeClaim pvc-zkl6h found but phase is Pending instead of Bound. I1105 21:01:55.114] Nov 5 21:01:55.114: INFO: PersistentVolumeClaim pvc-zkl6h found but phase is Pending instead of Bound. I1105 21:01:57.194] Nov 5 21:01:57.193: INFO: PersistentVolumeClaim pvc-zkl6h found but phase is Pending instead of Bound. I1105 21:01:59.272] Nov 5 21:01:59.272: INFO: PersistentVolumeClaim pvc-zkl6h found but phase is Pending instead of Bound. I1105 21:02:01.351] Nov 5 21:02:01.351: INFO: PersistentVolumeClaim pvc-zkl6h found but phase is Pending instead of Bound. I1105 21:02:03.430] Nov 5 21:02:03.430: INFO: PersistentVolumeClaim pvc-zkl6h found but phase is Pending instead of Bound. I1105 21:02:05.509] Nov 5 21:02:05.509: INFO: PersistentVolumeClaim pvc-zkl6h found but phase is Pending instead of Bound. I1105 21:02:07.588] Nov 5 21:02:07.588: INFO: PersistentVolumeClaim pvc-zkl6h found but phase is Pending instead of Bound. I1105 21:02:09.667] Nov 5 21:02:09.667: INFO: PersistentVolumeClaim pvc-zkl6h found but phase is Pending instead of Bound. I1105 21:02:11.746] Nov 5 21:02:11.746: INFO: PersistentVolumeClaim pvc-zkl6h found but phase is Pending instead of Bound. I1105 21:02:13.825] Nov 5 21:02:13.825: INFO: PersistentVolumeClaim pvc-zkl6h found but phase is Pending instead of Bound. I1105 21:02:15.904] Nov 5 21:02:15.904: INFO: PersistentVolumeClaim pvc-zkl6h found but phase is Pending instead of Bound. I1105 21:02:18.020] Nov 5 21:02:18.020: INFO: PersistentVolumeClaim pvc-zkl6h found but phase is Pending instead of Bound. I1105 21:02:20.101] Nov 5 21:02:20.101: INFO: PersistentVolumeClaim pvc-zkl6h found but phase is Pending instead of Bound. I1105 21:02:22.230] Nov 5 21:02:22.229: INFO: PersistentVolumeClaim pvc-zkl6h found but phase is Pending instead of Bound. I1105 21:02:24.309] Nov 5 21:02:24.308: INFO: PersistentVolumeClaim pvc-zkl6h found but phase is Pending instead of Bound. I1105 21:02:26.388] Nov 5 21:02:26.388: INFO: PersistentVolumeClaim pvc-zkl6h found but phase is Pending instead of Bound. I1105 21:02:28.471] Nov 5 21:02:28.471: INFO: PersistentVolumeClaim pvc-zkl6h found but phase is Pending instead of Bound. I1105 21:02:30.550] Nov 5 21:02:30.550: INFO: PersistentVolumeClaim pvc-zkl6h found but phase is Pending instead of Bound. I1105 21:02:32.629] Nov 5 21:02:32.629: INFO: PersistentVolumeClaim pvc-zkl6h found but phase is Pending instead of Bound. I1105 21:02:34.710] Nov 5 21:02:34.710: INFO: PersistentVolumeClaim pvc-zkl6h found but phase is Pending instead of Bound. I1105 21:02:36.789] Nov 5 21:02:36.788: INFO: PersistentVolumeClaim pvc-zkl6h found but phase is Pending instead of Bound. I1105 21:02:38.868] Nov 5 21:02:38.868: INFO: PersistentVolumeClaim pvc-zkl6h found but phase is Pending instead of Bound. I1105 21:02:40.947] Nov 5 21:02:40.947: INFO: PersistentVolumeClaim pvc-zkl6h found but phase is Pending instead of Bound. I1105 21:02:43.026] Nov 5 21:02:43.026: INFO: PersistentVolumeClaim pvc-zkl6h found but phase is Pending instead of Bound. I1105 21:02:45.106] Nov 5 21:02:45.106: INFO: PersistentVolumeClaim pvc-zkl6h found but phase is Pending instead of Bound. I1105 21:02:47.185] Nov 5 21:02:47.185: INFO: PersistentVolumeClaim pvc-zkl6h found but phase is Pending instead of Bound. I1105 21:02:49.264] Nov 5 21:02:49.264: INFO: PersistentVolumeClaim pvc-zkl6h found but phase is Pending instead of Bound. I1105 21:02:51.343] Nov 5 21:02:51.343: INFO: PersistentVolumeClaim pvc-zkl6h found but phase is Pending instead of Bound. I1105 21:02:53.422] Nov 5 21:02:53.421: INFO: PersistentVolumeClaim pvc-zkl6h found but phase is Pending instead of Bound. I1105 21:02:55.501] Nov 5 21:02:55.500: INFO: PersistentVolumeClaim pvc-zkl6h found but phase is Pending instead of Bound. I1105 21:02:57.581] Nov 5 21:02:57.580: INFO: PersistentVolumeClaim pvc-zkl6h found but phase is Pending instead of Bound. I1105 21:02:59.661] Nov 5 21:02:59.661: INFO: PersistentVolumeClaim pvc-zkl6h found but phase is Pending instead of Bound. I1105 21:03:01.740] Nov 5 21:03:01.740: INFO: PersistentVolumeClaim pvc-zkl6h found but phase is Pending instead of Bound. I1105 21:03:03.820] Nov 5 21:03:03.820: INFO: PersistentVolumeClaim pvc-zkl6h found but phase is Pending instead of Bound. I1105 21:03:05.900] Nov 5 21:03:05.899: INFO: PersistentVolumeClaim pvc-zkl6h found but phase is Pending instead of Bound. I1105 21:03:07.979] Nov 5 21:03:07.978: INFO: PersistentVolumeClaim pvc-zkl6h found but phase is Pending instead of Bound. I1105 21:03:10.057] Nov 5 21:03:10.057: INFO: PersistentVolumeClaim pvc-zkl6h found but phase is Pending instead of Bound. I1105 21:03:12.137] Nov 5 21:03:12.137: INFO: PersistentVolumeClaim pvc-zkl6h found but phase is Pending instead of Bound. I1105 21:03:14.219] Nov 5 21:03:14.218: INFO: PersistentVolumeClaim pvc-zkl6h found but phase is Pending instead of Bound. I1105 21:03:16.297] Nov 5 21:03:16.297: INFO: PersistentVolumeClaim pvc-zkl6h found but phase is Pending instead of Bound. I1105 21:03:18.377] Nov 5 21:03:18.376: INFO: PersistentVolumeClaim pvc-zkl6h found but phase is Pending instead of Bound. I1105 21:03:20.456] Nov 5 21:03:20.456: INFO: PersistentVolumeClaim pvc-zkl6h found but phase is Pending instead of Bound. I1105 21:03:22.535] Nov 5 21:03:22.535: INFO: PersistentVolumeClaim pvc-zkl6h found but phase is Pending instead of Bound. I1105 21:03:24.615] Nov 5 21:03:24.615: INFO: PersistentVolumeClaim pvc-zkl6h found but phase is Pending instead of Bound. I1105 21:03:26.694] Nov 5 21:03:26.694: INFO: PersistentVolumeClaim pvc-zkl6h found but phase is Pending instead of Bound. I1105 21:03:28.776] Nov 5 21:03:28.776: INFO: PersistentVolumeClaim pvc-zkl6h found but phase is Pending instead of Bound. I1105 21:03:30.855] Nov 5 21:03:30.855: INFO: PersistentVolumeClaim pvc-zkl6h found but phase is Pending instead of Bound. I1105 21:03:32.936] Nov 5 21:03:32.935: INFO: PersistentVolumeClaim pvc-zkl6h found but phase is Pending instead of Bound. I1105 21:03:35.014] Nov 5 21:03:35.014: INFO: PersistentVolumeClaim pvc-zkl6h found but phase is Pending instead of Bound. I1105 21:03:37.095] Nov 5 21:03:37.095: INFO: PersistentVolumeClaim pvc-zkl6h found but phase is Pending instead of Bound. I1105 21:03:39.175] Nov 5 21:03:39.175: INFO: PersistentVolumeClaim pvc-zkl6h found but phase is Pending instead of Bound. I1105 21:03:41.254] Nov 5 21:03:41.254: INFO: PersistentVolumeClaim pvc-zkl6h found but phase is Pending instead of Bound. I1105 21:03:43.334] Nov 5 21:03:43.333: INFO: PersistentVolumeClaim pvc-zkl6h found but phase is Pending instead of Bound. I1105 21:03:45.413] Nov 5 21:03:45.413: INFO: PersistentVolumeClaim pvc-zkl6h found but phase is Pending instead of Bound. I1105 21:03:47.492] Nov 5 21:03:47.491: INFO: PersistentVolumeClaim pvc-zkl6h found but phase is Pending instead of Bound. I1105 21:03:49.571] Nov 5 21:03:49.571: INFO: PersistentVolumeClaim pvc-zkl6h found but phase is Pending instead of Bound. I1105 21:03:51.651] Nov 5 21:03:51.650: INFO: PersistentVolumeClaim pvc-zkl6h found but phase is Pending instead of Bound. I1105 21:03:53.730] Nov 5 21:03:53.729: INFO: PersistentVolumeClaim pvc-zkl6h found but phase is Pending instead of Bound. I1105 21:03:55.809] Nov 5 21:03:55.808: INFO: PersistentVolumeClaim pvc-zkl6h found but phase is Pending instead of Bound. I1105 21:03:57.890] Nov 5 21:03:57.890: INFO: PersistentVolumeClaim pvc-zkl6h found but phase is Pending instead of Bound. I1105 21:03:59.969] Nov 5 21:03:59.969: INFO: PersistentVolumeClaim pvc-zkl6h found but phase is Pending instead of Bound. I1105 21:04:02.048] Nov 5 21:04:02.048: INFO: PersistentVolumeClaim pvc-zkl6h found but phase is Pending instead of Bound. I1105 21:04:04.128] Nov 5 21:04:04.127: INFO: PersistentVolumeClaim pvc-zkl6h found but phase is Pending instead of Bound. I1105 21:04:06.216] Nov 5 21:04:06.216: INFO: PersistentVolumeClaim pvc-zkl6h found but phase is Pending instead of Bound. I1105 21:04:08.295] Nov 5 21:04:08.295: INFO: PersistentVolumeClaim pvc-zkl6h found but phase is Pending instead of Bound. I1105 21:04:10.374] Nov 5 21:04:10.374: INFO: PersistentVolumeClaim pvc-zkl6h found but phase is Pending instead of Bound. I1105 21:04:12.453] Nov 5 21:04:12.453: INFO: PersistentVolumeClaim pvc-zkl6h found but phase is Pending instead of Bound. I1105 21:04:14.532] Nov 5 21:04:14.532: INFO: PersistentVolumeClaim pvc-zkl6h found but phase is Pending instead of Bound. I1105 21:04:16.611] Nov 5 21:04:16.611: INFO: PersistentVolumeClaim pvc-zkl6h found but phase is Pending instead of Bound. I1105 21:04:18.690] Nov 5 21:04:18.690: INFO: PersistentVolumeClaim pvc-zkl6h found but phase is Pending instead of Bound. I1105 21:04:20.769] Nov 5 21:04:20.769: INFO: PersistentVolumeClaim pvc-zkl6h found but phase is Pending instead of Bound. I1105 21:04:22.848] Nov 5 21:04:22.848: INFO: PersistentVolumeClaim pvc-zkl6h found but phase is Pending instead of Bound. I1105 21:04:24.927] Nov 5 21:04:24.927: INFO: PersistentVolumeClaim pvc-zkl6h found but phase is Pending instead of Bound. I1105 21:04:27.006] Nov 5 21:04:27.006: INFO: PersistentVolumeClaim pvc-zkl6h found but phase is Pending instead of Bound. I1105 21:04:29.086] Nov 5 21:04:29.085: INFO: PersistentVolumeClaim pvc-zkl6h found but phase is Pending instead of Bound. I1105 21:04:31.164] Nov 5 21:04:31.164: INFO: PersistentVolumeClaim pvc-zkl6h found but phase is Pending instead of Bound. I1105 21:04:33.244] Nov 5 21:04:33.243: INFO: PersistentVolumeClaim pvc-zkl6h found but phase is Pending instead of Bound. I1105 21:04:35.324] Nov 5 21:04:35.324: INFO: PersistentVolumeClaim pvc-zkl6h found but phase is Pending instead of Bound. I1105 21:04:37.403] Nov 5 21:04:37.403: INFO: PersistentVolumeClaim pvc-zkl6h found but phase is Pending instead of Bound. I1105 21:04:39.481] Nov 5 21:04:39.481: INFO: PersistentVolumeClaim pvc-zkl6h found but phase is Pending instead of Bound. I1105 21:04:41.560] Nov 5 21:04:41.560: INFO: PersistentVolumeClaim pvc-zkl6h found but phase is Pending instead of Bound. I1105 21:04:43.639] Nov 5 21:04:43.639: INFO: PersistentVolumeClaim pvc-zkl6h found but phase is Pending instead of Bound. I1105 21:04:45.718] Nov 5 21:04:45.718: INFO: PersistentVolumeClaim pvc-zkl6h found but phase is Pending instead of Bound. I1105 21:04:47.797] Nov 5 21:04:47.797: INFO: PersistentVolumeClaim pvc-zkl6h found but phase is Pending instead of Bound. I1105 21:04:49.876] Nov 5 21:04:49.876: INFO: PersistentVolumeClaim pvc-zkl6h found but phase is Pending instead of Bound. I1105 21:04:51.955] Nov 5 21:04:51.955: INFO: PersistentVolumeClaim pvc-zkl6h found but phase is Pending instead of Bound. I1105 21:04:54.035] Nov 5 21:04:54.035: INFO: PersistentVolumeClaim pvc-zkl6h found but phase is Pending instead of Bound. I1105 21:04:56.114] Nov 5 21:04:56.114: INFO: PersistentVolumeClaim pvc-zkl6h found but phase is Pending instead of Bound. I1105 21:04:58.212] Nov 5 21:04:58.212: INFO: PersistentVolumeClaim pvc-zkl6h found but phase is Pending instead of Bound. I1105 21:05:00.291] Nov 5 21:05:00.291: INFO: PersistentVolumeClaim pvc-zkl6h found but phase is Pending instead of Bound. I1105 21:05:02.370] Nov 5 21:05:02.370: INFO: PersistentVolumeClaim pvc-zkl6h found but phase is Pending instead of Bound. I1105 21:05:04.450] Nov 5 21:05:04.450: INFO: PersistentVolumeClaim pvc-zkl6h found but phase is Pending instead of Bound. I1105 21:05:06.534] Nov 5 21:05:06.534: INFO: PersistentVolumeClaim pvc-zkl6h found but phase is Pending instead of Bound. I1105 21:05:08.613] Nov 5 21:05:08.613: INFO: PersistentVolumeClaim pvc-zkl6h found but phase is Pending instead of Bound. I1105 21:05:10.701] Nov 5 21:05:10.701: INFO: PersistentVolumeClaim pvc-zkl6h found but phase is Pending instead of Bound. I1105 21:05:12.780] Nov 5 21:05:12.780: INFO: PersistentVolumeClaim pvc-zkl6h found but phase is Pending instead of Bound. I1105 21:05:14.860] Nov 5 21:05:14.860: INFO: PersistentVolumeClaim pvc-zkl6h found but phase is Pending instead of Bound. I1105 21:05:16.939] Nov 5 21:05:16.938: INFO: PersistentVolumeClaim pvc-zkl6h found but phase is Pending instead of Bound. I1105 21:05:19.018] Nov 5 21:05:19.018: INFO: PersistentVolumeClaim pvc-zkl6h found but phase is Pending instead of Bound. I1105 21:05:21.097] Nov 5 21:05:21.097: INFO: PersistentVolumeClaim pvc-zkl6h found but phase is Pending instead of Bound. I1105 21:05:23.176] Nov 5 21:05:23.176: INFO: PersistentVolumeClaim pvc-zkl6h found but phase is Pending instead of Bound. I1105 21:05:25.255] Nov 5 21:05:25.255: INFO: PersistentVolumeClaim pvc-zkl6h found but phase is Pending instead of Bound. I1105 21:05:27.334] Nov 5 21:05:27.334: INFO: PersistentVolumeClaim pvc-zkl6h found but phase is Pending instead of Bound. I1105 21:05:29.413] Nov 5 21:05:29.413: INFO: PersistentVolumeClaim pvc-zkl6h found but phase is Pending instead of Bound. I1105 21:05:31.492] Nov 5 21:05:31.492: INFO: PersistentVolumeClaim pvc-zkl6h found but phase is Pending instead of Bound. I1105 21:05:33.574] Nov 5 21:05:33.574: INFO: PersistentVolumeClaim pvc-zkl6h found but phase is Pending instead of Bound. I1105 21:05:35.653] Nov 5 21:05:35.653: INFO: PersistentVolumeClaim pvc-zkl6h found but phase is Pending instead of Bound. I1105 21:05:37.732] Nov 5 21:05:37.732: INFO: PersistentVolumeClaim pvc-zkl6h found but phase is Pending instead of Bound. I1105 21:05:39.813] Nov 5 21:05:39.813: INFO: PersistentVolumeClaim pvc-zkl6h found but phase is Pending instead of Bound. I1105 21:05:41.893] Nov 5 21:05:41.892: INFO: PersistentVolumeClaim pvc-zkl6h found but phase is Pending instead of Bound. I1105 21:05:43.972] Nov 5 21:05:43.972: INFO: PersistentVolumeClaim pvc-zkl6h found but phase is Pending instead of Bound. I1105 21:05:45.972] Nov 5 21:05:45.972: INFO: Unexpected error occurred: PersistentVolumeClaims [pvc-zkl6h] not all in phase Bound within 5m0s I1105 21:05:45.973] [AfterEach] [single-az] I1105 21:05:45.973] /go/src/sigs.k8s.io/azuredisk-csi-driver/vendor/k8s.io/kubernetes/test/e2e/framework/framework.go:151 I1105 21:05:45.973] STEP: Collecting events from namespace "azuredisk-6633". I1105 21:05:46.052] STEP: Found 3 events. I1105 21:05:46.053] Nov 5 21:05:46.052: INFO: At 2019-11-05 21:00:44 +0000 UTC - event for pvc-zkl6h: {disk.csi.azure.com_csi-azuredisk-controller-6f6587bfc6-62nrt_3930bc69-0009-11ea-8cf5-829a27ff7d5a } Provisioning: External provisioner is provisioning volume for claim "azuredisk-6633/pvc-zkl6h" I1105 21:05:46.053] Nov 5 21:05:46.052: INFO: At 2019-11-05 21:00:44 +0000 UTC - event for pvc-zkl6h: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "disk.csi.azure.com" or manually created by system administrator I1105 21:05:46.054] Nov 5 21:05:46.052: INFO: At 2019-11-05 21:00:44 +0000 UTC - event for pvc-zkl6h: {disk.csi.azure.com_csi-azuredisk-controller-6f6587bfc6-62nrt_3930bc69-0009-11ea-8cf5-829a27ff7d5a } ProvisioningFailed: failed to provision volume with StorageClass "azuredisk-6633-disk.csi.azure.com-dynamic-sc-wp4x5": rpc error: code = Unknown desc = compute.DisksClient#CreateOrUpdate: Failure sending request: StatusCode=400 -- Original Error: Code="InvalidAvailabilityZone" Message="The zone(s) '' for resource 'Microsoft.Compute/disks/pvc-2b5dcbda-7a31-417c-ab49-147fe574629a' is not supported. The supported zones for location 'eastus2' are '1,2,3'" I1105 21:05:46.131] Nov 5 21:05:46.131: INFO: POD NODE PHASE GRACE CONDITIONS I1105 21:05:46.131] Nov 5 21:05:46.131: INFO: I1105 21:05:46.369] Nov 5 21:05:46.369: INFO: I1105 21:05:46.369] Logging node info for node k8s-agentpool-97435445-vmss000000 I1105 21:05:46.451] Nov 5 21:05:46.447: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-agentpool-97435445-vmss000000,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-agentpool-97435445-vmss000000,UID:e955dc47-8c8f-4142-9241-3d409f32e42d,ResourceVersion:11377,Generation:0,CreationTimestamp:2019-11-05 20:07:34 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{agentpool: agentpool,beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_D4s_v3,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 0,kubernetes.azure.com/cluster: kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b,kubernetes.azure.com/role: agent,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-agentpool-97435445-vmss000000,kubernetes.io/os: linux,kubernetes.io/role: agent,node-role.kubernetes.io/agent: ,storageprofile: managed,storagetier: Premium_LRS,topology.disk.csi.azure.com/zone: 0,},Annotations:map[string]string{csi.volume.kubernetes.io/nodeid: {"disk.csi.azure.com":"k8s-agentpool-97435445-vmss000000"},node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.2.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool-97435445-vmss/virtualMachines/0,Unschedulable:false,Taints:[],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-csi-disk.csi.azure.com: {{8 0} {} 8 DecimalSI},cpu: {{4 0} {} 4 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{16797040640 0} {} BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-csi-disk.csi.azure.com: {{8 0} {} 8 DecimalSI},cpu: {{4 0} {} 4 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{16010608640 0} {} BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 20:08:01 +0000 UTC 2019-11-05 20:08:01 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 21:05:44 +0000 UTC 2019-11-05 20:07:27 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 21:05:44 +0000 UTC 2019-11-05 20:07:27 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 21:05:44 +0000 UTC 2019-11-05 20:07:27 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 21:05:44 +0000 UTC 2019-11-05 20:07:55 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.0.4} {Hostname k8s-agentpool-97435445-vmss000000}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:bfb55325809340228149d2d93f50df58,SystemUUID:CB2ED036-0439-F641-BC7F-640C3638C027,BootID:a917f4d1-549d-471d-b7dc-47ae1715d97d,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[k8sprowinternal.azurecr.io/azuredisk-csi@sha256:75d4ff5d7cd845d6e810c67796ecb037e852fb849ec1eb5b3194564008e6d1e0 k8sprowinternal.azurecr.io/azuredisk-csi:e2e-3e3ed045c6faa509dbd8d2890ee9f56aa2934887] 286930414} {[k8s.gcr.io/kubernetes-dashboard-amd64@sha256:0ae6b69432e78069c5ce2bcde0fe409c5c4d6f0f4d9cd50a17974fea38898747 k8s.gcr.io/kubernetes-dashboard-amd64:v1.10.1] 121711221} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/metrics-server-amd64@sha256:49a9f12f7067d11f42c803dbe61ed2c1299959ad85cb315b25ff7eef8e6b8892 k8s.gcr.io/metrics-server-amd64:v0.2.1] 42541759} {[k8s.gcr.io/coredns@sha256:e83beb5e43f8513fa735e77ffc5859640baea30a882a11cc75c4c3244a737d3c k8s.gcr.io/coredns:1.5.0] 42488424} {[mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume@sha256:4fd30d43947d4a54fc89ead7985beecfd3c9b2a93a0655a373b1608ab90bd5af mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume:v0.0.7] 22909487} {[quay.io/k8scsi/livenessprobe@sha256:8c9c8a0f330acc2e558ae6624309447d02522e76aa35af23fa19b88abf4381cb quay.io/k8scsi/livenessprobe:v1.0.2] 17415381} {[quay.io/k8scsi/csi-node-driver-registrar@sha256:13daf82fb99e951a4bff8ae5fc7c17c3a8fe7130be6400990d8f6076c32d4599 quay.io/k8scsi/csi-node-driver-registrar:v1.1.0] 15815995} {[mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume@sha256:23d8c6033f02a1ecad05127ebdc931bb871264228661bc122704b0974e4d9fdd mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume:1.0.8] 1159025} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 21:05:46.451] Nov 5 21:05:46.448: INFO: I1105 21:05:46.451] Logging kubelet events for node k8s-agentpool-97435445-vmss000000 I1105 21:05:46.531] Nov 5 21:05:46.531: INFO: I1105 21:05:46.531] Logging pods the kubelet thinks is on node k8s-agentpool-97435445-vmss000000 I1105 21:05:46.633] Nov 5 21:05:46.633: INFO: azure-ip-masq-agent-m7tv9 started at 2019-11-05 20:08:01 +0000 UTC (0+1 container statuses recorded) I1105 21:05:46.633] Nov 5 21:05:46.633: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 21:05:46.634] Nov 5 21:05:46.633: INFO: keyvault-flexvolume-smspt started at 2019-11-05 20:08:01 +0000 UTC (0+1 container statuses recorded) I1105 21:05:46.634] Nov 5 21:05:46.633: INFO: Container keyvault-flexvolume ready: true, restart count 0 I1105 21:05:46.634] Nov 5 21:05:46.633: INFO: metrics-server-864ffbc5c-wl84z started at 2019-11-05 20:08:01 +0000 UTC (0+1 container statuses recorded) I1105 21:05:46.634] Nov 5 21:05:46.633: INFO: Container metrics-server ready: true, restart count 0 I1105 21:05:46.634] Nov 5 21:05:46.633: INFO: kube-proxy-8mt7f started at 2019-11-05 20:08:01 +0000 UTC (0+1 container statuses recorded) I1105 21:05:46.634] Nov 5 21:05:46.633: INFO: Container kube-proxy ready: true, restart count 0 I1105 21:05:46.634] Nov 5 21:05:46.633: INFO: kubernetes-dashboard-66dd8b8df7-t6j4l started at 2019-11-05 20:08:01 +0000 UTC (0+1 container statuses recorded) I1105 21:05:46.635] Nov 5 21:05:46.633: INFO: Container kubernetes-dashboard ready: true, restart count 0 I1105 21:05:46.635] Nov 5 21:05:46.633: INFO: csi-azuredisk-node-5wwhz started at 2019-11-05 20:16:09 +0000 UTC (0+3 container statuses recorded) I1105 21:05:46.635] Nov 5 21:05:46.633: INFO: Container azuredisk ready: true, restart count 0 I1105 21:05:46.635] Nov 5 21:05:46.633: INFO: Container liveness-probe ready: true, restart count 0 I1105 21:05:46.635] Nov 5 21:05:46.633: INFO: Container node-driver-registrar ready: true, restart count 0 I1105 21:05:46.635] Nov 5 21:05:46.633: INFO: blobfuse-flexvol-installer-p7hdb started at 2019-11-05 20:08:01 +0000 UTC (0+1 container statuses recorded) I1105 21:05:46.635] Nov 5 21:05:46.633: INFO: Container blobfuse-flexvol-installer ready: true, restart count 0 I1105 21:05:46.635] Nov 5 21:05:46.633: INFO: coredns-7f68dcdbdb-zh8g5 started at 2019-11-05 20:08:01 +0000 UTC (0+1 container statuses recorded) I1105 21:05:46.636] Nov 5 21:05:46.633: INFO: Container coredns ready: true, restart count 0 I1105 21:05:46.893] Nov 5 21:05:46.893: INFO: I1105 21:05:46.894] Latency metrics for node k8s-agentpool-97435445-vmss000000 I1105 21:05:46.894] Nov 5 21:05:46.893: INFO: I1105 21:05:46.894] Logging node info for node k8s-agentpool-97435445-vmss000001 I1105 21:05:46.976] Nov 5 21:05:46.972: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-agentpool-97435445-vmss000001,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-agentpool-97435445-vmss000001,UID:75d3cc50-ca6d-45e0-a42a-6090e8d537dd,ResourceVersion:11375,Generation:0,CreationTimestamp:2019-11-05 20:07:34 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{agentpool: agentpool,beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_D4s_v3,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 1,kubernetes.azure.com/cluster: kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b,kubernetes.azure.com/role: agent,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-agentpool-97435445-vmss000001,kubernetes.io/os: linux,kubernetes.io/role: agent,node-role.kubernetes.io/agent: ,storageprofile: managed,storagetier: Premium_LRS,topology.disk.csi.azure.com/zone: 1,},Annotations:map[string]string{csi.volume.kubernetes.io/nodeid: {"disk.csi.azure.com":"k8s-agentpool-97435445-vmss000001"},node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.0.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool-97435445-vmss/virtualMachines/1,Unschedulable:false,Taints:[],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-csi-disk.csi.azure.com: {{8 0} {} 8 DecimalSI},cpu: {{4 0} {} 4 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{16797044736 0} {} 16403364Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-csi-disk.csi.azure.com: {{8 0} {} 8 DecimalSI},cpu: {{4 0} {} 4 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{16010612736 0} {} 15635364Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 20:08:25 +0000 UTC 2019-11-05 20:08:25 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 21:05:43 +0000 UTC 2019-11-05 20:07:26 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 21:05:43 +0000 UTC 2019-11-05 20:07:26 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 21:05:43 +0000 UTC 2019-11-05 20:07:26 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 21:05:43 +0000 UTC 2019-11-05 20:07:55 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.0.5} {Hostname k8s-agentpool-97435445-vmss000001}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:e78def54b2f545c6b94d88bbba523c60,SystemUUID:40AE8188-86AB-D848-84A4-D9254E1258BC,BootID:8587ec06-f4f4-43c1-86cd-06c295b05b63,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[k8sprowinternal.azurecr.io/azuredisk-csi@sha256:75d4ff5d7cd845d6e810c67796ecb037e852fb849ec1eb5b3194564008e6d1e0 k8sprowinternal.azurecr.io/azuredisk-csi:e2e-3e3ed045c6faa509dbd8d2890ee9f56aa2934887] 286930414} {[gcr.io/kubernetes-helm/tiller@sha256:f6d8f4ab9ba993b5f5b60a6edafe86352eabe474ffeb84cb6c79b8866dce45d1 gcr.io/kubernetes-helm/tiller:v2.11.0] 71821984} {[quay.io/k8scsi/csi-provisioner@sha256:d657c839dce87324fe2b677302913f9386f885f8746be7bea0ced5b0844e3433 quay.io/k8scsi/csi-provisioner:v1.3.1] 50596998} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[quay.io/k8scsi/csi-attacher@sha256:26fccd7a99d973845df1193b46ebdcc6ab8dc5f6e6be319750c471fce1742d13 quay.io/k8scsi/csi-attacher:v1.2.0] 46226754} {[quay.io/k8scsi/csi-cluster-driver-registrar@sha256:fafd75ae5442f192cfa8c2e792903aee30d5884b62e802e4464b0a895d21e3ef quay.io/k8scsi/csi-cluster-driver-registrar:v1.0.1] 45874691} {[quay.io/k8scsi/csi-snapshotter@sha256:a49e0da1af6f2bf717e41ba1eee8b5e6a1cbd66a709dd92cc43fe475fe2589eb quay.io/k8scsi/csi-snapshotter:v1.1.0] 44325352} {[mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume@sha256:4fd30d43947d4a54fc89ead7985beecfd3c9b2a93a0655a373b1608ab90bd5af mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume:v0.0.7] 22909487} {[quay.io/k8scsi/livenessprobe@sha256:8c9c8a0f330acc2e558ae6624309447d02522e76aa35af23fa19b88abf4381cb quay.io/k8scsi/livenessprobe:v1.0.2] 17415381} {[quay.io/k8scsi/csi-node-driver-registrar@sha256:13daf82fb99e951a4bff8ae5fc7c17c3a8fe7130be6400990d8f6076c32d4599 quay.io/k8scsi/csi-node-driver-registrar:v1.1.0] 15815995} {[quay.io/k8scsi/livenessprobe@sha256:dde617756e0f602adc566ab71fd885f1dad451ad3fb063ac991c95a2ff47aea5 quay.io/k8scsi/livenessprobe:v1.1.0] 14967303} {[mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume@sha256:23d8c6033f02a1ecad05127ebdc931bb871264228661bc122704b0974e4d9fdd mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume:1.0.8] 1159025} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 21:05:46.976] Nov 5 21:05:46.973: INFO: I1105 21:05:46.976] Logging kubelet events for node k8s-agentpool-97435445-vmss000001 I1105 21:05:47.056] Nov 5 21:05:47.056: INFO: I1105 21:05:47.056] Logging pods the kubelet thinks is on node k8s-agentpool-97435445-vmss000001 I1105 21:05:47.141] Nov 5 21:05:47.140: INFO: blobfuse-flexvol-installer-j4s7t started at 2019-11-05 20:07:59 +0000 UTC (0+1 container statuses recorded) I1105 21:05:47.141] Nov 5 21:05:47.140: INFO: Container blobfuse-flexvol-installer ready: true, restart count 0 I1105 21:05:47.141] Nov 5 21:05:47.140: INFO: azure-ip-masq-agent-gdhcq started at 2019-11-05 20:07:59 +0000 UTC (0+1 container statuses recorded) I1105 21:05:47.141] Nov 5 21:05:47.140: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 21:05:47.141] Nov 5 21:05:47.140: INFO: keyvault-flexvolume-6bffx started at 2019-11-05 20:07:59 +0000 UTC (0+1 container statuses recorded) I1105 21:05:47.142] Nov 5 21:05:47.140: INFO: Container keyvault-flexvolume ready: true, restart count 0 I1105 21:05:47.142] Nov 5 21:05:47.140: INFO: kube-proxy-kjrpb started at 2019-11-05 20:08:00 +0000 UTC (0+1 container statuses recorded) I1105 21:05:47.142] Nov 5 21:05:47.140: INFO: Container kube-proxy ready: true, restart count 0 I1105 21:05:47.142] Nov 5 21:05:47.140: INFO: tiller-deploy-54c96cb5df-pt7x6 started at 2019-11-05 20:08:00 +0000 UTC (0+1 container statuses recorded) I1105 21:05:47.142] Nov 5 21:05:47.140: INFO: Container tiller ready: true, restart count 0 I1105 21:05:47.142] Nov 5 21:05:47.140: INFO: csi-azuredisk-node-q2dmt started at 2019-11-05 20:16:09 +0000 UTC (0+3 container statuses recorded) I1105 21:05:47.142] Nov 5 21:05:47.140: INFO: Container azuredisk ready: true, restart count 0 I1105 21:05:47.143] Nov 5 21:05:47.140: INFO: Container liveness-probe ready: true, restart count 0 I1105 21:05:47.143] Nov 5 21:05:47.140: INFO: Container node-driver-registrar ready: true, restart count 0 I1105 21:05:47.143] Nov 5 21:05:47.140: INFO: csi-azuredisk-controller-6f6587bfc6-62nrt started at 2019-11-05 20:16:09 +0000 UTC (0+6 container statuses recorded) I1105 21:05:47.143] Nov 5 21:05:47.140: INFO: Container azuredisk ready: true, restart count 0 I1105 21:05:47.143] Nov 5 21:05:47.140: INFO: Container cluster-driver-registrar ready: true, restart count 0 I1105 21:05:47.143] Nov 5 21:05:47.140: INFO: Container csi-attacher ready: true, restart count 0 I1105 21:05:47.144] Nov 5 21:05:47.140: INFO: Container csi-provisioner ready: true, restart count 0 I1105 21:05:47.144] Nov 5 21:05:47.140: INFO: Container csi-snapshotter ready: true, restart count 0 I1105 21:05:47.144] Nov 5 21:05:47.140: INFO: Container liveness-probe ready: true, restart count 0 I1105 21:05:47.393] Nov 5 21:05:47.393: INFO: I1105 21:05:47.393] Latency metrics for node k8s-agentpool-97435445-vmss000001 I1105 21:05:47.393] Nov 5 21:05:47.393: INFO: I1105 21:05:47.393] Logging node info for node k8s-master-97435445-0 I1105 21:05:47.479] Nov 5 21:05:47.476: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-97435445-0,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-97435445-0,UID:df3ae36f-961c-4729-a8f5-34cda54f4ff9,ResourceVersion:11262,Generation:0,CreationTimestamp:2019-11-05 20:08:36 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 2,kubernetes.azure.com/cluster: kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-97435445-0,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.4.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b/providers/Microsoft.Compute/virtualMachines/k8s-master-97435445-0,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 20:08:46 +0000 UTC 2019-11-05 20:08:46 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 21:05:04 +0000 UTC 2019-11-05 20:08:28 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 21:05:04 +0000 UTC 2019-11-05 20:08:28 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 21:05:04 +0000 UTC 2019-11-05 20:08:28 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 21:05:04 +0000 UTC 2019-11-05 20:08:36 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.255.5} {Hostname k8s-master-97435445-0}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:3c381865192341c2868672e8626cef7d,SystemUUID:F3D259D3-7EB8-B844-A1F0-F954DCD0D76B,BootID:4948329e-cf57-48c3-aa83-5297bef70370,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 21:05:47.479] Nov 5 21:05:47.476: INFO: I1105 21:05:47.479] Logging kubelet events for node k8s-master-97435445-0 I1105 21:05:47.560] Nov 5 21:05:47.560: INFO: I1105 21:05:47.560] Logging pods the kubelet thinks is on node k8s-master-97435445-0 I1105 21:05:47.645] Nov 5 21:05:47.645: INFO: kube-addon-manager-k8s-master-97435445-0 started at 2019-11-05 20:08:48 +0000 UTC (0+1 container statuses recorded) I1105 21:05:47.645] Nov 5 21:05:47.645: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 21:05:47.645] Nov 5 21:05:47.645: INFO: kube-apiserver-k8s-master-97435445-0 started at 2019-11-05 20:08:22 +0000 UTC (0+1 container statuses recorded) I1105 21:05:47.645] Nov 5 21:05:47.645: INFO: Container kube-apiserver ready: true, restart count 0 I1105 21:05:47.646] Nov 5 21:05:47.645: INFO: kube-controller-manager-k8s-master-97435445-0 started at 2019-11-05 20:08:22 +0000 UTC (0+1 container statuses recorded) I1105 21:05:47.646] Nov 5 21:05:47.645: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 21:05:47.646] Nov 5 21:05:47.645: INFO: kube-scheduler-k8s-master-97435445-0 started at 2019-11-05 20:08:22 +0000 UTC (0+1 container statuses recorded) I1105 21:05:47.646] Nov 5 21:05:47.645: INFO: Container kube-scheduler ready: true, restart count 0 I1105 21:05:47.646] Nov 5 21:05:47.645: INFO: azure-ip-masq-agent-q6dcl started at 2019-11-05 20:08:38 +0000 UTC (0+1 container statuses recorded) I1105 21:05:47.646] Nov 5 21:05:47.645: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 21:05:47.646] Nov 5 21:05:47.645: INFO: kube-proxy-5wnd2 started at 2019-11-05 20:08:38 +0000 UTC (0+1 container statuses recorded) I1105 21:05:47.647] Nov 5 21:05:47.645: INFO: Container kube-proxy ready: true, restart count 0 I1105 21:05:47.647] Nov 5 21:05:47.645: INFO: cloud-controller-manager-k8s-master-97435445-0 started at 2019-11-05 20:08:22 +0000 UTC (0+1 container statuses recorded) I1105 21:05:47.647] Nov 5 21:05:47.645: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 21:05:47.908] Nov 5 21:05:47.908: INFO: I1105 21:05:47.909] Latency metrics for node k8s-master-97435445-0 I1105 21:05:47.909] Nov 5 21:05:47.908: INFO: I1105 21:05:47.909] Logging node info for node k8s-master-97435445-1 I1105 21:05:47.989] Nov 5 21:05:47.986: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-97435445-1,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-97435445-1,UID:05534a67-d493-4ab5-9b0f-5ff5490e234f,ResourceVersion:11342,Generation:0,CreationTimestamp:2019-11-05 20:07:34 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 0,kubernetes.azure.com/cluster: kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-97435445-1,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.1.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b/providers/Microsoft.Compute/virtualMachines/k8s-master-97435445-1,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 20:08:26 +0000 UTC 2019-11-05 20:08:26 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 21:05:32 +0000 UTC 2019-11-05 20:07:28 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 21:05:32 +0000 UTC 2019-11-05 20:07:28 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 21:05:32 +0000 UTC 2019-11-05 20:07:28 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 21:05:32 +0000 UTC 2019-11-05 20:07:55 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{Hostname k8s-master-97435445-1} {InternalIP 10.240.255.6}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:079b605103b74acabac18fd2ea63368a,SystemUUID:38220EEC-BEFF-4C46-AA05-5A214AAFDC28,BootID:b1b45df0-d925-4bcc-9830-0202df36ebb4,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 21:05:47.989] Nov 5 21:05:47.987: INFO: I1105 21:05:47.989] Logging kubelet events for node k8s-master-97435445-1 I1105 21:05:48.070] Nov 5 21:05:48.070: INFO: I1105 21:05:48.070] Logging pods the kubelet thinks is on node k8s-master-97435445-1 I1105 21:05:48.155] Nov 5 21:05:48.154: INFO: azure-ip-masq-agent-n29bm started at 2019-11-05 20:07:59 +0000 UTC (0+1 container statuses recorded) I1105 21:05:48.155] Nov 5 21:05:48.154: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 21:05:48.155] Nov 5 21:05:48.154: INFO: kube-proxy-h4s2c started at 2019-11-05 20:08:00 +0000 UTC (0+1 container statuses recorded) I1105 21:05:48.155] Nov 5 21:05:48.154: INFO: Container kube-proxy ready: true, restart count 0 I1105 21:05:48.155] Nov 5 21:05:48.154: INFO: cloud-controller-manager-k8s-master-97435445-1 started at 2019-11-05 20:07:22 +0000 UTC (0+1 container statuses recorded) I1105 21:05:48.156] Nov 5 21:05:48.154: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 21:05:48.156] Nov 5 21:05:48.154: INFO: kube-addon-manager-k8s-master-97435445-1 started at 2019-11-05 20:07:22 +0000 UTC (0+1 container statuses recorded) I1105 21:05:48.156] Nov 5 21:05:48.154: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 21:05:48.156] Nov 5 21:05:48.154: INFO: kube-apiserver-k8s-master-97435445-1 started at 2019-11-05 20:07:22 +0000 UTC (0+1 container statuses recorded) I1105 21:05:48.156] Nov 5 21:05:48.154: INFO: Container kube-apiserver ready: true, restart count 0 I1105 21:05:48.156] Nov 5 21:05:48.154: INFO: kube-controller-manager-k8s-master-97435445-1 started at 2019-11-05 20:07:22 +0000 UTC (0+1 container statuses recorded) I1105 21:05:48.157] Nov 5 21:05:48.154: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 21:05:48.157] Nov 5 21:05:48.154: INFO: kube-scheduler-k8s-master-97435445-1 started at 2019-11-05 20:07:22 +0000 UTC (0+1 container statuses recorded) I1105 21:05:48.157] Nov 5 21:05:48.154: INFO: Container kube-scheduler ready: true, restart count 0 I1105 21:05:48.413] Nov 5 21:05:48.413: INFO: I1105 21:05:48.413] Latency metrics for node k8s-master-97435445-1 I1105 21:05:48.413] Nov 5 21:05:48.413: INFO: I1105 21:05:48.413] Logging node info for node k8s-master-97435445-2 I1105 21:05:48.494] Nov 5 21:05:48.491: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-97435445-2,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-97435445-2,UID:fa55c895-1a93-4743-885e-03fce84b5332,ResourceVersion:11216,Generation:0,CreationTimestamp:2019-11-05 20:09:19 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 0,kubernetes.azure.com/cluster: kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-97435445-2,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.5.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b/providers/Microsoft.Compute/virtualMachines/k8s-master-97435445-2,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 20:09:19 +0000 UTC 2019-11-05 20:09:19 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 21:04:49 +0000 UTC 2019-11-05 20:09:12 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 21:04:49 +0000 UTC 2019-11-05 20:09:12 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 21:04:49 +0000 UTC 2019-11-05 20:09:12 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 21:04:49 +0000 UTC 2019-11-05 20:09:19 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.255.7} {Hostname k8s-master-97435445-2}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:d19f6709138f496da43c095cec7b2360,SystemUUID:10116777-A8D7-CB4A-A8B7-7EBCF2F23B62,BootID:50cd14a9-5497-4972-bf60-cf59a0cbcb5f,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 21:05:48.494] Nov 5 21:05:48.491: INFO: I1105 21:05:48.494] Logging kubelet events for node k8s-master-97435445-2 I1105 21:05:48.575] Nov 5 21:05:48.575: INFO: I1105 21:05:48.575] Logging pods the kubelet thinks is on node k8s-master-97435445-2 I1105 21:05:48.658] Nov 5 21:05:48.658: INFO: kube-scheduler-k8s-master-97435445-2 started at 2019-11-05 20:09:05 +0000 UTC (0+1 container statuses recorded) I1105 21:05:48.658] Nov 5 21:05:48.658: INFO: Container kube-scheduler ready: true, restart count 0 I1105 21:05:48.659] Nov 5 21:05:48.658: INFO: cloud-controller-manager-k8s-master-97435445-2 started at 2019-11-05 20:09:05 +0000 UTC (0+1 container statuses recorded) I1105 21:05:48.659] Nov 5 21:05:48.658: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 21:05:48.659] Nov 5 21:05:48.658: INFO: kube-addon-manager-k8s-master-97435445-2 started at 2019-11-05 20:09:05 +0000 UTC (0+1 container statuses recorded) I1105 21:05:48.659] Nov 5 21:05:48.658: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 21:05:48.659] Nov 5 21:05:48.658: INFO: azure-ip-masq-agent-sq7p5 started at 2019-11-05 20:09:19 +0000 UTC (0+1 container statuses recorded) I1105 21:05:48.659] Nov 5 21:05:48.658: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 21:05:48.660] Nov 5 21:05:48.658: INFO: kube-proxy-pz426 started at 2019-11-05 20:09:19 +0000 UTC (0+1 container statuses recorded) I1105 21:05:48.660] Nov 5 21:05:48.658: INFO: Container kube-proxy ready: true, restart count 0 I1105 21:05:48.660] Nov 5 21:05:48.658: INFO: kube-apiserver-k8s-master-97435445-2 started at 2019-11-05 20:09:05 +0000 UTC (0+1 container statuses recorded) I1105 21:05:48.660] Nov 5 21:05:48.658: INFO: Container kube-apiserver ready: true, restart count 0 I1105 21:05:48.660] Nov 5 21:05:48.658: INFO: kube-controller-manager-k8s-master-97435445-2 started at 2019-11-05 20:09:05 +0000 UTC (0+1 container statuses recorded) I1105 21:05:48.660] Nov 5 21:05:48.658: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 21:05:48.916] Nov 5 21:05:48.916: INFO: I1105 21:05:48.916] Latency metrics for node k8s-master-97435445-2 I1105 21:05:48.917] Nov 5 21:05:48.916: INFO: I1105 21:05:48.917] Logging node info for node k8s-master-97435445-3 I1105 21:05:48.998] Nov 5 21:05:48.995: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-97435445-3,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-97435445-3,UID:8ffd1d0d-4487-4680-a2f6-e724387be637,ResourceVersion:11217,Generation:0,CreationTimestamp:2019-11-05 20:09:21 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 1,kubernetes.azure.com/cluster: kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-97435445-3,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.6.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b/providers/Microsoft.Compute/virtualMachines/k8s-master-97435445-3,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 20:09:39 +0000 UTC 2019-11-05 20:09:39 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 21:04:49 +0000 UTC 2019-11-05 20:09:18 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 21:04:49 +0000 UTC 2019-11-05 20:09:18 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 21:04:49 +0000 UTC 2019-11-05 20:09:18 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 21:04:49 +0000 UTC 2019-11-05 20:09:21 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.255.8} {Hostname k8s-master-97435445-3}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:266064173bfb4297898c6c25769e8b20,SystemUUID:E6E44241-5015-7740-9A81-A668A8427206,BootID:579d658c-b0ad-47f1-a993-1ccea2b55789,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 21:05:48.998] Nov 5 21:05:48.995: INFO: I1105 21:05:48.998] Logging kubelet events for node k8s-master-97435445-3 I1105 21:05:49.079] Nov 5 21:05:49.079: INFO: I1105 21:05:49.079] Logging pods the kubelet thinks is on node k8s-master-97435445-3 I1105 21:05:49.163] Nov 5 21:05:49.162: INFO: kube-scheduler-k8s-master-97435445-3 started at 2019-11-05 20:09:05 +0000 UTC (0+1 container statuses recorded) I1105 21:05:49.163] Nov 5 21:05:49.162: INFO: Container kube-scheduler ready: true, restart count 0 I1105 21:05:49.163] Nov 5 21:05:49.162: INFO: azure-ip-masq-agent-747cl started at 2019-11-05 20:09:21 +0000 UTC (0+1 container statuses recorded) I1105 21:05:49.163] Nov 5 21:05:49.162: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 21:05:49.163] Nov 5 21:05:49.162: INFO: kube-proxy-k45g8 started at 2019-11-05 20:09:21 +0000 UTC (0+1 container statuses recorded) I1105 21:05:49.164] Nov 5 21:05:49.162: INFO: Container kube-proxy ready: true, restart count 0 I1105 21:05:49.164] Nov 5 21:05:49.162: INFO: cloud-controller-manager-k8s-master-97435445-3 started at 2019-11-05 20:09:04 +0000 UTC (0+1 container statuses recorded) I1105 21:05:49.164] Nov 5 21:05:49.162: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 21:05:49.164] Nov 5 21:05:49.162: INFO: kube-addon-manager-k8s-master-97435445-3 started at 2019-11-05 20:09:05 +0000 UTC (0+1 container statuses recorded) I1105 21:05:49.164] Nov 5 21:05:49.162: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 21:05:49.164] Nov 5 21:05:49.162: INFO: kube-apiserver-k8s-master-97435445-3 started at 2019-11-05 20:09:05 +0000 UTC (0+1 container statuses recorded) I1105 21:05:49.165] Nov 5 21:05:49.162: INFO: Container kube-apiserver ready: true, restart count 0 I1105 21:05:49.165] Nov 5 21:05:49.162: INFO: kube-controller-manager-k8s-master-97435445-3 started at 2019-11-05 20:09:05 +0000 UTC (0+1 container statuses recorded) I1105 21:05:49.165] Nov 5 21:05:49.162: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 21:05:49.418] Nov 5 21:05:49.418: INFO: I1105 21:05:49.419] Latency metrics for node k8s-master-97435445-3 I1105 21:05:49.419] Nov 5 21:05:49.418: INFO: I1105 21:05:49.419] Logging node info for node k8s-master-97435445-4 I1105 21:05:49.500] Nov 5 21:05:49.497: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-97435445-4,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-97435445-4,UID:6d547b41-3f92-4cc4-a2d7-dda2071776a2,ResourceVersion:11261,Generation:0,CreationTimestamp:2019-11-05 20:08:36 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 1,kubernetes.azure.com/cluster: kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-97435445-4,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.3.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b/providers/Microsoft.Compute/virtualMachines/k8s-master-97435445-4,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 20:08:59 +0000 UTC 2019-11-05 20:08:59 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 21:05:03 +0000 UTC 2019-11-05 20:08:29 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 21:05:03 +0000 UTC 2019-11-05 20:08:29 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 21:05:03 +0000 UTC 2019-11-05 20:08:29 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 21:05:03 +0000 UTC 2019-11-05 20:08:36 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.255.9} {Hostname k8s-master-97435445-4}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:56b393c61eb34ba49ebda5279eeedb30,SystemUUID:FD239C23-618C-0049-8734-9AF3FDC9E3C1,BootID:f342ea44-e719-4513-807d-5def7ac606e2,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 21:05:49.500] Nov 5 21:05:49.497: INFO: I1105 21:05:49.500] Logging kubelet events for node k8s-master-97435445-4 I1105 21:05:49.581] Nov 5 21:05:49.581: INFO: I1105 21:05:49.581] Logging pods the kubelet thinks is on node k8s-master-97435445-4 I1105 21:05:49.676] Nov 5 21:05:49.675: INFO: kube-addon-manager-k8s-master-97435445-4 started at 2019-11-05 20:08:23 +0000 UTC (0+1 container statuses recorded) I1105 21:05:49.676] Nov 5 21:05:49.675: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 21:05:49.676] Nov 5 21:05:49.675: INFO: kube-apiserver-k8s-master-97435445-4 started at 2019-11-05 20:08:23 +0000 UTC (0+1 container statuses recorded) I1105 21:05:49.676] Nov 5 21:05:49.675: INFO: Container kube-apiserver ready: true, restart count 0 I1105 21:05:49.676] Nov 5 21:05:49.675: INFO: kube-controller-manager-k8s-master-97435445-4 started at 2019-11-05 20:08:23 +0000 UTC (0+1 container statuses recorded) I1105 21:05:49.677] Nov 5 21:05:49.675: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 21:05:49.677] Nov 5 21:05:49.675: INFO: kube-scheduler-k8s-master-97435445-4 started at 2019-11-05 20:08:23 +0000 UTC (0+1 container statuses recorded) I1105 21:05:49.677] Nov 5 21:05:49.675: INFO: Container kube-scheduler ready: true, restart count 0 I1105 21:05:49.677] Nov 5 21:05:49.675: INFO: kube-proxy-vfbfr started at 2019-11-05 20:08:36 +0000 UTC (0+1 container statuses recorded) I1105 21:05:49.677] Nov 5 21:05:49.675: INFO: Container kube-proxy ready: true, restart count 0 I1105 21:05:49.678] Nov 5 21:05:49.675: INFO: azure-ip-masq-agent-kmkll started at 2019-11-05 20:08:36 +0000 UTC (0+1 container statuses recorded) I1105 21:05:49.678] Nov 5 21:05:49.675: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 21:05:49.678] Nov 5 21:05:49.675: INFO: cloud-controller-manager-k8s-master-97435445-4 started at 2019-11-05 20:08:46 +0000 UTC (0+1 container statuses recorded) I1105 21:05:49.678] Nov 5 21:05:49.675: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 21:05:49.933] Nov 5 21:05:49.933: INFO: I1105 21:05:49.934] Latency metrics for node k8s-master-97435445-4 I1105 21:05:49.934] Nov 5 21:05:49.933: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready I1105 21:05:50.015] STEP: Destroying namespace "azuredisk-6633" for this suite. I1105 21:06:04.289] Nov 5 21:06:04.289: INFO: Waiting up to 30s for server preferred namespaced resources to be successfully discovered I1105 21:06:07.354] Nov 5 21:06:07.353: INFO: namespace azuredisk-6633 deletion completed in 17.338828215s I1105 21:06:07.354] I1105 21:06:07.354] • Failure [323.781 seconds] I1105 21:06:07.354] [azuredisk-csi-e2e] Dynamic Provisioning I1105 21:06:07.354] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:33 I1105 21:06:07.354] [single-az] I1105 21:06:07.354] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:36 I1105 21:06:07.355] should create a volume on demand and mount it as readOnly in a pod [It] I1105 21:06:07.355] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:126 I1105 21:06:07.355] I1105 21:06:07.355] Unexpected error: I1105 21:06:07.355] <*errors.errorString | 0xc0008c8b20>: { I1105 21:06:07.355] s: "PersistentVolumeClaims [pvc-zkl6h] not all in phase Bound within 5m0s", I1105 21:06:07.355] } I1105 21:06:07.355] PersistentVolumeClaims [pvc-zkl6h] not all in phase Bound within 5m0s I1105 21:06:07.356] occurred I1105 21:06:07.356] I1105 21:06:07.356] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/testsuites/testsuites.go:278 I1105 21:06:07.356] ------------------------------ I1105 21:06:07.356] [azuredisk-csi-e2e] Dynamic Provisioning [single-az] I1105 21:06:07.356] [env] should retain PV with reclaimPolicy "Retain" I1105 21:06:07.356] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:240 I1105 21:06:07.356] [BeforeEach] [single-az] I1105 21:06:07.356] /go/src/sigs.k8s.io/azuredisk-csi-driver/vendor/k8s.io/kubernetes/test/e2e/framework/framework.go:150 I1105 21:06:07.357] STEP: Creating a kubernetes client I1105 21:06:07.357] Nov 5 21:06:07.353: INFO: >>> kubeConfig: /workspace/aks971561099/kubeconfig/kubeconfig.eastus2.json I1105 21:06:07.357] STEP: Building a namespace api object, basename azuredisk I1105 21:06:07.598] STEP: Binding the e2e-test-privileged-psp PodSecurityPolicy to the default service account in azuredisk-9582 I1105 21:06:07.871] STEP: Waiting for a default service account to be provisioned in namespace I1105 21:06:07.948] [BeforeEach] [single-az] I1105 21:06:07.948] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:58 I1105 21:06:07.948] [It] [env] should retain PV with reclaimPolicy "Retain" I1105 21:06:07.948] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:240 I1105 21:06:07.949] STEP: setting up the StorageClass I1105 21:06:07.949] STEP: creating a StorageClass I1105 21:06:08.034] STEP: setting up the PVC and PV I1105 21:06:08.034] STEP: creating a PVC I1105 21:06:08.119] STEP: waiting for PVC to be in phase "Bound" I1105 21:06:08.119] Nov 5 21:06:08.119: INFO: Waiting up to 5m0s for PersistentVolumeClaims [pvc-m4z2v] to have phase Bound I1105 21:06:08.198] Nov 5 21:06:08.198: INFO: PersistentVolumeClaim pvc-m4z2v found but phase is Pending instead of Bound. I1105 21:06:10.278] Nov 5 21:06:10.277: INFO: PersistentVolumeClaim pvc-m4z2v found but phase is Pending instead of Bound. I1105 21:06:12.357] Nov 5 21:06:12.356: INFO: PersistentVolumeClaim pvc-m4z2v found but phase is Pending instead of Bound. I1105 21:06:14.436] Nov 5 21:06:14.436: INFO: PersistentVolumeClaim pvc-m4z2v found but phase is Pending instead of Bound. I1105 21:06:16.531] Nov 5 21:06:16.531: INFO: PersistentVolumeClaim pvc-m4z2v found but phase is Pending instead of Bound. I1105 21:06:18.637] Nov 5 21:06:18.637: INFO: PersistentVolumeClaim pvc-m4z2v found but phase is Pending instead of Bound. I1105 21:06:20.721] Nov 5 21:06:20.720: INFO: PersistentVolumeClaim pvc-m4z2v found but phase is Pending instead of Bound. I1105 21:06:22.802] Nov 5 21:06:22.802: INFO: PersistentVolumeClaim pvc-m4z2v found but phase is Pending instead of Bound. I1105 21:06:24.881] Nov 5 21:06:24.881: INFO: PersistentVolumeClaim pvc-m4z2v found but phase is Pending instead of Bound. I1105 21:06:26.961] Nov 5 21:06:26.960: INFO: PersistentVolumeClaim pvc-m4z2v found but phase is Pending instead of Bound. I1105 21:06:29.040] Nov 5 21:06:29.039: INFO: PersistentVolumeClaim pvc-m4z2v found but phase is Pending instead of Bound. I1105 21:06:31.119] Nov 5 21:06:31.118: INFO: PersistentVolumeClaim pvc-m4z2v found but phase is Pending instead of Bound. I1105 21:06:33.198] Nov 5 21:06:33.197: INFO: PersistentVolumeClaim pvc-m4z2v found but phase is Pending instead of Bound. I1105 21:06:35.276] Nov 5 21:06:35.276: INFO: PersistentVolumeClaim pvc-m4z2v found but phase is Pending instead of Bound. I1105 21:06:37.356] Nov 5 21:06:37.356: INFO: PersistentVolumeClaim pvc-m4z2v found but phase is Pending instead of Bound. I1105 21:06:39.436] Nov 5 21:06:39.436: INFO: PersistentVolumeClaim pvc-m4z2v found but phase is Pending instead of Bound. I1105 21:06:41.514] Nov 5 21:06:41.514: INFO: PersistentVolumeClaim pvc-m4z2v found but phase is Pending instead of Bound. I1105 21:06:43.594] Nov 5 21:06:43.593: INFO: PersistentVolumeClaim pvc-m4z2v found but phase is Pending instead of Bound. I1105 21:06:45.672] Nov 5 21:06:45.672: INFO: PersistentVolumeClaim pvc-m4z2v found but phase is Pending instead of Bound. I1105 21:06:47.752] Nov 5 21:06:47.751: INFO: PersistentVolumeClaim pvc-m4z2v found but phase is Pending instead of Bound. I1105 21:06:49.831] Nov 5 21:06:49.830: INFO: PersistentVolumeClaim pvc-m4z2v found but phase is Pending instead of Bound. I1105 21:06:51.910] Nov 5 21:06:51.910: INFO: PersistentVolumeClaim pvc-m4z2v found but phase is Pending instead of Bound. I1105 21:06:53.989] Nov 5 21:06:53.989: INFO: PersistentVolumeClaim pvc-m4z2v found but phase is Pending instead of Bound. I1105 21:06:56.069] Nov 5 21:06:56.068: INFO: PersistentVolumeClaim pvc-m4z2v found but phase is Pending instead of Bound. I1105 21:06:58.153] Nov 5 21:06:58.152: INFO: PersistentVolumeClaim pvc-m4z2v found but phase is Pending instead of Bound. I1105 21:07:00.232] Nov 5 21:07:00.232: INFO: PersistentVolumeClaim pvc-m4z2v found but phase is Pending instead of Bound. I1105 21:07:02.312] Nov 5 21:07:02.312: INFO: PersistentVolumeClaim pvc-m4z2v found but phase is Pending instead of Bound. I1105 21:07:04.402] Nov 5 21:07:04.402: INFO: PersistentVolumeClaim pvc-m4z2v found but phase is Pending instead of Bound. I1105 21:07:06.481] Nov 5 21:07:06.481: INFO: PersistentVolumeClaim pvc-m4z2v found but phase is Pending instead of Bound. I1105 21:07:08.561] Nov 5 21:07:08.561: INFO: PersistentVolumeClaim pvc-m4z2v found but phase is Pending instead of Bound. I1105 21:07:10.639] Nov 5 21:07:10.639: INFO: PersistentVolumeClaim pvc-m4z2v found but phase is Pending instead of Bound. I1105 21:07:12.724] Nov 5 21:07:12.724: INFO: PersistentVolumeClaim pvc-m4z2v found but phase is Pending instead of Bound. I1105 21:07:14.803] Nov 5 21:07:14.803: INFO: PersistentVolumeClaim pvc-m4z2v found but phase is Pending instead of Bound. I1105 21:07:16.882] Nov 5 21:07:16.882: INFO: PersistentVolumeClaim pvc-m4z2v found but phase is Pending instead of Bound. I1105 21:07:18.961] Nov 5 21:07:18.961: INFO: PersistentVolumeClaim pvc-m4z2v found but phase is Pending instead of Bound. I1105 21:07:21.040] Nov 5 21:07:21.040: INFO: PersistentVolumeClaim pvc-m4z2v found but phase is Pending instead of Bound. I1105 21:07:23.120] Nov 5 21:07:23.120: INFO: PersistentVolumeClaim pvc-m4z2v found but phase is Pending instead of Bound. I1105 21:07:25.199] Nov 5 21:07:25.199: INFO: PersistentVolumeClaim pvc-m4z2v found but phase is Pending instead of Bound. I1105 21:07:27.279] Nov 5 21:07:27.279: INFO: PersistentVolumeClaim pvc-m4z2v found but phase is Pending instead of Bound. I1105 21:07:29.358] Nov 5 21:07:29.358: INFO: PersistentVolumeClaim pvc-m4z2v found but phase is Pending instead of Bound. I1105 21:07:31.437] Nov 5 21:07:31.437: INFO: PersistentVolumeClaim pvc-m4z2v found but phase is Pending instead of Bound. I1105 21:07:33.517] Nov 5 21:07:33.517: INFO: PersistentVolumeClaim pvc-m4z2v found but phase is Pending instead of Bound. I1105 21:07:35.597] Nov 5 21:07:35.596: INFO: PersistentVolumeClaim pvc-m4z2v found but phase is Pending instead of Bound. I1105 21:07:37.677] Nov 5 21:07:37.676: INFO: PersistentVolumeClaim pvc-m4z2v found but phase is Pending instead of Bound. I1105 21:07:39.756] Nov 5 21:07:39.755: INFO: PersistentVolumeClaim pvc-m4z2v found but phase is Pending instead of Bound. I1105 21:07:41.835] Nov 5 21:07:41.834: INFO: PersistentVolumeClaim pvc-m4z2v found but phase is Pending instead of Bound. I1105 21:07:43.914] Nov 5 21:07:43.914: INFO: PersistentVolumeClaim pvc-m4z2v found but phase is Pending instead of Bound. I1105 21:07:45.994] Nov 5 21:07:45.994: INFO: PersistentVolumeClaim pvc-m4z2v found but phase is Pending instead of Bound. I1105 21:07:48.073] Nov 5 21:07:48.073: INFO: PersistentVolumeClaim pvc-m4z2v found but phase is Pending instead of Bound. I1105 21:07:50.152] Nov 5 21:07:50.152: INFO: PersistentVolumeClaim pvc-m4z2v found but phase is Pending instead of Bound. I1105 21:07:52.231] Nov 5 21:07:52.230: INFO: PersistentVolumeClaim pvc-m4z2v found but phase is Pending instead of Bound. I1105 21:07:54.310] Nov 5 21:07:54.310: INFO: PersistentVolumeClaim pvc-m4z2v found but phase is Pending instead of Bound. I1105 21:07:56.389] Nov 5 21:07:56.389: INFO: PersistentVolumeClaim pvc-m4z2v found but phase is Pending instead of Bound. I1105 21:07:58.468] Nov 5 21:07:58.468: INFO: PersistentVolumeClaim pvc-m4z2v found but phase is Pending instead of Bound. I1105 21:08:00.547] Nov 5 21:08:00.547: INFO: PersistentVolumeClaim pvc-m4z2v found but phase is Pending instead of Bound. I1105 21:08:02.628] Nov 5 21:08:02.628: INFO: PersistentVolumeClaim pvc-m4z2v found but phase is Pending instead of Bound. I1105 21:08:04.717] Nov 5 21:08:04.717: INFO: PersistentVolumeClaim pvc-m4z2v found but phase is Pending instead of Bound. I1105 21:08:06.797] Nov 5 21:08:06.796: INFO: PersistentVolumeClaim pvc-m4z2v found but phase is Pending instead of Bound. I1105 21:08:08.876] Nov 5 21:08:08.876: INFO: PersistentVolumeClaim pvc-m4z2v found but phase is Pending instead of Bound. I1105 21:08:10.956] Nov 5 21:08:10.955: INFO: PersistentVolumeClaim pvc-m4z2v found but phase is Pending instead of Bound. I1105 21:08:13.035] Nov 5 21:08:13.035: INFO: PersistentVolumeClaim pvc-m4z2v found but phase is Pending instead of Bound. I1105 21:08:15.114] Nov 5 21:08:15.114: INFO: PersistentVolumeClaim pvc-m4z2v found but phase is Pending instead of Bound. I1105 21:08:17.193] Nov 5 21:08:17.192: INFO: PersistentVolumeClaim pvc-m4z2v found but phase is Pending instead of Bound. I1105 21:08:19.272] Nov 5 21:08:19.272: INFO: PersistentVolumeClaim pvc-m4z2v found but phase is Pending instead of Bound. I1105 21:08:21.351] Nov 5 21:08:21.351: INFO: PersistentVolumeClaim pvc-m4z2v found but phase is Pending instead of Bound. I1105 21:08:23.430] Nov 5 21:08:23.430: INFO: PersistentVolumeClaim pvc-m4z2v found but phase is Pending instead of Bound. I1105 21:08:25.509] Nov 5 21:08:25.509: INFO: PersistentVolumeClaim pvc-m4z2v found but phase is Pending instead of Bound. I1105 21:08:27.588] Nov 5 21:08:27.588: INFO: PersistentVolumeClaim pvc-m4z2v found but phase is Pending instead of Bound. I1105 21:08:29.667] Nov 5 21:08:29.667: INFO: PersistentVolumeClaim pvc-m4z2v found but phase is Pending instead of Bound. I1105 21:08:31.747] Nov 5 21:08:31.747: INFO: PersistentVolumeClaim pvc-m4z2v found but phase is Pending instead of Bound. I1105 21:08:33.830] Nov 5 21:08:33.829: INFO: PersistentVolumeClaim pvc-m4z2v found but phase is Pending instead of Bound. I1105 21:08:35.909] Nov 5 21:08:35.909: INFO: PersistentVolumeClaim pvc-m4z2v found but phase is Pending instead of Bound. I1105 21:08:37.988] Nov 5 21:08:37.987: INFO: PersistentVolumeClaim pvc-m4z2v found but phase is Pending instead of Bound. I1105 21:08:40.067] Nov 5 21:08:40.067: INFO: PersistentVolumeClaim pvc-m4z2v found but phase is Pending instead of Bound. I1105 21:08:42.150] Nov 5 21:08:42.150: INFO: PersistentVolumeClaim pvc-m4z2v found but phase is Pending instead of Bound. I1105 21:08:44.237] Nov 5 21:08:44.237: INFO: PersistentVolumeClaim pvc-m4z2v found but phase is Pending instead of Bound. I1105 21:08:46.316] Nov 5 21:08:46.315: INFO: PersistentVolumeClaim pvc-m4z2v found but phase is Pending instead of Bound. I1105 21:08:48.396] Nov 5 21:08:48.395: INFO: PersistentVolumeClaim pvc-m4z2v found but phase is Pending instead of Bound. I1105 21:08:50.475] Nov 5 21:08:50.475: INFO: PersistentVolumeClaim pvc-m4z2v found but phase is Pending instead of Bound. I1105 21:08:52.554] Nov 5 21:08:52.554: INFO: PersistentVolumeClaim pvc-m4z2v found but phase is Pending instead of Bound. I1105 21:08:54.633] Nov 5 21:08:54.633: INFO: PersistentVolumeClaim pvc-m4z2v found but phase is Pending instead of Bound. I1105 21:08:56.713] Nov 5 21:08:56.713: INFO: PersistentVolumeClaim pvc-m4z2v found but phase is Pending instead of Bound. I1105 21:08:58.792] Nov 5 21:08:58.792: INFO: PersistentVolumeClaim pvc-m4z2v found but phase is Pending instead of Bound. I1105 21:09:00.874] Nov 5 21:09:00.874: INFO: PersistentVolumeClaim pvc-m4z2v found but phase is Pending instead of Bound. I1105 21:09:02.953] Nov 5 21:09:02.953: INFO: PersistentVolumeClaim pvc-m4z2v found but phase is Pending instead of Bound. I1105 21:09:05.034] Nov 5 21:09:05.034: INFO: PersistentVolumeClaim pvc-m4z2v found but phase is Pending instead of Bound. I1105 21:09:07.113] Nov 5 21:09:07.113: INFO: PersistentVolumeClaim pvc-m4z2v found but phase is Pending instead of Bound. I1105 21:09:09.193] Nov 5 21:09:09.193: INFO: PersistentVolumeClaim pvc-m4z2v found but phase is Pending instead of Bound. I1105 21:09:11.272] Nov 5 21:09:11.272: INFO: PersistentVolumeClaim pvc-m4z2v found but phase is Pending instead of Bound. I1105 21:09:13.352] Nov 5 21:09:13.352: INFO: PersistentVolumeClaim pvc-m4z2v found but phase is Pending instead of Bound. I1105 21:09:15.433] Nov 5 21:09:15.432: INFO: PersistentVolumeClaim pvc-m4z2v found but phase is Pending instead of Bound. I1105 21:09:17.512] Nov 5 21:09:17.512: INFO: PersistentVolumeClaim pvc-m4z2v found but phase is Pending instead of Bound. I1105 21:09:19.592] Nov 5 21:09:19.591: INFO: PersistentVolumeClaim pvc-m4z2v found but phase is Pending instead of Bound. I1105 21:09:21.671] Nov 5 21:09:21.670: INFO: PersistentVolumeClaim pvc-m4z2v found but phase is Pending instead of Bound. I1105 21:09:23.749] Nov 5 21:09:23.749: INFO: PersistentVolumeClaim pvc-m4z2v found but phase is Pending instead of Bound. I1105 21:09:25.829] Nov 5 21:09:25.828: INFO: PersistentVolumeClaim pvc-m4z2v found but phase is Pending instead of Bound. I1105 21:09:27.909] Nov 5 21:09:27.909: INFO: PersistentVolumeClaim pvc-m4z2v found but phase is Pending instead of Bound. I1105 21:09:29.988] Nov 5 21:09:29.988: INFO: PersistentVolumeClaim pvc-m4z2v found but phase is Pending instead of Bound. I1105 21:09:32.067] Nov 5 21:09:32.067: INFO: PersistentVolumeClaim pvc-m4z2v found but phase is Pending instead of Bound. I1105 21:09:34.146] Nov 5 21:09:34.146: INFO: PersistentVolumeClaim pvc-m4z2v found but phase is Pending instead of Bound. I1105 21:09:36.226] Nov 5 21:09:36.225: INFO: PersistentVolumeClaim pvc-m4z2v found but phase is Pending instead of Bound. I1105 21:09:38.305] Nov 5 21:09:38.304: INFO: PersistentVolumeClaim pvc-m4z2v found but phase is Pending instead of Bound. I1105 21:09:40.384] Nov 5 21:09:40.383: INFO: PersistentVolumeClaim pvc-m4z2v found but phase is Pending instead of Bound. I1105 21:09:42.463] Nov 5 21:09:42.463: INFO: PersistentVolumeClaim pvc-m4z2v found but phase is Pending instead of Bound. I1105 21:09:44.543] Nov 5 21:09:44.542: INFO: PersistentVolumeClaim pvc-m4z2v found but phase is Pending instead of Bound. I1105 21:09:46.629] Nov 5 21:09:46.629: INFO: PersistentVolumeClaim pvc-m4z2v found but phase is Pending instead of Bound. I1105 21:09:48.722] Nov 5 21:09:48.721: INFO: PersistentVolumeClaim pvc-m4z2v found but phase is Pending instead of Bound. I1105 21:09:50.801] Nov 5 21:09:50.801: INFO: PersistentVolumeClaim pvc-m4z2v found but phase is Pending instead of Bound. I1105 21:09:52.883] Nov 5 21:09:52.883: INFO: PersistentVolumeClaim pvc-m4z2v found but phase is Pending instead of Bound. I1105 21:09:54.962] Nov 5 21:09:54.962: INFO: PersistentVolumeClaim pvc-m4z2v found but phase is Pending instead of Bound. I1105 21:09:57.042] Nov 5 21:09:57.042: INFO: PersistentVolumeClaim pvc-m4z2v found but phase is Pending instead of Bound. I1105 21:09:59.121] Nov 5 21:09:59.121: INFO: PersistentVolumeClaim pvc-m4z2v found but phase is Pending instead of Bound. I1105 21:10:01.199] Nov 5 21:10:01.199: INFO: PersistentVolumeClaim pvc-m4z2v found but phase is Pending instead of Bound. I1105 21:10:03.278] Nov 5 21:10:03.278: INFO: PersistentVolumeClaim pvc-m4z2v found but phase is Pending instead of Bound. I1105 21:10:05.358] Nov 5 21:10:05.357: INFO: PersistentVolumeClaim pvc-m4z2v found but phase is Pending instead of Bound. I1105 21:10:07.436] Nov 5 21:10:07.436: INFO: PersistentVolumeClaim pvc-m4z2v found but phase is Pending instead of Bound. I1105 21:10:09.515] Nov 5 21:10:09.515: INFO: PersistentVolumeClaim pvc-m4z2v found but phase is Pending instead of Bound. I1105 21:10:11.594] Nov 5 21:10:11.594: INFO: PersistentVolumeClaim pvc-m4z2v found but phase is Pending instead of Bound. I1105 21:10:13.674] Nov 5 21:10:13.674: INFO: PersistentVolumeClaim pvc-m4z2v found but phase is Pending instead of Bound. I1105 21:10:15.753] Nov 5 21:10:15.753: INFO: PersistentVolumeClaim pvc-m4z2v found but phase is Pending instead of Bound. I1105 21:10:17.833] Nov 5 21:10:17.832: INFO: PersistentVolumeClaim pvc-m4z2v found but phase is Pending instead of Bound. I1105 21:10:19.912] Nov 5 21:10:19.912: INFO: PersistentVolumeClaim pvc-m4z2v found but phase is Pending instead of Bound. I1105 21:10:21.991] Nov 5 21:10:21.990: INFO: PersistentVolumeClaim pvc-m4z2v found but phase is Pending instead of Bound. I1105 21:10:24.070] Nov 5 21:10:24.070: INFO: PersistentVolumeClaim pvc-m4z2v found but phase is Pending instead of Bound. I1105 21:10:26.161] Nov 5 21:10:26.161: INFO: PersistentVolumeClaim pvc-m4z2v found but phase is Pending instead of Bound. I1105 21:10:28.240] Nov 5 21:10:28.240: INFO: PersistentVolumeClaim pvc-m4z2v found but phase is Pending instead of Bound. I1105 21:10:30.319] Nov 5 21:10:30.319: INFO: PersistentVolumeClaim pvc-m4z2v found but phase is Pending instead of Bound. I1105 21:10:32.404] Nov 5 21:10:32.404: INFO: PersistentVolumeClaim pvc-m4z2v found but phase is Pending instead of Bound. I1105 21:10:34.482] Nov 5 21:10:34.482: INFO: PersistentVolumeClaim pvc-m4z2v found but phase is Pending instead of Bound. I1105 21:10:36.561] Nov 5 21:10:36.561: INFO: PersistentVolumeClaim pvc-m4z2v found but phase is Pending instead of Bound. I1105 21:10:38.640] Nov 5 21:10:38.640: INFO: PersistentVolumeClaim pvc-m4z2v found but phase is Pending instead of Bound. I1105 21:10:40.721] Nov 5 21:10:40.721: INFO: PersistentVolumeClaim pvc-m4z2v found but phase is Pending instead of Bound. I1105 21:10:42.801] Nov 5 21:10:42.800: INFO: PersistentVolumeClaim pvc-m4z2v found but phase is Pending instead of Bound. I1105 21:10:44.880] Nov 5 21:10:44.880: INFO: PersistentVolumeClaim pvc-m4z2v found but phase is Pending instead of Bound. I1105 21:10:46.959] Nov 5 21:10:46.959: INFO: PersistentVolumeClaim pvc-m4z2v found but phase is Pending instead of Bound. I1105 21:10:49.039] Nov 5 21:10:49.038: INFO: PersistentVolumeClaim pvc-m4z2v found but phase is Pending instead of Bound. I1105 21:10:51.118] Nov 5 21:10:51.117: INFO: PersistentVolumeClaim pvc-m4z2v found but phase is Pending instead of Bound. I1105 21:10:53.196] Nov 5 21:10:53.196: INFO: PersistentVolumeClaim pvc-m4z2v found but phase is Pending instead of Bound. I1105 21:10:55.272] Nov 5 21:10:55.272: INFO: PersistentVolumeClaim pvc-m4z2v found but phase is Pending instead of Bound. I1105 21:10:57.349] Nov 5 21:10:57.349: INFO: PersistentVolumeClaim pvc-m4z2v found but phase is Pending instead of Bound. I1105 21:10:59.425] Nov 5 21:10:59.425: INFO: PersistentVolumeClaim pvc-m4z2v found but phase is Pending instead of Bound. I1105 21:11:01.502] Nov 5 21:11:01.502: INFO: PersistentVolumeClaim pvc-m4z2v found but phase is Pending instead of Bound. I1105 21:11:03.578] Nov 5 21:11:03.578: INFO: PersistentVolumeClaim pvc-m4z2v found but phase is Pending instead of Bound. I1105 21:11:05.654] Nov 5 21:11:05.654: INFO: PersistentVolumeClaim pvc-m4z2v found but phase is Pending instead of Bound. I1105 21:11:07.731] Nov 5 21:11:07.731: INFO: PersistentVolumeClaim pvc-m4z2v found but phase is Pending instead of Bound. I1105 21:11:09.731] Nov 5 21:11:09.731: INFO: Unexpected error occurred: PersistentVolumeClaims [pvc-m4z2v] not all in phase Bound within 5m0s I1105 21:11:09.732] [AfterEach] [single-az] I1105 21:11:09.732] /go/src/sigs.k8s.io/azuredisk-csi-driver/vendor/k8s.io/kubernetes/test/e2e/framework/framework.go:151 I1105 21:11:09.732] STEP: Collecting events from namespace "azuredisk-9582". I1105 21:11:09.809] STEP: Found 3 events. I1105 21:11:09.809] Nov 5 21:11:09.809: INFO: At 2019-11-05 21:06:08 +0000 UTC - event for pvc-m4z2v: {disk.csi.azure.com_csi-azuredisk-controller-6f6587bfc6-62nrt_3930bc69-0009-11ea-8cf5-829a27ff7d5a } Provisioning: External provisioner is provisioning volume for claim "azuredisk-9582/pvc-m4z2v" I1105 21:11:09.809] Nov 5 21:11:09.809: INFO: At 2019-11-05 21:06:08 +0000 UTC - event for pvc-m4z2v: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "disk.csi.azure.com" or manually created by system administrator I1105 21:11:09.810] Nov 5 21:11:09.809: INFO: At 2019-11-05 21:06:08 +0000 UTC - event for pvc-m4z2v: {disk.csi.azure.com_csi-azuredisk-controller-6f6587bfc6-62nrt_3930bc69-0009-11ea-8cf5-829a27ff7d5a } ProvisioningFailed: failed to provision volume with StorageClass "azuredisk-9582-disk.csi.azure.com-dynamic-sc-xrwd8": rpc error: code = Unknown desc = compute.DisksClient#CreateOrUpdate: Failure sending request: StatusCode=400 -- Original Error: Code="InvalidAvailabilityZone" Message="The zone(s) '' for resource 'Microsoft.Compute/disks/pvc-90f0ea65-b7d9-4beb-abb1-52e61bb3b021' is not supported. The supported zones for location 'eastus2' are '1,2,3'" I1105 21:11:09.885] Nov 5 21:11:09.885: INFO: POD NODE PHASE GRACE CONDITIONS I1105 21:11:09.886] Nov 5 21:11:09.885: INFO: I1105 21:11:10.112] Nov 5 21:11:10.112: INFO: I1105 21:11:10.112] Logging node info for node k8s-agentpool-97435445-vmss000000 I1105 21:11:10.212] Nov 5 21:11:10.209: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-agentpool-97435445-vmss000000,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-agentpool-97435445-vmss000000,UID:e955dc47-8c8f-4142-9241-3d409f32e42d,ResourceVersion:12284,Generation:0,CreationTimestamp:2019-11-05 20:07:34 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{agentpool: agentpool,beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_D4s_v3,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 0,kubernetes.azure.com/cluster: kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b,kubernetes.azure.com/role: agent,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-agentpool-97435445-vmss000000,kubernetes.io/os: linux,kubernetes.io/role: agent,node-role.kubernetes.io/agent: ,storageprofile: managed,storagetier: Premium_LRS,topology.disk.csi.azure.com/zone: 0,},Annotations:map[string]string{csi.volume.kubernetes.io/nodeid: {"disk.csi.azure.com":"k8s-agentpool-97435445-vmss000000"},node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.2.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool-97435445-vmss/virtualMachines/0,Unschedulable:false,Taints:[],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-csi-disk.csi.azure.com: {{8 0} {} 8 DecimalSI},cpu: {{4 0} {} 4 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{16797040640 0} {} BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-csi-disk.csi.azure.com: {{8 0} {} 8 DecimalSI},cpu: {{4 0} {} 4 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{16010608640 0} {} BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 20:08:01 +0000 UTC 2019-11-05 20:08:01 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 21:10:45 +0000 UTC 2019-11-05 20:07:27 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 21:10:45 +0000 UTC 2019-11-05 20:07:27 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 21:10:45 +0000 UTC 2019-11-05 20:07:27 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 21:10:45 +0000 UTC 2019-11-05 20:07:55 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.0.4} {Hostname k8s-agentpool-97435445-vmss000000}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:bfb55325809340228149d2d93f50df58,SystemUUID:CB2ED036-0439-F641-BC7F-640C3638C027,BootID:a917f4d1-549d-471d-b7dc-47ae1715d97d,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[k8sprowinternal.azurecr.io/azuredisk-csi@sha256:75d4ff5d7cd845d6e810c67796ecb037e852fb849ec1eb5b3194564008e6d1e0 k8sprowinternal.azurecr.io/azuredisk-csi:e2e-3e3ed045c6faa509dbd8d2890ee9f56aa2934887] 286930414} {[k8s.gcr.io/kubernetes-dashboard-amd64@sha256:0ae6b69432e78069c5ce2bcde0fe409c5c4d6f0f4d9cd50a17974fea38898747 k8s.gcr.io/kubernetes-dashboard-amd64:v1.10.1] 121711221} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/metrics-server-amd64@sha256:49a9f12f7067d11f42c803dbe61ed2c1299959ad85cb315b25ff7eef8e6b8892 k8s.gcr.io/metrics-server-amd64:v0.2.1] 42541759} {[k8s.gcr.io/coredns@sha256:e83beb5e43f8513fa735e77ffc5859640baea30a882a11cc75c4c3244a737d3c k8s.gcr.io/coredns:1.5.0] 42488424} {[mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume@sha256:4fd30d43947d4a54fc89ead7985beecfd3c9b2a93a0655a373b1608ab90bd5af mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume:v0.0.7] 22909487} {[quay.io/k8scsi/livenessprobe@sha256:8c9c8a0f330acc2e558ae6624309447d02522e76aa35af23fa19b88abf4381cb quay.io/k8scsi/livenessprobe:v1.0.2] 17415381} {[quay.io/k8scsi/csi-node-driver-registrar@sha256:13daf82fb99e951a4bff8ae5fc7c17c3a8fe7130be6400990d8f6076c32d4599 quay.io/k8scsi/csi-node-driver-registrar:v1.1.0] 15815995} {[mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume@sha256:23d8c6033f02a1ecad05127ebdc931bb871264228661bc122704b0974e4d9fdd mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume:1.0.8] 1159025} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 21:11:10.212] Nov 5 21:11:10.209: INFO: I1105 21:11:10.213] Logging kubelet events for node k8s-agentpool-97435445-vmss000000 I1105 21:11:10.287] Nov 5 21:11:10.287: INFO: I1105 21:11:10.287] Logging pods the kubelet thinks is on node k8s-agentpool-97435445-vmss000000 I1105 21:11:10.369] Nov 5 21:11:10.369: INFO: azure-ip-masq-agent-m7tv9 started at 2019-11-05 20:08:01 +0000 UTC (0+1 container statuses recorded) I1105 21:11:10.369] Nov 5 21:11:10.369: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 21:11:10.370] Nov 5 21:11:10.369: INFO: keyvault-flexvolume-smspt started at 2019-11-05 20:08:01 +0000 UTC (0+1 container statuses recorded) I1105 21:11:10.370] Nov 5 21:11:10.369: INFO: Container keyvault-flexvolume ready: true, restart count 0 I1105 21:11:10.370] Nov 5 21:11:10.369: INFO: metrics-server-864ffbc5c-wl84z started at 2019-11-05 20:08:01 +0000 UTC (0+1 container statuses recorded) I1105 21:11:10.370] Nov 5 21:11:10.369: INFO: Container metrics-server ready: true, restart count 0 I1105 21:11:10.370] Nov 5 21:11:10.369: INFO: kube-proxy-8mt7f started at 2019-11-05 20:08:01 +0000 UTC (0+1 container statuses recorded) I1105 21:11:10.370] Nov 5 21:11:10.369: INFO: Container kube-proxy ready: true, restart count 0 I1105 21:11:10.371] Nov 5 21:11:10.369: INFO: kubernetes-dashboard-66dd8b8df7-t6j4l started at 2019-11-05 20:08:01 +0000 UTC (0+1 container statuses recorded) I1105 21:11:10.371] Nov 5 21:11:10.369: INFO: Container kubernetes-dashboard ready: true, restart count 0 I1105 21:11:10.371] Nov 5 21:11:10.369: INFO: csi-azuredisk-node-5wwhz started at 2019-11-05 20:16:09 +0000 UTC (0+3 container statuses recorded) I1105 21:11:10.371] Nov 5 21:11:10.369: INFO: Container azuredisk ready: true, restart count 0 I1105 21:11:10.371] Nov 5 21:11:10.369: INFO: Container liveness-probe ready: true, restart count 0 I1105 21:11:10.371] Nov 5 21:11:10.369: INFO: Container node-driver-registrar ready: true, restart count 0 I1105 21:11:10.372] Nov 5 21:11:10.369: INFO: blobfuse-flexvol-installer-p7hdb started at 2019-11-05 20:08:01 +0000 UTC (0+1 container statuses recorded) I1105 21:11:10.372] Nov 5 21:11:10.369: INFO: Container blobfuse-flexvol-installer ready: true, restart count 0 I1105 21:11:10.372] Nov 5 21:11:10.369: INFO: coredns-7f68dcdbdb-zh8g5 started at 2019-11-05 20:08:01 +0000 UTC (0+1 container statuses recorded) I1105 21:11:10.372] Nov 5 21:11:10.369: INFO: Container coredns ready: true, restart count 0 I1105 21:11:10.640] Nov 5 21:11:10.640: INFO: I1105 21:11:10.640] Latency metrics for node k8s-agentpool-97435445-vmss000000 I1105 21:11:10.640] Nov 5 21:11:10.640: INFO: I1105 21:11:10.640] Logging node info for node k8s-agentpool-97435445-vmss000001 I1105 21:11:10.718] Nov 5 21:11:10.715: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-agentpool-97435445-vmss000001,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-agentpool-97435445-vmss000001,UID:75d3cc50-ca6d-45e0-a42a-6090e8d537dd,ResourceVersion:12280,Generation:0,CreationTimestamp:2019-11-05 20:07:34 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{agentpool: agentpool,beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_D4s_v3,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 1,kubernetes.azure.com/cluster: kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b,kubernetes.azure.com/role: agent,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-agentpool-97435445-vmss000001,kubernetes.io/os: linux,kubernetes.io/role: agent,node-role.kubernetes.io/agent: ,storageprofile: managed,storagetier: Premium_LRS,topology.disk.csi.azure.com/zone: 1,},Annotations:map[string]string{csi.volume.kubernetes.io/nodeid: {"disk.csi.azure.com":"k8s-agentpool-97435445-vmss000001"},node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.0.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool-97435445-vmss/virtualMachines/1,Unschedulable:false,Taints:[],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-csi-disk.csi.azure.com: {{8 0} {} 8 DecimalSI},cpu: {{4 0} {} 4 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{16797044736 0} {} 16403364Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-csi-disk.csi.azure.com: {{8 0} {} 8 DecimalSI},cpu: {{4 0} {} 4 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{16010612736 0} {} 15635364Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 20:08:25 +0000 UTC 2019-11-05 20:08:25 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 21:10:44 +0000 UTC 2019-11-05 20:07:26 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 21:10:44 +0000 UTC 2019-11-05 20:07:26 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 21:10:44 +0000 UTC 2019-11-05 20:07:26 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 21:10:44 +0000 UTC 2019-11-05 20:07:55 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.0.5} {Hostname k8s-agentpool-97435445-vmss000001}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:e78def54b2f545c6b94d88bbba523c60,SystemUUID:40AE8188-86AB-D848-84A4-D9254E1258BC,BootID:8587ec06-f4f4-43c1-86cd-06c295b05b63,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[k8sprowinternal.azurecr.io/azuredisk-csi@sha256:75d4ff5d7cd845d6e810c67796ecb037e852fb849ec1eb5b3194564008e6d1e0 k8sprowinternal.azurecr.io/azuredisk-csi:e2e-3e3ed045c6faa509dbd8d2890ee9f56aa2934887] 286930414} {[gcr.io/kubernetes-helm/tiller@sha256:f6d8f4ab9ba993b5f5b60a6edafe86352eabe474ffeb84cb6c79b8866dce45d1 gcr.io/kubernetes-helm/tiller:v2.11.0] 71821984} {[quay.io/k8scsi/csi-provisioner@sha256:d657c839dce87324fe2b677302913f9386f885f8746be7bea0ced5b0844e3433 quay.io/k8scsi/csi-provisioner:v1.3.1] 50596998} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[quay.io/k8scsi/csi-attacher@sha256:26fccd7a99d973845df1193b46ebdcc6ab8dc5f6e6be319750c471fce1742d13 quay.io/k8scsi/csi-attacher:v1.2.0] 46226754} {[quay.io/k8scsi/csi-cluster-driver-registrar@sha256:fafd75ae5442f192cfa8c2e792903aee30d5884b62e802e4464b0a895d21e3ef quay.io/k8scsi/csi-cluster-driver-registrar:v1.0.1] 45874691} {[quay.io/k8scsi/csi-snapshotter@sha256:a49e0da1af6f2bf717e41ba1eee8b5e6a1cbd66a709dd92cc43fe475fe2589eb quay.io/k8scsi/csi-snapshotter:v1.1.0] 44325352} {[mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume@sha256:4fd30d43947d4a54fc89ead7985beecfd3c9b2a93a0655a373b1608ab90bd5af mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume:v0.0.7] 22909487} {[quay.io/k8scsi/livenessprobe@sha256:8c9c8a0f330acc2e558ae6624309447d02522e76aa35af23fa19b88abf4381cb quay.io/k8scsi/livenessprobe:v1.0.2] 17415381} {[quay.io/k8scsi/csi-node-driver-registrar@sha256:13daf82fb99e951a4bff8ae5fc7c17c3a8fe7130be6400990d8f6076c32d4599 quay.io/k8scsi/csi-node-driver-registrar:v1.1.0] 15815995} {[quay.io/k8scsi/livenessprobe@sha256:dde617756e0f602adc566ab71fd885f1dad451ad3fb063ac991c95a2ff47aea5 quay.io/k8scsi/livenessprobe:v1.1.0] 14967303} {[mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume@sha256:23d8c6033f02a1ecad05127ebdc931bb871264228661bc122704b0974e4d9fdd mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume:1.0.8] 1159025} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 21:11:10.719] Nov 5 21:11:10.715: INFO: I1105 21:11:10.719] Logging kubelet events for node k8s-agentpool-97435445-vmss000001 I1105 21:11:10.793] Nov 5 21:11:10.793: INFO: I1105 21:11:10.794] Logging pods the kubelet thinks is on node k8s-agentpool-97435445-vmss000001 I1105 21:11:10.875] Nov 5 21:11:10.875: INFO: azure-ip-masq-agent-gdhcq started at 2019-11-05 20:07:59 +0000 UTC (0+1 container statuses recorded) I1105 21:11:10.876] Nov 5 21:11:10.875: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 21:11:10.876] Nov 5 21:11:10.875: INFO: keyvault-flexvolume-6bffx started at 2019-11-05 20:07:59 +0000 UTC (0+1 container statuses recorded) I1105 21:11:10.876] Nov 5 21:11:10.875: INFO: Container keyvault-flexvolume ready: true, restart count 0 I1105 21:11:10.876] Nov 5 21:11:10.875: INFO: kube-proxy-kjrpb started at 2019-11-05 20:08:00 +0000 UTC (0+1 container statuses recorded) I1105 21:11:10.876] Nov 5 21:11:10.875: INFO: Container kube-proxy ready: true, restart count 0 I1105 21:11:10.876] Nov 5 21:11:10.875: INFO: tiller-deploy-54c96cb5df-pt7x6 started at 2019-11-05 20:08:00 +0000 UTC (0+1 container statuses recorded) I1105 21:11:10.877] Nov 5 21:11:10.875: INFO: Container tiller ready: true, restart count 0 I1105 21:11:10.877] Nov 5 21:11:10.875: INFO: csi-azuredisk-node-q2dmt started at 2019-11-05 20:16:09 +0000 UTC (0+3 container statuses recorded) I1105 21:11:10.877] Nov 5 21:11:10.875: INFO: Container azuredisk ready: true, restart count 0 I1105 21:11:10.877] Nov 5 21:11:10.875: INFO: Container liveness-probe ready: true, restart count 0 I1105 21:11:10.877] Nov 5 21:11:10.875: INFO: Container node-driver-registrar ready: true, restart count 0 I1105 21:11:10.877] Nov 5 21:11:10.875: INFO: csi-azuredisk-controller-6f6587bfc6-62nrt started at 2019-11-05 20:16:09 +0000 UTC (0+6 container statuses recorded) I1105 21:11:10.877] Nov 5 21:11:10.875: INFO: Container azuredisk ready: true, restart count 0 I1105 21:11:10.877] Nov 5 21:11:10.875: INFO: Container cluster-driver-registrar ready: true, restart count 0 I1105 21:11:10.878] Nov 5 21:11:10.875: INFO: Container csi-attacher ready: true, restart count 0 I1105 21:11:10.878] Nov 5 21:11:10.875: INFO: Container csi-provisioner ready: true, restart count 0 I1105 21:11:10.878] Nov 5 21:11:10.875: INFO: Container csi-snapshotter ready: true, restart count 0 I1105 21:11:10.878] Nov 5 21:11:10.875: INFO: Container liveness-probe ready: true, restart count 0 I1105 21:11:10.878] Nov 5 21:11:10.875: INFO: blobfuse-flexvol-installer-j4s7t started at 2019-11-05 20:07:59 +0000 UTC (0+1 container statuses recorded) I1105 21:11:10.878] Nov 5 21:11:10.875: INFO: Container blobfuse-flexvol-installer ready: true, restart count 0 I1105 21:11:11.134] Nov 5 21:11:11.134: INFO: I1105 21:11:11.134] Latency metrics for node k8s-agentpool-97435445-vmss000001 I1105 21:11:11.135] Nov 5 21:11:11.134: INFO: I1105 21:11:11.135] Logging node info for node k8s-master-97435445-0 I1105 21:11:11.213] Nov 5 21:11:11.211: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-97435445-0,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-97435445-0,UID:df3ae36f-961c-4729-a8f5-34cda54f4ff9,ResourceVersion:12341,Generation:0,CreationTimestamp:2019-11-05 20:08:36 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 2,kubernetes.azure.com/cluster: kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-97435445-0,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.4.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b/providers/Microsoft.Compute/virtualMachines/k8s-master-97435445-0,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 20:08:46 +0000 UTC 2019-11-05 20:08:46 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 21:11:05 +0000 UTC 2019-11-05 20:08:28 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 21:11:05 +0000 UTC 2019-11-05 20:08:28 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 21:11:05 +0000 UTC 2019-11-05 20:08:28 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 21:11:05 +0000 UTC 2019-11-05 20:08:36 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.255.5} {Hostname k8s-master-97435445-0}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:3c381865192341c2868672e8626cef7d,SystemUUID:F3D259D3-7EB8-B844-A1F0-F954DCD0D76B,BootID:4948329e-cf57-48c3-aa83-5297bef70370,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 21:11:11.213] Nov 5 21:11:11.211: INFO: I1105 21:11:11.214] Logging kubelet events for node k8s-master-97435445-0 I1105 21:11:11.288] Nov 5 21:11:11.288: INFO: I1105 21:11:11.289] Logging pods the kubelet thinks is on node k8s-master-97435445-0 I1105 21:11:11.370] Nov 5 21:11:11.369: INFO: kube-proxy-5wnd2 started at 2019-11-05 20:08:38 +0000 UTC (0+1 container statuses recorded) I1105 21:11:11.370] Nov 5 21:11:11.369: INFO: Container kube-proxy ready: true, restart count 0 I1105 21:11:11.370] Nov 5 21:11:11.369: INFO: cloud-controller-manager-k8s-master-97435445-0 started at 2019-11-05 20:08:22 +0000 UTC (0+1 container statuses recorded) I1105 21:11:11.370] Nov 5 21:11:11.369: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 21:11:11.370] Nov 5 21:11:11.369: INFO: kube-addon-manager-k8s-master-97435445-0 started at 2019-11-05 20:08:48 +0000 UTC (0+1 container statuses recorded) I1105 21:11:11.370] Nov 5 21:11:11.369: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 21:11:11.370] Nov 5 21:11:11.369: INFO: kube-apiserver-k8s-master-97435445-0 started at 2019-11-05 20:08:22 +0000 UTC (0+1 container statuses recorded) I1105 21:11:11.371] Nov 5 21:11:11.369: INFO: Container kube-apiserver ready: true, restart count 0 I1105 21:11:11.371] Nov 5 21:11:11.369: INFO: kube-controller-manager-k8s-master-97435445-0 started at 2019-11-05 20:08:22 +0000 UTC (0+1 container statuses recorded) I1105 21:11:11.371] Nov 5 21:11:11.369: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 21:11:11.371] Nov 5 21:11:11.369: INFO: kube-scheduler-k8s-master-97435445-0 started at 2019-11-05 20:08:22 +0000 UTC (0+1 container statuses recorded) I1105 21:11:11.371] Nov 5 21:11:11.369: INFO: Container kube-scheduler ready: true, restart count 0 I1105 21:11:11.371] Nov 5 21:11:11.369: INFO: azure-ip-masq-agent-q6dcl started at 2019-11-05 20:08:38 +0000 UTC (0+1 container statuses recorded) I1105 21:11:11.371] Nov 5 21:11:11.369: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 21:11:11.619] Nov 5 21:11:11.619: INFO: I1105 21:11:11.620] Latency metrics for node k8s-master-97435445-0 I1105 21:11:11.620] Nov 5 21:11:11.619: INFO: I1105 21:11:11.620] Logging node info for node k8s-master-97435445-1 I1105 21:11:11.702] Nov 5 21:11:11.699: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-97435445-1,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-97435445-1,UID:05534a67-d493-4ab5-9b0f-5ff5490e234f,ResourceVersion:12248,Generation:0,CreationTimestamp:2019-11-05 20:07:34 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 0,kubernetes.azure.com/cluster: kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-97435445-1,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.1.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b/providers/Microsoft.Compute/virtualMachines/k8s-master-97435445-1,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 20:08:26 +0000 UTC 2019-11-05 20:08:26 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 21:10:32 +0000 UTC 2019-11-05 20:07:28 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 21:10:32 +0000 UTC 2019-11-05 20:07:28 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 21:10:32 +0000 UTC 2019-11-05 20:07:28 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 21:10:32 +0000 UTC 2019-11-05 20:07:55 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{Hostname k8s-master-97435445-1} {InternalIP 10.240.255.6}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:079b605103b74acabac18fd2ea63368a,SystemUUID:38220EEC-BEFF-4C46-AA05-5A214AAFDC28,BootID:b1b45df0-d925-4bcc-9830-0202df36ebb4,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 21:11:11.702] Nov 5 21:11:11.699: INFO: I1105 21:11:11.702] Logging kubelet events for node k8s-master-97435445-1 I1105 21:11:11.777] Nov 5 21:11:11.777: INFO: I1105 21:11:11.777] Logging pods the kubelet thinks is on node k8s-master-97435445-1 I1105 21:11:11.859] Nov 5 21:11:11.859: INFO: azure-ip-masq-agent-n29bm started at 2019-11-05 20:07:59 +0000 UTC (0+1 container statuses recorded) I1105 21:11:11.859] Nov 5 21:11:11.859: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 21:11:11.859] Nov 5 21:11:11.859: INFO: kube-proxy-h4s2c started at 2019-11-05 20:08:00 +0000 UTC (0+1 container statuses recorded) I1105 21:11:11.860] Nov 5 21:11:11.859: INFO: Container kube-proxy ready: true, restart count 0 I1105 21:11:11.860] Nov 5 21:11:11.859: INFO: cloud-controller-manager-k8s-master-97435445-1 started at 2019-11-05 20:07:22 +0000 UTC (0+1 container statuses recorded) I1105 21:11:11.860] Nov 5 21:11:11.859: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 21:11:11.860] Nov 5 21:11:11.859: INFO: kube-addon-manager-k8s-master-97435445-1 started at 2019-11-05 20:07:22 +0000 UTC (0+1 container statuses recorded) I1105 21:11:11.860] Nov 5 21:11:11.859: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 21:11:11.860] Nov 5 21:11:11.859: INFO: kube-apiserver-k8s-master-97435445-1 started at 2019-11-05 20:07:22 +0000 UTC (0+1 container statuses recorded) I1105 21:11:11.861] Nov 5 21:11:11.859: INFO: Container kube-apiserver ready: true, restart count 0 I1105 21:11:11.861] Nov 5 21:11:11.859: INFO: kube-controller-manager-k8s-master-97435445-1 started at 2019-11-05 20:07:22 +0000 UTC (0+1 container statuses recorded) I1105 21:11:11.861] Nov 5 21:11:11.859: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 21:11:11.861] Nov 5 21:11:11.859: INFO: kube-scheduler-k8s-master-97435445-1 started at 2019-11-05 20:07:22 +0000 UTC (0+1 container statuses recorded) I1105 21:11:11.861] Nov 5 21:11:11.859: INFO: Container kube-scheduler ready: true, restart count 0 I1105 21:11:12.108] Nov 5 21:11:12.108: INFO: I1105 21:11:12.109] Latency metrics for node k8s-master-97435445-1 I1105 21:11:12.109] Nov 5 21:11:12.108: INFO: I1105 21:11:12.109] Logging node info for node k8s-master-97435445-2 I1105 21:11:12.187] Nov 5 21:11:12.184: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-97435445-2,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-97435445-2,UID:fa55c895-1a93-4743-885e-03fce84b5332,ResourceVersion:12296,Generation:0,CreationTimestamp:2019-11-05 20:09:19 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 0,kubernetes.azure.com/cluster: kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-97435445-2,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.5.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b/providers/Microsoft.Compute/virtualMachines/k8s-master-97435445-2,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 20:09:19 +0000 UTC 2019-11-05 20:09:19 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 21:10:50 +0000 UTC 2019-11-05 20:09:12 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 21:10:50 +0000 UTC 2019-11-05 20:09:12 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 21:10:50 +0000 UTC 2019-11-05 20:09:12 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 21:10:50 +0000 UTC 2019-11-05 20:09:19 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.255.7} {Hostname k8s-master-97435445-2}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:d19f6709138f496da43c095cec7b2360,SystemUUID:10116777-A8D7-CB4A-A8B7-7EBCF2F23B62,BootID:50cd14a9-5497-4972-bf60-cf59a0cbcb5f,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 21:11:12.187] Nov 5 21:11:12.184: INFO: I1105 21:11:12.187] Logging kubelet events for node k8s-master-97435445-2 I1105 21:11:12.267] Nov 5 21:11:12.267: INFO: I1105 21:11:12.267] Logging pods the kubelet thinks is on node k8s-master-97435445-2 I1105 21:11:12.347] Nov 5 21:11:12.347: INFO: cloud-controller-manager-k8s-master-97435445-2 started at 2019-11-05 20:09:05 +0000 UTC (0+1 container statuses recorded) I1105 21:11:12.348] Nov 5 21:11:12.347: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 21:11:12.348] Nov 5 21:11:12.347: INFO: kube-addon-manager-k8s-master-97435445-2 started at 2019-11-05 20:09:05 +0000 UTC (0+1 container statuses recorded) I1105 21:11:12.348] Nov 5 21:11:12.347: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 21:11:12.348] Nov 5 21:11:12.347: INFO: azure-ip-masq-agent-sq7p5 started at 2019-11-05 20:09:19 +0000 UTC (0+1 container statuses recorded) I1105 21:11:12.348] Nov 5 21:11:12.347: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 21:11:12.349] Nov 5 21:11:12.347: INFO: kube-proxy-pz426 started at 2019-11-05 20:09:19 +0000 UTC (0+1 container statuses recorded) I1105 21:11:12.349] Nov 5 21:11:12.347: INFO: Container kube-proxy ready: true, restart count 0 I1105 21:11:12.349] Nov 5 21:11:12.347: INFO: kube-apiserver-k8s-master-97435445-2 started at 2019-11-05 20:09:05 +0000 UTC (0+1 container statuses recorded) I1105 21:11:12.349] Nov 5 21:11:12.347: INFO: Container kube-apiserver ready: true, restart count 0 I1105 21:11:12.350] Nov 5 21:11:12.347: INFO: kube-controller-manager-k8s-master-97435445-2 started at 2019-11-05 20:09:05 +0000 UTC (0+1 container statuses recorded) I1105 21:11:12.350] Nov 5 21:11:12.347: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 21:11:12.350] Nov 5 21:11:12.347: INFO: kube-scheduler-k8s-master-97435445-2 started at 2019-11-05 20:09:05 +0000 UTC (0+1 container statuses recorded) I1105 21:11:12.350] Nov 5 21:11:12.347: INFO: Container kube-scheduler ready: true, restart count 0 I1105 21:11:12.602] Nov 5 21:11:12.602: INFO: I1105 21:11:12.602] Latency metrics for node k8s-master-97435445-2 I1105 21:11:12.602] Nov 5 21:11:12.602: INFO: I1105 21:11:12.602] Logging node info for node k8s-master-97435445-3 I1105 21:11:12.681] Nov 5 21:11:12.678: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-97435445-3,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-97435445-3,UID:8ffd1d0d-4487-4680-a2f6-e724387be637,ResourceVersion:12297,Generation:0,CreationTimestamp:2019-11-05 20:09:21 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 1,kubernetes.azure.com/cluster: kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-97435445-3,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.6.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b/providers/Microsoft.Compute/virtualMachines/k8s-master-97435445-3,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 20:09:39 +0000 UTC 2019-11-05 20:09:39 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 21:10:50 +0000 UTC 2019-11-05 20:09:18 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 21:10:50 +0000 UTC 2019-11-05 20:09:18 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 21:10:50 +0000 UTC 2019-11-05 20:09:18 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 21:10:50 +0000 UTC 2019-11-05 20:09:21 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.255.8} {Hostname k8s-master-97435445-3}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:266064173bfb4297898c6c25769e8b20,SystemUUID:E6E44241-5015-7740-9A81-A668A8427206,BootID:579d658c-b0ad-47f1-a993-1ccea2b55789,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 21:11:12.681] Nov 5 21:11:12.678: INFO: I1105 21:11:12.681] Logging kubelet events for node k8s-master-97435445-3 I1105 21:11:12.756] Nov 5 21:11:12.756: INFO: I1105 21:11:12.757] Logging pods the kubelet thinks is on node k8s-master-97435445-3 I1105 21:11:12.840] Nov 5 21:11:12.840: INFO: kube-controller-manager-k8s-master-97435445-3 started at 2019-11-05 20:09:05 +0000 UTC (0+1 container statuses recorded) I1105 21:11:12.840] Nov 5 21:11:12.840: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 21:11:12.840] Nov 5 21:11:12.840: INFO: kube-scheduler-k8s-master-97435445-3 started at 2019-11-05 20:09:05 +0000 UTC (0+1 container statuses recorded) I1105 21:11:12.840] Nov 5 21:11:12.840: INFO: Container kube-scheduler ready: true, restart count 0 I1105 21:11:12.841] Nov 5 21:11:12.840: INFO: azure-ip-masq-agent-747cl started at 2019-11-05 20:09:21 +0000 UTC (0+1 container statuses recorded) I1105 21:11:12.841] Nov 5 21:11:12.840: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 21:11:12.841] Nov 5 21:11:12.840: INFO: kube-proxy-k45g8 started at 2019-11-05 20:09:21 +0000 UTC (0+1 container statuses recorded) I1105 21:11:12.841] Nov 5 21:11:12.840: INFO: Container kube-proxy ready: true, restart count 0 I1105 21:11:12.841] Nov 5 21:11:12.840: INFO: cloud-controller-manager-k8s-master-97435445-3 started at 2019-11-05 20:09:04 +0000 UTC (0+1 container statuses recorded) I1105 21:11:12.842] Nov 5 21:11:12.840: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 21:11:12.842] Nov 5 21:11:12.840: INFO: kube-addon-manager-k8s-master-97435445-3 started at 2019-11-05 20:09:05 +0000 UTC (0+1 container statuses recorded) I1105 21:11:12.842] Nov 5 21:11:12.840: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 21:11:12.842] Nov 5 21:11:12.840: INFO: kube-apiserver-k8s-master-97435445-3 started at 2019-11-05 20:09:05 +0000 UTC (0+1 container statuses recorded) I1105 21:11:12.842] Nov 5 21:11:12.840: INFO: Container kube-apiserver ready: true, restart count 0 I1105 21:11:13.083] Nov 5 21:11:13.083: INFO: I1105 21:11:13.083] Latency metrics for node k8s-master-97435445-3 I1105 21:11:13.083] Nov 5 21:11:13.083: INFO: I1105 21:11:13.083] Logging node info for node k8s-master-97435445-4 I1105 21:11:13.161] Nov 5 21:11:13.158: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-97435445-4,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-97435445-4,UID:6d547b41-3f92-4cc4-a2d7-dda2071776a2,ResourceVersion:12339,Generation:0,CreationTimestamp:2019-11-05 20:08:36 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 1,kubernetes.azure.com/cluster: kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-97435445-4,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.3.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b/providers/Microsoft.Compute/virtualMachines/k8s-master-97435445-4,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 20:08:59 +0000 UTC 2019-11-05 20:08:59 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 21:11:04 +0000 UTC 2019-11-05 20:08:29 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 21:11:04 +0000 UTC 2019-11-05 20:08:29 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 21:11:04 +0000 UTC 2019-11-05 20:08:29 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 21:11:04 +0000 UTC 2019-11-05 20:08:36 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.255.9} {Hostname k8s-master-97435445-4}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:56b393c61eb34ba49ebda5279eeedb30,SystemUUID:FD239C23-618C-0049-8734-9AF3FDC9E3C1,BootID:f342ea44-e719-4513-807d-5def7ac606e2,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 21:11:13.161] Nov 5 21:11:13.159: INFO: I1105 21:11:13.161] Logging kubelet events for node k8s-master-97435445-4 I1105 21:11:13.236] Nov 5 21:11:13.236: INFO: I1105 21:11:13.236] Logging pods the kubelet thinks is on node k8s-master-97435445-4 I1105 21:11:13.318] Nov 5 21:11:13.317: INFO: kube-addon-manager-k8s-master-97435445-4 started at 2019-11-05 20:08:23 +0000 UTC (0+1 container statuses recorded) I1105 21:11:13.318] Nov 5 21:11:13.317: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 21:11:13.318] Nov 5 21:11:13.317: INFO: kube-apiserver-k8s-master-97435445-4 started at 2019-11-05 20:08:23 +0000 UTC (0+1 container statuses recorded) I1105 21:11:13.318] Nov 5 21:11:13.317: INFO: Container kube-apiserver ready: true, restart count 0 I1105 21:11:13.318] Nov 5 21:11:13.317: INFO: kube-controller-manager-k8s-master-97435445-4 started at 2019-11-05 20:08:23 +0000 UTC (0+1 container statuses recorded) I1105 21:11:13.319] Nov 5 21:11:13.317: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 21:11:13.319] Nov 5 21:11:13.317: INFO: kube-scheduler-k8s-master-97435445-4 started at 2019-11-05 20:08:23 +0000 UTC (0+1 container statuses recorded) I1105 21:11:13.319] Nov 5 21:11:13.317: INFO: Container kube-scheduler ready: true, restart count 0 I1105 21:11:13.319] Nov 5 21:11:13.317: INFO: kube-proxy-vfbfr started at 2019-11-05 20:08:36 +0000 UTC (0+1 container statuses recorded) I1105 21:11:13.319] Nov 5 21:11:13.317: INFO: Container kube-proxy ready: true, restart count 0 I1105 21:11:13.319] Nov 5 21:11:13.317: INFO: azure-ip-masq-agent-kmkll started at 2019-11-05 20:08:36 +0000 UTC (0+1 container statuses recorded) I1105 21:11:13.319] Nov 5 21:11:13.317: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 21:11:13.319] Nov 5 21:11:13.317: INFO: cloud-controller-manager-k8s-master-97435445-4 started at 2019-11-05 20:08:46 +0000 UTC (0+1 container statuses recorded) I1105 21:11:13.320] Nov 5 21:11:13.317: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 21:11:13.565] Nov 5 21:11:13.565: INFO: I1105 21:11:13.565] Latency metrics for node k8s-master-97435445-4 I1105 21:11:13.566] Nov 5 21:11:13.565: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready I1105 21:11:13.643] STEP: Destroying namespace "azuredisk-9582" for this suite. I1105 21:11:27.882] Nov 5 21:11:27.882: INFO: Waiting up to 30s for server preferred namespaced resources to be successfully discovered I1105 21:11:30.768] Nov 5 21:11:30.768: INFO: namespace azuredisk-9582 deletion completed in 17.125420008s I1105 21:11:30.768] I1105 21:11:30.768] • Failure [323.414 seconds] I1105 21:11:30.769] [azuredisk-csi-e2e] Dynamic Provisioning I1105 21:11:30.769] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:33 I1105 21:11:30.769] [single-az] I1105 21:11:30.769] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:36 I1105 21:11:30.769] [env] should retain PV with reclaimPolicy "Retain" [It] I1105 21:11:30.769] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:240 I1105 21:11:30.769] I1105 21:11:30.770] Unexpected error: I1105 21:11:30.770] <*errors.errorString | 0xc000a2d860>: { I1105 21:11:30.770] s: "PersistentVolumeClaims [pvc-m4z2v] not all in phase Bound within 5m0s", I1105 21:11:30.770] } I1105 21:11:30.770] PersistentVolumeClaims [pvc-m4z2v] not all in phase Bound within 5m0s I1105 21:11:30.770] occurred I1105 21:11:30.770] I1105 21:11:30.771] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/testsuites/testsuites.go:278 I1105 21:11:30.771] ------------------------------ I1105 21:11:30.771] [azuredisk-csi-e2e] Dynamic Provisioning [single-az] I1105 21:11:30.771] should create a volume on demand I1105 21:11:30.771] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:78 I1105 21:11:30.771] [BeforeEach] [single-az] I1105 21:11:30.772] /go/src/sigs.k8s.io/azuredisk-csi-driver/vendor/k8s.io/kubernetes/test/e2e/framework/framework.go:150 I1105 21:11:30.772] STEP: Creating a kubernetes client I1105 21:11:30.772] Nov 5 21:11:30.768: INFO: >>> kubeConfig: /workspace/aks971561099/kubeconfig/kubeconfig.eastus2.json I1105 21:11:30.772] STEP: Building a namespace api object, basename azuredisk I1105 21:11:31.009] STEP: Binding the e2e-test-privileged-psp PodSecurityPolicy to the default service account in azuredisk-3372 I1105 21:11:31.267] STEP: Waiting for a default service account to be provisioned in namespace I1105 21:11:31.341] [BeforeEach] [single-az] I1105 21:11:31.341] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:58 I1105 21:11:31.341] [It] should create a volume on demand I1105 21:11:31.342] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:78 I1105 21:11:31.342] STEP: setting up the StorageClass I1105 21:11:31.342] STEP: creating a StorageClass I1105 21:11:31.422] STEP: setting up the PVC and PV I1105 21:11:31.422] STEP: creating a PVC I1105 21:11:31.507] STEP: waiting for PVC to be in phase "Bound" I1105 21:11:31.507] Nov 5 21:11:31.507: INFO: Waiting up to 5m0s for PersistentVolumeClaims [pvc-n9hsv] to have phase Bound I1105 21:11:31.588] Nov 5 21:11:31.588: INFO: PersistentVolumeClaim pvc-n9hsv found but phase is Pending instead of Bound. I1105 21:11:33.665] Nov 5 21:11:33.665: INFO: PersistentVolumeClaim pvc-n9hsv found but phase is Pending instead of Bound. I1105 21:11:35.741] Nov 5 21:11:35.741: INFO: PersistentVolumeClaim pvc-n9hsv found but phase is Pending instead of Bound. I1105 21:11:37.818] Nov 5 21:11:37.817: INFO: PersistentVolumeClaim pvc-n9hsv found but phase is Pending instead of Bound. I1105 21:11:39.894] Nov 5 21:11:39.894: INFO: PersistentVolumeClaim pvc-n9hsv found but phase is Pending instead of Bound. I1105 21:11:41.971] Nov 5 21:11:41.971: INFO: PersistentVolumeClaim pvc-n9hsv found but phase is Pending instead of Bound. I1105 21:11:44.048] Nov 5 21:11:44.047: INFO: PersistentVolumeClaim pvc-n9hsv found but phase is Pending instead of Bound. I1105 21:11:46.125] Nov 5 21:11:46.124: INFO: PersistentVolumeClaim pvc-n9hsv found but phase is Pending instead of Bound. I1105 21:11:48.201] Nov 5 21:11:48.201: INFO: PersistentVolumeClaim pvc-n9hsv found but phase is Pending instead of Bound. I1105 21:11:50.278] Nov 5 21:11:50.277: INFO: PersistentVolumeClaim pvc-n9hsv found but phase is Pending instead of Bound. I1105 21:11:52.354] Nov 5 21:11:52.354: INFO: PersistentVolumeClaim pvc-n9hsv found but phase is Pending instead of Bound. I1105 21:11:54.431] Nov 5 21:11:54.431: INFO: PersistentVolumeClaim pvc-n9hsv found but phase is Pending instead of Bound. I1105 21:11:56.508] Nov 5 21:11:56.508: INFO: PersistentVolumeClaim pvc-n9hsv found but phase is Pending instead of Bound. I1105 21:11:58.611] Nov 5 21:11:58.611: INFO: PersistentVolumeClaim pvc-n9hsv found but phase is Pending instead of Bound. I1105 21:12:00.688] Nov 5 21:12:00.688: INFO: PersistentVolumeClaim pvc-n9hsv found but phase is Pending instead of Bound. I1105 21:12:02.765] Nov 5 21:12:02.765: INFO: PersistentVolumeClaim pvc-n9hsv found but phase is Pending instead of Bound. I1105 21:12:04.842] Nov 5 21:12:04.842: INFO: PersistentVolumeClaim pvc-n9hsv found but phase is Pending instead of Bound. I1105 21:12:06.919] Nov 5 21:12:06.919: INFO: PersistentVolumeClaim pvc-n9hsv found but phase is Pending instead of Bound. I1105 21:12:08.995] Nov 5 21:12:08.995: INFO: PersistentVolumeClaim pvc-n9hsv found but phase is Pending instead of Bound. I1105 21:12:11.074] Nov 5 21:12:11.073: INFO: PersistentVolumeClaim pvc-n9hsv found but phase is Pending instead of Bound. I1105 21:12:13.150] Nov 5 21:12:13.150: INFO: PersistentVolumeClaim pvc-n9hsv found but phase is Pending instead of Bound. I1105 21:12:15.227] Nov 5 21:12:15.226: INFO: PersistentVolumeClaim pvc-n9hsv found but phase is Pending instead of Bound. I1105 21:12:17.303] Nov 5 21:12:17.303: INFO: PersistentVolumeClaim pvc-n9hsv found but phase is Pending instead of Bound. I1105 21:12:19.379] Nov 5 21:12:19.379: INFO: PersistentVolumeClaim pvc-n9hsv found but phase is Pending instead of Bound. I1105 21:12:21.456] Nov 5 21:12:21.456: INFO: PersistentVolumeClaim pvc-n9hsv found but phase is Pending instead of Bound. I1105 21:12:23.533] Nov 5 21:12:23.532: INFO: PersistentVolumeClaim pvc-n9hsv found but phase is Pending instead of Bound. I1105 21:12:25.609] Nov 5 21:12:25.609: INFO: PersistentVolumeClaim pvc-n9hsv found but phase is Pending instead of Bound. I1105 21:12:27.686] Nov 5 21:12:27.686: INFO: PersistentVolumeClaim pvc-n9hsv found but phase is Pending instead of Bound. I1105 21:12:29.763] Nov 5 21:12:29.763: INFO: PersistentVolumeClaim pvc-n9hsv found but phase is Pending instead of Bound. I1105 21:12:31.840] Nov 5 21:12:31.840: INFO: PersistentVolumeClaim pvc-n9hsv found but phase is Pending instead of Bound. I1105 21:12:33.916] Nov 5 21:12:33.916: INFO: PersistentVolumeClaim pvc-n9hsv found but phase is Pending instead of Bound. I1105 21:12:35.992] Nov 5 21:12:35.992: INFO: PersistentVolumeClaim pvc-n9hsv found but phase is Pending instead of Bound. I1105 21:12:38.069] Nov 5 21:12:38.069: INFO: PersistentVolumeClaim pvc-n9hsv found but phase is Pending instead of Bound. I1105 21:12:40.146] Nov 5 21:12:40.146: INFO: PersistentVolumeClaim pvc-n9hsv found but phase is Pending instead of Bound. I1105 21:12:42.223] Nov 5 21:12:42.223: INFO: PersistentVolumeClaim pvc-n9hsv found but phase is Pending instead of Bound. I1105 21:12:44.300] Nov 5 21:12:44.300: INFO: PersistentVolumeClaim pvc-n9hsv found but phase is Pending instead of Bound. I1105 21:12:46.377] Nov 5 21:12:46.377: INFO: PersistentVolumeClaim pvc-n9hsv found but phase is Pending instead of Bound. I1105 21:12:48.454] Nov 5 21:12:48.454: INFO: PersistentVolumeClaim pvc-n9hsv found but phase is Pending instead of Bound. I1105 21:12:50.532] Nov 5 21:12:50.532: INFO: PersistentVolumeClaim pvc-n9hsv found but phase is Pending instead of Bound. I1105 21:12:52.609] Nov 5 21:12:52.609: INFO: PersistentVolumeClaim pvc-n9hsv found but phase is Pending instead of Bound. I1105 21:12:54.686] Nov 5 21:12:54.686: INFO: PersistentVolumeClaim pvc-n9hsv found but phase is Pending instead of Bound. I1105 21:12:56.762] Nov 5 21:12:56.762: INFO: PersistentVolumeClaim pvc-n9hsv found but phase is Pending instead of Bound. I1105 21:12:58.839] Nov 5 21:12:58.839: INFO: PersistentVolumeClaim pvc-n9hsv found but phase is Pending instead of Bound. I1105 21:13:00.916] Nov 5 21:13:00.916: INFO: PersistentVolumeClaim pvc-n9hsv found but phase is Pending instead of Bound. I1105 21:13:02.993] Nov 5 21:13:02.992: INFO: PersistentVolumeClaim pvc-n9hsv found but phase is Pending instead of Bound. I1105 21:13:05.069] Nov 5 21:13:05.069: INFO: PersistentVolumeClaim pvc-n9hsv found but phase is Pending instead of Bound. I1105 21:13:07.146] Nov 5 21:13:07.146: INFO: PersistentVolumeClaim pvc-n9hsv found but phase is Pending instead of Bound. I1105 21:13:09.223] Nov 5 21:13:09.223: INFO: PersistentVolumeClaim pvc-n9hsv found but phase is Pending instead of Bound. I1105 21:13:11.299] Nov 5 21:13:11.299: INFO: PersistentVolumeClaim pvc-n9hsv found but phase is Pending instead of Bound. I1105 21:13:13.376] Nov 5 21:13:13.376: INFO: PersistentVolumeClaim pvc-n9hsv found but phase is Pending instead of Bound. I1105 21:13:15.453] Nov 5 21:13:15.452: INFO: PersistentVolumeClaim pvc-n9hsv found but phase is Pending instead of Bound. I1105 21:13:17.529] Nov 5 21:13:17.529: INFO: PersistentVolumeClaim pvc-n9hsv found but phase is Pending instead of Bound. I1105 21:13:19.605] Nov 5 21:13:19.605: INFO: PersistentVolumeClaim pvc-n9hsv found but phase is Pending instead of Bound. I1105 21:13:21.681] Nov 5 21:13:21.681: INFO: PersistentVolumeClaim pvc-n9hsv found but phase is Pending instead of Bound. I1105 21:13:23.758] Nov 5 21:13:23.758: INFO: PersistentVolumeClaim pvc-n9hsv found but phase is Pending instead of Bound. I1105 21:13:25.835] Nov 5 21:13:25.834: INFO: PersistentVolumeClaim pvc-n9hsv found but phase is Pending instead of Bound. I1105 21:13:27.911] Nov 5 21:13:27.911: INFO: PersistentVolumeClaim pvc-n9hsv found but phase is Pending instead of Bound. I1105 21:13:29.988] Nov 5 21:13:29.988: INFO: PersistentVolumeClaim pvc-n9hsv found but phase is Pending instead of Bound. I1105 21:13:32.065] Nov 5 21:13:32.065: INFO: PersistentVolumeClaim pvc-n9hsv found but phase is Pending instead of Bound. I1105 21:13:34.142] Nov 5 21:13:34.142: INFO: PersistentVolumeClaim pvc-n9hsv found but phase is Pending instead of Bound. I1105 21:13:36.219] Nov 5 21:13:36.218: INFO: PersistentVolumeClaim pvc-n9hsv found but phase is Pending instead of Bound. I1105 21:13:38.295] Nov 5 21:13:38.295: INFO: PersistentVolumeClaim pvc-n9hsv found but phase is Pending instead of Bound. I1105 21:13:40.372] Nov 5 21:13:40.372: INFO: PersistentVolumeClaim pvc-n9hsv found but phase is Pending instead of Bound. I1105 21:13:42.448] Nov 5 21:13:42.448: INFO: PersistentVolumeClaim pvc-n9hsv found but phase is Pending instead of Bound. I1105 21:13:44.525] Nov 5 21:13:44.525: INFO: PersistentVolumeClaim pvc-n9hsv found but phase is Pending instead of Bound. I1105 21:13:46.623] Nov 5 21:13:46.622: INFO: PersistentVolumeClaim pvc-n9hsv found but phase is Pending instead of Bound. I1105 21:13:48.708] Nov 5 21:13:48.708: INFO: PersistentVolumeClaim pvc-n9hsv found but phase is Pending instead of Bound. I1105 21:13:50.784] Nov 5 21:13:50.784: INFO: PersistentVolumeClaim pvc-n9hsv found but phase is Pending instead of Bound. I1105 21:13:52.861] Nov 5 21:13:52.861: INFO: PersistentVolumeClaim pvc-n9hsv found but phase is Pending instead of Bound. I1105 21:13:54.938] Nov 5 21:13:54.938: INFO: PersistentVolumeClaim pvc-n9hsv found but phase is Pending instead of Bound. I1105 21:13:57.014] Nov 5 21:13:57.014: INFO: PersistentVolumeClaim pvc-n9hsv found but phase is Pending instead of Bound. I1105 21:13:59.091] Nov 5 21:13:59.091: INFO: PersistentVolumeClaim pvc-n9hsv found but phase is Pending instead of Bound. I1105 21:15:15.074] Nov 5 21:15:15.073: INFO: PersistentVolumeClaim pvc-n9hsv found but phase is Pending instead of Bound. I1105 21:15:17.153] Nov 5 21:15:17.153: INFO: PersistentVolumeClaim pvc-n9hsv found but phase is Pending instead of Bound. I1105 21:15:19.232] Nov 5 21:15:19.232: INFO: PersistentVolumeClaim pvc-n9hsv found but phase is Pending instead of Bound. I1105 21:15:21.312] Nov 5 21:15:21.312: INFO: PersistentVolumeClaim pvc-n9hsv found but phase is Pending instead of Bound. I1105 21:15:23.391] Nov 5 21:15:23.390: INFO: PersistentVolumeClaim pvc-n9hsv found but phase is Pending instead of Bound. I1105 21:15:25.470] Nov 5 21:15:25.470: INFO: PersistentVolumeClaim pvc-n9hsv found but phase is Pending instead of Bound. I1105 21:15:27.559] Nov 5 21:15:27.559: INFO: PersistentVolumeClaim pvc-n9hsv found but phase is Pending instead of Bound. I1105 21:15:29.639] Nov 5 21:15:29.638: INFO: PersistentVolumeClaim pvc-n9hsv found but phase is Pending instead of Bound. I1105 21:15:31.718] Nov 5 21:15:31.718: INFO: PersistentVolumeClaim pvc-n9hsv found but phase is Pending instead of Bound. I1105 21:15:33.797] Nov 5 21:15:33.796: INFO: PersistentVolumeClaim pvc-n9hsv found but phase is Pending instead of Bound. I1105 21:15:35.876] Nov 5 21:15:35.876: INFO: PersistentVolumeClaim pvc-n9hsv found but phase is Pending instead of Bound. I1105 21:15:37.956] Nov 5 21:15:37.955: INFO: PersistentVolumeClaim pvc-n9hsv found but phase is Pending instead of Bound. I1105 21:15:40.034] Nov 5 21:15:40.034: INFO: PersistentVolumeClaim pvc-n9hsv found but phase is Pending instead of Bound. I1105 21:15:42.113] Nov 5 21:15:42.113: INFO: PersistentVolumeClaim pvc-n9hsv found but phase is Pending instead of Bound. I1105 21:15:44.192] Nov 5 21:15:44.192: INFO: PersistentVolumeClaim pvc-n9hsv found but phase is Pending instead of Bound. I1105 21:15:46.279] Nov 5 21:15:46.279: INFO: PersistentVolumeClaim pvc-n9hsv found but phase is Pending instead of Bound. I1105 21:15:48.359] Nov 5 21:15:48.358: INFO: PersistentVolumeClaim pvc-n9hsv found but phase is Pending instead of Bound. I1105 21:15:50.438] Nov 5 21:15:50.437: INFO: PersistentVolumeClaim pvc-n9hsv found but phase is Pending instead of Bound. I1105 21:15:52.517] Nov 5 21:15:52.516: INFO: PersistentVolumeClaim pvc-n9hsv found but phase is Pending instead of Bound. I1105 21:15:54.618] Nov 5 21:15:54.618: INFO: PersistentVolumeClaim pvc-n9hsv found but phase is Pending instead of Bound. I1105 21:15:56.706] Nov 5 21:15:56.706: INFO: PersistentVolumeClaim pvc-n9hsv found but phase is Pending instead of Bound. I1105 21:15:58.784] Nov 5 21:15:58.784: INFO: PersistentVolumeClaim pvc-n9hsv found but phase is Pending instead of Bound. I1105 21:16:00.866] Nov 5 21:16:00.866: INFO: PersistentVolumeClaim pvc-n9hsv found but phase is Pending instead of Bound. I1105 21:16:02.945] Nov 5 21:16:02.945: INFO: PersistentVolumeClaim pvc-n9hsv found but phase is Pending instead of Bound. I1105 21:16:05.024] Nov 5 21:16:05.024: INFO: PersistentVolumeClaim pvc-n9hsv found but phase is Pending instead of Bound. I1105 21:16:07.103] Nov 5 21:16:07.103: INFO: PersistentVolumeClaim pvc-n9hsv found but phase is Pending instead of Bound. I1105 21:16:09.182] Nov 5 21:16:09.182: INFO: PersistentVolumeClaim pvc-n9hsv found but phase is Pending instead of Bound. I1105 21:16:11.261] Nov 5 21:16:11.261: INFO: PersistentVolumeClaim pvc-n9hsv found but phase is Pending instead of Bound. I1105 21:16:13.341] Nov 5 21:16:13.341: INFO: PersistentVolumeClaim pvc-n9hsv found but phase is Pending instead of Bound. I1105 21:16:15.420] Nov 5 21:16:15.420: INFO: PersistentVolumeClaim pvc-n9hsv found but phase is Pending instead of Bound. I1105 21:16:17.498] Nov 5 21:16:17.498: INFO: PersistentVolumeClaim pvc-n9hsv found but phase is Pending instead of Bound. I1105 21:16:19.578] Nov 5 21:16:19.578: INFO: PersistentVolumeClaim pvc-n9hsv found but phase is Pending instead of Bound. I1105 21:16:21.657] Nov 5 21:16:21.656: INFO: PersistentVolumeClaim pvc-n9hsv found but phase is Pending instead of Bound. I1105 21:16:23.736] Nov 5 21:16:23.736: INFO: PersistentVolumeClaim pvc-n9hsv found but phase is Pending instead of Bound. I1105 21:16:25.815] Nov 5 21:16:25.815: INFO: PersistentVolumeClaim pvc-n9hsv found but phase is Pending instead of Bound. I1105 21:16:27.903] Nov 5 21:16:27.903: INFO: PersistentVolumeClaim pvc-n9hsv found but phase is Pending instead of Bound. I1105 21:16:29.982] Nov 5 21:16:29.982: INFO: PersistentVolumeClaim pvc-n9hsv found but phase is Pending instead of Bound. I1105 21:16:31.982] Nov 5 21:16:31.982: INFO: Unexpected error occurred: PersistentVolumeClaims [pvc-n9hsv] not all in phase Bound within 5m0s I1105 21:16:31.983] [AfterEach] [single-az] I1105 21:16:31.983] /go/src/sigs.k8s.io/azuredisk-csi-driver/vendor/k8s.io/kubernetes/test/e2e/framework/framework.go:151 I1105 21:16:31.983] STEP: Collecting events from namespace "azuredisk-3372". I1105 21:16:32.062] STEP: Found 3 events. I1105 21:16:32.062] Nov 5 21:16:32.062: INFO: At 2019-11-05 21:11:31 +0000 UTC - event for pvc-n9hsv: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "disk.csi.azure.com" or manually created by system administrator I1105 21:16:32.062] Nov 5 21:16:32.062: INFO: At 2019-11-05 21:11:31 +0000 UTC - event for pvc-n9hsv: {disk.csi.azure.com_csi-azuredisk-controller-6f6587bfc6-62nrt_3930bc69-0009-11ea-8cf5-829a27ff7d5a } Provisioning: External provisioner is provisioning volume for claim "azuredisk-3372/pvc-n9hsv" I1105 21:16:32.063] Nov 5 21:16:32.062: INFO: At 2019-11-05 21:11:31 +0000 UTC - event for pvc-n9hsv: {disk.csi.azure.com_csi-azuredisk-controller-6f6587bfc6-62nrt_3930bc69-0009-11ea-8cf5-829a27ff7d5a } ProvisioningFailed: failed to provision volume with StorageClass "azuredisk-3372-disk.csi.azure.com-dynamic-sc-mklzd": rpc error: code = Unknown desc = compute.DisksClient#CreateOrUpdate: Failure sending request: StatusCode=400 -- Original Error: Code="InvalidAvailabilityZone" Message="The zone(s) '' for resource 'Microsoft.Compute/disks/pvc-886ff5f0-a896-492a-8061-9c66ac06d55e' is not supported. The supported zones for location 'eastus2' are '1,2,3'" I1105 21:16:32.141] Nov 5 21:16:32.141: INFO: POD NODE PHASE GRACE CONDITIONS I1105 21:16:32.141] Nov 5 21:16:32.141: INFO: I1105 21:16:32.603] Nov 5 21:16:32.603: INFO: I1105 21:16:32.603] Logging node info for node k8s-agentpool-97435445-vmss000000 I1105 21:16:32.685] Nov 5 21:16:32.682: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-agentpool-97435445-vmss000000,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-agentpool-97435445-vmss000000,UID:e955dc47-8c8f-4142-9241-3d409f32e42d,ResourceVersion:13181,Generation:0,CreationTimestamp:2019-11-05 20:07:34 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{agentpool: agentpool,beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_D4s_v3,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 0,kubernetes.azure.com/cluster: kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b,kubernetes.azure.com/role: agent,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-agentpool-97435445-vmss000000,kubernetes.io/os: linux,kubernetes.io/role: agent,node-role.kubernetes.io/agent: ,storageprofile: managed,storagetier: Premium_LRS,topology.disk.csi.azure.com/zone: 0,},Annotations:map[string]string{csi.volume.kubernetes.io/nodeid: {"disk.csi.azure.com":"k8s-agentpool-97435445-vmss000000"},node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.2.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool-97435445-vmss/virtualMachines/0,Unschedulable:false,Taints:[],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-csi-disk.csi.azure.com: {{8 0} {} 8 DecimalSI},cpu: {{4 0} {} 4 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{16797040640 0} {} BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-csi-disk.csi.azure.com: {{8 0} {} 8 DecimalSI},cpu: {{4 0} {} 4 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{16010608640 0} {} BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 20:08:01 +0000 UTC 2019-11-05 20:08:01 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 21:15:46 +0000 UTC 2019-11-05 20:07:27 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 21:15:46 +0000 UTC 2019-11-05 20:07:27 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 21:15:46 +0000 UTC 2019-11-05 20:07:27 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 21:15:46 +0000 UTC 2019-11-05 20:07:55 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.0.4} {Hostname k8s-agentpool-97435445-vmss000000}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:bfb55325809340228149d2d93f50df58,SystemUUID:CB2ED036-0439-F641-BC7F-640C3638C027,BootID:a917f4d1-549d-471d-b7dc-47ae1715d97d,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[k8sprowinternal.azurecr.io/azuredisk-csi@sha256:75d4ff5d7cd845d6e810c67796ecb037e852fb849ec1eb5b3194564008e6d1e0 k8sprowinternal.azurecr.io/azuredisk-csi:e2e-3e3ed045c6faa509dbd8d2890ee9f56aa2934887] 286930414} {[k8s.gcr.io/kubernetes-dashboard-amd64@sha256:0ae6b69432e78069c5ce2bcde0fe409c5c4d6f0f4d9cd50a17974fea38898747 k8s.gcr.io/kubernetes-dashboard-amd64:v1.10.1] 121711221} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/metrics-server-amd64@sha256:49a9f12f7067d11f42c803dbe61ed2c1299959ad85cb315b25ff7eef8e6b8892 k8s.gcr.io/metrics-server-amd64:v0.2.1] 42541759} {[k8s.gcr.io/coredns@sha256:e83beb5e43f8513fa735e77ffc5859640baea30a882a11cc75c4c3244a737d3c k8s.gcr.io/coredns:1.5.0] 42488424} {[mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume@sha256:4fd30d43947d4a54fc89ead7985beecfd3c9b2a93a0655a373b1608ab90bd5af mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume:v0.0.7] 22909487} {[quay.io/k8scsi/livenessprobe@sha256:8c9c8a0f330acc2e558ae6624309447d02522e76aa35af23fa19b88abf4381cb quay.io/k8scsi/livenessprobe:v1.0.2] 17415381} {[quay.io/k8scsi/csi-node-driver-registrar@sha256:13daf82fb99e951a4bff8ae5fc7c17c3a8fe7130be6400990d8f6076c32d4599 quay.io/k8scsi/csi-node-driver-registrar:v1.1.0] 15815995} {[mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume@sha256:23d8c6033f02a1ecad05127ebdc931bb871264228661bc122704b0974e4d9fdd mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume:1.0.8] 1159025} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 21:16:32.685] Nov 5 21:16:32.682: INFO: I1105 21:16:32.685] Logging kubelet events for node k8s-agentpool-97435445-vmss000000 I1105 21:16:32.764] Nov 5 21:16:32.763: INFO: I1105 21:16:32.764] Logging pods the kubelet thinks is on node k8s-agentpool-97435445-vmss000000 I1105 21:16:32.999] Nov 5 21:16:32.999: INFO: azure-ip-masq-agent-m7tv9 started at 2019-11-05 20:08:01 +0000 UTC (0+1 container statuses recorded) I1105 21:16:32.999] Nov 5 21:16:32.999: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 21:16:33.000] Nov 5 21:16:32.999: INFO: keyvault-flexvolume-smspt started at 2019-11-05 20:08:01 +0000 UTC (0+1 container statuses recorded) I1105 21:16:33.000] Nov 5 21:16:32.999: INFO: Container keyvault-flexvolume ready: true, restart count 0 I1105 21:16:33.000] Nov 5 21:16:32.999: INFO: metrics-server-864ffbc5c-wl84z started at 2019-11-05 20:08:01 +0000 UTC (0+1 container statuses recorded) I1105 21:16:33.000] Nov 5 21:16:32.999: INFO: Container metrics-server ready: true, restart count 0 I1105 21:16:33.000] Nov 5 21:16:32.999: INFO: kube-proxy-8mt7f started at 2019-11-05 20:08:01 +0000 UTC (0+1 container statuses recorded) I1105 21:16:33.000] Nov 5 21:16:32.999: INFO: Container kube-proxy ready: true, restart count 0 I1105 21:16:33.001] Nov 5 21:16:32.999: INFO: kubernetes-dashboard-66dd8b8df7-t6j4l started at 2019-11-05 20:08:01 +0000 UTC (0+1 container statuses recorded) I1105 21:16:33.001] Nov 5 21:16:32.999: INFO: Container kubernetes-dashboard ready: true, restart count 0 I1105 21:16:33.001] Nov 5 21:16:32.999: INFO: csi-azuredisk-node-5wwhz started at 2019-11-05 20:16:09 +0000 UTC (0+3 container statuses recorded) I1105 21:16:33.001] Nov 5 21:16:32.999: INFO: Container azuredisk ready: true, restart count 0 I1105 21:16:33.001] Nov 5 21:16:32.999: INFO: Container liveness-probe ready: true, restart count 0 I1105 21:16:33.002] Nov 5 21:16:32.999: INFO: Container node-driver-registrar ready: true, restart count 0 I1105 21:16:33.002] Nov 5 21:16:32.999: INFO: blobfuse-flexvol-installer-p7hdb started at 2019-11-05 20:08:01 +0000 UTC (0+1 container statuses recorded) I1105 21:16:33.002] Nov 5 21:16:32.999: INFO: Container blobfuse-flexvol-installer ready: true, restart count 0 I1105 21:16:33.002] Nov 5 21:16:32.999: INFO: coredns-7f68dcdbdb-zh8g5 started at 2019-11-05 20:08:01 +0000 UTC (0+1 container statuses recorded) I1105 21:16:33.002] Nov 5 21:16:32.999: INFO: Container coredns ready: true, restart count 0 I1105 21:16:33.327] Nov 5 21:16:33.327: INFO: I1105 21:16:33.327] Latency metrics for node k8s-agentpool-97435445-vmss000000 I1105 21:16:33.327] Nov 5 21:16:33.327: INFO: I1105 21:16:33.327] Logging node info for node k8s-agentpool-97435445-vmss000001 I1105 21:16:33.409] Nov 5 21:16:33.406: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-agentpool-97435445-vmss000001,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-agentpool-97435445-vmss000001,UID:75d3cc50-ca6d-45e0-a42a-6090e8d537dd,ResourceVersion:13176,Generation:0,CreationTimestamp:2019-11-05 20:07:34 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{agentpool: agentpool,beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_D4s_v3,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 1,kubernetes.azure.com/cluster: kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b,kubernetes.azure.com/role: agent,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-agentpool-97435445-vmss000001,kubernetes.io/os: linux,kubernetes.io/role: agent,node-role.kubernetes.io/agent: ,storageprofile: managed,storagetier: Premium_LRS,topology.disk.csi.azure.com/zone: 1,},Annotations:map[string]string{csi.volume.kubernetes.io/nodeid: {"disk.csi.azure.com":"k8s-agentpool-97435445-vmss000001"},node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.0.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool-97435445-vmss/virtualMachines/1,Unschedulable:false,Taints:[],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-csi-disk.csi.azure.com: {{8 0} {} 8 DecimalSI},cpu: {{4 0} {} 4 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{16797044736 0} {} 16403364Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-csi-disk.csi.azure.com: {{8 0} {} 8 DecimalSI},cpu: {{4 0} {} 4 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{16010612736 0} {} 15635364Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 20:08:25 +0000 UTC 2019-11-05 20:08:25 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 21:15:45 +0000 UTC 2019-11-05 20:07:26 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 21:15:45 +0000 UTC 2019-11-05 20:07:26 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 21:15:45 +0000 UTC 2019-11-05 20:07:26 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 21:15:45 +0000 UTC 2019-11-05 20:07:55 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.0.5} {Hostname k8s-agentpool-97435445-vmss000001}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:e78def54b2f545c6b94d88bbba523c60,SystemUUID:40AE8188-86AB-D848-84A4-D9254E1258BC,BootID:8587ec06-f4f4-43c1-86cd-06c295b05b63,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[k8sprowinternal.azurecr.io/azuredisk-csi@sha256:75d4ff5d7cd845d6e810c67796ecb037e852fb849ec1eb5b3194564008e6d1e0 k8sprowinternal.azurecr.io/azuredisk-csi:e2e-3e3ed045c6faa509dbd8d2890ee9f56aa2934887] 286930414} {[gcr.io/kubernetes-helm/tiller@sha256:f6d8f4ab9ba993b5f5b60a6edafe86352eabe474ffeb84cb6c79b8866dce45d1 gcr.io/kubernetes-helm/tiller:v2.11.0] 71821984} {[quay.io/k8scsi/csi-provisioner@sha256:d657c839dce87324fe2b677302913f9386f885f8746be7bea0ced5b0844e3433 quay.io/k8scsi/csi-provisioner:v1.3.1] 50596998} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[quay.io/k8scsi/csi-attacher@sha256:26fccd7a99d973845df1193b46ebdcc6ab8dc5f6e6be319750c471fce1742d13 quay.io/k8scsi/csi-attacher:v1.2.0] 46226754} {[quay.io/k8scsi/csi-cluster-driver-registrar@sha256:fafd75ae5442f192cfa8c2e792903aee30d5884b62e802e4464b0a895d21e3ef quay.io/k8scsi/csi-cluster-driver-registrar:v1.0.1] 45874691} {[quay.io/k8scsi/csi-snapshotter@sha256:a49e0da1af6f2bf717e41ba1eee8b5e6a1cbd66a709dd92cc43fe475fe2589eb quay.io/k8scsi/csi-snapshotter:v1.1.0] 44325352} {[mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume@sha256:4fd30d43947d4a54fc89ead7985beecfd3c9b2a93a0655a373b1608ab90bd5af mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume:v0.0.7] 22909487} {[quay.io/k8scsi/livenessprobe@sha256:8c9c8a0f330acc2e558ae6624309447d02522e76aa35af23fa19b88abf4381cb quay.io/k8scsi/livenessprobe:v1.0.2] 17415381} {[quay.io/k8scsi/csi-node-driver-registrar@sha256:13daf82fb99e951a4bff8ae5fc7c17c3a8fe7130be6400990d8f6076c32d4599 quay.io/k8scsi/csi-node-driver-registrar:v1.1.0] 15815995} {[quay.io/k8scsi/livenessprobe@sha256:dde617756e0f602adc566ab71fd885f1dad451ad3fb063ac991c95a2ff47aea5 quay.io/k8scsi/livenessprobe:v1.1.0] 14967303} {[mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume@sha256:23d8c6033f02a1ecad05127ebdc931bb871264228661bc122704b0974e4d9fdd mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume:1.0.8] 1159025} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 21:16:33.409] Nov 5 21:16:33.406: INFO: I1105 21:16:33.409] Logging kubelet events for node k8s-agentpool-97435445-vmss000001 I1105 21:16:33.486] Nov 5 21:16:33.485: INFO: I1105 21:16:33.486] Logging pods the kubelet thinks is on node k8s-agentpool-97435445-vmss000001 I1105 21:16:33.719] Nov 5 21:16:33.719: INFO: csi-azuredisk-node-q2dmt started at 2019-11-05 20:16:09 +0000 UTC (0+3 container statuses recorded) I1105 21:16:33.719] Nov 5 21:16:33.719: INFO: Container azuredisk ready: true, restart count 0 I1105 21:16:33.719] Nov 5 21:16:33.719: INFO: Container liveness-probe ready: true, restart count 0 I1105 21:16:33.719] Nov 5 21:16:33.719: INFO: Container node-driver-registrar ready: true, restart count 0 I1105 21:16:33.720] Nov 5 21:16:33.719: INFO: csi-azuredisk-controller-6f6587bfc6-62nrt started at 2019-11-05 20:16:09 +0000 UTC (0+6 container statuses recorded) I1105 21:16:33.720] Nov 5 21:16:33.719: INFO: Container azuredisk ready: true, restart count 0 I1105 21:16:33.720] Nov 5 21:16:33.719: INFO: Container cluster-driver-registrar ready: true, restart count 0 I1105 21:16:33.720] Nov 5 21:16:33.719: INFO: Container csi-attacher ready: true, restart count 0 I1105 21:16:33.720] Nov 5 21:16:33.719: INFO: Container csi-provisioner ready: true, restart count 0 I1105 21:16:33.720] Nov 5 21:16:33.719: INFO: Container csi-snapshotter ready: true, restart count 0 I1105 21:16:33.721] Nov 5 21:16:33.719: INFO: Container liveness-probe ready: true, restart count 0 I1105 21:16:33.721] Nov 5 21:16:33.719: INFO: blobfuse-flexvol-installer-j4s7t started at 2019-11-05 20:07:59 +0000 UTC (0+1 container statuses recorded) I1105 21:16:33.721] Nov 5 21:16:33.719: INFO: Container blobfuse-flexvol-installer ready: true, restart count 0 I1105 21:16:33.721] Nov 5 21:16:33.719: INFO: azure-ip-masq-agent-gdhcq started at 2019-11-05 20:07:59 +0000 UTC (0+1 container statuses recorded) I1105 21:16:33.721] Nov 5 21:16:33.719: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 21:16:33.722] Nov 5 21:16:33.719: INFO: keyvault-flexvolume-6bffx started at 2019-11-05 20:07:59 +0000 UTC (0+1 container statuses recorded) I1105 21:16:33.722] Nov 5 21:16:33.719: INFO: Container keyvault-flexvolume ready: true, restart count 0 I1105 21:16:33.722] Nov 5 21:16:33.719: INFO: kube-proxy-kjrpb started at 2019-11-05 20:08:00 +0000 UTC (0+1 container statuses recorded) I1105 21:16:33.722] Nov 5 21:16:33.719: INFO: Container kube-proxy ready: true, restart count 0 I1105 21:16:33.723] Nov 5 21:16:33.719: INFO: tiller-deploy-54c96cb5df-pt7x6 started at 2019-11-05 20:08:00 +0000 UTC (0+1 container statuses recorded) I1105 21:16:33.723] Nov 5 21:16:33.719: INFO: Container tiller ready: true, restart count 0 I1105 21:16:34.056] Nov 5 21:16:34.055: INFO: I1105 21:16:34.056] Latency metrics for node k8s-agentpool-97435445-vmss000001 I1105 21:16:34.056] Nov 5 21:16:34.055: INFO: I1105 21:16:34.056] Logging node info for node k8s-master-97435445-0 I1105 21:16:34.143] Nov 5 21:16:34.141: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-97435445-0,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-97435445-0,UID:df3ae36f-961c-4729-a8f5-34cda54f4ff9,ResourceVersion:13239,Generation:0,CreationTimestamp:2019-11-05 20:08:36 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 2,kubernetes.azure.com/cluster: kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-97435445-0,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.4.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b/providers/Microsoft.Compute/virtualMachines/k8s-master-97435445-0,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 20:08:46 +0000 UTC 2019-11-05 20:08:46 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 21:16:06 +0000 UTC 2019-11-05 20:08:28 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 21:16:06 +0000 UTC 2019-11-05 20:08:28 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 21:16:06 +0000 UTC 2019-11-05 20:08:28 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 21:16:06 +0000 UTC 2019-11-05 20:08:36 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.255.5} {Hostname k8s-master-97435445-0}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:3c381865192341c2868672e8626cef7d,SystemUUID:F3D259D3-7EB8-B844-A1F0-F954DCD0D76B,BootID:4948329e-cf57-48c3-aa83-5297bef70370,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 21:16:34.144] Nov 5 21:16:34.141: INFO: I1105 21:16:34.144] Logging kubelet events for node k8s-master-97435445-0 I1105 21:16:34.222] Nov 5 21:16:34.222: INFO: I1105 21:16:34.222] Logging pods the kubelet thinks is on node k8s-master-97435445-0 I1105 21:16:34.381] Nov 5 21:16:34.381: INFO: azure-ip-masq-agent-q6dcl started at 2019-11-05 20:08:38 +0000 UTC (0+1 container statuses recorded) I1105 21:16:34.382] Nov 5 21:16:34.381: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 21:16:34.382] Nov 5 21:16:34.381: INFO: kube-proxy-5wnd2 started at 2019-11-05 20:08:38 +0000 UTC (0+1 container statuses recorded) I1105 21:16:34.382] Nov 5 21:16:34.381: INFO: Container kube-proxy ready: true, restart count 0 I1105 21:16:34.382] Nov 5 21:16:34.381: INFO: cloud-controller-manager-k8s-master-97435445-0 started at 2019-11-05 20:08:22 +0000 UTC (0+1 container statuses recorded) I1105 21:16:34.383] Nov 5 21:16:34.381: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 21:16:34.383] Nov 5 21:16:34.381: INFO: kube-addon-manager-k8s-master-97435445-0 started at 2019-11-05 20:08:48 +0000 UTC (0+1 container statuses recorded) I1105 21:16:34.383] Nov 5 21:16:34.381: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 21:16:34.383] Nov 5 21:16:34.381: INFO: kube-apiserver-k8s-master-97435445-0 started at 2019-11-05 20:08:22 +0000 UTC (0+1 container statuses recorded) I1105 21:16:34.383] Nov 5 21:16:34.381: INFO: Container kube-apiserver ready: true, restart count 0 I1105 21:16:34.383] Nov 5 21:16:34.381: INFO: kube-controller-manager-k8s-master-97435445-0 started at 2019-11-05 20:08:22 +0000 UTC (0+1 container statuses recorded) I1105 21:16:34.383] Nov 5 21:16:34.381: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 21:16:34.384] Nov 5 21:16:34.381: INFO: kube-scheduler-k8s-master-97435445-0 started at 2019-11-05 20:08:22 +0000 UTC (0+1 container statuses recorded) I1105 21:16:34.384] Nov 5 21:16:34.381: INFO: Container kube-scheduler ready: true, restart count 0 I1105 21:16:34.713] Nov 5 21:16:34.713: INFO: I1105 21:16:34.713] Latency metrics for node k8s-master-97435445-0 I1105 21:16:34.713] Nov 5 21:16:34.713: INFO: I1105 21:16:34.713] Logging node info for node k8s-master-97435445-1 I1105 21:16:34.794] Nov 5 21:16:34.792: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-97435445-1,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-97435445-1,UID:05534a67-d493-4ab5-9b0f-5ff5490e234f,ResourceVersion:13147,Generation:0,CreationTimestamp:2019-11-05 20:07:34 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 0,kubernetes.azure.com/cluster: kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-97435445-1,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.1.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b/providers/Microsoft.Compute/virtualMachines/k8s-master-97435445-1,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 20:08:26 +0000 UTC 2019-11-05 20:08:26 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 21:15:35 +0000 UTC 2019-11-05 20:07:28 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 21:15:35 +0000 UTC 2019-11-05 20:07:28 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 21:15:35 +0000 UTC 2019-11-05 20:07:28 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 21:15:35 +0000 UTC 2019-11-05 20:07:55 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{Hostname k8s-master-97435445-1} {InternalIP 10.240.255.6}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:079b605103b74acabac18fd2ea63368a,SystemUUID:38220EEC-BEFF-4C46-AA05-5A214AAFDC28,BootID:b1b45df0-d925-4bcc-9830-0202df36ebb4,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 21:16:34.794] Nov 5 21:16:34.792: INFO: I1105 21:16:34.794] Logging kubelet events for node k8s-master-97435445-1 I1105 21:16:34.873] Nov 5 21:16:34.873: INFO: I1105 21:16:34.873] Logging pods the kubelet thinks is on node k8s-master-97435445-1 I1105 21:16:35.031] Nov 5 21:16:35.031: INFO: cloud-controller-manager-k8s-master-97435445-1 started at 2019-11-05 20:07:22 +0000 UTC (0+1 container statuses recorded) I1105 21:16:35.031] Nov 5 21:16:35.031: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 21:16:35.031] Nov 5 21:16:35.031: INFO: kube-addon-manager-k8s-master-97435445-1 started at 2019-11-05 20:07:22 +0000 UTC (0+1 container statuses recorded) I1105 21:16:35.031] Nov 5 21:16:35.031: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 21:16:35.032] Nov 5 21:16:35.031: INFO: kube-apiserver-k8s-master-97435445-1 started at 2019-11-05 20:07:22 +0000 UTC (0+1 container statuses recorded) I1105 21:16:35.032] Nov 5 21:16:35.031: INFO: Container kube-apiserver ready: true, restart count 0 I1105 21:16:35.032] Nov 5 21:16:35.031: INFO: kube-controller-manager-k8s-master-97435445-1 started at 2019-11-05 20:07:22 +0000 UTC (0+1 container statuses recorded) I1105 21:16:35.032] Nov 5 21:16:35.031: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 21:16:35.032] Nov 5 21:16:35.031: INFO: kube-scheduler-k8s-master-97435445-1 started at 2019-11-05 20:07:22 +0000 UTC (0+1 container statuses recorded) I1105 21:16:35.032] Nov 5 21:16:35.031: INFO: Container kube-scheduler ready: true, restart count 0 I1105 21:16:35.032] Nov 5 21:16:35.031: INFO: azure-ip-masq-agent-n29bm started at 2019-11-05 20:07:59 +0000 UTC (0+1 container statuses recorded) I1105 21:16:35.033] Nov 5 21:16:35.031: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 21:16:35.033] Nov 5 21:16:35.031: INFO: kube-proxy-h4s2c started at 2019-11-05 20:08:00 +0000 UTC (0+1 container statuses recorded) I1105 21:16:35.033] Nov 5 21:16:35.031: INFO: Container kube-proxy ready: true, restart count 0 I1105 21:16:35.367] Nov 5 21:16:35.367: INFO: I1105 21:16:35.367] Latency metrics for node k8s-master-97435445-1 I1105 21:16:35.367] Nov 5 21:16:35.367: INFO: I1105 21:16:35.367] Logging node info for node k8s-master-97435445-2 I1105 21:16:35.449] Nov 5 21:16:35.446: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-97435445-2,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-97435445-2,UID:fa55c895-1a93-4743-885e-03fce84b5332,ResourceVersion:13196,Generation:0,CreationTimestamp:2019-11-05 20:09:19 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 0,kubernetes.azure.com/cluster: kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-97435445-2,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.5.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b/providers/Microsoft.Compute/virtualMachines/k8s-master-97435445-2,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 20:09:19 +0000 UTC 2019-11-05 20:09:19 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 21:15:51 +0000 UTC 2019-11-05 20:09:12 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 21:15:51 +0000 UTC 2019-11-05 20:09:12 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 21:15:51 +0000 UTC 2019-11-05 20:09:12 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 21:15:51 +0000 UTC 2019-11-05 20:09:19 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.255.7} {Hostname k8s-master-97435445-2}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:d19f6709138f496da43c095cec7b2360,SystemUUID:10116777-A8D7-CB4A-A8B7-7EBCF2F23B62,BootID:50cd14a9-5497-4972-bf60-cf59a0cbcb5f,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 21:16:35.449] Nov 5 21:16:35.447: INFO: I1105 21:16:35.449] Logging kubelet events for node k8s-master-97435445-2 I1105 21:16:35.527] Nov 5 21:16:35.526: INFO: I1105 21:16:35.527] Logging pods the kubelet thinks is on node k8s-master-97435445-2 I1105 21:16:35.685] Nov 5 21:16:35.685: INFO: kube-controller-manager-k8s-master-97435445-2 started at 2019-11-05 20:09:05 +0000 UTC (0+1 container statuses recorded) I1105 21:16:35.686] Nov 5 21:16:35.685: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 21:16:35.686] Nov 5 21:16:35.685: INFO: kube-scheduler-k8s-master-97435445-2 started at 2019-11-05 20:09:05 +0000 UTC (0+1 container statuses recorded) I1105 21:16:35.686] Nov 5 21:16:35.685: INFO: Container kube-scheduler ready: true, restart count 0 I1105 21:16:35.686] Nov 5 21:16:35.685: INFO: cloud-controller-manager-k8s-master-97435445-2 started at 2019-11-05 20:09:05 +0000 UTC (0+1 container statuses recorded) I1105 21:16:35.686] Nov 5 21:16:35.685: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 21:16:35.686] Nov 5 21:16:35.685: INFO: kube-addon-manager-k8s-master-97435445-2 started at 2019-11-05 20:09:05 +0000 UTC (0+1 container statuses recorded) I1105 21:16:35.687] Nov 5 21:16:35.685: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 21:16:35.687] Nov 5 21:16:35.685: INFO: azure-ip-masq-agent-sq7p5 started at 2019-11-05 20:09:19 +0000 UTC (0+1 container statuses recorded) I1105 21:16:35.687] Nov 5 21:16:35.685: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 21:16:35.687] Nov 5 21:16:35.685: INFO: kube-proxy-pz426 started at 2019-11-05 20:09:19 +0000 UTC (0+1 container statuses recorded) I1105 21:16:35.687] Nov 5 21:16:35.685: INFO: Container kube-proxy ready: true, restart count 0 I1105 21:16:35.687] Nov 5 21:16:35.685: INFO: kube-apiserver-k8s-master-97435445-2 started at 2019-11-05 20:09:05 +0000 UTC (0+1 container statuses recorded) I1105 21:16:35.687] Nov 5 21:16:35.685: INFO: Container kube-apiserver ready: true, restart count 0 I1105 21:16:36.017] Nov 5 21:16:36.017: INFO: I1105 21:16:36.018] Latency metrics for node k8s-master-97435445-2 I1105 21:16:36.018] Nov 5 21:16:36.017: INFO: I1105 21:16:36.018] Logging node info for node k8s-master-97435445-3 I1105 21:16:36.099] Nov 5 21:16:36.097: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-97435445-3,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-97435445-3,UID:8ffd1d0d-4487-4680-a2f6-e724387be637,ResourceVersion:13195,Generation:0,CreationTimestamp:2019-11-05 20:09:21 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 1,kubernetes.azure.com/cluster: kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-97435445-3,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.6.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b/providers/Microsoft.Compute/virtualMachines/k8s-master-97435445-3,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 20:09:39 +0000 UTC 2019-11-05 20:09:39 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 21:15:50 +0000 UTC 2019-11-05 20:09:18 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 21:15:50 +0000 UTC 2019-11-05 20:09:18 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 21:15:50 +0000 UTC 2019-11-05 20:09:18 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 21:15:50 +0000 UTC 2019-11-05 20:09:21 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.255.8} {Hostname k8s-master-97435445-3}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:266064173bfb4297898c6c25769e8b20,SystemUUID:E6E44241-5015-7740-9A81-A668A8427206,BootID:579d658c-b0ad-47f1-a993-1ccea2b55789,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 21:16:36.099] Nov 5 21:16:36.097: INFO: I1105 21:16:36.100] Logging kubelet events for node k8s-master-97435445-3 I1105 21:16:36.177] Nov 5 21:16:36.177: INFO: I1105 21:16:36.177] Logging pods the kubelet thinks is on node k8s-master-97435445-3 I1105 21:16:36.336] Nov 5 21:16:36.335: INFO: kube-addon-manager-k8s-master-97435445-3 started at 2019-11-05 20:09:05 +0000 UTC (0+1 container statuses recorded) I1105 21:16:36.336] Nov 5 21:16:36.335: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 21:16:36.336] Nov 5 21:16:36.335: INFO: kube-apiserver-k8s-master-97435445-3 started at 2019-11-05 20:09:05 +0000 UTC (0+1 container statuses recorded) I1105 21:16:36.336] Nov 5 21:16:36.335: INFO: Container kube-apiserver ready: true, restart count 0 I1105 21:16:36.337] Nov 5 21:16:36.335: INFO: kube-controller-manager-k8s-master-97435445-3 started at 2019-11-05 20:09:05 +0000 UTC (0+1 container statuses recorded) I1105 21:16:36.337] Nov 5 21:16:36.335: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 21:16:36.337] Nov 5 21:16:36.335: INFO: kube-scheduler-k8s-master-97435445-3 started at 2019-11-05 20:09:05 +0000 UTC (0+1 container statuses recorded) I1105 21:16:36.337] Nov 5 21:16:36.335: INFO: Container kube-scheduler ready: true, restart count 0 I1105 21:16:36.338] Nov 5 21:16:36.335: INFO: azure-ip-masq-agent-747cl started at 2019-11-05 20:09:21 +0000 UTC (0+1 container statuses recorded) I1105 21:16:36.338] Nov 5 21:16:36.335: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 21:16:36.338] Nov 5 21:16:36.335: INFO: kube-proxy-k45g8 started at 2019-11-05 20:09:21 +0000 UTC (0+1 container statuses recorded) I1105 21:16:36.338] Nov 5 21:16:36.335: INFO: Container kube-proxy ready: true, restart count 0 I1105 21:16:36.339] Nov 5 21:16:36.335: INFO: cloud-controller-manager-k8s-master-97435445-3 started at 2019-11-05 20:09:04 +0000 UTC (0+1 container statuses recorded) I1105 21:16:36.339] Nov 5 21:16:36.335: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 21:16:36.594] Nov 5 21:16:36.594: INFO: I1105 21:16:36.594] Latency metrics for node k8s-master-97435445-3 I1105 21:16:36.594] Nov 5 21:16:36.594: INFO: I1105 21:16:36.594] Logging node info for node k8s-master-97435445-4 I1105 21:16:36.675] Nov 5 21:16:36.673: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-97435445-4,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-97435445-4,UID:6d547b41-3f92-4cc4-a2d7-dda2071776a2,ResourceVersion:13237,Generation:0,CreationTimestamp:2019-11-05 20:08:36 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 1,kubernetes.azure.com/cluster: kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-97435445-4,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.3.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b/providers/Microsoft.Compute/virtualMachines/k8s-master-97435445-4,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 20:08:59 +0000 UTC 2019-11-05 20:08:59 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 21:16:05 +0000 UTC 2019-11-05 20:08:29 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 21:16:05 +0000 UTC 2019-11-05 20:08:29 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 21:16:05 +0000 UTC 2019-11-05 20:08:29 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 21:16:05 +0000 UTC 2019-11-05 20:08:36 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.255.9} {Hostname k8s-master-97435445-4}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:56b393c61eb34ba49ebda5279eeedb30,SystemUUID:FD239C23-618C-0049-8734-9AF3FDC9E3C1,BootID:f342ea44-e719-4513-807d-5def7ac606e2,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 21:16:36.675] Nov 5 21:16:36.673: INFO: I1105 21:16:36.676] Logging kubelet events for node k8s-master-97435445-4 I1105 21:16:36.758] Nov 5 21:16:36.758: INFO: I1105 21:16:36.758] Logging pods the kubelet thinks is on node k8s-master-97435445-4 I1105 21:16:36.842] Nov 5 21:16:36.842: INFO: kube-scheduler-k8s-master-97435445-4 started at 2019-11-05 20:08:23 +0000 UTC (0+1 container statuses recorded) I1105 21:16:36.843] Nov 5 21:16:36.842: INFO: Container kube-scheduler ready: true, restart count 0 I1105 21:16:36.843] Nov 5 21:16:36.842: INFO: kube-proxy-vfbfr started at 2019-11-05 20:08:36 +0000 UTC (0+1 container statuses recorded) I1105 21:16:36.843] Nov 5 21:16:36.842: INFO: Container kube-proxy ready: true, restart count 0 I1105 21:16:36.843] Nov 5 21:16:36.842: INFO: azure-ip-masq-agent-kmkll started at 2019-11-05 20:08:36 +0000 UTC (0+1 container statuses recorded) I1105 21:16:36.843] Nov 5 21:16:36.842: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 21:16:36.844] Nov 5 21:16:36.842: INFO: cloud-controller-manager-k8s-master-97435445-4 started at 2019-11-05 20:08:46 +0000 UTC (0+1 container statuses recorded) I1105 21:16:36.844] Nov 5 21:16:36.842: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 21:16:36.844] Nov 5 21:16:36.842: INFO: kube-addon-manager-k8s-master-97435445-4 started at 2019-11-05 20:08:23 +0000 UTC (0+1 container statuses recorded) I1105 21:16:36.844] Nov 5 21:16:36.842: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 21:16:36.844] Nov 5 21:16:36.842: INFO: kube-apiserver-k8s-master-97435445-4 started at 2019-11-05 20:08:23 +0000 UTC (0+1 container statuses recorded) I1105 21:16:36.844] Nov 5 21:16:36.842: INFO: Container kube-apiserver ready: true, restart count 0 I1105 21:16:36.844] Nov 5 21:16:36.842: INFO: kube-controller-manager-k8s-master-97435445-4 started at 2019-11-05 20:08:23 +0000 UTC (0+1 container statuses recorded) I1105 21:16:36.845] Nov 5 21:16:36.842: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 21:16:37.104] Nov 5 21:16:37.104: INFO: I1105 21:16:37.104] Latency metrics for node k8s-master-97435445-4 I1105 21:16:37.104] Nov 5 21:16:37.104: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready I1105 21:16:37.183] STEP: Destroying namespace "azuredisk-3372" for this suite. I1105 21:16:51.427] Nov 5 21:16:51.426: INFO: Waiting up to 30s for server preferred namespaced resources to be successfully discovered I1105 21:16:54.527] Nov 5 21:16:54.527: INFO: namespace azuredisk-3372 deletion completed in 17.343631664s I1105 21:16:54.527] I1105 21:16:54.527] • Failure [323.759 seconds] I1105 21:16:54.528] [azuredisk-csi-e2e] Dynamic Provisioning I1105 21:16:54.528] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:33 I1105 21:16:54.528] [single-az] I1105 21:16:54.528] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:36 I1105 21:16:54.528] should create a volume on demand [It] I1105 21:16:54.528] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:78 I1105 21:16:54.528] I1105 21:16:54.528] Unexpected error: I1105 21:16:54.529] <*errors.errorString | 0xc000697090>: { I1105 21:16:54.529] s: "PersistentVolumeClaims [pvc-n9hsv] not all in phase Bound within 5m0s", I1105 21:16:54.529] } I1105 21:16:54.529] PersistentVolumeClaims [pvc-n9hsv] not all in phase Bound within 5m0s I1105 21:16:54.529] occurred I1105 21:16:54.529] I1105 21:16:54.529] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/testsuites/testsuites.go:278 I1105 21:16:54.529] ------------------------------ I1105 21:16:54.529] [azuredisk-csi-e2e] [single-az] Pre-Provisioned I1105 21:16:54.530] [env] should use a pre-provisioned volume and mount it as readOnly in a pod I1105 21:16:54.530] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/pre_provisioning.go:66 I1105 21:16:54.530] [BeforeEach] [azuredisk-csi-e2e] [single-az] Pre-Provisioned I1105 21:16:54.530] /go/src/sigs.k8s.io/azuredisk-csi-driver/vendor/k8s.io/kubernetes/test/e2e/framework/framework.go:150 I1105 21:16:54.530] STEP: Creating a kubernetes client I1105 21:16:54.530] Nov 5 21:16:54.527: INFO: >>> kubeConfig: /workspace/aks971561099/kubeconfig/kubeconfig.eastus2.json I1105 21:16:54.530] STEP: Building a namespace api object, basename azuredisk I1105 21:16:54.781] STEP: Binding the e2e-test-privileged-psp PodSecurityPolicy to the default service account in azuredisk-5349 I1105 21:16:55.055] STEP: Waiting for a default service account to be provisioned in namespace I1105 21:16:55.131] [BeforeEach] [azuredisk-csi-e2e] [single-az] Pre-Provisioned I1105 21:16:55.131] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/pre_provisioning.go:48 I1105 21:16:55.131] [It] [env] should use a pre-provisioned volume and mount it as readOnly in a pod I1105 21:16:55.132] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/pre_provisioning.go:66 I1105 21:17:06.986] STEP: Successfully provisioned AzureDisk volume: "/subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b/providers/Microsoft.Compute/disks/pre-provisioned-readOnly" I1105 21:17:06.986] I1105 21:17:06.986] STEP: setting up the PV I1105 21:17:06.986] STEP: creating a PV I1105 21:17:07.079] STEP: setting up the PVC I1105 21:17:07.080] STEP: creating a PVC I1105 21:17:07.170] STEP: waiting for PVC to be in phase "Bound" I1105 21:17:07.171] Nov 5 21:17:07.170: INFO: Waiting up to 5m0s for PersistentVolumeClaims [pvc-xkxzt] to have phase Bound I1105 21:17:07.254] Nov 5 21:17:07.254: INFO: PersistentVolumeClaim pvc-xkxzt found and phase=Bound (83.536933ms) I1105 21:17:07.254] STEP: checking the PVC I1105 21:17:07.338] STEP: validating provisioned PV I1105 21:17:07.429] STEP: checking the PV I1105 21:17:07.429] STEP: deploying the pod I1105 21:17:07.522] STEP: checking that the pods command exits with an error I1105 21:17:07.522] Nov 5 21:17:07.522: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-gxg4d" in namespace "azuredisk-5349" to be "Error status code" I1105 21:17:07.606] Nov 5 21:17:07.605: INFO: Pod "azuredisk-volume-tester-gxg4d": Phase="Pending", Reason="", readiness=false. Elapsed: 83.639334ms I1105 21:17:09.690] Nov 5 21:17:09.690: INFO: Pod "azuredisk-volume-tester-gxg4d": Phase="Pending", Reason="", readiness=false. Elapsed: 2.168057746s I1105 21:17:11.774] Nov 5 21:17:11.774: INFO: Pod "azuredisk-volume-tester-gxg4d": Phase="Pending", Reason="", readiness=false. Elapsed: 4.252400256s I1105 21:17:13.858] Nov 5 21:17:13.858: INFO: Pod "azuredisk-volume-tester-gxg4d": Phase="Pending", Reason="", readiness=false. Elapsed: 6.336531165s I1105 21:17:15.944] Nov 5 21:17:15.943: INFO: Pod "azuredisk-volume-tester-gxg4d": Phase="Pending", Reason="", readiness=false. Elapsed: 8.421717582s I1105 21:17:18.032] Nov 5 21:17:18.032: INFO: Pod "azuredisk-volume-tester-gxg4d": Phase="Pending", Reason="", readiness=false. Elapsed: 10.509912522s I1105 21:17:20.116] Nov 5 21:17:20.116: INFO: Pod "azuredisk-volume-tester-gxg4d": Phase="Pending", Reason="", readiness=false. Elapsed: 12.594122131s I1105 21:17:22.223] Nov 5 21:17:22.222: INFO: Pod "azuredisk-volume-tester-gxg4d": Phase="Pending", Reason="", readiness=false. Elapsed: 14.700767911s I1105 21:17:24.307] Nov 5 21:17:24.306: INFO: Pod "azuredisk-volume-tester-gxg4d": Phase="Pending", Reason="", readiness=false. Elapsed: 16.784674618s I1105 21:17:26.391] Nov 5 21:17:26.391: INFO: Pod "azuredisk-volume-tester-gxg4d": Phase="Pending", Reason="", readiness=false. Elapsed: 18.868894328s I1105 21:17:28.475] Nov 5 21:17:28.475: INFO: Pod "azuredisk-volume-tester-gxg4d": Phase="Pending", Reason="", readiness=false. Elapsed: 20.953177637s I1105 21:17:30.560] Nov 5 21:17:30.560: INFO: Pod "azuredisk-volume-tester-gxg4d": Phase="Pending", Reason="", readiness=false. Elapsed: 23.037932251s I1105 21:17:32.645] Nov 5 21:17:32.645: INFO: Pod "azuredisk-volume-tester-gxg4d": Phase="Pending", Reason="", readiness=false. Elapsed: 25.122820065s I1105 21:17:34.729] Nov 5 21:17:34.729: INFO: Pod "azuredisk-volume-tester-gxg4d": Phase="Pending", Reason="", readiness=false. Elapsed: 27.207263876s I1105 21:17:36.814] Nov 5 21:17:36.813: INFO: Pod "azuredisk-volume-tester-gxg4d": Phase="Pending", Reason="", readiness=false. Elapsed: 29.291545685s I1105 21:17:38.898] Nov 5 21:17:38.898: INFO: Pod "azuredisk-volume-tester-gxg4d": Phase="Pending", Reason="", readiness=false. Elapsed: 31.376127597s I1105 21:17:40.983] Nov 5 21:17:40.983: INFO: Pod "azuredisk-volume-tester-gxg4d": Phase="Pending", Reason="", readiness=false. Elapsed: 33.46087401s I1105 21:17:43.067] Nov 5 21:17:43.067: INFO: Pod "azuredisk-volume-tester-gxg4d": Phase="Pending", Reason="", readiness=false. Elapsed: 35.544953618s I1105 21:17:45.151] Nov 5 21:17:45.151: INFO: Pod "azuredisk-volume-tester-gxg4d": Phase="Pending", Reason="", readiness=false. Elapsed: 37.629114327s I1105 21:17:47.235] Nov 5 21:17:47.234: INFO: Pod "azuredisk-volume-tester-gxg4d": Phase="Pending", Reason="", readiness=false. Elapsed: 39.712712131s I1105 21:17:49.319] Nov 5 21:17:49.319: INFO: Pod "azuredisk-volume-tester-gxg4d": Phase="Pending", Reason="", readiness=false. Elapsed: 41.797104041s I1105 21:17:51.412] Nov 5 21:17:51.411: INFO: Pod "azuredisk-volume-tester-gxg4d": Phase="Pending", Reason="", readiness=false. Elapsed: 43.889796714s I1105 21:17:53.497] Nov 5 21:17:53.496: INFO: Pod "azuredisk-volume-tester-gxg4d": Phase="Pending", Reason="", readiness=false. Elapsed: 45.974641428s I1105 21:17:55.581] Nov 5 21:17:55.581: INFO: Pod "azuredisk-volume-tester-gxg4d": Phase="Pending", Reason="", readiness=false. Elapsed: 48.058949937s I1105 21:17:57.666] Nov 5 21:17:57.666: INFO: Pod "azuredisk-volume-tester-gxg4d": Phase="Pending", Reason="", readiness=false. Elapsed: 50.143948351s I1105 21:17:59.750] Nov 5 21:17:59.750: INFO: Pod "azuredisk-volume-tester-gxg4d": Phase="Pending", Reason="", readiness=false. Elapsed: 52.228025759s I1105 21:18:01.835] Nov 5 21:18:01.834: INFO: Pod "azuredisk-volume-tester-gxg4d": Phase="Pending", Reason="", readiness=false. Elapsed: 54.31261607s I1105 21:18:03.919] Nov 5 21:18:03.919: INFO: Pod "azuredisk-volume-tester-gxg4d": Phase="Pending", Reason="", readiness=false. Elapsed: 56.397363583s I1105 21:18:06.004] Nov 5 21:18:06.004: INFO: Pod "azuredisk-volume-tester-gxg4d": Phase="Pending", Reason="", readiness=false. Elapsed: 58.481840093s I1105 21:18:08.088] Nov 5 21:18:08.088: INFO: Pod "azuredisk-volume-tester-gxg4d": Phase="Pending", Reason="", readiness=false. Elapsed: 1m0.566144702s I1105 21:18:10.219] Nov 5 21:18:10.218: INFO: Pod "azuredisk-volume-tester-gxg4d": Phase="Pending", Reason="", readiness=false. Elapsed: 1m2.696560961s I1105 21:18:12.303] Nov 5 21:18:12.303: INFO: Pod "azuredisk-volume-tester-gxg4d": Phase="Pending", Reason="", readiness=false. Elapsed: 1m4.781284973s I1105 21:18:14.388] Nov 5 21:18:14.388: INFO: Pod "azuredisk-volume-tester-gxg4d": Phase="Pending", Reason="", readiness=false. Elapsed: 1m6.865894884s I1105 21:18:16.472] Nov 5 21:18:16.472: INFO: Pod "azuredisk-volume-tester-gxg4d": Phase="Pending", Reason="", readiness=false. Elapsed: 1m8.950381495s I1105 21:18:18.557] Nov 5 21:18:18.556: INFO: Pod "azuredisk-volume-tester-gxg4d": Phase="Pending", Reason="", readiness=false. Elapsed: 1m11.034798704s I1105 21:18:20.641] Nov 5 21:18:20.641: INFO: Pod "azuredisk-volume-tester-gxg4d": Phase="Pending", Reason="", readiness=false. Elapsed: 1m13.119370815s I1105 21:18:22.726] Nov 5 21:18:22.725: INFO: Pod "azuredisk-volume-tester-gxg4d": Phase="Failed", Reason="", readiness=false. Elapsed: 1m15.203703724s I1105 21:18:22.726] STEP: Saw pod failure I1105 21:18:22.726] Nov 5 21:18:22.725: INFO: Pod "azuredisk-volume-tester-gxg4d" satisfied condition "Error status code" I1105 21:18:22.726] STEP: checking that pod logs contain expected message I1105 21:18:22.816] Nov 5 21:18:22.816: INFO: deleting Pod "azuredisk-5349"/"azuredisk-volume-tester-gxg4d" I1105 21:18:22.905] Nov 5 21:18:22.905: INFO: Pod azuredisk-volume-tester-gxg4d has the following logs: /bin/sh: can't create /mnt/test-1/data: Read-only file system I1105 21:18:22.905] I1105 21:18:22.906] STEP: Deleting pod azuredisk-volume-tester-gxg4d in namespace azuredisk-5349 I1105 21:18:23.009] STEP: deleting PV "azuredisk-5349-disk.csi.azure.com-preprovsioned-pv-xb772" I1105 21:18:23.009] Nov 5 21:18:23.009: INFO: Deleting PersistentVolume "azuredisk-5349-disk.csi.azure.com-preprovsioned-pv-xb772" I1105 21:18:23.101] STEP: waiting for claim's PV "azuredisk-5349-disk.csi.azure.com-preprovsioned-pv-xb772" to be deleted I1105 21:18:23.101] Nov 5 21:18:23.101: INFO: Waiting up to 10m0s for PersistentVolume azuredisk-5349-disk.csi.azure.com-preprovsioned-pv-xb772 to get deleted I1105 21:18:23.185] Nov 5 21:18:23.184: INFO: PersistentVolume azuredisk-5349-disk.csi.azure.com-preprovsioned-pv-xb772 found and phase=Bound (83.826136ms) I1105 21:18:28.270] Nov 5 21:18:28.269: INFO: PersistentVolume azuredisk-5349-disk.csi.azure.com-preprovsioned-pv-xb772 found and phase=Bound (5.168773903s) I1105 21:18:33.354] Nov 5 21:18:33.354: INFO: PersistentVolume azuredisk-5349-disk.csi.azure.com-preprovsioned-pv-xb772 found and phase=Bound (10.253230567s) I1105 21:18:38.439] Nov 5 21:18:38.439: INFO: PersistentVolume azuredisk-5349-disk.csi.azure.com-preprovsioned-pv-xb772 found and phase=Bound (15.338281334s) I1105 21:18:43.523] Nov 5 21:18:43.523: INFO: PersistentVolume azuredisk-5349-disk.csi.azure.com-preprovsioned-pv-xb772 found and phase=Bound (20.422515195s) I1105 21:18:48.608] Nov 5 21:18:48.608: INFO: PersistentVolume azuredisk-5349-disk.csi.azure.com-preprovsioned-pv-xb772 found and phase=Bound (25.507025958s) I1105 21:18:53.698] Nov 5 21:18:53.698: INFO: PersistentVolume azuredisk-5349-disk.csi.azure.com-preprovsioned-pv-xb772 found and phase=Bound (30.597325965s) I1105 21:18:58.783] Nov 5 21:18:58.783: INFO: PersistentVolume azuredisk-5349-disk.csi.azure.com-preprovsioned-pv-xb772 found and phase=Bound (35.682320831s) I1105 21:19:03.867] Nov 5 21:19:03.867: INFO: PersistentVolume azuredisk-5349-disk.csi.azure.com-preprovsioned-pv-xb772 found and phase=Bound (40.766710592s) I1105 21:19:08.971] Nov 5 21:19:08.971: INFO: PersistentVolume azuredisk-5349-disk.csi.azure.com-preprovsioned-pv-xb772 found and phase=Bound (45.8704483s) I1105 21:19:14.055] Nov 5 21:19:14.055: INFO: PersistentVolume azuredisk-5349-disk.csi.azure.com-preprovsioned-pv-xb772 found and phase=Bound (50.954640459s) I1105 21:19:19.140] Nov 5 21:19:19.140: INFO: PersistentVolume azuredisk-5349-disk.csi.azure.com-preprovsioned-pv-xb772 found and phase=Bound (56.039524724s) I1105 21:19:24.225] Nov 5 21:19:24.225: INFO: PersistentVolume azuredisk-5349-disk.csi.azure.com-preprovsioned-pv-xb772 found and phase=Bound (1m1.124015985s) I1105 21:19:29.309] Nov 5 21:19:29.309: INFO: PersistentVolume azuredisk-5349-disk.csi.azure.com-preprovsioned-pv-xb772 found and phase=Bound (1m6.208699947s) I1105 21:19:34.394] Nov 5 21:19:34.394: INFO: PersistentVolume azuredisk-5349-disk.csi.azure.com-preprovsioned-pv-xb772 found and phase=Bound (1m11.292984306s) I1105 21:19:39.485] Nov 5 21:19:39.478: INFO: PersistentVolume azuredisk-5349-disk.csi.azure.com-preprovsioned-pv-xb772 found and phase=Bound (1m16.377452766s) I1105 21:19:44.562] Nov 5 21:19:44.562: INFO: PersistentVolume azuredisk-5349-disk.csi.azure.com-preprovsioned-pv-xb772 found and phase=Bound (1m21.461691524s) I1105 21:19:49.648] Nov 5 21:19:49.647: INFO: PersistentVolume azuredisk-5349-disk.csi.azure.com-preprovsioned-pv-xb772 found and phase=Bound (1m26.546897389s) I1105 21:19:54.732] Nov 5 21:19:54.732: INFO: PersistentVolume azuredisk-5349-disk.csi.azure.com-preprovsioned-pv-xb772 found and phase=Bound (1m31.631304548s) I1105 21:19:59.816] Nov 5 21:19:59.816: INFO: PersistentVolume azuredisk-5349-disk.csi.azure.com-preprovsioned-pv-xb772 found and phase=Bound (1m36.715514506s) I1105 21:20:04.901] Nov 5 21:20:04.901: INFO: PersistentVolume azuredisk-5349-disk.csi.azure.com-preprovsioned-pv-xb772 found and phase=Bound (1m41.800139866s) I1105 21:20:09.986] Nov 5 21:20:09.986: INFO: PersistentVolume azuredisk-5349-disk.csi.azure.com-preprovsioned-pv-xb772 found and phase=Bound (1m46.884987528s) I1105 21:20:15.070] Nov 5 21:20:15.069: INFO: PersistentVolume azuredisk-5349-disk.csi.azure.com-preprovsioned-pv-xb772 found and phase=Bound (1m51.968714181s) I1105 21:20:20.154] Nov 5 21:20:20.154: INFO: PersistentVolume azuredisk-5349-disk.csi.azure.com-preprovsioned-pv-xb772 found and phase=Bound (1m57.053256439s) I1105 21:20:25.247] Nov 5 21:20:25.247: INFO: PersistentVolume azuredisk-5349-disk.csi.azure.com-preprovsioned-pv-xb772 found and phase=Bound (2m2.146468564s) I1105 21:20:30.331] Nov 5 21:20:30.331: INFO: PersistentVolume azuredisk-5349-disk.csi.azure.com-preprovsioned-pv-xb772 found and phase=Bound (2m7.230576319s) I1105 21:20:35.416] Nov 5 21:20:35.416: INFO: PersistentVolume azuredisk-5349-disk.csi.azure.com-preprovsioned-pv-xb772 found and phase=Bound (2m12.315080377s) I1105 21:20:40.505] Nov 5 21:20:40.504: INFO: PersistentVolume azuredisk-5349-disk.csi.azure.com-preprovsioned-pv-xb772 found and phase=Bound (2m17.403953468s) I1105 21:20:45.589] Nov 5 21:20:45.588: INFO: PersistentVolume azuredisk-5349-disk.csi.azure.com-preprovsioned-pv-xb772 found and phase=Bound (2m22.487869021s) I1105 21:20:50.668] Nov 5 21:20:50.668: INFO: PersistentVolume azuredisk-5349-disk.csi.azure.com-preprovsioned-pv-xb772 found and phase=Bound (2m27.56737974s) I1105 21:20:55.747] Nov 5 21:20:55.747: INFO: PersistentVolume azuredisk-5349-disk.csi.azure.com-preprovsioned-pv-xb772 found and phase=Bound (2m32.646391256s) I1105 21:21:00.827] Nov 5 21:21:00.827: INFO: PersistentVolume azuredisk-5349-disk.csi.azure.com-preprovsioned-pv-xb772 found and phase=Bound (2m37.725975176s) I1105 21:21:05.906] Nov 5 21:21:05.906: INFO: PersistentVolume azuredisk-5349-disk.csi.azure.com-preprovsioned-pv-xb772 found and phase=Bound (2m42.805401294s) I1105 21:21:10.987] Nov 5 21:21:10.986: INFO: PersistentVolume azuredisk-5349-disk.csi.azure.com-preprovsioned-pv-xb772 found and phase=Bound (2m47.885737519s) I1105 21:21:16.066] Nov 5 21:21:16.066: INFO: PersistentVolume azuredisk-5349-disk.csi.azure.com-preprovsioned-pv-xb772 found and phase=Bound (2m52.965252437s) I1105 21:21:21.145] Nov 5 21:21:21.145: INFO: PersistentVolume azuredisk-5349-disk.csi.azure.com-preprovsioned-pv-xb772 found and phase=Bound (2m58.044421753s) I1105 21:21:26.224] Nov 5 21:21:26.224: INFO: PersistentVolume azuredisk-5349-disk.csi.azure.com-preprovsioned-pv-xb772 found and phase=Bound (3m3.123365467s) I1105 21:21:31.303] Nov 5 21:21:31.303: INFO: PersistentVolume azuredisk-5349-disk.csi.azure.com-preprovsioned-pv-xb772 found and phase=Bound (3m8.202702183s) I1105 21:21:36.384] Nov 5 21:21:36.383: INFO: PersistentVolume azuredisk-5349-disk.csi.azure.com-preprovsioned-pv-xb772 found and phase=Bound (3m13.282805606s) I1105 21:21:41.463] Nov 5 21:21:41.463: INFO: PersistentVolume azuredisk-5349-disk.csi.azure.com-preprovsioned-pv-xb772 found and phase=Bound (3m18.362157122s) I1105 21:21:46.542] Nov 5 21:21:46.542: INFO: PersistentVolume azuredisk-5349-disk.csi.azure.com-preprovsioned-pv-xb772 found and phase=Bound (3m23.441636439s) I1105 21:21:51.622] Nov 5 21:21:51.622: INFO: PersistentVolume azuredisk-5349-disk.csi.azure.com-preprovsioned-pv-xb772 found and phase=Bound (3m28.521011755s) I1105 21:21:56.726] Nov 5 21:21:56.726: INFO: PersistentVolume azuredisk-5349-disk.csi.azure.com-preprovsioned-pv-xb772 found and phase=Bound (3m33.625471061s) I1105 21:22:01.806] Nov 5 21:22:01.805: INFO: PersistentVolume azuredisk-5349-disk.csi.azure.com-preprovsioned-pv-xb772 found and phase=Bound (3m38.704867377s) I1105 21:22:06.885] Nov 5 21:22:06.885: INFO: PersistentVolume azuredisk-5349-disk.csi.azure.com-preprovsioned-pv-xb772 found and phase=Bound (3m43.783973591s) I1105 21:22:11.964] Nov 5 21:22:11.964: INFO: PersistentVolume azuredisk-5349-disk.csi.azure.com-preprovsioned-pv-xb772 found and phase=Bound (3m48.863032404s) I1105 21:22:17.043] Nov 5 21:22:17.043: INFO: PersistentVolume azuredisk-5349-disk.csi.azure.com-preprovsioned-pv-xb772 found and phase=Bound (3m53.942582721s) I1105 21:22:22.123] Nov 5 21:22:22.122: INFO: PersistentVolume azuredisk-5349-disk.csi.azure.com-preprovsioned-pv-xb772 found and phase=Bound (3m59.021796534s) I1105 21:22:27.202] Nov 5 21:22:27.202: INFO: PersistentVolume azuredisk-5349-disk.csi.azure.com-preprovsioned-pv-xb772 found and phase=Bound (4m4.10127565s) I1105 21:22:32.281] Nov 5 21:22:32.281: INFO: PersistentVolume azuredisk-5349-disk.csi.azure.com-preprovsioned-pv-xb772 found and phase=Bound (4m9.180106861s) I1105 21:22:37.361] Nov 5 21:22:37.360: INFO: PersistentVolume azuredisk-5349-disk.csi.azure.com-preprovsioned-pv-xb772 found and phase=Bound (4m14.259717677s) I1105 21:22:42.440] Nov 5 21:22:42.439: INFO: PersistentVolume azuredisk-5349-disk.csi.azure.com-preprovsioned-pv-xb772 found and phase=Bound (4m19.33883699s) I1105 21:22:47.518] Nov 5 21:22:47.518: INFO: PersistentVolume azuredisk-5349-disk.csi.azure.com-preprovsioned-pv-xb772 found and phase=Bound (4m24.417522399s) I1105 21:22:52.598] Nov 5 21:22:52.598: INFO: PersistentVolume azuredisk-5349-disk.csi.azure.com-preprovsioned-pv-xb772 found and phase=Bound (4m29.497093514s) I1105 21:22:57.677] Nov 5 21:22:57.677: INFO: PersistentVolume azuredisk-5349-disk.csi.azure.com-preprovsioned-pv-xb772 found and phase=Bound (4m34.576473928s) I1105 21:23:02.759] Nov 5 21:23:02.759: INFO: PersistentVolume azuredisk-5349-disk.csi.azure.com-preprovsioned-pv-xb772 found and phase=Bound (4m39.657968358s) I1105 21:23:07.838] Nov 5 21:23:07.838: INFO: PersistentVolume azuredisk-5349-disk.csi.azure.com-preprovsioned-pv-xb772 found and phase=Bound (4m44.737451872s) I1105 21:23:12.918] Nov 5 21:23:12.917: INFO: PersistentVolume azuredisk-5349-disk.csi.azure.com-preprovsioned-pv-xb772 found and phase=Bound (4m49.816917387s) I1105 21:23:17.998] Nov 5 21:23:17.997: INFO: PersistentVolume azuredisk-5349-disk.csi.azure.com-preprovsioned-pv-xb772 found and phase=Bound (4m54.896682503s) I1105 21:23:23.077] Nov 5 21:23:23.077: INFO: PersistentVolume azuredisk-5349-disk.csi.azure.com-preprovsioned-pv-xb772 found and phase=Bound (4m59.976403419s) I1105 21:23:28.156] Nov 5 21:23:28.156: INFO: PersistentVolume azuredisk-5349-disk.csi.azure.com-preprovsioned-pv-xb772 found and phase=Bound (5m5.055478029s) I1105 21:23:33.235] Nov 5 21:23:33.235: INFO: PersistentVolume azuredisk-5349-disk.csi.azure.com-preprovsioned-pv-xb772 found and phase=Bound (5m10.134687741s) I1105 21:23:38.315] Nov 5 21:23:38.315: INFO: PersistentVolume azuredisk-5349-disk.csi.azure.com-preprovsioned-pv-xb772 found and phase=Bound (5m15.214375656s) I1105 21:23:43.395] Nov 5 21:23:43.395: INFO: PersistentVolume azuredisk-5349-disk.csi.azure.com-preprovsioned-pv-xb772 found and phase=Bound (5m20.294610775s) I1105 21:23:48.475] Nov 5 21:23:48.474: INFO: PersistentVolume azuredisk-5349-disk.csi.azure.com-preprovsioned-pv-xb772 found and phase=Bound (5m25.373912087s) I1105 21:23:53.554] Nov 5 21:23:53.554: INFO: PersistentVolume azuredisk-5349-disk.csi.azure.com-preprovsioned-pv-xb772 found and phase=Bound (5m30.453611402s) I1105 21:23:58.634] Nov 5 21:23:58.633: INFO: PersistentVolume azuredisk-5349-disk.csi.azure.com-preprovsioned-pv-xb772 found and phase=Bound (5m35.532761712s) I1105 21:24:03.713] Nov 5 21:24:03.713: INFO: PersistentVolume azuredisk-5349-disk.csi.azure.com-preprovsioned-pv-xb772 found and phase=Bound (5m40.612287725s) I1105 21:24:08.792] Nov 5 21:24:08.792: INFO: PersistentVolume azuredisk-5349-disk.csi.azure.com-preprovsioned-pv-xb772 found and phase=Bound (5m45.691657137s) I1105 21:24:13.872] Nov 5 21:24:13.872: INFO: PersistentVolume azuredisk-5349-disk.csi.azure.com-preprovsioned-pv-xb772 found and phase=Bound (5m50.770994049s) I1105 21:24:18.952] Nov 5 21:24:18.952: INFO: PersistentVolume azuredisk-5349-disk.csi.azure.com-preprovsioned-pv-xb772 found and phase=Bound (5m55.850989765s) I1105 21:24:24.031] Nov 5 21:24:24.031: INFO: PersistentVolume azuredisk-5349-disk.csi.azure.com-preprovsioned-pv-xb772 found and phase=Bound (6m0.930501677s) I1105 21:24:29.111] Nov 5 21:24:29.111: INFO: PersistentVolume azuredisk-5349-disk.csi.azure.com-preprovsioned-pv-xb772 found and phase=Bound (6m6.010135091s) I1105 21:24:34.218] Nov 5 21:24:34.218: INFO: PersistentVolume azuredisk-5349-disk.csi.azure.com-preprovsioned-pv-xb772 found and phase=Bound (6m11.11701501s) I1105 21:24:39.297] Nov 5 21:24:39.297: INFO: PersistentVolume azuredisk-5349-disk.csi.azure.com-preprovsioned-pv-xb772 found and phase=Bound (6m16.196360921s) I1105 21:24:44.482] Nov 5 21:24:44.481: INFO: PersistentVolume azuredisk-5349-disk.csi.azure.com-preprovsioned-pv-xb772 found and phase=Bound (6m21.380745828s) I1105 21:24:49.561] Nov 5 21:24:49.561: INFO: PersistentVolume azuredisk-5349-disk.csi.azure.com-preprovsioned-pv-xb772 found and phase=Bound (6m26.460079839s) I1105 21:24:54.640] Nov 5 21:24:54.640: INFO: PersistentVolume azuredisk-5349-disk.csi.azure.com-preprovsioned-pv-xb772 found and phase=Bound (6m31.539146747s) I1105 21:24:59.729] Nov 5 21:24:59.729: INFO: PersistentVolume azuredisk-5349-disk.csi.azure.com-preprovsioned-pv-xb772 found and phase=Bound (6m36.628190331s) I1105 21:25:04.808] Nov 5 21:25:04.808: INFO: PersistentVolume azuredisk-5349-disk.csi.azure.com-preprovsioned-pv-xb772 found and phase=Bound (6m41.70743044s) I1105 21:25:09.887] Nov 5 21:25:09.887: INFO: PersistentVolume azuredisk-5349-disk.csi.azure.com-preprovsioned-pv-xb772 found and phase=Bound (6m46.786140445s) I1105 21:25:14.973] Nov 5 21:25:14.973: INFO: PersistentVolume azuredisk-5349-disk.csi.azure.com-preprovsioned-pv-xb772 found and phase=Bound (6m51.872528309s) I1105 21:25:20.052] Nov 5 21:25:20.052: INFO: PersistentVolume azuredisk-5349-disk.csi.azure.com-preprovsioned-pv-xb772 found and phase=Bound (6m56.951502416s) I1105 21:25:25.132] Nov 5 21:25:25.132: INFO: PersistentVolume azuredisk-5349-disk.csi.azure.com-preprovsioned-pv-xb772 found and phase=Bound (7m2.030996827s) I1105 21:25:30.218] Nov 5 21:25:30.218: INFO: PersistentVolume azuredisk-5349-disk.csi.azure.com-preprovsioned-pv-xb772 found and phase=Bound (7m7.117009687s) I1105 21:25:35.297] Nov 5 21:25:35.297: INFO: PersistentVolume azuredisk-5349-disk.csi.azure.com-preprovsioned-pv-xb772 found and phase=Bound (7m12.196402897s) I1105 21:25:40.376] Nov 5 21:25:40.376: INFO: PersistentVolume azuredisk-5349-disk.csi.azure.com-preprovsioned-pv-xb772 found and phase=Bound (7m17.275282702s) I1105 21:25:45.456] Nov 5 21:25:45.455: INFO: PersistentVolume azuredisk-5349-disk.csi.azure.com-preprovsioned-pv-xb772 found and phase=Bound (7m22.354687212s) I1105 21:25:50.535] Nov 5 21:25:50.534: INFO: PersistentVolume azuredisk-5349-disk.csi.azure.com-preprovsioned-pv-xb772 found and phase=Bound (7m27.43390402s) I1105 21:25:55.614] Nov 5 21:25:55.614: INFO: PersistentVolume azuredisk-5349-disk.csi.azure.com-preprovsioned-pv-xb772 found and phase=Bound (7m32.513617632s) I1105 21:26:00.706] Nov 5 21:26:00.706: INFO: PersistentVolume azuredisk-5349-disk.csi.azure.com-preprovsioned-pv-xb772 found and phase=Bound (7m37.605476236s) I1105 21:26:05.785] Nov 5 21:26:05.785: INFO: PersistentVolume azuredisk-5349-disk.csi.azure.com-preprovsioned-pv-xb772 found and phase=Bound (7m42.684508642s) I1105 21:26:10.865] Nov 5 21:26:10.864: INFO: PersistentVolume azuredisk-5349-disk.csi.azure.com-preprovsioned-pv-xb772 found and phase=Bound (7m47.76377525s) I1105 21:26:15.944] Nov 5 21:26:15.944: INFO: PersistentVolume azuredisk-5349-disk.csi.azure.com-preprovsioned-pv-xb772 found and phase=Bound (7m52.842967358s) I1105 21:26:21.023] Nov 5 21:26:21.023: INFO: PersistentVolume azuredisk-5349-disk.csi.azure.com-preprovsioned-pv-xb772 found and phase=Bound (7m57.922419567s) I1105 21:26:26.103] Nov 5 21:26:26.103: INFO: PersistentVolume azuredisk-5349-disk.csi.azure.com-preprovsioned-pv-xb772 found and phase=Bound (8m3.002176878s) I1105 21:26:31.183] Nov 5 21:26:31.182: INFO: PersistentVolume azuredisk-5349-disk.csi.azure.com-preprovsioned-pv-xb772 found and phase=Bound (8m8.081867589s) I1105 21:26:36.262] Nov 5 21:26:36.262: INFO: PersistentVolume azuredisk-5349-disk.csi.azure.com-preprovsioned-pv-xb772 found and phase=Bound (8m13.161280698s) I1105 21:26:41.342] Nov 5 21:26:41.342: INFO: PersistentVolume azuredisk-5349-disk.csi.azure.com-preprovsioned-pv-xb772 found and phase=Bound (8m18.24113171s) I1105 21:26:46.422] Nov 5 21:26:46.422: INFO: PersistentVolume azuredisk-5349-disk.csi.azure.com-preprovsioned-pv-xb772 found and phase=Bound (8m23.320962821s) I1105 21:26:51.502] Nov 5 21:26:51.502: INFO: PersistentVolume azuredisk-5349-disk.csi.azure.com-preprovsioned-pv-xb772 found and phase=Bound (8m28.401595339s) I1105 21:26:56.582] Nov 5 21:26:56.581: INFO: PersistentVolume azuredisk-5349-disk.csi.azure.com-preprovsioned-pv-xb772 found and phase=Bound (8m33.480855646s) I1105 21:27:01.662] Nov 5 21:27:01.662: INFO: PersistentVolume azuredisk-5349-disk.csi.azure.com-preprovsioned-pv-xb772 found and phase=Bound (8m38.56098766s) I1105 21:27:06.742] Nov 5 21:27:06.741: INFO: PersistentVolume azuredisk-5349-disk.csi.azure.com-preprovsioned-pv-xb772 found and phase=Bound (8m43.640954872s) I1105 21:27:11.822] Nov 5 21:27:11.822: INFO: PersistentVolume azuredisk-5349-disk.csi.azure.com-preprovsioned-pv-xb772 found and phase=Bound (8m48.721029685s) I1105 21:27:16.901] Nov 5 21:27:16.901: INFO: PersistentVolume azuredisk-5349-disk.csi.azure.com-preprovsioned-pv-xb772 found and phase=Bound (8m53.800676894s) I1105 21:27:21.982] Nov 5 21:27:21.982: INFO: PersistentVolume azuredisk-5349-disk.csi.azure.com-preprovsioned-pv-xb772 found and phase=Bound (8m58.881281311s) I1105 21:27:27.061] Nov 5 21:27:27.061: INFO: PersistentVolume azuredisk-5349-disk.csi.azure.com-preprovsioned-pv-xb772 found and phase=Bound (9m3.960683119s) I1105 21:27:32.142] Nov 5 21:27:32.142: INFO: PersistentVolume azuredisk-5349-disk.csi.azure.com-preprovsioned-pv-xb772 found and phase=Bound (9m9.041502037s) I1105 21:27:37.222] Nov 5 21:27:37.221: INFO: PersistentVolume azuredisk-5349-disk.csi.azure.com-preprovsioned-pv-xb772 found and phase=Bound (9m14.120822744s) I1105 21:27:42.302] Nov 5 21:27:42.302: INFO: PersistentVolume azuredisk-5349-disk.csi.azure.com-preprovsioned-pv-xb772 found and phase=Bound (9m19.201117858s) I1105 21:27:47.382] Nov 5 21:27:47.382: INFO: PersistentVolume azuredisk-5349-disk.csi.azure.com-preprovsioned-pv-xb772 found and phase=Bound (9m24.281354471s) I1105 21:27:52.467] Nov 5 21:27:52.467: INFO: PersistentVolume azuredisk-5349-disk.csi.azure.com-preprovsioned-pv-xb772 found and phase=Bound (9m29.366074219s) I1105 21:27:57.547] Nov 5 21:27:57.546: INFO: PersistentVolume azuredisk-5349-disk.csi.azure.com-preprovsioned-pv-xb772 found and phase=Bound (9m34.445765928s) I1105 21:28:02.631] Nov 5 21:28:02.630: INFO: PersistentVolume azuredisk-5349-disk.csi.azure.com-preprovsioned-pv-xb772 found and phase=Bound (9m39.52982617s) I1105 21:28:07.710] Nov 5 21:28:07.710: INFO: PersistentVolume azuredisk-5349-disk.csi.azure.com-preprovsioned-pv-xb772 found and phase=Bound (9m44.609054776s) I1105 21:28:12.789] Nov 5 21:28:12.789: INFO: PersistentVolume azuredisk-5349-disk.csi.azure.com-preprovsioned-pv-xb772 found and phase=Bound (9m49.688579483s) I1105 21:28:17.869] Nov 5 21:28:17.869: INFO: PersistentVolume azuredisk-5349-disk.csi.azure.com-preprovsioned-pv-xb772 found and phase=Bound (9m54.768464794s) I1105 21:28:22.950] Nov 5 21:28:22.950: INFO: PersistentVolume azuredisk-5349-disk.csi.azure.com-preprovsioned-pv-xb772 found and phase=Bound (9m59.849166488s) I1105 21:28:27.950] Nov 5 21:28:27.950: INFO: Unexpected error occurred: PersistentVolume azuredisk-5349-disk.csi.azure.com-preprovsioned-pv-xb772 still exists within 10m0s I1105 21:28:27.951] [AfterEach] [azuredisk-csi-e2e] [single-az] Pre-Provisioned I1105 21:28:27.951] /go/src/sigs.k8s.io/azuredisk-csi-driver/vendor/k8s.io/kubernetes/test/e2e/framework/framework.go:151 I1105 21:28:27.951] STEP: Collecting events from namespace "azuredisk-5349". I1105 21:28:28.030] STEP: Found 7 events. I1105 21:28:28.030] Nov 5 21:28:28.030: INFO: At 2019-11-05 21:17:07 +0000 UTC - event for azuredisk-volume-tester-gxg4d: {default-scheduler } Scheduled: Successfully assigned azuredisk-5349/azuredisk-volume-tester-gxg4d to k8s-agentpool-97435445-vmss000001 I1105 21:28:28.031] Nov 5 21:28:28.030: INFO: At 2019-11-05 21:17:22 +0000 UTC - event for azuredisk-volume-tester-gxg4d: {attachdetach-controller } FailedAttachVolume: AttachVolume.Attach failed for volume "azuredisk-5349-disk.csi.azure.com-preprovsioned-pv-xb772" : attachment timeout for volume /subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b/providers/Microsoft.Compute/disks/pre-provisioned-readOnly I1105 21:28:28.031] Nov 5 21:28:28.030: INFO: At 2019-11-05 21:17:48 +0000 UTC - event for azuredisk-volume-tester-gxg4d: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "azuredisk-5349-disk.csi.azure.com-preprovsioned-pv-xb772" I1105 21:28:28.032] Nov 5 21:28:28.030: INFO: At 2019-11-05 21:18:19 +0000 UTC - event for azuredisk-volume-tester-gxg4d: {kubelet k8s-agentpool-97435445-vmss000001} Pulling: Pulling image "docker.io/library/busybox:1.29" I1105 21:28:28.032] Nov 5 21:28:28.030: INFO: At 2019-11-05 21:18:19 +0000 UTC - event for azuredisk-volume-tester-gxg4d: {kubelet k8s-agentpool-97435445-vmss000001} Pulled: Successfully pulled image "docker.io/library/busybox:1.29" I1105 21:28:28.032] Nov 5 21:28:28.030: INFO: At 2019-11-05 21:18:21 +0000 UTC - event for azuredisk-volume-tester-gxg4d: {kubelet k8s-agentpool-97435445-vmss000001} Created: Created container volume-tester I1105 21:28:28.032] Nov 5 21:28:28.030: INFO: At 2019-11-05 21:18:21 +0000 UTC - event for azuredisk-volume-tester-gxg4d: {kubelet k8s-agentpool-97435445-vmss000001} Started: Started container volume-tester I1105 21:28:28.110] Nov 5 21:28:28.110: INFO: POD NODE PHASE GRACE CONDITIONS I1105 21:28:28.110] Nov 5 21:28:28.110: INFO: I1105 21:28:28.418] Nov 5 21:28:28.418: INFO: I1105 21:28:28.418] Logging node info for node k8s-agentpool-97435445-vmss000000 I1105 21:28:28.500] Nov 5 21:28:28.496: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-agentpool-97435445-vmss000000,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-agentpool-97435445-vmss000000,UID:e955dc47-8c8f-4142-9241-3d409f32e42d,ResourceVersion:15268,Generation:0,CreationTimestamp:2019-11-05 20:07:34 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{agentpool: agentpool,beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_D4s_v3,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 0,kubernetes.azure.com/cluster: kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b,kubernetes.azure.com/role: agent,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-agentpool-97435445-vmss000000,kubernetes.io/os: linux,kubernetes.io/role: agent,node-role.kubernetes.io/agent: ,storageprofile: managed,storagetier: Premium_LRS,topology.disk.csi.azure.com/zone: 0,},Annotations:map[string]string{csi.volume.kubernetes.io/nodeid: {"disk.csi.azure.com":"k8s-agentpool-97435445-vmss000000"},node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.2.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool-97435445-vmss/virtualMachines/0,Unschedulable:false,Taints:[],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-csi-disk.csi.azure.com: {{8 0} {} 8 DecimalSI},cpu: {{4 0} {} 4 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{16797040640 0} {} BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-csi-disk.csi.azure.com: {{8 0} {} 8 DecimalSI},cpu: {{4 0} {} 4 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{16010608640 0} {} BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 20:08:01 +0000 UTC 2019-11-05 20:08:01 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 21:27:48 +0000 UTC 2019-11-05 20:07:27 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 21:27:48 +0000 UTC 2019-11-05 20:07:27 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 21:27:48 +0000 UTC 2019-11-05 20:07:27 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 21:27:48 +0000 UTC 2019-11-05 20:07:55 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.0.4} {Hostname k8s-agentpool-97435445-vmss000000}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:bfb55325809340228149d2d93f50df58,SystemUUID:CB2ED036-0439-F641-BC7F-640C3638C027,BootID:a917f4d1-549d-471d-b7dc-47ae1715d97d,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[k8sprowinternal.azurecr.io/azuredisk-csi@sha256:75d4ff5d7cd845d6e810c67796ecb037e852fb849ec1eb5b3194564008e6d1e0 k8sprowinternal.azurecr.io/azuredisk-csi:e2e-3e3ed045c6faa509dbd8d2890ee9f56aa2934887] 286930414} {[k8s.gcr.io/kubernetes-dashboard-amd64@sha256:0ae6b69432e78069c5ce2bcde0fe409c5c4d6f0f4d9cd50a17974fea38898747 k8s.gcr.io/kubernetes-dashboard-amd64:v1.10.1] 121711221} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/metrics-server-amd64@sha256:49a9f12f7067d11f42c803dbe61ed2c1299959ad85cb315b25ff7eef8e6b8892 k8s.gcr.io/metrics-server-amd64:v0.2.1] 42541759} {[k8s.gcr.io/coredns@sha256:e83beb5e43f8513fa735e77ffc5859640baea30a882a11cc75c4c3244a737d3c k8s.gcr.io/coredns:1.5.0] 42488424} {[mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume@sha256:4fd30d43947d4a54fc89ead7985beecfd3c9b2a93a0655a373b1608ab90bd5af mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume:v0.0.7] 22909487} {[quay.io/k8scsi/livenessprobe@sha256:8c9c8a0f330acc2e558ae6624309447d02522e76aa35af23fa19b88abf4381cb quay.io/k8scsi/livenessprobe:v1.0.2] 17415381} {[quay.io/k8scsi/csi-node-driver-registrar@sha256:13daf82fb99e951a4bff8ae5fc7c17c3a8fe7130be6400990d8f6076c32d4599 quay.io/k8scsi/csi-node-driver-registrar:v1.1.0] 15815995} {[mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume@sha256:23d8c6033f02a1ecad05127ebdc931bb871264228661bc122704b0974e4d9fdd mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume:1.0.8] 1159025} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 21:28:28.500] Nov 5 21:28:28.497: INFO: I1105 21:28:28.500] Logging kubelet events for node k8s-agentpool-97435445-vmss000000 I1105 21:28:28.576] Nov 5 21:28:28.576: INFO: I1105 21:28:28.576] Logging pods the kubelet thinks is on node k8s-agentpool-97435445-vmss000000 I1105 21:28:28.743] Nov 5 21:28:28.743: INFO: azure-ip-masq-agent-m7tv9 started at 2019-11-05 20:08:01 +0000 UTC (0+1 container statuses recorded) I1105 21:28:28.744] Nov 5 21:28:28.743: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 21:28:28.744] Nov 5 21:28:28.743: INFO: keyvault-flexvolume-smspt started at 2019-11-05 20:08:01 +0000 UTC (0+1 container statuses recorded) I1105 21:28:28.744] Nov 5 21:28:28.743: INFO: Container keyvault-flexvolume ready: true, restart count 0 I1105 21:28:28.744] Nov 5 21:28:28.743: INFO: metrics-server-864ffbc5c-wl84z started at 2019-11-05 20:08:01 +0000 UTC (0+1 container statuses recorded) I1105 21:28:28.744] Nov 5 21:28:28.743: INFO: Container metrics-server ready: true, restart count 0 I1105 21:28:28.745] Nov 5 21:28:28.743: INFO: kube-proxy-8mt7f started at 2019-11-05 20:08:01 +0000 UTC (0+1 container statuses recorded) I1105 21:28:28.745] Nov 5 21:28:28.743: INFO: Container kube-proxy ready: true, restart count 0 I1105 21:28:28.745] Nov 5 21:28:28.743: INFO: kubernetes-dashboard-66dd8b8df7-t6j4l started at 2019-11-05 20:08:01 +0000 UTC (0+1 container statuses recorded) I1105 21:28:28.745] Nov 5 21:28:28.743: INFO: Container kubernetes-dashboard ready: true, restart count 0 I1105 21:28:28.745] Nov 5 21:28:28.743: INFO: csi-azuredisk-node-5wwhz started at 2019-11-05 20:16:09 +0000 UTC (0+3 container statuses recorded) I1105 21:28:28.746] Nov 5 21:28:28.743: INFO: Container azuredisk ready: true, restart count 0 I1105 21:28:28.746] Nov 5 21:28:28.743: INFO: Container liveness-probe ready: true, restart count 0 I1105 21:28:28.746] Nov 5 21:28:28.743: INFO: Container node-driver-registrar ready: true, restart count 0 I1105 21:28:28.746] Nov 5 21:28:28.743: INFO: blobfuse-flexvol-installer-p7hdb started at 2019-11-05 20:08:01 +0000 UTC (0+1 container statuses recorded) I1105 21:28:28.746] Nov 5 21:28:28.743: INFO: Container blobfuse-flexvol-installer ready: true, restart count 0 I1105 21:28:28.747] Nov 5 21:28:28.743: INFO: coredns-7f68dcdbdb-zh8g5 started at 2019-11-05 20:08:01 +0000 UTC (0+1 container statuses recorded) I1105 21:28:28.747] Nov 5 21:28:28.743: INFO: Container coredns ready: true, restart count 0 I1105 21:28:29.012] Nov 5 21:28:29.012: INFO: I1105 21:28:29.012] Latency metrics for node k8s-agentpool-97435445-vmss000000 I1105 21:28:29.012] Nov 5 21:28:29.012: INFO: I1105 21:28:29.013] Logging node info for node k8s-agentpool-97435445-vmss000001 I1105 21:28:29.094] Nov 5 21:28:29.091: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-agentpool-97435445-vmss000001,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-agentpool-97435445-vmss000001,UID:75d3cc50-ca6d-45e0-a42a-6090e8d537dd,ResourceVersion:15234,Generation:0,CreationTimestamp:2019-11-05 20:07:34 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{agentpool: agentpool,beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_D4s_v3,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 1,kubernetes.azure.com/cluster: kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b,kubernetes.azure.com/role: agent,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-agentpool-97435445-vmss000001,kubernetes.io/os: linux,kubernetes.io/role: agent,node-role.kubernetes.io/agent: ,storageprofile: managed,storagetier: Premium_LRS,topology.disk.csi.azure.com/zone: 1,},Annotations:map[string]string{csi.volume.kubernetes.io/nodeid: {"disk.csi.azure.com":"k8s-agentpool-97435445-vmss000001"},node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.0.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool-97435445-vmss/virtualMachines/1,Unschedulable:false,Taints:[],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-csi-disk.csi.azure.com: {{8 0} {} 8 DecimalSI},cpu: {{4 0} {} 4 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{16797044736 0} {} 16403364Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-csi-disk.csi.azure.com: {{8 0} {} 8 DecimalSI},cpu: {{4 0} {} 4 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{16010612736 0} {} 15635364Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 20:08:25 +0000 UTC 2019-11-05 20:08:25 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 21:27:37 +0000 UTC 2019-11-05 20:07:26 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 21:27:37 +0000 UTC 2019-11-05 20:07:26 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 21:27:37 +0000 UTC 2019-11-05 20:07:26 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 21:27:37 +0000 UTC 2019-11-05 20:07:55 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.0.5} {Hostname k8s-agentpool-97435445-vmss000001}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:e78def54b2f545c6b94d88bbba523c60,SystemUUID:40AE8188-86AB-D848-84A4-D9254E1258BC,BootID:8587ec06-f4f4-43c1-86cd-06c295b05b63,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[k8sprowinternal.azurecr.io/azuredisk-csi@sha256:75d4ff5d7cd845d6e810c67796ecb037e852fb849ec1eb5b3194564008e6d1e0 k8sprowinternal.azurecr.io/azuredisk-csi:e2e-3e3ed045c6faa509dbd8d2890ee9f56aa2934887] 286930414} {[gcr.io/kubernetes-helm/tiller@sha256:f6d8f4ab9ba993b5f5b60a6edafe86352eabe474ffeb84cb6c79b8866dce45d1 gcr.io/kubernetes-helm/tiller:v2.11.0] 71821984} {[quay.io/k8scsi/csi-provisioner@sha256:d657c839dce87324fe2b677302913f9386f885f8746be7bea0ced5b0844e3433 quay.io/k8scsi/csi-provisioner:v1.3.1] 50596998} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[quay.io/k8scsi/csi-attacher@sha256:26fccd7a99d973845df1193b46ebdcc6ab8dc5f6e6be319750c471fce1742d13 quay.io/k8scsi/csi-attacher:v1.2.0] 46226754} {[quay.io/k8scsi/csi-cluster-driver-registrar@sha256:fafd75ae5442f192cfa8c2e792903aee30d5884b62e802e4464b0a895d21e3ef quay.io/k8scsi/csi-cluster-driver-registrar:v1.0.1] 45874691} {[quay.io/k8scsi/csi-snapshotter@sha256:a49e0da1af6f2bf717e41ba1eee8b5e6a1cbd66a709dd92cc43fe475fe2589eb quay.io/k8scsi/csi-snapshotter:v1.1.0] 44325352} {[mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume@sha256:4fd30d43947d4a54fc89ead7985beecfd3c9b2a93a0655a373b1608ab90bd5af mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume:v0.0.7] 22909487} {[quay.io/k8scsi/livenessprobe@sha256:8c9c8a0f330acc2e558ae6624309447d02522e76aa35af23fa19b88abf4381cb quay.io/k8scsi/livenessprobe:v1.0.2] 17415381} {[quay.io/k8scsi/csi-node-driver-registrar@sha256:13daf82fb99e951a4bff8ae5fc7c17c3a8fe7130be6400990d8f6076c32d4599 quay.io/k8scsi/csi-node-driver-registrar:v1.1.0] 15815995} {[quay.io/k8scsi/livenessprobe@sha256:dde617756e0f602adc566ab71fd885f1dad451ad3fb063ac991c95a2ff47aea5 quay.io/k8scsi/livenessprobe:v1.1.0] 14967303} {[mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume@sha256:23d8c6033f02a1ecad05127ebdc931bb871264228661bc122704b0974e4d9fdd mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume:1.0.8] 1159025} {[busybox@sha256:8ccbac733d19c0dd4d70b4f0c1e12245b5fa3ad24758a11035ee505c629c0796 busybox:1.29] 1154361} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 21:28:29.095] Nov 5 21:28:29.091: INFO: I1105 21:28:29.095] Logging kubelet events for node k8s-agentpool-97435445-vmss000001 I1105 21:28:29.171] Nov 5 21:28:29.170: INFO: I1105 21:28:29.171] Logging pods the kubelet thinks is on node k8s-agentpool-97435445-vmss000001 I1105 21:28:29.330] Nov 5 21:28:29.330: INFO: keyvault-flexvolume-6bffx started at 2019-11-05 20:07:59 +0000 UTC (0+1 container statuses recorded) I1105 21:28:29.330] Nov 5 21:28:29.330: INFO: Container keyvault-flexvolume ready: true, restart count 0 I1105 21:28:29.330] Nov 5 21:28:29.330: INFO: kube-proxy-kjrpb started at 2019-11-05 20:08:00 +0000 UTC (0+1 container statuses recorded) I1105 21:28:29.331] Nov 5 21:28:29.330: INFO: Container kube-proxy ready: true, restart count 0 I1105 21:28:29.331] Nov 5 21:28:29.330: INFO: tiller-deploy-54c96cb5df-pt7x6 started at 2019-11-05 20:08:00 +0000 UTC (0+1 container statuses recorded) I1105 21:28:29.331] Nov 5 21:28:29.330: INFO: Container tiller ready: true, restart count 0 I1105 21:28:29.331] Nov 5 21:28:29.330: INFO: csi-azuredisk-node-q2dmt started at 2019-11-05 20:16:09 +0000 UTC (0+3 container statuses recorded) I1105 21:28:29.331] Nov 5 21:28:29.330: INFO: Container azuredisk ready: true, restart count 0 I1105 21:28:29.331] Nov 5 21:28:29.330: INFO: Container liveness-probe ready: true, restart count 0 I1105 21:28:29.332] Nov 5 21:28:29.330: INFO: Container node-driver-registrar ready: true, restart count 0 I1105 21:28:29.332] Nov 5 21:28:29.330: INFO: csi-azuredisk-controller-6f6587bfc6-62nrt started at 2019-11-05 20:16:09 +0000 UTC (0+6 container statuses recorded) I1105 21:28:29.332] Nov 5 21:28:29.330: INFO: Container azuredisk ready: true, restart count 0 I1105 21:28:29.332] Nov 5 21:28:29.330: INFO: Container cluster-driver-registrar ready: true, restart count 0 I1105 21:28:29.332] Nov 5 21:28:29.330: INFO: Container csi-attacher ready: true, restart count 0 I1105 21:28:29.333] Nov 5 21:28:29.330: INFO: Container csi-provisioner ready: true, restart count 0 I1105 21:28:29.333] Nov 5 21:28:29.330: INFO: Container csi-snapshotter ready: true, restart count 0 I1105 21:28:29.333] Nov 5 21:28:29.330: INFO: Container liveness-probe ready: true, restart count 0 I1105 21:28:29.333] Nov 5 21:28:29.330: INFO: blobfuse-flexvol-installer-j4s7t started at 2019-11-05 20:07:59 +0000 UTC (0+1 container statuses recorded) I1105 21:28:29.333] Nov 5 21:28:29.330: INFO: Container blobfuse-flexvol-installer ready: true, restart count 0 I1105 21:28:29.333] Nov 5 21:28:29.330: INFO: azure-ip-masq-agent-gdhcq started at 2019-11-05 20:07:59 +0000 UTC (0+1 container statuses recorded) I1105 21:28:29.334] Nov 5 21:28:29.330: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 21:28:29.603] Nov 5 21:28:29.603: INFO: I1105 21:28:29.603] Latency metrics for node k8s-agentpool-97435445-vmss000001 I1105 21:28:29.604] Nov 5 21:28:29.603: INFO: I1105 21:28:29.604] Logging node info for node k8s-master-97435445-0 I1105 21:28:29.685] Nov 5 21:28:29.682: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-97435445-0,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-97435445-0,UID:df3ae36f-961c-4729-a8f5-34cda54f4ff9,ResourceVersion:15321,Generation:0,CreationTimestamp:2019-11-05 20:08:36 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 2,kubernetes.azure.com/cluster: kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-97435445-0,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.4.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b/providers/Microsoft.Compute/virtualMachines/k8s-master-97435445-0,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 20:08:46 +0000 UTC 2019-11-05 20:08:46 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 21:28:07 +0000 UTC 2019-11-05 20:08:28 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 21:28:07 +0000 UTC 2019-11-05 20:08:28 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 21:28:07 +0000 UTC 2019-11-05 20:08:28 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 21:28:07 +0000 UTC 2019-11-05 20:08:36 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.255.5} {Hostname k8s-master-97435445-0}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:3c381865192341c2868672e8626cef7d,SystemUUID:F3D259D3-7EB8-B844-A1F0-F954DCD0D76B,BootID:4948329e-cf57-48c3-aa83-5297bef70370,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 21:28:29.685] Nov 5 21:28:29.682: INFO: I1105 21:28:29.686] Logging kubelet events for node k8s-master-97435445-0 I1105 21:28:29.762] Nov 5 21:28:29.762: INFO: I1105 21:28:29.762] Logging pods the kubelet thinks is on node k8s-master-97435445-0 I1105 21:28:29.847] Nov 5 21:28:29.847: INFO: cloud-controller-manager-k8s-master-97435445-0 started at 2019-11-05 20:08:22 +0000 UTC (0+1 container statuses recorded) I1105 21:28:29.847] Nov 5 21:28:29.847: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 21:28:29.847] Nov 5 21:28:29.847: INFO: kube-addon-manager-k8s-master-97435445-0 started at 2019-11-05 20:08:48 +0000 UTC (0+1 container statuses recorded) I1105 21:28:29.847] Nov 5 21:28:29.847: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 21:28:29.847] Nov 5 21:28:29.847: INFO: kube-apiserver-k8s-master-97435445-0 started at 2019-11-05 20:08:22 +0000 UTC (0+1 container statuses recorded) I1105 21:28:29.848] Nov 5 21:28:29.847: INFO: Container kube-apiserver ready: true, restart count 0 I1105 21:28:29.848] Nov 5 21:28:29.847: INFO: kube-controller-manager-k8s-master-97435445-0 started at 2019-11-05 20:08:22 +0000 UTC (0+1 container statuses recorded) I1105 21:28:29.848] Nov 5 21:28:29.847: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 21:28:29.848] Nov 5 21:28:29.847: INFO: kube-scheduler-k8s-master-97435445-0 started at 2019-11-05 20:08:22 +0000 UTC (0+1 container statuses recorded) I1105 21:28:29.848] Nov 5 21:28:29.847: INFO: Container kube-scheduler ready: true, restart count 0 I1105 21:28:29.848] Nov 5 21:28:29.847: INFO: azure-ip-masq-agent-q6dcl started at 2019-11-05 20:08:38 +0000 UTC (0+1 container statuses recorded) I1105 21:28:29.848] Nov 5 21:28:29.847: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 21:28:29.849] Nov 5 21:28:29.847: INFO: kube-proxy-5wnd2 started at 2019-11-05 20:08:38 +0000 UTC (0+1 container statuses recorded) I1105 21:28:29.849] Nov 5 21:28:29.847: INFO: Container kube-proxy ready: true, restart count 0 I1105 21:28:30.107] Nov 5 21:28:30.107: INFO: I1105 21:28:30.108] Latency metrics for node k8s-master-97435445-0 I1105 21:28:30.108] Nov 5 21:28:30.107: INFO: I1105 21:28:30.108] Logging node info for node k8s-master-97435445-1 I1105 21:28:30.214] Nov 5 21:28:30.212: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-97435445-1,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-97435445-1,UID:05534a67-d493-4ab5-9b0f-5ff5490e234f,ResourceVersion:15230,Generation:0,CreationTimestamp:2019-11-05 20:07:34 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 0,kubernetes.azure.com/cluster: kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-97435445-1,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.1.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b/providers/Microsoft.Compute/virtualMachines/k8s-master-97435445-1,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 20:08:26 +0000 UTC 2019-11-05 20:08:26 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 21:27:36 +0000 UTC 2019-11-05 20:07:28 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 21:27:36 +0000 UTC 2019-11-05 20:07:28 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 21:27:36 +0000 UTC 2019-11-05 20:07:28 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 21:27:36 +0000 UTC 2019-11-05 20:07:55 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{Hostname k8s-master-97435445-1} {InternalIP 10.240.255.6}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:079b605103b74acabac18fd2ea63368a,SystemUUID:38220EEC-BEFF-4C46-AA05-5A214AAFDC28,BootID:b1b45df0-d925-4bcc-9830-0202df36ebb4,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 21:28:30.215] Nov 5 21:28:30.212: INFO: I1105 21:28:30.215] Logging kubelet events for node k8s-master-97435445-1 I1105 21:28:30.292] Nov 5 21:28:30.292: INFO: I1105 21:28:30.292] Logging pods the kubelet thinks is on node k8s-master-97435445-1 I1105 21:28:30.376] Nov 5 21:28:30.376: INFO: cloud-controller-manager-k8s-master-97435445-1 started at 2019-11-05 20:07:22 +0000 UTC (0+1 container statuses recorded) I1105 21:28:30.376] Nov 5 21:28:30.376: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 21:28:30.376] Nov 5 21:28:30.376: INFO: kube-addon-manager-k8s-master-97435445-1 started at 2019-11-05 20:07:22 +0000 UTC (0+1 container statuses recorded) I1105 21:28:30.377] Nov 5 21:28:30.376: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 21:28:30.377] Nov 5 21:28:30.376: INFO: kube-apiserver-k8s-master-97435445-1 started at 2019-11-05 20:07:22 +0000 UTC (0+1 container statuses recorded) I1105 21:28:30.377] Nov 5 21:28:30.376: INFO: Container kube-apiserver ready: true, restart count 0 I1105 21:28:30.377] Nov 5 21:28:30.376: INFO: kube-controller-manager-k8s-master-97435445-1 started at 2019-11-05 20:07:22 +0000 UTC (0+1 container statuses recorded) I1105 21:28:30.377] Nov 5 21:28:30.376: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 21:28:30.377] Nov 5 21:28:30.376: INFO: kube-scheduler-k8s-master-97435445-1 started at 2019-11-05 20:07:22 +0000 UTC (0+1 container statuses recorded) I1105 21:28:30.378] Nov 5 21:28:30.376: INFO: Container kube-scheduler ready: true, restart count 0 I1105 21:28:30.378] Nov 5 21:28:30.376: INFO: azure-ip-masq-agent-n29bm started at 2019-11-05 20:07:59 +0000 UTC (0+1 container statuses recorded) I1105 21:28:30.378] Nov 5 21:28:30.376: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 21:28:30.378] Nov 5 21:28:30.376: INFO: kube-proxy-h4s2c started at 2019-11-05 20:08:00 +0000 UTC (0+1 container statuses recorded) I1105 21:28:30.378] Nov 5 21:28:30.376: INFO: Container kube-proxy ready: true, restart count 0 I1105 21:28:30.662] Nov 5 21:28:30.662: INFO: I1105 21:28:30.662] Latency metrics for node k8s-master-97435445-1 I1105 21:28:30.662] Nov 5 21:28:30.662: INFO: I1105 21:28:30.662] Logging node info for node k8s-master-97435445-2 I1105 21:28:30.743] Nov 5 21:28:30.741: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-97435445-2,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-97435445-2,UID:fa55c895-1a93-4743-885e-03fce84b5332,ResourceVersion:15280,Generation:0,CreationTimestamp:2019-11-05 20:09:19 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 0,kubernetes.azure.com/cluster: kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-97435445-2,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.5.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b/providers/Microsoft.Compute/virtualMachines/k8s-master-97435445-2,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 20:09:19 +0000 UTC 2019-11-05 20:09:19 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 21:27:53 +0000 UTC 2019-11-05 20:09:12 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 21:27:53 +0000 UTC 2019-11-05 20:09:12 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 21:27:53 +0000 UTC 2019-11-05 20:09:12 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 21:27:53 +0000 UTC 2019-11-05 20:09:19 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.255.7} {Hostname k8s-master-97435445-2}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:d19f6709138f496da43c095cec7b2360,SystemUUID:10116777-A8D7-CB4A-A8B7-7EBCF2F23B62,BootID:50cd14a9-5497-4972-bf60-cf59a0cbcb5f,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 21:28:30.744] Nov 5 21:28:30.741: INFO: I1105 21:28:30.744] Logging kubelet events for node k8s-master-97435445-2 I1105 21:28:30.821] Nov 5 21:28:30.821: INFO: I1105 21:28:30.821] Logging pods the kubelet thinks is on node k8s-master-97435445-2 I1105 21:28:30.905] Nov 5 21:28:30.905: INFO: kube-scheduler-k8s-master-97435445-2 started at 2019-11-05 20:09:05 +0000 UTC (0+1 container statuses recorded) I1105 21:28:30.905] Nov 5 21:28:30.905: INFO: Container kube-scheduler ready: true, restart count 0 I1105 21:28:30.905] Nov 5 21:28:30.905: INFO: cloud-controller-manager-k8s-master-97435445-2 started at 2019-11-05 20:09:05 +0000 UTC (0+1 container statuses recorded) I1105 21:28:30.905] Nov 5 21:28:30.905: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 21:28:30.906] Nov 5 21:28:30.905: INFO: kube-addon-manager-k8s-master-97435445-2 started at 2019-11-05 20:09:05 +0000 UTC (0+1 container statuses recorded) I1105 21:28:30.906] Nov 5 21:28:30.905: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 21:28:30.906] Nov 5 21:28:30.905: INFO: azure-ip-masq-agent-sq7p5 started at 2019-11-05 20:09:19 +0000 UTC (0+1 container statuses recorded) I1105 21:28:30.906] Nov 5 21:28:30.905: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 21:28:30.906] Nov 5 21:28:30.905: INFO: kube-proxy-pz426 started at 2019-11-05 20:09:19 +0000 UTC (0+1 container statuses recorded) I1105 21:28:30.906] Nov 5 21:28:30.905: INFO: Container kube-proxy ready: true, restart count 0 I1105 21:28:30.906] Nov 5 21:28:30.905: INFO: kube-apiserver-k8s-master-97435445-2 started at 2019-11-05 20:09:05 +0000 UTC (0+1 container statuses recorded) I1105 21:28:30.907] Nov 5 21:28:30.905: INFO: Container kube-apiserver ready: true, restart count 0 I1105 21:28:30.907] Nov 5 21:28:30.905: INFO: kube-controller-manager-k8s-master-97435445-2 started at 2019-11-05 20:09:05 +0000 UTC (0+1 container statuses recorded) I1105 21:28:30.907] Nov 5 21:28:30.905: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 21:28:31.161] Nov 5 21:28:31.161: INFO: I1105 21:28:31.162] Latency metrics for node k8s-master-97435445-2 I1105 21:28:31.162] Nov 5 21:28:31.161: INFO: I1105 21:28:31.162] Logging node info for node k8s-master-97435445-3 I1105 21:28:31.243] Nov 5 21:28:31.241: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-97435445-3,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-97435445-3,UID:8ffd1d0d-4487-4680-a2f6-e724387be637,ResourceVersion:15279,Generation:0,CreationTimestamp:2019-11-05 20:09:21 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 1,kubernetes.azure.com/cluster: kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-97435445-3,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.6.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b/providers/Microsoft.Compute/virtualMachines/k8s-master-97435445-3,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 20:09:39 +0000 UTC 2019-11-05 20:09:39 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 21:27:52 +0000 UTC 2019-11-05 20:09:18 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 21:27:52 +0000 UTC 2019-11-05 20:09:18 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 21:27:52 +0000 UTC 2019-11-05 20:09:18 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 21:27:52 +0000 UTC 2019-11-05 20:09:21 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.255.8} {Hostname k8s-master-97435445-3}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:266064173bfb4297898c6c25769e8b20,SystemUUID:E6E44241-5015-7740-9A81-A668A8427206,BootID:579d658c-b0ad-47f1-a993-1ccea2b55789,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 21:28:31.243] Nov 5 21:28:31.241: INFO: I1105 21:28:31.244] Logging kubelet events for node k8s-master-97435445-3 I1105 21:28:31.321] Nov 5 21:28:31.321: INFO: I1105 21:28:31.321] Logging pods the kubelet thinks is on node k8s-master-97435445-3 I1105 21:28:31.406] Nov 5 21:28:31.406: INFO: kube-addon-manager-k8s-master-97435445-3 started at 2019-11-05 20:09:05 +0000 UTC (0+1 container statuses recorded) I1105 21:28:31.406] Nov 5 21:28:31.406: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 21:28:31.407] Nov 5 21:28:31.406: INFO: kube-apiserver-k8s-master-97435445-3 started at 2019-11-05 20:09:05 +0000 UTC (0+1 container statuses recorded) I1105 21:28:31.407] Nov 5 21:28:31.406: INFO: Container kube-apiserver ready: true, restart count 0 I1105 21:28:31.407] Nov 5 21:28:31.406: INFO: kube-controller-manager-k8s-master-97435445-3 started at 2019-11-05 20:09:05 +0000 UTC (0+1 container statuses recorded) I1105 21:28:31.407] Nov 5 21:28:31.406: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 21:28:31.407] Nov 5 21:28:31.406: INFO: kube-scheduler-k8s-master-97435445-3 started at 2019-11-05 20:09:05 +0000 UTC (0+1 container statuses recorded) I1105 21:28:31.407] Nov 5 21:28:31.406: INFO: Container kube-scheduler ready: true, restart count 0 I1105 21:28:31.407] Nov 5 21:28:31.406: INFO: azure-ip-masq-agent-747cl started at 2019-11-05 20:09:21 +0000 UTC (0+1 container statuses recorded) I1105 21:28:31.408] Nov 5 21:28:31.406: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 21:28:31.408] Nov 5 21:28:31.406: INFO: kube-proxy-k45g8 started at 2019-11-05 20:09:21 +0000 UTC (0+1 container statuses recorded) I1105 21:28:31.408] Nov 5 21:28:31.406: INFO: Container kube-proxy ready: true, restart count 0 I1105 21:28:31.408] Nov 5 21:28:31.406: INFO: cloud-controller-manager-k8s-master-97435445-3 started at 2019-11-05 20:09:04 +0000 UTC (0+1 container statuses recorded) I1105 21:28:31.408] Nov 5 21:28:31.406: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 21:28:31.661] Nov 5 21:28:31.660: INFO: I1105 21:28:31.661] Latency metrics for node k8s-master-97435445-3 I1105 21:28:31.661] Nov 5 21:28:31.660: INFO: I1105 21:28:31.662] Logging node info for node k8s-master-97435445-4 I1105 21:28:31.741] Nov 5 21:28:31.739: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-97435445-4,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-97435445-4,UID:6d547b41-3f92-4cc4-a2d7-dda2071776a2,ResourceVersion:15316,Generation:0,CreationTimestamp:2019-11-05 20:08:36 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 1,kubernetes.azure.com/cluster: kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-97435445-4,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.3.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b/providers/Microsoft.Compute/virtualMachines/k8s-master-97435445-4,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 20:08:59 +0000 UTC 2019-11-05 20:08:59 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 21:28:06 +0000 UTC 2019-11-05 20:08:29 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 21:28:06 +0000 UTC 2019-11-05 20:08:29 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 21:28:06 +0000 UTC 2019-11-05 20:08:29 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 21:28:06 +0000 UTC 2019-11-05 20:08:36 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.255.9} {Hostname k8s-master-97435445-4}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:56b393c61eb34ba49ebda5279eeedb30,SystemUUID:FD239C23-618C-0049-8734-9AF3FDC9E3C1,BootID:f342ea44-e719-4513-807d-5def7ac606e2,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 21:28:31.741] Nov 5 21:28:31.739: INFO: I1105 21:28:31.742] Logging kubelet events for node k8s-master-97435445-4 I1105 21:28:31.818] Nov 5 21:28:31.818: INFO: I1105 21:28:31.819] Logging pods the kubelet thinks is on node k8s-master-97435445-4 I1105 21:28:31.903] Nov 5 21:28:31.903: INFO: cloud-controller-manager-k8s-master-97435445-4 started at 2019-11-05 20:08:46 +0000 UTC (0+1 container statuses recorded) I1105 21:28:31.904] Nov 5 21:28:31.903: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 21:28:31.904] Nov 5 21:28:31.903: INFO: kube-addon-manager-k8s-master-97435445-4 started at 2019-11-05 20:08:23 +0000 UTC (0+1 container statuses recorded) I1105 21:28:31.904] Nov 5 21:28:31.903: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 21:28:31.904] Nov 5 21:28:31.903: INFO: kube-apiserver-k8s-master-97435445-4 started at 2019-11-05 20:08:23 +0000 UTC (0+1 container statuses recorded) I1105 21:28:31.904] Nov 5 21:28:31.903: INFO: Container kube-apiserver ready: true, restart count 0 I1105 21:28:31.904] Nov 5 21:28:31.903: INFO: kube-controller-manager-k8s-master-97435445-4 started at 2019-11-05 20:08:23 +0000 UTC (0+1 container statuses recorded) I1105 21:28:31.904] Nov 5 21:28:31.903: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 21:28:31.905] Nov 5 21:28:31.903: INFO: kube-scheduler-k8s-master-97435445-4 started at 2019-11-05 20:08:23 +0000 UTC (0+1 container statuses recorded) I1105 21:28:31.905] Nov 5 21:28:31.903: INFO: Container kube-scheduler ready: true, restart count 0 I1105 21:28:31.905] Nov 5 21:28:31.903: INFO: kube-proxy-vfbfr started at 2019-11-05 20:08:36 +0000 UTC (0+1 container statuses recorded) I1105 21:28:31.905] Nov 5 21:28:31.903: INFO: Container kube-proxy ready: true, restart count 0 I1105 21:28:31.905] Nov 5 21:28:31.903: INFO: azure-ip-masq-agent-kmkll started at 2019-11-05 20:08:36 +0000 UTC (0+1 container statuses recorded) I1105 21:28:31.905] Nov 5 21:28:31.903: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 21:28:32.161] Nov 5 21:28:32.161: INFO: I1105 21:28:32.161] Latency metrics for node k8s-master-97435445-4 I1105 21:28:32.161] Nov 5 21:28:32.161: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready I1105 21:28:32.241] STEP: Destroying namespace "azuredisk-5349" for this suite. I1105 21:28:46.492] Nov 5 21:28:46.492: INFO: Waiting up to 30s for server preferred namespaced resources to be successfully discovered I1105 21:28:49.496] Nov 5 21:28:49.495: INFO: namespace azuredisk-5349 deletion completed in 17.254890662s I1105 21:28:49.496] [AfterEach] [azuredisk-csi-e2e] [single-az] Pre-Provisioned I1105 21:28:49.496] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/pre_provisioning.go:54 I1105 21:28:55.208] I1105 21:28:55.209] • Failure [720.681 seconds] I1105 21:28:55.209] [azuredisk-csi-e2e] [single-az] Pre-Provisioned I1105 21:28:55.209] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/pre_provisioning.go:36 I1105 21:28:55.209] [env] should use a pre-provisioned volume and mount it as readOnly in a pod [It] I1105 21:28:55.210] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/pre_provisioning.go:66 I1105 21:28:55.210] I1105 21:28:55.210] Unexpected error: I1105 21:28:55.210] <*errors.errorString | 0xc000ce6730>: { I1105 21:28:55.210] s: "PersistentVolume azuredisk-5349-disk.csi.azure.com-preprovsioned-pv-xb772 still exists within 10m0s", I1105 21:28:55.211] } I1105 21:28:55.211] PersistentVolume azuredisk-5349-disk.csi.azure.com-preprovsioned-pv-xb772 still exists within 10m0s I1105 21:28:55.211] occurred I1105 21:28:55.211] I1105 21:28:55.211] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/testsuites/testsuites.go:353 I1105 21:28:55.212] ------------------------------ I1105 21:28:55.212] [azuredisk-csi-e2e] Dynamic Provisioning [single-az] I1105 21:28:55.212] should delete PV with reclaimPolicy "Delete" I1105 21:28:55.212] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:221 I1105 21:28:55.212] [BeforeEach] [single-az] I1105 21:28:55.213] /go/src/sigs.k8s.io/azuredisk-csi-driver/vendor/k8s.io/kubernetes/test/e2e/framework/framework.go:150 I1105 21:28:55.213] STEP: Creating a kubernetes client I1105 21:28:55.213] Nov 5 21:28:55.208: INFO: >>> kubeConfig: /workspace/aks971561099/kubeconfig/kubeconfig.eastus2.json I1105 21:28:55.213] STEP: Building a namespace api object, basename azuredisk I1105 21:28:55.453] STEP: Binding the e2e-test-privileged-psp PodSecurityPolicy to the default service account in azuredisk-1513 I1105 21:28:55.724] STEP: Waiting for a default service account to be provisioned in namespace I1105 21:28:55.800] [BeforeEach] [single-az] I1105 21:28:55.800] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:58 I1105 21:28:55.801] [It] should delete PV with reclaimPolicy "Delete" I1105 21:28:55.801] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:221 I1105 21:28:55.801] STEP: setting up the StorageClass I1105 21:28:55.801] STEP: creating a StorageClass I1105 21:28:55.885] STEP: setting up the PVC and PV I1105 21:28:55.885] STEP: creating a PVC I1105 21:28:55.980] STEP: waiting for PVC to be in phase "Bound" I1105 21:28:55.980] Nov 5 21:28:55.979: INFO: Waiting up to 5m0s for PersistentVolumeClaims [pvc-xcqrj] to have phase Bound I1105 21:28:56.060] Nov 5 21:28:56.059: INFO: PersistentVolumeClaim pvc-xcqrj found but phase is Pending instead of Bound. I1105 21:28:58.140] Nov 5 21:28:58.140: INFO: PersistentVolumeClaim pvc-xcqrj found but phase is Pending instead of Bound. I1105 21:29:00.219] Nov 5 21:29:00.219: INFO: PersistentVolumeClaim pvc-xcqrj found but phase is Pending instead of Bound. I1105 21:29:02.306] Nov 5 21:29:02.306: INFO: PersistentVolumeClaim pvc-xcqrj found but phase is Pending instead of Bound. I1105 21:29:04.386] Nov 5 21:29:04.385: INFO: PersistentVolumeClaim pvc-xcqrj found but phase is Pending instead of Bound. I1105 21:29:06.466] Nov 5 21:29:06.466: INFO: PersistentVolumeClaim pvc-xcqrj found but phase is Pending instead of Bound. I1105 21:29:08.557] Nov 5 21:29:08.557: INFO: PersistentVolumeClaim pvc-xcqrj found but phase is Pending instead of Bound. I1105 21:29:10.636] Nov 5 21:29:10.636: INFO: PersistentVolumeClaim pvc-xcqrj found but phase is Pending instead of Bound. I1105 21:29:12.716] Nov 5 21:29:12.716: INFO: PersistentVolumeClaim pvc-xcqrj found but phase is Pending instead of Bound. I1105 21:29:14.796] Nov 5 21:29:14.796: INFO: PersistentVolumeClaim pvc-xcqrj found but phase is Pending instead of Bound. I1105 21:29:16.882] Nov 5 21:29:16.882: INFO: PersistentVolumeClaim pvc-xcqrj found but phase is Pending instead of Bound. I1105 21:29:18.961] Nov 5 21:29:18.961: INFO: PersistentVolumeClaim pvc-xcqrj found but phase is Pending instead of Bound. I1105 21:29:21.040] Nov 5 21:29:21.040: INFO: PersistentVolumeClaim pvc-xcqrj found but phase is Pending instead of Bound. I1105 21:29:23.122] Nov 5 21:29:23.122: INFO: PersistentVolumeClaim pvc-xcqrj found but phase is Pending instead of Bound. I1105 21:29:25.202] Nov 5 21:29:25.202: INFO: PersistentVolumeClaim pvc-xcqrj found but phase is Pending instead of Bound. I1105 21:29:27.281] Nov 5 21:29:27.281: INFO: PersistentVolumeClaim pvc-xcqrj found but phase is Pending instead of Bound. I1105 21:29:29.361] Nov 5 21:29:29.360: INFO: PersistentVolumeClaim pvc-xcqrj found but phase is Pending instead of Bound. I1105 21:29:31.441] Nov 5 21:29:31.441: INFO: PersistentVolumeClaim pvc-xcqrj found but phase is Pending instead of Bound. I1105 21:29:33.528] Nov 5 21:29:33.527: INFO: PersistentVolumeClaim pvc-xcqrj found but phase is Pending instead of Bound. I1105 21:29:35.607] Nov 5 21:29:35.607: INFO: PersistentVolumeClaim pvc-xcqrj found but phase is Pending instead of Bound. I1105 21:29:37.687] Nov 5 21:29:37.687: INFO: PersistentVolumeClaim pvc-xcqrj found but phase is Pending instead of Bound. I1105 21:29:39.766] Nov 5 21:29:39.766: INFO: PersistentVolumeClaim pvc-xcqrj found but phase is Pending instead of Bound. I1105 21:29:41.847] Nov 5 21:29:41.846: INFO: PersistentVolumeClaim pvc-xcqrj found but phase is Pending instead of Bound. I1105 21:29:43.926] Nov 5 21:29:43.926: INFO: PersistentVolumeClaim pvc-xcqrj found but phase is Pending instead of Bound. I1105 21:29:46.006] Nov 5 21:29:46.006: INFO: PersistentVolumeClaim pvc-xcqrj found but phase is Pending instead of Bound. I1105 21:29:48.086] Nov 5 21:29:48.086: INFO: PersistentVolumeClaim pvc-xcqrj found but phase is Pending instead of Bound. I1105 21:29:50.188] Nov 5 21:29:50.188: INFO: PersistentVolumeClaim pvc-xcqrj found but phase is Pending instead of Bound. I1105 21:29:52.268] Nov 5 21:29:52.268: INFO: PersistentVolumeClaim pvc-xcqrj found but phase is Pending instead of Bound. I1105 21:29:54.348] Nov 5 21:29:54.348: INFO: PersistentVolumeClaim pvc-xcqrj found but phase is Pending instead of Bound. I1105 21:29:56.427] Nov 5 21:29:56.427: INFO: PersistentVolumeClaim pvc-xcqrj found but phase is Pending instead of Bound. I1105 21:29:58.507] Nov 5 21:29:58.507: INFO: PersistentVolumeClaim pvc-xcqrj found but phase is Pending instead of Bound. I1105 21:30:00.587] Nov 5 21:30:00.587: INFO: PersistentVolumeClaim pvc-xcqrj found but phase is Pending instead of Bound. I1105 21:30:02.675] Nov 5 21:30:02.675: INFO: PersistentVolumeClaim pvc-xcqrj found but phase is Pending instead of Bound. I1105 21:30:04.755] Nov 5 21:30:04.755: INFO: PersistentVolumeClaim pvc-xcqrj found but phase is Pending instead of Bound. I1105 21:30:06.834] Nov 5 21:30:06.834: INFO: PersistentVolumeClaim pvc-xcqrj found but phase is Pending instead of Bound. I1105 21:30:08.915] Nov 5 21:30:08.915: INFO: PersistentVolumeClaim pvc-xcqrj found but phase is Pending instead of Bound. I1105 21:30:10.994] Nov 5 21:30:10.994: INFO: PersistentVolumeClaim pvc-xcqrj found but phase is Pending instead of Bound. I1105 21:30:13.073] Nov 5 21:30:13.073: INFO: PersistentVolumeClaim pvc-xcqrj found but phase is Pending instead of Bound. I1105 21:30:15.153] Nov 5 21:30:15.153: INFO: PersistentVolumeClaim pvc-xcqrj found but phase is Pending instead of Bound. I1105 21:30:17.232] Nov 5 21:30:17.232: INFO: PersistentVolumeClaim pvc-xcqrj found but phase is Pending instead of Bound. I1105 21:30:19.312] Nov 5 21:30:19.312: INFO: PersistentVolumeClaim pvc-xcqrj found but phase is Pending instead of Bound. I1105 21:30:21.391] Nov 5 21:30:21.391: INFO: PersistentVolumeClaim pvc-xcqrj found but phase is Pending instead of Bound. I1105 21:30:23.471] Nov 5 21:30:23.471: INFO: PersistentVolumeClaim pvc-xcqrj found but phase is Pending instead of Bound. I1105 21:30:25.550] Nov 5 21:30:25.550: INFO: PersistentVolumeClaim pvc-xcqrj found but phase is Pending instead of Bound. I1105 21:30:27.630] Nov 5 21:30:27.630: INFO: PersistentVolumeClaim pvc-xcqrj found but phase is Pending instead of Bound. I1105 21:30:29.709] Nov 5 21:30:29.709: INFO: PersistentVolumeClaim pvc-xcqrj found but phase is Pending instead of Bound. I1105 21:30:31.793] Nov 5 21:30:31.793: INFO: PersistentVolumeClaim pvc-xcqrj found but phase is Pending instead of Bound. I1105 21:30:33.873] Nov 5 21:30:33.873: INFO: PersistentVolumeClaim pvc-xcqrj found but phase is Pending instead of Bound. I1105 21:30:35.952] Nov 5 21:30:35.952: INFO: PersistentVolumeClaim pvc-xcqrj found but phase is Pending instead of Bound. I1105 21:30:38.032] Nov 5 21:30:38.031: INFO: PersistentVolumeClaim pvc-xcqrj found but phase is Pending instead of Bound. I1105 21:30:40.111] Nov 5 21:30:40.111: INFO: PersistentVolumeClaim pvc-xcqrj found but phase is Pending instead of Bound. I1105 21:30:42.217] Nov 5 21:30:42.217: INFO: PersistentVolumeClaim pvc-xcqrj found but phase is Pending instead of Bound. I1105 21:30:44.297] Nov 5 21:30:44.297: INFO: PersistentVolumeClaim pvc-xcqrj found but phase is Pending instead of Bound. I1105 21:30:46.376] Nov 5 21:30:46.376: INFO: PersistentVolumeClaim pvc-xcqrj found but phase is Pending instead of Bound. I1105 21:30:48.456] Nov 5 21:30:48.456: INFO: PersistentVolumeClaim pvc-xcqrj found but phase is Pending instead of Bound. I1105 21:30:50.536] Nov 5 21:30:50.536: INFO: PersistentVolumeClaim pvc-xcqrj found but phase is Pending instead of Bound. I1105 21:30:52.615] Nov 5 21:30:52.615: INFO: PersistentVolumeClaim pvc-xcqrj found but phase is Pending instead of Bound. I1105 21:30:54.695] Nov 5 21:30:54.694: INFO: PersistentVolumeClaim pvc-xcqrj found but phase is Pending instead of Bound. I1105 21:30:56.774] Nov 5 21:30:56.774: INFO: PersistentVolumeClaim pvc-xcqrj found but phase is Pending instead of Bound. I1105 21:30:58.858] Nov 5 21:30:58.858: INFO: PersistentVolumeClaim pvc-xcqrj found but phase is Pending instead of Bound. I1105 21:31:00.938] Nov 5 21:31:00.938: INFO: PersistentVolumeClaim pvc-xcqrj found but phase is Pending instead of Bound. I1105 21:31:03.017] Nov 5 21:31:03.017: INFO: PersistentVolumeClaim pvc-xcqrj found but phase is Pending instead of Bound. I1105 21:31:05.097] Nov 5 21:31:05.096: INFO: PersistentVolumeClaim pvc-xcqrj found but phase is Pending instead of Bound. I1105 21:31:07.176] Nov 5 21:31:07.176: INFO: PersistentVolumeClaim pvc-xcqrj found but phase is Pending instead of Bound. I1105 21:31:09.255] Nov 5 21:31:09.255: INFO: PersistentVolumeClaim pvc-xcqrj found but phase is Pending instead of Bound. I1105 21:31:11.335] Nov 5 21:31:11.335: INFO: PersistentVolumeClaim pvc-xcqrj found but phase is Pending instead of Bound. I1105 21:31:13.415] Nov 5 21:31:13.415: INFO: PersistentVolumeClaim pvc-xcqrj found but phase is Pending instead of Bound. I1105 21:31:15.497] Nov 5 21:31:15.496: INFO: PersistentVolumeClaim pvc-xcqrj found but phase is Pending instead of Bound. I1105 21:31:17.576] Nov 5 21:31:17.576: INFO: PersistentVolumeClaim pvc-xcqrj found but phase is Pending instead of Bound. I1105 21:31:19.655] Nov 5 21:31:19.655: INFO: PersistentVolumeClaim pvc-xcqrj found but phase is Pending instead of Bound. I1105 21:31:21.734] Nov 5 21:31:21.734: INFO: PersistentVolumeClaim pvc-xcqrj found but phase is Pending instead of Bound. I1105 21:31:23.814] Nov 5 21:31:23.813: INFO: PersistentVolumeClaim pvc-xcqrj found but phase is Pending instead of Bound. I1105 21:31:25.893] Nov 5 21:31:25.893: INFO: PersistentVolumeClaim pvc-xcqrj found but phase is Pending instead of Bound. I1105 21:31:27.973] Nov 5 21:31:27.972: INFO: PersistentVolumeClaim pvc-xcqrj found but phase is Pending instead of Bound. I1105 21:31:30.052] Nov 5 21:31:30.052: INFO: PersistentVolumeClaim pvc-xcqrj found but phase is Pending instead of Bound. I1105 21:31:32.131] Nov 5 21:31:32.131: INFO: PersistentVolumeClaim pvc-xcqrj found but phase is Pending instead of Bound. I1105 21:31:34.219] Nov 5 21:31:34.219: INFO: PersistentVolumeClaim pvc-xcqrj found but phase is Pending instead of Bound. I1105 21:31:36.298] Nov 5 21:31:36.298: INFO: PersistentVolumeClaim pvc-xcqrj found but phase is Pending instead of Bound. I1105 21:31:38.378] Nov 5 21:31:38.378: INFO: PersistentVolumeClaim pvc-xcqrj found but phase is Pending instead of Bound. I1105 21:31:40.457] Nov 5 21:31:40.457: INFO: PersistentVolumeClaim pvc-xcqrj found but phase is Pending instead of Bound. I1105 21:31:42.541] Nov 5 21:31:42.541: INFO: PersistentVolumeClaim pvc-xcqrj found but phase is Pending instead of Bound. I1105 21:31:44.623] Nov 5 21:31:44.623: INFO: PersistentVolumeClaim pvc-xcqrj found but phase is Pending instead of Bound. I1105 21:31:46.703] Nov 5 21:31:46.702: INFO: PersistentVolumeClaim pvc-xcqrj found but phase is Pending instead of Bound. I1105 21:31:48.781] Nov 5 21:31:48.781: INFO: PersistentVolumeClaim pvc-xcqrj found but phase is Pending instead of Bound. I1105 21:31:50.862] Nov 5 21:31:50.862: INFO: PersistentVolumeClaim pvc-xcqrj found but phase is Pending instead of Bound. I1105 21:31:52.941] Nov 5 21:31:52.941: INFO: PersistentVolumeClaim pvc-xcqrj found but phase is Pending instead of Bound. I1105 21:31:55.021] Nov 5 21:31:55.021: INFO: PersistentVolumeClaim pvc-xcqrj found but phase is Pending instead of Bound. I1105 21:31:57.100] Nov 5 21:31:57.100: INFO: PersistentVolumeClaim pvc-xcqrj found but phase is Pending instead of Bound. I1105 21:31:59.180] Nov 5 21:31:59.180: INFO: PersistentVolumeClaim pvc-xcqrj found but phase is Pending instead of Bound. I1105 21:32:01.259] Nov 5 21:32:01.259: INFO: PersistentVolumeClaim pvc-xcqrj found but phase is Pending instead of Bound. I1105 21:32:03.338] Nov 5 21:32:03.338: INFO: PersistentVolumeClaim pvc-xcqrj found but phase is Pending instead of Bound. I1105 21:32:05.418] Nov 5 21:32:05.418: INFO: PersistentVolumeClaim pvc-xcqrj found but phase is Pending instead of Bound. I1105 21:32:07.505] Nov 5 21:32:07.505: INFO: PersistentVolumeClaim pvc-xcqrj found but phase is Pending instead of Bound. I1105 21:32:09.584] Nov 5 21:32:09.584: INFO: PersistentVolumeClaim pvc-xcqrj found but phase is Pending instead of Bound. I1105 21:32:11.663] Nov 5 21:32:11.663: INFO: PersistentVolumeClaim pvc-xcqrj found but phase is Pending instead of Bound. I1105 21:32:13.742] Nov 5 21:32:13.742: INFO: PersistentVolumeClaim pvc-xcqrj found but phase is Pending instead of Bound. I1105 21:32:15.821] Nov 5 21:32:15.821: INFO: PersistentVolumeClaim pvc-xcqrj found but phase is Pending instead of Bound. I1105 21:32:17.903] Nov 5 21:32:17.903: INFO: PersistentVolumeClaim pvc-xcqrj found but phase is Pending instead of Bound. I1105 21:32:19.982] Nov 5 21:32:19.982: INFO: PersistentVolumeClaim pvc-xcqrj found but phase is Pending instead of Bound. I1105 21:32:22.061] Nov 5 21:32:22.061: INFO: PersistentVolumeClaim pvc-xcqrj found but phase is Pending instead of Bound. I1105 21:32:24.141] Nov 5 21:32:24.141: INFO: PersistentVolumeClaim pvc-xcqrj found but phase is Pending instead of Bound. I1105 21:32:26.221] Nov 5 21:32:26.221: INFO: PersistentVolumeClaim pvc-xcqrj found but phase is Pending instead of Bound. I1105 21:32:28.301] Nov 5 21:32:28.300: INFO: PersistentVolumeClaim pvc-xcqrj found but phase is Pending instead of Bound. I1105 21:32:30.380] Nov 5 21:32:30.379: INFO: PersistentVolumeClaim pvc-xcqrj found but phase is Pending instead of Bound. I1105 21:32:32.460] Nov 5 21:32:32.459: INFO: PersistentVolumeClaim pvc-xcqrj found but phase is Pending instead of Bound. I1105 21:32:34.539] Nov 5 21:32:34.538: INFO: PersistentVolumeClaim pvc-xcqrj found but phase is Pending instead of Bound. I1105 21:32:36.618] Nov 5 21:32:36.617: INFO: PersistentVolumeClaim pvc-xcqrj found but phase is Pending instead of Bound. I1105 21:32:38.697] Nov 5 21:32:38.697: INFO: PersistentVolumeClaim pvc-xcqrj found but phase is Pending instead of Bound. I1105 21:32:40.776] Nov 5 21:32:40.776: INFO: PersistentVolumeClaim pvc-xcqrj found but phase is Pending instead of Bound. I1105 21:32:42.859] Nov 5 21:32:42.859: INFO: PersistentVolumeClaim pvc-xcqrj found but phase is Pending instead of Bound. I1105 21:32:44.939] Nov 5 21:32:44.939: INFO: PersistentVolumeClaim pvc-xcqrj found but phase is Pending instead of Bound. I1105 21:32:47.018] Nov 5 21:32:47.018: INFO: PersistentVolumeClaim pvc-xcqrj found but phase is Pending instead of Bound. I1105 21:32:49.097] Nov 5 21:32:49.097: INFO: PersistentVolumeClaim pvc-xcqrj found but phase is Pending instead of Bound. I1105 21:32:51.176] Nov 5 21:32:51.175: INFO: PersistentVolumeClaim pvc-xcqrj found but phase is Pending instead of Bound. I1105 21:32:53.255] Nov 5 21:32:53.255: INFO: PersistentVolumeClaim pvc-xcqrj found but phase is Pending instead of Bound. I1105 21:32:55.334] Nov 5 21:32:55.334: INFO: PersistentVolumeClaim pvc-xcqrj found but phase is Pending instead of Bound. I1105 21:32:57.426] Nov 5 21:32:57.426: INFO: PersistentVolumeClaim pvc-xcqrj found but phase is Pending instead of Bound. I1105 21:32:59.506] Nov 5 21:32:59.505: INFO: PersistentVolumeClaim pvc-xcqrj found but phase is Pending instead of Bound. I1105 21:33:01.585] Nov 5 21:33:01.584: INFO: PersistentVolumeClaim pvc-xcqrj found but phase is Pending instead of Bound. I1105 21:33:03.664] Nov 5 21:33:03.664: INFO: PersistentVolumeClaim pvc-xcqrj found but phase is Pending instead of Bound. I1105 21:33:05.743] Nov 5 21:33:05.743: INFO: PersistentVolumeClaim pvc-xcqrj found but phase is Pending instead of Bound. I1105 21:33:07.822] Nov 5 21:33:07.822: INFO: PersistentVolumeClaim pvc-xcqrj found but phase is Pending instead of Bound. I1105 21:33:09.902] Nov 5 21:33:09.902: INFO: PersistentVolumeClaim pvc-xcqrj found but phase is Pending instead of Bound. I1105 21:33:11.981] Nov 5 21:33:11.981: INFO: PersistentVolumeClaim pvc-xcqrj found but phase is Pending instead of Bound. I1105 21:33:14.060] Nov 5 21:33:14.060: INFO: PersistentVolumeClaim pvc-xcqrj found but phase is Pending instead of Bound. I1105 21:33:16.139] Nov 5 21:33:16.139: INFO: PersistentVolumeClaim pvc-xcqrj found but phase is Pending instead of Bound. I1105 21:33:18.231] Nov 5 21:33:18.231: INFO: PersistentVolumeClaim pvc-xcqrj found but phase is Pending instead of Bound. I1105 21:33:20.312] Nov 5 21:33:20.311: INFO: PersistentVolumeClaim pvc-xcqrj found but phase is Pending instead of Bound. I1105 21:33:22.391] Nov 5 21:33:22.390: INFO: PersistentVolumeClaim pvc-xcqrj found but phase is Pending instead of Bound. I1105 21:33:24.470] Nov 5 21:33:24.470: INFO: PersistentVolumeClaim pvc-xcqrj found but phase is Pending instead of Bound. I1105 21:33:26.549] Nov 5 21:33:26.549: INFO: PersistentVolumeClaim pvc-xcqrj found but phase is Pending instead of Bound. I1105 21:33:28.628] Nov 5 21:33:28.628: INFO: PersistentVolumeClaim pvc-xcqrj found but phase is Pending instead of Bound. I1105 21:33:30.707] Nov 5 21:33:30.707: INFO: PersistentVolumeClaim pvc-xcqrj found but phase is Pending instead of Bound. I1105 21:33:32.787] Nov 5 21:33:32.786: INFO: PersistentVolumeClaim pvc-xcqrj found but phase is Pending instead of Bound. I1105 21:33:34.866] Nov 5 21:33:34.866: INFO: PersistentVolumeClaim pvc-xcqrj found but phase is Pending instead of Bound. I1105 21:33:36.946] Nov 5 21:33:36.946: INFO: PersistentVolumeClaim pvc-xcqrj found but phase is Pending instead of Bound. I1105 21:33:39.025] Nov 5 21:33:39.025: INFO: PersistentVolumeClaim pvc-xcqrj found but phase is Pending instead of Bound. I1105 21:33:41.104] Nov 5 21:33:41.104: INFO: PersistentVolumeClaim pvc-xcqrj found but phase is Pending instead of Bound. I1105 21:33:43.183] Nov 5 21:33:43.183: INFO: PersistentVolumeClaim pvc-xcqrj found but phase is Pending instead of Bound. I1105 21:33:45.262] Nov 5 21:33:45.262: INFO: PersistentVolumeClaim pvc-xcqrj found but phase is Pending instead of Bound. I1105 21:33:47.342] Nov 5 21:33:47.342: INFO: PersistentVolumeClaim pvc-xcqrj found but phase is Pending instead of Bound. I1105 21:33:49.422] Nov 5 21:33:49.421: INFO: PersistentVolumeClaim pvc-xcqrj found but phase is Pending instead of Bound. I1105 21:33:51.501] Nov 5 21:33:51.500: INFO: PersistentVolumeClaim pvc-xcqrj found but phase is Pending instead of Bound. I1105 21:33:53.581] Nov 5 21:33:53.580: INFO: PersistentVolumeClaim pvc-xcqrj found but phase is Pending instead of Bound. I1105 21:33:55.660] Nov 5 21:33:55.660: INFO: PersistentVolumeClaim pvc-xcqrj found but phase is Pending instead of Bound. I1105 21:33:57.661] Nov 5 21:33:57.660: INFO: Unexpected error occurred: PersistentVolumeClaims [pvc-xcqrj] not all in phase Bound within 5m0s I1105 21:33:57.661] [AfterEach] [single-az] I1105 21:33:57.661] /go/src/sigs.k8s.io/azuredisk-csi-driver/vendor/k8s.io/kubernetes/test/e2e/framework/framework.go:151 I1105 21:33:57.661] STEP: Collecting events from namespace "azuredisk-1513". I1105 21:33:57.747] STEP: Found 3 events. I1105 21:33:57.747] Nov 5 21:33:57.746: INFO: At 2019-11-05 21:28:55 +0000 UTC - event for pvc-xcqrj: {disk.csi.azure.com_csi-azuredisk-controller-6f6587bfc6-62nrt_3930bc69-0009-11ea-8cf5-829a27ff7d5a } Provisioning: External provisioner is provisioning volume for claim "azuredisk-1513/pvc-xcqrj" I1105 21:33:57.747] Nov 5 21:33:57.746: INFO: At 2019-11-05 21:28:55 +0000 UTC - event for pvc-xcqrj: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "disk.csi.azure.com" or manually created by system administrator I1105 21:33:57.748] Nov 5 21:33:57.746: INFO: At 2019-11-05 21:28:56 +0000 UTC - event for pvc-xcqrj: {disk.csi.azure.com_csi-azuredisk-controller-6f6587bfc6-62nrt_3930bc69-0009-11ea-8cf5-829a27ff7d5a } ProvisioningFailed: failed to provision volume with StorageClass "azuredisk-1513-disk.csi.azure.com-dynamic-sc-fr4h7": rpc error: code = Unknown desc = compute.DisksClient#CreateOrUpdate: Failure sending request: StatusCode=400 -- Original Error: Code="InvalidAvailabilityZone" Message="The zone(s) '' for resource 'Microsoft.Compute/disks/pvc-3c352b9d-7d5f-4b7f-afe2-585dacc93888' is not supported. The supported zones for location 'eastus2' are '1,2,3'" I1105 21:33:57.826] Nov 5 21:33:57.826: INFO: POD NODE PHASE GRACE CONDITIONS I1105 21:33:57.826] Nov 5 21:33:57.826: INFO: I1105 21:33:58.068] Nov 5 21:33:58.068: INFO: I1105 21:33:58.069] Logging node info for node k8s-agentpool-97435445-vmss000000 I1105 21:33:58.162] Nov 5 21:33:58.159: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-agentpool-97435445-vmss000000,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-agentpool-97435445-vmss000000,UID:e955dc47-8c8f-4142-9241-3d409f32e42d,ResourceVersion:16343,Generation:0,CreationTimestamp:2019-11-05 20:07:34 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{agentpool: agentpool,beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_D4s_v3,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 0,kubernetes.azure.com/cluster: kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b,kubernetes.azure.com/role: agent,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-agentpool-97435445-vmss000000,kubernetes.io/os: linux,kubernetes.io/role: agent,node-role.kubernetes.io/agent: ,storageprofile: managed,storagetier: Premium_LRS,topology.disk.csi.azure.com/zone: 0,},Annotations:map[string]string{csi.volume.kubernetes.io/nodeid: {"disk.csi.azure.com":"k8s-agentpool-97435445-vmss000000"},node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.2.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool-97435445-vmss/virtualMachines/0,Unschedulable:false,Taints:[],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-csi-disk.csi.azure.com: {{8 0} {} 8 DecimalSI},cpu: {{4 0} {} 4 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{16797040640 0} {} BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-csi-disk.csi.azure.com: {{8 0} {} 8 DecimalSI},cpu: {{4 0} {} 4 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{16010608640 0} {} BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 20:08:01 +0000 UTC 2019-11-05 20:08:01 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 21:33:49 +0000 UTC 2019-11-05 20:07:27 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 21:33:49 +0000 UTC 2019-11-05 20:07:27 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 21:33:49 +0000 UTC 2019-11-05 20:07:27 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 21:33:49 +0000 UTC 2019-11-05 20:07:55 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.0.4} {Hostname k8s-agentpool-97435445-vmss000000}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:bfb55325809340228149d2d93f50df58,SystemUUID:CB2ED036-0439-F641-BC7F-640C3638C027,BootID:a917f4d1-549d-471d-b7dc-47ae1715d97d,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[k8sprowinternal.azurecr.io/azuredisk-csi@sha256:75d4ff5d7cd845d6e810c67796ecb037e852fb849ec1eb5b3194564008e6d1e0 k8sprowinternal.azurecr.io/azuredisk-csi:e2e-3e3ed045c6faa509dbd8d2890ee9f56aa2934887] 286930414} {[k8s.gcr.io/kubernetes-dashboard-amd64@sha256:0ae6b69432e78069c5ce2bcde0fe409c5c4d6f0f4d9cd50a17974fea38898747 k8s.gcr.io/kubernetes-dashboard-amd64:v1.10.1] 121711221} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/metrics-server-amd64@sha256:49a9f12f7067d11f42c803dbe61ed2c1299959ad85cb315b25ff7eef8e6b8892 k8s.gcr.io/metrics-server-amd64:v0.2.1] 42541759} {[k8s.gcr.io/coredns@sha256:e83beb5e43f8513fa735e77ffc5859640baea30a882a11cc75c4c3244a737d3c k8s.gcr.io/coredns:1.5.0] 42488424} {[mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume@sha256:4fd30d43947d4a54fc89ead7985beecfd3c9b2a93a0655a373b1608ab90bd5af mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume:v0.0.7] 22909487} {[quay.io/k8scsi/livenessprobe@sha256:8c9c8a0f330acc2e558ae6624309447d02522e76aa35af23fa19b88abf4381cb quay.io/k8scsi/livenessprobe:v1.0.2] 17415381} {[quay.io/k8scsi/csi-node-driver-registrar@sha256:13daf82fb99e951a4bff8ae5fc7c17c3a8fe7130be6400990d8f6076c32d4599 quay.io/k8scsi/csi-node-driver-registrar:v1.1.0] 15815995} {[mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume@sha256:23d8c6033f02a1ecad05127ebdc931bb871264228661bc122704b0974e4d9fdd mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume:1.0.8] 1159025} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 21:33:58.162] Nov 5 21:33:58.159: INFO: I1105 21:33:58.163] Logging kubelet events for node k8s-agentpool-97435445-vmss000000 I1105 21:33:58.238] Nov 5 21:33:58.238: INFO: I1105 21:33:58.238] Logging pods the kubelet thinks is on node k8s-agentpool-97435445-vmss000000 I1105 21:33:58.397] Nov 5 21:33:58.396: INFO: blobfuse-flexvol-installer-p7hdb started at 2019-11-05 20:08:01 +0000 UTC (0+1 container statuses recorded) I1105 21:33:58.397] Nov 5 21:33:58.396: INFO: Container blobfuse-flexvol-installer ready: true, restart count 0 I1105 21:33:58.397] Nov 5 21:33:58.396: INFO: coredns-7f68dcdbdb-zh8g5 started at 2019-11-05 20:08:01 +0000 UTC (0+1 container statuses recorded) I1105 21:33:58.397] Nov 5 21:33:58.396: INFO: Container coredns ready: true, restart count 0 I1105 21:33:58.397] Nov 5 21:33:58.396: INFO: azure-ip-masq-agent-m7tv9 started at 2019-11-05 20:08:01 +0000 UTC (0+1 container statuses recorded) I1105 21:33:58.397] Nov 5 21:33:58.396: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 21:33:58.398] Nov 5 21:33:58.396: INFO: keyvault-flexvolume-smspt started at 2019-11-05 20:08:01 +0000 UTC (0+1 container statuses recorded) I1105 21:33:58.398] Nov 5 21:33:58.396: INFO: Container keyvault-flexvolume ready: true, restart count 0 I1105 21:33:58.398] Nov 5 21:33:58.396: INFO: metrics-server-864ffbc5c-wl84z started at 2019-11-05 20:08:01 +0000 UTC (0+1 container statuses recorded) I1105 21:33:58.398] Nov 5 21:33:58.396: INFO: Container metrics-server ready: true, restart count 0 I1105 21:33:58.398] Nov 5 21:33:58.396: INFO: kube-proxy-8mt7f started at 2019-11-05 20:08:01 +0000 UTC (0+1 container statuses recorded) I1105 21:33:58.398] Nov 5 21:33:58.396: INFO: Container kube-proxy ready: true, restart count 0 I1105 21:33:58.398] Nov 5 21:33:58.396: INFO: kubernetes-dashboard-66dd8b8df7-t6j4l started at 2019-11-05 20:08:01 +0000 UTC (0+1 container statuses recorded) I1105 21:33:58.398] Nov 5 21:33:58.396: INFO: Container kubernetes-dashboard ready: true, restart count 0 I1105 21:33:58.399] Nov 5 21:33:58.396: INFO: csi-azuredisk-node-5wwhz started at 2019-11-05 20:16:09 +0000 UTC (0+3 container statuses recorded) I1105 21:33:58.399] Nov 5 21:33:58.396: INFO: Container azuredisk ready: true, restart count 0 I1105 21:33:58.399] Nov 5 21:33:58.396: INFO: Container liveness-probe ready: true, restart count 0 I1105 21:33:58.399] Nov 5 21:33:58.396: INFO: Container node-driver-registrar ready: true, restart count 0 I1105 21:33:58.654] Nov 5 21:33:58.653: INFO: I1105 21:33:58.654] Latency metrics for node k8s-agentpool-97435445-vmss000000 I1105 21:33:58.654] Nov 5 21:33:58.653: INFO: I1105 21:33:58.654] Logging node info for node k8s-agentpool-97435445-vmss000001 I1105 21:33:58.736] Nov 5 21:33:58.733: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-agentpool-97435445-vmss000001,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-agentpool-97435445-vmss000001,UID:75d3cc50-ca6d-45e0-a42a-6090e8d537dd,ResourceVersion:16311,Generation:0,CreationTimestamp:2019-11-05 20:07:34 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{agentpool: agentpool,beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_D4s_v3,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 1,kubernetes.azure.com/cluster: kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b,kubernetes.azure.com/role: agent,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-agentpool-97435445-vmss000001,kubernetes.io/os: linux,kubernetes.io/role: agent,node-role.kubernetes.io/agent: ,storageprofile: managed,storagetier: Premium_LRS,topology.disk.csi.azure.com/zone: 1,},Annotations:map[string]string{csi.volume.kubernetes.io/nodeid: {"disk.csi.azure.com":"k8s-agentpool-97435445-vmss000001"},node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.0.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool-97435445-vmss/virtualMachines/1,Unschedulable:false,Taints:[],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-csi-disk.csi.azure.com: {{8 0} {} 8 DecimalSI},cpu: {{4 0} {} 4 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{16797044736 0} {} 16403364Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-csi-disk.csi.azure.com: {{8 0} {} 8 DecimalSI},cpu: {{4 0} {} 4 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{16010612736 0} {} 15635364Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 20:08:25 +0000 UTC 2019-11-05 20:08:25 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 21:33:38 +0000 UTC 2019-11-05 20:07:26 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 21:33:38 +0000 UTC 2019-11-05 20:07:26 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 21:33:38 +0000 UTC 2019-11-05 20:07:26 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 21:33:38 +0000 UTC 2019-11-05 20:07:55 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.0.5} {Hostname k8s-agentpool-97435445-vmss000001}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:e78def54b2f545c6b94d88bbba523c60,SystemUUID:40AE8188-86AB-D848-84A4-D9254E1258BC,BootID:8587ec06-f4f4-43c1-86cd-06c295b05b63,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[k8sprowinternal.azurecr.io/azuredisk-csi@sha256:75d4ff5d7cd845d6e810c67796ecb037e852fb849ec1eb5b3194564008e6d1e0 k8sprowinternal.azurecr.io/azuredisk-csi:e2e-3e3ed045c6faa509dbd8d2890ee9f56aa2934887] 286930414} {[gcr.io/kubernetes-helm/tiller@sha256:f6d8f4ab9ba993b5f5b60a6edafe86352eabe474ffeb84cb6c79b8866dce45d1 gcr.io/kubernetes-helm/tiller:v2.11.0] 71821984} {[quay.io/k8scsi/csi-provisioner@sha256:d657c839dce87324fe2b677302913f9386f885f8746be7bea0ced5b0844e3433 quay.io/k8scsi/csi-provisioner:v1.3.1] 50596998} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[quay.io/k8scsi/csi-attacher@sha256:26fccd7a99d973845df1193b46ebdcc6ab8dc5f6e6be319750c471fce1742d13 quay.io/k8scsi/csi-attacher:v1.2.0] 46226754} {[quay.io/k8scsi/csi-cluster-driver-registrar@sha256:fafd75ae5442f192cfa8c2e792903aee30d5884b62e802e4464b0a895d21e3ef quay.io/k8scsi/csi-cluster-driver-registrar:v1.0.1] 45874691} {[quay.io/k8scsi/csi-snapshotter@sha256:a49e0da1af6f2bf717e41ba1eee8b5e6a1cbd66a709dd92cc43fe475fe2589eb quay.io/k8scsi/csi-snapshotter:v1.1.0] 44325352} {[mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume@sha256:4fd30d43947d4a54fc89ead7985beecfd3c9b2a93a0655a373b1608ab90bd5af mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume:v0.0.7] 22909487} {[quay.io/k8scsi/livenessprobe@sha256:8c9c8a0f330acc2e558ae6624309447d02522e76aa35af23fa19b88abf4381cb quay.io/k8scsi/livenessprobe:v1.0.2] 17415381} {[quay.io/k8scsi/csi-node-driver-registrar@sha256:13daf82fb99e951a4bff8ae5fc7c17c3a8fe7130be6400990d8f6076c32d4599 quay.io/k8scsi/csi-node-driver-registrar:v1.1.0] 15815995} {[quay.io/k8scsi/livenessprobe@sha256:dde617756e0f602adc566ab71fd885f1dad451ad3fb063ac991c95a2ff47aea5 quay.io/k8scsi/livenessprobe:v1.1.0] 14967303} {[mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume@sha256:23d8c6033f02a1ecad05127ebdc931bb871264228661bc122704b0974e4d9fdd mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume:1.0.8] 1159025} {[busybox@sha256:8ccbac733d19c0dd4d70b4f0c1e12245b5fa3ad24758a11035ee505c629c0796 busybox:1.29] 1154361} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 21:33:58.736] Nov 5 21:33:58.733: INFO: I1105 21:33:58.737] Logging kubelet events for node k8s-agentpool-97435445-vmss000001 I1105 21:33:58.819] Nov 5 21:33:58.819: INFO: I1105 21:33:58.819] Logging pods the kubelet thinks is on node k8s-agentpool-97435445-vmss000001 I1105 21:33:58.979] Nov 5 21:33:58.979: INFO: keyvault-flexvolume-6bffx started at 2019-11-05 20:07:59 +0000 UTC (0+1 container statuses recorded) I1105 21:33:58.980] Nov 5 21:33:58.979: INFO: Container keyvault-flexvolume ready: true, restart count 0 I1105 21:33:58.980] Nov 5 21:33:58.979: INFO: kube-proxy-kjrpb started at 2019-11-05 20:08:00 +0000 UTC (0+1 container statuses recorded) I1105 21:33:58.980] Nov 5 21:33:58.979: INFO: Container kube-proxy ready: true, restart count 0 I1105 21:33:58.980] Nov 5 21:33:58.979: INFO: tiller-deploy-54c96cb5df-pt7x6 started at 2019-11-05 20:08:00 +0000 UTC (0+1 container statuses recorded) I1105 21:33:58.980] Nov 5 21:33:58.979: INFO: Container tiller ready: true, restart count 0 I1105 21:33:58.980] Nov 5 21:33:58.979: INFO: csi-azuredisk-node-q2dmt started at 2019-11-05 20:16:09 +0000 UTC (0+3 container statuses recorded) I1105 21:33:58.980] Nov 5 21:33:58.979: INFO: Container azuredisk ready: true, restart count 0 I1105 21:33:58.981] Nov 5 21:33:58.979: INFO: Container liveness-probe ready: true, restart count 0 I1105 21:33:58.981] Nov 5 21:33:58.979: INFO: Container node-driver-registrar ready: true, restart count 0 I1105 21:33:58.981] Nov 5 21:33:58.979: INFO: csi-azuredisk-controller-6f6587bfc6-62nrt started at 2019-11-05 20:16:09 +0000 UTC (0+6 container statuses recorded) I1105 21:33:58.981] Nov 5 21:33:58.979: INFO: Container azuredisk ready: true, restart count 0 I1105 21:33:58.981] Nov 5 21:33:58.979: INFO: Container cluster-driver-registrar ready: true, restart count 0 I1105 21:33:58.981] Nov 5 21:33:58.979: INFO: Container csi-attacher ready: true, restart count 0 I1105 21:33:58.982] Nov 5 21:33:58.979: INFO: Container csi-provisioner ready: true, restart count 0 I1105 21:33:58.982] Nov 5 21:33:58.979: INFO: Container csi-snapshotter ready: true, restart count 0 I1105 21:33:58.982] Nov 5 21:33:58.979: INFO: Container liveness-probe ready: true, restart count 0 I1105 21:33:58.982] Nov 5 21:33:58.979: INFO: blobfuse-flexvol-installer-j4s7t started at 2019-11-05 20:07:59 +0000 UTC (0+1 container statuses recorded) I1105 21:33:58.982] Nov 5 21:33:58.979: INFO: Container blobfuse-flexvol-installer ready: true, restart count 0 I1105 21:33:58.982] Nov 5 21:33:58.979: INFO: azure-ip-masq-agent-gdhcq started at 2019-11-05 20:07:59 +0000 UTC (0+1 container statuses recorded) I1105 21:33:58.982] Nov 5 21:33:58.979: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 21:33:59.246] Nov 5 21:33:59.246: INFO: I1105 21:33:59.246] Latency metrics for node k8s-agentpool-97435445-vmss000001 I1105 21:33:59.246] Nov 5 21:33:59.246: INFO: I1105 21:33:59.246] Logging node info for node k8s-master-97435445-0 I1105 21:33:59.328] Nov 5 21:33:59.325: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-97435445-0,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-97435445-0,UID:df3ae36f-961c-4729-a8f5-34cda54f4ff9,ResourceVersion:16225,Generation:0,CreationTimestamp:2019-11-05 20:08:36 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 2,kubernetes.azure.com/cluster: kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-97435445-0,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.4.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b/providers/Microsoft.Compute/virtualMachines/k8s-master-97435445-0,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 20:08:46 +0000 UTC 2019-11-05 20:08:46 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 21:33:08 +0000 UTC 2019-11-05 20:08:28 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 21:33:08 +0000 UTC 2019-11-05 20:08:28 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 21:33:08 +0000 UTC 2019-11-05 20:08:28 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 21:33:08 +0000 UTC 2019-11-05 20:08:36 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.255.5} {Hostname k8s-master-97435445-0}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:3c381865192341c2868672e8626cef7d,SystemUUID:F3D259D3-7EB8-B844-A1F0-F954DCD0D76B,BootID:4948329e-cf57-48c3-aa83-5297bef70370,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 21:33:59.328] Nov 5 21:33:59.325: INFO: I1105 21:33:59.328] Logging kubelet events for node k8s-master-97435445-0 I1105 21:33:59.405] Nov 5 21:33:59.405: INFO: I1105 21:33:59.405] Logging pods the kubelet thinks is on node k8s-master-97435445-0 I1105 21:33:59.489] Nov 5 21:33:59.489: INFO: kube-scheduler-k8s-master-97435445-0 started at 2019-11-05 20:08:22 +0000 UTC (0+1 container statuses recorded) I1105 21:33:59.490] Nov 5 21:33:59.489: INFO: Container kube-scheduler ready: true, restart count 0 I1105 21:33:59.490] Nov 5 21:33:59.489: INFO: azure-ip-masq-agent-q6dcl started at 2019-11-05 20:08:38 +0000 UTC (0+1 container statuses recorded) I1105 21:33:59.490] Nov 5 21:33:59.489: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 21:33:59.490] Nov 5 21:33:59.489: INFO: kube-proxy-5wnd2 started at 2019-11-05 20:08:38 +0000 UTC (0+1 container statuses recorded) I1105 21:33:59.490] Nov 5 21:33:59.489: INFO: Container kube-proxy ready: true, restart count 0 I1105 21:33:59.490] Nov 5 21:33:59.489: INFO: cloud-controller-manager-k8s-master-97435445-0 started at 2019-11-05 20:08:22 +0000 UTC (0+1 container statuses recorded) I1105 21:33:59.490] Nov 5 21:33:59.489: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 21:33:59.491] Nov 5 21:33:59.489: INFO: kube-addon-manager-k8s-master-97435445-0 started at 2019-11-05 20:08:48 +0000 UTC (0+1 container statuses recorded) I1105 21:33:59.491] Nov 5 21:33:59.489: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 21:33:59.491] Nov 5 21:33:59.489: INFO: kube-apiserver-k8s-master-97435445-0 started at 2019-11-05 20:08:22 +0000 UTC (0+1 container statuses recorded) I1105 21:33:59.491] Nov 5 21:33:59.489: INFO: Container kube-apiserver ready: true, restart count 0 I1105 21:33:59.491] Nov 5 21:33:59.489: INFO: kube-controller-manager-k8s-master-97435445-0 started at 2019-11-05 20:08:22 +0000 UTC (0+1 container statuses recorded) I1105 21:33:59.491] Nov 5 21:33:59.489: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 21:33:59.747] Nov 5 21:33:59.747: INFO: I1105 21:33:59.747] Latency metrics for node k8s-master-97435445-0 I1105 21:33:59.747] Nov 5 21:33:59.747: INFO: I1105 21:33:59.747] Logging node info for node k8s-master-97435445-1 I1105 21:33:59.829] Nov 5 21:33:59.826: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-97435445-1,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-97435445-1,UID:05534a67-d493-4ab5-9b0f-5ff5490e234f,ResourceVersion:16306,Generation:0,CreationTimestamp:2019-11-05 20:07:34 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 0,kubernetes.azure.com/cluster: kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-97435445-1,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.1.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b/providers/Microsoft.Compute/virtualMachines/k8s-master-97435445-1,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 20:08:26 +0000 UTC 2019-11-05 20:08:26 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 21:33:37 +0000 UTC 2019-11-05 20:07:28 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 21:33:37 +0000 UTC 2019-11-05 20:07:28 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 21:33:37 +0000 UTC 2019-11-05 20:07:28 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 21:33:37 +0000 UTC 2019-11-05 20:07:55 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{Hostname k8s-master-97435445-1} {InternalIP 10.240.255.6}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:079b605103b74acabac18fd2ea63368a,SystemUUID:38220EEC-BEFF-4C46-AA05-5A214AAFDC28,BootID:b1b45df0-d925-4bcc-9830-0202df36ebb4,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 21:33:59.829] Nov 5 21:33:59.826: INFO: I1105 21:33:59.829] Logging kubelet events for node k8s-master-97435445-1 I1105 21:33:59.907] Nov 5 21:33:59.907: INFO: I1105 21:33:59.907] Logging pods the kubelet thinks is on node k8s-master-97435445-1 I1105 21:33:59.992] Nov 5 21:33:59.991: INFO: kube-scheduler-k8s-master-97435445-1 started at 2019-11-05 20:07:22 +0000 UTC (0+1 container statuses recorded) I1105 21:33:59.992] Nov 5 21:33:59.991: INFO: Container kube-scheduler ready: true, restart count 0 I1105 21:33:59.992] Nov 5 21:33:59.991: INFO: azure-ip-masq-agent-n29bm started at 2019-11-05 20:07:59 +0000 UTC (0+1 container statuses recorded) I1105 21:33:59.992] Nov 5 21:33:59.991: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 21:33:59.992] Nov 5 21:33:59.991: INFO: kube-proxy-h4s2c started at 2019-11-05 20:08:00 +0000 UTC (0+1 container statuses recorded) I1105 21:33:59.992] Nov 5 21:33:59.991: INFO: Container kube-proxy ready: true, restart count 0 I1105 21:33:59.993] Nov 5 21:33:59.991: INFO: cloud-controller-manager-k8s-master-97435445-1 started at 2019-11-05 20:07:22 +0000 UTC (0+1 container statuses recorded) I1105 21:33:59.993] Nov 5 21:33:59.991: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 21:33:59.993] Nov 5 21:33:59.991: INFO: kube-addon-manager-k8s-master-97435445-1 started at 2019-11-05 20:07:22 +0000 UTC (0+1 container statuses recorded) I1105 21:33:59.993] Nov 5 21:33:59.991: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 21:33:59.993] Nov 5 21:33:59.991: INFO: kube-apiserver-k8s-master-97435445-1 started at 2019-11-05 20:07:22 +0000 UTC (0+1 container statuses recorded) I1105 21:33:59.993] Nov 5 21:33:59.991: INFO: Container kube-apiserver ready: true, restart count 0 I1105 21:33:59.993] Nov 5 21:33:59.991: INFO: kube-controller-manager-k8s-master-97435445-1 started at 2019-11-05 20:07:22 +0000 UTC (0+1 container statuses recorded) I1105 21:33:59.994] Nov 5 21:33:59.991: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 21:34:00.256] Nov 5 21:34:00.256: INFO: I1105 21:34:00.257] Latency metrics for node k8s-master-97435445-1 I1105 21:34:00.257] Nov 5 21:34:00.256: INFO: I1105 21:34:00.257] Logging node info for node k8s-master-97435445-2 I1105 21:34:00.338] Nov 5 21:34:00.336: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-97435445-2,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-97435445-2,UID:fa55c895-1a93-4743-885e-03fce84b5332,ResourceVersion:16356,Generation:0,CreationTimestamp:2019-11-05 20:09:19 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 0,kubernetes.azure.com/cluster: kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-97435445-2,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.5.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b/providers/Microsoft.Compute/virtualMachines/k8s-master-97435445-2,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 20:09:19 +0000 UTC 2019-11-05 20:09:19 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 21:33:54 +0000 UTC 2019-11-05 20:09:12 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 21:33:54 +0000 UTC 2019-11-05 20:09:12 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 21:33:54 +0000 UTC 2019-11-05 20:09:12 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 21:33:54 +0000 UTC 2019-11-05 20:09:19 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.255.7} {Hostname k8s-master-97435445-2}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:d19f6709138f496da43c095cec7b2360,SystemUUID:10116777-A8D7-CB4A-A8B7-7EBCF2F23B62,BootID:50cd14a9-5497-4972-bf60-cf59a0cbcb5f,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 21:34:00.338] Nov 5 21:34:00.336: INFO: I1105 21:34:00.338] Logging kubelet events for node k8s-master-97435445-2 I1105 21:34:00.415] Nov 5 21:34:00.415: INFO: I1105 21:34:00.415] Logging pods the kubelet thinks is on node k8s-master-97435445-2 I1105 21:34:00.499] Nov 5 21:34:00.498: INFO: azure-ip-masq-agent-sq7p5 started at 2019-11-05 20:09:19 +0000 UTC (0+1 container statuses recorded) I1105 21:34:00.499] Nov 5 21:34:00.498: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 21:34:00.499] Nov 5 21:34:00.499: INFO: kube-proxy-pz426 started at 2019-11-05 20:09:19 +0000 UTC (0+1 container statuses recorded) I1105 21:34:00.500] Nov 5 21:34:00.499: INFO: Container kube-proxy ready: true, restart count 0 I1105 21:34:00.500] Nov 5 21:34:00.499: INFO: kube-apiserver-k8s-master-97435445-2 started at 2019-11-05 20:09:05 +0000 UTC (0+1 container statuses recorded) I1105 21:34:00.500] Nov 5 21:34:00.499: INFO: Container kube-apiserver ready: true, restart count 0 I1105 21:34:00.500] Nov 5 21:34:00.499: INFO: kube-controller-manager-k8s-master-97435445-2 started at 2019-11-05 20:09:05 +0000 UTC (0+1 container statuses recorded) I1105 21:34:00.500] Nov 5 21:34:00.499: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 21:34:00.500] Nov 5 21:34:00.499: INFO: kube-scheduler-k8s-master-97435445-2 started at 2019-11-05 20:09:05 +0000 UTC (0+1 container statuses recorded) I1105 21:34:00.500] Nov 5 21:34:00.499: INFO: Container kube-scheduler ready: true, restart count 0 I1105 21:34:00.501] Nov 5 21:34:00.499: INFO: cloud-controller-manager-k8s-master-97435445-2 started at 2019-11-05 20:09:05 +0000 UTC (0+1 container statuses recorded) I1105 21:34:00.501] Nov 5 21:34:00.499: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 21:34:00.501] Nov 5 21:34:00.499: INFO: kube-addon-manager-k8s-master-97435445-2 started at 2019-11-05 20:09:05 +0000 UTC (0+1 container statuses recorded) I1105 21:34:00.501] Nov 5 21:34:00.499: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 21:34:00.756] Nov 5 21:34:00.756: INFO: I1105 21:34:00.757] Latency metrics for node k8s-master-97435445-2 I1105 21:34:00.757] Nov 5 21:34:00.756: INFO: I1105 21:34:00.757] Logging node info for node k8s-master-97435445-3 I1105 21:34:00.837] Nov 5 21:34:00.834: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-97435445-3,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-97435445-3,UID:8ffd1d0d-4487-4680-a2f6-e724387be637,ResourceVersion:16354,Generation:0,CreationTimestamp:2019-11-05 20:09:21 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 1,kubernetes.azure.com/cluster: kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-97435445-3,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.6.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b/providers/Microsoft.Compute/virtualMachines/k8s-master-97435445-3,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 20:09:39 +0000 UTC 2019-11-05 20:09:39 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 21:33:53 +0000 UTC 2019-11-05 20:09:18 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 21:33:53 +0000 UTC 2019-11-05 20:09:18 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 21:33:53 +0000 UTC 2019-11-05 20:09:18 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 21:33:53 +0000 UTC 2019-11-05 20:09:21 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.255.8} {Hostname k8s-master-97435445-3}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:266064173bfb4297898c6c25769e8b20,SystemUUID:E6E44241-5015-7740-9A81-A668A8427206,BootID:579d658c-b0ad-47f1-a993-1ccea2b55789,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 21:34:00.837] Nov 5 21:34:00.834: INFO: I1105 21:34:00.837] Logging kubelet events for node k8s-master-97435445-3 I1105 21:34:00.913] Nov 5 21:34:00.913: INFO: I1105 21:34:00.913] Logging pods the kubelet thinks is on node k8s-master-97435445-3 I1105 21:34:00.997] Nov 5 21:34:00.997: INFO: kube-proxy-k45g8 started at 2019-11-05 20:09:21 +0000 UTC (0+1 container statuses recorded) I1105 21:34:00.997] Nov 5 21:34:00.997: INFO: Container kube-proxy ready: true, restart count 0 I1105 21:34:00.997] Nov 5 21:34:00.997: INFO: cloud-controller-manager-k8s-master-97435445-3 started at 2019-11-05 20:09:04 +0000 UTC (0+1 container statuses recorded) I1105 21:34:00.998] Nov 5 21:34:00.997: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 21:34:00.998] Nov 5 21:34:00.997: INFO: kube-addon-manager-k8s-master-97435445-3 started at 2019-11-05 20:09:05 +0000 UTC (0+1 container statuses recorded) I1105 21:34:00.998] Nov 5 21:34:00.997: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 21:34:00.998] Nov 5 21:34:00.997: INFO: kube-apiserver-k8s-master-97435445-3 started at 2019-11-05 20:09:05 +0000 UTC (0+1 container statuses recorded) I1105 21:34:00.998] Nov 5 21:34:00.997: INFO: Container kube-apiserver ready: true, restart count 0 I1105 21:34:00.998] Nov 5 21:34:00.997: INFO: kube-controller-manager-k8s-master-97435445-3 started at 2019-11-05 20:09:05 +0000 UTC (0+1 container statuses recorded) I1105 21:34:00.999] Nov 5 21:34:00.997: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 21:34:00.999] Nov 5 21:34:00.997: INFO: kube-scheduler-k8s-master-97435445-3 started at 2019-11-05 20:09:05 +0000 UTC (0+1 container statuses recorded) I1105 21:34:00.999] Nov 5 21:34:00.997: INFO: Container kube-scheduler ready: true, restart count 0 I1105 21:34:00.999] Nov 5 21:34:00.997: INFO: azure-ip-masq-agent-747cl started at 2019-11-05 20:09:21 +0000 UTC (0+1 container statuses recorded) I1105 21:34:00.999] Nov 5 21:34:00.997: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 21:34:01.258] Nov 5 21:34:01.257: INFO: I1105 21:34:01.258] Latency metrics for node k8s-master-97435445-3 I1105 21:34:01.258] Nov 5 21:34:01.257: INFO: I1105 21:34:01.258] Logging node info for node k8s-master-97435445-4 I1105 21:34:01.338] Nov 5 21:34:01.336: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-97435445-4,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-97435445-4,UID:6d547b41-3f92-4cc4-a2d7-dda2071776a2,ResourceVersion:16220,Generation:0,CreationTimestamp:2019-11-05 20:08:36 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 1,kubernetes.azure.com/cluster: kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-97435445-4,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.3.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b/providers/Microsoft.Compute/virtualMachines/k8s-master-97435445-4,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 20:08:59 +0000 UTC 2019-11-05 20:08:59 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 21:33:07 +0000 UTC 2019-11-05 20:08:29 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 21:33:07 +0000 UTC 2019-11-05 20:08:29 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 21:33:07 +0000 UTC 2019-11-05 20:08:29 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 21:33:07 +0000 UTC 2019-11-05 20:08:36 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.255.9} {Hostname k8s-master-97435445-4}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:56b393c61eb34ba49ebda5279eeedb30,SystemUUID:FD239C23-618C-0049-8734-9AF3FDC9E3C1,BootID:f342ea44-e719-4513-807d-5def7ac606e2,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 21:34:01.338] Nov 5 21:34:01.336: INFO: I1105 21:34:01.339] Logging kubelet events for node k8s-master-97435445-4 I1105 21:34:01.415] Nov 5 21:34:01.415: INFO: I1105 21:34:01.415] Logging pods the kubelet thinks is on node k8s-master-97435445-4 I1105 21:34:01.499] Nov 5 21:34:01.499: INFO: kube-scheduler-k8s-master-97435445-4 started at 2019-11-05 20:08:23 +0000 UTC (0+1 container statuses recorded) I1105 21:34:01.499] Nov 5 21:34:01.499: INFO: Container kube-scheduler ready: true, restart count 0 I1105 21:34:01.499] Nov 5 21:34:01.499: INFO: kube-proxy-vfbfr started at 2019-11-05 20:08:36 +0000 UTC (0+1 container statuses recorded) I1105 21:34:01.500] Nov 5 21:34:01.499: INFO: Container kube-proxy ready: true, restart count 0 I1105 21:34:01.500] Nov 5 21:34:01.499: INFO: azure-ip-masq-agent-kmkll started at 2019-11-05 20:08:36 +0000 UTC (0+1 container statuses recorded) I1105 21:34:01.500] Nov 5 21:34:01.499: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 21:34:01.500] Nov 5 21:34:01.499: INFO: cloud-controller-manager-k8s-master-97435445-4 started at 2019-11-05 20:08:46 +0000 UTC (0+1 container statuses recorded) I1105 21:34:01.500] Nov 5 21:34:01.499: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 21:34:01.500] Nov 5 21:34:01.499: INFO: kube-addon-manager-k8s-master-97435445-4 started at 2019-11-05 20:08:23 +0000 UTC (0+1 container statuses recorded) I1105 21:34:01.500] Nov 5 21:34:01.499: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 21:34:01.501] Nov 5 21:34:01.499: INFO: kube-apiserver-k8s-master-97435445-4 started at 2019-11-05 20:08:23 +0000 UTC (0+1 container statuses recorded) I1105 21:34:01.501] Nov 5 21:34:01.499: INFO: Container kube-apiserver ready: true, restart count 0 I1105 21:34:01.501] Nov 5 21:34:01.499: INFO: kube-controller-manager-k8s-master-97435445-4 started at 2019-11-05 20:08:23 +0000 UTC (0+1 container statuses recorded) I1105 21:34:01.501] Nov 5 21:34:01.499: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 21:34:01.754] Nov 5 21:34:01.754: INFO: I1105 21:34:01.754] Latency metrics for node k8s-master-97435445-4 I1105 21:34:01.754] Nov 5 21:34:01.754: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready I1105 21:34:01.834] STEP: Destroying namespace "azuredisk-1513" for this suite. I1105 21:34:16.079] Nov 5 21:34:16.078: INFO: Waiting up to 30s for server preferred namespaced resources to be successfully discovered I1105 21:34:19.104] Nov 5 21:34:19.104: INFO: namespace azuredisk-1513 deletion completed in 17.270051397s I1105 21:34:19.104] I1105 21:34:19.104] • Failure [323.896 seconds] I1105 21:34:19.105] [azuredisk-csi-e2e] Dynamic Provisioning I1105 21:34:19.105] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:33 I1105 21:34:19.105] [single-az] I1105 21:34:19.105] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:36 I1105 21:34:19.105] should delete PV with reclaimPolicy "Delete" [It] I1105 21:34:19.105] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:221 I1105 21:34:19.105] I1105 21:34:19.106] Unexpected error: I1105 21:34:19.106] <*errors.errorString | 0xc000916f50>: { I1105 21:34:19.106] s: "PersistentVolumeClaims [pvc-xcqrj] not all in phase Bound within 5m0s", I1105 21:34:19.106] } I1105 21:34:19.106] PersistentVolumeClaims [pvc-xcqrj] not all in phase Bound within 5m0s I1105 21:34:19.106] occurred I1105 21:34:19.106] I1105 21:34:19.106] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/testsuites/testsuites.go:278 I1105 21:34:19.106] ------------------------------ I1105 21:34:19.106] [azuredisk-csi-e2e] Dynamic Provisioning [multi-az] I1105 21:34:19.107] should delete PV with reclaimPolicy "Delete" I1105 21:34:19.107] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:221 I1105 21:34:19.107] [BeforeEach] [multi-az] I1105 21:34:19.107] /go/src/sigs.k8s.io/azuredisk-csi-driver/vendor/k8s.io/kubernetes/test/e2e/framework/framework.go:150 I1105 21:34:19.107] STEP: Creating a kubernetes client I1105 21:34:19.107] Nov 5 21:34:19.104: INFO: >>> kubeConfig: /workspace/aks971561099/kubeconfig/kubeconfig.eastus2.json I1105 21:34:19.107] STEP: Building a namespace api object, basename azuredisk I1105 21:34:19.345] STEP: Binding the e2e-test-privileged-psp PodSecurityPolicy to the default service account in azuredisk-3090 I1105 21:34:19.611] STEP: Waiting for a default service account to be provisioned in namespace I1105 21:34:19.687] [BeforeEach] [multi-az] I1105 21:34:19.687] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:58 I1105 21:34:19.687] [It] should delete PV with reclaimPolicy "Delete" I1105 21:34:19.688] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:221 I1105 21:34:19.688] STEP: setting up the StorageClass I1105 21:34:19.688] STEP: creating a StorageClass I1105 21:34:19.775] STEP: setting up the PVC and PV I1105 21:34:19.775] STEP: creating a PVC I1105 21:34:19.864] STEP: waiting for PVC to be in phase "Bound" I1105 21:34:19.864] Nov 5 21:34:19.864: INFO: Waiting up to 5m0s for PersistentVolumeClaims [pvc-rxsnk] to have phase Bound I1105 21:34:19.943] Nov 5 21:34:19.943: INFO: PersistentVolumeClaim pvc-rxsnk found but phase is Pending instead of Bound. I1105 21:34:22.023] Nov 5 21:34:22.023: INFO: PersistentVolumeClaim pvc-rxsnk found but phase is Pending instead of Bound. I1105 21:34:24.102] Nov 5 21:34:24.102: INFO: PersistentVolumeClaim pvc-rxsnk found but phase is Pending instead of Bound. I1105 21:34:26.207] Nov 5 21:34:26.207: INFO: PersistentVolumeClaim pvc-rxsnk found but phase is Pending instead of Bound. I1105 21:34:28.286] Nov 5 21:34:28.286: INFO: PersistentVolumeClaim pvc-rxsnk found but phase is Pending instead of Bound. I1105 21:34:30.367] Nov 5 21:34:30.367: INFO: PersistentVolumeClaim pvc-rxsnk found but phase is Pending instead of Bound. I1105 21:34:32.446] Nov 5 21:34:32.446: INFO: PersistentVolumeClaim pvc-rxsnk found but phase is Pending instead of Bound. I1105 21:35:11.105] Nov 5 21:35:11.105: INFO: PersistentVolumeClaim pvc-rxsnk found but phase is Pending instead of Bound. I1105 21:35:13.184] Nov 5 21:35:13.184: INFO: PersistentVolumeClaim pvc-rxsnk found but phase is Pending instead of Bound. I1105 21:35:15.264] Nov 5 21:35:15.264: INFO: PersistentVolumeClaim pvc-rxsnk found but phase is Pending instead of Bound. I1105 21:35:17.343] Nov 5 21:35:17.343: INFO: PersistentVolumeClaim pvc-rxsnk found but phase is Pending instead of Bound. I1105 21:35:19.422] Nov 5 21:35:19.422: INFO: PersistentVolumeClaim pvc-rxsnk found but phase is Pending instead of Bound. I1105 21:35:21.502] Nov 5 21:35:21.502: INFO: PersistentVolumeClaim pvc-rxsnk found but phase is Pending instead of Bound. I1105 21:35:23.581] Nov 5 21:35:23.581: INFO: PersistentVolumeClaim pvc-rxsnk found but phase is Pending instead of Bound. I1105 21:35:25.661] Nov 5 21:35:25.661: INFO: PersistentVolumeClaim pvc-rxsnk found but phase is Pending instead of Bound. I1105 21:35:27.740] Nov 5 21:35:27.740: INFO: PersistentVolumeClaim pvc-rxsnk found but phase is Pending instead of Bound. I1105 21:35:29.819] Nov 5 21:35:29.819: INFO: PersistentVolumeClaim pvc-rxsnk found but phase is Pending instead of Bound. I1105 21:35:31.898] Nov 5 21:35:31.898: INFO: PersistentVolumeClaim pvc-rxsnk found but phase is Pending instead of Bound. I1105 21:35:33.978] Nov 5 21:35:33.978: INFO: PersistentVolumeClaim pvc-rxsnk found but phase is Pending instead of Bound. I1105 21:35:36.058] Nov 5 21:35:36.058: INFO: PersistentVolumeClaim pvc-rxsnk found but phase is Pending instead of Bound. I1105 21:35:38.138] Nov 5 21:35:38.138: INFO: PersistentVolumeClaim pvc-rxsnk found but phase is Pending instead of Bound. I1105 21:35:40.218] Nov 5 21:35:40.218: INFO: PersistentVolumeClaim pvc-rxsnk found but phase is Pending instead of Bound. I1105 21:35:42.296] Nov 5 21:35:42.296: INFO: PersistentVolumeClaim pvc-rxsnk found but phase is Pending instead of Bound. I1105 21:35:44.376] Nov 5 21:35:44.375: INFO: PersistentVolumeClaim pvc-rxsnk found but phase is Pending instead of Bound. I1105 21:35:46.455] Nov 5 21:35:46.455: INFO: PersistentVolumeClaim pvc-rxsnk found but phase is Pending instead of Bound. I1105 21:35:48.534] Nov 5 21:35:48.534: INFO: PersistentVolumeClaim pvc-rxsnk found but phase is Pending instead of Bound. I1105 21:35:50.618] Nov 5 21:35:50.618: INFO: PersistentVolumeClaim pvc-rxsnk found but phase is Pending instead of Bound. I1105 21:35:52.717] Nov 5 21:35:52.717: INFO: PersistentVolumeClaim pvc-rxsnk found but phase is Pending instead of Bound. I1105 21:35:54.795] Nov 5 21:35:54.795: INFO: PersistentVolumeClaim pvc-rxsnk found but phase is Pending instead of Bound. I1105 21:35:56.875] Nov 5 21:35:56.875: INFO: PersistentVolumeClaim pvc-rxsnk found but phase is Pending instead of Bound. I1105 21:35:58.955] Nov 5 21:35:58.954: INFO: PersistentVolumeClaim pvc-rxsnk found but phase is Pending instead of Bound. I1105 21:36:01.034] Nov 5 21:36:01.034: INFO: PersistentVolumeClaim pvc-rxsnk found but phase is Pending instead of Bound. I1105 21:36:03.115] Nov 5 21:36:03.114: INFO: PersistentVolumeClaim pvc-rxsnk found but phase is Pending instead of Bound. I1105 21:36:05.194] Nov 5 21:36:05.193: INFO: PersistentVolumeClaim pvc-rxsnk found but phase is Pending instead of Bound. I1105 21:36:07.273] Nov 5 21:36:07.273: INFO: PersistentVolumeClaim pvc-rxsnk found but phase is Pending instead of Bound. I1105 21:36:09.352] Nov 5 21:36:09.351: INFO: PersistentVolumeClaim pvc-rxsnk found but phase is Pending instead of Bound. I1105 21:36:11.431] Nov 5 21:36:11.431: INFO: PersistentVolumeClaim pvc-rxsnk found but phase is Pending instead of Bound. I1105 21:36:13.510] Nov 5 21:36:13.510: INFO: PersistentVolumeClaim pvc-rxsnk found but phase is Pending instead of Bound. I1105 21:36:15.590] Nov 5 21:36:15.590: INFO: PersistentVolumeClaim pvc-rxsnk found but phase is Pending instead of Bound. I1105 21:36:17.670] Nov 5 21:36:17.669: INFO: PersistentVolumeClaim pvc-rxsnk found but phase is Pending instead of Bound. I1105 21:36:19.750] Nov 5 21:36:19.750: INFO: PersistentVolumeClaim pvc-rxsnk found but phase is Pending instead of Bound. I1105 21:36:21.830] Nov 5 21:36:21.829: INFO: PersistentVolumeClaim pvc-rxsnk found but phase is Pending instead of Bound. I1105 21:36:23.909] Nov 5 21:36:23.908: INFO: PersistentVolumeClaim pvc-rxsnk found but phase is Pending instead of Bound. I1105 21:36:25.987] Nov 5 21:36:25.987: INFO: PersistentVolumeClaim pvc-rxsnk found but phase is Pending instead of Bound. I1105 21:36:28.067] Nov 5 21:36:28.066: INFO: PersistentVolumeClaim pvc-rxsnk found but phase is Pending instead of Bound. I1105 21:36:30.153] Nov 5 21:36:30.152: INFO: PersistentVolumeClaim pvc-rxsnk found but phase is Pending instead of Bound. I1105 21:36:32.233] Nov 5 21:36:32.232: INFO: PersistentVolumeClaim pvc-rxsnk found but phase is Pending instead of Bound. I1105 21:36:34.312] Nov 5 21:36:34.312: INFO: PersistentVolumeClaim pvc-rxsnk found but phase is Pending instead of Bound. I1105 21:36:36.391] Nov 5 21:36:36.391: INFO: PersistentVolumeClaim pvc-rxsnk found but phase is Pending instead of Bound. I1105 21:36:38.470] Nov 5 21:36:38.470: INFO: PersistentVolumeClaim pvc-rxsnk found but phase is Pending instead of Bound. I1105 21:36:40.549] Nov 5 21:36:40.549: INFO: PersistentVolumeClaim pvc-rxsnk found but phase is Pending instead of Bound. I1105 21:36:42.628] Nov 5 21:36:42.628: INFO: PersistentVolumeClaim pvc-rxsnk found but phase is Pending instead of Bound. I1105 21:36:44.723] Nov 5 21:36:44.722: INFO: PersistentVolumeClaim pvc-rxsnk found but phase is Pending instead of Bound. I1105 21:36:46.802] Nov 5 21:36:46.802: INFO: PersistentVolumeClaim pvc-rxsnk found but phase is Pending instead of Bound. I1105 21:36:48.881] Nov 5 21:36:48.881: INFO: PersistentVolumeClaim pvc-rxsnk found but phase is Pending instead of Bound. I1105 21:36:50.960] Nov 5 21:36:50.960: INFO: PersistentVolumeClaim pvc-rxsnk found but phase is Pending instead of Bound. I1105 21:36:53.039] Nov 5 21:36:53.039: INFO: PersistentVolumeClaim pvc-rxsnk found but phase is Pending instead of Bound. I1105 21:36:55.118] Nov 5 21:36:55.118: INFO: PersistentVolumeClaim pvc-rxsnk found but phase is Pending instead of Bound. I1105 21:36:57.198] Nov 5 21:36:57.198: INFO: PersistentVolumeClaim pvc-rxsnk found but phase is Pending instead of Bound. I1105 21:36:59.277] Nov 5 21:36:59.277: INFO: PersistentVolumeClaim pvc-rxsnk found but phase is Pending instead of Bound. I1105 21:37:01.357] Nov 5 21:37:01.357: INFO: PersistentVolumeClaim pvc-rxsnk found but phase is Pending instead of Bound. I1105 21:37:03.436] Nov 5 21:37:03.436: INFO: PersistentVolumeClaim pvc-rxsnk found but phase is Pending instead of Bound. I1105 21:37:05.518] Nov 5 21:37:05.518: INFO: PersistentVolumeClaim pvc-rxsnk found but phase is Pending instead of Bound. I1105 21:37:07.597] Nov 5 21:37:07.597: INFO: PersistentVolumeClaim pvc-rxsnk found but phase is Pending instead of Bound. I1105 21:37:09.676] Nov 5 21:37:09.676: INFO: PersistentVolumeClaim pvc-rxsnk found but phase is Pending instead of Bound. I1105 21:37:11.756] Nov 5 21:37:11.755: INFO: PersistentVolumeClaim pvc-rxsnk found but phase is Pending instead of Bound. I1105 21:37:13.835] Nov 5 21:37:13.835: INFO: PersistentVolumeClaim pvc-rxsnk found but phase is Pending instead of Bound. I1105 21:37:15.916] Nov 5 21:37:15.915: INFO: PersistentVolumeClaim pvc-rxsnk found but phase is Pending instead of Bound. I1105 21:37:17.995] Nov 5 21:37:17.995: INFO: PersistentVolumeClaim pvc-rxsnk found but phase is Pending instead of Bound. I1105 21:37:20.075] Nov 5 21:37:20.075: INFO: PersistentVolumeClaim pvc-rxsnk found but phase is Pending instead of Bound. I1105 21:37:22.155] Nov 5 21:37:22.155: INFO: PersistentVolumeClaim pvc-rxsnk found but phase is Pending instead of Bound. I1105 21:37:24.234] Nov 5 21:37:24.234: INFO: PersistentVolumeClaim pvc-rxsnk found but phase is Pending instead of Bound. I1105 21:37:26.314] Nov 5 21:37:26.314: INFO: PersistentVolumeClaim pvc-rxsnk found but phase is Pending instead of Bound. I1105 21:37:28.393] Nov 5 21:37:28.393: INFO: PersistentVolumeClaim pvc-rxsnk found but phase is Pending instead of Bound. I1105 21:37:30.472] Nov 5 21:37:30.472: INFO: PersistentVolumeClaim pvc-rxsnk found but phase is Pending instead of Bound. I1105 21:37:32.551] Nov 5 21:37:32.551: INFO: PersistentVolumeClaim pvc-rxsnk found but phase is Pending instead of Bound. I1105 21:37:34.632] Nov 5 21:37:34.631: INFO: PersistentVolumeClaim pvc-rxsnk found but phase is Pending instead of Bound. I1105 21:37:36.712] Nov 5 21:37:36.712: INFO: PersistentVolumeClaim pvc-rxsnk found but phase is Pending instead of Bound. I1105 21:37:38.791] Nov 5 21:37:38.791: INFO: PersistentVolumeClaim pvc-rxsnk found but phase is Pending instead of Bound. I1105 21:37:40.871] Nov 5 21:37:40.871: INFO: PersistentVolumeClaim pvc-rxsnk found but phase is Pending instead of Bound. I1105 21:37:42.950] Nov 5 21:37:42.950: INFO: PersistentVolumeClaim pvc-rxsnk found but phase is Pending instead of Bound. I1105 21:37:45.029] Nov 5 21:37:45.029: INFO: PersistentVolumeClaim pvc-rxsnk found but phase is Pending instead of Bound. I1105 21:37:47.109] Nov 5 21:37:47.109: INFO: PersistentVolumeClaim pvc-rxsnk found but phase is Pending instead of Bound. I1105 21:37:49.189] Nov 5 21:37:49.189: INFO: PersistentVolumeClaim pvc-rxsnk found but phase is Pending instead of Bound. I1105 21:37:51.268] Nov 5 21:37:51.268: INFO: PersistentVolumeClaim pvc-rxsnk found but phase is Pending instead of Bound. I1105 21:37:53.350] Nov 5 21:37:53.350: INFO: PersistentVolumeClaim pvc-rxsnk found but phase is Pending instead of Bound. I1105 21:37:55.429] Nov 5 21:37:55.429: INFO: PersistentVolumeClaim pvc-rxsnk found but phase is Pending instead of Bound. I1105 21:37:57.509] Nov 5 21:37:57.508: INFO: PersistentVolumeClaim pvc-rxsnk found but phase is Pending instead of Bound. I1105 21:37:59.588] Nov 5 21:37:59.587: INFO: PersistentVolumeClaim pvc-rxsnk found but phase is Pending instead of Bound. I1105 21:38:01.668] Nov 5 21:38:01.667: INFO: PersistentVolumeClaim pvc-rxsnk found but phase is Pending instead of Bound. I1105 21:38:03.746] Nov 5 21:38:03.746: INFO: PersistentVolumeClaim pvc-rxsnk found but phase is Pending instead of Bound. I1105 21:38:05.826] Nov 5 21:38:05.825: INFO: PersistentVolumeClaim pvc-rxsnk found but phase is Pending instead of Bound. I1105 21:38:07.905] Nov 5 21:38:07.905: INFO: PersistentVolumeClaim pvc-rxsnk found but phase is Pending instead of Bound. I1105 21:38:09.983] Nov 5 21:38:09.983: INFO: PersistentVolumeClaim pvc-rxsnk found but phase is Pending instead of Bound. I1105 21:38:12.063] Nov 5 21:38:12.063: INFO: PersistentVolumeClaim pvc-rxsnk found but phase is Pending instead of Bound. I1105 21:38:14.144] Nov 5 21:38:14.143: INFO: PersistentVolumeClaim pvc-rxsnk found but phase is Pending instead of Bound. I1105 21:38:16.223] Nov 5 21:38:16.223: INFO: PersistentVolumeClaim pvc-rxsnk found but phase is Pending instead of Bound. I1105 21:38:18.304] Nov 5 21:38:18.304: INFO: PersistentVolumeClaim pvc-rxsnk found but phase is Pending instead of Bound. I1105 21:38:20.383] Nov 5 21:38:20.382: INFO: PersistentVolumeClaim pvc-rxsnk found but phase is Pending instead of Bound. I1105 21:38:22.462] Nov 5 21:38:22.462: INFO: PersistentVolumeClaim pvc-rxsnk found but phase is Pending instead of Bound. I1105 21:38:24.542] Nov 5 21:38:24.541: INFO: PersistentVolumeClaim pvc-rxsnk found but phase is Pending instead of Bound. I1105 21:38:26.623] Nov 5 21:38:26.623: INFO: PersistentVolumeClaim pvc-rxsnk found but phase is Pending instead of Bound. I1105 21:38:28.712] Nov 5 21:38:28.712: INFO: PersistentVolumeClaim pvc-rxsnk found but phase is Pending instead of Bound. I1105 21:38:30.791] Nov 5 21:38:30.791: INFO: PersistentVolumeClaim pvc-rxsnk found but phase is Pending instead of Bound. I1105 21:38:32.870] Nov 5 21:38:32.870: INFO: PersistentVolumeClaim pvc-rxsnk found but phase is Pending instead of Bound. I1105 21:38:34.949] Nov 5 21:38:34.949: INFO: PersistentVolumeClaim pvc-rxsnk found but phase is Pending instead of Bound. I1105 21:38:37.028] Nov 5 21:38:37.028: INFO: PersistentVolumeClaim pvc-rxsnk found but phase is Pending instead of Bound. I1105 21:38:39.108] Nov 5 21:38:39.107: INFO: PersistentVolumeClaim pvc-rxsnk found but phase is Pending instead of Bound. I1105 21:38:41.186] Nov 5 21:38:41.186: INFO: PersistentVolumeClaim pvc-rxsnk found but phase is Pending instead of Bound. I1105 21:38:43.266] Nov 5 21:38:43.265: INFO: PersistentVolumeClaim pvc-rxsnk found but phase is Pending instead of Bound. I1105 21:38:45.345] Nov 5 21:38:45.345: INFO: PersistentVolumeClaim pvc-rxsnk found but phase is Pending instead of Bound. I1105 21:38:47.423] Nov 5 21:38:47.423: INFO: PersistentVolumeClaim pvc-rxsnk found but phase is Pending instead of Bound. I1105 21:38:49.503] Nov 5 21:38:49.503: INFO: PersistentVolumeClaim pvc-rxsnk found but phase is Pending instead of Bound. I1105 21:38:51.582] Nov 5 21:38:51.582: INFO: PersistentVolumeClaim pvc-rxsnk found but phase is Pending instead of Bound. I1105 21:38:53.662] Nov 5 21:38:53.661: INFO: PersistentVolumeClaim pvc-rxsnk found but phase is Pending instead of Bound. I1105 21:38:55.741] Nov 5 21:38:55.741: INFO: PersistentVolumeClaim pvc-rxsnk found but phase is Pending instead of Bound. I1105 21:38:57.820] Nov 5 21:38:57.820: INFO: PersistentVolumeClaim pvc-rxsnk found but phase is Pending instead of Bound. I1105 21:38:59.899] Nov 5 21:38:59.899: INFO: PersistentVolumeClaim pvc-rxsnk found but phase is Pending instead of Bound. I1105 21:39:01.978] Nov 5 21:39:01.978: INFO: PersistentVolumeClaim pvc-rxsnk found but phase is Pending instead of Bound. I1105 21:39:04.059] Nov 5 21:39:04.058: INFO: PersistentVolumeClaim pvc-rxsnk found but phase is Pending instead of Bound. I1105 21:39:06.138] Nov 5 21:39:06.137: INFO: PersistentVolumeClaim pvc-rxsnk found but phase is Pending instead of Bound. I1105 21:39:08.217] Nov 5 21:39:08.217: INFO: PersistentVolumeClaim pvc-rxsnk found but phase is Pending instead of Bound. I1105 21:39:10.296] Nov 5 21:39:10.296: INFO: PersistentVolumeClaim pvc-rxsnk found but phase is Pending instead of Bound. I1105 21:39:12.376] Nov 5 21:39:12.375: INFO: PersistentVolumeClaim pvc-rxsnk found but phase is Pending instead of Bound. I1105 21:39:14.455] Nov 5 21:39:14.455: INFO: PersistentVolumeClaim pvc-rxsnk found but phase is Pending instead of Bound. I1105 21:39:16.538] Nov 5 21:39:16.538: INFO: PersistentVolumeClaim pvc-rxsnk found but phase is Pending instead of Bound. I1105 21:39:18.632] Nov 5 21:39:18.631: INFO: PersistentVolumeClaim pvc-rxsnk found but phase is Pending instead of Bound. I1105 21:39:20.632] Nov 5 21:39:20.632: INFO: Unexpected error occurred: PersistentVolumeClaims [pvc-rxsnk] not all in phase Bound within 5m0s I1105 21:39:20.632] [AfterEach] [multi-az] I1105 21:39:20.632] /go/src/sigs.k8s.io/azuredisk-csi-driver/vendor/k8s.io/kubernetes/test/e2e/framework/framework.go:151 I1105 21:39:20.633] STEP: Collecting events from namespace "azuredisk-3090". I1105 21:39:20.712] STEP: Found 3 events. I1105 21:39:20.712] Nov 5 21:39:20.712: INFO: At 2019-11-05 21:34:19 +0000 UTC - event for pvc-rxsnk: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "disk.csi.azure.com" or manually created by system administrator I1105 21:39:20.712] Nov 5 21:39:20.712: INFO: At 2019-11-05 21:34:19 +0000 UTC - event for pvc-rxsnk: {disk.csi.azure.com_csi-azuredisk-controller-6f6587bfc6-62nrt_3930bc69-0009-11ea-8cf5-829a27ff7d5a } Provisioning: External provisioner is provisioning volume for claim "azuredisk-3090/pvc-rxsnk" I1105 21:39:20.713] Nov 5 21:39:20.712: INFO: At 2019-11-05 21:34:19 +0000 UTC - event for pvc-rxsnk: {disk.csi.azure.com_csi-azuredisk-controller-6f6587bfc6-62nrt_3930bc69-0009-11ea-8cf5-829a27ff7d5a } ProvisioningFailed: failed to provision volume with StorageClass "azuredisk-3090-disk.csi.azure.com-dynamic-sc-55fv2": rpc error: code = Unknown desc = compute.DisksClient#CreateOrUpdate: Failure sending request: StatusCode=400 -- Original Error: Code="InvalidAvailabilityZone" Message="The zone(s) '' for resource 'Microsoft.Compute/disks/pvc-c4dba06b-be89-41c6-a771-815a56bb0c81' is not supported. The supported zones for location 'eastus2' are '1,2,3'" I1105 21:39:20.790] Nov 5 21:39:20.790: INFO: POD NODE PHASE GRACE CONDITIONS I1105 21:39:20.790] Nov 5 21:39:20.790: INFO: I1105 21:39:21.248] Nov 5 21:39:21.248: INFO: I1105 21:39:21.248] Logging node info for node k8s-agentpool-97435445-vmss000000 I1105 21:39:21.330] Nov 5 21:39:21.327: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-agentpool-97435445-vmss000000,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-agentpool-97435445-vmss000000,UID:e955dc47-8c8f-4142-9241-3d409f32e42d,ResourceVersion:17244,Generation:0,CreationTimestamp:2019-11-05 20:07:34 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{agentpool: agentpool,beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_D4s_v3,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 0,kubernetes.azure.com/cluster: kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b,kubernetes.azure.com/role: agent,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-agentpool-97435445-vmss000000,kubernetes.io/os: linux,kubernetes.io/role: agent,node-role.kubernetes.io/agent: ,storageprofile: managed,storagetier: Premium_LRS,topology.disk.csi.azure.com/zone: 0,},Annotations:map[string]string{csi.volume.kubernetes.io/nodeid: {"disk.csi.azure.com":"k8s-agentpool-97435445-vmss000000"},node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.2.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool-97435445-vmss/virtualMachines/0,Unschedulable:false,Taints:[],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-csi-disk.csi.azure.com: {{8 0} {} 8 DecimalSI},cpu: {{4 0} {} 4 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{16797040640 0} {} BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-csi-disk.csi.azure.com: {{8 0} {} 8 DecimalSI},cpu: {{4 0} {} 4 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{16010608640 0} {} BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 20:08:01 +0000 UTC 2019-11-05 20:08:01 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 21:38:50 +0000 UTC 2019-11-05 20:07:27 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 21:38:50 +0000 UTC 2019-11-05 20:07:27 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 21:38:50 +0000 UTC 2019-11-05 20:07:27 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 21:38:50 +0000 UTC 2019-11-05 20:07:55 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.0.4} {Hostname k8s-agentpool-97435445-vmss000000}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:bfb55325809340228149d2d93f50df58,SystemUUID:CB2ED036-0439-F641-BC7F-640C3638C027,BootID:a917f4d1-549d-471d-b7dc-47ae1715d97d,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[k8sprowinternal.azurecr.io/azuredisk-csi@sha256:75d4ff5d7cd845d6e810c67796ecb037e852fb849ec1eb5b3194564008e6d1e0 k8sprowinternal.azurecr.io/azuredisk-csi:e2e-3e3ed045c6faa509dbd8d2890ee9f56aa2934887] 286930414} {[k8s.gcr.io/kubernetes-dashboard-amd64@sha256:0ae6b69432e78069c5ce2bcde0fe409c5c4d6f0f4d9cd50a17974fea38898747 k8s.gcr.io/kubernetes-dashboard-amd64:v1.10.1] 121711221} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/metrics-server-amd64@sha256:49a9f12f7067d11f42c803dbe61ed2c1299959ad85cb315b25ff7eef8e6b8892 k8s.gcr.io/metrics-server-amd64:v0.2.1] 42541759} {[k8s.gcr.io/coredns@sha256:e83beb5e43f8513fa735e77ffc5859640baea30a882a11cc75c4c3244a737d3c k8s.gcr.io/coredns:1.5.0] 42488424} {[mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume@sha256:4fd30d43947d4a54fc89ead7985beecfd3c9b2a93a0655a373b1608ab90bd5af mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume:v0.0.7] 22909487} {[quay.io/k8scsi/livenessprobe@sha256:8c9c8a0f330acc2e558ae6624309447d02522e76aa35af23fa19b88abf4381cb quay.io/k8scsi/livenessprobe:v1.0.2] 17415381} {[quay.io/k8scsi/csi-node-driver-registrar@sha256:13daf82fb99e951a4bff8ae5fc7c17c3a8fe7130be6400990d8f6076c32d4599 quay.io/k8scsi/csi-node-driver-registrar:v1.1.0] 15815995} {[mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume@sha256:23d8c6033f02a1ecad05127ebdc931bb871264228661bc122704b0974e4d9fdd mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume:1.0.8] 1159025} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 21:39:21.330] Nov 5 21:39:21.327: INFO: I1105 21:39:21.330] Logging kubelet events for node k8s-agentpool-97435445-vmss000000 I1105 21:39:21.405] Nov 5 21:39:21.405: INFO: I1105 21:39:21.406] Logging pods the kubelet thinks is on node k8s-agentpool-97435445-vmss000000 I1105 21:39:21.639] Nov 5 21:39:21.638: INFO: kubernetes-dashboard-66dd8b8df7-t6j4l started at 2019-11-05 20:08:01 +0000 UTC (0+1 container statuses recorded) I1105 21:39:21.639] Nov 5 21:39:21.639: INFO: Container kubernetes-dashboard ready: true, restart count 0 I1105 21:39:21.639] Nov 5 21:39:21.639: INFO: csi-azuredisk-node-5wwhz started at 2019-11-05 20:16:09 +0000 UTC (0+3 container statuses recorded) I1105 21:39:21.639] Nov 5 21:39:21.639: INFO: Container azuredisk ready: true, restart count 0 I1105 21:39:21.640] Nov 5 21:39:21.639: INFO: Container liveness-probe ready: true, restart count 0 I1105 21:39:21.640] Nov 5 21:39:21.639: INFO: Container node-driver-registrar ready: true, restart count 0 I1105 21:39:21.640] Nov 5 21:39:21.639: INFO: blobfuse-flexvol-installer-p7hdb started at 2019-11-05 20:08:01 +0000 UTC (0+1 container statuses recorded) I1105 21:39:21.640] Nov 5 21:39:21.639: INFO: Container blobfuse-flexvol-installer ready: true, restart count 0 I1105 21:39:21.640] Nov 5 21:39:21.639: INFO: coredns-7f68dcdbdb-zh8g5 started at 2019-11-05 20:08:01 +0000 UTC (0+1 container statuses recorded) I1105 21:39:21.640] Nov 5 21:39:21.639: INFO: Container coredns ready: true, restart count 0 I1105 21:39:21.641] Nov 5 21:39:21.639: INFO: azure-ip-masq-agent-m7tv9 started at 2019-11-05 20:08:01 +0000 UTC (0+1 container statuses recorded) I1105 21:39:21.641] Nov 5 21:39:21.639: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 21:39:21.641] Nov 5 21:39:21.639: INFO: keyvault-flexvolume-smspt started at 2019-11-05 20:08:01 +0000 UTC (0+1 container statuses recorded) I1105 21:39:21.641] Nov 5 21:39:21.639: INFO: Container keyvault-flexvolume ready: true, restart count 0 I1105 21:39:21.641] Nov 5 21:39:21.639: INFO: metrics-server-864ffbc5c-wl84z started at 2019-11-05 20:08:01 +0000 UTC (0+1 container statuses recorded) I1105 21:39:21.641] Nov 5 21:39:21.639: INFO: Container metrics-server ready: true, restart count 0 I1105 21:39:21.642] Nov 5 21:39:21.639: INFO: kube-proxy-8mt7f started at 2019-11-05 20:08:01 +0000 UTC (0+1 container statuses recorded) I1105 21:39:21.642] Nov 5 21:39:21.639: INFO: Container kube-proxy ready: true, restart count 0 I1105 21:39:21.979] Nov 5 21:39:21.979: INFO: I1105 21:39:21.980] Latency metrics for node k8s-agentpool-97435445-vmss000000 I1105 21:39:21.980] Nov 5 21:39:21.979: INFO: I1105 21:39:21.980] Logging node info for node k8s-agentpool-97435445-vmss000001 I1105 21:39:22.063] Nov 5 21:39:22.059: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-agentpool-97435445-vmss000001,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-agentpool-97435445-vmss000001,UID:75d3cc50-ca6d-45e0-a42a-6090e8d537dd,ResourceVersion:17214,Generation:0,CreationTimestamp:2019-11-05 20:07:34 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{agentpool: agentpool,beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_D4s_v3,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 1,kubernetes.azure.com/cluster: kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b,kubernetes.azure.com/role: agent,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-agentpool-97435445-vmss000001,kubernetes.io/os: linux,kubernetes.io/role: agent,node-role.kubernetes.io/agent: ,storageprofile: managed,storagetier: Premium_LRS,topology.disk.csi.azure.com/zone: 1,},Annotations:map[string]string{csi.volume.kubernetes.io/nodeid: {"disk.csi.azure.com":"k8s-agentpool-97435445-vmss000001"},node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.0.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool-97435445-vmss/virtualMachines/1,Unschedulable:false,Taints:[],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-csi-disk.csi.azure.com: {{8 0} {} 8 DecimalSI},cpu: {{4 0} {} 4 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{16797044736 0} {} 16403364Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-csi-disk.csi.azure.com: {{8 0} {} 8 DecimalSI},cpu: {{4 0} {} 4 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{16010612736 0} {} 15635364Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 20:08:25 +0000 UTC 2019-11-05 20:08:25 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 21:38:39 +0000 UTC 2019-11-05 20:07:26 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 21:38:39 +0000 UTC 2019-11-05 20:07:26 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 21:38:39 +0000 UTC 2019-11-05 20:07:26 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 21:38:39 +0000 UTC 2019-11-05 20:07:55 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.0.5} {Hostname k8s-agentpool-97435445-vmss000001}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:e78def54b2f545c6b94d88bbba523c60,SystemUUID:40AE8188-86AB-D848-84A4-D9254E1258BC,BootID:8587ec06-f4f4-43c1-86cd-06c295b05b63,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[k8sprowinternal.azurecr.io/azuredisk-csi@sha256:75d4ff5d7cd845d6e810c67796ecb037e852fb849ec1eb5b3194564008e6d1e0 k8sprowinternal.azurecr.io/azuredisk-csi:e2e-3e3ed045c6faa509dbd8d2890ee9f56aa2934887] 286930414} {[gcr.io/kubernetes-helm/tiller@sha256:f6d8f4ab9ba993b5f5b60a6edafe86352eabe474ffeb84cb6c79b8866dce45d1 gcr.io/kubernetes-helm/tiller:v2.11.0] 71821984} {[quay.io/k8scsi/csi-provisioner@sha256:d657c839dce87324fe2b677302913f9386f885f8746be7bea0ced5b0844e3433 quay.io/k8scsi/csi-provisioner:v1.3.1] 50596998} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[quay.io/k8scsi/csi-attacher@sha256:26fccd7a99d973845df1193b46ebdcc6ab8dc5f6e6be319750c471fce1742d13 quay.io/k8scsi/csi-attacher:v1.2.0] 46226754} {[quay.io/k8scsi/csi-cluster-driver-registrar@sha256:fafd75ae5442f192cfa8c2e792903aee30d5884b62e802e4464b0a895d21e3ef quay.io/k8scsi/csi-cluster-driver-registrar:v1.0.1] 45874691} {[quay.io/k8scsi/csi-snapshotter@sha256:a49e0da1af6f2bf717e41ba1eee8b5e6a1cbd66a709dd92cc43fe475fe2589eb quay.io/k8scsi/csi-snapshotter:v1.1.0] 44325352} {[mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume@sha256:4fd30d43947d4a54fc89ead7985beecfd3c9b2a93a0655a373b1608ab90bd5af mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume:v0.0.7] 22909487} {[quay.io/k8scsi/livenessprobe@sha256:8c9c8a0f330acc2e558ae6624309447d02522e76aa35af23fa19b88abf4381cb quay.io/k8scsi/livenessprobe:v1.0.2] 17415381} {[quay.io/k8scsi/csi-node-driver-registrar@sha256:13daf82fb99e951a4bff8ae5fc7c17c3a8fe7130be6400990d8f6076c32d4599 quay.io/k8scsi/csi-node-driver-registrar:v1.1.0] 15815995} {[quay.io/k8scsi/livenessprobe@sha256:dde617756e0f602adc566ab71fd885f1dad451ad3fb063ac991c95a2ff47aea5 quay.io/k8scsi/livenessprobe:v1.1.0] 14967303} {[mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume@sha256:23d8c6033f02a1ecad05127ebdc931bb871264228661bc122704b0974e4d9fdd mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume:1.0.8] 1159025} {[busybox@sha256:8ccbac733d19c0dd4d70b4f0c1e12245b5fa3ad24758a11035ee505c629c0796 busybox:1.29] 1154361} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 21:39:22.063] Nov 5 21:39:22.059: INFO: I1105 21:39:22.063] Logging kubelet events for node k8s-agentpool-97435445-vmss000001 I1105 21:39:22.139] Nov 5 21:39:22.139: INFO: I1105 21:39:22.139] Logging pods the kubelet thinks is on node k8s-agentpool-97435445-vmss000001 I1105 21:39:22.372] Nov 5 21:39:22.372: INFO: kube-proxy-kjrpb started at 2019-11-05 20:08:00 +0000 UTC (0+1 container statuses recorded) I1105 21:39:22.373] Nov 5 21:39:22.372: INFO: Container kube-proxy ready: true, restart count 0 I1105 21:39:22.373] Nov 5 21:39:22.372: INFO: tiller-deploy-54c96cb5df-pt7x6 started at 2019-11-05 20:08:00 +0000 UTC (0+1 container statuses recorded) I1105 21:39:22.373] Nov 5 21:39:22.372: INFO: Container tiller ready: true, restart count 0 I1105 21:39:22.373] Nov 5 21:39:22.372: INFO: csi-azuredisk-node-q2dmt started at 2019-11-05 20:16:09 +0000 UTC (0+3 container statuses recorded) I1105 21:39:22.373] Nov 5 21:39:22.372: INFO: Container azuredisk ready: true, restart count 0 I1105 21:39:22.373] Nov 5 21:39:22.372: INFO: Container liveness-probe ready: true, restart count 0 I1105 21:39:22.374] Nov 5 21:39:22.372: INFO: Container node-driver-registrar ready: true, restart count 0 I1105 21:39:22.374] Nov 5 21:39:22.372: INFO: csi-azuredisk-controller-6f6587bfc6-62nrt started at 2019-11-05 20:16:09 +0000 UTC (0+6 container statuses recorded) I1105 21:39:22.374] Nov 5 21:39:22.372: INFO: Container azuredisk ready: true, restart count 0 I1105 21:39:22.374] Nov 5 21:39:22.372: INFO: Container cluster-driver-registrar ready: true, restart count 0 I1105 21:39:22.374] Nov 5 21:39:22.372: INFO: Container csi-attacher ready: true, restart count 0 I1105 21:39:22.374] Nov 5 21:39:22.372: INFO: Container csi-provisioner ready: true, restart count 0 I1105 21:39:22.374] Nov 5 21:39:22.372: INFO: Container csi-snapshotter ready: true, restart count 0 I1105 21:39:22.375] Nov 5 21:39:22.372: INFO: Container liveness-probe ready: true, restart count 0 I1105 21:39:22.375] Nov 5 21:39:22.372: INFO: blobfuse-flexvol-installer-j4s7t started at 2019-11-05 20:07:59 +0000 UTC (0+1 container statuses recorded) I1105 21:39:22.375] Nov 5 21:39:22.372: INFO: Container blobfuse-flexvol-installer ready: true, restart count 0 I1105 21:39:22.375] Nov 5 21:39:22.372: INFO: azure-ip-masq-agent-gdhcq started at 2019-11-05 20:07:59 +0000 UTC (0+1 container statuses recorded) I1105 21:39:22.375] Nov 5 21:39:22.372: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 21:39:22.375] Nov 5 21:39:22.372: INFO: keyvault-flexvolume-6bffx started at 2019-11-05 20:07:59 +0000 UTC (0+1 container statuses recorded) I1105 21:39:22.375] Nov 5 21:39:22.372: INFO: Container keyvault-flexvolume ready: true, restart count 0 I1105 21:39:22.723] Nov 5 21:39:22.722: INFO: I1105 21:39:22.723] Latency metrics for node k8s-agentpool-97435445-vmss000001 I1105 21:39:22.723] Nov 5 21:39:22.722: INFO: I1105 21:39:22.723] Logging node info for node k8s-master-97435445-0 I1105 21:39:22.804] Nov 5 21:39:22.802: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-97435445-0,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-97435445-0,UID:df3ae36f-961c-4729-a8f5-34cda54f4ff9,ResourceVersion:17302,Generation:0,CreationTimestamp:2019-11-05 20:08:36 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 2,kubernetes.azure.com/cluster: kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-97435445-0,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.4.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b/providers/Microsoft.Compute/virtualMachines/k8s-master-97435445-0,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 20:08:46 +0000 UTC 2019-11-05 20:08:46 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 21:39:09 +0000 UTC 2019-11-05 20:08:28 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 21:39:09 +0000 UTC 2019-11-05 20:08:28 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 21:39:09 +0000 UTC 2019-11-05 20:08:28 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 21:39:09 +0000 UTC 2019-11-05 20:08:36 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.255.5} {Hostname k8s-master-97435445-0}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:3c381865192341c2868672e8626cef7d,SystemUUID:F3D259D3-7EB8-B844-A1F0-F954DCD0D76B,BootID:4948329e-cf57-48c3-aa83-5297bef70370,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 21:39:22.804] Nov 5 21:39:22.802: INFO: I1105 21:39:22.804] Logging kubelet events for node k8s-master-97435445-0 I1105 21:39:22.891] Nov 5 21:39:22.890: INFO: I1105 21:39:22.891] Logging pods the kubelet thinks is on node k8s-master-97435445-0 I1105 21:39:23.049] Nov 5 21:39:23.048: INFO: cloud-controller-manager-k8s-master-97435445-0 started at 2019-11-05 20:08:22 +0000 UTC (0+1 container statuses recorded) I1105 21:39:23.049] Nov 5 21:39:23.048: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 21:39:23.049] Nov 5 21:39:23.048: INFO: kube-addon-manager-k8s-master-97435445-0 started at 2019-11-05 20:08:48 +0000 UTC (0+1 container statuses recorded) I1105 21:39:23.049] Nov 5 21:39:23.048: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 21:39:23.049] Nov 5 21:39:23.048: INFO: kube-apiserver-k8s-master-97435445-0 started at 2019-11-05 20:08:22 +0000 UTC (0+1 container statuses recorded) I1105 21:39:23.050] Nov 5 21:39:23.048: INFO: Container kube-apiserver ready: true, restart count 0 I1105 21:39:23.050] Nov 5 21:39:23.048: INFO: kube-controller-manager-k8s-master-97435445-0 started at 2019-11-05 20:08:22 +0000 UTC (0+1 container statuses recorded) I1105 21:39:23.050] Nov 5 21:39:23.048: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 21:39:23.050] Nov 5 21:39:23.048: INFO: kube-scheduler-k8s-master-97435445-0 started at 2019-11-05 20:08:22 +0000 UTC (0+1 container statuses recorded) I1105 21:39:23.050] Nov 5 21:39:23.048: INFO: Container kube-scheduler ready: true, restart count 0 I1105 21:39:23.050] Nov 5 21:39:23.048: INFO: azure-ip-masq-agent-q6dcl started at 2019-11-05 20:08:38 +0000 UTC (0+1 container statuses recorded) I1105 21:39:23.050] Nov 5 21:39:23.048: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 21:39:23.051] Nov 5 21:39:23.048: INFO: kube-proxy-5wnd2 started at 2019-11-05 20:08:38 +0000 UTC (0+1 container statuses recorded) I1105 21:39:23.051] Nov 5 21:39:23.048: INFO: Container kube-proxy ready: true, restart count 0 I1105 21:39:23.385] Nov 5 21:39:23.385: INFO: I1105 21:39:23.386] Latency metrics for node k8s-master-97435445-0 I1105 21:39:23.386] Nov 5 21:39:23.385: INFO: I1105 21:39:23.386] Logging node info for node k8s-master-97435445-1 I1105 21:39:23.466] Nov 5 21:39:23.464: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-97435445-1,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-97435445-1,UID:05534a67-d493-4ab5-9b0f-5ff5490e234f,ResourceVersion:17208,Generation:0,CreationTimestamp:2019-11-05 20:07:34 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 0,kubernetes.azure.com/cluster: kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-97435445-1,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.1.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b/providers/Microsoft.Compute/virtualMachines/k8s-master-97435445-1,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 20:08:26 +0000 UTC 2019-11-05 20:08:26 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 21:38:38 +0000 UTC 2019-11-05 20:07:28 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 21:38:38 +0000 UTC 2019-11-05 20:07:28 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 21:38:38 +0000 UTC 2019-11-05 20:07:28 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 21:38:38 +0000 UTC 2019-11-05 20:07:55 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{Hostname k8s-master-97435445-1} {InternalIP 10.240.255.6}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:079b605103b74acabac18fd2ea63368a,SystemUUID:38220EEC-BEFF-4C46-AA05-5A214AAFDC28,BootID:b1b45df0-d925-4bcc-9830-0202df36ebb4,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 21:39:23.467] Nov 5 21:39:23.464: INFO: I1105 21:39:23.467] Logging kubelet events for node k8s-master-97435445-1 I1105 21:39:23.544] Nov 5 21:39:23.544: INFO: I1105 21:39:23.544] Logging pods the kubelet thinks is on node k8s-master-97435445-1 I1105 21:39:23.703] Nov 5 21:39:23.703: INFO: kube-apiserver-k8s-master-97435445-1 started at 2019-11-05 20:07:22 +0000 UTC (0+1 container statuses recorded) I1105 21:39:23.703] Nov 5 21:39:23.703: INFO: Container kube-apiserver ready: true, restart count 0 I1105 21:39:23.703] Nov 5 21:39:23.703: INFO: kube-controller-manager-k8s-master-97435445-1 started at 2019-11-05 20:07:22 +0000 UTC (0+1 container statuses recorded) I1105 21:39:23.704] Nov 5 21:39:23.703: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 21:39:23.704] Nov 5 21:39:23.703: INFO: kube-scheduler-k8s-master-97435445-1 started at 2019-11-05 20:07:22 +0000 UTC (0+1 container statuses recorded) I1105 21:39:23.704] Nov 5 21:39:23.703: INFO: Container kube-scheduler ready: true, restart count 0 I1105 21:39:23.704] Nov 5 21:39:23.703: INFO: azure-ip-masq-agent-n29bm started at 2019-11-05 20:07:59 +0000 UTC (0+1 container statuses recorded) I1105 21:39:23.704] Nov 5 21:39:23.703: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 21:39:23.704] Nov 5 21:39:23.703: INFO: kube-proxy-h4s2c started at 2019-11-05 20:08:00 +0000 UTC (0+1 container statuses recorded) I1105 21:39:23.704] Nov 5 21:39:23.703: INFO: Container kube-proxy ready: true, restart count 0 I1105 21:39:23.705] Nov 5 21:39:23.703: INFO: cloud-controller-manager-k8s-master-97435445-1 started at 2019-11-05 20:07:22 +0000 UTC (0+1 container statuses recorded) I1105 21:39:23.705] Nov 5 21:39:23.703: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 21:39:23.705] Nov 5 21:39:23.703: INFO: kube-addon-manager-k8s-master-97435445-1 started at 2019-11-05 20:07:22 +0000 UTC (0+1 container statuses recorded) I1105 21:39:23.705] Nov 5 21:39:23.703: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 21:39:24.032] Nov 5 21:39:24.031: INFO: I1105 21:39:24.032] Latency metrics for node k8s-master-97435445-1 I1105 21:39:24.032] Nov 5 21:39:24.031: INFO: I1105 21:39:24.032] Logging node info for node k8s-master-97435445-2 I1105 21:39:24.113] Nov 5 21:39:24.110: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-97435445-2,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-97435445-2,UID:fa55c895-1a93-4743-885e-03fce84b5332,ResourceVersion:17260,Generation:0,CreationTimestamp:2019-11-05 20:09:19 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 0,kubernetes.azure.com/cluster: kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-97435445-2,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.5.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b/providers/Microsoft.Compute/virtualMachines/k8s-master-97435445-2,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 20:09:19 +0000 UTC 2019-11-05 20:09:19 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 21:38:55 +0000 UTC 2019-11-05 20:09:12 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 21:38:55 +0000 UTC 2019-11-05 20:09:12 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 21:38:55 +0000 UTC 2019-11-05 20:09:12 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 21:38:55 +0000 UTC 2019-11-05 20:09:19 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.255.7} {Hostname k8s-master-97435445-2}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:d19f6709138f496da43c095cec7b2360,SystemUUID:10116777-A8D7-CB4A-A8B7-7EBCF2F23B62,BootID:50cd14a9-5497-4972-bf60-cf59a0cbcb5f,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 21:39:24.113] Nov 5 21:39:24.111: INFO: I1105 21:39:24.113] Logging kubelet events for node k8s-master-97435445-2 I1105 21:39:24.190] Nov 5 21:39:24.189: INFO: I1105 21:39:24.190] Logging pods the kubelet thinks is on node k8s-master-97435445-2 I1105 21:39:24.347] Nov 5 21:39:24.346: INFO: kube-controller-manager-k8s-master-97435445-2 started at 2019-11-05 20:09:05 +0000 UTC (0+1 container statuses recorded) I1105 21:39:24.347] Nov 5 21:39:24.346: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 21:39:24.347] Nov 5 21:39:24.346: INFO: kube-scheduler-k8s-master-97435445-2 started at 2019-11-05 20:09:05 +0000 UTC (0+1 container statuses recorded) I1105 21:39:24.347] Nov 5 21:39:24.346: INFO: Container kube-scheduler ready: true, restart count 0 I1105 21:39:24.347] Nov 5 21:39:24.346: INFO: cloud-controller-manager-k8s-master-97435445-2 started at 2019-11-05 20:09:05 +0000 UTC (0+1 container statuses recorded) I1105 21:39:24.348] Nov 5 21:39:24.347: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 21:39:24.348] Nov 5 21:39:24.347: INFO: kube-addon-manager-k8s-master-97435445-2 started at 2019-11-05 20:09:05 +0000 UTC (0+1 container statuses recorded) I1105 21:39:24.348] Nov 5 21:39:24.347: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 21:39:24.348] Nov 5 21:39:24.347: INFO: azure-ip-masq-agent-sq7p5 started at 2019-11-05 20:09:19 +0000 UTC (0+1 container statuses recorded) I1105 21:39:24.348] Nov 5 21:39:24.347: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 21:39:24.349] Nov 5 21:39:24.347: INFO: kube-proxy-pz426 started at 2019-11-05 20:09:19 +0000 UTC (0+1 container statuses recorded) I1105 21:39:24.349] Nov 5 21:39:24.347: INFO: Container kube-proxy ready: true, restart count 0 I1105 21:39:24.349] Nov 5 21:39:24.347: INFO: kube-apiserver-k8s-master-97435445-2 started at 2019-11-05 20:09:05 +0000 UTC (0+1 container statuses recorded) I1105 21:39:24.349] Nov 5 21:39:24.347: INFO: Container kube-apiserver ready: true, restart count 0 I1105 21:39:24.676] Nov 5 21:39:24.676: INFO: I1105 21:39:24.676] Latency metrics for node k8s-master-97435445-2 I1105 21:39:24.676] Nov 5 21:39:24.676: INFO: I1105 21:39:24.677] Logging node info for node k8s-master-97435445-3 I1105 21:39:24.757] Nov 5 21:39:24.754: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-97435445-3,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-97435445-3,UID:8ffd1d0d-4487-4680-a2f6-e724387be637,ResourceVersion:17255,Generation:0,CreationTimestamp:2019-11-05 20:09:21 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 1,kubernetes.azure.com/cluster: kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-97435445-3,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.6.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b/providers/Microsoft.Compute/virtualMachines/k8s-master-97435445-3,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 20:09:39 +0000 UTC 2019-11-05 20:09:39 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 21:38:54 +0000 UTC 2019-11-05 20:09:18 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 21:38:54 +0000 UTC 2019-11-05 20:09:18 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 21:38:54 +0000 UTC 2019-11-05 20:09:18 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 21:38:54 +0000 UTC 2019-11-05 20:09:21 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.255.8} {Hostname k8s-master-97435445-3}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:266064173bfb4297898c6c25769e8b20,SystemUUID:E6E44241-5015-7740-9A81-A668A8427206,BootID:579d658c-b0ad-47f1-a993-1ccea2b55789,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 21:39:24.757] Nov 5 21:39:24.755: INFO: I1105 21:39:24.757] Logging kubelet events for node k8s-master-97435445-3 I1105 21:39:24.833] Nov 5 21:39:24.833: INFO: I1105 21:39:24.834] Logging pods the kubelet thinks is on node k8s-master-97435445-3 I1105 21:39:24.991] Nov 5 21:39:24.991: INFO: kube-apiserver-k8s-master-97435445-3 started at 2019-11-05 20:09:05 +0000 UTC (0+1 container statuses recorded) I1105 21:39:24.991] Nov 5 21:39:24.991: INFO: Container kube-apiserver ready: true, restart count 0 I1105 21:39:24.992] Nov 5 21:39:24.991: INFO: kube-controller-manager-k8s-master-97435445-3 started at 2019-11-05 20:09:05 +0000 UTC (0+1 container statuses recorded) I1105 21:39:24.992] Nov 5 21:39:24.991: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 21:39:24.992] Nov 5 21:39:24.991: INFO: kube-scheduler-k8s-master-97435445-3 started at 2019-11-05 20:09:05 +0000 UTC (0+1 container statuses recorded) I1105 21:39:24.992] Nov 5 21:39:24.991: INFO: Container kube-scheduler ready: true, restart count 0 I1105 21:39:24.992] Nov 5 21:39:24.991: INFO: azure-ip-masq-agent-747cl started at 2019-11-05 20:09:21 +0000 UTC (0+1 container statuses recorded) I1105 21:39:24.992] Nov 5 21:39:24.991: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 21:39:24.993] Nov 5 21:39:24.991: INFO: kube-proxy-k45g8 started at 2019-11-05 20:09:21 +0000 UTC (0+1 container statuses recorded) I1105 21:39:24.993] Nov 5 21:39:24.991: INFO: Container kube-proxy ready: true, restart count 0 I1105 21:39:24.993] Nov 5 21:39:24.991: INFO: cloud-controller-manager-k8s-master-97435445-3 started at 2019-11-05 20:09:04 +0000 UTC (0+1 container statuses recorded) I1105 21:39:24.993] Nov 5 21:39:24.991: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 21:39:24.993] Nov 5 21:39:24.991: INFO: kube-addon-manager-k8s-master-97435445-3 started at 2019-11-05 20:09:05 +0000 UTC (0+1 container statuses recorded) I1105 21:39:24.994] Nov 5 21:39:24.991: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 21:39:25.254] Nov 5 21:39:25.254: INFO: I1105 21:39:25.254] Latency metrics for node k8s-master-97435445-3 I1105 21:39:25.255] Nov 5 21:39:25.254: INFO: I1105 21:39:25.255] Logging node info for node k8s-master-97435445-4 I1105 21:39:25.335] Nov 5 21:39:25.333: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-97435445-4,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-97435445-4,UID:6d547b41-3f92-4cc4-a2d7-dda2071776a2,ResourceVersion:17296,Generation:0,CreationTimestamp:2019-11-05 20:08:36 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 1,kubernetes.azure.com/cluster: kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-97435445-4,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.3.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b/providers/Microsoft.Compute/virtualMachines/k8s-master-97435445-4,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 20:08:59 +0000 UTC 2019-11-05 20:08:59 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 21:39:08 +0000 UTC 2019-11-05 20:08:29 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 21:39:08 +0000 UTC 2019-11-05 20:08:29 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 21:39:08 +0000 UTC 2019-11-05 20:08:29 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 21:39:08 +0000 UTC 2019-11-05 20:08:36 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.255.9} {Hostname k8s-master-97435445-4}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:56b393c61eb34ba49ebda5279eeedb30,SystemUUID:FD239C23-618C-0049-8734-9AF3FDC9E3C1,BootID:f342ea44-e719-4513-807d-5def7ac606e2,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 21:39:25.336] Nov 5 21:39:25.333: INFO: I1105 21:39:25.336] Logging kubelet events for node k8s-master-97435445-4 I1105 21:39:25.412] Nov 5 21:39:25.412: INFO: I1105 21:39:25.412] Logging pods the kubelet thinks is on node k8s-master-97435445-4 I1105 21:39:25.497] Nov 5 21:39:25.496: INFO: cloud-controller-manager-k8s-master-97435445-4 started at 2019-11-05 20:08:46 +0000 UTC (0+1 container statuses recorded) I1105 21:39:25.497] Nov 5 21:39:25.496: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 21:39:25.497] Nov 5 21:39:25.496: INFO: kube-addon-manager-k8s-master-97435445-4 started at 2019-11-05 20:08:23 +0000 UTC (0+1 container statuses recorded) I1105 21:39:25.497] Nov 5 21:39:25.496: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 21:39:25.498] Nov 5 21:39:25.496: INFO: kube-apiserver-k8s-master-97435445-4 started at 2019-11-05 20:08:23 +0000 UTC (0+1 container statuses recorded) I1105 21:39:25.498] Nov 5 21:39:25.496: INFO: Container kube-apiserver ready: true, restart count 0 I1105 21:39:25.498] Nov 5 21:39:25.496: INFO: kube-controller-manager-k8s-master-97435445-4 started at 2019-11-05 20:08:23 +0000 UTC (0+1 container statuses recorded) I1105 21:39:25.498] Nov 5 21:39:25.496: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 21:39:25.499] Nov 5 21:39:25.496: INFO: kube-scheduler-k8s-master-97435445-4 started at 2019-11-05 20:08:23 +0000 UTC (0+1 container statuses recorded) I1105 21:39:25.499] Nov 5 21:39:25.496: INFO: Container kube-scheduler ready: true, restart count 0 I1105 21:39:25.499] Nov 5 21:39:25.496: INFO: kube-proxy-vfbfr started at 2019-11-05 20:08:36 +0000 UTC (0+1 container statuses recorded) I1105 21:39:25.499] Nov 5 21:39:25.496: INFO: Container kube-proxy ready: true, restart count 0 I1105 21:39:25.499] Nov 5 21:39:25.496: INFO: azure-ip-masq-agent-kmkll started at 2019-11-05 20:08:36 +0000 UTC (0+1 container statuses recorded) I1105 21:39:25.500] Nov 5 21:39:25.496: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 21:39:25.747] Nov 5 21:39:25.747: INFO: I1105 21:39:25.747] Latency metrics for node k8s-master-97435445-4 I1105 21:39:25.747] Nov 5 21:39:25.747: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready I1105 21:39:25.828] STEP: Destroying namespace "azuredisk-3090" for this suite. I1105 21:39:40.070] Nov 5 21:39:40.070: INFO: Waiting up to 30s for server preferred namespaced resources to be successfully discovered I1105 21:39:43.177] Nov 5 21:39:43.177: INFO: namespace azuredisk-3090 deletion completed in 17.349357898s I1105 21:39:43.177] I1105 21:39:43.177] • Failure [324.073 seconds] I1105 21:39:43.177] [azuredisk-csi-e2e] Dynamic Provisioning I1105 21:39:43.178] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:33 I1105 21:39:43.178] [multi-az] I1105 21:39:43.178] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:40 I1105 21:39:43.178] should delete PV with reclaimPolicy "Delete" [It] I1105 21:39:43.178] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:221 I1105 21:39:43.178] I1105 21:39:43.178] Unexpected error: I1105 21:39:43.178] <*errors.errorString | 0xc000a3cf40>: { I1105 21:39:43.179] s: "PersistentVolumeClaims [pvc-rxsnk] not all in phase Bound within 5m0s", I1105 21:39:43.179] } I1105 21:39:43.179] PersistentVolumeClaims [pvc-rxsnk] not all in phase Bound within 5m0s I1105 21:39:43.179] occurred I1105 21:39:43.179] I1105 21:39:43.179] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/testsuites/testsuites.go:278 I1105 21:39:43.179] ------------------------------ I1105 21:39:43.179] [azuredisk-csi-e2e] Dynamic Provisioning [multi-az] I1105 21:39:43.179] should create a volume on demand and mount it as readOnly in a pod I1105 21:39:43.180] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:126 I1105 21:39:43.180] [BeforeEach] [multi-az] I1105 21:39:43.180] /go/src/sigs.k8s.io/azuredisk-csi-driver/vendor/k8s.io/kubernetes/test/e2e/framework/framework.go:150 I1105 21:39:43.180] STEP: Creating a kubernetes client I1105 21:39:43.180] Nov 5 21:39:43.177: INFO: >>> kubeConfig: /workspace/aks971561099/kubeconfig/kubeconfig.eastus2.json I1105 21:39:43.180] STEP: Building a namespace api object, basename azuredisk I1105 21:39:43.417] STEP: Binding the e2e-test-privileged-psp PodSecurityPolicy to the default service account in azuredisk-8482 I1105 21:39:43.680] STEP: Waiting for a default service account to be provisioned in namespace I1105 21:39:43.756] [BeforeEach] [multi-az] I1105 21:39:43.757] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:58 I1105 21:39:43.757] [It] should create a volume on demand and mount it as readOnly in a pod I1105 21:39:43.757] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:126 I1105 21:39:43.757] STEP: setting up the StorageClass I1105 21:39:43.757] STEP: creating a StorageClass I1105 21:39:43.844] STEP: setting up the PVC and PV I1105 21:39:43.845] STEP: creating a PVC I1105 21:39:43.931] STEP: waiting for PVC to be in phase "Bound" I1105 21:39:43.931] Nov 5 21:39:43.930: INFO: Waiting up to 5m0s for PersistentVolumeClaims [pvc-ckvm7] to have phase Bound I1105 21:39:44.010] Nov 5 21:39:44.010: INFO: PersistentVolumeClaim pvc-ckvm7 found but phase is Pending instead of Bound. I1105 21:39:46.089] Nov 5 21:39:46.088: INFO: PersistentVolumeClaim pvc-ckvm7 found but phase is Pending instead of Bound. I1105 21:39:48.168] Nov 5 21:39:48.168: INFO: PersistentVolumeClaim pvc-ckvm7 found but phase is Pending instead of Bound. I1105 21:39:50.247] Nov 5 21:39:50.247: INFO: PersistentVolumeClaim pvc-ckvm7 found but phase is Pending instead of Bound. I1105 21:39:52.326] Nov 5 21:39:52.326: INFO: PersistentVolumeClaim pvc-ckvm7 found but phase is Pending instead of Bound. I1105 21:39:54.405] Nov 5 21:39:54.405: INFO: PersistentVolumeClaim pvc-ckvm7 found but phase is Pending instead of Bound. I1105 21:39:56.484] Nov 5 21:39:56.484: INFO: PersistentVolumeClaim pvc-ckvm7 found but phase is Pending instead of Bound. I1105 21:39:58.564] Nov 5 21:39:58.563: INFO: PersistentVolumeClaim pvc-ckvm7 found but phase is Pending instead of Bound. I1105 21:40:00.643] Nov 5 21:40:00.643: INFO: PersistentVolumeClaim pvc-ckvm7 found but phase is Pending instead of Bound. I1105 21:40:02.722] Nov 5 21:40:02.722: INFO: PersistentVolumeClaim pvc-ckvm7 found but phase is Pending instead of Bound. I1105 21:40:04.802] Nov 5 21:40:04.802: INFO: PersistentVolumeClaim pvc-ckvm7 found but phase is Pending instead of Bound. I1105 21:40:06.882] Nov 5 21:40:06.882: INFO: PersistentVolumeClaim pvc-ckvm7 found but phase is Pending instead of Bound. I1105 21:40:08.962] Nov 5 21:40:08.961: INFO: PersistentVolumeClaim pvc-ckvm7 found but phase is Pending instead of Bound. I1105 21:40:11.041] Nov 5 21:40:11.041: INFO: PersistentVolumeClaim pvc-ckvm7 found but phase is Pending instead of Bound. I1105 21:40:13.120] Nov 5 21:40:13.120: INFO: PersistentVolumeClaim pvc-ckvm7 found but phase is Pending instead of Bound. I1105 21:40:15.201] Nov 5 21:40:15.200: INFO: PersistentVolumeClaim pvc-ckvm7 found but phase is Pending instead of Bound. I1105 21:40:17.297] Nov 5 21:40:17.296: INFO: PersistentVolumeClaim pvc-ckvm7 found but phase is Pending instead of Bound. I1105 21:40:19.376] Nov 5 21:40:19.375: INFO: PersistentVolumeClaim pvc-ckvm7 found but phase is Pending instead of Bound. I1105 21:40:21.456] Nov 5 21:40:21.456: INFO: PersistentVolumeClaim pvc-ckvm7 found but phase is Pending instead of Bound. I1105 21:40:23.535] Nov 5 21:40:23.535: INFO: PersistentVolumeClaim pvc-ckvm7 found but phase is Pending instead of Bound. I1105 21:40:25.620] Nov 5 21:40:25.620: INFO: PersistentVolumeClaim pvc-ckvm7 found but phase is Pending instead of Bound. I1105 21:40:27.699] Nov 5 21:40:27.699: INFO: PersistentVolumeClaim pvc-ckvm7 found but phase is Pending instead of Bound. I1105 21:40:29.778] Nov 5 21:40:29.778: INFO: PersistentVolumeClaim pvc-ckvm7 found but phase is Pending instead of Bound. I1105 21:40:31.853] Nov 5 21:40:31.852: INFO: PersistentVolumeClaim pvc-ckvm7 found but phase is Pending instead of Bound. I1105 21:40:33.926] Nov 5 21:40:33.926: INFO: PersistentVolumeClaim pvc-ckvm7 found but phase is Pending instead of Bound. I1105 21:40:36.000] Nov 5 21:40:35.999: INFO: PersistentVolumeClaim pvc-ckvm7 found but phase is Pending instead of Bound. I1105 21:40:38.074] Nov 5 21:40:38.074: INFO: PersistentVolumeClaim pvc-ckvm7 found but phase is Pending instead of Bound. I1105 21:40:40.154] Nov 5 21:40:40.154: INFO: PersistentVolumeClaim pvc-ckvm7 found but phase is Pending instead of Bound. I1105 21:40:42.227] Nov 5 21:40:42.227: INFO: PersistentVolumeClaim pvc-ckvm7 found but phase is Pending instead of Bound. I1105 21:40:44.301] Nov 5 21:40:44.301: INFO: PersistentVolumeClaim pvc-ckvm7 found but phase is Pending instead of Bound. I1105 21:40:46.375] Nov 5 21:40:46.375: INFO: PersistentVolumeClaim pvc-ckvm7 found but phase is Pending instead of Bound. I1105 21:40:48.449] Nov 5 21:40:48.449: INFO: PersistentVolumeClaim pvc-ckvm7 found but phase is Pending instead of Bound. I1105 21:40:50.523] Nov 5 21:40:50.522: INFO: PersistentVolumeClaim pvc-ckvm7 found but phase is Pending instead of Bound. I1105 21:40:52.597] Nov 5 21:40:52.597: INFO: PersistentVolumeClaim pvc-ckvm7 found but phase is Pending instead of Bound. I1105 21:40:54.671] Nov 5 21:40:54.671: INFO: PersistentVolumeClaim pvc-ckvm7 found but phase is Pending instead of Bound. I1105 21:40:56.745] Nov 5 21:40:56.745: INFO: PersistentVolumeClaim pvc-ckvm7 found but phase is Pending instead of Bound. I1105 21:40:58.819] Nov 5 21:40:58.819: INFO: PersistentVolumeClaim pvc-ckvm7 found but phase is Pending instead of Bound. I1105 21:41:00.892] Nov 5 21:41:00.892: INFO: PersistentVolumeClaim pvc-ckvm7 found but phase is Pending instead of Bound. I1105 21:41:02.966] Nov 5 21:41:02.966: INFO: PersistentVolumeClaim pvc-ckvm7 found but phase is Pending instead of Bound. I1105 21:41:05.040] Nov 5 21:41:05.040: INFO: PersistentVolumeClaim pvc-ckvm7 found but phase is Pending instead of Bound. I1105 21:41:07.114] Nov 5 21:41:07.114: INFO: PersistentVolumeClaim pvc-ckvm7 found but phase is Pending instead of Bound. I1105 21:41:09.188] Nov 5 21:41:09.188: INFO: PersistentVolumeClaim pvc-ckvm7 found but phase is Pending instead of Bound. I1105 21:41:11.262] Nov 5 21:41:11.262: INFO: PersistentVolumeClaim pvc-ckvm7 found but phase is Pending instead of Bound. I1105 21:41:13.336] Nov 5 21:41:13.336: INFO: PersistentVolumeClaim pvc-ckvm7 found but phase is Pending instead of Bound. I1105 21:41:15.411] Nov 5 21:41:15.410: INFO: PersistentVolumeClaim pvc-ckvm7 found but phase is Pending instead of Bound. I1105 21:41:17.484] Nov 5 21:41:17.484: INFO: PersistentVolumeClaim pvc-ckvm7 found but phase is Pending instead of Bound. I1105 21:41:19.561] Nov 5 21:41:19.561: INFO: PersistentVolumeClaim pvc-ckvm7 found but phase is Pending instead of Bound. I1105 21:41:21.635] Nov 5 21:41:21.635: INFO: PersistentVolumeClaim pvc-ckvm7 found but phase is Pending instead of Bound. I1105 21:41:23.709] Nov 5 21:41:23.708: INFO: PersistentVolumeClaim pvc-ckvm7 found but phase is Pending instead of Bound. I1105 21:41:25.784] Nov 5 21:41:25.784: INFO: PersistentVolumeClaim pvc-ckvm7 found but phase is Pending instead of Bound. I1105 21:41:27.857] Nov 5 21:41:27.857: INFO: PersistentVolumeClaim pvc-ckvm7 found but phase is Pending instead of Bound. I1105 21:41:29.931] Nov 5 21:41:29.931: INFO: PersistentVolumeClaim pvc-ckvm7 found but phase is Pending instead of Bound. I1105 21:41:32.005] Nov 5 21:41:32.005: INFO: PersistentVolumeClaim pvc-ckvm7 found but phase is Pending instead of Bound. I1105 21:41:34.079] Nov 5 21:41:34.079: INFO: PersistentVolumeClaim pvc-ckvm7 found but phase is Pending instead of Bound. I1105 21:41:36.152] Nov 5 21:41:36.152: INFO: PersistentVolumeClaim pvc-ckvm7 found but phase is Pending instead of Bound. I1105 21:41:38.225] Nov 5 21:41:38.225: INFO: PersistentVolumeClaim pvc-ckvm7 found but phase is Pending instead of Bound. I1105 21:41:40.299] Nov 5 21:41:40.299: INFO: PersistentVolumeClaim pvc-ckvm7 found but phase is Pending instead of Bound. I1105 21:41:42.373] Nov 5 21:41:42.373: INFO: PersistentVolumeClaim pvc-ckvm7 found but phase is Pending instead of Bound. I1105 21:41:44.447] Nov 5 21:41:44.447: INFO: PersistentVolumeClaim pvc-ckvm7 found but phase is Pending instead of Bound. I1105 21:41:46.521] Nov 5 21:41:46.521: INFO: PersistentVolumeClaim pvc-ckvm7 found but phase is Pending instead of Bound. I1105 21:41:48.596] Nov 5 21:41:48.596: INFO: PersistentVolumeClaim pvc-ckvm7 found but phase is Pending instead of Bound. I1105 21:41:50.670] Nov 5 21:41:50.669: INFO: PersistentVolumeClaim pvc-ckvm7 found but phase is Pending instead of Bound. I1105 21:41:52.744] Nov 5 21:41:52.743: INFO: PersistentVolumeClaim pvc-ckvm7 found but phase is Pending instead of Bound. I1105 21:41:54.817] Nov 5 21:41:54.817: INFO: PersistentVolumeClaim pvc-ckvm7 found but phase is Pending instead of Bound. I1105 21:41:56.891] Nov 5 21:41:56.891: INFO: PersistentVolumeClaim pvc-ckvm7 found but phase is Pending instead of Bound. I1105 21:41:58.964] Nov 5 21:41:58.964: INFO: PersistentVolumeClaim pvc-ckvm7 found but phase is Pending instead of Bound. I1105 21:42:01.038] Nov 5 21:42:01.038: INFO: PersistentVolumeClaim pvc-ckvm7 found but phase is Pending instead of Bound. I1105 21:42:03.113] Nov 5 21:42:03.112: INFO: PersistentVolumeClaim pvc-ckvm7 found but phase is Pending instead of Bound. I1105 21:42:05.190] Nov 5 21:42:05.190: INFO: PersistentVolumeClaim pvc-ckvm7 found but phase is Pending instead of Bound. I1105 21:42:07.264] Nov 5 21:42:07.263: INFO: PersistentVolumeClaim pvc-ckvm7 found but phase is Pending instead of Bound. I1105 21:42:09.337] Nov 5 21:42:09.337: INFO: PersistentVolumeClaim pvc-ckvm7 found but phase is Pending instead of Bound. I1105 21:42:11.411] Nov 5 21:42:11.411: INFO: PersistentVolumeClaim pvc-ckvm7 found but phase is Pending instead of Bound. I1105 21:42:13.485] Nov 5 21:42:13.484: INFO: PersistentVolumeClaim pvc-ckvm7 found but phase is Pending instead of Bound. I1105 21:42:15.558] Nov 5 21:42:15.558: INFO: PersistentVolumeClaim pvc-ckvm7 found but phase is Pending instead of Bound. I1105 21:42:17.633] Nov 5 21:42:17.632: INFO: PersistentVolumeClaim pvc-ckvm7 found but phase is Pending instead of Bound. I1105 21:42:19.707] Nov 5 21:42:19.706: INFO: PersistentVolumeClaim pvc-ckvm7 found but phase is Pending instead of Bound. I1105 21:42:21.780] Nov 5 21:42:21.780: INFO: PersistentVolumeClaim pvc-ckvm7 found but phase is Pending instead of Bound. I1105 21:42:23.854] Nov 5 21:42:23.854: INFO: PersistentVolumeClaim pvc-ckvm7 found but phase is Pending instead of Bound. I1105 21:42:25.928] Nov 5 21:42:25.928: INFO: PersistentVolumeClaim pvc-ckvm7 found but phase is Pending instead of Bound. I1105 21:42:28.002] Nov 5 21:42:28.002: INFO: PersistentVolumeClaim pvc-ckvm7 found but phase is Pending instead of Bound. I1105 21:42:30.077] Nov 5 21:42:30.077: INFO: PersistentVolumeClaim pvc-ckvm7 found but phase is Pending instead of Bound. I1105 21:42:32.151] Nov 5 21:42:32.151: INFO: PersistentVolumeClaim pvc-ckvm7 found but phase is Pending instead of Bound. I1105 21:42:34.225] Nov 5 21:42:34.224: INFO: PersistentVolumeClaim pvc-ckvm7 found but phase is Pending instead of Bound. I1105 21:42:36.301] Nov 5 21:42:36.301: INFO: PersistentVolumeClaim pvc-ckvm7 found but phase is Pending instead of Bound. I1105 21:42:38.376] Nov 5 21:42:38.376: INFO: PersistentVolumeClaim pvc-ckvm7 found but phase is Pending instead of Bound. I1105 21:42:40.450] Nov 5 21:42:40.449: INFO: PersistentVolumeClaim pvc-ckvm7 found but phase is Pending instead of Bound. I1105 21:42:42.523] Nov 5 21:42:42.523: INFO: PersistentVolumeClaim pvc-ckvm7 found but phase is Pending instead of Bound. I1105 21:42:44.597] Nov 5 21:42:44.597: INFO: PersistentVolumeClaim pvc-ckvm7 found but phase is Pending instead of Bound. I1105 21:42:46.671] Nov 5 21:42:46.671: INFO: PersistentVolumeClaim pvc-ckvm7 found but phase is Pending instead of Bound. I1105 21:42:48.744] Nov 5 21:42:48.744: INFO: PersistentVolumeClaim pvc-ckvm7 found but phase is Pending instead of Bound. I1105 21:42:50.818] Nov 5 21:42:50.818: INFO: PersistentVolumeClaim pvc-ckvm7 found but phase is Pending instead of Bound. I1105 21:42:52.897] Nov 5 21:42:52.897: INFO: PersistentVolumeClaim pvc-ckvm7 found but phase is Pending instead of Bound. I1105 21:42:54.971] Nov 5 21:42:54.970: INFO: PersistentVolumeClaim pvc-ckvm7 found but phase is Pending instead of Bound. I1105 21:42:57.044] Nov 5 21:42:57.044: INFO: PersistentVolumeClaim pvc-ckvm7 found but phase is Pending instead of Bound. I1105 21:42:59.118] Nov 5 21:42:59.118: INFO: PersistentVolumeClaim pvc-ckvm7 found but phase is Pending instead of Bound. I1105 21:43:01.192] Nov 5 21:43:01.192: INFO: PersistentVolumeClaim pvc-ckvm7 found but phase is Pending instead of Bound. I1105 21:43:03.266] Nov 5 21:43:03.266: INFO: PersistentVolumeClaim pvc-ckvm7 found but phase is Pending instead of Bound. I1105 21:43:05.340] Nov 5 21:43:05.340: INFO: PersistentVolumeClaim pvc-ckvm7 found but phase is Pending instead of Bound. I1105 21:43:07.415] Nov 5 21:43:07.415: INFO: PersistentVolumeClaim pvc-ckvm7 found but phase is Pending instead of Bound. I1105 21:43:09.488] Nov 5 21:43:09.488: INFO: PersistentVolumeClaim pvc-ckvm7 found but phase is Pending instead of Bound. I1105 21:43:11.562] Nov 5 21:43:11.562: INFO: PersistentVolumeClaim pvc-ckvm7 found but phase is Pending instead of Bound. I1105 21:43:13.637] Nov 5 21:43:13.636: INFO: PersistentVolumeClaim pvc-ckvm7 found but phase is Pending instead of Bound. I1105 21:43:15.711] Nov 5 21:43:15.710: INFO: PersistentVolumeClaim pvc-ckvm7 found but phase is Pending instead of Bound. I1105 21:43:17.785] Nov 5 21:43:17.784: INFO: PersistentVolumeClaim pvc-ckvm7 found but phase is Pending instead of Bound. I1105 21:43:19.858] Nov 5 21:43:19.858: INFO: PersistentVolumeClaim pvc-ckvm7 found but phase is Pending instead of Bound. I1105 21:43:21.932] Nov 5 21:43:21.931: INFO: PersistentVolumeClaim pvc-ckvm7 found but phase is Pending instead of Bound. I1105 21:43:24.006] Nov 5 21:43:24.005: INFO: PersistentVolumeClaim pvc-ckvm7 found but phase is Pending instead of Bound. I1105 21:43:26.080] Nov 5 21:43:26.080: INFO: PersistentVolumeClaim pvc-ckvm7 found but phase is Pending instead of Bound. I1105 21:43:28.154] Nov 5 21:43:28.153: INFO: PersistentVolumeClaim pvc-ckvm7 found but phase is Pending instead of Bound. I1105 21:43:30.228] Nov 5 21:43:30.228: INFO: PersistentVolumeClaim pvc-ckvm7 found but phase is Pending instead of Bound. I1105 21:43:32.304] Nov 5 21:43:32.304: INFO: PersistentVolumeClaim pvc-ckvm7 found but phase is Pending instead of Bound. I1105 21:43:34.378] Nov 5 21:43:34.377: INFO: PersistentVolumeClaim pvc-ckvm7 found but phase is Pending instead of Bound. I1105 21:43:36.451] Nov 5 21:43:36.451: INFO: PersistentVolumeClaim pvc-ckvm7 found but phase is Pending instead of Bound. I1105 21:43:38.526] Nov 5 21:43:38.526: INFO: PersistentVolumeClaim pvc-ckvm7 found but phase is Pending instead of Bound. I1105 21:43:40.599] Nov 5 21:43:40.599: INFO: PersistentVolumeClaim pvc-ckvm7 found but phase is Pending instead of Bound. I1105 21:43:42.674] Nov 5 21:43:42.674: INFO: PersistentVolumeClaim pvc-ckvm7 found but phase is Pending instead of Bound. I1105 21:43:44.749] Nov 5 21:43:44.748: INFO: PersistentVolumeClaim pvc-ckvm7 found but phase is Pending instead of Bound. I1105 21:43:46.822] Nov 5 21:43:46.822: INFO: PersistentVolumeClaim pvc-ckvm7 found but phase is Pending instead of Bound. I1105 21:43:48.896] Nov 5 21:43:48.896: INFO: PersistentVolumeClaim pvc-ckvm7 found but phase is Pending instead of Bound. I1105 21:43:50.970] Nov 5 21:43:50.970: INFO: PersistentVolumeClaim pvc-ckvm7 found but phase is Pending instead of Bound. I1105 21:43:53.044] Nov 5 21:43:53.044: INFO: PersistentVolumeClaim pvc-ckvm7 found but phase is Pending instead of Bound. I1105 21:43:55.118] Nov 5 21:43:55.118: INFO: PersistentVolumeClaim pvc-ckvm7 found but phase is Pending instead of Bound. I1105 21:43:57.191] Nov 5 21:43:57.191: INFO: PersistentVolumeClaim pvc-ckvm7 found but phase is Pending instead of Bound. I1105 21:43:59.265] Nov 5 21:43:59.265: INFO: PersistentVolumeClaim pvc-ckvm7 found but phase is Pending instead of Bound. I1105 21:44:01.339] Nov 5 21:44:01.339: INFO: PersistentVolumeClaim pvc-ckvm7 found but phase is Pending instead of Bound. I1105 21:44:03.413] Nov 5 21:44:03.412: INFO: PersistentVolumeClaim pvc-ckvm7 found but phase is Pending instead of Bound. I1105 21:44:05.489] Nov 5 21:44:05.489: INFO: PersistentVolumeClaim pvc-ckvm7 found but phase is Pending instead of Bound. I1105 21:44:07.563] Nov 5 21:44:07.563: INFO: PersistentVolumeClaim pvc-ckvm7 found but phase is Pending instead of Bound. I1105 21:44:09.637] Nov 5 21:44:09.637: INFO: PersistentVolumeClaim pvc-ckvm7 found but phase is Pending instead of Bound. I1105 21:44:11.710] Nov 5 21:44:11.710: INFO: PersistentVolumeClaim pvc-ckvm7 found but phase is Pending instead of Bound. I1105 21:44:13.784] Nov 5 21:44:13.784: INFO: PersistentVolumeClaim pvc-ckvm7 found but phase is Pending instead of Bound. I1105 21:44:15.858] Nov 5 21:44:15.858: INFO: PersistentVolumeClaim pvc-ckvm7 found but phase is Pending instead of Bound. I1105 21:44:17.933] Nov 5 21:44:17.933: INFO: PersistentVolumeClaim pvc-ckvm7 found but phase is Pending instead of Bound. I1105 21:44:20.007] Nov 5 21:44:20.007: INFO: PersistentVolumeClaim pvc-ckvm7 found but phase is Pending instead of Bound. I1105 21:44:22.081] Nov 5 21:44:22.081: INFO: PersistentVolumeClaim pvc-ckvm7 found but phase is Pending instead of Bound. I1105 21:44:24.155] Nov 5 21:44:24.155: INFO: PersistentVolumeClaim pvc-ckvm7 found but phase is Pending instead of Bound. I1105 21:44:26.229] Nov 5 21:44:26.229: INFO: PersistentVolumeClaim pvc-ckvm7 found but phase is Pending instead of Bound. I1105 21:44:28.303] Nov 5 21:44:28.303: INFO: PersistentVolumeClaim pvc-ckvm7 found but phase is Pending instead of Bound. I1105 21:44:30.377] Nov 5 21:44:30.377: INFO: PersistentVolumeClaim pvc-ckvm7 found but phase is Pending instead of Bound. I1105 21:44:32.450] Nov 5 21:44:32.450: INFO: PersistentVolumeClaim pvc-ckvm7 found but phase is Pending instead of Bound. I1105 21:44:34.524] Nov 5 21:44:34.524: INFO: PersistentVolumeClaim pvc-ckvm7 found but phase is Pending instead of Bound. I1105 21:44:36.598] Nov 5 21:44:36.598: INFO: PersistentVolumeClaim pvc-ckvm7 found but phase is Pending instead of Bound. I1105 21:44:38.673] Nov 5 21:44:38.673: INFO: PersistentVolumeClaim pvc-ckvm7 found but phase is Pending instead of Bound. I1105 21:44:40.748] Nov 5 21:44:40.747: INFO: PersistentVolumeClaim pvc-ckvm7 found but phase is Pending instead of Bound. I1105 21:44:42.822] Nov 5 21:44:42.821: INFO: PersistentVolumeClaim pvc-ckvm7 found but phase is Pending instead of Bound. I1105 21:44:44.822] Nov 5 21:44:44.822: INFO: Unexpected error occurred: PersistentVolumeClaims [pvc-ckvm7] not all in phase Bound within 5m0s I1105 21:44:44.822] [AfterEach] [multi-az] I1105 21:44:44.822] /go/src/sigs.k8s.io/azuredisk-csi-driver/vendor/k8s.io/kubernetes/test/e2e/framework/framework.go:151 I1105 21:44:44.823] STEP: Collecting events from namespace "azuredisk-8482". I1105 21:44:44.896] STEP: Found 3 events. I1105 21:44:44.897] Nov 5 21:44:44.896: INFO: At 2019-11-05 21:39:43 +0000 UTC - event for pvc-ckvm7: {disk.csi.azure.com_csi-azuredisk-controller-6f6587bfc6-62nrt_3930bc69-0009-11ea-8cf5-829a27ff7d5a } Provisioning: External provisioner is provisioning volume for claim "azuredisk-8482/pvc-ckvm7" I1105 21:44:44.897] Nov 5 21:44:44.896: INFO: At 2019-11-05 21:39:43 +0000 UTC - event for pvc-ckvm7: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "disk.csi.azure.com" or manually created by system administrator I1105 21:44:44.898] Nov 5 21:44:44.896: INFO: At 2019-11-05 21:39:44 +0000 UTC - event for pvc-ckvm7: {disk.csi.azure.com_csi-azuredisk-controller-6f6587bfc6-62nrt_3930bc69-0009-11ea-8cf5-829a27ff7d5a } ProvisioningFailed: failed to provision volume with StorageClass "azuredisk-8482-disk.csi.azure.com-dynamic-sc-xk2nr": rpc error: code = Unknown desc = compute.DisksClient#CreateOrUpdate: Failure sending request: StatusCode=400 -- Original Error: Code="InvalidAvailabilityZone" Message="The zone(s) '' for resource 'Microsoft.Compute/disks/pvc-c412946c-30a6-4ba1-8382-12c6afc4b96f' is not supported. The supported zones for location 'eastus2' are '1,2,3'" I1105 21:44:44.970] Nov 5 21:44:44.970: INFO: POD NODE PHASE GRACE CONDITIONS I1105 21:44:44.970] Nov 5 21:44:44.970: INFO: I1105 21:44:45.255] Nov 5 21:44:45.255: INFO: I1105 21:44:45.256] Logging node info for node k8s-agentpool-97435445-vmss000000 I1105 21:44:45.331] Nov 5 21:44:45.328: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-agentpool-97435445-vmss000000,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-agentpool-97435445-vmss000000,UID:e955dc47-8c8f-4142-9241-3d409f32e42d,ResourceVersion:18143,Generation:0,CreationTimestamp:2019-11-05 20:07:34 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{agentpool: agentpool,beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_D4s_v3,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 0,kubernetes.azure.com/cluster: kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b,kubernetes.azure.com/role: agent,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-agentpool-97435445-vmss000000,kubernetes.io/os: linux,kubernetes.io/role: agent,node-role.kubernetes.io/agent: ,storageprofile: managed,storagetier: Premium_LRS,topology.disk.csi.azure.com/zone: 0,},Annotations:map[string]string{csi.volume.kubernetes.io/nodeid: {"disk.csi.azure.com":"k8s-agentpool-97435445-vmss000000"},node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.2.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool-97435445-vmss/virtualMachines/0,Unschedulable:false,Taints:[],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-csi-disk.csi.azure.com: {{8 0} {} 8 DecimalSI},cpu: {{4 0} {} 4 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{16797040640 0} {} BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-csi-disk.csi.azure.com: {{8 0} {} 8 DecimalSI},cpu: {{4 0} {} 4 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{16010608640 0} {} BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 20:08:01 +0000 UTC 2019-11-05 20:08:01 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 21:43:51 +0000 UTC 2019-11-05 20:07:27 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 21:43:51 +0000 UTC 2019-11-05 20:07:27 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 21:43:51 +0000 UTC 2019-11-05 20:07:27 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 21:43:51 +0000 UTC 2019-11-05 20:07:55 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.0.4} {Hostname k8s-agentpool-97435445-vmss000000}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:bfb55325809340228149d2d93f50df58,SystemUUID:CB2ED036-0439-F641-BC7F-640C3638C027,BootID:a917f4d1-549d-471d-b7dc-47ae1715d97d,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[k8sprowinternal.azurecr.io/azuredisk-csi@sha256:75d4ff5d7cd845d6e810c67796ecb037e852fb849ec1eb5b3194564008e6d1e0 k8sprowinternal.azurecr.io/azuredisk-csi:e2e-3e3ed045c6faa509dbd8d2890ee9f56aa2934887] 286930414} {[k8s.gcr.io/kubernetes-dashboard-amd64@sha256:0ae6b69432e78069c5ce2bcde0fe409c5c4d6f0f4d9cd50a17974fea38898747 k8s.gcr.io/kubernetes-dashboard-amd64:v1.10.1] 121711221} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/metrics-server-amd64@sha256:49a9f12f7067d11f42c803dbe61ed2c1299959ad85cb315b25ff7eef8e6b8892 k8s.gcr.io/metrics-server-amd64:v0.2.1] 42541759} {[k8s.gcr.io/coredns@sha256:e83beb5e43f8513fa735e77ffc5859640baea30a882a11cc75c4c3244a737d3c k8s.gcr.io/coredns:1.5.0] 42488424} {[mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume@sha256:4fd30d43947d4a54fc89ead7985beecfd3c9b2a93a0655a373b1608ab90bd5af mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume:v0.0.7] 22909487} {[quay.io/k8scsi/livenessprobe@sha256:8c9c8a0f330acc2e558ae6624309447d02522e76aa35af23fa19b88abf4381cb quay.io/k8scsi/livenessprobe:v1.0.2] 17415381} {[quay.io/k8scsi/csi-node-driver-registrar@sha256:13daf82fb99e951a4bff8ae5fc7c17c3a8fe7130be6400990d8f6076c32d4599 quay.io/k8scsi/csi-node-driver-registrar:v1.1.0] 15815995} {[mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume@sha256:23d8c6033f02a1ecad05127ebdc931bb871264228661bc122704b0974e4d9fdd mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume:1.0.8] 1159025} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 21:44:45.332] Nov 5 21:44:45.328: INFO: I1105 21:44:45.332] Logging kubelet events for node k8s-agentpool-97435445-vmss000000 I1105 21:44:45.402] Nov 5 21:44:45.402: INFO: I1105 21:44:45.403] Logging pods the kubelet thinks is on node k8s-agentpool-97435445-vmss000000 I1105 21:44:45.550] Nov 5 21:44:45.550: INFO: metrics-server-864ffbc5c-wl84z started at 2019-11-05 20:08:01 +0000 UTC (0+1 container statuses recorded) I1105 21:44:45.551] Nov 5 21:44:45.550: INFO: Container metrics-server ready: true, restart count 0 I1105 21:44:45.551] Nov 5 21:44:45.550: INFO: kube-proxy-8mt7f started at 2019-11-05 20:08:01 +0000 UTC (0+1 container statuses recorded) I1105 21:44:45.551] Nov 5 21:44:45.550: INFO: Container kube-proxy ready: true, restart count 0 I1105 21:44:45.551] Nov 5 21:44:45.550: INFO: kubernetes-dashboard-66dd8b8df7-t6j4l started at 2019-11-05 20:08:01 +0000 UTC (0+1 container statuses recorded) I1105 21:44:45.551] Nov 5 21:44:45.550: INFO: Container kubernetes-dashboard ready: true, restart count 0 I1105 21:44:45.551] Nov 5 21:44:45.550: INFO: csi-azuredisk-node-5wwhz started at 2019-11-05 20:16:09 +0000 UTC (0+3 container statuses recorded) I1105 21:44:45.552] Nov 5 21:44:45.550: INFO: Container azuredisk ready: true, restart count 0 I1105 21:44:45.552] Nov 5 21:44:45.550: INFO: Container liveness-probe ready: true, restart count 0 I1105 21:44:45.552] Nov 5 21:44:45.550: INFO: Container node-driver-registrar ready: true, restart count 0 I1105 21:44:45.552] Nov 5 21:44:45.550: INFO: blobfuse-flexvol-installer-p7hdb started at 2019-11-05 20:08:01 +0000 UTC (0+1 container statuses recorded) I1105 21:44:45.552] Nov 5 21:44:45.550: INFO: Container blobfuse-flexvol-installer ready: true, restart count 0 I1105 21:44:45.552] Nov 5 21:44:45.550: INFO: coredns-7f68dcdbdb-zh8g5 started at 2019-11-05 20:08:01 +0000 UTC (0+1 container statuses recorded) I1105 21:44:45.553] Nov 5 21:44:45.550: INFO: Container coredns ready: true, restart count 0 I1105 21:44:45.553] Nov 5 21:44:45.550: INFO: azure-ip-masq-agent-m7tv9 started at 2019-11-05 20:08:01 +0000 UTC (0+1 container statuses recorded) I1105 21:44:45.553] Nov 5 21:44:45.550: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 21:44:45.553] Nov 5 21:44:45.550: INFO: keyvault-flexvolume-smspt started at 2019-11-05 20:08:01 +0000 UTC (0+1 container statuses recorded) I1105 21:44:45.553] Nov 5 21:44:45.550: INFO: Container keyvault-flexvolume ready: true, restart count 0 I1105 21:44:45.794] Nov 5 21:44:45.794: INFO: I1105 21:44:45.794] Latency metrics for node k8s-agentpool-97435445-vmss000000 I1105 21:44:45.794] Nov 5 21:44:45.794: INFO: I1105 21:44:45.794] Logging node info for node k8s-agentpool-97435445-vmss000001 I1105 21:44:45.870] Nov 5 21:44:45.867: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-agentpool-97435445-vmss000001,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-agentpool-97435445-vmss000001,UID:75d3cc50-ca6d-45e0-a42a-6090e8d537dd,ResourceVersion:18286,Generation:0,CreationTimestamp:2019-11-05 20:07:34 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{agentpool: agentpool,beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_D4s_v3,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 1,kubernetes.azure.com/cluster: kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b,kubernetes.azure.com/role: agent,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-agentpool-97435445-vmss000001,kubernetes.io/os: linux,kubernetes.io/role: agent,node-role.kubernetes.io/agent: ,storageprofile: managed,storagetier: Premium_LRS,topology.disk.csi.azure.com/zone: 1,},Annotations:map[string]string{csi.volume.kubernetes.io/nodeid: {"disk.csi.azure.com":"k8s-agentpool-97435445-vmss000001"},node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.0.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool-97435445-vmss/virtualMachines/1,Unschedulable:false,Taints:[],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-csi-disk.csi.azure.com: {{8 0} {} 8 DecimalSI},cpu: {{4 0} {} 4 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{16797044736 0} {} 16403364Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-csi-disk.csi.azure.com: {{8 0} {} 8 DecimalSI},cpu: {{4 0} {} 4 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{16010612736 0} {} 15635364Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 20:08:25 +0000 UTC 2019-11-05 20:08:25 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 21:44:40 +0000 UTC 2019-11-05 20:07:26 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 21:44:40 +0000 UTC 2019-11-05 20:07:26 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 21:44:40 +0000 UTC 2019-11-05 20:07:26 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 21:44:40 +0000 UTC 2019-11-05 20:07:55 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.0.5} {Hostname k8s-agentpool-97435445-vmss000001}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:e78def54b2f545c6b94d88bbba523c60,SystemUUID:40AE8188-86AB-D848-84A4-D9254E1258BC,BootID:8587ec06-f4f4-43c1-86cd-06c295b05b63,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[k8sprowinternal.azurecr.io/azuredisk-csi@sha256:75d4ff5d7cd845d6e810c67796ecb037e852fb849ec1eb5b3194564008e6d1e0 k8sprowinternal.azurecr.io/azuredisk-csi:e2e-3e3ed045c6faa509dbd8d2890ee9f56aa2934887] 286930414} {[gcr.io/kubernetes-helm/tiller@sha256:f6d8f4ab9ba993b5f5b60a6edafe86352eabe474ffeb84cb6c79b8866dce45d1 gcr.io/kubernetes-helm/tiller:v2.11.0] 71821984} {[quay.io/k8scsi/csi-provisioner@sha256:d657c839dce87324fe2b677302913f9386f885f8746be7bea0ced5b0844e3433 quay.io/k8scsi/csi-provisioner:v1.3.1] 50596998} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[quay.io/k8scsi/csi-attacher@sha256:26fccd7a99d973845df1193b46ebdcc6ab8dc5f6e6be319750c471fce1742d13 quay.io/k8scsi/csi-attacher:v1.2.0] 46226754} {[quay.io/k8scsi/csi-cluster-driver-registrar@sha256:fafd75ae5442f192cfa8c2e792903aee30d5884b62e802e4464b0a895d21e3ef quay.io/k8scsi/csi-cluster-driver-registrar:v1.0.1] 45874691} {[quay.io/k8scsi/csi-snapshotter@sha256:a49e0da1af6f2bf717e41ba1eee8b5e6a1cbd66a709dd92cc43fe475fe2589eb quay.io/k8scsi/csi-snapshotter:v1.1.0] 44325352} {[mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume@sha256:4fd30d43947d4a54fc89ead7985beecfd3c9b2a93a0655a373b1608ab90bd5af mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume:v0.0.7] 22909487} {[quay.io/k8scsi/livenessprobe@sha256:8c9c8a0f330acc2e558ae6624309447d02522e76aa35af23fa19b88abf4381cb quay.io/k8scsi/livenessprobe:v1.0.2] 17415381} {[quay.io/k8scsi/csi-node-driver-registrar@sha256:13daf82fb99e951a4bff8ae5fc7c17c3a8fe7130be6400990d8f6076c32d4599 quay.io/k8scsi/csi-node-driver-registrar:v1.1.0] 15815995} {[quay.io/k8scsi/livenessprobe@sha256:dde617756e0f602adc566ab71fd885f1dad451ad3fb063ac991c95a2ff47aea5 quay.io/k8scsi/livenessprobe:v1.1.0] 14967303} {[mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume@sha256:23d8c6033f02a1ecad05127ebdc931bb871264228661bc122704b0974e4d9fdd mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume:1.0.8] 1159025} {[busybox@sha256:8ccbac733d19c0dd4d70b4f0c1e12245b5fa3ad24758a11035ee505c629c0796 busybox:1.29] 1154361} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 21:44:45.871] Nov 5 21:44:45.867: INFO: I1105 21:44:45.871] Logging kubelet events for node k8s-agentpool-97435445-vmss000001 I1105 21:44:45.941] Nov 5 21:44:45.941: INFO: I1105 21:44:45.941] Logging pods the kubelet thinks is on node k8s-agentpool-97435445-vmss000001 I1105 21:44:46.091] Nov 5 21:44:46.091: INFO: tiller-deploy-54c96cb5df-pt7x6 started at 2019-11-05 20:08:00 +0000 UTC (0+1 container statuses recorded) I1105 21:44:46.091] Nov 5 21:44:46.091: INFO: Container tiller ready: true, restart count 0 I1105 21:44:46.091] Nov 5 21:44:46.091: INFO: csi-azuredisk-node-q2dmt started at 2019-11-05 20:16:09 +0000 UTC (0+3 container statuses recorded) I1105 21:44:46.091] Nov 5 21:44:46.091: INFO: Container azuredisk ready: true, restart count 0 I1105 21:44:46.091] Nov 5 21:44:46.091: INFO: Container liveness-probe ready: true, restart count 0 I1105 21:44:46.092] Nov 5 21:44:46.091: INFO: Container node-driver-registrar ready: true, restart count 0 I1105 21:44:46.092] Nov 5 21:44:46.091: INFO: csi-azuredisk-controller-6f6587bfc6-62nrt started at 2019-11-05 20:16:09 +0000 UTC (0+6 container statuses recorded) I1105 21:44:46.092] Nov 5 21:44:46.091: INFO: Container azuredisk ready: true, restart count 0 I1105 21:44:46.092] Nov 5 21:44:46.091: INFO: Container cluster-driver-registrar ready: true, restart count 0 I1105 21:44:46.092] Nov 5 21:44:46.091: INFO: Container csi-attacher ready: true, restart count 0 I1105 21:44:46.092] Nov 5 21:44:46.091: INFO: Container csi-provisioner ready: true, restart count 0 I1105 21:44:46.092] Nov 5 21:44:46.091: INFO: Container csi-snapshotter ready: true, restart count 0 I1105 21:44:46.092] Nov 5 21:44:46.091: INFO: Container liveness-probe ready: true, restart count 0 I1105 21:44:46.093] Nov 5 21:44:46.091: INFO: blobfuse-flexvol-installer-j4s7t started at 2019-11-05 20:07:59 +0000 UTC (0+1 container statuses recorded) I1105 21:44:46.093] Nov 5 21:44:46.091: INFO: Container blobfuse-flexvol-installer ready: true, restart count 0 I1105 21:44:46.093] Nov 5 21:44:46.091: INFO: azure-ip-masq-agent-gdhcq started at 2019-11-05 20:07:59 +0000 UTC (0+1 container statuses recorded) I1105 21:44:46.093] Nov 5 21:44:46.091: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 21:44:46.093] Nov 5 21:44:46.091: INFO: keyvault-flexvolume-6bffx started at 2019-11-05 20:07:59 +0000 UTC (0+1 container statuses recorded) I1105 21:44:46.093] Nov 5 21:44:46.091: INFO: Container keyvault-flexvolume ready: true, restart count 0 I1105 21:44:46.093] Nov 5 21:44:46.091: INFO: kube-proxy-kjrpb started at 2019-11-05 20:08:00 +0000 UTC (0+1 container statuses recorded) I1105 21:44:46.094] Nov 5 21:44:46.091: INFO: Container kube-proxy ready: true, restart count 0 I1105 21:44:46.382] Nov 5 21:44:46.382: INFO: I1105 21:44:46.383] Latency metrics for node k8s-agentpool-97435445-vmss000001 I1105 21:44:46.383] Nov 5 21:44:46.382: INFO: I1105 21:44:46.383] Logging node info for node k8s-master-97435445-0 I1105 21:44:46.459] Nov 5 21:44:46.457: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-97435445-0,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-97435445-0,UID:df3ae36f-961c-4729-a8f5-34cda54f4ff9,ResourceVersion:18201,Generation:0,CreationTimestamp:2019-11-05 20:08:36 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 2,kubernetes.azure.com/cluster: kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-97435445-0,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.4.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b/providers/Microsoft.Compute/virtualMachines/k8s-master-97435445-0,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 20:08:46 +0000 UTC 2019-11-05 20:08:46 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 21:44:10 +0000 UTC 2019-11-05 20:08:28 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 21:44:10 +0000 UTC 2019-11-05 20:08:28 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 21:44:10 +0000 UTC 2019-11-05 20:08:28 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 21:44:10 +0000 UTC 2019-11-05 20:08:36 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.255.5} {Hostname k8s-master-97435445-0}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:3c381865192341c2868672e8626cef7d,SystemUUID:F3D259D3-7EB8-B844-A1F0-F954DCD0D76B,BootID:4948329e-cf57-48c3-aa83-5297bef70370,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 21:44:46.459] Nov 5 21:44:46.457: INFO: I1105 21:44:46.459] Logging kubelet events for node k8s-master-97435445-0 I1105 21:44:46.530] Nov 5 21:44:46.530: INFO: I1105 21:44:46.531] Logging pods the kubelet thinks is on node k8s-master-97435445-0 I1105 21:44:46.613] Nov 5 21:44:46.612: INFO: kube-proxy-5wnd2 started at 2019-11-05 20:08:38 +0000 UTC (0+1 container statuses recorded) I1105 21:44:46.613] Nov 5 21:44:46.613: INFO: Container kube-proxy ready: true, restart count 0 I1105 21:44:46.613] Nov 5 21:44:46.613: INFO: cloud-controller-manager-k8s-master-97435445-0 started at 2019-11-05 20:08:22 +0000 UTC (0+1 container statuses recorded) I1105 21:44:46.613] Nov 5 21:44:46.613: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 21:44:46.613] Nov 5 21:44:46.613: INFO: kube-addon-manager-k8s-master-97435445-0 started at 2019-11-05 20:08:48 +0000 UTC (0+1 container statuses recorded) I1105 21:44:46.614] Nov 5 21:44:46.613: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 21:44:46.614] Nov 5 21:44:46.613: INFO: kube-apiserver-k8s-master-97435445-0 started at 2019-11-05 20:08:22 +0000 UTC (0+1 container statuses recorded) I1105 21:44:46.614] Nov 5 21:44:46.613: INFO: Container kube-apiserver ready: true, restart count 0 I1105 21:44:46.614] Nov 5 21:44:46.613: INFO: kube-controller-manager-k8s-master-97435445-0 started at 2019-11-05 20:08:22 +0000 UTC (0+1 container statuses recorded) I1105 21:44:46.614] Nov 5 21:44:46.613: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 21:44:46.614] Nov 5 21:44:46.613: INFO: kube-scheduler-k8s-master-97435445-0 started at 2019-11-05 20:08:22 +0000 UTC (0+1 container statuses recorded) I1105 21:44:46.614] Nov 5 21:44:46.613: INFO: Container kube-scheduler ready: true, restart count 0 I1105 21:44:46.615] Nov 5 21:44:46.613: INFO: azure-ip-masq-agent-q6dcl started at 2019-11-05 20:08:38 +0000 UTC (0+1 container statuses recorded) I1105 21:44:46.615] Nov 5 21:44:46.613: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 21:44:46.853] Nov 5 21:44:46.853: INFO: I1105 21:44:46.853] Latency metrics for node k8s-master-97435445-0 I1105 21:44:46.853] Nov 5 21:44:46.853: INFO: I1105 21:44:46.853] Logging node info for node k8s-master-97435445-1 I1105 21:44:46.929] Nov 5 21:44:46.926: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-97435445-1,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-97435445-1,UID:05534a67-d493-4ab5-9b0f-5ff5490e234f,ResourceVersion:18280,Generation:0,CreationTimestamp:2019-11-05 20:07:34 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 0,kubernetes.azure.com/cluster: kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-97435445-1,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.1.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b/providers/Microsoft.Compute/virtualMachines/k8s-master-97435445-1,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 20:08:26 +0000 UTC 2019-11-05 20:08:26 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 21:44:39 +0000 UTC 2019-11-05 20:07:28 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 21:44:39 +0000 UTC 2019-11-05 20:07:28 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 21:44:39 +0000 UTC 2019-11-05 20:07:28 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 21:44:39 +0000 UTC 2019-11-05 20:07:55 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{Hostname k8s-master-97435445-1} {InternalIP 10.240.255.6}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:079b605103b74acabac18fd2ea63368a,SystemUUID:38220EEC-BEFF-4C46-AA05-5A214AAFDC28,BootID:b1b45df0-d925-4bcc-9830-0202df36ebb4,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 21:44:46.929] Nov 5 21:44:46.927: INFO: I1105 21:44:46.929] Logging kubelet events for node k8s-master-97435445-1 I1105 21:44:47.000] Nov 5 21:44:47.000: INFO: I1105 21:44:47.000] Logging pods the kubelet thinks is on node k8s-master-97435445-1 I1105 21:44:47.079] Nov 5 21:44:47.079: INFO: kube-apiserver-k8s-master-97435445-1 started at 2019-11-05 20:07:22 +0000 UTC (0+1 container statuses recorded) I1105 21:44:47.079] Nov 5 21:44:47.079: INFO: Container kube-apiserver ready: true, restart count 0 I1105 21:44:47.079] Nov 5 21:44:47.079: INFO: kube-controller-manager-k8s-master-97435445-1 started at 2019-11-05 20:07:22 +0000 UTC (0+1 container statuses recorded) I1105 21:44:47.079] Nov 5 21:44:47.079: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 21:44:47.080] Nov 5 21:44:47.079: INFO: kube-scheduler-k8s-master-97435445-1 started at 2019-11-05 20:07:22 +0000 UTC (0+1 container statuses recorded) I1105 21:44:47.080] Nov 5 21:44:47.079: INFO: Container kube-scheduler ready: true, restart count 0 I1105 21:44:47.080] Nov 5 21:44:47.079: INFO: azure-ip-masq-agent-n29bm started at 2019-11-05 20:07:59 +0000 UTC (0+1 container statuses recorded) I1105 21:44:47.080] Nov 5 21:44:47.079: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 21:44:47.080] Nov 5 21:44:47.079: INFO: kube-proxy-h4s2c started at 2019-11-05 20:08:00 +0000 UTC (0+1 container statuses recorded) I1105 21:44:47.080] Nov 5 21:44:47.079: INFO: Container kube-proxy ready: true, restart count 0 I1105 21:44:47.081] Nov 5 21:44:47.079: INFO: cloud-controller-manager-k8s-master-97435445-1 started at 2019-11-05 20:07:22 +0000 UTC (0+1 container statuses recorded) I1105 21:44:47.081] Nov 5 21:44:47.079: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 21:44:47.081] Nov 5 21:44:47.079: INFO: kube-addon-manager-k8s-master-97435445-1 started at 2019-11-05 20:07:22 +0000 UTC (0+1 container statuses recorded) I1105 21:44:47.081] Nov 5 21:44:47.079: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 21:44:47.329] Nov 5 21:44:47.328: INFO: I1105 21:44:47.329] Latency metrics for node k8s-master-97435445-1 I1105 21:44:47.329] Nov 5 21:44:47.328: INFO: I1105 21:44:47.329] Logging node info for node k8s-master-97435445-2 I1105 21:44:47.404] Nov 5 21:44:47.402: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-97435445-2,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-97435445-2,UID:fa55c895-1a93-4743-885e-03fce84b5332,ResourceVersion:18158,Generation:0,CreationTimestamp:2019-11-05 20:09:19 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 0,kubernetes.azure.com/cluster: kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-97435445-2,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.5.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b/providers/Microsoft.Compute/virtualMachines/k8s-master-97435445-2,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 20:09:19 +0000 UTC 2019-11-05 20:09:19 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 21:43:55 +0000 UTC 2019-11-05 20:09:12 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 21:43:55 +0000 UTC 2019-11-05 20:09:12 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 21:43:55 +0000 UTC 2019-11-05 20:09:12 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 21:43:55 +0000 UTC 2019-11-05 20:09:19 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.255.7} {Hostname k8s-master-97435445-2}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:d19f6709138f496da43c095cec7b2360,SystemUUID:10116777-A8D7-CB4A-A8B7-7EBCF2F23B62,BootID:50cd14a9-5497-4972-bf60-cf59a0cbcb5f,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 21:44:47.405] Nov 5 21:44:47.402: INFO: I1105 21:44:47.405] Logging kubelet events for node k8s-master-97435445-2 I1105 21:44:47.476] Nov 5 21:44:47.475: INFO: I1105 21:44:47.476] Logging pods the kubelet thinks is on node k8s-master-97435445-2 I1105 21:44:47.553] Nov 5 21:44:47.553: INFO: cloud-controller-manager-k8s-master-97435445-2 started at 2019-11-05 20:09:05 +0000 UTC (0+1 container statuses recorded) I1105 21:44:47.553] Nov 5 21:44:47.553: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 21:44:47.553] Nov 5 21:44:47.553: INFO: kube-addon-manager-k8s-master-97435445-2 started at 2019-11-05 20:09:05 +0000 UTC (0+1 container statuses recorded) I1105 21:44:47.554] Nov 5 21:44:47.553: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 21:44:47.554] Nov 5 21:44:47.553: INFO: azure-ip-masq-agent-sq7p5 started at 2019-11-05 20:09:19 +0000 UTC (0+1 container statuses recorded) I1105 21:44:47.554] Nov 5 21:44:47.553: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 21:44:47.554] Nov 5 21:44:47.553: INFO: kube-proxy-pz426 started at 2019-11-05 20:09:19 +0000 UTC (0+1 container statuses recorded) I1105 21:44:47.554] Nov 5 21:44:47.553: INFO: Container kube-proxy ready: true, restart count 0 I1105 21:44:47.555] Nov 5 21:44:47.553: INFO: kube-apiserver-k8s-master-97435445-2 started at 2019-11-05 20:09:05 +0000 UTC (0+1 container statuses recorded) I1105 21:44:47.555] Nov 5 21:44:47.553: INFO: Container kube-apiserver ready: true, restart count 0 I1105 21:44:47.555] Nov 5 21:44:47.553: INFO: kube-controller-manager-k8s-master-97435445-2 started at 2019-11-05 20:09:05 +0000 UTC (0+1 container statuses recorded) I1105 21:44:47.555] Nov 5 21:44:47.553: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 21:44:47.555] Nov 5 21:44:47.553: INFO: kube-scheduler-k8s-master-97435445-2 started at 2019-11-05 20:09:05 +0000 UTC (0+1 container statuses recorded) I1105 21:44:47.556] Nov 5 21:44:47.553: INFO: Container kube-scheduler ready: true, restart count 0 I1105 21:44:47.794] Nov 5 21:44:47.794: INFO: I1105 21:44:47.794] Latency metrics for node k8s-master-97435445-2 I1105 21:44:47.794] Nov 5 21:44:47.794: INFO: I1105 21:44:47.795] Logging node info for node k8s-master-97435445-3 I1105 21:44:47.872] Nov 5 21:44:47.868: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-97435445-3,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-97435445-3,UID:8ffd1d0d-4487-4680-a2f6-e724387be637,ResourceVersion:18152,Generation:0,CreationTimestamp:2019-11-05 20:09:21 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 1,kubernetes.azure.com/cluster: kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-97435445-3,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.6.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b/providers/Microsoft.Compute/virtualMachines/k8s-master-97435445-3,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 20:09:39 +0000 UTC 2019-11-05 20:09:39 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 21:43:54 +0000 UTC 2019-11-05 20:09:18 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 21:43:54 +0000 UTC 2019-11-05 20:09:18 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 21:43:54 +0000 UTC 2019-11-05 20:09:18 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 21:43:54 +0000 UTC 2019-11-05 20:09:21 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.255.8} {Hostname k8s-master-97435445-3}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:266064173bfb4297898c6c25769e8b20,SystemUUID:E6E44241-5015-7740-9A81-A668A8427206,BootID:579d658c-b0ad-47f1-a993-1ccea2b55789,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 21:44:47.872] Nov 5 21:44:47.869: INFO: I1105 21:44:47.872] Logging kubelet events for node k8s-master-97435445-3 I1105 21:44:47.943] Nov 5 21:44:47.943: INFO: I1105 21:44:47.943] Logging pods the kubelet thinks is on node k8s-master-97435445-3 I1105 21:44:48.024] Nov 5 21:44:48.024: INFO: cloud-controller-manager-k8s-master-97435445-3 started at 2019-11-05 20:09:04 +0000 UTC (0+1 container statuses recorded) I1105 21:44:48.024] Nov 5 21:44:48.024: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 21:44:48.024] Nov 5 21:44:48.024: INFO: kube-addon-manager-k8s-master-97435445-3 started at 2019-11-05 20:09:05 +0000 UTC (0+1 container statuses recorded) I1105 21:44:48.024] Nov 5 21:44:48.024: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 21:44:48.025] Nov 5 21:44:48.024: INFO: kube-apiserver-k8s-master-97435445-3 started at 2019-11-05 20:09:05 +0000 UTC (0+1 container statuses recorded) I1105 21:44:48.025] Nov 5 21:44:48.024: INFO: Container kube-apiserver ready: true, restart count 0 I1105 21:44:48.025] Nov 5 21:44:48.024: INFO: kube-controller-manager-k8s-master-97435445-3 started at 2019-11-05 20:09:05 +0000 UTC (0+1 container statuses recorded) I1105 21:44:48.025] Nov 5 21:44:48.024: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 21:44:48.025] Nov 5 21:44:48.024: INFO: kube-scheduler-k8s-master-97435445-3 started at 2019-11-05 20:09:05 +0000 UTC (0+1 container statuses recorded) I1105 21:44:48.025] Nov 5 21:44:48.024: INFO: Container kube-scheduler ready: true, restart count 0 I1105 21:44:48.026] Nov 5 21:44:48.024: INFO: azure-ip-masq-agent-747cl started at 2019-11-05 20:09:21 +0000 UTC (0+1 container statuses recorded) I1105 21:44:48.026] Nov 5 21:44:48.024: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 21:44:48.026] Nov 5 21:44:48.024: INFO: kube-proxy-k45g8 started at 2019-11-05 20:09:21 +0000 UTC (0+1 container statuses recorded) I1105 21:44:48.026] Nov 5 21:44:48.024: INFO: Container kube-proxy ready: true, restart count 0 I1105 21:44:48.268] Nov 5 21:44:48.268: INFO: I1105 21:44:48.268] Latency metrics for node k8s-master-97435445-3 I1105 21:44:48.268] Nov 5 21:44:48.268: INFO: I1105 21:44:48.269] Logging node info for node k8s-master-97435445-4 I1105 21:44:48.344] Nov 5 21:44:48.341: INFO: Node Info: &Node{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:k8s-master-97435445-4,GenerateName:,Namespace:,SelfLink:/api/v1/nodes/k8s-master-97435445-4,UID:6d547b41-3f92-4cc4-a2d7-dda2071776a2,ResourceVersion:18196,Generation:0,CreationTimestamp:2019-11-05 20:08:36 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: Standard_DS2_v2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: eastus2,failure-domain.beta.kubernetes.io/zone: 1,kubernetes.azure.com/cluster: kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b,kubernetes.azure.com/role: master,kubernetes.io/arch: amd64,kubernetes.io/hostname: k8s-master-97435445-4,kubernetes.io/os: linux,kubernetes.io/role: master,node-role.kubernetes.io/master: ,},Annotations:map[string]string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:NodeSpec{PodCIDR:10.244.3.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/940f88ce-a64b-4e73-a258-9931349b9789/resourceGroups/kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b/providers/Microsoft.Compute/virtualMachines/k8s-master-97435445-4,Unschedulable:false,Taints:[{node-role.kubernetes.io/master true NoSchedule }],ConfigSource:nil,},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {} BinarySI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{7284363264 0} {} 7113636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {} 0 DecimalSI},hugepages-2Mi: {{0 0} {} 0 DecimalSI},memory: {{6497931264 0} {} 6345636Ki BinarySI},pods: {{110 0} {} 110 DecimalSI},},Phase:,Conditions:[{NetworkUnavailable False 2019-11-05 20:08:59 +0000 UTC 2019-11-05 20:08:59 +0000 UTC RouteCreated RouteController created a route} {MemoryPressure False 2019-11-05 21:44:09 +0000 UTC 2019-11-05 20:08:29 +0000 UTC KubeletHasSufficientMemory kubelet has sufficient memory available} {DiskPressure False 2019-11-05 21:44:09 +0000 UTC 2019-11-05 20:08:29 +0000 UTC KubeletHasNoDiskPressure kubelet has no disk pressure} {PIDPressure False 2019-11-05 21:44:09 +0000 UTC 2019-11-05 20:08:29 +0000 UTC KubeletHasSufficientPID kubelet has sufficient PID available} {Ready True 2019-11-05 21:44:09 +0000 UTC 2019-11-05 20:08:36 +0000 UTC KubeletReady kubelet is posting ready status. AppArmor enabled}],Addresses:[{InternalIP 10.240.255.9} {Hostname k8s-master-97435445-4}],DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:56b393c61eb34ba49ebda5279eeedb30,SystemUUID:FD239C23-618C-0049-8734-9AF3FDC9E3C1,BootID:f342ea44-e719-4513-807d-5def7ac606e2,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.15.0,KubeProxyVersion:v1.15.0,OperatingSystem:linux,Architecture:amd64,},Images:[{[gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:7f66ed6bd11edd309dad15db1dc906f89035cc9d9ae705c65c98264c775b9a8f gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.15.0] 593352254} {[gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64@sha256:d2397a13afbf962153355d029a62c7b7175c6c525faadce790671280475f5c02 gcrio.azureedge.net/google_containers/cloud-controller-manager-amd64:v1.15.0] 142865982} {[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1] 83076194} {[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0] 50144412} {[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1] 742472}],VolumesInUse:[],VolumesAttached:[],Config:nil,},} I1105 21:44:48.344] Nov 5 21:44:48.342: INFO: I1105 21:44:48.344] Logging kubelet events for node k8s-master-97435445-4 I1105 21:44:48.415] Nov 5 21:44:48.415: INFO: I1105 21:44:48.416] Logging pods the kubelet thinks is on node k8s-master-97435445-4 I1105 21:44:48.494] Nov 5 21:44:48.493: INFO: kube-apiserver-k8s-master-97435445-4 started at 2019-11-05 20:08:23 +0000 UTC (0+1 container statuses recorded) I1105 21:44:48.494] Nov 5 21:44:48.493: INFO: Container kube-apiserver ready: true, restart count 0 I1105 21:44:48.494] Nov 5 21:44:48.493: INFO: kube-controller-manager-k8s-master-97435445-4 started at 2019-11-05 20:08:23 +0000 UTC (0+1 container statuses recorded) I1105 21:44:48.494] Nov 5 21:44:48.493: INFO: Container kube-controller-manager ready: true, restart count 0 I1105 21:44:48.495] Nov 5 21:44:48.493: INFO: kube-scheduler-k8s-master-97435445-4 started at 2019-11-05 20:08:23 +0000 UTC (0+1 container statuses recorded) I1105 21:44:48.495] Nov 5 21:44:48.493: INFO: Container kube-scheduler ready: true, restart count 0 I1105 21:44:48.495] Nov 5 21:44:48.493: INFO: kube-proxy-vfbfr started at 2019-11-05 20:08:36 +0000 UTC (0+1 container statuses recorded) I1105 21:44:48.495] Nov 5 21:44:48.493: INFO: Container kube-proxy ready: true, restart count 0 I1105 21:44:48.495] Nov 5 21:44:48.493: INFO: azure-ip-masq-agent-kmkll started at 2019-11-05 20:08:36 +0000 UTC (0+1 container statuses recorded) I1105 21:44:48.495] Nov 5 21:44:48.493: INFO: Container azure-ip-masq-agent ready: true, restart count 0 I1105 21:44:48.495] Nov 5 21:44:48.493: INFO: cloud-controller-manager-k8s-master-97435445-4 started at 2019-11-05 20:08:46 +0000 UTC (0+1 container statuses recorded) I1105 21:44:48.496] Nov 5 21:44:48.493: INFO: Container cloud-controller-manager ready: true, restart count 0 I1105 21:44:48.496] Nov 5 21:44:48.493: INFO: kube-addon-manager-k8s-master-97435445-4 started at 2019-11-05 20:08:23 +0000 UTC (0+1 container statuses recorded) I1105 21:44:48.496] Nov 5 21:44:48.493: INFO: Container kube-addon-manager ready: true, restart count 0 I1105 21:44:48.728] Nov 5 21:44:48.728: INFO: I1105 21:44:48.728] Latency metrics for node k8s-master-97435445-4 I1105 21:44:48.728] Nov 5 21:44:48.728: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready I1105 21:44:48.802] STEP: Destroying namespace "azuredisk-8482" for this suite. I1105 21:45:03.032] Nov 5 21:45:03.032: INFO: Waiting up to 30s for server preferred namespaced resources to be successfully discovered I1105 21:45:05.856] Nov 5 21:45:05.856: INFO: namespace azuredisk-8482 deletion completed in 17.053919093s I1105 21:45:05.856] I1105 21:45:05.857] • Failure [322.679 seconds] I1105 21:45:05.857] [azuredisk-csi-e2e] Dynamic Provisioning I1105 21:45:05.857] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:33 I1105 21:45:05.857] [multi-az] I1105 21:45:05.857] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:40 I1105 21:45:05.857] should create a volume on demand and mount it as readOnly in a pod [It] I1105 21:45:05.857] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning.go:126 I1105 21:45:05.858] I1105 21:45:05.858] Unexpected error: I1105 21:45:05.858] <*errors.errorString | 0xc0007f2c10>: { I1105 21:45:05.858] s: "PersistentVolumeClaims [pvc-ckvm7] not all in phase Bound within 5m0s", I1105 21:45:05.858] } I1105 21:45:05.858] PersistentVolumeClaims [pvc-ckvm7] not all in phase Bound within 5m0s I1105 21:45:05.858] occurred I1105 21:45:05.859] I1105 21:45:05.859] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/testsuites/testsuites.go:278 I1105 21:45:05.859] ------------------------------ I1105 21:45:05.859] 2019/11/05 21:45:05 Uninstalling Azure Disk CSI Driver... I1105 21:45:05.859] make[1]: Entering directory '/go/src/sigs.k8s.io/azuredisk-csi-driver' I1105 21:45:05.862] helm delete --purge azuredisk-csi-driver I1105 21:45:12.920] release "azuredisk-csi-driver" deleted I1105 21:45:12.922] make[1]: Leaving directory '/go/src/sigs.k8s.io/azuredisk-csi-driver' I1105 21:45:12.922] 2019/11/05 21:45:12 Azure Disk CSI Driver uninstalled I1105 21:45:12.922] I1105 21:45:12.923] I1105 21:45:12.923] Summarizing 16 Failures: I1105 21:45:12.923] I1105 21:45:12.923] [Fail] [azuredisk-csi-e2e] Dynamic Provisioning [multi-az] [It] should create a volume on demand I1105 21:45:12.923] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/testsuites/testsuites.go:278 I1105 21:45:12.923] I1105 21:45:12.923] [Fail] [azuredisk-csi-e2e] Dynamic Provisioning [single-az] [It] should create a raw block volume on demand I1105 21:45:12.924] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/testsuites/testsuites.go:278 I1105 21:45:12.924] I1105 21:45:12.924] [Fail] [azuredisk-csi-e2e] Dynamic Provisioning [multi-az] [It] [env] should retain PV with reclaimPolicy "Retain" I1105 21:45:12.924] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/testsuites/testsuites.go:278 I1105 21:45:12.924] I1105 21:45:12.924] [Fail] [azuredisk-csi-e2e] Dynamic Provisioning [single-az] [It] should create multiple PV objects, bind to PVCs and attach all to different pods on the same node I1105 21:45:12.924] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/testsuites/testsuites.go:278 I1105 21:45:12.925] I1105 21:45:12.925] [Fail] [azuredisk-csi-e2e] Dynamic Provisioning [multi-az] [It] should create a deployment object, write and read to it, delete the pod and write and read to it again I1105 21:45:12.925] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/testsuites/testsuites.go:278 I1105 21:45:12.925] I1105 21:45:12.925] [Panic!] [azuredisk-csi-e2e] [single-az] Pre-Provisioned [It] [env] should use a pre-provisioned volume and retain PV with reclaimPolicy "Retain" I1105 21:45:12.925] /usr/local/go/src/runtime/panic.go:82 I1105 21:45:12.925] I1105 21:45:12.926] [Fail] [azuredisk-csi-e2e] Dynamic Provisioning [multi-az] [It] should create multiple PV objects, bind to PVCs and attach all to different pods on the same node I1105 21:45:12.926] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/testsuites/testsuites.go:278 I1105 21:45:12.926] I1105 21:45:12.926] [Fail] [azuredisk-csi-e2e] Dynamic Provisioning [single-az] [It] should create a deployment object, write and read to it, delete the pod and write and read to it again I1105 21:45:12.926] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/testsuites/testsuites.go:278 I1105 21:45:12.926] I1105 21:45:12.927] [Fail] [azuredisk-csi-e2e] Dynamic Provisioning [multi-az] [It] should create a raw block volume on demand I1105 21:45:12.927] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/testsuites/testsuites.go:278 I1105 21:45:12.927] I1105 21:45:12.927] [Fail] [azuredisk-csi-e2e] Dynamic Provisioning [single-az] [It] should create a volume on demand and mount it as readOnly in a pod I1105 21:45:12.927] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/testsuites/testsuites.go:278 I1105 21:45:12.927] I1105 21:45:12.928] [Fail] [azuredisk-csi-e2e] Dynamic Provisioning [single-az] [It] [env] should retain PV with reclaimPolicy "Retain" I1105 21:45:12.928] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/testsuites/testsuites.go:278 I1105 21:45:12.928] I1105 21:45:12.928] [Fail] [azuredisk-csi-e2e] Dynamic Provisioning [single-az] [It] should create a volume on demand I1105 21:45:12.928] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/testsuites/testsuites.go:278 I1105 21:45:12.929] I1105 21:45:12.929] [Fail] [azuredisk-csi-e2e] [single-az] Pre-Provisioned [It] [env] should use a pre-provisioned volume and mount it as readOnly in a pod I1105 21:45:12.929] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/testsuites/testsuites.go:353 I1105 21:45:12.929] I1105 21:45:12.929] [Fail] [azuredisk-csi-e2e] Dynamic Provisioning [single-az] [It] should delete PV with reclaimPolicy "Delete" I1105 21:45:12.930] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/testsuites/testsuites.go:278 I1105 21:45:12.930] I1105 21:45:12.930] [Fail] [azuredisk-csi-e2e] Dynamic Provisioning [multi-az] [It] should delete PV with reclaimPolicy "Delete" I1105 21:45:12.930] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/testsuites/testsuites.go:278 I1105 21:45:12.930] I1105 21:45:12.930] [Fail] [azuredisk-csi-e2e] Dynamic Provisioning [multi-az] [It] should create a volume on demand and mount it as readOnly in a pod I1105 21:45:12.931] /go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/testsuites/testsuites.go:278 I1105 21:45:12.931] I1105 21:45:12.931] Ran 16 of 16 Specs in 5444.012 seconds I1105 21:45:12.931] FAIL! -- 0 Passed | 16 Failed | 0 Pending | 0 Skipped I1105 21:45:12.931] --- FAIL: TestE2E (5444.01s) I1105 21:45:12.931] FAIL I1105 21:45:12.931] FAIL sigs.k8s.io/azuredisk-csi-driver/test/e2e 5444.141s W1105 21:45:13.147] make: *** [Makefile:59: e2e-test] Error 1 W1105 21:45:13.147] 2019/11/05 21:45:13 process.go:155: Step 'make e2e-test' finished in 1h33m35.936241521s W1105 21:45:13.148] 2019/11/05 21:45:13 azure.go:912: Skippng DumpClusterLogs due to CCM not being enabled. W1105 21:45:13.148] 2019/11/05 21:45:13 azure.go:906: Deleting resource group: kubetest-bbefcda2-0003-11ea-9fa5-0242359b656b. W1105 21:56:08.176] 2019/11/05 21:56:08 process.go:96: Saved XML output to /workspace/_artifacts/junit_runner.xml. W1105 21:56:08.176] 2019/11/05 21:56:08 process.go:153: Running: bash -c . hack/lib/version.sh && KUBE_ROOT=. kube::version::get_version_vars && echo "${KUBE_GIT_VERSION-}" W1105 21:56:08.814] 2019/11/05 21:56:08 process.go:155: Step 'bash -c . hack/lib/version.sh && KUBE_ROOT=. kube::version::get_version_vars && echo "${KUBE_GIT_VERSION-}"' finished in 637.738541ms W1105 21:56:08.814] 2019/11/05 21:56:08 main.go:319: Something went wrong: encountered 1 errors: [error during make e2e-test: exit status 2] W1105 21:56:08.816] Traceback (most recent call last): W1105 21:56:08.817] File "/workspace/./test-infra/jenkins/../scenarios/kubernetes_e2e.py", line 778, in W1105 21:56:08.817] main(parse_args()) W1105 21:56:08.817] File "/workspace/./test-infra/jenkins/../scenarios/kubernetes_e2e.py", line 626, in main W1105 21:56:08.817] mode.start(runner_args) W1105 21:56:08.818] File "/workspace/./test-infra/jenkins/../scenarios/kubernetes_e2e.py", line 262, in start W1105 21:56:08.818] check_env(env, self.command, *args) W1105 21:56:08.818] File "/workspace/./test-infra/jenkins/../scenarios/kubernetes_e2e.py", line 111, in check_env W1105 21:56:08.818] subprocess.check_call(cmd, env=env) W1105 21:56:08.818] File "/usr/lib/python2.7/subprocess.py", line 190, in check_call W1105 21:56:08.818] raise CalledProcessError(retcode, cmd) W1105 21:56:08.819] subprocess.CalledProcessError: Command '('kubetest', '--dump=/workspace/_artifacts', '--gcp-service-account=/etc/service-account/service-account.json', '--build=bazel', '--up', '--down', '--test', '--deployment=acsengine', '--provider=skeleton', '--cluster=bootstrap-e2e', '--gcp-network=bootstrap-e2e', '--ginkgo-parallel=1', '--acsengine-agentpoolcount=2', '--acsengine-admin-username=azureuser', '--acsengine-creds=/etc/azure-cred/credentials', '--acsengine-orchestratorRelease=1.16', '--acsengine-mastervmsize=Standard_DS2_v2', '--acsengine-agentvmsize=Standard_D4s_v3', '--acsengine-location=eastus2', '--acsengine-public-key=/etc/azure-ssh/azure-ssh-pub', '--acsengine-template-url=https://raw.githubusercontent.com/kubernetes-sigs/cloud-provider-azure/master/tests/k8s-azure/manifest/linux-vmss-multi-zones.json', '--acsengine-download-url=https://github.com/Azure/aks-engine/releases/download/v0.39.1/aks-engine-v0.39.1-linux-amd64.tar.gz', '--test-azure-disk-csi-driver=True', '--test_args=--ginkgo.focus=\\[multi-az\\]', '--timeout=420m')' returned non-zero exit status 1 E1105 21:56:08.827] Command failed I1105 21:56:08.827] process 683 exited with code 1 after 138.4m E1105 21:56:08.827] FAIL: chewong-pull-azuredisk-csi-driver-e2e-multi-az I1105 21:56:08.827] Call: gcloud auth activate-service-account --key-file=/etc/service-account/service-account.json W1105 21:56:09.444] Activated service account credentials for: [kubernetes-upstream@kubernetes-upstream.iam.gserviceaccount.com] I1105 21:56:09.477] process 13994 exited with code 0 after 0.0m I1105 21:56:09.477] Call: gcloud config get-value account I1105 21:56:09.730] process 14006 exited with code 0 after 0.0m I1105 21:56:09.730] Will upload results to gs://kubernetes-upstream/pr-logs using kubernetes-upstream@kubernetes-upstream.iam.gserviceaccount.com I1105 21:56:09.730] Upload result and artifacts... I1105 21:56:09.730] Gubernator results at https://gubernator.k8s.io/build/kubernetes-upstream/pr-logs/pull/sigs.k8s.io_azuredisk-csi-driver/180/chewong-pull-azuredisk-csi-driver-e2e-multi-az/1191801407074734080 I1105 21:56:09.730] Call: gsutil ls gs://kubernetes-upstream/pr-logs/pull/sigs.k8s.io_azuredisk-csi-driver/180/chewong-pull-azuredisk-csi-driver-e2e-multi-az/1191801407074734080/artifacts W1105 21:56:10.645] CommandException: One or more URLs matched no objects. E1105 21:56:10.732] Command failed I1105 21:56:10.732] process 14018 exited with code 1 after 0.0m W1105 21:56:10.732] Remote dir gs://kubernetes-upstream/pr-logs/pull/sigs.k8s.io_azuredisk-csi-driver/180/chewong-pull-azuredisk-csi-driver-e2e-multi-az/1191801407074734080/artifacts not exist yet I1105 21:56:10.732] Call: gsutil -m -q -o GSUtil:use_magicfile=True cp -r -c -z log,txt,xml /workspace/_artifacts gs://kubernetes-upstream/pr-logs/pull/sigs.k8s.io_azuredisk-csi-driver/180/chewong-pull-azuredisk-csi-driver-e2e-multi-az/1191801407074734080/artifacts I1105 21:56:12.158] process 14162 exited with code 0 after 0.0m I1105 21:56:12.158] Call: git rev-parse HEAD I1105 21:56:12.161] process 14691 exited with code 0 after 0.0m I1105 21:56:12.162] Call: git rev-parse HEAD I1105 21:56:12.165] process 14692 exited with code 0 after 0.0m I1105 21:56:12.166] Call: gsutil stat gs://kubernetes-upstream/pr-logs/directory/chewong-pull-azuredisk-csi-driver-e2e-multi-az/jobResultsCache.json I1105 21:56:12.971] process 14693 exited with code 0 after 0.0m I1105 21:56:12.971] Call: gsutil -q cat 'gs://kubernetes-upstream/pr-logs/directory/chewong-pull-azuredisk-csi-driver-e2e-multi-az/jobResultsCache.json#1572986279255921' I1105 21:56:14.306] process 14837 exited with code 0 after 0.0m I1105 21:56:14.307] Call: gsutil -q -h Content-Type:application/json -h x-goog-if-generation-match:1572986279255921 cp /tmp/gsutil_LyqP3H gs://kubernetes-upstream/pr-logs/directory/chewong-pull-azuredisk-csi-driver-e2e-multi-az/jobResultsCache.json I1105 21:56:15.509] process 14983 exited with code 0 after 0.0m I1105 21:56:15.509] Call: gsutil stat gs://kubernetes-upstream/pr-logs/pull/sigs.k8s.io_azuredisk-csi-driver/180/chewong-pull-azuredisk-csi-driver-e2e-multi-az/jobResultsCache.json I1105 21:56:16.369] process 15163 exited with code 0 after 0.0m I1105 21:56:16.369] Call: gsutil -q cat 'gs://kubernetes-upstream/pr-logs/pull/sigs.k8s.io_azuredisk-csi-driver/180/chewong-pull-azuredisk-csi-driver-e2e-multi-az/jobResultsCache.json#1572986282406578' I1105 21:56:17.403] process 15307 exited with code 0 after 0.0m I1105 21:56:17.405] Call: gsutil -q -h Content-Type:application/json -h x-goog-if-generation-match:1572986282406578 cp /tmp/gsutil_IHEPuG gs://kubernetes-upstream/pr-logs/pull/sigs.k8s.io_azuredisk-csi-driver/180/chewong-pull-azuredisk-csi-driver-e2e-multi-az/jobResultsCache.json I1105 21:56:18.720] process 15453 exited with code 0 after 0.0m I1105 21:56:18.720] Call: gsutil -q -h Content-Type:application/json cp /tmp/gsutil_TWJ5Ve gs://kubernetes-upstream/pr-logs/pull/sigs.k8s.io_azuredisk-csi-driver/180/chewong-pull-azuredisk-csi-driver-e2e-multi-az/1191801407074734080/finished.json I1105 21:56:19.899] process 15633 exited with code 0 after 0.0m I1105 21:56:19.900] Call: gsutil -q -h Content-Type:text/plain -h 'Cache-Control:private, max-age=0, no-transform' cp /tmp/gsutil_Dp9Te1 gs://kubernetes-upstream/pr-logs/pull/sigs.k8s.io_azuredisk-csi-driver/180/chewong-pull-azuredisk-csi-driver-e2e-multi-az/latest-build.txt I1105 21:56:21.123] process 15813 exited with code 0 after 0.0m I1105 21:56:21.124] Call: gsutil -q -h Content-Type:text/plain -h 'Cache-Control:private, max-age=0, no-transform' cp /tmp/gsutil_utf6yC gs://kubernetes-upstream/pr-logs/directory/chewong-pull-azuredisk-csi-driver-e2e-multi-az/latest-build.txt I1105 21:56:22.293] process 15993 exited with code 0 after 0.0m